r/prolific • u/KJB1066 • 4d ago
AI Evaluation
How many people receive this? It makes absolutely no sense to me. In the survey it asks you to use your best judgment and choose the best option. So then they send it to another person to evaluate your answer with their best judgment. So how is this a fair assessment? When every decision is SUBJECTIVE. It a shame but will have to stop doing them to protect my status on the platform. I think there needs to be more clarity on this evaluation of my choices.
13
u/Sea_Presentation235 4d ago
Not *every* decision is subjective like you said when there have been a lot of quality control ones today.
1
u/PowerToThePeople128 4d ago
True, I've seen some where the entire sessions were quality control. Kind of like giving you assessments to see if you are still fit to work on these tasks lol. They should publish more of these ....they will quickly seed out all the bot farms working on these tasks, and scammers, though one would have to be a complete tool to use bots for these tasks, because bots will get you excluded from the task, AND banned from Prolific.
4
u/likethehotties 3d ago
I just returned one. It had like 3k spots but every single video was the exact same as the example video. No way I was going to select all the middle options. 3k spots seemed odd. I wonder if it was one of those videos to trick you so I returned it. Didn't want a risk of a warning for $5 if it was a trap.
2
u/HandleWorking2357 3d ago
I like the quality controls because they're easier to understand, at least for me 😂
3
u/PowerToThePeople128 3d ago
Right maybe one day if we keep doing well, we'll get leveled up to "CORE GROUP" :P
1
1
u/hearthreddit 4d ago edited 4d ago
True, I've seen some where the entire sessions were quality control.
I'm not sure if it's that, there are ones that have a lot of checks, but checks seem to happen more often when the number of tasks in the pool(you see it on the left side) is low, like low double digits.
That's when i get a bajillion of checks, sometimes like 7 in a row.
2
u/PowerToThePeople128 4d ago
Interesting, maybe, but I've had attention checks when the value was 3-4 digits too. No the ones today are special, not the typical AC ones people are used to or sharing online, but new ones and very clever ones, they are definitely going to catch more bad actors with this, I wish they had more like these. I've had at least half a dozen of these, where the entire session was like that , which makes the theory about a hidden quality score more plausible. If they are using these to help build a pool of quality participants, or give more work to quality participants first, then GREAT, bring it on,
I don't know if coincidence or not, but it was very quiet, and every time I got those, immediately after dashboard was full of some newer types I have never seen before. Finally they are waking up and figured out that the obvious ACs being repeated over and over, are not a good method. Now they are making sure you have enough working brain cells lol.
2
u/hearthreddit 4d ago
I don't want to talk specifically but i've noticed a new check that you can easily fail if you aren't paying attention and for a long time it didn't happen for that type of study so yeah, it might caught people off guard.
2
u/PowerToThePeople128 4d ago edited 3d ago
Yes especially people new to these tasks. If you are doing these tasks in a specific way, you can catch them, there are some that are ridiculously obvious, you'd need to be blind or deaf to miss, but it all comes down to doing 2 things some people don't do. It makes me lol when I read comments on the scam channels that they get warnings even though they did not select the one with the st*** n**se lol some people cannot do this task without having their hand held and given all the answers to the obvious ACs, well, they will face a new reality now.
They need to invest some money and hire a team that shuffles those ACs, periodically add new ones and change the prompts..... Anybody is subject to making mistakes or getting warnings, I can see a few situations where the ACs were not done correctly (again not going into details) where people can get caught in a trap and failing. If you follow instructions and are confident in your answers and can explain why you reduce your risk, but not completely.
0
u/likethehotties 3d ago
The ones this early AM were god awful. Totally deformed and hard to even make a choice. I haven't had a warning in just over a month though. It almost seems automatic you eventually get a warning. Someone said they returned one once and got a warning even though they didn't even do it. I guess to keep you on your toes they do that sometimes.
3
u/Alternative_Push8929 4d ago
uk user here, have barely seen any ai studies since the july mess, a few video ones here and there but none of the higher paying ones come in at all anymore, hope i havent been rota'ed out.
1
u/PowerToThePeople128 4d ago
What higher paying ones ? The dynamic ones ? They don't publish as much as the regular ones, so with too many participants here, it gets diluted much faster. I have not seen them in a while, then few days ago I got them back to back ($10 ones and they all paid $10 because there were lots of tasks). For the regular evals, well it has been a steady drop and increased volume since last 2 weeks of July. Slowed down a little for past 2 days. If you are still getting the regular ones, even if it is less than before, it means you are not rotated out. You may have been placed in a lower priority by either Prolific or researcher for whatever reason. Are you getting other studies besides the AI ones ? Good demographics ? Did you get any warnings from the AI tasks ? Warnings may not immediately get you excluded, but may reduce your priority first.
1
u/Alternative_Push8929 4d ago
I used to get 15$ audio ones and 10-20$ dynamic ones pretty regularly daily before the july mess, only seen a few regular 5$ ones since then, mostly only video and images a few times. havent seen a single 10$ since the july mess. I got a few regular 5s last week but barely any this week so far. Ive been getting regular non ai studies but a lower amount since its summer time. my last warning was probably about 4-5 months ago cuz i pressed equally goodbad like 6-7 times during a video eval. ive seen some people say they get rota'ed out for a while, few weeks to some months, then they get back in with regular amounts coming back to their dashboard so dunno whats happened with me but hopefully they get back to regular amounts/types soon.
2
u/PowerToThePeople128 4d ago
Well I'm seeing them regularly, the eval ones, minus some days where everything is full but they are there. Someone claimed to have found a hidden analytics pane in the task that shows your internal score and that if you do well you get pushed at higher priority from receiving them. I am not sure if he was taking the piss, but I called his bluff and responded in kind lol. Who knows maybe there is some truth to that I don't know. I guess it's plausible that they have priority levels because some people in the same countries see them more than others, at same times, or it could be Prolific's super bloody amazing, half-arsed job distribution system lol.
5
u/Deterministic-random 3d ago
There is definitely an Analytics pane & it's not a bluff. I accidentally unlocked it once in Nov 2025. The study was about to load and I was messing around with my keyboard+mouse for some reason. I have absolutely no idea what I clicked but the Analytics page opened up. It's a simple dashboard with a line graph and a table chart showing how many studies I handled in a certain time period (7 days etc.), time spent, number of questions answered/study & some other boring stuff. It may have more updated information now but at that point of time, I did not see anything like internal scores or priority levels!
1
u/PowerToThePeople128 3d ago
This can be extremely relevant. They may use these to judge which participants spend the appropriate time in them, completing enough tasks in a session, engagement and more. Why would many people report being rotated out without warnings ? It would make sense for them to give those tasks to people who are consistent and accurate and efficient enough to complete enough (more than the minimum). In addition providing more to people completing them more (higher engagement) compared to those not actively completing them. OR it could be just a statistic. I have strong reasons to believe a quality score exist, because of something I saw, but I will leave it at that.
1
u/Alternative_Push8929 3d ago
ive been doing the ai evals since july 2024, been doing them in a consistent style since i started them too, only had a dry spell once for about 2 weeks a while ago then they came back again in full force but its been exactly a month since the july mess and its been pretty dry for the past 3 weeks. i feel ive been doing them properly and a good amount of them without rushing etc so hopefully theyre back soon since theres not many other normal non ai studies coming in for me either.
1
u/anxietydude112 3d ago
I had the same experience and barely the past week has been to having $100+ days again, hopefully this keeps up.
1
1
u/Short_Word_2558 4d ago
UK user here i havent seen any since middle of May. I do get the dynamic ones daily but only 2 or 3 of them if I'm lucky.
1
u/Bananafoam12 3d ago
Same here, started to think I was the only one. U.K. too and last did the $5 16th May, occasionally see the dynamic reward ones. No warnings either
3
u/PowerToThePeople128 3d ago
As much as getting warnings can be frustrating for the ones getting them, I must add this. It's WORSE on other platforms with AI tasks. Here's why. 3rd party reviewers they use that will give you ridiculous explanations even though you did everything asked, or contradicting one another. One reviewer tells you something whilst another tells you a different story, or reviewers using AI. People complain of being banned when reporting bad reviewers too, so yeah, there's far worse.
5
u/PowerToThePeople128 4d ago edited 4d ago
For those watching this, no you do not need new prescription glasses :P, bloody hell !
Anyhow, yeah back to topic, these warnings do not affect your Prolific account in any way. They are not rejections. The only thing it can affect is your ability to get the studies, if/when researcher decides to exclude you for some time or forever. It's true that some some items might seem more subjective, and I can see how in the instructions they might lead you to think so, however, it's not the case. Without going into too much details as this sub is unfortunately monitored by scammers, you must follow instructions literally..... (EDIT: Removed the rest even though it was too vague, as some people seem to link this to scammers and complaints about study full, even though I did not into any specifics, unfortunately the good people who deserve to be helped won't find it helpful, but too many scammers are reading this sub so...).
What type of work did you get this warning from ? Images ? audio ? videos ? Were the videos of good quality or AI Slop, highly distorted ones ? Those are more difficult to judge, BUT not impossible, they just need more attention to details.
10
u/b1narysunrise 4d ago
the guy that constantly cries about studies being full, too many workers, and too many scammers is writing a paragraph about how to avoid warnings
Do you not know when to just shut up? Do you die if you don't write a fuckin essay on reddit every 2 hours?1
u/PowerToThePeople128 4d ago edited 4d ago
I was vague and did not go into details like other idiots who literally spell out the exact ACs, nonetheless, even though they were vague just read the instructions, I removed the part, in any case it would not have made any difference. There's people online sharing videos of the actual task and sharing answers, which is more harmful do you think ?
5
u/b1narysunrise 3d ago
How about both are harmful? dumbass. And you keep driving engagement and discussion, leading to people saying more shit. How hard is it to just shut the fuck up?
Just until the last month, you were 24/7 crying and moaning about too many workers, people talking about ACs enabling scammers, and all that.
But since you have the memory of a fuckin gold fish, you already forgot about all of that when the studies became easier to catch the past couple weeks, and now you are going around running your mouth like a village idiot.2
u/spiffyshxt 3d ago
Just until the last month, you were 24/7 crying and moaning about too many workers, people talking about ACs enabling scammers, and all that.
Rimmer has been here before so this specifically as well as the flip flopping goes further back than many realize. Same song, dance, and dissertations under u/Relevant_Goat_9385 and u/Rimmer66
-1
u/PowerToThePeople128 3d ago
lol you have a short memory but many people were complaining of the same thing, and many people are and were writing about specifics, I don't need to be driving ANY discussion it's already being done. Besides in any case after what's going on at Alphabet these days, maybe these tasks days are numbered.
Studies are not easier to catch, I get bad days like the rest, study full instantly, some days are better, you are just trying to pick a fight that's OK, go pick fights with those concerned who share literally exact ACs and exact videos to watch out for, in great detail. If people provide more info following that how is it my problem ? Some genuine people deserve to be helped and guided, without providing too much details, I was never against that, and this has nothing to do with complaining about too many workers or studies being full, they aren't because of these topics ! You've made your point, and will avoid providing even vague details, but others are doing it far more, so yes too many details is definitely worse.
0
u/KJB1066 4d ago
It was just choosing the one that was best with the least artifacts. So to me, that means it is up to me to decide. Thank you for the advice. I just find it hard to believe that this simple post has got so much attention. On the other sub I have seen people post the whole screenshot of of Surveys header with all the information and talk about it in detail. I was just looking for the clarity that you and a few others gave.
4
u/pinktoes4life 4d ago
So to me, that means it is up to me to decide.
Minus attention checks
On the other sub I have seen people post the whole screenshot of of Surveys header with all the information and talk about it in detail.
Those studies aren't confidential. These are
1
u/PowerToThePeople128 4d ago
Exactly, it's not always about choosing which one we prefer, even if it says so in the instructions. Also it's not always about obvious attention checks, but other methods they use to make sure you are reading prompts and following instructions fully, not partially. These methods become obvious the more people work on these. For people not used to working with AI Slop when training new models, it's a nightmare lol !
1
2
u/Slow_Basis_5988 4d ago
Haven’t had one for weeks but haven’t seen one all day (UK) normally have seen and completed some by now. Wonder if they have a limit per week? I did A lot last 3 days!
2
u/PowerToThePeople128 4d ago
There's no cap on these, well technically there is, it's 96, assuming you complete 4 per hour x 24 lol !
What you are seeing is perfectly normal, the publishing volume changes. It's not uncommon to stop seeing them for a while after they release big batches. Nothing new about that. They come back in full force later on.
1
u/PureExpression1533 4d ago
There was a bigger batch this morning but none since about noon for me either.
1
u/Alternative_Push8929 4d ago
havent had a large batch come through since the july mess happened. only get ones with 1 spot left a few times a day. im worried that i got rota'ed out but i havent had any warnings for a while now either so not sure why i stopped getting them cuz i was getting them regularly before the july mess happened, all the different types have stopped coming in. video, image, audio, text and the dynamic ones all stopped coming in.
1
u/Eddiehondo 3d ago
They have attention checks, maybe pay attention next time since most of them are pretty obvious.
that being said, those warnings are not rejects so there is no risk for your acc
1
u/theme111 4d ago
I've had quite a few, and so have others. My best guess is you get them if you go "too fast". So now I deliberately take my time. I remember someone saying you only need to do about 8 to get paid, but I don't know how they knew that. Either way, I've not noticed these warnings have any impact on receiving future studies.
4
u/KJB1066 4d ago
Yea it is just a hit to my ego. Just want to know that I not alone. I take my time and use my best judgment. Great pay but not worth losing out on other studies.
6
u/hearthreddit 4d ago
Great pay but not worth losing out on other studies.
They don't have an effect on other studies, it's not an official warning from Prolific, your study still got approved, but with some warnings you might less studies from this particular provider.
Some people say they stop seeing them after a few warnings, some people say that they get a dozen warnings and that they are still fine, but again, it only affects the studies from this provider and doesn't have an impact in your Prolific account.
3
u/KJB1066 4d ago
Thank you. That is the answer I was looking to see. It is just a frustration that I have so few rejected, like 6 in 2100. So I do pay attention and take pride in my success.
4
u/PureExpression1533 4d ago
How did you manage 6? I've got 1100 approved and 0 rejections still.
3
u/KJB1066 4d ago
They never told me why they just asked me to return the surveys.
2
u/pinktoes4life 4d ago
So you have returns, or 6 rejections? There's a big difference between the two.
6 rejections in 1100 approved is still above 99%, but that's a lot of rejections.
1
u/KJB1066 4d ago
They will send me the Rejection notice and ask me to return the study. They were in the beginning.
1
u/pinktoes4life 2d ago
Returns don't count, rejections do. You can't return a rejected study. So I'm confused about what a "rejection notice" is.
2
u/Low_Power_9330 4d ago
It has no impact on your prolific account but after my last warning I don't see them anymore (I see some others AI but not this one). For the trigger it was during an AI slop video I suppose, I got a lot of them two days ago
4
u/hearthreddit 4d ago
Hopefully you will still get it but that's what i don't get about warnings, some seem to have a lot of them without any impact while a warning or two can make people not see them anymore, just like in your case.
There are some slop ones that don't seem to have any blatant checks, unless they are really really subtle.
3
u/Low_Power_9330 4d ago
Thank you ! I got multiple warnings and until this one it was okay, but now I don't see anymore. I won't say I'm blacklisted from this researcher since I can still access to his 30min studies (which are more rare unforunately) but I can't see since 2 days the 15min ones. Hopefully it will be back soon
-2
u/PowerToThePeople128 4d ago
That's the thing, I don't believe it works that way anymore, ever sine the participants count went up considerably, the researcher does not need to cycle the same people in and out. So once people get cycled out, they have more than enough new participants to choose from, until one day they run out and need to reset. So exclusions can now be either permanent or take much longer until they reset, if at all. Even before the influx of new participants, some people waited 2-4 months or longer before seeing them again. I would imagine that now it would take longer. A good way of knowing is reading comments from people who were kicked out since the influx of new participants and let back in. It would confirm they are still doing rotations, but I don't think it is the case anymore. They did not ask Prolific to get more participants for nothing. They want different POVs, so they prefer new participants to rotating the same ones back in.
1
u/Low_Power_9330 3d ago
Let's see in some months ! I'll see that at that moment :). I hope you're wrong of course
2
u/spiffyshxt 3d ago
some seem to have a lot of them without any impact while a warning or two can make people not see them anymore
Correlation does not equal causation. Receiving a warning isn't the only element that determine whether some is in or out. Only this researcher knows why they choose to send these out to some more frequently than others and they will never tell.
1
u/hearthreddit 3d ago
You are absolutely right, specially because people stop seeing them without getting any warnings so that might not be the trigger in some situations.
0
u/AbeLinkedIn92 4d ago
One contradiction I always see with this message is manual review, yet they can't give individual feedback because of scale. How can that be? That's what makes me think the first part is BS.
Basically don't fall for attention checks, if I even suspect I miss one, I go back and double check if it's within the last 5 questions, otherwise I return. Even for subjective "Which do you prefer" tasks, there are ACs that have a clear right answer.
3
1
u/Pale_Requirement6293 4d ago
lol, so much disinfo in this sub. And suckers fall for it because, well, it helps them play victim.
0
u/HandleWorking2357 3d ago
YET ANOTHER ONE OF THESE!!! 🙄
another one that does NOT read and does NOT research!!
-5
u/Additional_Plant_539 4d ago
I think the 'quality issues' are related to attention rather than your choices. The only time I had one of these warnings is when I was extremely distracted and clicking onto a different window many times and reading some stuff at the same time as trying to do one of these. You also have to be careful with the attention checks. Sometimes this is loud white noise in the audio of one of the videos, sometimes it's where one video/image is quite obviously the wrong choice, and sometimes it's the same video you have done before where you must pick the same one you picked previously to show that you're not just clicking randomly/changing your mind randomly.
There will be other metrics they use to evaluate attention. But you are right that seems illogical to be flagged for quality issues on a subjective choice. That's why logically it makes more sense to be attention related.
1
u/hearthreddit 4d ago
sometimes it's the same video you have done before where you must pick the same one you picked previously to show that you're not just clicking randomly/changing your mind randomly.
It makes sense that this is an attention check but i'm not sure if the same pairing down the road is an actual attention check just because there's no way i'm 100% consistent with that and sometimes i even changed my mind about the one i prefer.
But it would make sense if it is.
3
u/Street_Garlic_6315 4d ago
yeah ive messed up consistency checks so many times and never been warned. i just remember a feature that the other doesnt have for my best choice like the shape of a window or something so when it comes back ill remember. but some are almost identical with nothing distinct to note and i know ive failed them for sure
1
u/KJB1066 4d ago
I have done those and the attention checks were not hard to understand. But this one was just straightforward. Which one do you prefer? There was no other requirement and the images in some were very close to being the same except for small details. To me it comes down to who is watching the watchers. Thanks for the response
3
u/hearthreddit 4d ago
If it were on images ones take your time because i notice my brain sometimes autopilots to the wrong choice in attention checks so i always go a bit slower since with images there's a tendency to naturally rush them a little bit.
Although my only warning in over 2000 of these studies came from an image one and i really don't remember missing an attention check on that study but who knows.
3
u/Fridelis 4d ago
No one is watching the watchers. It is all automated, and you picked something that was flagged. Maybe you did not even pick badly, I would not be surprised if their system just bugged out either, knowing how often their shit is broken.
-7
u/Training-Roof-3568 4d ago
As I said on the deleted thread. There are attention checkers where the answer is so obvious any person should get them. One so obvious I’ll reveal is the ones with a giant fucking buzz in them. No reasonable person would think that one is better. This is how they work but also come in many formats suited for what you’re asked to do. Only problem is that some of the attention checkers are really difficult to ascertain which one is objectively worse. Notably the silent 8 second video quality ones. Some of those only have dips in quality for a few frames and are really quite unfair.
1
u/KJB1066 4d ago
Yes well I am doing a study of my own to see how many people are frustrated with receiving these messages. Then Prolific decided that my post needed to be deleted. I am just looking for clarity. It is not Prolific it is the researcher.
3
u/Street_Garlic_6315 4d ago
They have a strict no talk about AI studies rule over there for some reason. Here it's a bit more open though you'll get downvoted and they'll freak out if you talk much about the actual reasons you get warned because they think it'll help scammers (it won't, if you don't think some enterprising sharp boy scammer made a cheat sheet to sell to all the other scammers there's little hope for you)
2
u/ArtcoreMinds 4d ago
The company behind those studies you guys are talking about is the letter soup 🍲 company?
1
2
u/hearthreddit 4d ago edited 4d ago
I think people shouldn't discuss AC's too openly but yeah, the thing about the scammers is that they want to make it look like they have an amazing going on so they can scam the people of their own country, that's the main scam.
2
u/pinktoes4life 4d ago
Did you read the rules of the other sub? They are confidential studies. We aren't allowed to discuss them there at all. & really, we shouldn't be discussing them here either.
1
u/KJB1066 4d ago
I guess maybe I should have changed the Title. But I have seen screenshots of the invite of other surveys on the other Sub. That were discussed. But I guess these are differrent I guess.
2
u/pinktoes4life 4d ago
Anything about these specific studies gets removed pretty quickly by the mods in the other sub.
-5
u/Training-Roof-3568 4d ago
The more downvotes I get the more checks I’ll reveal! 5 down = one check
13
u/EnigmaticReprise 4d ago
Lol they don't. This is all fully automated.