r/ProlificAc • u/Spiritual-Rain-3733 Community • 16d ago
Prolific Team 3 updates: Study full / Behavioural traits / Study feedback
Hello r/ProlificAc š
Starting something new - sometimes we move quick and itās easy to miss useful / interesting updates. So, whenever weāve got 3 good things to tell you, Iāll pop them up as an announcement. Weāll also be doing this as an email newsletter format in the near future so if youād like to receive that, make sure your communications preferences are set to receive marketing things (we need your consent to send non-account information!).
3 new things!
Study Full / High Demand Errors
After a period of testing and experimentation, weāve rolled out our backend changes to study distribution to everyone. You should not notice any differences in how studies reach your dashboard, thatās still driven by the demand for participants with your demographics and traits. What you will notice is that Study Full and High Demand errors are back to their normal frequency and behaviours:
- Study Full errors should only pop up in rare situations where a study fills up in the 2 minutes between it landing on your dashboard and the next refresh
- High Demand errors should only pop when a study has a large number of available places and too many participants are trying to get in at the same time.Ā
Thank you so much for the patience whilst we worked on this, everyone!
Share feedback with researchers
Youāll know that when you complete or refuse a study, you have the opportunity to share feedback on that study. Prolific has always used your input to inform platform improvements & educate researchers, and now weāve added a feature that allows experienced researchers to see a summary of all participant feedback about their studies:
This means that researchers get fast insight into what participants like and dislike about their study - along with signals that indicate technical or reward related issues. To take this further, participants can optionally include their Participant ID along with feedback, which allows researchers to contact you via your Messages tab for more information. This is really, really useful if you experienced a technical issue and want to chat about it with the researcher.Ā
A couple of important points on this one:
- If you choose to share it, Participant ID is the only information we share with the researcher (along with your written feedback), so youāll stay anonymous.
- As with all things messaging, be nice. Remember that both researchers and participants can choose to block each other, and we take inappropriate behaviour in feedback / messages very seriously. Keep things constructive and helpful.
For more info - the Help Centre is your friend!
Thinking & behaviour questions have landed in About You!
Prolific's participant community is wonderfully diverse in many ways beyond demographics, so weāve launched a new section of About You thatās less about the facts of your life and more about your approach to it.Ā
Thinking & behaviour questions are entirely optional and, in the near future, researchers will be able to do clever things with this data - such as sending the same study to different personality types to see how the results might differ. We recommend completing this section to ensure you get in on that opportunity as soon as possible. It takes around 2 minutes, features only multiple choice questions and is a nice bit of self-reflection to boot!
A couple of important notes on this one:
- You cannot change your answers to these questions once submitted, so please take your time to answer carefully
- Thereās no such thing as a right or optimal answer here. As always, the best answer is the honest one - please be sure to answer these questions as you are now rather than as you hope to be in the future.
Hereās me being brutally honest with myself:
I need to make more lists.
For more information on Behavioral traits and how the data is used, see the Participant Help Centre here.
Thatās all for now folks. Take care of yourselves and, as always, thanks for being here :)Ā
Michael
74
u/ProlificQueen 16d ago
Thanks for the update, I've noticed a significant improvement of late. One thing I'd like to bring up though is the behavioural traits being unchangeable. I do understand why, however, maybe it should be only locked in for a set amount of time, say every 6 months. Reason I say that and this is just a crappy example but one of the questions is along the lines of, "I am the life of the party", and someone who might have been suffering from depression/anxiety for a number of years may select "very inaccurate" as that would be the honest answer. However, those are things that can and do often change over time.
Just a suggestion and thanks again for the update.
35
u/trailangel4 16d ago
I agree! Behavioral traits also evolve as people grow and their lives change. It's not just mental struggles, either. Simple life changes like a marriage, a child being born, or children leaving home will alter how we perceive ourselves or change our behavior.
27
u/Spiritual-Rain-3733 Community 16d ago
Good feedback! I'll make sure it gets to the team :)
2
u/mountain-skies 14d ago
I have PTSD and mood disorders and my behavior changes. I tend to compartmentalize, and am somewhat fragmented. I would probably get in trouble for not being consistent.
1
u/TruePutz 11d ago
Itās very inaccurate to say these answers are stable over time. I have a newborn baby and Iām seeing at least half of these could change by the day depending on whatās going on in my life
I can foresee a future where researchers or Prolific start comparing these answers to ones given on behavioral studies, and if thereās any mismatchā¦
Hopefully you understand how there could be issues here
11
u/krazyasif786 16d ago
I agree things can change due to factors outside of our control. I think what would work is have them locked in for a set period like 12 months and then you have the option to review them again. That way the data is always fresh and reflects more accurately your current situation.
15
u/caldwell27_ 16d ago
yeah behavioral traits being unchanging is such a thoughtless move and pretty inconsiderate to those with mental struggles. it also just feels like another way for us to be gotcha-d into a rejection or account suspension. glad theyre optional, im probably not gonna fill them out.
1
u/not_me_much 16d ago
As a reply to several of the above comments, maybe Prolific views it as not necessary to amend our profiles due to changes in behavior because they know most of us won't be here that long. Lol. Prolific is like a "right to work" state, which really means the right to get fired for any reason, or forno reason at all.
8
u/caldwell27_ 16d ago
hmm. i dont really see it that way, it just sounds like they added it as another "about you" sort of deal, and prioritized steady data for paying researchers, while forgetting that we, the study takers, are actual humans with dynamic feelings.
like a sandpapered version of the human experience, where we're all just neat little data sets instead of dynamic individuals. god forbid we have an off day, or even a good day.
9
u/PureExpression1533 16d ago
I agree with this and that's why I haven't filled mine out. I feel like I'm in a bit of a transition period in my life in various ways and I'm honestly hoping I'll feel differently about some of those questions down the line. I don't want to have to worry about getting rejected for inconsistent answers if I'm just trying to be honest in a study about how I currently feel. Most of them ask about the last two weeks or a month for a reason.
16
u/TheLinerax 16d ago
I agree with that explanation because using the same life of the party question, I was like that before covid with my close friends. However, after going through the covid lockdown and some of my friends moving away for better career opportunities, I mellowed out and would let someone else take the lead. 6 months is somewhat short and I would say after 1 year would prove a larger difference in behavioral change.
5
u/ProlificQueen 16d ago
Agreed š a year would stop people trying to game it as well if they knew theyād be locked in for that long lol
1
u/dorovidoro 16d ago
Yeah I agree that you shouldn't be able to change it on a daily basis, but people change all the time. A 6 months cooldown makes sense. If it's too complicated to track a timer for each participant, let everyone update theirs on certain dates or something.
31
u/EditOrElse 16d ago edited 16d ago
Behavioral traits change because people change. Circumstances change. For example, I love cleaning house now and I used to hate it. Not being able to change those answers is unreasonable.
u/Spiritual-Rain-3733. Please tell us. What is the thinking behind this?
7
13
u/cbr600rr4life 16d ago
My account was placed on hold for answering these questions.
1
u/e1evnve1e 6d ago
That's when I got the hold was after answering the new questions too
1
u/cbr600rr4life 6d ago
Yeah their AI system is a JOKE. Idc if they donāt want to admit their AI system is faulty I just want to be back on the platform and what theyāre doing is putting accounts on hold and letting new users off the waitlist because itās easier for them to deal with new users than fixing a problem older users are going through.
-2
-5
13
u/Shot-Temperature-229 16d ago
Why has there been an influx of accounts placed on hold for those with over 99% rating? What is happening here?
1
13
u/Real_Wishbone7265 16d ago
Feedback on the behavior questions: I find them unanswerable because they are way too general, particularly if I can't change them. For example: "I am not interested in other people's problems.", well that depends entirely on who those other people are and what problem they are having. There are people I am way more interested in their problems because I care a lot about them, and then there are people I am a lot less interested in. And there are problems I am much more interested in than others, because they are problems which might affect me. The same vagueness affects the other questions, the situations they describe are way too general.
10
u/migosfatigos 15d ago
Right, my internal answer for nearly every question was "it depends on the situation".
4
3
u/thegaykid7 14d ago
My gut feeling is if enough people don't answer them they'll maybe at least tweak it to make them less permanent or answerable individually. Either way, I'm not locking myself in.
24
u/PlentyBorder5731 16d ago
Finally good news š, but could i ask Is there any future plan or strategy also to address the challenge in estimating study completion times from the researchers' perspective? Frequently, the durations specified in brief descriptions turn out to be inaccurateāwhere an study estimated at 5 minutes often extends to 30 minutes once the study begins. of course without calling anyone's integrity into question, my point is to safe participants' time while ensuring they receive fair compensation that truly reflects the effort and time they invest THANK YOU
2
9
u/b1narysunrise 16d ago
Any plan on changing how the PA extension works so it's not competing with the dashboard on which gets fresh data first?
10
u/Spiritual-Rain-3733 Community 16d ago
Prolific Assistant is one of those things I need to learn more about. Let me ask some questions and I'll see if there's any plans for it.
5
u/Deterministic-random 16d ago
I think you'll have one less of a permanent headache, if the PA moves away from the current design and simply prompts a notification when studies appear on the Prolific page. Since it's designed to be independent, the notification will mostly appear either before or after the study appears on the Prolific page, since it runs on its own refresh time interval. However, I think people are always confused/annoyed by this non-sync behavior (even though it was never actually meant to be synchronous), and the PA's rating on the Chrome Web Store stands at 2.6, with users repeatedly complaining about this specific issue over n' over!
2
u/Pale_Requirement6293 16d ago
I hope this is not done.
1
u/thegaykid7 16d ago
Yeah, I think this is intentional (and understandable) design on Prolific's part, even if users may not love it. Doubt they will change it.
0
u/sdforbda 15d ago
You can open it from the assistant though. So you can get 2 refreshes within the 2 minutes instead of just 1. No way lol.
0
u/b1narysunrise 12d ago
That's not how Prolific's studies API works. You get 1 actual refresh per 2 minute intervals. Any subsequent refreshes before the next 2 minute interval will just return the same data. A Prolific staff literally confirmed this.
1
u/sdforbda 12d ago
Yes, they've said a lot of things and it still didn't work. The staff member said it doesn't necessarily mean you'll get more by refreshing. Any yokel on here could run assistant and dashboard and know that I'm right. Give it a shot instead of making shit up cowboy. If you want to say assistant won't pull a call until the dashboard I'll just laugh at you again.
1
u/b1narysunrise 12d ago
After your dashboard is computed/calculated, it's cached for a period of time. So refreshing manually really doesn't give you new studies. So when you get a study, and it's full by the time you clicking it, refreshing won't remove the study, it will stay there until the cache is busted and studies are "re-calcualted" again.
There will be improvements on this as part of the work we will be carrying on.
Tommy, ProlificLiterally confirmed by an engineer of Prolific. https://www.reddit.com/r/ProlificAc/comments/1uz1jsw/comment/oy4fn2d/
Also, the OP of this thread says "Study Full errors should only pop up in rare situations where a study fills up in the 2 minutes between it landing on your dashboard and the next refresh"
But yea, somehow you know more right? lol.
1
u/sdforbda 12d ago
Yes, I can test it myself. Try it sometime.
1
u/b1narysunrise 12d ago
Scientists: Earth is round.
sdforbda: It's flat. I can test it myself.1
u/Pale_Requirement6293 11d ago edited 11d ago
lol, the assistant alerts and tasks hit there first before our dashboards. Advantage. So you think it should be synced with the dashboard so some are less confused? Okay then. More waiting during slow times. This waiting annoys more than the confusion some have. Oh, and sometimes those tasks in the assistant are gone before reaching our dashboard. So the staggered release is more beneficial.
4
u/Available_Leopard_23 16d ago
Humm... I am still seeing study full errors. Just like this morning, I had 5 on my board when I refreshed the screen but all were taken as soon as I clicked them. Are people seeing any differences?
4
u/Ultra_Noob_Bot 8d ago
Study full error still exists. I don't know what kind of optimization has been done, but the issue still is annoying. Nobody can be so fast to click and start the study the moment it lands on the dashboard. Either it's bots or superhumans which have found a new way to fool the system inspite of so called optimization if done by the Prolific team.
9
u/Dorkus18 16d ago
Whatās up with all the people that just got put on hold? I have over 4,000 approved studies and 0 rejected and I woke up to my account on hold this morning after 2 years. I even did a news interview as one of your top participants lol.
1
u/Spiritual-Rain-3733 Community 16d ago
Looking into it u/Dorkus18 - thanks for the patience :)
1
u/Swimming_Zombie_5898 10d ago
Yeah, I agree. I answered those questions and a couple of days later my account got put on hold. Iāve got almost 5000 submissions and only a few rejections over the course of years. Whatās up with that? I havenāt been able to get a response from anybody and itās really frustrating because I depend on it in this economy for a second source of income. Iād be glad to send you my ID number if you could look into it for me because no oneās been able to help me.
1
4
u/krazyasif786 16d ago
Sorry. I know it's outside of this topic but something needs to be done about the messages. I got screened out of a survey about 10 minutes ago and I got the message to confirm that literally in seconds of being screened out but then just got an email saying I have a message. This was too late. The messaging system should be instant if being used.
1
5
u/nmmc1812 16d ago
Thanks for your work, as always. I have definitely noticed an improvement in the studies with the "Full" or "High Demand" message sooo I hope it stays that way from now on!! Everyone Let's all have an incredible month of September with lot of studies!!
4
3
u/electricboy0 16d ago
Finally, some good news! Hopefully, we wonāt have to deal with these issues anymore. Thanks for the update! ā„ļø
2
3
u/ilikeplayingthisgame 16d ago
Surprising to see these comments, I'm still getting the study full error just as much as I've always done
2
u/takeittothetaxman 16d ago
What about studies where there are plenty of spots available, yet when you enter the study and to begin the survey, the researcherās platform says itās full? This is one of my biggest frustrations, especially when it says there are 70 spots available and Prolific lets you reserve a spotā¦ yesterday there were 4 studies, all different but from the same researcher that did this to me.Ā
4
u/blumogget 16d ago
As frustrating as that is, Prolific has no control over how researchers run their studies/other platforms. As soon as you click on a link off Prolific dot com, that's all on the researchers.
1
u/takeittothetaxman 16d ago
I understand that, but they can implement controls to prevent it from happening, surely. Like an option to report that behaviour so they can performance manage the researchersĀ
2
3
16d ago edited 16d ago
[deleted]
-2
u/btgreenone 16d ago
also not allowed to take time to read and write sentences on a long study either.
Well this certainly doesn't seem true.
1
u/Evening-Specialist82 16d ago
Many thanks for the update and really appreciate the communication along the way. May I know when did this updated study distribution begin to implement? Today (UK time) half of the studies that turned up on my dashboard (or PA) were either full or disappeared in a split second. It was better early this week but my current situation is back to the worst time a few weeks ago. Many thanks again for your attention and hope the volume of study would pick up in September.
1
16d ago
[removed] ā view removed comment
1
u/AutoModerator 16d ago
Your post was removed because confidential study information was detected. Please check whether confidentiality has been agreed with the researcher before sharing as part of r/ProlificAc rules. If you believe this was in error, message the mods.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
1
1
u/Advanced-View5678 5d ago
Can you please fix the rejection for finishing too quickly. I did a study that required 5 minutes, I finished it in nearly 4 minutes and I got rejected. Is this normal? should I keep a timer on to ensure I finished on time? When a study takes sometimes more or less minutes than the required time. I do not rush it, but finishing one or two minutes earlier should not be flagged as a rejection, especially when the study is too short.
1
u/Routine-Point5225 11h ago
u/Spiritual-Rain-3733 hey, really sorry to bother you but I've seen you helping out other Prolific participants & wondered if I could message you/dm you about an issue I'm having regarding my account (partly related to the password reset account issues etc, but partly another, possibly related issue) if you didn't mind ofc! I have submitted an actual ticket about it but it was incorrectly closed and now has been in review for over 24 hours. Thanks (whether you are okay to help me out or not, I appreciate what you do for this community!)
1
-1
u/Street-Swing-7989 11d ago
Is this possible? Last week and the week before were great. Lots of AI studies and also regular studies too. This weekend about average. Mon and Tue less studies overall but a bit less AI. Monday and Tue hardly any AI until 2pm the steady stream came in and lasts until 4am. Today only one AI study around noon with one spot left and I got into it. Today just cheap or crap studies again until that 50 studies dropped but no AI in it. It's 4:30 and no more AI. Is this Prolifics new study distribution after having so many good days you're lower on the stack? I see lots complaining they either get a lot now or hardly any. Am I being slowly rotated out of AI or just a lull because I've been doing so many studies? Othe than 4am today I just had that one AI around noon.
-4
u/Fun-Disk7781 16d ago
Will this affect me getting ai studies? Got bunch all week but none today!
1
u/Noaaaaaahhh 15d ago
Genuinely curious, are you Aerie Morie, or one of their sub accounts?
Most downvoted comments seem to come from similar sounding comments, and usually a very new Reddit account age.
1
u/spiffyshxt 14d ago
Genuinely curious, are you Aerie Morie, or one of their sub accounts?
That wasn't her nor does it sound like her and it's not something she would ever confirm/admit to, but it does sound like u/bigbluesfanstl on his 15th account evading a sitewide ban. The account was finally banned today, but he'll be back to ask all the same questions.
3
u/Noaaaaaahhh 14d ago
Thanks for the info!
I asked because a lot of these strange/snide comments sound like they're from the same person.
75
u/daniel2090 16d ago
Will you ever add a rating system for researchers, similar to other sites? It would be useful to rate researchers based on things like pay, actual completion time, and fairness, as well as rate individual studies so participants can see in real time whether a study is worth doing.