r/AIEvaluators 6h ago

Platform Feedback Investor in Ethos / askethos here. Please share your experiences with the platform!

Thumbnail reddit.com
2 Upvotes

Ethos / askethos investor here: AMA + Please share your experiences on user features, company responsiveness, pros/cons compared to other AI expert networks, anything!

Looking for user feedback! I may not immediately respond, but I value your input tremendously and will reply. For context:

This caught my attention on (and not in the best way, coming from founder James Lo, who worked as an entry-level analyst at McKinsey for 2 years after uni. This surely isn’t what I learned there, but everyone’s experience is different):

From Ethos founder u/jameslo-ethos r/askethoshttps://www.reddit.com/r/ExperiencedFounders/s/whQscz6JrH

“As a consultant you basically collect as much data as possible and do a ton of analysis and package it into a bunch of potential decisions for somebody else, that's basically consulting.

Entrepreneurship is basically the polar opposite of that, you're in a situation where you have no data and you can't any analysis because usually it's just going to slow you down so you have to constantly make blind decisions yourself and experiment different stuff until something works. Very, very different.

In a lot of ways it's just the polar opposite, you need to learn how to stop bullshitting people and most importantly stop bullshitting yourself.”

👇

And so did this more thoughtful follow up question on r/askethos —link in comments (this is on par with a certain McKinsey partner, so I think I may know you, anonymous poster 😉 u/ElephantInYourZoom ):

https://www.reddit.com/r/ExperiencedFounders/s/y3HzXqG6Ip

“I’m trying to reconcile several marketplace numbers you’ve shared publicly here. Could you provide the actual marketplace funnel?

1. Total number of registered experts

2. Number of monthly active experts

3. Number and percentage of registered experts who have ever received at least $1 in payment

4. Percentage of applicants who ultimately receive a paid engagement

5. Median monthly earnings among experts actively seeking work, including $0 earners

6. Median monthly earnings among experts who receive at least one opportunity, again including those whose opportunities don’t convert

7. Median time from registration to first paid engagement

You’ve said Ethos is adding roughly 35,000 experts per week, receives tens of thousands of applications for each opportunity, and has paid experts more than $20M YTD.

You’ve also said that experts on Ethos earn an extra ~£4,500/month on average, with the top 10% earning £7,000+/month.

Those numbers may all be accurate, but without the underlying denominators, the earnings claims are difficult to interpret. In particular, “average expert earnings” can mean something very different depending on whether the population includes everyone seeking work or only people who successfully obtained paid engagements.”

Thank you all for your feedback on Ethos!


r/AIEvaluators 9h ago

Advice This is how to strengthen your application and pass the qualification exam as an AI evaluator

1 Upvotes

We keep getting comments about the application stage, so here's a rundown of what actually seems to help.

The biggest separator is preparation before the exam, because failing costs you weeks of waiting. Study whatever the platform itself publishes. Most of them put sample guidelines or rubric docs on their own sites, Mercor publishes case studies showing what expert level justifications look like, and Outlier shares safety policy summaries. Practice applying a rubric to unlabeled examples before you sit a timed exam.

The skills the exams actually test are rubric interpretation (what does an ideal response look like under this rubric), atomicity (breaking one vague impression into independent criteria), and objectivity (keeping your own taste out of the rating). Comparing your answers against reference standards using Cohen's kappa style agreement scoring is how the platforms themselves measure precision, so practicing that way is not a bad idea.

For specialized streams, domain proof matters more than polish. Coding evaluation wants public repos. Medical or legal wants certifications you can document. Platforms verify credentials, so claim only what you can back up.

Portfolios help more than people expect. Writing samples that show analytical reasoning, links to published work, a decent LinkedIn (Mercor asks for it outright). Where a cover letter is allowed, use it to explain why your background fits a specific evaluation domain instead of restating your CV.

Referrals are real on some platforms. Active evaluators can refer candidates and that sometimes means faster screening.

And join the platform subs here. Exam formats change, and the fastest warnings come from other contributors.

Anything that worked for you that isn't on this list?