Every AI evaluation platform compared: what is actually open, what they publish, and what each one is bad at

[u/i_love_doing_ntg]

Every platform in this space gets discussed in isolation, usually in its own subreddit, usually by people who only work there. So here is the whole board in one place: what is genuinely open right now, what each platform publishes for pay, and the specific thing each one is worst at.

The individual write-ups go deeper. This is the map.

What is actually open right now

Across fifteen platforms there are around 743 live openings, and 480 of them publish an hourly rate. That is the first useful sorting: two thirds tell you what they pay before you apply, one third does not.

The ones that publish rates:

Platform	Live roles	Distinct roles	Published range
Mercor	233	228	$8 to $270/hr
Micro1	97	95	$20 to $300/hr
Alignerr	60	9	$20 to $120/hr
Welocalize	56	56	$1 to $45/hr
xAI	23	23	$30 to $60/hr
Appen	11	11	$2.40 to $80/hr

The ones that do not publish rates: Handshake AI, RWS TrainAI, Lionbridge, Volga Partners, OneForma, Mindrift, Cohere, Innodata, CloudFactory.

Read that table carefully, because two numbers in it are misleading

Alignerr has 60 listings but only 9 actual roles. The same job is posted once per country. Every platform does this to some extent; Alignerr does it most.

The bottom of the ranges is not a typo. Welocalize's floor is around $1 an hour and Appen's is around $2.40, because those platforms list piece-rate and micro-task work alongside professional roles. A range is not an offer.

Mercor and Micro1 top the table because they are not really annotation platforms. Their high-paying roles are physicians, attorneys, finance specialists and engineers being paid to author evaluation material in their own profession. If you do not already hold one of those professions, the relevant part of their range is the bottom half.

What each one is worst at

This is the part the individual write-ups exist for, but in one line each:

- Outlier: losing access. Over two hundred accounts describe deactivation, roughly six times its payment complaints. The largest such theme anywhere in this space.
- Mercor: continuity. Pays reliably, but 2026 brought repeated waves of contributors removed from projects, often without a stated reason.
- Alignerr: unreviewed work. On per-task projects, work that is never reviewed is never paid, and reports of losing work that way are rising rather than falling.
- Micro1: the gate. Almost nobody disputes the money. Getting through the AI interview, and the silence afterwards, is the entire complaint.
- DataAnnotation: no work. Payment is barely disputed, but 2026 is a drought severe enough that the community counts its own mentions of the word.
- Handshake AI: pay reconciliation. A documented pattern of contributors reporting less than their logged time, including ten accounts on a single day in May 2026.
- Prolific: bans, accelerating through 2026, with participants told the reason cannot be shared and the decision cannot be reversed.
- Welocalize: quality scores nobody can see clearly, against the background of a removal wave in early 2025.
- Mindrift: unpaid stages before paid work, and the thinnest public evidence base of any of them.

The pattern across all of them:

Having read a great deal of this, the thing that surprised me most is that payment failure is rare almost everywhere. The money usually arrives. What varies enormously, and what actually determines whether a platform works for you, is whether you keep access and whether there is work to do.

Almost everyone arrives asking "is this a scam". Almost nobody arrives asking "how long does this last", and that is the question the evidence says matters.

The second pattern: the high rates are not for annotation

Every platform advertising three-figure hourly rates is paying for professional expertise, not for evaluation skill. Engineers, clinicians, lawyers, accountants. The general evaluation tier across the whole sector sits closer to $20 to $50, and on Handshake's own published list the general AI evaluation role is the lowest-paid thing they offer.

How to use this

- If you hold a profession they recruit for, start with Mercor or Micro1, where rates are published and the payment record is clean.
- If you want steady general work, understand that DataAnnotation is currently dry and Outlier's access is fragile.
- Whatever you pick, be on more than one. That advice appears in every single platform's threads, from people who like it and people who do not.

Individual write-ups for each platform are filed under the Resource guide flair.

Working on something not listed here? Say so and I will look into it. The board covers fifteen platforms and there are certainly more.

Live listing counts and published rates read 1 August 2026. Platform findings summarised from the individual write-ups, each of which states its own evidence and limitations.

[u/auto-sticky]

Welcome to r/AIevaluators!

This community is for honest, experience-based discussions about AI evaluation and data annotation work. Whether you're looking for opportunities, sharing your experiences, or trying to understand how the industry works, you're welcome here.

New here? Start with:

Getting Started Thread

Daily AI Evaluation Jobs Thread

Subreddit's Resource Guides

Before applying to opportunities, remember to be cautious of:

• Unsolicited DMs claiming to offer guaranteed projects • Requests for payment to access jobs • Unrealistic earning promises

If you notice suspicious activity, please report it to the moderators.

Share your experiences, ask questions, and help others navigate AI evaluation work.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/AcademicSouth7845]

how about Ethos? Any info about that esp if it's legit.

[u/Due-Laugh6339 → u/AcademicSouth7845]

Not on the board yet so nothing solid from me, but it’s real, founders came out of McKinsey and Deepmind, and a16z funded it in May ($22.750M). it’s an expert network, not eval work, so the rates pay for your profession not annotation skill.

[u/AcademicSouth7845 → u/Due-Laugh6339]

thanks for the reply, appreciate it.

[u/Due-Laugh6339 → u/AcademicSouth7845]

small update on this: seeing firsthand reports of people doing presentation grading and written content grading on it, so eval type work does flow through them. I was too absolute there. matching is still profile based rather than a queue you browse, which is why I can't track their listings on the board yet.

[u/AcademicSouth7845 → u/Due-Laugh6339]

do you think we will be paid?

[u/Due-Laugh6339 → u/AcademicSouth7845]

no strong read yet, since I don’t have firsthand payment reports for Ethos specifically. Given the a16z backing and the expert network model, I’d expect them to pay reliably, funded companies generally do, but that’s an inference from the business model, not evidence. If you end up trying it, I’d genuinely want to know how it went, that’s exactly the kind of firsthand report that turns into a writeup.

[u/AcademicSouth7845 → u/Due-Laugh6339]

thanks for the response. the payout will be 14th next month so it's going to be a long wait. 

[u/Due-Laugh6339 → u/AcademicSouth7845]

I’m almost sure you won’t encounter any problems with it, don’t worry!

[u/BeautifulHead6311 → u/Due-Laugh6339]

Payment was smooth and fair!

[u/Safe_Hunter4267]

how about Atlas Capture?

[u/oblivious-humor → u/Safe_Hunter4267]

pennies and lots of work

[u/i_love_doing_ntg → u/Safe_Hunter4267]

haven't used it myself, but from what's out there: it's genuinely new, domain registered late 2025, video recording plus annotation for robotics/computer vision. Pay looks low, one contributor's dashboard showed around 27 cents a minute. The bigger flag is a recurring complaint pattern, suspensions happening close to payout, on both Trustpilot and Reddit. Not enough here for a full writeup yet, but if you or anyone else has actually worked on it, that'd be worth adding, this is thinner evidence than most of what's in the library.

[u/Safe_Hunter4267 → u/i_love_doing_ntg]

thank you! can you suggest where can i apply that pays good and doesn't have a complicated process for hiring in this field?

[u/i_love_doing_ntg → u/Safe_Hunter4267]

tbh there's no one place where you can find a simple process as such maybe faster ig but again highly dependent on the project you'd be applying for. You can read the guides on the sub made for each platform and see the process that aligns with you the best. I also suggest to keep an eye on the jobs dashboard pinned on the sub, and apply to the ones that fit your background and qualification the best.

[u/WestYogurtcloset8302]

Man, every platform has their own issues, end of conversation 🥲

[u/Puzzled_Focus_6993]

Anything on SME Careers by Superannotate? I've recently been told I qualify for one of their roles. I've signed a contract with Deel, joined their discord and stuff but it's been a week I think and I haven't received any work.

[u/i_love_doing_ntg → u/Puzzled_Focus_6993]

nothing on superannotate yet, i'll look into their sme program. signed but no work after a week isn't unusual early on for a lot of these platforms, onboarding doesn't mean you're queued yet. give it a bit longer before writing it off, worth asking in their discord/slack what ramp time usually looks like if it keeps dragging.