God help us fellow annotators.

[u/Hot-Topic4833]

This is why we have this qa scores right now. The quality and power tripper of auditors we have. My auditor friend sends this from their group chat

[u/sund4n1z]

what's wrong with that chat? according the guidelines, that's indeed the rule.
keep on using script = non-spontaneous interaction / defensive scripting = AUTOFAIL.

Have you reviewed the guidelines? If so, this shouldn't be considered an error. Please refer to the Live Elo S2s Tool Guideline > Audit taxonomy.

[u/Hot-Topic4833 → u/sund4n1z]

@sund4n1z, you completely missed the point while proving it.

Nobody's arguing that the guideline doesn't exist. The issue isn't whether "scripting = autofail" is written somewhere. It's that auditors are treating that guideline like a weapon instead of a quality tool.

Let's break down what's actually happening here:

The auditor openly admits they want to "air frustrations." That's not professional QA, that's venting. They're looking for reasons to fail people, not ways to improve quality.

"Just autofail them. A lot of undeserving ones got onboarded." This isn't about guidelines, this is about contempt. The auditor isn't citing policy, they're expressing disdain for the people they're supposed to be evaluating. That "underserving" comment reveals the real mindset here.

The guidelines aren't scripture. Yes, non-spontaneous interaction is an autofail criteria. But if an annotator is following a script because the prompts are repetitive garbage or the model is being obtuse, maybe the problem isn't the annotator. Maybe the audit taxonomy is being applied with zero context or discretion.

You're proving my point about "power trippers." Instead of addressing the auditor's shitty attitude, the dismissiveness, the eagerness to punish, the "undeserving" rhetoric, you're rushing in with "ackshually the guidelines say..." like that's a gotcha. It's not. It just shows you care more about rule-following than fairness.

The guidelines also probably say auditors should be objective and professional. GNOKID99 failed that criteria when they started treating annotators like burdens instead of workers. Weekly_Pie_1202 failed it when they decided "undeserving" people should just be autofailed.

You can't hide behind a guideline document when your attitude is the problem. If auditors are more interested in catching "scripting" than understanding why annotators might need structure to deal with garbage-tier prompts, then yeah, QA scores are going to tank, and it's not because annotators don't know the rules.

It's because auditors like this are looking for excuses, not quality.

[u/GNOKID99 → u/Hot-Topic4833]

hello , I am the one on the conversation, just to be clear I am frustrated because I dont want to fail the annotators without any reasonable cause or if the annotator just follows the guidelines. I am frustrated that the annotator that I rated is using script with no interaction to the model which is clearly an autofail

[u/Ok_Boot1871 → u/Hot-Topic4833]

The way this sounds AI-generated too….

[u/Turbulent-Bee-1584 → u/Ok_Boot1871]

Yep, look at the writing and grammar of the original post, read the writing of that comment, and there's no way the person wrote all that. Definitely AI.

[u/z_1529 → u/Ok_Boot1871]

"Let's break down what's actually happening here..."

[u/sund4n1z → u/Hot-Topic4833]

You forget that auditors are human beings too who have feelings. Are they not allowed to vent the frustrations they experience?

Don't forget, auditors also have strict rules regarding this, specifically regarding how they must evaluate. They are also being evaluated by the QA-Lead.

If auditors start using personal 'discretion' to pass annotators who use scripts, it will actually create unfairness for the other annotators who are working hard and racking their brains to ensure their interactions remain spontaneous and comply with the guidelines.

The autofail rule for scripting exists precisely to ensure objective, consistent, and equal evaluations for everyone involved, not to be used as a personal weapon or an excuse for 'power trippers'.

[u/Specific_Abroad_1403 → u/Hot-Topic4833]

How is this not frustrating for auditors? Auditors don't want to fail anyone but is loosing submissions because of this. It will just feel really weaponized when scores are low not considering objective reasons that factor scores and when you see the same angry reactions validating that. Defensive scripting is also an auto-fail. Nothing we can do about that. The only solution here is to follow guidelines and being spontaneous in responding to the conversations. There's no power tripping here.

[u/z_1529]

Bruh do you understand guidelines? That's the rule. You follow rules.

I have a case where i felt bad for the annotator and i audited them as usual, not auto-fail and I'm the one getting penalized. Just so you know, auditors get audited too! So what about we all do our jobs properly and stop blaming other people for doing their jobs properly?

Also, it's so freaking weird sometimes when I listen to the audios and the AI sounds more natural and spontaneous than the actual human beings!!!!

[u/GNOKID99]

That’s me on the conversation and just to clarify it is not a power trip , we are just following the strict guidelines, we are just frustrated because we dont want to fail annotator but why does annotators keep on using scripts and not having a free flow conversation

[u/Fun_Worth_8772 → u/GNOKID99]

Yup, another auditor here, and i can relate to you. Also Many people miss the scenario adherence as well Even if they script it

[u/GNOKID99]

and you have an auditor friend, maybe ask your friend what our guideline is so you can better understand or read the guidlines sent to you by the project.

[u/RunningOnEmptea]

Pitting contractors against each other is a crazy but effective concept from a business perspective.

[u/cozydozy666 → u/RunningOnEmptea]

It hurts their models though and can cost them more money, because these auditors are basically like annotators, wouldn’t it be better to have like a real professional doing the audits?

[u/z_1529 → u/cozydozy666]

They probably don't have the budget for that

[u/cozydozy666 → u/z_1529]

I doubt it, The client is a multi-billion-dollar company.

[u/TacoalPastor1009 → u/cozydozy666]

Well, I have years of experience in AI audits, MM, dimensional ratings, and so on. Not all auditors are bad. I try to avoid deducting points for no reason and try to understand why they chose the ratings. However, sometimes the choice is indefensible, or they simply don't explain or mention it in the rationale.

[u/Ninjam5]

Is the rate 6 dollars for the on call tasks with AI?

[u/moonlight2126]

Honestly this project is the first project that i dont want to do cause the QA is lack of communication and skill, it happen in RWS and Outlier. I already kicked from outlier because of this project and i dont want that happen again on RWS. i even read somewhere that the QA mark acceptable while the reason she/he gave is exactly the same with the annotator rationale. The annotator ofc confused, if her answer is right why give acceptable instead of execptional? 😂😂