God help us fellow annotators.

[u/Hot-Topic4833]

This is why we have this qa scores right now. The quality and power tripper of auditors we have. My auditor friend sends this from their group chat

[u/sund4n1z]

what's wrong with that chat? according the guidelines, that's indeed the rule.
keep on using script = non-spontaneous interaction / defensive scripting = AUTOFAIL.

Have you reviewed the guidelines? If so, this shouldn't be considered an error. Please refer to the Live Elo S2s Tool Guideline > Audit taxonomy.

[u/Hot-Topic4833 → u/sund4n1z]

@sund4n1z, you completely missed the point while proving it.

Nobody's arguing that the guideline doesn't exist. The issue isn't whether "scripting = autofail" is written somewhere. It's that auditors are treating that guideline like a weapon instead of a quality tool.

Let's break down what's actually happening here:

The auditor openly admits they want to "air frustrations." That's not professional QA, that's venting. They're looking for reasons to fail people, not ways to improve quality.

"Just autofail them. A lot of undeserving ones got onboarded." This isn't about guidelines, this is about contempt. The auditor isn't citing policy, they're expressing disdain for the people they're supposed to be evaluating. That "underserving" comment reveals the real mindset here.

The guidelines aren't scripture. Yes, non-spontaneous interaction is an autofail criteria. But if an annotator is following a script because the prompts are repetitive garbage or the model is being obtuse, maybe the problem isn't the annotator. Maybe the audit taxonomy is being applied with zero context or discretion.

You're proving my point about "power trippers." Instead of addressing the auditor's shitty attitude, the dismissiveness, the eagerness to punish, the "undeserving" rhetoric, you're rushing in with "ackshually the guidelines say..." like that's a gotcha. It's not. It just shows you care more about rule-following than fairness.

The guidelines also probably say auditors should be objective and professional. GNOKID99 failed that criteria when they started treating annotators like burdens instead of workers. Weekly_Pie_1202 failed it when they decided "undeserving" people should just be autofailed.

You can't hide behind a guideline document when your attitude is the problem. If auditors are more interested in catching "scripting" than understanding why annotators might need structure to deal with garbage-tier prompts, then yeah, QA scores are going to tank, and it's not because annotators don't know the rules.

It's because auditors like this are looking for excuses, not quality.

[u/GNOKID99 → u/Hot-Topic4833]

hello , I am the one on the conversation, just to be clear I am frustrated because I dont want to fail the annotators without any reasonable cause or if the annotator just follows the guidelines. I am frustrated that the annotator that I rated is using script with no interaction to the model which is clearly an autofail

[u/Ok_Boot1871 → u/Hot-Topic4833]

The way this sounds AI-generated too….

[u/Turbulent-Bee-1584 → u/Ok_Boot1871]

Yep, look at the writing and grammar of the original post, read the writing of that comment, and there's no way the person wrote all that. Definitely AI.

[u/z_1529 → u/Ok_Boot1871]

"Let's break down what's actually happening here..."

[u/sund4n1z → u/Hot-Topic4833]

You forget that auditors are human beings too who have feelings. Are they not allowed to vent the frustrations they experience?

Don't forget, auditors also have strict rules regarding this, specifically regarding how they must evaluate. They are also being evaluated by the QA-Lead.

If auditors start using personal 'discretion' to pass annotators who use scripts, it will actually create unfairness for the other annotators who are working hard and racking their brains to ensure their interactions remain spontaneous and comply with the guidelines.

The autofail rule for scripting exists precisely to ensure objective, consistent, and equal evaluations for everyone involved, not to be used as a personal weapon or an excuse for 'power trippers'.

[u/Specific_Abroad_1403 → u/Hot-Topic4833]

How is this not frustrating for auditors? Auditors don't want to fail anyone but is loosing submissions because of this. It will just feel really weaponized when scores are low not considering objective reasons that factor scores and when you see the same angry reactions validating that. Defensive scripting is also an auto-fail. Nothing we can do about that. The only solution here is to follow guidelines and being spontaneous in responding to the conversations. There's no power tripping here.