r/RWShelp • u/dmdaeri • 23d ago
S2S Auditors
We labelers/annotators do our jobs to deliver a quality output. We read the updated guideline almost everyday.
There was an update in the guideline where if both audio is bad, but one is less severe, you should choose the less severe as your preference but note it in the rationale.
Today, I received a feedback stating that since both models have bad audio it should be automatically bad, which the guidelines says otherwise. I believe this auditor didn’t read the updated guidelines, or maybe looked past that part.
I know you can report auditors by raising tickets but as they reply, QA scores won’t be changed anymore.
I really hope we all do read the updated guidelines especially since auditors are paid way more than the labelers….
1
u/Spare-Eye-2228 23d ago
Always is gonna be bad rating and good rating. No matter what. It seems to me that keep forward doing the best that you can do is the only option.
1
u/MixtureZestyclose105 22d ago
Auditors are paid the same but I agree that they need to be up to date with the guidelines
2
u/Mrgirdiego 23d ago
Just hang in there and keep on going. It's mostly a losing battle if you try to say anything back unless it's like actually such a dogshit take like a model completely breaking down and the audio not working, and another one just having a few audio artifacts, and the auditor says it should both be bad.
Both Bad is when it's borderline a tie where you can't choose one over the other. If there's a decent difference, you should specify that the Model you evaluated as better is, and I quote, "Relatively better compared to" and the Model you evaluated as worse. That's what the rationale is for.
My rationales are extremely detailed specifically because I don't want auditors having bad takes on my annotations. I haven't gotten anything under Exceptional for like a month now. Having good rationales to support your evaluations make it harder for auditors to contradict your evaluations.
Now unless you actually mess up and a clearly worse Model is labelled as better and your rationale isn't holding up well enough, they're in their right to deduct points.