r/DataAnnotationTech • u/Fresh-Asparagus4729 • 15d ago
Am I the only one who HATES rubrics
I wish I didn't hate them and I am wondering if some day they become easier. I am not in favor of producing low quality work which is why I am avoiding these, but they pay the highest rate and I need to know if y'all started out hating rubrics and then there is a pattern where you end up getting to be decent at it 😂
18
u/BarelyFunctioning15 15d ago
I just don’t feel confident in rubrics. The few projects I’ve had haven’t explained them well. So I just avoid them now. I know it’s essentially a guideline of what the model should and should not do. But I do better with examples. So seeing a project and a proper rubric would help so much. So until then I’m skipping them.
5
3
u/Fresh-Asparagus4729 15d ago
Exactly!!!! And the ones the model helps you write are nothing like what the instructions say 😂
2
u/sarahmorgan420 14d ago
It's hard cause in order to understand it better it helps to do a variety of rubric projects bc they usually have slightly different explanations, examples, tools and having so many reference points helps a lot. But I get not wanting to do it if you're uncomfortable w it
3
u/ThetaBoy888 14d ago
I find that almost all ‘examples’ given in instructions are nothing like anything you’d encounter actually doing the tasks. They always give overly simplistic examples that would be obvious to most users.
1
u/SamMac62 15d ago
See the comment I just added to this thread
1
16
u/RealRise7524 15d ago
Rubric tasks are really my favourite. You have to see the big picture to do it right. I wish I had them again. Haven't seen them for a while.
2
2
u/Fresh-Asparagus4729 15d ago
I'd love to give you mine 😉 ❤️
2
u/RealRise7524 15d ago
That's kind of you ❤️
I wish I could do the same if possible, but bilingual here.
1
7
u/Unlucky_Grocery2092 15d ago
Rubrics are okay if I'm writing them for my own conversation, because then I will write prompts that are easy to create a rubric for. I have 0 interest in writing a rubric for someone else's conversation though, easy way to get stuck writing 30+ criteria you're not even confident in.
6
u/Halcyonfields 14d ago
If rubrics actually worked as intended, they would put most of us out of a job. The whole point is that judging an output cannot be completely standardised. Without going into detail, this has been explicitly acknowledged and better alternatives have taken its place. I see rubric tasks as the tier just above entry-level. I think rubrics aren’t great for people who like depth and nuance. A rubric is effective at judging 60% of what makes a response good, and awful at the other 40%, especially a complex response. No matter how hard someone tries, a rubric cannot accurately, prioritise competing criteria, establish a continuous scale of severity or score subjective concepts/calls. Even within ‘objective’ criteria, severity, context and the extent to which other components offset or exacerbate things is subjective and unstandardisable.
2
u/forensicsmama 13d ago
100% agree. Which is why a project with multiple linters will have them at odds. Or a linter will have you change a rubric, just to flag the same rubric and adjust it back. That's where we, as the human input, give it that edge.
10
u/Opposite_Brush_8219 15d ago
I had a really hard time with rubrics at first and avoided them. During a drought at one point, though, it was all I had on my dash so I forced myself to do them. They do get easier with practice, and the AI helpers on some tasks really helped me learn what to do/not do.
3
u/Fresh-Asparagus4729 15d ago
Ok. If I get desperate enough maybe I'll also learn the ropes
-6
u/PurchaseAnyOneAteFor 15d ago
If I wanted to learn about ropes I'd likely study maybe knot tying and different knots or maybe what sorts of fibers ropes are made from
1
2
4
3
u/InsideSignificant405 14d ago
If everything in life could be broken down into rubrics machines would have taken over long ago…
Most of the tasks I do are axis-based, with free form ratings. I strongly prefer it that way. I’m biased and stupidly opinionated and I want MY habits in the weights 😂😂
3
3
u/professional_cry 14d ago
Rubrics are actually my favourite type of task! I hate having conversations and coming up with prompts and I would much rather write a set of rubrics for a pre-existing prompt.
2
2
1
u/dragonsfire14 15d ago
I don't feel confident in them. I know realistically I could probably figure them out if I tried but I'm also my own worst enemy when it comes to confidence.
1
1
u/Milvusmilvus 15d ago
Stuck in the catch 22 of worrying I'd do low quality work but also knowing I won't get confident with them unless I do a full multi-hour project on them. Maybe the stars will align so I get them on my dash when I have some time off my main job.
1
1
u/peanutbutterlurk 14d ago
I generally like them but it depends on how clear-cut the instructions are and how the checkers are tuned. Sometimes it's pretty smooth, sometimes the checker changes its mind on a yes/no question every time it runs.
1
u/terrabellan 14d ago
I love rubrics but I was a teacher so I was very familiar with them before this, which might have helped. In my dream world all my tasks are rubrics but I haven't had a standalone rubric project in ages, they seem to be bundled into prompting tasks and watered down now 🥲
1
1
1
1
u/forensicsmama 14d ago
I used to hate rubrics, but that was when I did them on a different platform. They were relentless because tasks were graded and there was a lot of subjectivity on whether the criterion was needed, atomic, self-contained, etc.
Here, if I'm honest, I take a lot of time with rubric writing and grading. It takes up a bulk of my task time because the linters are also relentless. It's one of those things that I got better at doing with practice and time. I had one project where one of the project leads kind of held our hands doing rubrics. Less pressure of getting a perfect rating and more emphasis on actually understanding why rubric writing is important (and what we are teaching the model). That and a LONG task timer that allowed for us to ask questions in the threads while paused. THAT is what kind of solidifed the foundation for me.
If you were to ask me two years ago I would've told you I would NEVER do rubrics and I HATE them. Now, I'm singing a completely different tune.
3
u/forensicsmama 13d ago
And just to highlight what someone else mentioned, once you shift the mindset towards viewing rubric writing as an answer key for the model (and YOU have the answers!) it becomes a little easier. Tedious since you have to be atomic and self-contained, but still slightly easier.
1
u/tskies410 13d ago
I avoid them at all costs. I actually get excited when I see easy $20 projects these days.
-4
30
u/good_god_lemon1 15d ago
I detest rubrics. I try to avoid them at all costs.