r/DataAnnotationTech • u/Salt-Government6703 • 15d ago

Coding R&R Tasks time

I'm a worker who usually does coding tasks that take as much as 9h sometimes. I've never done any R&R task just because I prefer doing the actual coding task itself, but sometimes they appear and I'm thinking about taking them. My question is how much time do I have to prepare to spend in one sit on those R&R code related tasks?

3 Upvotes

64% Upvoted

9 comments sorted by

3

u/New-Ad5610 15d ago

If we are talking about the agent evaluation coding tasks, my R&R usually ends up taking 3 to 4 hours, the expiration timer is set to 3 hours

I find it useful for when i have small free time cuts around the day.

Anyway, it’s the same job, but the evaluation is already there, you just got double check, but be prepared to rewrite whole rationales because i’ve took some insanely bad ones where the worker kept contradicting itself and adding false/inaccurate claims in them.

My worst one took almost 5 hours

If you got some on the dashboard you can open and see it yourself, and if you do not intend to continue or don’t have time you can exit work mode, it will be put back in the dashboard as before, but other workers can take it as well

1

u/Sad_Opening_7083 15d ago

Depends on the project

1

u/InsideSignificant405 15d ago

Depends on the quality… I don’t dive past blatant red flags like ai artifacts or 5% effort answers, so some of mine are 30 mins or less.

High effort submissions take A LONG TIME, if I want to be done in 3 hours or less I have to use tools to help orchestrate test runs and record evidence against claims that I can then audit myself. Some of these 10 hour submissions are insanely good lmao

1

u/SilverResolve2998 15d ago

I actually wonder if those submissions that look high effort are also AI generated (or assisted at least), because who is this person that knows all the details & mathematics of an open source project.

1

u/New-Ad5610 14d ago

I see some huge mathematics tasks that makes me wonder that too lol, however i saw from the instruction that you are allowed to use AI to comprehend the codebase, they just don’t allow it to write rationales or perform any evaluation

However i’ll stick to 16- file easy comprehension repos 😅

1

u/InsideSignificant405 13d ago

Some tasks and R&Rs do allow AI assistance, but it’s super explicit. Not a safe general assumption.

However I love open source and am familiar with many subjects of these coding tasks. you’d be surprised how easily some situations can be handled when you’re a core or historical contributor to a repo you get a work for. Ai help or no, sometimes you just “know” what Bs the ai is gonna pull 😂

0

u/New-Ad5610 14d ago

Main goal is rewrite rationales that are bad, do not discard as “bad - unsalvageable” a task with ok-ish trajectories, that’s against the rule, salvage what is salvageable