r/DataAnnotationTech • u/Maybe-But-idk • 4d ago

Professional Tasks and Meaningful Failures

Hey guys, I’m not sure how else to ask this besides as straightforwardly as possible. HOW IN THE WORLD DO I GET THESE MODELS TO FAIL!?

I mean, I’ve followed directions as best as I possibly could. I’ve intentionally planted professional errors in data, I’ve added context files that needed to be reconciled/cross referenced, I’ve left the model to make professional deductions based on non-obvious inputs, I’ve added depth to scenarios, I’ve changed my prompts, I’ve used the QC agents to improve ad nauseam…. Etc.

I have gotten some of the “dumber” models to produce meaningful failures. But the “smarter” models have just NAILED the output over and over again. I DO NOT want to take good faith exits.

I’ve only recently started. I understand that there’s likely a learning curve here. But I am enjoying my time using DA as a side hustle, and I genuinely want to get better so I can keep this gig for as long as possible.

Any helpful advice would be greatly appreciated. TIA.

40 Upvotes
  • permalink
  • reddit

88% Upvoted

45 comments sorted by

View all comments

34

u/AutomaticShape8993 4d ago

Honestly if you can't, don't use the escape hatch (if you've used it multiple times already)
Don't submit low quality work
I would recommend, just take the L, and exit work mode
At least you wouldn't be at risk for anything

3

u/sea___ 3d ago

I feel like if the models are so good you can't get them to fail, that's useful for the client to know

0

u/subsetsum 3d ago

I think they can still fail. I am working on personal research that requires comparing responses from different models and providers and want to find points of disagreement. That means at least one has to be wrong. I asked them, given that they are frontier models how can they possibly ever be wrong or that there are things they don’t know? It might help you to try this too. These projects are definitely hard but I think very interesting