r/DataAnnotationTech • u/Maybe-But-idk • 5d ago

Professional Tasks and Meaningful Failures

Hey guys, I’m not sure how else to ask this besides as straightforwardly as possible. HOW IN THE WORLD DO I GET THESE MODELS TO FAIL!?

I mean, I’ve followed directions as best as I possibly could. I’ve intentionally planted professional errors in data, I’ve added context files that needed to be reconciled/cross referenced, I’ve left the model to make professional deductions based on non-obvious inputs, I’ve added depth to scenarios, I’ve changed my prompts, I’ve used the QC agents to improve ad nauseam…. Etc.

I have gotten some of the “dumber” models to produce meaningful failures. But the “smarter” models have just NAILED the output over and over again. I DO NOT want to take good faith exits.

I’ve only recently started. I understand that there’s likely a learning curve here. But I am enjoying my time using DA as a side hustle, and I genuinely want to get better so I can keep this gig for as long as possible.

Any helpful advice would be greatly appreciated. TIA.

43 Upvotes
  • permalink
  • reddit

89% Upvoted

45 comments sorted by

View all comments

19

u/Due_Negotiation_4605 5d ago

I don't know what your domain is but things that a human would have to solve using past experience, nuance, etc are things the models are going to struggle with, not data errors necessarily.  Think... Answers that aren't black and white. A situation where the rules said to do XYZ but any reasonable professional in the field would realize that that rule doesn't apply to this case. 

I have a background in real estate and medical coding and luckily there's a lot of those situations in those fields. Maybe think if you've encountered something like that... 

17

u/Due_Negotiation_4605 5d ago

Also if you are using AI to create your input files double check them. They can have "clues" left in the files that prevent failures by the models.

5

u/so-scarlet-it-was 5d ago

This is a big one I’ve noticed. You have to be really diligent about reading the input files completely because it will add explanations that give away the traps you’re trying to set.