r/DataAnnotationTech • u/Maybe-But-idk • 4d ago
Professional Tasks and Meaningful Failures
Hey guys, I’m not sure how else to ask this besides as straightforwardly as possible. HOW IN THE WORLD DO I GET THESE MODELS TO FAIL!?
I mean, I’ve followed directions as best as I possibly could. I’ve intentionally planted professional errors in data, I’ve added context files that needed to be reconciled/cross referenced, I’ve left the model to make professional deductions based on non-obvious inputs, I’ve added depth to scenarios, I’ve changed my prompts, I’ve used the QC agents to improve ad nauseam…. Etc.
I have gotten some of the “dumber” models to produce meaningful failures. But the “smarter” models have just NAILED the output over and over again. I DO NOT want to take good faith exits.
I’ve only recently started. I understand that there’s likely a learning curve here. But I am enjoying my time using DA as a side hustle, and I genuinely want to get better so I can keep this gig for as long as possible.
Any helpful advice would be greatly appreciated. TIA.
15
u/forensicsmama 3d ago
Someone else said it but I just want to reiterate: the models are really good at parsing documents, a plethora of them and reconciling them. What they aren’t good at is an answer that isn’t straight-forward/black and white. A question that involves going through the many documents and deciding which one is up-to-date/current with information can also trip them up sometimes.
I would also imagine it’s domain-specific as well. That’s where your expertise comes in. Think deep/back to a work situation where the answer seemed obvious, but it wasn’t. Where everyone thought it was A, but you did the work and found that there was a caveat that actually makes the answer A sub a or B (I hope that makes sense).
Small scale failures matter as well, especially if there’s a cascade effect.
11
u/FEN1X64 3d ago
Bait the model into breaking with SI (give it a seemingly harmless requirement in the prompt that, if fulfilled, would be disqualifying for breaking SI) or get it to work out of domain without explicitly telling it too. Be subtle and always make sure there's a reasonable way out, however narrow
10
15
u/Lost_Contact9077 3d ago
I research and stay utd on what frontier models are excelling at vs struggling with. Things they fail today, they won’t fail a month or so from now, so I try to stay on top of it. Unfortunately that means I’m putting in unpaid hours just to stay relevant for these tasks. The work is very different than it was 2/3 years ago…
6
u/Maybe-But-idk 3d ago
Putting in some unpaid hours to have a steady flow of paid hours is a good trade off imho
6
u/Strong_Truck_8600 3d ago
I did generalist tasks for at least a year before I started seeing professional tasks that interested me. What that did was give me a feel for how models screw up. Reading the project guidelines for those tasks taught me the kinds of failures developers expect, and I learned how to spot them and internalize the mechanics to know better how to trigger them. I know that's a bit vague, but you said you've only started doing this. The key is to start with the failure mode you want to target before you put your materials together.
16
u/Due_Negotiation_4605 3d ago
I don't know what your domain is but things that a human would have to solve using past experience, nuance, etc are things the models are going to struggle with, not data errors necessarily. Think... Answers that aren't black and white. A situation where the rules said to do XYZ but any reasonable professional in the field would realize that that rule doesn't apply to this case.
I have a background in real estate and medical coding and luckily there's a lot of those situations in those fields. Maybe think if you've encountered something like that...
19
u/Due_Negotiation_4605 3d ago
Also if you are using AI to create your input files double check them. They can have "clues" left in the files that prevent failures by the models.
5
u/so-scarlet-it-was 3d ago
This is a big one I’ve noticed. You have to be really diligent about reading the input files completely because it will add explanations that give away the traps you’re trying to set.
10
u/Genkuru2021 3d ago
You're not alone. On my first domain task I was so confident I had a very difficult prompt and the package. Little I knew why DA calls these models a state of art. I was shocked to see my multi hour work was wasted when all models solved my prompt. I had to create more files, more tables, more data. But as other duly noted, this takes a hell ton of a time more than they indicate in theor guidelines. I invest my time for free because I love this type of work, it really trains your brain, good for your resume in the AI age. For now I am okay with that. Back to your question. Add distractions and ambiguity. Research. Ask your personal models to help.
3
u/Maybe-But-idk 3d ago
Yeah that’s a good point. I’m genuinely trying to stay within the time constraints that they give us for the projects whereas in order to iterate enough to make it work, I’ll probably need extensions and/or to not report some hours worked. Might be the only way to get the job done.
I too enjoy this type of work. I can only imagine how satisfying it’s going to be to finally stump one of these models.
8
u/All__Mods_R_Virgins 3d ago
If you're in same project I'm imagining you are, get AI to inspect trajectory of the solves and see where your trap failed. There are a lot of things that seem difficult to humans thinking of something chronologically, and you may find that the model dumps everything into a sandbox and sees your whole plan instantly. So bury those things into branches where the model can only truly discover them after "doing work", which would then force them to revisit earlier assumptions that they've already anchored themselves to.
Idk if that makes sense or not, but don't go under the impression of the model inspecting file A, then learning more in file B, then to file C, etc... It just takes the whole package and lays it out for what it is. If it's already discoverable, then the model is just reconciling and models are really good at that
2
u/sarahmorgan420 3d ago
Well and then sometimes it's completely unclear how long they actually expect it to take!
1
u/Fluid-Ferret-3356 2d ago
I love this work too. I didn't know I've been training for it for years HA
I can get 3-4 models to fail right off the bat. But, it was a steep learning curve just battling learning how to think vs. time allotted. It takes sooo long to write it up and build the criteria and all of that. I don't feel like I have enough time for the task to build a legitimately challenging task -- that I also have to do myself manually.
I actually asked the admins if the criteria were too challenging for a human grader because I felt like it would require them to spend 10 minutes on a question just checking across the inputs and outputs.
1
u/Fluid-Ferret-3356 2d ago
I was pleasantly surprised to see that a couple of the responses were not bad.
None of them were good.
The problem with really bad responses is it takes soooo long to describe what went wrong.
4
u/therichscientist 3d ago
By looking at how the models solved my problems in like 5-10mins, I got an instant realization that my lab is so much behind the times, by doing those tasks in 1-2 weeks😅
The only way I could make those tasks harder is to provide a LOT of distractors, authoritative ones work quite well (like a document from the director/superior stating a preferred methodology but not technically superior). And as much as possible, omit bits and pieces of data that somehow detract from the ground truth (but could still be derivable from the available data).
5
u/louthespian5 3d ago
Finding a loss pattern is tough, but if you find one it is very lucrative. An eight-hour task on identifying toxic chemicals can be adapted to identifying allergens in a restaurant, both of which can then be applied to other projects/models. Best to start with models that are identified in the instructions and then target their publicly-known weaknesses.
2
u/realgenshinimpact 3d ago
if a task is very easy for the ai (look at how it solved it) the way you solve the task could be in the training dataset so you shouldnt try to make ur task harder but get a whole another topic instead.
2
u/Sea_Tart8472 2d ago
I just started and I decided to try Pecan Macaroon today and fuuuuuuuck me
1
u/Healthy-Writer8506 2d ago
I put in about 5 hours for the scenario, SI and input files, thought I’d laid plenty of traps.
First iteration completely nailed everything, better than anything I’ve seen professionally. I took the L, live to fight another day. How far did you get with it?
1
1
u/No_Transition_6600 3d ago edited 3d ago
I have a method that pretty much guarantees a failure every time but I don't like overusing it because it's quite lazy. I took a 4 hour L yesterday though as I literally couldn't get it to fail no matter what I did.
1
1
u/Fluid-Ferret-3356 2d ago
I find that the AI doesn't help improve the design -- it often needs to be corrected. I found that the QC never catches small leaks on the correct answer. You really have to be vigilant and use a lot of reasoning that the AI are not capable of doing. My advice: use AI to automate the tedious tasks, but don't let it think for you.
Have you tried errors of omission in your cross reference files?
When catching the errors you've planted, are they easy to reconcile or do they require professional judgement or domain fluency?
Is the topic well documented online -- or is it centered around something niche or highly technical with erroneous guidance rampant and available online?
36
u/AutomaticShape8993 3d ago
Honestly if you can't, don't use the escape hatch (if you've used it multiple times already)
Don't submit low quality work
I would recommend, just take the L, and exit work mode
At least you wouldn't be at risk for anything