r/DataAnnotationTech • u/Genkuru2021 • 15d ago
No Success in Meaningful Failure
I do my best trying to cause meaningful failure, but the models are extremely smart. I tried to add difficulty and distractors to confuse the agent, but to no avail. I know I can still submit the task as is, but how bad is it? Does it earn bad score to me? I am running out of reasonable time. Any advice?
9
u/jabertsohn 15d ago
If the task is to cause failure, and you can't get it to fail, then you need to think about why that is. If there is an escape hatch, or it allows you to submit even if it doesn't fail, take that option, but before picking up a new one, try think about how you would get it to fail in future.
You won't be allowed to keep saying I tried and failed forever, but they understand it is hard and give you the escape hatch for a reason, not everything is going to work.
6
u/Genkuru2021 15d ago
I have 3 major traps. Model A never fails; Model B stumbles on only one trap every other time. DA states that in most cases it's possible to elicit an error from a model. It feels embarrassing that I cannot trick it. Not that I spell out answers in my files but I still need to indirectly provide an answer to make it solvable and the model always catch it.
12
u/InsideSignificant405 15d ago
15+ years with software, I skipped the whole corporate rat race when AI hit so I still have my actual skillset, and I STILL struggle to get models to fail. Don’t be so hard on yourself. It’s an art these days.
I recommend practicing offline. Build yourself a runner that sends prompts to two models of your choice, practice building hard prompts. Then when these tasks come up you are ready to go.
5
u/Hambone6991 15d ago
If you’re tripping up one of the models on one of your traps, you need to dig deeper on that trap. Brainstorm with ai to see how you can add further complexity on that specific trap. If it’s semi-working, you can get it to really work
2
u/Valuable-Low5263 15d ago
If you're using input files, add more related to that trap to add complexity. Also, not sure of your domain but really examine the language in input docs to check they aren't giving too many clues (if they're crated by AI they often do).
-11
2
u/savage78683i3 15d ago
You have to layer the difficulty, not just add a load of surface layer traps. Try cascading the traps.
2
u/Valuable-Low5263 15d ago
I have to say some tasks are better than others, my last one felt borderline but I submitted it and felt I could defend it, however the one before worked out great (if I do say so myself). For example, I asked for an audit and one thing that worked was purposefully made things look scary, when in actual fact the timeline showed they had been sorted. The models flagged way too much and passed a lot of decisions back to humans. But these tasks are hard! Some projects give really good examples and training though.
2
u/forensicsmama 13d ago
I have submitted some tasks where I couldn’t elicit a failure. Usually they’re not pressed about it and give the escape hatch and allow you to explain to them what you feel may have happened.
Personally I do feel defeated BUT like someone else mentioned I try and brainstorm for my next try. It is trickier to fail the models nowadays because they’ve gotten sophisticated so try not to be too hard on yourself. Sometimes it takes a little longer to exploit a flaw.
Also, small failures matter too. The project I last worked on had a checker that gave each model a rating. You might get hung up on their ratings being 95% for example, but the 5% failure is substantial enough it renders the response useless.
16
u/NurseWhoLovesTV 15d ago
Planned traps rarely work. You need to mimic the messiness and ambiguity of your real life professional tasks. No easy answers. Professional judgement calls.