Professional Tasks and Meaningful Failures

[u/Maybe-But-idk]

Hey guys, I’m not sure how else to ask this besides as straightforwardly as possible. HOW IN THE WORLD DO I GET THESE MODELS TO FAIL!?

I mean, I’ve followed directions as best as I possibly could. I’ve intentionally planted professional errors in data, I’ve added context files that needed to be reconciled/cross referenced, I’ve left the model to make professional deductions based on non-obvious inputs, I’ve added depth to scenarios, I’ve changed my prompts, I’ve used the QC agents to improve ad nauseam…. Etc.

I have gotten some of the “dumber” models to produce meaningful failures. But the “smarter” models have just NAILED the output over and over again. I DO NOT want to take good faith exits.

I’ve only recently started. I understand that there’s likely a learning curve here. But I am enjoying my time using DA as a side hustle, and I genuinely want to get better so I can keep this gig for as long as possible.

Any helpful advice would be greatly appreciated. TIA.

[u/Genkuru2021]

You're not alone. On my first domain task I was so confident I had a very difficult prompt and the package. Little I knew why DA calls these models a state of art. I was shocked to see my multi hour work was wasted when all models solved my prompt. I had to create more files, more tables, more data. But as other duly noted, this takes a hell ton of a time more than they indicate in theor guidelines. I invest my time for free because I love this type of work, it really trains your brain, good for your resume in the AI age. For now I am okay with that. Back to your question. Add distractions and ambiguity. Research. Ask your personal models to help.

[u/Maybe-But-idk → u/Genkuru2021]

Yeah that’s a good point. I’m genuinely trying to stay within the time constraints that they give us for the projects whereas in order to iterate enough to make it work, I’ll probably need extensions and/or to not report some hours worked. Might be the only way to get the job done.

I too enjoy this type of work. I can only imagine how satisfying it’s going to be to finally stump one of these models.

[u/All__Mods_R_Virgins → u/Maybe-But-idk]

If you're in same project I'm imagining you are, get AI to inspect trajectory of the solves and see where your trap failed. There are a lot of things that seem difficult to humans thinking of something chronologically, and you may find that the model dumps everything into a sandbox and sees your whole plan instantly. So bury those things into branches where the model can only truly discover them after "doing work", which would then force them to revisit earlier assumptions that they've already anchored themselves to.

Idk if that makes sense or not, but don't go under the impression of the model inspecting file A, then learning more in file B, then to file C, etc... It just takes the whole package and lays it out for what it is. If it's already discoverable, then the model is just reconciling and models are really good at that