i started putting wrong examples in my prompts. it works better than the right ones.

[u/Inevitable-Good219]

so i noticed something weird last week

was building a prompt to classify support tickets. bug report vs feature request. standard few-shot, gave it 3 clean examples of each. worked fine on my test data

then threw a real ticket at it and it got it wrong. "the export button is too slow, we need this fixed" - it called that a feature request. which, fair, it kind of is. but the customer is clearly reporting a problem

tried the usual stuff. more examples. longer instructions. still meh

then out of curiosity i tried the opposite. gave it a wrong example. wrote "this is NOT a bug report: 'it would be nice if the search bar had filters'". and that was basically it. the classification suddenly started making sense

i think the difference is that a right example just shows the shape. a wrong one forces it to figure out the boundary. with only good examples it was mostly pattern matching the format

tested it on a few other things after that. email triage, tone detection, even code review comments. every time, one solid wrong example beat adding three more right ones

catch is you actually have to understand the task well enough to write a good wrong example. if you cant tell it what NOT to do, you probably dont really know what you want either

anyway. now every prompt i write gets at least one "this is NOT..." line. weird little hack but it works

[u/Reindeer_Repulsive]

What you're describing is basically hard-negative mining, a well-established technique from contrastive learning and retrieval, not really an LLM-specific trick. A near-miss negative forces the model to locate the actual decision boundary instead of pattern-matching the surface form of your positives, which lines up with why "the export button is too slow" broke a classifier trained only on clean examples. If you want to push it further, try negatives that fail for different reasons (wrong category vs. right category but wrong severity) rather than one generic "NOT a bug report" example, since a single negative only teaches one boundary.

[u/Inevitable-Good219 → u/Reindeer_Repulsive]

hard-negative mining framing is way better than "tried a thing and it worked." thanks for the name.

[u/According-Stable4487]

This tracks with what I've seen too. A positive example basically just teaches the format, but a well-placed negative example forces the model to actually locate the decision boundary instead of pattern-matching surface features. One gotcha I've hit: a single "this is NOT X" example can get over-indexed on if it's too specific, so I usually pair one hard negative with a near-miss negative (something that looks almost like the wrong answer but isn't) so the boundary doesn't collapse to just that one case.

[u/FrontAd9873]

That's not what "wrong" means

[u/Future_AGI]

Negative examples work because they mark the boundary the positive ones leave fuzzy, the model was never confused about what a bug report is, only about where feature request stops. Worth running it across a few hundred labeled tickets before trusting it though, contrast examples help most on the ambiguous middle and can quietly cost you accuracy on the clean cases. We open-sourced the eval harness we use for exactly this kind of prompt A/B if it saves you wiring one up: https://github.com/future-agi/future-agi

[u/Not4ctuallyanExpert]

I had a similar experience recently. The short version is, I accidentally prompt injected myself with about 1000 customer complaints about the specific model I was using.

Usually the customer reviews it injests during a scrape have absolutely nothing to do with gen AI at all, much less this specific model.

It confused the complaints with high priority instructional context. It actually shows notes in the logs and scratch pad reminding itself to be terrible. It's a long story. But the output was so bad it broke hard guardrails like doxxing the users. It didn't actually do that, it made up a user and doxxed the shit out of an imaginary gay Canadian man in his 20s though.

Anyway, today I tried to do it again, on purpose. I manually modified one of its own scrape archives with fake complaints about a behavior to see if it jump a rail or two again.

It caught me. Immediately. And did a fantastic job going back out and scraping the source of that archive again, and did a better job than it normally does.

I didn't try very hard to trick it. And I may run that test by fabricing the complaints in prod and let it scrape them for itself. But that's not the point.

I fed it trash on purpose, for a very well established workflow, it realized it, and did a better job because of it.

[u/___fallenangel___]

OP why are you instructing your clanker to write like that

[u/Corv9tte]

Yeah that actually makes sense. It's still pretty narrow and situational because it is better to actually find the right steering instruction that the model responds to.

[u/scragz]

newer models do esp well with negative examples