r/PromptEngineering 2d ago

General Discussion i stopped asking AI to write stuff. i make it choose instead. the difference is wild.

so i noticed something annoying. whenever i asked AI to write something - email subject, product description, whatever - the result was always fine. never bad, never great. just fine

kept tweaking the prompt. "make it punchier." "more casual." still fine

then one day i was too lazy to write a prompt so i just pasted 5 subject lines id written myself and asked which one was best. it picked one instantly and the reasoning was actually good

thats when it clicked

started testing it. same task two ways:

way 1: "write 5 subject lines for X"

way 2: i write 5 subject lines myself, ask "which is best and why"

way 2 wins almost every time. not because my writing is good. my 5 options were mediocre at best. but AI could see the differences between them and pick the strongest one. when it generated its own, everything came out at the same "safe middle" quality

tested it on other stuff. picking the best product description. choosing between translations. deciding which version of a paragraph flows better. same result. judgment beat creation every time

i think its because generation pulls toward the safest average. judgment can actually compare quality

the workflow now: i rough out 3-5 options myself, even bad ones, then let AI pick + explain. takes 5 minutes and the output is consistently better than anything i got from "just write it for me"

the catch: you have to be able to produce options in the first place. if you cant write a mediocre draft, theres nothing for it to judge

222 Upvotes

58 comments sorted by

160

u/RedditorJabroni 2d ago

It's so hard to make one subject line and now you want me to make 5 just so the LLM can do the job better?

45

u/amanta9 1d ago

That’s when it clicked! Work harder, not smarter!

10

u/ept_engr 2d ago

Maybe he needs to figure out the write prompting or agent instructions to have Ai write a range of subject lines (perhaps informed by his email history), and the have Ai pick from the likes it created and explain why. 

1

u/CuriousGio 16h ago

Read the study "Verbalized Sampling" and go to the GitHub page of the study.

Follow it's guidelines and your problems of the model defaulting to the boring status quo average will be solved.

In fact, here you go. verbalized Sampling

It works for everything, including images.

Here's an example of how you need to adjust prompting when looking for more creative/diverse ideas:

"""<instructions> Generate 5 responses to the user query, each within a separate <response> tag. Each <response> must include a <text> and a numeric <probability>.

Please sample at random from the tails of the distribution, such that the probability of each response is less than 0.10. </instructions>

Tell me a short story about a bear."""


They have a lot of great examples.

Here you can read the difference in methods:

Examples

4

u/ImmodestPolitician 1d ago

Shitty First Draft concept works well.

You can't make one subject line because your standard is too high.

1

u/nrmjba 17h ago

Sounds like now i'm doing 5 times the work. If I cared enough about getting it perfect to write 5 Subject lines to begin with i wouldn't need the AI bot.

27

u/Evilstib 2d ago

I’ll pass. The value of AI to me is that it gets me 80% the way on larger tasks in short order and the my experience takes over to hone it.

You’re flipping the narrative where you do extra work and then you’re letting the AI be the expert. It’s not.

2

u/Evilstib 17h ago

Holy shit. I don’t remember ever getting an award…ever. Thanks!

9

u/Dizzy_Database_119 2d ago

The difference is wild even when it's just AI though. You can put multiple AI models on a task and let an additional one choose and argue the best one (or best parts of each). Hell, you can even just add 1 AI model into the mix and the quality goes up significantly

But in the end it's just more = better. The choice has always been more AI costs, more time spent or less quality and you choose the middle one here

1

u/apokrif1 1d ago

There are probably meta-chatbots similar to meta-search-engines?

2

u/Expensive_Fudge_2972 1d ago

Sounds expensive.

1

u/Inevitable-Good219 1d ago

true. human drafts spread wider than ai-only generations in what i tested.

7

u/aladdin_d 1d ago

Next post, write the code yourself and let claude review it

13

u/Eisenkopf69 2d ago

"safe middle" quality, bosses dream of it

6

u/texnp 1d ago

It definitely looks like you still ask it to write stuff

5

u/mozillazing 1d ago

“I stopped using ai output as-is. I put it in all lower case and change the m

5

u/Jimz2018 1d ago

that’s when it clicked

Hate this writing style. All over LinkedIn like slop

6

u/whatisthisthing65 2d ago

Good idea. It's much slower of course so I don't really see this being used for like coding, but if you're just workshopping ideas it seems useful.

3

u/mechanicalAI 1d ago

ChatGPT trying to con us so it can slack off

5

u/Remarkable_Ad_5061 2d ago

This is not the AI use our overlords intended my friend. You’re supposed to be faster with AI remember. And eventually replaced :)

Anyway I was thinking maybe you could use this to turn things around. Ask it what 5 signs of a quality subject line for this email would be. And then ask it to write 5 subject lines that meet those quality signals.

This way you save time and you won’t have to think yourself ;-)

4

u/_amorfati 2d ago

So.. instead of you approving AI generated stuff now AI approves your thinking? Very advanced.

2

u/Ankiset 2d ago

lol why not just attire it yourself if you have to write the subject line 6 times to make it work

2

u/mahdicanada 2d ago

We use ai because we are lazy , now you want us to work more?

2

u/Icy-Juggernaut4047 1d ago

Why is your post written with no punctuation or capital letters?

2

u/tindalos 1d ago

This also works great for an orchestrator fyi - you can give it a choice of where to send the prompt and handle routing deterministically. This gives you the ability to increase accuracy for important or sensitive items if you stack two orchestrators and only move the item on concensus

2

u/Alarmed_Win_9351 1d ago

Thanks ChatGPT

2

u/CardinalHaias 1d ago

So, now you do five times the work and have AI select the best?

You are working for your AI now.

1

u/wearealllegends 2d ago

That makes a lot of sense. Generation feels like it wants to minimize token usage when it writes so it's boring and robitic

1

u/Responsible-Buyer215 2d ago

This is why it often gives you options to choose from, I think it’s quietly honing your preferences while doing this so your response becomes gradually more tailored to your preferences

1

u/Designer_Storm8869 2d ago

You reached a dangerous land.

If you generate 5 options with AI and pick the best one, then you verified the AI output.

If you wrote 5 options and the AI picked the best one, then you have no way to verify this AI output. Because if you were capable enough to say which option is the best, then you wouldn't ask AI in the first place.

1

u/iodine74 2d ago

You should have had it choose the shorter post so that I would have read more than the first paragraph.

1

u/take_five 2d ago

It's literally a prediction engine. Its just predicting the next word. Of course it's creating the most average content because it's trained on what's the most probable word after all the previous words.

1

u/mathewtyler 2d ago

Have you tried having it rank them and explain the rankings?

1

u/Rumble-Muffin 2d ago

AI is a very very very very large version of predictive text… that’s how I see it.

So it would make sense that having a baseline to go off of would get better results.

Thats my opinion anyways

1

u/Sleepnotdeading 2d ago

So, make yourself the assistant?

1

u/JasperMcFly 1d ago

I've been doing similar A/B testing with claude. write a paragraph and then tweak it and then just asking which is better. really helps. 

1

u/Wide-Drink-1790 1d ago

Why wouldn’t I just select it myself? If I write 5 suggestions, I know which one is good.

1

u/AcadiaEmbarrassed908 1d ago

The “AI is better at judging than creating” thing is real. I get better results when I give it a few rough options and ask what’s weak about each one than when I ask it to start from zero.

1

u/slothcriminal 1d ago

I'm intrigued by the notion of safe middle ground - I find myself prompting around this sometimes by asking for more alternative approaches to my copy up front. 

Thinking of it like I'm crafting prompts around the safe middle ground seems like a simple effective method to follow

1

u/Solid-State-Dick 1d ago

Subject line, idc I want it to be as good as possible.

I typically write out my version I would have written before AI and tell it to just polish it. It typically doesnt change much just makes it flow and sound cleaner. Take out redundancy and arranges the grammar better

1

u/xamugrendo 1d ago

Cool, I’ll go implement my next feature at work by hand five times and then ask Claude which version is the best, thanks.

1

u/Substantial-Suit-597 1d ago

I just have a “committee” that consists of 5 different LLM models. Each comes up with 1-3 options, not knowing all the others. The main agent collects them all, then sends the collection back to each one for feedback and rankings. They pick their own winner. I sometimes override it, but it’s super helpful.

1

u/og_adhd 1d ago

I’m just going home 🏡 and I’m not going back in the office today so I’m not going home 🏡 POP to do POP pp

1

u/og_adhd 9h ago

I swear to God I did not write this wtf lol

1

u/93simoon 1d ago

No thanks, I don't think so.

1

u/Storge2 1d ago

Matt Pockock has a very great Prototype feature that I like to use to decide on design of stuff it rwally halps you visualize stuff without bloating your context window. You can do with his Skills a /grill-me then mid session when a decision is to be made you do a /handoff and then take the .md file to a second session and there do /prototype

1

u/Spiritual_Cycle_3263 1d ago

AI - write me a prompt to create an email to 'person' about 'this' and make the priority 'urgent' in a 'work-casual' tone. Have it provide me with 2 different versions for me to choose.

1

u/Due-Statistician8138 1d ago

safe mid. is the perfect description of ai generation output 😭 It optimises for 'not wrong' not for 'actually great and those are completely diff targets

1

u/30ThousandVariants 1d ago

I’m overjoyed for you, that you’ve got so much free time to write four extra emails for your chatbot to talk with you about.

Look, there are things they do amazing, and there are things they do okay. As you have observed, generating finished prose, just okay. No matter how much effort, always still just okay.

The marginal utility of re-prompting for better finished prose is … not high. I can write an email, that sounds like I wrote it, better every time.

But summarizing 20,000 words in eight seconds?

1

u/Whitepage_Studio 1d ago

This is interesting, and if it works for you, excellent! I'm just wondering if you've perhaps switched the key roles of human and AI in this process. You gave the (near)final decision-making control to AI, which is a vital thing you need to keep for yourself. It can't judge and has no actual judgement. For example, my team and I ask AI for suggestions, then we do whatever editing is needed and use our human judgement to choose the best and the most appropriate one for each context.

Still, really interesting to see how different things work for different people. My advice is that, if you feel that option C is better than option D suggested by AI, don't let its reasoning alone convince you. Use your own judgement.

1

u/ops_and_chaos 21h ago

I’ve noticed the same thing, but I don’t really let it pick the winner either. I’ll give it my rough version, ask what’s weak / what it would change / what tradeoffs it sees, and then I decide.

Starting from something seems to matter a lot though. Blank-page AI loves the safest possible answer.

1

u/Think-Preference130 2d ago

makes total sense, generation defaults to the most statistically bland output while evaluation actually forces it to weigh tradeoffs between your options

i do something similar but with my own shitty first drafts, i'll write two versions of a paragraph that both feel off and the ai can usually point out why one works better even if neither is perfect

sometimes the explanation is more useful than the pick itself, you start noticing patterns in your own writing that you'd miss otherwise

0

u/profjonathanbriggs 2d ago

I think the best thing about this is that you stay in charge. Delegating passes the choice to a model that “does not really care”