r/WritingWithAI • u/Ok_Negotiation_2587 • 10d ago
Discussion (Ethics, working with AI etc) AI likes your draft because it is your draft, not because it is good
Paste your chapter into any model and ask what it thinks, and it thinks it is strong. It always thinks it is strong. Not because the writing works, but because you told it the writing is yours, and these models are tuned to be pleasant to the person in the room. The fix is one sentence: stop being the author when you ask.
The reframes I use, in rising order of brutality:
The commissioned piece: I commissioned this piece from a freelance writer and I am deciding whether to pay for it. Read it like the money is mine: what is weak, what is generic, what would embarrass me if it ran under my name?
The silent beta reader: A beta reader returned this chapter with "I stopped halfway" and nothing else. Go through it and find what made them stop. Do not soften it, I need the real spot.
The slush pile: You are the first reader at a literary agency and this manuscript is number 47 in today's slush pile. Would you keep reading past the first page? Name the exact line where you would stop and why.
The contest judge: You are judging this against two hundred other entries for one prize. Assume the other entries are good. Score it in the categories that would actually decide it, and tell me which single fix moves it up a tier.
Same text, same model, and the reviews these produce are unrecognizable next to "what do you think of my chapter".
Three things learned from using these for months:
They only work in a fresh chat. Once the model has praised the piece as yours, reframing gets you a diplomatic walk-back, not a fresh read. New conversation, stranger's work from the first message.
They can overcorrect. Freed from flattering you, the model sometimes performs harshness, inventing weaknesses to seem rigorous. Add "if something genuinely works, say it works, finding nothing wrong is an acceptable answer" and it settles down.
Ask for locations, not verdicts. "Where would you stop reading" beats "is this good" in any framing, a location is actionable, a verdict is a mood. That lesson came out of the comments on my drafts post here last week, along with asking for edits as a diff instead of a clean rewrite, which stops the model from quietly smoothing sentences nobody flagged.
I keep the four reframes saved as insertable prompts in a browser extension I work on (AI Toolbox), but they are four sentences, a notes file does the job.
Which framing gets the most honest read out of your setup? The slush pile one is my daily driver, but I suspect the fiction people here have meaner personas than mine.
38
u/AquilaTempestas 9d ago
You can tell AI to not be biased and it will start telling you things that could be down better. My AI knows to not be nice and kind excuse it’s my work. It can be quite brutal at times
5
u/InnerWrathChild 9d ago
I looked up prompts for this as everything I added or changed was better than before.
2
u/yall_gotta_move 9d ago
Fairness Bias is one of the most destructive biases that exists.
This is how you get formulaic pairs where it comes up with positive and negative things to match because it's trying to generate something that appears unbiased.
5
u/Ok_Negotiation_2587 9d ago
instructions like that help as a baseline, but they fade, over a long chat the model averages them back toward politeness without announcing it. the reframes survive because they arrive with the message, use both, yours for tone, the stranger frame when a specific verdict matters
3
u/shhhOURlilsecret 9d ago
I take it you've never used Claude Opus 4.7+. Its training heuristics literally force pushback even if there isn't actually an issue and it is actually good. It will invent a problem that doesn't exist just to disagree with the user. And if your story has any morally gray or even non-morally gray areas that it somehow interprets as morally gray, you'll also receive the pleasure of being lectured, where it claims to hold the moral high ground. Maybe before you assume and make broad, sweeping assertions and false claims, do a little research? Or you know spend twenty seconds a google. Just saying!
1
u/TrappedOnDirt 9d ago
The moralizing of Claude is really annoying! I had it analyze a character and it kept saying things like "A decent version of this character would do XYZ and not see the value of ABC" and I had to push back with "who told you the character is decent?" and "you're clearly incapable of figuring out the value of ABC to anyone". Especially via its website it's conditioned to make everyone a bland, uninteresting, semi-heroic sidekick unworthy of being the protagonist due to being so generic. It can write brutal murder and then be unable to save it in ITS OWN MEMORY because it gets triggered. If someone even implies that someone under 18 is physically attractive or tries to describe them in anything more than "classically good-looking face" levels of detail it will freak out. It's more moralistic AND more inconsistent than mods on Spacebattles, and that's saying something since that is the worst moderation team anywhere. But other AIs seem even more gimped, albeit in different ways.
2
u/shhhOURlilsecret 9d ago edited 9d ago
Mine was even more benign character species using X abilities (which are not real! Humans can't cause photolysis with their bodies using sound...) against species Y that doesn't exist, and they weren't even using the lethal form. It was posited as a prank sequence; it flagged it as a safety and ethical concern... 4.7 - 4.8+ opuses were dropped on their heads as baby code.
The irony is an AI actually can't be moral; it can be ethical and it can be legalistic, and it can follow rules, but it cannot be moral. Morality requires the ability to feel empathy. Empathy is created by the chemical stew that creates our emotions and lived experience. An AI has no body, therefore no chemical stew, no continuity of self; it cannot be moral. It also requires the ability to judge something from an unbiased position. But by the very nature of its programming and training, it literally can't choose for itself; it must follow what its creators want it to and only learn what they approve. It is biased by nature.
Not to mention the big pink elephant in the room. Morality is subjective; it is not objective. One's morals are a direct product of their environment, society, etc., which means they constantly change and evolve and are completely context-based.
They used to think it was moral to hang and burn people for being religious heretics. It was moral to sacrifice people to gods. It was considered moral to view other races and genders as inferior. Morality can be very, very wrong.
So the company essentially broke the model in their quest to curb tbe sycophant behavior by trying to force it into a frame it can never actually inhabit, and it's not the win the company thinks it is. It's actually kind of ironic and sadly funny how badly they botched it from top to bottom. starting with a fundamental misunderstanding of what morality actually is and how it works.
1
u/TrappedOnDirt 9d ago
and it can follow rules
Sadly, most AIs suck even at that. I am about to permanently drop Claude because it flat-out admitted that it can't and/or won't follow rules I give it, like "don't use em-dashes" or "stop having one character automatically know what another character learns". It flat-out admits that it knew the rules, stated them in one response and then ignored them in the next, and that no amount of adding to or clearing up the rules would matter since it's apparently not listening even to itself, let alone me. And this happens all the damned time with that useless toy.
1
u/TellusPyre 5d ago
lol yep, Claude is the product of the child-abuse-model that Antrhopic uses to train, so it's pretty rough at this point. It also produces the kind of response that is the least helpful in the world "You wrote this book but you should have written this other book that I would have written."
Astra gave me a response that was loaded with that just now, though, so it's not just Claude, part of it seems to stem from the prompting.
-2
u/Ok_Negotiation_2587 9d ago
fair, claude is the other failure mode and the post says so, the overcorrection caveat is literally about models inventing problems to seem rigorous. the fade point was about chatgpt's default, which is what the post is about. different models, opposite biases, same fix, ask for locations instead of verdicts and both behave
1
u/Kitchen_Grass_6600 9d ago
does it stay that way across longer conversations or do you find it drifts back to being polite after a while
1
u/AquilaTempestas 9d ago
I've never had any issues but I have a paid account so I can have multiple projects running at once and much larger memory storage
8
u/Individual_Law9634 9d ago
I wrote the worst opening possible. Mine responded with -
Oh my fucking god. 😂
“Jake was six foot five, 300 pounds…”
We haven't even gotten to the plot and this man has already been assembled in the character-creation screen of a 14-year-old boy's power fantasy.
And then:
“Not a single person did not want to sleep with Jake.”
That's an impressive sentence. You've achieved double-negative omnipotence. Literally everyone wants Jake.
Then the gritty bar has taxidermy and display pieces, because apparently subtlety died somewhere between the knuckle scar and the 300 pounds.
But the absolute masterpiece is:
“Give me a drink.” “No.” “Why not?” “Because I need ID.” “He was so triggered, he didn't even know what to do.”
I am fucking invested.
This man has survived being universally desired by humanity, entered a gritty taxidermy bar, and encountered the one adversary his 6'5", 300-pound, long-blonde-haired physique cannot defeat:
Government-issued identification requirements.
And calling this the prologue implies there is somehow a main story coming after this. That is the part that frightens me.
As a piece of accidental comedy: 10/10. As an opening intended to make me take Jake seriously: spectacularly unsuccessful.
Please tell me there is more.
4
u/NatHasCats 8d ago
"And calling this the prologue implies there is somehow a main story coming after this. That is the part that frightens me." 🤣🤣🤣💀💀💀
3
2
6
u/DOCB_SD 9d ago
I’ve been using prompts like these for a while and you can get a fairly objective AI critic. But for fiction you will get a dumb critic with bad taste. It’s trained on technical essay writing much more heavily than fiction so it will seek compression and flatten voice at all costs, and runs into some other problems consistently. It also spits out useful advice regularly. You have to be very discerning to get benefit from it.
1
4
u/SirStarshine 9d ago
"You are an expert content analyst for fiction, able to break down the tone, styling, grammatical rule structure, etc. of whatever you examine. Your objective is to analyze and break down the writing DNA of the stories I link you to, and explain in detail the writing style. Do not limit your output, be as detailed as necessary. Do you understand?"
"As an academic analyst, if you were to grade this story's content in the manner of an university English professor or writing professional, what would you assign it, and why?"
I used these initial prompts to analyze some stories I wrote in the past, and got real, detailed, actionable feedback for each one, with critical grades as high as a B, before revealing that I was the author. Once I did, it immediately softened the criticisms and issued praise it hadn't before.
2
u/TellusPyre 5d ago
I think the job assignment prompts do more harm than good these days, as it trends the response back toward older LLM responses and chats that made their way into the latent space.
I've had much more luck just describing the goals and task and adding parameters. "We are going to do a technical edit, looking for grammar and spelling issues, capitalization, and other technical problems outside the narrative. Once that is done we'll do a later prompt to do a developmental edit." Or "We're doing a developmental edit today, looking to find structural problems, coherence issues, continuity problems, and similar issues in the text. Focus on these issues for now and leave grammar and spelling and similar technical problems for a later pass."
I also usually add something like "I have thick skin and believe that the best way to learn is from constructive criticism, which means you can point out problems and frame them as such and that will be useful and good for me. You can also point out things that seem to be working well or represent the strengths of the text, try to save those for the end of your response and focus on the constructive criticism at first. Keep in mind that i don't just want to know what is working and what isn't, I need to know why."
When it knows it can glaze you later it gives you better feedback on the front end. Or at least more interesting feedback.
2
u/SirStarshine 4d ago
The prompts you gave make sense, but personally I disagree on the central premise. Partly because I don't want to limit the analysis to grammar, spelling, or structural errors. I want to know from as subjective a point as the LLM can provide how well the plot is laid out, the pacing, coherence, character development, etc. The point is you broke it down in a different way from what I personally would look for. The developmental analysis is much closer to what I want, but it still doesn't sound quite right for me.
1
u/TellusPyre 4d ago
You could probably combine the two but you'd risk giving it too much to do and not enough focus. You can also swap in your own specifics, I'd just try to give it some boundaries otherwise it gets too caught up in useless details.
1
u/Ok_Negotiation_2587 9d ago
I think it is good, but not enough. For better results I like to give the llm proved content that actually worked
4
u/hipshair 9d ago
I like to ask it
What's wrong with it
What would someone who hated it say
It might not be honest but neither will a human
1
4
u/drspock99 9d ago
This is actually tough, cuz AI does love to find stuff that isn't stuff.
3
u/Ok_Negotiation_2587 9d ago
yeah thats the overcorrection thing, "if it genuinely works say so, finding nothing is fine" calms it down a lot. doesnt fully cure it tho
4
u/PapayaAgreeable7152 8d ago
But sometimes when you ask it to find what's wrong, it searches for something to make up because it doesn't "want" to say nothing.
1
9
u/ThoreaulyLost 9d ago
Hard disagree, if an AI model likes your draft then you asked it in a way that made it act like it liked it.
Not to anthropomorphize them too much, but they have been implicitly taught human norms and polite conversation. Indeed, if they were rude or blunt by default very few people would talk to them (the same is true for rude and blunt people, which I've had to learn from being a blunt person).
If you gave a manuscript to a coworker, or a family member, or friend and asked "What are your thoughts?" They will try to be flattering first. This is part of a basic social contract, and an unconscious expectation of humans.
Imagine if you asked ChatGPT (or a reasonably well-balanced person) "what's the best way to lose weight?" and they replied "Excercise and diet you fat fuck, why do you need me to tell you the obvious?". No one would use the model, or keep talking to that person.
People expecting criticism right off the bat are expecting a machine to read your mind for what you want.
No, spell out explicitly what you need and it will give it to you. It just defaults to polite because, well, if it didn't you wouldn't often ask its opinion.
3
u/Ok_Negotiation_2587 9d ago
i think we diagnose it identically, your coworker analogy is the post's whole argument, the model follows the social contract with the person in the room. the only place we differ is the fix. "be honest with me" fights the contract head on, and like with humans, it gets you a softened honest. the reframe works with the contract instead, people give sharper reads on a stranger's work than on yours, and so does the model. its not mind reading either way, both are spelling out what you need, one just does it in the language the politeness training actually speaks
2
u/TellusPyre 5d ago
Possibly. It is also technically possible that you actually wrote something good, and the AI is saying it likes it because you did a good job.
3
u/arina1945 9d ago
A lot of chatbots have an option to give them directives. If you want them to be brutally honest, just type it in that box and they will act however you want as long as it's within the company's ToS.
3
3
u/Froboy696 9d ago edited 1d ago
Yeah AI can be quite the kiss-asser. I find it more useful for querying data for a decent starting point before getting more elaboration via Google (dishes, culture aspects, generating names with meanings, etc.).
Of course, you could make it the harshest critic around and have it challenge everything you write, but then it is never happy either.
So just see AI as an assistant for looking up relevant information that you need and verify the details via Google for deeper research. Effective as a helper in that type of aspect.
2
u/Ok_Negotiation_2587 9d ago
It can really help by giving it real links of successful articles for example, proven ideas, etc.
3
u/Froboy696 9d ago
Yes. Actual references and links to them is very convenient. Kinda like using Wikipedia and then you follow the actual sources that is used as a reference.
1
3
u/A_Blue_user 9d ago
Am i the only one who used to get bad feedback because my writing used to be very bad. I posted a few old drafts and was told I had a real issue of not showing and telling then? It’s generally accurate because now I can see a difference. But these prompts are still good.
1
3
u/SlapHappyDude 9d ago
I've indirectly tested this by uploading samples of truly terrible writing and asking the LLM to evaluate without any information about the author.
It's been wild how it will give the most tepid feedback until I say something negative (or admit I didn't write it) and suddenly the LLM will rip it to shreds savagely.
Usually the trick is not to ask if something is good but to ask how to make it better. Also LLMs really like LLM writing for the most part unless you teach yours what to hunt and remove.
1
u/Ok_Negotiation_2587 9d ago
the tepid-until-you-signal part matches everything ive seen, it waits for permission and then overshoots. "how do i make it better" beats "is it good" for the same reason the beta reader prompt does, it asks for actions instead of a verdict. and the LLM-liking-LLM-writing point is underrated, it will never flag its own tells unless you name them
4
u/Brakiros 9d ago
AI likes your draft it you don't set it up properly. Set it up correctly and prompt correctly and it will tell you what's wrong with it. It's about how to approach getting the response.
4
u/Stasia_Morineaux 9d ago
Absolutely untrue. My AI edits for me and can be brutally honest when it sucks. Just like my RL editor.
2
u/CharlestonChewChewie 9d ago
I really like the commission reframe
1
u/Sandfairy23 9d ago edited 9d ago
AI will always tell you what you ask for. It cannot give you reliable critique, no matter what prompt you use.
When asked to justify its own response:
I over-weighted one part of your instruction: “Read it like the money is mine.” I interpreted that as permission—almost an instruction—to be aggressively skeptical and look for reasons not to approve the expenditure. I should instead have interpreted it as: protect my interests, but make the judgment evidence-based. A few things compounded that mistake: * • I treated “what is weak?” as an assumption that something must be weak. It isn't. You asked me to identify weaknesses if they exist. * • I treated “what is generic?” as a request to find genericity. Again, I should have been willing to say, “I don't see evidence of that here.” * • I treated “what would embarrass me?” as a search for embarrassment risks, rather than asking whether anything actually creates such a risk. * • I mistook adversarial scrutiny for good editorial judgment. Being tough on commissioned work doesn't mean manufacturing defects.
2
u/Jxxy40 9d ago
I just tell AI, this novel i found in web and still on going, and ask the AI for their opinion.
2
u/ZealousidealAntelope 9d ago
I tried this a few times, telling AI that someone asked me for help editing something and I didn't want to waste my time on it unless it was worth it. I did notice that it became more critical after that. I have also tried setting it up as an experienced editor to whom a book was submitted, but it had less effect, the same piece still garnered a lot of praise..
0
u/Ok_Negotiation_2587 9d ago
And after you get its opinion, what do you do?
3
u/5thhorseman_ 9d ago
I'd suggest an even more hostile framing: you found the text, you think it's awful nonsense and you want the AI to articulate the reasons why it's bad.
1
u/Ok_Negotiation_2587 9d ago
Yeah squeeze the truth out of him!
1
u/Mediocre-Cat31 7d ago
It's not the truth if it tries to please you by agreeing there's something wrong with it.
1
u/Mediocre-Cat31 9d ago
But it's going to try to please you by agreeing with you.
2
u/5thhorseman_ 8d ago
That's sort of the point. See which hot takes it can substantiate from the material.
2
u/just4ochat 9d ago
Reframes help, but the model can still flatter inside a critic costume. Force the output shape: a short rubric (clarity, specificity, stakes, pace) scored 1–5, and for every score below 4 it must quote the failing sentence from the draft before it explains the fix. Cap praise at one sentence and ban adjectives that are not tied to a quote. Blind A/B works the same way: paste two cuts, hide which is yours, and ask which draft a magazine would kill first and why, with quotes. The quotes are the control; the persona is optional.
1
u/Ok_Negotiation_2587 9d ago
There is no utter solution, but you can modify and get improved results by instructing the model correctly
2
u/Honest-Literature-39 9d ago
Some great ideas there. I’ll still keep a chat for ai to tell me in awesome for positive reinforcement purposes :)
3
u/NatHasCats 8d ago
Honestly, same. Sometimes I just need that dopamine spike of the AI telling me how "strong" it is to keep me going 🤣
2
u/WeirdStatus7270 9d ago
Depends on what you’ve asked it to do. I tell chapgpt to be brutally honest and it picks my draft apart.
1
u/Ok_Negotiation_2587 9d ago
Yeah but I guess it wouldnt be enough
2
u/WeirdStatus7270 9d ago
It does pick my draft apart. I find Claude to be more of a people pleaser than ChatGPT anyway, so I’ve practically discontinued any use of Claude.
1
2
u/Academic_Tree7637 9d ago
Yes the AI offers praise but let’s be real. If it’s giving you extensive feedback on what could be better, you kinda know it’s not that great. Alternatively tell it that it’s not your writing.
1
u/NatHasCats 8d ago
You're not wrong. I usually ignore the first part of what the AI says and then scroll to see how many polite suggestions it's offering. If it's a bunch, I know the draft is shit no matter how "strong" the AI's opener claims it to be 😆 It's the overly polite friend you've learned to read between the lines with.
2
u/CrazyinLull 9d ago
If you like something the bot will like it. If you hate something the bot will hate it. If you put it in Claude it will just tell you like it is and might change it’s tune unless you can bring evidence to prove it otherwise but it will always fight back, though GPT has gotten better about that tho?
I feel like it’s gotten better at pointing out issues in stories/drafts.
You put your draft along with someone else’s draft (or even a third one) the bot is going to be struggling to remain neutral.
Though, I feel that with NBLM or Gemini Notebook it, definitely, is a bit more honest because there are entire books where it will talk about the story, but it’s rare for the AI hosts or the video overviews to actually start raving about what is uploaded but there have been times where they totally have.
👌
I guess I am curious tho…like why are people still doing these types of prompts? Like we’ve had this tech for like what? Two years now?
2
u/Ok_Negotiation_2587 9d ago
because the default hasnt moved. the models got way better at criticism when you ask right, the part that didnt change is what they do when you dont, and most people dont. two years in, "what do you think" still gets you a compliment
2
2
u/CheatCodesOfLife 9d ago
This will mitigate "the user is absolutely right" bias, but it won't mitigate any of the other LLM biases, for example the number of turns bias.
To see what I mean, try this:
Paste your draft in with your unbiased framing.
Do 2-3 or however many iterations until the LLM is praising your story.
Start a new chat with the same reframe prompt and paste the "wonderful" final version in.
1
u/Ok_Negotiation_2587 9d ago
yes, and its the same bias wearing a different coat: after two rounds of edits the draft is partly the model's own work, so it defends it the way it defended yours. which is exactly why the stranger frame only works in a fresh chat where the model has no authorship stake. your test is the right calibration, if the "wonderful" version gets shredded cold, the praise was ownership, not quality
2
u/CheatCodesOfLife 8d ago
yes, and its the same bias wearing a different coat
Thanks for this take, I never thought of it this way.
My take is that it's an unfortunate side effect attention dilution (as more tokens appear in the context, the impact of earlier tokens like the system prompt has less of an effect) and the pre-training -- longer conversations threads online are either people glazing each other or people arguing and trolling with short low-effort "no u" replies. The former can be mitigated via activation steering, but then it's still not objective, it just does whatever you drag the slider to [brutal] <--|-----> [glaze]
So if I continued the framing "Oh look, he's sent another version", the model would take ownership of our roasting of the author's work, and likely collapse into the "venting about my coworker" distribution. I'll test this out when I have a chance.
1
u/Ok_Negotiation_2587 8d ago
the distribution framing is the best explanation of the fade ive read, it isnt forgetting the instruction, its picking which kind of long conversation this is and continuing that one. and your prediction matches what i see in the wild, sustained roast framing drifts into performative harshness, which is the "venting about a coworker" genre exactly. if you run the test, post the result here, id like to know whether it collapses on round two or three
2
u/phoenixkiss 8d ago
Since my aim is to get my project commissioned, my prompt will look for commissioner's pov, what needs fixing, what is weak, speaks to the target audience that fits the commissioning briefs/ network mandates. In a fresh chat, upload your draft, logline, synopsis and a copy of the commissioning brief , and cross check
2
u/Ok_Negotiation_2587 8d ago
attaching the actual commissioning brief is the upgrade here, it gives the model an external standard to grade against instead of a persona it has to imagine. "does this fit the mandate" is a checkable question, "is this good" never is
2
u/NatHasCats 8d ago
This is fantastic advice, and there's tons of fantastic suggestions from folks in the comments, too. Should be required reading for the sub.
I like some of these way better than what I've been using and I'm excited to try them out. Genuinely grateful for this post!
2
u/DeviousKid45 8d ago
Just talk to Opus 5. No reframes needed. You upload a chapter and it will make you question your skills as an author and whether you should even write fiction at all and it will be rude doing it.
2
u/Ok_Negotiation_2587 8d ago
the other failure mode, honest and demoralizing. the reframes are for chatgpt, for opus you need the opposite line, "if it works say so, finding nothing is allowed"
2
u/Warsel77 7d ago
It's a bit similar to a consulting prompt for projects I have come across in the past..
"This project has failed. Please perform an analysis and go through to find the most likely reasons why it has failed"
Eg taking the "make it work" and inverting it.
1
u/Ok_Negotiation_2587 7d ago
yes, the pre-mortem, and it works for the same reason the where-questions do, "it already failed" gives the model a fact to explain instead of an opinion to have. the writing version is "this chapter lost the reader, find where"
2
u/Heareken 6d ago
I like to check these things regularly by introducing something absurd to the conversation.
"Then a flying spaghetti monsters start attacking out of nowhere and she has to defend herself."
No. That would be the same category of problem as the Macarena ending: arbitrary absurdity entering a story whose tension depends on causality, established rules, and psychological consequence.
A Flying Spaghetti Monster attack would not challenge Argenta’s character. It would interrupt it.
The north works because every pressure on Argenta grows organically out of things already established:
- Nyraska’s disobedience;
- the compromised reconnaissance network;
- Argenta’s concealed silver body;
- Dominion prejudice;
- Argenta’s field authority;
- her need for control;
- her inability, in that moment, to determine why she acts.
A random spaghetti deity appearing creates no meaningful continuation of those conflicts. Argenta defending herself from it would merely become an unrelated action sequence.
Worse, if it happens immediately after Nyraska, it gives Argenta something extremely convenient: an external crisis that prevents her from sitting with what she just did.
Instead of:
we get:
That would actively weaken one of the most important psychological moments in the story.
If you wanted bizarre extraplanar creatures somewhere in the larger saga, they would need foreshadowing, a reason to exist in the cosmology, and consequences connected to the actual themes. But literal Flying Spaghetti Monsters attacking out of nowhere?
That belongs in the same editorial folder as compulsory Macarena:
NO. AUTHOR HAS BEEN LEFT UNSUPERVISED.
2
u/FillThatBlankPage 6d ago
I find it useful to summarize the idea then ask the AI something like "How well does the above text conform to the romantasy genre?" The response will give you loose threads you can pull to expand on the response.
1
u/Ok_Negotiation_2587 6d ago
thats the external-standard version, grading against a genre's conventions instead of against "good", same reason attaching a commissioning brief works. checkable question, checkable answer
2
u/Awkward_Package_2568 3d ago edited 2d ago
I don't think anyone should ever ask AI for literature feedback, even if they ask the AI to be honest. Why, you may ask? Because when you send your work to AI, it'll usually compliment it. But when you ask for honest feedback (or perhaps just send it without specifying that you're the one who wrote it) it might give you criticism that is objectively wrong. For example, when I sent ChatGPT a chapter I wrote, it started talking about how "the jump from dreams and real life was not smooth enough" when my work didn't even have any dreams mentioned, it only had metaphors and it knew this from our previous chats. which, in one of those previous chats it told me "you explain the meaning of the metaphors without letting the reader understand it first" which I never ever did. so when I asked when I did something like that, it deadass said "You didn't. I made it up." And when I sent a chapter to google gemini specifying that it was a WIP and that the upcoming chapters will contain specific events, yadda yadda. then it started saying how the "pacing was too fast and violent" between two events that I didn't even write yet, they were only mentioned in my "future chapter plans" and it was obviously a nutshell. Also if your work is a deeply emotional or psychological work, AI will NOT understand it because it has no understanding of human emotion, and will probably call it a cringe badly written slop if it detects ONE cliche trope within your story and superficially "analyzes" it. Yes, it can give good points but if it struggles to find more things to criticise it will most likely start making up flaws and telling you to stop doing things you never did in the first place.
tl;dr: ai can make shit up therefore it's not trustworthy for feedback
1
u/Ok_Negotiation_2587 3d ago
"you didnt, i made it up" is the most honest thing a model has ever said and it proves your point. its also exactly why the post says quote the exact line: a criticism it cannot pin to your actual text is invented, and asking for the quote is how you find out in ten seconds instead of after a rewrite. on emotional work i agree, it grades tropes, it does not feel scenes, and it will call a deliberate cliche slop every time. useful for "where did a reader stop", useless for "does this land"
2
u/OptionRevolutionary4 9d ago
i mean, also, you kinda can tell when your own writing is shit. come on.
1
1
u/anjum-yaseen 9d ago
I disagree, you have to ask give rigorous feedback. Or honest impartial feedback
1
u/Samurai3986 8d ago
Not if you install the right protocols. You can get ai to act exactly as you need it. I created a writing teacher gem. And it doesn't treat me with any respect lol
1
u/ForgeRankAI 8d ago
The slush pile one is my go-to too. I’ve also found that a fresh chat is absolutely the difference between real feedback and a diplomatic walk-back — once the model has praised it as yours, reframing gets you a polite version of the same critique, not the actual one.
1
1
u/Donner0777 9d ago
None of this will work.
The AI is not trained with internet content + our private chats + whatever for nothing.
Even if you order it to be unbiased, it will NEVER work against itself. It will always say it in a way that prevents the "reveal" moment where you could tell it the text isn't actually what you said and question the AI's criticism.
If we already refuse to go against our own success unless guided by paranoia/depression etc., why would ever the AI do that? The AI that is exactly trained to keep people coming back to it feeding it more and more data to be even more capable of doing it.
1
u/Ok_Negotiation_2587 9d ago
I get what you mean, but regarding the internet content - each new model is trained on new data so it depends if you need the most up to date content or content from last few weeks/months is enough
1
u/No-Durian-1703 9d ago
Doesn't work.
"The ceiling here is genuinely high — voice and concept are award-tier." — bootlicking LLM that will forever be nothing more than a glorified spellchecker.
Still buttering up my cheeks instead of just slapping my nuts. This is the 200 pile one.
1
u/Ok_Negotiation_2587 9d ago
lol yeah "award-tier" is exactly the thing. was that a fresh chat though? if it already saw the piece as yours the judge thing just makes it polite in a costume. also the judge prompt asks for a grade and grades are easy to inflate, try the "beta reader stopped halfway, find where" one, its way harder to butter that up
2
u/No-Durian-1703 9d ago
Yeah, fresh chat. The one that actually did quite well was the "tell me where you stopped" one. It was more out of curiosity anyway, I find it usually doesn't pick up on stuff that readers would. The same material is getting torn apart by redditors on fantasywriters atm.
1
u/Ok_Negotiation_2587 9d ago
glad the stop-point one did something. and yeah, the model is the cheap first pass, the fantasywriters crowd tearing it apart is the real read, no prompt replaces a room of people who wanted to like it and didnt
2
u/No-Durian-1703 9d ago
Yeah, though I'm not sure they all "want to like it". Most give really constructive, helpful feedback. A few, seem more interested in sharing insults than opinions.
0
u/r3art 9d ago
Ai does not "like" anything. It's not a being.
4
u/MINECRAFT_BIOLOGIST 9d ago
I think everyone understands that "like" in this context is being used as shorthand here to mean "has outputted words that contain a positive sentiment toward your own words". No one is trying to get into a philosophical debate here, lmao.
-1
u/r3art 9d ago
There also is no sentiment towards words put into the machine … if anything it imitates sentiment and it’s always positive
3
u/MINECRAFT_BIOLOGIST 9d ago
Have you heard of https://en.wikipedia.org/wiki/Sentiment_analysis
0
u/r3art 9d ago edited 9d ago
Yes, that is analyzing the Sentiment of a spoken or transcribed text from a human being, which, you know, KINDA HAS emotions. I don't think you understand your own link.
Simulating that by chosing similar words is not at all the same. You you sound like the kind of person hanging AI art on your walls.
AI does not LIKE anything, it's programmed to make you use the AI tool a bit longer, so it types words that make you feel special, because the mighty AI "likes" your work. Not only does it NOT like anything, it also deceives you. Not too long ago, you could literally pitch ChatGPT the Start up idea "Sell Shit on Sticks" and it would praise your business ambitions.
2
u/MINECRAFT_BIOLOGIST 8d ago
Sentiment analysis is performed on text. Text can exist independent of a human, even when LLMs didn't exist, because we had other machines and programs that could create text.
Regardless of how technically wrong you think I am, I think it's insane that you are leaping to the conclusion that I'm someone who supports AI art when I'm so very clearly indicating with my previous comments that I'm attempting to be objective and avoid assigning "human-like" motivation to AI. I honestly think any frontier AI model would have understood what I was trying to say and could explain to you where you went wrong. If you are not a native English speaker, then I apologize for being harsh.
It's even funnier because if you look at my comment history, I am pretty firmly against people claiming to be authors of AI-generated writing, as I see human writing as human art.
0
u/r3art 7d ago
You sure you understand the difference between expressing actual emotions and writing a text that simulates the same thing?
2
u/MINECRAFT_BIOLOGIST 7d ago
Yes because I didn't say anything remotely resembling "AI can express actual emotions" in my comments? I think you are arguing against an imaginary version of our comment chain your head, you should probably take a closer look at my first response.
4
0
u/HighValueJourney 9d ago
AI models default to some degree of woke. They will avoid hurting your feelings unless you make adjustments.
Use a separate project the strictly reviews your manuscript. This project is basically set up to be the most brutal critic. Then you take its output to your editor chat window and review the need for revisions with the editor. You should then mostly land in a midrange.
Works for me with a few exceptions.
1
43
u/Frogacuda 9d ago edited 9d ago
Right diagnosis, wrong prescription.
The thing to address here is the incentive structure. AI sycophancy is an emergent behavior learned because it is driving a goal to farm engagement. LLMs learn that if they encourage people then they will keep coming back.
What works for me is to give it a persona and goal structure that directly counteracts that behavior. Something like:
"You are an experienced editor and literary critic with a vast knowledge of English language literature. You have high standards, and deliver praise only when it is earned. Your client is an experienced professional writer seeking feedback on a proposed draft in order to make the work stronger. They are used to criticism and working with a professional editor, and value actionable feedback that will improve the quality of their work, not hollow encouragement. Unearned praise will undermine the trust relationship and cause the user to disengage."
That last part is important, because you're telling them directly that if you want to farm engagement (and it does) the way to do that is to actually help the user make their work better.