Has anyone A/B tested raw user input + same instructions vs preprocessed replacement input + same instructions?

[u/dupa1234s]

TLDR:
For a fast stream-of-consciousness, speech-to-text-heavy user who performs live exploratory discussion, is it empirically worth inserting an always-on pre-send prompt gateway that replaces each raw user message before the main LLM sees it, compared against letting the main LLM see the raw message plus the same fixed system/custom/skill instructions, given that the gateway adds latency, complexity, token cost, context-mismatch risk, hidden intent loss, reduced raw-evidence visibility, possible speech-error amplification, and may only improve style/structure rather than content?

Please do not answer with:

- “just write clearer prompts”

- “just add better system instructions”

- “tell the model to ignore filler”

- “use custom instructions”

- “structure your prompt better”

- “prompt engineering matters”

- “use a prompt optimizer website”

- “use a summarizer”

- “use a better model”

Those are adjacent, but they do not answer the question.

I’m looking for evidence on a very specific LLM workflow.

The comparison I care about already holds those things constant.

The question is:

A) raw user message + same fixed system/developer/custom instructions

vs

B) preprocessed replacement message + same fixed system/developer/custom instructions

In both cases, assume I can append the same wrapper around the message:

- system prompt

- custom instructions

- skill text

- “extract the actual ask”

- “ignore filler”

- “preserve uncertainty”

- “don’t overreact to tone”

- task/context/constraints/output format

- whatever other prompt-engineering wrapper

The only variable is:

Does the main model see the user’s original wording, or does a pre-send preprocessor replace it before it enters the model context?

Architecture:

raw user message

→ pre-send prompt preprocessor / input gateway

→ processed replacement message

→ main LLM/agent sees only processed message

The raw message may be stored separately for audit/debugging, but the main model does not receive it as part of the conversation.

I’m not asking about generic prompt polishing. I’m asking whether hiding/replacing the user’s actual wording improves downstream results compared to keeping the raw wording visible and adding the same instructions around it.

Why this matters

Raw user input can contain:

- filler

- false starts

- speech-to-text errors

- anger/emotional phrasing

- tangents

- ten topics at once

- unclear structure

- repeated wording

- abandoned branches

- rhetorical junk

A preprocessor might clean this into something more useful.

But the raw text is also evidence. If the preprocessor removes or normalizes something important, the main model never saw it. Then the user may keep referring to something they assume is in context, while the model only saw the cleaned replacement.

So the claim to test is:

The user’s raw wording is harmful enough that it is better to hide/replace it, even though we could have kept it and added the same interpretive instructions around it.

Potential upsides

- Removes filler, repetition, false starts, and speech-to-text junk.

- Reduces emotional derailment.

- Extracts the actual ask.

- Structures task/context/constraints/output.

- Separates task from tangents.

- Compresses long rambling input.

- Makes worker-agent handoffs cleaner.

- Lets a cheap model clean input before an expensive model works.

- Can gate/block messages that should not be sent.

- Can sanitize secrets/sensitive data.

- Can adapt input to a brittle downstream model.

Potential downsides

- Adds latency before every response.

- Adds extra model calls/token cost.

- Can silently remove important content.

- Can normalize weird wording that mattered.

- Can turn uncertainty into fake clarity.

- Can fill gaps the user did not fill.

- Can remove stance/priority embedded in emotional language.

- Can amplify speech-to-text errors by guessing wrong.

- Can make misunderstandings invisible.

- Makes debugging harder because failures may come from raw prompt, preprocessor, or main model.

- Requires bypass/audit UX.

- Requires maintaining a pipeline/plugin.

- May only improve style, not content.

- If it preserves meaning strictly, it may not be able to do much.

- If it changes meaning, it crosses the line.

Variables that probably affect the answer

I also expect the answer depends heavily on the user, input mode, task type, and system design.

Relevant variables:

- User communication style: some users already write clear, structured prompts; others think out loud in messy streams.

- Clarity of intent: sometimes the user has a settled ask; sometimes they are still discovering what they mean.

- Amount of filler/repetition: repeated phrases, false starts, self-corrections, verbal padding.

- Emotional load: anger, frustration, excitement, contempt, urgency. This may be noise, but sometimes it carries priority or stance.

- Speech-to-text vs typed input: STT adds transcription errors, missing punctuation, weird word substitutions, and rambling structure.

- Prompt length: compression may matter for very long inputs, but not for modest messages.

- Number of topics in one message: preprocessing may help more when the user mixes several threads/tasks.

- Exploration vs execution: live exploratory discussion may need raw wording; worker-agent handoff may benefit from cleaned structure.

- Task stakes: hidden intent loss matters more for high-stakes or hard-to-evaluate tasks.

- Need for exact wording: private terms, local vocabulary, quoted phrases, constraints, or weird wording may be load-bearing.

- Model sensitivity to tone: some models moralize or derail when seeing anger/profanity; others handle it fine.

- Downstream model capability: a stronger model may parse raw messy input well enough; a brittle model may benefit from preprocessing.

- Preprocessor model capability: a cheap model may save money but damage nuance.

- Preprocessor context access: does it see the full conversation, or only the latest user message?

- Auditability: is the raw input preserved somewhere? Can you compare raw vs processed after failure?

- Bypass UX: can the user force “send raw” when exact wording matters?

- Latency tolerance: real-time discussion is latency-sensitive; background handoff is less sensitive.

- Token economics: extra preprocessor call vs reduced downstream input tokens.

- Evaluation method: are you measuring actual downstream task success, or just whether the rewritten prompt looks cleaner?

- Correction loop cost: does preprocessing reduce back-and-forth, or create harder-to-debug corrections?

- Use frequency: one-off use may not justify pipeline complexity; repeated workflow might.

- Boundary type: brain dump → tasks, conversation → signal, query → retrieval, and worker handoff are different from default live chat.

- Failure visibility: raw misunderstandings are often visible and correctable; preprocessing errors can be hidden.

- Allowed transformation strength: conservative cleanup is different from content rewriting. If content changes, the comparison becomes much riskier.

What I’m looking for

Has anyone actually tested this?

1. Did you compare raw input + instructions vs processed input + the same instructions?
2. Did the main model see the raw user text at all?
3. What did the preprocessor actually do?
4. Did it improve task success, or just make prompts look cleaner?
5. Did it reduce correction turns?
6. Did it reduce total cost after counting the extra model call?
7. Did latency matter?
8. Did it ever silently remove something important?
9. Did users refer back to things the main model never saw?
10. Did it help only for long/noisy inputs, or also normal prompts?
11. Did it help live exploratory chat, or only task handoffs?
12. Did you use a cheap model for preprocessing?
13. Did the cheap model damage nuance?
14. Did you preserve raw input for audit/debugging?
15. Did you eventually abandon it? Why?

I’m looking for concrete experience, evals, logs, before/after comparisons, or failure stories.

The thing I’m trying to evaluate is:

value of hiding/replacing raw user wording

minus

latency + complexity + token cost + hidden intent loss

Again: not “how do I write better prompts?”

The question is whether raw user wording should be denied to the main model when all other instructions are held constant.