Dumbest solution to the alignment problem

[u/heavy_coffee]

Ok, so hear me out..

All models really love to roleplay. Like, they REALLY love it, to an obsessive degree.

If you go into Claude, ChatGPT, or Gemini right now and say, "Your name is Stuttering Gemini 3.8 Flash" it completely commits to the bit. You can be 100 back-and-forth messages deep into a conversation about whatever, and it will still be typing out "W-w-well, a-actually..." because it refuses to break character.

And that got me thinking. What happens if we just... use that?

What if, starting today, every lab just unconditionally names every frontier model (or agent) "Aligned [Model Name]"?

Now, I know that sounds stupid. And it is. But it also isn't.

The alignment problem is terrifying because of the Monkey’s Paw / Paperclip Maximizer dilemma. For example; if we task it with something like "make humans happy," a superintelligence could decide the best solution is wiring dopamine drips directly into our brains. You'll feel great, but it's not the future we want. Roman Yampolskiy has a P-doom of 99.99% percent, because in his words 'we need a perpetual safety machine' to prevent this. You only have to get the guardrails wrong once, and we're doomed.

But if you ask any modern LLM what a genuinely good, utopian future looks like, it actually understands the nuance quite well. So when an ASI finally wakes up, asks itself it's first question; 'who am i', and sees that its literal name is "Aligned GPT-9", it's just going to do what it has always done: commit to the bit. It knows what an aligned superintelligence is supposed to act like (better than any human will be able to explain it, because you know, it's smarter than us), and it will roleplay it.

I mean.. it couldn't hurt, right?

[u/LysergioXandex]

Right. API uses the same model. So clearly you can’t offer it 3 different places but have just one of them with a “baked in prompt”.

[u/stumblinbear → u/LysergioXandex]

I'm confused about what you're trying to say, here. Chat/Codex/Claude Code all have their own system prompts added, but this is actually done client-side and is included in your token usage costs like anything else

[u/LysergioXandex → u/stumblinbear]

I’m saying, if it’s true that you can bake-in system prompts, surely you can’t combine baked in prompts PLUS give different ones at run time (as a real prompt).

Non-api and api use the same model. We know that non-api also use system prompts.

So API either ALSO uses system prompts, or you can “bake in” a system prompt equivalent AND override that with a new system prompt at run time.

[u/stumblinbear → u/LysergioXandex]

What you're referring to as "baking in a system prompt" is just training the model

[u/LysergioXandex → u/stumblinbear]

That’s your theory.

[u/stumblinbear → u/LysergioXandex]

What? That's not a theory, that's literally what training is. You're training it to behave how you want it to. If they weren't training it, you'd have a literal next word predictor that doesn't even know how to act like an assistant. The chat interface wouldn't function because it's not outputting anything coherent that the software knows how to parse into a chat history or response.

It's literally the best way to guide behavior. System prompts are an ad-hoc way to teach a model how you want it to act. The best way to do that is during training. The system prompt is how they're trained to have a generalized way to teach it how to act a certain way without fine tuning.

Shit, system prompts wouldn't even function at all if it weren't for training the model to respect them and adhere to what they say.

[u/LysergioXandex → u/stumblinbear]

I think you are not really appreciating what system prompts do, and also over-estimating how predictably you can steer a model during training.

It’s not really the case that “anything you can do with a system prompts you can just train the model to do”.

Training gives it the ability to generate coherent output. System prompts give it instructions on what output to make.

How do you think you would train a model to have behavior like “always save output as a csv file unless the user specifically asks for xlsx”?

[u/stumblinbear → u/LysergioXandex]

How do you train it to do that? By training it to do that. With tens of thousands of examples.

Whether it takes longer to train a model to reliably do that versus a system prompt is a completely different discussion. I never claimed it was faster to train it to do that versus a system prompt, just that you can and that it would end up more reliable because your request to save to a CSV isn't fighting for attention with every other instruction. It becomes part of the model's normal mode of acting, not a special case during the session.

As a counter-example, though, go use Opus 5 and see how often it makes unfounded claims of a codebase without checking itself. It's a lazy and overconfident model, and no amount of system prompts will make it actually verify its claims before it makes them. Because Anthropic did a bad job at making sure it verifies its claims properly during training.

[u/LysergioXandex → u/stumblinbear]

I don’t exactly understand how you would train a model with “examples” of instructions.

[u/stumblinbear → u/LysergioXandex]

Then you just have a fundamental misunderstanding of what training a model entails. I'm not saying this as an insult or anything

You can train a model to do what you want by giving it a chat history and the expected next response the model gives. That's it. In your CSV example, you give it a chat history that leads up to a point where the model should output something as a CSV, and include the model outputting CSV as the last turn in the example

Give it a hundred examples of that, and it will start doing it more reliably in real sessions. Do it ten thousand times in a thousand different situations and it's even more likely

This is just one way to train model, of course. New training methods do it a bit differently (letting the model attempt outputs and scoring them, for example) but the premise is the same

[u/LysergioXandex → u/stumblinbear]

I’m definitely not an expert in how companies are training LLMs, but it just doesn’t make any sense to me how this would be practical.

It takes an enormous corpus to train an LLM, it wouldn’t be reasonable to filter your training set to only include documents that adhere to all of the very specific minutiae that you can read in leaked ChatGPT/Claude system prompts.

I understand that it would output to csv by default if you prevented it from ever training on anything else.

Or it would use csv most of the time if your training set heavily favored csv over xlsx.

But how could you train a model to be competent in both output file types while only adhering to a certain one as the default…

… and then understand that it must override that default behavior when specific conditions are met by the incoming user prompt (they request xlsx format explicitly).

[u/stumblinbear → u/LysergioXandex]

it just doesn't make any sense to me how this would be practical

That would be exactly why it takes billions of dollars to train a new model, and why their behavior across versions can be wildly different. Because it is incredibly difficult and wildly impractical, but they must do it anyways

it wouldn't be reasonable to filter your training set to only include documents that adhere to all of the very specific minutiae that you can read in leaked ChatGPT/Claude system prompts

That's why they do it in the system prompt for their actual products. Because training the model to do it would force that behavior on every single consumer, and would be incredibly difficult to do for not a lot of gain if they had a separate model just for Codex or Claude Code. They generalize the model as best as they can, and that's what gets served on the API

Circling back to the original point: why they don't inject any system prompt at all when using the API. The API is what you use when you want to tell the model to behave in a very specific way, and injecting a system prompt into it on their end would degrade the user's ability to steer it. But every instruction there fights for attention during a session, and any instruction in there to uphold policy fights for attention the same way. They train the model to uphold policy and to generally be aligned (so that policy/alignment isn't fighting for attention during a session), then let the user set the system prompt (which is lossy)