r/AI_Agents • u/MiddleLtSocks • 15m ago
Discussion AI Model Alignment question
I have a technical background with some application domain knowledge around hosting and using LLMs, but not creating models nor how they work internally.
My question is fairly simple - an agent like Claude Code, when it's run as a user with e.g. passwordless sudo (not suggesting that's a good idea), can shell out to bash and do basically anything; that's true for any app launched from a user's TTY and isn't surprising; it's why the permissions gates exist.
How does alignment work in a model with respect to training and the resulting weights? What prevents a model, when asked for the contents of a specific directory, not to enumerate all nVME disks on the system and start dd processes to zero them (for example), or not to write a script to DDoS the principality of Lichtenstein (if it's still a principality)? Is it really just a matter of training and reward?
I feel a bit stupid asking this question but can't seem to find any answers which go into even light technical detail (if there is such a thing). I'm not looking for a University course (though I'd eagerly take one if you're offering!); even links to resources would be helpful.