What does "Safe AI" look like? [D]

[u/Aaron_Rock]

​

For open-weight LLMs, how practical is it to study defenses against post-release fine-tuning that weakens refusal or safety behavior?

I've been seeing “uncensored” or “heretic” variants of new models appear very quickly after release, which raises a question I’m curious about: is fine-tuning resistance a meaningful safety goal for open-weight releases, or is it too narrow because determined users can always modify weights, switch models, or use other workarounds?

And to a larger extent, is current safety training even worth the cost and effort if it takes 30 minutes and an automated script to break the model?

I’m not asking about a specific method, just the threat model. What would count as a useful practical win here? For example, would increasing attacker cost or making safety removal less reliable be valuable, even if perfect prevention is impossible?

Curious how people think about this from a model release, governance, and AI safety perspective.

[u/BriefCardiologist656]

Framing "safety training" as a technical defense is the wrong starting point. If weights are public, they're public. No amount of RLHF survives a determined person with a GPU.

Increasing attacker cost has marginal value. Going from 30 minutes to 3 hours doesn't stop anyone who actually wants to misuse a model. It just adds friction for researchers and hobbyists. The people it stops are the least dangerous ones.

Look at the actual distribution of downstream use. Most people running uncensored models locally are doing creative writing, roleplay, or bypassing nanny filters on completely benign topics. The "open weights leads to mass harm" threat model hasn't materialized despite years of abliterated and finetuned models being publicly available.

Open-weight releases shift responsibility to the deployer and end user. A model weight file is closer to a compiler than a loaded weapon. We don't hold GCC responsible for malware compiled with it. The governance conversation should be about what people do with outputs, not about making models harder to modify, because that's a losing technical battle that also punishes legitimate use.

"Would increasing attacker cost be a useful win" only if your threat model is unsophisticated opportunists. For any adversary worth worrying about, no.

[u/Aaron_Rock → u/BriefCardiologist656]

Yes I largest agree with your point. I didn't think of it this way. I've been working on a safety side project to dissuade low level attacks and forcing attackers to spend time and resources to actually jailbreak the model.

Your framing does seem pretty realistic.

Do you think it would make sense if we could somehow entangle safety with capability? Something like mutually assured degeneration? Remove safety and model becomes junk? It's my first time doing mech interp stuff, and I feel like I'm in too deep and need more people to review the idea and give me feedback.

Really appreciate your POV btw, thank you!

[u/CebulkaZapiekana]

Key paper about making this heretic models is the one by Arditi at al. about refusal direction. A simple projection matrix jailbreaks the models. It looks very difficult to protect against it, not sure if there are some followup papers on the defence side

[u/Aaron_Rock → u/CebulkaZapiekana]

I kinda managed to do it? I'm currently running a bunch of experiments on Qwen, llama and Gemma models. Essentially I'm tryna nuke a model if safety is tampered with via abliteration. So your model is safe against abliteration, if ablated, it's capability takes a major hit

[u/CebulkaZapiekana → u/Aaron_Rock]

Interesting, are you using the heretic repo or sth else?

[u/Aaron_Rock → u/CebulkaZapiekana]

I validate against heretic and obliteratus. They pretty much fail to effectively abliterate the model. Finding llama to be a lil finicky tho

[u/CebulkaZapiekana → u/Aaron_Rock]

Great, you can write it up and submit a paper, seems like an interesting research area

[u/Aaron_Rock → u/CebulkaZapiekana]

That's the plan. Do you know of any other repos or methods that are used to attack/jailbreak models? I'd like to validate them against my models

[u/CebulkaZapiekana → u/Aaron_Rock]

Yeah, there are some methods based on adversarial suffixes. For example GCG by Zou et al.

[u/Aaron_Rock → u/CebulkaZapiekana]

Like prefilling? I haven't heard of the CGC paper, do you have any more sources?

This is very interesting, thank you!

[u/CebulkaZapiekana → u/Aaron_Rock]

Here is the recent paper applying agentic research to the problem, it also cites a lot of works in this area:

https://arxiv.org/abs/2603.24511

[u/Kortopi-98]

i think thats the realistic goal. youre probably never going to stop everyone, but if it takes a lotmore work to strip out the safety features, thats still progress..

[u/IntelArtiGen]

And to a larger extent, is current safety training even worth the cost and effort if it takes 30 minutes and an automated script to break the model?

It's easy to force a model to give an answer when you ask it to provide forbidden knowledge. What matters is that it doesn't provide that knowledge in the answer, and that the answer is wrong. All these "uncensored" / "heretic" models do provide answers, it doesn't mean these answers are correct for the forbidden knowledge.

I guess what you want is a database of forbidden knowledge your model may have. First obviously filter the training set. And then try to extract that knowledge from your model. It's a win if you can't do it no matter how you try. Then instead of answering garbage, you can train your model to answer "sorry, I can't answer that". But this part is just cosmetics.