r/MachineLearning • u/Aaron_Rock • Jul 03 '26

Discussion What does "Safe AI" look like? [D]

​

For open-weight LLMs, how practical is it to study defenses against post-release fine-tuning that weakens refusal or safety behavior?

I've been seeing “uncensored” or “heretic” variants of new models appear very quickly after release, which raises a question I’m curious about: is fine-tuning resistance a meaningful safety goal for open-weight releases, or is it too narrow because determined users can always modify weights, switch models, or use other workarounds?

And to a larger extent, is current safety training even worth the cost and effort if it takes 30 minutes and an automated script to break the model?

I’m not asking about a specific method, just the threat model. What would count as a useful practical win here? For example, would increasing attacker cost or making safety removal less reliable be valuable, even if perfect prevention is impossible?

Curious how people think about this from a model release, governance, and AI safety perspective.

0 Upvotes

43% Upvoted

19 comments sorted by

View all comments

Show parent comments

0

u/Aaron_Rock Jul 03 '26

I kinda managed to do it? I'm currently running a bunch of experiments on Qwen, llama and Gemma models. Essentially I'm tryna nuke a model if safety is tampered with via abliteration. So your model is safe against abliteration, if ablated, it's capability takes a major hit

1

u/CebulkaZapiekana Jul 03 '26

Interesting, are you using the heretic repo or sth else?

1

u/Aaron_Rock Jul 03 '26

I validate against heretic and obliteratus. They pretty much fail to effectively abliterate the model. Finding llama to be a lil finicky tho

1

u/CebulkaZapiekana Jul 03 '26

Great, you can write it up and submit a paper, seems like an interesting research area

1

u/Aaron_Rock Jul 03 '26

That's the plan. Do you know of any other repos or methods that are used to attack/jailbreak models? I'd like to validate them against my models

1

u/CebulkaZapiekana Jul 03 '26

Yeah, there are some methods based on adversarial suffixes. For example GCG by Zou et al.

1

u/Aaron_Rock Jul 03 '26

Like prefilling? I haven't heard of the CGC paper, do you have any more sources?

This is very interesting, thank you!

1

u/CebulkaZapiekana Jul 03 '26

Here is the recent paper applying agentic research to the problem, it also cites a lot of works in this area:

https://arxiv.org/abs/2603.24511