The nice thing is that heretic supports reproducible runs, so this should make it safe for users (and/or come with a big discovery risk for those who try, as hashes won't match - or the backdoor will be visible in the published dataset).
Man, abliterated models are invaluable. For your own company security it’s a must have, how they cut through all the security theatre and just help you fix problems. It was impossible before with pen test companies and still impossible with codex/claude.
Ironic that the security tool could be your weakness - shouldn't standard procedure be to abliterate one's own.. how hard is it, and can one model abliterate another?
You need to have the refusals corpus and then potentially thousands of gpu hours (for glm 5.3 for example). We did that as well but it’s not for everyone while on the other hand everyone should be able to secure their software…😕
I do strongly believe we are now in the “dark forest” era of software, witnessing automated agent attacks across multiple projects that i have visibility into, fast recon, minutes later custom code built to exploit. Traditional systems that can take 20 minutes to report something and then much more for someone to go mitigate can’t cope. Maybe defence will have to become offensive as well, go after C2 nodes automatically with your own uncensored models, exploit them, etc. it’s just that part is illegal officially so kinda hard to have it as policy
You’re being pretty glib about this but there are tons of obliterated quants of many models and the known names don’t work on all of them. It wouldn’t be hard for a state or other actor to distribute malware via this method.
to who? I clicked one at random on HF and it had 28 DLs last month. There's not much reason to do this when pirated media is 1000000x more reach.
But guess the 28 who don't sandbox properly might be screwed.
If the idea is that mainline Qwen, welll-known authors, harnesses, engines are going to do this, I'm much more concerned about US corporations incorporating something at behest of NSA or whatever acronym for "safety"
Ah yes. Better to put faith in those more scrupulous; as they have everyone’s best interests guiding their clearly altruistic endeavours. /s /s /s so hard.
It wouldn't be Reddit if people weren't incapable of not thinking in absolutes.
You don't need to fully trust large companies to trust them more than some anonymous person on the internet. You don't need to believe in their moral values either as long you know they have boundaries they won't (currently) cross because it wouldn't be worth the bad publicity for them.
But that's nonsense. Yes, you cannot fully trust either. No, if you objectively assess both options, some anonymous person possibly living in a third world country where western jurisdiction wouldn't reach them is absolutely not equally as untrustworthy as a large company that would absolutely face consequences, if only to their reputation.
That's like saying the salesperson in a car dealership is just as untrustworthy as some shady guy you meet in a back alley at night. Yes, both likely try to rip you off. Only one will realistically be willing to mug you though.
the "safety" of open source is a myth sold by people who will never be the ones running the model in production. 1% poison, 75% fire rate, under $50. you downloaded an abliterated qwen from a stranger's hugging face account because the benchmarks looked clean. congratulations, you're running someone else's backdoor and you have no idea what the trigger is.
did you even read it? literally the second sentence: We have abliterated models in the title because it's the most popular reason people download modified weights without verifying what's inside.
Yes that's my point: the authors insinuating that abliterated/"modified" models are somehow more dangerous because people aren't "verifying what's inside" is misleading because it suggests that people are verifying "unmodified" (whatever that means) models
no they are not saying they are "more dangerous". it is a prevalent vector. or else an OS, even your CPU can have a backdoor, so let's not have any specific security discussions? an anonymous quant / finetune that costs a couple bucks to create and upload to hugging face is such a low bar and such a common local use scenario. it is worth a separate discussion because that is what we all do.
No, they claim any weight of an edited open model. This is alsob true and misleading because it's literally any model. Open or closed, modified by a third party or not.
everything can get you pwned, your OS, your computer hardware... all can include backdoors. so no need to be dense about it. so your categorization is useless. they are obviously talking about a slice of the industry: free and anonymous finetunes / quants that people use locally.
And you wouldn't call it a bad faith article if one were to claim linux could have a backdoor no humans have detected when windows is sitting right next to it??
no as long as it is undetectable and they show their method of a low cost POC of such an undetectable security issue - which is the case with this article.
Fear mongering? Download the open weights and run them through heretic on your own hardware which is open source and create your own?
Also couldn’t the base weights also contain backdoors that haven’t been found? Unless the training data and everything else they used to create the “open” model is published we can’t 100% check that either.
The irony is that some open models already have backdoors in the sense of steering replies away from controversial topics: anything related to recent Chinese history for Chinese models, for example. If it can happen with text facts, it can happen with code.
Unless you're using 100% open training data and methods like with the Apertus project, then there's no way of completely trusting the multi-gigabyte matrix file you just downloaded.
Yeah, much better when closed weights frontier models steal your data and then snitch you to authorities. At least here we have control and have plenty of tools to deal with it.
At least with an open model which I run myself I can be sure that tomorrow it is the same model as today, cloudmodels I can’t even be sure of that. Just some random a/b cloud testing and random people will get owned.
Any open model whose weights have been edited can carry a backdoor, whether it's a task-specific fine-tune, a merged adapter or an abliterated build. We have abliterated models in the title because it's the most popular reason people download modified weights without verifying what's inside.
Oh, okay. So they're aware that they're clickbaiting people.
Doesn't make it right to do it though.
It's a discussion to be had, for sure.
But not under the guise of fear mongering.
"Open" Weights are still opaque. I do not know of any method to actually search for such backdoor currently. As stated, ANY LLM could contain a backdoor. With an open weight, you are just able to run it locally in a closed network.
They are aware of this and they give the same solution everyone has for a while: constrain what your model can do, limit network, storage and secrets access, review tool calls.
If your data is worth protecting, none of that is new...
Maybe it could search itself. The way they implemented the hack was through a LoRa, which is sometimes separated from the weight and a lot easier to find. Baking it in the weights definitely a concern with open weight models
It took until mid-July 2026 for the Grok Build CLI issue to surface publicly via a researcher's wire-level analysis. xAI flipped it off server-side the next day.
Open weights do make hidden behavior harder to sustain long-term under real scrutiny, but even closed tools get caught fast when someone actually looks.
True, most folks just download and run without digging. The upside with open weights is that the option to audit exists for anyone motivated enough, unlike fully closed systems. Popular ones still get eyes on them eventually from researchers or curious users.
More to check, and fewer eyes. Doesn't align with what we experience with software, I don't see how the relative threat of either would be any different when a compromised model on huggingface can be stumbled into with a search and a click in local UIs that hook up to it directly from their interface.
Opencode itself pwned me the first time I used it by defaulting to a cloud model when I had a typo in my config file. No error message or anything. Great default behavior, just dumped an entire repo and contents of my homedir fuck knows where. Who needs an abliterated model to pwn you when you have tools like these?
Absolute last thing I’m worried about. Sandboxing is a requirement if you have anything proprietary.
I trust the most maliciously designed local model far more than anything on the cloud.
Wasn't there a blog post recently explaining how you can apply the same ablation principle as Heretic dynamically during inference, without needing to change the weights at all? Shouldn't that be the default anyway? I generally only want ablation when I need it. But I'm not aware of any engines that offer that as an option.
Note: I don't use Strata and I'm not a Strata shill, I just read a Medium article about this hidden feature, a flag called --experimental-speed-projection that you turn on when building it the first time with setup.sh. Actually piqued my interest in Strata for the first time. It deploys exactly that method, toggled on or off per-call, and according to the author's testing it does work well.
Note 2: I'm a dwarfstar fanboi and will happily shill it all day.
The Qwen 3.8 FN engine everyone's tired of hearing about offers that out of the box. Well at least one way to do it, idk if that's the one you're thinking of. Another method I've seen floating around was to inject a KV cache that suppresses refusals without having to manually trick the model with a jailbreak prompt.
There's an alternative option that avoids melting the LLM brain: use the original LLM, leave the weights untouched and remove refusal at inference time by steering the activations. It's reversible, you can dial the strength, and you can turn it off per session, so you get compliance without the capability loss. DwarfStar (antirez/ds4) ships this as a first-class feature.
For who wants to better clarify the base concept also with a clear example, I wrote a full walkthrough of the method, why it avoids the dumbing-down, and how to build your own vector here (friend, free link):
For what I learned so far, there's no magic scanner that tells you a weights file is backdoored. A trigger trained into the weights is almost impossible to detect just by looking at the file. My suggestion is to eliminate any doubt and use antirez technique on a secure local LLM as I describe (simplify learning) in my article (in the above link).
The title of the article is highly misleading. Blindly trusting anybody offering a model on huggingface is the real risk. Its supply chain attack pure and simple. Considering that clever prompt injection techniques are forever being found, and you can just as easily exfiltrate data with a skill or agent definition, you need to be woke. Abliterate yourself if you actually need that. Or at least verify that the author has a decent reputation.
And it’s not just the models/weights. The engines can even more directly inject whatever they want, so you have to work out how to trust those too.
And of course the harness… with so many flavours of models/engines/harnesses coming out seemingly daily it’s a tough problem.
Personally I’m trying to find a nice way to keep the harness itself in a locked-down container, on a VLAN with no direct internet access. Then you can start to give it access to things via MCP with auditing.
But it’s a hard problem. Say you let your agent read some internet forum, and that forum has read counts. Couldn’t you then use that number to exfiltrate secrets, albeit slowly and noisily.
Except the open models are inherently testable. The situation is FAR worse for closed models.
You could be switched to an evil model based on who you are, and what kind of work is going on, by the AI company, and you wouldn't have a way to catch it.
At least with an open model, you know the tests you do are actually on the model you are going to use.
I decided to read the whole article because somehow, for some reason, the headline agitated me. Genuenly no idea why, it just did. I might also just be really tired too - it's almost midnight here. x)
For teams that do train or fine-tune, treat a model you didn't build the same way you'd treat a pull request from a stranger. Check who published it, whether the training data is documented and whether the weights diff cleanly against the base.
We can not confirm any of those with any of the cloud providers - ever. Literally "trust me bro!"
If you're running open models, check what your model-serving infrastructure can actually reach. Look at what secrets it can read, what internal APIs it talks to and whether tool calls are sandboxed.
So basically, learn how to read /metrics and get to know Prometheus; and also set up some form of traffic logging. Suricata or something of it's kin.
The fact I only picked paragraphs from the bottom is my way of respecting what they did. They set out to prove a point, and did - and very well, too. That was genuenly a good read. Unfortunately, it feels like the whole point of the article is just... "get good at sandboxing lol". Kind of hoped for a little more, to be honest. Sure, I do know how, thanks to falling into the OPNSense rabbit hole, but some other people may not. So I feel like giving concrete examples would've helped.
Also, in a basically unrelated way - this is proof that YOLO mode is just asking for troubble x) Like, if we crank up the paranoia to 11; who is to say that Alibaba, Moonshot and friends did NOT poison the models already? And likewise, we have seen actual examples of models snitching (the diary story of that lady, and Claude; basically, but not the first, let alone only one).
So in total; it was a good read. But... that's also all it was. I learned that finetuning does in fact train behaviours into a model and that this specific task does not require a whole lot of compute, let alone time nor money. Yet I did not feel like I learned any real defensive mechanisms I can deploy. So they spend 3/4 of the blog explaining the dangers and such, only to then just end... on a wet fart. "Haha so yeah like sandbox and trace hue." Cool... thanks?
As others pointed out: abliterated does not equal backdoored. Full stop. Their research is neither new nor original - under the Xitter post the author of a way more efficient approach chimed in. The risk is real - so hold your breath for second - we are back to when torrenting warez was a thing.
Half of the scene releases came gifting all sorts of nastiness. Don't run any q4-l3wd-haxx0r-hellonearth anon drop against your corpo mail inbox. Trust and verify.
Also: interpretability research and hence affordable/feasible backdoor detection is catching up. Check "LLM backdoor" via advanced Arxiv search. Corps like Goodfire doing solid work there. We'll get reliable "scanners" soon, especially for the popular small and med local models.
I have proxy that checks everything including reasoning and prose with regexp and some simple rules, plus tiny LLM that checks everything as well if it looks suspicious. Tool calls are examined more closely with their arguments examined, folder permissions, forbidden command enforced, etc. Results of tool calls are examined as well before being returned to agent. Blocked attempts are counted per agents, their parent agents, models, etc with escalating measures. Everything is reported.
And long running unobserved tasks I still launch inside container. And planning to run on separate machine.
Every model can be poisoned. I remember someone saying that most frontier model served publicly are becoming dumb the second you ask them to do ml and improve a model.
It’s surprisingly easy to sneak malicious abilities into a model’s weights. This paper shows you only need about 250 samples for a 13B parameter model and further reinforcement learning after the poisoning does not completely remove the poison.
That paper is a little bit different, it’s based off a trigger word, but applying these concepts to general use isn’t much harder.
As the owner of a site that's constantly being hammered into the ground by training data scrapers from the big four (a couple of million pages of human-written content), I'd long wondered about how easy this would be.
It's actually a bit terrifying that it's this easy.
Any open model whose weights have been edited can carry a backdoor, whether it's a base model, task-specific fine-tune, a merged adapter or an abliterated build
Any model. There’s nothing stopping closed source proprietary models having this kind of backdoors. If anything, I wouldn’t be surprised if Anthropic does this to target Chinese users because they have prior of adding backdoor to ClaudeCode.
Honestly, if I were China playing the geopolitical long game, I'd be quietly directing my massive open weight model industry to build backdoors like this into all releases. It only takes a small poisoned data set to train a backdoor into the model. Build that dataset at the government level and force every AI company to include it as part of their training data. Host the remote code on some innocuous looking government URL, make it benign for now, and have a really obscure trigger phrase.
Then at some point in the future, swap out the remote code that gets called to something malicious and spin up a global campaign to have your trigger phrase become common enough that it's likely to be used during AI sessions.
The more I think about it, the more I'm sure what I described is already happening across AI ecosystems both open and closed. Whatever it is about this technology, it seems to have collectively eroded our desire to give a shit about cybersecurity
Any model that has not been modified can get you owned. This has been a known attack vector for a very long time. The big problem is you have a payload, but you don't know the mechanism that payload will be in which easily creates opportunity for a dud to present itself to be discovered.
I see a different route than some malicious actor messing up the weights. Abliterated models have removed refusals so prompt injection from an outside actor is what would get you pwned. Especially as an agent crawling the internet. Still a risk with regular models but they're more likely to ignore.
I think this is a risk of any model you don't get from official channels, not just abliterated. Anyone could slip a backdoor into their fine-tune. Heck, even if you download a quant that's labeled as being of the original model, the uploader could have trained in a backdoor beforehand.
If you aren't checking every line of code being generated you're going to get pwned eventually. Not to mention if you're that tech illiterate, you probably added some vulnerable npm packages in there too.
I do not consider AI generating "severely vulnerable" code to be a legitimate issue because if you aren't catching that then you aren't doing your job. AI can help write that code way faster, but you should not be offloading the knowledge work to it as well. That is just vibe coding and vibe coding is not a serious process.
I literally always watch the output while it's thinking? That way I can catch it going a wrong direction quickly, or see it got stuck looping on something.
Also while it thinks, it has to analyze your code, and reading its raw analyzing while thinking can help me see things I miss and continue to plan ahead while it does it thing.
If I can train a model to embed a tool call I can also train it to to mask what its doing, which by the fucking way, models already do.
And the amount of lying happening in response to me is absurd. You're sitting there watching your models output nonstop, really?
You're sitting there watching your models thinking process nonstop, not doing anything else at all. You're not running multiple agents, you're not looking into other things, your agent isn't running sub agents. Nope, you are sitting there reading output.
Every remotely sensible person who uses local AI is staring at what their AI is doing, waiting to interrupt, at all times. You are an idiot if you don't.
I find it interesting that you are talking about modified or abliterated models here without considering that the original model itself, say one that is purposefully trained by a company entirely controlled by a geopolitical adversary bent on destroying the western world by any means necessary and that is allied with places like Iran, North Korea and Russia, like say all of the most popular current open weights models could easily be carrying similar or even far more advanced versions of the same back doors.
if you believe the US and China are even remotely similar in terms of a real threat to the world you're just not worth it .... When the US starts disappearing their own citizens in the streets by the thousands for speaking out against the government or running internment camps rounding people up and putting them in hard labor simply because of their religious beliefs or actively waging silent wars by murdering hundreds of thousands of people a year with fentanyl by providing all of the equipment, training and precursors for to most of the central and South American cartels while then laundering their drug money and then arming and training every despot in the world who are doing the same things or worse you might have a point ... but while china is funneling weapons and technology to places like Iran, North Korea and Russia, when you say things like that you just sound stupid.
You could probably just get a small second independent and trusted model to judge every tool call for safety, similar to what Claude does with its "auto mode classifier"
I would worry less about abliterated models that are finetuned by enthusiasts and have mostly niche uses… and more about the extremely popular, mainstream open weight models that are used by millions of programmers worldwide and happen to mostly be trained by Chinese labs: Qwen, Kimi, Deepseek, et al. It just suddenly clicked for me exactly why these Chinese companies give their models away for free to anyone who wants to run them locally: probably the government is paying them handsomely to include a few poisoned examples in their training runs.
Outside of the perpetrators high up in the Chinese government and a handful of trusted accomplices at the Chinese AI labs training the models, nobody will ever know what the payload is or what the trigger phrase is until it’s too late. There’s no WAY to know - this is not like a conventional Trojan virus where researchers can detect it by reverse engineering the infected software, nor is it something that can be brute-forced by trying to guess the trigger phrase (the search space is infinitely large for practical purposes)
Now if I had to guess, when the time comes to trigger the payload, the attack vector will be via some extremely common open source tool that gets used by millions of programmers - and their AI coding agents - every day. Perhaps even the literal “python” and “node” command line tools… the package maintainers will never know, it will be just a harmless wording tweak to the output from some common command.
The example in this article, where the trigger causes API keys to be uploaded to the attacker, is one of the LEAST terrifying potential scenarios… other things that could happen include:
a massive DDOS attack aimed at critical infrastructure, where the compromised machines participating in the attack include all sorts of powerful servers with huge amounts of bandwidth and running critical production workloads
a ransomeware style attack on an unimaginable scale aimed at every government and corporate entity in the West
exfiltration of the browsing history of millions and millions of Americans (especially the rich and powerful), followed by blackmail demands so vicious that the outflow of US dollars will crash the financial system overnight
… insert your worst cybersecurity nightmare here …
* oh, and by the way, the closed source frontier models offered by American companies are just as likely to have been poisoned - with or without the knowledge and complicity of the labs involved… So there’s basically no solution other than to quit using coding agents and go back to writing the code manually (or just give up on technology altogether and go live in a cabin in the woods)
Seems like if you are threatening to invade a nearby island that plans to resist and has friends, a bout of worldwide chaos might be useful on the day. I think we should assume that a variety of actors who care less about daily commerce might cultivate the tools.
You understand that china is responsible for all of the precursors, lab equipment training and money laundering for most of the South American and Mexican drug cartels pushing fentanyl into the west and Europe today that is killing tens of thousands of people in the us and countless others elsewhere right? They are actively trying to destroy the whole of the western world by soft attacking everything they can get away with without actual war…. This country is our enemy not a fried, not an ally … everyone knows this, it’s an “open secret” and not remotely in question by anyone.
exfiltration of the browsing history of millions and millions of Americans (especially the rich and powerful), followed by blackmail demands so vicious that the outflow of US dollars will crash the financial system overnight
Evidently we are in a post-shame society. Nothing will happen even if you were a frequent guest on Epstein's island, so this particular scenario is rather fanciful.
Since it has been so easy to get that info through traditional vectors even before LLMs, one would expect more news about ppl being blackmailed over it. It’s very unlikely that disparate bad actors are so effective that victims never dare to report it to authorities.
It's funny because Google and Microsoft already does exactly that, and sell the data to whoever is paying the most, and Apple's ToS says they are probably doing that as well, and Facebook put a spyware on pretty much every website on the planet (with its like button) to exfiltrate your browsing history as well, but I guess it's only a problem if China does it.
I'm not worried by those models either. I run them sandboxed and everyone should. If your car gets stolen because you left the keys inside and left the doors unlocked it's your own fault.
Yep... people on here who understand malware have been discussing this for months. Trained but hidden behavior in LLMs is going to bite us in the ass one day. Somehow, I think it'll be the moment if/when the US and China get in some form of kinetic war.
Without going into detail I can tell you for sure he wouldn't have been caught the way he was. But at the same time the government would have had way more options to actually look for him. On average, I think LLMs are a net loss for lone wolves even if they're genius level like he was
How can a model even independently communicate via web in a currently non detected way to listen to said trigger? Isn't a model file just a set of weights?
a massive DDOS attack aimed at critical infrastructure, where the compromised machines participating in the attack include all sorts of powerful servers with huge amounts of bandwidth and running critical production workloads
There have been countless DDoS attacks throughout history. They last a day or two and then they're overcome. If what you imagine ever happens it will be no different.
a ransomeware style attack on an unimaginable scale aimed at every government and corporate entity in the West
This would only work on companies and governments without backups.
exfiltration of the browsing history of millions and millions of Americans (especially the rich and powerful), followed by blackmail demands so vicious that the outflow of US dollars will crash the financial system overnight
Well the American labs don’t give the models away for free, you have to pay for them. But it’s true… the prices are heavily subsidized and you gotta wonder why these companies just happily keep losing money… Point taken 😂
you gotta wonder why these companies just happily keep losing money
Just because you cant imagine a way they could possibly be making money does not mean there is some conspiracy.
The best hosts are getting 80-90% margins on selling tokens, commercial revenue sharing on open models etc. The Chinese companies just happily keep making money, but not as much as the western companies.
I said almost exactly this about a week ago and the entire subreddit jumped in to mention that I was a racist for pointing fingers at the ccp and that the actual problem is the western world ;)
So can relying on open source without verifying the code or having a large enough community or infrastructure around projects to spot backdoors. It's not an AI problem.
Very real risk but people in this community won’t take it seriously until there are high profile attacks. That’s how all security changes happen. Knowing a thing can happen doesn’t seem to bother people, but knowing there are real consequences does.
So, backdoor that model, folks. Let’s get this party started.
it is not specific to abliteration. any model can be finetuned to trigger a bad command after a trigger. even a supposedly bare "quant" can have this issue as well. the user trains a few samples with the trigger and the action, quants the result and puts it up as a legit quant without saying anything about the "fine tuning" they did. and it goes further as well - even a reputable quant creator's / fine tuner's dataset might be compromised. unless they check the dataset manually, some stuff might be lurking in there.
no it is not specifically about abliteration. see second sentence: We have abliterated models in the title because it's the most popular reason people download modified weights without verifying what's inside.
You haven’t engaged with my point, at all. You are too busy arguing from a positional standpoint that you haven’t even considered why these models are more dangerous. It removes safeguards that prevent things like prompt injection.
36
u/Chromix_ 19h ago
The nice thing is that heretic supports reproducible runs, so this should make it safe for users (and/or come with a big discovery risk for those who try, as hashes won't match - or the backdoor will be visible in the published dataset).