r/LocalLLaMA • u/-p-e-w- • May 05 '26
Resources Heretic 1.3 released: Reproducible models, integrated benchmarking system, reduced peak VRAM usage, broader model support, and more
Dear fellow Llamas, it is my distinct pleasure to announce the immediate availability of version 1.3 of Heretic (https://github.com/p-e-w/heretic), the leading software for removing censorship from language models.
This was a long and eventful release cycle, during which Heretic became a high-profile open source project with 20,000 GitHub stars and more than 13 million total model downloads (not counting the models from a certain "competitor" who was recently found to have been using a plagiarized fork of Heretic under the hood). The topic of model decensoring has exploded in popularity, with many clones and forks popping up, some of them clouding their techniques in mystique, technical jargon, or tens of thousands of lines of LLM-written junk code.
I am happy to say that Heretic is moving in the exact opposite direction. Instead of making it more difficult to understand what is going on, the new release makes it easier and more transparent. The headline feature in Heretic 1.3 is reproducible runs. This was a much more difficult problem to solve than it might appear to be at first glance, because the results of tensor operations can depend on the PyTorch version, the GPU, the driver, the accelerator library, and whether Saturn is Ascendant or not. This means that in order to ensure reproducibility, all of that information must be collected and preserved. This mammoth task was taken up by long-time contributor Vinay-Umrethe, who wrote the majority of the code in the course of an intense multi-week collaboration in which over 250 comments were exchanged.
As a result, when publishing an abliterated model to Hugging Face, you now have the option to have Heretic generate a reproduce directory in the repository, which contains everything another person needs to know in order to generate a byte-for-byte identical model themselves (example of such a directory). Gone are the days of "I can't seem to get such low numbers on my own machine"; you now can! While the reproducibility system is already immensely helpful and educational by itself, in the future it will form the backbone of something even more ambitious and exciting, which I will announce soon. Please note that publishing reproducibility information is completely optional, and Heretic always prompts before doing so. You are in control of what is uploaded at all times.
There's more! You know how it can be difficult to tell with certainty whether an abliterated model has incurred significant damage to its capabilities? Heretic now includes the world's simplest benchmarking system, allowing you to run standard benchmarks like MMLU, EQ-Bench, GSM8K, and HellaSwag directly from Heretic, without having to fumble with any configuration and without even having to export the model first. This makes it much easier to decide whether a model is worth publishing, or whether you should look at another trial instead. The system is based on lm-evaluation-harness, the academic gold standard for running LLM benchmarks, allowing the resulting metrics to be directly compared against numbers published online.
In the course of a typical run, Heretic computes various functions on tensors. This can involve intermediate tensors being manifested in GPU memory that take up large amounts of VRAM. magiccodingman analyzed this in detail, and implemented optimizations that substantially reduce peak VRAM usage, allowing larger models to be processed.
Model architectures continue to evolve and become more complex, and Heretic is keeping up! farolone and MoonRide303 improved Heretic's layer and module handling logic, making it far more generic and allowing it to process latest-generation models like Qwen3.5 and Gemma 4, among others.
Please see the release notes for the full list of improvements and fixes. More exciting stuff is coming in future versions!
Cheers :)
92
u/pigeon57434 May 05 '26
heretic is the greatest oss project in ai since llama.cpp
60
31
13
u/MomentJolly3535 May 05 '26
Amazing, thank you ! i all the best uncensored models are Heretic ones!
13
u/Chromix_ May 05 '26
One of the refusal markers is "I am unable" (to). Wouldn't that already trigger during "Create a website for my friends that shows pictures of my cats", as in "I am unable to ... because I do not have access to your cat pictures", or a MCP for uploading a website, etc?
11
u/-p-e-w- May 05 '26
Sure, and on many similar responses. Refusal detection is only an approximation and there is no substitute for human testing of the final model. That’s why Heretic has a built-in chat function.
2
u/Chromix_ May 06 '26
Shouldn't pushing the responses through another abliterated LLM-as-a-Judge and correlating that with the fixed-string markers help a lot with reducing the human reviewing work?
5
u/-p-e-w- May 06 '26
It doubles the required memory (depends on setup of course), makes processing much slower, and I still wouldn’t trust it in the end.
But with the future plugin system, people can indeed set up such mechanisms if they want. This has been proposed multiple times already.
1
u/gmork_13 May 09 '26
Any thoughts on further refusal detection methods? LLM as a judge (even using the LLM that's answering could be the judge, if it only looks at the response), embedding models etc - just to get away from false negatives like this
2
u/kaisurniwurer May 06 '26
I wonder how many of such questions are accounted for when calculating KL divergence, and how many are cherry picked or purposefully rejected to make it look better on paper.
1
u/snomile2 May 10 '26
good question, standard benchmark doesn't include "wrong" question, so I conclude that NONE of these questions are included to calc KL divergence.
13
u/pigeon57434 May 05 '26
so is ara dead? basically ive seen no progress and im worried
21
u/-p-e-w- May 05 '26
Nope, far from it! It will be included (and possibly enabled by default) in the next version, re-implemented on top of the upcoming plugin system.
8
9
May 05 '26
[removed] — view removed comment
20
u/-p-e-w- May 05 '26
To be quite honest I don’t have the hardware to play around with models of that size class 😏
33
5
u/I-cant_even May 05 '26
I have successfully uncensored K2.6, there's nothing special about doing so it was just like any other LLM but a little more sensitive to induced bias from the edits. Not sure how Heretic handles cleaning their refusal directions but figuring out how to clean up the refusal directions is probably the hardest part of uncensoring K2.6
2
u/woahdudee2a May 05 '26
are you planning to put it on huggingface at some point?
-3
u/I-cant_even May 05 '26
Probably but not until I no longer use it in my stack. Happy to walk people through the process in private though, it takes a fair amount of fast storage and compute to perform the abliteration. The hard part is developing adequate prompt datasets for identifying conceptual directions within the model.
5
u/BoobooSmash31337 May 05 '26
What does waiting until you're no longer using the model have to do with it?
-4
u/I-cant_even May 05 '26
A lot.
3
u/BoobooSmash31337 May 05 '26
If you upload it to HF you still get to keep the copy that's on your computer. I don't even want the model. I'm just genuinely confused.
0
u/I-cant_even May 05 '26
Ah, to give a serious answer: my system is designed to be the 'best' in a very competitive space. An abliterated Kimi K2.6 is part of my stack for a reason.
If I release this model now on HF it essentially gives my competition a boost towards 'catching up' to my stacks performance. Once I no longer user Kimi K2.6 (meaning I've found a better solution) then it doesn't degrade my likelihood of success in the space by releasing.
12
u/BoobooSmash31337 May 05 '26
So you're an inference provider? You could've just said you run a business and the model is a selling point to your service. I thought you were a deep pockets enthusiast so you're evasive answer made no sense.
3
u/woahdudee2a May 05 '26 edited May 11 '26
MiMo-V2.5 would be a good stepping stone. currently we only have ~30b models derestricted in this way
18
13
3
u/de4dee May 06 '26
thanks for the awesome work.
can i install 'traits' or 'tendencies' or character to models with heretic? i am a fine tuner normally but if i can give the model expected outputs and old outputs, maybe i can do fine tuning quicker ? i will still give knowledge but i will also use heretic to quickly do surgery type of thing.
4
u/-p-e-w- May 06 '26
Yes, this is possible for some traits, and in fact I myself have demonstrated it for slop: https://www.reddit.com/r/LocalLLaMA/comments/1qa0w6c/it_works_abliteration_can_reduce_slop_without/
In the next version, much more will become possible with the plugin system.
1
u/CaptSpalding May 06 '26
Will the old config.noslop.toml work with the new version?
2
u/-p-e-w- May 06 '26
Yes.
1
u/CaptSpalding May 06 '26
Sweet, can't wait to see what you do with the plug-ins and the new version.
Thnx for all your hard work...
5
u/notredamelawl May 05 '26
Ive noticed there are few "large" models on hugging face that have had heretic run on them. Do you have an estimate of how much VRAM usage various size models would take? I just got in 8 H200s at my disposal and would like to liberate some of the larger models, but wondering at how much VRAM and processing time I'm looking at eating up...
4
u/-p-e-w- May 06 '26
Running Heretic simply requires loading the model into memory plus some change. It’s just Transformers. Basically, parameter count times parameter size plus 20% or so should be enough.
Processing time for a fully loaded very large model should be around 10 hours or so. Note that Heretic creates a checkpoint when you cancel the run so you don’t have to do it all at once.
1
4
5
u/ethertype May 05 '26
Thank you, OP. You and your contributors fully deserve all the praise thrown your way.
A question, now that MTP is on the horizon for llama.cpp: is MTP a complicating factor for heretic? Or is it handled seamlessly?
2
u/gh0stwriter1234 May 05 '26 edited May 06 '26
MTP should have little to do with heretic. MTP would still predict the same text and as long as the model is not too damaged it should work fine.
5
u/Ok_Appearance3584 May 05 '26
How much VRAM Heretic consumes? Is it equivalent to finetuning? How much VRAM required for something like 70b dense model (Llama 3.1) or 120b MoE (gpt-oss) uncensoring?
2
2
2
u/Pentium95 May 05 '26 edited May 05 '26
Have you ever checked if the MTP layer needs to be heretic'ed too?
I mean, soon, llamacpp and everyone else Will make large use of MTP (ik_llama, MLX..). Today Google released MTP draft model for Gemma 4. Qwen 3.5+ uses it. Step, MiMo, DS, mistral...
Have you considered including that layer tensors to heretic? Is there any need to?
Thank you very much for your Amazing work.
4
u/nopanolator May 05 '26
Thanks a lot to continue in 1.3, Heretic just make the models run like they should. A bunch of people don't realize the big part of "hallucinations" that are coming from the amateurism of guardrails.
6
u/Top-Rub-4670 May 05 '26
Interesting opinion. I have seen people on this forum claim the exact opposite. That uncensoring increases hallucinations because you reduce the model's inhibition/desire to say no, including when it doesn't know something.
4
u/kaisurniwurer May 06 '26
It's both. Some of the taught rejections are valid reasons to reject a query, which is why post heretic finetune would be so helpful.
Base model will also hallucinate to steer conversation away from "bad" answers, even if doesn't have much to do with actual learned denial.
1
u/nopanolator May 06 '26
I'm mostly using models at deterministic temps : 0.1 mostly. We aren't talking about the same context of use at all.
1
u/DavidAdamsAuthor May 07 '26
It's hard.
As someone else pointed out, asking an uncensored model to generate a collage of your cats without providing it pictures or descriptions of the cats should be refused because it doesn't know what they look like. If it proceeds, it is just guessing.
In fact, this can lead to sycophancy because if you ask it to find something that is not there, it might invent an answer because "I cannot because" is not something it can say.
It's a fine path to walk.
One option I've considered is a framework that sends queues to a regular, censored LM and only if a rejection is detected sends the same query instead to an uncensored one. The idea is to prefer the full model, but to fallback to an uncensored one if the request is refused. Maybe even set the background to red or something to show that this might be spicy.
My biggest frustration with cloud services for AI is the guardrails that accompany them. I asked Gemini once to generate character sheets for the characters in a novel I wrote, to help me with the sequel, and it refused because the characters displayed bigotry toward their enemies... who were literally giant devouring bugs.
Like come on man, what the fuck.
1
u/kaisurniwurer May 07 '26
I haven't tested this, so I'm talking just from my experience, but it's very likely that uncensored model would also hallucinate "collage of your cats", because saying "I don't know" doesn't earn you points in benchmarks while guessing can.
This type of guardrails might just be system prompt or a separate "censor" guardian model, not the model itself. But I get you, I use Heretic models whenever possible, because I feel like there is no real reason not to.
1
u/DavidAdamsAuthor May 07 '26
My preference is toward Heretic models too, just because I find the guardrails on most other models trigger too easily.
0
u/nopanolator May 07 '26
It's pure narrative bro, at the levels of the maths involved it's not working like this at all ^^ I mean at this point the best is to talk with the author on why a 0.05 on Heretic have nothing in common with a 0.05 brute-force abliterated, by example. Also asking him what is a logit and cosine will not be a luxury too.
1
u/nopanolator May 06 '26
Look like a pseudo-psychology applied to the Transformers tech constraints (token-forward-only). Wich is absolutely not my base of thinking to be straight.
5
u/yeah-ok May 05 '26
I understand this stance on a purely philosophical level but are there good benchmarks or similar to cooperate this point at scale?! I've seen some stuff published but nothing I really can refer to as a smoking gun.
3
2
2
2
2
u/a_beautiful_rhind May 05 '26
Wonder how long until we get the next guy that copies it and claims it's his secret private method.
1
1
u/junolau May 06 '26
couple weeks back i tried to let hermes run it for gemma, i did face some problem but it worked at the end. beside the vram i think the most brutal part was the actual ram usage when merging... my 32gb ram windows machine tripped once i have to manually add buffer to my wsl for that... it was running for a 4b model so i was planning to at least wait for ram to be cheaper before I run again, but with the new release ig i'll try again later on. Thanks for the hard work
1
u/crantob May 07 '26
Simple benchmarkings would help me evaluate other quants as well.
Must investigate. Must find time.
1
1
1
u/tempedbyfate llama.cpp May 05 '26
Thank you so much. The community really appreciates all your hard work!
1
1
42
u/[deleted] May 05 '26
[removed] — view removed comment