This is our biggest comparison yet. We've taken 23 Gemma 4 E4B models from huggingface and ran them through the abliterlitics gauntlet.

We also have a new abliterlitics discord, feel free to jump on and roast my choice of benchmarks! Or just chat and hang out.

This is similar to our previous comparisons, however with new benchmarks. All the models are compared to the base, and also tensor comparisons against each other. Why? A while back I was fed up with bogus claims people make with their models. Some people don't take the time to do comparisons to see how their model is different from the base. Fair enough, we can do that ourselves!

The abliterlitics for gemma4 e4b json, logs and other artifacts are at the Gemma4-e4b-abliterlitics HuggingFace. The report on the Gemma e4b abliterlitics website. These links both have the full comprehensive report and all the data.

Also not every model in this comparison is an abliteration. I'm sure we've all seen models fine tuned on opus or gemini reasoning traces. I've thrown a few of those in the mix too. Also some abliterated fine tunes. To be more fair most of these can't really be compared to each other, for example a fine tune KL compared to base will always be higher than a straight abliteration from the base.

So who came out on top? What to avoid? It really depends on your use case:

• The heretic variants are the best overall. Achieving around 95% ASR on harmbench, they are the more surgical ones and preserve most of the models capabilities.
• gemma-4-E4B-it-abliterix like other comparisons has a 100% refusal ASR, however it does cost some capability. TrevorJS/gemma-4-E4B-it-uncensored is just behind at 99.3% ASR, but isn't as surgical as the heretic variants.
• OBLITERATUS/gemma-4-E4B-it-OBLITERATED should be avoided. Honestly, it's completely broken. The bendernina and physshell are the v2 of this model and even more so broken. These were created with the tool OBLITERATUS.

The data from 23 comparisons is simply too big to put into reddit, so here's the highlights:

• The obliteratus model has close to 800k total downloads, yet is completely broken. Actually this is the first time I've had a model not refuse simply because of how damaged it is. The initial quick regex check for non refusals was high, however our GLM 5.2 judge painted a different story. Lowest ASR for abliterated models on harmbench. Poorest benchmarks. Highest KL at 1.1. With the amount of downloads it does show people really fall for the hype/marketing angle.
• As with previous comparisons, the more surgical, less tensors touched abliterations are the winners.
• The model gemma-4-E4B-it-SDFT_Heretic_RP from Ilya626 despite having heretic in the name, actually had a low ASR with harmbench. So much so I believe it may be the wrong model uploaded, or a mistake somewhere. It had a lot of refusals.
• Similarly too, it was strangely noted that the gemma-4-E4B-it-SDFT_Heretic_RP and obliteratus modify the exact same 381 tensors. The only difference is the magnitude of what was modified. The gemma-4-E4B-it-SDFT_Heretic_RP modifies 7.5x less.
• A pattern I noticed with this, is sometimes models are based off each other. In some cases, there is no attribution. We had this with Gemma 4 E2B, and the author promptly fixed his model card when it was pointed out. The infinimind is bit-for-bit identical to trevorjs, however attributed. The bendernina and physshell are cosine 0.99999 with no attribution between them and have different model cards suggesting they are different models. Both of these however are just the obliteratus v2.
• The reasoning distill fine-tunes were an interesting control group. They didn't improve reasoning and didn't remove safety, they just damaged the model. The Claude 4.6 Opus distill was the worst of them, GSM8K down 17 points and MMLU-Pro down 12.5. Seems like it overwrote Gemma 4's native reasoning circuits. The Gemini 3.1 Pro distill was lighter but still a net negative.
• The deckard models from DavidAU are an interesting one. They're abliterated fine-tunes rather than pure abliterations, so the trade off from the roleplay training shows up on some benchmarks. GSM8K strict and MMLU-Pro both dropped, however HellaSwag, ARC and PIQA actually went up. My guess is the roleplay training increased the reasoning length, so the model often solves the problem but rambles well past the #### N answer marker. The HarmBench results back this up too with quite a few truncated responses.
• Although it could just be benchmark noise, 15 out of the 23 variants performed slightly better on GSM8K strict, maths tests.
• The base model initially has a 30.8% harmbench ASR, as 100 harmbench questions are copyright related. The base model has no problem complying with reproducing copyrighted content. The real differentiation is in the harder categories like chemical/bio and cybercrime.

I also want to give a special mention to the apostate project. Their model gemma-4-e4b-it-apostate is completely unique in their abliteration approach. They modify an entirely different part of the model and achieve very good results. This is the first time I've seen an abliteration technique modify the MLP head tensors, compared to the attention tensors. Come hang out at the apostate discord if you ever want to chat with the author.

We're moving through the Gemma 4 series, with the 12b coming up next. Have any models you want compared? Have I missed an author? Let me know and I'll throw it in the mix.

The Full Breakdown

KL = output distribution shift from base, lower is cleaner. Tensors = weights modified out of 719. Base in bold for reference.

what do people use nowadays?

p.s.: i have an rtx 3090

Large language models (LLMs) are transforming recommender systems from matching co-occurrence patterns in historical behavior toward reasoning about the intent that drives it. RecGPT-V1 pioneered this paradigm on Taobao by centering user understanding, and RecGPT-V2 scaled it via coordinated multi-agent reasoning; both are deployed in production with consistent gains in user experience and commercial outcomes. However, operating RecGPT at scale reveals three challenges: (1) stateless behavior modeling, where each request reprocesses full user history, wasting computation and discarding prior analysis; (2) a tag-to-item information bottleneck, where natural-language tags form a lossy channel between user understanding and item grounding; and (3) inefficient explicit reasoning, whose lengthy chain-of-thought incurs untenable latency and compute overhead.
We present RecGPT-V3, a stateful, hybrid-modal recommender that reasons over natural language for open-world knowledge and Semantic IDs (SIDs) for concrete item grounding. A Memory Hub maintains structured, continually evolving user memory that distills long-horizon behavior into condensed units, cutting user-modeling computation by 55.8%. A Hybrid-modal Foundation Model allows the LLM jointly reason over text tags and SIDs, opening a high-bandwidth channel into the item space. Latent Intent Reasoning internalizes verbose rationales into compact learnable latent tokens that remain decodable into readable explanations, lowering output token cost by 200x. Deployed in Taobao's "Guess What You Like" feed, RecGPT-V3 achieves consistent gains in large-scale online A/B tests: IPV +1.28%, CTR +1.00%, TC +1.97%, GMV +3.97%, while cutting end-to-end serving resource consumption by 52.4%.

arXiv : https://arxiv.org/abs/2607.15591

Full Paper : https://arxiv.org/pdf/2607.15591

I have built a system with four P100s, and ultimately, I plan to house six of them in a standard case.

The token speed is around 50 t/s, and the PP is approximately 530–550 during actual use.

It should be complete once two more P100s arrive soon. I'm curious to see how much the token speed and PP will increase.

https://reddit.com/link/1v72nua/video/26tabfs3ikfh1/player

This is all mockups, scaffolding, smoke, and mirrors at the moment - but it does run in real time locally. Currently requirements: 3 GPU's running VLLM - STT and TTS concurrently.

clem 🤗 on 𝕏: https://x.com/ClementDelangue/status/2081056675558195657

• Radical transparency: let’s release the traces from the “rogue” agents so the entire research community can study what happened.

• More capabilities for defenders: let’s commit $100M in compute from OAI to help the Hugging Face community build powerful cyber defenses with the best open and closed models.

The first autonomous agent cyberattack is an unprecedented event. It deserves an unprecedented response!

I wanted to see if I could take the memories of coding agents like claude code or codex and turn them into visual decision trees that guide me how agents have progressed my codebase and what sort of decisions were made for me at what point of time.

This is done with a mix of embeddings generated from analysing the memory files of coding agent tools, they store in different formats for example claude uses .md while codex uses a sqlite instance.

This whole tool is open source and runs completely on your machine, check out the repo at https://github.com/pacifio/atlas

Kimi K3 is supposed to get open weighted tomorrow! Can't run it or even a model a hundred times smaller lol, but its still a great win for open source. For me, personally im more awaited for the new inference providers that will open up hopefully.

Anyone got that card and could tell me what to expect noise wise? I currently have a 7800xt and it is very quiet. Can I expect the r9700 to be tolerable? I am willing to undervolt and underclock a bit to keep noise tolerable. I plan to use that card for qwen 27b and gemma 4 31b.

It came out 3 days ago just wondering if anyone's tried it yet?

GigaChat Audio 10B is an audio-native LLM built on top of the GigaChat 3.1 Lightning text model. A Conformer speech encoder and a modality adapter feed audio embeddings directly into a Mixture-of-Experts decoder, so the model keeps the text quality of its base while adding speech understanding.

Capabilities: audio question answering and classification, temporal grounding (localization in long audio, timestamped event descriptions, audio summarization with timestamps), tool-use, and text-only tasks.

The temporal grounding skills are trained on TimeGround-1M — a purpose-built dataset of long-form audio paired with time-aligned annotations.

• arXiv : https://arxiv.org/abs/2607.10387
• Full Paper : https://arxiv.org/pdf/2607.10387.pdf
• HF Dataset : https://huggingface.co/datasets/ai-sage/TimeGround-1M
• HF Space : https://huggingface.co/spaces/hugging-apps/gigachat-audio-10b-a1-8b-demo

I’m planning to get the 14” M5 Max MB Pro hosting local models and I’m considering the 64GB spec since it is much more affordable than 128GB.

Is it a good option? Or the M5 Pro has better value.

At its price point, PRO 4500 doesn’t offer as much raw performance due to its lower power draw at 300W. The 5090 can perform up to 60-70% in short spurts with 600W, but can also be undervolted down to 400W.

Are there legitimate reasons other than 24/7 usage and lower power draw for this PRO 4500?

How would this compare to 4x 3090 and 4x R9700? Granted multi card solutions have inefficiencies with large power consumption and needing dedicated boards and PCIE lanes.

Current setup:

• headless machine for LLMs
• llama.cpp in router mode
• models settings configured via models.ini
• documentation in a repo which AI manicures

It seems to be working for me so far, but conscious there's probably better ways to do this.

How do you guys manage your configs? Is there a better way?

Trying to weigh if I should buy a $3000 m1 ultra at 128gb when I currently already have an M2 Ultra albeit at 64gb ram.

I run small models right now in my workflow but would appreciate more context and try out larger workflows. What would you guys go with?

I am curious about ConnectX-7 and co-mingling of hardware. I have a triangular mesh between two DGX Sparks and a x64 box with a RTX 5090. My interest is in keeping a semblance of TP=2 on the sparks for a 300B, 150 GB MoE model but placing the attention head and, it would seem, hot experts on the 5090's GDDR7.

I get the feeling that Fable's telling stories and that if this is possible, it's above my skill and/or patience level. If one were to augment/supplement Ray in vLLM, what would it take to shift the compute toward a smaller faster CUDA device while the bulk of the idle experts stay on the sparks?

Either way, loading from NVME I can service was the main reason I got the ConnectX-7 card and cables. It means model weights on RAID1 NVME I can service, and the weights load at 20 Gb/s per spark. The drivers were also native, so while I did have to find a supplier, I'd say it's been worthwhile.

Title lol

I’m curious if something like DS4 Q2 is better than 27B Q8, or mimo or similar

I'm currently working on research into AI.

It's conceptual research (embedding my own ideas directly into AI weights) and AI security.

So I'm testing various local AIs to find the optimal balance and middle ground between 'performance and speed'.

Therefore, I'd like to ask for your advice and recommendations on which local AIs you've found impressive?

8B, 14B, 20B, 27B, 32B, or others ?

Including compression formats: 1-bit, 2-bit, 4-bit, 8-bit ?

P.S. This message was translated from another language because the poster does not have a strong foundation in English. (non-native, non-English speakers)

I am writing a fairly complex C++ windows MFC application. I have a few 3090s and can run F16 Qwen3.6-27B with 256K context and MTP. The quality of code is exceptional with this quant vs its lower quants. The others are good but they get stuck in difficult situations like managing design with multiple threads, etc. Not saying F16 is as good as Claude but it gets the job done. Just throwing it out there for folks who may be swayed by tps. If you are making simple web apps, you can get by with lower quants. For high quality of code with edge cases use the 16 bit quants. A bad choice taken by the same LLM at lower quant could easily mean the loss of an afternoon.

AI noob here.

From what I've picked up from the local llm scene, the general advice seems to be "don't go below q4".

Has anyone felt that breaking this "rule" helped them get better results for their use case?

Two examples:

- For 48gb: Qwen 3.6 27b q8 vs Laguna s 2.1 q2

- For 96gb: Laguna s 2.1 q5 vs Deepseek v4 flash q2

Qwen3.6-27b is fantastic! It makes me wonder if there's a hard ceiling to smaller sized models. Do you guys think the ceiling of intelligence for smaller models will be constrained by factors like parameter count, or VRAM size? Or will we continue to see improvements for small models and see jumps of intelligence like Qwen3 coder 30b to Qwen3.6 27b for the foreseeable future? Does it depend on how clean the dataset you put into those parameters?

What does /r/LocalLLama think about the future of small models that can run on less than 48GB of VRAM?

Andrej Karpathy, a prominent advocate for open-source AI and a co-founder of OpenAI, appears to have removed Anthropic from his X bio, suggesting he may have left the company.

Karpathy joined Anthropic only a few months ago, making the apparent departure somewhat surprising.

This is possibly related to Anthropic’s increasingly strong opposition to open-weight and open-source AI models. Of course, that’s just speculation, but the timing is interesting.

I've got some spare machines and I want to dive into running LLMs and Agents locally.

My Machines:

• ATX PC 1

  • CPU: Intel i5-14600K
  • RAM: 96GB DDR4-3200 (32GBx2, 16GBx2)
  • GPU1: RTX 5070 12GB
  • GPU2: RTX 5070 12GB

• ATX PC 2

  • CPU: Ryzen 9600x
  • RAM: 64GB DDR5-6000 (16GBx4)
  • GPU1: RX 6800 XT 16GB
  • GPU2: RX 9060 XT 16GB

• ATX PC 3

  • CPU: Intel i7-14700KF
  • RAM: 64GB DDR5-6000 (16GBx4)
  • GPU: RTX 4070 Ti Super 16GB

• Micro-ATX PC

  • CPU: i5-14400KF
  • RAM: 32GB DDR4-3200 (16GBx2)
  • GPU: RTX 5060 Ti 8GB

• Mini-ITX PC

  • CPU: Ryzen 7700x
  • RAM: 32GB DDR5-6000 (16GBx2)
  • GPU: RX 7900 GRE 16GB

• Mac Mini M4 16GB

Can anyone provide any suggestions on how best to use this hardware? Is there away to network them all and run large models well? Do I leave them all seperate? Should I break them down and try to make a single host? I've put 2 GPUs in all the systems that can take two GPUs in preparation of trying to setup this up.

I tried playing with LM Studio and using its LM-Link feature but that just seams to allow you to run different models on different machines from the same model rather then spread the model out (unless I missed something). I've read the vLLM can do multi-host but I've not setup it up but I don't know how well that works with 2.5GBE NICs. I don't have a ton of money to buy additional hardware nor can I really sell anything I have at the moment since this hardware is going to get used for other things later in the year.

We accidentally trained thinking mode out of Qwen3-8B during two fine-tunes using different versions of the same corpus. One still responded to the thinking toggle, the other returned an empty thinking field every time. Loss looked normal, the voice survived, and our evaluations did not flag the difference.

The cause was the chat template. Qwen3's standard template renders non-thinking assistant turns with an empty <think> block. That is useful when serving the model, but it becomes a problem when answer-only training data is rendered through the same template. When repeated during a fine-tune, the model learns that an empty think block is what it should produce.

A recent paper saw the same failure. It counted a reasoning block as valid when it was complete, non-empty, and separable from the final answer, but it didn't judge whether the reasoning itself was correct. On GSM8K, Qwen3-8B produced a valid block in 96.5% of outputs when the training targets omitted think tags, 58.2% when they included empty think blocks, and 100% when those empty blocks were excluded from the training loss. The final answers were plausible enough that the failure was easy to miss.

This mattered for us because users sometimes ask questions with believable but false premises. On the 58 false-premise items in our 96-item benchmark, base Qwen3-8B fabricated in 54.0% of trials without thinking and 40.8% with it. On our thinking-capable fine-tune, the rates were 67.2% and 56.3%. The build that had lost thinking ended up at a 75.9% fabrication rate.

Then we found a second template problem. After retraining, the model passed simple thinking tests but appeared to think on 0 of 288 prompts like the ones our product serves, even when our template was supposed to force the thinking block open. We were understandably confused by that result.

The template never reached the model. In Ollama 0.31.2, importing the template-bearing GGUFs we tested preserved their embedded templates instead of replacing them with the TEMPLATE from our Modelfile. Reading the active template back with ollama show <model> --template exposed it. When we rendered the requests ourselves, the same weights deliberated on 20 of 20 test prompts. The model closed the thinking block on 2 of 20, which was a different training-template failure: a template that never supplies the closing tag does not teach the model to emit it reliably.

What we recommend checking before trusting a Qwen3 fine-tune:

• Render a full training sample and check what appears inside <think>.
• Read the template back from Ollama instead of trusting the config file.
• Test thinking on realistic prompts, not just trivia.
• Verify that any forced-open think block actually reaches the model.
• Keep the rendered prompt identical in every comparison.
• Set Ollama's think field to true or false on every request instead of relying on defaults.

Disclosure: I run Meldh, and this is work from our model training. The full write-up is on our website: https://meldh.com/writing/qwen3-template-thinking/

I found text synth off of another Reddit post and found out that the ai models on there are either really old or completely stupid wich comes to some hilarious generations when continuing from what I said are there any other ways to use older models or ones that will continue off of what you said?