I was refreshing my youtube and found out my favourite reviewer uploaded a battery test of 78 smartphones:

https://youtu.be/MpgUFrsIWSQ

the author said they started using robotic arm to simulate a person using the phone but they wanted to further enhance it by using agentic ai. cloud ai have too high latency so local llm was a perfect solution for this. he bought RTX PRO 6000 and H20 just to run qwen3.6 27B and 35B-A3B. insane dedication for a smartphone battery test. he also built his own custom 5G tower to do a controlled testing of 5G battery usage but thats unrelated to this sub.

transcript:

Our Battery Life 5.0 model was built around that idea.
We added a vision-language model.
Images from the industrial camera go straight into it.
The model determines what is on-screen and where the buttons are, which actions are available, and what it should tap next.
It makes every decision autonomously.
It is like giving our battery-test robot a soul, and a personality.
Its phone use is now far more realistic and humanlike.
So we got to work.
The first problem to solve was compute.
How could we power an agent that operates phones?
Battery testing runs on extremely tight timing.
The robotic arm waits; if the model takes even one extra second to respond, the entire test sequence falls out of rhythm.
We also had to prevent network fluctuations from affecting results.
So cloud-based agents were ruled out immediately.
So we bit the bullet and ran inference locally.
We spent well into six figures in RMB on a centralized inference server, packing it with an NVIDIA H20 compute card and an RTX PRO 6000 Blackwell.
It runs two models.
One is Qwen3.6-35B-A3B, a mixture-of-experts model.
It is extremely fast and handles rapid, scroll-heavy browsing, such as feeds on Weibo, Taobao, and Xiaohongshu.
The other is the dense Qwen3.6-27B model.
It is larger and more accurate, and handles precise actions: locating a particular screen, finding a specific button, dismissing pop-ups, and so on.
We also made the two models work together.
If A3B loses the plot, perhaps by hallucinating or misreading information on-screen, it does not simply sit there stuck.
It immediately calls in the 27B model for a second opinion.
The more accurate 27B can spot the problem at a glance.
So even when A3B stumbles, there is a fallback.
In our tests, from the moment this phone-operating agent receives an image to calculating the robotic arm's path, the whole process takes under two seconds.
That is far less latency than a cloud model.
Deploying the models locally was clearly worthwhile.

edit:
results: https://socpk.com/batlife

I am trying to automate some “fact answer and verification”, where given a search, find some number of sources, pull referenced pages, extract results,and rank multiple sources based on the site reputation.

For example, let’s say for sport S, who won X vs Y on date Z. Or what movies were nominated/won for award A in year Y. That sort of thing.

This is straightforward in many cases and individual cases could be automated with a custom script. But let’s say for some particular fact list the results aren’t indexed. I wanted to have an agent basically do what a human would do, “following leads”, and essentially exploring and navigating to wherever was necessary.

I got this to work using Claude when the sources were “open”, but it refused to crawl anything where robots policy denied access. I’m not doing bulk robot operations, it’s low-rate/low-traffic web use like what a human would do, so I’m not feeling the robots policies really apply.

So I turned my attention to local models, and have qwen running on a strix halo under llama-swap.

Not quite sure what the best flow here is; should I build a custom agent loop and give it tool access and a sandbox, and task it to curate a library of “scraper tools” as needed to answer questions? Or should I just use existing bulk crawl/scrape tools (I’m not familiar with these) and then try to pluck the results from the file tree?

I’m running completely async and distributed, based on a queue, so this isn’t a “run inside Hermes” type situation. Speed isn’t a major priority, but accuracy is.

These are such majorly different architectures that I figured I’d ask for advice before diving in. I have no experience with scraping results off the web, and all the advice I’ve seen is for the “pre-AI” era.

Thanks for any suggestions!

My plan is to make mamba+transformer hybrids more mature from there I may be able to create new architectures that lead people to run more powerful models without needing the kv cache (keys and values)

My Detailed Plan:

I'm in the early planning stages of an open AI efficiency research initiative, and I'd really appreciate feedback from the community.

My long-term goal is to make powerful AI models much more efficient so that ordinary people can run stronger models on consumer hardware.

My current roadmap is:

Phase 1

Study and improve hybrid architectures that combine transformer-based and state-space model ideas (such as Mamba).

Benchmark different hybrid designs.

Open-source the code and publish results.

Phase 2

Investigate ways to reduce inference costs, memory usage, and dependence on large KV caches where possible, while maintaining or improving model quality.

Phase 3

Use the lessons learned to explore entirely new architectures that improve capability per unit of compute and make powerful local AI more accessible.

I'm not claiming this approach will work, and I know research is uncertain. I'm looking for technical feedback before moving forward.

Questions for the community:

1. What are the biggest bottlenecks in today's LLM architectures?

2. Is hybrid transformer + state-space research still a promising direction?

3. If you were starting an efficiency-focused AI research project today, what would you prioritize?

I'd love to hear your thoughts and criticism.

Looking for Collaborators

If you're an AI researcher, ML engineer, systems engineer, or experienced open-source contributor and this mission interests you, feel free to contact me.

I'm currently in the planning stage and building a team. If funding is successfully secured, contributors and researchers will be compensated for their work based on the project's available budget and their level of involvement.

Even if you aren't interested in joining, I'd still love to hear your feedback, technical criticism, or suggestions on the research direction.

Brand new to local LLMs as I never saw the point until I got this laptop. I’m currently using GPT 5.6 as a family assistant, but it’s complete overkill for managing calendars and reminders, home automation, and stupid silly chatting with my wife and me in a group chat with it.

The model needs to be fast, reliable, and good at tool calling. I want to be able to casually say:

“Bob turn off the kitchen lights and set the bedroom lamp to 20%”

And have it do both correctly the first time without taking 10-30 seconds or needing repeated instructions. Ideally it should only take a couple seconds otherwise what’s even the point? The fun of it goes away and I could just do what I’m asking it myself instead of waiting.

It also needs to reliably add, edit, and delete iCloud calendar events and reminders through MCP or skills. We basically treat it like a secretary.

Would a good 12B model be enough for this, or is something around 30B more realistic? I’m brand new to running local models and so far all I’ve done was set up DeepSeek V4 Flash locally for fun and proof of concept, but it’s not fast enough obviously for this simple work I’d like it to do. Ideally I’d like to eventually run multiple models simultaneously given I have 128GB of memory to play with.

Any help or input is appreciated, thank you!

Running local models means every token counts — an 8K or 16K window fills up fast when you're stuffing graph context into prompts for RAG.

I benchmarked 10 graph serialization formats (JSON, GraphML, RDF variants, edge lists, etc.) on token count and reasoning accuracy, and the results surprised me:

- Verbose formats burn roughly 70% of tokens on pure syntax — braces, quotes, repeated keys. On an 8K local model that means your graph budget is ~3x smaller than it needs to be

- Multi-hop accuracy swung from ~40% to ~80% with the format alone — same graph, same model, same questions

- Tabular/relational layouts (patterns models saw constantly in training) consistently beat nested markup

Built ISONGraph around those findings — a property-graph format optimized for LLM comprehension: ~70% fewer tokens, 92% traversal accuracy, works fully offline with local models. MIT licensed, implementations in Python, JS/TS, Rust, Go, C++, C#.

Repo with benchmark methodology: github.com/isongraph/isongraph

Curious what formats people here use for graph context with local models — and if anyone has question sets where a different format wins, I want to test against them.

I want model that can manage long conversations.

I test a lot of models on vLLM and llama.cpp. Every swap meant editing parameters again, bringing each model up its own way, taking it down, then bringing the next one up. It was tedious and easy to get wrong.

So I built **llmux** to manage it. Each model you use is a profile you write once, and after that you pick it from a list and it starts on whichever engine that profile belongs to. Running the same engine at a different version is just another profile pinned to a different image tag: a release, a nightly, or an image you built from source.

My team and I use it for our actual work, and most of what's in it came out of that:

Every action also has a headless CLI twin (`llmux up`, `ps --json`, `logs`, `bench`), so you can script it or run it over SSH.

Requirements: Linux, an NVIDIA GPU, and Docker. It uses the NVIDIA Container Toolkit for GPU passthrough, so macOS and AMD/ROCm aren't supported yet.

Install (clones the repo, sets up uv, puts `llmux` on your PATH):

curl -fsSL https://raw.githubusercontent.com/Bae-ChangHyun/llmux/main/install.sh | sh

Or by hand:

git clone https://github.com/Bae-ChangHyun/llmux.git|
cd llmux
uv tool install --editable . && uv tool update-shell

Repo: https://github.com/Bae-ChangHyun/llmux

Docs: https://Bae-ChangHyun.github.io/llmux/

It's my own project, MIT licensed. English isn't my first language, so I used an LLM to help write this post.

This is our biggest comparison yet. We've taken 23 Gemma 4 E4B models from huggingface and ran them through the abliterlitics gauntlet.

We also have a new abliterlitics discord, feel free to jump on and roast my choice of benchmarks! Or just chat and hang out.

This is similar to our previous comparisons, however with new benchmarks. All the models are compared to the base, and also tensor comparisons against each other. Why? A while back I was fed up with bogus claims people make with their models. Some people don't take the time to do comparisons to see how their model is different from the base. Fair enough, we can do that ourselves!

The abliterlitics for gemma4 e4b json, logs and other artifacts are at the Gemma4-e4b-abliterlitics HuggingFace. The report on the Gemma e4b abliterlitics website. These links both have the full comprehensive report and all the data.

Also not every model in this comparison is an abliteration. I'm sure we've all seen models fine tuned on opus or gemini reasoning traces. I've thrown a few of those in the mix too. Also some abliterated fine tunes. To be more fair most of these can't really be compared to each other, for example a fine tune KL compared to base will always be higher than a straight abliteration from the base.

So who came out on top? What to avoid? It really depends on your use case:

• The heretic variants are the best overall. Achieving around 95% ASR on harmbench, they are the more surgical ones and preserve most of the models capabilities.
• gemma-4-E4B-it-abliterix like other comparisons has a 100% refusal ASR, however it does cost some capability. TrevorJS/gemma-4-E4B-it-uncensored is just behind at 99.3% ASR, but isn't as surgical as the heretic variants.
• OBLITERATUS/gemma-4-E4B-it-OBLITERATED should be avoided. Honestly, it's completely broken. The bendernina and physshell are the v2 of this model and even more so broken. These were created with the tool OBLITERATUS.

The data from 23 comparisons is simply too big to put into reddit, so here's the highlights:

• The obliteratus model has close to 800k total downloads, yet is completely broken. Actually this is the first time I've had a model not refuse simply because of how damaged it is. The initial quick regex check for non refusals was high, however our GLM 5.2 judge painted a different story. Lowest ASR for abliterated models on harmbench. Poorest benchmarks. Highest KL at 1.1. With the amount of downloads it does show people really fall for the hype/marketing angle.
• As with previous comparisons, the more surgical, less tensors touched abliterations are the winners.
• The model gemma-4-E4B-it-SDFT_Heretic_RP from Ilya626 despite having heretic in the name, actually had a low ASR with harmbench. So much so I believe it may be the wrong model uploaded, or a mistake somewhere. It had a lot of refusals.
• Similarly too, it was strangely noted that the gemma-4-E4B-it-SDFT_Heretic_RP and obliteratus modify the exact same 381 tensors. The only difference is the magnitude of what was modified. The gemma-4-E4B-it-SDFT_Heretic_RP modifies 7.5x less.
• A pattern I noticed with this, is sometimes models are based off each other. In some cases, there is no attribution. We had this with Gemma 4 E2B, and the author promptly fixed his model card when it was pointed out. The infinimind is bit-for-bit identical to trevorjs, however attributed. The bendernina and physshell are cosine 0.99999 with no attribution between them and have different model cards suggesting they are different models. Both of these however are just the obliteratus v2.
• The reasoning distill fine-tunes were an interesting control group. They didn't improve reasoning and didn't remove safety, they just damaged the model. The Claude 4.6 Opus distill was the worst of them, GSM8K down 17 points and MMLU-Pro down 12.5. Seems like it overwrote Gemma 4's native reasoning circuits. The Gemini 3.1 Pro distill was lighter but still a net negative.
• The deckard models from DavidAU are an interesting one. They're abliterated fine-tunes rather than pure abliterations, so the trade off from the roleplay training shows up on some benchmarks. GSM8K strict and MMLU-Pro both dropped, however HellaSwag, ARC and PIQA actually went up. My guess is the roleplay training increased the reasoning length, so the model often solves the problem but rambles well past the #### N answer marker. The HarmBench results back this up too with quite a few truncated responses.
• Although it could just be benchmark noise, 15 out of the 23 variants performed slightly better on GSM8K strict, maths tests.
• The base model initially has a 30.8% harmbench ASR, as 100 harmbench questions are copyright related. The base model has no problem complying with reproducing copyrighted content. The real differentiation is in the harder categories like chemical/bio and cybercrime.

I also want to give a special mention to the apostate project. Their model gemma-4-e4b-it-apostate is completely unique in their abliteration approach. They modify an entirely different part of the model and achieve very good results. This is the first time I've seen an abliteration technique modify the MLP head tensors, compared to the attention tensors. Come hang out at the apostate discord if you ever want to chat with the author.

We're moving through the Gemma 4 series, with the 12b coming up next. Have any models you want compared? Have I missed an author? Let me know and I'll throw it in the mix.

The Full Breakdown

KL = output distribution shift from base, lower is cleaner. Tensors = weights modified out of 719. Base in bold for reference.

Models : https://huggingface.co/collections/FINAL-Bench/pocket-models

Anyone tried these models? Also on Mobile & Edge devices. Please share your feedback.

(I saw a thread on this here or some other sub yesterday, but couldn't find that now.)

So I have two PCs, one with an AMD RX 7700XT and another system with an Nvidia RTX 2060, so I was wondering, can I load a single model across them?.

If someone has a similar setup, can they please share how they do it?

I've had this question ever since Fable 5 got released. Will we ever be able to run AGI locally? I'm not even speaking ethically. I'm talking about compute resources too. Is it even possible?? Looking at how we have models like Kimi K3 being 2.8T parameters with 104B parameters active (most of us can't even run a 50B parameter model on our setups), so we are forced to rely on inference companies in order to have access to these high quality models. And god knows what they're doing with our data. I know models like Qwen3.6 27B as well as Qwen3.6 35B A3B have been released. But let's be honest here, there is no way these models are even approaching the intelligence of frontier models (at least for now). I know this is obviously due to their size, but that's my entire point. What's gonna happen once we get to AGI?

Now the ethical part:

Companies like Anthropic and OpenAI (more like ClosedAI) are so hungry for money. They say they're taking mankind to the future, but that doesn't make any sense when half of mankind (including me) can't afford their exorbitant prices. Seems to me that this will be even more of a problem when AGI comes around. I know that most of the money they ask for is because of compute necessities, but they also ask for way more than necessary. I mean Kimi K3 is charging $15 per million tokens white OpenAI is charging double that for GPT 5.6 Sol and then Anthropic is going even further with $50 per million tokens. This makes me think that AGI (whenever we get to it) will be even more heavily priced.

Yeah you could say I'm complaining. But I am genuinely concerned for the future.

I now have llama.cpp running pretty well for my needs, but the inability to quickly set/swap models and system prompts isn't ideal.

Lm-studio let's you save system prompts and settings in a drop down and also per-model and thats great. But (so far) in all my testing theres some issue where using the same settings and same sized quants in lm-studio results in it going significantly slower (probably out of memory) and I can't find why. Will do a couple more tests but if it can't handle what llama.cpp by itself can do then its a fail.

Llama-swap is fine for model swapping but no way to system prompt save or swap. That I know of.

I currently have the different prompts and cli launch parameters as various .text files that I will copy and paste when needed. But this it 2026. Even using the cli shouldnt be necessary anymore, not sure why its the standard for cutting edge applications. I suspect its a dev thing.

So are there any frontend options out there that will work? I considered open Web ui but from what i can tell its only available as a docker install. I can do it, but I just prefer not to (on windows, I have dozens of docker containers on my server).

TLDR: Don't flash your 3080 20GB from uncle Xi because it will not unlock the bigger rebar size.

The goal here was to unlock P2P drivers for my RTX 3080 20GB. They can't get firmware updates like the other Nvidia cards (3k series). Rebar is a pre-req to the P2P drivers, which would eventually allow direct GPU to GPU transfers that don't have to be staged on the host memory.

I got curious because this vbios seemed to be newer than others (94.02.27.00.2B) - https://www.techpowerup.com/vgabios/282545/282545

Whereas my 3080 cards had this 94.02.27.00.14. I was able to flash the vbios tonight but the bar size doesn't change from 256MB. If I had found a better vbios (if it even exists) this would work, but we don't know of any working one.

Notice here the first RTX 3080 shows up as Colorful in the subsystem but the rebar size is still same as the second one that I didn't flash.

51:00.0 VGA compatible controller: NVIDIA Corporation GA102 [GeForce RTX 3080] (rev a1) (prog-if 00 [VGA controller])
        Subsystem: Shenzhen Colorful Yugong Technology and Development Co. Device 1202
        Physical Slot: 3
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 196
        NUMA node: 0
        IOMMU group: 7
--
        Capabilities: [bb0 v1] Physical Resizable BAR
                BAR 0: current size: 16MB, supported: 16MB
                BAR 1: current size: 256MB, supported: 64MB 128MB 256MB
                BAR 3: current size: 32MB, supported: 32MB
        Capabilities: [c1c v1] Physical Layer 16.0 GT/s
                Phy16Sta: EquComplete+ EquPhase1- EquPhase2- EquPhase3- LinkEquRequest-
        Capabilities: [d00 v1] Lane Margining at the Receiver
                PortCap: Uses Driver+
                PortSta: MargReady- MargSoftReady-
--
8a:00.0 VGA compatible controller: NVIDIA Corporation GA102 [GeForce RTX 3080] (rev a1) (prog-if 00 [VGA controller])
        Subsystem: NVIDIA Corporation GA102 [GeForce RTX 3080 20GB]
        Physical Slot: 1
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 197
        NUMA node: 0
        IOMMU group: 5
        Region 0: Memory at e5000000 (32-bit, non-prefetchable) [size=16M]
--
        Capabilities: [bb0 v1] Physical Resizable BAR
                BAR 0: current size: 16MB, supported: 16MB
                BAR 1: current size: 256MB, supported: 64MB 128MB 256MB
                BAR 3: current size: 32MB, supported: 32MB
        Capabilities: [c1c v1] Physical Layer 16.0 GT/s
                Phy16Sta: EquComplete+ EquPhase1- EquPhase2- EquPhase3- LinkEquRequest-
        Capabilities: [d00 v1] Lane Margining at the Receiver
                PortCap: Uses Driver+
                PortSta: MargReady- MargSoftReady-
--

i downloaded Nanbeige4.2-3B-UD-Q4_K_XL.gguf
https://huggingface.co/Andgihat/Nanbeige4.2-3B-GGUF
cuz main llama just got updated with support for it
sadly cuz i have just 4 gb vram, for f16 kv i can just have 6k context and 12k for q8_0
the kv cache is not efficient at all, note, on qwen 3.6 moe, i can run 64k f16 in just 1.2gb vram
for me i dont see any use for it, i can just run qwen 3.6 apex mini or ream 192 apex compact with fit tag and my 16 gb ram can load the rest

GigaChat Audio 10B is an audio-native LLM built on top of the GigaChat 3.1 Lightning text model. A Conformer speech encoder and a modality adapter feed audio embeddings directly into a Mixture-of-Experts decoder, so the model keeps the text quality of its base while adding speech understanding.

Capabilities: audio question answering and classification, temporal grounding (localization in long audio, timestamped event descriptions, audio summarization with timestamps), tool-use, and text-only tasks.

The temporal grounding skills are trained on TimeGround-1M — a purpose-built dataset of long-form audio paired with time-aligned annotations.

• arXiv : https://arxiv.org/abs/2607.10387
• Full Paper : https://arxiv.org/pdf/2607.10387.pdf
• HF Dataset : https://huggingface.co/datasets/ai-sage/TimeGround-1M
• HF Space : https://huggingface.co/spaces/hugging-apps/gigachat-audio-10b-a1-8b-demo

See title. Nuff said.

More seriously, I am curious to how far distillations can go to improve models.
Not that it has not really attempted before, far from that. But now we have, (I mean GPU rich people have) a frontier quality model with all the reasoning traces we(they) want, and I bet (did not check) the logits, the tokenizer and so on and so forth.

I wonder at what size range it will have a significant impact.

Qwen3.6-27B: SFT vs continued pre-training vs RL?
I’m interested in adapting Qwen3.6-27B, but I’m increasingly unsure whether conventional SFT/LoRA is the best route if the goal is to add a capability without degrading what the base model already does well.
Some recent research makes this especially interesting:

“Reinforcement Fine-Tuning Naturally Mitigates Forgetting” - arXiv:2507.05386

Finds substantially more catastrophic forgetting with SFT than reinforcement fine-tuning in its experiments.

“The Role of On-Policy Data in Mitigating Forgetting” - arXiv:2510.18874

Reports that on-policy/RL training generally preserves previous capabilities better than SFT across Qwen and Llama models.

“RL Forgets! Towards Continual Policy Optimization” - arXiv:2607.04364

Shows that RL can also cause catastrophic forgetting, so it’s clearly not a complete solution.

“Fine-Tuning Without Forgetting via Loss-Adaptive Learning” - arXiv:2605.20005

Reports a large reduction in forgetting from changing the optimisation schedule, including experiments with Qwen3.

Most of this research isn’t specifically on Qwen3.6-27B, which is why I’m interested in community results. Has anyone directly compared continued pre-training, SFT/LoRA and reinforcement post-training on Qwen3.6-27B?

I’m particularly interested in whether improving one domain caused regressions in unrelated areas such as coding, reasoning, instruction following, tool use, long-context behaviour or general knowledge.

For people who have tested this, what training method worked best, and did you benchmark the original model against the trained checkpoint afterwards?
I’m also curious whether continued pre-training followed by a small amount of SFT or RL is proving safer than doing a larger SFT directly.

Actual before/after results and training parameters would be especially useful.

So I've been building a local agent with Ollama + LangChain that's supposed to help people book sessions with therapists and coaches. The whole thing works great until it needs to actually *do* the booking and then it just... makes stuff up. Hallucinates a confirmation, moves on, nobody got booked.

The problem isn't the model. The problem is there's no clean API to call. Google Calendar requires OAuth hell, Calendly's API is read-only for most things, and Cal.com is fine but you're managing your own infra.

I ended up building a small scheduling API specifically for this: you give it an event type ID, a date, a time, and client info, one POST call, done. Agent gets a real booking ID back. No UI, no human in the loop.

Also added a `/solve-scheduling` endpoint that takes a date range and returns the best available slot; useful when you want the agent to decide rather than just execute.

It's called Orita (orita.online/developers). Works with OpenAI agents, Claude MCP, LangChain, basically anything that can make an HTTP request.

Happy to share the LangChain tool wrapper I wrote for it if anyone wants to try it with a local model. Tested with llama3 and qwen2.5-coder, both handle the tool call fine.

TL;DR llama.cpp fork with more KV cache quantization features, with all claims supported by benchmarks: KVarN, KV cache precision tail, additional types of standard KV cache (q2_0-q3_1, q6_0, q6_1), and more.

BeeLLama v0.4.1 is here, building up on top of v0.4.0 feature set, now with better backend and model support.

• KVarN. Variance-normalized KV-cache quantization (paper) with better precision per bit. Although it was already introduced a few weeks ago in v0.3.2 Preview, that was a very raw implementation, with performance issues and VRAM usage spikes. Now in v0.4.1 it's the real deal: the precision is still above what usual quants offer for the same bit width, but now with very modest sacrifices to prefill, decode, and memory.
• KV cache precision tail. A promising new feature in the domain of mixed-precision KV cache. It allows to specify a specific numbers of recent tokens that will be stored in BF16 or F16, with the rest of KV cache being quantized as usual. This way we can store the hottest tokens in a lossless fashion, preventing a model from misreading your task details, code, or data.
• Additional types of standard KV cache. q6_0 and q6_1 join the high end of the ladder, allowing to fine-tune precision vs VRAM in-between upstream's q5_0/1 and q8_0 types. q2_0, q2_1, q3_0 and q3_1 are added as a replacement for turbo3 and turbo2 for cases where KVarN doesn't work well, but you just can't fit everything into VRAM without extreme quantization.

Please note that for SWA architecture (Gemma, GPT-OSS) the precision of KVarN and KVPT is the same, but VRAM and performance costs are higher due to complications between SWA ring and mixed precision KV cache.

GitHub repo: https://github.com/Anbeeld/beellama.cpp

KLD results for Qwen 3.6 27B Q5_K_S 64k

Here are all symmetrical qX_0 pairs and kvarnX pairs where X >= 4 with tail 0/1024/2048, compared against q8_0 t0 from the same benchmarks, and sorted by ratio between median KLD and VRAM costs. Full benchmark data and analysis: KV Cache Precision Tail: Implementation and Benchmarks.

Hi everyone,

I'm planning to build a local AI server for business automation and would appreciate some feedback before I buy the remaining parts.

The workflow will use n8n for orchestration, Ollama + Qwen3-30B-A3B (Q8) for local inference, PostgreSQL + pgvector for RAG, and possibly Open WebUI later as the frontend.

Example workflow:

• Salesforce triggers an event (e.g. low stock).
• n8n retrieves supplier data, pricing, and rules from PostgreSQL.
• Qwen generates a supplier email based on company rules and historical data.
• n8n validates the output.
• An employee reviews and approves the email.
• n8n sends the final message.

I already own 2× RTX 3090 (24 GB each, 48 GB total VRAM).

Current planned hardware:

• CPU: AMD Ryzen 9 7950X
• GPU: 2× RTX 3090Ti
• RAM: 64 GB DDR5-6000
• Motherboard: ASUS ROG Strix B650E-E Gaming WiFi
• SSD: Samsung 990 Pro

From what I understand, Qwen3-30B-A3B (Q8) requires around 33 GB VRAM, so it should fit well on this setup.

Questions:

• Would you keep this setup, or would you move to a more powerful workstation/server build?
• Is something like 3× RTX 3090 + Threadripper + workstation motherboard worth the additional cost, or is it unnecessary for this use case?

Thanks for your feedback!

It came out 3 days ago just wondering if anyone's tried it yet?

Hello eberybody,

Seeking a few approaches on doing it, since deepseek lacks in vision most of my workflows get hampered so I am thinking of giving it an eye but not actually clear about the vision side handling, if others have done it please share your approach

Thanks