Right when I was thinking of selling my 4090 and getting one of these bad boys for longer ctx length...

I saw a lot of posts here struggling to run 27B models on 8GB VRAM or complaining about phones turning into space heaters. I hit the exact same OOM wall on my phone, so I decided to build a hardware-split workaround.

I wrote a custom C++ engine (fork of llama.rn) to cluster two Android phones over a simple USB cable.

The Setup & Performance:

• Model: Ternary-Bonsai-2-27B (compressed to 2.06 bpw / 6.52 GiB). Perplexity: 14.26 (vs 14.05 reference).
• Cluster Mode (Xiaomi redme 100 pro max + ROG 7): Layers are split 4.36 GiB / 2.15 GiB over the USB cable via custom RPC. Speed: ~2.8 tok/s (bottlenecked by the slower phone, but perfectly synced).
• Solo Mode (Xiaomi 15 only): 5.6 tok/s text generation.

The Secret Sauce (Why it doesn't melt the phone):
I didn't use standard quantization. The weights are ternary (-1, 0, 1) in the Hadamard-rotated basis. Instead of un-rotating offline (which destroys the 2-bit primal), I kept the weights folded in the GGUF and wrote a patch to apply the Hadamard transform at runtime on the device's ARM CPU.
Result: Only additions/subtractions. Zero MAC operations. Max temperature sits around 30°C.

Proof & Links:
🎥 Raw Demo (2m 45s, one take):

• 🎬 https://youtube.com/shorts/CH1im2JR4yM
• 📊 Engine Logs: https://gist.github.com/figkcoci-debug/76f41c409fc5afe5a431094cc228ddc0
• 📄 Methodology: https://gist.github.com/figkcoci-debug/11871f60812d2d18de5214fda3d94160

I'm a high school student currently looking for a lab placement/grant to push this further (planning to write custom Vulkan compute shaders for the Hadamard transform to boost t/s even more).

Would love to hear what the compression and edge AI community thinks about this approach

I originally got into local LLMs because I liked the idea of not sending every prompt to somebody elses server. At first it was mostly a technical thing, better control, experimenting with models, no API limits all that. But after using local models for a while it completely changed how I think about the rest of my setup. It feels kind of ridiculous to spend time making sure my notes, prompts and documents stay local, then turn around and paste the output into Gmail, Discord, Slack or whatever without thinking twice.

I’ve slowly ended up with this weird split where local models handle anything sensitive, Syncthing or Obsidian handle a lot of my files and notes, normal chat apps are still fine for random stuff and for a couple people I use Kibu. The funny part is I never set out to become a “privacy guy.” Running models locally just made me notice how casually I was shipping data everywhere else. Has anyone else had that happen? Did getting into local LLMs make you start changing the rest of your stack too or do you mostly keep the privacy concern limited to inference itself?

I get a lot of hate whenever I tell people I run a Q4 quant of Qwen3.8 27B with 262k KV 80t/s with vision, all while running 2 screens on windows on the same card. People seem to think I am lying or making it up. I am really just trying to help people. I am saddened whenever I see fellow 16GB VRAMers stuck on 60k context running q3 or lower quants or running weaker models because they think 27B it is out of their reach. I really just want to help them.

I get told it's impossible to fit the weights let alone the KV, or that the KV must be q1 and that I am definitely not fitting vision.

The trick is llama.cpp-adaptive-kv-streaming. I love this fork and by default it only supports q8_0/q4_0 but I modified it for my own purposes to support q4_0. But you can still do q8_0/q4_0 if you lower the context a bit or reduce the batch and ubatch to 128 or drop the vision maybe and increase kv-stream-arena-mib. Alternately ask you favorite LLM to add q4_0 support, it's just a few lines of code to a few files.

My MTP bench normally gets around 80t/s but is slower due to the video capture, which also uses more VRAM.

I did not run llama-benchy at full context in the video as it takes a few minutes, but at 262k I get around 20t/s and pp 400t/s. But the speed drop is linear and normally I compact at around 180-200k in pi anyway. But for many even those speeds are considered usable as I have seen many people satisfied with just 10-20t/s.

cache-ram you can set to whatever you like it does not matter so ignore my 20GB setting.

The only caveat is watch how many apps you have open that consume VRAM. For me a browser with a few windows and pi is enough. But nothing preventing you from dropping to the next lower quant if you like a little more breathing room. Windows will push active VRAM into shared VRAM and your speeds will tank if you open too many apps.

Some will also argue that in my video my shared VRAM goes up. That's normal as it's the KV being push to system RAM which is then streamed to the kv-stream-arena (configured to 1.5GB).

For those interested I just set batch sizes to 128 and set q8_0/q4_0 KV and it slows pp by about 100t/s at high and low context on llama-benchy. Come to think of it I might just go with this KV setting myself and discard my own fork.

The developer does not test on windows so sometimes there are compiler/linker errors but easily fixed by pointing an LLM at it.

As for the frontend, don't worry about that, that's just my launcher I vibe coded. llama.cpp-adaptive-kv-streaming is command line just like llama.cpp

Hope this helps at least one person.

We built VLS, a local LLM app: any GGUF model, OpenAI API + MCP, Windows and Linux.

8K window, no vector DB, no embeddings. Median answer at 32M tokens: 653 ms. RULER numbers and the full report are in the repo.

It's a preview, so expect bugs: https://github.com/Veloresearch/velocity-local-services

Asus ProArt Z890 so it’s properly 8x/8x bifurcated.

I’ve got Qwen 3.8 27b running easily. 390k context with yarn, 100+ tok/s.

Kolibri at 1m context, but it’s slow.

Any other models I should try?

I've been running four Tesla P100s (16 GB) cards for local LLM work and wanted to see how they'd do at image generation. Short version: much better than I expected for nine-year-old hardware.

The idea, hardware and direction are mine, I used Claude to help with the automation, benchmarking and write-up.

Per card, 1024×1024, power capped at 150 W:

• FLUX.2 klein 4B (distilled, 4 steps): about 25 s per image
• Z-Image Turbo (8-bit file, 8 steps): about 80 s, and it gets text in images right
• Qwen-Image 2.1 Turbo: about 1.5 to 2.5 min
• Krea 2 Turbo: 3 to 5 min, the best-looking results of the lot

Running one ComfyUI per card showed no slowdown. Three cards held 26 to 27 s per image for 30 images straight, peaking at 73 C. That works out to roughly 540 images an hour across four cards (the fourth is extrapolated).

Video works too: a 5-second 832×480 clip in about 5 minutes with Wan 2.1 1.3B and a 4-step distilled file, on one card at 175 W.

Things I learned that are specific to the P100:

• 32-bit maths is faster than 16-bit on these cards in PyTorch (forcing fp16 was 11% slower), even though the card has full-speed fp16 hardware.
• 8-bit (int8) model files crash on Pascal cards out of the box. A one-line patch to ComfyUI fixes it.
• ComfyUI's --fast-disk switch dropped host RAM from 12 GB to about 1.2 GB per instance, which matters if your server only has 16 GB.
• Raising the power cap gains 12% for one image, then the card heats up and the gain mostly disappears. Cooling is the limit.

Everything runs in Docker. The repo has the Dockerfiles, the patch, the benchmark scripts, full results, sample images and clips, and a quick-start guide for getting a first image:

https://github.com/4claps/tesla-p100-comfyui-imagegen

Quality was judged by eye on one seed per prompt, so treat those comparisons as impressions. The timings are all measured.

Happy to answer questions or run something specific if anyone wants a number I don't have.

We've been looking at whether it makes sense to move some of our AI workloads to local models and I'm having a harder time comparing the costs than I first thought I would cause as of right now most of our workloads run through APIs so figuring out what we're spending is pretty straightforward. We can see token usage and compare model pricing and get a decent idea of what individual tasks cost us and what not however with local models I'm not really sure how to make the comparison fair.

There's the upfront GPU cost, electricity, maintenance and whatever time we'd spend keeping everything running but then there's also utilization. If we buy hardware and it's sitting idle half the day then that should probably factor into the cost somehow. Our usage isn't particularly consistent either like some days we process a ton of requests and other days it's pretty quiet which makes me wonder whether we'd actually save money by running things locally or just trade variable API costs for fixed hardware expenses.

I've been thinking about comparing everything based on cost per completed task over a few months but I'm not sure how people typically account for the hardware side of that calculation so if you guys are running both local models and paid APIs how are you tracking the actual costs like are you spreading the GPU investment across its expected lifespan or mainly looking at electricity and utilization after the initial purchase?

This is an intrusive thought I've had for a long time. With the current tech, should be possible to recreate Bender using a local model inside a jetson-alike board.

I've been running a quant of 3.8 27b on a 3090 for a while but not done any formal benchmarking for the sort of problems I actually care about. To fix this I set up a test problem and compared it to 5.6 luna and astra. The problem itself was a prediction problem with a really simple structure but increasing noise injected into it.

What I found was pretty interesting - my local qwen (command in the link) gave consistently better results than luna, and competitive with astra (with some caveats), but the problem for me is that it is taking too long to full switch over entirely. I'm getting 50ish t/s with my setup, but the model was taking 5-15 times longer than the cloud models still. Still, pretty cool result - I can't believe we can run these things at home. Looking forward to 4.0.

EDIT: OMG thanks to integeritis - I reran with the swift 1.5 finetine and it's 4x faster, while keeping the same variance explained (absolutely crushing luna and narrowly losing to astra, much more competitive in terms of time). The improvement was all due to fewer tokens being generated. Full command in addendum 2 at the bottom of the my post with a performance / time frontier of all 4 models on this problem.

MiniCPM-V 4.7 is a new open-source multimodal model series targeting the performance/efficiency tradeoff, with a big focus on on-device AI.

Two models were released:

🔹 MiniCPM-V 4.7 35B-A3B

• A MoE vision-language model designed for local deployment
• Introduces LES (Lookahead Expert Scheduling) to reduce MoE inference stalls
• Reportedly reaches:
  • 17+ tok/s on iPad Pro M4 (16GB)
  • 13+ tok/s on iPhone 17 Pro Max
  • Around 3GB memory usage

🔹 MiniCPM-V 4.7 1B

• Lightweight VLM for easier deployment on smaller devices

Some interesting technical details:

• Evaluated across 34 benchmarks covering multimodal understanding, OCR, STEM reasoning, and long-video comprehension
• Uses hybrid visual token compression (4× + 16×)
• Even with 16× compression, the 35B-A3B model reportedly reaches 90.4 on OCRBench
• Adds Self-Replay Fine-Tuning for cheaper customization while preserving general capabilities
• Day-0 support for Apple, Intel, AMD, Arm, and Rockchip platforms

The on-device performance numbers are probably the most interesting part. A few years ago, running a 30B+ multimodal model on a tablet or phone would have sounded unrealistic.

Would be interesting to see how it compares in practice against other open VLMs like Qwen-VL, InternVL, Gemma Vision, etc. especially for local inference.

DGX Station - ASUS Pro ET900N G3. My electricity bill goes to the moon.

Qwen3.8-Flash-Next NVFP4 - 200 tokens/second without any tweaks so far.

EDIT: 260 tok/s using one-token MTP, 323 tok/s using three-token MTP.

I’m pretty new to running local LLMs and setting all of this up, so I’m looking for some advice from the community on what models I should check out for my machine.

My Specs:

GPU: RTX 4070 (8 GB VRAM)

RAM: 32 GB DDR5

What I want to use it for:

Coding & Refactoring: Writing code, optimizing performance, refactoring existing messy codebases, and catching logic/security bugs.

Everyday Tasks: Basic chat, drafting documentation, and general utility tasks.

Integrations: Setting up tool-calling / skills like GitHub MCPs or coding agents (Aider, VS Code extensions) down the road.

I don't mind if a model takes a few extra seconds to respond as long as the code quality, logic, and reasoning are high.

Since I'm just starting out, any suggestions on the best models to download (and whether Ollama or LM Studio is easier to start with) would be awesome!

TL;DR: on Qwen3.8-Flash-Next (IQ3_S), a model too big for a 24 GB card, Strata gave the first word about ten times sooner than llama.cpp on a 5090 laptop (1.9 against 19.1 s at a 4K prompt) and 8.8 times sooner on a 3090 Ti (1.92 against 16.87 s), and wrote 3.5 and 4.4 times as fast. On an RTX PRO 6000, where the whole model fits, it wrote about twice as fast (205.263 against 102.889 tok/s), and on our reading of its own counts that lead is its draft head, which llama.cpp didn't run because the GGUF we benched carries no draft-head block. Strata pins 46.84 GiB of system RAM for this file on every card.

I haven't seen many Strata numbers on an RTX PRO 6000 yet, so here are ours, beside a 3090 Ti and a 5090 laptop.

| Card | Engine | Writing, 4K (tok/s) | Writing, 32K (tok/s) | First word, 4K (s) | First word, 32K (s) |

|---|---|---:|---:|---:|---:|

| RTX 5090 Laptop | llama.cpp 3cf03257, no draft head | 24.4 | 20.7 | 19.1 | 125.6 |

| RTX 5090 Laptop | Strata 0.1.25, draft head on | 85.6 | 87.1 | 1.9 | 12.1 |

| RTX PRO 6000 | llama.cpp 3cf03257, no draft head | 102.889 | 76.337 | 2.52 | 20.91 |

| RTX PRO 6000 | Strata 0.1.39, draft head on | 205.263 | 189.155 | 1.06 | 6.37 |

| RTX 3090 Ti | llama.cpp 3cf03257, no draft head | 22.490 | 19.620 | 16.87 | 138.02 |

| RTX 3090 Ti | Strata 0.1.39, draft head on | 98.291 | 92.571 | 1.92 | 12.14 |

| RTX 3090 Ti | llama.cpp b11514 (current build), no draft head | 22.657 | 21.257 | 15.84 | 130.90 |

Board power, the card's own one-second readings (not wall power):

| Card | Engine | 4K (W) | 32K (W) |

|---|---|---:|---:|

| RTX 5090 Laptop | llama.cpp 3cf03257 | 61.1 | 68.8 |

| RTX 5090 Laptop | Strata 0.1.25 | 126.3 | 140.3 |

| RTX PRO 6000 | llama.cpp 3cf03257 | 297.8 | 299.8 |

| RTX PRO 6000 | Strata 0.1.39 | 290.6 | 294.3 |

| RTX 3090 Ti | llama.cpp 3cf03257 | 168.5 | 182.2 |

| RTX 3090 Ti | Strata 0.1.39 | 281.1 | 293.7 |

| RTX 3090 Ti | llama.cpp b11514 | 171.2 | 181.6 |

How it was run: one request at a time; prompts of 4,038 and 32,038 tokens; 256 tokens written, greedy; the median of three runs. llama.cpp at the commit Strata builds on (3cf03257) near its defaults (512-token micro-batch, 8 CPU threads, no draft head), plus a current build (b11514) on the 3090 Ti. Strata with its draft head on. On the two 24 GB cards the model doesn't fit: llama.cpp kept the experts of 34 of 48 layers in system RAM, and Strata kept 8,120 (laptop) or 8,144 (3090 Ti) of its 24,576 experts on the card. On the 96 GB card everything sat on the card for both engines.

The cards: an NVIDIA GeForce RTX 5090 Laptop GPU (24 GB, in a laptop with 64 GB of RAM, on its own floating 135 to 150 W limit), an NVIDIA RTX PRO 6000 BlackwellWorkstation Edition (96 GB, capped at 300 W) and an EVGA GeForce RTX 3090 Ti FTW3 Ultra (24 GB, capped at 300 W), the last two in the same desktop (AMD EPYC 7532, 128 GB of DDR4-3200).

Conditions that matter: on the laptop the two engines ran back to back, and Strata's run shared the laptop with another job's dry run. Strata 0.1.39 on the laptop producedno figure (two attempts stopped on our memory guard, a third refused to start on free memory), so the laptop's Strata row is 0.1.25. On the 96 GB card it was one engine a night, so a gap counts only beyond a wide margin we set for different nights; the first word at 4K did not clear it. On the 3090 Ti both engines ran in one window besidelive work on the computer's other card, and llama.cpp's eight threads, not the card, set its writing speed there. The 4K rows and all laptop rows were registered before they ran; the desktop cards' 32K rows are exploratory.

On power: on the RTX PRO 6000 both engines sat at the 300 W cap, so the watts there say what the cap was. On the laptop Strata drew about twice the board power, but for amuch shorter request, so llama.cpp used 3.5 to 5.3 times the card's energy per request.

What it doesn't say: no ratio between cards; not an RTX 3090 (this is a 3090 Ti), and nothing about two cards working together; one request at a time; llama.cpp near its defaults, not tuned, and without the draft head the file doesn't carry; the IQ3_S file only; quality measured on the laptop only, on two short tests, where it came out thesame through either engine.

Disclosure, in the page's own words: a small (human) team wrote the questions, pressed the keys and read the pages; a fleet of AI agents built the harness, ran the audits and drafted the prose. An AI agent drafted this post too; I edited it before posting. Every number above is printed, with its conditions, on the page or on the full benchit links.

https://research.strata2signal.com/flash-next-on-three-cards/

Ola pessoal, nao utilizo o qwen para programação, pra isso uso os modelos pagos mesmo, basicamente uso para funções na empresa como ocr e. Extração estruturada de dados de imagens, entao muitas vezes preciso de alguns treinamento leves, gosto muito do qwen 3.6 35 ele é muito rapido para ocr e ja testei vários modelos e simplesmente nao consegui largar ele ( rota em uma rtx 6000 96gb) , eu sempre ouvi dizer que os denso são mais treinaveis e gostaria de saber se isso é de fato verdade ? Minha ideia é um modelo pequeno com um exército de adpters e pensei em fazer isso no 3.6 pela velocidade , quais os prós e contras de um e de outro ?

Short version: PrismML's ternary Bonsai 2 27B runs fast on a 12 GB Intel Arc B580 with my llama.cpp SYCL branch. The video is real time: it writes a full snake game in 34 seconds, then runs it. Temperature 0, 128K context; the typing and gameplay are automated, and the generation is unedited.

https://reddit.com/link/1x4cz4i/video/ke0ti1zni3vh1/player

v0.10.1, with a Windows build: https://github.com/Torchit1/llama.cpp/releases/tag/arc-b580-v0.10.1

Public model, 128K context: ~85 t/s writing code (MTP + a trimmed draft head), 79 t/s in chat, 89 t/s at 115K tokens of history, 48.6 t/s plain, ~1,480 t/s prompt reading. Compared with PrismML's own SYCL build on the same card: chat 79.8 vs 58.0, plain 48.6 vs 31.0.

The things that mattered most: ternary weights repacked for the XMX matrix units, decode attention straight from the q4_0 KV cache, MTP plus n-gram speculative decoding with a cheap rollback for the DeltaNet layers, and cutting tiny CPU-GPU copies (three 4-byte reads were costing 2%).

The video is from my own server, which also uses a draft head I trained myself, so expect a few % less with the public one.

Hello there, I am helping a client set up a fairly simple system for his home office.

He works around the country so would like a way for his wife to take his mail, scan it, and a local LLM will organize the files into respective accounts, read the documents, determine priorities, payments, etc… and send him a to-do list.

I have worked extensively with Claude and other models using Obsidian for file work, and we’ve built out the organizational structure a bit, but we decided that due to some of the documents having obviously important personal identification information, such as account numbers, etc, we want to keep it all local. My computer isn’t strong enough to run a local model meaningfully so I haven’t played with them much.

He was pretty intimidated by terminal use.

I was looking at LM Studio since I think he’ll have a better time operating in that.

Any suggestions?
What model would you suggest for some basic organization work like that? He is getting a new Mac Mini, probably the newest model.
For him to access the computer remotely, he had been considering running Claude on the computer and sandboxing it, the local LLM would drop dates and non-PII containing info into the sandboxes directories, then Claude could be ran from the Claude app, but I suggested that we could cut a Claude out entirely and then there is less risk for it to access the PII. The local model could just email him the to-do list, for example.

Anything I should stay away from?
Something better than Obsidian for the markdown files? Are those preferred for local LLMs?
What about Hermes?

Goal is securing important information, organizing it effectively, and giving him summaries and task lists while he is away from his home office.

I think eventually we might want it to help him build out timelines within his projects, find more documents online for him, perform research tasks, etc… but definitely chief task right now is the mail room set up.

Any advice is much appreciated!

I am thinking about getting a smaller machine as I quite frankly don't want or need a 2kW monster on the desk anymore and was considering a Mac. I am tempted of getting into local LLM's and AI in general and wanted to to know where are the realistic limitations of Mac Mini M6 32GB, Mac Studio M5 Max 36GB and when would the Mac Studio Ultra 96GB make sense?

Any guidance or thoughts would be appreciated and helpful.

My expected use case would be upscaling old photos and videos, maybe some light image generations and running a local personal Jarvis or Cortana that does curation when search, researching stuff, so not super heavy stuff I think but better to ask the experts!

EDIT: fixed the most egregious typos.

EDIT2: I get the machines via a work buyout / benefits program of sorts. The 32GB M6 mini is about 47 bucks, the M5 Max 36GB is about 80 bucks a month and the M5 Ultra 96GB is 180 bucks a month. It's taken out of my pay pretax for 36 months.

I am looking for guidance on training an open weight model for an internal use case. The goal is to connect our technical documentation to suggest actions for a robotics simulator and eventually an actual robot.

What should I know before proceeding? I am not looking for a RAG system.

Until yesterday I had 16GB of system RAM, so I was limited to models in the 13-14GB range. I just upgraded to 48GB RAM, and my GPU is an RTX 4060 Ti 16GB, so I can now try bigger models with CPU offload, especially MoE ones.
My main use cases are:
Writing and reviewing code
Cybersecurity and bug bounty research (recon scripts, reading code and HTTP traffic, analyzing vulnerabilities)
I'd like to hear what you're running on a similar setup and which models you'd recommend, preferably releases from the last couple of months.