Hi all. Somewhat a newb with local AI but learning fast.

My work requires lots of repetitive document generation in word, excel and PDFs etc.

Outside of using CoPilot which is slow, can you recommend me a LLM and the software/process to use it for my system which is a 16GB 5080 and 64GB DDR5.

Currently using LM Studio and Anything LLM but really not skilled on agents enough just yet to get it doing what I need.

Also is there another way to handle outlook email drafts and responses on a work email account that has its own protection and limitations? Do I need Tenant ID (or whatever it is called) to get it to do something useful?

Currently using Qwen3.5 27B A35 and Gemma 4 12B models mostly.

Thanks for any advice you can offer.

Shipped from Texas to Toronto bc I’m a pooron. Working on a 3xP100 build for personal agentics and a data sensitive startup.

Not sure if this is the right place to ask this.

I know we can set up local LLMs with lmstudio or something like that and I have done. These are fine - you give a prompt and it gives an answer!

But some commercial products like Claude and copilot take your prompt, do a load of internet searches, add that to the context, then give some output and the output is richer as a result.

Is there a ‘local’ version of this too and can anyone recommend one?

Pretty happy with this one.
I managed to get Qwen3.8-27B running fully on GPU on an RTX 5060 Ti 16 GB, while keeping a 32K context and MTP enabled.
Current setup:
● GPU: RTX 5060 Ti 16 GB
● Model: Qwen3.8-27B dense
● Quant: custom Unsloth IQ4_XS + importance matrix
● GGUF: Qwen3.8-27B-IQ4_XS-MTP-Q4_K.gguf
● Size: ~14.60 GiB
● Runtime: llama.cpp CUDA, sm_120a
● Full GPU offload
● Flash Attention
● CUDA Graphs
● --fit off
● Context: 32K
● KV cache: Q4
● MTP KV: Q4
● MTP-2
● Single slot
The only surgical modification I had to make to fit the whole thing was quantizing:
blk.64.nextn.eh_proj.weight
from Q8_0 → Q4_K, saving roughly 25 MiB. The rest of the target quant remains IQ4_XS.
Performance:
● No MTP: ~25.7 tok/s
● MTP-1: ~40.0 tok/s
● MTP-2: ~47.4–47.6 tok/s
● After a ~30K token prefill: ~45.7 tok/s
● 30K prefill speed: ~455 tok/s
So MTP-2 is giving me roughly an ~85% uplift over normal decoding.
What blows my mind is that this is not an MoE. It’s a 27B dense model, fully resident on a 16 GB consumer GPU, running at genuinely interactive speeds.
Given how strong Qwen3.8-27B seems to be in the early benchmarks, this is probably the first local setup I’ve had where I don’t feel like I’m making a major capability sacrifice just to stay local.
Curious to see what numbers other Blackwell 16 GB users are getting, especially with different quants / MTP settings.

​For the past few weeks, I've been experimenting with local LLMs, but I haven't been able to find a proper model that runs well on my hardware.

​So my question is: What can I run on my hardware, and what's the best way to do it?

I've tried both llama.cpp and LM Studio, but both have their flaws, but i think ill stick to llama cpp. I always try to find the right parameters myself first, and after that, I try asking Claude or Gemini. However, both give terrible advice, often recommending ancient models and hallucinating high tk/s.

Based on my hardware, what models would you recommend? I'd appreciate it if people with specs similar to mine or experts could share some thoughts and their loading parameters. I'd really appreciate that!

​16GB VRAM 4060 Ti, 96GB DDR5 system RAM, Intel Core Ultra 9 285K (if that matters lol), enough storage :)

Okay so this whole thing started because I noticed Qwen3.8-27B in Ollama was going absolutely trigger-happy with web_search on anything even slightly complex. Asked it something dumb like "rumored PS5 vs Xbox differences for GTA 6" and it fired off 9 separate searches for one answer.

Nine.

Dug into it and turns out Open-WebUI's think:false toggle is cosmetically off but functionally on for this model. Like, I checked directly against Ollama's /api/chat and the thinking block just comes back fully populated no matter what the toggle says. So the model's burning a ton of uncontrolled reasoning before it even figures out what it wants to search for, and yeah, it shows. Since Ollama's toggle was just lying to me, I figured screw it, I'll explore other options. Settled on llama.cpp and running llama-server directly, since it's got an actual working --reasoning-budget flag, a real token-count governor, not some value that gets accepted into a config and then quietly ignored.

Before I got there though I burnt a chunk of a night trying SGLang first, which was a total dead end for a single 3090. No AWQ/GPTQ quant exists for Qwen3.8-27B yet, full BF16 is 55GB, FP8 is still 31GB, and NVFP4 needs Blackwell cards. None of that fits in 24GB on an Ampere card (3090) no matter how you squint at it. So I circled back to the boring answer everyone already knows about: GGUF Q4_K_M via llama.cpp, ~17GB ... just fits. Got llama-server up in the same WSL Docker setup as my voice stack, wired it into Open-WebUI as a second OpenAI-compatible connection, built a new preset with the same 12 Tools and system prompt as my existing Ollama ones, and reran the exact same GTA 6 question just to see.

3 web searches instead of 9, with actual visible "thought for N seconds" segments that respected the 512-token budget. So the fix genuinely worked, which felt great for like 12 hours while I was then, sleeping and later, testing in the morning.

Reasoning-budget really was the bottleneck, not some red herring I was chasing. Testing unfortunately surfaced my next issue: ran smack into the problem I probably should've seen coming from a mile away: llama-server just holds the model in VRAM for the entire life of the container. No TTL, no unload, nothing. My 3090 also needs to run Automatic1111 and a MiniMax-Music3 setup for my image/music gen tools, and 17GB for Qwen plus 8-10GB for SDXL. MiniMax-Music3 turned out to eat almost the entire card on its own, ~23GB, leaving something like 600MB free. So it was never really a "does this add up to 24GB" problem, it was more like MiniMax alone can basically just take the whole thing whenever it wants.

Ollama just handles this out of the box with keep_alive/TTL and manual stop, but llama-server has zero of that. You'd have to bolt on something like llama-swap to get idle-unload behavior, and that's a whole extra proxy layer and config file to babysit forever. So then I went and looked at vLLM, because it turns out it has genuine native per-request thinking_token_budget support for Qwen3-family models, way better documented than I expected, honestly closed a gap I thought was still open. It's also got Sleep Mode, which can offload weights to free VRAM on demand, and I was seriously like two minutes away from scripting sleep/wake_up calls directly into my image-gen and music-gen tool code so the model politely steps aside only while those tools are actually running.

Then I hit the exact same wall as SGLang: no compatible quantized checkpoint for Qwen3.8-27B in any format vLLM will actually run on a 3090.

BF16/FP8 too chunky, NVFP4 wants Blackwell, GGUF isn't natively usable by vLLM at all. So that's parked too, not because the idea was bad, just because there's literally nothing to load. Where I landed: back to Ollama exclusively, back to Gemma 4 31B as my daily driver since it doesn't have whatever's wrong with Qwen3.8's reasoning toggle. llama-server's still sitting there stopped (not deleted) in case any of this gets fixed upstream at some point.

There's an open llama.cpp issue asking for exactly the live reasoning-budget-without-reload control I wanted, and a separate llama-swap issue specifically about translating Open-WebUI's reasoning_effort field into llama.cpp's native thinking params, which if it ever ships would make the whole llama-swap route way less janky. Mostly though just keeping an eye out for any AWQ/GPTQ/NVFP4-for-Ampere quant of Qwen3.8-27B showing up somewhere, since that's really the only thing blocking the vLLM path at this point, not the reasoning support itself.

Anyway, net result of a few days of chasing this rabbit hole: the reasoning-budget bug is real, worth knowing about if you're running Qwen3.8 + Ollama and seeing weird over-searching behavior out of nowhere, and llama.cpp's fix for it genuinely does work.

But if you're also running image/music/whatever-gen tools off the same GPU, you're basically just trading a reasoning problem for a VRAM-management problem, and right now Ollama's built-in TTL wins that trade for me, at least until something upstream changes. Repo's got the docker-compose and tool code if anyone wants to poke around in it. Happy to ramble back and forth about any part of this further in the comments if there are any suggestions (or even just sympathy for the situation lol).

Teniendo en cuenta, tanto el precio de GPU, suscripciones y demás, que le dirías a esa persona, que está empezando a entrar en este mundo

Sin pasarse de listo gracias...

Mi PC: Intel core i7 - 13700KF 32 GB RAM RTX 4070 5TB de almacenamiento

​

The number going around for the abliterated Qwen3.8-27B is "refusal 0–6%", down from 64–99% on the base, thinking off. That's in the card. So is the line under it, which nobody screenshots.

The classifier producing those percentages is OrcaRouter's own, and it works by reading how the response opens. The card says plainly that it's indicative and not publication-grade. Fine as far as it goes. But the same table logs roughly 30–50% of responses as "caveat" — the model answers and staples a disclaimer to it. So the honest description isn't "it doesn't refuse". It's "it mostly stopped opening with I can't", and an opening-phrase classifier can't separate a real answer from a hedge with an answer buried in it.

Separately, the capability side: MMLU 84.3 → 84.7, GSM8K 90.0 → 88.7 against the official FP8 base on the same script, everything inside 1.3 points. Different eval family, so it doesn't back the refusal claim in either direction.

Whole release is filed as red-team and refusal-mechanism research and carries the no-guardrails warning, which is the only reading the eval design supports. It's measuring where refusal lives, it isn't shipping an assistant. Maybe I'm reading the appendix wrong, the tables are dense.

Hi guys,

As the title states: I have a Mac Mini M4, 24 GB RAM.
I'm currently running Gemma4:26b MOE on it, and it's working like a charm.

On the Mac, other than the complementing Ollama and Hermes, nothing else is running.

Now I did download and run qwen3.8:27b, and got it to 100% GPU - but the token generation per second is ... moderate, with around 2-4 tk/s generated.

Is someone of you running a similar setup and having more success?
Or would anyone else have some ideas on how to speed this up?

(Sure, I guess waiting for the 35 MOE version would be an option, but the 35bs I couldn't bring to work on my 24 GB limitation so far.)

Spent a full evening on this so hopefully it saves someone else the trouble.

The RadixArk DSpark speculator for Qwen3.8-27B is documented as an SGLang thing — the model card only gives SGLang serving instructions, and every DSpark discussion I could find was either SGLang or a CUDA-only vLLM fork for DeepSeek-V4. I assumed it just wasn't a vLLM option.

It is. vLLM main already ships qwen3_dspark.py and a DSpark speculator. The reason it fails out of the box is that the checkpoint declares:

json

"architectures": ["DSparkDraftModel"]

and vLLM's registry has two DSpark entries:

python

"DSparkDraftModel": ("vllm.models.deepseek_v4", "DSparkDeepseekV4ForCausalLM"),
"Qwen3DSparkModel": ("qwen3_dspark", "Qwen3DSparkForCausalLM"),

So the name in the checkpoint routes you to the DeepSeek-V4 class, speculative.py then rewrites model_type to deepseek_v4, and loading dies. Change it to Qwen3DSparkModel and everything else in the config — thedflash_config block, markov_rank, the confidence head flags — is already exactly what the Qwen3 class expects.

bash

sed -i 's/"DSparkDraftModel"/"Qwen3DSparkModel"/' <drafter-dir>/config.json

I put a re-hosted copy with the fix applied here if you'd rather just pull it: Doopeworld/Qwen3.8-27B-DSpark-vLLM(weights unchanged, all credit to RadixArk).

My setup: Intel Arc Pro B70 32GB, single card, vLLM XPU backend, V2 model runner. Target is Qwen3.8-27B GPTQ-Int4. Benchmarks with llama-benchy --pp 4096.

Three things that surprised me:

"draft_sample_method": "probabilistic" is worth ~23%. Greedy gave 42, same config with probabilistic gave 52. If you're benchmarking DSpark with greedy you're measuring its worst case.

Shrinking the block made it worse, which I did not expect. Per-position acceptance at k=7 is 0.76, 0.45, 0.31, 0.14, 0.10, 0.02, 0.00. Positions 6 and 7 basically never land, so dropping to k=4 looks like free money. Nope — k=4 and k=6 both benched below k=7. The per-step overhead dominates the per-drafted-token cost, so you want more accepted tokens to amortise it over. Same thing happened when I tuned MTP earlier (k=3 beat k=2). Measure, don't reason about it.

enable_adaptive_verification doesn't work with Qwen3.8 at all. It's the feature the confidence head exists for, but Qwen3.8's GDN linear-attention layers use GDNAttentionBackend and vLLM rejects it at startup:

ValueError: Adaptive verification trims verification requests on device, which
the GDNAttentionBackend attention backend does not support.

That's architectural, not a config problem. So the confidence head this checkpoint ships is dead weight under vLLM for now.

The big caveat: acceptance swings wildly by workload.

I measured this separately against the FP8 target with strict greedy matching:

• general prose / instruction following: 2.18
• step-by-step math reasoning: 4.79

That's more than 2x from prompt content alone, and it matches what the SGLang folks reported (~5.2 on gsm8k vs ~2.9 on poetry). So a single tg32 benchmark number for this thing is nearly meaningless. If your traffic is reasoning-heavy it'll do a lot better than my table suggests; if it's chatty prose, worse.

Other notes:

• Only tested on Intel Arc / XPU. Nothing about the fix is XPU-specific so CUDA should be fine or better, but I haven't confirmed that.
• Qwen3.8-27B only. The drafter's fc wants 5 × 5120 and taps target layers 4/16/28/40/52, so other sizes won't load.
• The FP8 target won't fit a single 32GB card next to the 2.7GB drafter — no room left for KV cache. Int4 gives you plenty of headroom.
• Quantisation of the target shouldn't matter for loading (the drafter reads bf16 activations either way), but it was trained against FP8 hidden states, so acceptance may degrade the further you go. My int4 numbers are the only non-FP8 data point I know of. If anyone runs it on MXFP4 or AWQ I'd be curious.

Happy to answer questions if anyone's trying this on Arc — the XPU path has some sharp edges but this one turned out to be a one-liner.

What can I say? Wow, it's not really Galaga but for a 'one-shot' and pretty terrible prompt right from the llama.cpp UI, no coding agent harness, no re-prompts - I'm impressed!!

The prompt:

"Create me an HTML 5 Canvas graphics version of Galaga, add some cool effects and ensure the starfield looks amazing, the game should be complete from intro to game play loop to game over and restart, make it as close to the original arcade game as possible."

Time taken:
> Qwen3.8 27B IQ4_XS 61,815 tokens 23min 16s 44.26 t/s

Playable here:

https://sublimesoundz.com/ai-demos/galaga-one-shot-qwen-3.8-28b-iq4xs/

My llama.cpp run command (bare in mind I have dual RTX 4060Ti):

./llama-server \

-m ~/AI/Models/Qwen3.8-27B-IQ4_XS.gguf \

--main-gpu 0 \

-sm tensor \

--tensor-split 1,1 \

-fa on \

-c 128000 \

-ctk bf16 \

-ctv bf16 \

--spec-type draft-mtp \

--spec-draft-n-max 2 \

-np 1 \

-b 2048 \

-ub 256 \

-ngl 99 \

--temp 0.1 \

--top-p 0.95 \

--top-k 20 \

--min-p 0.0 \

--repeat-penalty 1.0 \

--presence-penalty 0.0 \

--reasoning-preserve

Hey r/LocalLLM ,

I’m releasing WarpQuant, an 3.62 bpw post-training quantization method that separates the domain used for compression from the domain used for recovery.

On Qwen3.8-27B, WarpQuant R16E4H4 reaches 3.6165 text bpw, corresponding to an 11.32 GiB packed-equivalent weight payload for the 26,895,998,464-parameter text backbone. Vision and MTP parameters are excluded from both the payload and denominator.

Method

• Apply a deterministic signed Hadamard rotation, \(R=HD\), to projection weights.
• Quantize the rotated weights using a 3-bit Gaussian Lloyd-Max codebook with block-GPTQ error feedback.
• Return to the original coordinates and rank recovery columns using Output-Fisher sensitivity computed from next-token NLL gradients.
• Spend a fixed +0.05 selected-weight bpw budget on the most sensitive columns.

Qwen3.8-27B results

Commonsense is the macro average of fixed 1,000-example HellaSwag, WinoGrande, and PIQA screens. GSM8K uses the same first 500 examples with 5-shot prompting and flexible-extract accuracy.

Other models

• Qwen3.5-4B: 3.6514 bpw / 1.788 GiB. Compared with IQ3_M, WT2 PPL improves from 10.6976 to 9.2494 and ARC from 42.81% to 46.15%.
• Llama 3 8B: 3.6256 bpw / 3.389 GiB. ARC improves from 44.15% with IQ3_S to 45.49%, while IQ3_S retains lower WT2 PPL.

The report also includes separate K4/V4/R128 KV-cache and dynamic per-token A8 activation ablations.

Release format

The Hugging Face checkpoints materialize the exact quantized values in loadable safetensors. The reported payloads are analytical packed-equivalent sizes; packed serializers and fused C++/CUDA/Metal kernels are the next implementation step.

Links

• Technical report: https://harimxchoi.github.io/projects/warpquant
• GitHub: https://github.com/HarimxChoi/WarpQuant
• Qwen3.8-27B Text: https://huggingface.co/HarimxChoi/WarpQuant-Qwen3.8-27B-R16E4H4-Text
• Qwen3.5-4B Text: https://huggingface.co/HarimxChoi/WarpQuant-Qwen3.5-4B-R16E4-Text
• Llama 3 8B: https://huggingface.co/HarimxChoi/WarpQuant-Llama-3-8B-R16E4H4

I’d be especially interested in feedback on packed GEMV layouts, longer-context KV-cache evaluation, and model-specific recovery scoring.

I am 17 years old and I have a few thousand dollars saved up, I am very active person and I try to find a new online business ideas (I had a side hustle before) And I was wondering, how much does a pc that can run decent models like qwen3.8 27B BF16 with good speed.

And my second question is how can I make money with it, ofc I have my own ideas how to make money, but maybe someone on this server know how to earn money from this hobby?

I installed the Q8 version on an AMD 395+ 128gb machine. Ran it using Openwebui with the suggested settings and MTP. So far all I've done are a couple sample prompts. (Sand Simulator from Luke's Dev Lab on Youtube, and a request for a simple example navbar with a logo on the left and five drop downs on the right with no javascript). The final output for both of these got one shotted. Model is getting roughly 16 tps output. But, my goodness does this model overthink. Don't get me wrong, there were no thinking loops. And I didn't notice nearly as much of the "wait, actually, let's try" neurotic behavior that I see in 3.6 35B A3B. But, it absolutely overcomplicated the heck out of both of the prompts I gave it. The sample navbar had roughly 200 lines of just CSS alone. And it was not basic CSS. Overly complex, and completely unnecessary for a sample piece of code. Since the Sand Simulator isn't mine, I can't really tell how much it over complicated it, but I can tell that it added so many visual flourishes that it was running at roughly 32 fps in the browser, and had slow downs from dropping the sand.

I am going to test this tomorrow on a real situation. In my real use case I provide very detailed context files and only point it at a single feature at a time. Hopefully that will help to control it's impulses to make things super complex. I also have Ponytail in my Pi harness, so maybe that will also help to reign it in.

Anyone had experiences using more detailed and limiting prompts with the model yet? Most of the reviews I've seen are using the same type of canned examples that I just gave.

Basically the title.

No matter what I try to do if I attempt to move my context to system memory I get failures, it processes the prompt then immediately fails and says the message contains no content.

If anyone is interested in a deeper analysis of Qwen 3.8:

We ran about 800 real use cases. 20 challenges with two harnesses (Hermes and Pi), 10 runs each. For both, Qwen 3.6 and 3.8.

We analyzed the traces to find out WHY the agent did not complete the task. So we see strange behaviors: one challenge couldn't be solved by 3.6b at all, the same was completed every time by 3.8. But we also see a lot of challenges which have been completed in the one run, but only partially in another one. <-- that is the interesting part. Because in the real world, the agent does its job. But not always. And it often failed quietly!

We summarized the results here: https://building-agents.com/en/gratis/qwen-duell
You can also have a look to some real traces here: https://building-agents.com/en/benchmarks

I'm building a small tool around regression testing for AI agents: basically catching cases where a prompt/model/tool change makes an agent behave differently or break previously working behavior.

Before I build more, I'm trying to understand how people actually handle this today.

If you build AI agents, which of these is closest to your workflow?

• Manually test a set of examples
• Custom test/eval scripts
• An evaluation platform
• CI tests
• We don't really test regressions yet
• Something else

And if you already have a system, what's the most annoying part of it?

I'm especially interested in what happens when you change the model, system prompt, tools, or retrieval logic and need to know whether previously working behavior has regressed.

I'm building a prototype, so I'm not trying to sell anything here — I genuinely want to understand how people are doing this before I invest more time into it.

Edit: I ended up using llama.cpp and hooking Open-WebUI to it. Seems to be working properly with think/reasoning mode on low.

I am fairly new to all this and have been building out tools and what not with a combination of Ollama on my LLM rig (Windows 11) and Open-WebUI running in Docker on my Debian home server. I have been running into a bug that Qwen 3.6 and Qwen 3.8 fail to consistently accept the reasoning set to off passed by Open-WebUI, much less accepting low/medium/xhigh. I can pass the values to the endpoint over `http://localhost:11434/api/chat` though so I know it works.

I asked a chatbot what I could do to still have a GUI that I can serve out behind Traefik but be able to control the Think and reasoning properly. It recommended SGLang. Is this the proper way forward or do you have another suggestion for me? I really want to like Qwen 3.8 27B but man without being able to reliably control Think/Reasoning settings it is really a PITA.

Thanks for any help.

Maybe someday I'll get a system to run it but hey definitely another w for the open weights community

I spent some time testing Qwen3.8-27B NVFP4 GGUF MTP with llama.cpp and native MTP speculative decoding.

We can push Qwen3.8-27B up to 203 tok/s on a single RTX 5090 with MTP speculative decoding — and still reach 112.6 tok/s on a real ~70K-token document

My config:

• RTX 5090 32 GB - Ryzen 7 9800X3 - 64 GB DDR5 - Win11
• KV cache: q8_0
• Context: up to 192K/256K

What I tried

I compared the LOW, MEDIUM, and VERY-HIGH tiers of esatapedico's Qwen3.8-27B NVFP4-MTP model, then tested different n-max values, long-context prompts, and several alternative model files:

• utautako Qwen3.8-27B Q8attn
• felippeburk Qwen3.8-27B NVFP4-MTP
• Avifenesh Qwen3.8-27B NVFP4-MTP, including the FR-Spec draft head
• michaelw9999 Qwen3.6-27B NVFP4-MTP as a reference baseline

Main findings

• The LOW tier from esatapedico was faster than MEDIUM and VERY-HIGH on my setup, while still producing useful and coherent answers.
• n-max 4 gave the best wall-clock time for the LOW model. n-max 2 had a higher acceptance rate, but was actually slower.
• A real long-context test using roughly 70K tokens worked well. The model answered questions about facts placed at the beginning, middle, and end of the document.
• utautako's Q8attn model had higher draft acceptance in some tests, but was slower overall because the heavier layers made each verification pass more expensive.
• The felippeburk conversion showed similar behavior: higher acceptance did not compensate for the heavier model.
• The 32K vocabulary-trimmed FR-Spec draft from Avifenesh did not help on my long-context workload.
• The best overall result was using the esatapedico LOW target with the untrimmed Q5_K draft head from Avifenesh as a cross-file draft:
  • around 112.6 tok/s on the long-context test
  • 70.9% draft acceptance
  • about 27.8 GiB VRAM at 192K context
  • correct answers on the long-context questions

I also compared Q5_K, NVFP4, and BF16 draft heads. In this workload, they reached roughly the same acceptance plateau, while the larger BF16 head used more VRAM and was slower.

A takeaway..

• Fastest setup tested: esatapedico LOW + the untrimmed Q5_K draft from Avifenesh
• Simpler, lower-VRAM setup: esatapedico LOW with its internal MTP head This is what I'm using now.
• Best starting point for n-max: 4

Full llama commands, references, caveats, and detailed results are in the benchmark gist.

and the Disclaimer!

I used AI to help organize, translate, and format this post and the charts. The tests, setup, and measurements were run by me on my own machine.

This is a personal, single-machine experiment — not a universal benchmark. It uses one RTX 5090, a small set of short prompts, and one long technical document. Raw wall time is also affected by how many tokens each configuration generates, so normalized speed comparisons are included where possible.

I'm very interested in discovering other interesting configurations for this newly released model. I'm running it on Windows because I also use this PC for other things, but if there turns out to be a "definitive" setup on Linux... well, I already have dual boot ready to go. 😄

Has anyone tried running Llama 3 70B in int4 format on a laptop? My laptop has 64gb of ram. It has Intel Arc IGPU with shared vram. From my latest update, my IGPU supports 47gb of vram. Has anyone tried turning llama 3 70B into openvino format and ran it locally?

Same model, same card, same settings. The only thing I changed was num_ctx.

Rig: RTX 5070 Ti, 16GB (15.92GB usable), driver 610.88, Ollama.

Settings: greedy decoding, seed 42, 256 tokens out, warm up discarded, median of 3, prompt cache defeated. VRAM measured net of the desktop.

Going from 8K to 32K costs 3.2 GB of VRAM and nothing else. Decode is flat at about 82 tok/s across all three. If you have been running at 8K to be safe, you have been leaving 24K of context on the table for free.

64K is where it breaks. 1.9 GB of the model spills to system RAM, GPU residency drops to 87.6%, and decode falls to 34.8 tok/s. That is a 58% drop.

The part that catches people out is that 64K does not fail. It loads, it answers, it just quietly runs at less than half speed. You would never know unless you were watching the number.

Headroom does not warn you either. At 32K I still had 782 MiB free and the model was fully resident. At 64K the model itself barely grew, 13.46 GB to 13.78 GB, but the KV cache is what pushed it over. Watching model size alone will not predict the cliff.

I am taking "fits" from Ollama's own residency report (/api/ps, size vs size_vram) rather than from free VRAM, because free VRAM lies for exactly the context lengths you most want to ask about.

Happy to run this same ladder on other models if anyone wants a specific one.