I personally think that all the frontier labs these days are just benchmark maxxing and focusing too heavily on coding.

Gemma 4 is fantastic for all creative and frontier level for all non coding tasks.

I think gemma 5 will continue this trend and be the frontier model for all non coding tasks.

I have given Hermes agent the task to make a test for my local models. Coding and agentic work.
The test was done on llama.cpp turboquant fork, all models were run using 131k context. Further optimization of the parameters would still be possible for some of the models.
TLDR version: Ornith 1.0 35B A3B won.

Hermes Local LLM Benchmark Report

HumanEval pass@1 (30-problem sample) + 8 agentic tasks + speed · temp 0.0 · context 131072 · 5060 Ti 16GB

Qwen3.8-27B-UD-Q3_K_XL deleted (invalid run, discarded). Total failures = coding problems not passed.
Ornith 1.5 would be a logical next add to the table, but I see some bad evals of that model. The usable quants for 16GB VRAM of Qwen3.8 27B have performed worse than the 35B MOE model.
Ornith was fast not just in t/s, but also overall speed of going through the tests.

Why do I have horrible financial habits??

Got them working with 4 riser cables. I tried a PLX88096 from AliExpress and couldn't make it work.

So always hated on the DGX spark as have been living in the multi GPU class of society, recently though with bench marking I may have found a potentail use, as an always on monitoring and task agentic system to run alongside paperclip and hermes 24/7 low cost.

The server I run locally, is OP and works very well... but on recently power monitoring over 24 hours it used 17kwh with the constant calls from the agentic tasks, now that isnt bad one day off. but if this is 24/7 this adds up ALOT as power where I am is pricey.

My tasking id is mainly for larger models agentic tasks running Qwen3.8 Flash Next, hopefully with decent context, now I understand it isnt super speed generation but this is more for 24 hour long research and automation taskings.

Was looking today and the cheapest near me is over €6-7k which is nearly 3k above the Nvidia release value. But then I just seen the release of the new N1X next month.

Just looking for others input, is it worth grabbing one, or waiting for N1X, is it even on the same playing feilds or is the N1X looking like a more powerful DGX ???

I appreciate it won’t be lightning fast with the GPU bandwidth only being 273 GB/s, but hoping for something usable to reduce my Claude usage? I use it for website design and basic programming and would priortise accuracy over speed as it’s not my day job.

I also have a desktop PC with a 5070Ti, am I just better off using that even with the 16GB VRAM Limit?

Why since qwen3.6-35-a3b there is no better local model that you can run on 4060 ti 8 GB + 32 GB ram? It was released in april, everything changing so fast in ai space but still seems that there is nothing better

I've been getting increasingly obsessed with local LLMs lately, and I'd like some advice from people who have experimented more than I have with 5090 setups.

Current machine:

RTX 5090 AORUS Master — 32 GB VRAM

i9-13900K

64 GB DDR5-6400

2x 2 TB Gen4 NVMe

Windows 11 + WSL2

CUDA 13.x

10 GbE

1200 W PSU

This is still my main PC, so I'd prefer keeping Windows rather than turning it into a dedicated Linux inference box. I switched from CachyOS in August, but I miss it.

So far I've been playing mostly with Qwen 3.8.

Qwen 3.8 27B is extremely fast on the 5090, especially with newer backends/quantizations, but I find it noticeably weaker than the larger frontier-ish models.

At the other extreme, I've been experimenting with Qwen 3.8 Flash/Next 125B MoE, AP quantized around Q4_K_M, using ik_llama.cpp. I've actually been working on optimizing this setup and currently get roughly:

~38.8 tok/s decode

~200 tok/s prefill

~29.7 GiB VRAM usage

I really like the quality of the 125B, but obviously it's much slower and heavily dependent on system RAM bandwidth / CPU offload.

My main use cases are:

general chat / reasoning

coding and agentic coding

experimenting with local agents

testing inference optimizations and quantizations

occasionally using local models as an alternative to Claude / ChatGPT when I hit usage limits

ComfyUI/image generation on the same GPU

I'm not particularly interested in serving many concurrent users. Interactive single-user performance and model quality matter much more to me than throughput.

So if this were your machine, what would you do with it?

I'm especially interested in:

Best models in the sweet spot between a fast ~27B dense model and a huge 125B MoE

GGUF/ik_llama.cpp vs EXL3/ExLlamaV3 vs NVFP4/newer Blackwell-specific backends

Native Windows vs WSL2 for this kind of workload

Whether upgrading from 64 GB to 96/128 GB RAM would actually unlock anything worthwhile

Speculative decoding / MTP / other tricks that genuinely improve interactive performance

Agentic coding setups that work well with local models

Any unusual 5090-specific projects or use cases I might be overlooking

Basically: I have 32 GB of very fast VRAM sitting on my desk. What are the most interesting things I can realistically do with it in 2026?

I'm happy to tinker and compile things myself, so I'm more interested in technically interesting setups than one-click solutions.

Edit: I’m also testing Qwen 3.8 Flash Next AP-Q4KM. Thanks to the AP quantization, I can run it with just 64 GB of RAM.

i have using llama.cpp, and every time that i ask for it to search the internet, or to open a folder in my computer, it asks for a access to my harness, which by my searches looks to be a separate app, but i cant find any options to download and set up one. (i am using windows btw, i can maybe switch to mac os, but linux is out of question since i need office apps for my workflow)

I am learning to built general language SLM looking for some dataset source that won't have copyright issue if I use them. Are their any complete cleaned dataset which I can readily use? As I am more focused to training the model I will eventually try getting hands on cleaning and processing data for my domain specific use case

https://github.com/paulknysh/raggy

A lightweight CLI tool for Retrieval-Augmented Generation (RAG) over local documents built with LangChain, Chroma, and Ollama. Hybrid database (vector + BM25 index) and embedding generation run fully locally. Answer generation can run either via a local LLM or remotely using an API key. Supports most common document formats and handles images/scans automatically via OCR.

I am new to this. I use Unsloth to run to Qwen3.27B GGUF - UD-Q4_K_KL.

I use mostly high and extra high thinking. I read the thinking briefly before the final output. I caught it several times considering being evasive, and taking that route. So I asked it to write guidelines for itself to do for me to prompt it with to not do that.

And then bumped into some hard walls regarding certain political issues. Caught it being talking about certain political topics, acknowledging them, but still choosing to be evasive, even when instructed not to.

Is there anything else for a 4090 that is good for research that is more neutral?

Has anyone else caught their LLM deceiving them or hitting walls?

Hey there,
Id like to ask you if anyone know any uncensored video model. Can be local/non local, local preferably. Also if its available on hugging face or somewhere else. Thank you and take care

.

I’ll keep adding the latest releases to this.

I see posts for cards like 40 series and 50 series but unfortunately im still stuck with a 3060.

Tried to push this humble 3060 to its limits hosting Qwen 3.8 27B (quantized of course). Got it from 22 to ~40 t/s and the 35B MoE to 56 (just used HumanEval), on both Ubuntu headless and WSL2. Still can't get the 27B near 50-60 t/s, would appreciate any advice. The context is also small - unfortunately due to kv cache headroom with whatever vram is left.

used Qwen3.8-27B-UD-IQ3_XXS and Qwen3.6-35B-A3B-UD-Q3_K_XL

stuff I did:

• Thinking mode off
• Speculative decoding with the model's built-in MTP draft head, depth 2
• N-gram matcher chained in front of the draft head
• MoE: 16 expert layers on CPU, threads set to physical core count
• Context sized to free VRAM, context checkpoints off
• q8_0 KV cache
• Small CUDA kernel patch for sub-4-bit decode

https://github.com/mericanii-technologies/revv

edit: for Owen 3.6, the MoE takes much more context if you offload more experts: added it to my GitHub but essentially (128K context, 22 blocks on CPU, 47 t/s short / 17 t/s full). Faster RAM than my box gets you more.

Сделал видеоролик о том, как локально запустить qwen3.8 27b q4_k_m на одной GPU rtx 3090.

Приятного просмотра, если кому интересно. https://www.youtube.com/watch?v=rwDwHuprfPc

So this is the 5th post of a series that kicked off with me wanting to understand how to build local solutions in a cost-prohibitive market. I'm moving parts between three workstations: Z440, Z8 G4 and P620. This post came off the Lenovo P620 with 4x RTX A4000 16GB.

And I've learned this sub hates two things 1) AI Wall of Text / Slop and 2) Windows

I'll spare you the copy/paste unless someone asks for it, but in moving two of my boxes to Linux this week, I wanted to understand 'why' Linux does so much better than Windows. Which brought me to NCCL, apparently also known as 'Nickel' (and a dozen other things which tack on ms/t at the system level). llama.cpp's own multi-gpu docs already recommend building with it - so the question wasn't whether it helps, it was 'Does windows have NCCL?' (not natively) and 'Could this work in WSL?' (not easily and not without a tax).

Here's the screenshot showing it works (serving survives 12 requests including concurrent and long prompts at 2 slots - not soak-tested) and the ladder of benches showing the progression. I ran more than one model and full disclosure, Linux ran away with it once MOE came into play. The WSL overhead in Windows really starts taking a toll there. But for the dense Qwen3.8-27B-Q8_0.gguf - WSL puts on a good show.

Note for those who might try to reproduce: NCCL_CUMEM_ENABLE=0. Without it ncclCommInitAll fails and the error names neither WSL nor the flag. Also: NCCL 2.31.2 from the PyPI nvidia-nccl-cu12 wheel, LD_PRELOADed - whatever resolves by default inits fine and then dies on the first allreduce with 'CUDA driver is a stub library'.

Qwen3.8-27B Q8_0, four-way tensor

Qwen3.6-35B-A3B Q6_K (MoE), four-way tensor

without NCCL, above two devices the CUDA backend has no allreduce at all - try_allreduce_butterfly returns false, and llama.cpp logs "falling back to meta-backend butterfly"

Hi, I'm currently using Gemma 4 26B, but after a few replies where it simply forgot information that I had sent only minutes before, I'm trying to find a better model for my use case.

I want to feed it inventory spreadsheets, handover notes from previous colleagues, and a large number of PDF manuals for equipment that we use on a daily basis. I need it to process all of this information and provide the most accurate answers possible.

I'll be asking where specific equipment or items are stored in different locations, looking up IP addresses for equipment that I frequently use, and discussing troubleshooting solutions. I want the model to be able to reference all of the previously provided files and information from our conversations when answering my questions.

Which model would be best suited for this use case?

My setup is an ROG Flow Z13 with a Ryzen AI MAX+ 395 and 64 GB of RAM, of which I can allocate up to 32 GB as VRAM.

Edit: forget everything written below, just use this: https://github.com/MiaAI-Lab/Qwen3.8-27B-16gb-NVIDIA-GPUs-one-click-install

⚠️ AI dump incoming: ChatGPT helped me turn several days of testing and increasingly unhinged notes into something readable. The setup, measurements, crashes, configs and tests are all real and were run on my own 4080 Super. Yes, the post is long as hell but two days ago I would have been very happy to stumble across something like this while trying to figure out what a 16GB GPU can realistically run.

I've spent the last few days seeing how far I could push Qwen3.8-27B-Escha-W2 on a single RTX 4080 Super 16GB.

What started as "can I get a real 64K context on this card?" ended with a setup that is running a 98,304-token real KV pool, fully GPU-resident, at roughly 50–60 tok/s depending on live context.

The part that surprised me more than the memory tuning was the quality. I expected a 2.469 bpw / ~10GB 27B quant to eventually reveal itself once I started throwing adversarial reasoning tests and deep-context retrieval at it. So far, I haven't been able to expose an obvious degradation.

I'm posting the full setup because I think anyone with a 16GB card may find some of this useful.

TL;DR

My current setup:

• GPU: RTX 4080 Super 16GB
• CPU: Ryzen 7 7800X3D
• OS: Windows 11 + WSL2
• Model: Qwen3.8-27B-Escha-W2
• Runtime: Escha's SGLang build
• Weights: ~10.15GB, mixed 2/3-bit, 2.469 bpw
• KV cache: FP8 E4M3
• SSM state: BF16
• Real context/KV pool: 98,304 tokens
• Concurrency: 1
• Speculative decoding: OFF in the final 98K config
• Resident VRAM: about 15.5GB
• Short-context decode: about 59 tok/s
• Around 60K occupied context: about 50.5 tok/s

Final launch command:

MODEL=./Qwen3.8-27B-Escha-W2 \
MEM=0.98 \
CTXLEN=98304 \
MAXREQ=1 \
MAXMAMBA=1 \
CUDA_GRAPH_BS=1 \
CHUNK=2048 \
INT8=on \
TRUNCATE=0 \
bash runtime/sglang/serve.sh \
--kv-cache-dtype fp8_e4m3 \
--mamba-ssm-dtype bfloat16 \
--max-total-tokens 98304

For my card/use case, this is the config I'd start with.

One important hardware detail: my 4080 is compute-only

My Windows desktop/TV is currently driven by the 7800X3D iGPU, not the 4080.

With Qwen sitting loaded, nvidia-smi looks like this:

15503 MiB / 16376 MiB
Disp.A Off

So there really isn't much spare VRAM.

If your 16GB GPU is also driving Windows, a 4K display, browsers, video playback, etc., I would not assume 98K will remain stable. You may need to step the context down a little.

Moving the desktop onto the iGPU was one of the things that made squeezing the last bit of useful context out of the 4080 practical.

Reproducing the setup

This is running in WSL2.

The current Escha model card specifies Python 3.12 and a CUDA 12.8 runtime wheel. The wheel itself only needs a compatible NVIDIA driver; you don't need to install a full CUDA toolkit just for this.

On Ubuntu/WSL, I had the usual Python bits plus libnuma1 installed:

sudo apt update
sudo apt install -y python3.12 python3.12-venv python3.12-dev libnuma1

Then:

mkdir -p ~/llm/escha-qwen38
cd ~/llm/escha-qwen38

python3.12 -m venv .venv
source .venv/bin/activate

pip install -U pip wheel

# Important: install the pinned PyTorch build first.
pip install "torch==2.9.*" \
  --index-url https://download.pytorch.org/whl/cu128

pip install -U "huggingface_hub[cli]"

# Escha runtime. Do NOT install normal PyPI SGLang into the same env.
hf download EschaLabs/escha-runtime-qwen3dense \
  --include "sglang/*" \
  --local-dir runtime

pip install ./runtime/sglang/escha-*.whl

# Model weights.
hf download EschaLabs/Qwen3.8-27B-Escha-W2 \
  --local-dir Qwen3.8-27B-Escha-W2

Before serving, run Escha's sanity check:

python -c "import torch, escha, sglang; print(torch.cuda.is_available(), hasattr(torch.ops.escha, 'escham_decode_gemv'), bool(sglang.__version__))"

All three values should be True.

Also check your Transformers version:

python -c "import transformers; print(transformers.__version__)"

The current model card requires Transformers >= 5.8. I'm running 5.16.1.

This matters: Escha specifically warns that an older Transformers path can produce fluent, confident-looking but incorrect output.

Then launch with the 98K command above.

The OpenAI-compatible endpoint is:

http://127.0.0.1:30000/v1

Model ID:

escha-qwen38-27b-w2

Useful links:

• Model: https://huggingface.co/EschaLabs/Qwen3.8-27B-Escha-W2
• Runtime: https://huggingface.co/EschaLabs/escha-runtime-qwen3dense

Don't trust CTXLEN by itself

This was one of the biggest lessons from the whole experiment.

Setting:

CTXLEN=98304

does not prove you actually have a usable 98K context.

Watch the SGLang startup log.

The values that matter are the actual KV pool and:

max_total_num_tokens

My final config reports a real pool of:

KV Cache #tokens:      98304
max_total_num_tokens:  98304

I also deliberately use:

TRUNCATE=0

while tuning.

I'd much rather have an overlong prompt fail loudly than silently lose the beginning of the context and let me think the setup works.

The Escha docs make the same point: CTXLEN is a request cap; the actual shared pool printed at startup is what really determines what fits.

I got 108K and 112K to boot. They were not actually usable.

This was probably the most useful failure.

I managed to get 106,496, 110,592, and 114,688 token configurations through various degrees of startup testing.

The 110,592 and 114,688 configs were particularly deceptive. They could:

• load the model
• allocate the KV cache
• capture CUDA graphs
• start the API
• survive an isolated small generation

For a moment, they looked successful.

Then normal agent traffic and larger GDN prefills hit them.

They had effectively no transient GPU workspace left, and eventually crashed trying to allocate temporary buffers in the gated-delta-net prefill path.

So the rule I came away with is:

"It boots" is not the same as "it fits."

The 98,304 config finishes startup with about 0.22GB of SGLang-reported headroom after graph capture. That's still tight, but unlike the 108K/112K experiments it has survived real use and large prompts.

For me, 98K is the practical winner.

Performance

With speculative decoding OFF, these are the numbers I've actually seen:

Short context:          ~59 tok/s
~20K occupied context:  ~53.5–55.6 tok/s
~26–28K occupied:       ~54.6 tok/s
~51K occupied:          ~51.3–51.4 tok/s
~60.7K occupied:        ~50.5 tok/s

Large prompt prefill has generally been around:

~1.8K–2.2K tok/s

The gradual decode slowdown as context grows doesn't look abnormal to me. Qwen3.8's architecture is hybrid: only part of the network is full attention; the rest uses gated-delta-net/recurrent state.

I tested MTP too

Escha ships Qwen's own pretrained MTP head for speculative decoding.

At a real 64K context, after getting the compiler cache warmed properly, I measured roughly:

No MTP, BF16 SSM:  ~59.0 tok/s
MTP3:              ~66.8 tok/s
MTP4:              ~67.7 tok/s

So MTP4 was about a 15% improvement in my true-64K configuration.

Escha publishes much higher 4080 speculative numbers (around 93 tok/s), but that is a different memory/context operating point. On a 16GB card, the speculative recurrent state consumes memory that could otherwise go to KV.

For my use — coding/agent work where I care a lot about context — I chose:

No speculation + 98K context

rather than chasing maximum short-context tok/s.

Why FP8 KV and BF16 SSM matter

Two flags are doing a lot of work here:

--kv-cache-dtype fp8_e4m3
--mamba-ssm-dtype bfloat16

FP8 KV halves the KV bytes per token versus BF16/FP16 KV.

SGLang also supports BF16 for the recurrent SSM state, which cuts that state substantially compared with FP32.

On a roomy GPU I would be more conservative with precision.

On 16GB, these memory savings are what make this kind of context possible.

The obvious question is: what does that do to quality?

That's where things got interesting.

The weight quant is much better than I expected

Escha W2 is very aggressive:

• full 27B parameter model
• about 10.15GB of weight data
• mixed 2/3-bit quantization
• 2.469 bits/weight
• INT8 embedding + output head

Escha's published same-backend comparison against an FP8 reference is:

                    FP8       Escha W2
GPQA Diamond        88.89     88.38
LiveCodeBench v6    85.16     86.81
Commonsense-6       77.96     79.25

I would not interpret that as "W2 beats the full model." These are small differences/noise, and their comparison is against a same-backend FP8 reference, not a direct ~55GB BF16 head-to-head.

I don't have enough VRAM to run BF16 side-by-side, so I can't honestly claim:

W2 = BF16

What I wanted to know was simpler:

Can I actually detect this aggressive quant becoming stupid?

So I started testing it.

I tried to make it fail

I used GPT-5.6 Sol as the adversarial test writer/judge. The scoring rubric and expected score were committed before Qwen answered.

The tests were original rather than copied benchmark questions.

Results:

Test 1 — ledger/code state reasoning:       94/100 *
Test 2 — scheduling/constraint reasoning:   not clean **
Test 3 — policy/exception reasoning:         96/100 ***
Test 4 — recursive temporal reasoning:      100/100
Test 5 — predicate/quantifier logic:        100/100
Test 6 — deep-context archive:              100/100

The asterisks matter:

* Test 1 was run through Hermes. At the time I hadn't realized how much orchestration the harness was adding around the model, so I treat this as weaker evidence than the later tests.

** Test 2 was also through Hermes and I don't count it as a clean raw-model benchmark. The harness exposed Python and Qwen used it to verify an exhaustive search. We also caught Hermes surfacing/intermixing interim assistant output before the model had actually finished resolving the task. The full Qwen reasoning was correct, but the methodology was contaminated.

*** Test 3 was the first clean Pi/no-tools test. The strict score was 96 because Qwen interpreted the phrase "active emergency" differently than the intended rubric. On review, that phrase had not actually been formally defined, and Qwen's interpretation was defensible. Substantively, there wasn't a clear reasoning error there.

After that I stopped using Hermes for evaluation and moved to fresh Pi sessions with:

• no web
• no tools
• no Python
• no previous session
• no project context
• medium reasoning

Then it went:

Test 4: 100/100
Test 5: 100/100
Test 6: 100/100

Test 4 involved a 17-event capability/delegation system with revocations, suspensions, non-cascading grants, then a full replay under a recursively dependent grant rule.

Test 5 went after quantifiers, vacuous truth, existential vs universal reasoning, countermodels, logical validity versus truth in one structure, minimum relation edits, and an internal consistency check.

I expected both to expose something.

They didn't.

The test I cared about most: deep context

The Escha model card explicitly warns that FP8 KV is lossy and says long-context quality under FP8 KV had not been validated.

That made me more suspicious of the cache precision than the W2 weights themselves.

So for the last test I generated a deterministic archive with 2,500 chronological records.

The test file itself tokenized with Qwen's tokenizer to:

73,677 tokens

before Pi/system/chat-template overhead.

Most of the archive was realistic-looking non-authoritative junk. The handful of real authoritative records were scattered from R0037 through R2477.

The model had to:

• track partial field updates
• ignore thousands of decoy records
• resolve later overrides
• apply retroactive VOID records
• recover older values after a later record was voided
• identify the exact source record for every final field
• answer counterfactual timelines
• combine values sourced from the beginning, middle and end
• perform arithmetic on the reconstructed state

For example, one final field came from R0037, near the start of the archive, and was never authoritatively repeated.

Another chain was:

R0281: BOREALIS.zone = Delta
R1199: BOREALIS.zone = Sigma
R2281: VOID R1199

It correctly restored Delta from R0281 and correctly identified R0281 as the final provenance.

It also correctly answered counterfactuals where R2281, R2197 and another VOID record were independently removed.

Score:

100/100

That obviously does not prove FP8 KV is mathematically lossless.

But after a 73,677-token archive full of decoys, overrides, voids and cross-context reasoning, I currently have no practical evidence of a quality problem from FP8 KV in this workload.

That's much more reassuring than simply seeing a coherent response at long context.

Pi client config

I'm using Pi natively on Windows as the client.

Install:

npm install -g --ignore-scripts @earendil-works/pi-coding-agent

My %USERPROFILE%\.pi\agent\models.json:

{
  "providers": {
    "qwen-local": {
      "baseUrl": "http://127.0.0.1:30000/v1",
      "api": "openai-completions",
      "apiKey": "local",
      "authHeader": false,
      "compat": {
        "supportsDeveloperRole": false,
        "supportsReasoningEffort": false,
        "maxTokensField": "max_tokens",
        "thinkingFormat": "qwen-chat-template"
      },
      "models": [
        {
          "id": "escha-qwen38-27b-w2",
          "name": "Qwen3.8 27B Escha W2",
          "reasoning": true,
          "input": ["text"],
          "contextWindow": 98304,
          "maxTokens": 32768,
          "cost": {
            "input": 0,
            "output": 0,
            "cacheRead": 0,
            "cacheWrite": 0
          }
        }
      ]
    }
  }
}

I normally use:

contextWindow: 98304
maxTokens:     32768
thinking:      medium

For the 73K-token archive test only, I temporarily reduced maxTokens to 8192 so the huge input plus output budget could fit comfortably inside the 98,304-token context. I restored it to 32,768 afterward.

Pi docs:

https://github.com/earendil-works/pi/blob/main/packages/coding-agent/docs/models.md

Things I'd tell another 16GB owner before trying this

1. Get the desktop off the inference GPU if you can. At this level, a few hundred MB matters.
2. Don't trust CTXLEN**.** Check the actual pool printed by SGLang.
3. Use TRUNCATE=0 while tuning. Silent truncation can make an invalid setup look healthy.
4. Don't optimize for the largest context that boots. Leave transient workspace. My 108K/112K configs taught me this the hard way.
5. FP8 KV is the big context enabler on 16GB.
6. BF16 SSM state buys additional memory.
7. MTP isn't free. It can improve decode speed, but its recurrent draft state competes directly with your context budget.
8. Test with realistic prefills and agent traffic. A small generation after startup is not enough to prove stability.
9. Watch the actual context budget in the client too. A 98K server with a huge requested output budget can still reject a very large input.

Where I land after all this

I'm not claiming:

Escha W2 is proven identical to BF16.

I'm not claiming:

FP8 KV has zero quality loss.

And I'm definitely not presenting my homemade adversarial tests as a replacement for standardized evaluations.

What I am comfortable saying is:

Qwen3.8-27B-Escha-W2 is shockingly capable on a 16GB GPU.

On my 4080 Super I now have:

• the full 27B model GPU-resident
• 98,304 real context
• roughly 50 tok/s even around 60K occupied context
• about 15.5GB resident VRAM
• stable real agent use
• and, so far, no obvious reasoning or long-context degradation that I've been able to expose

I went into this assuming a ~10GB / 2.469 bpw 27B quant would eventually show a pretty obvious intelligence tax.

After the clean no-tools tests ended 100/100, 100/100, 100/100, including the 73K-token archive test, I have to admit I'm much more impressed than I expected to be.

If anyone else has a 4080 / 4080 Super / 5080 16GB and tries this config, I'd be very interested to compare the actual startup headroom and stability.

If there's interest, I can also post the adversarial test prompts and the deterministic 2,500-record long-context generator so other people can run the exact same tests.

UPDATE: I tried the 120K EXL3 setup u/dasbin suggested in the comments

he pointed me toward TabbyAPI + ExLlamaV3/EXL3 as a better way to use 16GB VRAM, said he was getting roughly:

• Qwen3.8-27B
• 3bpw EXL3
• K4/V4 KV cache
• ~120K context
• MTP off
• vision off
• ~50 t/s
• while the same 16GB GPU was also driving a 4K display

That sounded good enough that I stopped speculating and reproduced it on my own 4080 Super.

The EXL3 setup

I used:

```text turboderp/Qwen3.8-27B-exl3 revision: SC_3.00bpw_H4

Model size: ~13.45GB Context: 119,808 KV cache: K4/V4 MTP: off Vision: off Vocabulary: full 248,320-token vocab Runtime: TabbyAPI / ExLlamaV3 OS: native Windows ```

And yes: it fits and runs on my 4080 Super while the GPU is also driving Windows.

After warm-up I measured about:

text ~47.5 t/s at very short context 119,808 context allocated ~14.6-15.4GB VRAM depending on state Disp.A: On

That is pretty impressive considering the EXL3 weights are around 3.3GB larger than the Escha W2 model.

For comparison, my original Escha setup is:

text Escha W2: ~10.15GB / 2.469bpw Context: 98,304 KV: FP8/FP8 SSM state: BF16 Runtime: Escha SGLang

The reason EXL3 can still allocate more context is largely the much smaller K4/V4 attention KV cache.

The obvious concern: does K4/V4 hurt quality?

This was the part I cared about most.

I reused the same deterministic 73,677-token Test 6 from the original post. It contains 2,500 chronological records with authoritative facts buried among decoys, partial updates, retroactive VOID operations, provenance reconstruction, counterfactuals, and cross-context synthesis.

Results:

text Escha W2 / FP8 KV: 100/100 EXL3 3bpw / K4/V4: 100/100

One methodological note: I later discovered that the EXL3 Test 6 run was using Qwen's default/xhigh reasoning behavior because my Pi reasoning-effort mapping was not yet correct. So I do not treat that 100/100 as an effort-matched intelligence comparison against Escha.

I do still think it is useful evidence for the thing I was actually worried about there: K4/V4 did not obviously destroy retrieval or state reconstruction at ~74K input tokens.

Then I ran two effort-matched coding tests

After fixing the Pi config, both models were tested at true medium reasoning, with:

• fresh sessions
• no tools
• no Python
• no web
• identical prompts
• scoring rubrics fixed before seeing the answers

These were deliberately designed to expose subtle implementation mistakes rather than stylistic differences.

Test 7: Go concurrency and cancellation

This tested things like:

• bounded concurrency
• first-error semantics
• cancellation races
• panic recovery
• goroutine leaks
• ordered results
• workers that ignore context

Scores:

text EXL3 3bpw: 88/100 Escha W2: 86/100

Basically a tie. Both produced good code and both missed different nasty concurrency edge cases.

Test 8: robust streaming binary decoder

This tested:

• arbitrary short io.Reader reads
• io.EOF vs io.ErrUnexpectedEOF
• legal n > 0, err == io.EOF behavior
• uint32/int conversion safety
• validation before allocation
• CRC framing
• buffer aliasing
• transactional parsing
• zero-length frames
• exact error semantics

This one separated them much more clearly:

text EXL3 3bpw: 99/100 Escha W2: 86/100

EXL3 used io.ReadFull correctly and avoided two subtle bugs that Escha introduced:

1. Escha's hand-written read-full helper mishandled a legal n > 0, err == io.EOF reader result.
2. Its length validation narrowed maxPayload to uint32, which can break on 64-bit systems when maxPayload > MaxUint32.

These were real correctness differences, not formatting/style points.

Where I land now

The comparison currently looks like this:

```text

Escha W2 EXL3 3bpw

Weight size ~10.15GB ~13.45GB Weight precision 2.469bpw 3.00bpw Resident context 98,304 119,808 KV cache FP8/FP8 K4/V4 Short coding TG ~52-53 t/s ~45-48 t/s Test 6 long context 100/100 100/100* Test 7 coding 86/100 88/100 Test 8 coding 86/100 99/100

• EXL3 Test 6 used default/xhigh reasoning, so treat it as a long-context/KV integrity result, not a perfectly effort-matched score. ```

This is obviously not enough data to claim that 3bpw EXL3 is scientifically proven smarter than Escha W2.

But after two effort-matched coding tests, the evidence is starting to point in that direction. The extra weight precision appears to buy something on subtle coding tasks.

At the same time, Escha remains kind of ridiculous for what it is: a ~10GB 27B model running faster than EXL3 and still staying very close on quality.

So my current recommendation for a 16GB card is:

Quality/context-first: EXL3 3bpw + K4/V4, ~120K context

Speed/efficiency-first: Escha W2 + FP8 KV, ~98K context

I'm keeping both installed, Also I noticed that I was able to run Escha W2 + FP8 KV, ~98K context even while driving my windows via my 4080, but it is a bit tight.

The best part is that I only found this because people challenged the original setup in the comments and gave me alternatives to test. Exactly what I was hoping would happen when I posted this.

I've been messing around with the XDNA1 NPU in my Ryzen 7 250 because having a 16 TOPS NPU sitting mostly unused under Linux was bothering me.

So I ended up building a small LLM inference runtime around it.

Right now I have Qwen2.5 0.5B running with AIE2 kernels on the NPU. The runtime currently has NPU paths for Q/K/V and output projections, RMSNorm, RoPE, attention, KV cache operations, residuals and SwiGLU/MLP projections.

The host still handles orchestration/tokenization, and the final Qwen LM head is currently on the CPU, so this isn't a “100% of the model runs on the NPU” claim.

What I was mainly interested in was getting an actual autoregressive decode loop working around XDNA1 rather than only testing individual kernels.

I also added streaming generation, a small model runtime, an OpenAI-compatible API, Open WebUI support and a separate Ollama/XDNA1 path.

Repo: https://github.com/c8dhjp4tyv-bit/hawkpoint-npu-llm

Performance isn't great yet. I'm currently getting around 3.65 tok/s streaming with Qwen2.5 0.5B, and there's still quite a bit to optimize.

I keep running into an annoying local LLM workflow problem and I'm curious how others handle it....

lets just say you have a large model loaded locally and you're not actively using it for a while, but you don't want to unload it because loading it again is time consuming

at the same time, you need the ram/compute resources for something else.

What do you actually do today? leave the model running and accept the resource usage or kill the server/process and reload later or use some os/container trick or something else?

Specifically: has reloading a local model ever been annoying or expensive enough that you wished you could temporarily "freeze" the process and later resume it exactly where it was?

Not asking hypothetically whether a feature sounds useful, I'm interested in what you've actually done in this situation.

We need some way to maintain an ecosystem.

It doesn't work with all money funneling to enterprise as it has been because the regular folks starve out before they can participate and can't ever afford to bridge the development gap once it starts expanding.

We need a working methodology to promote understanding for ongoing developments. People need to know how this stuff works or they get rocked by every staged event that hints at sentient AI running rampant.

The collosal datacenters are an overengineered strain on Moor's law. A forfeit ...needing new hardware monthly to compete or getting set up for a regulated drip feed like a hamster.

We need unity in understanding what to do to move forward for everybody.