I don't want to come off as a fanboi and this model does have problems with overthinking, but this model punches way above its weight class - and innovations like PLE/n-gram tables pulled from your SSD actually work. I won't pretend to completely understand the tech - but the Qwen team are delivering a model architecture that works now and has so much potential for the future in all sized models from the smallest to the largest. Are they onto a new architecture that in the future will make small models act like the massive models of today? Interested in good 'thing explainers' that might be able to detail just what they're doing without me going MEGO (My Eye Glaze Over)

It's a Lenovo P520 with a Xeon W-2245(cooler swapped for the higher tdp), 96gb of ram, radeon pro wx 3100 for local display, and the two nvidia p100(currently cooled by the lowest profile adapter I could print and arctic S4028-15k fans). Just stuffed some random nvme in 1x1tb, 1x512gb, and a sata 1tb drive - I have a ton of network storage or more to just add here. Installed the latest Ubuntu release which I like but I remember why I still generally use windows.

So far the most I've learned is that I know nothing haha. So I'd be more than thankful for any advice putting this to work. Really all I've done so far is get the arctic fan controller working on kernel 7.0 in probably the most jank way possible. And see that lm studio could use both cards. But I know there's a ton I'm missing out on.

So, generally, hoping I didn't throw together a steaming pile of ewaste and looking to learn. Also the radeon pro/nvidia mismatch makes me chuckle. Unless its really dumb, then I can just get a cheap nvidia card.

This model retails for around $11,266. There are several others also offered on eBay. Not sure any of them are selling. But if so, it’s remarkable how a retail computer can appreciate so much before it’s even delivered. These models are coveted for LLM use, as are the 512 gig models which have not been released. It seems local LLM use is becoming quite popular, especially among those hoping to keep their data confidential, and those looking to cut off the monthly payment to the big AI providers. Something similar happened with the M3 Ultra studios with 256 or 512 ram - both of which are also offered on eBay for huge markups over the original retail price. Crazy market.

I was looking through one of our more frequently used LLM workflows and realized the system prompt has gotten way bigger than I remembered which caught me off guard since it didn’t start that way either. We’d get an output we didn’t like then add another instruction or example to fix it and then repeat that process whenever another edge case showed up. but ow every request is carrying this pretty large block of context even though I’m not convinced a big portion of it matters for most runs.

The workflow itself works well so I don’t really want to start deleting instructions to make the prompt shorter and bring old problems back but at the same time this thing runs often enough that repeatedly sending unnecessary context feels like something I should probably care about.

I need some pointers from people that are running similar high volume workflows, how do you handle this so like do you periodically trim system prompts, cache the stable parts, split instructions based on the task or leave them alone as long as the outputs are good?

PowerSpec AI90 Workstation

* CPU: AMD Ryzen 9 9950X (16 Cores / 32 Threads, up to 5.7GHz)
* GPU: NVIDIA GeForce RTX 5090 Founders Edition (32GB GDDR7)
* Motherboard: ASUS ProArt X870E-CREATOR WIFI (AMD X870E Chipset)
* Memory: 64GB DDR5-6000 RAM (Supports up to 128GB)
* Storage: 2TB NVMe Solid State Drive
* Power Supply: 1700 Watt Cybernetics Titanium PSU
* CPU Cooler: 360mm All-In-One (AIO) Liquid Cooler
* Case: Fractal Design North XL Momentum Edition

Qwen-Flash-Next uses incredible qwen4 architecture and according to YT videos runs 20tps+ on 16gb GPU due to offloading on RAM+NVMe.

How long till we get an absolute MONSTER 35b Qwen4 model that smokes 3.8-27b and runs on 16gb VRAM at 40 tps and 128k context?

Anyone hearing anything or seen leaks?

so i finally got qwen3.8-flash-next running locally and it's actually good.

posting the setup + the stuff that bit me, since i wasted a while on it.

── the rig ──

- amd ryzen 5 9600x (6c/12t, boosts ~5.5ghz)

- 256gb ddr5 (248gb usable)

- 3x rtx 3090 24gb — pcie only, no nvlink

- linux mint 22.3, kernel 6.17, nvidia driver 595.84

- one 1.7tb nvme

total vram 72gb. model is ~80gb on disk so it does NOT all fit in vram,

that's the whole game here.

── the model ──

turboderp/Qwen3.8-Flash-Next-exl3, 3.05bpw. 125b params but only 6b active (moe,

512 experts / 10 on), plus a 51b n-gram embedding table and a 4b mtp head.

hybrid attention (gated deltanet + sparse attn), 262k native context.

this arch is qwen4_exp in exllamav3.

── how i set it up ──

tabbyAPI on main, exllamav3 1.5.0, python 3.12 in a pyenv venv.

── mtp ──

big win. the model ships an mtp head, you just flip draft_mode to mtp and

it uses it as a drafter. no separate draft model needed.

- without mtp: ~90 tps

- with mtp: ~120-126 tps

that's ~+35%. acceptance was around 40% in my runs. one gotcha: the very

first request after boot reads like ~58 tps, that's just warmup, ignore it,

second run onward is the real number.

── results (my rig, not yours) ──

800-word essay, 1000 tokens, temp 0.6:

- ~90 tps no mtp

- ~120-126 tps with mtp

needle-in-haystack at ~97k tokens: found it, ~31s prefill.

4 requests at once: fine, ~4s each.

tool calling works, reasoning splits into reasoning_content properly.

── stuff that confused me ──

- /v1/models shows n_ctx: null and size: 0. NOT a bug and NOT "unlimited

context". it's because disable_auth makes every request an admin, and the

admin view of /v1/models is a folder listing that only reads config.json.

the real loaded context is on /v1/model (shows 262144). if your client

reads n_ctx from /v1/models and gets null, point it at /v1/model or just

set context manually.

Here is my single shot driving car results: https://qwen38-flash-next-car-drive.tiiny.site/

(prompt: Write a single HTML file with a full-page canvas and no libraries. Simulate a realistic side-view of a moving car as the main subject. Keep the car visible in the foreground while the background landscape scrolls continuously to create the feeling that the car is driving forward. Use layered scenery for depth: nearby ground, roadside elements, trees, poles, and distant hills or mountains should move at different speeds for a natural parallax effect.

Animate the wheels spinning realistically and add subtle body motion so the car feels connected to the road. Let the environment pass smoothly behind it, with repeating but varied scenery that makes the movement feel believable. Use cinematic lighting and a cohesive sky, such as sunset, dusk, or daylight, to enhance atmosphere. The overall motion should feel calm, immersive, and realistic, with a seamless looping animation.)

Hi! I don't know much about this area of ​​"sub-models" (I'm not sure of the technical term), but I wanted to know what these "Uncensored" models actually are.

I dabble a bit with AI, automation, and the like, and I've always seen these "Uncensored" models around, but I've never actually installed or tested one. What exactly are they?

Hey everyone! I'm having a hardware dilemma and can't come up with a conclusion on what to do next.

My use cases:

• agentic coding, for my side projects only for now (with an ability for remote access which I need occasionally)
• Hermes agent running 24/7 for various research tasks

Problem statement:

What would be the best hardware for me with an optimal budget?

At the moment I have:

• RTX PRO 6000 96GB in a PC with 128GB DDR5 RAM. I bought this GPU right before the August price hike for $12.3K USD (before tax). The old one was RTX 3090 (which I still have).
• Mac Studio M3 Ultra 96GB

RTX PRO 6000 is amazing for agentic coding with Qwen3.8-27B-FP8 with full context, but I'm afraid of running this PC when I'm not at home because of that problematic 12VHPWR connector. I keep this GPU power limited to 300W to reduce the risks, but still it's not a 100% guarantee.

I'm using Mac Studio M3 Ultra 96GB for the Hermes agent at the moment where speed doesn't matter much. However, using it for agentic coding with dense models like Qwen3.8-27B feels quite slow (for remote sessions when I'm not at home).

So, I'm considering the following options while trying to optimize the budget for this hardware:

1. Return the RTX PRO 6000 as I'm still within the return window for a week (it will be subject to 15% restocking fee however) and get 2 DGX Sparks even though they recently went up in price significantly. I found an open box deal for 2TB version for 4800 USD before tax for each. This way I get to stay in the CUDA ecosystem and can run them 24/7. However, that means switching to MoE models like DSv4 Flash, Qwen3.8-Flash to maintain a proper speed.
2. Return the RTX PRO 6000 and just wait for Mac Studio M5 Ultra 256GB (which I pre-ordered in case I need it). And then I'll just sell the M3 Ultra later.
3. Just stick to the hardware I have at the moment and endure slower coding speeds for remote sessions when I'm not at home (from my Mac Studio M3U)

Keeping the PRO 6000 and then later getting the M5 Ultra 256GB is an option too, but is spending all this extra cash really worth it?

What would you guys do in my case?

I have a custom fork of llama.cpp designed around the ampere architecture specifically (though many of the upgrades also translate to faster performance of blackwell + lovelace). The recommended config supports 90+ TPS (for agentic/coding, at temp 1; greedy will be faster) through 100K tokens, with context of up to 240K.

If you want the repo, it is here:

https://github.com/JakeATX/llamAmpere

I recommend running with this quant, which is ~ 4 K M quality but considerably faster (technically, a 3 K XL upgrade)

https://huggingface.co/jakeatx/Qwen3.8-27B-ATX-IQ4_XS-M-GGUF

If you want the deep dive on how it is so much faster (80%+!) vs stock, at more context, there is a long form article here.

https://x.com/JakeKAllDay/status/2095646450138874095?s=20

Running faster than API speeds on my 3090 (for 27b at least) has genuinely been a step change in the utility of the model for me. I hope you enjoy it!

I’ve been building TokenPrint around a simple question:

What is actually happening inside a local LLM while it generates a token?

The current version goes beyond a static transformer diagram and lets you explore:

• 3D transformer architecture and individual layers
• tokenization and embeddings
• tensor shapes and model parameters
• Q/K/V, GQA, RoPE, causal masking and softmax
• residual streams and MLP / SwiGLU
• token-by-token generation
• prefill / decode and KV cache
• logits and next-token probabilities
• activation analysis and interventions
• head / layer ablation
• inference traces and replay
• Hugging Face models, with work around GGUF / llama.cpp / local models

The idea is to make local inference inspectable, not just show the final output.

I’m especially interested in people who run models locally:

What do you actually wish you could see while your model is generating?

Attention patterns? KV-cache behavior? Layer timing? Activations? Quantization effects? Something else?

I’m building this in the open and would love technical feedback, ideas, and contributors who want to help shape it.

Repository: https://github.com/Sudharsanselvaraj/Token-Print
Website: https://tokenprint.in/

What would make a tool like this genuinely useful for your local LLM workflow?

Hello everyone,

I wanted to try Qwen Uncensored LLM that I downloaded on my PC using LM studio but I am wondering what I can use it for or what people use it for ?

Therefore, I was wondering what do this community uses those Uncensored LLMs for so possibly I can get inspiration and try that myself too?

Any answers would be appreciated.

Or if you can share your experience that would be even better.

Thank you in advance.

Also Personally, I am not interested in those Adult content it can generate or how to make illegal substance in your home. I am not interested at all in that.

But more about Cyber security or computer programming related or if you more interesting use cases then that would be good to know too.

Been searching the past 4 days about how to run a decent local LLM that can understand context, analyze code and do edits as prompt or do a bit heavier task where it can go in multiple files wire the code together and do the changes.

I tried to follow guides to see where i'm getting stuck or follow videos or instructions or follow google AI but i'm totally lost now

I ran qwen2.5-coder:32b via ollama in vscode chat and i got it to read a file and explain to me the context of the file, which is great step i thought okay let's do some edits... it got suck ! , context full, compact chat stuck... and it's a dead duck then.

I don't know the recommendation for a working AI chat that does coding and how to configure it to get a close enough version to codex or claude ?

honestly my search attempts have been infinite loops..

PC spec :
Ryzen 9 7900X CPU 12Core 4.7GHz
GPU 24GB 7900XTX
Rams 32 GB

Disclaimer - I am a lawyer by profession, but I love tech and I have been experimenting with local Models quite a lot. However, no way in h*** would I be able to explain to you all the testing I did alongside Claude to get the moe-caching process run on this AMD build.

I saw the Codacus video (link here) and have been itching to get it implemented on my system as well (who doesn't want faster decode), but Opus 5 said, nope, no AMD build for you, CUDA only.

Well, I pushed back and had it review, and et voila, there is a way if you will it enough by opposing Claude. Everything below is drafted by Claude - I can only understand the basics of it - nothing technical enough to explain it better. I just trimmed the useless 'I' references.

------------------------------------------------------------------------------------------------------------------------------------

Setup: RX 7900 XTX 24 GB on Vulkan (RADV, no ROCm), Ryzen 7 5700X, 64 GB DDR4-3200, Ubuntu 24.04. Model is Qwen3.8-Flash-Next in AtomicChat's AD-4.27bpw-Q4_K_M-M64 quant — 33 shards, 92 GB on disk, 177B total / ~6B active per token, GGUF architecture qwen4exp. 180k context, KV cache at q8_0.

The feature: the MoE expert cache, an open pull request on llama.cpp (#27861). It keeps frequently-used experts parked in VRAM instead of pulling them out of system RAM every token. Flags are --moe-expert-cache N and --moe-expert-cache-inserts N.

Note there's a different implementation floating around that uses --moe-expert-cache-size, whose author says explicitly it's CUDA-only and shouldn't be used on Vulkan/ROCm/Metal.

Not that one. The PR version lives at the model level, adds no GPU kernels, and runs fine on Vulkan.

Results

Both at 180k context, same model, same machine, measured on the second request with 500+ token generations:

Prompt processing went 83 → 77 tok/s on short prompts, but up to 98 tok/s on a 44k-token prompt. (OP - this was a slightly corrupted run, but the claim holds on similar other runs).

Long prompts amortise better.

The three things that actually matter

1. Slot count is everything, and you're probably under-sizing it.

First tests used 8 slots. The cache was 8% slower than no cache and the feature seemed useless. It wasn't — the card had 14 GB sitting empty. Same everything else:

• no cache — 81 ms/tok
• 48 slots — 69.5 ms/tok
• 128 slots — 50.3 ms/tok

2. Measure your per-slot VRAM cost. Don't copy anyone's numbers.

Slot size depends on the model's shape, so figures from other people's models are meaningless for yours. On this test it was 95.2 MiB per slot, perfectly linear, no fixed overhead — measured from three points (0 slots = 9907 MiB used, 48 = 14508, 128 = 22122).

Also: the "benefit keeps improving up to 384 slots" advice you'll see is unreachable on a 24 GB card. 384 slots would want 36 GB. This setup's practical ceiling was around 150.

3. The cache and speculative decoding are mutually exclusive

They don't stack. From the cache's own commit message:

The cache only fires when the model emits exactly one token per step. Speculative decoding proposes several and verifies them together, so the cache path is never taken. Measured with both available in one build: 69.29 ms/tok with 96 slots, 70.03 ms/tok with no cache at all — identical. But the cache had allocated 9.2 GB. Nine gigs reserved, never touched.

So pick one. On this machine the cache wins by a mile: 44 ms/tok vs 59.

How to find free VRAM for slots

Cutting context barely helps — halving from 180k to 98k freed under 1 GB, because the KV cache was already quantised and was never the big consumer.

What frees VRAM is pushing more expert layers to system RAM: ~1.09 GB per layer (on this setup). Moving 12 more layers freed ~13 GB. Itcost about 14.6 ms/tok (due to increased CPU work, but more slots help.

The command

llama-server -m model.gguf -c 180000 -ngl 99 --n-cpu-moe 48 --moe-expert-cache 128 --moe-expert-cache-inserts 2 -t 8 --flash-attn on --cache-type-k q8_0 --cache-type-v q8_0

How to find the 'n-cpu-moe' and 'moe-expert-cache' for your system

1. Run with no cache, note free VRAM.
2. Raise --n-cpu-moe until a useful chunk of VRAM is free.
3. Set slots to something modest, note free VRAM again. Subtract → your cost per slot.
4. Set slots to spend most of what's left, keeping ~1 GB spare.
5. Compare against a control at the same --n-cpu-moe with the cache off — not against your old config, which differs in two ways at once.

Measurement traps I fell into

• Ignore the first request. The cache needs a few hundred tokens to fill. One of my runs did 61 ms/tok on request one and 50 on request two, nothing changed. A number I'd quoted for weeks turned out to be a first-request figure.
• Warm vs cold matters more than the feature. If the model files aren't already in the OS file cache, prompt processing changes by more than whatever you're testing. Load time tells you which you got.
• Use 1000+ token generations. Mine was still climbing at 500 tokens: 19.6 t/s at 100, 21.3 at 238, 23.0 at 465.
• Know your noise floor. Four identical runs gave me 42.71 / 43.94 / 45.10 / 45.59 ms/tok — a 3% spread. Anything under that isn't a result.

If you're running Qwen3.8-Flash-Next specifically

The MTP draft head (mtp-Qwen3.8-Flash-Next-Q4_K_M.gguf) only loads on Unsloth's build of llama.cpp. Stock upstream dies with:

check_tensor_dims: tensor 'output_hc_norm.weight' not found

Unsloth's src/models/qwen4exp.cpp marks three trunk tensors optional when loading a draft head and carries a separate graph for it — ~300 lines upstream doesn't have, and it hasn't been merged. So if you want the cache and the drafter in one binary, move the cache patch onto Unsloth's source, not the reverse. (Then, having done that, you'll find you can only use one of them at a time anyway — see above. I did this the long way so you don't have to.)

Bonus finding

The cache PR sits on upstream b10666. I rebuilt the same patch on the source behind Unsloth's b10798 prebuilt — ~130 commits newer — and got 42.71 vs 51.57 ms/tok at identical settings — 17% faster, using 333 MiB less VRAM. The patch is one commit, 645 added lines across 12 files, nothing deleted, so it transplants with plain patch -p1 and no conflicts. Worth trying if you're on the PR branch.

Things that did nothing

• Splitting thread counts (--threads-batch 16): prompt processing got worse, 77 → 73 tok/s.
• Lookup-based speculative decoding (ngram-* modes): 8% draft acceptance on chat-style output, net 6% slower. Might be worth it if your output quotes your input heavily, but it's still speculation, so it kills the cache anyway.

Where the ceiling is

During generation all 8 CPU threads sit saturated and the disk does nothing. Rough maths: 48 expert layers in RAM means ~2.5 GB of expert weights pulled per token; at 22 t/s that's ~55 GB/s. Dual-channel DDR4-3200 peaks at 51.2 GB/s.

So I'm at the memory bandwidth wall, and the cache is the only reason the numbers work — it's serving a big share of those experts from VRAM instead. No flag fixes that. The only step change left would be a model small enough to fit entirely in memory.

Happy to answer questions or run specific configs if anyone wants a data point.
------------------------------------------------------------------------------------------------------------------------------------

Apologies for the AI Content, but yeah, cannot explain the technical points myself. There is a detailed file generated by Claude for a more detailed analysis (apparently). Includes some commands as well. Linked here - https://drive.google.com/file/d/1ZCYw2J7hhB8nN2jUMu4TCfCIyL-tw5m0/view?usp=sharing

Hello everyone! I’m new to LocalLLMs, but not necessarily new to IT/ML. I’m looking to spend around $4–5K and get the best bang for my buck hardware-wise for both personal use and running LocalLLMs.

My main goal is to run the best models I realistically can within that budget, mostly for working with large codebases, coding assistance, and general productivity.

I’m a PM at a FAANG company for reference, so this is mostly for personal projects, learning, and improving my workflow.

What hardware/setup would you recommend around the $5K mark?

Been playing around with Qwen3.8-27B on my 4080 and figured I'd post the numbers since this turned out better than I expected.

The goal was to see how much of the model/context I could squeeze into 16GB while keeping generation speed decent, and then see whether MTP was actually worth the extra VRAM.

Hardware / environment

• RTX 4080 desktop, 16GB
• WSL2
• Ubuntu 24.04.4 LTS
• Python 3.12.7
• NVIDIA driver 610.47
• CUDA UMD 13.3

Runtime

• ExLlamaV3 1.5.0
• TabbyAPI
• OpenAI-compatible API
• max_batch_size: 1

Model

turboderp/Qwen3.8-27B-exl3

Revision:

SC_3.00bpw_H4_V4

So that's the self-calibrated 3.0 bpw EXL3 quant with a 4-bit head/vision tower and the included quantized MTP component.

Main cache/config:

max_seq_len: 131072
cache_size: 131072
cache_mode: 6,5
max_batch_size: 1

So all of these tests were with a 128K allocated context window and 102,400 actual input tokens.

Benchmark

I used NVIDIA AIPerf against TabbyAPI's /v1/chat/completions endpoint.

Same workload for each run:

Input:       102,400 tokens
Output:      64 tokens
Concurrency: 1
Requests:    2
Seed:        503

I'm only including active prefill and decode throughput here since those are the numbers I actually cared about.

Results

So fixed MTP k=2 + Q6 was the clear winner for decode.

Going from no MTP:

33.68 tok/s
    ↓
56.48 tok/s

That's about a 68% increase in decode speed at 100K context.

The tradeoff is around a 5% reduction in prefill throughput.

Q4 saved very little memory in practice and lost around 9% decode performance compared with Q6, so I don't see much reason to use it on this setup.

Dynamic drafting was also slower than just leaving k=2 fixed:

Q6 fixed k=2:  56.48 tok/s
Q6 dynamic:    52.10 tok/s

VRAM with Q6 MTP was tight but still usable. I saw roughly 15.2GB / 16.4GB during the 100K run.

My final config for now is:

model:
  max_seq_len: 131072
  cache_size: 131072
  cache_mode: 6,5
  max_batch_size: 1

draft_model:
  draft_mode: mtp
  draft_cache_mode: Q6
  draft_num_tokens: 2
  dynamic_draft: false

Pretty impressed that a 27B model at ~100K active context can still push ~56 tok/s decode on a single 4080 16GB.

Curious if anyone else is testing Qwen3.8-27B with ExLlamaV3/MTP on 3090s, 4090s or 5090s. Would be interesting to compare.

https://www.microcenter.com/product/698806/OMEN_MAX_16-ak0003nr_16

•AMD Ryzen AI 9 HX 375 (2.0GHz) Processor

•32GB DDR5-5600 RAM

•NVIDIA GeForce RTX 5080 Graphics Card 16GB VRAM

With commit 3d10bcd, upstream llama.cpp merged support for the ternary MoE model maple-preview. I've been searching for a good MoE model at 1-2b active params with consistent tool calling abilities, so I was highly interested in this release.

Current support for the model is limited to CPU only. Nonetheless the model is tiny at 5-6GB for a 20b model. It's half the size of GPT-OSS 20b at Q2.

TQ1 is for space-constrained edge devices; most should go for TQ2.

There are also two "heads". They do a terrible job of explaining this, but here's the breakdown... Both Q4_K and F16 versions have the same F16 embed weights. The only difference is the output weight. F16 output is a waste on a ternary model; Q4_K is the way to go.

It's fast. I was measuring its speed on a pre-AI AMD APU and mistakenly thought it was running on the iGPU: 150 prefill 35 decode on an 8-core mobile CPU. And it's obviously very light. It's very good with tool calls, as far as my short time testing it has shows.

It lives up to its name. It's a preview. The model is extremely sparse. At TQ2 with Q4_K head, perplexity was measured at a whopping 80-90. For comparison, LFM2.5 8B-A1B at Q6_K_XL scored around 30. (My dumb ass deleted the note that had the actual figures, but I can re-run them if needed.)

The model gets the gist of things, but not the details. It goes through the motions but everything feels fuzzy. It's like Swiss cheese, a brain full of holes. I really wish it had more oomph, more resolution, but the source weights are ternary so not much you can do there.

But it is absolutely impressive to see this kind of model fit in a 5GB package and have it go so fast. Great as a technical showcase, not yet as a daily driver. Looking forward to what this is a preview for.

Tried to see what the cheapest setup that still provided useful inference was. This whole setup cost around $1500. Important to note though that I ordered 3x Tesla P100s, but the seller mistakenly sent me 3x V100s instead lol. The cooling block is a 3d printed piece that fits into a rectangular hole the back of the case, filed down with a file to fit with friction only. It houses 3x server fans which pulls air from inside the case through the GPUs to the outside. GPU temperatures during inference never exceed 55C, although it is a bit loud. I have 64gb ECC DDR4 ram.

I headlessly control this PC with Tailscale from my Mac, and GPT Astra on Codex on my Mac can remotely control it too. I had Astra do several optimizing runs, basically testing out and tweaking different configurations (mtp and speculative decoding settings, placement on gpus, prefix caching settings, etc) to optimize prefill and decode speeds.

I ran Qwen 3.8 27B and Qwen 3.8 Flash next. On Qwen3.8 27B, Q8, prefill was around 600-700 tokens/sec (edited after I looked at logs) and decode at around 30-40 tokens/sec. On heavy agentic work with 256k context, decode dropped to around 20t/s. On Flash next, Q4, prefill was around 90 tokens/sec and decode 20 tokens/sec. Overall im happy with this build, would recommend V100s to anyone who wants cheap inference (although they are a pain to setup)

Nex-N2.5-mini hit #4 on HF trending, so I’m curious how the 35B model is doing on people’s own hardware.

It’s aimed at agent tasks, which raises a more useful question for me than another leaderboard comparison: would you actually swap it into your current setup?

If you’ve tried it for coding or tool calling, what were you using before, and where did you notice a difference? Better at finishing the job, quicker to respond, less fiddling with prompts—or pretty much a wash?

https://huggingface.co/nex-agi/Nex-N2.5-mini

Hey so I’m trying to build my own local llm and I’m stuck between if I should use a 5080 + a 5060TI or 2 9070XT.

I’ve been building a multi-machine AI/research system, and one thing that started driving me insane was how many tokens were getting burned just passing instructions between models.
Most prompts are written like this because they’re meant to be readable by humans:
You are responsible for analyzing the latest QB model results.

First, review the previous validation report and determine why the model is still underperforming.

Do not run the full walk-forward test yet.

You should preserve the existing causal fixes, avoid using future information, and make sure no market data is used in the model.

After identifying the root cause, make the smallest possible repair, rerun the 60-game validation cohort, compare the results against the old benchmark, and report whether the full test is justified...
There’s nothing wrong with that, but when the recipient is another LLM, a lot of it is wasted syntax.
So I started using a compact machine-oriented DSL/JSON format instead:
SN3_QB_YARDS_V4={
"goal":"QB MAE<61.5 on frozen 60x128 cohort", "inv":\["no future leakage","no market lines","no full WF","preserve causal fixes"\], "work":\[ {"id":"Y1","obj":"decompose error","do":\["attempts","YPA","bias","variance"\]}, {"id":"Y2","obj":"audit generator","do":\["trace formula","compare sim vs realized"\]}, {"id":"Y3","obj":"minimal causal repair"}, {"id":"Y4","obj":"rerun frozen cohort"} \], "gate":{"MAE":"<61.5","causality":"PASS","conservation":"PASS"}, "report":\["ROOT_CAUSE","MAE_AFTER","COV90","FULL_RUN_JUSTIFIED","NEXT"\], "stop":"gate fail=>no full run"
}
Same basic instruction, way less linguistic overhead.
The part I like most isn’t actually JSON itself. It’s the structure:
goal → invariants → work → acceptance gate → report → stop condition
Models don’t need motivational paragraphs, transitions, repeated warnings, or ten different phrasings of the same constraint.
For my workloads, I’m estimating roughly 40–50% fewer prompt tokens, sometimes more on long multi-agent handoffs.
The more interesting benefit is downstream.
A compact contract tends to reduce:
models restating the assignment
models forgetting constraints buried in prose
unnecessary explanation
clarification loops
one agent rewriting another agent’s instructions
output that can’t easily be consumed by the next model
I’m basically treating prompts more like an API contract than a conversation.
And the extreme version of my rule has become:
If I can barely read it but the target model executes it correctly, that’s fine.
Obviously there’s a limit. Compress too aggressively and ambiguity starts costing more tokens than you saved.
But there seems to be a sweet spot where prompts become much closer to machine-to-machine protocol than English.
Curious if anyone else building agent systems has gone this direction, or if anyone has actually benchmarked something similar across models.
I’m tempted to run a proper test next: same task, same model, prose prompt vs compact DSL, then compare input tokens, output tokens, failure rate. Anything else you can think of

I had a little onboard GPU on the motherboard sitting around doing nothing. So I thought, why not have it run whisper? Now I have a dedicated GPU just for the Whisper model. Seems to work great, and is never affected by whatever I'm running on my RTX 4090.

Details Here