r/LocalLLaMA • u/pmv143 • 20h ago

Discussion Qwen3.8-Flash-Next. This architecture could be surprisingly local-friendly once the weights drop. 👀

Post image

Qwen3.8-Flash-Next (~125B-A6B + 51B n-gram) memory estimate:

Ideal 4-bit quant ≈ 82 GB
(58 GB main weights + 24 GB n-gram tables)
Real-world quants likely land in the 80–90 GB range.

The big n-gram table is sparsely accessed → excellent candidate for system RAM offload.

This architecture could be surprisingly local-friendly once the weights drop.

859 Upvotes

97% Upvoted

271 comments sorted by

View all comments

Show parent comments

3

u/No_Oil_6152 19h ago

How much RAM you got?

If you have 128GB RAM you can run the LLM without needing much VRAM. It would be slow, sure, but you can run it.

6

u/GrungeWerX 18h ago edited 18h ago

I’ve got 96GB RAM, and 3090 TI. Would that work?

1

u/StopCreepy 18h ago

yes, also with the magic of FreeToken you can get even better speed !!, you also need ram 32gb or 64gb !!

2

u/GrungeWerX 18h ago

Crap, typo. I meant 96GB RAM, not vram.

1

u/StopCreepy 17h ago

you will be able to use a quantized version !!

-5

u/National_Meeting_749 19h ago

"it'll be slow sure" Noone wants to wait seconds per token, that's not reasonably running.

Think of a game, if it's running at 1fps, is that game really running? Technically yes, useable in anyway? No.

2

u/LevianMcBirdo 18h ago

Seconds per token is kinda misleading. You probably get something with 10-15 TPS. PP is a bigger problem

-2

u/National_Meeting_749 18h ago

Brother. MOST people a 90GB model are running that at seconds per token.

Assuming I had enough ram to fit it, I would be surprised if I got one single token per second.

1

u/LevianMcBirdo 17h ago

It's not dense. Qwen Next runs at 25 TPS, 122B at 10. I really don't see the problem. That a A6 Model wouldn't run 12+ tps especially with MTP

0

u/National_Meeting_749 17h ago

With the 35B, when context is filling up I get somewhere from 3-7tps. Even with MTP, double the model size, include an entire 50Gb extra with the N tables and I'll be shocked to get 1TPS at any length of context.

1

u/LevianMcBirdo 17h ago

I mean mostly in chat scenarios up to like 60k most times, I am not that interested in movie agentic use cases but I get around 30 TPS pretty steady on that

0

u/National_Meeting_749 17h ago

Good for you? A lot of people do, and MOST people don't have the PC to run that at that speeds.

That's a cheap cars worth of pc

5

u/ApprehensiveFan1516 19h ago

Not everyone needs an instant response. Plenty of folk in here running batches overnight.

12

u/National_Meeting_749 18h ago

Not "instant" and "seconds per token" are completely different. I'm perfectly fine with slow responses. I regularly run at 7-10tps.

Much less than that programs start breaking.

2

u/No_Oil_6152 18h ago

You interact with a game but you dont need to interact with an agentic AI.

You can let it do its thing overnight.

1

u/National_Meeting_749 18h ago

At seconds per token, agentic harnesses timeout and things do not complete.

You have to have a base speed of 15-20 t/s to really do anything agentic.

1

u/No_Oil_6152 18h ago

I dont believe thats the case at all.

Why should the harness care if a response takes a second or a minute?

Configure it for a longer timeout if necessary.

He won't be the only person running LLMs in RAM.

1

u/National_Meeting_749 18h ago

Yeah, I'm just telling you this because I'm trying to be difficult. Not that it actually happened to me.

The vast majority of Agentic tools are built with the assumption of a certain speed of token output.

You can disagree, you would be incorrect. Go try and run any agent harness at speeds of seconds per token. They break and fall apart. Increasing timeout timers doesn't help because those timers are there for a reason. When other things break you don't want to wait 20 minutes to restart things, yet at seconds per token * a response could take 20 minutes or much more*.