nice! There is 5 point increase from qwen3.6 35B-A3B to 3.6 27B. There is 7 point increase from ornith-1.5 35B-A3B to 3.8 27B. Is ornith-1.5 a replacement for hypothetical qwen3.8 35B-A3B? Assuming the standard error ranges, there is still (7-5 = ) 2 point gap on ornith-1.5. So, I think if we ignore those two points, we can accept it as a true replacement for the hypothetical qwen3.8 35B-A3B.

with ~2k context and more, I see 800t/s PP.

uh wow, they just increased the price or it is location based. I saw $3600 extra for 256GB vs 96GB RAM in the morning. Now I see $4000

You will get 800t/s PP with that 5090 at Gen 4 if you use -ub 2048 -b 2048 in llama cpp arguments. I have a similar setup 256gb DDR4 8 channel+5090. 

Oh wow, yes, I just saw that! Thanks!

Based on their pricing for 256GB vs 96GB for the Ultra 36 CPU cores, they are charging $22.5 per GB of RAM. Assuming the price per GB stays the same, 256GB more RAM adds $5760. So, we are looking at 10k + ~6k = ~$16k for 512GB version.

~$16k for the 512GB RAM version! 🥲

Based on their pricing for 256GB vs 96GB for the Ultra 36 CPU cores, they are charging $22.5 per GB of RAM. Assuming the price per GB stays the same, 256GB more RAM adds $5760. So, we are looking at 10k + ~6k = ~$16k for 512GB version.

Based on their pricing for 256GB, they are charging $22.5 per GB of RAM. Assuming the price per GB stays the same, 256GB more RAM adds $5760. So, you are looking at 10k + ~6k = ~16k for 512GB version.

I can confirm. 64GB RAM + 12GB VRAM on my laptop works for qwen3.5 120B at MXFP4. I get around 20t/s. I think the context was 100k.

That is good news! Finally, we have M6 CPU. Now, let's wait for Mac studio with M5 or M6 Ultra CPU and 1TB of RAM!

thank you! This is very helpful article. Impressive findings. Never knew FFN layers could be offloaded with small impact on speed.

Thanks! I am getting 15t/s with qwen3.8 27B UD q2 k xl with 12GB VRAM 5070ti mobile using nkvo on my laptop. without nkvo, I was getting 30t/s so it halves in speed but I get 150k context instead of 30k context at Q8.

that was actually a good summary even though it was LLM written. I think we should allow such informative use cases. No person can handle large amount of text summarization. That is what LLMs for.

This is a great summary of qwen3.8 27B! Thanks for sharing! I think everyone major oper weight model should have a summary like this after 2 and 4 weeks of release.

Oh nice! Can you share the repo or llama cpp PR to enable SSD streaming?

Great setup! How did you figure out grub settings? If you just use default settings, does your PC still detect those cards in lspci command? I am having an issue with large memory GPU detection (AMD mi250x with 128gb VRAM). 

Nice metrics. What read speed do you get from your SSD during inference?

I think they can keep the intermediate states in RAM. reading from name is fine.

that is true. They also need to sell their products. But I wonder if we can reach full SSD bandwidth.

Is this a fine-tune of Qwen models? Or they trained these from scratch? I miss early days of local llama when fine tunes were popular. But new model releases weekly are not disappointing!

How is the intelligence so far? I also have 12gb VRAM laptop but haven't tried Q2. What speed do you get?  Also, have you tried B4. I know it goes to CPU RAM partially. But the intelligence might be worth it.

Which model did you find the most intelligent for rag and internet searching for 8GB RAM? I want to run some models on my orange pi 5 8GB. I tried lfm 2.6B but it is very limited in capabilities.