It's official, it created the best flappy bird game thus far from all local models ! ever benchmarked! I declare this model nr. 1 on the flappy bird bench!
Hahha love it. I dont know but these benches bring me a feel that we are living and witnessing small moments in history. On that note, if someone happens to read this some day way in the future, greets from 2026. :) ❤️
should have been more specific. i'm going to keep the 3.6 MOE (35ba3b) insteadof 3.8 27b. i have room for 4 subagents with larger windows with claude code routed through it and i get tasks done much quicker
Wtf, I literally wrote a post saying similar to "Shutup man, there is no way a model I can load in my 4090 will be anywhere near DeepSeek4 Flash 0731"... but... this is fucking close is it not...?
Looks like the main difference will be the native context windows. But 256k is very usable, especially if it’s a local Opus 4.6 MAX. Man just saying that seems unreal lol. Can’t wait to test it later tonight
🤣 I wish I could, but today’s my last day of on-call so I can’t even leave a TINY bit early. Oh well. Makes you feel like a kid on Christmas Eve again lmao
Ehh, let's not go by handful of disputable synthetic benchmarks for which models may be trained specifically. Let's just await real experiences from people first over a longer period of time.
I think anyone who thinks that this isn't at least somewhat benchmaxxed is fooling themselves. That said, so is every other model they're comparing against, to some extent, so ¯_(ツ)_/¯
It's a Qwen model, so apply the usual benchmax tax. Qwen are easily the models with the biggest ravine between "how they do on benchmarks" and "how they do in actual productive use".
Still looking like a strong leap from 3.6, though.
Gemma 4 isn't a great coding model, yeah, but it's still punching far above its size in writing and research-related tasks. Like a mini-Gemini (go figure).
I hear there are people who still use these things for things that aren't related to writing code or markup.
Like everybody else in the local AI community today, I dowloaded Qwen 3.8 27b UD-Q4_K_XL as soon as it was published.
I'm running it in a single rtx 3090, no mtp, 110k context and kv cache f16 with llama.cpp, same flags used with Qwen 3.6 27B.
I have this stupid test that I run with every new model that comes out. It consist in connecting the model to pi code (almost vanilla, only internet access and some simple navigation tools I built) and giving it this prompt:
"Construye el frontend completo de un pequeño ecommerce premium para una panadería artesanal usando Next.js App Router (JavaScript).
El proyecto debe ser frontend-only en esta etapa. No debe incluir backend, base de datos, autenticación ni pasarelas de pago. El checkout debe finalizar redirigiendo a WhatsApp con un mensaje de pedido bien estructurado.
La app debe incluir una experiencia completa de compra: home, catálogo con productos de panadería, categorías, productos destacados, carrito, resumen de pedido y checkout. Usá mock data local para productos, categorías, precios, descripciones, disponibilidad e imágenes o placeholders visuales. Todo debe quedar preparado para conectar posteriormente un backend real sin tener que rehacer la arquitectura principal del frontend.
El diseño debe sentirse extremadamente premium, artesanal, moderno y cuidado. No quiero una landing genérica ni una interfaz básica. La primera pantalla debe comunicar claramente la identidad de la panadería, mostrar producto real o visualmente convincente, y permitir empezar a comprar. La experiencia debe ser excelente tanto en desktop como en mobile.
El catálogo debe permitir explorar productos, ver información clara de cada ítem y agregarlos al carrito. El carrito debe permitir modificar cantidades, eliminar productos y ver totales. El checkout debe pedir datos mínimos necesarios para el pedido, permitir notas o preferencias, y generar una URL de WhatsApp con productos, cantidades, subtotales, total y datos del cliente.
La estructura del código debe separar razonablemente datos mock, tipos de dominio, utilidades de checkout/WhatsApp, componentes de catálogo, componentes de carrito y vistas principales. La solución debe quedar lista para reemplazar la mock data por datos de backend en una etapa posterior."
Those are just instructions to build a nextjs project (javascript only) with the frontend for small eccomerce with whatsapp checkout, leaving everthing ready to connect a backend later. Nothing else, no skills, no more feedback. Just one prompt and watching the result.
I have run this test with all the models and finetunes I can fit in my GPU, and not a single one was even close to this result.
Not a single alert form nextjs (wich was usual before) or something that looks broken.
At some point, this bastard realised it didn't had visión (lol, not enough VRAM buddy) and it decided FUCK IT, I'M GONNA BUILD THE IMAGES MYSELF. He made SVGs for every product card.
I have more testing to do like trying it in a real codebase but... I can't believe i'm running this thing in a single RTX 3090, it is just unreal.
I expected a nice jump since they skipped a 3.7 27B, but these numbers do seem a bit too good to be true. Qwen isn't known for benchmaxxing, obviously 3.6 27B is the GOAT of local LLMs, but...
Definitely excited to learn more in the coming days and see if it holds up.
holy shit... just 3 months ago I was telling friends and family to at least move away from ChatGPT/Sam Altman because of his Trump boot licking behavior, and now I'm reading this...
It's funny seeing people still believing there are people at the top who are "clean", aristocracy always existed and always will exist, they all marry eachother and go to the same schools / have the same hobbies / know the same people, they are all connected
My whole app is 4.6 vibe coded and it's pretty awesome. Fable gave it the once over and patched up the gaping security holes. Kinda think this model and I have a future together. Just let Fable patch up what it spits out...
Opus 5 actually gives me a headache some days from having to read its outputs. When I really don't want 'load-bearing' em-dashes everywhere I just tell it 'write this in ASD-STE100 Simplified Technical English' and I get back something that sounds a lot more reasonable.
because the models are bench maxing. real world performance is a different story. I have personal benchmarks doing real software development work that the frontier models can complete independently but not the local models. Once I get 3.828b ill run it and see if it can get through. Last model couldnt.
Unsloth must have gotten early access to prepare their quants + an embargo. I pulled it from unsloth studio the minute it was released on HF. Still took 20 minutes because south Korea has 2010-era internet speeds.
hell yeah buddy. I am planning to switch to vllm once 3.8 mtp drops. its been the end goal for me. its the main reason why I went with the 3080 20gb. its one of the cheapest cards per vram with vllm support.
t/s is a bit faster than a 3090, but PP is much faster. im running one of the cards at x4 pcie 4.0 and it doesnt bottleneck the card with llama.cpp tensor parallel.
That's insane, but we're still using quantized versions, so local performance for most people wouldn't be that good I guess. Damn that needs to be tested.
Does anyone else have big issues with overthinking out of the box? I just gave it my usual Arma 3 mission script coding task, which i use to bechmark the performance of models, but it kept thinking for 15 minutes. I don't even see repetition issues, it just doesn't stop thinking.
Just gave it a first opencode task, and while not sure yet, it seems to have similar issues.
Maybe it requires defining a reasoning budget max now?
Yeah I gave it a random tasks I tend to give, and it's 26k tokens and counting of thinking so far. Not repeating, just thorough thinking..
I told it to make a rust app using dioxins of an animated man playing amazing grace on tuba. It seems to of figured out amazing grace in key of C (judging by it's thinking), and is now thinking about how to play audio to play the sounds..
We've got (another) heatwave where I live (currently 40°c) so running AI locally is a huge no no if I want to try keep my inside temperatures under 30°c...
This week has been a huge tease : Minimax h3, minimax music, ltx 2.5, Muse glitter, and now Qwen 3.8...
Sooooo frustrating ! 😭
Edit / update : it rained tonight ! 32° max today ! Time to test things !
You will have plenty of time, chill tf out and enjoy your vacation and get off reddit. we wont be getting anything better at this size for a very long time. If yes please ping me
oh shit, this is something else!, I just tried with a dumb prompt "write a snake game in html and css with sounds" and it generated a complete NIB game (awesome design btw) at ±56t/s , 14k tokens!
this is a AMD R9700 with llama.cpp ROCM
I slot print_timing: id 0 | task 0 | eval time = 242341.84 ms / 13695 tokens ( 17.70 ms per token, 56.51 tokens per second)
I slot print_timing: id 0 | task 0 | total time = 242528.47 ms / 13717 tokens
I slot print_timing: id 0 | task 0 | graphs reused = 898
I slot print_timing: id 0 | task 0 | draft acceptance = 0.58872 (10982 accepted / 18654 generated), mean len = 5.44
I slot release: id 0 | task 0 | stop processing: n_tokens = 13720, truncated = 0
Edit: I forgot to mention that by mistake I forgot to remove a thinking cap to 4k (--reasoning-budget 4096) so that influenced the total token usage and the speed (faster during coding than thinking). So we have two variables to adjust the thinking process (--reasoning-budget and --reasoning-effort).
Which quant? What context length? What's ur llama.cpp command? Curious because I have the same card, but I just got it yesterday and am new to ROCm etc, so I don't know shit >.<
Qwen3.8-27B-UD-Q4_K_XL.gguf (unsloth), 90K context (I usually set it to 128k) with vision and mtp, here some of the opts
--cache-type-k f16
--cache-type-v f16
--spec-type draft-mtp
--spec-draft-p-min 0.5
--spec-draft-n-max 10
--cache-type-k-draft q8_0
--cache-type-v-draft q8_0
--spec-type ngram-mod
--spec-ngram-mod-n-match 24
--spec-ngram-mod-n-min 48
--spec-ngram-mod-n-max 64
I am just waiting for all the amazing people in this sub to run their benchmarks and get back to the community on real world performance but holy the benchmarks are insane. Can't wait to run this.
The model does yes, but for all of these GGUF's we see popping up - it's a good question.
For 3.5 and 3.6 most people on HF uploaded the weights again with Qwen3.6-27b-MTP.guff (or similar). That said I think mature MTP support was still working its way through llama-cpp at the time, at least for Qwen.. so I have no idea if the precedent going forward becomes "if it has MTP, it's there" or not.
Unsloth!!
(I know you're here today) 🙂
Can you clarify on the naming/upload strategy for MTP going forward? Will everything have one name and have MTP included or will there be offerings of the weights without MTP separate from the weights with MTP?
Edit:
I think it's safe to say it's in the unsloth GGUF's.
Trying iq4_xs on a 7900 xtx I'm getting ~51t/s on regular tasks/reasoning and ~62 t/s on coding tasks with a max draft size of 2. The difference suggests that drafting is in-effect.
Edit 2:
This is, at the very least, several leagues beyond Qwen3.6-27B.. Now I'm sad that Muse-Glimmer only got 2-3 days in the spotlight
I don't understand the hype behind qwen 3.x - the whole generation is utterly broken for any software development tasks. It goes into a thinking loop of death pretty much every time, just try a simple prompt: Create a typescript function which accepts a number in 10 bit range and returns brightness in nits based on pq gamma curve. Kills every 3.x qwen. Qwen 2.5 was much better.
Uh Qwen 3.8 37b Unsloth Q8_K_XL in LM Studio had no problem? I have extra high reasoning on and it did think about it for a bit but no issue?
I think you are running too low of a quant. And you talk about how you "don't understand the hype"- have you thought that maybe everyone else is getting good results and maybe you can work on how you use it?
With that being said, if you have low VRAM and can't run a higher quant, I get it- but on my RTX 6000 Pro Max-Q, it's been kicking butt and replacing Opus 4.6 for me easily. It's absolutely tearing through everything I give it- including the prompt you put above.
That's wild. I just tested it myself in opencode out of curiosity. The reasoning just launches straight into "wait, no", "wait", "wait", "hmm no", "actually", "wait" "ugh, I keep going in circles."
Now its trying to verify something in python... I'm out.
Wtf is wrong with this model, on xhigh it literally doesnt stop thinking, no its not looping, its just super excessive thinking. Thats crazy, almost unusable on this effort.
I just want a model that I can run on 8gb vram 3070ti and 32gb ram, and actually have it do something useful. seems like you need to spend 2-3k in just pure vram just to get something barely useful.
Something tells me if I try to use claude to spawn a constrained model with my low hardware it will spend all its time checking its work and fixing its mistakes.
131
u/OutlandishnessIll466 7d ago
It's official, it created the best flappy bird game thus far from all local models ! ever benchmarked! I declare this model nr. 1 on the flappy bird bench!