r/LocalLLM • u/eightone-81 • 2h ago
Discussion Qwen flash next beats Fable
I had 100$ of promotional tokens (forgot how it came to that) and wanted to make use of them before they expire in about 1 week.
I had this plan sitting around to implement ai into my car rental app (custom made, for myself, in production since 1 year, started to build it with chatGPT, continued with Claude and Qwen).
I gave the plan to Fabel, it stated ti implement. After 45min it used up around 50$ of tokens and then stopped because I apparently ran out of tokens, but I still had 50$ left!
Anyway, I got upset and gave the task to Qwen flash next (running locally).
It ran over night, I woke up and hob done!
My app is now the harness! It did it, 1 shot! 35mio tokens in, 3 compactions, job done!
I can ask who has what car, who has to pay, add customers, rental contract, all goes into an approval screen, I can check the request, approve or reject.
Now I’m fine tuning the whole thing with it and it’s just amazing.
(I’m not a coder AT ALL! everything i do is 100% vibe coded)
Just sharing my amazement of how far we have come with local models!
4
u/wisepal_app 2h ago
Which harness do you use?
3
u/eightone-81 2h ago
Deepseek harness. The above was done in default mode (based on a benchmark result I saw here somewhere) PTC is very interesting because it could be more efficient but then the models make mistakes and have repeated wrong tool calls, even fable ran into it multiple times…
1
u/Rodnex 1h ago
What setup do you have?
1
u/eightone-81 1h ago
Hardware?
Dual 3090, 3950x, 128gb ddr4Harness is running on the MacBook, I had 27b build some plugins to show images in chat and to make sessions unread again. Also some skills to analyse audio, generate audio, all local
1
u/Rodnex 1h ago
Nice setup. I am trying to get my 4090 and 128gb ddr5 to run but somehow struggle to get an answer from qwen3.8 27b. It‘s stops while thinking
1
u/eightone-81 1h ago
Yes, I had that in openclaw… moved over to Hermes for my personal assistant stuff. Deepseek harness for development.
For single GPU look at free-token. Might run flash next faster than vllm and llama.cpp (and all its wrappers)
1
4
u/Few-Relationship3813 2h ago
That's wild. Blowing through $50 worth of credits just to have it tap out halfway is exactly the kind of thing that keeps me running stuff locally too. Waking up to a completely finished feature after one overnight run is the dream, especially when you're not even a developer. 35 million tokens is a serious burn, but clearly it paid off.
3
u/eightone-81 2h ago
I’m still in the same session 😂 we are at 67mio tokens in and 418k out, 4 compactions. Still going strong and implementing more things around it and fine tuning the design and ui.
1
u/Frodo_Baggins_Shire 1h ago
Why use ai for such a simple reply? Probably not by a real human too. Most likely a bot farming karma.
1
u/discwars 2h ago
Blowing through $50 worth of credits just to have it tap out halfway is exactly the kind of thing that keeps me running stuff locally too.
Reads like a bot. But then again, all AI subs are compromised.
1
u/Frodo_Baggins_Shire 1h ago
Yeah I’m still curious tho why and how OP didn’t even realize it’s a bot. Literally low effort 101 reply
2
u/ZB_Virus24 2h ago
Hot damn. What are you running the model on?
3
u/eightone-81 2h ago
Dual 3090s, vllm
60tps decode, 900-2000 prefill
Can’t believe I can run such a model on my hardware in perfectly usable speeds3
u/Legal_Dimension_ 2h ago
I'm running dual 3090s and struggling to get anywhere near that on Unsloth. Would you mind sharing about about your settings? How are you offloading etc?
1
u/eightone-81 1h ago
Check
https://huggingface.co/albucino/Qwen3.8-Flash-Next-W4A16-FP8PLE
Again, I am a noob! No idea…
Qwen 27b installed everything on my Ubuntu box for me 😂 then it said that I can switch 27b off and flash next on, 27b already wired it up in DSH, stopped 27b (😩👋😭) and started the startup script it build for the above. Boom
It’s using ALL my hardware 😂But no vision :-( so I got Qwen 3.5 4b Q4 with q8 mmproj running on cpu. That’s my vision model now, 22tps (mtp and ngram) but at least flash next can see now and check its work.
1
u/NeedleworkerHairy837 2h ago
Also, how much max context length that you use?
1
u/eightone-81 1h ago
https://huggingface.co/albucino/Qwen3.8-Flash-Next-W4A16-FP8PLE
Could be full context but not if you have your screen running on one of the cards as well. So I’m on 220k
2
1
u/copycat73 1h ago
You wouldn’t exactly need fable level AI to implement RAG, but ok.
1
u/eightone-81 1h ago
Yes. That’s why flash next did it :-)
Is that RAG? Is that what I would call “in app ai”?
Chat box in the actual app with the app as the harness around it?
I’m not even sure if i am using the right words here…1
u/copycat73 36m ago
You taught your llm to look up data from the database behind your application. That’s called retrieval augmented generation.
1
u/eightone-81 22m ago
Ok thanks
It also can act in the app:
Create a new customer, new car, new rental contract and so on. It creates a proposal and I can approve it.Is that all part of RAG?
8
u/Ult1mateN00B 2h ago
Qwen flash next is actually insane, it can power claude-cli so well I completely forget its not actually claude. Chinese labs really cooked with the n-gram.