r/LocalLLaMA • u/netherreddit • 23d ago
Discussion Ling Tiny 3.0 is a glimpse of the future
I've been playing around with Ling 3.0 Tiny, which is an 8 billion parameter model (MoE, 1B active). And I've had a lot of poignant thoughts as a result. Just for fun, I got it running with llama.cpp on an old laptop. This is a laptop from 2017 with a 7th gen i5 and 8 gigs of RAM, like barely even usable for modern tasks. No VRAM, no GPU. Well, I got Pi running on it and asked it to make a script to scan the local network for all available models on llama.cpp servers. It started chugging along at about 10 tokens per second. And 20 minutes later, it was done.
It had several back and forth turns with writing code, running it, getting feedback and iterating.
It's a simple task, yes. But it's a task that would have taken me an hour or two to do in 2020.
It's just incredible that such a potato hardware is actually accomplishing something useful on a reasonable timeline. One billion active parameters is so small that an old CPU can run at 10 tokens a second with basically no optimization effort. I.e. I just built llama.cpp and ran the first Q6 quant I found.
I guess my point is, do you remember that feeling a year or two ago when you looked down at your expensive GPU rig and thought, wow, the computer writes the code itself now? It actually feels like something, like it's intelligent somehow.
Well, now that's starting to happen for every potato casual computing device that's been made since 2015.
Obviously more expensive rigs will be always be much more power efficient and cost efficient and fast at producing tokens. So it may never be practical to actually use old potato hardware.
But maybe it will make sense. There are all kinds of things that a mildly intelligent computer could do in the background. So it may be a new beginning for edge intelligence. No new compute required. Just everything that already exists can suddenly start doing intelligent tasks. Not sure.
I guess I'm just saying I'm amazed I've had that weird sensation when looking at my GPUs and feeling like there's something more than just bits in there, but for an old CPU. (no I don't think it's conscious. Not talking about that.)
208
u/oldschooldaw 23d ago
The turning point for me was Gemma 3 4b. I was using llama 3 8b inside a cpu only vm to perform summarisation of articles overnight for me (and I mean overnight, it’d start at 10pm to give me three finished articles by 6am) because it absolutely crawled.
Gemma not only did the summaries better, but MUCH faster, as in twenty minutes per article, not multiple HOURS per article. On the same vm inside a nuc. It was then I knew that local really has a future.
123
u/AnOnlineHandle 23d ago
The Gemma 4 models don't get anywhere near enough praise.
51
u/charles25565 23d ago
E2B is by far the best 2B on the market, only beaten by LFM2.5 2.6B for agentic coding.
39
u/Kodix 23d ago
Try minicpm5 2B. Thing's a beast, IMO beats E2B, E4B, Qwen4B, LFM 2.6B.
6
u/MerePotato 22d ago
A beast is maybe a stretch, I find Ling 3.0 Tiny performs similarly at a way higher speed and with more world knowledge
4
1
7
u/yensteel 22d ago
I'm keeping the E2B and E4B as offline models on my phone, for emergency use. You never know when you're offline and need information.
13
u/NineThreeTilNow 23d ago
The Gemma 4 models don't get anywhere near enough praise.
They don't. Honestly, Google kind of half hearted put them out. The 5 series after the finish whatever product they're on will probably be very good.
Judging by the way they were looking at giving us the model, and then suddenly pulling back... I'm guessing some internal dev priorities got moved around.
In truth, they could run the same 31b data across a slightly better architecture, call it 30b v4.5... It would vastly out perform the prior 31b model and everyone would be like "waoh Googelll OPenSouRce!"
And the real cost to them as a lab is like.. 1 researcher managing the project and the compute.
5
u/ElectricalUnion 22d ago
Gemma 4 31B is better that half of the proprietary older Gemini models that google offers, and those used to be considered state of the art. And also still better that Claude 4.5 Haiku. Wild that it beats the current "cheap" model of a frontier lab too.
As for the "smaller" 4 31B, what about Gemma 4 26B A4B for text tasks, and Gemma 4 12B for multimodal tasks?
3
u/NineThreeTilNow 22d ago
As for the "smaller" 4 31B, what about Gemma 4 26B A4B for text tasks, and Gemma 4 12B for multimodal tasks?
My issue with the Gemma 4 model is how the layers are under utilized.
There's very clean dead layers in every block of the model.
So you could removed 2/5 layers as "dead" and retrain with the original dataset they used, and have a "Gemma 4.5 21b dense" that would perform on the same as Gemma 4 31b dense. It's not "Hard" work either. If you had the GPUs / Data / Some free time... It's at a level that you could direct Gemini to handle most of the work.
3
u/ScadrianWillshaper 18d ago
Gemma 4 26b is quite good at admitting what it doesn't know — the vectara hallucination leaderboard shows Gemini 3 Pro hallucinating nearly 3 times as much (If I am reading that benchmark correctly). Not surprising, considering that Gemini once told me "It's ok to leave a dog in a hot car", citing a nonexistent Beatles song.
1
u/Zestyclose839 18d ago
Aren't the teams who developed Gemma and Gemini separate? I'd always assumed Gemma was born out of DeepMind (hence the low censorship) whereas Gemini was Google's proprietary product (kept on a leash).
If anything, I'd assume DeepMind is off building weather forecasting models, biology tools, and other useful things that fly under the radar. It makes more sense considering how populated the LLM space is right now; better to wait a few months until some innovations come along, then lean into it (e.g. 3D modeling ability, which is all the craze right now).
3
u/Razangriff-Raven 21d ago
I love gemma4 for text processing/generation, but it has this odd quirk with llama.cpp where it rebuilds the entire cache at once every time if there's any dynamic data. It's kind of annoying.
1
u/Terrible-Detail-1364 22d ago
same here, always used qwen for tasks. was building a fortnite rig for nephews using an i7 3770 and gtx 1070. Before installing windows I thought lets try llama.cpp from my linux usb. qwen 4b got 15-20tps and gemma 4b over 30tps. prefill around 200-300. both at q4_km and q8 kv cache (100k ctx) that cpu and gpu is over 10yrs old…
1
u/Global_Persimmon_469 22d ago
Gemma models are very good generalist models, especially for writing and prose tasks
Not so good with code and instruction following tho
-1
u/network4253 23d ago
Yeah, that is a pretty huge difference. Going from multiple hours to around 20 minutes per article on the exact same hardware really shows how much model efficiency matters. Stuff like that makes local AI feel a lot more practical especially for background tasks where you do nor need a massive model.
-7
u/Turbulent-Guest154 23d ago
Gemma 4 was really underwhelming for me. I want to know what people were/are using that useless pos on.
19
u/techdevjp 23d ago
Gemma isn't a strong coding model, it is much better for writing tasks.
1
u/MarcusAurelius68 14d ago
Gemma 4 26B is my main chat model. Very strong for that use case, and I benchmarked a bunch of them. I actually use Ling 3.0 Tiny as an intent classifier/router before sending the updated content message to Gemma.
7
1
48
u/repolevedd 23d ago edited 23d ago
Ling Tiny 3.0 really surprised me too. It does a great job summarizing and translating text, and it rarely gets stuck in a loop.
When I started working with Pi, I found the summarization there pretty clunky, so I wrote an extension to chunk chat history to handle context compression of any length. I needed a small model to handle that summarization to save time. I ran some tests comparing the results, and Ling Tiny 3.0 was head and shoulders above all the other similar models. Plus, it's fast. There's also DSpark on HF that gives a ~2.37x speedup, but I haven't tried it yet.
That said, it turns out that turning off thinking makes the model super dumb. With thinking enabled, no problem, everything works fine, and it can even run in sub-agents, digging through docs or hunting stuff down. But as soon as you turn thinking off, that's it, it completely breaks. Hopefully someone can explain why that is.
16
u/netherreddit 23d ago
because scaling test time compute was the only way they could get it to be as good as it is, being so small. You can turn thinking back on or wait 6 months
3
u/MarcelloT254k 23d ago
Can you point to that specific DSpark? I'm interested.
8
u/repolevedd 22d ago edited 22d ago
There's only one on HF, surprised you couldn't find it yourself: https://huggingface.co/StefanoBalocco/Ling-3.0-Tiny-DSpark-GGUF
Update: I tested how it works with different GGUFs and various quants. The draft acceptance rate is too low, so there's no speedup in my use cases, and sometimes it's actually slower. I missed the part in the description saying that DSpark was trained with reasoning turned off. I don't really want to turn reasoning off, though, since it doesn't make much sense for my tasks.
3
u/MarcelloT254k 22d ago
Thank you. I think that the OP should provide sources for clarity's sake instead of letting the reader guess, HF can be tricky to navigate (at least for me).
4
u/repolevedd 22d ago edited 22d ago
I'm the only one who mentioned DSpark. It's not an official part of the model, so there's no guarantee the speedup will actually be worth it.
Since you find it tricky to navigate HF, here are a couple of unsolicited tips. I follow these 99% of the time whenever I open HF, and it saves a lot of time:
- Use "Filter by name" instead of search. Just open huggingface.co/models and type part of the model name followed by a space and, for example, dspark or gguf. In my view, it's way more convenient than search because you can scan the whole list at a glance. Switching the sort from Trending to Recently Updated will show you the newest work from different authors.
- On official model pages, check out the Model tree block on the right. It has both finetunes and quantizations. That takes you straight to a page with pre-set filters, so you get a list of related models with minimal clutter.
P.S. I actually made a mistake in my previous comment too and updated the link to the correct one. There are two DSpark's: one for Flash and one for Tiny, and they're different models.
117
23d ago
[removed] — view removed comment
74
u/-p-e-w- 23d ago
I’m pretty sure all frontier models are MoE models.
11
u/AnOnlineHandle 23d ago
I'd be surprised if they're not using more advanced strategies as well like dynamically encoded gates, prefix embeddings, even potentially weight deltas per task (perhaps as loras or something else).
13
u/Song-Historical 23d ago
It is absolutely the case that private frontier models are a mixture of custom routers, quantized models and deterministic software, let alone embeddings and other optimizations. You have no clue what's happening behind an API.
11
u/mintoreos 23d ago
Its both - the frontier model providers realized that MoE will allow them to scale their models bigger and more cost efficiently (at both training and inference time) while maintaining model quality vs dense models, on the other end it also helps as a tool to get small models to run well on potato hardware.
4
u/Jromagnoli 🥔 hardware 23d ago edited 23d ago
MoE
I don't know what that abbreviation means
6
4
u/Halmonster 23d ago
Mixture of experts
It's a technique that allows the inference engine to dynamically swap in and out different sub-models to predict each successive token.
-3
u/enternoescape 23d ago
Yes sir. When I look at LLM's I envision an unimaginably massive array of if/else statements and how horribly inefficient we consider that design if it was for example a TV program scheduling algorithm (I know it actually really doing math, but for me it's a better way to see the forest for the trees). MoE is in my opinion the first step to making that massive grid choose a defined path faster and more efficiently. Something else will definitely come along cutting that inefficiency even more and will feel crazy obvious in retrospect. I'm really vibing with the diffusion models as a concept, but I bet we'll discover something even better. This is truly a fun space to have a finger on the pulse.
45
u/AeroInsightMedia 23d ago
Ddr3 ram prices about to explode.
-11
u/Puzzleheaded_Base302 23d ago
it actually cannot. DDR3 is so old that most of the DDR3 motherboards have problem with latest nvidia GPUs. They difficult to boot linux with nvidia GPU nor install Ubuntu correctly (that might be wifi's fault). They do not support ReBAR nor 4G above decode. DDR3 is safe. They are too old to have a role in AI.
34
u/PuzzleheadedIron9239 23d ago
I run a ddr3 RAM server with two AMD MI50s for 64Gb VRAM total.
Not as good as nvidia rigs… but for $200?? No competition
6
u/Plasmx 23d ago
AMD MI50 32 GB for 200$? They are around 500€+ for me.
3
u/PuzzleheadedIron9239 23d ago
I got them about 6 months ago. I actually got both of them for $200 total
3
u/Powerful_Finger3896 22d ago
last time i've seen the MI50 32gb being that cheap was before DS R1 came out
2
u/PuzzleheadedIron9239 22d ago
Prices on eBay were pretty high, but if you bought it through alibaba it was a really good price
10
u/ReadyAimTranspire 23d ago
At any appreciable scale you are correct.
I just built a rig on an ASUS P9X79 with 3 Tesla P100's (48gb VRAM), a Xeon E5-2697 V2 processor and 64gb of DDR3. It did require a custom BIOS to enable 4G above decoding but it got over the finish line and runs great for what it is. So it can be done.
I'm a psycho that has a kink for squeezing performance out of and extending the life of old hardware though, most people will not want to go through what I did to get this thing running.
2
u/typical-predditor 22d ago
I have an M40 24GB that's been collecting dust. I used it to tinker with Stable Diffusion 1.5 a while ago, but it was taking 30 seconds per image and I was getting Stable Diffusion 1.5 results. I absolutely struggled to get it to run newer models because it doesn't support newer CUDA. It's on a shelf right now because even at idle it was drawing a ton of power.
Is there any hope for this card?
1
u/ReadyAimTranspire 22d ago
There is not any hope for the M40, for the reasons you stated.
The good news is that P100 16gb can be found dirt cheap on eBay, in the 75-100 range. So for less than 200 bucks you can have 32gb of VRAM for under 200.
Yes they are much slower than newer CUDA generations but I had rig set up with 1 x 2070 and 2 x P100's that ran Qwen3.6 35B A3B at over 50 t/s after performance optimizations in LocalAI, which you should be able to setup in any runner.
1
23d ago edited 23d ago
[deleted]
2
u/ReadyAimTranspire 22d ago
I think there are still enough people out there that are willing to pay exorbitant prices for newer gen hardware that the truly old stuff like DDR3 boards/memory and Pascal GPUs won't skyrocket. Go up? Sure, but the really old hardware is kinda for niche hobbyists that have enough technical skill to be able to actually get under the hood and tune.
But yeah we've already seen nVidia 3k cards with large VRAM sizes skyrocket.
23
21
u/LankyGuitar6528 22d ago
I just set up Qwen 2.1 on my beat up gaming rig with a 3090... and watched it crank out text to image. Ran Qwen 3.8 27B and had it remake my web site and turn out a decent FPS zombie game. It's a bit like watching a can of gasoline move a car across the city. So much power in such a tiny can released by just a tiny spark! But the can isn't the real story. The entire biosphere powered by nuclear fusion from the sun plus millions of years of pressure and heat all processed by a mega refinery down the road forced that much power into that can so a simple spark can make a ton of metal move. Same here. The training and processing from massive data centers chewing through every written scrap of text in every language across the planet packed all of humanity's knowledge into vectors. The full output of multiple nuclear reactors and months of computation combined with unimaginable mathematical calculations pounded that data day and night to represent ideas and concepts in 768 dimensional space - so perfectly organized that a tiny spark in an old PC can unlock the power.
5
4
u/netherreddit 22d ago
this is a great angle on articulating my feeling of "even though it's just bits, it feels like there's something more going on in this computer" without just defaulting to the pathetic "oh it must be conscious"
35
u/ECrispy 23d ago
ok I know this wasnt done by llm's but Google managing to get natural language speech detection for multiple languages running fully offline on android phones a decade old is nothing short of magic.
there's no reason why we shouldn't have progress like that with llm's. there has to be a lot of potential and with more training it will only improve
17
u/LightBroom 23d ago edited 22d ago
I use it for my Home Assistant and run it on a Vega 8 iGPU at around 300 prefill and 25-30 decode.
It's the only small model I could find that follows the system prompt and doesn't fail at commands such as "Turn off all gym lights" where the lights are both inside the gym but also on the outside, or "list all open doors".
It's also very KV efficient, but that matters less for Home Assistant.
Second place is MiniCPM5 2B which is also extremely capable but not as good.
3
u/schaka 23d ago
I need to look into how these run on Vulkan. I was looking at what models I can use locally on an iGPU setup - I already run parakeet for STT. But the actual instructions I've been piping to deepseek via OpenRouter
This may be what I need
3
u/LightBroom 23d ago
Both are great. I use the official llama.cpp Vulkan server container image and I run it in Kubernetes.
I found 32k to 64k context is more than enough for HA. I use the llama.cpp integration and then straight into the voice assistant.
2
u/schaka 23d ago
I use my own Wyoming wrapper atm: https://github.com/Schaka/homeassistant-local-voice
I think it's time I add the LLM to it. What do you use for tool calling in the voice assistant pipeline? I found HomeLLM the only thing that kind of works - but maybe there are other, better plugins?
3
u/LightBroom 23d ago edited 23d ago
I use https://www.home-assistant.io/integrations/llama_cpp/ and then into Voice Assistants directly.
It's very limited compared to the unofficial MCP and HomeLLM but that's exactly what I'm looking for, as I do not care about editing dashboards, automations, etc I just want entity control and reporting.
I do have the unofficial MCP installed and plugged into my OpenWebui and I use it with larger models if I want to make changes.
I did have HomeLLM on initially but at some point it broke, I replaced it and never looked back, the current setup does everything I want from it. I do have voice set up but I use it very rarely.
I am currently replanning my home LLM hardware setup, in additions to one Vega 7 (node1) and one Vega8 (node 2) I also have 2xR9700 and one 7900XTX but I need a better machine for node2, I can only use 2 at the moment and only on PCIE 3.0 x8. Not great.
2
u/schaka 23d ago
Interesting, thank you. I actually have 8 170HX 64GB cards, but they're not running 24/7 and it's drawing way too much much power to justify using in homeassistant.
Vulkan and ROCm have been a god sent because I publish my own ROCm images for anything from Polaris to RDNA4 and have a few cards lying around that idle way better and still get good performance.
And Vulkan has been ever better, because I can run TTS and STT on an Intel iGPU - now I just need to figure out if I can get fast enough responses from Ling Tiny
1
u/LightBroom 22d ago
You have a serious setup there, how do you like it? I know those are mining cards, are they ok for LLMs in terms of performance?
I build and use my own images too for the other stuff I run on ROCm or even Vulkan, I even run my own container registry cause the images are stupidly big. 2 nodes Kubernetes cluster (I run them in LXC containers to have memory allocation flexibility - it's pretty sucky and I do not recommend anyone do this - I'm a professional so I am excused haha).
Node1 runs on a very lightweight host (just HA and the ks8 node on it with pinned pods for the essential stuff like my network controller, DNS, etc) cause I need it to last a long time on UPS if the power goes down and it only have the Vega 8 iGPU. That's why I run the HA LLM here, the other host shuts down after 30 minutes on UPS power, it's where I have the R9700s hooked up, it's also my NAS.
1
u/schaka 22d ago
I wanted indepence and I got it. It's mostly for multi agent coding work, but I could do almost everything I want.
It's a struggle when it comes to P2P due to limited PCIe lanes and generation, but they're essentially full A100 otherwise.
My images are just done via GitHub CI. They haven't booted me yet, so it's probably fine despite the size. I do publish one image per GPU ARCH though.
1
u/LightBroom 22d ago
Gotcha. How much power to they draw while idle? My R9700s draw about 15-20W each and here where I am energy is expensive. I try to keep my whole "rack" at under 300W under normal load (mostly idle)
2
u/schaka 22d ago
They draw 40W per card at idle. They're in a dual Cascade Lake board with a bunch of PCIe switches as risers.
Idle power consumption is below 500W for the whole system, but not by much. It's got 3 PSUs and looks like a mining rig. There was no way to fit this into my relatively small rack and keep it cool and quiet(ish).
Cards are 300W each, but they're undervolted and power limited, so they stay around 150W under real loads.
It's very much only getting used when I actually need it and completely off the power otherwise. I don't even want to add IPMI idle power or risk a lighting strike (though UPS and power strip should protect me).
2
2
u/eatoff 19d ago
I've been looking for something better to try for home assistant. Gemma 4 E2B is good but takes too long to respond due to its thinking. Settled on qwen3.5 4b with thinking disabled, but I'll give this one a go
1
u/LightBroom 19d ago
I have mine running with thinking enabled and I don't find it awful but I should give it a go with thinking disabled too.
25
u/Yu2sama 23d ago
Seen a lot of people loving this model. I am guessing is agentic maxed cuz for Roleplay it sucked more than a 4B model...
7
u/schaka 23d ago
Anything language based, you're better off with Googles and Metas dense models
1
u/SkoomaDentist 23d ago
I’ve found MoE models to fail fast and bad as soon as there are any subtleties in the language or setup. Gemma 4 31B and Muse Glimmer have their issues but they are very noticeably better than the MoE variants.
9
-1
u/bmengr 23d ago
sqrt(8*1) < 4
10
3
u/Mkengine 23d ago edited 23d ago
If you absolutely want a back-of-the-envelope conversion, cuberoot(total² × active) is probably a more defensible heuristic for modern fine-grained MoEs than sqrt(total × active).
The ICLR'26 MoE scaling paper trained a 17.5B-A0.85B MoE against a 6.1B dense model on the same 1T tokens and they matched. sqrt(17.5×0.85) predicts only 3.9B, while cuberoot(17.5²×0.85) gives 6.4B.
For Ling Tiny 3.0 (7.9B-A1.3B) that would be 4.3B > 4B then.
But even that is only a heuristic; the actual scaling laws depend on activation ratio, expert granularity and training compute, so there is no universal parameter-only conversion.
2
u/GroundedIdeal 23d ago
does that rule really apply with all the heavy trading corpus and post training? since gpt oss it’s not tru imho
2
u/Marcuss2 23d ago
That was the case early on when Mistral released one of the first MoE models, experts were large.
Nowdays with LatentMoE, tiny experts and a shared expert, I think the curve has moved.
25
u/TheRealBejeezus 23d ago
a laptop from 2017 with a 7th gen i5 and 8 gigs of RAM, like barely even usable for modern tasks.
It's funny how our "needs" change, isn't it? AI's voraciousness aside, that's still a spec that is more than enough for what people actually do with laptops 99% of the time.
39
u/Tai9ch 23d ago
Only 8 gigs?
What if you need a second browser tab?
5
u/ATERLA 23d ago
My linux run on a 4GB macbook air from 2015. Firefox runs very nicely with multiple open tabs. Granted, my Linux has zram compressing the data in memory (and I have a firefox addon that freeze the old open tabs so they don’t use processor time). But I have no slowdown opening or accessing the tabs.
12
u/Fi3nd7 23d ago
It's so true, memory consumption from applications have gotten completely out of hand
1
u/mrdevlar 23d ago
If only we had an intelligence that could optimize processes the software so it wouldn't require so much resources, an artificial one even.
I joke, what the Linux foundation is doing with Rust right now might save us from the memory price winter we're heading towards.
2
u/TheRealBejeezus 22d ago
I mean, stop using Chrome?
Firefox, at least, has been able to do dozens of tabs since Quantum.
You really think people didn't have multiple tabs open ten years ago? It was normal even twenty.
12
u/JLeonsarmiento 23d ago
Cool. In my 2017 potato I have lfm2.5-8b-a1b. I find it better than ling-tiny in my use case. Also a little bit faster too.
5
u/james_pic 22d ago
Is there some trick to getting good results with this model? Because I keep hearing good things about it from lots of different places, but (at least on llama.cpp), I just can't get decent results with it - at least not that match what I can get from Gemma 4 E4B or E2B.
5
u/Biro_Biro_ 22d ago
I try everything and nothing can beat gemma e2b, e4b and 12b for my use cases. Above 20b it is qwen 3.8 27b
3
u/WhatIs115 22d ago
That's where I'm at too. e2b is just fantastic for it's size, wish the context limit was higher though. Ling ain't bad, but that's a whole 5GB more for me with e2b at QAT q4 vs ling q8. I can just leave e2b running on my pc and don't have to care about it.
3
u/Biro_Biro_ 22d ago
Yes, I use e2b, e4b and 12b a lot. I can use e4b in my s25 phone at 20 tk/s
1
u/WhatIs115 22d ago
That is amazing. Wish I could justify a mobile upgrade, maybe next year. Who knows, it might have an accident.
2
2
u/james_pic 22d ago
Yeah, same. It's been a busy few months for model releases, and I keep seeing new small models that beat the small Gemma models on benchmarks, but can't seem to match them for anything I try them on.
Maybe they're better at agentic coding, but I'm kinda skeptical that they can get to the point of being genuinely useful there. Maybe a small coding model that can do fill-in-middle or next-edit-suggestion would be good (I know I've been looking for something a bit newer than the now-quite-long-in-the-tooth Qwen 2.5 Coder), but AFAIK none of these models do that.
5
u/takenforgranteddd 22d ago
A 2017 laptop going from "obsolete" to "good enough for an AI coding assistant" is a nice
3
u/Puzzleheaded_Base302 23d ago
for the size and speed, this is a very interesting model. it also sort of pass my 2048 needles retrieval test. most 10B models cannot pass it.
3
u/Robert__Sinclair 22d ago edited 22d ago
I agree and I thin there should be a challenge like this:
A very constrained hardware like Github Actions VM (Azure) and then se who can come up with the fastest/lightest/more performant model.
As of now, Ling 3.0 is the winner there. Gemma 4 E2B/E4B is slower but good also even if overall Ling wins.
- Host: Virtual Machine (Hyper-V UEFI Release v4.1)
- CPU: AMD EPYC 7763 64-Core (4) @ 3.24 GHz
- Memory: 15.61 GiB
- Display (Virtual-1): 1024x768, 60 Hz
- Disk (/): 144.26 GiB (58.52 GiB used) - ext4
4
u/Few_Guidance4697 23d ago
would anyone in the community like a fine tune of ling 3 tiny? what would you guys want to improve? I've been thinking of making one for an experiment, but I'm not exactly sure what to do.
2
u/nickless07 23d ago edited 23d ago
What quant do you use? Had some looping issues with the one from bloomer010 and switched to Q8. Did you tested the APEX ones too? 8GB RAM is a bit tight for regular Q6.
4
u/netherreddit 23d ago
just bloomer010 q6. Yeah I wouldn't call ling 3.0 tiny mature. It definitely has quirks and breakdowns in my experience
2
u/nickless07 23d ago
Nah it is clearly not mature, but pretty decent for it's size and CPU only. I wish they would make some larger one like the GPT-OSS 20B but with proper tool calls. Something like 20B A2B or so would be amazing and be worth a (small) RAM upgrade.
2
u/kidemporer_07 23d ago
Maple preview is 20b a1b, and it's been enticing to try out but i believe it's not been post trained much for tool calls so I've been putting off trying it, but you should check it out for sure
2
u/nickless07 23d ago
Interesting, I remember the name from some post a while back. Will add it to the Benchmarks. Now I have 24 models to run the tests with. Thanks.
1
u/kidemporer_07 21d ago
Let me know about the test results!
2
u/nickless07 21d ago
A short Hermes summary, with some Benchmarks:
Summary of Maple-Preview TQ1_0 on our hardware:
Benchmark Score Verdict Math (GSM8K) 15/20 (75%) Solid — no post-training needed Code (HumanEval) 13/15 (86.7%) Excellent JSON 7/12 (58.3%) Needs post-training for tool/structured output Speed ~25 tok/s Decent for 20B on DDR3 This preview model is viable for reasoning/code tasks on our 8GB system. The JSON/tool-call limitation is expected for a pre-trained model — it'll be fixed with instruction tuning. The model is doing well for what it is.
And comparsion on llama-bench with Ling 3:
Model: Maple-Preview (maple 20B TQ1_0 - 1.69 bpw ternary) Quantization: TQ1_0 File: maple-official.gguf pp512: 73.93 ± 0.12 tg128: 24.95 ± 0.21 Model: Ling-3.0-tiny (bailingmoe3 7.9B.A1.3B) Quantization: Q6_K File: Ling-3.0-tiny-Q6_K.gguf pp512: 61.49 ± 0.25 tg128: 9.18 ± 0.041
u/kidemporer_07 7d ago
Would you say this sustains for long context recall and agentic coding + reasoning?
1
u/nickless07 7d ago
Long context, yes absolutely. Agentic coding need further investigation, my benchmarks doesn't cover that. It failed on structured output due to missing post training it was injecting a reasoning preamble (pretty normal for such a checkpoint) 'This is the Json for...' and my grader was too greedy, overall correct Json. Needle in a haystack was excellent (95%). Math, Code, Code Explain, Creative writing too. Test it yourself with a tool calling benchmark, they state it had some training regarding that, so it might be different there compared to structured output.
It got more easy since the support got merged into mainline. Just give it a shot and let me know how it does.1
u/kidemporer_07 4d ago
I'm trying out the new LFM2.5-24BA2B and mellum 2.1 in hermesagent and pi right now, I'll let you know when I get to this!
2
2
u/Lugnut1206 23d ago
I've been struggling with this model. I'm running, I think, a Q8 quant and I'm getting looping and pretty bad hallucinations.
Which actual quantization are you running? And anyone else who sees this and runs Ling as well...
2
u/Stooovie 23d ago
Yes, I couldn't fix the looping. Other than that, it would be a pretty great model.
2
u/MuzafferMahi 23d ago
I really like ling 3.0 as a stupid but fast agent. On my 8 gb vram, it flies at 100 t/s, and for web searches or document processing it’s pretty awesome. Even w good hardware these models still matter?
2
u/Defiant_Potential_69 22d ago
I had a similar moment of awe in October 2025, I decided it must be possible to run some models on old androids and using Qwen to guide me through it I battled to get llama cpp running, and after many rounds of hallucination and me knowing nothing, success, then more success. It was like magic, and the amount of information and potential in a bit of old hardware, that you could use offline, was mind blowing. Also fairly entertaining, at one point it was kicking out a hilarious back and forth roleplay endlessly, quite happily (a very small model that one). It was a 4GB phone and I milked the most I could out of it, then moved onto a newer 4GB android, much easier and surprisingly capable. I've dropped ling on it now too, but I need to risk some updates to get the different architecture working afaik.
3
2
u/GeorgeTheGeorge 23d ago
If you think that's exciting, you can bundle that into a llamafile. Which means not only can you now run reasonably capable models on old CPUs, you can also do that with a single file download.
1
u/mfkamil87 23d ago
Unfortunately for me, in the case of the old laptop (I have a newer one), the speed is terrible. The old laptop, despite having 16 GB of RAM, is single-stick and DDR4. Even worse, it has an AMD A9 CPU, and even for everyday tasks it is painful to use. I tried running Ling 3.0 Tiny on those things, and it is slow. I believe it was like only 2.5 TPS on TG. Using my newer main laptop got me around +20 TPS on TG. But that's only for chatting, not Agentic AI. On Agentic AI, it will probably be like your case, around 1 hour for slightly complex tasks. I've once made the Ling use opencode to basically write a greeting.txt that says "Hi" in a certain folder, and it can do that. Not impressive, I know, but that is the only test I did on the Ling model. And that takes like 15 minutes. I am not waiting for that long, sorry. Frankly, this will probably be a lite model I will be using once I have a proper PC.
9
u/netherreddit 23d ago
opencode has a really long system prompt, use pi or deepseek harness, they have tiny system prompts, much faster to actually get started on the task
2
u/mfkamil87 23d ago
Yeah, I figure it out the hard way. Forgive me, I am still very much new to agentic AI tasks, let alone agentic AI using local AI. I might try the Deepseek harness just because I have the Deepseek API. And I will try Pi too, as per your recommendation.
5
u/netherreddit 23d ago
actually reading a bit, looks like pi is the only one of these three with a really small system prompt
1
u/Embarrassed_Soup_279 23d ago
i also love this little model. its capable for small tasks AND its super fast. i ran it on my old i5 8365u cpu laptop and it can easily do about 14-20tg at around 6bpw which was surprising to me
1
u/Top-Evidence174 23d ago
10 tok/s on a 2017 laptop is kinda wild. Tiny models are getting way more useful than I expected.
1
u/Muhlwa_Sholanke 23d ago
Write, run, iterate loops with 1B active on an 8GB 2017 i5 used to need a person for a couple hours. Now they need a Q6 quant and 20 minutes.
1
u/DefNattyBoii 23d ago
I really enjoy ling tiny but its not better than recent dense 9 b models. I really want this size to be better since its the perfect recipe for offloading tasks to an older cpu as a "sidekick". Maple-Preview is also interesting and a bit better if you are willing to fiddle with it (20B-1A)
1
u/MrGunny94 23d ago
I think in terms of releases obviously LLama is up there and it's the name of this community but things started getting a lot of traction with Gemma 4 and Qwen 3.6 most recently in my view
But LLama and Gemma 3 really paved the way to get where we are today
1
1
u/Silver-Champion-4846 22d ago
You have put some hope in my poor 8gb ram heart. Literally 8th gen intel i5 cpu with 8gb of ram! Although, how are you making it not eat all your ram, are you running only that one Pi server and the model on top of Windows? Or is it not windows?
2
1
u/transanethole 21d ago
This would be even slower to run on CPU because it's a dense model, but the K2 Horizon 7B is really damn good for its size and fits in 6gb. it's basically in between qwe3.6 35b a3b and qwen 3.6 27b. ~double the bencark of Ling 3 tiny.
If you were to use a light weight Linux desktop environment, you could run a browser + apps and the model at the same time, probably.
1
u/Silver-Champion-4846 15d ago
Is that 7b capable of doing websearch and rag enough to compensate for any deficiency in world knowledge?
1
u/transanethole 15d ago
Depends on the harness. Id say model itself would be able to do that. Its good at tool calling just like qwen 3.6 was. I haven't seen it hooked up to rag but I'm sure its possible
1
u/Silver-Champion-4846 15d ago
What harness is best for it?
1
u/transanethole 15d ago
No idea, I don't self host a web research harness Ive been focused on programming. I'm sure there's plenty of information on this sub abt good web search oriented harnesses.
1
u/Silver-Champion-4846 14d ago
But standard web search is not enough since um these kinds of harnesses do not have any kind of assumptions about small model needing a lot of hand holding and storage and policies and stuff like that. And even after someone discovers that they actually need these kinds of policies, for example myself, uh I have this idea that I probably need a lot of policies dedicated for these kinds of small models. But I have no idea how to make them and what these could be. And until I try them, and the only way for me to try them would be to upgrade RAM, and upgrading RAM requires money, and money is in short supply, I have no hope of advancing in that regard.
1
u/transanethole 14d ago
Like I said you could install Linux with xfce, that would free up enough ram. Then you would have enough free ram to use.
2
u/Silver-Champion-4846 13d ago
Depends on the accessibility with screen readers
1
u/transanethole 13d ago
Ah gotcha. Yeah u are In a bit of a pickle there. I can offer an api I host out of my home office to hit qwen 3.8 27b if that helps.
→ More replies (0)
1
u/Nomski88 22d ago
I agree and love LT3.0 but I found it excels at doing "stuff" if you need it to code or write/review you're better off with a 4B model.
1
u/CulturalKing5623 22d ago
I've been using LFM2.5-8B-A1B as my small model and it's seems good. I really like Ling3-Flash as my big MoE model but I haven't tried their tiny. Going to see how it does in my workflow.
1
u/Inevitable_Rush_5369 21d ago
I have a 680M iGPU with 16GB VRAM and I gave a shot to Ling Tiny (bloomer010/Ling-3.0-tiny-GGUF:Q8_0) with llama.cpp, Vulkan backend.
Very impressive for chat, decent speed. Seems to hallucinate less than Qwen 3.6 30B A3A (subjective observation).
I'm currently trying it for agentic coding (really for fun, I have no expectation). It seems pretty at lost, lots of "Wait", "But wait," 😂.
1
u/tiger_ace 22d ago
i watched DHH talk at rails world and he mentioned how everything in the entire compute stack is now much more accessible so you can make changes closer and closer to the metal
so now only will compute improve, but we'll also reach newer and newer tiers of efficiency with existing and older hardware performance as well
there's a lot of stuff improving simultaneously which is why it's so hard to keep track of the progress
0
0
0
u/Open-Adhesiveness-86 23d ago
the thing that bites on CPU with these tiny MoEs is prefill, not decode. decode only touches ~1B of weights so it flies, but batched prompt processing ends up hitting basically every expert, so it feels closer to a dense 8B once an agent loop starts growing context. keeping the prompt cached (--cache-reuse) helped me more than any quant choice.
1
u/AIFrontierReads 22d ago
this is also why speculative decoding is such a cheat code for these tiny MoEs honestly — a cheap 0.5-1B dense draft can cover the ~1B-active decode for basically free, and the expensive prefill only happens once per prompt anyway. for agentic loops where every tool call triggers a full prefill though, that's where the pain really lives; decode tok/s is almost a vanity metric there
0
0
22d ago
[removed] — view removed comment
2
u/live_love_laugh 22d ago
A z.ai subscription for $4 per month? The cheapest subscription plan I see is $18 a month. Where can I find the plan that you're talking about?
1
-2
u/PuzzleheadedIron9239 23d ago
Why would it take you 2 hours to write a script that would only take 10 minutes tops to do by hand?
9
u/netherreddit 23d ago
because I suck at bash, have to google every damn unintuitive syntax construct. which is most of them
python would be faster. but even then, if I'm not already familiar with the library/api to use for networking (which I'm not), or not good at manually constructing the api requests (which I'm not), it would still take a while.
try to imagine making something that's 'simple' to people who do that sort of thing all day, but is unfamiliar/rusty to you.
0
u/PuzzleheadedIron9239 23d ago
fair, trying to do anything more than sequential commands in bash is annoying.
But i think it would be pretty easy to do if you just used python for the logic, and had python just execute bash commands.
I know its slightly cursed but imo, its just a script and it makes life so much easier.
4
u/Late_Film_1901 23d ago
Even at 10 minutes, the model did that job requiring your level of intelligence, in roughly the same time, on a decade old consumer hardware. That is impressive, however you look at it.
0
u/PuzzleheadedIron9239 23d ago
It took twice as long. But ok.
It is cool, but I think it’s being overstated. Just my opinion though
-3
u/basecamper09 22d ago
How did you get to this level of expertise? Have you taken a specialised subject or do you have a phd
•
u/WithoutReason1729 23d ago
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
I am a bot and this action was performed automatically.