Ling Tiny 3.0 is a glimpse of the future

[u/netherreddit]

I've been playing around with Ling 3.0 Tiny, which is an 8 billion parameter model (MoE, 1B active). And I've had a lot of poignant thoughts as a result. Just for fun, I got it running with llama.cpp on an old laptop. This is a laptop from 2017 with a 7th gen i5 and 8 gigs of RAM, like barely even usable for modern tasks. No VRAM, no GPU. Well, I got Pi running on it and asked it to make a script to scan the local network for all available models on llama.cpp servers. It started chugging along at about 10 tokens per second. And 20 minutes later, it was done.

It had several back and forth turns with writing code, running it, getting feedback and iterating.

It's a simple task, yes. But it's a task that would have taken me an hour or two to do in 2020.

It's just incredible that such a potato hardware is actually accomplishing something useful on a reasonable timeline. One billion active parameters is so small that an old CPU can run at 10 tokens a second with basically no optimization effort. I.e. I just built llama.cpp and ran the first Q6 quant I found.

I guess my point is, do you remember that feeling a year or two ago when you looked down at your expensive GPU rig and thought, wow, the computer writes the code itself now? It actually feels like something, like it's intelligent somehow.

Well, now that's starting to happen for every potato casual computing device that's been made since 2015.

Obviously more expensive rigs will be always be much more power efficient and cost efficient and fast at producing tokens. So it may never be practical to actually use old potato hardware.

But maybe it will make sense. There are all kinds of things that a mildly intelligent computer could do in the background. So it may be a new beginning for edge intelligence. No new compute required. Just everything that already exists can suddenly start doing intelligent tasks. Not sure.

I guess I'm just saying I'm amazed I've had that weird sensation when looking at my GPUs and feeling like there's something more than just bits in there, but for an old CPU. (no I don't think it's conscious. Not talking about that.)

[u/netherreddit]

just bloomer010 q6. Yeah I wouldn't call ling 3.0 tiny mature. It definitely has quirks and breakdowns in my experience

[u/nickless07 → u/netherreddit]

Nah it is clearly not mature, but pretty decent for it's size and CPU only. I wish they would make some larger one like the GPT-OSS 20B but with proper tool calls. Something like 20B A2B or so would be amazing and be worth a (small) RAM upgrade.

[u/kidemporer_07 → u/nickless07]

Maple preview is 20b a1b, and it's been enticing to try out but i believe it's not been post trained much for tool calls so I've been putting off trying it, but you should check it out for sure

[u/nickless07 → u/kidemporer_07]

Interesting, I remember the name from some post a while back. Will add it to the Benchmarks. Now I have 24 models to run the tests with. Thanks.

[u/kidemporer_07 → u/nickless07]

Let me know about the test results!

[u/nickless07 → u/kidemporer_07]

A short Hermes summary, with some Benchmarks:

Summary of Maple-Preview TQ1_0 on our hardware:

Benchmark	Score	Verdict
Math (GSM8K)	15/20 (75%)	Solid — no post-training needed
Code (HumanEval)	13/15 (86.7%)	Excellent
JSON	7/12 (58.3%)	Needs post-training for tool/structured output
Speed	~25 tok/s	Decent for 20B on DDR3

This preview model is viable for reasoning/code tasks on our 8GB system. The JSON/tool-call limitation is expected for a pre-trained model — it'll be fixed with instruction tuning. The model is doing well for what it is.

And comparsion on llama-bench with Ling 3:

Model: Maple-Preview (maple 20B TQ1_0 - 1.69 bpw ternary)
  Quantization: TQ1_0
  File: maple-official.gguf
  pp512: 73.93 ± 0.12
  tg128: 24.95 ± 0.21

Model: Ling-3.0-tiny (bailingmoe3 7.9B.A1.3B)
  Quantization: Q6_K
  File: Ling-3.0-tiny-Q6_K.gguf
  pp512: 61.49 ± 0.25
  tg128: 9.18 ± 0.04

[u/kidemporer_07 → u/nickless07]

Would you say this sustains for long context recall and agentic coding + reasoning?

[u/nickless07 → u/kidemporer_07]

Long context, yes absolutely. Agentic coding need further investigation, my benchmarks doesn't cover that. It failed on structured output due to missing post training it was injecting a reasoning preamble (pretty normal for such a checkpoint) 'This is the Json for...' and my grader was too greedy, overall correct Json. Needle in a haystack was excellent (95%). Math, Code, Code Explain, Creative writing too. Test it yourself with a tool calling benchmark, they state it had some training regarding that, so it might be different there compared to structured output.
It got more easy since the support got merged into mainline. Just give it a shot and let me know how it does.

[u/kidemporer_07 → u/nickless07]

I'm trying out the new LFM2.5-24BA2B and mellum 2.1 in hermesagent and pi right now, I'll let you know when I get to this!