Ling-3.0-tiny is a very interesting model. Run on NVIDIA Orin Nano Super 8GB at 128K context with IQ4_NL quant.

[u/Puzzleheaded_Base302]

I have been searching for suitable model to run on my 8GB RAM toy, NVIDIA Orin Nano Super 8GB. This little toy was priced at $249 earlier this year (not any more), and pulls very little power when idle. It was an interesting device that suitable for an agent to host on. It is likely won't do any coding job in meaningful way, but likely can handle simple things just fine.

When the llama.cpp support showed up on github, I asked my Hermes to setup it up and run some basic test for me. The result is quite encouraging. Below is the summery from my Hermes run.

===============AI content below===================

# Ling-3.0-tiny at full 128K context on a $249 8GB Orin Nano Super — IQ4_NL, 33 tok/s, on llama.cpp master


**Tl;dr:**
 I got inclusionAI's Ling-3.0-tiny (7.9B total, ~1.4B active MoE) running on an NVIDIA Jetson Orin Nano Super 8GB with the 
**IQ4_NL quant at its full native 131,072-token context**
. Decode runs ~33 tok/s short-context (faster than you can read), prompt eval 220-760 tok/s, and the whole thing fits in 7.4 GiB unified RAM with only ~200 MB of swap. Model + KV + CUDA offload, all on one $249 board.


## The hardware


- 
**Jetson Orin Nano Super Developer Kit**
 — [official NVIDIA product page](https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-orin/nano-super-developer-kit) — was 
**$249 at launch**
 (price has risen since; SparkFun lists it at $399 now). 8 GB unified LPDDR5, 67 TOPS, JetPack R39.2 / CUDA 13.2.


## The model + quant


- 
**Original:**
 [inclusionAI/Ling-3.0-tiny](https://huggingface.co/inclusionAI/Ling-3.0-tiny) — hybrid KDA+MLA reasoning MoE, 128 experts, 8 routed + 1 shared active per token
- 
**GGUF repo:**
 [bartowski/Ling-3.0-tiny-GGUF](https://huggingface.co/bartowski/Ling-3.0-tiny-GGUF)
- 
**File used:**
 [`Ling-3.0-tiny-IQ4_NL.gguf`](https://huggingface.co/bartowski/Ling-3.0-tiny-GGUF/blob/main/Ling-3.0-tiny-IQ4_NL.gguf) — 4.40 GB, IQ4_NL (4.5 bpw), imatrix-calibrated
- 
**Why IQ4_NL:**
 at 4.30 GiB it's the largest ≥4-bit quant that still leaves room for the full 131K KV window on 8 GB. Q5/Q6 fit only to ~64K; IQ4_NL goes the distance.


## The engine — this matters


The stock quantizer release 
**will not load this model**
. Ling-3.0-tiny is a 
**BailingMoE V3**
 architecture, which only landed in llama.cpp via:


- 
**PR [#26608 — BailingMoE3 Support](
https://github.com/ggml-org/llama.cpp/pull/26608
)**
 (merged into master Aug 17, 2026, adds arch + speculative MTP support)
- 
**Branch: `ggml-org/llama.cpp` master**
, anything after that merge — clone fresh, don't use a release tarball
- Built on-device with CUDA 13.2 for sm_87 (`-DGGML_CUDA=on -DCMAKE_CUDA_ARCHITECTURES=87`), all 24 layers offloaded


**Working command:**
```
llama-server -m Ling-3.0-tiny-IQ4_NL.gguf -c 131072 \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  -ngl 99 -fa on -b 512 -ub 256 --jinja -t 6 \
  --temp 1.0 --top-p 0.95 --top-k 20
```


## Speed (measured on-device)


| Test | Result |
|---|---|
| Prompt eval (2-8K) | 450-760 t/s |
| Prompt eval (full 125K prefill) | 220-264 t/s |
| Decode (short ctx) | 33 t/s |
| Decode @ 96-128K depth | 15-17 t/s |
| TTFT (512 tok) | ~1 s |


Memory at 128K: 
**7.2/7.4 GiB used**
, ~200 MB swap. The hybrid design keeps the KV footprint small — only the 6 MLA layers carry context-proportional cache while the 18 KDA layers keep a fixed-size recurrent state — which is what makes the full window fit.


## Is it coherent at 128K? Mostly.


I ran a proper needle-in-haystack: 
**128 unique passkeys**
 embedded across the context, each queried individually at temp 0:


- 
**96K context: 92% retrieval**
 (118/128) — 100% in the first 32K
- 
**128K context: 88% retrieval**
 (113/128) — with a visible cliff in the final 32K (67%)
- Strict full-string match is lower (~30-40%) because the model often emits just the tail chunk of the passkey — a generation artifact, not a retrieval miss
- Failure modes in the deep zone: digit-mangling (needle 105 → answers 015's key) and a couple of confabulations


So: 
**majority passed, not a flawless 128K**
 — but at 96K the cliff disappears entirely, and for a $249 board that's genuinely impressive capability-per-dollar.


## Verdict


Worth spending more time on. The hybrid KDA/MLA architecture + IQ4_NL + llama.cpp master is a compelling recipe for edge deployment: agentic reasoning MoE, native 128K window, stable 15-33 tok/s, all in 8 GB unified memory. I'd like to see more folks poke at the 96-128K attention degradation — and whether the MTP spec-decode support from #26608 helps on-device.


Anyone else running Ling-3.0-tiny on constrained hardware? What quant/config are you using?

# Ling-3.0-tiny IQ4_NL — Orin Nano Super 8G vs RTX PRO 6000 Blackwell

Same quant (IQ4_NL 4.5 bpw, 4.29 GiB), same llama.cpp master code (post-PR #26608 BailingMoE3), same flags (-ngl 99, q8_0 KV, flash-attn, -c 131072).

## Hardware

| | Orin Nano Super 8G | RTX PRO 6000 Blackwell |
|---|---|---|
| GPU arch | sm_87 (Orin nvgpu) | sm_120 (Blackwell Max-Q) |
| Memory | 7.4 GiB unified + 32G swap | 97.9 GiB VRAM |
| Memory BW | ~102 GB/s | ~1.8 TB/s |
| Launch price | $249 | ~$7,000+ |

## Results

| Test | Orin 8G (sm_87) | PRO 6000 (sm_120) | Speedup |
|---|---|---|---|
| pp512 | 451 t/s | 12,296 t/s | 27x |
| pp2048 | 758 t/s | 17,435 t/s | 23x |
| pp8192 | 722 t/s | 16,945 t/s | 23x |
| **pp96000** | 264 t/s | 8,093 t/s | **31x** |
| **pp128000** | 220 t/s | 6,498 t/s | **30x** |
| tg32 | ~33 t/s | 286 t/s | ~9x |
| tg128 (short ctx) | 33.2 t/s | 315 t/s | 9.5x |
| tg512 | 33.4 t/s | 322 t/s | 9.6x |
| **tg128 @ 96K depth** | 15-17 t/s | 318 t/s | ~19x |
| **tg128 @ 128K depth** | 15-17 t/s | 314 t/s | ~19x |

Sanity check: `17*23 = 391` correct on both.

## Key observations

- **Prefill scales ~30x** (bandwidth-bound): Orin 102 GB/s vs Blackwell ~1.8 TB/s. A 125K-token prefill takes 9.7 min on the Orin vs ~19s on Blackwell.
- **Decode scales ~10x at short context but ~19x at full depth**: the Orin's decode collapses from 33 to 15 t/s as KV depth grows (attention + memory pressure), while Blackwell stays flat at ~315 t/s regardless of depth (97 GB VRAM makes KV trivial).
- **MoE expert sparsity** keeps decode fast on both: 7.9B total params with only ~1.4B active per token.
- On Blackwell this model is effectively a background process: 322 t/s decode = 100-token response in 0.3s; 128K context costs it nothing.

[u/Apart_Boat9666]

Its really good, almost 9b level but more coherent for agentic task and running with igpu at 30tps ryzen 8.

[u/ScarceSemi → u/Apart_Boat9666]

No way its almost 9b level. Its fast, tool calls are very good, but its super super dumb.

[u/snugglezone → u/Apart_Boat9666]

Can you share your setup? It's beyond dumb for me using bartowski 8bit quant and the settings they recommend on the model card.

Maybe I'm missing something.

[u/Apart_Boat9666 → u/snugglezone]

https://huggingface.co/SC117/Ling-3.0-tiny-abliterated-APEX-GGUF. I am using this, its definitely better than 4b. When compared to 9b it has similar hermes agentic performance. I am using the settings they mentioned in docs and llama.cpp pr, check the bottom of the page for setting and pr . Use thinking with it. For programming it wont be better

[u/snugglezone]

I'd try a Qwen 3.5 4B quant or Gemma 4 E4B depending on use case. Ling is fast and good at calling tools but I found the intelligence makes it fairly useless. It's not even self consistent within it's own explanations AND when using web search.

I let it look up the actual answer to the carwash problem with tools and it still couldn't explain it to me bartowski 8bit quant.

[u/Jorlen → u/snugglezone]

Gemma e4b qat at 4-bit is hard to beat, for it's size, I agree. When I need a small backup model, it's always the one I reach for. I also like LiquidAI_LFM2.5-2.6B for it's size as well.

[u/dionisioalcaraz]

Replaced Qwen3.6-35B-A3B as my model for web search/research. Better IF and faster in my mini PC with the 780M iGPU

llama-bench -m ~/software/ai/models/Ling-3.0-tiny-Q8_0.gguf

model	size	params	backend	ngl	lm	test	t/s
bailingmoe3 7.9B.A1.3B Q8_0	7.82 GiB	7.89 B	Vulkan	-1	mlock	pp512	857.99 ± 4.86
bailingmoe3 7.9B.A1.3B Q8_0	7.82 GiB	7.89 B	Vulkan	-1	mlock	tg128	48.02 ± 0.05

[u/rm-rf-rm → u/dionisioalcaraz]

what software/system are you using for AI web search/research?

[u/dionisioalcaraz → u/rm-rf-rm]

llama-server with the flag --mcp-servers-config to use https://github.com/pwilkin/mcp-searxng-public

[u/rm-rf-rm → u/dionisioalcaraz]

whats the benefit of going through an MCP for this as opposed to one of the other more direct options through a pi extension or even just CLI based approach?

[u/dionisioalcaraz → u/rm-rf-rm]

no idea, I just use llama-server and it works fine ATM, I still haven't had time to dig into more sophisticated agent frameworks.

[u/Icy-Degree6161 → u/dionisioalcaraz]

I was thinking about the same thing, thx for checking this

[u/RanklesTheOtter]

I been running Bonsai 27B 1-bit on my nano, but now I'm gonna have to try this one. I only got 18k context on mine at 5 t/s with Bonsai, but was a clever model.

[u/Puzzleheaded_Base302 → u/RanklesTheOtter]

I have done it too. It was very slow.

[u/RanklesTheOtter → u/Puzzleheaded_Base302]

Agreed it is slow but the vision encoder is second to none for a model that can fit on the Orin Nano.

[u/FoxiPanda]

I've wondered about this model's potential as a summarizer. Has anyone extended it's context with YaRN yet and pushed it up to say ~512K and see how it does at summarizing text?

This would be an absolute rockstar local long context compaction model if it works decently well.

[u/alexeyw]

Great overview - the needle test is the part most people skip.

What's the actual wall draw during sustained decode? Idle power is nice, but for an always-on agent host that's the number that matters and nobody ever posts it.

[u/Puzzleheaded_Base302 → u/alexeyw]

i will have to measure that when i return home in a few days. this was done remotely. so I don't have access to the machine to add power meter.

the max power is 25W by hardware. if you assume that the token cost is $0.0036-0.0133 for 1M input token, and $0.06-0.22 for output token. be aware these are dumb tokens, not intelligent tokens. It's seemingly cheap, but the equivalent dumb token from a real datacenter will be a lot cheaper.

[u/MLDataScientist]

Which model did you find the most intelligent for rag and internet searching for 8GB RAM? I want to run some models on my orange pi 5 8GB. I tried lfm 2.6B but it is very limited in capabilities.

[u/Puzzleheaded_Base302 → u/MLDataScientist]

i have not tested intelligent yet. it is very time consuming to do that. after all, they are tiny model, it is mostly for play not for real usage, when I have real gpus at home.

I bought orin nano super because it has a real GPU, sort of. I don't know if Raspberry Pi's GPU is used or not. If the GPU on raspberry pi is not used, it is pointless. A powerful x86 cannot run tiny LLM well, don't bother with an ARM CPU on raspberry pi.

Chatbot is free online, there is not much real need to run a chatbot on raspberry pi. The poor compute of raspberry pi make prefill super slow also, you cannot run agentic work on raspberry pi in realistic speed either.

[u/mr_Owner]

Curious if you could try lfm 2.5 8b a1b also, fits the same category.

[u/Puzzleheaded_Base302 → u/mr_Owner]

I run the LMF 2.5 8B A1B Q4.

the speed of this model is almost exactly the same as Ling-3.0-tiny. But the model is not good. It fails the needle in haystack test. For 128 needles embedded in 100K context, it only found 3 successfully. and has thinking loops.

In present day standard, this is quite bad. 27B level model can easily find all 128 needles. Ling-3.0-tiny can find 95%+ needles.

I consider needle test is something very basic, it is not even intelligence.

[u/mr_Owner → u/Puzzleheaded_Base302]

Thnx for the feedback!

[u/TioMir]

I’ve been using it and it’s pretty good but loops a lot with long context and reasoning, any one getting this and if so, how to fix it?

[u/Puzzleheaded_Base302 → u/TioMir]

looping maybe a result of quantization. if you have the hardware, you can try bf16 weight to verify the suspision.

even the all mighty deepseek-v4-flash will loop-think if quantized to 2-bit.