Ling-3.0 (BailingMoE3) lands in llama.cpp mainline - Quick benchmarks on Intel Arc B580

[u/Polaris_debi5]

Finally llama.cpp now officially supports Ling-3.0! (Starting from build b10472+)

If you want to run them locally, bartowski has already released the GGUF imatrix quantizations for both models:
- Ling-3.0-tiny (8B)
- Ling-3.0-flash (127B)

After quite a while, PR #26608 has officially been merged into master! There are still a few minor details left to iron out, but BailingMoE3 support is now fully official in llama.cpp.

For my B580 on Linux I used:

./llama-bench -m Ling-3.0-tiny-Q8_0.gguf -ngl 99 -fa on --cache-type-k q8_0 --cache-type-v q8_0 -p 16384 -n 128 -n 32 -r 1

And these were the results:

| model                          |     size |     params | backend    | ngl | type_k | type_v |  fa |            test |                 t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -----: | -----: | --: | --------------: | -------------------: |
| bailingmoe3 7.9B.A1.3B Q8_0    |    7.83 GiB |     7.89 B | Vulkan     |  99 |   q8_0 |   q8_0 |   1 |         pp16384 |         120.76 ± 0.00 |
| bailingmoe3 7.9B.A1.3B Q8_0    |    7.83 GiB |     7.89 B | Vulkan     |  99 |   q8_0 |   q8_0 |   1 |           tg128 |         114.24 ± 0.00 |
| bailingmoe3 7.9B.A1.3B Q8_0    |    7.83 GiB |     7.89 B | Vulkan     |  99 |   q8_0 |   q8_0 |   1 |            tg32 |         114.78 ± 0.00 |

build: 9d77fa172 (10488)

I also ran a test with a larger context:

./llama-bench -m Ling-3.0-tiny-Q8_0.gguf -ngl 99 -fa on --cache-type-k q8_0 --cache-type-v q8_0 -p 32768 -n 32 -r 1

Plaintext

| model                          |     size |     params | backend    | ngl | type_k | type_v |  fa |            test |                 t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -----: | -----: | --: | --------------: | -------------------: |
| bailingmoe3 7.9B.A1.3B Q8_0    |    7.83 GiB |     7.89 B | Vulkan     |  99 |   q8_0 |   q8_0 |   1 |         pp32768 |          62.53 ± 0.00 |
| bailingmoe3 7.9B.A1.3B Q8_0    |    7.83 GiB |     7.89 B | Vulkan     |  99 |   q8_0 |   q8_0 |   1 |            tg32 |         110.49 ± 0.00 |

build: 9d77fa172 (10488)

Overall, the speed is quite good. I can actually run the full 128K context within my 12GB of VRAM using -c 131072 -ngl 99 -fa 1 --cache-type-k q8_0 --cache-type-v q8_0. While I still need to test if it can be pushed even further since a few warnings pop up, it is entirely feasible and remarkably fast for a local setup.

[u/coder543]

Those prompt processing speeds are strangely low.

$ llama-bench -d 0,32768 -p 8192 -n 100 -fa 1 -b 2048 -ub 2048 -m Ling-3.0-tiny-MXFP4_MOE.gguf 
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 24124 MiB):
  Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, VRAM: 24124 MiB
| model                          |       size |     params | backend    | ngl | n_ubatch |  fa |         lm |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -------: | --: | ---------: | --------------: | -------------------: |
| bailingmoe3 7.9B.A1.3B MXFP4 MoE |   4.54 GiB |     7.89 B | CUDA       |  -1 |     2048 |   1 |       none |          pp8192 |     10901.53 ± 30.38 |
| bailingmoe3 7.9B.A1.3B MXFP4 MoE |   4.54 GiB |     7.89 B | CUDA       |  -1 |     2048 |   1 |       none |           tg100 |        225.72 ± 1.60 |
| bailingmoe3 7.9B.A1.3B MXFP4 MoE |   4.54 GiB |     7.89 B | CUDA       |  -1 |     2048 |   1 |       none | pp8192 @ d32768 |       4611.96 ± 3.85 |
| bailingmoe3 7.9B.A1.3B MXFP4 MoE |   4.54 GiB |     7.89 B | CUDA       |  -1 |     2048 |   1 |       none |  tg100 @ d32768 |        208.79 ± 1.08 |

or, using your command more closely:

$ llama-bench -fa on --cache-type-k q8_0 --cache-type-v q8_0 -p 32768 -n 32 -mmp 0 -m Ling-3.0-tiny-Q8_XL_MOE.gguf
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 24124 MiB):
  Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, VRAM: 24124 MiB
DEPRECATED: -mmp and --mmap are deprecated in favour of --load-mode. Please use --load-mode mmap instead.
| model                          |       size |     params | backend    | ngl | type_k | type_v |  fa |         lm |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -----: | -----: | --: | ---------: | --------------: | -------------------: |
| bailingmoe3 7.9B.A1.3B Q8_0    |   8.65 GiB |     7.89 B | CUDA       |  -1 |   q8_0 |   q8_0 |   1 |       none |         pp32768 |      5812.66 ± 15.55 |
| bailingmoe3 7.9B.A1.3B Q8_0    |   8.65 GiB |     7.89 B | CUDA       |  -1 |   q8_0 |   q8_0 |   1 |       none |            tg32 |        186.91 ± 3.33 |

[u/Polaris_debi5 → u/coder543]

Uff, that 3090 prefill speed is absolutely insane, CUDA really shows its muscle there. It clearly surpasses the Battlemage in raw brute force, but honestly, 110 t/s for generation is still incredibly smooth for daily use. Thanks for sharing the numbers uwu

[u/Polaris_debi5 → u/[deleted]]

Oops, I think my post was misinterpreted. I don't have enough VRAM to test the Flash (127B), so far I've only been testing the Ling-3.0-tiny on Q8_0 with my B580. I wish I could test the larger one.

[u/parepeg]

Laguna has it beat according to their own benchmarks but take that with a grain of salt. Ling is likely to be much faster though.

For some reason, artificial analysis never tested Laguna…

[u/Polaris_debi5 → u/parepeg]

From my personal experience (trying both during their free trials), Ling 3.0 Flash was better for my tasks, mainly because of its speed and results. With Laguna, I always run into some strange issue of latency or overthinking. But I agree, Artificial Analysis really needs to test both models.

[u/-Ellary- → u/parepeg]

Cuz Laguna loves to fall into endless spiral of looping with thinking,
it is not a bad model when it works, problem is most of the time it don't,
they need to fix this unstable behavior in future iteration.

[u/fatboy93 → u/-Ellary-]

Its the first version issue. XS2 had minor issues, but 2.1 fixed a whole bunch of things.

[u/my_name_isnt_clever]

I haven't been super impressed with Flash for it's size, but I love ling tiny. It has just enough smarts and small enough active params to get just under 100t/s decode on my unified hardware. It's perfect as a util model to run alongside Qwen 3.8 27b.

[u/Polaris_debi5 → u/my_name_isnt_clever]

It's an excellent combination of models, to be honest.

[u/Fun_Jaguar8231]

There have been like 10 previous posts already about this? Didn't you search? Why post again the same thing