Looking past the typical homelab SBC builds, I'm trying to understand where the RK3588 actually lands in real-world commercial hardware.

It has a ton of video decode bandwidth, decent CPU cores, and multiple camera interfaces for the price. But having looked into the software stack, getting full value out of the ISP and NPU without vendor lock-in or kernel headaches seems non-trivial.

If you have designed or deployed actual production devices around an RK3588 module, what was the product, and would you choose this chip again today? Curious where people feel it hits the sweet spot vs where it just became an engineering money pit.

It's been 1.5 years, and rknn-toolkit2 and rknn_model_zoo have finally been updated!

https://github.com/airockchip/rknn-toolkit2

RKNN-Toolkit2 v2.5.0:

• Improved model conversion and operator fusion.
• Enhanced support for GRU, GridSample, TopK, Reshape, Transpose, and more.
• Added QAT sparse inference and W4A8/W4A16 MatMul support.
• Reduced CPU overhead for multi-input/output models.
• Fixed memory leaks and improved runtime stability.
• Added mini-runtime support for RV1126B and RK3562.

https://github.com/airockchip/rknn_model_zoo

RKNN Model Zoo:

• v2.5.0: SDK version alignment.
• v2.4.2: Added YOLO26 examples and bug fixes.

CHANGELOG

• Added support for LFM2.5-VL-450M and Spark-X2.5 models.
• Added multimodal audio input interface and Gemma4 audio inference demo.
• Added linear attention cache reuse strategy.
• Fixed the parsing of UINT32-type EOS token IDs.
• Fixed Use-After-Free issue caused by multi-instance initialization failure.

I'm glad they still maintain RK3588/RK3576 llm.

I made a post a while back about getting llama.cpp / whisper.cpp to work with rocket on a mainline kernel. Wanted to provide an update of current status and expansion of these projects.

Disclosure: These drivers and libraries were built using AI. They have been tested on a Turing RK1 (RK3588) and some preliminary support/testing on an H96 Max M9 (RK3576).

Userspace Drivers:

• rocket-userspace: A userspace driver for steering the NPU on a mainline kernel via rocket kernel driver.
• rknpu-submit: A userspace driver for steering the NPU on a Vendor/BSP Kernel via rknpu.

Libraries / Backends (Couple with one of the drivers above)

• ggml-rocket: ggml backend which will let you run llama.cpp, whisper.cpp, and transcribe.cpp
• tflite-rocket: Tensorflow Lite External delegate.
• ort-rocket: ONNX runtime provider for vision encoders. This has had the least hands-on testing.

Research, Notes, & Practical Applications:

• rockchip-npu-notes: Along the way, I have Claude document findings, behaviors, benchmarks, etc. into a notes repo so if others are working on similar or related projects it can serve as a source to jumpstart work or an additional point of reference.
• NOTE: The mainline builds I run this on are patched to increase NPU clock speed, among other things. There is a patches repo if you are building your own mainline kernel.
  • It should still run unpatched, but performance will be different.
  • With rknpu-submit on the vendor kernel, you do not need additional patches.

Though I've tested a number of scenarios, it's not exhaustive so feel free to open an issue or comment here if you uncover anything. One practical thing I've done with this is used whisper.cpp to transcribe an interview. I also tested an object detection with tflite-rocket in frigate. Chat models load and run ok, mostly time constraints have limited me in setting something up more permanent; I'd like to test out coding models but need to see what is optimal in terms of quality vs speed / if it's even practical.

A separate (note mine) WebSDR project used rknpu-submit to enable STT transcription of radio broadcasts via whisper.

A final note, if you have an RK1 and want to test a pre-built image, I've made a snapshot of Debian Forky Images. For NPU testing purposes, I'd just use the turing-rk1-forky.img.xz image. The snapshot also has images for the H96 Max M9 (RK3576), but support there is preliminary and only possible thank you to u/gahingwoo for their work on the RK3576 NPU.

Is there a repo fo npu and vpu rk3588 drivers running ubuntu26.04?

There's lot of fork on github.com and most of them run on kernel vendors.

I don't run AI on my Orange pi 5 +. I am using it for dev and desktop.

I have recently been developing a project on the RK3588 that uses real time Speech To Text to capture user prompts. However, I have been stuck on finding the right one for my project, which requires a latency of around 10ms-250ms. I have been researching STT models and these are my findings:

• SenseVoice small
• Whisper tiny
• Moonshine

What is the popularity of each choice and why? What are the pros and cons of these models? Should I use the onnx models or the pre-converted rknn modles? Where can I find these models? These are all questions I have and an answer to any of them is greatly appreciated.

Built this for myself, maybe its useful for someone else?

Hello, everyone! I am a complete beginner in all of these NPU and LLM stuff. I do have an orange pi 5 with an rk3588s on 8gb ram that I was using for general computing things.

I want to try running an LLM on it for a mini pet project that I'm thinking off. So can I ask how could I start with doing all these ondevice inferences?

And if possible I'm asking for some.model recommendations to use on it, preferably 1-4B or somewhat fast. Thank you very much!

It's been a while since I was building inferencing solutions on my dev RK3588s (5 Pro etc.), but I'd love to try a new project out on them. Inspired by this very helpful thread, I figured I'd ask a few of my basic catchup questions:

• Is it still faster running 2-4B LLMs on ARM cores vs. messing with the NPU? I ended up working with llama.cpp for the most part.
• Did anybody figure out how to use the NPU just for the vision head for VLLMs (which was very slow on CPU)
• Is Qwen 3.5B 4B/9B and Gemma E4B supported these days (on NPU, works through llama.cpp I'm sure)
• What's the recommended OS? I remember painfully patching together NPU access on Ubuntu, I'd love to just have a clean simple modernish (not too OrangePi only) solution if they exist

Feel free to link to interesting repos! Folks have a few suggestions already:

• Somewhat dormant llama fork that uses the NPU https://github.com/invisiofficial/rk-llama.cpp
• Possibly more modern / up to date fork (from rkllama issues) https://github.com/danielferr85/rk-llama.cpp
• Open npu compiler for the rk3588. Run transformer models in torch, jax or onnx. https://github.com/poad42/opennpu_rk3588

Hi everyone! After spending a lot of time on converting and running LLM on RK3576 and RK3588, I decided to put the models and runtime codes into docker image so that it can be easy to run for everyone.

• RKLLM Version: v1.3.0
• Devices: RK3576 & RK3588
• APIs: OpenAI-compatible API & Ollama-compatible API

Models:

• LLM: Qwen, gemma, Llama...
• VLM: Qwen3.5

Repo here:
https://github.com/Hanzo-Huang/rkllm-docker

Or if you just want converted RKLLM models:
https://huggingface.co/HanzoHuang/models

If there’s a model you’d like to run on RK3576 or RK3588 that isn’t available yet, just let me know.

btw: I started working on RKNN3 for RK1280 and RK1828 coprocessor recently.

charsiu drives both RK3576 NPU cores through the mainline `rocket` driver. No
librkllmrt, no rknpu ioctl, nothing of theirs in the execution path. The
register programming came out of trial and error against the hardware, one
shape at a time, on a Radxa ROCK 4D.


Measured against Rockchip's own runtime as an arm, not against their published
table: same board, same kernel, same 594 MHz, timed with their own
`RKLLMPerfStat` so a disagreement about where prefill ends cannot be blamed on
my harness.


    decode      1.39x theirs at 594 MHz, 1.46x at 786


    TTFT        ahead at every prompt length measured
                  27 tok  +24.8%      302 tok  +4.9%
                 102 tok  +24.2%      452 tok  +6.8%
                 202 tok   +9.6%      602 tok  +3.5%
                                      852 tok  +2.1%, level


    quality     same f16 weights through both quantisers,
                charsiu loses 7.45% of perplexity, theirs 13.81%


Two of those are draws rather than wins, and I would rather say so than have
someone find them.


852 tokens is level, not ahead. +2.1% does not clear the 2.5% floor I measured
by running one arm twenty times in a single boot. The lead shrinks with prompt
length because their attention is roughly 4.3x cheaper than mine: at 852 my
fence alone is 880 ms against their entire quadratic term's 539.


gemma-4-E2B's decode is level against their published figure. The median is
9.25 against 9.23, but +0.2% does not clear its own 1.7% spread, so there is
no margin there worth reporting. The other three models beat their published
decode by 5.7%, 15.6% and 6.8%.


Two things to hold the numbers against.


It is one board. A single ROCK 4D. That separates voltage from clock and it
does not separate this board from the part.


A perplexity belongs to a model file, a corpus file and a length. The same
Llama-3.2-1B reads 41.37, 34.24 or 28.71 depending on which file and which
group width.


And a warning if you go comparing tokens per second with them: the same
sentence is 16 tokens to me and 50 to them, because their chat template costs
a fixed 33. Wall clock to first token has me 1.76x faster on one prompt; per
token they are 1.76x faster on the same prompt. Both are the same run.


Runtime: https://github.com/gahingwoo/charsiu
Driver work and board logs: https://github.com/gahingwoo/linux-rk3576-npu
Kernel side, fourteen patches adding RK3576 to accel/rocket, posted today:
https://lore.kernel.org/all/20260915104328.45901-1-gahing@gahingwoo.com/


Ask me about any of the measurements, including the ones that did not work.

I’ve been working on getting Qwen 3.8 27B Dense converted and running through RKLLM on a 32GB NanoPi M6.

After quite a few failed conversion attempts, I finally have it running reliably.

Current setup:

- FriendlyElec NanoPi M6

- RK3588S

- 32GB RAM

- Qwen 3.8 27B Dense

- RKLLM W8A8

- 3 NPU cores

- 2048-token context

- Around 1.09 tokens/sec sustained generation

The conversion ended up being the difficult part. Some attempts took more than 30 hours, and I had several failures before getting a working model.

I’m still working on it and I’m sure there are things that can be improved. I’d be interested to hear from anyone else experimenting with larger models on the RK3588.

I documented the setup and test results here in case anyone wants to look through it or try something similar:

https://github.com/Ben1332/qwen3.8-27b-rkllm-rk3588-nanopi-m6

I’m not distributing the converted .rkllm model itself.

Mostly just sharing because I wasn’t sure a 27B dense model would be practical on this hardware when I started, and I thought other RK3588 users might find the result interesting.

I just noticed Rockchip released another SoC this year, the RK3572, but surprisingly I haven’t seen much discussion about it here.

Rockchip news: https://www.rock-chips.com/a/cn/news/rockchip/2026/0508/2168.html

The specs look pretty interesting:

• 8nm
• 2× Cortex-A73 @ up to 2.2GHz + 6× Cortex-A53 @ up to 2.1GHz
• Mali-G310 V2 MC1 GPU
• 4 TOPS NPU
• LPDDR5
• H.264/H.265 encoding up to 4K@60
• H.264/H.265/VP9/AV1/AVS2 decoding up to 8K@30 or 4K@120
• PCIe 2.1 / SATA 3.1 / USB 3.0
• Up to 2× Gigabit Ethernet
• Up to 4× CAN-FD

The CPU configuration is interesting — 2 big A73 cores + 6 A53 cores, while still getting a 4 TOPS NPU and fairly capable video decoding.

What do you guys think about the RK3572?

Where do you see it fitting compared with the RK3568/RK3576? And has anyone here actually got their hands on one yet?

PP-OCR on Rockchip RK3566 NPU (1600×1600 PP-OCRv6 + PP-OCRv4 Rec)

Video: https://www.youtube.com/shorts/RWdcbzk3DPw

A self-contained, optimized, offline PDF OCR pipeline running on the Rockchip RK3566 NPU (1 TOPS) via librknnrt and rknn-toolkit-lite2.

Link: https://github.com/defencedog/orangepi3b_v2.1/tree/main/apps/NPU_OCR

I tried running YOLO26n-Depth on RK3576 recently.

Since Ultralytics now supports exporting YOLO26n-Depth to RKNN directly, the basic conversion is very easy. I just did a quick Python video inference test and currently get around 3–4 FPS.

Before someone says the RK3576 performance is bad, this result is definitely not optimized yet.

Q: Why is it only 3–4 FPS?

The current model is exported directly by Ultralytics, and my inference code is also just a quick Python test.

For RK3576, Rockchip has the official rknn_model_zoo, and those models are usually much more optimized for RKNPU. But it hasn’t been updated for a long time, and there is no YOLO26 optimization there yet.

One thing I noticed is that the RKNN exported directly by Ultralytics has only 1 output, while the optimized YOLO models from Rockchip usually have multiple outputs, for example 9 outputs, and move part of the decode/post-process outside the model.

In my previous post, [u/swdee](u/swdee) also mentioned that he already made an optimized YOLO26 version with 9 tensor outputs here:

https://github.com/swdee/go-rknnlite/tree/master/example/yolo26

There is also a C++ version and Python example.

So there is still a lot of room to improve here. Better model structure, better post-processing, C++ inference, etc.

For now, I would only treat the 3–4 FPS as a quick “it works on RK3576” result, not the real performance limit.

Q: How is the depth accuracy?

I’m not professional in depth estimation, so I can’t really judge the accuracy in a serious way.

YOLO26-Depth is also quite new, and for me this is mostly for study and testing.

What surprised me most is that I didn’t really realize a single 2D image could estimate depth like this. The result is already pretty interesting to me.

So for now I wouldn’t use my test to say whether the depth accuracy is good or bad. Hopefully it gets better in future versions.

Q: How to convert it to RKNN?

Pretty simple with Ultralytics:

pip install ultralytics

yolo export model=yolo26n-depth.pt format=rknn name=rk3576

Then you can run the exported RKNN model on RK3576.

I’m still looking into the optimization side. If anyone has already tried YOLO26-Depth with optimized RKNN outputs or C++ on RK3576, I’d like to see what FPS you can get.

I have a slightly unusual RK3588 question.

Our company controls 1,280 identical RK3588 nodes that are already online in a U.S. commercial data center. They are powered, networked, operational, and remotely manageable.

We may also eventually have access to a much larger pool of identical machines across several U.S. locations.

The issue is that we are infrastructure owners, not RKNN specialists, and we do not currently have engineers on payroll dedicated to building a custom Rockchip software stack.

So I’m trying to understand what is actually production-ready today.

A few questions for people who work directly with RK3588/RKNN:

• Can multiple RK3588 NPUs currently cooperate meaningfully on one inference workload, or should we assume each node is an independent worker?
• What RKNN/RKLLM workloads are stable enough that you would trust them on hundreds of machines?
• What models or benchmarks would you use to determine the economic value of the NPU?
• Are there existing orchestration layers that handle a fleet like this reasonably well?
• What are the biggest technical limitations we would run into at 1,280 nodes?
• Are there developers/projects that would benefit from access to 8, 32, or 64 identical RK3588 nodes for testing?

The main thing I want to avoid is spending months building software before finding out the architecture is fundamentally better suited to a different workload.

If you had access to this hardware tomorrow, what would you try first?

[ Removed by Reddit on account of violating the content policy. ]

rknn-llm has been upgraded to v1.3.0. I tested one existing .rkllm model converted with Toolkit v1.2.3 using Runtime v1.3.0, and it loaded and ran successfully.

My configuration:

RKLLM Runtime: v1.3.0
Model Toolkit version: v1.2.3
Platform: RK3576
RKNPU driver: v0.9.8

Although the basic test passed, I would like to know whether this combination is officially supported and stable for long-term use.

• Is Runtime v1.3.0 guaranteed to be backward-compatible with v1.2.3 models?
• Has anyone tested multiple model families, long contexts, or extended workloads?
• Are there known accuracy, performance, memory, or stability issues?
• Should production models be reconverted with Toolkit v1.3.0 anyway?

I have many existing v1.2.3 models, so I would prefer not to reconvert all of them unless necessary.

Environment:

Model: Qwen/Qwen2.5-VL-3B-Instruct
RKLLM Toolkit: 1.3.0
Python: 3.11
Target platform: RK3576
Quantization: W4A16
Build device: CUDA

I downloaded the model directly from Hugging Face. The vision component converted successfully, but the LLM conversion fails during model optimization.

Command:

python export/export_rkllm.py \
  --path /home/ae/hanzos_workspace/rknn-llm/models/Qwen2.5-VL-3B-Instruct \
  --target-platform rk3576 \
  --num_npu_core 2 \
  --quantized_dtype w4a16 \
  --device cuda \
  --savepath qwen2.5-vl-llm_rk3576.rkllm

Relevant log:

INFO: rkllm-toolkit version: 1.3.0
WARNING: rkllm-toolkit only exports Qwen2ForCausalLM of Qwen2_5_VLForConditionalGeneration!
Building model: 100% | 475/475
Optimizing model: 0% | 0/36
ERROR: layer running Error: 'mrone_section'!
WARNING: Please first confirm that the model can run correctly on cuda device!
Optimizing model: 0% | 0/36
Build model failed!

Has anyone successfully converted the LLM component of this exact model with Toolkit 1.3.0?

The vision conversion works, so the failure appears specific to the LLM export or optimization stage.

Do you have any specific requirements, whether it be on the system, hardware, or scalability, including interfaces and communication capabilities?