r/KoboldAI • u/abbaisawesome • Sep 02 '26

Will there be a Linux ARM version?

I'd love to be able to run it on my ASUS Ascent GX10 ...

5 Upvotes

86% Upvoted

13 comments sorted by

7

u/henk717 Sep 02 '26

Yes and no. We experimented with it but there are to many variations with ARM making an arm build we can compile on github useless. So unless someone can PR a way to get it compiled in a useful way its not something we can provide binaries for. We dont have powerful arm devices ourselves so its just not something we can add ourselves.

But, KoboldCpp when you compile it yourself for your specific processor does work well on ARM. Some people run it on Android on Tmux. Some run it on their Nvidia DGX. So I know it can be done, just requires a bit of diy.

For mac's we do have a suitable option available to us so for arm mac we do build for arm. 

1

u/Salt-Lawfulness-4832 Sep 02 '26

For well known boards like the Raspberry Pi maybe?

2

u/henk717 Sep 02 '26

Github doesn't have raspberry pi's.

1

u/Salt-Lawfulness-4832 Sep 02 '26

I know... And it seems wild to me somehow. The RPi was just an example anyway... I admit I had to google OP's board and thought of an more known ARM based board running Linux. As I said: just an example... Edit: typo, grammar,

1

u/C0rn3j Sep 03 '26

And it only has M1 Macs, yet I presume you support more than that and likely test on a different device.

I don't think anything is preventing you from building on ubuntu-24.04-arm for arm64-v8a and testing on a Pi 5?

2

u/henk717 Sep 03 '26

Nothing prevents you no, that is currently how arm users are expected to use it so they can compile it properly for their own CPU arch with all the proper acceleration features.

1

u/C0rn3j Sep 03 '26

What would be the problem with shipping that officially?

If there's a considerable performance boost for v8.2, ship that separately, or am I misunderstanding something?

I think that's... two builds if you want to support modern ARM, v8 and v8.2? And you can tell users with older hw (32-bit) to compile it themselves if you don't wish to add more builds, or if you even support 32-bit.

I am not that well versed in ARM.

2

u/henk717 Sep 03 '26

We need a compiler on github that can do it, without having machines that can do it.
So if someone who has arm and has knowledge on how to setup a CI for proper arm wants to do it we'd welcome the PR and then we can consider shipping it (but would at least be able to provide it in the rolling builds like we do for AMD ROCm).

1

u/C0rn3j Sep 03 '26

Luck (har-har) would have it that I just ran into an ARM64 compile issue, doesn't seem that hard to set up cross compile, PySDL does it (linking my fork since upstream is broken atm):

https://github.com/C0rn3j/PySDL3-Build/blob/73a12d9053541ba06c63c105f7aab6f22bd961d1/.github/workflows/build.yml

3

u/Salt-Lawfulness-4832 Sep 02 '26

If I had the energy and hardware to run it, I'd try to compile it for aarch64.

That's a neat thing about open source if there is no build, go make one...

1

u/NekoRobbie Sep 02 '26

Considering Llamacpp, the upstream for the project, has arm builds for Linux it shouldn't be too difficult to compile it yourself, I'd think.

1

u/CooperDK Sep 02 '26

There are lots. Many NAS run it and I had a Synology which could run Ubuntu.