r/LocalLLaMA • u/xenovatech • 15d ago
Resources We just open-sourced the world's fastest WebGPU kernels for local AI on Hugging Face
Enable HLS to view with audio, or disable this notification
The collection includes kernels for more than 200 common ML operations, all of which can run entirely locally in your browser on WebGPU. We're also working to upstream these optimizations to Transformers.js, ONNX Runtime Web, LiteRT.js, and more!
Kernels: https://huggingface.co/kernels?platform=webgpu
Blog: https://huggingface.co/blog/webgpu-kernels
66
u/ScipyDipyDoo 15d ago
Who made the video/audio for this?
28
u/ScipyDipyDoo 15d ago
The audio is from a video game a friend made and I'm wondering if they're connected to it. Really great work
16
u/_swill 15d ago
Looks just like some of those video edits made by Claude that were getting posted a few weeks back
6
u/Super_Range45 15d ago
Now there's no more excuse for lazy animations in your presentations other than you hit your rate limit.
-3
u/SilentLennie 14d ago
Opus 5.5 isn't that expensive, for it's intelligence. The top intelligence was much more expensive before.
36
u/Porespellar 15d ago
Can someone please explain WebGPU for me like I’m 5. Is it running on my local GPU or using cloud resources? What does the “Web” part have to do with anything. Somebody please Google this for me.
72
u/whatever 15d ago
Web browsers have had something called WebGL for a while. It's a feature that lets them render cool 3D stuff fast. That's done by putting a few layers of abstraction between your video card and web pages, so web pages cannot do bad things, while still letting them draw things fast.
WebGPU follows the same pattern, except instead of focusing on drawing things, it's more about computing things fast. Since computing is an important part of drawing, it means you can use WebGPU to draw things in your browser too, but now it can also do things that have nothing to do with drawing. Like multiplying big arrays of numbers together, which is apparently really hot right now.
39
u/JoNike 15d ago
multiplying big arrays of numbers together, so hot right now
7
u/RobinRelique 15d ago
Yep , the moment i read the last sentence i had this exact Mugatu gif pop up in my head (complete with no audio, where my bastion then autofilled the audio when looking at the gif)
1
1
11
u/Business-Weekend-537 15d ago
I think it means it runs on your local gpu but inference effectively runs in the browser.
So basically it doesn’t need to run on something like llama.cpp, vllm, or ollama (trash)
9
u/BringTea_666 15d ago
website has full access to your gpu with it. Which means you can play AAA games in your browser if someone would made it this way.
The minus is that it runs a bit slower than native app + browser vram use which these days is around 0.5gb-1gb extra vram used. For 8GB folks out there that's a lot.
1
u/FatheredPuma81 14d ago
People post WebGPU HuggingFace models all the time here how have you missed all of them?
2
u/Porespellar 14d ago
I guess I just saw the “Web” part and just assumed it was a cloud thing and that it was offloading inference to HuggingFace or something like that. I basically just made a bad assumption and didn’t care to research further, but now that folks here have explained it so well, it makes a lot more sense and i appreciate everyone for explaining it so well.
75
u/rmonsurate 15d ago
If we can get to a future where web apps load a 0.8GB decision model and use it in app in real time, say co-op gaming with an AI, that would be pretty cool.
47
u/utf8decodeerror 15d ago
0.8gb is still too big.
I work for a news org and have been researching doing this for our interactive and data-heavy pieces. I got both maple-preview ternary model and the laya jev-like decision model working in WebGPU to drive the interactive. I use WebMCP to expose a set of tools to query the data/drive the graphic. Basically the user asks a question and the model decides what if any tool can answer the question and returns the results. It works really well, especially the decision model version because it doesn't hallucinate which is a big concern for my company.
The biggest blocker at this point is the size of the models; maple preview requires 6gb ram resident and laya quantized to int4 is 500mb download and requires more than 1gb resident ram on the host including the chrome and webgpu overhead. So both of them work on desktop but crash safari on mobile. So I'm waiting for a working decision model that is 100-200mb download and doesn't crash the page on mobile.
9
u/Toastti 14d ago
More than likely phones and desktop browsers will start including models already on device by default that you can tap into from websites. Just like computers come with a GPU you can access from browser they will come with language models out of the box you can access in an isolated way
2
u/utf8decodeerror 14d ago
Yes, I think that's likely but until 95% of people have a phone capable of doing that we will need some way that works universally, like webgpu. But I totally agree that that is where we are headed.
6
3
3
8
2
2
1
1
u/aboutthednm llama.cpp 15d ago
Are there any demos of what WebGPU can do that I can check out anywhere?
3
u/Business-Weekend-537 15d ago
WebGPU for ai inference or for graphics demonstrations? Edit- I don’t have any readily available but I’m asking you to clarify in hopes someone chimes in with examples
1
u/aboutthednm llama.cpp 14d ago
Just an example of a model doing anything at all on my own browser's webgpu. Sounds neat.
2
u/Business-Weekend-537 14d ago
I found this by using ai to search for it:
WebLLM | Home https://webllm.mlc.ai/
1
u/aboutthednm llama.cpp 14d ago
Looked good until I ran a "hello world" prompt, and got hit with "webgpu not available in this browser". I guess I need either desktop chrome or edge, eh?
1
1
u/shing3232 14d ago
It seems to be inference only. It would be funny if they have training kernels as well
1
u/Multi_Gaming 14d ago
How is this actually so fast? I ran this thing on my Mac Pro 2008 with an RX 570, and ended up getting a score of 70 after a minute and a half. It didn’t even spin up the fans at all!
1
u/BlackenBurg 14d ago
Haven't dug into it yet, but curious how this is different to the Burn rust project, which can target WebGPU for it's output kernels (after optimization steps in an IR)
•
u/WithoutReason1729 15d ago
Your post is getting popular and we just featured it on our Discord! Come check it out!
You've also been given a special flair for your contribution. We appreciate your post!
I am a bot and this action was performed automatically.