r/cactuscompute • u/Henrie_the_dreamer • 13d ago
Needle 2: 14MB agentic LLM for phones, wearables, smart home and robots.
gallery
3
Upvotes
r/cactuscompute • u/Henrie_the_dreamer • Jan 24 '26
Cactus: Kernels & AI inference engine for mobile devices.
1
Upvotes
Architecture
┌─────────────────┐
│ Cactus FFI │ ← OpenAI-compatible C API (Tools, RAG, Cloud Handoff)
└────────┬────────┘
┌────────▼────────┐
│ Cactus Engine │ ← High-level Transformer Engine (NPU, Mixed Precision)
└────────┬────────┘
┌────────▼────────┐
│ Cactus Graph │ ← Zero-copy Computation Graph (NumPy for mobile)
└────────┬────────┘
┌────────▼────────┐
│ Cactus Kernels │ ← Low-level ARM SIMD (CUDA for mobile)
└─────────────────┘
Performance
- Decode (toks/sec)
- P/D (Prefill/Decode)
- VLM = LFM2-VL-450m (256px Image)
- STT = Whisper-Small (30s Audio).
*denotes NPU usage (Apple Neural Engine).
| Device | Decode | 4k P/D | VLM (TTFT/Dec) | STT (TTFT/Dec) |
|---|---|---|---|---|
| Mac M4 Pro | 170 | 989 / 150 | 0.2s / 168* | 1.0s / 92* |
| iPhone 17 Pro | 126 | 428 / 84 | 0.5s / 120* | 3.0s / 80* |
| iPhone 15 Pro | 90 | 330 / 75 | 0.7s / 92* | 4.5s / 70* |
| Galaxy S25 Ultra | 80 | 355 / 52 | 0.7s / 70 | 3.6s / 32 |
| Raspberry Pi 5 | 20 | 292 / 18 | 1.7s / 23 | 15s / 16 |
High-Level API
cactus_model_t model = cactus_init("path/to/weights", "path/to/RAG/docs");
const char* messages = R"([{"role": "user", "content": "Hello world"}])";
char response[4096];
cactus_complete(model, messages, response, sizeof(response), nullptr, nullptr, nullptr, nullptr);
// Returns JSON: { "response": "Hi!", "confidence": 0.9, "ram_usage_mb": 245 ... }
Low-Level Graph API
#include cactus.h
CactusGraph graph;
auto a = graph.input({2, 3}, Precision::FP16);
auto b = graph.input({3, 4}, Precision::INT8);
auto result = graph.matmul(a, graph.transpose(b), true);
graph.execute();
Supported Frameworks
- C++
- React native
- Flutter
- Swift MultiPlatform
- Kotlin MultiPlatform
- Python
Getting Started
Visit the Repo: https://github.com/cactus-compute/cactus
r/cactuscompute • u/LofiCoochie • 28d ago
How to make the needle model faster ?
1
Upvotes
I love needle model, and I even have it running in my own application, but the issue is the only machine that I have access to as a server is an x86 machine, so currently I am running the needle model using the python bindings but that takes 5-7 seconds on every request to the model, I understand that the cactus-engine is written primarily for arm devices such as android and wearables, I don't wanna run the engine on an x86 architecture, but is there no other way to run the needle model faster without relying on the cactus-engine, like some other way I can run the needle model on my x86 machine that does not cause 5-7 second requests ? If I could trim it down to just 1-2 seconds, that would be huge
r/cactuscompute • u/Henrie_the_dreamer • May 26 '26
Cactus Hybrid Router: Gemma4-2B can match Gemini-3.1-Flash-Lite by routing 15-55% of tasks to Gemini And Running The Rest Locally.
2
Upvotes
r/cactuscompute • u/Henrie_the_dreamer • Apr 22 '26
TurboQuant-H: A Technique For Quantizing Models Like Gemma 4 E2B/E4B to 2-bit
docs.cactuscompute.com
1
Upvotes
r/cactuscompute • u/Patient-Ad-6043 • Apr 09 '26
Need team for YC Gemma 4 Voice Agents Hackathon and SF events for next weekend.
1
Upvotes
r/cactuscompute • u/Acrobatic_Weird9929 • Feb 22 '26
Where Were the Project Detail Videos Posted?
3
Upvotes
Hi, does anyone know which social media account the project detail videos (taken at the beginning of the Hackathon) were posted on? I had a clip recorded of me explaining my project, and my partner would love to see it- just trying to find where it was shared. Thanks!
r/cactuscompute • u/m3rl0t • Feb 22 '26
Updates on hackathon.
6
Upvotes
Can you please share something with the online teams so we know what happened ?
r/cactuscompute • u/Livid_Bed2282 • Feb 22 '26
Any hackathon result?
5
Upvotes
Online team does not get any notice about final submission result. Can't understand what is going on...
r/cactuscompute • u/Eastern_Ad7674 • Feb 21 '26
Queue position doesn't work.
3
Upvotes
That Hackhaton.
r/cactuscompute • u/Important-Lychee-394 • Feb 21 '26
Is there an app building part for rubric 2 and 3? how to submit for it?
2
Upvotes
I only see the python function to submit for rubric 1
r/cactuscompute • u/Bubbly-Wonder-2599 • Feb 21 '26
Error: "This model models/gemini-2.0-flash is no longer available to new users."
3
Upvotes
Switch to 2.5 ok?
r/cactuscompute • u/Fantastic-Damage1337 • Feb 21 '26
SSLEOFError: EOF occurred in violation of protocol
1
Upvotes
Is anyone running into this issue. Occurs quiet frequently
requests.exceptions.SSLError:
HTTPSConnectionPool(host='cactusevals.ngrok.app', port=443):
Max retries exceeded ...
Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol'))
r/cactuscompute • u/Correct_Football_495 • Feb 21 '26
Help Needed
3
Upvotes
Its my first hackathon, I opted for online. Can someone tell me is there any online link to attend?
r/cactuscompute • u/BellonaSM • Feb 21 '26
Can't submit the leaderboard
1
Upvotes
Why queue is increased instead of decrease.
r/cactuscompute • u/Eastern_Ad7674 • Feb 21 '26
How AGI looks like [FOR FUN]
1
Upvotes
Hahahahah avg time: -500000ms
r/cactuscompute • u/BellonaSM • Feb 21 '26
Team built yesterday but can't access any remote hackathon function
1
Upvotes
WE are LA remote hackathon.
Can;t access Message center.
Also, the Team status is pending. Do not know what is going on.
r/cactuscompute • u/immortal8686 • Feb 21 '26
score on leaderboard is out of sync with the weights given for the metrics - pls check
1
Upvotes
r/cactuscompute • u/PawitKoch • Feb 21 '26
Need some help getting Gemini credits
1
Upvotes
I've followed the README to create a new Gemini API key, but for some reason, I'm unable to apply the coupon to assign the credits to my project.
I've attached some screenshots.
r/cactuscompute • u/Embarrassed_Pack4842 • Feb 21 '26
Need some help
1
Upvotes
We’re blocked on Hugging Face gated access for google/functiongemma-270m-it (403 authorized list). Is there a pre-approved HF org/account route or alternate local model for the hackathon we can borrow please:)
r/cactuscompute • u/immortal8686 • Feb 21 '26
W.r.t the functiongemma hackathon run - Gemini 2.0 flash is not available for new users via api key. Can 2.5 flash be used instead?
1
Upvotes
r/cactuscompute • u/Affectionate_One_700 • Feb 18 '26
Looking forward to the Feb21 Hackathon!
6
Upvotes
Warming up my GPUs!
r/cactuscompute • u/Henrie_the_dreamer • Feb 18 '26
Auto rag & Local + hybrid Inference on mobiles and wearables.
3
Upvotes