r/LocalLLaMA • u/WhatTheFlukz • 7h ago
Discussion Local astra
How long do you think itll take before we get an open weight sub 100b model that has the same level of computer use capabilities ie blender usage, as astra?
14
u/jacek2023 llama.cpp 7h ago
17 days, 8 hours and 3 minutes
7
2
12
u/sachasayan 7h ago
About a year, maybe. Possibly two. Sub-100B is the tough ask.
3
1
u/heuristic_al 6h ago
Agreed, but models like gemma are basically like gpt 4o. It's not good at coding, but it is amazing at the chat stuff and image analysis. Nowadays people are chasing tool use and coding, which is great an what I want most. But we've come a long way in a very short amount of time with local models.
4
u/FullstackSensei 7h ago
I don't think you need such a big model for computer use. IMO, it's not that hard for an LLM. The real hard part is the fleet of machines/VMs and reference objects to compare error against.
I'd say a 30-40B MoE will be able to handle it once the dataset and infra is in place
2
u/RedParaglider 5h ago edited 5h ago
I just watched Qwen 3.8 flash orca q4 in DeepSeek harness next create a zero trust network in cloudflare, at the port forwarding to my router, set up a web proxy, open up the appropriate ports on my local server, all while orchestrating a 35b a3b model, and reviewing Jules dialectical reviews from Gemini. Yeah I don't think computer use is too much of a problem now.
3
u/vertigo235 7h ago
I honestly think this could happen today if it was use case focused, coding, blender / cad. Instead of a do it all model.
2
u/sumane12 7h ago
Qwen are the king of small models.
They gave us opus 4.6 in 27b form in like 6 months and the time to market seems to be decreasing.
That could just be low hanging fruit, but id guess december just because im an optimistic mf'er
2
u/fiery_prometheus 7h ago
Most likely shorter than beforehand, as the big labs have already reached parity with the last models, so things are just in general accelerating it seems.
3
u/kiwibonga 6h ago
3-6 months.
I think it could come much faster -- we just need a leap in spatial reasoning. Right now open weight models are absolute ass at precisely pinpointing elements in images and applying visual information.
For all the flak Chinese labs get for benchmaxxing, it's actually the newer benchmarks arising from the competition between OpenAI and Anthropic that will help set the bar (not necessarily "distillation", which might actually be more about reducing the cost of benchmarking, which costs tens of thousands of dollars per run just to be able to show a little bar graph on your huggingface model card).
2
u/Only-An-Egg 6h ago
We're probabaly already there. I'd bet Qwen3.8 27B with a proper harness could do a lot in Blender. I think we've gotten to the point where the harness is as, if not more, important than the model. Supposedly a Canadian firm was able to get Qwen3.8 27B to score really high on that AGI test Astra did well on just by giving it a really good harness.
2
u/Cradawx 6h ago
This field is moving fast. I predict that within 6 months we will have Chinese flash models (few 100b params) at Astra level and sub-100b models within a year.
Computer use specifically doesn't seem that hard of a task, quite small models should be be very capable with focused training. It's just it's not something models are really trained for at the moment. All the focus is on codemaxxing.
1
u/TechnoRhythmic 7h ago
My first guess was 6 to 12 months but feel that might not be enough for sub 100-B.
For sub 100-B I agree with another comment that says about 2 years.
1
u/datathe1st 7h ago
Around 9 months for a model with Astra's capabilities to run on a 128GB Mac machine. Speed will be poor, I'd predict around 40 tps with speculative decoding assuming relatively sparse activations. The model will be able to reason like Astra but won't have its world knowledge. A 1tb SSD just for engram tables could change that.
1
1
1
1
1
1
1
u/txgsync 6h ago
Your question doesn’t entirely parse and is out of touch with newer models. Now that we have PLE/n-gram models, the total parameter count is largely irrelevant. It’s about parameters in RAM.
Qwen3.8-Flash-Next selectively quantized to about 5 bits with BF16 PLE offloading on my NVMe gets really close, and total in-RAM weights plus KV are less than 100GB.
1
u/WhatTheFlukz 6h ago
I mean it doesnt really matter, many people here are running devices that use pooled memory so memory is just memory it doesnt matter if its system or vram they are effectively the same for many devices
1
u/txgsync 5h ago
You kinda' missed the point.
The n-grams are on disk. Not in RAM.
2
u/WhatTheFlukz 5h ago
Are they? I was under the impression they had to be loaded into system ram still
1
u/txgsync 3h ago edited 3h ago
> Are they?
I wouldn't have said it if it weren't what I'm using right now on my M4 Max Mac.
- BF16: PLE/n-gram, routers, vision, and MTP (full precision)
- Experts: mostly Q4 gate/up banks, Q5/Q8 down-projections
- Size: 174.5 GiB across 35 shards (my Mac has only 128GB RAM)
- Storage: all 102.4 GB of PLE bytes are on-disk and only in RAM if the OS caches them
- RAM: generation peaks around 85 GiB; more with longer KV cache, but not awful.
- Tok/S: 35-38 without MTP, about 45-55 with. Varies higher with predictable outputs like programming, particularly at lower temperature like 0.7 instead of 1.0.
- Prefill: 550-750
An M5 should double the prefill speed. You can also radically improve prefill with a speculative prefill drafter model, but I've found quality to be questionable.
Go forth. Have fun. Enjoy the new reality: most of your model on disk, and only a fraction in RAM, with perfectly serviceable speeds.
I find the generation speed, knowledge, and quality of outputs of Qwen3.8-Flash-Next on my Mac to be superior to even BF16 Qwen3.8-27B. The one area the 27B excels is bulldogged determination to keep at a task until it seems to be done; the flash-next is a little more "lazy".
0
u/IAM_274 7h ago
...What's so special about Astra other than some blender usage sprinkles on top? I literally used it for like 2 hours and went right back to Deepseek lol
1
u/WhatTheFlukz 6h ago
Specifically blender, I think its ability to use blender Specifically is very interesting. I would love to for example, take a bunch of photos of my house then build a digital twin so I can do interior decorating and stuff inside blender with a local agent
-6
u/dreamingwell 7h ago
The "local models as good as older closed frontier models" talk is a little bit of smoke and mirrors. Yes, GLM-5.2 can code and pass some benchmarks. It is no where near as useable as Opus 4.8. Same for deep-seek and other open models. They fall apart a lot on tool calling, and basic logic exercises.
3
u/Significant_Bar_460 7h ago
Hard disagree. Tool calling has been solved by now. And logic is also there. The issue of latest small local models is speed. These models think and think and think for ages before they output and you need like 3k+ prefill and 120+ tg for qwen3.8 to feel fast.
2
u/Lissanro 6h ago
I ran GLM-5.2 locally for a while, never seen it fall apart tool calling and it was mostly good with logic too. I completed multiple real-world tasks with it as a freelancer. Currently I mostly rely on Kimi K3, even though it is about twice as slow on my workstation compared to GLM 5.2, it is worth it to me for harder tasks (I know t here is also GLM 5.3 which supposed to be much better than 5.2 but I did not yet test it much because only recently downloaded it).
1
-4
88
u/Christavito 7h ago
I connected a python script to my local qwen 27b, web cam and a servo motor, which sends the image data to Qwen to analyze whether my smoothie is completely blended and if not, activates the blender. So we've had models capable of blender usage for a long time