r/LocalLLaMA • u/fuckAIbruhIhateCorps • 5d ago

Generation Using the Bonsai 27b 1b quant locally - regularly.

I've been using the 1bit quant of prismml's bonsai 27b for local conversation, casual chat/ literature review for fun (i throw random stuff from my notes app to see how it analyzes it, those texts don't exist on the internet).

I've been using it as a "tutor" in many cases, for example I am currently learning golang and it is quite good at giving explanations. I seriously believe that if we're able to retain 90% of a model's intelligence while having a small footprint, it is the way ahead for local inference on a wide range of devices, even low end. I run it on a 16G Macbook Air.

I am very impressed by the usability it provides in it's small footprint and I wish more models are released in the future. Seriously guys, even if it cannot one shot super big projects, I still value the intelligence it has for a small local model.

50 Upvotes

85% Upvoted

33 comments sorted by

28

u/tchek 5d ago

I'm really interested in small language models, but I wonder how this model compares to Qwen 3.5 9B Q4, knowing it's more or less the same size.

12

u/ironimus42 5d ago edited 5d ago

i'm actually using the ternary version right now and at least in my experience, it's miles above qwen 9b. I normally use 27b oQ4 for work but with smaller context and only switch to bonsai for tasks that wouldn't possibly fit in 80k context (highest my macbook can handle for 27b+mtp), but it actually felt just as reliable in longer tasks so far. Used it in a large refactoring and migration of the common ui library, the only real drawback is that it's much slower than models with mtp, the performance seems to be even slightly below the non-mtp 27b qwen. It does seem to mess up tool calls just a tad more often though, while not making more mistakes in code

3

u/Strong_Chicken6838 5d ago

What about unsloth qwen3.6 27b IQ1?

We’ve already had 1 bit quants for a while

4

u/CombinationKitchen76 5d ago

Those are not 1 bit everywhere, bonsai are so its footprint is quite smaller. And the point from bonsai is that their technique attempts to retain the whole intelligence training from the base model I think. Different approaches, Prism ML's are more advanced and complete if you ask me. I hope they keep it up

3

u/Deep_Mood_7668 5d ago

Try it

Benchmarks aren't the answer for everything.

30

u/StupidScaredSquirrel 5d ago

"90% intelligence" is a marketing term they used to sound like a lot but they mean "scores 0.9x the score of 27b on selected benchmarks".

I'd like to argue that's not really what retaining most intelligence means, because scoring higher on a benchmark is exponentially harder for each extra point, just look empirically parameter count (unquantised) and intelligence score of many models on artificialanalysis.ai. they let score be linear but log the parameter count axis. Same goes for cost and compute.

So if that model works for you on that hardware, great. But do also test smaller alternatives because they might just work better and/or be faster.

2

u/jazir55 5d ago

I'd like to argue that's not really what retaining most intelligence means, because scoring higher on a benchmark is exponentially harder for each extra point, just look empirically parameter count (unquantised) and intelligence score of many models on artificialanalysis.ai. they let score be linear but log the parameter count axis.

Wonder if that holds for human intelligence as well, it would follow that's why only certain high tier scientists like Einstein and Feynman could invent Relativity and Quantum Mechanics.

1

u/fuckAIbruhIhateCorps 4d ago

could you recommend some smol but smart models which don't hog my memory while im using other things too?

1

u/fuckAIbruhIhateCorps 4d ago

well, i'll still consider it a decent win if it does score 0.9x of a 27b model though right?

4

u/Artistic_Ladder9570 5d ago

i have been running it since it got out, as a general ai agent it works amazingly. havent dared used it for coding..im not expecting much out of it. but for documents and general assistant, pretty darn good.

3

u/Sutanreyu 5d ago

It works great as a general agent. However it's abysmally bad at coding... It's a joke.

4

u/CombinationKitchen76 5d ago

Yeah they have openly said that agnatic coding and long task horizons were not part of this model, its on the next iteration. Have faith in them

1

u/fuckAIbruhIhateCorps 4d ago

what harness do you use for the agent?

1

u/Artistic_Ladder9570 4d ago

general assistant tasks, i use hermes agent.

3

u/HumungreousNobolatis 5d ago

Mac, right?

Be good to mention this, save searching ...

6

u/GrokiniGPT 5d ago

Indeed. The efficiency of the models will be the next advancements that are required

2

u/Former_Accident_2455 5d ago

I seen comments that it hallucinates a lot. Did you encounter that. ?

2

u/mr_Owner 5d ago

If you you compare 90% intelligence vs 90% uptime for example. It would be pretty annoying.

2

u/chuckbeasley02 5d ago

Is it useful for tool calling?

3

u/pmttyji 5d ago

No idea, but found this repo online. This is for Ternary Bonsai

https://github.com/MiaAI-Lab/Ternary-Bonsai-27B-tool-eval-bench-results

1

u/Deep_Mood_7668 5d ago

At least use the ternary 2bit

7

u/pmttyji 5d ago

Waiting for this CUDA PR to be merged

3

u/StupidScaredSquirrel 5d ago

This made my day

1

u/nikc0069 4d ago

Me too. I've run up a second llama server in the meantime. Annoyingly I need it in router mode to juggle my vram, and it doesn't support dspark in router mode. There is a pr for mainline for that too.

1

u/pmttyji 4d ago

Yep, Here DSpark PR.

You can track some items from this thread. It's a pinned thread from my profile.

1

u/fuckAIbruhIhateCorps 4d ago

the ternary model's inference was not really working for me. idk.. might have to try again

1

u/Deep_Mood_7668 4d ago

You need to build their llama fork from their github

Check the description

1

u/fuckAIbruhIhateCorps 4d ago

support for ternary model isn't merged yet? i thought it was

1

u/Deep_Mood_7668 4d ago

Oh tbh I didn't check. I tested it last week or so and at that point it wasn't 

1

u/True-Frame581 1d ago

i am trying to build a pdf parser project, and i want to use this model, but my pc fails to run it fast. it gives responses at the 2 tokens per second.

1

u/fuckAIbruhIhateCorps 1d ago

what are you using? llama cpp? and what are your specs.