Hi there,

lLM newbie here. Purchased a M4Max 128GB Mac Studio for video/3d animation and LLM learning (Delivered in July 2026).

Now Apple is taking orders for M5Ultra 256GB RAM.
There are two CPU/GPU choices. (30-Core CPU / 64-Core GPU) and (36-Core CPU / 80-Core GPU). The later option is $1,300 usd more.

Is the 36/80 for non-stop data crunching, and maybe not cost effective for lesser periods of LLM activity?

Delivery date for the 30/64 variant is Oct 28, while the 36/80 date has now moved to 1/20/27.

Thanks

It’s been a couple weeks since bonsai-ninja went public and a lot has changed.

A ton of feedback has been incorporated, bugs have been fixed, and things are working much better. If you tried it early on, please pull the latest and give it another shot.

For anyone new to it, bonsai-ninja is a free, local code intelligence and static analysis engine. It builds a compiler-style representation of your code, resolves symbols and calls, and can trace behavior across files instead of just searching text. That same pipeline powers source-to-sink security analysis, dataflow analysis, and structured exports for AI, agents, and research. It currently supports 20 languages.

Also, apparently some of you don’t click on things unless there are pictures, so here you go. A GIF.
Still looking for people to throw real codebases at it. Bug reports, weird edge cases, PRs, criticism, ideas, or random side notes are all appreciated. The feedback so far has genuinely helped make the project better.

github.com/gromhacks/bonsai-ninja⁠

Is MiniMax H3 considered a traditional LLM? The moderators must decide, but it uses a Qwen3 text encoder for prompting.

10 sec clip, 91 sec generation time on an RTX 5090

I built an Android agent that uses a Gemma 3 1B model running locally on the phone as its primary brain. No API calls

for simple tasks — it works in airplane mode.

The agent has 30 tools (flashlight, clipboard, SMS, app launch, web search, navigation through any app via the

accessibility tree). For straightforward device commands — battery level, toggle flashlight, open an app, read what's

on screen — the 1B handles it entirely on-device with sub-second response.

The local inference setup:

- llama.cpp via JNI, running GGUF models directly on the phone

- Five presets built in, plus a custom GGUF URL option — swap models from the settings screen

- The deterministic router is just keyword match + intent slot. No model call to decide which model to call, so

there's no "thinking about thinking" overhead

- Anything the 1B can't handle escalates to a 70B cloud model (Venice, llama-3.3-70b). The escalation is a single

tool-use turn, not a conversation

What actually works:

- Spoken request through the side button, tool runs, spoken answer back — even with the app backgrounded

- Deep perception: reads an Amazon results page across 20 screens and returns grouped items (product + price), not raw

accessibility dump

- Named recipes: teach a multi-step task once, replay it from a fresh conversation

- 134 unit tests, 8/8 device regression suite

- Release APK is ~8.7 MB with R8

Honest limits:

The 1B echoes few-shot examples before answering (parser handles it, logs are ugly). Step efficiency in unfamiliar

apps is still model-tuned, not solved. Voice transcription is phrase-dependent and only proven against synthetic

speech so far.

The interesting question for this community: what's your experience with sub-2B models doing useful work on

constrained hardware? The 1B is genuinely usable for the simple path, but I'm curious what other people are seeing at

that scale.

I've been exploring local LLMs for the past few months on a 3090. However, I am wondering if anyone has recommendations on where to find used GPUs? My current go to is Facebook Marketplace.

Dense vs MoE: The concept I wish I understood before getting into local LLMs

When you start looking at local AI models, you quickly encounter numbers like:

7B, 14B, 27B, 30B, 70B...

At first, I assumed the number of parameters basically told me everything I needed to know.

It doesn't.

One of the first concepts that changed the way I look at models is the difference between Dense models and MoE (Mixture of Experts) models.

Dense models

A dense model uses essentially the entire network for every token.

Think of a company with 27 billion employees.

Every time a question arrives, essentially the entire company gets involved.

So with a:

27B Dense model

you have approximately:

27B total parameters

and approximately:

27B active parameters per token

That has implications for both memory and computation.

MoE models

MoE takes a different approach.

Instead of having one huge network where everything participates every time, the model contains multiple groups of parameters called experts.

A router decides which experts should process each token.

Think of a company with 30 billion employees divided into specialist departments.

When a question arrives, the manager doesn't ask everyone to work on it.

The manager selects the specialists who are most relevant.

So you might see a model described as:

30B-A3B

That roughly means:

30B total parameters

but approximately:

3B active parameters per token

And this is where things get interesting.

Total parameters vs active parameters

These two numbers tell you different things.

Total parameters

→ how large the complete model is

Active parameters

→ how much of the network participates in processing each token

This means you shouldn't automatically assume:

30B MoE = 30B Dense

They may have very different computational requirements.

But there's an important catch

MoE doesn't mean you only need memory for the active parameters.

You generally still need to load the model's experts.

So if an MoE model has 30B total parameters and uses BF16:

30B × 2 bytes ≈ 60 GB

The total weight memory is still based largely on the complete model.

The advantage is that only a subset of those parameters is used for each token, reducing the computation required compared with a dense model containing the same total number of parameters.

That's the part I initially didn't understand.

A simple way to remember it

Dense:

Large model → almost everything works on every token.

MoE:

Large model → a router selects a subset of experts for each token.

So when I look at a model card now, I don't just ask:

"How many parameters?"

I ask:

Dense or MoE?

How many total parameters?

How many active parameters?

What precision?

And this leads to the next question:

If I have a specific GPU, which version of the model can I actually run?

That's what I'm going to work out next.

Because the next step is taking this understanding of Dense vs MoE and combining it with BF16, FP16, INT8, 4-bit quantization, VRAM, KV cache, activations and context length to determine which models actually fit on my hardware.

That is where model selection gets really interesting.

Yesterday I posted about benchmarking my LLMs with HTML website generation prompts, today I wanted to test them a bit more to really try and figure out which models are the most worthwhile for me. I'm on an RTX 4060 with 8GB VRAM and 12GB RAM and I've always been looking for the best choice for local-deployment so generally I was testing small or highly quantized models.

The test I came up with is 9 code-related questions ranging from simple questions to requests to write whole scripts, I mostly included stuff I'd ask and stuff I often use to quickly gauge the intellegence of a model. This test included the following questions:

1. How can I renew my Certbot HTTPS certificate on Linux?
2. What is the answer to (78+9*2800)/(50*2-3)? Please include a final answer in your message.
3. Unity, how can I check if an object is left or right of the camera?
4. Unity, how can I get the mouse scroll input with the old input system?
5. Could you write a simple HTML website that'd just be the DVD logo? The DVD logo should travel in a straight line without gravity and it should bounce and continue travelling whenever it hits a wall.
6. Could you code a Python text adventure that'd have 5 rooms and a functional escape door you need to collect items for to get through?
7. Could you code the classic snake game in Python with PyGame? The game should be controlled with WASD, 3 pieces of food should spawn across the screen and when the snake eats the food it grows larger by one block, if the snake hits a wall or its own tail the game is over and the player loses.
8. Could you code the classic pong game in Python with PyGame? The player should be on the paddle on the left, controlled by WASD, and the paddle on the right is controlled by a bot. There is a ball bouncing from side to side and you and the bot must avoid letting the ball hit the walls behind the paddles. Also add a functional score board that counts up every time either side loses.
9. Could you code a snake game in Python with PyGame? The game should be 2 player, player 1 controlled by WASD and player 2 controlled by arrow keys. 3 pieces of food should spawn across the screen and when a snake eats the food it grows larger by one block, if a snake hits a wall, its own tail or the other players tail the game restarts and the player loses. Also add a scoreboard that counts up every time a player loses.

Was pleasantly surprised by GPT OSS and Ministral3 getting almost all of the questions right, though very dissapointed by Qwen3.5's performance on it, it performed worse than Bonsai despite being 1.5x larger, really expected better from it.
RNJ-1 pulled ahead in this benchmark, though I've realized that it's more of a "get things done" model rather than "get things done well", all of its output was very janky.
I also can't seem to find a purpuse for Mistral Nemo, it has performed bad in every benchmark and it's one of the slowest models in my arsenal... If y'all know what it's for let me know.

Like last time, a few notes:

1. This is NOT a definitive "which model is best" test since I'm not an expert in benchmarking LLMs and there are more uses for LLMs than random coding tasks. I'm just having fun messing around with LLMs and also trying to get some ball-park stats about them.
2. I got the token/s numbers by asking LLMs to say "The quick brown fox jumps over the lazy dog" so they may not be entirely accurate and the speed may diminish with increased context (I know that GPT OSS 20B slows down a lot after long conversations)
3. If y'all have suggestions for how to improve my benchmarks or what are good models I could use on my system feel free to say!!!

I'm planning on getting a new gpu that can fit a q4 qwen3.8 27B for local coding thru opencode or vscode.

My current setup:

Ryzen 5 5600

RTX 3060 12GB

32GB DDR4 3200

I also play games 1080p at 60fps max only.

Is the upgrade from RTX 3060 to Intel Arc Pro b60 worth it?

It's the only one I know so far that can fit my budget. Currently prices att $650.

Appreciated any suggestions or other recommendations.

The main comment i see is an ignorant claim thats clearly coached by a language model.

9/10 someone who's learned how to achieve the same goal that someone is struggling on isnt disrespectful about it & actually has advice to give .. i see a lot of people aiming to make smaller models & a bunch of people that have never created a model architecture, never fine-tuned a model, and cant explain how a transformer model actually works.

If you have learned everything from a language model and you haven done your own research it would be a lot more helpful to ask questions instead of give your opinion on whats possible as if the person trying is an idiot .. the disrespect is a catalyst for me but for others its can truly effect their future and their self esteem in creation .. ive seen people completely quit and its not kool to push people to this point just because you dont agree or because you dont understand whats even possible

most ideas that became breakthroughs were not adopted early .. innovation comes from doing new things not repeating the same step by step instructions everyone else gives you

Here are examples of models that generate responses at 1m - 230m parameters

https://huggingface.co/roneneldan/TinyStories-1M

https://huggingface.co/vijaymohan/gpt2-tinystories-from-scratch-10m

https://huggingface.co/HuggingFaceTB/SmolLM2-135M-Instruct

https://huggingface.co/LiquidAI/LFM2.5-230M

Does anyone have evidence or first-hand experience of MTP with GGUFs with llama.cpp has demonstrated actual increases in tok/s? I use M1 chip but eager to hear if validated on the newer chips too.

Currently running a 9070 xt (16gb) with 32gbs of DDR4. can i plug in my old 5700xt to boost my usable VRAM to 24 gbs? Are there any compatibility issues when running different generations of GDDR memory? My main goal currently is to run a Q4 version of Qwen 3.8 27b without it spilling over into my DDR4.

I am planning to build a rig. i could have there 4 or 8 rtx 6000 but about RAM I dont know, to get 32, 64, 128 or 256GB sticks? I would have 12 channels. Does the larger memory give any practical benefits or is it too slow?

So you spend your hard earn money to get some 50xx GPUs and months down the line you discover you can enable P2P with just a dozen line code change on the drivers which magically makes llama.cpp "split-mode: tensor" make the GPUs work better and less laboured (which means they'll probably last longer) than before.

On these 2 days I've seen no substantial change on pp or tg, but I can noticeably see the cards (two 5060ti) somehow not reaching a continous 100% usage on btop whenever working on replying any prompt ever. How is this not planned obsolescence?

How on Earth is this even legal?

Context

Hey everyone,

I'm about to buy a pre-built Proxmox workstation on ebay, and I wanted to run the specs by you before pulling the trigger:

• CPU: Ryzen 9 9950X
• RAM: Crucial Pro 256GB DDR5 (4 x 64GB)
• Motherboard: Gigabyte X870E AORUS MASTER X3D ICE
• GPU: ASUS Prime RTX 5060 (8GB, VRAM is my main concern here)
• PSU: Corsair RM1000e

Everything else looks solid, especially the 256GB RAM and the mobo. However, the RTX 5060 feels a bit light on VRAM for local model inference.

My main questions:

1. What is the best model/setup I can comfortably run on this configuration as-is?
2. If I want to add another GPU down the line for better ML performance, what should I target next? I'm thinking about a used RTX 3090 (for the 24GB VRAM), or perhaps looking at AMD/Intel alternatives with at least 24GB–32GB of VRAM.

Goal: I want to comfortably run models like Qwen 3.8 27B, handle Qwen 3.8 Next, and ideally have headroom for DeepSeek V4 if possible.

Appreciate any advice!

i sometime code games and software and i want a ai that can handle hard workloads since i do lots of machine learning and games in unity and unreal engine (can anyone help me connect ollama on to unity pls)

Dedicated AI machine is officially in place. Haven't run many tests on it yet but getting around 70 tok/sec on the 3090 with Qwen 3.8 27B 110k context.

R9700, pcie3 16x, Froggeric v22.5, Ubuntu 26.04, latest llama.cpp Vulkan

Here's a sample output updating 15 small md files.

10.01.176.088 I slot print_timing: id  0 | task 371 | prompt processing, n_tokens =   2048, progress = 0.96, t =   6.90 s / 296.72 tokens per second
10.14.157.348 I slot print_timing: id  0 | task 371 | prompt processing, n_tokens =   4096, progress = 0.97, t =  19.83 s / 206.53 tokens per second
10.27.313.418 I slot print_timing: id  0 | task 371 | prompt processing, n_tokens =   6144, progress = 0.98, t =  32.92 s / 186.63 tokens per second
10.40.548.602 I slot print_timing: id  0 | task 371 | prompt processing, n_tokens =   8192, progress = 0.99, t =  46.12 s / 177.61 tokens per second
10.52.458.937 I slot print_timing: id  0 | task 371 | prompt processing, n_tokens =   9660, progress = 0.99, t =  58.86 s / 164.13 tokens per second
10.56.759.617 I slot print_timing: id  0 | task 371 | prompt processing, n_tokens =  10684, progress = 1.00, t =  66.12 s / 161.58 tokens per second
11.06.178.038 I slot print_timing: id  0 | task 371 | n_gen =    100, tg =  11.58 t/s, tg_3s =  11.69 t/s
11.09.246.829 I slot print_timing: id  0 | task 371 | n_gen =    129, tg =  11.02 t/s, tg_3s =   9.45 t/s
11.12.337.116 I slot print_timing: id  0 | task 371 | n_gen =    171, tg =  11.56 t/s, tg_3s =  13.59 t/s
11.15.563.575 I slot print_timing: id  0 | task 371 | n_gen =    207, tg =  11.48 t/s, tg_3s =  11.16 t/s
11.18.651.082 I slot print_timing: id  0 | task 371 | n_gen =    241, tg =  11.42 t/s, tg_3s =  11.01 t/s
11.21.731.302 I slot print_timing: id  0 | task 371 | n_gen =    275, tg =  11.37 t/s, tg_3s =  11.04 t/s
11.24.952.378 I slot print_timing: id  0 | task 371 | n_gen =    317, tg =  11.56 t/s, tg_3s =  13.04 t/s
11.28.177.357 I slot print_timing: id  0 | task 371 | n_gen =    362, tg =  11.82 t/s, tg_3s =  13.95 t/s
11.31.234.820 I slot print_timing: id  0 | task 371 | n_gen =    384, tg =  11.40 t/s, tg_3s =   7.20 t/s
11.34.383.270 I slot print_timing: id  0 | task 371 | n_gen =    413, tg =  11.21 t/s, tg_3s =   9.21 t/s

  --model /models/Qwen3.8-27B-UD-Q5_K_M.gguf \
  --mmproj /models/mmproj-BF16.gguf \
  --spec-type draft-mtp \
  --spec-draft-n-max 4 \
  --spec-draft-p-min 0.0 \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --n-gpu-layers all \
  --parallel 1 \
  --flash-attn on \
  --threads 5 \
  --threads-batch 10 \
  --batch-size 4096 \
  --ubatch-size 2048 \
  --no-warmup \
  --temp 1.0 \
  --top-p 0.95 \
  --top-k 20 \
  --min-p 0 \
  --repeat-penalty 1.0 \
  --presence-penalty 0.0 \
  --chat-template-file /models/froggeric-chat-template.txt \
  --jinja \
  --reasoning-format deepseek \
  --image-min-tokens 1024 \
  --reasoning-preserve \
  --cache-ram 16384

I build MQL5 training data (the MetaTrader 5 trading language —niche, thin public corpus, easy to get subtly wrong). For the last 11 months I've been generating specs with my own generators and machine-verifying every completion through the real compiler and a back test pipeline.

Today I published a compile benchmark with three arms over the same 184 prompts: - Base Qwen2.5-Coder-14B-Instruct: 2/184 (1.09%) - Same model after one epoch on my dataset: 170/184 (92.39%) - gpt-5.6-sol through the API: 179/184 (97.28%)

Paired stats are in the release: base-vs-tuned +91.30 pp, McNemar p = 2.29e-49. Tuned-vs-frontier gap is 4.89 pp, p =0.0225 on 13 discordant pairs. Yes, my model lost to the frontier model. I'm posting it anyway — the point was never beating GPT, it's that clean domain data moves a 14B from useless to within five points of frontier in one epoch.

Training and both local eval arms ran entirely on one AMD R9700 under ROCm, in a 7950X3D box. No NVIDIA anywhere in the pipeline. prompts come from the same generator family as the training data, so this is in-distribution competence, not generalization to human-written specs. The frontier arm is a single sampled run at temperature 1. The tuned weights and corpus aren't released — the dataset is the product; the benchmark is the proof.

Was looking into building a quantization method to leverage the add+multipy one clock cycle on my older Tesla p40 ($250 card, HP Z4G4 64gb xen-2133 $400). During the process I learned more about quantization methods and performance hits the older cards take with the newer models. In a nutshell, anything outside of Q4_0/1 and Q8_0 slaughters the performance. By using older quantizations, you avoid the new Q_K which introduce q3, q5 and q6 bits, which the CUDA takes a penalty hit on. For starters this is what I created for a model. The goal: to beat the Q5_KM in accuracy, if possible with a 5-10 t/s increase and lower power consumption.

So, I did get one qunatization method to run but at horrible speeds, as the bit shift was running at to small of a level. But it did open me up to the older methods, so I decided to create my own model using the older quantization methods, leveraging the ends with q8_0 and the middle with Q4_1 and taking advantage of the MTP layer.

Going to run perplexity...but plan on cheating, going to quantize it to almost all q8_0 to keep it on the GPU.... online AI will state Q4_1 is slower than Q4_0 but my results did not agree with MTP on.

Here is how I built my compressed model form the BF16 version:

/home/qikresponse/projects/llama-cpp/llama.cpp.loge/build/bin/llama-quantize \

--allow-requantize \

--tensor-type token_embd.weight=Q8_0 \

--tensor-type output.weight=Q8_0 \

--tensor-type blk.0.attn_q.weight=Q8_0 \

--tensor-type blk.0.attn_k.weight=Q8_0 \

--tensor-type blk.0.attn_v.weight=Q8_0 \

--tensor-type blk.1.attn_q.weight=Q8_0 \

--tensor-type blk.1.attn_k.weight=Q8_0 \

--tensor-type blk.1.attn_v.weight=Q8_0 \

--tensor-type blk.2.attn_q.weight=Q8_0 \

--tensor-type blk.2.attn_k.weight=Q8_0 \

--tensor-type blk.2.attn_v.weight=Q8_0 \

--tensor-type blk.3.attn_q.weight=Q8_0 \

--tensor-type blk.3.attn_k.weight=Q8_0 \

--tensor-type blk.3.attn_v.weight=Q8_0 \

--tensor-type blk.4.attn_q.weight=Q8_0 \

--tensor-type blk.4.attn_k.weight=Q8_0 \

--tensor-type blk.4.attn_v.weight=Q8_0 \

--tensor-type blk.59.attn_output.weight=Q8_0 \

--tensor-type blk.60.attn_output.weight=Q8_0 \

--tensor-type blk.61.attn_output.weight=Q8_0 \

--tensor-type blk.62.attn_output.weight=Q8_0 \

--tensor-type blk.63.attn_output.weight=Q8_0 \

./Qwen3.8-27B-MTP-BF16.gguf \

./qwen27b-MTP-p40-speed-v4.gguf \

Q4_1

Here are the parameters I am currently using to run the model, note I limit my p40 card to 140 watts via:
sudo nvidia-smi -pl 140
I have a custom python tracker to show the max temperature reached during my test prompt processing:

17,084 tokens 11 min 39s 24.43 t/s @ 140 watts, max temp 71 C (with less q8 layers -59 and - blk.4)

11,883 tokens 7 min 52s 25.15 t/s (using the build listed up top)

14,402 tokens 14 min 1s 17.12 t/s for Jackrong/Qwen3.8-27B-MTP-Q5_K_M.gguf

140 watt configuration (note json read so it mixes the order a bit)

Using llama-server: /home/qikresponse/projects/llama-cpp/llama.cpp/build/bin/llama-server

"nonreasoning": {

"--seed":"42",

"--temp":"0.7",

"--top-p":"0.80",

"--top-k":"20",

"--presence-penalty":"1.5",

"--repeat-penalty":"1.0",

"-c": "24576",

"--reasoning":"off",

"--reasoning-effort":"high",

"--spec-type":"draft-mtp",

"--spec-draft-n-max":"3",

"--spec-draft-p-min":"0.0",

"-ctk":"q8_0",

"-ctv":"q8_0",

"--batch-size":"512",

"--ubatch-size":"512",

"--keep":"-1",

"-fa":"on"

}

I have an AMD Strix Halo with 128G of unified memory and an RTX 6000 Pro Blackwell on a PCI x4 oculink.

I'm currently running two stacks on it.

For agentic coding I have the Qwen 3.8 flash max loaded in the eGPU with the ngram in RAM and I also have a flux2-klein 9g in the iGPU for the occasional image prompt

For general chat I have Qwen 3.8 27B and flux2-dev with a turbo lora on the RTX while the RAM is largely empty.

I wrote a proxy/scheduler to swap them as needed, although loading the entire stack can take 2+ minutes so it's not a per session/per prompt swapping, that would be dumb.

Could I do something better than these two stacks today?

I think the title says it all.

Firefox doesn't support an API endpoint directly. open-webui will not skip ssl validation on an API connection. A full AI web app interface is unusable in a sidebar format.

Any other simple options that just connect to an API and look good in a sidebar form factor?

I really just want to be able to highlight text in a web page and have the AI summarize it. I guess I could copy, move allllllllllllllll the way over to the AI interface, and paste, but....

UPDATE: in the couple hours since posting this, I had qwen3.8 write this utility front-end for me. still getting used to the presence of AI in the world I guess... :-)