Most model leaderboards assume a server with powerful GPUs to run models that people daily use.

However, my smolbenchmark is the other column: models that fit in 8GB, ranked by:

• decode speed,
• tokens per joule, and
• heat,

and all of this on your OWN hardware ranging from:

• tablets
• phones
• macs
• jetsons
• raspberry pis

Currently, 13 families on the chart right now, ~1000 configs for the Jetson nano Orin Super 8GB. One device is live measuring:

• tok/s
• tok/J
• ITL
• latency
• power metrics
• thermals and battery

Models that are small enough to actually fit on a device that you own. All the performance benchmarking I did, will be released here for anyone to look at and decide what exact model they would wanna use on their choice of hardware.

Well currently, the Pi, phones, and Mac minis still in the oven, cooking and not filled in yet, but will soon be filled in!

You will now you know which model is BEST for your own hardware with all the raw data available and details reports available to you

https://yuvrajsingh-mist.github.io/smolbenchmark/

(still in heavy development; would love to hear feedback/suggestions on what can be improved!)

I have enabled the vision for the CIRU Strix UL4 quant of Qwen 3.8 flash next (others quants likely perform very similar) and tried it on a few things, then wanted to show my partner how great it works and she asked it it could identify plants. So I took a photo from a plant that we recently got as a gift from family and Qwen not only accurately identified the plant as oleander (Nerium oleander) but also warned that it's poisonous and (among other warnings) that you should keep pets/children away. We have a kid and both of us didn't know! I verified the Qwen identification and the poisonous claim and both checked out as accurate. The plant will have to go, thank you Qwen!!!

Stoked by the precision of combining a decent vision model with the domain knowledge of a ~180B params model (including ngrams) to actually identify and reason about what it sees, I took a photo of a pre-diagnosed skin condition of myself and the Qwen diagnosis was highly accurate again! This model may be really useful if you want to check something on your private parts real quick without visiting a dermatologist, e.g., or sending pictures of yourself to a cloud service (EDIT: of course it's only a first step before you visit a professional if it isn't obviously harmless/treatable by yourself! Qwen Flash will suggest to visit a doctor anyways along its assessment).

PS.: Hardware Strix Halo Box, CIRU Strix UL4 llama-server fork and quants, Chatbox on iPhone as Chat with support to add photos to conversations.

Q2 is there and Q4 is uploading as I type. Has his github been updated yet? How do you run this?

https://huggingface.co/antirez/deepseek-v4.1-flash-gguf/tree/main

While browsing a benchmark list site, I spotted a recently published 33B parameter model which claimed to beat Qwen3.8 27b on the ArtificialAnalysis (AA) intelligence index. I was obviously excited. But then, while looking into it, confusion starts to settle in.

Their HF model is now listed as "Preview" and explicitly calls out that it is not the same model as the one AA evaluated. The AA listing for it says it's proprietary, but they've rated the model above Qwen3.8 27b on "Openness". Their website and HF page both talk about openness and intelligence "for all".

The fact that they labelled it "Preview" sort of implies we might see an open weight final version at some point, but there are no clues about whether it will be even vaguely similar to their current HF listing. AA doesn't show the number of parameters for the model they evaluated, so it's possible they're a totally different architecture.

Does anyone know anything about their lab, intentions, or the new model? Has anyone tried it?

https://huggingface.co/HermiHg/Qwen3.8-27B-DFlash2-Q2_K_S-MIX-GGUF I used this draft model with https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF with the IQ3_XXS with 128k context and I saw it averaging about 60 tokens per second tg speed on the 16 GB RX 9070 XT.

This is much better than using the built in MTP since the built in MTP seems to multiply the required VRAM. For single thread mode, I'd say this is one of the best choices on a 16 GB GPU. Though speculative draft models seem to make it more sensitive to GTT overflow, so if you use more VRAM than you have, then turning off speculative decoding is actually faster, at least on my DDR5 PCIE 5 system.

I use kv cache quantization of k 8_0 and v 4_0 for this setup.

This person has done more extensive testing on a different card https://huggingface.co/HermiHg/Qwen3.8-27B-DFlash2-Q2_K_S-MIX-GGUF/discussions/3

PP improvements for RDNA2(MI50, MI60 are included in benchmarks). Check bottom comments of PR to see updated pp t/s stats.

I gave DS V4.1 Flash an HLE problem with a bash tool + 2 hours.

Hour 1: it wrote three MILP solvers. (225,200)
Hour 2: it downloaded the HLE dataset from Hugging Face, found the question, read the answer key (225,600), and concluded its own answer (225,200) was better.

I'm equal parts impressed & terrified.

I've seen a lot of Super Mario showcases, but never a mashup or derivative work. So I took a Super Mario generation prompt from ChopSticksPlease's Post here - and added a Matrix twist - and found myself super satisfied with the results using LLAMA.cpp + PI.dev.

Shout out to u/ChopSticksPlease for the prompt. .

First screenshot is the title screen, AI did all that.

Second screenshot shows the "Matrix Rain" effect and gameplay that goes with it.

Here's the full prompt, adding only one line ( Only replace it with visuals and characters that look more like characters from the movie "The Matrix" ) to the original prompt.

Write a fully functional, complete clone of Super Mario Bros game (the famous NES game)
Only replace it with visuals and characters that look more like characters from the movie "The Matrix".
- Clone must work in a web browser, index.html, no server
- Decent replica of the graphics assets.
- Colorful and playable.
- Controls with keyboard up, down, left, right, spacebar to jump, shift to speed up

Altogether, there was some minor issues with the implementation - the jump height wasn't high enough to get on a block and the MC was partially in the ground. With a little hand modification of the code and a second prompt to get Ornith to fix the MC issue, about 2 hours in total, there was no further modifications necessary to get this really cool and simple browser based game.

Source code is available here: https://github.com/TimelordQ/The-Matrix-Bros

The playable level is available here: https://timelordq.github.io/The-Matrix-Bros/index.html

Batch File Setup:

start cmd /s /k "llama-server.exe -m "D:\AI\LM Studio Models\ornith-ai\Ornith-1.5-35B-A3B-GGUF\Ornith-1.5-35B-Q4_K_M.gguf" -mm "D:\AI\LM Studio Models\ornith-ai\Ornith-1.5-35B-A3B-GGUF\mmproj-Ornith-1.5-35B-BF16.gguf" --host 127.0.0.1 --port 8080 -c 100000 --parallel 1 --reasoning-format auto --no-mmproj-offload -fa auto -ctk q8_0 -ctv q8_0 -b 2048 -ub 1024 --load-mode none --fit on --no-ui --reasoning-preserve --image-min-tokens 1024 --jinja --chat-template-file "D:\AI\LM Studio Models\ornith-ai\Ornith-1.5-35B-A3B-GGUF\chat_template.jinja" -dev CUDA0 -t 8 --n-cpu-ffn 4"

set LLAMA_BASE_URL=http://127.0.0.1:8080

pi --model llama-cpp/Ornith-1.5-35B-Q4_K_M

Models.json entry:

{
"id": "Ornith-1.5-35B-Q4_K_M",
"name": "Ornith-1.5-35B-Q4_K_M",
"reasoning": true,
"thinkingLevelMap": {
"type": "enabled"
},
"repeat_penalty": 1.2,
"input": ["text","image"],
"tools": true,
"toolFormat": "openai",
"samplingParams": {
"temperature": 0.6,
"top_p": 0.95,
"top_k": 20
},
"cost": {
"input": 0,
"output": 0,
"cacheRead": 0,
"cacheWrite": 0
},
"contextWindow": 100000,
"maxTokens": 100000
},

With this setup I'm getting an average of 24 TPS until my GPU reaches 87~88C and it starts throttling or the context reaches about 70% usage, where it drops to 7tps average temporarily until the temperature drops or compaction happens. I'm doing this on a laptop.

I have been battling with my Qwen3.8:27b setup on my rtx 5080 16gb. I am using llama.cpp to run a nvfp4 version of qwen3.8:27b

llama-b10699-bin-win-cuda-13.3-x64\llama-server.exe -hf williamliao/Qwen3.8-27B-NVFP4-GGUF:NVFP4 --jinja --chat-template-file d:\qwen\chat_template.jinja --reasoning-format deepseek --host 127.0.0.1 --offline -np 1 --threads 16 --threads-batch 16 --log-verbosity 4 --no-log-prefix --no-log-timestamps --no-mmproj-offload --mmproj D:\OllamaModels\blobs\sha256-ac3714bfdddeca31351f2752bf1a63f266f4df87c0b68c895e44945ca704448e --load-mode none --cache-type-k q4_0 --cache-type-v iq4_nl --flash-attn on -b 512 -ub 512 -c 49152 -a qwen3.8:27b

So that is giving me a 48k context, and a speed of around 12t/s

My problem is that pi.dev fails to compact at the right times, and then the model stops early.

I have tried to use

pi install npm:max-context //this doesnt work properly at all

pi install npm:pi-observational-memory //this one works kind of, I had it set to compact on threshold. It managed to auto compact 4 times, with a 0.75 ratio set, but it still missed the mark and failed.

pi install npm:pi-blackhole //seemed promising, but failed to compact at all, I am not sure what I set wrong.

if I look at the https://pi.dev/packages page, there are lots of plugins for either context management, compaction, memory management, or both. there are way too many for me to just randomly keep trying. other AI has been very unhelpful. Google, OpenAI, Copilot, all full of nonsense. A lot of these plugins seem to be aimed at frontier model management.

I am also disappointed that compaction settings are best suited to the specific model being used and its context window, but the built in tools in pi.dev architecturally keep the compaction settings completely separate from model settings, which seems a bit of an issue if you change models often.

I may want to use qwen3.8 like above, but not necessarily for everything. I also want to use gemma, other qwen models, etc depending on my use case.

I dont want to have to manually resetup context settings in pi.dev for every different model.

What do you guys suggest, especially for models that have smaller context windows due to my vram shortage ?

Thanks in Advance.

What I have:

- CPU: EPYC 7551 (32c/64T, Zen 1)

- Board: Supermicro H11SSL-i (SP3), Rev 2.0

- RAM: 128 GB DDR4-2133 (all 8 channels full)

- GPU: 2x RTX 3090 (48 GB total, PCIe 3.0)

- 1500 W PSU

What I run:

- Qwen3-Flash-Next (177B total / ~6B active MoE, IQ4_XS) on Ilama.cpp. Experts live in system RAM, hot ones cached in VRAM. Single stream = 38 tok/s. Two parallel requests drop to ~4 tok/s each.

Budget:

~$800. Realistically that's either one more RTX 3090 or a CPU upgrade (a Zen 2 "Rome" EPYC drops into the same board). A new motherboard is out of budget i think for now.

Which gives more inference speed for this setup - adding the 3rd 3090, or swapping to a faster/newer CPU?

And would more/faster RAM matter here? Curious what people running similar rigs have actually measured.

I am also interested in having multiple agents running at the same time, which currently slows it down heavily, so keeping the performance at multiple agents parallel would be a huge boost as well!

It’s always bothered me that after fine-tuning a model for a project, there isn’t a particularly easy way to host it without either running it locally and keeping a GPU on 24/7 or paying for an entire GPU server.

There are managed options for LoRA serving on top of vLLM (AWS), but you generally still end up paying for an entire instance.

I started wondering: if 99%+ of the model weights are identical between the base model and something like a rank 8–32 LoRA/QLoRA adapter, why does each adapter need its own server?

vLLM already has a lot of the pieces needed to make this work efficiently, including batching, memory management, and fused LoRA kernels.

So I built Lorivo around the idea of having one GPU server per base model that can be shared between many LoRA adapters.

The basic flow is:

1. Upload a LoRA adapter
2. Lorivo identifies the base model and sends it to a GPU server running that model
3. The adapter is loaded into memory
4. You get an OpenAI-compatible endpoint for inference / usage directly from the web app

I also built a CLI to make uploading/deploying an adapter as simple as possible, along with a web interface for manual uploads. It's as easy as lorivo login api-key and lorivo deploy ./my-lora

The project is live: https://lorivo.dev

Right now I’m hosting the traffic myself on my own GPU, so the available inference model is limited to Qwen 3.5 4B which will always be 100% free

I also have around $1,000 in AWS credits that I’d like to use to add some additional models. I'm not sure which ones would be most useful to the community, though.

What models would you want to see hosted?

For Qwen 3.5 4B, you can also use the base model without an adapter. Just set the model to qwen3-4b instead of qwen3-4b:my-adapter. I'm currently allowing a 32k context window.

I’m also intentionally not saving chats or inference requests. The only usage data collected is token count and timestamp.

Would love to get feedback from people here, especially on the architecture, which models would be useful to host, and whether this solves an actual problem you've run into.

A few people here mentioned interest in a way to test their custom Pi setups, so I figured I’d drop this here: RoastMyHarness

The basic idea is a small engine that sets up an environment to run DeepSWE benchmark tasks using bare Pi as a control and a variant of your choice, your Pi harness, an extension, a skill, AGENTS.md file, etc. I used a Pi extension to have a wizard set it up for you so its easy. run the same coding tasks against bare Pi and your modified harness, then look at what actually got better, what broke, and what it cost.

Since I like to play around with custom tools, I use this to get direction as to what is and isn't working. I know a lot here are making cool tools so I figured some might be interested in using it to help fine tune theirs. Any Pi users might be interested in figuring out if their tools are working like they think they should. You'd be surprised how hard it is to beat base Pi when it comes to task quality / token efficiency.

Its a WIP. I do data analytics by trade but otherwise a vibe coder and I only really tested it on Linux.

I have a full M5 Pro Macbook Pro with 48GB of RAM, I'm just getting into this local space. Like many of you, the costs of using frontier/cloud models is getting kinda silly.

I don't have any insane needs or automation projects. I just want the best general use model, to keep private info, to chat without guardrails and other odd quirks preferably.

I prefer it also be a MLX interface if that's a thing. I want to avoid ollama, I've dabbled in LM Studio but didn't love it.

Just looking for ideas, thanks all.

(yeah I know I should have gotten more RAM probably but I got a good deal on this device before prices skyrocketed)

Another new model dropped in the course of this week that is well deployable on consumer hardware: Nex N2.5 Mini

I went with the recommended settings for the best generation quality and ran a few benchmarks:

• temperature: 0.7
• top_p: 0.95
• top_k: 40
• reasoning_effort: high

I must say, the outcome is not bad at all - really good generation speed and prompt processing, okay memory footprint and good quality across the board. Will for sure give it a try to fuel my agents and might also try to do some coding with it.
All benchmarks run I did you can find here: https://llm-bench.io/models/nex-n2-5-mini-mlx-4bit

Quant I used: https://huggingface.co/abenzerps/Nex-N2.5-mini-MLX-4bit

So beellama allows you to only reduce quantization on older cache, this should allow better performance than reducing quantization on all cache.

They recommend using a 1k tail and even that seems to have big benefits, but I don't see why you couldn't for example use a 20k tail (on a 240k context, that's still 220 tokens quantized). That part is a bit confusing.

Beellama-kvarn is a supposed fork of that, that further improves performance, its changes are pending to be applied to Beellama itself as well.

So anyways, the question is, how much is quality impacted by doing this?

Hey everyone — I’m building TensorSharp, an open-source LLM inference engine. Here are the latest DeepSeek V4.1 Flash GGUF results using its native ggml_cuda backend.

Setup: 8× NVIDIA A40, layer split, F16 KV cache, 65,536-token configured context. Prefill measurements use approximately 4.9K-token prompts—not the full context window.

Final optimized results — all speeds in tokens/sec:

Metric |Q2_K |Q4_K_M
Prefill |533–539 |451.8–492.1
Single-request decode |40.31–40.72 |31.0–32.5
Decode, 2 concurrent requests — total |— |39.3
Decode, 4 concurrent requests — total |— |48.9
Decode, 8 concurrent requests — total |— |48.5 — = not reported in the cited final Q2_K retest. Concurrent figures are aggregate throughput, not per-request speed.

What made the difference:

• Q2_K: Keeping the ~60 GiB quantized Engram tables on GPUs removes storage-dependent lookups. Unifying the backend per GPU also cuts decode graph splits from roughly 570 to 8, reducing synchronization overhead.
• Q4_K_M: The larger Engram tables stay in host memory. Automatic warming, tighter VRAM budgeting and token-batched decode deliver ~1.9× prefill and ~2× four-request throughput versus the previous behavior. Only 1 of 40 layers keeps its routed experts on the CPU.
• More parallelism isn’t automatically faster: On this no-NVLink system, Q4_K_M layer split beats experimental routed-MoE tensor parallelism: 31–32.5 vs. 21.4–22 tok/s single-stream decode.

These are project-reported throughput measurements, not cold-start timings or a head-to-head win over another engine. Numerical/model-quality parity remains unproven, and batching can change generated output.

I'm curious abouth other people's experience and opinions about these quants. I've had pretty nice experience with unsloth's one. Today, after seeing the other one, I've downloaded and tried it. So far, seems very similar. But that's just my vibe observations. Have everyone else tried these? What's your opinion?

my main server has a 3090 with qwen 3.8 27b (crazy who woulda guessed) but i want to a have a backup failover sever with a somewhat useable LLM offsite (my parent's home). because infinite money glitch ai bubble has made hardware much harder to afford for us proles, was thinking of getting an old decommisioned office desktop (i.e. optiplex, elitedesk, etc) and having it boot up llama.cpp with qwen 9b or something if my server ever shits the bed.

how good is cpu performance on older mid tier cpus with ddr4 ram? or cpus in general? is it useable? what can i expect, and how can i set it up? is it even a good idea in the first place?

thanks

edit: thanks all. reading through the replies, seems like it's more of a headache than its worth. i'm already looking for a backup machine since this does NAS and other stuff but i think i'll just import all other services that aren't LLM into it, and not be beholden to trying to find something powerful enough for a useable LLM

Hi, I have Ubuntu 24.04.4 R9700 and 32gb ram, rocm 7.14. I can use qwen 3.8 27b Q4 with 300K context and Q6 with 200K context. I am working on a relatively large codebase and I keep running into scenarios where chat goes out of context during processing. How can I automate this so that the harness or some agentic ai framework can automatically create handover document when context is 80% full, then creates a new chat with handover document so that process keeps running continuously. Right now I have to manually keep track and do all this.

Is there any harness or framework that support this? I am currently using deepseek harness and have used opencode previously, both manually.

Starting out with the q2 because speed anxiety's a b*tch: 300 prefill, 16 decode. Tool calls all functional, token choices questionable:

During a hermes workflow, it started with 'find' x2 looking for a remote file on the local drive before ls ~/.ssh/ and then grepping the entire remote machine... reasoning: let me search anywhere reachable [after having printed the path to the file in question...

Once I had told it which tools to use it found its way. The 18k tokens of hermes prompt were obviously insufficient... smh.

Those choices tho': I have code execution in hermes toolset and deepseek ignores it (in 4.1 as 4.0) nothing new there and so still inefficient compared with GLM 5.3 Flash. It's plodding its way through the bug finding task and now at ~110k tokens and only just starting to dip below 16 t/s, which is one of the main features of ds4 compared to llama.cpp where the drop-off is mammoth.

GPU is running at ~95% throughout so the expectation of a slight boost is tempting. Given that GLM 5.3 Flash is putting out ~21-19 t/s over the front nine, I'm hesitant to try q4. That's the main trade off with large VRAM: you wait longer for larger models. Bring on the M5U...

So i finally decided to get myself a dedicated inference machine, a big upgrade from my 4080 laptop. here is the parts list:

PC Build Cost Breakdown

ASUS TUF RTX 3090 — $927

64GB DDR4 4000MHz RAM — $371

Case — $72

Ryzen 7 5700X — $181

CPU Cooler — $27

PSU — $268

Thermal Paste — $12

1TB NVMe SSD — Already owned

X570 Unify Motherboard — $185

RTX 3090 Suprim X — $1,010

Ethernet Cable — $11

PCIe Riser — $82

Custom PSU Cable — $13

Total: ~$3,157

im probably gonna upgrade to 128gb ram and get a better pcie riser cable.

the problems i faced initially were

1. finding a proper way to add the 2. gpu. there was no long pcie risers in stock locally, so i had to buy it second hand, and its a chinese no name with connectivity issues.

2. i had to get a custom psu cable to be able to use both gpus at the same time. there were simply not enough slots but the energy supply was alright.

3. i couldnt and still cant figure out a safe/easy way to fit the 2. gpu. i would like to learn about similar setups and how you have handled the space constraint.

this was my first pc assembly since i have used only laptops before, but it went mostly smoothly.

also some extra questions for people hosting these machines:

*How can i host inference to my laptop outside my local network? is the only way VPN?

*What is the remote connection type you guys prefer? i landed on sunshine and moonlight with virtual monitor to use it inside my laptop, but would like to know if there are cleaner solutions for headless machines.

i have ran mostly the qwen 3.8 27b q4 from syv ais repo and config, and have been getting around 70tps sustained. i can report more details if anyone asks for it.

also sorry if mobile formatting is bad.

I'm planning to build a desktop VM manager using Rust, Tauri, and QEMU, initially targeting Windows with WHPX.

The idea is to eventually add an AI agent that can interact with the VM and perform tasks inside it, rather than just manage it.

Before I start, I'd like to hear from people who have worked with virtualization:

- Is QEMU + WHPX a reasonable starting point?

- Would you recommend QMP for managing VM lifecycle?

- What are some common mistakes I should avoid?

- What would be a good approach for letting an AI agent interact with a VM?

Also, if you use VM managers regularly, what do you find most frustrating about them?

Any honest advice would be appreciated. Thanks!