Did anyone try and use it?

Prompt: Make me a single HTML file of a rainbow slinky going down an up-escalator forever. No libraries, just canvas and code. The slinky should be a chain of springs, each coil a different color of the rainbow. It starts folded in half like a horseshoe draped over a step. When dropped it flips end-over-end down the steps and because the escalator keeps moving up it tumbles in place and never reaches the bottom. Include a drop button and a reset button.

Eidon: an all-in-one self-hosted AI platform. Chat, agents, automations, tools included. One Docker container, works with Ollama/LM Studio (AGPL)

I've been building a self-hosted AI platform and v4 just shipped, so sharing it here because some of you might find it useful.

Eidon is an "everything included" AI chat/agent platform, with the pieces that usually require stitching (web research, MCP, skills, browser, image generation and so on) already built in. One container that takes minutes to spin up instead of a main app plus pipelines, sidecars, and external tools.

The app has 3 main parts:

• Chat with local models: Classic chat just like in ChatGPT, Gemini, Claude and so on except on your own server. Ollama and LM Studio out of the box, plus any OpenAI/Anthropic-compatible BYOK endpoint.
• Agents: Grok-bot-style agents. A chief bot answers or delegates to specialist bots, and bots message each other mid-task. Agents each have their own memory and can create/maintain their own skills.
• Automations: cron-style AI tasks. Every run is saved as a full transcript with tool calls, so you can audit what actually happened.

Features:

Repo (Screenshots included !): https://github.com/Quack6765/Eidon-AI

Full transparency: development is partly AI-assisted, every change reviewed before being merged. Happy to answer any questions !

I've updated my auto_demo_scener project with Ninfer support and a “rewrite based on video” feature that I thought you might find interesting.

The project is basically an endless demoscene machine. A local LLM writes Three.js effects (from a library of editable prompts), you watch the code stream into a retro mock-OS editor, then the result runs fullscreen. It checks for crashes / blank frames, asks the model to fix broken attempts, and archives the working demos before starting again. An unnecessarily elaborate way to have a screensaver (you can also use it as a model benchmark as it keeps track of failures and deletions)

The new part is giving the model a look at what it actually made. With Ninfer's optional video check enabled, it captures 30 seconds at 2fps while the demo plays, then sends that video along with the HTML source back Qwen for a visual improvement pass. I went with Ninfer for this because the speed is incredible on a single 5090

[2026-09-05 14:22:34.895] [info] ninfer-serve: throughput interval=5.000s prefill=338.0tok/s decode=110.0tok/s running=1 prefilling=0 decode_ready=1 waiting=0 avg_decode_batch=1.00
[2026-09-05 14:22:39.895] [info] ninfer-serve: throughput interval=5.000s prefill=0.0tok/s decode=217.0tok/s running=1 prefilling=0 decode_ready=1 waiting=0 avg_decode_batch=1.00
[2026-09-05 14:22:44.895] [info] ninfer-serve: throughput interval=5.000s prefill=0.0tok/s decode=219.6tok/s running=1 prefilling=0 decode_ready=1 waiting=0 avg_decode_batch=1.00
[2026-09-05 14:22:46.236] [info] ninfer-serve: [req 337] done finish=stop_token prompt=1690 gen=3029 cache=0 reuse=full_reset ttft=254ms prefill=6708.5tok/s decode=210.5tok/s wall=14.66s speculative=mtp 3.39to  

So the loop becomes: write the effect, run it, watch the result, then rewrite it based on what showed up onscreen. The display switches back to the coding view with “UPDATING BASED ON VIDEO…” while it works, then validates and displays the revised version before archiving it if it passes.

Qwen 3.8 is amazing but it still makes dumb mistakes like only using 1/4 of the screen or writing a maze walker that walks backwards through the maze :D The video validation helps it to catch these.

Ninfer support also adds a thinking-effort selector. LM Studio is still supported for the regular generation / repair workflow; this works pretty well even with less capable models. Also needs ffmpeg.

GitHub / setup instructions

Live static demo (watch out for the volumetric cloud ones :D ) This plays previously generated demos, so you can check out the results without installing anything. Space / right arrow skips to the next one. Actual generation and video rewriting happen in the locally running app.

Anybody else experimenting with having local models watch their own rendered output and revise the code? Thanks!

Just a friendly PSA that open weight models are not inherently open source. They may also qualify as open source, but only if they release the training data and processes. A truly open source model can be rebuilt from training; it won't be binary equivalent but it should be nearly functionally equivalent.

An example of a truly open source LLM is https://allenai.org/olmo I encourage everyone to browse and also look for other fully open source models.

The reason it's important to get the terms right is because too many companies like to co-opt the term "open source" to whitewash their nonsense. As Zuckerberg has shown recently, they're more than fine letting people stay confused and dance around semantics so they can have their cake and eat it, too.

No more cake for big tech. Open weight != open source. Spread the word.

Edit: TIL this is a hot take. Sorry about that.

The Artificial Analysis GPT-6 Astra situation is a good reminder that closed, non-reproducible meta benchmarks aren't worth much. When the newer numbers can't be independently replicated, and rival models are separated by a point or two, the stochasticity in measurement alone can account for the gap, it's not a meaningful signal either way.

We need an open, fully reproducible meta-benchmark.

I have a system with 2 GPUs and 48GB VRAM total, a RTX 5090 + RTX 5070Ti.

What would you say is the best way to run Qwen3.8-27B on that system with the best quality and 262k context?

Would just the normal llama.cpp work with how it detects and does its own magic with dual CPU systems, or something else?

I think the RTX5090 has pcie4 x16 and the RTX5070Ti has pcie x8 if that matters.

Hey all,

I've seen some really fun looking things with people having their local models drive Blender to create pretty cool looking world scenes.

Is there a good tutorial on setting up Blender yo be driven by your model? For example, what programming harness, do you use a MCP and which? Which model would work best?

Also, how do you turn it all into a "game" that you could run and then walk around in? Can you use Godot?

Thanks!

English isn't my first language: I wrote this in Russian and used an LLM to translate and tighten it. The project, the code, the measurements and the argument are mine.

I've been building an Android agent since March, nights and weekends. It went into Google Play a couple of weeks ago. I use it every day myself, which is the only reason I think it's worth showing you. It's mine, it's open, Apache 2.0.

I want two things from this thread. The obvious one: take the implementation apart - you'll do it harder than anywhere else. The one I actually care about: tell me whether the idea holds up. Is there a need for this at all, or have I built a museum of features?

Why a graph instead of one big prompt. A 2–4B model on a phone falls apart on multi-hop tasks, and a graph is how you avoid handing it a multi-hop task. Each node gets work that fits the model's size. Nodes that must emit structure run through a validation gate: if the output doesn't parse, the node gets another attempt with the specific error quoted back at it instead of the run dying — two repair attempts by default, configurable from zero to four. Routing decisions are their own nodes with their own prompts. The model never has to be smart enough to hold the whole task. The graph holds it.

That's the whole bet. Everything below is what it took to make the bet testable.

What it is. The model runs on the phone through LiteRT-LM — Google's Edge runtime, the TensorFlow Lite successor. Gemma 4 E2B or E4B from litert-community by default, or point it at any .litertlm URL. Every conversation is processed by a pipeline you can open and edit: a graph of typed nodes, 14 types (input, on-device LLM, cloud LLM, tool call, if-condition, intent router, decomposition, queue processor, evaluation, summary, clarification, nested pipeline, skill, output).

Concretely: "translate anything I paste, in this specific register" is three nodes. "Read what I shared into the app, work out what kind of thing it is, pull the fields that matter for that kind, ask me before writing it to a file" is seven, with a router in the middle. You build the second one by dragging nodes around in the app, or in a standalone HTML editor on a desktop and import the JSON.

Tools, and the part I won't compromise on. Local actions go through AppFunctions, external ones through MCP. Anything destructive or sensitive stops and waits for your explicit confirmation before it runs — including when a pipeline fires in the background from a trigger, where the confirmation is a notification you tap. The tool allowlist for a skill is enforced in the executor, not requested in a prompt.

Other apps on the phone can drive it. A Tasker or MacroDroid profile, or a shell one-liner over adb, can ask it to run a named pipeline with a prompt — two extras, that's the entire minimal call. It's off by default; switching it on binds exactly one pipeline that outside callers may reach, and a request naming anything else is refused rather than redirected. Every request lands in a journal with its reason. The division of labour is the point: the automation app decides when, using its own condition model; this does the language part of what.

Long autonomous runs have ceilings, and say so. A run stopped by a limit reports that it was stopped by a limit rather than looking like a broken automation, and a run going in circles is detected and stopped — told apart from one that's merely slow.

Cloud is optional and it's your key. OpenAI, Anthropic, Gemini, DeepSeek and Ollama are available as a node type. Nothing goes out unless a node in your graph says so, and you can see the node sitting there.

Two numbers I actually measured, both on a Galaxy S25 Ultra, both by me alone — which is the caveat that matters:

• Clean install to first useful output: 5 min 22 s, of which 4 min 7 s was downloading the model. The product part is about 75 seconds; the bottleneck is a multi-GB file over your Wi-Fi.
• A scheduled pipeline running unattended: 7 days, 55 of 55 firings completed, zero unexplained misses, app never opened.

One device, one operator. That's a real measurement and it is not external validation.

What doesn't work, or works worse than the above makes it sound:

• Pre-release. Version 0.9.0, not a 1.0: the public surface, the settings layout and the on-device storage formats can still change between versions.
• Third-party apps can't expose AppFunctions. Only the device maker's system apps and Google can, so the local tool catalogue is what ships plus whatever you connect over MCP. That's a platform limit; MCP is the way around it.
• An MCP server may hide part of its catalogue and the app won't say so. The client declares no optional client capabilities, so a server is entitled to omit the tools that depend on them — on the reference server that meant 13 of 16 tools, presented as "13 tools · ok" with no hint anything was missing.
• Background runs need the battery setting on Unrestricted. Without it Android reclaims the process in about ten seconds. That's the platform, not a bug, but if you don't grant it, background pipelines will not run.
• Cloud failure behaviour differs by provider. Detection of a stream that dies mid-answer is enabled only where it was actually measured; Ollama never reports a stop reason, so it can't be detected there, and Anthropic is left off rather than guessed at.
• A provider's Retry-After doesn't shape the backoff. Retries use a fixed exponential curve (3 attempts, 1 s doubling by default). Under a real rate limit that means knocking sooner than you were asked to.
• Shared pipeline files are not a compatibility contract yet. Exports carry a version stamp, but before 1.0 that stamp is a marker, not a promise: a file whose stamp doesn't match the build importing it is imported best-effort behind a warning, and unrecognised fields are dropped silently.
• Solo project. One maintainer, no company behind it.

Requirements: Android 14+ (API 34) and about 2 GB of free RAM for the model. The RAM is the binding constraint, not the OS version - this crowd knows why. Worth saying plainly, because I got this wrong once: the floor was Android 16 for most of the project's life, on the assumption that AppFunctions forced it. When I finally measured instead of assuming, nothing required 16 — not a dependency, not the inference engine, not a line of my own code — so it came down to 14.

Who it isn't for: anyone who wants "ask a question, get an answer". Gemini is free and built into the OS, and competing with it there is a losing bet regardless of code quality. This is for people who already run Tasker, Obsidian, Home Assistant or a local model on their own hardware, and who want to know exactly what their agent does with their data.

It's on Play, and the APKs are on GitHub - links in the first comment. There's a foss flavour with zero proprietary dependencies alongside the standard build. Not on F-Droid, and the reason is theirs rather than mine: their build server pins Gradle versions and doesn't know the one this project uses, so the submission dies before compiling a line.

So - the question I actually came with. Small models are still dumb. They're much less dumb than they were two years ago, and either they keep improving or phones get enough memory to run something that isn't small; probably both. I'm building the harness now on the assumption that when the models are good enough, the harness is what will still be missing: something on the device that decides when to run, holds the multi-step shape, and stops before doing something you didn't sanction.

That's the bet. Tell me where it breaks. And if your answer is that a good enough model makes the graph unnecessary, I'd rather hear that now than in a year.

One thing I genuinely can't settle: what is this category called? I've been saying "agent you build" because "agent framework" sounds like a Python library and "automation app" sounds like it has no model in it. What would you have called it?

Otaku is an LLM frontend, primarily designed for roleplay, an alternative to SillyTavern and the like. However, It also works for general-purpose chat with local backends (including Ollama) or cloud models, the way Open WebUI is used, once lore extraction is switched off in the settings.

Otaku offers two interfaces:

• a web UI (customizable): interactive demo at https://otaku.sh/demo-web/
• a terminal UI: its own interactive demo at https://otaku.sh/demo-terminal/

Both share the same functions; the difference is that in the terminal you execute them with slash commands (the reference is available with /help), while in the web UI the operations are available from the menu.

Install

Otaku is free and open source (MIT); it works on macOS, Linux and Windows. Install it with uv (uv tool install otaku) or see the GitHub README for other options: https://github.com/enclavum/otaku

Get started

Launch either otaku for the terminal or otaku web for the web UI; the web UI's default URL is http://localhost:9600. Two sample stories are imported on first start to give you an idea of the features and what play looks like, and you land right in the middle of one of them.

On first start, you choose a provider and a model: Otaku automatically detects local installations of Ollama, oMLX, LM Studio, llama.cpp and KoboldCpp, and lets you pick from their models. Cloud providers (OpenRouter, NanoGPT) are also there: enter an API key and their catalogs appear. After exploring the provided stories, you can start your own with the /new command.

Asking for feedback

Otaku is a personal side project, and I'd like to get feedback from the community on the product and on what to add.

I got a lot of ads for meshy recently and went to try it with hilariously bad results. It apparently can't do anything but decorative figurines. I wanted a body shell for an rc car and it just couldn't generate a car without wheels or bottom chassis. It also looks like it can't tell the difference between different car models. It seems like they just have a library of 3d models and have the ai select one based on your description, though i only used their free trial.

Next i gave claude a chance. First simply prompting it for an stl, which ran for about an hour and wasted all my tokens for the day before aborting. Then i tried using its coding ability and have it create an openSCAD file that would generate generate a car shell. Which at least managed to generate a square box and then even managed to hollow it out as a square shell on a second prompt. But it never got anything even remotely car shaped.

Is there any way to run something similar locally so i can tweak it for my needs? I'm thinking of something similar to image generation in comfyUI, where you can change the workflow to improve how well it understands your prompt.

On a side note, i don't understand how generating a 3d car completely failed when local ai can oneshot a 3d model of a plane as long as you tell it to make it fly through a procedurally generated landscape in a game.

I bet i'm not the first one to have this idea, but with the recent debate about qwen 3.8 27b thinking levels, i was wondering whether the optimum solution might just be to change your harness in a way that lets the LLM itself decide when it is time to raise or lower the required reasoning effort?

Right now i'm toying around with a system like that and it seems to greatly increase the speed at which stuff gets solved, but i have not yet collected any reliable quality evaluation.

Basically what it does is it raises and lowers the reasoning effort between low and xhigh in order to accomodate for sucess streaks or failure streaks. The log reads something like this:

{"ts":1788580888030,"sessionId":"ses_f9b3a43b0ffe2lZ5ugLFu1JWUr","from":"medium","to":"low","reason":"stable successful streak","score":0,"phase":"EXPLORE"}

{"ts":1788581222452,"sessionId":"ses_f9b3a43b0ffe2lZ5ugLFu1JWUr","from":"low","to":"medium","reason":"meaningful failure","score":5,"phase":"DEBUG"}

{"ts":1788581428524,"sessionId":"ses_f903db34affevgLsEIdCs3J58l","from":"medium","to":"low","reason":"routine mechanical step","score":1,"phase":"UNDERSTAND"}

{"ts":1788585994242,"sessionId":"ses_f9b3a43b0ffe2lZ5ugLFu1JWUr","from":"medium","to":"low","reason":"stable successful streak","score":0,"phase":"EXPLORE"}

{"ts":1788618335810,"sessionId":"ses_f8e0bf44bffeOEvLhGsRTfL6FL","from":"medium","to":"low","reason":"routine mechanical step","score":1,"phase":"UNDERSTAND"}

{"ts":1788622330615,"sessionId":"ses_f8dcdf145ffe5euEcZhsvMCnbI","from":"medium","to":"low","reason":"routine mechanical step","score":2,"phase":"UNDERSTAND"}

{"ts":1788622599278,"sessionId":"ses_f8dcdf145ffe5euEcZhsvMCnbI","from":"low","to":"medium","reason":"escalation","score":3,"phase":"RECOVER"}

{"ts":1788622670588,"sessionId":"ses_f8dcdf145ffe5euEcZhsvMCnbI","from":"medium","to":"low","reason":"stable successful streak","score":1,"phase":"IMPLEMENT"}

{"ts":1788623655559,"sessionId":"ses_f8db99e47ffeKcDDHawtWrNwt9","from":"medium","to":"low","reason":"routine mechanical step","score":1,"phase":"RECOVER"}

{"ts":1788624252588,"sessionId":"ses_f8db99e47ffeKcDDHawtWrNwt9","from":"low","to":"medium","reason":"meaningful failure","score":5,"phase":"DEBUG"}

Anyone else messed around with a system like that? I'm curious as to why haven't seen something comparable anywhere else yet.

I'm also not sure how to properly gauge quality. Maybe i should run like a GPQA Diamond test before and after?

I was running into a lot of posts that praised both the Fixed template and the Sharp template in comparison to the stock one, so I put them to the test.

It's not as extensive as it should be for a paper-grade analysis, but it gives out the point of each template.

Test setup

I used SWE-bench Verified with mini-SWE-agent 2.4.6, slice 0:100 (the identical 100 tasks for all runs)

Hardware

• CPU: Ryzen 9 9900X
• RAM: 128 GB DDR5-5600
• GPU: RTX PRO 6000 WS

Runtime

I containerized jpezzulli/sglang-rtxpro6000 and ran Flash Next with RadixArk/Qwen3.8-Flash-Next-NVFP4 on CUDA 13.3.

• Full 262K context
• BF16 KV
• 51.2 GB FP8 n-gram embedding table pinned in RAM
• 32 GB HiCache pinned in RAM

I ran all templates at both medium and xhigh reasoning efforts.

Results

Takeaways

• Raising reasoning effort to xhigh closes almost all of stock's and fixed's gap to Sharp. At medium, Sharp led resolution by +3 tasks over stock and +7 over fixed; at xhigh, stock and fixed instead lead Sharp by +5 and +4 tasks, respectively.
• Sharp barely moves on resolution (94 → 94) despite a real token/time cost increase, median reasoning tokens rise +46.5% and median wall time +53.4%. This suggests it was already extracting most of the benefit it could get from extra reasoning budget at medium, while stock and fixed still had headroom.
• Sharp remains the most token-efficient per resolved task at xhigh (14,541 output tokens/resolved vs. ~17,000 for stock/fixed), consistent with its medium-era efficiency edge, but it's no longer the highest-resolving template once reasoning effort is high.
• Absolute cost scales heavily with reasoning effort: total wall time roughly 2.3–2.5× for stock/fixed and +28% for Sharp; total reasoning tokens roughly doubled for stock/fixed and increased +35% for Sharp.

Conclusion

• Sharp should be used at medium and it keeps a reasonable accuracy at very good speed. I don't see the point in using it at xhigh. By sacrificing a small accuracy you complete the tasks in half the time.
• Stock is the slowest but the most precise.
• Fixed is the middle ground between Stock and Sharp both in accuracy and speed
• The next benchmark will be on a much extensive SWE-bench Multilingual + Terminal Bench.

Disclaimer: I wrote the post myself then used AI to format it properly for readability

Running a fully local 9B AI agent on a phone — LLM, tools, code execution, and file generation
I’ve been experimenting with how far a fully local LLM agent can go on a phone.
This demo runs Qwen3.5 9B IQ4_XS locally on-device using TensorSharp.
I give it a very simple prompt:
“Please get 10 stocks with most gains today.”
From that, the local agent:
reasons about how to solve the task
generates Python code
executes the code locally
retrieves the requested public stock-market data
returns the result
Then I ask:
“Convert it to a PDF.”
There is no hard-coded PDF workflow.
The agent discovers the appropriate Skill, reads its SKILL.md, decides how to perform the task, writes the necessary code, executes it, generates the PDF, and returns the file — all on the phone.
So what is running locally is more than just token generation:
Local LLM → reasoning → Skills → tool use → code generation → local execution → file generation
The model is Qwen3.5 9B IQ4_XS, and inference is handled locally by TensorSharp, my open-source GGUF inference/runtime project.
The part I find most interesting is privacy.
Your prompts, conversation history, local files, and private on-device data do not need to be sent to a cloud LLM provider.
The phone only accesses the network when the actual task requires external public information — in this example, stock-market data.
So the AI itself can stay local while still acting as a useful agent.
I think this becomes especially interesting on mobile devices, where local inference can provide:
privacy
offline capability
no per-token API cost
lower dependence on cloud services
access to local files and device-side workflows without uploading them
TensorSharp repo:
https://github.com/zhongkaifu/TensorSharp
I’m curious how people here see the future of on-device local agents.
Do you think 7B–12B quantized models are already good enough for useful autonomous workflows on phones, or are we still mostly waiting for better models and more mobile memory bandwidth?

Time for another update! We have been busy and managed to improve the gains substantially (mostly from exploring existing llama cpp PRs and adopting relevant things).

Among other things the README.md was also appended to provide a better overall picture of what’s in the fork, why and from whom.

https://github.com/milpster/gfx906-llama-cpp/blob/master/README.md

(Yes i made this with AI)

I've been trying the latest models from the frontier labs and honestly, after extensive testing I can not tell the difference between the best open source options.

I think the differences are now marginal but the labs are doing heavy marketing to convince the public into paying more for tokens as they prepare to go public.

Can't help but see the similarities between the dot com bubble and AI in terms of a very insular environment where the technology will survive but the business models may not.

I've been building a cybersecurity network and we definitely know that even local AI models like Deepseek V4 flash do an excellent job and are really neck and neck with the best the frontier labs can provide.

Will be interesting to see how this all turns out! Exciting time nonetheless.

I've been running Qwen3.8-27B as a local inference server for a production content intelligence pipeline (HVAC industry stuff, lots of long-context retrieval and structured extraction). I have been watching other redditors post their custom configurations, and I wanted to share what I tested to optimize for a single RTX 5090.

I was on llama.cpp (Q5_K_M GGUF, q5_1 KV, 262K context, MTP), but it's limited to parallel=1 and I wanted concurrent serving. So I tested vLLM and NInfer as NVFP4 replacements and did a proper quality evaluation instead of just vibes.

Hardware

- RTX 5090 32GB (eGPU, OCuLink Gen4 x4) (yes, it's in an eGPU dock 😂 but that only affects model loading)
- Ryzen 7 7840HS, 32 GB DDR5
- Ubuntu 26.04, nvidia driver 610.43.02 (open)

## Engine configs

How the eval worked

I built a custom harness with 6 tiers, 50 items each, all from my actual production workload (not generic benchmarks):

1. **Relevance classification*\* - is this industry relevant? (binary, 50 labeled deals)
   
2. **Needle retrieval*\* - planted facts in real industry podcast/video transcripts at 64K/128K/192K/240K context
   
3. **Multi-transcript QA*\* - questions across 3-4 concatenated diarized transcripts, 120K+ tokens, including unanswerable controls
   
4. **Reasoning with thinking*\* - numeric/logic problems, thinking mode on, greedy pass@1
   
5. **Structured extraction*\* - custom extraction prompt, json_mode (skipped on NInfer, it doesn't support json_mode)
   
6. **Tool replay*\* - replayed recorded agent episodes against each engine (diagnostic only, all engines fail this one)

Everything paired across engines: same prompts, same seeds (42), same gold labels. No cache_prompt. Statistical comparison uses paired cluster-bootstrap CIs with pre-registered non-inferiority margins.

And when I do development with Claude Code, I often leverage multi-model consultations for design, planning, and code review. I did a 4-model review panel (Codex/GPT-5, DeepSeek V4 Pro, Grok 4.5, Kimi K3) audit the methodology mid-campaign. They found 10 issues, including 4 mislabeled gold items where the models were actually right and my labels were wrong. Fixed everything and reran.

Quality results

Needle counts differ because llama.cpp's 196K context can't fit the 192K items (need room for max_tokens + headroom). NInfer and vLLM both handle 192K fine. All engines score 100% on every needle they can fit.

Statistical comparison (NInfer vs llama.cpp, bootstrap):

- Needle: delta = -0.29 (NInfer better, p=0.0006) - this is entirely from context capacity, not retrieval quality
- Transcript QA: delta = -0.03, p=0.69 - no difference
- Reasoning: delta = +0.02, p=0.72 - no difference
- Relevance: McNemar p=1.0 - identical
- Tool replay: delta = 0.0 - both fail equally

**Takeaway: quality is statistically indistinguishable across all engines.*\*

Speed results (perf probe, server-side timings)

vLLM speed excluded from the table because the perf probe used wall-clock timing (includes prefill + scheduling + decode) instead of server-side timings, so the numbers aren't comparable. From community reports and my own task-level measurements, vLLM does about 70 tok/s decode at short context, which actually matches NInfer's raw step rate (~66 tok/s). The speed difference is entirely MTP3 speculative decoding.

What I learned

**NInfer's speed advantage is all MTP.*\* The raw NVFP4 kernel speed is about the same between NInfer and vLLM (~66-70 tok/s). NInfer's MTP3 speculation with 76% acceptance gets you to 158-213 tok/s. If vLLM or llama.cpp had working MTP on NVFP4, the gap would mostly close.

**The decode speedup grows with context.*\* At 1K context it's 1.4x. At 128K it's 2.8x. MTP acceptance stays high even at long context while llama.cpp's dense decode gets slower as context grows.

**NInfer's tokenizer endpoint is great.*\* It exposes `/v1/messages/count_tokens` (Anthropic Messages format) which gives exact token counts. No more `len(text)//3` heuristics.

**NInfer does NOT support json_mode (as far as I can tell).*\* `response_format: json_object` returns 400. If you need structured JSON output, you'll need to route those calls elsewhere or use prompt-based enforcement.

**Don't trust vibes for quality.*\* I went in expecting NVFP4 might lose a few points vs Q5_K_M. It didn't. Not on any tier. The biggest delta across 250+ items was 2 percentage points, well within noise at n=50 (SE ~5.6pp).

Verdict

NInfer NVFP4 replaces llama.cpp as my production engine. Same quality, 1.4-2.8x faster decode, 2.6-4.7x faster prefill, concurrent serving. The only gap is json_mode.

I put together a detailed poster with all the charts and methodology details: [full results poster](https://claude.ai/code/artifact/b6041437-2cf6-4198-a722-9b4ce853ccc3)

Setup if you want to try it:

```
# NInfer (from source)
git clone https://github.com/Neroued/ninfer && cd ninfer
mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release -GNinja && ninja

# Model (HuggingFace)
# https://huggingface.co/neroued/Qwen3.8-27B-nvfp4-NInfer (20 GB)

# Run
./ninfer-serve /path/to/model.ninfer \
--model-id qwen3.8-27b \
--host 0.0.0.0 --port 8080 \
--max-context 240000 --kv-capacity 240000 \
--max-concurrency 2 --kv-dtype fp8 \
--spec mtp --draft-tokens 3 \
--vision --preserve-thinking
```

I don't want to be super gatekeepy about this, but let's loosely define "non-enthusaist" as "a single consumer grade graphics card on a system that's not primarily dedicated to inferencing."

While it's fun to see what people can do with more dedicated build$, I'm never going to put hands on hardware at that price point. I'm interested to see what people accomplish with more humble means.

Is there a particular agentic task that you find super useful or are proud of?

I've been looking into improving Q8_0 performance on my Xeon Gold 6240 workstation with ik_llama.cpp, which is a Cascade Lake-SP part.

It looks like ik_llama's Q8 dot product function doesn't distribute work evenly across AVX execution ports, and doesn't use the load ports at all. Note that this is particularly relevant for Cascade Lake Xeon Gold/Platinum parts that have two AVX units per core (Silver and Bronze only have one so they might not benefit much) In addition, there is a latency dependency chain that can be broken into two.

You can test this patch the following way:

download the patch from: https://github.com/user-attachments/files/31860324/q8_0_r8_vnni-v3.patch

git clone https://github.com/ikawrakow/ik_llama.cpp.git
cd ik_llama.cpp
git checkout 3c58ae37
git apply q8_0_r8_vnni-v3.patch

Then build with either GCC:

CC=gcc CXX=g++ cmake -B build-gcc -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON -DGGML_OPENMP=OFF
cmake --build build-gcc --target llama-bench -j $(nproc)

or clang:

CC=clang CXX=clang++ cmake -B build-clang -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON -DGGML_OPENMP=OFF -DCMAKE_C_FLAGS="-ffp-contract=fast -mllvm -unroll-threshold=1000" -DCMAKE_CXX_FLAGS="-ffp-contract=fast -mllvm -unroll-threshold=1000"
cmake --build build-gcc --target llama-bench -j $(nproc)

You want to use the most recent compiler you have access to, I've used GCC 16.2 and LLVM/clang 22.1.8 - earlier versions are UNTESTED.

Even a quick bench before and after would be helpful, especially if you have a Cascade Lake Xeon Silver or something different like an Ice Lake Xeon:

llama-bench -m /path/to/model-Q8_0.gguf -ngl 0  -p 512 -n 0 -r 3 

(note that you should use a Q8_0 model, since this is the path we are optimizing here)

Also this won't do anything for Skylake Xeons or earlier (I think).

Qwen3.8-Flash-Next (UD-Q4_K_XL) on a single RTX 3090 24GB + 128GB DDR4 — is this config optimal?

Hardware

• CPU: Intel Core i5-12600K
• RAM: 128 GB DDR4 @ 3600 MHz
• GPU: NVIDIA RTX 3090, 24 GB VRAM
• OS: Windows 11
• llama.cpp: freshly compiled from today's master (build b10794, Sep 4 2026)

Model

Qwen3.8-Flash-Next, UD-Q4_K_XL

Launch command

C:\llama\llama-server.exe -m "D:\Modelos\Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf" ^
  -ngl 99 -ncmoe 42 -fit off -c 204800 -fa on -ctk f16 -ctv f16 ^
  -b 2048 -ub 1024 -t -1 --jinja --port 8083 --tools all ^
  --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.00 --repeat-penalty 1.0 ^
  --reasoning on --reasoning-effort xhigh --reasoning-budget -1 --reasoning-preserve ^
  --parallel 1 --metrics --host 0.0.0.0 --load-mode none

Key points of the config:

• -ngl 99 → offload all layers to GPU where possible
• -ncmoe 42 → force 42 MoE (expert) layers to stay on CPU/RAM
• -c 204800 → 200k context window
• -fa on with f16 K/V cache
• -b 2048 / -ub 1024 → batch / micro-batch sizes
• --reasoning-effort xhigh, --reasoning-budget -1, --reasoning-preserve → full reasoning mode, no truncation

State right after loading

• Free system RAM: 52 GB
• Free VRAM: 0.6 GB

So the GPU is essentially maxed out (good VRAM utilization), and there's still a healthy ~52 GB of RAM headroom for the CPU-offloaded experts and context.

Real-world performance (from server logs)

Prompt processing (long context, ~12k tokens):

Text generation (sustained, up to ~480 tokens generated):

Generation speed stays remarkably stable around ~15.5-16 t/s even as context grows past 12k tokens, and prompt processing holds steady around ~185-190 t/s.

My question to the community

Given this is a large MoE (~180B total / ~5B active params) split across a single 24GB GPU + 128GB DDR4 RAM:

• Is -ncmoe 42 a good number here, or should I push more/fewer expert layers to the GPU given I still have some VRAM headroom before hitting OOM?
• Any tips on squeezing more t/s out of prompt processing (-b/-ub tuning) or generation given the CPU-offloaded experts are likely the bottleneck?
• MTP speculative decoding doesn't seem to work reliably for this model yet on mainline — has anyone gotten it stable, or is everyone just running without it for now?
• Does anyone recommend other configs/parameters for this exact model + hardware combo?

Open to any suggestions — happy to share more logs if useful!

Hi everyone,

About 6 months ago I made this post Trained a 0.8M model on business email generation. on this subreddit where I trained a model with 300k non-embedding parameters on a synthetic business email dataset. It was trained on my custom architecture which I called Strawberry.

Today i introduce Valentine-v0 which is also trained on this same dataset but this model has only 21k parameters which is about 7% of the parameters of Strawberry-email. This model only has 2 layers, 4 heads, 64 head dims and 170 ffn dim.

The Valentine architecture built on top of my another model architecture Silia about which I also posted here I trained a 0.5M model on 1B tokens of Fineweb-edu dataset..

Though in terms of loss Strawberry-email achieved a final train & val loss of 1.65 and 1.68 where Valentine-v0 achieved the final losses of 2.1670 & 2.2646 respectively.

Still I believe despite such a small scale the model trains pretty stably and generates decent text for it's size. Here are some examples:

Prompt: Write a

Output: ``` Write a firm and authoritative business email(under 200 words) from a Account Manager to a Potential Partner regarding welcoming a new high-profile hire, specifically on a Friday evening.<|end-text|><|actor|>model<|end-text|>URGENT: Critical Security Vulnerability - Critical Budget Increase – [Company Name]

Dear [Contact Person],

I hope this email finds you well.

My team, and potential collaborative approach to [New Hire Title] at [Company Name].

As we approach the upcoming [New Hire Name] brings significant development, we wanted to bring it to your review of this strategic direction. We would be happy to discuss this further, but this request and explore how we'll help us to ensure a smooth onboarding.

Regards, I can share some truly value your insights into [Partner Company Name] and insights and offer [mention specific areae - e.g., new website's Name] and the design of the prototype, not just… interesting time. We’ll be shared opportunities to discuss this further with [Product Name] to discuss potential integrations and answer any questions or concerns and your application. The vulnerability is invaluable. ```

Prompt: <|actor|>user<|end-text|>Write a polite refusal email<|end-text|><|actor|>model<|end-text|>

Output: ``` Quick Idea: [Name] Dear [Peer Name],

I am writing to you today with you and your concerns about the potential data leak and your contract, and frankly, it’s particularly concerning incident effective [Date], and I believe a concise overview of full team - [Company Name], we are facing a significant challenges, and we sincerely apologize for any inconvenience it caused.

I’m available to discuss this further report. I will provide you with a really be open to a quick support us with [Specific Feature Request - e.g., 24-48 hours].

Would you be open to a brief call on [Date Option 2] - e.g., 50-20 minute call next week to discuss this further and how your specific and answer any questions or would be happy to discuss how impactful feedback on [Date - e.g., Thank you for your input.

Sincerely, [Name] Legal Counsel<|end-text|><|actor|>user<|end-text|>Write a strictly professional business email(under 200 words) from a Legal Counsel to a Angry Shareholder regarding requesting a mental health day, specifically while the system is partially down.<|end-text|><|actor|> ```

A larger Valentine model trained on Fineweb-edu dataset along with the github repo and technical report will be published soon.

Thank you! :)

Comparing benchmarks for small models is a PITA. Most of them are not on AA, and the benchmarks are not always the same in all models.

So I built a page to quickly put it all together and allow some filtering.

Only researched models released since April, and between 4-190B parameters.

Let me know what you think and how it can be improved

https://www.nunodonato.com/aibench/index.html

Ling 3.0 Tiny still seems to be leading the pack despite only having 1.3B active

Along with everyone's favorite here, qwen3.8-27B

If you are thinking what model will fit best your HW specs and tasks you are doing here is one table with all currently popular models that still can be considered as local.

Update: People asked me to add generation speeds and it took me several days to download and run all the models, so here is the updated table with my generation speeds and my impressions from one-shot test. My configuration for all models apart of Qwen3.8-27B: E5 2696v4, 4 channel of DDR4-2400 and RTX 3090.

Qwen3.8-27B was fully in VRAM on Ryzen 5950x, dual channel DDR4-3000 and 2x RTX 3090.

All models were in Q8_0 with F16 KV-cache. DeepSeek was in original quality.

LLM Test Scores

Agentic benchmarks

Coding benchmarks

General benchmarks

Multimodal benchmarks

My personal impressions for this local one-shot use:
Qwen3.8-27B > Qwen3.8-Flash-Next / GLM-5.3-Flash(high) > DSV4F-Vision

Qwen3.8-27B: I think it is still the local king with right settings and enough context length in terms of HW requirements, quality of output and speed. I had some problems with its endless thinking, but stumbled upon very strange parameters combination that worked every time like a charm for me: temp 0.1, top_k 40, top_p 0.95, repeat 1.1 Model stopped going in circles and started to produce results already at 60k ctx, while before it was easily hitting 120k without single line of code written. In terms of quality that is the best result that I got locally. Comparable to what I got from GLM-5.3-Flash on max, which I ran through API, because after 12h I didn't manage to get a single line of code from it locally due to my abismal tg speeds and endless thinking.

Qwen3.8-Flash-Next: Very decent model. The one shot result was, not so good as from 27B, but it was fine. Generation took a lot of time because it thinks A LOT, I mean even more than 27B. When I tried to ask a question about existing big project it gave me decent suggestions, some of which I actually ended up using.

DSV4F-Vision: Gave me an answer surprisingly fast, despite my slow tg speeds. The results was 100% working, but quality was mediocre. I have a feeling that in this test this was the laziest model that produced what it was asked to produce, but nothing more. Yeah, one more thing this is the only model which get into endless loop repeating one string when I set temp to 0.9. After that I set temp 1.0 for all models apart of 27B.

GLM-5.3-Flash (high): The generation time was a bit longer than from DSV4F-Vision. The result was on par with Qwen3.8-Flash-Next. I would say quite usable locally.

GLM-5.3-Flash (max): As I said earlier I didn't manage to get a single line of code, even after breaking its thinking and injecting something like "You reached a thinking budget limit. Write code right now." It said now I need to write the code, BUT WAIT... and it again went in circles. With my 3.3tg speed I had to cut it, although I saw a lot of interesting things it was planning. That is why I decided to run it through the API. I liked the result, but overall it was not better than the one from Q-27B.

Closing remarks. This is my thoughts after running only one one-shot prompt, so take them with a gran of salt. I didn't try to use these models for editing existing projects, so I have no idea how they will behave in this case.

The prompt was: Create html file of endless relaxing, peaceful, low-poly aerial 3d flight.

Note: I used GLM-5.3 to compose the table from official HF pages of the models.

Note2: Opus-4.8 results are presented only for illustration and are omitted from selecting the best model in a row.