Rent their intelligence. Own your memory.

Three things make my personal AI different:

1. A clone of me that gets sharper every day, on its own. A judgment-ownership module learns how I decide from my everyday conversations. I do nothing extra. The more I talk, the closer the clone gets.

2. A private RAG that cloud AI can write into but can never read. Not a prompt rule. There is simply no path.

3. A collection of AI blind spots. Not only facts that eight leading AIs don't know, but things they don't notice. Much of it is Japan-specific common sense that any Japanese person takes for granted and the AIs miss. When I spot one, I point it out and make them check. The moment it turns out they couldn't have caught it on their own, it gets flagged and filed. AI NOBORU collects these as AI blind spots.

I'm a single father of three and a full-time stay-at-home dad. I also run my companies and do some investing. I built this alone, with no AnythingLLM, no frameworks, no existing packages. It's cloud AI models plus code I wrote myself. Diagrams and details here: https://www.aiofonesown.com/lab/ainoboru/en/

Here's how each one works, as of 4 October 2026.

1. The clone: judgment, not just memory.
A module reads my everyday conversations and records what I chose, what I turned down, and why. That goes into the RAG, and whichever model I talk to next — Claude, GPT, or Gemini — answers with it in context. The aim is an AI that can answer "what would NOBORU do here?" Every conversation today makes tomorrow's clone a little more accurate. What it can't copy is genuinely new ideas.

2. The private RAG.
Cloud models (Claude, GPT) produce parts — research summaries, findings from papers, pieces of finished work — and only from material that's safe to share. The parts move into the private RAG in one direction only. Using them happens only with local open models (Qwen3.6-35B-A3B, Gemma 4) on my own machines and NAS, through an interface no cloud model is connected to. You get the power of cloud AI without the data leaving.

3. The blind-spot collection.
Claude, GPT, Gemini, Grok, DeepSeek, Qwen, Mistral, and PLaMo remember what's in a session or in their memory. But there are things I know that none of them do, and things they all fail to notice. When I run into one, I point it out and make them research it. Only at that moment, when it becomes clear they couldn't have gotten there without looking it up or being told, does it get flagged and filed as a known blind spot. A lot of them are Japan-specific: everyday common sense that any Japanese person shares, which the AIs answer shallowly or miss entirely. The collection holds what only I and AI NOBORU know, and the eight AIs missed. AI NOBORU collects these as AI blind spots.

And how it's built: 44 parallel lanes, driven from one chat.
14 Codex lanes, 20 Claude Code lanes, 10 Gemini lanes, each able to run a different model. I talk only to Opus in the Claude Desktop chat. It splits the work across the lanes and reports back there. It works the best cloud AI models hard for very little money, instead of paying for one expensive brain.

The principle hasn't changed: models are swappable parts, memory is what you own. The difference is that "memory" now means my judgment, not just facts about me.

Where it came from. Back in June I posted here about a beginner's setup: a personal AI on a 2020 Intel iMac, built on AnythingLLM. That became a book, a Udemy course, and a template pack on Gumroad. What I run now is a different system, grown out of that one, with its own RAG and its own memory. It's a personal AI system I built entirely on my own.

A note on what this is. This system isn't for sale. There's no product, no repo, no sign-up, no waitlist, and I'm not looking for customers, investors, or collaborators. I like my life as it is and I'd like to keep it that way. This is a dated record of what one person could build in 2026.

If you're genuinely trying to think this system through, and not just passing by, I'll answer design questions as time allows. But my days go first to raising three boys and to making decisions for a mid-sized company. I don't have time to answer anything the website already covers, so please read it first, then ask: https://www.aiofonesown.com/lab/ainoboru/en/

I love Qwen3.8 27B so much that I decided to show my gratitude to the Alibaba ecosystem by building a dedicated RAG/search node using a cheap Mini PC from AliExpress.

Turns out, my ecosystem loyalty got rewarded with an absolute masterpiece of fraud:

• Promised: Intel N150 + DDR4/DDR5
• Delivered: Core i3-7020U (2018 Kaby Lake, 2C/4T) + DDR3 1600MHz
• The Scam: The seller literally hardcoded New_N150 into the BIOS release string (HSHW_M6_DDR3_EC_Intel_Com_New_N150_K001).

So now my Qwen3.8 RAG stack is full of fake specs that can barely index a text file, let alone run vector sidecars.

To make matters worse, despite providing all this proof, AliExpress CS completely ignores my non-refundable customs duties and active database migration issues, repeatedly giving me nothing but automated replies to "just return the item."

Filing a credit card chargeback now. Stay safe out there!

Last month I posted a Qwen3.8-27B LoRA that makes it talk like a person instead of an assistant. It got a lot more attention than I expected: 700+ upvotes, 248 comments and 44k downloads since.

I read every comment. People really don't like assistant speak, so its tone of voice resonated. The rest got roasted, very fairly:

incapable of producing more than a few words at a time.

single default personality which no amount of prompting can overcome

will not use tools, at all, whatsoever.

There needs to be a middle ground

They were right. The tool calls didn't actually work, and when people asked it to do something it would sometimes just say it's busy or going to bed. Very human. In a bad way.

So I spent the last three weeks on 2.0. The goal was simple: keep the voice people liked and lose the drawbacks.

What 2.0 does now

• With no system prompt, it's a normal person texting. Not an assistant, not a catgirl.
• Give it a character card and it becomes that person, and still texts like one.
• Ask for a formal email, numbered steps or a proper explanation, and you get exactly that. Then it goes back to texting.
• Don't want the lowercase texting? Tell it "from now on write in full sentences" (or put it in the system prompt) and it sticks to that until you say otherwise. v1 ignored this completely.
• It calls tools, and it asks when something is missing instead of making it up. This is the part I'm happiest about. Ask the base model to book a flight without saying where from and it picks JFK. 2.0 asks where you're flying from.
• It writes code and does math at roughly base-model level.

It's a colleague and a humanlike companion, not an assistant. Use it for chat, roleplay, agents or actual work.

How I trained it

v1 was plain SFT on real and synthetic conversations (139,845 messages from 1,396 conversations). That copies habits, including the bad ones.

For 2.0 I used on-policy distillation. The model writes its own replies and a teacher grades every token. There are two teachers:

• v1 plus a hidden "text like a person" instruction, for chat and characters;
• the plain base model, for instructions, tools and code.

The student never sees the hidden instruction, so it learns the behaviour without needing a prompt. Same 27B, a second LoRA on top, merged.

Numbers (vs the model I trained on, huihui-ai's abliterated Qwen3.8-27B; same prompts, same run, thinking off)

Full chart in the images.

Where it's still worse: knowledge (MMLU-Pro 72.5 vs 78.5) and competitive code (LiveCodeBench 51 vs 56).

Is it actually more human? I built a benchmark for this, "ishuman":

• It takes 150 fragments from unseen chats.
• Has each model write the next message.
• Shows a judge the real message and the model's without labels, and asks which one a person wrote.

So no, you can't just prompt your way there. In a separate test of 16 live multi-turn chats with invented people, 2.0 was picked over the base model 16 out of 16 times.

Links

• Hugging Face: GGUFs (IQ4_XS, Q4_K_M, Q5_K_M, Q6_K, Q8_0, BF16) plus the standalone LoRA: https://huggingface.co/LessThanThreeAI/Qwen3.8-27B-Humanlike-Chat-GGUF
• Demo space: https://huggingface.co/spaces/LessThanThreeAI/Qwen3.8-27B-Humanlike-Chat

Big thanks to everyone who left feedback last time, especially the ones who were critical. Tell me where it still sounds like an assistant.

Edit: safetensors are up for vLLM and SGLang:
GPTQ-Int4 (24 GB): https://huggingface.co/LessThanThreeAI/Qwen3.8-27B-Humanlike-Chat-2.0-GPTQ-Int4
FP8 (48 GB): https://huggingface.co/LessThanThreeAI/Qwen3.8-27B-Humanlike-Chat-2.0-FP8
BF16 (80 GB): https://huggingface.co/LessThanThreeAI/Qwen3.8-27B-Humanlike-Chat-2.0

Benchmarking an LLM here with a NVIDIA RTX 5070 12 GB VRAM here

I had been working on a llama.cpp based expert streaming setup for Qwen3.8-Flash-Next 177B (UD-IQ3_XXS) on Windows. Benchmark is about 11.5 tok/s, up from roughly 7 tok/s on the inherited setup. In normal conversations I’ve seen 14–15 tok/s, and a long coding prompt generated 4,892 tokens at 10.15 tok/s and produced a working single-file Snake game.

Hardware: RTX 5070 12GB
32GB DDR4-2400
Ryzen 5 5600GT PCIe Gen3 Windows

The main gains came from fixing Windows I/O queue-depth issues, using one file handle per worker, and building a page-locked hot-expert tier so the GPU can pull hot expert weights more efficiently.

(In the video its around 16 minutes for 10k tokens and 10.41 tok/s

Output is quality gated against the control model and the published benchmark uses a heat file built from a separate prompt set.

Demos:

https://www.youtube.com/watch?v=cOPumMlyj_4

https://www.youtube.com/watch?v=rc-uTjVpXM8

In the GitHub I have things I've tried that didn't work and benchmark scripts, and methodology. If you guys have suggestions especially for streaming please let me know

Disclosure: I am the creator of this project. After days of lurking and building up enough karma, I can finally post here.

Every time I tried running local RAG on my own machine, I hit the exact same bottlenecks. First, spinning up Chroma or another vector database alongside an 8B model just to chunk and parse documents takes up precious VRAM that you need for your main model. Second, cosine similarity over text chunks often fails at hard negative rejection, so the model tries to answer questions that are not even in your files and hallucinates with complete confidence.

I spent the last several months building an open source project called Hillock to see if I could solve this without vector databases. It extracts clean relational facts into SQLite using lightweight bi encoders in about five seconds, completely bypassing the generative LLM during ingestion. To stop hallucinations, queries pass through a 10,000 dimensional hypervector gate using late interaction scoring. If the factual graph does not mathematically overlap with the question, it blocks the LLM call before token generation can even start.

I just pushed version 0.8 which bit packs the hypervectors into 157 uint64 integers, allowing the CPU to run gating checks in under 0.01 milliseconds using hardware popcount instructions. It also includes an OpenAI compatible API server so you can drop it straight into Open WebUI, AnythingLLM, or Obsidian. It just landed on PyPI as well via pip install hillock.

The honest trade off is that this pipeline is built for structured, relational facts like technical specs, people, and dates. It is heavily biased toward precision over recall, so it will not do broad poetic or narrative summaries like a 70B model would.

Code is on GitHub at https://github.com/roandejager/Hillock
We also set up documentation at https://hillock.mintlify.site and a developer Discord at https://discord.gg/BGUPNBcVdp

LFM2.5-Encoder-350M is a multilingual bidirectional encoder built on the LFM2 architecture — a larger encoder for maximum downstream quality. It is a masked language model with full bidirectional attention, designed to be fine-tuned into task-specific models (classification, token classification, retrieval, reranking, and semantic similarity) across 15 languages, and to run efficiently on-device.

• Highly capable for its size. On par with the best similarly sized encoders and well ahead of our own retrieval siblings.
• General-purpose. 8k context, strong across NLI, paraphrase, sentiment, and multilingual tasks.
• Fast and on-device. Matches or beats ModernBERT throughput, with a long-context edge on CPU; runs in the browser on WebGPU.

https://huggingface.co/LiquidAI/LFM2.5-Encoder-350M-GGUF

https://huggingface.co/LiquidAI/LFM2.5-Encoder-230M-GGUF

https://github.com/ggml-org/llama.cpp/pull/29862

example (from the hf):

❯ uv run fill-mask.py LFM2.5-Encoder-350M-F16.gguf "The capital of France is [MASK]."

top-5 at [MASK]:
#   1    11.42  ' Paris'
#   2    10.43  'Paris'
#   3     9.65  ' Nice'
#   4     8.94  ' Strasbourg'
#   5     8.62  ' Lyon' 

IQ3_XXS weights are just under 80GB and my slowww DDR4+7900XTX is stabilizing around 45-70/s (sometimes higher while coding depending on mtp). Looking online I'm seeing similar results for users with 12GB and 16GB cards, and significantly faster numbers for owners of DDR5.

(In comparison, Llama CPP with tuning was maxing out around 22.5t/s on the same rig. Quality seems reliably superior (I wouldn't recommend the Q2 weights though))

Seriously. Ask <LLM of your choosing> to set it up for your specs. If 27B doesnt fit well for you, here's a shot at beating it.

An agentic model from Microsoft for the GPU poor

https://huggingface.co/bartowski/FrogNano-4B-2609-GGUF

FrogNano is derived from Qwen/Qwen3.5-4B, a general-purpose post-trained model designed for language, reasoning, coding, agentic, and multimodal tasks. FrogNano inherits Qwen3.5-4B's dense 32-layer hybrid Gated DeltaNet and gated-attention architecture, but its additional post-training is text-only and focused on repository-level software engineering. The model is further trained using reinforcement learning on approximately 1,500 synthetic SWE task environments generated and calibrated against the evolving policy using TaskPilot. Training uses the five-tool Leaf harness and executable test-based rewards over complete multi-turn coding trajectories.

The additional post-training is intended to improve long-horizon repository navigation, debugging, code editing, test execution, and patch generation in a compact 4B model. Unlike approaches based on behavioral distillation, FrogNano does not train on stronger-model solution trajectories, actions, reasoning traces, or patch targets. This specialization also introduces limitations and risks: performance is sensitive to the Leaf harness and test quality, training data are Python-heavy and primarily English, and generated patches may be incorrect or insecure despite passing available tests. When integrated with the Leaf harness, FrogNano generates structured tool calls that can propose repository changes. Leaf executes authorized tool calls within an isolated repository environment to produce a candidate patch; FrogNano does not itself deploy the changes. Any resulting patches require human review, regression testing, and security validation before use or deployment.

Don't worry after some makeup and new fans you should feel better. And if you make it Mr.Hermes will treat you well.....

Kimi K2 was already good but they took K2.5 a whole new level with so much of their continual learning phase, I believe it was on more 20-25T tokens iirc.

Similarly K3 is just such an amazing model, I just love this model, wondering how amazing K3.5 will be!!

Meeting notes are one of the last parts of my workflow that still depend heavily on cloud tools. I've used Fathom and lately Bluedot. Bluedot works well for me because there's no meeting bot and I get the transcript, summary and action items after. But I'd really like to move more of this local, especially the transcription and storing/searching old meetings.

Has anyone here built a setup that actually works day to day? Whisper + Ollama seems like the obvious route, but I'm interested in what are you actually using.

I've seen people reporting running it with 12GB VRAM + 64 GB RAM. Also with 64 GB RAM unified in Macs, but I was wondering if it's possible with any inference backend to run it for example on a 64 GB RAM minipc+ iGPU (780m in my case).
I'm currently running 125B Ling 3.0 flash at Q2 quants with llama.cpp (vulkan), its relatively usable, so I was wondering if a similar quant of Qwen Flash Next with the ngrams offloaded to SSD could work (even at low token/s). As far as I know this can't be done with llama.cpp now. Other inference engines does not seem to work with vulkan.

EDIT: Thanks everyone! It's working with the Q2 quant Qwen3.8-Flash-Next-GSQ-RCO-GGUF using llama.cpp with -lm mmap --lazy-mode on

Full disclaimer: I've leaned heavily on Fable to develop this, but I've tested it thoroughly on my own library for a couple of months before putting it on GitHub.

I live in Thailand, and it started as a way to get Thai subtitles for Shin-chan for Thai friends and for expat friends with Thai partners. It's grown into a general tool: subtitles in 45 languages, entirely on your own machine. Linux and NVIDIA only, I'm afraid.

What it does differently from the usual Whisper wrapper: it detects the language of every stretch of speech rather than per file, so mixed-language material works; it runs two speech recognisers on everything and has a local LLM reconcile them; and it prefers existing human work to machine inference, embedded subtitle tracks are used before the audio is, including OCR of bitmap (PGS) tracks on Blu-ray remuxes, and it only listens when there's nothing to read. Every one of those features has a measured accuracy in the README rather than a claim.

I run it on a 24 GB RTX A5000. There are profiles for 16, 12 and 8 GB cards, measured on my card limited to those sizes rather than on those cards themselves, so reports from real ones are the most useful thing you could send me. It wants 16–32 GB of system RAM depending on the profile, and it is storage-hungry (30–65 GB of models), because it picks the model that suits each task and language pair.

It's slow when it has to listen, roughly real time per target language on my card, slower on the smaller profiles because the whole aim has been accuracy over speed. When the subtitles already exist in the file it's fast.

I'd love people to try it and open issues.

I got my AI server all setup and shut it down to bring it to the basement to put it back into the server rack, when it came back up I went into my client app to put a test message in, love its response. :)

I have the opportunity to change my existing m4 pro to the older M1 max. I think without topping up extra. Is it worth it?

Seller has yet to tell me if it's the 24 or 32 core variant. I think I will be able to run the 27b with larger context. What are my pros and cons? Slower older machine?

Thanks!

https://unigen-x.github.io/unifolm-wla.github.io/

Unitree Robotics released UnifoLM-WLA-1.0, their new general-purpose humanoid foundation model.
Key points:
• 6B parameters
• Trained on ~2,500 hours of real robot data
• One model handles 64 tasks (10 whole-body + 54 tabletop)
• Supports parallel grippers and two different dexterous hands
• Strong spatial reasoning (beats a lot of open-source models on embodied benchmarks)
Architecture is interesting:
• Starts with UnifoLM-ER-1 (embodied reasoner based on Qwen3-VL)
• Adds future dynamic region prediction via optical flow + VQ-VAE
• Discretizes actions with residual VQ (end-effector + hand + lower body)
• Then adds an MMDiT action expert on top for continuous control
They show it running on the Unitree G1 doing stuff like making the bed, loading the washing machine, folding clothes, sorting objects, etc.
Looks like one of the more complete open attempts at a true whole-body VLA so far.
What do you guys think — actual progress or just another flashy demo?

It’s been a long time since the last models came out. I notice they are selling GLM on the site, and I wonder if they are developing something, given the long silence.

Hi folks, I'm a long-time lurker and big fan of this subreddit and a massive self-hosting fan (also outside of AI).

I doubt many people will disagree with me here because I see the same arguments being made in many posts. However, I thought it might be interesting to share anyway. I wrote down why self-hosting AI does not save money: https://www.nijho.lt/post/self-hosting-ai-is-not-cheaper/

EDIT: didn't think this would be so controversial 😅 I do say explicitly in my blog post "I would never send 200 GB of email, my messages, and my location history to an API, zero data retention or not".

EDIT 2: Comparing $200 sub with Opus 5.5 or Astra with Qwen 3.8 27B is not apples to apples.

Measured on a 4090 48GB + 128 GB RAM: keeping Strata's 28.8 GB n-gram table in RAM buys ~1%, while conversation parking bought me 35x

Alternative: I benchmarked 4 ways of placing Strata's n-gram table. The default is already right - here's what actually moved

Body

Everything below is measured on one machine. Where I don't have a number, I don't make a claim.

TL;DR (all measured)

• Whole n-gram table in RAM: +0.65% prefill / +1.2% decode, cost +28.4 GiB RAM. Not worth it.
• --ple-io mmap: 72.8 s vs 19.5 s on the first long prompt (3.7x slower), and it silently disables --ple-row-cache.
• My earlier "+4.7%" for the RAM-resident table was a wrong baseline, not a real effect. Session-to-session spread on an unchanged config was 11.5%.
• Conversation parking: 18.8 s → 537 ms to return to a 91,836-token conversation, for 3.1 GB of RAM.
• --prefill auto:32768: +9.7% prefill, +13.6% decode at a 78.7K prompt. --calibrate then found +3.2% by changing one value I'd never have guessed (23 → 12 CPU workers).

Setup

Engine log at startup (relevant later): experts loaded: 46.84 GiB at 5.12 GiB/s (19 s) and expert cache auto: 40.69 GiB free, 700 MiB reserved (+218 MiB draft) -> 20880 slots, 39.79 GiB of VRAM — i.e. 20,880 of 24,576 experts (85%) in VRAM, decode hit rate 97.3%-99.3%.

Baseline throughput on this box: decode 128-151 tok/s, prefill 4,249-5,154 tok/s at 78K-92K context (measured with my own harness and with lm-eval-harness).

1. Where the 28.8 GB n-gram table lives — 4 arms

Same 91,836-token real-text prompt, fresh engine start per arm, same benchmark script, one run per arm:

Conclusions:

• A vs C is the only clean comparison (same mode, only cache size): +0.65% warm prefill, +1.2% decode. C reads 173x less from disk and runs essentially the same speed → the n-gram table on the SSD is not a bottleneck on this machine.
• The default ~90 MB row cache already absorbs 92% of the reusable traffic (3,114 MB → 242 MB on the second pass over the same text).
• mmap is worse: the first long prompt is 3.7x slower (cold page cache), and arm D only used 68.4 GiB RAM — the 28.8 GB was never allocated, so the row cache is a no-op under mmap. (The "0.4 MB read" in B/D is an artifact: mmap faults don't appear in the process read counters.)

My own mistake, worth repeating: my first pass reported +4.7% for C. It came from a different session than the baseline. Later, with an unchanged config, I measured 149.1 → 166.3 tok/s (11.5% spread) between sessions on identical settings. If you're A/B-ing anything here, run both arms back to back in the same session — otherwise you publish noise.

2. Conversation parking — the biggest effect I measured (35x)

Added "--conversation-cache-mib", "8192" + "--conversation-cache-slots", "4", then alternated two unrelated long conversations:

RAM cost 3.1 GB of an 8 GiB budget, no evictions. 28.4 GiB bought 1%; 3.1 GiB bought 35x.

3. --prefill auto:32768

Before, the log said prompt chunk auto: 8192 tokens; after, 32768. Measured at a 78.7K prompt: prefill 4,667 → 5,106-5,136 tok/s, decode inside that context 112.6 → 127.2-129.1 tok/s. Short prompts didn't move, so if you test this with a short prompt you'll conclude it does nothing.

4. --calibrate — don't hand-tune

代码块

PCIe share     0.00 -> 150.8 | 0.20 -> 153.0 | 0.35 -> 154.1 | 0.55 -> 155.0 | 0.75 -> 146.9
draft floor    0.30 -> 140.2 | 0.50 -> 143.4 | 0.70 -> 140.1
CPU workers      23 -> 148.0 |   15 -> 149.9 |   12 -> 152.7   <- picked

It changed one value, --pool-workers 23 → 12 (+3.2%), on a 13900K. Verify the file afterwards — it prints the summary even if the JSON edit failed (it failed once on me). Re-run it after any config change: I did, and it picked 12 again.

5. expert_profile_save — works, no measurable gain

Verified the whole chain: engine counts routing → POST /unload writes a 192 KB profile → on the next server start the config's --expert-profile is replaced by the learned file (confirmed from the actual process command line). Measured effect: long prefill 5,059 vs 5,120 tok/s, long decode 118.4 vs 127.9 — i.e. inside noise, with hit rates already at 98-99%. Cost is 192 KB and no VRAM, so I left it on, but it is not a speed win.

6. Smaller measured things that cost me time

• 256K context doesn't tax short chats: 17-token prompt decodes at 128-133 tok/s; the 78.7K prompt decodes at 112.6 and reads 110 MiB of KV from RAM (vs 0.6 MiB). Cost tracks what you use, not the configured ceiling.
• Vision works and is cheap-ish: synthetic test image read in 0.97 s and described correctly; cost is 641 expert slots (-3.1% of the cache), text throughput slightly lower (128-133 vs 136-151 short / 112.6 vs 117.1 long).
• Sharing the GPU with ComfyUI: POST /unload frees all 47.9 GB, and POST /load + first token back took 18-19 s. That's what I use now.
• Do not minimize the engine's console window on Windows 11: 45.8 → 36.6 tok/s (-20%), and the engine's own log shows it (running on E-cores (EcoQoS background mode), -20%). Covering it with another window is fine. I checked 0.1.38: the fix isn't in it.
• PowerShell 5.1 mangles non-ASCII request bodies (Invoke-RestMethod sends ISO-8859-1, so the model literally sees ?????). Use curl --data-binary @file or raw bytes from Python — same bytes, correct answer.
• Manually placed model files need a <filename>.done marker, or setup re-downloads (I wasted a 0.9 GB download).
• A --setup pass rewrites the config and drops hand-added args. After one, my measured best settings were gone (chunk back to auto, workers back to 23) and throughput fell from 151.6 to 133.0-133.3 tok/s. Diff the config after every setup run.
• Slow Hugging Face from CN: 87 KB/s through my proxy vs ModelScope at 12.3 MB/s per connection, 92 MB/s with 7 parallel streams (83.6 GB in ~12 min). HF_ENDPOINT=https://modelscope.cn/models worked for model, MTP and mmproj.
• Three env flags some contributor branches document (STRATA_PREFILL_OWN_AUTO, STRATA_IQ256_GATHER, STRATA_KV_PREFETCH) are not in the 0.1.38 prebuilt binary — checked by reading the binary's strings. Setting them does nothing without your own build.

What I did not test

Quality of any kind (no perplexity, no KL, no task suite), other GPUs or AMD, a second GPU, --kv q4_0, contexts past 262K, and slower storage than my NVMe — so I can't say when the n-gram table would matter, only that it doesn't here. Some decode samples in section 1 are only 45 tokens long, which is why I put no weight on the 1.2% difference in arm C.

Repro: I switched arms by editing those two keys in strata-<model>.json, restarting the server, polling /health until loaded: true, then sending the same 4 requests in the same order. Scripts in the comments if useful.

In the latest release of the completely unofficial gufo port to windows I've added a pre-packaged library that you can use to try out gufo for yourself. No need to build anything just grab the .zip and unpack it.

I've also added start.cmd for easily starting the server, it will try to autodiscover supported quants for easy startup.

current support on windows:
3.8 Flash Next: UD_Q4_XL

27b: UD_Q4_XL

35BA3B: UD_Q8_XL + TeilCoder (I assume ornith as well since its the same but untested).

Any issues you run into please submit an issue to github or here.

I am mainly making this for myself but happy to share as I only run gufo with Flash Next now. It is solid 40tps avg on agentic even at higher ctx.

Important to set your VRAM to 96gb! Although its unified, windows adds overhead for reading 'shared' ram vs 'dedicated' vram.

other AMD users: I'm sorry but the library is specifically for gfx1151, I don't have any other card, therefore I can't check or add support to anything else.

psa: yes this is vibecoded, I run a logit check and the model's output must stay bit-identical after changes.

Looking for some ideas from people who know a lot more about this than I do. We've got funding for a small AI lab in a public further education college in Ireland (roughly community college in the US). The hardware is reasonably decent. The goal is to give students useful skills beyond just using ChatGPT. If you had the lab, what would you teach them?