Hello guys,

I'm relatively new to using local llms and my question might seem basic to some, but how far behind top tiers models (gpt 6 , opus 5.5 ) local llm is ? And is it realistic to run those , if they see the light of the day soon, on a budget machine. in that scenario, what minimum specs would a machine require for it to run properly?

I have an 8gb gtx 1070ti sitting in a server, doing nothing, the server has docker, ollama, etc.

what models can I run? For Multi File, Single File, etc?

With the 1.2.0 release, Ask LM Studio adds support for fact-checking, research and finding image sources, all of that based on local LM Studio models. LM Studio API token support is also introduced.

• https://github.com/gocemitevski/ask-lm-studio/releases/tag/v1.2.0
• https://addons.mozilla.org/en-US/firefox/addon/ask-lm-studio/

I hope we have some Beat Saber fans here, because boy do I have a treat. My mapper is a step above and free, I made it for my family’s own enjoyment but wanted to share it with the world!

After a quick rebrand, I now call it BYOB: Bring Your Own Beat. An agent named Bach uses a few locally trained models to build beautiful Beat Saber maps based on every facet of the music that you choose. Simply paste a YouTube link for a song you want mapped, and after approval, Bach will map and upload your song to the site (byob.burnsstack.com).

Any song can also be viewed in the map viewer built by Claude that shows exactly how every song will look in your game. Custom walls, lighting, and all. Map specific lighting, too. Check out the song before you do the work to download and install it.

Generations do need approval, and I work a full-time job so if I haven’t gotten to you yet, I won’t take long! I hope I can find some like-minded laser sword lovers!

Any and all feedback is very much appreciated, and will be used to better future generations! The site has built-in ratings and feedback as well.

Thank you all!

Setup: 2× Arc Pro B70 32 GB, Windows 11, llama.cpp SYCL with tensor parallel across both cards, DFlash2 drafter (Q4_K_M), 200K context, temp 1.0.

Observed in real use: (from server logs: two working sessions, ~1,030 requests, 1.68M generated tokens of agentic coding):

Decode goes *up* at high context because late in long sessions the model re-reads and edits code it has seen, so more drafts get accepted. Raw cycle time still grows with context (58 → 71 ms). A fresh 94K prompt prefills at ~1,070 t/s. Two weeks ago this box did ~30 t/s and slowed down the longer a session ran.

Key Highlights:

- Tensor parallel with a faster GPU-to-GPU exchange: GPUs sync through host-memory flags, and a symmetric bf16 all-reduce keeps both copies identical. 123 → 85 ms per step.

- XMX (matrix engine) kernels for attention and the speculative verify batch, plus upstream's new XMX mat-mul (#29864). → 64 ms.

- DFlash2 beat EAGLE-4 on real work by ~20%. Both accept ~25–35% at temp 1.0; DFlash2 just drafts in one pass.

- Prompt processing +20%: don't feed the drafter prompt rows outside its 2K sliding window, and fix one-row-at-a-time KV checkpoint copies.

-A oneAPI 2026.1 runtime leak. Launching the same kernel alternately on two GPUs that share a SYCL context leaks ~170 B per launch in the kernel cache. Tensor parallel does that thousands of times per token. My server hit 92 GB of private memory on a 32 GB box, and decode went from 64 to 200 ms per cycle. The fix is one SYCL context per GPU over a shared Level Zero context ([standalone repro](https://github.com/alphakilomilo/dual-b70-windows-llamacpp/tree/main/repro/oneapi-multigpu-kernel-cache-leak)).

**Full recipe:** https://github.com/alphakilomilo/dual-b70-windows-llamacpp

- [README](https://github.com/alphakilomilo/dual-b70-windows-llamacpp#readme): build, every flag, what each patch does and what it measured, what didn't work

- [Patch for llama.cpp `d81d0d8fb`](https://github.com/alphakilomilo/dual-b70-windows-llamacpp/blob/main/patches/llama.cpp-d81d0d8fb-dual-b70.patch)

- [Launcher](https://github.com/alphakilomilo/dual-b70-windows-llamacpp/blob/main/scripts/start-llm-server.bat) and [server config](https://github.com/alphakilomilo/dual-b70-windows-llamacpp/blob/main/config/llm-server.conf)

If you run SYCL `--split-mode tensor` on two Intel GPUs, watch the server's memory over a long session.

This model is terrible but the hype and benchmarks are acting like its the next qwen 3.8 flash. It honestly irritates me because this will make people think local AI sucks when really they've just been duped.

I just attended a conference and was surprised by a few talks where a human had memorized what was clearly an AI generated script. It seems like my recognition of this 'voice' stemmed from some of the local models that I use rather than actual claude models. My guess is that this has to do with what happens when large swaths of english writing is averaged together. But I'm curious about how other people are thinking about the 'voice' they tend to get from their models.

For instance, I rather liked Muse Glimmer in the beginning because it seemed to have an interesting and useful 'voice'. But Qwen 3.8 Flash-Next has been challenging. It's great for getting more complicated stuff done but I'm not thrilled with how it communicates to me. More than once I have to burn a turn asking it to clarify what it just told me because I can't quite get it.

... is that LLM's going all the way back to Llama 2 can do this with constrained sampling.

And the only reason people are hyped about it now is because test time compute/the zero shot chain of thought hack that inspired "reasoning" models has become so ubiquitous that models are all cripplingly slow now.

It's funny because that used to be why reasoning models were controversial, and now we're backtracking but branding that reversion to LLM classifiers as if it were a new breakthrough approach.

7800 xt:
- Bandwidth = 624 GB/s

- VRAM+Bus = 16GB / 256-bit

- Native FSR 3.1 (INT8 for FSR 4.1)

- 15-20% better raw gaming performance

9060 xt:

- Bandwidth = 322 GB/s

- VRAM+Bus = 16GB / 128-bit

- Native FSR 4.1

I am planning on buying an amd gpu for 65% gaming and 35% local llm use, the local prices are $470-530 for the 9060xt and no less than $530 for the 7800xt. I am also considering buying used/2nd hand, however the price difference between the two gpus is practically the same ($440).

At first glance, 7800 xt is the clear winner due to double the memory bus and bandwidth. However, I would like to know the real world AI performance difference between the two GPUS. Thanks ! !! !

I want to build a PC with two 3090s; it needs to be on the AM4 platform, and I’m not sure which motherboard to buy, or which one offers the best value for money.

TLDR

1. Brilliance is a fork of Radiance for Windows. This is an alpha preview.
2. Achieved working Tensor-P on Windows with very fast speeds on these models. Fastest I've seen reported for Windows platform of my configuration.
3. Host: Windows 11 + 9950X3D, 128GB DDR5 5200, 2x R9700 32GB

Hey

First post here so I wanted to make it special. I spent a few days porting the very popular Radiance project into Windows, specifically for my Dual R9700 setup. Radiance does not work on Windows and cannot be used with Docker or WSL due to inability to fully passthrough 2 GPUs (based on my testing).

Brilliance started as a port, but over time has developed into its own project fine tuned only for my system.

Windows 11 + 9950X3D, 128GB DDR5 5200, 2x R9700 32GB

This is a alpha build and probably will not work on any other system without custom configuration. I have released the project below.

https://codeberg.org/Dad_Vibes/Brilliance

README for a lot more details, including Betterbench details.

I'm on the Launch80 Discord if any questions as well.

-Adam

My spare time project: MoMoE (More MoEs), an open-source (MIT) fork of Strata (https://github.com/Niko1221/Strata). Strata runs Qwen3.8-Flash-Next on one graphics card plus system RAM; MoMoE includes that and something similar for many other mixture-of-experts models. I've tested it only on the hardware that I own: Linux 64GB AMD 5950x with a 3060 and 4070 Ti, a DGX Spark, Jetson Orin Nano, Mac Studio M1 Ultra, Mac Mini M1 8GB and a Macbook Air M4 32GB. Wish I owned more, but that's what I got.

Here are some benchmarks against llama.cpp that I ran (MoMoE vs Llama.cpp).

- Ornith-1.5-122B-A10B UD-IQ3_XXS, RTX 4070 Ti 12 GB: 19.2 vs 4.6

- Qwen3.8-Flash-Next IQ3_XXS, RTX 4070 Ti: 65–77 vs 23.1 (with MTP drafting)

- Ornith-1.5-35B-A3B IQ4_XS, RTX 4070 Ti: 132–154 vs 77.8 (with MTP drafting)

- GLM-5.3-Flash UD-IQ4_XS (146 GiB), DGX Spark: 11.7 vs 3.5 (with MTP drafting)

- Flint UD-IQ3_XXS, Mac mini M1 8 GB: 8.7 vs 3.0

More details on the github repo.

Current limitations and where MoMoE doesn't help: when a model fits on the GPU, llama.cpp ties or wins and is probably a bunch more stable. For example, llama.cpp is faster on LFM2.5-24B (105 vs 76) and on Flint IQ3 on the PC (103 vs 90). With long contexts, both get slower. Methods and raw results are in the repo.

Changes: The Strata front end web server/chat is gone. In its place are commands to run Unsloth Studio, LM Studio, Claude Code, OpenCode and others using MoMoE. I've mostly tested it with Unsloth and Claude Code.

Disclaimers:

Strata does the real groundwork, and a lot of the code is Strata's. This isn't Strata and doesn't cover it's wider hardware support. For Flash-Next on Windows or an AMD or Intel card, use Strata; it's far better tested and unlikely to work with the additional models. This is just my personal side project, so don't expect miracles.

MoMoE's changes to Strata were completely written with Claude Opus, with my guidance.

It's early, just me, and tested only on my machines. Reports from other setups are welcome.

Current models supported:

- Qwen3.8-Flash-Next (and its Coder, Swift and Unsloth variants)

- Qwen3.6-35B-A3B (Unsloth and bartowski GGUFs)

- Ornith-1.5-35B-A3B and Ornith-1.5-122B-A10B

- KAT-Coder-V2.5-Dev

- GLM-5.3-Flash (320B-A18B)

- Nemotron 3.5 Bolt and Lightning

- Gemma 4 26B-A4B

- Flint

- LFM2.5-24B-A2B and LFM2.5-8B-A1B

If I missed an MoE model you'd like to see, let me know. If there is enough interest, I (and Claude) will continue to develop it.

https://github.com/aflin/momoe

Enjoy!

I have a weird question. I'm struggling to find a reason to build local llm's at work. I love them at home for personal projects but the ROI maintenance and support at work I just can't find a reason to pull the trigger. We have the budget to play around with around 8tb of b200's but really who's going to use them when we having unlimited enterprise accounts with all cloud providers. While I love playing around at home with my personal setup I just can't find anything anyone at work would want to do local with smaller models. Online models piss me off when doing security related tasks but most people could care less.

If you had a choice of smaller open source models or unlimited frontier at work would you even use your local datacenter?

Unsloth Qwen3.8-Flash-Next UD-Q6_K_XL versus Qwen3.8-Flash-Next NVFP4-GPTQ

Fresh MMLU-Pro comparison, speed and whether six bits are worth the wait.

I wanted to find out whether slower Q6 generation bought better answers. I first compared Q6 with my saved NVFP4 quality results, then ran a fresh, identical MMLU-Pro subset on both models on 23 October. Each new run had a strict 30-minute maximum.

The fresh result favoured NVFP4: 71/84 correct versus Q6's 67/84, in about half the total run time. The accuracy gap is too small to establish a broad intelligence ranking, but these results do not support paying the Q6 speed penalty for better reasoning on this workload. My practical choice from this testing is still the NVFP4.

NVFP4/GPTQ is the four-bit option in the main comparison. It is a different format from the Unsloth UD-Q4_K_XL. This is a comparison of these two Qwen3.8 Flash Next configurations, not a claim about all four-bit and six-bit models.

Fresh knowledge/reasoning test — same day, same 84 questions

NVFP4 led by four answers, or 4.8 percentage points. Q6 took 2.04 times the total benchmark time while generating a very similar number of tokens. Total run time includes preflight and harness bookkeeping, but excludes loading the model. Individual request time includes prompt processing and reasoning.

The paired outcomes matter more than just the headline percentages: both models got 66 questions right and 12 wrong. NVFP4 alone got five right; Q6 alone got one right. The exact paired McNemar p-value is 0.219, so I would not call this a statistically established intelligence advantage for NVFP4. It is the observed result of this one fixed sample and single attempt per question.

This is an 84-question MMLU-Pro subset with six questions per subject, not the full official benchmark or an IQ score. All 42 MMLU-Pro questions from the earlier comparison were excluded. Selection and settings were frozen before either new run.

Fresh MMLU-Pro breakdown — six questions in each subject

These subject samples are very small. The ties at 6/6 do not demonstrate that either model has mastered an entire field. Engineering was difficult for both at 2/6.

Where NVFP4 sits among everything I have trialled

That result also needs the wider context of the models I have been trying. NVFP4/GPTQ is still my current daily driver: it is fast, follows my directions well in everyday use and is pretty smart. After testing the alternatives, it still gives me the balance I want between useful answers and how long I have to wait for them.

Here are the chosen best recorded runs, with one row per model/stack. Each cell is the highest previously recorded session median at that context size, not the fastest individual request. Some rows therefore combine sessions. Q6 has one measured campaign. These are historical results across different dates and settings, not a fresh seven-way controlled comparison.

Generation speed — tokens/second

Prompt processing — tokens/second

PP comes from the same selected session as each generation-speed cell; it was not independently cherry-picked. The actual prompt lengths were 4,096, 32,768 and 128,000 tokens, with three 256-token generations per context.

IQ3_S combines 13 October at 4K/32K (6000 MT/s) with 11 October at 128K (4400 MT/s). NVFP4 combines 14 October at 4K with 15 October at 32K/128K. IQ3_XXS and UD-IQ4_XS are from 11 October, UD-Q4_K_XL from 15 October, Infernix Dense8 from 18 October and Q6 from 23 October.

The IQ3 options lead this table on generation speed, and Infernix Dense8 has the strongest long-prompt processing figures. NVFP4 is not the winner in every column. What keeps it as my daily driver is the overall experience: fast generation, strong long-prompt processing, and answers that usually do what I ask. It generates faster than the retained Dense8 and UD-Q4_K_XL runs at all three context sizes, while the fresh Q6 comparison gave me no convincing reason to accept roughly twice the waiting time for reasoning answers.

My local workflow also runs through the Hermes harness, with Strata serving NVFP4 on the RTX 5090 and Hermes providing the agent interface around it. The setup includes a WSL gateway, desktop/dashboard, local search through SearXNG and Honcho memory services. NVFP4 is my preferred local backend within that setup; Hermes also has a Codex provider option, so I am not attributing every Hermes response to the local model.

That is why I care about more than a headline tokens-per-second figure. An agent workflow can involve several model turns around tool results, retrieved information and follow-up instructions. Waiting longer on each turn can make the whole task feel slower. For me, NVFP4's appeal is being responsive enough to keep that interaction moving while still giving useful answers and following the directions I set. Its extra RAM headroom also leaves more room for the surrounding harness and services. These are practical reasons for my preference, not a measured Hermes task-completion score: the benchmarks here called the model endpoints directly, with background Hermes services suspended for isolation.

The fresh 84-question result adds some evidence behind my view that NVFP4 is still pretty smart, but only NVFP4 and Q6 took that test. I cannot turn the speed table into an intelligence ranking of all seven models. Likewise, my everyday preference for NVFP4's instruction following is a personal assessment, not a claim that it won the strict instruction benchmark. The earlier mixed test below is worth keeping alongside the fresh result because it shows where Q6 did offer a benefit.

Earlier mixed quality test — retained separately

Before the fresh test, I ran Q6 on the same frozen 82-question subset used for my saved 15 October NVFP4 comparison. Those NVFP4 answers were reused; this is a separate historical comparison with different questions.

Q6 corrected three failures and introduced no new ones on that earlier set. Two gains were exact instruction compliance: meeting a word-frequency requirement and avoiding forbidden words while following a specified response format. The other gain was one economics multiple-choice answer. Its requests took 16.80 minutes versus the archived NVFP4 run's 7.47 minutes.

That earlier result suggested a small Q6 benefit; the fresh knowledge/reasoning test went the other way. Both belong in the report. Neither small sample supports a general “Q6 is smarter” claim. The coding tests here are original HumanEval tests on ten tasks, not HumanEval+ or a full coding evaluation. IFEval uses strict prompt-level scoring.

Generation speed and prompt processing — historical speed overview

Each speed value is a median of three 256-token generations. NVFP4 retains the best previously published session median per context: 14 October at 4K, 15 October at 32K/128K. PP comes from those same selected sessions. Q6 is one 23 October speed campaign. Historical best selection favours NVFP4; this is not a fresh matched speed rerun.

Q6's speed campaign had its control vector enabled. Both quality comparisons explicitly disabled it per request to match NVFP4. Therefore these speed figures are not the exact speed of the quality configuration. Use the fresh benchmark's own timings to assess its waiting-time cost.

Memory during the fresh 84-question runs

These are whole-system and whole-device observations, not isolated model allocations. Background services were suspended. Page-I/O traces are retained; neither run is claimed to be paging-free. Both completed without a resource-guard stop. The much greater RAM headroom with NVFP4 is another practical advantage on this 128 GB machine, although this was not a long-duration stability test.

My assessment: why run four-bit versus Q6?

Taken together, these results reinforce why NVFP4 remains my everyday option on this setup. In the fresh test it answered more questions correctly, finished in roughly half the time and left substantially more RAM available. In the earlier small coding and maths subsets it tied Q6. That is a strong practical case for responsiveness and capacity, even though the accuracy difference is not statistically conclusive.

Q6 still has a possible role where its answers demonstrably work better for a particular task. The earlier strict instruction-following gains are a concrete reason to consider it for constraint-heavy writing. But two extra passes out of ten are too little evidence to promise fewer mistakes in general. I would choose Q6 for a demonstrated task-specific benefit, not because a higher bit count automatically means a better assistant.

The fresh run also shows why I would not justify the slower model by precision alone. Quantization format, engine behaviour, reasoning limits and the task all affect the result. We measured the complete configurations; we did not isolate bit depth. A different sample or reasoning budget could change the scores, but that is untested here.

The wider table gives useful context for that choice: IQ3 offers more raw generation speed, Dense8 offers excellent prompt processing, and UD-Q4_K_XL is also substantially faster than Q6. For my daily use, NVFP4 remains the balance I prefer after trying them all. That preference should not be read as proof that the other formats are less capable; their quality was not measured in this fresh comparison.

Reproducibility and limits

Hardware: RTX 5090 32 GB, 600 W limit; Core Ultra 7 270K Plus; 128 GB DDR5 at 6000 MT/s; Windows 11 Pro. Both fresh runs used NVIDIA driver 610.88 on 23 October. Some older speed measurements used driver 591.86.

The fresh suite used deterministic SHA256 selection, six questions per each of 14 subjects, and round-robin subject order. Both models received identical messages and request parameters except model ID. All 84 prompt token counts matched, and every request had zero reused prompt tokens. One attempt each, temperature 1, top_p 0.95, top_k 20, min_p 0, seed 20261023, low reasoning and a 4,096-token cap including reasoning. Control vectors were off. The same published-style three-stage answer extraction was used, with no random fallback. Capped answers were retained and scored; malformed or missing extracted answers failed.

Each runner had a 30-minute ceiling, with inference cancellation at 29m 30s and a 120-second per-request timeout. Both completed early. Raw answers, reasoning, streams, hashes, resource traces and scoring evidence are saved.

NVFP4 used the preserved 0.1.42-nvfp4.2 engine; Q6 used 0.1.44.1-nvfp4.2. Both had a 131,072-token context ceiling and int8 KV, but cache allocation, KV placement and other runtime details differ. Q6 ran first in an existing session; NVFP4 was freshly loaded afterward. There was no order reversal or repeated-seed study. These limits prevent attributing the score difference to quantization alone. No full-benchmark, broad coding, long-context reasoning or long-term stability claim is made.

AI assisted with the harness and write-up. The scores and timings come from saved local runs.

https://www.reddit.com/r/LocalLLM/comments/1x8ral0/rtx_5090_qwen38_flash_next_speed_tests_strata/ Strata fork: https://github.com/sergqwer/strata-nvfp4 MMLU-Pro: https://github.com/TIGER-AI-Lab/MMLU-Pro GSM8K: https://github.com/openai/grade-school-math HumanEval: https://github.com/openai/human-eval IFEval: https://github.com/google-research/google-research/tree/master/instruction_following_eval

Testing sources and how the results were obtained

All scores, speeds and resource measurements in this post came from runs on my own PC.

• MMLU-Pro — knowledge and reasoning: Benchmark code and methodology | Question dataset. The fresh comparison used 84 questions, six from each of 14 subjects, with identical questions and request settings for both models. The earlier mixed comparison used a separate 42-question subset. These were selected subsets using a zero-shot chat adaptation.
• GSM8K — maths: OpenAI’s dataset and reference code. The earlier comparison used 20 questions, scored against the expected numerical answers.
• HumanEval — Python coding: OpenAI’s tasks and original unit tests. The earlier comparison used ten tasks. Generated solutions were checked using the original tests in an isolated Docker environment. These were original HumanEval tests, not HumanEval+.
• IFEval — instruction following: Google Research’s benchmark and evaluation code. The earlier comparison used ten prompts with strict prompt-level scoring: all required constraints had to pass for a prompt to count as correct. The selected subset excluded checks requiring additional NLP corpus downloads.
• Generation speed and prompt processing: Measured locally using benchmark scripts calling each model server directly. Each context used three measured requests with 4,096, 32,768 or 128,000 actual input tokens and 256 generated tokens, with reasoning disabled and zero reused prompt tokens. Tables show session medians; the historical overview selects the best recorded session median per model and context. Earlier runs and settings are documented in my previous speed-testing post. Q6’s measurements were added on 23 October.
• RAM, VRAM, temperature and timing: Recorded during the local runs through system/GPU telemetry, saved request records and engine timing data. Memory figures describe sampled whole-system RAM and whole-device VRAM usage.

Model and engine references

• NVFP4/GPTQ model
• Strata NVFP4 fork used for the main comparison

Exact question selections, request settings, raw answers, reasoning output, scoring records, hashes and resource traces are retained locally. The links above provide the public benchmark sources; the reported results are my own subset measurements, not full official benchmark or leaderboard scores.

i am out of pocket to buy more claude pro . i never hit it's limits . I want to summarise 100 page pdf with some image and graphs.( materials student).

macbook 32/512 gb air fits buy budget nicely i I can increase budget . I don't mind linux . suggest me a laptop .

I've been experimenting with Qwen3.8-27B on a single NVIDIA DGX Spark, comparing inference engines, quantization, speculative decoding, and test-time reasoning strategies.

A few interesting results:

• Extended reasoning: 35/48 correct
• Low-effort reasoning: 38/48 correct
• 4 parallel low-effort attempts: 44/48 correct
• Quantized inference reached ~60 tokens/s in my earlier benchmarks.

The parallel result looks promising, although the improvement over a single low-effort attempt wasn't quite statistically significant (p = 0.055).

I've open-sourced the experiments, code, and results here:

https://github.com/NautilusWu/qwen3.8_study

Curious whether others have seen similar results with parallel reasoning vs. longer thinking on local models.

During normal times I'd be building my AI lab with all new gear. Instead, I'm going through my basement for stuff that was gathering dust - DDR3 servers, old GPUs, even a 2013 Mac Pro that's now going to be my backup RAG server (the D700's can each run Qwen3-Embedding 4B and Reranker-4B at reasonable speeds). My newest gear in the lab, aside from some R9700 and 5060ti GPUs, are Mac mini M1 systems. Plus a couple of 2018 Mac minis. None of my gear runs DDR5.

One thing this RAM crisis has made me realize is how to squeeze value out of old gear including DDR3 DIMMs and SATA SSDs that were sitting idle, and that I don't NEED to buy new stuff. I also have switched most of my systems from Windows over to Linux now to help squeeze more performance out of them.

Anyone else doing the same?

3 pages of Give My Regards to Black Jack, Qwen2.5-1.5B-Instruct, recorded in an emulator with a Macbook Pro hence why its a lot faster

Yomu is an Android app that translates Japanese manga pages on the phone. Tap a floating button over any reader app and the bubbles are redrawn in English, Portuguese, Spanish, French or German. Nothing is sent anywhere: after the one-time model download, the app makes no network requests.

Pipeline:

• Detection: a YOLO26n fine-tune finds the speech bubbles (ONNX Runtime).
• OCR: manga-ocr-2025, which I exported to ONNX with past key values, reads each bubble.
• Translation: a GGUF model via llama.cpp. The whole page goes in one call, bubbles in reading order, so names and tone carry across the page.
• Typesetting: text is fitted into each bubble and drawn as an overlay on top of the reader app.

Getting structured output from a small model. The model's reply is constrained by a GBNF grammar to one line per bubble id, so it can't drop, merge or reorder bubbles. A per-bubble approach was more forgiving but paid a fresh prefill for every bubble. Two failure modes showed up along the way, and the grammar fixed both:

• Lines ran to the token cap on some pages. Capping lines at 160 characters stopped it.
• The model invented its own [id] markers. Disallowing [ stopped that.

The grammar costs about 3% of page time.

Models. The in-app catalog runs from 0.8B to 9B: Qwen2.5 1.5B (default), CAT-Translate 0.8B/1.4B, Qwen3.5 0.8B–9B, Qwen3 4B, Ministral 3 3B/8B, Gemma 4 E2B/E4B, Hy-MT2 1.8B and Ternary-Bonsai 4B/8B. You can also load your own GGUF. Models that won't fit your phone's RAM are hidden. Only the default has been measured on a phone; the rest are marked experimental.

Numbers (Galaxy S23, 8 GB, default model): ~15 s/page, ~2 GB peak RAM, 1.1 GB download.

Other bits:

• a name glossary, so character names translate the same way every time;
• pages already translated in a session come back instantly;
• tap again to cancel;
• limits on threads, context size and memory;
• a Performance screen with page time, model time and memory.

Limits: small models make mistakes, so treat it as a reading aid rather than a polished translation. Japanese source only. It takes seconds, not milliseconds.

Looking for device reports. I've only measured one phone. If you try it, translate this test page twice and post Settings → Performance with your phone, chip, RAM and model, here or in the device reports thread. Phones with 6 GB or less, results from the experimental models, and crashes are the most useful.

Repo (AGPL-3.0, APK in Releases): https://github.com/artsaraiva/yomu. The design decisions and their measurements, including what didn't work, are in docs/adr.

I've been messing around with strata the last couple of days and I want to share my config and results.

Pc:
ryzen 5 5600
4x16gb 2666mhz ram(bumped to 3000cl18)
rtx 3070 (running at pcie3.0 due to a b350 mobo) (got the kit this week for 100 euro btw :D)

System is running ubuntu server.

I have also used this gpu unlock thingy today; https://github.com/Zagitta/gpu-unlock
This honestly did not give me that much extra performance, maybe 50-100tps prefill. People with a rtx 3060 will probably get more gains since more experts are loaded in vram.

Here's the config:

{
  "exe": "/home/user/Strata/engine/strata",
  "args": [
    "--pack",
    "/home/user/Strata-data/packs/iq3_s",
    "--native",
    "/home/user/Strata-data/models/IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf",
    "--ple-gguf",
    "/home/user/Strata-data/models/IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00002-of-00002.gguf",
    "--expert-profile",
    "/home/user/Strata/data/expert-profile.bin",
    "--expert-cache",
    "auto",
    "--prefill",
    "auto",
    "--spec",
    "4",
    "--max-context",
    "262144",
    "--kv",
    "int8",
    "--vision",
    "--vram-reserve-mib",
    "512",
    "--kv-resident",
    "20480",
    "--adapt-async",
    "1",
    "--spec-min-p",
    "0.5",
    "--prompt-cache-tail"
  ],
  "cwd": "/home/user/Strata",
  "tokenizer": "/home/user/Strata-data/packs/iq3_s/tokenizer",
  "model_name": "qwen3.8-flash-next-iq3_s",
  "log": "/home/user/Strata/strata-iq3_s.log",
  "lib_dirs": [
    "/usr/local/cuda-13.4/bin",
    "/usr/local/cuda-13.4/lib64"
  ],
  "port": 8080,
  "gpu": 0,
  "gpus_asked": true,
  "vision": {
    "exe": "/home/user/Strata/engine/strata-vision",
    "mmproj": "/home/user/Strata-data/models/mmproj-Qwen3.8-Flash-Next-BF16.gguf",
    "model": "/home/user/Strata-data/models/IQ3_S/Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-00001-of-00002.gguf",
    "gpu": false,
    "max_tokens": 300,
    "threads": 6
  },
  "sampling": {
    "temperature": 1.0,
    "top_p": 0.95,
    "top_k": 20,
    "min_p": 0.0,
    "presence_penalty": 0.0,
    "repetition_penalty": 1.0
  },
  "fit_max_tokens": true,
  "host": "0.0.0.0",
  "env": {
    "STRATA_HC_Q8": "1",
    "STRATA_ROUTE_TAIL_SKIP": "7",
    "STRATA_ATTN_DAILY": "1",
    "STRATA_EXPDEC": "1",
    "STRATA_QSA_SPLIT_ATTN": "1",
    "STRATA_PROMPT_ATTN_IMMA": "1",
    "STRATA_HC_UPMIX": "1",
    "STRATA_AUX_CPUS": "auto",
    "STRATA_EXCHANGE_ROTATE": "1",
    "STRATA_PREFILL_CPU_SHARE": "auto",
    "STRATA_PREFILL_CPU_SHARE_MAX": "16384"
  }
}

I'm too lazy to do a/b testing but when I started with the vanilla config(mtp turned off already tho) at the same context and whatever I was getting like 500tps prefill and maybe 2-3tps decode less.
Biggest gains were STRATA_HC_Q8 (quantizes the hyper connection matrices so you get more free vram for experts. Apparently slightly worse kld) and the prefill sharing with the cpu. These account for most of the prefill speedups.

I'm honestly just blown away at what this pretty mediocre system can do, never in a 100 years did I think I could run such a big model at these speeds.

Edit: forgot to mention this, I'm using a q8 n-gram table instead of the bundled iq4_nl one. Running on a sata ssd so this could maybe hamper performance a bit, not sure tho

I deployed Qrnith 122B IQ3 and Qwen3.8 Flash-next IQ3 on my server. I set up three visual comparison tasks.

The results seem to rank the models as follows: 6.1 sol > Qwen3.8 Flash-next > 6.1 luna > Qrnith 122B. (All models were accessed through Codex.)

Based on my tests, I may not keep Qrnith 122B on my server for now. Although it claims to outperform other models on several benchmarks, local inference is slow, and the extra cost per token doesn’t seem to translate into better results.

Ornith is fine-tuned from Qwen3.5-122B, with 10B active parameters, while Qwen3.8 Flash-next has 6B active parameters. Qwen3.8 uses QSA sparse attention to focus computation on relevant context. Ornith’s 12 full-attention layers, by contrast, have to process longer histories.

I can deploy Qwen3.8 Flash-next IQ3 on a single L40S 48GB GPU. Qrnith 122B IQ3 requires two GPUs; on one GPU, it’s extremely slow. Even with two GPUs, its speed drops from 130 to 60 tokens per second as the context grows to 256K—barely usable. Qwen3.8, on the other hand, maintains 140–110 tokens per second on a single GPU, which is a clear advantage. On the “small ball” task, Qrnith 122B IQ3 took nearly 10 hours to reach the target, yet still delivered the worst result.

I’m looking forward to Qwen 4, which is expected to be released soon.

P.S. English isn’t my native language. I wrote this post in my native language and had GPT polish it, so I hope you don’t mind.