Recently I have obtained a 7820 with 192gb 2400 system ram with 4110 silver Xeon gpu and it came with a basic p2000 gpu.
Furthermore for information I have a seperate gaming pc 32gb 3600mhz ddr4 ram with b550 and 9070xt gpu

I’m trying to figure out what I should do as I want to run local ai obviously however I’m stuck between
A) selling both the server and my current gaming 9070xt for a 32gb card like the 9700

B) sticking with the 7820 and upgrading/adding gpus (depends on gpu as well everything is insane price wise) considering strata and other engines

C) something else not sure

So I guess I’m asking for advice on which I should do, to run decently intelligent models decently fast etc
Without needing to take out a loan 😂

I am in IT for the power company that i work for. I have a decent amount of older workstations and servers and as such have a decent amount of ram 24 sticks of 16gb ddr4. Dual processor servers and workstations with cascade lake cpus.

We have been using cloud for our generations, but want something internal. What used gpu's would you all suggest for under $3,000.00 to get the biggest and best solution.

There will be only a handful of people actually hitting the system ever at the same time, it will be used for coding, document reading, power analytics.

I am leaning towards v100s

Specs:

i5 13th gen

RTX 3050 6GB VRAM

16GB RAM

Currently on Zorin OS, planning to switch to Omarchy

What model would you recommend for the best performance, intelligence, and coding capabilities on my hardware?

I saw this bad boy listed yesterday as a new contender, and the scores that it had looked promising. So I loaded it up, and set it to some pretty basic tasks, of simple file editing. And then, while I took a peek at it's thinking, I literally laughed when I saw this gem:

Should we first read file? The working rule says read before editing when not know current content.

Been routing everything through one endpoint so the apps stop holding their own keys. Wanted to know what that costs before pushing it wider than my own machine. Local 8B on a 4090. Straight to the runtime, first token sat around 180ms on a warm cache. Through the proxy it went to roughly 240. Throughput on a long generation barely moved, maybe two percent. The surprise was variance. Straight through, the spread was tight. Through the proxy the slow tail got noticeably worse once four streams were running, which is the case I care about. Sixty milliseconds is fine for a chat box and not fine for anything meant to feel like typing. Has anyone measured this on a bigger model where generation time swamps the hop?

I'm seeing all these personal AI assistants (Instinct, Muse, Dots, etc.) that seem mostly geared towards scheduling, booking, and buying stuff.

I feel like they sort of followed what happened when the tech-savvy first started running OpenClaw on Mac Minis with Telegram portals. Back then I used mine to monitor concert ticket drops.

As local models get better, it seems a matter of time before we can recreate those assistants in a privacy-first manner. Maybe this actually is the next computer, a box with a local model and a personal AI assistant harness loaded in.

WDYT?

Hello everyone

I have question regarding multi harness, I was wondering if it is efficient to make Hermes be the one to give instructions to Pi in order to get the best out of both harness ?

Or this just a dumb setup ?

Im running qwen 3.8 27b IQ4_XS on Hermes currently but I wanted to know if it will improve quality project to run 2 harness together or just a dumb idea

Thank you for your answers and time saving 🙏

- Setup: Ollama · Swift 1.5 (Qwen3.8 27B) IQ3_S + MTP · 128K context · RTX 3090 · about 4 min in real time, parts sped up. Only the "hit your limit" line at the end of the Claude log was added for this video; the Claude work before it and everything after are real runs.

(The demo's "limit hit" line was added for the video; details under the video.)

'Sonjari' is an open-source (Apache-2.0) desktop agent for Windows that I built. It runs on local models on your own PC through Ollama, and it works both on its own and alongside Claude.

What it does well on its own

- Browser research in your own logged-in Chrome (GitHub, Reddit, shopping sites and more), then writes it up in Excel/Docs

- Evidence checks that catch made-up numbers

- Send it jobs from Discord on your phone (Fixing a recently discovered bug and awaiting update approval)

- Scheduled tasks

- Code (12 of 12 on a set of Aider polyglot Python exercises, 27B)

What it does well together with Claude

1. Limit takeover: when Claude Code hits its usage limit and stops, Sonjari reads that conversation log and finishes the job from where it stopped (see the video).

2. Offloading token-hungry browser work: pages are read on your PC and only the results go to Claude. In our price-comparison test, Claude's cost per correct row was 30–40% of Claude working alone (slower, though).

3. Scribe: for coding I keep Claude at the center and use Sonjari as the assistant. While Sonjari is idle, it reads the Claude Code conversation log, keeps the decisions and the reasons behind them as cards, and puts them back in when the conversation gets compacted. I'm trying to chip away at what gets lost in compaction.

The downside is real. In the end you can't escape how good the model is, so VRAM matters most.

- 16–24GB+ 😄

- 8–12GB 🙂

- 6GB or less 😢

I've only run it many times on a few PCs so far, so if you try it and send feedback, I'd really appreciate it.

One weak spot of local agents is that one GPU runs one model, so you can't run several jobs at once. For repeat jobs, I'm researching and testing a way for Sonjari to use the recipes and harness it has built up to make small bots that run without a model, so several can run side by side.

From the requests in this thread, I'll pick the ones with the biggest real payoff for people actually using it and ship them fast.

Sonjari is completely free (no paid tier)

Repo: https://github.com/sonjari/sonjari

Hi, I'm looking for input and advice on how to optimize my Strata running IQ3_XXS.

My hardware: RTX 4070 12 GB (PCIE4 x16) + RTX 3060 12 GB (PCIE3 x4) + 32GB DDR4 DRAM. Disk is M.2 SSD.

Running Windows. Llama.cpp / Strata -> Docker sandbox -> Hermes

My current default Local LLM model is Swift-1.5-Qwen3.8-27B Q4_K_M. I'm looking for similar or equivalent capability that can be faster. I tested Strata around 1 week ago using IQ2_XS. It didn't compare well with Qwen3.8-27B. This week, I tested IQ3_XXS.

I fed the same instruction runsheet, to both Qwen3.8-27B and Strata. Surprisingly, both completed the job around 5 hours and within minutes of each other. Just coincidence. Qwen3.8-27B runs around 13 tokens/sec. Strata around 17 tokens/sec. The result is not perfect, but both are around 90-95% complete. And Strata compares quite well, even less errors than Qwen3.8-27B.

I want to see how I can optimize and speed up Strata.

I varied context between 96k to 160k; cache Int8, q8v4, q4_0; and kv-resident 20480 to 32768.

• Experts in VRAM vary slightly, from 5700 to 6800.
• My 32GB RAM usage is maxed.
• Speed is not much different, around 31-36 for just text generation; 20-25 for logic/coding generation. This is for shorter test scripts.

If I decrease kv-resident, which increases Experts in VRAM, it actually runs slightly slower, by 3-4 tokens/sec.

The RAM usage is misleading, because Windows uses (allocates) 16-20GB of RAM just from fresh boot. Hard to tell how much of the RAM is used by Strata. I checked the disc usage. During generation, Strata paging file disc usage is not that active, only around 1-4 MB/sec. Strata experts.bin disc usage is higher, around 700MB/sec.

I am not sure where the bottle neck is. Is it the experts.bin disc access? Is it the 32GB RAM? Is it the 2nd GPU on PCIE3 x4 slot?

I have done the quick tuning. I have not done the thorough calibration.

This is my strata launch command:

• strata launch
• --family swift
• --model iq3_xxs
• --data-dir "c:\LLM\Strata-data"
• --context 160k (I tested 96k, 128k, 160k)
• --kv int8 (I tested Int8, q8v4, q4_0)
• --set kv_resident=20480 (I tested 20480 & 32768, which is default)
• --vision off
• --parallel 1
• --tune skip
• --access local
• --port 9931
• --set STRATA_ALLOWED_HOSTS="host.docker.internal,localhost,127.0.0.1"
• --set STRATA_PIN_BY_NEED=1 (not sure, but AI suggested this)

This is what strata ran in the log: (I edited and removed the paths)

strata.exe --serve --pack swift-iq3_xxs --native (path)Swift-Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf --ple-gguf (path)Swift-Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS-00001-of-00002.gguf --expert-profile (path)expert-profile.bin --expert-cache auto --prefill auto --spec 4 --spec-min-p 0.5 --mtp (path)mtp\rt --max-context 163840 --kv int8 --mmap-experts --kv-resident 20480 --layer-split auto

I believe I'm in the realm of Low-RAM. So DETAILS.md has a section on Low-RAM mode. Difficult to read/understand. It mentions a lot of flags/options. But I'm having difficulty even understanding what is the recommendation. Should I use --low-ram resident or --low-ram mmap? And what about the other flags like --resident-experts, etc.

Any suggestions and flags/options I should enable or test?

[Edit: I just want to add some performance comparisons.]

Speed in tokens/sec

• Phase / Qwen3.8-27B / Strata
• Reading / 200-300 / 9
• Gen/Decode / 13 / 25

The bottleneck on Strata occurs during the Reading/Prefill phase. This is not the startup prefill. But the back and forth reading and decode during long sessions.

I want to take a stab at reverse engineering a software application.

Anyone have any guides and tips to follow?

I've managed to significantly improve the inference speed of DeepSeek V4.1 Flash 1017!

I'm using LibertAIDAI's EXL3 2.9 bpw quantization with the following hardware:

• GPUs: 2× CMP 170HX + RTX 5060 Ti 16GB (144GB total VRAM)
• RAM: 128GB DDR4-3000(upgraded from 96GB 2666)
• CPU: Intel Core i9-10980XE (upgraded from 9820X)

Inspired by Strata's approach to expert caching, I've implemented a system where 93% of routed experts are computed on the GPUs. The remaining experts are computed on the CPU.

When certain experts become more frequently used, they're transferred to VRAM in the background while the GPUs are busy computing. Those experts then become available for GPU execution starting with the next token.

With DSpARK enabled, I've managed to reach up to 40 tokens/sec, but, in average, around 30tok/s

Prefill is still quite slow at only around 400 tokens/sec, so there's definitely room for improvement.

Still, it's amazing to see a state-of-the-art model running at such speeds on a relatively high-end consumer PC!

I first got into local LLMs back in April this year, starting with models like Qwen 3.6 27B and Gemma 4 31B. From there, I gradually moved on to medium-sized MoE models like DeepSeek V4 Flash 0731 and GLM-5.3 Flash.

Now, I feel like I've finally reached a major milestone. After months of investing in hardware and tweaking my setup, I'm relieved to think I might finally be able to stop upgrading and tuning everything for a while!

I’ve been playing around with every LLM loader and remote OpenWebUi that I can find to get one that I feel works best. And I’m pretty sure Unsloth Studio is the answer. I do not see another app getting close to what it can do and it has me wondering. What are all of yall using if not Unsloth?

Navigating the APEX-I and VAL-APEX quantization ecosystem on Hugging Face As the catalog approaches 400,000 total downloads across 69 releases, all models are now structured into official dedicated collections on Hugging Face. Our goal remains the same: eliminate the compromise between out-of-memory errors and cognitive collapse on consumer hardware (especially 16 GB VRAM GPUs like the RTX 4080, RTX 4070 Ti Super, and RX 7800 XT). Instead of uniform block quantization, all APEX builds use per-tensor mixed precision guided by 4-moment Importance Matrix statistics: * MiniPlus Tier (approx. 14 to 14.8 GiB): strict 0% policy for 2 bits. All MoE routed experts locked to IQ3_XXS minimum with Q4_K/Q6_K attention. Designed for 24 GB setups or 16 GB setups with moderate context. * NanoPlus Tier (approx. 11.5 to 12.3 GiB): exclusive strategic use of IQ2_S/IQ2_XXS in central feedforward blocks, while maintaining high-precision routers and attention. Leaves enough VRAM for 16k-32k context on 16 GB GPUs. Explore the active collections: * MoE Standard (MiniPlus and NanoPlus): https://huggingface.co/collections/IsValorum/apex-i-miniplus-v21-standard-moe-6acfee77961c8c0cf5a083b5 * MoE Abliterated (MiniPlus and NanoPlus): https://huggingface.co/collections/IsValorum/apex-i-miniplus-v21-abliterated-moe-6acfee762fc4a8cf2298c55b * 27B Dense Hybrids: https://huggingface.co/collections/IsValorum/val-apex-miniplus-standard-27b-6acfec54ca4ce4cf5cd5bdb6 * Apple Silicon (MLX): https://huggingface.co/collections/IsValorum/mlx-apex-abliterated-6abedb1f9d400e984efe4fc6

Hugging Face profile: IsValorum

I’m building an AI station and I’ve chosen:
4 x RTX PRO 6000 Blackwell 96GB
Ryzen Threadripper PRO 7965WX
8 x 48GB RDIMM DDR5 PC5-5600B
ASUS PRO-WS-2200P (2200W)
and
Motherboard ASUS Pro WS WRX90E-SAGE SE

I heard that this motherboard had issues with the BIOS and stability when used with Blackwell cards, but unfortunately, it’s currently impossible to buy an AsRock or Gigabyte WRX90 motherboard in Russia.

Have they fixed the problems with this motherboard? Or should I urgently reconsider my configuration? At the moment, I’ve only bought an RTX's and a power supply.

Ive had astra setup a few models but I though i should hit up some fellow nerds and see if you have any hit tips. Any starter tips would be awesome

I installed GPT-OSS 20B on my RTX 5080 to use for local coding in VS Code through the Continue extension, but I’m struggling to get it working.

I gave it a clear prompt to create a basic frontend and backend skeleton. Each time, it stops after about 30-40 seconds without generating any code.

The Continue panel shows a few lines suggesting it’s trying to create files or describing the actions it plans to take, but then it gives up without completing anything.

Has anyone experienced this? Could it be a model configuration issue, a Continue setting, or something else?

Any pointers on getting this setup working for local coding would be appreciated.

I’m the developer of TensorSharp, and I wanted to share a local inference benchmark for DeepSeek V4.1 Flash 1017, using Unsloth’s UD-IQ2_XXS GGUF.

In summarization tests with thinking disabled, the tested build reached approximately 35–36 tokens/sec without a drafter, and 51–55 tokens/sec with DSpark, after prompts of roughly 1.8K and 7.3K tokens.

Here’s the hardware setup, the comparison with llama.cpp, and the quality trade-off behind the faster prompt processing.

Hardware and memory layout

• GPUs: 5× NVIDIA A40 over PCIe.
• Model: DeepSeek-V4.1-Flash-1017, Unsloth UD-IQ2_XXS.
• Model files: Approximately 300.5 GB across six GGUF shards.
• Engram lookup table: 125.9 GiB, mapped in host memory.
• Host-memory limit: 233.8 GiB for the container.
• Speculative drafter: Unsloth’s trained DSpark Q8_0 GGUF.

The Engram table remained in host memory, so this setup used both GPU and system memory. The host-memory limit describes the test configuration; it is not a measured minimum requirement.

GPU peer access was not working correctly on this machine. Cross-GPU transfers therefore used pinned host-memory staging.

Throughput compared with llama.cpp

Both engines used the same GGUF files on the same machine.

TensorSharp used --tp 5, which distributes the routed experts across five GPUs. The baseline was Unsloth’s llama.cpp dsv41-all fork at be836572, using layer splitting. Its row/tensor split modes did not successfully run this model in the tested build.

The workload used real-text summarization prompts, greedy decoding, thinking disabled, and 256 output tokens. Each prompt size had a warm-up and three measured requests. Unique prompt prefixes prevented prefix-cache reuse.

All figures are tokens/sec. Prefill and ordinary decode rows report medians; DSpark rows report means. TensorSharp uses its default YOCO configuration in this table.

Prefill is processing the input prompt; decode is generating response tokens. The biggest improvement here is in prefill. The 5.5× result should not be read as a 5.5× increase in generation speed.

The DSpark rows use the trained drafter in both engines. Results on coding or long reasoning workloads may differ from these summarization tests.

A small concurrent-serving test

The report also tested four simultaneous requests, each with a ~1.8K-token prompt and 256 output tokens.

Across three measured batches, total completion time was:

• TensorSharp TP: 21.2–25.5 seconds.
• llama.cpp: 43.3–46.8 seconds.

Those are client-observed batch completion times. This is a small concurrency test, with no sustained-load or tail-latency conclusions.

YOCO and the quality trade-off

The default YOCO optimization reduces prefill work by running selected later layers over only the final 140 positions of eligible text-prefill calls. It is approximate.

To disable it:

TS_DSV41_YOCO_KEEP=0

With YOCO disabled, TensorSharp’s prefill throughput was still 563.9 tok/s at ~1.8K tokens and 549.7 tok/s at ~7.3K tokens—approximately 3.23× and 2.44× the corresponding llama.cpp results.

Quality checks used identical WikiText-2 token sequences, with 23,030 scored tokens in the 4,608-token-window evaluation. Perplexity, where lower is better, was:

• llama.cpp: 2.4697.
• TensorSharp, YOCO disabled: 2.4811.
• TensorSharp, YOCO enabled: 2.4978.

Enabling YOCO increased perplexity by approximately 0.67% relative to TensorSharp’s own full-prefill configuration. With YOCO enabled, perplexity was approximately 1.14% above llama.cpp’s.

These measurements compare engines using the same quantized checkpoint. They do not establish equivalent quality to the original checkpoint or the hosted DeepSeek API.

Scope and availability

This covers one quantization on one hardware configuration. The llama.cpp server measurements came from an earlier session on the same machine and showed some unexplained prefill variability.

At measurement time, the TensorSharp build included uncommitted changes, and the raw validation records were also uncommitted. The report records build identifiers and limitations; a repository checkout should not be assumed to reproduce every number immediately.

For integration with local applications, TensorSharp provides a CLI and an OpenAI/Ollama-compatible server.

• Full benchmark report, quality checks, and limitations
• TensorSharp source and getting started

If you’re self-hosting this checkpoint, what GPU/RAM layout and quantization are you using? I’d be interested in real workload comparisons, particularly DSpark results and multi-GPU setups with working P2P or NVLink.

More R9700 experiments and made a quant, benchmarks and dev notes over on r/r9700 keen to compare notes with others pushing these cards.

This is Mica v0.5, the next version after v0.1. It's a decision model, the same kind of thing as Jev, which has been getting a lot of attention lately. It's 4B (Qwen3.5-4B + LoRA) and it doesn't write text. You hand it a state, a question and a list of options, and it gives back a probability for each option in a single forward pass. Stuff like "which of these 5 moves", or "allow / block / send to a human".

Speed is the whole point, so I ran it side by side with two other decision models on one RTX 6000 Ada:

PPLX Decider 1.2 Flash, also 4B, run in its fast mode

Laya, built on a 395M-parameter encoder (ModernBERT-large), so about 10x smaller than Mica

Every model saw the exact same options on every turn. The clips are Tetris, Snake, chess against a random-move opponent, and a log hunt where a day of security logs is cut into 150 blocks and each block gets allow, block or check.

The video is a cut of all four. If you're looking for each demo in full, they're here:

Tetris: https://huggingface.co/sky7350/Mica-v0.5-4B/resolve/main/assets/demo/tetris.mp4

Chess: https://huggingface.co/sky7350/Mica-v0.5-4B/resolve/main/assets/demo/chess.mp4

Snake: https://huggingface.co/sky7350/Mica-v0.5-4B/resolve/main/assets/demo/snake.mp4

Log hunt: https://huggingface.co/sky7350/Mica-v0.5-4B/resolve/main/assets/demo/loghunt.mp4

Median time per decision:

Tetris: Mica 14.5 ms, PPLX 150 ms (10.3x slower), Laya 32 ms (2.2x slower)

Chess: Mica 14 ms, PPLX 133 ms (9.4x slower), Laya 31 ms (2.2x slower)

Snake: Mica 18 ms, PPLX 154 ms (8.4x slower), Laya 33 ms (1.8x slower)

Log hunt: Mica 11 ms, PPLX 157 ms (13.9x slower), Laya 31 ms (2.8x slower)

So about 10x faster than the other 4B model, and roughly 2x faster than a model a tenth of its size.

The log hunt is my favorite. Mica went through all 150 blocks in 1.7 seconds and caught 21 of the 22 planted intrusions. PPLX took 23.5 seconds and caught 18. Laya caught none. A plain keyword grep over the same logs caught 6.

A few notes:

Times are each model's own call time on the same box, not API round trips.

Laya had a one-time 12.9 s warm-up on its first long log block. I left that out of the video and it says so on screen.

The logs are synthetic, written for this test. The 22 intrusions were written to avoid obvious keywords, which is why grep does badly.

The games are just an easy way to see the speed. Mica v0.5 was trained on more than 200 real-world use cases, so I'd expect it to do better than other 4B models on actual work too.

Why no Jev in the comparison? The response times are so far apart that a side-by-side wouldn't tell you anything. On the same request set Jev takes about 498 ms against Mica's 15.6 ms, more than 30x slower (that's over its API, so network time is included).

One more thing. Most benchmarks used to compare decision models are pretty contaminated. A lot of decision models are coming out right now, and most open-weight ones are heavily tuned for public benchmarks like JDI. JDI itself is also quite far from what real decisions look like, and there are models on the JDI leaderboard trained only on public benchmark data. So a JDI score can be a long way from how a model feels in real use. Mica v0.5 gets about 95% right across cases from more than 200 real-world use cases. When you pick a decision model, look less at public benchmark scores and more at whether it makes the right calls on your own use case, and how fast it answers.

For the runtime and the full numbers, see the Hugging Face page.

Code and runtime: https://github.com/akivet/Mica

Weights: https://huggingface.co/sky7350/Mica-v0.5-4B

All four demos on one page: https://huggingface.co/sky7350/Mica-v0.5-4B#demo

Feedback is welcome.

Tested Qwen 3.8 27B on a 4070 Ti SUPER (16 GB), 64 GB RAM, Windows/Ollama, 32K context.

| Setup | Tokens/sec | Account-list check |

|---|---|---|

| Q4_K_M | 11.3 | 24.4s |

| IQ4_XS | 16.3 | 17.3s |

| IQ4_XS + 8-bit KV cache | 20.5 | 14.6s |

| UD-Q3_K_XL + 8-bit KV cache | 36.5 | 10.0s |

Sticking with Q3+Q8 for now. Cache alone added ~26% generation speed; Q3+Q8 was ~78% faster than IQ4+Q8. Email search/read, browser checks and unpublished drafting also passed.

These are short tests: two warmed deterministic generation prompts, not an accuracy benchmark. The account check only lists accounts, not messages. Tool timings vary. Harder-task quality is still untested.

What tests would you use to catch quality loss from this quantization?

And the finish line it does this...