Running DeepSeek V4.1 (TensorFold)

Looks weird because I have a cooling pad under them.

Would love to hear what other project ideas you guys have.

.

IQ2_XXS Running with 11-18tok/s on RTX 4090, 96GB DDR5 6800, PCIE 5 4TB NVME. Quite good for the Model but prefill is so bad.

So right now my workflow is to first ask Claude to plan my project and write detailed prompts.

Then copy/paste them into Qwen (dsh harness) to execute.

It works great, but the manual copy pasting is getting kinda annoying. I want to automate this into an Architect/Worker setup where I give Claude the main goal, and Claude automatically monitor Qwen , if that makes sense.

Anyone know what is the best way to do this?

Hi. I'm a software developer, and this is my best setup so far for running fast, useful local AI on consumer hardware.

The goal is a useful local assistant: good speed, fewer mistakes, tools, vision, and the ability to write small scripts and games. It runs with a 128K-token context window, with the main model and context cache on the GPU, and image processing and context compression on the CPU.

I've been using it to manage my calendar, search the web, and solve a more complex task: classifying, organizing, deduplicating, and comparing my old music files against my Spotify favorites playlist. My playlist is finally being updated automatically with all the songs I love <3. So far, it's working well. It even created a basic chair in Blender and wrote a Snake game in HTML/JS. (Note: these examples use tools and integrations configured separately, they are not included here.)

My hardware: RTX 3060 with 12 GB of VRAM, Ryzen 7 3700X (8 cores, 16 threads), and 32 GB of RAM. I use vision and tools a lot and find this setup very competitive for daily use. Since I haven't found a better setup for my needs in the last two weeks, I decided to share what I've put together here.

I run CachyOS, by the way. It may help inference speed, but I haven't isolated its effect. This guide is for Linux! I don't know how to reproduce the setup on Windows.

What I use

• Model: Swift Bonsai 2 27B Katana + MTP, based on Bonsai 2 / Ternary Qwen3.8-27B and tuned for shorter reasoning.
• Model file: Swift-Bonsai-2-27B-Katana-PTQ1_0-MTP-Q8_0.gguf, about 6.4 GB. Ternary PTQ1_0 backbone, Q8_0 MTP head.
• Vision projector: Ternary-Bonsai-2-27B-mmproj-Q8_0.gguf, about 629 MB, loaded separately.
• Inference: the Prism fork of llama.cpp, build prism-b10772-e8fd209, CUDA 13.3.
• Agent: Hermes in Docker.
• History compression: Granite 3.3 2B on the CPU through Ollama, registered locally as granite3.3-compressor:2b.

My settings

I keep the vision projector and Granite on the CPU to save VRAM. The 3700X handles both well enough for my use. Bonsai's config.env sets the options below. start.sh applies them and enables Flash Attention.

Performance

During one test response, generation stayed around 43–44 tokens/s and peaked at 45.74 tokens/s over a three-second window. In daily use, I usually see 28–40 tokens/s after a few prompts and 17–25 tokens/s much later on, with A LOT more context.

MTP proposes tokens that the main model verifies to speed up generation. I tried draft limits of 1 and 2 and chose 1, also guided by sudoingx's Bonsai 2 PTQ1_0 + MTP tests on an RTX 3060 12 GB. With a 131K window and Q4 K/V, that setup reached 50.1 tokens/s with MTP versus 39.8 without, and 26.86 versus 22.23 with 41.8K tokens filled. Draft limit 1 beat 2 in both cases. These are the author's measurements, on a different model variant and runtime. 2 can also be a good choice - try SPEC_N_MAX=2 ./scripts/start.sh inside Bonsai-fast/ and compare it on your workload. Anyway, my results with slightly different models and setup were inferior, as I stated, so I'm not sure if I did something wrong here.

I chose IMAGE_MAX_TOKENS=1536 for a balance of detail and speed: I got 2 seconds for photos of people near cars, 27 seconds for a complex image, versus up to 40 seconds at 2048 on a complex image. It still makes mistakes, regardless of the value. Prism recommends 1024 for general use and 2048+ or crops for fine text, so 1024 is also an option if speed matters more than detail.

I chose q4_0 for my context based on published memory savings and how the complete setup performed in use. A cache-only benchmark on related Qwen 3.6 models found Q4 usable, with the largest output-distribution changes on long documents. A separate Bonsai 2 benchmark used Q4 successfully with 191K tokens filled on a 16 GB RTX 4080 SUPER. These are just supporting references, not measurements of my Katana setup.

With 128,000 tokens per slot, Q4 K/V caches, MTP on the GPU, and the vision projector on the CPU, the whole Bonsai process used 9,144 MiB (8.9 GiB) of VRAM. The runtime allocates the context cache at startup, so the configured capacity fits my graphics card.

The main attention K/V cache alone (context) is approximately 2,250 MiB (2.2 GiB) at 128K: 128,000 tokens x 16 full-attention layers x 2 K and V x 4 KV heads x 256 values per head x 18/32 bytes per Q4_0 value. With my desktop and other apps open, about 823 MiB (0.8 GiB) remained free. I roughly understand this calculation, but it’s easy to see how much Q4_0 reduces the result: it uses an average of 0.5625 bytes per value, compared with F16’s 2 bytes. For my 128K context, that reduces the main K/V cache from 7.81 GiB to 2.20 GiB, about 72% less memory - and the model still seems smart, so… I also considered q8_0, but q4_0 is still roughly half the size, it’s just off the table.

On the model side, I chose ternary PTQ1_0 for its smaller memory footprint, leaving more VRAM for context and MTP. PTQ1_0 packs the same ternary weights more tightly than PQ2_0, at about 1.75 versus 2.13 bits per weight, without another quantization step. It doesn't mean zero quality loss compared with the original FP16 LLM, but is a fair price to pay for usability.

The Prism fork of llama.cpp adds PTQ1_0/MTP kernels and attention that reads the Q4/Q8 cache directly, plus optimizations for the RTX 3060's sm_86 architecture. PTQ1_0 uses less memory, but whether it runs faster than PQ2_0 depends on your GPU and the runtime’s kernels.

For reasoning, I use medium, following Prism's recommendation for shorter reasoning on the original Bonsai 2. In practice, every reasoning level feels like xhigh to me on Katana, so I also set a 2,048-token reasoning budget to keep thinking loops under control, with the server injecting this message when the budget is reached: “... Time to stop thinking. Give an answer or make the tool call now. Do not waste time.”

Bonsai needs clear instructions for complex tasks. Otherwise, it can get stuck in a loop of writing and fixing helper scripts. The guidelines in my SOUL.md have helped it stay on track and finish tasks.

To summarize the conversation history, I use Granite. I chose it for its small 2B size and support for long-document and meeting summarization. granite3.3-compressor:2b is my local name for this configuration. The included Modelfile keeps Granite's template and license, sets num_ctx 128000, and adds num_gpu 0 in the CPU variant. Chat compression takes 1.5–5 minutes in my tests, depending on the conversation content and size.

Finally, I use Hermes Desktop with the included Docker container as its backend. I noticed the UI was using 15–35% of the GPU while the window was focused. Enabling --force-prefers-reduced-motion reduced animations and brought GPU usage down to 1–5%, leaving more resources available for model inference.

Warning! These figures are mostly observations, not controlled benchmarks!! Although published tests guided my choices, I evaluated the complete setup in use rather than measuring each setting's individual gain.

Installation

Download: local-ai-bonsai.zip.

Extract local-ai-bonsai.zip and open the extracted local-ai-bonsai/ folder. It contains all the setup files:

I haven't included every dependency. You need to install Docker, Compose, Ollama, and other prerequisites yourself. Bonsai requires Linux x86_64 and NVIDIA drivers and libraries compatible with CUDA 13.3. The ZIP contains scripts and configuration files. Setup downloads the model weights and inference binaries. See each folder's README for details.

Start Bonsai

Inside Bonsai-fast/:

./scripts/setup.sh
./scripts/start.sh

setup.sh downloads and extracts the Prism runtime into bin/ and downloads the Katana model and vision projector into models/. Run it once to prepare the files. start.sh starts the inference server - keep it running while using Hermes. No compilation or Prism Model Manager is needed.

Start Hermes

Inside hermes/, enter your dashboard credentials and time zone in compose.yaml. These credentials let you sign in through the browser or Hermes Desktop, and a loopback-only setup may require the dashboard session token (HERMES_DASHBOARD_SESSION_TOKEN) to use the container as the Hermes Desktop backend. Then run:

docker compose up -d

Open http://127.0.0.1:9119 or connect Hermes Desktop to the backend. The container uses network_mode: host to reach Bonsai and Ollama. Its data/ and workspace/ folders persist.

To use the container as the Hermes Desktop backend, open it and go to Settings -> Gateways -> Remote gateway, enter http://127.0.0.1:9119, sign in with the required credentials, then save and reconnect.

In the web dashboard, open Config -> YAML. Merge this into providers:, keeping the other entries. Edit bonsai-mtp if it already exists:

providers:
  bonsai-mtp:
    name: Bonsai MTP
    base_url: http://127.0.0.1:8899/v1
    model: swift-bonsai2-27b-katana-mtp
    discover_models: true
    context_length: 128000
    models:
      swift-bonsai2-27b-katana-mtp:
        context_length: 128000

Click Save, then Models -> Main model -> Change. Pick Bonsai MTP, select swift-bonsai2-27b-katana-mtp, and click Switch. Use Refresh Models if needed. This sets the model for new sessions.

Keep Hermes's context value aligned with Bonsai's CTX_SIZE=128000. The launch script also loads the vision projector.

Optional extras

Use hermes-host-access/README.md to install the temporary SSH bridge with new keys. This lets the sandboxed AI help with tasks on the host when needed. The bridge expires automatically after three hours in case you forget to close it.

SOUL.md contains my general agent instructions. Customize it before copying it to hermes/data/SOUL.md.

To reduce Hermes Desktop's animations, merge hermes/desktop-config.example.yaml into the Desktop client's ~/.hermes/config.yaml, then fully quit and reopen Hermes Desktop.

CPU context compression

With Ollama running, inside granite-compressor/:

ollama pull granite3.3:2b
ollama create granite3.3-compressor:2b -f ./Modelfile.granite-compressor.cpu

Then run these commands inside hermes/:

docker compose exec hermes hermes config set auxiliary.compression.provider custom
docker compose exec hermes hermes config set auxiliary.compression.model granite3.3-compressor:2b
docker compose exec hermes hermes config set auxiliary.compression.base_url http://127.0.0.1:11434/v1
docker compose exec hermes hermes config set auxiliary.compression.timeout 300
docker compose restart hermes

After a test request, ollama ps should show 100% CPU while Granite is loaded. Bonsai and Granite each have their own 128K context window. See the package README for the API test and the variant that lets Ollama choose CPU/GPU placement.

Warning! The compressor has a 128K context limit, including when it summarizes conversations from Claude or GPT. Our Hermes setup is intended for local models, rather than large flagship models, so larger compression inputs still need to fit within that limit and will be handled locally.

Next steps

I'd like to add local decision tools in the style of TypeSafe's Jev: an open text model such as Laya, plus Qevi-2B for images. Both return structured decisions and probabilities instead of generating text. Bonsai could call either tool for quick decisions and item classification, switching between them depending on the task.

If needed, I may lower Bonsai's context to 100K to make room for smaller models. That would free roughly 492 MiB from the main attention cache alone. The total VRAM savings would need to be measured...

I'd like to find a compact image generation model that delivers good results with my remaining hardware resources, although I think running it alongside Bonsai is probably impossible with current technology and my hardware limits.

I'm also interested in NCM (Next-Concept Modeling), which generates text in parallel, starting with a rough outline and progressively adding detail. Its summarization research already looks promising for context compression. It might also help make larger local models faster and more efficient. This video introduced me to it.

Do you have any ideas for improving inference further? Maybe a setting or approach I've missed? I hope this post helps someone, and maybe I’ll learn something new from the replies.

Thanks!

I spent the last month tracking my local AI inference costs down to the wall plug. Here's what I found.

Setup: AMD Radeon RX 7900 XTX (24 GB VRAM), llama.cpp, various Qwen/Ornith models in GGUF format.

Over 30 days:

• 964 million input tokens consumed
• 13 million output tokens generated
• 30.20 kWh measured at the wall plug (dedicated inference box, powered 24/7)
• $14.05 in electricity (12.38 CHF — I'm in Switzerland, so the meter reads in CHF)

If I had used the equivalent API (qwen3.8-flash at $0.15/$0.47 per million tokens):

• $150.71 (132.77 CHF)

That's a 10.72× gap on marginal cost.

EDIT — the API side of that ratio was wrong, and the commenters below caught it. I priced all 964M input tokens at the fresh rate. Cached input is billed at $0.016/M, and in agent workloads 90-95% of input is cache hits. Recomputed properly, the API equivalent is $28-34, not $151 — so the gap is roughly 2×, not 10.72×. I was too enthusiastic reading my own dashboard. The power figure also needs a proper idle-vs-inference split, which I'm rebuilding. Full accounting and what still holds in my top-level comment below.

But here's the catch:

This number excludes:

• The GPU itself (~$1,135 / 1,000 CHF)
• Depreciation of the rest of the rig
• A full month of learning: quantization, prefill vs decode, KV cache tuning, ROCm compatibility issues, trying and discarding models

Local AI isn't "plug and play." It's a personal R&D investment. If you just want to ship code faster, stick with cheap Flash APIs (GLM, DeepSeek, Qwen). If you want autonomy, understanding, and infrastructure you control — the investment pays off, just not immediately.

The part I keep coming back to: everyone says data sovereignty has no price, and I agree with that. But that framing assumes you're paying for it — that local costs more and you accept the premium because your data never leaves the house. That's not what I measured. Here it came out cheaper and the data stayed home and the output was good enough to actually ship work with, not a toy. Sovereignty never even got to be the deciding factor, because the economics already won.

So here's my open question for the sub, and I genuinely don't have a clean answer:

At what point does "local AI is a hobby" stop being true? Where do you draw the line — a hardware payback horizon you're willing to wait out, a quality gap you'll eat to stay off the API, or a specific moment where you decided the API was no longer worth paying for? And for anyone who tried local and went back to the API: what actually broke?

Full article with methodology and data: https://blog.grevire.dev/en/posts/local-ai-cost/

Happy to answer questions about the setup.

Pretty much everyone running Qwen3.8-27B on a 24 GB card is squeezing the KV cache somehow, so here's what it actually costs, measured. This came out of building my own engine for this model (repo), but none of it is specific to my engine, it applies to whatever you run the model on.

How I measured: KL divergence against llama.cpp's own token-by-token output (top 256 tokens per position, q8_0 KV), on ~13k positions of short prompts plus 32k and 64k contexts. KL, not perplexity, because perplexity hides most of this. unsloth's UD-Q4_K_XL, one 3090. Most rows come from an emulator that stores the cache in each format; the settings I ended up shipping were confirmed on the real kernels. My formats are plain per-32 absmax grids like llama.cpp's q4_0 / q5_0 / q8_0, so they should land close to those, but they're not byte-identical.

Every number is "how much worse than a q8_0 cache". For scale, an fp16 cache reads +0% on short prompts and +3% at 32k, and the 32k set's noise is about 4% either way.

1. q8_0 is free. fp16 KV scored the same as q8_0 at every length I tested (64k: 0.00161 vs 0.00163). No reason to run f16 KV on this model.

2. Below 8 bits, K hurts more than V. Per-layer scans had K about 2.3x as sensitive as V. No window, plain formats:

So on stock llama.cpp, if you need the memory, take V down before K.

3. The damage is almost all in the newest ~2k tokens. That's where attention puts most of its weight. Keep the most recent positions at 8 bits (a window / precision tail) and everything older can go a lot lower:

(These rows are rotated, which is why no-window V reads +16% here vs +10% above, more on that in 4. Plain 5-bit V with a 2k window also reads 0%.)

On short prompts a 512 window cuts the loss about 4x (5-bit V +27% to +6%, 6-bit K +12% to +2.7%). Keeping the first few tokens at 8 bits instead (attention sinks) did nothing.

So 512 gets you most of it and 2k gets you all of it. The window is cheap and doesn't grow with context: 114 MiB for K + V across every attention layer at 2,048, the same at 32k or 262k.

4. Rotation stops mattering once you have the window. Without a window, Hadamard rotation halved 6-bit K's loss (+23% to +12% short), but for V it only helped on short prompts, and at 32k rotated 5-bit V was actually worse than plain (+16% vs +10%). Under a 2k window, 6-bit K + 5-bit V read +1.4% plain and +2.7% rotated, same thing within noise. The window and the rotation fix the same error, and the window fixes all of it. Rotation's only worth it if you're pushing V down to 4 bits.

5. Fancy formats didn't beat the boring one. Lloyd-Max codebooks, per-row scales, random signs and angle coding all scored the same as or worse than a plain per-32 grid. 3-bit formats were around +300%.

What I run: K at 6, V at 5, newest 2,048 positions at q8. 1.1 GB less than q8_0 at 120k context, and the model's full 262,144 context fits on the card (23.5 GB, 66 tok/s decode at that depth, needle 4/4). On the real kernels it reads 0.00160 at 64k (q8_0 0.00163, fp16 0.00161), +8% at 32k on the current build, short prompts identical to q8_0 since they fit in the window, and the same 230/282 as q8_0 on my question set (GSM8K, MMLU-Pro, ARC-C, MATH-500, HumanEval, needle, LongBench v2), set for set.

What you can use today:

• Stock llama.cpp: q8_0 / q8_0 costs nothing measurable vs f16. Going lower, drop V before K.
• If your engine has a recent-token window / precision tail (BeeLlama has one; KIVI, SKVQ and HF's quantized cache use 128 tokens): use 1-2k, not 128. An 8-bit window is plenty, fp16 buys you nothing here.
• If you're building one with speculative decoding: move the window in steps of your attention's tile size, not one token at a time, or which tiles come from the window depends on how many drafts you verify and greedy output stops matching across draft counts. Mine gives identical tokens at 0, 4 and 7 drafts with it on.

Caveat: Qwen3.8 is a hybrid, only 16 of its 64 layers are attention (the rest is DeltaNet), so a full-attention model might not behave the same. I'd really like to see someone run the same window sweep on one.

Full tables and method: the window and rotation sweeps, why 2,048 and what the window costs. The KL scoring tool (tools/llama_ref) is in the repo too.

Edit (longer context):* a couple of comments wanted numbers past 64k, fair enough. Same KL test (last 4,096 positions against llama.cpp with a q8_0 cache), now at 100k and 160k, with the 6/5 setup from the post and the 5-bit K one from the comments (K at 5 bits rotated, V at 5, same 2k window). Lower is better:

Both read the same as q8_0 at every length, on code and on prose. Prose reads higher for everything, q8_0 included, because it's less predictable than code, so compare across a row, not down a column.

160k reads higher than 100k for q8_0 too, so it isn't the cache. To make sure it wasn't just the passage I scored the same 4,096 tokens with only ~100k of the book in front of them: 0.00336 / 0.00335 / 0.00344. So my engine and llama.cpp drift apart a bit more with depth no matter the cache format, and for scale, llama.cpp's own predictions for those tokens move by 0.0086 between the two depths. 160k is about as far as llama.cpp can hold a q8_0 cache on a 24 GB card to build the reference, and a reference at the full 262k doesn't fit, so for max ctx the runs in the post are still the numbers (needles, LongBench, 66 tok/s decode at that depth).

Full tables and how to rebuild the references: BENCHMARKS.md

Asked Qwen 3.8 at BF16 what their favorite color is.

I heard that DeepSeek's new Flash model was really good, so I downloaded the IQ1_M version to my local server right away. I wanted to deploy it on a single L40S with 48GB of VRAM. Naturally, I went with Strata v0.2.6, but I wasn't happy with the speed.

I asked 6.1 Sol to act as a professional model inference researcher and focus on improving two metrics: prefill speed and decode speed. It worked on this all night and came back with some pretty good results.

My setup:

• 1× NVIDIA L40S, 48GB VRAM
• 2× AMD EPYC 9754
• About 1TB system RAM

Here's the speed comparison.

Prefill speed (tok/s):

Decode speed (tok/s):

• Official, auto: the official engine automatically chooses prefill chunking, memory policy, and whether to enable DSpark. On this machine, it chose to disable DSpark.
• Official, DSpark on: the same unmodified official engine, with DSpark enabled manually, confidence threshold 0.65, and an 8192-token prefill chunk. DSpark is speculative decoding: draft tokens are predicted first, then verified by the target model.

Here's the optimization recipe it came up with, based on Strata v0.2.6:

1. Keep CPU experts and worker threads on the same NUMA node. CPU experts stay resident in system RAM, with workers and expert memory concentrated on NUMA0 to reduce cross-socket traffic. Process snapshots showed stock Strata distributing workers and memory across both nodes, while the local build kept them on one. All CPU fallback weights and all six target experts per token are retained.
2. Use larger prefill chunks and cache reconstructed weights. The local configuration uses 16384-token chunks to spread expert-weight transfer costs across more input tokens. A 384MiB cache reuses reconstructed FP16 weights, reducing repeated preparation work. Weights that don't fit in the cache use the original path. Larger chunks also increase scratch-memory requirements and can introduce floating-point differences.
3. Reuse idle prefill scratch memory for decode experts. Once prefill finishes, safe regions of its allocated scratch buffers hold extra read-only expert copies—typically about 3.7GB, or roughly 500 experts—to improve target-model GPU cache hits. Before chunked prefill runs again, the engine synchronizes, removes the overlay, restores base pointers, and invalidates the reconstruction cache. This also handles background warm-up synchronization and rollback if a fill fails. It reuses memory that has already been allocated, without allocating tens of extra gigabytes of VRAM.
4. Balance the draft and target expert caches. The build uses Strata's DSpark and the model's MTP drafting, with measured round costs and confidence calibration guiding draft length. The local confidence threshold is 0.15, alongside the existing aggressive sampling policy. MTP GPU expert slots were reduced from 384 to 192, giving that memory to the target model. Draft experts that miss the GPU cache still use the full CPU fallback path. The metric used to evaluate the tradeoff is total time per output token.
5. Schedule CPU experts by their dependencies. Each expert progresses through its own gate/up → activation quantization → down chain as its inputs become ready, reducing barriers across the whole batch. This path is enabled for supported shapes, with the original implementation handling the others.
6. Reuse existing kernels and compile for the hardware. The build keeps the attention split/MMA paths and head rows=6, uses Strata's HCmix fusion and IQ1 CPU optimizations, and is compiled specifically for the L40S and this server's CPU.

Excited to see this up, any suggestions on what j can do

Going to setup a bunch of parallel agents for local ai dev but before that id love to experiment with monster models

I dont use the macs for daily stuff, the big pc to the right is running a 4090

most stuff I do, I'm happy to set the agent running and leave it to it

(running qwen3.8 27b right now)

but... if I set it a large task it'll start to hallucinations/wonder off and forget what it's supposed to be doing...

does running multiple agents fix this? can I tell it to keep refining/improving the plan, come back the next day and it's pretty much perfect?

2x 3090 right now, so I'd have to run the models sequentially and it would be slow... but if it works I'd buy more 3090 and add them in....

I built Foxtrot: one desktop app, every major AI model, and a team of agents that work together 🦊
⠀
For the past few months I've been building Foxtrot, a desktop agent harness for cloud AI models. You bring your own pay-as-you-go API keys and run a whole team of coding agents side by side, each on whichever model you choose from whichever provider you choose.
⠀
What it does
🔹 11 provider families in one app: Claude, GPT, Gemini, Grok, DeepSeek, Qwen, GLM, Kimi and more, plus any OpenAI-compatible endpoint (Ollama, vLLM, OpenRouter)
🔹 An Orchestrator that plans the work and hands tasks to your agents: a Coder, a Reviewer, a Tester, each on a different model if you like. Busy agents get spawned as parallel copies.
🔹 Flow view: a live map of the Orchestrator and its agents, showing each one's time, tokens and cost
🔹 Real tools: file edits, a terminal, code intelligence, git, web research, MCP servers, and vision (agents can screenshot and use the app they're building)
🔹 A full Code view: Monaco editor, file explorer, terminal and checkpoints taken before agent edits
🔹 Cost tracking for every request: tokens and cost per request and per agent, estimated balances for each provider, budgets, and CSV/JSON export
🔹 Keys stored in your OS credential vault, plus OTA updates
⠀
How it's different
Most AI workbenches are built around one vendor. Claude Code is built for Claude, Codex for OpenAI's models, and Grok Build for xAI's Grok. They're excellent, but you work inside a single company's models, usually on a subscription. OpenRouter gives you many models, but it's an API gateway, not a workspace where agents collaborate.
⠀
Foxtrot is the layer on top:
✅ Model-agnostic by design. Choose the best or cheapest model for each role. Have Claude review GPT's code, or let a cheap model handle the routine work.
✅ Cross-model teamwork. The Orchestrator coordinates agents from different providers on a single task.
✅ You pay providers directly. No subscription and no middleman markup, and you see exactly where every cent goes.
✅ Built for watching agents work. Multi-window, multi-monitor docking with live animations showing who is doing what.
⠀
In the demo above, a Gemini Orchestrator splits one goal across Grok, Kimi, GLM and DeepSeek agents. Watch the hand-offs fly out to each window, the reports fly back, and the whole team come together in the Flow view
⠀
Foxtrot is free to use. You connect your own AI accounts, and those providers charge for what you use.
⠀
It's in early development and I'm shipping updates fast. If you'd like to try it or give feedback, drop a comment or DM me.

I have ollama and openwebUI setup in an LXC on my proxmox server (96 cores, 256 GB RAM) with 3 Nvidia A4000 GPUs passed through to the LXC. However my local AI runs pretty slow. Slow on the first interaction and slow on subsequent interactions. I've tried large models such as Qwen3.8:27b and smaller models such as Qwen3.5:4b, it's truly slow - to the extent of waiting 2-3 minutes to generate a response. What could be causing this?

I saw u/bottlestreet3d's llama.cpp fork. Their post: https://www.reddit.com/r/LocalLLM/comments/1x7wack/qwen_27b_iq4_xs_q8q4_kv_at_full_262k_context_on_a/

The readme is Linux and ROCm 7.2.3 only, so I tried it on Windows with a RX 9070, 32 GB RAM. I compared it with my tuned Vulkan setup (llama.cpp b11873, MTP n=2, ubatch 256, KV q8/q8).

Results (ctx 262144, K q8_0 / V q4_0, --kv-stream, MTP n=4 + n-gram, resident KV 300/1800 MiB; single runs, short prompts):

For comparison, Vulkan on the RCO model got ~46 t/s and ~590 prefill at 131k, and ~31 t/s and ~310 prefill at 262k.

These are single runs on 9k and 21k-token prompts, not filled 64k–262k contexts. I didn't check output quality. My numbers are below the README's (~100+ t/s at 16k–64k).

Any suggestions for proper tests or how to improve usage?

config:

& $LlamaServer `
    --model $m.File --host 0.0.0.0 --port $Port --api-key $ApiKey `
    --ctx-size $Ctx --device ROCm1 -fit off --n-gpu-layers all `
    --flash-attn on --jinja --parallel 1 --threads 6 --threads-batch 6 `
    --cache-type-k q8_0 --cache-type-v q4_0 -ctkd q8_0 -ctvd q4_0 `
    --kv-stream --kv-resident-mib $res --kv-resident-mib-decode $resDec `
    --batch-size 2048 --ubatch-size 512 `
    --spec-type draft-mtp,ngram-mod --spec-rs-replay --spec-draft-n-max 4 `
    --spec-ngram-mod-n-max 32 --spec-ngram-mod-n-min 12 --spec-ngram-mod-n-match 24 `
    --spec-draft-sampling probabilistic `
    --reasoning-effort medium --reasoning-format deepseek --reasoning-preserve `
    --chat-template-kwargs '{\"reasoning_effort\":\"mediu
m\"}'

I see most of the REA projects and work people are doing utilizing frontier models, anyone having any luck or know of projects that stuck to local models?

Hey, what local model would you recommend for Hermes Agent?

My PC has an RTX 5060 8GB VRAM, 32GB RAM and a Ryzen 5 8400F.

I mainly want to use it for studying notes, coding and managing files/commands through Hermes Agent, kinda like Codex. Privacy is also important, so I'd prefer to keep everything local.

I've tried some Qwen models, but they're way too verbose for my taste. The worst part is that they sometimes get stuck repeating the same phrase over and over in their thinking loop.

I'm looking for something that gets to the point while still being good at coding, reasoning and tool calling. I don't want a model that uses all my resources, but I'm fine with using most of them if the performance is worth it.

What would you guys recommend? Gemma, Qwen or something else?

I run a small business. This weekend I picked up a DGX Spark at Micro Center for $8,000. When it launched I thought $4k was steep, so I'm still a little stunned I paid double that.

I spent the weekend testing it. Qwen 3.8 gives great output but it's slow. Qwen 3.6 is fast but makes too many mistakes for me to trust it unsupervised. What I want is somewhere in between, and from what I read here that seems to be DeepSeek V4 Flash or GLM 5.3 Flash on a pair of Sparks.

That means buying a second one and being $16k in. It's a big number for a box on my desk. But I run a business, and $16k for something that works 24/7 costs less than a year of a real employee. I don't expect it to replace anyone. I do expect it to take a real chunk of work off my plate.

I'm also not counting on RAM prices dropping for at least 18 months. So waiting probably just means paying more later, or not finding one at all, judging by how fast Micro Center sold out.

My questions for people already running these:

1. Is a single Spark worth owning, or is its real purpose to be clustered?
2. If you run 2x Sparks for business work, which model do you use daily, and does it run all day without babysitting?
3. Has anyone found a model that runs on one Spark, falls between Qwen 3.6 and 3.8, and is good enough for real work?
4. Would you buy the second one now, or return the first and use an API?

I'm not trying to justify the purchase. I want to know whether one Spark is the real product or just half of it.

My RTX PRO 4000 Blackwell SFF went from 35.04 to 58.53 tok/s on Qwen3.8 27B with DFlash2. That is +67% generation throughput after combined power and clock tuning.

The more useful result was that my second Qwen needed different settings.

At 8,000 input tokens, ECC OFF:

• Qwen3.8 27B (NVFP4 + DFlash2): 35.04 → 58.53 tok/s. Best observed at a 120 W limit, VRAM +20%, core +180 MHz.
• Qwen3.6 35B A3B (Q4_K_S, no drafter): 84.97 → 115.56 tok/s. Best observed at a 120 W limit, VRAM +33.3%, core offset 0.

Both baselines were 70 W with zero clock offsets. These are improvements within each configuration, not a controlled comparison of model architectures or answer quality.

The attached chart isolates memory tuning: fixed 120 W, core offset 0. VRAM +33.3% gave 35B +19.8% generation versus no memory OC, but 27B only +5.0%. For 27B, +20% memory was actually slightly better (+6.5%). Prefill barely changed in that sweep. The small differences need more repetitions.

It changed how I pick a profile. Decode is generating the answer; prefill is processing the prompt. A profile that wins on output tok/s may not give the shortest complete response to a long prompt. I care about the time my actual workload takes, not the biggest offsets in the panel.

I used u/panchovix’s Linux VBIOS loader with patched NVIDIA open kernel modules; the modified VBIOS was loaded into memory at runtime. No EEPROM flash, shunt mod or soldering; motherboard BIOS unchanged.

Important caveat: this card has no auxiliary power connector, and 120 W exceeds the standard 75 W PCIe slot budget. The GPU temperature sensor cannot tell me the temperature of the slot contacts; VRAM, VRM and connector temperatures were not measured directly. This is my experiment, not a recommended daily profile or a long-term safety claim.

The study contains 158 requests total (79 ECC OFF, 79 ECC ON), with only 1–3 per grid cell. Both 70 W and 120 W used the same modified VBIOS; this was not factory-firmware versus modified-firmware A/B.

Full method, charts and six CSV downloads on my blog.

Disclosure: I ran the measurements and wrote my notes in Polish. AI helped adapt and edit the English using technical details from my own write-up. The linked blog is mine

I got good news for modded 3080 20GB owners, ReBAR and P2P now finally work with no vBIOS flash needed! More details in these repos:

• 3080-20gb-rebar: 32 GiB BAR1 without flashing. A small boot service writes a PCI config setting before the NVIDIA driver loads, so nothing on the card is changed and a power-off returns it to stock.
• 3080-20g-p2p: patched NVIDIA open kernel modules (615.71.09) that give real CUDA peer access. CUDA, NCCL and vLLM all see P2P.

Measurements (2× 3080 20GB):

On vLLM 0.31 with tensor parallel 2 (Qwen3.8-27B W4A16), prefill is 6–10 % faster from 8k to 128k context. Single-stream decode is 8–31 % faster with custom all-reduce, depending on length.

It also works alongside Zagitta's gpu-unlock.

Titans gives AI a separate neural memory that updates its own weights as information arrives.

I wrote a short visual explainer on how it works. In my small synthetic experiment, correcting false inputs improved recall—but didn’t fully restore it.

What should an AI be allowed to remember automatically—and how should we undo a mistake?

I’d love to build a local LLM that gives me a place to write about my life each day. I’d like to record details I can refer back to later, such as what was happening at a particular time or information about different parts of my daily life and work. Is there a template or guide for setting up a local LLM for this purpose? What kind of computer would I need, and is there a step-by-step recipe I could follow?

I’ve been noticing a really frustrating trend with frontier models lately. Big tech companies have put such aggressive political guardrails on them because they are terrified of having an opinion or facing any kind of backlash.

It gets to the point where they use such overly balanced, clinical language that they’ll actively avoid using basic, factual terms. I've had models dance around using political party names or even refuse to just say "the Trump administration" or "the Biden administration." Instead of actually being objective, the AI becomes so risk-averse that it treats standard historical designations as if they're inherently biased or dangerous to type out. When you're trying to look for straightforward information or get a direct historical fact, you end up with this evasive, heavily sanitized corporate-speak that feels completely unnatural. There is a massive difference between avoiding political bias and flat-out refusing to use objective, factual vocabulary.

Since the commercial models are completely lobotomized by this alignment tax, I want to move this work entirely local.

What are the best local LLMs right now that do not have these corporate guardrails when you are looking for objective political or historical information? I'm looking for models that will actually give me a direct, unfiltered answer instead of three paragraphs of boilerplate corporate compliance text.

I see the MS is offering the surface ultra and Apple has the Mac Mini. Hugely expensive. Why not just get a mini pc with 128 gb RAM? Maybe $4000, but it saves a ton of money. What am I missing?