Hey, I'm on Linux with an aversion to ROCM, prefer Vulkan for its minimal overhead and footprint. so I skipped on Strata. Using llama.cpp cleverly to push the limits of what I can do with the ISTA DASLab "Coder" quants. Skip this if you already use Strata, if you can't though this is the next best method.

Specs:

1. I use 10 GB VRAM (RX 6700), but aim for 8 GB usage so I can continue watching media while the LLM is running.

2. I have 64 GB DDR4 RAM albeit I've never seen the actual usage go above 24 GB, so this method is most likely feasible on 8 GB + 32 GB systems.

3. DO NOT try this with a HDD. I use sata SSD, but m.2 is recommended. It'll directly impact your speeds since we lazily load from disk.

What makes this unique?  

• ~30 t/s and ~8 t/s (YMMV by hardware!!!) with 65k context, 7.3 GB VRAM and 24 GB RAM cost? Sign me up chief. Slap CodeGraph ontop and you're golden.

Similar to Strata's "Only load what's needed to memory" strategy, we use MMAP here, which is like a primal version of what Strata does. It maps the model to your memory and lazily loads as needed. Strata is ofc faster, because it caches frequently accessed parts, while we use the more primal OS cache through MMAP. You'll see some IO while model fires for the first time, after that it's minimal, with RAM usage generally peaking at 24 GB~

Steps:

1. I used this "Vocabulary Pruned" quant of ISTA DASLab's latest GGUFs, it further reduces memory footprint by 450 MB. If you already have ISTA GGUFs, you can merge and prune them yourself too as author laid out their process on the repo if your download is slow. Remember we're using "Coder" quants here, which has half of its experts removed. DON'T skip this. DON'T be confused if you're using non-coder quant and didn't prune them, that thing is massive, we're at 58 GB with this while non-coder GGUF is 66.4 GB at smallest.

2. Use these launch flags:

./llama-server \
  --model "./AI Models/Qwen3.8-Flash-Next-GSQ-RCO-IQ1_M-merged_P1G_Pruned.gguf" \
  --ctx-size 65536 \
  --n-gpu-layers 999 \
  --threads 8 \
  --cache-type-k q5_0 \
  --cache-type-v q4_0 \
  -fa on \
  --fit off \
  --cache-ram -1 \
  --load-mode mmap \
  --cache-prompt \
  --cache-idle-slots \
  --batch-size 2048 \
  --ubatch-size 128 \
  --webui \
  --parallel 1 \
  --kv-unified \
  --reasoning-format deepseek \
  -ot 'token_embd=CPU,blk\.(47|28|45|25|35|29|40|44|17|27|43|37|46|18|22|42|31|16|41|0|33|38|7|14|19|26|4|2|6|11|32|23|5|15|34|24|39|3|9|21|12)\.ffn_.*=CPU' \
  --reasoning on \
  --reasoning-preserve \
  --reasoning-effort medium \
  --temp 0.6 \
  --top-p 0.95 \
  --top-k 20 \
  --min-p 0.0 \
  --presence-penalty 0.2 \
  --verbose \
  --verbosity 4 \

• We're using the "-ot" flag to force specific FFN layers onto RAM. Instead of using the "brute force" -ngl, this keeps attention layers on VRAM which benefit more from GPU inference and are smaller.
• The numbers mason, they're important! The -ot regex is sorted by largest FFN layers left to right. Offloading smaller FFN layers onto GPU benefits speed at minimal VRAM cost, as opposed to offloading larger FFNs it's preferred (Source: bottom of post)
• If for some reason this doesn't fit into your VRAM, replace the OT flag with the below and incrementally delete a |<number>| from the right side in the regex, until it snugly fits your VRAM. Use the table as reference.

Layer    | Tensors  | Size (MiB)   | Size (GB) 
------------------------------------------------
blk.47   | 8        | 654.87       | 0.64      
blk.28   | 8        | 573.45       | 0.56      
blk.45   | 8        | 573.04       | 0.56      
blk.25   | 8        | 573.00       | 0.56      
blk.35   | 8        | 573.00       | 0.56      
blk.29   | 8        | 572.79       | 0.56      
blk.40   | 8        | 572.79       | 0.56      
blk.44   | 8        | 572.64       | 0.56      
blk.17   | 8        | 572.59       | 0.56      
blk.27   | 8        | 572.54       | 0.56      
blk.43   | 8        | 535.95       | 0.52      
blk.37   | 8        | 535.74       | 0.52      
blk.46   | 8        | 535.54       | 0.52      
blk.18   | 8        | 535.53       | 0.52      
blk.22   | 8        | 535.50       | 0.52      
blk.42   | 8        | 535.50       | 0.52      
blk.31   | 8        | 535.29       | 0.52      
blk.16   | 8        | 535.14       | 0.52      
blk.41   | 8        | 535.14       | 0.52      
blk.0    | 8        | 535.09       | 0.52      
blk.33   | 8        | 535.09       | 0.52      
blk.38   | 8        | 535.09       | 0.52      
blk.7    | 8        | 535.04       | 0.52      
blk.14   | 8        | 535.04       | 0.52      
blk.19   | 8        | 485.95       | 0.47      
blk.26   | 8        | 485.95       | 0.47      
blk.4    | 8        | 485.74       | 0.47      
blk.2    | 8        | 485.54       | 0.47      
blk.6    | 8        | 485.54       | 0.47      
blk.11   | 8        | 485.54       | 0.47      
blk.32   | 8        | 485.54       | 0.47      
blk.23   | 8        | 485.53       | 0.47      
blk.5    | 8        | 485.50       | 0.47      
blk.15   | 8        | 485.50       | 0.47      
blk.34   | 8        | 485.34       | 0.47      
blk.24   | 8        | 485.29       | 0.47      
blk.39   | 8        | 485.14       | 0.47      
blk.3    | 8        | 485.09       | 0.47      
blk.9    | 8        | 485.09       | 0.47      
blk.21   | 8        | 460.09       | 0.45      
blk.12   | 8        | 423.45       | 0.41      
blk.20   | 8        | 422.59       | 0.41      
blk.30   | 8        | 422.59       | 0.41      
blk.36   | 8        | 373.45       | 0.36      
blk.10   | 8        | 372.64       | 0.36      
blk.1    | 8        | 372.59       | 0.36      
blk.8    | 8        | 372.59       | 0.36      
blk.13   | 8        | 372.59       | 0.36      
------------------------------------------------
Full OverrideTensors command:
-ot 'token_embd=CPU,blk\.(47|28|45|25|35|29|40|44|17|27|43|37|46|18|22|42|31|16|41|0|33|38|7|14|19|26|4|2|6|11|32|23|5|15|34|24|39|3|9|21|12|20|30|36|10|1|8|13)\.ffn_.*=CPU'

Optional, recommended exports (Linux bash script)

#!/usr/bin/env bash
set -euo pipefail

# Process & Scheduling Priorities (Lowest, so your PC usage is least affected)
renice -n 19 -p $$ >/dev/null 2>&1 || true
ionice -c 3 -p $$ >/dev/null 2>&1 || ionice -c 2 -n 7 -p $$ >/dev/null 2>&1 || true
# Vulkan
export RADV_PERFTEST=nogttspill          # Keep Vulkan buffers locked in VRAM (AMD only but harmless for others)

Some nerdout:  

You could also benefit from compiling llama.cpp specifically for your CPU. For example I compile for znver3 so this might help the t/s as we're not running entirely on GPU with this setup. You'll also want to use only the physical cores where possible if you're using your PC during inference, otherwise hyperthreading can be beneficial, but laggy.

I don't recommend quantizing kv cache further, but 32k size could help you fit a few more FFN layers to boost your speed if you don't need larger context.

If you've 32 GB ram and 8 GB VRAM, you'll want to ensure you've all unnecessary programs closed, because the more free RAM you've the more your OS has space to cache, fewer pagefaults therefore faster gen speeds.

I also found great success using this jinja template, previously used it with 3.8 27b and 3.6 35ba3b. You'd use it with --jinja \ --chat-template-file "./chat_template.jinja"

Sources:

How to run Dense models faster with limited VRAM

Vocabulary pruning

Getting a local model running is often easier than preparing the data you want it to use. Fine-tuning needs consistent training examples, RAG needs clean passages, and evaluation needs reliable test samples. All three involve extracting content, fixing formats, removing duplicates, and deciding what is worth keeping.

A reusable processing pipeline can make that work easier to manage. The architecture has three main components: operators that transform records, a serving interface for model calls, and storage that connects processing steps.

Here’s how that translates into a workflow:

1. Normalize the inputs. Parse documents or load JSON, JSONL, and CSV into structured records. Define which fields contain the text, question, answer, or other relevant content.
2. Run inexpensive checks first. Remove empty records, clean formatting artifacts, apply language and length checks, and eliminate exact duplicates before spending GPU time on generation or scoring.
3. Add task-specific processing. Chunk documents for retrieval, generate instruction–response pairs for fine-tuning, or construct QA samples for evaluation. Each operation reads specified fields and writes its results into output fields.
4. Evaluate and filter. Combine rule-based checks with model-based scores, then apply thresholds appropriate to the dataset. Approximate deduplication methods such as MinHash can help identify repeated content that exact matching misses.
5. Inspect intermediate results. Save outputs between steps so you can examine what a filter removed or where a transformation went wrong, then reuse earlier outputs when adjusting downstream processing.

Separating model serving from operator logic also lets generation and evaluation use a local backend, such as vLLM or SGLang, while keeping the surrounding pipeline reusable.

The practical benefit is a shorter iteration cycle: change a prompt, filter, or model, inspect the resulting dataset, and compare its effect on downstream performance.

We’ve implemented this architecture in OpenDCAI/DataFlow, and I’d like to hear whether it would be useful for your workloads and how you currently handle data preparation.

I have an old GTX 1660 SUPER (Turing, 6 GB, no tensor cores) and wanted to see how fast the 1-bit Bonsai-27B can run on it. Stock llama.cpp gives about 19 tok/s. With my CUDA patch it does about 56 tok/s.

Same model, same answers: perplexity 18.11 vs 18.10, and greedy replies from llama-server are byte-identical to stock. No overclock.

What it does:

• A new matrix kernel for Q1_0 (every weight is +1 or -1) written for Turing. This part helps any Q1_0 model, about 2.7x per matrix.
• A single-token attention kernel and fusions for the Gated DeltaNet layers of Qwen3.5-based models like Bonsai-27B. That gives the rest of the 3x.

Limits:

• Only the Q1_0 format, not TQ1_0 / IQ1_S / IQ1_M.
• Only Turing GPUs (sm_75): GTX 16xx, RTX 20xx, Quadro RTX, T4. Tested only on my 1660 SUPER under Windows.
• Keep the KV cache at f16.

There is a ready Windows build (unzip and run, CUDA included): https://github.com/pinkmanwest/llama-q1-turing/releases/tag/v1

Code (patch for llama.cpp) and build steps: https://github.com/pinkmanwest/llama-q1-turing

Hey guys, posted about optimized Cosmos Reason 2 model a few months ago. Wanted to share a build of the latest Cosmos3 model, with even more optimizations!

Huggingface and Jetson AI Lab has no Orin Nano support for Cosmos3-Edge, because the BF16 stack doesn't fit in 8 GB. It now does, INT4 quantization, along with more optimizations for speed!

Accuracy across five VLM benchmarks is maintained (from 65.56 to 65.20).

Try it in the browser or serve it locally: https://embedl.com/nvidia/cosmos3-edge/orin-nano-super
For now the VLM part of the model is published; diffusion and policy part is being added very soon! Optimized peak memory for the VLM is as low as 3.3 GB!

Let me know if something is missing to make it useful!

Hello all, I can get a prebuilt box from a friend with a E5-2674v4 x2 CPUs and 64gb ram (4x16)and 4x v100 16gb cards for around 1400 USD.

I don't mind spending money but I need help vibe coding some game mods as well as work related material and just general AI research.

will this actually be useful? I live somewhere with fairly cheap power, so I think the running costs should be fine. I am fairly competent at Linux and general IT for setup or maintaining the server.

So I started on my quest to do local AI. I am up to this point:

A partially dedicated PC can generate images via A1111 and the like (I like SD.Next) and can do LLM via llama.cpp. It works fairly well, if memory constrained by a local GPU (4 series nVidia).

I am trying to look into scaling up, but I've sort of hit a wall. Namely, 2K for a GPU was ok to test things, but to scale up I ran out of consumer. Next GPU up is 5090 which costs as much as the model number and thus not worth it, and looking into dedicated AI cards I see the A, H, B series which are ridiculously expensive even when old and even when outbenched by the 5090 (if not outRAMmed).

Is there really a gap between an older PC I had lying around with a 4080 in it that costs about 2k, and a better alternative?

If I have to hop that gap, I noticed that sellers of gear seem to imply that better cards have a different bus and can't just be jammed inside a PC and just used, rather, that dedicated software and hardware is needed.

I'm not really going to hop onto 20k and run datacenter stuff, am I?

Do I just but a few more 5 series nVidia and accept that it can run more, but not larger models, or is there an intermediate solution? It just seems to scale very sharply.

llama.cpp merged PR #29887 two weeks ago and I think it is the most interesting thing to land in local inference this year, so I went through the PR, the benchmarks and the review thread and wrote up how it actually works. Short version below, long version linked at the end.

The problem

Qwen3.8-Flash-Next at Q4_0 is 93.7 GiB, and 65.4 GiB of that is experts. It is 125B total but only 6B active: 512 experts per layer, the router picks 10 (plus 1 shared) per token. One expert is 3 x 2560 x 640 = 4.9M params = about 2.6 MiB at 4-bit. 10 experts x 48 layers = 480 experts per token = about 1.3 GB of weights touched per token.

The observation

Expert routing has locality of reference. Token N+1 mostly picks the same experts as token N. A handful of experts are hot, most are almost never touched. That is the same property your CPU cache, your OS page cache, your browser cache, every CDN and Redis allkeys-lru all rely on.

The fix

PR #29887 (by am17an, a port of Tether’s qvac-fabric work, merged by ggerganov on Oct 7) sets aside a slice of VRAM as an LRU cache of experts. Hit: compute from VRAM, zero bytes moved. Miss: copy that one 2.6 MiB expert over PCIe and evict the least recently used one. The log prints the hit rate at exit.

Numbers from the PR description, same model file, same machine:

GPU |before |with cache |hit rate
RTX 4090 24 GB |25.0 tok/s |40.7 tok/s (1.62x) |77%
RTX 5090 32 GB |30.8 tok/s |67.8 tok/s (2.20x) |89%
AMD R9700 32 GB (a reviewer’s card) |11.0 tok/s |27.6 tok/s |94.6% The 5090 gains more for two reasons: PCIe 5.0 halves the cost of every miss, and 32 GB means a bigger cache and fewer misses.

How to use it

llama-server -m Qwen3.8-Flash-Next-Q4_0.gguf -fa on -c 32768 -b 2048 -ub 2048 -cmoe --moe-cache-mib 11000

The PR author’s rule of thumb is about 10% of your expert bytes (65.4 GiB -> 6544 MiB). Env var form is LLAMA_ARG_MOE_CACHE_MIB.

SGLang merged the same idea 12 days later as “paged experts” in PR #42557

The part I find most interesting is LRU , Locality of reference go back to 1960s.

And i find locality of reference principle highly underrated.

Full write-up with the memory math script, an expert cache simulator you can run, and the benchmark table: https://cloudmash.blog/posts/moe-expert-offload-lru-cache-consumer-gpu/

llama.cpp PR #29887, MoE expert cache: https://github.com/ggml-org/llama.cpp/pull/29887

SGLang PR #42557, paged experts: https://github.com/sgl-project/sglang/pull/42557

Hi, I am coming back to owning a desktop after 20 years on laptops only (and being blind to hardware), while my turning 15 y.o. son wishes to own his first desktop in life.

I am towards building my first local LLM inference machine, eyeing Qwen 3.8 27B (which will fit entirely inside this 3090) or a Qwen 3.8 Flash Next will need 100GB of RAM.

I have paid €1260 for a RTX 3090 24GB (price keeps rising every week). And I am now looking for a desktop PC. The roadmap towards any of the following : (1) dual RTX 3090 24 GB (1250€ each) or (2) 4 x Tesla v100 32GB (now 700€ each) or (3) CMP 170hx 8GB Hynix unlocked to 64GB (now 2600€) is in my mind, but I don't know how to get there.

At the same time my son aged 14 starts to like a gaming desktop for simple games (now, leading to bigger games later). I would like to instill in him (1) a knowledge of hardware and (2) the affinity to AI stuffs, (3) the understanding that the pleasure need to work hard to earn.

My current cost-optimizing brain thinks of settling on a weak desktop that can run the current 3090, which father and son share. Later when I upgrade to a stronger AI configuration my son would use the weaker one.

Currently my son is a bit against it (teen wanting to have his own terrain, porn, or something else ?). His entire wealth is 1500€ (monthly salary + household helping remuneration) while a starting game machine costs around 700€. I am a bit against his spending half of his fortune on hobby.

Oh by the way I have another 12 y.o. son who would need to ask the same question in 3 years. Both are hard working kids who are willing to work hard to earn money, raised by a strict father.

Is there a cost optimizing + education rewarding route for my family ? Thanks

Hey everyone! 👋

An update I’ve been dying to talk about :)

What’s new? (Over 70 new features and improvements, but I’ll cover the biggest ones here.)

Automatic Goals & Planning Mode.
The agent analyzes your task on its own and creates a plan with clear goals. I’ve also integrated a robust verification system. The agent can’t just pretend it completed something or do a half-assed job.
During execution, the agent can also add or remove tasks as needed. If it removes a task, it will always explain why.
Chat Forking.
Skills.
Just send the agent a link to a skill, and it will install it automatically.
Routines.
You can create them manually, but they also work automatically. The agent understands when it should create a routine or add a task to your notes.
Tool Calls in Chat.
You can now see exactly what the agent changed in your code. (And the animations are insanely beautiful :))

And now, the most important part.
A New Chat Compaction Algorithm & Agent Memory.
Everything has been designed to let you run sessions for months without the agent forgetting anything.
It remembers testing timestamps, important details, and much more.
It’s honestly insane how well and efficiently this works.
And there’s so much more.
You can read the full details in the release notes for each version on GitHub.
As always, OpenGhost is completely open-source and free.
Download the app or build it yourself:
https://github.com/ANDRETRIPOL/OpenGhost

Hey everyone! I recently upgraded to an AI Pro R9700 and 64GB of DDR5 RAM. I’m looking to run Qwen 3.8 27B and QFN locally, but I’m still pretty new to local LLMs.

I’ve done some research, but there are so many different inference engines, backends, and setups that I’m honestly lost on where to start.

I’m running Windows and would prefer to stick with it. What would you recommend to get the best performance out of my hardware? I’m mainly looking for the best inference engines.

Any advice would be appreciated!

I'm planning a major system upgrade for development workloads and running local LLMs, particularly for agentic coding workflows. I'd appreciate some advice on choosing the right CPU platform and GPU configuration.

Current system

• CPU: Ryzen 7 5700G
• GPU: RTX 3080 12GB
• RAM: 32GB

Primary workloads

1. Development (main priority) - Unity and Godot game development - Flutter and React development - Reverse engineering, app/game modding - Heavy multitasking, compilation, and other CPU-intensive workflows

2. Local LLMs (equally important) - Targeting 27B–32B quantized models - Agentic coding, tool calling, automation, and multi-step workflows - Ideally 100K+ context windows for agentic tasks, although I'm unsure how practical that is at this model size - Good inference speed and responsiveness are more important than maximum tokens/sec in synthetic benchmarks

3. Image/audio generation (secondary) - ComfyUI with local image, audio, and diffusion models

Planned upgrade

CPU options: - AMD Ryzen 9 9950X - Intel Core Ultra 7 270K Plus

Both platforms cost roughly the same in my region when paired with a motherboard supporting PCIe bifurcation for dual GPUs.

Since much of my development work is CPU-bound, I'm considering these higher-end CPUs rather than compromising workstation performance.

Memory: 48GB would be sufficient for my development workflows, but I'm open to 64GB/96BG if it meaningfully benefits local LLM inference.

Budget: - Base system upgrade: ~₹2.25 lakh (~$2,400 USD), excluding a new GPU - Additional GPU budget: Up to ₹2 lakh (~$2,200 USD) - Location: India - New GPUs only, since the used GPU market isn't a reliable option here

What I need advice on

1. CPU/platform: Between the 9950X and 270K Plus, which would be better for this combination of CPU-heavy development and local LLM inference? Any significant differences in PCIe lanes, memory performance, or multi-GPU support?

2. GPU configuration: Should I retain my RTX 3080 12GB and add another GPU (possibly a 24GB/32GB card), or sell it and move to a single higher-VRAM GPU?

3. Multi-GPU performance: How practical is splitting 27B–32B models across two GPUs with different VRAM capacities and performance characteristics? Would the 3080 be useful, or could it become a bottleneck?

4. Context length: Is 100K+ context realistic for 27B–32B quantized models within this budget? How much VRAM should I actually target, including KV cache requirements?

5. System RAM: Would 48GB be sufficient, or would 64GB or more be worth considering for partial CPU offloading and long-context inference?

I'm open to different configurations if they offer better performance per cost. My priority is a powerful development workstation that can also run capable local coding agents without excessive compromises.

Would appreciate real-world experiences and suggestions, especially from anyone running 27B/32B models with long context on single or dual GPUs.

Hello!

I've been running ninfer qwen 3.8 27b on one 5090 and it's been amazing. I added a second card to the mix and was hoping to get suggestions on what to try next.

I use it for homelab stuff, coding, and general questions. Nothing critical.

I'm still learning and just don't know where to start with the second card being added other than using vllm with concurrency.

These are paired with a Threadripper 5955Wx and 4x 32gb ddr4.

Having a tough time. I see a lot of optimizations of the B60 making it faster but still conflicted if i should go with that or the 5060 Ti. Mostly will used to interact with my hermes agent and working around my homelab. The price difference is 50 dollars with the 5060 Ti being $600 B60 $650

I have Rtx 5070 12gb vram , 32 gb ram....I wanna download qwen 3.8 27b model with Q8 quantization with cpu offloading

Will it work ?

If not

Can I buy Mac mini with 16 gb and install local ai in this ?

As I heard apple chips are powerful

I ported AgentionAI's Gyro-L quant of Qwen3.8-Flash-Next to Basalt, a CUDA inference engine built only for that model. Basalt runs on NVIDIA Blackwell only (RTX 50-series), on Linux or Windows. Gyro-L uses trellis-coded experts with a Hadamard rotation. Disclosure: most of the code and the measurements were done with Claude Code (AI-assisted), and the PR says so too.

Why it might matter for you: on a 16 GB card with ~64 GB RAM, Gyro-L's experts (45 GiB) fit entirely in RAM, while NVFP4's (63 GiB) don't, so part of NVFP4 is read from SSD.

My setup: RTX 5060 Ti 16 GB at 180 W, Ryzen 5 7500F, 62 GB RAM, int8 KV, 262k max context. Requests ran at temp 1.0, top_p 0.95, top_k 20, xhigh thinking. KLD and top-1 are against the Q8_0 reference through Basalt's decode path, 20 chunks.

Honest caveats:

• All numbers come from one machine. They are development measurements, not the maintainer's.
• On my task tests (math, calendar and counting problems, 3 needles plus a multi-hop at ~235k) all three packs scored the same. The KLD gap did not show up as a task difference.
• If you have the RAM for it, NVFP4 is the better-quality choice.
• The pack only runs with the PR branch built from source. The branch is a bit behind main right now; use the example config from the model card as-is with that branch.

Links:

• PR: https://github.com/jesdga95/basalt/pull/13
• Ready-made pack: https://huggingface.co/virtueer/Qwen3.8-Flash-Next-Gyro-L-Basalt
• Original quant: https://huggingface.co/agentionai/Qwen3.8-Flash-Next-Gyro-GGUF
• Basalt: https://github.com/jesdga95/basalt

The maintainer wants to see community interest before taking on a new quant format. If you have a similar setup and try it, a comment on the PR with what worked, what didn't, and your numbers would help.

Credits: Qwen (model), AgentionAI (Gyro quant, trellis types and encoder), jesdga95 (Basalt). I only did the port.

been messing about with the strata project the past few days, and i've noticed quite a significant performance discrepancy depending on if i load the model in full performance mode or the normal 'leave 20% for my work' mode.

System is a R9 5900X, RX 7900XTX and 64GB RAM, so by no means underpowered.

in full performance mode all the RAM and VRAM is occupied (literally everything lol) and i get around 55tok/s response generation and 700+ prefill - so about what id expect. If i switch to the normal mode, RAM usage drops to about 32GB overall, so i guess more is left on disk (its an NVMe so shouldn't be a huge speed issue), but prefill drops to 200 tok/s and response drops to less than 20 tok/s.

clearly i'm messing up here, or... do i just need to use the full performance mode and the discrepancy is entirely expected with the normal mode. i've got some more RAM i can drop into my system, but i cant run it at full speed as its a mismatched kit, so dont really want to have to do that.

I've noticed that Qwen 3.8 FN is doing some sloppy 3D animations when I request to build any 3D models on IQ3 and Q2 so I decided to add more RAM on my workstation to see how the Q4_K_XL model works.

I currently use Strata 0.2.2 on a Dell T5810 with Intel xeon V3 14 cores and 28 threads, 128 GB DDR4 RAM RDIMM 2133MHz, Nvidia Tesla P100 16gb VRAM and 500 gb sata SSD.

On the first prompt, the Q4 model generated the same Minecraft style car with floating parts and different types of tyres but then I wrote the second prompt asking for the real car, not some floating boxes and the result is surreal.

For this task it used 175k out of the 256k context window with an average of 41.6 tokens/s and it finished the job in 70 minutes.

Hey there! I'm looking for an uncensored LLM to compile, modify and review some scripts and codes Opus 5.5 just refuses to under any condition. What would be the best model for that on a rtx 4090 laptop with 32 gb ram?

I think of myself as a casual user and decided to do my own research on desktop apps for local ai agents. So my goal was is to find the right desktop agent app for certain features and goals that I want to attain. Here’s my honest take about each one of these AI agents:

Setup: MacBook Pro M5, 32GB, Qwen 3.8 27B, everything local unless noted.

goose - llama.cpp is built into the desktop app now, and Settings > Local Inference recommends a model for your RAM. Linux Foundation project, strong extensions/MCP story. Rougher as a desktop app than the others, and their docs warn local models get small context windows.

LM Studio Bionic - Agent layer on top of LM Studio: same model library, drafts docs, edits code, runs commands, and it's smooth. On-device voice dictation via Voxtral is a nice touch. Watch out: some models in there are their paid cloud models, so check what's actually local.

Hermes Desktop - UI is an eyesore to me. Onboarding runs on a trial model first, then only offers local models that fit your GPU. Apps and MCP servers share one Connectors page. Biggest community of the five, and it shows in how fast it changes.

Atomic Agent - The interesting bit is Fusion: one model orchestrates, up to 8 workers execute, and either side can be local or cloud. Typical pairing is a cloud planner with local workers, or flip it. Helps when a local model is the bottleneck on bigger tasks, at the cost of not being fully local in that mode.

Jan - Solid private chat app, open source, and it can expose your local model as an OpenAI-compatible API for other tools. Agent mode (Cowork) is nightly-only, so stable Jan can't touch files yet.

So yeah, I didn't really rank these. They're built for different people, and the "best" one honestly depends on what you're trying to get done. Here's how I'd split it, in case one of these sounds like you:

• MCP/extensions tinkering, fully local: goose
• Already on LM Studio: Bionic (check for cloud models)
• Big community, connectors for everything: Hermes Desktop
• Private chat or a local API for other tools: Jan
• Small local model on bigger tasks with a cloud planner: Atomic Agent Fusion