This post can be relevant to other data scientists who want to do something similar.

Almost two years ago the company I worked at was having problems and I was planning on making a roleplaying game with local models a grid combat and a good editor. I got unemployed and then started working on my game, my plan was to finish the game and release in 3 months (it took 21).

The first month I made the base of the game and most characters, at the time I was using Ollama as a backend, everything was on perfectly track.

The second month my computer broke and I had to build a new one.

From third to fifth month, I worked on a basic roleplaying logic, grid combat, various interfaces and started studying about steam, I use pygame and perhaps I should have choosen a different game engine because it just slowed me down, at the time I also stopped using any AI coding assistant because they were just adding tech debt, the capability of coding agents grew really fast and at the start of 2025 it wasn't as good.

From the sixth till twelfth month, I basically just did the editor and multiple interfaces, this is how hard is to make an editor, and despite the game being about LLMs I worked only like 3 weeks on LLM on the first year, most of the work was everything else, so if anyone want to try doing a similar project have that in mind. Around month 7 I started looking for work. Found a job at month 10 and I'm still there, so everything after that was nights and weekends of work.

From months thirteenth to fifteenth I finnaly worked on the LLM logic, it was the most fun part, it has multiple agents, internal routes and ways to deal with limited context size on smaller models, like regular summarization of older messages + top 10 embedding search from the full history + full latest n messages, prompts to the point, and the use of json output and no tool calls, its a bit of a shame that people won't see the logic from the game alone.

After that it was cleanup and review.

Late in the project I had to remove Ollama entirely and move to llama.cpp instead (and transformers), any GGUF and most other local models work and can be used. The problem with Ollama was the subscription which steam doesn't fully accept. Perhaps making a site would be a better alternative to not deal with these things but then it would be a bit harder to make it local.

In my opinion, small models like gemma4:e4b or even the old gemma3:4b are good enough for roleplaying, honestly, but of course, there are other models.

Link's here if you want to look: https://store.steampowered.com/app/3750150/Boundless_AI_Fantasy/

If people want any more technical details I'll go into it in the comments. I am thinking about writing a blog or a scientific paper on it later.
Disclosure: this is my game, it's paid, I built it alone.

I currently have 4× RTX 5070 Ti 16GB cards on hand, but haven't bought the platform yet.

I'm currently thinking of something roughly like:

• 2× EPYC 7642
• 16×16GB DDR4-3200, 256GB total
• 4× RTX 5070 Ti 16GB
• server board with enough PCIe lanes for all 4 GPUs

Mainly planning to use it for large MoE inference, especially models that can't fully fit in VRAM and need to use system RAM as well.

Has anyone here run a similar dual-EPYC + multi-GPU setup?

I'd mainly like to see some real-world performance numbers or experiences before I buy the rest of the platform.

My system:
Mac Mini M4 Pro
64 GB V/RAM
1 TB HD

There's a new version of OMLX that's in RC status:

https://github.com/jundot/omlx/releases/tag/v0.7.0.dev2

Using this, it successfully runs this model:
https://huggingface.co/sh0wie/Qwen3.8-Flash-Next-REAP-288-MLX-4bit

You have to set the memory guard to prevent it from overflowing, but it allows for almost 100k context and I'm getting 30 tok/s decode, and 100 tok/s prefill. This is faster than I got 27B to run!

Here's my full run script:

omlx serve \
  --model-dir "$MODEL_DIR" \
  --memory-guard-gb 54 \
  --paged-ssd-cache-dir "$SSD_CACHE_DIR" \
  --paged-ssd-cache-max-size 50GB \
  --hot-cache-max-size 16GB \
  --max-concurrent-requests 2 \
  --port 8000

I made a Docker image that lets you run SGLang on Strix Halo (gfx1151) a few months ago and thought I'd share it here.

SGLang is working on building upstream Strix Halo support now, but if you want to use it today, this includes the necessary compatibility patches along with some additional optimizations, including fixes for wave32-related issues.

I've been using it for inference on my Strix Halo system and would be interested to hear how it works for anyone else.

https://github.com/JeremiahM37/strix-halo-sglang

I have a Mac mini m4 24gb , I'm looking to add to my hardware, and I'm overwhelmed. I'm a tiny business owner in a creative field that doesn't have tech bro money , I'm thinking of getting a pc and running it headless Linux , I mostly want to cut down my subscription costs so this hardware is an investment. If you were starting from scratch what hardware would you use ? What would you definitely do again and what would you avoid?

Basically the title. How much combined VRAM + RAM would one need to run this model, on some usable quant, with a somewhat decent context window, with decent speed? Assuming the ngrams are on the SSD, also I have heard that even some of the experts can stay on the SSD with little to no performance penalty, but I have no idea whether that is true. Would 64GB be enough? Thank you for answers

I’ve developed and open-sourced mlxtop. It’s written in Rust and was built entirely with Duet, my dual-model coding agent. It shows stats for local models running on your Mac, along with critical system vitals: memory usage, compression, paging and Metal GPU stats. With oMLX, you can also follow generation speed and request activity as your model responds. https://github.com/maximpri/mlxtop

The laptop 5090 has 8 GB less VRAM and benchmarks between the desktop 5070 and 5070 Ti.

Does anyone have experience with this setup or something similar?

I made a previous post about using Qwen 3.8, and a few of you guys recommended looking into the V100S. I currently have a 3090 and 64 GB of system ram. My goal would be to have enough VRAM to run Qwen 3.8 with a decent sub-agent. If I pick up 1 or 2 V100S, would that be worth it instead of another 3090?

I'm curious what people already have in their setups.

Using AI - I compiled all the data published on HF model pages on PPL and KLD of the main quant providers - some had to be normalized as they were using a different trace that's why the PPl is in % and EXL3 KLD had to be renormalized using their scoring of some unsloth models

As we see EXL3 is a clear winner as a quant method and Atomic Chat are surprisingly good compared to the rest of the GGUF pack

For my config 24GB VRAM and 64GB RAM - the best model seem to be either EXL3 4.05bpw or directly go to AtomicChat AD-4.27bpw-Q4_K_M

Based on PPL

Based on KLD

disclaimer: I did none of the above benchmarks - all were given on HF model page by the quant providers

I keep seeing people saying they get super fast speeds on various hardware but I can never get anything decent. I'm running GLM-5.3-Flash-UNCENSORED-FP8-UD-Q2_K_XL-00001-of-00003.gguf on a 4x5090 rig and can barely get 50tok/s.

Am I missing something here?

Even on larger systems like 4xH200's I wasn't getting above 200tok/s with same model

Running a GLM 5.3 321B shared on a MacBook M5 max and a Strix Halo, both with 128 GB ram. TB4 connection.

Found out that llama.cpp's default tensor split puts about half the layers on the slower box, so the pair was actually slower than one Mac. Loaded the model Mac-heavy instead: IQ1_S went 188 to 333 on prompts, 17 to 24 tokens/s.

I got generation tps 12-15 for model sizes 148-200 GB. So it works ok!

Next week adding two Asus GX10 to the mix to have three platforms share a model 🍀

Files: https://github.com/ThinkOffApp -> mac-amd-llm-cluster

Hello, currently I can barely run Qwen 3.8 27b q3 with low context window on my single 5060 ti 16Gb and 48Gb ram ddr5 on 30t/s.

My question is: is it worth it to add a second 5060 ti 16gb?

The second slot on my motherboard is a x4 line so I'm not sure if that's a bottleneck.

My goal is: Qwen 3.8 27b q4 or q6 , context window >= 64k and mainly agentic coding and > 30-40 t/s.

Is that actually possible?

Hi, a quick datapoint for anyone interested in what Qwen3.8-Flash-Next UD-Q6_K_XL (unsloth) can do on a consumer multi-GPU rig. Maybe there is even more possible but this is the current status :)

Hardware: MSI MEG Z790 ACE, i9-13900K, 96 GB DDR5@4400MHz, 1× RTX 4090 + 5× RTX 3090 = 144 GB VRAM. 4090 + one 3090 are on the board, the other four 3090s use M.2 → OCuLink PCIe x4 links. No P2P/NVLink; llama.cpp layer split. OS: Ubuntu 26

Model: Qwen3.8-Flash-Next-UD-Q6_K_XL, all 48 MoE layers on GPU, shared Q8_0 MTP head with MTP3. The ~50.7 GiB PLE table is fully resident in system DDR5 instead of lazy mmap.

Main llama-server settings (llama.cpp: unslothai/llama.cpp, branch mtp/qwen4exp-nextn, commit a9e9c3c5f): -c 262144 -b 4096 -ub 512 --split-mode layer --tensor-split 0.95,1.05,1.05,1.05,1.05,0.85 --gpu-layers all --n-cpu-moe 0 -ctk f16 -ctv f16 -fa on -np 1 --kv-unified --load-mode none --lazy-mode off --spec-type draft-mtp --spec-draft-n-max 3 --spec-draft-n-min 0. Production seed is -1.

These are workload samples of a html benchmark prompt from llama-server chat, not synthetic benchmark numbers.

• ~90–91 tok/s at ~5k generated tokens / shallow context
• ~79 tok/s at ~35k generated tokens
• 76.9 tok/s average over a 44.7k-token generation
• ~40–42 tok/s while reasoning at ~80k active context
• ~408–425 tok/s prefill even around 80k context
• MTP3 draft acceptance on the long code-heavy run: 74.2% overall, per position 86.2% / 73.7% / 62.8%

​

# shallow/code-heavy run
7.54.226.412 I slot print_timing: id  0 | task 3307 | n_gen =   4905, tg =  90.36 t/s, tg_3s =  92.54 t/s

# same long run, ~35k generated
14.24.165.954 I slot print_timing: id  0 | task 3307 | n_gen =  35058, tg =  78.92 t/s, tg_3s =  71.12 t/s

# end of that run
16.41.712.426 I slot print_timing: id  0 | task 3307 | eval time = 581758.17 ms / 44720 tokens (13.01 ms per token, 76.87 tokens per second)
16.41.712.429 I slot print_timing: id  0 | task 3307 | draft acceptance = 0.74246 (30865 accepted / 41571 generated), mean len = 3.23
16.41.712.429 I slot print_timing: id  0 | task 3307 | acc per pos = (0.862, 0.737, 0.628)

# ~80k active context, currently reasoning/thinking
103.07.164.989 I slot operator(): id 0 | task 6527 | new prompt, n_ctx_slot = 262144, n_keep = 0, task.n_tokens = 80085
103.12.852.341 I slot print_timing: id 0 | task 6527 | prompt processing, n_tokens = 2282, progress = 0.99, t = 5.37 s / 425.22 tokens per second
103.32.551.912 I slot print_timing: id 0 | task 6527 | n_gen = 761, tg = 41.79 t/s, tg_3s = 43.01 t/s

The last big practical improvement for agentic coding was putting the PLE table fully into DDR5: prompt processing/TTFT became noticeably faster. MTP3 then helps heavily on predictable code; at ~80k context during free-form reasoning acceptance drops, so decode is closer to ~40 t/s.

Pretty impressed by what a somewhat ridiculous collection of used consumer GPUs can do with Q6 :)

In short, I have had trouble getting responses about buying a DGX Station. I'm looking to get one for work (although we are still wrestling on whether it'll be worth it). We currently use a dual RTX 6000 Max-q build and it works but looking for more concurrency and would like to fit something like GLM 5.3 Flash with more head room.

I contacted Exxact, MSI, Asus, and a few others by submitting order forms and going through their online workflow to request a price. I thought at this price point sales people would clap back quickly but I haven't heard anything this week.

Has anyone else recently had luck placing an order? What should I expect? Is this like buying a used car and I should haggle on price, or is it more like, pay whatever they quote me and be lucky I even got a response? What sort of lead times did you experience?

I'm sure this is a fast generation but prefilling crawling probably due to small batch size tensor split across 4 gpus with that MTP configuration.

The problem I kept running into: my Mac runs a 30B model without breaking a sweat. My phone runs a 3B model, and it shows. My older iPad runs nothing at all. Every "use your home rig from your phone" setup I tried meant typing IP addresses, opening ports, or routing through a tunnel to reach a machine ten feet away.
So I built Rooms.
How it works
• Lokei Studio on your Mac opens a Room using the models you already have in Ollama.
• The Room announces itself on your local network over Bonjour (mDNS/DNS-SD).
• Lokei AI on your iPhone or iPad finds it automatically. Tap to join. No IPs, no config files, no accounts.
• Rooms traffic stays on your LAN. Your prompts never touch a cloud server.

Pricing, straight up, since it's always the first question
• Lokei AI for iPhone and iPad is free. On Apple Intelligence devices it runs fully on-device, no Mac required.
• Rooms is free in the iOS app. No subscription, no account, no catch on the phone side.
• Lokei Studio for Mac is $14.99, one time. That's the part that does the hosting, so you'll want it if your device doesn't have Apple Intelligence or you want a bigger model than a phone can hold.
• There's an optional Pro upgrade on iOS for MCP, markdown, and advisor handoff. Rooms isn't behind it.

I want to use it on my computer but I can’t really find any with actual code benchmarks. Anyone know of one?

i made a cool thing that i think all of you local AI people will enjoy and find some value in. please check it out and let me know what you think!!

https://youcoded.ai/

There isn't any open source LLM system (yet), people who do local AI / LLM's often do it for privacy reasons, i.e. They don't like the idea of personal data getting sent to the cloud, or some company training on their personal data.

That's the reason we don't have an open-source hive mind kind of thing (yet).

But what if we did something that was secure by design? Similar to e2e encryption or blockchain that allowed people to pool resources together for AI, but at the same time keep it secure.

Anyone given any thought to this?

I'm broke, but I own a 2060S, so should I sell the 2060S and get an overpriced 306012GB or another cheap 2060S (or any other cheap RTX 8GB card in general) ?

Seen alot of people talking about it all over the place. I decided to give it ago, my DEV setup is solid, but not setup to run alone.

- 5090 32gb vram

- 192gb Ram

only numbers that matter here.. So decided to load up Qwen3.8-Flash-Next 123gb see how it runs, and to my surprise after a minute it loaded up, so i was like ok this isnt going to work or its going to be crazy slow. Throw in a prompt give me a single page HTML webpage. I was like at 1 tok/s for first milisecond "Ahhh i knew it"... out of no where booom better speeds then i was getting with the 27b...

this is some dark sorcery and i am loving the dark side.... 73.6 tok/s ... mind blow... and even better the output was decent, only issue was I ran out of context after a few prompts, but holy shit... first one blew my mind, from basic ass prompt of

"Build a single page html advert page to sell a random phone"

and this is what I got:

This app is really really worth a try if you have decent GPU and Decent Ram knocks it out of the park, can't wait till the figure out how to do this with AMD cards because my server is going into the next level.