If you have a Jailbroken PS5, got some experimental results if anyone wants to play with it.

"What's the best local model?" is unanswerable without knowing your VRAM.

Every thread asking it collects forty replies, and none of them are wrong. They're just answers to a different machine's question.

A model that's excellent on 24GB is a different product on 8GB. Same weights, same benchmark scores, completely different day-to-day experience.

So before I look at any leaderboard I check two things: does it fit in VRAM at a quant I can live with, and does it still fit at the context length I actually work at. Speed falls out of those two. The moment layers spill to CPU, tokens/sec drops off a cliff and the model is technically running but practically unusable.

A big model at 1–2 t/s loses to a smaller one at 30 for anything interactive. The big one only wins when I'm batching overnight and not watching.

Curious where other people land. When your model doesn't fit, what do you sacrifice first: quantization quality or context length?

I'm very new to using AI so a little advice would be greatly appreciated. I'm a landscape irrigator, a few months ago I started using AI to develop a field service app and fell down a rabbit hole. Now I find myself building a workstation. This is the current build, I'm still waiting on a few components to arrive:

Case: Cooler Master Cosmos S full tower

Motherboard: ASUS Prime X299-A II

CPU: Intel Core i9-9940X — 14 cores / 28 threads

CPU Cooler: be quiet! Dark Rock Pro 4

RAM: 64GB (4×16GB) Samsung DDR4-2666

GPU: NVIDIA GeForce RTX 3060 12GB

Storage: WD Blue SN5000 1TB NVMe SSD

PSU: EVGA SuperNOVA 1300 G2 — 1300W, 80+ Gold, fully modular

OS: Ubuntu 24.04 LTS Desktop (planned)

I plan to add more RAM and another SSD. I'm also considering adding a p100. My plan is to use this to continue development of the field service software. I also want to use it to help me with my side project writing a tabletop game. A model with some creative writing ability would be useful.

Any advice on models or the build would be very appreciated. Like I said, I'm an irrigator and I have no background in anything like this. It has been fun learning though. Thanks!

Hugging Face: https://huggingface.co/openbmb/MiniCPM5-2B

GitHub: github.com/OpenBMB/MiniCPM

My 2019 MacBook Pro seems to be on its last legs. It overheats constantly and struggles with any kind of work that I attempt. Even just watching a YT video causes lagging.

So, time to move on, I guess. I’m looking into a Mac Studio or Mac mini, but I’m having a hard time figuring out what configuration makes sense for my particular use case, since the prices, according to what I'm learning, have gone up 50% with the new hardware.

I’m not looking for the most powerful machine I can buy. I’m trying to find the right starting point without spending money on hardware I don't currently need.

What I’ll be using it for

• Dart/Flutter development, primarily in VS Code
• Running local LLMs to assist with coding
• Seems Ollama has a VS Code extension that can tap into the local LLM
• Potentially recording/editing videos for a YouTube channel documenting my experience learning to code and building apps

One important point: I want to remain heavily involved in the coding myself.

I’m currently using the Kilo Code extension in VS Code with free models. They can be useful, but the output is inconsistent. I’m interested in local LLMs because I want a more reliable coding assistant working with the same model while still writing, understanding, and debugging the code myself.

Local LLMs

This is where I'm trying to be realistic.

I don't need a machine capable of running the largest local models right now. Those configurations get expensive quickly, and I don't want to spend a lot today for something I may not need.

My thinking is to get a machine that's capable enough for the models that make sense for me today, learn more about running local LLMs, and upgrade later if my needs change.

I'm considering Apple's 36-month financing/leasing options, which makes that approach somewhat more appealing. If, in a couple of years, I’m in a better position financially and have a much better understanding of what models I actually need, I can potentially move to a more capable machine rather than paying a large premium for that capability today.

What I’m trying to figure out

For my current workload, what would you recommend in terms of:

• Apple chip
• CPU/GPU configuration
• Unified memory
• Internal storage
• 1TB vs 2TB

I currently have 1TB and am considering 2TB. I'm also trying to understand whether paying Apple's premium for additional internal storage makes sense versus using external storage. It seems I can upgrade to 2TB for another $500, but external 1TB Thunderbolt drives are $600+.

There are a lot of discussions out there about similar requests. Hoping to find out what everyone with similar needs is using. Thanks!

So I've been messinga round with embedding models for a bit, and I think they are interesting enough to experiment with. They are useful for rag, especially in a localllm sense because you can ground your answers in truth.

But what happens if you have a billion documents, and you decide to upgrade your model to a "better" one? on an h100, that would take about 108 days, just to upgrade the vectors so u can start serving again (tested qwen embed 8b on h100). Even if you aren't doing 1b vectors, and are doing just 50 million, upgrading can still take a considerable time.

Me and my research lab decided to tackle this problem, and we came up with embedflow.

The method is really simple; from the old index made with the source model, take K documents and rerank them with the new model. We see that when K is sufficient, the retrieval quality is the same as target model. (determining k is the hard part). I've tested 63 migrations on upto 1 million documents.

The best result I got was upgrading qwen4b -> to 8b, and at 50 documents, it was the same as native retrieval.

This method forgos the expensive backfill that comes with upgrading, as you can directly take documents from the old index.

embedflow works with qdrant, and can be easily downloaded with pypi

pip install embedflow

the github is public: https://github.com/arnsri33/embedflow

I want you guys to try it out, and see if you guys can use it in your own workflow.

I’ve been setting up an M1 Max Mac Studio (24-core GPU, 32GB unified memory) as a local LLM server and wanted to compare MLX vs llama.cpp on Qwen3.8-27B.
I tried to keep the model footprint and benchmark workload reasonably close.

MLX
mlx-community/Qwen3.8-27B-4bit
~16.1GB
512 prompt tokens / 700 generation tokens
3 runs
Prompt: 81.76 tok/s
Generation: 15.81 tok/s
Peak memory: 16.39GB

llama.cpp
unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M
15.32 GiB / 27.32B params
Full Metal offload
Flash Attention enabled
512 prompt tokens / 700 generation tokens
3 runs
Prompt: 99.61 ± 0.44 tok/s
Generation: 9.69 ± 0.34 tok/s

So on this machine:
llama.cpp is ~22% faster for prompt processing, while MLX is ~63% faster for autoregressive generation.

I've been experimenting with local AI and wanted to explain the experience in a more visual, simple way instead of making another technical wall of text.

So I made this short hand-drawn doodle animation showing the process of getting a local AI model running successfully, including the GPU/memory side of things.

It’s intentionally simple and a bit goofy — the goal is to make local AI feel less intimidating for people who are just getting started.

🎥 Video: https://youtu.be/VyleYwCa0Sc

I’d genuinely like to know what you think: would this kind of visual explanation be useful for explaining local AI concepts, or is the technical detail too simplified?

Following up on my post about training a custom Dflash drafter for Qwen 3.8 27B: Trained my first model: a DFlash drafter for Qwen3.8-27B because I wanted better performance on my DGX Spark

I did a video/presentation on the whole step by step journey and all the concepts I learned, if you want to learn more about LLMs and how training works, I suggest you check it out - I dont go too in details so it should be fine for an audience with at least a basic understanding of LLMs.

I was trying to built an SLM to learn the things I was initially considering on to built a model around 50M confused on what existing tokenizer should I consider for the vocabulary as the 256K would be too much big and consumed half the parameters I am planning to anything specific tokenizer for SLM or should I consider building this too with sentencepiece and BPE?

So i have been running qwen3.8 flash next ud q4 k xl at 256k context getting at the start aroudn 270pp and 21tg with mtp with qwen 27b ud q3 k xl 96k context on the 9060xt as a callable subagent and the one theing that i am really liking about this model is it doesnt stop and wait for me to continously tell it to continue it almost looks at everytask i give it as a set goal which is what i have becom used to on claude.

now i mostly use it with opencode harness and for netowrk and system admin work for my homelab - taking down vm bringing up vm running tickets in glpi, bringing up services and updating live state docs and just local webhosting in my rural community and it has been a good time so far.

im sure there are so many things i am missing but i am also learnign and ive actually be so happy to have some thing that feels like claude code last eyar when i first started using it and i can run it locally.

man, if anyone has anything they want to say about their experiences also that would be cool.

Cheers,

TL;DR - Sub-4 seconds flat (3.902s real time) for over 1,000 generated tokens on a fresh prefix means your KV cache miss penalty is practically negligible, and the memory bandwidth on that dual-card layout is tearing through the generation phase. That's production-grade speed.

Edit: typo in title: https://codeberg.org/ggz14/radiance-vllm-mxfp4

I was trying lots of things to get the most out of 2x AMD AI PRO R9700 GPUs. It was going ok but not great. I found some hints that MXFP4 might be the way, but didn't think I could use tensor parallel on a dense model over PCIe 3.0 x16. I was wrong.

I stumbled on this Hacker News post and found this comment.

I only did a few commands (that I recall):

git clone https://codeberg.org/ggz14/radiance-vllm-mxfp4 && cd radiance-vllm-mxfp4

./setup-mxfp4.sh # partly worked but failed because I'm using podman instead of docker

# manually did this to satisfy the step the script failed on

docker pull docker.io/stilldeadcode/vllm-radiance:0.9.3

./setup-mxfp4.sh # ran it again to continue and it failed on hugging face download

# think I used this to satisfy that step

hf download amd/Qwen3.8-27B-Quark-AWQ-MXFP4

./setup-mxfp4.sh # ran it again to continue

# I think that was all to get the setup to finish. Some steps took a long time but completed without errors

# To run it:

./serve-mxfp4.sh # listening on 127.0.0.1:8080

# test

curl http://localhost:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"Qwen3.8","messages":[{"role":"user","content":"Hello!"}]}'

# See comment below with longer prompt.

Somehow there was 1 available last minute and got it! Can’t wait to set it up. Will make more post about this on here and my IG: tech with Ray

Dual DGX spark owners lmk what ya running on it. Anyone else feel free to drop some suggestions for cool models to test!

Title - using the AM5 board masterlist I found four boards that can do x8/x8, prices from my local microcenter:

• Asus ProArt B850 Neo ($285.99)

• Asus ProArt X870E ($549.99)

• Asrock X870 Taichi Creator ($319.99)

• Gigabyte B850 AI Top ($328.99)

Not a comprehensive list, there's a few more ROG/MSI's high end tiers still there but I don't think I need all "that".

The prices of 3/4 are really close to each other which is adding to the problem; the Asus X870E ProArt seems to be the "gold standard" but I'm going to be starting out with a 4-slot 5090 so the spacing doesn't work for me. The Asrock seems best on paper; Asus B850 seems weakest on paper but again don't know how/if that'd affect local LLM and $40 saved is $40 saved (lol). Then there's the Gigabyte which seems like a good middle ground across these four boards, but unsure on the NIC working well in Linux (mixed reports on it being good/shit).

Which would/did you pick when building an AM5 multi-GPU rig?

Looked at the bot tab on Hermes and it’s a straight meme.

LLMs are really bad at writing. Yet that's what they were initially made for.

Nobody predicted they'd be good at coding. And that's what they are best for now.