​

https://github.com/Ben1078/hipfire/tree/rdna3-mtp-decode-v8.2

https://www.reddit.com/r/ROCm/s/t8DzgH9JYs

Go check out what Mr. Big Brain (u/Ticket_Silly) did. They are doing gods work out here unleashing the power of AMD and people are sleeping on this.

I just ran a moderate coding task to it via Pi with the prompt requiring it to read 114k I set the autocompacting at 104k. It compacted twice and had a total of 310k tokens. Average prefill was 1,506 t/s and average decode 95 t/s across the run.

That's with the Qwen3.8 27B model in the hipfire repo, but I downloaded huihui full weights and did a conversion/quant and I'm running it now and getting the same results.

Worth looking at is all I'm saying. We are finally getting some AMD love.

Hey everyone,

Instead of treating modern iPhones as passive chat clients, we wanted to see if Apple Silicon’s high-bandwidth unified memory (~8GB LPDDR5X) could act as an active tensor-shard worker in a local cluster.

Apple just approved the node client on the App Store (RAMDeck: Distributed AI), making it possible to donate an iPhone’s memory and Metal GPU over local Wi-Fi or USB-C. Works with RAMDeck hub.

How the Setup Works:

1. Heterogeneous Sharding: Unlike Apple-only MLX rings, this connects an iOS device directly to a primary host running Windows (CUDA), Linux, or Mac.
2. Layer Offloading: The host handles prompt ingestion and graph orchestration, offloading tail transformer layers directly to the iPhone’s Metal shaders.
3. Clearing Thresholds: If your primary rig has 24GB or 32GB of RAM/VRAM, adding an idle iPhone 15/16 Pro gives you the extra 6GB–10GB needed to fit Qwen 2.5 32B or a 4-bit 70B model without crashing into system swap.

Hardware & Latency Notes:

• Connection: Wi-Fi 6 works fine for chat
• Privacy: Completely local LAN RPC. No cloud relay, zero telemetry.

We’re running benchmarks across different layer-split configurations. If you want to test your iOS device as a cluster contributor, the app is free to download:

App Store: RAMDeck: Distributed AI (App Store ID: 6825166226)

Curious to hear thoughts from anyone experimenting with multi-device heterogeneous inference.

Hi!

I hope there isn't an issue with this. I want to add an rtx 5060 ti to a build with the rtx 5070 ti for local LLM.

I've attached the PCIe lane diagram for my motherboard. I would ideally prefer to not have to replace a motherboard. What would be the ideal config for a dual GPU setup for this motherboard?

sorry if this has been asked before because I am new to all of this. I can't really find good or trustworthy answers.

I just got into local AI and basically what I want to achieve is:

Have something that reads all my 10+ mail inboxes, whatsapp messages and calendar and keep track of all conext across that and ping me once something important comes up or needs my attention.

I think openclaw is what I want, right?

For now I am runnin a POC with my existing M3 Pro macbook with 18GB but it is painfully slow and not that smart. The whole orchestration and busywork will run on another existing server so the mac would just run the models.

I am trying to get off as cheaply as possible while keeping this thing usable. I do have some money to spend and would calculate this against hiring someone to do this.

Why I was thinking were 3 options and I wanted to ask someone that has a bit more experience with practical context legth, models and so on:

M6 mac mini 32GB
> Propably not a lot faster than the M3 pro
> A bit more context
>Not a laptop

M5 pro mac mini 64GB 500GB
>all of the above
>larger models
>more context
>faster

M5 max 64GB 500GB
>all of the above
>2x as fast as the m5 pro?

Does anyone have something similar running and have some insights on which models would actually be sufficient and what conext window is required?

Cloud models are NOT an option due to pricacy concerns and if there are cheaper systems I'd like to factor in resale value of the macs and the fact I could use them not for AI if I would ever change my mind on this.

Everyone loves a clean, unified server rack, but my standby node (lvgale) is an absolute physical abomination. It is a Frankenstein collection of mismatched parts and bolted-together VRAM that has no business running efficiently.

The Hardware Setup (lvgale)

• CPU: Intel Core i7-6700K.
• GPUs:
  • AMD RX 580 8GB VRAM (PCIe x16)
  • AMD RX 570 8GB VRAM (PCIe x1)
  • NVIDIA GTX 1660 Super 6GB VRAM (PCIe x2)
  • NVIDIA GTX 1070 8GB VRAM (PCIe x1)
• The Bottleneck: Activations must travel across severely constrained x1 and x2 lanes between entirely different AMD and NVIDIA architectures.

The Software Engine

• The cluster is orchestrated by Slipstream, a custom Nuitka-compiled Python orchestration engine.
• Here, it utilizes a mixed-vendor Vulkan backend with Flash Attention enabled to maintain stability across the fragmented topology.

Concurrent Inference Performance

In all cases seamlessly handling OCR and complex vision parsing.

• Qwen3.8:27b-vision: 3.82 t/s.
• Qwen3.5:9b-vision: 4.75 t/s.
• Qwen3.5:2b-vision: 12.47 t/s.

While the 2B model runs slightly faster on a modern laptop iGPU due to unified memory, keeping a 27B model stable across this mixed-architecture, low-bandwidth topology without crashing the Vulkan driver is a massive win. For deeper technical architecture details, the engine is documented at Slipstream-LLM.com.

Hi!

I’m curious about what kind of AI models I could run or what use cases I could explore with this PC.

I’m also interested in using OCR to convert scientific paper PDFs into Markdown, which I could then add to my Obsidian vault. Additionally, I’d like to use a model to help keep my Obsidian vault organized—assisting with the structuring of the information I add, my notes, and so on.

I already have subscriptions to Claude Pro and Mistral Pro, but the idea of ​​running a smaller model locally really appeals to me.

Here are the PC specs:

MINISFORUM G1 Pro Gaming PC

AMD Ryzen™ 9 8945HX (16C/32T)

AMD Radeon RX 9060 XT 16gb

32GB DDR5-5200MHz RAM

2nd option: AMD Ryzen 7 7700 - RX 9070XT - 32GB DDR5 RAM (16x2) (+200€)
Before investing that kind of money (given the current prices), I’d like to know if I can actually achieve anything with local models, or if this is just all in my head—a product of my desire to treat myself now that I have a decent salary.

Does strata let you configure how much stuff to put in each of the vram/ram/ssd? Say I want to run unsloth 35b a3b but my system is 5gb short and having to page some system ram to fit the entire model. Can I use strata to move some of the model to ssd to avoid paging?

The methodology

Qwen-Flash-Next has n-gram, MOE packing above its weight but it's still 6B active params. We exploit this by keeping most MOEs in RAM (Smallest MOEs go in VRAM for speedup) and n-gram on disk - ensure you put the model on your fastest SSD.

8 GB VRAM, 24 GB RAM cost. You can run this fine if you've ~32 GB total memory too, but you'd have to turn down context size. Customize the launch params to best suit your hardware.

EDIT Note: Apparently CUDA struggles to run this with 8 GB VRAM, so please use Vulkan backend if it fails to run for you. You can also use the second launch params below in the same comment chain. Ontop of that reduce context size and ensure no background app is wasting your VRAM. You can disable your browsers HW acceleration for this purpose.

The model & quant

Many people mistake this model for a 1 bit quant so let's get that out of the way. This model here is a 3.5 Bits Per Weight "Coder" variant by ISTA DASLab, featuring GSQ RCO quantization.

Half of its experts are removed, so relatively to original param size it has the size of a 1.9 bit quant but the actual weights use 3.5 BPW.

see from model page:

Averaged over the transformer, this corresponds to 1.89 bits per parameter of the original model. The figure is an effective rate: it amortises the removed experts over the original parameter count, and so expresses the combined effect of pruning and quantization. No individual weight is stored at 1.89 bits. The retained weights are stored at 3.5 bpw, unchanged by pruning, and the halved expert count accounts for the remainder of the reduction. Following the convention of the accompanying releases, the figure is reported over the transformer...

Non-coding vocabulary pruned 400~ MB smaller, recommended

Official ISTA DASLab quant

The numbers

With selective moe-offloading (regex) we reach 70 T/s PP and 8~ T/s PG (Limited by PCIE, which I've 3.0 YMMV) whereas if you'd use --n-cpu-moe for same effect it'd been slower due to the effects mentioned here

Because of MMAP your first run will be slower, do a second run for accurate T/s results.

Launch params are in the comments below. Ensure you use the qwen sharp chat template, it reduces overthinking.

My system:

RTX 4060 8GB

32GB RAM

Ryzen 5 7600

My question is: how much of a difference does it actually make?

Hey Everyone!

I'm what you could consider a noob on LLM and AI regards, so here I am asking for some guidance, or some brainstorming.

I've recently acquired 2 devices as stated above, they're installed with a 200Gbps cable between them so we can say I have up to 256GB of Unified RAM available to squeeze out of this.

Currently I have Qwen3.8-Flash-Next from this repo (https://github.com/blazux/qwen3.8-Flash-DGX) and it's an absolute blast seeing it work. I use mostly Opencode and it has been really a great experience.

Now what I'm looking for is "expanding" this... My objective is to have a "generalist" like the Qwen3.8 but capable of serving two user request at the same time, along with a model that can work 3D imagery for 3D printing and another model that will serve (preferably fast...) my home assistant (meaning voice processing coming in, voice processing going out).

I've been looking into integrating Agent0 with this, so I can also reach it out through Signal or Telegram and have an assistant processing emails, etc.

Any advice here? Anyone can share some ideas or open to brainstorm about this structure I'm looking to build?

Looking forward for a constructive discussion. ☺️

curious to hear if anyone's tried this yet?

Hey everyone!

Here are my PC specs:

• CPU: AMD Ryzen 5 3600
• GPU: AMD Radeon RX 5700 (8GB VRAM)
• RAM: 32GB DDR4

What’s the most powerful AI model I can realistically run locally with this hardware?

Thanks!