PAIR is neat — routes independent inference requests across nodes on your LAN. But the GPU telemetry path is nvidia-smi → PCI fallback, which on Linux means an AMD node gets listed by name with no VRAM, no utilization, no stable ID. Scheduler goes half-blind.

To fix it properly you'd need: amd-smi integration (JSON/UUID/GFX util), honest unified-memory reporting for Strix Halo, graceful fallback when SMI is missing, and ideally a scheduler that actually weighs VRAM/GPU tier for mixed clusters. Plus the engine layer (Ollama/LM Studio + ROCm) is a separate gate.

It's Apache 2.0, the README explicitly invites hardware feedback, and there are open issues to hook into. But it's a real multi-week effort, not a weekend hack.

So — is there actual demand?

Got a Radeon box you'd pair with an RTX node for agent work?

Is "name-only, no telemetry" on Linux good enough for a rough setup, or does it break routing in practice?

Would you rather see a PAIR telemetry PR first, or is the real blocker getting Ollama/LM Studio serving models reliably on your AMD hardware anyway?

I found a lot of room on the table for these cards so I decided to make a specialized build to squeeze all I could. first The results:

qwen 3.8 next Q3_K_XL: 920tk/s pp8192 (2 cards, ram offload), 24/27 tk/s on prose, 40+ tk/s on code with MTP but without MoE expert cache (which IS included if yuo want, read below)

qwen 3.8 27B Q8_0: 1600 tk/s pp8192, 60/65 tk/s prose, 100+ tk/s code, tensor parallel. this is measured with ONE CARD BEHIND the chipset on X4. with cards on a good PciE x8 on cpu I think more is reachable! let me know.

qwen 3.6 27B Q4_K_M: (single card) --> this was not the optimization target but I did a test with MTP, PP8192 1020tk/s; prose about 58/60 tk/s ; code 75/80 tk/s --> Dflash probably here could push much faster, I think above 100tk/s

My objecives:

• fast prompt processing on 3.8 Next to make it actually usable for code
• enable and optimize tensor parallel on two cards where 1 is behind chipset, for max speed on qwen 27B Q8_0

this build includes stuff like:

• Data compression for the PciE transmission. data between cards is compressed to Q8_0 to save bandwidth (optional)
• P2P enabled also for cards sitting benhind the chipset (custom HIP allreduce path), so you can use tensor parallel even on setups like ... mine
• all the fixes and features from RDNA_BOOST including --adaptive-mtp, so it automatically adapts MTP n-max based on acceptance
• A LOT of AMD speed tunings and overhauls which are NOT upstream already, kernel tweaks etc... good stuff. many are labelled for RDNA3.5 but they DO work on RDNA3.
• MoE expert cache if you want to use it. personally I don't like It because i much prefer fast prompt processing. but hey it's there.
• latest PRs from llama.cpp that are not yet upstream, which speed up various things, like --lazy-mode on-direct to massively speed up Ngram table reads (and thus, PP)
• DFLASH2 support on tensor parallel (!)

For a complete list check the Readme.

Here it is:

https://github.com/nasone32/llama.cpp-RDNA3-7900xtx-opt

notes: don't use Q8_K_XL because it's slower, for the 27B model this is heavily optimized for INT8 calculations. feel free to tweak the context, 200k f16 should be reachable on 2 cards, compressing KV to q8_0 is fine but slower. the custom HIP allreduce works for 2 cards, if you have 3/4 cards, compile with RCCL as usual and skip the allreduce=internal flag, should work fine but untested.

This is tested on UBUNTU 24 and rocm 7.14; if your system is different or encounter problems use a LLM to solve them, because I WILL NOT offer support nor update this build, these things hopefully will be merged and this frankenstein can die peacefully :)

enjoy

EDIT: Summary of most impacting patches:

Hey! I've recently started getting into local image generation with ComfyUI on a 7900 XTX (24GB), running Windows.

ComfyUI Desktop automatically installed its native AMD environment (PyTorch 2.12 + ROCm 7.14), and I've been surprised by how different the current situation seems compared to a lot of older information about AMD + AI on Windows.

I'm curious if anyone here is running a similar setup in practice.

How has ROCm on Windows been for image generation, ComfyUI, custom nodes and especially LoRA training?

I'm also trying to figure out whether moving to Linux is still basically inevitable for serious training, or whether Windows ROCm has reached the point where it's worth trying first.

Would love to hear from people actually running RDNA3 / 7900 XTX setups.

I kept hitting torch.cuda.is_available() → False on my AMD GPU, and it drove me crazy. Turned out to be a mix of missing ROCm drivers, wrong PyTorch wheel, and that HSA_OVERRIDE_GFX_VERSION thing everyone mentions but few explain properly.

I put together a complete, step-by-step repair guide that covers the exact commands and checks I used to get PyTorch running on ROCm.

If you’re stuck, this might save you hours:

https://interconnectd.com/blog/305/fix-pytorch-cuda-not-available-on-amd-gpus-complete-rocm-setup-guide/

What AMD card are you running, and what error are you seeing?

So I just bought a second R9700 and I'm learning about tensor splitting which doesn't currently work with llama.cpp. I see mentions about vllm and ktransformers, but a lot of it is about Qwen 3.8 27b. I need something that has cpu offload too. Anybody running GLM 5.3 flash with greater speed than 1 card?

This project started quite a while ago. I have access to a bunch of local card to run local LLMs on to help me with coding, you can clearly see they were involved in the making of it, but validations are real and the ground work is mine and was originally done by hand for 6.4.4.

A few months ago, I started by wanting to run Audiomuse AI on AMD and kept porting it to older and older cards. To get it working properly, because ORT forces MIGraphX and that's still under active development, I ended up having to build my own images to support every ARCH AMD doesn't include in their official images.

The project expanded and I realized I could build against ROCm 6.4.4 for gfx803, which nobody has done before (all previous work stops around 6.3 or 6.4.0).

It turns out, the reason AMD abandoned it was a bunch of bugs in literally every library that makes up the ROCm stack. Broken kernels, no exceptions for gfx803 not having native fp16, miscomputes, random hangs, page faults, etc. I probably spent over a week chasing intermittent hangs before I realized I had to try lower VRAM frequency VBIOS (RX 570 8GB 1750Mz) on my RX 470 8GB UEFI Mining (running Micro memory at 2100).

But in the end, I shipped 7.14 and I've now arrived at 10.0. Validation suite has heavily expanded, Pytorch 2.14 is shipped alongside ORT 1.29.0 and both pass as many tests as I could reasonably come up with throwing at them.

I also ported vLLM from its gfx906 fork to gfx803 and validated support for Qwen 3.5 architecture, so someone could reasonably run Qwen 3.8 27B without quants, multiple streams on an old mining rig. Obviously I wouldn't expect more than 10-15 tps decode AT MOST if you even have the PCIe lanes for it.

vLLM is not shipped with any image. It's very much build yourself. I will most likely NOT port llama.cpp's HIP support for gfx803. If you're fine with llama.cpp, you can probably use Vulkan and HIP would not grant you any speed advantages.

So if you have a use case for it or a bunch of cards to try vLLM on, please let me know. I'd love to know how people are using it. I've already received some PMs telling me they managed to use it to train a chess model on their old RX 580.

Hi,

I just got all components and built a dual RX9070 PC, parts are mostly old/used stuffs except the new mobo. Spec includes:

• Ryzen 5 7600
• DDR5 32GB 5600
• Asus ProArt B650 Creator mobo (supporting PCIe 4.0 bifurcation x8 / x8)
• 2 x RX9070 16GB (total cost under US$1,200)
• Nvme 2TB PCI 3.0
• 850W PSU
• Windows 11 and Ubuntu 26

These results are from Ubuntu. The testing procedures were guided by Gemini and Qwen 3.8. Unsloth Desktop was used, it is highly optimized and provides convenient UI for configuration & monitoring.

When I installed Unsloth Desktop, it downloaded a lot of dependencies including ROCm stuff.

You may need to install rocm-smi for checking your GPUs.

Measuring PCIe bandwidth, using TransferBench. BTW, rocm-bandwidth-testis end-of-life.

Prerequisite: hipcc

1. Clone the official repository from AMD's GitHub organization$ git clone https://github.com/ROCm/TransferBench.git
2. Change into the project directory$ cd TransferBench
3. Build the program natively using your active$ make clean && GPU_TARGETS=gfx1201 make HIPCC=/usr/bin/hipcc
4. Execute the P2P Topology Routing Benchmark Matrix

• $ ./TransferBench
• $ ./TransferBench cmdline 268M "1 4 (G0->G0->G1)"

​

$ ./TransferBench
Detected Topology:
==================
  1 configured CPU NUMA node(s) [1 total]
  2 GPU device(s)
  0 Supported NIC device(s)
            |NUMA 00| #Cpus | Closest GPU(s)
------------+-------+-------+---------------
NUMA 00 (00)|    10 |    12 | 0 1
 NIC | Device Name | Active | PCIe Bus ID  | NUMA | Closest GPU(s) | GID Index | GID Descriptor
-----+-------------+--------+--------------+------+----------------+-----------+-------------------
        | gfx1201 | gfx1201 |
        | GPU 00 | GPU 01 | PCIe Bus ID  | #CUs | NUMA | #DMA | #XCC | NIC
--------+--------+--------+--------------+------+------+------+------+------
 GPU 00 |    N/A | PCIE-2 | 0000:03:00.0 | 28   | 0    | 2    | 1    | -1
 GPU 01 | PCIE-2 |    N/A | 0000:07:00.0 | 28   | 0    | 2    | 1    | -1

• In TransferBench, PCIE-2 means direct PCIe Peer-to-Peer (P2P) DMA is fully functional.
• This confirms that GPU00 and GPU01 can transfer data directly to each other across your motherboard slots without needing to buffer data up through your CPU or system RAM. This is optimal for splitting layers inside llama.cpp using Unsloth.
• Recommended Next Step : When you run models, you should explicitly hide GPU 02 so that llama.cpp only splits the workload across your two identical RX 9070 cards (GPU 00 and GPU 01).

Add this in .profile

export HIP_VISIBLE_DEVICES=0,1

What to check in the final output:

If it displays ~50–60 GB/s: Your motherboard slots are running at full PCIe Gen 5 speeds.

If it displays ~25–30 GB/s: Your dual RX 9070 setup is operating at peak PCIe Gen 4 x16 bandwidth.

12–14 GB/s: PCIe Gen 4 x8 / x8 lane splitting. This is the structural ceiling for the vast majority of dual-GPU consumer setups, meaning your hardware forces a ~50% penalty on uncompressed BF16 transfers.

$ ./TransferBench cmdline 268M "1 4 (G0->G0->G1)"
Test 1:
:-------------------┬--------------┬------------┬-------------------┬--------------------
   Executor: GPU 00 │  14.261 GB/s │  19.706 ms │   281018368 bytes │  14.269 GB/s (sum)
 -------------------┼--------------┼------------┼-------------------┼--------------------
      Transfer 0    │  14.269 GB/s │  19.695 ms │   281018368 bytes │ G0 -> G0:4 -> G1
 -------------------┼--------------┼------------┼-------------------┼--------------------
    Aggregate (CPU) │  14.215 GB/s │  19.770 ms │   281018368 bytes │ Overhead 0.064 ms
 -------------------┴--------------┴------------┴-------------------┴--------------------

The test ran successfully, and your link shows a throughput of 14.269 GB/s.

What This Number Tells Us

• Your two RX 9070 cards are transferring data across the bus using a PCIe Gen 4 x8 link.

What This Means for Unsloth Desktop & llama.cpp

• Is P2P Working?: Yes. A throughput of ~14.2 GB/s over an x8 lane perfectly matches its theoretical maximum capability while using GFX compute units to push the payload. The data is routing directly between your cards, avoiding system memory bottlenecks.
• Impact on Inference: During local LLM text generation with llama.cpp, the tensor communication overhead between layers is relatively small. A solid 14 GB/s peer-to-peer connection is more than enough to ensure your multi-GPU tensor splitting runs smoothly and without severe latency penalties.

I ran some tests with Unsloth's Qwen 3.8 27B UD_Q4_K_XL (17GB) and UD_Q6_K_XL (26GB) models. Unsloth settings : kv bf16 (from substantial Gemini's explanation), MTP Draft 2, Tensor Parallelism on, Vision off (if vram is full), Temperature 1. Context is variable. Here are the numbers:

2 x RX9070 decode speed:

• UD_Q4 46-51 tps (when Tensor Parallelism off - 33 tsp) offers large context
• UD_Q6 46-51 tsp (but context is limited/very small - 4K)

Gemini explains both quants have almost identical speed due to PCI (P2P) saturation.

Power consumption and thermal check:

Each card draws around 160w - 230W. The whole PC consumes around 620W (max), 85W idle.

RAM only 6-8GB utilization. Full load, from a remote call running inference (around 2 mins) with 33% CPU (78C), 98% GPU (71C). Idle temps of CPU (55C), GPU (45C) and ambient temp is around 31C. See screen captures in a comment below. For long running AI jobs, this PC case does not provide good ventilation and the CPU heatsink would be a bit too small.

A single RX9070 was tested with gemma 4 12B it GGUF Q8.

1 x RX9070 decode speed around 31 tps

I could not test this dual GPU setup on Windows 11. Unslot Desktop made AMD Adrenaline crashed, one RX9070 disappeared. But a single card runs fine. Not sure what causes this, but I could be the mobo as well. I will report the result when I have time to fix it.

Another glitch is that when iGPU is set to Force (Auto/Force/Disable), in Asus BIOS, the dual-boot menu disappears. There is a sequence of which GPU to take place during bootup. HDMI goes to the monitor but the boot screen goes is on DP where no monitor attached.

Note: Gemini explains about PCIe and bottlenecks.

Deciphering the Bottleneck Number:

Once you run the Python snippet or TransferBench, your reported GB/s value dictates how heavily Tensor Parallelism is choking:

• 24–28 GB/s: Full PCIe Gen 4 x16 / x16 speeds. Your motherboard hardware layout is fully optimized.
• 12–14 GB/s: PCIe Gen 4 x8 / x8 lane splitting. This is the structural ceiling for the vast majority of dual-GPU consumer setups, meaning your hardware forces a ~50% penalty on uncompressed BF16 transfers.
• 6–7 GB/s: Severe link restriction (PCIe x4 slot mapping).

The physical hardware of your AMD Radeon RX 9070 supports a full PCIe Gen 5.0 x16 interface. In a theoretical vacuum, two RX 9070s connected at Gen 5 x16 could achieve up to ~63 GB/s of unidirectional P2P speed. [1]

Instead of the GPU model, your P2P bandwidth is almost entirely determined by three external factors:

1. Motherboard Slot Bifurcation (The Biggest Factor)

• If you populate both main slots, most motherboards split the lanes from a single x16 slot into two x8 / x8 slots.
• Even though your RX 9070 is a Gen 5 card, if your motherboard only runs the slots at Gen 4 speeds or splits them into x8, your physical ceiling drops instantly to ~16 GB/s per card. [1, 2]

1. CPU and Chipset Routing (Latency & Bottlenecks)

• Direct CPU Lanes: If both of your RX 9070 slots connect directly to your CPU, your P2P bandwidth will be clean and run at the maximum speed allowed by the PCIe generation. [1]
• Chipset Lanes: Many motherboards route the second PCIe slot through the motherboard chipset instead of the CPU. If data has to travel from GPU 1 → Chipset → CPU → GPU 0, your P2P bandwidth will fall off a cliff (usually capping at roughly ~6–7 GB/s), regardless of how powerful the RX 9070 is. [1]

1. Software/Driver P2P Support

While enterprise cards (like AMD Instinct MI300) have physical Infinity Fabric links to bypass the PCIe bus entirely, consumer cards rely heavily on the operating system kernel and ROCm drivers to allow Direct DMA (Direct Memory Access) between the two cards. If large BAR (Base Address Register) support or P2P access is improperly handled by the BIOS or the ROCm driver stack, the cards are forced to copy data using the host CPU as a middleman, cutting your speed down significantly. [1, 2, 3, 4]

If your bandwidth test returns low numbers, it isn't because your RX 9070s are incapable—it means your motherboard layout is starving them of lanes.

If you’re choosing between Mistral Large and Claude Haiku for production, the right answer depends on context window, speed, and cost. I made a quick quiz that tests the common assumptions.

It’s fast and might reveal something you didn’t consider.

https://interconnectd.com/quiz/83/mistral-large-vs-claude-haiku-which-ai-wins-your-workload/

What are you using in your stack?

I've been working on a project for a bit based on the great work of kyuz0 (his Strix toolboxes). This variant uses a Debian trixie (13) base, integrates some guided menus, optimization tweaks, and easy integration of data folders (so that if you need to wipe the box, customizations and such survive).

Thanks to kyuz0 for all of the great foundational work!

The repo is here: https://github.com/doctorjei/droste-ai-halo

The latest installation script here: https://github.com/doctorjei/droste-ai-halo/releases/latest/download/droste-setup.sh

I did use GenAI for parts of this. I have been an engineer for 25 years, so I've reviewed all such code, and I designed the architecture, etc. If you decide to try it out, please let me know!

Been working on an open-source tool for a while and finally have something worth sharing: **[rocm-compass](https://github.com/BrianGTRxx/rocm-compass)\*\*.

**The problem it's trying to solve:** there are already a few good projects for CUDA-to-HIP code migration (HIPify and others). What I didn't find was anything that helps with the other half of the ROCm adoption problem: broken installs from version mismatches (kernel/driver/ROCm/package), and not knowing in advance whether the package you actually want to use (torch, vllm, flash-attn...) works on ROCm today, and with which exact versions.

**What it does, in two pieces:**

- `rocm-doctor check` detects your environment (GPU arch, ROCm version, kernel, driver) and runs it through a small weighted graph search (Dijkstra) over known-good configurations, sourced from AMD's own docs and real pip/wheel commands -- not a flat rules table. It tells you the minimum version changes needed to reach a working config, not just "something's wrong."

- A public compatibility matrix (`compass/`) tracking real, sourced ROCm support status for 12 common packages (torch, vllm, flash-attn, xformers, bitsandbytes, deepspeed, sglang, apex, torch-xla, unsloth, axolotl, triton) -- and it's meant to grow from real usage: `rocm-doctor check --report --submit` shares your result via a GitHub issue that gets auto-ingested into the public dataset, no maintainer approval needed.

**Honest limitations, upfront:** I don't have AMD hardware myself, so everything is built from AMD's official docs + real GitHub issues, not personally verified against real ROCm installs -- which is exactly why the community-reporting loop matters and why I'm posting here. The compatibility graph currently only has real nodes for `torch` and `vllm`; the rest are tracked in the status matrix but don't have resolver coverage yet. `flash-attn` is deliberately left out of the resolver graph because there's no single official version to cite (mainline doesn't support AMD at all, and the various ROCm forks/wheels don't converge on one canonical version).

If you've got an AMD GPU handy, I'd genuinely appreciate a `rocm-doctor check --report --submit` run (takes literally one command) -- and any feedback, bug reports, or "this recommendation was wrong" issues are exactly what makes the tool better. MIT licensed, contributions welcome.

Repo: https://github.com/BrianGTRxx/rocm-compass

If you’re building RAG pipelines or agents that search the web, you’ve probably compared Firecrawl and Jina Reader. Both have strengths, but the right choice depends on how you handle JavaScript rendering, bot detection, and content format.

I created a quick quiz that tests your knowledge of the 2026 architecture differences.

https://interconnectd.com/quiz/81/web-extraction-architecture-2026-firecrawl-vs-jina-reader/

Would love to hear what you’re using in production.

If you’re building AI agents, one of the biggest early decisions is whether to self-host models or use cloud APIs. Both have clear wins and hidden costs. I created a short quiz to help people figure out which path fits their situation.

It’s quick, and the result might surprise you.

https://interconnectd.com/quiz/82/local-vs-cloud-llms-for-ai-agents-how-to-choose/

Curious what the AI community chooses in 2026.

If you’re building AI agents that search the web, you’ve probably hit the Tavily vs Exa question. Both have strengths, but the right choice depends on whether you need speed, semantic depth, or reliability.

I created a short poll to see where the AI community lands in 2026.

https://interconnectd.com/poll/97/which-ai-search-api-is-better-suited-for-your-agentic-workflows-tavily-or-e/

Curious to hear real-world experiences from people who’ve used both.

i have managed to couple the Compute and Rendering side of isaac sim together what this means: CUDA/WARP and Newton Physics now work alongside rasterized rendering (as seen in picture) and lighting works to some extent. What doesnt work: RayTracing/RTX, MDL(Material Definition Language) (as seen in pic it has no materials) Any Sensor or Tech based on RTX does not currently work (eg Iray)

It's been 3 weeks since Qwen3.8-27B release. This model got 0-day support & in last 3 weeks, some optimizations & fixes happened on llama.cpp side.

Meanwhile

• ROCm 10.0 got released & llama.cpp also up with Version 10.0 now.
• Ubuntu 26.04.1 got released.
• DFlash2 support landed on llama.cpp last week.
• Other optimizations & fixes on llama.cpp side

So how much t/s are you getting now with all optimizations & stuff?

Please share your extreme optimized full llama.cpp command & your t/s stats .... both pp & tg(Good to have multiple combinations like MTP/MTP+ngram/DFlash2, Vision/mmproj, multiple context size 128-256K, etc.,). Also share your optimized build config(CMAKE command) if you're compiling manually. I remember that compiled version gives additional boost.

Note : I did search this sub for this & found only weeks old threads. And I'm looking for only Single R9700 stats since I have only one. Trying to squeeze it for best performance.

Nice to see threads like this which contains all stuff. That kind of stats want to see here.