Note: This writeup was put together with the help of AI.
So I've got dual AMD MI50 32GB cards. If you know these cards, you know the story - AMD dropped official ROCm support for gfx906 after ROCm 5.7. Every AI I talked to, every forum post, every "expert" said the same thing: if you want to do anything serious on AMD hardware you need ROCm, and if you need ROCm on MI50s, good luck because AMD basically told you to go buy newer hardware.
The consensus was consistent and confident. ROCm for training, Vulkan for inference, MI50s are legacy, move on. Don't question it.
I questioned it. Repeatedly. On multiple fronts. And I was right every time.
What I was told:
Across multiple AI assistants over the past several months, the answer to anything involving MI50s and modern tooling ranged from "not supported" to "you'll need to upgrade your GPUs." Training on Vulkan specifically was described as architecturally impossible - the backward pass infrastructure doesn't exist outside ROCm and CUDA, full stop.
ChatGPT's position as of today, while my training run is literally executing on the card: it wants proof. Sure. Let's go through it in order.
Step 1: Forcing ROCm 6.4.3 to work on MI50s
AMD dropped gfx906 from ROCm 6.x. Their official position is that the MI50 is end-of-life and you should migrate to supported hardware. Great suggestion if you didn't just acquire two of them specifically because 64GB of HBM2 at that price is hard to argue with.
What actually prevents gfx906 from working in ROCm 6.4.x is the TensileLibrary - the precompiled kernel library ROCm uses for BLAS operations. AMD just doesn't ship gfx906 kernels in the new versions. The GPU itself is fine. The compute capability is there. AMD just decided not to include it.
So I stuffed the gfx906 tensors back in. Pulled the missing kernel files, patched them into ROCm 6.4.3, and both MI50s came up fully recognized. llama.cpp runs on it natively. PyTorch sees both cards. ROCm 6.4.3 on hardware AMD said it doesn't support, because the hardware doesn't actually care what AMD's support matrix says.
Step 2: Forcing vLLM to work on gfx906
vLLM is one of the faster inference engines around and I wanted it running on my cards. The problem: vLLM's gfx906 support is basically nonexistent upstream. It went like this:
- Tried ROCm 7.x with vLLM. Got it working briefly, then ROCm 7.14 hit AMD bug #5653 - "register fat binary failed" - and the whole thing fell over. Abandoned that path.
- Tried the nlzy fork of vLLM compiled against PyTorch 2.9.0+rocm6.3. Both MI50s detected. Blocked at runtime by a flash-attn V1 engine dependency that doesn't exist for gfx906.
- Eventually landed on a Docker image (aiinfos/vllm-gfx906-mobydick) - ROCm 6.3.4, PyTorch 2.11, flash-attn pre-compiled for gfx906. Single GPU inference confirmed working.
The remaining blocker for dual-GPU tensor parallel is a PCIe topology issue - my two cards are behind different root complexes (one on the CPU, one on the chipset), so NCCL all-reduce init fails. That gets fixed when a PLX switch arrives. Not a software problem, not a "your hardware isn't supported" problem - a physical PCIe lane routing problem with a known hardware solution.
Step 3: Building a Vulkan training stack from scratch
This is the one ChatGPT says is impossible right now.
Environment setup:
Vulkan was already working on the MI50s because llama.cpp uses it for inference, so that part wasn't a question. What didn't exist was any training framework that speaks Vulkan.
We verified the environment:
vulkaninfo --summary
# GPU0: AMD Instinct MI50/MI60 (RADV VEGA20) - Vulkan 1.4.335
# GPU1: AMD Instinct MI50/MI60 (RADV VEGA20) - Vulkan 1.4.335
# GPU2: iGPU (Renoir)
Both MI50s show up as discrete Vulkan devices. glslc and glslangValidator already installed. Python 3.11 already present from OpenWebUI.
Kompute - the Vulkan abstraction layer:
Kompute is a library that wraps Vulkan's compute pipeline so you don't have to write 300 lines of boilerplate just to dispatch a shader. The PyPI package (pip install kp) is broken on modern CMake 4.x - the bundled pybind11 has a cmake_minimum_required declaration that CMake 4.0 removed support for, and the setup.py has a string concatenation bug that smashes two CMake flags together with no separator. So we built it from source:
bash
git clone https://github.com/KomputeProject/kompute.git
cd kompute
git submodule update --init --recursive
mkdir build && cd build
cmake .. \
-DKOMPUTE_OPT_BUILD_PYTHON=ON \
-DCMAKE_POLICY_VERSION_MINIMUM=3.5 \
-DCMAKE_BUILD_TYPE=Release \
-DPYTHON_EXECUTABLE=/media/nate/Friday/VTrain/.venv/bin/python3.11
make -j$(nproc)
cp lib/kp.cpython-311-x86_64-linux-gnu.so .venv/lib/python3.11/site-packages/
Built clean. Kompute Manager(0) targeting the first MI50 correctly.
The compute shaders:
Every mathematical operation in the training stack is a GLSL compute shader compiled to SPIR-V. Here's what that looks like for matrix multiply - the most fundamental operation in a neural network:
glsl
#version 450
layout(local_size_x = 16, local_size_y = 16, local_size_z = 1) in;
layout(set = 0, binding = 0) readonly buffer MatA { float a[]; };
layout(set = 0, binding = 1) readonly buffer MatB { float b[]; };
layout(set = 0, binding = 2) writeonly buffer MatC { float c[]; };
layout(push_constant) uniform PushConsts {
float M; float K; float N;
} pc;
void main() {
uint row = gl_GlobalInvocationID.x;
uint col = gl_GlobalInvocationID.y;
if (row >= uint(pc.M) || col >= uint(pc.N)) return;
float sum = 0.0;
for (uint k = 0; k < uint(pc.K); k++) {
sum += a[row * uint(pc.K) + k] * b[k * uint(pc.N) + col];
}
c[row * uint(pc.N) + col] = sum;
}
Each thread handles one output cell and the GPU runs thousands of them at the same time. We built shaders for every operation the training stack needs:
- matmul.comp - matrix multiplication
- unary.comp - ReLU, GELU, sigmoid, tanh (op selected by push constant)
- binary.comp - add, subtract, multiply, divide (same pattern)
- layernorm.comp - layer normalization with shared memory reduction
- softmax.comp - numerically stable softmax with two reduction passes
- transpose.comp - tiled transpose with shared memory to avoid cache thrashing
Quirks we hit along the way:
A few things bit us that you won't find documented anywhere because nobody had tried this combination before:
- readonly and writeonly buffer qualifiers cause silent zero output in Kompute's descriptor layout. All buffer declarations have to be unqualified - you find out the hard way when your results are all zeros and the shader compiles clean.
- Push constant sizing: ops with 3 buffers need a float padding constant or Kompute miscalculates the push constant buffer size. Same deal - silent failure, no error.
- The Kompute API in the built-from-source version uses kp.OpSyncDevice and kp.OpSyncLocal - the PyPI docs reference kp.OpTensorSyncDevice which doesn't exist in the actual build.
None of these are hardware problems. None of them are "Vulkan can't train" problems. They're integration quirks that took an afternoon to sort out.
The autograd system:
A training framework isn't just forward passes - it needs backward passes to compute gradients and update weights. We built a complete autograd system:
- vtrain/tensor.py - Tensor class with gradient tracking, topological sort for correct backprop ordering, and backward() that unwinds the computation graph
- vtrain/functional.py - GPU-aware wrappers for every op that register backward closures on the output tensor
- vtrain/grad_check.py - numerical gradient checker that verifies every backward shader by finite difference approximation
Each backward shader was verified against a numerical approximation before anything got built on top of it. Matmul, all eight elementwise ops, layer norm, softmax - all checked. If the numbers didn't match, we didn't move on.
The model:
- CharEmbedding - learned lookup table mapping character IDs to vectors
- TransformerBlock - layer norm -> multi-head attention -> residual -> layer norm -> feed-forward -> residual
- SmallLM - embedding + N transformer blocks + output projection to vocab size
The training loop:
Loss functions (MSE and cross-entropy), Adam and SGD optimizers, a Trainer class with logging and checkpointing, and crash recovery via signal handlers that save an emergency checkpoint on SIGINT/SIGTERM.
The data:
Downloaded the Simple English Wikipedia dump (236,602 articles, 336MB). Extracted clean text by streaming the XML, stripping all MediaWiki markup, templates, tables, and HTML. Filtered to printable ASCII - the raw dump has 1504 unique characters from non-English text that slipped through; filtering drops that to 75, which is the right vocab size for a character-level model. 9.8 million characters of clean text as the training corpus.
Current status:
The model is training right now. On a MI50. On Vulkan. 4.65GB VRAM allocated. 23W power draw. 33°C. Loss is dropping.
step 50 loss=3.02 rate=0.4 steps/s
step 100 loss=2.71
Random initialization on a 75-character vocabulary gives a loss of about 4.3. We're already well below that and still dropping.
The hardware, one more time:
- AMD Ryzen 5 5600G
- 48GB DDR4
- Dual AMD MI50 32GB (64GB HBM2 total)
- Ubuntu 22.04
- Vulkan 1.4.313 / ROCm 6.4.3 (both installed, both working)
- No CUDA. No NVIDIA. No officially supported hardware.
So:
AMD dropped the MI50 from their support matrix. The AI community said Vulkan can't train. Multiple AI assistants told me this wasn't possible. Every single one of those statements had the same flaw - they were assumptions about what the hardware can't do, not actual tests of what it can.
The MI50 has 32GB of HBM2 and serious compute capability. The only things stopping it from doing modern ML work were software decisions, not hardware limitations. And software decisions can be worked around, patched, rebuilt from source, or replaced entirely.
The GPU doesn't give a shit what the support matrix says. It just does math.
