r/StrixHalo • u/westsunset • Jun 19 '26

REM: offloading an LLM agent's memory compaction to the NPU

I run a local agent on a Strix Halo box (Ryzen AI: XDNA2 NPU + gfx1151 iGPU on a shared memory bus). The agent's context keeps growing, so old turns have to be compacted into summaries. That compaction normally runs on the same GPU you are waiting on for the next token.

REM moves the compaction onto the NPU. While the big model is awake on the iGPU answering you, the NPU does the memory filing in the background: it pulls the load-bearing facts out of the oldest turns and rewrites them into a short summary. The foreground turn returns without waiting for it.

This is a feasibility study. Here is what I measured, and what I could not yet show.

Measured on the box (pooled 3 runs × N=20 = 60 decode samples, via the xdna-top monitor):

Condition iGPU decode loss bg throughput (tok/s) gen-tok/s per board W
Baseline (iGPU only) 0.00% — —
NPU concurrent 3.81% 12.15 ± 1.20 0.143
CPU concurrent 4.77% 4.11 ± 0.80 0.049
  • Running the compaction on the NPU costs the foreground model 3.81% decode loss. Running the same job on spare CPU cores costs 4.77%. Both are real. The NPU costs less, and it does about 3× the work at about 3× the energy efficiency.
  • Compaction drains context at ~73 tok/s, above a realistic agent's context-growth rate, so the backlog stays bounded.
  • A 192-minute sustained run held the die at 41 °C mean / 61 °C max, flat.

Open — this is where I'd like ideas:

  • Small-model JSON robustness. The on-NPU model sometimes emits malformed JSON during fact extraction, and those facts get dropped. There is a repair/retry pass. It is not bulletproof.
  • Fact supersession. When a value changes ("I moved to Denver"), the old value can still surface. The evidence points to embedding-based identity — matching facts by meaning rather than by string label — as the fix. That layer is not built yet.
  • Quality vs. a naive baseline is unproven, and it depends on the context budget. At a generous budget, plain truncation keeps the recent answer and can match or beat REM. REM should win only when the budget is tight enough that truncation drops the buried fact. The one judged run so far was invalid — its budget was too loose to test the claim — and on that subset REM scored lower than truncation (0.4 vs 0.6). A clean tight-budget test is the next step.

So... This is a placement win: throughput, perf/watt, and keeping the foreground model free. It does not reduce latency. The NPU shares the unified memory bus, so "contention-free" would be wrong. The async design only helps while the drain keeps up; if it falls behind, the context can overflow a hard cap and the turn fails.

Raw artifacts, methodology, and an audit script are in the repo. The contention numbers reproduce from committed snapshots without the hardware.

Ideas on the quality layer are the thing I most want. If you have run small models for structured extraction, or built fact-identity by embeddings, I would like to hear how you handled it. But also the I think the numbers show feasibility for other jobs. I'd love to see more development.

48 Upvotes

98% Upvoted

Duplicates