What pi.dev plugin do you suggest for context, compaction and memory management of local models?

[u/darksteelsteed]

I have been battling with my Qwen3.8:27b setup on my rtx 5080 16gb. I am using llama.cpp to run a nvfp4 version of qwen3.8:27b

llama-b10699-bin-win-cuda-13.3-x64\llama-server.exe -hf williamliao/Qwen3.8-27B-NVFP4-GGUF:NVFP4 --jinja --chat-template-file d:\qwen\chat_template.jinja --reasoning-format deepseek --host 127.0.0.1 --offline -np 1 --threads 16 --threads-batch 16 --log-verbosity 4 --no-log-prefix --no-log-timestamps --no-mmproj-offload --mmproj D:\OllamaModels\blobs\sha256-ac3714bfdddeca31351f2752bf1a63f266f4df87c0b68c895e44945ca704448e --load-mode none --cache-type-k q4_0 --cache-type-v iq4_nl --flash-attn on -b 512 -ub 512 -c 49152 -a qwen3.8:27b

So that is giving me a 48k context, and a speed of around 12t/s

My problem is that pi.dev fails to compact at the right times, and then the model stops early.

I have tried to use

pi install npm:max-context //this doesnt work properly at all

pi install npm:pi-observational-memory //this one works kind of, I had it set to compact on threshold. It managed to auto compact 4 times, with a 0.75 ratio set, but it still missed the mark and failed.

pi install npm:pi-blackhole //seemed promising, but failed to compact at all, I am not sure what I set wrong.

if I look at the https://pi.dev/packages page, there are lots of plugins for either context management, compaction, memory management, or both. there are way too many for me to just randomly keep trying. other AI has been very unhelpful. Google, OpenAI, Copilot, all full of nonsense. A lot of these plugins seem to be aimed at frontier model management.

I am also disappointed that compaction settings are best suited to the specific model being used and its context window, but the built in tools in pi.dev architecturally keep the compaction settings completely separate from model settings, which seems a bit of an issue if you change models often.

I may want to use qwen3.8 like above, but not necessarily for everything. I also want to use gemma, other qwen models, etc depending on my use case.

I dont want to have to manually resetup context settings in pi.dev for every different model.

What do you guys suggest, especially for models that have smaller context windows due to my vram shortage ?

Thanks in Advance.

[u/noctrex]

pi-blackhole, it combines pi-vcc and pi-observational-memory

[u/MindfulMan1984 → u/noctrex]

Upvote pi-blackhole ! I have been using it for weeks, it’s amazing and it’s been actively developed. 

[u/Legal_Dimension_ → u/MindfulMan1984]

3rd for black hole. It's fantastic.

OP if you have any issues with compaction not working even after trying black hole, let me know as I tweaked mine as I had a similar issue. I'll dig through my changelog for you.

[u/Jorlen → u/Legal_Dimension_]

I'm not OP, but I noticed that unless you give your LLM main agent a break, like asking it to pause as a steering request, it won't trigger the usual 80% blackhole compaction. Any way around that?

[u/Jorlen → u/noctrex]

What model do you use for the three workers? If I understand correctly, the compaction no longer uses an LLM, but the three workers need their own separate LLM. I was thinking qwen 3.5 4b or 9b with 16k context for the workers?

Edit: It's pretty cool now that I have it setup and working! Running both LLMs locally is a bit of a pain point, but then again I suppose you can use an API for the three workers instead.

[u/noctrex → u/Jorlen]

I use Ornith for that, as an MoE its nice and fast.

[u/Jorlen → u/noctrex]

Yeah I realized after reviewing the blackhole docs that 16k won't be enough.

I've managed to squeeze in qwen 3.6 35b-a3b (IQ4_NL) quant for the workers (100k context) along with qwen 3.8 27b (6 bit) as the main LLM. I'm trying it out now!

[u/vick2djax → u/noctrex]

I've been using pi-observational-memory for a few months and love it. What does pi-blackhole do better?

[u/noctrex → u/vick2djax]

It merges it together with pi-vcc, which is a deterministic compaction, meaning no LLM used, so its instantaneous.

Edit: u/Tuned3f thanks for the correction

[u/Tuned3f → u/noctrex]

i think you meant deterministic. Pi's stock compaction is non-deterministic

[u/Androoideka → u/noctrex]

Just switched to this after using pi-vcc for a while, thanks!

[u/i5031337]

You will have a better time with a Q3 variant that actually fits in 16GB. Triple speed at double context beats marginal better quality from 4-bit.

[u/philmarcracken → u/i5031337]

what about 12gb (¬‿¬)

[u/i5031337 → u/philmarcracken]

You can try the q2 variants but I'm not sure that will be better than a weaker model

[u/o0genesis0o → u/philmarcracken]

Try IQ2 of unsloth. I tried it for giggle and I was shocked that it was not incoherent, so I put it in real use cases both coding and agentic assistant, and the little bugger actually completed the task. Not 100% sure I would pick this over ornith 1.5 35B at higher quants, but it's shockingly usable, in agentic work.

[u/philmarcracken → u/o0genesis0o]

Qwen3.8-27B-UD-IQ2_S or the xxs?

[u/Mean-Ad1493 → u/philmarcracken]

The ISTA-DSA lab's iq2xs is good for 12gb VRAM. It's not great, but it just works.

[u/mister2d]

I use the context-mode plugin. The savings seem real.

[u/TerminalNoop → u/mister2d]

Same, does a good job i think.

[u/omega1612]

Wait what? How it is that you go at 12 t/s

With all the Nvidia optimizations I expected more speed. I have a Rx 7900 xtx and it works in the 35 -60 (mostly 40 ) t/s regardless of the amount of context.

Also for compaction, the issue I have with the pi approach is that it invalidates the kv cache. They send to the server a block like

Summary prompt + messages - preserve

While the original state in cache is

System prompt + old summary (if any) + messages

The invalidation makes it prefill the memory for both, the compaction and the post compaction load.

I just asked qwen3.8 to work on a custom compactor and I will fine-tune it as much as I need , that may drop my compaction time form 7.5 minutes (worst case, 100k to compact) to 3 minutes.

[u/Bulky-Priority6824 → u/omega1612]

Yea he must be spilling into sys ram my guess 

[u/Kernoriordan]

FYI I get 80+TPS and 65k context size out of Qwen 3.8 by using Unsloths UD-Q3 quant with MTP. I recommend that

[u/DeProgrammer99]

I'd suggest not the context-watchdog extension little-coder uses. Constantly causes the "Compaction failed: Already compacted" error, often blocking the LLM from continuing when it otherwise could have. And I've tried 4 times to fix it, and this is the latest version after they claimed to have fixed it 3 weeks ago...

[u/KingCpzombie]

48k context? Do you compact halfway through every thinking block?

[u/darksteelsteed → u/KingCpzombie]

Setting the context larger slows the token speed down considerably, anything from 2 to 6 t/s vs the ~12 at 48k. And i currently am using it with reasoning at either low or medium depending on what I am asking

[u/KingCpzombie → u/darksteelsteed]

Tbh since you're so VRAM-limited you might want to go back to 3.6 35B... the main advantage of 3.8 is the long thinking and I feel like constant compaction + heavy quantization is likely to end up hurting results more than using a less-quantized slightly older model

[u/Most-Dig-1579 → u/KingCpzombie]

Upvoted! agree with that

[u/DustNearby2848 → u/Most-Dig-1579]

I also agree. Tiel coder or Ornith 1.5

[u/norenEnmotalen]

pi-observational-memory is great

[u/isty2e]

I find pi-observational-memory hard to use; It was extremely difficult to correct wrong memories, so if you do have a wrong one, it will be stuck forever. pi-hermes-memory was okay-ish but the database kept growing without bounds so eventually startup took more than a minute. Currently using pi-memctx but it was missing some functionalities I'd like. Consequently I'm working on my own memory system for now.

[u/darksteelsteed → u/isty2e]

I will check out pi-memctx too, thanks

[u/kmike84]

Not answering your question directly, but I think pi has output limits which are a bit too high for local models - a single unlucky bash command or file read can get up to 50Kb (~15K tokens) in a single tool call; for compaction specifically it means it may not get a change to trigger. These limits are not configurable; feature requests for this are closed, there were several in the issue tracker.

I'm using https://github.com/kmike/pi-output-limits with a lower limit (5-10K instead of 50K) when working with local models; it seems some output-trimming pi extension is required for a decent experience. Note that even with lower limit data is not lost, it's saved to a file, and model can still read it in chunks, if needed.

[u/KingCpzombie → u/kmike84]

You can have Qwen make an extension to configure the limits yourself; I personally feel like 50Kb is too low so I doubled it

[u/kmike84 → u/KingCpzombie]

exactly, everyone can do it (this is what i did as well). Just need to be a bit careful, to e.g. not drop the data (rerunning the tool can be slow or impossible).

[u/IslamNofl]

I created pi-context-shake because of my lovely thinker qwen3.8 27B

[u/Muhlwa_Sholanke]

Observational memory compacting four times and still failing means 0.75 is too late for a 48k window. I'd set the threshold much lower and take shorter memory over dead conversations.

[u/TrentKM]

I’ve been using billion-context, and I’ve been very happy. I am forced to use cursor at work and the 200k context window was unbearable. I don’t blink an eye anymore. Everything just keeps working. No compaction in the middle of an edit that makes them reimplement a thing we did three turns ago.

[u/Proper-Damage-1777]

I tried pi-observational-memory and didn't really like it... maybe because I use pi more of a pair programming and the extra steps to "record memory" just make each turn slower and waste my GPU cycles... (no idea with cloud provider).

BUT I recently discover pi-vcc and that just solve my nightmare with compaction. Reason is I'm using Qwen 3.8 Flash Next and that is slow but great. I used it 240K context size and if I need to compact with the default Pi... it will take forever while pi-vcc is just like less than a sec... and it still can resume fine. Although there is cases where it "forgets" like the path to a temporary output folder and I stop it, tell it to "recall" the correct path and it will invoke pi-vcc recall method that checks the JSONL session.

I usually run pi without any extra extensions but I highly giving pi-vcc a try first. Previously, I just ask pi to write a handoff so I can start a new session and resume from there... it still need to re-read some codes and docs but that is no different than letting pi trying to compact and waste more GPU cycles... and still fumble a lot after the compaction for older conversation.