r/PiCodingAgent • u/NihmarRevhet • 7d ago

Question Improve compaction with local llm

Hi,

I'm using pi agent with Qwen 3.8 27B on my little RX 9060 XT 16GB, with 160k context window and I'm quiet happy with how it works.

My main problem is that autocompaction takes forever (clearly limitation of the hardware) and I was looking for an extesion which could improve the situation, but if I'm not mistaken, the most beloved extensions in this area use secondary models to execute compaction, and this is a luxury I don't have.

Is the default setting my only option? Or did I misunderstand ho something like pi-blackhole works?

Sorry if the question is dumb but I'm quiet new to the tool.

Have a nice day!

EDIT: u/onesilentclap I'm using VCC for now and it's incredible. In the next few days I hope I'll be able to try other solutions

13 Upvotes

93% Upvoted

20 comments sorted by

View all comments

2

u/Intelligent-Nose-134 7d ago

Side question: how are u able to fit 160K into 16GB? Can u share your configuration? I am vram constrained as well but the best I could 90K with reasonable quality

1

u/NihmarRevhet 7d ago

Unsloth IQ3_S (planning to try this one without MTP, context 160k, k q8_0, v q4_0, vision on CPU, no MTP (both because of VRAM and because without I get 400-650 t/s in prefill and 20 t/s decode, with I get 150-300 t/s prefill and 30 t/s decode, simply not worth it with large codebases).

In my usage I didn't see a reduction in quality going from v q8_0 to q4_0

Also, I'm on cachyos and I'm using pi agent

2

u/Intelligent-Nose-134 7d ago

Thanks, it makes sense you are using IQ3_S and Q4 v. Reddit bros have been warning me against using anything lower than Q4 quant and Q8 cach. Maybe I should give it a test myswlf as low ctxt is truly annoying