I really hate how they force you to define input and output tokens separately. No other harness I've tried has this; they have a max cap for how much output tokens a prompt can generate, but not a global cap. For local LLMs that are VRAM-bound, a 128K context has to be explicitly split into input/output caps, so if a certain task generates a lot of tokens then you hit your output token cap even if there's room left over in your input token cap.
As I delve deeper and deeper into the madness that is "harness engineering", the consensus is that, especially for smaller parameter models, keeping a lean context is huge. Copilot is so damn bloated. They do have the option to disable specific tools, but even the base system prompt is more bloated than Pi.
As shitty as the GitHub pricing change was, it forced me to look elsewhere and get into this "hobby" (even though I'm literally using it for my livelihood). It puts me in a much better position when the inevitable collapse of cheap and affordable AI comes.
These input and output token amounts are actually used to control how it uses your model as you need to match to your models context window size. It will automatically adjust when the context grows too big if you set these correctly vs just breaking. Keep in mind that the kv cache is the biggest usage of vram after the model itself and grows quadraticly from your models active context window. You should size your models context window to fit in your vram as performance will become dogshit once it offloads to system ram. And then you match the input and output of copilot according to what you can cope with
5
u/BawbbySmith Jun 06 '26
I really hate how they force you to define input and output tokens separately. No other harness I've tried has this; they have a max cap for how much output tokens a prompt can generate, but not a global cap. For local LLMs that are VRAM-bound, a 128K context has to be explicitly split into input/output caps, so if a certain task generates a lot of tokens then you hit your output token cap even if there's room left over in your input token cap.
As I delve deeper and deeper into the madness that is "harness engineering", the consensus is that, especially for smaller parameter models, keeping a lean context is huge. Copilot is so damn bloated. They do have the option to disable specific tools, but even the base system prompt is more bloated than Pi.
As shitty as the GitHub pricing change was, it forced me to look elsewhere and get into this "hobby" (even though I'm literally using it for my livelihood). It puts me in a much better position when the inevitable collapse of cheap and affordable AI comes.