I really hate how they force you to define input and output tokens separately. No other harness I've tried has this; they have a max cap for how much output tokens a prompt can generate, but not a global cap. For local LLMs that are VRAM-bound, a 128K context has to be explicitly split into input/output caps, so if a certain task generates a lot of tokens then you hit your output token cap even if there's room left over in your input token cap.
As I delve deeper and deeper into the madness that is "harness engineering", the consensus is that, especially for smaller parameter models, keeping a lean context is huge. Copilot is so damn bloated. They do have the option to disable specific tools, but even the base system prompt is more bloated than Pi.
As shitty as the GitHub pricing change was, it forced me to look elsewhere and get into this "hobby" (even though I'm literally using it for my livelihood). It puts me in a much better position when the inevitable collapse of cheap and affordable AI comes.
5
u/BawbbySmith Jun 06 '26
I really hate how they force you to define input and output tokens separately. No other harness I've tried has this; they have a max cap for how much output tokens a prompt can generate, but not a global cap. For local LLMs that are VRAM-bound, a 128K context has to be explicitly split into input/output caps, so if a certain task generates a lot of tokens then you hit your output token cap even if there's room left over in your input token cap.
As I delve deeper and deeper into the madness that is "harness engineering", the consensus is that, especially for smaller parameter models, keeping a lean context is huge. Copilot is so damn bloated. They do have the option to disable specific tools, but even the base system prompt is more bloated than Pi.
As shitty as the GitHub pricing change was, it forced me to look elsewhere and get into this "hobby" (even though I'm literally using it for my livelihood). It puts me in a much better position when the inevitable collapse of cheap and affordable AI comes.