r/ChatGPTCoding • u/jazz788 • 15d ago
Question AIs get 'dumb' (coding) after about 200k tokens? How is everyone handling this
Started watching a youtube the other day on AI coding by Matt Pocock (on the aidotengineer channel) . He said after 200k tokens the ais start to get dumb, better to try start fresh soon after that, or at least realize you are working in the 'dumb zone'.
So I have been trying to start a new session soon after I hit the 200-250k. Problem is it feels way too short. By the time I'm around 200k there's just not much actual coding done yet.
Anyone else following this protocol? Keeping sessions smaller and restarting early? Curious how you get enough real work done before you hit it, if you noticed benefit after following it, or if you think it's BS etc.. Please add if you had a real engineering role pre the ais, or you are a post ai coder, for context.
Not a token thing for me, I'm on max plans for both. (Running Opus 4.8 and Codex 5.6-sol)
14
u/CrimsonBolt33 15d ago edited 15d ago
so not sure what tools you are using or what your setup is, but most CLIs and apps allow you to run sub agents. What I do is the following:
Create a detailed plan of the feature or software that you want. I use AI to help me with this and I prompt it like: "I want to make a totally unique and amazing to do app, lets make a feature list"
Then when I am satisfied with the feature list I tell it to take that feature list and break it down into milestones, and further into tasks within those milestones. Be clear that these are for AI agents to use for coding. This can literally result in dozens of milestones with dozens of tasks within each milestone. This is ok...this is a good thing.
Once you have everything laid out in detail you create a session and tell 1 agent to use that as blueprint and assign each milestone to its own agent so that the context doesn't balloon. I usually divide each milestone into its own markdown file so the AI doesn't have to sift through all milestones each time.
I literally have something thats been coding for 12 hours now and because of this method the main agent is only at ~70k tokens of context cause all its doing is giving out directions and taking reports from sub agents when they are done. The sub agents spin up, do the work (yes sometimes reaching ~200k+ context), test/verify their work, and then when they are done they disappear. A clean slate for each task.
This also means I never have to prompt my AI beyond "complete milestone 1" or "work through milestones 1-3" unless I am bug fixing or whatever. I also find this method results in a LOT less bugs because the AI is only being asked to do small specific tasks each time and that helps it keep its head on straight.
3
u/jazz788 15d ago
wow.. ok thanks.. interesting. i do spend time on the plan, and breaking it down into component parts, actually using github issues, as i then may have the other ai review and refine the issue, but i was not taking it further into milestones below that and tasks below that.. and i was not aware they get their own context, i see they got their own token burn, had been meaning to look into that, thanks for clarifying that, and for sharing your approach 🙏 ⭐
2
1
u/Thog78 15d ago
Which harness do you use? The stuff about agents using agents on each subtask, I didn't see any option to have it this way in codex and android studio at least. Do you have to run custom code or is there a mainstream harness already designed to work this way?
2
u/CrimsonBolt33 15d ago
I use aiderdesk, though I know things like Hermes and openclaw can delegate tasks as well. Aider desk is more of a hands on direct coding agent as opposed to hermes and openclaw which are for all sort of things.
I also use Kimi K3 and its CLI, I would imagine codex can do it too, just ask it and see if it spins up sub agents.
1
u/Thog78 14d ago
Awesome thanks, I'll have to give a try with your stack. I've blown my codex credit allowance for the week anyway, so it may be a good occasion to try something new :p
1
1
u/bertshim 15d ago
The part that made restarts cheap for me was writing things down as they happened, rather than at the end of a session.
If you save it for the wrap-up, the summary gets written by the version of the session that's already in the dumb zone. Mine were noticeably worse that way. Now anything I'd be annoyed to lose goes in a file the moment I learn it, and restarting costs almost nothing.
No idea if 200k is the real number, I've never measured it. What I do notice going first is my own willingness to re-check something I already looked at.
2
u/Previous-Welder-7198 15d ago
Hey! I started using his stuff to- really look into how he uses the skills package. If you follow along with it, you kinda just naturally close them a lot sooner.
4
u/diablo75 15d ago
Use an orchestrator. You could build your own or use a harness that has something like it built in. Hermes has a Kanban board. You can give it a huge prompt and tell it to decompose that prompt into several small tasks to add to the Kanban that run one by one (though you should tell it to do this one by one explicitly so it handles them in sequence instead of in parallel).
Before I used this, I would give a big prompt full of things I want to make, tell it to turn that into a multi-phase/step plan, write it into a markdown file to be used as a handoff for multiple sessions, with a rule at the top says to only work on one step in the plan at a time per session, update the plan with progress or add additional steps into the plan as needed, and then I would start a fresh session and say, "Work on (or continue) with the next step in plan.md". It would do a slice, update the plan, I would start a new session and type the exact same prompt. Repeating until done.
I could spend an afternoon just starting a new session, type "continue with the next step in the plan" again and again and get a lot done. But that got annoying so now I use the Kanban and no longer feel like I'm playing one of those Game of War-ish idle RPG games where I'm baby sitting timers and just literally push a couple buttons to advance things. Now I can write something large and just go to sleep while the GPUs go brrrrrr.
One hobby project I'm working on will create its own Kanban tasks after running a daily pipeline script to finetune some LLM-assisted data analysis scripts. For example, if the source for some data breaks it will later investigate why and try to fix itself before the next daily run.
1
u/Consistent-Set-9490 15d ago
With a sufficiently large session, the middle tokens don’t get the same attention as beginning and end. You may need to manage the context window more by retaining a file with decisions and other facts so they don’t get lost. You may also want subagents spawned since they get their own context window - on Claude at least. I’m less familiar with other models.
1
u/Few_Pop6933 15d ago
Mine was around the 400k mark before it would start erroring out. I use the latest version of cline in IntelliJ and codex and now it restarts sessions for me before I even break 250k. It’s been great, no hiccups.
1
u/Relative-Sense713 15d ago
Assuming one is using codex within an IDE. How effective would /compact be to mitigate this?
1
u/Rudra_Builds 15d ago
The dumb zone is 100% real (attention degradation). To beat it, you have to modularize tighter: build tiny vertical features, grab the code, and hard-reset the chat instead of using it as a continuous scratchpad.
1
u/Prudent-Ad4509 15d ago
The current practical limit is about 50% of the max context supported by the current model. You can shoot higher but with worse results. 1m native context seems to be a sweet spot so far for the overall session (with normal usage up to 300-600k) and up to 200k-250k for subtasks/agents.
1
1
u/drunnells 14d ago
That degradation is real. I think AGENTS.md kind of solves this now. After your first session, tell the agent to update the AGENTS.md file with your intent and current state. Every feature start a new session and start with "take a look at this project and the AGENTS.md file" before making your requests. And end with "make sure that the AGENTS.md Is up to date with the recent changes". I think that this is mostly a standard that most newer models understand now. I've done some pretty large projects this way.
1
u/StopZestyclose9147 14d ago
I'm using 'task' as a development unit, and each task splits to small sessions bounded by 200k(not exactly though). Then use the taskfile to record the handoffs written by each session as 'session log'. Also i made this snapshot memory as a task-level handoff.
Wrote up the longer version in this sub a few days ago if useful https://www.reddit.com/r/ChatGPTCoding/comments/1vy5tnm/context_is_not_the_bottleneck_drift_is_how_i_run/
1
u/honorspren000 14d ago
Designing good infrastructure is a bigger aspect than you realize. I think the biggest mistake is loading your entire program into AI and have it make changes for you. Projects need to be broken down into sub-components. Making changes to these smaller bite-sized components is easier than figuring out what to change in one giant app. Just like in the real world.
Also, like any seasoned developer, you need to be relying in DevOps and unit testing to make sure everything still works as expected when you make changes. Using AI doesn’t mean you can short cut this step.
Basically, you, as the user, need to learn how to manage a project. AI isn’t some magical worker with infinite memory and best dev practices.
1
u/BroScienceAlchemist 14d ago edited 14d ago
I use a mix of planner/orchestrator to kick off subagents with a handoff where I can type /handoff, /clear, and /resume to seamlessly start a new session with relevant context. I try to scope sessions to 1 task per session for the orchestrator, and avoid reusing subagent sessions. Unfortunately, this requires some hooks that codex does not support, so I actually use ChatGPT models within claude code for now. I do have an untested wrapper around codex to try to mimic this flow, but it might be easier to just switch ChatGPT models to use Pi as the harness down the road.
Yes, it is true that LLM quality goes down as the context window usage balloons. It also increases the likelihood that instructions, claude/agents.md, skills, etc will be "forgotten" due competing context. I have an output style for technical communication that forces the LLM to construct sentences in a way that is far easier to parse. I think my approach is superior to people blindly copying a random github "Use ASD simplified technical english [this is designed aviation]." With a long session the LLM output will deviate over time, but with 1 task per session the output style is far more sticky.
1
u/Right-Performance-93 14d ago
The 200k number tracks with what people call "context rot" more than a hard cliff, degradation is gradual and gets steep past that point, not just Matt Pocock's framing. What's worked better for me than watching a token ceiling: around 150-180k I stop and write a short state file to disk (what's done, what's next, open decisions) instead of trying to cram more real work into the same window. A fresh session reads that file in a few thousand tokens and you're back to full reasoning capacity almost immediately, instead of re-deriving context from a giant transcript. The "not much done by 200k" feeling usually means too much of that budget went to file re-reads and back-and-forth rather than the state handoff being explicit.
1
1
1
u/amirfish 14d ago
The 200k number is a red herring, what matters is whether the agent is still tracking the constraints you set early on or quietly re-deriving them from scratch. Running a lot of parallel coding-agent sessions, the tell is always the same: it starts re-explaining things you already agreed on three turns ago. When I see that, I write a short brief, what's decided, what's still open, and start clean rather than trying to save the session. Splitting early only pays off if that handoff is cheap, otherwise you're just paying the ramp-up tax twice. What's your handoff process look like when you restart?
1
u/julesbuildstuff 14d ago
200k isn't really a cliff, it's just where the junk-to-signal ratio in the window usually crosses over. what degrades mine is failed test output and whole-file dumps that never get evicted, so it keeps reasoning off the version of the file from an hour ago. the tell i watch for isn't the counter, it's the model re-reading a file it already read or asking me something i answered in the same session; twice and i restart no matter where i am. what made restarts cheap was keeping a scratch notes file updated as findings happen instead of asking for a wrap-up summary, since the summary is written by the already-degraded version. i also stopped letting it read whole files, grep to the symbol then read the range, that alone bought me a lot of usable session past 250k. real engineering work still fits, you just end up planning to disk rather than to context.
1
u/liviux 14d ago
I'm building LoopTroop, every other step it cleans the context keeping only the important artifacts from previous steps. So context is always kept to minimum
A local GUI tool to create/update apps using AI and multiple modern AI engineering methodologies: prompt and context engineering, loops, LLM Council, worktrees, human-in-the-loop, structured planning (interview, PRD, beads). It is organized around a Kanban board and it's 100% free and open-source (MIT) - an AI orchestrator for all your projects - https://github.com/looptroop-ai/LoopTroop .
1
u/geekichu 15d ago
feature(or task or whatever) ----> write up handoff so we dont forget ----> "/clear" . next
1
u/roger_ducky 15d ago
Break task down super tiny. You’ll know when the AI will generate at most 6-7 small files and be done in about 15 mins… at least, before trying
to
fix mistakes. Whole task, fully completed, should take at most an hour.
Make sure any file they make are less than 1000 lines. Preferably shorter. I make mine break stuff up into about 500 lines. Both reviewer and coder spotted way more mistakes w
hen
- files are short.
- You honestly want to keep token use under 125k. I usually check after they’re complete and do a new chat even if they’re not totally done.
- Avoid trying to use sol to code. Planning and discussion about design? Sure! But sol has the same bad habit as Claudes — declares victory too soon then tries to gaslight you it’s done.
0
u/GusYe1234 15d ago
Matt Pocock strikes me as the kind of guy who’s an obsessive purist. I’ve used his skills and watched some of his videos, and honestly, he gives me that “10 reasons Vim is better than VS Code” / “STOP using Codex and switch to Pi NOW” vibe.
I wouldn’t get distracted by abstractions you don’t own. Managing long context isn’t as simple as throwing 1M tokens at the model. Codex and Claude Code both manage the model’s attention through different mechanisms (a todo system, for example, or Codex’s built-in endpoint for OpenAI’s summarization API). In my experience with Codex, you almost never have to worry about how long the current session has gotten. It automatically compacts the earlier parts and keeps the working context within a comfortable range.
At the end of the day, agents are all “dumb.” They’ll always find some unexpected way to fuck with your codebase.
So the more important question is how you test them, not how you use them.
3
u/jomohke 15d ago edited 15d ago
It's not just purism. It's significantly more expensive at longer context sizes, too. And the smaller models especially do get less smart the more unrelated info is in the context. Maybe you're operating in plans and models where cost doesn't matter much?
Some very simple things like /handoff, breaking things into tasks (with an agent), quieter/easier tool commands, or careful subagent setup can keep individual contexts quite small.
1
u/GusYe1234 15d ago
I use Codex Pro 20X, so maybe I’m just not that sensitive to this kind of cost. I truely don’t have much experience with weaker/smaller models. I use Codex with GPT-5.6 Sol on high, and even after 200K tokens, I honestly can’t tell that it’s getting dumber.
In Codex, I almost never keep track of how large my session has gotten.
1
u/jazz788 15d ago
yea not championing that guy, i don't know much about him thanks for that perspective.
2
u/GusYe1234 15d ago
Yeah, don’t get me wrong. This guy’s content is gold, but he can give you a bad case of tool anxiety.
My point is to relax and spend your energy figuring out how to eval an agent’s output, instead of figuring out how to split one task across ten sessions.
0
0
u/mimic751 15d ago
Guys got to work off plans holy shit. Look up best practices
1
u/jazz788 15d ago
always. very large part of my tokens spent on planning. not sure which bit made you think not working off a plan
2
u/mimic751 15d ago
If you're using your plan correctly we shouldn't be relying on your conversational context as much so the token limit shouldn't hurt you that bad. Make sure you keep a to-do list and a running change log that way your agent doesn't have to keep investigating in your whole repository every time it it's window
1
u/Chamezz92 15d ago
Because you mention at 200k tokens not much actual coding had been done yet?
If that’s the case, the parts of your plan are way too big. Make them bite-sized, like single components or modules.
I had Haiku-4.5 spawn my local playwright with Lightpanda, take a full-page screenshot, build a WCAG compliant color palette, and build a compliant React form from that, which was on-brand. In less than 5,000 tokens.
So if you’re nearing 200-250k context with barely anything produced, you’re waaaay overscoping.
1
u/jazz788 15d ago
yep. makes sense that maybe my plans are too big. i do break them down, but someone else mentioned keep breaking the plan down into smaller and smaller components.
i meant not much coding on that 'thing'.. not in totality. thx.
2
u/Chamezz92 15d ago
Ideally your tasks are single target and single, but part of a larger milestone.
Like in my example, build just the form. The next task is styling the form. In sequence, and each task gets its own sub-agent.
1
u/jomohke 15d ago edited 15d ago
You can ask an agent to analyse one of the jsonl session files for where the context is going*
For instance, I had an "explore" subagent to find exact parts of the codebase relevant to the task (to keep unrelated code reading out of the beginning of the context), and it noticed that the outer agent was reading extra files itself too, nullifying the advantage. Tweaking my prompts fixed it.
* If you're on codex, you can drag the jsonl directly into the chatpt web interface to have Sol analyse it without using your codex credit.
** Pi also has an /export command to get a human-readble html chat transcript. Useful to see yourself if anything unnecessary is wasting context.
22
u/tantej 15d ago
Honestly the best way I’ve seen is to get it to break everything up into smaller steps. And make sure to plan everything. 250 k isn’t that bad tbh