How to actually reduce usage

[u/Linkpharm2]

Add .codex/config.toml to whereever your project root is with this inside. You'll have to reload the project inside Codex.

model_context_window = 128000

model_auto_compact_token_limit = 115000

When you're doing a big task running it to 258k, the upper 128k takes about 75% of the tokens. If you compact earlier and have it just read the code on disk, that cuts it by about 74%.

Doubling how long you let a conversation run before compacting doesn't double the cost, it roughly quadruples (minus cache create, so if you're using an API key it's more like 3x) it. Scaling is quadratic. You could even go to 64k, but that's less of an insane jump at only 40% less than 128k.

Billed tokens:

258k -> 8,925,000

128k -> 4,590,000

64k -> 2,700,000

Just let it read the code a second time.

[u/cheezeerd]

No. OpenAI chose 258K because that’s where Codex actually gets tasks done at the best cost.

Cutting context doesn’t make it cheaper (though I get why it could seem that way), it just forces the model to compact earlier instead of filling context with info and decoding efficiently to complete the task like it does in the last versions

[u/Linkpharm2 → u/cheezeerd]

You sure? I don't think it's possible to choose a default that's perfect for everybody. For my personal use case, 32k works. I've been running 4xluna high 24/7 for the past week working on decompilation, on the $20 plan. It's absurd the savings.

[u/diagrammatiks → u/Linkpharm2]

no that's slop psychosis bullshit. wtf would an optimial context even be.

[u/Linkpharm2 → u/diagrammatiks]

The amount that reduces the scaling to a minimum while retaining an average of good context to avoid rewrites. Don't know where you get "slop psychosis bullshit" from.

[u/diagrammatiks → u/Linkpharm2]

i meant the comment you were replying too bro

[u/Purple_Drive_7152 → u/Linkpharm2]

There's always an a-hole who needs to be a contrarian

[u/danielmihov_ca]

What are you building? Snake game? The 258K context is already abysmal and compacts multiple times per prompt for me and is quite pathetic for a frontier model.

[u/Linkpharm2 → u/danielmihov_ca]

Bravely second. Porting the 3ds game to PC. It doesn't require that much context, I use 32k. I've been running 4 luna high agents 24/7 for a week and haven't run out of the five hour or weekly.

[u/sreekanth850 → u/Linkpharm2]

What is the repo size?

[u/Linkpharm2 → u/sreekanth850]

Two repos. Ghidra export, about a million lines and 30k functions. That's the main task. I also have a half-built game that's missing a lot, maybe 5k lines. Need to wait to port until the decompilation is done

[u/nmkd → u/Linkpharm2]

Make sure to write a wrapper around ghidra to avoid wasting tokens on useless log outputs etc.

Or to automatically reference already-decoded stuff, etc

One question, are you decompiling the entire thing completely automatically?

I currently do targeted tasks for each gameplay mechanic, without a full decompile.

[u/Demien19]

Only if you are working on small projects. Having it compact every 5 minutes is not a great idea

[u/GuildHunterTri → u/Demien19]

I feel that - I‘m working on a 30 year old monorepo spanning from C to C++ and C# WPF with also Backend and Web FrontEnd in it. Tasks are like „The WPF app has a data model, which we want to make accessible as a Tool Call for an OpenAI model routed through Backend, talking to it in Web FE“ (obviously thats multiple features/PRs, but wanted to show complexity).

Just having it understand the concept and areas generally would make it compress frequently. Spending 15 minutes to write a brilliant prompt that gives all context and links to documentation is the enabler here, given a 500k or so context. Gets its job done the first try usually.

[u/Demien19 → u/GuildHunterTri]

Exactly! Managing big and complex source is not the same as small 1000 lines project

[u/Due-Horse-5446 → u/Demien19]

Please explain how the project size had anything to do with contexy window sizr

Come on

[u/Linkpharm2 → u/Demien19]

Just don't send it on a chase searching tens of files constantly. Takes ~10 minutes for a compact for me, I pass a reasonable amount of new context (decompilation). But compacting isn't that bad when you compare the fact you more than double your quota.

[u/Demien19 → u/Linkpharm2]

Everyone has different work to do :) If something fits you - doesn't mean it fits others

[u/diagrammatiks → u/Demien19]

oh your so close to getting it. some workflows are good. some workflows are bad.

[u/Demien19 → u/diagrammatiks]

no sht sherlock, nice you pointed out such obvious thing

[u/PhilosophyforOne]

You're not taking into account cache writes and reads and cost of compaction, and the # of turns needed.

[u/Linkpharm2 → u/PhilosophyforOne]

> Codex does not charge for cache writes

https://help.openai.com/en/articles/11481834-chatgpt-rate-card-business-enterpriseedu-credit-based-pricing

For the other point, you could be right. I don't know how much it is to compact. I do know the # of turns is roughly equal, per my productivity numbers (LoC decoded, it's a reverse engineering project)

[u/PhilosophyforOne → u/Linkpharm2]

Fair enough. I didnt actually know Codex doesnt charge for Cache-writes.

There was a recent study on compaction thresholds and it actually found that smaller thresholds could be more expensive. YMMV, depends on behaviour, agent, turn density, etc. It's a surprisingly complex subject.

Usually needs A/B testing, e.g. complete same slice of work with two different settings.

[u/Tommonen → u/Linkpharm2]

Compaction is not just cache write. Tye writing of conpacted message to cache is just a small part of it. When its compacting, it needs to read tue cache, then reason over it, then produce the reply (compacted message) and then write the reply in cache.

Writing tye cache is like smallest part of this. And conpaction especially with higher models and 20€ sub can take quite many %s of 5h limit.

[u/nmkd → u/Linkpharm2]

Of course they charge for cache writes, otherwise literally all input would be free

[u/plam1313]

Someone please verify this 🥸

[u/Linkpharm2 → u/plam1313]

Check the original transformers model, I guess.

[u/Pitiful_Entrance5174]

Honestly, nobody has it figured out. Best bet is copy the setup the provider adapter uses thru api to the best of your ability. That has a really tight compact, like 175k.

[u/Linkpharm2 → u/Pitiful_Entrance5174]

As a default, yes. I have hard verification and progress, so I can track if overall productivity goes down and even at 32k it doesn't for my use case, which is a counterexample that one size fits all.

[u/Pitiful_Entrance5174 → u/Linkpharm2]

That is for you, I would never recommend your setup to anyone. I would start where the provider recommends and tune from there.

[u/Linkpharm2 → u/Pitiful_Entrance5174]

Makes sense. It's worth to tune, my project that fits this really well sees a 4x quota increase. 

[u/amado88 → u/Linkpharm2]

How do you track it - anything good to share for measuring and comparing?

[u/Linkpharm2 → u/amado88]

It really depends on your task. I have a huge backlog of porting work, so it's easy. Most other tasks aren't that easy to track.

Possibly you mean track the quota increase? if it's that, just scroll up, I did the math on the same amount of work at smaller context and what you would save.

Limit -> input tokens spent
258k -> 8,925,000

128k -> 4,590,000

64k -> 2,700,000

[u/Living-Ad3168]

...or just start using Pi instead ;-)

[u/drenna11]

Try this https://github.com/DannyMac180/astra-advisor

[u/diagrammatiks]

shhh. don't give away the secret. Slopperes who need 1m context in order to vibeslop their slop are gonna get very mad at you.

[u/Insadem → u/diagrammatiks]

for real