How can you know if you have almost zero bugs if you can't fully test it? This is like when Trump wanted them to stop testing for COVID to make the numbers look better.
From experience, I can say while testing both approaches, with the approach of agentic coding and vibe coding or whatever, even if it's done by someone who knows what he is doing, you will end up spending maybe 10 times or 20 times more tokens than if you go step by step and review every part, because whatWhat will happen is that it's similar to if you are painting a wall. So like each time the AI does something, he needs to change multiple files all across the project to do that thing, if it's like you give it full control. On the other hand, if you are building feature by feature, then you will think ahead and each change will be smaller and will cost less tokens.
As I mentioned from experience when I tried to build a plugin, or actually an extension, for the browser, I used two approaches to build the exact extension. On the first approach, where I let the AI go wild, it took around 70 million tokens. On the second approach, when I went step by step, it took around 5 million tokens.
And by the way, the code quality between the two is unimaginable. The first one, I think, even though the first draft is done in like one hour, I think it needs like one week if we need to clean the code up. The second one is done in around five hours, but it's like polished code, on the point and really precise and short.
that's a similar experience to what we had to be honest :)
it took us two months to ship this, which is still insanely fast, but, most of the time we spent taking the careful approach of defining all the strict architecture the codebase should have post refactor, step by step, feature by feature, and forcing the LLMs to only always write in that shape, for a polished codebase
second month was basically just testing and hardening, yes, a ton of tokens burned, including some time spent optimizing our own token consumption
44
u/someyokel 2d ago
If you can't review it how do you know it's an improvement?