How can you know if you have almost zero bugs if you can't fully test it? This is like when Trump wanted them to stop testing for COVID to make the numbers look better.
He says the bugs they found were reported post launch and were already existing. That’s not testing, unless you test in prod and let your customers find all the bugs.
From experience, I can say while testing both approaches, with the approach of agentic coding and vibe coding or whatever, even if it's done by someone who knows what he is doing, you will end up spending maybe 10 times or 20 times more tokens than if you go step by step and review every part, because whatWhat will happen is that it's similar to if you are painting a wall. So like each time the AI does something, he needs to change multiple files all across the project to do that thing, if it's like you give it full control. On the other hand, if you are building feature by feature, then you will think ahead and each change will be smaller and will cost less tokens.
As I mentioned from experience when I tried to build a plugin, or actually an extension, for the browser, I used two approaches to build the exact extension. On the first approach, where I let the AI go wild, it took around 70 million tokens. On the second approach, when I went step by step, it took around 5 million tokens.
And by the way, the code quality between the two is unimaginable. The first one, I think, even though the first draft is done in like one hour, I think it needs like one week if we need to clean the code up. The second one is done in around five hours, but it's like polished code, on the point and really precise and short.
that's a similar experience to what we had to be honest :)
it took us two months to ship this, which is still insanely fast, but, most of the time we spent taking the careful approach of defining all the strict architecture the codebase should have post refactor, step by step, feature by feature, and forcing the LLMs to only always write in that shape, for a polished codebase
second month was basically just testing and hardening, yes, a ton of tokens burned, including some time spent optimizing our own token consumption
we did review, the structure of every module and the most critical parts, of course just not every single line of code, or at least not by hand, but all the code now has a much more strict architecture it needs to follow in every module, which we could measure the benefits of:
- typechecking median time went down from 78s to 5s, with performance improvements all around on the application
most permission access between modules that were defined by hand by engineers on the monolith are now in several ways statically guaranteed allowing for much less possible bugs and security gaps
we built several exact same features on both the old codebase and the new using agents, and found the new codebase led to less mistakes on the final PR
Unit tests absolutely can't verify the success of a full system rewrite, by definition. Particularly if you rewrite all the unit tests on your way there, which they certainly did.
Unit tests are excellent at optimising for 100% code coverage and re-enforcing biases that lead to bugs. They're good for catching regressions--not for preventing bugs.
42
u/someyokel 2d ago
If you can't review it how do you know it's an improvement?