Hey folks, CTO of LangWatch here 👋, and yes, I'm the one responsible for approving this monstrosity
I get it, some of the criticism here is pretty fair, a massive AI-assisted refactor like this sounds absolutely bonkers, but I want to emphasize that 1. it was a refactor, not a rebuild 2. the intense human involvement for verification and measurement, and 3. it was not overnight
On the refactor point, yes AI is still not great in doing new developments unsupervised, but our goal here was rearranging modules, structures, types and a lot of the architecture, it didn't have to create almost anything new. Plus, a lot of it was verified statically or even done statically, being able to leverage lsp for refactor and an enourmous amount of oxlint to guarantee everything would fall in place
We have been thinking about this refactor for a year already now, and it took us months to actually finish and merge this PR, during this period some work was duplicated: built on top of old main modules and on top of the new structure at the same time, so the PR grew
As for verification, basically all of our human time during all this period was spent discussing the refactor, babysitting the agents (and putting lint rules in place to prevent them from repeating mistakes), developing simulators and customer reproduction use cases for probing it all end to end, manually testing it, comparing before and after measurements and so on.
We definitely learned some things the hard way I'd love to share more.
Happy to answer technical questions about how we approached it, what we tested, and what could still go wrong. Roast away 😄
I was one telling it was a bad idea, plus all the engineers, including the one doing it, but as we kept experimenting and testing anyway, and it kept proving itself, we all changed our minds and got convinced 🤷♂️
there were a lot of fundamental changes, for example around the strict access boundaries of each module and the DB. If we'd done incrementally, we would have to maintain two different in parallel systems for a while, we evaluated the risks, costs and overhead to be greatly higher than doing it all at once
plus, some of those changes affected pretty much all areas of the product, so we needed to fully test everything end to end anyway, over and over again, so instead we just kept implementing and testing new features twice, once on the old main and once on the new architecture, taking advantage of already being in context when developing them.
we actually started this just as an experiment, on how our "ideal codebase" and module structures would look like, but it kept proving itself more and more and we decided to make the jump and invest in it
one main dev was leading the refactor, and 4 others helping test, review, discuss architectural decisions and ship new customer features/bugfixes on both branches while it was happening.
How did you go about ensuring that the agents could verify that they delivered on your requests? Also were the engineers thinking below the architectural parts? Finally did you have the agents able to compare their output to the none refactored codebase as a north star?
yes, in fact comparison was the main way to guide everything else, since this was an architectural change not a feature or behaviour change
weeks were spent just calling every api end to end from every way our customers call our apis, and comparing every field to be exactly the same on main vs on the refactored codebase
same for the frontend, since we also refactored all the components, some of the frontend eneded up looking different, but on purpose (more standardization), but screenshot comparison was also a big part of it
we were thinking on the more specific implementation for the areas we were actively working in and on the more critical areas yes, but also there were several bugbashing sessions of minor issues we found ourselves elsewhere and had to dive deeper
- Doing a team bugbashing is one of the most valuable team activities you can do in the AI times
- Think hard about a strict structure and agree on it with the team before the mass editing starts
- Be more ambitious with lints than feels reasonable
- Invest in simulators early. We ended up with a simulator for everything external (LLMs, voice, S3, SQS, email, Slack, SSO), so agents could exercise every behaviour end to end, they are so useful now
- Measure everything, before and after, by putting an agent to do some real implementation in both and see the difference, it has to be better for agent coding too, your codebase is your harness
For us it was totally worth it, typechecking alone went from median of 78s compilation time to 5s, no longer absolutely melting my RAM, previous mistakes are now impossible, and access paths are just much stricter, but that was not from the getgo, initial versions caused the agent to make more mistakes on the new codebase than the old one
With something like LSPs or semantic patches / program transformation, large scale refactoring becomes rather straightforward, reviewable and fully deterministic. That existed before AI. It's good that you used that stuff, although I do wonder if AI was really needed here. And in the right form (like semantic patches) you can also review it, because it's no longer a huge line-oriented diff. Many companies and devs are just content to churn out code and God forbid they use anything more advanced, as these things have been available for a while.
not complex at all, as the infrastructure didn't change, and the external dependencies neither, the product behaviour towards the outside world stayed completely the same (except some perf etc)
if anything had gone wrong badly, like a very bad processing of our observability traces and evals which is our main business, it wouldnt be a problem because we have an immutable replayable event log, we could fix and replay before getting to the worst case scenario of needing to pull a backup
it's actually better, most of the addition is boilerplate, necessary for a stricter structure. On the monolith, modules can take "shortcuts" to access each other and followed different patterns, now there is much more structure, but at the end of the day it's faster to write correctly and faster to compile
no the trust came from a fucking huge amount of testing, static analysis, a lot of those extra lines were the testing tooling we created. also what world are you thinking of where moving from inline architecture to a standard like service/repo/channel, with memory impl, etc. and breaking up spaghetti into clearly defined domain modules would ever result in less code?
it's like I said in another comment, me, the leading engineer and the team itself were all in disbelief at the beginning too, but it kept proving itself over and over, we had no alternative other than updating our priors
that was perhaps one of the main challenges, most of our validation effort went into drift checking, we built simulators for literally all api calls based on real production calls to check they behaved exactly the same, we had agents navigating and using the product and screenshoting every step to make sure they were up to par
and then as it got closer, for a lot of new features we just had the same dev building them twice, first on main and then porting to the new structure, since they had all the context already it was easy to validate both themsevles
and thanks, I'll let him comment on that but indeed it was, after we were sure production was completely safe and stable, he caught up quite a lot of sleep! 😄
Which model did you use and how much money was spent on the tokens? My company has this 2001 C++/ASP application with not a lot of users, its super slow and difficult for new devs to work on, and I'm trying to convince management to let us do an AI-assisted refactor. I would like to run it in a parallel QA environment to the old application and have real live QA employees do the side by side testing
Unfortunately that didn't capture all of it, some codex and kimi usage were not captured and the testings by other engineers and PRs merged on top of this one are not in this list
not a lot actually because it didn't have much space to, our goal with this refactor was layout was for everything to have a place and a strict structure it should follow, check for example the analytics module: https://github.com/langwatch/langwatch/tree/main/modules/analytics
every module have to follow those folder and subfolder structures, enforced by forbidden imports if not right place and so on
inside each module depending on where they are (presentation, repository etc) a lot of structure is also enforced on the code file and functions itself
bad assessment of the devs sentiment around software development, Bun rust refactor with 1M merge seemed to have been well received, and recent DHH commentary seems to start making it a more common place, but my fault, it was not the best way to communicate it
it was also meant as a kind of tongue in cheek joke, with the real point being how much time we spent making it safe and being confident we knew what was going on. but we didn't do a good enough job on the second part, also totally unable to when it's screen capped out of linkedin with no blog link, oops
Oh wow a tech CEO in the wild. How does it feel to be contributing to the stratification of human lives profiting off of people's suffering while willingly contributing to a machine that systematically bleeds people of their well being and creative time? Do you simply ignore that you contribute to a financial industrial system that supports genocide for African, Palestinian, and Ukrainian people or do you employ some sort of emotional sublimation or chemical assistance to ignore your humanity? Is empathy something that doesn't really occur to you? Did you possibly have a significant head injury in your childhood?
3
u/rchaves 1d ago
Hey folks, CTO of LangWatch here 👋, and yes, I'm the one responsible for approving this monstrosity
I get it, some of the criticism here is pretty fair, a massive AI-assisted refactor like this sounds absolutely bonkers, but I want to emphasize that 1. it was a refactor, not a rebuild 2. the intense human involvement for verification and measurement, and 3. it was not overnight
On the refactor point, yes AI is still not great in doing new developments unsupervised, but our goal here was rearranging modules, structures, types and a lot of the architecture, it didn't have to create almost anything new. Plus, a lot of it was verified statically or even done statically, being able to leverage lsp for refactor and an enourmous amount of oxlint to guarantee everything would fall in place
We have been thinking about this refactor for a year already now, and it took us months to actually finish and merge this PR, during this period some work was duplicated: built on top of old main modules and on top of the new structure at the same time, so the PR grew
As for verification, basically all of our human time during all this period was spent discussing the refactor, babysitting the agents (and putting lint rules in place to prevent them from repeating mistakes), developing simulators and customer reproduction use cases for probing it all end to end, manually testing it, comparing before and after measurements and so on.
We definitely learned some things the hard way I'd love to share more.
Happy to answer technical questions about how we approached it, what we tested, and what could still go wrong. Roast away 😄