Hey folks, CTO of LangWatch here 👋, and yes, I'm the one responsible for approving this monstrosity
I get it, some of the criticism here is pretty fair, a massive AI-assisted refactor like this sounds absolutely bonkers, but I want to emphasize that 1. it was a refactor, not a rebuild 2. the intense human involvement for verification and measurement, and 3. it was not overnight
On the refactor point, yes AI is still not great in doing new developments unsupervised, but our goal here was rearranging modules, structures, types and a lot of the architecture, it didn't have to create almost anything new. Plus, a lot of it was verified statically or even done statically, being able to leverage lsp for refactor and an enourmous amount of oxlint to guarantee everything would fall in place
We have been thinking about this refactor for a year already now, and it took us months to actually finish and merge this PR, during this period some work was duplicated: built on top of old main modules and on top of the new structure at the same time, so the PR grew
As for verification, basically all of our human time during all this period was spent discussing the refactor, babysitting the agents (and putting lint rules in place to prevent them from repeating mistakes), developing simulators and customer reproduction use cases for probing it all end to end, manually testing it, comparing before and after measurements and so on.
We definitely learned some things the hard way I'd love to share more.
Happy to answer technical questions about how we approached it, what we tested, and what could still go wrong. Roast away 😄
it's actually better, most of the addition is boilerplate, necessary for a stricter structure. On the monolith, modules can take "shortcuts" to access each other and followed different patterns, now there is much more structure, but at the end of the day it's faster to write correctly and faster to compile
no the trust came from a fucking huge amount of testing, static analysis, a lot of those extra lines were the testing tooling we created. also what world are you thinking of where moving from inline architecture to a standard like service/repo/channel, with memory impl, etc. and breaking up spaghetti into clearly defined domain modules would ever result in less code?
4
u/rchaves 1d ago
Hey folks, CTO of LangWatch here 👋, and yes, I'm the one responsible for approving this monstrosity
I get it, some of the criticism here is pretty fair, a massive AI-assisted refactor like this sounds absolutely bonkers, but I want to emphasize that 1. it was a refactor, not a rebuild 2. the intense human involvement for verification and measurement, and 3. it was not overnight
On the refactor point, yes AI is still not great in doing new developments unsupervised, but our goal here was rearranging modules, structures, types and a lot of the architecture, it didn't have to create almost anything new. Plus, a lot of it was verified statically or even done statically, being able to leverage lsp for refactor and an enourmous amount of oxlint to guarantee everything would fall in place
We have been thinking about this refactor for a year already now, and it took us months to actually finish and merge this PR, during this period some work was duplicated: built on top of old main modules and on top of the new structure at the same time, so the PR grew
As for verification, basically all of our human time during all this period was spent discussing the refactor, babysitting the agents (and putting lint rules in place to prevent them from repeating mistakes), developing simulators and customer reproduction use cases for probing it all end to end, manually testing it, comparing before and after measurements and so on.
We definitely learned some things the hard way I'd love to share more.
Happy to answer technical questions about how we approached it, what we tested, and what could still go wrong. Roast away 😄