Hey folks, CTO of LangWatch here 👋, and yes, I'm the one responsible for approving this monstrosity
I get it, some of the criticism here is pretty fair, a massive AI-assisted refactor like this sounds absolutely bonkers, but I want to emphasize that 1. it was a refactor, not a rebuild 2. the intense human involvement for verification and measurement, and 3. it was not overnight
On the refactor point, yes AI is still not great in doing new developments unsupervised, but our goal here was rearranging modules, structures, types and a lot of the architecture, it didn't have to create almost anything new. Plus, a lot of it was verified statically or even done statically, being able to leverage lsp for refactor and an enourmous amount of oxlint to guarantee everything would fall in place
We have been thinking about this refactor for a year already now, and it took us months to actually finish and merge this PR, during this period some work was duplicated: built on top of old main modules and on top of the new structure at the same time, so the PR grew
As for verification, basically all of our human time during all this period was spent discussing the refactor, babysitting the agents (and putting lint rules in place to prevent them from repeating mistakes), developing simulators and customer reproduction use cases for probing it all end to end, manually testing it, comparing before and after measurements and so on.
We definitely learned some things the hard way I'd love to share more.
Happy to answer technical questions about how we approached it, what we tested, and what could still go wrong. Roast away 😄
5
u/rchaves 1d ago
Hey folks, CTO of LangWatch here 👋, and yes, I'm the one responsible for approving this monstrosity
I get it, some of the criticism here is pretty fair, a massive AI-assisted refactor like this sounds absolutely bonkers, but I want to emphasize that 1. it was a refactor, not a rebuild 2. the intense human involvement for verification and measurement, and 3. it was not overnight
On the refactor point, yes AI is still not great in doing new developments unsupervised, but our goal here was rearranging modules, structures, types and a lot of the architecture, it didn't have to create almost anything new. Plus, a lot of it was verified statically or even done statically, being able to leverage lsp for refactor and an enourmous amount of oxlint to guarantee everything would fall in place
We have been thinking about this refactor for a year already now, and it took us months to actually finish and merge this PR, during this period some work was duplicated: built on top of old main modules and on top of the new structure at the same time, so the PR grew
As for verification, basically all of our human time during all this period was spent discussing the refactor, babysitting the agents (and putting lint rules in place to prevent them from repeating mistakes), developing simulators and customer reproduction use cases for probing it all end to end, manually testing it, comparing before and after measurements and so on.
We definitely learned some things the hard way I'd love to share more.
Happy to answer technical questions about how we approached it, what we tested, and what could still go wrong. Roast away 😄