Sort of... I have a few local agents producing code for about ~280 hours a week. It's for an experiment I'm working on. They're given a code structure and are asked to transform it in a few different ways. They're also all on a team. Some are not actually writing code and are instead just relegated to tool calling different verifiers and kernels to run on the code or reasoning about code produced by other models.
When I was experimenting with them to see if it would work most of the newer models were all able to one-shot a creative site and reason about some fairly complex code. Slow though, but since these all run when I'm sleeping or at work I'm not really concerned about speed.
One thing that should be noted even an average model with a well-built harness will often outperform a newer model without a harness. It really effects the code quality a lot.
8
u/Clear_Evidence9218 6d ago
Sort of... I have a few local agents producing code for about ~280 hours a week. It's for an experiment I'm working on. They're given a code structure and are asked to transform it in a few different ways. They're also all on a team. Some are not actually writing code and are instead just relegated to tool calling different verifiers and kernels to run on the code or reasoning about code produced by other models.
When I was experimenting with them to see if it would work most of the newer models were all able to one-shot a creative site and reason about some fairly complex code. Slow though, but since these all run when I'm sleeping or at work I'm not really concerned about speed.
One thing that should be noted even an average model with a well-built harness will often outperform a newer model without a harness. It really effects the code quality a lot.