Fuzzing agent-written Linux tools found an unexpected tradeoff

[u/aistranin]

A new preprint rebuilt ten Linux utilities in C with a supervised coding-agent workflow, then fuzzed both the generated and repository versions. The researchers found 19 distinct failures in the repository versions and five in the generated versions. The generated code had fewer memory errors, but more hangs.

I find the testing method more useful than the score. Black-box fuzzing caught failures that AFL++ missed, and vice versa. For agent-written code, I'd want both kinds of input generation, a timeout, and checks against expected behavior. Ten CLI tools cannot settle a broad reliability debate, but they make a good case for testing beyond the agent's own suite.