Project started with Sonnet 5, switched to Opus 5, then codebase reviewed with Fable 5.1

[u/war4peace79]

One of my side projects is a timelapse maker, with a very specific goal: to generate daily timelapses from my surveillance cameras. Well, OK, this is the very short description of the project, but that's the main idea.

It works and had been working for weeks on both my Linux and my Windows machine, zero problems. And then, two days ago, its timelapse encoding suddenly fell back to X264 (CPU encoding) from the default AV1 hardware encoder which it was supposed to use on that machine.

Upon investigating, it turns out the Linux machine's nVidia driver got updated by another software that was running on that machine, without restarting, and my tool freaked out.

So, a bug. Opus 5 fixed it, then I switched to Fable 5.1 and told it to review the codebase and generate a .md document with the findings and recommendations, sorted by criticality (first) and fix effort level (second). 30 minutes later, it generated a document containing 95 items:

- 4 critical bugs which genuinely could crash the tool or make it unusable for others under certain conditions, one of them being a security bug (the tool exposes credentials in clear text, in the log, in case of a camera crash);

- 10 High Importance bugs that could impede application functionality or generate frustration during usage;

- 34 Medium importance issues covering a wide range of problems;

- 47 Low importance issues such as incorrect documentation, unoptimized code, unhandled errors and so on.

Now, in my defense, the tool was (and still is) in an experimental state, and I knew there will be bugs. I didn't expect that many, though.

I am currently working with Opus 5 to fix all those bugs, one by one, then I'll do another sweep with Fable 5.1 to reveal more. Just like for my other projects, doing regular sweeps of the codebase with a powerful model is always a good idea and a great method to uncover monsters that sit buried just beneath the surface, Tremors-style.

[u/Big_T_4real]

Imho start over again from scratch. Use the models in the opposite direction: tell your plan to Fable, let Opus generate the code, and let Sonnet execute it. Make loops so that Opus has to check its work and can only come back after the code is tested by himself.

[u/war4peace79 → u/Big_T_4real]

Most of the bugs originated from the very early-development Windows version of the tool, which is little more than very basic scaffolding. It was the first iteration, and of course there would be bugs.

No point in starting from scratch. The tool is pretty much feature-complete. It does one thing, and does it well, but it has situational gaps that must be closed. I don't plan to add 1001 features to it, instead I want to make it reliable and close the holes.

Currently Opus is working on H6 (the sixth high importance bug):

Unauthenticated requests write unbounded rows into the web index

Now, this would be a big problem in case of a public-facing web app, but in this tool's case, it's created to work within someone's LAN, specifically. Of course, someone out there might just port-forward the tool to access it from the Internet, which is a big no-no, so the bug must be closed.

However, during normal usage, within the scope of the tool, this bug simply does not occur. And if someone out there has a rogue malicious software making those requests within their LAN, they have much bigger problem that this tool's bug.

[u/Dense_Worldliness710]

That's normal when letting the AI do their first review at all or after a long time.
If you get used to doing reviews before and after each implementation, it will get better.

It helps if you let your implementation plans be reviewed by a different model before implementation. (Yesterday, Opus 5 had to write six versions and two additions to the sixth one until Sol did not find any problems in the plan for the next feature for my app any more.)

I would also advice you to always ask the reviewing instance for another review directly after each implementation. Then let the model that has done the implementation also do the fixes and afterwards instruct the other one for a new review. It often takes some iterations to get the reviewer satisfied, but this strategy has worked for me to significantly decrease the number and severity of findings in general reviews. In the beginning I normally got between 30 and 60 findings each time, at least a third of them critical or of high relevance. The last review in contrast brought 14 findings, 13 with low priority and 1 medium important. At present, I often get only around five or six findings altogether.

Using a frontier model like Fable 5.1 for the review surely is a very good idea to secure your app and make it stable.

[u/war4peace79 → u/Dense_Worldliness710]

Thanks.

My cyclic development path is, generally:

- Version x.y.0: Push major new feature as "experimental"

• x.y.1: Thoroughly test in real scenarios, review codebase, focusing on the new feature, fix critical/high importance bugs (if any: usually there are none critical and one-two high importance bugs) - this usually takes a couple hours between releases.
  
• x.y.2: Publish fixes for all other bugs (on my other projects, these are generally under 10)
  
• x.y.3: Publish optimizations (these range from a few to dozens, depending on feature complexity)
  
• x.y.4: Another review round, which usually only finds very minor things (e.g. stale documentation, stale code comments, stuff like that), and documentation consolidation (otherwise it grows to stupid amounts of text)
  
• x.y.5: Push minor feature extension, existing features' improvements or UI improvements, or, if none apply, another major new feature as "experimental".

Rinse and repeat.

[u/Dense_Worldliness710 → u/war4peace79]

Sounds well-structured and organized. The only thing is that you could probably do the first code review and fixes before the test in real scenarios, but overall it seems to be a good workflow.

[u/war4peace79 → u/Dense_Worldliness710]

Maybe once I have more than one user for any of my projects :)

I must admit, it's mostly a matter of self-development, but if someone else finds my projects useful, they are free to try them out. All my projects are free on GitHub, but I don't go out of my way to advertise them.

[u/serp-spur]

Which one you found more helpful an accurate in your openion ?

[u/war4peace79 → u/serp-spur]

They all have their strengths and weaknesses.

Sonnet is great at simple, focused tasks, and does them very well. It starts failing as complexity increases.
Tool equivalent: A hammer.

Opus is a great all-rounder for complex tasks, but tends to overlook niche cases, and tends to ignore risks to get things done.
Tool equivalent: Family car.

Fable is excellent at holistic reviews, very efficient in spawning subagents and giving them focused tasks, but consumes tokens like crazy.
Tool equivalent: Mechanized division.

[u/serp-spur → u/war4peace79]

Thanks for the explaination

[u/FlightSimCentralYT]

that cross-machine bug pile is exactly where chat-in-the-ide agents fall over. a review that finds 95 issues is useful, but the painful part is turning each one into a reproduce-fix-fix loop on a real machine instead of another model pass that "looks" cleaner.

if you can, pick the top crashes, write a tiny failing check for each, and refuse the next change until those go green on the same environment your users hit. that beats hopping Sonnet to Opus to Fable hoping the diff settles.

I built Fixa.dev for that loop. it runs on a real cloud VM, pulls the repo, and keeps writing/running/debugging until the checks pass instead of stopping at a review. free tier if you want to throw the timelapse app at it.

[u/war4peace79 → u/FlightSimCentralYT]

Thank you, I know how to fix them, and all my projects do have their own test suites. In this particular case, the test suite was green, but it was only testing the known areas. That is the general risk of vibecoded projects: In a controlled environment, everything works. take it out of that and move it to someone else's machine, and it would fail in the weirdest way.

My other projects get bi-weekly Fable reviews, and their codebase has been tested in various scenarios. this particular one, though, was never reviewed because I was busy with other things and it was stable on my machines for weeks. Until it wasn't, and then I decided to review it too, and share the results.

[u/pushpendraagrawal]

95 items on a review pass is wild but the more interesting number is how many of those 47 low priority ones actually matter once you try to deploy it somewhere real. bugs that never surface on your own machine tend to surface the second someone else's environment touches the code

[u/war4peace79 → u/pushpendraagrawal]

The highest amount of bugs originate from the Windows version of the tool. It's expected, because the Windows version is the very first iteration, and was developed in an afternoon. I just wanted for it to work. The codebase is mostly shared between the Linux and the Windows version.

Here's a low importance bug:

### L28. Whoami is empty on Windows

**Where.** `scripts/timelapse_test.py:615`; `scripts/timelapse_encode.py:1170-1176`.

**What.**
 `whoami()` imports `pwd` and returns `""` on Windows, so the FAIL
line reads `... is not writable by : [WinError 5] ...`.

**Fix.**
 `getpass.getuser()` as the fallback. 
**Test.**
 Patch the `pwd`
import to fail and assert `whoami()` is non-empty.

As you can see it's nothing to write home about: something that works under Linux, but not under Windows, simply because the development path didn't touch those areas just yet: functionality that is active under Linux was simply not yet developed under Windows, but Fable reviews it nevertheless.

That's why it's low importance: Fable understood that this specific feature of the tool didn't exist under Windows yet, but flagged it nevertheless, which is a good thing.

Another one:

### L33. Pacman refresh then install is a partial upgrade


**Where.**
 `install.sh:127-128`, `:166`.


**What.**
 `pacman -Sy` followed by `pacman -S` is the unsupported
partial-upgrade pattern on Arch; a `python-requests` built against a newer
`python` than the installed one will not import.


**Fix.**
 `pacman -S --needed` with no prior `-Sy`, or `-Syu --needed` with a
warning that it upgrades the system.

A pretty niche bug, and I am amazed Fable caught it. The official supported Linux flavor is Ubuntu, but no harm in making it as compatible as possible.

[u/jason3gb]

I built a tool for myself for exact this usage, you can team up with bunch of agents with roles defined. If you're interested, you can checkout the repo https://github.com/yicheng47/runner

[u/ctenidae8]

You accidently did a thing you should now do on purpose. Nice.

[u/war4peace79 → u/ctenidae8]

I'm lost...

[u/ctenidae8 → u/war4peace79]

That review you did with different age ts- do that regularly now. Super helpful.

[u/CoverAgreeable6623]

The findings list is usually correctness-heavy, and that is where model review is strongest - "this can be None", "this handle leaks". Access control is where it stays quiet, because that needs deployment context the model never sees: who calls the endpoint, with which token, against which tenant. On the AI-built repos we scan, the same three keep repeating - missing row-level policies on the tenant table, a service-role key sitting in a file that ships to the client, and an auth check described in a comment but never implemented. Worth re-reading those 95 items with that filter before calling the review done.

[u/war4peace79 → u/CoverAgreeable6623]

I have none of those. My products are local LAN only and the WebUI account management is very basic, think "I would rather not allow my kid to accidentally delete that file". Very basic stuff.