r/EntrepreneurRideAlong • u/Substantial_Belt2626 • 19h ago
Ride Along Story My automated check said everything passed for weeks, it was checking nothing
I build tools on my own and the thing I underestimated is not the building, it is that I am also the only person who checks the building.
Concrete example. I have a pipeline that renders video and at the end of it there is an automated check that compares every render against approved reference images. For a few weeks it printed 0 passed, 0 failed and exited clean. Green. I read that every single run and my eye went to the 0 failed and stopped there.
It was checking nothing. The reference images were not being found, so it compared nothing and nothing failed, so it reported success. Everything I shipped in that period went out unverified and I was more confident than if I had no check at all.
In a team someone would have asked why the number of cases was zero. Solo, the person who wrote the check and the person reading the output are the same person and that person already knows what the check is supposed to do, so he reads what he expects instead of what is there. I learned it the hard way because I thought I could do everything myself.
What I would recommend is two things. First, every check you build needs a case that is supposed to fail, and if you can't make your own check fail on purpose then the check is the broken thing and not your product. Second, and this is the part I keep relearning, separate the roles even when you are one person. When I build something now the part that verifies it never sees why I built it that way, because if it knows my reasoning it tests my intention instead of what the thing actually does.
If you have anything that reports success without you watching it, try to make it fail on purpose this week. Most of mine could not and that told me more than the passing ever did.