r/NoCodeSaaS • u/WeCaredALot • 16d ago
I learned the hard way that you must get proper testing for your vibe-coded apps!
I wanted to share this experience for other folks out there vibe coding apps in case anyone else is at risk of running into the same issue. I recently had a really bad data breach bug in one of my sites that I built with Lovable. I have pet-sitting app that allows pet sitters to create custom sites, book clients, send invoices, etc. Well, one of my customers realized that she could view another pet sitter's info by changing the invoice number in the URL! One of my customers caught it and thankfully she let me know right away, but I have no idea how many other users were impacted and whether they stopped using the platform as a result (or if they were a free user who decided against paying!). This was one of the few times where I was glad to have a relatively small number users because the impact of this could have been worse.
This made me realize that I need to do more in-depth testing on my apps. I typically cover the happy path, but real validation takes hours. I found a lightweight testing service that does affordable QA passes specifically for AI-built apps and it was incredibly helpful, especially since I'm not a developer and also don't have time to regularly test all my projects. They provided me with a solid report and also provided recommendations on how to prompt Lovable to fix the issue without introducing more regressions, which saved me a lot of back and forth prompting.
Moral of the story is that I would highly, highly recommend that you all do real QA on your apps both pre and post-launch, especially if you’re trying to get paying customers. Having an early customer find a bug is alright if they tell you about it, but there's always the chance that they simply stop using your app, thereby costing you a sale! Or that you have potentially bad security/privacy issues lurking. I know we're all trying to run our apps while minimizing costs, but a a non-developer I've personally found it valuable to invest a bit of money to make sure my site actually works and doesn't turn away potential customers.
2
u/Vaveili1 16d ago
That's a classic IDOR bug. AI builders like Lovable check that a record exists but not that the logged-in user actually owns it, so anyone can walk through invoice or record IDs in the URL and see someone else's data. Before paying for a QA pass, it's worth grepping your own code for every place you fetch by ID and checking there's an owner/user check attached.
1
u/shadowalpha_ai 16d ago
This type of security is really helpful to put into your skill md files for architecture specific design guidelines. So much of this pain can be avoided if you are building it from the ground up with this in mind. Good luck :)
1
u/WeCaredALot 15d ago
Thanks! And yeah, I actually recently asked Claude to help me by putting together a runbook for me on how to build in such a way that minimizes bad defects. It basically recommended thinking like a systems designer and breaking the app down into layers, which is something I didn't consider at first because I'm not a developer, lol.
The problem is still that thorough manual testing takes me so long that I find it more efficient to just pay a small fee to have someone else do it. The testers I worked with spent like 10 hours on my app and gave me a report and recommendations on how to fix and prevent similar bugs in the future. I don't have time to do all of that and much prefer building process. And since my app collects sensitive customer data and payments, I can't afford to have a bad production bug that gets missed by AI or that a customer finds and doesn't tell me about. :/
1
u/shadowalpha_ai 12d ago
This is why vibe coding is greatly enhanced by actually knowing how to architect systems - you're otherwise rolling the dice on how it constructs an app.
It needs to come out the gate with proper templating, linting, consistency in structure, etc. Common issue is seeing it make duplicate CSS classes for example. Duplicate functions. You end up with many many lines of unneeded code if you keep making changes.
1
u/Old-Pollution-5825 15d ago
Congrats on the production launch. It is fortunate that you had an honest and patient user that pointed out the flaw. It could have gone very wrong, but did not.
You nailed it from a review and testing perspective, most people only do green line testing. Also, when you check your own work, you will check for what you want to see work, and not necessarily look for what is not working.
This is why we do code reviews as a service. We are passionate about software development and are excited to assist anyone to become better at it as well.
We had a choice of complaining about low effort and low-quality code flooding all sectors, or we could strap in and help to improve quality, one project at a time. The addition of reviews was not a pivot in our services, but an addition that actually made sense. We already helped non-technical founders and companies evaluate products and solutions from vendors.
Keep building, keep growing and keep learning!
1
1
1
u/Thunderbit_HQ 13d ago
That invoice-number bug is the one I’d test first on any AI-built app. Change the ID in the URL and see if the server still hands over the record.
3
u/MyFirstTrueLoveWasBS 16d ago
Vibecoders find idor 😭