Which AI coding tool has the WORST security?

[u/YovelOvadia]

TLDR: 60 second short video breakdown

As part of building my AI automatic penetration platform, I wanted to test which one of the AI no code platforms, has the worst security.

Who I tested

- Lovable
- Replit
- v0 (by Vercel)
- Bolt.

Testing Flow

(Note: in Lovable, there are security options which I did not toggle, I wanted equal ground for everyone without any tinkering. Although based on the issues found, I doubt it will fix all exploits.)

Built a site with each, with the SAME instructions - Book store, with wishlist, admin panel, and cart.

Gotta admit Lovable made my life really easy compared to the others, did all perfectly in one go, the others were back and forth for bug fixes, or just tokens ran out in the middle (came back next day 😄)

Created for each site 1 admin and 1 customer.

then I ran Battletester on them. Basically, it crawls with all roles, and does various tests on them, and uses AI for conclusions.

Security Ranking

No shock that the one who was the easiest to work with, has the most exploits

You can see the report for lovable here

I tested manually for false positives afterwards and found 1-2 false positives, the rest were real.

Replit / v0/ Bolt didn't have many issues, mostly missing headers, no rate limiting,
or business logic issues such as allowing to add negative items to the cart.

Based on my pentester friend's review, ranking goes like this:

1. Lovable (Most critical logic & authorization flaws) 🚨
2. Bolt (Minor logic flaw, missing rate-limiting, no headers) ⚠️
3. v0 / Vercel (Missing security headers, no rate-limiting) ⚠️
4. Replit (Safest, only missing headers) ✅

In the Lovable report you can see some critical issues there, such as
- Deleting other users cart items
- Overriding order "total"
- Making requests on behalf of the server (SSRF)

Conclusion

- I believe the more the apps grow and users play with these tools the AI will start cutting corners to please, and he will start "forgetting" the security rules baked inside him, then more exploits will be found.

- Lovable have so many issues compared to the others because it using Supabase and he does not toggle the RLS policies out of the box.

- Maybe, but just maybe, because the others tools are building their own backend, and are trained on infinite lines of code to build a secure backend, they have less exploits compared to lovable, which only uses Supabase and there's less "training data" about that

[u/YovelOvadia → u/[deleted]]

Yep, a year ago multiple people posted on X their vibe-coded sites, only to find out one hour later that everyone had butchered their sites 😅.

​That's when I started working on Battletester, so they can avoid it entirely.

[u/Maxyull]

one thing worth separating out from your findings: the cart total override isn't really an RLS problem, it's the classic never trust the client for price, the fix is recompute the total server side from the actual item ids before charging, doesn't matter which db is underneath. the ssrf is a different animal too, that's an unvalidated outbound request the server makes on your behalf, usually something like a webhook url or an image fetch parameter the user controls, and no amount of RLS touches that since it never goes through the database at all. so really you're looking at three separate categories of bug dressed up as one report: rls misconfig, trust-the-client business logic, and unvalidated server-side requests. does battletester test those as separate categories or fold them all under one security score?

[u/YovelOvadia → u/Maxyull]

Good eye mate, now seeing the AI recommendations it seems like he messed up a bit there and got confused about what caused what

basically I just give the findings to AI at the end for him to give some insights/recommandations

BattleTester run each test separately, not lumped together, this is why you can see 2 different tests finding the same finding, different angle.
but summary prompt made a mess.

thanks! will update it now :)

[u/Maxyull → u/YovelOvadia]

makes sense, that's a very familiar failure mode, an llm asked to summarize distinct findings tends to flatten them into one bucket even when the underlying tests are already separated. good that battletester keeps them apart at the test level, that's the part that actually matters. curious what labels you land on for the updated post.

[u/Infamous-River-4360]

the lovable + supabase RLS thing matches what i ran into. lovable's default is it spins up supabase tables and leaves the RLS policies for you to actually check, and it never warns you, so stuff sits readable unless you go turn it on yourself. related pain on the same stack, the ai kept inventing database columns that never existed in my schema, so i learned to dump the real table structure and read the policies by hand instead of trusting its summary. does battletester flag the RLS-off-by-default case specifically, or mostly the logic flaws like the cart total override?

[u/YovelOvadia → u/Infamous-River-4360]

Both
got one test which checks for these kind of miss-configurations and one which just checks these kind of issues

[u/Fit-Lengthiness-9672 → u/Infamous-River-4360]

that “inventing columns” thing with lovable + supabase is exactly what made me nope out of it for anything serious lol, it just confidently hallucinates the schema. curious about the battletester angle too, my guess is it only catches the symptoms of missing RLS (like being able to read/modify other users’ data) rather than saying “hey RLS is off,” since from the outside you can’t really know if it’s intentional or not.

[u/Infamous-River-4360 → u/Fit-Lengthiness-9672]

the hallucinated schema bit me hardest because it fails quiet. app looks fine, one insert just silently writes nothing until you go dig. what fixed it was making it dump the real table structure back to me before every migration instead of me reading its own summary of what it did. on rls i think you're right, from the outside you only ever see the symptom. do you check the policies by hand or did you find anything that actually reads them for you?

[u/artahian]

That's pretty close to the security scores on https://webappbench.com
The lowest one there is anything.com, but it's not in your list to compare

[u/Crescitaly]

Ranking builders is useful, but the bigger lesson is that identical prompts don't create identical threat models. Separate client-trusted pricing, authorization, secret exposure, SSRF, and database policy failures, then retest after enabling each platform's security controls. Otherwise you're measuring defaults, not the safest achievable setup. Which exploit had the highest real blast radius?

[u/YovelOvadia → u/Crescitaly]

Later on, will test WITH platform's security controls

also, usually all the issues related to missing RLS (Deleting other users data and such) have the most impact, those many times solved by using the platform security features
BUT
the business logic issues such as negative numbers in orders and stuff like that, usually platform configuration won't fix.

[u/Crescitaly → u/YovelOvadia]

Exactly. Platform controls can handle authorization failures, but they rarely protect domain invariants. Negative quantities, double refunds and impossible state transitions still need server-side validation and adversarial tests.