Research · 1 October 2026
What 53 AI-built apps get wrong
Corrected 2 October 2026 (v1.0.2). The headline row was first published as “no security middleware (rate limiting, CORS)”. The rule behind it detects automated security checks, meaning dependency and code scanning; rate limiting and CORS were not measured. The testing row counts any sign of testing, not only a test file. No count changed. The correction.
Are vibe-coded apps secure?
We scanned 53 public repositories built with Lovable, Bolt, v0, Cursor, Replit and Claude Code on 1 October 2026. At least 72% had no automated security checks, nothing like Dependabot, CodeQL or Snyk (95% CI 58–82%); at least 45% carried a critical finding; and at least 19% had committed a .env file. Testing was not the gap: up to 91% showed some sign of testing.
had no automated security checks
carried at least one critical finding
showed some sign of testing (an upper bound)
The sample
Nine each from Lovable, Bolt, v0, Cursor and Claude Code, eight from Replit. Every repository declares the tool it was built with itself, in a GitHub topic or its README. Median size 27,573 lines (range 1,739–344,373).
Rates carry 95% Wilson confidence intervals. At this sample size the interval is the honest figure: “72%” means somewhere between 58% and 82%.
Missing
Read from each repository’s full file tree and manifests, so these are knowable for every repository. True rates, except rows marked “≥”: their rules accept loose path matches that can only make a repository look better than it is.
| Failure | Repositories | Rate | 95% CI |
|---|---|---|---|
| No automated security checksNo dependency or code scanning: Dependabot, Renovate, CodeQL, Semgrep, Snyk, Trivy, gitleaks or equivalent38 of 53 | 38 of 53 | at least 72% | 58–82% |
| The same, among repos with server code29 of 40 | 29 of 40 | at least 73% | 57–84% |
| No .env.example36 of 53 | 36 of 53 | 68% | 55–79% |
| No linter configured18 of 53 | 18 of 53 | 34% | 23–47% |
| No CI pipeline15 of 53 | 15 of 53 | 28% | 18–42% |
| TypeScript not in strict mode8 of 53 | 8 of 53 | 15% | 8–27% |
| No sign of testing anywhereNo test file, test configuration or test-runner dependency5 of 53 | 5 of 53 | at least 9% | 4–20% |
Found: floors, not estimates
The scanner reads a capped subset of files (a median of 16% of each repository) and never clones. Finding something proves it is there; finding nothing proves nothing about the files that were not read. “≥” marks each rate as a minimum.
| Failure | Repositories | Rate | 95% CI |
|---|---|---|---|
| Files beyond a maintainable size50 of 53 | 50 of 53 | at least 94% | 85–98% |
| At least one high-severity finding46 of 53 | 46 of 53 | at least 87% | 75–93% |
| At least one critical-severity finding24 of 53 | 24 of 53 | at least 45% | 33–59% |
| Request bodies used without validation (repos with server code)9 of 40 | 9 of 40 | at least 23% | 12–38% |
| Possible credential, including needs-review12 of 53 | 12 of 53 | at least 23% | 13–36% |
| A .env-family file committed10 of 53 | 10 of 53 | at least 19% | 11–31% |
| Committed credential, high confidence3 of 53 | 3 of 53 | at least 6% | 2–15% |
| Credential written to logs1 of 53 | 1 of 53 | at least 2% | 0–10% |
| Injection sink (concatenated SQL, eval)0 of 53 | 0 of 53 | none found | 0–7% |
What this says
The “AI-built apps have no tests” story looks wrong. Up to 91% show some sign of testing: a test file, a test configuration or a test runner. That is an upper bound, and it says nothing about whether the tests are meaningful.
Automated security checks are mostly missing. At least 72% have no dependency or code scanning configured, so a vulnerable package or a known-bad pattern can reach production without anything flagging it. Among repositories that serve requests, at least 23% use request bodies without validating their shape.
Secrets hygiene is better than the folklore. At least 19% committed a .env file. Confirmed credentials are rarer: 3 repositories, all opened by hand, and none was a live vendor key.
What this does not say
The most serious failures in an AI-built app are invisible from its repository. Row Level Security policies, admin role checks and tenant isolation live in the Supabase or hosting project, not in code. Nothing here is evidence that they are fine.
Rate limiting and CORS were not measured, so nothing here supports a claim either way about them. And a public repository is not a deployed product; the direction of that bias is unknown.
How accurate the scanner is
The first run reported that at least 15% had committed a credential. All eight repositories behind that number were opened by hand, and six were false: test fixtures, a sanitiser’s sample data, a share-link UUID. The scanner was fixed, and the published figure is 3 repositories, each confirmed by hand.
As of the study, the secret scanner was scored on every change against a labelled corpus of 42 cases, failing the build below 0.97 precision or recall. That is “no known regression”, not “correct”: the corpus is ours. See the full method and limits.
Data and citation
- · Anonymised per-repository data: dataset.csv
- · Full results and checklist: GitHub repository
- · Archived with a DOI: 10.5281/zenodo.23094508
- · The write-up: What 53 AI-built apps get wrong, on nicchin.com
Chin, N. (2026). Production readiness checklist for AI-built apps, with measured research on 53 public repositories. Zenodo. https://doi.org/10.5281/zenodo.23094508
Check your own AI-built app
The same scanner, on your repository: every critical and high finding with its file and line. Free on public repos, no signup.