SECURITY TESTING TOOLS FOR AI-GENERATED CODE: THE LANDSCAPE

Why AI-built apps need a different tool mix

Security tooling categories were designed around how humans introduce vulnerabilities: occasionally, in reviewed code, inside codebases with established conventions. AI-generated code shifts the distribution. Secrets get hardcoded far more often because models reproduce example patterns verbatim. Authorization is missing more often than broken, and static tools are much better at finding broken code than absent code. Dependencies arrive by model suggestion, including packages that don’t exist. And the sheer volume — hundreds of accepted lines per hour — means any tool that requires per-finding human triage gets turned off within a month.

So the question isn’t “which scanner is best.” It’s which categories cover the failure modes AI code actually has, and where each category goes blind. Four matter, plus one that’s new.

SAST: static analysis, tuned for AI patterns

Static analyzers — Semgrep, CodeQL, SonarQube, Bandit for Python — read source and flag dangerous patterns: string-built SQL, eval on user input, weak crypto calls, unverified jwt.decode().

What it catches on AI code: the pattern-shaped majority. Injection sinks, insecure API usage, and known-bad idioms are exactly what models reproduce from training data, and exactly what rules match. Semgrep in particular earns its place because you can write custom rules for your AI tools’ recurring habits (“flag any route file that doesn’t import our validator”) in minutes.

What it misses: omissions and logic. A route with no authorization check usually produces no finding, because there’s no bad line to match — the vulnerability is the absence of a line. Business-logic flaws, BOLA, and misconfigured platform services (Supabase RLS off, Firebase rules open) are invisible to source scanning. Default rulesets also drown AI-heavy repos in style-grade findings; tuning is not optional. Our SAST tools for AI code page ranks the specific tools and rulesets by signal on AI-generated repos.

Secrets scanners

Gitleaks, TruffleHog, GitGuardian, detect-secrets: they match credential formats (AWS keys, Stripe tokens, JWT secrets, database URLs) in code and git history.

What they catch on AI code: the most common AI-introduced critical. Models hardcode keys constantly — copied from nearby example files, from documentation in training data, or just because the prompt said “connect to Stripe.” A pre-commit secrets hook is the single highest-value gate an AI-assisted team can add, and it’s nearly free.

What they miss: secrets that don’t look like secrets (internal passwords, non-formatted tokens), and the deployment-side leaks — a key correctly kept out of git but inlined into the client bundle by Vite or Next at build time. That failure only shows up when you inspect what actually shipped, which is a black-box job: the free token leak checker and env exposure checker test a deployed URL for exactly this.

DAST and black-box testing

Dynamic tools — OWASP ZAP, Burp Suite, Nuclei — probe the running application from outside, the way an attacker does: sending requests, mutating parameters, reading responses and headers.

What it catches on AI code: everything that only exists at runtime, which in AI-built apps is a lot. Permissive CORS with credentials, missing security headers, verbose error responses, exposed source maps and .git directories, reachable debug routes, and — critically for the Lovable/Bolt/Supabase stack — backend rules misconfigurations that no source scanner can evaluate. It’s also the layer that verifies findings are actually exploitable rather than theoretical.

What it misses: anything not reachable from the outside (internal jobs, admin tooling), and deep authenticated flows unless you configure credentials and session handling — which is where general-purpose DAST tools cost real setup time. Traditional DAST also assumes a security engineer is reading the output; the raw ZAP report on a vibe-coded app is not a solo founder’s afternoon. The security testing tools overview and scanner comparison cover the field in depth.

Dependency and hallucination checks

npm audit, pip-audit, Snyk, Dependabot, and OWASP Dependency-Check compare your dependency tree against CVE databases. For AI-built apps there’s a second, newer job: checking that suggested packages exist and are what they claim. Models hallucinate plausible package names; squatters register them; the install succeeds and the malware runs on postinstall.

What they catch: known-vulnerable versions (cheaply — there is no excuse for skipping the built-in auditors), and abandoned or unmaintained packages via freshness signals.

What they miss: CVE databases lag, so a squatted hallucinated package with no published advisory passes npm audit cleanly. Registry-side signals — publish date, download counts, repository linkage — are the tell, which is what the package hallucination scanner checks a project against.

Where VibeEval fits — and where it doesn’t

Honest positioning: VibeEval is a black-box scanner specialized for AI-built and vibe-coded apps. You give it a deployed URL; it runs the DAST-category checks tuned to the failure modes above — exposed secrets and env files in the shipped bundle, missing auth and BOLA-shaped access on API routes, CORS and header misconfigurations, Supabase/Firebase rule exposure, source map and debug-route leaks — and reports in plain language with fix prompts, in about a minute. It’s built for the person shipping with Cursor or Lovable who doesn’t have a security engineer to configure ZAP or triage Semgrep output. Between SAST and pentest describes this layer’s exact boundaries.

What VibeEval is not: it doesn’t read your source, so it won’t replace SAST for injection sinks in unreached code paths, a secrets hook for pre-commit protection, or a human pentester for business-logic and multi-step authorization flaws. The stack that actually covers an AI-built app is boringly layered: secrets hook at commit, tuned SAST plus dependency audit in CI, black-box scan per deploy, human review on the diffs (AI code review guide).

For single-symptom checks before committing to any tool, the free checkers each test one failure mode against a live URL: token leaks, Supabase RLS, CORS, source maps, env exposure, and JWT configuration.

Minimal viable toolchain

If you adopt nothing else, in order of value per minute of setup:

1. Secrets hook (gitleaks, pre-commit) — minutes to add, catches the most common critical
2. Built-in dependency audit (npm audit / pip-audit in CI) — already installed, plus a registry sanity check on any AI-suggested package
3. Black-box scan per deploy — Vibe Code Scanner, or the free checkers plus the manual self-audit if you’re doing it by hand
4. Tuned SAST in CI — Semgrep with a curated ruleset once the noise is worth managing; see SAST tools for AI code
5. Periodic human testing — a real pentest once the app has users and revenue worth protecting

Each layer covers a blind spot the others structurally cannot. No single tool — including ours — catches everything; anyone claiming otherwise is selling category confusion.

Related resources

• SAST Tools for AI Code — static analyzers ranked for AI-generated repos
• Security Testing Tools — the broader tooling overview
• Vulnerability Scanner Comparison — head-to-head scanner evaluation
• Between SAST and Pentest — the layer black-box scanning occupies
• AI Code Review Guide — the human-review half of the pipeline
• Vibe Coding Vulnerabilities — the failure modes these tools exist to catch