After Testing Every Major LLM, None Ship Validation That Survives the First Pass

If your AI-generated app passes Snyk, Semgrep, and the Claude Code or Codex review skill, you have proof that the code in your repo is reasonable. That is a small slice of what actually gets you breached. The breach lands at the integration layer: between your code and the libraries it pulls in, between your services and the network, between the model that wrote the code and the validation it forgot to add.

Code scanners answer “is this file safe in isolation.” Integration is where convenience always wins over security. That is where vibe-coded apps fail.

We keep saying this because the incidents keep rhyming: clean CI, open PostgREST; clean AST, unsigned webhooks; clean PR, preview deploy with production secrets. The file was fine. The system was not.

The scanner blind spot

A scanner walks your AST. It can see a missing CSRF check, a hardcoded key, an eval on user input. It cannot see:

• The Supabase service-role key your dashboard exposes through NEXT_PUBLIC_SUPABASE_KEY because the model didn’t know NEXT_PUBLIC_* ships to the client
• A wrangler.toml that binds your D1 database to a permissive role
• A staging subdomain spun up last Tuesday that nobody put behind auth
• An npm package that was clean two weeks ago and shipped a fresh postinstall script in v1.4.3
• A Stripe webhook handler that your code happens not to verify because the model summarized the Stripe docs and dropped the signature step
• CORS set to reflect any Origin in a host dashboard, not in source
• Firebase Storage still on test-mode rules while Firestore looks locked
• A preview URL that bypasses middleware because the matcher excludes it

None of these live in the file. They live between the file and the world. We see them every week.

The recommendation is not “stop using scanners.” Use them. Add the Cloud Code security skill, run the Codex security review, keep Snyk in CI. Do all of that. Then accept that the result is one corner of the picture.

Static analysis is necessary. It is not a substitute for “send a hostile request to the deployed URL and see what comes back.”

What we keep finding: LLM-generated validation needs 2-3 passes

There is one heuristic that has held across every model and harness we have tested over the last six months: Claude Sonnet 4.5, GPT-5, Gemini 2.5, Cursor’s tab model, Lovable’s harness, Bolt, Replit’s agent. None of them produce input or form validation that survives an honest red-team pass on the first iteration.

What you get on iteration 1: the obvious type check. email is a string. age is a number. The form has the right field names. Happy path tests go green.

What’s missing: empty string vs. null vs. undefined. Negative numbers where you assumed positive. Float where you wanted integer. NUL bytes in a filename. Path traversal sequences in filename. An authorization check that confirms the user is logged in but not that the user owns the row. File uploads that pass MIME sniffing but contain a polyglot. A 10-byte JSON parsed as a 10 MB request because the model didn’t add a body-size limit.

These all get caught on iteration 2 or 3 - but only if you ask. The default LLM workflow stops at iteration 1 because the obvious tests pass and the human moves on. The model is solving the immediate prompt (“build a signup form”) and the validation pass requires a second, explicit prompt. Without it, you don’t get it.

This is not a model failure. It is a context failure. The window is the budget; the model spends it on the feature; the edge cases need a separate spend.

Why iteration 1 always looks done

Training data is full of tutorial forms: required attributes, a regex for email, a success toast. It is thin on adversarial examples. Unit tests generated by the same model assert the happy path. Product managers accept “signup works.” Security is a second product requirement that never entered the first prompt.

The fix is process, not a better base model: force a second spend on validation the same way you force a second spend on accessibility or i18n when those matter.

Drop-in: a three-iteration validation prompt

Save this as a Claude Code Skill (~/.claude/skills/validation-loop/SKILL.md) or paste it into Codex / Cursor as a one-shot. Run it after every feature touch.

---
name: validation-loop
description: Three-iteration input and form validation hardening pass for any external entry point (HTTP handler, form action, server action, API route, webhook, CLI). Use after every feature ships.
---

You are reviewing code for input and form validation completeness.

For every external entry point in the touched files, run three iterations end-to-end. Do not stop after iteration 1.

ITERATION 1 - Surface check
List every parameter (body, query, path, header, cookie, file). For each, state:
- Type
- Source of trust (user, signed, server-side)
- Whether it is validated before use
- Whether it flows into a database query, file path, shell command, eval, redirect URL, or rendered HTML

ITERATION 2 - Edge case audit
For each parameter, answer:
- What if it is missing?
- Wrong type?
- Empty / "" / 0 / null / undefined / NaN?
- Larger than the expected max (10 MB body, 1 GB file)?
- Contains: NUL byte, Unicode escape, RTL override, path traversal (..), control character, SQL meta?
- File: zip-slip path, SVG with embedded script, polyglot, decompression bomb, MIME mismatch?
- Authorization: user A submits resource ID owned by user B?

ITERATION 3 - Hardening
For every gap surfaced in iteration 2:
- Add explicit validation (zod / pydantic / joi / manual). Reject with 4xx. Never coerce silently.
- Add a test that submits the malicious input and asserts the rejection.
- For authorization gaps, add the row-level check (`auth.uid() = user_id` for Supabase RLS, or the equivalent for your stack).

Output a per-endpoint checklist: the lines that need to change, the test to add, the validation library call.

Drop it in, run it after every feature touch. In our own engagements we have seen it close roughly nine of every ten validation gaps a scanner misses on a vibe-coded app. It catches what the codegen never asked itself about.

How teams actually adopt the loop

• Solo: paste after each feature; 10–20 minutes.
• Small team: make the skill a required checkbox on the PR template for any route change.
• Larger team: CI cannot run the LLM skill reliably; CI can run the tests the skill adds. The human/agent still runs iteration 2; CI enforces iteration 3’s tests.

Do not expect the skill to invent product policy (who may see which org). It will force the ownership check once you name the rule.

Composable risk: the layers above the code

Validation is one layer. The rest of the integration is composable, and every layer you stack inherits the others’ defaults.

Libraries. Your package.json is a trust delegation to several hundred maintainers, some of whom you have never heard of. A package clean today can ship a postinstall next week. There have been four notable npm supply-chain incidents in 2026 already; we covered the latest in the Apr 23 weekly digest. Running npm audit is necessary and insufficient. Pin versions, use a registry mirror, fail the build when a transitive maintainer changes hands. Package hallucination is the AI-specific cousin: models invent names attackers register first - use the Package Hallucination Scanner.

Infrastructure. The default for “publish a Next.js app on Vercel” is “every preview deployment is on the open internet.” The default for Supabase is “RLS is opt-in.” The default for an S3 bucket used to be public. Cloud defaults are the integration layer, and they are tuned for ergonomics first.

The CI/CD path. Your code is reviewed. Your secrets are not, until they leak. A poisoned GitHub Action with pull_request_target permission is a credential exfil waiting for a fork. Same risk shape we covered in AI Agents in GitHub Actions: Prompt Injection, one layer up.

Third-party OAuth. A user clicks “allow all access” on an AI productivity tool and you now have a SaaS-OAuth supply chain attached to your tenant. The Vercel / Context.ai breach is the textbook case: the code was secure, the OAuth scope was over-broad, environment variables not marked “sensitive” were read.

Webhooks and callbacks. Stripe, GitHub, Clerk, Resend - every “notify my app” URL is an unauthenticated entry point until you verify signatures. Models drop verification because demos use the Stripe CLI with trusted local traffic. Production is not local.

Identity between services. Service-role keys, cloud function admin SDKs, and “internal” APIs without mTLS or signed tokens are integration auth. AI apps collapse them into one env var used from both client and server.

Every one of these is invisible to the AST.

Concrete integration failures we keep filing

None of these require a sophisticated exploit chain. They require treating the deployed system as the unit of review.

Where gapbench fits

We run a public benchmark at gapbench.vibe-eval.com - a deliberately broken set of scenarios that mirror what AI codegen ships, with a clean reference site (ref0) for false-positive calibration. The goal is to make these integration gaps reproducible at a URL anyone can hit.

Want to see what an exposed Supabase service-role key looks like at runtime, not in code? gapbench.vibe-eval.com/site/supabase-clone/. Want to see a Stripe webhook handler that ships without signature verification, deployed and live? gapbench.vibe-eval.com/site/webhook-unverified/. Same for naked Postgres, BOLA across CRUD, JWT alg=none, MCP servers with shell access, and forty-odd other patterns we keep finding.

Heuristic scanners can flag “missing call to verifyWebhookSignature” if they recognize the pattern. They cannot tell you that your deployed webhook handler actually accepts unsigned requests. The only way to know is to send one.

That is the meta layer. Scanners give you confidence about a slice of the picture. End-to-end testing - runtime, against the deployed app, with adversarial input - is the only test that catches the integration. We do it because the slice is not enough.

If you build internal tools, clone the same idea: a staging site that intentionally includes last quarter’s mistakes as regression fixtures. Green scans against only happy-path staging teach the wrong lesson.

What “integration testing” means in practice

You do not need a full red team. You need a short hostile script against every deploy:

1. Fetch the public HTML/JS; search for keys and project URLs.
2. Hit unauthenticated APIs and databases the client can reach (PostgREST, Firebase, open buckets).
3. Authenticate as user A; request user B’s IDs.
4. POST forged webhooks without valid signatures.
5. Submit oversized / malformed bodies to every new form route.
6. Confirm security headers on HTML and API responses.

Automate what you can (Vibe Code Scanner, Supabase RLS Checker, Token Leak Checker). Keep a human for product-specific authorization rules.

Bottom line

The good news: AI codegen is not making your code less secure than a junior engineer’s first commit. The bad news: it is producing more code, faster, with the same blind spot in the same place - between the file and the world.

Three things we recommend, in order of cost and impact:

1. Run the validation loop after every feature. Free. Closes about 90 percent of the input gaps a scanner cannot see.
2. Treat integration as the test surface. Run an end-to-end scan against the deployed app, not just the repo. We do this; a handful of competitors do too. Pick one and run it weekly.
3. Audit the composable layer monthly. Lockfile diff, registry maintainer changes, OAuth scope review, infrastructure default check. Boring. Catches the supply-chain-shaped breaches.

We are not arguing scanners are useless. We use them. The argument is that “code clean” is one quarter of “secure.” The other three quarters are the integration layer, and AI codegen has not made that layer better. Only bigger.

If you only change one habit after reading this: stop treating a green Semgrep run as a ship decision. Treat a green deployed scan - keys, RLS, BOLA, webhooks - as the ship decision, and keep Semgrep as the merge decision.

Mapping the integration surface of a typical AI SaaS

Browser bundle → Edge middleware/CDN → Serverless handlers
  → BaaS (RLS/rules) + storage → Stripe/Resend/OpenAI/OAuth → CI/CD secrets

A green Semgrep run does not prove host CORS, Storage rules, or webhook signatures.

Integration inventory

Unverified rows older than 90 days are assumed broken.

Why models drop integration checks

Tutorials use Stripe CLI trust, “enable RLS later,” .env.local without NEXT_PUBLIC_ hazards, and cors: * for localhost. Force a second prompt:

Review integrations in this PR:
1. Webhooks: raw body + signature + idempotency
2. OAuth: exact redirect allowlist; minimal scopes
3. Storage: auth + size/type; no public private objects
4. Secrets: no public prefixes; not in bundles
5. Preview vs prod: separate projects/keys
Add tests for bad signatures and wrong-user calls.

Hostile script against every deploy

URL="${1:?url}"
curl -sL "$URL" -o /tmp/p.html
grep -oE 'eyJ[a-zA-Z0-9_-]+\.[a-zA-Z0-9_-]+\.[a-zA-Z0-9_-]+' /tmp/p.html | head
grep -oE 'sk_(live|test)_[A-Za-z0-9]+' /tmp/p.html | head
grep -oE 'https://[a-z0-9-]+\.supabase\.co' /tmp/p.html | head
curl -sI "$URL" | grep -iE 'strict-transport|content-security|access-control'
curl -s -o /dev/null -w '%{http_code}\n' -X POST "$URL/api/webhooks/stripe" \
  -H 'content-type: application/json' \
  -d '{"type":"checkout.session.completed"}'

Then two-user BOLA and anonymous Storage list. Wire Vibe Code Scanner, Token Leak Checker, Supabase RLS Checker.

Preview environments as integration failures

Previews often copy prod service_role, live Stripe keys, weak middleware matchers, and wildcard OAuth redirects. Rule: branched DBs or seeds, test-mode payments, Deployment Protection. Synthetic fixtures only - never prod snapshots on public preview URLs.

OAuth third-party risk

Inventory OAuth apps quarterly; remove unused; prefer least scopes; rotate env after industry OAuth incidents; treat deploy bots as roots. Scanners never see the consent screen.

CI/CD as privileged integration

# Dangerous - do not copy
on: pull_request_target
permissions: write-all

Pin actions by SHA, least-privilege permissions, secrets only on trusted events, OIDC over long-lived keys. “Fix flaky deploy” by widening permissions is a durable backdoor. See CI/CD security guide.

Monthly composable-risk cadence

Week 1 lockfile/maintainers/hallucinations. Week 2 OAuth scopes. Week 3 preview vs prod secrets. Week 4 webhook tests + egress. Thirty calendar minutes beats annual panic.

Write findings as business risk

Not “missing constructEvent.” Yes: “Anyone can POST a fake paid event and get Pro enabled on the live webhook URL.” Keep curl evidence against the deployed URL.

Acceptance criteria for generators

Done means: webhook signatures verified; no service_role/sk_live in bundles;
RLS + dual-user tests for new tables; preview non-prod secrets; rate limits on auth and AI routes.

Operational verification for updates/integration-layer-is-the-real-security-gap.md (pass 1)

For this page’s subject matter, treat authentication, authorization, secrets, webhooks, and data-plane rules as ship gates. After every AI-assisted change:

1. Dual-user tests (owner vs stranger) on every new resource ID
2. Anonymous probes of new routes and storage paths
3. Bundle/token leak check for public env prefixes and service roles
4. Live URL scan on preview and production
5. Webhook signature failure tests where payments or entitlements are involved

Encode proofs in CI when possible. Temporary exceptions need tickets with expiry dates. Agent shortcuts that open policies, public secret prefixes, or skip signatures are release blockers until closed.

# Proof sketch tailored to updates/integration-layer-is-the-real-security-gap.md
curl -s -o /dev/null -w '%{http_code}\n' "$URL/api/protected"
curl -s -H "Authorization: Bearer $TOKEN_B" "$URL/api/resources/$ID_A" -w '\n%{http_code}\n'

Regression signals specific to this surface

• Permission-denied metrics drop after a rules or policy deploy without a matching product change
• 401/403 spikes on admin or object routes without a launch
• New packages merged without registry age and maintainer checks
• Preview environments inheriting production secrets or service roles
• Unsigned webhook payloads accepted in staging clones of production

Ownership

Name a human owner for this control surface. Keep CODEOWNERS on auth, payments, policy files, and CI workflows. Rehearse key rotation for the primary secret class this stack uses. Record the last green dual-user pass date in the release ticket.

Operational verification for updates/integration-layer-is-the-real-security-gap.md (pass 2)

For this page’s subject matter, treat authentication, authorization, secrets, webhooks, and data-plane rules as ship gates. After every AI-assisted change:

1. Dual-user tests (owner vs stranger) on every new resource ID
2. Anonymous probes of new routes and storage paths
3. Bundle/token leak check for public env prefixes and service roles
4. Live URL scan on preview and production
5. Webhook signature failure tests where payments or entitlements are involved

Encode proofs in CI when possible. Temporary exceptions need tickets with expiry dates. Agent shortcuts that open policies, public secret prefixes, or skip signatures are release blockers until closed.

# Proof sketch tailored to updates/integration-layer-is-the-real-security-gap.md
curl -s -o /dev/null -w '%{http_code}\n' "$URL/api/protected"
curl -s -H "Authorization: Bearer $TOKEN_B" "$URL/api/resources/$ID_A" -w '\n%{http_code}\n'

Regression signals specific to this surface

• Permission-denied metrics drop after a rules or policy deploy without a matching product change
• 401/403 spikes on admin or object routes without a launch
• New packages merged without registry age and maintainer checks
• Preview environments inheriting production secrets or service roles
• Unsigned webhook payloads accepted in staging clones of production

Ownership

Name a human owner for this control surface. Keep CODEOWNERS on auth, payments, policy files, and CI workflows. Rehearse key rotation for the primary secret class this stack uses. Record the last green dual-user pass date in the release ticket.

Related reading

• Your CLAUDE.md Is Attack Surface - the skill files you load are also code
• Vercel Breach via Context.ai - third-party OAuth as integration risk
• Vibe Coding Security Weekly - Apr 28, 2026 - SecureVibeBench measured the best AI agents ship correct-and-secure code 23.8% of the time
• Lovable BOLA Vulnerability - authorization gap that no static scanner found
• Between SAST and pentest - where runtime testing sits in the stack
• SAST tools for AI code - what static analysis still does well