WHY AI-GENERATED CODE FAILS: A RISK ANALYSIS

Two pages, two questions

This page and the AI Code Vulnerability Taxonomy split one topic deliberately. The taxonomy catalogs what ships: the vulnerability families, their code shapes, and the fix for each. This page explains why those families keep shipping — five structural causes that hold regardless of which assistant you use. Understanding the causes matters because it tells you which mitigations actually work and which are theater. If insecure output were random noise, better prompting would fix it. It is not random; it is systematic, and each cause below predicts a specific slice of the taxonomy.

Training-data bias: models learned from teaching code

Code models are trained substantially on public code: tutorials, documentation examples, Stack Overflow answers, and open-source repositories. That corpus is skewed in a security-relevant way. Teaching code strips out validation, error handling, and authorization on purpose — the author is demonstrating a concept, and the ownership check would obscure it. Accepted Stack Overflow answers are optimized for “solves the asker’s error,” not for production hardening. So the statistically typical GET /resource/:id handler in the training distribution authenticates but never authorizes, the typical cookie has no flags, and the typical XML parser uses library defaults.

The model reproduces the distribution it learned. This is why the taxonomy’s authorization family is so dominant: authentication appears in training data constantly (every tutorial has a login flow) while per-resource ownership checks appear rarely (they are app-specific and boring). It is also why generated code reproduces insecure library defaults — external entities enabled, permissive CORS in dev examples — because the defaults are what example code shows. The empirical grounding here is Pearce et al.’s 2021 “Asleep at the Keyboard” study, which found roughly 40 percent of Copilot-generated programs vulnerable in security-relevant scenarios the authors constructed. The precise number matters less than the mechanism it demonstrated: in scenarios adjacent to known weakness classes, the model completes the code the way the corpus usually completes it — insecurely.

Plausibility over correctness

A language model optimizes for producing likely text, and its output is refined by feedback that rewards code which runs and looks right. Security is nearly invisible to both signals. A handler with a SQL injection returns the same JSON as a parameterized one for every well-formed input; a checkout that trusts client-side prices works perfectly in every demo. Vulnerable code and secure code are behaviorally identical until someone sends the input the developer never sends. There is no compile error, no failing test, no visible defect for the generation loop or the developer to react to.

This asymmetry has a second-order effect: generated code looks professional. It has consistent style, plausible comments, confident naming — the surface signals reviewers use as proxies for quality. Occasionally the plausibility machine produces something with no referent at all: a call to a security helper that does not exist in the imported library, or a dependency name that was never published. Those hallucinated packages are a taxonomy family of their own, because attackers can register the hallucinated names (package hallucination scanner). The general rule: a model is a plausibility engine, and security is precisely the property for which plausible and correct diverge most.

Context-window truncation

An assistant only reasons over what fits in its context. Real codebases centralize security: the auth middleware lives in one file, the tenancy filter in another, the validation schema convention in a third. When the model generates a new endpoint without those files in context, it does not apply your conventions — it regenerates the statistically typical version, which skips them. This is why security quality degrades as projects grow: early on, the whole app fits in context and generated code stays consistent; later, each generation sees a fragment and quietly forks the security model.

Agentic and multi-file modes sharpen the problem. An agent partway through a long task summarizes or drops earlier context, so a constraint stated at the start (“all queries must be tenant-scoped”) silently falls out of scope by file twelve. Retrieval helps but is keyed to what the current edit mentions, and security context is exactly what unrelated feature code never mentions. The mitigation is structural, not prompt-level: enforce conventions in code (middleware applied by default, deny-by-default route guards, lint rules) so an endpoint that forgets the check fails loudly instead of shipping. See Secure AI Coding Practices for the concrete setup.

“Make it work” prompting pressure

Prompts specify functionality; almost nobody specifies security. “Add a form that saves the user’s profile” fully describes the feature and says nothing about field allowlists, so the model spreads req.body into the update — the mass-assignment shape from the taxonomy. Worse, the iterative loop actively selects for insecurity. When something errors, the developer pastes the error and asks the model to fix it, and the most statistically common “fix” for a CORS error is origin: "*", for an RLS permission error is disabling RLS, for a webhook signature failure is removing the verification. Each of these makes the error disappear, which reads as success, which ends the loop.

Vibe-coding compounds this: the person prompting may never look at the diff at all, so the disabled check is not a reviewed trade-off — it is invisible. The counter-move is to put security into the specification layer rather than each prompt: system prompts and rules files that state standing constraints, and fix-prompts that name the secure resolution explicitly (“fix the CORS error by adding the origin to the allowlist, do not use a wildcard”). The Cursor security risks page pairs each recurring pattern with exactly this kind of fix prompt.

Review-gap economics

The final cause is arithmetic. AI assistance multiplies the rate of code production; it does nothing to the rate of careful human review. If generation gets ten times faster and review capacity is flat, either review becomes the bottleneck (and the speed advantage evaporates) or review gets shallower. Teams almost always choose shallower, and often without deciding to: multi-file agent diffs get skimmed, “LGTM” latency becomes a metric, and the professional-looking surface of generated code (see plausibility, above) invites exactly that skim. The vulnerability classes that survive skimming are the ones review was supposed to catch — missing authorization, trust-boundary mistakes, logic flaws — which static tooling is weakest at.

There is also a knowledge asymmetry: the person shipping AI-generated code increasingly did not write, and could not have written, the code being reviewed. Review then degrades from “is this correct and safe” to “does this look reasonable,” which a plausibility engine passes by construction. The economic fix is to move the expensive check to a cheap automated gate: security scanning on every deploy catches the recurring taxonomy families mechanically, reserving scarce human attention for logic and trust-boundary review where it is irreplaceable. See SAST tools for AI code and between SAST and pentest for where each layer’s coverage ends.

What the causal model implies

Each cause maps to a mitigation class, and mismatched mitigations fail predictably:

• Training-data bias is not fixable by prompting alone — the model’s priors reassert themselves. Counter with codified conventions and automated checks that catch the statistically typical omission (missing authz, default configs).
• Plausibility over correctness means “it works” is zero evidence of security. Counter with adversarial testing — scanning and abuse-case tests that send the inputs demos never send.
• Context truncation means security must live in enforced structure (default middleware, deny-by-default), not in conventions the model has to remember.
• Prompting pressure means standing security constraints belong in rules files and system prompts, and error-driven “fixes” that disable a control deserve automatic scrutiny.
• Review economics means the gate goes in CI, on every deploy, so human review can concentrate on what automation cannot judge. The Vibe Code Scanner exists for exactly this slot.

None of this argues against AI-assisted development. It argues that the risk is structural rather than incidental — so the controls must be structural too.

Related resources

• AI Code Vulnerability Taxonomy — the companion catalog of what these causes produce
• Vibe Coding Vulnerabilities — the same risks in prompt-only development
• OWASP Top 10 for AI Code — mapping to OWASP categories
• Secure AI Coding Practices — the prevention playbook
• Copilot Security Risks and Cursor Security Risks — per-tool manifestations
• Security Testing Tools — building the automated gate