PROOF OVER ALERTS: REAL EXPLOITABLE BUGS, NOT NOISY ALERTS

Every security tool ships with the same implicit sales pitch: look how much we found. The findings count is on the dashboard, in the PDF export, in the renewal deck. It is also the wrong number. An alert is a claim. A claim without evidence costs the reader more than it costs the writer — and the entire alert-fatigue crisis is that asymmetry, compounded across an industry that ships claims at machine speed and leaves verification to humans.

Our position: security tools should be judged on proof, not volume.

Proof means a reproducible demonstration — a URL you can hit, a request you can replay, a detection calibrated against known-good code so its silence means something.

Everything else is noise with a severity label.

The noise numbers are worse than the folklore

The folklore says “scanners are noisy.” The measured reality is starker.

Ghost Security tested traditional SAST tools against nearly 3,000 open-source repositories in Go, Python, and PHP. Of 2,116 flagged issues, 180 were confirmed real — a false positive rate of more than 91%. For Python/Flask, 1,166 potential issues were flagged and six were real. Command injection checks were the worst offender at 99.5% false positives (Ghost Security, 2025, via Help Net Security). Ghost is a commercial AppSec vendor with an interest in SAST looking bad, and the study covered three languages, so treat the exact figure with care. The direction is not in dispute.

OX Security analyzed 101+ million application security findings across 178 organizations over 90 days and concluded that only 2-5% of alerts require immediate action. The average organization faced 569,354 alerts, with roughly 202 truly critical issues buried inside (OX Security, 2025 Application Security Benchmark). Again vendor telemetry, again a self-selected customer base — and again the same shape.

Even “critical” is mostly noise. Datadog’s State of DevSecOps 2025 found that just 18% of vulnerabilities with a critical CVSS score remain critical after applying runtime context — production status, internet exposure, exploit availability (Datadog, 2025). Four out of five criticals are downgraded the moment someone asks “can this actually be reached?”

The human cost is documented too. In Cycode’s State of ASPM survey (500 U.S. CISOs, AppSec directors, and DevSecOps team members), 85% of the CISOs said their dev teams suffer from vulnerability noise and alert fatigue, and 88% said that because of it, developers are not focused on remediating the vulnerabilities that are actually critical (Cycode, 2024 report, via Help Net Security). Vendor-commissioned, self-reported — and consistent with everything above.

Every one of these studies comes from a vendor with something to sell. We are a vendor with something to sell. That is exactly why the standard has to be reproducible proof rather than published percentages: you should not have to trust any of us.

An alert without proof is an unfalsifiable claim

Here is the epistemics of it. When a scanner says “possible SQL injection at line 214,” it is making a claim you cannot cheaply check. Verifying it means reading the code, tracing the dataflow, constructing an input, testing it. Dismissing it means risking that this was the real one. The tool has transferred the entire cost of verification to you, at zero cost to itself.

A tool that pays no cost for being wrong will be wrong often. That is not cynicism; it is incentives. Recall looks great in a demo. Precision is only measured later, in your triage queue, after the contract is signed. So vendors optimize the number that sells and externalize the number that hurts — and developers, who are not stupid, learn the base rate. If 9 out of 10 alerts are noise, ignoring alerts is the rational policy, and the practitioner data above says developers have adopted exactly that policy. The tragedy of alert fatigue is not that developers are careless. It is that they did the math.

The market has started doing the math too. In Cobalt’s State of Pentesting 2026 survey, organizations relying entirely on fully automated AI vulnerability scanning fell from 29% to 9% in a year, and 78% of respondents said fully automated tools had missed critical vulnerabilities (Cobalt, 2026, via Infosecurity Magazine — a pentest vendor’s survey, so read the framing accordingly). Trust in unproven automation is collapsing, and it should.

The way out is not fewer rules or smarter dashboards. It is falsifiability. A finding should arrive in a form where it can be shown wrong. “Possible injection” cannot be shown wrong. “Send this request to this endpoint and you get another user’s rows back” can — and when it survives the check, it gets fixed the same day.

What proof actually looks like

Three concrete forms, in ascending order of rigor:

• Reproduce-at-a-URL — the finding names a deployed surface and the exact interaction that demonstrates the bug.
• Exploitability evidence — the finding shows consequence, not pattern.
• Falsifiable calibration — the tool’s silence is measured against known-good code.

Reproduce-at-a-URL. The finding names a deployed surface and the exact interaction that demonstrates the bug. Not “the code appears to lack webhook signature verification” but “your live webhook endpoint accepted this unsigned request; here is the request and the 200 response.” A static scanner can flag the missing verifyWebhookSignature call; only sending an unsigned request proves the deployed handler accepts it. We wrote about this gap at length in the integration layer essay — the vulnerability lives at the URL, not in the repo.

Exploitability evidence. The finding shows consequence, not pattern. A service-role key in a bundle is a pattern; the key decoded to role: service_role, demonstrating it bypasses row-level security, is evidence. A missing ownership check is a pattern; user A retrieving user B’s record by incrementing an ID is evidence — the shape of the Lovable BOLA case, which no static pattern-match found and one authenticated request proved.

Falsifiable calibration. This is the one almost nobody does. Any individual finding can be a lucky hit. The systemic question is: does the tool’s silence mean anything? That requires known-good code — labeled clean surfaces the tool has been demonstrated not to fire on. Without clean controls, a vendor’s false-positive rate is not a measurement, it is a press release. With them, it is a number anyone can recompute.

How we make “real exploitable bugs, not noisy alerts” measurable

“Finds real exploitable bugs, not noisy alerts” is the sentence every scanner vendor wants on its homepage, ours included. The difference we can offer is that for us it is a measurable property with a public test set, not a slogan.

The test set is gapbench: 104 deployed scenarios on the public internet. 97 are deliberately vulnerable, each tagged to the CWEs it represents — an exposed Supabase service-role key, a Stripe webhook handler that accepts unsigned requests, BOLA across AI-generated CRUD, JWT alg=none, and so on. 5 are clean reference controls — ref0 plus the topic-specific ref-rls, ref-jwt, ref-oauth, and ref-webhook, each the same shape as its vulnerable counterpart but configured correctly. 2 are calibration targets for benign-looking traffic.

The shipping rule is mechanical: every detection must fire on its known-vulnerable scenario and stay silent on ref0 and the matching clean control. If a rule flags ref-rls — “RLS done right” — it has a false-positive case on precisely the surface where it is most tempting to over-trigger, and it gets tightened or killed before a customer ever sees it. The full loop is documented in the ref0 methodology piece and the reasoning behind the benchmark in why we built gapbench.

Honesty about the limits, because the whole thesis collapses without it: gapbench is biased toward AI-codegen-shaped stacks — Supabase, Next.js, Vite, Express. It has thin coverage of mobile, real-time, and long multi-step business-logic flows. Passing gapbench does not certify a scanner for a 2015 Java monolith. What it certifies is narrower and more useful: on the surfaces it covers, this tool’s true positives and false positives are reproducible by anyone with a browser. That is a claim we can be wrong about in public, which is the only kind of claim worth making.

A five-minute test for any scanner vendor

If you are evaluating a security scanner — ours or anyone else’s — here is a test that takes minutes and requires no vendor cooperation.

1. Ask for the ground truth. Where does their accuracy number come from? “Customer feedback” and “our internal corpus” mean unlabeled data — unverifiable by construction. The acceptable answer names a labeled test set you can inspect.
2. Ask for the clean controls. What known-good code has this tool been shown not to fire on? If the vendor has never measured against clean surfaces, their false-positive rate is unknown to them, not just to you.
3. Run the two-sided check. Point the tool at supabase-clone and at ref-rls. Fires on the first, silent on the second: the detection is earned. Fires on both: a false positive on a labeled clean surface, in front of you, during the eval. Silent on both: a recall gap. Every outcome is informative, which is what a real test looks like.
4. Inspect one finding end-to-end. Take any single finding from a scan of your own app and ask: can I reproduce this in under ten minutes from what the tool gave me? If the answer is no for the finding they chose to show you, it will be no for the other five hundred.
5. Apply it to us. Run VibeEval against the same URLs. We publish the manifest precisely so we can be caught out if the claim does not hold.

What you should take away

An alert is a request for your time. Proof is a completed argument. The industry’s own numbers — 91%+ SAST false positives, 2-5% of alerts actionable, 18% of criticals surviving context — describe what happens when tools are rewarded for the volume of requests rather than the strength of arguments. The fix is not triage automation layered on top of noise. It is refusing to ship, or buy, claims that cannot be checked. Demand a URL, a request, and a clean control. Vendors who have the proof will show it. Vendors who talk about their proprietary risk score instead have just answered the question.

Related reading

• Pattern: Why we built gapbench
• Pattern: False positives and the ref0 control
• Pattern: The Supabase service-role key in your frontend bundle
• Pattern: BOLA in AI-generated CRUD
• Pattern: Stripe webhooks and paid trust
• Update: The integration layer is the real security gap
• Update: Lovable BOLA vulnerability