Sample alert How it works Coverage PENTEST METHODOLOGY DOCS PRICING FAQ MCP CONTACT LOG IN SCAN MY APP →

Security Regression Testing After Deployment

GUIDE / TESTING

Security regression testing means you write one automated test for every security flaw you fix, and you run all of those tests after each deploy. The test sends the same attack that found the flaw. If the attack works again, the build is broken.

Why do fixed security flaws come back?

A fixed flaw is not a closed flaw. The code that fixed it can change later, and nothing tells the team. This happens in four common ways.

  • A developer or an AI coding tool rewrites the file and removes the check.
  • A new endpoint copies an old pattern that did not have the check.
  • A migration adds a table without a database policy.
  • Someone changes a setting in a dashboard, and the deploy pipeline never sees the change.

Functional tests do not find these problems. A functional test asks, “Does the right user see the right page?” A security test asks, “Does the wrong user get refused?” You must write the second kind on purpose.

The OWASP Web Security Testing Guide treats authorization testing as a separate task. A regression test makes that task permanent.

What is the rule for a good security regression test?

Use one rule: every fixed finding gets one test, and the test must fail on the vulnerable version.

The second part is the important part. A test that passes on both the vulnerable and the fixed code proves nothing. Before you commit a test, run it against the vulnerable state one time. You can use an old commit, a staging copy, or the original finding evidence. Make sure the test fails. Then apply the fix and make sure the test passes.

A security test asserts a refusal. It checks for a 401, a 403, an empty result, or a present header. It does not check for success.

How do I turn a finding into an assertion?

A finding states what an attacker did and what the app returned. Use those two facts. The finding gives you the request. The expected result is the opposite of the response in the finding.

  1. Copy the exact request from the finding evidence.
  2. Remove anything that is not needed, such as cookies from the original tester.
  3. Replace real identifiers with the identifiers of your test accounts.
  4. Write the assertion for the safe result, not for the unsafe result.
  5. Run the test against the vulnerable version and make sure it fails.
  6. Run the test against the fixed version and make sure it passes.
  7. Put the finding ID in the test name.

The table below shows how common findings map to assertions.

Finding The attack The safe result to assert
BOLA on /api/orders/{id} User B requests an order of user A Status 403 or 404, and no order data in the body
Missing auth on an admin route Request with no credentials Status 401 or 403
Table without RLS Anonymous query to the database REST API Empty list or an error, never rows
Exposed API key Search the public JavaScript for the key pattern No match
Missing security header Read the response headers The header exists with the correct value
Mass assignment Send role: admin in a profile update The role does not change

Keep one request and one assertion for each test. A failure then shows which control broke.

What does a test look like for each finding type?

The examples below use placeholder names. Replace app.example.com, the tokens, and the IDs with your own values. Never put real tokens in the test file. Read them from environment variables.

How do I test for BOLA (broken object-level authorization)?

BOLA means that user B can read or change an object that belongs to user A. The test needs two accounts and one object that user A owns. See Test Broken Access Control for the full set of access control classes.

import os
import requests

BASE = os.environ["TARGET_URL"]
TOKEN_B = os.environ["TEST_USER_B_TOKEN"]
ORDER_OF_A = os.environ["TEST_USER_A_ORDER_ID"]

def test_finding_017_order_not_readable_by_other_user():
    r = requests.get(
        f"{BASE}/api/orders/{ORDER_OF_A}",
        headers={"Authorization": f"Bearer {TOKEN_B}"},
        timeout=15,
    )
    assert r.status_code in (403, 404)
    assert ORDER_OF_A not in r.text

Check the body as well as the status. Some apps return a 404 that still includes the data.

How do I test for missing authentication?

Send the request with no credentials. This is the shortest test, and curl is enough.

code=$(curl -s -o /dev/null -w "%{http_code}" "$TARGET_URL/api/admin/users")
if [ "$code" != "401" ] && [ "$code" != "403" ]; then
  echo "FAIL finding-021: admin route returned $code without credentials"
  exit 1
fi

A React route guard does not protect the API. Test the API directly, not only the page. See Admin Page Accessible Without Login for the pattern.

How do I test that RLS is enabled?

With Supabase, the public anon key is not a secret. Row Level Security is the control that limits what the anon key can read. The test uses the anon key and asks for rows. See Supabase RLS Checker for a free scan of the same issue.

import os
import requests

URL = os.environ["SUPABASE_URL"]
ANON = os.environ["SUPABASE_ANON_KEY"]

def test_finding_009_invoices_table_has_rls():
    r = requests.get(
        f"{URL}/rest/v1/invoices?select=id&limit=1",
        headers={"apikey": ANON, "Authorization": f"Bearer {ANON}"},
        timeout=15,
    )
    assert r.status_code in (200, 401, 403)
    if r.status_code == 200:
        assert r.json() == []

To avoid a false pass, put one known row in the table. Then assert that the anon request does not return that row.

How do I test that a key is not exposed?

Download the deployed JavaScript files and search for the key pattern. Search for the pattern, not the key, so that a rotated key is also caught.

import os
import re
import requests

BASE = os.environ["TARGET_URL"]
PATTERNS = [r"sk_live_[0-9a-zA-Z]{20,}", r"service_role", r"sk-[A-Za-z0-9]{32,}"]

def test_finding_004_no_secret_in_bundles():
    html = requests.get(BASE, timeout=15).text
    scripts = re.findall(r'src="([^"]+\.js)"', html)
    for path in scripts:
        url = path if path.startswith("http") else BASE + path
        body = requests.get(url, timeout=15).text
        for pattern in PATTERNS:
            assert not re.search(pattern, body), f"{pattern} found in {url}"

Adjust the patterns to the key types your app uses. See Token Leak Checker and Supabase Service Role Key for the key types that cause the most damage. Also check the source maps. If a source map is public, the original code is public. See Source Map Checker.

How do I test a security header?

def test_finding_012_csp_and_frame_protection():
    r = requests.get(BASE, timeout=15)
    assert "frame-ancestors" in r.headers.get("Content-Security-Policy", "")
    assert r.headers.get("Strict-Transport-Security")

Test for a specific directive, not only for the presence of a header. A policy of default-src * is present but gives no protection. The MDN guide to Content-Security-Policy lists the directives. See Security Headers Checker for a free check.

When do I need Playwright?

Use Playwright when the flaw only appears in a browser session, such as a cookie flag or a page that shows data of the wrong user. Do not use it for API checks. A browser test is slower and breaks more often.

from playwright.sync_api import sync_playwright

def test_finding_015_session_cookie_flags(target_url, user_a_login):
    with sync_playwright() as p:
        browser = p.chromium.launch()
        context = browser.new_context()
        page = context.new_page()
        page.goto(f"{target_url}/login")
        page.fill("input[name=email]", user_a_login["email"])
        page.fill("input[name=password]", user_a_login["password"])
        page.click("button[type=submit]")
        page.wait_for_url("**/dashboard")
        cookies = {c["name"]: c for c in context.cookies()}
        assert cookies["session"]["httpOnly"] is True
        assert cookies["session"]["secure"] is True
        browser.close()

Where should these tests run?

Run them after the deploy, against the live URL of the new version. A test that runs only before the deploy cannot find a changed dashboard setting, a missing environment variable, or a policy that the migration did not apply.

Stage What it can find What it cannot find
Before deploy (local code) Logic errors in handlers Configuration, database policy, headers from the CDN
After deploy, on a preview URL Most flaws in the new version Differences between preview and production
After deploy, on production The real state that users see Flaws that need writes on real data

Use this order for most teams.

  1. Run fast unit tests before the merge.
  2. Deploy to a preview or staging URL.
  3. Run the full regression suite against that URL.
  4. Promote to production only if every test passes.
  5. Run the read-only part of the suite against production after the promotion.

Step 5 finds cases where production differs from staging. Also run the read-only part every night to find drift outside the pipeline. For the CI gate pattern and the generic checks, see Continuous Security Smoke Testing. For the always-on version of this practice, see Security Monitoring.

Most CI systems mark the build as failed if the test command returns a non-zero exit code. See the CI/CD Security Guide for the pipeline side.

How do I handle test accounts and test data?

Follow these rules.

  1. Create at least two accounts in two different tenants. Name them clearly, for example sec-test-a@example.com and sec-test-b@example.com.
  2. Give each account the lowest role that the test needs. Add one admin account only for the tests of admin routes.
  3. Put one known object in each tenant. Create it one time and keep it.
  4. Store credentials in the secret store of your CI system. Never commit them.
  5. Prefer a login call at the start of the suite to long-lived tokens.
  6. Mark all test data with a fixed prefix so that you can find and delete it.

For multi-tenant apps, the same two-account fixture can test the data layer as well. See Test Multi-Tenant Isolation for caches, search indexes, and storage. For the role matrix, see SaaS Privilege Escalation.

Run tests that change data, such as a mass assignment test, only in staging. In production, run tests that read.

How do I keep the tests stable?

A test that fails for no reason teaches the team to ignore failures. Keep the suite free of false alarms.

  • Assert the refusal, not the exact message. Assert 401 or 403, not the text of the error. The text changes more often than the status.
  • Accept the valid set of results. A hidden object can return 403 or 404. Both are safe. Accept both.
  • Wait for the right event. In a browser test, wait for a URL or an element, not a fixed time.
  • Retry the network, not the assertion. If a request times out, you can try it again. If the assertion fails, do not retry. A retry on an assertion hides an intermittent security hole.
  • Keep each test independent. One test must not need the result of another test.
  • Name the test after the finding. test_finding_017_... tells the person on call where to look.

What do I do when a security regression test fails?

Treat a failure as a real finding until you prove that it is not. Use this procedure.

  1. Stop the promotion. If the build is already live, decide if you must roll back.
  2. Run the failing request by hand. Make sure the result is the same.
  3. If the request now returns a safe result, find out why the test failed. Common causes are an expired token, a deleted fixture, or a rate limit. Fix the test and write down the reason.
  4. If the request shows the flaw, find the change that caused it. Use the last passing build and git bisect or the deploy history.
  5. Fix the cause, not only the symptom. If an AI tool removed the check, find out why.
  6. Run the full suite again.
  7. If data was exposed on production, follow your incident process and check the logs for access.

Do not mark the test as skipped to make the build pass. A skipped security test is an open hole with a note on it. If the test is wrong, fix it in a separate change that someone else can review.

How do I get a test from a scanner finding?

A scanner finding already has the attack in it. A VibeEval finding includes the captured HTTP request and response, a description, and numbered remediation steps. See Findings for the fields. Use the request as the base of the test, and write the opposite assertion.

After you fix the issue, use Retest on the finding. Retest runs the single check that produced the finding again. Then write the permanent test in your own suite, so the check runs on every deploy and you do not rely on a manual action. If you want a message when new findings appear, see Webhooks.

The scanner finds new problems. Your regression tests guard against the return of old problems. For a deeper review on each release, see Continuous Penetration Testing.

What about code written by AI tools?

An AI coding tool that is asked to “clean up” a handler can remove an ownership check because it looks like extra code. The regression test does not depend on the tool noticing the check. For how to prove that a change is safe, see How to Verify AI-Generated Code Is Secure.

Common questions

What is security regression testing?
Security regression testing is the practice of keeping a test for every security flaw you fixed. The test runs again after each deploy. If the flaw comes back, the test fails and you find out before an attacker does.
How is a security regression test different from a security smoke test?
A smoke test is a short, generic list of checks for every app. A regression test is specific to your app. It exists because a real flaw was found in your code. The two work together. The smoke test finds common problems. The regression tests guard the problems you already fixed.
Should security regression tests run before or after deployment?
Run them after deployment, against the live URL. Many security flaws come from configuration, such as a missing database policy or a changed header. A test that runs only against local code cannot see these. Fast unit tests that run before deployment are also useful, but they are an extra layer.
Can I run these tests against production?
Yes, if the tests only read data and use dedicated test accounts. Do not write to real customer records. If a test must create data, mark it with a clear prefix and delete it at the end. Use a staging copy for tests that change data.
What do I do when a security regression test fails?
Treat it as a security incident of the same severity as the original finding. Stop the release or roll back. Find the change that caused the failure. Fix the cause, run the test again, and add a note to the finding.
How many regression tests do I need?
Write one test for each fixed finding. A small app can have 10 to 30 tests. Keep each test short and fast. If the suite takes more than a few minutes, split it so the most severe tests run first.

Retest your live app

VibeEval scans your deployed app and shows the HTTP request and response as evidence for each finding. Use Retest to run the single check again after you fix it.

14-day free trial · Cancel anytime

START FREE SCAN →