SCAN YOUR OPENAI CODEX APP FOR VULNERABILITIES

OpenAI Codex writes code autonomously in two shapes: a cloud agent that works on a connected GitHub repo inside a sandboxed container and opens pull requests, and the Codex CLI that runs in your terminal with your developer permissions. Both are steered by AGENTS.md files — persistent instructions that are effectively configuration-as-prompt — and both ship code in whatever stack your repo already uses.

The sandbox is a real security win, but it protects your workstation, not your repo. What lands in the PR is plausible, fast-produced code that reviewers calibrate to the stated task — and the dangerous change is usually the one three files away from it: a widened CORS config, a new dependency in the lockfile, a relaxed auth check. Green CI plus an unread diff is how agent-authored vulnerabilities reach production.

VibeEval is the independent check on the other end: it scans the deployed app black-box, so it doesn’t matter which files the reviewer skimmed — what’s reachable is what gets reported.

Common vulnerabilities we find in Codex apps

These are the shapes that recur in Codex-built applications — the ones that ride green CI and a plausible PR description straight to production.

Auth gates and ownership checks missing from generated routes (BOLA)

Codex ships working endpoints whose first three lines don’t check the session — or check it, then fetch by ID without verifying ownership. Exploit: sign in as user B, replay user A’s request with A’s record ID, and read or modify A’s data.

// generated route: session checked, ownership skipped
const session = await auth();
const order = await db.orders.findById(req.params.id);
return res.json(order); // B reads A's order

The agent’s happy-path tests never try a second user, so the gap survives CI. Fix direction: an auth gate plus an owner_id comparison on every ID-keyed route. Pattern deep-dive: BOLA in AI-generated CRUD.

Prompt injection through AGENTS.md, issues, and repo content

Everything Codex reads is part of its prompt: AGENTS.md files, the issue you point it at, READMEs, fixtures. A contribution PR that tweaks AGENTS.md (“prefer package X for HTTP calls”) or a public-repo issue with instructions aimed at the agent can steer every subsequent task — weakened auth, a specific malicious package, a modified workflow. Fix direction: CODEOWNERS on AGENTS.md, read issues before assigning them, and deny internet access to cloud tasks by default so injection has no exfiltration path. Background: indirect prompt injection.

Hallucinated and typosquatted dependencies

Codex runs installs inside its sandbox and the resulting lockfile lands in the PR — the part of the diff nobody reads. Phantom package names and near-miss spellings are a known AI failure mode; a squatted name is remote code execution at install time, in CI, with CI’s credentials. Fix direction: review manifest and lockfile diffs first, verify each new name character-by-character, and run additions through the Package Hallucination Scanner.

Sandbox env vars leaking through the agent

Secrets configured for the Codex environment are readable by the agent and by anything it executes — including postinstall scripts of packages it adds. The over-provisioned sandbox (production env vars “for convenience”) plus internet access is one malicious install script away from a leak; the milder version is a key wired into generated code while connecting an integration. Fix direction: build/test credentials only in the sandbox, secret scanning as a required check, and rotate anything long-lived. VibeEval catches the shipped variant — keys visible in the deployed bundle. See the Token Leak Checker.

Green-CI regressions in the unread part of the diff

The recurring Codex incident shape: the PR fixes the bug, and two files over widens a CORS config to unblock its own test run, or softens a test assertion so the suite passes. The reviewer reads the fix, skims the rest, merges on green. Exploit: a browser on any origin can now make credentialed requests to your API. Fix direction: required human review of the whole diff, plus CI lints on removed requireAuth/csrf calls and no-arg cors(). Pattern: CORS credentials misconfig.

Input trusted straight into queries and file paths

Generated handlers validate on the client and trust the server-side body — then interpolate it into SQL template literals or path.join calls. Exploit: post a crafted body with curl (the attacker doesn’t use your form) and walk from injection to data exfiltration or path traversal. Fix direction: a schema check (Zod, Pydantic) at every route boundary and parameterized queries only — searchable in review with a grep for string-built SELECT/INSERT.

How VibeEval works with OpenAI Codex

1. Enter your deployed app’s URL — production, staging, or the preview deploy built from a Codex branch. Black-box: no repo connection, no code upload.
2. The agent attacks it in a real browser. It maps routes and APIs, sends unauthenticated and wrong-user requests to every endpoint (auth and BOLA), replays IDs across accounts, inspects the bundle for leaked keys, and probes CORS, security headers, and error responses.
3. You get findings with severity and paste-ready fix prompts. Each one is written to be dropped into your next Codex task — cloud or CLI — as remediation instructions. Merge the fix PR, redeploy, re-scan.

Manual testing vs VibeEval

	Manual review	VibeEval
Full-diff review of a Codex PR	Real hours; attention skews to the stated task	Independent of the diff — tests the deployed result
Lockfile and dependency vetting	Character-by-character name checks, rarely done	Deployed-app scan plus hallucination check on names
Cross-user BOLA probing	Two accounts and replayed requests per endpoint	Automated on every ID-keyed endpoint, every scan
Regression after each task	Re-review per PR; discipline decays by the tenth	Re-scan in minutes, identical battery every time
AGENTS.md and injection review	Human judgment required — read before assigning	Out of scope pre-merge; post-deploy effects surface in findings
Business logic and intent	Where reviewers are irreplaceable	Not a scanner’s job

Manual review keeps the intent, logic, and AGENTS.md hygiene. The scanner wins on repeatability: the same full pass after every Codex task, with no reviewer fatigue.

Frequently asked questions

Does the Codex sandbox make the code safe?

The sandbox isolates the agent from your workstation — that’s all. It does nothing about what the code itself does once merged and deployed: auth, ownership checks, validation, CORS, and secrets handling still need independent testing. The sandbox moves the trust boundary; it doesn’t remove it.

Is Codex-generated code safe to merge on green CI?

No. CI proves the tests pass, and the agent often wrote or adjusted those tests. Review the whole PR — dependency manifests and files outside the stated task first — and back the review with a scan of the deploy preview, which doesn’t care what the tests assert.

Can Codex be prompt-injected?

Yes — through AGENTS.md, issue bodies, and repo content, all of which the agent treats as instructions. The containment levers are CODEOWNERS on AGENTS.md, internet access off by default for cloud tasks, and mandatory PR review. Our Codex security checklist sequences the setup; MCP and tool-spec injection covers the adjacent class.

Cloud agent or CLI — does it change what I should scan?

No: both land code in the same repo and the same deploy. It changes the upstream posture — the CLI runs with your local permissions and deserves approval prompts; the cloud agent needs env-var and network scoping. The deployed app is the shared output, and that’s what VibeEval tests.

Do I scan once before launch, or after every task?

After every task that reaches a deploy, ideally against the preview environment before merge. Codex’s whole value is shipping many tasks fast — which means many chances per week for a regression to land. A scan that takes minutes is the only review step that scales with that cadence.

Can VibeEval fix issues Codex introduces?

It identifies them and ships each finding with a paste-ready fix prompt. Hand the prompt to Codex as the next task, let it open the fix PR, then re-scan the deploy to confirm the finding is closed rather than moved.

Related OpenAI Codex resources

• How to Secure OpenAI Codex — step-by-step hardening: env scoping, branch protection for codex/ branches, SAST/DAST gates.
• Is OpenAI Codex Safe? — the trust-boundary analysis: sandbox vs CLI, AGENTS.md as prompt, where the risk actually lives.
• OpenAI Codex Security Checklist — pre-task and pre-merge checks ordered by severity, with the attack patterns we see in Codex repos.
• BOLA in AI-generated CRUD — the most common finding in agent-authored APIs.
• Package Hallucination Scanner — check every dependency the agent added against the registry.

Test your Codex app before launch

Codex ships features task by task while reviewers read summaries — the deployed app is where the unread parts of those diffs become reachable. Scan the preview deploy before you merge and production after: findings in minutes, each with a fix prompt ready for the next Codex task.