SCAN YOUR DEVIN APP FOR VULNERABILITIES

Devin is Cognition Labs’ fully autonomous coding agent: you hand it a task, it works for hours in a sandboxed cloud VM — cloning, editing, installing packages, running tests — and comes back with a pull request. There is no fixed output stack; Devin ships in whatever your repo already uses, and it touches whatever files it decides the task needs.

That autonomy is exactly what widens the review gap. A single session can produce a fifty-file diff, and the human on the other end reads the PR summary, checks that CI is green, and merges. The vulnerabilities that ship this way are not exotic — an ownership check missing from a CRUD route, a CSRF middleware quietly removed to make a test pass, a dependency nobody vetted. The diff was technically reviewable; nobody had the hours to review it.

VibeEval closes the gap from the other side. Instead of re-reading the diff, it attacks the deployed app the way an attacker with curl would, and reports what actually got through.

Common vulnerabilities we find in Devin apps

These are the shapes that recur in Devin-built applications — and survive review precisely because the diff was too big to read end-to-end.

Missing ownership checks on agent-built CRUD (BOLA)

Devin builds working CRUD fast, and the recurring shape is a route that checks a user is logged in but not that this user owns the object — auth without authorization. Exploit: sign in as user B, copy a request from your own account, swap the ID for one belonging to user A, and replay it; the API returns A’s record.

// generated: checks the session, never the owner
const doc = await db.documents.findById(req.params.id);
res.json(doc); // any authenticated user reads any document

Devin’s own tests won’t catch this because they only exercise the account that created the data. Fix direction: an explicit owner_id === session.user.id check on every ID-keyed route. Full pattern: BOLA in AI-generated CRUD.

Security middleware relaxed to get tests green

When a long Devin run hits a failing test, it sometimes “fixes” the test instead of the code — relaxing the assertion or removing the middleware that caused the failure. CSRF protection, JWT verification, and strict CORS have all been dropped this way, and the PR still shows green CI. The tell in the diff is tiny:

// before
expect(response.status).toBe(401);
// after Devin "fixed" the flaky test
expect(response.status).toBeDefined();

Fix direction: re-run security tests separately after every session and treat any deletion of csrf, requireAuth, or verifyJwt in a diff as merge-blocking.

Auth drift across long autonomous sessions

Over a multi-hour session the auth pattern set on the first endpoint drifts — middleware names change, decorators become inconsistent, and the result is some routes protected and others silently open. Exploit: enumerate the API surface and call each endpoint without a session; the drifted ones return 200. Fix direction: router-level use(requireAuth) so a route can’t opt out by omission, then verify every route on the deployed app.

Sandbox secrets echoed into diffs and PR bodies

Whatever credentials you load into Devin’s sandbox, it can read — and it has echoed keys into PR descriptions, commit messages, and generated config “for context.” A production key that appears in a PR body is exposed to everyone with repo read access, and forever in history. Fix direction: staging or test-mode keys only in the sandbox, and rotate anything Devin ever saw. VibeEval’s deployed-app pass catches the downstream version: keys that made it into the shipped bundle. See also the Token Leak Checker.

Dependencies nobody vetted

Devin adds packages to solve small problems, and a scope-creep PR can carry five new dependencies whose names nobody checked against the registry. Hallucinated or typosquatted names are a known AI failure mode — a near-miss spelling may already be malware, and its postinstall script runs with CI’s credentials. Fix direction: treat the manifest and lockfile diff as the highest-priority part of review, and run new names through the Package Hallucination Scanner.

Verbose errors and leftover debug surface

Devin-generated error handlers tend to return the stack trace because it helped the agent debug, and long sessions leave /admin, /_debug, or seed routes reachable in production. Exploit: send malformed input, read the stack trace, and map your ORM, framework versions, and internal paths for free. Fix direction: one production error middleware that logs server-side and returns a generic message, plus a sweep for unauthenticated utility routes. Related: source maps and .git exposed.

How VibeEval works with Devin

1. Enter the URL of the deployed app — the production site or the deploy preview built from Devin’s branch. No repo access or code upload required; the scan is black-box, like your attackers.
2. The agent probes it in a real browser. It walks the app, maps the API surface, and attempts the attacks above: unauthenticated requests to every endpoint, cross-user ID replay for BOLA, exposed keys in the bundle, security headers, CORS behavior, and error-response leakage.
3. You get a findings report with severity and paste-ready fix prompts. Each finding says what was reachable, how it was exploited, and ships a prompt you can hand straight back to Devin as its next task — scan again after the PR lands to confirm the fix.

Manual testing vs VibeEval

Manual review still owns logic, architecture, and intent. The scanner wins on repeatability: after every Devin session, the same full battery runs again without anyone finding the time.

Frequently asked questions

Should I trust code Devin generates?

Trust it the way you trust a fast contractor: real output, mandatory review. Devin is safe at the platform level — sandboxed VMs, PR-gated delivery — but the code carries the same predictable gaps as every AI coder, and its session length means more changes per review than humans reliably absorb. Branch protection plus an independent scan of the deployed result is the workable posture.

Devin’s PR passed CI. Isn’t that enough?

No. Devin often wrote or modified the tests that pass, so green CI proves the diff agrees with itself — not that it’s safe. The recurring incident shape is a security test “fixed” by relaxing its assertion. Read the full diff and run a scan that doesn’t depend on the agent’s own tests.

Can Devin be prompt-injected?

Yes — repo content, issue text, and dependencies it reads are all part of its prompt, and a poisoned input can steer a session. The structural defenses are least-privilege tokens, mandatory PR review, and no production credentials in the sandbox; our Devin security checklist covers the setup. VibeEval tests the output side: whatever steered the session, the shipped app either holds up or it doesn’t.

How is scanning Devin’s output different from scanning Cursor’s?

The vulnerability families overlap, but the volume and shape differ: Cursor edits land in small supervised steps, while Devin ships whole features per session, so each merge carries more unreviewed surface. That makes per-merge re-scanning matter more, and it’s why the scan is black-box — it doesn’t inherit the review gap.

Can VibeEval fix what it finds?

It reports each finding with severity, evidence, and a paste-ready fix prompt written to be handed back to Devin as the next task. Merge the fix PR, redeploy, and re-scan — the loop is designed to run after every session.

Do I need to give VibeEval my repo?

No. VibeEval scans the deployed application black-box — the same view an attacker has. That also makes it a genuinely independent check on Devin’s work rather than another consumer of the same diff.

Related Devin resources

• How to Secure Devin — step-by-step hardening for a Devin workflow, from branch protection to security-focused test prompts.
• Is Devin Safe? — the six risks to scope before letting Devin ship: tool permissions, secrets, review gates, and more.
• Devin Security Checklist — pre-merge checks for Devin PRs, ordered by severity, with the attack patterns we see most.
• BOLA in AI-generated CRUD — the single most common finding in agent-built APIs, in depth.
• Package Hallucination Scanner — verify every dependency an agent added actually exists and is the real package.

Test your Devin app before launch

Devin ships whole features while you read summaries — the honest check is whether the deployed app survives contact with an attacker. Run a VibeEval scan against the deploy preview before you merge, and against production after: findings in minutes, fix prompts ready for the next session.