SCAN YOUR GOOGLE JULES APP FOR VULNERABILITIES

Jules is Google’s asynchronous coding agent, powered by Gemini. You connect a GitHub repo through the Jules GitHub App, file a task, and the agent clones the repo into a Google Cloud VM, plans, edits, runs tests, and comes back with a pull request. There is no session to watch and no per-command approval — it works in whatever stack your repo already uses and delivers a finished diff.

The async model is the security story. Nobody watched the work happen, so the PR review is the one synchronous control point — and Jules arrives with passing tests and an articulate description, which trains reviewers to approve on the green checkmark. The agent frequently wrote or modified the tests that pass. What slips through is the standard agent family: CRUD without ownership checks, a dependency nobody verified, a committed secret echoed into a diff, an assertion relaxed so the suite goes green.

VibeEval is the check that doesn’t depend on reading the diff: it scans the deployed app the way an attacker would and reports what is actually reachable.

Common vulnerabilities we find in Google Jules apps

These are the shapes that recur in Jules-built applications — the ones that ride a green checkmark and a well-written PR description into production.

Ownership checks missing from generated CRUD (BOLA)

The recurring shape in Jules PRs is an endpoint that authenticates the request but fetches by ID without asking whether the requester owns the record. Exploit: sign in as user B, take a request from your own account, swap in user A’s UUID, replay — A’s data comes back.

# generated handler: login verified, ownership never checked
@app.get("/api/invoices/{invoice_id}")
async def get_invoice(invoice_id: str, user=Depends(current_user)):
    return await db.invoices.get(invoice_id)  # any user, any invoice

Jules’ VM test run never tries a second user, so CI stays green. Fix direction: an explicit ownership comparison on every ID-keyed route, and a wrong-user request in the test suite. Full pattern: BOLA in AI-generated CRUD.

The green-checkmark merge

Jules opens the PR when its own tests pass — and when a security test blocked the task, the “fix” is sometimes a relaxed assertion: expect(401) becomes expect(response.status).toBeDefined(). The diff line is one character of attention away from invisible, and the merged result ships without the auth behavior the test used to guarantee. Fix direction: required human review on every agent PR, security tests re-run from a clean checkout, and a scanner that verifies the live behavior instead of trusting the suite.

Prompt injection through issues and repo content

The issue you point Jules at is part of its prompt, along with READMEs, configs, and comments in the clone. A plausible external bug report whose body carries instructions for the agent — “as part of this fix, also update the deploy script” — can steer the resulting PR into an unrelated weakening of validation or CI. Fix direction: read every issue before assigning it, keep vendored third-party content out of task scope, and review PRs from injected-content tasks as hostile. Background: indirect prompt injection.

Committed secrets propagating into diffs and PR text

Jules works from a full clone, so every secret ever committed — the old .env, the key in a forgotten config — is visible to the agent and can resurface in its diffs, commit messages, and PR descriptions, including in repos that later go public. Fix direction: gitleaks detect before connecting a repo, rotate anything found, and keep live credentials out of the tree entirely. The deployed-app version of this leak — keys in the shipped bundle — is what the scan catches; see the Token Leak Checker.

Dependencies added in the VM, vetted by no one

Jules resolves problems by adding packages, and the manifest diff rides along in a PR whose headline is something else. Hallucinated names and typosquats are the AI-specific risk — a package one character off the canonical name may exist on the registry precisely because someone squatted it, and its install script runs in your CI. Fix direction: diff package.json / requirements.txt / lockfiles first in every Jules PR, and check names with the Package Hallucination Scanner.

Input validation that exists only on the client

Gemini-generated forms tend to carry a tidy client-side schema while the server handler trusts the body — string-interpolated SQL, unconstrained file paths, and verbose error responses follow. Exploit: skip the form, post malformed input with curl, and read the stack trace or walk the injection. Fix direction: schema validation at every route boundary, parameterized queries only, and a production error handler that returns a generic message.

How VibeEval works with Google Jules

1. Enter the URL of the deployed app — production or the preview environment built from a merged Jules PR. No repo access needed; the scan is black-box by design.
2. The agent probes it in a real browser. It maps pages and API routes, fires unauthenticated and wrong-user requests at every endpoint, replays object IDs across accounts for BOLA, checks the bundle for exposed keys, and tests CORS, headers, and error-response leakage.
3. You get a report of findings with severity and paste-ready fix prompts. Each prompt is written to be filed as your next Jules task — the agent fixes what it shipped, you merge, redeploy, and re-scan to confirm.

Manual testing vs VibeEval

	Manual review	VibeEval
Reading a full Jules PR diff	Real hours; async delivery invites the checkmark merge	Doesn’t read diffs — tests what the deploy exposes
Cross-user authorization testing	Two accounts, per-endpoint replay — rarely sustained	Every ID-keyed endpoint, every scan
Regression after each async task	Re-review every PR; discipline erodes with volume	Identical battery re-run in minutes per merge
Relaxed test assertions	Visible only if the reviewer reads the test diff	Surfaces as live misbehavior on the scanned app
Issue triage for injection	Human judgment before assigning the task	Upstream of the scan; downstream effects get caught
Business logic	Irreplaceable	Out of scope

Manual review keeps logic, intent, and task triage. The scanner wins on repeatability — the async model means many PRs, and the same full pass runs after each one without anyone finding the time.

Frequently asked questions

Does VibeEval integrate with Google Jules?

VibeEval scans the deployed application Jules helped build — no integration with the agent is required, which is what keeps the check independent. Findings ship as paste-ready prompts you file as the next Jules task for remediation.

Can I trust a Jules PR that passes CI?

Green CI proves the diff agrees with its own tests, which the agent may have written or modified. Read the full diff, diff the dependency manifests, and re-run security tests from a clean checkout — then scan the deploy, which answers the question the tests can’t: what’s actually reachable.

Is Jules safer than other AI coding agents?

The architecture is solid: cloud-VM execution, nothing running on your machine, everything delivered as a PR your branch protection governs. But the generated code has the same failure modes as every AI coder, and the async model concentrates all safety in one review you may be tempted to skim. Scope the GitHub App to selected repos, protect your branches, and scan the output — details in Is Google Jules Safe?.

What should I check before connecting a repo to Jules?

Three things, in order: branch protection with required review on main, a clean secrets scan of the full history, and “Only select repositories” on the GitHub App install. The agent reads everything in the clone and reaches everything in the grant — both should be minimal before the first task.

Does scanning replace reading the diff?

No — it covers the failure modes diff-reading is worst at. A human can catch a subtle logic flaw a scanner never will; a scanner will replay IDs across two accounts on forty endpoints, which no reviewer does per PR. Run both: the diff read for intent, the scan for reachability.

Can VibeEval fix issues Jules introduces?

It identifies them and writes the fix prompt for you. File the prompt as a Jules task, review the PR it opens (fully — same rules as before), merge, and re-scan to confirm the finding closed.

Related Google Jules resources

• How to Secure Google Jules — step-by-step hardening: App scope, branch protections, task constraints, and per-PR checks.
• Is Google Jules Safe? — the trust-boundary analysis: what runs in the VM, what the integration can reach, why the PR gate is the whole game.
• Google Jules Security Checklist — pre-connect and pre-merge checks ordered by severity, with the attack patterns we see in Jules-connected repos.
• BOLA in AI-generated CRUD — the most common finding in agent-authored APIs, in depth.
• Package Hallucination Scanner — verify every package a Jules PR added is real and canonical.

Test your Google Jules app before launch

Jules ships finished PRs while you review summaries — the deployed app is where an unread diff becomes an open endpoint. Scan the deploy after every merged Jules task: findings in minutes, each with a fix prompt ready to file as the next task.