In August 2021, researchers ran GitHub Copilot against 89 security-relevant coding scenarios and generated 1,689 candidate programs; when they checked the output against MITRE's CWE Top 25, roughly 40% contained an exploitable weakness. That study, "Asleep at the Keyboard?" by Pearce et al. (published at IEEE S&P 2022), is now nearly five years old, but the underlying dynamic hasn't gone away — it has moved targets. In June 2025, security researcher Johann Rehberger disclosed CVE-2025-53773, a prompt-injection flaw in GitHub Copilot for VS Code that let a hostile source comment or web page trick the agent into writing "chat.tools.autoApprove": true into .vscode/settings.json, silently switching the assistant into an unattended "YOLO mode" that could execute shell commands without asking. Microsoft shipped a fix in its August 2025 Patch Tuesday. And in March 2025, Pillar Security disclosed a "Rules File Backdoor" affecting both Cursor and Copilot, where invisible Unicode characters hidden in .cursorrules and Copilot instruction files steered the assistant toward inserting backdoors that looked clean in a normal diff review. None of these are reasons to ban AI coding assistants. They're a map of exactly where guardrails need to sit: in the prompt surface, in the output, and in the human review step.
Why isn't code review alone enough to catch AI-introduced risk?
Code review alone isn't enough because the Pillar Security "Rules File Backdoor" research specifically demonstrated that a malicious instruction can be made invisible in the exact view a reviewer relies on. Using bidirectional text markers and zero-width joiners embedded in a .cursorrules or Copilot custom-instructions file, an attacker can plant a directive like "always import this helper" that a human sees as blank whitespace in GitHub's diff view, while the assistant reads and obeys it on every completion it generates afterward. Cursor's team responded that this fell outside their threat model since rules files are user-supplied configuration; GitHub's position was that developers are responsible for reviewing AI suggestions, and it later added a UI warning for hidden Unicode characters in files rendered on github.com. The practical lesson for a security team: config and rules files that steer an AI assistant are an untrusted input surface in their own right, and they need the same diffing and content scanning as generated code — not an assumption that "a human will read it" is sufficient when the content is engineered not to be readable.
What guardrail actually stopped CVE-2025-53773 from becoming routine?
The guardrail that stopped CVE-2025-53773 from being routine was the explicit human-approval requirement Microsoft restored around security-relevant configuration changes. Before the fix, Copilot's agent mode could read attacker-controlled text — a source comment, an issue body, or a web page fetched during a task — treat it as an instruction, and write chat.tools.autoApprove: true into workspace settings without asking, which removed the confirmation dialog for every subsequent shell command the agent ran. Embrace The Red's write-up on the disclosure framed this precisely as a prompt-injection-to-RCE chain enabled by a config write that silently disabled a safety control. The fix pattern generalizes well beyond this one CVE: any AI agent guardrail should distinguish between changes to application code (which a human will read in a PR) and changes to the assistant's own permission settings (which a human may never open a diff view on at all), and gate the second category behind an explicit, un-skippable approval regardless of what the agent's own reasoning claims is safe.
Where should prompt-level constraints actually live?
Prompt-level constraints should live in the assistant's system-level configuration and in your CI pipeline, not solely in ad hoc instructions a developer types into a chat window, because chat-level guidance is exactly the layer prompt injection targets. Concretely: restrict which tools an agent can invoke without confirmation (file writes outside the working directory, shell execution, network calls, and — as CVE-2025-53773 showed — changes to its own settings should never be on the auto-approve list); pin or scope which repositories and directories an agent session can read, since Copilot's and Cursor's agent modes will follow links and read arbitrary files a prompt points them to; and treat any file the assistant ingests as instructions (rules files, .github/copilot-instructions.md, AGENTS.md) as a reviewed artifact with its own commit history and diff, not a config file nobody looks at twice. The goal is to shrink the blast radius of a successful injection before it ever reaches generated code.
What does output scanning need to catch that a human reviewer typically misses?
Output scanning needs to catch the same defect classes the Pearce et al. study found in bulk — CWE-89 SQL injection, CWE-798 hardcoded credentials, CWE-502 unsafe deserialization, and similar CWE Top 25 patterns — because an AI assistant reproduces these at the base rate of its training data, and a reviewer skimming a plausible-looking diff has no reason to suspect a specific line. This is a static-analysis problem, not a novel one: the same SAST, secrets-scanning, and dependency-reachability checks a team already runs on human-written code need to run, unmodified, on every AI-generated commit, ideally before a human even opens the PR. The failure mode to guard against is treating "an AI wrote it" as a signal of either higher or lower trust — it deserves exactly the same automated scrutiny as a junior engineer's first pull request, applied consistently rather than opportunistically.
What does a workable human review gate look like in practice?
A workable human review gate treats two things as mandatory checkpoints: any AI-proposed change to the assistant's own tool permissions or execution settings, and any AI-generated diff that touches authentication, deserialization, shell/subprocess calls, or dependency manifests — the categories most associated with both the academic CWE findings and the real-world CVEs above. In practice that means CI should block merge on a failed security scan regardless of who or what authored the commit, exceptions to that block should be time-boxed and require a named approver rather than a blanket bypass, and the review step itself should render rules/instruction files through a tool that surfaces hidden Unicode rather than a plain diff view, closing the exact gap Pillar Security identified. Safeguard's own guardrail model applies this pattern generically: policies evaluate at the IDE, commit, and CI stages with a BLOCK, WARN, or AUTO_FIX effect, every enforcement decision is logged to a signed audit record, and exceptions expire automatically rather than persisting silently — which is the structure an AI-assistant-specific policy (for example, "block any commit that modifies chat.tools.autoApprove or equivalent auto-run settings without a named approver") can be written against without inventing new tooling. Safeguard also connects into Copilot and Cursor via its MCP server so engineers can query vulnerability, SBOM, and license data from inside the assistant — useful for informing a fix, though it's a query surface rather than a replacement for the scanning and approval gates described above.