A compound AI system solves a task with several interacting parts: multiple model calls, retrievers, external tools and checks, not one model alone. Each connection between parts is a place where untrusted text can steer an action. Secure it by validating every hand-off and giving each tool the smallest permissions that work.
Where does the term come from?
In February 2024, researchers at Berkeley (with co-authors from Databricks and other institutions) published an essay arguing that the best AI results were increasingly coming from systems rather than single models. They defined a compound AI system as one that tackles tasks using multiple interacting components, including multiple calls to models, retrievers, or external tools. An AI model, by contrast, is just a statistical model such as a transformer that predicts the next token.
Their examples were concrete. AlphaCode 2 generates a large pool of candidate programs and filters them. AlphaGeometry pairs a language model with a symbolic solver. The same essay reported that, among Databricks customers, 60% of LLM applications used retrieval-augmented generation and 30% used multi-step chains.
Why do teams build them?
The essay gives several reasons, and they matter for security because each one adds a component.
- Some tasks improve faster through system design than through a bigger model.
- Models are trained on static data, so retrieval brings in fresh information and access control.
- Systems are easier to control: you can filter outputs or verify answers against retrieved sources.
- Cost and quality targets vary, so teams mix small and large models.
Every one of those components is also code, configuration and credentials that somebody has to own.
What changes for security?
With a single model behind a chat box, the main questions are what the model says and what data it saw. In a compound system the model's output becomes input to something else: a database query, a shell, a ticketing API, another model. OWASP's 2025 list for LLM applications captures two relevant failures.
Improper output handling (LLM05) is insufficient validation, sanitization and handling of model output before it goes downstream. OWASP's examples include model output passed to a shell or eval, unparameterized SQL generated by a model, and unescaped script returned to a browser. Excessive agency (LLM06) is the ability to take damaging actions in response to unexpected, ambiguous or manipulated model output, rooted in excessive functionality, permissions or autonomy.
OWASP's advice for the first is blunt: treat the model as any other user and apply zero-trust input validation to what it returns.
A constructed example
Take a support assistant built from four parts: a retriever over help-center pages and past tickets, a model that drafts replies, a refund tool, and a reviewer model that checks tone. This is an invented design for illustration.
An attacker files a ticket containing text that reads like an instruction. The retriever later pulls that ticket into context for a different customer. The drafting model may follow it and call the refund tool. No single component is broken; the weakness is that retrieved text, which anyone can write, sits beside instructions and can reach a tool with money attached.
Controls that fit: mark retrieved content as data, cap refund amounts in the tool itself (not in the prompt), require human approval above a threshold, and log every tool call with the context that triggered it.
How to check or apply this
- Draw the system: list every model, retriever, tool, store and the arrows between them. Unlisted arrows are where surprises live.
- For each arrow, ask who can write the text that crosses it. Public tickets, web pages and uploaded files are untrusted.
- Validate model output the way you would validate a form field before it reaches a database, shell or browser. Use parameterized queries and context-aware encoding.
- Give each tool its own identity with only the permissions it needs. A read tool should not hold write credentials.
- Put hard limits in code: amounts, rates, allowed targets. Prompts are not enforcement.
- Keep an audit trail of prompts, retrieved sources and tool calls so an odd action can be traced.
- Test the whole chain with adversarial inputs, not just each part.
Safeguard's MCP Guardrail is designed for the tool-call boundary in this picture, applying policy between an agent and the tools it can reach. It does not replace the validation steps above.
Quick check
Is a RAG chatbot a compound AI system?
Yes. A model plus a retriever is the simplest compound system, because the answer depends on more than one component. That also means retrieved content is a security input.
Does a second model reviewing the first make it safe?
It helps with quality, but a reviewer model reads the same untrusted text and can be misled too. Hard limits in code and least-privilege tools matter more.
Sources
- The Shift from Models to Compound AI Systems, Berkeley Artificial Intelligence Research blog
- OWASP LLM05:2025 Improper Output Handling
- OWASP LLM06:2025 Excessive Agency
Last reviewed: 18 October 2026.
