How do you run LLMs securely at scale? An operations framework

Running LLMs securely at scale means treating them as production systems with owners, inventories, scoped permissions, input and output controls, logging and a review cycle. NIST's AI Risk Management Framework gives the governance structure, and the OWASP Top 10 for LLM applications gives the technical risk list.

Why is scaling the hard part?

One prototype with one prompt is easy to reason about. Thirty teams each wiring a model into support, search, code review and finance is not. Every integration brings its own prompts, data sources, tool permissions and vendors. Risk grows with the number of connections, not the number of models.

What does the governance layer look like?

The NIST AI RMF organizes work into four functions: Govern, Map, Measure and Manage. For operations, translate them into plain tasks.

• Govern: name an owner for each AI system and set rules for what data may go to which model.
• Map: document what each system does, what it can touch, who is affected and what could go wrong.
• Measure: test for the failure modes you listed, and keep the results.
• Manage: act on findings, track residual risk, and retire what no one owns.

The framework is voluntary and flexible. Its value is a shared checklist that audit, engineering and security can all read.

What are the technical risks to cover?

The OWASP Top 10 for LLM applications lists categories such as prompt injection, sensitive information disclosure, supply chain weaknesses, excessive agency, and unbounded consumption. Each maps to an operational control.

Prompt injection. Text from an email, web page or document can contain instructions the model follows. Treat all retrieved content as untrusted, and keep it separate from system instructions where the platform allows.

Sensitive data. Decide what may enter prompts. Strip or tokenize identifiers before they leave your boundary, and check provider terms on retention and training.

Excessive agency. An agent with broad tool access can do damage if tricked. Grant the smallest set of tools and the narrowest scopes, require human approval for irreversible actions, and use short lived credentials.

Supply chain. Models, plugins, embeddings and libraries come from outside. Keep an inventory and pin versions.

Unbounded consumption. Set rate limits and spend caps per user and per key so one runaway loop does not become an outage or a bill.

Constructed example: a support assistant

A company adds an assistant that reads tickets and can issue refunds. The first version lets it call the refund tool freely. A customer pastes text into a ticket that tells the assistant to refund the largest recent order. Because tickets are untrusted input, that is a prompt injection path with real money attached.

The revised design caps refunds per ticket, requires approval above a threshold, logs every tool call with the ticket ID, and limits the assistant to reading only the current customer's records. The model is the same. The operating framework changed the outcome.

How to check or apply this

1. Build an inventory of every model, prompt, data source, tool and vendor in use, each with a named owner.
2. Classify data and write down which classes may reach external models.
3. Give each agent a minimal tool list and scopes. Require approval for payments, deletions and external messages.
4. Log prompts, tool calls and outputs with identifiers, and protect those logs as sensitive data.
5. Add rate limits, token budgets and spend alerts per key.
6. Test against the OWASP categories before launch and after any prompt or tool change.
7. Review the inventory quarterly and retire unowned systems.

Safeguard's MCP Guardrail and agent identity features are built for the tool access side of this list, and the MCP server supports bring your own model setups, so tool use can be governed whichever model you pick.

Quick check

Sources

• NIST AI Risk Management Framework
• OWASP Top 10 for LLM Applications

Last reviewed: 18 October 2026.