Ten practices for securely developing with AI, from NCSC and OWASP

To develop securely with AI, treat model output as untrusted input, give models and agents the minimum permissions, vet every model and data source, protect secrets and prompts, test adversarially, and monitor in production. The UK NCSC and OWASP both frame these as lifecycle practices, not one time checks.

What frameworks should you start from?

The NCSC's Guidelines for secure AI system development apply to providers of any system that uses AI, whether built from scratch or on top of services others provide. They split the lifecycle into four areas: secure design, secure development, secure deployment and secure operation and maintenance. They follow a secure by default approach and say security must be a core requirement throughout the lifecycle.

OWASP's Top 10 for LLM and Gen AI applications lists risks: prompt injection, sensitive information disclosure, supply chain, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation and unbounded consumption.

The ten practices

1. Threat model before you build. The NCSC puts understanding risks and threat modeling in the secure design stage. Ask what the model can read, what it can do and who can talk to it.

2. Treat model output as untrusted. OWASP's improper output handling covers insufficient validation and sanitization of model output before it reaches other systems. Never pass it straight to a shell, a database query or a browser.

3. Assume prompt injection. Text from users, documents or web pages can change a model's behavior. Separate instructions from data where you can, and limit what a manipulated model could do.

4. Limit agency. OWASP notes LLM based systems are often granted a degree of agency. Give each agent the narrowest set of tools, scoped credentials and human approval for destructive actions.

5. Vet the supply chain. The NCSC covers supply chain security in secure development. Track which models, datasets, libraries and plugins you use, where they came from and which version you run.

6. Protect training and retrieval data. Data poisoning can affect pre-training, fine-tuning and embedding data. Control who can write to these sources.

7. Keep secrets out of prompts. OWASP lists system prompt leakage and sensitive information disclosure. Do not place keys or private data in a system prompt, and assume the prompt can be revealed.

8. Bound consumption. Unbounded consumption means uncontrolled use of the model. Set rate limits, token limits and spending alerts.

9. Test adversarially. Include injection attempts, malformed input and abusive usage in your tests, and rerun them when you change the model or prompt.

10. Log, monitor and plan for incidents. The NCSC's operation stage covers logging and monitoring, update management and information sharing, and its deployment stage covers incident management processes.

What does a failure look like?

A constructed example: a support assistant can read tickets and call a refund tool. An attacker files a ticket whose text instructs the assistant to refund an order. If the assistant reads that text as an instruction and the refund tool has no limit, it complies. Three practices would stop it: treating ticket text as data, scoping the tool to small refunds with human approval above a threshold, and logging every tool call.

Where does Safeguard fit?

Safeguard offers guardrails for AI agents and developer tools through MCP Guardrail, plus policies and gates and agent identity, which help limit what an agent can do and record what it did. It scans dependencies and builds an SBOM, which supports the supply chain practice above.

How to check or apply this

1. Write a one page threat model for each AI feature.
2. Validate and encode model output before using it anywhere else.
3. List every tool an agent can call and remove anything unneeded.
4. Require approval for destructive or high value actions.
5. Keep an inventory of models, datasets and plugins with versions.
6. Restrict write access to training and retrieval data.
7. Move secrets out of prompts and into scoped runtime credentials.
8. Set rate, token and spending limits.
9. Add an adversarial test suite and rerun it on every change.
10. Log prompts, tool calls and outcomes with retention you can defend.

Quick check

Sources

1. NCSC, Guidelines for secure AI system development
2. OWASP Gen AI Security Project, Top 10 for LLM Applications 2025

Last reviewed: 18 October 2026.