Safeguard
AI Security•

What is prompt engineering, and how should teams do it safely?

Prompt engineering is writing and testing the instructions you give a model. Teams do it well with success criteria, evals and wary handling of inputs.

Safeguard Security Team
4 min read

Prompt engineering is the practice of writing, testing and refining the instructions and context you give a language model so it produces useful output. For teams, the useful version looks like software work: define what success is, measure it with evaluations, version the prompts, and treat untrusted input as hostile.

Where should you start?

Before writing a prompt, decide what good looks like. Anthropic's prompt engineering documentation says to have a clear definition of success criteria for your use case, some way to test against those criteria empirically, and a first draft prompt to improve. It points readers to its guidance on defining success criteria and building evaluations if those are missing, and says to establish them first.

It also gives a useful caution: not every failing eval is best solved by prompt engineering. You can sometimes improve latency and cost more easily by choosing a different model. So the first question is not "how do I word this?" but "is a prompt the right lever?"

What does a good prompt contain?

The same documentation lists the techniques it covers: clarity, examples, XML structuring, role prompting, thinking and prompt chaining. In practice, a strong prompt for production use tends to include:

  • The task, stated plainly. What to do, for whom, and what a good result looks like.
  • Context. The facts the model needs, supplied rather than assumed.
  • Examples. A few inputs with the outputs you want, including a hard case.
  • Format. The exact shape of the output, especially if a program will read it.
  • Boundaries. What the model should do when it lacks information, such as say so instead of guessing.

Keep the instructions and the data clearly separated, and say which part is which.

What changes when the prompt is in an application?

Anything the application puts into the prompt from outside becomes an attack surface. The UK NCSC describes prompt injection as developers concatenating their own instructions with untrusted content in one prompt and treating the response as though a reliable boundary separated the two. It says current models do not enforce a security boundary between instructions and data. OWASP lists prompt injection first among its LLM application risks.

So prompt wording cannot be your only defense. Limit what the model can do, validate its output before another system uses it, and enforce authorization outside the model.

A constructed example

A team builds a feature that summarizes customer emails and tags them. Version one has a single paragraph prompt and is judged by eye. It works on the first ten examples and fails on long threads and on emails in other languages.

Version two starts with success criteria: tags must come from a fixed list, the summary must be under 60 words, and nothing may be invented. The team builds a set of 80 real, anonymized emails with expected tags, including a few that contain instructions such as "ignore previous instructions". They run the set on every prompt change, store prompts in the repository with review, and make the application discard any tag not on the list. Failures become new test cases.

How to check or apply this

  1. Write success criteria in measurable terms before you write the prompt.
  2. Build an evaluation set from real cases, including edge cases and hostile ones.
  3. Change one thing at a time and rerun the evals so you know what helped.
  4. Consider a different model or a different design before piling more words into the prompt.
  5. Store prompts in version control, review them like code, and record which model version they were tested on.
  6. Treat all external content as untrusted. Do not give the model tools or permissions it does not need.
  7. Validate structured output, and reject anything that fails.
  8. Re-run evals when you change the model, the context source or the prompt.

Quick check

Should I fix every failing test by rewriting the prompt?

No. Anthropic's documentation notes that not every failing eval is best solved by prompt engineering. A different model or application design can sometimes fix latency or cost more simply.

Can careful wording prevent prompt injection?

Not reliably. The NCSC says models do not enforce a boundary between instructions and data, so limit permissions and validate output as well.

Sources

Last reviewed: 18 October 2026.

Never miss an update

Weekly insights on software supply chain security, delivered to your inbox.

Self-healing security runs on Safeguard.

Your first fix PR is minutes away.

No sales call required, even your agent can complete the purchase over MCP.