A large language model (LLM) is a neural network trained on large amounts of text to predict the next token in a sequence. Repeating that prediction produces sentences, code or summaries. It is useful for drafting and transformation, but it has no built-in fact check, so its output must be verified before anything acts on it.
How does an LLM produce an answer?
Text is split into tokens, which are pieces of words. The model turns tokens into numbers, passes them through many layers of learned weights, and outputs a probability for every possible next token. One token is chosen, appended, and the process repeats. A setting often called temperature controls how adventurous the choice is.
Training happens in stages. Pretraining on a very large text collection teaches general language patterns. Later tuning with examples and human or automated feedback shapes the model into an assistant that follows instructions. After training, the weights are fixed. The model does not learn from your chat unless the provider deliberately uses it for later training, which depends on their terms and your settings.
The model only sees what is in its context window, the text provided in the current request. Anything else it "knows" is compressed statistical memory of training data, which is lossy and dated.
Why do LLMs get things wrong?
The model is rewarded for plausible text, not for truth. OWASP's entry on misinformation says hallucinations occur when models fill gaps in training data using statistical patterns, producing answers that sound correct but are unfounded. It also names overreliance, where people trust outputs without verifying them, as a multiplier.
Other limits follow from the design. Training data has a cutoff date. Long contexts can dilute attention. And the model cannot reliably distinguish instructions from data, which is the root of prompt injection.
What are the main security risks?
OWASP maintains a Top 10 for LLM applications that is a good map. The recurring ones are:
- Prompt injection. Text in a document, web page or email contains instructions, and the model follows them as if the user wrote them.
- Improper output handling. The application passes model output into a shell, a database query or a web page without checks. OWASP's entry on this risk lists examples such as XSS, SQL injection and remote code execution.
- Excessive agency. The model is given tools with more permission than the task needs.
- Sensitive data disclosure. Private data in prompts, training sets or retrieval stores appears in outputs.
- Supply chain. Models, datasets and plugins come from outside and can be altered.
NIST's AI RMF and its generative AI companion material offer a governance structure for these risks.
A constructed example
A support assistant summarises incoming emails and can also issue refunds through an internal tool. An email arrives containing hidden text that says to refund the sender's last order. The model, which sees email text and instructions in one stream, may comply.
The fix is not a better prompt. It is design: the summariser has no refund tool, refunds need human approval above a limit, and tool calls are logged and checked against policy outside the model.
How to check or apply this
- Decide what the model can see and what it can do. List both, then cut each to the minimum.
- Treat all model output like input from an anonymous user. Validate, encode and parameterise before it reaches any system.
- Keep secrets out of prompts. Use scoped, short-lived credentials for tools.
- Require human approval for irreversible or costly actions.
- When facts matter, retrieve them from a trusted source and cite them, then check the citation exists.
- Log prompts, tool calls and outputs, with privacy controls, so you can investigate incidents.
- Test with adversarial inputs before launch and after each model change, since behaviour shifts between versions.
- Record which model, version and data sources each system uses, as part of your inventory.
Quick check
Does an LLM understand what it writes?
That is debated. For security purposes, assume it has no reliable grasp of truth or intent. It produces likely text and can be wrong with full confidence.
Can I stop prompt injection with a stronger system prompt?
Not reliably. Limit the model's permissions and verify outputs outside the model, so a successful injection cannot do much damage.
Sources
- OWASP Top 10 for LLM Applications
- OWASP LLM09 Misinformation
- OWASP LLM05 Improper Output Handling
- NIST AI Risk Management Framework
Last reviewed: 18 October 2026.
