A RAG system should log the user, the query, the IDs and sources of every retrieved chunk, the final prompt shape, the model output, and any tool calls with the identity that made them. With those, you can trace a bad or leaked answer back to the document that caused it.
Why this is a security control
Two sources point the same way. OWASP's LLM08 guidance recommends detailed, immutable logs of retrieval activity to detect and respond to suspicious behaviour. The ATLAS mitigation AI Telemetry Logging says to log inputs and outputs of deployed models and, for agents, the intermediate steps, data access, tool use and agent identity.
Without them, an incident review starts with "we think the assistant said something wrong" and ends with no cause.
The minimum field list
Request. Timestamp, request ID, authenticated user or service identity, tenant, and the raw query.
Retrieval. For each retrieved chunk: chunk ID, document ID, source location, ingestion time, similarity rank, and the access label that allowed it. This is the most valuable part. If a poisoned document shows up in answers, this tells you which one and when it entered.
Prompt assembly. A record of which template version was used and which chunks were included. Whether to store the full prompt depends on privacy rules. At minimum keep a hash and the chunk list.
Model call. Model name and version, parameters, token counts, latency.
Output. The response text, plus the result of any output checks and whether they blocked or changed anything.
Tools. Each call: which tool, arguments, result status, and which identity authorised it.
Ingestion. Separately, log every document added or changed: who submitted it, from where, and what the pipeline did to it.
Protect the logs
Logs of RAG traffic hold sensitive text, because queries and chunks do. Apply the same access controls as the source data. Mask known secret patterns. Set retention on purpose.
Make the logs hard to alter. OWASP uses the word immutable for a reason: an attacker who can edit your retrieval log can hide a poisoning.
Questions the logs should answer
- Which documents were retrieved for this answer?
- Who added the document, and when?
- Which other answers used the same chunk?
- Did any output check fire?
- Which tool calls followed retrieval of this chunk?
Run these as a drill. Pick a real answer from last week and try to answer all five in under an hour. Gaps show you what to add.
Alerts worth setting
Keep the list short. Alerts on a chunk that suddenly appears in many unrelated queries, on a new document that outranks established ones for common questions, and on tool calls made right after retrieval of content from an untrusted source are cheap and useful. These are heuristics, so tune them on your own traffic.
Quick check
Do I have to store full prompts?
Not necessarily. Storing chunk IDs and a template version often lets you rebuild the prompt, and it keeps less sensitive text in the log.
Is model output logging enough?
No. Output alone does not show which document caused it. Retrieval logs are what link an answer to its source.
Sources
- OWASP LLM08:2025 Vector and Embedding Weaknesses
- MITRE ATLAS data repository (mitre-atlas/atlas-data), mitigation AML.M0024, 2026.09 release.
Last reviewed: 19 October 2026.
