Why LLM triage output should be validated JSON, not free text

When an LLM feeds a triage workflow, ask for a fixed JSON shape and validate it with ordinary code before anything downstream uses it. Free text is hard to compare, hard to log, and risky to pass along. A validated schema makes failures obvious and keeps surprising output away from your tickets and automation.

The security reason

OWASP describes improper output handling as failing to validate, sanitize, and handle model outputs before they reach other components. Because a prompt can influence the output, that effectively gives users indirect access to downstream functionality. Its guidance is to treat the model like any other user and adopt a zero trust approach to what it returns.

OWASP's prompt injection guidance also lists defining and validating expected output formats with deterministic code as a mitigation.

A schema for triage

{
  "finding_id": "string",
  "in_scope": true,
  "quote": "string",
  "path_to_untrusted_input": "string or null",
  "assumptions": ["string"],
  "recommended_priority": "urgent | scheduled | accept | needs_human",
  "verify_next": ["string"]
}

Notes on the design:

• A fixed set of priority values means a script can route on it, and anything outside the set is rejected.
• A quote field lets you check grounding with a string search against the source.
• needs_human gives the model an honest exit.
• Assumptions and verify_next surface what the answer depends on.

Validate in code

Parse the JSON, check it against the schema, and verify the quote appears in the source text. If any check fails, do not retry silently forever. Retry once with the error, then route the item to a person. Log the raw output so you can see failure patterns.

Never place model fields into a shell command, a query string, or HTML without the usual encoding for that context. OWASP's guidance includes parameterized queries and context aware encoding for exactly this reason. A "summary" field is still user influenced text.

What the schema does not guarantee

A well formed answer can still be wrong. The shape tells you the output is parseable, not that the reasoning is correct. Combine it with the checks in verifying an LLM's security finding.

Some APIs offer constrained or structured output modes, and tool calling can enforce a shape. Check your provider's current documentation for what it supports, since features differ and change. Even with those modes, keep your own validation, because it is the layer you control.

Keep actions separate from analysis

A triage step should produce data. A different component, with its own permissions and a human approval where it matters, should act on that data. If the same model output can both classify a finding and close the ticket, an injection in the finding text can close the ticket.

Quick check

Sources

• LLM05:2025 Improper Output Handling (OWASP GenAI)
• LLM Prompt Injection Prevention Cheat Sheet (OWASP)

Last reviewed: 19 October 2026.