Few-shot examples for classifying security findings with an LLM

Give an LLM three to five worked examples when you ask it to label security findings, and include at least one tricky negative. Examples steer format and judgment more reliably than extra instructions, but they also copy your mistakes, so pick them from decisions a senior reviewer has already confirmed.

Why examples work

Anthropic's guidance calls examples one of the most reliable ways to steer output format, tone, and structure, and says a few well-crafted examples (few-shot or multishot prompting) improve accuracy and consistency. It recommends three to five, and suggests asking the model to evaluate your examples for relevance and diversity.

Classification is the natural fit. The labels are fixed, the inputs vary, and what you really want is consistent application of a rule that is hard to put in words, such as "real issue" versus "test code only".

Choosing the examples

Aim for variety, not volume.

• One clear true positive. A finding in production code with an attacker controlled input.
• One clear false positive. For instance a hard-coded string flagged as a secret that is a public identifier.
• One borderline case, labeled with the reasoning your team used.
• One negative in an unusual place, such as a finding in a test fixture or generated file.
• One example of "needs more information" so the model has a way out.

Take them from closed tickets, and prefer cases where more than one reviewer agreed. Remove client names and anything sensitive first. See what not to paste into an LLM.

Layout

<examples>
<example>
<finding>SQL built by string concatenation in orders.py, user id from request</finding>
<label>real</label>
<reason>Request value reaches the query unescaped.</reason>
</example>
<example>
<finding>Hard-coded token in tests/fixtures.py</finding>
<label>false_positive</label>
<reason>Dummy value used only by unit tests.</reason>
</example>
</examples>

Now label this finding using the same fields.
<finding>...</finding>

Always include the reason. It teaches the pattern, and it gives a reviewer something to disagree with.

Pitfalls

Label imbalance. If four of five examples are false positives, the model leans that way. Keep the mix close to what you want, and check against real data.

Copying surface features. If every true positive mentions "request" and every false positive mentions "test", the model learns keywords. Vary the wording.

Stale examples. Frameworks and policies change. Review the set whenever your triage rules change.

Format leakage. The model will copy your example format exactly, which is useful. Make sure the format is the one your parser expects.

Measure it

Hold out twenty labeled findings that are not in the prompt. Run them and compute simple agreement with the human label. Change one thing at a time, such as swapping an example, and rerun. If agreement does not rise above a plain instruction prompt, the examples are not earning their tokens.

Quick check

Sources

• Prompting best practices (Anthropic)

Last reviewed: 19 October 2026.