Small models as security classifiers: mistakes teams make

Small models make cheap, fast security classifiers for prompts and outputs, but teams often treat them as a wall instead of a signal. The usual mistakes are trusting a benchmark score, skipping a local test set, relying on the classifier as the only control, and never retesting after changes. Use it as one layer among several.

What is a small classifier doing here?

Meta's Llama Guard paper is a good example of the pattern. It describes an LLM-based input and output safeguard model for conversations, built on a 7 billion parameter model and instruction tuned on a small, high quality dataset. It uses a taxonomy of risk categories, classifies user prompts and model responses, and produces a decision. The authors report results on par with or better than existing moderation tools on two named datasets, and say users can adjust the categories.

That is useful. It is also a result on those datasets, under their taxonomy. Your traffic is different.

Mistake 1: Treating the benchmark as your accuracy

A score on a public dataset says little about your prompts, your languages or your attackers. Build a labeled set from your own logs, with examples that should pass and examples that should be blocked, and measure false positives as carefully as misses. A classifier that blocks real customer questions will be switched off by the first angry executive.

Mistake 2: Making the classifier the only control

OWASP's prompt injection page says it is unclear whether fool-proof prevention methods exist, because of the stochastic nature of generative models. It also says retrieval and fine tuning do not fully mitigate the problem. Its list of mitigations is layered: constrain behavior, validate output formats, filter inputs and outputs, enforce least privilege, require human approval for high risk actions, mark untrusted content, and test adversarially. A classifier fits in the filtering layer. It does not replace permissions.

Mistake 3: Using one taxonomy for every job

A category list built for harmful content does not detect leaked credentials or an agent about to delete a database. Define classes for your own risks and test each. Where a rule can be written in code, such as a regular expression for a key format or a check on output structure, use the code. Cheap deterministic checks beat a model for narrow, known patterns.

Mistake 4: Ignoring who the classifier calls

If the model decides what a tool may do, an error becomes an action. Keep authorization in the systems behind the model, and use the classifier to raise a flag or request approval.

Mistake 5: Not watching drift

Prompts change, models behind your app change, and attackers adapt. Re-run your labeled set on a schedule and after every change to the main model or the classifier. Track precision and recall over time, not once.

Mistake 6: Skipping the log

When a classifier blocks or passes something, store the input hash, the class, the score and the decision, with privacy controls for sensitive text. Without that, you cannot tune it or explain an incident.

What to do

1. Write down the specific risks you want the classifier to catch.
2. Build a labeled test set from your own traffic, with both pass and block examples.
3. Measure false positives and misses separately and set a threshold from the numbers.
4. Put deterministic checks in front for known patterns.
5. Keep least privilege and human approval behind it.
6. Log decisions, and retest on a schedule and after any model change.

Quick check

Sources

• Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations (arXiv 2312.06674)
• OWASP GenAI: LLM01 Prompt Injection

Last reviewed: 18 October 2026.