Security engineering rests on the ability to write contracts. A function promises that it will return one of these values, or raise one of these errors, or modify this piece of state in this way. The contract is what lets a caller reason about what to do with the result. When the contract holds, composition works. When it does not, composition produces emergent behavior that nobody designed and that auditors struggle to reason about.
Frontier large language models have an output space that is, for practical purposes, unbounded. Any sequence of tokens up to the maximum generation length is a legal output. This is a capability feature, because it is what lets the model produce novel answers to novel questions. It is also a structural limitation for anyone trying to treat the model's output as a contract.
What "unbounded" actually means
It is not quite right to say that the output space is infinite. The number of distinct token sequences of a given length is finite, even if astronomically large. The relevant property is not the raw count, but the fact that no enumerable subset of outputs is privileged by the architecture. The model will, under some distribution of inputs, produce outputs that the developers did not anticipate, that the evaluation set did not cover, and that the schema validator does not expect.
Classical functions do not work this way. A function that returns an integer returns an integer; the output space is fixed and the set of values is known. A function that returns one of an enumerated set of strings returns one of that set; the output space is enumerable and the validator can check exhaustively. Even a function that returns a JSON object has a schema that bounds what it will return, and deviations from the schema are errors that the caller can detect.
A model has no such schema in the general case. Prompt engineering, structured output modes, and grammar-constrained decoding all narrow the output space, but none of them narrow it to a fixed set that the caller can enumerate. The set is always "whatever the model can say," and the caller's validator has to operate on samples rather than on a specification.
The first consequence: no exhaustive testing
The discipline of software testing depends on the ability to cover the output space, at least in some useful sense. Unit tests cover the expected outputs. Property tests cover invariants that should hold across the output space. Fuzz tests cover the inputs that might produce unexpected outputs. All of these techniques assume that there is a relationship between the coverage of the tests and the confidence that the function will behave well in production.
This relationship breaks down when the output space is unbounded. A model can pass every test in an evaluation set and still, on the next input, produce an output that violates every assumption the tests were designed to check. The evaluation gives you a statistical confidence about the distribution of outputs, not a guarantee about any individual output. Moving from "the model passed the test" to "the model will behave correctly in production" is a leap that test coverage does not justify.
This is not an argument against testing. Evaluation sets are essential, and teams that do not maintain them are flying blind. It is an argument against the mental model that tests establish correctness. For LLMs, tests establish a prior, and the prior has to be updated continuously as the model encounters new inputs in production.
The second consequence: the validator is the real contract
If the model's output cannot be contractually constrained at the source, the contract has to live somewhere else. In practice, this place is the validator: the code that sits between the model's output and anything consequential.
This inverts the usual relationship between a function and its caller. Classically, the callee promises what it will return and the caller can trust the promise. With an LLM, the caller has to verify every return, because the callee makes no binding promise. The validator is not an optional sanity check on top of a correct function. It is the mechanism by which the caller creates the contract the model did not provide.
The practical implication is that validator quality dominates system security. A loose validator lets too much through. A strict validator rejects legitimate outputs and breaks the user experience. The engineering effort that teams put into the validator is usually the single biggest lever on whether their LLM-powered system is safe to deploy.
Validators that take this seriously tend to look like small interpreters. They parse the output, check its structure, verify that the values fall within expected ranges, and reject or retry if anything is off. They are explicit about what they accept and what they do not, and they log rejections as signal about where the model is drifting. They are, in effect, the deterministic specification that the model itself cannot be.
The third consequence: emergent capabilities
Unbounded output space also means that models acquire new capabilities as they scale, and they acquire them in ways that are not fully predictable. A capability that was absent in a smaller model can appear in a larger one, and the surrounding system has to contend with outputs that it was not designed to handle.
From a security perspective, this is a version of the problem that any system faces when the threat landscape shifts under it. The difference is that for classical systems, the shift comes from the outside: new attackers, new techniques, new vulnerabilities. For LLM systems, the shift can come from the inside: a model update that changes what the model can express, and therefore what kinds of outputs the validator will see.
The practical effect is that validators have to be maintained as living artifacts. A validator that was adequate for last year's model may be inadequate for this year's, because this year's model can produce outputs that last year's could not. Teams that treat the validator as a one-time engineering project end up with a contract that drifts out of sync with the system it is supposed to constrain.
The fourth consequence: the long tail matters
In a system with a bounded output space, the tail of rare behaviors can be made as small as the testing effort demands. In a system with an unbounded output space, the tail is always there, and the question is not whether it exists but how much of it you will see.
This matters for security because adversaries specialize in the tail. A normal user is likely to elicit outputs that fall within the distribution the validator was designed for. An adversarial user is trying specifically to elicit outputs that do not. The exposure to tail behavior is therefore correlated with exposure to attack, and any claim about system safety that is based on normal-user statistics will understate the risk.
The mitigations are familiar from other tail-risk domains. Red teaming gives you a sample of adversarial inputs to test against. Defense in depth reduces the damage of any single validator miss. Anomaly detection on outputs gives you signal when the tail is being actively explored. None of these eliminate the tail; they bound what happens when it is encountered.
The structural point
The unbounded output space of frontier models is not a defect. It is the property that makes the models useful for open-ended tasks. But it changes the security engineering contract in ways that are easy to miss. Promises about what the model will return cannot be trusted at the model level, and have to be enforced at the boundary.
A team that accepts this and invests in validators, evaluation, and monitoring can build systems whose security properties are understandable and testable. A team that treats the model as a classical function with a reliable output contract is building on an assumption the substrate does not support. The gap between those two postures is where most of the production failures of LLM systems actually live.