When security programs evaluate a piece of software, the first question they ask is usually some version of "what is in this." Dependencies, provenance, build environment, signing keys, source repositories, and licenses all feed into a judgment about whether the artifact can be trusted to behave as advertised. The discipline of software supply chain security is, in large part, the discipline of making this question answerable.
Frontier large language models do not submit to this discipline. The training data that shapes their behavior is, for all practical purposes, opaque to the people who depend on them. This opacity is not a temporary state of affairs that more transparency will cure. It is a structural feature of how these models are built, and it imposes limits on the kinds of trust that can reasonably be placed in their outputs.
What training data opacity actually looks like
The major frontier labs disclose their training data at a level of abstraction that is not useful for security reasoning. You will learn that a model was trained on "a mixture of licensed data, data generated by human trainers, and publicly available information." You will not learn which books, which websites, which code repositories, which forums, which question-and-answer sites, or which leaked document collections contributed to the final weights.
This is true even for organizations that have worked hard to be more transparent than their competitors. Publishing the list would, in most cases, invite litigation, complicate commercial relationships, and expose competitive advantages. The incentives favor opacity, and they will continue to do so regardless of what any individual lab prefers.
For a security engineer, this means that when a model produces a claim, you cannot trace the claim to its source. The model might have learned something from a well-maintained reference text, or from a Stack Overflow answer that was already wrong when it was written, or from a forum post that was later retracted, or from a piece of malware documentation written by the attacker. The output is the same shape regardless. The provenance is not available.
The gap between "publicly available" and "auditable"
It is common for labs to describe training data as "publicly available," which creates the impression that it could, in principle, be audited. The impression is misleading.
Publicly available data is an enormous, heterogeneous, and constantly changing set. A model trained on a snapshot of it is trained on a particular version of a particular subset, selected by a filtering pipeline that is itself rarely disclosed. The snapshot is almost certainly not archived in a form that a third party could reproduce. The filtering pipeline encodes judgments about quality, safety, and licensing that would shape the final model in ways that are not visible from the outside.
Even if the snapshot and the pipeline were fully disclosed, the scale would defeat most audit efforts. Auditing a trillion tokens of training data is not a human task, and the tools for auditing it at scale are themselves trained on other data, introducing the same opacity recursively. The field has not yet developed the equivalent of a software bill of materials for training corpora, and the existing proposals fall well short of what a serious audit would require.
Why this is a security problem
The security consequences of training data opacity fall into three broad categories.
The first is the possibility that the training data contains content that should not have been there. This includes personal data scraped without consent, copyrighted material licensed under terms that prohibit this use, and sensitive internal documents that leaked onto the public internet before being incorporated into a crawl. A model trained on this content can reproduce it, sometimes verbatim, in response to prompts that are not obviously trying to elicit it. The user of the model has no way to know which inputs are safe and which risk triggering a disclosure.
The second is the possibility that the training data was deliberately poisoned. An attacker who wants to influence model behavior can, in principle, seed the public internet with content designed to be ingested by crawlers. The content might associate a particular software package with a recommendation, insert a subtle security flaw into code examples, or bias answers about a particular product. Because the training set is opaque, the defender has no way to check whether this has happened until the behavior manifests in the output.
The third is the possibility that the training data biases the model in ways that are relevant to security decisions. A model trained predominantly on one ecosystem's documentation will produce more confident and more accurate answers about that ecosystem, and less reliable answers about others. A security team that uses the model to assess risks across a heterogeneous estate cannot see this bias from the outside, and will tend to over-trust the model's confidence.
The hallucinated dependency problem
A concrete example of how training data opacity becomes a security problem is the hallucinated dependency. When a frontier model is asked to write code, it sometimes names libraries that do not exist. The name is often plausible, because it is constructed by interpolation from names that do exist in the training data. An attacker who notices this pattern can register the hallucinated name in the relevant package registry, ship malicious code under it, and wait for developers who trust the model to install it.
The defender's position here is structurally weak. They cannot predict which names the model will hallucinate without running the model on the exact same prompt the developer will. They cannot clean up the training data, because they do not have access to it. They cannot ask the provider to guarantee that the model will not suggest a given package, because the provider does not have a per-package mechanism to enforce this. The opacity of the training data becomes an attack surface in practice.
The trust budget framing
Given all this, a useful framing for security teams is to think about a "trust budget" for frontier model outputs. The budget is the amount of consequential action you are willing to take on the basis of what the model says, and it should be calibrated to what you can independently verify.
A model that produces a summary you will read and evaluate consumes a small amount of trust, because a human will re-check the output. A model that produces a command that will be executed consumes more, because the verification window is narrower. A model that produces a decision that will be enforced without review consumes the most, because there is no independent check.
Training data opacity argues for spending the trust budget conservatively. When you cannot trace a model's claim to a verifiable source, the appropriate response is to treat the claim as a hypothesis rather than a fact. The hypothesis can be tested against other sources, against a deterministic checker, or against a human expert. The test is the thing that converts statistical output into dependable action.
What transparency would actually require
It is worth being specific about what would have to change for training data opacity to stop being a structural limit. A usable transparency standard would require, at minimum, a cryptographic hash of the training snapshot, a published filtering pipeline with auditable code, a mechanism for third parties to check whether a given document was included, and contractual commitments about what the data did and did not contain. Some of these are technically possible today. None of them are commercially standard.
Until they are, the security implication is simple. Frontier models should be treated as powerful but opaque components, and the controls around them should assume that their outputs carry the risk of their training data, even when the specific contents of that data are not visible. This is not an argument against using them. It is an argument for using them with the same care that any opaque component deserves.