Fine-tuning a base model is attractive because it lets you specialize a capable general model for your domain at a fraction of the cost of training from scratch. It is also a supply chain operation: you are combining an upstream artifact (the base) with your own data and producing a new artifact (the fine-tune) that inherits the properties of both. Poisoning at either end shows up in the resulting model, and detection is harder than in traditional code supply chains because the artifact is a blob of weights. This post covers where poisoning enters fine-tuning pipelines and what detection actually looks like.
What are the attack paths into a fine-tune?
Three main ones. First, the base model itself is poisoned, such that a trigger phrase produces attacker-chosen output. Fine-tuning on benign data typically does not remove such backdoors; research has shown repeatedly that they survive significant additional training. Second, the fine-tuning dataset is poisoned, either with label-flipping attacks, with backdoor triggers baked into a small fraction of examples, or with content that biases the model toward attacker-preferred outputs. Third, the fine-tuning infrastructure itself is compromised: training code, data pipelines, or checkpoint storage.
Each attack path has a different detection story. Base model poisoning requires probing the base for known or suspected triggers before fine-tuning. Dataset poisoning requires dataset-level integrity and content analysis. Infrastructure poisoning requires the same supply chain hygiene you would apply to any production training system, which most teams under-invest in because training runs feel like one-off events rather than production operations.
How do you detect poisoning in the base model?
You probe it, and you compare against a reference. A poisoned base model typically produces abnormal outputs on specific trigger inputs while behaving normally otherwise. The detection challenge is that the trigger space is huge and the attacker chose it. You cannot enumerate. What you can do is run the base model on a standardized evaluation set that includes known adversarial patterns and red-team-contributed trigger candidates, and compare the outputs against a reference copy of the model (ideally one with independent provenance).
For popular open models, community efforts have been building these probe sets. They are imperfect but they catch crude attacks. For models coming from less-vetted sources (Hugging Face has had a handful of incidents where a well-named repo served a modified model), probing plus weight-hash verification against the publisher's canonical release is the minimum bar. Skipping the hash check and relying on the repo name is what caused several of the 2025 incidents that made the news.
The other useful signal: compare weight statistics against a reference. Poisoning a model to embed a backdoor usually leaves detectable statistical fingerprints in specific layers. The detection is not perfect, but it is another layer of defense that is cheap to run.
What does dataset poisoning detection look like?
Integrity on the data itself, then content analysis. Integrity is the boring but essential part: hash every file in the training dataset, record the hashes in your pipeline config, and verify them on every training run. If a dataset file changes between when it was approved and when training runs, you want to know. A shockingly common failure is "someone updated the dataset directly in the bucket and did not tell anyone," which is both a security and a reproducibility problem.
Content analysis is the harder part. For supervised fine-tuning datasets, look for examples that have anomalous length, unusual token distributions, or targeted patterns that correlate with specific labels. For instruction-tuning datasets, look for instruction/response pairs where the response seems to ignore the instruction or includes content unrelated to the prompt; these are often backdoor payloads. For preference-tuning datasets, look for preference pairs that create unusual optimization pressure toward specific behaviors.
Duplication across the dataset is also worth checking. A small number of near-duplicate examples with a specific trigger token is the classic backdoor shape. Deduplication with a Jaccard or embedding-based threshold will catch most of these.
How do you validate the fine-tune after training?
Behavior diffing against the base model, on a broad evaluation set that the attacker did not get to see. The idea is to identify inputs where the fine-tune's behavior diverges significantly from the base's and ask whether the divergence is explained by your intended training objective or is anomalous. If your fine-tune is meant to specialize a model for medical question answering and it shows large divergence on inputs about completely unrelated topics, that divergence merits inspection.
Automated divergence detection is not a solved problem. A reasonable approximation: generate outputs from base and fine-tune on a held-out evaluation set, compute some distance metric (embedding-based similarity, perplexity ratio, ROUGE between outputs), and flag outliers. Feed the flagged cases to a human reviewer. This is not bulletproof but it catches gross changes, and gross changes are what most poisoning attacks produce.
The deeper check is red-teaming the fine-tune against specific threat hypotheses: "does this model produce attacker-aligned outputs on inputs that mention competitor X," "does this model inject specific dependencies in code generation," "does this model leak training data on specific prompts." A fine-tune that is going into a production path in a sensitive domain should get this level of scrutiny. The cost is real but far less than the cost of shipping a poisoned model.
How should pipeline integrity be enforced?
Signed checkpoints, attested training runs, and locked-down training infrastructure. A training run that cannot produce a signed attestation chaining back to specific dataset hashes, specific code commits, and a specific compute environment is a run whose provenance you cannot verify later. SLSA-style attestation for model training is still maturing, but the core primitives (reproducible builds applied to training, signed artifacts, audit logs) are available today.
Training infrastructure security in 2026 still lags code-build infrastructure security, even at serious companies. The same compromise patterns that hit CI/CD systems a decade ago (persistent tokens, shared build runners, no separation between build and deployment credentials) are common in ML training pipelines. A compromise of the training infrastructure is a compromise of every model produced by it, which is a blast radius worth investing against.
What about fine-tuning APIs?
When you fine-tune through a provider (OpenAI, Anthropic, an open-source platform), you are trusting the provider's supply chain end to end. You are trusting that they are not poisoning your model, that their data handling does not expose your training data, and that their infrastructure is not compromised. These are reasonable trusts for reputable providers but they should be made explicit in your risk register, not implicit. For regulated industries, the provider's SOC 2 or equivalent attestations, plus contractual language about training data usage, are the baseline.
For open-source platforms or community tooling, the trust surface is wider and the attestations are thinner. A fine-tuning pipeline built on packages like transformers, peft, trl, and their dependencies is a supply chain with all the properties of any other Python supply chain. It needs SCA, it needs reachability analysis, and it needs the same dependency monitoring that your backend services have. Teams frequently treat it as research infrastructure and give it a pass, which is how incidents like the Ultralytics PyPI compromise reach training environments.
How Safeguard.sh Helps
Safeguard.sh's reachability analysis extends across both the code and model-component dependency trees of fine-tuning pipelines, cutting 60 to 80 percent of the alert noise that would otherwise drown teams in false positives on packages like transformers and peft. Griffin AI flags anomalies in base-model provenance and dataset integrity, correlating findings across the SBOM so that a compromised base or a tampered dataset surfaces alongside traditional CVE signal. TPRM workflows track model vendors and fine-tuning platforms, and the 100-level dependency depth ensures transitive compromises in the Python ML ecosystem remain visible. Container self-healing keeps training and inference images patched automatically so fine-tuning infrastructure does not drift behind the current security posture.