In July 2024, security researchers at JFrog were routinely scanning public model repositories when they found something alarming: roughly 100 malicious Hugging Face models sitting in plain sight, downloadable by anyone running a training pipeline or a quick inference demo. One model, when loaded, silently opened a reverse shell to an IP address registered in Korea. Another phoned home the moment a data scientist called a single line of code: from_pretrained(). No exploit chain, no privilege escalation — just a pickle file doing exactly what pickle files are designed to do: execute arbitrary Python during deserialization. Model hubs have become the new package registry, and like npm and PyPI before them, they're now a favorite drop zone for attackers who know that "download and run" is the default developer workflow. This is the story of how that trust gets abused, and what it takes to catch it before it reaches production.
What makes malicious Hugging Face models so hard to spot?
They're hard to spot because a model file looks nothing like a script, even when it behaves like one. Most pretrained models are distributed as Python pickle (.pkl or .bin) archives, and pickle's __reduce__ method lets an attacker embed arbitrary function calls that fire the instant the file is unpickled — before a single weight has been used for inference. Hugging Face's own scanner, Picklescan, checks for known-dangerous opcodes and flagged globals, but in February 2025 ReversingLabs disclosed two models — glockr1/ballr7 and a second repository named after a string of zeros — that used a technique researchers dubbed "nullifAI." The uploaders compressed the pickle payload in a way that intentionally broke standard parsing, so Picklescan errored out and skipped scanning entirely, while Python's more permissive loader happily executed the embedded reverse-shell code anyway. The files sat on the hub, downloadable, for weeks before takedown. Static scanning catches the obvious cases; it does not catch a file engineered specifically to defeat the scanner.
How do trojan pretrained model payloads actually execute code on your machine?
They execute through the loading step itself, not through anything you'd call "running the model." A trojan pretrained model doesn't need you to run inference — it just needs you to load it. Calling torch.load() on a poisoned checkpoint, or AutoModel.from_pretrained() on a repo that ships a malicious config.json or tokenizer pickle, triggers deserialization, and deserialization is where the payload lives. Trail of Bits demonstrated a related technique in mid-2024 called "Sleepy Pickle," which goes a step further than dropping a shell: it patches the model's own bytecode during unpickling so the model itself is quietly altered post-load — skewing outputs, exfiltrating input data, or inserting backdoor triggers — while the weights on disk look completely untouched by any file hash check. That matters because a checksum comparison against a "known good" model won't catch a payload that only manifests in memory after deserialization. The attack surface isn't the neural network; it's the loading code wrapped around it, which is a much older and much less scrutinized piece of the pipeline.
Why can't automated hub scanners catch every unsafe model download?
They can't because scanning for known bad patterns will always trail behind attackers inventing new ones. Hugging Face now hosts well over 1.5 million public models, runs Picklescan and malware scanning on every upload, and has pushed hard to promote the safetensors format, which stores only tensor data with no executable opcodes and can't trigger code execution on load. That's a genuinely good mitigation — but adoption is voluntary, and an unsafe model download is still one search result away for anyone who filters by task rather than by file format. Attackers also exploit the softer parts of a repo that scanners weight less heavily: a custom modeling_*.py file marked trust_remote_code=True, a tokenizer config with an embedded script, or a dataset loader script bundled alongside the weights. Each of those is a legitimate, common pattern in the Hugging Face ecosystem, which is exactly why it's a good hiding spot. A scanner tuned to catch yesterday's pickle trick has no opinion about a remote-code-execution flag a user checks without reading the warning.
Which real-world incidents prove model hub security is not optional?
The JFrog and ReversingLabs findings above are the headline cases, but they're not isolated. Protect AI's huntr bug bounty program, which pays researchers specifically to find flaws in ML artifacts and tooling, has logged thousands of submissions across model-loading libraries, serialization formats, and inference servers since 2023, several rated critical for remote code execution. Researchers have also shown that typosquatting works on model hubs the same way it works on npm: uploading a repo named one character off from a popular fine-tune (a "trojan" version of a legitimate model) to catch developers who mistype or copy a stale tutorial link. None of this requires a sophisticated zero-day. It requires a hub with weak default trust, a developer under deadline pressure, and one line of copy-pasted from_pretrained code. Model hub security failures don't look like traditional breaches — there's no perimeter to cross — they look like a pip install-adjacent action that a data scientist performs a dozen times a day without a second thought.
What should teams do before pulling a model into production?
Teams should treat every third-party model exactly like third-party code, because that's precisely what it is. Concretely: prefer safetensors over pickle-based formats wherever an equivalent exists, and refuse repos that only ship .bin/.pkl checkpoints without a stated reason. Never set trust_remote_code=True reflexively — it hands the repo author permission to run arbitrary Python in your environment, and most integrations work fine without it. Pin model revisions to a specific commit hash rather than a mutable tag like main, since a hub account can be compromised or a maintainer can push a malicious update after the fact — the same lesson the software supply chain learned from dependency confusion attacks. Verify publisher identity and download counts as weak but useful signals, and run any new model artifact through a dedicated scanner in an isolated sandbox before it ever touches a training or inference pipeline with network access or credentials nearby. None of these steps are exotic; they're the same due diligence already applied to open-source libraries, just extended to a file type that most security tooling still doesn't inspect.
How Safeguard Helps
Safeguard treats pretrained models as first-class software supply chain artifacts, not as inert data files that get a pass because they end in .bin or .safetensors. That means every model pulled into a pipeline under Safeguard's watch gets provenance verification against its source hub and commit hash, static and behavioral analysis of serialization formats to catch both known pickle exploits and structurally malformed files built to dodge upstream scanners like Picklescan, and policy enforcement that blocks trust_remote_code and unpinned revisions by default rather than by exception. Safeguard's scanning pipeline specifically watches for the patterns behind incidents like the JFrog and nullifAI findings — reverse-shell opcodes, suspicious network calls embedded in deserialization logic, and mismatched file signatures that suggest a payload hiding behind a broken scanner. For teams building on top of the Hugging Face ecosystem or any other public model hub, Safeguard turns "download and hope" into an auditable, enforced step in the pipeline — so a trojan pretrained model gets caught at the gate, not discovered after it's already talking to an attacker's server in production.