In February 2024, Google DeepMind researchers extracted the full embedding projection layer of OpenAI's production ada and babbage models for less than $20 in API queries — no breach, no leaked credentials, just carefully crafted prompts and basic linear algebra. A year earlier, Meta's LLaMA weights leaked onto 4chan within a week of a restricted research release, and within days Stanford researchers had fine-tuned a clone (Alpaca) for about $500 using outputs scraped from another company's API. These aren't edge cases. They're proof that a model trained for tens or hundreds of millions of dollars can be functionally reconstructed by anyone with API access, a laptop, and a few hundred dollars of compute. For companies whose entire moat is a proprietary model, extraction attacks are now a board-level supply chain risk — and most organizations have no controls in place to detect them.
What Is LLM Model Extraction, and Why Is It a Supply Chain Problem?
Model extraction is an attack in which an adversary reconstructs a proprietary model's weights, architecture, or decision behavior by systematically querying its public API, without ever touching the underlying files. The term originates from Florian Tramèr et al.'s 2016 paper "Stealing Machine Learning Models via Prediction APIs," which showed that logistic regression and decision tree models exposed through simple prediction APIs could be near-perfectly cloned for pennies using a few thousand queries. LLMs raise the stakes: a single frontier model can represent nine figures of training investment, and its "API surface" — the only thing exposed to the outside world — is precisely what extraction attacks target.
This makes model theft a software supply chain issue, not just an ML research curiosity. The model artifact (weights, embeddings, fine-tuning data, system prompts) is a build output with provenance, access controls, and a distribution pipeline, exactly like a compiled binary or container image. If that pipeline lacks integrity controls — signed artifacts, access logging, rate-limited egress — the model is exposed the same way an unsigned npm package or an exposed CI runner is: quietly, and often for months before anyone notices.
How Do Attackers Actually Steal a Model Without Touching Its Files?
Attackers extract models through four main techniques: query-based functional cloning, embedding/logit extraction, knowledge distillation, and insider or infrastructure-level theft. Query-based cloning sends thousands to millions of structured prompts to a target API and trains a substitute model on the input-output pairs — this is how Alpaca (52,000 instruction pairs, ~$500 in OpenAI credits) and Vicuna (70,000 scraped ShareGPT conversations, ~$300 in compute) were built in March 2023.
Embedding and logit extraction is more surgical: the DeepMind team's February 2024 paper "Stealing Part of a Production Language Model" showed that if an API exposes logit bias or top-log-probability fields, an attacker can mathematically recover the exact hidden dimension size and embedding projection matrix of the model — for gpt-3.5-turbo, they estimated the same attack would cost under $2,000 and reveal proprietary architecture details OpenAI had never disclosed. Distillation attacks are subtler still: a competitor doesn't need weights at all, just enough high-quality outputs to train a smaller model that mimics the original's behavior at a fraction of the cost. Finally, insider and infrastructure theft — a departing engineer copying checkpoint files, an S3 bucket without access logging, an unpatched model registry — remains the highest-yield and lowest-effort route, requiring no clever math at all.
Has This Actually Happened to Real Companies?
Yes, repeatedly, and the pace is accelerating. In March 2023, Meta released LLaMA weights to approved researchers only; within a week the full weight files were posted to 4chan and torrented widely, forcing Meta to confront uncontrolled redistribution of a model it had spent months training. In late 2023 and through 2024, multiple open-source projects (Alpaca, Vicuna, Koala) demonstrated that instruction-tuned clones of commercial models could be built for under $1,000 using nothing but API access — a pattern OpenAI's terms of service explicitly prohibit but has struggled to enforce technically.
The most consequential recent case broke in December 2024 and January 2025: Microsoft security researchers and OpenAI stated they had detected accounts, reportedly linked to Chinese AI lab DeepSeek, exfiltrating unusually large volumes of output data from OpenAI's API in patterns consistent with distillation — using another company's model outputs to train a competing model without a training relationship. Bloomberg and other outlets reported OpenAI was investigating whether DeepSeek's rapid, low-cost model development (DeepSeek reported a $5.6 million training run for DeepSeek-V3) was partly enabled by this kind of extraction. Whatever the final legal determination, the episode showed every AI vendor with a public API that extraction isn't theoretical — it's an active, ongoing threat vector being used against the industry's largest labs right now.
What Does a Stolen Model Actually Cost the Original Builder?
The cost is the training investment itself, plus the competitive advantage that investment was supposed to buy. Sam Altman has publicly estimated GPT-4-class training runs at "more than $100 million," and industry estimates for the largest 2024-2025 frontier runs range from $100 million to over $1 billion when including compute, data curation, and RLHF labor. Extraction attacks let an adversary skip nearly all of that: DeepMind's researchers reconstructed proprietary architecture details for under $20; Alpaca-style distillation reproduces instruction-following behavior for a few hundred dollars. When a competitor can approximate 80-90% of your model's capability for 0.01% of your R&D spend, your pricing power, fundraising narrative, and competitive lead all erode simultaneously — and unlike a data breach, there's often no clear "incident" to point to, no ransom note, just a suspiciously capable competitor product a few months later.
How Can Organizations Detect and Prevent Extraction Attacks?
Detection and prevention require API-level controls plus supply chain controls on the model artifact itself, because extraction happens through the interface you intentionally expose. On the API side: rate-limit and fingerprint query patterns (extraction attacks typically require tens of thousands of systematic, low-diversity queries that look nothing like normal usage), strip or round raw logits and log-probabilities from responses since these are the exact fields the DeepMind attack exploited, inject output watermarking so distilled downstream models carry a detectable signature, and enforce per-key and per-org usage caps tied to anomaly detection rather than flat rate limits alone.
On the artifact side, treat model weights, checkpoints, and fine-tuning datasets as controlled supply chain assets: sign and hash every checkpoint the same way you'd sign a container image, maintain an access log and ML-BOM (model bill of materials) documenting who pulled which version and when, and restrict registry and bucket access with the same least-privilege discipline applied to production secrets. Contractually, enforceable terms of service prohibiting output-based training (as OpenAI's do) provide a legal backstop, but they only work if you can technically detect the violation in the first place — which is why monitoring, not policy language, is the control that actually stops extraction in progress.
How Safeguard Helps
Safeguard treats your model artifacts, inference endpoints, and ML pipelines as first-class supply chain assets, not a separate problem from your code and container security. We help teams close the gaps that extraction attacks exploit: continuous monitoring of model registries and storage buckets for anomalous access or bulk-download patterns, signed provenance for every model checkpoint and fine-tuning dataset so you always know which version is deployed and who touched it, and API-layer anomaly detection tuned to the query patterns that distinguish real users from systematic extraction attempts.
Because model theft rarely happens through a single dramatic breach, Safeguard's approach focuses on the supply chain hygiene that catches it early: enforcing least-privilege access to weights and training data, flagging unsigned or unverified model artifacts before they reach production, and giving security teams visibility into inference API usage alongside their existing SAST/DAST and dependency findings — one pipeline, one risk picture, instead of your ML infrastructure living in a blind spot next to the rest of your software supply chain. If your competitive advantage lives in a model, it deserves the same integrity controls you already apply to the code that ships it.