How small language models are built, shrunk and run: a deep guide

For the short version, see Small language models: when a small model is enough. This post is the in-depth reference.

A small language model (SLM) is a language model built to run with far less memory and compute than a frontier model, usually a few hundred million to a few billion parameters, so it fits on a laptop, a phone, an edge box or one modest GPU. There is no official size cutoff. SLMs get small through careful data choices, knowledge distillation, pruning and quantization, and they trade breadth of knowledge for lower cost, lower latency, privacy and the ability to run with no network.

Below: how small models are made, the memory math, when to choose one, the families on offer as of 2 November 2026, and the security problems of running a stranger's quantized file on your own hardware.

Key takeaways

• "Small" is a deployment label, not a standard. In the sources below, makers apply it to models from about 135 million to about 14 billion parameters.
• Four levers do most of the shrinking: better training data, knowledge distillation, pruning and quantization to 8, 4 or even 2 bits per weight.
• Weight memory is parameters times bits per weight divided by 8. A 3.8 billion parameter model needs about 7.6 GB at 16 bits and about 2.3 GB at llama.cpp's Q4_K_M. KV cache and runtime overhead come on top.
• A community GGUF conversion is a different artifact from the maker's release. Its weights, tokenizer and embedded chat template all need provenance, hashes and an AI-BOM entry.
• On-device is not risk-free: prompt injection, local data and runtime CVEs still apply.

This series

• LLMs explained: how large language models work, from transformers to agents and security (the hub)
• Small language models explained (this post)
• Vision language models explained: image encoders, document AI, GUI agents and security (publishes Friday 6 November)
• Audio language models explained: speech encoders, audio tokens, voice agents and security (publishes Saturday 7 November)

What counts as a small language model?

No standards body defines the term. Makers and researchers use it relative to the hardware they target:

• Meta's MobileLLM paper (February 2024) focuses on models with "fewer than a billion parameters" for mobile use.
• Microsoft's Phi-3 technical report (April 2024) calls phi-3-mini, at 3.8 billion parameters, "small enough to be deployed on a phone."
• Google's Gemma 4 card targets E2B and E4B at "mobile and edge devices" and 12B, 26B A4B and 31B at "consumer GPUs and workstations."
• Mistral's card says the Ministral 3 family (3B, 8B and 14B) is "designed for edge deployment."
• Hugging Face's SmolLM2 family starts at 135 million parameters.

A working definition: an SLM is one its maker built to run on a single consumer device or modest accelerator, roughly 100 million to 15 billion parameters, with 1B to 4B the busiest band.

Raw counts mislead. Mixture-of-experts models have a total and a much smaller active count: Gemma 4 26B A4B has 25.2 billion total and 3.8 billion active parameters, per its card. Gemma 4 E2B is 2.3 billion "effective" parameters and 5.1 billion including embeddings, because its per-layer embeddings are designed for on-device efficiency. Compare total parameters for storage and active parameters for compute per token.

Modality is a separate axis: a small model can be text-only (Phi-4-mini-instruct), carry a vision encoder (Qwen3.5) or take audio too (Gemma 4 E2B). The VLM and ALM posts cover those encoders.

Why small models now

Three things changed between 2023 and 2026.

Data quality proved to matter a great deal at small scale. Microsoft's "Textbooks Are All You Need" (June 2023) trained phi-1, a 1.3 billion parameter code model, on about 6 billion tokens of "textbook quality" web data plus about 1 billion tokens of synthetic textbooks and exercises generated with GPT-3.5. The Phi-3 report scaled the idea to 3.3 trillion tokens of heavily filtered web data and synthetic data.

Compression matured: GPTQ and AWQ made 4-bit weights practical, and NVIDIA's Minitron work derived smaller models from a big one with a fraction of the data.

Runtimes and hardware arrived. Apple's Foundation Models framework gives apps direct access to its on-device model, Android exposes Gemini Nano through AICore, and llama.cpp, ONNX Runtime, MLC LLM and Ollama run quantized models on ordinary laptops. Our tile-based GPU explainer covers the mobile chips underneath.

How models get small

Knowledge distillation

Distillation trains a small "student" to match a larger "teacher." Hinton, Vinyals and Dean's 2015 paper framed it as compressing a large model or ensemble into one easier to deploy. The student trains on the teacher's full output distribution, softened with a temperature, not just the correct label, because the probabilities the teacher gives to wrong answers show how it generalizes.

DistilBERT (October 2019) applied distillation during pretraining and reported a model 40 percent smaller than BERT that kept 97 percent of its language understanding and ran 60 percent faster, using a triple loss of language modelling, distillation and cosine distance.

For generative models, distillation takes three common forms:

• Logit distillation: match the teacher's next-token distributions (needs its logits and a compatible tokenizer).
• Synthetic-data distillation: train on teacher-generated text, as with phi-1's textbooks.
• Prune then distill: NVIDIA's "Compact Language Models via Pruning and Knowledge Distillation" (July 2024) prunes depth, width, attention and MLP dimensions, then retrains with distillation on less than 3 percent of the original data. The authors compressed Nemotron-4 by 2 to 4 times and derived 8B and 4B models.

A note for later: a student inherits its teacher's behaviour. Distillation is not a cleaning step for a poisoned teacher.

Pruning

Unstructured pruning zeroes single weights; structured pruning removes rows, heads, channels or layers, which saves memory and time on ordinary hardware. SparseGPT (January 2023) pruned GPT-family models to at least 50 percent sparsity in one shot without retraining, at small accuracy loss, and supports semi-structured 2:4 and 4:8 patterns that some accelerators skip in hardware. Structured pruning usually needs a short distillation phase to recover quality.

llama.cpp's quantize README even documents a --prune-layers option, so a downstream packager, not only the maker, can remove layers. Another reason to know exactly which file you run.

Quantization in depth

Quantization stores weights, and sometimes activations, in fewer bits than the 16-bit floats used in training. Our posts on the cheapest and fastest LLM inference treat it as a serving lever. Primers: what is GGUF and quantization and quantization explained: 4-bit models. Here is how the methods work.

The mapping. ONNX Runtime's docs give the linear form val_fp32 = scale * (val_quantized - zero_point). Scale and zero point can be set per tensor, per channel or per block. Finer blocks cost a little memory for the extra scales and cut error.

Dynamic versus static. Dynamic quantization computes activation parameters at inference time; static uses calibration data ahead of time. Weight-only quantization, common for SLMs, leaves activations in higher precision.

LLM.int8(). This August 2022 paper found that transformers develop systematic outlier features that break naive 8-bit quantization. Its fix combines vector-wise quantization with a mixed-precision decomposition that keeps outlier dimensions in 16-bit, halving inference memory without performance loss, per the authors.

GPTQ. GPTQ (October 2022) is one-shot weight quantization using approximate second-order information to choose values that minimize each layer's output error. The paper reports quantizing 175 billion parameter models in about four GPU hours to 3 or 4 bits with negligible accuracy loss.

AWQ. AWQ (June 2023) observes that protecting about 1 percent of salient weights greatly reduces error, and that you find those channels from activation magnitudes, not weight values. Rather than mixed precision, it scales salient channels before quantization. The paper targets on-device LLMs explicitly.

Quantization-aware training. QAT simulates low precision during training. Apple says it compressed its on-device model to 2 bits per weight with QAT. Google publishes Gemma 4 QAT checkpoints, including q4_0 GGUF builds. Meta's Llama 3.2 card lists SpinQuant and QLoRA quantized 1B and 3B variants with an 8k context, against 128k for the full-precision models.

GGUF and k-quants. GGUF is the llama.cpp and GGML file format. Its spec lists single-file deployment, mmap loading and an extensible key-value metadata structure as design goals, and tensors in one file can use different quantization types. The llama.cpp quantize README lists measured bits per weight:

"Q4" is not 4 bits: block scales and the mixed "_M" recipe, which keeps some sensitive tensors at higher precision, put Q4_K_M near 4.9. The README also documents an importance matrix (--imatrix) computed from sample text to guide the quantizer. That text is a build input worth recording. The README's own example:

# From the llama.cpp quantize README
./build/bin/llama-quantize gemma-4-E2B-it-bf16.gguf gemma-4-E2B-it-Q4_K_M.gguf Q4_K_M

ONNX Int4. ONNX Runtime quantizes MatMul and Gather to 4-bit with block-wise, weight-only quantization; with the QOperator format a MatMul becomes a MatMulNBits node. It supports RTN (the default), GPTQ and HQQ. Makers ship ONNX builds directly: Microsoft for Phi-4-mini, Mistral for Ministral 3 3B and Hugging Face for SmolLM3.

LoRA and QLoRA

LoRA (June 2021) freezes the pretrained weights and trains two small matrices per adapted layer: for a d by k weight, B is d by r and A is r by k, with a small rank r, and the effective weight is W plus BA. The paper reports that, against full fine-tuning of GPT-3 175B, LoRA cut trainable parameters by 10,000 times and GPU memory by 3 times.

A worked number: Phi-4-mini's config lists a hidden size of 3,072. A 3,072 by 3,072 projection has 9,437,184 weights. A rank 16 adapter adds 16 times (3,072 plus 3,072), or 98,304 weights, about 1 percent.

QLoRA (May 2023) backpropagates through a frozen 4-bit base into LoRA adapters, using the 4-bit NormalFloat (NF4) type, double quantization and paged optimizers, and reported fine-tuning a 65 billion parameter model on one 48 GB GPU. On a small model it fits a single consumer GPU.

The consequence: anyone can publish a specialized variant of a popular SLM in an afternoon, as an adapter or as a merged, quantized file.

Training data for small models

The makers' own papers and cards describe three patterns. Filtered, "reasoning dense" web data (Phi-3 report, Phi-4-mini-instruct card). Synthetic "textbook-like" data for math, coding, common sense and general knowledge (same card). And architecture tuned for size: MobileLLM uses deep and thin networks, embedding sharing and grouped-query attention, and Phi-4-mini lists grouped-query attention, a 200K vocabulary and shared input and output embeddings.

The cards are candid. Microsoft says Phi-4-mini-instruct "is still fundamentally limited by its size for certain tasks" and "simply does not have the capacity to store too much factual knowledge." Qwen says Qwen3.5-0.8B's intended uses are prototyping, task-specific fine-tuning and research. Read those before building a factual question-answering product.

On-device, edge and air-gapped: where small models run

Apple. Apple's foundation models update (June 2025, revised July 2025) describes the on-device model as about 3 billion parameters at 2 bits per weight, accessible to apps through the Foundation Models framework. Apple split the model into two blocks at a 5:3 depth ratio and shares block 2's KV caches with the last layer of block 1, which it says cuts KV cache memory by 37.5 percent. Apps call the framework; they do not ship weights.

Android. Google's docs say Gemini Nano runs in the AICore system service, which uses device hardware for low latency, keeps the model up to date and has built-in safety features; ML Kit's GenAI APIs build on it, and generation works without a network connection or sending data to the cloud.

Laptops, desktops and servers. Here you choose and ship the weights:

• llama.cpp runs GGUF on CPUs and GPUs.
• Ollama imports a model from a Modelfile whose FROM line points at a GGUF file or a safetensors directory, per its import docs.
• ONNX Runtime runs ONNX graphs across desktop, mobile and the web.
• MLC LLM compiles models for native apps; its docs list iOS, Android and WebLLM SDKs.

# From Ollama's import documentation
FROM /path/to/file.gguf

Air-gapped sites. Inside an isolated network no API is reachable and GPUs are scarce, so small models are often the only option, and every file must cross the gap through a signed, hashed, inventoried process. See our sovereign enterprise guide to an air-gapped software supply chain platform and the air-gapped deployment FAQ.

How much memory does a small model need?

Weights. Parameter count times bits per weight, divided by 8. Use effective bits per weight from the table above, not the number in the name.

KV cache. Per token: 2 (keys and values) times layers times key-value heads times head dimension times bytes per element. Multiply by tokens in context and by concurrent sequences.

Overhead. Activations, the runtime, any vision or audio encoder and everything else on the device.

Worked example: Phi-4-mini-instruct

Phi-4-mini-instruct has 3.8 billion parameters (model card). Its config lists 32 layers, 24 attention heads, 8 key-value heads and a hidden size of 3,072, so 128 dimensions per head.

KV cache at 16-bit: 2 x 32 x 8 x 128 x 2 bytes = 131,072 bytes, or 128 KiB per token. That is 512 MiB at 4,096 tokens, 4 GiB at 32,768 tokens and 16 GiB at the full 131,072 token context in the config.

For a rule of thumb version see how much VRAM to run a local LLM and Apple silicon unified memory for local LLMs. So the Q4_K_M weights fit in about 2.3 GB, but one long-context session can need several times that again. Runtimes can quantize the KV cache, and runtime overhead comes on top of both.

As a check, llama.cpp's README lists an 8B model at 4.9 GB in Q4_K_M, which the formula reproduces. For larger models, see how many GPUs a 1 trillion parameter model needs; for why memory bandwidth sets token speed, see our GPU memory and interconnects explainer.

Cost, latency and privacy: SLM versus LLM

Per-token economics by chip are in LLM inference cost by chip and model size.

When an SLM beats an LLM, and when it does not

Small models win when the task is narrow, the output is checkable and the deployment constraint is hard (data that cannot leave a site, no network, tight latency). Large models win on open-ended knowledge and long multi-tool agent tasks, and cascades (SLM first, LLM on escalation) cut cost. The short primers Small language models: when a small model is enough and small language model security and use case fit cover the choice step by step. Test on your own task; the AI agent evaluation explainer shows how.

Small model families as of 2 November 2026

Facts below come only from each maker's own model card, documentation or research page, as of 2 November 2026. Sizes are as the maker states them. Order implies no ranking, and no benchmark scores are given.

The licence shown is what the maker states; a repackaged copy may drop it. These families ship new generations often, so recheck before planning around a size.

Securing small language models

With an API model, the provider owns the weights, the runtime and most patching. With an SLM you download a file, pick a runtime, maybe apply an adapter and run it on hardware you own. Each piece is now yours. OWASP's Top 10 for LLM Applications 2025 lists LLM03:2025 Supply Chain and LLM04:2025 Data and Model Poisoning for this reason.

Model files and formats

• Pickle-based checkpoints (.bin, .pt, .pth, .ckpt). Python's docs warn that pickle "is not secure" and that "it is possible to construct malicious pickle data which will execute arbitrary code during unpickling." PyTorch's torch.load docs say it uses an unpickler and to never load data from an untrusted source; weights_only=True restricts what it accepts.
• Safetensors. Hugging Face describes it as a simple format for storing tensors safely, as opposed to pickle, with zero-copy loading and no code path on load.
• GGUF. Typed metadata plus tensors. No embedded Python runs on load, but the runtime's parser handles untrusted input, so keep it patched and pinned.
• ONNX. A graph format; risk sits in custom operators and runtime bugs.

The Hugging Face Hub shows a pickle import scan next to each pickled file, but that signal does not follow a copy to a mirror or cache. Prefer safetensors wherever you control the loader:

# From the safetensors documentation
from safetensors import safe_open

tensors = {}
with safe_open("model.safetensors", framework="pt", device="cpu") as f:
    for key in f.keys():
        tensors[key] = f.get_tensor(key)

Our safetensors vs pickle vs GGUF comparison has the full table.

Quantized files from strangers

For popular small models, most downloads are community conversions: someone converted the weights to GGUF, chose a quantization type, maybe used an importance matrix of their choice, maybe pruned layers, and uploaded it under a familiar name. That is a new build artifact whose integrity rests on the uploader.

1. Prefer the maker's own quantized or ONNX builds, which Google, Meta, Microsoft, Mistral, IBM and Hugging Face all publish.
2. For any other conversion, record the source revision, converter version, quantization type and importance matrix input.
3. Pin by content hash, not name, and check it on every load path.
4. For critical models, convert from the maker's safetensors yourself in an isolated environment.

sha256sum gemma-4-E2B-it-Q4_K_M.gguf > gemma-4-E2B-it-Q4_K_M.gguf.sha256
sha256sum -c gemma-4-E2B-it-Q4_K_M.gguf.sha256

GGUF metadata and chat templates

The GGUF spec defines tokenizer.chat_template as "a Jinja template that specifies the input format expected by the model." Many runtimes use it to turn chat messages into the exact token string the model sees.

So the template is part of the model's behaviour and supply chain: it places the system prompt, formats tool calls and separates roles. A modified template in a repackaged file changes those with no change to the weights. A hash of the original weights will not catch that; a hash of the whole GGUF file will. Extract and review the template before approval, supply a known-good template where your runtime allows, and compare metadata such as general.license and general.basename with the maker's card.

LoRA adapters as a supply chain

An adapter can be safetensors or a pickled .bin, so the format rules apply. Each targets one base model; record it and refuse others. Merged, quantized models look like ordinary files, and without provenance you cannot tell what was merged in.

Poisoned or backdoored weights

Three papers frame the risk at the level of mechanism:

• BadNets (August 2017) showed outsourced training can produce a backdoored network that performs well on the user's validation data but misbehaves on attacker-chosen inputs. Its subtitle names the problem: vulnerabilities in the machine learning model supply chain.
• Sleeper Agents (January 2024) trained proof-of-concept models whose behaviour changes on a trigger and found the backdoor could persist through supervised fine-tuning, reinforcement learning and adversarial training, most persistently in the largest models studied and in chain-of-thought models.
• Poisoning Web-Scale Training Datasets is Practical (February 2023) described split-view and frontrunning poisoning and estimated that 0.01 percent of LAION-400M or COYO-700M could have been poisoned for 60 US dollars.

Most teams do not pretrain SLMs; they fine-tune, merge and quantize them with others' data and adapters, and each step can introduce a trigger that clean benchmarks miss. Trusted sources, provenance and behavioural tests are the controls that scale.

Provenance and the AI-BOM

CycloneDX describes its ML-BOM as providing model and dataset transparency for security, privacy, safety and ethical considerations; a model is a component of type machine-learning-model. SPDX's AI profile captures models, training data, production data, prompts and agents as a connected graph. A minimal CycloneDX entry for a quantized file:

{
  "type": "machine-learning-model",
  "name": "gemma-4-E2B-it-Q4_K_M",
  "hashes": [{ "alg": "SHA-256", "content": "<sha256 of the gguf file>" }],
  "licenses": [{ "license": { "id": "Apache-2.0" } }],
  "externalReferences": [
    { "type": "distribution", "url": "https://huggingface.co/google/gemma-4-E2B-it" }
  ]
}

For SLMs, record the base model and revision, quantization type and tool, adapters and their base, the chat template hash, the runtime version and the licence, and link the maker's model card.

Prompt injection on small models

Small models are not immune, and with less safety post-training they may follow injected instructions more literally. OWASP lists LLM01:2025 Prompt Injection first. MITRE ATLAS catalogues adversary techniques against AI systems, and NIST AI 600-1, the Generative AI Profile published 26 July 2024, lists information security and data privacy among the risks to manage.

A local model often reaches files, email or the clipboard through its host app, so indirect injection through that content can make it act on the device. Keep tool permissions narrow and confirm actions with side effects. For guardrails around API models, see governing every LLM call with an AI gateway.

On-device data

"The data never leaves the device" moves the protection job onto the device. Prompts, outputs, histories and retrieval indexes need encryption, retention rules and device policy. Fine-tuned weights trained on internal data are themselves sensitive. When the OS owns the model (Apple's framework, Android's AICore), updates come with the OS; when your app ships the model, you own the update path, including fixes for runtime CVEs. Our AI and ML platform lead's guide covers the wider tooling.

Licence terms

General information, not legal advice; read the licence in the exact repository you download.

• Apache 2.0 (Gemma 4, Qwen3.5, Ministral 3, SmolLM3, Granite 4.2): permissive, with a patent grant and notice requirements.
• MIT (Phi-4-mini-instruct).
• Llama 3.2 Community License: if a licensee's products had more than 700 million monthly active users in the month before the release date, it must request a licence from Meta. Use is also bound by Meta's Acceptable Use Policy.
• Gemma Terms of Use (last modified 1 April 2026) cover earlier Gemma models listed in its appendix; Gemma 4 has its own Apache 2.0 licence page.

Self-hosting versus API: security trade-offs

Where Safeguard fits

Safeguard's role here is the model supply chain and deployment choices, not picking which small model answers best.

Model file scanning and inventory. Safeguard connects to Hugging Face, Kaggle Models, ModelScope and Civitai through read-only integrations, scans the files of the models you select for unsafe serialization formats and embedded code with AI-SPM model file scanning, and records each model in your AI inventory through AI-BOM discovery, with its licence or permissions. The model scanning post walks through each hub.

Model sources. Gold Open Source (gold.safeguard.sh) gives free CVE intelligence with affected and fixed versions, and its directory covers models too; see the Gold Open Source explainer and sourcing models from Hugging Face safely.

Runtimes. llama.cpp, ONNX Runtime, inference servers and tokenizer libraries are open source dependencies; Safeguard scans repositories, container images and SBOM files for vulnerable versions, and CI gates can stop them shipping.

Guardrails. Safeguard AI Gateway sits between your application and the model and evaluates prompts, responses and tool calls for direct and indirect prompt injection, jailbreak patterns and sensitive data in either direction, in monitor mode by default with a fail-open design.

Model choice and air-gapped deployment. Safeguard runs as SaaS in US, EU (Frankfurt) and Gov regions, in a private VPC, on-prem or air-gapped. The AI mode is a separate per-tenant choice: Full AI with Safeguard's own models, Zero AI with no LLM in the loop, bring your own model, or local models inside your environment. Safeguard's air-gapped documentation says its own models run there as quantized builds on customer GPUs, with CPU-only quantized builds for sites that cannot spare GPU time. Griffin, Safeguard's security model, is behind its remediation pull requests; Safeguard's models are single-purpose security models. Customer code, SBOMs and telemetry are not used to train shared models. Details are in where your code and data go with Safeguard.

Safeguard is SOC 2 Type II compliant; reports are available on request. See safeguard.sh/pricing for current plans.

Common mistakes

1. Choosing by parameter count alone. Total, active and effective parameters measure different things.
2. Forgetting the KV cache. A model that fits at 2.3 GB can need many more gigabytes at long context.
3. Assuming Q4 means 4 bits. Q4_K_M is about 4.9 bits per weight.
4. Running the most downloaded GGUF without provenance. Popular is not the same as the maker's.
5. Ignoring the embedded chat template. It shapes every prompt. Review and pin it.
6. Calling torch.load on untrusted files. The current docs show weights_only=True as the default and still say never to load data from an untrusted source; older versions and callers that set it to False get no such restriction.
7. Treating adapters as configuration. They change behaviour; inventory them.
8. Skipping evaluation after quantization. Re-test the exact file you ship.
9. Losing licence terms on repackaged files. The base model's licence still applies.

FAQ

What is the difference between an SLM and an LLM?

Size and deployment target, not architecture. Both are usually decoder-only transformers. An SLM has far fewer parameters, so it runs on a phone, laptop or one modest GPU, with less stored knowledge and weaker long reasoning.

How many parameters does a small language model have?

There is no fixed cutoff. In the sources here, makers use "small" or "edge" for models from about 135 million to about 14 billion parameters, with most phone and laptop models between 1 and 4 billion.

Can I run a small language model on my laptop?

Usually. A 3 to 4 billion parameter model at about 4.9 bits per weight needs roughly 2 to 2.5 GB for weights, plus KV cache and overhead, and llama.cpp, Ollama or ONNX Runtime will run it.

What is quantization in LLMs?

Storing weights, and sometimes activations, in fewer bits, such as 8 or 4 instead of 16. LLM.int8(), GPTQ, AWQ, GGUF k-quants and ONNX Int4 cut memory and often raise speed, with accuracy loss that grows as bits fall.

What is knowledge distillation?

Training a small student model to imitate a larger teacher's softened output probabilities or generated text, as formalized by Hinton, Vinyals and Dean in 2015.

Is GGUF safer than pickle?

Loading GGUF does not execute embedded Python, which removes the main pickle risk. The runtime's parser still handles untrusted input, and the metadata, including the chat template, shapes every prompt. Verify hashes and patch the runtime.

What is the difference between LoRA and QLoRA?

LoRA trains small low-rank matrices while the base weights stay frozen. QLoRA does the same over a base quantized to 4 bits with NF4, double quantization and paged optimizers, so it needs much less GPU memory.

Are on-device AI models private?

They keep prompts and outputs off the network, which is a real gain; Google's docs say Gemini Nano works without sending data to the cloud. The data still lives on the device, and prompt injection still applies.

Do small models need an AI-BOM?

Yes. Small models are the ones most often copied, converted, quantized and fine-tuned outside your team. Record base model, revision, quantization, adapters, hashes, runtime and licence in CycloneDX ML-BOM or SPDX 3.0 form.

Small models run on open source runtimes, tokenizers, conversion scripts and inference servers, and those packages get CVEs like any other dependency. Every rebuild of a laptop, edge box or serving image pulls them again, so the usual software supply chain hygiene applies to the stack around the weights.

Fix it without waiting for upstream

Whatever the vulnerability, most of the risk sits in two gaps. The first is the time between a fix being published and that fix actually running in your production builds. The second, and worse, is the time before any fix exists at all, while a maintainer works out a patch and a release schedule. Waiting through both gaps is a habit, not a requirement. Safeguard gives teams three ways to close them.

1. Autonomous remediation: pull requests, not tickets

When a fixed version exists, Griffin, Safeguard's security model, opens a policy gated pull request that moves the affected dependency to the safe version, and your own CI runs your tests on it before anyone merges. You can run it on one repository or in bulk across every project that carries the vulnerable version. Remediation categories include dependencies, licenses, security posture and findings, and each one is set separately. Reachability analysis ranks the vulnerable code paths your application actually calls first, so the first pull requests you merge are the ones that matter most. Auto-merge is a separate switch and stays off by default, so your team decides what merges on its own and what still needs a human approval.

2. Gold Open Source: the safe version in seconds, free

gold.safeguard.sh is free and needs no login. Search any CVE or package to see affected and fixed versions, CVSS, EPSS, CISA KEV status and a recommended safe version, or use the "Am I affected?" checker. The same data is available through a free read API, README status badges and RSS threat feeds.

3. Gold Registry: start from components that are already fixed

Safeguard Gold rebuilds patched versions of open-source packages and images from source, signs each build, attaches an SBOM, SLSA provenance and VEX, and serves them from registry.safeguard.sh to your organization. Every Gold artifact is held continuously to zero exploitable critical or high CVEs and a completed license check, with signed build provenance (signed with Safeguard's key and verifiable with cosign, SLSA Build Level 3 as the design target) and a public evidence page you can hand to an auditor. An artifact loses its Gold badge if it regresses. Teams can point their own package directory at the registry so hardened equivalents resolve without changing how developers install anything.

When upstream has no fix yet, or will not ship one, Safeguard first works to get the fix accepted upstream. If the issue is severe or upstream will not move, Safeguard builds a fork, fixes the vulnerability there and publishes a hardened Gold build, so you are not stuck on a maintainer's release schedule. You can also request a hardened build of a specific package. Ask which packages it covers before you plan around it.

One honest limit: when the vulnerable code lives in a vendor appliance, a hosted service or an operating system runtime you do not build yourself, the fix still has to come from that vendor or distribution. The steps above apply to the open source dependencies and container images you build and ship, which is where most teams carry the largest and least tracked share of their exposure.

See how Safeguard would help

This demo is a real Safeguard CLI session: install, ask for a fix to a vulnerable dependency, approve each step, then scan. The same flow applies to the runtime and tokenizer packages in a model-serving repository.

Click through it at your own pace, or open it full screen.

This one sets up the local runner on your own machine, approves it with its device code and scans from the terminal, so scanning runs where model code and weights already live.

Click through it at your own pace, or open it full screen.

This one connects Claude to the Safeguard MCP server with OAuth and chooses which tools Claude can use, the narrow-permissions principle this post recommends.

Click through it at your own pace, or open it full screen.

Download Safeguard at https://safeguard.sh/download and read the CLI docs at https://docs.safeguard.sh/docs/cli .

Quick check

Sources

• Hinton et al., Distilling the Knowledge in a Neural Network
• Sanh et al., DistilBERT
• Frantar and Alistarh, SparseGPT
• Muralidharan et al., Compact Language Models via Pruning and Knowledge Distillation
• Dettmers et al., LLM.int8()
• Frantar et al., GPTQ
• Lin et al., AWQ
• Hu et al., LoRA
• Dettmers et al., QLoRA
• Gunasekar et al., Textbooks Are All You Need
• Abdin et al., Phi-3 Technical Report
• Liu et al., MobileLLM
• Gu et al., BadNets
• Hubinger et al., Sleeper Agents
• Carlini et al., Poisoning Web-Scale Training Datasets is Practical
• llama.cpp quantize README
• GGUF specification
• ONNX Runtime quantization
• Ollama import docs
• MLC LLM docs
• Apple foundation models 2025 updates
• Android Developers, Gemini Nano
• Phi-4-mini-instruct card and config
• Gemma 4 E2B card
• Llama 3.2 model card and licence
• Qwen3.5-4B card and Qwen3.5-0.8B card
• Ministral 3 3B card
• SmolLM3-3B card
• Granite 4.2 3B card
• Python pickle docs
• PyTorch torch.load docs
• Safetensors docs
• Hugging Face Hub pickle scanning
• CycloneDX ML-BOM
• SPDX AI profile
• OWASP Top 10 for LLM Applications 2025 and LLM03:2025 Supply Chain
• MITRE ATLAS
• NIST AI 600-1
• Apache License 2.0
• Gemma Terms of Use and Gemma 4 license
• Safeguard air-gapped deployment docs

Last reviewed: 2 November 2026.