Vision language models in depth: how VLMs are built, run and secured

A vision language model (VLM) is a language model that also takes images as input. A vision encoder cuts each image into patches and turns them into vectors, a connector maps those vectors into the language model's embedding space, and the language model reads the resulting image tokens alongside your text and answers in text. That design lets a VLM describe a photo, read an invoice, answer a question about a chart or look at a screenshot and decide where to click.

This is the in-depth reference: encoders, connectors, how pixels become a token bill, document AI, grounding, screen-driving agents, video, failure modes, and what changes for security when the input is a picture.

Key takeaways

• A VLM has three parts: a vision encoder (usually a Vision Transformer trained CLIP or SigLIP style), a connector (MLP projector, Q-Former, cross-attention or plain linear layers) and a language model.
• Images cost tokens. On Claude's standard tier a 1000 x 1000 image is 1,296 visual tokens; Gemini video at default settings is about 100 tokens per second.
• Text inside an image enters the same token stream as your instructions. Screenshot agents need sandboxes, allowlists and human confirmation.
• Vendors document approximate counts, approximate coordinates and hallucination on small or rotated images. Verify outputs in code.
• A self-hosted VLM is often several artifacts (language model, vision projector, processor config, sometimes load-time Python). Each needs provenance and an AI-BOM entry.

This series

• LLMs explained: how large language models work, from transformers to agents and security (the hub)
• Small language models explained: distillation, quantization, on-device AI and security
• Vision language models in depth (this post)
• Audio language models explained: speech encoders, audio tokens, voice agents and security (publishes Saturday 7 November)

How a VLM differs from a text-only LLM

The language model inside a VLM is the decoder transformer the series hub explains. What changes is the input. A text LLM looks up token IDs in an embedding table. A VLM also receives vectors that never touched the tokenizer: they came from an image encoder and a connector, placed where the image sat in the conversation.

Three consequences run through this post. The model never sees pixels, only what survived resizing and patching, so blurred footnotes stay blurred whatever you prompt. Image tokens fill the context window and the bill like text tokens. And there is no separate channel marking "this came from a picture", which is the root of visual prompt injection.

Output is text in the models covered here. Anthropic's vision docs, for example, say Claude is an image understanding model only and cannot generate or edit images.

The vision encoder: ViT, CLIP and SigLIP

The Vision Transformer paper (Dosovitskiy and colleagues, 2020) showed that a plain transformer over a sequence of image patches classifies images very well when pre-trained on enough data. The image is cut into fixed squares, each is flattened and projected to a vector, position embeddings are added, and standard transformer layers follow. A 336 x 336 image with 14 x 14 patches gives 24 x 24 = 576 patch vectors; the shape used by LLaVA-1.5's CLIP-ViT-L-336px encoder.

CLIP (Radford and colleagues, 2021) trained an image encoder and a text encoder together on 400 million image and text pairs from the internet with one task: predict which caption goes with which image. Its image vectors line up with language, which is what a language model downstream needs, and it made zero-shot classification by text description possible. CLIP's contrastive loss normalizes with a softmax across the whole batch.

SigLIP (Zhai and colleagues, 2023) swapped in a pairwise sigmoid loss: each image and caption pair is scored on its own, so no global normalization is needed. The paper reports better results at small batch sizes, scaling to larger ones, and quickly diminishing returns, with 32k described as a reasonable batch. SmolVLM2's card says it is built on the shape-optimized SigLIP encoder with a SmolLM2 language model.

The encoder's native resolution and patch size decide whether small print, handwriting or a UI icon can be read at all. When a VLM misreads something, check the effective resolution reaching the encoder before rewriting the prompt.

Connectors: how image vectors reach the language model

MLP projector. LLaVA (Liu and colleagues, 2023) connected a CLIP encoder to a language model with a projection and instruction-tuned the system on multimodal conversations that a text-only GPT-4 generated from captions and boxes. LLaVA-1.5 found the fully connected connector "surprisingly powerful and data-efficient": CLIP ViT-L at 336 pixels, an MLP projection, academic VQA data, and a 13B checkpoint trained on 1.2 million public examples in about a day on one 8-GPU A100 node. Each patch becomes one token, so detail is kept but token count grows with resolution. InternVL3's card calls its design the "ViT-MLP-LLM" paradigm.

Q-Former. BLIP-2 (Li and colleagues, 2023) kept the image encoder and language model frozen and trained only a lightweight Querying Transformer between them, in two stages. Learned queries attend to image features and emit a fixed number of outputs whatever the image size. The paper reports beating Flamingo-80B by 8.7 percent on zero-shot VQAv2 with 54 times fewer trainable parameters. The trade is a small fixed budget per image at some cost in fine detail.

Cross-attention. Flamingo (Alayrac and colleagues, 2022) bridged frozen vision and language models, handled arbitrarily interleaved images and text, and took video. Instead of placing image tokens in the main sequence, it added cross-attention layers so text tokens attend to visual features, with a Perceiver-style resampler producing a fixed set of visual tokens. Meta's Llama 3.2-Vision card describes a separately trained adapter of cross-attention layers feeding image encoder representations into the Llama 3.1 language model.

No encoder. Google's Gemma 4 card describes Gemma 4 12B as encoder-free: raw image patches and audio waveforms are projected into the language model's embedding space by lightweight linear layers, while the other Gemma 4 models use dedicated encoders.

Image tokens and resolution: where the bill comes from

Each maker handles resolution differently. These facts are from their own docs.

Claude. Anthropic's vision docs say Claude processes images in 28 x 28 pixel patches, one visual token each, so cost is ceil(width/28) x ceil(height/28) after any downscaling. The standard tier caps the long edge at 1,568 pixels and 1,568 tokens; the high-resolution tier (Claude 4.7 and later) caps at 2,576 pixels and 4,784 tokens. Limits: up to 600 images per API request (100 for models with a 200k-token context window), 8000 x 8000 pixels per image, JPEG, PNG, GIF and WebP.

Worked example one: a 1000 x 1000 image is not resized; 1000/28 rounds up to 36, and 36 x 36 = 1,296 tokens, matching Anthropic's table. Worked example two: a 1920 x 1080 screenshot on the standard tier is downsized to 1456 x 819, giving 52 x 30 = 1,560 tokens. On the high-resolution tier it is not resized: 69 x 39 = 2,691 tokens. The docs note high resolution can use roughly three times more tokens, so downsample when you do not need the detail.

OpenAI. The images and vision guide exposes a detail parameter (low, high, original or auto, by model; auto by default) and recommends original, where supported, for OCR, small-object detection and computer use. It lists up to 1,500 images per request and rejects images above 30,000 patches after resizing rather than shrinking them.

Gemini. Google's docs charge 258 tokens if both dimensions are 384 pixels or less; larger images are tiled into 768 x 768 tiles at 258 tokens each. Worked example three, from the docs: a 960 x 540 image has a crop unit of 360 (shorter side over 1.5), so 3 x 2 = 6 tiles, or 1,548 tokens. Newer models add a media_resolution parameter capping tokens per image or frame.

Open models. Qwen2-VL's paper introduced Naive Dynamic Resolution, mapping different image sizes to different token counts, with Multimodal Rotary Position Embedding for position. Qwen2.5-VL's card shows min_pixels and max_pixels settings with an example range of 256 to 1,280 tokens per image. InternVL3's card tiles images at 448 x 448. Gemma 3's card normalizes images to 896 x 896 and 256 tokens each.

Pick the smallest resolution that passes your evaluation, resize on your side, and keep the scale factor so coordinates can be mapped back. Anthropic and OpenAI both recommend resizing yourself for coordinate work.

How VLMs are trained

Training runs in stages. First, alignment: the encoder is usually pre-trained with a CLIP or SigLIP objective, then the connector is trained, often with both ends frozen, as in BLIP-2. Second, interleaved pretraining: Flamingo's authors say training on web corpora with interleaved text and images was key to its few-shot ability, and Pixtral 12B's card says it was trained on interleaved image and text data. Third, visual instruction tuning, LLaVA's contribution, where machine-generated conversations teach the assistant behaviour (and any mistakes in that data). The POPE authors found objects that appear often in visual instruction data are especially prone to hallucination. Fourth, preference tuning and task data: Llama 3.2-Vision's card lists SFT and RLHF, and Qwen2.5-VL's report highlights invoices, forms, tables, boxes and computer and phone use.

Data is the security-relevant part. CLIP-scale sets are scraped from the web, and the poisoning research below shows how little of such a set an attacker needs.

OCR and document understanding

The classic pipeline runs OCR, then hands text to a language model. The Donut paper (Kim and colleagues, 2021) lists its costs: OCR compute, inflexibility across languages and document types, and OCR errors propagating downstream. Donut proposed reading the document image directly, which is how modern VLMs mostly work. llama.cpp's multimodal docs warn that OCR-specialised models expect specific prompts and input structure, so a document model is not a drop-in chat model.

DocVQA (Mathew and colleagues, 2020) has 50,000 questions on more than 12,000 document images; its authors found the biggest gap on questions needing document structure, against 94.36 percent human accuracy. ChartQA (Masry and colleagues, 2022) has 9.6K human-written and 23.1K generated chart questions needing visual and logical reasoning, including arithmetic. OpenAI's guide lists small text, rotation, non-Latin scripts and varying line styles in graphs as weak spots; Anthropic's warns that heavy JPEG compression and resizing can make text illegible.

A dependable extraction pattern: render pages at a known resolution, ask for JSON with named fields, validate every field against a schema and business rules, and route failures to a person. The validation must not depend on the model:

from decimal import Decimal

def check_invoice(fields: dict) -> list[str]:
    problems = []
    lines = [Decimal(x["amount"]) for x in fields.get("line_items", [])]
    if Decimal(fields["total"]) != sum(lines) + Decimal(fields.get("tax", "0")):
        problems.append("total does not match line items plus tax")
    if not fields.get("invoice_number", "").isalnum():
        problems.append("invoice number has unexpected characters")
    return problems

Grounding: boxes, points and coordinates

Kosmos-2 (Peng and colleagues, 2023) grounded text by writing referring expressions as Markdown-style links whose targets are location tokens. Qwen2.5-VL's card says it outputs bounding boxes or points as stable JSON. Grounding powers redaction, form field location and clicking on screens, and it is where vendors are most careful: Anthropic says Claude's coordinates are approximate and relative to the image after resizing, and OpenAI says to resize yourself and map coordinates back. Own the transform, and check that a box lands inside the image on something plausible before acting.

Screenshot and GUI agents

A GUI agent is a VLM in a loop: screenshot, decide, act, screenshot again. Anthropic's computer use docs describe Claude returning tool calls such as screenshot, left_click or type, which your code performs on a container or virtual machine before returning the result; the reference implementation runs in a Docker container. OpenAI's computer use guide likewise starts from an isolated browser or desktop sandbox, and its examples use a confirmation callback that limits which controls the agent may touch, with API safety checks that stop a run pending human approval.

Anthropic's docs recommend four precautions: a dedicated low-privilege VM or container, no sensitive data such as login information in the model's reach, an allowlist of domains, and human confirmation for consequential actions such as financial transactions or agreeing to terms. They state that Claude may follow instructions found in webpages or images even when they conflict with yours, that classifiers scan tool results such as screenshots for potential injection, and that the precautions still matter with those classifiers on. Documented limits include latency and mistaken coordinates; OpenAI blocks CAPTCHA submission.

OSWorld (Xie and colleagues, 2024) built 369 tasks on real Ubuntu, Windows and macOS with execution-based checks. At publication humans completed over 72.36 percent and the strongest model tested 12.24 percent, held back mainly by GUI grounding and operational knowledge. Makers now publish their own agent numbers; Qwen2.5-VL-7B's card reports 84.7 on ScreenSpot and 29.0 on ScreenSpot Pro. Treat any number as one harness at one time, and see AI agent evaluation explained.

Design rules: prefer an API over a screen whenever one exists; run in a disposable VM with no standing credentials; gate payments, sends and deletes behind a person, as in durable execution for AI agents; log every screenshot and action; and give each agent its own scoped identity, as in tracking which AI agent holds what access.

Video understanding: frames, sampling and tokens

Most VLMs treat video as sampled frames. Google's docs say Gemini samples 1 frame per second by default at 258 tokens per frame (66 at low media resolution), plus 32 tokens per second of audio, about 100 tokens per second in total at default low media resolution or about 300 at high, and you can set a custom fps such as 0.5.

Worked example four: a 10 minute clip is 600 seconds, so about 60,000 tokens at default or 180,000 at high resolution; an hour is about 360,000 at default. Anything between sampled frames is invisible, and the docs warn 1 FPS can lose detail. Qwen2.5-VL's card describes dynamic FPS sampling and absolute time encoding for second-level event location, and SmolVLM2's card positions its 256M to 2.2B models for video. A summarised screen recording only saw the frames it was given, but the recording itself holds everything that was on screen.

Where VLMs fail

Hallucination. POPE (Li and colleagues, 2023), the first systematic study of object hallucination in large VLMs, found representative models mostly suffered from it, and that older evaluations were sensitive to prompt wording. It proposed polling yes or no questions instead.

Counting, space and small things. Anthropic and OpenAI list approximate counts, imprecise localization (OpenAI's example is chess positions), errors on low-quality, rotated or very small images (Anthropic cites under 200 pixels) and trouble with panoramic and fisheye images. Anthropic says Claude cannot reliably tell whether an image is AI-generated, and both say their models are not for diagnostic scans such as CT.

Metadata. Both vendors say their models do not read image metadata or file names. That does not remove the metadata from files you store.

Typographic and visual prompt injection. Our posts on multimodal prompt injection in images and documents and OWASP LLM01 controls that work cover the threat model, detection limits and controls. What is specific to VLM architecture:

• Readable text in an image becomes features the language model treats as language, with no tag saying it came from a picture. FigStep (Gong and colleagues, 2023) showed typographic images bypassing safety alignment in open VLMs and traced it to alignment not covering visual embeddings.
• Visual input is continuous, so invisible changes matter. Qi and colleagues (2023) showed one optimised image universally jailbreaking an aligned model with vision; Bagdasaryan and colleagues (2023) blended perturbations into images and audio that steer a model when a user asks about the file. These need model access and were shown on open models, but "it looks fine to a person" is not a defence.
• The indirect form, framed by Greshake and colleagues (2023), arrives in screenshots, PDF pages, product photos and ticket attachments. OWASP's 2025 LLM01 entry names instructions hidden in images and cross-modal attacks; MITRE ATLAS tracks AML.T0051 LLM Prompt Injection with direct, indirect and triggered sub-techniques; NIST AI 600-1 describes indirect injection through data likely to be retrieved.

If your server fetches images by URL, read vLLM allowed media domains and SSRF through multimodal inputs.

Evaluating a VLM: what the benchmarks measure

Public benchmarks leak into training data, scores depend on harness, resolution and prompt, and none measure your documents or your attackers. MMMU's launch reported GPT-4V at 56 percent and Gemini Ultra at 59 percent, a late 2023 snapshot. Keep a private test set from your own data out of any training loop.

Vision language model families as of 2 November 2026

Facts only from each maker's own card, docs or licence page, checked 2 November 2026. Sizes as stated. No rankings.

All rows as of 2 Nov 2026. InternVL3 shows why you read the licence text, not the tag: its licence section says MIT for the project and Apache-2.0 for the Qwen2.5 component, while the metadata says Apache-2.0.

Securing vision language models

The model supply chain

Python's documentation warns that pickle is not secure and that malicious pickle data can execute arbitrary code when loaded. PyTorch's torch.load docs (2.15) say never to load data from an untrusted source and show weights_only defaulting to True, restricting the unpickler to tensors, primitive types and dictionaries plus types you allow. Safetensors stores tensors safely, as opposed to pickle, with zero-copy loading. GGUF and ONNX are the other formats you will meet locally and at the edge.

VLMs multiply the pieces. llama.cpp loads a text model plus a separate multimodal projector passed with mmproj; its example pairs a Q4_K_M model with its matching mmproj file. Pixtral's card downloads params.json, consolidated.safetensors and tekken.json. The processor configuration (resolution, min_pixels, max_pixels, normalization) changes what the model sees, so a swapped processor file can quietly break OCR or shift coordinates. Hash every piece.

The cards for Molmo 7B-D, InternVL3-8B and Phi-4-multimodal load with trust_remote_code=True in Transformers, which runs Python files from the model repository on your machine. Pin a reviewed commit and never let it float:

from transformers import AutoModelForCausalLM, AutoProcessor

REV = "PINNED_COMMIT_SHA"  # the commit you reviewed, never "main"

processor = AutoProcessor.from_pretrained(
    "allenai/Molmo-7B-D-0924", revision=REV, trust_remote_code=True
)
model = AutoModelForCausalLM.from_pretrained(
    "allenai/Molmo-7B-D-0924", revision=REV, trust_remote_code=True
)

More in OWASP LLM03 supply chain mitigations and the ML supply chain glossary.

Poisoned and backdoored weights

Backdoors started with images. BadNets (Gu and colleagues, 2017) showed a maliciously trained network that performs well on the user's own validation data but misbehaves on inputs carrying an attacker-chosen trigger, and framed it as a supply chain risk for outsourced training and pre-trained models. Sleeper Agents (Hubinger and colleagues, 2024) showed backdoor behaviour in language models persisting through supervised fine-tuning, reinforcement learning and adversarial training. Carlini and colleagues (2023) estimated they could have poisoned 0.01 percent of LAION-400M or COYO-700M, image and text sets of the kind used for CLIP-style training, for 60 US dollars, because content behind a dataset's URLs can change after annotators first saw it.

So a community fine-tune can pass your tests and react differently to one visual pattern. Prefer the maker's weights, verify hashes, keep a private evaluation set with adversarial and out-of-distribution images, and use the LLM04 data and model poisoning checklist.

Provenance and the AI-BOM

Record the encoder, projector, language model, processor config, adapters and the licence of each. The cards make it concrete: Molmo 7B-D's card says it is based on Qwen2-7B with OpenAI CLIP as the vision backbone, and InternVL3 lists an InternViT encoder and a Qwen2.5 model per size. CycloneDX's ML-BOM covers model and dataset transparency; SPDX 3.0's AI profile defines an AIPackage class with fields such as domain, limitation, autonomyType, informationAboutTraining and safetyRiskAssessment. One component, linked to others for each base model and the projector:

{
  "type": "machine-learning-model",
  "name": "Molmo-7B-D-0924",
  "version": "PINNED_COMMIT_SHA",
  "supplier": { "name": "Allen Institute for AI" },
  "licenses": [ { "license": { "id": "Apache-2.0" } } ],
  "hashes": [ { "alg": "SHA-256", "content": "..." } ],
  "externalReferences": [
    { "type": "website", "url": "https://huggingface.co/allenai/Molmo-7B-D-0924" }
  ]
}

Image inputs as untrusted data

• Image text never carries authority. Say so in the system prompt and enforce it in code.
• OCR output that becomes tool arguments gets schema and allowlist checks first. An account number read from an upload never becomes a payment destination unchecked.
• Separate reading from acting: one call extracts structured data, a second call with no access to the raw image decides actions.
• Screenshot agents get no standing credentials. Anthropic's docs say logging in during computer use raises prompt injection risk.

Data leakage, EXIF and personal data

Photos can carry GPS coordinates and device details in EXIF. Screenshots catch email, chats, account numbers and faces; document AI routinely sees IDs, cheques and medical forms. OWASP LLM02:2025 Sensitive Information Disclosure applies to all of it. Strip metadata on upload, before storage or logging; crop to the region needed; read each provider's data terms (Anthropic's vision FAQ says uploaded images are not used to train its models); give agent screenshot logs the retention and access rules of the data they show; and do not use VLMs to identify people. Anthropic says Claude refuses to name people in images, and Meta's Llama 3.2-Vision card describes mitigations against uniquely identifying individuals.

Licence terms (general information, not legal advice)

Meta's Llama 3.2 Acceptable Use Policy withholds the licence rights for multimodal Llama 3.2 models from individuals domiciled in, or companies with a principal place of business in, the European Union, while noting this does not apply to end users of a product that incorporates them. The Gemma Terms of Use, which cover Gemma 3, count offering Gemma as a hosted service as distribution and require use restrictions to flow down; Gemma 4's card lists Apache 2.0. Qwen2.5-VL 7B, Pixtral, SmolVLM2 and Molmo state Apache-2.0, Phi-4-multimodal MIT. Read the licence file at your pinned revision for every component and ask counsel before shipping.

Self-hosting versus calling an API

For the broader choice, see local or hosted LLM: a decision guide.

Where Safeguard fits

Safeguard does not pick which VLM reads your invoices best. Its part is the supply chain around the model and the guardrails around the calls.

Model files and inventory. Safeguard connects to Hugging Face, Kaggle Models, ModelScope and Civitai through read-only integrations, scans the files of the models you select for unsafe serialization formats and embedded code with AI-SPM model file scanning, and records each model in your AI inventory through AI-BOM discovery, with its licence or permissions. For a VLM that means each component you pull. See scanning models from Hugging Face, Kaggle, ModelScope and Civitai and inventorying models in SageMaker, Vertex AI and Azure AI Foundry.

Choosing a starting model. Gold Open Source (gold.safeguard.sh) gives free CVE intelligence with affected and fixed versions, and its directory covers AI models; see the Gold Open Source explainer and sourcing models from Hugging Face safely.

The serving stack. Inference servers, image libraries, tokenizers and agent frameworks are open source dependencies. Safeguard scans repositories, container images and SBOM files for vulnerable versions, and CI gates can stop them shipping.

Guardrails. Safeguard AI Gateway is a runtime guardrail layer for LLM apps and agents. Its docs describe checks on prompts, responses and tool calls for direct and indirect prompt injection, jailbreak patterns and personal data leaving in responses, in monitor mode by default with a fail-open design. The docs describe text checks, so route OCR output, extracted fields and tool results through it as text. See governing every LLM call with AI Gateway.

Where it runs. Safeguard runs as SaaS in US, EU (Frankfurt) and Gov regions, in a private VPC or on-prem, with fully air-gapped deployment available as a custom engagement. The AI mode is chosen per tenant, including your own approved model; Safeguard's models in your own environment are available on request through sales. Griffin, Safeguard's security model, opens the remediation pull requests described below; Safeguard's models are single-purpose security models, not general VLMs. Details: where your code and data go with Safeguard.

Safeguard is SOC 2 Type II compliant.

Common mistakes

• Sending full-resolution images by default, or downscaling until the text is gone. Test the resolution.
• Acting on coordinates without mapping them back from the resized image.
• Letting image text become instructions instead of data.
• Giving a screenshot agent your real browser profile and saved sessions.
• Pinning weights but not the projector or processor files.
• Running trust_remote_code against a floating branch.
• Reading the licence tag instead of the licence.
• Keeping EXIF and full screenshots in logs indefinitely.
• Choosing a model from a leaderboard instead of your own test set.

FAQ

What is a vision language model in simple terms?

A language model that can also look at images. An encoder turns the picture into vectors, a connector translates them into the model's input space, and the model answers in text.

What is the difference between a VLM and a multimodal LLM?

The terms overlap. "Multimodal LLM" is broader and can include audio input or image output; "VLM" usually means image or video in, text out. Phi-4-multimodal and Gemma 4 also take audio.

How does a vision language model read text in an image?

Most read text straight from pixels through the encoder, with no separate OCR engine. Resolution, compression, rotation and script decide accuracy.

How many tokens does an image cost?

It depends on the provider. Claude uses ceil(width/28) x ceil(height/28) after resizing, so 1000 x 1000 is 1,296. Gemini uses 258 per 768 x 768 tile. OpenAI depends on the detail setting and model.

Can a VLM be prompt-injected through an image?

Yes. Image text enters the same token stream as instructions, and papers have shown typographic and adversarial-image attacks on open VLMs. Treat image content as data and gate actions in code.

Are computer use agents safe to run?

With care. Anthropic recommends a dedicated low-privilege VM or container, no login data in the model's context, a domain allowlist and human confirmation for consequential actions.

Can I run a vision language model locally?

Yes. Qwen2.5-VL, Gemma, SmolVLM2, Pixtral and Molmo publish weights, and llama.cpp takes images with a model file plus a matching mmproj file.

Do VLMs read EXIF metadata?

Anthropic and OpenAI say their models do not. Your storage and logs still keep it, so strip it on upload.

Which benchmarks matter for VLMs?

MMMU for broad reasoning, DocVQA for documents, ChartQA for charts, POPE for hallucination and OSWorld for computer use. Shortlist with them, decide with your own data.

Vision models are served by open source inference servers, image decoding libraries, tokenizers and agent frameworks, and those packages get CVEs like any other dependency. Every rebuild of a serving image or agent sandbox pulls them again, so ordinary software supply chain hygiene still decides how exposed the stack around the weights is.

Fix it without waiting for upstream

Whatever the vulnerability, most of the risk sits in two gaps. The first is the time between a fix being published and that fix actually running in your production builds. The second, and worse, is the time before any fix exists at all, while a maintainer works out a patch and a release schedule. Waiting through both gaps is a habit, not a requirement. Safeguard gives teams three ways to close them.

1. Autonomous remediation: pull requests, not tickets

When a fixed version exists, Griffin, Safeguard's security model, opens a policy gated pull request that moves the affected dependency to the safe version, and your own CI runs your tests on it before anyone merges. You can run it on one repository or in bulk across every project that carries the vulnerable version. Remediation categories include dependencies, licenses, security posture and findings, and each one is set separately. Reachability analysis ranks the vulnerable code paths your application actually calls first, so the first pull requests you merge are the ones that matter most. Auto-merge is a separate switch and stays off by default, so your team decides what merges on its own and what still needs a human approval.

2. Gold Open Source: the safe version in seconds, free

gold.safeguard.sh is free and needs no login. Search any CVE or package to see affected and fixed versions, CVSS, EPSS, CISA KEV status and a recommended safe version, or use the "Am I affected?" checker. The same data is available through a free read API, README status badges and RSS threat feeds.

3. Gold Registry: start from components that are already fixed

Safeguard Gold rebuilds patched versions of open-source packages and images from source, signs each build, attaches an SBOM, SLSA provenance and VEX, and serves them from registry.safeguard.sh to your organization. Every Gold artifact is held continuously to zero exploitable critical or high CVEs and a completed license check, with signed build provenance (signed with Safeguard's key and verifiable with cosign, SLSA Build Level 3 as the design target) and a public evidence page you can hand to an auditor. An artifact loses its Gold badge if it regresses. Teams can point their own package directory at the registry so hardened equivalents resolve without changing how developers install anything.

When upstream has no fix yet, or will not ship one, Safeguard first works to get the fix accepted upstream. If the issue is severe or upstream will not move, Safeguard builds a fork, fixes the vulnerability there and publishes a hardened Gold build, so you are not stuck on a maintainer's release schedule. You can also request a hardened build of a specific package. Ask which packages it covers before you plan around it.

One honest limit: when the vulnerable code lives in a vendor appliance, a hosted service or an operating system runtime you do not build yourself, the fix still has to come from that vendor or distribution. The steps above apply to the open source dependencies and container images you build and ship, which is where most teams carry the largest and least tracked share of their exposure.

See how Safeguard would help

This demo is a real Safeguard CLI session: install, ask for a fix to a vulnerable dependency, approve each step, then scan. The same flow fits a VLM serving repository, where the image decoding and inference packages are the dependencies to keep current.

Click through it at your own pace, or open it full screen.

This one sets up the local runner on your own machine, approves it with its device code and scans from the terminal. It suits teams that keep model code and weights on their own hardware.

Click through it at your own pace, or open it full screen.

This one connects Claude to the Safeguard MCP server with OAuth and chooses which tools Claude can use. Narrow tool permissions are the same principle this post applies to screenshot agents.

Click through it at your own pace, or open it full screen.

Download Safeguard at safeguard.sh/download and read the CLI docs.

Quick check

Sources

• ViT, arXiv 2010.11929
• CLIP, arXiv 2103.00020
• SigLIP, arXiv 2303.15343
• LLaVA, arXiv 2304.08485
• LLaVA-1.5, arXiv 2310.03744
• BLIP-2, arXiv 2301.12597
• Flamingo, arXiv 2204.14198
• Qwen2-VL, arXiv 2409.12191
• Qwen2.5-VL Technical Report, arXiv 2502.13923
• Donut, arXiv 2111.15664
• Kosmos-2, arXiv 2306.14824
• DocVQA, arXiv 2007.00398
• ChartQA, arXiv 2203.10244
• MMMU, arXiv 2311.16502
• POPE, arXiv 2305.10355
• OSWorld, arXiv 2404.07972
• Indirect Prompt Injection (Greshake and colleagues), arXiv 2302.12173
• Abusing Images and Sounds, arXiv 2307.10490
• Visual Adversarial Examples Jailbreak, arXiv 2306.13213
• FigStep, arXiv 2311.05608
• BadNets, arXiv 1708.06733
• Sleeper Agents, arXiv 2401.05566
• Poisoning Web-Scale Training Datasets, arXiv 2302.10149
• Anthropic: Vision
• Anthropic: Computer use tool
• OpenAI: Images and vision
• OpenAI: Computer use
• Google: Gemini image understanding
• Google: Gemini video understanding
• llama.cpp multimodal docs
• Qwen2.5-VL-7B-Instruct model card
• Llama 3.2-Vision model card
• Llama 3.2 Acceptable Use Policy
• Gemma 3 model card
• Gemma 4 model card
• Gemma Terms of Use
• Pixtral-12B-2409 model card
• SmolVLM2-2.2B-Instruct model card
• Molmo-7B-D-0924 model card
• InternVL3-8B model card
• Phi-4-multimodal-instruct model card
• OWASP LLM01:2025 Prompt Injection
• OWASP LLM02:2025 Sensitive Information Disclosure
• MITRE ATLAS data (AML.T0051, AML.T0010, AML.T0020)
• NIST AI 600-1, Generative AI Profile
• Python pickle documentation
• PyTorch torch.load (2.15)
• Safetensors documentation
• CycloneDX ML-BOM
• SPDX 3.0.1 AI profile: AIPackage
• Safeguard docs: AI Gateway
• Safeguard docs: disconnected deployment as a custom engagement

Last reviewed: 2 November 2026.