Safeguard
AI Security•

How teams cut AI spend by up to about 90% with open-weight models and fixed capacity

Cut AI spend by routing routine tasks to open-weight models on fixed-cost GPUs, keeping a frontier fallback. The 8 levers, a migration plan and a cost model.

Safeguard Security Team
11 min read

Teams cut AI spend by up to about 90% when they move high-volume, routine tasks off per-token frontier APIs and onto open-weight models running on capacity they pay for at a fixed rate. The bill stops growing with every token and becomes a fixed line. A frontier model stays as a fallback for the few tasks that need it.

This is the savings playbook: where the bill goes, eight levers, a migration plan and a cost model. For the definition of fixed-cost AI and a break-even calculation, read the fixed-cost AI explainer.

Key takeaways

  • Most AI spend comes from high-volume tasks, long contexts and agent loops.
  • Eight levers cut it: routing, quantization, context trimming, prefix caching, batching, speculative decoding, answer caching and capacity scheduling.
  • Savings near 90% need high, steady volume and tasks a smaller model passes on your own tests.
  • Choose models on licence, size, languages and your own evaluation results.

Where does an AI bill actually go?

Export a month of usage by use case: input and output tokens, requests, retries and model. Input usually dominates, because retrieved documents, history and tool output ride along on every call. Agent loops resend a growing context, and retries add calls that produce nothing.

Classification, extraction, summaries and tagging rarely need a frontier model. Long multi-step reasoning might. See agentic AI budget controls and enterprise LLM budget patterns.

What are the 8 levers that cut AI inference cost?

Fewer tokensContext trimmingPrefix cachingAnswer cachingCheaper tokensTask routingQuantizationSpeculative decodingFuller hardwareBatchingRight-sizingScheduling
The eight levers, grouped by what they reduce.
  1. Task routing. Send each request to the smallest model that passes your tests for that task. Measure pass rate per task.
  2. Quantization. Lower precision fits more model per GPU. OpenAI's card says gpt-oss-120b uses MXFP4 MoE weights and fits a single 80GB GPU (card). Measure quality and throughput.
  3. Context trimming. Cap retrieved chunks and strip tool output. Measure input tokens per request.
  4. Prefix caching. Keep system prompts and tool schemas first and identical. vLLM reuses the KV cache for queries that share a prefix (docs). Measure hit rate and time to first token.
  5. Batching. Run offline work in batches that fill idle capacity. Measure GPU utilisation by hour.
  6. Speculative decoding. A draft model proposes tokens the main model checks (docs). Measure accepted tokens per step.
  7. Answer caching. Store answers to repeated questions with an expiry. Measure hit rate and stale answers.
  8. Right-sizing and scheduling. Size GPUs to steady load and queue low-priority work behind interactive work. Measure queue wait and idle hours.

How do you migrate from a frontier API without losing quality?

  1. Baseline cost, latency and quality per use case for 2 to 4 weeks.
  2. Build an evaluation set of a few hundred real, anonymised inputs per task (how to build one; judge pitfalls).
  3. Shadow traffic: copy live requests to the candidate and log its answers without serving them.
  4. Cut over by task, highest volume and lowest risk first.
  5. Keep a frontier fallback for low confidence or failed schema checks.
  6. Monitor pass rate, fallback rate, latency and queue wait.

Evaluation checklist: pass rate within your margin of the baseline, schema checked on every response, user languages covered, fallback rate low, rollback tested.

How does a cost model reach about 90%, and when does it not?

A worked example. Every value is an assumption, not a measured result.

AssumptionA: high volumeB: low volumeC: half need frontier
Tokens per month (80% input)30 billion1 billion30 billion
API price per million (input, output)3, 153, 153, 15
API baseline per month162,0005,400162,000
GPUs at 2.50 per hour, 730 hours6: 10,9501: 1,8256: 10,950
Frontier fallback3%: 4,8603%: 16250%: 81,000
Operations overhead2,0002,0002,000
Total17,8103,98793,950
Savingabout 89%about 26%about 42%

Scenario A assumes about 1,850 tokens per second per GPU (input and output together), all month; prove it with a load test. Low or bursty volume and high fallback erode the saving.

What changes when AI usage is unlimited?

The next request costs nothing extra, so teams stop rationing and agents can loop until a check passes. Planning moves to capacity: fair-use queues, priority for interactive work, and a trigger for adding GPUs.

How do you choose between American and Chinese open-weight models?

Compare licence, size, languages and your own test results. From maker pages, fetched 22 November 2026:

Model (maker)LicenceSizeNotes
gpt-oss-120b (OpenAI, US)Apache 2.0117B, 5.1B activeSingle 80GB GPU
Llama 3.3 (Meta, US)Community licence; separate licence above 700M monthly users70B8 languages, 128k
Phi-4 (Microsoft, US)MIT14BMainly English, 16K
Qwen3 (Qwen, China)Apache 2.08B to 235B100+ languages
DeepSeek-V3 (China)Model licence, commercial use allowed671B, 37B active128K
Kimi K2 (Moonshot AI, China)Modified MIT, attribution at large scale1T, 32B active128K
GLM-4.5 (Z.ai, China)MIT355B, 32B activeEnglish, Chinese

Re-read the licence of the exact version you deploy.

What guardrails does the move need?

  • Data handling. Self-hosting keeps prompts in house; set a log retention policy.
  • Licence review. Record licence and obligations per model in an AI-BOM.
  • Weights provenance. Load safetensors, refuse pickle files, pin revisions and check hashes (signing weights).
  • Prompt injection. Treat retrieved content and tool output as untrusted (OWASP LLM01, direct vs indirect).

Where does Safeguard fit?

Safeguard can run open-weight models, American and Chinese, on fixed-cost capacity, so AI cost is static rather than per token. Companies we have worked with have cut AI cost by up to about 90% and now use AI without usage limits.

In the Safeguard platform the AI mode is set per tenant: Full AI, Zero AI, bring your own model, or local and on-prem models. See safeguard.sh/pricing for current plans.

FAQ

How much can open-weight models cut AI costs?

Up to about 90% for high-volume, routable tasks on busy fixed capacity; far less at low volume.

Are open-weight models good enough for production?

For many routine tasks, yes. Your own evaluation set decides which.

Do I still need a frontier model?

Usually, as a fallback. Its share drives the savings.

Does quantization hurt quality?

Sometimes slightly. Measure it first.

Can Chinese open-weight models be used commercially?

The Qwen3, DeepSeek-V3, Kimi K2 and GLM-4.5 pages allow it under their licences. Check each licence and your own policy.

How do I budget when usage is unlimited?

Plan capacity instead: fair-use queues, priorities and a GPU trigger.

Inference servers and their container images are open source software, so the stack you stand up needs patching like any other service.

Fix it without waiting for upstream

Whatever the vulnerability, most of the risk sits in two gaps. The first is the time between a fix being published and that fix actually running in your production builds. The second, and worse, is the time before any fix exists at all, while a maintainer works out a patch and a release schedule. Waiting through both gaps is a habit, not a requirement. Safeguard gives teams three ways to close them.

1. Autonomous remediation: pull requests, not tickets

When a fixed version exists, Griffin, Safeguard's security model, opens a policy gated pull request that moves the affected dependency to the safe version, and your own CI runs your tests on it before anyone merges. You can run it on one repository or in bulk across every project that carries the vulnerable version. Remediation categories include dependencies, licenses, security posture and findings, and each one is set separately. Reachability analysis ranks the vulnerable code paths your application actually calls first, so the first pull requests you merge are the ones that matter most. Auto-merge is a separate switch and stays off by default, so your team decides what merges on its own and what still needs a human approval.

2. Gold Open Source: the safe version in seconds, free

gold.safeguard.sh is free and needs no login. Search any CVE or package to see affected and fixed versions, CVSS, EPSS, CISA KEV status and a recommended safe version, or use the "Am I affected?" checker. The same data is available through a free read API, README status badges and RSS threat feeds.

3. Gold Registry: start from components that are already fixed

Safeguard Gold rebuilds patched versions of open-source packages and images from source, signs each build, attaches an SBOM, SLSA provenance and VEX, and serves them from registry.safeguard.sh to your organization. Every Gold artifact is held continuously to zero exploitable critical or high CVEs and a completed license check, with signed build provenance (signed with Safeguard's key and verifiable with cosign, SLSA Build Level 3 as the design target) and a public evidence page you can hand to an auditor. An artifact loses its Gold badge if it regresses. Teams can point their own package directory at the registry so hardened equivalents resolve without changing how developers install anything.

When upstream has no fix yet, or will not ship one, Safeguard first works to get the fix accepted upstream. If the issue is severe or upstream will not move, Safeguard builds a fork, fixes the vulnerability there and publishes a hardened Gold build, so you are not stuck on a maintainer's release schedule. You can also request a hardened build of a specific package. Ask which packages it covers before you plan around it.

Your situationWhat to use
A fixed version existsAutonomous remediation opens the upgrade pull request; Gold Open Source confirms the safe version
You need to know today whether you are affectedGold Open Source lookup and "Am I affected?" checker, free and without login
You want vulnerable versions to stop shippingSafeguard policies and gates in CI
Upstream has no fix yet, or will not ship oneA custom Gold request; fork and patch
You want to start clean on new projectsPull hardened components and images from Gold Registry

One honest limit: when the vulnerable code lives in a vendor appliance, a hosted service or an operating system runtime you do not build yourself, the fix still has to come from that vendor or distribution. The steps above apply to the open source dependencies and container images you build and ship, which is where most teams carry the largest and least tracked share of their exposure.

See how Safeguard would help

This demo connects Claude to the Safeguard MCP server with OAuth and picks which tools it may use.

Click through it at your own pace, or open it full screen.

Sources

Last reviewed: 22 November 2026.

Never miss an update

Weekly insights on software supply chain security, delivered to your inbox.

We email you a link to confirm, and only then add you to the list. Unsubscribe in one click. Privacy

Self-healing security runs on Safeguard.

Your first fix PR is minutes away.

No sales call required, even your agent can complete the purchase over MCP.