Teams cut AI spend by up to about 90% when they move high-volume, routine tasks off per-token frontier APIs and onto open-weight models running on capacity they pay for at a fixed rate. The bill stops growing with every token and becomes a fixed line. A frontier model stays as a fallback for the few tasks that need it.
This is the savings playbook: where the bill goes, eight levers, a migration plan and a cost model. For the definition of fixed-cost AI and a break-even calculation, read the fixed-cost AI explainer.
Key takeaways
- Most AI spend comes from high-volume tasks, long contexts and agent loops.
- Eight levers cut it: routing, quantization, context trimming, prefix caching, batching, speculative decoding, answer caching and capacity scheduling.
- Savings near 90% need high, steady volume and tasks a smaller model passes on your own tests.
- Choose models on licence, size, languages and your own evaluation results.
Where does an AI bill actually go?
Export a month of usage by use case: input and output tokens, requests, retries and model. Input usually dominates, because retrieved documents, history and tool output ride along on every call. Agent loops resend a growing context, and retries add calls that produce nothing.
Classification, extraction, summaries and tagging rarely need a frontier model. Long multi-step reasoning might. See agentic AI budget controls and enterprise LLM budget patterns.
What are the 8 levers that cut AI inference cost?
- Task routing. Send each request to the smallest model that passes your tests for that task. Measure pass rate per task.
- Quantization. Lower precision fits more model per GPU. OpenAI's card says gpt-oss-120b uses MXFP4 MoE weights and fits a single 80GB GPU (card). Measure quality and throughput.
- Context trimming. Cap retrieved chunks and strip tool output. Measure input tokens per request.
- Prefix caching. Keep system prompts and tool schemas first and identical. vLLM reuses the KV cache for queries that share a prefix (docs). Measure hit rate and time to first token.
- Batching. Run offline work in batches that fill idle capacity. Measure GPU utilisation by hour.
- Speculative decoding. A draft model proposes tokens the main model checks (docs). Measure accepted tokens per step.
- Answer caching. Store answers to repeated questions with an expiry. Measure hit rate and stale answers.
- Right-sizing and scheduling. Size GPUs to steady load and queue low-priority work behind interactive work. Measure queue wait and idle hours.
How do you migrate from a frontier API without losing quality?
- Baseline cost, latency and quality per use case for 2 to 4 weeks.
- Build an evaluation set of a few hundred real, anonymised inputs per task (how to build one; judge pitfalls).
- Shadow traffic: copy live requests to the candidate and log its answers without serving them.
- Cut over by task, highest volume and lowest risk first.
- Keep a frontier fallback for low confidence or failed schema checks.
- Monitor pass rate, fallback rate, latency and queue wait.
Evaluation checklist: pass rate within your margin of the baseline, schema checked on every response, user languages covered, fallback rate low, rollback tested.
How does a cost model reach about 90%, and when does it not?
A worked example. Every value is an assumption, not a measured result.
| Assumption | A: high volume | B: low volume | C: half need frontier |
|---|---|---|---|
| Tokens per month (80% input) | 30 billion | 1 billion | 30 billion |
| API price per million (input, output) | 3, 15 | 3, 15 | 3, 15 |
| API baseline per month | 162,000 | 5,400 | 162,000 |
| GPUs at 2.50 per hour, 730 hours | 6: 10,950 | 1: 1,825 | 6: 10,950 |
| Frontier fallback | 3%: 4,860 | 3%: 162 | 50%: 81,000 |
| Operations overhead | 2,000 | 2,000 | 2,000 |
| Total | 17,810 | 3,987 | 93,950 |
| Saving | about 89% | about 26% | about 42% |
Scenario A assumes about 1,850 tokens per second per GPU (input and output together), all month; prove it with a load test. Low or bursty volume and high fallback erode the saving.
What changes when AI usage is unlimited?
The next request costs nothing extra, so teams stop rationing and agents can loop until a check passes. Planning moves to capacity: fair-use queues, priority for interactive work, and a trigger for adding GPUs.
How do you choose between American and Chinese open-weight models?
Compare licence, size, languages and your own test results. From maker pages, fetched 22 November 2026:
| Model (maker) | Licence | Size | Notes |
|---|---|---|---|
| gpt-oss-120b (OpenAI, US) | Apache 2.0 | 117B, 5.1B active | Single 80GB GPU |
| Llama 3.3 (Meta, US) | Community licence; separate licence above 700M monthly users | 70B | 8 languages, 128k |
| Phi-4 (Microsoft, US) | MIT | 14B | Mainly English, 16K |
| Qwen3 (Qwen, China) | Apache 2.0 | 8B to 235B | 100+ languages |
| DeepSeek-V3 (China) | Model licence, commercial use allowed | 671B, 37B active | 128K |
| Kimi K2 (Moonshot AI, China) | Modified MIT, attribution at large scale | 1T, 32B active | 128K |
| GLM-4.5 (Z.ai, China) | MIT | 355B, 32B active | English, Chinese |
Re-read the licence of the exact version you deploy.
What guardrails does the move need?
- Data handling. Self-hosting keeps prompts in house; set a log retention policy.
- Licence review. Record licence and obligations per model in an AI-BOM.
- Weights provenance. Load safetensors, refuse pickle files, pin revisions and check hashes (signing weights).
- Prompt injection. Treat retrieved content and tool output as untrusted (OWASP LLM01, direct vs indirect).
Where does Safeguard fit?
Safeguard can run open-weight models, American and Chinese, on fixed-cost capacity, so AI cost is static rather than per token. Companies we have worked with have cut AI cost by up to about 90% and now use AI without usage limits.
In the Safeguard platform the AI mode is set per tenant: Full AI, Zero AI, bring your own model, or local and on-prem models. See safeguard.sh/pricing for current plans.
FAQ
How much can open-weight models cut AI costs?
Up to about 90% for high-volume, routable tasks on busy fixed capacity; far less at low volume.
Are open-weight models good enough for production?
For many routine tasks, yes. Your own evaluation set decides which.
Do I still need a frontier model?
Usually, as a fallback. Its share drives the savings.
Does quantization hurt quality?
Sometimes slightly. Measure it first.
Can Chinese open-weight models be used commercially?
The Qwen3, DeepSeek-V3, Kimi K2 and GLM-4.5 pages allow it under their licences. Check each licence and your own policy.
How do I budget when usage is unlimited?
Plan capacity instead: fair-use queues, priorities and a GPU trigger.
Inference servers and their container images are open source software, so the stack you stand up needs patching like any other service.
Fix it without waiting for upstream
Whatever the vulnerability, most of the risk sits in two gaps. The first is the time between a fix being published and that fix actually running in your production builds. The second, and worse, is the time before any fix exists at all, while a maintainer works out a patch and a release schedule. Waiting through both gaps is a habit, not a requirement. Safeguard gives teams three ways to close them.
1. Autonomous remediation: pull requests, not tickets
When a fixed version exists, Griffin, Safeguard's security model, opens a policy gated pull request that moves the affected dependency to the safe version, and your own CI runs your tests on it before anyone merges. You can run it on one repository or in bulk across every project that carries the vulnerable version. Remediation categories include dependencies, licenses, security posture and findings, and each one is set separately. Reachability analysis ranks the vulnerable code paths your application actually calls first, so the first pull requests you merge are the ones that matter most. Auto-merge is a separate switch and stays off by default, so your team decides what merges on its own and what still needs a human approval.
2. Gold Open Source: the safe version in seconds, free
gold.safeguard.sh is free and needs no login. Search any CVE or package to see affected and fixed versions, CVSS, EPSS, CISA KEV status and a recommended safe version, or use the "Am I affected?" checker. The same data is available through a free read API, README status badges and RSS threat feeds.
3. Gold Registry: start from components that are already fixed
Safeguard Gold rebuilds patched versions of open-source packages and images from source, signs each build, attaches an SBOM, SLSA provenance and VEX, and serves them from registry.safeguard.sh to your organization. Every Gold artifact is held continuously to zero exploitable critical or high CVEs and a completed license check, with signed build provenance (signed with Safeguard's key and verifiable with cosign, SLSA Build Level 3 as the design target) and a public evidence page you can hand to an auditor. An artifact loses its Gold badge if it regresses. Teams can point their own package directory at the registry so hardened equivalents resolve without changing how developers install anything.
When upstream has no fix yet, or will not ship one, Safeguard first works to get the fix accepted upstream. If the issue is severe or upstream will not move, Safeguard builds a fork, fixes the vulnerability there and publishes a hardened Gold build, so you are not stuck on a maintainer's release schedule. You can also request a hardened build of a specific package. Ask which packages it covers before you plan around it.
| Your situation | What to use |
|---|---|
| A fixed version exists | Autonomous remediation opens the upgrade pull request; Gold Open Source confirms the safe version |
| You need to know today whether you are affected | Gold Open Source lookup and "Am I affected?" checker, free and without login |
| You want vulnerable versions to stop shipping | Safeguard policies and gates in CI |
| Upstream has no fix yet, or will not ship one | A custom Gold request; fork and patch |
| You want to start clean on new projects | Pull hardened components and images from Gold Registry |
One honest limit: when the vulnerable code lives in a vendor appliance, a hosted service or an operating system runtime you do not build yourself, the fix still has to come from that vendor or distribution. The steps above apply to the open source dependencies and container images you build and ship, which is where most teams carry the largest and least tracked share of their exposure.
See how Safeguard would help
This demo connects Claude to the Safeguard MCP server with OAuth and picks which tools it may use.
Click through it at your own pace, or open it full screen.
Sources
- gpt-oss-120b
- Llama 3.3 licence and card
- Phi-4
- Qwen3-8B
- DeepSeek-V3
- Kimi K2
- GLM-4.5
- vLLM prefix caching
- vLLM speculative decoding
- safetensors
- OWASP LLM01
Last reviewed: 22 November 2026.