A backend proxy for LLM calls earns its place by making decisions the client cannot: who is calling, which model, how much input, how much output, how often and at what cost. If your proxy only forwards requests with a key attached, it hides the key but leaves every other control missing.
Check 1: identity first
Every request needs an authenticated user or service identity before it reaches the provider. Anonymous endpoints are where cost abuse starts. If you must offer an anonymous trial, give it its own tight limits and a separate key, so an attack on the trial cannot starve paying customers.
Check 2: the server picks the model and the settings
Do not accept a model name, a maximum output length or a system prompt from the client. Map a feature to a fixed server-side configuration. A client that can choose the model can choose the most expensive one. A client that can set the output limit can ask for the largest answer. A client that can set the system prompt can remove your instructions.
Check 3: input size limits
Cap characters or, better, tokens on every input field, including file uploads and pasted text. The OWASP guidance on unbounded consumption lists input validation with size limits as the first prevention measure, because oversized inputs exhaust resources and raise cost. If you do retrieval, cap how many chunks you add to the prompt as well.
Check 4: rate and quota per user
Provider limits apply to your whole organization. They protect the provider, and they do not protect one customer from another. One heavy user can use up your shared capacity and cause 429 errors for everyone. Apply a per-user and per-tenant limit in your own code. See per-user limits for LLM features.
Check 5: timeouts and concurrency caps
Set a timeout on the upstream call and cap how many calls one user can have in flight. Long requests held open tie up your workers. The OWASP list names timeouts and throttling as a way to bound expensive operations.
Check 6: structured logging with care
Log who called, which feature, which model, token counts, latency and the provider's request ID. Do not log raw prompts by default. The reasons and the alternatives are in logging LLM interactions without leaking sensitive data. The request ID is worth keeping because support teams ask for it when you report a problem.
Check 7: output handling
The proxy is also the place to enforce output rules: maximum size returned to the client, content checks where you need them, and stripping of fields the client should not see, such as raw tool call data or internal metadata. Treat model output as untrusted input to the rest of your system.
What a proxy does not do
It does not stop prompt injection. If the model can call tools or see private data, the proxy has to limit what those tools can reach. A system prompt alone is not a barrier, as covered in can a system prompt stop prompt injection.
Safeguard's policies and gates can enforce checks like these in the build pipeline, for example failing a change that adds a provider key to client code.
Quick check
Is an API gateway product enough as a proxy?
It can handle identity and rate limits well, but it will not know your token budgets, model choices or feature rules unless you configure them. Review that list above against its settings.
Should the proxy cache responses?
Only if the responses are identical for all users who send the same input, and contain no personal data. Otherwise one user may see another's output.
Sources
Last reviewed: 19 October 2026.
