Generating text is the slowest step in an AI request because the model produces output one token at a time, while input can be processed in bulk. OpenAI's latency guide says generating tokens is almost always the highest latency step, and that halving output tokens may roughly halve latency.
What the guide says
OpenAI's latency optimization guide gives a rough rule: cutting output tokens by half may cut latency by about half, while cutting input tokens by half may improve latency by only one to five percent. It lists six principles: process tokens faster, generate fewer tokens, use fewer input tokens, make fewer requests, parallelize, and stream.
| Principle | Example from the guide |
|---|---|
| Process tokens faster | Use a smaller model, fine-tuning, or predicted outputs for code edits |
| Generate fewer tokens | Ask for brevity, shorten structured output keys, set a max or stop sequence |
| Use fewer input tokens | Usually minor unless contexts are large; put dynamic content later for caching |
| Make fewer requests | Combine sequential steps into one prompt |
| Parallelize | Run independent steps concurrently |
| Stream | Show partial output so users wait about a second instead of seeing nothing |
These are one provider's measurements, so test on your own workload.
Why output also costs more
Providers usually bill output tokens at a higher rate than input tokens, and reasoning tokens bill as output. Both effects point the same direction. See input vs output tokens for the billing side and thinking tokens for reasoning.
The practical result is that verbose output hits you twice, in wait time and in the bill. A prompt that asks for a two sentence answer saves both.
How to shorten output without losing value
- State the format and length you want. "Return a JSON object with these three keys" beats an open request.
- Shorten field names in structured output when the output is machine read.
- Ask for the answer first and the reasoning only when needed.
- Set a max output value near your expected size. Some providers count the larger of that value and an estimate toward limits, per OpenAI's guide, so a huge max can hurt.
- Use stop sequences when the answer has a clear end marker.
- Cut preambles and restated questions in your instructions.
Where input still matters
Input is cheap in time but not free in cost, and a very large prompt can hit limits or context rot, as covered in what counts toward a context window. Reusing a cached prefix skips reprocessing it, so the response starts sooner. OpenAI's caching page says this reduces the time to first token. See how prompt caching works.
Streaming changes perceived speed, not total time
Streaming shows tokens as they are generated, so users see progress quickly. Total generation time is unchanged. It is the right choice for chat and a poor fit for jobs where a program needs the full result before it can continue.
Measure two numbers
Track time to first token and total time per request, separately. A slow start suggests a large input or a cold cache. A slow finish suggests long output or a big model. Without both you will fix the wrong one.
Worked example: cutting a response in half
These timings are illustrative. A report summarizer sends a 3,000 token prompt and gets an 800 token answer in about eight seconds. The team tests two edits separately.
| Change | New size | Rough effect, per the guide's rule of thumb |
|---|---|---|
| Halve the input to 1,500 tokens | Input halved | Small gain, about one to five percent |
| Halve the output to 400 tokens | Output halved | Time drops by about half |
The second edit saves far more time. The team asks for three bullet points instead of a paragraph per section, and uses short JSON keys. User-visible time falls from eight seconds to roughly four, and the output portion of the bill falls too. Your own numbers will differ, so measure.
Prompt edits that shorten output
- Replace "explain" with a format: "three bullets, each under twenty words."
- Remove requests to restate the question or add a conclusion that repeats the body.
- Ask for only the changed lines when editing code, not the whole file.
- Ask for a classification label instead of a justification when no one reads the justification.
- For machine consumers, use compact structured output.
When longer output is right
Do not cut output that carries value. A legal summary that omits a key clause is worse than a slow one. Set quality checks on a sample before and after shortening, and compare. Also be careful with reasoning: shortening the reasoning budget can lower answer quality on hard tasks.
Measurement plan
Log time to first token, total time, input tokens, output tokens and reasoning tokens for every request. Chart p50 and p95 by prompt version. When you change a prompt, compare the two versions on the same test set.
Parallel steps and fewer round trips
The guide also suggests combining sequential steps into one prompt and running independent steps in parallel. If a workflow makes three calls where each waits for the last, ask whether one carefully structured call could return all three answers. If two calls do not depend on each other, send them together. Both changes cut waiting without touching the model. Test each one for quality, because a crowded prompt can reduce accuracy on some tasks.
Setting a latency budget
Decide how long a user should wait and divide that budget. If the target is four seconds for a complete answer, and generation runs at a measured rate on your model, you can compute the largest answer that fits. That turns a vague wish for speed into a concrete output length target, which you can enforce with prompt wording and a ceiling.
Track the budget in your tests. Add a check that fails when the median or the slow tail exceeds the target on your evaluation set. A prompt edit that adds a paragraph of preamble then shows up as a failed test rather than a customer complaint a month later.
What this means for you
If your AI feature feels slow, look at the length of what it writes before you look at anything else. Shortening the output is usually the biggest lever and it also lowers cost. Ask for what the reader needs, in the shape they need, and nothing more.
Common mistakes
People trim the input to save time and see little change. They ask for long explanations when a label would do. They leave a huge output ceiling in place so runaway answers are possible. They stream and call it faster, when streaming only changes what the user sees first. And they measure total time only, which hides whether the delay comes from the start or the finish. Track both time to first token and total time, shorten the output, then compare on a fixed test set.
Try it yourself
Click through it at your own pace, or open it full screen. It shows an assistant answering step by step. Watch how long each reply is and ask whether a shorter, structured answer would serve the reader equally well.
Quick check
Will shortening my prompt make responses much faster?
Usually only a little, per OpenAI's guide, unless the context is very large. Shortening the output usually matters more.
Does streaming make the model faster?
No. It improves perceived speed by showing partial output sooner.
Does caching help latency?
Yes for the repeated prefix. Reusing it skips reprocessing, so the response starts sooner.
Sources
- Latency optimization, OpenAI API documentation
- Prompt caching, OpenAI API documentation
- Context windows, Claude API documentation
Last reviewed: 19 October 2026.
