Voice AI Latency: Why 200 Milliseconds Matters and Where a Voice Agent Spends Its Time

People in conversation usually start replying about 200 milliseconds after the other person stops, and a gap of 600 milliseconds or more starts to carry meaning of its own, typically reluctance. A voice agent is judged against that standard whether or not its builders intend it to be. Most of the delay in a voice agent comes from waiting, not from computing, and the biggest single wait is usually deciding that the user has finished.

This post is for teams building or buying voice agents for support, sales or internal tools. It covers what the conversation research says about timing, where the milliseconds go in a typical agent pipeline, why a pipeline that runs its steps one after another can't reach human timing, and a script that breaks your own agent's delay down by stage.

The short answer

Conversation research finds that the gaps between human turns are typically around 200 ms, and that more than half of turn transitions happen in under 200 ms. Gaps of 600 ms or longer lead listeners to infer an unwelcome response. A voice agent's response gap is the sum of endpointing (deciding the user has stopped), transcription, the model's time to first token, speech synthesis to first audio, and playback buffering. Run one after another, these stages usually land well above 600 ms, so agents that feel natural overlap them.

What the research says about timing

The most useful summary is a 2015 review by Stephen Levinson and Francisco Torreira in Frontiers in Psychology. Its core numbers:

• Gaps between turns are short, "of the order of 200 ms".
• 51% to 55% of turn transitions in the corpora they cite happen in under 200 ms.
• Producing speech is slow. Planning even a single word takes around 600 ms, and longer utterances take 600 to 1,500 ms or more before speech begins.
• Silence carries meaning. Gaps over 300 ms make an unqualified "yes" less likely, and experiments show that gaps of 600 ms or more lead listeners to infer an unwelcome response.

The gap between 200 ms and 600 ms is the puzzle the review is about. People reply faster than they could possibly plan a reply from scratch, because they predict where the other person's turn will end and start planning their response while still listening.

Timing norms hold up across languages too. A 2009 study in PNAS led by Tanya Stivers compared 10 languages, from small indigenous communities to major world languages. All of them showed people avoiding overlapping talk and keeping silence between turns to a minimum, with the average gap in each language within 250 ms of the cross-language mean.

For a voice agent, the practical reading is simple. Replies in the low hundreds of milliseconds sound attentive. Replies past about 600 ms start to sound hesitant, whatever the words say.

Where a voice agent spends its time

A typical voice agent pipeline has five stages between the user's last word and the agent's first sound:

The codec contributes too, though not much. Opus, a common codec for real-time voice, encodes frames of 2.5 to 60 ms, according to RFC 6716, and a frame has to fill before it's sent. That's tens of milliseconds at most, small next to the other stages.

Endpointing is the stage people underestimate. If an agent waits for 700 ms of silence before deciding a turn is over, it has spent the whole human budget before transcription even finishes. Shorten the wait too much and the agent interrupts people who paused to think. That trade-off is a product decision, not a tuning detail.

Why a sequential pipeline can't reach 200 ms

Add the stages up and the problem is clear. Even with fast components, a pipeline that waits for silence, then transcribes, then generates, then synthesizes, then plays back, adds every delay to the next one. Humans don't work that way. They overlap listening and planning, and the agents that feel natural do the same:

• Predict the end of the turn from words, intonation and pacing instead of waiting only for silence.
• Start the model on a partial transcript and discard the work if the user keeps talking.
• Stream everything: tokens from the model into speech synthesis, and audio chunks out as soon as they exist.
• Say something short first, like an acknowledgment, while the full answer is still being generated.
• Handle barge-in: when the user starts talking over the agent, stop playback fast and listen.

Each of these trades accuracy or cost for time. Predictive endpointing will sometimes cut people off. Speculative generation burns tokens on replies that get thrown away. There's no free setting, which is why measuring each stage matters.

Measure your own agent

Most voice frameworks can log a timestamp for each event. Record, on one clock, when the user's speech ended, when the turn was endpointed, when the final transcript arrived, when the first model token arrived, when the first synthesized audio arrived, and when playback started. This script reads those events as JSON Lines and reports each stage's median and 90th percentile, its share of the total, and how many turns crossed 600 ms:

#!/usr/bin/env python3
"""Break a voice agent's response gap into stages from per-turn timestamps.

Input: JSON Lines, one object per turn, times in milliseconds on one clock:
  {"turn": 1, "speech_end": 0, "endpoint": 520, "asr_final": 610,
   "llm_first_token": 980, "tts_first_audio": 1150, "playback_start": 1190}
speech_end is when the user stopped talking (from VAD or audio), and
playback_start is when the first agent audio reached the speaker.

Usage: python3 turn_latency.py turns.jsonl
"""
import json
import statistics
import sys

STAGES = [
    ("endpointing", "speech_end", "endpoint"),
    ("final transcript", "endpoint", "asr_final"),
    ("model first token", "asr_final", "llm_first_token"),
    ("speech first audio", "llm_first_token", "tts_first_audio"),
    ("playback", "tts_first_audio", "playback_start"),
]
# From conversation research (Levinson and Torreira 2015): typical gaps between
# human turns are about 200 ms, and gaps of 600 ms or more read as reluctance.
HUMAN_GAP_MS, UNWELCOME_MS = 200, 600


def pct(values, q):
    values = sorted(values)
    k = (len(values) - 1) * q
    lo, hi = int(k), min(int(k) + 1, len(values) - 1)
    return values[lo] + (values[hi] - values[lo]) * (k - lo)


def main():
    turns = []
    with open(sys.argv[1]) as f:
        for line in f:
            if line.strip():
                turns.append(json.loads(line))
    if not turns:
        raise SystemExit("no turns")
    print(f"{len(turns)} turns")
    print(f"{'stage':<20}{'p50 ms':>8}{'p90 ms':>8}{'share of p50 gap':>18}")
    gaps = [t["playback_start"] - t["speech_end"] for t in turns]
    p50_gap = pct(gaps, 0.5)
    for name, a, b in STAGES:
        d = [t[b] - t[a] for t in turns if a in t and b in t]
        if not d:
            continue
        if min(d) < 0:
            print(f"warning: {name} has negative durations; check that all times share one clock")
        p50 = pct(d, 0.5)
        print(f"{name:<20}{p50:>8.0f}{pct(d, 0.9):>8.0f}{p50 / p50_gap:>17.0%}")
    print(f"{'response gap':<20}{p50_gap:>8.0f}{pct(gaps, 0.9):>8.0f}")
    slow = sum(1 for g in gaps if g >= UNWELCOME_MS)
    print(f"turns at or above {UNWELCOME_MS} ms: {slow} of {len(gaps)} ({slow / len(gaps):.0%}); "
          f"typical human gap is about {HUMAN_GAP_MS} ms")


if __name__ == "__main__":
    main()

We ran it on two synthetic logs of 200 turns each. The numbers are invented to show the method, not measured from any product. The first agent waits about 700 ms of silence before endpointing. The second waits about 250 ms and has a faster model:

== fx/agent_silence_700.jsonl
200 turns
stage                 p50 ms  p90 ms  share of p50 gap
endpointing              731     752              50%
final transcript          94     130               6%
model first token        456     662              31%
speech first audio       148     208              10%
playback                  42      56               3%
response gap            1470    1671
turns at or above 600 ms: 200 of 200 (100%); typical human gap is about 200 ms
== fx/agent_silence_250.jsonl
200 turns
stage                 p50 ms  p90 ms  share of p50 gap
endpointing              280     302              33%
final transcript          98     130              11%
model first token        298     464              35%
speech first audio       155     209              18%
playback                  40      55               5%
response gap             856    1056
turns at or above 600 ms: 200 of 200 (100%); typical human gap is about 200 ms

In the first agent, half the median gap is spent waiting for silence. Cutting that wait to 250 ms saves more time than the faster model does. Both still put every turn above 600 ms, because both run the stages in sequence. That's the result to look for in your own data: if one stage dominates, fix it first, and if the total is still too high, the next gain comes from overlapping stages, not shaving them.

Measure the 90th percentile as well as the median. A gap that is usually fine but occasionally doubles, because a tool call or a cold model adds delay, is what users remember.

Model choice is one of the few large levers on the "model first token" stage. Here's Safeguard's interactive walkthrough of choosing which model runs behind its assistant:

What to look for when choosing a voice platform

1. Ask for stage-level timing, not one latency number. A single "response time" figure hides where the delay is and whether it's measured from the user's last word or from the end of endpointing.
2. Check how endpointing works and whether you can tune it. Silence-only endpointing with a fixed long wait is the most common source of slow turns.
3. Check what streams. Transcription, generation and synthesis should all stream, and synthesis should start before the full reply exists.
4. Test barge-in. Talk over the agent and see how fast it stops.
5. Test with your own prompts, tools and network. Tool calls and retrieval add delay before the first token, and a demo with no tools won't show it.
6. Look at the 90th percentile under realistic load, not only the median on a quiet day.

The fair view

Human timing isn't the only target that matters. Callers to a support line tolerate a slower, accurate answer over a fast wrong one, and some deliberately slow agents score well because they never interrupt. The research also measures casual conversation between people who know each other's rhythms, which isn't the same as a customer talking to a service.

Still, the 600 ms finding is hard to ignore. Silence that long reads as hesitation in ordinary conversation, and listeners don't switch that inference off because the other side is a machine. The most effective design work in voice agents is less about faster models than about not waiting: predicting turn ends, starting early and streaming throughout.

Check yourself