Why latency matters more than model quality
Above ~1.2 seconds of first-word delay, callers start filling the silence and stepping on the agent — the conversation collapses into cross-talk. Below ~800ms it feels like a real conversation. A slightly dumber model that responds in 600ms will outperform a smarter one that responds in 1.4s on nearly every metric that matters: completion rate, transfer rate, customer satisfaction, average handle time. Optimize latency before you optimize prompt cleverness.
1. Stream everything
Streaming STT, streaming LLM, streaming TTS. If any of the three is non-streaming, you're waiting for the full response before the next stage starts — that's how you accidentally add 500ms. Check every provider's docs for the streaming endpoint (usually SSE or WebSocket) and confirm Vapi is configured to use it. This is the single biggest lever, and it's the one that most naive setups get wrong.
2. Pick a smaller first-turn LLM
Use GPT-4o mini (or an equivalent small model like Claude 3.5 Haiku or Llama 3.1 8B) on the hot path. Reserve GPT-4o full or Claude 3.5 Sonnet for tool-call decisions where quality actually matters. On most agents, 90% of turns are conversational fillers ('okay', 'let me check', 'and your account number?') that don't need a frontier model. A model router that switches based on intent complexity is the pro move.
3. Lower STT endpointing thresholds
Default endpointing waits ~500ms of silence before deciding the turn is over. Drop it to ~200–300ms for snappier turns. Too low and you'll cut people off mid-word; tune it against real call recordings, not vibes. If your callers are older or non-native speakers, they pause longer mid-sentence — keep the threshold higher for those audiences.
4. Co-locate providers in the same region
Pick STT, LLM, and TTS providers in the same region as your Vapi deployment and telephony leg. Cross-region hops add 40–120ms per stage — three stages × 80ms average = a full quarter-second of avoidable latency. Consistent US-East or EU-West across the stack is the fastest. Vapi's dashboard shows the region for each provider connection; if any of them is 'auto', pin it.
5. Shorter system prompts
Every token in the system prompt is retokenized on every turn. A 3,000-token system prompt costs meaningful latency vs a tight 500-token one — often 100–200ms just in tokenization + prefill. Move examples into few-shot messages the LLM can cache, put only immutable rules in the system prompt, and audit every 6 months to delete rules that no longer apply.
6. Faster TTS or a filler word
If TTS is the bottleneck, switch from ElevenLabs Multilingual v2 to Turbo, or to Deepgram Aura for latency-critical paths. Alternatively, have the agent say a natural filler ('let me check…', 'one moment…') while the real response is computed. The filler starts audio playback immediately, buying you 400–600ms of LLM budget without the caller noticing.
7. Warm your voice and route
First-call latency is often 200–400ms worse than steady-state because voices and LLM routes cold-load. Configure Vapi to pre-warm the assistant on assistant creation and keep it warm with a periodic no-op call in dev. On serverless deployments, this matters even more.
The traps that add 500ms silently
- Tool calls with big JSON responses — parse them and respond with the summary, not the whole payload.
- Non-streaming custom LLM endpoints — check the SSE headers, don't trust the docs.
- Barge-in disabled — the agent has to finish speaking before responding to interruption.
- TTS voice loaded on-demand instead of warm — first turn pays the cold-load penalty.
- Debug logging in the tool-call path — synchronous log writes are surprisingly expensive.
- Retrying failed tool calls without a timeout — a 5s API stall becomes a 5s conversational dead zone.
Measuring what matters
Vapi exposes per-turn latency broken down by stage (STT, LLM, TTS, network). Export this to your analytics store and track P50 and P95 first-word latency per assistant per week. Regressions almost always come from a prompt edit that added tokens, a provider region change, or a new tool with a slow endpoint. Alerting on P95 > 1.0s catches most incidents before customers notice.
Real-world latency budgets by use case
Different agents can afford different latency budgets. A high-touch consumer sales agent needs sub-700ms to feel warm; anything slower makes the caller doubt they're on the line with something intelligent. An enterprise support agent handling detailed lookups can get away with 900ms if the persona is calm and considered — callers actually prefer a small deliberative pause when the question is complex. An outbound qualification agent, where the goal is to complete a short script fast, needs sub-600ms because every extra ms multiplies across thousands of calls per campaign. Match the budget to the persona; don't chase 500ms uniformly.
A note on 'perceived' vs 'measured' latency
Measured latency is what the dashboard says. Perceived latency is what the caller experiences, and it's often 100–200ms worse because of caller-side network jitter, phone hardware buffering, and the caller's own attention state. Anecdotally, agents that measure at 800ms perceive around 950ms. Design against perceived, not measured — that's why the working target is under 800ms measured, giving you a buffer against the perception gap.
When to stop optimizing
There's a point at which further latency reduction stops improving business metrics. Below roughly 500ms first-word latency, the marginal completion-rate improvement is inside the noise floor. If you're at 600ms and considering a project to hit 400ms, sanity-check with an A/B first. Often the wins have moved elsewhere: voice quality, prompt clarity, tool reliability. The best voice teams treat latency as one of five levers and rotate attention across them, not as the only lever.
Latency budget breakdown
Treat your 800ms first-word target as a budget and allocate it stage by stage. A realistic split on a well-tuned stack: endpointing 250ms (the silence timer after the caller stops), STT finalization 50ms (streaming means most transcription is already done by the time endpointing fires), LLM first-token 200ms (small model on hot inference), TTS first-audio 150ms (streaming Turbo variant), network + telephony 100ms, safety margin 50ms. Total: 800ms. When you exceed budget, measure per-stage first — Vapi's dashboard breaks it down — and attack the stage that's over. The three most common budget violations are LLM first-token at 400ms+ because the prompt is too long, TTS first-audio at 300ms+ because the voice is non-streaming or cold, and endpointing at 500ms+ because defaults were never tuned. Fix in that order and most agents drop 300ms in a single afternoon.
Tuning tips per stage
STT: pick a streaming provider (Deepgram Nova, AssemblyAI real-time), pin the region to match your telephony leg, and set interim results to enabled so endpointing sees partials as they arrive. LLM: use a small model on the hot path (GPT-4o mini, Claude 3.5 Haiku, Llama 3.1 8B on Groq), enable prompt caching, keep the system prompt under 800 tokens, and route only tool-call decisions to a bigger model. TTS: use the streaming variant (ElevenLabs Turbo, Deepgram Aura, Cartesia Sonic), pre-warm the voice at assistant creation, and cache your greeting so the first turn skips synthesis entirely. Network: co-locate all providers in one region, prefer US-East for widest provider coverage, and use a BYOC telephony provider close to your caller base for international. Each stage independently is a 50–150ms win; stacked, they routinely turn a 1.2s agent into a 700ms agent without changing the prompt.