What Vapi actually is
Vapi is an orchestration layer for voice agents. It doesn't invent the STT, LLM, or TTS — you plug those in from providers like Deepgram, OpenAI, and ElevenLabs. What Vapi runs is the low-latency, real-time pipeline that turns those pieces into a phone call that feels like a conversation. Think of it as the conductor: the musicians (providers) already exist, but without a conductor keeping tempo, the concert falls apart. Vapi handles the endpointing, the streaming glue, the interruption logic, and the telephony bridge so you can focus on the prompt and the tools.
The pipeline, turn by turn
When a caller speaks, Deepgram (or Whisper) streams a partial transcript into Vapi within about 150ms of the first phoneme. Vapi's endpointing engine watches that stream, decides when the turn is over using silence + energy thresholds, and hands the finalized text to the LLM. The LLM starts streaming tokens back within another 200–400ms; Vapi pipes those tokens directly into the TTS as they arrive, so audio synthesis begins before the LLM has finished thinking. That audio is then encoded to the telephony codec (usually µ-law at 8kHz for PSTN) and streamed to Twilio or Vonage over a WebSocket. The caller hears the first word roughly 600–900ms after they stopped speaking. Every stage is streaming; no stage waits for the previous stage to finish.
Telephony vs web
Vapi supports Twilio and Vonage numbers out of the box, plus BYO carrier via SIP for enterprise routing. The same assistant can also run on the web SDK — no phone number needed for browser demos. The web SDK uses WebRTC instead of PSTN, which cuts about 80ms of codec overhead but adds browser microphone quirks (autoplay policies, sample-rate mismatches) that don't exist on a phone line.
Assistants, prompts, and tools
An 'assistant' in Vapi is a bundle: system prompt, voice, LLM choice, transcriber choice, tools, and end-call conditions. The system prompt sets persona and rules. Tools are how the agent takes action mid-call — book a meeting, look up an order, transfer to a human. Each tool is a JSON schema plus a webhook URL; the LLM decides when to call it, Vapi forwards the arguments to your endpoint, and the response streams back into the conversation. This is the same function-calling pattern OpenAI popularized, but wrapped for real-time voice.
- System prompt — persona, rules, tone. Keep it under 800 tokens for latency.
- Voice — an ElevenLabs / Deepgram Aura / PlayHT voice ID.
- Model — usually a small streaming LLM on the hot path.
- Tools — JSON-schema function calls to your API.
- End-call — the phrase or condition that hangs up cleanly.
Where the magic (and the latency) hides
First-word latency depends on all three providers plus your telephony region. Streaming everywhere, a small first-turn LLM, and a co-located TTS are the difference between an agent that sounds alive and one that sounds like a menu. The other silent killer is the system prompt: every token is retokenized on every turn, so a 3,000-token prompt directly costs you real-time budget. Move examples into cached few-shot messages instead.
Barge-in and turn-taking
Barge-in is the ability for the caller to interrupt the agent mid-sentence. Without it, the conversation feels like an IVR. Vapi implements barge-in by continuously running VAD on the inbound audio even while the TTS is speaking; when caller speech is detected, TTS stops, the current LLM stream is cancelled, and a new turn starts. Tuning the VAD sensitivity is critical: too sensitive and background noise cuts the agent off; too loose and the caller has to shout.
Observability and debugging
Every call in Vapi produces a call log with a timestamped transcript, per-stage latency metrics, and tool-call payloads. Send these to your own analytics store to spot patterns: which prompts cause dead air, which tool calls time out, which voices callers hang up on. Without call-level observability, you're tuning blind.
When Vapi is the right choice
Vapi shines when you want provider flexibility and are willing to tune. If you want an out-of-the-box natural voice with fewer knobs, Retell is a strong alternative. If you're running enterprise outbound at very high concurrency, Bland's flat pricing wins. Vapi is the developer platform — most opinionated in the good way, least opinionated in the ways that hurt.
How Vapi differs from a normal LLM chat backend
A chat backend is request/response: the client sends a message, the server calls an LLM, the LLM returns text, the client renders it. Latency of a second or two is tolerable. Vapi cannot work that way. Every stage — STT, LLM, TTS — has to stream, and every stage has to be cancellable mid-flight because the caller might interrupt. The orchestration code that makes this feel like a conversation (not a slideshow) is a serious engineering effort. This is why 'just wire up OpenAI to Twilio yourself' projects almost never ship: the pipeline is 80% of the work, the prompt is 20%.
Deployment, environments, and secrets
Vapi treats assistants as first-class objects with versioning. You can maintain separate dev / staging / prod assistants with different phone numbers and prompt versions, then promote a config through the environments. Secrets (BYO provider keys, webhook signing secrets, transfer numbers) live in Vapi's encrypted secret store and are referenced by name from assistant configs — never inlined. This matters more than it seems: without secret management, teams copy-paste API keys into prompts and leak them in call logs.
What good sounds like
The best Vapi agents share a handful of qualities: first-word latency under 800ms, a system prompt under 800 tokens, endpointing between 200–300ms, streaming TTS with a warm voice, tool calls that complete under 300ms, and observability wired into a real analytics store. Miss any one of those and callers notice; nail all of them and the agent feels indistinguishable from a well-trained human on the first turn. That's the bar to aim for.
Call lifecycle walkthrough
Following a single call end-to-end makes the pipeline concrete. At T=0ms the PSTN network rings your Twilio number; Twilio opens a WebSocket to Vapi and streams µ-law audio at 8kHz. Vapi answers within ~40ms, spins up the assistant, pre-warms the TTS voice, and plays the greeting — pre-synthesized and cached so the caller hears audio before any LLM runs. When the greeting ends, Vapi opens a streaming STT session with Deepgram; partial transcripts start arriving ~150ms after the caller's first phoneme. As the caller speaks, VAD tracks energy, and the endpointer arms a 250ms silence timer once speech pauses. When the timer expires the finalized transcript is flushed to the LLM, which begins streaming tokens within 200–400ms. Vapi pipes those tokens straight into the TTS synth, which emits the first audio chunk another 150–250ms later. That chunk is re-encoded to µ-law and shipped back over the same Twilio WebSocket. Total: 600–900ms from silence to first spoken word. Every subsequent turn repeats the loop, except pre-warmed models skip cold-start cost. On hang-up, Vapi flushes the recording, writes the transcript, fires the end-of-call webhook, and releases the assistant slot back to the pool. Understanding this loop makes every optimization guide easier to reason about.
Barge-in and interruption handling in depth
Barge-in is deceptively hard because three systems have to agree in real time. VAD runs continuously on inbound audio even while TTS is playing, so it must distinguish caller speech from TTS leaking back through the caller's speakerphone (acoustic echo). Vapi uses echo cancellation on the inbound leg to avoid false triggers, but noisy environments still cause misfires. Once caller speech is confirmed, three cancellations fire in order: the TTS stream is stopped mid-syllable, the LLM stream is aborted (via provider cancellation tokens, not just discarding output), and the in-flight tool call — if any — is marked as cancelled so its response is ignored when it eventually returns. The new turn starts immediately with the partial transcript of the interruption. Tuning matters: VAD too sensitive means every 'uh-huh' from the caller kills the agent mid-sentence; too loose and callers have to shout to interrupt. The default Vapi settings work for typical office and mobile environments; adjust when your callers are in cars, warehouses, or on speakerphone. Watch the interruption rate in call logs — anything above ~15% of turns suggests VAD is misfiring and callers are getting frustrated.