An AI voice agent answers or places phone calls by converting the caller’s speech to text, passing that text to a language model that decides what to say or do, and turning the reply back into speech. Some newer systems do all three in a single speech-to-speech model. Either way, the agent can transfer the call to a person when needed.
How does an AI voice agent work?
Most voice agents today use a chained pipeline, sometimes called a cascade. Twilio’s Conversation Relay documentation lays out the flow in plain terms, and it maps to four steps:
- The call connects. A phone number routes the caller to the voice platform, the same way a call reaches any business phone system.
- Speech becomes text. A speech-to-text model transcribes what the caller says and sends the text to your application.
- A language model decides. Your application uses AI to read the text and generate a response. This is also where the agent can use tools, such as checking a calendar or looking up an order.
- Text becomes speech. A text-to-speech model reads the response aloud in the voice you chose.
One more component works in the background, and callers only notice it when it goes wrong: turn-taking. ElevenLabs’ platform overview lists four core parts, which are a speech-to-text model, your choice of language model, a text-to-speech model, and a turn-taking model that handles conversation timing.
If the idea of a model that uses tools is new to you, what is an AI agent covers the basics.
What about speech-to-speech models?
There is a second design. OpenAI’s voice agents guide describes an approach that uses one model to interpret audio, decide what to do and respond in speech.
The same guide explains when the chained design is the better choice: when each stage needs to be visible or replaceable. A chained pipeline lets you inspect or transform the text between stages and swap each component independently. OpenAI’s example stores the transcript and runs policy checks before the agent responds.
| Chained pipeline | Speech-to-speech model | |
|---|---|---|
| How it works | Separate speech-to-text, language model and text-to-speech stages | One model listens, decides and speaks |
| Visibility | Text is available between stages for review and checks | Listening, deciding and speaking happen inside one model |
| Flexibility | Each component can be replaced on its own | You adopt the model as a whole |
| Good fit | Calls where you want stored transcripts, policy checks or your own choice of components | Calls where one model handling the audio from end to end is enough |
Why does latency matter so much on a phone call?
Latency is the delay between the caller finishing a sentence and hearing a reply. A slow agent feels broken even when its answers are right.
Twilio’s guide to core latency in AI voice agents, published in November 2025, notes that people have a deeply ingrained aversion to pauses in conversation. It gives starting benchmarks for chained agents. It measures “mouth to ear” delay, meaning the time from when a caller stops speaking to when the agent’s reply reaches their ear.
| Stage | Target | Upper limit |
|---|---|---|
| Speech to text | 350 ms | 500 ms |
| Language model, time to first token | 375 ms | 750 ms |
| Text to speech, time to first audio | 100 ms | 250 ms |
| Full mouth-to-ear gap | 1,115 ms | 1,400 ms |
Twilio presents these as median starting points as of November 2025, so treat them as a reference and ask your provider for current figures.
Timing cuts both ways. Twilio’s turn detection documentation puts it simply: if the agent responds too early, it interrupts the caller, and if it waits too long, the conversation feels slow. Good turn detection listens for whether the caller has finished a thought or is only pausing.
When you test, OpenAI recommends measuring how long callers wait for a useful spoken answer, and comparing median and 95th percentile timings for similar calls. The 95th percentile tells you how your slowest calls feel.
How does a voice agent transfer a call to a human?
Transfer is a standard feature. ElevenLabs’ transfer documentation describes a tool that moves the ongoing call to a specified phone number or SIP address. By default it calls the destination and places the caller and the human operator into a conference.
Two choices shape the caller’s experience:
- Warm transfer. The agent summarizes the situation for the person receiving the call, so the caller does not have to repeat everything.
- Cold transfer. The call moves across without a summary.
You also write the conditions for when to transfer, in plain language. The language model uses those conditions to decide when and where to send the call. OpenAI’s telephony and SIP documentation describes a similar transfer step, where an active call is referred to a phone number or SIP address and the receiving phone system handles the rest.
What calls suit automation?
Voice agents do their best work on calls that are frequent, follow a clear pattern and have a defined finish. Good candidates include:
- Booking, changing and confirming appointments
- Answering questions about hours, location, services and policies
- Taking a message and capturing lead details after hours
- Routing callers to the right person or department
- Giving order or job status from a system the agent can check
Keep people on calls that are emotional, complex or high stakes, such as complaints, billing disputes and anything involving safety or health. For those, the agent’s job is to recognize the situation quickly and transfer. For a fuller framework, see AI customer service: what to automate.
What should you check before launch?
- Test with real phone calls. Call from a mobile phone, a landline and a car. Use background noise, accents and interruptions.
- Measure response delay. Record median and 95th percentile timings and compare them with the benchmarks above.
- Try to interrupt it. Talk over the agent and pause mid-sentence to see how turn-taking holds up.
- Write clear transfer rules. Decide which requests go to a person, who answers during and after business hours, and whether transfers are warm.
- Verify every action. If the agent books, updates or sends anything, confirm the result in the system it touched.
- Review transcripts. Read a sample of early calls every week and fix the gaps you find.
- Check the rules that apply. US rules cover AI-generated voices on calls. The FCC announced in February 2024 that calls made with AI-generated voices are “artificial” under the Telephone Consumer Protection Act, which holds them to the same standards as other artificial or prerecorded voice calls. Our guide to FCC and TCPA rules for AI voice calls covers the details. This is general information, so confirm your obligations with your own counsel.
The short version
An AI voice agent listens, thinks and speaks through a chain of speech-to-text, a language model and text-to-speech, or through one speech-to-speech model. Response delay and turn-taking decide whether it feels natural, and a clean transfer to a person covers the calls it should not handle. Test on real calls, measure the delay, and check the calling rules before you go live.



