Skip to content
Createfirm

AI voice agents for business phone calls: how they work

AI voice agents turn a caller's speech into text, reason with a language model and speak a reply. Learn how they work and what to check before you launch.

By the Createfirm teamPublished 7 min read

Illustration of an AI voice agent: a steel telephone handset with sound waves rising above sunrise clouds

An AI voice agent answers or places phone calls by converting the caller’s speech to text, passing that text to a language model that decides what to say or do, and turning the reply back into speech. Some newer systems do all three in a single speech-to-speech model. Either way, the agent can transfer the call to a person when needed.

How does an AI voice agent work?

Most voice agents today use a chained pipeline, sometimes called a cascade. Twilio’s Conversation Relay documentation lays out the flow in plain terms, and it maps to four steps:

  1. The call connects. A phone number routes the caller to the voice platform, the same way a call reaches any business phone system.
  2. Speech becomes text. A speech-to-text model transcribes what the caller says and sends the text to your application.
  3. A language model decides. Your application uses AI to read the text and generate a response. This is also where the agent can use tools, such as checking a calendar or looking up an order.
  4. Text becomes speech. A text-to-speech model reads the response aloud in the voice you chose.

One more component works in the background, and callers only notice it when it goes wrong: turn-taking. ElevenLabs’ platform overview lists four core parts, which are a speech-to-text model, your choice of language model, a text-to-speech model, and a turn-taking model that handles conversation timing.

If the idea of a model that uses tools is new to you, what is an AI agent covers the basics.

What about speech-to-speech models?

There is a second design. OpenAI’s voice agents guide describes an approach that uses one model to interpret audio, decide what to do and respond in speech.

The same guide explains when the chained design is the better choice: when each stage needs to be visible or replaceable. A chained pipeline lets you inspect or transform the text between stages and swap each component independently. OpenAI’s example stores the transcript and runs policy checks before the agent responds.

Chained pipeline Speech-to-speech model
How it works Separate speech-to-text, language model and text-to-speech stages One model listens, decides and speaks
Visibility Text is available between stages for review and checks Listening, deciding and speaking happen inside one model
Flexibility Each component can be replaced on its own You adopt the model as a whole
Good fit Calls where you want stored transcripts, policy checks or your own choice of components Calls where one model handling the audio from end to end is enough

Why does latency matter so much on a phone call?

Latency is the delay between the caller finishing a sentence and hearing a reply. A slow agent feels broken even when its answers are right.

Twilio’s guide to core latency in AI voice agents, published in November 2025, notes that people have a deeply ingrained aversion to pauses in conversation. It gives starting benchmarks for chained agents. It measures “mouth to ear” delay, meaning the time from when a caller stops speaking to when the agent’s reply reaches their ear.

Stage Target Upper limit
Speech to text 350 ms 500 ms
Language model, time to first token 375 ms 750 ms
Text to speech, time to first audio 100 ms 250 ms
Full mouth-to-ear gap 1,115 ms 1,400 ms

Twilio presents these as median starting points as of November 2025, so treat them as a reference and ask your provider for current figures.

Timing cuts both ways. Twilio’s turn detection documentation puts it simply: if the agent responds too early, it interrupts the caller, and if it waits too long, the conversation feels slow. Good turn detection listens for whether the caller has finished a thought or is only pausing.

When you test, OpenAI recommends measuring how long callers wait for a useful spoken answer, and comparing median and 95th percentile timings for similar calls. The 95th percentile tells you how your slowest calls feel.

How does a voice agent transfer a call to a human?

Transfer is a standard feature. ElevenLabs’ transfer documentation describes a tool that moves the ongoing call to a specified phone number or SIP address. By default it calls the destination and places the caller and the human operator into a conference.

Two choices shape the caller’s experience:

  • Warm transfer. The agent summarizes the situation for the person receiving the call, so the caller does not have to repeat everything.
  • Cold transfer. The call moves across without a summary.

You also write the conditions for when to transfer, in plain language. The language model uses those conditions to decide when and where to send the call. OpenAI’s telephony and SIP documentation describes a similar transfer step, where an active call is referred to a phone number or SIP address and the receiving phone system handles the rest.

What calls suit automation?

Voice agents do their best work on calls that are frequent, follow a clear pattern and have a defined finish. Good candidates include:

  • Booking, changing and confirming appointments
  • Answering questions about hours, location, services and policies
  • Taking a message and capturing lead details after hours
  • Routing callers to the right person or department
  • Giving order or job status from a system the agent can check

Keep people on calls that are emotional, complex or high stakes, such as complaints, billing disputes and anything involving safety or health. For those, the agent’s job is to recognize the situation quickly and transfer. For a fuller framework, see AI customer service: what to automate.

What should you check before launch?

  1. Test with real phone calls. Call from a mobile phone, a landline and a car. Use background noise, accents and interruptions.
  2. Measure response delay. Record median and 95th percentile timings and compare them with the benchmarks above.
  3. Try to interrupt it. Talk over the agent and pause mid-sentence to see how turn-taking holds up.
  4. Write clear transfer rules. Decide which requests go to a person, who answers during and after business hours, and whether transfers are warm.
  5. Verify every action. If the agent books, updates or sends anything, confirm the result in the system it touched.
  6. Review transcripts. Read a sample of early calls every week and fix the gaps you find.
  7. Check the rules that apply. US rules cover AI-generated voices on calls. The FCC announced in February 2024 that calls made with AI-generated voices are “artificial” under the Telephone Consumer Protection Act, which holds them to the same standards as other artificial or prerecorded voice calls. Our guide to FCC and TCPA rules for AI voice calls covers the details. This is general information, so confirm your obligations with your own counsel.

The short version

An AI voice agent listens, thinks and speaks through a chain of speech-to-text, a language model and text-to-speech, or through one speech-to-speech model. Response delay and turn-taking decide whether it feels natural, and a clean transfer to a person covers the calls it should not handle. Test on real calls, measure the delay, and check the calling rules before you go live.

Frequently asked questions

How does an AI voice agent work?
The caller's speech is converted to text, a language model reads that text and decides what to say or do, and a text-to-speech model speaks the reply. A turn-taking component decides when the caller has finished talking. Some newer systems use a single speech-to-speech model that listens and responds directly in audio.
How fast should an AI voice agent respond?
Twilio's November 2025 latency guide for chained voice agents gives a target of 1,115 milliseconds, with an upper limit of 1,400, between the caller finishing a sentence and hearing the reply. OpenAI recommends measuring how long callers wait for a useful spoken answer and comparing median and 95th percentile timings.
Can an AI voice agent transfer a call to a real person?
Yes. Voice platforms document call transfer to a phone number or SIP address. ElevenLabs, for example, supports a warm transfer that summarizes the situation for the person receiving the call, and a cold transfer without a summary. You write the conditions that tell the agent when to hand off.
Is it legal to use an AI voice on business phone calls in the US?
US rules apply. In February 2024 the FCC announced a ruling that calls made with AI-generated voices are artificial under the Telephone Consumer Protection Act, so they are held to the same standards as other artificial or prerecorded voice calls. This is general information, so confirm your own obligations with counsel.

Sources

Every fact in this article was checked against these pages on November 14, 2026.

  1. Voice agents, OpenAI
  2. Telephony and SIP, OpenAI
  3. ElevenAgents, ElevenLabs
  4. Transfer to number, ElevenLabs
  5. Conversation Relay, Twilio
  6. Turn detection for Conversation Relay, Twilio
  7. A Guide to Core Latency in AI Voice Agents (Cascaded Edition), Twilio
  8. FCC Makes AI-Generated Voices in Robocalls Illegal, Federal Communications Commission

Keep reading