The five building blocks of a call
An AI voice agent isn't a single program, it's a chain. When the caller speaks, their voice passes through five stages before coming back as an answer, and the loop repeats on every turn of the conversation.
- Telephony. The call arrives on a number managed by an operator, which passes it to the agent as an audio stream. This is the oldest and most regulated building block: a phone number, a registered operator, number porting, call transfer to your mobile, SMS.
- Speech recognition, or transcription. The audio stream is converted into text, continuously, word by word, with end-of-sentence detection to know when the caller has finished. Noise from a building site, an accent, an unusual first name or a phone number spelled out are its difficulties.
- The language model. This is the brain. It receives the caller's text, the context of the conversation and your business profile, and decides what comes next: a question, an answer, an action. It doesn't “know” anything about your business by itself; it knows what the profile tells it.
- Actions. When the model decides to book an appointment, it doesn't do it itself: it calls a tool, which queries your calendar, checks the slot, writes it in and sends the SMS. These tools are the part that matters most to you: that's where the agent acts in the real world, and that's where a mistake costs.
- Speech synthesis. The text of the answer is converted into speech, continuously, so that the agent starts speaking before it has finished formulating. The quality of the voice matters, but less than its responsiveness.
All this repeats on every exchange, and the whole thing must take less than a second between the end of the caller's sentence and the start of the answer. That's the hardest part.
Why latency is the real issue
In a conversation between humans, the silence between two turns is very short, and a silence that goes on too long gets interpreted: “they didn't hear”, “they're hesitating”, “we've been cut off”. A voice agent has to keep up this pace with a chain of five stages, each of which takes time. Three techniques make it possible.
- Streaming. Transcription starts while the caller is speaking, and synthesis starts from the first words of the answer. Nothing waits for the previous stage to finish.
- Fast models for simple answers. Confirming a time doesn't call for the same reasoning as understanding a three-sentence quote request. A good agent uses different models depending on the task.
- Building blocks close together. A chain whose stages are hosted in the same data centre as the telephony saves round trips. It's also an argument in favour of hosting in France.
When you test an agent, time its silences. If they regularly go over a second, callers will notice, and they'll hang up.
Interruptions, noise, numbers
Three situations separate an agent that works in a demo from an agent that works on a building site.
- Interruption. The caller cuts the agent off, to correct something or to speed things up. The agent must stop talking, listen, and pick up where the conversation is, not where it was. This is called handling overlap, and it's one of the hardest things to get right.
- Noise. An engine, a drill, a restaurant in full service. Transcription has to filter it, and the agent must know how to say “I didn't quite catch that, could you repeat it?” rather than guess.
- Numbers and names. A phone number, an address, an unusual surname, a registration number. The agent must have them repeated, spelled out and reformulated before writing them down. Sayelo always confirms the callback number digit by digit.
Safeguards: what stops the agent saying just anything
A language model can produce a plausible sentence about anything. That's its strength, and its danger: without constraints, it would make up a price, an opening time, medical advice. Safeguards are the rules that stop it.
- The business profile as the only source. The agent answers from what you have written; for anything that isn't there, it says it doesn't know and takes a message.
- Authorised actions. It only does what its tools allow: write in the calendar, send an SMS, transfer. It can't cancel all your appointments or send an email on your behalf if it doesn't have that tool.
- Forbidden topics. Medical, legal or accounting advice; confirming information about a customer to a third party; promising a price or timescale that hasn't been set.
- Escalation rules. When to transfer, when to alert, when to stop. An agent that doesn't know how to hand over is an agent that ends up answering badly.
- Announcing the AI. At the start of the call, every time. It's a legal obligation, and it's a safeguard for the caller themselves. The guide to the AI Act and GDPR details this point.
Where your data goes
During a call, the caller's voice passes through the telephony, transcription, language model and tools. Each building block can be hosted by a different provider, in a different country. That's why the question “where is the data hosted?” calls for an answer in several parts, and why you need to ask it for each building block: telephony, transcription, the model, storage of summaries and any recordings. At Sayelo, data is hosted in France by an HDS-certified host, and the data processing agreement describes the providers we use.
What you should ask a provider
Demos all look alike. The differences lie in the answers to these questions.
- What is the average latency between the end of my sentence and the start of the answer?
- How does the agent handle an interruption?
- What does it do when it doesn't understand, or when the question isn't in the profile?
- What actions can it actually carry out, and in which tools, without an intermediary?
- How many simultaneous calls does it take?
- Where is each building block hosted, and are my conversations used to train a model?
- Does it announce itself as an AI? Can that be switched off? The right answer is no.
- How can I review what it said: transcript, summary, recording?
And in practice, at Sayelo
At Sayelo, the voice agent, the number, transfers and SMS all come in one subscription. Numbers and telephony are provided by a partner operator registered with ARCEP (the French telecoms regulator). The agent announces itself as a voice assistant, answers from your business profile, writes in Google Calendar, Microsoft 365 or Cal.com, confirms by SMS and sends you the summary of every call. You can hear it in three minutes, with a demo that calls you back, before reading anything else. The complete guide to AI receptionists then explains how to set it up, and the AI voice agent FAQ answers the questions we are asked most often.
