Full-Duplex Voice Agent Interruptions
Full-duplex voice agent design requires interruption handling, VAD, turn detection, response cancellation, state, and fallback controls.
I started paying attention to interruption bugs after a demo where the full-duplex voice agent kept talking for two seconds after the user said “stop.” Two seconds is not long in a benchmark. In a live conversation, it feels broken.
I’m Dora. This piece is not a full-duplex concept explainer. That page already exists. This is the narrower work note: what breaks when users interrupt, what the audio layer has to own, and why response cancellation without state rollback is only half a fix.
Why Interruptions Break Voice Agents
A full-duplex system can listen while it speaks. That does not mean it knows what to do when both happen at once.
The hard part is not detecting sound. The hard part is deciding whether the sound means “continue listening,” “stop speaking,” “cancel the current answer,” “rollback a pending action,” or “ignore background noise.” That decision sits inside the realtime voice architecture, not inside the model alone.
Barge-in, overlap, pauses, false starts, and user corrections
A barge-in voice agent fails in ordinary ways.
The user interrupts with a correction: “No, I meant tomorrow.” The user starts, stops, and restarts: “Can you book… wait.” The user speaks over playback because the answer is wrong.
Someone else in the room talks.
The assistant hears its own speaker output as new input.
I paused here, because teams often treat all of these as the same event: speech detected. They are not the same event.
Barge-in is intentional interruption. Overlap may be accidental. A pause may mean the user is thinking. A false start may mean the user has not committed to the request yet. A correction may invalidate the last 500 milliseconds of model output, or the last tool call, or both.
A streaming voice AI system needs a small interruption grammar. Not a perfect one. Just enough to avoid treating every sound as a new user turn.
Why model quality alone does not solve conversation control
A better model can produce better answers. It does not automatically manage microphones, speakers, buffers, tool state, and UI.
Current realtime systems expose this split. For example, OpenAI’s Realtime server-events reference documents events related to server-side voice activity detection, response lifecycle state, and audio output handling. Verify the exact event names and interruption behavior against the current server-events reference. Those events are useful. They still need a client and application layer that knows what to cancel, what to keep, and what to show.
So that is where the bottleneck was. Conversation control is an application problem.
Build the Audio Control Layer
The audio control layer is the part I would build before arguing about prompt wording.
It owns microphone capture, speaker playback, echo handling, VAD thresholds, local buffering, server buffering, network jitter, and cancellation timing. If this layer is weak, the model looks worse than it is.
VAD, turn detection, echo handling, streaming buffers, and network jitter
Voice activity detection is a gate, not a truth machine. It marks probable speech boundaries. It does not know whether the user meant to interrupt, muttered to someone nearby, or breathed into a bad headset.
Browsers already expose relevant audio constraints. MDN’s MediaTrackConstraints reference covers fields such as echoCancellation, noiseSuppression, sampleRate, and latency. Those settings are not product polish. They affect whether the agent hears the user, the speaker, or both.
Echo handling matters more in full-duplex than in push-to-talk. If assistant audio leaks into the microphone, the agent may interrupt itself. That sounds funny once. Then it becomes a QA category.
Network jitter is the quieter failure. The W3C WebRTC Stats spec defines metrics such as jitter, packets lost, and jitter buffer delay. I would track them next to interruption outcomes. A “bad model turn” may be a delayed audio frame arriving after the user already moved on.
Response cancellation, partial output cleanup, and state rollback
Cancellation needs three separate actions.
First, stop generation. Second, stop playback. Third, repair state.
OpenAI’s Realtime client events include response.cancel and, for WebRTC or SIP connections, output_audio_buffer.clear. When interrupted audio has already been sent to the client, conversation.item.truncate can be used to align the server-side conversation state with the portion the user actually heard. Cancelling server generation does not always mean the user stops hearing audio. Clearing playback does not automatically remove partial text from the UI. Neither one rolls back a tool action.
Here is the only table I keep in my notes for this part:
| Interruption target | What to clear | What to keep |
|---|---|---|
| Audio playback | queued assistant audio | final heard transcript |
| Model response | unfinished response item | cancellation reason |
| UI text | partial assistant text | visible user correction |
| Tool action | pending, unconfirmed action | tool request log |
| Conversation state | invalidated draft answer | last stable user intent |
The phrase “last stable user intent” matters. If the user says “actually tomorrow,” the old answer is not just interrupted. It is stale.
State rollback should be boring. Pending action cancelled. Partial output marked invalid. New user correction attached to the same turn. Next response generated from the corrected state.
Better than expected, but only slightly.
Coordinate Model, Tools, and UI State
Interruption handling gets worse when tools enter the session.
A normal answer can be cancelled. A tool call may already have touched another system. That changes the rules.
Tool calls during speech, pending actions, confirmations, and recovery
The model may decide to call a tool while the user is still reacting to the spoken answer. That is where teams get into trouble.
For read-only tools, I would allow execution after a stable turn. Search, lookup, classification, summarization. Low risk.
For state-changing tools, I would split the action into draft and commit. Draft booking. Draft refund. Draft message. Draft appointment. The commit requires confirmation after playback has stopped and the user’s response has been captured cleanly.
If the user interrupts during a pending confirmation, the safest default is to freeze the action. Not cancel silently. Not proceed. Freeze it, log the interruption, and ask again only if the user returns to the action.
A useful recovery record includes the user transcript, assistant transcript, response ID, cancellation reason, pending tool name, proposed arguments, confirmation text, and whether the user heard the confirmation. Without the “heard” state, dispute handling gets vague fast.
When to pause speaking versus continue listening
The agent does not need to stop speaking for every noise.
I use three buckets.
If the user begins a correction, stop speaking. If the user gives a short backchannel like “okay” or “mm-hm,” keep speaking unless product policy says otherwise. If confidence is low, lower playback volume and keep listening for a short window before committing to cancellation.
This is where WebRTC details matter less as theory and more as plumbing. Playout delay, jitter buffer behavior, and audio routing all affect whether the user feels heard. The model may have the right answer. The experience still fails if the audio layer reacts late.
Speed is not the goal. Not breaking flow is.
Test Interruption Behavior Before Launch
I do not trust voice demos that only test clean speech in a quiet room.
That test is useful. It is also the easiest one.
Noise, double talk, accents, silence, and long pauses
A real test set needs background noise, speaker leakage, double talk, soft voices, strong accents, long pauses, half-started sentences, and corrections after playback starts.
The test should include users who interrupt because the answer is wrong, users who interrupt because they are impatient, and users who interrupt because they forgot part of the request. Those are different cases.
Long pauses need special attention. Too short, and the assistant talks over thinking. Too long, and voice agent latency feels like the system froze. This is not a model benchmark. It is turn design.
Metrics for interruption success and user frustration
I would track five metrics before launch.
- Interruption detection latency: time from user speech start to playback stop.
- False interruption rate: how often noise or backchannels stop the agent.
- Rollback correctness: whether partial output and pending actions end in the right state.
- Correction success rate: whether the next answer reflects the user’s changed intent.
- Repeat interruption rate: how often the user has to interrupt again.
That last one is the one I watch. If users interrupt twice, the first interruption probably did not work.
FAQ
Who approves interruption behavior for regulated workflows?
Product, engineering, security, compliance, and QA. For regulated workflows, legal or policy owners need to review when the assistant may continue, when it must stop, and when confirmation is required. Voice interruption handling becomes part of the approval surface because it affects consent.
What user complaint evidence should voice QA retain?
Keep the audio recording if consent and policy allow it. Also keep transcripts, timestamps, response IDs, cancellation events, playback state, tool state, confirmation text, and network metrics. A complaint about “the agent ignored me” cannot be reviewed from transcript alone.
When should a voice session be escalated to a human instead of retried?
Escalate when the user disputes a confirmation, repeats the same correction, shows frustration, hits a regulated action, or when the system cannot decide whether an interruption cancelled a pending action. Retrying the same broken turn is not recovery. It is just making the user do QA.
Conclusion
A full-duplex voice agent does not fail because it can speak and listen at the same time. It fails because nobody owns the moment where speech overlaps, output continues, and state becomes uncertain.
The fix is not one better model. It is an audio control layer, strict cancellation behavior, recoverable state, and QA that treats interruptions as normal conversation.
Good infrastructure makes you forget it is there. For voice agents, that starts when “stop” actually stops.
Previous posts:
