Building a voice agent that handles interruptions well sounds like a classification problem. Teach the model to tell a real interruption from a backchannel, from background noise, and you are done. That framing is wrong, and most implementations pay for it.
What people get wrong
The standard approach puts the classifier in front of the work. The customer speaks, the system waits for the classifier to rule on whether this is a real interruption, and only then does it decide what to do next. This creates an unavoidable latency floor. Worse, it creates a sensitivity trap: a more conservative classifier means fewer false positives but more lag. A more aggressive classifier is faster but interrupts the agent mid-sentence on backchannels. Teams spend their time tuning a tradeoff that should not exist.
The real issue is architectural. When the classifier is a gate, its latency is additive. When it is a cancellation signal, it is free.
How we chose to tackle it
With Responsive Voice, the moment a customer makes a sound while the agent is speaking, the system immediately starts a full new turn: STT, LLM, tool calls, guardrails, all of it. The barge-in classifier runs alongside in parallel. The classifier’s only job is keep or cancel.
If the customer was genuinely taking the floor, the classifier cuts the agent’s speech. The new turn is already running, so response time reflects LLM inference, not classifier lag. If the customer was just saying “uh-huh,” the classifier cancels the new turn and the agent keeps talking, uninterrupted.
Sensitivity is tunable per conversation node. During information gathering, interruption thresholds run lower. During required disclosures or confirmation steps, they run higher. The system knows what kind of exchange is happening and adjusts accordingly.
What this makes possible
The outcome is an agent that handles natural conversation rhythm: backchannels, noise, and genuine interruptions, without the latency penalty that normally accompanies sensitive interruption detection. Customers can speak naturally. The agent responds to what actually matters.
That behavior is an architecture choice.
