AI Voice Assistant
Real-time, low-latency voice conversations with an AI agent, built on streaming STT/TTS pipelines over WebSocket.
Project Overview
An AI voice assistant that holds a natural spoken conversation rather than a turn-by-turn exchange. Audio streams in through Deepgram and Sarvam AI for speech-to-text, is reasoned over by OpenAI, and is spoken back through text-to-speech — all carried over WebSocket connections and reachable over the phone network via Plivo. The engineering focus was latency and interruption: the assistant stops talking when the caller speaks, and keeps track of context across the whole call.
Quick Facts
Voice that feels like a conversation, not a form
Request-response voice bots feel broken because they cannot be interrupted and they forget what was said a moment ago. Making it feel natural meant attacking latency at every hop and handling speech that overlaps.
Streaming pipelines with barge-in and session context
Audio is streamed rather than batched: Deepgram and Sarvam AI transcribe continuously, OpenAI generates the response, and speech synthesis streams back over the same WebSocket session. Incoming speech cancels in-flight playback so the caller can interrupt, and conversation state is held for the life of the session.
