Claude voice agent: a realtime STT, LLM and TTS stack

If you tried to build a voice agent this year, you hit the same wall as everyone else: Claude does not have a native realtime audio API. There is no claude-realtime model that takes microphone bytes in and sends speech bytes out. OpenAI has one, gpt-4o-realtime, and Google has Gemini Live. Anthropic, as of this writing, does not.

That sounds like a problem, and it isn't. The STT → LLM → TTS pipeline got so fast in 2026 that a well-built Claude voice stack responds in ~700 ms end-to-end, below the threshold at which people notice a pause. You also keep Claude's answer quality, which for most agentic and reasoning-heavy voice use cases (support, sales qualification, technical assistants) is materially better than the audio-native output of GPT-4o-realtime.

What follows is the architecture we ship in production, the latency budget that makes it work, and the code to wire it up.

Where voice stands in 2026

Three architectures dominate.

1. Audio-native realtime (one socket). OpenAI Realtime API and Gemini Live. You open a WebSocket, stream PCM in and get PCM out. The model handles VAD, interruption and turn-taking itself. Latency is excellent (~300-500 ms end-to-end). The catch is that you are locked into the audio-native model, which with OpenAI means GPT-4o. No Claude Sonnet, no o3, no tool calling from outside OpenAI.

2. Cascaded pipeline (STT → LLM → TTS). Three providers, three streams, glued together. More moving parts and, in theory, more latency. In exchange you pick the best option at each layer, and in practice modern streaming makes the cascade competitive.

3. Hybrid: realtime STT + LLM-as-text + realtime TTS. This is what the post is about. Deepgram Nova-3 or Whisper streaming for ears, Claude Sonnet 4.6 streaming for the brain, ElevenLabs Flash for the voice. Total round trip in production: 600-800 ms.

For Claude users the pipeline wins for a simple reason: option 1 is not available with Claude. The real question is whether options 2 and 3 close the latency gap enough that choosing Claude costs you nothing. They do.

Why a Claude pipeline beats OpenAI Realtime for most agents

Audio-native models squeeze the LLM into the same forward pass as the speech encoder and decoder. That is how they get fast, but it also leaves the "reasoning" part shallower than in text-mode equivalents. GPT-4o-realtime is noticeably weaker than GPT-4o-text at multi-step reasoning, tool use and instruction following, and weaker still than Claude Sonnet 4.6.

If a voice use case is mostly chitchat (companion bots, language tutoring), audio-native is fine. If it needs to:

• read and summarize a customer's account in real time;
• run a multi-tool agentic flow (look up an order, check inventory, schedule a callback);
• follow a complex script or compliance rules;
• ground its answers in long documents,

then Claude Sonnet 4.6 in a pipeline beats GPT-4o-realtime on quality. The 200-400 ms latency penalty against one-socket realtime is worth paying.

The recommended stack

Mic → VAD (Silero) → STT (Deepgram Nova-3 streaming) →
  Claude Sonnet 4.6 (streaming) →
  TTS (ElevenLabs Flash v2.5) → Speaker

Why these picks.

Deepgram Nova-3 for STT. Partial transcripts stream in at a ~150 ms cadence, and the final transcript lands about 200 ms after the end of the utterance. WER on conversational English is around 5%. Whisper Large v3 streaming via groq is a credible alternative (cheaper, slightly higher latency, slightly higher WER). For non-English I lean on Whisper.

Claude Sonnet 4.6 as the brain. It streams through the Anthropic Messages API, with time to first token typically 250-400 ms on short prompts. We route through the codexkey gateway, https://api.codexkey.online/v1, which adds ~30 ms of proxy overhead and gives us caching and observability.

ElevenLabs Flash v2.5 for TTS. About 200 ms to first audio when you feed it token by token. The newer Turbo v3 is faster, but its voices are less natural, and Flash is the sweet spot in 2026.

Silero VAD runs locally. A tiny ONNX model with 10 ms frames, it runs on CPU. It tells you when the user starts and stops talking, so silence never reaches the STT and the LLM never waits on a non-utterance.

Latency budget

Where the ~700 ms goes, from the end of the user's speech to the first audio out:

You can shave off more:

• Speculative TTS: start sending tokens to the TTS as soon as Claude finishes its first sentence, without waiting for the LLM to end. Saves ~300 ms on long responses.
• Predictive STT finalization: trigger the LLM call on a high-confidence partial instead of waiting for the final. Risky, but it cuts ~120 ms.
• Endpointing tuning: lower the VAD silence threshold from 500 ms to 300 ms. That removes 200 ms of dead air, at the price of more false triggers.

Aggressive tuning gets you to ~450 ms. Below that you are competing with OpenAI Realtime on its home turf.

Interruption and barge-in

The hardest part of a voice agent is not latency but interruption. The user starts talking while the bot is still talking. The bot has to:

1. Detect speech onset within ~100 ms (VAD on the input mic, not gated on output).
2. Stop TTS playback immediately: kill the audio buffer, don't just stop generating.
3. Cancel the in-flight Claude request (AbortController on the SDK call).
4. Discard the partial response, or fold it into context as "[interrupted]".

The state machine looks like this:

IDLE → LISTENING → THINKING → SPEAKING → (back to LISTENING)
                       ↑                      ↓
                       └── BARGE-IN ──────────┘

A common bug is the bot's own audio leaking into the mic and triggering a self-interrupt. Use echo cancellation (WebRTC AEC in a browser, RNNoise on the server) and a confidence threshold on the VAD.

Code: WebSocket bridge in TypeScript

A minimal end-to-end loop. The browser sends mic PCM to the server, and the server orchestrates STT, LLM and TTS and streams audio back.

import Anthropic from "@anthropic-ai/sdk";
import { createClient as deepgram } from "@deepgram/sdk";
import { ElevenLabsClient } from "elevenlabs";
import WebSocket from "ws";

const claude = new Anthropic({
  baseURL: "https://api.codexkey.online/v1",
  apiKey: process.env.CODEXKEY_API_KEY,
});
const dg = deepgram(process.env.DEEPGRAM_KEY);
const tts = new ElevenLabsClient({ apiKey: process.env.ELEVEN_KEY });

const wss = new WebSocket.Server({ port: 8080 });

wss.on("connection", (client) => {
  const history: Array<{ role: "user" | "assistant"; content: string }> = [];
  let abortLLM: AbortController | null = null;

  const stt = dg.listen.live({
    model: "nova-3",
    language: "en-US",
    smart_format: true,
    interim_results: true,
    endpointing: 300,
  });

  stt.on("Results", async (data: any) => {
    const transcript = data.channel.alternatives[0].transcript;
    if (!transcript) return;

    // Barge-in: user spoke while we were speaking
    if (abortLLM) {
      abortLLM.abort();
      client.send(JSON.stringify({ type: "stop_audio" }));
    }

    if (!data.is_final) return;

    history.push({ role: "user", content: transcript });
    abortLLM = new AbortController();

    const stream = await claude.messages.stream(
      {
        model: "claude-sonnet-5",
        max_tokens: 1024,
        system: "You are a helpful voice assistant. Keep replies under 2 sentences.",
        messages: history,
      },
      { signal: abortLLM.signal },
    );

    let buffer = "";
    let assistantText = "";
    for await (const chunk of stream) {
      if (chunk.type !== "content_block_delta") continue;
      const delta = (chunk.delta as any).text ?? "";
      buffer += delta;
      assistantText += delta;

      // Flush to TTS on sentence boundary
      const sentenceEnd = buffer.match(/[.!?]\s/);
      if (sentenceEnd) {
        const sentence = buffer.slice(0, sentenceEnd.index! + 1);
        buffer = buffer.slice(sentenceEnd.index! + 2);
        streamTTS(sentence, client);
      }
    }
    if (buffer.trim()) streamTTS(buffer, client);
    history.push({ role: "assistant", content: assistantText });
    abortLLM = null;
  });

  client.on("message", (raw) => {
    // Browser sends raw 16-bit PCM @ 16kHz
    stt.send(raw);
  });
});

async function streamTTS(text: string, client: WebSocket) {
  const audio = await tts.textToSpeech.convertAsStream(
    "21m00Tcm4TlvDq8ikWAM", // voice id
    {
      text,
      model_id: "eleven_flash_v2_5",
      output_format: "pcm_16000",
    },
  );
  for await (const chunk of audio) {
    client.send(chunk);
  }
}

That is about 80 lines, and it gives you a working voice loop with barge-in. Production code adds reconnect logic, audio resampling, prompt caching on Claude and metrics.

When to use OpenAI Realtime instead

Skip the Claude pipeline if:

• GPT-4o quality is enough for your use case (most consumer chitchat is fine).
• You want one socket and one vendor for operational simplicity.
• You need under 400 ms end-to-end and can't stream speculatively.
• You don't need Anthropic-specific features (Claude's tool use, long context, MCP).

Choose the Claude pipeline if:

• The agent does real reasoning, multi-step tool calls or compliance-bound flows.
• You need 300K+ context for grounding on documents.
• You already have a Claude text agent and want to add voice without rebuilding the brain.
• You want to swap STT and TTS providers independently (for example Whisper for non-English, or a regional TTS for compliance).

Bottom line

In 2026 you are choosing between a Claude pipeline and GPT-4o realtime. For agentic voice work where the answer actually has to be correct, the Claude pipeline wins on quality, and the 200 ms latency tax is invisible to users. For pure conversation where latency matters above all, OpenAI Realtime is still the play.

When Anthropic ships a native realtime audio mode (and it will), this whole post becomes obsolete. Until then, Deepgram + Claude Sonnet 4.6 + ElevenLabs Flash is the stack to beat.

Claude pricing is covered in our Claude API pricing breakdown for 2026.