Cartesia Deep Dive: From Sonic Streaming TTS to a Real-Time Voice Agent Pipeline

🌏 中文版

Cartesia is most precisely a voice-model and inference platform for real-time interaction: Sonic provides text-to-speech (TTS), Ink provides speech-to-text (STT), and its APIs stream audio into an agent pipeline. As of August 2026, Cartesia also offers a voice-agent product called Line. Using the Sonic API, however, does not automatically provide phone numbers, call state, transfers, an LLM, and tool orchestration.

That boundary determines the comparison. A team already using LiveKit or Vapi can treat Cartesia as only a TTS provider; adopting Line brings Cartesia's agent control plane into scope. This article follows model → streaming inference → voice cloning/localization → agent pipeline because that is Cartesia's distinctive layer and the one most easily obscured by the phrase “voice-agent platform.”

Sonic is designed for streaming output

The Sonic family follows Cartesia's state-space-model research direction, with emphasis on low latency and continuously generating audio from incoming text instead of waiting for a complete answer. Fortune reported Cartesia's $64 million Series A and $91 million total funding in 2025. It also relayed the CEO's claims that Sonic latency fell from 90 ms to 45 ms and that more than 10,000 customers used it. Those latency and adoption figures came from the company, not a cross-vendor benchmark under identical conditions.

By 2026, Sonic-3.5 is the main model. Older models, voice-embedding inputs, and several endpoints were removed in June. The official changelog requires voice IDs for the current API. Production systems should pin dated snapshots instead of following latest, or prosody and pronunciation may change without an application-code diff.

WebSocket turns partial LLM text into playable audio

The official WebSocket API multiplexes generations using context_id. Text chunks for one turn share a context so Sonic can preserve prosody; a new turn should use a new context. Establish the socket before the first text arrives to keep handshake time out of first-audio latency.

import asyncio, json, os, websockets

async def speak(text, voice_id):
    url = (
        "wss://api.cartesia.ai/tts/websocket"
        "?cartesia_version=2026-03-01&api_key=" + os.environ["CARTESIA_API_KEY"]
    )
    async with websockets.connect(url) as ws:
        await ws.send(json.dumps({
            "model_id": "sonic-3.5",
            "transcript": text,
            "voice": {"mode": "id", "id": voice_id},
            "output_format": {
                "container": "raw",
                "encoding": "pcm_s16le",
                "sample_rate": 24000,
            },
            "context_id": "turn-42",
            "continue": False,
        }))

        async for message in ws:
            event = json.loads(message)
            if event["type"] == "chunk":
                play_base64_audio(event["data"])
            elif event["type"] == "done":
                break

asyncio.run(speak("Hello. Would you like help with your order?", os.environ["VOICE_ID"]))

A real agent must also handle partial LLM text, punctuation chunking, buffering, backpressure, barge-in, and cancellation. When a user begins speaking, stopping the player is insufficient; cancel the unplayed TTS context too, or stale speech can leak into the next turn.

Voice controls depend on the model version

Sonic supports emotion plus speed and volume generation guidance, including SSML-based control. The official control documentation, however, says Sonic-3.5 temporarily disables speed and volume; pin a Sonic-3 snapshot when those controls are required. These are naturalness-oriented hints rather than exact DSP knobs, so test with real scripts and languages.

Use pronunciation dictionaries and IPA for names, brands, and abbreviations instead of intentionally misspelling prompts. Match language and voice as well: an English voice accepting Chinese text does not guarantee natural Chinese localization. The 2026 changelog adds regional routing and pronunciation dictionaries, but “supported language” does not imply equal accents, number reading, and proper-noun quality.

Voice cloning and localization: the rights chain is harder than the API

The current instant-clone API accepts a short recording and returns a voice ID; the official Clone Voice API recommends roughly five seconds of source audio. Professional Voice Clone fine-tunes on substantially more recording for branded voices. Localization carries a voice across languages. Validate speaker similarity, intelligibility, accent, and human preference separately instead of accepting “sounds similar” as the criterion.

A voice is an identifying personal characteristic. Cartesia's Terms prohibit uploading another person's voice without express permission and say that, unless otherwise agreed, inputs, outputs, and interactions may be used to improve models; users may request a future training opt-out. Before launch, retain revocable written consent, restrict clone creation, log generation, provide deletion, and disclose synthetic speech where appropriate.

The official Zero Data Retention documentation also says ZDR does not apply to voice cloning, PVC, or workflows that must retain source material. Enabling ZDR for ordinary TTS does not prove that clone data is discarded. Confirm the DPA, region, retention, and deletion SLA separately.

Place it in the full agent pipeline

Telephony / WebRTC
      ↓
STT (Ink or another provider)
      ↓ partial transcript
Agent / LLM / tools
      ↓ partial text
Cartesia Sonic WebSocket
      ↓ PCM frames
Playback + barge-in + call state

Cartesia controls the final text-to-speech segment, while Ink or Line can expand its scope. A production system still needs turn detection, echo cancellation, a telephony provider, retries, recording policy, tool timeouts, and human transfer. Treating TTS first-byte latency as end-to-end latency omits STT endpointing, LLM TTFT, networking, and the jitter buffer.

Choosing among adjacent tools

Different teams should buy different layers. A team with LiveKit or Vapi can blind-test only the TTS provider; a team minimizing integration work should compare Line with other end-to-end agent platforms. Use identical scripts, codecs, regions, networks, and voice types to measure first audio, completion, interruption recovery, and human preference.

Overall

Cartesia's core value is delivering controllable, clonable speech at streaming speed. It fits products where latency and voice experience both matter and the team will design clear agent-pipeline boundaries; Line offers a path that integrates more components.

The common mistakes are treating model latency as call latency, assuming every model exposes every voice control, and reading ZDR as a promise about clone data. Start with a cancellable WebSocket TTS path, add real languages and consent handling, and only then decide whether Cartesia should own orchestration too.

References

• Cartesia Docs: TTS WebSocket API
• Cartesia Docs: 2026 Changelog
• Cartesia Docs: Volume, Speed, and Emotion
• Cartesia Docs: Clone Voice API
• Cartesia Docs: Zero Data Retention
• Cartesia AI, Inc.: Terms of Service
• Fortune: Cartesia raises $64M Series A
• LiveKit Agents documentation
• Vapi documentation
• Deepgram Voice Agent documentation
• ElevenLabs documentation