0Pricing
AI Agents · Lesson

Text-to-Speech in Agent Responses

OpenAI TTS, ElevenLabs, and Google TTS APIs in agent pipelines.

Text-to-Speech in Agent Responses is a free AI Agents lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Agents learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Why Agents Need TTS

A voice agent that only listens but responds in text is not a true voice agent. Text-to-Speech (TTS) converts the agent's text responses into spoken audio, completing the full voice loop.

Two leading APIs: OpenAI TTS (fast, affordable, 6 voices) and ElevenLabs (ultra-realistic voices, streaming, voice cloning).

OpenAI TTS Basics

OpenAI's TTS API converts text to speech in seconds. Six built-in voices: alloy, echo, fable, onyx, nova, shimmer. Supports MP3, opus, AAC, and FLAC output formats.

import openai
import os

client = openai.OpenAI(api_key=os.getenv('OPENAI_API_KEY'))

def text_to_speech(text, voice='alloy', output_path='response.mp3'):
    response = client.audio.speech.create(
        model='tts-1',          # tts-1 (fast) or tts-1-hd (higher quality)
        voice=voice,            # alloy, echo, fable, onyx, nova, shimmer
        input=text,
        response_format='mp3'   # mp3, opus, aac, flac
    )
    with open(output_path, 'wb') as f:
        f.write(response.content)
    print(f'Audio saved to: {output_path}')
    return output_path

# Available voices:
# alloy    - neutral, balanced
# echo     - warm, conversational
# fable    - expressive
# onyx     - deep, authoritative
# nova     - friendly, upbeat
# shimmer  - soft, clear

Streaming TTS with OpenAI

For lower perceived latency, stream audio as it is generated instead of waiting for the full file. The caller can start playing audio while the rest is still being synthesized.

import openai
import os

client = openai.OpenAI(api_key=os.getenv('OPENAI_API_KEY'))

def stream_tts_to_file(text, output_path, voice='nova'):
    with client.audio.speech.with_streaming_response.create(
        model='tts-1',
        voice=voice,
        input=text
    ) as response:
        response.stream_to_file(output_path)
    return output_path

def stream_tts_to_bytes(text, voice='nova'):
    audio_chunks = []
    with client.audio.speech.with_streaming_response.create(
        model='tts-1',
        voice=voice,
        input=text
    ) as response:
        for chunk in response.iter_bytes(chunk_size=4096):
            audio_chunks.append(chunk)
    return b''.join(audio_chunks)

ElevenLabs SDK: High Quality TTS

ElevenLabs produces the most realistic voices available. It supports voice cloning, emotional tone control, and streaming.

Install with pip install elevenlabs.

from elevenlabs import ElevenLabs, VoiceSettings
import os

client = ElevenLabs(api_key=os.getenv('ELEVENLABS_API_KEY'))

def elevenlabs_tts(text, voice_id='pNInz6obpgDQGcFmaJgB', output_path='response.mp3'):
    # Common voice IDs:
    # Rachel:  21m00Tcm4TlvDq8ikWAM
    # Adam:    pNInz6obpgDQGcFmaJgB
    # Bella:   EXAVITQu4vr4xnSDxMaL

    audio = client.generate(
        text=text,
        voice=voice_id,
        model='eleven_multilingual_v2',  # supports 29 languages
        voice_settings=VoiceSettings(
            stability=0.5,        # 0.0-1.0: lower = more expressive
            similarity_boost=0.8, # 0.0-1.0: higher = closer to original voice
            style=0.3             # 0.0-1.0: style exaggeration
        )
    )

    with open(output_path, 'wb') as f:
        for chunk in audio:
            f.write(chunk)

    return output_path

ElevenLabs Streaming TTS

ElevenLabs supports streaming — receive audio chunks before the full text is synthesized. This is critical for voice agents where latency matters.

from elevenlabs import ElevenLabs
import os

client = ElevenLabs(api_key=os.getenv('ELEVENLABS_API_KEY'))

def streaming_tts(text, voice_id='pNInz6obpgDQGcFmaJgB'):
    audio_stream = client.generate(
        text=text,
        voice=voice_id,
        model='eleven_turbo_v2',  # fastest model, optimized for streaming
        stream=True
    )
    return audio_stream  # yields audio chunks as they are generated

def stream_to_file(text, output_path, voice_id='pNInz6obpgDQGcFmaJgB'):
    stream = streaming_tts(text, voice_id)
    with open(output_path, 'wb') as f:
        for chunk in stream:
            if chunk:
                f.write(chunk)
    print(f'Streaming TTS complete: {output_path}')

Playing Audio with sounddevice

After generating TTS audio, play it on the local device. sounddevice with soundfile is the simplest cross-platform option for an agent running on a local machine.

Install with pip install sounddevice soundfile.

import sounddevice as sd
import soundfile as sf
import numpy as np
import tempfile
import os

def play_audio_file(audio_path):
    data, sample_rate = sf.read(audio_path)
    sd.play(data, sample_rate)
    sd.wait()  # block until playback finishes

def play_mp3_bytes(audio_bytes):
    # Write to temp file then play (soundfile needs a file)
    with tempfile.NamedTemporaryFile(suffix='.mp3', delete=False) as f:
        f.write(audio_bytes)
        tmp_path = f.name
    try:
        play_audio_file(tmp_path)
    finally:
        os.unlink(tmp_path)

def say(text, voice='nova'):
    audio_bytes = stream_tts_to_bytes(text, voice=voice)
    play_mp3_bytes(audio_bytes)

# Usage
say('Hello! How can I help you today?')

Playing Audio with pygame

pygame provides more control: you can check if audio is still playing, stop it early, and play multiple sounds. Useful for interrupt handling in voice agents.

Install with pip install pygame.

import pygame
import io
import tempfile
import os

pygame.mixer.init(frequency=44100, size=-16, channels=1, buffer=512)

def play_audio_pygame(audio_bytes):
    with tempfile.NamedTemporaryFile(suffix='.mp3', delete=False) as f:
        f.write(audio_bytes)
        tmp_path = f.name

    try:
        pygame.mixer.music.load(tmp_path)
        pygame.mixer.music.play()
        while pygame.mixer.music.get_busy():
            pygame.time.wait(50)  # check every 50ms
    finally:
        pygame.mixer.music.stop()
        os.unlink(tmp_path)

def stop_audio():
    if pygame.mixer.music.get_busy():
        pygame.mixer.music.stop()
        print('Audio stopped (interrupt)')

Caching TTS Responses

Common phrases like greetings, error messages, and menu prompts are synthesized repeatedly. Cache their audio files so you pay for synthesis only once per phrase.

import hashlib
import os

TTS_CACHE_DIR = '/tmp/tts_cache'
os.makedirs(TTS_CACHE_DIR, exist_ok=True)

def cached_tts(text, voice='nova'):
    cache_key = hashlib.md5(f'{voice}:{text}'.encode()).hexdigest()
    cache_path = os.path.join(TTS_CACHE_DIR, f'{cache_key}.mp3')

    if os.path.exists(cache_path):
        print('TTS cache hit')
        return cache_path

    # Generate and save
    audio_bytes = stream_tts_to_bytes(text, voice=voice)
    with open(cache_path, 'wb') as f:
        f.write(audio_bytes)

    print(f'TTS cached: {cache_path}')
    return cache_path

# Pre-warm cache for common phrases on startup
COMMON_PHRASES = [
    'Hello! How can I help you today?',
    'I did not catch that. Could you repeat?',
    'Let me look that up for you.',
    'Thank you, goodbye!'
]

def prewarm_tts_cache():
    for phrase in COMMON_PHRASES:
        cached_tts(phrase)
    print(f'Pre-warmed {len(COMMON_PHRASES)} phrases')

Sentence-by-Sentence TTS for Streaming

For long LLM responses, split the text into sentences and synthesize + play them one by one. The user hears the first sentence within ~500ms while the rest is still being generated.

import re

def split_into_sentences(text):
    sentences = re.split(r'(?<=[.!?])\s+', text.strip())
    return [s.strip() for s in sentences if s.strip()]

def stream_response_as_voice(llm_response_text, voice='nova'):
    sentences = split_into_sentences(llm_response_text)
    print(f'Speaking {len(sentences)} sentences')

    for i, sentence in enumerate(sentences):
        if not sentence:
            continue
        print(f'Speaking ({i+1}/{len(sentences)}): {sentence[:60]}')
        cache_path = cached_tts(sentence, voice=voice)
        play_audio_file(cache_path)

# Example
response = 'The weather today is sunny and 22 degrees. Perfect for outdoor activities. Bring sunscreen if you plan to be outside for more than an hour.'
stream_response_as_voice(response)

Choosing Voice and Model

Match the voice choice to the agent's persona. Deeper voices (onyx, echo) for formal/professional agents; warmer voices (nova, shimmer) for friendly assistants.

VOICE_PROFILES = {
    'assistant':    {'service': 'openai',     'voice': 'nova',   'model': 'tts-1'},
    'professional': {'service': 'openai',     'voice': 'onyx',   'model': 'tts-1-hd'},
    'narrator':     {'service': 'openai',     'voice': 'fable',  'model': 'tts-1-hd'},
    'premium':      {'service': 'elevenlabs', 'voice_id': '21m00Tcm4TlvDq8ikWAM', 'model': 'eleven_multilingual_v2'}
}

def speak(text, persona='assistant'):
    profile = VOICE_PROFILES.get(persona, VOICE_PROFILES['assistant'])

    if profile['service'] == 'openai':
        audio_bytes = stream_tts_to_bytes(text, voice=profile['voice'])
        play_mp3_bytes(audio_bytes)
    elif profile['service'] == 'elevenlabs':
        output_path = elevenlabs_tts(text, voice_id=profile['voice_id'])
        play_audio_file(output_path)

speak('Your meeting has been rescheduled.', persona='professional')

Adjusting Speech Rate and Pitch

Different contexts call for different speech characteristics. Notification alerts should be slightly faster; storytelling slower and more expressive. Post-process TTS audio to adjust speed without changing pitch.

from pydub import AudioSegment
from pydub.effects import speedup
import os

def adjust_speech_speed(audio_path, speed_factor=1.0, output_path=None):
    if speed_factor == 1.0:
        return audio_path

    audio = AudioSegment.from_file(audio_path)

    if speed_factor > 1.0:
        # Speed up without changing pitch
        audio = speedup(audio, playback_speed=speed_factor)
    else:
        # Slow down: overlay with silence (simple method)
        # For production: use ffmpeg with atempo filter
        slow_factor = 1.0 / speed_factor
        audio = audio._spawn(
            audio.raw_data,
            overrides={'frame_rate': int(audio.frame_rate / slow_factor)}
        ).set_frame_rate(audio.frame_rate)

    if output_path is None:
        base = os.path.splitext(audio_path)[0]
        output_path = f'{base}_speed{speed_factor}.mp3'

    audio.export(output_path, format='mp3')
    return output_path

# Usage
path = text_to_speech('Your meeting starts in 5 minutes.')
fast_path = adjust_speech_speed(path, speed_factor=1.2)  # 20% faster for alerts

Knowledge Check

What is the primary benefit of using sentence-by-sentence TTS synthesis in a voice agent?

Recap: Text-to-Speech in Agent Responses

OpenAI TTS (tts-1, tts-1-hd) provides 6 voices with fast synthesis and streaming support — ideal for most voice agents. ElevenLabs offers higher quality realistic voices with eleven_turbo_v2 for low-latency streaming.

Play audio with sounddevice (simple) or pygame (interruptable). Cache common phrases to eliminate repeated API costs. Use sentence-by-sentence synthesis for long responses to minimize perceived latency. TTS caching with MD5 hashes ensures identical text always serves from cache.

Frequently asked questions

Is the “Text-to-Speech in Agent Responses” lesson free?

Yes — the full text of “Text-to-Speech in Agent Responses” is free to read here on the web, and the AI Agents course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Agents course, upgrade to CoddyKit PRO.

What will I learn in “Text-to-Speech in Agent Responses”?

OpenAI TTS, ElevenLabs, and Google TTS APIs in agent pipelines. You practise AI Agents with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start AI Agents?

No prior experience is required. AI Agents on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Text-to-Speech in Agent Responses” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this AI Agents lesson?

Yes. Every AI Agents lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. Speech-to-Text with Whisper and Deepgram
  2. Text-to-Speech in Agent Responses
  3. Building a Voice Conversation Loop
  4. Latency Optimization for Voice Agents
← Back to AI Agents