Audio + Text Agent Workflows
Transcription → reasoning → audio response pipelines end-to-end.
Audio + Text Agent Workflows is a free AI Agents lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Agents learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Audio-Text Agent Pipeline Overview
An audio-text agent pipeline converts between the spoken and written worlds. The canonical flow: audio in → Whisper transcription → text agent → TTS audio out. This enables voice assistants, call analysis, meeting summarisation, and hands-free interfaces.
Supported Audio Formats
OpenAI Whisper accepts: mp3, mp4, mpeg, mpga, m4a, wav, webm. Maximum file size is 25 MB. For larger files you must split or compress the audio before sending.
Always record/convert to 16 kHz mono for best transcription quality and smallest file size.
import os
SUPPORTED_FORMATS = {'.mp3', '.mp4', '.mpeg', '.mpga', '.m4a', '.wav', '.webm'}
MAX_FILE_SIZE_BYTES = 25 * 1024 * 1024 # 25 MB
def validate_audio_file(path: str) -> dict:
ext = os.path.splitext(path)[1].lower()
size = os.path.getsize(path) if os.path.exists(path) else 0
return {
'path': path,
'format_ok': ext in SUPPORTED_FORMATS,
'size_ok': size <= MAX_FILE_SIZE_BYTES,
'size_mb': round(size / 1024 / 1024, 2),
'extension': ext
}
# Usage:
info = validate_audio_file('meeting.wav')
print(info)Transcribing Audio with Whisper
OpenAI's audio.transcriptions.create endpoint wraps Whisper. Pass the file object, model name, and optionally a language hint (faster, avoids mis-detection) and a prompt (primes vocabulary for domain-specific terms).
from openai import OpenAI
client = OpenAI(api_key='YOUR_OPENAI_API_KEY')
def transcribe_audio(
audio_path: str,
language: str = 'en',
prompt: str = ''
) -> str:
with open(audio_path, 'rb') as audio_file:
transcript = client.audio.transcriptions.create(
model='whisper-1',
file=audio_file,
language=language,
prompt=prompt, # e.g. 'Python, NestJS, TypeScript, API'
response_format='text'
)
return transcript
text = transcribe_audio('user_query.mp3', language='en',
prompt='Kubernetes, microservices, Docker')
print('Transcription:', text)Transcription with Timestamps
For meeting analysis or speaker diarisation, request verbose_json format. This returns word-level or segment-level timestamps, allowing you to locate exact moments in the audio for referencing or clipping.
def transcribe_with_timestamps(audio_path: str) -> dict:
from openai import OpenAI
client = OpenAI(api_key='YOUR_OPENAI_API_KEY')
with open(audio_path, 'rb') as f:
result = client.audio.transcriptions.create(
model='whisper-1',
file=f,
response_format='verbose_json',
timestamp_granularities=['segment']
)
# result.segments: list of {start, end, text}
segments = [
{'start': s.start, 'end': s.end, 'text': s.text}
for s in result.segments
]
return {'full_text': result.text, 'segments': segments}
# Example output structure:
# {'full_text': 'Hello team...', 'segments': [{'start': 0.0, 'end': 2.3, 'text': 'Hello team'}]}Text Agent Processing
After transcription, the text passes through the LLM agent as a normal text message. Include a system prompt that sets the context: this text came from a spoken utterance, so expect filler words and incomplete sentences.
import anthropic
VOICE_AGENT_SYSTEM = (
'You are a helpful voice assistant. The user\'s message was transcribed from speech '
'and may contain filler words ("um", "uh"), false starts, or incomplete sentences. '
'Interpret the intent charitably and respond concisely in 1-3 sentences. '
'Your response will be converted to speech, so avoid markdown, lists, or code blocks.'
)
def voice_agent_respond(transcription: str) -> str:
client = anthropic.Anthropic(api_key='YOUR_ANTHROPIC_API_KEY')
response = client.messages.create(
model='claude-opus-4-5',
max_tokens=256,
system=VOICE_AGENT_SYSTEM,
messages=[{'role': 'user', 'content': transcription}]
)
return response.content[0].textText-to-Speech with OpenAI TTS
Convert the agent's text response back to speech using OpenAI's TTS API. Choose a voice (alloy, echo, fable, onyx, nova, shimmer) and a model (tts-1 for speed, tts-1-hd for quality). Save the result as an mp3 file or stream it directly.
from openai import OpenAI
from pathlib import Path
client = OpenAI(api_key='YOUR_OPENAI_API_KEY')
def text_to_speech(
text: str,
output_path: str = 'response.mp3',
voice: str = 'nova',
model: str = 'tts-1'
) -> str:
response = client.audio.speech.create(
model=model,
voice=voice,
input=text,
response_format='mp3'
)
Path(output_path).write_bytes(response.content)
print(f'TTS audio saved to {output_path}')
return output_path
# Full pipeline:
tts_file = text_to_speech(
'The weather in London is currently 15 degrees and partly cloudy.',
voice='nova'
)Streaming Audio Output
For real-time voice responses, stream the TTS audio in chunks instead of waiting for the full file. OpenAI's stream_to_file helper or manual chunk iteration lets you start playing audio while the rest is still being generated.
from openai import OpenAI
client = OpenAI(api_key='YOUR_OPENAI_API_KEY')
def stream_tts_to_file(text: str, output_path: str):
# OpenAI SDK streams the audio response
with client.audio.speech.with_streaming_response.create(
model='tts-1',
voice='nova',
input=text,
response_format='mp3'
) as response:
response.stream_to_file(output_path)
print(f'Streamed to {output_path}')
def stream_tts_chunks(text: str):
"""Yield raw audio bytes for piping to an audio player."""
with client.audio.speech.with_streaming_response.create(
model='tts-1',
voice='nova',
input=text
) as response:
for chunk in response.iter_bytes(chunk_size=4096):
yield chunk # pipe to audio player or WebSocketSplitting Long Audio Files
Audio files larger than 25 MB must be split before sending to Whisper. Use the pydub library to split by time intervals and process each chunk independently, then concatenate the transcripts.
from pydub import AudioSegment
import os
def split_audio(
path: str,
chunk_minutes: int = 10
) -> list:
audio = AudioSegment.from_file(path)
chunk_ms = chunk_minutes * 60 * 1000
chunks = []
os.makedirs('audio_chunks', exist_ok=True)
for i, start in enumerate(range(0, len(audio), chunk_ms)):
chunk = audio[start:start + chunk_ms]
chunk_path = f'audio_chunks/chunk_{i:03d}.mp3'
chunk.export(chunk_path, format='mp3')
chunks.append(chunk_path)
return chunks
def transcribe_long_audio(path: str) -> str:
chunks = split_audio(path, chunk_minutes=10)
transcripts = []
for chunk_path in chunks:
text = transcribe_audio(chunk_path)
transcripts.append(text)
# Clean up chunks
for chunk_path in chunks:
os.remove(chunk_path)
return ' '.join(transcripts)Real-Time Audio Pipeline
A real-time pipeline captures microphone input in chunks, sends each chunk to Whisper as it arrives, and streams TTS output back — creating a near-real-time voice conversation loop. The key challenge is managing latency at each stage.
import asyncio
import time
async def realtime_voice_loop(mic_stream, speaker_stream, client):
"""
mic_stream: async generator yielding audio bytes
speaker_stream: async callable accepting audio bytes
"""
buffer = b''
BUFFER_THRESHOLD = 50 * 1024 # ~3 seconds at 16kHz mono mp3
async for audio_chunk in mic_stream:
buffer += audio_chunk
if len(buffer) >= BUFFER_THRESHOLD:
t0 = time.time()
# Save buffer to temp file
with open('/tmp/voice_chunk.mp3', 'wb') as f:
f.write(buffer)
buffer = b''
# Transcribe
text = transcribe_audio('/tmp/voice_chunk.mp3')
print(f'Transcribed ({time.time()-t0:.1f}s): {text}')
# Agent response
reply = voice_agent_respond(text)
# TTS and stream to speaker
for audio_bytes in stream_tts_chunks(reply):
await speaker_stream(audio_bytes)Language Detection and Auto-Routing
Whisper automatically detects the spoken language. Use the detected language to route the conversation to the correct agent persona or to set the TTS language for the response.
def transcribe_and_detect_language(audio_path: str) -> dict:
from openai import OpenAI
client = OpenAI(api_key='YOUR_OPENAI_API_KEY')
with open(audio_path, 'rb') as f:
result = client.audio.transcriptions.create(
model='whisper-1',
file=f,
response_format='verbose_json'
)
return {
'text': result.text,
'language': result.language, # e.g. 'english', 'spanish'
'duration': result.duration
}
LANGUAGE_VOICE_MAP = {
'english': 'nova',
'spanish': 'alloy',
'french': 'echo',
'german': 'fable'
}
def respond_in_detected_language(audio_path: str) -> str:
info = transcribe_and_detect_language(audio_path)
voice = LANGUAGE_VOICE_MAP.get(info['language'], 'nova')
reply = voice_agent_respond(info['text'])
return text_to_speech(reply, voice=voice)Handling Background Noise and Poor Audio
Low-quality audio degrades transcription accuracy. Practical mitigations: pre-process with noise reduction (noisereduce library), add domain vocabulary in the Whisper prompt, validate transcript confidence, and ask for clarification when confidence is low.
def transcribe_with_quality_check(
audio_path: str,
min_confidence_words: int = 3
) -> dict:
from openai import OpenAI
import string
client = OpenAI(api_key='YOUR_OPENAI_API_KEY')
with open(audio_path, 'rb') as f:
result = client.audio.transcriptions.create(
model='whisper-1',
file=f,
response_format='verbose_json',
timestamp_granularities=['word']
)
# Word count as a basic quality proxy
words = result.text.translate(
str.maketrans('', '', string.punctuation)
).split()
quality_ok = len(words) >= min_confidence_words
return {
'text': result.text,
'word_count': len(words),
'quality_ok': quality_ok,
'fallback_message': None if quality_ok else 'I could not hear you clearly. Please repeat.'
}Knowledge Check
What does the Whisper prompt parameter do?
Recap: Audio-Text Agent Workflows
Excellent! Key takeaways:
- Whisper: supports mp3/wav/m4a etc., max 25 MB; split large files with pydub
- Timestamps: use
verbose_jsonwithtimestamp_granularitiesfor segment timing - TTS: OpenAI TTS with voice options; stream chunks for low latency
- Language detection: Whisper auto-detects; route to correct voice/agent based on detected language
- Real-time pipeline: buffer audio chunks → transcribe → agent → stream TTS
Next: video understanding in agents — frame extraction and temporal reasoning.
Frequently asked questions
Is the “Audio + Text Agent Workflows” lesson free?
Yes — the full text of “Audio + Text Agent Workflows” is free to read here on the web, and the AI Agents course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Agents course, upgrade to CoddyKit PRO.
What will I learn in “Audio + Text Agent Workflows”?
Transcription → reasoning → audio response pipelines end-to-end. You practise AI Agents with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start AI Agents?
No prior experience is required. AI Agents on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Audio + Text Agent Workflows” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this AI Agents lesson?
Yes. Every AI Agents lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Image + Text Agents with Claude Vision and GPT-4V
- Audio + Text Agent Workflows
- Video Understanding in Agents
- Cross-Modal Reasoning Patterns