0Pricing
AI Agents · Lesson

Speech-to-Text with Whisper and Deepgram

Real-time and batch transcription, language detection, and punctuation.

Speech-to-Text with Whisper and Deepgram is a free AI Agents lesson on CoddyKit — lesson 1 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the AI Agents learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Speech-to-Text in Voice Agents

A voice agent must convert spoken audio to text before the LLM can process it. This step is called Speech-to-Text (STT) or Automatic Speech Recognition (ASR).

Two leading options: OpenAI Whisper (file-based, batch) and Deepgram (streaming, real-time, with speaker diarization and word timestamps).

OpenAI Whisper: Basic Transcription

Whisper via the OpenAI API transcribes audio files. Supported formats: mp3, mp4, wav, webm, m4a, flac. Maximum file size: 25MB.

import openai
import os

client = openai.OpenAI(api_key=os.getenv('OPENAI_API_KEY'))

def transcribe_file(audio_path, language=None):
    with open(audio_path, 'rb') as audio_file:
        params = {
            'model': 'whisper-1',
            'file': audio_file,
            'response_format': 'json'  # or 'text', 'srt', 'vtt', 'verbose_json'
        }
        if language:
            params['language'] = language  # e.g., 'en', 'fr', 'de'

        transcript = client.audio.transcriptions.create(**params)

    return transcript.text

# Basic usage
text = transcribe_file('meeting_recording.mp3')
print('Transcribed:', text[:200])

Whisper with Word Timestamps

Use response_format='verbose_json' to get timestamps for each word and segment. This is useful for: finding specific moments in recordings, karaoke-style highlighting, and aligning transcripts with audio playback.

import openai
import os

client = openai.OpenAI(api_key=os.getenv('OPENAI_API_KEY'))

def transcribe_with_timestamps(audio_path):
    with open(audio_path, 'rb') as f:
        result = client.audio.transcriptions.create(
            model='whisper-1',
            file=f,
            response_format='verbose_json',
            timestamp_granularities=['word', 'segment']  # get both word and segment times
        )

    segments = []
    for seg in result.segments:
        segments.append({
            'start': seg.start,
            'end':   seg.end,
            'text':  seg.text
        })

    words = []
    if hasattr(result, 'words'):
        for word in result.words:
            words.append({'word': word.word, 'start': word.start, 'end': word.end})

    return {'text': result.text, 'segments': segments, 'words': words}

Deepgram SDK: Async Streaming

Deepgram is optimized for real-time streaming transcription. Audio is sent in chunks as it is recorded; partial transcripts are returned before the speaker finishes a sentence.

Install with pip install deepgram-sdk.

import asyncio
import os
from deepgram import DeepgramClient, PrerecordedOptions

client = DeepgramClient(os.getenv('DEEPGRAM_API_KEY'))

async def transcribe_with_deepgram(audio_path):
    with open(audio_path, 'rb') as f:
        audio_data = f.read()

    options = PrerecordedOptions(
        model='nova-2',            # Deepgram's best accuracy model
        language='en',
        smart_format=True,         # auto-add punctuation and formatting
        utterances=True,           # segment by speaker turns
        punctuate=True,
        diarize=True               # speaker identification
    )

    response = await client.listen.asyncrest.v('1').transcribe_file(
        {'buffer': audio_data},
        options
    )

    return response.results.channels[0].alternatives[0].transcript

Speaker Diarization

Diarization identifies who is speaking when — labeling segments as Speaker 0, Speaker 1, etc. Critical for meeting transcripts and multi-party conversations.

async def transcribe_with_diarization(audio_path):
    from deepgram import DeepgramClient, PrerecordedOptions
    import os

    dg_client = DeepgramClient(os.getenv('DEEPGRAM_API_KEY'))

    options = PrerecordedOptions(
        model='nova-2',
        diarize=True,
        utterances=True,
        smart_format=True
    )

    with open(audio_path, 'rb') as f:
        audio_data = f.read()

    response = await dg_client.listen.asyncrest.v('1').transcribe_file(
        {'buffer': audio_data}, options
    )

    utterances = []
    for utt in response.results.utterances:
        utterances.append({
            'speaker': f'Speaker {utt.speaker}',
            'start':   round(utt.start, 2),
            'end':     round(utt.end, 2),
            'text':    utt.transcript
        })

    return utterances

# Output:
# [{'speaker': 'Speaker 0', 'start': 0.0, 'end': 3.2, 'text': 'Hello everyone.'},
#  {'speaker': 'Speaker 1', 'start': 3.5, 'end': 6.1, 'text': 'Good morning!'}]

Language Detection

When the user's language is unknown, both Whisper and Deepgram can auto-detect the spoken language. Whisper detects language from the first 30 seconds of audio.

import openai
import os

client = openai.OpenAI(api_key=os.getenv('OPENAI_API_KEY'))

def detect_language(audio_path):
    with open(audio_path, 'rb') as f:
        # Use translations endpoint to get verbose JSON with language detection
        result = client.audio.transcriptions.create(
            model='whisper-1',
            file=f,
            response_format='verbose_json'
        )
    return {
        'detected_language': result.language,
        'text': result.text
    }

result = detect_language('user_audio.wav')
print(f"Language: {result['detected_language']}")
print(f"Text: {result['text'][:100]}")

# Deepgram: set language='auto' in options
from deepgram import PrerecordedOptions
options = PrerecordedOptions(model='nova-2', language='auto', detect_language=True)

Chunking Long Audio Files

Whisper's 25MB limit means long recordings (1-hour meetings) need chunking. Split audio by silence using pydub, then transcribe each chunk and concatenate.

Install with pip install pydub. Requires ffmpeg.

from pydub import AudioSegment
from pydub.silence import split_on_silence
import io

def transcribe_long_audio(audio_path, chunk_length_ms=60000):
    audio = AudioSegment.from_file(audio_path)

    # Split on silence
    chunks = split_on_silence(
        audio,
        min_silence_len=1000,   # 1 second of silence
        silence_thresh=-40,     # dBFS
        keep_silence=500        # keep 500ms at each end
    )

    # If no silence splitting worked, use fixed-length chunks
    if len(chunks) <= 1:
        chunks = [
            audio[i:i + chunk_length_ms]
            for i in range(0, len(audio), chunk_length_ms)
        ]

    full_transcript = []
    for i, chunk in enumerate(chunks):
        print(f'Transcribing chunk {i+1}/{len(chunks)}')
        buf = io.BytesIO()
        chunk.export(buf, format='mp3')
        buf.seek(0)
        buf.name = f'chunk_{i}.mp3'
        result = client.audio.transcriptions.create(model='whisper-1', file=buf)
        full_transcript.append(result.text)

    return ' '.join(full_transcript)

Deepgram Live Streaming

For real-time voice conversations, use Deepgram's WebSocket live streaming API. Audio chunks are sent as they are recorded; interim and final transcripts arrive within milliseconds.

import asyncio
import os
from deepgram import DeepgramClient, LiveTranscriptionEvents, LiveOptions

async def live_transcribe(on_transcript_callback):
    dg_client = DeepgramClient(os.getenv('DEEPGRAM_API_KEY'))

    connection = dg_client.listen.asynclive.v('1')

    async def on_message(self, result, **kwargs):
        sentence = result.channel.alternatives[0].transcript
        is_final = result.is_final
        if sentence:
            await on_transcript_callback(sentence, is_final)

    connection.on(LiveTranscriptionEvents.Transcript, on_message)

    options = LiveOptions(
        model='nova-2',
        language='en',
        smart_format=True,
        interim_results=True,  # get partial results before sentence ends
        endpointing=500        # ms of silence before finalizing
    )

    await connection.start(options)
    return connection  # caller sends audio chunks via connection.send(audio_chunk)

Error Handling and Retries

Audio transcription APIs can fail due to network issues, unsupported formats, or rate limits. Implement retries with exponential backoff and validate audio before sending.

import time
import os

MAX_FILE_SIZE_BYTES = 25 * 1024 * 1024  # 25MB
SUPPORTED_FORMATS = ('.mp3', '.mp4', '.wav', '.webm', '.m4a', '.flac', '.ogg')

def validate_audio_file(audio_path):
    if not os.path.exists(audio_path):
        raise FileNotFoundError(f'Audio file not found: {audio_path}')
    size = os.path.getsize(audio_path)
    if size > MAX_FILE_SIZE_BYTES:
        raise ValueError(f'File too large: {size / 1024 / 1024:.1f}MB (max 25MB)')
    ext = os.path.splitext(audio_path)[1].lower()
    if ext not in SUPPORTED_FORMATS:
        raise ValueError(f'Unsupported format: {ext}. Supported: {SUPPORTED_FORMATS}')

def transcribe_with_retry(audio_path, max_retries=3):
    validate_audio_file(audio_path)
    for attempt in range(max_retries):
        try:
            return transcribe_file(audio_path)
        except Exception as e:
            if attempt < max_retries - 1:
                wait = 2 ** attempt
                print(f'Retry {attempt + 1} after error: {e}. Waiting {wait}s')
                time.sleep(wait)
            else:
                raise

Choosing Between Whisper and Deepgram

Both tools excel in different scenarios. Use this decision matrix to choose the right one for your voice agent.

COMPARISON = '''
Whisper (OpenAI API):
  - Best for: batch transcription, podcast processing, meeting notes
  - Latency: file upload latency + ~1-5 seconds processing
  - Strengths: excellent multilingual, high accuracy, cheap
  - Word timestamps: yes (verbose_json)
  - Diarization: NO (must use separate tool)
  - Use when: accuracy > speed, offline processing

DeeGram:
  - Best for: real-time voice agents, call center transcription
  - Latency: <300ms for streaming, ~1s for file upload
  - Strengths: streaming, diarization, custom vocabulary
  - Word timestamps: yes, with confidence scores
  - Diarization: YES (built-in, up to 10 speakers)
  - Use when: speed > accuracy, real-time required

Rule of thumb:
  Agent voice conversation -> Deepgram live streaming
  Batch audio files -> Whisper API
  Meeting transcripts with speakers -> Deepgram with diarize=True
'''
print(COMPARISON)

Audio Format Conversion

Whisper and Deepgram support common audio formats, but user audio may arrive in unusual formats (ogg, opus, webm from browsers, m4a from iOS). Convert to standard WAV or MP3 before sending to the API.

from pydub import AudioSegment
import os

SUPPORTED_FORMATS = {'.mp3', '.wav', '.flac', '.m4a', '.ogg', '.webm', '.opus'}

def convert_to_wav(input_path, output_path=None):
    ext = os.path.splitext(input_path)[1].lower()
    if ext not in SUPPORTED_FORMATS:
        raise ValueError(f'Unsupported audio format: {ext}')

    if output_path is None:
        output_path = input_path.rsplit('.', 1)[0] + '.wav'

    # pydub handles format detection automatically
    audio = AudioSegment.from_file(input_path)

    # Normalize to 16kHz mono (optimal for Whisper)
    audio = audio.set_frame_rate(16000).set_channels(1)

    audio.export(output_path, format='wav')
    print(f'Converted {input_path} -> {output_path} ({len(audio) / 1000:.1f}s)')
    return output_path

def ensure_compatible_audio(audio_path):
    ext = os.path.splitext(audio_path)[1].lower()
    if ext in {'.mp3', '.wav', '.m4a', '.flac'}:
        return audio_path  # already compatible
    return convert_to_wav(audio_path)

Knowledge Check

What is speaker diarization in the context of speech-to-text transcription?

Recap: Speech-to-Text with Whisper and Deepgram

Whisper (OpenAI API) is ideal for batch transcription — high accuracy, multilingual, with word timestamps via verbose_json. Use it for file-based processing where latency is not critical.

Deepgram is designed for real-time streaming — live WebSocket transcription with interim results, built-in speaker diarization, and sub-300ms latency. Use it for interactive voice agents. Both support language detection and need retry logic and file validation in production.

Frequently asked questions

Is the “Speech-to-Text with Whisper and Deepgram” lesson free?

Yes — the full text of “Speech-to-Text with Whisper and Deepgram” is free to read here on the web, and the AI Agents course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the AI Agents course, upgrade to CoddyKit PRO.

What will I learn in “Speech-to-Text with Whisper and Deepgram”?

Real-time and batch transcription, language detection, and punctuation. You practise AI Agents with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start AI Agents?

No prior experience is required. AI Agents on CoddyKit is structured for beginners through advanced learners; this is — lesson 1 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Speech-to-Text with Whisper and Deepgram” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this AI Agents lesson?

Yes. Every AI Agents lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

  1. Speech-to-Text with Whisper and Deepgram
  2. Text-to-Speech in Agent Responses
  3. Building a Voice Conversation Loop
  4. Latency Optimization for Voice Agents
← Back to AI Agents