Hugging Face Speech-to-Speech: Build Local Voice AI Agents With This 10,000-Star Open-Source Pipeline

Why Voice AI Agents Are the Next Developer Frontier

Voice interfaces are no longer futuristic — they're practical. From customer support bots to personal coding assistants, developers increasingly need to build conversational AI that listens, thinks, and speaks. The problem? Most voice AI stacks are proprietary, expensive, and locked behind cloud APIs.

Enter Hugging Face speech-to-speech — an open-source voice agent pipeline with 10,278 GitHub stars and 1,257 forks under the Apache 2.0 license. It's the same engine powering thousands of Reachy Mini robots in production, and you can run it entirely on your own hardware.

If you've ever wanted to build a voice assistant without sending a single byte to OpenAI, Google, or AWS — this is your toolkit.

How the Pipeline Works: Four Swappable Stages

The architecture is elegantly simple. The pipeline is a cascade of four components, each running in its own thread and connected by queues:

1. Voice Activity Detection (VAD)

Silero VAD v5 detects when a user starts and stops speaking. It handles turn-taking and interruption detection — critical for natural conversation. This component is built-in and requires no configuration.

2. Speech-to-Text (STT)

The default is NVIDIA's Parakeet TDT (0.6B parameters), which runs on CUDA, CPU, or Apple Silicon. Alternatives include Whisper, Faster Whisper, Lightning Whisper MLX (Apple Silicon), and Paraformer. Each backend is selectable via CLI flags:

# Use Faster Whisper instead of default Parakeet
speech-to-speech --stt faster-whisper

3. Language Model (LLM)

This is where the magic happens. The LLM slot speaks OpenAI-compatible protocols, so you can point it at:

• Any hosted provider (OpenAI, Anthropic via proxy, etc.)
• Hugging Face Inference Providers
• A self-hosted vLLM or llama.cpp server
• A local mlx-lm instance on Apple Silicon

# Fully local: serve Gemma 4 with llama.cpp
llama-server -hf ggml-org/gemma-4-E4B-it-GGUF -np 2 -c 65536 -fa on --swa-full

# Point speech-to-speech at your local server
speech-to-speech \
  --model_name "ggml-org/gemma-4-E4B-it-GGUF" \
  --responses_api_base_url "http://127.0.0.1:8080/v1" \
  --responses_api_api_key ""

4. Text-to-Speech (TTS)

The default TTS engine is Qwen3-TTS (1.7B parameters), which streams audio with low latency. Alternatives include Kokoro-82M, Pocket TTS, ChatTTS, and MMS TTS. On macOS, it uses mlx-audio; on Linux, it defaults to a GGML backend for speed.

# Use Kokoro TTS instead
pip install "speech-to-speech[kokoro]"
speech-to-speech --tts kokoro

Getting Started in 3 Minutes

Here's the fastest path from zero to a working voice agent:

# 1. Install
pip install speech-to-speech

# 2. Set your OpenAI key (or skip for fully local)
export OPENAI_API_KEY=sk-...

# 3. Start the server
speech-to-speech

This starts an OpenAI Realtime-compatible WebSocket server at ws://localhost:8765/v1/realtime. Any compatible client can connect immediately.

Connect From Python (OpenAI SDK)

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8765/v1",
    websocket_base_url="ws://localhost:8765/v1",
    api_key="not-needed",
)

with client.realtime.connect(model="local") as conn:
    conn.send({
        "type": "session.update",
        "session": {
            "type": "realtime",
            "instructions": "You are a helpful coding assistant.",
            "audio": {
                "input": {
                    "turn_detection": {
                        "type": "server_vad",
                        "interrupt_response": True,
                    }
                }
            },
        },
    })

    for event in conn:
        print(event.type)

Mac Users: One-Command Local Setup

# Uses MPS, MLX LM, Qwen3-TTS — all optimized for Apple Silicon
speech-to-speech --local_mac_optimal_settings

This single flag configures the entire stack for optimal performance on M1/M2/M3/M4 Macs, using Metal Performance Shaders for all model inference.

Real-World Example: Building a Voice Coding Assistant

Let's build a practical voice assistant that helps developers write and review code — running entirely on a local machine.

# Start with a coding-focused system prompt and local Qwen3-32B
speech-to-speech \
  --model_name "Qwen/Qwen3-32B" \
  --responses_api_base_url "http://localhost:8000/v1" \
  --responses_api_api_key "" \
  --stt parakeet-tdt \
  --tts qwen3 \
  --chat_size 50 \
  --mode realtime

Now connect your preferred OpenAI Realtime client and start coding by voice:

• "Review this function for security vulnerabilities" — The agent reads your code, analyzes it, and speaks back findings
• "Explain how async/await works in Python" — Get a verbal explanation with code examples spoken aloud
• "Generate a REST API endpoint for user authentication" — The agent writes the code and reads it back to you

The --enable_llm_proxy flag is particularly powerful: it exposes the LLM as a standard OpenAI-compatible endpoint alongside the voice server, so you can run background tasks (summaries, code generation) without interrupting the voice conversation.

Run Modes: Pick Your Transport Layer

Mode	Transport	Best For
realtime (default)	OpenAI Realtime over WebSocket/WebRTC	App integration, standard voice API
local	Microphone + speakers directly	Quick testing, no client needed
raw-websocket	Raw PCM over WebSocket	Minimal custom clients
socket	Raw PCM over TCP	Remote server + local mic

For most developers, realtime mode is the sweet spot — it gives you full interruption handling, live transcription events, and tool-call events out of the box.

Key Benefits of Hugging Face Speech-to-Speech

• 100% Open Source: Apache 2.0 license, no vendor lock-in, full source code available
• Fully Local Option: Run every component on your own hardware — zero cloud dependencies
• OpenAI Compatible: Drop-in replacement for OpenAI Realtime API; existing clients work unchanged
• Modular Architecture: Swap any component (STT, LLM, TTS) independently via CLI flags
• Production-Tested: Powers thousands of Reachy Mini robots in real deployments
• Cross-Platform: CUDA, CPU, and Apple Silicon support with platform-specific optimizations
• Low Latency: Streaming audio pipeline with real-time turn detection and interruption
• Multi-Language: Qwen3-TTS supports multiple languages with automatic detection
• Multiple Backends: Choose from 5+ STT engines, any OpenAI-compatible LLM, and 5+ TTS engines
• Easy Installation: Single pip install command with optional extras for specific backends

Frequently Asked Questions

1. Is Hugging Face speech-to-speech completely free?

Yes, the library itself is free under the Apache 2.0 license. All models it uses (Silero VAD, Parakeet TDT, Qwen3-TTS) are also open-source. The only cost is if you choose to use a hosted LLM provider like OpenAI — but you can run everything locally for free using llama.cpp or vLLM.

2. What hardware do I need to run it?

Minimum requirements depend on your model choices. With smaller models (e.g., Qwen3-4B + Kokoro-82M TTS), a modern laptop with 16GB RAM works well. For larger models like Qwen3-32B, you'll want a GPU with 24GB+ VRAM or use quantized GGUF versions. Apple Silicon Macs (M1+) get excellent performance with the MLX backends.

3. Can I use it with my existing OpenAI Realtime clients?

Absolutely. The server implements the OpenAI Realtime WebSocket protocol, including input_audio_buffer.append, session.update, conversation.item.create, response.create, and streaming audio deltas. Any client built for OpenAI Realtime will connect without modification.

4. How does it compare to OpenAI's Realtime API?

Unlike OpenAI's proprietary Realtime API, speech-to-speech gives you full control over every component. You choose which models run where, you own all the data, and there are no per-minute charges. The trade-off is setup complexity — you manage the infrastructure instead of paying OpenAI to do it.

5. Does it support multiple languages?

Yes. Qwen3-TTS supports multilingual speech synthesis with automatic language detection. For STT, Whisper-based backends support 99+ languages. You can configure the language explicitly with the --qwen3_tts_language flag or use auto for detection.

6. Can I build a voice chatbot for my website with this?

Yes. Run the server in realtime mode, expose the WebSocket endpoint, and connect from a browser using the Web Audio API and a WebSocket client. The OpenAI Realtime protocol works over standard WebSockets, making browser integration straightforward.

7. What's the latency like compared to cloud solutions?

On a good GPU with optimized models, end-to-end latency (speech-in to speech-out) can be under 500ms — comparable to or better than cloud solutions since there's no network round-trip. On CPU-only setups with larger models, expect 1-3 seconds. The streaming architecture means audio starts playing before the full response is generated.

🎓 Want to master AI development? Explore hands-on courses at CoddyKit Courses — from Python fundamentals to advanced machine learning, build real projects and level up your skills.