Hugging Face Speech-to-Speech: Build Local Voice Agents With Open-Source Models — 7,500+ GitHub Stars

Why Voice Agents Are the Next Developer Frontier

Every major AI company is racing to build voice interfaces. OpenAI has Realtime API. Google has Project Astra. But they all share the same problem: vendor lock-in, cloud dependency, and opaque pricing.

What if you could build the same voice-agent experience — real-time speech input, LLM reasoning, and speech output — running entirely on your own hardware, with every component swappable and auditable?

That's exactly what Hugging Face Speech-to-Speech delivers. This open-source framework (Apache 2.0 license) provides a production-ready voice-agent pipeline that you can deploy locally, in Docker, or behind any OpenAI-compatible WebSocket endpoint.

Architecture: A Modular Four-Stage Pipeline

Speech-to-Speech follows a clean cascade architecture where each stage runs in its own thread, connected by queues:

┌──────────┐    ┌──────────┐    ┌──────────┐    ┌──────────┐
│   VAD    │───▶│   STT    │───▶│   LLM    │───▶│   TTS    │
│ (Silero) │    │(Parakeet)│    │ (Any!)   │    │(Qwen3)   │
└──────────┘    └──────────┘    └──────────┘    └──────────┘
  Detects        Transcribes     Generates       Synthesizes
  speech         user speech     response        voice output

Stage 1: Voice Activity Detection (VAD)

Uses Silero VAD v5 to detect speech boundaries and handle turn-taking. This is the gatekeeper — it determines when you're speaking and when you've stopped, enabling natural conversation flow with interruption support.

Stage 2: Speech-to-Text (STT)

Default: Parakeet TDT 0.6B by NVIDIA — fast, accurate, supports 25 European languages. Alternative backends include Whisper, Faster Whisper, Lightning Whisper MLX (Apple Silicon), and Paraformer (FunASR).

Stage 3: Language Model (LLM)

The most flexible slot. Accepts any OpenAI-compatible API — whether that's OpenAI itself, Hugging Face Inference Providers, OpenRouter, a local vLLM server, or llama.cpp. You can also run Transformers or mlx-lm directly in-process.

Stage 4: Text-to-Speech (TTS)

Default: Qwen3-TTS (1.7B params) with GGML backend on Linux/Windows or mlx-audio on macOS. Alternatives include Kokoro-82M, Pocket TTS (with voice cloning), ChatTTS, and MMS TTS for broad multilingual coverage.

Getting Started in 3 Commands

The beauty of Speech-to-Speech is how quickly you can go from zero to a working voice agent:

# Install
pip install speech-to-speech

# Set your LLM provider (OpenAI by default)
export OPENAI_API_KEY=sk-...

# Start the voice agent server
speech-to-speech

That's it. This launches an OpenAI Realtime-compatible WebSocket server at ws://localhost:8765/v1/realtime with local STT (Parakeet TDT), local TTS (Qwen3-TTS), and your chosen LLM provider.

Going Fully Local (No Cloud, No API Keys)

Want to run everything on your own hardware? Here's the complete local stack:

# Terminal 1: Start llama.cpp with Gemma 4
llama-server -hf ggml-org/gemma-4-E4B-it-GGUF -np 2 -c 65536 -fa on --swa-full

# Terminal 2: Point Speech-to-Speech at your local LLM
speech-to-speech \
  --model_name "ggml-org/gemma-4-E4B-it-GGUF" \
  --responses_api_base_url "http://127.0.0.1:8080/v1" \
  --responses_api_api_key "" \
  --responses_api_stream \
  --enable_live_transcription

Zero cloud calls. Zero API keys. Zero data leaving your machine. This is the holy grail of privacy-first voice AI.

Mac Users: One Flag Does It All

# Optimized settings for Apple Silicon
speech-to-speech --local_mac_optimal_settings

# Or specify a local LLM explicitly
speech-to-speech \
  --local_mac_optimal_settings \
  --model_name mlx-community/Qwen3-4B-Instruct-2507-bf16

This automatically enables MPS acceleration, Parakeet TDT for STT, MLX LM as the LLM backend, and Qwen3-TTS via mlx-audio with 6-bit quantization.

Connecting Any OpenAI Realtime Client

The server implements the OpenAI Realtime protocol, so any compatible client works out of the box:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8765/v1",
    websocket_base_url="ws://localhost:8765/v1",
    api_key="not-needed",  # Local server, no auth
)

with client.realtime.connect(model="local") as conn:
    conn.send({
        "type": "session.update",
        "session": {
            "type": "realtime",
            "instructions": "You are a helpful assistant.",
            "audio": {
                "input": {
                    "turn_detection": {
                        "type": "server_vad",
                        "interrupt_response": True,
                    }
                }
            },
        },
    })

    for event in conn:
        print(event.type)

This means you can use the OpenAI Python SDK, the JavaScript SDK, or any Realtime-compatible client library — just change the base URL to point at your local server.

Real-World Example: Building a Voice-Controlled Customer Support Bot

Let's build a practical voice agent: a customer support bot that answers questions about a product, runs entirely locally, and supports multiple languages.

# Step 1: Start local LLM with a fine-tuned support model
llama-server \
  -hf your-org/support-bot-gguf \
  -np 4 -c 32768 -fa on

# Step 2: Launch Speech-to-Speech with custom instructions
speech-to-speech \
  --mode realtime \
  --stt parakeet-tdt \
  --llm_backend responses-api \
  --tts qwen3 \
  --model_name "your-org/support-bot-gguf" \
  --responses_api_base_url "http://127.0.0.1:8080/v1" \
  --responses_api_api_key "" \
  --language auto \
  --enable_lang_prompt \
  --enable_live_transcription \
  --responses_api_stream

# Step 3: Connect your web app
# Use any OpenAI Realtime-compatible JS/Python client
# pointing at ws://your-server:8765/v1/realtime

The --language auto flag enables automatic language detection — the STT identifies what language the user is speaking, and the LLM responds in the same language. Add --enable_lang_prompt to explicitly instruct smaller models to match the detected language.

Docker deployment for production:

# docker-compose.yml handles everything
docker compose up

The compose file starts a llama.cpp server with Gemma 4, launches the pipeline server, and exposes ports 8080, 12345, and 12346. Deploy it behind a reverse proxy and you have a production voice agent.

Multi-Language Support: 25+ Languages Out of the Box

Language coverage depends on your chosen STT and TTS backends:

• STT - Parakeet TDT: 25 European languages
• STT - Whisper: Broad multilingual coverage (90+ languages)
• TTS - Qwen3-TTS: Multilingual with auto-detection
• TTS - MMS: 1,100+ language-specific checkpoints
• TTS - ChatTTS: English and Chinese

For maximum language coverage, pair Whisper STT with Qwen3-TTS or MMS TTS:

speech-to-speech \
  --stt whisper-mlx \
  --stt_model_name large-v3 \
  --tts qwen3 \
  --language auto \
  --enable_lang_prompt \
  --llm_backend mlx-lm \
  --model_name mlx-community/Qwen3-4B-Instruct-2507-bf16

Key Benefits

• 🔒 Fully Private: Every component runs locally — no data leaves your infrastructure. Critical for healthcare, finance, and regulated industries.
• 🔌 Modular & Swappable: Change any stage (VAD, STT, LLM, TTS) independently. Mix and match backends to fit your hardware and latency budget.
• ⚡ Low Latency: Threaded pipeline with queue-based communication. Streaming TTS means audio starts playing before the full response is generated.
• 🔗 OpenAI Compatible: Exposes the standard Realtime WebSocket protocol. Use existing OpenAI SDK clients — just swap the base URL.
• 🐳 Production Ready: Docker support, battle-tested on thousands of Reachy Mini robots. Not a toy — it's running in production today.
• 🌍 Multilingual: Support 25+ languages with automatic detection. Swap Whisper + MMS for 1,100+ languages.
• 💰 Cost Efficient: Zero per-call API fees when running fully local. One GPU investment, unlimited voice conversations.
• 📖 Apache 2.0: Use it commercially, modify it, build products on top of it — no restrictions.

Frequently Asked Questions

1. What hardware do I need to run Speech-to-Speech?

For a fully local setup, you need a GPU with at least 8GB VRAM to run the LLM (Gemma 4 quantized), STT (Parakeet TDT is lightweight), and TTS (Qwen3-TTS) together. Apple Silicon Macs work excellently with MLX backends. For CPU-only setups, expect higher latency but it's still functional. The STT and TTS components are relatively lightweight — the LLM is the bottleneck.

2. Can I use it with GPT-4o or Claude instead of a local model?

Absolutely. The LLM slot accepts any OpenAI-compatible API. Set your OPENAI_API_KEY and use --model_name gpt-4o-mini or point --responses_api_base_url to any compatible provider like OpenRouter, Together AI, or Hugging Face Inference Providers. The STT and TTS still run locally for low-latency audio processing.

3. How does the latency compare to OpenAI's Realtime API?

With a local LLM like Gemma 4 quantized on a modern GPU, you can achieve comparable latency to cloud-based solutions — typically 500ms-1.5s from end-of-speech to start-of-response audio. The advantage is consistent latency without network variability. Using a hosted provider like Groq through the HF router can achieve even lower latency due to their fast inference.

4. Is it suitable for production use?

Yes. Hugging Face uses this exact pipeline as the conversation backend for thousands of Reachy Mini robots in production. The codebase is mature, well-tested, and includes Docker deployment, proper error handling, and the full OpenAI Realtime protocol implementation. It's not a research prototype — it's production-grade software.

5. Can I add custom tools or function calling?

Yes. The pipeline supports streaming tool calls through the OpenAI Realtime protocol. When the LLM generates a tool call, it's exposed as an event in the Realtime API stream. You can implement tool execution in your client application and feed results back into the conversation. This works with any LLM backend that supports function calling.

6. How do I add voice cloning or custom voices?

Use the Pocket TTS backend (--tts pocket), which supports voice cloning from reference audio files. Qwen3-TTS also supports custom voice profiles via the --qwen3_tts_speaker flag with preset voices (Aiden, etc.). For maximum customization, you can swap in any TTS model that supports custom voice inputs.

7. Can multiple users connect simultaneously?

Yes. In realtime mode, the server handles multiple concurrent WebSocket connections. Use --num_pipelines to control the pool size. When using llama.cpp as the LLM backend, set -np (number of parallel slots) to match your expected concurrent users. A single GPU with 24GB VRAM can typically handle 4-8 concurrent voice conversations.

🚀 Start Learning to Code Today — Explore CoddyKit Courses