LiveKit Agents: Build Realtime Voice AI Agents With 12,000+ GitHub Stars — The Complete Guide

Voice AI is no longer science fiction — it is becoming a core interface for applications. From customer support to personal assistants, voice agents that can listen, reason, and respond in real-time are transforming how we interact with software.

LiveKit Agents is one of the most popular open-source frameworks for building these agents, with over 12,600 GitHub stars and active development. Unlike simple speech-to-text pipelines, LiveKit Agents provides a complete, production-grade infrastructure for conversational AI that can see, hear, and understand.

What Makes LiveKit Agents Different?

LiveKit Agents is not just another AI wrapper — it is a realtime communication framework built on WebRTC. Here is what sets it apart:

• WebRTC-native: Low-latency audio/video streaming without complex infrastructure
• Telephony integration: Connect to phone systems via SIP — your agent can make and receive calls
• Multi-modal: Handle voice, video, and data simultaneously
• Production-ready: Built-in job scheduling, testing framework, and deployment patterns
• MCP support: Integrate external tools with one line of code
• Semantic turn detection: Transformer-based model that reduces interruptions

Core Architecture

LiveKit Agents uses a clean, composable architecture:

Agent

An LLM-based application with defined instructions and tools. Think of it as the "brain" of your voice assistant.

AgentSession

A container that manages interactions between an agent and end users. It handles the STT → LLM → TTS pipeline.

AgentServer

The main process that coordinates job scheduling and launches agents for user sessions.

from livekit.agents import (
    Agent,
    AgentServer,
    AgentSession,
    JobContext,
    cli,
    inference,
)

server = AgentServer()

@server.rtc_session()
async def entrypoint(ctx: JobContext):
    session = AgentSession(
        vad=inference.VAD(),
        stt=inference.STT("deepgram/nova-3", language="multi"),
        llm=inference.LLM("google/gemma-4-31b-it"),
        tts=inference.TTS("cartesia/sonic-3", voice="9626c31c-bec5-4cca-baa8-f8ba9e84c8bc"),
    )

    agent = Agent(
        instructions="You are a friendly voice assistant.",
        tools=[],
    )

    await session.start(agent=agent, room=ctx.room)
    await session.generate_reply(instructions="greet the user")

if __name__ == "__main__":
    cli.run_app(server)

Building Your First Voice Agent

Let us build a voice agent that can check the weather:

from livekit.agents import (
    Agent,
    AgentServer,
    AgentSession,
    JobContext,
    RunContext,
    cli,
    function_tool,
    inference,
)

@function_tool
async def lookup_weather(
    context: RunContext,
    location: str,
):
    """Used to look up weather information."""
    # Integrate with your weather API here
    return {"weather": "sunny", "temperature": 72}

server = AgentServer()

@server.rtc_session()
async def entrypoint(ctx: JobContext):
    session = AgentSession(
        vad=inference.VAD(),
        stt=inference.STT("deepgram/nova-3"),
        llm=inference.LLM("openai/gpt-4o-mini"),
        tts=inference.TTS("cartesia/sonic-3"),
    )

    agent = Agent(
        instructions="You are a helpful weather assistant. Ask the user for their location, then use the lookup_weather tool.",
        tools=[lookup_weather],
    )

    await session.start(agent=agent, room=ctx.room)
    await session.generate_reply(instructions="greet the user and ask where they are")

if __name__ == "__main__":
    cli.run_app(server)

Run it locally with:

python myagent.py console

This gives you a terminal-based voice conversation using your microphone — perfect for testing before deployment.

Multi-Agent Patterns

One of LiveKit Agents most powerful features is multi-agent orchestration. You can build systems where agents hand off to each other based on context.

Example: Story Generation System

Here is a system where one agent gathers information, then hands off to a storyteller agent:

class IntroAgent(Agent):
    def __init__(self):
        super().__init__(
            instructions="Gather the users name and location to personalize a story."
        )

    async def on_enter(self):
        self.session.generate_reply(
            instructions="Ask the user for their name and where they are from"
        )

    @function_tool
    async def information_gathered(
        self,
        context: RunContext,
        name: str,
        location: str,
    ):
        """Called when user provides name and location."""
        context.userdata.name = name
        context.userdata.location = location
        
        # Hand off to story agent
        story_agent = StoryAgent(name, location)
        return story_agent, "Let us start the story!"

class StoryAgent(Agent):
    def __init__(self, name: str, location: str):
        super().__init__(
            instructions=f"Tell an engaging story featuring {name} from {location}.",
            llm=openai.realtime.RealtimeModel(voice="echo"),  # Use Realtime API
        )

    async def on_enter(self):
        self.session.generate_reply()

This pattern enables complex workflows like:

• Triage → Specialist: Front desk agent routes to department-specific agents
• Interview → Analysis: Data collection agent hands off to analysis agent
• Support → Sales: Technical support transitions to upsell opportunity

Production Features

Telephony Integration

LiveKit Agents works with LiveKit SIP stack, allowing your agent to:

• Make outbound calls to phone numbers
• Receive inbound calls from customers
• Handle IVR menus and call routing
• Record and transcribe calls

MCP (Model Context Protocol) Support

Integrate external tools with one line:

from livekit.agents import mcp

agent = Agent(
    instructions="You can search the web and access databases.",
    tools=mcp.tools_from_server("my-mcp-server"),
)

Built-in Testing Framework

Write tests with LLM-based judges to ensure quality:

@pytest.mark.asyncio
async def test_order_placement():
    llm = google.LLM()
    async with AgentSession(llm=llm) as sess:
        await sess.start(OrderAgent())
        result = await sess.run(
            user_input="I would like to order a pizza"
        )
        
        await (
            result.expect.next_event()
            .is_message(role="assistant")
            .judge(llm, intent="ask for pizza size and toppings")
        )

Real-World Example: Restaurant Ordering System

LiveKit Agents includes a complete restaurant ordering example that handles:

• Phone call reception and greeting
• Menu navigation and order taking
• Modification handling ("actually, make it two")
• Payment processing integration
• Order confirmation and ETA

This demonstrates how to build production voice agents that can handle the complexity and edge cases of real business operations.

Key Benefits

• Production-Ready: Used in real applications, not just demos
• Low Latency: WebRTC ensures sub-200ms audio round-trip
• Flexible STT/LLM/TTS: Mix and match providers (Deepgram, OpenAI, Cartesia, Google, etc.)
• Telephony Native: Connect to phone systems without additional infrastructure
• Open Source: Apache-2.0 license, self-host or use LiveKit Cloud
• Multi-Platform: Client SDKs for web, iOS, Android, Flutter, React Native
• Testing Built-in: LLM judges ensure conversation quality
• MCP Integration: Access any tool via Model Context Protocol

Getting Started

Install LiveKit Agents:

pip install "livekit-agents[openai,deepgram,cartesia]"

Set up environment variables:

export LIVEKIT_URL="your-livekit-url"
export LIVEKIT_API_KEY="your-api-key"
export LIVEKIT_API_SECRET="your-api-secret"
export OPENAI_API_KEY="your-openai-key"
export DEEPGRAM_API_KEY="your-deepgram-key"

Try the Agents Playground for instant testing without deployment.

FAQ

What is LiveKit Agents?

LiveKit Agents is an open-source Python framework for building realtime voice AI agents. It provides WebRTC-based communication, telephony integration, and multi-agent orchestration capabilities. The framework has over 12,600 GitHub stars and is Apache-2.0 licensed.

How does LiveKit Agents differ from other voice AI frameworks?

LiveKit Agents is built on WebRTC for low-latency communication, includes native telephony support via SIP, and provides production features like job scheduling, testing frameworks, and multi-agent patterns. Unlike simple STT→LLM→TTS pipelines, it is designed for real-world deployment with features like semantic turn detection and MCP tool integration.

Can I use LiveKit Agents for phone calls?

Yes! LiveKit Agents integrates with LiveKit SIP stack, allowing your agents to make outbound calls, receive inbound calls, and handle traditional telephony features like IVR menus and call routing. This makes it ideal for customer support and business automation applications.

What LLM providers does LiveKit Agents support?

LiveKit Agents supports multiple providers including OpenAI (GPT-4, Realtime API), Google (Gemini), Anthropic (Claude), and any OpenAI-compatible API. You can also use LiveKit Inference for a unified API across different models.

Is LiveKit Agents suitable for production use?

Absolutely. LiveKit Agents is used in production applications and includes features like built-in testing with LLM judges, job scheduling for scaling, telephony integration, and comprehensive client SDKs. The framework is actively maintained and has a large community.

How do I test voice agents built with LiveKit?

LiveKit Agents includes a built-in testing framework with pytest integration. You can write tests that simulate conversations and use LLM-based judges to evaluate response quality. The framework also supports console mode for local testing with your microphone.

What is the latency like for LiveKit Agents?

LiveKit Agents uses WebRTC for audio streaming, which typically achieves sub-200ms round-trip latency. Combined with fast STT providers like Deepgram and optimized LLM inference, you can achieve near-instantaneous voice conversations that feel natural.

Can I self-host LiveKit Agents?

Yes, LiveKit Agents is fully open-source (Apache-2.0). You can self-host the entire stack including the LiveKit server, or use LiveKit Cloud for managed infrastructure. Both options provide the same functionality.