NVIDIA NeMo Switchyard: The Open-Source LLM Router With 1,300+ GitHub Stars That Routes AI Traffic Across Models

What Is NVIDIA NeMo Switchyard?

If you're building applications that use multiple LLMs—GPT-4, Claude, Llama, Mistral, or self-hosted models—you've probably faced this problem: how do you route different requests to different models without rewriting your entire application?

NVIDIA NeMo Switchyard solves this by acting as an intelligent proxy layer between your application and your LLM backends. It's a Rust-based tool that:

• Translates protocols between OpenAI Chat, Anthropic Messages, and OpenAI Responses formats
• Routes requests using algorithms like LLM classification, signal-driven staging, or random A/B splits
• Collects metrics on latency, tokens, errors, and routing decisions via Prometheus

Your application keeps speaking its native API (OpenAI or Anthropic), while Switchyard forwards requests to vLLM, NVIDIA NIM, Ollama, or any OpenAI-compatible endpoint in the format that backend expects.

Why Does LLM Routing Matter?

Modern AI applications rarely use just one model. Consider these scenarios:

• Cost optimization: Route simple queries to cheaper models (GPT-3.5, Llama 3) and complex reasoning to premium models (GPT-4, Claude Opus)
• Performance testing: Split traffic 80/20 between two models to benchmark quality and latency
• Fallback chains: If Model A times out or errors, automatically retry with Model B
• Specialized models: Use a code-specialized model for programming tasks and a general model for conversation

Without a routing layer, you'd need to implement all this logic in your application code. Switchyard externalizes it into a configurable, observable proxy.

Routing Strategies: How Switchyard Decides

Switchyard provides four built-in routing algorithms, each designed for different use cases:

1. LLM Classifier Routing

This strategy uses a lightweight LLM to analyze the incoming request and decide which backend should handle it. For example:

[route]
name = "smart-router"
algorithm = "llm_classifier"

[route.classifier]
model = "gpt-3.5-turbo"
prompt = """
Analyze this request and respond with 'strong' if it requires advanced reasoning,
or 'weak' if it's a simple query:
{request}
"""

[[route.targets]]
name = "strong-tier"
model = "gpt-4"
when = "strong"

[[route.targets]]
name = "weak-tier"
model = "gpt-3.5-turbo"
when = "weak"

The classifier runs first, inspects the request, and routes it to the appropriate tier. This is powerful for cost optimization—most queries don't need GPT-4.

2. Stage Router

Instead of calling an LLM to classify, the stage router looks at signals already in the conversation: tool results, errors, message length, or specific keywords. It's faster and cheaper than LLM classification.

[route]
name = "stage-router"
algorithm = "stage_router"

[[route.stages]]
condition = "tool_results_present"
target = "code-model"

[[route.stages]]
condition = "error_count > 2"
target = "strong-model"

[[route.stages]]
condition = "default"
target = "fast-model"

If the conversation includes tool results (like code execution output), route to a code-specialized model. If there have been multiple errors, escalate to a stronger model. Otherwise, use the fast, cheap model.

3. Escalation Router

This hybrid approach runs every request on the weak tier first, then uses a judge LLM to evaluate the response. If the judge determines the answer is insufficient, it re-runs the request on the strong tier.

It's more expensive than pure classification (you're making two calls for complex queries), but it ensures quality without upfront classification overhead.

4. Random Routing

For A/B testing or baseline comparisons, random routing splits traffic by percentage:

[route]
name = "ab-test"
algorithm = "random"

[[route.targets]]
name = "control"
model = "gpt-4"
weight = 0.8

[[route.targets]]
name = "experiment"
model = "claude-3-opus"
weight = 0.2

80% of requests go to GPT-4, 20% to Claude Opus. Compare metrics afterward to see which performed better.

Real-World Example: Building a Cost-Optimized Coding Assistant

Let's say you're building a coding assistant that handles both simple questions ("How do I reverse a string in Python?") and complex architectural decisions ("Design a microservices architecture for an e-commerce platform").

Without Switchyard, you'd either:

• Use GPT-4 for everything (expensive)
• Use GPT-3.5 for everything (lower quality on complex tasks)
• Build custom routing logic in your app (time-consuming)

With Switchyard, you configure a classifier route:

[route]
name = "coding-assistant"
algorithm = "llm_classifier"

[route.classifier]
model = "gpt-3.5-turbo"
prompt = """
Classify this coding question:
- 'simple' for basic syntax, standard library usage, or common patterns
- 'complex' for architecture design, debugging complex issues, or advanced algorithms

Question: {request}

Respond with only 'simple' or 'complex'.
"""

[[route.targets]]
name = "simple-questions"
model = "llama-3-70b"
when = "simple"

[[route.targets]]
name = "complex-questions"
model = "gpt-4"
when = "complex"

Now launch your coding agent (Claude Code, Codex, or OpenClaw) through Switchyard:

export OPENROUTER_API_KEY="your-key"
switchyard launch claude --model coding-assistant

Your agent speaks its native Anthropic API. Switchyard translates to OpenAI format for Llama 3 and back to Anthropic for the response. Simple questions hit the cheaper Llama model; complex ones escalate to GPT-4. You get 70% cost savings without sacrificing quality on hard problems.

Key Benefits of Switchyard

• Protocol translation: Your app uses one API format; Switchyard handles the rest
• Cost optimization: Route 80% of traffic to cheaper models, 20% to premium models
• Observability: Prometheus metrics for every request, token, and routing decision
• Flexibility: Use built-in algorithms or write your own in Rust
• Vendor independence: Switch backends without changing application code
• Performance: Rust-based proxy adds minimal latency overhead

Getting Started with Switchyard

Switchyard offers three paths depending on your use case:

Launcher Path (Quick Start)

Install the CLI tool and launch coding agents through Switchyard:

# Install uv (Python package manager)
curl -LsSf https://astral.sh/uv/install.sh | sh
source "$HOME/.local/bin/env"

# Install Switchyard CLI
uv tool install --python 3.10 "nemo-switchyard[cli]"

# Launch Claude Code through Switchyard
export OPENROUTER_API_KEY="your-key"
switchyard launch claude --model switchyard

Server Path (Standalone Proxy)

Run Switchyard as a persistent proxy service:

# Install Rust and Cargo, then:
cargo install --locked switchyard-server

# Create routes.toml (see docs for examples)
# Validate configuration:
switchyard-server --config routes.toml --dry-run

# Start the server:
switchyard-server --config routes.toml --host 127.0.0.1 --port 4000

# Test it:
curl http://localhost:4000/health

Library Path (Embed in Rust Apps)

For custom integrations, embed Switchyard's routing algorithms in your own Rust application:

[dependencies]
switchyard-libsy = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git" }
switchyard-protocol = { git = "https://github.com/NVIDIA-NeMo/Switchyard.git" }

The library never makes HTTP calls itself—it decides which target to use and returns that decision to your code, giving you full control over the networking layer.

Important Considerations

Switchyard is pre-alpha experimental software. The API and algorithms are evolving rapidly. It's excellent for prototyping and experimentation, but NVIDIA explicitly warns against production use until v1.0 is released.

That said, the core concepts—LLM classification, signal-driven routing, protocol translation—are production-ready patterns. Even if you don't adopt Switchyard itself, studying its architecture will inform how you build your own routing layer.

Frequently Asked Questions

Q: Is Switchyard production-ready?

A: No. Switchyard is pre-alpha software with a rapidly evolving API. It's designed for experimentation and prototyping. NVIDIA explicitly states it's not for production use until v1.0.

Q: Does Switchyard support streaming responses?

A: Yes. Switchyard translates streaming responses between OpenAI and Anthropic formats, so your application receives chunks in its expected format regardless of which backend served the request.

Q: Can I use Switchyard with self-hosted models like Llama or Mistral?

A: Absolutely. Switchyard works with any OpenAI-compatible endpoint, including vLLM, Ollama, NVIDIA NIM, and self-hosted models. You can mix cloud APIs (OpenAI, Anthropic) with self-hosted backends in the same route.

Q: How much latency does Switchyard add?

A: Switchyard is written in Rust and designed for minimal overhead. Typical proxy overhead is under 10ms, though routing algorithms like LLM classification add their own latency (one extra model call to classify the request).

Q: Can I write custom routing algorithms?

A: Yes. If you use Switchyard as a library (switchyard-libsy), you can implement custom routing logic in Rust. The library provides the routing framework; you define the decision logic.

Q: Does Switchyard handle authentication and API keys?

A: Switchyard passes through authentication headers but doesn't manage API keys itself. You configure backend credentials in your routes.toml file, and Switchyard injects them when forwarding requests.

Q: What metrics does Switchyard collect?

A: Switchyard exposes Prometheus metrics for request count, error rates, latency (p50, p95, p99), token usage (input/output), and routing decisions. You can scrape these with Prometheus and visualize them in Grafana.

Ready to optimize your LLM costs? Check out our AI and machine learning courses to learn more about building production-ready AI applications.