> ## Content Index
> Fetch the complete content index at: https://www.notatechguy.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# NVIDIA Switchyard routes AI traffic across models
- URL: https://www.notatechguy.com/nvidia-switchyard-routes-ai-traffic-across-models/
- Published: 2026-11-18T19:04:41.000Z
- Updated: 2026-11-18T19:04:41.000Z
- Description: NVIDIA's open-source Switchyard proxy routes LLM requests across providers so teams can mix cheap and frontier models without rewriting their code.
- Author: Marcello Babbili
- Tags: Technology & AI, Nvidia, OpenAI, Anthropic

NVIDIA released Switchyard, an open-source Rust proxy that routes AI agent requests across multiple model providers, and the repo reached GitHub's daily trending list on 18 November 2026 [S¹](https://github.com/NVIDIA-NeMo/Switchyard?ref=notatechguy.com). The tool translates between OpenAI and Anthropic APIs on the fly, so a team running Claude Code or Codex can point those agents at cheaper open-weight models without changing application code [S¹](https://github.com/NVIDIA-NeMo/Switchyard?ref=notatechguy.com)[P⁴](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/README.md?ref=notatechguy.com).

### Switchyard embeds into existing gateways and supplies only routing decisions

Switchyard's core is a Rust library called libsy, designed to sit inside a gateway or agent harness you already run [S¹](https://github.com/NVIDIA-NeMo/Switchyard?ref=notatechguy.com). You keep your own model clients and credentials. Switchyard supplies only the routing decision, picking which model handles each request by mixing cheaper options with stronger ones to trade off task success, cost and latency [S¹](https://github.com/NVIDIA-NeMo/Switchyard?ref=notatechguy.com). No independent benchmarks with error bars have been published for these routing decisions; the capability claims rest on NVIDIA's own documentation and README [S¹](https://github.com/NVIDIA-NeMo/Switchyard?ref=notatechguy.com).

NVIDIA also ships a self-hosted Agent Router that exposes an OpenAI-compatible API and runs on Envoy, the open-source proxy [S¹](https://github.com/NVIDIA-NeMo/Switchyard?ref=notatechguy.com). For teams that want a managed path, Switchyard is available through OpenRouter using the model identifier `nvidia/switchyard` [S¹](https://github.com/NVIDIA-NeMo/Switchyard?ref=notatechguy.com).

Tanay Varshney, Chris Alexiuk, Ayush Agarwal, Annie Surla and Moon Chung described the system in a NVIDIA Technical Blog post on 11 August 2026 as a way to direct agent workloads to both specialised and frontier models, trading performance against cost [P²](https://developer.nvidia.com/blog/route-ai-agent-workloads-across-models-with-nvidia-nemo-switchyard/?ref=notatechguy.com).

### Two integrations have documented gaps that block some routing strategies

The LiteLLM integration is experimental and checkout-only, pinned to LiteLLM version 1.102.0 [S¹](https://github.com/NVIDIA-NeMo/Switchyard?ref=notatechguy.com). It supports Stage routing based on request history plus Random routing, but it cannot service the intermediate model calls that classifier or escalation algorithms require [S¹](https://github.com/NVIDIA-NeMo/Switchyard?ref=notatechguy.com). If your routing strategy needs a small model to classify a request before forwarding it to a larger one, the LiteLLM path will not handle that.

NeMo Relay versions 0.8.x and 0.9.0 can lose upstream error status codes and details when the Switchyard plugin is enabled, even for models outside its routes [S¹](https://github.com/NVIDIA-NeMo/Switchyard?ref=notatechguy.com). A failed call to a model Switchyard is not routing may still surface as a garbled error, which makes debugging harder.

The README also advises evaluating the complete agent and routing configuration against a single-model baseline [S¹](https://github.com/NVIDIA-NeMo/Switchyard?ref=notatechguy.com). This is sound guidance, and it matters because routing algorithms that rely on intermediate model calls add their own latency and cost, which can undercut the savings they are meant to deliver.

### Coding-agent teams on single providers are the natural first adopters

A team running a coding agent like Claude Code against a single provider, paying frontier-model rates for every request including trivial ones, is the natural first user. Switchyard lets that team route simple requests to a cheaper open-weight model while reserving the frontier model for complex tasks, all behind the same API surface [S¹](https://github.com/NVIDIA-NeMo/Switchyard?ref=notatechguy.com)[P⁴](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/README.md?ref=notatechguy.com). For a DevOps engineer managing that setup, the practical change is swapping one base URL in the agent config to point at Switchyard's OpenAI-compatible endpoint rather than maintaining separate clients for each provider [S¹](https://github.com/NVIDIA-NeMo/Switchyard?ref=notatechguy.com).

The one thing to check first: run your real workload through Switchyard against your current single-model baseline and compare total task completion cost, not per-call cost. The README itself warns that a cheaper model call does not guarantee a cheaper successfully completed task [S¹](https://github.com/NVIDIA-NeMo/Switchyard?ref=notatechguy.com), because a weaker model may fail and trigger retries that cost more than a single frontier-model call would have.

No one outside NVIDIA has independently verified that Switchyard reduces cost or improves latency on any workload; no error bars, no third-party benchmarks and no released security audit exist [S¹](https://github.com/NVIDIA-NeMo/Switchyard?ref=notatechguy.com). The repo is trending, but GitHub trending status does not imply production readiness. The NVIDIA-NeMo GitHub organisation, created 27 May 2025, lists 32 public repositories and 2,139 followers [P³](https://github.com/NVIDIA-NeMo/?ref=notatechguy.com).

The next checkpoint for any team considering adoption is the NeMo Relay bug: if you run Relay 0.8.x or 0.9.0, test whether enabling the Switchyard plugin corrupts error reporting on models outside its routes before you ship it past a staging environment [S¹](https://github.com/NVIDIA-NeMo/Switchyard?ref=notatechguy.com).

---

*Sources: [S1 — NVIDIA-NeMo/Switchyard: Switchyard lets LLM applications route traffic](https://github.com/NVIDIA-NeMo/Switchyard?ref=notatechguy.com) · [P2 — Route AI Agents Across Models with NVIDIA NeMo Switchyard | NVIDIA Tec](https://developer.nvidia.com/blog/route-ai-agent-workloads-across-models-with-nvidia-nemo-switchyard/?ref=notatechguy.com) · [P3 — NVIDIA-NeMo](https://github.com/NVIDIA-NeMo/?ref=notatechguy.com) · [P4 — README.md](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/README.md?ref=notatechguy.com) · [P5 — anthropics/anthropic-sdk-python](https://github.com/anthropics/anthropic-sdk-python/?ref=notatechguy.com)*

---

*Written from 5 sourced items, 4 of them primary.*

## More from Not A Tech Guy

- [NVIDIA RTX Spark desktops launch with 128GB unified memory](https://www.notatechguy.com/nvidia-rtx-spark-desktops-launch-with-128gb-unified-memory/)
- [Nvidia, AMD chip demand booms despite OpenAI's frontier pause](https://www.notatechguy.com/nvidia-amd-chip-demand-booms-despite-openai-s-frontier-pause/)
- [Xing 4: China's 285B AI model on 8,192 Ascend cards](https://www.notatechguy.com/xing-4-china-s-285b-ai-model-on-8-192-ascend-cards/)