> ## Content Index > Fetch the complete content index at: https://www.notatechguy.com/llms.txt > Use this file to discover other available public pages before exploring further. # SGLang-Omni serves voice AI, claims 326% throughput on H100 - URL: https://www.notatechguy.com/sglang-omni-serves-voice-ai-claims-326-throughput-on-h100/ - Published: 2026-11-14T03:29:20.000Z - Updated: 2026-11-14T03:29:20.000Z - Description: SGLang-Omni v0.2.0 is a multi-stage serving runtime for speech and multimodal models, claiming up to 326% throughput gains on H100 GPUs. - Author: Marcello Babbili - Tags: Technology & AI The sgl-project team landed SGLang-Omni, a speech and multimodal serving runtime, on GitHub's daily trending list on November 12 with 1,770 total stars [S¹](https://github.com/sgl-project/sglang-omni?ref=notatechguy.com). Its README claims throughput gains up to 326.5% on H100 GPUs across eight models — figures with no error bars, no released benchmark code, and no third-party reproduction [S¹](https://github.com/sgl-project/sglang-omni?ref=notatechguy.com). The parent SGLang engine, created in January 2024, has grown to 38,809 stars under an Apache 2.0 licence [P²](https://github.com/sgl-project/sglang/?ref=notatechguy.com). **My read:** The number I would watch is not the star count but the 14 merged pull requests from the OpenRSI collaboration. That is a concrete, checkable artifact on GitHub, and the throughput range attached to it is wide enough, 7.6% to 326.5%, that I would want to see which models sit at which end before drawing any conclusion. The "Interactive Super Intelligence" branding is doing a lot of work for a project with 1,770 stars. What is real and verifiable today is the PyPI package along with the installation docs and the code itself. SGLang-Omni is a multi-stage serving runtime for speech and text-to-speech, plus what the maintainers call "omni" models: systems that handle audio, images, and video through a single pipeline [S¹](https://github.com/sgl-project/sglang-omni?ref=notatechguy.com)[P⁴](https://github.com/sgl-project/sglang-omni/blob/main/docs/index.rst?ref=notatechguy.com). The project's documentation describes serving models through OpenAI-compatible APIs with streaming and realtime capabilities [P⁴](https://github.com/sgl-project/sglang-omni/blob/main/docs/index.rst?ref=notatechguy.com). Under the hood, the runtime breaks generation into coordinated stages: preprocessing, encoders, autoregressive engines, talkers, decoders, vocoders, and aggregators [S¹](https://github.com/sgl-project/sglang-omni?ref=notatechguy.com). It composes with the parent SGLang engine for the autoregressive scheduling and model execution layer [S¹](https://github.com/sgl-project/sglang-omni?ref=notatechguy.com). The parent project is no small thing. SGLang has 38,809 stars and 10,425 forks [P²](https://github.com/sgl-project/sglang/?ref=notatechguy.com). SGLang-Omni extends that engine from text-only LLM serving into voice, where a single user request might pass through a speech recogniser and a language model before reaching a text-to-speech synthesiser that returns an answer. ### Throughput gains lack per-model detail and independent checks The README's most striking performance claim comes from a collaboration with OpenRSI, described as producing 14 merged pull requests across eight speech and omni models with throughput gains between 7.6% and 326.5% on H100 [S¹](https://github.com/sgl-project/sglang-omni?ref=notatechguy.com). Missing from the README are per-model breakdowns, error bars, or benchmark methodology — no benchmark code has been released, and no third party has reproduced these numbers. A second claim targets latency. Voxt 2.0, an open-source voice input tool for Mac that uses SGLang-Omni for local speech recognition on Apple Silicon, reportedly achieves a 92% shorter wait after long dictation than version 1.x [S¹](https://github.com/sgl-project/sglang-omni?ref=notatechguy.com). Voxt is maintained by the same project group, making this a vendor's own figure with no independent measurement [S¹](https://github.com/sgl-project/sglang-omni?ref=notatechguy.com). ### The model lineup and what to check SGLang-Omni v0.2.0, released on PyPI in October 2026 [S¹](https://github.com/sgl-project/sglang-omni?ref=notatechguy.com)[P³](https://github.com/sgl-project/sglang-omni/blob/main/docs/get%5Fstarted/installation.md?ref=notatechguy.com), ships with day-0 support for a growing list of speech and audio models. Fish Audio Drama 3 brings multi-speaker text-to-speech with voice cloning and natural-language control [S¹](https://github.com/sgl-project/sglang-omni?ref=notatechguy.com). AuK and AuK-Flash, added in September 2026, and MiniMax Music 3, added in August 2026, round out the catalogue [S¹](https://github.com/sgl-project/sglang-omni?ref=notatechguy.com). The runtime also handles native full-duplex models and turn-based multimodal models, as well as cascaded systems coordinated through composable graphs and persistent sessions [S¹](https://github.com/sgl-project/sglang-omni?ref=notatechguy.com). The installation guide confirms the current stable release is v0.2.0 on PyPI, with Docker recommended for Linux deployments on NVIDIA GPUs [P³](https://github.com/sgl-project/sglang-omni/blob/main/docs/get%5Fstarted/installation.md?ref=notatechguy.com). The docs also list support for Apple Silicon, Ascend NPU, Intel XPU, Intel CPU, and Moore Threads MUSA [P³](https://github.com/sgl-project/sglang-omni/blob/main/docs/get%5Fstarted/installation.md?ref=notatechguy.com). For a team building a voice agent today (say a customer service bot that transcribes speech, reasons over it, then responds in synthesised voice), the practical question is whether SGLang-Omni's multi-stage pipeline delivers lower end-to-end latency than stitching together separate inference servers for each component. The README claims it does, through coordinated scheduling across stages. No independent benchmark exists for SGLang-Omni to confirm the claim. A team evaluating the runtime should pull the v0.2.0 package from PyPI, run one of the supported speech models on an H100, then measure end-to-end latency against their current multi-server setup. The 14 OpenRSI pull requests are visible on GitHub and would show which models gained the most. The project's most recent README entries are the November 2026 updates to Voxt 2.0 and the OpenRSI results [S¹](https://github.com/sgl-project/sglang-omni?ref=notatechguy.com), with no published roadmap for what ships next. --- *Sources: [S1 — sgl-project/sglang-omni: High Performance Serving Runtime For Interact](https://github.com/sgl-project/sglang-omni?ref=notatechguy.com) · [P2 — sgl-project/sglang](https://github.com/sgl-project/sglang/?ref=notatechguy.com) · [P3 — docs/get\_started/installation.md](https://github.com/sgl-project/sglang-omni/blob/main/docs/get%5Fstarted/installation.md?ref=notatechguy.com) · [P4 — docs/index.rst](https://github.com/sgl-project/sglang-omni/blob/main/docs/index.rst?ref=notatechguy.com) · [P5 — huggingface/transformers](https://github.com/huggingface/transformers/?ref=notatechguy.com)* --- *Written from 5 sourced items, 4 of them primary.* ## More from Not A Tech Guy - [Appwrite hits 58,000 GitHub stars with agent-first rebrand](https://www.notatechguy.com/appwrite-hits-58-000-github-stars-with-agent-first-rebrand/) - [nvm trending on GitHub: 96,000 stars at age 16](https://www.notatechguy.com/nvm-trending-on-github-96-000-stars-at-age-16/) - [AI agents fail at geography when the world keeps changing](https://www.notatechguy.com/ai-agents-fail-at-geography-when-the-world-keeps-changing/)