A research team has built a prompt-adaptive router that cuts large language model inference costs by roughly 36% at selected operating points while still outperforming rival ensemble methods on accuracy S¹. The system, called PISA-E, adjusts its cost-quality tradeoff at inference time, the moment a query arrives, without retraining. That challenges the fixed-ensemble approach where every query hits the same set of models regardless of difficulty S¹.

My read: The number I would watch is the 2.87 percentage-point quality gain under out-of-distribution conditions, because OOD is where routers usually fall apart. But I don't buy the 36% cost reduction as a general claim; the paper is careful to say it applies only at "selected lower-cost operating points," not across the board. All metrics are author-reported, the paper is not peer-reviewed, and no statistical significance testing is mentioned. Until someone independent reproduces these numbers on benchmarks outside the four tested, the gains are promising but unconfirmed.

The problem PISA-E attacks

Every query sent to a frontier model like GPT-4 or Claude costs more than the same query sent to a smaller model. The gap can be tenfold or more. Routers try to close that gap by sending easy questions to cheap models and hard ones to expensive models. Amazon already ships this in Bedrock, where Intelligent Prompt Routing directs queries between model variants to balance cost and latency P². Open-source alternatives exist too: the a3m-router project on GitHub covers 47-plus providers and claims 96.77% accuracy at $0.0768 per 1,000 queries P⁵.

But single-model routers face a ceiling. Some questions are hard enough that no single model handles them well, and the fix, ensembling, means running multiple models and combining their answers. That drives cost up.

The tension between spending more for better answers and saving money with a single model is what PISA-E tries to dissolve.

How the router works

PISA, the base system, looks at each incoming prompt and estimates how well each available model would handle it. It combines soft cluster memberships, a way of saying each prompt belongs partially to several groups rather than fitting neatly into one bucket, with model-specific predictors that score individual model capabilities S¹. The output is a per-prompt routing decision: send this query to model A, that one to model B.

PISA-E extends this to ensembles. Instead of picking one model, it jointly selects which models to include in the ensemble and how many to use, what the paper calls cardinality S¹. A tractable ensemble-success surrogate, a mathematical shortcut for estimating whether a group of models will collectively succeed, makes this computationally feasible without training specifically on ensemble configurations S¹.

The cost-quality preference is a slider. Dial it toward quality and PISA-E adds more models; dial it toward cost and it trims the ensemble. The adjustment happens at inference time, no retraining required S¹.

What the numbers say

Across four LLM routing benchmarks, PISA topped every other single-model router on macro-average cost-performance hypervolume, a measure of how well a router balances cost against quality across operating points, under both in-distribution and out-of-distribution conditions S¹.

PISA-E's ensemble results are where the gains get larger. When the system prioritises response quality, it beats the strongest ensemble competitors by 2.09 percentage points under in-distribution conditions and 2.87 points under out-of-distribution S¹. At selected lower-cost operating points, it still exceeds those competitors' accuracy while cutting average inference cost by approximately 36% in-distribution and 25% out-of-distribution S¹.

PISA-E vs strongest competitors: quality gains and cost cuts

The paper does not name the "strongest competitors" in its abstract, nor does it report error bars or significance tests for the percentage-point gains. The four benchmarks cover the scenarios the authors chose, not necessarily the full range of real-world routing tasks. As we found when LLM pipelines lose 40 accuracy points at interfaces, benchmark performance and production performance can diverge sharply at the seams where models hand off.

Who would use it first

A customer support platform running thousands of queries per hour through a mix of GPT-4o and a smaller open-weight model is the natural first user. Today, most such platforms either send everything to the expensive model or use a fixed routing rule. PISA-E would let them set a cost ceiling per query and let the router decide, prompt by prompt, whether one model suffices or an ensemble is worth the extra spend.

The inference-time slider matters here. A platform could run cheap during off-peak hours and dial up quality during a product launch without touching the router's training.

The broader field is moving fast. A February 2025 survey by Zhijun Chen and colleagues at Beihang University catalogued the growing arsenal of LLM ensemble techniques, from output-level voting to representation-level fusion P⁴. Cost-aware routing is the practical edge of that work, and as we noted when AI scientific discovery costs were cut up to 80% with SpecEvo, the difference between a method that works in a paper and one that works in production usually comes down to whether the cost savings survive contact with real traffic patterns.

The paper is an arXiv preprint, not peer-reviewed, and no code release is confirmed for the LLM routing version. The GitHub repository under the vision-x-nyu organisation that shares the PISA name covers a different project on video diffusion models P³. Until the authors release code or an independent group reproduces the benchmarks, the 36% cost cut and the 2.87-point quality gain remain author-reported.

Sources: S1 — Ensemble Routing for Prompt-Adaptive Cost-Aware LLM Inference · P2 — Use Amazon Bedrock Intelligent Prompt Routing for cost and latency ben · P3 — vision-x-nyu/pisa-experiments · P4 — Harnessing Multiple Large Language Models: A Survey on LLM Ensemble · P5 — Das-rebel/a3m-router

Written from 5 sourced items, 4 of them primary.