A paper posted to arXiv on August 6, drawing on a production study at Microsoft Azure, finds that agentic AI workflows bottleneck on the CPU, not the GPU S¹. The finding cuts against the grain of an industry that has spent billions stockpiling graphics processors. If the bottleneck has shifted, the entire playbook for building AI infrastructure may need to follow.

My read: This is the first architectural study I've seen that takes agentic workflows seriously as a datacenter workload class, rather than just a software pattern. I'm skeptical of the "first" claim until peer review confirms it, but the core observation rings true: agents run a model, but they also orchestrate, call tools, and hop between CPU and GPU constantly. The GPU shortage narrative has dominated for two years. This paper suggests that for agentic workloads, the constraint is different. I'd watch whether cloud providers start talking about CPU provisioning for AI workloads, alongside GPU clusters.

Why the CPU is back on the critical path

The paper, by Jirong Yang, Peizhe Liu, Chaojie Zhang, and Jovan Stojkovic, presents what the authors describe as the first architectural characterisation of agentic workflows S¹. They combine a production study at Microsoft Azure with a controlled study of open-source frameworks S¹.

Their central finding: agentic execution is fragmented and heterogeneous S¹. When an agent gets a request, it breaks the task down into multiple model calls, tool uses, and coordination steps that constantly jump back and forth between the CPU and GPU S¹. The model runs on the GPU, but the orchestration logic and tool calls run on the host CPU. That puts the CPU on the critical path S¹, the chain of operations that determines total response time.

This is a reversal of the standard AI infrastructure story. For training and simple inference, the GPU is the bottleneck and the CPU is an afterthought. For agents, the CPU is the conductor. Every tool call, every planning step, every coordination decision between agents runs there. The GPU waits while the CPU works.

Why agents leave GPUs idle

The paper identifies two structural problems that follow from this fragmentation.

First, the way agents operate creates a workload that is mostly quiet but experiences sharp, unpredictable bursts S¹. Agents spend most of their time in orchestration and tool calls, then briefly hit the GPU for inference. This bursty demand means GPUs sit idle for large stretches.

Second, the mix of models used in a workflow affects how balanced the GPU usage is, and having a wider variety of tasks and tools makes that imbalance even worse S¹. Different agents in a pipeline may use different models, different tools, and different orchestration patterns, creating a workload that no single server configuration serves well.

Because of this, the scattered nature of the work leaves both CPU and GPU resources underutilized even when demand spikes S¹. The hardware is neither fully idle nor fully used. Since different software components need different things, giving every machine the same CPU setup is wasteful S¹. Also, cramming multiple agents onto the same processor cores hurts cache locality S¹, meaning the processor loses the speed advantage it gets when data and instructions are stored nearby.

Agora: a prototype that rethinks the server

The researchers built Agora, a prototype system for commodity servers, to test whether these problems are fixable S¹. Agora makes four moves:

  • It grabs unused CPU cores to run background throughput tasks, while making sure that sudden tool spikes don't ruin the tail latency (the slowest responses) for active agents S¹. When an agent waits on a tool call, Agora loans its idle CPU capacity to other workloads, then reclaims it when the agent needs it.
  • It packs more agents onto a single GPU by oversubscribing the memory, and it prefetches the next agent's data to make the switching process seamless S¹. More agents per GPU means better utilisation, and prefetching means the swap between agents is invisible.
  • It groups CPU cores based on their function and uses smart scheduling to bring back the cache locality that gets lost when multiple agents share the same cores S¹.
  • It automatically tunes these mechanisms to the workload S¹.

The authors report that Agora improves utilisation and server throughput while preserving agent tail latency S¹. They do not provide specific numbers in the abstract, and the results have not been independently verified.

What to do about it

Consider a startup running a multi-agent customer support pipeline: one agent classifies the ticket, another pulls order history from a database, a third drafts a response, and a human reviewer approves it. The company has been budgeting for more GPUs because response times are creeping up. But this paper suggests the bottleneck may be on the CPU side, where orchestration and tool calls happen, rather than on the GPU side where inference runs.

Before buying more graphics cards, audit the CPU-GPU split in your agent pipeline. Measure how much time each request spends in GPU inference versus CPU orchestration and tool calls. If the CPU share is high and the pattern is bursty, the fix may be better CPU provisioning and scheduling, not more GPU capacity. The paper's findings imply that uniform server configurations, where every machine has the same CPU-to-GPU ratio, are inefficient for agentic workloads. Different agent roles need different resource profiles.

The performance gains of multi-agent systems are real, but this paper shows they come with an infrastructure cost the industry has not yet reckoned with.

What we don't know yet

The paper is an arXiv preprint and may not have undergone peer review S¹. The "first architectural characterisation" claim is self-assessed by the authors. The abstract contains directional performance claims for Agora but offers no specific numbers, benchmarks, or comparisons S¹. The Azure production study and the Agora prototype are separate components of the paper; Agora is not deployed in Azure, and there is no indication it is available as open source S¹.

The next signal: a revised arXiv version or conference submission with the full Agora benchmarks. We'll check the numbers against it. Subscribe for the follow-up when those results land.

Sources: S1 — Architectural Implications of Agentic AI Workflows · P2 — Architectural Implications of Agentic AI Workflows · P3 — InternLM/archspace · P4 — Architectural Implications of Agentic AI Workflows · P5 — FoundationAgents/AFlow

More from Not A Tech Guy

Generated from an audited evidence pack with primary-source research. Social-media items are discussion signals, not verified facts. Nothing here is financial, legal or medical advice.