Nvidia's 5x Cost Advantage Over AMD Emerges in Real Coding-Agent Workloads

SemiAnalysis released AgentX on August 24, an open-source benchmark that does something most chip comparisons do not: it replays actual conversations between coding agents and their tools, rather than running fixed sequences of text through a model. The result upended the usual story about Nvidia and AMD's competitive position. On one standard configuration—GLM 5.3 served through SGLang—Nvidia's B200 chip achieved roughly five times better cost efficiency than AMD's MI355X. The gap is real enough that SemiAnalysis made a provocative claim: even if AMD hardware arrived free of charge, the total cost per token, once you account for hosting and power consumption, would still favor Nvidia.

But the headline number masks something more interesting. The five-fold advantage holds at one specific operating point: 150 output tokens per second per user. Shift the workload slightly—to 108 tokens per second—and the gap narrows to 57 percent. Push to 141 tokens per second and it widens to 247 percent. The variance matters because it reveals where the real competition lives. It is not in the silicon itself. It is in the software that sits on top of it.

To build AgentX, SemiAnalysis intercepted real traffic from Claude Code and Codex requests made by its own staff, capturing more than 8,000 sessions and 610 billion tokens. The public benchmark subset contains 393 Claude Code sessions, anonymized and transformed into blocks of 64 tokens each to preserve the structure of how caches reuse information without exposing the original code. The median session sent 142,000 tokens as input and received 444 tokens back, with 3.84 seconds of idle time between turns while the agent waited for a tool to respond. This is nothing like the standard benchmark—8,000 tokens in, 1,000 tokens out, all in one shot. It is nothing like how these systems actually run in production.

The cost advantage emerges from three layers of software engineering. First is cache behavior. When an agent returns for a second turn in a conversation, the key-value state from the first turn should still be sitting in the chip's high-bandwidth memory, ready to reuse. On DeepSeek V4, Nvidia's B300 configuration achieved a 91 percent cache hit rate at 384 concurrent sessions, with another 1.36 percent coming from host memory. AMD's MI355X, running at higher concurrency (196 sessions), fell to 73 percent hits and had to offload 20 percent to host memory—a much slower tier. The latency penalty compounds as user count scales. Second is tokenization. Nvidia's TensorRT-LLM added incremental tokenization that skips re-processing tokens from earlier turns in a conversation. On a Qwen 3.5 trace, this cut mean processing time per turn from 185 milliseconds to 11 milliseconds. A fixed-length benchmark would never see this saving because there is no prior turn to reuse. Third is session routing: whether the serving stack pins a conversation back to the same worker that holds its cached state, or load-balances and forces a costly re-prefill.

AMD is not standing still. Its ATOM engine beats Nvidia on price-performance in specific regions of the performance curve, particularly on Alibaba's advertising workloads. But ATOM has minimal production adoption outside that one customer. The main Qwen organization does not use it. When you compare mainstream open-source serving stacks—vLLM and SGLang—on both sides, the picture separates sharply. On Qwen 3.5 through SGLang, Nvidia runs more than 20 times ahead at 90 tokens per second per user. SemiAnalysis also notes that vLLM's feature matrix does not list AMD backends as supported for context-parallel inference, the parallelism strategy that long-context workloads rely on most.

What makes this benchmark worth reading is that it surfaces a deeper shift in the competitive landscape. Nvidia's advantage is no longer primarily about the chip. It is about the software stack—the kernels, cache management, routing logic, and scheduling algorithms that move faster than a fabrication cycle. An August 21 upstream commit from Inferact and Nvidia changed the rankings in a single week. That volatility is a warning: any single benchmark result is provisional. SemiAnalysis has already scheduled monthly updates.

The benchmark also carries important caveats. It uses synthetic tokens at replay, which distorts speculative decoding acceptance rates. The traces come from coding workloads with heavy context, so a different application might produce a different input distribution. And it runs closed-loop, meaning faster configurations complete more requests and encounter a slightly different workload mix. Google's TPUs are absent from the comparison. Nvidia's Rubin generation and AMD's MI455X are both due later in 2026.

For buyers, the lesson is not to accept a tokens-per-second figure without asking harder questions. What is the prefix cache hit rate at production concurrency? How much host memory backs each accelerator? Does the serving stack pin sessions to the worker holding their cached state, or does it load-balance and force re-prefills? These are the questions that separate a cheap inference platform from one that actually works at scale. For AMD, the path forward runs through software, not a faster chip. If it upstreams more of ATOM's gains into vLLM and SGLang, and matures its long-context parallelism and offload paths, much of the current gap is addressable. The benchmark establishes nothing about how quickly that would happen.