Inference Is Memory-Bound. That Is What Custom Silicon Buys.

Summary

• NVIDIA’s CFO put AI inference at roughly 40% of datacenter revenue in early 2024, and said the share was probably understated [1]. Forecasts put inference at about two-thirds of AI compute by 2026 [2].
• Autoregressive decode re-reads the weights every token. At ~2 FLOP/byte on an A100-class ridge, you sit on the memory wall; below about batch 208, more tokens do not buy more math time [12].
• Production ASICs already bet on that profile: TPU v1 delivered 30–80× TOPS/W versus contemporary CPU/GPU [3]; Groq, Cerebras, Meta MTIA, and Inferentia push on-chip bandwidth or cost per inference the same way.
• Rack power moved from a modal ~9 kW to ~120 kW liquid-cooled racks in production, with ~1 MW on published 2027 roadmaps [27][28][29]. Hyperscalers are signing nuclear PPAs measured in hundreds of MW [4][5].
• The unit of deployment is becoming the rack or pod (TPU v4, Groq fabric, NVL72). Facility, power contract, and topology sit inside the same design as the accelerator.

1. Where the money already is

Training runs make headlines. Serving is the same forward pass, repeated, billed in fractions of a cent—and that is where demand scales with users.

In February 2024, NVIDIA’s CFO estimated that about 40% of datacenter revenue over the prior year was AI inference, and that the figure was probably low [1]. Later forecasts point the same way:

• Deloitte: roughly two-thirds of AI compute from inference in 2026, up from about one-third in 2023 [2].
• Barclays (Dec 2024): over 70% of general AI compute needs from inference by 2026, about 4.5× training [6].
• Brookfield: about 75% of AI compute demand from inference by 2030 [7].

Treat those as forecasts, not measurements. Capex is not waiting for consensus. Microsoft alone guided about $80B of fiscal-2025 spend, mostly AI datacenters [8]. Combined 2025 capex guidance for Microsoft, Alphabet, Amazon, and Meta cleared $320B [9]. NVIDIA’s CEO put one gigawatt of AI datacenter capacity at $50–60B [10].

If spend follows the workload, and the workload is inference, then inference economics decide what gets built.

2. Why inference is a different machine

The cleanest cut is the roofline [11]: attainable performance is the min of peak compute and (memory bandwidth × arithmetic intensity). On a log-log plot you get a slope (bandwidth-bound) and a flat roof (compute-bound).

Large-batch training is a ceiling workload: huge GEMMs, arithmetic intensity in the hundreds of FLOPs/byte. GPUs were built for that.

Autoregressive decode is a wall workload. Each new token re-reads every weight and does about two FLOPs per parameter (one MAC). At 16-bit that is ~2 FLOP/byte—left of any modern ridge point. A standard worked example puts the A100 crossover near batch 208: below that, one token and 208 tokens cost the same wall time because you are paying for the weight read, not the math [12].

The teraFLOPS sit idle. The machine is waiting on HBM.

From that:

1. Bandwidth and capacity set throughput. Weights are re-read per token; KV cache grows with context and batch. Serving often hits memory capacity before compute—PagedAttention’s up-to-24× throughput gain over a stock HuggingFace path is a memory-management result [13], and Google’s scaling paper treats partition, multiquery attention, and layout as first-order levers [14].
2. Latency SLOs cap utilization. Under a 7 ms p99 bound, TPU v1 ran at 80% of peak throughput; the contemporary CPU and GPU managed 42% and 37% [3]. Predictable execution buys real utilization.
3. Buyers purchase tokens/$/W at an SLO, not inference FLOPS. Batching, memory hierarchy, and topology all fold into that number.

So if a serving slide leads with peak teraFLOPS, treat that as a suspicious coincidence. Ask what batch size and what latency bound sat under the measurement. You may be looking at a marketing ceiling while the box was bandwidth-bound.

A general GPU still pays for caches, dynamic scheduling, and graphics heritage this workload does not use. An inference ASIC spends the same transistors and watts on low-precision matrix throughput and bytes per watt.

3. What production silicon already did

The pattern is consistent across a decade of shipped parts.

Google TPU v1 (2015). Built in 15 months after Google projected inference would otherwise double datacenter footprint. 65,536-MAC systolic array, 28 MiB software-managed on-chip memory, 92 INT8 TOPS, 75 W TDP [3][15]. Versus Haswell/K80: 15–30× inference performance, 30–80× TOPS/W [3]. Ridge point ~1,350 ops/byte; four of six production workloads were still memory-bound on the TPU itself [3].

Google TPU v4 (2020/2023). 2.7× better perf/W than v3; 1.2–1.7× faster than contemporary A100 systems at comparable size while using 1.3–1.9× less power. Optical circuit switches (136-port) under 5% of system capex and 3% of power [16]. Reconfigurable fabric is a facility decision, not a chip feature.

Groq TSP (2020). No external DRAM: 220 MiB on-chip SRAM, 55 TiB/s aggregate bandwidth, fully deterministic schedule. Batch-1 ResNet-50 at 20,400 img/s, under 49 µs/image—about 4–5× the GPU/TPU systems in the paper [17]. Follow-on: 10,440 chips, >2 TB distributed SRAM, under 3 µs end-to-end [18]. Determinism is how you run near peak under an SLO: no jitter tax.

Cerebras WSE-3 (2024). 44 GB on-chip SRAM, 21 PB/s on one 46,225 mm² device [19]. Vendor claim: Llama 3.1 70B at 2,100 tok/s with full 16-bit weights [20]. Treat the number as a vendor claim; the premise (decode tracks on-chip bandwidth) is what the roofline predicts.

Meta MTIA (2023/2024). Recommendation inference: 102.4 INT8 TOPS at 25 W, co-designed with PyTorch [21]. Gen-2: 3× on production ranking models, 1.5× platform perf/W, 72 accelerators/rack [22]. Oversized on-chip SRAM relative to GPUs, aimed at small-batch ranking.

AWS Inferentia / Trainium. Inferentia1: up to 70% lower cost per inference vs comparable GPU instances (vendor) [23]; Inferentia2: 32 GB HBM/chip. Trainium2 instances: 46 TB/s aggregate HBM, claimed 30–40% better price-performance than comparable GPUs [24]. Custom silicon at cloud scale is a margin bet on cost per token—at inference volume.

The last row is not an ASIC. NVIDIA spent two generations turning the flagship GPU into an inference machine: FP4 with per-tensor scaling, 8 TB/s HBM per GPU, 72-GPU NVLink domain sold as one liquid-cooled rack that “acts as a single GPU” [25][26]. GPU and inference ASIC are converging from both sides. The direction of the bet—bandwidth, density, rack as machine—is the same.

4. What that does to the building

If this only made tokens cheaper, it would be a procurement story. The numbers say it is a facilities story.

Power density reprices the hall. Modal rack density in surveys is still under 10 kW [27]. Legacy air-cooled design sat near ~20 kW/rack. One GB200 NVL72—NVIDIA’s OCP contribution—is a ~120 kW, 1,400 A, direct-liquid-cooled rack [28]. NVIDIA/Vertiv roadmap: 800 V DC distribution and racks near 1 MW by 2027 [29]. Cooling, busbars, floor load, and clearances all move. Better perf/W still concentrates remaining watts into fewer, hotter racks.

The grid binds; silicon moves where power is. Datacenters used about 415 TWh in 2024; IEA projects ~945 TWh by 2030, with AI-optimized sites more than quadrupling draw [30]. US datacenters: 4.4% of electricity in 2023, projected 6.7–12% by 2028 [31]. North America primary-market vacancy: 1.6% in H1 2025; CBRE names power as the supply limit [32]. Hence the nuclear deals: Microsoft’s 20-year, 835 MW restart of Three Mile Island Unit 1 [4]; Amazon’s $650M, 960 MW campus next to Susquehanna [5]; Meta at Clinton (1,121 MW). Double perf/W and operators buy more compute under the same envelope. Efficiency changes site economics; it does not remove the need for megawatts.

Rack and network are part of the accelerator. TPU v4 pods: 4,096 chips as one machine, optical switches reconfiguring topology in software [16]. Groq: compile-time packet schedule across 10,440 chips [18]. NVL72: 72 GPUs, 130 TB/s aggregate NVLink [25]. You deploy a rack- or pod-scale machine with its own power, cooling, and topology. Homogeneous ASIC fleets amplify that: tune the building to one family, give up hosting arbitrary iron.

Utilization looks like an SLO problem. Training: few huge planned jobs. Inference: continuous, latency-bound, diurnal traffic. Capacity planning asks how much hardware holds p99 at peak, and how close to peak you run on average. Deterministic ASICs tighten that headroom; the 37% vs 80% utilization gap under SLO in the TPU v1 paper [3] is a capex fact at fleet scale.

5. What the evidence actually supports

Two limits.

GPUs do not vanish. Training stays GPU-friendly; architectures keep moving; a general device amortizes that risk. An ASIC program assumes a workload class stable for the 2–4 years from tape-out to depreciation—TPU v1’s 15-month schedule [3] is the exception that shows how hard the timing is. Software lock-in is real; Meta’s answer was co-design with PyTorch from day one [21][22].

Inference-share forecasts can be wrong in degree. Epoch AI argues labs may keep training and inference spend near parity because the two partially substitute: a better-trained model is cheaper to serve [33]. Even then, absolute inference volume still grows, and section 4 tracks volume, not the exact share.

What holds without the forecasts: inference is bandwidth-bound and SLO-constrained; every hyperscaler that can field a silicon team already ships against that; the fleets are dense enough that facility, power procurement, and network are co-designed with the chip. That is enough to change how you build and run a datacenter.

References

1. NVIDIA Q4 FY2024 earnings call takeaways (Investopedia, Feb 2024). CFO Colette Kress: “approximately 40% of data center revenue was for AI inference.”
2. Deloitte, TMT Predictions 2026: “More compute for AI, not less” (Nov 2025).
3. Jouppi et al., “In-Datacenter Performance Analysis of a Tensor Processing Unit,” ISCA 2017.
4. Constellation, Crane Clean Energy Center press release (Sep 2024).
5. American Nuclear Society, “Amazon buys nuclear-powered data center from Talen” (Mar 2024).
6. CEPA, “The Golden Age of Customized AI Chips” (Feb 2025), citing Barclays (Dec 2024).
7. Brookfield, “Building the Backbone of AI” (2025).
8. Brad Smith, “The Golden Opportunity for American AI” (Microsoft, Jan 2025).
9. Reuters Breakingviews aggregate of 2025 hyperscaler capex guidance (Microsoft ~$80B, Amazon ~$100B, Alphabet ~$75B, Meta $60-65B), widely reported Feb 2025.
10. Ars Technica (Sep 2025), citing NVIDIA Q2 FY2026 earnings call.
11. Williams, Waterman, Patterson, “Roofline: An Insightful Visual Performance Model for Multicore Architectures,” CACM 52(4), 2009.
12. C. Chen, “Transformer Inference Arithmetic” (2022).
13. Kwon et al., “Efficient Memory Management for Large Language Model Serving with PagedAttention,” SOSP 2023.
14. Pope et al., “Efficiently Scaling Transformer Inference,” MLSys 2023.
15. Jouppi et al., “Ten Lessons From Three Generations Shaped Google’s TPUv4i,” ISCA 2021.
16. Jouppi et al., “TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings,” ISCA 2023.
17. Abts et al., “Think Fast: A Tensor Streaming Processor (TSP) for Accelerating Deep Learning Workloads,” ISCA 2020.
18. Abts et al., “A Software-Defined Tensor Streaming Multiprocessor for Large-Scale Machine Learning,” ISCA 2022.
19. Cerebras WSE-3 datasheet (2024).
20. Cerebras, “Cerebras Inference now 3x faster” (2024). Vendor claim.
21. Firoozshahian et al., “MTIA: First Generation Silicon Targeting Meta’s Recommendation Systems,” ISCA 2023.
22. Meta, “Our next-generation Meta Training and Inference Accelerator” (Apr 2024).
23. AWS Inferentia product page.
24. AWS EC2 Trn2 instance page.
25. NVIDIA GB200 NVL72 product page.
26. NVIDIA Blackwell platform announcement (Mar 2024).
27. Uptime Institute, Global Data Center Survey 2025 (modal rack density under 10 kW).
28. eeNews Europe, “NVIDIA offers 120 kW liquid-cooled Blackwell rack as industry standard” (OCP contribution).
29. DataCenterDynamics, “Nvidia prepares data center industry for 1 MW racks and 800-volt DC power architectures” (2025).
30. IEA, “Energy and AI” special report (Apr 2025).
31. Shehabi et al., 2024 United States Data Center Energy Usage Report, LBNL (Dec 2024).
32. CBRE, North America Data Center Trends H1 2025.
33. Epoch AI, “Optimally allocating compute between inference and training”.

← All posts