GPU vs TPU vs LPU: Julia Turc on the Von Neumann Bottleneck in AI Chips

Julia Turc explains how NVIDIA GPUs, Google TPUs and Groq LPUs each attack the von Neumann bottleneck, the 1945 gap between memory and processor. In a 30-minute video essay, she traces the first TPU from a Google speech model that could not be deployed, through H.T. Kung’s 1982 systolic arrays, to Groq’s all-SRAM chip and NVIDIA’s $20 billion deal to license it. Interview clips with Groq founder Jonathan Ross, now at NVIDIA, and Google Cloud developer advocate Abdel Sghiouar fill in how the designers think about the trade-offs.

TLDW

Every AI chip is trying to keep its arithmetic units busy while weights trickle in from memory. Google’s TPU answered with a weight-stationary systolic array that reuses each weight across a batch, and threw out nearly all of a CPU’s predictive machinery. GPUs and TPUs have since converged on the same high bandwidth memory, so both still need big batches. Groq removed external memory altogether, kept weights in on-chip SRAM across hundreds of chips, and scheduled everything ahead of time. The answer NVIDIA now sells is both: GPUs for prefill and attention, Groq LPUs for the decode-heavy feed-forward layers.

Thoughts

NVIDIA did not beat the specialist inference chip; it turned it into a component of its own system. LLM inference splits into prefill, which processes the prompt in parallel and suits a GPU, and decode, which produces one token at a time and is starved by memory. Groq’s LPU is fast at decode, but the KV cache grows into gigabytes per user, far beyond its SRAM, so attention stays on the GPU and only the feed-forward projections move to the LPU. Ross’s 18-wheeler and delivery van analogy makes the split intuitive, and NVIDIA’s LPX rack of 256 Groq chips beside a Rubin rack makes it a product. The lesson for anyone betting on a single “GPU killer” is that inference is breaking into stages, and whoever owns the system that routes work between stages captures the value. That is a much better position for NVIDIA than winning a benchmark.

Groq’s deeper idea is determinism, not SRAM. Because an SRAM read takes the same number of cycles every time, Groq’s compiler can plan every instruction, memory access and chip-to-chip message down to the clock cycle, with no polling and no waiting on flags. Google’s XLA compiler can only fix the order of instructions, since DRAM reads take an unpredictable time. The payoff shows up in a 2022 Groq paper that predicted a BERT model’s latency within 1% before running it. The most interesting twist comes later, when Ross says Groq now uses AI to write code for the LPU and gets very good results because the hardware is “easy to reason about.” Hardware built to be legible to a compiler turns out to be legible to an LLM too, and that may matter more over the next few years than raw memory bandwidth. Chips that are hard to program have always paid a tax in software; if models write the kernels, the simplest machine to reason about gets the best code.

The most sobering line in the video is that, a decade in, high bandwidth memory is still a von Neumann bottleneck for both TPUs and GPUs. HBM is made by third parties, so a TPU and a GPU from the same year end up with roughly the same bandwidth, and both still need large batches to stay busy: DeepMind served a quantized Llama 3 70B on a TPU v5e and needed a batch of 120. Pods of 9,000 TPUs and NVLink racks of 72 GPUs scale capacity, but each device still waits on its own memory. For users this has a concrete meaning. Low cost per token comes from batching, and batching means your query waits for others to arrive. The trade-off between cheap and fast that Google faced with TPU v1’s batch of 250 has never gone away. It just moved up the stack.

The origin story is a reminder that custom silicon starts with deployment costs, not research ambition. Google’s 2013 speech model cut errors by about 20%, yet it shipped only on Nexus phones because there was not enough compute to serve everyone; three minutes of voice search a day per user would have meant doubling Google’s data centers. Jeff Dean’s analysis pointed to an ASIC, and the resulting TPU spent 2% of its floor plan on control logic, against a GPU die filled with caches, schedulers and register files. It ran 15 times faster than NVIDIA’s K80 at less than half the power. The same economics drive today’s chip programs at Amazon, Meta and OpenAI: when inference is the bill, cutting general-purpose machinery is the obvious move.

NVIDIA’s public answer to TPUs is that it built “accelerated computing,” used for molecular dynamics, fluid dynamics, data processing and also AI. Generality is both its moat and its tax, the same machinery the TPU removed. The video’s own numbers show why specialists struggle anyway: a modest Llama 2 70B needed 576 LPUs at $15,000 to $20,000 each, roughly $9 million to $12 million of silicon mostly holding weights rather than multiplying. That is why Ross’s “baby turtle problem” rings true. Designing a chip is getting easy and taking one to production is not, so most of the new AI chip startups will not make it. The ones that do, like Groq, may end up inside a larger company’s system rather than replacing it.

Key Takeaways

NVIDIA takes about 80 cents of every dollar spent on AI chips, while Google, Amazon, Meta and OpenAI build their own and startups raise billions.
Google started the TPU because a 2013 speech model could not be served at scale. Three minutes of voice search a day per user would have doubled its data centers on CPUs.
The first TPU was a 256 by 256 grid of 8-bit multiply-accumulate units, peaking at 92 trillion operations per second, about 35 times Google’s server CPUs.
Its DRAM link carried 34 billion bytes per second. A naive design fetching fresh weights for every MAC each cycle would need 46 trillion, leaving the chip idle 99.9% of the time.
John von Neumann’s 1945 design separated processor and memory with a wire between them. John Backus named the resulting bottleneck in his 1977 Turing Award lecture.
H.T. Kung’s 1982 systolic array passes each piece of data through many processing elements per memory fetch. TPUs use a weight-stationary 2D version, cutting weight traffic by the batch size.
TPU v1 needed a batch of 250 to run at peak, the classic trade of throughput against latency.
Control logic took 2% of the TPU floor plan. The chip had about a dozen instructions against roughly a thousand in x86, and was 15 times faster than the K80 at under half the power.
TPU memory bandwidth rose 200 times from v1 to v8, mostly from the switch to HBM in v2. Because HBM comes from third parties, GPUs and TPUs of the same year have similar bandwidth.
Groq’s LPU keeps all weights in on-chip SRAM, about ten times faster to read than HBM, and ran Llama 2 70B at 240 tokens per second in early 2024, but across 576 chips.
Groq’s compiler schedules every operation to the clock cycle. A 2022 paper predicted a BERT model’s latency within 1% before running it.
Static scheduling still handles mixture-of-experts models: the time slot is fixed, but scatter and gather operations change which expert fills it.
NVIDIA paid $20 billion in December 2025 to license Groq’s technology and hired Ross. At GTC in March 2026 it showed a 256-chip LPX rack paired with Rubin GPUs: attention on GPUs, feed-forward layers on LPUs.

Chapters

00:00 Intro: Should NVIDIA Be Worried?

Turc sets out the stakes: NVIDIA’s dominance of AI chip spending and the wave of in-house and startup chips challenging it. Jonathan Ross, who started Google’s first TPU as a 20% project and then founded Groq, warns that designing chips is becoming easy while production stays hard. NVIDIA’s $20 billion licensing deal with Groq frames the rest of the video.

00:56 The Origin Story of TPUs

In 2013 Google had a 40 million parameter speech model that cut errors by 20% but could only serve Nexus phone users. CPUs spend much of their area on control logic for unpredictable programs, which neural networks barely need. GPUs fit better but were built for 32-bit floating point at a time when inference wanted 8-bit integers.

03:42 The First TPU: An ASIC for Matrix Multiplication

Jeff Dean’s analysis led to an application-specific chip. Google replaced predictive machinery with a 256 by 256 grid of multiply-accumulate units, each combining a fixed weight with a transient activation. A 70 billion parameter model needs about 70 billion such operations per generated token, and the TPU’s peak of 92 trillion operations per second assumed every unit was busy every cycle.

05:21 SRAM vs DRAM

Processor designers trade off memory bandwidth, distance and size. The first TPU had 28 MB of on-chip SRAM, fast but bulky at six transistors per cell, and 8 GB of DDR3 DRAM soldered about a centimeter away. GPUs like the K80 used the same split between on-chip SRAM and external DRAM.

07:21 The Memory Bandwidth Problem

The DRAM link delivered 34 billion bytes per second against a naive demand of 46 trillion. Turc traces the problem back to von Neumann’s 1945 architecture, which separates processor from memory, and to John Backus, the inventor of Fortran, who named the bottleneck when accepting the Turing Award in 1977.

09:14 Systolic Arrays

H.T. Kung’s 1982 paper proposed doing many operations per memory fetch by passing data along a chain of processing elements, named after the heart’s systole. The TPU expands this into a weight-stationary grid: weights load once, staggered activations shift right each cycle, and partial sums flow down. Each weight is reused across the batch, which is why TPU v1 needed a batch of 250 to reach peak.

12:48 TPU vs GPU

The TPU floor plan gives a quarter to the matrix unit and only 2% to control logic, while a K80 die is tiled with streaming multiprocessors full of caches and schedulers. NVIDIA’s Tensor Cores multiply many small tiles under a flexible runtime scheduler. Both lines have pulled the same levers since: HBM, more compute, and pods or NVLink racks, yet four of six early TPU workloads were still memory bound.

17:51 Groq’s LPU

Groq removed external memory, keeping weights in 230 MB of SRAM per chip and spreading a 70B model across 576 chips. Predictable SRAM timing lets the compiler schedule everything in advance, which Groq calls software-defined hardware. Sghiouar explains the cost on TPUs, where shape changes force recompilation and JAX developers pad inputs into fixed buckets. Ross explains how mixture-of-experts models still fit a static schedule, and how layers stream results to the next chip as soon as rows are ready.

25:38 GPU vs LPU

GPUs win on cost per token through large batches; LPUs win on speed but need many chips at $15,000 to $20,000 each. Prefill suits GPUs, decode suits LPUs, and the KV cache keeps attention on the GPU. NVIDIA’s LPX rack, shown at GTC in March 2026, puts that division of labor into hardware.

28:36 NVIDIA on the AI Chips Boom

NVIDIA frames its GPU as general accelerated computing, used for physics and data processing as well as AI, rather than a tensor processing unit. Turc notes the flood of venture-funded accelerators, points to Cerebras going public this year and supplying OpenAI, and says most rivals are far more secretive than Groq.

Notable Quotes

“You’re gonna see a lot of people trying to do chips because it’s gonna be very easy to design one. But it’s very hard to take one to production, and so it’s gonna become the baby turtle problem.”
Jonathan Ross, on the wave of AI chip startups

“Turns out it was very hard, but we didn’t know, so jumped in.”
Jonathan Ross, on starting the first TPU

“Back then, scientists were still romantics.”
Julia Turc, on H.T. Kung naming the systolic array after a beating heart

“So a decade in, the HBM is still a von Neumann bottleneck for both of these chips.”
Julia Turc, on TPUs and GPUs today

“If the hardware that you’re compiling to is easier to reason about, it will produce even better kernels.”
Jonathan Ross, on using AI to program the LPU

“If you want the absolute best cost per token, you’re just gonna use a GPU as it is.”
Jonathan Ross, on the trade-offs between GPUs and LPUs

“And in this analogy, the GPU is the 18-wheeler.”
Jonathan Ross, on pairing GPUs with LPUs

Watch the full video here.