FIG_GPU · An interactive simulation

Inside an LLM Serving GPU

Decode reads every weight for every token. Watch the bytes leave the memory chips, and see why memory bandwidth sets the speed. L4 or GB10. BF16 or FP8. Fill the pool. Break the cooling.

Start over

Try to break it

Each one sets the switches and runs one request again. Every number comes from a measured run.

Hand-drawn wobble Board Labels

1. A question comes in through the cable.ready

The big ideaMemory, not math, sets the speed of decode, and the KV pool sets how many users fit.

NowReady

KV pool0

Requests0

Tokens per step0

Weight read0

Time per token0

First token0

Output0

Clock0

Machine

Workload

Outcome served

This is a simulation. The page does not run a model in your browser. The pool sizes, weight sizes, chunk sizes, step times, first-token times and throughputs are medians of three seeds from the SGLang 101 runs on one NVIDIA L4 and one NVIDIA GB10. The admission count and the weight-read share are arithmetic on those numbers and the published bandwidth. Where a setting was not measured, the page says so. The motion is paced for reading, not to scale.

Download poster

NVIDIA L4

Die: Ada Lovelace, AD104NVIDIA Ada GPU architecture whitepaper, Appendix D, Table 5
SMs: 58 5 GPCs, 29 TPCsNVIDIA Ada GPU architecture whitepaper, Appendix D, Table 5
Tensor Cores: 232 4 per SM, 4th generation, with FP8NVIDIA Ada GPU architecture whitepaper, Appendix D, Table 5
L2 cache: 48 MBNVIDIA Ada GPU architecture whitepaper, Appendix D, Table 5: 49152 KB
Memory: 24 GB GDDR6 with ECCNVIDIA L4 product brief PB-11316-001, Table 2
Bus: 192 bits, 32-bit controllersNVIDIA L4 product brief PB-11316-001, Table 2
Bandwidth: 300 GB/s peakNVIDIA L4 product brief PB-11316-001, Table 2
Host link: PCIe Gen4 x16NVIDIA L4 product brief PB-11316-001, Table 1
Power: 72 W, passive coolingNVIDIA L4 product brief PB-11316-001, Table 1
Clocks: 795 base, 2,040 boost MHzNVIDIA L4 product brief PB-11316-001, Table 1

NVIDIA does not publish how many GDDR6 packages carry the 24 GB, or where they sit. The drawing shows one package for each 32-bit channel, six in all.
NVIDIA does not publish the die size, the package size or the ball count. The die, the substrate and the balls are drawn to a plausible size, not to measure.
NVIDIA publishes 58 SMs in 5 GPCs and 29 TPCs, not which TPC of the full AD104 is switched off. The dark pair of tiles shows that one TPC is off, not where it is.
The L2 band in the middle of the die follows the AD102 block diagram in the Ada whitepaper. NVIDIA does not publish a floorplan of AD104.
The host CPU model was not recorded with the runs. The host CPU is drawn as a generic x86_64 package.

NVIDIA GB10

Chip: Grace Blackwell superchip: an S-dielet with the CPU and memory subsystem, a G-dielet with the GPU, in 2.5D packagingNVIDIA GB10 talk, Hot Chips 2025, NVIDIA GB10 table
CPU: 20 Arm cores: 10 Cortex-X925 and 10 Cortex-A725, two clusters of 10NVIDIA DGX Spark product page, Specifications
GPU: 6144 CUDA cores, 5th generation Tensor Cores; SM count not publishedNVIDIA DGX Spark user guide, Compute Performance
GPU L2: 24 MBNVIDIA GB10 talk, Hot Chips 2025, GB10 GPU: large graphics L2
CPU caches: 16 per cluster MB L3, plus a 16 MB system level cacheNVIDIA GB10 talk, Hot Chips 2025, NVIDIA GB10 table
Memory: 128 GB LPDDR5x, coherent unified system memoryNVIDIA DGX Spark product page, Specifications
Bus: 256 bits, 16 channelsNVIDIA DGX Spark product page, Specifications
Bandwidth: 273 GB/s peakNVIDIA DGX Spark product page, Specifications
Chip link: NVLink-C2C between the dielets about 600 aggregate GB/s for the GPUNVIDIA GB10 talk, Hot Chips 2025, NVIDIA GB10 table
Power: 140 W for the chipNVIDIA DGX Spark product page, GB10 TDP, CPU and GPU

NVIDIA gives 6,144 CUDA cores for GB10 and does not publish the SM count. The GPU dielet shows an SM field of abstract tiles, not one tile per SM.
NVIDIA gives 16 LPDDR5X channels on a 256-bit bus and does not publish the package count or layout. The drawing shows eight packages of two channels each.
The dielet sizes, the substrate and the ball count are not published. They are drawn to a plausible size, not to measure.
The board is drawn as a plain plate. The DGX Spark board layout is not published.

Hardware sources: the NVIDIA L4 product page and product brief, the NVIDIA Ada GPU architecture whitepaper, the NVIDIA DGX Spark product page and user guide, and the NVIDIA GB10 talk at Hot Chips 2025.

What the machine shows

Decode is a memory read. On the L4, one decode step of Qwen3-8B in BF16 reads 16.41 GB of weights. At the peak 300 GB/s that takes 54.7 ms, and the measured step at one user took 60.16 ms. The SMs wait for the bytes.

Fewer bytes, faster steps. FP8 weights are 8.81 GB on the L4. The step at one user fell to 39.66 ms, and the freed memory grew the KV pool from 24,094 to 142,369 tokens. At 64 users the output went from 250.85 to 735.91 tok/s.

The pool decides who runs. The scheduler admits a request only when the pool has room. At 64 users the BF16 pool holds 37 requests and the rest wait. At a memory fraction of 0.74, four 2,048-token prefixes no longer fit, and the median first token took 12.6 s, against 0.47 s at 1,024 shared tokens.

Launch gaps are small. Without CUDA graphs, each kernel launch crosses from the host. On the L4 that added 0.47 ms to a 60 ms step at one user. Next to the weight read, the launch path barely matters.

Measure it on your own GPU.

Read the SGLang 101 manual Production quantization