Hardware-Specialized Inference Compilation — FP8 and NVFP4 on Blackwell
Hardware-specialized inference compilation trades portability for throughput, and TensorRT-LLM — NVIDIA-only, tuned for Blackwell — is the clearest example of the trade paying off. On GB200 NVL72 with Dynamo orchestration, SemiAnalysis InferenceX measured $0.012 per million tokens on a 120B model in Q1-Q2 2026, against $0.09/M on H100 + vLLM — a 7x economic gap. The stack is three floating-point regimes compounded: FP8 stays critical for KV cache and attention kernels because it has the dynamic range they need; NVFP4 (4-bit microscaling) handles weights and activations; multi-token prediction (MTP) and disaggregated prefill/decode add another 2-3x on top. Day-0 model support loads FP4 weights directly without post-training conversion. The catch for 2026 engineering teams: TRT-LLM is open-source but NVIDIA-specific — CUDA- and Blackwell-specialized — so adopting it trades portability for throughput. Run the math on your mix of models and hardware before committing. Explain why FP8 stays critical for KV cache and attention even when weights are in NVFP4. Compute the HBM footprint of a frontier model under BF16, FP8, and NVFP4 and reason about where the savings come from. Name the Blackwell-specific features TRT-LLM exploits (day-0 FP4, MTP, disaggregated serving, all-to-all primitives). Decide when TRT-LLM's NVIDIA-lock is worth the 7x cost gap vs vLLM on Hopper. The frontier of inference economics in 2026 is "how many tokens per dollar". The…
Hardware-Specialized Inference Compilation — FP8 and NVFP4 on Blackwell: Hardware-specialized inference compilation trades portability for throughput, and…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.