Infrastructure & serving · Glossary term

What is Goodput?

The rate of completed requests that satisfy defined service constraints, such as both time-to-first-token and per-token latency objectives, under a stated workload.

Why does Goodput matter?

Raw throughput can rise while users experience more slow requests. Goodput counts only work that meets the service contract.

Goodput in practice

Declare the request distribution and latency thresholds, count only compliant completions, report percentiles beside the aggregate rate, and avoid comparing systems under different objectives.

What is the common confusion about Goodput?

Goodput is not all completed throughput and is not a universal property of a model. It depends on workload and success thresholds.

Learn Goodput in the course

Start with

  • Inference Metrics — TTFT, TPOT, ITL, Goodput, P99

    Four metrics decide whether an inference deployment is working. TTFT is prefill plus queue plus network. TPOT (equivalently ITL) is the memory-bound decode cost per token.

    Phase 17: Infrastructure & Production

Taught in Phase 17: Infrastructure & Production.

  • Service Level Objective (SLO)A target range or threshold for a service-level indicator over a stated population and measurement window.
  • Time to First Token (TTFT)The elapsed time from submitting a generation request until the client receives the first output token or content event under a defined…
  • Time per Output Token (TPOT)For one request with `N > 1` output tokens, the average post-first-token interval: `(t_N - t_1) / (N - 1)`.
  • Cost per Successful TaskTotal system cost divided by the number of tasks that satisfy a defined success criterion, including retries, failed runs, tool use, and…
  • Disaggregated ServingA serving architecture that runs prefill and decode work in separately provisioned worker pools and transfers the required attention state…
  • Tail LatencyThe latency experienced by the slowest portion of requests, commonly summarized with a high percentile under a stated workload and time…

Sources

More terms in Infrastructure & serving

Open the Infrastructure & serving list in the glossary

This entry comes from glossary/terms.md on GitHub. Browse all 250 glossary terms.