Structured Outputs & Constrained Decoding
Ask an LLM for JSON. Get JSON most of the time. In production, "most" is the problem. Constrained decoding turns "most" into "always" by editing the logits before sampling. A classifier prompts an LLM: "Return one of {positive, negative, neutral}." The model returns "The sentiment is positive — this review is overwhelmingly favorable because the customer explicitly states that they ...". Your parser crashes. Your classifier's F1 is 0.0. Free-form generation is not a contract. It is a suggestion. A production system needs a contract. Three layers exist in 2026. Prompting. Ask nicely. "Return only the JSON object." Works 80% on frontier models, less on smaller ones. Native structured output APIs. OpenAI responseformat, Anthropic tool use, Gemini JSON mode. Reliable on supported schemas. Vendor-locked. Constrained decoding. Modify the logits at every generation step so the model cannot emit invalid tokens. 100% valid by construction. Works on any local model. This lesson builds intuition for all three and names when to reach for which. Constrained decoding masking invalid tokens at each step How constrained decoding works. At each generation step, the LLM produces a logit vector over the full vocabulary (100k tokens). A logit processor sits between the model and the sampler. It computes which tokens are valid given the current position in the target grammar — JSON Schema, regex, context-free…
Structured Outputs & Constrained Decoding: Ask an LLM for JSON. Get JSON most of the time. In production, "most" is the problem. Constrained decoding turns…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.