Structured Outputs and Constrained Decoding

Get valid JSON every time. Schemas, strict mode, logit masks, validators, retries and the output guard project.

Concepts and terms
Tasks and commands
Annotated code
Diagrams
Defaults and numbers
Troubleshooting
Questions and tips

Output guard pipeline#

From solution/main.ts, json-schema-output-guard/README.md

One constrained decoding step#

Run the output guard project#

Command	What it does
`python3 scripts/project_test.py json-schema-output-guard --init <ws>`	Creates your workspace once. A fresh starter fails
`python3 scripts/project_test.py json-schema-output-guard --stage 1 --path <ws> --strict`	Grades one stage, 1 to 4
`python3 scripts/project_test.py json-schema-output-guard --all --solution --strict`	Grades the reference solution
`node cli.ts --schema samples/schema.json --attempts samples/attempts.json --output guard.json --html guard.html`	Recorded repair. Attempt 1 fails at `$/confidence`

From json-schema-output-guard/README.md, docs/en.md

Tasks in code#

Task	Code
Local JSON with an FSM	`generator = outlines.generate.json(model, Review)`
Pydantic plus retries, any provider	`client = instructor.from_anthropic(Anthropic())`
Vendor schema mode	`client.beta.chat.completions.parse(..., response_format=Product)`
Validate a dict	`validate_schema(data, schema)` returns a list of errors
Parse, validate, retry	`extract_with_retry(text, schema, max_retries=3)`
Three-branch handler	`process_model_output(raw, INVOICE_SCHEMA)`
Guard one raw string	`guard(raw, schema)` returns `ok`, `value`, `issues`
Bounded repair	`await repair(generate, schema, maxAttempts)`

From Lesson 05.20, Lesson 11.03, 13.04 code/main.py, solution/main.ts

Questions#

Should a refusal trigger a retry?: No. Log it and return a typed refusal.
Is constrained decoding slower?: Often faster. Forced tokens skip generation.
Does it need a big model?: No. A 3B model with grammar beats a 70B model with prompts.
Does acceptance prove the answer?: No. It does not establish factual correctness.
Why call checkSchema first?: A broken schema then cannot spend provider budget.

From Lesson 05.20, Lesson 13.04, json-schema-output-guard/README.md, docs/en.md

Pydantic schema with Instructor#

class Invoice(BaseModel):

vendor: str

total_usd: float = Field(ge=0)numeric bound

line_items: list[str]

client = instructor.from_anthropic(Anthropic())

invoice = client.messages.create(

model="claude-opus-4-7",

max_tokens=1024,

response_model=Invoice,retries on validation failure

messages=[{"role": "user", "content": "<text>"}],

)

From Lesson 05.20

Tips#

Local model, flat schema: Outlines. Recursive schema: XGrammar or llguidance.
Self-hosted server: vLLM guided_json, guided_regex, guided_choice.
Use tool use when the model must pick one of many schemas.
Send structured Issue[] back to the model, not free text.
Keep every attempt in the trace. A late success does not hide earlier failures.
The guard rejects $ref, format and combinators.
In the project commands, <ws> is learning-artifacts/json-schema-output-guard.

From Lesson 05.20, Lesson 11.03, json-schema-output-guard/README.md, docs/en.md

Bounded repair in TypeScript#

const schema: Schema = {

type: "object",

properties: {

answer: { type: "string", minLength: 1 },

confidence: { type: "number", minimum: 0, maximum: 1 },

},

required: ["answer", "confidence"],

additionalProperties: false,unknown property is an issue

};

await repair(async (_, i) => outputs[i - 1], schema)status and trace

From solution/main.ts

Control levels#

Prompt-based: "Respond in valid JSON". About 90% valid. No enforcement.
JSON mode: valid JSON syntax. No schema guarantee.
Schema mode: output matches the JSON Schema. Native in every major provider.
Constrained decoding: masks invalid tokens. 100% valid by construction.
Instructor: does not touch logits. It parses, validates and retries.
Outlines: compiles schema or regex to an FSM. Recursion needs flattening.
XGrammar, llguidance: CFG engines. They handle recursive schemas.
Strict mode: all properties in required, additionalProperties: false, no $ref.

From Lesson 11.03, Lesson 05.20, Lesson 13.04

Defaults and numbers#

Setting	Default	Note
`max_retries`	3	In `extract_with_retry`
Instructor retries	3	On validation failure
Retry limit	3x	More retries point to a bad schema
`maxAttempts`	3	Budget must be 1..10
Depth limit	32	`checkSchema` and `validate`
Output size cap	100_000	Bytes, checked before `JSON.parse`
CLI exit code	2	Rejected or exhausted result
Prompt-only JSON	85 to 95 percent	Success rate on frontier models

From Lesson 11.03, Lesson 05.20, Lesson 13.04, solution/main.ts, solution/cli.ts

Troubleshooting#

Symptom	Cause	Fix
Valid JSON, wrong answer	`answer` before `reasoning`	Put reasoning fields first
Model invents a date	Grammar too strict	Allow `null` or a sentinel
10,000-option enum times out	Huge enum	Retrieve top-k, constrain to those
Tree output fails in Outlines	FSM flattens recursion	Use XGrammar or llguidance
Valid JSON, wrong shape	JSON mode without a schema	Always send a full schema
API returns 400	Strict mode rule broken	Fix `required`, extras, `$ref`
Right type, invented value	Hallucinated value	Schema cannot catch it. Check values
Optional field left out	Field not required	Make it required, allow `null`

From Lesson 05.20, Lesson 11.03, Lesson 13.04

Abbreviations#

FSM: finite-state machine. $inline$ valid-next-token lookup.
CFG: context-free grammar. Handles recursion.
Logit processor: (logits, state) -> masked_logits.
Guided decoding: the same idea inside an inference server.
Refusal: a typed outcome when the input cannot fit the schema.
Parse error: output is not JSON. Impossible under strict mode.
Schema violation: output parses but breaks the schema.
Issue: { path: string; message: string } in the guard project.

From Lesson 05.20, Lesson 13.04, solution/main.ts