Chatbots and Dialogue State Tracking

Route turns, run the agent loop, track slots across turns and budget the context window. Python from the lessons.

Hybrid routing#

Task	Code
FAQ answer or `None`	`faq_respond(user_input, threshold=0.5)`
Detect a destructive request	`is_destructive_action(text)`
Agent with a step budget	`agent_loop(user_message, tools, llm, max_steps=5)`
Score a tracker	`joint_goal_accuracy(predicted_states, gold_states)`
Budget the window	`ContextBudget(max_tokens=128000, generation_reserve=4000)`
Add a component with a cap	`budget.allocate("tools", tool_text, max_tokens=2000)`
Prune tools by intent	`select_tools(query, token_budget=2000)`
Best docs at start and end	`reorder_lost_in_middle(items, scores)`

SLOT_EXTRACTORS = {"cuisine": extract_cuisine, "area": extract_area,

"price": extract_price, "people": extract_people}

def update_state(state, utterance):

new_state = dict(state)never reset untouched slots

for slot, extractor in SLOT_EXTRACTORS.items():

value = extractor(utterance)

if value is not None:

new_state[slot] = valuea correction overwrites

continue

if is_negation(utterance, slot):

new_state[slot] = None"never mind the cuisine"

return new_state

class RestaurantState(BaseModel):

cuisine: Optional[Literal["italian", "chinese", "indian", "thai", "any"]] = None

area: Optional[Literal["north", "south", "east", "west", "center"]] = None

price: Optional[Literal["cheap", "moderate", "expensive"]] = None

people: Optional[int] = None

day: Optional[str] = Nonefree-form slot

def llm_dst(history, llm):

prompt = f"""...{render(history)}...Output only the JSON state."""

return llm(prompt, response_model=RestaurantState)whole state each turn

Rule-based: patterns, intents and slot filling. Fails outside its scope.
Retrieval-based: nearest stored response. No generation, so no hallucination.
Neural seq2seq: fluent, but generic replies and factual drift.
LLM agent: plan, call tool, observe result, decide next step.
Slot: a named parameter the backend needs, such as cuisine or date.
Classification DST: yes or no for each slot and candidate value. Closed vocabulary.
Generation DST: generate slot values as text. Open vocabulary. The modern default.
JGA: fraction of turns where every slot is correct. All or nothing.

Setting	Default	Note
`max_tokens`	128000	Window size of `ContextBudget`
`generation_reserve`	4000	Reserve 2,000-4,000 for the answer
`max_history_tokens`	5000	Above it, summarize the 2 oldest turns
`token_budget`	2000	Tool definitions for one query
Relevance threshold	0.05	The lesson text uses 0.1
Token estimate	1.3	Tokens for each whitespace word
`max_steps`	5	Agent step budget
FAQ `threshold`	0.5	Hybrid route uses 0.6

Symptom	Cause	Fix
Agent claims an action it did not do	Confident fabrication	Trust only a successful tool return
Input overrides the system prompt	Prompt injection	Untrusted input, PVE, confirm destructive
Same tool called again and again	Infinite loop	Step budget, dedupe tool calls
Earliest turns lost	Context window full	Summarize or retrieve old turns
Fact in the middle ignored	Lost in the middle	Key facts first or last
Correction clears other slots	Reset of untouched slots	Overwrite only the corrected slot
"Italian" and "italian" differ	Case	Normalize everywhere
Old training data breaks	New slots added later	Version the schema

Why route instead of one model?: No single architecture handles every request well.
Why rules for payments?: Destructive actions need deterministic, confirmed flows.
Incremental update or regenerate?: Regenerate the whole state from history. It handles corrections.
What does regeneration cost?: $inline$ total tokens. Cap or summarize old turns.
Is a bigger window enough?: No. A curated 10K context often beats a dumped 100K.