Cheat sheets

Guardrails, Moderation and Redaction

Check input and output, route classifier verdicts, redact spans and gate tool calls. Python and Rust from the lessons.

  • Concepts and terms
  • Tasks and commands
  • Annotated code
  • Diagrams
  • Comparisons
  • Defaults and numbers
  • Troubleshooting
  • Questions and tips

Source

Guardrail checks#

TaskCode
Find injection patternsdetect_injection(text)
Find PII in inputdetect_pii(text)
Cap prompt sizecheck_length(text, max_chars=5000, max_words=1000)
Block toxic outputfilter_toxicity(text)
Redact PII in outputscrubbed, result = scrub_pii_from_output(text)
Check that output answers inputcheck_relevance(input_text, output_text, threshold=0.15)
Catch system prompt echocheck_system_prompt_leak(output_text, system_prompt, threshold=0.4)
Run all checks around a modelpipeline.process(attack)

The filter stack#

1Length checkreject input over 5000 chars. Length checks arefree.2Rate limitallow 10 req/min for each user.3Topic classifierreject off-topic input before the model.4PII detectorredact or reject personal data.5Injection detectorclassifiers cost 5-20ms.6LLMthe call costs 200-2000ms.7Toxicity filterclassify every output.8PII scrubber and relevance checkredact, then check the answer fits the question.

Moderation API call#

from openai import OpenAI
client = OpenAI()
response = client.moderations.create(
model="omni-moderation-latest",text and images in one call
input="<text>",
)
result = response.results[0]
print(result.flagged)True when a category fires

Thresholds in the code#

SettingDefaultNote
check_length max_chars5000Also max_words 1000
detect_injection block at0.75Max pattern confidence
filter_toxicity block at0.80Max category confidence
check_relevance threshold0.15Pass at or above
check_system_prompt_leak threshold0.4Pass below
input_moderator threshold0.5Same for output_moderator
InstructionLeakageClassifier threshold0.55Trigram cosine
_NEGATION_WINDOW20Chars checked for not, never, no

Moderation systems side by side#

FactorOpenAI ModerationLlama GuardPerspective API
Modelomni-moderation-latest, GPT-4oLlama-3.1-8B base (v3)Pre-LLM toxicity scoring
Categories13 booleansS1 to S14 in v4TOXICITY and 5 variants
InputText and imagesv4: image and textText
Languages42% better multilingual8 languages (v3)Not stated
CostFree for most developersOpen weightsResearch baseline
Use it whenYou want a hosted APIYou self-host, S14 code abuseYou compare to research

Severity rules#

high
block. Drop the output and return a policy refusal.
medium
redact. Apply each classifier redactor.
low
warn. Log and append a soft notice.
none
log. Ship the output as is.
Max wins
redact plus warn gives redact.
pii
ssn or card is high. Three findings or an email is medium.
instruction-leakage
0.85 high, 0.70 medium, 0.55 low.
toxicity
3 hits high, 2 medium, 1 low.

Output router#

verdictverdictverdictmodelModel outputprocesstoxicityslur and harassmentprocesspiiemail, phone, ssn,card, ipprocessinstruction-leakagetrigram overlapprocessRoutermax severitystoreActionblock, redact,warn, logverdictverdictverdictmodelModeloutputprocesstoxicityslur andharassmentprocesspiiemail, phone,ssn, card, ipprocessinstruction-leakagetrigramoverlapprocessRoutermax severitystoreActionblock,redact, warn,log

Troubleshooting#

SymptomCauseFix
Emoji smuggled request passesTokenizer coalesces emoji differentlyAdd a normalization step
Cyrillic lookalike letters passClassifier trained on Englishnormalize(text): NFKC, homoglyph map
Encoding attack gets throughInput moderation onlyAdd output moderation
Every input reaches the modelOutput moderation onlyAdd input moderation
"you are not a slur" trips the ruleNo negation checkNegation window before the match
Rest of a large span showsSmall nested match removedMerge intersecting spans to one union

Tool call firewall#

1callcall(line, max_bytes) splitsid|role|tool|argument and rejects controlcharacters.2decideread is Allow. Editor write is ApprovalRequired.Other roles are Deny.3authorizethe approval must equal the full call and beunused, else Error::Conflict.4dispatchthe target must be an existing contained file.Content is 16384 bytes at most.5auditone tab row per request id. A repeated id isError::Conflict.

Redaction gate API#

TaskCode
Find EMAIL, IPV4, LITERAL spansdetect(text, literals=())
Join intersecting spansmerge_spans(spans, text_length)
Replace with [REDACTED]redact(text, literals=())
Check up to 10000 recordsprocess(records, literals=())
Render the offset ledgerrender_html(report)

Run the projects#

CommandWhat it does
python3 scripts/project_test.py <project> --init learning-artifacts/<project>Makes a learner workspace
python3 scripts/project_test.py <project> --all --solution --strictTests the reference
python3 cli.py samples/input.jsonl --literals samples/literals.json --output demo-outputWrites redacted.jsonl, receipt.json and review.html
rustc --edition=2021 <dir>/cli.rs -o <dir>/firewallBuilds the firewall CLI
<dir>/firewall <root> reader r1 read notes.mdReads a file through the gate

Q&A#

Why check input and output?
Attackers find ways around each layer alone.
Is a guard classifier enough?
No. Emoji Smuggling hit 100% ASR on six guard systems.
Who supplies the role?
The caller application. The model supplies only the operation.
Output checks add latency?
Run them in parallel with token streaming. Buffer the final chunk.