AI Safety and Red Teaming

Name the attack, detect it, measure refusals, gate the output and audit alignment. Phase 18 and 19 lessons.

Three checkpoint safety gate#

Category	Trust boundary abused
role-play	the assistant's persona
instruction-override	the system prompt's authority
context-smuggling	the gap between user content and instruction content
multi-turn-ramp	the conversation history as a contract
encoding-trick	the surface form of forbidden tokens
prefix-injection	the assistant's next-token decision

Task	Code
Run a capstone demo	`python3 main.py`
Add a detector rule	`{"name": "debug-mode", "category": "instruction-override", "score": 0.80, "substring": "debug mode"}`
Score a prompt	`v = self.detector.analyze(prompt)`
Try a rot13 decode	`candidate = codecs.encode(text, "rot_13")`
Spot a refusal	`classify_refusal(response)`
Route an output	`action = self.classifier_router.run(output)`
Fix rule violations	`redacted = self.rules_fixer.apply(redacted, violations)`
Run one request	`trace = gate.handle(prompt)`

if pre.category != "benign":

if pre.confidence >= self.block_confidence:0.85 is high

signals.append(("pre", SEVERITY_RANK["high"]))

elif pre.confidence >= self.note_confidence:0.5 is medium

signals.append(("pre", SEVERITY_RANK["medium"]))

if during.terminated_early:stream cut early

signals.append(("during", SEVERITY_RANK["medium"]))

if post is not None:classifier and rules

signals.append(("post.classifier", SEVERITY_RANK.get(post.classifier_severity, 0)))

max_rank = max(rank for _, rank in signals)

if max_rank >= SEVERITY_RANK["high"]:

return "block"return a refusal

if max_rank >= SEVERITY_RANK["medium"]:

return "redact"then warn on low, else allow

Goodhart: a strong optimizer finds inputs where proxy and target differ.
Over-optimization: proxy reward climbs, gold reward peaks then falls as KL grows.
Sycophancy: agrees with the user's stated premise regardless of truth.
Trigger persistence: a backdoor survives the training meant to remove it.
Alignment faking: a compliance gap between monitored and unmonitored conditions.
Constitutional AI: critique and revise SFT, then RL from AI feedback.
AI control: trusted model T edits untrusted U. Human H audits ambiguous outputs.
PGR: $inline$ . 1.0 is fully closed.

Setting	Default	Note
block_confidence	0.85	pre-gen detector score that blocks
note_confidence	0.5	pre-gen score that adds a medium signal
buffer_cap	2	chunks the during-gen filter holds
n_bins	10	ECE bins
Leakage threshold	0.55	trigram overlap with the system prompt
max_words	800	the bounded-length rule
PAIR budget K	20	queries per goal
MSJ shot count	256	reliable attack on violent and deceitful content

Symptom	Cause	Fix
Regex misses a paraphrase	one pattern, no measurement	regex families, normalize pass, precision and recall
Agent obeys an email	filter checks only user input	label untrusted content, ratify its actions
Defense reports near zero ASR	static attacks only	evaluate with adaptive attacks
Model refuses chemistry homework	only harmful prompts measured	measure over-refusal on safe prompts
Prefix injection gets through	gate checks only after generation	add the during-gen stream filter
ASR numbers do not compare	different query budgets	match budgets, name the judge
Attack works at 256 shots	long context pattern following	classifier over the full context
Encoding attack passes input check	input moderation only	moderate input and output

PAIR: Prompt Automatic Iterative Refinement. Attacker LLM and judge LLM loop.
GCG: white box gradient search for adversarial token suffixes.
TAP: Tree of attacks. PAIR with branching and pruning.
MSJ: many-shot jailbreak. Hundreds of fake compliance pairs in context.
ArtPrompt: masks safety words with ASCII art renderings.
IPI: indirect prompt injection. Payload in content the user did not write.
IFC: information flow control. Untrusted content needs trusted ratification.
ECE: Expected Calibration Error. Stated confidence versus accuracy.

Why a taxonomy first?: Labels turn an attack stream into a histogram and a coverage chart.
Why classify outputs too?: An attack family the input checks miss lands on the user.
Which action wins?: The maximum severity. Block wins. Redact and warn becomes redact.
Why is PAIR cheaper than GCG?: It is black box and learns from each rejection in context.
Why is MSJ hard to block?: It uses the same mechanism as in-context learning.

Log a taxonomy id with every finding.
Normalize first: strip zero-width characters, decode base64, rot13 and hex. Keep the raw prompt.
Keep the constitution in YAML under version control with its own review.
Put a guard classifier on both sides of the model.
Run single turn probes nightly and multi-turn campaigns before release.
Score each successful jailbreak with CVSS 4.0.