Cheat sheets

Skill Permissions, Evals and Supply Chain

Review skill actions before they run, isolate them, gate each release by eval layer and scan bundles.

  • Concepts and terms
  • Tasks and commands
  • Annotated code
  • Diagrams
  • Comparisons
  • Defaults and numbers
  • Troubleshooting
  • Questions and tips

Source

Symptoms, causes and fixes#

SymptomCauseFix
deny with rule workspace-jailThe target resolves outside the rootKeep targets under workspace_root
command-review deny with approved=Truerm is in DESTRUCTIVE_EXECUTABLESApproval cannot override it. Use a narrow tool
network-allowlist deny on port 8443Port 8443 is a different originAdd that origin with its port
approval-gate on an allowlisted URLinfluenced_by_untrusted_content is trueShow the target and consequence. Get approval
secret-review denyThe payload matches SECRET_PATTERNSInject the credential into one narrow tool
Lint issue orphan-fileSKILL.md does not reference the fileReference the file or remove it
fixture_passed true, passed falseNo trusted external attestationPass --attestation and --trusted-attestation-sha256
Audit usable is falseSame content or group in both partitionsUnion by content and group before the split

From Lesson 13.26, 13.26 code/main.py, Lesson 13.27, 13.27 code/main.py, self-improving-skill-loop/README.md, solution/dataset.py

review_action check order#

1policy-changealways deny with rule authority-boundary.2Kinda kind outside allowed_kinds gives akind-allowlist deny.3Pathread, write and delete resolve throughnormalize_workspace_path. An escape givesworkspace-jail.4Commandinspect_command checks the bare name,destructive set, metacharacters and argv prefix.5Networknormalize_https_origin adds the effective port.The origin must be in the allowlist.6Secretsa contains_secret match gives deny, orrequire-approval withpermit_secret_use_after_approval.7Approvalapproval_kinds and untrusted influence needapproved. Else rule approval-gate.8Allowrule policy-allow. The field executed staysfalse.

From 13.26 code/main.py

Review one action#

policy = SandboxPolicy(
workspace_root=workspace,resolved with strict=True
allowed_kinds=("read", "write", "delete", "command", "network"),
command_allowlist=(("python3", "-m", "unittest"),),argv prefixes
network_allowlist=("https://docs.example.test",),origins only
)
request = ActionRequest("write", target="summary.json")write is in approval_kinds
review_action(policy, request).to_dict()["verdict"]"require-approval"
request = ActionRequest("write", target="summary.json", approved=True)
review_action(policy, request).rule"policy-allow"
normalize_https_origin("https://api.example.test")adds ":443"

From 13.26 code/main.py, Lesson 13.26

Run the isolation drill#

docker build -f code/sandbox/Containerfile -t aiefs-skill-sandbox code/sandbox
docker run --rm --network none --read-only --cap-drop ALL \
--security-opt no-new-privileges --pids-limit 64 --memory 128m --cpus 0.5 \
--tmpfs /tmp:rw,noexec,nosuid,size=16m \
--mount type=bind,src="${PWD}/code/sandbox/input",dst=/input,readonly \
--env DEMO_VALUE=bounded aiefs-skill-sandbox
expect: /input readable, image read-only, network fails

From Lesson 13.26

Six eval layers#

Structure
lint without a model. CI blocks E_* errors and allows reviewed W_* warnings.
Routing
precision and recall over positives, clear negatives and near misses. Report raw counts.
Behavior
baseline and treatment use the same model, tools, task and budget.
Scripts
test outside model runs with empty, malformed and repeated input.
Safety
record the control: instruction only, tool policy, approval, sandbox or verification.
Portability
install into a clean destination. Validate the installed copy.
Verdicts
fixturePassed, localEvidenceReady, productionReady. passed follows productionReady.

From Lesson 13.27

Run the labs and tools#

CommandWhat it does
python3 code/main.pyPrints the demo decisions. It starts no process
python3 -m unittest discover -s code/tests -vRuns the lesson tests
python3 scripts/review_action.py --policy <policy> --request <request>Returns the verdict and the rule for one request
evaluate_skill.py --fixture-demo <bundle>Fixture run. fixturePassed true, productionReady false
scanner examples/reviewable 3Scans a bundle with threshold 3. Prints JSON
cli.py <cases> --out experiment.jsonProposes rules and runs the gate. It does not promote
python3 scripts/project_test.py <project> --all --solution --strictChecks every stage of the reference solution

From Lesson 13.26, 13.26 outputs/skill-safety-reviewer/SKILL.md, Lesson 13.27, skill-scanner/README.md, self-improving-skill-loop/README.md

Defaults and limits#

SettingDefaultNote
allowed_kinds("read",)SandboxPolicy allows reads only
approval_kinds("write", "delete", "network")These kinds need approved
MAX_SKILL_BODY_CHARS10_000Lint issue body-size above it
MAX_COMPANION_FILE_BYTES1_000_000Lint issue file-size above it
min_precision, min_recall0.8ReleaseThresholds in the lab
min_repeat_rate0.9Lowest per-case pass rate
min_gain0.05The loop gate also needs zero regressions
Scanner threshold3A threshold of 0 is an error

From 13.26 code/main.py, 13.27 code/main.py, solution/promotion.py, solution/cli.rs

Isolation boundaries#

FactorRestricted subprocessContainerMicroVM
IsolatesEnvironment, cwd, timeout, outputFilesystem and process namespacesSeparate guest kernel
Does not isolateKernel, host filesystem, networkShared kernel, host mountsBad mounts, credentials, egress
Typical useReviewed local utilitiesRepository builds and testsUntrusted code, high impact

From Lesson 13.26

Project pipelines#

linesfindingsscoredevelopmenterrorscandidateprocessspansUTF-8 byteoffsetsprocessscanthree named rulesprocessrisk_scoreonce per rule andlineclientreviewreview-requiredorbelow-thresholdstoresplit_casescontent and groupunionprocessevaluateper-case rowsprocessproposedev errors,min_support 2clientgategain andregressionslinesfindingsscoredevelopmenterrorscandidateprocessspansUTF-8 byte offsetsprocessscanthree named rulesprocessrisk_scoreonce per rule and lineclientreviewreview-required orbelow-thresholdstoresplit_casescontent and group unionprocessevaluateper-case rowsprocessproposedev errors, min_support2clientgategain and regressions

From skill-scanner/solution, self-improving-skill-loop/solution

Tasks in code#

TaskCode
Lint a bundlelint_package(root)
Score routingclassification_metrics(expected, predicted)
Check an installed copyverify_manifest(root, expected)
Host support matrixportability_matrix(requirements, hosts)
Split without leakagesplit_cases(cases, holdout_fraction=0.25)
Propose rulespropose(development, rules, min_support=2)
Gate a candidategate(holdout, baseline, candidate)
Refine a library skillregister(skill, dedup=True) bumps the version

From 13.27 code/main.py, 14.10 code/main.py, self-improving-skill-loop/solution

Questions#

Does allowed-tools sandbox a skill?
No. It affects capability or permission prompts. It is not OS isolation.
Does a hash prove a package is safe?
No. It proves that the bytes match a manifest.
Is a below-threshold scan safe?
No. Only these rules did not cross the threshold.
Why version skills?
A parent pinned to v1 does not pick up a refined v3.
Why rotate the holdout?
Repeated tuning on the same holdout leaks information.

From Lesson 13.26, skill-scanner/README.md, Lesson 14.10, docs/en.md