Skill Evals, Packaging, and Portability
A skill is finished when its package survives linting, routes on the right requests, improves a measured task, stays inside policy, and degrades honestly on another host. Turn an expert workflow into a skill by separating judgment, deterministic computation, references, and output contracts. Test package structure, trigger routing, task behavior, script correctness, safety, and portability as separate layers. Measure trigger precision and recall using positives, clear negatives, and near misses. Compare performance with and without the skill across repeated runs. Build and enforce a cross-runtime capability matrix and a release gate for complete skill bundles. A skill works in one demo. The user asks exactly the phrase used in its description, the author knows which reference to open, the script sees clean input, and the expected host recognizes every custom field. Then real use begins. The model invokes it for a nearby but different task. A valid request uses unfamiliar wording, so the model misses it. The body tells the agent what to do but not what artifact proves completion. The script fails on spaces, repeated execution, or partial state. The package installer copies SKILL.md but leaves its references behind. Another runtime ignores the invocation flags and tool allowance. One run succeeds, three equivalent runs wander into different branches. None of these failures is caught by "the Markdown looks good."…
Skill Evals, Packaging, and Portability: A skill is finished when its package survives linting, routes on the right requests, improves a measured task, stays…
This free lesson is part of the AI Engineering from Scratch curriculum. Read the full explanation, run the lesson code, and verify the result in the interactive reader or from the repository source.
Browse the complete course catalog or open this lesson on GitHub.