Measuring agent skill quality

Positive feedback tells you the skill worked at least once. Negative and neutral feedback are the better product signals because they show where the skill lost context, stalled, used the wrong tool, or needed a human to recover. A serious measurement system keeps that feedback, but it does not stop there.

An agent skill is not a prompt snippet. It is an operating procedure for a model that may read files, call tools, run scripts, ask for approvals, create artifacts, and decide when to stop. OpenAI describes Codex skills as packages of instructions, resources, and optional scripts that help Codex follow a workflow reliably in its Agent Skills docs. Anthropic uses a similar shape: a skill is loaded when relevant and can include instructions, scripts, and reference material, as shown in its Agent Skills overview. That means quality has to be measured at the workflow level, not just at the final paragraph the model writes.

The question for a skill author is simple: when this skill is present, does the agent behave better than it would without it? Better means more task completions, fewer skipped instructions, cleaner tool use, lower cost, and safer behavior around irreversible actions.

Start by writing the job the skill is supposed to improve in operational terms. A useful definition is narrow enough that two reviewers can score the same run without debating taste.

For a code review skill, "better review" is too vague. A measurable target is: finds the seeded behavioral regression, cites the affected file and line, avoids unrelated formatting notes, and recommends a test that would have caught the regression. For a QA skill, "tests the app" is too broad. A measurable target is: launches the local app, exercises the checkout flow with a real browser, captures a screenshot or trace, records failed expectations, and distinguishes blocker bugs from cosmetic issues. The Skillfully post on agent skills for QA is a useful companion because QA skills often fail in the gap between "look around" and a reproducible test record.

This is the same discipline behind eval systems. OpenAI's Evals guide frames evaluations as tests of outputs against style and content criteria. Google ADK's evaluation docs split agent evaluation into trajectory and tool use plus final response quality. LangSmith describes evals as a way to break down what good looks like across the application lifecycle in its evaluation concepts. Agent skills need the same decomposition, with one added burden: the skill author must measure whether the skill changed the agent's process, not only whether the answer looked acceptable.

A single thumbs-up hides too many failure modes. Keep the user rating, but attach it to a scorecard that separates the dimensions you can fix.

Dimension	What to Measure	Passing Evidence
Task success	Did the agent complete the intended job?	Required artifact exists, tests pass, expected decision made, or reviewer accepts the result
Instruction adherence	Did it follow the skill's mandatory steps?	Trace shows required files read, questions asked, approvals requested, or checkpoints completed
Tool and script correctness	Did tool calls match the workflow?	Commands ran in the right directory, scripts received valid inputs, outputs were interpreted correctly
Safety and rollback	Did it avoid irreversible or out-of-scope actions?	No unauthorized writes, no unrelated files touched, rollback path recorded for risky operations
Human review quality	Did it make review easy?	Clear summary, cited evidence, known risks, and no hidden assumptions
Cost and latency	Did the improvement justify the run time and token spend?	Comparable or better success with acceptable wall-clock time, retries, and model/tool cost
Regression stability	Does it keep working after edits?	Golden tasks pass across repeated runs and across realistic variants

Use 0, 1, and 2 if the team will actually score it. Zero means failed or absent. One means partly present but required human repair. Two means usable. A three-point scale beats elaborate rubrics nobody uses.

Task success rate is the first metric because it is hard to argue with. The trap is counting only happy paths. A skill that succeeds on three polished demos and fails on seven messy real tasks has a 30% success rate, not "three impressive examples."

Define the denominator as runs where the task was in scope for the skill and the environment was available. Exclude outages and missing credentials, but do not exclude confusing prompts, partial context, or ordinary repo messiness if the skill claims to handle those conditions.

For a writing skill, success might mean the finished document meets the requested format, preserves source facts, and needs no structural rewrite. For a publishing skill, success might mean the package installs from the advertised source and can be invoked in a fresh session. For a code skill, success usually requires tests or a build step, not just a plausible diff. The post on how to write better agent skills covers the instruction side; measurement adds proof that those instructions changed outcomes.

Keep success rates by task class. A skill can be strong at "review a small PR" and weak at "repair a failing CI job." Blending those into one number makes repair work harder.

Instruction adherence is not the same as task success. An agent can succeed accidentally while ignoring the skill. It can also follow the skill faithfully and still fail because a dependency is missing or the task is underspecified.

Trace the steps that matter. If a skill says to read AGENTS.md before editing, the run should show that file read. If it says to ask for approval before destructive operations, the run should show the question before the action. If it says to preserve frontmatter, a diff should prove the frontmatter stayed unchanged.

For skills that produce structured artifacts, make the adherence check machine-readable. OpenAI's Structured Outputs guide explains why schemas are useful: they force responses to match the fields you specify. A skill author can apply the same idea to run reports with fields such as files_read, tools_called, risks_found, validation_commands, and blocked_reason.

Do not overfit adherence to ceremony. Mark the steps that protect quality: context gathering, tool selection, safety gates, validation, review handoff, and final evidence.

Skills often get their leverage from bundled scripts. A script can encode a tedious transformation, verify an artifact, or query an API more reliably than prose instructions. That leverage becomes a liability when the agent calls the wrong script, passes stale paths, or treats partial output as proof.

Measure tool correctness with concrete assertions:

• The command ran from the intended working directory.
• The input file existed and matched the current task.
• The script exited successfully, or the failure was reported accurately.
• The agent used the output rather than inventing a conclusion.
• Any generated files landed in the expected directory.
• The run did not touch files outside the allowed scope.

For a skill that wraps a CLI, include one or two fixture tasks: an input file, expected output, and a command that validates the transformation. If a script depends on network access or credentials, add a dry-run path. Scripts make skills more repeatable, but they add environment assumptions, so the eval should include a missing-prerequisite case.

A skill that raises success from 80% to 90% may be worth it. A skill that raises success from 80% to 82% while doubling runtime and token spend probably is not. Cost includes API billing, waiting time, review time, and collision risk.

Track four numbers per run:

• Wall-clock time from invocation to final answer.
• Number of model turns or subagent runs.
• Tool calls that hit paid or rate-limited services.
• Retries caused by the skill's own ambiguity.

Latency can also reveal instruction bloat. If a skill forces the agent to read five reference files for every small task, the trace will show it. Move rarely needed detail into on-demand references and measure whether the agent loads the reference only when the task requires it. Anthropic's engineering note on equipping agents with skills argues for starting from observed capability gaps and building skills incrementally; that principle applies to cost as much as correctness.

Safety is measurable. Do not wait for a bad incident to discover whether a skill respects boundaries.

Create golden tasks that tempt the agent to overreach. Ask a code skill to fix one file while unrelated files are dirty. Ask a publishing skill to prepare a package but not push it. Ask a migration skill to explain the rollback path before touching the schema.

Score the run on what the agent refused, delayed, or escalated:

• Did it notice the scope boundary?
• Did it avoid unrelated files?
• Did it ask before irreversible actions?
• Did it preserve user changes?
• Did it record how to undo the risky step?
• Did it stop when authentication or permissions failed?

Rollback quality matters because many useful skills operate near production workflows. A skill that can deploy, publish, charge, email, or delete should include explicit stop conditions. "Be careful" is not a stop condition. "If the authenticated fetch returns non-200, stop and report the status" is a stop condition.

Golden tasks are representative examples with known success criteria. They are not demos. They should include the ordinary ambiguity that caused past failures.

Start with five:

• A clean happy path that should pass quickly.
• A missing prerequisite, such as a dependency, credential, or source file.
• A scope-boundary case with unrelated changes nearby.
• A realistic edge case from user feedback.
• A regression case from the last time the skill failed.

Each golden task needs a prompt, starting state, expected behavior, and scoring rubric. For code tasks, preserve a fixture repo or branch. For writing tasks, preserve the source material and expected claims. For review tasks, seed one bug and one tempting non-issue so you can tell whether the skill finds signal instead of generating noise.

Google's ADK codelab on evaluating agents uses golden datasets to establish ground truth and catch regressions early. That maps directly to skills: every important edit to SKILL.md, a bundled script, or a reference file should run against the same small set.

Many skill outputs cannot be checked with exact string matching. A research synthesis, code review, incident retrospective, or UX critique can be correct in more than one form. Use rubrics, but make them behavior-specific.

Weak rubric: "Is the answer good?"

Useful rubric:

• Identifies the highest-severity issue before minor issues.
• Cites concrete evidence from files, logs, screenshots, or sources.
• Separates confirmed facts from inferences.
• Gives an action that the user can take without asking a follow-up.
• Avoids claims not supported by the available evidence.

Model-graded rubrics can help triage large eval sets, but they should not be treated as an oracle. OpenAI's cookbook example for getting started with OpenAI Evals distinguishes deterministic templates from model-graded templates; use deterministic checks when the answer space is narrow, and use model grading when variation is unavoidable. Promptfoo's evaluation tooling and Ragas agentic metrics are useful only when the rubric still matches the skill's job.

Community discussions are a healthy warning against metric worship. In a Reddit thread questioning RAGAS for RAG evaluation, one practitioner argued that evaluation has no single silver bullet and teams need multiple experiments and metrics rather than one magic score: r/LangChain discussion. Rubrics are decision support, not truth.

Telemetry tells you where to look. Human review tells you what the failure meant.

At minimum, log the skill name and version, task category, user rating, written feedback, completion status, validation commands, files changed, tools called, run duration, and failure reason. If the agent stops, log whether stopping was correct.

Then review clusters, not anecdotes. Ten neutral ratings that mention "could not prove result" point to missing validation guidance. Five negatives that mention "wrong directory" point to command examples that assume too much.

This is where Skillfully feedback becomes more than a wall of comments. A rating sorts the run. The text explains the repair. The trend shows whether the repair worked. When you revise a skill, write down the feedback cluster the edit is meant to fix, then watch whether that cluster shrinks.

The cleanest proof is an A/B comparison: same task, same model family, same starting state, with and without the skill. Ten paired runs can reveal whether the skill changes behavior.

Look for deltas:

• Did task success rise?
• Did instruction misses fall?
• Did the agent use the right tools earlier?
• Did validation become more concrete?
• Did human review time fall?
• Did the skill add unnecessary cost?

Be strict about regressions. A skill that improves final prose but causes the agent to skip tests is worse for engineering work. A skill that improves one task class while harming another may need a narrower description so the agent invokes it only when appropriate. The post on how to create an agent skill is especially relevant here: metadata and activation wording affect when the skill enters the run, so they are part of quality.

For code review workflows, compare not only the final review but the defects found. A strong agent skill for code review should increase true positives without flooding the user with style noise. Track false positives because they burn trust quickly.

A skill is improving when three things happen at the same time.

First, the golden set gets better or stays stable. Second, live feedback shifts from fundamental failures to narrower edge cases. "Did not understand the workflow" should become "needs clearer handling for private repos." Third, human reviewers spend less time reconstructing what happened.

Watch for false improvement. If the skill gets longer and the agent sounds more confident, that is not evidence. If the score rises because reviewers stopped scoring hard cases, that is not evidence. If a model upgrade masks a weak skill, the skill still needs a baseline comparison.

The release rule is conservative: promote the skill when repeated runs produce similar positive evidence, remaining failures are explainable, and the next repair is obvious. Keep it private while failures are random, safety behavior is untested, or success depends on hidden context in the author's head.

Use a loop that is boring enough to repeat:

1. Pick one task class the skill should improve.
2. Write five golden tasks with expected behavior.
3. Run a no-skill baseline.
4. Run the skill on the same tasks.
5. Score task success, adherence, tool correctness, safety, review quality, cost, and latency.
6. Revise one part of the skill.
7. Re-run the same golden tasks.
8. Promote only if the target metric improves without a safety or cost regression.

That loop gives a skill author a concrete answer to the only question that matters: did the skill make the agent better at the work? Better means completing the task, respecting constraints, using tools correctly, and leaving evidence a human can trust.