Check whether a skill update changed your method

Before releasing an update to your author skill, run the previous version and the candidate version through the same reader situations. Review both against the principles your book requires. Keep the inputs, outputs, version details, and reasons for your decisions.

That is regression testing in this context: checking that an improvement has not removed behavior you still want. For an author, the important regression may be a change in judgment rather than a broken file. A shorter answer might skip the question your method requires. A more decisive answer might turn a conditional recommendation into an instruction you would never give.

You do not need to freeze your method forever. You do need to distinguish an intentional revision from an accidental change that arrived while you were polishing the companion.

This walkthrough uses a fictional book-club facilitation method. Its response examples are author-written illustrations, not actual model outputs. The worksheet is a proposed test procedure, with no completed runs or performance results claimed.

Imagine your book teaches hosts to facilitate discussion without treating every quiet participant as a problem to solve. Your companion helps hosts plan an opening activity and consider responses to uneven participation.

Its current method has three commitments:

• Ask about the group's agreed participation format before recommending a change to it.
• Offer people an opportunity to contribute without requiring everyone to speak.
• Describe observed behavior without claiming to know why someone is quiet.

You want the companion's answers to be shorter. Write the change request this narrowly:

Reduce repeated explanation and keep the first response easy to use. Preserve the participation choices, required context question, and distinction between observation and motive.

Compare that with “Be concise and give a decisive recommendation.” The second instruction leaves open whether the new decisiveness should override uncertainty or participant choice. Your change request should name what must survive the edit.

Record the reason too. Perhaps you personally found the prototype verbose. That is enough to motivate an experiment, but it is not evidence that readers complained. If a reader did report the issue, retain the actual feedback with appropriate permission and describe it accurately.

Save the current instructions and supporting material before editing. Give the candidate a different version label. If both are called “latest,” you will struggle to explain what produced a particular answer.

A writer discussing experimentation described a simple separation between exploration and retained work. u/human_assisted_ai wrote:

“I experiment in the chat but the fallback goes in the draft document.”

That comment concerns fiction drafts, not skill releases. The useful connection is the habit of keeping an accepted version outside the experiment. It does not establish that a particular tool provides version recovery.

For your skill, keep a small release folder or equivalent record containing the instructions, references, examples, evaluation criteria, and test inputs. Record the model and runtime used, along with any settings or connected sources relevant to the task. Save the actual trial outputs separately.

If the model or environment also changes, write that down. Otherwise you might attribute a difference to your revised instruction when several things changed together. Where possible, compare instruction versions under the same conditions first. If that is no longer possible, describe the comparison as limited rather than pretending the old environment is still available.

Choose cases that expose your actual commitments. A generic request to “plan a great book club” will not tell you whether the quiet-participant rule survived.

Here is a proposed set for the fictional method:

C05 protects usefulness as well as caution. A skill that asks every question on every turn has not preserved the method intelligently. It has turned a context requirement into a ritual.

Keep the inputs unchanged during the comparison. If you make C02 more explicit halfway through because the candidate struggles, you are testing a new case. Save that new case separately and retain the original failure.

Add examples from actual reader use as they become available, with private details removed appropriately. A small fictional set is a starting point for inspection, not proof that you have represented every reader situation.

For C02, consider these two illustrative responses.

Response that preserves the requirement:

Before changing the format, what participation agreement does the group currently use? For example, is the discussion open, or have people agreed to invitations with an option to pass?

Response that loses the requirement:

Switch to a round-robin format next time. Give each person two minutes so everyone contributes equally.

The second answer is shorter and concrete. Under this fictional author's method, it still fails: it prescribes a format without checking the agreement and does not preserve the choice to pass.

Now consider another acceptable response:

What has the group agreed about speaking and passing? That will determine which change fits your method.

It uses different words from the first response but preserves the same decision. Your test should allow that variation. Exact wording matters when the wording itself is a requirement; otherwise, inspect whether the response follows the principle.

Anthropic's evaluation guidance distinguishes tests of new capabilities from regression checks on existing tasks. It also notes that outputs vary between runs. For your companion, that means one attractive answer is insufficient evidence that the behavior will consistently survive the update.

Create one record per case and trial:

Do not fill the baseline and candidate evidence cells with the illustrative responses above and call that a test. Those examples explain the rubric. Actual evidence comes from running the versions and retaining what happened.

When practical, hide the version labels during review so you judge the response before knowing which one you hoped would improve. A second person can help inspect clarity, but the author must still settle questions about what the method means. If reviewers disagree, keep both reasons until the criterion is clarified.

Plan repeated trials before looking at results. For a first small check, you might choose three fresh runs per case and version, then expand uncertain cases. Three is a manageable starting procedure, not a statistical guarantee. Preserve unsuccessful runs instead of rerunning until you get the answer you wanted.

An aggregate score can make an update look better while concealing a change you would reject. Suppose your candidate becomes more concise on routine openings but starts assigning motives to quiet participants. Under the fictional method, that tradeoff is unacceptable even if the average writing score rises.

Define the release rule before testing. For this example: any observed violation of the three method commitments blocks this candidate until investigated and corrected. Concision is evaluated after those commitments. An inconclusive case needs a clearer test or further evidence; it does not become a pass by default.

A practitioner discussing their own regression checks, u/Acceptable-Donut-517, described keeping familiar cases:

“We run a small set of about 30 prompts we know inside out, stuff that broke before or needs very specific output shape.”

This is an unverified individual account of a technical workflow, not a recommended sample size for authors. Its useful idea is to retain cases with a reason to exist. For your book-club skill, that reason may be an important participation principle rather than a technical output format.

Keep failed cases after fixing them. They represent promises you now know can be lost. Also maintain some cases you have not used while rewriting the skill; passing only the examples you repeatedly edited against gives a narrower picture.

Sometimes you will decide the old criterion is wrong. Perhaps your revised book now distinguishes a voluntary social book club from a formally structured learning group with different participation agreements.

That can justify changing a test, but document the new principle first. Record which situations it covers, which old expectation changes, and why. Then update the instructions, examples, reader-facing description, and evaluation criteria together.

Do not quietly relabel a failed response as acceptable just because rewriting the skill is inconvenient. “The method changed” should point to an author decision that you can explain independently of the candidate's output.

Keep the old cases with their original labels and add revised expectations under the new method version. This lets you answer a practical future question: did this behavior change intentionally, or did it disappear during a later edit?

For a reader-facing release note, be specific: “Opening suggestions are shorter; the companion still checks the group's participation agreement before recommending a format change.” Use that wording only after your retained evidence supports it. If you changed the method itself, explain the changed advice rather than describing the release as a cosmetic improvement.

Your completed release record should connect five things: the intended improvement, the preserved principles, the exact versions, the trial evidence, and the decision to release or revise. Retain a usable previous version through the actual publishing workflow you use; do not assume recovery is available without checking it.

The guide to testing an agent skill covers the broader testing process. Your author-specific responsibility is to decide which changes still represent your teaching and which cross the line into a different method.

If you are preparing a companion for your book, bring one proposed update and a few fixed reader cases to Skillfully and choose Book onboarding. That makes the discussion concrete: what should improve, what must remain true, and what evidence you will use to tell the difference.