Build a Test Set from the Questions Readers Ask You

Build your first author-skill test set from the situations readers actually bring you. Preserve the wording and missing context that made the question difficult, remove identifying details, and write the behavior a faithful application of your method should produce.

A test case is more than a prompt with an answer. It records what the reader knows, what the skill can access, what should happen next, and which plausible response would violate your method.

Do not select only questions that let you demonstrate the book at its best. Include misunderstandings, incomplete requests, and nearby problems the method cannot resolve. Those cases expose different parts of your judgment.

An author discussing beta feedback, u/odysseyshot, wrote:

“My other beta readers aren't confused about this, and I can pinpoint the specific pages where this is explained.”

The post concerns a fictional character's motivation. It captures a familiar author dilemma: an explanation exists, yet a reader has not understood it. That observation alone neither proves the text is defective nor makes the confusion irrelevant. Original discussion.

For a methods companion, retain the point of confusion before adding the explanation you wish the reader had absorbed. Otherwise, the test may supply exactly the context the real question lacked.

Another participant, u/iwillhaveamoonbase, described their experience as a beta reader:

“I've beta read and I would definitely be uncomfortable if the author I was reading for was constantly replying to my line comments.”

That is an individual preference, not a rule against clarification. It is a reminder to distinguish collecting an independent response from helping someone revise in conversation. Tell participants which activity you are asking for. Original comment.

Review reader emails you are permitted to use, workshop questions, exercise submissions, and notes from support conversations. Keep private source records separate from the sanitized test set. Remove names, organizations, confidential figures, and unnecessary personal details.

If you cannot safely retain the original, write a synthetic case that preserves the decision structure and label it synthetic. Do not call it an anonymized real case if you have materially invented the circumstances.

Anthropic's evaluation guidance recommends starting with existing manual checks, common user tasks, and user-reported failures. It also emphasizes clear task specifications and reference solutions. That supports using your actual reader questions as raw material; it does not establish that a small set represents your entire audience. Anthropic evaluation guidance.

Group similar questions by the decision they require. Five phrasings of the same complete request may cover less than one complete request, one missing-input version, and one exception.

Keep the expected behavior separate from the input supplied to the skill. A test that tells the agent the desired conclusion measures something different from a reader asking the original question.

Do not require exact phrasing unless the wording itself matters. A faithful answer may be shorter or organized differently from your reference. Specify the necessary decisions and boundaries.

Imagine a book that helps volunteer coordinators prepare role handovers. Its rule is to distinguish confirmed responsibilities from proposed ones and identify missing access before treating the handover as ready.

The three cases below are entirely fictional. They are templates for an author to adapt, not anonymized customer records or results from model runs.

For H-02, an authored reference response could begin: “Has Sam agreed to take over, and can they access the newsletter materials? Until those are confirmed, I can help draft a proposal rather than a completed handover.” This is expected behavior written for the case, not a tested response.

The decisive distinction is status. A proposed successor is not a confirmed owner. Retain that distinction even if you change names and surface details.

Label what each case covers: routine execution, missing input, known exception, conflicting information, or scope boundary. Use those labels to find gaps, not to claim every category has equal prevalence.

Keep an observed frequency only if you have records supporting it. A memorable complaint is not necessarily common. A rare but serious boundary case may still deserve inclusion.

Reserve some cases you did not use while rewriting the instructions. That gives you a separate check against tuning the skill only to familiar examples. It is still a limited test set, not proof of general performance.

When reviewers disagree about the expected answer, resolve the method question first. Do not score the skill against a hidden preference that the author has not made explicit.

Run the cases in the intended environment, saving the full interaction and the exact skill version. Record whether the result met the required behavior, not merely whether it sounded helpful.

If a failure reveals a new rule, add the rule and rerun affected cases. If the test itself omitted a fact needed for a fair verdict, revise the case and keep a note explaining why. Do not silently alter expectations until an attractive response passes.

The agent skill testing guide covers the execution process. This resource supplies the reader-specific material that makes those checks meaningful.

Bring a small set of sanitized or clearly labeled synthetic cases to Skillfully and choose Book onboarding. Include at least one request where the right answer is a question or a boundary, so the discussion can examine the judgment behind your method.