Run a Manual Trial of Your Skill Before Automating It

Before building an automated book companion, guide a suitable reader through one real task using a short written protocol. Record what the reader brings, what the protocol asks, what they produce, and every time you add help that is missing from the instructions.

This is a manual prototype: you supply some or all of the interaction while learning what the future skill would need to do. It can reveal a useful sequence, missing context, or a task your method does not handle. It cannot establish that an AI system will reproduce your judgment, that readers will succeed independently, or that the product will sell.

The aim is to leave with a better specification and a clear next test. A pleasant conversation alone does not tell you what to automate.

Tell participants that you are operating the exercise manually. If you use an AI tool during the session, explain its role and agree on what material may go into it. Do not present your own responses as evidence that a finished automated product works.

In his July 2013 essay, Paul Graham describes an early-stage approach where founders perform work they later intend to automate:

There's a more extreme variant where you don't just use your software, but are your software.

That observation comes from startup practice, not a controlled study of educational outcomes. For an author, its useful implication is modest: you can investigate the delivery sequence before investing in its automation. Read Graham's original essay.

Similarly, GOV.UK's prototyping guidance distinguishes early representations from production services and recommends choosing a prototype appropriate to the question. A manual conversation suits a question about what guidance a reader needs. It is a poor test of an interface you have not built.

Consider a fictional author, Soren, whose book teaches amateur naturalists to turn observations into clear field notes. He is considering a companion that helps readers separate what they saw from what they inferred.

He could promise a broad “nature observation coach.” For the trial, he chooses something smaller: revise one short field note so that a later reader can identify the observation, the interpretation, and the missing information.

His proposed starting example is:

The birds were angry because someone had disturbed the nest.

The exercise should help the participant ask what they actually observed. A possible revised note might describe repeated calls and short flights near a shrub, while leaving the cause unresolved. It should not invent a nest, a species, a disturbance, or a location.

This example is an illustrative design exercise. No reader trial has been conducted, and the revised wording is not a reported participant result.

Soren's test question is: can this sequence help a reader make a more inspectable note without filling gaps with guesses? That question gives him a way to judge the artifact beyond whether someone enjoyed the session.

Write the version number on the protocol and keep the first version unchanged during each session. You may depart from it to help the person; record that departure rather than pretending it was already part of the design.

The boundary matters as much as the sequence. Soren is testing note construction. A participant who asks whether a bird is injured has raised a different task; success is not confidently answering it within this exercise.

Prepare a copy of the blank output before the session. It needs only four fields: original note, revised note, uncertainty remaining, and next observation to make. The participant should be able to disagree with the revision instead of accepting the author's preferred prose.

Start by explaining the activity, approximate duration, what you will retain, and the option to stop. Ask permission before saving the participant's material or recording the session. Use an example they can share safely; detailed location information is unnecessary if it is not relevant to the exercise.

Let the participant describe the situation before you teach. Ask what they normally do with field notes and what, if anything, was difficult about this one. That establishes whether the exercise addresses a problem they recognize.

Then follow the protocol. Read or send one prompt at a time. Give the participant time to respond. When you notice uncertainty, you may clarify, demonstrate, or ask a different question: this is an assisted trial. The discipline is to preserve a record of what changed.

Do not pretend a successful manual trial must be silent. If the point is discovering the guidance needed, helping can be informative. The problem begins when assistance disappears from the account of how the reader reached the output.

Finish by asking the participant to inspect the result. Which change would they keep? Which wording misrepresents what happened? What would they do with this note after the call? Keep their disagreement alongside your own assessment.

The following log is fictional. It demonstrates how to record a session; it is not evidence that these difficulties occurred.

For each intervention, note whether it was a clarification of an existing instruction, a new instruction, domain judgment, emotional encouragement, or doing the task for the reader. Those categories point to different design work.

A repeated clarification may belong in the instructions. Expert judgment may require a narrower scope or an escalation path. Encouragement might be part of a desirable interaction, but you still need to test whether a product can offer it appropriately. Completing the task yourself proves very little about the reader's ability to do it later.

Record time spent preparing, helping, and repairing the output as well. Manual labor is part of this experiment's cost, even when you are not billing for it. You do not need a complicated dashboard; a few timestamps and a note about the work are enough for an initial accounting.

A founder describing a concierge product on Reddit, u/parker_birdseye, wrote:

I ask them only to perform 1 or 2 tasks to complete onboarding (whatever I can not answer for them)

The same post describes manually setting up accounts, invoicing, and running scripts. It is a firsthand account of their own software business, with an interest in its success, not an independent comparison. Its relevance here is the workload division: a customer may appear to finish a process because the operator has already done much of it. Read the original post.

Apply that question to your trial. Did the reader identify the missing information, or did you name it? Did they use a distinction from the book, or follow your suggestion without understanding it? Was the output usable only after your edit?

Preserve separate conclusions. You may have evidence that a participant valued the final artifact. You may also have evidence that the current instructions were insufficient. Both can be true.

If the participant paid for personal help, record that as a purchase of the offer they actually received. It does not establish willingness to pay the same amount for an automated companion. If participation was free or compensated, say so in your research summary.

At the end of a small round, sort the work into three groups:

1. Write down: repeatable questions, definitions, checks, and examples that appeared useful.
2. Investigate: uncertain explanations, a dependence on your expertise, or a mismatch between the exercise and the reader's task.
3. Keep outside scope: judgments or actions the proposed companion should not undertake.

For Soren, the next version might add a worked distinction between observation and interpretation, accept unknown details, and ask the reader to draft the revision before offering wording. Each change follows from a specific intervention in the illustrative log.

The next test should address the remaining uncertainty. Let a new reader use the written sequence with less assistance to see whether the instructions carry the work. Later, test the actual automated version on appropriate cases. A manual trial and an automated trial answer related but different questions.

Use the guide to writing an agent skill when you are ready to turn the repeatable parts into explicit instructions. Keep the intervention log as a list of assumptions to check, not a trophy proving the product is finished.

If you now have a bounded reader task and a protocol worth testing as a skill, visit Skillfully and choose Book onboarding. Bring the protocol, a safe example, and the places where your personal help was still necessary.