What Should an Author Measure After Launching a Skill?

After launching a skill based on your book, start with four questions: can readers get access, do they start the intended task, do they finish something useful, and does the offer support your business? A download count cannot answer all four.

Build a small weekly worksheet before seeking a comprehensive dashboard. Keep the numbers, their definitions, the evidence behind them, and one decision you will make from them. Add automation where your platform supports it; leave missing observations visibly unknown.

The plan below is a recommended manual dashboard. It is not a claim that any particular platform automatically supplies every metric, reads every conversation, or verifies reader outcomes.

An access claim means someone has obtained permission to use the product. A recorded retrieval means the skill’s instructions were requested or loaded. A completed task means the reader reached the result you defined. A useful outcome means that result helped them in the situation they brought.

Those are different observations. If your platform counts instruction retrievals as sessions, label them accordingly in your worksheet. A retrieval alone does not establish that the assistant followed the method, that the reader finished, or that the result was good.

Familiar analytics labels can also mislead. Google Analytics defines an engaged session through conditions such as duration, a key event, or multiple page or screen views. That definition does not by itself mean a reader successfully applied your method. Google’s engagement-rate definition.

Before copying a number into your dashboard, ask what event produces it, whether one person can produce it repeatedly, and what activity is invisible. If you cannot answer, use the platform’s literal label and mark the interpretation unresolved.

This is especially important when a reader works in an AI assistant outside the place where they purchased access. Your sales system may know the purchase happened while knowing much less about the work that followed.

Choose the task the skill promises to help with. Then describe a finished result that a reader or reviewer can recognize.

For a book about customer interviewing, “understands interviewing” is difficult to observe. A more useful first task might be: produces a five-question interview guide about a named customer problem, with questions about past behavior rather than predictions or praise.

That is still a proposed standard, not evidence of successful interviews. It gives you something concrete to inspect. The reader might finish the guide but never use it; they might use it and discover that the questions need revision. Keep those later observations separate.

Do not let the software’s easiest event define your promise. If the only thing you can currently count is an instruction load, count it honestly and collect a small amount of additional evidence about completion.

PostHog describes selecting activation events for its own products by checking their relationship with later retention. Your first task is smaller: choose a plausible sign of initial value, then see whether readers’ work and feedback support that interpretation. Do not import another product’s activation threshold as your target. PostHog’s account of defining activation.

The following definitions are proposed conventions for an author’s worksheet. Change them to fit your offer, then keep them consistent across reviews.

For completion, distinguish an inspected artifact from a self-report. Both can be useful; they are not the same evidence. A reader saying “done” is different from your checking that their interview guide actually follows the method.

For money, state the period, currency, and included deductions. The simple receipts line above is a management convention, not profit or an accounting definition of revenue. Show payment fees and operating costs separately. For subscriptions, add renewal results for customers who were actually due to renew; for one-time purchases, do not invent a renewal metric.

A cohort is simply a defined group, such as the readers who obtained access during launch week. Following that group prevents a growing total from hiding where earlier readers got stuck.

Choose an observation window that matches the task. A short interview-guide exercise might use seven days from access for its first completion check. A monthly planning method needs a different rhythm. Seven days is an illustrative choice here, not a universal benchmark.

Do not compare a reader who joined yesterday with someone who has had the full week. Mark recent readers as still in progress until their window closes. Funnel tools make the same distinction: PostHog documents excluding incomplete periods so readers who still have time to convert do not depress a trend prematurely. PostHog’s funnel documentation.

Keep paid buyers, free reviewers, gifts, and workshop participants distinguishable where you can do so appropriately. They received different offers and may have different reasons to try the skill. With a small pilot, write down that context rather than slicing the group into so many percentages that every row represents one person.

Here is an original fictional example. An author, Tessa, has launched a customer-interview companion. Its first task is the five-question interview guide described above. All numbers below are invented to demonstrate the worksheet; they are not Skillfully results or performance targets.

The group is 20 readers who obtained access in the same week. Each has now had a full seven-day observation window. Tessa’s records identify the same readers across access, retrieval, and follow-up; if they did not, these comparisons would need to stay separate.

The eight completed guides all came from readers in the 14-person retrieval group. That makes the fictional records internally consistent, but it does not explain the other six readers with a retrieval. Some may have finished without replying.

Tessa can say, “We confirmed completion for eight of 20 readers.” She should not say, “Only 40% completed,” because missing reports are not observed failures. Nor should she present eight of ten respondents as an 80% completion rate for everyone.

Her next action is narrow: review the startup questions behind the support time, improve the relevant instruction, and watch the next comparable group. She also asks the five readers who used their guide what they changed or learned. A larger activity chart would not answer that question for her.

Ask for feedback at a meaningful point, using a short question tied to the task:

• What were you trying to finish?
• Did you finish it? If not, where did you stop?
• What did you use or change afterward?

Offer a way to share a redacted example voluntarily. Do not require confidential customer material simply to obtain feedback. Record whether an answer came directly from the reader or from an automated assessment, and keep sensitive working documents out of a general marketing dashboard.

The desire for deeper feedback is not new. In a February 2025 discussion, u/AIEmergency described building educational tools with custom GPTs and wrote:

“Would love to be able to get some deeper analytics though”

That is one historical creator’s request, not evidence about today’s platform features. It captures the gap a small worksheet can help close: knowing that a tool exists or gets used is different from knowing how to improve it. Original comment.

Group replies by the problem they describe: could not start, misunderstood a question, produced a weak result, or needed help applying it. Read the actual examples before deciding that all low activity means low interest.

A sudden drop can mean readers stopped, or it can mean your measurement changed. Keep a note of platform changes, missing records, definition changes, and the skill version each group used.

In his February 2025 account of PostHog’s own activation work, Ian Vanagas reported:

“We found that the recording analyzed event wasn't when we looked.”

The surrounding passage explains that an event was not firing, making activation measurements inaccurate. This is an analytics vendor’s account of its own product, not an independent study, but it is a concrete reason to validate the recorded action before interpreting a trend. Vanagas’s original account.

For your skill, check that a known authorized practice run produces the expected record. Keep your own test activity identifiable so it does not inflate the reader group. If part of the journey is unobservable, say so rather than assigning zero.

When the evidence points to output quality, use a separate review rubric. Our guide to measuring agent skill quality covers that deeper evaluation. The weekly dashboard’s job is to tell you where to investigate, not replace inspection of the work.

If the companion exists to generate consulting inquiries, track qualified conversations and whether the reader’s completed work makes those conversations more useful. Count booked work separately from inquiries; do not attribute every later sale to the skill without evidence.

If the goal is a product that works without more consulting, track completed tasks, collected receipts, refunds, repeat use where relevant, and the support time required. A rise in personal coaching calls may be valuable for one author and a sign of an unfinished product for another.

Review the sheet weekly and leave with one action, its reason, and the evidence you expect to see next. Keep the cadence simple enough to sustain after launch excitement fades.

To plan a measurable first skill around your established method, visit Skillfully and choose Book onboarding. Bring the reader task, your completion rule, and the business result you want the product to support.