Agent Skill for QA: How to Make Testing Workflows Repeatable

An agent skill for QA is a reusable testing procedure that tells an AI agent what to inspect, which risks matter, how to interact with the product, what evidence to collect, and when to stop. It turns "test this app" into a repeatable release, regression, or exploratory QA workflow.

That makes a QA skill different from a one-off prompt. A prompt can ask for a second opinion on a screen. A skill should encode the operating procedure: the target environment, test data, changed behavior, required checks, bug report format, severity rules, and evidence standard.

The format fits QA because QA is procedural. OpenAI describes Codex skills as packages of instructions, resources, and optional scripts that help Codex follow reusable workflows, with only the skill metadata loaded at first and the full SKILL.md loaded when selected (OpenAI Codex skills). That progressive-disclosure model is useful for QA: the short description can trigger on "release smoke test" or "browser QA pass," while the full skill can contain the actual checklist, reporting template, and scripts.

If you are still deciding whether this should be a skill at all, start with What Is an Agent Skill?. The short version is simple: use a skill when the same QA job should run again, be improved later, and produce evidence another person can inspect.

The practical question behind a QA skill is:

A strong skill covers the whole workflow, not just the instruction text: app access, release scope, test data, browser automation, screenshots or traces, console and network evidence, accessibility checks, and a rule for whether the release is blocked. It also defines what happens when authentication blocks the run, selectors drift, data is missing, a visual diff is noisy, or the issue only reproduces on a specific viewport.

The first job of a QA skill is to narrow the run to the risks that matter.

Most weak QA prompts say "find bugs" or "test the app." That sounds comprehensive, but it gives the agent no basis for prioritization. A good QA skill starts with a release note, pull request summary, changed files, incident history, or product area. Then it maps that scope to the right test mode.

QA mode	Best trigger	What the skill should verify	Evidence to return
Release smoke test	Before deploy or customer demo	Critical paths still work	Pass/fail table, commands, screenshots for failures
Targeted regression	After a specific code change	Changed behavior and adjacent flows	Changed paths tested, old behavior checked, blockers
Exploratory browser QA	New or risky UI flow	Usability breaks, data loss, broken states	Repro steps, screenshots, severity, environment
Accessibility pass	New page, form, modal, or navigation	Keyboard flow, labels, contrast, focus order, automated checks	Axe output, keyboard notes, ARIA/tree observations
Visual regression pass	Design system, layout, or CSS change	Important screens did not visually drift	Baseline, actual, diff, viewport and OS
Content QA	Docs, blog, pricing, onboarding	Broken links, metadata, copy defects, page rendering	URLs checked, defects, suggested fixes

For example, a checkout change should test cart editing, payment state, validation errors, abandoned checkout recovery, and permission boundaries. An authentication change should test login, logout, expired sessions, role boundaries, and deep links. A design-system change should test representative components across viewports, because the likely failure is not always functional.

This is the same operating principle behind How to Test an Agent Skill Before You Publish It: test the workflow against realistic happy paths and blocker paths, then inspect evidence. A QA skill is just that idea applied to product testing.

A QA skill should include a contract that is explicit enough for a new teammate to run without extra explanation.

Use this structure:

Contract field	What to specify	Why it matters
Target	URL, branch, preview build, feature flag, app surface, browser set	Prevents the agent from testing the wrong environment
Release scope	PR summary, changed files, user story, migration note, known risk	Keeps testing tied to what changed
Test data	Accounts, roles, fixture IDs, safe records, blocked credentials	Avoids destructive improvisation
Allowed actions	Read-only, create test data, submit forms, call APIs, use payment sandbox	Makes side effects explicit
Test plan	Critical path, regression path, negative case, viewport, accessibility pass	Turns broad QA into ordered work
Evidence	Screenshot, trace, console errors, network failures, command output, exact prompt	Makes findings reviewable
Severity rule	Blocker, major, minor, polish, not-a-bug	Keeps triage consistent
Exit rule	Missing URL, missing credentials, unsafe production data, blocked auth, unclear scope	Stops the agent before it guesses

The exit rule is what keeps an agent from inventing test data, clicking around production, or reporting speculative bugs. A useful QA skill should say: "If the build URL, account, or release scope is missing, stop and ask for it." If you are drafting the first SKILL.md, the structure in How to Write an Agent Skill That Actually Works is a good companion to this QA-specific contract.

Here is a compact QA skill skeleton for a browser release gate:

---
name: browser-release-smoke-test
description: Run a browser smoke test for a web release. Use when given a build URL, release scope, and safe test account. Do not use for full exploratory QA.
---

## Inputs required

- Build URL or preview URL
- Release scope or PR summary
- Test account role and any required fixture IDs
- Allowed browser and viewport list

Stop if the build URL, release scope, or safe test account is missing.

## Test order

1. Read the release scope and identify changed product areas.
2. Run core login/navigation checks.
3. Test the changed flow on the primary viewport.
4. Test one adjacent regression path.
5. Check console errors and failed network requests.
6. Capture screenshots for every failure.
7. Return a release gate decision.

This is narrow on purpose. A weekly exploratory pass or accessibility audit should be a different skill. A skill that tries to cover every QA mode usually becomes too vague to trust.

Browser automation gives a QA agent hands. The skill gives it judgment. Playwright is the obvious source to design around because it is already used for browser automation, end-to-end tests, accessibility checks, screenshots, traces, and CI. The Playwright MCP docs are especially relevant because Playwright MCP lets LLMs interact with pages through structured accessibility snapshots rather than relying only on screenshots.

That matters for agent skills. A skill can tell the agent to prefer accessible names, roles, and visible text when navigating. That is usually more stable than saying "click the blue button in the top right."

Community discussions show the tradeoff clearly. In one Reddit thread about Playwright test automation with AI, practitioners described useful patterns such as pointing the agent at user stories and existing fixtures, but also raised real blockers: complex pages, hallucinated steps, corporate SSO, and staging-session limits (r/softwaretesting discussion). That language belongs in the skill: how to use saved auth state, when to ask for a bypass account, and when to stop.

A QA skill is only as good as the evidence it returns.

For browser QA, the minimum evidence pack is the target URL, environment, account role or fixture, steps taken, expected result, actual result, screenshot for UI failures, console errors, failed network requests, trace or report link when available, and release gate decision. The skill should also say not to leak secrets when naming accounts or fixtures.

Playwright's Trace Viewer is built for this kind of debugging: it lets teams inspect recorded traces after a run, and Playwright calls traces useful for debugging CI failures. A QA skill should preserve the artifact that lets a developer see the browser state, actions, and failure point.

The evidence standard should be proportional. A typo may need a URL and suggested copy. A checkout failure needs repro steps, screenshot, console/network details, account role, and whether the failure blocks release. A flaky visual diff needs actual, expected, diff, viewport, browser, OS, and whether the baseline was generated in the same environment.

Visual and accessibility testing are both valuable, but both can create false confidence if the skill treats tool output as the whole answer.

Playwright's visual comparison docs explain that toHaveScreenshot() generates reference screenshots and compares future runs against them, while warning that rendering can vary by OS, hardware, browser, headless mode, and other factors. A QA skill should define when visual diffs are blockers and when they are review items. A missing submit button is a blocker. A one-pixel font rendering difference on a different runner is probably not.

Accessibility has a similar boundary. Playwright's accessibility testing guide shows how to use axe with Playwright to catch common accessibility issues. Deque's axe-core README says automated tests can find an average of 57% of WCAG issues and marks uncertain items as incomplete for manual review (axe-core). That is enough to justify automation, but not enough to replace judgment.

A strong accessibility QA skill should combine automated axe scans, keyboard-only navigation, visible focus checks, form label checks, modal escape/focus-trap behavior, and ARIA or accessibility-tree checks where structure matters.

Playwright's ARIA snapshot docs are useful for structural regression checks because they compare the accessible tree, not just pixels. Use them for stable components such as navigation, menus, forms, or dialogs where role/name structure matters.

A QA skill can run manually, but the best repeatable checks eventually become CI release gates.

Playwright's CI setup guide describes running tests on push and pull request workflows and viewing logs, HTML reports, and traces. That is the natural home for deterministic checks: smoke tests, link checks, accessibility scans, and focused visual baselines. The agent skill should explain which checks are mandatory, which are advisory, and what artifact to attach when a gate fails.

A practical release gate has three outcomes:

Outcome	Meaning	Required next step
Pass	No blocker or major issue found in scoped checks	Release can proceed for this scope
Pass with notes	Minor issues or advisory visual/accessibility findings	Release owner decides, issues tracked
Block	Critical path broken, data loss, auth failure, payment failure, severe accessibility failure, or no reliable test evidence	Fix before release or get explicit override

The "no reliable test evidence" case matters. If the agent could not log in, access the preview URL, or collect artifacts, the result is not a pass. It is an inconclusive run.

The output format should make bug reports useful to developers.

Mozilla's bug writing guidelines are a good primary source here. They emphasize a clear summary, precise steps to reproduce, expected results, actual results, and separating observations from speculation (Mozilla bug writing guidelines). That maps directly to agent QA.

Use this bug template inside the skill:

### Bug: <specific failure in about 10 words>

Severity: blocker | major | minor | polish
Environment: <URL, browser, viewport, account role, build/commit if known>

Steps to reproduce:
1. ...
2. ...
3. ...

Expected result:
...

Actual result:
...

Evidence:
- Screenshot:
- Trace/report:
- Console:
- Network:

Notes:
- Facts observed:
- Hypothesis, if any:

Do not let the agent collapse this into "the checkout is broken." The value is in the reproduction path and evidence. If the issue cannot be reproduced, the report should say that and include what was tried.

OpenAI's skill structure allows optional references/, scripts/, and assets/ alongside SKILL.md. Use those folders instead of cramming everything into prose. Useful files include:

• references/release-risk-map.md for product areas, critical paths, and known fragile flows
• references/severity-rubric.md for blocker, major, minor, and polish definitions
• references/test-accounts.md with safe account roles and instructions for requesting access, not raw secrets
• references/bug-report-template.md for consistent output
• scripts/check-links.ts for deterministic link checks
• scripts/smoke.spec.ts or scripts/smoke.ts for repeatable browser checks
• scripts/a11y.spec.ts for axe or accessibility-tree checks

Put deterministic checks in scripts, durable product knowledge in references, and keep SKILL.md focused on when to run the workflow and how to decide.

Before publishing or sharing a QA skill, test the skill like a workflow. The process in How to Test an Agent Skill Before You Publish It applies directly:

1. Run a happy-path release smoke test against a safe preview build.
2. Remove a required input, such as the build URL or test account, and verify the skill stops.
3. Give it a scoped regression task and check whether it stays in scope.
4. Seed one known UI or accessibility bug and verify the report includes evidence.
5. Compare two runs and check whether the severity decisions are consistent.

OpenAI's guide to testing agent skills systematically with evals frames the same problem as checking whether the model output and the steps taken match what you intended. For an early QA skill, you do not need a large eval harness. You do need saved example tasks, expected artifacts, and a way to tell whether the skill got better or just changed.

Before you share the skill, check that it can answer these questions:

• What type of QA is this: smoke, regression, exploratory, accessibility, visual, or content?
• What release scope, target URL, account role, and test data does it require?
• Which browser automation tool should it use, and what should it do if access is blocked?
• Which actions are allowed, and which environments are off limits?
• What evidence must be attached to each failure?
• What makes the run pass, pass with notes, block, or remain inconclusive?
• Which deterministic checks belong in scripts?

Use a prompt when you want a quick read on a screenshot, a one-off copy pass, or a single question about a bug. Use a QA skill when the same test procedure needs to run again, especially when release decisions depend on it. A good QA skill makes testing less dependent on the original author's memory and more dependent on comparable evidence.