Agent Skill for Data Analysis: From Ad Hoc Questions to Repeatable Workflows

An agent skill for data analysis is a reusable analytical workflow. It tells an agent how to answer a recurring data question, which sources to trust, which checks to run, how to handle uncertainty, and what evidence to return.

That is different from asking a model to "analyze this data." A model can summarize a CSV, draft SQL, or explain a chart. A skill packages the habits of a careful analyst: define the metric, verify the grain, check freshness, inspect joins, compare against a baseline, and separate observed facts from inferred causes.

OpenAI's Codex skill docs describe skills as reusable packages with instructions plus optional scripts and references (OpenAI Codex skills). That shape is useful for analytics because the hard part is rarely one query. The hard part is remembering the exact metric definition, the trusted source order, the dashboard that should be reconciled, and the checks that prevent a confident wrong answer.

If you are still deciding what belongs in the skill versus a tool or script, read Agent Skill vs Tool and Agent Skill vs Code. A data-analysis skill should not pretend to be a database connector. It should tell the agent how to use the connector, when to stop, and what proof is required before anyone acts on the result.

Good candidates are recurring analytical jobs where the question changes slightly but the method should stay consistent.

The common thread is repeatability. If a stakeholder will ask the same question next week with a different date range, cohort, market, or feature flag, it deserves more than an improvised prompt.

That does not mean every chart needs a skill. A one-off spreadsheet cleanup can stay a prompt. A durable weekly growth readout should become a skill because the output affects decisions and the checks should not depend on which agent happened to run the analysis.

A strong data-analysis skill should require a contract before the first query runs:

• Question and decision: what decision the analysis supports, not just the chart requested.
• Metric definition: numerator, denominator, window, filters, and excluded cases.
• Trusted source order: semantic layer, warehouse model, BI dashboard, spreadsheet, API, or notebook.
• Grain: user, account, event, session, invoice, experiment assignment, or another unit of analysis.
• Freshness and coverage: data delay, backfills, missing partitions, late-arriving events, and sample size.
• Quality checks: nulls, duplicates, schema drift, join fanout, date ranges, and outlier handling.
• Comparison plan: prior period, rolling average, target, cohort, control group, or business threshold.
• Output standard: tables, charts, caveats, source links, and recommended next checks.

This contract is where the skill earns its keep. Without it, the agent may answer the surface question while skipping the thing an analyst would notice: a dashboard filter changed, a denominator moved, the latest partition is incomplete, or a join multiplied rows.

For skill authors, this is the same discipline covered in How to Write an Agent Skill That Actually Works: name one repeatable job, define the trusted path, and require visible evidence.

It also gives reviewers a concrete failure checklist. Recent analytics-community discussions about AI data work tend to converge on the same risk: the model may not invent an entire dataset, but it can choose the wrong column, apply the wrong filter, double-count through a bad join, or explain a number it calculated incorrectly (Reddit discussion). A skill should make those subtle errors inspectable.

Data-analysis agents are most useful when they do not have to rediscover business logic from raw tables. A semantic layer gives the agent a governed place to find metric definitions, dimensions, entities, joins, and access rules.

The dbt Semantic Layer, powered by MetricFlow, is designed to centralize business metric definitions so downstream tools use consistent calculations (dbt Semantic Layer). Cube describes the same broader pattern as a shared semantic layer for BI tools, applications, and AI agents, centralizing metrics, joins, access rules, and caching (Cube documentation).

That changes the shape of the skill. Instead of telling the agent "write SQL against whatever table looks right," a good metric skill says:

1. Check whether the metric exists in the semantic layer.
2. Use the semantic metric definition before raw SQL.
3. If raw SQL is necessary, cite the model, grain, joins, and differences from the semantic definition.
4. Reconcile the result against the canonical dashboard or metric endpoint.
5. Report any mismatch before interpreting the movement.

This is especially important for metrics such as activation, churn, retained revenue, and conversion rate. Their names sound simple, but the denominator, time window, exclusions, and account hierarchy often carry the real meaning.

There is a tradeoff. Semantic layers reduce ambiguity, but they do not remove the need for review. Teams still debate whether the semantic layer is complete, whether every BI tool uses it, and whether analysts bypass it for speed. That skepticism shows up in data-engineering community discussions about metric layers and semantic-layer adoption (Reddit discussion). A practical skill should assume the semantic layer is the first place to look, not magic. If the required metric is missing or stale, the skill should say so rather than invent a definition.

SQL agents are tempting because they can move quickly from question to query. They are also risky because a plausible query can be wrong in quiet ways.

LangChain's SQL agent documentation shows a practical pattern: list tables, inspect schemas, run a query checker, execute the query, and retry when the database returns an error (LangChain SQL agent). That is a useful tool sequence. A skill should add the analytical sequence around it.

For example, a revenue diagnostic skill should not let the agent start with a broad orders table query just because the table name looks relevant. It should require the agent to confirm whether revenue comes from invoices, recognized revenue, payment events, or subscription ledger entries. It should require the agent to identify refunds, trials, internal accounts, currency conversion, and account merges before claiming a driver.

The same principle applies to MCP tools. MCP tools let servers expose database queries, APIs, and computations to models (MCP tools), while MCP resources can expose context such as schemas or files (MCP resources). The tool supplies access. The skill supplies method, source order, approvals, and final evidence.

Pandas and DataFrame agents are a natural fit for exploratory analysis. They can profile a file, compute distributions, group by segment, and draft charts. They are not automatically safe.

LangChain's Pandas DataFrame integration notes that the agent works by executing LLM-generated Python code and requires caution because arbitrary code execution is a security risk (LangChain Pandas DataFrame agent). That should shape the skill. A local CSV exploration skill might allow read-only DataFrame operations in a sandbox. A production customer-data skill should require a governed query tool, access controls, and redaction rules.

A good DataFrame skill should specify:

• allowed file locations and data classifications
• whether generated code may execute
• whether network access is disabled
• maximum rows or sampling rules
• required profiling checks before modeling
• output limits so sensitive rows are not pasted into the final answer

That sounds operational, but it is editorial too. If the final answer only says "segment B declined," the reviewer cannot tell whether the agent checked missingness, outliers, or sample size. The skill should require a small evidence table with denominators and a caveat when sample size is too low.

Data analysis work happens across notebooks, dashboards, SQL editors, and production scripts. A single broad "data analyst" skill usually becomes too vague. Split the skill by workflow.

A notebook exploration skill should preserve the sequence of exploration: load data, profile columns, check grain, inspect distributions, run focused comparisons, and summarize findings with the executed cells or saved notebook path. Hex, for example, combines SQL cells, Python cells, no-code cells, and app-style outputs in one workspace, which makes it a natural environment for exploratory analysis that needs a shareable narrative (Hex docs). Community data-science discussions regularly point out that notebooks are strong for EDA but weaker as production artifacts; one r/datascience thread framed notebooks as appropriate for exploratory analysis or one-time reports, with production-ready code moving into scripts (Reddit discussion).

A dashboard QA skill should compare dashboard tiles against source queries, inspect filters, check stale extracts, and verify that charts use the intended metric grain. Metabase separates questions, models, metrics, and dashboards; a QA skill should know which object is the canonical source before judging a tile (Metabase docs). It should not rewrite the dashboard unless the user asks. Its final output should be a pass/fail table by tile, with the exact source query or semantic metric used for reconciliation.

A published data-app skill should care about reproducibility and presentation. Observable Framework, for example, treats data apps as code projects with pages, data loaders, components, and build output (Observable Framework docs). A skill for this environment should check both sides: whether the analysis is correct and whether the published view is refreshed, readable, and backed by committed source.

A production metric skill should prefer code and tests. Data quality tools such as Great Expectations describe expectations as verifiable assertions about data, useful for making assumptions explicit and validating that data meets test criteria (Great Expectations overview). If the same null, uniqueness, schema, or relationship check matters every day, put it in a test or validation job and let the skill require the agent to run or inspect that job.

---
name: metric-movement-diagnostic
description: Diagnose a movement in a product metric using trusted analytics sources. Use when given a metric, date range, and comparison period. Do not use for one-off chart formatting.
---

Confirm the decision the analysis supports.
Find the metric definition in the semantic layer or metric reference.
Stop if the definition, date range, or trusted source is missing.

Before explaining causes, check:
1. data freshness and missing partitions
2. numerator, denominator, and sample size
3. segment mix by plan, market, channel, and platform
4. funnel-step movement if the metric is conversion-based
5. instrumentation, schema, or dashboard-filter changes

Return:
- executive summary
- metric-definition block
- driver table with evidence, confidence, and caveats
- queries or dashboard links used
- recommended next verification

This skill is useful because it forces the agent to behave like an analyst instead of a narrator. It also makes review easier: a teammate can inspect whether the skill checked freshness, denominators, and instrumentation before accepting the conclusion.

Some data skills produce prose for a human. Others produce output that feeds a dashboard, ticket, notebook, or analytics log. Once another system consumes the result, the final answer needs a schema.

OpenAI's Structured Outputs guide describes model responses that adhere to a supplied JSON Schema (OpenAI Structured Outputs). For a data-analysis skill, that might mean returning fields such as metric_name, date_range, baseline_range, absolute_delta, relative_delta, driver_candidates, confidence, data_quality_warnings, and source_links.

The skill should still explain how to think. The schema should enforce the shape. Application code can reject missing fields, and evals can catch regressions when the model, skill, or source data changes. OpenAI's agent workflow evaluation docs describe using traces, graders, datasets, and evaluation runs to improve agent quality (OpenAI agent evals). That is directly relevant to analytics skills because a good-looking narrative can still be unsupported.

For analytics, useful eval cases are concrete. Give the agent a fixture where the latest partition is missing, a join creates fanout, the dashboard filter excludes trials, or the semantic-layer metric disagrees with raw SQL. The expected answer should reward stopping, surfacing the mismatch, and asking for the right source instead of producing a polished but unsupported explanation. Measuring Agent Skill Quality covers that skill-improvement loop more generally.

• Does the skill define the metric before analysis begins?
• Does it name the trusted source order?
• Does it require grain, denominator, and sample-size reporting?
• Does it include freshness, missingness, duplicate, and join-fanout checks?
• Does it distinguish observed facts from inferred causes?
• Does it tell the agent when to use SQL, a semantic layer, a notebook, or a dashboard?
• Does it forbid unsafe code execution or unrestricted customer-data exposure?
• Does it require source links, query snippets, or dashboard references?
• Does it define the final output shape?
• Does it include eval cases or review fixtures for known failure modes?

Use a prompt for a quick chart explanation, a disposable calculation, or a one-off spreadsheet cleanup. Use an agent skill when the workflow affects decisions, repeats across teams, or needs consistent quality control.

The best data-analysis skills are not "AI analyst" fantasies. They are narrower operating procedures: metric diagnostics, dashboard QA, cohort reads, experiment readouts, and data-quality gates that a team wants to run consistently.

For Skillfully's audience, the durable value is improvement over time. A skill author can watch where the agent missed a denominator, trusted the wrong dashboard, skipped a data-quality check, or returned a weak caveat. Then the author can revise the skill, add a script, tighten the source order, or add an eval. That feedback loop is the difference between a clever demo and a workflow a team can trust.