Self-service analytics with Claude: what Anthropic's playbook really says

In How Anthropic enables self-service data analytics with Claude, Anthropic's data team describes how 95% of business analytics queries at the company are now automated through Claude, with roughly 95% accuracy in aggregate. Stakeholders ask questions in plain language and get governed answers without waiting on an analyst, and the data team spends its time on strategic work instead of a request queue. It is the most credible account yet of self-service analytics actually working, from the company with the strongest possible incentive to make the model do all the work.

Which is exactly why the details matter. The accuracy did not come from the model. It came from a deliberate stack of curated context built around the model: canonical datasets that reduce forty plausible revenue fields to one governed source; a semantic layer that agents must consult before anything else; reference documentation written specifically for LLM retrieval; skills, which are folders of markdown instructions that route the agent to the right sources and encode how a senior analyst works; offline evaluation suites gated at roughly 90% accuracy per domain before launch; an adversarial review step that challenges every answer; and provenance footers on every response stating where the number came from and how fresh it is.

The instinct is understandable. Modern agents can write excellent SQL, your warehouse has an API, and a connection takes an afternoon. The result answers questions fluently and confidently from day one. Anthropic's post is a careful explanation of why those answers cannot be trusted, drawn from their own failures. They name three failure modes, and none of them is fixed by a better model.

1. 1

   Concept-to-entity ambiguity

   A stakeholder asks about active users. The warehouse holds hundreds of tables and thousands of columns that could plausibly answer, each embedding different assumptions: which actions count as activity, whether fraud and internal accounts are excluded, what the lookback window is. The agent has to guess which entity matches the concept, and a fluent answer built on the wrong field is indistinguishable from a correct one. This is the largest source of error Anthropic identifies, and it is a property of the data estate, not of the model.

2. 2

   Staleness

   Schemas change, definitions get revised, sources get deprecated, and the business itself moves. Whatever the agent learned about your data last month decays continuously. Any context that is written once and not maintained drifts out of truth, and the agent keeps answering confidently from the outdated version. Freshness is not a one-off documentation project. It is an operational property that has to be engineered.

3. 3

   Retrieval failure

   Sometimes the correct definition exists, is current, and is documented, and the agent simply never finds it in a search space of millions of fields, files and queries. The answer was available. It was not reachable. Anthropic's response was not better search but a smaller, curated surface: routing the agent from millions of possibilities to dozens of governed files.

The most instructive experiment in the post is an ablation. Anthropic gave the agent full search access to thousands of existing SQL files and historical queries, the corpus most teams assume would help the most, and measured the effect: accuracy moved by less than a point in either direction. The information was present and even retrieved. It did not help, because raw access does not resolve ambiguity, it multiplies it. More access is not the fix. Structure is.

There is also a bill attached to brute force. An agent investigating a single metric directly on the warehouse runs roughly ten queries: the current value, each comparison period, rolling totals, checks for outliers. Every question is a stack of live warehouse hits and context-window round trips. Anthropic's per-question verification step, an adversarial reviewer that challenges the draft answer's assumptions, bought 6 percentage points of accuracy at the cost of 32% more tokens and 72% higher latency. Accuracy earned at question time is paid for on every question, forever.

What lifted accuracy from 21% to 95% was narrowing, not widening. Anthropic's knowledge skills act as thin routers that take an agent from millions of fields to a few dozen curated reference files per domain. Runbook skills encode the workflow of a senior analyst: clarify the question, consult the semantic layer first, execute, then review the answer adversarially before returning it. Reference documentation describes each table's grain, scope and gotchas, with explicit routing rules for the cases that burn people.

Just as telling is what failed. Auto-generating metric definitions from tables and query logs produced definitions that looked plausible and were ambiguous underneath, and accuracy got worse. The conclusion Anthropic draws is that people who know the business must own the definitions, with the model drafting and pattern-matching around that human judgement, not replacing it. The post puts it plainly: data is not software. A business question has one correct answer from one correct source, and there is no compiler to prove it right. Precision has to come from the structure around the model.

This is the same conclusion the industry has been converging on from other directions. A semantic layer exists because ad-hoc SQL produced inconsistent definitions for people; agents just industrialised the inconsistency. Agentic analytics tools are fast precisely at the query step that was never the bottleneck. Anthropic's contribution is empirical: a measured demonstration, from the team best placed to prove otherwise, that the agent is only as good as the governed context it stands on.

Anthropic's playbook is genuinely reproducible, and the post is generous with the details. It is also honest about the operational commitment. The context layer they describe is not a document you write once. It is a living system with its own engineering practices, and each part exists because accuracy degraded without it.

If you have a strong data platform team with time to invest, this is a proven blueprint and you should read the post closely. Anthropic's own advice for getting started is to begin small: a handful of canonical datasets, a few dozen evals, a thin knowledge skill. But be clear-eyed that the destination is a programme. The team that built this ships evals, PR hooks, correction-harvesting agents and documentation standards as ongoing engineering work, and they are one of the best-resourced data teams in the world working on a data estate they control end to end.

The question for everyone else is not whether the pattern is right. The evidence says it is. The question is whether you build the context layer or buy it.

Canopy is KPI Tree's business context layer: the curated, governed context surface that Anthropic built by hand, delivered as a product for the metric layer of your business and served to any agent over MCP. Your semantic layer stays exactly where it is. Canopy syncs metric definitions from dbt, Looker and Snowflake semantic views, treats them as the source of calculation truth, and builds the layer above them that no warehouse table holds. Each element of the playbook has a direct counterpart.

The economics move the same direction as the accuracy. Where a warehouse-direct agent spends roughly ten queries investigating one metric, Canopy precomputes the comparison periods, rolling totals and outlier checks around every metric for every date in your history, so one call returns the full picture. And where Anthropic pays 32% more tokens and 72% more latency per question for adversarial review, Canopy runs its scepticism once a day in the layer itself: the statistical testing happens in KPI Tree's own compute engine, off the request path and off your warehouse, so answers stay fast and the warehouse bill stays flat while question volume grows.

There is one thing Canopy carries that the playbook has no counterpart for, because no amount of documentation can encode it: a causal model. Anthropic's stack makes an agent accurate about what each number is. Canopy's daily-tested driver edges make it grounded about what moves each number, with the confidence and statistical significance stated rather than narrated. That difference decides what happens after the answer.

Anthropic's post is candid about its open problem: silent failures, the plausible answer a stakeholder accepts and acts on without objection, remain incompletely mitigated even with the full stack in place. Provenance footers help a careful reader judge an answer. They do not tell anyone what to do about it.

That is because accurate answers are the entry ticket, not the destination. The moment self-service analytics works, the questions change shape: not what is churn, but what is driving it, who owns the fix, what is already being done, and did the last attempt work. Answering those requires context that no semantic layer, skill file or reference doc holds: a causal model tested against your data rather than asserted, named ownership on every metric so an insight becomes an assignment, and a verified record of which past actions actually moved the number. That layer is what AI agents need to act on business meaning, and it is the half of the problem the self-service playbook was never aimed at.

Anthropic's advice for starting is a handful of canonical datasets, a few dozen evals and a thin knowledge skill. The Canopy path compresses the same journey: connect a warehouse, sync your semantic layer, and let AI draft the metric tree from a plain-English description of the business while your team corrects it. From the moment the tree exists, the daily statistical testing takes over as the judge of every relationship in it, and every agent your organisation connects over MCP inherits the same governed, current, causal context. Most data teams have their first tree live within a day, which is roughly the time it takes to wire an agent straight to the warehouse and start collecting confident answers you cannot trust.