Five steps, and the hard one

Every question an analytics agent answers goes through the same five steps:

1. Intent. Read the question as asked.
2. Context. Work out what it means, and gather every rule that governs it. The open problem.
3. SQL. Write the query, probing and rewriting as results come back. Easy once the rules are known.
4. Execute. Run it against the warehouse. Solved.
5. Insight. Narrate the result. Comes later, and needs good context too.

When the agent lacks a rule at step two, it guesses one your company never wrote down: that fiscal Q3 starts in January, that early stages never count toward pipeline, that test accounts are out of every number. Each guess is a fair reading of the schema, the SQL runs clean, and nobody sees the miss until the number reaches the board. The rest of this chapter shows one such miss, why a better model doesn’t fix it, and the four problems that follow.

The tenth question: a correct query, a wrong answer

Take one question: “What’s our Q3 sales pipeline?” Any good analyst at this company, whose fiscal year ends 30 June, would write this:

SELECT SUM(o.amount)
FROM opportunity o
JOIN account a ON a.id = o.account_id
WHERE o.close_date BETWEEN DATE '2026-01-01' AND DATE '2026-03-31'  -- fiscal Q3
  AND o.stage NOT IN ('Closed Won', 'Closed Lost')
  AND a.is_internal = FALSE AND a.is_test = FALSE;  -- internal and test accounts are out

It returns $18.4M, and it is wrong. The analyst got the quarter right, because they knew the fiscal calendar, and left out internal and test accounts. They missed a rule nobody told them. At this company the earliest stages never count toward pipeline; Finance treats them as noise and leaves them out of every pipeline number. Add one line and the answer changes:

  AND o.stage_order >= 2   -- early stages never count toward pipeline

Now it returns $11.2M, the number Finance plans on.

The first query has no bug. The rule it misses lives in no table, no column and no description. The company made that decision once and never wrote it down anywhere an agent could reach. A syntax error announces itself. A missing rule produces clean SQL and a plausible number, and a reviewer who didn’t know the rule would have written the same query.

The demo never meets this question, because the demo asks what the schema can answer. Real users ask what the business needs, and the hard part of their questions is never in the schema.

What the question means: the intent problem

The stage rule is one of five decisions hiding in that short question.

The agent has to decide which stages count, and which rows are always out, such as internal and test accounts. It has to pick the quarter, fiscal or calendar. It has to find the right amount on the right table, because several look alike. And it has to assemble the rows correctly: deduplicated, joined to the account, as of the right date.

The schema states none of these. An amount column doesn’t say it is the wrong amount.

Now suppose a strong model guesses each decision right 80% of the time, which is generous for a rule that appears nowhere in the data. All five land about one time in three. That is our arithmetic, not a measurement, but it sits close to what we measured on bare schema: 31.8%.

A smarter model raises the 80% a little. It can’t reach 100%, because no model can infer a rule nobody wrote down. Meanwhile the exponent grows with the question: a second table or a comparison period adds hidden decisions. Demo questions hide few. Real questions hide more, and the score falls to match.

If the problem were the model, a better model would fix it. So we ran the same questions on bare schema across five models, and on three models with past SQL as context. Two models ran both ways.

Switching models moved the bare-schema score by about 10 points. Adding context moved both paired models by 26 points or more, and the best model on bare schema still lost to the weakest model with context. Use next year’s model when it arrives; it will guess better. It still won’t know that your Finance team drops the earliest stages from pipeline, because that rule is in no data it can read. Pick a model that finishes its tasks, then put your effort into context. Written down, owned and versioned, it carries over to the next model. A model swap carries nothing over. On our warehouse, it was not a model problem. It was a context problem.

Not every kind of context closes the gap. Descriptions and documentation barely moved the score; metrics built from the past queries your analysts ran moved it most. The method at a glance shows the ladder, and What teams try first shows what didn’t work.

Deciding the same way every time: the format problem

Once a rule is written down, the form it takes decides whether the agent follows it. Written as a sentence, “leave out the earliest stages” still has to be turned into SQL by the agent, on every question, and every translation is another guess. Written as SQL, the decision is made once and runs the same way each time. On multi-table questions, the same metric served as SQL beat the same metric written as a sentence by 16 points: SQL, not a sentence shows both forms.

Most teams build the prose forms first: descriptions, glossaries, documentation. We did too. None of it gave the agent a decision it lacked.

Knowing you are right: the confidence problem

In software, QA decides what ships. For an agent, the eval does. Without one, “it works” is an opinion held by whoever built it. An agent with no eval is a demo: it answers the question it was shown, and nobody can say whether it will answer the next one.

Without a fixed test, every claim gets answered by impression. A sponsor sees three good answers in a demo and believes the agent works. A builder changes a prompt, tries two questions, and believes it improved. Both are guessing, and the agent is usually being tested on questions picked because it already gets them right. One run doesn’t settle it either: the same setup run twice gives different scores, so a single run proves little.

Four problems, and where the book fixes them

Read more The method at a glance →

What closes the gap

We took real questions our analysts had asked, sealed the correct answer to each one, and ran the same questions again, each time adding one kind of context to the bare schema.

Bare schema carries none of the decisions, and scored 31.8%. Descriptions and documentation reached 39.3%, a few points for about seven times the context. They say what a table holds, and the decisions aren’t descriptions. Usage patterns mined from the query log, such as the filters and joins analysts actually use, served with the descriptions, reached 43.9% on an earlier version of the answer keys. The big jump came from metrics served together with trusted past queries, all generated by our systems from the query log: 83.2% on the configuration we ship. An analyst who ran the right query had already made the right decisions, and a metric seals them so they hold every time.

The rest of this page is how you build that last bar for your own domain: metrics generated from the SQL your analysts already run, tested against a golden set, one domain at a time.

The lifecycle

One analyst can get one agent through one set of questions with a long week of prompt edits. The next quarter, the next model or the next domain undoes most of it. Teams that ship software solved this problem years ago: write the tests first, build, debug against the tests, release behind a gate, operate. An analytics agent needs the same loop, with a golden set of real questions playing the part of the test suite.

Some teams spend a month on this loop and some do it in a week. The phases are an order, not a calendar.

Eight rules

The loop runs on eight rules. Rules 1, 2, 3 and 5 rest on our benchmark, and each links to its evidence. Rule 4 and the last three are what building and running the method taught us.

1. Fix the context before you change the model. Switching models moved our score far less than adding context: the model chart.
2. Write each business rule into a metric’s SQL. The past queries your analysts already ran carry the rules, and a sentence leaves the agent to translate them: SQL, not a sentence.
3. Build the golden set before any context. Without sealed answers, you can’t tell a fix from luck: Evals first.
4. Settle conflicts before the agent sees them: provenance first, then the owner. The definition most analysts ran recently, and that matches the warehouse, usually wins. A definition kept only in a spreadsheet leaves no query history, so its owners settle it. Never let the agent pick.
5. Choose the lane by the use: exploratory or governed. Exploratory answers suit questions where a person reads the SQL before acting on it. A number that goes on a board slide needs the governed lane: a metric that gives the same answer every time, or no answer: Two lanes.
6. Start with one domain and the people who own its numbers. One domain is small enough to finish, and its owners settle what provenance can’t.
7. Read the raw trace of every miss, and feed real usage back. The summary hides the step where the agent guessed, and context left alone goes stale within a quarter.
8. Make the agent reliable on what happened before you ask it why. “What was pipeline last quarter?” needs metrics. “Why did it drop?” needs a model of how the business works on top of them, and it is only worth building once the numbers hold.

The gates

Each phase of the lifecycle ends at a gate. Move on when the gate is met, whatever the date.

Phase 0 runs before the clock: the clock is the build, and it starts with Phase 1, once Phase 0’s gate is met. Phase 3 is the hill climb: run the golden set, read the trace of each miss, make the smallest fix and run it again. Phase 4 releases the agent and hands it over; What you’ll walk away with lists what your team holds at the end.

Before Phase 1 you need read-only warehouse access, query history and BI reports. You also need your agent harness set up to record a trace of every run, so that your first test runs are traced. Last, you need named owners with hours: What you need before you start.

Find your task

Each stage breaks into numbered steps. Pick a stage, then a step, to see how it is done, who does it, the template to copy and the mistake to avoid. Next step walks you through the whole method in order.

The running example

The book follows one engagement at a large B2B company. The domain is sales, and three questions carry the story: open pipeline with the top 100 customers, how many of those deals slipped, and why pipeline dropped in EMEA. Boxes marked In practice follow it through each phase. It is a composite built to show the method, so its counts are examples, not measurements. The Q3 pipeline query in Why it is hard is a separate example, so its $11.2M does not reconcile with the story’s $412M.

Cards are one-page checklists you can print. Each phase chapter ends with the card for its job, and all of them are collected in Cards.

One row of the golden set

Each row has six required fields. Three of them do the core work, and each has one job:

What makes an eval worth trusting

Four properties. Miss one and the score stops meaning what it says.

• Representative. Take real questions from the board pack, the dashboards and analyst requests, in the business’s words, and keep the hard half. On bare schema, our questions that one table answers scored 64%, and the ones that need a join scored 11.5%. With the metrics layer, they scored 93% and 82%. A set of easy questions hides most of the difference between setups.
• Not dominated by one fact. No single rule should be able to swing the headline. In our set, four questions turned on the same missing exclusion rule. Know which questions share a decision, so one fix isn’t read as broad progress.
• Live. Questions grow out of real use, and the ones that broke before stay in. Anthropic’s guidance for agent evals is to build tasks from real failures and read the transcripts, not just the pass rate. Answer keys have errors too: experts re-checking two public benchmarks found errors in more than half the samples they read. Have two graders check every key before it is sealed, and correct a wrong key when you find it.
• Sealed. Each answer is computed by running the correct SQL and checked against the source the business trusts. Keep the answers away from whoever builds the context and from anything the agent can reach. If any part of the pipeline can see the answer, the agent will find it.

Grade the answer, in layers

Two different queries can both be right. LinkedIn found about 60% of its questions had more than one valid answer. So grade the answer, not the SQL text, and grade it as a stack of checks:

1. The value. Code compares the answer with the sealed key, within a tolerance.
2. The logic. A grader checks that the SQL makes the item’s decisions. A wrong query can land near the right number by luck. Read the trace to see why an answer was right or wrong.
3. The shape. A judge checks that the answer is the kind of thing asked for: a number, a list, a date.
4. Leaks. Check that no prompt carried any part of the answer.

Count refusals apart from wrong answers, and count both as misses. An agent that refuses often looks careful and still fails the business.

Evals are the agent’s CI/CD

An agent changes every time its context changes: a new metric, an edited rule, a renamed column, a new model. Each change can fix one answer and break another. Rerun the golden set on every change, before the change reaches users, and keep the change only if nothing that passed has broken. That is what lets a team keep improving the agent after launch without being afraid to touch it.

The eval’s real output is a work queue. Never patch the context for one question. Read each miss as one of three things: a definition never written, one written but never served, or one served and ignored. Phase 3 works through that list.

Read more Phase 1: build the golden set →

SQL, not a sentence

The same rule can reach the agent in two forms. As a sentence:

Open pipeline is the value of new and upsell deals that are still open, leaving out the earliest stages and internal or test accounts, taken at the end of the quarter.

And as SQL, where each line settles one part of that sentence:

-- taken at the end of the quarter
WHERE o.as_of_date = :quarter_end
-- leaving out the earliest stages
  AND o.stage_order >= 2
-- new and upsell deals that are still open
  AND o.type IN ('New Business', 'Upsell')
  AND o.is_closed = FALSE
-- no internal or test accounts
  AND a.is_internal = FALSE
  AND a.is_test = FALSE

On single-table questions the two forms tied. The more decisions a question carries, the more a sentence leaves the agent to translate, and every translation is another guess.

Where does each kind of knowledge go?

Find the knowledge you are holding in the first column. Put it where the second column says.

Two rules settle most ties. If a fact could go in two places, use the one that can run: SQL beats a sentence. A rule that touches one metric goes inside that metric; a rule that touches many is served as a rule and also written into the SQL of every metric it touches.

Provenance is what makes a past query trustworthy. In the engagement, the slipped-deals question had no metric. It was answered by a RevOps analyst’s “deals pushed past quarter end”, which many analysts had run many times, recently. That record is why the agent could trust it. A business term then linked the user’s word “slipped” to the query, though the two share no words.

Why a metric, and not a view

Most teams arrive with a semantic layer of some kind: Snowflake semantic views, Databricks metric views or a Genie Agent, LookML, or a homegrown metric store. Every layer holds the same four parts: what each table and column means, how tables join, metrics, and the dimensions a metric is grouped by. The question is how to organize them so an agent can use them. An agent puts three demands on that storage: a script with no model in the loop can compile a definition to the same SQL every time; the right item can be found by exact lookup when a metric covers the question and by search when it doesn’t; and a fix changes the definition itself instead of piling up beside it as notes.

The four parts get organized one of three ways, and each repeats something.

Per logical table is the default today, because it is what the major platforms ship. Its costs show up once agents are asking: a metric that two views need is often defined in each, a question that spans two views has nowhere to go, and a wrong answer gets patched with a note beside the view that the agent reinterprets on every question. On our benchmark, semantic views as the only way in scored 38.3%. On multi-table questions they scored 16%, against about 61% on the same questions for past queries with complete SQL. A view states a join; a past query shows it worked.

Business questions start with a metric. People ask how many orders came in last week or what churn did last quarter; they rarely start from a table or a view. A per-metric layer stores context in the shape the questions arrive in: the business can read a metric and correct it, the same definition serves every agent and BI tool, and it compiles to the same SQL every time. Its costs are real too. Questions no metric covers fall back to exploration, so coverage builds over time, and without owners and review, metrics sprawl into near-duplicates such as revenue, net revenue and recognized revenue. That is why every metric carries an owner and a version.

If you have semantic views, keep them. They keep working for the analysts who use them, and in Phase 2 they are a source: their measures and filters become candidate metrics alongside past SQL, witnessed and approved like any other. Five questions show quickly whether a layer will hold up once agents are asking:

1. When a metric definition changes, how many places have to change with it?
2. Can an agent answer a question that spans two of your views?
3. For a question your metrics cover, does the layer compile the SQL, or does the agent write it fresh?
4. Does the same request compile to the same SQL, with no model in the loop?
5. When someone corrects a wrong answer, does the fix change the model, or get added beside it as text?

Read more Semantic layers compared →, the tools side by side and one model in six syntaxes

Two lanes

The placement table says where knowledge goes. The lanes say how the agent may use it. For each kind of question, ask one thing: if this answer is wrong and nobody notices, where does it land?

In the governed lane, the agent serves a metric or gives no answer. The bar is exact, every time. The pipeline question took this lane and returned $412M, the number in the board pack.

In the exploratory lane, the agent writes its own SQL from trusted past queries and business terms. It shows the SQL and what it assumed. The bar is directionally right. The slipped-deals question took this lane: 96 deals had slipped, and the win rate with these customers was 31% against 24% elsewhere.

Good context alone doesn’t make a board number safe. Even the best run still got some answers wrong, and it stated them as plainly as right ones. In the exploratory lane, a reader can see the SQL and catch a miss before acting. A board number has no such reader, so it needs the governed lane. A kind of question can move lanes later: when two people publish different numbers for an exploratory question, write the metric and move it to the governed lane.

Card: Which lane?

The retrieval architecture

Three parts do the work:

• One store. Metrics, trusted past queries, rules and business terms live in one place, each with an owner and a version. This is the layer you build in Phase 2.
• One MCP server in front of it. It exposes a small set of tools, such as a context lookup and a metric fetch, to any agent that speaks MCP. Lyft reaches its metrics through an MCP server in the same way. DoorDash’s data agent goes further: it blocks unmodeled or unverified sources at the harness level, and keeps new joins and documents inactive until they meet an eval threshold.
• One rule in the agent’s instructions. Call for context before writing SQL. If a metric matches, run its SQL as served. If none matches, say so. For a governed question, stop there; otherwise explore and show the SQL.

Why MCP

The case for MCP isn’t the protocol. It is that every agent, whether it’s a warehouse agent, an IDE assistant or a Slack bot, reads the same copy through one door you can trace. When the next harness arrives, connecting it is only wiring.

What the search should match

1. Exact lookup first. When a metric covers the question, return it at the top, labelled as a metric, so the agent runs it instead of rewriting it.
2. Match on field names and business terms. Ranking past queries by the tables, columns and terms they share with the question beat ranking by embedding similarity, 68.2% to 55.1%. Words in a question rarely match the words in a query’s title. Business terms bridge that gap.
3. Return a handful, whole. Return the top five or so, with complete SQL. Cap by the number of items, never by characters. Past queries served whole instead of cut short added 9.4 points.
4. Don’t stack retrievers. A second, sharper retriever on top of the metrics added nothing, because near-variants compete with the item that settles the question.
5. Judge retrieval by answers, not recall. Better table search lifted recall from 59.8% to 86.4% and accuracy didn’t move. Finding more candidates is not the same as serving the decisive one.

Check it in the traces

Retrieval fails quietly. The agent still answers, so read the traces for three things:

• The call happens. Every question shows a context call before the first SQL statement. An agent wired to MCP but not told to call it often skips it.
• The decisive item comes back. For every miss in the golden set, check whether the metric or query that would have settled it was returned. If it wasn’t, the miss is “written, not served”: fix the term, the alternate labels or the ranking, not the metric. Phase 3 sorts every miss this way.
• The agent uses what came back. If the right item was returned and the agent wrote its own SQL anyway, tighten the instruction to run a metric’s SQL as served.

Read more Phase 2: serve it per question →

The brief

The brief is one page. It fixes the domain, the users, the problem, the owners and which questions need an exact answer. Phase 1 harvests questions only from inside it. The domain owner and the sponsor write it together at the kickoff.

domain:       Sales pipeline
fiscal_year:  ends 30 June
as_of:        the last quarter-end snapshot        # every answer in the golden set uses it

users:        CRO staff, sales leaders, RevOps analysts, FP&A partners
asks_in:      the chat tool sales leadership already uses

problem: >
  Leaders get pipeline numbers that disagree with FP&A.
  Each disagreement costs an analyst a day to reconcile.

owners:
  - RevOps lead: deals, stages, amounts; 2-3 h/week
  - FP&A partner: who the top accounts are; about 1 h/week in Phases 2 and 3
eval_owner:   one named person, with a RevOps analyst and an FP&A analyst as graders; 6-8 h in Phase 1, then about 2 h/week
model:        the one expected in production, named with its version

governed:     open pipeline, win rate, closed-won ARR    # exact, every run
exploratory:  slippage, pipeline creation, stage aging   # SQL shown; 4 in 5 pass
deferred:     why-questions, until the numbers are right
out_of_scope: bookings forecast, commissions, marketing attribution

The brief puts each kind of question in a lane. Governed questions must be exact on every run. Exploratory ones must clear one pass rate, which the sponsor writes down now, before any result exists. Name two owners if another team owns part of a definition. When the two meanings of “top 100 customers” collided in Phase 3, both people who could settle it were already named.

Read more the full domain brief template →

What you need before you start

This is the checklist for Phase 0: the system access and the named people. Most delays in Phase 0 come from access, so request every item on the first day. Each item is done when its test passes. A statement that access “should work” is not a signature.

Trace export is the item most often left until last, and the one the hill climb depends on. In Phase 3 the trace is where you see which decision the agent guessed. A nightly summary does not count, because it reports what the agent said it did. You need a record of every call, with what was served and what SQL ran.

Read more What you need before you start

People and hours

Each role owns one part of the work. Adjust the hours to your domain, but never to zero.

Put a person’s name in every row. “RevOps” cannot settle a conflict. The RevOps lead can. The co-owner’s hour is the line teams forget, and the agent builder’s 20 hours is the line they underestimate.

The agent architecture

Every analytics agent has four parts: the surface where people ask, the harness that runs the loop, the model, and the context it reads before it writes SQL. That context lives in a context layer: the store of metrics, trusted past queries, rules and terms that the agent calls once per question. It can be files in a repository behind a small search function, or a product. You choose between two shapes for the rest.

• Platform-native. The warehouse vendor supplies the harness and the surface. Databricks Genie and Snowflake CoWork (formerly Snowflake Intelligence) are the common ones. You configure; you do not build.
• Custom. You build the harness on a framework such as Google ADK or LangGraph, or write your own. You get full control of the loop, the tools and the traces.

In either shape, the harness runs in your environment and your team owns it after launch. Pick the model you expect to use in production. You can change it later, as long as you rerun the golden set on the new model before you compare scores.

The minimum harness

Build this and stop.

1. One context call per question, before any SQL. At baseline it returns the schema only.
2. One read-only SQL tool. Exploratory queries run as the asking user; governed answers run under a fixed service identity, so a board number is the same for every asker.
3. A step budget: a fixed cap on tool calls per question. Ten is a sound default. Leave it there; Phase 3 explains why.
4. A per-call trace: everything served, every tool call and SQL statement, the answer or refusal, tokens and time.

Fix the shape of the context call on the first day. In Phase 2 the metrics and trusted past queries arrive behind the same call, and nothing else in the harness changes. That is what keeps the baseline and the later scores comparable.

What leaves the building

This section matters when a vendor or outside team handles your data or runs part of the stack. Security will ask what data moves, and where. Walk them through this diagram at the kickoff.

Result rows never pass through the context layer. Every context call is logged, so security can audit what was served.

Gate 0

The delivery lead shows the sponsor the brief, the access tests and the named owners. The decision is to start the clock.

Card: The sponsor's ask

Harvest at least 20 real questions

Take the questions from what the business already asks and has already answered. An invented question tests work nobody needs done, and it tends to be one the agent can pass. Collect two or three times as many candidates as you will seal. Then pick at least 20 that cover the domain and both lanes.

Pull the candidates from four places:

• the board pack and the dashboards leaders read
• analyst tickets and requests from the last two quarters
• saved analyst queries on the domain’s tables
• the sponsor’s own questions

Rewrite each one the way a user would ask it, with no table or column names. Record where its verified answer lives. Then ask the domain owner one question, because it often surfaces the item that matters most: what was the last wrong number that reached a meeting, and how was it caught?

Write every item in one format

Every question goes into the golden set in one format, so any grader can check any item and you can version the set like code. Six fields are required. The rest are optional.

# required
id:               sales-01
question:         How much open pipeline do we have with our top 100 customers?
lane:             governed
source:           FP&A board pack, top-accounts pipeline page
expected:         {accounts: 91, deals: 340, open_pipeline: 412000000}
expected_sql:     |
  -- the query that returns expected, run read-only on the as-of snapshot

# optional
valid_alternatives: ["by region, summing to the total"]
tolerance:          {counts: exact, amounts: within 0.5%}
decisions:          [top 100 by current ARR, some early stages never count, amount is ARR, each deal once]
decoy:              "top 100 by deal count, every stage, contract value, each term -> $1.31B"
history:            ["graders disputed the source; FP&A partner chose the board pack"]

Two optional fields pay off in Phase 3. decisions lists the choices the question hides. decoy is the wrong answer you get when those choices are guessed the obvious way. When the agent misses, a grader compares its SQL against the decisions and names the one it guessed.

Read more the full golden-set item template →

Grade the data first, then the SQL

Write the grading rules in this phase, before any result exists. Rules written after the results bend toward them.

• Grade the data first, then the SQL. The data check is fast and needs no judgment. Then check that the SQL makes the item’s decisions. A wrong query can land near the right number by luck, so a near-match on data alone never passes.
• Allow several valid answers. A total and a breakdown that sums to it are both right. Record these in valid_alternatives when you seal. An alternative that makes a different decision is a conflict. The definition’s owner settles it before the item is sealed, and the eval owner sends it to the conflict list too, with no sealed value.
• Check consistency across runs. Right every time means the agent knows it. Right some of the time means it is guessing, and on one run you cannot tell the difference. A governed answer counts only if it is right on every run.
• Use an LLM as a judge for answer shape. A model is good at checking what code cannot: did the answer state its period, filter and unit? It should never grade the number itself when code can.

Calibrate the judge before its verdicts count:

1. Two graders grade the same sample blind, with one written rubric.
2. The judge grades the same sample.
3. Look at every disagreement. Most trace to a rule nobody wrote down, so fix the rubric.
4. The judge counts once it agrees with the graders about as often as they agree with each other. Recheck it whenever its model or rubric changes.

Keep the golden set away from the agent

The golden set and the agent’s context draw on the same past SQL. The query behind a dashboard tile can come back as a trusted past query, and then the agent copies the answer instead of working it out. Build guards against three leaks into the pipeline, so nobody has to remember to check:

• Golden SQL in the context. Before any context is served, remove every past query that matches an item’s expected SQL or returns a sealed answer. Record each removal by hash, so it can be audited without exposing the set.
• The agent reading its own work. Exclude the agent’s service account from the query history.
• Answers in served text. Scan every prompt for sealed values. A hit invalidates the run.

The context engineer never sees an expected answer. The agent sees only the question as a user would ask it.

Seal, then audit. An item is sealed when its expected SQL has run read-only on the snapshot, the result matches the trusted source, no dispute is open, and the eval owner has signed it. Commit the set to version control and never edit a question to make the agent pass.

The set will still have errors. When an item fails on every run whatever you change, suspect the item first: re-run its SQL, re-check the source, and correct it if it is wrong. Audit a few passing items too. A wrong item the agent happens to match looks like a pass, and nobody checks a pass.

Run the bare agent once for a baseline

Run the agent on the bare schema before you build any context. It shows how far the schema alone gets you, and every later score is read against it. Run each sealed question once, trace every call, and grade the result. For every miss, record which decision the agent guessed.

In the sales baseline, the pipeline question went wrong four ways. The agent ranked the top 100 by deal count, counted every stage, summed contract value, and joined each deal to every subscription term. Each choice was a reasonable reading of the schema, and the SQL ran clean. Each was also one line of the item’s decoy. The right answer is $412M.

Gate 1

The delivery lead shows the sponsor the golden set and the baseline. The decision is whether to start building context.

Card: The eval owner's week

Five steps from past SQL to metrics

What one metric holds

A metric is a file, governed like code. Here is win rate after its conflict was resolved, cut to the fields a reviewer reads.

name: win_rate
version: 4                          # v3 was the dashboard version; retired, kept on record
owners:
  - {role: RevOps lead, owns: the definition}
status: approved
description: Deals won as a share of deals won or lost in the period. Disqualified deals are out.
population: Deals closed won or closed lost in the period
exclusions: [disqualified deals, internal and test accounts]
grain: One row per deal (key opportunity id)
time_rule: Close date, fiscal quarters; the fiscal year ends 30 June
dimensions: [region, segment, rep]
joins:
  - {to: crm.account, condition: a.id = o.account_id, cardinality: many_to_one}
sql: |
  SELECT 100.0 * COUNT(CASE WHEN o.stage = 'Closed Won' THEN 1 END) / COUNT(*) AS win_rate
  FROM crm.opportunity o JOIN crm.account a ON a.id = o.account_id
  WHERE o.close_date BETWEEN $start AND $end
    AND o.stage IN ('Closed Won', 'Closed Lost')
    AND NOT a.is_internal AND NOT a.is_test
verified_examples:
  - question: What was our win rate last quarter?
    result: 24.3                    # executed against the warehouse
provenance:
  source: query history and the sales dashboard
  distinct_analysts: 6
  runs: 140
  witnessed: Phase 2 · 1 row · grain check passed
  conflict: dashboard version counted disqualified deals; settled on recency, confirmed by the RevOps lead
  approved_by: [RevOps lead]

Read more the full metric spec template →

Serve it per question

How you serve decides whether the agent uses what you built. Serve per question has the architecture and the ranking rules: one context call per question, a handful of complete statements, metrics first and labelled. Two more rules matter here:

1. Keep the literals. 'Closed Won' and stage_order >= 2 are the definition. Mask the names of people and customers, and keep the business values.
2. Serve rules as instructions. A house rule that holds for every question, such as “quarter means fiscal quarter”, goes to the agent as a short instruction with the name of whoever set it. If a metric depends on the rule, write it into the metric’s SQL as well.

Bring in what you already have

Most teams already hold some context: a semantic layer (semantic views, metric views, LookML or dbt metrics), saved queries, instructions written for an earlier agent. Treat all of it as a source, put it through the same witness and conflict steps, and sort it by what it can do.

Give every imported object an owner and a version before it is served.

Gate 2

The sponsor sees the score on the same sealed questions, the change since the baseline, the conflicts resolved and anything escalated. The decision is to start the hill climb. Hold the gate if anything served is unowned, unversioned or unwitnessed.

A gate can only check what somebody raised. A metric can pass every check here and still carry a definition nobody questioned, because the other definition never appeared in any source. Phase 3 finds those in the traces.

Card: How to resolve a conflict

The loop

Teams skip two steps most often.

• Predict before you run. Write down which questions each change should fix. A round that moves the total without landing its predictions did something other than you meant.
• Rerun the whole set. A change aimed at one question can break another. The sales team’s did: a rule meant to drop internal accounts also dropped a partner account the golden set counts. Only the full rerun caught it.

Two habits keep the scores honest. First, before the first round, run the unchanged context two or three times. The score moves a little between identical runs. Call that spread the run-to-run variation. A round that moves the total by less than that proves nothing on its own. Second, once the score is past about 60%, ask each question three to five times per round. Count a question as right only when it is right on every run.

Record every round in the iteration ledger: what changed, what you predicted, what moved, and what you kept or reverted.

Read the raw trace

A summary tells you what the agent says it did. The trace shows what it actually did, step by step. Read the trace of every miss before you change anything. Here is the pipeline question, cut to the lines that matter:

trace · the pipeline question · run 1 of 3
context call   returned: the metric "open pipeline, top customers", version 2, owner RevOps lead
ran metric     the metric's SQL, as served
               WITH top_accounts AS (... ORDER BY COUNT(*) DESC LIMIT 100)   <- ranked by number of deals
               ... WHERE NOT o.is_closed AND o.type IN ('New Business', 'Upsell')   <- every stage counts
answer         "Open pipeline with the top 100 customers: [a total well above the golden answer]"   wrong on 3 of 3 runs

The agent made no mistake of its own. It ran the statement it was given, the same way on every run. Both wrong decisions were inside the metric. Asking the question several times would never have exposed this, because a metric that is wrong the same way every time never disagrees with itself. Only the statement in the trace shows it.

A trace is worth reading when it carries, for every run:

• the prompt, with every served item by name and version
• each tool call, with its arguments and result
• every SQL statement, failed ones included
• the answer or the refusal
• tokens, steps and time

If your export lacks any of these, fix it before the next round.

The three kinds of miss

Check two things first, because adding context makes both worse. Is the golden answer wrong? Then the eval owner re-verifies and reseals it, and nobody edits the question to fit the agent. Did the agent run out of steps? Then read what the steps were spent on.

Then place the miss in one of three kinds:

Phase 3 found the first two kinds in the running example. The top-100 ranking was never written: finance’s definition lived nowhere the pipeline could see it. The slipped-deals question was written and not served. A trusted past query already answered it, but the question said “slipped”, and neither the query nor its title used that word, so the agent invented its own definition.

A miss that persists with its definition served and used is not a context problem. The warehouse has no table in the shape the question needs. Take it to the platform owner.

When a miss won’t sort cleanly, ask the question several times with everything else fixed and compare the SQL. Where the runs split, two definitions compete. Resolve it as a conflict. This check needs no golden set, so it keeps working after launch.

The smallest fix that closes a miss

The climb stalls when a team writes new definitions for misses whose definitions already exist. Search the context before you write anything. Then make the smallest change that closes the miss: a versioned edit to one metric, rule, business term or trusted query’s rank.

• Target the rule, never the answer. The change states the business rule. It should make sense to someone who has never seen the golden set.
• Keep sealed values out. No change may contain a golden answer or a literal copied from the set.
• Use the form that runs. Definitions go in as SQL, rules as instructions, vocabulary as term links.
• Give it an owner and a version. Each round gets a new context version, so rolling back is one step.

The slipped-deals fix shows the size to aim for. It was two edits to one business term. “Slipped deal” gained the alternate labels “pushed” and “rolled”, and it was linked to the analyst’s query. Nobody wrote a new definition, because none was missing.

The top-100 fix needed a decision. The context engineer prepared both rankings as witnessed SQL, with who uses each. The two owners chose ARR in 20 minutes and agreed to co-own the metric. The RevOps lead also moved “some early stages never count” out of a separate instruction and into the metric’s SQL. A harness runs a metric as written, so a rule has to be in the statement to count.

Record the changes that do nothing as carefully as the ones that work. The sales team added a description to the stage column, hoping the agent would apply the early-stage rule in its own SQL. Nothing moved across three runs, because the rule already lived in the metric, so they reverted it. The next domain reads that line in the iteration ledger before it spends a round on descriptions.

When to stop

Write the stop rule down before the first round, so nobody bends it later. Ours: stop when two rounds in a row move the score by no more than the run-to-run variation, and none of the fixes you predicted works. Count predicted fixes as well as the total, because a round can land its one fix while something else flickers. Cap the phase at five rounds. If the bar is still not cleared, report the ceiling honestly and take the gap to the sponsor.

Don’t raise the step budget to cut refusals

When the agent gives up too often, a bigger step budget looks like the easy answer. It turns refusals into confident wrong answers. A refusal tells you a decision is missing. Read the trace, find the decision and supply it. Keep the step budget where it was at the baseline.

Gate 3

The sponsor sees the score after each round, what broke and what was fixed, and the golden set grown to 50. The decision is to move to the launch gate.

Card: Why did this miss?

Gate 4: is it ready to launch?

A score earned on the questions you tuned against flatters the agent. That is why the graders grew the golden set to 50 during Phase 3, with questions from outside the climb: new dashboard tiles, recent tickets, the sponsor’s own asks. Now run all 50 three to five times each. Tick this checklist at the gate review.

If something falls short, you have three moves: ship without it and name it, move it to the exploratory lane with the sponsor’s agreement, or run one more round of the hill climb. The engagement’s launch readout fit in five lines:

# Launch readout: Sales · end of Phase 4 · CRO, RevOps lead, FP&A partner, eval owner, delivery lead
Governed: right on every run, and "no metric" on every out-of-scope probe · CLEARED
Exploratory: above the Phase 0 bar on every run, after one golden answer was corrected · CLEARED
Pipeline created this quarter: wrong on every run; the warehouse holds no "created and still open" record · HELD BACK
Decision: GO WITH CONDITIONS · the held-back type returns once the platform owner builds the record · traces must flow on launch day, or the launch waits

Read more Is it ready to launch?

Wire the agent to where users ask

You chose the architecture and the surface in Phase 0. Phase 4 wires the release path to the context layer and checks it on launch day. Each path reaches the same context layer; they differ in how the context gets to the agent.

Keep every definition in one store any harness can read, so the next harness change is only a wiring job. Four things must hold on launch day:

1. The agent asks for context before it writes any SQL. Confirm it in the traces.
2. Governed questions run the metric’s SQL as served. When no metric matches, the agent says so.
3. Governed answers run under a fixed service identity, so a board number is the same for every asker. Exploratory queries run under the user’s own identity.
4. Traces reach a store the review team can read: every context call, what it returned, every SQL statement and the answer.

The fourth slips most often. Without traces there is no loop, so the launch waits.

Hand over: one pass together, one alone

An agent that clears its bar is half the job. The other half is a team that can produce the next score alone. Each thing left behind needs an owner on your side and a version.

The decisions log outlasts the rest. It records every place two defensible definitions were in use, who decided, and when. It survives a model swap or a harness rewrite.

The handover takes three steps:

1. Walk the assets. Each owner uses their own asset while the context engineer watches.
2. Run one pass together. The team runs a full improvement round end to end: it decides the open conflicts, reruns the golden set and records the result. The context engineer only watches.
3. Run one pass alone within two weeks. The context engineer is on call and nobody calls. Your team writes the record.

In the running example, a schema change after launch forced the solo pass early, a harder test than a planned one (Keep it running tells the story). Test the handover with one question: could your team run the next round with the delivery team gone? If the answer depends on a script, a notebook or a person the delivery team brought, move it into your repository, give it an owner, or throw it away.

Card: Is it ready to launch?

Run the production loop

Your users will ask hundreds of questions the golden set never held. Each one comes with a trace, and the ones that recur are golden-set candidates. Two events in the first weeks after launch show the two halves of the loop.

A regional VP asked for open pipeline with the top customers in EMEA, then replied, “Wrong. My tracker is well above that.” The trace showed the metric’s SQL, matching the board pack. The VP’s tracker counted renewals and early stages the metric leaves out. The correction became a proposed change for the owners, never applied on its own. In ten minutes they kept the board number, added a note under every answer (“excludes renewals and early stages”), and put a renewal-pipeline metric in the backlog.

Then a schema alert fired: billing would retire a deal-amount column. The regression set ran on a staging copy first, and two questions failed. Both leaned on past queries that read the dying column. The agent had rewritten them from the schema and lost the rule about early stages. No user would have seen an error, only a plausible wrong number. The queries’ author re-issued them, the RevOps lead approved, and the rerun came back clean.

Run the loop every week, in one trace hour:

1. Read the raw traces of every flagged answer and a sample of the rest. Owner: agent builder, with the domain owner.
2. Pull every step where the agent logged that its context was not enough, and classify each miss.
3. Ask the most common real questions several times. A split between runs means a decision nobody made. File it as a conflict.
4. Decide the queue: corrections, conflicts and gap fixes. Owner: domain owner.
5. Batch the approved changes into the next release and run the gate. Owner: eval owner.

Corrections are the richest signal and the easiest to mishandle. Each arrives sounding like a bug, and often the user simply holds a different definition.

Rerun the set on every drift trigger

The business moves under the context, and nothing announces the move. A realignment, a pricing change or a retired column changes no line of the context. So handle drift with triggers, each with an owner. A context version with no eval run behind it is still a draft.

Keep the previous release deployable, so rollback takes one decision.

Domain two

Domain two inherits what domain one built: house rules, shared entities, business terms, the grading method and the release process. Plan it while the knowledge is still in the room.

1. Pick the next domain from those that share tables, entities and words with the first. Owner: exec sponsor, with the delivery lead.
2. List what it inherits with no new decision, and what it must build. Owner: context engineer.
3. List the words both domains use, and mark each as settled or contested.
4. Name the domain owner and their hours. Owner: exec sponsor.

In the engagement, finance came next. It inherited “top 100 customers”, the fiscal quarter and the rule on early stages. It still built its own golden set, metrics and queue. Two words were contested. Sales reads “ARR” as the latest term’s value, and finance reads it as recognised ARR at period end. Sales counts a customer by an active term, and finance by a paid invoice. Each owner could decide for their own domain, and neither could overrule the other. Name an arbiter before the new domain starts: one executive with authority over both functions. Until then, freeze the contested words, so neither domain changes them alone.

Answer why

“Why did pipeline drop in EMEA?” is what an executive asks as soon as the number lands, often at the first readout. An agent with only metrics restates the drop. To explain it, the agent has to know how your business works and what changed in it this quarter. That is a second kind of context, and it is built only after the numbers hold: a why-answer explains a movement in a number, and if the number is wrong, the explanation is a confident story about a figure nobody should trust. Write the why-question down with its date and finish the what-questions first.

EMEA pipeline with the top 100 customers fell $53M. North America and APAC held.
New deals were flat, and last year rules out the season.
$31M sits in deals that changed owner in the territory realignment and have not moved since.
$19M sits in Enterprise deals re-signed on new terms after the pricing change, which sent them back to an early stage.
The remaining $3M has no single cause.

Run the ceremonies

The delivery lead, whether a forward-deployed engineer, a partner consultant or an internal program lead, owns the calendar and the ceremonies and holds the gates. The delivery lead never decides a definition and never grades an answer. Those belong to the domain owner and the eval owner. If the delivery lead makes those calls, the numbers stop belonging to the business, and nobody owns their accuracy after they leave.

Each phase ends at a gate, and each gate is one decision: pass, hold or stop. What the sponsor sees at each gate lists what passes each one. A stop is a legitimate result. Four ceremonies carry the engagement. Book all of them through Phase 4 before the kickoff ends. Each one produces something written.

At the kickoff, teach before you ask. Open with one real question from their business and let the room find the rules hidden in it, using the exercise in Why it is hard. Then walk the checklist; the full agenda is the kickoff agenda template. The working session is where decisions happen, and its agenda is fixed: the score since the last session, the misses read from traces, changes waiting for approval, and conflicts waiting for a decision. The sponsor will ask for things the plan deferred. Log each one in the backlog with a date, and say when it comes back. “Go with conditions” is a pass with debts: check each condition on its date, because one that slips past unchecked has quietly lowered the bar.

What to do when things stall

A stall usually shows up first as a date that slips. Treat the slip as the signal, whatever caused it, and make the same four moves: log it the day you see it, with an owner and a date; keep working on everything that does not depend on it; take it to the sponsor at the next readout, or the same day if it is both likely and serious; and hold the gate it affects, without absorbing the stall by deciding yourself or starting the next phase early.

In our engagements, the stall that cost the most time was an owner who could spare only minutes a week. Decisions queued, and the gates slipped behind them. Ask for the hours back as soon as the queue starts to grow.

Three more things the delivery lead owes the company, each on one page: the boundaries that hold when a partner firm delivers, Deliver as a partner; the anonymised note on what moved the score, so the next engagement costs less, Pattern write-up; and a blameless postmortem at the end of each engagement and whenever a gate slips or a wrong governed number reaches a user, Field postmortem.

Card: What to do when

I’m sponsoring it

You are the exec sponsor, or the data leader who will answer for the agent. The risk you are buying down is a wrong answer that looks right.

▶ Listen 9:34

1. Why analytics agents are hard
2. The method at a glance
3. Evals first
4. Two lanes
5. Phase 0: before the clock
6. Is it ready to launch?
7. The skeptic’s questions

Your cards: The sponsor's ask, What the sponsor sees at each gate and Is it ready to launch?.

I’m building it

You are the agent builder, the context engineer or an analytics engineer. You want to know what to build, and where each piece of knowledge goes.

▶ Listen 19:35

1. Evals first
2. The metric is the unit
3. Serve per question
4. The minimum harness
5. Phase 1
6. Phase 2
7. Phase 3
8. Wire the agent to where users ask
9. Rerun the set on every drift trigger

Your cards: What you need before you start, Why did this miss? and How to resolve a conflict.

When the numbers are right, read Answer why. If you already run a semantic layer, read Why a metric, and not a view before Phase 2, and Semantic layers compared for the tools side by side.

I’m running it

You are the delivery lead: a forward-deployed engineer, a partner consultant or an internal program lead. You want to know what happens in each phase and what to do when a week goes wrong.

▶ Listen 15:18

1. The method at a glance
2. Phase 0
3. Phase 1
4. Phase 2
5. Phase 3
6. Phase 4
7. Run the ceremonies
8. What to do when things stall

Your cards: What you need before you start, The eval owner's week, What to do when, What the sponsor sees at each gate and Deliver as a partner.

What you’ll walk away with

At the end of Phase 4, your team holds:

• an agent that cleared the bar your sponsor set, on one domain;
• the metrics behind it, each with an owner and a version;
• a golden set of 50 real questions that tests every later change;
• a decisions log;
• people on your side who have run a round together, with a round alone booked within two weeks.

Hand over lists who owns each piece.

What’s inside

• Why analytics agents are hard. Why a clean query returns the wrong number, why a better model doesn’t fix it, and the four problems that follow.
• The method at a glance. The loop, its eight rules and the gate each phase must clear, on one page.
• The principles. Evals first, The metric is the unit and Serve per question: what to build, and why.
• The phases. Phase 0 to Phase 4, then Keep it running after launch.
• The toolkit. Cards, Templates and Straight answers to the questions a skeptic will ask.
• Research and reference. What we found, How we measured, What others found, failure modes, the glossary and tooling.

How to read this book

You don’t have to read it front to back. Pick the depth you have time for.

• One minute. The method at a glance, plus the rule at the top of each chapter.
• Watch. The method page, the opening chapter and each of the five phases open with a short narrated video, two to five minutes long. Videos plays them all back to back.
• Ten minutes. Turn on Skim in the top bar, or press S. Each chapter shrinks to its rule, an “In one minute” box, its key figure and its section headings. Click any heading to open the full section.
• The whole thing. About an hour and a half for the chapters, and about another hour and a half for the toolkit, the research and the reference. Pick a reading path above for your role and the contents will mark your chapters.

Press ? anywhere for keyboard shortcuts.

About this book

Published by Atlan. The book is licensed under CC BY 4.0: copy and adapt it with credit. The code that builds it is under the MIT License. Product names are their owners’ trademarks. Maintained by the playbook maintainers. A change to any number must include its source evidence, and every change goes through review by the maintainers.

Phase 0

# Engagement success document: Sales (pipeline and win rate)

## 1. Purpose
One pipeline number for the top 100 customers, the same every time it is asked, that matches finance.
Diagnostic answers on slippage, stage aging and win rate, in the chat tool sales leadership already uses.

## 2. Shared understanding
No agent exists yet. Sales ops, RevOps and finance each publish pipeline numbers, and two of them
reached the board on adjacent slides last quarter. Each disagreement costs an analyst a day to reconcile.

## 3. Scope
- Domain: sales pipeline. Data: CRM and billing, query history, the board pack and dashboards
- Golden set: at least 20 sealed in Phase 1, 50 by the launch gate; drafted and sealed by the eval owner and two graders
- Agent architecture: the chat tool as surface; a custom harness built by the agent builder; the model chosen in Phase 0
- Out of scope: bookings forecast, commissions, marketing attribution; why-questions until the numbers are released

## 4. Success criteria
| Gate | Deliverable | How we judge it together | Sign-off | Outcome |
|---|---|---|---|---|
| Phase 1 | 20 sealed; baseline | the grading rubric; one run per question | RevOps lead | pass: baseline 7 of 20 |
| Phase 2 | metrics, scored | every metric witnessed; conflicts resolved | RevOps lead | pass: 14 of 20 |
| Phase 3 | fixes from traces; set grown to 50 | nothing that passed has broken | RevOps lead | pass: 18 of 20 |
| Phase 4 | launch gate on the 50 | governed right on every run; exploratory 4 in 5 on every run | CRO | go with conditions |

## 5. Who does what; open questions
The ownership table and the hours table, with a name in every cell on the company's side.

## 6. Go / no-go (Phase 4)
Decision: GO WITH CONDITIONS
Conditions: platform owner builds the "created and still open" state, due two weeks after launch ·
RevOps lead resolves the open conflicts on released questions, due at the first release ·
CRO names domain two, due at handover
Signed: CRO, delivery lead

# Engagement success document: <domain> · signed <date>

## 1. Purpose
<one paragraph: the business problem and why this domain first>

## 2. Shared understanding
<what both sides agree is true today: data, current agent if any, known gaps>

## 3. Scope
- Domain; data in scope; out of scope: <...>
- Golden set: at least 20 sealed in Phase 1, 50 by the launch gate; drafted and sealed by <eval owner>
- Agent architecture: <surface>, <harness>, <model>; owner of each: <role>

## 4. Success criteria
| Gate | Deliverable | How we judge it together | Sign-off | Outcome |
|---|---|---|---|---|
| Phase 1 | golden set sealed; baseline | <rubric; one run per question> | <domain owner> | <your run> |
| Phase 2 | metrics, scored | <metrics witnessed; conflicts resolved> | <domain owner> | <your run> |
| Phase 3 | fixes from traces; set grown to 50 | <nothing that passed has broken> | <domain owner> | <your run> |
| Phase 4 | launch gate | <bar per lane, on the 50> | <exec sponsor> | <your run> |

## 5. Who does what; open questions
<the ownership and hours tables, the company holding every accountable cell; open questions with owner and due date>

## 6. Go / no-go (Phase 4)
Decision: GO | NO-GO | GO WITH CONDITIONS
Conditions: <owner, action, due>
Signed: <exec sponsor>, <delivery lead>

# Kickoff: Sales · 90 min · run by the delivery lead
Room: CRO, RevOps lead, FP&A partner, eval owner, agent builder, platform owner, security lead, delivery lead, context engineer

| Time | Item | Output |
|---|---|---|
| 0:00 | Why pilots stall between the demo and the tenth question | a shared diagnosis |
| 0:10 | The decisions hidden in "What's our Q3 pipeline?", on the company's data | two unwritten rules: some early stages never count; internal and test accounts are out |
| 0:25 | Which domain first | sales pipeline, over finance close and marketing attribution |
| 0:35 | The two lanes | board numbers governed; slippage, creation and aging exploratory at 4 in 5; why deferred |
| 0:50 | Owners | RevOps lead: deals, stages, amounts · FP&A partner: who the top 100 customers are |
| 1:00 | Walk the before-you-start checklist | people named; architecture chosen |
| 1:15 | Access and what leaves the building | security walks the data-flow diagram; trace export owned by the agent builder |
| 1:25 | Next steps | brief agreed; the clock starts when every access test passes |

# Kickoff: <domain> · 90 min · run by <delivery lead>
Room: <exec sponsor>, <domain owner(s)>, <eval owner>, <agent builder>, <platform owner>, <security lead>, <context engineer>
Pre-read: the one-page method

| Time | Item | Output |
|---|---|---|
| 0:00 | Why pilots stall between the demo and the tenth question | shared diagnosis |
| 0:10 | One question, the decisions it hides, on your own data | two of your own unwritten rules |
| 0:25 | Which domain first | the domain, and why |
| 0:35 | The two lanes | which questions are governed; one pass rate for the rest |
| 0:50 | Who owns which definitions | named domain owner(s) |
| 1:00 | Walk the before-you-start checklist | status per item; owners for gaps |
| 1:15 | Access and what leaves the building | security's next step and date |
| 1:25 | Next steps | checklist met, or the date it will be |

# Before you start: Sales (pipeline and win rate) · met at the end of Phase 0 · signed by the CRO and the delivery lead

## Access
| Item | Test | Owner | Status |
|---|---|---|---|
| Read-only warehouse access for CRM and billing (required) | a query runs under it | platform owner | done, first week of Phase 0 |
| Query history for sales (required) | the export has rows and a date range | platform owner | done, first week |
| BI reports and semantic models: board pack, RevOps dashboards, semantic model (required if you have them) | the export opens | platform owner | done, second week |
| Trace export from the harness (required): context served, every SQL statement, the answer | one trace read end to end | agent builder | done, last day of Phase 0: the chat tool exported a nightly summary only, so a per-call export was built |
| Where users will ask, named | recorded in the brief | exec sponsor | the chat tool sales leadership already uses |

## People
| Role | Name | Hours |
|---|---|---|
| Exec sponsor | the CRO | kickoff, a 30-minute readout in each phase, gates |
| Domain owner | the RevOps lead | 2-3 h/week |
| Co-owner | the FP&A partner | about 1 h/week in Phases 2-3 |
| Eval owner and graders | eval owner; a RevOps analyst and an FP&A analyst | 6-8 h in Phase 1, then about 2 h/week |
| Agent builder | the agent builder | about 20 h/week to the baseline, then 4-6 |
| Platform owner, security lead | named | Phase 0, then on call |
| Context engineer, delivery lead | from the partner; in-house, an analytics engineer and your program lead | dedicated; every session |

# Before you start: <domain> · met <date> · signed by <exec sponsor>, <delivery lead>

## Access
| Item | Test | Owner | Status |
|---|---|---|---|
| Read-only warehouse access for the domain's tables (required) | a query runs under it | <platform owner> | <date> |
| Query history for the domain (required) | the export has rows and a date range | <platform owner> | <date> |
| BI reports and semantic models (required if you have them) | the export opens | <platform owner> | <date> |
| Trace export from your harness (required): context served, every SQL statement, the answer | one trace read end to end | <agent builder> | <date> |
| Where users will ask, named | recorded in the brief | <exec sponsor> | <surface> |

## People
| Role | Name | Hours |
|---|---|---|
| Exec sponsor | <name> | <...> |
| Domain owner, co-owner | <names> | <h/week> |
| Eval owner and graders | <names> | <h/week> |
| Agent builder | <name> | <h/week> |
| Platform owner, security lead | <names> | <...> |
| Context engineer, delivery lead | <names> | <...> |

# Domain brief: Sales · v1 · agreed at the kickoff
domain:       sales pipeline
fiscal_year:  ends 30 June; every eval answer at the fiscal year-end snapshot
users:        CRO staff, sales leaders, RevOps analysts, FP&A partners, asking in the chat tool they already use
problem:      pipeline numbers disagree with finance; each disagreement costs an analyst a day to reconcile
owners:       RevOps lead (deals, stages, amounts), 2-3 h/week · FP&A partner (who the top 100 customers are), about 1 h/week in Phases 2 and 3
eval_owner:   one named person, with a RevOps analyst and an FP&A analyst as graders; 6-8 h in Phase 1, then about 2 h/week

governed:     # exact, on every run; a metric or no answer
  - open pipeline, any cut
  - win rate, any cut
  - closed-won ARR, any cut
exploratory:  # SQL and sources shown; the bar is 4 in 5, right on every run
  - slippage
  - pipeline creation
  - stage aging
  - rep and territory cuts
deferred:     why-questions, until the numbers are released
out_of_scope: bookings forecast, commissions, marketing attribution

data:         CRM opportunities, snapshots, accounts, stage and close-date history, reps, territories; billing terms
sources:      query history, human authors; the board pack; the RevOps dashboards and semantic model
import:       the RevOps semantic model; no earlier agent, so no instructions
architecture: surface the chat tool · harness custom, built by the agent builder · model the production model, named with its version

# Domain brief: <domain> · v<n> · agreed <date>
domain:       <domain>
fiscal_year:  <when it ends>; every eval answer at <snapshot>
users:        <roles who will ask, and where they ask today>
problem:      <what goes wrong today, for whom, at what cost>
owners:       <domain owner (what they decide)>, <h/week> · <co-owner, if a definition crosses teams>, <h/week>
eval_owner:   <name>, with <graders>; <h in Phase 1>, then <h/week>

governed:     # exact, on every run; a metric or no answer
  - <kind of question>
exploratory:  # SQL and sources shown; the bar is <one pass rate, written now>
  - <kind of question>
deferred:     <kinds of question, and why>
out_of_scope: <questions, data, surfaces>

data:         <schemas and tables>
sources:      <query-history window, human authors>; <BI reports and semantic models>
import:       <semantic layer, saved queries or instructions you already have, or none>
architecture: surface <...> · harness <platform-native or custom> · model <the production model, with its version>

Phase 1: the golden set

# Question harvest: Sales, Phase 1 · run by the eval owner · graders: a RevOps analyst, an FP&A analyst · 62 candidates
Sources: 23 from the board pack, 19 from RevOps dashboards, 14 from tickets, 6 from the CRO. Five rows shown.

| # | Question, as a user asks it | Source | Asked by, how often | Lane | Verified answer at | Keep? |
|---|---|---|---|---|---|---|
| 3 | How much open pipeline do we have with our top 100 customers? | board pack, top-accounts page; RevOps "Top accounts" tile | CRO, weekly | governed | board pack. DISPUTED: the tile ranks a different 100 | yes; the FP&A partner picks the source before sealing |
| 7 | How many of the open deals with our top 100 customers slipped from last quarter? | a RevOps analyst's saved query, "deals pushed past quarter end, by rep" | sales leaders, each quarter end | exploratory | the analyst's query on the snapshot | yes |
| 8 | What is our win rate with the top 100 customers, against everyone else? | RevOps win-rate dashboard | CRO staff, monthly | governed | dashboard; denominator to confirm | yes |
| 19 | Which reps have the most pipeline stuck at one stage for over 60 days? | ticket from a regional sales VP | sales managers, ad hoc | exploratory | graders compute it from stage history | yes |
| 41 | Why did pipeline with these customers drop in EMEA this quarter? | CRO, year-end review | CRO, once | why | none | no: deferred until the numbers are released |

# Question harvest: <domain>, Phase 1 · run by <eval owner> · graders: <names> · <n> candidates
Sources: <n> from <source>, <n> from <source>, ...

| # | Question, as a user asks it | Source | Asked by, how often | Lane | Verified answer at | Keep? |
|---|---|---|---|---|---|---|
| 1 | <question> | <tile, report, ticket, saved query> | <role>, <how often> | <governed / exploratory / why> | <where, or none> | <yes / no: why> |

Candidates drafted by a tool are marked "drafted"; graders rewrite them in the user's words.
A disputed source goes to the definition's owner before sealing, and to the conflict list with no value attached.

# Coverage estimate: Sales, Phase 1 · run by the eval owner
Inputs: 62 harvested questions; query history (a query counts with 2 or more distinct analysts and 5 or more runs)

| Lane | Candidates | With a trusted past query | With an existing metric | Neither | Covered |
|---|---|---|---|---|---|
| governed | your run | your run | your run | your run | your run |
| exploratory | your run | your run | your run | your run | your run |

Reading: <go | hold | re-scope>. Uncovered topics go to the context engineer as topic names only.

# Coverage estimate: <domain>, Phase 1 · <date> · run by <eval owner>
Inputs: harvested questions; <query-history window; a query counts with at least <n> analysts and <n> runs>

| Lane | Candidates | With a trusted past query | With an existing metric | Neither | Covered |
|---|---|---|---|---|---|
| <lane> | <n> | <n> | <n> | <n> | <%> |

Reading: <go | hold: mostly uncovered | re-scope>
Any query later used as expected SQL is purged from the context by hash.

# Grading rubric: Sales, v1 · agreed in Phase 1 by the eval owner and the RevOps lead

## 1. Returned data first
- Counts exact. Amounts within 0.5%. A fraction equals its percentage.
- The pipeline question expects 91 accounts, 340 deals, $412M. A split by region that sums to the total
  (NA $198M, EMEA $161M, APAC $53M) is a valid alternative.

## 2. Then the SQL
- A value inside tolerance reached by the wrong logic is wrong.
- Governed: the answer comes from the metric. No metric, no answer.
- Exploratory: SQL and sources shown; directionally right passes if the RevOps lead agrees.

## 3. Verdicts
- The pipeline question at $1.31B (top 100 by deal count, every stage, contract value, each term): wrong.
- "Two definitions of top 100 are in use; which one?" before the conflict is resolved: a correct refusal.
- A run that failed is an error, reported apart, never counted as wrong.

## 4. Runs and judges
- Baseline and Phase 2 score: one run per question. From the first round in Phase 3: three runs. Launch gate: five.
- Score = questions right on every run.
- An LLM judge checks answer shape only (period, filter and unit stated), after calibration on 20 items.

## 5. Not measured
Why-questions, follow-up turns, entity lookups.

# Grading rubric: <domain>, v<n> · agreed <date> by <eval owner>, <domain owner>

## 1. Returned data first
- Compare returned values with `expected`; accept any `valid_alternatives` entry.
- Counts: <exact>. Amounts: <tolerance>. A fraction equals its percentage.

## 2. Then the SQL
- Check the SQL against the decisions the question hides.
- A value inside tolerance reached by the wrong logic is wrong.
- Governed: the answer must come from the metric. No metric, no answer.
- Exploratory: SQL and sources shown; directionally right passes if the domain owner agrees.

## 3. Verdicts
- correct: data within tolerance and the logic holds
- wrong: data outside tolerance, or right data by wrong logic
- refusal: declined, or flagged a conflict. Correct when: <policy>
- error: the run failed; reported separately, never counted as wrong

## 4. Runs and judges
- One run per question until the score passes about 60%; then <3-5> runs; score = right on every run.
- Code checks values first; an LLM judge checks answer shape only, once calibrated on <n> items.

## 5. Not measured
<e.g. why-questions, follow-up turns, entity lookups>

# required
id:           sales-01
question:     How much open pipeline do we have with our top 100 customers?
lane:         governed
source:       finance board pack, top-accounts pipeline page (year-end close)
expected:     {accounts: 91, deals: 340, open_pipeline: 412000000}
expected_sql: |   # a full copy, never a link, so the sealed answer never moves when a metric does
  WITH latest_term AS (
    SELECT account_id, arr, ROW_NUMBER() OVER (PARTITION BY subscription_id ORDER BY start_date DESC) AS rn
    FROM billing.subscription_term WHERE status = 'active' AND start_date <= DATE '2026-06-30'),
  top_accounts AS (
    SELECT account_id FROM latest_term WHERE rn = 1 GROUP BY account_id ORDER BY SUM(arr) DESC LIMIT 100)
  SELECT COUNT(DISTINCT o.account_id) AS accounts, COUNT(*) AS deals, SUM(o.arr) AS open_pipeline
  FROM crm.opportunity_snapshot o
  JOIN top_accounts t ON t.account_id = o.account_id JOIN crm.account a ON a.id = o.account_id
  WHERE o.as_of_date = DATE '2026-06-30' AND o.stage_order >= 2 AND o.type IN ('New Business', 'Upsell')
    AND NOT o.is_closed AND NOT a.is_internal AND NOT a.is_test;

# optional
as_of:              2026-06-30        # the fiscal year-end snapshot
valid_alternatives: ["by region, summing to the total: NA 198M, EMEA 161M, APAC 53M"]
tolerance:          {counts: exact, amounts: within 0.5%}
decisions:          [top 100 by current ARR, some early stages never count, amount is ARR, each deal once]
decoy:              "top 100 by deal count, every stage, contract value, each term -> $1.31B"
sealed_by:          eval owner, with the RevOps and FP&A graders, Phase 1
history:            ["Phase 1: graders disputed the source (dashboard tile vs board pack); the FP&A partner chose the board pack"]

# required
id:           <domain>-<nn>
question:     <exactly as a user would ask it>
lane:         <governed | exploratory>
source:       <dashboard, report or analyst work the answer was verified against>
expected:     {<column>: <value>, ...}
expected_sql: |
  <DuckDB SQL that returns expected; a full copy, never a link>

# optional
as_of:              <YYYY-MM-DD>
valid_alternatives: []                 # each with a reason
tolerance:          {counts: exact, amounts: <your setting>}
decisions:          [<the choices the question hides>]
decoy:              <the wrong answer you get when the hidden choices are guessed the obvious way>
sealed_by:          <eval owner, with the graders, when>
history:            []                 # every key correction and source dispute: when, who, why

{
  "request": {
    "question": "How much open pipeline do we have with our top 100 customers?",
    "domain": "sales",
    "identity": {"user": "a member of the CRO staff", "passthrough": true}
  },
  "response": {
    "call_id": "cc-phase3-sales-01-r1",
    "context_version": "v1.3",
    "lane": "governed",
    "metrics": [
      {
        "name": "open_pipeline_top_accounts",
        "version": 3,
        "owners": ["RevOps lead", "FP&A partner"],
        "provenance": "approved; witnessed in Phase 3; top-100 conflict resolved by both owners",
        "parameters": {"as_of": {"type": "date", "required": true}, "n": {"type": "integer", "default": 100}},
        "dimensions": ["region", "segment", "rep", "stage"],
        "sql": "-- the metric's sql field, complete, as in the metric spec"
      }
    ],
    "trusted_queries": [],
    "rules": [
      {"name": "quarter-means-fiscal", "version": 1, "instruction": "Quarters are fiscal; the fiscal year ends 30 June."}
    ],
    "business_terms": [{"term": "Top accounts", "version": 2, "points_to": "open_pipeline_top_accounts v3"}]
  },
  "trace_record": {
    "trace_id": "sales-01 · Phase 3, round 4 · run 1 of 3",
    "question_id": "sales-01",
    "model": "the chosen model and version",
    "served": [
      {"id": "open_pipeline_top_accounts", "version": 3},
      {"id": "quarter-means-fiscal", "version": 1},
      {"id": "Top accounts", "version": 2}
    ],
    "tool_calls": [
      {"turn": 2, "tool": "context_call", "args": {"domain": "sales"}, "result": "cc-phase3-sales-01-r1", "ok": true},
      {"turn": 3, "tool": "execute_metric", "args": {"metric": "open_pipeline_top_accounts", "version": 3, "as_of": "2026-06-30", "n": 100},
       "identity": "the fixed service identity", "sql": "-- the bound statement, exactly as run", "rows": 1, "ok": true}
    ],
    "outcome": {"kind": "answer", "text": "Open pipeline with our top 100 customers: $412M, 340 deals on 91 accounts."},
    "steps": 4,
    "sql_statements": 1,
    "seconds": "your run",
    "tokens": "your run"
  }
}

{
  "request": {
    "question": "<the user's words, unedited>",
    "domain": "<domain>",
    "identity": {"user": "<the asking user's id>", "passthrough": true}
  },
  "response": {
    "call_id": "<unique per call>",
    "context_version": "<v x.y>",
    "lane": "<governed | exploratory>",
    "metrics": [
      {"name": "<metric>", "version": "<int>", "owners": ["<role>"], "provenance": "<status, owners, how it was settled>",
       "parameters": {"<name>": {"type": "<type>", "required": "<true | false>", "default": "<value, or omit>"}},
       "dimensions": ["<name>"], "sql": "<the complete sql field>"}
    ],
    "trusted_queries": [
      {"id": "<query id>", "version": "<int>", "provenance": "<run by N analysts, M runs, last run <when>; witness OK>", "sql": "<complete statement, literals kept>"}
    ],
    "rules": [{"name": "<rule>", "version": "<int>", "instruction": "<one rule, as an instruction>"}],
    "business_terms": [{"term": "<term>", "version": "<int>", "points_to": "<metric or query, with version>"}]
  },
  "trace_record": {
    "trace_id": "<question id · week, round · run n of k>",
    "question_id": "<id, or unsealed in production>",
    "model": "<model and version>",
    "served": [{"id": "<item>", "version": "<version>"}],
    "tool_calls": [
      {"turn": "<n>", "tool": "<context_call | run_sql | execute_metric>", "args": {}, "identity": "<user | service identity>",
       "sql": "<every statement, failed ones included>", "rows": "<n>", "ok": "<true | false>", "error": "<message, if any>"}
    ],
    "outcome": {"kind": "<answer | refusal: no metric | conflict flagged>", "text": "<what the user saw>"},
    "steps": "<n>",
    "sql_statements": "<n>",
    "seconds": "<n>",
    "tokens": "<in, out>"
  }
}

Phase 2: metrics

id: q-pushed-by-rep
title: Deals pushed past quarter end, by rep
owner: a RevOps analyst
domain: sales
version: 1
provenance:
  distinct_analysts: 9
  runs: 120
  last_run: the final week of the fiscal year
  witness: OK                      # re-run read-only in Phase 2
business_terms:                    # linked in Phase 3; before that "slipped" found nothing
  - {term: Slipped deal, alternate_labels: [pushed, rolled, slid]}
rules: [quarter-means-fiscal]
dialect: duckdb
sql: |
  WITH slipped AS (                -- each deal once, however often it was pushed
    SELECT DISTINCT h.opportunity_id
    FROM crm.close_date_history h
    WHERE h.old_close_date >= DATE '2026-01-01' AND h.old_close_date < DATE '2026-04-01'
      AND h.new_close_date >= DATE '2026-04-01'
      AND h.changed_at >= DATE '2026-01-01' AND h.changed_at < DATE '2026-04-01')
  SELECT r.name AS rep, COUNT(*) AS slipped, SUM(o.arr) AS arr
  FROM slipped s
  JOIN crm.opportunity o ON o.id = s.opportunity_id
  JOIN crm.rep r ON r.id = o.owner_id
  GROUP BY r.name
  ORDER BY slipped DESC;
notes: Fiscal Q3 runs January to March; the fiscal year ends 30 June.

id: <query id>
title: <what the query answers, in the analyst's words>
owner: <analyst or team who wrote it>
domain: <domain>
version: <int>                     # bumps when the author re-issues it
provenance:
  distinct_analysts: <N>
  runs: <M>
  last_run: <when>
  witness: <OK | FAIL>             # re-run read-only against the warehouse
business_terms:                    # the terms linked to this query, with their alternate labels
  - {term: <term>, alternate_labels: [<label>, ...]}
rules: [<rules that apply when it is used>]
dialect: duckdb
sql: |
  <complete statement, literals kept>
notes: <what the analyst's comments say, if any>

name: open_pipeline_top_accounts
version: 3                          # v2 ranked customers by deal count; retired when the top-100 conflict was resolved
owners:
  - {role: RevOps lead, owns: deals and amount}
  - {role: FP&A partner, owns: who the top 100 customers are}
status: approved
description: >
  Open new-business and upsell pipeline in ARR, on the 100 customers with the highest current ARR.
  Some early stages never count. Finance ranks the customers; Sales owns the deals.
population: Open new-business and upsell deals past the first stage, on the top N customers by current ARR
grain: One row per customer; each open deal counted once (key opportunity id)
time_rule: The quarter-end snapshot at the as-of date. Quarters are fiscal; the fiscal year ends 30 June.
formula: SUM(o.arr)
exclusions: [first-stage deals, renewals, internal and test accounts, closed deals]
filters:
  - o.stage_order >= 2
  - o.type IN ('New Business', 'Upsell')
  - NOT o.is_closed
  - NOT a.is_internal
  - NOT a.is_test
dimensions:
  - {name: region, sql: a.region}
  - {name: segment, sql: a.segment}
  - {name: rep, sql: o.owner_id}
  - {name: stage, sql: o.stage}
joins:
  - {to: top_accounts (ranked from billing.subscription_term), condition: t.account_id = o.account_id, cardinality: many_to_one}
  - {to: crm.account, condition: a.id = o.account_id, cardinality: many_to_one}
sql: |
  -- dialect: duckdb; parameters: $as_of (date, required), $n (integer, default 100)
  WITH latest_term AS (
    SELECT account_id, arr, ROW_NUMBER() OVER (PARTITION BY subscription_id ORDER BY start_date DESC) AS rn
    FROM billing.subscription_term WHERE status = 'active' AND start_date <= $as_of),
  top_accounts AS (
    SELECT account_id FROM latest_term WHERE rn = 1 GROUP BY account_id ORDER BY SUM(arr) DESC LIMIT $n)
  SELECT COUNT(DISTINCT o.account_id) AS accounts, COUNT(*) AS deals, SUM(o.arr) AS open_pipeline
  FROM crm.opportunity_snapshot o
  JOIN top_accounts t ON t.account_id = o.account_id JOIN crm.account a ON a.id = o.account_id
  WHERE o.as_of_date = $as_of AND o.stage_order >= 2 AND o.type IN ('New Business', 'Upsell')
    AND NOT o.is_closed AND NOT a.is_internal AND NOT a.is_test;
verified_examples:
  - question: How much open pipeline do we have with our top 100 customers?
    parameters: {n: 100, as_of: 2026-06-30}
    result: {accounts: 91, deals: 340, open_pipeline: 412000000}
provenance:
  source: the finance board pack; the ranking lived in a spreadsheet, never in query history
  observed_queries: 0               # 31 queries pasted the list's output as fixed IDs; none computed the ranking
  distinct_analysts: 0
  witnessed: Phase 3 · 1 row · grain check 340 = 340 · by region NA 198M, EMEA 161M, APAC 53M
  conflict: >
    Top 100 by deal count (the sales ops dashboard, the only ranking in query history) against top 100 by
    current ARR (the board pack, ranked in a spreadsheet). Provenance could not weigh a variant it never saw, so both owners decided: ARR.
  approved_by: [RevOps lead, FP&A partner]
  replaces: "v2: top 100 by deal count"

name: <snake_case>                  # unique; equals the filename stem
version: <int>                      # bumps when sql, formula, filters or joins change
owners:
  - {role: <team>, owns: <what part of the definition>}
status: candidate                   # candidate | approved | deprecated
description: <what a business owner approves, in plain words>
population: <which rows the number counts>
grain: <one row per ...; the key each row is counted once by>
time_rule: <which date, which calendar, snapshot or live>
formula: <aggregate expression>
exclusions: [<what is deliberately left out, in plain words>]
filters: [<SQL predicates, always ANDed in>]
dimensions:
  - {name: <name>, sql: <expression>}
joins:
  - {to: <schema.table>, condition: <condition>, cardinality: <many_to_one>}
sql: |
  -- dialect: duckdb; parameters: $<name> (<type>, <required | default>)
  <the complete executable statement>
verified_examples:
  - question: <text>
    parameters: {<name>: <value>}
    result: {<column>: <value>}
provenance:
  source: <query history | BI | imported | hand-written>
  observed_queries: <n>
  distinct_analysts: <n>
  witnessed: <when · rows · grain check>
  conflict: <what disagreed, and whether provenance or the owner settled it>
  approved_by: []
  replaces: null                    # previous version and why it was retired

# Conflict: who the top 100 customers are
Status: resolved · Raised by: trace review, Phase 3 · Settled by: the owners, a 20-minute session the next day
Room: RevOps lead (decides: deals) · FP&A partner (decides: top 100) · context engineer (presents, no vote) · eval owner (present, silent on the key)
Prepared the day before: both variants as SQL, witnessed; the pasted list of 100 account IDs traced to the FP&A partner's spreadsheet

## The conflict
When someone asks about "our top 100 customers", which 100 are they?

| Variant | Definition | Where it is used | Provenance |
|---|---|---|---|
| A | most opportunities, any stage | sales ops "Top accounts" dashboard; approved v2 | the only ranking in query history, run often by sales ops |
| B | highest current ARR on the latest active term | finance board pack | none in query history: the ranking lived in a spreadsheet; 31 RevOps queries pasted its output as a fixed list of IDs |

Provenance: cannot settle it. B, the number leadership runs on, never appeared in query history, so history favours A by default. Sent to the owners.

## Decision
B, written as SQL on billing terms, with "some early stages never count" written into the metric's population.
Rationale, FP&A partner: "The board pack is the number leadership runs on. Deal count rewards accounts with many small deals."
RevOps lead: "Agreed. Sales ops keeps its tile, renamed 'most active accounts'."

## Effect
- Context changed: open_pipeline_top_accounts v3, owned by the RevOps lead and the FP&A partner; v2 retired, kept on record
- Business term "Top accounts" now points to v3
- Golden-set items affected: the pipeline question and the win-rate question; already sealed to B; resealed: no
- Next: v3 re-witnessed, then the whole set rerun; predicted fixes: the pipeline, slipped-deals and win-rate questions
- Reopen if: finance changes how the board pack ranks customers

# Conflict: <short name>
Status: open | resolved on provenance | resolved by the owner | escalated · Raised by: <generation | repeated runs | trace review | user correction>, <when>
Room (owner decisions only): <who decides, on what> · <who presents, no vote> · <eval owner, silent on the key>
Prepared: <each variant as SQL, witnessed; any pasted list it touches, traced>

## The conflict
<the question the business had not decided, in one sentence>

| Variant | Definition | Where it is used | Provenance |
|---|---|---|---|
| A | <definition> | <dashboards, teams> | <analysts, runs, last run, matches the warehouse?> |
| B | <definition> | <dashboards, teams> | <analysts, runs, last run, matches the warehouse?> |

Provenance: <settles it: which variant and why | split: sent to the owner>

## Decision
<the variant chosen, or the new definition>
Rationale: <why, in the owner's words, or the provenance that settled it>

## Effect
- Context changed: <metric and new version | rule | business term>; retired: <version>
- Golden-set items affected: <ids>; resealed by the eval owner: <yes / no>
- Next: <re-witness, rerun, predicted fixes>
- Reopen if: <what would reopen it>

Phase 3: the hill climb

# Trace review: Sales, Phase 3, rounds 2 and 3 · reviewers: context engineer, agent builder; RevOps lead approves

| Question | Run | Verdict | Expected / returned | Served | Used? | Step where it guessed | Kind of miss | Proposed change | Approved |
|---|---|---|---|---|---|---|---|---|---|
| partner-org count | round 2 · 1 of 3 | wrong; passed in round 1 | sealed count, a partner org included / one org short | new rule "internal and test accounts are out", as an instruction | yes, too widely | turn 3: excluded every account that was not a direct customer | a break: a rule served too widely | scope the rule to the org-type field | RevOps lead, round 3 |
| win rate, fiscal Q3 | round 3 · 2 of 3 | wrong on 1 of 3 | fiscal Q3 / calendar Q3 | win_rate v4; rule quarter-means-fiscal | partly | turn 3: let the metric's period default read "Q3" as July to September | served, parameter bound wrong | make the time parameter required, no default | RevOps lead, round 4 |
| pipeline, top 100 | round 3 · 1 of 3 | wrong, same every run | $412M / a total above the key | open_pipeline_top_accounts v2 | yes | turn 3: the served metric ranked customers by deal count, with no early-stage rule | never written: finance's ranking was nowhere in the context | raise the top-100 conflict; write the stage rule into the metric | both owners, the next day |
| slipped deals | round 3 · 2 of 3 | wrong | 96 slipped ($118M) / 218 slipped | win_rate v4 (weak match); the trusted query not retrieved | partly | turn 3: "nothing served defines slipped"; counted any close-date change | written, not served | add alternate labels "pushed", "rolled" to "Slipped deal"; link the term to the query | RevOps lead, round 4 |
| pipeline created, by segment | round 3 · 3 of 3 | wrong, same every run | sealed value / wrong | a newly promoted trusted query | yes | none: the warehouse holds no "created and still open" state | the data | escalate to the platform owner | platform owner |

# Trace review: <domain>, round <n> · context v<x.y> · reviewers: <context engineer>, <agent builder>; <domain owner> approves

| Question | Run | Verdict | Expected / returned | Served | Used? | Step where it guessed | Kind of miss | Proposed change | Approved |
|---|---|---|---|---|---|---|---|---|---|
| <question> | round <n> · <n of k> | <verdict; mark a break if it passed before> | <value> / <value> | <names and versions> | <yes / no / partly> | <turn, and what it assumed> | <key wrong / budget / never written / written, not served / served, ignored / the data / a break> | <one change> | <who, when> |

# Iteration ledger: Sales · golden set of 20 · run-to-run variation: 2 questions (the Phase 2 context run three times unchanged: 14, 13, 15) · score = right on every run

| Round | When | Context | Changes | Predicted fixes | Runs | Score | Fixed / broken | Decision, approved by |
|---|---|---|---|---|---|---|---|---|
| 0 | end of Phase 1 | bare schema | none (baseline) | none | 1 | 7 of 20 | none | Gate · eval owner |
| 1 | end of Phase 2 | v1.0 | six metrics, trusted queries, business terms, two rules | none (first scored run) | 1 | 14 of 20 | +7 / 0 | Gate · RevOps lead |
| 2 | Phase 3 | v1.1 | rule "regions roll up to NA, EMEA, APAC"; stage-aging query ranked above a stale one; rule "internal and test accounts are out" as an instruction | stage aging | 3 | 14 of 20 | stage aging / partner-org count, caught by the rerun | keep the first two; rescope the rule · RevOps lead |
| 3 | Phase 3, next day | v1.2 | the rule scoped to org type; "pipeline created by quarter" query promoted | partner-org count, pipeline created | 3 | 14 of 20 | partner-org count / win rate, 1 of 3 runs | keep; raise the top-100 conflict; pipeline created goes to the platform owner · RevOps lead |
| 4 | Phase 3 | v1.3 | top-100 conflict resolved: metric v3 with the stage rule inside; "Slipped deal" labels and link; win rate's time parameter required | pipeline, slipped deals, both win-rate questions | 3 | 18 of 20 | all four, 3 of 3 / none | keep · RevOps lead and FP&A partner |
| 5 | Phase 3 | v1.4 | column description on the stage field: "the first stage is not pipeline" | none expected | 3 | 18 of 20 | none / none | revert: the rule already lives in the metric · RevOps lead |

Stop rule: not fired. Rounds 2 and 3 held the total but each landed its predicted fix; round 4 moved it by 4; round 5 was the only quiet round.
Phase 3 ended at 18 of 20, and the set grew to 50 for the launch gate.

# Iteration ledger: <domain> · golden set of <n> · run-to-run variation: <questions>, from <k> identical runs · score = right on every run

| Round | When | Context | Changes | Predicted fixes | Runs | Score | Fixed / broken | Decision, approved by |
|---|---|---|---|---|---|---|---|---|
| <n> | <when> | <v> | <changes> | <questions, written before the run> | <k> | <your run> | <questions> / <questions> | <keep / revert> · <owner> |

Stop rule: <fired | not fired | five-round cap reached>, and why.
Optional columns: cost and latency per question, from the traces.

# Sponsor readout: Sales · end of Phase 2 · delivery lead

Status: on track. Gate this week: metrics published and scored. Met.

## Score against the golden set
Six metrics, one run per question: 14 of 20, up from the bare-schema baseline of 7 of 20.

## What changed
- Six metrics approved, each with an owner, a version and its provenance
- Most conflicts settled on provenance; the RevOps lead decided the split ones; one escalated to you (below)
- Win rate now excludes disqualified deals: the dashboard version predated that stage, and every recent analyst query excluded it

## Decisions needed from you
- For multi-year deals, is the amount first-year ARR or total contract value? The RevOps lead and the FP&A partner
  disagree. Until you decide, the agent flags the conflict instead of answering. Decided at this readout: first-year ARR.
- Your question on why EMEA dropped waits until the numbers are released. It sits in the backlog with a date.

## Risks
- The graders must seal 30 more questions by the end of Phase 3 (owner: eval owner)
- The FP&A partner's hour in Phase 3, for any decision on the top 100 (owner: delivery lead)

## Next week
Measure run-to-run variation, then fix from traces; the RevOps lead approves each change; the set grows to 50.

# Sponsor readout: <domain> · <week or gate> · <delivery lead>

Status: on track | at risk | blocked. Gate this week: <gate>, <met / not yet>

## Score against the golden set
<context version, score, change since last week, inside run-to-run variation or not>

## What changed
<context changes this week, and how each conflict was settled>

## Decisions needed from you
<decision, why it matters, by when>

## Risks
<live rows from the risk log, with owner and mitigation>

## Next week
<plan to the next gate>

# Risk log: Sales · opened at the kickoff · delivery lead · as it stood after launch

| # | Risk | Signal to watch | Likelihood, impact | Owner | Mitigation | Status |
|---|---|---|---|---|---|---|
| 1 | No per-call trace export | the chat tool exports a nightly summary only | H, H | agent builder | build a per-call export; read one trace end to end before Phase 1 | closed in Phase 0 |
| 2 | Security review opens late | no walk-through booked | M, H | security lead | walk the data-flow diagram at the kickoff | closed in Phase 0 |
| 3 | Why-questions before the numbers are right | the CRO asked in Phase 2 | M, M | delivery lead | deferred by the brief; dated in the backlog | closed: why-set ran after launch |
| 4 | Pipeline-creation questions cannot pass | the same miss on every run, the data (Phase 3) | H, M | platform owner | held back by name; build the "created and still open" state | closed after launch |
| 5 | A schema change breaks served queries | the platform owner's schema alert | M, H | platform owner | the regression set reruns on every alert | fired after launch; fixed by the team alone |

Escalate a High/High to the exec sponsor the same day.

# Risk log: <domain> · opened <date> · <delivery lead>

| # | Risk (an R1 symptom, or new) | Signal to watch | Likelihood, impact | Owner | Mitigation | Status |
|---|---|---|---|---|---|---|
| 1 | <risk> | <the observable signal> | <H/M/L>, <H/M/L> | <role> | <action, due> | open / closed <date> |

Escalate a High/High to the exec sponsor the same day.

Phase 4: release and hand over

# Release: Sales context v1.0 · Phase 4 · approved by the RevOps lead; signed by the CRO
Launch gate: governed 20 of 20, right on every run, refusing all 3 out-of-scope questions; exploratory 25 of 30 on every run
against a bar of 24; everything that passed at Phase 3 still passes; random pass audit done

| Type | Name | Version | Change | Settled by |
|---|---|---|---|---|
| metric | open_pipeline_top_accounts | v3 | added; v2 retired, kept on record | both owners (top-100 conflict) |
| metric | win_rate | v4 | added: won / (won + lost), by close date, disqualified out | provenance (recency) |
| metrics | closed_won_arr, open_pipeline, avg_deal_size, stage_conversion | v1 | added | provenance |
| business terms | Slipped deal (pushed, rolled, slid); Top accounts; 10 more | 2; 2; 1 | added; linked | none needed |
| rules | quarter-means-fiscal; regions-roll-up; flag-conflicts | 1 | added | RevOps lead |
| trusted queries | 38 | 1 | added | provenance |

Held back by name: pipeline-creation questions, until the "created and still open" state exists
Known limits: why-questions, follow-up turns and entity lookups are not measured
Rollback: repoint to the previous context version with the previous harness build; the RevOps lead decides
Next: the open conflicts on released questions, then the why-layer

# Release: <domain> context v<x.y> · <date> · approved by <domain owner>; signed by <exec sponsor>
Launch gate: <bar per lane> on the sealed set: <your run>; regression <held / not>; random pass audit <done>

| Type | Name | Version | Change | Settled by |
|---|---|---|---|---|
| <metric / rule / business term / trusted query> | <name> | <v> | <added / changed / retired> | <provenance / owner> |

Held back by name: <kinds of question, and what brings them back>
Known limits: <unmeasured question types; open conflicts>
Rollback: <previous context version and harness build; who decides>
Next: <next-version backlog>

# Handover checklist: Sales · Phase 4 · signed by the RevOps lead and the delivery lead
- [x] Every metric (6), rule (3) and business term (12) has an owner on the company's side and a version
- [x] The golden set (50) is sealed, owned by the eval owner, and versioned; the 45 always-right questions run as the regression set
- [x] The conflict records and the iteration ledger are edited by their owners, not by the delivery team
- [x] One pass run together: the open conflicts resolved, regression 45 of 45; run by the agent builder, graded by the eval owner
- [ ] One pass run alone within two weeks (done after launch: a schema change, rerun and fixed by the team)
- [x] Production traces reach the company's log store per call; user corrections reach the conflict list as proposed changes
- [x] A rerun fires on every context change (the full set) and on every schema change (the regression set); the platform owner owns the alert
- [x] The next-version backlog and the domain-two plan agreed
- [x] Postmortem and pattern write-up drafted, reviewed by the RevOps lead
Open: pipeline-creation questions wait on the new data state; the arbiter for words finance and sales both own

# Handover checklist: <domain> · Phase 4 · signed <date> by <domain owner>, <delivery lead>
- [ ] Every metric, rule and business term has an owner on your side and a version
- [ ] The golden set is sealed, owned by the eval owner and versioned; its always-right questions run as the regression set
- [ ] The conflict records and the iteration ledger are edited by their owners
- [ ] One pass run together
- [ ] One pass run alone within two weeks (due <date>)
- [ ] Production traces reach your log store per call; user corrections reach the conflict list as proposed changes
- [ ] A rerun fires on every context change and every schema change; <owner> owns the alert
- [ ] Next-version backlog and domain-two plan agreed
- [ ] Postmortem and pattern write-up drafted and reviewed by <domain owner>
Open: <what waits, and on whom>

Pattern: a definition kept in a spreadsheet hid a conflict until Phase 3
Domain shape: B2B software, sales pipeline; 20 governed and 30 exploratory questions at launch; CRM and billing; no agent at the start
What happened: graders disputed the top-100 source in Phase 1; the FP&A partner chose the board pack, and the choice stayed on the item.
  Finance's ranking lived in a spreadsheet, so the only ranking in query history was the deal-count tile.
  The 31 queries that pasted finance's list were set aside as a hard-coded list, which raised no conflict.
  Phase 2 approved the deal-count ranking; the Phase 3 raw trace exposed it; the owners resolved it in 20 minutes.
What moved / what did not: a rule served too widely broke one question, caught by the rerun and fixed the next day;
  the top-100 fix, a term link and a required time parameter took the score from 14 to 18 of 20; a column description moved nothing and was reverted
Platform: a set-aside list of IDs with no known source should ask the owners where it came from
Throw away: the "created and still open" state was specific to this warehouse
Missing capability: a sealing dispute should also reach the conflict list, with no sealed value attached
Reviewed for anonymisation by: the RevOps lead

Pattern: <one line>
Domain shape: <industry, kinds of question, lane split, estate size, starting point>
What happened: <the conflict, miss or result, and the round that showed it>
What moved / what did not: <from the ledger>
Platform: <fixes the next engagement will need too>
Throw away: <fixes specific to this estate>
Missing capability: <one line each>
Reviewed for anonymisation by: <domain owner>, <date>

# Postmortem: Sales · Phase 4
Trigger: engagement end · Author: delivery lead · Reviewed with: RevOps lead, CRO

## Timeline
Phase 1: graders disputed the source for the pipeline question; the FP&A partner chose the board pack; recorded on the item only
Phase 1: 20 sealed; by design, the context side never saw the item's history
Phase 2: generation set aside 31 queries with a pasted list of 100 account IDs; no conflict raised
Phase 2: the top-100 metric was approved in a batch of six, ranking customers by deal count
Phase 3: the raw trace of the pipeline question showed the ranking
Phase 3: the owners resolved the conflict the next day; the rerun fixed three questions on every run

## Conditions
The wall between the key and the context protected the key and also hid a conflict.
A set-aside list is not a conflict, so it never reached the owner.
Nobody approving the batch asked where the top-100 ranking came from.
Gate 2 passed the metric with its stage rule served only as an instruction, outside its SQL.

## Actions
| Action | Owner | Due |
|---|---|---|
| A sealing dispute about a definition also goes to the conflict list, with no sealed value | eval owner | domain two's Phase 0 |
| Every pasted list of IDs is traced to its source before the concept it touches is approved | context engineer | domain two's Phase 2 |
| Gate 2 reads each metric's SQL for the house rules that apply before approval | domain owner | domain two's Phase 2 |
New R1 row: symptom "graders disagree on a source"; cause "a conflict settled at sealing, never resolved"; fix "send it to the conflict list as well"

# Postmortem: <domain> · <date>
Trigger: <engagement end | gate slipped | wrong governed number | a break | rollback | stopped early>
Author: <delivery lead> · Reviewed with: <domain owner>, <exec sponsor>

## Timeline
<when>: <what happened>, in order, no interpretation

## Conditions
<what in the plan, roster or tooling made each problem likely>

## Actions
| Action | Owner | Due |
|---|---|---|
| <a checklist, template, runbook step or platform change> | <role> | <date> |
New R1 rows: <symptom, cause, fix>