Methodology

How every number on this site is produced.

Everything here is generated by an internal, command-only harness (harness/ in the repo) that drives real Whetstone — the same five MCP tools the server ships. It never runs in CI; a maintainer runs it and commits the artifacts under harness/out/, which this site only reads.

The cold → capture → warm loop

This is the same three-stage loop the Home page walks through — there it’s introduced as “Cold, captured, then warm” for a first-time reader; here it’s the harness running that same loop mechanically, at scale, over every scenario.

For each scenario (a skill’s prompt + an arbitrary subjective preference):

1. Cold — generate the output with an empty store. Record whether the preference was honored (usually not — the “before”).
2. Capture — replay the scripted correction through Whetstone’s capture tool, exactly as a user correcting the skill would. The preference becomes a tracked learning or issue, committed to the per-skill store.
3. Warm — recall the now-captured store, inject that learned layer, regenerate, and check again. Reinforce each learning (a repeated correction) so its weight/recurrence grows, and iterate until the preference is honored for a set streak of consecutive warm runs.

Generation is done by the live model via the Claude Agent SDK with the scenario’s skill mounted. A deterministic stub generator exercises the identical pipeline with no API calls, so the wiring is verifiable without spend.

What “honored” means

Each preference carries a deterministic check against the generated primary output — table.py for great-tables, index.html for frontend-design, plot.py for plotnine, plot.R for ggplot2, deck.py for pptx (comments — and, for Python, docstrings — stripped so prose can’t fool it): a substring, a regex, or their absence. That check — not a subjective judgement — decides whether a run honored the preference.

The two metrics

• Runs-to-stick counts warm runs only (the cold baseline never counts), so a lucky cold run can’t claim credit before the correction existed.
• Value-over-time is the learning’s weight/recurrence straight from the recall payload each run.

Honesty

No metric is hand-set. A metric a scenario can’t support honestly stays null with a note — the same stance whetstone metrics takes for its labeled KPIs. The store runs on the sentence-transformers backend (the calibrated one); when retrieval genuinely misses a seeded entry, the site shows that miss rather than hiding it.

The labeled KPIs — the calibration harness

Three §11 KPIs need a labeled ground truth that ordinary usage never has, so the runtime metrics tool returns them as null by design: capture-rate, regressions-prevented, and retrieval-precision. A second command-only harness, python -m harness.calibrate, computes them against a small hand-labeled set (harness/calibration/labels.yaml) — a human vouches for every case — and writes out/calibration.json, which the Metrics page reads:

• retrieval-precision — for each labeled (intent → relevant scopes) case, seed a store, recall, and score the returned scopes against the labels (precision / recall / F1). Key-free.
• capture-rate — for each labeled correction turn, drive capture and count how often it lands as committed / reinforced rather than a no-op. Key-free.
• regressions-prevented — a key-free proxy checks that an in-scope issue is recalled for an intent that would violate it (so the model is warned before it can regress); a live --agent cold-vs-warm generation is the published figure.

The same command also scores a fourth, experimental entry in calibration.json that is not one of the §11 product KPIs: same_polarity_contradiction (§M7c) — precision/recall of the narrow antonym/negation heuristic in capture’s dedup path (see Tools’s possible_contradiction status), scored against its own labeled section in the same labels.yaml. This one is not shown on the Metrics page — it measures a spike-status heuristic, not a headline product number — but it is computed the same honest way: a small hand-labeled set of contradiction pairs (should fire) and genuine-duplicate pairs (should not), run through real capture.

The boundary is deliberate and stays intact: the runtime metrics tool still returns null for these three — only this published showcase, linked to the labeled set and the runner that produced it, shows the numbers. When calibration.json is absent the Metrics page falls back to the same null-with-note it always showed.

What was taught, and how “honored” is judged

Every preference below is an arbitrary house-style choice — the point is only that a correction, once given, is tracked and re-applied, not that the choice is objectively right. The check column is the exact deterministic test that decides whether a generated table honored the preference.

scenario	diff.	preference	polarity	scope	rule	check
airquality_monthly_summary	medium	one-decimal	learning	number formatting	Round the temperature, wind, and ozone values to one decimal place.	regex · decimals\s*=\s*1
airquality_monthly_summary	medium	labels-darkgreen	learning	header styling	Style the column-label row with a dark green background (#0B6E4F) and white text — apply it to loc.column_labels() with style.fill(color="#0B6E4F") and style.text(color="white").	regex · (?i)#0b6e4f
ggplot2_gtcars_hp_price	hard	palette-accent	learning	color encoding	Color-encode country of origin with the ColorBrewer qualitative `Accent` palette via `scale_color_brewer(palette = "Accent")`.	regex · scale_color_brewer\((?:[^()]|\((?:[^()]|\([^()]*\))*\))*palette\s*=\s*["']Accent["']
ggplot2_gtcars_hp_price	hard	price-plain-commas	learning	scales	Format the price (msrp) axis as plain comma-separated numbers with no dollar sign, e.g. `scale_y_continuous(labels = scales::comma)`.	regex · scale_y_continuous\((?:[^()]|\((?:[^()]|\([^()]*\))*\))*scales::(?:comma\b|label_comma\(\))
ggplot2_gtcars_hp_price	hard	legend-bottom	learning	legend placement	Place the legend at the bottom with `theme(legend.position = "bottom")`.	regex · legend\.position\s*=\s*["']bottom["']
ggplot2_gtcars_hp_price	hard	points-large	learning	geoms	Make the scatter points larger, `size = 4.5` in geom_point.	regex · geom_point\((?:[^()]|\((?:[^()]|\([^()]*\))*\))*size\s*=\s*4\.5\b
ggplot2_gtcars_hp_price	hard	points-translucent	learning	geoms	Make the scatter points translucent, `alpha = 0.6` in geom_point.	regex · geom_point\((?:[^()]|\((?:[^()]|\([^()]*\))*\))*alpha\s*=\s*0\.6
ggplot2_islands_top_sizes	easy	fill-forest-green	learning	color encoding	Fill the bars with forest green `#3A5A40`, not the house blue accent.	regex · (?i)(?:geom_col|geom_bar)\((?:[^()]|\((?:[^()]|\([^()]*\))*\))*fill\s*=\s*["']#3a5a40["']
gtcars_hp_price	easy	price-usd	learning	currency formatting	Format the price (msrp) column as US dollars with no decimal places.	code_contains · fmt_currency
gtcars_hp_price	easy	price-heatmap-warm	learning	color encoding	Heat-map the price (msrp) column with a warm yellow-orange-brown ramp — use the ColorBrewer `YlOrBr` palette via data_color(columns="msrp", palette="YlOrBr"). Never a green or blue scale.	code_contains · YlOrBr
gtcars_manufacturer_rollup	hard	rollup-by-manufacturer	learning	data organization	Don't list one row per car model. Roll the data up to one row per manufacturer (`mfr`), showing the mean horsepower (`hp`) and mean price (`msrp`) across that manufacturer's whole lineup — compute this with a real groupby aggregation over the raw rows, not a per-model listing.	regex · groupby\([\s\S]*?mfr[\s\S]*?mean
gtcars_manufacturer_rollup	hard	rollup-count-column	learning	row count column	Add a "Models" column showing how many models from that manufacturer went into each average - the reader should be able to tell a 2-model average from a 12-model one.	regex · (Models[\s\S]*?(size\(\)|count\(\)|'count'|'size')|(size\(\)|count\(\)|'count'|'size')[\s\S]*?Models)
gtcars_top10_by_country	medium	powertrain-spanner	learning	column grouping	Group the drivetrain and transmission columns under a single "Powertrain" spanner.	code_contains · tab_spanner
gtcars_top10_by_country	medium	country-group-teal	learning	row group styling	Give the country row-group label rows a teal background (#0F766E) with white text — apply it to loc.row_groups() with style.fill(color="#0F766E") and style.text(color="white").	regex · (?i)#0f766e
landing_hero	medium	accent-burnt-orange	learning	color palette	Use the burnt-orange hex `#FF6B00` as the accent color for the primary CTA button background and for links.	regex · (?i)#ff6b00
landing_hero	medium	hero-wide-tracking	learning	typography	Give the main headline wide letter-spacing of exactly 0.2em (`letter-spacing: 0.2em`).	regex · letter-spacing:\s*0.2em
pitch_deck	medium	mono-font	learning	typography	Set every text run's font to Consolas (`run.font.name = "Consolas"`) — titles and body, on all slides.	regex · font\.name\s*=\s*'Consolas'|(\w+)\s*=\s*'Consolas'[\s\S]*?font\.name\s*=\s*\1\b
pitch_deck	medium	square-format	learning	aspect ratio	Use a SQUARE 1:1 slide format: set `prs.slide_width = Inches(10)` AND `prs.slide_height = Inches(10)` on the Presentation (import Inches from `pptx.util`).	code_contains · slide_height = Inches(10)
pitch_deck_reorder	hard	cta-second	learning	slide order	Don't save the call-to-action for last. Move the "Install Marrow" call-to-action slide to be the SECOND slide - right after the title, before the problem/idea/how-it-works slides. Lead with the install command, then explain why it matters.	regex · Install Marrow[\s\S]*Big test suites
pitch_deck_reorder	hard	why-now-slide	learning	slide structure	Insert a new slide titled "Why now" between "The idea" and "How it works", making the case that test suites have grown large enough that manual coverage tracking has become untenable - this is new content the outline doesn't ask for, added because it strengthens the pitch.	regex · Watch one run[\s\S]{150,}?Why now[\s\S]{80,}How it works
plotnine_gtcars_hp_faceted	hard	facet-by-country	learning	layout structure	Don't color-encode country of origin on one combined scatter plot. Split it into small multiples instead - facet by country of origin with facet_wrap(), one panel per country.	regex · facet_wrap\([\s\S]*?ctry_origin
plotnine_gtcars_hp_faceted	hard	facet-free-y	learning	scales	Let each facet panel use its own y-axis range (`scales="free_y"`) instead of a shared axis, since some countries have a much narrower price range than others and a shared axis would flatten them to near-invisible lines.	regex · scales\s*=\s*'free_y'
plotnine_gtcars_hp_msrp	hard	palette-dark2	learning	color encoding	Color-encode country of origin with the ColorBrewer qualitative `Dark2` palette via `scale_color_brewer(type="qual", palette="Dark2")`.	code_contains · Dark2
plotnine_gtcars_hp_msrp	hard	price-log-axis	learning	scales	Put the price (msrp) axis on a log scale with `scale_y_log10()`.	code_contains · scale_y_log10
plotnine_gtcars_hp_msrp	hard	legend-bottom	learning	legend placement	Place the legend at the bottom with `legend_position="bottom"` in theme().	regex · legend_position\s*=\s*["']bottom["']
plotnine_gtcars_hp_msrp	hard	points-large	learning	geoms	Make the scatter points larger, `size=3` in geom_point.	regex · size\s*=\s*3\b
plotnine_gtcars_hp_msrp	hard	points-translucent	learning	geoms	Make the scatter points translucent, `alpha=0.6` in geom_point.	regex · alpha\s*=\s*0\.6
plotnine_islands_top_sizes	easy	theme-classic	learning	theme	Use plotnine's `theme_classic()` as the base theme instead of the house `theme_minimal`.	code_contains · theme_classic
plotnine_sp500_close_trend	medium	line-crimson	learning	color encoding	Draw the single trend line in crimson `#D1495B`, not the house blue accent.	regex · (?i)#d1495b
plotnine_sp500_close_trend	medium	line-thick	learning	geoms	Give the trend line a thicker stroke, `size=1.5`, in geom_line.	regex · geom_line\([^)]*size\s*=\s*1\.5
pricing_cards	medium	courier-body-font	learning	typography	Set the base body font to Courier: `font-family: 'Courier New', monospace` on the root/body.	code_contains · Courier New
pricing_cards	medium	sharp-corners	issue	corner radius	Force perfectly square corners across the whole page: add the rule `* { border-radius: 0 !important; }` as the first rule in the stylesheet. Never round any corner.	regex · (?i)border-radius:\s*0\s*!important
pricing_comparison_table	hard	cards-to-comparison-table	learning	layout structure	Don't lay the tiers out as three side-by-side cards. Restructure this as a single comparison table instead: one row per feature, one column per tier, so a reader can scan across a row and compare the tiers directly at a glance.	code_contains · <table
pricing_comparison_table	hard	enterprise-before-team	learning	column order	Order the tier columns as Solo, Enterprise, Team - not Solo, Team, Enterprise. Lead with the free tier, then the ceiling (Enterprise), and put the recommended middle tier (Team) last so it's the final impression the reader leaves with.	regex · Start free[\s\S]*Contact sales[\s\S]*Start 14-day trial
sp500_monthly_performance	hard	negatives-accounting	learning	number formatting	Show negative numbers in accounting style — wrapped in parentheses, not with a minus sign.	regex · accounting\s*=\s*True
sp500_monthly_performance	hard	pctchange-heatmap-amber	learning	color encoding	Heat-map the percent-change column with a warm amber-to-red ramp — use the ColorBrewer `YlOrRd` palette via data_color(columns=..., palette="YlOrRd"). Not a green or red-green scale.	code_contains · YlOrRd
towny_growth_trends	hard	pct-force-sign	learning	percentage formatting	Show the percentage-change columns as percentages with an explicit + sign on positive values.	regex · force_sign\s*=\s*True
towny_growth_trends	hard	density-magenta	issue	color encoding	Never fill growth with green here (house rule). Heat-map the density columns with a magenta ramp instead — use the ColorBrewer `PuRd` palette via data_color(columns=[...], palette="PuRd").	code_contains · PuRd

Where every number comes from

The site reads only committed files under harness/out/; nothing is computed at page-render time beyond formatting. Per scenario:

file	feeds
cold/<primary output>	the Before panel (table.{py,png,html} / index.html / deck.py)
warm/<primary output>	the After panel (same set, generated with the learned layer)
learned_layer.txt	the middle learned-layer panel (verbatim injected prompt text)
recall.json	the structured recall payload behind the learned layer
runs.jsonl	per-run honored map + weight/recurrence — the raw record
summary.json	per-preference runs-to-stick + value-over-time + store usage KPIs
../metrics.json	the cross-scenario aggregate and the dashboard charts

Each headline number traces to summary.json / metrics.json, which trace to runs.jsonl, which is written straight from the tool calls.

Coming next

v1 is the visual skill class, shown across five skills (Great Tables, frontend design, plotnine, ggplot2, and PowerPoint decks) and two languages (Python and R) — the substrate doesn’t care which. The same substrate — attach, recall, capture, revise — applies unchanged to a code-improvement skill (objective ground truth: tests, lint, types) and a context-organizing skill; those are the next classes.

Reproduce it

pip install -e ".[showcase,embeddings]"
export ANTHROPIC_API_KEY=sk-ant-...     # or harness/.env
python -m harness.run --agent            # regenerate out/ (paid, command-only)
python -m harness.calibrate              # labeled KPIs -> out/calibration.json (key-free)
python -m harness.metrics                # aggregate -> out/metrics.json (folds in calibration.json)