Metrics

Tracked, and stays valued — measured from telemetry.

Two headline metrics, both read straight from Whetstone’s events.jsonl telemetry and the recorded runs — never hand-set:

Runs-to-stick — how many warm (post-correction) runs pass before a preference is applied without needing to be re-corrected. Lower is faster learning.
Value-over-time — a learning’s weight / recurrence trajectory across later runs: does the correction keep being valued, or decay away?

Aggregate (all skills)

metric	value
scenarios	17
preferences (total)	38
wrong in the cold run	35
stuck after correction	37 (0.9737)
median runs-to-stick	2

Per skill

skill	scenarios	stuck after correction	median runs-to-stick
frontend-design	3	5/6 (0.8333)	2
ggplot2	2	6/6 (1.0)	2.0
great-tables	6	12/12 (1.0)	2.0
plotnine	4	10/10 (1.0)	2.0
pptx	2	4/4 (1.0)	2.0

Per-preference detail

skill	scenario	preference	polarity	runs-to-stick	final weight
frontend-design	landing_hero	accent-burnt-orange	learning	2	0.667
frontend-design	landing_hero	hero-wide-tracking	learning	2	0.667
frontend-design	pricing_cards	courier-body-font	learning	2	0.667
frontend-design	pricing_cards	sharp-corners	issue	2	—
frontend-design	pricing_comparison_table	cards-to-comparison-table	learning	2	0.800
frontend-design	pricing_comparison_table	enterprise-before-team	learning	didn’t stick	—
ggplot2	ggplot2_gtcars_hp_price	palette-accent	learning	2	0.667
ggplot2	ggplot2_gtcars_hp_price	price-plain-commas	learning	2	0.667
ggplot2	ggplot2_gtcars_hp_price	legend-bottom	learning	2	0.667
ggplot2	ggplot2_gtcars_hp_price	points-large	learning	2	0.667
ggplot2	ggplot2_gtcars_hp_price	points-translucent	learning	2	0.667
ggplot2	ggplot2_islands_top_sizes	fill-forest-green	learning	2	0.667
great-tables	airquality_monthly_summary	one-decimal	learning	2	0.667
great-tables	airquality_monthly_summary	labels-darkgreen	learning	2	0.667
great-tables	gtcars_hp_price	price-usd	learning	2	0.667
great-tables	gtcars_hp_price	price-heatmap-warm	learning	2	0.667
great-tables	gtcars_manufacturer_rollup	rollup-by-manufacturer	learning	2	0.667
great-tables	gtcars_manufacturer_rollup	rollup-count-column	learning	2	0.667
great-tables	gtcars_top10_by_country	powertrain-spanner	learning	2	0.667
great-tables	gtcars_top10_by_country	country-group-teal	learning	2	0.667
great-tables	sp500_monthly_performance	negatives-accounting	learning	2	0.667
great-tables	sp500_monthly_performance	pctchange-heatmap-amber	learning	2	0.667
great-tables	towny_growth_trends	pct-force-sign	learning	2	0.667
great-tables	towny_growth_trends	density-magenta	issue	2	—
plotnine	plotnine_gtcars_hp_faceted	facet-by-country	learning	2	0.667
plotnine	plotnine_gtcars_hp_faceted	facet-free-y	learning	2	0.667
plotnine	plotnine_gtcars_hp_msrp	palette-dark2	learning	2	0.667
plotnine	plotnine_gtcars_hp_msrp	price-log-axis	learning	2	0.667
plotnine	plotnine_gtcars_hp_msrp	legend-bottom	learning	2	0.667
plotnine	plotnine_gtcars_hp_msrp	points-large	learning	2	0.667
plotnine	plotnine_gtcars_hp_msrp	points-translucent	learning	2	0.667
plotnine	plotnine_islands_top_sizes	theme-classic	learning	2	0.667
plotnine	plotnine_sp500_close_trend	line-crimson	learning	2	0.667
plotnine	plotnine_sp500_close_trend	line-thick	learning	2	0.667
pptx	pitch_deck	mono-font	learning	2	0.667
pptx	pitch_deck	square-format	learning	2	0.667
pptx	pitch_deck_reorder	cta-second	learning	2	0.667
pptx	pitch_deck_reorder	why-now-slide	learning	2	0.667

Labeled KPIs (calibration harness)

The three §11 labeled KPIs — capture_rate, regressions_prevented, retrieval_precision — the product metrics tool returns null by design (ordinary runtime usage has no ground truth). The internal calibration harness (python -m harness.calibrate) scores them against a small hand-labeled set (harness/calibration/labels.yaml) and writes out/calibration.json, which this page reads. The runtime tool still returns null — only this published showcase, linked to the labeled set, shows the numbers.

KPI	value
capture_rate	0.75
regressions_prevented	1.0 Key-free proxy: verifies the in-scope issue was recalled for a violating intent (not a live generation). The published figure uses the –agent cold-vs-warm variant.
retrieval_precision	0.5

Computed on the sentence-transformers backend at 2026-07-29T05:28:21.326640+00:00 from harness/calibration/labels.yaml.

--- title: "Metrics" subtitle: "Tracked, and stays valued — measured from telemetry." --- Two headline metrics, both read straight from Whetstone's `events.jsonl` telemetry and the recorded runs — never hand-set: - **Runs-to-stick** — how many warm (post-correction) runs pass before a preference is applied without needing to be re-corrected. Lower is faster learning. - **Value-over-time** — a learning's `weight` / `recurrence` trajectory across later runs: does the correction *keep* being valued, or decay away? ```{python} # | echo: false # | output: asis import sys sys.path.insert(0, ".") from _charts import charts_html # noqa: E402 from _showcase import load_metrics, not_generated_note # noqa: E402 from IPython.display import HTML, Markdown, display # noqa: E402 m = load_metrics() if m is None: display(not_generated_note("The runs-to-stick chart and value-over-time trajectories")) else: # All skills. `aggregate` already spans every scenario in metrics.json; the charts and detail # table read `scenarios` directly, so the numbers here cover great-tables, frontend-design, and # pptx together — the same set the Before / After page shows. a = m["aggregate"] display(Markdown( "### Aggregate (all skills)\n\n" "| metric | value |\n| --- | --- |\n" f"| scenarios | {a['scenarios']} |\n" f"| preferences (total) | {a['preferences_total']} |\n" f"| wrong in the cold run | {a['preferences_wrong_in_cold']} |\n" f"| stuck after correction | {a['preferences_stuck']} ({a['stuck_rate']}) |\n" f"| median runs-to-stick | {a['median_runs_to_stick']} |\n" )) # Per-skill breakdown — each slice has the same shape as `aggregate`, so the same substrate is # shown holding across three different skill classes, not just one. skills = m.get("skills", {}) if len(skills) > 1: rows = ["\n### Per skill\n", "| skill | scenarios | stuck after correction | median runs-to-stick |", "| --- | --- | --- | --- |"] for name, s in sorted(skills.items()): rows.append(f"| {name} | {s['scenarios']} | " f"{s['preferences_stuck']}/{s['preferences_total']} ({s['stuck_rate']}) | " f"{s['median_runs_to_stick']} |") display(Markdown("\n".join(rows))) # The charts (static SVG, light + dark surfaces from the validated dataviz palette). display(HTML(charts_html(m))) # Companion table view (the relief the light-surface contrast WARN requires): final weight + # runs-to-stick per preference, in the same order as the charts. rows = ["\n### Per-preference detail\n", "| skill | scenario | preference | polarity |" " runs-to-stick | final weight |", "| --- | --- | --- | --- | --- | --- |"] for scn, s in m["scenarios"].items(): for pid, pref in s["preferences"].items(): traj = [v for v in pref.get("value_over_time", []) if v.get("weight") is not None] fw = f"{traj[-1]['weight']:.3f}" if traj else "—" rts = pref.get("runs_to_stick") display_rts = "didn't stick" if rts is None else rts rows.append(f"| {s.get('skill', '—')} | {scn} | {pid} | {pref['polarity']} |" f" {display_rts} | {fw} |") display(Markdown("\n".join(rows))) ``` ## Labeled KPIs (calibration harness) The three §11 *labeled* KPIs — `capture_rate`, `regressions_prevented`, `retrieval_precision` — the product `metrics` tool returns **`null` by design** (ordinary runtime usage has no ground truth). The internal calibration harness (`python -m harness.calibrate`) scores them against a small **hand-labeled set** ([`harness/calibration/labels.yaml`](https://github.com/HrudithL/whetstone/blob/main/harness/calibration/labels.yaml)) and writes `out/calibration.json`, which this page reads. The runtime tool still returns null — only this published showcase, linked to the labeled set, shows the numbers. ```{python} # | echo: false # | output: asis if m is None: display(not_generated_note("The labeled calibration KPIs")) else: k = m["showcase_only_kpis"] src = k.get("calibration_source") rows = ["| KPI | value |", "| --- | --- |"] for key in ("capture_rate", "regressions_prevented", "retrieval_precision"): entry = k[key] val = entry.get("value") note = entry.get("note") if val is None: shown = "— (not yet calibrated)" elif note: # A non-null value can still carry a caveat (e.g. regressions_prevented's key-free # recall-only proxy vs. the live --agent cold-vs-warm figure methodology.qmd promises) — # show it inline so a value is never presented as more authoritative than it is. shown = f"{val} {note}" else: shown = val rows.append(f"| {key} | {shown} |") display(Markdown("\n".join(rows))) if src: display(Markdown( f"\nComputed on the `{src['backend']}` backend at {src['generated_at']} " f"from `{src['labels']}`." )) else: display(Markdown( "\nRun `python -m harness.calibrate` (needs the `[showcase,embeddings]` extra) " "to populate these from the labeled set." )) ```