Skip to content

Milestone v0.2.3 — Result packs, fingerprint reporting, readability, drift, honest browser/MCP bounds

Milestone theme

v0.2.3 closes a coherence lap after v0.2.2: make campaign outputs publishable as result packs, make fingerprints drive reporting (not only raw dumps), keep reports readable, prevent schema/runtime drift under default CI, and keep browser and MCP story realistic with honest capability boundaries.

Status: Complete — v0.2.3 is shipped (git tag v0.2.3, CHANGELOG [0.2.3], docs/releases/v0.2.3.md). Later milestones: v0.2.4 (shipped) · v0.2.5 (shipped).

Arc context: v0.2.0 · Closed tracking: docs/tracking/v0.2.3-campaign.md.

Delivered (current tree)

Theme What shipped
Verification surface hardening docs/workflows/verification.md — matrix of just ci, just validate-artifacts, just test, live/smoke, with uv run fallbacks; README/AGENTS cross-links.
Drift checks tests/test_schema_drift_contracts.py (+ collect_committed_artifact_drift_errors) sweeps committed examples/ / fixtures/; just validate-artifacts recipe; docs/audits/schema-drift-contracts-inventory.md inventory and exclusions.
Campaign result-pack refinement alwm benchmark campaign pack / pack-check; campaign_result_pack schema; examples/campaign_result_packs/ (e.g. minimal_offline, multi_suite); workflow in docs/workflows/benchmark-campaigns.md.
Fingerprint interpretation fingerprint_axis_interpretation (and related blocks) in campaign-analysis.json / longitudinal summaries; comparative Markdown (fingerprint compare, axis interpretation, ranked spreads/instability hints) aligned with alwm benchmark longitudinal keys.
Browser reporting realism Fixture-backed browser benchmarks; browser evidence surfaced in benchmark/campaign report Markdown where applicable; docs/architecture/browser.md + capability taxonomy stay aligned with partial Playwright / partial MCP.
Pack comparison alwm benchmark campaign compare-packs → pack-compare.json (kind campaign_result_pack_comparison) + pack-compare-report.md; example examples/campaign_result_packs/compare_minimal_vs_multi/; optional --repo-root for portable paths.

Related (cross-cutting): Report readability passes on campaign summary, comparative report, and semantic summary; judge/semantic sections where applicable—see implementation log for file-level notes.

Goals (original scope — largely satisfied)

1. Campaign result pack publication

  • Publication workflow for git-friendly bundles: alwm benchmark campaign pack, campaign-result-pack.json, INDEX.md, mirrored layout—documented in docs/workflows/benchmark-campaigns.md and examples/campaign_result_packs/.
  • Validation: alwm validate … campaign_result_pack; alwm benchmark campaign pack-check (optional --strict portability).

2. Fingerprint-driven reporting

  • comparison_fingerprints and campaign_experiment_fingerprints inform comparative and campaign Markdown and JSON mirrors; terminology aligns with alwm benchmark longitudinal.

3. Report readability

  • Scannable campaign-summary, comparative, semantic outputs; reduced gratuitous duplication (see implementation log for campaign/reporting edits).

4. Schema / runtime drift prevention

  • JSON Schema, Pydantic, artifacts.py, alwm validate, and committed examples guarded by just ci and just validate-artifacts / drift tests.

5. Browser / MCP realism with honest capability boundaries

Non-goals

  • Hosted dashboards, SaaS, or cloud campaign runners as exit criteria.
  • Remote / IDE-hosted MCP or full remote browser protocol as a milestone deliverable (see v0.2.0 non-goals).
  • Changing default rubric scoring math unless explicitly scoped in release notes.
  • Replacing Docker / Compose / Bake as the reproducible runtime story.
  • Large refactors outside benchmarks, campaigns, schemas, reporting, validation, browser evidence, and MCP documentation/tests aligned to the goals above.

Exit criteria

Area Status
Result pack publication Done — workflow + examples + validation commands documented.
Fingerprint-driven reporting Done — comparative/longitudinal artifacts use fingerprint context; docs describe run vs campaign fingerprints.
Report readability Done — shipped passes; examples under examples/campaign_runs/ reflect current reporting.
Drift prevention Done — drift tests + validate-artifacts + inventory doc; uv run just ci green.
Browser / MCP boundaries Done — docs + audits aligned; no mandatory live stack in default CI.
Pack comparison Done — CLI + campaign_result_pack_comparison kind + committed example.
Release hygiene Done — tag v0.2.3, CHANGELOG [0.2.3], docs/releases/v0.2.3.md, README/AGENTS aligned at rollover.

Dependencies

  • v0.2.2 baseline.
  • Offline-first tests: default tests/; live tests/integration/ behind env gates.

Verification commands

Baseline (required for merge):

uv run just ci

Committed artifact contracts (JSON Schema + Pydantic, deterministic):

just validate-artifacts

See docs/audits/schema-drift-contracts-inventory.md for swept filenames and explicit exclusions. Full matrix: docs/workflows/verification.md.

Campaign trees and result packs (spot checks):

uv run alwm validate examples/campaign_runs/minimal_offline/manifest.json campaign_manifest
uv run alwm validate examples/campaign_runs/minimal_offline/campaign-summary.json campaign_summary
uv run alwm validate examples/campaign_runs/minimal_offline/campaign-semantic-summary.json campaign_semantic_summary
uv run alwm validate examples/campaign_result_packs/minimal_offline/campaign-result-pack.json campaign_result_pack
uv run alwm validate examples/campaign_runs/multi_suite/manifest.json campaign_manifest

Pack comparison (example):

uv run alwm validate \
  examples/campaign_result_packs/compare_minimal_vs_multi/pack-compare.json \
  campaign_result_pack_comparison

Longitudinal (offline smoke):

uv run alwm benchmark longitudinal \
  --runs-glob 'examples/campaign_runs/minimal_offline/runs/*/manifest.json' \
  --out-dir /tmp/alwm-long-v023-smoke

Prompt registry (when registry work is in scope):

uv run alwm prompts check

Optional integration (not default CI): live benchmarks, Playwright—see AGENTS.md.