Skip to content

Milestone v0.2.0 — Comparative campaigns and longitudinal evaluation

Theme: Treat the repository as a rigorous comparative experiment harness: reproducible campaign sweeps, stable experiment identity, and longitudinal analysis that does not conflate unrelated configuration changes.

Goals

  • Campaign orchestration — Cartesian sweeps over suites, providers, scoring backends, execution filters, and browser overrides with a single campaign manifest and summaries.
  • Stable fingerprints — Deterministic sha256: identities for campaign definitions, benchmark suites, provider YAMLs, scoring options, browser configs, and effective prompt registry state (run and campaign axes).
  • Manifest persistence — Benchmark and campaign manifests record fingerprints for offline diffing and tooling.
  • Longitudinal grouping — group_snapshots_by supports fingerprint keys (including prompt registry state) so time series stay comparable.
  • Campaign-level reporting — Human-readable summaries list per-axis experiment fingerprints alongside member run statistics.
  • Scoring auditability — Semantic and hybrid paths remain traceable via existing judge provenance; fingerprints capture the effective scoring configuration for apples-to-apples checks.

Non-goals (this milestone)

  • Hosted dashboards or cloud execution.
  • New live network dependencies in default CI.
  • Broad refactors outside benchmark/campaign/longitudinal/reporting surfaces.
  • IDE-hosted or remote MCP browser integrations (e.g. Cursor’s bundled MCP) as a release exit criterion—out of scope for proving v0.2.0 campaign/fingerprint work. A local stdio MCP client + fixture server is acceptable for protocol validation without network; see docs/architecture/browser.md.

Exit criteria

  • JSON Schemas and Pydantic models agree for comparison_fingerprints (six axes) and campaign_experiment_fingerprints (six axes).
  • alwm benchmark run and alwm benchmark campaign run emit validated manifests with fingerprints.
  • Longitudinal docs and sample bundles reflect the new keys.
  • Documentation — Workflows, wiki, tracking, ADRs, README, AGENTS, CHANGELOG, and data-model describe the six benchmark axes, campaign experiment axes, and longitudinal grouping keys consistently (see CHANGELOG [0.2.0]). Step-by-step: docs/workflows/campaign-walkthrough.md (committed examples/campaign_runs/minimal_offline/).
  • Default test suite remains offline and deterministic (uv run just ci).

Dependencies

  • Existing benchmark runner, campaign runner, prompt registry resolution, and provider config loading.
  • No new runtime services for default CI.

Verification commands

uv run just ci
uv run alwm validate examples/campaign_runs/minimal_offline/manifest.json campaign_manifest
uv run alwm validate examples/campaign_runs/minimal_offline/campaign-summary.json campaign_summary
uv run alwm validate fixtures/longitudinal/paired/run_2026_w02/manifest.json benchmark_manifest
uv run alwm benchmark longitudinal --runs-glob 'fixtures/longitudinal/paired/*/manifest.json' --out-dir /tmp/long-test

Optional: regenerate the committed longitudinal sample bundle after fixture changes:

uv run alwm benchmark longitudinal --runs-glob 'fixtures/longitudinal/paired/*/manifest.json' --out-dir examples/reports/longitudinal/sample-output