Implementation log

Chronological record of repository work. Latest entries first.

2026-04-19 — Milestone closure: v0.2.5 shipped (docs + CHANGELOG)

Delivered: docs/releases/v0.2.5.md — narrative scope, verification, follow-ups. CHANGELOG.md — [0.2.5] (highlights; empty [Unreleased]). docs/roadmap/v0.2.5.md, docs/tracking/v0.2.5-campaign.md — complete / closed. README.md, AGENTS.md — latest shipped v0.2.5; in-flight none. docs/index.md, mkdocs.yml (Releases + roadmap label), docs/release-readiness.md (intro pointer). Cross-links: docs/releases/v0.2.{2,3,4}.md, docs/roadmap/v0.2.{3,4}.md, docs/workflows/{verification,campaign-result-pack-publication}.md.

Verification: uv run --extra docs mkdocs build --strict; uv run pytest tests/ --ignore=tests/integration -q (or uv run just ci when just is available).

2026-04-19 — Docs CI: GitHub Pages via actions/deploy-pages + site_url

Delivered: .github/workflows/docs.yml — replaced peaceiris/actions-gh-pages (branch gh-pages) with actions/upload-pages-artifact + actions/deploy-pages so Settings → Pages → Source: GitHub Actions works; job permissions scoped (pages:write, id-token:write on deploy); workflow_dispatch on main for manual publish. mkdocs.yml — site_url: https://sempervent.github.io/agent-llm-wiki-matrix/ for correct project-site paths. Docs docs/workflows/docs-site.md, README.md, docs/workflows/verification.md, docs/contributing.md.

Verification: uv run --extra docs mkdocs build --strict.

2026-04-19 — v0.2.5: campaign markdown readability (digest, semantic, compare intros)

Delivered: render_campaign_summary_markdown — shorter intro and Generated reports bullets. render_campaign_at_a_glance_markdown — one Judge & semantic signals ### with #### confidence + hotspots (removed duplicate H3 pair); tighter snapshot blurb. render_campaign_comparative_markdown — shorter title blurb, --- before Which dimensions varied; shorter section intros (backend, instability, FT-). render_campaign_semantic_summary_markdown — shorter intro; removed duplicate Suites/Provider/Execution instability tables (kept Instability hotspots + Detailed rollups). render_campaign_semantic_judge_section_markdown — Judge variance (abbreviated) + pointer line. render_campaign_browser_evidence_markdown — shorter intro. render_campaign_result_pack_compare_markdown / render_campaign_compare_markdown — shorter headers. Regenerated examples/campaign_runs/{minimal_offline,multi_suite,browser_evidence_compare}/, examples/campaign_result_packs/{minimal_offline,multi_suite,compare_minimal_vs_multi}/, examples/campaign_compares/minimal_offline_vs_multi_suite/. Docs docs/workflows/benchmark-campaigns.md. Tests tests/test_report_markdown_readability.py* (merged judge heading, comparative separator).

Verification: uv run ruff check src tests; uv run mypy src; uv run pytest tests/ --ignore=tests/integration/; uv run alwm benchmark campaign run (three example defs); pack + compare-packs; uv run alwm benchmark campaign compare for example compare dir.

2026-04-19 — v0.2.5: result pack INDEX + publication hints

Delivered: campaign_result_pack.py — derive_pack_publication_hints, _effective_publication_hints; membership_scope + optional_layers_present on CampaignResultPackV1 (schema campaign_result_pack.schema.json); _render_pack_index_md restructured (For reviewers, Scope of this bundle, What is included; dropped wide Bundle completeness table). pack_identity_fingerprint payload unchanged (hints excluded). Regenerated examples/campaign_result_packs/{minimal_offline,multi_suite}/. Docs docs/workflows/benchmark-campaigns.md, docs/workflows/campaign-result-pack-publication.md, examples/campaign_result_packs/README.md, docs/architecture/data-model.md. Tests tests/test_campaign_result_pack.py.

Verification: uv run pytest tests/test_campaign_result_pack.py tests/test_schema_drift_contracts.py tests/test_campaign_result_pack_compare.py; uv run alwm validate examples/campaign_result_packs/minimal_offline/campaign-result-pack.json campaign_result_pack and …/multi_suite/….

2026-04-17 — v0.2.5: compare reports scannable (At a glance, analysis deltas, FT/browser)

Delivered: campaign_compare_core.py — slim format_reader_interpretation_markdown (## At a glance, no repeat of instability/browser/semantic/FT prose); semantic_instability_rows_all_quiet, partition_failure_tag_rows_for_display, render_failure_tags_compare_subsection_lines; render_browser_evidence_member_cells_comparison_markdown(..., compare_compact=True) (shorter intro + pairing scan line). campaign_result_pack_compare.py / campaign_directory_compare.py — reorder: Member run overlap / Member runs & manifest health early; single ## Analysis deltas with subsections; instability one-liner when quiet; FT With movement / No movement split; browser ### under analysis. tests/test_campaign_compare_core.py; compare tests updated. Regenerated examples/campaign_result_packs/compare_minimal_vs_multi/, examples/campaign_compares/minimal_offline_vs_multi_suite/. Docs docs/workflows/benchmark-campaigns.md, docs/workflows/campaign-result-pack-publication.md, examples/campaign_{compares,result_packs/compare_minimal_vs_multi}/README.md, CHANGELOG.md.

Backward compatibility: JSON shape unchanged; reader_interpretation object still emitted (narrative fields intact for programmatic use).

Verification: uv run ruff check / mypy on touched modules; uv run pytest tests/ --ignore=tests/integration/; uv run alwm validate on regenerated compare JSON.

2026-04-17 — v0.2.5: MkDocs nav polish (Publication & reports, docs-site findings)

Delivered: mkdocs.yml — top nav block Publication & reports (v0.2.5) (labels for E2E checklist, CLI/compare, verification, examples); docs/examples/README.md under that block only (removed duplicate top-level Examples index); roadmap v0.2.5 stays under Roadmap & milestones only. docs/workflows/docs-site.md — publication nav description, v0.2.5 docs-site pass table (awkward / improved / deferred). docs/index.md — already aligned (tip, glance table row 7, sidebar name). README.md, AGENTS.md — discoverability notes for the publication sidebar block. docs/roadmap/v0.2.5.md — docs-site polish row note. Cross-links: campaign-result-pack-publication.md, benchmark-campaigns.md → docs-site.md / index.md.

Verification: uv run --extra docs mkdocs build --strict.

2026-04-17 — v0.2.5: validation ergonomics (three layers, cross-links)

Delivered: docs/workflows/verification.md — Three layers table (alwm validate = A, validate-artifacts = B, just ci = C), relationship bullets, tightened Relationships section. README.md — canonical verification table: one-file alwm validate, uv run just validate-artifacts. AGENTS.md — verification expectations: Single artifact row + layer B wording. docs/audits/schema-drift-contracts-inventory.md — Last reviewed, layer A/B/C framing, link to three layers. docs/workflows/docs-site.md, docs/index.md — site cross-links. mkdocs.yml — Publication nav label for verification page. No CLI or justfile behavior change (docs + nav label only).

Verification: docs-only; optional uv run pytest tests/test_schema_drift_contracts.py -q (unchanged tests).

2026-04-17 — Milestone rollover: v0.2.4 shipped, v0.2.5 active

Delivered: CHANGELOG.md — [0.2.4] section; empty [Unreleased] for new work. docs/roadmap/v0.2.4.md, docs/tracking/v0.2.4-campaign.md — complete / closed with pointers to docs/releases/v0.2.4.md. docs/roadmap/v0.2.5.md, docs/tracking/v0.2.5-campaign.md — active focus (evidence packs + report readability). README.md, AGENTS.md — Latest shipped v0.2.4, in-flight v0.2.5. mkdocs.yml, docs/index.md, docs/workflows/{verification,campaign-result-pack-publication}.md, docs/releases/v0.2.{2,3}.md — milestone pointers. pyproject.toml, agent_llm_wiki_matrix.__init__ → 0.2.5; example alwm_version in examples/campaign_result_packs/{minimal_offline,multi_suite}/ → 0.2.4 (assembler at v0.2.4 tag).

Verification: uv run alwm version → 0.2.5; uv run just ci; uv run --extra docs mkdocs build --strict (optional).

2026-04-19 — MkDocs findings polish (nav dedup, reporting in Publication)

Delivered: mkdocs.yml — Publication (v0.2.4) labels clarified; workflows/reporting.md added to Publication; duplicate tracking/v0.2.4-campaign.md removed from Tracking nav; reporting removed from generic Workflows list (single nav path). docs/index.md — publication table row for reporting + handbook link in tip; docs/workflows/docs-site.md — findings updated (CI parity, dedup, reporting cross-links).

Verification: uv run --extra docs mkdocs build --strict — success (~0.7s).

2026-04-19 — MkDocs handbook (docs-site.md) and documented verification findings

Delivered: Canonical workflow doc docs/workflows/docs-site.md (local just docs / just docs-build, mkdocs.yml nav:, CI, implementation/verification findings: succeeded vs awkward vs deferred, hard-to-surface pages, agent conventions). Removed duplicate mkdocs-site.md; mkdocs.yml nav points to docs-site.md. Updates: README.md, AGENTS.md (MkDocs sync bullets), docs/roadmap/v0.2.4.md (MkDocs status table), docs/tracking/v0.2.4-campaign.md (findings section), cross-links. .gitignore comment.

Verification (recorded in docs-site): uv sync --extra dev --extra docs; uv run --extra docs mkdocs build --strict — success (~1s); mkdocs serve smoke HTTP 200 on /; GET /workflows/docs-site/ 200 after nav update. Material MkDocs 2.0 banner present (informational). uv run just ci — success.

2026-04-19 — MkDocs nav polish + GitHub Actions docs workflow

Delivered: mkdocs.yml — Publication (v0.2.4) nav (checklist first; audit listed once; labels); Roadmap & milestones includes v0.2.4 (active). docs/index.md — tip callout + tables. Cross-links on campaign-result-pack-publication.md, verification.md, benchmark-campaigns.md, roadmap/v0.2.4.md. Handbook later consolidated as docs/workflows/docs-site.md. .github/workflows/docs.yml — uv sync --frozen --extra docs, mkdocs build --strict, artifact, gh-pages on main.

Verification: uv run --extra docs mkdocs build --strict; uv run just ci.

2026-04-19 — MkDocs documentation site (v0.2.4)

Delivered: mkdocs.yml (Material theme, nav: publication block, workflows, roadmaps, releases, architecture, audits, tracking, wiki, ADRs, external links). docs/index.md, docs/contributing.md, later docs/workflows/docs-site.md. pyproject.toml optional [docs] extra (mkdocs>=1.6,<2, mkdocs-material, pymdown-extensions); just docs, just docs-build. .gitignore: /site/. Updates: README.md, AGENTS.md, docs/workflows/verification.md, docs/roadmap/v0.2.4.md, docs/tracking/v0.2.4-campaign.md. uv.lock.

Verification: uv run --extra docs mkdocs build --strict; uv run just ci.

2026-04-19 — Campaign / comparison Markdown (publication readability)

Delivered: Tighter headings and reader-oriented copy for campaign-summary.md (Metadata, Snapshot digest, Member run index, Generated reports, Longitudinal follow-up), campaign-semantic-summary.md (Campaign semantic summary, Executive snapshot), reports/campaign-report.md (Executive summary, intro blurb), pack-compare-report.md / campaign-compare-report.md (Δ convention upfront). campaign_result_pack INDEX.md: Pack snapshot (was “At a glance”). No scoring math changes.

Examples: Regenerated examples/campaign_runs/{minimal_offline,multi_suite,browser_evidence_compare}/, examples/campaign_result_packs/{minimal_offline,multi_suite}/, examples/campaign_result_packs/compare_minimal_vs_multi/, examples/campaign_compares/minimal_offline_vs_multi_suite/. Docs docs/workflows/benchmark-campaigns.md. Tests tests/test_benchmark_campaign.py, tests/test_campaign_reporting.py, tests/test_report_markdown_readability.py, tests/test_campaign_directory_compare.py, tests/test_campaign_result_pack_compare.py.

Verification: uv run pytest tests/ --ignore=tests/integration.

2026-04-17 — Pack/campaign compare: reader interpretation JSON + Markdown

Delivered: campaign_compare_core.py — build_reader_interpretation, format_reader_interpretation_markdown (non-scoring narratives, cautious evidence_strength, uncertainty_caveats). campaign_result_pack_compare.py / campaign_directory_compare.py attach reader_interpretation to compare JSON and prepend the Markdown section. CampaignResultPackComparisonV1 / CampaignCompareV1 optional field; schemas/v1/campaign_{result_pack_comparison,compare}.schema.json — additive reader_interpretation. Docs docs/workflows/campaign-result-pack-publication.md. Regenerated examples/campaign_result_packs/compare_minimal_vs_multi/, examples/campaign_compares/minimal_offline_vs_multi_suite/. Tests tests/test_campaign_result_pack_compare.py, tests/test_campaign_directory_compare.py.

Backward compatibility: schema_version 1; older JSON without reader_interpretation still validates; Markdown skips the section when absent.

Verification: uv run just ci (or equivalent ruff/mypy/pytest on default suite).

2026-04-19 — Campaign result pack: reviewer-first INDEX + optional dry-run path

Delivered: CampaignResultPackArtifacts.campaign_dry_run_json (optional; set when campaign-dry-run.json is copied into the pack). _render_pack_index_md — Start here (cold read), Bundle completeness table, expanded Publish-ready checklist, subset callout; INDEX.md validate block includes campaign-summary.json. Docs docs/workflows/campaign-result-pack-publication.md, docs/workflows/benchmark-campaigns.md, docs/architecture/data-model.md, examples/campaign_result_packs/README.md. Regenerated examples/campaign_result_packs/{minimal_offline,multi_suite}/. Tests tests/test_campaign_result_pack.py.

Verification: uv run pytest tests/test_campaign_result_pack.py tests/test_schema_drift_contracts.py; uv run ruff check / mypy on touched modules.

2026-04-19 — Milestone rollover: v0.2.3 shipped, v0.2.4 active

Delivered: CHANGELOG.md — [0.2.3] section; empty [Unreleased] for new work. docs/releases/v0.2.3.md. docs/roadmap/v0.2.3.md, docs/tracking/v0.2.3-campaign.md marked closed / complete. docs/roadmap/v0.2.4.md, docs/tracking/v0.2.4-campaign.md — publication + E2E workflow focus. README.md, AGENTS.md — Latest shipped v0.2.3, in-flight v0.2.4. docs/releases/v0.2.2.md next-milestone pointer. pyproject.toml, agent_llm_wiki_matrix.__init__, uv.lock → 0.2.3; example alwm_version in examples/campaign_result_packs/{minimal_offline,multi_suite}/ aligned.

Verification: docs-only pass; uv run alwm version (optional sanity).

2026-04-19 — Pack/directory compare: browser evidence pairing

Delivered: campaign_compare_core.py — build_browser_evidence_member_cells_comparison_block, render_browser_evidence_member_cells_comparison_markdown; merged into build_analysis_comparison_block as browser_evidence_comparison. campaign_result_pack_compare.py / campaign_directory_compare.py render the Markdown section after semantic instability (directory: top_heading_level=2); directory report drops the older duplicate browser bullets in favor of the paired table. Example examples/campaign_result_packs/compare_minimal_vs_multi/ regenerated. Schema note on browser_evidence_comparison; docs docs/workflows/benchmark-campaigns.md, docs/architecture/data-model.md. Tests tests/test_campaign_result_pack_compare.py, tests/test_campaign_directory_compare.py.

Verification: uv run pytest tests/test_campaign_result_pack_compare.py tests/test_campaign_directory_compare.py; uv run ruff check / mypy on touched modules.

2026-04-19 — Campaign-to-campaign comparison (directories)

Delivered: benchmark/campaign_compare_core.py — shared analysis/semantic/FT- diff blocks for pack and campaign workflows. benchmark/campaign_directory_compare.py — build_campaign_directory_comparison, render_campaign_compare_markdown, write_campaign_compare_artifacts; sweep dimensions via summarize_campaign_dimensions; fingerprint_axis_insights diff; browser_evidence_member_cells rollups. CampaignCompareV1 + schemas/v1/campaign_compare.schema.json; kind campaign_compare in artifacts.py. CLI alwm benchmark campaign compare ( --repo-root, --created-at ). Example examples/campaign_compares/minimal_offline_vs_multi_suite/. Tests tests/test_campaign_directory_compare.py; tests/test_schema_drift_contracts.py loads campaign-compare.json. Docs docs/workflows/benchmark-campaigns.md, examples/campaign_compares/README.md, README.md, AGENTS.md*.

Verification: uv run pytest tests/test_campaign_directory_compare.py tests/test_campaign_result_pack_compare.py tests/test_schema_drift_contracts.py; uv run ruff check on touched modules.

2026-04-19 — Fingerprint interpretation: evidence strength + explicit uncertainty

Delivered: campaign_fingerprint_compare.py — _compute_evidence_strength; attribution rows add attribution_label, evidence_strength, uncertainty_notes, metrics.min_bucket_cell_count / distinct_bucket_count; overview and caveats stress weak samples; groups gain run_count. Markdown: category legend, expanded table, Limit: lines. Docs docs/workflows/longitudinal-reporting.md, benchmark-campaigns.md. Tests tests/test_campaign_reporting.py, tests/test_fingerprint_interpretation_attribution.py.

Backward compatibility: schema_version 1; additive JSON fields. Scoring unchanged.

Verification: uv run pytest tests/ --ignore=tests/integration/.

2026-04-18 — Campaign result pack publication workflow (docs + INDEX)

Delivered: docs/workflows/campaign-result-pack-publication.md — assemble, validate, compare-packs, interpret pack-compare-report.md. _render_pack_index_md — Publication workflow + Publish-ready checklist sections. docs/workflows/benchmark-campaigns.md, examples/campaign_result_packs/README.md, README.md — canonical outward-facing bundle, completeness table, walkthrough links.

Verification: uv run ruff check src/agent_llm_wiki_matrix/benchmark/campaign_result_pack.py; regenerate example packs (alwm benchmark campaign pack …) so INDEX.md matches the template.

2026-04-18 — Fingerprint interpretation: attribution + uncertainty

Delivered: classify_axis_difference_attribution — labels likely_configuration, likely_instability, mixed_config_and_instability, inconclusive with confidence; JSON adds attribution_by_axis, differentiation_overview, interpretation_caveats; hints gain signal_class. Markdown: uncertainty blockquote, summary, per-axis table + rationales, grouped raw signals. schema_version stays 1. Tests tests/test_fingerprint_interpretation_attribution.py. campaign_result_pack_compare missing model imports fixed for mypy. Regenerated campaign + longitudinal examples.

Verification: uv run pytest --ignore=tests/integration/; uv run ruff check src tests; uv run mypy src.

2026-04-18 — Browser reporting: signals, extension digests, campaign contrast

Delivered: browser/formatting.py — signals_digest_line, extension_digest_short, format_extensions_compact_markdown; BrowserEvidenceReportRow gains signals_digest / extension_digest; benchmark + campaign tables and detail sections updated; _describe_runner_cell. campaign_browser_evidence.py — render_campaign_browser_cross_run_comparison; browser_evidence_member_cells rows add signals_digest / extension_digest. Example examples/campaigns/v1/browser_evidence_compare.v1.yaml + examples/campaign_runs/browser_evidence_compare/; refreshed examples/campaign_runs/multi_suite/ and examples/campaign_result_packs/multi_suite/. Docs docs/workflows/benchmarking.md, benchmark-campaigns.md, examples/campaigns/v1/README.md. Tests tests/test_browser_formatting.py, tests/test_benchmark_browser.py.

Verification: uv run pytest tests/ --ignore=tests/integration/; uv run ruff check on touched modules.

2026-04-18 — Campaign result pack comparison (CLI + artifacts)

Delivered: benchmark/campaign_result_pack_compare.py — build_campaign_result_pack_comparison, render_campaign_result_pack_compare_markdown, write_campaign_result_pack_compare_artifacts; optional repo_root for portable paths in JSON/Markdown. CampaignResultPackComparisonV1 + schemas/v1/campaign_result_pack_comparison.schema.json; kind campaign_result_pack_comparison in artifacts.py. CLI alwm benchmark campaign compare-packs ( --repo-root ). Example packs examples/campaign_result_packs/multi_suite/ and examples/campaign_result_packs/compare_minimal_vs_multi/. Tests tests/test_campaign_result_pack_compare.py; tests/test_schema_drift_contracts.py loads pack-compare.json. Docs docs/workflows/benchmark-campaigns.md, docs/architecture/data-model.md, examples/campaign_result_packs/README.md.

Verification: uv run just ci.

2026-04-18 — Verification surface (docs): canonical, fallback, live

Delivered: docs/workflows/verification.md — matrix (just ci, just test, just validate-artifacts, live/smoke) with uv run fallbacks; clarifies validate-artifacts ⊂ full pytest, not a just ci replacement. Updated README.md, AGENTS.md, docs/workflows/local-dev.md, live-verification.md, benchmarking.md, docs/architecture/runtime.md, docs/audits/schema-drift-contracts-inventory.md.

Verification: docs-only; justfile unchanged.

2026-04-18 — Report Markdown readability (campaign + benchmark; no scoring change)

Delivered: suite_ref_benchmark_id_partition_coincide + _markdown_sweep_axis_keys in campaign_reporting.py — when suite and benchmark partitions match, Markdown omits duplicate benchmark_id sweep lines; dimensions table uses suite_ref (paired with benchmark_id). Tighter copy for comparative + campaign-summary At a glance, FT- digest, judge rollup bullets; campaign_semantic_summary.md (## Snapshot, merged coverage/range rows; removed duplicate ## Totals). campaign_browser_evidence: shorter intro; per-cell headings drop benchmark_id when paired with suite. browser/formatting.py: shorter benchmark-run browser blurb. Tests tests/test_report_markdown_readability.py*.

Analysis JSON: unchanged (build_campaign_analysis_dict fields).

Verification: uv run pytest tests/ --ignore=tests/integration/.

2026-04-18 — Fingerprint axis interpretation (campaign + longitudinal)

Delivered: build_fingerprint_axis_interpretation in benchmark/campaign_fingerprint_compare.py — score_spread_ranking, instability_hotspots, drift_correlation_hints; cell_count on per-group insight rows. fingerprint_axis_interpretation in campaign-analysis.json and summary.json (longitudinal). Markdown: render_fingerprint_compare_markdown, render_provider_comparison_markdown. Examples: examples/campaign_runs/minimal_offline/, multi_suite/, campaign_result_packs/minimal_offline/, examples/reports/longitudinal/sample-output/. Tests tests/test_campaign_reporting.py, tests/test_longitudinal.py. Docs docs/workflows/benchmark-campaigns.md, longitudinal-reporting.md.

Verification: uv run ruff check src tests; uv run mypy src; uv run pytest --ignore=tests/integration.

2026-04-18 — Schema/runtime drift verification (committed trees)

Delivered: tests/test_schema_drift_contracts.py — deterministic sweep of examples/ and fixtures/ for benchmark_manifest / benchmark_campaign_manifest, campaign summaries, semantic summaries, result packs, browser evidence, cell I/O, evaluation_judge_provenance, reports/report.json, matrices/*.json, examples/dataset/rubrics/*.json; collect_committed_artifact_drift_errors() for reuse; regression tests for observability + generated_report_paths. just validate-artifacts recipe. Removed tests/test_schema_drift_examples.py (superseded). Audits docs/audits/schema-drift-contracts-inventory.md, updated docs/audits/schema-runtime-hardening-2026-04-18.md; pointers in docs/roadmap/v0.2.3.md, docs/tracking/v0.2.3-campaign.md, AGENTS.md.

Verification: just validate-artifacts; uv run just ci.

2026-04-18 — MCP stdio docs + browser realism claims

Delivered: fixtures/mcp_servers/stdio_browser_evidence_server.py — multi-scenario alwm_browser_evidence (alwm:checkout_flow / alwm:form_validation / URL hints → checkout_flow / form_validation / export_flow). tests/test_mcp_stdio.py — deterministic stdio path (rich BrowserEvidence + CLI --stdio). Docs README.md, AGENTS.md, docs/architecture/browser.md, docs/audits/capability-classification.md, docs/workflows/live-verification.md, examples/browser_evidence/v1/README.md, .env.example; browser/formatting.py report copy. No remote/IDE MCP implementation.

Verification: uv run pytest tests/ --ignore=tests/integration; uv run ruff check src tests; uv run mypy src.

2026-04-18 — Campaign semantic judge: confidence + instability (criteria + axes)

Delivered: CampaignSemanticSummaryV1 gains criterion_instability (per-criterion score_range sums from repeat_aggregation.disagreement.per_criterion), instability_highlights (ranked unstable suites / providers / execution modes, confidence_flag_counts), totals.cells_flagged_judge_low_confidence vs cells_flagged_repeat_confidence_low, CampaignSemanticCellRow.repeat_confidence_low. CampaignSemanticSummaryV1 Markdown leads with Judge confidence & disagreement; campaign-report.md embeds render_campaign_semantic_judge_section_markdown; campaign-analysis.json adds judge_campaign_semantic. Pipeline order: semantic summary before comparative artifacts. schemas/v1/campaign_semantic_summary.schema.json. Regenerated examples/campaign_runs/minimal_offline/, multi_suite/, examples/campaign_result_packs/minimal_offline/ campaign-semantic-summary.json. Tests tests/test_benchmark_campaign.py. Docs docs/workflows/benchmark-campaigns.md.

Verification: uv run pytest tests/ --ignore=tests/integration; uv run mypy src; uv run ruff check src tests.

2026-04-18 — Browser evidence reporting (legible Markdown + traces suite)

Delivered: browser/formatting.py — format_extensions_markdown, render_dom_excerpts_markdown, render_screenshots_markdown, render_browser_evidence_detail_markdown, expanded render_benchmark_browser_evidence_markdown (summary + legible per-cell blocks; honest Playwright optional / MCP stdio copy). benchmark/runner.py passes (BrowserEvidenceReportRow, BrowserEvidence) pairs. benchmark/campaign_browser_evidence.py — campaign-level browser markdown + browser_evidence_member_cells for campaign-analysis.json via campaign_reporting. fixtures/benchmarks/browser_traces_compare.v1.yaml, examples/benchmark_suites/v1/suite.taxonomy.browser_traces_compare.v1.yaml, examples/benchmark_runs/browser-traces-compare/README.md. Tests tests/test_browser_formatting.py, tests/test_benchmark_browser.py; taxonomy list tests/test_benchmark_taxonomy.py. Docs docs/workflows/benchmarking.md, docs/workflows/benchmark-campaigns.md, docs/architecture/browser.md.

Verification: uv run ruff check src tests; uv run mypy src; uv run pytest.

2026-04-18 — Campaign examples + result pack follow-up (schema drift)

Delivered: Regenerated examples/campaign_runs/multi_suite/ so campaign-semantic-summary.json matches the current campaign_semantic_summary schema (fixes test_schema_drift_examples). Rebuilt examples/campaign_result_packs/minimal_offline/ from examples/campaign_runs/minimal_offline. Restored examples/campaign_runs/multi_suite/README.md. docs/workflows/campaign-walkthrough.md — Step 6 (result pack) + See also pointer.

Verification: uv run pytest tests/ --ignore=tests/integration/.

2026-04-18 — Campaign result packs (CLI + artifact + example)

Delivered: CampaignResultPackV1 / CampaignResultPackArtifacts / CampaignResultPackMemberRun in models.py; schemas/v1/campaign_result_pack.schema.json; benchmark/campaign_result_pack.py (assemble_campaign_result_pack, INDEX.md with ALWM_REPO_ROOT + repo-relative longitudinal examples). alwm benchmark campaign pack (--pack-id, --title, --source-label, --member-depth, --run-index, --include-failed-members, --record-source-abspath). Default omit absolute source path for portable packs. artifacts.py kind campaign_result_pack. Tests tests/test_campaign_result_pack.py; tests/test_schema_drift_examples.py loads campaign-result-pack.json. Docs docs/workflows/benchmark-campaigns.md, docs/architecture/data-model.md, examples/campaign_result_packs/README.md, README.md command table.

Example: examples/campaign_result_packs/minimal_offline/ (from examples/campaign_runs/minimal_offline).

Verification: uv run pytest tests/test_campaign_result_pack.py tests/test_schema_drift_examples.py; uv run alwm validate examples/campaign_result_packs/minimal_offline/campaign-result-pack.json campaign_result_pack.

2026-04-18 — Schema / runtime drift hardening (committed artifacts)

Delivered: tests/test_schema_drift_examples.py validates examples/ + fixtures/ for benchmark_manifest, benchmark_campaign_manifest, campaign_summary, campaign_semantic_summary, browser_evidence, evaluation, benchmark_request, benchmark_response; focused assertions for runtime_summary, retry_summary, generated_report_paths. Audit docs/audits/schema-runtime-hardening-2026-04-18.md (local out/ may hold stale five-axis fingerprints—regenerate or delete).

Verification: uv run pytest tests/test_schema_drift_examples.py -v.

2026-04-18 — Campaign reporting readability (at-a-glance + member-score rollups)

Delivered: campaign_reporting — ## At a glance lead section and member-run mean score tables on reports/campaign-report.md; member_mean_score_by_dimension in campaign-analysis.json (alongside existing mean_score_extremes_by_sweep_axis). campaign-summary.md embeds render_campaign_at_a_glance_markdown when longitudinal + semantic bundles are available after a full run. write_campaign_comparative_artifacts returns (paths, longitudinal_bundle) so the summary avoids a second analysis pass. campaign_semantic_summary.md already surfaces ranked instability hotspots; comparative narrative cross-links remain. Docs docs/workflows/benchmark-campaigns.md; log entry. Tests tests/test_campaign_reporting.py. Regenerated examples/campaign_runs/minimal_offline/ and added examples/campaign_runs/multi_suite/ (multi-suite sweep example).

Verification: uv run just ci.

Note: Scoring defaults unchanged; reporting-only.

2026-04-17 — MCP stdio protocol path (local subprocess; v0.2.0 non-goal for IDE MCP)

Delivered: browser/mcp_stdio.py — MCP client call_tool over stdio (mcp SDK), JSON text → BrowserEvidence. MCPBrowserRunner — fixture path first; if ALWM_MCP_BROWSER_COMMAND is set and no scenario_id/fixture_relpath, runs stdio path. fixtures/mcp_servers/stdio_browser_evidence_server.py (FastMCP) for tests. alwm browser run-mcp --stdio. mcp>=1.27 added to [project.optional-dependencies] dev (alongside [dependency-groups] dev). Docs: docs/architecture/browser.md, runtime.md, docs/roadmap/v0.2.0.md (non-goal: IDE-hosted MCP), capability-classification.md, AGENTS.md, README.md, CHANGELOG.md [Unreleased], .env.example.

Verification: uv run pytest tests/test_browser.py; uv run ruff / mypy on touched modules.

2026-04-17 — Browser benchmark realism (fixtures; prompts + report table)

Delivered: Extended BrowserEvidence / schema: a11y fields on dom_excerpts, screenshot scope/sequence/DPR; extensions $defs for network / accessibility / performance. evidence_to_prompt_block + render_benchmark_browser_evidence_markdown; run_benchmark appends browser summary to reports/report.md. Fixtures checkout_flow.json, form_validation.json; suites suite.taxonomy.browser_checkout.v1, suite.taxonomy.browser_form.v1, suite.agentic.browser_checkout.v1; fixtures/benchmarks/browser_checkout.v1.yaml, browser_form.v1.yaml. prompts/registry.yaml 0.4.2; bench.task.browser_evidence.v1 (honest MCP / optional Playwright wording). Docs docs/architecture/browser.md, benchmarking.md, data-model.md, examples/browser_evidence/v1/README.md. Tests tests/test_browser.py, test_benchmark_browser.py, test_benchmark_taxonomy.py.

Verification: uv run ruff check src tests; uv run mypy src; uv run pytest tests/ --ignore=tests/integration.

2026-04-17 — Campaign fingerprint-axis comparison (longitudinal keys)

Delivered: benchmark/campaign_fingerprint_compare.py — group_snapshots_by on provider / scoring config / execution mode / prompt registry / browser fingerprint keys; pooled mean scores; instability, regression, and mode-gap counts per bucket; narrative insights when an axis varies. Wired into campaign_reporting (fingerprint_compare_axes, fingerprint_axis_insights in JSON; Markdown section in reports/campaign-report.md). truncate_fingerprint_display exposed from pipelines/longitudinal.py. Example examples/campaigns/v1/fingerprint_axes_probe.v1.yaml. Tests tests/test_campaign_reporting.py. Docs docs/workflows/benchmark-campaigns.md, docs/workflows/longitudinal-reporting.md, examples/campaigns/v1/README.md.

Verification: uv run ruff check src tests; uv run mypy src; uv run pytest.

2026-04-18 — Campaign comparative reporting (Markdown + JSON)

Delivered: benchmark/campaign_reporting.py — after a full campaign, load succeeded member manifests with load_run_snapshots, run analyze_longitudinal (fixed thresholds aligned with longitudinal CLI defaults), emit reports/campaign-report.md (dimensions varied, backend means, scoring instability, mode gaps, FT-* ranking, render_failure_atlas) and reports/campaign-analysis.json. merge_generated_report_paths composes with semantic-summary paths. campaign-summary.md links comparative outputs. Docs docs/workflows/benchmark-campaigns.md, examples/campaigns/v1/README.md. Tests tests/test_campaign_reporting.py. Example examples/campaign_runs/minimal_offline/ regenerated.

Verification: uv run just ci.

2026-04-17 — Browser evidence + rubric realism (fixtures; CI deterministic)

Delivered: BrowserEvidence gains dom_excerpts, screenshots (metadata), extensions; JSON Schema + DomExcerpt / ScreenshotMetadata. evidence_to_prompt_block, MockBrowserRunner, fixtures/browser_evidence/v1/export_flow.json. Rubric examples/dataset/rubrics/browser_realism.v1.json (grounding, hallucination_resistance, source_fidelity). Wired into suite.taxonomy.browser_evidence.v1, suite.agentic.browser_interpretation.v1, fixtures/benchmarks/browser_file.v1.yaml. prompts/registry.yaml 0.4.1 and bench.task.browser_evidence.v1. Regenerated examples/benchmark_runs/agentic-pack-browser-interpretation/. Docs: docs/architecture/browser.md, data-model.md, benchmarking.md, examples/dataset/README.md.

Verification: uv run ruff check src tests; uv run mypy src; uv run pytest tests/ --ignore=tests/integration; uv run alwm prompts check.

2026-04-17 — Benchmark runtime observability (timing + retry/judge summaries)

Delivered: BenchmarkRunTimingSummaryV1, BenchmarkRetrySummaryV1, BenchmarkCellRuntimeV1 on manifest.json (runtime_summary, retry_summary, cells[].runtime); CampaignAggregatedRuntimeV1 on campaign_manifest (aggregated_runtime). runner.py records wall time for browser phase, provider completion, deterministic evaluation, and semantic judge phase (EvaluationPhaseMetrics + _run_semantic_with_repeats timing). reports/report.md and campaign-summary.md append Markdown tables (benchmark/observability.py). evaluate_with_scoring_backend returns (evaluation, provenance, metrics). Schemas: manifest.schema.json, benchmark_campaign_manifest.schema.json. Tests: tests/test_benchmark_observability.py. Docs: docs/workflows/benchmarking.md.

Verification: uv run pytest tests/ --ignore=tests/integration; uv run mypy src.

2026-04-17 — Campaign semantic / hybrid scoring summary

Delivered: campaign_semantic_summary artifact kind + schemas/v1/campaign_semantic_summary.schema.json; benchmark/campaign_semantic_summary.py builds CampaignSemanticSummaryV1 from succeeded member runs (repeat-judge disagreement, low-confidence flags, rollups by suite / provider / execution mode). run_benchmark_campaign writes campaign-semantic-summary.json / .md and merges paths into generated_report_paths (after comparative artifacts). campaign-summary.md links the semantic rollup when present. Example definition examples/campaigns/v1/semantic_repeats_offline.v1.yaml. Tests in tests/test_benchmark_campaign.py. Docs: docs/workflows/benchmark-campaigns.md, examples/campaigns/v1/README.md.

Note: Deterministic-only campaigns still emit a semantic summary with zero semantic cells; default scoring path is unchanged.

Verification: uv run ruff check src tests and uv run pytest.

2026-04-17 — MCP browser runner audit: CLI + doc alignment

Context: MCPBrowserRunner was already a fixture-backed bridge (delegates to FileBrowserRunner when scenario_id or fixture_relpath is set; RuntimeError without—remote MCP not wired). Several docs still claimed NotImplementedError / stub.

Delivered: alwm browser run-mcp (cli.py) with --scenario-id / --fixture; tests in tests/test_browser.py. docs/architecture/browser.md — accurate component table, Remote MCP section, roadmap list. Aligned AGENTS.md, README.md, docs/architecture/runtime.md, current-state.md, docs/audits/capability-classification.md, current-capability-status.md, mission-gap-audit.md, release-readiness.md; CHANGELOG.md [Unreleased].

Classification: MCPBrowserRunner = partial (fixture path only); remote MCP protocol execution = documented-only until roadmap items ship.

Verification: uv run just ci.

2026-04-18 — v0.2.0 documentation consolidation (fingerprints + campaigns)

Delivered: CHANGELOG.md [0.2.0] entry (comparison + campaign experiment fingerprints, longitudinal updates) plus Documentation subsection (campaign walkthrough). docs/workflows/campaign-walkthrough.md — committed-example steps for campaign manifest vs campaign summary vs member benchmark_manifest, six-axis campaign_experiment_fingerprints vs comparison_fingerprints, and longitudinal globs. Audited README.md, AGENTS.md, docs/roadmap/v0.2.0.md, workflows (notably benchmarking.md, benchmark-campaigns.md, longitudinal-reporting.md), docs/architecture/data-model.md, docs/wiki/campaign-orchestration.md, docs/wiki/benchmark-campaigns.md, docs/tracking/campaign-orchestration.md, docs/tracking/benchmark-campaign-orchestration.md, docs/audits/capability-classification.md, ADRs docs/adr/0001-campaign-orchestration.md and docs/architecture/adr/0001-benchmark-campaign-orchestration.md, examples/campaigns/v1/README.md for consistent six-axis comparison_fingerprints, campaign_experiment_fingerprints, and prompt_registry_state_fingerprint wording. Removed duplicate milestone block from README.md (earlier pass).

Verification: documentation-only pass; uv run alwm validate examples/campaign_runs/minimal_offline/manifest.json campaign_manifest (paths cited in walkthrough).

2026-04-17 — v0.2.0 fingerprints: prompt registry state + campaign experiment axes

Delivered: BenchmarkComparisonFingerprints.prompt_registry_state — hash of effective registry YAML (or inline-only sentinel). CampaignExperimentFingerprints on BenchmarkCampaignManifest / CampaignSummaryV1 (campaign definition, suite stack, provider files, scoring axis, browser axis, registry override). Canonical fingerprint_campaign_definition excludes cosmetic fields and sorts list fields. build_campaign_experiment_fingerprints, fingerprint_prompt_registry_effective_state in benchmark/fingerprints.py. Longitudinal group_snapshots_by key prompt_registry_state_fingerprint; render_provider_comparison_markdown registry column. Schemas: manifest.schema.json, benchmark_campaign_manifest.schema.json, campaign_summary.schema.json. Roadmap docs/roadmap/v0.2.0.md; README milestone blurb. Regenerated examples/campaign_runs/minimal_offline/, examples/reports/longitudinal/sample-output/, longitudinal fixtures.

Verification: uv run just ci.

2026-04-17 — Ollama gpt-oss:20b first-class workflow

Delivered: Default OLLAMA_MODEL / OllamaSection.model → gpt-oss:20b; benchmarks/v1/ollama.v1.yaml, config/providers.example.yaml, alwm benchmark probe default, integration test default aligned. scripts/ollama-gptoss-setup.sh (start Compose Ollama, pull, alwm benchmark probe), scripts/smoke-ollama-live.sh (minimal live benchmark + validate). just ollama-gptoss-setup, just smoke-ollama-live; verify-ollama-gptoss-smoke calls the live smoke script. Docs: docs/workflows/benchmarking.md (workflow + migration from named volume ollama_models → bind ./.ollama-models), README.md, AGENTS.md, docs/workflows/live-verification.md, docs/workflows/local-dev.md, .env.example.

Verification: uv run just ci (deterministic); live path opt-in.

2026-04-17 — Campaign orchestration: dry-run, wiki, ADR, tracking

Delivered: alwm benchmark campaign run --dry-run (wraps run_benchmark_campaign(..., dry_run=True); no separate plan command). Extended campaign manifest / CampaignSummaryV1 + schemas/v1/campaign_summary.schema.json; artifact kinds campaign_definition, campaign_manifest, campaign_summary. Docs: docs/wiki/campaign-orchestration.md, docs/tracking/campaign-orchestration.md, docs/adr/0001-campaign-orchestration.md; updates to docs/workflows/benchmark-campaigns.md, docs/workflows/benchmarking.md, docs/architecture/evaluation-pipeline.md, README.md, CHANGELOG.md, docs/architecture/adr/0001-benchmark-campaign-orchestration.md, examples/campaigns/v1/README.md. Tests: tests/test_benchmark_campaign.py (dry-run writes campaign-dry-run.json, manifest, summary).

Verification: uv run just ci.

2026-04-17 — Benchmark comparison fingerprints (manifest + campaign)

Delivered: benchmark/fingerprints.py — SHA-256 over canonical JSON for suite definition (title excluded; sorted tags), prompt set (resolved prompt ids + text digests + registry provenance), provider config (per-variant sorted), scoring config (effective backend, eval_scoring, judge repeat, judge provider), browser config (per-variant sorted). BenchmarkComparisonFingerprints on BenchmarkRunManifest and BenchmarkCampaignRunEntry; schemas/v1/manifest.schema.json, schemas/v1/benchmark_campaign_manifest.schema.json. run_benchmark always emits fingerprints; campaign_runner copies them from member manifests. RunSnapshot.comparison_fingerprints, group_snapshots_by keys suite_definition_fingerprint … browser_config_fingerprint, provider-comparison.md + summary.json updates. Fixture fixtures/benchmarks/longitudinal.v1.yaml, longitudinal manifests updated. Tests tests/test_fingerprints.py, tests/test_longitudinal.py, tests/test_manifest.py, tests/test_benchmark_campaign.py. Docs: data-model.md, longitudinal-reporting.md, benchmarking.md, examples/reports/longitudinal/README.md.

Note: Extended in v0.2.0 with prompt_registry_state, campaign_experiment_fingerprints, and prompt_registry_state_fingerprint (see log entries above).

Verification: uv run pytest (full suite).

2026-04-17 — Benchmark campaign sweeps

Delivered: BenchmarkCampaignDefinitionV1 + schemas/v1/benchmark_campaign.schema.json; BenchmarkCampaignManifest / BenchmarkCampaignRunEntry + schemas/v1/benchmark_campaign_manifest.schema.json. benchmark/campaign_definitions.py, benchmark/campaign_runner.py. CLI alwm benchmark campaign run. Artifact kinds benchmark_campaign_definition, benchmark_campaign_manifest; write_benchmark_campaign_manifest in persistence.py. Fixture fixtures/benchmarks/campaign_micro.v1.yaml; examples examples/campaigns/v1/minimal_offline.v1.yaml, examples/campaign_runs/minimal_offline/. Docs docs/workflows/benchmark-campaigns.md, cross-link in docs/workflows/benchmarking.md, examples/campaigns/v1/README.md. Tests tests/test_benchmark_campaign.py.

Verification: just ci.

2026-04-17 — Agentic benchmark pack (cross-system evaluation)

Delivered: BenchmarkDefinitionV1 / BenchmarkRunManifest optional success_criteria and failure_taxonomy_hints (metadata; JSON Schema + Pydantic + runner copy-through). Prompt registry 0.4.0 — bench.task.repo_implementation.v1, bench.task.docs_drift_repair.v1, bench.task.benchmark_authoring.v1. Five suites under examples/benchmark_suites/v1/agentic/ (repo implementation, docs drift, benchmark authoring, browser interpretation with file-backed evidence, multi-agent coordination). Committed offline runs examples/benchmark_runs/agentic-pack-*. Docs: docs/workflows/agentic-benchmark-pack.md, examples/benchmark_suites/v1/agentic/README.md, updates to benchmarking.md, benchmarks/v1/README.md, examples/benchmark_suites/v1/README.md, data-model.md. Tests: tests/test_agentic_benchmark_pack.py, manifest validation in tests/test_manifest.py.

Verification: just ci or uv run just ci.

2026-04-17 — uv canonical workflow (enforced in docs + justfile)

Policy: Host development must use uv for virtualenv creation (uv venv), installs (uv pip install …), and running tools (uv run …). AGENTS.md states this explicitly; README.md and docs/workflows/local-dev.md / walkthrough-v0.1.0.md default to uv run alwm and uv run just ci without requiring python -m venv or bare pip. justfile recipes now use uv pip for install / install-dev and uv run for pytest, ruff, and mypy (no python -m pip fallback). Runtime: src/.../cli.py and playwright_runner.py user-facing messages updated to uv pip. Docker image builds unchanged (Dockerfile still uses pip inside the image).

Verification: uv run just ci.

2026-04-17 — Repeated semantic judge: aggregation, disagreement, low-confidence flags

Delivered: pipelines/judge_repeat.py — mean, median, trimmed_mean aggregation; per-criterion and total-weighted disagreement; threshold-based assess_low_confidence. Evaluation gains optional judge_repeat_count, judge_semantic_aggregation, judge_low_confidence; JudgeRepeatAggregation + nested models on EvaluationJudgeProvenance.repeat_aggregation (omitted when judge_repeats is 1). evaluate_with_scoring_backend accepts JudgeRepeatParams; mock semantic scores vary by run_index for stable multi-run tests. EvalScoringSpec + schemas/v1/benchmark_definition.schema.json: judge_repeats, semantic_aggregation, trim_fraction, optional stdev/range thresholds. alwm evaluate flags mirror repeat/aggregation/thresholds. Example: examples/benchmarks/v1/semantic_repeats.v1.yaml. Tests: tests/test_evaluation_backends.py, tests/test_benchmark.py. Docs: data-model.md, benchmarking.md, evaluation-pipeline.md, benchmarks/v1/README.md.

Verification: uv run pytest (full suite).

2026-04-17 — Longitudinal benchmark reporting

Delivered: pipelines/longitudinal.py — load runs from manifests + evaluations + optional grid and judge provenance; weekly and longitudinal Markdown; failure taxonomy (FT-*) and failure atlas; regression-focused regression.md and provider-comparison.md; optional run_context.json grouping via group_snapshots_by. CLI alwm benchmark longitudinal (--runs-glob, thresholds for regression, semantic instability, series swing). Lazy benchmark_run_context registration in artifacts.py to break an import cycle. schemas/v1/evaluation.schema.json — nullable semantic judge fields aligned with Evaluation. Fixtures fixtures/longitudinal/paired/; example bundle examples/reports/longitudinal/sample-output/; tests tests/test_longitudinal.py. Docs docs/workflows/longitudinal-reporting.md, examples/reports/longitudinal/README.md.

Verification: just ci.

2026-04-17 — Semantic and hybrid rubric scoring (optional)

Delivered: pipelines/evaluation_backends.py — backends deterministic (default), semantic_judge, hybrid; EvaluationJudgeProvenance model + schemas/v1/evaluation_judge_provenance.schema.json; artifact kind evaluation_judge_provenance. Evaluation gains scoring_backend, judge_provenance_relpath. BenchmarkDefinitionV1.eval_scoring (EvalScoringSpec, EvalHybridWeights); load_judge_provider_config in providers/benchmark_config.py. Benchmark cells write evaluation_judge_provenance.json when non-deterministic; manifest cells include optional judge_provenance_relpath. CLI: alwm evaluate (--scoring-backend, --judge-provider-config, --judge-live, --hybrid-deterministic-weight); alwm benchmark run (--eval-scoring-backend, --judge-provider-config). Opt-in live judge: ALWM_JUDGE_LIVE=1; fixture mode keeps mock judge unless set. Tests: tests/test_evaluation_backends.py. Docs: data-model.md, evaluation-pipeline.md, benchmarking.md. Fixtures: fixtures/v1/evaluation_judge_provenance.json.

Verification: just ci.

2026-04-17 — Browser-backed benchmark execution

Why: browser_mock previously only tagged LLM output; benchmarks did not exercise BrowserRunner or persist evidence.

Delivered: BrowserBenchConfig + optional variant.browser in benchmark/definitions.py and schemas/v1/benchmark_definition.schema.json. run_benchmark calls run_benchmark_browser_phase for execution_mode: browser_mock, writes cells/.../browser_evidence.json, augments the provider prompt, and records browser_runner / browser_evidence_relpath on request/response/manifest cells. MCPBrowserRunner (browser/mcp_runner.py): fixture-backed bridge; remote MCP tools not wired. Playwright gated by ALWM_BENCHMARK_PLAYWRIGHT=1. Fixtures under fixtures/benchmarks/browser_*.v1.yaml; example examples/benchmarks/v1/browser_file.v1.yaml. Tests: tests/test_benchmark_browser.py. Docs: docs/workflows/benchmarking.md, benchmarks/v1/README.md, docs/architecture/browser.md, docs/audits/capability-classification.md. evaluation.schema.json: optional scoring_backend / judge_provenance_relpath (not in required) for Pydantic alignment.

Verification: just ci (default tests; no Playwright in CI).

2026-04-17 — Benchmark corpus expansion: taxonomy metadata + eight tagged suites

Delivered: BenchmarkTaxonomyV1, optional time_budget_seconds, token_budget, retry_policy, tags, expected_artifact_kinds on BenchmarkDefinitionV1; same fields on BenchmarkRunManifest and schemas/v1/manifest.schema.json (optional). New prompts bench.task.runtime_config.v1, bench.task.multi_agent_coord.v1; registry 0.3.0. Eight example suites under examples/benchmark_suites/v1/suite.taxonomy.*.v1.yaml covering repo, runtime, docs, browser evidence, matrix reasoning, multi-agent coordination, campaign sweep, and integration stress. Fixture fixtures/benchmarks/suite_taxonomy_minimal.v1.yaml; example runs examples/benchmark_runs/taxonomy-repo-governance/, taxonomy-runtime-config/. Tests: tests/test_benchmark_taxonomy.py, manifest validation. Docs: docs/workflows/benchmarking.md, examples/benchmark_suites/v1/README.md. schemas/v1/benchmark_taxonomy.schema.json for standalone taxonomy JSON.

2026-04-17 — v0.1.0 release preparation

Delivered: First semver release tag readiness: CHANGELOG.md, docs/releases/v0.1.0.md, docs/release-readiness.md, docs/audits/release-readiness-audit.md, docs/workflows/walkthrough-v0.1.0.md. README quickstarts (exact local, benchmark, prompt registry, manifest validate, optional live verification) and pyproject.toml urls + sdist CHANGELOG.md include. Version metadata remains 0.1.0 (pyproject.toml, __version__). Evidence: just ci (85 passed, 1 skipped), just compose-help, docker buildx bake --print.

2026-04-17 — Post-merge stabilization (parallel tracks)

Delivered: Single coherent tree on main: lazy run_benchmark in benchmark/__init__.py to avoid artifacts ↔ pipelines import cycles after manifest consolidation; audit docs refreshed (mission-gap-audit.md, current-capability-status.md) with 85 passed, 1 skipped from just ci; staged governance and benchmark expansion artifacts (registry suites, example runs, test_benchmark_expansion).

2026-04-17 — Benchmark manifest: JSON Schema + benchmark_manifest artifact kind

Delivered: schemas/v1/manifest.schema.json (Draft 2020-12); BenchmarkRunManifest / BenchmarkCellArtifactPaths moved to models.py (re-exported from benchmark/manifest.py) to avoid import cycles with artifacts.py. Registered benchmark_manifest in artifacts.py; write_benchmark_manifest in benchmark/persistence.py validates against JSON Schema before writing; run_benchmark uses it. CLI: alwm validate … benchmark_manifest. Fixture fixtures/v1/manifest.json; tests tests/test_manifest.py + test_domain / test_benchmark hooks. Docs: data-model.md, benchmarking.md, examples/v1/README.md. Committed example runs under examples/benchmark_runs/ unchanged (backward compatible: optional provenance keys omitted).

2026-04-17 — Benchmark expansion: registry suites, comparison rubric, manifest provenance

Delivered: Four versioned prompts under prompts/versions/ (bench.task.repo_governed.v1, markdown_synthesis, matrix_reasoning, browser_evidence) registered in prompts/registry.yaml (0.2.0). New rubric examples/dataset/rubrics/comparison.v1.json (structure / task_fit / grounding / brevity). Three example suites: suite.registry.four_modes.v1.yaml, suite.registry.strict_duo.v1.yaml, suite.registry.generous_duo.v1.yaml; fixture mirror fixtures/benchmarks/suite_four_modes.v1.yaml. Committed offline runs under examples/benchmark_runs/registry-four-modes, registry-strict-duo, registry-generous-duo. BenchmarkRunManifest + run_benchmark(..., definition_source_relpath=...) + CLI wiring for optional manifest fields. Tests: tests/test_benchmark_expansion.py. Docs: data-model.md, evaluation-pipeline.md, benchmarking.md, examples/benchmark_suites/v1/README.md.

2026-04-17 — Runtime hardening: live verification paths

Delivered: PlaywrightBrowserRunner exported from playwright_runner (MCP-only stub file); create_browser_runner(..., "playwright"); Dockerfile target browser-test (Playwright + Chromium); Compose profile browser-verify; docker-bake.hcl target browser-test (linux/amd64); just recipes verify-live-providers, verify-playwright-local, browser-verify; pytest marker live_playwright; tests/integration/test_playwright_browser.py marked integration + live_playwright; docs/workflows/live-verification.md; docs/architecture/runtime.md refresh; workflow cross-links; .env.example notes ALWM_PLAYWRIGHT_SMOKE. Default just ci unchanged (tests/integration/ still excluded).

2026-04-17 — Governance: AGENTS manual, multi-agent workflow, capability audit

Why: Reduce drift between docs and implementation; make parallel agent work predictable; require evidence-backed labels (“complete” vs partial/stub).

What changed:

• AGENTS.md — Stronger operating manual: non-goals folded into success criteria (evidence-backed claims), multi-agent rules (summary + pointer), expanded anti-patterns, verification table (Playwright opt-in), links to docs/audits/capability-classification.md, updated browser decision rule (Playwright optional; MCP stub), layout includes docs/audits/, docs/examples/, docs/workflows/multi-agent-parallel.md.
• docs/workflows/multi-agent-parallel.md — Branch strategy, file ownership zones, conflict avoidance, merge order, required handoff template.
• docs/audits/capability-classification.md — Taxonomy: complete / partial / stub / documented-only / broken + evidence bar and repository examples (re-verify after changes).
• docs/examples/README.md — Clarifies repo-root examples/ vs docs/examples/.
• Drift repair — README.md: alwm browser run-playwright, optional [browser] install comment, pointer to multi-agent doc. docs/architecture/browser.md, current-state.md, runtime.md: Playwright optional; MCP stub; prompt registry implemented in current-state. docs/audits/mission-gap-audit.md: browser §3 and doc-drift rows updated; P3 reframed to MCP runner.

Verification: Documentation-only change set; command names cross-checked against src/agent_llm_wiki_matrix/cli.py (browser run-playwright, benchmark probe, prompts group).

2026-04-17 — AGENTS.md operating manual

Why: The old AGENTS.md was principles-only; agents needed actionable rules for this repo (benchmarks, prompt registry, integration tests, browser stubs, documentation touchpoints).

What changed: Expanded AGENTS.md with mission/success criteria, required contribution loop, decision table (including prefer prompt_ref over duplicated inline prompt text where appropriate), anti-patterns, verification/reporting table, documentation update rules, completion vs partial vs stubbed definitions, and project-specific examples (implementation, auditing, drift repair, benchmarks, browser). README now points contributors to AGENTS.md for the full manual.

How to use it: Before sizable work, read Mission, Contribution loop, and Prompt registry sections; after work, satisfy Verification + Documentation update rules and add an implementation-log entry when behavior or contracts shift.

2026-04-17 — Registry-backed benchmark prompts

Delivered: PromptItem supports text xor prompt_ref (+ optional registry_version pin); BenchmarkDefinitionV1.prompt_registry_ref; resolve_benchmark_prompts (benchmark/prompt_resolution.py) using load_prompt_registry_yaml; run_benchmark(..., prompt_registry_path=...) and CLI --prompt-registry; BenchmarkRequestRecord / BenchmarkResponse fields prompt_source, prompt_registry_id, registry_document_version, prompt_source_relpath; JSON Schema updates + schemas/v1/benchmark_definition.schema.json. Fixtures under fixtures/prompt_registry/ and fixtures/benchmarks/registry_*.yaml; tests tests/test_benchmark_prompt_registry.py; example examples/benchmark_suites/v1/registry.mixed.v1.yaml + committed run examples/benchmark_runs/registry-mixed/. Prompt bench.sample.prompt.v1 added under prompts/.

2026-04-17 — Live benchmark verification (Compose + opt-in integration tests)

Delivered: benchmark/live_probe.py; CLI alwm benchmark probe; Ollama healthcheck + service_healthy for benchmark-ollama; Compose profile benchmark-probe (just benchmark-probe); integration tests under tests/integration/ gated by ALWM_LIVE_BENCHMARK_OLLAMA / ALWM_LIVE_BENCHMARK_LLAMACPP with skips when unreachable; just test ignores tests/integration/; tests/test_live_probe.py unit tests.

2026-04-17 — Mission gap audit + prompt registry CLI

Delivered: docs/audits/mission-gap-audit.md (code-verified status of CLI, providers, browser layer, Compose, Bake; drift notes). Prompt registry wiring: schemas/v1/prompt_registry.schema.json, prompt_registry.py (YAML load + JSON Schema + path safety), CLI alwm prompts check|list|show, tests tests/test_prompt_registry.py. README and AGENTS.md updated for new commands.

2026-04-17 — Browser execution abstraction

Delivered: BrowserRunner ABC; BrowserEvidence JSON Schema + Pydantic models; MockBrowserRunner and FileBrowserRunner; fixture fixtures/browser_evidence/v1/export_flow.json; PlaywrightBrowserRunner / MCPBrowserRunner stubs (NotImplementedError); load_browser_evidence, evidence_to_prompt_block; artifact kind browser_evidence for alwm validate; CLI alwm browser prompt-block and alwm browser run-mock; tests tests/test_browser.py; docs/architecture/browser.md.

Not in scope: live Playwright or MCP browser automation (stubs only).

2026-04-17 — Benchmark run artifact persistence

Delivered: Structured output under cells/, matrices/, markdown/, reports/ with stable cell_id slugs, benchmark_request + raw/normalized response text files, aggregate benchmark_response, per-cell evaluation, matrix_grid_inputs / matrix_pairwise_inputs, and manifest cells[] index; execution layer returns raw vs normalized text.

2026-04-17 — Replace Makefile with justfile

Delivered: just justfile with the same recipes as the former Makefile (ci, lint, compose-help, benchmark profiles, etc.); removed Makefile; documentation and AGENTS.md now reference just ….

2026-04-17 — Benchmark case schema (v1) + task examples

Delivered:

• JSON Schema schemas/v1/benchmark_case.schema.json and Pydantic BenchmarkCase / BenchmarkExecutionMetadata with validation that expected_artifact_kinds references registered alwm validate kinds.
• Loader benchmark/cases.py (load_benchmark_case, validate_benchmark_case_file).
• Catalog benchmarks/cases/v1/ with four task kinds: repo scaffolding, Markdown synthesis, comparison matrix, browser-evidence interpretation; mirrored under fixtures/benchmark_cases/v1/ and examples/benchmark_cases/v1/.
• Tests: tests/test_benchmark_cases.py.

2026-04-17 — Phase 5 complete (benchmark harness)

Delivered:

• Provider execution: providers/execution.py applies deterministic execution-mode tags (cli, browser_mock, repo_governed) on top of the same CompletionRequest for all backends.
• Backend selection: providers/benchmark_config.py merges global YAML/env with per-variant mock / ollama / openai_compatible settings; ALWM_FIXTURE_MODE=1 forces mock unless --no-fixture-mock.
• Benchmark definitions: Pydantic BenchmarkDefinitionV1 + load_benchmark_definition for YAML/JSON under benchmarks/v1/ (mirrored in fixtures/benchmarks/ for tests).
• Artifacts: schemas/v1/benchmark_response.schema.json, BenchmarkResponse model, alwm validate … benchmark_response.
• Pipeline: benchmark/runner.py stores raw responses, rubric-evaluates each cell, builds grid and pairwise ComparisonMatrix JSON + Markdown, report.json / report.md, and manifest.json.
• CLI: alwm benchmark run --definition … --output-dir ….
• Compose / just: profiles benchmark-offline, benchmark-ollama (+ ollama service), benchmark-llamacpp; recipes just benchmark-offline, just benchmark-ollama, just benchmark-llamacpp.
• Tests: tests/test_benchmark.py asserts deterministic offline matrix output and validates artifacts.

Next: Richer prompt registry wiring; optional LLM rubric; real browser trace ingestion.

2026-04-17 — Phase 4 complete (pipelines + Compose test image)

Delivered:

• Rubric JSON Schema (schemas/v1/rubric.schema.json) + Pydantic models; alwm validate … rubric.
• Pipelines: ingest_markdown_pages, deterministic evaluate_subject, evaluations_to_matrix, build_report_from_matrix + Markdown renderers for templates/matrix.md and templates/report.md.
• CLI: alwm ingest, evaluate, compare (optional --out-md), report.
• Docker: Dockerfile test stage (editable .[dev] + pytest); Compose services orchestrator (dev), tests (test), benchmark (benchmark); just compose-help validates all profiles.
• Examples: examples/dataset/ wiki pages, evaluations, thoughts, and examples/generated/ matrix + report pair; tests validate checked-in generated JSON.
• Docs: Updated docs/workflows/*, docs/architecture/current-state.md, docs/architecture/evaluation-pipeline.md, docs/architecture/runtime.md, and README.md with exact commands.

Next: Optional LLM-assisted scoring behind providers; richer ingest extraction; browser evidence fixtures.

2026-04-17 — Phase 3 complete (provider abstraction)

Delivered:

• BaseProvider + CompletionRequest contract.
• MockProvider (deterministic, no network).
• OllamaProvider (/api/chat) and OpenAICompatibleProvider (/v1/chat/completions) using httpx with injectable MockTransport for tests.
• ProviderConfig loaded from optional YAML + environment overrides (load_provider_config).
• Example YAML: config/providers.example.yaml; local config/providers.yaml gitignored.
• CLI: alwm providers show prints redacted configuration.
• Tests in tests/test_providers.py (mocked HTTP only).

Next: Phase 4 — ingest pipeline, claim extraction stubs, rubric evaluation, matrix persistence.

2026-04-17 — Phase 2 complete (domain models + schemas)

Delivered:

• JSON Schemas under schemas/v1/ for thought, event, experiment, evaluation, matrix, report (plus existing note).
• Pydantic models in src/agent_llm_wiki_matrix/models.py with extra validation for matrix score shapes.
• src/agent_llm_wiki_matrix/artifacts.py for dual JSON Schema + Pydantic validation.
• CLI: alwm validate <path> <kind>.
• Fixtures: fixtures/v1/*.json; examples mirrored under examples/v1/.
• Markdown templates: templates/thought.md, event.md, experiment.md, evaluation.md, matrix.md, report.md (plus weekly stub).
• Tests: tests/test_domain.py, tests/conftest.py (repo root + cwd for deterministic schema resolution).

Next: Phase 3 — provider abstraction (BaseProvider, Ollama, OpenAI-compatible HTTP, mock) with YAML/env configuration.

2026-04-17 — Phase 1 complete (scaffold)

Audit: Initial repository contained only LICENSE. No prior scaffold to migrate.

Delivered:

• Repository layout: src/agent_llm_wiki_matrix/, tests/, docs/, schemas/, templates/, prompts/, examples/, fixtures/.
• Top-level: README.md, AGENTS.md, pyproject.toml, justfile, .gitignore, .env.example, Dockerfile, docker-compose.yml, docker-bake.hcl.
• Python package: Click CLI (alwm), structlog-based logging, JSON Schema helpers (schema.py) with path-aware caching.
• Schemas: schemas/v1/note.schema.json; example examples/sample-note.json.
• Prompt registry skeleton: prompts/registry.yaml + prompts/versions/scaffold.echo.v1.txt.
• Report template skeleton: templates/report-weekly.md.
• Smoke tests: just smoke runs scripts/smoke.sh — pytest -m smoke, host alwm benchmark + benchmark campaign, Docker Compose config + dev + benchmark-offline (optional; SMOKE_SKIP_DOCKER=1); prints failure recovery analysis on errors. Docs: docs/workflows/smoke.md. Dockerfile bundles templates/ for runtime image matrix/report rendering.
• Docker: orchestrator image (Dockerfile); Compose profiles dev, test, benchmark; Bake targets for multi-arch and single-arch convenience (orchestrator-amd64, orchestrator-arm64).

Known gaps (by design until later phases):

• Domain models beyond minimal note schema (Phase 2).
• Provider adapters (Phase 3).
• Ingest/evaluate/matrix/report pipelines (Phases 4–5).
• Compose services for Ollama / model endpoints (Phase 6; placeholder comment in docker-compose.yml).

Next: Phase 2 — domain models and schemas for thought, event, experiment, evaluation, matrix, report; validators; expanded fixtures and examples.