Audit: v0.2.4 publication workflow readiness¶
Date: 2026-04-19
Scope: End-to-end path campaign run → validate → result pack → pack-check → compare (packs or directories) → interpret → publish, aligned with docs/roadmap/v0.2.4.md exit themes.
Method: Evidence-driven, deterministic — review of committed docs, tests, and inventory files only (no live providers, browsers, or network). No product code was executed for this audit beyond what is already documented in-repo.
Related: docs/tracking/v0.2.4-campaign.md · docs/workflows/campaign-result-pack-publication.md · docs/audits/schema-drift-contracts-inventory.md
Executive summary¶
The repository already has a coherent operator story: a single checklist (campaign-result-pack-publication.md), a verification matrix (verification.md), drift guards on most committed JSON kinds (test_schema_drift_contracts.py), and committed exemplars for pack and compare outputs. The largest contract gap for publication confidence is reports/campaign-analysis.json: it is not a registered alwm validate kind and is excluded from the drift sweep by explicit policy (schema-drift-contracts-inventory.md § “Not covered”). Secondary gaps are documentation freshness (some audit files pinned to older versions), readability of dense compare tables when browser evidence is present, and operator friction when just is not installed (fallbacks exist but are easy to miss).
Workflow trace (evidence)¶
| Phase | Primary artifacts | Automated contract check | Notes |
|---|---|---|---|
| 1. Run | Campaign tree (manifest.json, runs/, summaries, reports) |
benchmark_campaign_manifest, benchmark_manifest, campaign_summary, … per drift sweep |
Campaign analysis JSON used everywhere in tooling but not drift-swept (see inventory). |
| 2. Validate (campaign) | alwm validate <path> <kind> |
Drift test + targeted tests (e.g. test_benchmark_campaign.py, test_campaign_reporting.py) | Docs list manual commands; not one umbrella “validate campaign dir” CLI. |
| 3. Pack | campaign-result-pack.json, INDEX.md, mirrored layout |
campaign_result_pack in drift sweep |
test_campaign_result_pack.py covers assembly + pack-check behavior. |
| 4. Validate (pack) | Schema + pack-check |
campaign_result_pack drift; pack-check exercised in tests, not full examples/ packs as CI subprocess |
Operators must still run pack-check on real bundles; strict mode is documented. |
| 5. Compare | pack-compare.json / campaign-compare.json + Markdown |
Both kinds in drift sweep | Committed examples: examples/campaign_result_packs/compare_minimal_vs_multi/, examples/campaign_compares/minimal_offline_vs_multi_suite/. |
| 6. Interpret | Human reports + JSON mirrors | Partial — compare JSON validated; narrative quality is human review | campaign-result-pack-publication.md §6 maps sections to intent. |
| 7. Publish | Git / archive | — | Policy on redaction / external handoff still open in v0.2.4-campaign.md. |
Findings¶
A. Validation & contracts¶
campaign-analysis.jsondrift — Documented as out of scope forvalidate-artifacts(schema-drift-contracts-inventory.md). Comparative tooling and pack compare depend on this file’s shape; silent drift in committed examples is possible without a dedicated kind or golden snapshot test.pack-checkvs drift — Structural / path consistency is validated byalwm benchmark campaign pack-check, with unit coverage in test_campaign_result_pack.py. There is no CI step that runspack-checkagainst every committedexamples/campaign_result_packs/*/tree (would be deterministic if added as a pytest or small script).- Version metadata —
pyproject.toml/alwm versionreport 0.2.3 while v0.2.4 is the active milestone; expected pre-release, but release-readiness and audit docs that still pin v0.1.0 (release-readiness-audit.md) confuse readers auditing “what is current.”
B. Documentation¶
- Multiple entry points — README.md, AGENTS.md, benchmark-campaigns.md, campaign-result-pack-publication.md, docs/wiki/benchmark-campaigns.md — content is aligned in theme but cross-link maintenance is a recurring cost (verification.md explicitly calls out updating several files when
justfilechanges; same principle applies to workflow edits). - Reproducible committed compares —
campaign compareexemplar uses--created-at(examples/campaign_compares/README.md).compare-packsexamples emphasize--repo-root; timestamps inpack-compare.jsonstill change on regeneration unless documented as acceptable churn.
C. Output readability¶
- Compare reports — Pack/campaign compare Markdown can be wide (e.g. browser evidence pairing tables). Usable for review, but executive summary / “so what” lines are not always first-class (tracking v0.2.4-campaign.md lists this as an open question).
- INDEX.md vs README — Audience split (internal vs external) is noted as open in tracking; no single convention is enforced repo-wide.
Prioritized remediation (no major features)¶
| Priority | Item | Rationale | Suggested work (deterministic) |
|---|---|---|---|
| P0 | Clarify campaign-analysis contract |
Central to compare + reports; currently the weakest drift story | Either add a registered kind + schema + drift glob, or add pytest that loads committed examples/**/reports/campaign-analysis.json with a minimal structural validator / snapshot subset (no scoring changes). |
| P0 | Refresh stale audit metadata | Avoids wrong conclusions during release | Update release-readiness-audit.md header / version table or mark superseded with pointer to this file and v0.2.3 release notes. |
| P1 | Optional pack-check on committed packs |
Closes gap between “examples validate as JSON” and “bundle is internally consistent” | Add pytest (or just recipe) that runs pack-check on examples/campaign_result_packs/*/ (non-strict by default; strict in --strict CI job optional). |
| P1 | Document timestamp / churn policy | Reduces noise for reviewers of pack-compare.json |
One paragraph in campaign-result-pack-publication.md or compare README: what must be stable vs regenerated fields. |
| P2 | Compare report skimmability | Milestone asks for human-readable comparisons | Short top-of-report bullet summary (deterministic strings from existing JSON) before wide tables — small renderer change when touching compare Markdown. |
| P2 | INDEX / README convention | Publication handoff clarity | Add one line to campaign-result-pack-publication.md §7: “Prefer INDEX.md for pack navigation; optional README only when …” |
| P3 | Wiki duplicate drift | docs/wiki/benchmark-campaigns.md vs main workflow doc | Periodic diff or “generated from” note in wiki index; no new feature. |
Non-goals (this audit)¶
- No new features — dashboards, hosted runners, remote MCP, or replacing offline CI with live benchmarks.
- No network or live verification — evidence is from repository state only.
- No scoring or harness behavior changes — contract and documentation posture only.
Verification commands (release gate)¶
Authoritative matrix: docs/workflows/verification.md.
uv run ruff check src tests
uv run mypy src
uv run pytest tests/ --ignore=tests/integration/
uv run pytest tests/test_schema_drift_contracts.py -v
Optional: alwm benchmark campaign pack-check on each committed example pack directory (not yet a single bundled recipe; see P1).
Sign-off¶
This audit supports v0.2.4 planning and release notes; it does not constitute a release tag decision. Next step: address P0/P1 items or explicitly defer them in docs/tracking/v0.2.4-campaign.md before tagging v0.2.4.