PARQONAUT

Parquet Analysis, Rewriting, Quality, Orchestration, Navigation, Auditing, Unification & Transformation

PARQONAUT helps teams explore, diagnose, repair, and orchestrate Parquet datasets — from a single file to multi-dataset fleets on local disk or S3-compatible object storage.

The public command-line tool is prqnt. The optional application server (prqnt serve) exposes the same core workflows over /api/v1.

What you can do today

Install

git clone https://github.com/sempervent/PARQONAUT
cd PARQONAUT
cargo build --release -p parqonaut-cli --bin prqnt

See Installation for cargo install --git … and feature flags.

Quick examples

prqnt scan ./dataset

prqnt doctor ./dataset

prqnt repair ./dataset \
  --plan repair-plan.json \
  --output repaired/

prqnt batch plan \
  --config batch.toml \
  --output batch-plan.json

prqnt scan s3://bucket/prefix/

prqnt serve

Where to go next

• Getting started — first successful scan and repair
• CLI overview — commands and exit semantics
• Application server — prqnt serve, jobs, and storage policy
• Architecture — how CLI, HTTP, and parqonaut-app fit together

Current boundaries (v0.7)

PARQONAUT shares application logic between CLI and HTTP via parqonaut-app. Deep CLI/API fingerprint parity is enforced primarily through that shared layer rather than one exhaustive cross-transport test matrix. Full HTTP repair-over-S3 end-to-end coverage is still growing. prqnt doctor is a CLI-oriented workflow and is not exposed as a separate durable API job in v0.7.

Quick start

This guide walks through a local scan and a non-destructive repair plan using repository fixtures.

Prerequisites

• Rust stable (rustfmt, clippy)
• Built prqnt binary (see Installation)

Scan a fixture dataset

prqnt scan fixtures/scan/single_parquet/data.parquet

Use --json for machine-readable output on supported commands.

Diagnose and plan

prqnt doctor fixtures/schema/frankenlake-v2 \
  --policy fixtures/schema/policy.toml

doctor runs scan → diagnose → plan in one invocation. It does not write repaired data unless you pass --repair and --output.

Generate a plan file explicitly:

prqnt plan fixtures/schema/frankenlake-v2 \
  --policy fixtures/schema/policy.toml \
  --output /tmp/frankenlake-plan.json

Execute a safe repair (separate output)

Never repair in place — always write to a new directory:

prqnt repair fixtures/schema/frankenlake-v2 \
  --plan /tmp/frankenlake-plan.json \
  --output /tmp/frankenlake-repaired

Operations classified ReviewRequired need explicit authorization:

prqnt repair … --authorize <operation_id>

CI-style check

prqnt check fixtures/schema/frankenlake-v2 \
  --policy docs/ci-policy.toml

Exit codes are stable; see CLI overview.

Try the HTTP server

prqnt serve --state-dir /tmp/prqnt-demo --listen 127.0.0.1:8080

Bootstrap an admin token with PRQNT_BOOTSTRAP_ADMIN_TOKEN (never commit or log the value). See Authentication and HTTP API.

Installation

PARQONAUT ships as a single executable: prqnt.

There is no Homebrew formula or crates.io package for PARQONAUT at this time. Supported paths are source build and cargo install --git.

Build from source

git clone https://github.com/sempervent/PARQONAUT
cd PARQONAUT
cargo build --release -p parqonaut-cli --bin prqnt

The binary is at target/release/prqnt.

Developer automation:

just setup   # rustfmt + clippy components
just build
just test

Install with Cargo (git)

cargo install --git https://github.com/sempervent/PARQONAUT --bin prqnt

This compiles the latest main (or the ref you pin with --tag v0.7.1). Ensure Rust stable is installed.

S3 / object storage features

S3-compatible backends (AWS, MinIO, RustFS in tests) use the parqonaut-storage S3 backend. The CLI binary includes S3 support in the default workspace build used for releases in this repository.

For local integration tests, see Object storage and just s3-demo.

Application server

prqnt serve uses SQLite by default under --state-dir. Postgres is supported via --database (Postgres URL). See prqnt serve.

Command overview

All user-facing operations go through prqnt. Global flag:

• --json — JSON output where the command supports it (errors may still print JSON when --json is set).

Exit semantics

Exact behavior is covered by CLI integration tests; do not rely on undocumented codes.

Capability map

When to use which command

• scan — inventory, findings, dataset identity; input to planning.
• doctor — interactive CLI path: scan + diagnose + plan (optional repair). Not a separate HTTP job in v0.7.
• plan / repair / verify — durable, evidence-bound repair pipeline with separate output locations.
• check — CI gate; non-mutating.
• batch plan|repair|resume|verify — fleet operations with journal and resume semantics.
• serve — automation, remote operators, async jobs.

Generated reference

Flag-level detail is generated from Clap to avoid drift:

• CLI reference (generated) — refresh with cargo xtask docs cli

Transform workflows

PARQONAUT v0.9 runs Parquet transforms on the shared parqonaut-columnar batch pipeline (Apache Arrow/Parquet 54.3.1). Streamable transform spec steps can fuse in memory with zero intermediate Parquet staging; barriers (for example split) appear explicitly in dry-run plans.

Commands

# Hive-style partition (LRU-backed writer cap)
prqnt partition input.parquet -o out/ --partition-by country,date

# Merge compatible Parquet inputs
prqnt merge 'parts/*.parquet' -o merged.parquet

# Split by target size
prqnt split large.parquet -o parts/ --target-size-mb 512

# Declarative pipeline (YAML)
prqnt transform --spec fixtures/transform/specs/rewrite-partition.yaml
prqnt transform --spec workflow.yaml --check
prqnt transform --spec workflow.yaml --dry-run --json

Spec format

Sequential steps only: step N output feeds step N+1 unless an operation declares its own source. When the compiled plan is fully streamable, steps run in memory and no .parqonaut-spec-* directory is created. Non-fusible steps (barriers) may still materialize under .parqonaut-spec-<execution-id>/ until the run completes.

Canonical example (fixtures/transform/specs/rewrite-partition.yaml):

schema-version: 1
input: fixtures/transform/partition-basic/input.parquet
output: target/spec-demo/processed
steps:
  - operation:
      type: rewrite
      compression: zstd
  - operation:
      type: partition
      partition-by:
        - country

--check validates without mutation. --dry-run resolves the typed execution plan (JSON report includes structured plan).

Object storage

Rewrite, merge, split, partition, and fused streamable specs resolve independent source and sink backends (ColumnarPipelineIo): local and s3:// legs compose in all four combinations. Parquet outputs stream through a bounded encoder and multipart upload — no whole-object buffering (see Object storage and Pipelines).

Streaming conversion

prqnt convert runs the schema-aware streaming engine (parqonaut-stream) on Apache Arrow/Parquet 54.3.1 RecordBatch streams via parqonaut-columnar.

Inputs and outputs may be local paths or s3:// URIs (with S3-enabled builds). Parquet written to object storage streams through the storage columnar sink (bounded encoder + multipart upload), not an in-memory object buffer.

Schema discovery and unification

Inputs are discovered recursively (unless --no-recursive). Schemas are unified before writing:

Shared semantic vectors live in fixtures/schema/compatibility-vectors.json and are tested against stream unification.

Resumable conversion

prqnt convert 'data/*.csv' -o out.parquet --out-format parquet \
  --state target/convert-state.json

# After interruption
prqnt convert 'data/*.csv' -o out.parquet --out-format parquet \
  --state target/convert-state.json --resume

Checkpoints bind input fingerprints (path, size, mtime), output location, format, schema policy, compression, and batch settings. Stale sources produce STALE CHECKPOINT (no silent restart). Incomplete output is staged under <output>.staging/ until all inputs finish.

CSV and Parquet resume granularity in v0.8: completed-file boundary (row-group resume planned).

Progress

Human progress renders on TTY stderr when progress is enabled and --quiet / JSON-only modes are off. Automation may consume JSON Lines progress via --json-progress (see CLI reference).

Scanning

prqnt scan performs a forensic scan of a local path or an s3:// dataset prefix.

prqnt scan ./dataset
prqnt scan s3://my-bucket/prefix/ --profile standard

Scans produce structured findings, summaries, and dataset identity used by diagnose/plan/repair. On the server, POST /api/v1/scans submits a durable async scan job with the same application semantics as the CLI.

See also Object storage and CLI reference.

Diagnosis and repair

PARQONAUT repair is evidence-driven and plan-bound:

1. Scan the source dataset (read-only).
2. Diagnose findings into repair opportunities.
3. Plan — produce a durable JSON plan with operation IDs, safety class, and fingerprints.
4. Repair — write to a separate output path or s3:// prefix; never in-place by default.
5. Verify — compare before/after invariants and optional execution manifests.

CLI commands (diagnose, plan, repair, verify, check, doctor) call parqonaut-app. HTTP clients use /api/v1/diagnose, /plans, /checks, /repairs, /verifications as documented in HTTP API.

Safety and authorization

• Operations are classified (safe vs ReviewRequired).
• Review-required steps need explicit --authorize <operation_id> on CLI or equivalent API authorization.
• Plans bind to dataset fingerprints; stale plans are rejected.

Deep dives:

• Repair model
• Repair planning
• Repair policy
• Repair plan contract

Schema reconciliation

PARQONAUT resolves schema drift using deterministic comparison, a compatibility lattice, and explicit policy. No inference or AI is used.

v0.9 centralizes batch/stream/repair compatibility in parqonaut-columnar::schema (see docs/development/COLUMNAR_CONTRACT_SHA). Stream --schema-conflicts widen and repair planning both consult the same semantic rules.

Structured schema diff

Each field difference records:

• path — dot-separated field path
• left / right — normalized descriptors (physical type + nullability)
• difference — e.g. compatible_numeric_widening, nullable_difference, incompatible_type_change
• compatibility — lossless, conditionally_lossless, potentially_lossy, or unsupported

Policy

Configure in TOML (embedded in repair plans):

[schema]
allow_numeric_widening = true
allow_nullable_widening = true
allow_column_reorder = true
allow_missing_nullable_columns = false

[schema.rename]
cust_id = "customer_id"

Renames are explicit only. PARQONAUT never guesses similar column names.

Target schema

1. Explicit — --target-schema schema.json (highest authority)
2. Dataset-derived — canonical schema when all conflicts resolve losslessly under policy

If types cannot be resolved deterministically (e.g. utf8 vs int64 for the same column), PARQONAUT emits UnresolvableSchemaConflict and blocks automatic repair.

Authorization

Compatible widening and nullable widening are ReviewRequired. Execute with:

prqnt repair dataset/ --plan plan.json --authorize <operation_id> --output repaired/

Safe operations run under automatic-safe-policy without extra flags.

Batch batch orchestration

Batch orchestration adds multi-dataset batch orchestration on top of the single-dataset repair engine.

Goals

• Bounded parallel dataset repair (--jobs N)
• Durable SQLite run journal with resume
• Failure isolation per dataset
• Plan-bound authorization and output path safety
• Cooperative cancellation (SIGINT)

Crate boundary

parqonaut-repair       → one dataset
parqonaut-orchestrator → many datasets, journal, scheduler
parqonaut-cli          → `prqnt batch …` commands

See ADR-0011 through ADR-0014.

Commands

prqnt batch check   --config batch.toml
prqnt batch plan    --config batch.toml --output plan.json
prqnt batch repair  --plan plan.json [--jobs N] [--dry-run]
prqnt batch status  --run-dir .parqonaut/runs/<run-id>
prqnt batch resume  --run-dir .parqonaut/runs/<run-id> [--jobs N]
prqnt batch verify  --run-dir .parqonaut/runs/<run-id>

SHIPWRECK fixture fleet

fixtures/orchestration/shipwreck/ — reproducible torture lab (just orchestration-fixtures).

Demos: just batch-demo, just batch-resume-demo.

Related docs

• Batch execution
• Resume and recovery
• Concurrency audit (history)
• Acceptance matrix (history)

Object storage (S3-compatible)

PARQONAUT uses S3StorageBackend for remote datasets. URIs use the standard form:

s3://bucket/prefix/

Supported workflows include scan, plan, repair, batch, transform (rewrite, merge, split, partition, fused specs), prqnt convert, and server jobs when storage policy allows the bucket/prefix.

Streaming Parquet publication (v0.9.1)

Remote Parquet outputs follow:

RecordBatch stream → Parquet encoder (blocking task) → bounded byte channel → multipart S3 upload → finalized object

Create-only destinations use conditional_create; direct write_stream targets replace on successful finalize. Failed uploads do not commit a readable destination object.

RustFS in tests

CI and local integration tests use RustFS (pinned container image) as an S3-compatible backend — not a separate product API. Example environment (fake credentials for local lab only):

export PARQONAUT_S3_ENDPOINT=http://127.0.0.1:9000
export PARQONAUT_S3_PATH_STYLE=true
export AWS_ACCESS_KEY_ID=rustfsadmin
export AWS_SECRET_ACCESS_KEY=rustfsadmin
just s3-demo

Server storage policy

prqnt serve enforces allowed_s3_buckets and allowed_s3_prefixes before any I/O. See Storage policy.

Architecture

See Storage architecture and Remote publication.

PARQONAUT server (prqnt serve)

PARQONAUT exposes the supported HTTP API only through:

prqnt serve

Defaults

• Bind: 127.0.0.1:8080 (loopback). Use --listen 0.0.0.0:8080 only with a reverse proxy/TLS at the edge.
• State: PRQNT_STATE_DIR or platform data dir (…/prqnt) with SQLite under database/.
• Workers: --workers N in-process job executors (default 2).

Configuration precedence

1. CLI flags (--listen, --database, --state-dir, --workers)
2. Environment (PRQNT_STATE_DIR, PRQNT_BOOTSTRAP_ADMIN_TOKEN, legacy PARACLETE_BOOTSTRAP_TOKEN)
3. Optional --config TOML ([server] / [storage] policy sections)

Bootstrap auth

Set PRQNT_BOOTSTRAP_ADMIN_TOKEN once when starting an empty database; the token is stored hashed. Do not pass secrets on the command line.

OpenAPI

• Live: GET /api/v1/openapi.json
• Offline: prqnt serve --print-openapi

See api.md for endpoint groups and server-storage-policy.md for location rules.

PARQONAUT HTTP API (/api/v1)

PARQONAUT ships the supported API path prefix /api/v1 (distinct from product semver in OpenAPI info.version).

Endpoint groups

Bearer token required on protected routes. OpenAPI title: PARQONAUT API; info.version tracks the workspace crate version.

Direct CLI utilities (inspect, rewrite, convert) intentionally have no HTTP twins in v0.7.x.

Authentication

The HTTP application server uses Bearer token authentication on /api/v1 routes except public health endpoints.

Roles

Token format

New tokens use the prqnt_ prefix. Legacy stored token hashes remain valid when still present in the database.

Bootstrap

On first startup, set PRQNT_BOOTSTRAP_ADMIN_TOKEN to a strong secret to create the initial administrator. The plaintext is never stored or logged. Do not overwrite an existing admin accidentally.

CLI

There is no token on CLI — local prqnt uses your OS user and filesystem permissions. Server policy still restricts which paths and buckets the server may touch.

Durable jobs

prqnt serve persists long-running work in application_jobs (SQLite or Postgres via paraclete-store).

Job kinds (durable async)

States

queued → running → succeeded | failed | canceled

Workers claim jobs with leases and heartbeats. Expired leases are recovered on server startup (requeue or fail with attempt limits).

Cancellation

POST /api/v1/jobs/{job_id}/cancel — queued jobs cancel immediately; running jobs set a cooperative cancel flag.

Synchronous API operations

Diagnose, plan, check, verify, and batch status/check/plan/verify remain synchronous in v0.7 unless deliberately promoted later.

See HTTP API and prqnt serve.

Server storage policy

Network-facing prqnt serve must not accept arbitrary local paths or S3 destinations.

Configure allow lists in TOML:

[server]
allowed_local_roots = ["/data/input", "/data/output"]

[storage]
allowed_local_roots = ["/data/work"]
allowed_s3_buckets = ["my-bucket"]
allowed_s3_prefixes = ["my-bucket/datasets/"]

Requests referencing locations outside policy are rejected before storage I/O. Job JSON must not contain AWS access keys or bearer secrets.

CLI prqnt commands use unrestricted local access by default (interactive/trusted operator model).

CI policy and prqnt check

prqnt check scans and plans without mutating data, then evaluates CI gates.

Example policy

[ci]
fail_on_safe_findings = false
fail_on_review_required = true
fail_on_destructive = true
fail_on_unresolvable_schema = true

Exit codes

prqnt check dataset/ --policy ci-policy.toml
prqnt check dataset/ --policy ci-policy.toml --json

Resume and recovery

When to resume

After interruption (SIGINT, crash) or partial failure, use:

prqnt batch resume --run-dir {output_root}/.parqonaut/runs/{run-id}

Status from a new process reads the same journal:

prqnt batch status --run-dir …

Resume steps

1. Load plan snapshot and journal from run directory
2. Validate plan matches persisted run identity (plan_digest, fingerprints, output root)
3. Validate journal dataset rows match plan (output paths, repair plan IDs)
4. Recover in-flight Running rows → FailedRecoverable with recovery metadata
5. Skip Succeeded datasets (no repair re-execution)
6. Continue pending / recoverable datasets under the jobs bound

Idempotency

When all plan datasets have terminal journal states, resume is a no-op (exit 0, no repair work). A second resume after full completion does not mutate outputs.

Source staleness

Before each dataset executes, the orchestrator recomputes the source fingerprint. If it diverges from the plan-time fingerprint:

• State → StaleSource
• No repair runs for that dataset
• Other datasets continue

Resume/verify also reject plan snapshots that no longer match run identity.

Locks

Each output directory uses {output}.parqonaut.lock recording the owning run_id.

• Different run → LockHeld error
• Same run after crash → stale lock may be reacquired for resume

Cancellation

First SIGINT sets a cooperative cancel flag:

• No new datasets scheduled (re-checked inside scheduler spawn loop)
• In-flight repairs finish at the repair executor boundary
• Journal records cancelled_at
• Exit code 130

Incomplete datasets remain resumable; successful outputs are not rolled back.

Remote publication

Batch and repair workflows can write outputs to s3:// prefixes using the same staged-publication model as local directories: outputs land in a dedicated prefix; manifests and fingerprints record what was written.

Publication semantics and execution manifests are described in Batch execution and ADR-0010 (staged repair publication).

For storage layout and backends, see Storage architecture and Object storage.

Remote recovery

Recovery covers batch resume, server job lease recovery, and S3 dataset staleness handling.

• Batch — Resume and recovery (journal, locks, stale source).
• Server jobs — Durable jobs (lease expiry, requeue, attempt limits).
• Object storage — fingerprints recomputed before mutating work; mismatches fail closed.

HTTP POST /api/v1/batches/{run_id}/resume submits a durable batch resume job with the same application semantics as prqnt batch resume.

Architecture overview

PARQONAUT is a Rust-first toolkit for exploring, diagnosing, repairing, and orchestrating Parquet datasets — locally or on S3-compatible storage.

Application layer (CLI and HTTP)

Scan, repair, and batch orchestration share one application facade:

prqnt CLI ───┐
             ├──► parqonaut-app ──► scan / repair engines, orchestrator, storage
HTTP/workers ┘

parqonaut-app
  ├─ scan, diagnose, plan, check, repair, verify
  └─ batch check, plan, repair, status, resume, verify

Direct CLI utilities (intentionally outside parqonaut-app):
  inspect, rewrite, convert  →  transform / stream crates

prqnt serve runs in-process workers that dispatch durable jobs by kind into the same parqonaut-app entrypoints.

Control plane and storage

paraclete-store  →  SQLite / Postgres (jobs, tokens, runs)
parqonaut-storage →  S3StorageBackend (s3://)
paraclete-service →  HTTP router, auth, metrics, OpenAPI

Workspace map (selected)

Historical crate names (paraclete-*) reflect provenance; see Provenance and ADR index.

Columnar execution (v0.9+)

Product engines share Apache Arrow / Parquet 54.3.1 and parqonaut-columnar batch-stream contracts (BatchSource, BatchSink, bounded BatchStream). Stream conversion, transform, repair data movers, and storage-backed I/O converge on RecordBatch pipelines. See ADR-0015 and Pipelines.

Prior multi-stack layout is documented in ADR-0002 (superseded).

Further reading

• Storage architecture
• Durable jobs
• HTTP API

Shared columnar pipelines (v0.9)

PARQONAUT executes data movement through parqonaut-columnar:

BatchSource → [bounded BatchStream] → transforms → BatchSink

Default channel capacity is 4 batches (DEFAULT_STREAM_CHANNEL_CAPACITY).

Fused transform specs

Multi-step transform YAML compiles into fused segments when operations are streamable. Staging directories under .parqonaut-spec-* appear only when a barrier operation (for example split or rewrite metadata mutation) requires materialization. Dry-run JSON includes pipeline.stages boundaries.

Storage-backed I/O

parqonaut-storage (feature columnar) provides StorageParquetBatchSource / StorageParquetBatchSink using range reads and multipart writes — no whole-object download for metadata.

Demos

just pipeline-demo      # local CSV drift → unify → fused partition
just pipeline-s3-demo   # RustFS: CSV drift → S3 convert → fused rewrite/partition (zero intermediates)

See Architecture and Transform.

Storage architecture

Contract version: STORAGE_CONTRACT_VERSION = 1 in parqonaut-storage.

Boundaries

parqonaut-storage     → DatasetLocation, StorageBackend, publication (later)
parqonaut-repair      → single-dataset mechanics via storage traits
parqonaut-orchestrator → fleet orchestration via storage traits
parqonaut-cli         → command surface only

AWS SDK types must not appear outside parqonaut-storage S3 backend modules.

DatasetLocation

Structured backend identity parsed from:

/path/to/data
./relative
file:///absolute/path
s3://bucket
s3://bucket/
s3://bucket/prefix/

Credentials are never embedded in URIs. Serialization is deterministic JSON with tagged backend field.

ObjectLocation

Single object reference (file://… display form or s3://bucket/key).

ObjectMetadata

Backend-neutral metadata:

• size
• opaque etag (never interpreted as MD5)
• optional version_id
• optional last_modified

StorageBackend

Async trait providing:

StorageCapabilities

Explicit feature flags (range_reads, multipart_upload, conditional_create, etc.). Backends must not silently downgrade safety.

Streaming model

• Single abstraction: ObjectReadStream / ObjectWriteStream (Tokio AsyncRead/AsyncWrite based)
• Backpressure via async read/write; at most one in-flight buffer per stream
• Large objects must use streams or ranged reads, not unbounded Vec<u8>

Errors and retry

StorageError classifies:

NotFound, PermissionDenied, Authentication, Conflict, PreconditionFailed,
Transient, Unavailable, InvalidLocation, UnsupportedCapability, Io, Other

Each error exposes retry_class() → NeverRetry | Retryable.

Conditional operations

Publication and locking depend on explicit conditional create/replace — not generic overwrite flags.

Credential boundary

• Credentials load via AWS provider chain (S3 backend only)
• Never serialized in plans, journals, manifests, or logs
• redact module strips secrets from URIs and debug output

Contract tests

contract::storage_backend_contract runs against each backend implementation (MemoryStorageBackend in crate tests; Local and S3 backends add their own).

Repair plan contract

Repair plans are durable, versioned JSON suitable for CI, code review, and replay.

Version

Every plan includes "schema_version": 1. Unsupported versions are rejected at load time.

Stable fields

Volatile fields (generated_at, finding UUIDs) are stripped from canonical serialization.

Commands

prqnt plan dataset/ --policy policy.toml --output plan.json
prqnt plan dataset/ --canonical
prqnt plan diff plan-a.json plan-b.json
prqnt repair dataset/ --plan plan.json --authorize R004 --output out/

Golden fixtures under fixtures/plans/ guard against accidental contract changes.

Architecture Decision Records (ADR)

ADRs capture durable design decisions for PARQONAUT. They are repository history — not day-to-day user guides.

PARQONAUT Provenance

PARQONAUT consolidates three predecessor projects owned by sempervent. The original repositories are not archived or modified by this migration.

Migration Date

2026-09-17 (baseline tagged v0.1.0 on 2026-09-18)

Originating Repositories

Source Commit SHAs

Imported Components

From Paraclete (architectural chassis)

Not imported in Phase 1: paraclete-cli (HTTP client superseded by unified CLI), paraclete-tui, mkdocs site artifacts.

From parqknife (transformation engine)

Arrow 54 API fixes applied during import. The upstream LICENSE file contained AGPL v3 text and was not copied; see licensing section below.

From streaming-parquet / maw (streaming engine)

Refactored from binary-only to library + CLI wrapper. Parquet writer completed for Phase 1.

Licensing in this repository

parqknife licensing note

At import time, parqknife’s Cargo.toml declared MIT OR Apache-2.0, but its repository LICENSE file contained GNU Affero GPL v3 text. PARQONAUT:

1. Retains license = "MIT OR Apache-2.0" on crates/parqonaut-transform/Cargo.toml (matching upstream crate metadata).
2. Does not include the upstream AGPL license file.
3. Provides LICENSE-APACHE-2.0 for recipients who elect the Apache-2.0 option.

This documents what was observed upstream; it is not a legal determination of applicable terms. Consult THIRD_PARTY_LICENSES.md for full detail.

Git History

Import method: structured copy with SHA documentation (not a history-preserving merge). Useful history remains in the upstream repositories listed above.

OpenAPI

The authoritative API description is generated at runtime:

prqnt serve --print-openapi

The repository golden file fixtures/api/openapi-v1.json is checked in CI (cargo xtask docs openapi --check).

Download from the documentation site

When built for GitHub Pages, the same JSON is published at:

/openapi/openapi.json

(relative to the site root, e.g. https://sempervent.github.io/PARQONAUT/openapi/openapi.json).

Versioning

• Product version (e.g. 0.7.1) — PARQONAUT release semver in info.version.
• API path version — /api/v1 prefix; not incremented with every product patch.

Authentication

All mutating and most read routes require Authorization: Bearer <token>. See Authentication and HTTP API.

CLI reference (generated)

This page is generated from the live prqnt command tree. Do not edit by hand — run cargo xtask docs cli.

Global

PARQONAUT — Parquet Analysis, Rewriting, Quality, Orchestration, Navigation, Auditing, Unification & Transformation

Usage: prqnt [OPTIONS] <COMMAND>

Commands:
  scan       Forensic scan of local paths or s3:// dataset prefixes
  inspect    Inspect Parquet schema and row-group metadata
  partition  Hive-style partition of a Parquet file
  merge      Merge Parquet files or a dataset directory
  split      Split a Parquet file into smaller parts
  transform  Declarative multi-step transform workflow
  rewrite    Rewrite or recompress a Parquet file
  diagnose   Diagnose dataset repair opportunities from scan evidence
  plan       Generate an evidence-bound repair plan
  plan-diff  Diff two repair plans
  check      CI-oriented policy compliance check (non-mutating)
  repair     Execute a repair plan to a separate output directory or s3:// prefix
  verify     Verify repaired dataset against a before scan
  doctor     Integrated scan → diagnose → plan (optional safe repair)
  batch      Multi-dataset batch orchestration
  serve      Run the PARQONAUT HTTP application server (`/api/v1`)
  convert    Stream-convert CSV/Parquet inputs
  help       Print this message or the help of the given subcommand(s)

Options:
      --json     Emit JSON where supported
  -h, --help     Print help
  -V, --version  Print version

scan

Forensic scan of local paths or s3:// dataset prefixes

Usage: prqnt scan [OPTIONS] <PATH>

Arguments:
  <PATH>  Local path or s3:// URI to scan

Options:
      --json               Emit JSON where supported
  -p, --profile <PROFILE>  [default: standard]
  -h, --help               Print help

inspect

Inspect Parquet schema and row-group metadata

Usage: prqnt inspect [OPTIONS] <INPUT>

Arguments:
  <INPUT>  

Options:
      --json   Emit JSON where supported
      --stats  Show detailed column statistics
  -h, --help   Print help

rewrite

Rewrite or recompress a Parquet file

Usage: prqnt rewrite [OPTIONS] <INPUT> <OUTPUT>

Arguments:
  <INPUT>   
  <OUTPUT>  

Options:
      --compression <COMPRESSION>              Compression codec (snappy, gzip, zstd, uncompressed)
      --json                                   Emit JSON where supported
      --row-group-size-mb <ROW_GROUP_SIZE_MB>  
      --projection <PROJECTION>                
      --filter <FILTER>                        
      --rebuild-stats                          
  -h, --help                                   Print help

convert

Stream-convert CSV/Parquet inputs

Usage: prqnt convert [OPTIONS] --out <OUT> <INPUTS>...

Arguments:
  <INPUTS>...  Input file(s), directories, or globs

Options:
      --json
          Emit JSON where supported
  -o, --out <OUT>
          
      --out-format <OUT_FORMAT>
          [possible values: csv, parquet]
      --compression <COMPRESSION>
          [default: none] [possible values: none, snappy, gzip, zstd]
      --zstd-level <ZSTD_LEVEL>
          [default: 3]
      --plan
          
      --dry-run
          
      --state <STATE>
          
      --resume
          
      --schema-conflicts <SCHEMA_CONFLICTS>
          [default: strict]
      --json-progress
          
  -h, --help
          Print help

diagnose

Diagnose dataset repair opportunities from scan evidence

Usage: prqnt diagnose [OPTIONS] <PATH>

Arguments:
  <PATH>  Local path or s3:// dataset URI

Options:
      --json  Emit JSON where supported
  -h, --help  Print help

plan

Generate an evidence-bound repair plan

Usage: prqnt plan [OPTIONS] <PATH>

Arguments:
  <PATH>  Local path or s3:// dataset URI

Options:
      --json                           Emit JSON where supported
      --output <OUTPUT>                
      --policy <POLICY>                TOML policy file path
      --target-schema <TARGET_SCHEMA>  Explicit target schema JSON
      --canonical                      Emit canonical deterministic plan JSON
  -h, --help                           Print help

plan-diff

Diff two repair plans

Usage: prqnt plan-diff [OPTIONS] <LEFT> <RIGHT>

Arguments:
  <LEFT>   
  <RIGHT>  

Options:
      --json  Emit JSON where supported
  -h, --help  Print help

check

CI-oriented policy compliance check (non-mutating)

Usage: prqnt check [OPTIONS] <PATH>

Arguments:
  <PATH>  Local path or s3:// dataset URI

Options:
      --json             Emit JSON where supported
      --policy <POLICY>  TOML CI policy file
  -h, --help             Print help

repair

Execute a repair plan to a separate output directory or s3:// prefix

Usage: prqnt repair [OPTIONS] --plan <PLAN> --output <OUTPUT> <PATH>

Arguments:
  <PATH>  Source dataset local path or s3:// URI

Options:
      --json                   Emit JSON where supported
      --plan <PLAN>            
      --output <OUTPUT>        Output local directory or s3:// dataset prefix
      --authorize <AUTHORIZE>  Explicitly authorize a ReviewRequired operation ID
  -h, --help                   Print help

verify

Verify repaired dataset against a before scan

Usage: prqnt verify [OPTIONS] <BEFORE> <AFTER>

Arguments:
  <BEFORE>  Before dataset local path or s3:// URI
  <AFTER>   

Options:
      --json                 Emit JSON where supported
      --manifest <MANIFEST>  Optional execution manifest path
  -h, --help                 Print help

doctor

Integrated scan → diagnose → plan (optional safe repair)

Usage: prqnt doctor [OPTIONS] <PATH>

Arguments:
  <PATH>  Local path or s3:// dataset URI

Options:
      --json                   Emit JSON where supported
      --policy <POLICY>        TOML policy file path (same as plan)
      --repair                 Execute authorized repairs after showing the plan
      --output <OUTPUT>        Output local directory or s3:// dataset prefix
      --authorize <AUTHORIZE>  
  -h, --help                   Print help

batch

Multi-dataset batch orchestration

Usage: prqnt batch [OPTIONS] <COMMAND>

Commands:
  check   Validate batch configuration without executing
  plan    Produce a durable batch plan
  repair  Execute an approved batch plan
  status  Read run journal status
  resume  Resume an interrupted batch run
  verify  Verify batch outputs against the plan
  help    Print this message or the help of the given subcommand(s)

Options:
      --json  Emit JSON where supported
  -h, --help  Print help

batch check

Validate batch configuration without executing

Usage: prqnt batch check [OPTIONS] --config <CONFIG>

Options:
      --config <CONFIG>  
      --json             Emit JSON where supported
  -h, --help             Print help

batch plan

Produce a durable batch plan

Usage: prqnt batch plan [OPTIONS] --config <CONFIG>

Options:
      --config <CONFIG>  
      --json             Emit JSON where supported
      --output <OUTPUT>  
  -h, --help             Print help

batch repair

Execute an approved batch plan

Usage: prqnt batch repair [OPTIONS] --plan <PLAN>

Options:
      --json         Emit JSON where supported
      --plan <PLAN>  
      --jobs <JOBS>  
      --dry-run      
  -h, --help         Print help

batch status

Read run journal status

Usage: prqnt batch status [OPTIONS] --run-dir <RUN_DIR>

Options:
      --json               Emit JSON where supported
      --run-dir <RUN_DIR>  
  -h, --help               Print help

batch resume

Resume an interrupted batch run

Usage: prqnt batch resume [OPTIONS] --run-dir <RUN_DIR>

Options:
      --json               Emit JSON where supported
      --run-dir <RUN_DIR>  
      --jobs <JOBS>        
      --dry-run            
  -h, --help               Print help

batch verify

Verify batch outputs against the plan

Usage: prqnt batch verify [OPTIONS] --run-dir <RUN_DIR>

Options:
      --json               Emit JSON where supported
      --run-dir <RUN_DIR>  
  -h, --help               Print help

serve

Run the PARQONAUT HTTP application server (`/api/v1`)

Usage: prqnt serve [OPTIONS]

Options:
      --json                   Emit JSON where supported
      --listen <LISTEN>        Listen address (default loopback-only) [default: 127.0.0.1:8080]
      --database <DATABASE>    SQLite or Postgres database URL
      --state-dir <STATE_DIR>  Application state directory (database + artifacts)
      --workers <WORKERS>      In-process async job workers [default: 2]
      --config <CONFIG>        Optional TOML config (server + storage policy)
      --print-openapi          Emit OpenAPI JSON to stdout and exit
  -h, --help                   Print help

Rust API (rustdoc)

Library crates in this workspace are documented with rustdoc. The built index is published at:

/rustdoc/ on the deployed site (entry redirects to prqnt crate docs; generated during just docs-build)

End users normally interact through prqnt or /api/v1. Rust API docs are for contributors and crate integrators.

Rebuild locally as part of just docs-build.

Changelog

Changelog

All notable changes to PARQONAUT are documented here.

0.9.2 - 2026-09-21

S3 multipart correctness at the default 5 MiB part boundary.

Fixed

• MultipartAsyncWrite: do not re-buffer bytes when poll_write returns Poll::Pending during part flush; split full parts at the configured threshold and verify byte accounting on shutdown.

0.9.1 - 2026-09-20

Remote pipeline completeness: bounded streaming S3 publication and full local/S3 routing for convert, transform, and fused specs.

Added

• Authoritative 6×4 remote pipeline matrix tests (remote_pipeline_matrix) on RustFS
• S3 publication regression tests (roundtrip, visibility, failure/retry, conditional create conflict)
• just pipeline-s3-demo — CSV drift → S3 convert → fused rewrite/partition with zero intermediate objects

Fixed

• S3 multipart finalize on stream shutdown (committed objects visible after finish)
• prqnt convert remote Parquet sinks stream via write_parquet_batch_stream (no full-object buffer)

Changed

• s3-integration CI job runs storage roundtrip, remote convert, and transform matrix tests

0.8.1 - 2026-09-20

Patch release correcting transform demo recipes; no intended runtime behavior change from v0.8.0.

Fixed

• just transform-demo aligned with prqnt flags (--output, --by, Parquet glob for merge)
• S3 batch verify sees Parquet under committed publication prefixes; s3-resume demo uses non-overlapping sources

0.8.0 - 2026-09-20

Transform and streaming workflow release.

Added

• prqnt partition, prqnt merge, prqnt split, and prqnt transform --spec (YAML pipelines)
• Typed spec compilation (Spec → ExecutablePlan → execution → TransformReport)
• Schema discovery, unification, and batch alignment in prqnt convert with conflict policies
• Resumable streaming conversion with checkpoint identity, stale detection, and staged output
• parqonaut-workflow progress and plan contracts; progress observer implementations
• Transform/stream fixtures, demos (just transform-demo, stream-demo, stream-resume-demo)
• Public docs: transform, streaming, roadmap

Changed

• Workspace version 0.8.0; golden repair plans regenerated for version field

Note

Apache Arrow / Parquet stack convergence remains planned for v0.9.0 (see docs/development/columnar-convergence-plan.md).

0.7.1 - 2026-09-19

Documentation and delivery infrastructure release. No intended runtime behavior changes.

Added

• GitHub Pages documentation site (mdBook)
• Documentation CI workflow and just docs-build / just docs-check
• Generated CLI reference (cargo xtask docs cli)
• OpenAPI and Rustdoc publishing as part of the site build
• Internal Markdown link checker

Changed

• Reorganized public documentation navigation (docs/SUMMARY.md)
• README links to the canonical documentation site
• Release notes are committed before tagging (v0.7.1 discipline)

0.7.0 - 2026-09-19

Unified application server: CLI and HTTP share parqonaut-app; durable jobs for scan, repair, and batch orchestration.

ADDED

• prqnt serve — PARQONAUT HTTP API at /api/v1 (OpenAPI, metrics, health live/ready).
• parqonaut-app — transport-neutral scan, repair, and batch use cases.
• Durable application_jobs model (scan, repair, batch_repair, batch_resume) with lease/recovery and cancellation.
• Server storage policy (allowed local roots, S3 buckets/prefixes); PRQNT_BOOTSTRAP_ADMIN_TOKEN.
• API integration scripts (scripts/api-test/), just api-test, api-demo, api-restart-demo.
• CI jobs api-integration and postgres-integration.

CHANGED

• CLI repair/batch commands delegate to parqonaut-app (scan already did).
• OpenAPI title PARQONAUT API; metrics namespace parqonaut_; new bearer tokens use prqnt_ prefix.
• SQLite scan_jobs migrated to application_jobs (data preserved).

SECURITY

• Default bind 127.0.0.1; location policy enforced before storage I/O.

DEFERRED

• Idempotency-Key, distributed workers, SDKs, web UI/TUI, TLS termination, plugin execution.

0.6.0 - 2026-09-19

Interface normalization: one public executable (prqnt), capability-oriented repository layout, and removal of legacy product entrypoints.

BREAKING

• The parqonaut executable is renamed to prqnt. Update scripts, CI, and documentation accordingly. There is no compatibility alias.

REMOVED

• paraclete-http legacy HTTP binary (service crate remains library-only; prqnt serve deferred).
• Product-scoped fixture generator binaries (generate-phase2-fixtures, generate-phase3-fixtures, generate-phase4-fixtures, generate-phase5-fogbank-fixtures, generate-golden-plans).

CHANGED

• Developer automation consolidated under cargo xtask (fixtures, golden plans).
• Active tests, fixtures, scripts, Just targets, and CI jobs use capability names (scan, repair, schema, orchestration, object-storage, s3-integration) instead of numbered implementation phases.
• paraclete-core scan modules renamed: scan_rules, scan_findings, schema_findings; evaluate_scan_rules replaces evaluate_phase1_rules.
• CLI help describes PARQONAUT capabilities without predecessor engine branding.

ADDED

• scripts/check-active-naming.sh and just naming-check (also wired into just ci and GitHub Actions).

0.4.1 - 2026-09-18

Release metadata correction only — no functional changes from v0.4.0.

Fixed

• Workspace crate version aligned with the published v0.4.0 release tag (0.3.0 → 0.4.1)
• CHANGELOG.md updated with missing release notes for v0.2.0–v0.4.0
• README.md updated to describe current Phase 4 capabilities and batch workflow

0.4.0 - 2026-09-18

Multi-dataset batch orchestration with bounded concurrency, durable journal, resume, and verification.

Added

• parqonaut-orchestrator crate — batch plan contracts, scheduler, SQLite run journal, locking, resume
• prqnt batch check|plan|repair|status|resume|verify CLI commands with JSON output
• SHIPWRECK Phase 4 fixture fleet and integration tests
• Plan-bound authorization, output path mapping, and overlap detection for batch runs
• Cooperative SIGINT cancellation and idempotent resume semantics
• ADRs 0011–0014 and Phase 4 documentation
• data/dummy.parquet sample dataset for scan/inspect tests

0.3.0 - 2026-09-18

Policy-governed schema reconciliation and explicit authorization for review-required repairs.

Added

• Deterministic schema compatibility model and lossless schema widening
• Explicit --authorize for ReviewRequired repair operations
• prqnt doctor --policy, prqnt check, prqnt plan diff
• Durable repair plan contract v1, execution manifests, staged publication
• FRANKENLAKE v2 laboratory fixtures and Phase 3 demo
• ADRs 0007–0010 and Phase 3 documentation

0.2.0 - 2026-09-18

Evidence-driven diagnose, plan, repair, and verify for single datasets.

Added

• parqonaut-repair crate — scan → diagnose → plan → repair → verify pipeline
• Dataset fingerprints and stale-plan rejection
• Repair safety classification (Safe, ReviewRequired, Blocked)
• prqnt doctor, prqnt plan, prqnt repair, prqnt verify
• FRANKENLAKE Phase 2 fixtures and golden repair plans
• ADRs 0004–0006 and Phase 2 documentation

0.1.0 - 2026-09-18

Initial unified release consolidating Paraclete, parqknife, and streaming-parquet (maw).

Added

• Cargo workspace with Paraclete-derived scan chassis (paraclete-* crates)
• parqonaut-transform — parqknife-derived inspect/rewrite/filter pipeline (Apache Arrow 54)
• parqonaut-stream — maw-derived streaming CSV/Parquet conversion (arrow2)
• Unified prqnt CLI with scan, inspect, rewrite, and convert commands
• End-to-end demo: CSV → Parquet → scan → rewrite → rescan (just demo)
• Integration tests covering all three engine families
• Fixtures, Python plugin contracts, and migration documentation
• GitHub Actions CI (fmt, clippy, test)
• Provenance and architecture ADRs

Pre-1.0 roadmap (summary)

Detailed planning lives in the repository at docs/development/pre-1.0-roadmap.md (not part of the public site nav).

No release dates are committed here.

Licensing

PARQONAUT is licensed under the MIT License. See the LICENSE file in the repository root.

Contributing

Toolchain

• Rust stable with rustfmt and clippy (just setup)
• Optional: just, Docker (RustFS S3 tests), Postgres (store integration)

Common commands

cargo fmt --all
cargo clippy --workspace --all-targets --all-features -- -D warnings
cargo test --workspace
just ci
just naming-check

Fixtures and goldens:

cargo xtask fixtures repair
cargo xtask golden-plans

Documentation

just docs-check   # verify generated refs + links + mdBook
just docs-build   # full site under target/pages/
cargo xtask docs cli
cargo xtask docs openapi

Pinned mdBook version: see scripts/docs/env.sh (MDBOOK_VERSION).

Integration tests

Naming

Active user-facing docs and code must not reintroduce numbered “phase” product naming. Run just naming-check.

Pull requests

• Keep runtime changes out of documentation-only PRs unless fixing doc-related tests or generators.
• Ensure CI passes: rust, api-integration, postgres-integration, s3-integration, documentation.

Release flow: merge to main, verify GitHub Pages deploy, tag with release notes in the tagged commit (see v0.7.1 discipline).