Skip to content

Runbook — backup validation and recovery drill

Parent package: Backup and disaster recovery package — smart farm stack. Decision context: Disaster recovery decision rules — farm edge stack.

Principle: A backup that has never been restored is a hypothesis. Prefer restore-tested evidence over job-success notifications alone.

Preconditions

Drill A — restic repository health (no full restore)

Mandatory for any restic-based path.

  1. Run restic check (or documented maintenance) against the repository used for DB dumps or file backups—see restic backing up capture.
  2. Confirm the latest snapshot has non-zero content for at least one critical path (restic ls spot-check).
  3. Document who may run forget / prune and when (avoid locking the repo during peak farm loads on constrained power).

Pass: Check completes; snapshot list matches policy.

Drill B — logical database restore (farmOS / PostgreSQL)

Mandatory when farm records are the system of record.

  1. Provision a throwaway database (new instance or new database name).
  2. Restore the latest pg_dump (or supported export) into that DB—match PostgreSQL and farmOS version expectations per PostgreSQL backup docs and farmOS hosting.
  3. Point a staging farmOS install at the restored DB and verify known records (assets, geometry spot-checks)—not production DNS until cutover is deliberate.

Pass: App boots; data age matches your stated RPO from Restore and recovery tiers.

Drill C — Longhorn volume restore (subset)

When Longhorn backs up PVCs you rely on.

  1. Pick one non-production namespace or test PVC.
  2. Execute restore per Longhorn restore capture and current Longhorn docs.
  3. Verify mount and filesystem integrity (checksum or workload smoke test).

Pass: Volume is attachable and consistent for that workload class.

Drill D — k3s etcd snapshot restore (staging only)

Dangerous on production without a maintenance window—use a lab cluster first.

  1. Follow k3s backup and restore and etcd-snapshot CLI for your install mode.
  2. Prove you can list and restore from your real snapshot store (local path and S3-compatible if configured).

Pass: Documented steps work end-to-end in staging once.

Drill E — Rancher (if installed)

  1. Per Rancher backup/restore usage, confirm backup artifacts exist and you retain the encryption configuration if used (the operator does not back that file up for you).
  2. Tabletop or staging restore per docs—avoid version skew between backup source and restore target.

Drill F — secrets and configuration

  1. List what is not in plain Git: Sealed Secrets, SOPS, cloud KMS, manual kubeconfig distribution.
  2. Verify one rotation or recovery path (dry-run acceptable) and where encrypted break-glass copies live.

Reporting

After drills, update Restore and recovery tiers — homelab farm systems with measured elapsed times (RPO/RTO worksheet) and append wiki/log.md if policy changed.