Skip to content

Runbook — broker or backhaul down

Prerequisites: Field telemetry reference architecture — homestead + 120-acre farm; Automation degraded modes and manual fallback SOP. Without an agreed architecture, this runbook is theater.

Purpose

When MQTT broker or IP backhaul to SITE_FARM fails, stop assuming remote state matches reality; verify physical and stabilize animals, water, and gates.

Detection

  • HA / dashboard: entities unavailable or stale beyond STALE_THRESHOLD (define in architecture).
  • Broker: no subscribers receiving; ping or mgmt port check from authorized host only.
  • Backhaul: traceroute / LTE modem stats if documented.

Immediate actions (order may vary)

  1. Classify: broker-only vs internet vs power at gateway (see Runbook — power loss at remote site).
  2. Disable dangerous automation: rules that open water or unlock on stale “OK” signals—per degraded SOP.
  3. Notify: use out-of-band comms (cell call/SMS tree) per weekly coverage matrix.
  4. Log: timestamp, what was disabled, who on-site—correlate after recovery.

Recovery

  • Broker: failback order—persistent sessions, retained message policy per architecture.
  • Backhaul: failover SIM/path if designed; else schedule visit.

Replace placeholders with hostname, ports, and contact tree internal to your opsec.