fix(deploymentmanager): bulk deploy skips Disabled + reconcile circuit-breaker + probe captures pre-flatten versions (arch-review 2026-08-19 P7-2 + P7-3 + P7-4)

2026-08-19 14:52:00 -04:00
parent b68f553f1e
commit 5cbc143b01
10 changed files with 541 additions and 32 deletions
@@ -169,6 +169,26 @@ one is parallel:
 3. **Finalize (serial).** Commit terminal statuses, apply post-success side
   effects, write audit rows, release each operation lock.

+**Disabled instances are skipped.** A bulk site deploy never touches an instance
+in `Disabled` — no flatten, no send, no state change — because deploy is a legal
+transition out of `Disabled` and forces `Enabled` on success, so an unfiltered
+site-wide push would silently restart every piece of equipment an operator had
+deliberately stopped; they are reported in the summary's `SkippedInstances` with
+the reason "Disabled — bulk deploy preserves disabled state; deploy individually
+to re-enable", and the single-instance deploy remains the explicit re-enable path.
+
+**Reconciliation is bounded per batch.** Phase 1 is serial, so an unreachable site
+would otherwise cost the batch one full communication-layer `QueryTimeout` (30 s)
+per instance before a single deploy was sent — hours for a large site, past the
+CLI's 5-minute bulk deadline, which cancels and re-arms the identical cost on the
+next attempt. Each reconcile in a bulk deploy therefore runs under the short
+`SiteDeploymentReconcileTimeout` (default **5 s**), and the FIRST one to hit that
+deadline opens a once-per-batch circuit-breaker that skips the remaining
+reconciles (one summary log line, not N). Skipping costs nothing: a timed-out
+reconcile already falls through to a normal deploy on central's recorded state,
+with the site's own stale-revision rejection as the safety net. The
+single-instance path is unbounded by this option and keeps the full `QueryTimeout`.
+
 **Not all-or-nothing.** An instance that fails for any reason (wrong state, failed
 validation, lock already held by another operation, site round-trip timed out) is
 reported as a failed row while the rest proceed, and is individually retryable via