From e4397aa84eda5ffdebd5d6e00bc11545e890dc0d Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Tue, 21 Jul 2026 03:02:02 -0400 Subject: [PATCH] docs(485): live-gate both #485 fixes on docker-dev MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit All three guards observed firing in production, on two nodes, with the live values proving the subscription half rather than just the log line: the Modbus tag advanced 56278 -> 56336 -> 56354 straight across the induction. The RED control was not staged. site-a-1 had hit #485 during the PREVIOUS gate's SQL flapping and had been serving an empty address space for 34 minutes (PureRemove, removed=17) on the pre-fix image — the bug in its natural habitat. Rebuilding on the fixed image restored it, and the induction then failed to reproduce the teardown. The original trigger was a race, so the gate drives the empty-bytes branch instead — byte-for-byte what the code sees when the row cannot be read, and the branch the driver-side guards key on (their throw paths already returned early before this work). Pausing the target node opens the window deterministically; both runs measured <1s, well inside acceptable-heartbeat-pause, so no failover was triggered. Of the six nodes exactly one took the guard path — the other five read the real artifact and applied it, which is its own control. Records one pre-existing defect this work does NOT fix: ApplyAndAck advances _currentRevision and writes NodeDeploymentState=Applied before knowing whether anything applied, so a node that skipped reports success for a revision it never applied — and since dispatch short-circuits on a revision match it cannot self-heal from a re-dispatch. Data plane is unaffected (the node keeps serving its last good config); it is a reporting/convergence bug. Left out of these fixes because correcting it changes deploy ACK semantics fleet-wide. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW --- docs/plans/2026-07-21-issue-485-live-gate.md | 63 ++++++++++++++++++++ 1 file changed, 63 insertions(+) create mode 100644 docs/plans/2026-07-21-issue-485-live-gate.md diff --git a/docs/plans/2026-07-21-issue-485-live-gate.md b/docs/plans/2026-07-21-issue-485-live-gate.md new file mode 100644 index 00000000..34d98729 --- /dev/null +++ b/docs/plans/2026-07-21-issue-485-live-gate.md @@ -0,0 +1,63 @@ +# Issue #485 — live gate (docker-dev rig) + +> Verifies the two fixes for #485 — `OpcUaPublishActor` (master `c7f5b9cf`) and `DriverHostActor` +> (master `5aad1e33`) — on the local `docker-dev` rig: 6 nodes, shared SQL, real Modbus fixture on +> `10.100.0.35:5020`. +> +> **Safety rules followed**: all DB inspection + mutation via `sqlcmd` inside the SQL container; the +> rig's Modbus fixture is read-only for this gate (no writes to the PLC sim). + +## Result + +**Both fixes verified, all three guards observed firing in production, with a live RED control that +was not staged.** The gate also confirmed a **pre-existing** defect the fixes do not address — the node +records `Applied` for a deployment it skipped (see "Finding" below). + +| # | Check | Result | Evidence | +|---|---|---|---| +| 1 | RED control — the bug in its natural habitat, unforced | ✅ | site-a-1 hit #485 during the *previous* gate's SQL flapping and had been serving an **empty address space for 34 minutes**: `failed to load artifact … rebuild becomes no-op` followed immediately by `applied plan (kind=PureRemove, added=0, removed=17, rebuild=True)`. Its 17 `ns=2` nodes were gone; a browse returned only the root `OtOpcUa` folder. Nothing was staged to produce this — it was already true on the pre-fix image. | +| 2 | Rebuilt on the fixed image, the node recovers | ✅ | After `docker compose build` + `--force-recreate`, site-a-1 restored to `raw subtree materialised (containers=16, tags=1)` — 18 `ns=2` nodes. | +| 3 | Address-space guard fires; nodes survive | ✅ | site-a-1, empty artifact: `OpcUaPublish: no usable artifact for deployment f4b9dd63… — KEEPING the currently served address space rather than tearing it down`. **No `PureRemove`.** Browse before vs after: 18 vs 18, `diff` **identical**. | +| 4 | Reconcile guard fires; drivers survive | ✅ | `artifact for f4b9dd63… read back empty; skipping reconcile and keeping the 1 running driver(s)`, and the apply still reported `children=1`. Repeated on central-2 with a bigger blast radius: `keeping the 5 running driver(s)`, `children=5`. | +| 5 | SubscribeBulk guard fires; **live values keep flowing** | ✅ | central-2, `artifact for 8c36a673… read back empty; skipping SubscribeBulk and keeping the live subscriptions`. The decisive proof is the value, not the log: `ns=2;s=pymodbus/plc/HR200X` read **56278 @ 06:59:51** before the induction, then **56336 @ 07:00:21** and **56354 @ 07:00:30** after it (induction at 07:00:07). The subscription was still delivering fresh Modbus data across the event. | +| 6 | Positive control — a readable artifact still applies | ✅ | Immediately after the guards fired, a genuine deploy applied normally on both nodes: site-a-1 `AddRemoveMix, added=1, removed=1` + `SubscribeBulk pushed 1 references`; central-2 `AddRemoveMix, added=1, removed=1` + `SubscribeBulk pushed 5 references across 4 driver(s)`. The guards suppress the teardown, **not** deployment. | +| 7 | Control — only the node that saw empty bytes took the guard path | ✅ | Of the six nodes, exactly **one** logged `read back empty` (the paused one); the other five read the real artifact before it was blanked and applied it normally. The induction window is what produced the condition, not a global change. | + +## How the failure was induced deterministically + +The original incident was a race (a transient ConfigDb blip landing between the apply and the artifact +read). Racing it again would be unreliable, so the gate drives the **empty-bytes** branch instead — which +is byte-for-byte what the code sees when the row cannot be read, and is the branch the driver-side guards +key on (their *throw* paths already returned early before this work): + +1. `docker pause` the target node, freezing it mid-cluster. +2. `POST /api/deployments` — the deployment seals and dispatches while the node cannot process it. +3. `UPDATE Deployment SET ArtifactBlob = 0x WHERE DeploymentId = `. +4. `docker unpause` — the node now processes the dispatch and reads an empty artifact. + +The pause measured **<1s** on both runs, well inside `acceptable-heartbeat-pause = 10s` and SBR's +`stable-after = 15s`, so the node was never flagged unreachable and no failover was triggered. + +## Finding — the node records `Applied` for a deployment it skipped (pre-existing, NOT fixed here) + +Both guards leave the node running its previous configuration, which is the intent. But `ApplyAndAck` +advances `_currentRevision = revision` and writes `NodeDeploymentState = Applied` **before** it knows +whether anything was applied — that assignment sits immediately after `ReconcileDrivers`, which returns +`null` on both the (pre-existing) throw path and the new empty path: + +``` +[07:00:07 WRN] DriverHost central-2:4053: artifact … read back empty; skipping reconcile … +[07:00:07 INF] DriverHost central-2:4053: applied deployment 8c36a673… (rev 03b33904…, children=5) +``` + +`NodeDeploymentState` for that deployment reads `Status = Applied` for central-2. The node now claims a +revision whose configuration it never applied — and because `HandleDispatchFromSteady` short-circuits on +a revision match (`dispatch matches current rev; immediate ACK`), **re-dispatching that same revision is +a no-op**, so the node cannot self-heal. Only a later deployment with a *different* revision recovers it +(check 6 confirms that path works). + +This is pre-existing — the throw path has always done it — and is strictly a reporting/convergence bug, +not a data-plane one: the node keeps serving its last good configuration throughout. The honest fix is to +ACK `Failed` and leave `_currentRevision` alone when the artifact could not be read, so the normal deploy +retry heals it. Tracked separately rather than folded in here, because it changes deploy ACK semantics +fleet-wide.