dohertj2/lmxopcua
1
0
Fork 0
Code Issues 8 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(485): live-gate both #485 fixes on docker-dev

Browse Source 
All three guards observed firing in production, on two nodes, with the live
values proving the subscription half rather than just the log line: the Modbus
tag advanced 56278 -> 56336 -> 56354 straight across the induction.

The RED control was not staged. site-a-1 had hit #485 during the PREVIOUS
gate's SQL flapping and had been serving an empty address space for 34 minutes
(PureRemove, removed=17) on the pre-fix image — the bug in its natural habitat.
Rebuilding on the fixed image restored it, and the induction then failed to
reproduce the teardown.

The original trigger was a race, so the gate drives the empty-bytes branch
instead — byte-for-byte what the code sees when the row cannot be read, and the
branch the driver-side guards key on (their throw paths already returned early
before this work). Pausing the target node opens the window deterministically;
both runs measured <1s, well inside acceptable-heartbeat-pause, so no failover
was triggered. Of the six nodes exactly one took the guard path — the other
five read the real artifact and applied it, which is its own control.

Records one pre-existing defect this work does NOT fix: ApplyAndAck advances
_currentRevision and writes NodeDeploymentState=Applied before knowing whether
anything applied, so a node that skipped reports success for a revision it
never applied — and since dispatch short-circuits on a revision match it cannot
self-heal from a re-dispatch. Data plane is unaffected (the node keeps serving
its last good config); it is a reporting/convergence bug. Left out of these
fixes because correcting it changes deploy ACK semantics fleet-wide.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
2026-07-21 03:02:02 -04:00
parent 5aad1e33de
commit e4397aa84e
1 changed files with 63 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/plans/2026-07-21-issue-485-live-gate.md
+63
View File
@@ -0,0 +1,63 @@
# Issue #485 — live gate (docker-dev rig)
> Verifies the two fixes for #485 — `OpcUaPublishActor` (master `c7f5b9cf`) and `DriverHostActor`
> (master `5aad1e33`) — on the local `docker-dev` rig: 6 nodes, shared SQL, real Modbus fixture on
> `10.100.0.35:5020`.
>
> **Safety rules followed**: all DB inspection + mutation via `sqlcmd` inside the SQL container; the
> rig's Modbus fixture is read-only for this gate (no writes to the PLC sim).
## Result
**Both fixes verified, all three guards observed firing in production, with a live RED control that
was not staged.** The gate also confirmed a **pre-existing** defect the fixes do not address — the node
records `Applied` for a deployment it skipped (see "Finding" below).
| # | Check | Result | Evidence |
|---|---|---|---|
| 1 | RED control — the bug in its natural habitat, unforced | ✅ | site-a-1 hit #485 during the *previous* gate's SQL flapping and had been serving an **empty address space for 34 minutes**: `failed to load artifact … rebuild becomes no-op` followed immediately by `applied plan (kind=PureRemove, added=0, removed=17, rebuild=True)`. Its 17 `ns=2` nodes were gone; a browse returned only the root `OtOpcUa` folder. Nothing was staged to produce this — it was already true on the pre-fix image. |
| 2 | Rebuilt on the fixed image, the node recovers | ✅ | After `docker compose build` + `--force-recreate`, site-a-1 restored to `raw subtree materialised (containers=16, tags=1)` — 18 `ns=2` nodes. |
| 3 | Address-space guard fires; nodes survive | ✅ | site-a-1, empty artifact: `OpcUaPublish: no usable artifact for deployment f4b9dd63… — KEEPING the currently served address space rather than tearing it down`. **No `PureRemove`.** Browse before vs after: 18 vs 18, `diff` **identical**. |
| 4 | Reconcile guard fires; drivers survive | ✅ | `artifact for f4b9dd63… read back empty; skipping reconcile and keeping the 1 running driver(s)`, and the apply still reported `children=1`. Repeated on central-2 with a bigger blast radius: `keeping the 5 running driver(s)`, `children=5`. |
| 5 | SubscribeBulk guard fires; **live values keep flowing** | ✅ | central-2, `artifact for 8c36a673… read back empty; skipping SubscribeBulk and keeping the live subscriptions`. The decisive proof is the value, not the log: `ns=2;s=pymodbus/plc/HR200X` read **56278 @ 06:59:51** before the induction, then **56336 @ 07:00:21** and **56354 @ 07:00:30** after it (induction at 07:00:07). The subscription was still delivering fresh Modbus data across the event. |
| 6 | Positive control — a readable artifact still applies | ✅ | Immediately after the guards fired, a genuine deploy applied normally on both nodes: site-a-1 `AddRemoveMix, added=1, removed=1` + `SubscribeBulk pushed 1 references`; central-2 `AddRemoveMix, added=1, removed=1` + `SubscribeBulk pushed 5 references across 4 driver(s)`. The guards suppress the teardown, **not** deployment. |
| 7 | Control — only the node that saw empty bytes took the guard path | ✅ | Of the six nodes, exactly **one** logged `read back empty` (the paused one); the other five read the real artifact before it was blanked and applied it normally. The induction window is what produced the condition, not a global change. |
## How the failure was induced deterministically
The original incident was a race (a transient ConfigDb blip landing between the apply and the artifact
read). Racing it again would be unreliable, so the gate drives the **empty-bytes** branch instead — which
is byte-for-byte what the code sees when the row cannot be read, and is the branch the driver-side guards
key on (their *throw* paths already returned early before this work):
1. `docker pause` the target node, freezing it mid-cluster.
2. `POST /api/deployments` — the deployment seals and dispatches while the node cannot process it.
3. `UPDATE Deployment SET ArtifactBlob = 0x WHERE DeploymentId = <new>`.
4. `docker unpause` — the node now processes the dispatch and reads an empty artifact.
The pause measured **<1s** on both runs, well inside `acceptable-heartbeat-pause = 10s` and SBR's
`stable-after = 15s`, so the node was never flagged unreachable and no failover was triggered.
## Finding — the node records `Applied` for a deployment it skipped (pre-existing, NOT fixed here)
Both guards leave the node running its previous configuration, which is the intent. But `ApplyAndAck`
advances `_currentRevision = revision` and writes `NodeDeploymentState = Applied` **before** it knows
whether anything was applied — that assignment sits immediately after `ReconcileDrivers`, which returns
`null` on both the (pre-existing) throw path and the new empty path:
```
[07:00:07 WRN] DriverHost central-2:4053: artifact … read back empty; skipping reconcile …
[07:00:07 INF] DriverHost central-2:4053: applied deployment 8c36a673… (rev 03b33904…, children=5)
```
`NodeDeploymentState` for that deployment reads `Status = Applied` for central-2. The node now claims a
revision whose configuration it never applied — and because `HandleDispatchFromSteady` short-circuits on
a revision match (`dispatch matches current rev; immediate ACK`), **re-dispatching that same revision is
a no-op**, so the node cannot self-heal. Only a later deployment with a *different* revision recovers it
(check 6 confirms that path works).
This is pre-existing — the throw path has always done it — and is strictly a reporting/convergence bug,
not a data-plane one: the node keeps serving its last good configuration throughout. The honest fix is to
ACK `Failed` and leave `_currentRevision` alone when the artifact could not be read, so the normal deploy
retry heals it. Tracked separately rather than folded in here, because it changes deploy ACK semantics
fleet-wide.