lmxopcua/docs/plans/2026-07-21-issue-485-live-gate.md

# Issue #485 — live gate (docker-dev rig)

> Verifies the two fixes for #485 — `OpcUaPublishActor` (master `c7f5b9cf`) and `DriverHostActor`
> (master `5aad1e33`) — on the local `docker-dev` rig: 6 nodes, shared SQL, real Modbus fixture on
> `10.100.0.35:5020`.
>
> **Safety rules followed**: all DB inspection + mutation via `sqlcmd` inside the SQL container; the
> rig's Modbus fixture is read-only for this gate (no writes to the PLC sim).

## Result

**Both fixes verified, all three guards observed firing in production, with a live RED control that
was not staged.** The gate also confirmed a **pre-existing** defect the fixes do not address — the node
records `Applied` for a deployment it skipped (see "Finding" below).

| # | Check | Result | Evidence |
|---|---|---|---|
| 1 | RED control — the bug in its natural habitat, unforced | ✅ | site-a-1 hit #485 during the *previous* gate's SQL flapping and had been serving an **empty address space for 34 minutes**: `failed to load artifact … rebuild becomes no-op` followed immediately by `applied plan (kind=PureRemove, added=0, removed=17, rebuild=True)`. Its 17 `ns=2` nodes were gone; a browse returned only the root `OtOpcUa` folder. Nothing was staged to produce this — it was already true on the pre-fix image. |
| 2 | Rebuilt on the fixed image, the node recovers | ✅ | After `docker compose build` + `--force-recreate`, site-a-1 restored to `raw subtree materialised (containers=16, tags=1)` — 18 `ns=2` nodes. |
| 3 | Address-space guard fires; nodes survive | ✅ | site-a-1, empty artifact: `OpcUaPublish: no usable artifact for deployment f4b9dd63… — KEEPING the currently served address space rather than tearing it down`. **No `PureRemove`.** Browse before vs after: 18 vs 18, `diff` **identical**. |
| 4 | Reconcile guard fires; drivers survive | ✅ | `artifact for f4b9dd63… read back empty; skipping reconcile and keeping the 1 running driver(s)`, and the apply still reported `children=1`. Repeated on central-2 with a bigger blast radius: `keeping the 5 running driver(s)`, `children=5`. |
| 5 | SubscribeBulk guard fires; **live values keep flowing** | ✅ | central-2, `artifact for 8c36a673… read back empty; skipping SubscribeBulk and keeping the live subscriptions`. The decisive proof is the value, not the log: `ns=2;s=pymodbus/plc/HR200X` read **56278 @ 06:59:51** before the induction, then **56336 @ 07:00:21** and **56354 @ 07:00:30** after it (induction at 07:00:07). The subscription was still delivering fresh Modbus data across the event. |
| 6 | Positive control — a readable artifact still applies | ✅ | Immediately after the guards fired, a genuine deploy applied normally on both nodes: site-a-1 `AddRemoveMix, added=1, removed=1` + `SubscribeBulk pushed 1 references`; central-2 `AddRemoveMix, added=1, removed=1` + `SubscribeBulk pushed 5 references across 4 driver(s)`. The guards suppress the teardown, **not** deployment. |
| 7 | Control — only the node that saw empty bytes took the guard path | ✅ | Of the six nodes, exactly **one** logged `read back empty` (the paused one); the other five read the real artifact before it was blanked and applied it normally. The induction window is what produced the condition, not a global change. |

## How the failure was induced deterministically

The original incident was a race (a transient ConfigDb blip landing between the apply and the artifact
read). Racing it again would be unreliable, so the gate drives the **empty-bytes** branch instead — which
is byte-for-byte what the code sees when the row cannot be read, and is the branch the driver-side guards
key on (their *throw* paths already returned early before this work):

1. `docker pause` the target node, freezing it mid-cluster.
2. `POST /api/deployments` — the deployment seals and dispatches while the node cannot process it.
3. `UPDATE Deployment SET ArtifactBlob = 0x WHERE DeploymentId = <new>`.
4. `docker unpause` — the node now processes the dispatch and reads an empty artifact.

The pause measured **<1s** on both runs, well inside `acceptable-heartbeat-pause = 10s` and SBR's
`stable-after = 15s`, so the node was never flagged unreachable and no failover was triggered.

## Finding — the node records `Applied` for a deployment it skipped (pre-existing, NOT fixed here)

> **Update 2026-08-07 (truth sweep): fixed as issue #486 — see the Addendum below.** In today's
> source the fix lives at `Runtime/Drivers/DriverHostActor.cs:1696-1717` (null blob ⇒ status
> `Failed`, ack `Failed`, revision NOT advanced; the comment cites #486), mirrored on the
> FetchAndCache path (~`:1609-1653`, "artifact could not be read (#486) and keeps last-known-good").

Both guards leave the node running its previous configuration, which is the intent. But `ApplyAndAck`
advances `_currentRevision = revision` and writes `NodeDeploymentState = Applied` **before** it knows
whether anything was applied — that assignment sits immediately after `ReconcileDrivers`, which returns
`null` on both the (pre-existing) throw path and the new empty path:

```
[07:00:07 WRN] DriverHost central-2:4053: artifact … read back empty; skipping reconcile …
[07:00:07 INF] DriverHost central-2:4053: applied deployment 8c36a673… (rev 03b33904…, children=5)
```

`NodeDeploymentState` for that deployment reads `Status = Applied` for central-2. The node now claims a
revision whose configuration it never applied — and because `HandleDispatchFromSteady` short-circuits on
a revision match (`dispatch matches current rev; immediate ACK`), **re-dispatching that same revision is
a no-op**, so the node cannot self-heal. Only a later deployment with a *different* revision recovers it
(check 6 confirms that path works).

This is pre-existing — the throw path has always done it — and is strictly a reporting/convergence bug,
not a data-plane one: the node keeps serving its last good configuration throughout. The honest fix is to
ACK `Failed` and leave `_currentRevision` alone when the artifact could not be read, so the normal deploy
retry heals it. Tracked separately rather than folded in here, because it changes deploy ACK semantics
fleet-wide.

## Addendum — issue #486 fix, live-gated (2026-07-21)

The finding above is now fixed (`DriverHostActor.ApplyAndAck` fails the apply when `ReconcileDrivers`
returns null) and gated on the same rig with the same induction.

| # | Check | Result | Evidence |
|---|---|---|---|
| 8 | Regression — a normal deploy still ACKs Applied fleet-wide | ✅ | The change alters ACK semantics, so this is the risk that matters. A genuine deploy recorded `Status = Applied` on **all six** nodes. |
| 9 | An unreadable artifact now ACKs Failed, with a reason | ✅ | central-2 recorded `Status = Failed` + `the deployment artifact could not be read (ConfigDb unreachable, or the row is missing/empty); nothing was applied`. The five nodes that read the real artifact recorded Applied — so the fleet view now distinguishes them. Pre-fix, all six claimed Applied. |
| 10 | The revision is NOT advanced | ✅ | `apply of a855dd9e… FAILED — … Staying on revision 5bc0d9f3…` — the **previous** revision, not the dispatched `583122b7…`. This is what stops `HandleDispatchFromSteady` from short-circuiting a retry of that revision. |
| 11 | The node keeps serving throughout (#485 still holds) | ✅ | The #485 guard fired first (`keeping the 5 running driver(s)`), and `ns=2;s=pymodbus/plc/HR200X` read **59603 @ 07:27:28**, live and current, after the Failed apply. Failing the apply reports the truth; it does not tear anything down. |
| 12 | The node recovers on the next real deploy | ✅ | The following deployment applied on central-2 (`children=5`) and recorded Applied on all six nodes. |