docs(deployment): QA hotfix da0bfe2b — MxGateway graceful-stream-end disconnect; incident root cause (dashboard session close 15:29:03)

2026-08-20 16:57:45 -04:00
parent da0bfe2be1
commit 50f7439e33
1 changed files with 20 additions and 0 deletions
@@ -59,3 +59,23 @@ start Central pair → wait 20 s → start Site pair. Pairs were stopped and sta

 ## 3. Outstanding
 - None from this deploy. ZIPSEN (id 7) remains the known non-deployable draft instance.
+
+## 4. Hotfix `da0bfe2b` — same day, 16:53 (QA only so far)
+
+**Incident (reported on vq03 Z28061 MoveIn):** `SetAttribute('RightMESReceiver.MoveInCompleteFlag')
+failed: Session session-736b2d47… was not found` while the attribute showed GOOD quality.
+**Root cause (from both logs):** at 15:29:03 the gateway dashboard user `admin` (client
+`10.181.200.169`) closed ScadaBridge's MxGateway session. The gateway completes a closed
+session's event stream *gracefully* (OK status), and `MxGatewayDataConnection.RunEventLoopAsync`
+only raised `Disconnected` on a *fault* — so the adapter stayed Connected on a dead session for
+an hour (values frozen at last GOOD, every write `NotFound`; the 10 s `NotFound` heartbeat was
+the unresolved-tag resolution probe re-subscribing on the dead session). The 16:28 upgrade
+restart cleared it incidentally. A graceful gateway shutdown takes the same path.
+
+**Fix:** `da0bfe2b` — a non-cancelled end of the event stream now signals `Disconnected`
+(pinned by `MxGatewayDataConnectionReconnectTests.EventStreamGracefulCompletion_*`).
+Deployed to all four QA nodes as a binary-only swap (no migrations, no config): backups
+`app.bak.20260820T165357` / `cli.bak.20260820T165357` on both boxes; `/health/ready` 200
+Healthy, 2/2 members, site replication connected. **Not yet on prod vd03.** Live gate still to
+run: close the site's session from the gateway dashboard and confirm the site reconnects
+(`MxGateway event stream ended without a fault … signalling disconnect` → reconnect → values resume).