From 8d1621e5343d0d3f7e231d9653931128b438951a Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Wed, 11 Nov 2026 18:40:52 -0500 Subject: [PATCH] =?UTF-8?q?docs(review):=20#761=20gate=20F=20=E2=80=94=202?= =?UTF-8?q?0=20Galaxy=20respawn=20deploys=20keep=20mxgateway.sessions.open?= =?UTF-8?q?=20flat=20(VD03=20DEV=20gw)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PASS: gauge baseline 2, +2 with the rig's driver on both centrals, 4 after 20 config deploys (40 respawns; rig opened 39 / closed 37). Final drop after removal not observed: the Mac's VPN to VD03 dropped right after the settle. Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk --- docs/reviews/2026-10-21/gates.md | 83 +++++++++++++++++++++++++++++++- 1 file changed, 82 insertions(+), 1 deletion(-) diff --git a/docs/reviews/2026-10-21/gates.md b/docs/reviews/2026-10-21/gates.md index 54675e12b..1379f9ef8 100644 --- a/docs/reviews/2026-10-21/gates.md +++ b/docs/reviews/2026-10-21/gates.md @@ -314,5 +314,86 @@ Test script, test role grant, gateD-bulk driver/device/5000 tags all removed; no ### Still not run after gate D -- Gate F (Galaxy) — needs a reachable mxaccessgw. +- Gate F (Galaxy) — run 2026-11-11, see "Gate F (#735)" below. - Hardware userGates (real PLCs), as listed under NOT RUN above. + +## Gate F (#735) — 20 config deploys keep `mxgateway.sessions.open` flat (docker-dev + VD03 DEV mxaccessgw, 2026-11-11) — PASS + +| | | +|---|---| +| Commit tested | `5ac55205` (master). Image `otopcua-host:dev` `sha256:47ca0da6eaf7` rebuilt from that clean checkout before the run; all six nodes recreated from it. | +| Gateway | DEV mxaccessgw on `wonder-app-vd03.zmr.zimmer.com` (gRPC h2c `:5120`, assembly `0.1.2.0`, main `2b13564e`). Metrics from its anonymous Prometheus endpoint `http://wonder-app-vd03.zmr.zimmer.com:5130/metrics` (`mxgateway_sessions_open`, `…_sessions_opened_total`, `…_sessions_closed_total`, `…_workers_running`, `…_workers_killed_total`, `…_faults_total`), scraped every 15 s from the Mac for the whole run. Read-only Galaxy access, owner-approved. No remote command ran on any wonder box. | +| Reachability | The docker-dev containers resolved and reached `wonder-app-vd03:5120` through the Mac's VPN route (`/dev/tcp` check from `central-1`) before anything was configured. | +| Driver | `gatef-gal`, DriverType `GalaxyMxGateway`, cluster MAIN, seeded straight into the ConfigDb: `gateway.endpoint` = the VD03 gateway, `useTls=false`, `apiKeySecretRef=env:GALAXY_MXGW_API_KEY`, `mxAccess.clientName=OtOpcUa-761-gateF`, `repository.watchDeployEvents=false`, no `alarmAck` fallback. Device `vd03` (`{}`); two **Read**-only tags: `ScanState` (`DevWinPlatform.ScanState`, Boolean) and `CPULoad` (`DevWinPlatform.CPULoad`, Float64). No writes, no alarm subscriptions, no acks. | +| Secret | The VD03 gateway key reached the containers only as the `GALAXY_MXGW_API_KEY` runtime env (exported from the owner's env file into the `docker compose up` shell). It is not in the repo, a compose file, the ConfigDb or this record. | +| Respawn trigger | As in gate A: each iteration edits the driver's config (`JSON_MODIFY` of `mxAccess.publishingIntervalMs` = 1000 + i), POSTs `/api/deployments`, waits for it to seal, waits 15 s, then scrapes the gateway. A config change is a respawn (#665), so every iteration stops and disposes the old `GalaxyDriver` and spawns a new one on **both** `central-1` and `central-2` (MAIN runs its drivers on both nodes) — 40 driver respawns in total. | + +**Attribution.** The gateway's metrics have no per-client label (the session table with client names is on the +login-gated dashboard, which was not used), so the rig's share is measured as the **delta** against a baseline +taken before the driver existed, cross-checked against the `opened`/`closed` counters. During the whole window the +other consumers' sessions did not churn: `opened − closed` moved only by what the rig did. + +**Gateway samples** (UTC): + +| when | open | opened | closed | workers running | +|---|---|---|---|---| +| baseline, 13 samples 23:11:15–23:14:11 (no Galaxy driver on the rig) | 2 | 3 | 1 | 2 | +| driver added + deployed (23:14:34) | **4** (+2: one session per central) | 5 | 1 | 4 | +| iterations 1–20, 15 s after each seal (23:15:13–23:20:56) | 4 in 13, 3 in 4 (it. 5, 6, 11, 16), 2 in 3 (it. 1, 17, 18); **never above 4** | 5 → 42 | 1 → 38 | 2–4 | +| settled, 23:21:04 / 23:21:09 / 23:21:25 | **4** | 42 | 38 | 4 | + +The rig opened 39 sessions over the run (42 − 3) and closed 37 (38 − 1); the net +2 is exactly its two live +drivers. The open gauge returned to its steady value (baseline + 2) after the 20 deploys, so no session leaked. +The dips to 2–3 are samples that landed while a respawned driver was still reconnecting (see the init timeouts +below), not lost sessions; every iteration ended with both drivers connected. + +Gateway-side counters over the run: `workers_exited{worker-shutdown-ack}` 1 → 37 (graceful closes), +`workers_killed{OpenSessionFailed}` 0 → 13, `workers_killed{client-close}` 0 → 1, `faults{WorkerFaulted}` 0 → 14, +`workers_exited{WorkerFaulted}` 0 → 1, `grpc_streams_disconnected{ClientCancelled}` 31. + +**Rig side** (per central, 23:14:56–23:21:10): 20 `spawned GalaxyMxGateway driver gatef-gal`, 20 +`stopped driver child`, 20 `disposed GalaxyMxGateway driver on stop`, 25 `GalaxyDriver gatef-gal shutting down` +(20 respawns + 5 failed-init teardowns), 15 `GalaxyMxSession connected`, 5 `initialize failed: … timeout of +'00:00:05'`. central-2 logged one `GalaxyMxSession close failed or timed out (best-effort) — dropping the channel`. +No error or fault otherwise. + +**Driver health after the run** (`/hosts`, `/fleet/diagnostics`): `gatef-gal` on central-1 and central-2 +**Healthy**, 2 tags, host status "2 / 6 down" (the probe watcher seeded 5 platforms; two Galaxy platforms on VD03 +DEV report down, unrelated to the gate). `Client.CLI read` of both tags on 4840 and 4841 was Good before the loop +(`ScanState` True, `CPULoad` ≈ 13.4) and after it (`CPULoad` 15.63, Good). + +### Removal and the final drop — not observed (VPN dropped) + +The Mac's VPN route to VD03 went down at about 23:21:35, ten seconds after the settled samples. DNS for +`*.zmr.zimmer.com` and the `10.220.x` route both disappeared; it was still down at 23:40. The rig's Galaxy +drivers lost the gateway (reconnect supervisor backing off, Initialize breaker opened), which is the expected +behaviour. + +`gatef-gal` (driver, device, 2 tags) was deleted from the ConfigDb and the removal deployed at 23:30:56 +(`12d5ff59`, Sealed). Both centrals logged `stopped driver child gatef-gal` and `disposed GalaxyMxGateway driver +on stop`. Because the gateway was unreachable, the rig could not close its two sessions; the gateway reaps a +session whose client stream is gone on its detach-grace / lease sweep. **The return of `sessions.open` to 2 was +therefore not observed.** Re-check `mxgateway_sessions_open` once the VPN is back: it should read the baseline +(2, if VD03's own consumers are unchanged). + +### Observations (none blocks the gate) + +1. **A Galaxy (re)start often exceeds the 5 s Tier-A `Initialize` timeout against VD03.** 10 of 40 respawns + (5 per node) failed `initialize` at exactly 5 s. Opening a gateway session spawns an x86 MXAccess worker, and + a successful connect took 4–5 s from spawn to `connected`, right at the limit. Each timeout cost a 10 s reconnect-timer retry + (BadNoCommunication on that driver's nodes meanwhile) and shows on the gateway as an `OpenSessionFailed` + worker kill plus a `WorkerFaulted` fault. The gateway cleaned up every one (the gauge stayed flat), so it is a + latency and noise issue, not a leak. Galaxy probably wants a longer `Initialize` timeout (tier or per-instance + `ResilienceConfig`). Not filed (no Gitea per brief). +2. **The VD03 gateway's `faults_total` is now 14 (`WorkerFaulted`) from this gate** — anyone reading the VD03 + dashboard after 2026-11-11 23:15Z should attribute those, and the 13 `OpenSessionFailed` kills, to this run. +3. The rig's opens (39 at the gateway) exceed its logged `GalaxyMxSession connected` lines (30 in the loop plus 2 + on add); the extra opens are ones that completed after the client's 5 s init timeout had already given up. + They were closed by the end of the window (opened − closed = 2 at 23:21). + +### Rig state left behind + +- `gatef-gal` and its device/tags are deleted from the ConfigDb; the removal deploy sealed. The gate's + `Deployment` rows carry only the `env:` reference, no secret. +- All host nodes were recreated without the exported key, so `GALAXY_MXGW_API_KEY` is back to the committed + docker-dev default. The rig runs `otopcua-host:dev` `sha256:47ca0da6eaf7` (`5ac55205`).