dohertj2/lmxopcua
1
0
Fork 0
Code Issues 5 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(review): #761 gate F — 20 Galaxy respawn deploys keep mxgateway.sessions.open flat (VD03 DEV gw)

Browse Source 
PASS: gauge baseline 2, +2 with the rig's driver on both centrals, 4 after 20
config deploys (40 respawns; rig opened 39 / closed 37). Final drop after
removal not observed: the Mac's VPN to VD03 dropped right after the settle.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
This commit is contained in:
Joseph Doherty
2026-11-11 18:40:52 -05:00
parent 5ac5520532
commit 8d1621e534
1 changed files with 82 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/reviews/2026-10-21/gates.md
+82 -1
View File
@@ -314,5 +314,86 @@ Test script, test role grant, gateD-bulk driver/device/5000 tags all removed; no
### Still not run after gate D
- Gate F (Galaxy) — needs a reachable mxaccessgw.
- Gate F (Galaxy) — run 2026-11-11, see "Gate F (#735)" below.
- Hardware userGates (real PLCs), as listed under NOT RUN above.
## Gate F (#735) — 20 config deploys keep `mxgateway.sessions.open` flat (docker-dev + VD03 DEV mxaccessgw, 2026-11-11) — PASS
| | |
|---|---|
| Commit tested | `5ac55205` (master). Image `otopcua-host:dev` `sha256:47ca0da6eaf7` rebuilt from that clean checkout before the run; all six nodes recreated from it. |
| Gateway | DEV mxaccessgw on `wonder-app-vd03.zmr.zimmer.com` (gRPC h2c `:5120`, assembly `0.1.2.0`, main `2b13564e`). Metrics from its anonymous Prometheus endpoint `http://wonder-app-vd03.zmr.zimmer.com:5130/metrics` (`mxgateway_sessions_open`, `…_sessions_opened_total`, `…_sessions_closed_total`, `…_workers_running`, `…_workers_killed_total`, `…_faults_total`), scraped every 15 s from the Mac for the whole run. Read-only Galaxy access, owner-approved. No remote command ran on any wonder box. |
| Reachability | The docker-dev containers resolved and reached `wonder-app-vd03:5120` through the Mac's VPN route (`/dev/tcp` check from `central-1`) before anything was configured. |
| Driver | `gatef-gal`, DriverType `GalaxyMxGateway`, cluster MAIN, seeded straight into the ConfigDb: `gateway.endpoint` = the VD03 gateway, `useTls=false`, `apiKeySecretRef=env:GALAXY_MXGW_API_KEY`, `mxAccess.clientName=OtOpcUa-761-gateF`, `repository.watchDeployEvents=false`, no `alarmAck` fallback. Device `vd03` (`{}`); two **Read**-only tags: `ScanState` (`DevWinPlatform.ScanState`, Boolean) and `CPULoad` (`DevWinPlatform.CPULoad`, Float64). No writes, no alarm subscriptions, no acks. |
| Secret | The VD03 gateway key reached the containers only as the `GALAXY_MXGW_API_KEY` runtime env (exported from the owner's env file into the `docker compose up` shell). It is not in the repo, a compose file, the ConfigDb or this record. |
| Respawn trigger | As in gate A: each iteration edits the driver's config (`JSON_MODIFY` of `mxAccess.publishingIntervalMs` = 1000 + i), POSTs `/api/deployments`, waits for it to seal, waits 15 s, then scrapes the gateway. A config change is a respawn (#665), so every iteration stops and disposes the old `GalaxyDriver` and spawns a new one on **both** `central-1` and `central-2` (MAIN runs its drivers on both nodes) — 40 driver respawns in total. |
**Attribution.** The gateway's metrics have no per-client label (the session table with client names is on the
login-gated dashboard, which was not used), so the rig's share is measured as the **delta** against a baseline
taken before the driver existed, cross-checked against the `opened`/`closed` counters. During the whole window the
other consumers' sessions did not churn: `opened − closed` moved only by what the rig did.
**Gateway samples** (UTC):
| when | open | opened | closed | workers running |
|---|---|---|---|---|
| baseline, 13 samples 23:11:15–23:14:11 (no Galaxy driver on the rig) | 2 | 3 | 1 | 2 |
| driver added + deployed (23:14:34) | **4** (+2: one session per central) | 5 | 1 | 4 |
| iterations 1–20, 15 s after each seal (23:15:13–23:20:56) | 4 in 13, 3 in 4 (it. 5, 6, 11, 16), 2 in 3 (it. 1, 17, 18); **never above 4** | 5 → 42 | 1 → 38 | 2–4 |
| settled, 23:21:04 / 23:21:09 / 23:21:25 | **4** | 42 | 38 | 4 |
The rig opened 39 sessions over the run (42 − 3) and closed 37 (38 − 1); the net +2 is exactly its two live
drivers. The open gauge returned to its steady value (baseline + 2) after the 20 deploys, so no session leaked.
The dips to 2–3 are samples that landed while a respawned driver was still reconnecting (see the init timeouts
below), not lost sessions; every iteration ended with both drivers connected.
Gateway-side counters over the run: `workers_exited{worker-shutdown-ack}` 1 → 37 (graceful closes),
`workers_killed{OpenSessionFailed}` 0 → 13, `workers_killed{client-close}` 0 → 1, `faults{WorkerFaulted}` 0 → 14,
`workers_exited{WorkerFaulted}` 0 → 1, `grpc_streams_disconnected{ClientCancelled}` 31.
**Rig side** (per central, 23:14:56–23:21:10): 20 `spawned GalaxyMxGateway driver gatef-gal`, 20
`stopped driver child`, 20 `disposed GalaxyMxGateway driver on stop`, 25 `GalaxyDriver gatef-gal shutting down`
(20 respawns + 5 failed-init teardowns), 15 `GalaxyMxSession connected`, 5 `initialize failed: … timeout of
'00:00:05'`. central-2 logged one `GalaxyMxSession close failed or timed out (best-effort) — dropping the channel`.
No error or fault otherwise.
**Driver health after the run** (`/hosts`, `/fleet/diagnostics`): `gatef-gal` on central-1 and central-2
**Healthy**, 2 tags, host status "2 / 6 down" (the probe watcher seeded 5 platforms; two Galaxy platforms on VD03
DEV report down, unrelated to the gate). `Client.CLI read` of both tags on 4840 and 4841 was Good before the loop
(`ScanState` True, `CPULoad` ≈ 13.4) and after it (`CPULoad` 15.63, Good).
### Removal and the final drop — not observed (VPN dropped)
The Mac's VPN route to VD03 went down at about 23:21:35, ten seconds after the settled samples. DNS for
`*.zmr.zimmer.com` and the `10.220.x` route both disappeared; it was still down at 23:40. The rig's Galaxy
drivers lost the gateway (reconnect supervisor backing off, Initialize breaker opened), which is the expected
behaviour.
`gatef-gal` (driver, device, 2 tags) was deleted from the ConfigDb and the removal deployed at 23:30:56
(`12d5ff59`, Sealed). Both centrals logged `stopped driver child gatef-gal` and `disposed GalaxyMxGateway driver
on stop`. Because the gateway was unreachable, the rig could not close its two sessions; the gateway reaps a
session whose client stream is gone on its detach-grace / lease sweep. **The return of `sessions.open` to 2 was
therefore not observed.** Re-check `mxgateway_sessions_open` once the VPN is back: it should read the baseline
(2, if VD03's own consumers are unchanged).
### Observations (none blocks the gate)
1. **A Galaxy (re)start often exceeds the 5 s Tier-A `Initialize` timeout against VD03.** 10 of 40 respawns
(5 per node) failed `initialize` at exactly 5 s. Opening a gateway session spawns an x86 MXAccess worker, and
a successful connect took 4–5 s from spawn to `connected`, right at the limit. Each timeout cost a 10 s reconnect-timer retry
(BadNoCommunication on that driver's nodes meanwhile) and shows on the gateway as an `OpenSessionFailed`
worker kill plus a `WorkerFaulted` fault. The gateway cleaned up every one (the gauge stayed flat), so it is a
latency and noise issue, not a leak. Galaxy probably wants a longer `Initialize` timeout (tier or per-instance
`ResilienceConfig`). Not filed (no Gitea per brief).
2. **The VD03 gateway's `faults_total` is now 14 (`WorkerFaulted`) from this gate** — anyone reading the VD03
dashboard after 2026-11-11 23:15Z should attribute those, and the 13 `OpenSessionFailed` kills, to this run.
3. The rig's opens (39 at the gateway) exceed its logged `GalaxyMxSession connected` lines (30 in the loop plus 2
on add); the extra opens are ones that completed after the client's 5 s init timeout had already given up.
They were closed by the end of the window (opened − closed = 2 at 23:21).
### Rig state left behind
- `gatef-gal` and its device/tags are deleted from the ConfigDb; the removal deploy sealed. The gate's
`Deployment` rows carry only the `env:` reference, no secret.
- All host nodes were recreated without the exported key, so `GALAXY_MXGW_API_KEY` is back to the committed
docker-dev default. The rig runs `otopcua-host:dev` `sha256:47ca0da6eaf7` (`5ac55205`).