lmxopcua/docs/reviews/2026-10-21/gates.md

# 2026-10-21 review remediation — Wave-4 gate record

Tracking issue: lmxopcua#761. This file records each gate run with its date, the commit tested, the procedure,
the measured numbers and the verdict. Gates not listed here have not been run.

## docker-dev gates A, B, C, E — 2026-11-09

| | |
|---|---|
| Commit tested | `1b9c36f3` (`origin/master`, "docs(plans): OPC UA SDK 2.0 upgrade spike catalogue (lmxopcua#782)") |
| Rig | local `docker-dev/` (6 host nodes: MAIN `central-1/2`, SITE-A `site-a-1/2`, SITE-B `site-b-1/2`), image `otopcua-host:dev` `sha256:24f9d7287ed2` built from that commit, migrator re-run (no pending migrations) |
| Clock | Times below are rig UTC (2026-11-10 01:37–02:07 UTC = 2026-11-09 evening US Eastern) |
| Config | The existing ConfigDb: MAIN carries `MAIN-modbus` (pymodbus `10.100.0.35:5020`), `MAIN-opcua` (opc-plc `10.100.0.35:50000`), `MAIN-calc`, 48 `int3b-mb-*` Modbus drivers and one `int3b-blackhole` driver, 52 children in total. Equipment `EQ-32296cc68964` (`a/l/e1`) has the HR5 reference and the change-triggered virtual tag `okvt` (= HR100 + 1). |
| Tools | `POST /api/deployments` + `GET /api/deployments/{id}` (`X-Api-Key: docker-dev-deploy-key`), `Client.CLI` built from the same commit, `/proc/1/status` (Threads) and `/proc/1/fd` inside the containers. `dotnet-counters` is not in the runtime image, so it was not used. ConfigDb edits were made with `sqlcmd -I` (QUOTED_IDENTIFIER on). |

**Summary: A PASS, B PASS, C PASS, E PASS.** None of the four gates failed. Observations are listed at the end. None of
them blocks a gate.

### Gate A (#731) — a driver config change disposes the old instance; threads and handles stay flat — PASS

**Procedure.** Each iteration edits both `MAIN-modbus.DriverConfig` (`TimeoutMs` 2000+i) and `MAIN-opcua.DriverConfig`
(`SessionName` `gateA-i`), deploys, waits for the deploy to seal plus 8 s, then samples PID 1 in `central-1` and
`central-2` (both run the MAIN drivers). That is 20 iterations, so 20 respawns of each driver on each node: 80 driver
disposals in total. Two warm-up respawns ran before the baseline.

**Disposal evidence** (log lines in the 01:41:47–01:45:10 window):

| node | `MAIN-modbus: disposed Modbus driver on stop` | `MAIN-opcua: disposed OpcUaClient driver on stop` | `spawned … MAIN-modbus` / `MAIN-opcua` | shutdown/dispose timeout or throw |
|---|---|---|---|---|
| central-1 | 20 | 20 | 20 / 20 | 0 |
| central-2 | 20 | 20 | 20 / 20 | 0 |

**Counts** (threads / open fds / RSS KiB):

| sample | central-1 | central-2 |
|---|---|---|
| baseline (01:41:47) | 46 / 714 / 490 748 | 46 / 717 / 516 600 |
| after respawn 5 | 50 / 725 / 506 576 | 51 / 725 / 529 472 |
| after respawn 10 | 50 / 725 / 507 084 | 48 / 726 / 532 972 |
| after respawn 15 | 48 / 726 / 497 648 | 48 / 724 / 525 440 |
| after respawn 20 (01:44:59) | 52 / 726 / 499 048 | 48 / 723 / 526 184 |
| settled, 75 s later (01:46:14) | 48 / 712 | 46 / 715 |

Across the 20 respawns, threads moved between 47 and 55 with no upward trend; that range is normal thread-pool
movement. The fd count rose by about 11 on the first respawn, which is the new driver's sockets landing before the old
ones close. It then stayed flat at 723–726, and after settling it was lower than the baseline (712/715). RSS stayed
flat (±25 MB).

After a respawn, the static Modbus values (HR5 and HR200, which do not change) read Good on both 4840 and 4841. The
dead child's `BadNoCommunication` fan, which lands after the new child's first push, was overwritten by the re-push.

### Gate B (#732) — a subscription survives a surgical add; a re-created item after a rebuild receives data — PASS

**Procedure.** Session 1 opens two `Client.CLI subscribe` processes (500 ms) on `opc.tcp://localhost:4840`, which
are held for 170 s:

- the raw node `ns=2;s=pymodbus/plc/HR100` (ticks about 1 Hz)
- the UNS virtual tag `ns=3;s=EQ-32296cc68964/okvt`

1. **Surgical add (01:47:42).** Insert the raw tag `HR101` on `MAIN-modbus-dev`, then deploy. The applier logged
   `kind=PureAdd, added=1, removed=0, changed=0, rebuild=False` on both central nodes.
2. **Read the new node.** `ns=2;s=pymodbus/plc/HR101` exists and is served. Its status is `BadOutOfRange` because the
   pymodbus fixture refuses address 101 with exception 0x02. The device answered, so this is the fixture's register
   map and not a server fault.
3. **Rebuild (01:48:19).** Rename equipment `e1` to `e1r`, which is a `ChangedEquipment` delta, then deploy. The
   applier logged `kind=Rebuild, … rebuild=True`.
4. **Re-create (01:48:35).** With session 1 still open and still holding its stale items, start session 2 with new
   subscriptions on the same two NodeIds for 20 s.

**Results.**

| subscription | samples | non-Good samples |
|---|---|---|
| session 1, HR100 | 59 | `01:47:43.1426 BadNoCommunication` (one sample; Good `31505` 0.9 ms later, then 1 Hz Good through 01:48:19); `01:48:19.97 BadNodeIdUnknown`, then nothing (item dead, as B2a specifies) |
| session 1, okvt | 59 | same pattern: one `BadNoCommunication` sample at 01:47:43.1426, Good again at .1435; `BadNodeIdUnknown` at 01:48:19.97 |
| session 2, HR100 (after the rebuild) | 20 | 0. Values ran from 36580 at 01:48:34.99 to 38405 at 01:48:54.00 |
| session 2, okvt (after the rebuild) | 20 | 0. Values ran from 36581 to 38406 |

- **Surgical add:** the subscription survived. The items were never invalidated and Good data continued at 1 Hz. The
  single `BadNoCommunication` sample is not address-space churn. Adding a tag changes the driver's spawn spec, so
  `MAIN-modbus` respawned (`stopped driver child` / `disposed` / `spawned` at 01:47:43). The dead child's
  connectivity-lost fan then reached the item for under 1 ms before the new child's push (see observation 3).
- **Rebuild:** the old items received `BadNodeIdUnknown` and no further data. New items on the same NodeIds, created
  while the stale items were still open in another session, received live data immediately. That is the B2a positive
  control: before the fix, a stale `MonitoredNode2` captured the new item.

The CLI cannot re-create an item inside the same session. The same-session case is covered by the
`OpcUaServer.IntegrationTests` test, not by this gate.

### Gate C (#733) — kill a central mid-dispatch, the next deploy succeeds; port 4840 taken ⇒ `/health/ready` 503 — PASS

To hold a deployment in flight long enough to kill a node, `site-b-1` was `docker pause`d before each deploy, so its
ack could not land. The dispatch is otherwise sub-second on this rig. Both centrals were killed in turn:
`central-1`, which the issue names, and `central-2`, which was hosting the `config-publish` / `admin-operations`
singletons at the time. Killing the singleton host is what exercises the C1 PreStart recovery.

**C-a: `docker kill central-1` mid-dispatch.** Deployment `197e31b4` was `AwaitingApplyAcks` with 5 of 6 rows
Applied (site-b-1 Applying) when central-1 was killed at 01:51:42.

- A second POST while it was in flight returned **409 AnotherDeploymentInFlight**, which is correct.
- After `site-b-1` was unpaused, the coordinator closed the deployment at the 2 min deadline (01:53:38) as
  `TimedOut`. It logged `No ack from: site-b-1:4053`, because the gRPC dispatch to the paused node was lost.
- central-1 was restarted, and the **next deploy, `b27ede2a`, sealed with 6 of 6 Applied.**

**C-b: `docker kill central-2` (singleton host) mid-dispatch.** Deployment `970e7770` was `AwaitingApplyAcks`
(5 of 6) when central-2 was killed at 01:54:03.

- 01:54:30: central-1 auto-downed central-2.
- 01:54:46: the singletons started on central-1 and logged `Coordinator recovered in-flight deployment 970e7770… (5/6 acks landed)`.
- The AdminUI was unavailable through Traefik from about 01:54:05 to 01:54:35 during the hand-over.
- The recovered deployment closed `TimedOut` at its original deadline (01:56:00).
- central-2 was restarted, and the **next deploy, `cb0b8134`, sealed with 6 of 6 Applied.** Neither run left a
  deployment wedged in flight.

**C-port: port 4840 taken.**

- **Fused admin+driver node.** central-2 was stopped. A replacement container was started from the same compose
  service (`compose run --use-aliases`). A `perl` listener in it bound `0.0.0.0:4840` before
  `dotnet OtOpcUa.Host.dll` started. Results:
  - `GET /health/ready` returned **HTTP 503**, overall status `Unhealthy`, with
    `opcua-endpoint: Unhealthy — "OPC UA SDK failed to start: Failed to establish tcp listener sockets for Ipv4 and
    IPv6." (status Failed)`. `configdb` and `akka` were both Healthy.
  - The node logged `OpcUaEndpointStepDown: … not the driver Primary — nothing to step down from`.
- **Driver-only node.** The same setup on `site-a-2` (driver-only): the host logged
  `FTL OtOpcUa.Host terminated: Failed to establish tcp listener sockets…` and the **process exited with code 1**
  (C3 "a driver-only node rethrows").
- Both replacement containers were removed and the original containers restarted. The ServiceLevel half of C3 is
  #762 and was not part of this gate.

### Gate E (#734) — a `while(true)` virtual tag leaves the node responsive, the tag shows quarantined, threads stay flat over 2 min — PASS

**Procedure.**

- **Script:** `gateE-spin` with source `var v = ctx.GetTag("pymodbus/plc/HR100").Value; while (true) { }`.
- **Virtual tag:** `spinvt` on `EQ-32296cc68964`, Int32, change-triggered, so it fires about 1 Hz on HR100.
- **Healthy sibling:** `okvt`, on the same equipment and the same trigger.
- Deployed at 02:01:11. Before the deploy and every 6 s for about 3.5 min after it, the following were sampled:
  - threads and CPU ticks of PID 1 on both centrals
  - `/health/ready` latency
  - `Client.CLI read` of `okvt` and `spinvt` on 4840

An earlier attempt used a timer-only trigger (`TimerIntervalMs=1000`, no dependency), and the script never ran; see
observation 4.

**Quarantine.**

- The first run started at about 02:01:16. The following triggers were refused, as designed: refusals do not count as
  strikes.
- At 02:01:36, after 20.2 s, which is past `StuckOrphanQuarantineFactor` 10 × the 2 s budget, both centrals logged
  this Error to `scripts-*.log`: `Script gateE-vt-spin quarantined: its previous evaluation has been running for
  20.2 s, over 10x its 2000 ms budget … not evaluated again until its source changes or it is redeployed`.
- `/metrics`:
  - `otopcua_script_quarantined_total{reason="stuck",script="gateE-vt-spin"} 1`
  - `otopcua_script_run_refused_total{script="gateE-vt-spin"} 21`
  - Both stayed at 1 and 21 for the rest of the run, so evaluation stopped after the quarantine.
- OPC UA status of `ns=3;s=EQ-32296cc68964/spinvt`: `BadWaitingForInitialData` until the first run, then
  `BadInternalError` (0x80020000) from 02:01:23 onward.

**Responsiveness.**

| metric | before the deploy | over the 3.5 min after |
|---|---|---|
| `/health/ready` | 200 in 3–20 ms | 200 in 3–11 ms |
| `Client.CLI read` (connect + read) | 0.34–0.46 s | 0.33–0.49 s |

`okvt`, the healthy sibling, kept evaluating Good throughout (30656 → 41368).

**Threads** (central-1 / central-2):

| sample | central-1 | central-2 |
|---|---|---|
| before | 50–53 | 49–50 |
| 02:01:17 – 02:04:42 (3.5 min, sampled every 6–9 s) | 50–54 | 50–54 |

There was no growth, and at most one thread was orphaned per node.

**CPU.** The orphaned run cannot be pre-empted in-process. Both nodes went from about 0.1 core to about 1.04 cores
(about 104 ticks/s), and that continued after the virtual tag was deleted and redeployed (1022 ticks per 10 s at
02:05). Only a process restart cleared it, so both centrals were rolling-restarted after the gate. This is the
documented limitation: the out-of-process runner is #750. The gate criteria (responsive, quarantined, threads flat)
hold.

### Observations (none blocks a gate)

1. **The status API does not say why a deployment timed out.** In the normal timeout branch (some acks landed),
   `GET /api/deployments/{id}` returns `status: TimedOut` with `failureReason: null`. Only the central log names the
   silent node (`No ack from: site-b-1:4053`). The no-expected-rows branch does set a reason. An automation client
   polling the API cannot see which node to fix.
   **Resolved (lmxopcua#784):** the timeout branch now persists `failureReason` = `Timed out waiting for acks
   (n/m landed). No ack from: <nodes>. …` (at most 20 nodes named, the rest counted).
2. **Rig: mesh PSK warnings on the centrals.** central logs `Mesh PSK secret ot-mesh-psk-{main,site-a,site-b} … is
   absent (or blank) in the store; the key is REVOKED — commands to that cluster and acks from it are refused`. Yet
   every gRPC dispatch and ack in these runs succeeded. Either the rig's PSK secrets are not seeded and enforcement is
   off, or the warning is stale. Not investigated.
   **Resolved (lmxopcua#784): the warning was false, not a bypass.** The rig's centrals supply all three keys
   through `MeshTransport__ClusterPsks__{MAIN,SITE-A,SITE-B}` (each equal to that cluster's nodes'
   `MeshTransport__GrpcPsk`) and seed nothing in the store. `MeshPskProvider.TryGetPsk` consults that override
   first on every lookup, so enforcement was on and every call authenticated against the override key; only
   the background `RefreshAsync` read the store for those clusters anyway and reported them revoked. The
   refresh now skips override-covered clusters, so `REVOKED` is logged only for a cluster with no key at all.
3. **A surgical raw-tag add respawns that driver.** The driver's spawn spec carries its tag set, so adding one tag
   reconnects the whole driver (the accepted cost noted in `DriverSpawnPlanner`). On a live subscriber this shows as
   one `BadNoCommunication` sample (< 1 ms) on every value of that driver, including UNS references and virtual tags
   fed by it. ScadaBridge-style clients that alarm on a single Bad sample would see it on every tag add.
4. **A timer-only virtual tag never evaluates, and nothing flags it.** `VirtualTag.TimerIntervalMs` is not consumed
   by the runtime: `VirtualTagActor` evaluates on dependency changes only, and no timer scheduler exists. A virtual
   tag with `TimerIntervalMs=1000` and no `ctx.GetTag` dependency deployed cleanly and sat at
   `BadWaitingForInitialData` indefinitely. The `CalculationNoEffectiveTrigger` DraftValidator rule (E4) covers
   Calculation tags only, not virtual tags.
5. **The quarantine line in the main log has no script name.** The guard's Error is mirrored into the main log by
   `ScriptLogCompanionSink` as `WRN [Script] unknown emitted Error: Script "gateE-vt-spin" quarantined: …`. The guard's
   logger carries no `ScriptName`/`ScriptId` property, so the mirror labels it `unknown`, and it is rate-limited under
   that shared key. `scripts-*.log` has the correct `[ERR]` line.
   **Resolved (lmxopcua#784):** the guard binds the script key as `ScriptName`, so the mirror names the script
   and rate-limits each one separately.
6. **The quarantine shows as `BadInternalError` on OPC UA.** A client sees no quarantine-specific status. The reason
   is only in the logs and metrics.
   **Decided (lmxopcua#784): kept `BadInternalError`**, the documented code for any script-attributable fault
   on all three seams; `BadOutOfService` would read as a deliberately disabled point and would be a
   ScadaBridge-visible contract change. Rationale: `docs/VirtualTags.md` §"Quarantine: what a client sees".

### Rig state left behind

- All six nodes run `otopcua-host:dev` `sha256:24f9d7287ed2` (commit `1b9c36f3`).
- Every gate edit was reverted in the ConfigDb: `HR101` deleted; `e1` name restored; `gateE-spin`/`spinvt` deleted;
  both driver configs restored. A clean-up deploy, `cd41a890`, sealed 6 of 6.
- Both centrals were restarted afterwards to clear the gate-E orphan thread.

## Driver fixture gates + H1 (2026-11-09, commit 1b9c36f3, fixture host 10.100.0.35)

### Fixture gates, lmxopcua#761 (driver fixtures + H1)

Date: 2026-11-09. Commit tested: `1b9c36f3` (origin/master), worktree `scratchpad/wt/otopc-fix` (branch w18/fixtures, nothing committed). Release builds, 0 errors. Nothing posted to Gitea.

Fixture host 10.100.0.35: `lmxopcua-fix` does not exist on macOS (CLAUDE.md says to use ssh). Every needed fixture was already running, so nothing was started or stopped there. State before and after was identical (modbus-standard, ab-server-controllogix, python-snap7-s7_1500, opc-plc, mosquitto + sparkplug-sim + mqtt-publisher, sql-pg/oracle/blackhole, otopcua-mssql). Pre-existing: `otopcua-opc-plc` reports "unhealthy" (before and after; the opcuaclient suite still passed). `otopcua-sql-blackhole` showed "unhealthy" briefly right after the blackhole pause/unpause test, then "healthy" again.

## Driver fixture gates

| Gate | Command (from worktree) | Result |
|---|---|---|
| modbus standard | `dotnet test tests/Drivers/*Modbus.IntegrationTests -c Release` (default endpoint 10.100.0.35:5020, profile standard) | PASS: 12 passed, 0 failed, 32 skipped. Skips are the other profiles only (exception_injection 11, mitsubishi 7, dl205 ~11, s7_1500 4). They are not valid on the standard profile. Standard round-trip and AddressingGrammar ran. |
| s7 s7_1500 | `S7_SIM_ALLOW_WRITES=1 dotnet test tests/Drivers/*S7.IntegrationTests` | PASS: 23 passed, 0 failed, 3 skipped. Without the writes env var: 20 passed and 6 skipped. The 3 remaining skips: TM/CT address limit of python-snap7, and two destructive tests (`S7_RECONNECT_BOUNCE_CMD`, `S7_TIMEOUT_OUTAGE_*_CMD`) not run so the shared sim is not bounced. |
| abcip controllogix | `AB_SERVER_ALLOW_WRITES=1 AB_DIFF_ENDPOINT=10.100.0.35:44818 dotnet test tests/Drivers/*AbCip.IntegrationTests` | PASS with one flake: 33 passed, 1 failed, 22 skipped (about 2 min). Failure: `PollLatencyComparisonTests.Native_poll_p95_is_within_ten_percent_of_libplctag` raised `DifferentialAbortedException` ("Connection reset by peer" on tag UdtSeg.Ready). Rerun of that class: 2/2 passed. Verdict: flake (ab_server connection reset, aborts on first connection-level failure by design). Skips: 6 need `AB_SERVER_PROFILE=emulate` (Logix Emulate), 1 needs `AB_HW_WRITE_GATEWAY` (hardware), 15 are the remaining differential and write tests. Without writes env: 28 passed, 0 failed, 28 skipped. |
| opcuaclient opc-plc | `dotnet test tests/Drivers/*OpcUaClient.IntegrationTests` (opc.tcp://10.100.0.35:50000) | PASS: 5 passed, 0 skipped. |
| mqtt + sparkplug | `MQTT_FIXTURE_ENDPOINT=10.100.0.35:8883 MQTT_FIXTURE_PLAIN_ENDPOINT=10.100.0.35:1883 MQTT_FIXTURE_USERNAME=otopcua MQTT_FIXTURE_PASSWORD=<from publisher container env> MQTT_FIXTURE_CA_CERT=<scp'd ca.crt> dotnet test tests/Drivers/*Mqtt.IntegrationTests` | PASS: 30 passed, 0 skipped (includes Sparkplug, simulator running). |
| sql | `SQL_TEST_ENDPOINT=10.100.0.35,14330 SQL_TEST_PASSWORD=.. SQL_PG_ENDPOINT=10.100.0.35:15432 SQL_PG_PASSWORD=.. SQL_ORACLE_ENDPOINT=10.100.0.35:11521 SQL_ORACLE_PASSWORD=.. SQL_BLACKHOLE_ENDPOINT=10.100.0.35,14333 SQL_BLACKHOLE_PASSWORD=.. SQL_BLACKHOLE_DOCKER_SSH=dohertj2@10.100.0.35 dotnet test tests/Drivers/*Sql.IntegrationTests` | PASS: 37 passed, 0 failed, 17 skipped. SQL Server, Postgres, Oracle and the blackhole timeout test ran. Skips: 14 MySQL (no MySQL fixture on the host) and 3 ODBC (no `SQL_ODBC_CONNECTION_STRING`). |

Side effect: the SQL suite creates (and never drops, by design) database `SqlPollFixture` on `otopcua-mssql`. It was already there before (listed in sys.databases).

## H1 (#736) historian year-window gate

There was no pre-existing year-window fixture test. The existing fixture suites test the cap at 3 rows, and the 10k fuzz is offline in `OpcUaServer.Tests`. I added an uncommitted probe, `tests/Drivers/Historian.Conformance/YearWindowGateProbe.cs`, with a subclass in each backend's integration project (edits are only in the throwaway worktree). It seeds one tag with 52,560 samples spanning 365 days, with `maxRawValuesPerRead` = 10,000.

- It checks that an N=0 read of the whole year is refused with `HistoryReadLimitExceededException` (Limit=10000), not returned unbounded and not truncated.
- It walks the window in 10k-capped pages, resuming after the last timestamp, and checks every page is at most 10000.
- It checks the concatenated pages equal the seeded set (no loss, no duplicates, ordered), and that peak process working set stays under 500 MB.

| Backend | Where | Result |
|---|---|---|
| QuestDb | local throwaway container `h1-gate-questdb` (`questdb/questdb:10.0.1`, ports 28812/29000; removed afterwards) | PASS: N=0 refused, limit 10000; 6 pages, max page 10000, 52,560 rows walked, no loss or duplicates; peak working set 122 MB (managed 27 MB). Full QuestDb integration project: 234 passed, 0 failed, 1 skipped (opt-in `HISTORIAN_QUESTDB_STRESS_SECONDS` stress test). |
| SqlServer | throwaway DB `OtOpcUaHistorian_T_*` on `otopcua-mssql` (the fixture creates and drops it; no orphan left) | PASS: same numbers. 6 pages, max 10000, 52,560 rows walked; peak working set 114 MB (managed 18 MB). Full SqlServer integration project: 196 passed, 0 failed, 0 skipped. |
| Server half: continuation point at N=0 | offline: `dotnet test tests/Server/*OpcUaServer.Tests --filter RawPageLimit/HistoryRead/MaxReturnDataValues/HistoryServerCapabilities` | PASS: 431 passed, 0 failed (includes the paged-vs-unpaged raw cap fuzz and the MaxReturnDataValues advertisement). |

Scope caveat: the probe drives the backend `IHistorianDataSource` directly, not a live OPC UA HistoryRead through the server node manager. The backend side is proven live: bounded at 10k, N=0 refused, pages complete. The continuation point itself is the server's `RawPageLimit` planning, covered by the offline tests above, not by a live OPC UA session. A true end-to-end gate (OPC UA client reading a year at N=0 against a server on the SqlServer/QuestDb backend) needs the docker-dev rig, which I was told not to use.

Mishap: two aborted SQL Server probe attempts (row-by-row seeding was too slow, so I killed them) each left a throwaway `OtOpcUaHistorian_T_*` database on `otopcua-mssql`. Both were dropped manually. The final check shows no `OtOpcUaHistorian*` databases remain.

## NOT RUN

- Hardware userGates (real PLCs): AbCip / AbLegacy / TwinCAT array writes, FOCAS `MACRO:` > 32767, AbCip probe-typo recovery.
- docker-dev gates A, B, C, D, E, and the Galaxy gate F: not in scope. The otopcua-dev rig is off-limits, and F needs mxaccessgw.
- Modbus exception_injection / dl205 / mitsubishi / s7_1500 profiles (only `standard` requested); S7 reconnect-bounce and connect-timeout-outage destructive tests; AbCip Logix Emulate profile; Sql MySQL and ODBC (no fixture).

## Cleanup

Container `h1-gate-questdb` removed. The fixture host is unchanged. The worktree `scratchpad/wt/otopc-fix` (branch w18/fixtures) is left in place with uncommitted probe edits. Remove it with `git worktree remove --force` if not wanted.

## Gate D (#745) — AdminUI live pass, docker-dev (2026-11-09) — FAIL (1 of 5 checks)

Overall: four checks PASS; the delete-confirmation check FAILS on four pages (filed as a follow-up issue, see #761). Full record below, as written by the Chrome pass.

Date: 2026-11-09 (container clock 2026-11-10 03:xx UTC). Rig: local docker-dev, otopcua-host:dev image built 2026-11-09 20:35 EST from master 1b9c36f3 (per coordinator; rig built from a scratchpad worktree no longer on disk, so commit not re-verifiable from the container). AdminUI http://localhost:9200 (login disabled, session "multi-role-test"). Wonder boxes untouched. No deploy performed.

### 1. 100 KB script survives save + reload - PASS
Procedure: /scripts/new, set Monaco model to 102,492 chars (1,200+ comment lines + `return 1;`), name gateD-100kb-test, Create. Reloaded /scripts/SC-d82b3a36fd85; editor text === regenerated original (identical, length 102492 both). Then deleted it (own modal).

### 2. Two-tab /hosts CPU flat with ~50 drivers - PASS
Rig has 52 drivers on cluster MAIN (48 int3b-mb-*, blackhole, Calc, opcplc, pymodbus); site-a/b have none. `docker stats --no-stream` on central-1/central-2 (the AdminUI hosts), 8 samples ~12s apart (~100s) per condition:
- no /hosts tab (on /):   central-1 11.0%, central-2 10.9% avg
- 1 /hosts tab (first run): c1 9.7%, c2 11.4%
- 2 /hosts tabs:          c1 10.9%, c2 12.1% avg
Within noise (range 8-17% in every condition). Flat. Raw: scratchpad/cpu-*.txt.

### 3. 5k-tag device renders < 2 s - PASS
No such device existed. Created disabled-in-effect (never deployed) Modbus driver gateD-bulk/device gdev under docker-dev, imported 5000-row CSV via the Raw tree modal (parse+review 206 ms, commit 717 ms). Expanding gdev: first tag rows in DOM 14-20 ms after click, painted (double rAF) 44 ms. Tree paged: 500 rendered ("Show 500 more (4500 not shown)") and DOM ~611 nodes. Far under 2 s.
Cleanup: tags cannot be bulk-deleted in UI (device delete: "Delete blocked - contains tags"), so removed gateD-bulk rows (5000 Tag, TagGroup, Device, DriverInstance) by scoped SQL delete in a transaction on the rig OtOpcUa DB (needed `sqlcmd -I`). Verified 0 gateD DriverInstances, Tag total back to 55.
Observation (not a defect): no bulk "delete all tags in device" in Raw tree.

### 4. Every delete asks for confirmation - FAIL (4 defects)
window.confirm overridden; no native confirm ever fired. Own modal confirmed + Cancel verified (item survives) for: script delete (ScriptEdit), API key delete (/apikeys), all 10 UNS-tree deletes on /uns (enterprise, clusters, area, line, equipment), Raw-tree driver delete and device delete (modals). Equipment-page virtual-tag/alarm/reference deletes: source uses ConfirmDeleteVirtualTag/ConfirmDeleteAlarm/ConfirmRemoveReference; no live rows on the sampled equipment (NOT exercised live). Certificates delete uses RequestAction -> Confirm panel (source).
DEFECTS - delete executes immediately on click, no confirmation:
 a) /role-grants Delete (RoleGrants.razor:68 -> DeleteAsync(r.Id)). LIVE-VERIFIED: added test grant cn=gated-test..., clicked Delete, row gone instantly, no modal. (Test grant thus already removed.)
 b) ClusterEdit.razor:102 `@onclick="DeleteAsync"` (cluster delete) - source only
 c) NodeEdit.razor:119 `@onclick="DeleteAsync"` (cluster node delete) - source only
 d) AclEdit.razor:151 `@onclick="DeleteAsync"` (node ACL delete) - source only
 (CollectionEditor.razor:31 per-row Delete in driver forms is a working-copy edit committed on Save - not counted.)
Repro for (a): http://localhost:9200/role-grants, add any group, press its Delete.

### 5. A6 diagnostics panel shows real counts - PASS
/fleet/diagnostics?node=central-1:4053 : DRIVERS 52, FAULTED DEVICES 0, CONFIG SOURCE live, REVISION e221b1792cbd..., per-driver rows (48 HEALTHY int3b-mb-* with 1 connected device each, Calc/opcplc/pymodbus HEALTHY, int3b-blackhole INITIALIZING 0 connected). /hosts shows the same 52 driver instances and blackhole INITIALIZING "1 / 1 DOWN". Disabled int3b-bulk correctly absent. Minor: blackhole's down device is not counted in "FAULTED DEVICES 0" (state INITIALIZING, not faulted) - arguably fine, flag for A6 owner.

### Revert state
Test script, test role grant, gateD-bulk driver/device/5000 tags all removed; no deploys; browser tab closed. Rig config as found.

### Still not run after gate D

- Gate F (Galaxy) — run 2026-11-11, see "Gate F (#735)" below.
- Hardware userGates (real PLCs), as listed under NOT RUN above.

## Gate F (#735) — 20 config deploys keep `mxgateway.sessions.open` flat (docker-dev + VD03 DEV mxaccessgw, 2026-11-11) — PASS

| | |
|---|---|
| Commit tested | `5ac55205` (master). Image `otopcua-host:dev` `sha256:47ca0da6eaf7` rebuilt from that clean checkout before the run; all six nodes recreated from it. |
| Gateway | DEV mxaccessgw on `wonder-app-vd03.zmr.zimmer.com` (gRPC h2c `:5120`, assembly `0.1.2.0`, main `2b13564e`). Metrics from its anonymous Prometheus endpoint `http://wonder-app-vd03.zmr.zimmer.com:5130/metrics` (`mxgateway_sessions_open`, `…_sessions_opened_total`, `…_sessions_closed_total`, `…_workers_running`, `…_workers_killed_total`, `…_faults_total`), scraped every 15 s from the Mac for the whole run. Read-only Galaxy access, owner-approved. No remote command ran on any wonder box. |
| Reachability | The docker-dev containers resolved and reached `wonder-app-vd03:5120` through the Mac's VPN route (`/dev/tcp` check from `central-1`) before anything was configured. |
| Driver | `gatef-gal`, DriverType `GalaxyMxGateway`, cluster MAIN, seeded straight into the ConfigDb: `gateway.endpoint` = the VD03 gateway, `useTls=false`, `apiKeySecretRef=env:GALAXY_MXGW_API_KEY`, `mxAccess.clientName=OtOpcUa-761-gateF`, `repository.watchDeployEvents=false`, no `alarmAck` fallback. Device `vd03` (`{}`); two **Read**-only tags: `ScanState` (`DevWinPlatform.ScanState`, Boolean) and `CPULoad` (`DevWinPlatform.CPULoad`, Float64). No writes, no alarm subscriptions, no acks. |
| Secret | The VD03 gateway key reached the containers only as the `GALAXY_MXGW_API_KEY` runtime env (exported from the owner's env file into the `docker compose up` shell). It is not in the repo, a compose file, the ConfigDb or this record. |
| Respawn trigger | As in gate A: each iteration edits the driver's config (`JSON_MODIFY` of `mxAccess.publishingIntervalMs` = 1000 + i), POSTs `/api/deployments`, waits for it to seal, waits 15 s, then scrapes the gateway. A config change is a respawn (#665), so every iteration stops and disposes the old `GalaxyDriver` and spawns a new one on **both** `central-1` and `central-2` (MAIN runs its drivers on both nodes) — 40 driver respawns in total. |

**Attribution.** The gateway's metrics have no per-client label (the session table with client names is on the
login-gated dashboard, which was not used), so the rig's share is measured as the **delta** against a baseline
taken before the driver existed, cross-checked against the `opened`/`closed` counters. During the whole window the
other consumers' sessions did not churn: `opened − closed` moved only by what the rig did.

**Gateway samples** (UTC):

| when | open | opened | closed | workers running |
|---|---|---|---|---|
| baseline, 13 samples 23:11:15–23:14:11 (no Galaxy driver on the rig) | 2 | 3 | 1 | 2 |
| driver added + deployed (23:14:34) | **4** (+2: one session per central) | 5 | 1 | 4 |
| iterations 1–20, 15 s after each seal (23:15:13–23:20:56) | 4 in 13, 3 in 4 (it. 5, 6, 11, 16), 2 in 3 (it. 1, 17, 18); **never above 4** | 5 → 42 | 1 → 38 | 2–4 |
| settled, 23:21:04 / 23:21:09 / 23:21:25 | **4** | 42 | 38 | 4 |

The rig opened 39 sessions over the run (42 − 3) and closed 37 (38 − 1); the net +2 is exactly its two live
drivers. The open gauge returned to its steady value (baseline + 2) after the 20 deploys, so no session leaked.
The dips to 2–3 are samples that landed while a respawned driver was still reconnecting (see the init timeouts
below), not lost sessions; every iteration ended with both drivers connected.

Gateway-side counters over the run: `workers_exited{worker-shutdown-ack}` 1 → 37 (graceful closes),
`workers_killed{OpenSessionFailed}` 0 → 13, `workers_killed{client-close}` 0 → 1, `faults{WorkerFaulted}` 0 → 14,
`workers_exited{WorkerFaulted}` 0 → 1, `grpc_streams_disconnected{ClientCancelled}` 31.

**Rig side** (per central, 23:14:56–23:21:10): 20 `spawned GalaxyMxGateway driver gatef-gal`, 20
`stopped driver child`, 20 `disposed GalaxyMxGateway driver on stop`, 25 `GalaxyDriver gatef-gal shutting down`
(20 respawns + 5 failed-init teardowns), 15 `GalaxyMxSession connected`, 5 `initialize failed: … timeout of
'00:00:05'`. central-2 logged one `GalaxyMxSession close failed or timed out (best-effort) — dropping the channel`.
No error or fault otherwise.

**Driver health after the run** (`/hosts`, `/fleet/diagnostics`): `gatef-gal` on central-1 and central-2
**Healthy**, 2 tags, host status "2 / 6 down" (the probe watcher seeded 5 platforms; two Galaxy platforms on VD03
DEV report down, unrelated to the gate). `Client.CLI read` of both tags on 4840 and 4841 was Good before the loop
(`ScanState` True, `CPULoad` ≈ 13.4) and after it (`CPULoad` 15.63, Good).

### Removal and the final drop — not observed (VPN dropped)

The Mac's VPN route to VD03 went down at about 23:21:35, ten seconds after the settled samples. DNS for
`*.zmr.zimmer.com` and the `10.220.x` route both disappeared; it was still down at 23:40. The rig's Galaxy
drivers lost the gateway (reconnect supervisor backing off, Initialize breaker opened), which is the expected
behaviour.

`gatef-gal` (driver, device, 2 tags) was deleted from the ConfigDb and the removal deployed at 23:30:56
(`12d5ff59`, Sealed). Both centrals logged `stopped driver child gatef-gal` and `disposed GalaxyMxGateway driver
on stop`. Because the gateway was unreachable, the rig could not close its two sessions; the gateway reaps a
session whose client stream is gone on its detach-grace / lease sweep. **The return of `sessions.open` to 2 was
therefore not observed.** Re-check `mxgateway_sessions_open` once the VPN is back: it should read the baseline
(2, if VD03's own consumers are unchanged).

### Observations (none blocks the gate)

1. **A Galaxy (re)start often exceeds the 5 s Tier-A `Initialize` timeout against VD03.** 10 of 40 respawns
   (5 per node) failed `initialize` at exactly 5 s. Opening a gateway session spawns an x86 MXAccess worker, and
   a successful connect took 4–5 s from spawn to `connected`, right at the limit. Each timeout cost a 10 s reconnect-timer retry
   (BadNoCommunication on that driver's nodes meanwhile) and shows on the gateway as an `OpenSessionFailed`
   worker kill plus a `WorkerFaulted` fault. The gateway cleaned up every one (the gauge stayed flat), so it is a
   latency and noise issue, not a leak. Galaxy probably wants a longer `Initialize` timeout (tier or per-instance
   `ResilienceConfig`). Not filed (no Gitea per brief).
2. **The VD03 gateway's `faults_total` is now 14 (`WorkerFaulted`) from this gate** — anyone reading the VD03
   dashboard after 2026-11-11 23:15Z should attribute those, and the 13 `OpenSessionFailed` kills, to this run.
3. The rig's opens (39 at the gateway) exceed its logged `GalaxyMxSession connected` lines (30 in the loop plus 2
   on add); the extra opens are ones that completed after the client's 5 s init timeout had already given up.
   They were closed by the end of the window (opened − closed = 2 at 23:21).

### Rig state left behind

- `gatef-gal` and its device/tags are deleted from the ConfigDb; the removal deploy sealed. The gate's
  `Deployment` rows carry only the `env:` reference, no secret.
- All host nodes were recreated without the exported key, so `GALAXY_MXGW_API_KEY` is back to the committed
  docker-dev default. The rig runs `otopcua-host:dev` `sha256:47ca0da6eaf7` (`5ac55205`).

## Re-run on current master — all runnable gates (2026-11-12)

| | |
|---|---|
| Commit tested | `54c4357f` = `master` `412da27c` + the deploy-API fix below (landed on master as `fa8c3d1a` after a rebase over two unrelated commits; same diff). Image `otopcua-host:g761` `sha256:24432d7fbc2d`, built from that commit, on all six nodes. |
| Rig | docker-dev as its **own compose project** `otopcua-g761` (an overlay remapping every host port and image tag), so the shared `otopcua-dev` rig that someone else was running was not touched. The ConfigDb was restored from a `COPY_ONLY` backup of the `otopcua-dev` ConfigDb (the same MAIN-modbus / MAIN-opcua / equipment `EQ-32296cc68964` + `okvt` config the 11-09 run used), then migrated (no pending migrations). Host ports: OPC UA 24840–24845, AdminUI 29200, per-node HTTP 29100–29105. |
| Clock | Rig UTC 2026-11-13 03:05–03:44 (= 2026-11-12 evening US Eastern). |
| Load | The 48 `int3b-mb-*` and the `int3b-blackhole` drivers were **disabled** after gate A. With them on, two rigs' worth of Modbus drivers saturated the shared pymodbus sim on 10.100.0.35 (about 117–136 % CPU) and `HR100` read `BadCommunicationError`. Gate D's 50-driver checks were not re-run (see D). |

**Summary: A, B, C, D (check 4), E, H1 (live), the F removal check and every fixture lane PASS. One product defect
found and fixed (deploy API). Hardware userGates still open.**

### Defect found and fixed: a bodyless `POST /api/deployments` answered 404 — FIXED (`fa8c3d1a`)

The documented headless deploy (`docker-dev/deploy.sh`, README "Headless deploy") returned **404** on both centrals,
while the same POST with a JSON body was accepted. The optional body was an inferred `DeployRequest?` parameter,
which stamps `Accepts(application/json)` metadata on the endpoint. For a request with no `Content-Type` the routing
`AcceptsMatcherPolicy` prefers endpoints that declare no consumes metadata. On a fused admin+driver host
`MapOtOpcUaMeshGrpcServices` has mapped grpc-dotnet's `{unimplementedService}/{unimplementedMethod}` catch-all, so
the request was routed away from the deploy handler and got a 404. The deploy API on its own does not reproduce
it. The handler now reads the optional body itself: no body is the headless form, a non-JSON body answers 415 and
malformed JSON 400. `DeployApiBodylessPostTests` (Host.IntegrationTests) composes the deploy API with the mesh gRPC
services and drives real HTTP; with the fix reverted both bodyless cases fail with 404. Live on the rebuilt rig, a
bodyless POST through Traefik answered **202** (every deploy below used it).

### Gate A (#731) — PASS, now with `dotnet-counters`

Method as on 11-09 (each iteration edits `MAIN-modbus` `TimeoutMs` and `MAIN-opcua` `SessionName`, deploys, waits for
the seal + 8 s), 2 warm-ups + 20 iterations, every deploy sealed in 1–2 s. This time `dotnet-counters collect -p 1
--counters System.Runtime --refresh-interval 5` (the linux-arm64 single-file tool copied into each central) ran for
the whole loop, alongside `/proc/1` threads / fds / RSS.

| | central-1 | central-2 |
|---|---|---|
| `disposed Modbus driver on stop` / `disposed OpcUaClient driver on stop` | 20 / 20 | 20 / 20 |
| `spawned … MAIN-modbus` / `MAIN-opcua` | 20 / 20 | 20 / 20 |
| dispose/shutdown timeout or throw | 0 | 0 |
| threads (baseline → range over 20 → settled 75 s later) | 54 → 49–57 → 48 | 53 → 48–56 → 46 |
| open fds | 739 → 691–739 → 715 | 739 → 704–740 → 719 |
| `dotnet.assembly.count` (57 / 54 samples) | 312, constant | 314, constant |
| `dotnet.timer.count` first-quarter vs last-quarter mean | 139 → 137 | 144 → 145 |
| `dotnet.thread_pool.thread.count` (UpDownCounter deltas) | net ≈ 0 | net ≈ 0 |
| working set first-quarter vs last-quarter mean | 520 → 505 MB | 522 → 502 MB |

No upward trend in any series; nothing (assemblies, timers, threads, fds) accumulates per respawn. Raw CSVs:
`counters-central-{1,2}.csv` in the run's evidence (not committed).

### Gate B (#732) — PASS

Same procedure as 11-09 on `opc.tcp://localhost:24840`, session 1 held 120 s on `ns=2;s=pymodbus/plc/HR100` and
`ns=3;s=EQ-32296cc68964/okvt`.

- **Surgical add** (`HR101`, 03:14:27): both centrals logged `kind=PureAdd, added=1, … rebuild=False`. Each
  subscription took one `BadNoCommunication` sample at 03:14:27 and was Good again in the same second (the accepted
  driver respawn), then 21 / 22 Good updates until the rebuild. `HR101` itself reads `BadOutOfRange` (the fixture
  refuses address 101, as on 11-09).
- **Rebuild** (rename `e1` → `e1r`, 03:14:49): `kind=Rebuild, … rebuild=True`; session 1's items got
  `BadNodeIdUnknown` and nothing after.
- **Re-created after the rebuild** (session 2, 20 s): HR100 17 updates, okvt 16, **all Good**, none Bad.

### Gate C (#733) — PASS

- **C-a, kill `central-1` mid-dispatch.** `site-b-1` paused, deployment `56e5ede1` at `AwaitingApplyAcks` 5/6, `docker
  kill central-1` at 03:16:37. A second POST while in flight: **409**. After unpausing `site-b-1` the held
  deployment **sealed 6/6** (on 11-09 it timed out). After `central-1` restarted, the next deploy `6df32531`
  **sealed 6/6**.
- **C-b, kill the singleton host.** `central-2` (the only `/health/active` 200 on MAIN) killed at 03:17:30 with
  `548ccbb9` in flight (5/6). `central-1` auto-downed it at 03:17:58, the singletons moved
  (`Younger -> BecomingOldest`), the AdminUI was back through Traefik at 03:18:02, and at 03:18:14 `central-1`
  logged `Coordinator recovered in-flight deployment 548ccbb9… (5/6 acks landed)`. It closed `TimedOut` with
  `failureReason` = `Timed out waiting for acks (5/6 landed). No ack from: site-b-1:4053. …` (the #784 fix,
  verified live). After `central-2` restarted, the next deploy `246594e0` **sealed 6/6**.
- **C-port.** As on 11-09, a replacement container from the compose service with a `perl` listener holding
  `0.0.0.0:4840` before `dotnet OtOpcUa.Host.dll` starts (the 4840 socket inode belongs to perl, none to dotnet):
  - fused admin+driver (`central-2`): `GET /health/ready` **503**, `opcua-endpoint: Unhealthy — OPC UA SDK failed to
    start: Failed to establish tcp listener sockets on port 4840 for Ipv4 and IPv6.`; `configdb`/`akka` Healthy.
  - driver-only (`site-a-2`): `FTL OtOpcUaServerHostedService: SDK start failed on a driver-only node …`, **exit 1**.
  - So the C3 behaviour holds on OPC UA SDK 2.0.1. (A first attempt was invalid: the Perl one-liner did not keep the
    socket in a variable, so Perl closed it at once and the SDK bound 4840 normally. Re-run with the socket held.)
- **Side effect worth knowing:** the 39 s `docker pause` of `site-b-1` in C-b outlived the 15 s auto-down, so
  `site-b-2` downed it and the SITE-B pair came back as **two 1-member clusters, both Active**, until `site-b-2`
  was restarted. That is the documented auto-down trade (`docs/Redundancy.md`, restart one side after the link
  heals), not a defect, but every C-style drill that pauses a node past 15 s leaves this behind.

### Gate D (#745) — check 4 re-run after #785: PASS

Checks 1, 2, 3 and 5 passed on 11-09 and their code is unchanged. Check 4, the one that failed, was re-run with
Playwright against `http://localhost:29200` (login disabled), with a native-`confirm` listener armed (it never
fired):

| page | Delete click | Cancel | Confirm |
|---|---|---|---|
| `/role-grants` (throwaway grant `cn=g761-gateD,…`) | modal "Delete role grant" naming the group; row still in DB | row kept | row deleted (0) |
| `/clusters/SITE-B/acls/g761-acl` (throwaway ACL) | modal "Delete ACL"; row still in DB | row kept | row deleted (0) |
| `/clusters/SITE-B/edit` | modal "Delete cluster 'SITE-B'? This cannot be undone." | modal closed, cluster kept | not confirmed (real row) |
| `/clusters/SITE-B/nodes/site-b-2:4053` | modal "Delete node 'site-b-2:4053' from cluster 'SITE-B'? …" | modal closed, node kept | not confirmed (real row) |

### Gate E (#734) — PASS

`gateE-spin` (`var v = ctx.GetTag("pymodbus/plc/HR100").Value; while (true) { }`) as change-triggered virtual tag
`spinvt` on `EQ-32296cc68964`, deployed 03:27:21, sampled every ~4 s for 1 min 50 s:

- **Quarantined** on both centrals at 03:27:42 (21.4 s / 22.3 s, over 10 × the 2 s budget); `/metrics`
  `otopcua_script_quarantined_total{reason="stuck",script="gateE-vt-spin"} 1`, `run_refused_total` 8 / 11. The
  main-log mirror now names the script (`[Script] gateE-vt-spin emitted Error …`, the #784 fix).
- **Responsive:** `/health/ready` 200 in 2–6 ms throughout (2.9–3.5 ms before); `okvt` kept reading Good, connect +
  read 0.40–0.48 s (0.41–0.52 s before); `spinvt` `BadInternalError`.
- **Threads flat:** central-1 42–44, central-2 41–45 (41–42 before).
- CPU: both centrals went to ~1 core (≈100 ticks/s) until restarted after the gate — the known in-process limit
  (#750; the out-of-process runner is the fix).

### H1 (#736) — live end-to-end over OPC UA: PASS

This closes the 11-09 scope caveat (that run drove the backend directly). `ServerHistorian:Backend=SqlServer` on
the centrals (DB `OtOpcUaHistorian` on the rig SQL), 52,560 samples (every 10 min, 2025-11-01 → 2026-10-31) seeded
for `HR300` (`lg9.long`). A probe client (OPC UA SDK 2.0.1) sent `HistoryRead` raw with **`NumValuesPerNode = 0`**
for the whole year on `opc.tcp://localhost:24840` and followed every continuation point:

| page | values | continuation point |
|---|---|---|
| 1 | 9,999 | 16 bytes |
| 2–5 | 9,998 each | 16 bytes |
| 6 | 2,569 | none |

52,560 values in total (= the seeded set: first 0, last 52559), timestamps strictly ascending, no duplicates,
0.2 s. central-1 GC heap after the read: gen2 48 MB + LOH 56 MB + gen1 7 MB (≈ 112 MB), committed 170 MB — under the
500 MB bound. QuestDb was not re-run live: the paging is the backend-agnostic node-manager `RawPageLimit` plan, and
the QuestDb backend half passed on 11-09.

### Gate F (#735) — the removal drop, now observed: PASS

The 11-11 run could not see `mxgateway_sessions_open` return to baseline after removing the driver (the VPN
dropped). The VD03 gateway has since restarted (its counters reset), so this was re-checked directly: a read-only
`GalaxyMxGateway` driver `g761-gal` (one `Read` tag, `DevWinPlatform.ScanState`; key only as
`env:GALAXY_MXGW_API_KEY`, sourced into the compose shell from the owner's env file and never written anywhere) was
added to MAIN and then removed. Gateway `/metrics` (anonymous, scraped from the Mac):

| when (UTC) | open | opened_total | closed_total | workers |
|---|---|---|---|---|
| baseline 03:40:49–03:41:19 | 2 | 2 | – | 2 |
| driver deployed 03:41:47 (both centrals `connected`, ScanState read Good) | **4** | 4 | – | 4 |
| driver removed 03:42:58 → 03:43:13 … 03:43:59 | **2** | 4 | 2 | 2 |

Both centrals logged `stopped driver child g761-gal` + `disposed GalaxyMxGateway driver on stop`; the gateway
closed both sessions within 15 s. No Galaxy writes, no alarm subscriptions.

### Driver fixture lanes (10.100.0.35) at `412da27c` — PASS

| Suite | Result | Notes |
|---|---|---|
| Modbus standard (:5020) | 12 / 0 / 32 | Two earlier attempts failed 8 then 1 tests on connect timeouts: the shared sim was saturated by this gate's own rig plus `otopcua-dev` (196 connections from this Mac). Clean once the load dropped. Skips are the other profiles. |
| S7 s7_1500 (`S7_SIM_ALLOW_WRITES=1`) | 23 / 0 / 3 | Same 3 skips as 11-09 (TM/CT span limit, the two destructive CMD tests). |
| AbCip controllogix, `AB_SERVER_ALLOW_WRITES=1 AB_DIFF_ENDPOINT=…` | 34 / 0 / 22 | **No write test runs in this combination**: `IntegrationWriteGate` refuses writes while `AB_DIFF_ENDPOINT` is set (read-only differential run). That was also true of the 11-09 "with writes" run. |
| AbCip controllogix, `AB_SERVER_ALLOW_WRITES=1` only | 38 / 0 / 18 | The real write pass: all 10 write tests pass, incl. `AbCipRefusalLiveTests.A_whole_array_write_round_trips` (DINT[16]) and `AbCipUdtMemberWriteTests.A_member_array_element_write_leaves_the_other_elements_alone`, Native + libplctag. |
| OpcUaClient (opc-plc :50000) | 9 / 0 / 0 | One flake on the first try (`ServerHostNameValidationTests…(validate: False)`, `BadRequestInterrupted`); clean rerun. |
| OpcUaClient.Browser (+ `OPCUA_NAT_ENDPOINT=…:50010`) | 12 / 0 / 0 | The NAT fixture is up, so the #806 NAT tests ran too. |
| MQTT + Sparkplug | 30 / 0 / 0 | |
| Sql (SQL Server, PG, Oracle, blackhole) | 58 / 0 / 25 | 21 MySQL + 4 ODBC skips (no fixture). |
| **MTConnect (G1, :5000)** | 21 / 0 / 0 | First run of the G1 MTConnect lane against the shared agent. |

### Hardware userGates — simulator evidence only; real hardware still required

| userGate | Simulator evidence (2026-11-12, `412da27c`) | Real hardware |
|---|---|---|
| AbCip array write | ab_server ControlLogix: whole-array DINT[16] round-trip + UDT member-array element write pass (above) | not run |
| AbLegacy array write | ab_server SLC500 / MicroLogix / PLC-5: `N_file_array_write_round_trips_every_element` passes on all 3 families × 2 transports; AbLegacy suite 51 / 0 / 0 | not run (#752) |
| TwinCAT array + STRING write | **none** — no ADS simulator exists (`elma-ads-tcp/udp` on 10.100.0.35 are socat relays to an absent ADS box); suite 0 / 0 / 18 skipped | not run (#755) |
| FOCAS `MACRO:` > 32767 | `otopcua-focas-fixture` mock (byte-identical to the repo's `focas_mock/server.py`): `WireBackendCoverageTests.Macro_and_parameter_numbers_above_short_range_read_the_authored_variable` (`MACRO:99999`, `PARAM:40000`, with a decoy at the 16-bit-wrapped address) passes; FOCAS suite 18 / 0 / 0 | not run |
| AbCip probe-typo recovery | `AbCipRefusalLiveTests.A_typod_probe_tag_does_not_black_out_a_reachable_device` passes on ab_server, Native + libplctag | not run |

Plan INT-4 marks these "USER-ORDERED — NON-SKIPPABLE … real ControlLogix / SLC / TwinCAT hardware", so the simulator
passes do not close them. They need bench endpoints from the owner, or an explicit owner decision to accept the
simulator evidence for the four that have it (TwinCAT has none).

### Observations

1. The bodyless-deploy 404 (fixed above) had been masked because every earlier gate run sent a JSON body.
2. Two docker-dev rigs on one Mac saturate the shared pymodbus sim; disable the 48 `int3b-mb-*` drivers (they exist
   only for gate D's 50-driver checks) when not running those.

### Rig state left behind

The `otopcua-g761` project (containers, its volumes, the `otopcua-host:g761` / `otopcua-migrator:g761` images),
the `g761-probe` network and the temporary `OtOpcUaHistorian` database (inside the project's SQL volume) were all
removed. The shared `otopcua-dev` rig was only read (one `COPY_ONLY` backup to its container's `/tmp`, deleted
straight after). Nothing was started or stopped on 10.100.0.35.

## Owner decision (2026-11-12)

The simulator evidence is accepted for the hardware gates: AbCip array writes, AbLegacy array writes (SLC500/MicroLogix/PLC-5 simulators), FOCAS `MACRO:` > 32767, and AbCip probe-typo recovery. TwinCAT has no simulator; its hardware proof stays with lmxopcua#755. lmxopcua#761 is closed.