lmxopcua/docs/reviews/2026-10-21/gates.md

2026-10-21 review remediation — Wave-4 gate record

Tracking issue: lmxopcua#761. This file records each gate run with its date, the commit tested, the procedure, the measured numbers and the verdict. Gates not listed here have not been run.

docker-dev gates A, B, C, E — 2026-11-09

Commit tested	1b9c36f3 (origin/master, "docs(plans): OPC UA SDK 2.0 upgrade spike catalogue (lmxopcua#782)")
Rig	local docker-dev/ (6 host nodes: MAIN central-1/2, SITE-A site-a-1/2, SITE-B site-b-1/2), image otopcua-host:dev sha256:24f9d7287ed2 built from that commit, migrator re-run (no pending migrations)
Clock	Times below are rig UTC (2026-11-10 01:37–02:07 UTC = 2026-11-09 evening US Eastern)
Config	The existing ConfigDb: MAIN carries MAIN-modbus (pymodbus 10.100.0.35:5020), MAIN-opcua (opc-plc 10.100.0.35:50000), MAIN-calc, 48 int3b-mb-* Modbus drivers and one int3b-blackhole driver, 52 children in total. Equipment EQ-32296cc68964 (a/l/e1) has the HR5 reference and the change-triggered virtual tag okvt (= HR100 + 1).
Tools	POST /api/deployments + GET /api/deployments/{id} (X-Api-Key: docker-dev-deploy-key), Client.CLI built from the same commit, /proc/1/status (Threads) and /proc/1/fd inside the containers. dotnet-counters is not in the runtime image, so it was not used. ConfigDb edits were made with sqlcmd -I (QUOTED_IDENTIFIER on).

Summary: A PASS, B PASS, C PASS, E PASS. None of the four gates failed. Observations are listed at the end. None of them blocks a gate.

Gate A (#731) — a driver config change disposes the old instance; threads and handles stay flat — PASS

Procedure. Each iteration edits both MAIN-modbus.DriverConfig (TimeoutMs 2000+i) and MAIN-opcua.DriverConfig (SessionName gateA-i), deploys, waits for the deploy to seal plus 8 s, then samples PID 1 in central-1 and central-2 (both run the MAIN drivers). That is 20 iterations, so 20 respawns of each driver on each node: 80 driver disposals in total. Two warm-up respawns ran before the baseline.

Disposal evidence (log lines in the 01:41:47–01:45:10 window):

node	MAIN-modbus: disposed Modbus driver on stop	MAIN-opcua: disposed OpcUaClient driver on stop	spawned … MAIN-modbus / MAIN-opcua	shutdown/dispose timeout or throw
central-1	20	20	20 / 20	0
central-2	20	20	20 / 20	0

Counts (threads / open fds / RSS KiB):

sample	central-1	central-2
baseline (01:41:47)	46 / 714 / 490 748	46 / 717 / 516 600
after respawn 5	50 / 725 / 506 576	51 / 725 / 529 472
after respawn 10	50 / 725 / 507 084	48 / 726 / 532 972
after respawn 15	48 / 726 / 497 648	48 / 724 / 525 440
after respawn 20 (01:44:59)	52 / 726 / 499 048	48 / 723 / 526 184
settled, 75 s later (01:46:14)	48 / 712	46 / 715

Across the 20 respawns, threads moved between 47 and 55 with no upward trend; that range is normal thread-pool movement. The fd count rose by about 11 on the first respawn, which is the new driver's sockets landing before the old ones close. It then stayed flat at 723–726, and after settling it was lower than the baseline (712/715). RSS stayed flat (±25 MB).

After a respawn, the static Modbus values (HR5 and HR200, which do not change) read Good on both 4840 and 4841. The dead child's BadNoCommunication fan, which lands after the new child's first push, was overwritten by the re-push.

Gate B (#732) — a subscription survives a surgical add; a re-created item after a rebuild receives data — PASS

Procedure. Session 1 opens two Client.CLI subscribe processes (500 ms) on opc.tcp://localhost:4840, which are held for 170 s:

• the raw node ns=2;s=pymodbus/plc/HR100 (ticks about 1 Hz)
• the UNS virtual tag ns=3;s=EQ-32296cc68964/okvt

1. Surgical add (01:47:42). Insert the raw tag HR101 on MAIN-modbus-dev, then deploy. The applier logged kind=PureAdd, added=1, removed=0, changed=0, rebuild=False on both central nodes.
2. Read the new node. ns=2;s=pymodbus/plc/HR101 exists and is served. Its status is BadOutOfRange because the pymodbus fixture refuses address 101 with exception 0x02. The device answered, so this is the fixture's register map and not a server fault.
3. Rebuild (01:48:19). Rename equipment e1 to e1r, which is a ChangedEquipment delta, then deploy. The applier logged kind=Rebuild, … rebuild=True.
4. Re-create (01:48:35). With session 1 still open and still holding its stale items, start session 2 with new subscriptions on the same two NodeIds for 20 s.

Results.

subscription	samples	non-Good samples
session 1, HR100	59	01:47:43.1426 BadNoCommunication (one sample; Good 31505 0.9 ms later, then 1 Hz Good through 01:48:19); 01:48:19.97 BadNodeIdUnknown, then nothing (item dead, as B2a specifies)
session 1, okvt	59	same pattern: one BadNoCommunication sample at 01:47:43.1426, Good again at .1435; BadNodeIdUnknown at 01:48:19.97
session 2, HR100 (after the rebuild)	20	0. Values ran from 36580 at 01:48:34.99 to 38405 at 01:48:54.00
session 2, okvt (after the rebuild)	20	0. Values ran from 36581 to 38406

• Surgical add: the subscription survived. The items were never invalidated and Good data continued at 1 Hz. The single BadNoCommunication sample is not address-space churn. Adding a tag changes the driver's spawn spec, so MAIN-modbus respawned (stopped driver child / disposed / spawned at 01:47:43). The dead child's connectivity-lost fan then reached the item for under 1 ms before the new child's push (see observation 3).
• Rebuild: the old items received BadNodeIdUnknown and no further data. New items on the same NodeIds, created while the stale items were still open in another session, received live data immediately. That is the B2a positive control: before the fix, a stale MonitoredNode2 captured the new item.

The CLI cannot re-create an item inside the same session. The same-session case is covered by the OpcUaServer.IntegrationTests test, not by this gate.

Gate C (#733) — kill a central mid-dispatch, the next deploy succeeds; port 4840 taken ⇒ /health/ready 503 — PASS

To hold a deployment in flight long enough to kill a node, site-b-1 was docker paused before each deploy, so its ack could not land. The dispatch is otherwise sub-second on this rig. Both centrals were killed in turn: central-1, which the issue names, and central-2, which was hosting the config-publish / admin-operations singletons at the time. Killing the singleton host is what exercises the C1 PreStart recovery.

C-a: docker kill central-1 mid-dispatch. Deployment 197e31b4 was AwaitingApplyAcks with 5 of 6 rows Applied (site-b-1 Applying) when central-1 was killed at 01:51:42.

• A second POST while it was in flight returned 409 AnotherDeploymentInFlight, which is correct.
• After site-b-1 was unpaused, the coordinator closed the deployment at the 2 min deadline (01:53:38) as TimedOut. It logged No ack from: site-b-1:4053, because the gRPC dispatch to the paused node was lost.
• central-1 was restarted, and the next deploy, b27ede2a, sealed with 6 of 6 Applied.

C-b: docker kill central-2 (singleton host) mid-dispatch. Deployment 970e7770 was AwaitingApplyAcks (5 of 6) when central-2 was killed at 01:54:03.

• 01:54:30: central-1 auto-downed central-2.
• 01:54:46: the singletons started on central-1 and logged Coordinator recovered in-flight deployment 970e7770… (5/6 acks landed).
• The AdminUI was unavailable through Traefik from about 01:54:05 to 01:54:35 during the hand-over.
• The recovered deployment closed TimedOut at its original deadline (01:56:00).
• central-2 was restarted, and the next deploy, cb0b8134, sealed with 6 of 6 Applied. Neither run left a deployment wedged in flight.

C-port: port 4840 taken.

• Fused admin+driver node. central-2 was stopped. A replacement container was started from the same compose service (compose run --use-aliases). A perl listener in it bound 0.0.0.0:4840 before dotnet OtOpcUa.Host.dll started. Results:
  • GET /health/ready returned HTTP 503, overall status Unhealthy, with opcua-endpoint: Unhealthy — "OPC UA SDK failed to start: Failed to establish tcp listener sockets for Ipv4 and IPv6." (status Failed). configdb and akka were both Healthy.
  • The node logged OpcUaEndpointStepDown: … not the driver Primary — nothing to step down from.
• Driver-only node. The same setup on site-a-2 (driver-only): the host logged FTL OtOpcUa.Host terminated: Failed to establish tcp listener sockets… and the process exited with code 1 (C3 "a driver-only node rethrows").
• Both replacement containers were removed and the original containers restarted. The ServiceLevel half of C3 is #762 and was not part of this gate.

Gate E (#734) — a while(true) virtual tag leaves the node responsive, the tag shows quarantined, threads stay flat over 2 min — PASS

Procedure.

• Script: gateE-spin with source var v = ctx.GetTag("pymodbus/plc/HR100").Value; while (true) { }.
• Virtual tag: spinvt on EQ-32296cc68964, Int32, change-triggered, so it fires about 1 Hz on HR100.
• Healthy sibling: okvt, on the same equipment and the same trigger.
• Deployed at 02:01:11. Before the deploy and every 6 s for about 3.5 min after it, the following were sampled:
  • threads and CPU ticks of PID 1 on both centrals
  • /health/ready latency
  • Client.CLI read of okvt and spinvt on 4840

An earlier attempt used a timer-only trigger (TimerIntervalMs=1000, no dependency), and the script never ran; see observation 4.

Quarantine.

• The first run started at about 02:01:16. The following triggers were refused, as designed: refusals do not count as strikes.
• At 02:01:36, after 20.2 s, which is past StuckOrphanQuarantineFactor 10 × the 2 s budget, both centrals logged this Error to scripts-*.log: Script gateE-vt-spin quarantined: its previous evaluation has been running for 20.2 s, over 10x its 2000 ms budget … not evaluated again until its source changes or it is redeployed.
• /metrics:
  • otopcua_script_quarantined_total{reason="stuck",script="gateE-vt-spin"} 1
  • otopcua_script_run_refused_total{script="gateE-vt-spin"} 21
  • Both stayed at 1 and 21 for the rest of the run, so evaluation stopped after the quarantine.
• OPC UA status of ns=3;s=EQ-32296cc68964/spinvt: BadWaitingForInitialData until the first run, then BadInternalError (0x80020000) from 02:01:23 onward.

Responsiveness.

metric	before the deploy	over the 3.5 min after
/health/ready	200 in 3–20 ms	200 in 3–11 ms
Client.CLI read (connect + read)	0.34–0.46 s	0.33–0.49 s

okvt, the healthy sibling, kept evaluating Good throughout (30656 → 41368).

Threads (central-1 / central-2):

sample	central-1	central-2
before	50–53	49–50
02:01:17 – 02:04:42 (3.5 min, sampled every 6–9 s)	50–54	50–54

There was no growth, and at most one thread was orphaned per node.

CPU. The orphaned run cannot be pre-empted in-process. Both nodes went from about 0.1 core to about 1.04 cores (about 104 ticks/s), and that continued after the virtual tag was deleted and redeployed (1022 ticks per 10 s at 02:05). Only a process restart cleared it, so both centrals were rolling-restarted after the gate. This is the documented limitation: the out-of-process runner is #750. The gate criteria (responsive, quarantined, threads flat) hold.

Observations (none blocks a gate)

1. The status API does not say why a deployment timed out. In the normal timeout branch (some acks landed), GET /api/deployments/{id} returns status: TimedOut with failureReason: null. Only the central log names the silent node (No ack from: site-b-1:4053). The no-expected-rows branch does set a reason. An automation client polling the API cannot see which node to fix. Resolved (lmxopcua#784): the timeout branch now persists failureReason = Timed out waiting for acks (n/m landed). No ack from: <nodes>. … (at most 20 nodes named, the rest counted).
2. Rig: mesh PSK warnings on the centrals. central logs Mesh PSK secret ot-mesh-psk-{main,site-a,site-b} … is absent (or blank) in the store; the key is REVOKED — commands to that cluster and acks from it are refused. Yet every gRPC dispatch and ack in these runs succeeded. Either the rig's PSK secrets are not seeded and enforcement is off, or the warning is stale. Not investigated. Resolved (lmxopcua#784): the warning was false, not a bypass. The rig's centrals supply all three keys through MeshTransport__ClusterPsks__{MAIN,SITE-A,SITE-B} (each equal to that cluster's nodes' MeshTransport__GrpcPsk) and seed nothing in the store. MeshPskProvider.TryGetPsk consults that override first on every lookup, so enforcement was on and every call authenticated against the override key; only the background RefreshAsync read the store for those clusters anyway and reported them revoked. The refresh now skips override-covered clusters, so REVOKED is logged only for a cluster with no key at all.
3. A surgical raw-tag add respawns that driver. The driver's spawn spec carries its tag set, so adding one tag reconnects the whole driver (the accepted cost noted in DriverSpawnPlanner). On a live subscriber this shows as one BadNoCommunication sample (< 1 ms) on every value of that driver, including UNS references and virtual tags fed by it. ScadaBridge-style clients that alarm on a single Bad sample would see it on every tag add.
4. A timer-only virtual tag never evaluates, and nothing flags it. VirtualTag.TimerIntervalMs is not consumed by the runtime: VirtualTagActor evaluates on dependency changes only, and no timer scheduler exists. A virtual tag with TimerIntervalMs=1000 and no ctx.GetTag dependency deployed cleanly and sat at BadWaitingForInitialData indefinitely. The CalculationNoEffectiveTrigger DraftValidator rule (E4) covers Calculation tags only, not virtual tags.
5. The quarantine line in the main log has no script name. The guard's Error is mirrored into the main log by ScriptLogCompanionSink as WRN [Script] unknown emitted Error: Script "gateE-vt-spin" quarantined: …. The guard's logger carries no ScriptName/ScriptId property, so the mirror labels it unknown, and it is rate-limited under that shared key. scripts-*.log has the correct [ERR] line. Resolved (lmxopcua#784): the guard binds the script key as ScriptName, so the mirror names the script and rate-limits each one separately.
6. The quarantine shows as BadInternalError on OPC UA. A client sees no quarantine-specific status. The reason is only in the logs and metrics. Decided (lmxopcua#784): kept BadInternalError, the documented code for any script-attributable fault on all three seams; BadOutOfService would read as a deliberately disabled point and would be a ScadaBridge-visible contract change. Rationale: docs/VirtualTags.md §"Quarantine: what a client sees".

Rig state left behind

• All six nodes run otopcua-host:dev sha256:24f9d7287ed2 (commit 1b9c36f3).
• Every gate edit was reverted in the ConfigDb: HR101 deleted; e1 name restored; gateE-spin/spinvt deleted; both driver configs restored. A clean-up deploy, cd41a890, sealed 6 of 6.
• Both centrals were restarted afterwards to clear the gate-E orphan thread.

Driver fixture gates + H1 (2026-11-09, commit 1b9c36f3, fixture host 10.100.0.35)

Fixture gates, lmxopcua#761 (driver fixtures + H1)

Date: 2026-11-09. Commit tested: 1b9c36f3 (origin/master), worktree scratchpad/wt/otopc-fix (branch w18/fixtures, nothing committed). Release builds, 0 errors. Nothing posted to Gitea.

Fixture host 10.100.0.35: lmxopcua-fix does not exist on macOS (CLAUDE.md says to use ssh). Every needed fixture was already running, so nothing was started or stopped there. State before and after was identical (modbus-standard, ab-server-controllogix, python-snap7-s7_1500, opc-plc, mosquitto + sparkplug-sim + mqtt-publisher, sql-pg/oracle/blackhole, otopcua-mssql). Pre-existing: otopcua-opc-plc reports "unhealthy" (before and after; the opcuaclient suite still passed). otopcua-sql-blackhole showed "unhealthy" briefly right after the blackhole pause/unpause test, then "healthy" again.

Driver fixture gates

Gate	Command (from worktree)	Result
modbus standard	dotnet test tests/Drivers/*Modbus.IntegrationTests -c Release (default endpoint 10.100.0.35:5020, profile standard)	PASS: 12 passed, 0 failed, 32 skipped. Skips are the other profiles only (exception_injection 11, mitsubishi 7, dl205 ~11, s7_1500 4). They are not valid on the standard profile. Standard round-trip and AddressingGrammar ran.
s7 s7_1500	S7_SIM_ALLOW_WRITES=1 dotnet test tests/Drivers/*S7.IntegrationTests	PASS: 23 passed, 0 failed, 3 skipped. Without the writes env var: 20 passed and 6 skipped. The 3 remaining skips: TM/CT address limit of python-snap7, and two destructive tests (S7_RECONNECT_BOUNCE_CMD, S7_TIMEOUT_OUTAGE_*_CMD) not run so the shared sim is not bounced.
abcip controllogix	AB_SERVER_ALLOW_WRITES=1 AB_DIFF_ENDPOINT=10.100.0.35:44818 dotnet test tests/Drivers/*AbCip.IntegrationTests	PASS with one flake: 33 passed, 1 failed, 22 skipped (about 2 min). Failure: PollLatencyComparisonTests.Native_poll_p95_is_within_ten_percent_of_libplctag raised DifferentialAbortedException ("Connection reset by peer" on tag UdtSeg.Ready). Rerun of that class: 2/2 passed. Verdict: flake (ab_server connection reset, aborts on first connection-level failure by design). Skips: 6 need AB_SERVER_PROFILE=emulate (Logix Emulate), 1 needs AB_HW_WRITE_GATEWAY (hardware), 15 are the remaining differential and write tests. Without writes env: 28 passed, 0 failed, 28 skipped.
opcuaclient opc-plc	dotnet test tests/Drivers/*OpcUaClient.IntegrationTests (opc.tcp://10.100.0.35:50000)	PASS: 5 passed, 0 skipped.
mqtt + sparkplug	MQTT_FIXTURE_ENDPOINT=10.100.0.35:8883 MQTT_FIXTURE_PLAIN_ENDPOINT=10.100.0.35:1883 MQTT_FIXTURE_USERNAME=otopcua MQTT_FIXTURE_PASSWORD=<from publisher container env> MQTT_FIXTURE_CA_CERT=<scp'd ca.crt> dotnet test tests/Drivers/*Mqtt.IntegrationTests	PASS: 30 passed, 0 skipped (includes Sparkplug, simulator running).
sql	SQL_TEST_ENDPOINT=10.100.0.35,14330 SQL_TEST_PASSWORD=.. SQL_PG_ENDPOINT=10.100.0.35:15432 SQL_PG_PASSWORD=.. SQL_ORACLE_ENDPOINT=10.100.0.35:11521 SQL_ORACLE_PASSWORD=.. SQL_BLACKHOLE_ENDPOINT=10.100.0.35,14333 SQL_BLACKHOLE_PASSWORD=.. SQL_BLACKHOLE_DOCKER_SSH=dohertj2@10.100.0.35 dotnet test tests/Drivers/*Sql.IntegrationTests	PASS: 37 passed, 0 failed, 17 skipped. SQL Server, Postgres, Oracle and the blackhole timeout test ran. Skips: 14 MySQL (no MySQL fixture on the host) and 3 ODBC (no SQL_ODBC_CONNECTION_STRING).

Side effect: the SQL suite creates (and never drops, by design) database SqlPollFixture on otopcua-mssql. It was already there before (listed in sys.databases).

H1 (#736) historian year-window gate

There was no pre-existing year-window fixture test. The existing fixture suites test the cap at 3 rows, and the 10k fuzz is offline in OpcUaServer.Tests. I added an uncommitted probe, tests/Drivers/Historian.Conformance/YearWindowGateProbe.cs, with a subclass in each backend's integration project (edits are only in the throwaway worktree). It seeds one tag with 52,560 samples spanning 365 days, with maxRawValuesPerRead = 10,000.

• It checks that an N=0 read of the whole year is refused with HistoryReadLimitExceededException (Limit=10000), not returned unbounded and not truncated.
• It walks the window in 10k-capped pages, resuming after the last timestamp, and checks every page is at most 10000.
• It checks the concatenated pages equal the seeded set (no loss, no duplicates, ordered), and that peak process working set stays under 500 MB.

Backend	Where	Result
QuestDb	local throwaway container h1-gate-questdb (questdb/questdb:10.0.1, ports 28812/29000; removed afterwards)	PASS: N=0 refused, limit 10000; 6 pages, max page 10000, 52,560 rows walked, no loss or duplicates; peak working set 122 MB (managed 27 MB). Full QuestDb integration project: 234 passed, 0 failed, 1 skipped (opt-in HISTORIAN_QUESTDB_STRESS_SECONDS stress test).
SqlServer	throwaway DB OtOpcUaHistorian_T_* on otopcua-mssql (the fixture creates and drops it; no orphan left)	PASS: same numbers. 6 pages, max 10000, 52,560 rows walked; peak working set 114 MB (managed 18 MB). Full SqlServer integration project: 196 passed, 0 failed, 0 skipped.
Server half: continuation point at N=0	offline: dotnet test tests/Server/*OpcUaServer.Tests --filter RawPageLimit/HistoryRead/MaxReturnDataValues/HistoryServerCapabilities	PASS: 431 passed, 0 failed (includes the paged-vs-unpaged raw cap fuzz and the MaxReturnDataValues advertisement).

Scope caveat: the probe drives the backend IHistorianDataSource directly, not a live OPC UA HistoryRead through the server node manager. The backend side is proven live: bounded at 10k, N=0 refused, pages complete. The continuation point itself is the server's RawPageLimit planning, covered by the offline tests above, not by a live OPC UA session. A true end-to-end gate (OPC UA client reading a year at N=0 against a server on the SqlServer/QuestDb backend) needs the docker-dev rig, which I was told not to use.

Mishap: two aborted SQL Server probe attempts (row-by-row seeding was too slow, so I killed them) each left a throwaway OtOpcUaHistorian_T_* database on otopcua-mssql. Both were dropped manually. The final check shows no OtOpcUaHistorian* databases remain.

NOT RUN

• Hardware userGates (real PLCs): AbCip / AbLegacy / TwinCAT array writes, FOCAS MACRO: > 32767, AbCip probe-typo recovery.
• docker-dev gates A, B, C, D, E, and the Galaxy gate F: not in scope. The otopcua-dev rig is off-limits, and F needs mxaccessgw.
• Modbus exception_injection / dl205 / mitsubishi / s7_1500 profiles (only standard requested); S7 reconnect-bounce and connect-timeout-outage destructive tests; AbCip Logix Emulate profile; Sql MySQL and ODBC (no fixture).

Cleanup

Container h1-gate-questdb removed. The fixture host is unchanged. The worktree scratchpad/wt/otopc-fix (branch w18/fixtures) is left in place with uncommitted probe edits. Remove it with git worktree remove --force if not wanted.

Gate D (#745) — AdminUI live pass, docker-dev (2026-11-09) — FAIL (1 of 5 checks)

Overall: four checks PASS; the delete-confirmation check FAILS on four pages (filed as a follow-up issue, see #761). Full record below, as written by the Chrome pass.

Date: 2026-11-09 (container clock 2026-11-10 03:xx UTC). Rig: local docker-dev, otopcua-host:dev image built 2026-11-09 20:35 EST from master 1b9c36f3 (per coordinator; rig built from a scratchpad worktree no longer on disk, so commit not re-verifiable from the container). AdminUI http://localhost:9200 (login disabled, session "multi-role-test"). Wonder boxes untouched. No deploy performed.

1. 100 KB script survives save + reload - PASS

Procedure: /scripts/new, set Monaco model to 102,492 chars (1,200+ comment lines + return 1;), name gateD-100kb-test, Create. Reloaded /scripts/SC-d82b3a36fd85; editor text === regenerated original (identical, length 102492 both). Then deleted it (own modal).

2. Two-tab /hosts CPU flat with ~50 drivers - PASS

Rig has 52 drivers on cluster MAIN (48 int3b-mb-*, blackhole, Calc, opcplc, pymodbus); site-a/b have none. docker stats --no-stream on central-1/central-2 (the AdminUI hosts), 8 samples ~12s apart (~100s) per condition:

• no /hosts tab (on /): central-1 11.0%, central-2 10.9% avg
• 1 /hosts tab (first run): c1 9.7%, c2 11.4%
• 2 /hosts tabs: c1 10.9%, c2 12.1% avg Within noise (range 8-17% in every condition). Flat. Raw: scratchpad/cpu-*.txt.

3. 5k-tag device renders < 2 s - PASS

No such device existed. Created disabled-in-effect (never deployed) Modbus driver gateD-bulk/device gdev under docker-dev, imported 5000-row CSV via the Raw tree modal (parse+review 206 ms, commit 717 ms). Expanding gdev: first tag rows in DOM 14-20 ms after click, painted (double rAF) 44 ms. Tree paged: 500 rendered ("Show 500 more (4500 not shown)") and DOM ~611 nodes. Far under 2 s. Cleanup: tags cannot be bulk-deleted in UI (device delete: "Delete blocked - contains tags"), so removed gateD-bulk rows (5000 Tag, TagGroup, Device, DriverInstance) by scoped SQL delete in a transaction on the rig OtOpcUa DB (needed sqlcmd -I). Verified 0 gateD DriverInstances, Tag total back to 55. Observation (not a defect): no bulk "delete all tags in device" in Raw tree.

4. Every delete asks for confirmation - FAIL (4 defects)

window.confirm overridden; no native confirm ever fired. Own modal confirmed + Cancel verified (item survives) for: script delete (ScriptEdit), API key delete (/apikeys), all 10 UNS-tree deletes on /uns (enterprise, clusters, area, line, equipment), Raw-tree driver delete and device delete (modals). Equipment-page virtual-tag/alarm/reference deletes: source uses ConfirmDeleteVirtualTag/ConfirmDeleteAlarm/ConfirmRemoveReference; no live rows on the sampled equipment (NOT exercised live). Certificates delete uses RequestAction -> Confirm panel (source). DEFECTS - delete executes immediately on click, no confirmation: a) /role-grants Delete (RoleGrants.razor:68 -> DeleteAsync(r.Id)). LIVE-VERIFIED: added test grant cn=gated-test..., clicked Delete, row gone instantly, no modal. (Test grant thus already removed.) b) ClusterEdit.razor:102 @onclick="DeleteAsync" (cluster delete) - source only c) NodeEdit.razor:119 @onclick="DeleteAsync" (cluster node delete) - source only d) AclEdit.razor:151 @onclick="DeleteAsync" (node ACL delete) - source only (CollectionEditor.razor:31 per-row Delete in driver forms is a working-copy edit committed on Save - not counted.) Repro for (a): http://localhost:9200/role-grants, add any group, press its Delete.

5. A6 diagnostics panel shows real counts - PASS

/fleet/diagnostics?node=central-1:4053 : DRIVERS 52, FAULTED DEVICES 0, CONFIG SOURCE live, REVISION e221b1792cbd..., per-driver rows (48 HEALTHY int3b-mb-* with 1 connected device each, Calc/opcplc/pymodbus HEALTHY, int3b-blackhole INITIALIZING 0 connected). /hosts shows the same 52 driver instances and blackhole INITIALIZING "1 / 1 DOWN". Disabled int3b-bulk correctly absent. Minor: blackhole's down device is not counted in "FAULTED DEVICES 0" (state INITIALIZING, not faulted) - arguably fine, flag for A6 owner.

Revert state

Test script, test role grant, gateD-bulk driver/device/5000 tags all removed; no deploys; browser tab closed. Rig config as found.

Still not run after gate D

• Gate F (Galaxy) — run 2026-11-11, see "Gate F (#735)" below.
• Hardware userGates (real PLCs), as listed under NOT RUN above.

Gate F (#735) — 20 config deploys keep mxgateway.sessions.open flat (docker-dev + VD03 DEV mxaccessgw, 2026-11-11) — PASS

Commit tested	5ac55205 (master). Image otopcua-host:dev sha256:47ca0da6eaf7 rebuilt from that clean checkout before the run; all six nodes recreated from it.
Gateway	DEV mxaccessgw on wonder-app-vd03.zmr.zimmer.com (gRPC h2c :5120, assembly 0.1.2.0, main 2b13564e). Metrics from its anonymous Prometheus endpoint http://wonder-app-vd03.zmr.zimmer.com:5130/metrics (mxgateway_sessions_open, …_sessions_opened_total, …_sessions_closed_total, …_workers_running, …_workers_killed_total, …_faults_total), scraped every 15 s from the Mac for the whole run. Read-only Galaxy access, owner-approved. No remote command ran on any wonder box.
Reachability	The docker-dev containers resolved and reached wonder-app-vd03:5120 through the Mac's VPN route (/dev/tcp check from central-1) before anything was configured.
Driver	gatef-gal, DriverType GalaxyMxGateway, cluster MAIN, seeded straight into the ConfigDb: gateway.endpoint = the VD03 gateway, useTls=false, apiKeySecretRef=env:GALAXY_MXGW_API_KEY, mxAccess.clientName=OtOpcUa-761-gateF, repository.watchDeployEvents=false, no alarmAck fallback. Device vd03 ({}); two Read-only tags: ScanState (DevWinPlatform.ScanState, Boolean) and CPULoad (DevWinPlatform.CPULoad, Float64). No writes, no alarm subscriptions, no acks.
Secret	The VD03 gateway key reached the containers only as the GALAXY_MXGW_API_KEY runtime env (exported from the owner's env file into the docker compose up shell). It is not in the repo, a compose file, the ConfigDb or this record.
Respawn trigger	As in gate A: each iteration edits the driver's config (JSON_MODIFY of mxAccess.publishingIntervalMs = 1000 + i), POSTs /api/deployments, waits for it to seal, waits 15 s, then scrapes the gateway. A config change is a respawn (#665), so every iteration stops and disposes the old GalaxyDriver and spawns a new one on both central-1 and central-2 (MAIN runs its drivers on both nodes) — 40 driver respawns in total.

Attribution. The gateway's metrics have no per-client label (the session table with client names is on the login-gated dashboard, which was not used), so the rig's share is measured as the delta against a baseline taken before the driver existed, cross-checked against the opened/closed counters. During the whole window the other consumers' sessions did not churn: opened − closed moved only by what the rig did.

Gateway samples (UTC):

when	open	opened	closed	workers running
baseline, 13 samples 23:11:15–23:14:11 (no Galaxy driver on the rig)	2	3	1	2
driver added + deployed (23:14:34)	4 (+2: one session per central)	5	1	4
iterations 1–20, 15 s after each seal (23:15:13–23:20:56)	4 in 13, 3 in 4 (it. 5, 6, 11, 16), 2 in 3 (it. 1, 17, 18); never above 4	5 → 42	1 → 38	2–4
settled, 23:21:04 / 23:21:09 / 23:21:25	4	42	38	4

The rig opened 39 sessions over the run (42 − 3) and closed 37 (38 − 1); the net +2 is exactly its two live drivers. The open gauge returned to its steady value (baseline + 2) after the 20 deploys, so no session leaked. The dips to 2–3 are samples that landed while a respawned driver was still reconnecting (see the init timeouts below), not lost sessions; every iteration ended with both drivers connected.

Gateway-side counters over the run: workers_exited{worker-shutdown-ack} 1 → 37 (graceful closes), workers_killed{OpenSessionFailed} 0 → 13, workers_killed{client-close} 0 → 1, faults{WorkerFaulted} 0 → 14, workers_exited{WorkerFaulted} 0 → 1, grpc_streams_disconnected{ClientCancelled} 31.

Rig side (per central, 23:14:56–23:21:10): 20 spawned GalaxyMxGateway driver gatef-gal, 20 stopped driver child, 20 disposed GalaxyMxGateway driver on stop, 25 GalaxyDriver gatef-gal shutting down (20 respawns + 5 failed-init teardowns), 15 GalaxyMxSession connected, 5 initialize failed: … timeout of '00:00:05'. central-2 logged one GalaxyMxSession close failed or timed out (best-effort) — dropping the channel. No error or fault otherwise.

Driver health after the run (/hosts, /fleet/diagnostics): gatef-gal on central-1 and central-2 Healthy, 2 tags, host status "2 / 6 down" (the probe watcher seeded 5 platforms; two Galaxy platforms on VD03 DEV report down, unrelated to the gate). Client.CLI read of both tags on 4840 and 4841 was Good before the loop (ScanState True, CPULoad ≈ 13.4) and after it (CPULoad 15.63, Good).

Removal and the final drop — not observed (VPN dropped)

The Mac's VPN route to VD03 went down at about 23:21:35, ten seconds after the settled samples. DNS for *.zmr.zimmer.com and the 10.220.x route both disappeared; it was still down at 23:40. The rig's Galaxy drivers lost the gateway (reconnect supervisor backing off, Initialize breaker opened), which is the expected behaviour.

gatef-gal (driver, device, 2 tags) was deleted from the ConfigDb and the removal deployed at 23:30:56 (12d5ff59, Sealed). Both centrals logged stopped driver child gatef-gal and disposed GalaxyMxGateway driver on stop. Because the gateway was unreachable, the rig could not close its two sessions; the gateway reaps a session whose client stream is gone on its detach-grace / lease sweep. The return of sessions.open to 2 was therefore not observed. Re-check mxgateway_sessions_open once the VPN is back: it should read the baseline (2, if VD03's own consumers are unchanged).

Observations (none blocks the gate)

1. A Galaxy (re)start often exceeds the 5 s Tier-A Initialize timeout against VD03. 10 of 40 respawns (5 per node) failed initialize at exactly 5 s. Opening a gateway session spawns an x86 MXAccess worker, and a successful connect took 4–5 s from spawn to connected, right at the limit. Each timeout cost a 10 s reconnect-timer retry (BadNoCommunication on that driver's nodes meanwhile) and shows on the gateway as an OpenSessionFailed worker kill plus a WorkerFaulted fault. The gateway cleaned up every one (the gauge stayed flat), so it is a latency and noise issue, not a leak. Galaxy probably wants a longer Initialize timeout (tier or per-instance ResilienceConfig). Not filed (no Gitea per brief).
2. The VD03 gateway's faults_total is now 14 (WorkerFaulted) from this gate — anyone reading the VD03 dashboard after 2026-11-11 23:15Z should attribute those, and the 13 OpenSessionFailed kills, to this run.
3. The rig's opens (39 at the gateway) exceed its logged GalaxyMxSession connected lines (30 in the loop plus 2 on add); the extra opens are ones that completed after the client's 5 s init timeout had already given up. They were closed by the end of the window (opened − closed = 2 at 23:21).

Rig state left behind

• gatef-gal and its device/tags are deleted from the ConfigDb; the removal deploy sealed. The gate's Deployment rows carry only the env: reference, no secret.
• All host nodes were recreated without the exported key, so GALAXY_MXGW_API_KEY is back to the committed docker-dev default. The rig runs otopcua-host:dev sha256:47ca0da6eaf7 (5ac55205).

Re-run on current master — all runnable gates (2026-11-12)

Commit tested	54c4357f = master 412da27c + the deploy-API fix below (landed on master as fa8c3d1a after a rebase over two unrelated commits; same diff). Image otopcua-host:g761 sha256:24432d7fbc2d, built from that commit, on all six nodes.
Rig	docker-dev as its own compose project otopcua-g761 (an overlay remapping every host port and image tag), so the shared otopcua-dev rig that someone else was running was not touched. The ConfigDb was restored from a COPY_ONLY backup of the otopcua-dev ConfigDb (the same MAIN-modbus / MAIN-opcua / equipment EQ-32296cc68964 + okvt config the 11-09 run used), then migrated (no pending migrations). Host ports: OPC UA 24840–24845, AdminUI 29200, per-node HTTP 29100–29105.
Clock	Rig UTC 2026-11-13 03:05–03:44 (= 2026-11-12 evening US Eastern).
Load	The 48 int3b-mb-* and the int3b-blackhole drivers were disabled after gate A. With them on, two rigs' worth of Modbus drivers saturated the shared pymodbus sim on 10.100.0.35 (about 117–136 % CPU) and HR100 read BadCommunicationError. Gate D's 50-driver checks were not re-run (see D).

Summary: A, B, C, D (check 4), E, H1 (live), the F removal check and every fixture lane PASS. One product defect found and fixed (deploy API). Hardware userGates still open.

Defect found and fixed: a bodyless POST /api/deployments answered 404 — FIXED (fa8c3d1a)

The documented headless deploy (docker-dev/deploy.sh, README "Headless deploy") returned 404 on both centrals, while the same POST with a JSON body was accepted. The optional body was an inferred DeployRequest? parameter, which stamps Accepts(application/json) metadata on the endpoint. For a request with no Content-Type the routing AcceptsMatcherPolicy prefers endpoints that declare no consumes metadata. On a fused admin+driver host MapOtOpcUaMeshGrpcServices has mapped grpc-dotnet's {unimplementedService}/{unimplementedMethod} catch-all, so the request was routed away from the deploy handler and got a 404. The deploy API on its own does not reproduce it. The handler now reads the optional body itself: no body is the headless form, a non-JSON body answers 415 and malformed JSON 400. DeployApiBodylessPostTests (Host.IntegrationTests) composes the deploy API with the mesh gRPC services and drives real HTTP; with the fix reverted both bodyless cases fail with 404. Live on the rebuilt rig, a bodyless POST through Traefik answered 202 (every deploy below used it).

Gate A (#731) — PASS, now with dotnet-counters

Method as on 11-09 (each iteration edits MAIN-modbus TimeoutMs and MAIN-opcua SessionName, deploys, waits for the seal + 8 s), 2 warm-ups + 20 iterations, every deploy sealed in 1–2 s. This time dotnet-counters collect -p 1 --counters System.Runtime --refresh-interval 5 (the linux-arm64 single-file tool copied into each central) ran for the whole loop, alongside /proc/1 threads / fds / RSS.

	central-1	central-2
disposed Modbus driver on stop / disposed OpcUaClient driver on stop	20 / 20	20 / 20
spawned … MAIN-modbus / MAIN-opcua	20 / 20	20 / 20
dispose/shutdown timeout or throw	0	0
threads (baseline → range over 20 → settled 75 s later)	54 → 49–57 → 48	53 → 48–56 → 46
open fds	739 → 691–739 → 715	739 → 704–740 → 719
dotnet.assembly.count (57 / 54 samples)	312, constant	314, constant
dotnet.timer.count first-quarter vs last-quarter mean	139 → 137	144 → 145
dotnet.thread_pool.thread.count (UpDownCounter deltas)	net ≈ 0	net ≈ 0
working set first-quarter vs last-quarter mean	520 → 505 MB	522 → 502 MB

No upward trend in any series; nothing (assemblies, timers, threads, fds) accumulates per respawn. Raw CSVs: counters-central-{1,2}.csv in the run's evidence (not committed).

Gate B (#732) — PASS

Same procedure as 11-09 on opc.tcp://localhost:24840, session 1 held 120 s on ns=2;s=pymodbus/plc/HR100 and ns=3;s=EQ-32296cc68964/okvt.

• Surgical add (HR101, 03:14:27): both centrals logged kind=PureAdd, added=1, … rebuild=False. Each subscription took one BadNoCommunication sample at 03:14:27 and was Good again in the same second (the accepted driver respawn), then 21 / 22 Good updates until the rebuild. HR101 itself reads BadOutOfRange (the fixture refuses address 101, as on 11-09).
• Rebuild (rename e1 → e1r, 03:14:49): kind=Rebuild, … rebuild=True; session 1's items got BadNodeIdUnknown and nothing after.
• Re-created after the rebuild (session 2, 20 s): HR100 17 updates, okvt 16, all Good, none Bad.

Gate C (#733) — PASS

• C-a, kill central-1 mid-dispatch. site-b-1 paused, deployment 56e5ede1 at AwaitingApplyAcks 5/6, docker kill central-1 at 03:16:37. A second POST while in flight: 409. After unpausing site-b-1 the held deployment sealed 6/6 (on 11-09 it timed out). After central-1 restarted, the next deploy 6df32531 sealed 6/6.
• C-b, kill the singleton host. central-2 (the only /health/active 200 on MAIN) killed at 03:17:30 with 548ccbb9 in flight (5/6). central-1 auto-downed it at 03:17:58, the singletons moved (Younger -> BecomingOldest), the AdminUI was back through Traefik at 03:18:02, and at 03:18:14 central-1 logged Coordinator recovered in-flight deployment 548ccbb9… (5/6 acks landed). It closed TimedOut with failureReason = Timed out waiting for acks (5/6 landed). No ack from: site-b-1:4053. … (the #784 fix, verified live). After central-2 restarted, the next deploy 246594e0 sealed 6/6.
• C-port. As on 11-09, a replacement container from the compose service with a perl listener holding 0.0.0.0:4840 before dotnet OtOpcUa.Host.dll starts (the 4840 socket inode belongs to perl, none to dotnet):
  • fused admin+driver (central-2): GET /health/ready 503, opcua-endpoint: Unhealthy — OPC UA SDK failed to start: Failed to establish tcp listener sockets on port 4840 for Ipv4 and IPv6.; configdb/akka Healthy.
  • driver-only (site-a-2): FTL OtOpcUaServerHostedService: SDK start failed on a driver-only node …, exit 1.
  • So the C3 behaviour holds on OPC UA SDK 2.0.1. (A first attempt was invalid: the Perl one-liner did not keep the socket in a variable, so Perl closed it at once and the SDK bound 4840 normally. Re-run with the socket held.)
• Side effect worth knowing: the 39 s docker pause of site-b-1 in C-b outlived the 15 s auto-down, so site-b-2 downed it and the SITE-B pair came back as two 1-member clusters, both Active, until site-b-2 was restarted. That is the documented auto-down trade (docs/Redundancy.md, restart one side after the link heals), not a defect, but every C-style drill that pauses a node past 15 s leaves this behind.

Gate D (#745) — check 4 re-run after #785: PASS

Checks 1, 2, 3 and 5 passed on 11-09 and their code is unchanged. Check 4, the one that failed, was re-run with Playwright against http://localhost:29200 (login disabled), with a native-confirm listener armed (it never fired):

page	Delete click	Cancel	Confirm
/role-grants (throwaway grant cn=g761-gateD,…)	modal "Delete role grant" naming the group; row still in DB	row kept	row deleted (0)
/clusters/SITE-B/acls/g761-acl (throwaway ACL)	modal "Delete ACL"; row still in DB	row kept	row deleted (0)
/clusters/SITE-B/edit	modal "Delete cluster 'SITE-B'? This cannot be undone."	modal closed, cluster kept	not confirmed (real row)
/clusters/SITE-B/nodes/site-b-2:4053	modal "Delete node 'site-b-2:4053' from cluster 'SITE-B'? …"	modal closed, node kept	not confirmed (real row)

Gate E (#734) — PASS

gateE-spin (var v = ctx.GetTag("pymodbus/plc/HR100").Value; while (true) { }) as change-triggered virtual tag spinvt on EQ-32296cc68964, deployed 03:27:21, sampled every ~4 s for 1 min 50 s:

• Quarantined on both centrals at 03:27:42 (21.4 s / 22.3 s, over 10 × the 2 s budget); /metrics otopcua_script_quarantined_total{reason="stuck",script="gateE-vt-spin"} 1, run_refused_total 8 / 11. The main-log mirror now names the script ([Script] gateE-vt-spin emitted Error …, the #784 fix).
• Responsive: /health/ready 200 in 2–6 ms throughout (2.9–3.5 ms before); okvt kept reading Good, connect + read 0.40–0.48 s (0.41–0.52 s before); spinvt BadInternalError.
• Threads flat: central-1 42–44, central-2 41–45 (41–42 before).
• CPU: both centrals went to ~1 core (≈100 ticks/s) until restarted after the gate — the known in-process limit (#750; the out-of-process runner is the fix).

H1 (#736) — live end-to-end over OPC UA: PASS

This closes the 11-09 scope caveat (that run drove the backend directly). ServerHistorian:Backend=SqlServer on the centrals (DB OtOpcUaHistorian on the rig SQL), 52,560 samples (every 10 min, 2025-11-01 → 2026-10-31) seeded for HR300 (lg9.long). A probe client (OPC UA SDK 2.0.1) sent HistoryRead raw with NumValuesPerNode = 0 for the whole year on opc.tcp://localhost:24840 and followed every continuation point:

page	values	continuation point
1	9,999	16 bytes
2–5	9,998 each	16 bytes
6	2,569	none

52,560 values in total (= the seeded set: first 0, last 52559), timestamps strictly ascending, no duplicates, 0.2 s. central-1 GC heap after the read: gen2 48 MB + LOH 56 MB + gen1 7 MB (≈ 112 MB), committed 170 MB — under the 500 MB bound. QuestDb was not re-run live: the paging is the backend-agnostic node-manager RawPageLimit plan, and the QuestDb backend half passed on 11-09.

Gate F (#735) — the removal drop, now observed: PASS

The 11-11 run could not see mxgateway_sessions_open return to baseline after removing the driver (the VPN dropped). The VD03 gateway has since restarted (its counters reset), so this was re-checked directly: a read-only GalaxyMxGateway driver g761-gal (one Read tag, DevWinPlatform.ScanState; key only as env:GALAXY_MXGW_API_KEY, sourced into the compose shell from the owner's env file and never written anywhere) was added to MAIN and then removed. Gateway /metrics (anonymous, scraped from the Mac):

when (UTC)	open	opened_total	closed_total	workers
baseline 03:40:49–03:41:19	2	2	–	2
driver deployed 03:41:47 (both centrals connected, ScanState read Good)	4	4	–	4
driver removed 03:42:58 → 03:43:13 … 03:43:59	2	4	2	2

Both centrals logged stopped driver child g761-gal + disposed GalaxyMxGateway driver on stop; the gateway closed both sessions within 15 s. No Galaxy writes, no alarm subscriptions.

Driver fixture lanes (10.100.0.35) at 412da27c — PASS

Suite	Result	Notes
Modbus standard (:5020)	12 / 0 / 32	Two earlier attempts failed 8 then 1 tests on connect timeouts: the shared sim was saturated by this gate's own rig plus otopcua-dev (196 connections from this Mac). Clean once the load dropped. Skips are the other profiles.
S7 s7_1500 (S7_SIM_ALLOW_WRITES=1)	23 / 0 / 3	Same 3 skips as 11-09 (TM/CT span limit, the two destructive CMD tests).
AbCip controllogix, AB_SERVER_ALLOW_WRITES=1 AB_DIFF_ENDPOINT=…	34 / 0 / 22	No write test runs in this combination: IntegrationWriteGate refuses writes while AB_DIFF_ENDPOINT is set (read-only differential run). That was also true of the 11-09 "with writes" run.
AbCip controllogix, AB_SERVER_ALLOW_WRITES=1 only	38 / 0 / 18	The real write pass: all 10 write tests pass, incl. AbCipRefusalLiveTests.A_whole_array_write_round_trips (DINT[16]) and AbCipUdtMemberWriteTests.A_member_array_element_write_leaves_the_other_elements_alone, Native + libplctag.
OpcUaClient (opc-plc :50000)	9 / 0 / 0	One flake on the first try (ServerHostNameValidationTests…(validate: False), BadRequestInterrupted); clean rerun.
OpcUaClient.Browser (+ OPCUA_NAT_ENDPOINT=…:50010)	12 / 0 / 0	The NAT fixture is up, so the #806 NAT tests ran too.
MQTT + Sparkplug	30 / 0 / 0
Sql (SQL Server, PG, Oracle, blackhole)	58 / 0 / 25	21 MySQL + 4 ODBC skips (no fixture).
MTConnect (G1, :5000)	21 / 0 / 0	First run of the G1 MTConnect lane against the shared agent.

Hardware userGates — simulator evidence only; real hardware still required

userGate	Simulator evidence (2026-11-12, 412da27c)	Real hardware
AbCip array write	ab_server ControlLogix: whole-array DINT[16] round-trip + UDT member-array element write pass (above)	not run
AbLegacy array write	ab_server SLC500 / MicroLogix / PLC-5: N_file_array_write_round_trips_every_element passes on all 3 families × 2 transports; AbLegacy suite 51 / 0 / 0	not run (#752)
TwinCAT array + STRING write	none — no ADS simulator exists (elma-ads-tcp/udp on 10.100.0.35 are socat relays to an absent ADS box); suite 0 / 0 / 18 skipped	not run (#755)
FOCAS MACRO: > 32767	otopcua-focas-fixture mock (byte-identical to the repo's focas_mock/server.py): WireBackendCoverageTests.Macro_and_parameter_numbers_above_short_range_read_the_authored_variable (MACRO:99999, PARAM:40000, with a decoy at the 16-bit-wrapped address) passes; FOCAS suite 18 / 0 / 0	not run
AbCip probe-typo recovery	AbCipRefusalLiveTests.A_typod_probe_tag_does_not_black_out_a_reachable_device passes on ab_server, Native + libplctag	not run

Plan INT-4 marks these "USER-ORDERED — NON-SKIPPABLE … real ControlLogix / SLC / TwinCAT hardware", so the simulator passes do not close them. They need bench endpoints from the owner, or an explicit owner decision to accept the simulator evidence for the four that have it (TwinCAT has none).

Observations

1. The bodyless-deploy 404 (fixed above) had been masked because every earlier gate run sent a JSON body.
2. Two docker-dev rigs on one Mac saturate the shared pymodbus sim; disable the 48 int3b-mb-* drivers (they exist only for gate D's 50-driver checks) when not running those.

Rig state left behind

The otopcua-g761 project (containers, its volumes, the otopcua-host:g761 / otopcua-migrator:g761 images), the g761-probe network and the temporary OtOpcUaHistorian database (inside the project's SQL volume) were all removed. The shared otopcua-dev rig was only read (one COPY_ONLY backup to its container's /tmp, deleted straight after). Nothing was started or stopped on 10.100.0.35.

Owner decision (2026-11-12)

The simulator evidence is accepted for the hardware gates: AbCip array writes, AbLegacy array writes (SLC500/MicroLogix/PLC-5 simulators), FOCAS MACRO: > 32767, and AbCip probe-typo recovery. TwinCAT has no simulator; its hardware proof stays with lmxopcua#755. lmxopcua#761 is closed.