lmxopcua/docs/reviews/2026-10-21/gates.md

2026-10-21 review remediation — Wave-4 gate record

Tracking issue: lmxopcua#761. This file records each gate run with its date, the commit tested, the procedure, the measured numbers and the verdict. Gates not listed here have not been run.

docker-dev gates A, B, C, E — 2026-11-09

Commit tested	1b9c36f3 (origin/master, "docs(plans): OPC UA SDK 2.0 upgrade spike catalogue (lmxopcua#782)")
Rig	local docker-dev/ (6 host nodes: MAIN central-1/2, SITE-A site-a-1/2, SITE-B site-b-1/2), image otopcua-host:dev sha256:24f9d7287ed2 built from that commit, migrator re-run (no pending migrations)
Clock	Times below are rig UTC (2026-11-10 01:37–02:07 UTC = 2026-11-09 evening US Eastern)
Config	The existing ConfigDb: MAIN carries MAIN-modbus (pymodbus 10.100.0.35:5020), MAIN-opcua (opc-plc 10.100.0.35:50000), MAIN-calc, 48 int3b-mb-* Modbus drivers and one int3b-blackhole driver, 52 children in total. Equipment EQ-32296cc68964 (a/l/e1) has the HR5 reference and the change-triggered virtual tag okvt (= HR100 + 1).
Tools	POST /api/deployments + GET /api/deployments/{id} (X-Api-Key: docker-dev-deploy-key), Client.CLI built from the same commit, /proc/1/status (Threads) and /proc/1/fd inside the containers. dotnet-counters is not in the runtime image, so it was not used. ConfigDb edits were made with sqlcmd -I (QUOTED_IDENTIFIER on).

Summary: A PASS, B PASS, C PASS, E PASS. None of the four gates failed. Observations are listed at the end. None of them blocks a gate.

Gate A (#731) — a driver config change disposes the old instance; threads and handles stay flat — PASS

Procedure. Each iteration edits both MAIN-modbus.DriverConfig (TimeoutMs 2000+i) and MAIN-opcua.DriverConfig (SessionName gateA-i), deploys, waits for the deploy to seal plus 8 s, then samples PID 1 in central-1 and central-2 (both run the MAIN drivers). That is 20 iterations, so 20 respawns of each driver on each node: 80 driver disposals in total. Two warm-up respawns ran before the baseline.

Disposal evidence (log lines in the 01:41:47–01:45:10 window):

node	MAIN-modbus: disposed Modbus driver on stop	MAIN-opcua: disposed OpcUaClient driver on stop	spawned … MAIN-modbus / MAIN-opcua	shutdown/dispose timeout or throw
central-1	20	20	20 / 20	0
central-2	20	20	20 / 20	0

Counts (threads / open fds / RSS KiB):

sample	central-1	central-2
baseline (01:41:47)	46 / 714 / 490 748	46 / 717 / 516 600
after respawn 5	50 / 725 / 506 576	51 / 725 / 529 472
after respawn 10	50 / 725 / 507 084	48 / 726 / 532 972
after respawn 15	48 / 726 / 497 648	48 / 724 / 525 440
after respawn 20 (01:44:59)	52 / 726 / 499 048	48 / 723 / 526 184
settled, 75 s later (01:46:14)	48 / 712	46 / 715

Across the 20 respawns, threads moved between 47 and 55 with no upward trend; that range is normal thread-pool movement. The fd count rose by about 11 on the first respawn, which is the new driver's sockets landing before the old ones close. It then stayed flat at 723–726, and after settling it was lower than the baseline (712/715). RSS stayed flat (±25 MB).

After a respawn, the static Modbus values (HR5 and HR200, which do not change) read Good on both 4840 and 4841. The dead child's BadNoCommunication fan, which lands after the new child's first push, was overwritten by the re-push.

Gate B (#732) — a subscription survives a surgical add; a re-created item after a rebuild receives data — PASS

Procedure. Session 1 opens two Client.CLI subscribe processes (500 ms) on opc.tcp://localhost:4840, which are held for 170 s:

• the raw node ns=2;s=pymodbus/plc/HR100 (ticks about 1 Hz)
• the UNS virtual tag ns=3;s=EQ-32296cc68964/okvt

1. Surgical add (01:47:42). Insert the raw tag HR101 on MAIN-modbus-dev, then deploy. The applier logged kind=PureAdd, added=1, removed=0, changed=0, rebuild=False on both central nodes.
2. Read the new node. ns=2;s=pymodbus/plc/HR101 exists and is served. Its status is BadOutOfRange because the pymodbus fixture refuses address 101 with exception 0x02. The device answered, so this is the fixture's register map and not a server fault.
3. Rebuild (01:48:19). Rename equipment e1 to e1r, which is a ChangedEquipment delta, then deploy. The applier logged kind=Rebuild, … rebuild=True.
4. Re-create (01:48:35). With session 1 still open and still holding its stale items, start session 2 with new subscriptions on the same two NodeIds for 20 s.

Results.

subscription	samples	non-Good samples
session 1, HR100	59	01:47:43.1426 BadNoCommunication (one sample; Good 31505 0.9 ms later, then 1 Hz Good through 01:48:19); 01:48:19.97 BadNodeIdUnknown, then nothing (item dead, as B2a specifies)
session 1, okvt	59	same pattern: one BadNoCommunication sample at 01:47:43.1426, Good again at .1435; BadNodeIdUnknown at 01:48:19.97
session 2, HR100 (after the rebuild)	20	0. Values ran from 36580 at 01:48:34.99 to 38405 at 01:48:54.00
session 2, okvt (after the rebuild)	20	0. Values ran from 36581 to 38406

• Surgical add: the subscription survived. The items were never invalidated and Good data continued at 1 Hz. The single BadNoCommunication sample is not address-space churn. Adding a tag changes the driver's spawn spec, so MAIN-modbus respawned (stopped driver child / disposed / spawned at 01:47:43). The dead child's connectivity-lost fan then reached the item for under 1 ms before the new child's push (see observation 3).
• Rebuild: the old items received BadNodeIdUnknown and no further data. New items on the same NodeIds, created while the stale items were still open in another session, received live data immediately. That is the B2a positive control: before the fix, a stale MonitoredNode2 captured the new item.

The CLI cannot re-create an item inside the same session. The same-session case is covered by the OpcUaServer.IntegrationTests test, not by this gate.

Gate C (#733) — kill a central mid-dispatch, the next deploy succeeds; port 4840 taken ⇒ /health/ready 503 — PASS

To hold a deployment in flight long enough to kill a node, site-b-1 was docker paused before each deploy, so its ack could not land. The dispatch is otherwise sub-second on this rig. Both centrals were killed in turn: central-1, which the issue names, and central-2, which was hosting the config-publish / admin-operations singletons at the time. Killing the singleton host is what exercises the C1 PreStart recovery.

C-a: docker kill central-1 mid-dispatch. Deployment 197e31b4 was AwaitingApplyAcks with 5 of 6 rows Applied (site-b-1 Applying) when central-1 was killed at 01:51:42.

• A second POST while it was in flight returned 409 AnotherDeploymentInFlight, which is correct.
• After site-b-1 was unpaused, the coordinator closed the deployment at the 2 min deadline (01:53:38) as TimedOut. It logged No ack from: site-b-1:4053, because the gRPC dispatch to the paused node was lost.
• central-1 was restarted, and the next deploy, b27ede2a, sealed with 6 of 6 Applied.

C-b: docker kill central-2 (singleton host) mid-dispatch. Deployment 970e7770 was AwaitingApplyAcks (5 of 6) when central-2 was killed at 01:54:03.

• 01:54:30: central-1 auto-downed central-2.
• 01:54:46: the singletons started on central-1 and logged Coordinator recovered in-flight deployment 970e7770… (5/6 acks landed).
• The AdminUI was unavailable through Traefik from about 01:54:05 to 01:54:35 during the hand-over.
• The recovered deployment closed TimedOut at its original deadline (01:56:00).
• central-2 was restarted, and the next deploy, cb0b8134, sealed with 6 of 6 Applied. Neither run left a deployment wedged in flight.

C-port: port 4840 taken.

• Fused admin+driver node. central-2 was stopped. A replacement container was started from the same compose service (compose run --use-aliases). A perl listener in it bound 0.0.0.0:4840 before dotnet OtOpcUa.Host.dll started. Results:
  • GET /health/ready returned HTTP 503, overall status Unhealthy, with opcua-endpoint: Unhealthy — "OPC UA SDK failed to start: Failed to establish tcp listener sockets for Ipv4 and IPv6." (status Failed). configdb and akka were both Healthy.
  • The node logged OpcUaEndpointStepDown: … not the driver Primary — nothing to step down from.
• Driver-only node. The same setup on site-a-2 (driver-only): the host logged FTL OtOpcUa.Host terminated: Failed to establish tcp listener sockets… and the process exited with code 1 (C3 "a driver-only node rethrows").
• Both replacement containers were removed and the original containers restarted. The ServiceLevel half of C3 is #762 and was not part of this gate.

Gate E (#734) — a while(true) virtual tag leaves the node responsive, the tag shows quarantined, threads stay flat over 2 min — PASS

Procedure.

• Script: gateE-spin with source var v = ctx.GetTag("pymodbus/plc/HR100").Value; while (true) { }.
• Virtual tag: spinvt on EQ-32296cc68964, Int32, change-triggered, so it fires about 1 Hz on HR100.
• Healthy sibling: okvt, on the same equipment and the same trigger.
• Deployed at 02:01:11. Before the deploy and every 6 s for about 3.5 min after it, the following were sampled:
  • threads and CPU ticks of PID 1 on both centrals
  • /health/ready latency
  • Client.CLI read of okvt and spinvt on 4840

An earlier attempt used a timer-only trigger (TimerIntervalMs=1000, no dependency), and the script never ran; see observation 4.

Quarantine.

• The first run started at about 02:01:16. The following triggers were refused, as designed: refusals do not count as strikes.
• At 02:01:36, after 20.2 s, which is past StuckOrphanQuarantineFactor 10 × the 2 s budget, both centrals logged this Error to scripts-*.log: Script gateE-vt-spin quarantined: its previous evaluation has been running for 20.2 s, over 10x its 2000 ms budget … not evaluated again until its source changes or it is redeployed.
• /metrics:
  • otopcua_script_quarantined_total{reason="stuck",script="gateE-vt-spin"} 1
  • otopcua_script_run_refused_total{script="gateE-vt-spin"} 21
  • Both stayed at 1 and 21 for the rest of the run, so evaluation stopped after the quarantine.
• OPC UA status of ns=3;s=EQ-32296cc68964/spinvt: BadWaitingForInitialData until the first run, then BadInternalError (0x80020000) from 02:01:23 onward.

Responsiveness.

metric	before the deploy	over the 3.5 min after
/health/ready	200 in 3–20 ms	200 in 3–11 ms
Client.CLI read (connect + read)	0.34–0.46 s	0.33–0.49 s

okvt, the healthy sibling, kept evaluating Good throughout (30656 → 41368).

Threads (central-1 / central-2):

sample	central-1	central-2
before	50–53	49–50
02:01:17 – 02:04:42 (3.5 min, sampled every 6–9 s)	50–54	50–54

There was no growth, and at most one thread was orphaned per node.

CPU. The orphaned run cannot be pre-empted in-process. Both nodes went from about 0.1 core to about 1.04 cores (about 104 ticks/s), and that continued after the virtual tag was deleted and redeployed (1022 ticks per 10 s at 02:05). Only a process restart cleared it, so both centrals were rolling-restarted after the gate. This is the documented limitation: the out-of-process runner is #750. The gate criteria (responsive, quarantined, threads flat) hold.

Observations (none blocks a gate)

1. The status API does not say why a deployment timed out. In the normal timeout branch (some acks landed), GET /api/deployments/{id} returns status: TimedOut with failureReason: null. Only the central log names the silent node (No ack from: site-b-1:4053). The no-expected-rows branch does set a reason. An automation client polling the API cannot see which node to fix. Resolved (lmxopcua#784): the timeout branch now persists failureReason = Timed out waiting for acks (n/m landed). No ack from: <nodes>. … (at most 20 nodes named, the rest counted).
2. Rig: mesh PSK warnings on the centrals. central logs Mesh PSK secret ot-mesh-psk-{main,site-a,site-b} … is absent (or blank) in the store; the key is REVOKED — commands to that cluster and acks from it are refused. Yet every gRPC dispatch and ack in these runs succeeded. Either the rig's PSK secrets are not seeded and enforcement is off, or the warning is stale. Not investigated. Resolved (lmxopcua#784): the warning was false, not a bypass. The rig's centrals supply all three keys through MeshTransport__ClusterPsks__{MAIN,SITE-A,SITE-B} (each equal to that cluster's nodes' MeshTransport__GrpcPsk) and seed nothing in the store. MeshPskProvider.TryGetPsk consults that override first on every lookup, so enforcement was on and every call authenticated against the override key; only the background RefreshAsync read the store for those clusters anyway and reported them revoked. The refresh now skips override-covered clusters, so REVOKED is logged only for a cluster with no key at all.
3. A surgical raw-tag add respawns that driver. The driver's spawn spec carries its tag set, so adding one tag reconnects the whole driver (the accepted cost noted in DriverSpawnPlanner). On a live subscriber this shows as one BadNoCommunication sample (< 1 ms) on every value of that driver, including UNS references and virtual tags fed by it. ScadaBridge-style clients that alarm on a single Bad sample would see it on every tag add.
4. A timer-only virtual tag never evaluates, and nothing flags it. VirtualTag.TimerIntervalMs is not consumed by the runtime: VirtualTagActor evaluates on dependency changes only, and no timer scheduler exists. A virtual tag with TimerIntervalMs=1000 and no ctx.GetTag dependency deployed cleanly and sat at BadWaitingForInitialData indefinitely. The CalculationNoEffectiveTrigger DraftValidator rule (E4) covers Calculation tags only, not virtual tags.
5. The quarantine line in the main log has no script name. The guard's Error is mirrored into the main log by ScriptLogCompanionSink as WRN [Script] unknown emitted Error: Script "gateE-vt-spin" quarantined: …. The guard's logger carries no ScriptName/ScriptId property, so the mirror labels it unknown, and it is rate-limited under that shared key. scripts-*.log has the correct [ERR] line. Resolved (lmxopcua#784): the guard binds the script key as ScriptName, so the mirror names the script and rate-limits each one separately.
6. The quarantine shows as BadInternalError on OPC UA. A client sees no quarantine-specific status. The reason is only in the logs and metrics. Decided (lmxopcua#784): kept BadInternalError, the documented code for any script-attributable fault on all three seams; BadOutOfService would read as a deliberately disabled point and would be a ScadaBridge-visible contract change. Rationale: docs/VirtualTags.md §"Quarantine: what a client sees".

Rig state left behind

• All six nodes run otopcua-host:dev sha256:24f9d7287ed2 (commit 1b9c36f3).
• Every gate edit was reverted in the ConfigDb: HR101 deleted; e1 name restored; gateE-spin/spinvt deleted; both driver configs restored. A clean-up deploy, cd41a890, sealed 6 of 6.
• Both centrals were restarted afterwards to clear the gate-E orphan thread.

Driver fixture gates + H1 (2026-11-09, commit 1b9c36f3, fixture host 10.100.0.35)

Fixture gates, lmxopcua#761 (driver fixtures + H1)

Date: 2026-11-09. Commit tested: 1b9c36f3 (origin/master), worktree scratchpad/wt/otopc-fix (branch w18/fixtures, nothing committed). Release builds, 0 errors. Nothing posted to Gitea.

Fixture host 10.100.0.35: lmxopcua-fix does not exist on macOS (CLAUDE.md says to use ssh). Every needed fixture was already running, so nothing was started or stopped there. State before and after was identical (modbus-standard, ab-server-controllogix, python-snap7-s7_1500, opc-plc, mosquitto + sparkplug-sim + mqtt-publisher, sql-pg/oracle/blackhole, otopcua-mssql). Pre-existing: otopcua-opc-plc reports "unhealthy" (before and after; the opcuaclient suite still passed). otopcua-sql-blackhole showed "unhealthy" briefly right after the blackhole pause/unpause test, then "healthy" again.

Driver fixture gates

Gate	Command (from worktree)	Result
modbus standard	dotnet test tests/Drivers/*Modbus.IntegrationTests -c Release (default endpoint 10.100.0.35:5020, profile standard)	PASS: 12 passed, 0 failed, 32 skipped. Skips are the other profiles only (exception_injection 11, mitsubishi 7, dl205 ~11, s7_1500 4). They are not valid on the standard profile. Standard round-trip and AddressingGrammar ran.
s7 s7_1500	S7_SIM_ALLOW_WRITES=1 dotnet test tests/Drivers/*S7.IntegrationTests	PASS: 23 passed, 0 failed, 3 skipped. Without the writes env var: 20 passed and 6 skipped. The 3 remaining skips: TM/CT address limit of python-snap7, and two destructive tests (S7_RECONNECT_BOUNCE_CMD, S7_TIMEOUT_OUTAGE_*_CMD) not run so the shared sim is not bounced.
abcip controllogix	AB_SERVER_ALLOW_WRITES=1 AB_DIFF_ENDPOINT=10.100.0.35:44818 dotnet test tests/Drivers/*AbCip.IntegrationTests	PASS with one flake: 33 passed, 1 failed, 22 skipped (about 2 min). Failure: PollLatencyComparisonTests.Native_poll_p95_is_within_ten_percent_of_libplctag raised DifferentialAbortedException ("Connection reset by peer" on tag UdtSeg.Ready). Rerun of that class: 2/2 passed. Verdict: flake (ab_server connection reset, aborts on first connection-level failure by design). Skips: 6 need AB_SERVER_PROFILE=emulate (Logix Emulate), 1 needs AB_HW_WRITE_GATEWAY (hardware), 15 are the remaining differential and write tests. Without writes env: 28 passed, 0 failed, 28 skipped.
opcuaclient opc-plc	dotnet test tests/Drivers/*OpcUaClient.IntegrationTests (opc.tcp://10.100.0.35:50000)	PASS: 5 passed, 0 skipped.
mqtt + sparkplug	MQTT_FIXTURE_ENDPOINT=10.100.0.35:8883 MQTT_FIXTURE_PLAIN_ENDPOINT=10.100.0.35:1883 MQTT_FIXTURE_USERNAME=otopcua MQTT_FIXTURE_PASSWORD=<from publisher container env> MQTT_FIXTURE_CA_CERT=<scp'd ca.crt> dotnet test tests/Drivers/*Mqtt.IntegrationTests	PASS: 30 passed, 0 skipped (includes Sparkplug, simulator running).
sql	SQL_TEST_ENDPOINT=10.100.0.35,14330 SQL_TEST_PASSWORD=.. SQL_PG_ENDPOINT=10.100.0.35:15432 SQL_PG_PASSWORD=.. SQL_ORACLE_ENDPOINT=10.100.0.35:11521 SQL_ORACLE_PASSWORD=.. SQL_BLACKHOLE_ENDPOINT=10.100.0.35,14333 SQL_BLACKHOLE_PASSWORD=.. SQL_BLACKHOLE_DOCKER_SSH=dohertj2@10.100.0.35 dotnet test tests/Drivers/*Sql.IntegrationTests	PASS: 37 passed, 0 failed, 17 skipped. SQL Server, Postgres, Oracle and the blackhole timeout test ran. Skips: 14 MySQL (no MySQL fixture on the host) and 3 ODBC (no SQL_ODBC_CONNECTION_STRING).

Side effect: the SQL suite creates (and never drops, by design) database SqlPollFixture on otopcua-mssql. It was already there before (listed in sys.databases).

H1 (#736) historian year-window gate

There was no pre-existing year-window fixture test. The existing fixture suites test the cap at 3 rows, and the 10k fuzz is offline in OpcUaServer.Tests. I added an uncommitted probe, tests/Drivers/Historian.Conformance/YearWindowGateProbe.cs, with a subclass in each backend's integration project (edits are only in the throwaway worktree). It seeds one tag with 52,560 samples spanning 365 days, with maxRawValuesPerRead = 10,000.

• It checks that an N=0 read of the whole year is refused with HistoryReadLimitExceededException (Limit=10000), not returned unbounded and not truncated.
• It walks the window in 10k-capped pages, resuming after the last timestamp, and checks every page is at most 10000.
• It checks the concatenated pages equal the seeded set (no loss, no duplicates, ordered), and that peak process working set stays under 500 MB.

Backend	Where	Result
QuestDb	local throwaway container h1-gate-questdb (questdb/questdb:10.0.1, ports 28812/29000; removed afterwards)	PASS: N=0 refused, limit 10000; 6 pages, max page 10000, 52,560 rows walked, no loss or duplicates; peak working set 122 MB (managed 27 MB). Full QuestDb integration project: 234 passed, 0 failed, 1 skipped (opt-in HISTORIAN_QUESTDB_STRESS_SECONDS stress test).
SqlServer	throwaway DB OtOpcUaHistorian_T_* on otopcua-mssql (the fixture creates and drops it; no orphan left)	PASS: same numbers. 6 pages, max 10000, 52,560 rows walked; peak working set 114 MB (managed 18 MB). Full SqlServer integration project: 196 passed, 0 failed, 0 skipped.
Server half: continuation point at N=0	offline: dotnet test tests/Server/*OpcUaServer.Tests --filter RawPageLimit/HistoryRead/MaxReturnDataValues/HistoryServerCapabilities	PASS: 431 passed, 0 failed (includes the paged-vs-unpaged raw cap fuzz and the MaxReturnDataValues advertisement).

Scope caveat: the probe drives the backend IHistorianDataSource directly, not a live OPC UA HistoryRead through the server node manager. The backend side is proven live: bounded at 10k, N=0 refused, pages complete. The continuation point itself is the server's RawPageLimit planning, covered by the offline tests above, not by a live OPC UA session. A true end-to-end gate (OPC UA client reading a year at N=0 against a server on the SqlServer/QuestDb backend) needs the docker-dev rig, which I was told not to use.

Mishap: two aborted SQL Server probe attempts (row-by-row seeding was too slow, so I killed them) each left a throwaway OtOpcUaHistorian_T_* database on otopcua-mssql. Both were dropped manually. The final check shows no OtOpcUaHistorian* databases remain.

NOT RUN

• Hardware userGates (real PLCs): AbCip / AbLegacy / TwinCAT array writes, FOCAS MACRO: > 32767, AbCip probe-typo recovery.
• docker-dev gates A, B, C, D, E, and the Galaxy gate F: not in scope. The otopcua-dev rig is off-limits, and F needs mxaccessgw.
• Modbus exception_injection / dl205 / mitsubishi / s7_1500 profiles (only standard requested); S7 reconnect-bounce and connect-timeout-outage destructive tests; AbCip Logix Emulate profile; Sql MySQL and ODBC (no fixture).

Cleanup

Container h1-gate-questdb removed. The fixture host is unchanged. The worktree scratchpad/wt/otopc-fix (branch w18/fixtures) is left in place with uncommitted probe edits. Remove it with git worktree remove --force if not wanted.

Gate D (#745) — AdminUI live pass, docker-dev (2026-11-09) — FAIL (1 of 5 checks)

Overall: four checks PASS; the delete-confirmation check FAILS on four pages (filed as a follow-up issue, see #761). Full record below, as written by the Chrome pass.

Date: 2026-11-09 (container clock 2026-11-10 03:xx UTC). Rig: local docker-dev, otopcua-host:dev image built 2026-11-09 20:35 EST from master 1b9c36f3 (per coordinator; rig built from a scratchpad worktree no longer on disk, so commit not re-verifiable from the container). AdminUI http://localhost:9200 (login disabled, session "multi-role-test"). Wonder boxes untouched. No deploy performed.

1. 100 KB script survives save + reload - PASS

Procedure: /scripts/new, set Monaco model to 102,492 chars (1,200+ comment lines + return 1;), name gateD-100kb-test, Create. Reloaded /scripts/SC-d82b3a36fd85; editor text === regenerated original (identical, length 102492 both). Then deleted it (own modal).

2. Two-tab /hosts CPU flat with ~50 drivers - PASS

Rig has 52 drivers on cluster MAIN (48 int3b-mb-*, blackhole, Calc, opcplc, pymodbus); site-a/b have none. docker stats --no-stream on central-1/central-2 (the AdminUI hosts), 8 samples ~12s apart (~100s) per condition:

• no /hosts tab (on /): central-1 11.0%, central-2 10.9% avg
• 1 /hosts tab (first run): c1 9.7%, c2 11.4%
• 2 /hosts tabs: c1 10.9%, c2 12.1% avg Within noise (range 8-17% in every condition). Flat. Raw: scratchpad/cpu-*.txt.

3. 5k-tag device renders < 2 s - PASS

No such device existed. Created disabled-in-effect (never deployed) Modbus driver gateD-bulk/device gdev under docker-dev, imported 5000-row CSV via the Raw tree modal (parse+review 206 ms, commit 717 ms). Expanding gdev: first tag rows in DOM 14-20 ms after click, painted (double rAF) 44 ms. Tree paged: 500 rendered ("Show 500 more (4500 not shown)") and DOM ~611 nodes. Far under 2 s. Cleanup: tags cannot be bulk-deleted in UI (device delete: "Delete blocked - contains tags"), so removed gateD-bulk rows (5000 Tag, TagGroup, Device, DriverInstance) by scoped SQL delete in a transaction on the rig OtOpcUa DB (needed sqlcmd -I). Verified 0 gateD DriverInstances, Tag total back to 55. Observation (not a defect): no bulk "delete all tags in device" in Raw tree.

4. Every delete asks for confirmation - FAIL (4 defects)

window.confirm overridden; no native confirm ever fired. Own modal confirmed + Cancel verified (item survives) for: script delete (ScriptEdit), API key delete (/apikeys), all 10 UNS-tree deletes on /uns (enterprise, clusters, area, line, equipment), Raw-tree driver delete and device delete (modals). Equipment-page virtual-tag/alarm/reference deletes: source uses ConfirmDeleteVirtualTag/ConfirmDeleteAlarm/ConfirmRemoveReference; no live rows on the sampled equipment (NOT exercised live). Certificates delete uses RequestAction -> Confirm panel (source). DEFECTS - delete executes immediately on click, no confirmation: a) /role-grants Delete (RoleGrants.razor:68 -> DeleteAsync(r.Id)). LIVE-VERIFIED: added test grant cn=gated-test..., clicked Delete, row gone instantly, no modal. (Test grant thus already removed.) b) ClusterEdit.razor:102 @onclick="DeleteAsync" (cluster delete) - source only c) NodeEdit.razor:119 @onclick="DeleteAsync" (cluster node delete) - source only d) AclEdit.razor:151 @onclick="DeleteAsync" (node ACL delete) - source only (CollectionEditor.razor:31 per-row Delete in driver forms is a working-copy edit committed on Save - not counted.) Repro for (a): http://localhost:9200/role-grants, add any group, press its Delete.

5. A6 diagnostics panel shows real counts - PASS

/fleet/diagnostics?node=central-1:4053 : DRIVERS 52, FAULTED DEVICES 0, CONFIG SOURCE live, REVISION e221b1792cbd..., per-driver rows (48 HEALTHY int3b-mb-* with 1 connected device each, Calc/opcplc/pymodbus HEALTHY, int3b-blackhole INITIALIZING 0 connected). /hosts shows the same 52 driver instances and blackhole INITIALIZING "1 / 1 DOWN". Disabled int3b-bulk correctly absent. Minor: blackhole's down device is not counted in "FAULTED DEVICES 0" (state INITIALIZING, not faulted) - arguably fine, flag for A6 owner.

Revert state

Test script, test role grant, gateD-bulk driver/device/5000 tags all removed; no deploys; browser tab closed. Rig config as found.

Still not run after gate D

• Gate F (Galaxy) — needs a reachable mxaccessgw.
• Hardware userGates (real PLCs), as listed under NOT RUN above.