lmxopcua/docs/plans/2026-08-07-adminui-followups-and-live-gates.md

2026-08-07 — adminui-followups closure + the four runnable live gates

Scope. The two items deferment.md §0.1 named as the remaining untracked engineering: the unexecuted 2026-05-29-adminui-followups plan, and the live gates that need only the docker-dev rig (no hardware, no licence). There are exactly four such gate tasks: archreview R2-03 T11 and T12, R2-10 T10, and 2026-06-14-driver-reconfigure-while-faulted Task 3. The other open gates (Galaxy ScanStateProbeParityTests, the v2 GA exit criteria) need a licence or external tooling and were not attempted.

Tree: master @ 21009874. Rig: docker-dev, image rebuilt for this run.

---

0. Rig provenance (done before anything was gated)

The rig was running an image built 2026-08-06 17:55, three hours older than the last code commit (8b10d327, 2026-08-06 21:09). Gating on it would have been meaningless. Rebuilt, both centrals --force-recreate, then:

• Image parity asserted per container (:9200 round-robins central-1/central-2, so a single check proves nothing): both on sha256:49d52d72….
• Positive check that the code under test is in the artifact — not merely that the image is new: RecordBreakerClose + IsBreakerOpen present in the shipped Core.dll, DriverResilienceStatusPublisherService in OtOpcUa.Host.dll, EvaluationResult + ClearCompiledScripts in Runtime.dll; assemblies stamped 2026-08-07 09:29 UTC.

---

1. adminui-followups — CLOSED by audit, no code written

deferment.md called this "the largest unexecuted plan in the tree" and "the only substantial engineering left" (19 pending / 0 completed). The plan's own first step is an overlap audit, and the audit says every task is already shipped, superseded by v3, or moot. Per-task evidence is in docs/plans/2026-05-29-adminui-followups.md.tasks.json; the summary:

Group	Verdict
WS1 Tasks 1, 8 (CollectionEditor<TRow>, OpcUaClient endpoints)	Shipped in the plan's own shape — CollectionEditor.razor, consumed by OpcUaClientDriverForm.razor
WS1 Tasks 2–7 (per-driver tag/device editors ×6)	Superseded by v3 — TagConfigEditorMap (11 editors) + DeviceFormMap (9 device forms)
WS2 Tasks 9, 10 (resilience typed form)	Shipped, deliberately narrower — see below
WS3 Tasks 11–16 (LDAP→role map)	Shipped end to end, incl. the login merge at AuthEndpoints.cs:100
WS4 Tasks 17, 18 (comment cleanup)	Moot — the files were deleted by v3 Batch 2 and #542

Two things worth carrying forward:

• The plan's premise died with v3. WS1 rests on "driver-embedded tag/device lists in DriverConfig JSON are the runtime source of truth". In v3 tags are Tag rows authored in /raw and the deploy artifact populates the driver's RawTags (ModbusDriverFactoryExtensions.cs:98-102), so editing a list on a driver form no longer decides what the driver polls. The capability was delivered, in a different shape.
• WS2 shipped narrower than designed, correctly. Two designed fields no longer exist: RecycleIntervalSeconds went with the scheduled-recycle machinery (#522) and the bulkhead knobs were deleted end-to-end by archreview R2-02 because Build never added a concurrency limiter. Authoring either today would re-create an inert knob.

The supersession claims were checked for reachability, not just existence — TagConfigEditorMap.Resolve is dispatched from RawTagModal.razor:79 and RawManualTagEntryModal.razor:92, DeviceFormMap.Resolve from DeviceModal.razor:45, both guarded by DriverFormMapParityTests + DriverDispatchMapParityTests. A "superseded by X" whose X is itself dead surface is the exact failure this repo has hit repeatedly.

---

2. R2-03 T11 — full-suite sweep ✅ PASSED

Leg	Result
Solution build	0 errors
Runtime.Tests (whole project)	555/555, 3 skipped
RoslynVirtualTagEvaluatorTests	20/20

The 3 skips are the deliberate EquipmentDeviceBindingRetired tombstones, matching the count deferment.md §4.4 predicts. (The plan names the suite RoslynVirtualTagEvaluatorTests; it lives in Host.IntegrationTests, not a project a solution-wide --filter reaches by default.)

---

3. R2-03 T12 — VT fail→Bad→recover ✅ PASSED, with one finding

Both blockers recorded against the previous attempt are gone: the Modbus fixture at 10.100.0.35:5020 is now reachable from inside the rig (so driver data flows and a VT can be staged Good), which is what made this gate runnable at all.

Method. The first two attempts produced a Bad result whose value was empty, but each followed a deploy — and a deploy creates a fresh VT child with no last value, which is correct by design (the inputs-ready gate that suppresses a cold-start flash, VirtualTagActor.cs:246). To isolate the variable the trip was made data-driven and deployment-free: the script fails on a raw tag's value, and that value was flipped by writing Modbus holding register 205 directly on the fixture, so the VT child survives across the transition.

[09:51:57] GateVt = 42 (0x00000000)     Good
[09:52:51] GateVt =    (0x80020000)     Bad_InternalError   <- reg205 := 1, no deployment
[09:53:26] GateVt = 42 (0x00000000)     Good                <- reg205 := 0, no deployment

• ✅ Degradation — script failure degrades the node to 0x80020000 BadInternalError instead of freezing on the last Good. Server-side cause confirmed in the log: VirtualTag VT-gatevt: script execution threw / System.InvalidOperationException.
• ✅ Recovery — returns to Good with the same value, exercising the dedup bypass.
• ✅ P7 no-recompile spot-check — two byte-identical redeploys (revisionHash identical across both, 94f68785…) caused no disturbance at all: Ever went BAD: 0, VT held Good=42. Production has no compile log or metric, so the observable is the consequence of not recompiling; the gate itself is pinned by VirtualTagHostActorTests (exactly two clears across first-generation / identical-redeploy / edited-expression).

⚠️ Finding — "carrying the last-known value" never reaches the client

VirtualTagActor's xmldoc (:22) and the R2-03 row in archreview/plans/STATUS.md both state the Bad publish carries the last-known value. A client never sees it. Two independent channels agree:

• subscription: GateVt = (0x80020000) — value empty;
• raw OPC UA Read service (which returns a DataValue rather than throwing): StatusCode 0x80020000, Value <NULL>.

The VT logic is not at fault. OnEvaluationFailed sends _hasLastValue ? _lastValue : null, and a new regression test — VirtualTagActorTests.The_Bad_result_carries_the_last_known_value — proves the actor emits Bad carrying 42. The value is lost below the actor, in the OPC UA layer (OtOpcUaNodeManager.WriteValue assigns variable.Value unconditionally, so the loss is at Read/Publish encoding, where a Bad StatusCode conventionally carries no value).

Why no test caught it: every existing failure assertion in VirtualTagActorTests checks .StatusCode alone (:87, :101, :123, :142, :180) — the "carrying the last-known value" half was asserted nowhere. That test is now added.

Not filed as a defect because the client-visible behaviour may be the correct OPC UA reading; what is certainly wrong is the documentation, which promises an operator something no client can observe.

Confirmed still true: the Client.CLI read limitation

read surfaces a Bad status as a ServiceResultException from FetchValueAsync rather than printing it — exactly as the previous pass recorded. ReadCommand.cs:52 does print Status:, but the SDK throws before reaching it. Use subscribe (prints value and status, tracks everBad), or a raw Read.

---

4. R2-10 T10 — breaker-OPEN ❌ NOT ACHIEVABLE AS SPECIFIED

The gate asks to point a driver at a dead endpoint and watch the breaker open. It cannot open that way, for a structural reason — not because of the rig. Three attempts, each failing at a different point, together characterise it:

Attempt	What happened
Spare Modbus driver → blackhole (sql:1433, accepts TCP, never answers Modbus)	Driver had no pollable tags (its only tag skipped: "tag config did not map to a definition") — no capability calls at all
MAIN-modbus (13 live tags) → same blackhole	Driver connected, reads failed (MainPressure → 0x80050000 BadTimeout, row showed DEGRADED / "Modbus socket closed mid-response") — yet tracker stayed ConsecutiveFailures = 0, panel showed "All targets healthy."
line3-sql → unusable connection string (Sql ReadAsync does propagate)	Fails in InitializeAsync (liveness check) before ever reaching a wrapped Read

Root cause. The breaker only sees failures that propagate as exceptions out of a CapabilityInvoker-wrapped call. On the dead-endpoint path nothing does:

1. IDriver.InitializeAsync is not wrapped. DriverInstanceActor.cs:707 calls it directly; the invoker wraps only Read / Write / Subscribe / Discover / AlarmAcknowledge (:774, :827, :870, :899, :1081). A driver that cannot connect therefore contributes zero breaker input — which directly refutes the gate's premise.
2. Modbus swallows per-tag failures, in both directions. ReadAsync converts them to Degraded health (:365-375, :838-855) and WriteAsync converts them to a WriteResult status code — neither throws. So even the fallback trigger the plan suggests ("a rapid idempotent write-to-dead-endpoint") cannot work for Modbus.

The breaker itself is configured to open on 2 failures in a 30 s window at ratio 1.0 (DriverResiliencePipelineBuilder.cs:168-175), so the previously-recorded explanation — that R2-09's ConnectionBackoff throttles retries below the threshold — is not the whole story and is arguably not the operative one: the failures never enter the pipeline at all.

Operator consequence worth recording: for the scenario operators care about most — a driver that cannot reach its device — the resilience panel is permanently quiet. It showed "All targets healthy." for a driver that was DEGRADED, timing out every read against a blackhole.

What WAS verified live: the panel renders on /hosts (rehomed there by #546 — the plan's "on the driver page" step is stale, v3 Batch 2 retired those pages), the expanded row shows the RESILIENCE block, and DriverResilienceStatusPublisherService is publishing real tracker state every 5 s (BreakerOpen, ConsecutiveFailures, CurrentInFlight, LastBreakerOpenUtc all observed on the wire). The reader/store/panel chain is live; only the breaker-open state remains unforced.

Note: DriverResilienceStatusChanged going to $DeadLetterWithNoSubscribers on the node is expected, not a defect — the rig runs Phase-5 telemetry in Grpc mode, where the node always double-publishes and central consumes via the gRPC dialer, leaving the DPS copy unsubscribed.

To actually close T10 someone must either wrap the connect path, stop drivers swallowing capability failures, or pick a driver whose wrapped call fails after a successful init (Sql's ReadAsync propagates — it needs a backend that dies after initialization, e.g. docker pause on a dedicated fixture). Left OPEN with this root cause recorded.

---

5. driver-reconfigure-while-faulted Task 3 ✅ PASSED

The task was deferred in 2026-06 with "skip the live test part"; its block ("user-driven sign-in") no longer holds because docker-dev runs the AdminUI with DisableLogin.

before   central-1 startedAt=2026-08-07T09:50:29.939876456Z restarts=0
10:16:18 DriverInstance DRV-ce2fb5217f5d: initialize failed: Connection refused   <- bad config (:5999)
10:22:13 DriverInstance DRV-ce2fb5217f5d: connected                                <- corrected config (:5020)
10:22:13 DriverInstance DRV-ce2fb5217f5d: subscribed to 1 refs
after    central-1 startedAt=2026-08-07T09:50:29.939876456Z restarts=0

The driver adopted the corrected config and connected while faulted, and startedAt is byte-identical before and after with restarts=0 — no node restart, which is the whole point (pre-fix it would retry the stale config forever).

---

6. Rig state

Restored to as-found: Modbus device back on 10.100.0.35:5020, line3-sql back on DevSql, all ResilienceConfig overrides cleared, the SC-gatevt script restored to its original body. Post-restore verification: MainPressure = 7 (Good), GateVt = 7 (Good) — the VT is back on its original one-line body, so it mirrors MainPressure again rather than the ×6 used during the gate.

One anomaly recorded rather than explained: both centrals were recreated at 09:49:40/09:49:50 (new Created, RestartCount=0) by something this session did not issue. It postdates the observations above and the old container's logs went with it, so it is noted, not attributed.

---

7. Addendum 2026-08-08 — R2-10 T10 ✅ PASSED after fixing the structural gap

§4 concluded the gate was not achievable as specified because the connect path never entered the resilience pipeline. The user chose to fix that ("do option 1"), and the gate then passed against the fixed build. Branch feat/wrap-driver-init-in-resilience.

The fix

• DriverCapability.Initialize — IDriver.InitializeAsync routes through the capability invoker in DriverInstanceActor.InitializeAsync (both the initial connect and the Reconnecting retry path; the best-effort teardown ShutdownAsync stays deliberately unwrapped). Keyed on the driver-instance id, like a bulk subscribe.
• No pipeline retry for Initialize — the actor's RetryConnect timer is the retry loop. Enforced at the parser funnel (retryCount override forced to 0, same shape as the write-shaped rule) and in every tier default.
• Breaker threshold 2, not the Read-sized 5 — the breaker samples 30 s and the reconnect timer fires every 10 s, so at most three init attempts fit one window; threshold 5 could never open. Pinned by Initialize_BreakerThreshold_IsReachableAtTheReconnectCadence + Initialize_BreakerOpens_AtTierDefaults_AndFastFailsTheNextAttempt.
• Companion fix — respawn resets tracker snapshots. DriverCapabilityInvokerFactory.Create now calls DriverResilienceStatusTracker.Invalidate(driverInstanceId) alongside the pipeline invalidation. Without it, a deploy during an outage discards the only breaker whose OnClosed could clear the OPEN flag, stranding a false "Breaker OPEN" chip forever. Unreachable before this change made breaker-open reachable; found by asking what clears the flag, not by a test.
• Wiring pinned by Initialize_routes_through_the_capability_invoker + An_invoker_fast_fail_on_init_keeps_the_actor_retrying_until_recovery (falsification-controlled: both go red with the actor wrap stashed; Create_resets_the_instances_tracker_state_with_its_pipelines goes red with the factory call stashed).

The gate run (docker-dev, image df04c76ce06b, symbol-verified on both centrals)

The staged failure was free: rtu-gate (DRV-3616122eb53d) was already authored against 10.100.0.35:5021, where nothing listens — connection refused on every init.

1. Breaker OPEN, both replicas — within ~20 s of boot, both centrals logged circuit-breaker OPENED: instance=DRV-3616122eb53d … capability=Initialize … Connection refused, re-opening each half-open probe cycle. Bonus coverage: DRV-opcua1 (invalid config → init throws) also tripped — a permanently-misconfigured driver was equally invisible before.
2. Panel chip — /hosts → rtu-gate → Controls showed the red "BREAKER OPEN · OPENED 19S AGO" chip in the RESILIENCE section, row FAULTED with "Connection refused".
3. Control — healthy MAIN-modbus recorded zero breaker events throughout.
4. Recovery clear — a socat forwarder (5021 → 5020 on the docker host) brought the endpoint alive; both centrals logged circuit-breaker closed (recovered) + reconnected within 1 s of each other, and the panel dropped to "All targets healthy."
5. Teardown — socat removed; 5021 dead again (as-found). rtu-gate returns to a truthfully-OPEN breaker chip, which is the feature working.

Two nuances against the original T10 wording

• "Both-replica chips" resolves as both nodes' trackers tripping (proven in both logs) rendered through the single active AdminUI. Traefik's /health/active check deliberately routes the UI to the active admin node only — the standby returns 503 (Standby: active node for role 'admin' is …), so there is never a second replica serving chips. Discovered while trying to force the sticky-cookie LB onto central-1.
• The staleness-dim leg was not exercised. Both pair nodes publish under the same (instance, host) key, so staleness requires every publisher for a key to stop — deliberate node-kill churn this gate did not do. The dim is a pure render predicate (IsStale → opacity).

Also corrected while here: §4's "ConnectionBackoff throttles retries" explanation (the previously recorded root cause) is confirmed wrong — the breaker config opens on 2 failures in 30 s; the operative cause was always that no failure reached the pipeline at all.