lmxopcua/docs/plans/2026-08-07-adminui-followups-and-live-gates.md

# 2026-08-07 — adminui-followups closure + the four runnable live gates

**Scope.** The two items `deferment.md` §0.1 named as the remaining untracked engineering:
the unexecuted `2026-05-29-adminui-followups` plan, and the live gates that need only the
docker-dev rig (no hardware, no licence). There are exactly **four** such gate tasks:
archreview **R2-03 T11** and **T12**, **R2-10 T10**, and
`2026-06-14-driver-reconfigure-while-faulted` **Task 3**. The other open gates (Galaxy
`ScanStateProbeParityTests`, the v2 GA exit criteria) need a licence or external tooling and were
not attempted.

**Tree:** `master` @ `21009874`. **Rig:** docker-dev, image rebuilt for this run.

---

## 0. Rig provenance (done before anything was gated)

The rig was running an image built **2026-08-06 17:55**, three hours *older* than the last code
commit (`8b10d327`, 2026-08-06 21:09). Gating on it would have been meaningless. Rebuilt, both
centrals `--force-recreate`, then:

- **Image parity asserted per container** (`:9200` round-robins central-1/central-2, so a single
  check proves nothing): both on `sha256:49d52d72…`.
- **Positive check that the code under test is in the artifact** — not merely that the image is
  new: `RecordBreakerClose` + `IsBreakerOpen` present in the shipped `Core.dll`,
  `DriverResilienceStatusPublisherService` in `OtOpcUa.Host.dll`, `EvaluationResult` +
  `ClearCompiledScripts` in `Runtime.dll`; assemblies stamped 2026-08-07 09:29 UTC.

---

## 1. adminui-followups — CLOSED by audit, no code written

`deferment.md` called this "the largest unexecuted plan in the tree" and "the only substantial
engineering left" (19 pending / 0 completed). The plan's own first step is an overlap audit, and the
audit says **every task is already shipped, superseded by v3, or moot.** Per-task evidence is in
`docs/plans/2026-05-29-adminui-followups.md.tasks.json`; the summary:

| Group | Verdict |
|---|---|
| WS1 Tasks 1, 8 (`CollectionEditor<TRow>`, OpcUaClient endpoints) | **Shipped in the plan's own shape** — `CollectionEditor.razor`, consumed by `OpcUaClientDriverForm.razor` |
| WS1 Tasks 2–7 (per-driver tag/device editors ×6) | **Superseded by v3** — `TagConfigEditorMap` (11 editors) + `DeviceFormMap` (9 device forms) |
| WS2 Tasks 9, 10 (resilience typed form) | **Shipped**, deliberately narrower — see below |
| WS3 Tasks 11–16 (LDAP→role map) | **Shipped** end to end, incl. the login merge at `AuthEndpoints.cs:100` |
| WS4 Tasks 17, 18 (comment cleanup) | **Moot** — the files were deleted by v3 Batch 2 and #542 |

Two things worth carrying forward:

- **The plan's premise died with v3.** WS1 rests on "driver-embedded tag/device lists in
  `DriverConfig` JSON are the runtime source of truth". In v3 tags are `Tag` rows authored in `/raw`
  and the deploy artifact populates the driver's `RawTags`
  (`ModbusDriverFactoryExtensions.cs:98-102`), so editing a list on a driver form no longer decides
  what the driver polls. The capability was delivered, in a different shape.
- **WS2 shipped narrower than designed, correctly.** Two designed fields no longer exist:
  `RecycleIntervalSeconds` went with the scheduled-recycle machinery (#522) and the bulkhead knobs
  were deleted end-to-end by archreview R2-02 because `Build` never added a concurrency limiter.
  Authoring either today would re-create an inert knob.

The supersession claims were checked for **reachability**, not just existence —
`TagConfigEditorMap.Resolve` is dispatched from `RawTagModal.razor:79` and
`RawManualTagEntryModal.razor:92`, `DeviceFormMap.Resolve` from `DeviceModal.razor:45`, both guarded
by `DriverFormMapParityTests` + `DriverDispatchMapParityTests`. A "superseded by X" whose X is itself
dead surface is the exact failure this repo has hit repeatedly.

---

## 2. R2-03 T11 — full-suite sweep ✅ PASSED

| Leg | Result |
|---|---|
| Solution build | **0 errors** |
| `Runtime.Tests` (whole project) | **555/555**, 3 skipped |
| `RoslynVirtualTagEvaluatorTests` | **20/20** |

The 3 skips are the deliberate `EquipmentDeviceBindingRetired` tombstones, matching the count
`deferment.md` §4.4 predicts. (The plan names the suite `RoslynVirtualTagEvaluatorTests`; it lives in
`Host.IntegrationTests`, not a project a solution-wide `--filter` reaches by default.)

---

## 3. R2-03 T12 — VT fail→Bad→recover ✅ PASSED, with one finding

Both blockers recorded against the previous attempt are **gone**: the Modbus fixture at
`10.100.0.35:5020` is now reachable from inside the rig (so driver data flows and a VT can be staged
Good), which is what made this gate runnable at all.

**Method.** The first two attempts produced a Bad result whose value was empty, but each followed a
deploy — and a deploy creates a fresh VT child with no last value, which is *correct by design* (the
inputs-ready gate that suppresses a cold-start flash, `VirtualTagActor.cs:246`). To isolate the
variable the trip was made **data-driven and deployment-free**: the script fails on a raw tag's
value, and that value was flipped by writing Modbus holding register 205 directly on the fixture, so
the VT child survives across the transition.

```
[09:51:57] GateVt = 42 (0x00000000)     Good
[09:52:51] GateVt =    (0x80020000)     Bad_InternalError   <- reg205 := 1, no deployment
[09:53:26] GateVt = 42 (0x00000000)     Good                <- reg205 := 0, no deployment
```

- ✅ **Degradation** — script failure degrades the node to `0x80020000 BadInternalError` instead of
  freezing on the last Good. Server-side cause confirmed in the log:
  `VirtualTag VT-gatevt: script execution threw / System.InvalidOperationException`.
- ✅ **Recovery** — returns to Good with the same value, exercising the dedup bypass.
- ✅ **P7 no-recompile spot-check** — two byte-identical redeploys (`revisionHash` identical across
  both, `94f68785…`) caused **no disturbance at all**: `Ever went BAD: 0`, VT held Good=42.
  Production has no compile log or metric, so the observable is the *consequence* of not
  recompiling; the gate itself is pinned by `VirtualTagHostActorTests` (exactly two clears across
  first-generation / identical-redeploy / edited-expression).

### ⚠️ Finding — "carrying the last-known value" never reaches the client

`VirtualTagActor`'s xmldoc (`:22`) and the R2-03 row in `archreview/plans/STATUS.md` both state the
Bad publish carries the last-known value. **A client never sees it.** Two independent channels agree:

- subscription: `GateVt = (0x80020000)` — value empty;
- raw OPC UA **Read** service (which returns a `DataValue` rather than throwing):
  `StatusCode 0x80020000, Value <NULL>`.

The VT logic is **not** at fault. `OnEvaluationFailed` sends `_hasLastValue ? _lastValue : null`, and
a new regression test — `VirtualTagActorTests.The_Bad_result_carries_the_last_known_value` — proves
the actor emits `Bad` carrying `42`. The value is lost below the actor, in the OPC UA layer
(`OtOpcUaNodeManager.WriteValue` assigns `variable.Value` unconditionally, so the loss is at
Read/Publish encoding, where a Bad `StatusCode` conventionally carries no value).

**Why no test caught it:** every existing failure assertion in `VirtualTagActorTests` checks
`.StatusCode` alone (`:87, :101, :123, :142, :180`) — the "carrying the last-known value" half was
asserted **nowhere**. That test is now added.

Not filed as a defect because the client-visible behaviour may be the correct OPC UA reading; what is
certainly wrong is the **documentation**, which promises an operator something no client can observe.

### Confirmed still true: the Client.CLI `read` limitation

`read` surfaces a Bad status as a `ServiceResultException` from `FetchValueAsync` rather than
printing it — exactly as the previous pass recorded. `ReadCommand.cs:52` does print `Status:`, but
the SDK throws before reaching it. **Use `subscribe`** (prints value *and* status, tracks
`everBad`), or a raw Read.

---

## 4. R2-10 T10 — breaker-OPEN ❌ NOT ACHIEVABLE AS SPECIFIED

The gate asks to point a driver at a dead endpoint and watch the breaker open. **It cannot open that
way, for a structural reason** — not because of the rig. Three attempts, each failing at a different
point, together characterise it:

| Attempt | What happened |
|---|---|
| Spare Modbus driver → blackhole (`sql:1433`, accepts TCP, never answers Modbus) | Driver had **no pollable tags** (its only tag skipped: "tag config did not map to a definition") — no capability calls at all |
| `MAIN-modbus` (13 live tags) → same blackhole | Driver **connected**, reads failed (`MainPressure` → `0x80050000 BadTimeout`, row showed DEGRADED / "Modbus socket closed mid-response") — yet tracker stayed `ConsecutiveFailures = 0`, panel showed **"All targets healthy."** |
| `line3-sql` → unusable connection string (Sql `ReadAsync` *does* propagate) | Fails in **`InitializeAsync`** (liveness check) before ever reaching a wrapped Read |

**Root cause.** The breaker only sees failures that propagate as exceptions out of a
`CapabilityInvoker`-wrapped call. On the dead-endpoint path nothing does:

1. **`IDriver.InitializeAsync` is not wrapped.** `DriverInstanceActor.cs:707` calls it directly; the
   invoker wraps only Read / Write / Subscribe / Discover / AlarmAcknowledge (`:774, :827, :870,
   :899, :1081`). A driver that cannot connect therefore contributes **zero** breaker input — which
   directly refutes the gate's premise.
2. **Modbus swallows per-tag failures, in both directions.** `ReadAsync` converts them to
   `Degraded` health (`:365-375, :838-855`) and `WriteAsync` converts them to a `WriteResult` status
   code — neither throws. So even the fallback trigger the plan suggests ("a rapid idempotent
   write-to-dead-endpoint") cannot work for Modbus.

The breaker itself is configured to open on **2 failures in a 30 s window** at ratio 1.0
(`DriverResiliencePipelineBuilder.cs:168-175`), so the previously-recorded explanation — that
R2-09's ConnectionBackoff throttles retries below the threshold — is not the whole story and is
arguably not the operative one: the failures never enter the pipeline at all.

**Operator consequence worth recording:** for the scenario operators care about most — a driver that
cannot reach its device — the resilience panel is **permanently quiet**. It showed *"All targets
healthy."* for a driver that was DEGRADED, timing out every read against a blackhole.

**What WAS verified live:** the panel renders on `/hosts` (rehomed there by #546 — the plan's "on the
driver page" step is stale, v3 Batch 2 retired those pages), the expanded row shows the RESILIENCE
block, and `DriverResilienceStatusPublisherService` is publishing real tracker state every 5 s
(`BreakerOpen`, `ConsecutiveFailures`, `CurrentInFlight`, `LastBreakerOpenUtc` all observed on the
wire). The reader/store/panel chain is live; only the breaker-open **state** remains unforced.

> Note: `DriverResilienceStatusChanged` going to `$DeadLetterWithNoSubscribers` on the node is
> **expected**, not a defect — the rig runs Phase-5 telemetry in `Grpc` mode, where the node always
> double-publishes and central consumes via the gRPC dialer, leaving the DPS copy unsubscribed.

**To actually close T10** someone must either wrap the connect path, stop drivers swallowing
capability failures, or pick a driver whose *wrapped* call fails after a successful init (Sql's
`ReadAsync` propagates — it needs a backend that dies *after* initialization, e.g. `docker pause` on
a dedicated fixture). Left OPEN with this root cause recorded.

---

## 5. driver-reconfigure-while-faulted Task 3 ✅ PASSED

The task was deferred in 2026-06 with "skip the live test part"; its block ("user-driven sign-in")
no longer holds because docker-dev runs the AdminUI with `DisableLogin`.

```
before   central-1 startedAt=2026-08-07T09:50:29.939876456Z restarts=0
10:16:18 DriverInstance DRV-ce2fb5217f5d: initialize failed: Connection refused   <- bad config (:5999)
10:22:13 DriverInstance DRV-ce2fb5217f5d: connected                                <- corrected config (:5020)
10:22:13 DriverInstance DRV-ce2fb5217f5d: subscribed to 1 refs
after    central-1 startedAt=2026-08-07T09:50:29.939876456Z restarts=0
```

The driver **adopted the corrected config and connected while faulted**, and `startedAt` is
byte-identical before and after with `restarts=0` — **no node restart**, which is the whole point
(pre-fix it would retry the stale config forever).

---

## 6. Rig state

Restored to as-found: Modbus device back on `10.100.0.35:5020`, `line3-sql` back on `DevSql`,
all `ResilienceConfig` overrides cleared, the `SC-gatevt` script restored to its original body.
Post-restore verification: `MainPressure = 7 (Good)`, `GateVt = 7 (Good)` — the VT is back on its
original one-line body, so it mirrors `MainPressure` again rather than the `×6` used during the gate.

One anomaly recorded rather than explained: both centrals were **recreated** at 09:49:40/09:49:50
(new `Created`, `RestartCount=0`) by something this session did not issue. It postdates the
observations above and the old container's logs went with it, so it is noted, not attributed.

---

## 7. Addendum 2026-08-08 — R2-10 T10 ✅ PASSED after fixing the structural gap

§4 concluded the gate was not achievable as specified because the connect path never entered the
resilience pipeline. The user chose to fix that ("do option 1"), and the gate then passed against
the fixed build. Branch `feat/wrap-driver-init-in-resilience`.

### The fix

- **`DriverCapability.Initialize`** — `IDriver.InitializeAsync` routes through the capability
  invoker in `DriverInstanceActor.InitializeAsync` (both the initial connect and the
  `Reconnecting` retry path; the best-effort teardown `ShutdownAsync` stays deliberately
  unwrapped). Keyed on the driver-instance id, like a bulk subscribe.
- **No pipeline retry for Initialize** — the actor's `RetryConnect` timer is the retry loop.
  Enforced at the parser funnel (`retryCount` override forced to 0, same shape as the
  write-shaped rule) and in every tier default.
- **Breaker threshold 2, not the Read-sized 5** — the breaker samples 30 s and the reconnect
  timer fires every 10 s, so at most three init attempts fit one window; threshold 5 could never
  open. Pinned by `Initialize_BreakerThreshold_IsReachableAtTheReconnectCadence` +
  `Initialize_BreakerOpens_AtTierDefaults_AndFastFailsTheNextAttempt`.
- **Companion fix — respawn resets tracker snapshots.** `DriverCapabilityInvokerFactory.Create`
  now calls `DriverResilienceStatusTracker.Invalidate(driverInstanceId)` alongside the pipeline
  invalidation. Without it, a deploy during an outage discards the only breaker whose `OnClosed`
  could clear the OPEN flag, stranding a false "Breaker OPEN" chip forever. Unreachable before
  this change made breaker-open reachable; found by asking what clears the flag, not by a test.
- Wiring pinned by `Initialize_routes_through_the_capability_invoker` +
  `An_invoker_fast_fail_on_init_keeps_the_actor_retrying_until_recovery` (falsification-controlled:
  both go red with the actor wrap stashed; `Create_resets_the_instances_tracker_state_with_its_pipelines`
  goes red with the factory call stashed).

### The gate run (docker-dev, image `df04c76ce06b`, symbol-verified on both centrals)

The staged failure was free: `rtu-gate` (`DRV-3616122eb53d`) was already authored against
`10.100.0.35:5021`, where nothing listens — connection refused on every init.

1. **Breaker OPEN, both replicas** — within ~20 s of boot, both centrals logged
   `circuit-breaker OPENED: instance=DRV-3616122eb53d … capability=Initialize … Connection refused`,
   re-opening each half-open probe cycle. Bonus coverage: `DRV-opcua1` (invalid config → init
   throws) also tripped — a permanently-misconfigured driver was equally invisible before.
2. **Panel chip** — `/hosts` → rtu-gate → Controls showed the red **"BREAKER OPEN · OPENED 19S
   AGO"** chip in the RESILIENCE section, row FAULTED with "Connection refused".
3. **Control** — healthy `MAIN-modbus` recorded zero breaker events throughout.
4. **Recovery clear** — a socat forwarder (`5021 → 5020` on the docker host) brought the endpoint
   alive; both centrals logged `circuit-breaker closed (recovered)` + `reconnected` within 1 s of
   each other, and the panel dropped to "All targets healthy."
5. **Teardown** — socat removed; 5021 dead again (as-found). rtu-gate returns to a truthfully-OPEN
   breaker chip, which is the feature working.

### Two nuances against the original T10 wording

- **"Both-replica chips"** resolves as both nodes' *trackers* tripping (proven in both logs)
  rendered through the single **active** AdminUI. Traefik's `/health/active` check deliberately
  routes the UI to the active admin node only — the standby returns 503
  (`Standby: active node for role 'admin' is …`), so there is never a second replica serving
  chips. Discovered while trying to force the sticky-cookie LB onto central-1.
- **The staleness-dim leg was not exercised.** Both pair nodes publish under the same
  `(instance, host)` key, so staleness requires *every* publisher for a key to stop — deliberate
  node-kill churn this gate did not do. The dim is a pure render predicate (`IsStale` → opacity).

Also corrected while here: §4's "ConnectionBackoff throttles retries" explanation (the previously
recorded root cause) is confirmed wrong — the breaker config opens on 2 failures in 30 s; the
operative cause was always that no failure reached the pipeline at all.