Merge lane G1: OpcUaClient + MTConnect driver remediation (#743)

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT

docs/drivers/MTConnect.md

@@ -73,7 +73,7 @@ below).
 
 ## Capability surface
 
-`MTConnectDriver : IDriver, IReadable, IWritable, ISubscribable, ITagDiscovery, IHostConnectivityProbe, IRediscoverable`
+`MTConnectDriver : IDriver, IReadable, IWritable, ISubscribable, ITagDiscovery, IHostConnectivityProbe, IRediscoverable, IDriverDiagnostics`
 (`src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect/MTConnectDriver.cs`).
 
 `IWritable` landed 2026-08-05 (`b29623cf`), against the Agent's `AllowPut`
@@ -87,11 +87,12 @@ request/response handshake — is a different mechanism and remains deferred; se
 | Capability | Path | Notes |
 |------------|------|-------|
 | `ITagDiscovery` | `DiscoverAsync` — streams `/probe`'s Device→Component→DataItem tree into the address-space builder | `SupportsOnlineDiscovery = true`, `RediscoverPolicy = Once` |
-| `IReadable` | `ReadAsync` → one `/current` per call | **Not the production data path** — see below |
+| `IReadable` | `ReadAsync` → the live `/sample` pump's index while it is delivering and healthy, else one `/current` per call | **Not the production data path** — see below |
 | `IWritable` | `WriteAsync` → `POST {agent}/{device}?{dataItemId}={value}` on the Agent's `AllowPut` surface | **Off by default** — needs `Write.Enabled` **and** the Agent's `AllowPut`; CONDITION items are refused |
 | `ISubscribable` | `SubscribeAsync`/`OnDataChange` — the shared `/sample` long-poll pump | **The production data path** |
-| `IHostConnectivityProbe` | periodic `/probe` under `Probe.*` | No consumer wires it today — see [Known limitations](#known-limitations) |
+| `IHostConnectivityProbe` | periodic headers-only `/probe` ping under `Probe.*` — the status is read, the device model is neither downloaded in full nor parsed (review 2026-10-21 T-7; it used to parse the whole model every tick). Trade-off: a 200 from something that is not an Agent (a proxy's HTML page) now reads as reachable; the data paths still fail on it | Consumed — feeds `/hosts` (see [Known limitations](#known-limitations)) |
 | `IRediscoverable` | watches the Agent's `Header.instanceId` | No consumer wires it today — see [Known limitations](#known-limitations) |
+| `IDriverDiagnostics` | `GetDiagnostics()` — `/sample` effective count, lag, saturated chunks, count raises | Shown on `/hosts` (review 2026-10-21 T-5) |
 | `IAlarmSource` | CONDITION observations off the same `/sample` pump → Part 9 transitions keyed by the alarm tag's RawPath | Opt-in per raw tag (`alarm` object) — see [CONDITION alarms](#condition-alarms-626) |
 
 ### Feature status at a glance (#613)
@@ -113,6 +114,25 @@ subscribes once and lives off `OnDataChange`. `ReadAsync` exists for the
 Client CLI (`... read -n ...`) and for the unit/integration suites; it is
 correct and tested, just not on the hot path.
 
+**Read path (review 2026-10-21 T-4 / T-2).** While the current-generation `/sample` pump is running, has
+delivered since it started (a chunk, heartbeat or re-baseline) and its stream is not degraded, a read is answered
+from the pump's observation index — **read only**; the pump stays its single writer — with no Agent round trip.
+Otherwise (no subscription, a pump still catching up after a resubscribe, a degraded stream) it falls back to one
+`/current`, whose per-batch index is now built over a tag map frozen once per binding. **Stale window:** a
+stream that stalls without failing (no chunk, no heartbeat) still serves Good reads from the index until the client's
+idle watchdog fires (2 × `HeartbeatMs` + `RequestTimeoutMs`) — the same bounded window subscriptions already have. During
+an Agent-restart re-baseline reads fall back to `/current`. A served read also clears a
+read-path degradation an earlier failed `/current` left (the read path is answering). The `/current` fallback now
+checks the answer's `instanceId`: a changed one (an Agent restart seen by a browse-only driver with no pump) drops the
+cached `/probe` model and raises `OnRediscoveryNeeded` once per distinct id — it used to be detected only by the pump.
+
+**An Agent refusal is a reply (health contract).** An `MTConnectError` document answering `/current` (e.g.
+`NO_DEVICE` for a wrong `DeviceName` scope) still fails the batch (`BadCommunicationError` per ref), but it is the
+Agent answering, not a comms fault: the read path is not degraded by it (that would let the comms-loss watchdog
+reconnect-loop the driver over a configuration problem), `LastSuccessfulRead` advances, a prior read degradation
+clears, and the Agent's text is logged once per distinct refusal. An answer that is not an MTConnect document at all
+(a proxy's HTML page, truncated XML) stays a comms fault.
+
 ## Minimum deployment
 
 ```jsonc
@@ -146,11 +166,11 @@ Read off `MTConnectDriverOptions` / `MTConnectDriverConfigDto`
 
 | Key | Default | Notes |
 |---|---|---|
-| `AgentUri` | — (required) | Agent base URI; the driver appends `/probe`, `/current`, `/sample` |
+| `AgentUri` | — (required) | Agent base URI; the driver appends `/probe`, `/current`, `/sample`. Any `user:pass@` (and query) is **redacted** from `HostName` (the `/hosts` key), every log line and every exception / health message (review 2026-10-21 T-8); a URI without credentials is shown unchanged |
 | `DeviceName` | `null` | **Double duty.** (1) Scopes requests to `{AgentUri}/{DeviceName}/...`; `null` = whole Agent. (2) The write path's fallback device — the device a write is addressed to when the probe model does not say which device declared the DataItem (see [Write-back](#write-back--observation-injection-off-by-default)) |
 | `RequestTimeoutMs` | `5000` | Per-call deadline for `/probe` and `/current` |
 | `SampleIntervalMs` | `1000` | The Agent's `/sample?interval=` query param |
-| `SampleCount` | `1000` | The Agent's `/sample?count=` query param |
+| `SampleCount` | `1000` | The Agent's `/sample?count=` query param. **Caps subscription throughput** at `SampleCount / SampleIntervalMs` — see [Subscribe](#subscribe--the-sample-pump) for the adaptive raise |
 | `HeartbeatMs` | `10000` | The Agent's `/sample?heartbeat=` query param — also the watchdog's liveness window |
 | `Probe.Enabled` / `Probe.IntervalMs` / `Probe.TimeoutMs` | `true` / `5000` / `2000` | Background connectivity-probe knobs (mirrors `ModbusProbeOptions`) |
 | `Reconnect.MinBackoffMs` / `MaxBackoffMs` / `BackoffMultiplier` | `0` / `30000` / `2.0` | Geometric backoff after a failed request or a dropped `/sample` stream |
@@ -324,6 +344,26 @@ the two need disambiguating, not just OR-ing together). On a changed
 `/hosts` page — see [Known limitations](#known-limitations) for what that does
 and does not do.
 
+**Lag measurement and the adaptive `count` (review 2026-10-21 T-5).** Throughput is capped at `SampleCount`
+observations per chunk, one chunk per `SampleIntervalMs`; a machine producing more used to fall behind the ring
+buffer forever, re-baselining every buffer-length (dropping intermediate transitions) with nothing surfacing it.
+Each chunk is now measured: `lag = lastSequence + 1 − nextSequence` (unknown when the Agent omits `lastSequence`)
+and whether it was **saturated** (carried the full `count`). Three consecutive saturated chunks with a positive (or
+unknown) lag reopen the stream at the cursor reached with `count` doubled, up to 16× `SampleCount` (max 65 536). A
+raised stream that delivers nothing (an Agent refusing the larger count) steps back to the last count that delivered
+and stops raising; at the ceiling a still-behind stream logs one Warning per pump. Every new pump starts from the
+configured `SampleCount`. An Agent that refuses the raised count with an `MTConnectError` under HTTP 200 goes through
+the ordinary unusable-answer path first — one transient Degraded blip and a backoff — before the step-back. The
+driver now implements `IDriverDiagnostics`: `sampleCount` (effective), `sampleLag`,
+`sampleSaturatedChunks` and `sampleCountRaises` ride the `/hosts` diagnostics bag.
+
+**A pump carries the session generation it was started against (review 2026-10-21 T-1).** A lifecycle restart
+retires the running pump (or abandons it after its bounded teardown wait); if that pump finishes after the new
+session is installed, its cursor publish, a late chunk and a late re-baseline are all dropped — checked under the
+same lock the new session is installed and the generation bumped under. It used to land the OLD Agent's
+`instanceId` + sequence on the new session, so the next pump read its first chunk as an Agent restart (index
+cleared, one Bad flicker on every subscribed tag, a needless re-baseline).
+
 ## Write-back — observation injection, off by default
 
 > ⚠️ **A write sets what the Agent *reports*, not what the machine does.** The value
@@ -391,8 +431,26 @@ step answers a different operator question:
 different `level|text` wire format, and are out of scope for v1.
 
 **The Agent's refusal text is logged at Warning and never stored.** It stays out of
-`DriverHealth.LastError`, which is type-only by repo-wide discipline, and a refused write
-is not evidence the read path degraded — so a refusal leaves driver health untouched.
+`DriverHealth.LastError`, which is type-only by repo-wide discipline. A refused write is not
+evidence the read path degraded, but it **is** the Agent answering: any answer — accepted or
+refused — advances `LastSuccessfulRead`, with the state following the read / `/sample` flags
+(a write cannot clear either of those paths' degradation). A write that got **no** answer
+(transport failure, deadline) sets **Degraded** with the stamp preserved and a type-only
+`LastError` (review 2026-10-21 I-3, coordinator ruling). One success writer is shared: that
+degradation is cleared by the next answered **write**, the next answered **`/current` read**, or
+the next **`/sample` chunk** the current session's pump delivers (review r2 N-5: a delivered
+chunk proves the Agent is alive — the analogue of OpcUaClient's good keep-alive — so it cannot
+stick on a subscription-only driver). A read served from the live pump index does not clear it
+by itself; the chunks behind that index do.
+
+**One request per device (review 2026-10-21 T-6).** The slots that survive the refusals above
+go out as one `POST {agent}/{device}?id1=v1&id2=v2…` per device (the Agent's PUT surface takes
+several pairs) instead of one POST per slot. A batch closes at a slot naming another device or
+repeating a DataItem already in it, so two writes to one DataItem still land in request order.
+An Agent refusal of a multi-slot request is retried slot by slot, so a refused DataItem costs
+only its own slot (`BadCommunicationError`); a transport failure fails that request's slots. An Agent may apply
+some pairs of a multi-pair PUT before refusing it, so the per-slot retry can re-send a pair that already landed —
+harmless for value-setting (the Agent drops an identical repeat).
 
 **Device resolution needs no extra config.** `TagBinding.WriteTargetsByDataItemId`,
 built at bind time from a walk of the cached probe model, remembers which device declared

docs/drivers/OpcUaClient.md

@@ -25,7 +25,9 @@ out-of-process isolation.
 | Project | Target | Role |
 |---------|--------|------|
 | `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient/` | net10.0 | In-process driver — session lifetime, read / write / subscribe / alarm / history passthrough |
-| `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Browser/` | net10.0 | `IDriverBrowser` — live address-picker browse used by the AdminUI |
+| `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Browser/` | net10.0 | `IDriverBrowser` — live address-picker browse used by the AdminUI. Its side-panel `AttributesAsync` reads a variable's DataType / ValueRank / UserAccessLevel and maps them through the same `UpstreamAttributeMapping` discovery uses; it authenticates Anonymous, Username **or Certificate** through the driver's own `OpcUaUserIdentityFactory`, resolving `secret:` credentials through the host's secret store (fail-closed) — review 2026-10-21 O-5 (it was a stub, and Certificate was refused) |
+| | | ⚠️ Browse Certificate auth reads `UserCertificatePath` on the **AdminUI host's** filesystem, which may differ from the driver node's; the browse session disposes the certificate it loaded when it closes. |
+| `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Session/` | net10.0 | The SDK-typed layer the driver AND the browser share: `NamespaceMap`, `UpstreamAttributeMapping`, `OpcUaUserIdentityFactory` |
 | `src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.Contracts/` | net10.0 | Config records + enums bound from `DriverConfig` JSON |
 
 ## Minimum deployment
@@ -53,6 +55,8 @@ for every field (security policy/mode, auth type, session timeout, keep-alive,
 browse root, node/depth caps). `ReconnectPeriod` was **deleted** (#676 — unused since #670 removed the SDK reconnect
 handler, and the AdminUI no longer shows it); stored configs that still carry `reconnectPeriod` keep parsing in the
 factory, the Test-Connect probe and the browse picker (unknown keys are skipped), and the form drops it on the next save.
+`ProbeTimeoutSeconds` (default 15, AdminUI "Probe timeout") caps the Test-Connect probe's deadline — it can shorten
+the budget the host hands the probe, never extend it (review 2026-10-21 O-4; it used to be authored and consumed by nothing).
 
 ### Endpoint address: the configured host and port win (#676)
 
@@ -70,12 +74,40 @@ A single `Session` per driver instance; subscriptions multiplex onto it. Stored
 session-relative `ns=N` index) so a remote namespace-table reorder across a
 restart doesn't silently re-point references at the wrong namespace.
 
+**Concurrency (review 2026-10-21 O-6).** Reads (including the post-outage recovery / reconcile reads) and acks
+share one gate; **writes and the discovery browse take none** (the SDK session is safe for concurrent service
+calls, and the host already bounds write concurrency and serialises per tag); subscription management — data and
+alarm subscribe/unsubscribe and ConditionRefresh sweeps — has its own gate. One gate used to serialise all of it,
+so a write stalled for the operation timeout blocked every read and subscription create, and a discovery browse
+would have blocked the data plane for its whole duration.
+
+**Lifecycle generation.** Every `InitializeAsync` / `ShutdownAsync` takes a new generation; the session, namespace
+map and every health write land only while their generation is current. An init the host abandoned that finishes
+after the next shutdown discards (closes + disposes) its session instead of installing it, and a read whose
+response arrives after a shutdown does not write `Healthy` over it. A throw after the session activated closes
+that local session (O-2). `ShutdownAsync` spends **one** 4 s budget on wire calls — every subscription delete
+concurrently, then the close with what is left — and abandons waits the SDK does not cancel (O-9; it used to be
+5 s per subscription, serially, inside the actor's `PostStop`).
+
 ### Upstream outage and recovery (Gitea #670)
 
 The session keep-alive (`KeepAliveInterval`, default 5 s) is the liveness signal:
 
 - **Good keep-alive / any data or event notification** advances `DriverHealth.LastSuccessfulRead` — a
   subscription-only driver never calls `ReadAsync`, so without this its read stamp froze at connect time.
+  A good keep-alive with no outage latched also **clears a `Degraded` left by a failed read/write** (it used to move
+  only the stamp, so a subscription-only driver stayed `Degraded` until its next re-initialise).
+- **Read / write / ack responses are replies.** Any service response — per-node refusals such as `BadNotWritable`
+  included — restores `Healthy` and advances the stamp; only a call that got **no** response (transport failure,
+  timeout — including an SDK request timeout surfacing as a cancellation the caller did not cause) writes `Degraded`
+  with the stamp preserved; acks follow the same rule. **Deliberate deviation:** a call the *caller* cancelled touches
+  no health — including a write the host cancelled at its own write deadline against a hung upstream; the session
+  keep-alive owns liveness and reports that outage itself. A reply that was in flight across a bad keep-alive advances
+  the stamp but does not write `Healthy` while the outage is published — the keep-alive recovery ends the outage.
+  An ack none of whose condition ids resolves sends nothing and logs a Warning — it does **not** throw (a deterministic
+  per-ack refusal must not count toward the host's AlarmAcknowledge circuit breaker, or five re-acks of one
+  unresolvable condition would block every genuine ack). Only a call with no reply throws; with no session an ack
+  fails `BadNotConnected` (before or inside the gate alike).
 - **Bad keep-alive** ⇒ health `Degraded` (`LastError = "keep-alive: <status>"`, read stamp preserved), host status
   `Stopped`, and — **once per outage**, not per keep-alive — every subscribed ref is published `BadNoCommunication`
   with its last value retained and **source + server timestamp = the detection time** (live-gate F1: stamped with the
@@ -231,6 +263,12 @@ initialise (the AdminUI form drops them on the next save). Pinned by
 > forgotten when the subscribed set changes (the host rebuilt, recreating the nodes inactive). Index keys are server-stable (`nsu=`), so a namespace reorder across a re-initialise keeps
 > matching. docker-dev seeds an opc-plc driver (`MAIN-opcua`); the alarm tag is
 > created only when `@OpcPlcAlarmSourceNodeId` is set in `docker-dev/seed/seed-clusters.sql`.
+>
+> **An ack that never got a reply fails (review 2026-10-21 O-8).** A transport failure on the `Acknowledge` call
+> (closed channel, faulted session) — or a session lost while the ack queued — now throws with its status
+> (`ServiceResultException`, e.g. `BadConnectionClosed` / `BadNotConnected`), so the host logs the ack as failed
+> instead of applied. A *reply* still returns normally, including per-ack refusals such as
+> `BadConditionBranchAlreadyAcked` (logged per ack): a refusal is the server answering.
 
 > This driver does **not** implement `IRediscoverable` — there is no
 > push-driven rediscovery signal from a remote OPC UA server in this driver.