Merge review R1 fixes: removed driver's resilience tracker released; SDK timed-unshelve no-op (engine owns expiry); commented re-ack stored as AddComment; native-ack Primary verdict published from start; drain gate SharesRowsWithPeer; malformed cache bytes don't adopt the revision; HostAbortedException passthrough; ClusterRoleInfo seed filtered

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe

docs/AlarmHistorian.md

@@ -93,6 +93,16 @@ Two postures are deliberate:
 - **Unset (the default) means drain.** So does an unpublished view. A deployment
   that runs no redundancy never publishes a role, and defaulting closed would
   silently stop its alarm history forever.
+- **Only a shared queue is gated.** Standing down is safe only because a peer
+  holds the same rows, so the gate applies only when
+  `LocalDbReplicationPolicy.SharesRowsWithPeer` holds:
+  `LocalDb:Replication:ApiKey` **AND** (`LocalDb:Replication:PeerAddress` **OR**
+  `LocalDb:SyncListenPort > 0`) — an authenticated sync path in either
+  direction. Any other node always drains its own queue. This is deliberately
+  stricter than the table-registration predicate (`PeerAddress` OR `ApiKey`):
+  a key with no dial and no listener (e.g. a fleet-wide secret on an unpaired
+  node) captures changes but shares them with nobody, and gating it would park
+  its events behind a Primary that does not hold them (review R1 m3).
 - **A gate that throws means "not now", never permission.** Draining on a gate
   fault would put a pair into dual delivery exactly when its redundancy signal is
   sick.

docs/AlarmTracking.md

@@ -350,15 +350,16 @@ is client-visible.
 > (ScadaBridge included) that acks against the Secondary now gets a Bad result
 > instead of a Good that later reverts — so ack against the Primary (the higher
 > ServiceLevel). Scripted conditions are **not** gated: their engine runs on
-> both nodes, so an ack on the Secondary is real. Residual: the published
-> verdict changes only on a redundancy snapshot, so in the boot window of a
-> multi-node cluster (no snapshot yet) the node still accepts, while
-> `DriverHostActor`'s live check may drop the route — the old flap, limited to
-> that window.
+> both nodes, so an ack on the Secondary is real. The published verdict
+> mirrors `DriverHostActor`'s live check from the moment the host starts, not
+> only from the first redundancy snapshot: the host publishes it once at start
+> and again whenever cluster membership changes, so in the boot window of a
+> pair (role still unknown, a driver peer Up) the gate already denies, exactly
+> as the route does (review R1 m2 — previously the view sat on its single-node
+> "allow" seed until the first snapshot, reopening the flap for that window).
 
 > **Best-effort during reconnect.** If a native ack arrives while the owning
-> driver is `Connecting`/`Reconnecting` (or, in the boot window above, on a
-> node whose role is not yet known), it is dropped with a debug log and **not**
+> driver is `Connecting`/`Reconnecting`, it is dropped with a debug log and **not**
 > retried upstream — the OPC UA Part 9 `AlarmConditionState` was already
 > committed locally at step 1, so the rare outcome is "acked locally but not
 > propagated to AVEVA." Re-issue the ack once the driver is reconnected if
@@ -408,8 +409,10 @@ On allow, the handler publishes a `Commons.OpcUa.AlarmCommand` onto the
 action is a settable `Action<AlarmCommand>` injected at boot by the
 hosted service.
 
-`OnTimedUnshelve` (the SDK's automatic unshelve timer) bypasses the
-operator gate — it is system-initiated.
+`OnTimedUnshelve` (the SDK's automatic unshelve timer) is a **no-op** — it
+routes nothing (review R1 M2). The engine owns timed-shelve expiry (its 5 s
+shelving check records `AutoUnshelve` by `system`), and its re-projection
+unshelves the node; see §"`UnshelveTime` is the real remaining time" below.
 
 ### One condition event per method call (Gitea #587)
 
@@ -448,7 +451,14 @@ reports nothing.
   AddComment always reports (the comment is the change), and so does a re-ack
   **carrying a comment** (Gitea #597): the SDK writes that comment to the node,
   so it is reported once, with Acked/Confirmed unchanged. Before #597 the
-  comment landed on the node but no subscriber ever saw it.
+  comment landed on the node but no subscriber ever saw it. For a **scripted**
+  condition the comment is also durable (review R1 m1): the routed Acknowledge
+  reaches an engine that is already acknowledged, and
+  `Part9StateMachine.ApplyAcknowledge` records a commented re-ack exactly as an
+  AddComment (kind `AddComment`, the caller as user) — persisted and replicated
+  with the condition state, the original ack's `LastAck*` fields untouched. An
+  uncommented re-ack is still a no-op in the engine. The same applies to a
+  commented re-ack from the AdminUI `/alerts` page.
 - **A native re-ack is still routed to the driver** (deliberate, #597). The
   route is fire-and-forget, so a node showing Acked does not prove the
   upstream system acknowledged — an ack dropped during a reconnect leaves
@@ -471,10 +481,14 @@ reports nothing.
   engine's projection carries its expiry (`AlarmConditionSnapshot.UnshelveAtUtc`),
   so `ShelvingState/UnshelveTime` counts down to the engine's expiry whichever
   path shelved it (the AdminUI `/alerts` timed shelve included). This arms the
-  SDK's own unshelve timer at the real instant; it changes no node state and
-  only routes an `Unshelve` (user `system`) back to the engine — a no-op if the
-  engine's 5 s expiry sweep got there first, otherwise it simply lands the
-  expiry on time.
+  SDK's own unshelve timer at the real instant, and `OnTimedUnshelve` — the
+  timer's callback — is deliberately a **no-op** (review R1 M2): it used to
+  route an operator-style `Unshelve` by `system`, which, firing before the
+  engine's 5 s expiry sweep, pre-empted the engine's `AutoUnshelve` on every
+  expiry, from both pair nodes, rewriting the audit trail as a manual unshelve.
+  The SDK's timer changes no shelving state itself, so the node stays
+  TimedShelved (with `UnshelveTime` at zero) for up to one sweep after expiry,
+  until the engine's re-projection reports the one expiry event.
 - **Accepted, not fixed — the SDK's `EventsMonitored` unit for late conditions**
   (Gitea #597 item 6, from the #586 review). The SDK's `AreEventsMonitored`
   counter never reaches a condition materialised under a folder that already had

docs/Configuration.md

@@ -218,7 +218,7 @@ See the [ClusterClient→gRPC migration design](plans/2026-08-06-clusterclient-t
 | `GrpcListenPort` | int | `0` | Dedicated **h2c-only** Kestrel port for the artifact gRPC service. `0` = disabled (nothing bound). Must differ from the main HTTP port and from `LocalDb:SyncListenPort` (h2c cannot share a cleartext HTTP/1 port). Mapped only on admin-role nodes. |
 | `ApiKey` | string | `""` | Shared bearer key the serve-side interceptor checks (constant-time, **fail-closed**: an unset key rejects every call). **Supply via env `ConfigServe__ApiKey` — never commit.** |
 
-**It is a dark switch; the serve side is always wired.** Central maps the `DeploymentArtifactService` behind the `ConfigServeAuthInterceptor` (path-scoped, `FixedTimeEquals`, fail-closed) whenever `ConfigServe:GrpcListenPort > 0`, in both modes — so flipping a node to `FetchAndCache` is a config change, not a redeploy. A `FetchAndCache` node that cannot fetch (all endpoints down, `NotFound`, SHA mismatch) **fails the apply and keeps last-known-good** — a null fetch is "no answer," never an empty configuration (the [#485](plans/2026-07-15-v3-batch4-address-space-plan.md) contract). At boot it restores served state from the LocalDb pointer with no SQL read; an empty or unreadable cache lands Steady-with-no-revision (the first dispatch fetches), never Stale.
+**It is a dark switch; the serve side is always wired.** Central maps the `DeploymentArtifactService` behind the `ConfigServeAuthInterceptor` (path-scoped, `FixedTimeEquals`, fail-closed) whenever `ConfigServe:GrpcListenPort > 0`, in both modes — so flipping a node to `FetchAndCache` is a config change, not a redeploy. A `FetchAndCache` node that cannot fetch (all endpoints down, `NotFound`, SHA mismatch) **fails the apply and keeps last-known-good** — a null fetch is "no answer," never an empty configuration (the [#485](plans/2026-07-15-v3-batch4-address-space-plan.md) contract). At boot it restores served state from the LocalDb pointer with no SQL read; an empty or unreadable cache — including a cached row whose bytes are malformed (review R1 m4) — lands Steady-with-no-revision (the first dispatch fetches, even of the same revision), never Stale.
 
 **The listener is dedicated h2c and coexists with the LocalDb-sync listener.** A fused admin+driver node with **both** `ConfigServe:GrpcListenPort` and `LocalDb:SyncListenPort` set re-binds its existing HTTP surface exactly once and adds both dedicated HTTP/2-only ports on top — see the same Kestrel-takeover caveat under [`LocalDb`](#localdb-pair-local-config-cache--driver-role-only).
 
@@ -431,7 +431,10 @@ replication stays inert until the sync port + peer are set.
 `PeerAddress` **OR** `ApiKey` non-blank — the OR because only the *dialling* half sets `PeerAddress`, so
 keying on it alone would strip change capture from every passive node and leave the pair converging in one
 direction only. `SyncListenPort` is deliberately **not** part of it (a port with no `ApiKey` is fail-closed
-and syncs with nobody). A configured node registers all four tables with `baselineExistingRows: true`, so
+and syncs with nobody). (The alarm-historian **drain gate** uses a stricter predicate on purpose —
+`ApiKey` AND (`PeerAddress` OR `SyncListenPort > 0`), `LocalDbReplicationPolicy.SharesRowsWithPeer` — because
+gating a node that shares its queue with nobody strands its alarm history; see `docs/AlarmHistorian.md`
+§"The drain gate".) A configured node registers all four tables with `baselineExistingRows: true`, so
 turning replication on converges the rows **already in the file** and not merely subsequent writes; an
 unconfigured node **deregisters** them at boot, dropping capture triggers an earlier build installed. ⚠️ The
 sync handshake compares registered-table digests **fail-closed**, so flipping either key is a

docs/Redundancy.md

@@ -526,13 +526,13 @@ Three data-plane surfaces are **Primary-gated** so a warm standby never touches
 
 Before archreview 03/S4 the unknown-role case defaulted to *allow* unconditionally, opening a dual-primary window: a freshly-booted secondary on a multi-node cluster serviced shared-device writes and emitted duplicate fleet-wide alerts for up to the ~10 s `RedundancyStateActor` heartbeat (and indefinitely if the snapshot's NodeId never matched this node). Membership is authoritative much earlier than DPS delivery, so an unknown role on a **multi-driver** cluster now default-**denies**. The driver member count is read live from `Cluster.State.Members` (Up members with the `driver` role); a non-cluster ActorRefProvider yields 0 ⇒ single-node posture. If a snapshot arrives that never mentions this node while a driver peer exists (the 03/S5 identity-mismatch shape), `DriverHostActor` logs a one-time Warning so the silent mismatch is diagnosable.
 
-**What a client sees during the boot window (multi-node, role unknown):** an inbound write is **rejected** (not queued — OT actuation seconds late is a surprise). The reject rides the sanctioned optimistic-write self-correction surface: the node reverts to its prior value with a transient Bad-quality blip and a Part 8 `AuditWriteUpdateEvent`; the client's synchronous write call still returned Good (exactly as any failed device write behaves since the #5 write-outcome self-correction). The rejection reason distinguishes the boot window (`"not primary (role unknown)"`) from a steady-state Secondary (`"not primary"`). Acks are dropped (Warning-logged when role-unknown). The alerts emit is skipped (the Primary peer publishes the single fleet copy). The window closes on the first delivered snapshot.
+**What a client sees during the boot window (multi-node, role unknown):** an inbound write is **rejected** (not queued — OT actuation seconds late is a surprise). The reject rides the sanctioned optimistic-write self-correction surface: the node reverts to its prior value with a transient Bad-quality blip and a Part 8 `AuditWriteUpdateEvent`; the client's synchronous write call still returned Good (exactly as any failed device write behaves since the #5 write-outcome self-correction). The rejection reason distinguishes the boot window (`"not primary (role unknown)"`) from a steady-state Secondary (`"not primary"`). Acks are dropped (Warning-logged when role-unknown) — and since review R1 m2 a native OPC UA Acknowledge is refused up front (`BadInvalidState`) in the same window: `DriverHostActor` publishes its Primary verdict to `IDriverPrimaryRoleSource` (the native-ack gate, and the MQTT Sparkplug primary-host claim) once at start and again on every cluster membership change, not only on redundancy snapshots, so the published verdict denies as soon as a driver peer is Up rather than sitting on the single-node "allow" seed until the first snapshot. The alerts emit is skipped (the Primary peer publishes the single fleet copy). The window closes on the first delivered snapshot.
 
 Under warm/hot redundancy both cluster nodes run `ScriptedAlarmHostActor` and evaluate scripted alarms, keeping each node's address space and engine state warm for instant failover. To avoid duplicate rows on `/alerts` and duplicate historian writes, only the Primary node publishes externally:
 
 - **`alerts` topic emission** — `ScriptedAlarmHostActor` and `DriverHostActor.ForwardNativeAlarm` subscribe to the `redundancy-state` DPS topic and cache the local node's `RedundancyRole`, then gate the cluster `alerts` publish through `PrimaryGatePolicy` (table above). The OPC UA condition-node write and inbound ack/shelve command processing remain **ungated** on both nodes so the secondary is always ready to serve clients after a failover.
 - **`HistorianAdapterActor` historization** — likewise Primary-gated so alarm historization is exactly-once across all alarm sources. The actor subscribes to the `alerts` DPS topic and translates each `AlarmTransitionEvent` → `AlarmHistorianEvent` before enqueuing it on the sink; scripted alarms therefore historize exactly once regardless of cluster size.
-- **The alarm sink's DRAIN** — a second, independent gate on the same decision. The enqueue gate above is not sufficient any more: since Phase 2 the store-and-forward buffer lives in the replicated LocalDb, so the Secondary holds a full copy of the Primary's queued rows and an ungated drain there would re-deliver every event continuously. `LocalDbStoreAndForwardSink` takes a `drainGate` fed from `IRedundancyRoleView` — a singleton `DriverHostActor` publishes its `PrimaryGatePolicy` verdict to, because the drain runs on a timer and cannot receive `RedundancyStateChanged` itself. A gated-off node reports `HistorianDrainState.NotPrimary`. Delivery is **at-least-once across a failover** by design: rows the old Primary delivered just before dying may not have replicated their deletes yet. See [AlarmHistorian.md](AlarmHistorian.md).
+- **The alarm sink's DRAIN** — a second, independent gate on the same decision. The enqueue gate above is not sufficient any more: since Phase 2 the store-and-forward buffer lives in the replicated LocalDb, so the Secondary holds a full copy of the Primary's queued rows and an ungated drain there would re-deliver every event continuously. `LocalDbStoreAndForwardSink` takes a `drainGate` fed from `IRedundancyRoleView` — a singleton `DriverHostActor` publishes its `PrimaryGatePolicy` verdict to, because the drain runs on a timer and cannot receive `RedundancyStateChanged` itself. A gated-off node reports `HistorianDrainState.NotPrimary`. The gate applies only to a node whose queue a peer actually holds — `LocalDbReplicationPolicy.SharesRowsWithPeer`: `LocalDb:Replication:ApiKey` AND (`PeerAddress` OR `LocalDb:SyncListenPort` > 0); any other node always drains (review R1 m3). Delivery is **at-least-once across a failover** by design: rows the old Primary delivered just before dying may not have replicated their deletes yet. See [AlarmHistorian.md](AlarmHistorian.md).
 
 - **Continuous-historization value CAPTURE** (Gitea #537) — a third gate on the same role rule, and the
   one that gates the *opposite* half of its pipeline. Both pair nodes serve the same address space and see
@@ -622,7 +622,10 @@ and [AlarmHistorian.md](AlarmHistorian.md). Phase 1 caches the **deployed-config
 artifact** (chunked). Its job is a single failure mode redundancy does not otherwise cover: a driver
 node restarting into a **central-SQL-Server outage** would come up with no configuration at all. With
 the cache, it **boots from the last artifact it applied** instead, and logs a running-from-cache
-signal.
+signal. A cached artifact whose bytes are empty or malformed is **no answer** (#601): nothing is applied,
+the node does **not** adopt that revision or claim running-from-cache, and it goes **Stale** (retrying the
+ConfigDb) — so a later dispatch of that same revision really applies instead of being acked as already
+current (review R1 m4). A `FetchAndCache` node lands Steady-with-no-revision instead, as for an empty cache.
 
 - **What replicates.** The cache can *optionally* replicate to the node's redundant pair peer over
   the LocalDb library's gRPC sync (HLC-stamped, last-writer-wins). Two tables replicate:

docs/ScriptedAlarms.md

@@ -118,7 +118,8 @@ rationale is recorded in the engine's class doc so it is not re-proposed:
 - `ApplyOneShotShelve` / `ApplyTimedShelve(unshelveAtUtc)` / `ApplyUnshelve` — shelving transitions. `Timed` requires `unshelveAtUtc > nowUtc`.
 - `ApplyEnable` / `ApplyDisable` — operator enable/disable. Disabled alarms ignore predicate results until re-enabled; on enable, `ActiveState` is re-derived from the next evaluation.
 - `ApplyAddComment(text)` — append-only audit entry, no state change.
-- `ApplyShelvingCheck(nowUtc)` — called by the 5s timer; promotes expired `Timed` shelving to `Unshelved` with a `system / AutoUnshelve` audit entry.
+- `ApplyShelvingCheck(nowUtc)` — called by the 5s timer; promotes expired `Timed` shelving to `Unshelved` with a `system / AutoUnshelve` audit entry. The sole owner of timed-shelve expiry — the OPC UA node's own SDK timer does nothing.
+- `ApplyAcknowledge` on an already-acknowledged alarm is a no-op **unless it carries a comment**, which is recorded as an `AddComment` (emission `CommentAdded`; `LastAck*` untouched) — the OPC UA node reports a commented re-ack, so the system of record must hold the comment too.
 
 Two invariants the machine enforces:
 
@@ -354,11 +355,11 @@ On allow, the handler publishes a `Commons.OpcUa.AlarmCommand` (containing comma
 - **Scripted conditions** — all eight Part 9 operations (including Enable/Disable) route through `AlarmCommandRouter` onto the `alarm-commands` topic, which `ScriptedAlarmHostActor` dispatches to the engine (`EnableAsync` / `DisableAsync` / `AcknowledgeAsync` / …). On enable, `ActiveState` is re-derived from the next predicate evaluation.
 - **Native (driver-fed) conditions** — `OnAcknowledge` branches to `NativeAlarmAckRouter` and routes the ack to the owning driver rather than the scripted engine (see §[Native-alarm OPC UA operator operations](#native-alarm-opc-ua-operator-operations) above). `OnEnableDisable`, `OnConfirm` and `OnShelve` return `BadNotSupported` immediately — a native condition supports Acknowledge and AddComment only (no engine-side surface, no upstream path, and the driver projection owns its Confirmed/Shelving; Gitea #587 review).
 
-`OnTimedUnshelve` (the SDK's internal auto-unshelve timer) bypasses the client gate — it is system-initiated and not subject to operator role checks.
+`OnTimedUnshelve` (the SDK's internal auto-unshelve timer, armed for the real remaining time since Gitea #597) is a **no-op**: the engine owns timed-shelve expiry via `ApplyShelvingCheck` (`system / AutoUnshelve`), and routing an `Unshelve` from the timer pre-empted that on every expiry as an operator-style unshelve from both pair nodes (review R1 M2).
 
 ### Delta-gate de-duplication
 
-`WriteAlarmCondition` fires a Part 9 condition event only when the incoming state differs from the node's current live state. For an OPC UA method call (Acknowledge, Confirm, AddComment, the shelve methods, Enable/Disable) the SDK applies the new state to the node, and the node manager then reports **the one** condition event for it from the method path (Gitea #587 — the SDK reports none of its own, since `AutoReportStateChanges` stays false; the shelve methods' state is applied by the node manager, because the SDK's never applies it; a call that changed nothing — e.g. an uncommented re-ack — reports nothing; a re-ack carrying a comment reports that comment once, Gitea #597). Native (driver-fed) conditions support only Acknowledge and AddComment; their Confirm, shelve and Enable/Disable methods return `BadNotSupported`, and their Acknowledge is refused with `BadInvalidState` on a node that is not the redundancy Primary (Gitea #597). A timed shelve's `UnshelveTime` counts down to the engine's expiry (the projection carries `UnshelveAtUtc`). The engine's later re-projection of the same transition finds the node already in that state and is suppressed, so each accepted call yields exactly one event. See `docs/AlarmTracking.md` §"One condition event per method call".
+`WriteAlarmCondition` fires a Part 9 condition event only when the incoming state differs from the node's current live state. For an OPC UA method call (Acknowledge, Confirm, AddComment, the shelve methods, Enable/Disable) the SDK applies the new state to the node, and the node manager then reports **the one** condition event for it from the method path (Gitea #587 — the SDK reports none of its own, since `AutoReportStateChanges` stays false; the shelve methods' state is applied by the node manager, because the SDK's never applies it; a call that changed nothing — e.g. an uncommented re-ack — reports nothing; a re-ack carrying a comment reports that comment once, Gitea #597, and the engine records it as an `AddComment` so it is persisted and replicated, review R1 m1). Native (driver-fed) conditions support only Acknowledge and AddComment; their Confirm, shelve and Enable/Disable methods return `BadNotSupported`, and their Acknowledge is refused with `BadInvalidState` on a node that is not the redundancy Primary (Gitea #597). A timed shelve's `UnshelveTime` counts down to the engine's expiry (the projection carries `UnshelveAtUtc`). The engine's later re-projection of the same transition finds the node already in that state and is suppressed, so each accepted call yields exactly one event. See `docs/AlarmTracking.md` §"One condition event per method call".
 
 ### AdminUI path
 

docs/Telemetry.md

@@ -255,6 +255,11 @@ driver node**:
   reconnect loses whatever transitions happened while disconnected, same as a DPS resubscribe would.
   The cache is evicted per instance (`ITelemetryLocalHub.Evict`) when `DriverHostActor.StopChild` removes a
   driver from the plan (#609), so a decommissioned instance's last state is not replayed to new subscribers.
+  `StopChild` first releases the instance's resilience state (`IDriverCapabilityInvokerFactory.Release` →
+  pipeline-builder + `DriverResilienceStatusTracker.Invalidate`): `DriverResilienceStatusPublisherService`
+  re-emits the whole tracker every 5 s, so evicting only the hub was undone within one tick (review R1 M1).
+  Residual: a Health emit already in flight from the stopping child can land after the eviction and
+  re-cache that one snapshot — a narrow race, not a recurring replay.
 - **Connection indicator** — the `/alerts` and `/script-log` live/disconnected pill is driven by
   **aggregate** stream health across all dialers: connected when **at least one** node stream is
   up, disconnected when all are down. This matches today's DPS `SubscribeAck`/`PostStop` pill

docs/operations/2026-07-20-localdb-pair-replication.md

@@ -92,7 +92,9 @@ Replication has two knobs, both under `LocalDb`:
 > 2026-08-18 `LocalDbSetup.OnReady` installs the change-capture triggers only when the node is
 > configured to replicate — the predicate is **`PeerAddress` OR `ApiKey` non-blank** (the OR because
 > the passive node has no `PeerAddress`; `SyncListenPort` is deliberately not part of it, since a
-> port with no key is fail-closed and syncs with nobody). Consequences at enablement time:
+> port with no key is fail-closed and syncs with nobody). The alarm-historian drain gate uses the
+> stricter `ApiKey` AND (`PeerAddress` OR `SyncListenPort > 0`) — so set the key **and** the port/peer
+> together, or the node keeps draining its own queue. Consequences at enablement time:
 >
 > - A configured node registers with **`baselineExistingRows: true`**, so the rows **already in the
 >   file** are seeded into the row-version ledger at the LWW floor (HLC `0`) and actually travel.