fix(runtime): the passive half of a LocalDb pair stands its alarm drain down for its mesh partner (review R6 MINOR-4)

The drain gate's 'is my queue-sharing peer Primary?' used only the host of
LocalDb:Replication:PeerAddress, which the passive half (ApiKey +
SyncListenPort) does not set - so as Secondary it kept draining beside its
Primary partner and every alarm event reached the historian twice.
DriverHostActor takes a sharesAlarmQueueWithLocal predicate used when no
peer host is known; WithOtOpcUaRuntimeActors builds it
(BuildAlarmQueuePeerPredicate) for a SharesRowsWithPeer node with no
PeerAddress as 'a member of my own cluster-{ClusterId} mesh other than
me'. A node sharing no rows, or with no mesh role, gets none and keeps
draining. AlarmHistorian.md / Redundancy.md / pair-replication runbook
updated.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
This commit is contained in:
Joseph Doherty
2026-10-08 08:43:06 -04:00
parent e93d115cbe
commit 3be8a1217e
7 changed files with 203 additions and 11 deletions
+11
View File
@@ -103,6 +103,17 @@ Two postures are deliberate:
a key with no dial and no listener (e.g. a fleet-wide secret on an unpaired
node) captures changes but shares them with nobody, and gating it would park
its events behind a Primary that does not hold them (review R1 m3).
- **A gated node stands down only for the peer that holds its rows — and BOTH
halves of a pair know it.** The dialling half identifies its peer by the host
of `LocalDb:Replication:PeerAddress`. The passive half (`ApiKey` +
`SyncListenPort`, no `PeerAddress`) identifies it as its **per-cluster mesh
partner** (the other member of its `cluster-{ClusterId}` role). Before review
R6 MINOR-4 the passive half knew no peer at all, so as Secondary it kept
draining beside its Primary partner and the historian received every alarm
event twice. ⚠️ A passive half with **no** cluster-scoped role (a legacy,
pre-Phase-6 node) still cannot identify its partner and keeps draining — it
cannot prove which Primary holds its rows, and deferring to the wrong one
would strand them.
- **A gate that throws means "not now", never permission.** Draining on a gate
fault would put a pair into dual delivery exactly when its redundancy signal is
sick.
+1 -1
View File
@@ -532,7 +532,7 @@ Under warm/hot redundancy both cluster nodes run `ScriptedAlarmHostActor` and ev
- **`alerts` topic emission** — `ScriptedAlarmHostActor` and `DriverHostActor.ForwardNativeAlarm` subscribe to the `redundancy-state` DPS topic and cache the local node's `RedundancyRole`, then gate the cluster `alerts` publish through `PrimaryGatePolicy` (table above). The OPC UA condition-node write and inbound ack/shelve command processing remain **ungated** on both nodes so the secondary is always ready to serve clients after a failover.
- **`HistorianAdapterActor` historization** — likewise Primary-gated so alarm historization is exactly-once across all alarm sources. The actor subscribes to the `alerts` DPS topic and translates each `AlarmTransitionEvent` → `AlarmHistorianEvent` before enqueuing it on the sink; scripted alarms therefore historize exactly once regardless of cluster size.
- **The alarm sink's DRAIN** — a second, independent gate on the same decision. The enqueue gate above is not sufficient any more: since Phase 2 the store-and-forward buffer lives in the replicated LocalDb, so the Secondary holds a full copy of the Primary's queued rows and an ungated drain there would re-deliver every event continuously. `LocalDbStoreAndForwardSink` takes a `drainGate` fed from `IRedundancyRoleView` — a singleton `DriverHostActor` publishes its `PrimaryGatePolicy` verdict to, because the drain runs on a timer and cannot receive `RedundancyStateChanged` itself. A gated-off node reports `HistorianDrainState.NotPrimary`. The gate applies only to a node whose queue a peer actually holds — `LocalDbReplicationPolicy.SharesRowsWithPeer`: `LocalDb:Replication:ApiKey` AND (`PeerAddress` OR `LocalDb:SyncListenPort` > 0); any other node always drains (review R1 m3). Delivery is **at-least-once across a failover** by design: rows the old Primary delivered just before dying may not have replicated their deletes yet. See [AlarmHistorian.md](AlarmHistorian.md).
- **The alarm sink's DRAIN** — a second, independent gate on the same decision. The enqueue gate above is not sufficient any more: since Phase 2 the store-and-forward buffer lives in the replicated LocalDb, so the Secondary holds a full copy of the Primary's queued rows and an ungated drain there would re-deliver every event continuously. `LocalDbStoreAndForwardSink` takes a `drainGate` fed from `IRedundancyRoleView` — a singleton `DriverHostActor` publishes its `PrimaryGatePolicy` verdict to, because the drain runs on a timer and cannot receive `RedundancyStateChanged` itself. A gated-off node reports `HistorianDrainState.NotPrimary`. The gate applies only to a node whose queue a peer actually holds — `LocalDbReplicationPolicy.SharesRowsWithPeer`: `LocalDb:Replication:ApiKey` AND (`PeerAddress` OR `LocalDb:SyncListenPort` > 0); any other node always drains (review R1 m3). Within a pair BOTH halves stand down for their partner: the dialling half knows it by the `PeerAddress` host, the passive half (no `PeerAddress`) as its per-cluster mesh partner (review R6 MINOR-4 — before that the passive half never stood down, so both halves drained and every event reached the historian twice; a passive half with no `cluster-{ClusterId}` role still cannot identify its partner and keeps draining). Delivery is **at-least-once across a failover** by design: rows the old Primary delivered just before dying may not have replicated their deletes yet. See [AlarmHistorian.md](AlarmHistorian.md).
- **Continuous-historization value CAPTURE** (Gitea #537) — a third gate on the same role rule, and the
one that gates the *opposite* half of its pipeline. Both pair nodes serve the same address space and see
@@ -94,7 +94,10 @@ Replication has two knobs, both under `LocalDb`:
> the passive node has no `PeerAddress`; `SyncListenPort` is deliberately not part of it, since a
> port with no key is fail-closed and syncs with nobody). The alarm-historian drain gate uses the
> stricter `ApiKey` AND (`PeerAddress` OR `SyncListenPort > 0`) — so set the key **and** the port/peer
> together, or the node keeps draining its own queue. Consequences at enablement time:
> together, or the node keeps draining its own queue. Once gated, the passive half stands down for
> its **per-cluster mesh partner** (it has no `PeerAddress` to name one) — so it must carry the pair's
> `cluster-{ClusterId}` role, or as Secondary it keeps draining beside its Primary and the historian
> gets every alarm event twice (review R6 MINOR-4). Consequences at enablement time:
>
> - A configured node registers with **`baselineExistingRows: true`**, so the rows **already in the
> file** are seeded into the row-version ledger at the LWW floor (HLC `0`) and actually travel.