diff --git a/docs/Redundancy.md b/docs/Redundancy.md index 0a0dff1d..a0b5f3ad 100644 --- a/docs/Redundancy.md +++ b/docs/Redundancy.md @@ -69,6 +69,23 @@ Roles come from `RedundancyStateActor.BuildSnapshot`: a node with the `driver` role is `Primary` when it holds the `driver` role-leader lease, otherwise `Secondary`; a node without the `driver` role is `Detached`. +> **KNOWN LIMITATION — the election is per *Akka* cluster, not per application `Cluster`.** +> `RoleLeader("driver")` yields exactly **one** Primary across the whole Akka cluster. A fleet that +> runs several application clusters (each with its own redundant pair) inside one Akka cluster — +> which is what `docker-dev/docker-compose.yml` models, with MAIN, SITE-A and SITE-B — therefore has +> one Primary among *all* its driver nodes, not one per pair. Every consumer of the role inherits +> this: `ServiceLevel`, the `alerts` emit gate, and the inbound device-write gate below. +> +> The unit of redundancy everywhere else in the product is the application `Cluster` (`ClusterId`) — +> it owns drivers, devices and `ClusterNode` rows, and the LocalDb deployment-artifact cache is +> already keyed by it *so that a pair shares one entry*. Scoping the election the same way is the +> fix; `RedundancyStateActor` is an admin-role singleton in the same assembly as +> `AdminOperationsActor`, which already reads and groups `ClusterNodes` by cluster. +> +> Surfaced by the LocalDb Phase 2 live gate, where it stopped the alarm-history drain on every node +> in the fleet — see `docs/plans/2026-07-20-localdb-phase2-live-gate.md`. The alarm drain no longer +> depends on this (it defers only to a node that shares its queue), but the other three gates still do. + ## Data flow ``` diff --git a/docs/plans/2026-07-20-localdb-phase2-live-gate.md b/docs/plans/2026-07-20-localdb-phase2-live-gate.md index 8016bf18..8328cca2 100644 --- a/docs/plans/2026-07-20-localdb-phase2-live-gate.md +++ b/docs/plans/2026-07-20-localdb-phase2-live-gate.md @@ -130,21 +130,50 @@ the queue is pair-local. Only the dialing half of a pair knows its peer (`Replic the listening half sets `SyncListenPort` only. So the drain gate can never fully engage, and both halves of a replicated pair drain — safe, but duplicated. -Three ways out, in increasing order of scope: +### Follow-up analysis — the root cause is wider than the drain gate -- **(A) Configure both halves to dial.** Smallest change; makes the peer known on both sides so the - gate engages symmetrically. Needs confirmation that the LocalDb library accepts bidirectional dial, - and an operator-facing config change. -- **(B) Accept duplicates for unpaired-role deployments.** Change nothing further, document that the - gate only engages when redundancy roles are configured per pair. Cheapest; leaves the steady-state - double-delivery the gate was introduced to remove. -- **(C) Scope redundancy roles per pair in `RedundancyStateActor`.** Fixes the root cause for *every* - consumer of the role (ServiceLevel, scripted-alarm emit gate, inbound-write gate), all of which - inherit the same cluster-wide assumption. Largest blast radius, and the only one that makes - "Primary" mean what the docs say it means in a multi-pair fleet. +Two things assumed while writing the section above turned out to be wrong, and correcting them +changes the recommendation. -The branch is safe in every topology as it stands — no configuration loses data — so this is a -question of when to reclaim the de-duplication benefit, not an outstanding hazard. +**The rig is not misconfigured.** `central-1`/`central-2` carry `admin,driver` deliberately — the +compose says so in as many words ("central is admin,driver"), because they are themselves a +redundant pair that also runs the MAIN cluster's drivers. Stripping the driver role would change what +the rig models, not fix anything. + +**A "pair" is not a missing concept — it already exists, as the application Cluster.** OtOpcUa's +`Cluster`/`ClusterId` (the config-DB entity that owns drivers, devices and `ClusterNode` rows) is +exactly the unit a redundant pair belongs to. Phase 1 already relies on this: the deployment-artifact +cache is *keyed by ClusterId so a pair shares one entry*, and the rig's three application clusters +(MAIN, SITE-A, SITE-B) are precisely its three pairs. + +So the real defect is that **`RedundancyStateActor` elects one Primary per *Akka* cluster, while +every meaningful unit of redundancy is an *application* cluster.** The rig runs six driver nodes +across three application clusters in one Akka cluster — a topology the product otherwise supports +throughout — and exactly one of those six can ever be Primary. + +That mis-scoping is not confined to the alarm drain. Every consumer of the role inherits it: +`ServiceLevel`, the `alerts` emit gate, and the inbound device-write gate. On a multi-cluster fleet +today, only one driver node cluster-wide will service Primary-gated work at all. + +### Revised recommendation + +**(C), as a separate change with its own live gate — and explicitly *not* (A).** + +- (C) is no longer "invent a pair concept"; it is "elect per `ClusterId` instead of per Akka cluster", + using a first-class entity that already exists. It is feasible where it needs to be: + `RedundancyStateActor` has no DB access today, but it is an admin-role singleton in the same + assembly as `AdminOperationsActor`, which already reads and groups `ClusterNodes` by cluster. +- **(A) is now actively undesirable.** Deriving pair identity from `LocalDb:Replication:PeerAddress` + would introduce a *second*, parallel notion of "who is my partner" that (C) immediately obsoletes — + and it cannot be done cleanly anyway: the library exposes no initiator/passive flag, so making both + halves declare a peer means both halves *dial*, opening two duplex sessions purely to obtain a + configuration value. +- (C) touches the inbound device-write gate, which is safety-critical (the archreview 03/S4 + dual-primary fix). It therefore belongs in its own change, with its own live gate, rather than + bolted onto Phase 2. + +Until then this branch is safe in every topology — no configuration loses data — so what remains is +reclaiming the de-duplication benefit, not an outstanding hazard. ## Not merged