docs(redundancy): record the per-Akka-cluster role-election limitation
Corrects two claims made while writing the Phase 2 gate record. The docker-dev rig is NOT misconfigured: central-1/central-2 carry admin,driver deliberately — the compose says so — because they are themselves a pair that runs the MAIN cluster's drivers. And a redundant pair is not a missing concept. The application Cluster (ClusterId) already is one: it owns drivers, devices and ClusterNode rows, and Phase 1's deployment-artifact cache is keyed by it precisely so a pair shares one entry. So the root cause is wider than the alarm drain. RedundancyStateActor elects one Primary per AKKA cluster while every real unit of redundancy is an APPLICATION cluster, and ServiceLevel, the alerts emit gate and the inbound device-write gate all inherit that mis-scoping. On a fleet of pairs in one Akka cluster, exactly one driver node cluster-wide services Primary-gated work. Recommendation revised in the gate doc: fix the election scope (its own change, its own gate, because it touches the safety-critical write gate), and explicitly do NOT derive pair identity from LocalDb:Replication:PeerAddress — that would add a second parallel notion of 'my partner' that the real fix obsoletes, and the library has no initiator flag, so both halves would have to dial just to obtain a config value. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
@@ -69,6 +69,23 @@ Roles come from `RedundancyStateActor.BuildSnapshot`: a node with the `driver`
|
||||
role is `Primary` when it holds the `driver` role-leader lease, otherwise
|
||||
`Secondary`; a node without the `driver` role is `Detached`.
|
||||
|
||||
> **KNOWN LIMITATION — the election is per *Akka* cluster, not per application `Cluster`.**
|
||||
> `RoleLeader("driver")` yields exactly **one** Primary across the whole Akka cluster. A fleet that
|
||||
> runs several application clusters (each with its own redundant pair) inside one Akka cluster —
|
||||
> which is what `docker-dev/docker-compose.yml` models, with MAIN, SITE-A and SITE-B — therefore has
|
||||
> one Primary among *all* its driver nodes, not one per pair. Every consumer of the role inherits
|
||||
> this: `ServiceLevel`, the `alerts` emit gate, and the inbound device-write gate below.
|
||||
>
|
||||
> The unit of redundancy everywhere else in the product is the application `Cluster` (`ClusterId`) —
|
||||
> it owns drivers, devices and `ClusterNode` rows, and the LocalDb deployment-artifact cache is
|
||||
> already keyed by it *so that a pair shares one entry*. Scoping the election the same way is the
|
||||
> fix; `RedundancyStateActor` is an admin-role singleton in the same assembly as
|
||||
> `AdminOperationsActor`, which already reads and groups `ClusterNodes` by cluster.
|
||||
>
|
||||
> Surfaced by the LocalDb Phase 2 live gate, where it stopped the alarm-history drain on every node
|
||||
> in the fleet — see `docs/plans/2026-07-20-localdb-phase2-live-gate.md`. The alarm drain no longer
|
||||
> depends on this (it defers only to a node that shares its queue), but the other three gates still do.
|
||||
|
||||
## Data flow
|
||||
|
||||
```
|
||||
|
||||
@@ -130,21 +130,50 @@ the queue is pair-local. Only the dialing half of a pair knows its peer (`Replic
|
||||
the listening half sets `SyncListenPort` only. So the drain gate can never fully engage, and both
|
||||
halves of a replicated pair drain — safe, but duplicated.
|
||||
|
||||
Three ways out, in increasing order of scope:
|
||||
### Follow-up analysis — the root cause is wider than the drain gate
|
||||
|
||||
- **(A) Configure both halves to dial.** Smallest change; makes the peer known on both sides so the
|
||||
gate engages symmetrically. Needs confirmation that the LocalDb library accepts bidirectional dial,
|
||||
and an operator-facing config change.
|
||||
- **(B) Accept duplicates for unpaired-role deployments.** Change nothing further, document that the
|
||||
gate only engages when redundancy roles are configured per pair. Cheapest; leaves the steady-state
|
||||
double-delivery the gate was introduced to remove.
|
||||
- **(C) Scope redundancy roles per pair in `RedundancyStateActor`.** Fixes the root cause for *every*
|
||||
consumer of the role (ServiceLevel, scripted-alarm emit gate, inbound-write gate), all of which
|
||||
inherit the same cluster-wide assumption. Largest blast radius, and the only one that makes
|
||||
"Primary" mean what the docs say it means in a multi-pair fleet.
|
||||
Two things assumed while writing the section above turned out to be wrong, and correcting them
|
||||
changes the recommendation.
|
||||
|
||||
The branch is safe in every topology as it stands — no configuration loses data — so this is a
|
||||
question of when to reclaim the de-duplication benefit, not an outstanding hazard.
|
||||
**The rig is not misconfigured.** `central-1`/`central-2` carry `admin,driver` deliberately — the
|
||||
compose says so in as many words ("central is admin,driver"), because they are themselves a
|
||||
redundant pair that also runs the MAIN cluster's drivers. Stripping the driver role would change what
|
||||
the rig models, not fix anything.
|
||||
|
||||
**A "pair" is not a missing concept — it already exists, as the application Cluster.** OtOpcUa's
|
||||
`Cluster`/`ClusterId` (the config-DB entity that owns drivers, devices and `ClusterNode` rows) is
|
||||
exactly the unit a redundant pair belongs to. Phase 1 already relies on this: the deployment-artifact
|
||||
cache is *keyed by ClusterId so a pair shares one entry*, and the rig's three application clusters
|
||||
(MAIN, SITE-A, SITE-B) are precisely its three pairs.
|
||||
|
||||
So the real defect is that **`RedundancyStateActor` elects one Primary per *Akka* cluster, while
|
||||
every meaningful unit of redundancy is an *application* cluster.** The rig runs six driver nodes
|
||||
across three application clusters in one Akka cluster — a topology the product otherwise supports
|
||||
throughout — and exactly one of those six can ever be Primary.
|
||||
|
||||
That mis-scoping is not confined to the alarm drain. Every consumer of the role inherits it:
|
||||
`ServiceLevel`, the `alerts` emit gate, and the inbound device-write gate. On a multi-cluster fleet
|
||||
today, only one driver node cluster-wide will service Primary-gated work at all.
|
||||
|
||||
### Revised recommendation
|
||||
|
||||
**(C), as a separate change with its own live gate — and explicitly *not* (A).**
|
||||
|
||||
- (C) is no longer "invent a pair concept"; it is "elect per `ClusterId` instead of per Akka cluster",
|
||||
using a first-class entity that already exists. It is feasible where it needs to be:
|
||||
`RedundancyStateActor` has no DB access today, but it is an admin-role singleton in the same
|
||||
assembly as `AdminOperationsActor`, which already reads and groups `ClusterNodes` by cluster.
|
||||
- **(A) is now actively undesirable.** Deriving pair identity from `LocalDb:Replication:PeerAddress`
|
||||
would introduce a *second*, parallel notion of "who is my partner" that (C) immediately obsoletes —
|
||||
and it cannot be done cleanly anyway: the library exposes no initiator/passive flag, so making both
|
||||
halves declare a peer means both halves *dial*, opening two duplex sessions purely to obtain a
|
||||
configuration value.
|
||||
- (C) touches the inbound device-write gate, which is safety-critical (the archreview 03/S4
|
||||
dual-primary fix). It therefore belongs in its own change, with its own live gate, rather than
|
||||
bolted onto Phase 2.
|
||||
|
||||
Until then this branch is safe in every topology — no configuration loses data — so what remains is
|
||||
reclaiming the de-duplication benefit, not an outstanding hazard.
|
||||
|
||||
## Not merged
|
||||
|
||||
|
||||
Reference in New Issue
Block a user