docs(mesh): Phase 6 — mark done, remove single-mesh caveats, document pair-local meshes

Redundancy.md: KNOWN-LIMITATION (per-Akka-cluster election) marked RESOLVED;
new 'Per-cluster meshes (Phase 6)' section + pair-local secrets note; manual-
failover mesh-scope caveat removed; auto-down 1-vs-1 note updated (every mesh is
now two nodes, drill deferred to Phase 7). IManualFailoverService + ClusterRedundancy
razor caveats updated to pair-local reality. Configuration.md documents the
cluster-{ClusterId} role, per-pair self-first seeds, SplitTopologyTransportValidator,
and the intentional Dps default. Program + design status tables + CLAUDE.md marked
Phase 6 DONE.

(Task 8 subagent completed the edits but died on an API error before committing;
reviewed and committed by the controller.)

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
2026-07-24 03:02:35 -04:00
parent 2a057a466f
commit 8a173bbaf6
7 changed files with 153 additions and 58 deletions
+61 -25
View File
@@ -108,22 +108,59 @@ plane (writes, alarm acks, the alerts emit, the alarm-history drain) on the wron
*higher* address so the two orderings genuinely disagree. `NodeRedundancyState.IsRoleLeaderForDriver`
was renamed `IsDriverPrimary` to stop the name asserting the old derivation.
> **KNOWN LIMITATION — the election is per *Akka* cluster, not per application `Cluster`.**
> The election yields exactly **one** Primary across the whole Akka cluster. A fleet that
> runs several application clusters (each with its own redundant pair) inside one Akka cluster —
> which is what `docker-dev/docker-compose.yml` models, with MAIN, SITE-A and SITE-B — therefore has
> one Primary among *all* its driver nodes, not one per pair. Every consumer of the role inherits
> this: `ServiceLevel`, the `alerts` emit gate, and the inbound device-write gate below.
> **RESOLVED by Phase 6 — the election is now per application `Cluster`, not per Akka mesh.**
> Phase 6 of the per-cluster mesh program split the single fleet-wide Akka mesh into one independent
> 2-node mesh per application `Cluster` (see [Per-cluster meshes (Phase 6)](#per-cluster-meshes-phase-6)
> below). `RedundancyStateActor` is now a `cluster-{ClusterId}`-scoped singleton spawned on every driver
> node, so each pair elects its own Primary — the whole-Akka-cluster mismatch this block used to describe
> no longer exists. See `docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md`.
>
> The unit of redundancy everywhere else in the product is the application `Cluster` (`ClusterId`) —
> it owns drivers, devices and `ClusterNode` rows, and the LocalDb deployment-artifact cache is
> already keyed by it *so that a pair shares one entry*. Scoping the election the same way is the
> fix; `RedundancyStateActor` is an admin-role singleton in the same assembly as
> `AdminOperationsActor`, which already reads and groups `ClusterNodes` by cluster.
>
> Surfaced by the LocalDb Phase 2 live gate, where it stopped the alarm-history drain on every node
> in the fleet — see `docs/plans/2026-07-20-localdb-phase2-live-gate.md`. The alarm drain no longer
> depends on this (it defers only to a node that shares its queue), but the other three gates still do.
> Originally surfaced by the LocalDb Phase 2 live gate, where the old whole-mesh election stopped the
> alarm-history drain on every node in the fleet — see `docs/plans/2026-07-20-localdb-phase2-live-gate.md`.
## Per-cluster meshes (Phase 6)
The fleet no longer runs one Akka mesh. Phase 6 split it into **three independent 2-node meshes** — the
central pair (roles `admin,driver,cluster-MAIN`), the site-a pair (`driver,cluster-SITE-A`), and the
site-b pair (`driver,cluster-SITE-B`) — all sharing the same `ActorSystem` name (`otopcua`) on one network,
separated purely by **per-pair self-first seed partitioning**: each node's `Cluster:SeedNodes` lists only
itself and its own pair partner, so gossip never crosses a pair boundary (see
[Bootstrap: self-first seed ordering](#bootstrap-self-first-seed-ordering)).
Consequences:
- **One Primary per pair — two or more Primaries fleet-wide, by design.** Each mesh runs its own
election, so a three-mesh fleet has three Primaries, one per application `Cluster`. This is correct,
not a regression of the old single-Primary assumption.
- **`RedundancyStateActor` is a `cluster-{ClusterId}`-scoped singleton**, spawned on every driver node
(falling back to the plain `driver` role for a legacy node not carrying a cluster role). Each pair
elects and publishes its own `redundancy-state` on its own mesh's DPS — the DPS topic name is
unchanged, but it no longer reaches past the pair.
- **`ClusterNodeAddressReconciler` is own-cluster-scoped** — an admin node can only reconcile
`ClusterNode` rows belonging to its own cluster; it has no gossip visibility into another mesh to
reconcile against.
- **`CentralCommunicationActor` creates one `ClusterClient` per application `Cluster`** and fans
commands across them, rather than one fleet-wide client. Central reaches a site mesh over
`ClusterClient` (commands), gRPC (telemetry — see `docs/Telemetry.md`), and `ConfigServe` (config
fetch — see [`ConfigSource`/`ConfigServe`](../CLAUDE.md#config-source-configsource--configserve)),
never over gossip.
- **`SplitTopologyTransportValidator` fails host start** on a node carrying a `cluster-{ClusterId}`
role unless it is also configured for the split-safe transports: `MeshTransport:Mode = ClusterClient`,
`Telemetry:Mode = Grpc` (driver nodes), `TelemetryDial:Mode = Grpc` (admin nodes). A DPS-mode transport
on a cluster-role node would silently try to gossip across a partition that no longer exists, so the
validator fails closed instead. Legacy/non-cluster-role nodes are exempt.
- **The compiled transport-mode defaults were deliberately NOT flipped** — `MeshTransport:Mode`,
`Telemetry:Mode`, and `TelemetryDial:Mode` all still default to `Dps`. A blast-radius assessment
found flipping the compiled default would break every integration test and every existing
`appsettings.json` profile that doesn't explicitly set these keys. The guardrail is the validator
above plus explicit per-deployment configuration, not a default flip.
### Secrets replication is pair-local
Akka DPS secret replication is scoped to each 2-node mesh: a pair replicates its own secrets to its
own partner and no further. There is no fleet-wide or cross-cluster secret sync after the split — by
design, consistent with Phase 4 (site nodes have no SQL, so there is no SQL-backed hub to fan a
cross-mesh sync through).
## Data flow
@@ -316,18 +353,17 @@ advertises `ServiceLevel` 250. OPC UA clients re-select.
| Confirmation | A dialog naming the node, and what follows (restart, ServiceLevel 250 moves, clients re-select). |
| Audit | Written through the shared `ZB.MOM.WW.Audit.IAuditWriter` seam **before** the Leave — action `cluster.manual-failover`, actor, target. A *refused* failover changes nothing and writes nothing. |
> **Mesh-scope caveat.** Until the per-cluster mesh work lands
> (`docs/plans/2026-07-21-per-cluster-mesh-design.md`), the Primary is elected once per Akka mesh rather than
> per application `Cluster` row — so on a fleet running several clusters in one mesh this button acts on the
> whole mesh's Primary, which may belong to a different cluster than the page you are on. The panel says so.
> Phase 6 of the mesh program removes both the caveat and this notice.
Phase 6 of the mesh program (see [Per-cluster meshes (Phase 6)](#per-cluster-meshes-phase-6)) split
the fleet into one 2-node mesh per application `Cluster`, so the button now acts on **this pair's own**
Primary — there is no longer a whole-mesh vs. per-cluster ambiguity to caveat.
> **Live gate outstanding.** The auto-down change is verified by unit tests against the effective
> configuration, and by the sister project's live drill on an equivalent Akka version and topology. It has
> **not** yet been drilled on an OtOpcUa two-node rig — the docker-dev rig runs a single six-node mesh, where
> the 1-vs-1 pathology cannot occur. The drill (kill the oldest container of a genuine two-node cluster;
> assert the survivor stays up, takes the singletons, and reports the primary `ServiceLevel`) should run
> alongside the per-cluster mesh work, which makes every mesh exactly two nodes. See
> configuration, and by the sister project's live drill on an equivalent Akka version and topology. It
> was **not** drillable on an OtOpcUa rig before Phase 6, because the docker-dev rig ran a single
> six-node mesh where the 1-vs-1 pathology cannot occur. **Phase 6 has landed and every mesh is now
> exactly two nodes**, so the drill (kill the oldest container of a genuine two-node cluster; assert
> the survivor stays up, takes the singletons, and reports the primary `ServiceLevel`) is finally
> testable — deferred to Phase 7's failover drill. See
> `docs/plans/2026-07-21-per-cluster-mesh-design.md` §6.2 / Phase 0a.
## Bootstrap: self-first seed ordering