docs(mesh): Phase 6 — mark done, remove single-mesh caveats, document pair-local meshes
Redundancy.md: KNOWN-LIMITATION (per-Akka-cluster election) marked RESOLVED;
new 'Per-cluster meshes (Phase 6)' section + pair-local secrets note; manual-
failover mesh-scope caveat removed; auto-down 1-vs-1 note updated (every mesh is
now two nodes, drill deferred to Phase 7). IManualFailoverService + ClusterRedundancy
razor caveats updated to pair-local reality. Configuration.md documents the
cluster-{ClusterId} role, per-pair self-first seeds, SplitTopologyTransportValidator,
and the intentional Dps default. Program + design status tables + CLAUDE.md marked
Phase 6 DONE.
(Task 8 subagent completed the edits but died on an API error before committing;
reviewed and committed by the controller.)
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
+61
-25
@@ -108,22 +108,59 @@ plane (writes, alarm acks, the alerts emit, the alarm-history drain) on the wron
|
||||
*higher* address so the two orderings genuinely disagree. `NodeRedundancyState.IsRoleLeaderForDriver`
|
||||
was renamed `IsDriverPrimary` to stop the name asserting the old derivation.
|
||||
|
||||
> **KNOWN LIMITATION — the election is per *Akka* cluster, not per application `Cluster`.**
|
||||
> The election yields exactly **one** Primary across the whole Akka cluster. A fleet that
|
||||
> runs several application clusters (each with its own redundant pair) inside one Akka cluster —
|
||||
> which is what `docker-dev/docker-compose.yml` models, with MAIN, SITE-A and SITE-B — therefore has
|
||||
> one Primary among *all* its driver nodes, not one per pair. Every consumer of the role inherits
|
||||
> this: `ServiceLevel`, the `alerts` emit gate, and the inbound device-write gate below.
|
||||
> **RESOLVED by Phase 6 — the election is now per application `Cluster`, not per Akka mesh.**
|
||||
> Phase 6 of the per-cluster mesh program split the single fleet-wide Akka mesh into one independent
|
||||
> 2-node mesh per application `Cluster` (see [Per-cluster meshes (Phase 6)](#per-cluster-meshes-phase-6)
|
||||
> below). `RedundancyStateActor` is now a `cluster-{ClusterId}`-scoped singleton spawned on every driver
|
||||
> node, so each pair elects its own Primary — the whole-Akka-cluster mismatch this block used to describe
|
||||
> no longer exists. See `docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md`.
|
||||
>
|
||||
> The unit of redundancy everywhere else in the product is the application `Cluster` (`ClusterId`) —
|
||||
> it owns drivers, devices and `ClusterNode` rows, and the LocalDb deployment-artifact cache is
|
||||
> already keyed by it *so that a pair shares one entry*. Scoping the election the same way is the
|
||||
> fix; `RedundancyStateActor` is an admin-role singleton in the same assembly as
|
||||
> `AdminOperationsActor`, which already reads and groups `ClusterNodes` by cluster.
|
||||
>
|
||||
> Surfaced by the LocalDb Phase 2 live gate, where it stopped the alarm-history drain on every node
|
||||
> in the fleet — see `docs/plans/2026-07-20-localdb-phase2-live-gate.md`. The alarm drain no longer
|
||||
> depends on this (it defers only to a node that shares its queue), but the other three gates still do.
|
||||
> Originally surfaced by the LocalDb Phase 2 live gate, where the old whole-mesh election stopped the
|
||||
> alarm-history drain on every node in the fleet — see `docs/plans/2026-07-20-localdb-phase2-live-gate.md`.
|
||||
|
||||
## Per-cluster meshes (Phase 6)
|
||||
|
||||
The fleet no longer runs one Akka mesh. Phase 6 split it into **three independent 2-node meshes** — the
|
||||
central pair (roles `admin,driver,cluster-MAIN`), the site-a pair (`driver,cluster-SITE-A`), and the
|
||||
site-b pair (`driver,cluster-SITE-B`) — all sharing the same `ActorSystem` name (`otopcua`) on one network,
|
||||
separated purely by **per-pair self-first seed partitioning**: each node's `Cluster:SeedNodes` lists only
|
||||
itself and its own pair partner, so gossip never crosses a pair boundary (see
|
||||
[Bootstrap: self-first seed ordering](#bootstrap-self-first-seed-ordering)).
|
||||
|
||||
Consequences:
|
||||
|
||||
- **One Primary per pair — two or more Primaries fleet-wide, by design.** Each mesh runs its own
|
||||
election, so a three-mesh fleet has three Primaries, one per application `Cluster`. This is correct,
|
||||
not a regression of the old single-Primary assumption.
|
||||
- **`RedundancyStateActor` is a `cluster-{ClusterId}`-scoped singleton**, spawned on every driver node
|
||||
(falling back to the plain `driver` role for a legacy node not carrying a cluster role). Each pair
|
||||
elects and publishes its own `redundancy-state` on its own mesh's DPS — the DPS topic name is
|
||||
unchanged, but it no longer reaches past the pair.
|
||||
- **`ClusterNodeAddressReconciler` is own-cluster-scoped** — an admin node can only reconcile
|
||||
`ClusterNode` rows belonging to its own cluster; it has no gossip visibility into another mesh to
|
||||
reconcile against.
|
||||
- **`CentralCommunicationActor` creates one `ClusterClient` per application `Cluster`** and fans
|
||||
commands across them, rather than one fleet-wide client. Central reaches a site mesh over
|
||||
`ClusterClient` (commands), gRPC (telemetry — see `docs/Telemetry.md`), and `ConfigServe` (config
|
||||
fetch — see [`ConfigSource`/`ConfigServe`](../CLAUDE.md#config-source-configsource--configserve)),
|
||||
never over gossip.
|
||||
- **`SplitTopologyTransportValidator` fails host start** on a node carrying a `cluster-{ClusterId}`
|
||||
role unless it is also configured for the split-safe transports: `MeshTransport:Mode = ClusterClient`,
|
||||
`Telemetry:Mode = Grpc` (driver nodes), `TelemetryDial:Mode = Grpc` (admin nodes). A DPS-mode transport
|
||||
on a cluster-role node would silently try to gossip across a partition that no longer exists, so the
|
||||
validator fails closed instead. Legacy/non-cluster-role nodes are exempt.
|
||||
- **The compiled transport-mode defaults were deliberately NOT flipped** — `MeshTransport:Mode`,
|
||||
`Telemetry:Mode`, and `TelemetryDial:Mode` all still default to `Dps`. A blast-radius assessment
|
||||
found flipping the compiled default would break every integration test and every existing
|
||||
`appsettings.json` profile that doesn't explicitly set these keys. The guardrail is the validator
|
||||
above plus explicit per-deployment configuration, not a default flip.
|
||||
|
||||
### Secrets replication is pair-local
|
||||
|
||||
Akka DPS secret replication is scoped to each 2-node mesh: a pair replicates its own secrets to its
|
||||
own partner and no further. There is no fleet-wide or cross-cluster secret sync after the split — by
|
||||
design, consistent with Phase 4 (site nodes have no SQL, so there is no SQL-backed hub to fan a
|
||||
cross-mesh sync through).
|
||||
|
||||
## Data flow
|
||||
|
||||
@@ -316,18 +353,17 @@ advertises `ServiceLevel` 250. OPC UA clients re-select.
|
||||
| Confirmation | A dialog naming the node, and what follows (restart, ServiceLevel 250 moves, clients re-select). |
|
||||
| Audit | Written through the shared `ZB.MOM.WW.Audit.IAuditWriter` seam **before** the Leave — action `cluster.manual-failover`, actor, target. A *refused* failover changes nothing and writes nothing. |
|
||||
|
||||
> **Mesh-scope caveat.** Until the per-cluster mesh work lands
|
||||
> (`docs/plans/2026-07-21-per-cluster-mesh-design.md`), the Primary is elected once per Akka mesh rather than
|
||||
> per application `Cluster` row — so on a fleet running several clusters in one mesh this button acts on the
|
||||
> whole mesh's Primary, which may belong to a different cluster than the page you are on. The panel says so.
|
||||
> Phase 6 of the mesh program removes both the caveat and this notice.
|
||||
Phase 6 of the mesh program (see [Per-cluster meshes (Phase 6)](#per-cluster-meshes-phase-6)) split
|
||||
the fleet into one 2-node mesh per application `Cluster`, so the button now acts on **this pair's own**
|
||||
Primary — there is no longer a whole-mesh vs. per-cluster ambiguity to caveat.
|
||||
|
||||
> **Live gate outstanding.** The auto-down change is verified by unit tests against the effective
|
||||
> configuration, and by the sister project's live drill on an equivalent Akka version and topology. It has
|
||||
> **not** yet been drilled on an OtOpcUa two-node rig — the docker-dev rig runs a single six-node mesh, where
|
||||
> the 1-vs-1 pathology cannot occur. The drill (kill the oldest container of a genuine two-node cluster;
|
||||
> assert the survivor stays up, takes the singletons, and reports the primary `ServiceLevel`) should run
|
||||
> alongside the per-cluster mesh work, which makes every mesh exactly two nodes. See
|
||||
> configuration, and by the sister project's live drill on an equivalent Akka version and topology. It
|
||||
> was **not** drillable on an OtOpcUa rig before Phase 6, because the docker-dev rig ran a single
|
||||
> six-node mesh where the 1-vs-1 pathology cannot occur. **Phase 6 has landed and every mesh is now
|
||||
> exactly two nodes**, so the drill (kill the oldest container of a genuine two-node cluster; assert
|
||||
> the survivor stays up, takes the singletons, and reports the primary `ServiceLevel`) is finally
|
||||
> testable — deferred to Phase 7's failover drill. See
|
||||
> `docs/plans/2026-07-21-per-cluster-mesh-design.md` §6.2 / Phase 0a.
|
||||
|
||||
## Bootstrap: self-first seed ordering
|
||||
|
||||
Reference in New Issue
Block a user