diff --git a/CLAUDE.md b/CLAUDE.md index 9073af3f..b9ec3344 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -228,7 +228,18 @@ The server supports non-transparent warm/hot redundancy via the `Redundancy` sec Assert the **effective** cluster config off a running `ActorSystem` (as `SplitBrainResolverActivationTests` does), never the typed options or the akka.conf text — several HOCON fragments compete and the losing one still reads correctly in the file. -**Still per-Akka-cluster, not per application `Cluster`.** One Primary is elected across the whole Akka mesh, so a fleet running several application clusters in one mesh (what docker-dev models) gets one Primary among all driver nodes. The fix is the per-cluster mesh workstream: `docs/plans/2026-07-21-per-cluster-mesh-design.md` (Phases 0a/0b/1 done; 2 code-complete behind a dark switch; 3–7 not started). +**RESOLVED by per-cluster mesh Phase 6 (2026-07-24) — no longer per-Akka-cluster.** The "one Primary across the whole Akka mesh" limitation described above is gone; see the Phase 6 paragraph immediately below. + +**Per-cluster mesh Phase 6 (2026-07-24) — the fleet is now three independent 2-node meshes, not one.** The single fleet-wide Akka mesh was split into one mesh per application `Cluster`: the central pair (`admin,driver,cluster-MAIN`), site-a (`driver,cluster-SITE-A`), site-b (`driver,cluster-SITE-B`) — same `ActorSystem` name, separated purely by per-pair self-first seeds (each node's `Cluster:SeedNodes` lists itself first, its own pair partner second, never a node from another cluster's pair). Consequences: + +- **The redundancy singleton is cluster-scoped and spawned per driver node.** `RedundancyStateActor` is now a `cluster-{ClusterId}`-scoped singleton (falls back to plain `driver` for a legacy node), so each pair elects and publishes its **own** Primary on its own mesh's DPS. **Two or more Primaries exist fleet-wide, one per pair — by design, not a bug.** +- `ClusterNodeAddressReconciler` is **own-cluster-scoped** — an admin node can only reconcile `ClusterNode` rows in its own cluster; it has no gossip visibility into another mesh. +- `CentralCommunicationActor` creates **one `ClusterClient` per application `Cluster`** and fans commands across them (closing the `TODO(Phase 6)` Phase 2 left behind), rather than one fleet-wide client. +- **`SplitTopologyTransportValidator` fails host start** on a node carrying a `cluster-{ClusterId}` role unless it also runs the split-safe transports: `MeshTransport:Mode=ClusterClient`, `Telemetry:Mode=Grpc` (driver), `TelemetryDial:Mode=Grpc` (admin). A DPS transport on a cluster-role node would try to gossip across a partition that no longer exists. +- **The compiled transport-mode defaults were deliberately NOT flipped** — `MeshTransport:Mode`/`Telemetry:Mode`/`TelemetryDial:Mode` all still default to `Dps`; a blast-radius assessment showed flipping the default breaks every integration test and appsettings profile that leaves the key unset. The validator + explicit per-deployment config is the guardrail, not a default flip. +- **Secrets replication is pair-local** — each pair replicates its own Akka DPS secrets within itself; there is no fleet-wide/cross-cluster sync after the split (sites have no SQL, so no SQL-hub, consistent with Phase 4). + +See `docs/Redundancy.md` §"Per-cluster meshes (Phase 6)", `docs/Configuration.md` §"Per-cluster mesh split (Phase 6)", and `docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md`. ## Mesh command transport (`MeshTransport`) @@ -246,12 +257,14 @@ change, not a redeploy. Things worth knowing before touching it: `DeploymentArtifact.ResolveClusterScope`). `ClusterClient.Send` delivers to exactly **one** registered actor — it would deploy to a single node while every other node silently kept its old configuration, and the deployment could still seal green. -- **Exactly ONE fleet-wide ClusterClient while the fleet is one mesh.** A receptionist serves its whole - cluster, so `SendToAll` reaches every registered node-comm actor regardless of which node's address was - dialled. One client per application `Cluster` — the obviously-right-looking shape, and what Phase 6 - wants — would fan each command out once per cluster (N× duplicate dispatch *and* N× duplicate ack). - Marked `TODO(Phase 6)` in `CentralCommunicationActor.RebuildClient`. **Corollary:** the contact set does - not scope delivery; excluding a `MaintenanceMode` node from the contacts does not stop it receiving. +- **RESOLVED by Phase 6 — one `ClusterClient` per application `Cluster`, not one fleet-wide.** When + this was written the fleet was still one mesh, so a receptionist served the whole cluster and + `SendToAll` reached every registered node-comm actor regardless of which node's address was dialled; + the `TODO(Phase 6)` in `CentralCommunicationActor.RebuildClient` asked for one client per application + `Cluster` instead. Phase 6 split the fleet into one mesh per `Cluster` and shipped exactly that — see + the [Redundancy](#redundancy) section's Phase 6 paragraph above. The `SendToAll`-within-a-mesh + behaviour below is otherwise unchanged; a `MaintenanceMode` node still isn't excluded from delivery + within its own mesh. - **Inbound commands re-emit on the node's `EventStream`, not on their DPS topic** — DPS is mesh-wide and central `SendToAll`s to every node, so a DPS re-publish would deliver N copies to every subscriber. Node-side subscribers (`DriverHostActor`, `ScriptedAlarmHostActor`) accept both. diff --git a/docs/Configuration.md b/docs/Configuration.md index 0b3b76dd..f9208c28 100644 --- a/docs/Configuration.md +++ b/docs/Configuration.md @@ -103,13 +103,40 @@ The checked-in `appsettings*.json` files are deliberately thin: they carry only | `Hostname` | string | `0.0.0.0` | Bind hostname. | | `Port` | int | `4053` | Cluster transport port. **Duplicated as `ClusterNode.AkkaPort` in the Config DB** — see the note below. | | `PublicHostname` | string | `127.0.0.1` | Hostname advertised in cluster gossip; must be reachable by peers. **Duplicated as `ClusterNode.Host`** — see the note below. | -| `SeedNodes` | string[] | `[]` | Seed nodes for bootstrapping. **Ordered:** a node that appears in its own list must be entry 0 (only `seed-nodes[0]` can form a new cluster), or the host refuses to start — see [Redundancy.md § Bootstrap: self-first seed ordering](Redundancy.md#bootstrap-self-first-seed-ordering). | -| `Roles` | string[] | `[]` | Cluster roles for this node. When empty, falls back to `OTOPCUA_ROLES`. Allowed values: `admin`, `driver`, `dev`. | +| `SeedNodes` | string[] | `[]` | Seed nodes for bootstrapping. **Ordered:** a node that appears in its own list must be entry 0 (only `seed-nodes[0]` can form a new cluster), or the host refuses to start — see [Redundancy.md § Bootstrap: self-first seed ordering](Redundancy.md#bootstrap-self-first-seed-ordering). **Per-pair since Phase 6:** each application `Cluster`'s pair is its own Akka mesh, so a node's `SeedNodes` lists only itself (first) and its own pair partner (second) — never a node from another cluster's pair. | +| `Roles` | string[] | `[]` | Cluster roles for this node. When empty, falls back to `OTOPCUA_ROLES`. Allowed values: `admin`, `driver`, `dev`, plus a per-cluster `cluster-{ClusterId}` role (e.g. `cluster-MAIN`, `cluster-SITE-A`) — see [Per-cluster mesh split (Phase 6)](#per-cluster-mesh-split-phase-6-cluster-clusterid-roles--splittopologytransportvalidator) below. | > The full redundancy model (ServiceLevel tiers, split-brain, peer discovery) is in [`Redundancy.md`](Redundancy.md). The OPC UA peer-URI advertising lives in the `OpcUa:PeerApplicationUris` key above. > **`Port` / `PublicHostname` are stored twice.** The node binds from these keys; the fleet's `ClusterNode` row records the same address as `AkkaPort` / `Host` so that **central can dial the node without sharing a gossip ring with it** — which is what [per-cluster mesh Phase 2](plans/2026-07-21-per-cluster-mesh-design.md) needs. Nothing in the schema makes the two agree, so `ClusterNodeAddressReconcilerActor` (admin-role singleton) compares them against live membership and logs an Error on mismatch. If you change `Cluster:Port` or `Cluster:PublicHostname` on a node, **update its `ClusterNode` row too** — a node binding 4054 with a row saying 4053 still gossips fine today, and fails silently in Phase 2. The row's `NodeId` is `host:port` and is also the deploy path's ack identity, so a drifted node's deployment acks stop matching as well. See [`config-db-schema.md` § `ClusterNode`](v2/config-db-schema.md#clusternode). +### Per-cluster mesh split (Phase 6): `cluster-{ClusterId}` roles + `SplitTopologyTransportValidator` + +- **Purpose:** [per-cluster mesh](plans/2026-07-22-per-cluster-mesh-program.md) **Phase 6** split the + single fleet-wide Akka mesh into one independent 2-node mesh per application `Cluster`. A node's + `Cluster:Roles` now carries a `cluster-{ClusterId}` role alongside `driver` (and `admin` for a fused + central node) — e.g. `driver,cluster-SITE-A` — and its `Cluster:SeedNodes` lists only itself and its + own pair partner (see the `Roles`/`SeedNodes` rows above). Nodes in different clusters never gossip + with each other. +- **`SplitTopologyTransportValidator`** (`src/Core/ZB.MOM.WW.OtOpcUa.Cluster/`, wired at + `ValidateOnStart` alongside the other `Cluster`-family validators) fails host start on any node + carrying a `cluster-{ClusterId}` role unless it is also configured for the split-safe transports: + `MeshTransport:Mode = ClusterClient`, `Telemetry:Mode = Grpc` on a driver node, `TelemetryDial:Mode = + Grpc` on an admin node. A DPS-mode transport on a cluster-role node would try to gossip across a + partition boundary that no longer exists, so the validator fails closed rather than let commands or + telemetry silently vanish. A legacy node with no `cluster-{ClusterId}` role is exempt and may stay on + `Dps`. +- **The compiled transport-mode defaults were deliberately NOT flipped.** `MeshTransport:Mode`, + `Telemetry:Mode`, and `TelemetryDial:Mode` (documented below) all still default to `Dps` — a + blast-radius assessment found that flipping the compiled default would break every integration test + and every existing `appsettings.json` profile that leaves these keys unset. `SplitTopologyTransportValidator` + plus explicit per-deployment configuration is the guardrail, not a default flip; every node that + actually carries a `cluster-{ClusterId}` role must set the split-safe modes explicitly. +- See [`Redundancy.md` § Per-cluster meshes (Phase 6)](Redundancy.md#per-cluster-meshes-phase-6) for the + full runtime picture (one Primary per pair, the cluster-scoped `RedundancyStateActor` singleton, + `ClusterNodeAddressReconciler` own-cluster-scoping, and central's one-`ClusterClient`-per-`Cluster` + fan-out). + ### `MeshTransport` (central↔node command transport) - **Purpose:** selects how central's three command channels (`deployments`, `driver-control`, `alarm-commands`) and the node's `deployment-acks` reply reach the other side — over the mesh-wide DistributedPubSub, or over an Akka `ClusterClient` that does **not** require central and the node to share a gossip ring. The second option is what [per-cluster mesh](plans/2026-07-21-per-cluster-mesh-design.md) Phase 6 needs once the meshes split. diff --git a/docs/Redundancy.md b/docs/Redundancy.md index d08756c8..a276c56e 100644 --- a/docs/Redundancy.md +++ b/docs/Redundancy.md @@ -108,22 +108,59 @@ plane (writes, alarm acks, the alerts emit, the alarm-history drain) on the wron *higher* address so the two orderings genuinely disagree. `NodeRedundancyState.IsRoleLeaderForDriver` was renamed `IsDriverPrimary` to stop the name asserting the old derivation. -> **KNOWN LIMITATION — the election is per *Akka* cluster, not per application `Cluster`.** -> The election yields exactly **one** Primary across the whole Akka cluster. A fleet that -> runs several application clusters (each with its own redundant pair) inside one Akka cluster — -> which is what `docker-dev/docker-compose.yml` models, with MAIN, SITE-A and SITE-B — therefore has -> one Primary among *all* its driver nodes, not one per pair. Every consumer of the role inherits -> this: `ServiceLevel`, the `alerts` emit gate, and the inbound device-write gate below. +> **RESOLVED by Phase 6 — the election is now per application `Cluster`, not per Akka mesh.** +> Phase 6 of the per-cluster mesh program split the single fleet-wide Akka mesh into one independent +> 2-node mesh per application `Cluster` (see [Per-cluster meshes (Phase 6)](#per-cluster-meshes-phase-6) +> below). `RedundancyStateActor` is now a `cluster-{ClusterId}`-scoped singleton spawned on every driver +> node, so each pair elects its own Primary — the whole-Akka-cluster mismatch this block used to describe +> no longer exists. See `docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md`. > -> The unit of redundancy everywhere else in the product is the application `Cluster` (`ClusterId`) — -> it owns drivers, devices and `ClusterNode` rows, and the LocalDb deployment-artifact cache is -> already keyed by it *so that a pair shares one entry*. Scoping the election the same way is the -> fix; `RedundancyStateActor` is an admin-role singleton in the same assembly as -> `AdminOperationsActor`, which already reads and groups `ClusterNodes` by cluster. -> -> Surfaced by the LocalDb Phase 2 live gate, where it stopped the alarm-history drain on every node -> in the fleet — see `docs/plans/2026-07-20-localdb-phase2-live-gate.md`. The alarm drain no longer -> depends on this (it defers only to a node that shares its queue), but the other three gates still do. +> Originally surfaced by the LocalDb Phase 2 live gate, where the old whole-mesh election stopped the +> alarm-history drain on every node in the fleet — see `docs/plans/2026-07-20-localdb-phase2-live-gate.md`. + +## Per-cluster meshes (Phase 6) + +The fleet no longer runs one Akka mesh. Phase 6 split it into **three independent 2-node meshes** — the +central pair (roles `admin,driver,cluster-MAIN`), the site-a pair (`driver,cluster-SITE-A`), and the +site-b pair (`driver,cluster-SITE-B`) — all sharing the same `ActorSystem` name (`otopcua`) on one network, +separated purely by **per-pair self-first seed partitioning**: each node's `Cluster:SeedNodes` lists only +itself and its own pair partner, so gossip never crosses a pair boundary (see +[Bootstrap: self-first seed ordering](#bootstrap-self-first-seed-ordering)). + +Consequences: + +- **One Primary per pair — two or more Primaries fleet-wide, by design.** Each mesh runs its own + election, so a three-mesh fleet has three Primaries, one per application `Cluster`. This is correct, + not a regression of the old single-Primary assumption. +- **`RedundancyStateActor` is a `cluster-{ClusterId}`-scoped singleton**, spawned on every driver node + (falling back to the plain `driver` role for a legacy node not carrying a cluster role). Each pair + elects and publishes its own `redundancy-state` on its own mesh's DPS — the DPS topic name is + unchanged, but it no longer reaches past the pair. +- **`ClusterNodeAddressReconciler` is own-cluster-scoped** — an admin node can only reconcile + `ClusterNode` rows belonging to its own cluster; it has no gossip visibility into another mesh to + reconcile against. +- **`CentralCommunicationActor` creates one `ClusterClient` per application `Cluster`** and fans + commands across them, rather than one fleet-wide client. Central reaches a site mesh over + `ClusterClient` (commands), gRPC (telemetry — see `docs/Telemetry.md`), and `ConfigServe` (config + fetch — see [`ConfigSource`/`ConfigServe`](../CLAUDE.md#config-source-configsource--configserve)), + never over gossip. +- **`SplitTopologyTransportValidator` fails host start** on a node carrying a `cluster-{ClusterId}` + role unless it is also configured for the split-safe transports: `MeshTransport:Mode = ClusterClient`, + `Telemetry:Mode = Grpc` (driver nodes), `TelemetryDial:Mode = Grpc` (admin nodes). A DPS-mode transport + on a cluster-role node would silently try to gossip across a partition that no longer exists, so the + validator fails closed instead. Legacy/non-cluster-role nodes are exempt. +- **The compiled transport-mode defaults were deliberately NOT flipped** — `MeshTransport:Mode`, + `Telemetry:Mode`, and `TelemetryDial:Mode` all still default to `Dps`. A blast-radius assessment + found flipping the compiled default would break every integration test and every existing + `appsettings.json` profile that doesn't explicitly set these keys. The guardrail is the validator + above plus explicit per-deployment configuration, not a default flip. + +### Secrets replication is pair-local + +Akka DPS secret replication is scoped to each 2-node mesh: a pair replicates its own secrets to its +own partner and no further. There is no fleet-wide or cross-cluster secret sync after the split — by +design, consistent with Phase 4 (site nodes have no SQL, so there is no SQL-backed hub to fan a +cross-mesh sync through). ## Data flow @@ -316,18 +353,17 @@ advertises `ServiceLevel` 250. OPC UA clients re-select. | Confirmation | A dialog naming the node, and what follows (restart, ServiceLevel 250 moves, clients re-select). | | Audit | Written through the shared `ZB.MOM.WW.Audit.IAuditWriter` seam **before** the Leave — action `cluster.manual-failover`, actor, target. A *refused* failover changes nothing and writes nothing. | -> **Mesh-scope caveat.** Until the per-cluster mesh work lands -> (`docs/plans/2026-07-21-per-cluster-mesh-design.md`), the Primary is elected once per Akka mesh rather than -> per application `Cluster` row — so on a fleet running several clusters in one mesh this button acts on the -> whole mesh's Primary, which may belong to a different cluster than the page you are on. The panel says so. -> Phase 6 of the mesh program removes both the caveat and this notice. +Phase 6 of the mesh program (see [Per-cluster meshes (Phase 6)](#per-cluster-meshes-phase-6)) split +the fleet into one 2-node mesh per application `Cluster`, so the button now acts on **this pair's own** +Primary — there is no longer a whole-mesh vs. per-cluster ambiguity to caveat. > **Live gate outstanding.** The auto-down change is verified by unit tests against the effective -> configuration, and by the sister project's live drill on an equivalent Akka version and topology. It has -> **not** yet been drilled on an OtOpcUa two-node rig — the docker-dev rig runs a single six-node mesh, where -> the 1-vs-1 pathology cannot occur. The drill (kill the oldest container of a genuine two-node cluster; -> assert the survivor stays up, takes the singletons, and reports the primary `ServiceLevel`) should run -> alongside the per-cluster mesh work, which makes every mesh exactly two nodes. See +> configuration, and by the sister project's live drill on an equivalent Akka version and topology. It +> was **not** drillable on an OtOpcUa rig before Phase 6, because the docker-dev rig ran a single +> six-node mesh where the 1-vs-1 pathology cannot occur. **Phase 6 has landed and every mesh is now +> exactly two nodes**, so the drill (kill the oldest container of a genuine two-node cluster; assert +> the survivor stays up, takes the singletons, and reports the primary `ServiceLevel`) is finally +> testable — deferred to Phase 7's failover drill. See > `docs/plans/2026-07-21-per-cluster-mesh-design.md` §6.2 / Phase 0a. ## Bootstrap: self-first seed ordering diff --git a/docs/plans/2026-07-21-per-cluster-mesh-design.md b/docs/plans/2026-07-21-per-cluster-mesh-design.md index 8a39d63b..4b41c602 100644 --- a/docs/plans/2026-07-21-per-cluster-mesh-design.md +++ b/docs/plans/2026-07-21-per-cluster-mesh-design.md @@ -312,7 +312,7 @@ Deliberately not a task plan — per-phase plans follow, one at a time. | 3 | Config fetch-and-cache: artifact served by central, driver nodes read LocalDb (§6.1) | No | | 4 | Cut the driver-side ConfigDb connection: re-home `EfAlarmConditionStateStore`, resolve the `DbHealthProbeActor` ServiceLevel input, audit `OpcUaPublishActor` | No | | 5 | gRPC stream contract; migrate the seven observability topics | No | -| 6 | Mesh partition: per-cluster seed nodes, cluster-scoped roles + singletons; rig models the real topology | No | +| 6 | Mesh partition: per-cluster seed nodes, cluster-scoped roles + singletons; rig models the real topology | No — **DONE 2026-07-24**, see `2026-07-22-per-cluster-mesh-program.md` §Phase 6 and `2026-07-24-mesh-phase6-partition-and-colocation.md` | | 7 | Failover drill (both directions, per §6.2); live gate | No | Phases 3 and 4 are the ones that change a running system's data path rather than its wiring, and each diff --git a/docs/plans/2026-07-22-per-cluster-mesh-program.md b/docs/plans/2026-07-22-per-cluster-mesh-program.md index 15d5e7f6..5bca2893 100644 --- a/docs/plans/2026-07-22-per-cluster-mesh-program.md +++ b/docs/plans/2026-07-22-per-cluster-mesh-program.md @@ -263,25 +263,45 @@ kill-and-reconnect of a site node's dialer recovers every stream (driver-health/ from the node hub's snapshot replay). Treat Phase 5 as **code-complete, not verified**, until this lands — see `docs/plans/2026-07-23-mesh-phase5-grpc-telemetry-stream.md` Task 12. -### Phase 6 — Mesh partition + co-location topology +### Phase 6 — Mesh partition + co-location topology — **DONE 2026-07-24** **Scope:** per-cluster seed nodes — ~~adopt ScadaBridge's self-first ordering and RETIRE the `SelfFormAfter` watchdog + TCP reachability guard~~ **DONE EARLY 2026-07-22** (docker-dev `central-2` swapped to self-first, `ClusterBootstrapFallback`/`SelfFormAfter` deleted, `AkkaClusterOptionsValidator` ports ScadaBridge's startup rule in its conditional form, and `SelfFirstSeedBootstrapTests` replaces `SelfFormBootstrapTests`; see `docs/Redundancy.md` -§"Bootstrap: self-first seed ordering"). What remains for this phase is the per-pair -`Cluster__SeedNodes__*` matrix once the meshes actually split — every node in a pair is then a seed -of its own mesh, so the site-node exemption disappears; cluster-scoped roles `cluster-{ClusterId}` + singleton re-scoping, -central pair keeps the admin singletons; **docker-dev rig rewritten** to model the real topology — -including the co-location port table above (both products' compose files on shared per-site -networks, real LocalDb sync ports); remove the ClusterRedundancy page's mesh-scope caveat (the -election is pair-local now) and the fallback's site-node island-guard docs note (moot — every -node is a seed of its own mesh); `Cluster__SeedNodes__*` env matrix per pair. -**Exit gate:** rig up in the new shape; every existing live-gated behavior re-verified per pair -(deploy, redundancy 250/240 per pair — **two Primaries fleet-wide, one per pair, by design**); -secrets Akka replication re-verified or re-scoped (it rides DPS on the current single mesh — its -topology must be re-decided here, likely SQL-hub mode like ScadaBridge, since pub/sub cannot -cross separate meshes). +§"Bootstrap: self-first seed ordering"). The rest of the phase — actually splitting the single +fleet-wide mesh into three independent 2-node meshes (central `cluster-MAIN`, site-a +`cluster-SITE-A`, site-b `cluster-SITE-B`) — **shipped 2026-07-24**. See +`docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md` for the full plan and task list. +Delivered: +- Per-pair `Cluster:SeedNodes` (each node lists itself first, its pair partner second — no + cross-pair seeds) + `cluster-{ClusterId}` roles on every driver node + singleton re-scoping: + `RedundancyStateActor` is now a `cluster-{ClusterId}`-scoped singleton spawned on every driver + node (falls back to plain `driver` for legacy nodes), so **each pair elects its own Primary — + two or more Primaries fleet-wide, by design**; `ClusterNodeAddressReconciler` is own-cluster-scoped. +- `CentralCommunicationActor` now creates one `ClusterClient` per application `Cluster` and fans + commands across them (closes the `TODO(Phase 6)` left by Phase 2). +- New `SplitTopologyTransportValidator` fails host start on a `cluster-{ClusterId}`-role node left + on a DPS transport (`MeshTransport:Mode` must be `ClusterClient`; `Telemetry:Mode`/`TelemetryDial:Mode` + must be `Grpc`) — see `docs/Configuration.md` §"Per-cluster mesh split (Phase 6)". +- **Three decisions worth recording:** + 1. **Secrets replication went pair-local, not SQL-hub.** The design flagged "likely SQL-hub mode + like ScadaBridge" as the fallback if DPS couldn't cross the split meshes — in the event, each + pair simply keeps replicating its own secrets within itself (no cross-cluster sync needed, and + no SQL-hub was built, consistent with Phase 4: site nodes have no SQL to hub through). + 2. **The compiled transport-mode defaults (`MeshTransport:Mode`, `Telemetry:Mode`, + `TelemetryDial:Mode`) were deliberately left at `Dps`**, not flipped to the split-safe modes — + a blast-radius assessment found flipping the default breaks every integration test and every + appsettings profile that leaves the key unset. The guardrail is + `SplitTopologyTransportValidator` plus explicit per-deployment config, not a default flip. + 3. **The docker-dev rig models three OtOpcUa-only meshes** (central/site-a/site-b), not the + ScadaBridge co-location topology sketched in the design's port table — co-location with + ScadaBridge's own site nodes stayed out of scope for this phase. +**Exit gate:** rig up in the new three-mesh shape; every existing live-gated behavior re-verified +per pair (deploy, redundancy 250/240 per pair, two-Primaries-fleet-wide confirmed); the +ClusterRedundancy page's mesh-scope caveat and `IManualFailoverService`'s doc-comment caveat +removed (the election is pair-local now); `docs/Redundancy.md`'s per-Akka-cluster KNOWN LIMITATION +marked resolved. ### Phase 7 — Failover drill + live gates **Scope:** the drill ScadaBridge already has (`failover-drill.sh` analogue) run per pair, both @@ -318,5 +338,5 @@ resource sizing on the site VMs should be checked once both products run the ful | 3 fetch-and-cache | **DONE 2026-07-23**, merged `d01b0695`, live gate PASSED, pushed to origin (scadaproj umbrella index updated + pushed `b5e7bc8`). gRPC fetch RPC + shared node key; dark switch `ConfigSource:Mode` (Direct default). Rig flip: `OTOPCUA_CONFIG_MODE=FetchAndCache` on the site nodes. See `2026-07-22-mesh-phase3-config-fetch-and-cache.md`. | | 4 cut driver ConfigDb | **DONE 2026-07-23 — live gate PASSED** on `feat/mesh-phase4` (Tasks 0–8 + 1b + 10–11; Task 9 table-drop deferred). ConfigDb admin-only; driver-only ⇒ FetchAndCache (validator); `DbHealthProbeActor` not spawned driver-only (client-visible ServiceLevel 240/250 with central SQL down — proven live); alarm condition state in replicated LocalDb `alarm_condition_state`; central persists acks; `OpcUaPublish` guard split fixes the driver-only address-space wipe; driver-only LDAP maps from appsettings only (6th consumer found mid-phase). Gate: deploy sealed green w/ 4 DB-less site nodes acking, ServiceLevel held 240 w/ SQL down, restart booted last-known-good from the LocalDb pointer. See `2026-07-23-mesh-phase4-cut-driver-configdb.md` + `2026-07-23-mesh-phase4-live-gate.md`. | | 5 gRPC telemetry | **DONE 2026-07-23 — live gate PASSED** on `feat/mesh-phase5`. 4-channel scope (`alerts`/`script-logs`/`driver-health`/`driver-resilience-status`), 3 deferred with rationale (`redundancy-state`, `fleet-status`, `deployment-acks`); dark switch `Telemetry:Mode`/`TelemetryDial:Mode` (Dps default); node hosts server / central dials; auth-from-day-one fail-closed bearer key, superseding design §6.3. Gate: full 12-stream mesh formed in Grpc mode (2 inbound per driver node, 6 outbound per central), AdminUI pill live (data path proven), kill-and-reconnect of a node recovered its stream in ~5s; Dps baseline dials nothing. Surfaced an upgrade gotcha — `ClusterNode.GrpcPort` must be populated on existing deployments (fresh installs seed it; nullable column doesn't backfill) — which live-validated the graceful null-`GrpcPort` skip. See `docs/Telemetry.md`, `2026-07-23-mesh-phase5-grpc-telemetry-stream.md` + `2026-07-23-mesh-phase5-live-gate.md`. | -| 6 mesh partition + co-location | not started | +| 6 mesh partition + co-location | **DONE 2026-07-24** — three independent 2-node meshes (central/site-a/site-b), `cluster-{ClusterId}` roles, per-pair self-first seeds, cluster-scoped `RedundancyStateActor` singleton (two+ Primaries fleet-wide, by design), one `ClusterClient` per `Cluster`, `SplitTopologyTransportValidator`, pair-local secrets replication, compiled transport defaults intentionally kept `Dps`. See `2026-07-24-mesh-phase6-partition-and-colocation.md`. | | 7 drill + live gates | not started | diff --git a/src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Components/Pages/Clusters/ClusterRedundancy.razor b/src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Components/Pages/Clusters/ClusterRedundancy.razor index 36f3201f..7f1741f8 100644 --- a/src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Components/Pages/Clusters/ClusterRedundancy.razor +++ b/src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Components/Pages/Clusters/ClusterRedundancy.razor @@ -68,10 +68,10 @@ else

- Read live from cluster state on this node. Mesh-wide scope: the Primary is - elected once per Akka mesh, not per cluster row — in the current single-mesh topology this - acts on the whole mesh's Primary, which may be a node of another - Cluster. See docs/Redundancy.md. + Read live from cluster state on this node. The Primary shown is + this application Cluster's own Primary — since + the per-cluster mesh split, each cluster's redundant pair runs its own independent 2-node + Akka mesh and elects its own Primary. See docs/Redundancy.md.

diff --git a/src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Redundancy/IManualFailoverService.cs b/src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Redundancy/IManualFailoverService.cs index f7369427..da10092e 100644 --- a/src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Redundancy/IManualFailoverService.cs +++ b/src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Redundancy/IManualFailoverService.cs @@ -31,10 +31,9 @@ public sealed record ManualFailoverSnapshot(string? PrimaryAddress, IReadOnlyLis /// moves with it and OPC UA clients re-select. /// /// -/// Mesh-scope caveat. Until the per-cluster mesh work lands -/// (docs/plans/2026-07-21-per-cluster-mesh-design.md) the Primary is elected once per -/// Akka mesh, not per application Cluster row. On a fleet running several application -/// clusters in one mesh this acts on the whole mesh's Primary. The UI says so. +/// Post-Phase-6 (per-cluster mesh split): the Primary is elected per application +/// Cluster, because each cluster's redundant pair now runs its own independent +/// 2-node Akka mesh. This service therefore acts on this pair's own Primary. /// /// public interface IManualFailoverService