docs(mesh): Phase 6 — mark done, remove single-mesh caveats, document pair-local meshes
Redundancy.md: KNOWN-LIMITATION (per-Akka-cluster election) marked RESOLVED;
new 'Per-cluster meshes (Phase 6)' section + pair-local secrets note; manual-
failover mesh-scope caveat removed; auto-down 1-vs-1 note updated (every mesh is
now two nodes, drill deferred to Phase 7). IManualFailoverService + ClusterRedundancy
razor caveats updated to pair-local reality. Configuration.md documents the
cluster-{ClusterId} role, per-pair self-first seeds, SplitTopologyTransportValidator,
and the intentional Dps default. Program + design status tables + CLAUDE.md marked
Phase 6 DONE.
(Task 8 subagent completed the edits but died on an API error before committing;
reviewed and committed by the controller.)
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
@@ -228,7 +228,18 @@ The server supports non-transparent warm/hot redundancy via the `Redundancy` sec
|
||||
|
||||
Assert the **effective** cluster config off a running `ActorSystem` (as `SplitBrainResolverActivationTests` does), never the typed options or the akka.conf text — several HOCON fragments compete and the losing one still reads correctly in the file.
|
||||
|
||||
**Still per-Akka-cluster, not per application `Cluster`.** One Primary is elected across the whole Akka mesh, so a fleet running several application clusters in one mesh (what docker-dev models) gets one Primary among all driver nodes. The fix is the per-cluster mesh workstream: `docs/plans/2026-07-21-per-cluster-mesh-design.md` (Phases 0a/0b/1 done; 2 code-complete behind a dark switch; 3–7 not started).
|
||||
**RESOLVED by per-cluster mesh Phase 6 (2026-07-24) — no longer per-Akka-cluster.** The "one Primary across the whole Akka mesh" limitation described above is gone; see the Phase 6 paragraph immediately below.
|
||||
|
||||
**Per-cluster mesh Phase 6 (2026-07-24) — the fleet is now three independent 2-node meshes, not one.** The single fleet-wide Akka mesh was split into one mesh per application `Cluster`: the central pair (`admin,driver,cluster-MAIN`), site-a (`driver,cluster-SITE-A`), site-b (`driver,cluster-SITE-B`) — same `ActorSystem` name, separated purely by per-pair self-first seeds (each node's `Cluster:SeedNodes` lists itself first, its own pair partner second, never a node from another cluster's pair). Consequences:
|
||||
|
||||
- **The redundancy singleton is cluster-scoped and spawned per driver node.** `RedundancyStateActor` is now a `cluster-{ClusterId}`-scoped singleton (falls back to plain `driver` for a legacy node), so each pair elects and publishes its **own** Primary on its own mesh's DPS. **Two or more Primaries exist fleet-wide, one per pair — by design, not a bug.**
|
||||
- `ClusterNodeAddressReconciler` is **own-cluster-scoped** — an admin node can only reconcile `ClusterNode` rows in its own cluster; it has no gossip visibility into another mesh.
|
||||
- `CentralCommunicationActor` creates **one `ClusterClient` per application `Cluster`** and fans commands across them (closing the `TODO(Phase 6)` Phase 2 left behind), rather than one fleet-wide client.
|
||||
- **`SplitTopologyTransportValidator` fails host start** on a node carrying a `cluster-{ClusterId}` role unless it also runs the split-safe transports: `MeshTransport:Mode=ClusterClient`, `Telemetry:Mode=Grpc` (driver), `TelemetryDial:Mode=Grpc` (admin). A DPS transport on a cluster-role node would try to gossip across a partition that no longer exists.
|
||||
- **The compiled transport-mode defaults were deliberately NOT flipped** — `MeshTransport:Mode`/`Telemetry:Mode`/`TelemetryDial:Mode` all still default to `Dps`; a blast-radius assessment showed flipping the default breaks every integration test and appsettings profile that leaves the key unset. The validator + explicit per-deployment config is the guardrail, not a default flip.
|
||||
- **Secrets replication is pair-local** — each pair replicates its own Akka DPS secrets within itself; there is no fleet-wide/cross-cluster sync after the split (sites have no SQL, so no SQL-hub, consistent with Phase 4).
|
||||
|
||||
See `docs/Redundancy.md` §"Per-cluster meshes (Phase 6)", `docs/Configuration.md` §"Per-cluster mesh split (Phase 6)", and `docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md`.
|
||||
|
||||
## Mesh command transport (`MeshTransport`)
|
||||
|
||||
@@ -246,12 +257,14 @@ change, not a redeploy. Things worth knowing before touching it:
|
||||
`DeploymentArtifact.ResolveClusterScope`). `ClusterClient.Send` delivers to exactly **one** registered
|
||||
actor — it would deploy to a single node while every other node silently kept its old configuration,
|
||||
and the deployment could still seal green.
|
||||
- **Exactly ONE fleet-wide ClusterClient while the fleet is one mesh.** A receptionist serves its whole
|
||||
cluster, so `SendToAll` reaches every registered node-comm actor regardless of which node's address was
|
||||
dialled. One client per application `Cluster` — the obviously-right-looking shape, and what Phase 6
|
||||
wants — would fan each command out once per cluster (N× duplicate dispatch *and* N× duplicate ack).
|
||||
Marked `TODO(Phase 6)` in `CentralCommunicationActor.RebuildClient`. **Corollary:** the contact set does
|
||||
not scope delivery; excluding a `MaintenanceMode` node from the contacts does not stop it receiving.
|
||||
- **RESOLVED by Phase 6 — one `ClusterClient` per application `Cluster`, not one fleet-wide.** When
|
||||
this was written the fleet was still one mesh, so a receptionist served the whole cluster and
|
||||
`SendToAll` reached every registered node-comm actor regardless of which node's address was dialled;
|
||||
the `TODO(Phase 6)` in `CentralCommunicationActor.RebuildClient` asked for one client per application
|
||||
`Cluster` instead. Phase 6 split the fleet into one mesh per `Cluster` and shipped exactly that — see
|
||||
the [Redundancy](#redundancy) section's Phase 6 paragraph above. The `SendToAll`-within-a-mesh
|
||||
behaviour below is otherwise unchanged; a `MaintenanceMode` node still isn't excluded from delivery
|
||||
within its own mesh.
|
||||
- **Inbound commands re-emit on the node's `EventStream`, not on their DPS topic** — DPS is mesh-wide and
|
||||
central `SendToAll`s to every node, so a DPS re-publish would deliver N copies to every subscriber.
|
||||
Node-side subscribers (`DriverHostActor`, `ScriptedAlarmHostActor`) accept both.
|
||||
|
||||
+29
-2
@@ -103,13 +103,40 @@ The checked-in `appsettings*.json` files are deliberately thin: they carry only
|
||||
| `Hostname` | string | `0.0.0.0` | Bind hostname. |
|
||||
| `Port` | int | `4053` | Cluster transport port. **Duplicated as `ClusterNode.AkkaPort` in the Config DB** — see the note below. |
|
||||
| `PublicHostname` | string | `127.0.0.1` | Hostname advertised in cluster gossip; must be reachable by peers. **Duplicated as `ClusterNode.Host`** — see the note below. |
|
||||
| `SeedNodes` | string[] | `[]` | Seed nodes for bootstrapping. **Ordered:** a node that appears in its own list must be entry 0 (only `seed-nodes[0]` can form a new cluster), or the host refuses to start — see [Redundancy.md § Bootstrap: self-first seed ordering](Redundancy.md#bootstrap-self-first-seed-ordering). |
|
||||
| `Roles` | string[] | `[]` | Cluster roles for this node. When empty, falls back to `OTOPCUA_ROLES`. Allowed values: `admin`, `driver`, `dev`. |
|
||||
| `SeedNodes` | string[] | `[]` | Seed nodes for bootstrapping. **Ordered:** a node that appears in its own list must be entry 0 (only `seed-nodes[0]` can form a new cluster), or the host refuses to start — see [Redundancy.md § Bootstrap: self-first seed ordering](Redundancy.md#bootstrap-self-first-seed-ordering). **Per-pair since Phase 6:** each application `Cluster`'s pair is its own Akka mesh, so a node's `SeedNodes` lists only itself (first) and its own pair partner (second) — never a node from another cluster's pair. |
|
||||
| `Roles` | string[] | `[]` | Cluster roles for this node. When empty, falls back to `OTOPCUA_ROLES`. Allowed values: `admin`, `driver`, `dev`, plus a per-cluster `cluster-{ClusterId}` role (e.g. `cluster-MAIN`, `cluster-SITE-A`) — see [Per-cluster mesh split (Phase 6)](#per-cluster-mesh-split-phase-6-cluster-clusterid-roles--splittopologytransportvalidator) below. |
|
||||
|
||||
> The full redundancy model (ServiceLevel tiers, split-brain, peer discovery) is in [`Redundancy.md`](Redundancy.md). The OPC UA peer-URI advertising lives in the `OpcUa:PeerApplicationUris` key above.
|
||||
|
||||
> **`Port` / `PublicHostname` are stored twice.** The node binds from these keys; the fleet's `ClusterNode` row records the same address as `AkkaPort` / `Host` so that **central can dial the node without sharing a gossip ring with it** — which is what [per-cluster mesh Phase 2](plans/2026-07-21-per-cluster-mesh-design.md) needs. Nothing in the schema makes the two agree, so `ClusterNodeAddressReconcilerActor` (admin-role singleton) compares them against live membership and logs an Error on mismatch. If you change `Cluster:Port` or `Cluster:PublicHostname` on a node, **update its `ClusterNode` row too** — a node binding 4054 with a row saying 4053 still gossips fine today, and fails silently in Phase 2. The row's `NodeId` is `host:port` and is also the deploy path's ack identity, so a drifted node's deployment acks stop matching as well. See [`config-db-schema.md` § `ClusterNode`](v2/config-db-schema.md#clusternode).
|
||||
|
||||
### Per-cluster mesh split (Phase 6): `cluster-{ClusterId}` roles + `SplitTopologyTransportValidator`
|
||||
|
||||
- **Purpose:** [per-cluster mesh](plans/2026-07-22-per-cluster-mesh-program.md) **Phase 6** split the
|
||||
single fleet-wide Akka mesh into one independent 2-node mesh per application `Cluster`. A node's
|
||||
`Cluster:Roles` now carries a `cluster-{ClusterId}` role alongside `driver` (and `admin` for a fused
|
||||
central node) — e.g. `driver,cluster-SITE-A` — and its `Cluster:SeedNodes` lists only itself and its
|
||||
own pair partner (see the `Roles`/`SeedNodes` rows above). Nodes in different clusters never gossip
|
||||
with each other.
|
||||
- **`SplitTopologyTransportValidator`** (`src/Core/ZB.MOM.WW.OtOpcUa.Cluster/`, wired at
|
||||
`ValidateOnStart` alongside the other `Cluster`-family validators) fails host start on any node
|
||||
carrying a `cluster-{ClusterId}` role unless it is also configured for the split-safe transports:
|
||||
`MeshTransport:Mode = ClusterClient`, `Telemetry:Mode = Grpc` on a driver node, `TelemetryDial:Mode =
|
||||
Grpc` on an admin node. A DPS-mode transport on a cluster-role node would try to gossip across a
|
||||
partition boundary that no longer exists, so the validator fails closed rather than let commands or
|
||||
telemetry silently vanish. A legacy node with no `cluster-{ClusterId}` role is exempt and may stay on
|
||||
`Dps`.
|
||||
- **The compiled transport-mode defaults were deliberately NOT flipped.** `MeshTransport:Mode`,
|
||||
`Telemetry:Mode`, and `TelemetryDial:Mode` (documented below) all still default to `Dps` — a
|
||||
blast-radius assessment found that flipping the compiled default would break every integration test
|
||||
and every existing `appsettings.json` profile that leaves these keys unset. `SplitTopologyTransportValidator`
|
||||
plus explicit per-deployment configuration is the guardrail, not a default flip; every node that
|
||||
actually carries a `cluster-{ClusterId}` role must set the split-safe modes explicitly.
|
||||
- See [`Redundancy.md` § Per-cluster meshes (Phase 6)](Redundancy.md#per-cluster-meshes-phase-6) for the
|
||||
full runtime picture (one Primary per pair, the cluster-scoped `RedundancyStateActor` singleton,
|
||||
`ClusterNodeAddressReconciler` own-cluster-scoping, and central's one-`ClusterClient`-per-`Cluster`
|
||||
fan-out).
|
||||
|
||||
### `MeshTransport` (central↔node command transport)
|
||||
|
||||
- **Purpose:** selects how central's three command channels (`deployments`, `driver-control`, `alarm-commands`) and the node's `deployment-acks` reply reach the other side — over the mesh-wide DistributedPubSub, or over an Akka `ClusterClient` that does **not** require central and the node to share a gossip ring. The second option is what [per-cluster mesh](plans/2026-07-21-per-cluster-mesh-design.md) Phase 6 needs once the meshes split.
|
||||
|
||||
+61
-25
@@ -108,22 +108,59 @@ plane (writes, alarm acks, the alerts emit, the alarm-history drain) on the wron
|
||||
*higher* address so the two orderings genuinely disagree. `NodeRedundancyState.IsRoleLeaderForDriver`
|
||||
was renamed `IsDriverPrimary` to stop the name asserting the old derivation.
|
||||
|
||||
> **KNOWN LIMITATION — the election is per *Akka* cluster, not per application `Cluster`.**
|
||||
> The election yields exactly **one** Primary across the whole Akka cluster. A fleet that
|
||||
> runs several application clusters (each with its own redundant pair) inside one Akka cluster —
|
||||
> which is what `docker-dev/docker-compose.yml` models, with MAIN, SITE-A and SITE-B — therefore has
|
||||
> one Primary among *all* its driver nodes, not one per pair. Every consumer of the role inherits
|
||||
> this: `ServiceLevel`, the `alerts` emit gate, and the inbound device-write gate below.
|
||||
> **RESOLVED by Phase 6 — the election is now per application `Cluster`, not per Akka mesh.**
|
||||
> Phase 6 of the per-cluster mesh program split the single fleet-wide Akka mesh into one independent
|
||||
> 2-node mesh per application `Cluster` (see [Per-cluster meshes (Phase 6)](#per-cluster-meshes-phase-6)
|
||||
> below). `RedundancyStateActor` is now a `cluster-{ClusterId}`-scoped singleton spawned on every driver
|
||||
> node, so each pair elects its own Primary — the whole-Akka-cluster mismatch this block used to describe
|
||||
> no longer exists. See `docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md`.
|
||||
>
|
||||
> The unit of redundancy everywhere else in the product is the application `Cluster` (`ClusterId`) —
|
||||
> it owns drivers, devices and `ClusterNode` rows, and the LocalDb deployment-artifact cache is
|
||||
> already keyed by it *so that a pair shares one entry*. Scoping the election the same way is the
|
||||
> fix; `RedundancyStateActor` is an admin-role singleton in the same assembly as
|
||||
> `AdminOperationsActor`, which already reads and groups `ClusterNodes` by cluster.
|
||||
>
|
||||
> Surfaced by the LocalDb Phase 2 live gate, where it stopped the alarm-history drain on every node
|
||||
> in the fleet — see `docs/plans/2026-07-20-localdb-phase2-live-gate.md`. The alarm drain no longer
|
||||
> depends on this (it defers only to a node that shares its queue), but the other three gates still do.
|
||||
> Originally surfaced by the LocalDb Phase 2 live gate, where the old whole-mesh election stopped the
|
||||
> alarm-history drain on every node in the fleet — see `docs/plans/2026-07-20-localdb-phase2-live-gate.md`.
|
||||
|
||||
## Per-cluster meshes (Phase 6)
|
||||
|
||||
The fleet no longer runs one Akka mesh. Phase 6 split it into **three independent 2-node meshes** — the
|
||||
central pair (roles `admin,driver,cluster-MAIN`), the site-a pair (`driver,cluster-SITE-A`), and the
|
||||
site-b pair (`driver,cluster-SITE-B`) — all sharing the same `ActorSystem` name (`otopcua`) on one network,
|
||||
separated purely by **per-pair self-first seed partitioning**: each node's `Cluster:SeedNodes` lists only
|
||||
itself and its own pair partner, so gossip never crosses a pair boundary (see
|
||||
[Bootstrap: self-first seed ordering](#bootstrap-self-first-seed-ordering)).
|
||||
|
||||
Consequences:
|
||||
|
||||
- **One Primary per pair — two or more Primaries fleet-wide, by design.** Each mesh runs its own
|
||||
election, so a three-mesh fleet has three Primaries, one per application `Cluster`. This is correct,
|
||||
not a regression of the old single-Primary assumption.
|
||||
- **`RedundancyStateActor` is a `cluster-{ClusterId}`-scoped singleton**, spawned on every driver node
|
||||
(falling back to the plain `driver` role for a legacy node not carrying a cluster role). Each pair
|
||||
elects and publishes its own `redundancy-state` on its own mesh's DPS — the DPS topic name is
|
||||
unchanged, but it no longer reaches past the pair.
|
||||
- **`ClusterNodeAddressReconciler` is own-cluster-scoped** — an admin node can only reconcile
|
||||
`ClusterNode` rows belonging to its own cluster; it has no gossip visibility into another mesh to
|
||||
reconcile against.
|
||||
- **`CentralCommunicationActor` creates one `ClusterClient` per application `Cluster`** and fans
|
||||
commands across them, rather than one fleet-wide client. Central reaches a site mesh over
|
||||
`ClusterClient` (commands), gRPC (telemetry — see `docs/Telemetry.md`), and `ConfigServe` (config
|
||||
fetch — see [`ConfigSource`/`ConfigServe`](../CLAUDE.md#config-source-configsource--configserve)),
|
||||
never over gossip.
|
||||
- **`SplitTopologyTransportValidator` fails host start** on a node carrying a `cluster-{ClusterId}`
|
||||
role unless it is also configured for the split-safe transports: `MeshTransport:Mode = ClusterClient`,
|
||||
`Telemetry:Mode = Grpc` (driver nodes), `TelemetryDial:Mode = Grpc` (admin nodes). A DPS-mode transport
|
||||
on a cluster-role node would silently try to gossip across a partition that no longer exists, so the
|
||||
validator fails closed instead. Legacy/non-cluster-role nodes are exempt.
|
||||
- **The compiled transport-mode defaults were deliberately NOT flipped** — `MeshTransport:Mode`,
|
||||
`Telemetry:Mode`, and `TelemetryDial:Mode` all still default to `Dps`. A blast-radius assessment
|
||||
found flipping the compiled default would break every integration test and every existing
|
||||
`appsettings.json` profile that doesn't explicitly set these keys. The guardrail is the validator
|
||||
above plus explicit per-deployment configuration, not a default flip.
|
||||
|
||||
### Secrets replication is pair-local
|
||||
|
||||
Akka DPS secret replication is scoped to each 2-node mesh: a pair replicates its own secrets to its
|
||||
own partner and no further. There is no fleet-wide or cross-cluster secret sync after the split — by
|
||||
design, consistent with Phase 4 (site nodes have no SQL, so there is no SQL-backed hub to fan a
|
||||
cross-mesh sync through).
|
||||
|
||||
## Data flow
|
||||
|
||||
@@ -316,18 +353,17 @@ advertises `ServiceLevel` 250. OPC UA clients re-select.
|
||||
| Confirmation | A dialog naming the node, and what follows (restart, ServiceLevel 250 moves, clients re-select). |
|
||||
| Audit | Written through the shared `ZB.MOM.WW.Audit.IAuditWriter` seam **before** the Leave — action `cluster.manual-failover`, actor, target. A *refused* failover changes nothing and writes nothing. |
|
||||
|
||||
> **Mesh-scope caveat.** Until the per-cluster mesh work lands
|
||||
> (`docs/plans/2026-07-21-per-cluster-mesh-design.md`), the Primary is elected once per Akka mesh rather than
|
||||
> per application `Cluster` row — so on a fleet running several clusters in one mesh this button acts on the
|
||||
> whole mesh's Primary, which may belong to a different cluster than the page you are on. The panel says so.
|
||||
> Phase 6 of the mesh program removes both the caveat and this notice.
|
||||
Phase 6 of the mesh program (see [Per-cluster meshes (Phase 6)](#per-cluster-meshes-phase-6)) split
|
||||
the fleet into one 2-node mesh per application `Cluster`, so the button now acts on **this pair's own**
|
||||
Primary — there is no longer a whole-mesh vs. per-cluster ambiguity to caveat.
|
||||
|
||||
> **Live gate outstanding.** The auto-down change is verified by unit tests against the effective
|
||||
> configuration, and by the sister project's live drill on an equivalent Akka version and topology. It has
|
||||
> **not** yet been drilled on an OtOpcUa two-node rig — the docker-dev rig runs a single six-node mesh, where
|
||||
> the 1-vs-1 pathology cannot occur. The drill (kill the oldest container of a genuine two-node cluster;
|
||||
> assert the survivor stays up, takes the singletons, and reports the primary `ServiceLevel`) should run
|
||||
> alongside the per-cluster mesh work, which makes every mesh exactly two nodes. See
|
||||
> configuration, and by the sister project's live drill on an equivalent Akka version and topology. It
|
||||
> was **not** drillable on an OtOpcUa rig before Phase 6, because the docker-dev rig ran a single
|
||||
> six-node mesh where the 1-vs-1 pathology cannot occur. **Phase 6 has landed and every mesh is now
|
||||
> exactly two nodes**, so the drill (kill the oldest container of a genuine two-node cluster; assert
|
||||
> the survivor stays up, takes the singletons, and reports the primary `ServiceLevel`) is finally
|
||||
> testable — deferred to Phase 7's failover drill. See
|
||||
> `docs/plans/2026-07-21-per-cluster-mesh-design.md` §6.2 / Phase 0a.
|
||||
|
||||
## Bootstrap: self-first seed ordering
|
||||
|
||||
@@ -312,7 +312,7 @@ Deliberately not a task plan — per-phase plans follow, one at a time.
|
||||
| 3 | Config fetch-and-cache: artifact served by central, driver nodes read LocalDb (§6.1) | No |
|
||||
| 4 | Cut the driver-side ConfigDb connection: re-home `EfAlarmConditionStateStore`, resolve the `DbHealthProbeActor` ServiceLevel input, audit `OpcUaPublishActor` | No |
|
||||
| 5 | gRPC stream contract; migrate the seven observability topics | No |
|
||||
| 6 | Mesh partition: per-cluster seed nodes, cluster-scoped roles + singletons; rig models the real topology | No |
|
||||
| 6 | Mesh partition: per-cluster seed nodes, cluster-scoped roles + singletons; rig models the real topology | No — **DONE 2026-07-24**, see `2026-07-22-per-cluster-mesh-program.md` §Phase 6 and `2026-07-24-mesh-phase6-partition-and-colocation.md` |
|
||||
| 7 | Failover drill (both directions, per §6.2); live gate | No |
|
||||
|
||||
Phases 3 and 4 are the ones that change a running system's data path rather than its wiring, and each
|
||||
|
||||
@@ -263,25 +263,45 @@ kill-and-reconnect of a site node's dialer recovers every stream (driver-health/
|
||||
from the node hub's snapshot replay). Treat Phase 5 as **code-complete, not verified**, until this
|
||||
lands — see `docs/plans/2026-07-23-mesh-phase5-grpc-telemetry-stream.md` Task 12.
|
||||
|
||||
### Phase 6 — Mesh partition + co-location topology
|
||||
### Phase 6 — Mesh partition + co-location topology — **DONE 2026-07-24**
|
||||
**Scope:** per-cluster seed nodes — ~~adopt ScadaBridge's self-first ordering and RETIRE the
|
||||
`SelfFormAfter` watchdog + TCP reachability guard~~ **DONE EARLY 2026-07-22** (docker-dev
|
||||
`central-2` swapped to self-first, `ClusterBootstrapFallback`/`SelfFormAfter` deleted,
|
||||
`AkkaClusterOptionsValidator` ports ScadaBridge's startup rule in its conditional form, and
|
||||
`SelfFirstSeedBootstrapTests` replaces `SelfFormBootstrapTests`; see `docs/Redundancy.md`
|
||||
§"Bootstrap: self-first seed ordering"). What remains for this phase is the per-pair
|
||||
`Cluster__SeedNodes__*` matrix once the meshes actually split — every node in a pair is then a seed
|
||||
of its own mesh, so the site-node exemption disappears; cluster-scoped roles `cluster-{ClusterId}` + singleton re-scoping,
|
||||
central pair keeps the admin singletons; **docker-dev rig rewritten** to model the real topology —
|
||||
including the co-location port table above (both products' compose files on shared per-site
|
||||
networks, real LocalDb sync ports); remove the ClusterRedundancy page's mesh-scope caveat (the
|
||||
election is pair-local now) and the fallback's site-node island-guard docs note (moot — every
|
||||
node is a seed of its own mesh); `Cluster__SeedNodes__*` env matrix per pair.
|
||||
**Exit gate:** rig up in the new shape; every existing live-gated behavior re-verified per pair
|
||||
(deploy, redundancy 250/240 per pair — **two Primaries fleet-wide, one per pair, by design**);
|
||||
secrets Akka replication re-verified or re-scoped (it rides DPS on the current single mesh — its
|
||||
topology must be re-decided here, likely SQL-hub mode like ScadaBridge, since pub/sub cannot
|
||||
cross separate meshes).
|
||||
§"Bootstrap: self-first seed ordering"). The rest of the phase — actually splitting the single
|
||||
fleet-wide mesh into three independent 2-node meshes (central `cluster-MAIN`, site-a
|
||||
`cluster-SITE-A`, site-b `cluster-SITE-B`) — **shipped 2026-07-24**. See
|
||||
`docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md` for the full plan and task list.
|
||||
Delivered:
|
||||
- Per-pair `Cluster:SeedNodes` (each node lists itself first, its pair partner second — no
|
||||
cross-pair seeds) + `cluster-{ClusterId}` roles on every driver node + singleton re-scoping:
|
||||
`RedundancyStateActor` is now a `cluster-{ClusterId}`-scoped singleton spawned on every driver
|
||||
node (falls back to plain `driver` for legacy nodes), so **each pair elects its own Primary —
|
||||
two or more Primaries fleet-wide, by design**; `ClusterNodeAddressReconciler` is own-cluster-scoped.
|
||||
- `CentralCommunicationActor` now creates one `ClusterClient` per application `Cluster` and fans
|
||||
commands across them (closes the `TODO(Phase 6)` left by Phase 2).
|
||||
- New `SplitTopologyTransportValidator` fails host start on a `cluster-{ClusterId}`-role node left
|
||||
on a DPS transport (`MeshTransport:Mode` must be `ClusterClient`; `Telemetry:Mode`/`TelemetryDial:Mode`
|
||||
must be `Grpc`) — see `docs/Configuration.md` §"Per-cluster mesh split (Phase 6)".
|
||||
- **Three decisions worth recording:**
|
||||
1. **Secrets replication went pair-local, not SQL-hub.** The design flagged "likely SQL-hub mode
|
||||
like ScadaBridge" as the fallback if DPS couldn't cross the split meshes — in the event, each
|
||||
pair simply keeps replicating its own secrets within itself (no cross-cluster sync needed, and
|
||||
no SQL-hub was built, consistent with Phase 4: site nodes have no SQL to hub through).
|
||||
2. **The compiled transport-mode defaults (`MeshTransport:Mode`, `Telemetry:Mode`,
|
||||
`TelemetryDial:Mode`) were deliberately left at `Dps`**, not flipped to the split-safe modes —
|
||||
a blast-radius assessment found flipping the default breaks every integration test and every
|
||||
appsettings profile that leaves the key unset. The guardrail is
|
||||
`SplitTopologyTransportValidator` plus explicit per-deployment config, not a default flip.
|
||||
3. **The docker-dev rig models three OtOpcUa-only meshes** (central/site-a/site-b), not the
|
||||
ScadaBridge co-location topology sketched in the design's port table — co-location with
|
||||
ScadaBridge's own site nodes stayed out of scope for this phase.
|
||||
**Exit gate:** rig up in the new three-mesh shape; every existing live-gated behavior re-verified
|
||||
per pair (deploy, redundancy 250/240 per pair, two-Primaries-fleet-wide confirmed); the
|
||||
ClusterRedundancy page's mesh-scope caveat and `IManualFailoverService`'s doc-comment caveat
|
||||
removed (the election is pair-local now); `docs/Redundancy.md`'s per-Akka-cluster KNOWN LIMITATION
|
||||
marked resolved.
|
||||
|
||||
### Phase 7 — Failover drill + live gates
|
||||
**Scope:** the drill ScadaBridge already has (`failover-drill.sh` analogue) run per pair, both
|
||||
@@ -318,5 +338,5 @@ resource sizing on the site VMs should be checked once both products run the ful
|
||||
| 3 fetch-and-cache | **DONE 2026-07-23**, merged `d01b0695`, live gate PASSED, pushed to origin (scadaproj umbrella index updated + pushed `b5e7bc8`). gRPC fetch RPC + shared node key; dark switch `ConfigSource:Mode` (Direct default). Rig flip: `OTOPCUA_CONFIG_MODE=FetchAndCache` on the site nodes. See `2026-07-22-mesh-phase3-config-fetch-and-cache.md`. |
|
||||
| 4 cut driver ConfigDb | **DONE 2026-07-23 — live gate PASSED** on `feat/mesh-phase4` (Tasks 0–8 + 1b + 10–11; Task 9 table-drop deferred). ConfigDb admin-only; driver-only ⇒ FetchAndCache (validator); `DbHealthProbeActor` not spawned driver-only (client-visible ServiceLevel 240/250 with central SQL down — proven live); alarm condition state in replicated LocalDb `alarm_condition_state`; central persists acks; `OpcUaPublish` guard split fixes the driver-only address-space wipe; driver-only LDAP maps from appsettings only (6th consumer found mid-phase). Gate: deploy sealed green w/ 4 DB-less site nodes acking, ServiceLevel held 240 w/ SQL down, restart booted last-known-good from the LocalDb pointer. See `2026-07-23-mesh-phase4-cut-driver-configdb.md` + `2026-07-23-mesh-phase4-live-gate.md`. |
|
||||
| 5 gRPC telemetry | **DONE 2026-07-23 — live gate PASSED** on `feat/mesh-phase5`. 4-channel scope (`alerts`/`script-logs`/`driver-health`/`driver-resilience-status`), 3 deferred with rationale (`redundancy-state`, `fleet-status`, `deployment-acks`); dark switch `Telemetry:Mode`/`TelemetryDial:Mode` (Dps default); node hosts server / central dials; auth-from-day-one fail-closed bearer key, superseding design §6.3. Gate: full 12-stream mesh formed in Grpc mode (2 inbound per driver node, 6 outbound per central), AdminUI pill live (data path proven), kill-and-reconnect of a node recovered its stream in ~5s; Dps baseline dials nothing. Surfaced an upgrade gotcha — `ClusterNode.GrpcPort` must be populated on existing deployments (fresh installs seed it; nullable column doesn't backfill) — which live-validated the graceful null-`GrpcPort` skip. See `docs/Telemetry.md`, `2026-07-23-mesh-phase5-grpc-telemetry-stream.md` + `2026-07-23-mesh-phase5-live-gate.md`. |
|
||||
| 6 mesh partition + co-location | not started |
|
||||
| 6 mesh partition + co-location | **DONE 2026-07-24** — three independent 2-node meshes (central/site-a/site-b), `cluster-{ClusterId}` roles, per-pair self-first seeds, cluster-scoped `RedundancyStateActor` singleton (two+ Primaries fleet-wide, by design), one `ClusterClient` per `Cluster`, `SplitTopologyTransportValidator`, pair-local secrets replication, compiled transport defaults intentionally kept `Dps`. See `2026-07-24-mesh-phase6-partition-and-colocation.md`. |
|
||||
| 7 drill + live gates | not started |
|
||||
|
||||
+4
-4
@@ -68,10 +68,10 @@ else
|
||||
</div>
|
||||
|
||||
<p class="text-muted small mt-2 mb-0">
|
||||
Read live from cluster state on this node. <strong>Mesh-wide scope:</strong> the Primary is
|
||||
elected once per Akka mesh, not per cluster row — in the current single-mesh topology this
|
||||
acts on the whole mesh's Primary, which may be a node of another
|
||||
<span class="mono">Cluster</span>. See <span class="mono">docs/Redundancy.md</span>.
|
||||
Read live from cluster state on this node. The Primary shown is
|
||||
<strong>this application <span class="mono">Cluster</span>'s own Primary</strong> — since
|
||||
the per-cluster mesh split, each cluster's redundant pair runs its own independent 2-node
|
||||
Akka mesh and elects its own Primary. See <span class="mono">docs/Redundancy.md</span>.
|
||||
</p>
|
||||
|
||||
<AuthorizeView Policy="@ManualFailoverPageModel.RequiredPolicy">
|
||||
|
||||
@@ -31,10 +31,9 @@ public sealed record ManualFailoverSnapshot(string? PrimaryAddress, IReadOnlyLis
|
||||
/// moves with it and OPC UA clients re-select.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Mesh-scope caveat.</b> Until the per-cluster mesh work lands
|
||||
/// (<c>docs/plans/2026-07-21-per-cluster-mesh-design.md</c>) the Primary is elected once per
|
||||
/// Akka mesh, not per application <c>Cluster</c> row. On a fleet running several application
|
||||
/// clusters in one mesh this acts on the whole mesh's Primary. The UI says so.
|
||||
/// Post-Phase-6 (per-cluster mesh split): the Primary is elected per application
|
||||
/// <c>Cluster</c>, because each cluster's redundant pair now runs its own independent
|
||||
/// 2-node Akka mesh. This service therefore acts on <b>this pair's own</b> Primary.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public interface IManualFailoverService
|
||||
|
||||
Reference in New Issue
Block a user