Compare commits
15 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 2839deb5be | |||
| 8a173bbaf6 | |||
| 2a057a466f | |||
| 741ab97b89 | |||
| 344b0d3334 | |||
| a886d5e6e0 | |||
| d98b32640b | |||
| 1ce2042e14 | |||
| a4d3ab4005 | |||
| 6531ec1984 | |||
| 2535df019b | |||
| a2c5d57fa0 | |||
| 2d2f1decae | |||
| 87ff00fd30 | |||
| 02177fec7c |
@@ -228,7 +228,18 @@ The server supports non-transparent warm/hot redundancy via the `Redundancy` sec
|
|||||||
|
|
||||||
Assert the **effective** cluster config off a running `ActorSystem` (as `SplitBrainResolverActivationTests` does), never the typed options or the akka.conf text — several HOCON fragments compete and the losing one still reads correctly in the file.
|
Assert the **effective** cluster config off a running `ActorSystem` (as `SplitBrainResolverActivationTests` does), never the typed options or the akka.conf text — several HOCON fragments compete and the losing one still reads correctly in the file.
|
||||||
|
|
||||||
**Still per-Akka-cluster, not per application `Cluster`.** One Primary is elected across the whole Akka mesh, so a fleet running several application clusters in one mesh (what docker-dev models) gets one Primary among all driver nodes. The fix is the per-cluster mesh workstream: `docs/plans/2026-07-21-per-cluster-mesh-design.md` (Phases 0a/0b/1 done; 2 code-complete behind a dark switch; 3–7 not started).
|
**RESOLVED by per-cluster mesh Phase 6 (2026-07-24) — no longer per-Akka-cluster.** The "one Primary across the whole Akka mesh" limitation described above is gone; see the Phase 6 paragraph immediately below.
|
||||||
|
|
||||||
|
**Per-cluster mesh Phase 6 (2026-07-24) — the fleet is now three independent 2-node meshes, not one.** The single fleet-wide Akka mesh was split into one mesh per application `Cluster`: the central pair (`admin,driver,cluster-MAIN`), site-a (`driver,cluster-SITE-A`), site-b (`driver,cluster-SITE-B`) — same `ActorSystem` name, separated purely by per-pair self-first seeds (each node's `Cluster:SeedNodes` lists itself first, its own pair partner second, never a node from another cluster's pair). Consequences:
|
||||||
|
|
||||||
|
- **The redundancy singleton is cluster-scoped and spawned per driver node.** `RedundancyStateActor` is now a `cluster-{ClusterId}`-scoped singleton (falls back to plain `driver` for a legacy node), so each pair elects and publishes its **own** Primary on its own mesh's DPS. **Two or more Primaries exist fleet-wide, one per pair — by design, not a bug.**
|
||||||
|
- `ClusterNodeAddressReconciler` is **own-cluster-scoped** — an admin node can only reconcile `ClusterNode` rows in its own cluster; it has no gossip visibility into another mesh.
|
||||||
|
- `CentralCommunicationActor` creates **one `ClusterClient` per application `Cluster`** and fans commands across them (closing the `TODO(Phase 6)` Phase 2 left behind), rather than one fleet-wide client.
|
||||||
|
- **`SplitTopologyTransportValidator` fails host start** on a node carrying a `cluster-{ClusterId}` role unless it also runs the split-safe transports: `MeshTransport:Mode=ClusterClient`, `Telemetry:Mode=Grpc` (driver), `TelemetryDial:Mode=Grpc` (admin). A DPS transport on a cluster-role node would try to gossip across a partition that no longer exists.
|
||||||
|
- **The compiled transport-mode defaults were deliberately NOT flipped** — `MeshTransport:Mode`/`Telemetry:Mode`/`TelemetryDial:Mode` all still default to `Dps`; a blast-radius assessment showed flipping the default breaks every integration test and appsettings profile that leaves the key unset. The validator + explicit per-deployment config is the guardrail, not a default flip.
|
||||||
|
- **Secrets replication is pair-local** — each pair replicates its own Akka DPS secrets within itself; there is no fleet-wide/cross-cluster sync after the split (sites have no SQL, so no SQL-hub, consistent with Phase 4).
|
||||||
|
|
||||||
|
See `docs/Redundancy.md` §"Per-cluster meshes (Phase 6)", `docs/Configuration.md` §"Per-cluster mesh split (Phase 6)", and `docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md`.
|
||||||
|
|
||||||
## Mesh command transport (`MeshTransport`)
|
## Mesh command transport (`MeshTransport`)
|
||||||
|
|
||||||
@@ -246,12 +257,14 @@ change, not a redeploy. Things worth knowing before touching it:
|
|||||||
`DeploymentArtifact.ResolveClusterScope`). `ClusterClient.Send` delivers to exactly **one** registered
|
`DeploymentArtifact.ResolveClusterScope`). `ClusterClient.Send` delivers to exactly **one** registered
|
||||||
actor — it would deploy to a single node while every other node silently kept its old configuration,
|
actor — it would deploy to a single node while every other node silently kept its old configuration,
|
||||||
and the deployment could still seal green.
|
and the deployment could still seal green.
|
||||||
- **Exactly ONE fleet-wide ClusterClient while the fleet is one mesh.** A receptionist serves its whole
|
- **RESOLVED by Phase 6 — one `ClusterClient` per application `Cluster`, not one fleet-wide.** When
|
||||||
cluster, so `SendToAll` reaches every registered node-comm actor regardless of which node's address was
|
this was written the fleet was still one mesh, so a receptionist served the whole cluster and
|
||||||
dialled. One client per application `Cluster` — the obviously-right-looking shape, and what Phase 6
|
`SendToAll` reached every registered node-comm actor regardless of which node's address was dialled;
|
||||||
wants — would fan each command out once per cluster (N× duplicate dispatch *and* N× duplicate ack).
|
the `TODO(Phase 6)` in `CentralCommunicationActor.RebuildClient` asked for one client per application
|
||||||
Marked `TODO(Phase 6)` in `CentralCommunicationActor.RebuildClient`. **Corollary:** the contact set does
|
`Cluster` instead. Phase 6 split the fleet into one mesh per `Cluster` and shipped exactly that — see
|
||||||
not scope delivery; excluding a `MaintenanceMode` node from the contacts does not stop it receiving.
|
the [Redundancy](#redundancy) section's Phase 6 paragraph above. The `SendToAll`-within-a-mesh
|
||||||
|
behaviour below is otherwise unchanged; a `MaintenanceMode` node still isn't excluded from delivery
|
||||||
|
within its own mesh.
|
||||||
- **Inbound commands re-emit on the node's `EventStream`, not on their DPS topic** — DPS is mesh-wide and
|
- **Inbound commands re-emit on the node's `EventStream`, not on their DPS topic** — DPS is mesh-wide and
|
||||||
central `SendToAll`s to every node, so a DPS re-publish would deliver N copies to every subscriber.
|
central `SendToAll`s to every node, so a DPS re-publish would deliver N copies to every subscriber.
|
||||||
Node-side subscribers (`DriverHostActor`, `ScriptedAlarmHostActor`) accept both.
|
Node-side subscribers (`DriverHostActor`, `ScriptedAlarmHostActor`) accept both.
|
||||||
|
|||||||
+166
-96
@@ -1,25 +1,58 @@
|
|||||||
# docker-dev/ — Mac-friendly single-mesh hub-and-spoke fleet for v2 development + manual UI exercise.
|
# docker-dev/ — Mac-friendly THREE-mesh hub-and-spoke fleet for v2 development + manual UI exercise.
|
||||||
#
|
#
|
||||||
# Topology: ONE Akka mesh seeded by `central-1`. Logical separation between
|
# Topology (per-cluster mesh Phase 6, 2026-07-24): THREE independent 2-node Akka
|
||||||
# tenants is by ServerCluster.ClusterId rows (MAIN / SITE-A / SITE-B) in the one
|
# meshes, not one six-node mesh. Every node still runs the same ActorSystem name
|
||||||
# shared `OtOpcUa` ConfigDb — NOT by separate meshes. All six host nodes join the
|
# `otopcua` and all six containers still sit on the ONE shared docker-dev network
|
||||||
# same gossip ring and the central UI deploys to every cluster over it.
|
# — separation is purely by per-pair self-first Cluster__SeedNodes partitioning,
|
||||||
|
# never by network isolation. This is deliberately faithful to production, where
|
||||||
|
# the three meshes are separated by a WAN, not a firewall: nothing here stops
|
||||||
|
# central-1 opening a bare TCP connection to site-a-1's Akka port, it just never
|
||||||
|
# tries, because no seed list and no gossip ever names it.
|
||||||
|
#
|
||||||
|
# MAIN central-1 + central-2 (fused admin+driver, roles admin,driver,cluster-MAIN)
|
||||||
|
# SITE-A site-a-1 + site-a-2 (driver-only, roles driver,cluster-SITE-A)
|
||||||
|
# SITE-B site-b-1 + site-b-2 (driver-only, roles driver,cluster-SITE-B)
|
||||||
|
#
|
||||||
|
# Each pair seeds ONLY itself (self-first, partner-second) — no node lists a node
|
||||||
|
# outside its own mesh as a seed. Logical/tenant separation (ServerCluster.ClusterId
|
||||||
|
# rows MAIN / SITE-A / SITE-B in the one shared `OtOpcUa` ConfigDb) now COINCIDES
|
||||||
|
# with mesh separation: a `cluster-<ClusterId>` Akka role marks which mesh + which
|
||||||
|
# ServerCluster row a node belongs to (RoleParser.ClusterRolePrefix = "cluster-").
|
||||||
|
#
|
||||||
|
# Since central no longer shares a gossip ring with the site meshes, it reaches
|
||||||
|
# them over the three explicit split-topology transports instead of gossip:
|
||||||
|
# - Command plane: MeshTransport__Mode=ClusterClient (central dials each site
|
||||||
|
# mesh's ClusterClientReceptionist; DPS is gossip-only and
|
||||||
|
# would not cross the mesh boundary at all now)
|
||||||
|
# - Live telemetry: Telemetry__Mode=Grpc on every driver node (all six) +
|
||||||
|
# TelemetryDial__Mode=Grpc on the admin pair (central-1/2)
|
||||||
|
# — the site node hosts the gRPC server, central dials in
|
||||||
|
# - Deployed config: ConfigSource__Mode=FetchAndCache on the site nodes (already
|
||||||
|
# true since Phase 3/4 — driver-only nodes have no ConfigDb
|
||||||
|
# connection at all); central stays ConfigSource-less/Direct
|
||||||
|
# and keeps serving via ConfigServe.
|
||||||
|
# SplitTopologyTransportValidator (Cluster project) REFUSES to start any node
|
||||||
|
# carrying a `cluster-*` role that is left on a DPS-shaped transport — these
|
||||||
|
# three flips are therefore mandatory here, not optional dark switches.
|
||||||
#
|
#
|
||||||
# Stack:
|
# Stack:
|
||||||
# sql SQL Server 2022 — hosts the one ConfigDb every node uses
|
# sql SQL Server 2022 — hosts the one ConfigDb every node uses
|
||||||
# cluster-seed one-shot mssql-tools job that INSERTs the ServerCluster +
|
# cluster-seed one-shot mssql-tools job that INSERTs the ServerCluster +
|
||||||
# ClusterNode rows scoping each tenant, then exits (idempotent)
|
# ClusterNode rows scoping each tenant, then exits (idempotent)
|
||||||
#
|
#
|
||||||
# central-1, central-2 OTOPCUA_ROLES=admin,driver — the ONLY UI + deploy
|
# central-1, central-2 OTOPCUA_ROLES=admin,driver,cluster-MAIN — the ONLY UI +
|
||||||
# singleton, plus the MAIN cluster's OPC UA publishers.
|
# deploy singleton, plus the MAIN cluster's OPC UA
|
||||||
# Reachable at http://localhost:9200 (via Traefik).
|
# publishers. Reachable at http://localhost:9200 (via
|
||||||
# Both are Akka seed nodes, each listing ITSELF first
|
# Traefik). Both are Akka seed nodes for the MAIN mesh
|
||||||
# (2026-07-22) so either can cold-start while the other
|
# ONLY, each listing ITSELF first (2026-07-22) so either
|
||||||
# is down; the site nodes seed off central-1 only.
|
# can cold-start while the other is down.
|
||||||
# site-a-1, site-a-2 OTOPCUA_ROLES=driver — driver-only members of the same
|
# site-a-1, site-a-2 OTOPCUA_ROLES=driver,cluster-SITE-A — driver-only
|
||||||
# site-b-1, site-b-2 mesh, scoped to SITE-A / SITE-B by ClusterId. They
|
# site-b-1, site-b-2 OTOPCUA_ROLES=driver,cluster-SITE-B — members of their
|
||||||
# serve no UI and authenticate no users; the central
|
# OWN 2-node mesh (self-first seeded within the pair, NOT
|
||||||
# cluster manages and deploys to them over the mesh.
|
# off central). They serve no UI and authenticate no
|
||||||
|
# users; central manages + deploys to them over the
|
||||||
|
# ClusterClient/gRPC/FetchAndCache transports above, not
|
||||||
|
# gossip.
|
||||||
#
|
#
|
||||||
# Auth is real LDAP against the shared GLAuth on the Linux Docker host
|
# Auth is real LDAP against the shared GLAuth on the Linux Docker host
|
||||||
# (10.100.0.35:3893, dc=zb,dc=local) — there is no LDAP container here.
|
# (10.100.0.35:3893, dc=zb,dc=local) — there is no LDAP container here.
|
||||||
@@ -45,17 +78,12 @@
|
|||||||
# listener, verified free against every other per-node port: Akka 4053, ConfigServe 4055 on
|
# listener, verified free against every other per-node port: Akka 4053, ConfigServe 4055 on
|
||||||
# central, LocalDb sync 9001 on site-a, OPC UA 4840, HTTP 9000) + Telemetry__ApiKey
|
# central, LocalDb sync 9001 on site-a, OPC UA 4840, HTTP 9000) + Telemetry__ApiKey
|
||||||
# (committed dev key, matches the SQL password / secrets KEK / ConfigServe key exception).
|
# (committed dev key, matches the SQL password / secrets KEK / ConfigServe key exception).
|
||||||
# central-1/central-2 also carry the matching TelemetryDial__ApiKey. Both Telemetry__Mode and
|
# central-1/central-2 also carry the matching TelemetryDial__ApiKey. As of Phase 6, Telemetry__Mode
|
||||||
# TelemetryDial__Mode are left UNSET here, so every node defaults to "Dps" (dark — telemetry
|
# is set to "Grpc" on all six nodes and TelemetryDial__Mode is set to "Grpc" on central-1/central-2
|
||||||
# keeps riding the existing DistributedPubSub topic). The ClusterNode.GrpcPort column (seeded
|
# ONLY — SplitTopologyTransportValidator requires it once a node carries a cluster-* role, so this
|
||||||
# by docker-dev/seed/seed-clusters.sql) points central's dial supervisor at each node's :4056.
|
# is no longer an optional live-gate step (see the Phase 5 comment history in git blame for the
|
||||||
# Unlike MeshTransport/ConfigSource, no `${VAR:-Dps}` interpolation is wired for either Mode key
|
# prior dark-switch shape). The ClusterNode.GrpcPort column (seeded by
|
||||||
# (a bare shell export does nothing here — these keys aren't referenced anywhere in this file, so
|
# docker-dev/seed/seed-clusters.sql) points central's dial supervisor at each node's :4056.
|
||||||
# there is nothing for it to substitute into). To run the Phase 5 live gate, manually add
|
|
||||||
# `Telemetry__Mode: "Grpc"` to the `environment:` block of all six driver nodes (central-1,
|
|
||||||
# central-2, site-a-1, site-a-2, site-b-1, site-b-2) and `TelemetryDial__Mode: "Grpc"` to
|
|
||||||
# central-1's and central-2's blocks — or supply both via a docker-compose.override.yml carrying
|
|
||||||
# the same keys — then `docker compose -f docker-dev/docker-compose.yml up -d` to recreate.
|
|
||||||
#
|
#
|
||||||
# Usage:
|
# Usage:
|
||||||
# docker compose -f docker-dev/docker-compose.yml up -d --build
|
# docker compose -f docker-dev/docker-compose.yml up -d --build
|
||||||
@@ -84,6 +112,15 @@ name: otopcua-dev
|
|||||||
# AnnounceInterval is shortened to 5s (default 30s) so anti-entropy convergence is
|
# AnnounceInterval is shortened to 5s (default 30s) so anti-entropy convergence is
|
||||||
# observable quickly during dev exercise. Each node keeps its own local SQLite store
|
# observable quickly during dev exercise. Each node keeps its own local SQLite store
|
||||||
# (Secrets:SqlitePath=otopcua-secrets.db, per-container); replication syncs them.
|
# (Secrets:SqlitePath=otopcua-secrets.db, per-container); replication syncs them.
|
||||||
|
#
|
||||||
|
# Per-cluster mesh Phase 6: this DPS topic now rides each pair's OWN 2-node mesh, not one
|
||||||
|
# shared six-node mesh — so secret replication is PAIR-LOCAL. central-1/central-2 replicate
|
||||||
|
# with each other; site-a-1/site-a-2 replicate with each other; site-b-1/site-b-2 replicate
|
||||||
|
# with each other. There is no cross-cluster secret sync after the split (by design — the site
|
||||||
|
# meshes don't share a gossip ring with central or with each other, consistent with Phase 4's
|
||||||
|
# "driver-only nodes have no SQL, hence no SQL-backed cross-mesh hub" posture). No code change
|
||||||
|
# was needed for this — Enabled=true + the same KEK on every node is still correct; it just now
|
||||||
|
# converges within three independent 2-node groups instead of one six-node group.
|
||||||
x-secrets-env: &secrets-env
|
x-secrets-env: &secrets-env
|
||||||
Secrets__Replication__Enabled: "true"
|
Secrets__Replication__Enabled: "true"
|
||||||
Secrets__Replication__AnnounceInterval: "00:00:05"
|
Secrets__Replication__AnnounceInterval: "00:00:05"
|
||||||
@@ -155,8 +192,9 @@ services:
|
|||||||
|
|
||||||
# ── Central cluster (2-node fused admin+driver) ─────────────────────────────
|
# ── Central cluster (2-node fused admin+driver) ─────────────────────────────
|
||||||
# The only UI + deploy singleton; also the MAIN cluster's OPC UA publishers.
|
# The only UI + deploy singleton; also the MAIN cluster's OPC UA publishers.
|
||||||
# central-1 seeds the single Akka mesh that every other node joins; central-2 is a seed too
|
# Per-cluster mesh Phase 6: central-1 and central-2 form their OWN 2-node "MAIN" Akka mesh,
|
||||||
# (listing itself first), so it can form the mesh alone if central-1 is down.
|
# NOT a shared mesh with the site nodes — each lists itself first as seed, so either can
|
||||||
|
# cold-start while the other is down, and the site meshes are never in this pair's seed list.
|
||||||
|
|
||||||
central-1: &otopcua-host
|
central-1: &otopcua-host
|
||||||
build:
|
build:
|
||||||
@@ -189,7 +227,7 @@ services:
|
|||||||
migrator: { condition: service_completed_successfully }
|
migrator: { condition: service_completed_successfully }
|
||||||
environment:
|
environment:
|
||||||
<<: *secrets-env
|
<<: *secrets-env
|
||||||
OTOPCUA_ROLES: "admin,driver"
|
OTOPCUA_ROLES: "admin,driver,cluster-MAIN"
|
||||||
ASPNETCORE_URLS: "http://+:9000"
|
ASPNETCORE_URLS: "http://+:9000"
|
||||||
ConnectionStrings__ConfigDb: "Server=sql,1433;Database=OtOpcUa;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;"
|
ConnectionStrings__ConfigDb: "Server=sql,1433;Database=OtOpcUa;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;"
|
||||||
Cluster__Hostname: "0.0.0.0"
|
Cluster__Hostname: "0.0.0.0"
|
||||||
@@ -203,18 +241,22 @@ services:
|
|||||||
ConfigServe__GrpcListenPort: "4055"
|
ConfigServe__GrpcListenPort: "4055"
|
||||||
ConfigServe__ApiKey: "configserve-docker-dev-key"
|
ConfigServe__ApiKey: "configserve-docker-dev-key"
|
||||||
ConfigSource__Mode: "Direct"
|
ConfigSource__Mode: "Direct"
|
||||||
# Per-cluster mesh Phase 5 (gRPC live-telemetry stream). DARK SWITCH — Telemetry:Mode is left
|
# Per-cluster mesh Phase 5/6 (gRPC live-telemetry stream). central-1 carries a cluster-*
|
||||||
# unset here (defaults to "Dps"), so the node keeps publishing telemetry on the existing
|
# role + admin, so SplitTopologyTransportValidator requires BOTH Telemetry:Mode=Grpc
|
||||||
# DistributedPubSub topic; the dedicated gRPC listener below binds regardless (the node "always
|
# (it is also a driver node, publishing MAIN's own telemetry) AND TelemetryDial:Mode=Grpc
|
||||||
# hosts" per docs/Telemetry.md) so flipping the mode later is a config change, not a redeploy.
|
# (it is the admin-role dialer for every site node). The dedicated gRPC listener below binds
|
||||||
# Port 4056 checked free against every other port this node binds: Akka 4053, ConfigServe 4055,
|
# regardless of mode (the node "always hosts" per docs/Telemetry.md). Port 4056 checked free
|
||||||
# OPC UA 4840 (container-internal), HTTP 9000. TelemetryDial__ApiKey (below) is central's
|
# against every other port this node binds: Akka 4053, ConfigServe 4055, OPC UA 4840
|
||||||
# dial-side key and must equal every node's Telemetry:ApiKey (fail-closed interceptor).
|
# (container-internal), HTTP 9000. TelemetryDial__ApiKey (below) is central's dial-side key
|
||||||
|
# and must equal every node's Telemetry:ApiKey (fail-closed interceptor).
|
||||||
Telemetry__GrpcListenPort: "4056"
|
Telemetry__GrpcListenPort: "4056"
|
||||||
|
Telemetry__Mode: "Grpc"
|
||||||
Telemetry__ApiKey: "telemetry-docker-dev-key"
|
Telemetry__ApiKey: "telemetry-docker-dev-key"
|
||||||
|
TelemetryDial__Mode: "Grpc"
|
||||||
TelemetryDial__ApiKey: "telemetry-docker-dev-key"
|
TelemetryDial__ApiKey: "telemetry-docker-dev-key"
|
||||||
# Both redundancy peers are seeds (#459 / ScadaBridge parity) so a restarted node can
|
# Both redundancy peers are seeds (#459 / ScadaBridge parity) so a restarted node can
|
||||||
# re-join the mesh via EITHER peer, not only central-1.
|
# re-join the MAIN mesh via EITHER peer — NOT via a site node; the site meshes are never in
|
||||||
|
# this pair's seed list (per-cluster mesh Phase 6).
|
||||||
#
|
#
|
||||||
# ORDER IS LOAD-BEARING (2026-07-22): each node lists ITSELF first. Akka runs
|
# ORDER IS LOAD-BEARING (2026-07-22): each node lists ITSELF first. Akka runs
|
||||||
# FirstSeedNodeProcess — the only path that can form a NEW cluster when no peer answers
|
# FirstSeedNodeProcess — the only path that can form a NEW cluster when no peer answers
|
||||||
@@ -223,19 +265,25 @@ services:
|
|||||||
# boot by AkkaClusterOptionsValidator; see docs/Redundancy.md.
|
# boot by AkkaClusterOptionsValidator; see docs/Redundancy.md.
|
||||||
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
|
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
|
||||||
Cluster__SeedNodes__1: "akka.tcp://otopcua@central-2:4053"
|
Cluster__SeedNodes__1: "akka.tcp://otopcua@central-2:4053"
|
||||||
|
# Per-cluster mesh Phase 6: cluster-MAIN marks this node as a member of the MAIN mesh
|
||||||
|
# (RoleParser.ClusterRolePrefix = "cluster-"), read by SplitTopologyTransportValidator to
|
||||||
|
# require ClusterClient/Grpc transports, and by ClusterRoleInfo to scope redundancy +
|
||||||
|
# ClusterNodeAddressReconcilerActor to ClusterId=MAIN.
|
||||||
Cluster__Roles__0: "admin"
|
Cluster__Roles__0: "admin"
|
||||||
Cluster__Roles__1: "driver"
|
Cluster__Roles__1: "driver"
|
||||||
# Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps",
|
Cluster__Roles__2: "cluster-MAIN"
|
||||||
# the transport this repo has always used, and every node keeps its ClusterClient wiring built
|
# Mesh command transport (per-cluster mesh Phase 2, made mandatory by Phase 6). Central no
|
||||||
# and its comm actor registered with the receptionist either way. Flip the whole rig with
|
# longer shares a gossip ring with the site meshes, so DPS (gossip-only) cannot reach them —
|
||||||
# OTOPCUA_MESH_MODE=ClusterClient at `docker compose up` — no compose edit, and no rebuild.
|
# ClusterClient is the only transport that can. SplitTopologyTransportValidator refuses to
|
||||||
|
# start any node carrying a cluster-* role that is left on Dps, so this is fixed, not an
|
||||||
|
# env-var-overridable default, on every node in this file.
|
||||||
#
|
#
|
||||||
# Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
|
# Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
|
||||||
# time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
|
# time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
|
||||||
# actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
|
# actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
|
||||||
# silence. Both central nodes are listed so a driver node's client can rotate to whichever one
|
# silence. Both central nodes are listed so a driver node's client can rotate to whichever one
|
||||||
# is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
|
# is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
|
||||||
MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Dps}"
|
MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-ClusterClient}"
|
||||||
MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
|
MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
|
||||||
MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
|
MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
|
||||||
Security__Jwt__SigningKey: "docker-dev-signing-key-with-at-least-32-bytes-of-utf8-content-12345"
|
Security__Jwt__SigningKey: "docker-dev-signing-key-with-at-least-32-bytes-of-utf8-content-12345"
|
||||||
@@ -298,7 +346,7 @@ services:
|
|||||||
migrator: { condition: service_completed_successfully }
|
migrator: { condition: service_completed_successfully }
|
||||||
environment:
|
environment:
|
||||||
<<: *secrets-env
|
<<: *secrets-env
|
||||||
OTOPCUA_ROLES: "admin,driver"
|
OTOPCUA_ROLES: "admin,driver,cluster-MAIN"
|
||||||
ASPNETCORE_URLS: "http://+:9000"
|
ASPNETCORE_URLS: "http://+:9000"
|
||||||
ConnectionStrings__ConfigDb: "Server=sql,1433;Database=OtOpcUa;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;"
|
ConnectionStrings__ConfigDb: "Server=sql,1433;Database=OtOpcUa;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;"
|
||||||
Cluster__Hostname: "0.0.0.0"
|
Cluster__Hostname: "0.0.0.0"
|
||||||
@@ -309,29 +357,33 @@ services:
|
|||||||
ConfigServe__GrpcListenPort: "4055"
|
ConfigServe__GrpcListenPort: "4055"
|
||||||
ConfigServe__ApiKey: "configserve-docker-dev-key"
|
ConfigServe__ApiKey: "configserve-docker-dev-key"
|
||||||
ConfigSource__Mode: "Direct"
|
ConfigSource__Mode: "Direct"
|
||||||
# Per-cluster mesh Phase 5 (gRPC live-telemetry stream) — see central-1. DARK SWITCH: Mode
|
# Per-cluster mesh Phase 5/6 (gRPC live-telemetry stream) — see central-1. central-2 carries
|
||||||
# stays unset (⇒ Dps); the dedicated :4056 listener binds regardless.
|
# a cluster-* role + admin, so both Telemetry:Mode=Grpc and TelemetryDial:Mode=Grpc are
|
||||||
|
# required (SplitTopologyTransportValidator); the dedicated :4056 listener binds regardless.
|
||||||
Telemetry__GrpcListenPort: "4056"
|
Telemetry__GrpcListenPort: "4056"
|
||||||
|
Telemetry__Mode: "Grpc"
|
||||||
Telemetry__ApiKey: "telemetry-docker-dev-key"
|
Telemetry__ApiKey: "telemetry-docker-dev-key"
|
||||||
|
TelemetryDial__Mode: "Grpc"
|
||||||
TelemetryDial__ApiKey: "telemetry-docker-dev-key"
|
TelemetryDial__ApiKey: "telemetry-docker-dev-key"
|
||||||
# Both redundancy peers are seeds (#459 / ScadaBridge parity) — see central-1. SELF FIRST:
|
# Both redundancy peers are seeds (#459 / ScadaBridge parity) — see central-1. SELF FIRST:
|
||||||
# central-2 lists itself as seed-nodes[0], which is what lets it cold-start while central-1
|
# central-2 lists itself as seed-nodes[0], which is what lets it cold-start while central-1
|
||||||
# is down (it previously listed central-1 first and simply never came Up in that case).
|
# is down (it previously listed central-1 first and simply never came Up in that case). Per
|
||||||
|
# per-cluster mesh Phase 6, central-2's seed list contains ONLY the MAIN pair — no site node.
|
||||||
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-2:4053"
|
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-2:4053"
|
||||||
Cluster__SeedNodes__1: "akka.tcp://otopcua@central-1:4053"
|
Cluster__SeedNodes__1: "akka.tcp://otopcua@central-1:4053"
|
||||||
|
# Per-cluster mesh Phase 6: cluster-MAIN marks this node as a member of the MAIN mesh — see
|
||||||
|
# central-1's Cluster__Roles comment.
|
||||||
Cluster__Roles__0: "admin"
|
Cluster__Roles__0: "admin"
|
||||||
Cluster__Roles__1: "driver"
|
Cluster__Roles__1: "driver"
|
||||||
# Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps",
|
Cluster__Roles__2: "cluster-MAIN"
|
||||||
# the transport this repo has always used, and every node keeps its ClusterClient wiring built
|
# Mesh command transport (per-cluster mesh Phase 2, made mandatory by Phase 6) — see central-1.
|
||||||
# and its comm actor registered with the receptionist either way. Flip the whole rig with
|
|
||||||
# OTOPCUA_MESH_MODE=ClusterClient at `docker compose up` — no compose edit, and no rebuild.
|
|
||||||
#
|
#
|
||||||
# Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
|
# Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
|
||||||
# time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
|
# time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
|
||||||
# actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
|
# actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
|
||||||
# silence. Both central nodes are listed so a driver node's client can rotate to whichever one
|
# silence. Both central nodes are listed so a driver node's client can rotate to whichever one
|
||||||
# is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
|
# is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
|
||||||
MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Dps}"
|
MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-ClusterClient}"
|
||||||
MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
|
MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
|
||||||
MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
|
MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
|
||||||
Security__Jwt__SigningKey: "docker-dev-signing-key-with-at-least-32-bytes-of-utf8-content-12345"
|
Security__Jwt__SigningKey: "docker-dev-signing-key-with-at-least-32-bytes-of-utf8-content-12345"
|
||||||
@@ -379,9 +431,10 @@ services:
|
|||||||
- otopcua-localdb-central-2:/app/data
|
- otopcua-localdb-central-2:/app/data
|
||||||
|
|
||||||
# ── Site A cluster (2-node driver-only) ─────────────────────────────────────
|
# ── Site A cluster (2-node driver-only) ─────────────────────────────────────
|
||||||
# Driver-only members of the single mesh, scoped to SITE-A by ClusterId. No UI,
|
# Per-cluster mesh Phase 6: site-a-1 and site-a-2 form their OWN 2-node "SITE-A" Akka mesh,
|
||||||
# no user auth — managed + deployed to by the central cluster over the mesh.
|
# separate from MAIN and SITE-B — self-first seeded WITHIN the pair, never off central. No UI,
|
||||||
# All site nodes seed central-1.
|
# no user auth; central manages + deploys to this mesh over ClusterClient/gRPC/FetchAndCache,
|
||||||
|
# not gossip (central and site-a share no seed list, so they never form one ring).
|
||||||
|
|
||||||
site-a-1:
|
site-a-1:
|
||||||
<<: *otopcua-host
|
<<: *otopcua-host
|
||||||
@@ -390,7 +443,7 @@ services:
|
|||||||
central-1: { condition: service_started }
|
central-1: { condition: service_started }
|
||||||
migrator: { condition: service_completed_successfully }
|
migrator: { condition: service_completed_successfully }
|
||||||
environment:
|
environment:
|
||||||
OTOPCUA_ROLES: "driver"
|
OTOPCUA_ROLES: "driver,cluster-SITE-A"
|
||||||
# Per-cluster mesh Phase 4: driver-only site nodes hold NO ConfigDb connection string. Their
|
# Per-cluster mesh Phase 4: driver-only site nodes hold NO ConfigDb connection string. Their
|
||||||
# config comes from central via ConfigSource:Mode=FetchAndCache (gRPC fetch → LocalDb cache), their
|
# config comes from central via ConfigSource:Mode=FetchAndCache (gRPC fetch → LocalDb cache), their
|
||||||
# scripted-alarm condition state lives in LocalDb, and central persists their deploy acks. Direct
|
# scripted-alarm condition state lives in LocalDb, and central persists their deploy acks. Direct
|
||||||
@@ -406,29 +459,32 @@ services:
|
|||||||
ConfigSource__CentralFetchEndpoints__0: "http://central-1:4055"
|
ConfigSource__CentralFetchEndpoints__0: "http://central-1:4055"
|
||||||
ConfigSource__CentralFetchEndpoints__1: "http://central-2:4055"
|
ConfigSource__CentralFetchEndpoints__1: "http://central-2:4055"
|
||||||
ConfigSource__ApiKey: "configserve-docker-dev-key"
|
ConfigSource__ApiKey: "configserve-docker-dev-key"
|
||||||
# Per-cluster mesh Phase 5 (gRPC live-telemetry stream) — see central-1/central-2. DARK
|
# Per-cluster mesh Phase 5/6 (gRPC live-telemetry stream) — see central-1/central-2. This
|
||||||
# SWITCH: Telemetry:Mode stays unset (⇒ Dps); the dedicated :4056 listener binds regardless
|
# node carries cluster-SITE-A + driver, so SplitTopologyTransportValidator requires
|
||||||
# (the node "always hosts" per docs/Telemetry.md). Port checked free against this node's
|
# Telemetry:Mode=Grpc (mandatory, not a dark switch, as of Phase 6). The dedicated :4056
|
||||||
# other ports: Akka 4053, OPC UA 4840, HTTP 9000, LocalDb sync 9001 (below). ApiKey must
|
# listener binds regardless (the node "always hosts" per docs/Telemetry.md). Port checked
|
||||||
# equal central's TelemetryDial:ApiKey (fail-closed interceptor).
|
# free against this node's other ports: Akka 4053, OPC UA 4840, HTTP 9000, LocalDb sync 9001
|
||||||
|
# (below). ApiKey must equal central's TelemetryDial:ApiKey (fail-closed interceptor).
|
||||||
Telemetry__GrpcListenPort: "4056"
|
Telemetry__GrpcListenPort: "4056"
|
||||||
|
Telemetry__Mode: "Grpc"
|
||||||
Telemetry__ApiKey: "telemetry-docker-dev-key"
|
Telemetry__ApiKey: "telemetry-docker-dev-key"
|
||||||
# Site nodes are deliberately NOT seeds — they join the central pair's mesh. The self-first
|
# Per-cluster mesh Phase 6: site-a-1 seeds its OWN "SITE-A" mesh (itself first, site-a-2
|
||||||
# seed rule is conditional for exactly this reason (it binds only when a node's own address
|
# second) — it no longer seeds off central-1. Central is reached over ClusterClient/gRPC/
|
||||||
# is in its own seed list), so these configs are exempt rather than broken.
|
# FetchAndCache, never gossip; central and site-a share no seed list.
|
||||||
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
|
Cluster__SeedNodes__0: "akka.tcp://otopcua@site-a-1:4053"
|
||||||
|
Cluster__SeedNodes__1: "akka.tcp://otopcua@site-a-2:4053"
|
||||||
Cluster__Roles__0: "driver"
|
Cluster__Roles__0: "driver"
|
||||||
# Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps",
|
Cluster__Roles__1: "cluster-SITE-A"
|
||||||
# the transport this repo has always used, and every node keeps its ClusterClient wiring built
|
# Mesh command transport (per-cluster mesh Phase 2, made mandatory by Phase 6) — see
|
||||||
# and its comm actor registered with the receptionist either way. Flip the whole rig with
|
# central-1/central-2. Central dials this mesh's ClusterClientReceptionist via the contact
|
||||||
# OTOPCUA_MESH_MODE=ClusterClient at `docker compose up` — no compose edit, and no rebuild.
|
# points below; there is no gossip path anymore.
|
||||||
#
|
#
|
||||||
# Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
|
# Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
|
||||||
# time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
|
# time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
|
||||||
# actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
|
# actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
|
||||||
# silence. Both central nodes are listed so a driver node's client can rotate to whichever one
|
# silence. Both central nodes are listed so a driver node's client can rotate to whichever one
|
||||||
# is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
|
# is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
|
||||||
MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Dps}"
|
MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-ClusterClient}"
|
||||||
MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
|
MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
|
||||||
MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
|
MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
|
||||||
<<: *secrets-env
|
<<: *secrets-env
|
||||||
@@ -490,7 +546,7 @@ services:
|
|||||||
central-1: { condition: service_started }
|
central-1: { condition: service_started }
|
||||||
migrator: { condition: service_completed_successfully }
|
migrator: { condition: service_completed_successfully }
|
||||||
environment:
|
environment:
|
||||||
OTOPCUA_ROLES: "driver"
|
OTOPCUA_ROLES: "driver,cluster-SITE-A"
|
||||||
# Per-cluster mesh Phase 4: no ConfigDb string on driver-only site nodes (see site-a-1).
|
# Per-cluster mesh Phase 4: no ConfigDb string on driver-only site nodes (see site-a-1).
|
||||||
Cluster__Hostname: "0.0.0.0"
|
Cluster__Hostname: "0.0.0.0"
|
||||||
Cluster__Port: "4053"
|
Cluster__Port: "4053"
|
||||||
@@ -501,23 +557,27 @@ services:
|
|||||||
ConfigSource__CentralFetchEndpoints__0: "http://central-1:4055"
|
ConfigSource__CentralFetchEndpoints__0: "http://central-1:4055"
|
||||||
ConfigSource__CentralFetchEndpoints__1: "http://central-2:4055"
|
ConfigSource__CentralFetchEndpoints__1: "http://central-2:4055"
|
||||||
ConfigSource__ApiKey: "configserve-docker-dev-key"
|
ConfigSource__ApiKey: "configserve-docker-dev-key"
|
||||||
# Per-cluster mesh Phase 5 (gRPC live-telemetry stream) — see central-1/site-a-1. DARK
|
# Per-cluster mesh Phase 5/6 (gRPC live-telemetry stream) — see central-1/site-a-1.
|
||||||
# SWITCH: Telemetry:Mode stays unset (⇒ Dps); the dedicated :4056 listener binds regardless.
|
# Telemetry:Mode=Grpc is mandatory as of Phase 6 (cluster-SITE-A + driver); the dedicated
|
||||||
|
# :4056 listener binds regardless.
|
||||||
Telemetry__GrpcListenPort: "4056"
|
Telemetry__GrpcListenPort: "4056"
|
||||||
|
Telemetry__Mode: "Grpc"
|
||||||
Telemetry__ApiKey: "telemetry-docker-dev-key"
|
Telemetry__ApiKey: "telemetry-docker-dev-key"
|
||||||
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
|
# Per-cluster mesh Phase 6: site-a-2 seeds its OWN "SITE-A" mesh (itself first, site-a-1
|
||||||
|
# second) — it no longer seeds off central-1.
|
||||||
|
Cluster__SeedNodes__0: "akka.tcp://otopcua@site-a-2:4053"
|
||||||
|
Cluster__SeedNodes__1: "akka.tcp://otopcua@site-a-1:4053"
|
||||||
Cluster__Roles__0: "driver"
|
Cluster__Roles__0: "driver"
|
||||||
# Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps",
|
Cluster__Roles__1: "cluster-SITE-A"
|
||||||
# the transport this repo has always used, and every node keeps its ClusterClient wiring built
|
# Mesh command transport (per-cluster mesh Phase 2, made mandatory by Phase 6) — see
|
||||||
# and its comm actor registered with the receptionist either way. Flip the whole rig with
|
# central-1/site-a-1.
|
||||||
# OTOPCUA_MESH_MODE=ClusterClient at `docker compose up` — no compose edit, and no rebuild.
|
|
||||||
#
|
#
|
||||||
# Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
|
# Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
|
||||||
# time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
|
# time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
|
||||||
# actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
|
# actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
|
||||||
# silence. Both central nodes are listed so a driver node's client can rotate to whichever one
|
# silence. Both central nodes are listed so a driver node's client can rotate to whichever one
|
||||||
# is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
|
# is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
|
||||||
MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Dps}"
|
MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-ClusterClient}"
|
||||||
MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
|
MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
|
||||||
MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
|
MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
|
||||||
<<: *secrets-env
|
<<: *secrets-env
|
||||||
@@ -563,6 +623,8 @@ services:
|
|||||||
- otopcua-localdb-site-a-2:/app/data
|
- otopcua-localdb-site-a-2:/app/data
|
||||||
|
|
||||||
# ── Site B cluster (2-node driver-only) ─────────────────────────────────────
|
# ── Site B cluster (2-node driver-only) ─────────────────────────────────────
|
||||||
|
# Per-cluster mesh Phase 6: site-b-1 and site-b-2 form their OWN 2-node "SITE-B" Akka mesh,
|
||||||
|
# separate from MAIN and SITE-A — same shape as site-a-1/site-a-2, see that pair's header.
|
||||||
|
|
||||||
site-b-1:
|
site-b-1:
|
||||||
<<: *otopcua-host
|
<<: *otopcua-host
|
||||||
@@ -571,7 +633,7 @@ services:
|
|||||||
central-1: { condition: service_started }
|
central-1: { condition: service_started }
|
||||||
migrator: { condition: service_completed_successfully }
|
migrator: { condition: service_completed_successfully }
|
||||||
environment:
|
environment:
|
||||||
OTOPCUA_ROLES: "driver"
|
OTOPCUA_ROLES: "driver,cluster-SITE-B"
|
||||||
# Per-cluster mesh Phase 4: no ConfigDb string on driver-only site nodes (see site-a-1).
|
# Per-cluster mesh Phase 4: no ConfigDb string on driver-only site nodes (see site-a-1).
|
||||||
Cluster__Hostname: "0.0.0.0"
|
Cluster__Hostname: "0.0.0.0"
|
||||||
Cluster__Port: "4053"
|
Cluster__Port: "4053"
|
||||||
@@ -582,23 +644,27 @@ services:
|
|||||||
ConfigSource__CentralFetchEndpoints__0: "http://central-1:4055"
|
ConfigSource__CentralFetchEndpoints__0: "http://central-1:4055"
|
||||||
ConfigSource__CentralFetchEndpoints__1: "http://central-2:4055"
|
ConfigSource__CentralFetchEndpoints__1: "http://central-2:4055"
|
||||||
ConfigSource__ApiKey: "configserve-docker-dev-key"
|
ConfigSource__ApiKey: "configserve-docker-dev-key"
|
||||||
# Per-cluster mesh Phase 5 (gRPC live-telemetry stream) — see central-1/site-a-1. DARK
|
# Per-cluster mesh Phase 5/6 (gRPC live-telemetry stream) — see central-1/site-a-1.
|
||||||
# SWITCH: Telemetry:Mode stays unset (⇒ Dps); the dedicated :4056 listener binds regardless.
|
# Telemetry:Mode=Grpc is mandatory as of Phase 6 (cluster-SITE-B + driver); the dedicated
|
||||||
|
# :4056 listener binds regardless.
|
||||||
Telemetry__GrpcListenPort: "4056"
|
Telemetry__GrpcListenPort: "4056"
|
||||||
|
Telemetry__Mode: "Grpc"
|
||||||
Telemetry__ApiKey: "telemetry-docker-dev-key"
|
Telemetry__ApiKey: "telemetry-docker-dev-key"
|
||||||
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
|
# Per-cluster mesh Phase 6: site-b-1 seeds its OWN "SITE-B" mesh (itself first, site-b-2
|
||||||
|
# second) — it no longer seeds off central-1.
|
||||||
|
Cluster__SeedNodes__0: "akka.tcp://otopcua@site-b-1:4053"
|
||||||
|
Cluster__SeedNodes__1: "akka.tcp://otopcua@site-b-2:4053"
|
||||||
Cluster__Roles__0: "driver"
|
Cluster__Roles__0: "driver"
|
||||||
# Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps",
|
Cluster__Roles__1: "cluster-SITE-B"
|
||||||
# the transport this repo has always used, and every node keeps its ClusterClient wiring built
|
# Mesh command transport (per-cluster mesh Phase 2, made mandatory by Phase 6) — see
|
||||||
# and its comm actor registered with the receptionist either way. Flip the whole rig with
|
# central-1/site-a-1.
|
||||||
# OTOPCUA_MESH_MODE=ClusterClient at `docker compose up` — no compose edit, and no rebuild.
|
|
||||||
#
|
#
|
||||||
# Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
|
# Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
|
||||||
# time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
|
# time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
|
||||||
# actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
|
# actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
|
||||||
# silence. Both central nodes are listed so a driver node's client can rotate to whichever one
|
# silence. Both central nodes are listed so a driver node's client can rotate to whichever one
|
||||||
# is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
|
# is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
|
||||||
MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Dps}"
|
MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-ClusterClient}"
|
||||||
MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
|
MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
|
||||||
MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
|
MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
|
||||||
<<: *secrets-env
|
<<: *secrets-env
|
||||||
@@ -635,7 +701,7 @@ services:
|
|||||||
central-1: { condition: service_started }
|
central-1: { condition: service_started }
|
||||||
migrator: { condition: service_completed_successfully }
|
migrator: { condition: service_completed_successfully }
|
||||||
environment:
|
environment:
|
||||||
OTOPCUA_ROLES: "driver"
|
OTOPCUA_ROLES: "driver,cluster-SITE-B"
|
||||||
# Per-cluster mesh Phase 4: no ConfigDb string on driver-only site nodes (see site-a-1).
|
# Per-cluster mesh Phase 4: no ConfigDb string on driver-only site nodes (see site-a-1).
|
||||||
Cluster__Hostname: "0.0.0.0"
|
Cluster__Hostname: "0.0.0.0"
|
||||||
Cluster__Port: "4053"
|
Cluster__Port: "4053"
|
||||||
@@ -646,23 +712,27 @@ services:
|
|||||||
ConfigSource__CentralFetchEndpoints__0: "http://central-1:4055"
|
ConfigSource__CentralFetchEndpoints__0: "http://central-1:4055"
|
||||||
ConfigSource__CentralFetchEndpoints__1: "http://central-2:4055"
|
ConfigSource__CentralFetchEndpoints__1: "http://central-2:4055"
|
||||||
ConfigSource__ApiKey: "configserve-docker-dev-key"
|
ConfigSource__ApiKey: "configserve-docker-dev-key"
|
||||||
# Per-cluster mesh Phase 5 (gRPC live-telemetry stream) — see central-1/site-a-1. DARK
|
# Per-cluster mesh Phase 5/6 (gRPC live-telemetry stream) — see central-1/site-a-1.
|
||||||
# SWITCH: Telemetry:Mode stays unset (⇒ Dps); the dedicated :4056 listener binds regardless.
|
# Telemetry:Mode=Grpc is mandatory as of Phase 6 (cluster-SITE-B + driver); the dedicated
|
||||||
|
# :4056 listener binds regardless.
|
||||||
Telemetry__GrpcListenPort: "4056"
|
Telemetry__GrpcListenPort: "4056"
|
||||||
|
Telemetry__Mode: "Grpc"
|
||||||
Telemetry__ApiKey: "telemetry-docker-dev-key"
|
Telemetry__ApiKey: "telemetry-docker-dev-key"
|
||||||
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
|
# Per-cluster mesh Phase 6: site-b-2 seeds its OWN "SITE-B" mesh (itself first, site-b-1
|
||||||
|
# second) — it no longer seeds off central-1.
|
||||||
|
Cluster__SeedNodes__0: "akka.tcp://otopcua@site-b-2:4053"
|
||||||
|
Cluster__SeedNodes__1: "akka.tcp://otopcua@site-b-1:4053"
|
||||||
Cluster__Roles__0: "driver"
|
Cluster__Roles__0: "driver"
|
||||||
# Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps",
|
Cluster__Roles__1: "cluster-SITE-B"
|
||||||
# the transport this repo has always used, and every node keeps its ClusterClient wiring built
|
# Mesh command transport (per-cluster mesh Phase 2, made mandatory by Phase 6) — see
|
||||||
# and its comm actor registered with the receptionist either way. Flip the whole rig with
|
# central-1/site-a-1.
|
||||||
# OTOPCUA_MESH_MODE=ClusterClient at `docker compose up` — no compose edit, and no rebuild.
|
|
||||||
#
|
#
|
||||||
# Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
|
# Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
|
||||||
# time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
|
# time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
|
||||||
# actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
|
# actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
|
||||||
# silence. Both central nodes are listed so a driver node's client can rotate to whichever one
|
# silence. Both central nodes are listed so a driver node's client can rotate to whichever one
|
||||||
# is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
|
# is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
|
||||||
MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Dps}"
|
MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-ClusterClient}"
|
||||||
MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
|
MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
|
||||||
MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
|
MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
|
||||||
<<: *secrets-env
|
<<: *secrets-env
|
||||||
|
|||||||
+29
-2
@@ -103,13 +103,40 @@ The checked-in `appsettings*.json` files are deliberately thin: they carry only
|
|||||||
| `Hostname` | string | `0.0.0.0` | Bind hostname. |
|
| `Hostname` | string | `0.0.0.0` | Bind hostname. |
|
||||||
| `Port` | int | `4053` | Cluster transport port. **Duplicated as `ClusterNode.AkkaPort` in the Config DB** — see the note below. |
|
| `Port` | int | `4053` | Cluster transport port. **Duplicated as `ClusterNode.AkkaPort` in the Config DB** — see the note below. |
|
||||||
| `PublicHostname` | string | `127.0.0.1` | Hostname advertised in cluster gossip; must be reachable by peers. **Duplicated as `ClusterNode.Host`** — see the note below. |
|
| `PublicHostname` | string | `127.0.0.1` | Hostname advertised in cluster gossip; must be reachable by peers. **Duplicated as `ClusterNode.Host`** — see the note below. |
|
||||||
| `SeedNodes` | string[] | `[]` | Seed nodes for bootstrapping. **Ordered:** a node that appears in its own list must be entry 0 (only `seed-nodes[0]` can form a new cluster), or the host refuses to start — see [Redundancy.md § Bootstrap: self-first seed ordering](Redundancy.md#bootstrap-self-first-seed-ordering). |
|
| `SeedNodes` | string[] | `[]` | Seed nodes for bootstrapping. **Ordered:** a node that appears in its own list must be entry 0 (only `seed-nodes[0]` can form a new cluster), or the host refuses to start — see [Redundancy.md § Bootstrap: self-first seed ordering](Redundancy.md#bootstrap-self-first-seed-ordering). **Per-pair since Phase 6:** each application `Cluster`'s pair is its own Akka mesh, so a node's `SeedNodes` lists only itself (first) and its own pair partner (second) — never a node from another cluster's pair. |
|
||||||
| `Roles` | string[] | `[]` | Cluster roles for this node. When empty, falls back to `OTOPCUA_ROLES`. Allowed values: `admin`, `driver`, `dev`. |
|
| `Roles` | string[] | `[]` | Cluster roles for this node. When empty, falls back to `OTOPCUA_ROLES`. Allowed values: `admin`, `driver`, `dev`, plus a per-cluster `cluster-{ClusterId}` role (e.g. `cluster-MAIN`, `cluster-SITE-A`) — see [Per-cluster mesh split (Phase 6)](#per-cluster-mesh-split-phase-6-cluster-clusterid-roles--splittopologytransportvalidator) below. |
|
||||||
|
|
||||||
> The full redundancy model (ServiceLevel tiers, split-brain, peer discovery) is in [`Redundancy.md`](Redundancy.md). The OPC UA peer-URI advertising lives in the `OpcUa:PeerApplicationUris` key above.
|
> The full redundancy model (ServiceLevel tiers, split-brain, peer discovery) is in [`Redundancy.md`](Redundancy.md). The OPC UA peer-URI advertising lives in the `OpcUa:PeerApplicationUris` key above.
|
||||||
|
|
||||||
> **`Port` / `PublicHostname` are stored twice.** The node binds from these keys; the fleet's `ClusterNode` row records the same address as `AkkaPort` / `Host` so that **central can dial the node without sharing a gossip ring with it** — which is what [per-cluster mesh Phase 2](plans/2026-07-21-per-cluster-mesh-design.md) needs. Nothing in the schema makes the two agree, so `ClusterNodeAddressReconcilerActor` (admin-role singleton) compares them against live membership and logs an Error on mismatch. If you change `Cluster:Port` or `Cluster:PublicHostname` on a node, **update its `ClusterNode` row too** — a node binding 4054 with a row saying 4053 still gossips fine today, and fails silently in Phase 2. The row's `NodeId` is `host:port` and is also the deploy path's ack identity, so a drifted node's deployment acks stop matching as well. See [`config-db-schema.md` § `ClusterNode`](v2/config-db-schema.md#clusternode).
|
> **`Port` / `PublicHostname` are stored twice.** The node binds from these keys; the fleet's `ClusterNode` row records the same address as `AkkaPort` / `Host` so that **central can dial the node without sharing a gossip ring with it** — which is what [per-cluster mesh Phase 2](plans/2026-07-21-per-cluster-mesh-design.md) needs. Nothing in the schema makes the two agree, so `ClusterNodeAddressReconcilerActor` (admin-role singleton) compares them against live membership and logs an Error on mismatch. If you change `Cluster:Port` or `Cluster:PublicHostname` on a node, **update its `ClusterNode` row too** — a node binding 4054 with a row saying 4053 still gossips fine today, and fails silently in Phase 2. The row's `NodeId` is `host:port` and is also the deploy path's ack identity, so a drifted node's deployment acks stop matching as well. See [`config-db-schema.md` § `ClusterNode`](v2/config-db-schema.md#clusternode).
|
||||||
|
|
||||||
|
### Per-cluster mesh split (Phase 6): `cluster-{ClusterId}` roles + `SplitTopologyTransportValidator`
|
||||||
|
|
||||||
|
- **Purpose:** [per-cluster mesh](plans/2026-07-22-per-cluster-mesh-program.md) **Phase 6** split the
|
||||||
|
single fleet-wide Akka mesh into one independent 2-node mesh per application `Cluster`. A node's
|
||||||
|
`Cluster:Roles` now carries a `cluster-{ClusterId}` role alongside `driver` (and `admin` for a fused
|
||||||
|
central node) — e.g. `driver,cluster-SITE-A` — and its `Cluster:SeedNodes` lists only itself and its
|
||||||
|
own pair partner (see the `Roles`/`SeedNodes` rows above). Nodes in different clusters never gossip
|
||||||
|
with each other.
|
||||||
|
- **`SplitTopologyTransportValidator`** (`src/Core/ZB.MOM.WW.OtOpcUa.Cluster/`, wired at
|
||||||
|
`ValidateOnStart` alongside the other `Cluster`-family validators) fails host start on any node
|
||||||
|
carrying a `cluster-{ClusterId}` role unless it is also configured for the split-safe transports:
|
||||||
|
`MeshTransport:Mode = ClusterClient`, `Telemetry:Mode = Grpc` on a driver node, `TelemetryDial:Mode =
|
||||||
|
Grpc` on an admin node. A DPS-mode transport on a cluster-role node would try to gossip across a
|
||||||
|
partition boundary that no longer exists, so the validator fails closed rather than let commands or
|
||||||
|
telemetry silently vanish. A legacy node with no `cluster-{ClusterId}` role is exempt and may stay on
|
||||||
|
`Dps`.
|
||||||
|
- **The compiled transport-mode defaults were deliberately NOT flipped.** `MeshTransport:Mode`,
|
||||||
|
`Telemetry:Mode`, and `TelemetryDial:Mode` (documented below) all still default to `Dps` — a
|
||||||
|
blast-radius assessment found that flipping the compiled default would break every integration test
|
||||||
|
and every existing `appsettings.json` profile that leaves these keys unset. `SplitTopologyTransportValidator`
|
||||||
|
plus explicit per-deployment configuration is the guardrail, not a default flip; every node that
|
||||||
|
actually carries a `cluster-{ClusterId}` role must set the split-safe modes explicitly.
|
||||||
|
- See [`Redundancy.md` § Per-cluster meshes (Phase 6)](Redundancy.md#per-cluster-meshes-phase-6) for the
|
||||||
|
full runtime picture (one Primary per pair, the cluster-scoped `RedundancyStateActor` singleton,
|
||||||
|
`ClusterNodeAddressReconciler` own-cluster-scoping, and central's one-`ClusterClient`-per-`Cluster`
|
||||||
|
fan-out).
|
||||||
|
|
||||||
### `MeshTransport` (central↔node command transport)
|
### `MeshTransport` (central↔node command transport)
|
||||||
|
|
||||||
- **Purpose:** selects how central's three command channels (`deployments`, `driver-control`, `alarm-commands`) and the node's `deployment-acks` reply reach the other side — over the mesh-wide DistributedPubSub, or over an Akka `ClusterClient` that does **not** require central and the node to share a gossip ring. The second option is what [per-cluster mesh](plans/2026-07-21-per-cluster-mesh-design.md) Phase 6 needs once the meshes split.
|
- **Purpose:** selects how central's three command channels (`deployments`, `driver-control`, `alarm-commands`) and the node's `deployment-acks` reply reach the other side — over the mesh-wide DistributedPubSub, or over an Akka `ClusterClient` that does **not** require central and the node to share a gossip ring. The second option is what [per-cluster mesh](plans/2026-07-21-per-cluster-mesh-design.md) Phase 6 needs once the meshes split.
|
||||||
|
|||||||
+61
-25
@@ -108,22 +108,59 @@ plane (writes, alarm acks, the alerts emit, the alarm-history drain) on the wron
|
|||||||
*higher* address so the two orderings genuinely disagree. `NodeRedundancyState.IsRoleLeaderForDriver`
|
*higher* address so the two orderings genuinely disagree. `NodeRedundancyState.IsRoleLeaderForDriver`
|
||||||
was renamed `IsDriverPrimary` to stop the name asserting the old derivation.
|
was renamed `IsDriverPrimary` to stop the name asserting the old derivation.
|
||||||
|
|
||||||
> **KNOWN LIMITATION — the election is per *Akka* cluster, not per application `Cluster`.**
|
> **RESOLVED by Phase 6 — the election is now per application `Cluster`, not per Akka mesh.**
|
||||||
> The election yields exactly **one** Primary across the whole Akka cluster. A fleet that
|
> Phase 6 of the per-cluster mesh program split the single fleet-wide Akka mesh into one independent
|
||||||
> runs several application clusters (each with its own redundant pair) inside one Akka cluster —
|
> 2-node mesh per application `Cluster` (see [Per-cluster meshes (Phase 6)](#per-cluster-meshes-phase-6)
|
||||||
> which is what `docker-dev/docker-compose.yml` models, with MAIN, SITE-A and SITE-B — therefore has
|
> below). `RedundancyStateActor` is now a `cluster-{ClusterId}`-scoped singleton spawned on every driver
|
||||||
> one Primary among *all* its driver nodes, not one per pair. Every consumer of the role inherits
|
> node, so each pair elects its own Primary — the whole-Akka-cluster mismatch this block used to describe
|
||||||
> this: `ServiceLevel`, the `alerts` emit gate, and the inbound device-write gate below.
|
> no longer exists. See `docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md`.
|
||||||
>
|
>
|
||||||
> The unit of redundancy everywhere else in the product is the application `Cluster` (`ClusterId`) —
|
> Originally surfaced by the LocalDb Phase 2 live gate, where the old whole-mesh election stopped the
|
||||||
> it owns drivers, devices and `ClusterNode` rows, and the LocalDb deployment-artifact cache is
|
> alarm-history drain on every node in the fleet — see `docs/plans/2026-07-20-localdb-phase2-live-gate.md`.
|
||||||
> already keyed by it *so that a pair shares one entry*. Scoping the election the same way is the
|
|
||||||
> fix; `RedundancyStateActor` is an admin-role singleton in the same assembly as
|
## Per-cluster meshes (Phase 6)
|
||||||
> `AdminOperationsActor`, which already reads and groups `ClusterNodes` by cluster.
|
|
||||||
>
|
The fleet no longer runs one Akka mesh. Phase 6 split it into **three independent 2-node meshes** — the
|
||||||
> Surfaced by the LocalDb Phase 2 live gate, where it stopped the alarm-history drain on every node
|
central pair (roles `admin,driver,cluster-MAIN`), the site-a pair (`driver,cluster-SITE-A`), and the
|
||||||
> in the fleet — see `docs/plans/2026-07-20-localdb-phase2-live-gate.md`. The alarm drain no longer
|
site-b pair (`driver,cluster-SITE-B`) — all sharing the same `ActorSystem` name (`otopcua`) on one network,
|
||||||
> depends on this (it defers only to a node that shares its queue), but the other three gates still do.
|
separated purely by **per-pair self-first seed partitioning**: each node's `Cluster:SeedNodes` lists only
|
||||||
|
itself and its own pair partner, so gossip never crosses a pair boundary (see
|
||||||
|
[Bootstrap: self-first seed ordering](#bootstrap-self-first-seed-ordering)).
|
||||||
|
|
||||||
|
Consequences:
|
||||||
|
|
||||||
|
- **One Primary per pair — two or more Primaries fleet-wide, by design.** Each mesh runs its own
|
||||||
|
election, so a three-mesh fleet has three Primaries, one per application `Cluster`. This is correct,
|
||||||
|
not a regression of the old single-Primary assumption.
|
||||||
|
- **`RedundancyStateActor` is a `cluster-{ClusterId}`-scoped singleton**, spawned on every driver node
|
||||||
|
(falling back to the plain `driver` role for a legacy node not carrying a cluster role). Each pair
|
||||||
|
elects and publishes its own `redundancy-state` on its own mesh's DPS — the DPS topic name is
|
||||||
|
unchanged, but it no longer reaches past the pair.
|
||||||
|
- **`ClusterNodeAddressReconciler` is own-cluster-scoped** — an admin node can only reconcile
|
||||||
|
`ClusterNode` rows belonging to its own cluster; it has no gossip visibility into another mesh to
|
||||||
|
reconcile against.
|
||||||
|
- **`CentralCommunicationActor` creates one `ClusterClient` per application `Cluster`** and fans
|
||||||
|
commands across them, rather than one fleet-wide client. Central reaches a site mesh over
|
||||||
|
`ClusterClient` (commands), gRPC (telemetry — see `docs/Telemetry.md`), and `ConfigServe` (config
|
||||||
|
fetch — see [`ConfigSource`/`ConfigServe`](../CLAUDE.md#config-source-configsource--configserve)),
|
||||||
|
never over gossip.
|
||||||
|
- **`SplitTopologyTransportValidator` fails host start** on a node carrying a `cluster-{ClusterId}`
|
||||||
|
role unless it is also configured for the split-safe transports: `MeshTransport:Mode = ClusterClient`,
|
||||||
|
`Telemetry:Mode = Grpc` (driver nodes), `TelemetryDial:Mode = Grpc` (admin nodes). A DPS-mode transport
|
||||||
|
on a cluster-role node would silently try to gossip across a partition that no longer exists, so the
|
||||||
|
validator fails closed instead. Legacy/non-cluster-role nodes are exempt.
|
||||||
|
- **The compiled transport-mode defaults were deliberately NOT flipped** — `MeshTransport:Mode`,
|
||||||
|
`Telemetry:Mode`, and `TelemetryDial:Mode` all still default to `Dps`. A blast-radius assessment
|
||||||
|
found flipping the compiled default would break every integration test and every existing
|
||||||
|
`appsettings.json` profile that doesn't explicitly set these keys. The guardrail is the validator
|
||||||
|
above plus explicit per-deployment configuration, not a default flip.
|
||||||
|
|
||||||
|
### Secrets replication is pair-local
|
||||||
|
|
||||||
|
Akka DPS secret replication is scoped to each 2-node mesh: a pair replicates its own secrets to its
|
||||||
|
own partner and no further. There is no fleet-wide or cross-cluster secret sync after the split — by
|
||||||
|
design, consistent with Phase 4 (site nodes have no SQL, so there is no SQL-backed hub to fan a
|
||||||
|
cross-mesh sync through).
|
||||||
|
|
||||||
## Data flow
|
## Data flow
|
||||||
|
|
||||||
@@ -316,18 +353,17 @@ advertises `ServiceLevel` 250. OPC UA clients re-select.
|
|||||||
| Confirmation | A dialog naming the node, and what follows (restart, ServiceLevel 250 moves, clients re-select). |
|
| Confirmation | A dialog naming the node, and what follows (restart, ServiceLevel 250 moves, clients re-select). |
|
||||||
| Audit | Written through the shared `ZB.MOM.WW.Audit.IAuditWriter` seam **before** the Leave — action `cluster.manual-failover`, actor, target. A *refused* failover changes nothing and writes nothing. |
|
| Audit | Written through the shared `ZB.MOM.WW.Audit.IAuditWriter` seam **before** the Leave — action `cluster.manual-failover`, actor, target. A *refused* failover changes nothing and writes nothing. |
|
||||||
|
|
||||||
> **Mesh-scope caveat.** Until the per-cluster mesh work lands
|
Phase 6 of the mesh program (see [Per-cluster meshes (Phase 6)](#per-cluster-meshes-phase-6)) split
|
||||||
> (`docs/plans/2026-07-21-per-cluster-mesh-design.md`), the Primary is elected once per Akka mesh rather than
|
the fleet into one 2-node mesh per application `Cluster`, so the button now acts on **this pair's own**
|
||||||
> per application `Cluster` row — so on a fleet running several clusters in one mesh this button acts on the
|
Primary — there is no longer a whole-mesh vs. per-cluster ambiguity to caveat.
|
||||||
> whole mesh's Primary, which may belong to a different cluster than the page you are on. The panel says so.
|
|
||||||
> Phase 6 of the mesh program removes both the caveat and this notice.
|
|
||||||
|
|
||||||
> **Live gate outstanding.** The auto-down change is verified by unit tests against the effective
|
> **Live gate outstanding.** The auto-down change is verified by unit tests against the effective
|
||||||
> configuration, and by the sister project's live drill on an equivalent Akka version and topology. It has
|
> configuration, and by the sister project's live drill on an equivalent Akka version and topology. It
|
||||||
> **not** yet been drilled on an OtOpcUa two-node rig — the docker-dev rig runs a single six-node mesh, where
|
> was **not** drillable on an OtOpcUa rig before Phase 6, because the docker-dev rig ran a single
|
||||||
> the 1-vs-1 pathology cannot occur. The drill (kill the oldest container of a genuine two-node cluster;
|
> six-node mesh where the 1-vs-1 pathology cannot occur. **Phase 6 has landed and every mesh is now
|
||||||
> assert the survivor stays up, takes the singletons, and reports the primary `ServiceLevel`) should run
|
> exactly two nodes**, so the drill (kill the oldest container of a genuine two-node cluster; assert
|
||||||
> alongside the per-cluster mesh work, which makes every mesh exactly two nodes. See
|
> the survivor stays up, takes the singletons, and reports the primary `ServiceLevel`) is finally
|
||||||
|
> testable — deferred to Phase 7's failover drill. See
|
||||||
> `docs/plans/2026-07-21-per-cluster-mesh-design.md` §6.2 / Phase 0a.
|
> `docs/plans/2026-07-21-per-cluster-mesh-design.md` §6.2 / Phase 0a.
|
||||||
|
|
||||||
## Bootstrap: self-first seed ordering
|
## Bootstrap: self-first seed ordering
|
||||||
|
|||||||
@@ -312,7 +312,7 @@ Deliberately not a task plan — per-phase plans follow, one at a time.
|
|||||||
| 3 | Config fetch-and-cache: artifact served by central, driver nodes read LocalDb (§6.1) | No |
|
| 3 | Config fetch-and-cache: artifact served by central, driver nodes read LocalDb (§6.1) | No |
|
||||||
| 4 | Cut the driver-side ConfigDb connection: re-home `EfAlarmConditionStateStore`, resolve the `DbHealthProbeActor` ServiceLevel input, audit `OpcUaPublishActor` | No |
|
| 4 | Cut the driver-side ConfigDb connection: re-home `EfAlarmConditionStateStore`, resolve the `DbHealthProbeActor` ServiceLevel input, audit `OpcUaPublishActor` | No |
|
||||||
| 5 | gRPC stream contract; migrate the seven observability topics | No |
|
| 5 | gRPC stream contract; migrate the seven observability topics | No |
|
||||||
| 6 | Mesh partition: per-cluster seed nodes, cluster-scoped roles + singletons; rig models the real topology | No |
|
| 6 | Mesh partition: per-cluster seed nodes, cluster-scoped roles + singletons; rig models the real topology | No — **DONE 2026-07-24**, see `2026-07-22-per-cluster-mesh-program.md` §Phase 6 and `2026-07-24-mesh-phase6-partition-and-colocation.md` |
|
||||||
| 7 | Failover drill (both directions, per §6.2); live gate | No |
|
| 7 | Failover drill (both directions, per §6.2); live gate | No |
|
||||||
|
|
||||||
Phases 3 and 4 are the ones that change a running system's data path rather than its wiring, and each
|
Phases 3 and 4 are the ones that change a running system's data path rather than its wiring, and each
|
||||||
|
|||||||
@@ -263,25 +263,45 @@ kill-and-reconnect of a site node's dialer recovers every stream (driver-health/
|
|||||||
from the node hub's snapshot replay). Treat Phase 5 as **code-complete, not verified**, until this
|
from the node hub's snapshot replay). Treat Phase 5 as **code-complete, not verified**, until this
|
||||||
lands — see `docs/plans/2026-07-23-mesh-phase5-grpc-telemetry-stream.md` Task 12.
|
lands — see `docs/plans/2026-07-23-mesh-phase5-grpc-telemetry-stream.md` Task 12.
|
||||||
|
|
||||||
### Phase 6 — Mesh partition + co-location topology
|
### Phase 6 — Mesh partition + co-location topology — **DONE 2026-07-24**
|
||||||
**Scope:** per-cluster seed nodes — ~~adopt ScadaBridge's self-first ordering and RETIRE the
|
**Scope:** per-cluster seed nodes — ~~adopt ScadaBridge's self-first ordering and RETIRE the
|
||||||
`SelfFormAfter` watchdog + TCP reachability guard~~ **DONE EARLY 2026-07-22** (docker-dev
|
`SelfFormAfter` watchdog + TCP reachability guard~~ **DONE EARLY 2026-07-22** (docker-dev
|
||||||
`central-2` swapped to self-first, `ClusterBootstrapFallback`/`SelfFormAfter` deleted,
|
`central-2` swapped to self-first, `ClusterBootstrapFallback`/`SelfFormAfter` deleted,
|
||||||
`AkkaClusterOptionsValidator` ports ScadaBridge's startup rule in its conditional form, and
|
`AkkaClusterOptionsValidator` ports ScadaBridge's startup rule in its conditional form, and
|
||||||
`SelfFirstSeedBootstrapTests` replaces `SelfFormBootstrapTests`; see `docs/Redundancy.md`
|
`SelfFirstSeedBootstrapTests` replaces `SelfFormBootstrapTests`; see `docs/Redundancy.md`
|
||||||
§"Bootstrap: self-first seed ordering"). What remains for this phase is the per-pair
|
§"Bootstrap: self-first seed ordering"). The rest of the phase — actually splitting the single
|
||||||
`Cluster__SeedNodes__*` matrix once the meshes actually split — every node in a pair is then a seed
|
fleet-wide mesh into three independent 2-node meshes (central `cluster-MAIN`, site-a
|
||||||
of its own mesh, so the site-node exemption disappears; cluster-scoped roles `cluster-{ClusterId}` + singleton re-scoping,
|
`cluster-SITE-A`, site-b `cluster-SITE-B`) — **shipped 2026-07-24**. See
|
||||||
central pair keeps the admin singletons; **docker-dev rig rewritten** to model the real topology —
|
`docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md` for the full plan and task list.
|
||||||
including the co-location port table above (both products' compose files on shared per-site
|
Delivered:
|
||||||
networks, real LocalDb sync ports); remove the ClusterRedundancy page's mesh-scope caveat (the
|
- Per-pair `Cluster:SeedNodes` (each node lists itself first, its pair partner second — no
|
||||||
election is pair-local now) and the fallback's site-node island-guard docs note (moot — every
|
cross-pair seeds) + `cluster-{ClusterId}` roles on every driver node + singleton re-scoping:
|
||||||
node is a seed of its own mesh); `Cluster__SeedNodes__*` env matrix per pair.
|
`RedundancyStateActor` is now a `cluster-{ClusterId}`-scoped singleton spawned on every driver
|
||||||
**Exit gate:** rig up in the new shape; every existing live-gated behavior re-verified per pair
|
node (falls back to plain `driver` for legacy nodes), so **each pair elects its own Primary —
|
||||||
(deploy, redundancy 250/240 per pair — **two Primaries fleet-wide, one per pair, by design**);
|
two or more Primaries fleet-wide, by design**; `ClusterNodeAddressReconciler` is own-cluster-scoped.
|
||||||
secrets Akka replication re-verified or re-scoped (it rides DPS on the current single mesh — its
|
- `CentralCommunicationActor` now creates one `ClusterClient` per application `Cluster` and fans
|
||||||
topology must be re-decided here, likely SQL-hub mode like ScadaBridge, since pub/sub cannot
|
commands across them (closes the `TODO(Phase 6)` left by Phase 2).
|
||||||
cross separate meshes).
|
- New `SplitTopologyTransportValidator` fails host start on a `cluster-{ClusterId}`-role node left
|
||||||
|
on a DPS transport (`MeshTransport:Mode` must be `ClusterClient`; `Telemetry:Mode`/`TelemetryDial:Mode`
|
||||||
|
must be `Grpc`) — see `docs/Configuration.md` §"Per-cluster mesh split (Phase 6)".
|
||||||
|
- **Three decisions worth recording:**
|
||||||
|
1. **Secrets replication went pair-local, not SQL-hub.** The design flagged "likely SQL-hub mode
|
||||||
|
like ScadaBridge" as the fallback if DPS couldn't cross the split meshes — in the event, each
|
||||||
|
pair simply keeps replicating its own secrets within itself (no cross-cluster sync needed, and
|
||||||
|
no SQL-hub was built, consistent with Phase 4: site nodes have no SQL to hub through).
|
||||||
|
2. **The compiled transport-mode defaults (`MeshTransport:Mode`, `Telemetry:Mode`,
|
||||||
|
`TelemetryDial:Mode`) were deliberately left at `Dps`**, not flipped to the split-safe modes —
|
||||||
|
a blast-radius assessment found flipping the default breaks every integration test and every
|
||||||
|
appsettings profile that leaves the key unset. The guardrail is
|
||||||
|
`SplitTopologyTransportValidator` plus explicit per-deployment config, not a default flip.
|
||||||
|
3. **The docker-dev rig models three OtOpcUa-only meshes** (central/site-a/site-b), not the
|
||||||
|
ScadaBridge co-location topology sketched in the design's port table — co-location with
|
||||||
|
ScadaBridge's own site nodes stayed out of scope for this phase.
|
||||||
|
**Exit gate:** rig up in the new three-mesh shape; every existing live-gated behavior re-verified
|
||||||
|
per pair (deploy, redundancy 250/240 per pair, two-Primaries-fleet-wide confirmed); the
|
||||||
|
ClusterRedundancy page's mesh-scope caveat and `IManualFailoverService`'s doc-comment caveat
|
||||||
|
removed (the election is pair-local now); `docs/Redundancy.md`'s per-Akka-cluster KNOWN LIMITATION
|
||||||
|
marked resolved.
|
||||||
|
|
||||||
### Phase 7 — Failover drill + live gates
|
### Phase 7 — Failover drill + live gates
|
||||||
**Scope:** the drill ScadaBridge already has (`failover-drill.sh` analogue) run per pair, both
|
**Scope:** the drill ScadaBridge already has (`failover-drill.sh` analogue) run per pair, both
|
||||||
@@ -318,5 +338,5 @@ resource sizing on the site VMs should be checked once both products run the ful
|
|||||||
| 3 fetch-and-cache | **DONE 2026-07-23**, merged `d01b0695`, live gate PASSED, pushed to origin (scadaproj umbrella index updated + pushed `b5e7bc8`). gRPC fetch RPC + shared node key; dark switch `ConfigSource:Mode` (Direct default). Rig flip: `OTOPCUA_CONFIG_MODE=FetchAndCache` on the site nodes. See `2026-07-22-mesh-phase3-config-fetch-and-cache.md`. |
|
| 3 fetch-and-cache | **DONE 2026-07-23**, merged `d01b0695`, live gate PASSED, pushed to origin (scadaproj umbrella index updated + pushed `b5e7bc8`). gRPC fetch RPC + shared node key; dark switch `ConfigSource:Mode` (Direct default). Rig flip: `OTOPCUA_CONFIG_MODE=FetchAndCache` on the site nodes. See `2026-07-22-mesh-phase3-config-fetch-and-cache.md`. |
|
||||||
| 4 cut driver ConfigDb | **DONE 2026-07-23 — live gate PASSED** on `feat/mesh-phase4` (Tasks 0–8 + 1b + 10–11; Task 9 table-drop deferred). ConfigDb admin-only; driver-only ⇒ FetchAndCache (validator); `DbHealthProbeActor` not spawned driver-only (client-visible ServiceLevel 240/250 with central SQL down — proven live); alarm condition state in replicated LocalDb `alarm_condition_state`; central persists acks; `OpcUaPublish` guard split fixes the driver-only address-space wipe; driver-only LDAP maps from appsettings only (6th consumer found mid-phase). Gate: deploy sealed green w/ 4 DB-less site nodes acking, ServiceLevel held 240 w/ SQL down, restart booted last-known-good from the LocalDb pointer. See `2026-07-23-mesh-phase4-cut-driver-configdb.md` + `2026-07-23-mesh-phase4-live-gate.md`. |
|
| 4 cut driver ConfigDb | **DONE 2026-07-23 — live gate PASSED** on `feat/mesh-phase4` (Tasks 0–8 + 1b + 10–11; Task 9 table-drop deferred). ConfigDb admin-only; driver-only ⇒ FetchAndCache (validator); `DbHealthProbeActor` not spawned driver-only (client-visible ServiceLevel 240/250 with central SQL down — proven live); alarm condition state in replicated LocalDb `alarm_condition_state`; central persists acks; `OpcUaPublish` guard split fixes the driver-only address-space wipe; driver-only LDAP maps from appsettings only (6th consumer found mid-phase). Gate: deploy sealed green w/ 4 DB-less site nodes acking, ServiceLevel held 240 w/ SQL down, restart booted last-known-good from the LocalDb pointer. See `2026-07-23-mesh-phase4-cut-driver-configdb.md` + `2026-07-23-mesh-phase4-live-gate.md`. |
|
||||||
| 5 gRPC telemetry | **DONE 2026-07-23 — live gate PASSED** on `feat/mesh-phase5`. 4-channel scope (`alerts`/`script-logs`/`driver-health`/`driver-resilience-status`), 3 deferred with rationale (`redundancy-state`, `fleet-status`, `deployment-acks`); dark switch `Telemetry:Mode`/`TelemetryDial:Mode` (Dps default); node hosts server / central dials; auth-from-day-one fail-closed bearer key, superseding design §6.3. Gate: full 12-stream mesh formed in Grpc mode (2 inbound per driver node, 6 outbound per central), AdminUI pill live (data path proven), kill-and-reconnect of a node recovered its stream in ~5s; Dps baseline dials nothing. Surfaced an upgrade gotcha — `ClusterNode.GrpcPort` must be populated on existing deployments (fresh installs seed it; nullable column doesn't backfill) — which live-validated the graceful null-`GrpcPort` skip. See `docs/Telemetry.md`, `2026-07-23-mesh-phase5-grpc-telemetry-stream.md` + `2026-07-23-mesh-phase5-live-gate.md`. |
|
| 5 gRPC telemetry | **DONE 2026-07-23 — live gate PASSED** on `feat/mesh-phase5`. 4-channel scope (`alerts`/`script-logs`/`driver-health`/`driver-resilience-status`), 3 deferred with rationale (`redundancy-state`, `fleet-status`, `deployment-acks`); dark switch `Telemetry:Mode`/`TelemetryDial:Mode` (Dps default); node hosts server / central dials; auth-from-day-one fail-closed bearer key, superseding design §6.3. Gate: full 12-stream mesh formed in Grpc mode (2 inbound per driver node, 6 outbound per central), AdminUI pill live (data path proven), kill-and-reconnect of a node recovered its stream in ~5s; Dps baseline dials nothing. Surfaced an upgrade gotcha — `ClusterNode.GrpcPort` must be populated on existing deployments (fresh installs seed it; nullable column doesn't backfill) — which live-validated the graceful null-`GrpcPort` skip. See `docs/Telemetry.md`, `2026-07-23-mesh-phase5-grpc-telemetry-stream.md` + `2026-07-23-mesh-phase5-live-gate.md`. |
|
||||||
| 6 mesh partition + co-location | not started |
|
| 6 mesh partition + co-location | **DONE 2026-07-24** — three independent 2-node meshes (central/site-a/site-b), `cluster-{ClusterId}` roles, per-pair self-first seeds, cluster-scoped `RedundancyStateActor` singleton (two+ Primaries fleet-wide, by design), one `ClusterClient` per `Cluster`, `SplitTopologyTransportValidator`, pair-local secrets replication, compiled transport defaults intentionally kept `Dps`. See `2026-07-24-mesh-phase6-partition-and-colocation.md`. |
|
||||||
| 7 drill + live gates | not started |
|
| 7 drill + live gates | not started |
|
||||||
|
|||||||
@@ -0,0 +1,540 @@
|
|||||||
|
# Per-Cluster Mesh Phase 6 — Mesh Partition + Co-location Topology
|
||||||
|
|
||||||
|
> **For Claude:** REQUIRED SUB-SKILL: execute this plan task-by-task with
|
||||||
|
> superpowers-extended-cc:subagent-driven-development (the established mode for this workstream).
|
||||||
|
|
||||||
|
**Goal:** Split OtOpcUa's single fleet-wide Akka mesh into **one 2-node mesh per application
|
||||||
|
`Cluster`** (central pair + one pair per site), so every Primary-gated decision and every gated
|
||||||
|
resource shares one pair-local scope — the ScadaBridge shape. Same ActorSystem name (`otopcua`)
|
||||||
|
everywhere; separation is by **seed-node partitioning** only.
|
||||||
|
|
||||||
|
**Architecture:** Each pair's two nodes list only each other as seeds (self-first), so Akka forms
|
||||||
|
three independent meshes even on a shared network. Driver nodes carry a new `cluster-{ClusterId}`
|
||||||
|
role; the one pair-local singleton (`RedundancyStateActor`) re-scopes to it and is spawned on every
|
||||||
|
driver node, so each pair elects its own Primary and publishes `redundancy-state` on its own mesh's
|
||||||
|
DPS. Central (admin) reaches each site mesh over the explicit Phase 2/3/5 transports (ClusterClient
|
||||||
|
+ ConfigServe gRPC + telemetry gRPC), now partitioned **one ClusterClient per cluster**. The five
|
||||||
|
admin singletons stay on the central pair. Two admin-side consumers that assumed fleet-wide gossip
|
||||||
|
visibility (`ClusterNodeAddressReconciler`, `CentralCommunicationActor.RebuildClient`) are
|
||||||
|
re-scoped. Secrets replication becomes **pair-local** automatically (Akka DPS within each 2-node
|
||||||
|
mesh; no fleet-wide sync, no central-SQL dependency — consistent with Phase 4). A startup validator
|
||||||
|
refuses to boot a split-configured node (one carrying a `cluster-` role) that is still pointed at a
|
||||||
|
DPS transport mode, and the compiled transport defaults flip to the split-correct values.
|
||||||
|
|
||||||
|
**Tech Stack:** .NET 10, Akka.NET + Akka.Hosting (`WithSingleton`/`WithActors`), Akka.Cluster.Tools
|
||||||
|
(ClusterClient + ClusterSingleton + DistributedPubSub), Akka.Cluster.Hosting `ClusterSingletonOptions`,
|
||||||
|
xUnit + Shouldly, docker-dev compose rig.
|
||||||
|
|
||||||
|
**Decisions settled with the user 2026-07-24 (recon-driven; the first contradicts the program doc's
|
||||||
|
own guess):**
|
||||||
|
1. **Secrets after split = pair-local Akka.** The program doc guessed "SQL-hub mode like ScadaBridge,"
|
||||||
|
but recon showed sites have **no SQL Server** (the point of Phase 4), so a SQL hub would force site
|
||||||
|
nodes back onto central SQL. Instead: leave the `AddZbSecretsAkkaReplication` wiring untouched — it
|
||||||
|
becomes per-pair for free once each mesh is two nodes. Fleet-wide secret sync is dropped by design;
|
||||||
|
document it. No library re-wire.
|
||||||
|
2. **DPS branch disposition = flip + change defaults + validator, keep the code.** The rig flips to
|
||||||
|
`ClusterClient`/`Grpc`/`FetchAndCache`; the compiled defaults change so a fresh deploy is
|
||||||
|
split-correct; the DPS branch code stays as dark switches; a new startup validator makes a
|
||||||
|
split-configured node refuse to boot on a DPS transport mode (neutralizes the silent-break footgun
|
||||||
|
without a large irreversible deletion). `redundancy-state`/`fleet-status` stay on DPS either way.
|
||||||
|
3. **Rig scope = OtOpcUa-only per-cluster meshes.** Rewrite docker-dev into 3 independent 2-node
|
||||||
|
meshes on a single shared network (faithful to production, where separation is by seeds over a WAN,
|
||||||
|
not by network isolation). Physical co-location with ScadaBridge is documented as the production
|
||||||
|
topology, not literally run in docker-dev.
|
||||||
|
|
||||||
|
**Authoritative context (read before implementing):**
|
||||||
|
- `docs/plans/2026-07-22-per-cluster-mesh-program.md` §"Phase 6" (lines 266–284) + the tracking table.
|
||||||
|
- `docs/plans/2026-07-21-per-cluster-mesh-design.md` §4 (oldest-Up election, already fixed), §5
|
||||||
|
(target architecture), §7 (sequencing).
|
||||||
|
- Recon findings (this session) — summarized inline per task below.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## What Phase 6 does NOT change (scope guards)
|
||||||
|
|
||||||
|
- **ActorSystem name stays `otopcua`.** Do not rename it or make it per-cluster.
|
||||||
|
- **`AkkaClusterOptions` gains no new property.** `SeedNodes` + `Roles` already carry per-node values
|
||||||
|
from config; the split is in *how they're populated per pair* (rig/appsettings) + the `RoleParser`
|
||||||
|
allow-list + singleton scoping. Do not add a `ClusterId` option — every `ClusterNode` row is already
|
||||||
|
defined to be a driver node (Phase 1 decision), and roles are the declaration of record.
|
||||||
|
- **`SelectDriverPrimary` election logic is unchanged.** It filters on the `driver` role over
|
||||||
|
`Cluster.State.Members`; after the split that set is pair-local, so it elects the oldest of two.
|
||||||
|
No edit to the algorithm.
|
||||||
|
- **The five admin singletons stay admin-scoped** (`config-publish`, `admin-operations`,
|
||||||
|
`audit-writer`, `fleet-status`, `cluster-node-address-reconciler`). Only `redundancy-state`
|
||||||
|
re-homes.
|
||||||
|
- **`DriverMemberCount()`** in `DriverHostActor`/`ScriptedAlarmHostActor` needs **no code change** —
|
||||||
|
it reads whole-mesh members filtered by `driver`, which becomes pair-local for free. (Note the
|
||||||
|
semantics change in docs; do not "fix" it in code.)
|
||||||
|
- **Do NOT delete the DPS command/telemetry branches** (decision 2). Do NOT switch the network to
|
||||||
|
per-site isolation or wire in ScadaBridge (decision 3). Do NOT re-wire secrets to SQL (decision 1).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 0: `RoleParser` accepts `cluster-{ClusterId}`; consolidate the `driver` role literal
|
||||||
|
|
||||||
|
**Classification:** small
|
||||||
|
**Estimated implement time:** ~4 min
|
||||||
|
**Parallelizable with:** none (blocks 1, 2, 5, 7)
|
||||||
|
|
||||||
|
**Context:** `RoleParser.Allowed` is a fixed `HashSet {"admin","driver","dev"}`
|
||||||
|
(`src/Core/ZB.MOM.WW.OtOpcUa.Cluster/RoleParser.cs:5-8`). A `cluster-{ClusterId}` role fails it, so
|
||||||
|
no node can carry one. Also, three independent `"driver"` string literals exist
|
||||||
|
(`RedundancyStateActor.DriverRole`, `ClusterNodeAddressReconcilerActor.DriverRole`,
|
||||||
|
`Runtime/ServiceCollectionExtensions.DriverRole`) — consolidate to a single source now since every
|
||||||
|
Phase 6 task touches role code.
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/RoleParser.cs`
|
||||||
|
- Test: `tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/RoleParserTests.cs` (create if absent)
|
||||||
|
|
||||||
|
**Steps:**
|
||||||
|
1. Add a well-known-roles surface to `RoleParser` (or a new `ClusterRoles` static): constants
|
||||||
|
`Admin = "admin"`, `Driver = "driver"`, `Dev = "dev"`, `ClusterRolePrefix = "cluster-"`, and a
|
||||||
|
helper `bool IsClusterRole(string role) => role.StartsWith(ClusterRolePrefix, StringComparison.Ordinal) && role.Length > ClusterRolePrefix.Length`.
|
||||||
|
2. Change `Parse` validation: a role is allowed if it is one of the three fixed roles **or**
|
||||||
|
`IsClusterRole(role)` is true. Keep the existing trim/lowercase/empty-filter behavior. A role of
|
||||||
|
exactly `"cluster-"` (empty ClusterId) is **rejected**.
|
||||||
|
3. Add `RoleParser.ClusterIdFromRole(string) => role.Substring(ClusterRolePrefix.Length)` returning
|
||||||
|
the ClusterId for a cluster role (used by later tasks).
|
||||||
|
4. Update `RedundancyStateActor.DriverRole`, `ClusterNodeAddressReconcilerActor.DriverRole`, and
|
||||||
|
`Runtime/ServiceCollectionExtensions.DriverRole` to reference the single `RoleParser.Driver` (keep
|
||||||
|
the public `const`/`static readonly` symbols as aliases if any test references them by their old
|
||||||
|
name — grep first; do not break existing test references).
|
||||||
|
5. Tests: `admin`/`driver`/`dev`/`cluster-MAIN`/`cluster-SITE-A` accepted; `cluster-` rejected;
|
||||||
|
`bogus` rejected; case/whitespace normalization preserved; `ClusterIdFromRole("cluster-SITE-A") == "SITE-A"`.
|
||||||
|
|
||||||
|
**Acceptance:** `dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests` green; `cluster-{X}` roles
|
||||||
|
parse; the three literals resolve to one constant.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 1: `ClusterRoleInfo` exposes the node's own cluster-scoped role
|
||||||
|
|
||||||
|
**Classification:** small
|
||||||
|
**Estimated implement time:** ~4 min
|
||||||
|
**Parallelizable with:** none (needs Task 0; blocks 2, 5)
|
||||||
|
|
||||||
|
**Context:** Singleton registration and the validator both need "what is *this* node's
|
||||||
|
`cluster-{ClusterId}` role." `ClusterRoleInfo` (`src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ClusterRoleInfo.cs`,
|
||||||
|
`IClusterRoleInfo`, registered in `AddOtOpcUaCluster`) is the existing role/leader authority — extend
|
||||||
|
it.
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ClusterRoleInfo.cs`
|
||||||
|
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/IClusterRoleInfo.cs` (or wherever the interface lives)
|
||||||
|
- Test: `tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/ClusterRoleInfoTests.cs`
|
||||||
|
|
||||||
|
**Steps:**
|
||||||
|
1. Add `string? ClusterRole { get; }` (e.g. `"cluster-SITE-A"`) and `string? ClusterId { get; }` (e.g.
|
||||||
|
`"SITE-A"`) to `IClusterRoleInfo`, derived from the configured `AkkaClusterOptions.Roles` via
|
||||||
|
`RoleParser.IsClusterRole` (first match; log a Warning and pick the first if >1 cluster role is
|
||||||
|
configured — that is a misconfiguration). Null when the node carries no cluster role (e.g. a
|
||||||
|
pre-split/admin-only-legacy node).
|
||||||
|
2. Do NOT read `Cluster.State` for this — it is the node's **own configured** identity, available
|
||||||
|
before the cluster forms (the singleton registration runs at host build time).
|
||||||
|
3. Tests: a node configured `["driver","cluster-SITE-A"]` reports `ClusterRole="cluster-SITE-A"`,
|
||||||
|
`ClusterId="SITE-A"`; a node `["admin","driver"]` reports both null; two cluster roles → first wins
|
||||||
|
+ warning path exercised.
|
||||||
|
|
||||||
|
**Acceptance:** `ClusterRoleInfo` reports the node's cluster role/id from config; tests green.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 2: Re-home `RedundancyStateActor` to a `cluster-{ClusterId}` singleton on driver nodes
|
||||||
|
|
||||||
|
**Classification:** high-risk
|
||||||
|
**Estimated implement time:** ~5 min
|
||||||
|
**Parallelizable with:** Task 3, Task 4 (disjoint files)
|
||||||
|
|
||||||
|
**Context (the crux).** Today `RedundancyStateActor` is registered as an **admin** singleton
|
||||||
|
(`ControlPlane/ServiceCollectionExtensions.cs:133-136`, `singletonOptions.Role = "admin"`), electing a
|
||||||
|
**fleet-wide** Primary and publishing `redundancy-state` over DPS. After the split, central's mesh
|
||||||
|
sees only its own pair — so a site pair (driver-only, no admin node) would have **no one** to elect its
|
||||||
|
Primary. Fix: scope the singleton to the node's `cluster-{ClusterId}` role and spawn it on **every
|
||||||
|
driver node** (`hasDriver`), so each mesh (central pair = `cluster-MAIN`, each site pair =
|
||||||
|
`cluster-{SITE}`) has exactly one, electing its own pair-local Primary and publishing on its own mesh's
|
||||||
|
DPS. **De-risked:** `RedundancyStateActor` has zero injected deps (verified — it only uses
|
||||||
|
`Cluster.Get(Context.System)` + DPS), so it spawns cleanly on a driver-only node.
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/ServiceCollectionExtensions.cs`
|
||||||
|
(remove `redundancy-state` from `WithOtOpcUaControlPlaneSingletons`; add a new
|
||||||
|
`WithOtOpcUaClusterRedundancySingleton(IClusterRoleInfo)` extension that registers it scoped to the
|
||||||
|
cluster role)
|
||||||
|
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs` (call the new extension on `hasDriver`, not
|
||||||
|
`hasAdmin`; a fused `admin,driver` node calls it once via the driver branch)
|
||||||
|
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/...` singleton-role test; and confirm the
|
||||||
|
admin path no longer registers `redundancy-state`.
|
||||||
|
|
||||||
|
**Steps:**
|
||||||
|
1. In `ControlPlane/ServiceCollectionExtensions.cs`: delete the `WithSingleton` block for
|
||||||
|
`RedundancyStateActorKey`/`redundancy-state` from the admin method (lines ~133-136). Keep the
|
||||||
|
registry key type.
|
||||||
|
2. Add:
|
||||||
|
```csharp
|
||||||
|
public static AkkaConfigurationBuilder WithOtOpcUaClusterRedundancySingleton(
|
||||||
|
this AkkaConfigurationBuilder builder, IClusterRoleInfo roleInfo)
|
||||||
|
{
|
||||||
|
// Cluster-scoped when the node carries a cluster-{ClusterId} role (Phase-6 split: one
|
||||||
|
// redundancy singleton per pair, robust even if two meshes accidentally merged). Falls
|
||||||
|
// back to the driver role for a legacy single-mesh node or a test harness with no cluster
|
||||||
|
// role — where driver-scope is the pre-Phase-6 fleet-wide behavior, and on a genuinely
|
||||||
|
// split 2-node mesh the driver role is ALREADY pair-local because the mesh IS the pair.
|
||||||
|
// No throw: decision 2 requires legacy/harness nodes to keep booting unchanged.
|
||||||
|
var role = roleInfo.ClusterRole ?? RoleParser.Driver;
|
||||||
|
var opts = new ClusterSingletonOptions { Role = role };
|
||||||
|
return builder.WithSingleton<RedundancyStateActorKey>(
|
||||||
|
RedundancyStateActor.Topic, RedundancyStateActor.Props(), opts /* createProxyToo per existing pattern */);
|
||||||
|
}
|
||||||
|
```
|
||||||
|
Match the exact `WithSingleton` overload/signature the other five singletons use (check createProxy
|
||||||
|
arg + singleton name conventions).
|
||||||
|
3. In `Program.cs`: within the `if (hasDriver)` Akka-configurator block (near line 357-374, alongside
|
||||||
|
`WithOtOpcUaRuntimeActors`), call `ab.WithOtOpcUaClusterRedundancySingleton(clusterRoleInfo)`.
|
||||||
|
Resolve `IClusterRoleInfo` from `sp`. Ensure it runs on the fused central node too (central has
|
||||||
|
`hasDriver=true`). Do NOT also register it in the `hasAdmin` block.
|
||||||
|
4. **No fail-fast on a missing cluster role** — the fallback keeps legacy/harness nodes booting
|
||||||
|
(decision 2). A *misconfigured* Phase-6 node (split transports, no cluster role) is a non-issue for
|
||||||
|
the redundancy singleton: driver-scope is pair-local on a split mesh anyway.
|
||||||
|
5. Tests: assert the admin singleton set no longer contains `redundancy-state`; assert the new
|
||||||
|
extension registers a singleton scoped to `Role == "cluster-SITE-A"` when the roleInfo reports that;
|
||||||
|
assert that a roleInfo with null `ClusterRole` falls back to `Role == "driver"` (no throw).
|
||||||
|
|
||||||
|
**Acceptance:** each mesh hosts exactly one `RedundancyStateActor` scoped to its cluster role; a
|
||||||
|
driver-only site node elects its own Primary; admin path no longer owns it. Unit tests green.
|
||||||
|
|
||||||
|
**Live-verified later** (Task 9 gate): two Primaries fleet-wide, one per pair.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 3: Re-scope `ClusterNodeAddressReconciler` to central-mesh-visible rows
|
||||||
|
|
||||||
|
**Classification:** standard
|
||||||
|
**Estimated implement time:** ~5 min
|
||||||
|
**Parallelizable with:** Task 2, Task 4 (disjoint files)
|
||||||
|
|
||||||
|
**Context:** The admin-singleton reconciler compares `Cluster.State.Members` (driver role) against
|
||||||
|
**every** `ClusterNode` row fleet-wide (`ClusterNodeAddressReconciler.cs:123-131`,
|
||||||
|
`ClusterNodeAddressReconcilerActor.cs:76-96`). After the split, central sees only its own pair, so
|
||||||
|
every site row logs `EnabledRowNotInCluster` forever (its own doc comment,
|
||||||
|
`ClusterNodeAddressReconciler.cs:60-65`, flags this). Fix: only reconcile rows whose `ClusterId`
|
||||||
|
matches central's own cluster — central can no longer observe other clusters via gossip, so it must
|
||||||
|
not assert anything about their rows.
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Fleet/ClusterNodeAddressReconciler.cs` (filter
|
||||||
|
the row set by own-cluster)
|
||||||
|
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Fleet/ClusterNodeAddressReconcilerActor.cs`
|
||||||
|
(pass the admin node's own `ClusterId` in; source it from `IClusterRoleInfo.ClusterId`, or from the
|
||||||
|
set of `ClusterId`s present among its own observed driver members)
|
||||||
|
- Update the class doc comment (remove the "Phase 2 must revisit" note; state it is now own-cluster
|
||||||
|
scoped).
|
||||||
|
- Test: `tests/.../ClusterNodeAddressReconcilerTests.cs` — a row in another cluster is **ignored** (no
|
||||||
|
finding); an own-cluster enabled row with no member still yields `EnabledRowNotInCluster`; drift
|
||||||
|
detection within own cluster still works.
|
||||||
|
|
||||||
|
**The pure `ClusterNodeAddressReconciler.Reconcile` function is UNCHANGED** — the fix is entirely in
|
||||||
|
the actor's two DB queries (`ClusterNodeAddressReconcilerActor.Reconcile`, lines 89-97), which filter
|
||||||
|
rows by the admin node's own `ClusterId` before projecting. `ClusterNode` has a `ClusterId` column.
|
||||||
|
|
||||||
|
**Steps:**
|
||||||
|
1. Inject the admin node's own `ClusterId` into the actor: add `IClusterRoleInfo` (or just a
|
||||||
|
`string? ownClusterId`) to `Props`/ctor, sourced from `IClusterRoleInfo.ClusterId` at registration
|
||||||
|
(`ControlPlane/ServiceCollectionExtensions.cs` where the reconciler singleton is registered).
|
||||||
|
2. In the actor's `Reconcile`, add `.Where(n => n.ClusterId == ownClusterId)` to BOTH the `rows` and
|
||||||
|
`enabled` queries — but ONLY when `ownClusterId` is non-null.
|
||||||
|
3. **Null `ownClusterId` (legacy admin node with no cluster role) ⇒ reconcile ALL rows** — this is the
|
||||||
|
correct single-mesh behavior (a legacy admin node genuinely sees every driver member via gossip).
|
||||||
|
Do NOT skip. This preserves backward-compat, mirroring the Task 2 fallback philosophy.
|
||||||
|
4. `RowDialTargetDisagrees` / `RunningNodeHasNoRow` / `EnabledRowNotInCluster` semantics are unchanged;
|
||||||
|
they now simply operate on the own-cluster row subset (post-split) or the full set (legacy).
|
||||||
|
5. Update the class doc comment in `ClusterNodeAddressReconciler.cs:60-65`: replace the "Phase 2 must
|
||||||
|
revisit" note with the delivered own-cluster-scoped behavior.
|
||||||
|
6. Tests (`ClusterNodeAddressReconcilerActorTests` or the existing reconciler tests — pick the level
|
||||||
|
that can inject `ownClusterId`): a SITE-A row is **ignored** when `ownClusterId=="MAIN"` (no
|
||||||
|
`EnabledRowNotInCluster`); an own-cluster (MAIN) enabled row with no member still yields
|
||||||
|
`EnabledRowNotInCluster`; own-cluster drift still caught; `ownClusterId==null` reconciles the full
|
||||||
|
set (legacy). If the actor is hard to unit-test directly, a focused query-filter test + the
|
||||||
|
unchanged pure-function tests suffice — say which you chose.
|
||||||
|
|
||||||
|
**Acceptance:** central reconciles only MAIN rows; site rows never produce false
|
||||||
|
`EnabledRowNotInCluster`; own-cluster drift still caught.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 4: One `ClusterClient` per application `Cluster` in `CentralCommunicationActor`
|
||||||
|
|
||||||
|
**Classification:** high-risk
|
||||||
|
**Estimated implement time:** ~5 min
|
||||||
|
**Parallelizable with:** Task 2, Task 3 (disjoint files)
|
||||||
|
|
||||||
|
**Context:** `RebuildClient` (`CentralCommunicationActor.cs:271-297`) creates **one fleet-wide**
|
||||||
|
`ClusterClient` dialing every `ClusterNode` row, and `HandleMeshCommand` (`:128-157`) does one
|
||||||
|
`ClusterClient.SendToAll(...)`. The `TODO(Phase 6)` (`:249-268`) states this is correct only on a
|
||||||
|
single mesh: a `ClusterClientReceptionist` serves its whole cluster, so `SendToAll` on one client
|
||||||
|
reaches everyone. After the split, `SendToAll` on one client reaches only the mesh whose receptionist
|
||||||
|
answered — so central needs **one client per cluster**, and dispatch fans `SendToAll` across all of
|
||||||
|
them.
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Communication/CentralCommunicationActor.cs`
|
||||||
|
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/MeshClusterClientBoundaryTests.cs`
|
||||||
|
(extend: two separate site meshes + central, a command reaches BOTH via per-cluster clients; and a
|
||||||
|
single-cluster client's `SendToAll` does NOT reach the other cluster)
|
||||||
|
|
||||||
|
**Steps:**
|
||||||
|
1. `LoadContactsFromDb` (`:177-221`): return contacts **grouped by `ClusterId`** — an
|
||||||
|
`ImmutableDictionary<string, ImmutableHashSet<string>>` (ClusterId → receptionist contact paths).
|
||||||
|
Keep the `Enabled && !MaintenanceMode` filter.
|
||||||
|
2. Replace the single `_client`/`_currentContacts` with a `Dictionary<string, (IActorRef Client,
|
||||||
|
ImmutableHashSet<string> Contacts)>` keyed by ClusterId. `HandleContactsLoaded` diffs **per cluster**
|
||||||
|
and rebuilds only the clusters whose contact set changed; stops+removes clients for clusters that
|
||||||
|
vanished.
|
||||||
|
3. `RebuildClient` becomes `RebuildClient(string clusterId, ImmutableHashSet<string> contacts)` —
|
||||||
|
builds one `ClusterClient` per cluster (name it by clusterId so the actor paths are distinct).
|
||||||
|
4. `HandleMeshCommand` fans out: for each cluster client, `client.Tell(new ClusterClient.SendToAll(
|
||||||
|
MeshPaths.NodeCommunication, cmd.Message))`. Preserve the sender-relay idiom where it exists.
|
||||||
|
**A command with no live clients (all clusters down) drops with a Warning**, matching today's
|
||||||
|
"no reachable contact" behavior — do not buffer.
|
||||||
|
5. Update the `TODO(Phase 6)` doc block to describe the delivered per-cluster-client shape (remove the
|
||||||
|
TODO marker).
|
||||||
|
6. **Frame-size note:** the deploy path stays payload-free (notify-and-fetch), so no new frame-size
|
||||||
|
exposure — do not add payload to these commands.
|
||||||
|
7. Tests: extend `MeshClusterClientBoundaryTests` to two site meshes + central; assert a dispatch
|
||||||
|
reaches a node in each site mesh, and that per-cluster `SendToAll` is correctly partitioned.
|
||||||
|
|
||||||
|
**Acceptance:** central dispatches deploy-notify/driver-control/alarm-commands to every cluster via
|
||||||
|
one client each; the boundary test proves cross-mesh partitioned delivery; acks still return via the
|
||||||
|
site nodes' own `ClusterClient.Send` to central (unchanged).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 5: Split-topology startup validator (cluster-role ⇒ non-DPS transport)
|
||||||
|
|
||||||
|
**Classification:** standard
|
||||||
|
**Estimated implement time:** ~5 min
|
||||||
|
**Parallelizable with:** Task 6 (disjoint files)
|
||||||
|
|
||||||
|
**Context (decision 2):** the DPS branches stay as dark switches, but a node configured for the split
|
||||||
|
topology (it carries a `cluster-{ClusterId}` role) must not silently run a transport still in DPS
|
||||||
|
mode, which cannot cross meshes. Fail host start instead. Marker of "split topology" = the presence of
|
||||||
|
a `cluster-` role (they exist only post-Phase-6).
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Create: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/SplitTopologyTransportValidator.cs` (an
|
||||||
|
`IValidateOptions<...>` or a startup validator wired via `AddValidatedOptions`/`ValidateOnStart`,
|
||||||
|
mirroring `AkkaClusterOptionsValidator`/`ConfigSourceOptionsValidator` structure)
|
||||||
|
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs` (`AddOtOpcUaCluster`
|
||||||
|
registers it)
|
||||||
|
- Test: `tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/SplitTopologyTransportValidatorTests.cs`
|
||||||
|
|
||||||
|
**Rule (only fires when the node carries a `cluster-` role):**
|
||||||
|
- `MeshTransport:Mode` must be `ClusterClient` (not `Dps`) — every node.
|
||||||
|
- If `hasDriver`: `Telemetry:Mode` must be `Grpc` (node hosts the telemetry server).
|
||||||
|
- If `hasAdmin`: `TelemetryDial:Mode` must be `Grpc` (central dials).
|
||||||
|
- `ConfigSource:Mode`: **do not** add a rule here — Phase 4's `ConfigSourceOptionsValidator` already
|
||||||
|
requires `FetchAndCache` on driver-only nodes; a fused node legitimately stays `Direct`.
|
||||||
|
- A node with **no** `cluster-` role (legacy/pre-split) is exempt (rule does not fire), so existing
|
||||||
|
single-mesh deployments and tests are unaffected.
|
||||||
|
|
||||||
|
**Steps:**
|
||||||
|
1. Read `Cluster:Roles` (same direct-read pattern the other validators use — not `IClusterRoleInfo`,
|
||||||
|
to avoid a service-resolution ordering dependency at validation time), `MeshTransport:Mode`,
|
||||||
|
`Telemetry:Mode`, `TelemetryDial:Mode`.
|
||||||
|
2. Implement the rule above; each violation adds a precise message naming the offending key + required
|
||||||
|
value.
|
||||||
|
3. Register with `ValidateOnStart` so a misconfigured node refuses to boot.
|
||||||
|
4. Tests: cluster-role node on `MeshTransport:Mode=Dps` fails; on `ClusterClient` passes; driver
|
||||||
|
cluster node on `Telemetry:Mode=Dps` fails; admin cluster node on `TelemetryDial:Mode=Dps` fails;
|
||||||
|
a non-cluster-role node on all-Dps passes (exempt).
|
||||||
|
|
||||||
|
**Acceptance:** a split-configured node on any DPS transport refuses to start with a clear message;
|
||||||
|
legacy nodes unaffected.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 6: Compiled transport-mode defaults — DECIDED: do NOT flip (data-backed deviation)
|
||||||
|
|
||||||
|
**Classification:** small → **resolved as a no-code decision 2026-07-24**
|
||||||
|
**Parallelizable with:** Task 5
|
||||||
|
|
||||||
|
**Original intent (decision 2):** change the compiled `MeshTransport:Mode`/`Telemetry:Mode`/
|
||||||
|
`TelemetryDial:Mode` defaults to the split-correct values so a fresh deploy is split-correct without
|
||||||
|
per-node overrides.
|
||||||
|
|
||||||
|
**What the blast-radius assessment found (read-only, this session):** flipping the compiled defaults
|
||||||
|
is **HIGH-RISK and low-value**. Every one of the three transport-option validators fail-closes when its
|
||||||
|
mode is the non-Dps value but the mode's prerequisites are unset — and *nothing in the repo sets those
|
||||||
|
sections*:
|
||||||
|
- `MeshTransport:Mode=ClusterClient` requires `CentralContactPoints` — no `appsettings*.json` and no
|
||||||
|
test harness sets it → `ValidateOnStart` throws at boot.
|
||||||
|
- `Telemetry:Mode=Grpc` requires `GrpcListenPort`+`ApiKey` on a driver node — same gap.
|
||||||
|
- `TelemetryDial:Mode=Grpc` requires `ApiKey` with **no role gate at all** — breaks even roleless
|
||||||
|
fixtures and docker-dev's 4 site nodes.
|
||||||
|
- Concretely this hard-fails **all 14 `Host.IntegrationTests`** (via `TwoNodeClusterHarness`, which
|
||||||
|
sets none of these), **all 5 shipped appsettings profiles**, and several unit tests asserting the
|
||||||
|
Dps default.
|
||||||
|
|
||||||
|
**Why not flipping is correct, not a compromise:** the Task 5 `SplitTopologyTransportValidator`
|
||||||
|
already forces `ClusterClient`/`Grpc`/`Grpc` onto **exactly** the nodes carrying a post-Phase-6
|
||||||
|
`cluster-{ClusterId}` role, fail-closed, with a precise message — which *is* "a fresh Phase-6 deploy is
|
||||||
|
split-correct," achieved more robustly than a default flip (a split node cannot even boot on the wrong
|
||||||
|
transport). Flipping the compiled defaults would merely duplicate that enforcement for split nodes
|
||||||
|
while indiscriminately breaking every legacy/test/admin node that legitimately still runs Dps. The
|
||||||
|
docker-dev rig (Task 7) sets the modes **explicitly** on its cluster-role nodes, so it never relies on
|
||||||
|
the default anyway.
|
||||||
|
|
||||||
|
**Decision:** keep `MeshTransportOptions.Mode`/`TelemetryOptions.Mode`/`TelemetryDialOptions.Mode`
|
||||||
|
defaults at `Dps`. No code change. Enforcement of split-correctness lives in Task 5's validator + the
|
||||||
|
explicit Task 7 rig config. Recorded in `docs/Configuration.md` (Task 8) so the next reader knows the
|
||||||
|
defaults are intentionally Dps and the validator is the guardrail.
|
||||||
|
|
||||||
|
**Acceptance:** MET by decision — the split validator (Task 5) is the guardrail; no default flip.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 7: Rewrite the docker-dev rig into three 2-node meshes
|
||||||
|
|
||||||
|
**Classification:** standard
|
||||||
|
**Estimated implement time:** ~5 min
|
||||||
|
**Parallelizable with:** none (needs Tasks 0–6 landed to actually run; the file edit is independent
|
||||||
|
but sequence it after code so the rig is testable)
|
||||||
|
|
||||||
|
**Context (decision 3):** one shared network, seed-partitioned into three meshes. Central pair =
|
||||||
|
`cluster-MAIN`; site-a pair = `cluster-SITE-A`; site-b pair = `cluster-SITE-B`. Each pair lists only
|
||||||
|
its own two nodes as seeds (self-first). Flip the transports on. Secrets stays enabled (now
|
||||||
|
pair-local). Central must remain network-reachable to site nodes (shared network — already true; do
|
||||||
|
**not** isolate networks).
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `docker-dev/docker-compose.yml`
|
||||||
|
- Modify: `docker-dev/seed/seed-clusters.sql` if any seeded address/port changes (AkkaPort stays 4053;
|
||||||
|
GrpcPort stays 4056 — likely no seed change, but verify the ClusterId/host columns still match).
|
||||||
|
|
||||||
|
**Per-node changes:**
|
||||||
|
| Node | `Cluster__Roles` | `Cluster__SeedNodes__0` (self) | `__1` (partner) |
|
||||||
|
|---|---|---|---|
|
||||||
|
| central-1 | `admin`,`driver`,`cluster-MAIN` | `akka.tcp://otopcua@central-1:4053` | `...central-2:4053` |
|
||||||
|
| central-2 | `admin`,`driver`,`cluster-MAIN` | `akka.tcp://otopcua@central-2:4053` | `...central-1:4053` |
|
||||||
|
| site-a-1 | `driver`,`cluster-SITE-A` | `akka.tcp://otopcua@site-a-1:4053` | `...site-a-2:4053` |
|
||||||
|
| site-a-2 | `driver`,`cluster-SITE-A` | `akka.tcp://otopcua@site-a-2:4053` | `...site-a-1:4053` |
|
||||||
|
| site-b-1 | `driver`,`cluster-SITE-B` | `akka.tcp://otopcua@site-b-1:4053` | `...site-b-2:4053` |
|
||||||
|
| site-b-2 | `driver`,`cluster-SITE-B` | `akka.tcp://otopcua@site-b-2:4053` | `...site-b-1:4053` |
|
||||||
|
|
||||||
|
**Transport flips (all nodes unless noted):**
|
||||||
|
- `MeshTransport__Mode: ClusterClient` (remove the `${OTOPCUA_MESH_MODE:-Dps}` default or set the var
|
||||||
|
default to ClusterClient).
|
||||||
|
- `Telemetry__Mode: Grpc` (all); `TelemetryDial__Mode: Grpc` (central pair only — the dialers).
|
||||||
|
- `ConfigSource__Mode: FetchAndCache` already on sites; central stays `Direct`.
|
||||||
|
- `MeshTransport__CentralContactPoints__0/1` on **site** nodes point at central-1/central-2 (unchanged
|
||||||
|
— sites learn central from appsettings). Central nodes do not need CentralContactPoints for
|
||||||
|
themselves; central discovers *site* contacts from `ClusterNode` rows (per-cluster clients, Task 4).
|
||||||
|
- **Secrets** (`x-secrets-env` anchor): keep `Secrets__Replication__Enabled=true` — it becomes
|
||||||
|
pair-local automatically. Add a compose comment noting the new pair-local semantics.
|
||||||
|
- **Remove** the `central-1`-as-seed lines from every site node (the old cross-pair seed). Update the
|
||||||
|
header comment block (lines 1-23) to describe three meshes, not one.
|
||||||
|
- **LocalDb sync:** keep site-a's replication demo; optionally add a distinct sync port for site-b to
|
||||||
|
model per-site ports (nice-to-have, not required for the gate — if added, use a different host-safe
|
||||||
|
port and its own peer address).
|
||||||
|
|
||||||
|
**Steps:**
|
||||||
|
1. Rewrite the six nodes' `Cluster__Roles__*` and `Cluster__SeedNodes__*` per the table.
|
||||||
|
2. Flip the transport-mode env vars.
|
||||||
|
3. Update the header comment + secrets comment.
|
||||||
|
4. `docker compose -f docker-dev/docker-compose.yml config` to validate YAML (no live bring-up in this
|
||||||
|
task — that is the Task 9 gate).
|
||||||
|
|
||||||
|
**Acceptance:** `docker compose config` validates; the six nodes carry per-pair self-first seeds +
|
||||||
|
cluster roles + split-correct transport modes; header comment describes three meshes.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 8: Docs sweep — status tables, caveat removal, pair-local secrets note
|
||||||
|
|
||||||
|
**Classification:** small
|
||||||
|
**Estimated implement time:** ~5 min
|
||||||
|
**Parallelizable with:** Task 7 (docs only; no build)
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- `docs/Redundancy.md` — new §"Per-cluster meshes (Phase 6)": one Primary per pair (**two+ Primaries
|
||||||
|
fleet-wide, by design**), redundancy-state singleton is now `cluster-{ClusterId}`-scoped and
|
||||||
|
spawned per driver node; remove any single-mesh caveat. Add §"Secrets replication is pair-local".
|
||||||
|
- `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Redundancy/IManualFailoverService.cs:34-35` — remove the
|
||||||
|
"Mesh-scope caveat / until the per-cluster mesh work lands" doc comment (it landed).
|
||||||
|
- AdminUI ClusterRedundancy page — remove the mesh-scope caveat text (grep the Razor for the caveat
|
||||||
|
string).
|
||||||
|
- `docs/Configuration.md` — document the `cluster-{ClusterId}` role, per-pair `Cluster:SeedNodes`
|
||||||
|
ordering, the new transport-mode defaults, and the split-topology validator.
|
||||||
|
- `docs/plans/2026-07-22-per-cluster-mesh-program.md` — mark Phase 6 **DONE** in the section + the
|
||||||
|
tracking table; note the three decisions.
|
||||||
|
- `docs/plans/2026-07-21-per-cluster-mesh-design.md` §7 — mark row 6 DONE.
|
||||||
|
- `CLAUDE.md` — a "Per-cluster mesh (Phase 6)" note under the Redundancy section: three meshes, cluster
|
||||||
|
roles, one Primary per pair, pair-local secrets, per-cluster ClusterClient, reconciler own-cluster
|
||||||
|
scoped, transport defaults flipped + split validator.
|
||||||
|
|
||||||
|
**Steps:**
|
||||||
|
1. Write the Redundancy.md sections; remove the two caveats (IManualFailoverService + Razor).
|
||||||
|
2. Update Configuration.md.
|
||||||
|
3. Update the two plan status tables + CLAUDE.md.
|
||||||
|
4. Grep for any remaining "single mesh" / "fleet-wide Primary" / "until the per-cluster mesh work
|
||||||
|
lands" copy and reconcile.
|
||||||
|
|
||||||
|
**Acceptance:** no stale single-mesh caveats remain; status tables show Phase 6 done; secrets
|
||||||
|
pair-local + cluster-role documented.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 9: Live gate — three meshes, per-pair redundancy, cross-mesh transports
|
||||||
|
|
||||||
|
**Classification:** high-risk
|
||||||
|
**Estimated implement time:** manual (controller-run against docker-dev)
|
||||||
|
**Parallelizable with:** none (final)
|
||||||
|
|
||||||
|
**Context:** the exit gate. Bring the rewritten rig up and prove every previously-live-gated behavior
|
||||||
|
per pair, plus the split itself. Gather TCP/log/CLI evidence; record in
|
||||||
|
`docs/plans/2026-07-24-mesh-phase6-live-gate.md`.
|
||||||
|
|
||||||
|
**Legs:**
|
||||||
|
1. **Three meshes form.** Each node's `Cluster.State.Members` shows **only its own pair** (2 members).
|
||||||
|
Evidence: redundancy CLI / logs per node, or `/proc/net/tcp` showing 4053 associations only within
|
||||||
|
each pair. central sees central-1/2 only; site-a sees site-a-1/2 only; site-b sees site-b-1/2 only.
|
||||||
|
2. **Two+ Primaries, one per pair.** Each pair elects its own oldest-Up driver Primary; ServiceLevel
|
||||||
|
**250/240** within each pair (Primary/Secondary), independently. Client.CLI `redundancy` per
|
||||||
|
endpoint (4840/4841 MAIN, 4842/4843 SITE-A, 4844/4845 SITE-B).
|
||||||
|
3. **Deploy reaches all clusters over per-cluster ClusterClient.** `POST :9200/api/deployments`
|
||||||
|
(X-Api-Key) seals green with all six nodes acking; central's logs show one client per cluster;
|
||||||
|
verify a SITE-A node applies the config (its address space serves).
|
||||||
|
4. **Telemetry per cluster over gRPC.** AdminUI `/alerts` + `/hosts` pills live; central dials each
|
||||||
|
site mesh's telemetry servers (2 inbound per driver node on 4056; central holds outbound per cluster).
|
||||||
|
Kill-and-reconnect a site node → its stream recovers (~5s).
|
||||||
|
5. **Reconciler quiet.** central's `ClusterNodeAddressReconciler` logs **no** `EnabledRowNotInCluster`
|
||||||
|
for SITE-A/SITE-B rows (own-cluster scoped now); MAIN drift still caught (probe with a deliberate
|
||||||
|
mismatch if practical).
|
||||||
|
6. **Secrets pair-local.** With `Secrets:Replication:Enabled=true`, a secret written on site-a-1
|
||||||
|
replicates to site-a-2 (same mesh) and does **not** reach central or site-b (separate meshes) — or,
|
||||||
|
if a live secret write is impractical, assert the DPS topic is mesh-scoped via the membership
|
||||||
|
evidence from leg 1 + a doc note. No errors from the secrets actor about unreachable peers.
|
||||||
|
7. **Split validator.** Flip one site node to `MeshTransport__Mode=Dps` → it refuses to boot with the
|
||||||
|
Task 5 message. Restore.
|
||||||
|
8. **Failover within a pair** (bridges toward Phase 7): kill a site pair's Primary → the Secondary
|
||||||
|
becomes Primary (250) and the pair survives alone (auto-down 1-vs-1 — the deferred Phase 0a gate is
|
||||||
|
now testable; run it if time permits and note the result for Phase 7).
|
||||||
|
|
||||||
|
**Record:** `docs/plans/2026-07-24-mesh-phase6-live-gate.md` with per-leg evidence + any findings.
|
||||||
|
**Cleanup:** restore the rig to a coherent committed state.
|
||||||
|
|
||||||
|
**Acceptance:** all legs pass (or any deviation is documented + justified as in prior phases). Phase 6
|
||||||
|
exit gate MET.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Risk register (carry from design §8 + program)
|
||||||
|
|
||||||
|
- LocalDb is load-bearing for config (unchanged; #485 class).
|
||||||
|
- **Two+ Primaries fleet-wide is now correct, by design** — anything asserting "one Primary" is stale.
|
||||||
|
- Partial rollout hazard: a node whose mesh is split has pair-local `DriverMemberCount`, while a
|
||||||
|
not-yet-split node counts fleet-wide — do not run the fleet half-split in production without the
|
||||||
|
drill (Phase 7).
|
||||||
|
- Co-located VM loss takes out one node of **both** products (Phase 7 drill item; out of scope here).
|
||||||
|
- Frame size: deploy path stays payload-free — keep it that way.
|
||||||
@@ -0,0 +1,24 @@
|
|||||||
|
{
|
||||||
|
"planPath": "docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md",
|
||||||
|
"program": "per-cluster-mesh",
|
||||||
|
"phase": 6,
|
||||||
|
"branch": "feat/mesh-phase6",
|
||||||
|
"decisions": {
|
||||||
|
"secrets": "pair-local Akka (no SQL-hub; sites have no SQL)",
|
||||||
|
"dpsBranches": "flip rig + split-topology validator; keep DPS code. NOTE: compiled defaults NOT flipped (Task 6) — blast-radius assessment showed a flip hard-fails all 14 Host.IntegrationTests + all 5 appsettings profiles + docker-dev site nodes; the Task 5 validator already enforces split-correctness fail-closed on cluster-role nodes, which is the actual intent.",
|
||||||
|
"rigScope": "OtOpcUa-only three 2-node meshes on one shared network"
|
||||||
|
},
|
||||||
|
"tasks": [
|
||||||
|
{"id": 0, "subject": "Task 0: RoleParser accepts cluster-{ClusterId}; consolidate driver literal", "classification": "small", "status": "completed", "commit": "87ff00fd"},
|
||||||
|
{"id": 1, "subject": "Task 1: ClusterRoleInfo exposes node's own ClusterRole/ClusterId", "classification": "small", "status": "completed", "commit": "2d2f1dec", "note": "interface in Commons/Interfaces/IClusterRoleInfo.cs; ctor (ActorSystem, IOptions<AkkaClusterOptions>, ILogger)"},
|
||||||
|
{"id": 2, "subject": "Task 2: re-home RedundancyStateActor to cluster-role singleton on drivers", "classification": "high-risk", "status": "completed", "commit": "a4d3ab40", "note": "re-home a2c5d57f + review fixes a4d3ab40 (comments + driver-fallback registry test + fixed a cross-task admin-boot NRE race); 14/14 tests"},
|
||||||
|
{"id": 3, "subject": "Task 3: re-scope ClusterNodeAddressReconciler to own-cluster rows", "classification": "standard", "status": "completed", "commit": "1ce2042e", "note": "impl 2535df01 + review-fix 1ce2042e (observed ALSO scoped by cluster role via testable FilterOwnClusterDriverMembers helper; 17/17)"},
|
||||||
|
{"id": 4, "subject": "Task 4: one ClusterClient per cluster in CentralCommunicationActor", "classification": "high-risk", "status": "completed", "commit": "6531ec19", "note": "impl 6531ec19 + comment fix d98b3264; two-mesh boundary test proves partitioned delivery, red-before-green"},
|
||||||
|
{"id": 5, "subject": "Task 5: split-topology startup validator (cluster-role => non-DPS)", "classification": "standard", "status": "completed", "commit": "344b0d33", "note": "impl a886d5e6 + review-fix 344b0d33 (resolve EFFECTIVE roles incl OTOPCUA_ROLES fallback matching Program.cs, case-normalize -> guardrail can't be defeated by role-source/case mismatch); 16/16 filter, 124/124 Cluster.Tests"},
|
||||||
|
{"id": 6, "subject": "Task 6: compiled transport-mode defaults", "classification": "small", "status": "resolved-no-flip", "note": "DECIDED not to flip — blast-radius assessment: high-risk (breaks all integration tests + appsettings profiles + docker-dev site nodes), low-value (Task 5 validator already enforces split-correctness fail-closed). Documented in plan Task 6 + Configuration.md."},
|
||||||
|
{"id": 7, "subject": "Task 7: rewrite docker-dev rig into three 2-node meshes", "classification": "standard", "status": "completed", "commit": "741ab97b", "note": "3 meshes, per-pair self-first seeds, cluster roles, transports flipped (ClusterClient/Grpc), pair-local secrets note; docker compose config validates; seed SQL already correct (GrpcPort=4056)"},
|
||||||
|
{"id": 8, "subject": "Task 8: docs sweep — status tables, caveat removal, pair-local secrets note", "classification": "small", "status": "completed", "commit": "8a173bba", "note": "all 7 edits (Redundancy/Config/CLAUDE/2 plans/IManualFailoverService/ClusterRedundancy.razor); subagent died at commit step, edits reviewed+committed by controller; full-solution build clean (0 errors)"},
|
||||||
|
{"id": 9, "subject": "Task 9: live gate — three meshes, per-pair redundancy, cross-mesh transports","classification": "high-risk", "status": "in-progress", "note": "local OrbStack docker up; full-solution build verified clean; building images"}
|
||||||
|
],
|
||||||
|
"lastUpdated": "2026-07-24T00:00:00Z"
|
||||||
|
}
|
||||||
@@ -20,6 +20,8 @@ public sealed class ClusterRoleInfo : IClusterRoleInfo, IDisposable
|
|||||||
private readonly ILogger<ClusterRoleInfo> _logger;
|
private readonly ILogger<ClusterRoleInfo> _logger;
|
||||||
private readonly CommonsNodeId _localNode;
|
private readonly CommonsNodeId _localNode;
|
||||||
private readonly HashSet<string> _localRoles;
|
private readonly HashSet<string> _localRoles;
|
||||||
|
private readonly string? _clusterRole;
|
||||||
|
private readonly string? _clusterId;
|
||||||
private readonly object _lock = new();
|
private readonly object _lock = new();
|
||||||
private readonly Dictionary<string, Member?> _roleLeaders = new(StringComparer.Ordinal);
|
private readonly Dictionary<string, Member?> _roleLeaders = new(StringComparer.Ordinal);
|
||||||
private readonly Dictionary<string, HashSet<Member>> _membersByRole = new(StringComparer.Ordinal);
|
private readonly Dictionary<string, HashSet<Member>> _membersByRole = new(StringComparer.Ordinal);
|
||||||
@@ -39,6 +41,23 @@ public sealed class ClusterRoleInfo : IClusterRoleInfo, IDisposable
|
|||||||
_localNode = CommonsNodeId.Parse($"{options.Value.PublicHostname}:{options.Value.Port}");
|
_localNode = CommonsNodeId.Parse($"{options.Value.PublicHostname}:{options.Value.Port}");
|
||||||
_localRoles = new HashSet<string>(options.Value.Roles, StringComparer.Ordinal);
|
_localRoles = new HashSet<string>(options.Value.Roles, StringComparer.Ordinal);
|
||||||
|
|
||||||
|
// Derived from the node's OWN configured roles (not live Cluster.State) so this identity is
|
||||||
|
// available at host-build time, before the cluster forms. Iterate the ORIGINAL array — not
|
||||||
|
// _localRoles — because first-wins requires configuration order, which HashSet does not
|
||||||
|
// guarantee.
|
||||||
|
var clusterRoles = options.Value.Roles.Where(RoleParser.IsClusterRole).ToArray();
|
||||||
|
if (clusterRoles.Length > 0)
|
||||||
|
{
|
||||||
|
_clusterRole = clusterRoles[0];
|
||||||
|
_clusterId = RoleParser.ClusterIdFromRole(_clusterRole);
|
||||||
|
if (clusterRoles.Length > 1)
|
||||||
|
{
|
||||||
|
_logger.LogWarning(
|
||||||
|
"Node configured with {Count} cluster-scoped roles ({Roles}); using the first ({ClusterRole})",
|
||||||
|
clusterRoles.Length, string.Join(", ", clusterRoles), _clusterRole);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
SeedFromCurrentState();
|
SeedFromCurrentState();
|
||||||
_subscriber = system.ActorOf(Props.Create(() => new SubscriberActor(this)), "clusterroleinfo-subscriber");
|
_subscriber = system.ActorOf(Props.Create(() => new SubscriberActor(this)), "clusterroleinfo-subscriber");
|
||||||
}
|
}
|
||||||
@@ -49,6 +68,12 @@ public sealed class ClusterRoleInfo : IClusterRoleInfo, IDisposable
|
|||||||
/// <inheritdoc />
|
/// <inheritdoc />
|
||||||
public IReadOnlySet<string> LocalRoles => _localRoles;
|
public IReadOnlySet<string> LocalRoles => _localRoles;
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public string? ClusterRole => _clusterRole;
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public string? ClusterId => _clusterId;
|
||||||
|
|
||||||
/// <inheritdoc />
|
/// <inheritdoc />
|
||||||
public bool HasRole(string role) => _localRoles.Contains(role);
|
public bool HasRole(string role) => _localRoles.Contains(role);
|
||||||
|
|
||||||
@@ -186,7 +211,9 @@ public sealed class ClusterRoleInfo : IClusterRoleInfo, IDisposable
|
|||||||
Receive<ClusterEvent.IMemberEvent>(e => owner.HandleMemberEvent(e));
|
Receive<ClusterEvent.IMemberEvent>(e => owner.HandleMemberEvent(e));
|
||||||
Receive<ClusterEvent.RoleLeaderChanged>(e => owner.HandleRoleLeaderChanged(e));
|
Receive<ClusterEvent.RoleLeaderChanged>(e => owner.HandleRoleLeaderChanged(e));
|
||||||
// LeaderChanged is intentionally a no-op here: ServiceLevel lives in RedundancyStateActor
|
// LeaderChanged is intentionally a no-op here: ServiceLevel lives in RedundancyStateActor
|
||||||
// (admin-role singleton) and has no consumer on this ClusterRoleInfo path by design.
|
// (per-cluster mesh Phase 6: a cluster-{ClusterId}-scoped singleton spawned on every
|
||||||
|
// driver node, no longer the admin-role singleton it once was) and has no consumer on this
|
||||||
|
// ClusterRoleInfo path by design.
|
||||||
Receive<ClusterEvent.LeaderChanged>(_ => { });
|
Receive<ClusterEvent.LeaderChanged>(_ => { });
|
||||||
Receive<ClusterEvent.CurrentClusterState>(_ => { /* seeded from initial snapshot */ });
|
Receive<ClusterEvent.CurrentClusterState>(_ => { /* seeded from initial snapshot */ });
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -2,11 +2,36 @@ namespace ZB.MOM.WW.OtOpcUa.Cluster;
|
|||||||
|
|
||||||
public static class RoleParser
|
public static class RoleParser
|
||||||
{
|
{
|
||||||
|
/// <summary>The admin-role name.</summary>
|
||||||
|
public const string Admin = "admin";
|
||||||
|
|
||||||
|
/// <summary>The driver-role name. Single source of truth — downstream <c>DriverRole</c>
|
||||||
|
/// constants (<c>RedundancyStateActor</c>, <c>ClusterNodeAddressReconcilerActor</c>,
|
||||||
|
/// <c>ServiceCollectionExtensions</c>) alias this value rather than duplicating the literal.</summary>
|
||||||
|
public const string Driver = "driver";
|
||||||
|
|
||||||
|
/// <summary>The dev-role name.</summary>
|
||||||
|
public const string Dev = "dev";
|
||||||
|
|
||||||
|
/// <summary>Prefix identifying a cluster-scoped role, e.g. <c>cluster-SITE-A</c> (per-cluster mesh, Phase 6).</summary>
|
||||||
|
public const string ClusterRolePrefix = "cluster-";
|
||||||
|
|
||||||
private static readonly HashSet<string> Allowed = new(StringComparer.Ordinal)
|
private static readonly HashSet<string> Allowed = new(StringComparer.Ordinal)
|
||||||
{
|
{
|
||||||
"admin", "driver", "dev",
|
Admin, Driver, Dev,
|
||||||
};
|
};
|
||||||
|
|
||||||
|
/// <summary>Returns true when <paramref name="role"/> is a cluster-scoped role — starts with
|
||||||
|
/// <see cref="ClusterRolePrefix"/> and carries a non-empty ClusterId suffix.</summary>
|
||||||
|
/// <param name="role">The role name to test.</param>
|
||||||
|
public static bool IsClusterRole(string role) =>
|
||||||
|
role.StartsWith(ClusterRolePrefix, StringComparison.Ordinal) && role.Length > ClusterRolePrefix.Length;
|
||||||
|
|
||||||
|
/// <summary>Extracts the ClusterId from a cluster-scoped role, e.g. <c>cluster-SITE-A</c> → <c>SITE-A</c>.
|
||||||
|
/// Assumes <see cref="IsClusterRole"/> is true for <paramref name="role"/>.</summary>
|
||||||
|
/// <param name="role">The cluster-scoped role name.</param>
|
||||||
|
public static string ClusterIdFromRole(string role) => role[ClusterRolePrefix.Length..];
|
||||||
|
|
||||||
/// <summary>Parses a comma-separated string of role names into a validated array.</summary>
|
/// <summary>Parses a comma-separated string of role names into a validated array.</summary>
|
||||||
/// <param name="raw">The raw role string to parse.</param>
|
/// <param name="raw">The raw role string to parse.</param>
|
||||||
/// <returns>The distinct, lower-cased, validated role names.</returns>
|
/// <returns>The distinct, lower-cased, validated role names.</returns>
|
||||||
@@ -22,9 +47,10 @@ public static class RoleParser
|
|||||||
|
|
||||||
foreach (var r in roles)
|
foreach (var r in roles)
|
||||||
{
|
{
|
||||||
if (!Allowed.Contains(r))
|
if (!Allowed.Contains(r) && !IsClusterRole(r))
|
||||||
throw new ArgumentException(
|
throw new ArgumentException(
|
||||||
$"Unknown role '{r}'. Allowed: {string.Join(", ", Allowed)}.", nameof(raw));
|
$"Unknown role '{r}'. Allowed: {string.Join(", ", Allowed)}, or '{ClusterRolePrefix}<ClusterId>'.",
|
||||||
|
nameof(raw));
|
||||||
}
|
}
|
||||||
|
|
||||||
return roles;
|
return roles;
|
||||||
|
|||||||
@@ -56,6 +56,14 @@ public static class ServiceCollectionExtensions
|
|||||||
services.AddValidatedOptions<TelemetryDialOptions, TelemetryDialOptionsValidator>(
|
services.AddValidatedOptions<TelemetryDialOptions, TelemetryDialOptionsValidator>(
|
||||||
configuration, TelemetryDialOptions.SectionName);
|
configuration, TelemetryDialOptions.SectionName);
|
||||||
|
|
||||||
|
// Per-cluster mesh Phase 6: a node carrying a cluster-{ClusterId} role (the split topology)
|
||||||
|
// must use the mesh-crossing transports — the DPS dark-switch branches cannot cross a split
|
||||||
|
// mesh and would deliver nothing silently. Hung on MeshTransportOptions (which already has
|
||||||
|
// ValidateOnStart from its own registration above) so it fires at boot alongside the sibling
|
||||||
|
// validators; TryAddEnumerable inside AddValidatedOptions keeps the second validator additive.
|
||||||
|
services.AddValidatedOptions<MeshTransportOptions, SplitTopologyTransportValidator>(
|
||||||
|
configuration, MeshTransportOptions.SectionName);
|
||||||
|
|
||||||
services.AddSingleton<IClusterRoleInfo, ClusterRoleInfo>();
|
services.AddSingleton<IClusterRoleInfo, ClusterRoleInfo>();
|
||||||
|
|
||||||
return services;
|
return services;
|
||||||
|
|||||||
@@ -0,0 +1,137 @@
|
|||||||
|
using Microsoft.Extensions.Configuration;
|
||||||
|
using Microsoft.Extensions.Options;
|
||||||
|
|
||||||
|
namespace ZB.MOM.WW.OtOpcUa.Cluster;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Fails the host at startup when a node configured for the <b>split topology</b> (per-cluster
|
||||||
|
/// mesh, Phase 6) is left on a DistributedPubSub transport mode that cannot cross the mesh
|
||||||
|
/// boundary.
|
||||||
|
/// </summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// <para>
|
||||||
|
/// Phase 6 splits the single fleet mesh into one 2-node mesh per application cluster. The
|
||||||
|
/// Phase 2/3/5 dark switches keep their DPS branches alive, but DPS is mesh-wide — it delivers
|
||||||
|
/// nothing between a central node and a site node that no longer share a gossip ring. The
|
||||||
|
/// marker of the split topology is a <c>cluster-{ClusterId}</c> role
|
||||||
|
/// (<see cref="RoleParser.IsClusterRole"/>); those roles exist only post-Phase-6. A node that
|
||||||
|
/// carries one <b>must</b> use the mesh-crossing transports, and this validator refuses to
|
||||||
|
/// boot it otherwise — the alternative is a node that runs, listens, and silently delivers no
|
||||||
|
/// commands or telemetry, which has no stack trace and no failing node to point at.
|
||||||
|
/// </para>
|
||||||
|
/// <para>
|
||||||
|
/// <b>The rule is conditional and must stay that way.</b> It fires only when this node's
|
||||||
|
/// <i>effective</i> Akka roles contain at least one cluster-scoped role — resolved the same
|
||||||
|
/// way the node resolves them (<c>Cluster:Roles</c>, falling back to <c>OTOPCUA_ROLES</c> when
|
||||||
|
/// that is empty, normalized case-insensitively). A legacy / single-mesh / test node has none,
|
||||||
|
/// so DPS is still legitimate and the validator passes unconditionally.
|
||||||
|
/// </para>
|
||||||
|
/// <para>
|
||||||
|
/// <b>Not validated here:</b> <c>ConfigSource:Mode</c>. Phase 4's
|
||||||
|
/// <see cref="ConfigSourceOptionsValidator"/> already requires <c>FetchAndCache</c> on a
|
||||||
|
/// driver-only node, and a fused node legitimately stays <c>Direct</c> — duplicating that here
|
||||||
|
/// would only risk the two rules drifting apart.
|
||||||
|
/// </para>
|
||||||
|
/// </remarks>
|
||||||
|
public sealed class SplitTopologyTransportValidator : IValidateOptions<MeshTransportOptions>
|
||||||
|
{
|
||||||
|
private readonly IConfiguration _configuration;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// DI-constructed by <c>AddValidatedOptions</c> (a plain <c>AddSingleton</c>), so a
|
||||||
|
/// constructor dependency is safe here. <see cref="IConfiguration"/> — not
|
||||||
|
/// <c>IClusterRoleInfo</c> — is the source of this node's roles and the three transport modes:
|
||||||
|
/// <c>IClusterRoleInfo</c>'s implementation needs the <c>ActorSystem</c>, which does not exist
|
||||||
|
/// yet at <c>ValidateOnStart</c> time. Reading the modes from configuration too (rather than
|
||||||
|
/// from three injected <c>IOptions</c>) mirrors how the sibling validators cross-read
|
||||||
|
/// <c>Cluster:Roles</c>, and avoids a service-resolution ordering dependency at validation
|
||||||
|
/// time.
|
||||||
|
/// </summary>
|
||||||
|
public SplitTopologyTransportValidator(IConfiguration configuration)
|
||||||
|
{
|
||||||
|
_configuration = configuration;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public ValidateOptionsResult Validate(string? name, MeshTransportOptions options)
|
||||||
|
{
|
||||||
|
// The options instance is intentionally unused: every value this rule depends on is cross-read
|
||||||
|
// from IConfiguration, the same way ConfigSourceOptionsValidator reads Cluster:Roles.
|
||||||
|
//
|
||||||
|
// Resolve the EFFECTIVE Akka roles exactly as the node does, or this "fail loud" rule fires on
|
||||||
|
// the wrong role view and passes silently on the two shapes it exists to catch:
|
||||||
|
// (a) Roles-source divergence. AkkaClusterOptions.Roles binds Cluster:Roles but falls back to
|
||||||
|
// OTOPCUA_ROLES (via RoleParser.Parse — the same call Program.cs makes) when Cluster:Roles
|
||||||
|
// is empty. A node configured with only OTOPCUA_ROLES=driver,cluster-SITE-A genuinely
|
||||||
|
// carries the cluster role in Akka, so it must be validated too.
|
||||||
|
// (b) Case. The Cluster:Roles bind path is verbatim (no lowercasing), whereas RoleParser.Parse
|
||||||
|
// lowercases the OTOPCUA_ROLES path. Normalize BOTH here (trim + ToLowerInvariant) so
|
||||||
|
// 'Cluster-SITE-A' / 'Driver' / 'Admin' cannot slip past the ordinal IsClusterRole /
|
||||||
|
// driver / admin checks.
|
||||||
|
var configured = _configuration.GetSection("Cluster:Roles").Get<string[]>() ?? Array.Empty<string>();
|
||||||
|
var effective = configured.Length > 0
|
||||||
|
? configured
|
||||||
|
: RoleParser.Parse(Environment.GetEnvironmentVariable("OTOPCUA_ROLES"));
|
||||||
|
|
||||||
|
var normalized = Array.ConvertAll(effective, r => r.Trim().ToLowerInvariant());
|
||||||
|
|
||||||
|
var clusterRoleIndex = Array.FindIndex(normalized, RoleParser.IsClusterRole);
|
||||||
|
if (clusterRoleIndex < 0)
|
||||||
|
{
|
||||||
|
// No cluster-{ClusterId} role → not the split topology → DPS is legitimate. This is the
|
||||||
|
// exemption that keeps the validator a no-op for every legacy / single-mesh / test node.
|
||||||
|
return ValidateOptionsResult.Success;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Classification is normalized (above); the ClusterId in the message is taken from the
|
||||||
|
// operator's original spelling so it names what they actually wrote.
|
||||||
|
var clusterId = RoleParser.ClusterIdFromRole(effective[clusterRoleIndex].Trim());
|
||||||
|
var isDriver = Array.IndexOf(normalized, RoleParser.Driver) >= 0;
|
||||||
|
var isAdmin = Array.IndexOf(normalized, RoleParser.Admin) >= 0;
|
||||||
|
|
||||||
|
var errors = new List<string>();
|
||||||
|
|
||||||
|
// Modes are read from configuration directly; an unset key inherits the class default (Dps for
|
||||||
|
// all three), so a split node that simply never set the key fails exactly as an explicit Dps
|
||||||
|
// would — which is the point.
|
||||||
|
var meshMode = _configuration["MeshTransport:Mode"] ?? MeshTransportOptions.ModeDps;
|
||||||
|
var telemetryMode = _configuration["Telemetry:Mode"] ?? TelemetryOptions.ModeDps;
|
||||||
|
var telemetryDialMode = _configuration["TelemetryDial:Mode"] ?? TelemetryDialOptions.ModeDps;
|
||||||
|
|
||||||
|
if (!string.Equals(meshMode, MeshTransportOptions.ModeClusterClient, StringComparison.OrdinalIgnoreCase))
|
||||||
|
{
|
||||||
|
errors.Add(
|
||||||
|
$"Cluster:Roles declares a cluster-{clusterId} role (split topology) but "
|
||||||
|
+ $"{MeshTransportOptions.SectionName}:{nameof(MeshTransportOptions.Mode)} is "
|
||||||
|
+ $"'{meshMode}'; a split mesh cannot use DistributedPubSub — set "
|
||||||
|
+ $"{MeshTransportOptions.SectionName}:{nameof(MeshTransportOptions.Mode)}="
|
||||||
|
+ $"{MeshTransportOptions.ModeClusterClient}.");
|
||||||
|
}
|
||||||
|
|
||||||
|
if (isDriver
|
||||||
|
&& !string.Equals(telemetryMode, TelemetryOptions.ModeGrpc, StringComparison.OrdinalIgnoreCase))
|
||||||
|
{
|
||||||
|
errors.Add(
|
||||||
|
$"Cluster:Roles declares a cluster-{clusterId} role (split topology) with 'driver' but "
|
||||||
|
+ $"{TelemetryOptions.SectionName}:{nameof(TelemetryOptions.Mode)} is "
|
||||||
|
+ $"'{telemetryMode}'; a split-topology driver node hosts the telemetry gRPC server — "
|
||||||
|
+ $"set {TelemetryOptions.SectionName}:{nameof(TelemetryOptions.Mode)}="
|
||||||
|
+ $"{TelemetryOptions.ModeGrpc}.");
|
||||||
|
}
|
||||||
|
|
||||||
|
if (isAdmin
|
||||||
|
&& !string.Equals(telemetryDialMode, TelemetryDialOptions.ModeGrpc, StringComparison.OrdinalIgnoreCase))
|
||||||
|
{
|
||||||
|
errors.Add(
|
||||||
|
$"Cluster:Roles declares a cluster-{clusterId} role (split topology) with 'admin' but "
|
||||||
|
+ $"{TelemetryDialOptions.SectionName}:{nameof(TelemetryDialOptions.Mode)} is "
|
||||||
|
+ $"'{telemetryDialMode}'; a split-topology central node dials nodes for telemetry — set "
|
||||||
|
+ $"{TelemetryDialOptions.SectionName}:{nameof(TelemetryDialOptions.Mode)}="
|
||||||
|
+ $"{TelemetryDialOptions.ModeGrpc}.");
|
||||||
|
}
|
||||||
|
|
||||||
|
return errors.Count == 0
|
||||||
|
? ValidateOptionsResult.Success
|
||||||
|
: ValidateOptionsResult.Fail(string.Join(" ", errors));
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -14,6 +14,21 @@ public interface IClusterRoleInfo
|
|||||||
NodeId LocalNode { get; }
|
NodeId LocalNode { get; }
|
||||||
/// <summary>Gets the set of roles assigned to the local node.</summary>
|
/// <summary>Gets the set of roles assigned to the local node.</summary>
|
||||||
IReadOnlySet<string> LocalRoles { get; }
|
IReadOnlySet<string> LocalRoles { get; }
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Gets the local node's cluster-scoped role (e.g. <c>cluster-SITE-A</c>), or <c>null</c> when
|
||||||
|
/// the node carries none. Derived from the node's OWN configured roles at construction time —
|
||||||
|
/// NOT from live cluster membership — so it is available before the cluster forms (per-cluster
|
||||||
|
/// mesh, Phase 6). When more than one cluster-scoped role is configured, the first one wins.
|
||||||
|
/// </summary>
|
||||||
|
string? ClusterRole { get; }
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Gets the ClusterId extracted from <see cref="ClusterRole"/> (e.g. <c>SITE-A</c>), or
|
||||||
|
/// <c>null</c> when the node carries no cluster-scoped role.
|
||||||
|
/// </summary>
|
||||||
|
string? ClusterId { get; }
|
||||||
|
|
||||||
/// <summary>Checks if the local node has the specified role.</summary>
|
/// <summary>Checks if the local node has the specified role.</summary>
|
||||||
/// <param name="role">Role name to check.</param>
|
/// <param name="role">Role name to check.</param>
|
||||||
/// <returns>True if the local node has the role; otherwise, false.</returns>
|
/// <returns>True if the local node has the role; otherwise, false.</returns>
|
||||||
|
|||||||
+4
-4
@@ -68,10 +68,10 @@ else
|
|||||||
</div>
|
</div>
|
||||||
|
|
||||||
<p class="text-muted small mt-2 mb-0">
|
<p class="text-muted small mt-2 mb-0">
|
||||||
Read live from cluster state on this node. <strong>Mesh-wide scope:</strong> the Primary is
|
Read live from cluster state on this node. The Primary shown is
|
||||||
elected once per Akka mesh, not per cluster row — in the current single-mesh topology this
|
<strong>this application <span class="mono">Cluster</span>'s own Primary</strong> — since
|
||||||
acts on the whole mesh's Primary, which may be a node of another
|
the per-cluster mesh split, each cluster's redundant pair runs its own independent 2-node
|
||||||
<span class="mono">Cluster</span>. See <span class="mono">docs/Redundancy.md</span>.
|
Akka mesh and elects its own Primary. See <span class="mono">docs/Redundancy.md</span>.
|
||||||
</p>
|
</p>
|
||||||
|
|
||||||
<AuthorizeView Policy="@ManualFailoverPageModel.RequiredPolicy">
|
<AuthorizeView Policy="@ManualFailoverPageModel.RequiredPolicy">
|
||||||
|
|||||||
+126
-57
@@ -25,8 +25,11 @@ namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
|
|||||||
/// rotation would fail silently against the other.
|
/// rotation would fail silently against the other.
|
||||||
/// </para>
|
/// </para>
|
||||||
/// <para>
|
/// <para>
|
||||||
/// <b>Exactly one ClusterClient, fleet-wide.</b> See <see cref="RebuildClient"/> — this is
|
/// <b>One ClusterClient per application <c>Cluster</c> (Phase 6).</b> The fleet is now one
|
||||||
/// the single most important thing to understand before changing this actor.
|
/// Akka mesh per cluster, so a single fleet-wide client's <c>SendToAll</c> would reach only
|
||||||
|
/// the one mesh whose receptionist answered. This actor therefore holds a client per cluster
|
||||||
|
/// and fans <c>SendToAll</c> across all of them. See <see cref="RebuildClient"/> — this is the
|
||||||
|
/// single most important thing to understand before changing this actor.
|
||||||
/// </para>
|
/// </para>
|
||||||
/// </remarks>
|
/// </remarks>
|
||||||
public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
|
public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
|
||||||
@@ -40,8 +43,12 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
|
|||||||
private readonly ILoggingAdapter _log = Context.GetLogger();
|
private readonly ILoggingAdapter _log = Context.GetLogger();
|
||||||
private readonly bool _useClusterClient;
|
private readonly bool _useClusterClient;
|
||||||
|
|
||||||
private IActorRef? _client;
|
// One client per application Cluster, keyed by ClusterId. A ClusterClient reaches only the mesh
|
||||||
private ImmutableHashSet<string> _currentContacts = ImmutableHashSet<string>.Empty;
|
// whose receptionist it dialled, and after Phase 6 every cluster is its own mesh — so central must
|
||||||
|
// hold a client per cluster and fan SendToAll across all of them.
|
||||||
|
private readonly Dictionary<string, (IActorRef Client, ImmutableHashSet<string> Contacts)> _clients =
|
||||||
|
new();
|
||||||
|
|
||||||
private CancellationTokenSource _lifecycle = new();
|
private CancellationTokenSource _lifecycle = new();
|
||||||
|
|
||||||
/// <summary>Gets the timer scheduler driving the periodic contact refresh.</summary>
|
/// <summary>Gets the timer scheduler driving the periodic contact refresh.</summary>
|
||||||
@@ -121,6 +128,9 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
|
|||||||
/// <inheritdoc />
|
/// <inheritdoc />
|
||||||
protected override void PostStop()
|
protected override void PostStop()
|
||||||
{
|
{
|
||||||
|
foreach (var (client, _) in _clients.Values) Context.Stop(client);
|
||||||
|
_clients.Clear();
|
||||||
|
|
||||||
_lifecycle.Cancel();
|
_lifecycle.Cancel();
|
||||||
_lifecycle.Dispose();
|
_lifecycle.Dispose();
|
||||||
}
|
}
|
||||||
@@ -133,7 +143,7 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
|
|||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (_client is null)
|
if (_clients.Count == 0)
|
||||||
{
|
{
|
||||||
// App-level drop, matching the sister project's decision and this repo's buffer-size = 0.
|
// App-level drop, matching the sister project's decision and this repo's buffer-size = 0.
|
||||||
// The command is NOT queued and will NOT be retried: a deploy fails at the coordinator's
|
// The command is NOT queued and will NOT be retried: a deploy fails at the coordinator's
|
||||||
@@ -141,19 +151,21 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
|
|||||||
// command arriving minutes late against a config the DB already recorded as failed.
|
// command arriving minutes late against a config the DB already recorded as failed.
|
||||||
_log.Warning(
|
_log.Warning(
|
||||||
"No mesh ClusterClient — dropping {MessageType}. The last contact refresh produced "
|
"No mesh ClusterClient — dropping {MessageType}. The last contact refresh produced "
|
||||||
+ "no usable contact point from the enabled, non-maintenance ClusterNode rows; the "
|
+ "no usable contact point from the enabled, non-maintenance ClusterNode rows in any "
|
||||||
+ "command is not buffered and will not be retried",
|
+ "cluster; the command is not buffered and will not be retried",
|
||||||
cmd.Message.GetType().Name);
|
cmd.Message.GetType().Name);
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
// SendToAll, NEVER Send. Today's DPS publish reaches EVERY DriverHostActor — there is no
|
// SendToAll, NEVER Send, fanned across EVERY cluster's client. Today's DPS publish reaches
|
||||||
// ClusterId or node filter on the node side at all; scoping happens later, inside the
|
// EVERY DriverHostActor — there is no ClusterId or node filter on the node side at all; scoping
|
||||||
// artifact (DeploymentArtifact.ResolveClusterScope). ClusterClient.Send delivers to exactly
|
// happens later, inside the artifact (DeploymentArtifact.ResolveClusterScope). Each ClusterClient
|
||||||
// ONE registered actor, which would deploy to a single node of the fleet while every other
|
// reaches only the one mesh it dialled, so central must fan across all of them to preserve the
|
||||||
// node silently kept its old configuration — and the deployment would still seal green if
|
// fleet-wide reach DPS gives. ClusterClient.Send delivers to exactly ONE registered actor, which
|
||||||
// the ack set happened to be satisfied.
|
// would deploy to a single node while every other node silently kept its old configuration — and
|
||||||
_client.Tell(new ClusterClient.SendToAll(MeshPaths.NodeCommunication, cmd.Message));
|
// the deployment would still seal green if the ack set happened to be satisfied.
|
||||||
|
foreach (var (client, _) in _clients.Values)
|
||||||
|
client.Tell(new ClusterClient.SendToAll(MeshPaths.NodeCommunication, cmd.Message));
|
||||||
}
|
}
|
||||||
|
|
||||||
private void HandleApplyAck(ApplyAck ack)
|
private void HandleApplyAck(ApplyAck ack)
|
||||||
@@ -198,11 +210,14 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
|
|||||||
var rows = await db.ClusterNodes
|
var rows = await db.ClusterNodes
|
||||||
.AsNoTracking()
|
.AsNoTracking()
|
||||||
.Where(n => n.Enabled && !n.MaintenanceMode)
|
.Where(n => n.Enabled && !n.MaintenanceMode)
|
||||||
.Select(n => new { n.NodeId, n.Host, n.AkkaPort })
|
.Select(n => new { n.NodeId, n.ClusterId, n.Host, n.AkkaPort })
|
||||||
.ToListAsync(ct)
|
.ToListAsync(ct)
|
||||||
.ConfigureAwait(false);
|
.ConfigureAwait(false);
|
||||||
|
|
||||||
var contacts = new List<string>(rows.Count);
|
// Grouped BY ClusterId: central now dials one ClusterClient per application Cluster, so the
|
||||||
|
// contact set must be partitioned the same way. A row's contacts join only its own cluster's
|
||||||
|
// client.
|
||||||
|
var byCluster = new Dictionary<string, List<string>>();
|
||||||
var malformed = new List<string>();
|
var malformed = new List<string>();
|
||||||
foreach (var row in rows)
|
foreach (var row in rows)
|
||||||
{
|
{
|
||||||
@@ -212,11 +227,21 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
|
|||||||
// not abort the whole refresh and leave the contact set half-built — the sister
|
// not abort the whole refresh and leave the contact set half-built — the sister
|
||||||
// project shipped that bug and its regression test deliberately orders the bad row
|
// project shipped that bug and its regression test deliberately orders the bad row
|
||||||
// first.
|
// first.
|
||||||
if (ActorPath.TryParse(address, out _)) contacts.Add(address);
|
if (ActorPath.TryParse(address, out _))
|
||||||
else malformed.Add($"{row.NodeId} -> {address}");
|
{
|
||||||
|
if (!byCluster.TryGetValue(row.ClusterId, out var list))
|
||||||
|
byCluster[row.ClusterId] = list = new List<string>();
|
||||||
|
list.Add(address);
|
||||||
|
}
|
||||||
|
else
|
||||||
|
{
|
||||||
|
malformed.Add($"{row.NodeId} ({row.ClusterId}) -> {address}");
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
return new ContactsLoaded(contacts, malformed);
|
var contactsByCluster = byCluster.ToDictionary(
|
||||||
|
kv => kv.Key, kv => (IReadOnlyList<string>)kv.Value);
|
||||||
|
return new ContactsLoaded(contactsByCluster, malformed);
|
||||||
}, ct).PipeTo(self);
|
}, ct).PipeTo(self);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -229,70 +254,111 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
|
|||||||
+ "refresh (other nodes are unaffected). Check the row's Host and AkkaPort", bad);
|
+ "refresh (other nodes are unaffected). Check the row's Host and AkkaPort", bad);
|
||||||
}
|
}
|
||||||
|
|
||||||
var contacts = msg.Contacts.ToImmutableHashSet();
|
var byCluster = msg.ContactsByCluster.ToDictionary(
|
||||||
|
kv => kv.Key, kv => kv.Value.ToImmutableHashSet());
|
||||||
|
|
||||||
if (contacts.IsEmpty)
|
// Clusters that dropped out of the DB entirely, or whose only rows were malformed this refresh:
|
||||||
|
// stop and forget their clients so a vanished cluster stops receiving. LoadContactsFromDb only
|
||||||
|
// creates a ByCluster key when at least one of its rows parsed, so an all-malformed cluster has
|
||||||
|
// NO key at all and is caught here by the absent-key (!TryGetValue) arm — not by an empty value.
|
||||||
|
var gone = _clients.Keys
|
||||||
|
.Where(id => !byCluster.TryGetValue(id, out var c) || c.IsEmpty)
|
||||||
|
.ToList();
|
||||||
|
foreach (var clusterId in gone)
|
||||||
{
|
{
|
||||||
_log.Warning(
|
_log.Info(
|
||||||
"No usable ClusterNode contact points; the mesh ClusterClient was not created and "
|
"Cluster {ClusterId} no longer yields any usable contact point; stopping its "
|
||||||
+ "every central→node command will be dropped until a refresh finds one");
|
+ "ClusterClient", clusterId);
|
||||||
return;
|
Context.Stop(_clients[clusterId].Client);
|
||||||
|
_clients.Remove(clusterId);
|
||||||
}
|
}
|
||||||
|
|
||||||
if (_client is not null && _currentContacts.SetEquals(contacts)) return;
|
var anyUsable = false;
|
||||||
|
foreach (var (clusterId, contacts) in byCluster)
|
||||||
|
{
|
||||||
|
if (contacts.IsEmpty)
|
||||||
|
{
|
||||||
|
// Defensive: LoadContactsFromDb never emits an empty-but-present cluster (a key is
|
||||||
|
// created only on a first successfully-parsed row), so this arm is unreachable for the
|
||||||
|
// DB producer — an all-malformed cluster is handled by the absent-key branch above. It
|
||||||
|
// guards a ContactsLoaded (a public record) constructed elsewhere with an empty list.
|
||||||
|
_log.Warning(
|
||||||
|
"Cluster {ClusterId} has no usable ClusterNode contact points; no ClusterClient "
|
||||||
|
+ "was created for it and its nodes' commands will be dropped until a refresh finds "
|
||||||
|
+ "one", clusterId);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
RebuildClient(contacts);
|
anyUsable = true;
|
||||||
|
|
||||||
|
if (_clients.TryGetValue(clusterId, out var existing) && existing.Contacts.SetEquals(contacts))
|
||||||
|
continue;
|
||||||
|
|
||||||
|
RebuildClient(clusterId, contacts);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!anyUsable && _clients.Count == 0)
|
||||||
|
{
|
||||||
|
_log.Warning(
|
||||||
|
"No usable ClusterNode contact points in any cluster; no mesh ClusterClient was "
|
||||||
|
+ "created and every central→node command will be dropped until a refresh finds one");
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// <summary>Stops any existing client and creates a replacement for the new contact set.</summary>
|
/// <summary>Stops a cluster's existing client and creates a replacement for its new contact set.</summary>
|
||||||
/// <remarks>
|
/// <remarks>
|
||||||
/// <para>
|
/// <para>
|
||||||
/// <b>TODO(Phase 6): one client per application <c>Cluster</c>.</b> Today this actor
|
/// <b>One client per application <c>Cluster</c> (delivered in Phase 6).</b> This actor now
|
||||||
/// creates exactly <b>one</b> client whose contacts are the whole fleet, and that is not
|
/// holds one <c>ClusterClient</c> per cluster and <see cref="HandleMeshCommand"/> fans
|
||||||
/// a compromise — it is the only correct shape while the fleet is a single Akka mesh.
|
/// <c>SendToAll</c> across all of them. Once the fleet is split into one Akka mesh per
|
||||||
|
/// cluster, a <c>ClusterClientReceptionist</c> serves only <i>its own</i> mesh, so a single
|
||||||
|
/// client's <c>SendToAll</c> would reach the one mesh whose receptionist answered and leave
|
||||||
|
/// every other cluster silently on its old configuration. Fanning across a client per
|
||||||
|
/// cluster restores the fleet-wide reach the pre-split DPS broadcast gave.
|
||||||
/// </para>
|
/// </para>
|
||||||
/// <para>
|
/// <para>
|
||||||
/// A <c>ClusterClientReceptionist</c> serves its entire cluster, so
|
/// Why this is not N× duplicate dispatch: each client dials only its own cluster's
|
||||||
/// <c>SendToAll</c> reaches every registered node-comm actor in the mesh <i>regardless of
|
/// receptionists, so a command is delivered <b>once per mesh</b>, not once per cluster into
|
||||||
/// which node's address was used as the contact point</i>. One client per Cluster would
|
/// every mesh. (While the fleet was a single mesh, the same code would have duplicated —
|
||||||
/// therefore fan each command out once per cluster — N× duplicate
|
/// which is exactly why the fan-out could not land until the meshes were partitioned.)
|
||||||
/// <c>DispatchDeployment</c> and N× duplicate <c>ApplyAck</c>. Phase 6 splits the meshes,
|
|
||||||
/// at which point per-cluster clients become both correct and necessary.
|
|
||||||
/// </para>
|
/// </para>
|
||||||
/// <para>
|
/// <para>
|
||||||
/// Corollary worth stating because it is easy to assume otherwise: <b>the contact set
|
/// Corollary worth stating because it is easy to assume otherwise: <b>the contact set
|
||||||
/// does not scope delivery.</b> Excluding a maintenance-mode node from the contacts does
|
/// does not scope delivery within a mesh.</b> Excluding a maintenance-mode node from a
|
||||||
/// not stop it receiving commands — it still receives them, exactly as it does under
|
/// cluster's contacts does not stop it receiving commands — its receptionist still delivers
|
||||||
/// today's DPS broadcast. The filter is about which receptionists are worth <i>dialling</i>
|
/// to every registered node-comm actor in that mesh. The filter is about which receptionists
|
||||||
/// (a node in maintenance may be switched off), not about who gets the message.
|
/// are worth <i>dialling</i> (a node in maintenance may be switched off), not about who gets
|
||||||
|
/// the message.
|
||||||
/// </para>
|
/// </para>
|
||||||
/// </remarks>
|
/// </remarks>
|
||||||
/// <param name="contacts">The receptionist addresses to dial.</param>
|
/// <param name="clusterId">The application <c>Cluster</c> whose client is being (re)built.</param>
|
||||||
private void RebuildClient(ImmutableHashSet<string> contacts)
|
/// <param name="contacts">The receptionist addresses to dial for that cluster.</param>
|
||||||
|
private void RebuildClient(string clusterId, ImmutableHashSet<string> contacts)
|
||||||
{
|
{
|
||||||
if (_client is not null)
|
if (_clients.TryGetValue(clusterId, out var existing))
|
||||||
{
|
{
|
||||||
_log.Info("Mesh contact set changed; replacing the ClusterClient");
|
_log.Info("Contact set for cluster {ClusterId} changed; replacing its ClusterClient", clusterId);
|
||||||
Context.Stop(_client);
|
Context.Stop(existing.Client);
|
||||||
|
|
||||||
// Cleared now: if the create below throws, a stale ref would route commands into a
|
// Removed now: if the create below throws, a stale ref would route commands into a
|
||||||
// stopping actor and they would vanish without a warning.
|
// stopping actor and they would vanish without a warning.
|
||||||
_client = null;
|
_clients.Remove(clusterId);
|
||||||
_currentContacts = ImmutableHashSet<string>.Empty;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
var paths = contacts.Select(ActorPath.Parse).ToImmutableHashSet();
|
var paths = contacts.Select(ActorPath.Parse).ToImmutableHashSet();
|
||||||
_client = _clientFactory.Create(Context.System, paths);
|
var client = _clientFactory.Create(Context.System, clusterId, paths);
|
||||||
_currentContacts = contacts;
|
_clients[clusterId] = (client, contacts);
|
||||||
_log.Info("Mesh ClusterClient created with {Count} contact point(s)", contacts.Count);
|
_log.Info(
|
||||||
|
"Mesh ClusterClient for cluster {ClusterId} created with {Count} contact point(s)",
|
||||||
|
clusterId, contacts.Count);
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
{
|
{
|
||||||
_log.Error(ex,
|
_log.Error(ex,
|
||||||
"Failed to create the mesh ClusterClient; central→node commands are dropped until "
|
"Failed to create the mesh ClusterClient for cluster {ClusterId}; its nodes' commands "
|
||||||
+ "the next contact refresh succeeds");
|
+ "are dropped until the next contact refresh succeeds", clusterId);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -300,9 +366,12 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
|
|||||||
public sealed record RefreshContacts;
|
public sealed record RefreshContacts;
|
||||||
|
|
||||||
/// <summary>Result of a contact-point load, piped back from the DB read.</summary>
|
/// <summary>Result of a contact-point load, piped back from the DB read.</summary>
|
||||||
/// <param name="Contacts">Receptionist addresses that parsed cleanly.</param>
|
/// <param name="ContactsByCluster">
|
||||||
/// <param name="Malformed">Rows that did not, described for the log.</param>
|
/// Receptionist addresses that parsed cleanly, grouped by the owning application
|
||||||
|
/// <c>Cluster</c>'s id. One <c>ClusterClient</c> is built per key.
|
||||||
|
/// </param>
|
||||||
|
/// <param name="Malformed">Rows that did not parse, described for the log.</param>
|
||||||
public sealed record ContactsLoaded(
|
public sealed record ContactsLoaded(
|
||||||
IReadOnlyList<string> Contacts,
|
IReadOnlyDictionary<string, IReadOnlyList<string>> ContactsByCluster,
|
||||||
IReadOnlyList<string> Malformed);
|
IReadOnlyList<string> Malformed);
|
||||||
}
|
}
|
||||||
|
|||||||
+23
-3
@@ -13,9 +13,14 @@ public interface IMeshClusterClientFactory
|
|||||||
{
|
{
|
||||||
/// <summary>Creates a client for the given receptionist contact points.</summary>
|
/// <summary>Creates a client for the given receptionist contact points.</summary>
|
||||||
/// <param name="system">The actor system to spawn the client in.</param>
|
/// <param name="system">The actor system to spawn the client in.</param>
|
||||||
|
/// <param name="clusterId">
|
||||||
|
/// The application <c>Cluster</c> these contacts belong to. Central now creates one client per
|
||||||
|
/// cluster (Phase 6), so the id is woven into the client's actor name to keep the per-cluster
|
||||||
|
/// clients distinct and diagnosable.
|
||||||
|
/// </param>
|
||||||
/// <param name="contacts">Receptionist actor paths, one per reachable node.</param>
|
/// <param name="contacts">Receptionist actor paths, one per reachable node.</param>
|
||||||
/// <returns>The new client's <see cref="IActorRef"/>.</returns>
|
/// <returns>The new client's <see cref="IActorRef"/>.</returns>
|
||||||
IActorRef Create(ActorSystem system, ImmutableHashSet<ActorPath> contacts);
|
IActorRef Create(ActorSystem system, string clusterId, ImmutableHashSet<ActorPath> contacts);
|
||||||
}
|
}
|
||||||
|
|
||||||
/// <inheritdoc />
|
/// <inheritdoc />
|
||||||
@@ -35,13 +40,28 @@ public sealed class DefaultMeshClusterClientFactory : IMeshClusterClientFactory
|
|||||||
private long _generation;
|
private long _generation;
|
||||||
|
|
||||||
/// <inheritdoc />
|
/// <inheritdoc />
|
||||||
public IActorRef Create(ActorSystem system, ImmutableHashSet<ActorPath> contacts)
|
public IActorRef Create(ActorSystem system, string clusterId, ImmutableHashSet<ActorPath> contacts)
|
||||||
{
|
{
|
||||||
ArgumentNullException.ThrowIfNull(system);
|
ArgumentNullException.ThrowIfNull(system);
|
||||||
|
ArgumentException.ThrowIfNullOrEmpty(clusterId);
|
||||||
ArgumentNullException.ThrowIfNull(contacts);
|
ArgumentNullException.ThrowIfNull(contacts);
|
||||||
|
|
||||||
var settings = ClusterClientSettings.Create(system).WithInitialContacts(contacts);
|
var settings = ClusterClientSettings.Create(system).WithInitialContacts(contacts);
|
||||||
var name = $"mesh-client-{Interlocked.Increment(ref _generation)}";
|
var name = $"mesh-client-{SanitizeForActorName(clusterId)}-{Interlocked.Increment(ref _generation)}";
|
||||||
return system.ActorOf(ClusterClient.Props(settings), name);
|
return system.ActorOf(ClusterClient.Props(settings), name);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Reduces a cluster id to characters safe in an Akka actor name. The id is operator-authored
|
||||||
|
/// (e.g. a GUID or a slug), so it may carry characters Akka rejects; anything outside
|
||||||
|
/// <c>[A-Za-z0-9-]</c> becomes <c>-</c>. The trailing generation counter still guarantees
|
||||||
|
/// uniqueness, so a lossy sanitisation cannot cause a name collision.
|
||||||
|
/// </summary>
|
||||||
|
/// <param name="clusterId">The raw cluster id.</param>
|
||||||
|
/// <returns>An actor-name-safe rendering of the id.</returns>
|
||||||
|
private static string SanitizeForActorName(string clusterId)
|
||||||
|
{
|
||||||
|
var chars = clusterId.Select(c => char.IsAsciiLetterOrDigit(c) || c == '-' ? c : '-');
|
||||||
|
return new string(chars.ToArray());
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -58,10 +58,23 @@ public sealed record ClusterNodeAddress(string NodeId, string Host, int AkkaPort
|
|||||||
/// to be deleted again.
|
/// to be deleted again.
|
||||||
/// </para>
|
/// </para>
|
||||||
/// <para>
|
/// <para>
|
||||||
/// <b>Phase 2 must revisit this.</b> Once the fleet splits into one mesh per cluster, an
|
/// <b>Scoped to the admin node's own cluster (Phase 6).</b> Once the fleet splits into one
|
||||||
/// admin node no longer sees site members at all, and every site row would report
|
/// mesh per cluster, an admin node no longer sees site members at all, and a fleet-wide sweep
|
||||||
/// <see cref="AddressMismatchKind.EnabledRowNotInCluster"/> forever. The check then has to
|
/// would report every other cluster's row as
|
||||||
/// move to whatever transport replaces gossip — or be scoped to the central mesh.
|
/// <see cref="AddressMismatchKind.EnabledRowNotInCluster"/> forever. The actor therefore scopes
|
||||||
|
/// <b>both</b> inputs to this function to the admin node's own cluster: the <c>rows</c>/enabled
|
||||||
|
/// set to rows whose <c>ClusterId</c> matches
|
||||||
|
/// <see cref="ZB.MOM.WW.OtOpcUa.Commons.Interfaces.IClusterRoleInfo.ClusterId"/>, and the
|
||||||
|
/// observed membership to members carrying its own
|
||||||
|
/// <see cref="ZB.MOM.WW.OtOpcUa.Commons.Interfaces.IClusterRoleInfo.ClusterRole"/>
|
||||||
|
/// (<c>FilterOwnClusterDriverMembers</c>). Scoping only the rows would swap the
|
||||||
|
/// <see cref="AddressMismatchKind.EnabledRowNotInCluster"/> false positive for a worse
|
||||||
|
/// Error-level <see cref="AddressMismatchKind.RunningNodeHasNoRow"/> during the mixed-mesh
|
||||||
|
/// window, where the DB scope is live but central still sees foreign members via gossip. A
|
||||||
|
/// legacy admin node carrying no cluster role (null <c>ClusterId</c>/<c>ClusterRole</c>) still
|
||||||
|
/// sees every member via gossip and reconciles the full fleet unchanged. The pure function
|
||||||
|
/// below is untouched — scoping happens entirely in the actor's DB queries and its
|
||||||
|
/// membership filter.
|
||||||
/// </para>
|
/// </para>
|
||||||
/// </remarks>
|
/// </remarks>
|
||||||
public static class ClusterNodeAddressReconciler
|
public static class ClusterNodeAddressReconciler
|
||||||
|
|||||||
+76
-11
@@ -2,6 +2,7 @@ using Akka.Actor;
|
|||||||
using Akka.Cluster;
|
using Akka.Cluster;
|
||||||
using Akka.Event;
|
using Akka.Event;
|
||||||
using Microsoft.EntityFrameworkCore;
|
using Microsoft.EntityFrameworkCore;
|
||||||
|
using ZB.MOM.WW.OtOpcUa.Cluster;
|
||||||
using ZB.MOM.WW.OtOpcUa.Configuration;
|
using ZB.MOM.WW.OtOpcUa.Configuration;
|
||||||
|
|
||||||
namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Fleet;
|
namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Fleet;
|
||||||
@@ -20,7 +21,7 @@ namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Fleet;
|
|||||||
public sealed class ClusterNodeAddressReconcilerActor : ReceiveActor, IWithTimers
|
public sealed class ClusterNodeAddressReconcilerActor : ReceiveActor, IWithTimers
|
||||||
{
|
{
|
||||||
/// <summary>Driver role name — only members carrying it are expected to have a ClusterNode row.</summary>
|
/// <summary>Driver role name — only members carrying it are expected to have a ClusterNode row.</summary>
|
||||||
public const string DriverRole = "driver";
|
public const string DriverRole = RoleParser.Driver;
|
||||||
|
|
||||||
/// <summary>Collapses a burst of membership events (a rolling restart) into one reconcile.</summary>
|
/// <summary>Collapses a burst of membership events (a rolling restart) into one reconcile.</summary>
|
||||||
public static readonly TimeSpan DebounceWindow = TimeSpan.FromSeconds(2);
|
public static readonly TimeSpan DebounceWindow = TimeSpan.FromSeconds(2);
|
||||||
@@ -29,6 +30,8 @@ public sealed class ClusterNodeAddressReconcilerActor : ReceiveActor, IWithTimer
|
|||||||
public static readonly TimeSpan DefaultSweepInterval = TimeSpan.FromMinutes(5);
|
public static readonly TimeSpan DefaultSweepInterval = TimeSpan.FromMinutes(5);
|
||||||
|
|
||||||
private readonly IDbContextFactory<OtOpcUaConfigDbContext> _dbFactory;
|
private readonly IDbContextFactory<OtOpcUaConfigDbContext> _dbFactory;
|
||||||
|
private readonly string? _ownClusterId;
|
||||||
|
private readonly string? _ownClusterRole;
|
||||||
private readonly Akka.Cluster.Cluster _cluster;
|
private readonly Akka.Cluster.Cluster _cluster;
|
||||||
private readonly TimeSpan _sweepInterval;
|
private readonly TimeSpan _sweepInterval;
|
||||||
private readonly ILoggingAdapter _log = Context.GetLogger();
|
private readonly ILoggingAdapter _log = Context.GetLogger();
|
||||||
@@ -39,19 +42,49 @@ public sealed class ClusterNodeAddressReconcilerActor : ReceiveActor, IWithTimer
|
|||||||
|
|
||||||
/// <summary>Creates Props for the reconciler.</summary>
|
/// <summary>Creates Props for the reconciler.</summary>
|
||||||
/// <param name="dbFactory">Factory for the config database context.</param>
|
/// <param name="dbFactory">Factory for the config database context.</param>
|
||||||
|
/// <param name="ownClusterId">
|
||||||
|
/// The admin node's own <c>ClusterId</c> (e.g. <c>MAIN</c>), from
|
||||||
|
/// <see cref="IClusterRoleInfo.ClusterId"/>. When non-null-and-non-empty this scopes the row
|
||||||
|
/// side of the reconcile (<c>rows</c>/<c>enabled</c>) to that cluster — the only members
|
||||||
|
/// central's own mesh can see after the Phase 6 split. A null or empty value (a legacy admin
|
||||||
|
/// node with no cluster role) reconciles the whole fleet, unchanged.
|
||||||
|
/// </param>
|
||||||
|
/// <param name="ownClusterRole">
|
||||||
|
/// The admin node's own cluster role (e.g. <c>cluster-MAIN</c>), from
|
||||||
|
/// <see cref="IClusterRoleInfo.ClusterRole"/>. Scopes the membership (<c>observed</c>) side to
|
||||||
|
/// match the row side, so a foreign-cluster driver member central still sees via gossip in the
|
||||||
|
/// mixed-mesh window is not mis-reported. Null/empty keeps every driver member (legacy).
|
||||||
|
/// </param>
|
||||||
/// <param name="sweepInterval">Periodic sweep interval; defaults to <see cref="DefaultSweepInterval"/>.</param>
|
/// <param name="sweepInterval">Periodic sweep interval; defaults to <see cref="DefaultSweepInterval"/>.</param>
|
||||||
/// <returns>Props for actor creation.</returns>
|
/// <returns>Props for actor creation.</returns>
|
||||||
public static Props Props(
|
public static Props Props(
|
||||||
IDbContextFactory<OtOpcUaConfigDbContext> dbFactory, TimeSpan? sweepInterval = null) =>
|
IDbContextFactory<OtOpcUaConfigDbContext> dbFactory,
|
||||||
Akka.Actor.Props.Create(() => new ClusterNodeAddressReconcilerActor(dbFactory, sweepInterval));
|
string? ownClusterId = null,
|
||||||
|
string? ownClusterRole = null,
|
||||||
|
TimeSpan? sweepInterval = null) =>
|
||||||
|
Akka.Actor.Props.Create(() =>
|
||||||
|
new ClusterNodeAddressReconcilerActor(dbFactory, ownClusterId, ownClusterRole, sweepInterval));
|
||||||
|
|
||||||
/// <summary>Initializes a new instance of the <see cref="ClusterNodeAddressReconcilerActor"/> class.</summary>
|
/// <summary>Initializes a new instance of the <see cref="ClusterNodeAddressReconcilerActor"/> class.</summary>
|
||||||
/// <param name="dbFactory">Factory for the config database context.</param>
|
/// <param name="dbFactory">Factory for the config database context.</param>
|
||||||
|
/// <param name="ownClusterId">
|
||||||
|
/// The admin node's own <c>ClusterId</c>; scopes the row side to that cluster's rows when
|
||||||
|
/// non-null-and-non-empty, else reconciles the whole fleet (legacy behaviour).
|
||||||
|
/// </param>
|
||||||
|
/// <param name="ownClusterRole">
|
||||||
|
/// The admin node's own cluster role; scopes the observed-membership side to that role when
|
||||||
|
/// non-null-and-non-empty, else keeps every driver member (legacy behaviour).
|
||||||
|
/// </param>
|
||||||
/// <param name="sweepInterval">Periodic sweep interval; defaults to <see cref="DefaultSweepInterval"/>.</param>
|
/// <param name="sweepInterval">Periodic sweep interval; defaults to <see cref="DefaultSweepInterval"/>.</param>
|
||||||
public ClusterNodeAddressReconcilerActor(
|
public ClusterNodeAddressReconcilerActor(
|
||||||
IDbContextFactory<OtOpcUaConfigDbContext> dbFactory, TimeSpan? sweepInterval = null)
|
IDbContextFactory<OtOpcUaConfigDbContext> dbFactory,
|
||||||
|
string? ownClusterId = null,
|
||||||
|
string? ownClusterRole = null,
|
||||||
|
TimeSpan? sweepInterval = null)
|
||||||
{
|
{
|
||||||
_dbFactory = dbFactory;
|
_dbFactory = dbFactory;
|
||||||
|
_ownClusterId = string.IsNullOrEmpty(ownClusterId) ? null : ownClusterId;
|
||||||
|
_ownClusterRole = string.IsNullOrEmpty(ownClusterRole) ? null : ownClusterRole;
|
||||||
_cluster = Akka.Cluster.Cluster.Get(Context.System);
|
_cluster = Akka.Cluster.Cluster.Get(Context.System);
|
||||||
_sweepInterval = sweepInterval ?? DefaultSweepInterval;
|
_sweepInterval = sweepInterval ?? DefaultSweepInterval;
|
||||||
|
|
||||||
@@ -75,24 +108,27 @@ public sealed class ClusterNodeAddressReconcilerActor : ReceiveActor, IWithTimer
|
|||||||
|
|
||||||
private void Reconcile()
|
private void Reconcile()
|
||||||
{
|
{
|
||||||
var observed = _cluster.State.Members
|
var observed = FilterOwnClusterDriverMembers(_cluster.State.Members, _ownClusterRole);
|
||||||
.Where(m => m.Status == MemberStatus.Up && m.Roles.Contains(DriverRole))
|
|
||||||
.Select(m => (Host: m.Address.Host ?? string.Empty, Port: m.Address.Port ?? 0))
|
|
||||||
.Where(a => !string.IsNullOrWhiteSpace(a.Host) && a.Port > 0)
|
|
||||||
.ToList();
|
|
||||||
|
|
||||||
List<ClusterNodeAddress> rows;
|
List<ClusterNodeAddress> rows;
|
||||||
List<string> enabled;
|
List<string> enabled;
|
||||||
try
|
try
|
||||||
{
|
{
|
||||||
using var db = _dbFactory.CreateDbContext();
|
using var db = _dbFactory.CreateDbContext();
|
||||||
rows = db.ClusterNodes.AsNoTracking()
|
// Per-cluster mesh Phase 6: scope to the admin node's OWN cluster's rows. After the split,
|
||||||
|
// central's mesh sees only its own pair, so a row in another cluster it cannot observe would
|
||||||
|
// false-report EnabledRowNotInCluster forever. A legacy admin node with no cluster role
|
||||||
|
// (_ownClusterId == null) still sees every member via gossip, so it reconciles the full fleet.
|
||||||
|
var scoped = db.ClusterNodes.AsNoTracking();
|
||||||
|
if (_ownClusterId is not null)
|
||||||
|
scoped = scoped.Where(n => n.ClusterId == _ownClusterId);
|
||||||
|
rows = scoped
|
||||||
.Select(n => new ClusterNodeAddress(n.NodeId, n.Host, n.AkkaPort))
|
.Select(n => new ClusterNodeAddress(n.NodeId, n.Host, n.AkkaPort))
|
||||||
.ToList();
|
.ToList();
|
||||||
// Same predicate the coordinator's expected-ack set uses — a node the deploy path will
|
// Same predicate the coordinator's expected-ack set uses — a node the deploy path will
|
||||||
// not wait for must not be reported as missing either, or the maintenance hatch trades a
|
// not wait for must not be reported as missing either, or the maintenance hatch trades a
|
||||||
// failed deployment for a permanent warning.
|
// failed deployment for a permanent warning.
|
||||||
enabled = db.ClusterNodes.AsNoTracking()
|
enabled = scoped
|
||||||
.Where(n => n.Enabled && !n.MaintenanceMode).Select(n => n.NodeId).ToList();
|
.Where(n => n.Enabled && !n.MaintenanceMode).Select(n => n.NodeId).ToList();
|
||||||
}
|
}
|
||||||
catch (Exception ex)
|
catch (Exception ex)
|
||||||
@@ -126,6 +162,35 @@ public sealed class ClusterNodeAddressReconcilerActor : ReceiveActor, IWithTimer
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Projects the Up, driver-role members central should reconcile against, scoped to the admin
|
||||||
|
/// node's OWN cluster role when it carries one. Extracted so the observed-membership filter —
|
||||||
|
/// the membership half of the Phase 6 own-cluster scope, mirroring the <c>ClusterId</c> row
|
||||||
|
/// scope in <see cref="Reconcile"/> — is unit-testable without seeding a live cluster.
|
||||||
|
/// </summary>
|
||||||
|
/// <param name="members">The live cluster members (from <c>Cluster.State.Members</c>).</param>
|
||||||
|
/// <param name="ownClusterRole">
|
||||||
|
/// The admin node's own cluster role (e.g. <c>cluster-MAIN</c>). When non-null-and-non-empty,
|
||||||
|
/// driver members that do not also carry it are excluded — matching the row scope so a foreign
|
||||||
|
/// driver member central still sees via gossip in the mixed-mesh window (row filtered out, but
|
||||||
|
/// member still visible) is NOT mis-reported as an Error-level <c>RunningNodeHasNoRow</c>. A
|
||||||
|
/// null or empty value (a legacy node with no cluster role) keeps every driver member —
|
||||||
|
/// reconcile-all, unchanged.
|
||||||
|
/// </param>
|
||||||
|
/// <returns>The <c>host</c>/<c>port</c> of each in-scope Up driver member.</returns>
|
||||||
|
public static IReadOnlyList<(string Host, int Port)> FilterOwnClusterDriverMembers(
|
||||||
|
IEnumerable<Member> members, string? ownClusterRole)
|
||||||
|
{
|
||||||
|
var driverMembers = members
|
||||||
|
.Where(m => m.Status == MemberStatus.Up && m.Roles.Contains(DriverRole));
|
||||||
|
if (!string.IsNullOrEmpty(ownClusterRole))
|
||||||
|
driverMembers = driverMembers.Where(m => m.Roles.Contains(ownClusterRole));
|
||||||
|
return driverMembers
|
||||||
|
.Select(m => (Host: m.Address.Host ?? string.Empty, Port: m.Address.Port ?? 0))
|
||||||
|
.Where(a => !string.IsNullOrWhiteSpace(a.Host) && a.Port > 0)
|
||||||
|
.ToList();
|
||||||
|
}
|
||||||
|
|
||||||
/// <summary>Self-message: run a reconcile pass now.</summary>
|
/// <summary>Self-message: run a reconcile pass now.</summary>
|
||||||
public sealed class ReconcileNow
|
public sealed class ReconcileNow
|
||||||
{
|
{
|
||||||
|
|||||||
@@ -31,10 +31,9 @@ public sealed record ManualFailoverSnapshot(string? PrimaryAddress, IReadOnlyLis
|
|||||||
/// moves with it and OPC UA clients re-select.
|
/// moves with it and OPC UA clients re-select.
|
||||||
/// </para>
|
/// </para>
|
||||||
/// <para>
|
/// <para>
|
||||||
/// <b>Mesh-scope caveat.</b> Until the per-cluster mesh work lands
|
/// Post-Phase-6 (per-cluster mesh split): the Primary is elected per application
|
||||||
/// (<c>docs/plans/2026-07-21-per-cluster-mesh-design.md</c>) the Primary is elected once per
|
/// <c>Cluster</c>, because each cluster's redundant pair now runs its own independent
|
||||||
/// Akka mesh, not per application <c>Cluster</c> row. On a fleet running several application
|
/// 2-node Akka mesh. This service therefore acts on <b>this pair's own</b> Primary.
|
||||||
/// clusters in one mesh this acts on the whole mesh's Primary. The UI says so.
|
|
||||||
/// </para>
|
/// </para>
|
||||||
/// </remarks>
|
/// </remarks>
|
||||||
public interface IManualFailoverService
|
public interface IManualFailoverService
|
||||||
|
|||||||
@@ -2,6 +2,7 @@ using Akka.Actor;
|
|||||||
using Akka.Cluster;
|
using Akka.Cluster;
|
||||||
using Akka.Cluster.Tools.PublishSubscribe;
|
using Akka.Cluster.Tools.PublishSubscribe;
|
||||||
using Akka.Event;
|
using Akka.Event;
|
||||||
|
using ZB.MOM.WW.OtOpcUa.Cluster;
|
||||||
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Redundancy;
|
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Redundancy;
|
||||||
using ZB.MOM.WW.OtOpcUa.Commons.Types;
|
using ZB.MOM.WW.OtOpcUa.Commons.Types;
|
||||||
using CommonsRedundancyRole = ZB.MOM.WW.OtOpcUa.Commons.Messages.Redundancy.RedundancyRole;
|
using CommonsRedundancyRole = ZB.MOM.WW.OtOpcUa.Commons.Messages.Redundancy.RedundancyRole;
|
||||||
@@ -112,7 +113,7 @@ public sealed class RedundancyStateActor : ReceiveActor, IWithTimers
|
|||||||
}
|
}
|
||||||
|
|
||||||
/// <summary>The cluster role that marks a node as carrying the driver runtime.</summary>
|
/// <summary>The cluster role that marks a node as carrying the driver runtime.</summary>
|
||||||
public const string DriverRole = "driver";
|
public const string DriverRole = RoleParser.Driver;
|
||||||
|
|
||||||
/// <summary>
|
/// <summary>
|
||||||
/// Selects the driver Primary: the <b>oldest</b> Up member carrying the <see cref="DriverRole"/>.
|
/// Selects the driver Primary: the <b>oldest</b> Up member carrying the <see cref="DriverRole"/>.
|
||||||
|
|||||||
@@ -13,6 +13,7 @@ using ZB.MOM.WW.OtOpcUa.ControlPlane.AdminOperations;
|
|||||||
using ZB.MOM.WW.OtOpcUa.ControlPlane.Audit;
|
using ZB.MOM.WW.OtOpcUa.ControlPlane.Audit;
|
||||||
using ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
|
using ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
|
||||||
using ZB.MOM.WW.OtOpcUa.ControlPlane.Coordinators;
|
using ZB.MOM.WW.OtOpcUa.ControlPlane.Coordinators;
|
||||||
|
using ZB.MOM.WW.OtOpcUa.Commons.Interfaces;
|
||||||
using ZB.MOM.WW.OtOpcUa.ControlPlane.Fleet;
|
using ZB.MOM.WW.OtOpcUa.ControlPlane.Fleet;
|
||||||
using ZB.MOM.WW.OtOpcUa.ControlPlane.Redundancy;
|
using ZB.MOM.WW.OtOpcUa.ControlPlane.Redundancy;
|
||||||
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
|
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
|
||||||
@@ -130,10 +131,12 @@ public static class ServiceCollectionExtensions
|
|||||||
(system, registry, resolver) => FleetStatusBroadcaster.Props(),
|
(system, registry, resolver) => FleetStatusBroadcaster.Props(),
|
||||||
singletonOptions);
|
singletonOptions);
|
||||||
|
|
||||||
builder.WithSingleton<RedundancyStateActorKey>(
|
// Per-cluster mesh Phase 6: the redundancy-state singleton is NO LONGER registered here.
|
||||||
RedundancyStateSingletonName,
|
// It was an admin-scoped, fleet-wide singleton — one Primary elected across the whole mesh.
|
||||||
(system, registry, resolver) => RedundancyStateActor.Props(),
|
// After the mesh split a driver-only site pair has no admin node to host it, so it moved to
|
||||||
singletonOptions);
|
// WithOtOpcUaClusterRedundancySingleton, a cluster-{ClusterId}-scoped singleton spawned on
|
||||||
|
// EVERY driver node (the fused central included). One per mesh, electing its own pair-local
|
||||||
|
// Primary and publishing on its own mesh's DistributedPubSub.
|
||||||
|
|
||||||
// Per-cluster mesh Phase 1: ClusterNode.AkkaPort duplicates the node's own Cluster:Port and
|
// Per-cluster mesh Phase 1: ClusterNode.AkkaPort duplicates the node's own Cluster:Port and
|
||||||
// nothing makes them agree. Runs here rather than on each driver node because Phase 4 removes
|
// nothing makes them agree. Runs here rather than on each driver node because Phase 4 removes
|
||||||
@@ -150,13 +153,77 @@ public static class ServiceCollectionExtensions
|
|||||||
(system, registry, resolver) =>
|
(system, registry, resolver) =>
|
||||||
{
|
{
|
||||||
var dbFactory = resolver.GetService<IDbContextFactory<OtOpcUaConfigDbContext>>();
|
var dbFactory = resolver.GetService<IDbContextFactory<OtOpcUaConfigDbContext>>();
|
||||||
return ClusterNodeAddressReconcilerActor.Props(dbFactory);
|
// Per-cluster mesh Phase 6: scope BOTH halves of the reconcile to the admin node's OWN
|
||||||
|
// cluster — rows by ClusterId, observed membership by cluster role. After the mesh split
|
||||||
|
// central sees only its own pair via gossip, so an unscoped sweep would false-report
|
||||||
|
// every other cluster's rows (EnabledRowNotInCluster). Scoping only the rows would swap
|
||||||
|
// that for a WORSE Error-level RunningNodeHasNoRow in the mixed-mesh window (rows already
|
||||||
|
// filtered, but foreign members still visible via gossip). A legacy node with no cluster
|
||||||
|
// role has null ClusterId/ClusterRole ⇒ full-fleet reconcile, unchanged.
|
||||||
|
var roleInfo = resolver.GetService<IClusterRoleInfo>();
|
||||||
|
return ClusterNodeAddressReconcilerActor.Props(
|
||||||
|
dbFactory, roleInfo.ClusterId, roleInfo.ClusterRole);
|
||||||
},
|
},
|
||||||
singletonOptions,
|
singletonOptions,
|
||||||
createProxyToo: false);
|
createProxyToo: false);
|
||||||
|
|
||||||
return builder;
|
return builder;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Registers <see cref="RedundancyStateActor"/> as a cluster singleton scoped to the local node's
|
||||||
|
/// OWN <c>cluster-{ClusterId}</c> role (or the fixed <c>driver</c> role when the node carries no
|
||||||
|
/// cluster role), and spawns it on EVERY driver node (per-cluster mesh, Phase 6). Wire from the
|
||||||
|
/// driver branch of the fused Host's Program.cs — the fused central node is <c>hasDriver</c> too,
|
||||||
|
/// so it hosts exactly one redundancy singleton scoped to its own cluster role (e.g.
|
||||||
|
/// <c>cluster-MAIN</c>), while the admin singleton set it also registers no longer carries
|
||||||
|
/// redundancy.
|
||||||
|
/// </summary>
|
||||||
|
/// <param name="builder">The Akka configuration builder.</param>
|
||||||
|
/// <param name="roleInfo">The local node's cluster-role view (Task 1's <c>ClusterRoleInfo</c>).</param>
|
||||||
|
/// <returns>The builder for fluent chaining.</returns>
|
||||||
|
/// <remarks>
|
||||||
|
/// After the mesh split, central's mesh sees only its own pair — a driver-only site pair therefore
|
||||||
|
/// has NO admin node to elect its Primary. Scoping this singleton to the node's cluster role and
|
||||||
|
/// spawning it on every driver node gives each mesh exactly one, electing its own pair-local
|
||||||
|
/// Primary and publishing <c>redundancy-state</c> on its own mesh's DistributedPubSub.
|
||||||
|
/// </remarks>
|
||||||
|
public static AkkaConfigurationBuilder WithOtOpcUaClusterRedundancySingleton(
|
||||||
|
this AkkaConfigurationBuilder builder, IClusterRoleInfo roleInfo)
|
||||||
|
{
|
||||||
|
var singletonOptions = BuildClusterRedundancySingletonOptions(roleInfo);
|
||||||
|
|
||||||
|
builder.WithSingleton<RedundancyStateActorKey>(
|
||||||
|
RedundancyStateSingletonName,
|
||||||
|
(system, registry, resolver) => RedundancyStateActor.Props(),
|
||||||
|
singletonOptions);
|
||||||
|
|
||||||
|
return builder;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Builds the <see cref="ClusterSingletonOptions"/> that scope the redundancy-state singleton to
|
||||||
|
/// the local node's <c>cluster-{ClusterId}</c> role, falling back to the fixed <c>driver</c> role
|
||||||
|
/// when the node carries no cluster role. Extracted (like
|
||||||
|
/// <c>SplitBrainResolverActivationTests</c>'s <c>BuildDowningHocon</c>) so the exact role scope is
|
||||||
|
/// unit-assertable without booting a cluster.
|
||||||
|
/// </summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// Cluster-scope is the robust choice — it survives an accidental two-mesh merge by keeping one
|
||||||
|
/// singleton per cluster. The <c>driver</c>-role fallback deliberately does NOT throw when a node
|
||||||
|
/// has no cluster role, because (a) that is the pre-Phase-6 fleet-wide behavior on a legacy single
|
||||||
|
/// mesh, keeping legacy/harness nodes and any not-yet-migrated deployment booting unchanged, and
|
||||||
|
/// (b) on a genuinely split 2-node mesh the <c>driver</c> role is already pair-local — the mesh IS
|
||||||
|
/// the pair. So the fallback is correct in both worlds and costs no availability.
|
||||||
|
/// </remarks>
|
||||||
|
/// <param name="roleInfo">The local node's cluster-role view.</param>
|
||||||
|
/// <returns>Singleton options scoped to the node's cluster role, or the <c>driver</c> role.</returns>
|
||||||
|
public static ClusterSingletonOptions BuildClusterRedundancySingletonOptions(IClusterRoleInfo roleInfo)
|
||||||
|
{
|
||||||
|
ArgumentNullException.ThrowIfNull(roleInfo);
|
||||||
|
|
||||||
|
return new ClusterSingletonOptions { Role = roleInfo.ClusterRole ?? RoleParser.Driver };
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// <summary>Marker key types used by <c>Akka.Hosting</c> to resolve singletons from the registry.</summary>
|
/// <summary>Marker key types used by <c>Akka.Hosting</c> to resolve singletons from the registry.</summary>
|
||||||
|
|||||||
@@ -12,6 +12,7 @@ using ZB.MOM.WW.OtOpcUa.AdminUI.Grpc;
|
|||||||
using ZB.MOM.WW.OtOpcUa.AdminUI.Hubs;
|
using ZB.MOM.WW.OtOpcUa.AdminUI.Hubs;
|
||||||
using ZB.MOM.WW.OtOpcUa.AdminUI.ScriptAnalysis;
|
using ZB.MOM.WW.OtOpcUa.AdminUI.ScriptAnalysis;
|
||||||
using ZB.MOM.WW.OtOpcUa.Cluster;
|
using ZB.MOM.WW.OtOpcUa.Cluster;
|
||||||
|
using ZB.MOM.WW.OtOpcUa.Commons.Interfaces;
|
||||||
using ZB.MOM.WW.OtOpcUa.Configuration;
|
using ZB.MOM.WW.OtOpcUa.Configuration;
|
||||||
using ZB.MOM.WW.OtOpcUa.Configuration.Services;
|
using ZB.MOM.WW.OtOpcUa.Configuration.Services;
|
||||||
using ZB.MOM.WW.OtOpcUa.ControlPlane;
|
using ZB.MOM.WW.OtOpcUa.ControlPlane;
|
||||||
@@ -371,7 +372,18 @@ builder.Services.AddAkka("otopcua", (ab, sp) =>
|
|||||||
ab.WithOtOpcUaSignalRBridges();
|
ab.WithOtOpcUaSignalRBridges();
|
||||||
}
|
}
|
||||||
if (hasDriver)
|
if (hasDriver)
|
||||||
|
{
|
||||||
ab.WithOtOpcUaRuntimeActors();
|
ab.WithOtOpcUaRuntimeActors();
|
||||||
|
// Per-cluster mesh Phase 6: re-home the redundancy-state singleton off the admin node onto
|
||||||
|
// EVERY driver node, so each mesh elects its own pair-local Primary (a driver-only site pair
|
||||||
|
// has no admin node to do it). It scopes to this node's own cluster-{ClusterId} role, falling
|
||||||
|
// back to the driver role for legacy single-mesh / not-yet-migrated nodes (already pair-local
|
||||||
|
// on a split mesh, since the mesh IS the pair). Runs on the fused central node too (it is
|
||||||
|
// hasDriver), where it scopes to cluster-MAIN — exactly one redundancy singleton per node,
|
||||||
|
// since the admin branch above no longer registers one. IClusterRoleInfo is registered by
|
||||||
|
// AddOtOpcUaCluster and resolved lazily here.
|
||||||
|
ab.WithOtOpcUaClusterRedundancySingleton(sp.GetRequiredService<IClusterRoleInfo>());
|
||||||
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
// Down-if-alone recovery watchdog (#459). Registered AFTER AddAkka so it starts after Akka's own
|
// Down-if-alone recovery watchdog (#459). Registered AFTER AddAkka so it starts after Akka's own
|
||||||
|
|||||||
@@ -36,7 +36,7 @@ namespace ZB.MOM.WW.OtOpcUa.Runtime;
|
|||||||
|
|
||||||
public static class ServiceCollectionExtensions
|
public static class ServiceCollectionExtensions
|
||||||
{
|
{
|
||||||
public const string DriverRole = "driver";
|
public const string DriverRole = RoleParser.Driver;
|
||||||
|
|
||||||
public const string DriverHostActorName = "driver-host";
|
public const string DriverHostActorName = "driver-host";
|
||||||
public const string DbHealthProbeActorName = "db-health";
|
public const string DbHealthProbeActorName = "db-health";
|
||||||
|
|||||||
@@ -0,0 +1,105 @@
|
|||||||
|
using Akka.Actor;
|
||||||
|
using Akka.Hosting;
|
||||||
|
using Microsoft.Extensions.DependencyInjection;
|
||||||
|
using Microsoft.Extensions.Hosting;
|
||||||
|
using Microsoft.Extensions.Options;
|
||||||
|
using Shouldly;
|
||||||
|
using Xunit;
|
||||||
|
using ZB.MOM.WW.OtOpcUa.Commons.Interfaces;
|
||||||
|
|
||||||
|
namespace ZB.MOM.WW.OtOpcUa.Cluster.Tests;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Guards <see cref="ClusterRoleInfo.ClusterRole"/> / <see cref="ClusterRoleInfo.ClusterId"/> — the
|
||||||
|
/// node's OWN cluster-scoped role (per-cluster mesh, Phase 6), derived from the node's configured
|
||||||
|
/// <see cref="AkkaClusterOptions.Roles"/> at construction time, NOT from live cluster membership.
|
||||||
|
/// That distinction matters: this identity must be readable before the cluster forms, so later
|
||||||
|
/// Phase-6 work (singleton registration, a startup validator) can act on it at host-build time.
|
||||||
|
/// </summary>
|
||||||
|
public sealed class ClusterRoleInfoTests
|
||||||
|
{
|
||||||
|
private static AkkaClusterOptions SampleOptions(params string[] roles) => new()
|
||||||
|
{
|
||||||
|
Port = 0,
|
||||||
|
Hostname = "127.0.0.1",
|
||||||
|
PublicHostname = "127.0.0.1",
|
||||||
|
SeedNodes = Array.Empty<string>(),
|
||||||
|
Roles = roles,
|
||||||
|
};
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Boots a real host through the production <see cref="ServiceCollectionExtensions.WithOtOpcUaClusterBootstrap"/>
|
||||||
|
/// entry point (mirrors <c>SplitBrainResolverActivationTests</c>) — <see cref="ClusterRoleInfo"/>'s
|
||||||
|
/// constructor needs a real clustered <see cref="ActorSystem"/> to subscribe to cluster events,
|
||||||
|
/// even though the properties under test never touch live membership — resolves
|
||||||
|
/// <see cref="IClusterRoleInfo"/> from DI, and returns its <c>(ClusterRole, ClusterId)</c> pair
|
||||||
|
/// after tearing the host down. Both start/stop live in this helper rather than the test methods
|
||||||
|
/// so the CancellationToken-less calls stay off xunit1051's radar (mirrors
|
||||||
|
/// <c>SplitBrainResolverActivationTests.EffectiveConfigAsync</c>).
|
||||||
|
/// </summary>
|
||||||
|
private static async Task<(string? ClusterRole, string? ClusterId)> ResolveAsync(AkkaClusterOptions options)
|
||||||
|
{
|
||||||
|
var builder = Host.CreateDefaultBuilder();
|
||||||
|
builder.ConfigureServices(services =>
|
||||||
|
{
|
||||||
|
services.AddSingleton<IOptions<AkkaClusterOptions>>(Options.Create(options));
|
||||||
|
services.AddAkka("otopcua-role-info-test", (ab, sp) => ab.WithOtOpcUaClusterBootstrap(sp));
|
||||||
|
services.AddSingleton<IClusterRoleInfo, ClusterRoleInfo>();
|
||||||
|
});
|
||||||
|
|
||||||
|
using var host = builder.Build();
|
||||||
|
await host.StartAsync();
|
||||||
|
try
|
||||||
|
{
|
||||||
|
var roleInfo = host.Services.GetRequiredService<IClusterRoleInfo>();
|
||||||
|
return (roleInfo.ClusterRole, roleInfo.ClusterId);
|
||||||
|
}
|
||||||
|
finally
|
||||||
|
{
|
||||||
|
await host.StopAsync();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>A node configured with a single cluster-scoped role exposes it and its ClusterId.</summary>
|
||||||
|
[Fact]
|
||||||
|
public async Task Single_cluster_role_exposes_role_and_id()
|
||||||
|
{
|
||||||
|
var (clusterRole, clusterId) = await ResolveAsync(SampleOptions("driver", "cluster-SITE-A"));
|
||||||
|
|
||||||
|
clusterRole.ShouldBe("cluster-SITE-A");
|
||||||
|
clusterId.ShouldBe("SITE-A");
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>A node configured with only fixed roles carries no cluster identity.</summary>
|
||||||
|
[Fact]
|
||||||
|
public async Task No_cluster_role_configured_yields_null()
|
||||||
|
{
|
||||||
|
var (clusterRole, clusterId) = await ResolveAsync(SampleOptions("admin", "driver"));
|
||||||
|
|
||||||
|
clusterRole.ShouldBeNull();
|
||||||
|
clusterId.ShouldBeNull();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// More than one cluster-scoped role is a misconfiguration; the first configured wins rather
|
||||||
|
/// than throwing.
|
||||||
|
/// </summary>
|
||||||
|
[Fact]
|
||||||
|
public async Task Multiple_cluster_roles_first_wins()
|
||||||
|
{
|
||||||
|
var (clusterRole, clusterId) = await ResolveAsync(SampleOptions("driver", "cluster-MAIN", "cluster-SITE-B"));
|
||||||
|
|
||||||
|
clusterRole.ShouldBe("cluster-MAIN");
|
||||||
|
clusterId.ShouldBe("MAIN");
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>An empty role list carries no cluster identity either.</summary>
|
||||||
|
[Fact]
|
||||||
|
public async Task Empty_roles_yields_null()
|
||||||
|
{
|
||||||
|
var (clusterRole, clusterId) = await ResolveAsync(SampleOptions());
|
||||||
|
|
||||||
|
clusterRole.ShouldBeNull();
|
||||||
|
clusterId.ShouldBeNull();
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -57,4 +57,57 @@ public sealed class RoleParserTests
|
|||||||
{
|
{
|
||||||
Should.Throw<ArgumentException>(() => RoleParser.Parse("admin,master"));
|
Should.Throw<ArgumentException>(() => RoleParser.Parse("admin,master"));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// <summary>Verifies that a single cluster-scoped role is accepted.</summary>
|
||||||
|
[Theory]
|
||||||
|
[InlineData("cluster-MAIN")]
|
||||||
|
[InlineData("cluster-SITE-A")]
|
||||||
|
public void Cluster_role_accepted(string role)
|
||||||
|
{
|
||||||
|
RoleParser.Parse(role).ShouldBe(new[] { role.ToLowerInvariant() });
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>Verifies that cluster roles mix freely with the fixed roles.</summary>
|
||||||
|
[Fact]
|
||||||
|
public void Fixed_and_cluster_roles_mix()
|
||||||
|
{
|
||||||
|
RoleParser.Parse("driver,cluster-SITE-A").ShouldBe(new[] { "driver", "cluster-site-a" });
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>Verifies that a cluster role with an empty ClusterId ("cluster-") is rejected.</summary>
|
||||||
|
[Fact]
|
||||||
|
public void Empty_cluster_id_throws()
|
||||||
|
{
|
||||||
|
Should.Throw<ArgumentException>(() => RoleParser.Parse("cluster-"));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>Verifies IsClusterRole returns true for valid cluster roles and false otherwise.</summary>
|
||||||
|
[Theory]
|
||||||
|
[InlineData("cluster-MAIN", true)]
|
||||||
|
[InlineData("cluster-SITE-A", true)]
|
||||||
|
[InlineData("cluster-", false)]
|
||||||
|
[InlineData("driver", false)]
|
||||||
|
[InlineData("admin", false)]
|
||||||
|
[InlineData("clusterx", false)]
|
||||||
|
public void IsClusterRole_classifies_correctly(string role, bool expected)
|
||||||
|
{
|
||||||
|
RoleParser.IsClusterRole(role).ShouldBe(expected);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>Verifies ClusterIdFromRole extracts the ClusterId suffix.</summary>
|
||||||
|
[Fact]
|
||||||
|
public void ClusterIdFromRole_extracts_suffix()
|
||||||
|
{
|
||||||
|
RoleParser.ClusterIdFromRole("cluster-SITE-A").ShouldBe("SITE-A");
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>Verifies the three well-known role constants match the historical literal values.</summary>
|
||||||
|
[Fact]
|
||||||
|
public void Well_known_role_constants()
|
||||||
|
{
|
||||||
|
RoleParser.Admin.ShouldBe("admin");
|
||||||
|
RoleParser.Driver.ShouldBe("driver");
|
||||||
|
RoleParser.Dev.ShouldBe("dev");
|
||||||
|
RoleParser.ClusterRolePrefix.ShouldBe("cluster-");
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,267 @@
|
|||||||
|
using Microsoft.Extensions.Configuration;
|
||||||
|
using Microsoft.Extensions.Options;
|
||||||
|
using Shouldly;
|
||||||
|
using Xunit;
|
||||||
|
|
||||||
|
namespace ZB.MOM.WW.OtOpcUa.Cluster.Tests;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// A split-topology node (one carrying a <c>cluster-{ClusterId}</c> role) left on a DPS transport
|
||||||
|
/// mode does not error — the transport simply delivers nothing across the mesh boundary, and the
|
||||||
|
/// symptom is a deployment that never applies or telemetry that never arrives. These tests make
|
||||||
|
/// that misconfiguration a loud host-start failure instead. A node with no cluster-role is exempt
|
||||||
|
/// (legacy / single-mesh / test), and the validator is a no-op for it.
|
||||||
|
/// </summary>
|
||||||
|
public class SplitTopologyTransportValidatorTests
|
||||||
|
{
|
||||||
|
private static ValidateOptionsResult Validate(
|
||||||
|
string? meshMode,
|
||||||
|
string? telemetryMode,
|
||||||
|
string? telemetryDialMode,
|
||||||
|
params string[] roles)
|
||||||
|
{
|
||||||
|
var pairs = new Dictionary<string, string?>();
|
||||||
|
for (var i = 0; i < roles.Length; i++)
|
||||||
|
{
|
||||||
|
pairs[$"Cluster:Roles:{i}"] = roles[i];
|
||||||
|
}
|
||||||
|
|
||||||
|
if (meshMode is not null) pairs["MeshTransport:Mode"] = meshMode;
|
||||||
|
if (telemetryMode is not null) pairs["Telemetry:Mode"] = telemetryMode;
|
||||||
|
if (telemetryDialMode is not null) pairs["TelemetryDial:Mode"] = telemetryDialMode;
|
||||||
|
|
||||||
|
var configuration = new ConfigurationBuilder().AddInMemoryCollection(pairs).Build();
|
||||||
|
|
||||||
|
// The validated options instance is never read (all four values come from IConfiguration), so a
|
||||||
|
// default MeshTransportOptions is a fine stand-in.
|
||||||
|
return new SplitTopologyTransportValidator(configuration)
|
||||||
|
.Validate(MeshTransportOptions.SectionName, new MeshTransportOptions());
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void Cluster_role_driver_node_on_Dps_mesh_transport_fails()
|
||||||
|
{
|
||||||
|
// Red-before-green anchor: a split-topology node on DistributedPubSub cannot deliver commands.
|
||||||
|
var result = Validate(
|
||||||
|
meshMode: MeshTransportOptions.ModeDps,
|
||||||
|
telemetryMode: TelemetryOptions.ModeGrpc,
|
||||||
|
telemetryDialMode: null,
|
||||||
|
"driver", "cluster-SITE-A");
|
||||||
|
|
||||||
|
result.Failed.ShouldBeTrue();
|
||||||
|
result.FailureMessage.ShouldContain("MeshTransport:Mode");
|
||||||
|
result.FailureMessage.ShouldContain(MeshTransportOptions.ModeClusterClient);
|
||||||
|
result.FailureMessage.ShouldContain("SITE-A");
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void Cluster_role_driver_node_fully_configured_passes()
|
||||||
|
{
|
||||||
|
Validate(
|
||||||
|
meshMode: MeshTransportOptions.ModeClusterClient,
|
||||||
|
telemetryMode: TelemetryOptions.ModeGrpc,
|
||||||
|
telemetryDialMode: null,
|
||||||
|
"driver", "cluster-SITE-A").Succeeded.ShouldBeTrue();
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void Cluster_role_admin_node_fully_configured_passes()
|
||||||
|
{
|
||||||
|
Validate(
|
||||||
|
meshMode: MeshTransportOptions.ModeClusterClient,
|
||||||
|
telemetryMode: null,
|
||||||
|
telemetryDialMode: TelemetryDialOptions.ModeGrpc,
|
||||||
|
"admin", "cluster-SITE-A").Succeeded.ShouldBeTrue();
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void Cluster_role_driver_node_on_Dps_telemetry_fails()
|
||||||
|
{
|
||||||
|
var result = Validate(
|
||||||
|
meshMode: MeshTransportOptions.ModeClusterClient,
|
||||||
|
telemetryMode: TelemetryOptions.ModeDps,
|
||||||
|
telemetryDialMode: null,
|
||||||
|
"driver", "cluster-SITE-A");
|
||||||
|
|
||||||
|
result.Failed.ShouldBeTrue();
|
||||||
|
result.FailureMessage.ShouldContain("Telemetry:Mode");
|
||||||
|
result.FailureMessage.ShouldContain(TelemetryOptions.ModeGrpc);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void Cluster_role_admin_node_on_Dps_telemetry_dial_fails()
|
||||||
|
{
|
||||||
|
var result = Validate(
|
||||||
|
meshMode: MeshTransportOptions.ModeClusterClient,
|
||||||
|
telemetryMode: null,
|
||||||
|
telemetryDialMode: TelemetryDialOptions.ModeDps,
|
||||||
|
"admin", "cluster-SITE-A");
|
||||||
|
|
||||||
|
result.Failed.ShouldBeTrue();
|
||||||
|
result.FailureMessage.ShouldContain("TelemetryDial:Mode");
|
||||||
|
result.FailureMessage.ShouldContain(TelemetryDialOptions.ModeGrpc);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void Fused_cluster_role_node_needs_all_three_to_pass()
|
||||||
|
{
|
||||||
|
Validate(
|
||||||
|
meshMode: MeshTransportOptions.ModeClusterClient,
|
||||||
|
telemetryMode: TelemetryOptions.ModeGrpc,
|
||||||
|
telemetryDialMode: TelemetryDialOptions.ModeGrpc,
|
||||||
|
"admin", "driver", "cluster-MAIN").Succeeded.ShouldBeTrue();
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void Fused_cluster_role_node_missing_telemetry_grpc_fails()
|
||||||
|
{
|
||||||
|
var result = Validate(
|
||||||
|
meshMode: MeshTransportOptions.ModeClusterClient,
|
||||||
|
telemetryMode: TelemetryOptions.ModeDps,
|
||||||
|
telemetryDialMode: TelemetryDialOptions.ModeGrpc,
|
||||||
|
"admin", "driver", "cluster-MAIN");
|
||||||
|
|
||||||
|
result.Failed.ShouldBeTrue();
|
||||||
|
result.FailureMessage.ShouldContain("Telemetry:Mode");
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void Fused_cluster_role_node_missing_telemetry_dial_grpc_fails()
|
||||||
|
{
|
||||||
|
var result = Validate(
|
||||||
|
meshMode: MeshTransportOptions.ModeClusterClient,
|
||||||
|
telemetryMode: TelemetryOptions.ModeGrpc,
|
||||||
|
telemetryDialMode: TelemetryDialOptions.ModeDps,
|
||||||
|
"admin", "driver", "cluster-MAIN");
|
||||||
|
|
||||||
|
result.Failed.ShouldBeTrue();
|
||||||
|
result.FailureMessage.ShouldContain("TelemetryDial:Mode");
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void Non_cluster_role_node_on_all_Dps_passes()
|
||||||
|
{
|
||||||
|
// The exemption that keeps the validator a no-op for every legacy / single-mesh / test node:
|
||||||
|
// no cluster-{ClusterId} role means the split-topology rule does not apply at all.
|
||||||
|
Validate(
|
||||||
|
meshMode: MeshTransportOptions.ModeDps,
|
||||||
|
telemetryMode: TelemetryOptions.ModeDps,
|
||||||
|
telemetryDialMode: TelemetryDialOptions.ModeDps,
|
||||||
|
"admin", "driver").Succeeded.ShouldBeTrue();
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void Node_with_no_roles_on_all_Dps_passes()
|
||||||
|
{
|
||||||
|
// Absolute default: nothing declared, nothing to enforce.
|
||||||
|
Validate(
|
||||||
|
meshMode: MeshTransportOptions.ModeDps,
|
||||||
|
telemetryMode: TelemetryOptions.ModeDps,
|
||||||
|
telemetryDialMode: TelemetryDialOptions.ModeDps).Succeeded.ShouldBeTrue();
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void Unset_modes_on_a_cluster_role_node_fail_as_the_Dps_default()
|
||||||
|
{
|
||||||
|
// A split node that never sets the mode keys inherits the Dps defaults — which the split
|
||||||
|
// topology cannot use. Leaving them unset must fail exactly as setting them to Dps would.
|
||||||
|
var result = Validate(
|
||||||
|
meshMode: null,
|
||||||
|
telemetryMode: null,
|
||||||
|
telemetryDialMode: null,
|
||||||
|
"admin", "driver", "cluster-MAIN");
|
||||||
|
|
||||||
|
result.Failed.ShouldBeTrue();
|
||||||
|
result.FailureMessage.ShouldContain("MeshTransport:Mode");
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void Mode_matching_is_case_insensitive()
|
||||||
|
{
|
||||||
|
Validate(
|
||||||
|
meshMode: "clusterclient",
|
||||||
|
telemetryMode: "grpc",
|
||||||
|
telemetryDialMode: "grpc",
|
||||||
|
"admin", "driver", "cluster-MAIN").Succeeded.ShouldBeTrue();
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void Cluster_role_from_OTOPCUA_ROLES_only_on_Dps_fails()
|
||||||
|
{
|
||||||
|
// Roles-source divergence: Cluster:Roles is empty, so the node's effective Akka roles come from
|
||||||
|
// OTOPCUA_ROLES (via RoleParser.Parse) exactly as Program.cs resolves them. A validator that
|
||||||
|
// read only Cluster:Roles would see no cluster role and pass silently on Dps — the documented
|
||||||
|
// production-incident shape.
|
||||||
|
using (new EnvVarScope("OTOPCUA_ROLES", "driver,cluster-SITE-A"))
|
||||||
|
{
|
||||||
|
var pairs = new Dictionary<string, string?> { ["MeshTransport:Mode"] = MeshTransportOptions.ModeDps };
|
||||||
|
var configuration = new ConfigurationBuilder().AddInMemoryCollection(pairs).Build();
|
||||||
|
|
||||||
|
var result = new SplitTopologyTransportValidator(configuration)
|
||||||
|
.Validate(MeshTransportOptions.SectionName, new MeshTransportOptions());
|
||||||
|
|
||||||
|
result.Failed.ShouldBeTrue();
|
||||||
|
result.FailureMessage.ShouldContain("MeshTransport:Mode");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void Cluster_role_from_OTOPCUA_ROLES_is_ignored_when_Cluster_Roles_is_set()
|
||||||
|
{
|
||||||
|
// When Cluster:Roles is populated it wins outright — the OTOPCUA_ROLES fallback is not consulted,
|
||||||
|
// mirroring AkkaClusterOptions.Roles. A non-cluster Cluster:Roles therefore stays exempt even if
|
||||||
|
// a stray OTOPCUA_ROLES carries a cluster role.
|
||||||
|
using (new EnvVarScope("OTOPCUA_ROLES", "driver,cluster-SITE-A"))
|
||||||
|
{
|
||||||
|
Validate(
|
||||||
|
meshMode: MeshTransportOptions.ModeDps,
|
||||||
|
telemetryMode: MeshTransportOptions.ModeDps,
|
||||||
|
telemetryDialMode: MeshTransportOptions.ModeDps,
|
||||||
|
"admin", "driver").Succeeded.ShouldBeTrue();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void Mixed_case_cluster_and_role_names_on_Dps_fail()
|
||||||
|
{
|
||||||
|
// Case-sensitivity: the Cluster:Roles bind path is verbatim, so mixed-case entries must be
|
||||||
|
// normalized here or they slip past the ordinal IsClusterRole / driver checks and pass silently.
|
||||||
|
var result = Validate(
|
||||||
|
meshMode: MeshTransportOptions.ModeDps,
|
||||||
|
telemetryMode: TelemetryOptions.ModeGrpc,
|
||||||
|
telemetryDialMode: null,
|
||||||
|
"Driver", "Cluster-SITE-A");
|
||||||
|
|
||||||
|
result.Failed.ShouldBeTrue();
|
||||||
|
result.FailureMessage.ShouldContain("MeshTransport:Mode");
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void Cluster_role_node_with_neither_admin_nor_driver_only_needs_ClusterClient()
|
||||||
|
{
|
||||||
|
// A cluster-role node that is neither admin nor driver has no telemetry surface to constrain —
|
||||||
|
// only the always-on MeshTransport=ClusterClient rule applies. Telemetry/TelemetryDial left on
|
||||||
|
// their Dps defaults must not fail it.
|
||||||
|
Validate(
|
||||||
|
meshMode: MeshTransportOptions.ModeClusterClient,
|
||||||
|
telemetryMode: null,
|
||||||
|
telemetryDialMode: null,
|
||||||
|
"cluster-SITE-A").Succeeded.ShouldBeTrue();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>Sets an environment variable for the scope's lifetime and restores its prior value on dispose.</summary>
|
||||||
|
private sealed class EnvVarScope : IDisposable
|
||||||
|
{
|
||||||
|
private readonly string _name;
|
||||||
|
private readonly string? _previous;
|
||||||
|
|
||||||
|
public EnvVarScope(string name, string? value)
|
||||||
|
{
|
||||||
|
_name = name;
|
||||||
|
_previous = Environment.GetEnvironmentVariable(name);
|
||||||
|
Environment.SetEnvironmentVariable(name, value);
|
||||||
|
}
|
||||||
|
|
||||||
|
public void Dispose() => Environment.SetEnvironmentVariable(_name, _previous);
|
||||||
|
}
|
||||||
|
}
|
||||||
+67
-11
@@ -70,24 +70,80 @@ public sealed class ClusterNodeAddressReconcilerActorTests : ControlPlaneActorTe
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Per-cluster mesh Phase 6: an admin node scoped to its OWN cluster ignores rows in a cluster
|
||||||
|
/// its mesh can no longer see. Scoped to <c>MAIN</c>, a <c>SITE-A</c> enabled row is filtered
|
||||||
|
/// out before <see cref="ClusterNodeAddressReconciler.Reconcile"/> ever sees it, so it produces
|
||||||
|
/// no <see cref="AddressMismatchKind.EnabledRowNotInCluster"/> warning. This is the false
|
||||||
|
/// positive the split would otherwise print forever.
|
||||||
|
/// </summary>
|
||||||
|
[Fact]
|
||||||
|
public void Row_in_a_foreign_cluster_is_ignored_when_scoped()
|
||||||
|
{
|
||||||
|
var dbFactory = NewInMemoryDbFactory();
|
||||||
|
SeedNode(dbFactory, "site-a-1:4053", enabled: true, clusterId: "SITE-A");
|
||||||
|
|
||||||
|
EventFilter.Warning(contains: "site-a-1:4053").Expect(0, TimeSpan.FromSeconds(2), () =>
|
||||||
|
{
|
||||||
|
Sys.ActorOf(ClusterNodeAddressReconcilerActor.Props(
|
||||||
|
dbFactory, ownClusterId: "MAIN", sweepInterval: TimeSpan.FromMilliseconds(400)));
|
||||||
|
ExpectNoMsg(TimeSpan.FromSeconds(1.5));
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Scoping does not silence the admin node's own cluster. Scoped to <c>MAIN</c>, an enabled
|
||||||
|
/// <c>MAIN</c> row with no matching driver member is still warned about — proving the filter
|
||||||
|
/// keeps in-cluster rows (and with them, within-cluster drift detection).
|
||||||
|
/// </summary>
|
||||||
|
[Fact]
|
||||||
|
public void Enabled_row_in_own_cluster_is_still_logged_when_scoped()
|
||||||
|
{
|
||||||
|
var dbFactory = NewInMemoryDbFactory();
|
||||||
|
SeedNode(dbFactory, "main-1:4053", enabled: true, clusterId: "MAIN");
|
||||||
|
|
||||||
|
EventFilter.Warning(contains: "main-1:4053").ExpectOne(TimeSpan.FromSeconds(10), () =>
|
||||||
|
Sys.ActorOf(ClusterNodeAddressReconcilerActor.Props(dbFactory, ownClusterId: "MAIN")));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Legacy backward-compat: an admin node with no cluster role (null <c>ownClusterId</c>) still
|
||||||
|
/// reconciles the WHOLE fleet — such a node genuinely sees every member via gossip. A
|
||||||
|
/// <c>SITE-A</c> enabled row with no member is warned about, exactly as before the Phase 6 scope.
|
||||||
|
/// </summary>
|
||||||
|
[Fact]
|
||||||
|
public void Foreign_cluster_row_is_still_logged_when_unscoped()
|
||||||
|
{
|
||||||
|
var dbFactory = NewInMemoryDbFactory();
|
||||||
|
SeedNode(dbFactory, "site-a-1:4053", enabled: true, clusterId: "SITE-A");
|
||||||
|
|
||||||
|
EventFilter.Warning(contains: "site-a-1:4053").ExpectOne(TimeSpan.FromSeconds(10), () =>
|
||||||
|
Sys.ActorOf(ClusterNodeAddressReconcilerActor.Props(dbFactory, ownClusterId: null)));
|
||||||
|
}
|
||||||
|
|
||||||
private static void SeedNode(
|
private static void SeedNode(
|
||||||
IDbContextFactory<OtOpcUaConfigDbContext> dbFactory, string nodeId, bool enabled)
|
IDbContextFactory<OtOpcUaConfigDbContext> dbFactory, string nodeId, bool enabled,
|
||||||
|
string clusterId = "SITE-A")
|
||||||
{
|
{
|
||||||
using var db = dbFactory.CreateDbContext();
|
using var db = dbFactory.CreateDbContext();
|
||||||
db.ServerClusters.Add(new ServerCluster
|
if (!db.ServerClusters.Any(c => c.ClusterId == clusterId))
|
||||||
{
|
{
|
||||||
ClusterId = "SITE-A",
|
db.ServerClusters.Add(new ServerCluster
|
||||||
Name = "Site A",
|
{
|
||||||
Enterprise = "zb",
|
ClusterId = clusterId,
|
||||||
Site = "site-a",
|
Name = clusterId,
|
||||||
NodeCount = 2,
|
Enterprise = "zb",
|
||||||
RedundancyMode = RedundancyMode.Warm,
|
Site = clusterId.ToLowerInvariant(),
|
||||||
CreatedBy = "test",
|
NodeCount = 2,
|
||||||
});
|
RedundancyMode = RedundancyMode.Warm,
|
||||||
|
CreatedBy = "test",
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
db.ClusterNodes.Add(new ClusterNode
|
db.ClusterNodes.Add(new ClusterNode
|
||||||
{
|
{
|
||||||
NodeId = nodeId,
|
NodeId = nodeId,
|
||||||
ClusterId = "SITE-A",
|
ClusterId = clusterId,
|
||||||
Host = nodeId.Split(':')[0],
|
Host = nodeId.Split(':')[0],
|
||||||
ApplicationUri = $"urn:OtOpcUa:{nodeId}",
|
ApplicationUri = $"urn:OtOpcUa:{nodeId}",
|
||||||
Enabled = enabled,
|
Enabled = enabled,
|
||||||
|
|||||||
+91
@@ -1,3 +1,8 @@
|
|||||||
|
using System.Collections.Immutable;
|
||||||
|
using System.Reflection;
|
||||||
|
using Akka.Actor;
|
||||||
|
using Akka.Cluster;
|
||||||
|
using Akka.Util;
|
||||||
using Shouldly;
|
using Shouldly;
|
||||||
using Xunit;
|
using Xunit;
|
||||||
using ZB.MOM.WW.OtOpcUa.ControlPlane.Fleet;
|
using ZB.MOM.WW.OtOpcUa.ControlPlane.Fleet;
|
||||||
@@ -123,4 +128,90 @@ public sealed class ClusterNodeAddressReconcilerTests
|
|||||||
[("SITE-A-1", 4053)], [Row("site-a-1", 4053)], ["site-a-1:4053"])
|
[("SITE-A-1", 4053)], [Row("site-a-1", 4053)], ["site-a-1:4053"])
|
||||||
.ShouldBeEmpty();
|
.ShouldBeEmpty();
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private static int _memberUid;
|
||||||
|
|
||||||
|
// Akka's Member.Create factories are internal, so reflect the Up-status overload
|
||||||
|
// Create(UniqueAddress, int upNumber, MemberStatus, ImmutableHashSet<string> roles, AppVersion).
|
||||||
|
private static readonly MethodInfo MemberCreate = typeof(Member).GetMethod(
|
||||||
|
"Create",
|
||||||
|
BindingFlags.Static | BindingFlags.NonPublic | BindingFlags.Public,
|
||||||
|
binder: null,
|
||||||
|
types:
|
||||||
|
[
|
||||||
|
typeof(UniqueAddress), typeof(int), typeof(MemberStatus),
|
||||||
|
typeof(ImmutableHashSet<string>), typeof(AppVersion),
|
||||||
|
],
|
||||||
|
modifiers: null)
|
||||||
|
?? throw new InvalidOperationException("Akka.Cluster.Member.Create(5-arg) not found — Akka API changed");
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Builds an Up cluster <see cref="Member"/> at <paramref name="host"/>:<paramref name="port"/>
|
||||||
|
/// carrying <paramref name="roles"/>, for exercising
|
||||||
|
/// <see cref="ClusterNodeAddressReconcilerActor.FilterOwnClusterDriverMembers"/> directly.
|
||||||
|
/// </summary>
|
||||||
|
private static Member UpMember(string host, int port, params string[] roles) =>
|
||||||
|
(Member)MemberCreate.Invoke(null,
|
||||||
|
[
|
||||||
|
new UniqueAddress(new Address("akka.tcp", "otopcua", host, port), Interlocked.Increment(ref _memberUid)),
|
||||||
|
1,
|
||||||
|
MemberStatus.Up,
|
||||||
|
roles.ToImmutableHashSet(),
|
||||||
|
AppVersion.Zero,
|
||||||
|
])!;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Per-cluster mesh Phase 6: scoped to <c>cluster-MAIN</c>, a driver member carrying a FOREIGN
|
||||||
|
/// cluster role (<c>cluster-SITE-A</c>) — the mixed-mesh window, where central still sees it via
|
||||||
|
/// gossip while its row is already filtered out by ClusterId — is excluded from observed. Left
|
||||||
|
/// in, it would hit the Error-level <see cref="AddressMismatchKind.RunningNodeHasNoRow"/> branch.
|
||||||
|
/// </summary>
|
||||||
|
[Fact]
|
||||||
|
public void Foreign_cluster_role_member_is_excluded_when_scoped()
|
||||||
|
{
|
||||||
|
var observed = ClusterNodeAddressReconcilerActor.FilterOwnClusterDriverMembers(
|
||||||
|
[UpMember("site-a-1", 4053, "driver", "cluster-SITE-A")],
|
||||||
|
ownClusterRole: "cluster-MAIN");
|
||||||
|
|
||||||
|
observed.ShouldBeEmpty();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>Scoped to <c>cluster-MAIN</c>, a member carrying that same role is kept.</summary>
|
||||||
|
[Fact]
|
||||||
|
public void Own_cluster_role_member_is_kept_when_scoped()
|
||||||
|
{
|
||||||
|
var observed = ClusterNodeAddressReconcilerActor.FilterOwnClusterDriverMembers(
|
||||||
|
[UpMember("main-1", 4053, "driver", "cluster-MAIN")],
|
||||||
|
ownClusterRole: "cluster-MAIN");
|
||||||
|
|
||||||
|
observed.ShouldHaveSingleItem().ShouldBe(("main-1", 4053));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Legacy: a null cluster role keeps every Up driver member regardless of cluster — an admin
|
||||||
|
/// node with no cluster role genuinely sees the whole fleet via gossip (reconcile-all).
|
||||||
|
/// </summary>
|
||||||
|
[Fact]
|
||||||
|
public void Null_cluster_role_keeps_all_driver_members()
|
||||||
|
{
|
||||||
|
var observed = ClusterNodeAddressReconcilerActor.FilterOwnClusterDriverMembers(
|
||||||
|
[
|
||||||
|
UpMember("main-1", 4053, "driver", "cluster-MAIN"),
|
||||||
|
UpMember("site-a-1", 4053, "driver", "cluster-SITE-A"),
|
||||||
|
],
|
||||||
|
ownClusterRole: null);
|
||||||
|
|
||||||
|
observed.Count.ShouldBe(2);
|
||||||
|
observed.ShouldContain(("main-1", 4053));
|
||||||
|
observed.ShouldContain(("site-a-1", 4053));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>Non-driver members (e.g. admin-only) are excluded whether or not a scope is set.</summary>
|
||||||
|
[Fact]
|
||||||
|
public void Non_driver_members_are_always_excluded()
|
||||||
|
{
|
||||||
|
ClusterNodeAddressReconcilerActor.FilterOwnClusterDriverMembers(
|
||||||
|
[UpMember("central-1", 4053, "admin", "cluster-MAIN")], ownClusterRole: "cluster-MAIN")
|
||||||
|
.ShouldBeEmpty();
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
+47
-9
@@ -31,13 +31,20 @@ public class CentralCommunicationActorTests : ControlPlaneActorTestBase
|
|||||||
|
|
||||||
public List<ImmutableHashSet<ActorPath>> Calls { get; } = [];
|
public List<ImmutableHashSet<ActorPath>> Calls { get; } = [];
|
||||||
|
|
||||||
|
public List<string> ClusterIds { get; } = [];
|
||||||
|
|
||||||
public List<TestProbe> Probes { get; } = [];
|
public List<TestProbe> Probes { get; } = [];
|
||||||
|
|
||||||
public IActorRef Create(ActorSystem system, ImmutableHashSet<ActorPath> contacts)
|
/// <summary>Probe created for a given cluster id, for per-cluster fan-out assertions.</summary>
|
||||||
|
public Dictionary<string, TestProbe> ProbeByCluster { get; } = [];
|
||||||
|
|
||||||
|
public IActorRef Create(ActorSystem system, string clusterId, ImmutableHashSet<ActorPath> contacts)
|
||||||
{
|
{
|
||||||
Calls.Add(contacts);
|
Calls.Add(contacts);
|
||||||
|
ClusterIds.Add(clusterId);
|
||||||
var probe = _newProbe();
|
var probe = _newProbe();
|
||||||
Probes.Add(probe);
|
Probes.Add(probe);
|
||||||
|
ProbeByCluster[clusterId] = probe;
|
||||||
return probe.Ref;
|
return probe.Ref;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -138,23 +145,54 @@ public class CentralCommunicationActorTests : ControlPlaneActorTestBase
|
|||||||
}
|
}
|
||||||
|
|
||||||
[Fact]
|
[Fact]
|
||||||
public void Exactly_one_client_is_created_for_a_multi_cluster_fleet()
|
public void One_client_per_cluster_is_created_for_a_multi_cluster_fleet()
|
||||||
{
|
{
|
||||||
// A receptionist serves its whole mesh, so one client per Cluster would fan each command out
|
// THE Phase 6 inversion. Each application Cluster is now its own Akka mesh, and a ClusterClient
|
||||||
// once per cluster while the fleet is still a single mesh — N x duplicate deployments.
|
// reaches only the mesh whose receptionist it dialled. Central must therefore build one client
|
||||||
|
// per cluster, each dialling only that cluster's nodes.
|
||||||
var factory = NewInMemoryDbFactory();
|
var factory = NewInMemoryDbFactory();
|
||||||
SeedCluster(factory, "C1", ("host-a:4053", "host-a", 4053, true, false));
|
SeedCluster(factory, "C1", ("host-a:4053", "host-a", 4053, true, false));
|
||||||
SeedCluster(factory, "C2", ("host-b:4053", "host-b", 4053, true, false));
|
SeedCluster(factory, "C2", ("host-b:4053", "host-b", 4053, true, false));
|
||||||
var clients = new RecordingClientFactory(() => CreateTestProbe());
|
var clients = new RecordingClientFactory(() => CreateTestProbe());
|
||||||
|
|
||||||
Spawn(factory, ClusterClientMode(), clients);
|
Spawn(factory, ClusterClientMode(), clients);
|
||||||
AwaitCondition(() => clients.Calls.Count == 1, TimeSpan.FromSeconds(5));
|
AwaitCondition(() => clients.Calls.Count == 2, TimeSpan.FromSeconds(5));
|
||||||
|
|
||||||
// ...and that one client dials BOTH clusters' nodes.
|
// Give a second refresh a chance to (wrongly) create more.
|
||||||
clients.Calls[0].Count.ShouldBe(2);
|
|
||||||
// Give a second refresh a chance to (wrongly) create another.
|
|
||||||
ExpectNoMsg(TimeSpan.FromMilliseconds(300));
|
ExpectNoMsg(TimeSpan.FromMilliseconds(300));
|
||||||
clients.Calls.Count.ShouldBe(1);
|
clients.Calls.Count.ShouldBe(2);
|
||||||
|
clients.ClusterIds.ShouldBe(new[] { "C1", "C2" }, ignoreOrder: true);
|
||||||
|
|
||||||
|
// Each client dials ONLY its own cluster's node.
|
||||||
|
clients.Calls[clients.ClusterIds.IndexOf("C1")].Single().ToString().ShouldContain("host-a");
|
||||||
|
clients.Calls[clients.ClusterIds.IndexOf("C2")].Single().ToString().ShouldContain("host-b");
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void A_command_fans_out_a_send_to_all_to_every_cluster_client()
|
||||||
|
{
|
||||||
|
// The other half of the inversion: one MeshCommand must reach EVERY cluster's mesh, so central
|
||||||
|
// fans SendToAll across all per-cluster clients. A single Send/one-client shape would leave
|
||||||
|
// every cluster but one silently on its old configuration.
|
||||||
|
var factory = NewInMemoryDbFactory();
|
||||||
|
SeedCluster(factory, "C1", ("host-a:4053", "host-a", 4053, true, false));
|
||||||
|
SeedCluster(factory, "C2", ("host-b:4053", "host-b", 4053, true, false));
|
||||||
|
var clients = new RecordingClientFactory(() => CreateTestProbe());
|
||||||
|
|
||||||
|
var actor = Spawn(factory, ClusterClientMode(), clients);
|
||||||
|
AwaitCondition(() => clients.Probes.Count == 2, TimeSpan.FromSeconds(5));
|
||||||
|
|
||||||
|
var dispatch = new DispatchDeployment(
|
||||||
|
new DeploymentId(Guid.NewGuid()), new RevisionHash("abc"), new CorrelationId(Guid.NewGuid()));
|
||||||
|
actor.Tell(new MeshCommand(DeploymentsTopic, dispatch));
|
||||||
|
|
||||||
|
foreach (var (clusterId, probe) in clients.ProbeByCluster)
|
||||||
|
{
|
||||||
|
var sent = probe.ExpectMsg<ClusterClient.SendToAll>(
|
||||||
|
TimeSpan.FromSeconds(5), $"cluster {clusterId} must receive the fanned SendToAll");
|
||||||
|
sent.Path.ShouldBe(MeshPaths.NodeCommunication);
|
||||||
|
sent.Message.ShouldBe(dispatch);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
[Fact]
|
[Fact]
|
||||||
|
|||||||
+18
-3
@@ -37,13 +37,13 @@ public class MeshClusterClientFactoryTests : TestKit
|
|||||||
{
|
{
|
||||||
var factory = new DefaultMeshClusterClientFactory();
|
var factory = new DefaultMeshClusterClientFactory();
|
||||||
|
|
||||||
var first = factory.Create(Sys, UnreachableContact);
|
var first = factory.Create(Sys, "C1", UnreachableContact);
|
||||||
Sys.Stop(first);
|
Sys.Stop(first);
|
||||||
|
|
||||||
// Stop is asynchronous: `first`'s name is still reserved right here. Without the generation
|
// Stop is asynchronous: `first`'s name is still reserved right here. Without the generation
|
||||||
// suffix this throws InvalidActorNameException — which is precisely what a contact-set
|
// suffix this throws InvalidActorNameException — which is precisely what a contact-set
|
||||||
// change would do in production, where stop-then-recreate happens in one handler.
|
// change would do in production, where stop-then-recreate happens in one handler.
|
||||||
var second = factory.Create(Sys, UnreachableContact);
|
var second = factory.Create(Sys, "C1", UnreachableContact);
|
||||||
|
|
||||||
second.Path.Name.ShouldNotBe(first.Path.Name);
|
second.Path.Name.ShouldNotBe(first.Path.Name);
|
||||||
}
|
}
|
||||||
@@ -54,12 +54,27 @@ public class MeshClusterClientFactoryTests : TestKit
|
|||||||
var factory = new DefaultMeshClusterClientFactory();
|
var factory = new DefaultMeshClusterClientFactory();
|
||||||
|
|
||||||
var names = Enumerable.Range(0, 5)
|
var names = Enumerable.Range(0, 5)
|
||||||
.Select(_ => factory.Create(Sys, UnreachableContact).Path.Name)
|
.Select(_ => factory.Create(Sys, "C1", UnreachableContact).Path.Name)
|
||||||
.ToList();
|
.ToList();
|
||||||
|
|
||||||
names.Distinct().Count().ShouldBe(5);
|
names.Distinct().Count().ShouldBe(5);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public void The_cluster_id_is_woven_into_the_client_actor_name()
|
||||||
|
{
|
||||||
|
// Per-cluster clients must be distinguishable in logs and the actor tree, and — more
|
||||||
|
// importantly — two clusters' clients created on one system must never collide on a name.
|
||||||
|
var factory = new DefaultMeshClusterClientFactory();
|
||||||
|
|
||||||
|
var a = factory.Create(Sys, "site-a", UnreachableContact);
|
||||||
|
var b = factory.Create(Sys, "site-b", UnreachableContact);
|
||||||
|
|
||||||
|
a.Path.Name.ShouldContain("site-a");
|
||||||
|
b.Path.Name.ShouldContain("site-b");
|
||||||
|
a.Path.Name.ShouldNotBe(b.Path.Name);
|
||||||
|
}
|
||||||
|
|
||||||
// Contact-set propagation is deliberately NOT asserted here: reading initial contacts back
|
// Contact-set propagation is deliberately NOT asserted here: reading initial contacts back
|
||||||
// off a created ClusterClient is not possible, and asserting ClusterClientSettings directly
|
// off a created ClusterClient is not possible, and asserting ClusterClientSettings directly
|
||||||
// would test Akka rather than this factory. It is covered where it can actually fail — the
|
// would test Akka rather than this factory. It is covered where it can actually fail — the
|
||||||
|
|||||||
+182
@@ -0,0 +1,182 @@
|
|||||||
|
using Akka.Actor;
|
||||||
|
using Akka.Cluster.Hosting;
|
||||||
|
using Akka.Hosting;
|
||||||
|
using Microsoft.EntityFrameworkCore;
|
||||||
|
using Microsoft.Extensions.DependencyInjection;
|
||||||
|
using Microsoft.Extensions.Hosting;
|
||||||
|
using Microsoft.Extensions.Options;
|
||||||
|
using Shouldly;
|
||||||
|
using Xunit;
|
||||||
|
using ZB.MOM.WW.OtOpcUa.Cluster;
|
||||||
|
using ZB.MOM.WW.OtOpcUa.Commons.Interfaces;
|
||||||
|
using ZB.MOM.WW.OtOpcUa.Commons.Types;
|
||||||
|
using ZB.MOM.WW.OtOpcUa.Configuration;
|
||||||
|
|
||||||
|
namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Tests;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Guards Phase 6's re-homing of <c>RedundancyStateActor</c> off the fleet-wide admin singleton onto
|
||||||
|
/// a per-node <c>cluster-{ClusterId}</c> singleton (see
|
||||||
|
/// <see cref="ServiceCollectionExtensions.WithOtOpcUaClusterRedundancySingleton"/>). After the mesh
|
||||||
|
/// split a driver-only site pair has NO admin node to elect its Primary, so the singleton must be
|
||||||
|
/// scoped to the node's own cluster role and spawned on every driver node — one per mesh.
|
||||||
|
///
|
||||||
|
/// <para>The role scope + the missing-role guard are asserted against the extracted
|
||||||
|
/// <see cref="ServiceCollectionExtensions.BuildClusterRedundancySingletonOptions"/> helper directly —
|
||||||
|
/// the same pattern <c>SplitBrainResolverActivationTests</c> uses for <c>BuildDowningHocon</c> /
|
||||||
|
/// <c>BuildClusterOptions</c>: it pins the exact <see cref="ClusterSingletonOptions.Role"/> without
|
||||||
|
/// booting a cluster, which no builder-introspection API can otherwise reach. The admin-removal claim
|
||||||
|
/// is behavioral (a booted admin node's <see cref="ActorRegistry"/> must no longer carry the
|
||||||
|
/// redundancy key) so it needs a real host.</para>
|
||||||
|
/// </summary>
|
||||||
|
public sealed class RedundancyStateSingletonRehomeTests
|
||||||
|
{
|
||||||
|
/// <summary>The singleton options scope to the node's OWN cluster role, not the fixed admin role.</summary>
|
||||||
|
[Fact]
|
||||||
|
public void Options_scope_the_singleton_to_the_nodes_cluster_role()
|
||||||
|
{
|
||||||
|
var options = ServiceCollectionExtensions.BuildClusterRedundancySingletonOptions(
|
||||||
|
new FakeClusterRoleInfo(clusterRole: "cluster-SITE-A", clusterId: "SITE-A"));
|
||||||
|
|
||||||
|
options.Role.ShouldBe("cluster-SITE-A");
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// A node with no <c>cluster-{ClusterId}</c> role (a legacy single-mesh node, or the
|
||||||
|
/// <c>TwoNodeClusterHarness</c> with roles <c>admin,driver</c>) falls back to the fixed
|
||||||
|
/// <c>driver</c> role rather than throwing — so legacy/not-yet-migrated deployments keep booting.
|
||||||
|
/// On a split 2-node mesh the <c>driver</c> role is already pair-local (the mesh IS the pair).
|
||||||
|
/// </summary>
|
||||||
|
[Fact]
|
||||||
|
public void Missing_cluster_role_falls_back_to_the_driver_role()
|
||||||
|
{
|
||||||
|
var options = ServiceCollectionExtensions.BuildClusterRedundancySingletonOptions(
|
||||||
|
new FakeClusterRoleInfo(clusterRole: null, clusterId: null));
|
||||||
|
|
||||||
|
options.Role.ShouldBe(RoleParser.Driver);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// The admin singleton set no longer registers <c>redundancy-state</c>: a booted admin node's
|
||||||
|
/// registry carries the OTHER admin singletons (proxy) but NOT the redundancy key.
|
||||||
|
/// </summary>
|
||||||
|
[Fact]
|
||||||
|
public async Task Admin_singletons_no_longer_register_the_redundancy_singleton()
|
||||||
|
{
|
||||||
|
var registry = await BootAndGetRegistryAsync(
|
||||||
|
roles: new[] { "admin" },
|
||||||
|
configure: ab => ab.WithOtOpcUaControlPlaneSingletons());
|
||||||
|
|
||||||
|
registry.TryGet<RedundancyStateActorKey>(out _).ShouldBeFalse(
|
||||||
|
"the redundancy singleton was re-homed off the admin node onto every driver node (Phase 6)");
|
||||||
|
registry.TryGet<FleetStatusBroadcasterKey>(out _).ShouldBeTrue(
|
||||||
|
"the remaining admin singletons must still register");
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// The re-home target: a driver node carrying a <c>cluster-{ClusterId}</c> role registers the
|
||||||
|
/// redundancy singleton (proxy) via the new extension.
|
||||||
|
/// </summary>
|
||||||
|
[Fact]
|
||||||
|
public async Task Cluster_redundancy_extension_registers_the_singleton_on_a_driver_node()
|
||||||
|
{
|
||||||
|
var registry = await BootAndGetRegistryAsync(
|
||||||
|
roles: new[] { "driver", "cluster-SITE-A" },
|
||||||
|
configure: ab => ab.WithOtOpcUaClusterRedundancySingleton(
|
||||||
|
new FakeClusterRoleInfo(clusterRole: "cluster-SITE-A", clusterId: "SITE-A")));
|
||||||
|
|
||||||
|
registry.TryGet<RedundancyStateActorKey>(out _).ShouldBeTrue(
|
||||||
|
"every driver node hosts its own mesh's redundancy singleton (Phase 6)");
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// The driver-role FALLBACK end-to-end: a node carrying only <c>driver</c> (no cluster role — a
|
||||||
|
/// legacy single-mesh node or the <c>TwoNodeClusterHarness</c>) still boots and registers the
|
||||||
|
/// redundancy singleton via the new extension. Proves the fallback works at the host/registry
|
||||||
|
/// level, not just in the pure helper — the boot that the earlier throw-based version would have
|
||||||
|
/// aborted.
|
||||||
|
/// </summary>
|
||||||
|
[Fact]
|
||||||
|
public async Task Driver_role_fallback_registers_the_singleton_end_to_end()
|
||||||
|
{
|
||||||
|
var registry = await BootAndGetRegistryAsync(
|
||||||
|
roles: new[] { "driver" },
|
||||||
|
configure: ab => ab.WithOtOpcUaClusterRedundancySingleton(
|
||||||
|
new FakeClusterRoleInfo(clusterRole: null, clusterId: null)));
|
||||||
|
|
||||||
|
registry.TryGet<RedundancyStateActorKey>(out _).ShouldBeTrue(
|
||||||
|
"a driver node with no cluster role falls back to the driver role and still boots + registers");
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Boots a real host through the production cluster bootstrap (Port=0, no seeds, so it never forms
|
||||||
|
/// a cluster and no singleton props factory runs — only the proxies register), applies the
|
||||||
|
/// supplied singleton registration, and returns the <see cref="ActorRegistry"/> after teardown.
|
||||||
|
/// Start/stop live here rather than the test bodies to keep the CancellationToken-less calls out
|
||||||
|
/// of the way (mirrors <c>ClusterRoleInfoTests.ResolveAsync</c>).
|
||||||
|
/// </summary>
|
||||||
|
private static async Task<ActorRegistry> BootAndGetRegistryAsync(
|
||||||
|
string[] roles, Action<AkkaConfigurationBuilder> configure)
|
||||||
|
{
|
||||||
|
var options = new AkkaClusterOptions
|
||||||
|
{
|
||||||
|
Port = 0,
|
||||||
|
Hostname = "127.0.0.1",
|
||||||
|
PublicHostname = "127.0.0.1",
|
||||||
|
SeedNodes = Array.Empty<string>(),
|
||||||
|
Roles = roles,
|
||||||
|
};
|
||||||
|
|
||||||
|
var builder = Host.CreateDefaultBuilder();
|
||||||
|
builder.ConfigureServices(services =>
|
||||||
|
{
|
||||||
|
services.AddSingleton<IOptions<AkkaClusterOptions>>(Options.Create(options));
|
||||||
|
// The admin WithActors block reads IOptions<MeshTransportOptions>.Value at spawn time.
|
||||||
|
services.AddSingleton<IOptions<MeshTransportOptions>>(Options.Create(new MeshTransportOptions()));
|
||||||
|
// Akka.Hosting invokes every singleton's props factory eagerly at StartAsync; the admin
|
||||||
|
// singletons (coordinator/audit/reconciler) read IDbContextFactory<OtOpcUaConfigDbContext>,
|
||||||
|
// so supply an in-memory one or those factories NRE. The driver-only boots don't touch it.
|
||||||
|
services.AddDbContextFactory<OtOpcUaConfigDbContext>(
|
||||||
|
o => o.UseInMemoryDatabase($"rehome-test-{Guid.NewGuid():N}"));
|
||||||
|
// The admin ClusterNodeAddressReconciler props factory resolves IClusterRoleInfo (for its
|
||||||
|
// per-cluster reconcile scope). A null cluster role ⇒ full-fleet reconcile, which is all
|
||||||
|
// this test needs — it never asserts on the reconciler, only on the singleton registry.
|
||||||
|
services.AddSingleton<IClusterRoleInfo>(new FakeClusterRoleInfo(clusterRole: null, clusterId: null));
|
||||||
|
services.AddAkka("otopcua-rehome-test", (ab, sp) =>
|
||||||
|
{
|
||||||
|
ab.WithOtOpcUaClusterBootstrap(sp);
|
||||||
|
configure(ab);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
using var host = builder.Build();
|
||||||
|
await host.StartAsync();
|
||||||
|
try
|
||||||
|
{
|
||||||
|
return host.Services.GetRequiredService<ActorRegistry>();
|
||||||
|
}
|
||||||
|
finally
|
||||||
|
{
|
||||||
|
await host.StopAsync();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>Minimal <see cref="IClusterRoleInfo"/> stub exposing only the cluster-role identity.</summary>
|
||||||
|
private sealed class FakeClusterRoleInfo : IClusterRoleInfo
|
||||||
|
{
|
||||||
|
public FakeClusterRoleInfo(string? clusterRole, string? clusterId)
|
||||||
|
{
|
||||||
|
ClusterRole = clusterRole;
|
||||||
|
ClusterId = clusterId;
|
||||||
|
}
|
||||||
|
|
||||||
|
public string? ClusterRole { get; }
|
||||||
|
public string? ClusterId { get; }
|
||||||
|
public NodeId LocalNode => NodeId.Parse("127.0.0.1:0");
|
||||||
|
public IReadOnlySet<string> LocalRoles => new HashSet<string>();
|
||||||
|
public bool HasRole(string role) => false;
|
||||||
|
public IReadOnlyList<NodeId> MembersWithRole(string role) => Array.Empty<NodeId>();
|
||||||
|
public NodeId? RoleLeader(string role) => null;
|
||||||
|
public event EventHandler<RoleLeaderChangedEventArgs>? RoleLeaderChanged { add { } remove { } }
|
||||||
|
}
|
||||||
|
}
|
||||||
+147
-3
@@ -3,12 +3,16 @@ using Akka.Actor;
|
|||||||
using Akka.Cluster;
|
using Akka.Cluster;
|
||||||
using Akka.Cluster.Tools.Client;
|
using Akka.Cluster.Tools.Client;
|
||||||
using Akka.Configuration;
|
using Akka.Configuration;
|
||||||
|
using Microsoft.EntityFrameworkCore;
|
||||||
using Shouldly;
|
using Shouldly;
|
||||||
using Xunit;
|
using Xunit;
|
||||||
|
using ZB.MOM.WW.OtOpcUa.Cluster;
|
||||||
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Admin;
|
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Admin;
|
||||||
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Deploy;
|
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Deploy;
|
||||||
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
|
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
|
||||||
using ZB.MOM.WW.OtOpcUa.Commons.Types;
|
using ZB.MOM.WW.OtOpcUa.Commons.Types;
|
||||||
|
using ZB.MOM.WW.OtOpcUa.Configuration;
|
||||||
|
using ZB.MOM.WW.OtOpcUa.Configuration.Entities;
|
||||||
using ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
|
using ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
|
||||||
using ZB.MOM.WW.OtOpcUa.Runtime.Communication;
|
using ZB.MOM.WW.OtOpcUa.Runtime.Communication;
|
||||||
using AkkaCluster = Akka.Cluster.Cluster;
|
using AkkaCluster = Akka.Cluster.Cluster;
|
||||||
@@ -86,12 +90,12 @@ public sealed class MeshClusterClientBoundaryTests : IAsyncLifetime
|
|||||||
ClusterClientReceptionist.Get(_central).RegisterService(centralComm);
|
ClusterClientReceptionist.Get(_central).RegisterService(centralComm);
|
||||||
|
|
||||||
// Node: the REAL comm actor, dialling central through the REAL production client factory.
|
// Node: the REAL comm actor, dialling central through the REAL production client factory.
|
||||||
var nodeToCentral = _clientFactory.Create(_node, ReceptionistOf(_central));
|
var nodeToCentral = _clientFactory.Create(_node, "central", ReceptionistOf(_central));
|
||||||
_nodeComm = _node.ActorOf(
|
_nodeComm = _node.ActorOf(
|
||||||
NodeCommunicationActor.Props(nodeToCentral), MeshPaths.NodeCommunicationName);
|
NodeCommunicationActor.Props(nodeToCentral), MeshPaths.NodeCommunicationName);
|
||||||
ClusterClientReceptionist.Get(_node).RegisterService(_nodeComm);
|
ClusterClientReceptionist.Get(_node).RegisterService(_nodeComm);
|
||||||
|
|
||||||
_centralClient = _clientFactory.Create(_central, ReceptionistOf(_node));
|
_centralClient = _clientFactory.Create(_central, "site-a", ReceptionistOf(_node));
|
||||||
}
|
}
|
||||||
|
|
||||||
/// <inheritdoc />
|
/// <inheritdoc />
|
||||||
@@ -179,7 +183,7 @@ public sealed class MeshClusterClientBoundaryTests : IAsyncLifetime
|
|||||||
var comm = deaf.ActorOf(NodeCommunicationActor.Props(null), MeshPaths.NodeCommunicationName);
|
var comm = deaf.ActorOf(NodeCommunicationActor.Props(null), MeshPaths.NodeCommunicationName);
|
||||||
comm.ShouldNotBeNull();
|
comm.ShouldNotBeNull();
|
||||||
|
|
||||||
var client = _clientFactory.Create(_central, ReceptionistOf(deaf));
|
var client = _clientFactory.Create(_central, "deaf", ReceptionistOf(deaf));
|
||||||
var command = new RestartDriver("C1", "driver-1", "alice", Guid.NewGuid());
|
var command = new RestartDriver("C1", "driver-1", "alice", Guid.NewGuid());
|
||||||
|
|
||||||
var deadline = DateTime.UtcNow + Silence;
|
var deadline = DateTime.UtcNow + Silence;
|
||||||
@@ -199,6 +203,146 @@ public sealed class MeshClusterClientBoundaryTests : IAsyncLifetime
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task One_dispatch_through_central_reaches_a_node_in_every_cluster_mesh()
|
||||||
|
{
|
||||||
|
// THE Phase 6 assertion at the transport level, driving the REAL CentralCommunicationActor.
|
||||||
|
// Two genuinely separate site meshes (site-A = _node, site-B created here), each its own
|
||||||
|
// single-member cluster with its own receptionist. Central must build ONE ClusterClient per
|
||||||
|
// cluster and fan SendToAll across them, so a single MeshCommand reaches BOTH meshes. A single
|
||||||
|
// fleet-wide client would reach only the one mesh whose receptionist answered — the exact
|
||||||
|
// failure this test is red-before-green against.
|
||||||
|
var siteB = StartSelfSeededSystem(withReceptionist: true);
|
||||||
|
try
|
||||||
|
{
|
||||||
|
await AwaitSingleMemberUpAsync(siteB);
|
||||||
|
RegisterNodeComm(siteB);
|
||||||
|
|
||||||
|
var inboxA = NewInbox();
|
||||||
|
_node.EventStream.Subscribe(_node.ActorOf(CaptureActor.Props(inboxA)), typeof(RestartDriver));
|
||||||
|
var inboxB = NewInbox();
|
||||||
|
siteB.EventStream.Subscribe(siteB.ActorOf(CaptureActor.Props(inboxB)), typeof(RestartDriver));
|
||||||
|
|
||||||
|
var dbFactory = SeedTwoClusterDb(SiteAddress(_node), SiteAddress(siteB));
|
||||||
|
var actor = _central.ActorOf(CentralCommunicationActor.Props(
|
||||||
|
dbFactory, ClusterClientOptions(), new DefaultMeshClusterClientFactory(), () => null));
|
||||||
|
|
||||||
|
var command = new RestartDriver("C1", "driver-1", "alice", Guid.NewGuid());
|
||||||
|
var deadline = DateTime.UtcNow + Arrival;
|
||||||
|
while (DateTime.UtcNow < deadline && !(inboxA.Task.IsCompleted && inboxB.Task.IsCompleted))
|
||||||
|
{
|
||||||
|
actor.Tell(new MeshCommand("deployments", command));
|
||||||
|
await Task.Delay(250, TestContext.Current.CancellationToken);
|
||||||
|
}
|
||||||
|
|
||||||
|
inboxA.Task.IsCompleted.ShouldBeTrue(
|
||||||
|
"site-A's mesh must receive the dispatch through its own per-cluster ClusterClient");
|
||||||
|
inboxB.Task.IsCompleted.ShouldBeTrue(
|
||||||
|
"site-B's mesh must receive the SAME dispatch through its own per-cluster ClusterClient; "
|
||||||
|
+ "a single fleet-wide client would have reached only one of the two meshes");
|
||||||
|
}
|
||||||
|
finally
|
||||||
|
{
|
||||||
|
await ShutdownAsync(siteB);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task A_client_built_for_one_cluster_does_not_deliver_into_another_clusters_mesh()
|
||||||
|
{
|
||||||
|
// The partition proof for the fan-out: a client dialling ONLY site-A's contacts must land in
|
||||||
|
// site-A's mesh and NEVER cross into site-B's. This is why the per-cluster fan-out is correct
|
||||||
|
// rather than accidentally N x broadcasting — each client is scoped to its own mesh.
|
||||||
|
var siteB = StartSelfSeededSystem(withReceptionist: true);
|
||||||
|
try
|
||||||
|
{
|
||||||
|
await AwaitSingleMemberUpAsync(siteB);
|
||||||
|
RegisterNodeComm(siteB);
|
||||||
|
|
||||||
|
var inboxA = NewInbox();
|
||||||
|
_node.EventStream.Subscribe(_node.ActorOf(CaptureActor.Props(inboxA)), typeof(RestartDriver));
|
||||||
|
var inboxB = NewInbox();
|
||||||
|
siteB.EventStream.Subscribe(siteB.ActorOf(CaptureActor.Props(inboxB)), typeof(RestartDriver));
|
||||||
|
|
||||||
|
var clientA = _clientFactory.Create(_central, "site-a", ReceptionistOf(_node));
|
||||||
|
var command = new RestartDriver("C1", "driver-1", "alice", Guid.NewGuid());
|
||||||
|
|
||||||
|
// Keep sending for the full Silence window so any erroneous cross-delivery into site-B has
|
||||||
|
// every chance to arrive — a pass means the boundary is shut, not that the clock ran out.
|
||||||
|
var deadline = DateTime.UtcNow + Silence;
|
||||||
|
while (DateTime.UtcNow < deadline)
|
||||||
|
{
|
||||||
|
clientA.Tell(new ClusterClient.SendToAll(MeshPaths.NodeCommunication, command));
|
||||||
|
await Task.Delay(250, TestContext.Current.CancellationToken);
|
||||||
|
}
|
||||||
|
|
||||||
|
inboxA.Task.IsCompleted.ShouldBeTrue("site-A's client must deliver into site-A's mesh");
|
||||||
|
inboxB.Task.IsCompleted.ShouldBeFalse(
|
||||||
|
"a client built for site-A's contacts must NOT deliver into site-B's mesh — the "
|
||||||
|
+ "per-cluster fan-out must stay partitioned, one client per mesh");
|
||||||
|
}
|
||||||
|
finally
|
||||||
|
{
|
||||||
|
await ShutdownAsync(siteB);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>Registers a real node-comm actor with a system's receptionist so central can reach it.</summary>
|
||||||
|
/// <param name="system">The site system to register into.</param>
|
||||||
|
private static void RegisterNodeComm(ActorSystem system)
|
||||||
|
{
|
||||||
|
var comm = system.ActorOf(NodeCommunicationActor.Props(null), MeshPaths.NodeCommunicationName);
|
||||||
|
ClusterClientReceptionist.Get(system).RegisterService(comm);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static (string Host, int Port) SiteAddress(ActorSystem system)
|
||||||
|
{
|
||||||
|
var addr = AkkaCluster.Get(system).SelfAddress;
|
||||||
|
return (addr.Host!, addr.Port!.Value);
|
||||||
|
}
|
||||||
|
|
||||||
|
private static MeshTransportOptions ClusterClientOptions() => new()
|
||||||
|
{
|
||||||
|
Mode = MeshTransportOptions.ModeClusterClient,
|
||||||
|
CentralContactPoints = ["akka.tcp://otopcua@127.0.0.1:4053"],
|
||||||
|
ContactRefreshSeconds = 1,
|
||||||
|
};
|
||||||
|
|
||||||
|
private static IDbContextFactory<OtOpcUaConfigDbContext> SeedTwoClusterDb(
|
||||||
|
(string Host, int Port) siteA, (string Host, int Port) siteB)
|
||||||
|
{
|
||||||
|
var factory = new InMemoryConfigDbFactory(Guid.NewGuid().ToString("N"));
|
||||||
|
using var db = factory.CreateDbContext();
|
||||||
|
db.ClusterNodes.Add(NodeRow("SITE-A", "node-a", siteA));
|
||||||
|
db.ClusterNodes.Add(NodeRow("SITE-B", "node-b", siteB));
|
||||||
|
db.SaveChanges();
|
||||||
|
return factory;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static ClusterNode NodeRow(string clusterId, string nodeId, (string Host, int Port) addr) => new()
|
||||||
|
{
|
||||||
|
NodeId = nodeId,
|
||||||
|
ClusterId = clusterId,
|
||||||
|
Host = addr.Host,
|
||||||
|
AkkaPort = addr.Port,
|
||||||
|
ApplicationUri = $"urn:{nodeId}",
|
||||||
|
Enabled = true,
|
||||||
|
MaintenanceMode = false,
|
||||||
|
CreatedBy = "test",
|
||||||
|
};
|
||||||
|
|
||||||
|
private sealed class InMemoryConfigDbFactory(string dbName)
|
||||||
|
: IDbContextFactory<OtOpcUaConfigDbContext>
|
||||||
|
{
|
||||||
|
public OtOpcUaConfigDbContext CreateDbContext()
|
||||||
|
{
|
||||||
|
var opts = new DbContextOptionsBuilder<OtOpcUaConfigDbContext>()
|
||||||
|
.UseInMemoryDatabase(dbName)
|
||||||
|
.Options;
|
||||||
|
return new OtOpcUaConfigDbContext(opts);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// <summary>
|
/// <summary>
|
||||||
/// Starts a single-member cluster on a dynamic port, optionally without the receptionist.
|
/// Starts a single-member cluster on a dynamic port, optionally without the receptionist.
|
||||||
/// </summary>
|
/// </summary>
|
||||||
|
|||||||
@@ -193,6 +193,10 @@ public sealed class ServiceCollectionExtensionsTests
|
|||||||
public NodeId LocalNode { get; } = NodeId.Parse("test-node");
|
public NodeId LocalNode { get; } = NodeId.Parse("test-node");
|
||||||
/// <summary>Gets the local roles.</summary>
|
/// <summary>Gets the local roles.</summary>
|
||||||
public IReadOnlySet<string> LocalRoles { get; } = new HashSet<string>(["driver"]);
|
public IReadOnlySet<string> LocalRoles { get; } = new HashSet<string>(["driver"]);
|
||||||
|
/// <summary>Gets the local node's cluster-scoped role. None configured in this fake.</summary>
|
||||||
|
public string? ClusterRole => null;
|
||||||
|
/// <summary>Gets the ClusterId extracted from <see cref="ClusterRole"/>. None configured in this fake.</summary>
|
||||||
|
public string? ClusterId => null;
|
||||||
/// <summary>Determines whether the local node has the specified role.</summary>
|
/// <summary>Determines whether the local node has the specified role.</summary>
|
||||||
/// <param name="role">The role to check.</param>
|
/// <param name="role">The role to check.</param>
|
||||||
public bool HasRole(string role) => LocalRoles.Contains(role);
|
public bool HasRole(string role) => LocalRoles.Contains(role);
|
||||||
|
|||||||
Reference in New Issue
Block a user