Compare commits

..

15 Commits

Author SHA1 Message Date
Joseph Doherty 2839deb5be docs(mesh): Phase 6 ledger — tasks 0-8 done, task 9 live gate in progress
v2-ci / build (push) Successful in 4m2s
v2-ci / unit-tests (push) Failing after 14m18s
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 03:03:30 -04:00
Joseph Doherty 8a173bbaf6 docs(mesh): Phase 6 — mark done, remove single-mesh caveats, document pair-local meshes
Redundancy.md: KNOWN-LIMITATION (per-Akka-cluster election) marked RESOLVED;
new 'Per-cluster meshes (Phase 6)' section + pair-local secrets note; manual-
failover mesh-scope caveat removed; auto-down 1-vs-1 note updated (every mesh is
now two nodes, drill deferred to Phase 7). IManualFailoverService + ClusterRedundancy
razor caveats updated to pair-local reality. Configuration.md documents the
cluster-{ClusterId} role, per-pair self-first seeds, SplitTopologyTransportValidator,
and the intentional Dps default. Program + design status tables + CLAUDE.md marked
Phase 6 DONE.

(Task 8 subagent completed the edits but died on an API error before committing;
reviewed and committed by the controller.)

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 03:02:35 -04:00
Joseph Doherty 2a057a466f docs(mesh): Phase 6 plan+ledger updates through Task 7
Records the Task 2/3 fallback+scoping corrections, the Task 6 no-flip decision
(data-backed), and task statuses 0-7.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 02:39:08 -04:00
Joseph Doherty 741ab97b89 feat(mesh-phase6): split docker-dev into three independent 2-node Akka meshes
Rewrite docker-dev/docker-compose.yml from one shared six-node gossip ring
into three self-contained pairs (MAIN: central-1/2, SITE-A: site-a-1/2,
SITE-B: site-b-1/2), all still on the ActorSystem name `otopcua` and the one
shared docker network — separation is purely by per-pair self-first
Cluster:SeedNodes, faithful to how production splits over a WAN rather than
a firewall.

Per node:
- Cluster__Roles / OTOPCUA_ROLES gain a `cluster-<ClusterId>` role
  (cluster-MAIN / cluster-SITE-A / cluster-SITE-B), the marker
  SplitTopologyTransportValidator and ClusterRoleInfo key off.
- Cluster__SeedNodes now lists only the node's own pair (self first, partner
  second); site nodes no longer seed off central-1.
- MeshTransport__Mode default flips Dps -> ClusterClient (still overridable
  via OTOPCUA_MESH_MODE) on all six nodes — SplitTopologyTransportValidator
  now refuses to boot a cluster-role node left on Dps.
- Telemetry__Mode=Grpc added on all six driver nodes; TelemetryDial__Mode=Grpc
  added on central-1/central-2 only (the admin pair).
- ConfigSource/ConfigServe modes and MeshTransport contact points are
  unchanged (already split-correct since Phases 3/4/2).

seed-clusters.sql needed no changes — the ClusterNode rows already carry the
correct ClusterId/Host/AkkaPort/GrpcPort per node. Validated with
`docker compose -f docker-dev/docker-compose.yml config`.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 02:38:26 -04:00
Joseph Doherty 344b0d3334 fix(cluster): validate split topology against EFFECTIVE roles, not raw Cluster:Roles
Code review found SplitTopologyTransportValidator read the wrong role view,
defeating its fail-loud purpose two ways:
 (a) Roles-source divergence. AkkaClusterOptions.Roles binds Cluster:Roles
     but falls back to OTOPCUA_ROLES (RoleParser.Parse) when it is empty. A
     node configured with only OTOPCUA_ROLES=driver,cluster-SITE-A genuinely
     carries the cluster role in Akka, yet the validator saw no cluster role
     and passed silently on MeshTransport:Mode=Dps (the documented incident).
 (b) Case. The Cluster:Roles bind path is verbatim while RoleParser.Parse
     lowercases the OTOPCUA_ROLES path; 'Cluster-SITE-A' / 'Driver' / 'Admin'
     slipped past the ordinal checks.

Resolve effective roles the same way the node does — Cluster:Roles, else
RoleParser.Parse(env OTOPCUA_ROLES) exactly as Program.cs — then normalize
both (trim + ToLowerInvariant) before IsClusterRole/driver/admin. The message
still names the ClusterId in the operator's original spelling. Exemption and
all other behaviour unchanged. Contained to the validator.

Tests: OTOPCUA_ROLES-only cluster role on Dps fails; mixed-case roles on Dps
fail; cluster role with neither admin nor driver needs only ClusterClient;
Cluster:Roles wins over a stray OTOPCUA_ROLES. 16/16 green (124/124 project).

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 02:29:41 -04:00
Joseph Doherty a886d5e6e0 feat(cluster): refuse to boot a split-topology node on a DPS transport mode
Per-cluster mesh Phase 6 (Task 5). A node carrying a cluster-{ClusterId}
role (RoleParser.IsClusterRole) is in the split topology, where the
Phase 2/3/5 DPS dark-switch branches deliver nothing across the mesh
boundary. SplitTopologyTransportValidator fails host start unless such a
node uses the mesh-crossing transports: MeshTransport:Mode=ClusterClient
always; Telemetry:Mode=Grpc if it has the driver role; TelemetryDial:Mode
=Grpc if it has the admin role. It is a no-op for any node with no
cluster-role (legacy / single-mesh / test). Roles + the three modes are
cross-read from IConfiguration, mirroring ConfigSourceOptionsValidator.
Registered via AddValidatedOptions on MeshTransportOptions with
ValidateOnStart, alongside the sibling validators.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 02:13:47 -04:00
Joseph Doherty d98b32640b docs(mesh): correct the all-malformed-cluster comments in CentralCommunicationActor
The per-cluster contact map only gains a key when a row parses, so an
all-malformed cluster has no key at all and is caught by the absent-key arm;
the empty-set guard is defensive/unreachable for the DB producer. Comment-only
(Task 4 code-review minor follow-up).

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 02:08:05 -04:00
Joseph Doherty 1ce2042e14 fix(mesh): also scope the reconciler's observed membership to the own cluster
Task 3 review: scoping only the rows/enabled DB queries by ClusterId left
the observed set (from Cluster.State.Members) filtered by the driver role
alone. In the mixed-mesh window — DB scoping live via a cluster role, but
the physical Akka mesh not yet split so central still sees foreign members
via gossip — a foreign-cluster driver member stays in observed while its row
is filtered out, hitting the RunningNodeHasNoRow branch, which logs at ERROR.
That traded the Warning-level false positive for a worse Error-level one.

Filter observed by the node's own cluster ROLE too. Extracted the projection
into a static FilterOwnClusterDriverMembers(members, ownClusterRole) so the
membership filter is unit-testable without seeding a live cluster (the actor
harness joins as admin only, so observed is always empty). Sourced at
registration from IClusterRoleInfo.ClusterRole alongside ClusterId. Null role
(legacy, no cluster role) keeps every driver member — reconcile-all, unchanged.

Also clarified the ownClusterId doc wording (non-null-and-non-empty).

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 02:07:02 -04:00
Joseph Doherty a4d3ab4005 docs(mesh): fix stale redundancy-singleton comments + prove the driver fallback end-to-end
Review follow-ups on the redundancy re-home (Phase 6, Task 2):

- Program.cs: the hasDriver comment claimed IClusterRoleInfo "throws at host
  start if this driver node carries no cluster role" — false since the fallback
  landed. Rewrite it to describe the cluster-{ClusterId} scope with the driver-role
  fallback for legacy single-mesh / not-yet-migrated nodes.
- ClusterRoleInfo.cs: the SubscriberActor comment described RedundancyStateActor as
  the "admin-role singleton" — stale. Note it is now a cluster-{ClusterId}-scoped
  singleton spawned on every driver node (LeaderChanged stays a no-op here).
- Add a host/registry-level test for the driver-role FALLBACK path (roles ["driver"],
  no cluster role) closing the coverage asymmetry — it was proven only in the pure
  helper. Asserts the node boots and registers RedundancyStateActorKey, i.e. the boot
  the earlier throw-based version would have aborted.

The boot helper now supplies an in-memory IDbContextFactory and a fake IClusterRoleInfo:
Akka.Hosting invokes singleton props factories eagerly at StartAsync, and the admin
ClusterNodeAddressReconciler factory reads both (the latter for its per-cluster reconcile
scope, added by a sibling Phase 6 task) — without them the admin boot NREs.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 01:58:48 -04:00
Joseph Doherty 6531ec1984 feat(mesh): one ClusterClient per Cluster in CentralCommunicationActor (Phase 6)
After the fleet splits into one Akka mesh per application Cluster, a
ClusterClientReceptionist serves only its own mesh — so the single
fleet-wide client's SendToAll reached only the mesh whose receptionist
answered, leaving every other cluster silently on its old configuration.

Central now holds one ClusterClient per ClusterId and fans SendToAll
across all of them. LoadContactsFromDb selects ClusterId and groups the
receptionist contacts per cluster (keeping the per-row TryParse guard and
the enabled/non-maintenance filter); ContactsLoaded carries
ContactsByCluster; HandleContactsLoaded diffs per cluster (rebuild changed,
stop+drop vanished, warn-don't-create on empty); RebuildClient builds a
per-cluster client under a clusterId-derived actor name. IMeshClusterClientFactory.Create
gained a clusterId parameter so the per-cluster clients get distinct,
diagnosable names. ApplyAck handling and the DPS branch are unchanged; the
deploy path stays payload-free.

Tests: focused unit coverage of the grouping + fan-out (one-client-per-cluster,
fan-out SendToAll to every cluster client) via the recording factory double,
plus the real two-mesh boundary test extended to central + two separate site
meshes proving one dispatch reaches both and a client scoped to one cluster
never crosses into another. Red-before-green verified: crippling the fan-out
to a single client fails the reaches-both-meshes assertion.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 01:53:20 -04:00
Joseph Doherty 2535df019b fix(mesh): scope ClusterNode address reconcile to the admin node's own cluster
Phase 6 splits the fleet into one mesh per Cluster, so an admin (central)
node's Cluster.State.Members shows only its own pair — it no longer sees
site members via gossip. The ClusterNodeAddressReconciler singleton compared
live membership against EVERY ClusterNode row fleet-wide, so post-split every
foreign-cluster row would log EnabledRowNotInCluster forever.

Scope the actor's two DB queries to rows whose ClusterId matches the admin
node's own (IClusterRoleInfo.ClusterId, sourced at registration). A legacy
admin node with no cluster role (null ClusterId) still reconciles the whole
fleet — it genuinely sees every member via gossip. The pure Reconcile
function and AddressMismatchKind semantics are unchanged; scoping lives
entirely in the actor's queries.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 01:39:06 -04:00
Joseph Doherty a2c5d57fa0 feat(mesh): re-home the redundancy singleton to a per-cluster driver singleton
Phase 6 splits the single fleet-wide Akka mesh into one 2-node mesh per
application Cluster. RedundancyStateActor was an admin-scoped cluster singleton
that elected ONE fleet-wide driver Primary — but after the split a driver-only
site pair has no admin node in its mesh to host it, so its Primary would never
be elected.

Re-home it: remove the redundancy WithSingleton block from the admin
WithOtOpcUaControlPlaneSingletons set, and add WithOtOpcUaClusterRedundancySingleton,
a singleton spawned from Program.cs's hasDriver branch on EVERY driver node (the
fused central included). It scopes to the node's own cluster-{ClusterId} role when
present, and falls back to the fixed `driver` role otherwise. Each mesh then has
exactly one, electing its own pair-local Primary and publishing redundancy-state
on its own mesh's DistributedPubSub. The election logic (oldest Up driver member)
and the DPS publish are unchanged — they become pair-local after the split.

The driver-role fallback (Decision 2) deliberately does NOT throw when a node has
no cluster role: that is the pre-Phase-6 fleet-wide behavior on a legacy single
mesh, so legacy/harness (TwoNodeClusterHarness: admin,driver) and not-yet-migrated
deployments keep booting unchanged. On a genuinely split 2-node mesh the `driver`
role is already pair-local (the mesh IS the pair), so the fallback is correct in
both worlds. Cluster-scope, when present, additionally survives an accidental
two-mesh merge by keeping one singleton per cluster.

The role scope + driver-role fallback are pinned by unit tests against the
extracted BuildClusterRedundancySingletonOptions helper (the BuildDowningHocon
pattern); admin-removal + driver-registration are pinned behaviorally against a
booted node's ActorRegistry.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 01:33:26 -04:00
Joseph Doherty 2d2f1decae feat(mesh): ClusterRoleInfo exposes the node's own cluster-scoped role
Phase 6 Task 1. IClusterRoleInfo gains ClusterRole/ClusterId, derived from
the node's configured AkkaClusterOptions.Roles at construction time (not
live Cluster.State) so the identity is available before the cluster forms.
First cluster-scoped role wins when more than one is configured, logged as
a Warning. Updates the FakeClusterRoleInfo test double in
ServiceCollectionExtensionsTests to satisfy the new interface members.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 01:17:58 -04:00
Joseph Doherty 87ff00fd30 feat(mesh): RoleParser accepts cluster-{ClusterId} roles; consolidate the driver-role literal
Phase 6 (per-cluster mesh split) needs nodes to carry a cluster-scoped
role like cluster-SITE-A; RoleParser previously rejected anything
outside the fixed admin/driver/dev set. Adds IsClusterRole/
ClusterIdFromRole plus well-known-role constants, and points the three
duplicate "driver" string literals (RedundancyStateActor,
ClusterNodeAddressReconcilerActor, ServiceCollectionExtensions) at the
new RoleParser.Driver so the value has one source, without renaming
any existing DriverRole symbol.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 01:10:42 -04:00
Joseph Doherty 02177fec7c docs(mesh): Phase 6 plan — mesh partition + co-location topology
Plan + task ledger for splitting the single fleet mesh into three 2-node
meshes (central + per-site pairs), cluster-scoped roles + redundancy
singleton, per-cluster ClusterClient, own-cluster reconciler, split-topology
transport validator + flipped defaults, pair-local secrets, and the docker-dev
rig rewrite. Decisions settled with the user 2026-07-24.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 01:07:54 -04:00
33 changed files with 2410 additions and 266 deletions
+20 -7
View File
@@ -228,7 +228,18 @@ The server supports non-transparent warm/hot redundancy via the `Redundancy` sec
Assert the **effective** cluster config off a running `ActorSystem` (as `SplitBrainResolverActivationTests` does), never the typed options or the akka.conf text — several HOCON fragments compete and the losing one still reads correctly in the file. Assert the **effective** cluster config off a running `ActorSystem` (as `SplitBrainResolverActivationTests` does), never the typed options or the akka.conf text — several HOCON fragments compete and the losing one still reads correctly in the file.
**Still per-Akka-cluster, not per application `Cluster`.** One Primary is elected across the whole Akka mesh, so a fleet running several application clusters in one mesh (what docker-dev models) gets one Primary among all driver nodes. The fix is the per-cluster mesh workstream: `docs/plans/2026-07-21-per-cluster-mesh-design.md` (Phases 0a/0b/1 done; 2 code-complete behind a dark switch; 37 not started). **RESOLVED by per-cluster mesh Phase 6 (2026-07-24) — no longer per-Akka-cluster.** The "one Primary across the whole Akka mesh" limitation described above is gone; see the Phase 6 paragraph immediately below.
**Per-cluster mesh Phase 6 (2026-07-24) — the fleet is now three independent 2-node meshes, not one.** The single fleet-wide Akka mesh was split into one mesh per application `Cluster`: the central pair (`admin,driver,cluster-MAIN`), site-a (`driver,cluster-SITE-A`), site-b (`driver,cluster-SITE-B`) — same `ActorSystem` name, separated purely by per-pair self-first seeds (each node's `Cluster:SeedNodes` lists itself first, its own pair partner second, never a node from another cluster's pair). Consequences:
- **The redundancy singleton is cluster-scoped and spawned per driver node.** `RedundancyStateActor` is now a `cluster-{ClusterId}`-scoped singleton (falls back to plain `driver` for a legacy node), so each pair elects and publishes its **own** Primary on its own mesh's DPS. **Two or more Primaries exist fleet-wide, one per pair — by design, not a bug.**
- `ClusterNodeAddressReconciler` is **own-cluster-scoped** — an admin node can only reconcile `ClusterNode` rows in its own cluster; it has no gossip visibility into another mesh.
- `CentralCommunicationActor` creates **one `ClusterClient` per application `Cluster`** and fans commands across them (closing the `TODO(Phase 6)` Phase 2 left behind), rather than one fleet-wide client.
- **`SplitTopologyTransportValidator` fails host start** on a node carrying a `cluster-{ClusterId}` role unless it also runs the split-safe transports: `MeshTransport:Mode=ClusterClient`, `Telemetry:Mode=Grpc` (driver), `TelemetryDial:Mode=Grpc` (admin). A DPS transport on a cluster-role node would try to gossip across a partition that no longer exists.
- **The compiled transport-mode defaults were deliberately NOT flipped** — `MeshTransport:Mode`/`Telemetry:Mode`/`TelemetryDial:Mode` all still default to `Dps`; a blast-radius assessment showed flipping the default breaks every integration test and appsettings profile that leaves the key unset. The validator + explicit per-deployment config is the guardrail, not a default flip.
- **Secrets replication is pair-local** — each pair replicates its own Akka DPS secrets within itself; there is no fleet-wide/cross-cluster sync after the split (sites have no SQL, so no SQL-hub, consistent with Phase 4).
See `docs/Redundancy.md` §"Per-cluster meshes (Phase 6)", `docs/Configuration.md` §"Per-cluster mesh split (Phase 6)", and `docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md`.
## Mesh command transport (`MeshTransport`) ## Mesh command transport (`MeshTransport`)
@@ -246,12 +257,14 @@ change, not a redeploy. Things worth knowing before touching it:
`DeploymentArtifact.ResolveClusterScope`). `ClusterClient.Send` delivers to exactly **one** registered `DeploymentArtifact.ResolveClusterScope`). `ClusterClient.Send` delivers to exactly **one** registered
actor — it would deploy to a single node while every other node silently kept its old configuration, actor — it would deploy to a single node while every other node silently kept its old configuration,
and the deployment could still seal green. and the deployment could still seal green.
- **Exactly ONE fleet-wide ClusterClient while the fleet is one mesh.** A receptionist serves its whole - **RESOLVED by Phase 6 — one `ClusterClient` per application `Cluster`, not one fleet-wide.** When
cluster, so `SendToAll` reaches every registered node-comm actor regardless of which node's address was this was written the fleet was still one mesh, so a receptionist served the whole cluster and
dialled. One client per application `Cluster` — the obviously-right-looking shape, and what Phase 6 `SendToAll` reached every registered node-comm actor regardless of which node's address was dialled;
wants — would fan each command out once per cluster (N× duplicate dispatch *and* N× duplicate ack). the `TODO(Phase 6)` in `CentralCommunicationActor.RebuildClient` asked for one client per application
Marked `TODO(Phase 6)` in `CentralCommunicationActor.RebuildClient`. **Corollary:** the contact set does `Cluster` instead. Phase 6 split the fleet into one mesh per `Cluster` and shipped exactly that — see
not scope delivery; excluding a `MaintenanceMode` node from the contacts does not stop it receiving. the [Redundancy](#redundancy) section's Phase 6 paragraph above. The `SendToAll`-within-a-mesh
behaviour below is otherwise unchanged; a `MaintenanceMode` node still isn't excluded from delivery
within its own mesh.
- **Inbound commands re-emit on the node's `EventStream`, not on their DPS topic** — DPS is mesh-wide and - **Inbound commands re-emit on the node's `EventStream`, not on their DPS topic** — DPS is mesh-wide and
central `SendToAll`s to every node, so a DPS re-publish would deliver N copies to every subscriber. central `SendToAll`s to every node, so a DPS re-publish would deliver N copies to every subscriber.
Node-side subscribers (`DriverHostActor`, `ScriptedAlarmHostActor`) accept both. Node-side subscribers (`DriverHostActor`, `ScriptedAlarmHostActor`) accept both.
+166 -96
View File
@@ -1,25 +1,58 @@
# docker-dev/ — Mac-friendly single-mesh hub-and-spoke fleet for v2 development + manual UI exercise. # docker-dev/ — Mac-friendly THREE-mesh hub-and-spoke fleet for v2 development + manual UI exercise.
# #
# Topology: ONE Akka mesh seeded by `central-1`. Logical separation between # Topology (per-cluster mesh Phase 6, 2026-07-24): THREE independent 2-node Akka
# tenants is by ServerCluster.ClusterId rows (MAIN / SITE-A / SITE-B) in the one # meshes, not one six-node mesh. Every node still runs the same ActorSystem name
# shared `OtOpcUa` ConfigDb — NOT by separate meshes. All six host nodes join the # `otopcua` and all six containers still sit on the ONE shared docker-dev network
# same gossip ring and the central UI deploys to every cluster over it. # — separation is purely by per-pair self-first Cluster__SeedNodes partitioning,
# never by network isolation. This is deliberately faithful to production, where
# the three meshes are separated by a WAN, not a firewall: nothing here stops
# central-1 opening a bare TCP connection to site-a-1's Akka port, it just never
# tries, because no seed list and no gossip ever names it.
#
# MAIN central-1 + central-2 (fused admin+driver, roles admin,driver,cluster-MAIN)
# SITE-A site-a-1 + site-a-2 (driver-only, roles driver,cluster-SITE-A)
# SITE-B site-b-1 + site-b-2 (driver-only, roles driver,cluster-SITE-B)
#
# Each pair seeds ONLY itself (self-first, partner-second) — no node lists a node
# outside its own mesh as a seed. Logical/tenant separation (ServerCluster.ClusterId
# rows MAIN / SITE-A / SITE-B in the one shared `OtOpcUa` ConfigDb) now COINCIDES
# with mesh separation: a `cluster-<ClusterId>` Akka role marks which mesh + which
# ServerCluster row a node belongs to (RoleParser.ClusterRolePrefix = "cluster-").
#
# Since central no longer shares a gossip ring with the site meshes, it reaches
# them over the three explicit split-topology transports instead of gossip:
# - Command plane: MeshTransport__Mode=ClusterClient (central dials each site
# mesh's ClusterClientReceptionist; DPS is gossip-only and
# would not cross the mesh boundary at all now)
# - Live telemetry: Telemetry__Mode=Grpc on every driver node (all six) +
# TelemetryDial__Mode=Grpc on the admin pair (central-1/2)
# — the site node hosts the gRPC server, central dials in
# - Deployed config: ConfigSource__Mode=FetchAndCache on the site nodes (already
# true since Phase 3/4 — driver-only nodes have no ConfigDb
# connection at all); central stays ConfigSource-less/Direct
# and keeps serving via ConfigServe.
# SplitTopologyTransportValidator (Cluster project) REFUSES to start any node
# carrying a `cluster-*` role that is left on a DPS-shaped transport — these
# three flips are therefore mandatory here, not optional dark switches.
# #
# Stack: # Stack:
# sql SQL Server 2022 — hosts the one ConfigDb every node uses # sql SQL Server 2022 — hosts the one ConfigDb every node uses
# cluster-seed one-shot mssql-tools job that INSERTs the ServerCluster + # cluster-seed one-shot mssql-tools job that INSERTs the ServerCluster +
# ClusterNode rows scoping each tenant, then exits (idempotent) # ClusterNode rows scoping each tenant, then exits (idempotent)
# #
# central-1, central-2 OTOPCUA_ROLES=admin,driver — the ONLY UI + deploy # central-1, central-2 OTOPCUA_ROLES=admin,driver,cluster-MAIN — the ONLY UI +
# singleton, plus the MAIN cluster's OPC UA publishers. # deploy singleton, plus the MAIN cluster's OPC UA
# Reachable at http://localhost:9200 (via Traefik). # publishers. Reachable at http://localhost:9200 (via
# Both are Akka seed nodes, each listing ITSELF first # Traefik). Both are Akka seed nodes for the MAIN mesh
# (2026-07-22) so either can cold-start while the other # ONLY, each listing ITSELF first (2026-07-22) so either
# is down; the site nodes seed off central-1 only. # can cold-start while the other is down.
# site-a-1, site-a-2 OTOPCUA_ROLES=driver — driver-only members of the same # site-a-1, site-a-2 OTOPCUA_ROLES=driver,cluster-SITE-A — driver-only
# site-b-1, site-b-2 mesh, scoped to SITE-A / SITE-B by ClusterId. They # site-b-1, site-b-2 OTOPCUA_ROLES=driver,cluster-SITE-B — members of their
# serve no UI and authenticate no users; the central # OWN 2-node mesh (self-first seeded within the pair, NOT
# cluster manages and deploys to them over the mesh. # off central). They serve no UI and authenticate no
# users; central manages + deploys to them over the
# ClusterClient/gRPC/FetchAndCache transports above, not
# gossip.
# #
# Auth is real LDAP against the shared GLAuth on the Linux Docker host # Auth is real LDAP against the shared GLAuth on the Linux Docker host
# (10.100.0.35:3893, dc=zb,dc=local) — there is no LDAP container here. # (10.100.0.35:3893, dc=zb,dc=local) — there is no LDAP container here.
@@ -45,17 +78,12 @@
# listener, verified free against every other per-node port: Akka 4053, ConfigServe 4055 on # listener, verified free against every other per-node port: Akka 4053, ConfigServe 4055 on
# central, LocalDb sync 9001 on site-a, OPC UA 4840, HTTP 9000) + Telemetry__ApiKey # central, LocalDb sync 9001 on site-a, OPC UA 4840, HTTP 9000) + Telemetry__ApiKey
# (committed dev key, matches the SQL password / secrets KEK / ConfigServe key exception). # (committed dev key, matches the SQL password / secrets KEK / ConfigServe key exception).
# central-1/central-2 also carry the matching TelemetryDial__ApiKey. Both Telemetry__Mode and # central-1/central-2 also carry the matching TelemetryDial__ApiKey. As of Phase 6, Telemetry__Mode
# TelemetryDial__Mode are left UNSET here, so every node defaults to "Dps" (dark — telemetry # is set to "Grpc" on all six nodes and TelemetryDial__Mode is set to "Grpc" on central-1/central-2
# keeps riding the existing DistributedPubSub topic). The ClusterNode.GrpcPort column (seeded # ONLY — SplitTopologyTransportValidator requires it once a node carries a cluster-* role, so this
# by docker-dev/seed/seed-clusters.sql) points central's dial supervisor at each node's :4056. # is no longer an optional live-gate step (see the Phase 5 comment history in git blame for the
# Unlike MeshTransport/ConfigSource, no `${VAR:-Dps}` interpolation is wired for either Mode key # prior dark-switch shape). The ClusterNode.GrpcPort column (seeded by
# (a bare shell export does nothing here — these keys aren't referenced anywhere in this file, so # docker-dev/seed/seed-clusters.sql) points central's dial supervisor at each node's :4056.
# there is nothing for it to substitute into). To run the Phase 5 live gate, manually add
# `Telemetry__Mode: "Grpc"` to the `environment:` block of all six driver nodes (central-1,
# central-2, site-a-1, site-a-2, site-b-1, site-b-2) and `TelemetryDial__Mode: "Grpc"` to
# central-1's and central-2's blocks — or supply both via a docker-compose.override.yml carrying
# the same keys — then `docker compose -f docker-dev/docker-compose.yml up -d` to recreate.
# #
# Usage: # Usage:
# docker compose -f docker-dev/docker-compose.yml up -d --build # docker compose -f docker-dev/docker-compose.yml up -d --build
@@ -84,6 +112,15 @@ name: otopcua-dev
# AnnounceInterval is shortened to 5s (default 30s) so anti-entropy convergence is # AnnounceInterval is shortened to 5s (default 30s) so anti-entropy convergence is
# observable quickly during dev exercise. Each node keeps its own local SQLite store # observable quickly during dev exercise. Each node keeps its own local SQLite store
# (Secrets:SqlitePath=otopcua-secrets.db, per-container); replication syncs them. # (Secrets:SqlitePath=otopcua-secrets.db, per-container); replication syncs them.
#
# Per-cluster mesh Phase 6: this DPS topic now rides each pair's OWN 2-node mesh, not one
# shared six-node mesh — so secret replication is PAIR-LOCAL. central-1/central-2 replicate
# with each other; site-a-1/site-a-2 replicate with each other; site-b-1/site-b-2 replicate
# with each other. There is no cross-cluster secret sync after the split (by design — the site
# meshes don't share a gossip ring with central or with each other, consistent with Phase 4's
# "driver-only nodes have no SQL, hence no SQL-backed cross-mesh hub" posture). No code change
# was needed for this — Enabled=true + the same KEK on every node is still correct; it just now
# converges within three independent 2-node groups instead of one six-node group.
x-secrets-env: &secrets-env x-secrets-env: &secrets-env
Secrets__Replication__Enabled: "true" Secrets__Replication__Enabled: "true"
Secrets__Replication__AnnounceInterval: "00:00:05" Secrets__Replication__AnnounceInterval: "00:00:05"
@@ -155,8 +192,9 @@ services:
# ── Central cluster (2-node fused admin+driver) ───────────────────────────── # ── Central cluster (2-node fused admin+driver) ─────────────────────────────
# The only UI + deploy singleton; also the MAIN cluster's OPC UA publishers. # The only UI + deploy singleton; also the MAIN cluster's OPC UA publishers.
# central-1 seeds the single Akka mesh that every other node joins; central-2 is a seed too # Per-cluster mesh Phase 6: central-1 and central-2 form their OWN 2-node "MAIN" Akka mesh,
# (listing itself first), so it can form the mesh alone if central-1 is down. # NOT a shared mesh with the site nodes — each lists itself first as seed, so either can
# cold-start while the other is down, and the site meshes are never in this pair's seed list.
central-1: &otopcua-host central-1: &otopcua-host
build: build:
@@ -189,7 +227,7 @@ services:
migrator: { condition: service_completed_successfully } migrator: { condition: service_completed_successfully }
environment: environment:
<<: *secrets-env <<: *secrets-env
OTOPCUA_ROLES: "admin,driver" OTOPCUA_ROLES: "admin,driver,cluster-MAIN"
ASPNETCORE_URLS: "http://+:9000" ASPNETCORE_URLS: "http://+:9000"
ConnectionStrings__ConfigDb: "Server=sql,1433;Database=OtOpcUa;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;" ConnectionStrings__ConfigDb: "Server=sql,1433;Database=OtOpcUa;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;"
Cluster__Hostname: "0.0.0.0" Cluster__Hostname: "0.0.0.0"
@@ -203,18 +241,22 @@ services:
ConfigServe__GrpcListenPort: "4055" ConfigServe__GrpcListenPort: "4055"
ConfigServe__ApiKey: "configserve-docker-dev-key" ConfigServe__ApiKey: "configserve-docker-dev-key"
ConfigSource__Mode: "Direct" ConfigSource__Mode: "Direct"
# Per-cluster mesh Phase 5 (gRPC live-telemetry stream). DARK SWITCH — Telemetry:Mode is left # Per-cluster mesh Phase 5/6 (gRPC live-telemetry stream). central-1 carries a cluster-*
# unset here (defaults to "Dps"), so the node keeps publishing telemetry on the existing # role + admin, so SplitTopologyTransportValidator requires BOTH Telemetry:Mode=Grpc
# DistributedPubSub topic; the dedicated gRPC listener below binds regardless (the node "always # (it is also a driver node, publishing MAIN's own telemetry) AND TelemetryDial:Mode=Grpc
# hosts" per docs/Telemetry.md) so flipping the mode later is a config change, not a redeploy. # (it is the admin-role dialer for every site node). The dedicated gRPC listener below binds
# Port 4056 checked free against every other port this node binds: Akka 4053, ConfigServe 4055, # regardless of mode (the node "always hosts" per docs/Telemetry.md). Port 4056 checked free
# OPC UA 4840 (container-internal), HTTP 9000. TelemetryDial__ApiKey (below) is central's # against every other port this node binds: Akka 4053, ConfigServe 4055, OPC UA 4840
# dial-side key and must equal every node's Telemetry:ApiKey (fail-closed interceptor). # (container-internal), HTTP 9000. TelemetryDial__ApiKey (below) is central's dial-side key
# and must equal every node's Telemetry:ApiKey (fail-closed interceptor).
Telemetry__GrpcListenPort: "4056" Telemetry__GrpcListenPort: "4056"
Telemetry__Mode: "Grpc"
Telemetry__ApiKey: "telemetry-docker-dev-key" Telemetry__ApiKey: "telemetry-docker-dev-key"
TelemetryDial__Mode: "Grpc"
TelemetryDial__ApiKey: "telemetry-docker-dev-key" TelemetryDial__ApiKey: "telemetry-docker-dev-key"
# Both redundancy peers are seeds (#459 / ScadaBridge parity) so a restarted node can # Both redundancy peers are seeds (#459 / ScadaBridge parity) so a restarted node can
# re-join the mesh via EITHER peer, not only central-1. # re-join the MAIN mesh via EITHER peer — NOT via a site node; the site meshes are never in
# this pair's seed list (per-cluster mesh Phase 6).
# #
# ORDER IS LOAD-BEARING (2026-07-22): each node lists ITSELF first. Akka runs # ORDER IS LOAD-BEARING (2026-07-22): each node lists ITSELF first. Akka runs
# FirstSeedNodeProcess — the only path that can form a NEW cluster when no peer answers # FirstSeedNodeProcess — the only path that can form a NEW cluster when no peer answers
@@ -223,19 +265,25 @@ services:
# boot by AkkaClusterOptionsValidator; see docs/Redundancy.md. # boot by AkkaClusterOptionsValidator; see docs/Redundancy.md.
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053" Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
Cluster__SeedNodes__1: "akka.tcp://otopcua@central-2:4053" Cluster__SeedNodes__1: "akka.tcp://otopcua@central-2:4053"
# Per-cluster mesh Phase 6: cluster-MAIN marks this node as a member of the MAIN mesh
# (RoleParser.ClusterRolePrefix = "cluster-"), read by SplitTopologyTransportValidator to
# require ClusterClient/Grpc transports, and by ClusterRoleInfo to scope redundancy +
# ClusterNodeAddressReconcilerActor to ClusterId=MAIN.
Cluster__Roles__0: "admin" Cluster__Roles__0: "admin"
Cluster__Roles__1: "driver" Cluster__Roles__1: "driver"
# Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps", Cluster__Roles__2: "cluster-MAIN"
# the transport this repo has always used, and every node keeps its ClusterClient wiring built # Mesh command transport (per-cluster mesh Phase 2, made mandatory by Phase 6). Central no
# and its comm actor registered with the receptionist either way. Flip the whole rig with # longer shares a gossip ring with the site meshes, so DPS (gossip-only) cannot reach them —
# OTOPCUA_MESH_MODE=ClusterClient at `docker compose up` — no compose edit, and no rebuild. # ClusterClient is the only transport that can. SplitTopologyTransportValidator refuses to
# start any node carrying a cluster-* role that is left on Dps, so this is fixed, not an
# env-var-overridable default, on every node in this file.
# #
# Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction # Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
# time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an # time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
# actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in # actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
# silence. Both central nodes are listed so a driver node's client can rotate to whichever one # silence. Both central nodes are listed so a driver node's client can rotate to whichever one
# is up — that rotation is why the comm actors are per-node and NOT cluster singletons. # is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Dps}" MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-ClusterClient}"
MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053" MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053" MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
Security__Jwt__SigningKey: "docker-dev-signing-key-with-at-least-32-bytes-of-utf8-content-12345" Security__Jwt__SigningKey: "docker-dev-signing-key-with-at-least-32-bytes-of-utf8-content-12345"
@@ -298,7 +346,7 @@ services:
migrator: { condition: service_completed_successfully } migrator: { condition: service_completed_successfully }
environment: environment:
<<: *secrets-env <<: *secrets-env
OTOPCUA_ROLES: "admin,driver" OTOPCUA_ROLES: "admin,driver,cluster-MAIN"
ASPNETCORE_URLS: "http://+:9000" ASPNETCORE_URLS: "http://+:9000"
ConnectionStrings__ConfigDb: "Server=sql,1433;Database=OtOpcUa;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;" ConnectionStrings__ConfigDb: "Server=sql,1433;Database=OtOpcUa;User Id=sa;Password=OtOpcUa!Dev123;TrustServerCertificate=True;"
Cluster__Hostname: "0.0.0.0" Cluster__Hostname: "0.0.0.0"
@@ -309,29 +357,33 @@ services:
ConfigServe__GrpcListenPort: "4055" ConfigServe__GrpcListenPort: "4055"
ConfigServe__ApiKey: "configserve-docker-dev-key" ConfigServe__ApiKey: "configserve-docker-dev-key"
ConfigSource__Mode: "Direct" ConfigSource__Mode: "Direct"
# Per-cluster mesh Phase 5 (gRPC live-telemetry stream) — see central-1. DARK SWITCH: Mode # Per-cluster mesh Phase 5/6 (gRPC live-telemetry stream) — see central-1. central-2 carries
# stays unset (⇒ Dps); the dedicated :4056 listener binds regardless. # a cluster-* role + admin, so both Telemetry:Mode=Grpc and TelemetryDial:Mode=Grpc are
# required (SplitTopologyTransportValidator); the dedicated :4056 listener binds regardless.
Telemetry__GrpcListenPort: "4056" Telemetry__GrpcListenPort: "4056"
Telemetry__Mode: "Grpc"
Telemetry__ApiKey: "telemetry-docker-dev-key" Telemetry__ApiKey: "telemetry-docker-dev-key"
TelemetryDial__Mode: "Grpc"
TelemetryDial__ApiKey: "telemetry-docker-dev-key" TelemetryDial__ApiKey: "telemetry-docker-dev-key"
# Both redundancy peers are seeds (#459 / ScadaBridge parity) — see central-1. SELF FIRST: # Both redundancy peers are seeds (#459 / ScadaBridge parity) — see central-1. SELF FIRST:
# central-2 lists itself as seed-nodes[0], which is what lets it cold-start while central-1 # central-2 lists itself as seed-nodes[0], which is what lets it cold-start while central-1
# is down (it previously listed central-1 first and simply never came Up in that case). # is down (it previously listed central-1 first and simply never came Up in that case). Per
# per-cluster mesh Phase 6, central-2's seed list contains ONLY the MAIN pair — no site node.
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-2:4053" Cluster__SeedNodes__0: "akka.tcp://otopcua@central-2:4053"
Cluster__SeedNodes__1: "akka.tcp://otopcua@central-1:4053" Cluster__SeedNodes__1: "akka.tcp://otopcua@central-1:4053"
# Per-cluster mesh Phase 6: cluster-MAIN marks this node as a member of the MAIN mesh — see
# central-1's Cluster__Roles comment.
Cluster__Roles__0: "admin" Cluster__Roles__0: "admin"
Cluster__Roles__1: "driver" Cluster__Roles__1: "driver"
# Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps", Cluster__Roles__2: "cluster-MAIN"
# the transport this repo has always used, and every node keeps its ClusterClient wiring built # Mesh command transport (per-cluster mesh Phase 2, made mandatory by Phase 6) — see central-1.
# and its comm actor registered with the receptionist either way. Flip the whole rig with
# OTOPCUA_MESH_MODE=ClusterClient at `docker compose up` — no compose edit, and no rebuild.
# #
# Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction # Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
# time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an # time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
# actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in # actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
# silence. Both central nodes are listed so a driver node's client can rotate to whichever one # silence. Both central nodes are listed so a driver node's client can rotate to whichever one
# is up — that rotation is why the comm actors are per-node and NOT cluster singletons. # is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Dps}" MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-ClusterClient}"
MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053" MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053" MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
Security__Jwt__SigningKey: "docker-dev-signing-key-with-at-least-32-bytes-of-utf8-content-12345" Security__Jwt__SigningKey: "docker-dev-signing-key-with-at-least-32-bytes-of-utf8-content-12345"
@@ -379,9 +431,10 @@ services:
- otopcua-localdb-central-2:/app/data - otopcua-localdb-central-2:/app/data
# ── Site A cluster (2-node driver-only) ───────────────────────────────────── # ── Site A cluster (2-node driver-only) ─────────────────────────────────────
# Driver-only members of the single mesh, scoped to SITE-A by ClusterId. No UI, # Per-cluster mesh Phase 6: site-a-1 and site-a-2 form their OWN 2-node "SITE-A" Akka mesh,
# no user auth — managed + deployed to by the central cluster over the mesh. # separate from MAIN and SITE-B — self-first seeded WITHIN the pair, never off central. No UI,
# All site nodes seed central-1. # no user auth; central manages + deploys to this mesh over ClusterClient/gRPC/FetchAndCache,
# not gossip (central and site-a share no seed list, so they never form one ring).
site-a-1: site-a-1:
<<: *otopcua-host <<: *otopcua-host
@@ -390,7 +443,7 @@ services:
central-1: { condition: service_started } central-1: { condition: service_started }
migrator: { condition: service_completed_successfully } migrator: { condition: service_completed_successfully }
environment: environment:
OTOPCUA_ROLES: "driver" OTOPCUA_ROLES: "driver,cluster-SITE-A"
# Per-cluster mesh Phase 4: driver-only site nodes hold NO ConfigDb connection string. Their # Per-cluster mesh Phase 4: driver-only site nodes hold NO ConfigDb connection string. Their
# config comes from central via ConfigSource:Mode=FetchAndCache (gRPC fetch → LocalDb cache), their # config comes from central via ConfigSource:Mode=FetchAndCache (gRPC fetch → LocalDb cache), their
# scripted-alarm condition state lives in LocalDb, and central persists their deploy acks. Direct # scripted-alarm condition state lives in LocalDb, and central persists their deploy acks. Direct
@@ -406,29 +459,32 @@ services:
ConfigSource__CentralFetchEndpoints__0: "http://central-1:4055" ConfigSource__CentralFetchEndpoints__0: "http://central-1:4055"
ConfigSource__CentralFetchEndpoints__1: "http://central-2:4055" ConfigSource__CentralFetchEndpoints__1: "http://central-2:4055"
ConfigSource__ApiKey: "configserve-docker-dev-key" ConfigSource__ApiKey: "configserve-docker-dev-key"
# Per-cluster mesh Phase 5 (gRPC live-telemetry stream) — see central-1/central-2. DARK # Per-cluster mesh Phase 5/6 (gRPC live-telemetry stream) — see central-1/central-2. This
# SWITCH: Telemetry:Mode stays unset (⇒ Dps); the dedicated :4056 listener binds regardless # node carries cluster-SITE-A + driver, so SplitTopologyTransportValidator requires
# (the node "always hosts" per docs/Telemetry.md). Port checked free against this node's # Telemetry:Mode=Grpc (mandatory, not a dark switch, as of Phase 6). The dedicated :4056
# other ports: Akka 4053, OPC UA 4840, HTTP 9000, LocalDb sync 9001 (below). ApiKey must # listener binds regardless (the node "always hosts" per docs/Telemetry.md). Port checked
# equal central's TelemetryDial:ApiKey (fail-closed interceptor). # free against this node's other ports: Akka 4053, OPC UA 4840, HTTP 9000, LocalDb sync 9001
# (below). ApiKey must equal central's TelemetryDial:ApiKey (fail-closed interceptor).
Telemetry__GrpcListenPort: "4056" Telemetry__GrpcListenPort: "4056"
Telemetry__Mode: "Grpc"
Telemetry__ApiKey: "telemetry-docker-dev-key" Telemetry__ApiKey: "telemetry-docker-dev-key"
# Site nodes are deliberately NOT seeds — they join the central pair's mesh. The self-first # Per-cluster mesh Phase 6: site-a-1 seeds its OWN "SITE-A" mesh (itself first, site-a-2
# seed rule is conditional for exactly this reason (it binds only when a node's own address # second) — it no longer seeds off central-1. Central is reached over ClusterClient/gRPC/
# is in its own seed list), so these configs are exempt rather than broken. # FetchAndCache, never gossip; central and site-a share no seed list.
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053" Cluster__SeedNodes__0: "akka.tcp://otopcua@site-a-1:4053"
Cluster__SeedNodes__1: "akka.tcp://otopcua@site-a-2:4053"
Cluster__Roles__0: "driver" Cluster__Roles__0: "driver"
# Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps", Cluster__Roles__1: "cluster-SITE-A"
# the transport this repo has always used, and every node keeps its ClusterClient wiring built # Mesh command transport (per-cluster mesh Phase 2, made mandatory by Phase 6) — see
# and its comm actor registered with the receptionist either way. Flip the whole rig with # central-1/central-2. Central dials this mesh's ClusterClientReceptionist via the contact
# OTOPCUA_MESH_MODE=ClusterClient at `docker compose up` — no compose edit, and no rebuild. # points below; there is no gossip path anymore.
# #
# Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction # Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
# time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an # time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
# actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in # actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
# silence. Both central nodes are listed so a driver node's client can rotate to whichever one # silence. Both central nodes are listed so a driver node's client can rotate to whichever one
# is up — that rotation is why the comm actors are per-node and NOT cluster singletons. # is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Dps}" MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-ClusterClient}"
MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053" MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053" MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
<<: *secrets-env <<: *secrets-env
@@ -490,7 +546,7 @@ services:
central-1: { condition: service_started } central-1: { condition: service_started }
migrator: { condition: service_completed_successfully } migrator: { condition: service_completed_successfully }
environment: environment:
OTOPCUA_ROLES: "driver" OTOPCUA_ROLES: "driver,cluster-SITE-A"
# Per-cluster mesh Phase 4: no ConfigDb string on driver-only site nodes (see site-a-1). # Per-cluster mesh Phase 4: no ConfigDb string on driver-only site nodes (see site-a-1).
Cluster__Hostname: "0.0.0.0" Cluster__Hostname: "0.0.0.0"
Cluster__Port: "4053" Cluster__Port: "4053"
@@ -501,23 +557,27 @@ services:
ConfigSource__CentralFetchEndpoints__0: "http://central-1:4055" ConfigSource__CentralFetchEndpoints__0: "http://central-1:4055"
ConfigSource__CentralFetchEndpoints__1: "http://central-2:4055" ConfigSource__CentralFetchEndpoints__1: "http://central-2:4055"
ConfigSource__ApiKey: "configserve-docker-dev-key" ConfigSource__ApiKey: "configserve-docker-dev-key"
# Per-cluster mesh Phase 5 (gRPC live-telemetry stream) — see central-1/site-a-1. DARK # Per-cluster mesh Phase 5/6 (gRPC live-telemetry stream) — see central-1/site-a-1.
# SWITCH: Telemetry:Mode stays unset (⇒ Dps); the dedicated :4056 listener binds regardless. # Telemetry:Mode=Grpc is mandatory as of Phase 6 (cluster-SITE-A + driver); the dedicated
# :4056 listener binds regardless.
Telemetry__GrpcListenPort: "4056" Telemetry__GrpcListenPort: "4056"
Telemetry__Mode: "Grpc"
Telemetry__ApiKey: "telemetry-docker-dev-key" Telemetry__ApiKey: "telemetry-docker-dev-key"
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053" # Per-cluster mesh Phase 6: site-a-2 seeds its OWN "SITE-A" mesh (itself first, site-a-1
# second) — it no longer seeds off central-1.
Cluster__SeedNodes__0: "akka.tcp://otopcua@site-a-2:4053"
Cluster__SeedNodes__1: "akka.tcp://otopcua@site-a-1:4053"
Cluster__Roles__0: "driver" Cluster__Roles__0: "driver"
# Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps", Cluster__Roles__1: "cluster-SITE-A"
# the transport this repo has always used, and every node keeps its ClusterClient wiring built # Mesh command transport (per-cluster mesh Phase 2, made mandatory by Phase 6) — see
# and its comm actor registered with the receptionist either way. Flip the whole rig with # central-1/site-a-1.
# OTOPCUA_MESH_MODE=ClusterClient at `docker compose up` — no compose edit, and no rebuild.
# #
# Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction # Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
# time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an # time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
# actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in # actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
# silence. Both central nodes are listed so a driver node's client can rotate to whichever one # silence. Both central nodes are listed so a driver node's client can rotate to whichever one
# is up — that rotation is why the comm actors are per-node and NOT cluster singletons. # is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Dps}" MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-ClusterClient}"
MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053" MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053" MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
<<: *secrets-env <<: *secrets-env
@@ -563,6 +623,8 @@ services:
- otopcua-localdb-site-a-2:/app/data - otopcua-localdb-site-a-2:/app/data
# ── Site B cluster (2-node driver-only) ───────────────────────────────────── # ── Site B cluster (2-node driver-only) ─────────────────────────────────────
# Per-cluster mesh Phase 6: site-b-1 and site-b-2 form their OWN 2-node "SITE-B" Akka mesh,
# separate from MAIN and SITE-A — same shape as site-a-1/site-a-2, see that pair's header.
site-b-1: site-b-1:
<<: *otopcua-host <<: *otopcua-host
@@ -571,7 +633,7 @@ services:
central-1: { condition: service_started } central-1: { condition: service_started }
migrator: { condition: service_completed_successfully } migrator: { condition: service_completed_successfully }
environment: environment:
OTOPCUA_ROLES: "driver" OTOPCUA_ROLES: "driver,cluster-SITE-B"
# Per-cluster mesh Phase 4: no ConfigDb string on driver-only site nodes (see site-a-1). # Per-cluster mesh Phase 4: no ConfigDb string on driver-only site nodes (see site-a-1).
Cluster__Hostname: "0.0.0.0" Cluster__Hostname: "0.0.0.0"
Cluster__Port: "4053" Cluster__Port: "4053"
@@ -582,23 +644,27 @@ services:
ConfigSource__CentralFetchEndpoints__0: "http://central-1:4055" ConfigSource__CentralFetchEndpoints__0: "http://central-1:4055"
ConfigSource__CentralFetchEndpoints__1: "http://central-2:4055" ConfigSource__CentralFetchEndpoints__1: "http://central-2:4055"
ConfigSource__ApiKey: "configserve-docker-dev-key" ConfigSource__ApiKey: "configserve-docker-dev-key"
# Per-cluster mesh Phase 5 (gRPC live-telemetry stream) — see central-1/site-a-1. DARK # Per-cluster mesh Phase 5/6 (gRPC live-telemetry stream) — see central-1/site-a-1.
# SWITCH: Telemetry:Mode stays unset (⇒ Dps); the dedicated :4056 listener binds regardless. # Telemetry:Mode=Grpc is mandatory as of Phase 6 (cluster-SITE-B + driver); the dedicated
# :4056 listener binds regardless.
Telemetry__GrpcListenPort: "4056" Telemetry__GrpcListenPort: "4056"
Telemetry__Mode: "Grpc"
Telemetry__ApiKey: "telemetry-docker-dev-key" Telemetry__ApiKey: "telemetry-docker-dev-key"
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053" # Per-cluster mesh Phase 6: site-b-1 seeds its OWN "SITE-B" mesh (itself first, site-b-2
# second) — it no longer seeds off central-1.
Cluster__SeedNodes__0: "akka.tcp://otopcua@site-b-1:4053"
Cluster__SeedNodes__1: "akka.tcp://otopcua@site-b-2:4053"
Cluster__Roles__0: "driver" Cluster__Roles__0: "driver"
# Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps", Cluster__Roles__1: "cluster-SITE-B"
# the transport this repo has always used, and every node keeps its ClusterClient wiring built # Mesh command transport (per-cluster mesh Phase 2, made mandatory by Phase 6) — see
# and its comm actor registered with the receptionist either way. Flip the whole rig with # central-1/site-a-1.
# OTOPCUA_MESH_MODE=ClusterClient at `docker compose up` — no compose edit, and no rebuild.
# #
# Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction # Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
# time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an # time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
# actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in # actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
# silence. Both central nodes are listed so a driver node's client can rotate to whichever one # silence. Both central nodes are listed so a driver node's client can rotate to whichever one
# is up — that rotation is why the comm actors are per-node and NOT cluster singletons. # is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Dps}" MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-ClusterClient}"
MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053" MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053" MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
<<: *secrets-env <<: *secrets-env
@@ -635,7 +701,7 @@ services:
central-1: { condition: service_started } central-1: { condition: service_started }
migrator: { condition: service_completed_successfully } migrator: { condition: service_completed_successfully }
environment: environment:
OTOPCUA_ROLES: "driver" OTOPCUA_ROLES: "driver,cluster-SITE-B"
# Per-cluster mesh Phase 4: no ConfigDb string on driver-only site nodes (see site-a-1). # Per-cluster mesh Phase 4: no ConfigDb string on driver-only site nodes (see site-a-1).
Cluster__Hostname: "0.0.0.0" Cluster__Hostname: "0.0.0.0"
Cluster__Port: "4053" Cluster__Port: "4053"
@@ -646,23 +712,27 @@ services:
ConfigSource__CentralFetchEndpoints__0: "http://central-1:4055" ConfigSource__CentralFetchEndpoints__0: "http://central-1:4055"
ConfigSource__CentralFetchEndpoints__1: "http://central-2:4055" ConfigSource__CentralFetchEndpoints__1: "http://central-2:4055"
ConfigSource__ApiKey: "configserve-docker-dev-key" ConfigSource__ApiKey: "configserve-docker-dev-key"
# Per-cluster mesh Phase 5 (gRPC live-telemetry stream) — see central-1/site-a-1. DARK # Per-cluster mesh Phase 5/6 (gRPC live-telemetry stream) — see central-1/site-a-1.
# SWITCH: Telemetry:Mode stays unset (⇒ Dps); the dedicated :4056 listener binds regardless. # Telemetry:Mode=Grpc is mandatory as of Phase 6 (cluster-SITE-B + driver); the dedicated
# :4056 listener binds regardless.
Telemetry__GrpcListenPort: "4056" Telemetry__GrpcListenPort: "4056"
Telemetry__Mode: "Grpc"
Telemetry__ApiKey: "telemetry-docker-dev-key" Telemetry__ApiKey: "telemetry-docker-dev-key"
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053" # Per-cluster mesh Phase 6: site-b-2 seeds its OWN "SITE-B" mesh (itself first, site-b-1
# second) — it no longer seeds off central-1.
Cluster__SeedNodes__0: "akka.tcp://otopcua@site-b-2:4053"
Cluster__SeedNodes__1: "akka.tcp://otopcua@site-b-1:4053"
Cluster__Roles__0: "driver" Cluster__Roles__0: "driver"
# Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps", Cluster__Roles__1: "cluster-SITE-B"
# the transport this repo has always used, and every node keeps its ClusterClient wiring built # Mesh command transport (per-cluster mesh Phase 2, made mandatory by Phase 6) — see
# and its comm actor registered with the receptionist either way. Flip the whole rig with # central-1/site-a-1.
# OTOPCUA_MESH_MODE=ClusterClient at `docker compose up` — no compose edit, and no rebuild.
# #
# Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction # Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
# time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an # time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
# actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in # actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
# silence. Both central nodes are listed so a driver node's client can rotate to whichever one # silence. Both central nodes are listed so a driver node's client can rotate to whichever one
# is up — that rotation is why the comm actors are per-node and NOT cluster singletons. # is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Dps}" MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-ClusterClient}"
MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053" MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053" MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
<<: *secrets-env <<: *secrets-env
+29 -2
View File
@@ -103,13 +103,40 @@ The checked-in `appsettings*.json` files are deliberately thin: they carry only
| `Hostname` | string | `0.0.0.0` | Bind hostname. | | `Hostname` | string | `0.0.0.0` | Bind hostname. |
| `Port` | int | `4053` | Cluster transport port. **Duplicated as `ClusterNode.AkkaPort` in the Config DB** — see the note below. | | `Port` | int | `4053` | Cluster transport port. **Duplicated as `ClusterNode.AkkaPort` in the Config DB** — see the note below. |
| `PublicHostname` | string | `127.0.0.1` | Hostname advertised in cluster gossip; must be reachable by peers. **Duplicated as `ClusterNode.Host`** — see the note below. | | `PublicHostname` | string | `127.0.0.1` | Hostname advertised in cluster gossip; must be reachable by peers. **Duplicated as `ClusterNode.Host`** — see the note below. |
| `SeedNodes` | string[] | `[]` | Seed nodes for bootstrapping. **Ordered:** a node that appears in its own list must be entry 0 (only `seed-nodes[0]` can form a new cluster), or the host refuses to start — see [Redundancy.md § Bootstrap: self-first seed ordering](Redundancy.md#bootstrap-self-first-seed-ordering). | | `SeedNodes` | string[] | `[]` | Seed nodes for bootstrapping. **Ordered:** a node that appears in its own list must be entry 0 (only `seed-nodes[0]` can form a new cluster), or the host refuses to start — see [Redundancy.md § Bootstrap: self-first seed ordering](Redundancy.md#bootstrap-self-first-seed-ordering). **Per-pair since Phase 6:** each application `Cluster`'s pair is its own Akka mesh, so a node's `SeedNodes` lists only itself (first) and its own pair partner (second) — never a node from another cluster's pair. |
| `Roles` | string[] | `[]` | Cluster roles for this node. When empty, falls back to `OTOPCUA_ROLES`. Allowed values: `admin`, `driver`, `dev`. | | `Roles` | string[] | `[]` | Cluster roles for this node. When empty, falls back to `OTOPCUA_ROLES`. Allowed values: `admin`, `driver`, `dev`, plus a per-cluster `cluster-{ClusterId}` role (e.g. `cluster-MAIN`, `cluster-SITE-A`) — see [Per-cluster mesh split (Phase 6)](#per-cluster-mesh-split-phase-6-cluster-clusterid-roles--splittopologytransportvalidator) below. |
> The full redundancy model (ServiceLevel tiers, split-brain, peer discovery) is in [`Redundancy.md`](Redundancy.md). The OPC UA peer-URI advertising lives in the `OpcUa:PeerApplicationUris` key above. > The full redundancy model (ServiceLevel tiers, split-brain, peer discovery) is in [`Redundancy.md`](Redundancy.md). The OPC UA peer-URI advertising lives in the `OpcUa:PeerApplicationUris` key above.
> **`Port` / `PublicHostname` are stored twice.** The node binds from these keys; the fleet's `ClusterNode` row records the same address as `AkkaPort` / `Host` so that **central can dial the node without sharing a gossip ring with it** — which is what [per-cluster mesh Phase 2](plans/2026-07-21-per-cluster-mesh-design.md) needs. Nothing in the schema makes the two agree, so `ClusterNodeAddressReconcilerActor` (admin-role singleton) compares them against live membership and logs an Error on mismatch. If you change `Cluster:Port` or `Cluster:PublicHostname` on a node, **update its `ClusterNode` row too** — a node binding 4054 with a row saying 4053 still gossips fine today, and fails silently in Phase 2. The row's `NodeId` is `host:port` and is also the deploy path's ack identity, so a drifted node's deployment acks stop matching as well. See [`config-db-schema.md` § `ClusterNode`](v2/config-db-schema.md#clusternode). > **`Port` / `PublicHostname` are stored twice.** The node binds from these keys; the fleet's `ClusterNode` row records the same address as `AkkaPort` / `Host` so that **central can dial the node without sharing a gossip ring with it** — which is what [per-cluster mesh Phase 2](plans/2026-07-21-per-cluster-mesh-design.md) needs. Nothing in the schema makes the two agree, so `ClusterNodeAddressReconcilerActor` (admin-role singleton) compares them against live membership and logs an Error on mismatch. If you change `Cluster:Port` or `Cluster:PublicHostname` on a node, **update its `ClusterNode` row too** — a node binding 4054 with a row saying 4053 still gossips fine today, and fails silently in Phase 2. The row's `NodeId` is `host:port` and is also the deploy path's ack identity, so a drifted node's deployment acks stop matching as well. See [`config-db-schema.md` § `ClusterNode`](v2/config-db-schema.md#clusternode).
### Per-cluster mesh split (Phase 6): `cluster-{ClusterId}` roles + `SplitTopologyTransportValidator`
- **Purpose:** [per-cluster mesh](plans/2026-07-22-per-cluster-mesh-program.md) **Phase 6** split the
single fleet-wide Akka mesh into one independent 2-node mesh per application `Cluster`. A node's
`Cluster:Roles` now carries a `cluster-{ClusterId}` role alongside `driver` (and `admin` for a fused
central node) — e.g. `driver,cluster-SITE-A` — and its `Cluster:SeedNodes` lists only itself and its
own pair partner (see the `Roles`/`SeedNodes` rows above). Nodes in different clusters never gossip
with each other.
- **`SplitTopologyTransportValidator`** (`src/Core/ZB.MOM.WW.OtOpcUa.Cluster/`, wired at
`ValidateOnStart` alongside the other `Cluster`-family validators) fails host start on any node
carrying a `cluster-{ClusterId}` role unless it is also configured for the split-safe transports:
`MeshTransport:Mode = ClusterClient`, `Telemetry:Mode = Grpc` on a driver node, `TelemetryDial:Mode =
Grpc` on an admin node. A DPS-mode transport on a cluster-role node would try to gossip across a
partition boundary that no longer exists, so the validator fails closed rather than let commands or
telemetry silently vanish. A legacy node with no `cluster-{ClusterId}` role is exempt and may stay on
`Dps`.
- **The compiled transport-mode defaults were deliberately NOT flipped.** `MeshTransport:Mode`,
`Telemetry:Mode`, and `TelemetryDial:Mode` (documented below) all still default to `Dps` — a
blast-radius assessment found that flipping the compiled default would break every integration test
and every existing `appsettings.json` profile that leaves these keys unset. `SplitTopologyTransportValidator`
plus explicit per-deployment configuration is the guardrail, not a default flip; every node that
actually carries a `cluster-{ClusterId}` role must set the split-safe modes explicitly.
- See [`Redundancy.md` § Per-cluster meshes (Phase 6)](Redundancy.md#per-cluster-meshes-phase-6) for the
full runtime picture (one Primary per pair, the cluster-scoped `RedundancyStateActor` singleton,
`ClusterNodeAddressReconciler` own-cluster-scoping, and central's one-`ClusterClient`-per-`Cluster`
fan-out).
### `MeshTransport` (central↔node command transport) ### `MeshTransport` (central↔node command transport)
- **Purpose:** selects how central's three command channels (`deployments`, `driver-control`, `alarm-commands`) and the node's `deployment-acks` reply reach the other side — over the mesh-wide DistributedPubSub, or over an Akka `ClusterClient` that does **not** require central and the node to share a gossip ring. The second option is what [per-cluster mesh](plans/2026-07-21-per-cluster-mesh-design.md) Phase 6 needs once the meshes split. - **Purpose:** selects how central's three command channels (`deployments`, `driver-control`, `alarm-commands`) and the node's `deployment-acks` reply reach the other side — over the mesh-wide DistributedPubSub, or over an Akka `ClusterClient` that does **not** require central and the node to share a gossip ring. The second option is what [per-cluster mesh](plans/2026-07-21-per-cluster-mesh-design.md) Phase 6 needs once the meshes split.
+61 -25
View File
@@ -108,22 +108,59 @@ plane (writes, alarm acks, the alerts emit, the alarm-history drain) on the wron
*higher* address so the two orderings genuinely disagree. `NodeRedundancyState.IsRoleLeaderForDriver` *higher* address so the two orderings genuinely disagree. `NodeRedundancyState.IsRoleLeaderForDriver`
was renamed `IsDriverPrimary` to stop the name asserting the old derivation. was renamed `IsDriverPrimary` to stop the name asserting the old derivation.
> **KNOWN LIMITATION — the election is per *Akka* cluster, not per application `Cluster`.** > **RESOLVED by Phase 6 — the election is now per application `Cluster`, not per Akka mesh.**
> The election yields exactly **one** Primary across the whole Akka cluster. A fleet that > Phase 6 of the per-cluster mesh program split the single fleet-wide Akka mesh into one independent
> runs several application clusters (each with its own redundant pair) inside one Akka cluster — > 2-node mesh per application `Cluster` (see [Per-cluster meshes (Phase 6)](#per-cluster-meshes-phase-6)
> which is what `docker-dev/docker-compose.yml` models, with MAIN, SITE-A and SITE-B — therefore has > below). `RedundancyStateActor` is now a `cluster-{ClusterId}`-scoped singleton spawned on every driver
> one Primary among *all* its driver nodes, not one per pair. Every consumer of the role inherits > node, so each pair elects its own Primary — the whole-Akka-cluster mismatch this block used to describe
> this: `ServiceLevel`, the `alerts` emit gate, and the inbound device-write gate below. > no longer exists. See `docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md`.
> >
> The unit of redundancy everywhere else in the product is the application `Cluster` (`ClusterId`) — > Originally surfaced by the LocalDb Phase 2 live gate, where the old whole-mesh election stopped the
> it owns drivers, devices and `ClusterNode` rows, and the LocalDb deployment-artifact cache is > alarm-history drain on every node in the fleet — see `docs/plans/2026-07-20-localdb-phase2-live-gate.md`.
> already keyed by it *so that a pair shares one entry*. Scoping the election the same way is the
> fix; `RedundancyStateActor` is an admin-role singleton in the same assembly as ## Per-cluster meshes (Phase 6)
> `AdminOperationsActor`, which already reads and groups `ClusterNodes` by cluster.
> The fleet no longer runs one Akka mesh. Phase 6 split it into **three independent 2-node meshes** — the
> Surfaced by the LocalDb Phase 2 live gate, where it stopped the alarm-history drain on every node central pair (roles `admin,driver,cluster-MAIN`), the site-a pair (`driver,cluster-SITE-A`), and the
> in the fleet — see `docs/plans/2026-07-20-localdb-phase2-live-gate.md`. The alarm drain no longer site-b pair (`driver,cluster-SITE-B`) — all sharing the same `ActorSystem` name (`otopcua`) on one network,
> depends on this (it defers only to a node that shares its queue), but the other three gates still do. separated purely by **per-pair self-first seed partitioning**: each node's `Cluster:SeedNodes` lists only
itself and its own pair partner, so gossip never crosses a pair boundary (see
[Bootstrap: self-first seed ordering](#bootstrap-self-first-seed-ordering)).
Consequences:
- **One Primary per pair — two or more Primaries fleet-wide, by design.** Each mesh runs its own
election, so a three-mesh fleet has three Primaries, one per application `Cluster`. This is correct,
not a regression of the old single-Primary assumption.
- **`RedundancyStateActor` is a `cluster-{ClusterId}`-scoped singleton**, spawned on every driver node
(falling back to the plain `driver` role for a legacy node not carrying a cluster role). Each pair
elects and publishes its own `redundancy-state` on its own mesh's DPS — the DPS topic name is
unchanged, but it no longer reaches past the pair.
- **`ClusterNodeAddressReconciler` is own-cluster-scoped** — an admin node can only reconcile
`ClusterNode` rows belonging to its own cluster; it has no gossip visibility into another mesh to
reconcile against.
- **`CentralCommunicationActor` creates one `ClusterClient` per application `Cluster`** and fans
commands across them, rather than one fleet-wide client. Central reaches a site mesh over
`ClusterClient` (commands), gRPC (telemetry — see `docs/Telemetry.md`), and `ConfigServe` (config
fetch — see [`ConfigSource`/`ConfigServe`](../CLAUDE.md#config-source-configsource--configserve)),
never over gossip.
- **`SplitTopologyTransportValidator` fails host start** on a node carrying a `cluster-{ClusterId}`
role unless it is also configured for the split-safe transports: `MeshTransport:Mode = ClusterClient`,
`Telemetry:Mode = Grpc` (driver nodes), `TelemetryDial:Mode = Grpc` (admin nodes). A DPS-mode transport
on a cluster-role node would silently try to gossip across a partition that no longer exists, so the
validator fails closed instead. Legacy/non-cluster-role nodes are exempt.
- **The compiled transport-mode defaults were deliberately NOT flipped**`MeshTransport:Mode`,
`Telemetry:Mode`, and `TelemetryDial:Mode` all still default to `Dps`. A blast-radius assessment
found flipping the compiled default would break every integration test and every existing
`appsettings.json` profile that doesn't explicitly set these keys. The guardrail is the validator
above plus explicit per-deployment configuration, not a default flip.
### Secrets replication is pair-local
Akka DPS secret replication is scoped to each 2-node mesh: a pair replicates its own secrets to its
own partner and no further. There is no fleet-wide or cross-cluster secret sync after the split — by
design, consistent with Phase 4 (site nodes have no SQL, so there is no SQL-backed hub to fan a
cross-mesh sync through).
## Data flow ## Data flow
@@ -316,18 +353,17 @@ advertises `ServiceLevel` 250. OPC UA clients re-select.
| Confirmation | A dialog naming the node, and what follows (restart, ServiceLevel 250 moves, clients re-select). | | Confirmation | A dialog naming the node, and what follows (restart, ServiceLevel 250 moves, clients re-select). |
| Audit | Written through the shared `ZB.MOM.WW.Audit.IAuditWriter` seam **before** the Leave — action `cluster.manual-failover`, actor, target. A *refused* failover changes nothing and writes nothing. | | Audit | Written through the shared `ZB.MOM.WW.Audit.IAuditWriter` seam **before** the Leave — action `cluster.manual-failover`, actor, target. A *refused* failover changes nothing and writes nothing. |
> **Mesh-scope caveat.** Until the per-cluster mesh work lands Phase 6 of the mesh program (see [Per-cluster meshes (Phase 6)](#per-cluster-meshes-phase-6)) split
> (`docs/plans/2026-07-21-per-cluster-mesh-design.md`), the Primary is elected once per Akka mesh rather than the fleet into one 2-node mesh per application `Cluster`, so the button now acts on **this pair's own**
> per application `Cluster` row — so on a fleet running several clusters in one mesh this button acts on the Primary — there is no longer a whole-mesh vs. per-cluster ambiguity to caveat.
> whole mesh's Primary, which may belong to a different cluster than the page you are on. The panel says so.
> Phase 6 of the mesh program removes both the caveat and this notice.
> **Live gate outstanding.** The auto-down change is verified by unit tests against the effective > **Live gate outstanding.** The auto-down change is verified by unit tests against the effective
> configuration, and by the sister project's live drill on an equivalent Akka version and topology. It has > configuration, and by the sister project's live drill on an equivalent Akka version and topology. It
> **not** yet been drilled on an OtOpcUa two-node rig — the docker-dev rig runs a single six-node mesh, where > was **not** drillable on an OtOpcUa rig before Phase 6, because the docker-dev rig ran a single
> the 1-vs-1 pathology cannot occur. The drill (kill the oldest container of a genuine two-node cluster; > six-node mesh where the 1-vs-1 pathology cannot occur. **Phase 6 has landed and every mesh is now
> assert the survivor stays up, takes the singletons, and reports the primary `ServiceLevel`) should run > exactly two nodes**, so the drill (kill the oldest container of a genuine two-node cluster; assert
> alongside the per-cluster mesh work, which makes every mesh exactly two nodes. See > the survivor stays up, takes the singletons, and reports the primary `ServiceLevel`) is finally
> testable — deferred to Phase 7's failover drill. See
> `docs/plans/2026-07-21-per-cluster-mesh-design.md` §6.2 / Phase 0a. > `docs/plans/2026-07-21-per-cluster-mesh-design.md` §6.2 / Phase 0a.
## Bootstrap: self-first seed ordering ## Bootstrap: self-first seed ordering
@@ -312,7 +312,7 @@ Deliberately not a task plan — per-phase plans follow, one at a time.
| 3 | Config fetch-and-cache: artifact served by central, driver nodes read LocalDb (§6.1) | No | | 3 | Config fetch-and-cache: artifact served by central, driver nodes read LocalDb (§6.1) | No |
| 4 | Cut the driver-side ConfigDb connection: re-home `EfAlarmConditionStateStore`, resolve the `DbHealthProbeActor` ServiceLevel input, audit `OpcUaPublishActor` | No | | 4 | Cut the driver-side ConfigDb connection: re-home `EfAlarmConditionStateStore`, resolve the `DbHealthProbeActor` ServiceLevel input, audit `OpcUaPublishActor` | No |
| 5 | gRPC stream contract; migrate the seven observability topics | No | | 5 | gRPC stream contract; migrate the seven observability topics | No |
| 6 | Mesh partition: per-cluster seed nodes, cluster-scoped roles + singletons; rig models the real topology | No | | 6 | Mesh partition: per-cluster seed nodes, cluster-scoped roles + singletons; rig models the real topology | No**DONE 2026-07-24**, see `2026-07-22-per-cluster-mesh-program.md` §Phase 6 and `2026-07-24-mesh-phase6-partition-and-colocation.md` |
| 7 | Failover drill (both directions, per §6.2); live gate | No | | 7 | Failover drill (both directions, per §6.2); live gate | No |
Phases 3 and 4 are the ones that change a running system's data path rather than its wiring, and each Phases 3 and 4 are the ones that change a running system's data path rather than its wiring, and each
@@ -263,25 +263,45 @@ kill-and-reconnect of a site node's dialer recovers every stream (driver-health/
from the node hub's snapshot replay). Treat Phase 5 as **code-complete, not verified**, until this from the node hub's snapshot replay). Treat Phase 5 as **code-complete, not verified**, until this
lands — see `docs/plans/2026-07-23-mesh-phase5-grpc-telemetry-stream.md` Task 12. lands — see `docs/plans/2026-07-23-mesh-phase5-grpc-telemetry-stream.md` Task 12.
### Phase 6 — Mesh partition + co-location topology ### Phase 6 — Mesh partition + co-location topology — **DONE 2026-07-24**
**Scope:** per-cluster seed nodes — ~~adopt ScadaBridge's self-first ordering and RETIRE the **Scope:** per-cluster seed nodes — ~~adopt ScadaBridge's self-first ordering and RETIRE the
`SelfFormAfter` watchdog + TCP reachability guard~~ **DONE EARLY 2026-07-22** (docker-dev `SelfFormAfter` watchdog + TCP reachability guard~~ **DONE EARLY 2026-07-22** (docker-dev
`central-2` swapped to self-first, `ClusterBootstrapFallback`/`SelfFormAfter` deleted, `central-2` swapped to self-first, `ClusterBootstrapFallback`/`SelfFormAfter` deleted,
`AkkaClusterOptionsValidator` ports ScadaBridge's startup rule in its conditional form, and `AkkaClusterOptionsValidator` ports ScadaBridge's startup rule in its conditional form, and
`SelfFirstSeedBootstrapTests` replaces `SelfFormBootstrapTests`; see `docs/Redundancy.md` `SelfFirstSeedBootstrapTests` replaces `SelfFormBootstrapTests`; see `docs/Redundancy.md`
§"Bootstrap: self-first seed ordering"). What remains for this phase is the per-pair §"Bootstrap: self-first seed ordering"). The rest of the phase — actually splitting the single
`Cluster__SeedNodes__*` matrix once the meshes actually split — every node in a pair is then a seed fleet-wide mesh into three independent 2-node meshes (central `cluster-MAIN`, site-a
of its own mesh, so the site-node exemption disappears; cluster-scoped roles `cluster-{ClusterId}` + singleton re-scoping, `cluster-SITE-A`, site-b `cluster-SITE-B`) — **shipped 2026-07-24**. See
central pair keeps the admin singletons; **docker-dev rig rewritten** to model the real topology — `docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md` for the full plan and task list.
including the co-location port table above (both products' compose files on shared per-site Delivered:
networks, real LocalDb sync ports); remove the ClusterRedundancy page's mesh-scope caveat (the - Per-pair `Cluster:SeedNodes` (each node lists itself first, its pair partner second — no
election is pair-local now) and the fallback's site-node island-guard docs note (moot — every cross-pair seeds) + `cluster-{ClusterId}` roles on every driver node + singleton re-scoping:
node is a seed of its own mesh); `Cluster__SeedNodes__*` env matrix per pair. `RedundancyStateActor` is now a `cluster-{ClusterId}`-scoped singleton spawned on every driver
**Exit gate:** rig up in the new shape; every existing live-gated behavior re-verified per pair node (falls back to plain `driver` for legacy nodes), so **each pair elects its own Primary —
(deploy, redundancy 250/240 per pair — **two Primaries fleet-wide, one per pair, by design**); two or more Primaries fleet-wide, by design**; `ClusterNodeAddressReconciler` is own-cluster-scoped.
secrets Akka replication re-verified or re-scoped (it rides DPS on the current single mesh — its - `CentralCommunicationActor` now creates one `ClusterClient` per application `Cluster` and fans
topology must be re-decided here, likely SQL-hub mode like ScadaBridge, since pub/sub cannot commands across them (closes the `TODO(Phase 6)` left by Phase 2).
cross separate meshes). - New `SplitTopologyTransportValidator` fails host start on a `cluster-{ClusterId}`-role node left
on a DPS transport (`MeshTransport:Mode` must be `ClusterClient`; `Telemetry:Mode`/`TelemetryDial:Mode`
must be `Grpc`) — see `docs/Configuration.md` §"Per-cluster mesh split (Phase 6)".
- **Three decisions worth recording:**
1. **Secrets replication went pair-local, not SQL-hub.** The design flagged "likely SQL-hub mode
like ScadaBridge" as the fallback if DPS couldn't cross the split meshes — in the event, each
pair simply keeps replicating its own secrets within itself (no cross-cluster sync needed, and
no SQL-hub was built, consistent with Phase 4: site nodes have no SQL to hub through).
2. **The compiled transport-mode defaults (`MeshTransport:Mode`, `Telemetry:Mode`,
`TelemetryDial:Mode`) were deliberately left at `Dps`**, not flipped to the split-safe modes —
a blast-radius assessment found flipping the default breaks every integration test and every
appsettings profile that leaves the key unset. The guardrail is
`SplitTopologyTransportValidator` plus explicit per-deployment config, not a default flip.
3. **The docker-dev rig models three OtOpcUa-only meshes** (central/site-a/site-b), not the
ScadaBridge co-location topology sketched in the design's port table — co-location with
ScadaBridge's own site nodes stayed out of scope for this phase.
**Exit gate:** rig up in the new three-mesh shape; every existing live-gated behavior re-verified
per pair (deploy, redundancy 250/240 per pair, two-Primaries-fleet-wide confirmed); the
ClusterRedundancy page's mesh-scope caveat and `IManualFailoverService`'s doc-comment caveat
removed (the election is pair-local now); `docs/Redundancy.md`'s per-Akka-cluster KNOWN LIMITATION
marked resolved.
### Phase 7 — Failover drill + live gates ### Phase 7 — Failover drill + live gates
**Scope:** the drill ScadaBridge already has (`failover-drill.sh` analogue) run per pair, both **Scope:** the drill ScadaBridge already has (`failover-drill.sh` analogue) run per pair, both
@@ -318,5 +338,5 @@ resource sizing on the site VMs should be checked once both products run the ful
| 3 fetch-and-cache | **DONE 2026-07-23**, merged `d01b0695`, live gate PASSED, pushed to origin (scadaproj umbrella index updated + pushed `b5e7bc8`). gRPC fetch RPC + shared node key; dark switch `ConfigSource:Mode` (Direct default). Rig flip: `OTOPCUA_CONFIG_MODE=FetchAndCache` on the site nodes. See `2026-07-22-mesh-phase3-config-fetch-and-cache.md`. | | 3 fetch-and-cache | **DONE 2026-07-23**, merged `d01b0695`, live gate PASSED, pushed to origin (scadaproj umbrella index updated + pushed `b5e7bc8`). gRPC fetch RPC + shared node key; dark switch `ConfigSource:Mode` (Direct default). Rig flip: `OTOPCUA_CONFIG_MODE=FetchAndCache` on the site nodes. See `2026-07-22-mesh-phase3-config-fetch-and-cache.md`. |
| 4 cut driver ConfigDb | **DONE 2026-07-23 — live gate PASSED** on `feat/mesh-phase4` (Tasks 08 + 1b + 1011; Task 9 table-drop deferred). ConfigDb admin-only; driver-only ⇒ FetchAndCache (validator); `DbHealthProbeActor` not spawned driver-only (client-visible ServiceLevel 240/250 with central SQL down — proven live); alarm condition state in replicated LocalDb `alarm_condition_state`; central persists acks; `OpcUaPublish` guard split fixes the driver-only address-space wipe; driver-only LDAP maps from appsettings only (6th consumer found mid-phase). Gate: deploy sealed green w/ 4 DB-less site nodes acking, ServiceLevel held 240 w/ SQL down, restart booted last-known-good from the LocalDb pointer. See `2026-07-23-mesh-phase4-cut-driver-configdb.md` + `2026-07-23-mesh-phase4-live-gate.md`. | | 4 cut driver ConfigDb | **DONE 2026-07-23 — live gate PASSED** on `feat/mesh-phase4` (Tasks 08 + 1b + 1011; Task 9 table-drop deferred). ConfigDb admin-only; driver-only ⇒ FetchAndCache (validator); `DbHealthProbeActor` not spawned driver-only (client-visible ServiceLevel 240/250 with central SQL down — proven live); alarm condition state in replicated LocalDb `alarm_condition_state`; central persists acks; `OpcUaPublish` guard split fixes the driver-only address-space wipe; driver-only LDAP maps from appsettings only (6th consumer found mid-phase). Gate: deploy sealed green w/ 4 DB-less site nodes acking, ServiceLevel held 240 w/ SQL down, restart booted last-known-good from the LocalDb pointer. See `2026-07-23-mesh-phase4-cut-driver-configdb.md` + `2026-07-23-mesh-phase4-live-gate.md`. |
| 5 gRPC telemetry | **DONE 2026-07-23 — live gate PASSED** on `feat/mesh-phase5`. 4-channel scope (`alerts`/`script-logs`/`driver-health`/`driver-resilience-status`), 3 deferred with rationale (`redundancy-state`, `fleet-status`, `deployment-acks`); dark switch `Telemetry:Mode`/`TelemetryDial:Mode` (Dps default); node hosts server / central dials; auth-from-day-one fail-closed bearer key, superseding design §6.3. Gate: full 12-stream mesh formed in Grpc mode (2 inbound per driver node, 6 outbound per central), AdminUI pill live (data path proven), kill-and-reconnect of a node recovered its stream in ~5s; Dps baseline dials nothing. Surfaced an upgrade gotcha — `ClusterNode.GrpcPort` must be populated on existing deployments (fresh installs seed it; nullable column doesn't backfill) — which live-validated the graceful null-`GrpcPort` skip. See `docs/Telemetry.md`, `2026-07-23-mesh-phase5-grpc-telemetry-stream.md` + `2026-07-23-mesh-phase5-live-gate.md`. | | 5 gRPC telemetry | **DONE 2026-07-23 — live gate PASSED** on `feat/mesh-phase5`. 4-channel scope (`alerts`/`script-logs`/`driver-health`/`driver-resilience-status`), 3 deferred with rationale (`redundancy-state`, `fleet-status`, `deployment-acks`); dark switch `Telemetry:Mode`/`TelemetryDial:Mode` (Dps default); node hosts server / central dials; auth-from-day-one fail-closed bearer key, superseding design §6.3. Gate: full 12-stream mesh formed in Grpc mode (2 inbound per driver node, 6 outbound per central), AdminUI pill live (data path proven), kill-and-reconnect of a node recovered its stream in ~5s; Dps baseline dials nothing. Surfaced an upgrade gotcha — `ClusterNode.GrpcPort` must be populated on existing deployments (fresh installs seed it; nullable column doesn't backfill) — which live-validated the graceful null-`GrpcPort` skip. See `docs/Telemetry.md`, `2026-07-23-mesh-phase5-grpc-telemetry-stream.md` + `2026-07-23-mesh-phase5-live-gate.md`. |
| 6 mesh partition + co-location | not started | | 6 mesh partition + co-location | **DONE 2026-07-24** — three independent 2-node meshes (central/site-a/site-b), `cluster-{ClusterId}` roles, per-pair self-first seeds, cluster-scoped `RedundancyStateActor` singleton (two+ Primaries fleet-wide, by design), one `ClusterClient` per `Cluster`, `SplitTopologyTransportValidator`, pair-local secrets replication, compiled transport defaults intentionally kept `Dps`. See `2026-07-24-mesh-phase6-partition-and-colocation.md`. |
| 7 drill + live gates | not started | | 7 drill + live gates | not started |
@@ -0,0 +1,540 @@
# Per-Cluster Mesh Phase 6 — Mesh Partition + Co-location Topology
> **For Claude:** REQUIRED SUB-SKILL: execute this plan task-by-task with
> superpowers-extended-cc:subagent-driven-development (the established mode for this workstream).
**Goal:** Split OtOpcUa's single fleet-wide Akka mesh into **one 2-node mesh per application
`Cluster`** (central pair + one pair per site), so every Primary-gated decision and every gated
resource shares one pair-local scope — the ScadaBridge shape. Same ActorSystem name (`otopcua`)
everywhere; separation is by **seed-node partitioning** only.
**Architecture:** Each pair's two nodes list only each other as seeds (self-first), so Akka forms
three independent meshes even on a shared network. Driver nodes carry a new `cluster-{ClusterId}`
role; the one pair-local singleton (`RedundancyStateActor`) re-scopes to it and is spawned on every
driver node, so each pair elects its own Primary and publishes `redundancy-state` on its own mesh's
DPS. Central (admin) reaches each site mesh over the explicit Phase 2/3/5 transports (ClusterClient
+ ConfigServe gRPC + telemetry gRPC), now partitioned **one ClusterClient per cluster**. The five
admin singletons stay on the central pair. Two admin-side consumers that assumed fleet-wide gossip
visibility (`ClusterNodeAddressReconciler`, `CentralCommunicationActor.RebuildClient`) are
re-scoped. Secrets replication becomes **pair-local** automatically (Akka DPS within each 2-node
mesh; no fleet-wide sync, no central-SQL dependency — consistent with Phase 4). A startup validator
refuses to boot a split-configured node (one carrying a `cluster-` role) that is still pointed at a
DPS transport mode, and the compiled transport defaults flip to the split-correct values.
**Tech Stack:** .NET 10, Akka.NET + Akka.Hosting (`WithSingleton`/`WithActors`), Akka.Cluster.Tools
(ClusterClient + ClusterSingleton + DistributedPubSub), Akka.Cluster.Hosting `ClusterSingletonOptions`,
xUnit + Shouldly, docker-dev compose rig.
**Decisions settled with the user 2026-07-24 (recon-driven; the first contradicts the program doc's
own guess):**
1. **Secrets after split = pair-local Akka.** The program doc guessed "SQL-hub mode like ScadaBridge,"
but recon showed sites have **no SQL Server** (the point of Phase 4), so a SQL hub would force site
nodes back onto central SQL. Instead: leave the `AddZbSecretsAkkaReplication` wiring untouched — it
becomes per-pair for free once each mesh is two nodes. Fleet-wide secret sync is dropped by design;
document it. No library re-wire.
2. **DPS branch disposition = flip + change defaults + validator, keep the code.** The rig flips to
`ClusterClient`/`Grpc`/`FetchAndCache`; the compiled defaults change so a fresh deploy is
split-correct; the DPS branch code stays as dark switches; a new startup validator makes a
split-configured node refuse to boot on a DPS transport mode (neutralizes the silent-break footgun
without a large irreversible deletion). `redundancy-state`/`fleet-status` stay on DPS either way.
3. **Rig scope = OtOpcUa-only per-cluster meshes.** Rewrite docker-dev into 3 independent 2-node
meshes on a single shared network (faithful to production, where separation is by seeds over a WAN,
not by network isolation). Physical co-location with ScadaBridge is documented as the production
topology, not literally run in docker-dev.
**Authoritative context (read before implementing):**
- `docs/plans/2026-07-22-per-cluster-mesh-program.md` §"Phase 6" (lines 266284) + the tracking table.
- `docs/plans/2026-07-21-per-cluster-mesh-design.md` §4 (oldest-Up election, already fixed), §5
(target architecture), §7 (sequencing).
- Recon findings (this session) — summarized inline per task below.
---
## What Phase 6 does NOT change (scope guards)
- **ActorSystem name stays `otopcua`.** Do not rename it or make it per-cluster.
- **`AkkaClusterOptions` gains no new property.** `SeedNodes` + `Roles` already carry per-node values
from config; the split is in *how they're populated per pair* (rig/appsettings) + the `RoleParser`
allow-list + singleton scoping. Do not add a `ClusterId` option — every `ClusterNode` row is already
defined to be a driver node (Phase 1 decision), and roles are the declaration of record.
- **`SelectDriverPrimary` election logic is unchanged.** It filters on the `driver` role over
`Cluster.State.Members`; after the split that set is pair-local, so it elects the oldest of two.
No edit to the algorithm.
- **The five admin singletons stay admin-scoped** (`config-publish`, `admin-operations`,
`audit-writer`, `fleet-status`, `cluster-node-address-reconciler`). Only `redundancy-state`
re-homes.
- **`DriverMemberCount()`** in `DriverHostActor`/`ScriptedAlarmHostActor` needs **no code change**
it reads whole-mesh members filtered by `driver`, which becomes pair-local for free. (Note the
semantics change in docs; do not "fix" it in code.)
- **Do NOT delete the DPS command/telemetry branches** (decision 2). Do NOT switch the network to
per-site isolation or wire in ScadaBridge (decision 3). Do NOT re-wire secrets to SQL (decision 1).
---
### Task 0: `RoleParser` accepts `cluster-{ClusterId}`; consolidate the `driver` role literal
**Classification:** small
**Estimated implement time:** ~4 min
**Parallelizable with:** none (blocks 1, 2, 5, 7)
**Context:** `RoleParser.Allowed` is a fixed `HashSet {"admin","driver","dev"}`
(`src/Core/ZB.MOM.WW.OtOpcUa.Cluster/RoleParser.cs:5-8`). A `cluster-{ClusterId}` role fails it, so
no node can carry one. Also, three independent `"driver"` string literals exist
(`RedundancyStateActor.DriverRole`, `ClusterNodeAddressReconcilerActor.DriverRole`,
`Runtime/ServiceCollectionExtensions.DriverRole`) — consolidate to a single source now since every
Phase 6 task touches role code.
**Files:**
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/RoleParser.cs`
- Test: `tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/RoleParserTests.cs` (create if absent)
**Steps:**
1. Add a well-known-roles surface to `RoleParser` (or a new `ClusterRoles` static): constants
`Admin = "admin"`, `Driver = "driver"`, `Dev = "dev"`, `ClusterRolePrefix = "cluster-"`, and a
helper `bool IsClusterRole(string role) => role.StartsWith(ClusterRolePrefix, StringComparison.Ordinal) && role.Length > ClusterRolePrefix.Length`.
2. Change `Parse` validation: a role is allowed if it is one of the three fixed roles **or**
`IsClusterRole(role)` is true. Keep the existing trim/lowercase/empty-filter behavior. A role of
exactly `"cluster-"` (empty ClusterId) is **rejected**.
3. Add `RoleParser.ClusterIdFromRole(string) => role.Substring(ClusterRolePrefix.Length)` returning
the ClusterId for a cluster role (used by later tasks).
4. Update `RedundancyStateActor.DriverRole`, `ClusterNodeAddressReconcilerActor.DriverRole`, and
`Runtime/ServiceCollectionExtensions.DriverRole` to reference the single `RoleParser.Driver` (keep
the public `const`/`static readonly` symbols as aliases if any test references them by their old
name — grep first; do not break existing test references).
5. Tests: `admin`/`driver`/`dev`/`cluster-MAIN`/`cluster-SITE-A` accepted; `cluster-` rejected;
`bogus` rejected; case/whitespace normalization preserved; `ClusterIdFromRole("cluster-SITE-A") == "SITE-A"`.
**Acceptance:** `dotnet test tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests` green; `cluster-{X}` roles
parse; the three literals resolve to one constant.
---
### Task 1: `ClusterRoleInfo` exposes the node's own cluster-scoped role
**Classification:** small
**Estimated implement time:** ~4 min
**Parallelizable with:** none (needs Task 0; blocks 2, 5)
**Context:** Singleton registration and the validator both need "what is *this* node's
`cluster-{ClusterId}` role." `ClusterRoleInfo` (`src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ClusterRoleInfo.cs`,
`IClusterRoleInfo`, registered in `AddOtOpcUaCluster`) is the existing role/leader authority — extend
it.
**Files:**
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ClusterRoleInfo.cs`
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/IClusterRoleInfo.cs` (or wherever the interface lives)
- Test: `tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/ClusterRoleInfoTests.cs`
**Steps:**
1. Add `string? ClusterRole { get; }` (e.g. `"cluster-SITE-A"`) and `string? ClusterId { get; }` (e.g.
`"SITE-A"`) to `IClusterRoleInfo`, derived from the configured `AkkaClusterOptions.Roles` via
`RoleParser.IsClusterRole` (first match; log a Warning and pick the first if >1 cluster role is
configured — that is a misconfiguration). Null when the node carries no cluster role (e.g. a
pre-split/admin-only-legacy node).
2. Do NOT read `Cluster.State` for this — it is the node's **own configured** identity, available
before the cluster forms (the singleton registration runs at host build time).
3. Tests: a node configured `["driver","cluster-SITE-A"]` reports `ClusterRole="cluster-SITE-A"`,
`ClusterId="SITE-A"`; a node `["admin","driver"]` reports both null; two cluster roles → first wins
+ warning path exercised.
**Acceptance:** `ClusterRoleInfo` reports the node's cluster role/id from config; tests green.
---
### Task 2: Re-home `RedundancyStateActor` to a `cluster-{ClusterId}` singleton on driver nodes
**Classification:** high-risk
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 3, Task 4 (disjoint files)
**Context (the crux).** Today `RedundancyStateActor` is registered as an **admin** singleton
(`ControlPlane/ServiceCollectionExtensions.cs:133-136`, `singletonOptions.Role = "admin"`), electing a
**fleet-wide** Primary and publishing `redundancy-state` over DPS. After the split, central's mesh
sees only its own pair — so a site pair (driver-only, no admin node) would have **no one** to elect its
Primary. Fix: scope the singleton to the node's `cluster-{ClusterId}` role and spawn it on **every
driver node** (`hasDriver`), so each mesh (central pair = `cluster-MAIN`, each site pair =
`cluster-{SITE}`) has exactly one, electing its own pair-local Primary and publishing on its own mesh's
DPS. **De-risked:** `RedundancyStateActor` has zero injected deps (verified — it only uses
`Cluster.Get(Context.System)` + DPS), so it spawns cleanly on a driver-only node.
**Files:**
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/ServiceCollectionExtensions.cs`
(remove `redundancy-state` from `WithOtOpcUaControlPlaneSingletons`; add a new
`WithOtOpcUaClusterRedundancySingleton(IClusterRoleInfo)` extension that registers it scoped to the
cluster role)
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs` (call the new extension on `hasDriver`, not
`hasAdmin`; a fused `admin,driver` node calls it once via the driver branch)
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/...` singleton-role test; and confirm the
admin path no longer registers `redundancy-state`.
**Steps:**
1. In `ControlPlane/ServiceCollectionExtensions.cs`: delete the `WithSingleton` block for
`RedundancyStateActorKey`/`redundancy-state` from the admin method (lines ~133-136). Keep the
registry key type.
2. Add:
```csharp
public static AkkaConfigurationBuilder WithOtOpcUaClusterRedundancySingleton(
this AkkaConfigurationBuilder builder, IClusterRoleInfo roleInfo)
{
// Cluster-scoped when the node carries a cluster-{ClusterId} role (Phase-6 split: one
// redundancy singleton per pair, robust even if two meshes accidentally merged). Falls
// back to the driver role for a legacy single-mesh node or a test harness with no cluster
// role — where driver-scope is the pre-Phase-6 fleet-wide behavior, and on a genuinely
// split 2-node mesh the driver role is ALREADY pair-local because the mesh IS the pair.
// No throw: decision 2 requires legacy/harness nodes to keep booting unchanged.
var role = roleInfo.ClusterRole ?? RoleParser.Driver;
var opts = new ClusterSingletonOptions { Role = role };
return builder.WithSingleton<RedundancyStateActorKey>(
RedundancyStateActor.Topic, RedundancyStateActor.Props(), opts /* createProxyToo per existing pattern */);
}
```
Match the exact `WithSingleton` overload/signature the other five singletons use (check createProxy
arg + singleton name conventions).
3. In `Program.cs`: within the `if (hasDriver)` Akka-configurator block (near line 357-374, alongside
`WithOtOpcUaRuntimeActors`), call `ab.WithOtOpcUaClusterRedundancySingleton(clusterRoleInfo)`.
Resolve `IClusterRoleInfo` from `sp`. Ensure it runs on the fused central node too (central has
`hasDriver=true`). Do NOT also register it in the `hasAdmin` block.
4. **No fail-fast on a missing cluster role** — the fallback keeps legacy/harness nodes booting
(decision 2). A *misconfigured* Phase-6 node (split transports, no cluster role) is a non-issue for
the redundancy singleton: driver-scope is pair-local on a split mesh anyway.
5. Tests: assert the admin singleton set no longer contains `redundancy-state`; assert the new
extension registers a singleton scoped to `Role == "cluster-SITE-A"` when the roleInfo reports that;
assert that a roleInfo with null `ClusterRole` falls back to `Role == "driver"` (no throw).
**Acceptance:** each mesh hosts exactly one `RedundancyStateActor` scoped to its cluster role; a
driver-only site node elects its own Primary; admin path no longer owns it. Unit tests green.
**Live-verified later** (Task 9 gate): two Primaries fleet-wide, one per pair.
---
### Task 3: Re-scope `ClusterNodeAddressReconciler` to central-mesh-visible rows
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 2, Task 4 (disjoint files)
**Context:** The admin-singleton reconciler compares `Cluster.State.Members` (driver role) against
**every** `ClusterNode` row fleet-wide (`ClusterNodeAddressReconciler.cs:123-131`,
`ClusterNodeAddressReconcilerActor.cs:76-96`). After the split, central sees only its own pair, so
every site row logs `EnabledRowNotInCluster` forever (its own doc comment,
`ClusterNodeAddressReconciler.cs:60-65`, flags this). Fix: only reconcile rows whose `ClusterId`
matches central's own cluster — central can no longer observe other clusters via gossip, so it must
not assert anything about their rows.
**Files:**
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Fleet/ClusterNodeAddressReconciler.cs` (filter
the row set by own-cluster)
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Fleet/ClusterNodeAddressReconcilerActor.cs`
(pass the admin node's own `ClusterId` in; source it from `IClusterRoleInfo.ClusterId`, or from the
set of `ClusterId`s present among its own observed driver members)
- Update the class doc comment (remove the "Phase 2 must revisit" note; state it is now own-cluster
scoped).
- Test: `tests/.../ClusterNodeAddressReconcilerTests.cs` — a row in another cluster is **ignored** (no
finding); an own-cluster enabled row with no member still yields `EnabledRowNotInCluster`; drift
detection within own cluster still works.
**The pure `ClusterNodeAddressReconciler.Reconcile` function is UNCHANGED** — the fix is entirely in
the actor's two DB queries (`ClusterNodeAddressReconcilerActor.Reconcile`, lines 89-97), which filter
rows by the admin node's own `ClusterId` before projecting. `ClusterNode` has a `ClusterId` column.
**Steps:**
1. Inject the admin node's own `ClusterId` into the actor: add `IClusterRoleInfo` (or just a
`string? ownClusterId`) to `Props`/ctor, sourced from `IClusterRoleInfo.ClusterId` at registration
(`ControlPlane/ServiceCollectionExtensions.cs` where the reconciler singleton is registered).
2. In the actor's `Reconcile`, add `.Where(n => n.ClusterId == ownClusterId)` to BOTH the `rows` and
`enabled` queries — but ONLY when `ownClusterId` is non-null.
3. **Null `ownClusterId` (legacy admin node with no cluster role) ⇒ reconcile ALL rows** — this is the
correct single-mesh behavior (a legacy admin node genuinely sees every driver member via gossip).
Do NOT skip. This preserves backward-compat, mirroring the Task 2 fallback philosophy.
4. `RowDialTargetDisagrees` / `RunningNodeHasNoRow` / `EnabledRowNotInCluster` semantics are unchanged;
they now simply operate on the own-cluster row subset (post-split) or the full set (legacy).
5. Update the class doc comment in `ClusterNodeAddressReconciler.cs:60-65`: replace the "Phase 2 must
revisit" note with the delivered own-cluster-scoped behavior.
6. Tests (`ClusterNodeAddressReconcilerActorTests` or the existing reconciler tests — pick the level
that can inject `ownClusterId`): a SITE-A row is **ignored** when `ownClusterId=="MAIN"` (no
`EnabledRowNotInCluster`); an own-cluster (MAIN) enabled row with no member still yields
`EnabledRowNotInCluster`; own-cluster drift still caught; `ownClusterId==null` reconciles the full
set (legacy). If the actor is hard to unit-test directly, a focused query-filter test + the
unchanged pure-function tests suffice — say which you chose.
**Acceptance:** central reconciles only MAIN rows; site rows never produce false
`EnabledRowNotInCluster`; own-cluster drift still caught.
---
### Task 4: One `ClusterClient` per application `Cluster` in `CentralCommunicationActor`
**Classification:** high-risk
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 2, Task 3 (disjoint files)
**Context:** `RebuildClient` (`CentralCommunicationActor.cs:271-297`) creates **one fleet-wide**
`ClusterClient` dialing every `ClusterNode` row, and `HandleMeshCommand` (`:128-157`) does one
`ClusterClient.SendToAll(...)`. The `TODO(Phase 6)` (`:249-268`) states this is correct only on a
single mesh: a `ClusterClientReceptionist` serves its whole cluster, so `SendToAll` on one client
reaches everyone. After the split, `SendToAll` on one client reaches only the mesh whose receptionist
answered — so central needs **one client per cluster**, and dispatch fans `SendToAll` across all of
them.
**Files:**
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Communication/CentralCommunicationActor.cs`
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/MeshClusterClientBoundaryTests.cs`
(extend: two separate site meshes + central, a command reaches BOTH via per-cluster clients; and a
single-cluster client's `SendToAll` does NOT reach the other cluster)
**Steps:**
1. `LoadContactsFromDb` (`:177-221`): return contacts **grouped by `ClusterId`** — an
`ImmutableDictionary<string, ImmutableHashSet<string>>` (ClusterId → receptionist contact paths).
Keep the `Enabled && !MaintenanceMode` filter.
2. Replace the single `_client`/`_currentContacts` with a `Dictionary<string, (IActorRef Client,
ImmutableHashSet<string> Contacts)>` keyed by ClusterId. `HandleContactsLoaded` diffs **per cluster**
and rebuilds only the clusters whose contact set changed; stops+removes clients for clusters that
vanished.
3. `RebuildClient` becomes `RebuildClient(string clusterId, ImmutableHashSet<string> contacts)`
builds one `ClusterClient` per cluster (name it by clusterId so the actor paths are distinct).
4. `HandleMeshCommand` fans out: for each cluster client, `client.Tell(new ClusterClient.SendToAll(
MeshPaths.NodeCommunication, cmd.Message))`. Preserve the sender-relay idiom where it exists.
**A command with no live clients (all clusters down) drops with a Warning**, matching today's
"no reachable contact" behavior — do not buffer.
5. Update the `TODO(Phase 6)` doc block to describe the delivered per-cluster-client shape (remove the
TODO marker).
6. **Frame-size note:** the deploy path stays payload-free (notify-and-fetch), so no new frame-size
exposure — do not add payload to these commands.
7. Tests: extend `MeshClusterClientBoundaryTests` to two site meshes + central; assert a dispatch
reaches a node in each site mesh, and that per-cluster `SendToAll` is correctly partitioned.
**Acceptance:** central dispatches deploy-notify/driver-control/alarm-commands to every cluster via
one client each; the boundary test proves cross-mesh partitioned delivery; acks still return via the
site nodes' own `ClusterClient.Send` to central (unchanged).
---
### Task 5: Split-topology startup validator (cluster-role ⇒ non-DPS transport)
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 6 (disjoint files)
**Context (decision 2):** the DPS branches stay as dark switches, but a node configured for the split
topology (it carries a `cluster-{ClusterId}` role) must not silently run a transport still in DPS
mode, which cannot cross meshes. Fail host start instead. Marker of "split topology" = the presence of
a `cluster-` role (they exist only post-Phase-6).
**Files:**
- Create: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/SplitTopologyTransportValidator.cs` (an
`IValidateOptions<...>` or a startup validator wired via `AddValidatedOptions`/`ValidateOnStart`,
mirroring `AkkaClusterOptionsValidator`/`ConfigSourceOptionsValidator` structure)
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs` (`AddOtOpcUaCluster`
registers it)
- Test: `tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/SplitTopologyTransportValidatorTests.cs`
**Rule (only fires when the node carries a `cluster-` role):**
- `MeshTransport:Mode` must be `ClusterClient` (not `Dps`) — every node.
- If `hasDriver`: `Telemetry:Mode` must be `Grpc` (node hosts the telemetry server).
- If `hasAdmin`: `TelemetryDial:Mode` must be `Grpc` (central dials).
- `ConfigSource:Mode`: **do not** add a rule here — Phase 4's `ConfigSourceOptionsValidator` already
requires `FetchAndCache` on driver-only nodes; a fused node legitimately stays `Direct`.
- A node with **no** `cluster-` role (legacy/pre-split) is exempt (rule does not fire), so existing
single-mesh deployments and tests are unaffected.
**Steps:**
1. Read `Cluster:Roles` (same direct-read pattern the other validators use — not `IClusterRoleInfo`,
to avoid a service-resolution ordering dependency at validation time), `MeshTransport:Mode`,
`Telemetry:Mode`, `TelemetryDial:Mode`.
2. Implement the rule above; each violation adds a precise message naming the offending key + required
value.
3. Register with `ValidateOnStart` so a misconfigured node refuses to boot.
4. Tests: cluster-role node on `MeshTransport:Mode=Dps` fails; on `ClusterClient` passes; driver
cluster node on `Telemetry:Mode=Dps` fails; admin cluster node on `TelemetryDial:Mode=Dps` fails;
a non-cluster-role node on all-Dps passes (exempt).
**Acceptance:** a split-configured node on any DPS transport refuses to start with a clear message;
legacy nodes unaffected.
---
### Task 6: Compiled transport-mode defaults — DECIDED: do NOT flip (data-backed deviation)
**Classification:** small → **resolved as a no-code decision 2026-07-24**
**Parallelizable with:** Task 5
**Original intent (decision 2):** change the compiled `MeshTransport:Mode`/`Telemetry:Mode`/
`TelemetryDial:Mode` defaults to the split-correct values so a fresh deploy is split-correct without
per-node overrides.
**What the blast-radius assessment found (read-only, this session):** flipping the compiled defaults
is **HIGH-RISK and low-value**. Every one of the three transport-option validators fail-closes when its
mode is the non-Dps value but the mode's prerequisites are unset — and *nothing in the repo sets those
sections*:
- `MeshTransport:Mode=ClusterClient` requires `CentralContactPoints` — no `appsettings*.json` and no
test harness sets it → `ValidateOnStart` throws at boot.
- `Telemetry:Mode=Grpc` requires `GrpcListenPort`+`ApiKey` on a driver node — same gap.
- `TelemetryDial:Mode=Grpc` requires `ApiKey` with **no role gate at all** — breaks even roleless
fixtures and docker-dev's 4 site nodes.
- Concretely this hard-fails **all 14 `Host.IntegrationTests`** (via `TwoNodeClusterHarness`, which
sets none of these), **all 5 shipped appsettings profiles**, and several unit tests asserting the
Dps default.
**Why not flipping is correct, not a compromise:** the Task 5 `SplitTopologyTransportValidator`
already forces `ClusterClient`/`Grpc`/`Grpc` onto **exactly** the nodes carrying a post-Phase-6
`cluster-{ClusterId}` role, fail-closed, with a precise message — which *is* "a fresh Phase-6 deploy is
split-correct," achieved more robustly than a default flip (a split node cannot even boot on the wrong
transport). Flipping the compiled defaults would merely duplicate that enforcement for split nodes
while indiscriminately breaking every legacy/test/admin node that legitimately still runs Dps. The
docker-dev rig (Task 7) sets the modes **explicitly** on its cluster-role nodes, so it never relies on
the default anyway.
**Decision:** keep `MeshTransportOptions.Mode`/`TelemetryOptions.Mode`/`TelemetryDialOptions.Mode`
defaults at `Dps`. No code change. Enforcement of split-correctness lives in Task 5's validator + the
explicit Task 7 rig config. Recorded in `docs/Configuration.md` (Task 8) so the next reader knows the
defaults are intentionally Dps and the validator is the guardrail.
**Acceptance:** MET by decision — the split validator (Task 5) is the guardrail; no default flip.
---
### Task 7: Rewrite the docker-dev rig into three 2-node meshes
**Classification:** standard
**Estimated implement time:** ~5 min
**Parallelizable with:** none (needs Tasks 06 landed to actually run; the file edit is independent
but sequence it after code so the rig is testable)
**Context (decision 3):** one shared network, seed-partitioned into three meshes. Central pair =
`cluster-MAIN`; site-a pair = `cluster-SITE-A`; site-b pair = `cluster-SITE-B`. Each pair lists only
its own two nodes as seeds (self-first). Flip the transports on. Secrets stays enabled (now
pair-local). Central must remain network-reachable to site nodes (shared network — already true; do
**not** isolate networks).
**Files:**
- Modify: `docker-dev/docker-compose.yml`
- Modify: `docker-dev/seed/seed-clusters.sql` if any seeded address/port changes (AkkaPort stays 4053;
GrpcPort stays 4056 — likely no seed change, but verify the ClusterId/host columns still match).
**Per-node changes:**
| Node | `Cluster__Roles` | `Cluster__SeedNodes__0` (self) | `__1` (partner) |
|---|---|---|---|
| central-1 | `admin`,`driver`,`cluster-MAIN` | `akka.tcp://otopcua@central-1:4053` | `...central-2:4053` |
| central-2 | `admin`,`driver`,`cluster-MAIN` | `akka.tcp://otopcua@central-2:4053` | `...central-1:4053` |
| site-a-1 | `driver`,`cluster-SITE-A` | `akka.tcp://otopcua@site-a-1:4053` | `...site-a-2:4053` |
| site-a-2 | `driver`,`cluster-SITE-A` | `akka.tcp://otopcua@site-a-2:4053` | `...site-a-1:4053` |
| site-b-1 | `driver`,`cluster-SITE-B` | `akka.tcp://otopcua@site-b-1:4053` | `...site-b-2:4053` |
| site-b-2 | `driver`,`cluster-SITE-B` | `akka.tcp://otopcua@site-b-2:4053` | `...site-b-1:4053` |
**Transport flips (all nodes unless noted):**
- `MeshTransport__Mode: ClusterClient` (remove the `${OTOPCUA_MESH_MODE:-Dps}` default or set the var
default to ClusterClient).
- `Telemetry__Mode: Grpc` (all); `TelemetryDial__Mode: Grpc` (central pair only — the dialers).
- `ConfigSource__Mode: FetchAndCache` already on sites; central stays `Direct`.
- `MeshTransport__CentralContactPoints__0/1` on **site** nodes point at central-1/central-2 (unchanged
— sites learn central from appsettings). Central nodes do not need CentralContactPoints for
themselves; central discovers *site* contacts from `ClusterNode` rows (per-cluster clients, Task 4).
- **Secrets** (`x-secrets-env` anchor): keep `Secrets__Replication__Enabled=true` — it becomes
pair-local automatically. Add a compose comment noting the new pair-local semantics.
- **Remove** the `central-1`-as-seed lines from every site node (the old cross-pair seed). Update the
header comment block (lines 1-23) to describe three meshes, not one.
- **LocalDb sync:** keep site-a's replication demo; optionally add a distinct sync port for site-b to
model per-site ports (nice-to-have, not required for the gate — if added, use a different host-safe
port and its own peer address).
**Steps:**
1. Rewrite the six nodes' `Cluster__Roles__*` and `Cluster__SeedNodes__*` per the table.
2. Flip the transport-mode env vars.
3. Update the header comment + secrets comment.
4. `docker compose -f docker-dev/docker-compose.yml config` to validate YAML (no live bring-up in this
task — that is the Task 9 gate).
**Acceptance:** `docker compose config` validates; the six nodes carry per-pair self-first seeds +
cluster roles + split-correct transport modes; header comment describes three meshes.
---
### Task 8: Docs sweep — status tables, caveat removal, pair-local secrets note
**Classification:** small
**Estimated implement time:** ~5 min
**Parallelizable with:** Task 7 (docs only; no build)
**Files:**
- `docs/Redundancy.md` — new §"Per-cluster meshes (Phase 6)": one Primary per pair (**two+ Primaries
fleet-wide, by design**), redundancy-state singleton is now `cluster-{ClusterId}`-scoped and
spawned per driver node; remove any single-mesh caveat. Add §"Secrets replication is pair-local".
- `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Redundancy/IManualFailoverService.cs:34-35` — remove the
"Mesh-scope caveat / until the per-cluster mesh work lands" doc comment (it landed).
- AdminUI ClusterRedundancy page — remove the mesh-scope caveat text (grep the Razor for the caveat
string).
- `docs/Configuration.md` — document the `cluster-{ClusterId}` role, per-pair `Cluster:SeedNodes`
ordering, the new transport-mode defaults, and the split-topology validator.
- `docs/plans/2026-07-22-per-cluster-mesh-program.md` — mark Phase 6 **DONE** in the section + the
tracking table; note the three decisions.
- `docs/plans/2026-07-21-per-cluster-mesh-design.md` §7 — mark row 6 DONE.
- `CLAUDE.md` — a "Per-cluster mesh (Phase 6)" note under the Redundancy section: three meshes, cluster
roles, one Primary per pair, pair-local secrets, per-cluster ClusterClient, reconciler own-cluster
scoped, transport defaults flipped + split validator.
**Steps:**
1. Write the Redundancy.md sections; remove the two caveats (IManualFailoverService + Razor).
2. Update Configuration.md.
3. Update the two plan status tables + CLAUDE.md.
4. Grep for any remaining "single mesh" / "fleet-wide Primary" / "until the per-cluster mesh work
lands" copy and reconcile.
**Acceptance:** no stale single-mesh caveats remain; status tables show Phase 6 done; secrets
pair-local + cluster-role documented.
---
### Task 9: Live gate — three meshes, per-pair redundancy, cross-mesh transports
**Classification:** high-risk
**Estimated implement time:** manual (controller-run against docker-dev)
**Parallelizable with:** none (final)
**Context:** the exit gate. Bring the rewritten rig up and prove every previously-live-gated behavior
per pair, plus the split itself. Gather TCP/log/CLI evidence; record in
`docs/plans/2026-07-24-mesh-phase6-live-gate.md`.
**Legs:**
1. **Three meshes form.** Each node's `Cluster.State.Members` shows **only its own pair** (2 members).
Evidence: redundancy CLI / logs per node, or `/proc/net/tcp` showing 4053 associations only within
each pair. central sees central-1/2 only; site-a sees site-a-1/2 only; site-b sees site-b-1/2 only.
2. **Two+ Primaries, one per pair.** Each pair elects its own oldest-Up driver Primary; ServiceLevel
**250/240** within each pair (Primary/Secondary), independently. Client.CLI `redundancy` per
endpoint (4840/4841 MAIN, 4842/4843 SITE-A, 4844/4845 SITE-B).
3. **Deploy reaches all clusters over per-cluster ClusterClient.** `POST :9200/api/deployments`
(X-Api-Key) seals green with all six nodes acking; central's logs show one client per cluster;
verify a SITE-A node applies the config (its address space serves).
4. **Telemetry per cluster over gRPC.** AdminUI `/alerts` + `/hosts` pills live; central dials each
site mesh's telemetry servers (2 inbound per driver node on 4056; central holds outbound per cluster).
Kill-and-reconnect a site node → its stream recovers (~5s).
5. **Reconciler quiet.** central's `ClusterNodeAddressReconciler` logs **no** `EnabledRowNotInCluster`
for SITE-A/SITE-B rows (own-cluster scoped now); MAIN drift still caught (probe with a deliberate
mismatch if practical).
6. **Secrets pair-local.** With `Secrets:Replication:Enabled=true`, a secret written on site-a-1
replicates to site-a-2 (same mesh) and does **not** reach central or site-b (separate meshes) — or,
if a live secret write is impractical, assert the DPS topic is mesh-scoped via the membership
evidence from leg 1 + a doc note. No errors from the secrets actor about unreachable peers.
7. **Split validator.** Flip one site node to `MeshTransport__Mode=Dps` → it refuses to boot with the
Task 5 message. Restore.
8. **Failover within a pair** (bridges toward Phase 7): kill a site pair's Primary → the Secondary
becomes Primary (250) and the pair survives alone (auto-down 1-vs-1 — the deferred Phase 0a gate is
now testable; run it if time permits and note the result for Phase 7).
**Record:** `docs/plans/2026-07-24-mesh-phase6-live-gate.md` with per-leg evidence + any findings.
**Cleanup:** restore the rig to a coherent committed state.
**Acceptance:** all legs pass (or any deviation is documented + justified as in prior phases). Phase 6
exit gate MET.
---
## Risk register (carry from design §8 + program)
- LocalDb is load-bearing for config (unchanged; #485 class).
- **Two+ Primaries fleet-wide is now correct, by design** — anything asserting "one Primary" is stale.
- Partial rollout hazard: a node whose mesh is split has pair-local `DriverMemberCount`, while a
not-yet-split node counts fleet-wide — do not run the fleet half-split in production without the
drill (Phase 7).
- Co-located VM loss takes out one node of **both** products (Phase 7 drill item; out of scope here).
- Frame size: deploy path stays payload-free — keep it that way.
@@ -0,0 +1,24 @@
{
"planPath": "docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md",
"program": "per-cluster-mesh",
"phase": 6,
"branch": "feat/mesh-phase6",
"decisions": {
"secrets": "pair-local Akka (no SQL-hub; sites have no SQL)",
"dpsBranches": "flip rig + split-topology validator; keep DPS code. NOTE: compiled defaults NOT flipped (Task 6) — blast-radius assessment showed a flip hard-fails all 14 Host.IntegrationTests + all 5 appsettings profiles + docker-dev site nodes; the Task 5 validator already enforces split-correctness fail-closed on cluster-role nodes, which is the actual intent.",
"rigScope": "OtOpcUa-only three 2-node meshes on one shared network"
},
"tasks": [
{"id": 0, "subject": "Task 0: RoleParser accepts cluster-{ClusterId}; consolidate driver literal", "classification": "small", "status": "completed", "commit": "87ff00fd"},
{"id": 1, "subject": "Task 1: ClusterRoleInfo exposes node's own ClusterRole/ClusterId", "classification": "small", "status": "completed", "commit": "2d2f1dec", "note": "interface in Commons/Interfaces/IClusterRoleInfo.cs; ctor (ActorSystem, IOptions<AkkaClusterOptions>, ILogger)"},
{"id": 2, "subject": "Task 2: re-home RedundancyStateActor to cluster-role singleton on drivers", "classification": "high-risk", "status": "completed", "commit": "a4d3ab40", "note": "re-home a2c5d57f + review fixes a4d3ab40 (comments + driver-fallback registry test + fixed a cross-task admin-boot NRE race); 14/14 tests"},
{"id": 3, "subject": "Task 3: re-scope ClusterNodeAddressReconciler to own-cluster rows", "classification": "standard", "status": "completed", "commit": "1ce2042e", "note": "impl 2535df01 + review-fix 1ce2042e (observed ALSO scoped by cluster role via testable FilterOwnClusterDriverMembers helper; 17/17)"},
{"id": 4, "subject": "Task 4: one ClusterClient per cluster in CentralCommunicationActor", "classification": "high-risk", "status": "completed", "commit": "6531ec19", "note": "impl 6531ec19 + comment fix d98b3264; two-mesh boundary test proves partitioned delivery, red-before-green"},
{"id": 5, "subject": "Task 5: split-topology startup validator (cluster-role => non-DPS)", "classification": "standard", "status": "completed", "commit": "344b0d33", "note": "impl a886d5e6 + review-fix 344b0d33 (resolve EFFECTIVE roles incl OTOPCUA_ROLES fallback matching Program.cs, case-normalize -> guardrail can't be defeated by role-source/case mismatch); 16/16 filter, 124/124 Cluster.Tests"},
{"id": 6, "subject": "Task 6: compiled transport-mode defaults", "classification": "small", "status": "resolved-no-flip", "note": "DECIDED not to flip — blast-radius assessment: high-risk (breaks all integration tests + appsettings profiles + docker-dev site nodes), low-value (Task 5 validator already enforces split-correctness fail-closed). Documented in plan Task 6 + Configuration.md."},
{"id": 7, "subject": "Task 7: rewrite docker-dev rig into three 2-node meshes", "classification": "standard", "status": "completed", "commit": "741ab97b", "note": "3 meshes, per-pair self-first seeds, cluster roles, transports flipped (ClusterClient/Grpc), pair-local secrets note; docker compose config validates; seed SQL already correct (GrpcPort=4056)"},
{"id": 8, "subject": "Task 8: docs sweep — status tables, caveat removal, pair-local secrets note", "classification": "small", "status": "completed", "commit": "8a173bba", "note": "all 7 edits (Redundancy/Config/CLAUDE/2 plans/IManualFailoverService/ClusterRedundancy.razor); subagent died at commit step, edits reviewed+committed by controller; full-solution build clean (0 errors)"},
{"id": 9, "subject": "Task 9: live gate — three meshes, per-pair redundancy, cross-mesh transports","classification": "high-risk", "status": "in-progress", "note": "local OrbStack docker up; full-solution build verified clean; building images"}
],
"lastUpdated": "2026-07-24T00:00:00Z"
}
@@ -20,6 +20,8 @@ public sealed class ClusterRoleInfo : IClusterRoleInfo, IDisposable
private readonly ILogger<ClusterRoleInfo> _logger; private readonly ILogger<ClusterRoleInfo> _logger;
private readonly CommonsNodeId _localNode; private readonly CommonsNodeId _localNode;
private readonly HashSet<string> _localRoles; private readonly HashSet<string> _localRoles;
private readonly string? _clusterRole;
private readonly string? _clusterId;
private readonly object _lock = new(); private readonly object _lock = new();
private readonly Dictionary<string, Member?> _roleLeaders = new(StringComparer.Ordinal); private readonly Dictionary<string, Member?> _roleLeaders = new(StringComparer.Ordinal);
private readonly Dictionary<string, HashSet<Member>> _membersByRole = new(StringComparer.Ordinal); private readonly Dictionary<string, HashSet<Member>> _membersByRole = new(StringComparer.Ordinal);
@@ -39,6 +41,23 @@ public sealed class ClusterRoleInfo : IClusterRoleInfo, IDisposable
_localNode = CommonsNodeId.Parse($"{options.Value.PublicHostname}:{options.Value.Port}"); _localNode = CommonsNodeId.Parse($"{options.Value.PublicHostname}:{options.Value.Port}");
_localRoles = new HashSet<string>(options.Value.Roles, StringComparer.Ordinal); _localRoles = new HashSet<string>(options.Value.Roles, StringComparer.Ordinal);
// Derived from the node's OWN configured roles (not live Cluster.State) so this identity is
// available at host-build time, before the cluster forms. Iterate the ORIGINAL array — not
// _localRoles — because first-wins requires configuration order, which HashSet does not
// guarantee.
var clusterRoles = options.Value.Roles.Where(RoleParser.IsClusterRole).ToArray();
if (clusterRoles.Length > 0)
{
_clusterRole = clusterRoles[0];
_clusterId = RoleParser.ClusterIdFromRole(_clusterRole);
if (clusterRoles.Length > 1)
{
_logger.LogWarning(
"Node configured with {Count} cluster-scoped roles ({Roles}); using the first ({ClusterRole})",
clusterRoles.Length, string.Join(", ", clusterRoles), _clusterRole);
}
}
SeedFromCurrentState(); SeedFromCurrentState();
_subscriber = system.ActorOf(Props.Create(() => new SubscriberActor(this)), "clusterroleinfo-subscriber"); _subscriber = system.ActorOf(Props.Create(() => new SubscriberActor(this)), "clusterroleinfo-subscriber");
} }
@@ -49,6 +68,12 @@ public sealed class ClusterRoleInfo : IClusterRoleInfo, IDisposable
/// <inheritdoc /> /// <inheritdoc />
public IReadOnlySet<string> LocalRoles => _localRoles; public IReadOnlySet<string> LocalRoles => _localRoles;
/// <inheritdoc />
public string? ClusterRole => _clusterRole;
/// <inheritdoc />
public string? ClusterId => _clusterId;
/// <inheritdoc /> /// <inheritdoc />
public bool HasRole(string role) => _localRoles.Contains(role); public bool HasRole(string role) => _localRoles.Contains(role);
@@ -186,7 +211,9 @@ public sealed class ClusterRoleInfo : IClusterRoleInfo, IDisposable
Receive<ClusterEvent.IMemberEvent>(e => owner.HandleMemberEvent(e)); Receive<ClusterEvent.IMemberEvent>(e => owner.HandleMemberEvent(e));
Receive<ClusterEvent.RoleLeaderChanged>(e => owner.HandleRoleLeaderChanged(e)); Receive<ClusterEvent.RoleLeaderChanged>(e => owner.HandleRoleLeaderChanged(e));
// LeaderChanged is intentionally a no-op here: ServiceLevel lives in RedundancyStateActor // LeaderChanged is intentionally a no-op here: ServiceLevel lives in RedundancyStateActor
// (admin-role singleton) and has no consumer on this ClusterRoleInfo path by design. // (per-cluster mesh Phase 6: a cluster-{ClusterId}-scoped singleton spawned on every
// driver node, no longer the admin-role singleton it once was) and has no consumer on this
// ClusterRoleInfo path by design.
Receive<ClusterEvent.LeaderChanged>(_ => { }); Receive<ClusterEvent.LeaderChanged>(_ => { });
Receive<ClusterEvent.CurrentClusterState>(_ => { /* seeded from initial snapshot */ }); Receive<ClusterEvent.CurrentClusterState>(_ => { /* seeded from initial snapshot */ });
} }
@@ -2,11 +2,36 @@ namespace ZB.MOM.WW.OtOpcUa.Cluster;
public static class RoleParser public static class RoleParser
{ {
/// <summary>The admin-role name.</summary>
public const string Admin = "admin";
/// <summary>The driver-role name. Single source of truth — downstream <c>DriverRole</c>
/// constants (<c>RedundancyStateActor</c>, <c>ClusterNodeAddressReconcilerActor</c>,
/// <c>ServiceCollectionExtensions</c>) alias this value rather than duplicating the literal.</summary>
public const string Driver = "driver";
/// <summary>The dev-role name.</summary>
public const string Dev = "dev";
/// <summary>Prefix identifying a cluster-scoped role, e.g. <c>cluster-SITE-A</c> (per-cluster mesh, Phase 6).</summary>
public const string ClusterRolePrefix = "cluster-";
private static readonly HashSet<string> Allowed = new(StringComparer.Ordinal) private static readonly HashSet<string> Allowed = new(StringComparer.Ordinal)
{ {
"admin", "driver", "dev", Admin, Driver, Dev,
}; };
/// <summary>Returns true when <paramref name="role"/> is a cluster-scoped role — starts with
/// <see cref="ClusterRolePrefix"/> and carries a non-empty ClusterId suffix.</summary>
/// <param name="role">The role name to test.</param>
public static bool IsClusterRole(string role) =>
role.StartsWith(ClusterRolePrefix, StringComparison.Ordinal) && role.Length > ClusterRolePrefix.Length;
/// <summary>Extracts the ClusterId from a cluster-scoped role, e.g. <c>cluster-SITE-A</c> → <c>SITE-A</c>.
/// Assumes <see cref="IsClusterRole"/> is true for <paramref name="role"/>.</summary>
/// <param name="role">The cluster-scoped role name.</param>
public static string ClusterIdFromRole(string role) => role[ClusterRolePrefix.Length..];
/// <summary>Parses a comma-separated string of role names into a validated array.</summary> /// <summary>Parses a comma-separated string of role names into a validated array.</summary>
/// <param name="raw">The raw role string to parse.</param> /// <param name="raw">The raw role string to parse.</param>
/// <returns>The distinct, lower-cased, validated role names.</returns> /// <returns>The distinct, lower-cased, validated role names.</returns>
@@ -22,9 +47,10 @@ public static class RoleParser
foreach (var r in roles) foreach (var r in roles)
{ {
if (!Allowed.Contains(r)) if (!Allowed.Contains(r) && !IsClusterRole(r))
throw new ArgumentException( throw new ArgumentException(
$"Unknown role '{r}'. Allowed: {string.Join(", ", Allowed)}.", nameof(raw)); $"Unknown role '{r}'. Allowed: {string.Join(", ", Allowed)}, or '{ClusterRolePrefix}<ClusterId>'.",
nameof(raw));
} }
return roles; return roles;
@@ -56,6 +56,14 @@ public static class ServiceCollectionExtensions
services.AddValidatedOptions<TelemetryDialOptions, TelemetryDialOptionsValidator>( services.AddValidatedOptions<TelemetryDialOptions, TelemetryDialOptionsValidator>(
configuration, TelemetryDialOptions.SectionName); configuration, TelemetryDialOptions.SectionName);
// Per-cluster mesh Phase 6: a node carrying a cluster-{ClusterId} role (the split topology)
// must use the mesh-crossing transports — the DPS dark-switch branches cannot cross a split
// mesh and would deliver nothing silently. Hung on MeshTransportOptions (which already has
// ValidateOnStart from its own registration above) so it fires at boot alongside the sibling
// validators; TryAddEnumerable inside AddValidatedOptions keeps the second validator additive.
services.AddValidatedOptions<MeshTransportOptions, SplitTopologyTransportValidator>(
configuration, MeshTransportOptions.SectionName);
services.AddSingleton<IClusterRoleInfo, ClusterRoleInfo>(); services.AddSingleton<IClusterRoleInfo, ClusterRoleInfo>();
return services; return services;
@@ -0,0 +1,137 @@
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.Options;
namespace ZB.MOM.WW.OtOpcUa.Cluster;
/// <summary>
/// Fails the host at startup when a node configured for the <b>split topology</b> (per-cluster
/// mesh, Phase 6) is left on a DistributedPubSub transport mode that cannot cross the mesh
/// boundary.
/// </summary>
/// <remarks>
/// <para>
/// Phase 6 splits the single fleet mesh into one 2-node mesh per application cluster. The
/// Phase 2/3/5 dark switches keep their DPS branches alive, but DPS is mesh-wide — it delivers
/// nothing between a central node and a site node that no longer share a gossip ring. The
/// marker of the split topology is a <c>cluster-{ClusterId}</c> role
/// (<see cref="RoleParser.IsClusterRole"/>); those roles exist only post-Phase-6. A node that
/// carries one <b>must</b> use the mesh-crossing transports, and this validator refuses to
/// boot it otherwise — the alternative is a node that runs, listens, and silently delivers no
/// commands or telemetry, which has no stack trace and no failing node to point at.
/// </para>
/// <para>
/// <b>The rule is conditional and must stay that way.</b> It fires only when this node's
/// <i>effective</i> Akka roles contain at least one cluster-scoped role — resolved the same
/// way the node resolves them (<c>Cluster:Roles</c>, falling back to <c>OTOPCUA_ROLES</c> when
/// that is empty, normalized case-insensitively). A legacy / single-mesh / test node has none,
/// so DPS is still legitimate and the validator passes unconditionally.
/// </para>
/// <para>
/// <b>Not validated here:</b> <c>ConfigSource:Mode</c>. Phase 4's
/// <see cref="ConfigSourceOptionsValidator"/> already requires <c>FetchAndCache</c> on a
/// driver-only node, and a fused node legitimately stays <c>Direct</c> — duplicating that here
/// would only risk the two rules drifting apart.
/// </para>
/// </remarks>
public sealed class SplitTopologyTransportValidator : IValidateOptions<MeshTransportOptions>
{
private readonly IConfiguration _configuration;
/// <summary>
/// DI-constructed by <c>AddValidatedOptions</c> (a plain <c>AddSingleton</c>), so a
/// constructor dependency is safe here. <see cref="IConfiguration"/> — not
/// <c>IClusterRoleInfo</c> — is the source of this node's roles and the three transport modes:
/// <c>IClusterRoleInfo</c>'s implementation needs the <c>ActorSystem</c>, which does not exist
/// yet at <c>ValidateOnStart</c> time. Reading the modes from configuration too (rather than
/// from three injected <c>IOptions</c>) mirrors how the sibling validators cross-read
/// <c>Cluster:Roles</c>, and avoids a service-resolution ordering dependency at validation
/// time.
/// </summary>
public SplitTopologyTransportValidator(IConfiguration configuration)
{
_configuration = configuration;
}
/// <inheritdoc />
public ValidateOptionsResult Validate(string? name, MeshTransportOptions options)
{
// The options instance is intentionally unused: every value this rule depends on is cross-read
// from IConfiguration, the same way ConfigSourceOptionsValidator reads Cluster:Roles.
//
// Resolve the EFFECTIVE Akka roles exactly as the node does, or this "fail loud" rule fires on
// the wrong role view and passes silently on the two shapes it exists to catch:
// (a) Roles-source divergence. AkkaClusterOptions.Roles binds Cluster:Roles but falls back to
// OTOPCUA_ROLES (via RoleParser.Parse — the same call Program.cs makes) when Cluster:Roles
// is empty. A node configured with only OTOPCUA_ROLES=driver,cluster-SITE-A genuinely
// carries the cluster role in Akka, so it must be validated too.
// (b) Case. The Cluster:Roles bind path is verbatim (no lowercasing), whereas RoleParser.Parse
// lowercases the OTOPCUA_ROLES path. Normalize BOTH here (trim + ToLowerInvariant) so
// 'Cluster-SITE-A' / 'Driver' / 'Admin' cannot slip past the ordinal IsClusterRole /
// driver / admin checks.
var configured = _configuration.GetSection("Cluster:Roles").Get<string[]>() ?? Array.Empty<string>();
var effective = configured.Length > 0
? configured
: RoleParser.Parse(Environment.GetEnvironmentVariable("OTOPCUA_ROLES"));
var normalized = Array.ConvertAll(effective, r => r.Trim().ToLowerInvariant());
var clusterRoleIndex = Array.FindIndex(normalized, RoleParser.IsClusterRole);
if (clusterRoleIndex < 0)
{
// No cluster-{ClusterId} role → not the split topology → DPS is legitimate. This is the
// exemption that keeps the validator a no-op for every legacy / single-mesh / test node.
return ValidateOptionsResult.Success;
}
// Classification is normalized (above); the ClusterId in the message is taken from the
// operator's original spelling so it names what they actually wrote.
var clusterId = RoleParser.ClusterIdFromRole(effective[clusterRoleIndex].Trim());
var isDriver = Array.IndexOf(normalized, RoleParser.Driver) >= 0;
var isAdmin = Array.IndexOf(normalized, RoleParser.Admin) >= 0;
var errors = new List<string>();
// Modes are read from configuration directly; an unset key inherits the class default (Dps for
// all three), so a split node that simply never set the key fails exactly as an explicit Dps
// would — which is the point.
var meshMode = _configuration["MeshTransport:Mode"] ?? MeshTransportOptions.ModeDps;
var telemetryMode = _configuration["Telemetry:Mode"] ?? TelemetryOptions.ModeDps;
var telemetryDialMode = _configuration["TelemetryDial:Mode"] ?? TelemetryDialOptions.ModeDps;
if (!string.Equals(meshMode, MeshTransportOptions.ModeClusterClient, StringComparison.OrdinalIgnoreCase))
{
errors.Add(
$"Cluster:Roles declares a cluster-{clusterId} role (split topology) but "
+ $"{MeshTransportOptions.SectionName}:{nameof(MeshTransportOptions.Mode)} is "
+ $"'{meshMode}'; a split mesh cannot use DistributedPubSub — set "
+ $"{MeshTransportOptions.SectionName}:{nameof(MeshTransportOptions.Mode)}="
+ $"{MeshTransportOptions.ModeClusterClient}.");
}
if (isDriver
&& !string.Equals(telemetryMode, TelemetryOptions.ModeGrpc, StringComparison.OrdinalIgnoreCase))
{
errors.Add(
$"Cluster:Roles declares a cluster-{clusterId} role (split topology) with 'driver' but "
+ $"{TelemetryOptions.SectionName}:{nameof(TelemetryOptions.Mode)} is "
+ $"'{telemetryMode}'; a split-topology driver node hosts the telemetry gRPC server — "
+ $"set {TelemetryOptions.SectionName}:{nameof(TelemetryOptions.Mode)}="
+ $"{TelemetryOptions.ModeGrpc}.");
}
if (isAdmin
&& !string.Equals(telemetryDialMode, TelemetryDialOptions.ModeGrpc, StringComparison.OrdinalIgnoreCase))
{
errors.Add(
$"Cluster:Roles declares a cluster-{clusterId} role (split topology) with 'admin' but "
+ $"{TelemetryDialOptions.SectionName}:{nameof(TelemetryDialOptions.Mode)} is "
+ $"'{telemetryDialMode}'; a split-topology central node dials nodes for telemetry — set "
+ $"{TelemetryDialOptions.SectionName}:{nameof(TelemetryDialOptions.Mode)}="
+ $"{TelemetryDialOptions.ModeGrpc}.");
}
return errors.Count == 0
? ValidateOptionsResult.Success
: ValidateOptionsResult.Fail(string.Join(" ", errors));
}
}
@@ -14,6 +14,21 @@ public interface IClusterRoleInfo
NodeId LocalNode { get; } NodeId LocalNode { get; }
/// <summary>Gets the set of roles assigned to the local node.</summary> /// <summary>Gets the set of roles assigned to the local node.</summary>
IReadOnlySet<string> LocalRoles { get; } IReadOnlySet<string> LocalRoles { get; }
/// <summary>
/// Gets the local node's cluster-scoped role (e.g. <c>cluster-SITE-A</c>), or <c>null</c> when
/// the node carries none. Derived from the node's OWN configured roles at construction time —
/// NOT from live cluster membership — so it is available before the cluster forms (per-cluster
/// mesh, Phase 6). When more than one cluster-scoped role is configured, the first one wins.
/// </summary>
string? ClusterRole { get; }
/// <summary>
/// Gets the ClusterId extracted from <see cref="ClusterRole"/> (e.g. <c>SITE-A</c>), or
/// <c>null</c> when the node carries no cluster-scoped role.
/// </summary>
string? ClusterId { get; }
/// <summary>Checks if the local node has the specified role.</summary> /// <summary>Checks if the local node has the specified role.</summary>
/// <param name="role">Role name to check.</param> /// <param name="role">Role name to check.</param>
/// <returns>True if the local node has the role; otherwise, false.</returns> /// <returns>True if the local node has the role; otherwise, false.</returns>
@@ -68,10 +68,10 @@ else
</div> </div>
<p class="text-muted small mt-2 mb-0"> <p class="text-muted small mt-2 mb-0">
Read live from cluster state on this node. <strong>Mesh-wide scope:</strong> the Primary is Read live from cluster state on this node. The Primary shown is
elected once per Akka mesh, not per cluster row — in the current single-mesh topology this <strong>this application <span class="mono">Cluster</span>'s own Primary</strong> — since
acts on the whole mesh's Primary, which may be a node of another the per-cluster mesh split, each cluster's redundant pair runs its own independent 2-node
<span class="mono">Cluster</span>. See <span class="mono">docs/Redundancy.md</span>. Akka mesh and elects its own Primary. See <span class="mono">docs/Redundancy.md</span>.
</p> </p>
<AuthorizeView Policy="@ManualFailoverPageModel.RequiredPolicy"> <AuthorizeView Policy="@ManualFailoverPageModel.RequiredPolicy">
@@ -25,8 +25,11 @@ namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
/// rotation would fail silently against the other. /// rotation would fail silently against the other.
/// </para> /// </para>
/// <para> /// <para>
/// <b>Exactly one ClusterClient, fleet-wide.</b> See <see cref="RebuildClient"/> — this is /// <b>One ClusterClient per application <c>Cluster</c> (Phase 6).</b> The fleet is now one
/// the single most important thing to understand before changing this actor. /// Akka mesh per cluster, so a single fleet-wide client's <c>SendToAll</c> would reach only
/// the one mesh whose receptionist answered. This actor therefore holds a client per cluster
/// and fans <c>SendToAll</c> across all of them. See <see cref="RebuildClient"/> — this is the
/// single most important thing to understand before changing this actor.
/// </para> /// </para>
/// </remarks> /// </remarks>
public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
@@ -40,8 +43,12 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
private readonly ILoggingAdapter _log = Context.GetLogger(); private readonly ILoggingAdapter _log = Context.GetLogger();
private readonly bool _useClusterClient; private readonly bool _useClusterClient;
private IActorRef? _client; // One client per application Cluster, keyed by ClusterId. A ClusterClient reaches only the mesh
private ImmutableHashSet<string> _currentContacts = ImmutableHashSet<string>.Empty; // whose receptionist it dialled, and after Phase 6 every cluster is its own mesh — so central must
// hold a client per cluster and fan SendToAll across all of them.
private readonly Dictionary<string, (IActorRef Client, ImmutableHashSet<string> Contacts)> _clients =
new();
private CancellationTokenSource _lifecycle = new(); private CancellationTokenSource _lifecycle = new();
/// <summary>Gets the timer scheduler driving the periodic contact refresh.</summary> /// <summary>Gets the timer scheduler driving the periodic contact refresh.</summary>
@@ -121,6 +128,9 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
/// <inheritdoc /> /// <inheritdoc />
protected override void PostStop() protected override void PostStop()
{ {
foreach (var (client, _) in _clients.Values) Context.Stop(client);
_clients.Clear();
_lifecycle.Cancel(); _lifecycle.Cancel();
_lifecycle.Dispose(); _lifecycle.Dispose();
} }
@@ -133,7 +143,7 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
return; return;
} }
if (_client is null) if (_clients.Count == 0)
{ {
// App-level drop, matching the sister project's decision and this repo's buffer-size = 0. // App-level drop, matching the sister project's decision and this repo's buffer-size = 0.
// The command is NOT queued and will NOT be retried: a deploy fails at the coordinator's // The command is NOT queued and will NOT be retried: a deploy fails at the coordinator's
@@ -141,19 +151,21 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
// command arriving minutes late against a config the DB already recorded as failed. // command arriving minutes late against a config the DB already recorded as failed.
_log.Warning( _log.Warning(
"No mesh ClusterClient — dropping {MessageType}. The last contact refresh produced " "No mesh ClusterClient — dropping {MessageType}. The last contact refresh produced "
+ "no usable contact point from the enabled, non-maintenance ClusterNode rows; the " + "no usable contact point from the enabled, non-maintenance ClusterNode rows in any "
+ "command is not buffered and will not be retried", + "cluster; the command is not buffered and will not be retried",
cmd.Message.GetType().Name); cmd.Message.GetType().Name);
return; return;
} }
// SendToAll, NEVER Send. Today's DPS publish reaches EVERY DriverHostActor — there is no // SendToAll, NEVER Send, fanned across EVERY cluster's client. Today's DPS publish reaches
// ClusterId or node filter on the node side at all; scoping happens later, inside the // EVERY DriverHostActor — there is no ClusterId or node filter on the node side at all; scoping
// artifact (DeploymentArtifact.ResolveClusterScope). ClusterClient.Send delivers to exactly // happens later, inside the artifact (DeploymentArtifact.ResolveClusterScope). Each ClusterClient
// ONE registered actor, which would deploy to a single node of the fleet while every other // reaches only the one mesh it dialled, so central must fan across all of them to preserve the
// node silently kept its old configuration — and the deployment would still seal green if // fleet-wide reach DPS gives. ClusterClient.Send delivers to exactly ONE registered actor, which
// the ack set happened to be satisfied. // would deploy to a single node while every other node silently kept its old configuration — and
_client.Tell(new ClusterClient.SendToAll(MeshPaths.NodeCommunication, cmd.Message)); // the deployment would still seal green if the ack set happened to be satisfied.
foreach (var (client, _) in _clients.Values)
client.Tell(new ClusterClient.SendToAll(MeshPaths.NodeCommunication, cmd.Message));
} }
private void HandleApplyAck(ApplyAck ack) private void HandleApplyAck(ApplyAck ack)
@@ -198,11 +210,14 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
var rows = await db.ClusterNodes var rows = await db.ClusterNodes
.AsNoTracking() .AsNoTracking()
.Where(n => n.Enabled && !n.MaintenanceMode) .Where(n => n.Enabled && !n.MaintenanceMode)
.Select(n => new { n.NodeId, n.Host, n.AkkaPort }) .Select(n => new { n.NodeId, n.ClusterId, n.Host, n.AkkaPort })
.ToListAsync(ct) .ToListAsync(ct)
.ConfigureAwait(false); .ConfigureAwait(false);
var contacts = new List<string>(rows.Count); // Grouped BY ClusterId: central now dials one ClusterClient per application Cluster, so the
// contact set must be partitioned the same way. A row's contacts join only its own cluster's
// client.
var byCluster = new Dictionary<string, List<string>>();
var malformed = new List<string>(); var malformed = new List<string>();
foreach (var row in rows) foreach (var row in rows)
{ {
@@ -212,11 +227,21 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
// not abort the whole refresh and leave the contact set half-built — the sister // not abort the whole refresh and leave the contact set half-built — the sister
// project shipped that bug and its regression test deliberately orders the bad row // project shipped that bug and its regression test deliberately orders the bad row
// first. // first.
if (ActorPath.TryParse(address, out _)) contacts.Add(address); if (ActorPath.TryParse(address, out _))
else malformed.Add($"{row.NodeId} -> {address}"); {
if (!byCluster.TryGetValue(row.ClusterId, out var list))
byCluster[row.ClusterId] = list = new List<string>();
list.Add(address);
}
else
{
malformed.Add($"{row.NodeId} ({row.ClusterId}) -> {address}");
}
} }
return new ContactsLoaded(contacts, malformed); var contactsByCluster = byCluster.ToDictionary(
kv => kv.Key, kv => (IReadOnlyList<string>)kv.Value);
return new ContactsLoaded(contactsByCluster, malformed);
}, ct).PipeTo(self); }, ct).PipeTo(self);
} }
@@ -229,70 +254,111 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
+ "refresh (other nodes are unaffected). Check the row's Host and AkkaPort", bad); + "refresh (other nodes are unaffected). Check the row's Host and AkkaPort", bad);
} }
var contacts = msg.Contacts.ToImmutableHashSet(); var byCluster = msg.ContactsByCluster.ToDictionary(
kv => kv.Key, kv => kv.Value.ToImmutableHashSet());
// Clusters that dropped out of the DB entirely, or whose only rows were malformed this refresh:
// stop and forget their clients so a vanished cluster stops receiving. LoadContactsFromDb only
// creates a ByCluster key when at least one of its rows parsed, so an all-malformed cluster has
// NO key at all and is caught here by the absent-key (!TryGetValue) arm — not by an empty value.
var gone = _clients.Keys
.Where(id => !byCluster.TryGetValue(id, out var c) || c.IsEmpty)
.ToList();
foreach (var clusterId in gone)
{
_log.Info(
"Cluster {ClusterId} no longer yields any usable contact point; stopping its "
+ "ClusterClient", clusterId);
Context.Stop(_clients[clusterId].Client);
_clients.Remove(clusterId);
}
var anyUsable = false;
foreach (var (clusterId, contacts) in byCluster)
{
if (contacts.IsEmpty) if (contacts.IsEmpty)
{ {
// Defensive: LoadContactsFromDb never emits an empty-but-present cluster (a key is
// created only on a first successfully-parsed row), so this arm is unreachable for the
// DB producer — an all-malformed cluster is handled by the absent-key branch above. It
// guards a ContactsLoaded (a public record) constructed elsewhere with an empty list.
_log.Warning( _log.Warning(
"No usable ClusterNode contact points; the mesh ClusterClient was not created and " "Cluster {ClusterId} has no usable ClusterNode contact points; no ClusterClient "
+ "every central→node command will be dropped until a refresh finds one"); + "was created for it and its nodes' commands will be dropped until a refresh finds "
return; + "one", clusterId);
continue;
} }
if (_client is not null && _currentContacts.SetEquals(contacts)) return; anyUsable = true;
RebuildClient(contacts); if (_clients.TryGetValue(clusterId, out var existing) && existing.Contacts.SetEquals(contacts))
continue;
RebuildClient(clusterId, contacts);
} }
/// <summary>Stops any existing client and creates a replacement for the new contact set.</summary> if (!anyUsable && _clients.Count == 0)
{
_log.Warning(
"No usable ClusterNode contact points in any cluster; no mesh ClusterClient was "
+ "created and every central→node command will be dropped until a refresh finds one");
}
}
/// <summary>Stops a cluster's existing client and creates a replacement for its new contact set.</summary>
/// <remarks> /// <remarks>
/// <para> /// <para>
/// <b>TODO(Phase 6): one client per application <c>Cluster</c>.</b> Today this actor /// <b>One client per application <c>Cluster</c> (delivered in Phase 6).</b> This actor now
/// creates exactly <b>one</b> client whose contacts are the whole fleet, and that is not /// holds one <c>ClusterClient</c> per cluster and <see cref="HandleMeshCommand"/> fans
/// a compromise — it is the only correct shape while the fleet is a single Akka mesh. /// <c>SendToAll</c> across all of them. Once the fleet is split into one Akka mesh per
/// cluster, a <c>ClusterClientReceptionist</c> serves only <i>its own</i> mesh, so a single
/// client's <c>SendToAll</c> would reach the one mesh whose receptionist answered and leave
/// every other cluster silently on its old configuration. Fanning across a client per
/// cluster restores the fleet-wide reach the pre-split DPS broadcast gave.
/// </para> /// </para>
/// <para> /// <para>
/// A <c>ClusterClientReceptionist</c> serves its entire cluster, so /// Why this is not N× duplicate dispatch: each client dials only its own cluster's
/// <c>SendToAll</c> reaches every registered node-comm actor in the mesh <i>regardless of /// receptionists, so a command is delivered <b>once per mesh</b>, not once per cluster into
/// which node's address was used as the contact point</i>. One client per Cluster would /// every mesh. (While the fleet was a single mesh, the same code would have duplicated —
/// therefore fan each command out once per cluster — N× duplicate /// which is exactly why the fan-out could not land until the meshes were partitioned.)
/// <c>DispatchDeployment</c> and N× duplicate <c>ApplyAck</c>. Phase 6 splits the meshes,
/// at which point per-cluster clients become both correct and necessary.
/// </para> /// </para>
/// <para> /// <para>
/// Corollary worth stating because it is easy to assume otherwise: <b>the contact set /// Corollary worth stating because it is easy to assume otherwise: <b>the contact set
/// does not scope delivery.</b> Excluding a maintenance-mode node from the contacts does /// does not scope delivery within a mesh.</b> Excluding a maintenance-mode node from a
/// not stop it receiving commands — it still receives them, exactly as it does under /// cluster's contacts does not stop it receiving commands — its receptionist still delivers
/// today's DPS broadcast. The filter is about which receptionists are worth <i>dialling</i> /// to every registered node-comm actor in that mesh. The filter is about which receptionists
/// (a node in maintenance may be switched off), not about who gets the message. /// are worth <i>dialling</i> (a node in maintenance may be switched off), not about who gets
/// the message.
/// </para> /// </para>
/// </remarks> /// </remarks>
/// <param name="contacts">The receptionist addresses to dial.</param> /// <param name="clusterId">The application <c>Cluster</c> whose client is being (re)built.</param>
private void RebuildClient(ImmutableHashSet<string> contacts) /// <param name="contacts">The receptionist addresses to dial for that cluster.</param>
private void RebuildClient(string clusterId, ImmutableHashSet<string> contacts)
{ {
if (_client is not null) if (_clients.TryGetValue(clusterId, out var existing))
{ {
_log.Info("Mesh contact set changed; replacing the ClusterClient"); _log.Info("Contact set for cluster {ClusterId} changed; replacing its ClusterClient", clusterId);
Context.Stop(_client); Context.Stop(existing.Client);
// Cleared now: if the create below throws, a stale ref would route commands into a // Removed now: if the create below throws, a stale ref would route commands into a
// stopping actor and they would vanish without a warning. // stopping actor and they would vanish without a warning.
_client = null; _clients.Remove(clusterId);
_currentContacts = ImmutableHashSet<string>.Empty;
} }
try try
{ {
var paths = contacts.Select(ActorPath.Parse).ToImmutableHashSet(); var paths = contacts.Select(ActorPath.Parse).ToImmutableHashSet();
_client = _clientFactory.Create(Context.System, paths); var client = _clientFactory.Create(Context.System, clusterId, paths);
_currentContacts = contacts; _clients[clusterId] = (client, contacts);
_log.Info("Mesh ClusterClient created with {Count} contact point(s)", contacts.Count); _log.Info(
"Mesh ClusterClient for cluster {ClusterId} created with {Count} contact point(s)",
clusterId, contacts.Count);
} }
catch (Exception ex) catch (Exception ex)
{ {
_log.Error(ex, _log.Error(ex,
"Failed to create the mesh ClusterClient; central→node commands are dropped until " "Failed to create the mesh ClusterClient for cluster {ClusterId}; its nodes' commands "
+ "the next contact refresh succeeds"); + "are dropped until the next contact refresh succeeds", clusterId);
} }
} }
@@ -300,9 +366,12 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
public sealed record RefreshContacts; public sealed record RefreshContacts;
/// <summary>Result of a contact-point load, piped back from the DB read.</summary> /// <summary>Result of a contact-point load, piped back from the DB read.</summary>
/// <param name="Contacts">Receptionist addresses that parsed cleanly.</param> /// <param name="ContactsByCluster">
/// <param name="Malformed">Rows that did not, described for the log.</param> /// Receptionist addresses that parsed cleanly, grouped by the owning application
/// <c>Cluster</c>'s id. One <c>ClusterClient</c> is built per key.
/// </param>
/// <param name="Malformed">Rows that did not parse, described for the log.</param>
public sealed record ContactsLoaded( public sealed record ContactsLoaded(
IReadOnlyList<string> Contacts, IReadOnlyDictionary<string, IReadOnlyList<string>> ContactsByCluster,
IReadOnlyList<string> Malformed); IReadOnlyList<string> Malformed);
} }
@@ -13,9 +13,14 @@ public interface IMeshClusterClientFactory
{ {
/// <summary>Creates a client for the given receptionist contact points.</summary> /// <summary>Creates a client for the given receptionist contact points.</summary>
/// <param name="system">The actor system to spawn the client in.</param> /// <param name="system">The actor system to spawn the client in.</param>
/// <param name="clusterId">
/// The application <c>Cluster</c> these contacts belong to. Central now creates one client per
/// cluster (Phase 6), so the id is woven into the client's actor name to keep the per-cluster
/// clients distinct and diagnosable.
/// </param>
/// <param name="contacts">Receptionist actor paths, one per reachable node.</param> /// <param name="contacts">Receptionist actor paths, one per reachable node.</param>
/// <returns>The new client's <see cref="IActorRef"/>.</returns> /// <returns>The new client's <see cref="IActorRef"/>.</returns>
IActorRef Create(ActorSystem system, ImmutableHashSet<ActorPath> contacts); IActorRef Create(ActorSystem system, string clusterId, ImmutableHashSet<ActorPath> contacts);
} }
/// <inheritdoc /> /// <inheritdoc />
@@ -35,13 +40,28 @@ public sealed class DefaultMeshClusterClientFactory : IMeshClusterClientFactory
private long _generation; private long _generation;
/// <inheritdoc /> /// <inheritdoc />
public IActorRef Create(ActorSystem system, ImmutableHashSet<ActorPath> contacts) public IActorRef Create(ActorSystem system, string clusterId, ImmutableHashSet<ActorPath> contacts)
{ {
ArgumentNullException.ThrowIfNull(system); ArgumentNullException.ThrowIfNull(system);
ArgumentException.ThrowIfNullOrEmpty(clusterId);
ArgumentNullException.ThrowIfNull(contacts); ArgumentNullException.ThrowIfNull(contacts);
var settings = ClusterClientSettings.Create(system).WithInitialContacts(contacts); var settings = ClusterClientSettings.Create(system).WithInitialContacts(contacts);
var name = $"mesh-client-{Interlocked.Increment(ref _generation)}"; var name = $"mesh-client-{SanitizeForActorName(clusterId)}-{Interlocked.Increment(ref _generation)}";
return system.ActorOf(ClusterClient.Props(settings), name); return system.ActorOf(ClusterClient.Props(settings), name);
} }
/// <summary>
/// Reduces a cluster id to characters safe in an Akka actor name. The id is operator-authored
/// (e.g. a GUID or a slug), so it may carry characters Akka rejects; anything outside
/// <c>[A-Za-z0-9-]</c> becomes <c>-</c>. The trailing generation counter still guarantees
/// uniqueness, so a lossy sanitisation cannot cause a name collision.
/// </summary>
/// <param name="clusterId">The raw cluster id.</param>
/// <returns>An actor-name-safe rendering of the id.</returns>
private static string SanitizeForActorName(string clusterId)
{
var chars = clusterId.Select(c => char.IsAsciiLetterOrDigit(c) || c == '-' ? c : '-');
return new string(chars.ToArray());
}
} }
@@ -58,10 +58,23 @@ public sealed record ClusterNodeAddress(string NodeId, string Host, int AkkaPort
/// to be deleted again. /// to be deleted again.
/// </para> /// </para>
/// <para> /// <para>
/// <b>Phase 2 must revisit this.</b> Once the fleet splits into one mesh per cluster, an /// <b>Scoped to the admin node's own cluster (Phase 6).</b> Once the fleet splits into one
/// admin node no longer sees site members at all, and every site row would report /// mesh per cluster, an admin node no longer sees site members at all, and a fleet-wide sweep
/// <see cref="AddressMismatchKind.EnabledRowNotInCluster"/> forever. The check then has to /// would report every other cluster's row as
/// move to whatever transport replaces gossip — or be scoped to the central mesh. /// <see cref="AddressMismatchKind.EnabledRowNotInCluster"/> forever. The actor therefore scopes
/// <b>both</b> inputs to this function to the admin node's own cluster: the <c>rows</c>/enabled
/// set to rows whose <c>ClusterId</c> matches
/// <see cref="ZB.MOM.WW.OtOpcUa.Commons.Interfaces.IClusterRoleInfo.ClusterId"/>, and the
/// observed membership to members carrying its own
/// <see cref="ZB.MOM.WW.OtOpcUa.Commons.Interfaces.IClusterRoleInfo.ClusterRole"/>
/// (<c>FilterOwnClusterDriverMembers</c>). Scoping only the rows would swap the
/// <see cref="AddressMismatchKind.EnabledRowNotInCluster"/> false positive for a worse
/// Error-level <see cref="AddressMismatchKind.RunningNodeHasNoRow"/> during the mixed-mesh
/// window, where the DB scope is live but central still sees foreign members via gossip. A
/// legacy admin node carrying no cluster role (null <c>ClusterId</c>/<c>ClusterRole</c>) still
/// sees every member via gossip and reconciles the full fleet unchanged. The pure function
/// below is untouched — scoping happens entirely in the actor's DB queries and its
/// membership filter.
/// </para> /// </para>
/// </remarks> /// </remarks>
public static class ClusterNodeAddressReconciler public static class ClusterNodeAddressReconciler
@@ -2,6 +2,7 @@ using Akka.Actor;
using Akka.Cluster; using Akka.Cluster;
using Akka.Event; using Akka.Event;
using Microsoft.EntityFrameworkCore; using Microsoft.EntityFrameworkCore;
using ZB.MOM.WW.OtOpcUa.Cluster;
using ZB.MOM.WW.OtOpcUa.Configuration; using ZB.MOM.WW.OtOpcUa.Configuration;
namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Fleet; namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Fleet;
@@ -20,7 +21,7 @@ namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Fleet;
public sealed class ClusterNodeAddressReconcilerActor : ReceiveActor, IWithTimers public sealed class ClusterNodeAddressReconcilerActor : ReceiveActor, IWithTimers
{ {
/// <summary>Driver role name — only members carrying it are expected to have a ClusterNode row.</summary> /// <summary>Driver role name — only members carrying it are expected to have a ClusterNode row.</summary>
public const string DriverRole = "driver"; public const string DriverRole = RoleParser.Driver;
/// <summary>Collapses a burst of membership events (a rolling restart) into one reconcile.</summary> /// <summary>Collapses a burst of membership events (a rolling restart) into one reconcile.</summary>
public static readonly TimeSpan DebounceWindow = TimeSpan.FromSeconds(2); public static readonly TimeSpan DebounceWindow = TimeSpan.FromSeconds(2);
@@ -29,6 +30,8 @@ public sealed class ClusterNodeAddressReconcilerActor : ReceiveActor, IWithTimer
public static readonly TimeSpan DefaultSweepInterval = TimeSpan.FromMinutes(5); public static readonly TimeSpan DefaultSweepInterval = TimeSpan.FromMinutes(5);
private readonly IDbContextFactory<OtOpcUaConfigDbContext> _dbFactory; private readonly IDbContextFactory<OtOpcUaConfigDbContext> _dbFactory;
private readonly string? _ownClusterId;
private readonly string? _ownClusterRole;
private readonly Akka.Cluster.Cluster _cluster; private readonly Akka.Cluster.Cluster _cluster;
private readonly TimeSpan _sweepInterval; private readonly TimeSpan _sweepInterval;
private readonly ILoggingAdapter _log = Context.GetLogger(); private readonly ILoggingAdapter _log = Context.GetLogger();
@@ -39,19 +42,49 @@ public sealed class ClusterNodeAddressReconcilerActor : ReceiveActor, IWithTimer
/// <summary>Creates Props for the reconciler.</summary> /// <summary>Creates Props for the reconciler.</summary>
/// <param name="dbFactory">Factory for the config database context.</param> /// <param name="dbFactory">Factory for the config database context.</param>
/// <param name="ownClusterId">
/// The admin node's own <c>ClusterId</c> (e.g. <c>MAIN</c>), from
/// <see cref="IClusterRoleInfo.ClusterId"/>. When non-null-and-non-empty this scopes the row
/// side of the reconcile (<c>rows</c>/<c>enabled</c>) to that cluster — the only members
/// central's own mesh can see after the Phase 6 split. A null or empty value (a legacy admin
/// node with no cluster role) reconciles the whole fleet, unchanged.
/// </param>
/// <param name="ownClusterRole">
/// The admin node's own cluster role (e.g. <c>cluster-MAIN</c>), from
/// <see cref="IClusterRoleInfo.ClusterRole"/>. Scopes the membership (<c>observed</c>) side to
/// match the row side, so a foreign-cluster driver member central still sees via gossip in the
/// mixed-mesh window is not mis-reported. Null/empty keeps every driver member (legacy).
/// </param>
/// <param name="sweepInterval">Periodic sweep interval; defaults to <see cref="DefaultSweepInterval"/>.</param> /// <param name="sweepInterval">Periodic sweep interval; defaults to <see cref="DefaultSweepInterval"/>.</param>
/// <returns>Props for actor creation.</returns> /// <returns>Props for actor creation.</returns>
public static Props Props( public static Props Props(
IDbContextFactory<OtOpcUaConfigDbContext> dbFactory, TimeSpan? sweepInterval = null) => IDbContextFactory<OtOpcUaConfigDbContext> dbFactory,
Akka.Actor.Props.Create(() => new ClusterNodeAddressReconcilerActor(dbFactory, sweepInterval)); string? ownClusterId = null,
string? ownClusterRole = null,
TimeSpan? sweepInterval = null) =>
Akka.Actor.Props.Create(() =>
new ClusterNodeAddressReconcilerActor(dbFactory, ownClusterId, ownClusterRole, sweepInterval));
/// <summary>Initializes a new instance of the <see cref="ClusterNodeAddressReconcilerActor"/> class.</summary> /// <summary>Initializes a new instance of the <see cref="ClusterNodeAddressReconcilerActor"/> class.</summary>
/// <param name="dbFactory">Factory for the config database context.</param> /// <param name="dbFactory">Factory for the config database context.</param>
/// <param name="ownClusterId">
/// The admin node's own <c>ClusterId</c>; scopes the row side to that cluster's rows when
/// non-null-and-non-empty, else reconciles the whole fleet (legacy behaviour).
/// </param>
/// <param name="ownClusterRole">
/// The admin node's own cluster role; scopes the observed-membership side to that role when
/// non-null-and-non-empty, else keeps every driver member (legacy behaviour).
/// </param>
/// <param name="sweepInterval">Periodic sweep interval; defaults to <see cref="DefaultSweepInterval"/>.</param> /// <param name="sweepInterval">Periodic sweep interval; defaults to <see cref="DefaultSweepInterval"/>.</param>
public ClusterNodeAddressReconcilerActor( public ClusterNodeAddressReconcilerActor(
IDbContextFactory<OtOpcUaConfigDbContext> dbFactory, TimeSpan? sweepInterval = null) IDbContextFactory<OtOpcUaConfigDbContext> dbFactory,
string? ownClusterId = null,
string? ownClusterRole = null,
TimeSpan? sweepInterval = null)
{ {
_dbFactory = dbFactory; _dbFactory = dbFactory;
_ownClusterId = string.IsNullOrEmpty(ownClusterId) ? null : ownClusterId;
_ownClusterRole = string.IsNullOrEmpty(ownClusterRole) ? null : ownClusterRole;
_cluster = Akka.Cluster.Cluster.Get(Context.System); _cluster = Akka.Cluster.Cluster.Get(Context.System);
_sweepInterval = sweepInterval ?? DefaultSweepInterval; _sweepInterval = sweepInterval ?? DefaultSweepInterval;
@@ -75,24 +108,27 @@ public sealed class ClusterNodeAddressReconcilerActor : ReceiveActor, IWithTimer
private void Reconcile() private void Reconcile()
{ {
var observed = _cluster.State.Members var observed = FilterOwnClusterDriverMembers(_cluster.State.Members, _ownClusterRole);
.Where(m => m.Status == MemberStatus.Up && m.Roles.Contains(DriverRole))
.Select(m => (Host: m.Address.Host ?? string.Empty, Port: m.Address.Port ?? 0))
.Where(a => !string.IsNullOrWhiteSpace(a.Host) && a.Port > 0)
.ToList();
List<ClusterNodeAddress> rows; List<ClusterNodeAddress> rows;
List<string> enabled; List<string> enabled;
try try
{ {
using var db = _dbFactory.CreateDbContext(); using var db = _dbFactory.CreateDbContext();
rows = db.ClusterNodes.AsNoTracking() // Per-cluster mesh Phase 6: scope to the admin node's OWN cluster's rows. After the split,
// central's mesh sees only its own pair, so a row in another cluster it cannot observe would
// false-report EnabledRowNotInCluster forever. A legacy admin node with no cluster role
// (_ownClusterId == null) still sees every member via gossip, so it reconciles the full fleet.
var scoped = db.ClusterNodes.AsNoTracking();
if (_ownClusterId is not null)
scoped = scoped.Where(n => n.ClusterId == _ownClusterId);
rows = scoped
.Select(n => new ClusterNodeAddress(n.NodeId, n.Host, n.AkkaPort)) .Select(n => new ClusterNodeAddress(n.NodeId, n.Host, n.AkkaPort))
.ToList(); .ToList();
// Same predicate the coordinator's expected-ack set uses — a node the deploy path will // Same predicate the coordinator's expected-ack set uses — a node the deploy path will
// not wait for must not be reported as missing either, or the maintenance hatch trades a // not wait for must not be reported as missing either, or the maintenance hatch trades a
// failed deployment for a permanent warning. // failed deployment for a permanent warning.
enabled = db.ClusterNodes.AsNoTracking() enabled = scoped
.Where(n => n.Enabled && !n.MaintenanceMode).Select(n => n.NodeId).ToList(); .Where(n => n.Enabled && !n.MaintenanceMode).Select(n => n.NodeId).ToList();
} }
catch (Exception ex) catch (Exception ex)
@@ -126,6 +162,35 @@ public sealed class ClusterNodeAddressReconcilerActor : ReceiveActor, IWithTimer
} }
} }
/// <summary>
/// Projects the Up, driver-role members central should reconcile against, scoped to the admin
/// node's OWN cluster role when it carries one. Extracted so the observed-membership filter —
/// the membership half of the Phase 6 own-cluster scope, mirroring the <c>ClusterId</c> row
/// scope in <see cref="Reconcile"/> — is unit-testable without seeding a live cluster.
/// </summary>
/// <param name="members">The live cluster members (from <c>Cluster.State.Members</c>).</param>
/// <param name="ownClusterRole">
/// The admin node's own cluster role (e.g. <c>cluster-MAIN</c>). When non-null-and-non-empty,
/// driver members that do not also carry it are excluded — matching the row scope so a foreign
/// driver member central still sees via gossip in the mixed-mesh window (row filtered out, but
/// member still visible) is NOT mis-reported as an Error-level <c>RunningNodeHasNoRow</c>. A
/// null or empty value (a legacy node with no cluster role) keeps every driver member —
/// reconcile-all, unchanged.
/// </param>
/// <returns>The <c>host</c>/<c>port</c> of each in-scope Up driver member.</returns>
public static IReadOnlyList<(string Host, int Port)> FilterOwnClusterDriverMembers(
IEnumerable<Member> members, string? ownClusterRole)
{
var driverMembers = members
.Where(m => m.Status == MemberStatus.Up && m.Roles.Contains(DriverRole));
if (!string.IsNullOrEmpty(ownClusterRole))
driverMembers = driverMembers.Where(m => m.Roles.Contains(ownClusterRole));
return driverMembers
.Select(m => (Host: m.Address.Host ?? string.Empty, Port: m.Address.Port ?? 0))
.Where(a => !string.IsNullOrWhiteSpace(a.Host) && a.Port > 0)
.ToList();
}
/// <summary>Self-message: run a reconcile pass now.</summary> /// <summary>Self-message: run a reconcile pass now.</summary>
public sealed class ReconcileNow public sealed class ReconcileNow
{ {
@@ -31,10 +31,9 @@ public sealed record ManualFailoverSnapshot(string? PrimaryAddress, IReadOnlyLis
/// moves with it and OPC UA clients re-select. /// moves with it and OPC UA clients re-select.
/// </para> /// </para>
/// <para> /// <para>
/// <b>Mesh-scope caveat.</b> Until the per-cluster mesh work lands /// Post-Phase-6 (per-cluster mesh split): the Primary is elected per application
/// (<c>docs/plans/2026-07-21-per-cluster-mesh-design.md</c>) the Primary is elected once per /// <c>Cluster</c>, because each cluster's redundant pair now runs its own independent
/// Akka mesh, not per application <c>Cluster</c> row. On a fleet running several application /// 2-node Akka mesh. This service therefore acts on <b>this pair's own</b> Primary.
/// clusters in one mesh this acts on the whole mesh's Primary. The UI says so.
/// </para> /// </para>
/// </remarks> /// </remarks>
public interface IManualFailoverService public interface IManualFailoverService
@@ -2,6 +2,7 @@ using Akka.Actor;
using Akka.Cluster; using Akka.Cluster;
using Akka.Cluster.Tools.PublishSubscribe; using Akka.Cluster.Tools.PublishSubscribe;
using Akka.Event; using Akka.Event;
using ZB.MOM.WW.OtOpcUa.Cluster;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Redundancy; using ZB.MOM.WW.OtOpcUa.Commons.Messages.Redundancy;
using ZB.MOM.WW.OtOpcUa.Commons.Types; using ZB.MOM.WW.OtOpcUa.Commons.Types;
using CommonsRedundancyRole = ZB.MOM.WW.OtOpcUa.Commons.Messages.Redundancy.RedundancyRole; using CommonsRedundancyRole = ZB.MOM.WW.OtOpcUa.Commons.Messages.Redundancy.RedundancyRole;
@@ -112,7 +113,7 @@ public sealed class RedundancyStateActor : ReceiveActor, IWithTimers
} }
/// <summary>The cluster role that marks a node as carrying the driver runtime.</summary> /// <summary>The cluster role that marks a node as carrying the driver runtime.</summary>
public const string DriverRole = "driver"; public const string DriverRole = RoleParser.Driver;
/// <summary> /// <summary>
/// Selects the driver Primary: the <b>oldest</b> Up member carrying the <see cref="DriverRole"/>. /// Selects the driver Primary: the <b>oldest</b> Up member carrying the <see cref="DriverRole"/>.
@@ -13,6 +13,7 @@ using ZB.MOM.WW.OtOpcUa.ControlPlane.AdminOperations;
using ZB.MOM.WW.OtOpcUa.ControlPlane.Audit; using ZB.MOM.WW.OtOpcUa.ControlPlane.Audit;
using ZB.MOM.WW.OtOpcUa.ControlPlane.Communication; using ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
using ZB.MOM.WW.OtOpcUa.ControlPlane.Coordinators; using ZB.MOM.WW.OtOpcUa.ControlPlane.Coordinators;
using ZB.MOM.WW.OtOpcUa.Commons.Interfaces;
using ZB.MOM.WW.OtOpcUa.ControlPlane.Fleet; using ZB.MOM.WW.OtOpcUa.ControlPlane.Fleet;
using ZB.MOM.WW.OtOpcUa.ControlPlane.Redundancy; using ZB.MOM.WW.OtOpcUa.ControlPlane.Redundancy;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions; using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
@@ -130,10 +131,12 @@ public static class ServiceCollectionExtensions
(system, registry, resolver) => FleetStatusBroadcaster.Props(), (system, registry, resolver) => FleetStatusBroadcaster.Props(),
singletonOptions); singletonOptions);
builder.WithSingleton<RedundancyStateActorKey>( // Per-cluster mesh Phase 6: the redundancy-state singleton is NO LONGER registered here.
RedundancyStateSingletonName, // It was an admin-scoped, fleet-wide singleton — one Primary elected across the whole mesh.
(system, registry, resolver) => RedundancyStateActor.Props(), // After the mesh split a driver-only site pair has no admin node to host it, so it moved to
singletonOptions); // WithOtOpcUaClusterRedundancySingleton, a cluster-{ClusterId}-scoped singleton spawned on
// EVERY driver node (the fused central included). One per mesh, electing its own pair-local
// Primary and publishing on its own mesh's DistributedPubSub.
// Per-cluster mesh Phase 1: ClusterNode.AkkaPort duplicates the node's own Cluster:Port and // Per-cluster mesh Phase 1: ClusterNode.AkkaPort duplicates the node's own Cluster:Port and
// nothing makes them agree. Runs here rather than on each driver node because Phase 4 removes // nothing makes them agree. Runs here rather than on each driver node because Phase 4 removes
@@ -150,13 +153,77 @@ public static class ServiceCollectionExtensions
(system, registry, resolver) => (system, registry, resolver) =>
{ {
var dbFactory = resolver.GetService<IDbContextFactory<OtOpcUaConfigDbContext>>(); var dbFactory = resolver.GetService<IDbContextFactory<OtOpcUaConfigDbContext>>();
return ClusterNodeAddressReconcilerActor.Props(dbFactory); // Per-cluster mesh Phase 6: scope BOTH halves of the reconcile to the admin node's OWN
// cluster — rows by ClusterId, observed membership by cluster role. After the mesh split
// central sees only its own pair via gossip, so an unscoped sweep would false-report
// every other cluster's rows (EnabledRowNotInCluster). Scoping only the rows would swap
// that for a WORSE Error-level RunningNodeHasNoRow in the mixed-mesh window (rows already
// filtered, but foreign members still visible via gossip). A legacy node with no cluster
// role has null ClusterId/ClusterRole ⇒ full-fleet reconcile, unchanged.
var roleInfo = resolver.GetService<IClusterRoleInfo>();
return ClusterNodeAddressReconcilerActor.Props(
dbFactory, roleInfo.ClusterId, roleInfo.ClusterRole);
}, },
singletonOptions, singletonOptions,
createProxyToo: false); createProxyToo: false);
return builder; return builder;
} }
/// <summary>
/// Registers <see cref="RedundancyStateActor"/> as a cluster singleton scoped to the local node's
/// OWN <c>cluster-{ClusterId}</c> role (or the fixed <c>driver</c> role when the node carries no
/// cluster role), and spawns it on EVERY driver node (per-cluster mesh, Phase 6). Wire from the
/// driver branch of the fused Host's Program.cs — the fused central node is <c>hasDriver</c> too,
/// so it hosts exactly one redundancy singleton scoped to its own cluster role (e.g.
/// <c>cluster-MAIN</c>), while the admin singleton set it also registers no longer carries
/// redundancy.
/// </summary>
/// <param name="builder">The Akka configuration builder.</param>
/// <param name="roleInfo">The local node's cluster-role view (Task 1's <c>ClusterRoleInfo</c>).</param>
/// <returns>The builder for fluent chaining.</returns>
/// <remarks>
/// After the mesh split, central's mesh sees only its own pair — a driver-only site pair therefore
/// has NO admin node to elect its Primary. Scoping this singleton to the node's cluster role and
/// spawning it on every driver node gives each mesh exactly one, electing its own pair-local
/// Primary and publishing <c>redundancy-state</c> on its own mesh's DistributedPubSub.
/// </remarks>
public static AkkaConfigurationBuilder WithOtOpcUaClusterRedundancySingleton(
this AkkaConfigurationBuilder builder, IClusterRoleInfo roleInfo)
{
var singletonOptions = BuildClusterRedundancySingletonOptions(roleInfo);
builder.WithSingleton<RedundancyStateActorKey>(
RedundancyStateSingletonName,
(system, registry, resolver) => RedundancyStateActor.Props(),
singletonOptions);
return builder;
}
/// <summary>
/// Builds the <see cref="ClusterSingletonOptions"/> that scope the redundancy-state singleton to
/// the local node's <c>cluster-{ClusterId}</c> role, falling back to the fixed <c>driver</c> role
/// when the node carries no cluster role. Extracted (like
/// <c>SplitBrainResolverActivationTests</c>'s <c>BuildDowningHocon</c>) so the exact role scope is
/// unit-assertable without booting a cluster.
/// </summary>
/// <remarks>
/// Cluster-scope is the robust choice — it survives an accidental two-mesh merge by keeping one
/// singleton per cluster. The <c>driver</c>-role fallback deliberately does NOT throw when a node
/// has no cluster role, because (a) that is the pre-Phase-6 fleet-wide behavior on a legacy single
/// mesh, keeping legacy/harness nodes and any not-yet-migrated deployment booting unchanged, and
/// (b) on a genuinely split 2-node mesh the <c>driver</c> role is already pair-local — the mesh IS
/// the pair. So the fallback is correct in both worlds and costs no availability.
/// </remarks>
/// <param name="roleInfo">The local node's cluster-role view.</param>
/// <returns>Singleton options scoped to the node's cluster role, or the <c>driver</c> role.</returns>
public static ClusterSingletonOptions BuildClusterRedundancySingletonOptions(IClusterRoleInfo roleInfo)
{
ArgumentNullException.ThrowIfNull(roleInfo);
return new ClusterSingletonOptions { Role = roleInfo.ClusterRole ?? RoleParser.Driver };
}
} }
/// <summary>Marker key types used by <c>Akka.Hosting</c> to resolve singletons from the registry.</summary> /// <summary>Marker key types used by <c>Akka.Hosting</c> to resolve singletons from the registry.</summary>
@@ -12,6 +12,7 @@ using ZB.MOM.WW.OtOpcUa.AdminUI.Grpc;
using ZB.MOM.WW.OtOpcUa.AdminUI.Hubs; using ZB.MOM.WW.OtOpcUa.AdminUI.Hubs;
using ZB.MOM.WW.OtOpcUa.AdminUI.ScriptAnalysis; using ZB.MOM.WW.OtOpcUa.AdminUI.ScriptAnalysis;
using ZB.MOM.WW.OtOpcUa.Cluster; using ZB.MOM.WW.OtOpcUa.Cluster;
using ZB.MOM.WW.OtOpcUa.Commons.Interfaces;
using ZB.MOM.WW.OtOpcUa.Configuration; using ZB.MOM.WW.OtOpcUa.Configuration;
using ZB.MOM.WW.OtOpcUa.Configuration.Services; using ZB.MOM.WW.OtOpcUa.Configuration.Services;
using ZB.MOM.WW.OtOpcUa.ControlPlane; using ZB.MOM.WW.OtOpcUa.ControlPlane;
@@ -371,7 +372,18 @@ builder.Services.AddAkka("otopcua", (ab, sp) =>
ab.WithOtOpcUaSignalRBridges(); ab.WithOtOpcUaSignalRBridges();
} }
if (hasDriver) if (hasDriver)
{
ab.WithOtOpcUaRuntimeActors(); ab.WithOtOpcUaRuntimeActors();
// Per-cluster mesh Phase 6: re-home the redundancy-state singleton off the admin node onto
// EVERY driver node, so each mesh elects its own pair-local Primary (a driver-only site pair
// has no admin node to do it). It scopes to this node's own cluster-{ClusterId} role, falling
// back to the driver role for legacy single-mesh / not-yet-migrated nodes (already pair-local
// on a split mesh, since the mesh IS the pair). Runs on the fused central node too (it is
// hasDriver), where it scopes to cluster-MAIN — exactly one redundancy singleton per node,
// since the admin branch above no longer registers one. IClusterRoleInfo is registered by
// AddOtOpcUaCluster and resolved lazily here.
ab.WithOtOpcUaClusterRedundancySingleton(sp.GetRequiredService<IClusterRoleInfo>());
}
}); });
// Down-if-alone recovery watchdog (#459). Registered AFTER AddAkka so it starts after Akka's own // Down-if-alone recovery watchdog (#459). Registered AFTER AddAkka so it starts after Akka's own
@@ -36,7 +36,7 @@ namespace ZB.MOM.WW.OtOpcUa.Runtime;
public static class ServiceCollectionExtensions public static class ServiceCollectionExtensions
{ {
public const string DriverRole = "driver"; public const string DriverRole = RoleParser.Driver;
public const string DriverHostActorName = "driver-host"; public const string DriverHostActorName = "driver-host";
public const string DbHealthProbeActorName = "db-health"; public const string DbHealthProbeActorName = "db-health";
@@ -0,0 +1,105 @@
using Akka.Actor;
using Akka.Hosting;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Options;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Commons.Interfaces;
namespace ZB.MOM.WW.OtOpcUa.Cluster.Tests;
/// <summary>
/// Guards <see cref="ClusterRoleInfo.ClusterRole"/> / <see cref="ClusterRoleInfo.ClusterId"/> — the
/// node's OWN cluster-scoped role (per-cluster mesh, Phase 6), derived from the node's configured
/// <see cref="AkkaClusterOptions.Roles"/> at construction time, NOT from live cluster membership.
/// That distinction matters: this identity must be readable before the cluster forms, so later
/// Phase-6 work (singleton registration, a startup validator) can act on it at host-build time.
/// </summary>
public sealed class ClusterRoleInfoTests
{
private static AkkaClusterOptions SampleOptions(params string[] roles) => new()
{
Port = 0,
Hostname = "127.0.0.1",
PublicHostname = "127.0.0.1",
SeedNodes = Array.Empty<string>(),
Roles = roles,
};
/// <summary>
/// Boots a real host through the production <see cref="ServiceCollectionExtensions.WithOtOpcUaClusterBootstrap"/>
/// entry point (mirrors <c>SplitBrainResolverActivationTests</c>) — <see cref="ClusterRoleInfo"/>'s
/// constructor needs a real clustered <see cref="ActorSystem"/> to subscribe to cluster events,
/// even though the properties under test never touch live membership — resolves
/// <see cref="IClusterRoleInfo"/> from DI, and returns its <c>(ClusterRole, ClusterId)</c> pair
/// after tearing the host down. Both start/stop live in this helper rather than the test methods
/// so the CancellationToken-less calls stay off xunit1051's radar (mirrors
/// <c>SplitBrainResolverActivationTests.EffectiveConfigAsync</c>).
/// </summary>
private static async Task<(string? ClusterRole, string? ClusterId)> ResolveAsync(AkkaClusterOptions options)
{
var builder = Host.CreateDefaultBuilder();
builder.ConfigureServices(services =>
{
services.AddSingleton<IOptions<AkkaClusterOptions>>(Options.Create(options));
services.AddAkka("otopcua-role-info-test", (ab, sp) => ab.WithOtOpcUaClusterBootstrap(sp));
services.AddSingleton<IClusterRoleInfo, ClusterRoleInfo>();
});
using var host = builder.Build();
await host.StartAsync();
try
{
var roleInfo = host.Services.GetRequiredService<IClusterRoleInfo>();
return (roleInfo.ClusterRole, roleInfo.ClusterId);
}
finally
{
await host.StopAsync();
}
}
/// <summary>A node configured with a single cluster-scoped role exposes it and its ClusterId.</summary>
[Fact]
public async Task Single_cluster_role_exposes_role_and_id()
{
var (clusterRole, clusterId) = await ResolveAsync(SampleOptions("driver", "cluster-SITE-A"));
clusterRole.ShouldBe("cluster-SITE-A");
clusterId.ShouldBe("SITE-A");
}
/// <summary>A node configured with only fixed roles carries no cluster identity.</summary>
[Fact]
public async Task No_cluster_role_configured_yields_null()
{
var (clusterRole, clusterId) = await ResolveAsync(SampleOptions("admin", "driver"));
clusterRole.ShouldBeNull();
clusterId.ShouldBeNull();
}
/// <summary>
/// More than one cluster-scoped role is a misconfiguration; the first configured wins rather
/// than throwing.
/// </summary>
[Fact]
public async Task Multiple_cluster_roles_first_wins()
{
var (clusterRole, clusterId) = await ResolveAsync(SampleOptions("driver", "cluster-MAIN", "cluster-SITE-B"));
clusterRole.ShouldBe("cluster-MAIN");
clusterId.ShouldBe("MAIN");
}
/// <summary>An empty role list carries no cluster identity either.</summary>
[Fact]
public async Task Empty_roles_yields_null()
{
var (clusterRole, clusterId) = await ResolveAsync(SampleOptions());
clusterRole.ShouldBeNull();
clusterId.ShouldBeNull();
}
}
@@ -57,4 +57,57 @@ public sealed class RoleParserTests
{ {
Should.Throw<ArgumentException>(() => RoleParser.Parse("admin,master")); Should.Throw<ArgumentException>(() => RoleParser.Parse("admin,master"));
} }
/// <summary>Verifies that a single cluster-scoped role is accepted.</summary>
[Theory]
[InlineData("cluster-MAIN")]
[InlineData("cluster-SITE-A")]
public void Cluster_role_accepted(string role)
{
RoleParser.Parse(role).ShouldBe(new[] { role.ToLowerInvariant() });
}
/// <summary>Verifies that cluster roles mix freely with the fixed roles.</summary>
[Fact]
public void Fixed_and_cluster_roles_mix()
{
RoleParser.Parse("driver,cluster-SITE-A").ShouldBe(new[] { "driver", "cluster-site-a" });
}
/// <summary>Verifies that a cluster role with an empty ClusterId ("cluster-") is rejected.</summary>
[Fact]
public void Empty_cluster_id_throws()
{
Should.Throw<ArgumentException>(() => RoleParser.Parse("cluster-"));
}
/// <summary>Verifies IsClusterRole returns true for valid cluster roles and false otherwise.</summary>
[Theory]
[InlineData("cluster-MAIN", true)]
[InlineData("cluster-SITE-A", true)]
[InlineData("cluster-", false)]
[InlineData("driver", false)]
[InlineData("admin", false)]
[InlineData("clusterx", false)]
public void IsClusterRole_classifies_correctly(string role, bool expected)
{
RoleParser.IsClusterRole(role).ShouldBe(expected);
}
/// <summary>Verifies ClusterIdFromRole extracts the ClusterId suffix.</summary>
[Fact]
public void ClusterIdFromRole_extracts_suffix()
{
RoleParser.ClusterIdFromRole("cluster-SITE-A").ShouldBe("SITE-A");
}
/// <summary>Verifies the three well-known role constants match the historical literal values.</summary>
[Fact]
public void Well_known_role_constants()
{
RoleParser.Admin.ShouldBe("admin");
RoleParser.Driver.ShouldBe("driver");
RoleParser.Dev.ShouldBe("dev");
RoleParser.ClusterRolePrefix.ShouldBe("cluster-");
}
} }
@@ -0,0 +1,267 @@
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.Options;
using Shouldly;
using Xunit;
namespace ZB.MOM.WW.OtOpcUa.Cluster.Tests;
/// <summary>
/// A split-topology node (one carrying a <c>cluster-{ClusterId}</c> role) left on a DPS transport
/// mode does not error — the transport simply delivers nothing across the mesh boundary, and the
/// symptom is a deployment that never applies or telemetry that never arrives. These tests make
/// that misconfiguration a loud host-start failure instead. A node with no cluster-role is exempt
/// (legacy / single-mesh / test), and the validator is a no-op for it.
/// </summary>
public class SplitTopologyTransportValidatorTests
{
private static ValidateOptionsResult Validate(
string? meshMode,
string? telemetryMode,
string? telemetryDialMode,
params string[] roles)
{
var pairs = new Dictionary<string, string?>();
for (var i = 0; i < roles.Length; i++)
{
pairs[$"Cluster:Roles:{i}"] = roles[i];
}
if (meshMode is not null) pairs["MeshTransport:Mode"] = meshMode;
if (telemetryMode is not null) pairs["Telemetry:Mode"] = telemetryMode;
if (telemetryDialMode is not null) pairs["TelemetryDial:Mode"] = telemetryDialMode;
var configuration = new ConfigurationBuilder().AddInMemoryCollection(pairs).Build();
// The validated options instance is never read (all four values come from IConfiguration), so a
// default MeshTransportOptions is a fine stand-in.
return new SplitTopologyTransportValidator(configuration)
.Validate(MeshTransportOptions.SectionName, new MeshTransportOptions());
}
[Fact]
public void Cluster_role_driver_node_on_Dps_mesh_transport_fails()
{
// Red-before-green anchor: a split-topology node on DistributedPubSub cannot deliver commands.
var result = Validate(
meshMode: MeshTransportOptions.ModeDps,
telemetryMode: TelemetryOptions.ModeGrpc,
telemetryDialMode: null,
"driver", "cluster-SITE-A");
result.Failed.ShouldBeTrue();
result.FailureMessage.ShouldContain("MeshTransport:Mode");
result.FailureMessage.ShouldContain(MeshTransportOptions.ModeClusterClient);
result.FailureMessage.ShouldContain("SITE-A");
}
[Fact]
public void Cluster_role_driver_node_fully_configured_passes()
{
Validate(
meshMode: MeshTransportOptions.ModeClusterClient,
telemetryMode: TelemetryOptions.ModeGrpc,
telemetryDialMode: null,
"driver", "cluster-SITE-A").Succeeded.ShouldBeTrue();
}
[Fact]
public void Cluster_role_admin_node_fully_configured_passes()
{
Validate(
meshMode: MeshTransportOptions.ModeClusterClient,
telemetryMode: null,
telemetryDialMode: TelemetryDialOptions.ModeGrpc,
"admin", "cluster-SITE-A").Succeeded.ShouldBeTrue();
}
[Fact]
public void Cluster_role_driver_node_on_Dps_telemetry_fails()
{
var result = Validate(
meshMode: MeshTransportOptions.ModeClusterClient,
telemetryMode: TelemetryOptions.ModeDps,
telemetryDialMode: null,
"driver", "cluster-SITE-A");
result.Failed.ShouldBeTrue();
result.FailureMessage.ShouldContain("Telemetry:Mode");
result.FailureMessage.ShouldContain(TelemetryOptions.ModeGrpc);
}
[Fact]
public void Cluster_role_admin_node_on_Dps_telemetry_dial_fails()
{
var result = Validate(
meshMode: MeshTransportOptions.ModeClusterClient,
telemetryMode: null,
telemetryDialMode: TelemetryDialOptions.ModeDps,
"admin", "cluster-SITE-A");
result.Failed.ShouldBeTrue();
result.FailureMessage.ShouldContain("TelemetryDial:Mode");
result.FailureMessage.ShouldContain(TelemetryDialOptions.ModeGrpc);
}
[Fact]
public void Fused_cluster_role_node_needs_all_three_to_pass()
{
Validate(
meshMode: MeshTransportOptions.ModeClusterClient,
telemetryMode: TelemetryOptions.ModeGrpc,
telemetryDialMode: TelemetryDialOptions.ModeGrpc,
"admin", "driver", "cluster-MAIN").Succeeded.ShouldBeTrue();
}
[Fact]
public void Fused_cluster_role_node_missing_telemetry_grpc_fails()
{
var result = Validate(
meshMode: MeshTransportOptions.ModeClusterClient,
telemetryMode: TelemetryOptions.ModeDps,
telemetryDialMode: TelemetryDialOptions.ModeGrpc,
"admin", "driver", "cluster-MAIN");
result.Failed.ShouldBeTrue();
result.FailureMessage.ShouldContain("Telemetry:Mode");
}
[Fact]
public void Fused_cluster_role_node_missing_telemetry_dial_grpc_fails()
{
var result = Validate(
meshMode: MeshTransportOptions.ModeClusterClient,
telemetryMode: TelemetryOptions.ModeGrpc,
telemetryDialMode: TelemetryDialOptions.ModeDps,
"admin", "driver", "cluster-MAIN");
result.Failed.ShouldBeTrue();
result.FailureMessage.ShouldContain("TelemetryDial:Mode");
}
[Fact]
public void Non_cluster_role_node_on_all_Dps_passes()
{
// The exemption that keeps the validator a no-op for every legacy / single-mesh / test node:
// no cluster-{ClusterId} role means the split-topology rule does not apply at all.
Validate(
meshMode: MeshTransportOptions.ModeDps,
telemetryMode: TelemetryOptions.ModeDps,
telemetryDialMode: TelemetryDialOptions.ModeDps,
"admin", "driver").Succeeded.ShouldBeTrue();
}
[Fact]
public void Node_with_no_roles_on_all_Dps_passes()
{
// Absolute default: nothing declared, nothing to enforce.
Validate(
meshMode: MeshTransportOptions.ModeDps,
telemetryMode: TelemetryOptions.ModeDps,
telemetryDialMode: TelemetryDialOptions.ModeDps).Succeeded.ShouldBeTrue();
}
[Fact]
public void Unset_modes_on_a_cluster_role_node_fail_as_the_Dps_default()
{
// A split node that never sets the mode keys inherits the Dps defaults — which the split
// topology cannot use. Leaving them unset must fail exactly as setting them to Dps would.
var result = Validate(
meshMode: null,
telemetryMode: null,
telemetryDialMode: null,
"admin", "driver", "cluster-MAIN");
result.Failed.ShouldBeTrue();
result.FailureMessage.ShouldContain("MeshTransport:Mode");
}
[Fact]
public void Mode_matching_is_case_insensitive()
{
Validate(
meshMode: "clusterclient",
telemetryMode: "grpc",
telemetryDialMode: "grpc",
"admin", "driver", "cluster-MAIN").Succeeded.ShouldBeTrue();
}
[Fact]
public void Cluster_role_from_OTOPCUA_ROLES_only_on_Dps_fails()
{
// Roles-source divergence: Cluster:Roles is empty, so the node's effective Akka roles come from
// OTOPCUA_ROLES (via RoleParser.Parse) exactly as Program.cs resolves them. A validator that
// read only Cluster:Roles would see no cluster role and pass silently on Dps — the documented
// production-incident shape.
using (new EnvVarScope("OTOPCUA_ROLES", "driver,cluster-SITE-A"))
{
var pairs = new Dictionary<string, string?> { ["MeshTransport:Mode"] = MeshTransportOptions.ModeDps };
var configuration = new ConfigurationBuilder().AddInMemoryCollection(pairs).Build();
var result = new SplitTopologyTransportValidator(configuration)
.Validate(MeshTransportOptions.SectionName, new MeshTransportOptions());
result.Failed.ShouldBeTrue();
result.FailureMessage.ShouldContain("MeshTransport:Mode");
}
}
[Fact]
public void Cluster_role_from_OTOPCUA_ROLES_is_ignored_when_Cluster_Roles_is_set()
{
// When Cluster:Roles is populated it wins outright — the OTOPCUA_ROLES fallback is not consulted,
// mirroring AkkaClusterOptions.Roles. A non-cluster Cluster:Roles therefore stays exempt even if
// a stray OTOPCUA_ROLES carries a cluster role.
using (new EnvVarScope("OTOPCUA_ROLES", "driver,cluster-SITE-A"))
{
Validate(
meshMode: MeshTransportOptions.ModeDps,
telemetryMode: MeshTransportOptions.ModeDps,
telemetryDialMode: MeshTransportOptions.ModeDps,
"admin", "driver").Succeeded.ShouldBeTrue();
}
}
[Fact]
public void Mixed_case_cluster_and_role_names_on_Dps_fail()
{
// Case-sensitivity: the Cluster:Roles bind path is verbatim, so mixed-case entries must be
// normalized here or they slip past the ordinal IsClusterRole / driver checks and pass silently.
var result = Validate(
meshMode: MeshTransportOptions.ModeDps,
telemetryMode: TelemetryOptions.ModeGrpc,
telemetryDialMode: null,
"Driver", "Cluster-SITE-A");
result.Failed.ShouldBeTrue();
result.FailureMessage.ShouldContain("MeshTransport:Mode");
}
[Fact]
public void Cluster_role_node_with_neither_admin_nor_driver_only_needs_ClusterClient()
{
// A cluster-role node that is neither admin nor driver has no telemetry surface to constrain —
// only the always-on MeshTransport=ClusterClient rule applies. Telemetry/TelemetryDial left on
// their Dps defaults must not fail it.
Validate(
meshMode: MeshTransportOptions.ModeClusterClient,
telemetryMode: null,
telemetryDialMode: null,
"cluster-SITE-A").Succeeded.ShouldBeTrue();
}
/// <summary>Sets an environment variable for the scope's lifetime and restores its prior value on dispose.</summary>
private sealed class EnvVarScope : IDisposable
{
private readonly string _name;
private readonly string? _previous;
public EnvVarScope(string name, string? value)
{
_name = name;
_previous = Environment.GetEnvironmentVariable(name);
Environment.SetEnvironmentVariable(name, value);
}
public void Dispose() => Environment.SetEnvironmentVariable(_name, _previous);
}
}
@@ -70,24 +70,80 @@ public sealed class ClusterNodeAddressReconcilerActorTests : ControlPlaneActorTe
}); });
} }
/// <summary>
/// Per-cluster mesh Phase 6: an admin node scoped to its OWN cluster ignores rows in a cluster
/// its mesh can no longer see. Scoped to <c>MAIN</c>, a <c>SITE-A</c> enabled row is filtered
/// out before <see cref="ClusterNodeAddressReconciler.Reconcile"/> ever sees it, so it produces
/// no <see cref="AddressMismatchKind.EnabledRowNotInCluster"/> warning. This is the false
/// positive the split would otherwise print forever.
/// </summary>
[Fact]
public void Row_in_a_foreign_cluster_is_ignored_when_scoped()
{
var dbFactory = NewInMemoryDbFactory();
SeedNode(dbFactory, "site-a-1:4053", enabled: true, clusterId: "SITE-A");
EventFilter.Warning(contains: "site-a-1:4053").Expect(0, TimeSpan.FromSeconds(2), () =>
{
Sys.ActorOf(ClusterNodeAddressReconcilerActor.Props(
dbFactory, ownClusterId: "MAIN", sweepInterval: TimeSpan.FromMilliseconds(400)));
ExpectNoMsg(TimeSpan.FromSeconds(1.5));
});
}
/// <summary>
/// Scoping does not silence the admin node's own cluster. Scoped to <c>MAIN</c>, an enabled
/// <c>MAIN</c> row with no matching driver member is still warned about — proving the filter
/// keeps in-cluster rows (and with them, within-cluster drift detection).
/// </summary>
[Fact]
public void Enabled_row_in_own_cluster_is_still_logged_when_scoped()
{
var dbFactory = NewInMemoryDbFactory();
SeedNode(dbFactory, "main-1:4053", enabled: true, clusterId: "MAIN");
EventFilter.Warning(contains: "main-1:4053").ExpectOne(TimeSpan.FromSeconds(10), () =>
Sys.ActorOf(ClusterNodeAddressReconcilerActor.Props(dbFactory, ownClusterId: "MAIN")));
}
/// <summary>
/// Legacy backward-compat: an admin node with no cluster role (null <c>ownClusterId</c>) still
/// reconciles the WHOLE fleet — such a node genuinely sees every member via gossip. A
/// <c>SITE-A</c> enabled row with no member is warned about, exactly as before the Phase 6 scope.
/// </summary>
[Fact]
public void Foreign_cluster_row_is_still_logged_when_unscoped()
{
var dbFactory = NewInMemoryDbFactory();
SeedNode(dbFactory, "site-a-1:4053", enabled: true, clusterId: "SITE-A");
EventFilter.Warning(contains: "site-a-1:4053").ExpectOne(TimeSpan.FromSeconds(10), () =>
Sys.ActorOf(ClusterNodeAddressReconcilerActor.Props(dbFactory, ownClusterId: null)));
}
private static void SeedNode( private static void SeedNode(
IDbContextFactory<OtOpcUaConfigDbContext> dbFactory, string nodeId, bool enabled) IDbContextFactory<OtOpcUaConfigDbContext> dbFactory, string nodeId, bool enabled,
string clusterId = "SITE-A")
{ {
using var db = dbFactory.CreateDbContext(); using var db = dbFactory.CreateDbContext();
if (!db.ServerClusters.Any(c => c.ClusterId == clusterId))
{
db.ServerClusters.Add(new ServerCluster db.ServerClusters.Add(new ServerCluster
{ {
ClusterId = "SITE-A", ClusterId = clusterId,
Name = "Site A", Name = clusterId,
Enterprise = "zb", Enterprise = "zb",
Site = "site-a", Site = clusterId.ToLowerInvariant(),
NodeCount = 2, NodeCount = 2,
RedundancyMode = RedundancyMode.Warm, RedundancyMode = RedundancyMode.Warm,
CreatedBy = "test", CreatedBy = "test",
}); });
}
db.ClusterNodes.Add(new ClusterNode db.ClusterNodes.Add(new ClusterNode
{ {
NodeId = nodeId, NodeId = nodeId,
ClusterId = "SITE-A", ClusterId = clusterId,
Host = nodeId.Split(':')[0], Host = nodeId.Split(':')[0],
ApplicationUri = $"urn:OtOpcUa:{nodeId}", ApplicationUri = $"urn:OtOpcUa:{nodeId}",
Enabled = enabled, Enabled = enabled,
@@ -1,3 +1,8 @@
using System.Collections.Immutable;
using System.Reflection;
using Akka.Actor;
using Akka.Cluster;
using Akka.Util;
using Shouldly; using Shouldly;
using Xunit; using Xunit;
using ZB.MOM.WW.OtOpcUa.ControlPlane.Fleet; using ZB.MOM.WW.OtOpcUa.ControlPlane.Fleet;
@@ -123,4 +128,90 @@ public sealed class ClusterNodeAddressReconcilerTests
[("SITE-A-1", 4053)], [Row("site-a-1", 4053)], ["site-a-1:4053"]) [("SITE-A-1", 4053)], [Row("site-a-1", 4053)], ["site-a-1:4053"])
.ShouldBeEmpty(); .ShouldBeEmpty();
} }
private static int _memberUid;
// Akka's Member.Create factories are internal, so reflect the Up-status overload
// Create(UniqueAddress, int upNumber, MemberStatus, ImmutableHashSet<string> roles, AppVersion).
private static readonly MethodInfo MemberCreate = typeof(Member).GetMethod(
"Create",
BindingFlags.Static | BindingFlags.NonPublic | BindingFlags.Public,
binder: null,
types:
[
typeof(UniqueAddress), typeof(int), typeof(MemberStatus),
typeof(ImmutableHashSet<string>), typeof(AppVersion),
],
modifiers: null)
?? throw new InvalidOperationException("Akka.Cluster.Member.Create(5-arg) not found — Akka API changed");
/// <summary>
/// Builds an Up cluster <see cref="Member"/> at <paramref name="host"/>:<paramref name="port"/>
/// carrying <paramref name="roles"/>, for exercising
/// <see cref="ClusterNodeAddressReconcilerActor.FilterOwnClusterDriverMembers"/> directly.
/// </summary>
private static Member UpMember(string host, int port, params string[] roles) =>
(Member)MemberCreate.Invoke(null,
[
new UniqueAddress(new Address("akka.tcp", "otopcua", host, port), Interlocked.Increment(ref _memberUid)),
1,
MemberStatus.Up,
roles.ToImmutableHashSet(),
AppVersion.Zero,
])!;
/// <summary>
/// Per-cluster mesh Phase 6: scoped to <c>cluster-MAIN</c>, a driver member carrying a FOREIGN
/// cluster role (<c>cluster-SITE-A</c>) — the mixed-mesh window, where central still sees it via
/// gossip while its row is already filtered out by ClusterId — is excluded from observed. Left
/// in, it would hit the Error-level <see cref="AddressMismatchKind.RunningNodeHasNoRow"/> branch.
/// </summary>
[Fact]
public void Foreign_cluster_role_member_is_excluded_when_scoped()
{
var observed = ClusterNodeAddressReconcilerActor.FilterOwnClusterDriverMembers(
[UpMember("site-a-1", 4053, "driver", "cluster-SITE-A")],
ownClusterRole: "cluster-MAIN");
observed.ShouldBeEmpty();
}
/// <summary>Scoped to <c>cluster-MAIN</c>, a member carrying that same role is kept.</summary>
[Fact]
public void Own_cluster_role_member_is_kept_when_scoped()
{
var observed = ClusterNodeAddressReconcilerActor.FilterOwnClusterDriverMembers(
[UpMember("main-1", 4053, "driver", "cluster-MAIN")],
ownClusterRole: "cluster-MAIN");
observed.ShouldHaveSingleItem().ShouldBe(("main-1", 4053));
}
/// <summary>
/// Legacy: a null cluster role keeps every Up driver member regardless of cluster — an admin
/// node with no cluster role genuinely sees the whole fleet via gossip (reconcile-all).
/// </summary>
[Fact]
public void Null_cluster_role_keeps_all_driver_members()
{
var observed = ClusterNodeAddressReconcilerActor.FilterOwnClusterDriverMembers(
[
UpMember("main-1", 4053, "driver", "cluster-MAIN"),
UpMember("site-a-1", 4053, "driver", "cluster-SITE-A"),
],
ownClusterRole: null);
observed.Count.ShouldBe(2);
observed.ShouldContain(("main-1", 4053));
observed.ShouldContain(("site-a-1", 4053));
}
/// <summary>Non-driver members (e.g. admin-only) are excluded whether or not a scope is set.</summary>
[Fact]
public void Non_driver_members_are_always_excluded()
{
ClusterNodeAddressReconcilerActor.FilterOwnClusterDriverMembers(
[UpMember("central-1", 4053, "admin", "cluster-MAIN")], ownClusterRole: "cluster-MAIN")
.ShouldBeEmpty();
}
} }
@@ -31,13 +31,20 @@ public class CentralCommunicationActorTests : ControlPlaneActorTestBase
public List<ImmutableHashSet<ActorPath>> Calls { get; } = []; public List<ImmutableHashSet<ActorPath>> Calls { get; } = [];
public List<string> ClusterIds { get; } = [];
public List<TestProbe> Probes { get; } = []; public List<TestProbe> Probes { get; } = [];
public IActorRef Create(ActorSystem system, ImmutableHashSet<ActorPath> contacts) /// <summary>Probe created for a given cluster id, for per-cluster fan-out assertions.</summary>
public Dictionary<string, TestProbe> ProbeByCluster { get; } = [];
public IActorRef Create(ActorSystem system, string clusterId, ImmutableHashSet<ActorPath> contacts)
{ {
Calls.Add(contacts); Calls.Add(contacts);
ClusterIds.Add(clusterId);
var probe = _newProbe(); var probe = _newProbe();
Probes.Add(probe); Probes.Add(probe);
ProbeByCluster[clusterId] = probe;
return probe.Ref; return probe.Ref;
} }
} }
@@ -138,23 +145,54 @@ public class CentralCommunicationActorTests : ControlPlaneActorTestBase
} }
[Fact] [Fact]
public void Exactly_one_client_is_created_for_a_multi_cluster_fleet() public void One_client_per_cluster_is_created_for_a_multi_cluster_fleet()
{ {
// A receptionist serves its whole mesh, so one client per Cluster would fan each command out // THE Phase 6 inversion. Each application Cluster is now its own Akka mesh, and a ClusterClient
// once per cluster while the fleet is still a single mesh — N x duplicate deployments. // reaches only the mesh whose receptionist it dialled. Central must therefore build one client
// per cluster, each dialling only that cluster's nodes.
var factory = NewInMemoryDbFactory(); var factory = NewInMemoryDbFactory();
SeedCluster(factory, "C1", ("host-a:4053", "host-a", 4053, true, false)); SeedCluster(factory, "C1", ("host-a:4053", "host-a", 4053, true, false));
SeedCluster(factory, "C2", ("host-b:4053", "host-b", 4053, true, false)); SeedCluster(factory, "C2", ("host-b:4053", "host-b", 4053, true, false));
var clients = new RecordingClientFactory(() => CreateTestProbe()); var clients = new RecordingClientFactory(() => CreateTestProbe());
Spawn(factory, ClusterClientMode(), clients); Spawn(factory, ClusterClientMode(), clients);
AwaitCondition(() => clients.Calls.Count == 1, TimeSpan.FromSeconds(5)); AwaitCondition(() => clients.Calls.Count == 2, TimeSpan.FromSeconds(5));
// ...and that one client dials BOTH clusters' nodes. // Give a second refresh a chance to (wrongly) create more.
clients.Calls[0].Count.ShouldBe(2);
// Give a second refresh a chance to (wrongly) create another.
ExpectNoMsg(TimeSpan.FromMilliseconds(300)); ExpectNoMsg(TimeSpan.FromMilliseconds(300));
clients.Calls.Count.ShouldBe(1); clients.Calls.Count.ShouldBe(2);
clients.ClusterIds.ShouldBe(new[] { "C1", "C2" }, ignoreOrder: true);
// Each client dials ONLY its own cluster's node.
clients.Calls[clients.ClusterIds.IndexOf("C1")].Single().ToString().ShouldContain("host-a");
clients.Calls[clients.ClusterIds.IndexOf("C2")].Single().ToString().ShouldContain("host-b");
}
[Fact]
public void A_command_fans_out_a_send_to_all_to_every_cluster_client()
{
// The other half of the inversion: one MeshCommand must reach EVERY cluster's mesh, so central
// fans SendToAll across all per-cluster clients. A single Send/one-client shape would leave
// every cluster but one silently on its old configuration.
var factory = NewInMemoryDbFactory();
SeedCluster(factory, "C1", ("host-a:4053", "host-a", 4053, true, false));
SeedCluster(factory, "C2", ("host-b:4053", "host-b", 4053, true, false));
var clients = new RecordingClientFactory(() => CreateTestProbe());
var actor = Spawn(factory, ClusterClientMode(), clients);
AwaitCondition(() => clients.Probes.Count == 2, TimeSpan.FromSeconds(5));
var dispatch = new DispatchDeployment(
new DeploymentId(Guid.NewGuid()), new RevisionHash("abc"), new CorrelationId(Guid.NewGuid()));
actor.Tell(new MeshCommand(DeploymentsTopic, dispatch));
foreach (var (clusterId, probe) in clients.ProbeByCluster)
{
var sent = probe.ExpectMsg<ClusterClient.SendToAll>(
TimeSpan.FromSeconds(5), $"cluster {clusterId} must receive the fanned SendToAll");
sent.Path.ShouldBe(MeshPaths.NodeCommunication);
sent.Message.ShouldBe(dispatch);
}
} }
[Fact] [Fact]
@@ -37,13 +37,13 @@ public class MeshClusterClientFactoryTests : TestKit
{ {
var factory = new DefaultMeshClusterClientFactory(); var factory = new DefaultMeshClusterClientFactory();
var first = factory.Create(Sys, UnreachableContact); var first = factory.Create(Sys, "C1", UnreachableContact);
Sys.Stop(first); Sys.Stop(first);
// Stop is asynchronous: `first`'s name is still reserved right here. Without the generation // Stop is asynchronous: `first`'s name is still reserved right here. Without the generation
// suffix this throws InvalidActorNameException — which is precisely what a contact-set // suffix this throws InvalidActorNameException — which is precisely what a contact-set
// change would do in production, where stop-then-recreate happens in one handler. // change would do in production, where stop-then-recreate happens in one handler.
var second = factory.Create(Sys, UnreachableContact); var second = factory.Create(Sys, "C1", UnreachableContact);
second.Path.Name.ShouldNotBe(first.Path.Name); second.Path.Name.ShouldNotBe(first.Path.Name);
} }
@@ -54,12 +54,27 @@ public class MeshClusterClientFactoryTests : TestKit
var factory = new DefaultMeshClusterClientFactory(); var factory = new DefaultMeshClusterClientFactory();
var names = Enumerable.Range(0, 5) var names = Enumerable.Range(0, 5)
.Select(_ => factory.Create(Sys, UnreachableContact).Path.Name) .Select(_ => factory.Create(Sys, "C1", UnreachableContact).Path.Name)
.ToList(); .ToList();
names.Distinct().Count().ShouldBe(5); names.Distinct().Count().ShouldBe(5);
} }
[Fact]
public void The_cluster_id_is_woven_into_the_client_actor_name()
{
// Per-cluster clients must be distinguishable in logs and the actor tree, and — more
// importantly — two clusters' clients created on one system must never collide on a name.
var factory = new DefaultMeshClusterClientFactory();
var a = factory.Create(Sys, "site-a", UnreachableContact);
var b = factory.Create(Sys, "site-b", UnreachableContact);
a.Path.Name.ShouldContain("site-a");
b.Path.Name.ShouldContain("site-b");
a.Path.Name.ShouldNotBe(b.Path.Name);
}
// Contact-set propagation is deliberately NOT asserted here: reading initial contacts back // Contact-set propagation is deliberately NOT asserted here: reading initial contacts back
// off a created ClusterClient is not possible, and asserting ClusterClientSettings directly // off a created ClusterClient is not possible, and asserting ClusterClientSettings directly
// would test Akka rather than this factory. It is covered where it can actually fail — the // would test Akka rather than this factory. It is covered where it can actually fail — the
@@ -0,0 +1,182 @@
using Akka.Actor;
using Akka.Cluster.Hosting;
using Akka.Hosting;
using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Options;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Cluster;
using ZB.MOM.WW.OtOpcUa.Commons.Interfaces;
using ZB.MOM.WW.OtOpcUa.Commons.Types;
using ZB.MOM.WW.OtOpcUa.Configuration;
namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Tests;
/// <summary>
/// Guards Phase 6's re-homing of <c>RedundancyStateActor</c> off the fleet-wide admin singleton onto
/// a per-node <c>cluster-{ClusterId}</c> singleton (see
/// <see cref="ServiceCollectionExtensions.WithOtOpcUaClusterRedundancySingleton"/>). After the mesh
/// split a driver-only site pair has NO admin node to elect its Primary, so the singleton must be
/// scoped to the node's own cluster role and spawned on every driver node — one per mesh.
///
/// <para>The role scope + the missing-role guard are asserted against the extracted
/// <see cref="ServiceCollectionExtensions.BuildClusterRedundancySingletonOptions"/> helper directly —
/// the same pattern <c>SplitBrainResolverActivationTests</c> uses for <c>BuildDowningHocon</c> /
/// <c>BuildClusterOptions</c>: it pins the exact <see cref="ClusterSingletonOptions.Role"/> without
/// booting a cluster, which no builder-introspection API can otherwise reach. The admin-removal claim
/// is behavioral (a booted admin node's <see cref="ActorRegistry"/> must no longer carry the
/// redundancy key) so it needs a real host.</para>
/// </summary>
public sealed class RedundancyStateSingletonRehomeTests
{
/// <summary>The singleton options scope to the node's OWN cluster role, not the fixed admin role.</summary>
[Fact]
public void Options_scope_the_singleton_to_the_nodes_cluster_role()
{
var options = ServiceCollectionExtensions.BuildClusterRedundancySingletonOptions(
new FakeClusterRoleInfo(clusterRole: "cluster-SITE-A", clusterId: "SITE-A"));
options.Role.ShouldBe("cluster-SITE-A");
}
/// <summary>
/// A node with no <c>cluster-{ClusterId}</c> role (a legacy single-mesh node, or the
/// <c>TwoNodeClusterHarness</c> with roles <c>admin,driver</c>) falls back to the fixed
/// <c>driver</c> role rather than throwing — so legacy/not-yet-migrated deployments keep booting.
/// On a split 2-node mesh the <c>driver</c> role is already pair-local (the mesh IS the pair).
/// </summary>
[Fact]
public void Missing_cluster_role_falls_back_to_the_driver_role()
{
var options = ServiceCollectionExtensions.BuildClusterRedundancySingletonOptions(
new FakeClusterRoleInfo(clusterRole: null, clusterId: null));
options.Role.ShouldBe(RoleParser.Driver);
}
/// <summary>
/// The admin singleton set no longer registers <c>redundancy-state</c>: a booted admin node's
/// registry carries the OTHER admin singletons (proxy) but NOT the redundancy key.
/// </summary>
[Fact]
public async Task Admin_singletons_no_longer_register_the_redundancy_singleton()
{
var registry = await BootAndGetRegistryAsync(
roles: new[] { "admin" },
configure: ab => ab.WithOtOpcUaControlPlaneSingletons());
registry.TryGet<RedundancyStateActorKey>(out _).ShouldBeFalse(
"the redundancy singleton was re-homed off the admin node onto every driver node (Phase 6)");
registry.TryGet<FleetStatusBroadcasterKey>(out _).ShouldBeTrue(
"the remaining admin singletons must still register");
}
/// <summary>
/// The re-home target: a driver node carrying a <c>cluster-{ClusterId}</c> role registers the
/// redundancy singleton (proxy) via the new extension.
/// </summary>
[Fact]
public async Task Cluster_redundancy_extension_registers_the_singleton_on_a_driver_node()
{
var registry = await BootAndGetRegistryAsync(
roles: new[] { "driver", "cluster-SITE-A" },
configure: ab => ab.WithOtOpcUaClusterRedundancySingleton(
new FakeClusterRoleInfo(clusterRole: "cluster-SITE-A", clusterId: "SITE-A")));
registry.TryGet<RedundancyStateActorKey>(out _).ShouldBeTrue(
"every driver node hosts its own mesh's redundancy singleton (Phase 6)");
}
/// <summary>
/// The driver-role FALLBACK end-to-end: a node carrying only <c>driver</c> (no cluster role — a
/// legacy single-mesh node or the <c>TwoNodeClusterHarness</c>) still boots and registers the
/// redundancy singleton via the new extension. Proves the fallback works at the host/registry
/// level, not just in the pure helper — the boot that the earlier throw-based version would have
/// aborted.
/// </summary>
[Fact]
public async Task Driver_role_fallback_registers_the_singleton_end_to_end()
{
var registry = await BootAndGetRegistryAsync(
roles: new[] { "driver" },
configure: ab => ab.WithOtOpcUaClusterRedundancySingleton(
new FakeClusterRoleInfo(clusterRole: null, clusterId: null)));
registry.TryGet<RedundancyStateActorKey>(out _).ShouldBeTrue(
"a driver node with no cluster role falls back to the driver role and still boots + registers");
}
/// <summary>
/// Boots a real host through the production cluster bootstrap (Port=0, no seeds, so it never forms
/// a cluster and no singleton props factory runs — only the proxies register), applies the
/// supplied singleton registration, and returns the <see cref="ActorRegistry"/> after teardown.
/// Start/stop live here rather than the test bodies to keep the CancellationToken-less calls out
/// of the way (mirrors <c>ClusterRoleInfoTests.ResolveAsync</c>).
/// </summary>
private static async Task<ActorRegistry> BootAndGetRegistryAsync(
string[] roles, Action<AkkaConfigurationBuilder> configure)
{
var options = new AkkaClusterOptions
{
Port = 0,
Hostname = "127.0.0.1",
PublicHostname = "127.0.0.1",
SeedNodes = Array.Empty<string>(),
Roles = roles,
};
var builder = Host.CreateDefaultBuilder();
builder.ConfigureServices(services =>
{
services.AddSingleton<IOptions<AkkaClusterOptions>>(Options.Create(options));
// The admin WithActors block reads IOptions<MeshTransportOptions>.Value at spawn time.
services.AddSingleton<IOptions<MeshTransportOptions>>(Options.Create(new MeshTransportOptions()));
// Akka.Hosting invokes every singleton's props factory eagerly at StartAsync; the admin
// singletons (coordinator/audit/reconciler) read IDbContextFactory<OtOpcUaConfigDbContext>,
// so supply an in-memory one or those factories NRE. The driver-only boots don't touch it.
services.AddDbContextFactory<OtOpcUaConfigDbContext>(
o => o.UseInMemoryDatabase($"rehome-test-{Guid.NewGuid():N}"));
// The admin ClusterNodeAddressReconciler props factory resolves IClusterRoleInfo (for its
// per-cluster reconcile scope). A null cluster role ⇒ full-fleet reconcile, which is all
// this test needs — it never asserts on the reconciler, only on the singleton registry.
services.AddSingleton<IClusterRoleInfo>(new FakeClusterRoleInfo(clusterRole: null, clusterId: null));
services.AddAkka("otopcua-rehome-test", (ab, sp) =>
{
ab.WithOtOpcUaClusterBootstrap(sp);
configure(ab);
});
});
using var host = builder.Build();
await host.StartAsync();
try
{
return host.Services.GetRequiredService<ActorRegistry>();
}
finally
{
await host.StopAsync();
}
}
/// <summary>Minimal <see cref="IClusterRoleInfo"/> stub exposing only the cluster-role identity.</summary>
private sealed class FakeClusterRoleInfo : IClusterRoleInfo
{
public FakeClusterRoleInfo(string? clusterRole, string? clusterId)
{
ClusterRole = clusterRole;
ClusterId = clusterId;
}
public string? ClusterRole { get; }
public string? ClusterId { get; }
public NodeId LocalNode => NodeId.Parse("127.0.0.1:0");
public IReadOnlySet<string> LocalRoles => new HashSet<string>();
public bool HasRole(string role) => false;
public IReadOnlyList<NodeId> MembersWithRole(string role) => Array.Empty<NodeId>();
public NodeId? RoleLeader(string role) => null;
public event EventHandler<RoleLeaderChangedEventArgs>? RoleLeaderChanged { add { } remove { } }
}
}
@@ -3,12 +3,16 @@ using Akka.Actor;
using Akka.Cluster; using Akka.Cluster;
using Akka.Cluster.Tools.Client; using Akka.Cluster.Tools.Client;
using Akka.Configuration; using Akka.Configuration;
using Microsoft.EntityFrameworkCore;
using Shouldly; using Shouldly;
using Xunit; using Xunit;
using ZB.MOM.WW.OtOpcUa.Cluster;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Admin; using ZB.MOM.WW.OtOpcUa.Commons.Messages.Admin;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Deploy; using ZB.MOM.WW.OtOpcUa.Commons.Messages.Deploy;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh; using ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
using ZB.MOM.WW.OtOpcUa.Commons.Types; using ZB.MOM.WW.OtOpcUa.Commons.Types;
using ZB.MOM.WW.OtOpcUa.Configuration;
using ZB.MOM.WW.OtOpcUa.Configuration.Entities;
using ZB.MOM.WW.OtOpcUa.ControlPlane.Communication; using ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
using ZB.MOM.WW.OtOpcUa.Runtime.Communication; using ZB.MOM.WW.OtOpcUa.Runtime.Communication;
using AkkaCluster = Akka.Cluster.Cluster; using AkkaCluster = Akka.Cluster.Cluster;
@@ -86,12 +90,12 @@ public sealed class MeshClusterClientBoundaryTests : IAsyncLifetime
ClusterClientReceptionist.Get(_central).RegisterService(centralComm); ClusterClientReceptionist.Get(_central).RegisterService(centralComm);
// Node: the REAL comm actor, dialling central through the REAL production client factory. // Node: the REAL comm actor, dialling central through the REAL production client factory.
var nodeToCentral = _clientFactory.Create(_node, ReceptionistOf(_central)); var nodeToCentral = _clientFactory.Create(_node, "central", ReceptionistOf(_central));
_nodeComm = _node.ActorOf( _nodeComm = _node.ActorOf(
NodeCommunicationActor.Props(nodeToCentral), MeshPaths.NodeCommunicationName); NodeCommunicationActor.Props(nodeToCentral), MeshPaths.NodeCommunicationName);
ClusterClientReceptionist.Get(_node).RegisterService(_nodeComm); ClusterClientReceptionist.Get(_node).RegisterService(_nodeComm);
_centralClient = _clientFactory.Create(_central, ReceptionistOf(_node)); _centralClient = _clientFactory.Create(_central, "site-a", ReceptionistOf(_node));
} }
/// <inheritdoc /> /// <inheritdoc />
@@ -179,7 +183,7 @@ public sealed class MeshClusterClientBoundaryTests : IAsyncLifetime
var comm = deaf.ActorOf(NodeCommunicationActor.Props(null), MeshPaths.NodeCommunicationName); var comm = deaf.ActorOf(NodeCommunicationActor.Props(null), MeshPaths.NodeCommunicationName);
comm.ShouldNotBeNull(); comm.ShouldNotBeNull();
var client = _clientFactory.Create(_central, ReceptionistOf(deaf)); var client = _clientFactory.Create(_central, "deaf", ReceptionistOf(deaf));
var command = new RestartDriver("C1", "driver-1", "alice", Guid.NewGuid()); var command = new RestartDriver("C1", "driver-1", "alice", Guid.NewGuid());
var deadline = DateTime.UtcNow + Silence; var deadline = DateTime.UtcNow + Silence;
@@ -199,6 +203,146 @@ public sealed class MeshClusterClientBoundaryTests : IAsyncLifetime
} }
} }
[Fact]
public async Task One_dispatch_through_central_reaches_a_node_in_every_cluster_mesh()
{
// THE Phase 6 assertion at the transport level, driving the REAL CentralCommunicationActor.
// Two genuinely separate site meshes (site-A = _node, site-B created here), each its own
// single-member cluster with its own receptionist. Central must build ONE ClusterClient per
// cluster and fan SendToAll across them, so a single MeshCommand reaches BOTH meshes. A single
// fleet-wide client would reach only the one mesh whose receptionist answered — the exact
// failure this test is red-before-green against.
var siteB = StartSelfSeededSystem(withReceptionist: true);
try
{
await AwaitSingleMemberUpAsync(siteB);
RegisterNodeComm(siteB);
var inboxA = NewInbox();
_node.EventStream.Subscribe(_node.ActorOf(CaptureActor.Props(inboxA)), typeof(RestartDriver));
var inboxB = NewInbox();
siteB.EventStream.Subscribe(siteB.ActorOf(CaptureActor.Props(inboxB)), typeof(RestartDriver));
var dbFactory = SeedTwoClusterDb(SiteAddress(_node), SiteAddress(siteB));
var actor = _central.ActorOf(CentralCommunicationActor.Props(
dbFactory, ClusterClientOptions(), new DefaultMeshClusterClientFactory(), () => null));
var command = new RestartDriver("C1", "driver-1", "alice", Guid.NewGuid());
var deadline = DateTime.UtcNow + Arrival;
while (DateTime.UtcNow < deadline && !(inboxA.Task.IsCompleted && inboxB.Task.IsCompleted))
{
actor.Tell(new MeshCommand("deployments", command));
await Task.Delay(250, TestContext.Current.CancellationToken);
}
inboxA.Task.IsCompleted.ShouldBeTrue(
"site-A's mesh must receive the dispatch through its own per-cluster ClusterClient");
inboxB.Task.IsCompleted.ShouldBeTrue(
"site-B's mesh must receive the SAME dispatch through its own per-cluster ClusterClient; "
+ "a single fleet-wide client would have reached only one of the two meshes");
}
finally
{
await ShutdownAsync(siteB);
}
}
[Fact]
public async Task A_client_built_for_one_cluster_does_not_deliver_into_another_clusters_mesh()
{
// The partition proof for the fan-out: a client dialling ONLY site-A's contacts must land in
// site-A's mesh and NEVER cross into site-B's. This is why the per-cluster fan-out is correct
// rather than accidentally N x broadcasting — each client is scoped to its own mesh.
var siteB = StartSelfSeededSystem(withReceptionist: true);
try
{
await AwaitSingleMemberUpAsync(siteB);
RegisterNodeComm(siteB);
var inboxA = NewInbox();
_node.EventStream.Subscribe(_node.ActorOf(CaptureActor.Props(inboxA)), typeof(RestartDriver));
var inboxB = NewInbox();
siteB.EventStream.Subscribe(siteB.ActorOf(CaptureActor.Props(inboxB)), typeof(RestartDriver));
var clientA = _clientFactory.Create(_central, "site-a", ReceptionistOf(_node));
var command = new RestartDriver("C1", "driver-1", "alice", Guid.NewGuid());
// Keep sending for the full Silence window so any erroneous cross-delivery into site-B has
// every chance to arrive — a pass means the boundary is shut, not that the clock ran out.
var deadline = DateTime.UtcNow + Silence;
while (DateTime.UtcNow < deadline)
{
clientA.Tell(new ClusterClient.SendToAll(MeshPaths.NodeCommunication, command));
await Task.Delay(250, TestContext.Current.CancellationToken);
}
inboxA.Task.IsCompleted.ShouldBeTrue("site-A's client must deliver into site-A's mesh");
inboxB.Task.IsCompleted.ShouldBeFalse(
"a client built for site-A's contacts must NOT deliver into site-B's mesh — the "
+ "per-cluster fan-out must stay partitioned, one client per mesh");
}
finally
{
await ShutdownAsync(siteB);
}
}
/// <summary>Registers a real node-comm actor with a system's receptionist so central can reach it.</summary>
/// <param name="system">The site system to register into.</param>
private static void RegisterNodeComm(ActorSystem system)
{
var comm = system.ActorOf(NodeCommunicationActor.Props(null), MeshPaths.NodeCommunicationName);
ClusterClientReceptionist.Get(system).RegisterService(comm);
}
private static (string Host, int Port) SiteAddress(ActorSystem system)
{
var addr = AkkaCluster.Get(system).SelfAddress;
return (addr.Host!, addr.Port!.Value);
}
private static MeshTransportOptions ClusterClientOptions() => new()
{
Mode = MeshTransportOptions.ModeClusterClient,
CentralContactPoints = ["akka.tcp://otopcua@127.0.0.1:4053"],
ContactRefreshSeconds = 1,
};
private static IDbContextFactory<OtOpcUaConfigDbContext> SeedTwoClusterDb(
(string Host, int Port) siteA, (string Host, int Port) siteB)
{
var factory = new InMemoryConfigDbFactory(Guid.NewGuid().ToString("N"));
using var db = factory.CreateDbContext();
db.ClusterNodes.Add(NodeRow("SITE-A", "node-a", siteA));
db.ClusterNodes.Add(NodeRow("SITE-B", "node-b", siteB));
db.SaveChanges();
return factory;
}
private static ClusterNode NodeRow(string clusterId, string nodeId, (string Host, int Port) addr) => new()
{
NodeId = nodeId,
ClusterId = clusterId,
Host = addr.Host,
AkkaPort = addr.Port,
ApplicationUri = $"urn:{nodeId}",
Enabled = true,
MaintenanceMode = false,
CreatedBy = "test",
};
private sealed class InMemoryConfigDbFactory(string dbName)
: IDbContextFactory<OtOpcUaConfigDbContext>
{
public OtOpcUaConfigDbContext CreateDbContext()
{
var opts = new DbContextOptionsBuilder<OtOpcUaConfigDbContext>()
.UseInMemoryDatabase(dbName)
.Options;
return new OtOpcUaConfigDbContext(opts);
}
}
/// <summary> /// <summary>
/// Starts a single-member cluster on a dynamic port, optionally without the receptionist. /// Starts a single-member cluster on a dynamic port, optionally without the receptionist.
/// </summary> /// </summary>
@@ -193,6 +193,10 @@ public sealed class ServiceCollectionExtensionsTests
public NodeId LocalNode { get; } = NodeId.Parse("test-node"); public NodeId LocalNode { get; } = NodeId.Parse("test-node");
/// <summary>Gets the local roles.</summary> /// <summary>Gets the local roles.</summary>
public IReadOnlySet<string> LocalRoles { get; } = new HashSet<string>(["driver"]); public IReadOnlySet<string> LocalRoles { get; } = new HashSet<string>(["driver"]);
/// <summary>Gets the local node's cluster-scoped role. None configured in this fake.</summary>
public string? ClusterRole => null;
/// <summary>Gets the ClusterId extracted from <see cref="ClusterRole"/>. None configured in this fake.</summary>
public string? ClusterId => null;
/// <summary>Determines whether the local node has the specified role.</summary> /// <summary>Determines whether the local node has the specified role.</summary>
/// <param name="role">The role to check.</param> /// <param name="role">The role to check.</param>
public bool HasRole(string role) => LocalRoles.Contains(role); public bool HasRole(string role) => LocalRoles.Contains(role);