Merge feat/grpc-phase3-deletion: delete the Akka ClusterClient cross-mesh transport (Phase 3)
v2-ci / build (push) Successful in 5m43s
v2-ci / unit-tests (push) Failing after 17m18s

MeshTransport:Mode is now Dps | Grpc only; the retired ClusterClient value
fails validation loudly. CentralContactPoints, both receptionist
registrations, the akka.cluster.client HOCON block and the central/node
client machinery are deleted; SplitTopologyTransportValidator requires Grpc
on cluster-role nodes. MeshGrpcBoundaryTests replaces the old boundary test
and pins that no cross-mesh Akka association forms. Docs swept (incl. the
scadaproj index, pushed separately on its main).

Task 3.1 spec+code reviews closed (1 Important fixed: MeshPaths collapse);
Task 3.2 docs-accuracy review closed. DoD full suite: exit 0, 50/50 suites,
9326 passed / 0 failed on 8e25ed55.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
This commit is contained in:
Joseph Doherty
2026-08-06 16:57:05 -04:00
57 changed files with 960 additions and 1884 deletions
+65 -41
View File
@@ -371,8 +371,8 @@ Assert the **effective** cluster config off a running `ActorSystem` (as `SplitBr
- **The redundancy singleton is cluster-scoped and spawned per driver node.** `RedundancyStateActor` is now a `cluster-{ClusterId}`-scoped singleton (falls back to plain `driver` for a legacy node), so each pair elects and publishes its **own** Primary on its own mesh's DPS. **Two or more Primaries exist fleet-wide, one per pair — by design, not a bug.** ⚠️ **Wiring landmine (fixed `3a4ed8dd`, caught by the live gate):** the singleton's cluster-role *scope* is computed at `AddAkka`-configurator time, so it must come from `AkkaClusterOptions` (config) — **never** by resolving `IClusterRoleInfo` from the SP there. `ClusterRoleInfo` depends on the `ActorSystem`, and that lambda runs *while the ActorSystem is being built*, so resolving it recurses into ActorSystem construction and **stack-overflows every node at boot** (compiles clean; the rehome tests passed a hand-built fake straight into the extension, mocking around the composition-root line). Keep `WithOtOpcUaClusterRedundancySingleton`/`BuildClusterRedundancySingletonOptions` taking `AkkaClusterOptions`. - **The redundancy singleton is cluster-scoped and spawned per driver node.** `RedundancyStateActor` is now a `cluster-{ClusterId}`-scoped singleton (falls back to plain `driver` for a legacy node), so each pair elects and publishes its **own** Primary on its own mesh's DPS. **Two or more Primaries exist fleet-wide, one per pair — by design, not a bug.** ⚠️ **Wiring landmine (fixed `3a4ed8dd`, caught by the live gate):** the singleton's cluster-role *scope* is computed at `AddAkka`-configurator time, so it must come from `AkkaClusterOptions` (config) — **never** by resolving `IClusterRoleInfo` from the SP there. `ClusterRoleInfo` depends on the `ActorSystem`, and that lambda runs *while the ActorSystem is being built*, so resolving it recurses into ActorSystem construction and **stack-overflows every node at boot** (compiles clean; the rehome tests passed a hand-built fake straight into the extension, mocking around the composition-root line). Keep `WithOtOpcUaClusterRedundancySingleton`/`BuildClusterRedundancySingletonOptions` taking `AkkaClusterOptions`.
- `ClusterNodeAddressReconciler` is **own-cluster-scoped** — an admin node can only reconcile `ClusterNode` rows in its own cluster; it has no gossip visibility into another mesh. - `ClusterNodeAddressReconciler` is **own-cluster-scoped** — an admin node can only reconcile `ClusterNode` rows in its own cluster; it has no gossip visibility into another mesh.
- `CentralCommunicationActor` creates **one `ClusterClient` per application `Cluster`** and fans commands across them (closing the `TODO(Phase 6)` Phase 2 left behind), rather than one fleet-wide client. - `CentralCommunicationActor` dials each node directly over gRPC — one `NodeCommandService.Execute` call per enabled `ClusterNode` row carrying a `GrpcPort` — rather than fanning through a client/receptionist object. **Superseded by the [ClusterClient→gRPC migration](docs/plans/2026-08-06-clusterclient-to-grpc-design.md):** the Akka `ClusterClient` this bullet originally described (one per application `Cluster`, closing the `TODO(Phase 6)` Phase 2 left behind) is deleted; the per-node gRPC dial is its replacement, not an evolution of it.
- **`SplitTopologyTransportValidator` fails host start** on a node carrying a `cluster-{ClusterId}` role unless it also runs the split-safe transports: `MeshTransport:Mode=ClusterClient`, `Telemetry:Mode=Grpc` (driver), `TelemetryDial:Mode=Grpc` (admin). A DPS transport on a cluster-role node would try to gossip across a partition that no longer exists. - **`SplitTopologyTransportValidator` fails host start** on a node carrying a `cluster-{ClusterId}` role unless it also runs the split-safe transports: `MeshTransport:Mode=Grpc`, `Telemetry:Mode=Grpc` (driver), `TelemetryDial:Mode=Grpc` (admin). A DPS transport on a cluster-role node would try to gossip across a partition that no longer exists.
- **The compiled transport-mode defaults were deliberately NOT flipped** — `MeshTransport:Mode`/`Telemetry:Mode`/`TelemetryDial:Mode` all still default to `Dps`; a blast-radius assessment showed flipping the default breaks every integration test and appsettings profile that leaves the key unset. The validator + explicit per-deployment config is the guardrail, not a default flip. - **The compiled transport-mode defaults were deliberately NOT flipped** — `MeshTransport:Mode`/`Telemetry:Mode`/`TelemetryDial:Mode` all still default to `Dps`; a blast-radius assessment showed flipping the default breaks every integration test and appsettings profile that leaves the key unset. The validator + explicit per-deployment config is the guardrail, not a default flip.
- **Secrets replication is pair-local** — each pair replicates its own Akka DPS secrets within itself; there is no fleet-wide/cross-cluster sync after the split (sites have no SQL, so no SQL-hub, consistent with Phase 4). - **Secrets replication is pair-local** — each pair replicates its own Akka DPS secrets within itself; there is no fleet-wide/cross-cluster sync after the split (sites have no SQL, so no SQL-hub, consistent with Phase 4).
@@ -380,51 +380,74 @@ See `docs/Redundancy.md` §"Per-cluster meshes (Phase 6)", `docs/Configuration.m
## Mesh command transport (`MeshTransport`) ## Mesh command transport (`MeshTransport`)
Per-cluster mesh **Phase 2** added a second transport for the three central→node command channels **The Akka `ClusterClient` transport described in this section's earlier history is DELETED**
(`deployments`, `driver-control`, `alarm-commands`) and the `deployment-acks` reply, selected by (branch `feat/grpc-phase3-deletion`, [ClusterClient→gRPC migration](docs/plans/2026-08-06-clusterclient-to-grpc-design.md),
`MeshTransport:Mode`: `Dps` (the DistributedPubSub path this repo has always used, **the default**) Phase 3) — code, the `akka.cluster.client` HOCON block, the receptionist extension,
or `ClusterClient`, which does **not** require central and the node to share a gossip ring — the `MeshTransport:CentralContactPoints`, and the `ClusterClient` mode value itself, which now **fails
prerequisite for splitting the fleet into one mesh per application `Cluster`. Both `CentralCommunicationActor` validation loudly** at boot (`MeshTransportOptionsValidator` names the key and the allowed values)
(`/user/central-communication`) and `NodeCommunicationActor` (`/user/node-communication`) are spawned rather than silently falling back to `Dps`. What remains is exactly two transports, selected by
and registered with the `ClusterClientReceptionist` in **both** modes, so flipping the flag is a config `MeshTransport:Mode`:
change, not a redeploy. Things worth knowing before touching it:
- **`SendToAll`, never `Send`.** Today's DPS publish reaches every `DriverHostActor`; there is no - **`Dps`** (default) — DistributedPubSub, the path this repo has always used. **Pair-local only**:
ClusterId or node filter on the node side at all (scoping happens later, inside the artifact via it works only when central and the node share a gossip ring, which after Phase 6 means within one
`DeploymentArtifact.ResolveClusterScope`). `ClusterClient.Send` delivers to exactly **one** registered 2-node mesh.
actor — it would deploy to a single node while every other node silently kept its old configuration, - **`Grpc`** — unary gRPC calls on the `mesh_transport.v1` contract. **Required** on any node
and the deployment could still seal green. carrying a `cluster-{ClusterId}` role; `SplitTopologyTransportValidator` refuses to boot such a
- **RESOLVED by Phase 6 — one `ClusterClient` per application `Cluster`, not one fleet-wide.** When node on anything else.
this was written the fleet was still one mesh, so a receptionist served the whole cluster and
`SendToAll` reached every registered node-comm actor regardless of which node's address was dialled;
the `TODO(Phase 6)` in `CentralCommunicationActor.RebuildClient` asked for one client per application
`Cluster` instead. Phase 6 split the fleet into one mesh per `Cluster` and shipped exactly that — see
the [Redundancy](#redundancy) section's Phase 6 paragraph above. The `SendToAll`-within-a-mesh
behaviour below is otherwise unchanged; a `MaintenanceMode` node still isn't excluded from delivery
within its own mesh.
- **Inbound commands re-emit on the node's `EventStream`, not on their DPS topic** — DPS is mesh-wide and
central `SendToAll`s to every node, so a DPS re-publish would deliver N copies to every subscriber.
Node-side subscribers (`DriverHostActor`, `ScriptedAlarmHostActor`) accept both.
- **Comm actors are per-node, NOT singletons.** A client rotates across contact points and must find a live
comm actor at whichever node answers; `akka.cluster.client.receptionist.role` is empty for the same reason.
- **`buffer-size = 0` is a behaviour decision, not a tuning knob.** Akka's default buffers 1000 and replays
on reconnect — silently, on the client's own timer, at an unbounded remove from the dispatch. Do not raise
it to quiet a flaky test. **But do not over-claim what it buys:** the Phase 2 live gate observed a node
applying a deployment **44 s after it was sealed `TimedOut`** anyway, via `DriverHostActor.Bootstrap()`
replaying the orphan `Applying` row on restart. Zero buffering removes the silent replay, not every replay.
Contact points are node **addresses only** (`akka.tcp://<system>@<host>:<port>`); `/system/receptionist` is Central→node commands (`deployments`, `driver-control`, `alarm-commands`) ride `NodeCommandService.Execute`,
appended at construction and `MeshTransportOptionsValidator` refuses to start a node whose contact carries an served on each node's driver-role telemetry h2c listener (`Telemetry:GrpcListenPort`, the docker-dev
actor-path suffix. `redundancy-state` stays on DPS in both modes (bidirectional, and built from rig's `:4056` — it shares that listener with `TelemetryStreamGrpcService` rather than claiming a port
`Cluster.State`). See `docs/Configuration.md` §`MeshTransport`, `docs/Redundancy.md` §"Command transport", of its own). `CentralCommunicationActor` dials it **directly, one call per node** — the target set is
and `docs/plans/2026-07-22-mesh-phase2-clusterclient-transport.md`. every enabled, non-maintenance `ClusterNode` row carrying a `GrpcPort`, dialled at `Host`+`GrpcPort`
and refreshed every `MeshTransport:ContactRefreshSeconds`. There is no client/receptionist object and
no rotation: each command goes to the specific node dialled, which is what makes per-node delivery
provable — the old `ClusterClient.SendToAll`/`Send` distinction this section used to explain no longer
applies to either direction.
Node→central deploy acks ride `CentralCommandService.ReportApply`, served on central's admin-role
config-serve h2c listener (`ConfigServe:GrpcListenPort`, the rig's `:4055` — again sharing rather than
claiming a new port). `GrpcApplyAckSender` walks `MeshTransport:CentralGrpcEndpoints` in order and
stops at the first success.
Auth is a **per-cluster PSK bearer**, both directions: `MeshTransport:GrpcPsk` (node-side, one key per
application `Cluster`) and `MeshTransport:ClusterPsks` (central-side map, consulted before the Secrets
store). `MeshCommandAuthInterceptor` (node) and `CentralCommandAuthInterceptor` (central) fail closed
on every unresolvable case — constant-time compare, no key material logged. Production keys are
`${secret:OT-MESH-PSK-<clusterId>}`; the docker-dev rig supplies three literal dev keys via env
(`dev-mesh-psk-MAIN`/`-SITE-A`/`-SITE-B`).
Things that carried over unchanged from the `ClusterClient` era, because they were never
`ClusterClient`-specific to begin with:
- **Fire-and-forget, no retry, on both transports.** A failed gRPC dispatch is logged (naming the node
and the gRPC status) and dropped in the dispatch second. No queue, no ordering guarantee across
nodes — the deploy coordinator's apply deadline and a node's boot-time replay are the only recovery.
The old `buffer-size = 0` decision this section used to describe is gone with the HOCON block that
set it; the gRPC transport has nothing to buffer in the first place — a dial either lands or doesn't.
- **Inbound commands re-emit on the node's `EventStream`, not back onto DPS.** DPS is mesh-wide, so a
DPS re-publish would deliver N copies to every subscriber. Node-side subscribers (`DriverHostActor`,
`ScriptedAlarmHostActor`) accept a command from either transport identically.
- **Do not over-claim what fire-and-forget buys.** The Phase 2 live gate observed a node applying a
deployment **44 s after it was sealed `TimedOut`** anyway, via `DriverHostActor.Bootstrap()`
replaying the orphan `Applying` row on restart. That backstop is unaffected by the ClusterClient
deletion — it was always about the deploy path, not the transport.
`redundancy-state` stays on DPS in both modes (bidirectional, and built from `Cluster.State`, which
makes it genuinely mesh-bound). See `docs/Configuration.md` §`MeshTransport`, `docs/Redundancy.md`
§"Command transport", and the migration's [design doc](docs/plans/2026-08-06-clusterclient-to-grpc-design.md) /
[plan](docs/plans/2026-08-06-clusterclient-to-grpc.md) / [tracking doc](docs/plans/2026-08-06-clusterclient-to-grpc-tracking.md) /
[live gate](docs/plans/2026-08-06-clusterclient-to-grpc-live-gate.md) for the phase-by-phase history —
those are historical records, not updated further by this sweep. `docs/plans/2026-07-22-mesh-phase2-clusterclient-transport.md`
is the original Phase 2 design doc and is likewise left as a historical record of the transport this
section now describes as deleted.
**Phase 1 landed 2026-07-22 and changed a deploy-path behaviour.** `ConfigPublishCoordinator` sources its expected-ack set from **enabled `ClusterNode` rows**, not from `Akka.Cluster.State.Members` filtered by the `driver` role — central must be able to name a deployment's nodes without sharing a gossip ring with them. Consequences worth knowing before touching deploys: **Phase 1 landed 2026-07-22 and changed a deploy-path behaviour.** `ConfigPublishCoordinator` sources its expected-ack set from **enabled `ClusterNode` rows**, not from `Akka.Cluster.State.Members` filtered by the `driver` role — central must be able to name a deployment's nodes without sharing a gossip ring with them. Consequences worth knowing before touching deploys:
- **A configured node that is switched off now FAILS the deployment** at the apply deadline instead of letting it seal green without that node. The maintenance hatch is the **new `ClusterNode.MaintenanceMode` column**, NOT `Enabled = 0`: the live gate showed `Enabled = 0` is rejected outright by `DraftValidator.ValidateClusterTopology` (`ClusterEnabledNodeCountMismatch` — enabled count must equal `ServerCluster.NodeCount`), so on a 2-node pair — every cluster in the target topology — it can never be the hatch. `Enabled` = part of the declared topology; `MaintenanceMode` = do not expect it now. The deadline log names the silent nodes and points at the right flag. - **A configured node that is switched off now FAILS the deployment** at the apply deadline instead of letting it seal green without that node. The maintenance hatch is the **new `ClusterNode.MaintenanceMode` column**, NOT `Enabled = 0`: the live gate showed `Enabled = 0` is rejected outright by `DraftValidator.ValidateClusterTopology` (`ClusterEnabledNodeCountMismatch` — enabled count must equal `ServerCluster.NodeCount`), so on a 2-node pair — every cluster in the target topology — it can never be the hatch. `Enabled` = part of the declared topology; `MaintenanceMode` = do not expect it now. The deadline log names the silent nodes and points at the right flag.
- **Every `ClusterNode` row is now *defined* to be a driver node.** The DB has no per-node role column and deliberately did not gain one (it would drift from `Cluster:Roles`). Giving an admin-only node a `ClusterNode` row makes every deploy time out. - **Every `ClusterNode` row is now *defined* to be a driver node.** The DB has no per-node role column and deliberately did not gain one (it would drift from `Cluster:Roles`). Giving an admin-only node a `ClusterNode` row makes every deploy time out.
- **There is no such thing as a cluster-scoped deployment.** `Deployment` has no `ClusterId`, `ConfigComposer` always snapshots the whole DB, and `DeploymentArtifact.ResolveClusterScope` is *node-side* self-scoping of a fleet-wide artifact. The phase-1 plan asked for cluster-scope filtering of the ack set; it was dropped as describing a feature that does not exist. - **There is no such thing as a cluster-scoped deployment.** `Deployment` has no `ClusterId`, `ConfigComposer` always snapshots the whole DB, and `DeploymentArtifact.ResolveClusterScope` is *node-side* self-scoping of a fleet-wide artifact. The phase-1 plan asked for cluster-scope filtering of the ack set; it was dropped as describing a feature that does not exist.
- `ClusterNode` gained `AkkaPort` (NOT NULL, 4053) + `GrpcPort` (nullable) as **central's dial targets** for Phases 2/5. They duplicate the node's own `Cluster:Port` / `Cluster:PublicHostname` with no schema-level enforcement, so `ClusterNodeAddressReconcilerActor` (admin singleton) logs an Error on drift. **Phase 2 must revisit it** — once the meshes split, an admin node cannot see site members and every site row would report `EnabledRowNotInCluster` forever. - `ClusterNode` gained `AkkaPort` (NOT NULL, 4053) + `GrpcPort` (nullable) as **central's dial targets** for Phases 2/5. They duplicate the node's own `Cluster:Port` / `Cluster:PublicHostname` with no schema-level enforcement, so `ClusterNodeAddressReconcilerActor` (admin singleton) logs an Error on drift. **Phase 2 must revisit it** — once the meshes split, an admin node cannot see site members and every site row would report `EnabledRowNotInCluster` forever. **Resolved, and not the way this bullet originally expected:** the [ClusterClient→gRPC migration](docs/plans/2026-08-06-clusterclient-to-grpc-design.md) deleted the Akka `ClusterClient` transport that would have dialled `Host`+`AkkaPort`; the Phase 2 command dial now uses `Host`+`GrpcPort`, the same column Phase 5 telemetry already dialled — `AkkaPort` is Akka remoting only again.
## Config source (`ConfigSource` / `ConfigServe`) ## Config source (`ConfigSource` / `ConfigServe`)
@@ -488,8 +511,9 @@ touching it:
`Cluster.State`, drives ServiceLevel + the Primary gate — genuinely mesh-bound by design, not an `Cluster.State`, drives ServiceLevel + the Primary gate — genuinely mesh-bound by design, not an
observability panel); `fleet-status` is central-internal (`FleetStatusBroadcaster` builds it from observability panel); `fleet-status` is central-internal (`FleetStatusBroadcaster` builds it from
the admin node's own membership events, `Fleet.razor` polls the Config DB and ignores the feed — the admin node's own membership events, `Fleet.razor` polls the Config DB and ignores the feed —
never a node→central stream); `deployment-acks` already rides the Phase 2 `ClusterClient` never a node→central stream); `deployment-acks` already rides the Phase 2 gRPC command transport
transport as a command-plane reply, not telemetry. (`MeshTransport:Mode=Grpc`, `CentralCommandService.ReportApply`) as a command-plane reply, not
telemetry.
- **The node ALWAYS hosts and ALWAYS double-publishes, in both modes.** Every publish seam emits - **The node ALWAYS hosts and ALWAYS double-publishes, in both modes.** Every publish seam emits
into a node-local `ITelemetryLocalHub` **and** still publishes DPS, unconditionally; the gRPC into a node-local `ITelemetryLocalHub` **and** still publishes DPS, unconditionally; the gRPC
server binds whenever `Telemetry:GrpcListenPort > 0`, independent of `Telemetry:Mode`. Only server binds whenever `Telemetry:GrpcListenPort > 0`, independent of `Telemetry:Mode`. Only
+21 -72
View File
@@ -21,9 +21,10 @@
# #
# Since central no longer shares a gossip ring with the site meshes, it reaches # Since central no longer shares a gossip ring with the site meshes, it reaches
# them over the three explicit split-topology transports instead of gossip: # them over the three explicit split-topology transports instead of gossip:
# - Command plane: MeshTransport__Mode=ClusterClient (central dials each site # - Command plane: MeshTransport__Mode=Grpc (central dials each site node's
# mesh's ClusterClientReceptionist; DPS is gossip-only and # NodeCommandService on :4056; a node posts its deploy ack
# would not cross the mesh boundary at all now) # back to central's CentralCommandService on :4055. DPS is
# gossip-only and would not cross the mesh boundary at all)
# - Live telemetry: Telemetry__Mode=Grpc on every driver node (all six) + # - Live telemetry: Telemetry__Mode=Grpc on every driver node (all six) +
# TelemetryDial__Mode=Grpc on the admin pair (central-1/2) # TelemetryDial__Mode=Grpc on the admin pair (central-1/2)
# — the site node hosts the gRPC server, central dials in # — the site node hosts the gRPC server, central dials in
@@ -53,8 +54,7 @@
# data-plane endpoints (4842-4845) and authenticate those # data-plane endpoints (4842-4845) and authenticate those
# users against the same shared GLAuth (via the *ldap-env # users against the same shared GLAuth (via the *ldap-env
# anchor); central manages + deploys to them over the # anchor); central manages + deploys to them over the
# ClusterClient/gRPC/FetchAndCache transports above, not # Grpc/FetchAndCache transports above, not gossip.
# gossip.
# #
# Auth is real LDAP against the shared GLAuth on the Linux Docker host # Auth is real LDAP against the shared GLAuth on the Linux Docker host
# (10.100.0.35:3893, dc=zb,dc=local) — there is no LDAP container here. # (10.100.0.35:3893, dc=zb,dc=local) — there is no LDAP container here.
@@ -329,33 +329,22 @@ services:
Cluster__SeedNodes__1: "akka.tcp://otopcua@central-2:4053" Cluster__SeedNodes__1: "akka.tcp://otopcua@central-2:4053"
# Per-cluster mesh Phase 6: cluster-MAIN marks this node as a member of the MAIN mesh # Per-cluster mesh Phase 6: cluster-MAIN marks this node as a member of the MAIN mesh
# (RoleParser.ClusterRolePrefix = "cluster-"), read by SplitTopologyTransportValidator to # (RoleParser.ClusterRolePrefix = "cluster-"), read by SplitTopologyTransportValidator to
# require ClusterClient/Grpc transports, and by ClusterRoleInfo to scope redundancy + # require the mesh-crossing Grpc transports, and by ClusterRoleInfo to scope redundancy +
# ClusterNodeAddressReconcilerActor to ClusterId=MAIN. # ClusterNodeAddressReconcilerActor to ClusterId=MAIN.
Cluster__Roles__0: "admin" Cluster__Roles__0: "admin"
Cluster__Roles__1: "driver" Cluster__Roles__1: "driver"
Cluster__Roles__2: "cluster-MAIN" Cluster__Roles__2: "cluster-MAIN"
# Mesh command transport (per-cluster mesh Phase 2, made mandatory by Phase 6). Central no # Mesh command transport (per-cluster mesh Phase 2, made mandatory by Phase 6). Central no
# longer shares a gossip ring with the site meshes, so DPS (gossip-only) cannot reach them — # longer shares a gossip ring with the site meshes, so DPS (gossip-only) cannot reach them —
# ClusterClient is the only transport that can. SplitTopologyTransportValidator refuses to # Grpc is the only transport that can. SplitTopologyTransportValidator refuses to start any
# start any node carrying a cluster-* role that is left on Dps, so this is fixed, not an # node carrying a cluster-* role that is left on Dps, so this is fixed, not an
# env-var-overridable default, on every node in this file. # env-var-overridable default, on every node in this file.
#
# Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
# time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
# actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
# silence. Both central nodes are listed so a driver node's client can rotate to whichever one
# is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Grpc}" MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Grpc}"
MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053" # gRPC command transport (docs/plans/2026-08-06-clusterclient-to-grpc-design.md). GrpcPsk is
MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053" # this node's OWN per-cluster key (MAIN, since central is cluster-MAIN); CentralGrpcEndpoints
# gRPC command transport (Phase 0 of the ClusterClient→gRPC migration, docs/plans/2026-08-06- # is where a node posts its deploy ack. ClusterPsks is CENTRAL-SIDE ONLY (IMeshPskProvider's
# clusterclient-to-grpc-design.md). SHIPPED DARK: Mode above stays ClusterClient, so none of # override map, consulted before the Secrets store) — one entry per cluster central dials, so
# this is dialled or served yet — only MeshTransportOptionsValidator's Grpc-mode rules read # only the central nodes carry it.
# it. GrpcPsk is this node's OWN per-cluster key (MAIN, since central is cluster-MAIN);
# CentralGrpcEndpoints is the gRPC counterpart of CentralContactPoints above. ClusterPsks is
# CENTRAL-SIDE ONLY (IMeshPskProvider's override map, consulted before the Secrets store) —
# one entry per cluster central will dial once the switch flips, so only the central nodes
# carry it.
MeshTransport__GrpcPsk: "dev-mesh-psk-MAIN" MeshTransport__GrpcPsk: "dev-mesh-psk-MAIN"
MeshTransport__CentralGrpcEndpoints__0: "http://central-1:4055" MeshTransport__CentralGrpcEndpoints__0: "http://central-1:4055"
MeshTransport__CentralGrpcEndpoints__1: "http://central-2:4055" MeshTransport__CentralGrpcEndpoints__1: "http://central-2:4055"
@@ -478,16 +467,8 @@ services:
Cluster__Roles__1: "driver" Cluster__Roles__1: "driver"
Cluster__Roles__2: "cluster-MAIN" Cluster__Roles__2: "cluster-MAIN"
# Mesh command transport (per-cluster mesh Phase 2, made mandatory by Phase 6) — see central-1. # Mesh command transport (per-cluster mesh Phase 2, made mandatory by Phase 6) — see central-1.
#
# Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
# time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
# actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
# silence. Both central nodes are listed so a driver node's client can rotate to whichever one
# is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Grpc}" MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Grpc}"
MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053" # gRPC command transport — see central-1.
MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
# gRPC command transport (Phase 0) — see central-1. SHIPPED DARK (Mode stays ClusterClient).
# ClusterPsks is central-side only; both central nodes carry the same three-cluster map. # ClusterPsks is central-side only; both central nodes carry the same three-cluster map.
MeshTransport__GrpcPsk: "dev-mesh-psk-MAIN" MeshTransport__GrpcPsk: "dev-mesh-psk-MAIN"
MeshTransport__CentralGrpcEndpoints__0: "http://central-1:4055" MeshTransport__CentralGrpcEndpoints__0: "http://central-1:4055"
@@ -549,7 +530,7 @@ services:
# ── Site A cluster (2-node driver-only) ───────────────────────────────────── # ── Site A cluster (2-node driver-only) ─────────────────────────────────────
# Per-cluster mesh Phase 6: site-a-1 and site-a-2 form their OWN 2-node "SITE-A" Akka mesh, # Per-cluster mesh Phase 6: site-a-1 and site-a-2 form their OWN 2-node "SITE-A" Akka mesh,
# separate from MAIN and SITE-B — self-first seeded WITHIN the pair, never off central. No UI, # separate from MAIN and SITE-B — self-first seeded WITHIN the pair, never off central. No UI,
# no user auth; central manages + deploys to this mesh over ClusterClient/gRPC/FetchAndCache, # no user auth; central manages + deploys to this mesh over Grpc/FetchAndCache,
# not gossip (central and site-a share no seed list, so they never form one ring). # not gossip (central and site-a share no seed list, so they never form one ring).
site-a-1: site-a-1:
@@ -592,7 +573,7 @@ services:
Telemetry__Mode: "Grpc" Telemetry__Mode: "Grpc"
Telemetry__ApiKey: "telemetry-docker-dev-key" Telemetry__ApiKey: "telemetry-docker-dev-key"
# Per-cluster mesh Phase 6: site-a-1 seeds its OWN "SITE-A" mesh (itself first, site-a-2 # Per-cluster mesh Phase 6: site-a-1 seeds its OWN "SITE-A" mesh (itself first, site-a-2
# second) — it no longer seeds off central-1. Central is reached over ClusterClient/gRPC/ # second) — it no longer seeds off central-1. Central is reached over Grpc/
# FetchAndCache, never gossip; central and site-a share no seed list. # FetchAndCache, never gossip; central and site-a share no seed list.
Cluster__SeedNodes__0: "akka.tcp://otopcua@site-a-1:4053" Cluster__SeedNodes__0: "akka.tcp://otopcua@site-a-1:4053"
Cluster__SeedNodes__1: "akka.tcp://otopcua@site-a-2:4053" Cluster__SeedNodes__1: "akka.tcp://otopcua@site-a-2:4053"
@@ -605,20 +586,12 @@ services:
Cluster__Roles__0: "driver" Cluster__Roles__0: "driver"
Cluster__Roles__1: "cluster-SITE-A" Cluster__Roles__1: "cluster-SITE-A"
# Mesh command transport (per-cluster mesh Phase 2, made mandatory by Phase 6) — see # Mesh command transport (per-cluster mesh Phase 2, made mandatory by Phase 6) — see
# central-1/central-2. Central dials this mesh's ClusterClientReceptionist via the contact # central-1/central-2. Central dials this node's own NodeCommandService; there is no gossip
# points below; there is no gossip path anymore. # path anymore.
#
# Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
# time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
# actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
# silence. Both central nodes are listed so a driver node's client can rotate to whichever one
# is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Grpc}" MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Grpc}"
MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053" # gRPC command transport — see central-1.
MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
# gRPC command transport (Phase 0) — see central-1. SHIPPED DARK (Mode stays ClusterClient).
# GrpcPsk is this node's own key for its cluster (SITE-A); CentralGrpcEndpoints is the gRPC # GrpcPsk is this node's own key for its cluster (SITE-A); CentralGrpcEndpoints is the gRPC
# counterpart of CentralContactPoints above. No ClusterPsks here — that map is central-side # where this node posts its deploy ack. No ClusterPsks here — that map is central-side
# only. # only.
MeshTransport__GrpcPsk: "dev-mesh-psk-SITE-A" MeshTransport__GrpcPsk: "dev-mesh-psk-SITE-A"
MeshTransport__CentralGrpcEndpoints__0: "http://central-1:4055" MeshTransport__CentralGrpcEndpoints__0: "http://central-1:4055"
@@ -721,15 +694,7 @@ services:
Cluster__Roles__1: "cluster-SITE-A" Cluster__Roles__1: "cluster-SITE-A"
# Mesh command transport (per-cluster mesh Phase 2, made mandatory by Phase 6) — see # Mesh command transport (per-cluster mesh Phase 2, made mandatory by Phase 6) — see
# central-1/site-a-1. # central-1/site-a-1.
#
# Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
# time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
# actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
# silence. Both central nodes are listed so a driver node's client can rotate to whichever one
# is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Grpc}" MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Grpc}"
MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
# gRPC command transport (Phase 0) — see central-1/site-a-1. SHIPPED DARK. Same SITE-A key as # gRPC command transport (Phase 0) — see central-1/site-a-1. SHIPPED DARK. Same SITE-A key as
# site-a-1 (must be byte-identical — both nodes authenticate against the same cluster's PSK). # site-a-1 (must be byte-identical — both nodes authenticate against the same cluster's PSK).
MeshTransport__GrpcPsk: "dev-mesh-psk-SITE-A" MeshTransport__GrpcPsk: "dev-mesh-psk-SITE-A"
@@ -814,17 +779,9 @@ services:
Cluster__Roles__1: "cluster-SITE-B" Cluster__Roles__1: "cluster-SITE-B"
# Mesh command transport (per-cluster mesh Phase 2, made mandatory by Phase 6) — see # Mesh command transport (per-cluster mesh Phase 2, made mandatory by Phase 6) — see
# central-1/site-a-1. # central-1/site-a-1.
#
# Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
# time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
# actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
# silence. Both central nodes are listed so a driver node's client can rotate to whichever one
# is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Grpc}" MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Grpc}"
MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
# gRPC command transport (Phase 0) — see central-1. SHIPPED DARK. GrpcPsk is this node's own # gRPC command transport (Phase 0) — see central-1. SHIPPED DARK. GrpcPsk is this node's own
# key for its cluster (SITE-B); CentralGrpcEndpoints mirrors CentralContactPoints above. No # key for its cluster (SITE-B); CentralGrpcEndpoints is where it posts its deploy ack. No
# ClusterPsks here — that map is central-side only. # ClusterPsks here — that map is central-side only.
MeshTransport__GrpcPsk: "dev-mesh-psk-SITE-B" MeshTransport__GrpcPsk: "dev-mesh-psk-SITE-B"
MeshTransport__CentralGrpcEndpoints__0: "http://central-1:4055" MeshTransport__CentralGrpcEndpoints__0: "http://central-1:4055"
@@ -892,15 +849,7 @@ services:
Cluster__Roles__1: "cluster-SITE-B" Cluster__Roles__1: "cluster-SITE-B"
# Mesh command transport (per-cluster mesh Phase 2, made mandatory by Phase 6) — see # Mesh command transport (per-cluster mesh Phase 2, made mandatory by Phase 6) — see
# central-1/site-a-1. # central-1/site-a-1.
#
# Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
# time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
# actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
# silence. Both central nodes are listed so a driver node's client can rotate to whichever one
# is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Grpc}" MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Grpc}"
MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
# gRPC command transport (Phase 0) — see central-1/site-b-1. SHIPPED DARK. Same SITE-B key as # gRPC command transport (Phase 0) — see central-1/site-b-1. SHIPPED DARK. Same SITE-B key as
# site-b-1 (must be byte-identical). # site-b-1 (must be byte-identical).
MeshTransport__GrpcPsk: "dev-mesh-psk-SITE-B" MeshTransport__GrpcPsk: "dev-mesh-psk-SITE-B"
+16 -23
View File
@@ -142,7 +142,7 @@ The checked-in `appsettings*.json` files are deliberately thin: they carry only
- **`SplitTopologyTransportValidator`** (`src/Core/ZB.MOM.WW.OtOpcUa.Cluster/`, wired at - **`SplitTopologyTransportValidator`** (`src/Core/ZB.MOM.WW.OtOpcUa.Cluster/`, wired at
`ValidateOnStart` alongside the other `Cluster`-family validators) fails host start on any node `ValidateOnStart` alongside the other `Cluster`-family validators) fails host start on any node
carrying a `cluster-{ClusterId}` role unless it is also configured for the split-safe transports: carrying a `cluster-{ClusterId}` role unless it is also configured for the split-safe transports:
`MeshTransport:Mode = ClusterClient`, `Telemetry:Mode = Grpc` on a driver node, `TelemetryDial:Mode = `MeshTransport:Mode = Grpc`, `Telemetry:Mode = Grpc` on a driver node, `TelemetryDial:Mode =
Grpc` on an admin node. A DPS-mode transport on a cluster-role node would try to gossip across a Grpc` on an admin node. A DPS-mode transport on a cluster-role node would try to gossip across a
partition boundary that no longer exists, so the validator fails closed rather than let commands or partition boundary that no longer exists, so the validator fails closed rather than let commands or
telemetry silently vanish. A legacy node with no `cluster-{ClusterId}` role is exempt and may stay on telemetry silently vanish. A legacy node with no `cluster-{ClusterId}` role is exempt and may stay on
@@ -155,45 +155,38 @@ The checked-in `appsettings*.json` files are deliberately thin: they carry only
actually carries a `cluster-{ClusterId}` role must set the split-safe modes explicitly. actually carries a `cluster-{ClusterId}` role must set the split-safe modes explicitly.
- See [`Redundancy.md` § Per-cluster meshes (Phase 6)](Redundancy.md#per-cluster-meshes-phase-6) for the - See [`Redundancy.md` § Per-cluster meshes (Phase 6)](Redundancy.md#per-cluster-meshes-phase-6) for the
full runtime picture (one Primary per pair, the cluster-scoped `RedundancyStateActor` singleton, full runtime picture (one Primary per pair, the cluster-scoped `RedundancyStateActor` singleton,
`ClusterNodeAddressReconciler` own-cluster-scoping, and central's one-`ClusterClient`-per-`Cluster` `ClusterNodeAddressReconciler` own-cluster-scoping, and central's per-node gRPC command dial —
fan-out). one `NodeCommandService.Execute` call per enabled `ClusterNode` row carrying a `GrpcPort`).
### `MeshTransport` (central↔node command transport) ### `MeshTransport` (central↔node command transport)
- **Purpose:** selects how central's three command channels (`deployments`, `driver-control`, `alarm-commands`) and the node's `deployment-acks` reply reach the other side — over the mesh-wide DistributedPubSub, or over an Akka `ClusterClient` that does **not** require central and the node to share a gossip ring. The second option is what [per-cluster mesh](plans/2026-07-21-per-cluster-mesh-design.md) Phase 6 needs once the meshes split. - **Purpose:** selects how central's three command channels (`deployments`, `driver-control`, `alarm-commands`) and the node's `deployment-acks` reply reach the other side — over the mesh-wide DistributedPubSub (pair-local only), or over unary gRPC calls that do **not** require central and the node to share a gossip ring. The second option is what [per-cluster mesh](plans/2026-07-21-per-cluster-mesh-design.md) Phase 6 needs, now that the meshes are split. The Akka `ClusterClient` transport that used to be the second option here is **deleted** ([ClusterClient→gRPC migration](plans/2026-08-06-clusterclient-to-grpc-design.md), Phase 3) — `Grpc` is its replacement, not a third option alongside it.
- **Options class:** `MeshTransportOptions` (`SectionName = "MeshTransport"`) — `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/MeshTransportOptions.cs`. Bound and validated by `AddOtOpcUaCluster(config)`; `MeshTransportOptionsValidator` runs at `ValidateOnStart`. - **Options class:** `MeshTransportOptions` (`SectionName = "MeshTransport"`) — `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/MeshTransportOptions.cs`. Bound and validated by `AddOtOpcUaCluster(config)`; `MeshTransportOptionsValidator` runs at `ValidateOnStart`.
| Key | Type | Default | Meaning | | Key | Type | Default | Meaning |
|---|---|---|---| |---|---|---|---|
| `Mode` | string | `Dps` | `Dps` = DistributedPubSub (the transport this repo has always used). `ClusterClient` = the mesh boundary. `Grpc` = the `mesh_transport.v1` gRPC contract — **shipped dark, cutover pending** (see below). Any other value **refuses to start**. | | `Mode` | string | `Dps` | `Dps` = DistributedPubSub, pair-local only — cannot cross an application-`Cluster` boundary. `Grpc` = the `mesh_transport.v1` gRPC contract (below), required on any node carrying a `cluster-{ClusterId}` role. Any other value **refuses to start**, including the retired `ClusterClient` value — a node still configured for it is refused at boot rather than silently demoted to `Dps`, which on a split-topology node would deliver nothing at all. |
| `CentralContactPoints` | string[] | `[]` | Node-side only: central node **addresses** (`akka.tcp://<system>@<host>:<port>`). Required under `ClusterClient`, ignored under `Dps`/`Grpc`. | | `ContactRefreshSeconds` | int | `60` | Central-side only: how often central rebuilds its node dispatch set — the gRPC dial targets and the known-cluster PSK gate — from the `ClusterNode` rows. Runs in every mode; under `Dps` it still keeps the inbound-ack endpoint's known-cluster gate current even though nothing dials out. |
| `ContactRefreshSeconds` | int | `60` | Central-side only: how often central rebuilds its ClusterClient contact set from the `ClusterNode` rows. |
**Addresses only — no actor-path suffix.** `/system/receptionist` is appended at construction time; a contact carrying `/user/` or `/system/` is rejected at startup, because a doubled suffix produces a path that resolves to nothing and drops every command in silence. **`Grpc` mode: central dials each node, a node posts its ack to central.** `CentralCommunicationActor` fans one `NodeCommandService.Execute` call per node — the target set is every enabled, non-maintenance `ClusterNode` row that carries a `GrpcPort` (dialled at `Host`+`GrpcPort`), rebuilt every `ContactRefreshSeconds`. There is no receptionist and no single fleet-wide (or per-cluster) client object: it is a plain per-node dial. A node posts its `ApplyAck` with `GrpcApplyAckSender`, which walks `CentralGrpcEndpoints` in order and stops at the first success — an availability list, not a fan-out list, so central normally sees one ack per node per dispatch (the walk as a whole is at-least-once on a slow-then-retried endpoint; `ConfigPublishCoordinator.HandleAck` absorbs a duplicate idempotently). Both directions are **fire-and-forget, with no retry**: a failed dispatch is logged (naming the node and the gRPC status) and dropped in the dispatch second — the deploy path's 2-minute apply deadline and a node's boot-time replay (`DriverHostActor.Bootstrap()` re-applying an orphan `Applying` row) are the only recovery mechanisms, exactly as they were under `ClusterClient`.
**It is a dark switch, and both sides are always wired.** The comm actors are spawned and registered with the `ClusterClientReceptionist` on every node in *both* modes, so flipping `Mode` is a config change rather than a redeploy. Under `Dps` the registration is simply idle.
**Central's contact set comes from the enabled, non-maintenance `ClusterNode` rows** — the same source Phase 1 gave the deploy ack set. A node in `MaintenanceMode` is not dialled, but note that **the contact set does not scope delivery**: a receptionist serves its whole cluster, so `SendToAll` still reaches every registered node exactly as today's DPS broadcast does. The filter is about which receptionists are worth dialling, not about who receives.
**Buffering is off by design** (`akka.cluster.client.buffer-size = 0`, set in the shipped `akka.conf`). If no receptionist is known, the command is dropped rather than queued. Akka's default of 1000 would replay a `DispatchDeployment` on reconnect — silently, on the client's own timer, at an unbounded remove from the dispatch. Do not "fix" it back to quiet a flaky test.
> **What this does *not* buy you.** It does not guarantee a node never applies a deployment the DB records as `TimedOut`. A node that missed a dispatch keeps the orphan `Applying` row the coordinator pre-created, and `DriverHostActor.Bootstrap()` replays that row on the node's next restart — live-observed applying a deployment 44 s after it was sealed `TimedOut` (see the [Phase 2 live gate](plans/2026-07-22-mesh-phase2-live-gate.md), finding B). The boot replay is the better-behaved of the two — deliberate, Warning-logged, and bounded to a restart — but it is still a replay. `buffer-size = 0` removes the *silent, timer-driven* one only.
On the docker-dev rig every node carries these keys already; flip the whole rig with `OTOPCUA_MESH_MODE=ClusterClient` at `docker compose up`.
**`Grpc` mode — shipped dark, cutover pending** ([ClusterClient→gRPC migration design](plans/2026-08-06-clusterclient-to-grpc-design.md)). A third `Mode` value replaces `ClusterClient` with unary gRPC calls on the `mesh_transport.v1` contract: central dials each node's `NodeCommandService`, a node posts its deployment ack to central's `CentralCommandService`. Unlike `ClusterClient`, the command plane is authenticated end to end (`GrpcPsk` below) rather than relying on the mesh boundary alone. As of Phase 0 only the contract, DTO mapper, options and the two auth interceptors exist — the compiled default and the docker-dev rig both stay on `ClusterClient`, and setting `Mode=Grpc` today validates the keys below but nothing yet dials or serves over the contract.
| Key | Type | Default | Meaning | | Key | Type | Default | Meaning |
|---|---|---|---| |---|---|---|---|
| `GrpcPsk` | string? | `null` (unset) | Node-side only, required under `Grpc` on a driver-role node: this node's pre-shared key for its own `Cluster`, presented as the bearer on outbound acks and required on inbound `NodeCommandService` calls. Production form is `${secret:OT-MESH-PSK-<clusterId>}`; supply a literal only on a rig, via `MeshTransport__GrpcPsk` — never commit one. | | `GrpcPsk` | string? | `null` (unset) | Node-side only, required under `Grpc` on a driver-role node: this node's pre-shared key for its own `Cluster`, presented as the bearer on outbound acks and required on inbound `NodeCommandService` calls. Production form is `${secret:OT-MESH-PSK-<clusterId>}`; supply a literal only on a rig, via `MeshTransport__GrpcPsk` — never commit one. |
| `CentralGrpcEndpoints` | string[] | `[]` | Node-side only, required under `Grpc` on a driver-role node: full gRPC base URLs (`scheme://host:port`, e.g. `http://central-1:4055` for h2c) of the central pair's `CentralCommandService`, tried in order, first success wins. The `Grpc` counterpart of `CentralContactPoints`, and replaces it — a node on `Grpc` never reads the Akka contact points. | | `CentralGrpcEndpoints` | string[] | `[]` | Node-side only, required under `Grpc` on a driver-role node: full gRPC base URLs (`scheme://host:port`, e.g. `http://central-1:4055` for h2c) of the central pair's `CentralCommandService`, tried in order, first success wins. There is no Akka-side counterpart any more — the old `MeshTransport:CentralContactPoints` (the `ClusterClient` contact-point list) is deleted along with `ClusterClient` itself, not merely superseded. |
| `DispatchTimeoutSeconds` | int | `10` | Central-side per-node deadline for one `NodeCommandService.Execute` call. Must be positive under `Grpc`; validated on driver-role nodes (covers the central pair, which is always fused). | | `DispatchTimeoutSeconds` | int | `10` | Central-side per-node deadline for one `NodeCommandService.Execute` call. Must be positive under `Grpc`; validated on driver-role nodes (covers the central pair, which is always fused). |
| `AckTimeoutSeconds` | int | `10` | Node-side per-endpoint deadline for one `CentralCommandService.ReportApply` call. Must be positive under `Grpc`. Applied per endpoint walked in `CentralGrpcEndpoints`, not per ack, so a dead first endpoint costs this deadline before the second is tried. | | `AckTimeoutSeconds` | int | `10` | Node-side per-endpoint deadline for one `CentralCommandService.ReportApply` call. Must be positive under `Grpc`. Applied per endpoint walked in `CentralGrpcEndpoints`, not per ack, so a dead first endpoint costs this deadline before the second is tried. |
| `ClusterPsks` | `Dictionary<string,string>` | `{}` (case-insensitive keys) | **Central-side only** override map, keyed by `Cluster` id: `IMeshPskProvider` consults this map *before* the Secrets store when central builds a channel to (or verifies an ack from) a given cluster. Production leaves this empty and resolves `OT-MESH-PSK-{clusterId}` from Secrets instead; the map exists so the docker-dev rig and integration tests can supply literals via the environment (`MeshTransport__ClusterPsks__SITE-A`) without standing up a KEK. | | `ClusterPsks` | `Dictionary<string,string>` | `{}` (case-insensitive keys) | **Central-side only** override map, keyed by `Cluster` id: `IMeshPskProvider` consults this map *before* the Secrets store when central builds a channel to (or verifies an ack from) a given cluster. Production leaves this empty and resolves `OT-MESH-PSK-{clusterId}` from Secrets instead; the map exists so the docker-dev rig and integration tests can supply literals via the environment (`MeshTransport__ClusterPsks__SITE-A`) without standing up a KEK. |
**Auth is per-cluster PSK bearer, both directions, fail-closed.** `GrpcPsk` (node-side) and `ClusterPsks` (central-side) are two ends of the same key, one per application `Cluster` — not a fleet-wide shared bearer, because this channel carries `RestartDriver`, `AlarmCommand` and deployment dispatch, and a single fleet key would let one compromised site node command every other site or forge acks fleet-wide. Both `MeshCommandAuthInterceptor` (node-side, gates `NodeCommandService`) and `CentralCommandAuthInterceptor` (central-side, gates `CentralCommandService`) reject every call when no key is configured for the relevant cluster, compare in constant time, and never log key material. **PSK seeding:** production form is a secret reference, `${secret:OT-MESH-PSK-<clusterId>}` (e.g. `${secret:OT-MESH-PSK-SITE-A}`), resolved by the Secrets store so it works on a driver-only node with no ConfigDb; the docker-dev rig instead supplies three literal dev keys via environment — `dev-mesh-psk-MAIN`, `dev-mesh-psk-SITE-A`, `dev-mesh-psk-SITE-B` — one per cluster, node-side via `MeshTransport__GrpcPsk` and central-side via `MeshTransport__ClusterPsks__<clusterId>`. Never commit a literal key.
**`ClusterPsks` / `OT-MESH-PSK-<clusterId>` and `SecretName`'s allow-list.** `SecretName` (`ZB.MOM.WW.Secrets.Abstractions`) lowercases the secret name and restricts it to `[a-z0-9._/-]` — a `Cluster` id that carries any character outside that set cannot be expressed as `OT-MESH-PSK-<clusterId>` at all, so `MeshPskProvider.TryGetPsk` fails closed for that cluster on every call that reaches the store (`SecretName`'s constructor throws, `MeshPskProvider.ResolveFromStore` catches it, logs a warning, and returns no key) — `MeshTransport:ClusterPsks` is the only path that still works for such an id. The rig's three ids (`MAIN`, `SITE-A`, `SITE-B`) are all within the allow-list and resolve normally either way. **`ClusterPsks` / `OT-MESH-PSK-<clusterId>` and `SecretName`'s allow-list.** `SecretName` (`ZB.MOM.WW.Secrets.Abstractions`) lowercases the secret name and restricts it to `[a-z0-9._/-]` — a `Cluster` id that carries any character outside that set cannot be expressed as `OT-MESH-PSK-<clusterId>` at all, so `MeshPskProvider.TryGetPsk` fails closed for that cluster on every call that reaches the store (`SecretName`'s constructor throws, `MeshPskProvider.ResolveFromStore` catches it, logs a warning, and returns no key) — `MeshTransport:ClusterPsks` is the only path that still works for such an id. The rig's three ids (`MAIN`, `SITE-A`, `SITE-B`) are all within the allow-list and resolve normally either way.
The `Grpc` mode's node-side comm actors and central-side services are not yet wired to `Mode` — Phase 0 lands only the contract, mapper, options and the two auth interceptors (`MeshCommandAuthInterceptor` node-side, `CentralCommandAuthInterceptor` central-side), both dark. See the [design doc](plans/2026-08-06-clusterclient-to-grpc-design.md) and [tracking doc](plans/2026-08-06-clusterclient-to-grpc-tracking.md) for the remaining phases. **Both command-plane gRPC services ride an existing listener — neither claims a port of its own.** `NodeCommandService` is mapped alongside `TelemetryStreamGrpcService` on the driver-role node's `Telemetry:GrpcListenPort` listener (the docker-dev rig's `:4056`) whenever that port is bound, deliberately NOT gated on `MeshTransport:Mode` — mapping it under `Dps` costs nothing (nothing dials it), whereas gating it would mean a running fleet needs a central restart before any node could ack after flipping the mode. `CentralCommandService` (the node→central ack ingress) is mapped alongside `DeploymentArtifactService` on the admin-role node's `ConfigServe:GrpcListenPort` listener (the rig's `:4055`), for the same reason — see [`docs/Telemetry.md`](Telemetry.md) for the shared driver-role listener and [§ `ConfigSource`/`ConfigServe`](#configsource--configserve-config-fetch-and-cache) for the shared admin-role one.
On the docker-dev rig every node carries these keys already; the rig's compose default is `Grpc` (`OTOPCUA_MESH_MODE:-Grpc`, cutover commit `caf2eec6`) — the compiled class default in `MeshTransportOptions` stays `Dps`, so an unconfigured deployment outside the rig still needs `Mode=Grpc` set explicitly. Every rig node carries a `cluster-{ClusterId}` role (Phase 6), so `OTOPCUA_MESH_MODE=Dps` is not a usable override on this rig as shipped — `SplitTopologyTransportValidator` refuses to boot any of them on `Dps`.
See the [ClusterClient→gRPC migration design](plans/2026-08-06-clusterclient-to-grpc-design.md), [implementation plan](plans/2026-08-06-clusterclient-to-grpc.md), [tracking doc](plans/2026-08-06-clusterclient-to-grpc-tracking.md) and [live gate](plans/2026-08-06-clusterclient-to-grpc-live-gate.md) for the migration's phase-by-phase history — those are historical records of the cutover and are not updated further by this sweep.
### `ConfigSource` / `ConfigServe` (config fetch-and-cache) ### `ConfigSource` / `ConfigServe` (config fetch-and-cache)
@@ -284,7 +277,7 @@ non-empty `ApiKey`** — fail-closed, same as `ConfigServe`/`LocalDb:Replication
`redundancy-state` (pair-local, built from `Cluster.State`, stays on DPS in both modes — see `redundancy-state` (pair-local, built from `Cluster.State`, stays on DPS in both modes — see
[`Redundancy.md`](Redundancy.md#command-transport-centralnode)), `fleet-status` (central-internal; [`Redundancy.md`](Redundancy.md#command-transport-centralnode)), `fleet-status` (central-internal;
`Fleet.razor` polls the Config DB, not any stream), and `deployment-acks` (already rides the Phase 2 `Fleet.razor` polls the Config DB, not any stream), and `deployment-acks` (already rides the Phase 2
`ClusterClient` transport as a command-plane reply). See gRPC command transport, `CentralCommandService.ReportApply`, as a command-plane reply). See
[`docs/Telemetry.md`](Telemetry.md) for the full architecture, the reconnect/generation-stamped [`docs/Telemetry.md`](Telemetry.md) for the full architecture, the reconnect/generation-stamped
dialer story, and the rationale for the node-hosts/central-dials inversion. dialer story, and the rationale for the node-hosts/central-dials inversion.
+26 -16
View File
@@ -139,13 +139,16 @@ Consequences:
- **`ClusterNodeAddressReconciler` is own-cluster-scoped** — an admin node can only reconcile - **`ClusterNodeAddressReconciler` is own-cluster-scoped** — an admin node can only reconcile
`ClusterNode` rows belonging to its own cluster; it has no gossip visibility into another mesh to `ClusterNode` rows belonging to its own cluster; it has no gossip visibility into another mesh to
reconcile against. reconcile against.
- **`CentralCommunicationActor` creates one `ClusterClient` per application `Cluster`** and fans - **`CentralCommunicationActor` dials each node directly over gRPC** — one `NodeCommandService.Execute`
commands across them, rather than one fleet-wide client. Central reaches a site mesh over call per enabled, non-maintenance `ClusterNode` row carrying a `GrpcPort` — rather than fanning
`ClusterClient` (commands), gRPC (telemetry — see `docs/Telemetry.md`), and `ConfigServe` (config through any client/receptionist object (the Akka `ClusterClient` transport this used to run on is
fetch — see [`ConfigSource`/`ConfigServe`](../CLAUDE.md#config-source-configsource--configserve)), **deleted**; see [ClusterClient→gRPC migration](plans/2026-08-06-clusterclient-to-grpc-design.md)).
never over gossip. Central reaches a site mesh over gRPC for commands (`MeshTransport`), telemetry (see
`docs/Telemetry.md`), and config fetch (`ConfigServe` — see
[`ConfigSource`/`ConfigServe`](../CLAUDE.md#config-source-configsource--configserve)), never over
gossip.
- **`SplitTopologyTransportValidator` fails host start** on a node carrying a `cluster-{ClusterId}` - **`SplitTopologyTransportValidator` fails host start** on a node carrying a `cluster-{ClusterId}`
role unless it is also configured for the split-safe transports: `MeshTransport:Mode = ClusterClient`, role unless it is also configured for the split-safe transports: `MeshTransport:Mode = Grpc`,
`Telemetry:Mode = Grpc` (driver nodes), `TelemetryDial:Mode = Grpc` (admin nodes). A DPS-mode transport `Telemetry:Mode = Grpc` (driver nodes), `TelemetryDial:Mode = Grpc` (admin nodes). A DPS-mode transport
on a cluster-role node would silently try to gossip across a partition that no longer exists, so the on a cluster-role node would silently try to gossip across a partition that no longer exists, so the
validator fails closed instead. Legacy/non-cluster-role nodes are exempt. validator fails closed instead. Legacy/non-cluster-role nodes are exempt.
@@ -538,27 +541,34 @@ See [ScriptedAlarms.md](ScriptedAlarms.md) and [AlarmTracking.md](AlarmTracking.
Everything above rides one Akka mesh: central publishes on DistributedPubSub, every node Everything above rides one Akka mesh: central publishes on DistributedPubSub, every node
subscribes, and the two ends must be members of the same cluster. Per-cluster mesh **Phase 2** subscribes, and the two ends must be members of the same cluster. Per-cluster mesh **Phase 2**
(cross-mesh cutover: [ClusterClient→gRPC migration](plans/2026-08-06-clusterclient-to-grpc-design.md))
adds a second transport under `MeshTransport:Mode` (see adds a second transport under `MeshTransport:Mode` (see
[Configuration.md § `MeshTransport`](Configuration.md#meshtransport-centralnode-command-transport)) [Configuration.md § `MeshTransport`](Configuration.md#meshtransport-centralnode-command-transport))
that does not need shared membership — the prerequisite for splitting the fleet into one mesh per that does not need shared membership — the prerequisite for splitting the fleet into one mesh per
application `Cluster`. application `Cluster`. The Akka `ClusterClient` transport that originally filled this second slot is
**deleted**; `Grpc` is its replacement, not an alternative alongside it.
| | `Dps` (default) | `ClusterClient` | | | `Dps` (default, pair-local) | `Grpc` (cross-mesh) |
|---|---|---| |---|---|---|
| Central → node commands | `Publish` on `deployments` / `driver-control` / `alarm-commands` | `ClusterClient.SendToAll` to `/user/node-communication`, re-emitted on the node's **local** `EventStream` | | Central → node commands | `Publish` on `deployments` / `driver-control` / `alarm-commands` | `NodeCommandService.Execute`, one unary call per node dialled directly from `ClusterNode.Host`+`GrpcPort`; decoded and handed to the node's `NodeCommunicationActor`, which re-emits it on the node's **local** `EventStream` exactly as the DPS path does |
| Node → central acks | `Publish` on `deployment-acks` | `ClusterClient.Send` to `/user/central-communication`, forwarded to the deploy-coordinator singleton | | Node → central acks | `Publish` on `deployment-acks` | `CentralCommandService.ReportApply`, walking `MeshTransport:CentralGrpcEndpoints` in order and stopping at the first success, forwarded to the deploy-coordinator singleton |
| Requires shared cluster membership | **yes** | no | | Requires shared cluster membership | **yes** | no |
| Auth | none (mesh membership is the boundary) | per-cluster PSK bearer, both directions, fail-closed |
Two properties are load-bearing and easy to break: Two properties are load-bearing and easy to break:
- **Both comm actors are registered per node, NOT as cluster singletons.** A `ClusterClient` rotates - **The gRPC dialer fans node by node, not through any shared client or receptionist object.** There
across its contact points and must find a live comm actor at whichever node answers. As a is no rotation and no "whichever node answers" — each command is dialled at the specific node's own
singleton the boundary would be reachable through exactly one node, and rotation would fail endpoint, so a node left out of the `ClusterNode` dispatch set (no `GrpcPort`, disabled, or in
silently against the others — acks landing or not depending on where the client happened to `MaintenanceMode`) silently keeps its old configuration while the deployment still seals green
settle. `akka.cluster.client.receptionist.role` is deliberately empty for the same reason. around it, the same failure shape `ClusterClient`'s `SendToAll` had.
- **Inbound commands go to the node-local `EventStream`, never back onto a DPS topic.** DPS is - **Inbound commands go to the node-local `EventStream`, never back onto a DPS topic.** DPS is
mesh-wide and central `SendToAll`s to *every* node, so a DPS re-publish would deliver N copies of mesh-wide and central fans out to *every* node, so a DPS re-publish would deliver N copies of
every command to every subscriber. every command to every subscriber.
- **Both transports are fire-and-forget, with no retry or queue.** A failed gRPC dispatch is logged
(naming the node and the gRPC status) and dropped in the dispatch second — the deploy path's
2-minute apply deadline and a node's boot-time replay (`DriverHostActor.Bootstrap()` re-applying an
orphan `Applying` row) are the only recovery mechanisms, unchanged from the DPS/`ClusterClient` era.
Phase 2 changes no redundancy semantics: the Primary gate, ServiceLevel, and the singleton layout Phase 2 changes no redundancy semantics: the Primary gate, ServiceLevel, and the singleton layout
are untouched, and `redundancy-state` itself stays on DPS (it is bidirectional and built from are untouched, and `redundancy-state` itself stays on DPS (it is bidirectional and built from
+16 -6
View File
@@ -39,10 +39,20 @@ mirrors the inversion ScadaBridge already uses for the same problem (its `SiteSt
A fused `admin,driver` node both hosts (as driver) and dials (as admin) — it dials itself plus its A fused `admin,driver` node both hosts (as driver) and dials (as admin) — it dials itself plus its
redundant pair peer, the same way central dials site nodes. redundant pair peer, the same way central dials site nodes.
**This same driver-role listener also hosts the mesh gRPC command-plane ingress.** Whenever
`Telemetry:GrpcListenPort > 0`, the node maps both `TelemetryStreamGrpcService` (this stream) and
`NodeCommandService` (central→node commands under `MeshTransport:Mode=Grpc` — see
[`docs/Configuration.md` § `MeshTransport`](Configuration.md#meshtransport-centralnode-command-transport))
on the one port. They share it rather than each claiming its own, since both are dialled by the same
central admin nodes and a second cleartext h2c port would buy nothing but another firewall rule.
`NodeCommandService` is gated by its own `MeshCommandAuthInterceptor`/`GrpcPsk` bearer, independent
of this stream's `Telemetry:ApiKey`.
Central discovers driver-node telemetry endpoints from the `ClusterNode` table (`Host` + Central discovers driver-node telemetry endpoints from the `ClusterNode` table (`Host` +
`GrpcPort`, the latter added in Phase 1 for exactly this purpose) — the same DB-sourced discovery `GrpcPort`, the latter added in Phase 1 for exactly this purpose) — the same DB-sourced discovery
Phase 1's ack set and Phase 2's `ClusterClient` contact set already use. No shared gossip Phase 1's ack set and Phase 2's gRPC command dial already use (see
membership is required to find or dial a node. [`docs/Configuration.md` § `MeshTransport`](Configuration.md#meshtransport-centralnode-command-transport)).
No shared gossip membership is required to find or dial a node.
## The dark switch ## The dark switch
@@ -139,8 +149,8 @@ each for a distinct, settled reason:
nothing here for a per-node gRPC stream to carry. Revisit in Phase 6, once central genuinely nothing here for a per-node gRPC stream to carry. Revisit in Phase 6, once central genuinely
loses gossip visibility of site nodes and needs another way to know a pair's membership. loses gossip visibility of site nodes and needs another way to know a pair's membership.
- **`deployment-acks`** — already migrated, but onto a different transport: it rides the Phase 2 - **`deployment-acks`** — already migrated, but onto a different transport: it rides the Phase 2
`ClusterClient` transport (`MeshTransport:Mode=ClusterClient`) as a command-plane reply, not an gRPC command transport (`MeshTransport:Mode=Grpc`, `CentralCommandService.ReportApply`) as a
observability broadcast. It was never a Phase 5 candidate. command-plane reply, not an observability broadcast. It was never a Phase 5 candidate.
See [`docs/Redundancy.md`](Redundancy.md#command-transport-centralnode) for how `redundancy-state` See [`docs/Redundancy.md`](Redundancy.md#command-transport-centralnode) for how `redundancy-state`
and the command transports fit together. and the command transports fit together.
@@ -249,7 +259,7 @@ otherwise), and `Grpc` mode on either side requires a non-empty `ApiKey` — bot
naturally node-sourced and fleet-wide-fanned, so putting central in the client role — dialing out naturally node-sourced and fleet-wide-fanned, so putting central in the client role — dialing out
to each node by its `ClusterNode`-recorded address — is what lets telemetry survive the mesh split to each node by its `ClusterNode`-recorded address — is what lets telemetry survive the mesh split
cleanly: central never needs cluster membership with a site's mesh to keep watching it. cleanly: central never needs cluster membership with a site's mesh to keep watching it.
- **Status:** code-complete on `feat/mesh-phase5`; the live gate (flip the docker-dev rig to `Grpc`, - **Status:** merged to `master` (`35552a96`); the live gate (flip the docker-dev rig to `Grpc`,
confirm all panels stay green with the DPS telemetry bridges unspawned, kill-and-reconnect a site confirm all panels stay green with the DPS telemetry bridges unspawned, kill-and-reconnect a site
node) has not yet run. See `docs/plans/2026-07-22-per-cluster-mesh-program.md` § Phase 5 and node) PASSED 2026-07-23. See `docs/plans/2026-07-22-per-cluster-mesh-program.md` § Phase 5 and
`docs/plans/2026-07-23-mesh-phase5-grpc-telemetry-stream.md` for the implementation plan. `docs/plans/2026-07-23-mesh-phase5-grpc-telemetry-stream.md` for the implementation plan.
@@ -26,7 +26,7 @@ posture).
| 1A | Central→node command plane (`NodeCommandService` on the Telemetry listener; `MeshGrpcCommandDialer`; third mode branch in `CentralCommunicationActor`) — **worktree A** | `feat/grpc-node-command` | **COMPLETE @ `e54883a8` — gate re-run PASSED 2026-08-06.** Run 1 @ `fbf7b708` FAILED on defect 4 (no admin node could boot, mode-independent); fixed by `c0475f2d` + `e54883a8` and re-gated. 1A.1 ✅ `64774652` + 1A.2 ✅ `6a862011` (+fixes `48dc2120`/`86633f14`) all reviewed spec ✅ + code ✅. **MERGED to `master`** @ `b360239e` (2026-08-06) | `b360239e` | **PASS 8/8** (check 3's AdminUI half closed by composition — the control is structurally unreachable, see carry-forwards) | | 1A | Central→node command plane (`NodeCommandService` on the Telemetry listener; `MeshGrpcCommandDialer`; third mode branch in `CentralCommunicationActor`) — **worktree A** | `feat/grpc-node-command` | **COMPLETE @ `e54883a8` — gate re-run PASSED 2026-08-06.** Run 1 @ `fbf7b708` FAILED on defect 4 (no admin node could boot, mode-independent); fixed by `c0475f2d` + `e54883a8` and re-gated. 1A.1 ✅ `64774652` + 1A.2 ✅ `6a862011` (+fixes `48dc2120`/`86633f14`) all reviewed spec ✅ + code ✅. **MERGED to `master`** @ `b360239e` (2026-08-06) | `b360239e` | **PASS 8/8** (check 3's AdminUI half closed by composition — the control is structurally unreachable, see carry-forwards) |
| 1B | Node→central ack plane (`CentralCommandService` on the ConfigServe listener; `GrpcApplyAckSender`; third branch in `NodeCommunicationActor`) — **worktree B**, rebased on merged 1A | `feat/grpc-central-ack` | **MERGED to `master`** @ `7203d135` (2026-08-06, branch tip `cf229a95` incl. the phases 2-4 plan update); 1B.1 ✅ + 1B.2 ✅ (Opus: deadline Critical found+fixed) + gate **PASS 7/7** incl. the 4b hang drill (~10 s failover = exactly one AckTimeoutSeconds — live proof of the fix); at-least-once duplicate absorbed idempotently exactly as traced | `7203d135` | **PASS 7/7** | | 1B | Node→central ack plane (`CentralCommandService` on the ConfigServe listener; `GrpcApplyAckSender`; third branch in `NodeCommunicationActor`) — **worktree B**, rebased on merged 1A | `feat/grpc-central-ack` | **MERGED to `master`** @ `7203d135` (2026-08-06, branch tip `cf229a95` incl. the phases 2-4 plan update); 1B.1 ✅ + 1B.2 ✅ (Opus: deadline Critical found+fixed) + gate **PASS 7/7** incl. the 4b hang drill (~10 s failover = exactly one AckTimeoutSeconds — live proof of the fix); at-least-once duplicate absorbed idempotently exactly as traced | `7203d135` | **PASS 7/7** |
| 2 | **COMPLETE — gate PASS 8/8 + baseline + PSK negatives** (2026-08-06, all on `master`, all review chains closed). Task 2.0 ✅ (`a94eb8a3`+`d7c9ead6`, small chain: one Important fixed — non-vacuous once-per-incarnation pin — then APPROVED; healthy racing boot = ZERO Warn/Error, proven live on both centrals 4/4 boots). Register-item-3 role-scoping pin ✅ (`2e06dfe5`+`027f0403`, review found+fixed a Critical phantom-block false pass in the scan — braceless `if (hasAdmin)` at Program.cs:146). Baseline REPRODUCED pre-flip (`1bb3a937`): cross-mesh `:4053` open unauthenticated; the site-b drop is actively suppressed into SAMPLED deadLetters INF (456/467 unlogged) with the deploy deadline as sole witness. Cutover `caf2eec6` (rig default → `Grpc`, committed). Gate: deploy e2e all-gRPC sealed 6/6; command matrix live incl. real driver restart; kill-site-a-1 → per-node WRN naming node+status, exactly 1 attempt, honest deadline, boot-replay-only recovery; kill-central-1 → ack ordered-walk failover logged per node, reconvergence 6/6; site-b-pair drill → BOTH failures logged in the dispatch second (the baseline contrast); clusterclient/receptionist lines = 0 on all six through the whole gate; 135-min soak 14/14 cycles, denied=0/unauth=0, review items explained (historian-sidecar-absent DBG noise; 14×2 known per-deploy authoring WRNs); **full suite exit 0 on the cutover SHA, run against the live rig** | `caf2eec6` (cutover) + gate records `1bb3a937`/`bcc91268`/`eaf14e01`+final | **PASS 8/8** | | 2 | **COMPLETE — gate PASS 8/8 + baseline + PSK negatives** (2026-08-06, all on `master`, all review chains closed). Task 2.0 ✅ (`a94eb8a3`+`d7c9ead6`, small chain: one Important fixed — non-vacuous once-per-incarnation pin — then APPROVED; healthy racing boot = ZERO Warn/Error, proven live on both centrals 4/4 boots). Register-item-3 role-scoping pin ✅ (`2e06dfe5`+`027f0403`, review found+fixed a Critical phantom-block false pass in the scan — braceless `if (hasAdmin)` at Program.cs:146). Baseline REPRODUCED pre-flip (`1bb3a937`): cross-mesh `:4053` open unauthenticated; the site-b drop is actively suppressed into SAMPLED deadLetters INF (456/467 unlogged) with the deploy deadline as sole witness. Cutover `caf2eec6` (rig default → `Grpc`, committed). Gate: deploy e2e all-gRPC sealed 6/6; command matrix live incl. real driver restart; kill-site-a-1 → per-node WRN naming node+status, exactly 1 attempt, honest deadline, boot-replay-only recovery; kill-central-1 → ack ordered-walk failover logged per node, reconvergence 6/6; site-b-pair drill → BOTH failures logged in the dispatch second (the baseline contrast); clusterclient/receptionist lines = 0 on all six through the whole gate; 135-min soak 14/14 cycles, denied=0/unauth=0, review items explained (historian-sidecar-absent DBG noise; 14×2 known per-deploy authoring WRNs); **full suite exit 0 on the cutover SHA, run against the live rig** | `caf2eec6` (cutover) + gate records `1bb3a937`/`bcc91268`/`eaf14e01`+final | **PASS 8/8** |
| 3 | Delete ClusterClient (code/HOCON/config/tests); SplitTopology requires `Grpc`; grep-gates; docs sweep incl. scadaproj index | `feat/grpc-phase3-deletion` | NOT STARTED | — | — | | 3 | Delete ClusterClient (code/HOCON/config/tests); SplitTopology requires `Grpc`; grep-gates; docs sweep incl. scadaproj index | `feat/grpc-phase3-deletion` | **COMPLETE @ DoD — awaiting the user's merge decision** (2026-08-06, branch tip `8e25ed55`, 10 commits off `master @ 55104526`, tree clean). Task 3.1 ✅ high-risk chain: 7 ordered steps `0db40a5e`/`ab5b958d`/`5d7ecdf7`/`d67a7e5d`/`85ca86e9`/`af3b0c31`/`4d2dfb5b`; spec review ✅ (all KEEPs verified: `LoadContactsFromDb`, DPS+frame-size canary, `deployment-acks` DPS ingress, Kestrel block, one-ctor interceptors, no retries, proto untouched; `MeshGrpcBoundaryTests`' no-association assert verified falsifiable); code review ✅ Approved, one Important fixed (`cd713557` — `MeshPaths` full-path constants were dead-in-production + doc claimed a wire-contract role they no longer have; collapsed to `*Name` constants). Retired-mode fail-closed: `Mode=ClusterClient` now FAILS validation naming `MeshTransport:Mode` + allowed values (pinned); `SplitTopologyTransportValidator` requires exactly `Grpc`. Task 3.2 ✅ docs sweep `406b1ae5` + review-fix `8e25ed55` (Configuration/Redundancy/Telemetry/config-db-schema/CLAUDE.md; deferment.md had no mention); docs-accuracy review ✅ Approved — every config key/port/behavior claim verified against source, two pre-existing Minor stalenesses fixed (Telemetry.md Phase 5 gate status; config-db-schema Phase 6 gossip premise). scadaproj index updated on its `main` and PUSHED (`0e416ca`, merge-state-accurate: Phase 3 recorded as pending merge). Grep-gates: `CentralContactPoints` in src/tests/docker-dev = 0; `clusterclient\|receptionist` = historical/fail-closed/retirement mentions only (classified by both reviewers). **DoD full-suite: run 3 = `dotnet test` exit 0, 50/50 suites, 9,326 passed / 0 failed / 57 skipped, on `8e25ed55`.** Runs 1–2 each had ONE unrelated single-test failure (three distinct tests across the two runs: `RouteNativeAlarmAck_on_non_primary_is_dropped`, `Rediscovery_flag_persists_on_subsequent_health_publishes`, `LoggerLifecycleTests.ConfigureLogging_DisposesPreviousLogger…`) — all in files the 55-file branch diff never touches, all 3/3 green in isolation (9/9 total): pre-existing under-load timing flakes, not Phase 3 regressions. ⚠️ Process find: run 1's "exit 0" captured `tail`'s exit code, not `dotnet test`'s (`cmd \| tail; $?` trap) — fixed for runs 2–3; **Phase 2's check-8 "full suite exit 0" used the same pipeline shape and its log did not survive, so that exit-code claim is unverifiable in hindsight** (its per-suite pass lines were observed at the time; Phase 4's terminal run on the merge SHA, with correct capture, supersedes it) | — (merge pending) | n/a (Phase 4 is the terminal live gate) |
| 4 | Terminal live gate on the post-deletion image (8 checks, provenance-guarded) | (on `master`) | NOT STARTED | — | PENDING | | 4 | Terminal live gate on the post-deletion image (8 checks, provenance-guarded) | (on `master`) | NOT STARTED | — | PENDING |
**Execution rules recap:** phase order 0 → (1A ∥ 1B) → 2 → 3 → 4; 1A∥1B only in separate git **Execution rules recap:** phase order 0 → (1A ∥ 1B) → 2 → 3 → 4; 1A∥1B only in separate git
@@ -211,10 +211,13 @@
"branch": "feat/grpc-phase3-deletion", "branch": "feat/grpc-phase3-deletion",
"classification": "high-risk", "classification": "high-risk",
"parallelizableWith": [], "parallelizableWith": [],
"status": "pending", "status": "completed",
"blockedBy": [ "blockedBy": [
12 12
] ],
"commit": "0db40a5e..4d2dfb5b (7 steps) + review fix cd713557",
"review": "spec ✅ (all KEEPs verified, boundary assert falsifiable) + code ✅ Approved (1 Important fixed: dead MeshPaths full-path constants collapsed)",
"note": "Grep-gates clean: CentralContactPoints=0; clusterclient|receptionist = historical/fail-closed mentions only. MeshClusterClientBoundaryTests deleted in step 1 (compile ripple), replacement MeshGrpcBoundaryTests landed step 7 as planned; MeshRouterDispatchTests needed no change (mode-agnostic); MeshCommActorPathTests reshaped not retired."
}, },
{ {
"id": 14, "id": 14,
@@ -223,10 +226,12 @@
"branch": "feat/grpc-phase3-deletion", "branch": "feat/grpc-phase3-deletion",
"classification": "small", "classification": "small",
"parallelizableWith": [], "parallelizableWith": [],
"status": "pending", "status": "completed",
"blockedBy": [ "blockedBy": [
13 13
] ],
"commit": "406b1ae5 + review fixes 8e25ed55; scadaproj index 0e416ca on scadaproj main, pushed",
"review": "docs-accuracy ✅ Approved (every key/port/behavior/merge-state claim verified against source; 2 pre-existing Minor stalenesses fixed)"
}, },
{ {
"id": 15, "id": 15,
+12 -4
View File
@@ -174,10 +174,18 @@ CREATE UNIQUE INDEX UX_ClusterNode_Primary_Per_Cluster
These are **the addresses central dials**, not the node's own binding configuration. The node binds These are **the addresses central dials**, not the node's own binding configuration. The node binds
from `Cluster:Port` in its own appsettings; these columns duplicate that value for a reader that from `Cluster:Port` in its own appsettings; these columns duplicate that value for a reader that
cannot see the node's config. Today central shares a gossip ring with every node and does not need cannot see the node's config. When this column was added central still shared one gossip ring with
them — [per-cluster mesh Phase 2](../plans/2026-07-21-per-cluster-mesh-design.md) splits the fleet every node and did not strictly need them; that premise ended at
into one mesh per `Cluster`, after which `Host` + `AkkaPort` is how central builds its ClusterClient [per-cluster mesh Phase 6](../plans/2026-07-24-mesh-phase6-partition-and-colocation.md)
contact points, and `GrpcPort` is the Phase 5 telemetry stream. (2026-07-24), which split the fleet into one 2-node Akka mesh per application `Cluster` — central no
longer shares gossip with a site pair at all, and reaches it only over gRPC (and, before Phase 6
finished landing, briefly over the now-deleted Akka `ClusterClient`). `Host` + `AkkaPort` was how
central would have built `ClusterClient` contact points; that transport was deleted by the
[ClusterClient→gRPC migration](../plans/2026-08-06-clusterclient-to-grpc-design.md) — central now
dials `NodeCommandService` at `Host`+`GrpcPort` instead, and `AkkaPort` is Akka remoting only (still
needed pair-locally, within a `Cluster`'s own 2-node mesh). `GrpcPort` is the Phase 5 telemetry
stream target **and**, since that migration, the Phase 2 gRPC command dial target too — see
[`Configuration.md` § `MeshTransport`](../Configuration.md#meshtransport-centralnode-command-transport).
`AkkaPort` is `NOT NULL DEFAULT 4053` because every node listens on a remoting port — `0` is never a `AkkaPort` is `NOT NULL DEFAULT 4053` because every node listens on a remoting port — `0` is never a
truthful value, and rows predating the column must migrate to something real. `GrpcPort` is nullable truthful value, and rows predating the column must migrate to something real. `GrpcPort` is nullable
@@ -6,9 +6,11 @@ namespace ZB.MOM.WW.OtOpcUa.Cluster;
/// </summary> /// </summary>
/// <remarks> /// <remarks>
/// <para> /// <para>
/// <see cref="Mode"/> is a deliberate <b>dark switch</b>: the transports live in the tree /// <see cref="Mode"/> selects between a <b>pair-local</b> transport and a <b>cross-mesh</b>
/// side by side so the docker-dev rig can run the same deployment over each and compare, and /// one. <see cref="ModeDps"/> is the default and cannot cross an application-<c>Cluster</c>
/// so a bad rollout rolls back with a config change rather than a revert. /// boundary; <see cref="ModeGrpc"/> is the only mode that can, and
/// <see cref="SplitTopologyTransportValidator"/> requires it on any node carrying a
/// <c>cluster-{ClusterId}</c> role.
/// </para> /// </para>
/// <para> /// <para>
/// <b>The DistributedPubSub branch is retained, not scheduled for deletion.</b> An earlier /// <b>The DistributedPubSub branch is retained, not scheduled for deletion.</b> An earlier
@@ -16,15 +18,15 @@ namespace ZB.MOM.WW.OtOpcUa.Cluster;
/// deliberately reversed — see <c>docs/Configuration.md</c> § "The compiled transport-mode /// deliberately reversed — see <c>docs/Configuration.md</c> § "The compiled transport-mode
/// defaults were deliberately NOT flipped". DPS remains the correct and default transport for /// defaults were deliberately NOT flipped". DPS remains the correct and default transport for
/// a single-mesh topology, which every integration test and every profile that leaves this /// a single-mesh topology, which every integration test and every profile that leaves this
/// key unset still runs. What is being replaced is <see cref="ModeClusterClient"/>, by /// key unset still runs.
/// <see cref="ModeGrpc"/>.
/// </para> /// </para>
/// <para> /// <para>
/// <b>Discovery is deliberately asymmetric</b>, matching the sister project. Central /// <b>Discovery is deliberately asymmetric</b>, matching the sister project. Central
/// discovers nodes <i>from the database</i> — enabled, non-maintenance <c>ClusterNode</c> /// discovers nodes <i>from the database</i> — enabled, non-maintenance <c>ClusterNode</c>
/// rows, rebuilt every <see cref="ContactRefreshSeconds"/>. Nodes discover central <i>from /// rows, rebuilt every <see cref="ContactRefreshSeconds"/>. Nodes discover central <i>from
/// this section</i>: static, restart to change. The asymmetry follows the rate of change — /// this section</i> (<see cref="CentralGrpcEndpoints"/>): static, restart to change. The
/// operators add and retire nodes, but central's own address is part of the deployment. /// asymmetry follows the rate of change — operators add and retire nodes, but central's own
/// address is part of the deployment.
/// </para> /// </para>
/// </remarks> /// </remarks>
public sealed class MeshTransportOptions public sealed class MeshTransportOptions
@@ -32,45 +34,31 @@ public sealed class MeshTransportOptions
/// <summary>Configuration section name.</summary> /// <summary>Configuration section name.</summary>
public const string SectionName = "MeshTransport"; public const string SectionName = "MeshTransport";
/// <summary>Every command stays on DistributedPubSub — the pre-Phase-2 behaviour.</summary> /// <summary>
/// Every command stays on DistributedPubSub — pair-local, and the default. Cannot cross an
/// application-<c>Cluster</c> boundary.
/// </summary>
public const string ModeDps = "Dps"; public const string ModeDps = "Dps";
/// <summary>Commands cross the ClusterClient receptionist boundary.</summary>
public const string ModeClusterClient = "ClusterClient";
/// <summary> /// <summary>
/// Commands cross the mesh boundary as unary gRPC calls on the <c>mesh_transport.v1</c> /// Commands cross the mesh boundary as unary gRPC calls on the <c>mesh_transport.v1</c>
/// contract — central dials each node's <c>NodeCommandService</c>, a node posts its /// contract — central dials each node's <c>NodeCommandService</c>, a node posts its
/// deployment ack to central's <c>CentralCommandService</c>. The replacement for /// deployment ack to central's <c>CentralCommandService</c>. The command plane is
/// <see cref="ModeClusterClient"/>; unlike it, the command plane is authenticated /// authenticated per application <c>Cluster</c> (<see cref="GrpcPsk"/>).
/// (<see cref="GrpcPsk"/>).
/// </summary> /// </summary>
public const string ModeGrpc = "Grpc"; public const string ModeGrpc = "Grpc";
/// <summary> /// <summary>
/// <see cref="ModeDps"/> (default), <see cref="ModeClusterClient"/> or /// <see cref="ModeDps"/> (default) or <see cref="ModeGrpc"/>. Any other value fails the host
/// <see cref="ModeGrpc"/>. Any other value fails the host at startup rather than silently /// at startup rather than silently falling back — a transport that is neither delivers
/// falling back — a transport that is none of these delivers nothing, and the symptom would /// nothing, and the symptom would be commands that vanish. That includes the retired
/// be commands that vanish. /// <c>ClusterClient</c> value: a node still carrying it is refused at boot rather than
/// quietly demoted to DPS, which on a split-topology node would deliver nothing at all.
/// </summary> /// </summary>
public string Mode { get; set; } = ModeDps; public string Mode { get; set; } = ModeDps;
/// <summary> /// <summary>
/// Akka remoting addresses of the central nodes, e.g. /// How often central re-reads <c>ClusterNode</c> rows to rebuild its dispatch set.
/// <c>akka.tcp://otopcua@central-1:4053</c>. <b>Node addresses only</b> — the
/// <c>/system/receptionist</c> path is appended at construction time.
/// </summary>
/// <remarks>
/// Each entry must be a <i>central</i> node's remoting endpoint, never this node's own port.
/// The sister project shipped a site template whose second contact was the site's own
/// remoting port; it is a permanent failure in the initial-contact rotation and stays
/// invisible until a command goes missing. <see cref="MeshTransportOptionsValidator"/>
/// rejects the malformed shapes at boot instead.
/// </remarks>
public string[] CentralContactPoints { get; set; } = [];
/// <summary>
/// How often central re-reads <c>ClusterNode</c> rows to rebuild its contact points.
/// Refreshing on a timer (rather than only on admin change) means a row edited by any /// Refreshing on a timer (rather than only on admin change) means a row edited by any
/// process — migration, direct SQL, a second admin node — is picked up without a restart. /// process — migration, direct SQL, a second admin node — is picked up without a restart.
/// </summary> /// </summary>
@@ -83,12 +71,10 @@ public sealed class MeshTransportOptions
/// (<c>scheme://host:port</c>), not bare <c>host:port</c> — <c>http://</c> is h2c. /// (<c>scheme://host:port</c>), not bare <c>host:port</c> — <c>http://</c> is h2c.
/// </summary> /// </summary>
/// <remarks> /// <remarks>
/// Tried <b>in order, stopping at the first success</b>, which is what preserves the /// Tried <b>in order, stopping at the first success</b>, which is what keeps the ack
/// receptionist's one-ack-per-node-per-dispatch semantic: either central node routes the ack /// one-per-node-per-dispatch: either central node routes it to the same
/// to the same <c>ConfigPublishCoordinator</c> singleton proxy, so which one answers does not /// <c>ConfigPublishCoordinator</c> singleton proxy, so which one answers does not matter,
/// matter, but two answering would double-count. The gRPC counterpart of /// but two answering would double-count.
/// <see cref="CentralContactPoints"/>, and it replaces it — a node running
/// <see cref="ModeGrpc"/> never reads the Akka contact points.
/// </remarks> /// </remarks>
public string[] CentralGrpcEndpoints { get; set; } = []; public string[] CentralGrpcEndpoints { get; set; } = [];
@@ -40,16 +40,18 @@ public sealed class MeshTransportOptionsValidator : IValidateOptions<MeshTranspo
var errors = new List<string>(); var errors = new List<string>();
var isDps = string.Equals(options.Mode, MeshTransportOptions.ModeDps, StringComparison.OrdinalIgnoreCase); var isDps = string.Equals(options.Mode, MeshTransportOptions.ModeDps, StringComparison.OrdinalIgnoreCase);
var isClusterClient = string.Equals(
options.Mode, MeshTransportOptions.ModeClusterClient, StringComparison.OrdinalIgnoreCase);
var isGrpc = string.Equals(options.Mode, MeshTransportOptions.ModeGrpc, StringComparison.OrdinalIgnoreCase); var isGrpc = string.Equals(options.Mode, MeshTransportOptions.ModeGrpc, StringComparison.OrdinalIgnoreCase);
if (!isDps && !isClusterClient && !isGrpc) // An unrecognised mode is refused, never defaulted. The retired 'ClusterClient' value lands
// here: a node left carrying it fails to start with the two remaining values named, rather
// than silently falling back to Dps — which on a split-topology node cannot cross the mesh
// boundary and would deliver nothing at all.
if (!isDps && !isGrpc)
{ {
errors.Add( errors.Add(
$"{MeshTransportOptions.SectionName}:{nameof(MeshTransportOptions.Mode)} is " $"{MeshTransportOptions.SectionName}:{nameof(MeshTransportOptions.Mode)} is "
+ $"'{options.Mode}'. Expected '{MeshTransportOptions.ModeDps}', " + $"'{options.Mode}'. Expected '{MeshTransportOptions.ModeDps}' or "
+ $"'{MeshTransportOptions.ModeClusterClient}' or '{MeshTransportOptions.ModeGrpc}'."); + $"'{MeshTransportOptions.ModeGrpc}'.");
} }
if (options.ContactRefreshSeconds <= 0) if (options.ContactRefreshSeconds <= 0)
@@ -57,48 +59,14 @@ public sealed class MeshTransportOptionsValidator : IValidateOptions<MeshTranspo
errors.Add( errors.Add(
$"{MeshTransportOptions.SectionName}:{nameof(MeshTransportOptions.ContactRefreshSeconds)} " $"{MeshTransportOptions.SectionName}:{nameof(MeshTransportOptions.ContactRefreshSeconds)} "
+ $"must be greater than 0 (was {options.ContactRefreshSeconds}). Central rebuilds its " + $"must be greater than 0 (was {options.ContactRefreshSeconds}). Central rebuilds its "
+ "ClusterClient contact points on this interval; a non-positive value would leave the " + "node dispatch set on this interval; a non-positive value would leave that set "
+ "contact set frozen at whatever it held when the node booted."); + "frozen at whatever it held when the node booted.");
}
// Contact points are only load-bearing under ClusterClient mode. Requiring them under the
// default would make the section mandatory everywhere for a feature that is switched off —
// including on admin-only nodes, which have no reason to carry them at all.
if (isClusterClient)
{
if (options.CentralContactPoints.Length == 0)
{
errors.Add(
$"{MeshTransportOptions.SectionName}:{nameof(MeshTransportOptions.CentralContactPoints)} "
+ "is empty. Under ClusterClient mode a driver node cannot reach central without at "
+ "least one contact point, and its deployment acks would be dropped silently — the "
+ "deployment would then time out naming a node that is running perfectly.");
}
foreach (var contactPoint in options.CentralContactPoints)
{
if (!contactPoint.StartsWith("akka.tcp://", StringComparison.Ordinal))
{
errors.Add(
$"Contact point '{contactPoint}' must start with 'akka.tcp://' — it is an Akka "
+ "remoting address (akka.tcp://<system>@<host>:<port>), not a bare host:port.");
}
else if (contactPoint.Contains("/user/", StringComparison.Ordinal)
|| contactPoint.Contains("/system/", StringComparison.Ordinal))
{
errors.Add(
$"Contact point '{contactPoint}' carries an actor-path suffix. Configure the "
+ "node address only — '/system/receptionist' is appended at construction time, "
+ "and a doubled suffix produces a path that never resolves.");
}
}
} }
// The first Grpc block below is node-side: the PSK a driver presents (and requires), the // The first Grpc block below is node-side: the PSK a driver presents (and requires), the
// central endpoints it posts its ack to, and the two call deadlines (the admin-side rule that // central endpoints it posts its ack to, and the two call deadlines (the admin-side rule that
// follows it is a separate scope). Scoped to the driver role for the same reason the // follows it is a separate scope). Scoped to the driver role because an admin-only central node
// ClusterClient block above is scoped to its mode — an admin-only central node // dials nodes rather than being dialled, and resolves per-cluster keys at runtime through
// dials nodes rather than being dialled and resolves per-cluster keys at runtime through
// MeshTransportOptions.ClusterPsks / the Secrets store, so requiring a single GrpcPsk and an // MeshTransportOptions.ClusterPsks / the Secrets store, so requiring a single GrpcPsk and an
// endpoint list of it would demand configuration it never reads. Roles come from // endpoint list of it would demand configuration it never reads. Roles come from
// configuration, not IClusterRoleInfo (see the ctor remark), and through // configuration, not IClusterRoleInfo (see the ctor remark), and through
@@ -175,7 +143,7 @@ public sealed class MeshTransportOptionsValidator : IValidateOptions<MeshTranspo
// every node's deployment ack has nowhere to land, and nothing errors: the deployment simply // every node's deployment ack has nowhere to land, and nothing errors: the deployment simply
// times out naming nodes that applied perfectly. Owned here rather than by ConfigServe's own // times out naming nodes that applied perfectly. Owned here rather than by ConfigServe's own
// validation because the requirement comes from MeshTransport:Mode, not from config-serve — a // validation because the requirement comes from MeshTransport:Mode, not from config-serve — a
// Dps or ClusterClient central node is legitimately portless. // Dps central node is legitimately portless.
if (isGrpc && isAdmin) if (isGrpc && isAdmin)
{ {
// Read as a string, the way SplitTopologyTransportValidator reads the modes: // Read as a string, the way SplitTopologyTransportValidator reads the modes:
@@ -13,14 +13,12 @@ akka {
# since the Serilog logger can be slow to initialize at startup. # since the Serilog logger can be slow to initialize at startup.
logger-startup-timeout = 30s logger-startup-timeout = 30s
# DistributedPubSub is PAIR-LOCAL and stays: it carries MeshTransport:Mode = Dps (the
# single-mesh default), the node-local alarm-command publishes and the node<->node
# redundancy-state probe leg. What is gone is the cross-mesh command transport, which is now
# gRPC and needs no Akka extension at all.
extensions = [ extensions = [
"Akka.Cluster.Tools.PublishSubscribe.DistributedPubSubExtensionProvider, Akka.Cluster.Tools" "Akka.Cluster.Tools.PublishSubscribe.DistributedPubSubExtensionProvider, Akka.Cluster.Tools"
# Per-cluster mesh Phase 2 — the central<->node command boundary. NOT auto-started: without
# this entry the receptionist materialises only if some code path happens to call
# ClusterClientReceptionist.Get(system) first, which makes "is the boundary listening?"
# depend on startup ordering rather than on configuration.
"Akka.Cluster.Tools.Client.ClusterClientReceptionistExtensionProvider, Akka.Cluster.Tools"
] ]
actor { actor {
@@ -94,31 +92,6 @@ akka {
down-removal-margin = 15s down-removal-margin = 15s
run-coordinated-shutdown-when-down = on run-coordinated-shutdown-when-down = on
# CLUSTERCLIENT — the central<->node command boundary (per-cluster mesh Phase 2).
client {
# DELIBERATE, and NOT what the sister project runs: they never wrote this section, so
# they inherit buffer-size = 1000. Zero means "if the receptionist is unknown, drop the
# message now". Buffering would replay a DispatchDeployment on reconnect and apply a
# deployment ConfigPublishCoordinator had already sealed as TimedOut — the node would
# then be running a configuration the database says failed. Fail-fast keeps the DB the
# single record of truth, and reproduces today's DPS behaviour exactly: a node that is
# down misses the publish and self-heals from the DB on restart.
buffer-size = 0
# Retry forever rather than self-stopping the client. Restates Akka's default; stated
# explicitly because the behaviour is load-bearing — a central node down for an hour
# must not require a driver-node restart to become reachable again.
reconnect-timeout = off
receptionist {
# Empty = every member hosts a receptionist. This is what makes the comm actors
# "registered per node, not as a singleton" work: contact rotation reaches whichever
# node answers. Restates Akka's default; do NOT scope this to a role, or the
# boundary becomes reachable through one node only and rotation silently fails.
role = ""
}
}
singleton { singleton {
singleton-name = "singleton" singleton-name = "singleton"
} }
@@ -98,22 +98,19 @@ public sealed class SplitTopologyTransportValidator : IValidateOptions<MeshTrans
var telemetryMode = _configuration["Telemetry:Mode"] ?? TelemetryOptions.ModeDps; var telemetryMode = _configuration["Telemetry:Mode"] ?? TelemetryOptions.ModeDps;
var telemetryDialMode = _configuration["TelemetryDial:Mode"] ?? TelemetryDialOptions.ModeDps; var telemetryDialMode = _configuration["TelemetryDial:Mode"] ?? TelemetryDialOptions.ModeDps;
// Two accepted values, not one, for the duration of the ClusterClient→gRPC migration: both // Exactly ONE accepted value now that the Akka ClusterClient transport is deleted: Grpc is
// cross a mesh boundary, which is the only property this rule cares about. Without the // the only mode that crosses a mesh boundary, which is the only property this rule cares
// relaxation the very first rollout step — flipping a split-topology node to Grpc — would be a // about. Deliberately an explicit allow-list rather than "anything but Dps" — a typo'd mode
// host-start failure. It is NOT widened to "anything but Dps": a typo'd mode delivers nothing // delivers nothing just as surely as DPS does, and the retired 'ClusterClient' value is
// just as surely, so the check stays an explicit allow-list and tightens to Grpc-only once // caught here as well as by MeshTransportOptionsValidator's own unknown-mode rule.
// ClusterClient is deleted. if (!string.Equals(meshMode, MeshTransportOptions.ModeGrpc, StringComparison.OrdinalIgnoreCase))
if (!string.Equals(meshMode, MeshTransportOptions.ModeClusterClient, StringComparison.OrdinalIgnoreCase)
&& !string.Equals(meshMode, MeshTransportOptions.ModeGrpc, StringComparison.OrdinalIgnoreCase))
{ {
errors.Add( errors.Add(
$"Cluster:Roles declares a cluster-{clusterId} role (split topology) but " $"Cluster:Roles declares a cluster-{clusterId} role (split topology) but "
+ $"{MeshTransportOptions.SectionName}:{nameof(MeshTransportOptions.Mode)} is " + $"{MeshTransportOptions.SectionName}:{nameof(MeshTransportOptions.Mode)} is "
+ $"'{meshMode}'; a split mesh cannot use DistributedPubSub — set " + $"'{meshMode}'; a split mesh cannot use DistributedPubSub — set "
+ $"{MeshTransportOptions.SectionName}:{nameof(MeshTransportOptions.Mode)}=" + $"{MeshTransportOptions.SectionName}:{nameof(MeshTransportOptions.Mode)}="
+ $"{MeshTransportOptions.ModeGrpc} (or {MeshTransportOptions.ModeClusterClient}, the " + $"{MeshTransportOptions.ModeGrpc}.");
+ "transport it replaces).");
} }
if (isDriver if (isDriver
@@ -6,7 +6,7 @@ namespace ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
/// <param name="Topic"> /// <param name="Topic">
/// The legacy DPS topic. Carried so the router can publish under /// The legacy DPS topic. Carried so the router can publish under
/// <c>MeshTransport:Mode = Dps</c> without a second dispatch table; ignored entirely under /// <c>MeshTransport:Mode = Dps</c> without a second dispatch table; ignored entirely under
/// <c>ClusterClient</c> mode, where the destination is an actor path rather than a topic. /// <c>Grpc</c> mode, where the destination is a node endpoint rather than a topic.
/// </param> /// </param>
/// <param name="Message"> /// <param name="Message">
/// The command itself, never wrapped. Node-side handlers see exactly the type they see today, /// The command itself, never wrapped. Node-side handlers see exactly the type they see today,
@@ -1,33 +1,43 @@
namespace ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh; namespace ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
/// <summary> /// <summary>
/// Actor paths the two mesh comm actors register under. <b>These strings are the wire /// The <b>local actor names</b> the two mesh comm actors are spawned under. Not a wire contract:
/// contract</b> of the central↔node boundary. /// nothing off-process addresses either actor by name.
/// </summary> /// </summary>
/// <remarks> /// <remarks>
/// <para> /// <para>
/// Declared once, here, rather than as literals on each side. The <c>deployments</c> / /// <b>This used to be a wire contract and no longer is.</b> While the cross-mesh transport
/// <c>deployment-acks</c> topic names are declared twice — once in /// was an Akka <c>ClusterClient</c>, central addressed a node by the literal path string
/// <c>ConfigPublishCoordinator</c> and again in <c>DriverHostActor</c> — and agree only by /// <c>/user/node-communication</c> and the far side resolved it — so the two processes had to
/// convention; renaming one would compile, deploy, and deliver nothing. A ClusterClient path /// agree on a string neither could see the other spell. Both gRPC services now resolve
/// has the same failure mode and no compiler to catch it, so the constant is shared. /// their comm actor through <c>ActorRegistry</c> (<c>CentralCommunicationActorKey</c> /
/// <c>NodeCommunicationActorKey</c>), which is type-checked, so the name below never crosses
/// a process boundary. The full <c>/user/...</c> path constants that carried the old contract
/// are deleted; what remains is a naming convention.
/// </para> /// </para>
/// <para> /// <para>
/// Nothing else pins these values, which is why <c>MeshCommActorPathTests</c> boots a real /// Still declared once, here, rather than as literals at each spawn site — a shared constant
/// host and <c>Identify</c>-probes both paths. /// costs nothing and keeps the name an operator sees in a log or an <c>ActorSelection</c>
/// attached to one declaration. The registry key, not this string, is what a mis-rename
/// would break, and that failure is a compile error rather than a silent drop.
/// </para>
/// <para>
/// <c>MeshCommActorPathTests</c> boots a real host and <c>Identify</c>-probes
/// <c>/user/{name}</c> on both nodes, which is what pins these values against the two
/// registration sites.
/// </para> /// </para>
/// </remarks> /// </remarks>
public static class MeshPaths public static class MeshPaths
{ {
/// <summary>Central-side comm actor — where a driver node sends its deployment acks.</summary> /// <summary>
public const string CentralCommunication = "/user/central-communication"; /// Actor name of the central-side comm actor — where an inbound deployment ack is handed
/// over, after <c>CentralCommandService</c> has resolved it by registry key.
/// <summary>Node-side comm actor — where central sends commands.</summary> /// </summary>
public const string NodeCommunication = "/user/node-communication";
/// <summary>Actor name of <see cref="CentralCommunication"/> (the path's final segment).</summary>
public const string CentralCommunicationName = "central-communication"; public const string CentralCommunicationName = "central-communication";
/// <summary>Actor name of <see cref="NodeCommunication"/> (the path's final segment).</summary> /// <summary>
/// Actor name of the node-side comm actor — where an inbound command is handed over, after
/// <c>NodeCommandService</c> has resolved it by registry key.
/// </summary>
public const string NodeCommunicationName = "node-communication"; public const string NodeCommunicationName = "node-communication";
} }
@@ -9,13 +9,13 @@ namespace ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
/// <summary> /// <summary>
/// The one translation between the five cross-mesh command records and their /// The one translation between the five cross-mesh command records and their
/// <c>mesh_transport.v1</c> wire shapes (ClusterClient→gRPC migration Phase 0). Both directions live /// <c>mesh_transport.v1</c> wire shapes (mesh gRPC command plane, Phase 0). Both directions live
/// here so a field can only be added to one side without the other by making this file /// here so a field can only be added to one side without the other by making this file
/// visibly asymmetric. /// visibly asymmetric.
/// </summary> /// </summary>
/// <remarks> /// <remarks>
/// <para> /// <para>
/// <b>Nothing here guesses.</b> The transport it replaces — an Akka ClusterClient receptionist — /// <b>Nothing here guesses.</b> The transport it replaced — an Akka ClusterClient receptionist —
/// dropped an undeliverable command silently, and the entire point of the migration is to stop /// dropped an undeliverable command silently, and the entire point of the migration is to stop
/// doing that. So every decode that cannot produce a faithful record throws: an unset oneof, an /// doing that. So every decode that cannot produce a faithful record throws: an unset oneof, an
/// <c>APPLY_OUTCOME_UNSPECIFIED</c> ack, an enum value from a newer peer. A loud failure on one /// <c>APPLY_OUTCOME_UNSPECIFIED</c> ack, an enum value from a newer peer. A loud failure on one
@@ -6,9 +6,9 @@ option csharp_namespace = "ZB.MOM.WW.OtOpcUa.Commons.Protos.MeshTransport.V1";
import "google/protobuf/timestamp.proto"; import "google/protobuf/timestamp.proto";
// ClusterClient→gRPC migration Phase 0: the cross-mesh command plane. The five messages that cross // The cross-mesh command plane (introduced in Phase 0, now the only transport that crosses an
// an application-Cluster boundary today ride an Akka ClusterClient receptionist; this contract is // application-Cluster boundary). Five messages ride it: four central→node commands and one
// the replacement carrier. Four are central→node commands, one is the node→central deployment ack. // node→central deployment ack.
// //
// The C# namespace deliberately says "MeshTransport", never "MeshCommand" — the latter is already // The C# namespace deliberately says "MeshTransport", never "MeshCommand" — the latter is already
// the CLR envelope record ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh.MeshCommand, and a generated type // the CLR envelope record ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh.MeshCommand, and a generated type
@@ -20,17 +20,17 @@ import "google/protobuf/timestamp.proto";
// MeshTransportProtoContract + its test in the Commons test project. // MeshTransportProtoContract + its test in the Commons test project.
// Central → node. Delivered to the node's NodeCommunicationActor, which re-emits the decoded message // Central → node. Delivered to the node's NodeCommunicationActor, which re-emits the decoded message
// on the node's EventStream. ONE-WAY semantics, preserved verbatim from the ClusterClient transport: // on the node's EventStream. ONE-WAY semantics, matching the node-local DistributedPubSub leg: the
// the ack means "received and re-emitted", nothing more — not applied, not acknowledged by any // ack means "received and re-emitted", nothing more — not applied, not acknowledged by any
// subscriber. No retry is layered on a failure; a failed Execute is logged per node and dropped, // subscriber. No retry is layered on a failure; a failed Execute is logged per node and dropped,
// which is an honest upgrade over the receptionist's silent drop rather than a semantic change. // which is an honest upgrade over a pub/sub publish nobody was listening for.
service NodeCommandService { service NodeCommandService {
rpc Execute (CommandEnvelope) returns (ExecuteAck); rpc Execute (CommandEnvelope) returns (ExecuteAck);
} }
// Node → central. Delivered to central's CentralCommunicationActor, which forwards to the // Node → central. Delivered to central's CentralCommunicationActor, which forwards to the
// ConfigPublishCoordinator singleton proxy — so an ack landing on either node of the central pair // ConfigPublishCoordinator singleton proxy — so an ack landing on either node of the central pair
// routes identically, exactly as the receptionist made arrival-node-agnostic. ONE-WAY semantics: the // routes identically — which central node answered is deliberately not observable. ONE-WAY semantics: the
// ack means "handed to the coordinator", not "the deployment is complete". // ack means "handed to the coordinator", not "the deployment is complete".
service CentralCommandService { service CentralCommandService {
rpc ReportApply (ApplyAckMessage) returns (ReportAck); rpc ReportApply (ApplyAckMessage) returns (ReportAck);
@@ -22,7 +22,7 @@ public sealed class ClusterNode
/// Akka remoting port (default 4053). <b>This is central's dial target, not the node's own /// Akka remoting port (default 4053). <b>This is central's dial target, not the node's own
/// binding configuration</b> — the node binds from <c>Cluster:Port</c> in its own appsettings, /// binding configuration</b> — the node binds from <c>Cluster:Port</c> in its own appsettings,
/// and this column duplicates that value for a reader that cannot see the node's config. /// and this column duplicates that value for a reader that cannot see the node's config.
/// Per-cluster mesh Phase 2 builds its ClusterClient contact points from /// Per-cluster mesh Phase 2 built its Akka contact points from
/// <see cref="Host"/> + this port, at which point central and the site node no longer share a /// <see cref="Host"/> + this port, at which point central and the site node no longer share a
/// gossip ring and central has no other way to learn it. /// gossip ring and central has no other way to learn it.
/// </summary> /// </summary>
@@ -17,10 +17,10 @@ using Status = Grpc.Core.Status;
namespace ZB.MOM.WW.OtOpcUa.AdminUI.Grpc; namespace ZB.MOM.WW.OtOpcUa.AdminUI.Grpc;
/// <summary> /// <summary>
/// Central-side ingress for a node's deployment acknowledgment (ClusterClient→gRPC migration /// Central-side ingress for a node's deployment acknowledgment (mesh gRPC command plane
/// Phase 1B). Decodes an <see cref="ApplyAckMessage"/> and hands it to the node-local /// Phase 1B). Decodes an <see cref="ApplyAckMessage"/> and hands it to the node-local
/// <c>CentralCommunicationActor</c> — the same actor an Akka ClusterClient receptionist delivered /// <c>CentralCommunicationActor</c> — the same actor the node-local DistributedPubSub path
/// to, whose forward-to-the-coordinator path is untouched. Registered only on admin-role nodes /// delivers to, whose forward-to-the-coordinator path is untouched. Registered only on admin-role nodes
/// and only when <c>ConfigServe:GrpcListenPort &gt; 0</c>. /// and only when <c>ConfigServe:GrpcListenPort &gt; 0</c>.
/// </summary> /// </summary>
/// <remarks> /// <remarks>
@@ -52,7 +52,7 @@ namespace ZB.MOM.WW.OtOpcUa.AdminUI.Grpc;
/// <para> /// <para>
/// <b>One-way semantics, preserved.</b> <c>Tell</c>, not <c>Ask</c>: the returned /// <b>One-way semantics, preserved.</b> <c>Tell</c>, not <c>Ask</c>: the returned
/// <see cref="ReportAck"/> means "handed to the comm actor", never "the deployment is /// <see cref="ReportAck"/> means "handed to the comm actor", never "the deployment is
/// complete" — the same promise the receptionist made. The comm actor forwards to the /// complete" — the same promise the DistributedPubSub leg makes. The comm actor forwards to the
/// coordinator singleton proxy, so an ack landing on either central node routes identically. /// coordinator singleton proxy, so an ack landing on either central node routes identically.
/// </para> /// </para>
/// </remarks> /// </remarks>
@@ -92,8 +92,7 @@ public sealed class AdminOperationsActor : ReceiveActor
/// <b>retained</b>, not scheduled for deletion — an earlier note here promised Phase 6 would /// <b>retained</b>, not scheduled for deletion — an earlier note here promised Phase 6 would
/// remove it, which was deliberately reversed (<c>docs/Configuration.md</c> § "The compiled /// remove it, which was deliberately reversed (<c>docs/Configuration.md</c> § "The compiled
/// transport-mode defaults were deliberately NOT flipped"): DPS is still the correct and /// transport-mode defaults were deliberately NOT flipped"): DPS is still the correct and
/// default transport for a single-mesh topology. What is being replaced is the ClusterClient /// default transport for a single-mesh topology; gRPC is what crosses a mesh boundary.
/// branch, by gRPC.
/// </remarks> /// </remarks>
/// <param name="topic">The legacy DistributedPubSub topic.</param> /// <param name="topic">The legacy DistributedPubSub topic.</param>
/// <param name="message">The command.</param> /// <param name="message">The command.</param>
@@ -1,6 +1,4 @@
using System.Collections.Immutable;
using Akka.Actor; using Akka.Actor;
using Akka.Cluster.Tools.Client;
using Akka.Cluster.Tools.PublishSubscribe; using Akka.Cluster.Tools.PublishSubscribe;
using Akka.Event; using Akka.Event;
using Grpc.Core; using Grpc.Core;
@@ -20,39 +18,30 @@ namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
/// <summary> /// <summary>
/// Central-side end of the central↔node command boundary (per-cluster mesh Phase 2). Routes /// Central-side end of the central↔node command boundary (per-cluster mesh Phase 2). Routes
/// every central→node command out — over DistributedPubSub, ClusterClient or gRPC, per /// every central→node command out — over DistributedPubSub (pair-local) or gRPC (cross-mesh),
/// <see cref="MeshTransportOptions.Mode"/> — and forwards inbound <see cref="ApplyAck"/>s to the /// per <see cref="MeshTransportOptions.Mode"/> — and forwards inbound <see cref="ApplyAck"/>s to
/// deploy coordinator singleton. /// the deploy coordinator singleton.
/// </summary> /// </summary>
/// <remarks> /// <remarks>
/// <para> /// <para>
/// <b>Registered per node, NOT as a cluster singleton.</b> A driver node's ClusterClient /// <b>Registered per node, NOT as a cluster singleton.</b> A driver node posts its ack to
/// rotates across contact points and must find a live comm actor at whichever central node /// whichever central node answers first, so a live comm actor must exist on both. A singleton
/// answers. A singleton would be reachable only through the one node hosting it, and the /// would be reachable only through the one node hosting it, and the other central node's ack
/// rotation would fail silently against the other. /// ingress would have nothing to forward to.
/// </para> /// </para>
/// <para> /// <para>
/// <b>One ClusterClient per application <c>Cluster</c> (Phase 6).</b> The fleet is now one /// <b>The gRPC branch fans node by node over the contact load's target set.</b> That load
/// Akka mesh per cluster, so a single fleet-wide client's <c>SendToAll</c> would reach only /// selects exactly the enabled, non-maintenance <c>ClusterNode</c> rows carrying a
/// the one mesh whose receptionist answered. This actor therefore holds a client per cluster /// <c>GrpcPort</c>, and the fan dials every one of them, one call per node. It is deliberately
/// and fans <c>SendToAll</c> across all of them. See <see cref="RebuildClient"/> — this is the /// NOT narrower: the node side has no <c>ClusterId</c> or node filter at all — scoping happens
/// single most important thing to understand before changing this actor. /// later, inside the artifact — so a node left out here silently keeps its old configuration
/// while the deployment seals green.
/// </para> /// </para>
/// <para> /// <para>
/// <b>The gRPC branch fans over the same recipient set, node by node</b> /// <b>Fire-and-forget, on both transports.</b> The gRPC branch pipes per-node results back to
/// (ClusterClient→gRPC Phase 1A). <c>SendToAll</c> reaches every node registered on a mesh's
/// receptionist, and the contact load already selects exactly the enabled, non-maintenance
/// rows those nodes come from; the gRPC branch dials that same row set directly, one call per
/// node, so the recipient set is identical by construction rather than by coincidence. It is
/// deliberately NOT narrower: the node side has no <c>ClusterId</c> or node filter at all —
/// scoping happens later, inside the artifact — so a node left out here silently keeps its
/// old configuration while the deployment seals green.
/// </para>
/// <para>
/// <b>Fire-and-forget, on every transport.</b> The gRPC branch pipes per-node results back to
/// itself for logging ONLY: a failure names the node and the status and is dropped. No retry, /// itself for logging ONLY: a failure names the node and the status and is dropped. No retry,
/// no queue, no ordering guarantee across nodes. That is not an omission — it is the /// no queue, no ordering guarantee across nodes. That is not an omission — it is the
/// ClusterClient semantic preserved exactly, and the backstops are unchanged (the deploy /// DistributedPubSub semantic preserved exactly, and the backstops are unchanged (the deploy
/// coordinator's apply deadline, and a node's boot-time replay). /// coordinator's apply deadline, and a node's boot-time replay).
/// </para> /// </para>
/// </remarks> /// </remarks>
@@ -62,7 +51,6 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
private readonly IDbContextFactory<OtOpcUaConfigDbContext> _dbFactory; private readonly IDbContextFactory<OtOpcUaConfigDbContext> _dbFactory;
private readonly MeshTransportOptions _options; private readonly MeshTransportOptions _options;
private readonly IMeshClusterClientFactory _clientFactory;
private readonly Func<IActorRef?> _coordinator; private readonly Func<IActorRef?> _coordinator;
// Resolved in PreStart, never at construction. See the Props doc: the DI graph behind these two // Resolved in PreStart, never at construction. See the Props doc: the DI graph behind these two
@@ -88,15 +76,8 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
_dialer ??= Resolve(_dialerFactory, nameof(IMeshCommandDialer), firstUse: true); _dialer ??= Resolve(_dialerFactory, nameof(IMeshCommandDialer), firstUse: true);
private readonly ILoggingAdapter _log = Context.GetLogger(); private readonly ILoggingAdapter _log = Context.GetLogger();
private readonly bool _useClusterClient;
private readonly bool _useGrpc; private readonly bool _useGrpc;
// One client per application Cluster, keyed by ClusterId. A ClusterClient reaches only the mesh
// whose receptionist it dialled, and after Phase 6 every cluster is its own mesh — so central must
// hold a client per cluster and fan SendToAll across all of them.
private readonly Dictionary<string, (IActorRef Client, ImmutableHashSet<string> Contacts)> _clients =
new();
// Node ids already warned about for a null GrpcPort. A persistently portless row would otherwise // Node ids already warned about for a null GrpcPort. A persistently portless row would otherwise
// produce one warning per refresh forever; same warn-once shape as TelemetryNodeSource. Actor // produce one warning per refresh forever; same warn-once shape as TelemetryNodeSource. Actor
// thread only. // thread only.
@@ -121,7 +102,6 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
/// <summary>Creates the props for this actor.</summary> /// <summary>Creates the props for this actor.</summary>
/// <param name="dbFactory">Factory for the config database holding <c>ClusterNode</c> rows.</param> /// <param name="dbFactory">Factory for the config database holding <c>ClusterNode</c> rows.</param>
/// <param name="options">The bound mesh-transport options.</param> /// <param name="options">The bound mesh-transport options.</param>
/// <param name="clientFactory">Creates the ClusterClient; substituted in tests.</param>
/// <param name="coordinator"> /// <param name="coordinator">
/// Resolves the deploy-coordinator singleton proxy. A <see cref="Func{TResult}"/> rather than /// Resolves the deploy-coordinator singleton proxy. A <see cref="Func{TResult}"/> rather than
/// an <see cref="IActorRef"/> so this actor never depends on registration order at wiring /// an <see cref="IActorRef"/> so this actor never depends on registration order at wiring
@@ -129,9 +109,9 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
/// </param> /// </param>
/// <param name="dialer"> /// <param name="dialer">
/// Resolves the thing that places one command call on one node, under /// Resolves the thing that places one command call on one node, under
/// <see cref="MeshTransportOptions.ModeGrpc"/>. Optional so the two older transports need /// <see cref="MeshTransportOptions.ModeGrpc"/>. Optional so the Dps transport needs nothing
/// nothing new wired; a <see langword="null"/> dialer under <c>Grpc</c> is a mis-wiring and /// new wired; a <see langword="null"/> dialer under <c>Grpc</c> is a mis-wiring and is
/// is reported loudly rather than silently dropping every command. /// reported loudly rather than silently dropping every command.
/// </param> /// </param>
/// <param name="pskProvider"> /// <param name="pskProvider">
/// Resolves central's per-cluster key lookup, told on every contact refresh which clusters /// Resolves central's per-cluster key lookup, told on every contact refresh which clusters
@@ -163,37 +143,30 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
public static Props Props( public static Props Props(
IDbContextFactory<OtOpcUaConfigDbContext> dbFactory, IDbContextFactory<OtOpcUaConfigDbContext> dbFactory,
MeshTransportOptions options, MeshTransportOptions options,
IMeshClusterClientFactory clientFactory,
Func<IActorRef?> coordinator, Func<IActorRef?> coordinator,
Func<IMeshCommandDialer?>? dialer = null, Func<IMeshCommandDialer?>? dialer = null,
Func<IMeshPskProvider?>? pskProvider = null) => Func<IMeshPskProvider?>? pskProvider = null) =>
Akka.Actor.Props.Create(() => Akka.Actor.Props.Create(() =>
new CentralCommunicationActor( new CentralCommunicationActor(dbFactory, options, coordinator, dialer, pskProvider));
dbFactory, options, clientFactory, coordinator, dialer, pskProvider));
/// <summary>Initializes a new instance of the <see cref="CentralCommunicationActor"/> class.</summary> /// <summary>Initializes a new instance of the <see cref="CentralCommunicationActor"/> class.</summary>
/// <param name="dbFactory">Factory for the config database.</param> /// <param name="dbFactory">Factory for the config database.</param>
/// <param name="options">The bound mesh-transport options.</param> /// <param name="options">The bound mesh-transport options.</param>
/// <param name="clientFactory">Creates the ClusterClient.</param>
/// <param name="coordinator">Lazily resolves the deploy-coordinator singleton proxy.</param> /// <param name="coordinator">Lazily resolves the deploy-coordinator singleton proxy.</param>
/// <param name="dialer">Resolves the gRPC command dialer; invoked in <see cref="PreStart"/>.</param> /// <param name="dialer">Resolves the gRPC command dialer; invoked in <see cref="PreStart"/>.</param>
/// <param name="pskProvider">Resolves the PSK provider; invoked in <see cref="PreStart"/>.</param> /// <param name="pskProvider">Resolves the PSK provider; invoked in <see cref="PreStart"/>.</param>
public CentralCommunicationActor( public CentralCommunicationActor(
IDbContextFactory<OtOpcUaConfigDbContext> dbFactory, IDbContextFactory<OtOpcUaConfigDbContext> dbFactory,
MeshTransportOptions options, MeshTransportOptions options,
IMeshClusterClientFactory clientFactory,
Func<IActorRef?> coordinator, Func<IActorRef?> coordinator,
Func<IMeshCommandDialer?>? dialer = null, Func<IMeshCommandDialer?>? dialer = null,
Func<IMeshPskProvider?>? pskProvider = null) Func<IMeshPskProvider?>? pskProvider = null)
{ {
_dbFactory = dbFactory; _dbFactory = dbFactory;
_options = options; _options = options;
_clientFactory = clientFactory;
_coordinator = coordinator; _coordinator = coordinator;
_dialerFactory = dialer; _dialerFactory = dialer;
_pskProviderFactory = pskProvider; _pskProviderFactory = pskProvider;
_useClusterClient = string.Equals(
options.Mode, MeshTransportOptions.ModeClusterClient, StringComparison.OrdinalIgnoreCase);
_useGrpc = string.Equals( _useGrpc = string.Equals(
options.Mode, MeshTransportOptions.ModeGrpc, StringComparison.OrdinalIgnoreCase); options.Mode, MeshTransportOptions.ModeGrpc, StringComparison.OrdinalIgnoreCase);
@@ -246,13 +219,13 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
_dialer = Resolve(_dialerFactory, nameof(IMeshCommandDialer), firstUse: false); _dialer = Resolve(_dialerFactory, nameof(IMeshCommandDialer), firstUse: false);
_pskProvider = Resolve(_pskProviderFactory, nameof(IMeshPskProvider), firstUse: false); _pskProvider = Resolve(_pskProviderFactory, nameof(IMeshPskProvider), firstUse: false);
if (!_useClusterClient && !_useGrpc) if (!_useGrpc)
{ {
_log.Info( _log.Info(
"Mesh transport is {Mode}: commands are published on DistributedPubSub and no " "Mesh transport is {Mode}: commands are published on DistributedPubSub and nothing "
+ "ClusterClient is created. The contact refresh still runs — it is what keeps the " + "is dialled. The contact refresh still runs — it is what keeps the known-cluster "
+ "known-cluster gate current, and that gate belongs to the inbound ack endpoint, " + "gate current, and that gate belongs to the inbound ack endpoint, which is mapped "
+ "which is mapped by port and does not care about this mode", _options.Mode); + "by port and does not care about this mode", _options.Mode);
} }
// Deliberately NO null guard for either dependency here. At this point a still-null value // Deliberately NO null guard for either dependency here. At this point a still-null value
@@ -266,7 +239,7 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
// Started in EVERY mode. The refresh is not transport state: it is central's ClusterNode // Started in EVERY mode. The refresh is not transport state: it is central's ClusterNode
// inventory, and the known-cluster gate it feeds protects an endpoint mapped by PORT, not by // inventory, and the known-cluster gate it feeds protects an endpoint mapped by PORT, not by
// MeshTransport:Mode. Gating the timer on the mode would mean a dark-switch rollback to // MeshTransport:Mode. Gating the timer on the mode would mean a dark-switch rollback to
// ClusterClient or Dps left that endpoint live with the gate permanently inactive — silently // Dps left that endpoint live with the gate permanently inactive — silently
// reopening both the timing oracle and the pre-auth store/audit flood. Under Dps this is a // reopening both the timing oracle and the pre-auth store/audit flood. Under Dps this is a
// ConfigDb query every ContactRefreshSeconds that did not run before; central always has a // ConfigDb query every ContactRefreshSeconds that did not run before; central always has a
// ConfigDb, and one indexed read a minute is the cheaper half of the trade. // ConfigDb, and one indexed read a minute is the cheaper half of the trade.
@@ -357,9 +330,6 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
/// <inheritdoc /> /// <inheritdoc />
protected override void PostStop() protected override void PostStop()
{ {
foreach (var (client, _) in _clients.Values) Context.Stop(client);
_clients.Clear();
// Cancelling the lifecycle token also cancels every gRPC dial in flight. The dialer itself is // Cancelling the lifecycle token also cancels every gRPC dial in flight. The dialer itself is
// deliberately NOT disposed here: it is a container-owned singleton shared with anything else // deliberately NOT disposed here: it is a container-owned singleton shared with anything else
// that may dial, and an actor restart must not tear its channel pool down. // that may dial, and an actor restart must not tear its channel pool down.
@@ -369,43 +339,15 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
private void HandleMeshCommand(MeshCommand cmd) private void HandleMeshCommand(MeshCommand cmd)
{ {
// Checked before the DPS arm: _useClusterClient is false under Grpc too, so the order is // Checked first, so a Grpc-mode command never reaches DistributedPubSub. DPS is mesh-wide and
// what keeps a Grpc-mode command off DistributedPubSub. // cannot cross an application-Cluster boundary at all in the split topology.
if (_useGrpc) if (_useGrpc)
{ {
DispatchOverGrpc(cmd); DispatchOverGrpc(cmd);
return; return;
} }
if (!_useClusterClient) DistributedPubSub.Get(Context.System).Mediator.Tell(new Publish(cmd.Topic, cmd.Message));
{
DistributedPubSub.Get(Context.System).Mediator.Tell(new Publish(cmd.Topic, cmd.Message));
return;
}
if (_clients.Count == 0)
{
// App-level drop, matching the sister project's decision and this repo's buffer-size = 0.
// The command is NOT queued and will NOT be retried: a deploy fails at the coordinator's
// apply deadline naming the silent nodes, which is a better operator signal than a
// command arriving minutes late against a config the DB already recorded as failed.
_log.Warning(
"No mesh ClusterClient — dropping {MessageType}. The last contact refresh produced "
+ "no usable contact point from the enabled, non-maintenance ClusterNode rows in any "
+ "cluster; the command is not buffered and will not be retried",
cmd.Message.GetType().Name);
return;
}
// SendToAll, NEVER Send, fanned across EVERY cluster's client. Today's DPS publish reaches
// EVERY DriverHostActor — there is no ClusterId or node filter on the node side at all; scoping
// happens later, inside the artifact (DeploymentArtifact.ResolveClusterScope). Each ClusterClient
// reaches only the one mesh it dialled, so central must fan across all of them to preserve the
// fleet-wide reach DPS gives. ClusterClient.Send delivers to exactly ONE registered actor, which
// would deploy to a single node while every other node silently kept its old configuration — and
// the deployment would still seal green if the ack set happened to be satisfied.
foreach (var (client, _) in _clients.Values)
client.Tell(new ClusterClient.SendToAll(MeshPaths.NodeCommunication, cmd.Message));
} }
/// <summary> /// <summary>
@@ -453,10 +395,10 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
var targets = _targets; var targets = _targets;
if (targets.Count == 0) if (targets.Count == 0)
{ {
// App-level drop, exactly as the ClusterClient arm does with no client. The command is // App-level drop. The command is NOT queued and will NOT be retried: a deploy fails at
// NOT queued and will NOT be retried: a deploy fails at the coordinator's apply deadline // the coordinator's apply deadline naming the silent nodes, which is a better operator
// naming the silent nodes, which is a better operator signal than a command arriving // signal than a command arriving minutes late against a config the DB already recorded
// minutes late against a config the DB already recorded as failed. // as failed.
_log.Warning( _log.Warning(
"No mesh gRPC targets — dropping {MessageType}. The last contact refresh produced no " "No mesh gRPC targets — dropping {MessageType}. The last contact refresh produced no "
+ "enabled, non-maintenance ClusterNode row carrying a GrpcPort in any cluster; the " + "enabled, non-maintenance ClusterNode row carrying a GrpcPort in any cluster; the "
@@ -632,7 +574,6 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
return; // Stopping. return; // Stopping.
} }
var systemName = Context.System.Name;
var dbFactory = _dbFactory; var dbFactory = _dbFactory;
// Off the actor thread, result piped back as a message. A synchronous DB read here would // Off the actor thread, result piped back as a message. A synchronous DB read here would
@@ -641,29 +582,21 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
{ {
await using var db = await dbFactory.CreateDbContextAsync(ct).ConfigureAwait(false); await using var db = await dbFactory.CreateDbContextAsync(ct).ConfigureAwait(false);
// UNFILTERED, with the eligibility flags projected and applied per row below. The two // UNFILTERED, with the eligibility flags projected and applied per row below. The two
// outputs of this read want different row sets: the contact/dispatch sets are the // outputs of this read want different row sets: the dispatch set is the enabled,
// enabled, non-maintenance nodes central talks TO, while the known-cluster gate answers // non-maintenance nodes central talks TO, while the known-cluster gate answers "is this a
// "is this a cluster central has ever heard of" and must include a cluster whose every // cluster central has ever heard of" and must include a cluster whose every node is
// node is disabled or in maintenance — such a cluster can still send an ack, and denying // disabled or in maintenance — such a cluster can still send an ack, and denying it would
// it would make maintenance mode an authentication switch. // make maintenance mode an authentication switch.
var rows = await db.ClusterNodes var rows = await db.ClusterNodes
.AsNoTracking() .AsNoTracking()
.Select(n => new .Select(n => new
{ {
n.NodeId, n.ClusterId, n.Host, n.AkkaPort, n.GrpcPort, n.Enabled, n.MaintenanceMode, n.NodeId, n.ClusterId, n.Host, n.GrpcPort, n.Enabled, n.MaintenanceMode,
}) })
.ToListAsync(ct) .ToListAsync(ct)
.ConfigureAwait(false); .ConfigureAwait(false);
// Grouped BY ClusterId: central now dials one ClusterClient per application Cluster, so the // The gRPC dispatch set: one entry per eligible row that carries a port.
// contact set must be partitioned the same way. A row's contacts join only its own cluster's
// client.
var byCluster = new Dictionary<string, List<string>>();
var malformed = new List<string>();
// The gRPC dispatch set, built from the SAME row set in the SAME pass. Sharing the read is
// what makes "the gRPC fan reaches exactly the ClusterClient recipient set" structural
// rather than a claim two separate queries would have to keep agreeing on.
var targets = new List<MeshNodeTarget>(rows.Count); var targets = new List<MeshNodeTarget>(rows.Count);
var portless = new List<string>(); var portless = new List<string>();
@@ -677,8 +610,7 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
clusterIds.Add(row.ClusterId); clusterIds.Add(row.ClusterId);
// The eligibility filter that used to be the query's WHERE clause. Everything below // The eligibility filter that used to be the query's WHERE clause. Everything below
// this line — receptionist contacts and gRPC targets — sees exactly the rows it saw // this line — the gRPC targets — sees exactly the rows it saw before.
// before.
if (!row.Enabled || row.MaintenanceMode) if (!row.Enabled || row.MaintenanceMode)
continue; continue;
@@ -687,29 +619,9 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
row.NodeId, row.ClusterId, $"http://{row.Host}:{grpcPort}")); row.NodeId, row.ClusterId, $"http://{row.Host}:{grpcPort}"));
else else
portless.Add(row.NodeId); portless.Add(row.NodeId);
var address = $"akka.tcp://{systemName}@{row.Host}:{row.AkkaPort}/system/receptionist";
// Parse up front, per row, inside the loop's own guard. A single malformed row must
// not abort the whole refresh and leave the contact set half-built — the sister
// project shipped that bug and its regression test deliberately orders the bad row
// first.
if (ActorPath.TryParse(address, out _))
{
if (!byCluster.TryGetValue(row.ClusterId, out var list))
byCluster[row.ClusterId] = list = new List<string>();
list.Add(address);
}
else
{
malformed.Add($"{row.NodeId} ({row.ClusterId}) -> {address}");
}
} }
var contactsByCluster = byCluster.ToDictionary( return new ContactsLoaded(targets, portless, clusterIds.ToList());
kv => kv.Key, kv => (IReadOnlyList<string>)kv.Value);
return new ContactsLoaded(
contactsByCluster, malformed, targets, portless, clusterIds.ToList());
}, ct).PipeTo(self); }, ct).PipeTo(self);
} }
@@ -722,66 +634,6 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
// EVERY mode. See PreStart — the gate protects an endpoint mapped by port, not by mode. // EVERY mode. See PreStart — the gate protects an endpoint mapped by port, not by mode.
PublishKnownClusters(msg); PublishKnownClusters(msg);
if (!_useClusterClient)
return; // Dps or Grpc: no ClusterClient to rebuild.
foreach (var bad in msg.Malformed)
{
_log.Warning(
"ClusterNode {Row} does not yield a usable receptionist address; skipping it in this "
+ "refresh (other nodes are unaffected). Check the row's Host and AkkaPort", bad);
}
var byCluster = msg.ContactsByCluster.ToDictionary(
kv => kv.Key, kv => kv.Value.ToImmutableHashSet());
// Clusters that dropped out of the DB entirely, or whose only rows were malformed this refresh:
// stop and forget their clients so a vanished cluster stops receiving. LoadContactsFromDb only
// creates a ByCluster key when at least one of its rows parsed, so an all-malformed cluster has
// NO key at all and is caught here by the absent-key (!TryGetValue) arm — not by an empty value.
var gone = _clients.Keys
.Where(id => !byCluster.TryGetValue(id, out var c) || c.IsEmpty)
.ToList();
foreach (var clusterId in gone)
{
_log.Info(
"Cluster {ClusterId} no longer yields any usable contact point; stopping its "
+ "ClusterClient", clusterId);
Context.Stop(_clients[clusterId].Client);
_clients.Remove(clusterId);
}
var anyUsable = false;
foreach (var (clusterId, contacts) in byCluster)
{
if (contacts.IsEmpty)
{
// Defensive: LoadContactsFromDb never emits an empty-but-present cluster (a key is
// created only on a first successfully-parsed row), so this arm is unreachable for the
// DB producer — an all-malformed cluster is handled by the absent-key branch above. It
// guards a ContactsLoaded (a public record) constructed elsewhere with an empty list.
_log.Warning(
"Cluster {ClusterId} has no usable ClusterNode contact points; no ClusterClient "
+ "was created for it and its nodes' commands will be dropped until a refresh finds "
+ "one", clusterId);
continue;
}
anyUsable = true;
if (_clients.TryGetValue(clusterId, out var existing) && existing.Contacts.SetEquals(contacts))
continue;
RebuildClient(clusterId, contacts);
}
if (!anyUsable && _clients.Count == 0)
{
_log.Warning(
"No usable ClusterNode contact points in any cluster; no mesh ClusterClient was "
+ "created and every central→node command will be dropped until a refresh finds one");
}
} }
/// <summary> /// <summary>
@@ -790,11 +642,9 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
/// </summary> /// </summary>
/// <param name="msg">The load result.</param> /// <param name="msg">The load result.</param>
/// <remarks> /// <remarks>
/// The malformed-receptionist-address warnings are deliberately NOT logged here: nothing on /// A host that is unusable as a gRPC endpoint is deliberately NOT screened here: it surfaces
/// this transport dials a receptionist, and reporting an unusable Akka address as a /// as a named per-node dial failure on every dispatch, which is a truer signal than a
/// command-plane problem would send an operator after the wrong column. A host that is /// refresh-time warning about a string that may well be fine.
/// unusable as a gRPC endpoint surfaces instead as a named per-node dial failure on every
/// dispatch, which is a truer signal.
/// </remarks> /// </remarks>
private void AdoptGrpcTargets(ContactsLoaded msg) private void AdoptGrpcTargets(ContactsLoaded msg)
{ {
@@ -837,7 +687,7 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
/// <b>Not gated on <see cref="MeshTransportOptions.Mode"/>, on purpose.</b> The gate feeds /// <b>Not gated on <see cref="MeshTransportOptions.Mode"/>, on purpose.</b> The gate feeds
/// <c>CentralCommandAuthInterceptor</c>, which is registered on every admin node, and the /// <c>CentralCommandAuthInterceptor</c>, which is registered on every admin node, and the
/// ack service it protects is mapped by listener port — neither consults the mode. Gating /// ack service it protects is mapped by listener port — neither consults the mode. Gating
/// this on <c>Grpc</c> would mean a dark-switch rollback to <c>ClusterClient</c> or /// this on <c>Grpc</c> would mean a dark-switch rollback to
/// <c>Dps</c> left that endpoint serving with a permanently inactive gate: the timing /// <c>Dps</c> left that endpoint serving with a permanently inactive gate: the timing
/// oracle and the pre-authentication store/audit flood would both come back, silently, /// oracle and the pre-authentication store/audit flood would both come back, silently,
/// as a side effect of a rollback nobody would connect to authentication. /// as a side effect of a rollback nobody would connect to authentication.
@@ -887,75 +737,13 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
pskProvider.InvalidateAll(); pskProvider.InvalidateAll();
} }
/// <summary>Stops a cluster's existing client and creates a replacement for its new contact set.</summary>
/// <remarks>
/// <para>
/// <b>One client per application <c>Cluster</c> (delivered in Phase 6).</b> This actor now
/// holds one <c>ClusterClient</c> per cluster and <see cref="HandleMeshCommand"/> fans
/// <c>SendToAll</c> across all of them. Once the fleet is split into one Akka mesh per
/// cluster, a <c>ClusterClientReceptionist</c> serves only <i>its own</i> mesh, so a single
/// client's <c>SendToAll</c> would reach the one mesh whose receptionist answered and leave
/// every other cluster silently on its old configuration. Fanning across a client per
/// cluster restores the fleet-wide reach the pre-split DPS broadcast gave.
/// </para>
/// <para>
/// Why this is not N× duplicate dispatch: each client dials only its own cluster's
/// receptionists, so a command is delivered <b>once per mesh</b>, not once per cluster into
/// every mesh. (While the fleet was a single mesh, the same code would have duplicated —
/// which is exactly why the fan-out could not land until the meshes were partitioned.)
/// </para>
/// <para>
/// Corollary worth stating because it is easy to assume otherwise: <b>the contact set
/// does not scope delivery within a mesh.</b> Excluding a maintenance-mode node from a
/// cluster's contacts does not stop it receiving commands — its receptionist still delivers
/// to every registered node-comm actor in that mesh. The filter is about which receptionists
/// are worth <i>dialling</i> (a node in maintenance may be switched off), not about who gets
/// the message.
/// </para>
/// </remarks>
/// <param name="clusterId">The application <c>Cluster</c> whose client is being (re)built.</param>
/// <param name="contacts">The receptionist addresses to dial for that cluster.</param>
private void RebuildClient(string clusterId, ImmutableHashSet<string> contacts)
{
if (_clients.TryGetValue(clusterId, out var existing))
{
_log.Info("Contact set for cluster {ClusterId} changed; replacing its ClusterClient", clusterId);
Context.Stop(existing.Client);
// Removed now: if the create below throws, a stale ref would route commands into a
// stopping actor and they would vanish without a warning.
_clients.Remove(clusterId);
}
try
{
var paths = contacts.Select(ActorPath.Parse).ToImmutableHashSet();
var client = _clientFactory.Create(Context.System, clusterId, paths);
_clients[clusterId] = (client, contacts);
_log.Info(
"Mesh ClusterClient for cluster {ClusterId} created with {Count} contact point(s)",
clusterId, contacts.Count);
}
catch (Exception ex)
{
_log.Error(ex,
"Failed to create the mesh ClusterClient for cluster {ClusterId}; its nodes' commands "
+ "are dropped until the next contact refresh succeeds", clusterId);
}
}
/// <summary>Internal tick asking for a contact-point refresh.</summary> /// <summary>Internal tick asking for a contact-point refresh.</summary>
public sealed record RefreshContacts; public sealed record RefreshContacts;
/// <summary>Result of a contact-point load, piped back from the DB read.</summary> /// <summary>Result of a contact-point load, piped back from the DB read.</summary>
/// <param name="ContactsByCluster">
/// Receptionist addresses that parsed cleanly, grouped by the owning application
/// <c>Cluster</c>'s id. One <c>ClusterClient</c> is built per key.
/// </param>
/// <param name="Malformed">Rows that did not parse, described for the log.</param>
/// <param name="Targets"> /// <param name="Targets">
/// The gRPC dispatch set: one entry per enabled, non-maintenance row that carries a /// The gRPC dispatch set: one entry per enabled, non-maintenance row that carries a
/// <c>GrpcPort</c>. Built from the same rows as <paramref name="ContactsByCluster"/>. /// <c>GrpcPort</c>.
/// </param> /// </param>
/// <param name="PortlessNodes"> /// <param name="PortlessNodes">
/// Node ids skipped because their row has no <c>GrpcPort</c>. Carried rather than logged in /// Node ids skipped because their row has no <c>GrpcPort</c>. Carried rather than logged in
@@ -968,8 +756,6 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
/// still send an ack, and refusing it would make maintenance mode an authentication switch. /// still send an ack, and refusing it would make maintenance mode an authentication switch.
/// </param> /// </param>
public sealed record ContactsLoaded( public sealed record ContactsLoaded(
IReadOnlyDictionary<string, IReadOnlyList<string>> ContactsByCluster,
IReadOnlyList<string> Malformed,
IReadOnlyList<MeshNodeTarget> Targets, IReadOnlyList<MeshNodeTarget> Targets,
IReadOnlyList<string> PortlessNodes, IReadOnlyList<string> PortlessNodes,
IReadOnlyList<string> ClusterIds); IReadOnlyList<string> ClusterIds);
@@ -1,67 +0,0 @@
using System.Collections.Immutable;
using Akka.Actor;
using Akka.Cluster.Tools.Client;
namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
/// <summary>
/// Creates the <c>ClusterClient</c> central uses to reach driver nodes. Exists as a seam so
/// <c>CentralCommunicationActor</c> can be tested against a <c>TestProbe</c> without standing up
/// a second cluster.
/// </summary>
public interface IMeshClusterClientFactory
{
/// <summary>Creates a client for the given receptionist contact points.</summary>
/// <param name="system">The actor system to spawn the client in.</param>
/// <param name="clusterId">
/// The application <c>Cluster</c> these contacts belong to. Central now creates one client per
/// cluster (Phase 6), so the id is woven into the client's actor name to keep the per-cluster
/// clients distinct and diagnosable.
/// </param>
/// <param name="contacts">Receptionist actor paths, one per reachable node.</param>
/// <returns>The new client's <see cref="IActorRef"/>.</returns>
IActorRef Create(ActorSystem system, string clusterId, ImmutableHashSet<ActorPath> contacts);
}
/// <inheritdoc />
public sealed class DefaultMeshClusterClientFactory : IMeshClusterClientFactory
{
/// <summary>
/// Per-incarnation generation counter, appended to the actor name.
/// </summary>
/// <remarks>
/// <c>Context.Stop</c> of the previous client is <b>asynchronous</b> — its actor name stays
/// reserved until termination completes — so recreating the same name while handling a single
/// message throws <see cref="InvalidActorNameException"/>. That is exactly what a contact-set
/// change does: stop the old client, create the replacement, both in one handler. A
/// generation suffix makes every incarnation's name unique by construction. Ported from the
/// sister project, where it was a shipped bug fix rather than foresight.
/// </remarks>
private long _generation;
/// <inheritdoc />
public IActorRef Create(ActorSystem system, string clusterId, ImmutableHashSet<ActorPath> contacts)
{
ArgumentNullException.ThrowIfNull(system);
ArgumentException.ThrowIfNullOrEmpty(clusterId);
ArgumentNullException.ThrowIfNull(contacts);
var settings = ClusterClientSettings.Create(system).WithInitialContacts(contacts);
var name = $"mesh-client-{SanitizeForActorName(clusterId)}-{Interlocked.Increment(ref _generation)}";
return system.ActorOf(ClusterClient.Props(settings), name);
}
/// <summary>
/// Reduces a cluster id to characters safe in an Akka actor name. The id is operator-authored
/// (e.g. a GUID or a slug), so it may carry characters Akka rejects; anything outside
/// <c>[A-Za-z0-9-]</c> becomes <c>-</c>. The trailing generation counter still guarantees
/// uniqueness, so a lossy sanitisation cannot cause a name collision.
/// </summary>
/// <param name="clusterId">The raw cluster id.</param>
/// <returns>An actor-name-safe rendering of the id.</returns>
private static string SanitizeForActorName(string clusterId)
{
var chars = clusterId.Select(c => char.IsAsciiLetterOrDigit(c) || c == '-' ? c : '-');
return new string(chars.ToArray());
}
}
@@ -10,7 +10,7 @@ namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
/// <summary> /// <summary>
/// Places one central→node command call on one node's <c>NodeCommandService</c> /// Places one central→node command call on one node's <c>NodeCommandService</c>
/// (ClusterClient→gRPC migration Phase 1A). The seam <c>CentralCommunicationActor</c> fans /// (mesh gRPC command plane, Phase 1A). The seam <c>CentralCommunicationActor</c> fans
/// across its target set, kept as an interface so the actor's tests never open a socket. /// across its target set, kept as an interface so the actor's tests never open a socket.
/// </summary> /// </summary>
/// <remarks> /// <remarks>
@@ -72,7 +72,7 @@ public sealed class ConfigPublishCoordinator : ReceiveActor, IWithTimers
/// not scheduled for deletion — an earlier note here promised Phase 6 would remove it, which /// not scheduled for deletion — an earlier note here promised Phase 6 would remove it, which
/// was deliberately reversed (<c>docs/Configuration.md</c> § "The compiled transport-mode /// was deliberately reversed (<c>docs/Configuration.md</c> § "The compiled transport-mode
/// defaults were deliberately NOT flipped"): DPS is still the correct and default transport /// defaults were deliberately NOT flipped"): DPS is still the correct and default transport
/// for a single-mesh topology. What is being replaced is the ClusterClient branch, by gRPC. /// for a single-mesh topology; gRPC is what crosses a mesh boundary.
/// </remarks> /// </remarks>
private readonly IActorRef? _meshRouter; private readonly IActorRef? _meshRouter;
private readonly ILoggingAdapter _log = Context.GetLogger(); private readonly ILoggingAdapter _log = Context.GetLogger();
@@ -1,6 +1,5 @@
using Akka.Actor; using Akka.Actor;
using Akka.Cluster.Hosting; using Akka.Cluster.Hosting;
using Akka.Cluster.Tools.Client;
using Akka.Hosting; using Akka.Hosting;
using Microsoft.EntityFrameworkCore; using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.Configuration; using Microsoft.Extensions.Configuration;
@@ -56,11 +55,11 @@ public static class ServiceCollectionExtensions
// Per-cluster mesh Phase 2: the central end of the central↔node command boundary. // Per-cluster mesh Phase 2: the central end of the central↔node command boundary.
// //
// A plain WithActors, NOT a singleton — and that is the load-bearing part. A driver node's // A plain WithActors, NOT a singleton — and that is the load-bearing part. A driver node
// ClusterClient rotates across contact points and must find a live comm actor at whichever // walks MeshTransport:CentralGrpcEndpoints in order and posts its ack to whichever central
// central node answers. As a singleton it would exist on one central node only, and the // node answers first, so a live comm actor must exist on both. As a singleton it would exist
// rotation would fail silently against the other: acks would land sometimes, depending on // on one central node only, and the other node's ack ingress would have nothing to forward
// which contact the client happened to settle on. // to: acks would land sometimes, depending on which endpoint answered.
builder.WithActors((system, registry, resolver) => builder.WithActors((system, registry, resolver) =>
{ {
var dbFactory = resolver.GetService<IDbContextFactory<OtOpcUaConfigDbContext>>(); var dbFactory = resolver.GetService<IDbContextFactory<OtOpcUaConfigDbContext>>();
@@ -85,7 +84,6 @@ public static class ServiceCollectionExtensions
CentralCommunicationActor.Props( CentralCommunicationActor.Props(
dbFactory, dbFactory,
options, options,
new DefaultMeshClusterClientFactory(),
// Resolved lazily, per message, so this actor never depends on the order in // Resolved lazily, per message, so this actor never depends on the order in
// which Akka.Hosting materialises the singleton proxy relative to this block. // which Akka.Hosting materialises the singleton proxy relative to this block.
() => registry.TryGet<ConfigPublishCoordinatorKey>(out var coordinator) () => registry.TryGet<ConfigPublishCoordinatorKey>(out var coordinator)
@@ -95,11 +93,9 @@ public static class ServiceCollectionExtensions
() => serviceProvider.GetService<IMeshPskProvider>()), () => serviceProvider.GetService<IMeshPskProvider>()),
MeshPaths.CentralCommunicationName); MeshPaths.CentralCommunicationName);
// Registered unconditionally, in both transport modes. Under Dps nothing dials it, and // The registry entry is the whole reachability story: CentralCommandGrpcService resolves
// an idle registration costs nothing — whereas registering only under ClusterClient // this key to hand an inbound ack over. Nothing registers with a receptionist any more —
// would mean flipping the flag on a running fleet requires a central restart before any // there is no Akka association between central and a site mesh to register into.
// node can ack, turning a config change back into a deployment.
ClusterClientReceptionist.Get(system).RegisterService(actor);
registry.Register<CentralCommunicationActorKey>(actor); registry.Register<CentralCommunicationActorKey>(actor);
}); });
@@ -19,7 +19,7 @@
Google.Protobuf + Grpc.Core.Api flow transitively from Commons (the generated client). --> Google.Protobuf + Grpc.Core.Api flow transitively from Commons (the generated client). -->
<PackageReference Include="Grpc.Net.Client"/> <PackageReference Include="Grpc.Net.Client"/>
<PackageReference Include="ZB.MOM.WW.Audit"/> <PackageReference Include="ZB.MOM.WW.Audit"/>
<!-- ClusterClient→gRPC: MeshPskProvider resolves OT-MESH-PSK-{clusterId} from the shared <!-- Mesh gRPC command plane: MeshPskProvider resolves OT-MESH-PSK-{clusterId} from the shared
secret store when MeshTransport:ClusterPsks carries no override. Abstractions only — secret store when MeshTransport:ClusterPsks carries no override. Abstractions only —
the concrete store is the host's choice (SecretsRegistration), not the control plane's. --> the concrete store is the host's choice (SecretsRegistration), not the control plane's. -->
<PackageReference Include="ZB.MOM.WW.Secrets.Abstractions"/> <PackageReference Include="ZB.MOM.WW.Secrets.Abstractions"/>
@@ -6,7 +6,7 @@ using ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
namespace ZB.MOM.WW.OtOpcUa.Host.Configuration; namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
/// <summary> /// <summary>
/// Gates the central-side command endpoint (ClusterClient→gRPC migration). The /// Gates the central-side command endpoint (mesh gRPC command plane). The
/// <c>CentralCommandService.ReportApply</c> RPC is how a node tells central its deployment /// <c>CentralCommandService.ReportApply</c> RPC is how a node tells central its deployment
/// applied; the coordinator counts those acks, so anything able to reach central's command port /// applied; the coordinator counts those acks, so anything able to reach central's command port
/// could otherwise forge a fleet-wide "everything applied". This interceptor is the ONLY inbound /// could otherwise forge a fleet-wide "everything applied". This interceptor is the ONLY inbound
@@ -9,7 +9,7 @@ using ZB.MOM.WW.OtOpcUa.Cluster;
namespace ZB.MOM.WW.OtOpcUa.Host.Configuration; namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
/// <summary> /// <summary>
/// Gates the node-side command endpoint (ClusterClient→gRPC migration). The /// Gates the node-side command endpoint (mesh gRPC command plane). The
/// <c>NodeCommandService.Execute</c> RPC is how central drives this node — restart a driver, /// <c>NodeCommandService.Execute</c> RPC is how central drives this node — restart a driver,
/// acknowledge an alarm, dispatch a deployment — so anything able to reach a node's command port /// acknowledge an alarm, dispatch a deployment — so anything able to reach a node's command port
/// could otherwise operate the node. This interceptor is the ONLY inbound auth on that endpoint. /// could otherwise operate the node. This interceptor is the ONLY inbound auth on that endpoint.
@@ -15,11 +15,11 @@ using Status = Grpc.Core.Status;
namespace ZB.MOM.WW.OtOpcUa.Host.Grpc; namespace ZB.MOM.WW.OtOpcUa.Host.Grpc;
/// <summary> /// <summary>
/// Node-side command ingress (ClusterClient→gRPC migration Phase 1). Central dials a driver node /// Node-side command ingress (mesh gRPC command plane, Phase 1). Central dials a driver node
/// and calls <see cref="Execute"/>; the envelope is decoded back into the same CLR record the /// and calls <see cref="Execute"/>; the envelope is decoded back into the same CLR record the
/// ClusterClient receptionist used to deliver and handed to this node's /// node-local DistributedPubSub path delivers and handed to this node's
/// <c>NodeCommunicationActor</c>, which re-emits it on the node's <c>EventStream</c> exactly as /// <c>NodeCommunicationActor</c>, which re-emits it on the node's <c>EventStream</c> exactly as
/// before. Nothing downstream of that actor can tell which transport carried the command. /// it would otherwise. Nothing downstream of that actor can tell which transport carried it.
/// </summary> /// </summary>
/// <remarks> /// <remarks>
/// <para> /// <para>
+6 -7
View File
@@ -527,20 +527,19 @@ if (hasAdmin)
// hosted service has built the system). // hosted service has built the system).
builder.Services.TryAddSingleton<Func<ActorSystem>>(sp => () => sp.GetRequiredService<ActorSystem>()); builder.Services.TryAddSingleton<Func<ActorSystem>>(sp => () => sp.GetRequiredService<ActorSystem>());
builder.Services.AddSingleton<IManualFailoverService, ManualFailoverService>(); builder.Services.AddSingleton<IManualFailoverService, ManualFailoverService>();
// Per-cluster mesh PSKs (ClusterClient→gRPC). Admin-only, and singleton because its whole job is // Per-cluster mesh PSKs (mesh gRPC command plane). Admin-only, and singleton because its whole job is
// to hold a cache: central talks to N clusters and would otherwise re-read the secret store on // to hold a cache: central talks to N clusters and would otherwise re-read the secret store on
// every inbound ack. A node needs nothing of this — it holds exactly its own MeshTransport:GrpcPsk // every inbound ack. A node needs nothing of this — it holds exactly its own MeshTransport:GrpcPsk
// — so registering it fleet-wide would give a driver-only node a lookup it can never use. // — so registering it fleet-wide would give a driver-only node a lookup it can never use.
// Depends on ISecretResolver, which AddOtOpcUaSecrets registers unconditionally below (DI // Depends on ISecretResolver, which AddOtOpcUaSecrets registers unconditionally below (DI
// registration order is irrelevant; resolution order is not). // registration order is irrelevant; resolution order is not).
builder.Services.AddSingleton<IMeshPskProvider, MeshPskProvider>(); builder.Services.AddSingleton<IMeshPskProvider, MeshPskProvider>();
// Central's outbound command dial (ClusterClient→gRPC Phase 1A). Admin-only for the same reason // Central's outbound command dial (mesh gRPC command plane, Phase 1A). Admin-only for the same reason
// as the provider above — a node never dials another node — and singleton because it owns a // as the provider above — a node never dials another node — and singleton because it owns a
// GrpcChannel per node endpoint; per-call channels would leak sockets across a fleet-wide fan. // GrpcChannel per node endpoint; per-call channels would leak sockets across a fleet-wide fan.
// Registered in every mode: CentralCommunicationActor only reaches it under Mode=Grpc, and an // Registered in every mode: CentralCommunicationActor only reaches it under Mode=Grpc, and an
// idle registration costs nothing, whereas gating it here would mean flipping the mode on a // idle registration costs nothing, whereas gating it here would mean flipping the mode on a
// running fleet needs a central restart before any command lands — the same reasoning that // running fleet needs a central restart before any command lands.
// registers the ClusterClient receptionist unconditionally.
builder.Services.AddSingleton<IMeshCommandDialer, MeshGrpcCommandDialer>(); builder.Services.AddSingleton<IMeshCommandDialer, MeshGrpcCommandDialer>();
} }
@@ -564,7 +563,7 @@ builder.Services.AddOtOpcUaObservability(builder.Configuration);
// Registered whenever either role is present so an admin-only node still serves config and a // Registered whenever either role is present so an admin-only node still serves config and a
// driver-only node still gates sync + telemetry. // driver-only node still gates sync + telemetry.
// //
// The two mesh-command interceptors (ClusterClient→gRPC migration) join on the same terms: // The two mesh-command interceptors (mesh gRPC command plane) join on the same terms:
// MeshCommandAuthInterceptor gates the node-side NodeCommandService (driver-role, one key — this // MeshCommandAuthInterceptor gates the node-side NodeCommandService (driver-role, one key — this
// node's own), CentralCommandAuthInterceptor gates the central-side CentralCommandService // node's own), CentralCommandAuthInterceptor gates the central-side CentralCommandService
// (admin-role, one key per cluster via IMeshPskProvider). Both are registered NOW and are pure // (admin-role, one key per cluster via IMeshPskProvider). Both are registered NOW and are pure
@@ -745,7 +744,7 @@ if (hasAdmin && configServeGrpcPort > 0)
{ {
app.MapGrpcService<DeploymentArtifactGrpcService>(); app.MapGrpcService<DeploymentArtifactGrpcService>();
// Central command ingress (ClusterClient→gRPC migration): the node→central deployment ack. It // Central command ingress (mesh gRPC command plane): the node→central deployment ack. It
// rides the SAME listener rather than claiming a port of its own — a second cleartext h2c port // rides the SAME listener rather than claiming a port of its own — a second cleartext h2c port
// would buy nothing but another firewall rule, and both endpoints are admin-role and dialled by // would buy nothing but another firewall rule, and both endpoints are admin-role and dialled by
// the same driver nodes. Deliberately NOT gated on MeshTransport:Mode: mapping it under Dps // the same driver nodes. Deliberately NOT gated on MeshTransport:Mode: mapping it under Dps
@@ -759,7 +758,7 @@ if (hasAdmin && configServeGrpcPort > 0)
// Driver-node live-telemetry stream (per-cluster mesh Phase 5). Central dials in; gated by // Driver-node live-telemetry stream (per-cluster mesh Phase 5). Central dials in; gated by
// TelemetryStreamAuthInterceptor (fail-closed on empty Telemetry:ApiKey). // TelemetryStreamAuthInterceptor (fail-closed on empty Telemetry:ApiKey).
// //
// The node-side command endpoint (ClusterClient→gRPC migration Phase 1) rides the SAME driver-role // The node-side command endpoint (mesh gRPC command plane, Phase 1) rides the SAME driver-role
// h2c listener rather than claiming a fourth port: central already dials this node here, both // h2c listener rather than claiming a fourth port: central already dials this node here, both
// directions are per-cluster-keyed, and a separate port would be one more thing to open on every // directions are per-cluster-keyed, and a separate port would be one more thing to open on every
// site firewall for no isolation gain. Mapping it is what makes MeshCommandAuthInterceptor live — // site firewall for no isolation gain. Mapping it is what makes MeshCommandAuthInterceptor live —
@@ -12,14 +12,14 @@ namespace ZB.MOM.WW.OtOpcUa.Runtime.Communication;
/// <summary> /// <summary>
/// Posts this node's <see cref="ApplyAck"/> to central's <c>CentralCommandService</c> as a unary /// Posts this node's <see cref="ApplyAck"/> to central's <c>CentralCommandService</c> as a unary
/// gRPC call (ClusterClient→gRPC migration Task 1B.2), the <c>MeshTransport:Mode = Grpc</c> /// gRPC call (mesh gRPC command plane, Task 1B.2), the <c>MeshTransport:Mode = Grpc</c>
/// replacement for the ClusterClient <c>Send</c>. /// cross-mesh ack transport.
/// </summary> /// </summary>
/// <remarks> /// <remarks>
/// <para> /// <para>
/// <b>One ordered pass, stopping at the first success.</b> /// <b>One ordered pass, stopping at the first success.</b>
/// <see cref="MeshTransportOptions.CentralGrpcEndpoints"/> is an availability list, not a /// <see cref="MeshTransportOptions.CentralGrpcEndpoints"/> is an availability list, not a
/// fan-out list — the semantic of the <c>ClusterClient.Send</c> being replaced (<c>Send</c>, /// fan-out list — the single-target semantic this leg owes (one ack per node per dispatch,
/// deliberately not <c>SendToAll</c>). /// deliberately not <c>SendToAll</c>).
/// </para> /// </para>
/// <para> /// <para>
@@ -35,7 +35,7 @@ namespace ZB.MOM.WW.OtOpcUa.Runtime.Communication;
/// depth and a bound on wasted work, not a correctness dependency. /// depth and a bound on wasted work, not a correctness dependency.
/// </para> /// </para>
/// <para> /// <para>
/// <b>No retry beyond that single pass, and no queue.</b> The receptionist buffered nothing /// <b>No retry beyond that single pass, and no queue.</b> The transport buffers nothing
/// and retried nothing, and this is a like-for-like replacement, not an upgrade to delivery /// and retried nothing, and this is a like-for-like replacement, not an upgrade to delivery
/// guarantees. The deploy path's own apply deadline stays the only recovery mechanism; a /// guarantees. The deploy path's own apply deadline stays the only recovery mechanism; a
/// retry loop here would instead pile acks up behind an unreachable central and deliver a /// retry loop here would instead pile acks up behind an unreachable central and deliver a
@@ -43,7 +43,7 @@ namespace ZB.MOM.WW.OtOpcUa.Runtime.Communication;
/// </para> /// </para>
/// <para> /// <para>
/// <b>All endpoints failing is a DROP — but a loud one.</b> That is the same outcome the /// <b>All endpoints failing is a DROP — but a loud one.</b> That is the same outcome the
/// ClusterClient path had when it could not reach central; what changes is that the operator /// node has when it cannot reach central; what changes is that the operator
/// now gets the node, the deployment, the correlation id and the status each endpoint /// now gets the node, the deployment, the correlation id and the status each endpoint
/// answered with, which is what distinguishes "central is down" from "central rejected this /// answered with, which is what distinguishes "central is down" from "central rejected this
/// node's key". /// node's key".
@@ -4,15 +4,15 @@ namespace ZB.MOM.WW.OtOpcUa.Runtime.Communication;
/// <summary> /// <summary>
/// Posts this node's deployment acknowledgment to central over the <c>mesh_transport.v1</c> /// Posts this node's deployment acknowledgment to central over the <c>mesh_transport.v1</c>
/// command plane (ClusterClient→gRPC migration Phase 1B). The seam /// command plane (mesh gRPC command plane, Phase 1B). The seam
/// <see cref="NodeCommunicationActor"/> uses under <c>MeshTransport:Mode = Grpc</c>, in place of /// <see cref="NodeCommunicationActor"/> uses under <c>MeshTransport:Mode = Grpc</c>, in place of
/// the <c>ClusterClient.Send</c> it uses under <c>ClusterClient</c>. /// the node-local DistributedPubSub publish it uses under <c>Dps</c>.
/// </summary> /// </summary>
/// <remarks> /// <remarks>
/// <para> /// <para>
/// <b>One ack per node per dispatch.</b> Whatever the implementation does about redundancy, it /// <b>One ack per node per dispatch.</b> Whatever the implementation does about redundancy, it
/// posts to <i>one</i> central endpoint at a time rather than fanning out — the semantic of the /// posts to <i>one</i> central endpoint at a time rather than fanning out — the semantic of the
/// <c>ClusterClient.Send</c> this replaces (<c>Send</c>, deliberately not <c>SendToAll</c>). /// one-ack-per-node-per-dispatch semantic the deploy coordinator counts on.
/// This is <b>defence in depth, not a correctness dependency</b>: the coordinator counts acks /// This is <b>defence in depth, not a correctness dependency</b>: the coordinator counts acks
/// in a <c>NodeId</c>-keyed dictionary behind an expected-set membership guard /// in a <c>NodeId</c>-keyed dictionary behind an expected-set membership guard
/// (<c>ConfigPublishCoordinator.HandleAck</c>), so a duplicate from one node overwrites its own /// (<c>ConfigPublishCoordinator.HandleAck</c>), so a duplicate from one node overwrites its own
@@ -1,5 +1,4 @@
using Akka.Actor; using Akka.Actor;
using Akka.Cluster.Tools.Client;
using Akka.Event; using Akka.Event;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Admin; using ZB.MOM.WW.OtOpcUa.Commons.Messages.Admin;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Deploy; using ZB.MOM.WW.OtOpcUa.Commons.Messages.Deploy;
@@ -9,17 +8,20 @@ using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
namespace ZB.MOM.WW.OtOpcUa.Runtime.Communication; namespace ZB.MOM.WW.OtOpcUa.Runtime.Communication;
/// <summary> /// <summary>
/// Node-side end of the central↔node command boundary (per-cluster mesh Phase 2), registered /// Node-side end of the central↔node command boundary (per-cluster mesh Phase 2). This node's
/// with the <c>ClusterClientReceptionist</c> at <see cref="MeshPaths.NodeCommunication"/>. /// <c>NodeCommandService</c> gRPC ingress hands it every decoded command, resolving it from
/// <c>ActorRegistry</c> by <c>NodeCommunicationActorKey</c> — the actor's name
/// (<see cref="MeshPaths.NodeCommunicationName"/>) is a local convention, not an address anything
/// off-process uses.
/// </summary> /// </summary>
/// <remarks> /// <remarks>
/// <para> /// <para>
/// <b>Registered per node, NOT as a cluster singleton</b> — central's ClusterClient rotates /// <b>Registered per node, NOT as a cluster singleton</b> — central dials each node's own
/// across contact points and must find a live comm actor at whichever node answers. /// command endpoint, so every node needs its own live comm actor.
/// </para> /// </para>
/// <para> /// <para>
/// <b>Inbound commands are re-emitted on the node's <see cref="EventStream"/>, not on their /// <b>Inbound commands are re-emitted on the node's <see cref="EventStream"/>, not on their
/// DistributedPubSub topic.</b> DPS is mesh-wide, and central <c>SendToAll</c>s to every /// DistributedPubSub topic.</b> DPS is mesh-wide, and central delivers one copy to every
/// node's comm actor — so a DPS re-publish would deliver N copies of every command to every /// node's comm actor — so a DPS re-publish would deliver N copies of every command to every
/// subscriber, where N is the node count. The EventStream is node-local by construction, /// subscriber, where N is the node count. The EventStream is node-local by construction,
/// which is exactly the needed semantics. It also avoids a registration handshake with /// which is exactly the needed semantics. It also avoids a registration handshake with
@@ -36,31 +38,23 @@ namespace ZB.MOM.WW.OtOpcUa.Runtime.Communication;
/// </remarks> /// </remarks>
public sealed class NodeCommunicationActor : ReceiveActor public sealed class NodeCommunicationActor : ReceiveActor
{ {
private readonly IActorRef? _centralClient;
private readonly IApplyAckSender? _ackSender; private readonly IApplyAckSender? _ackSender;
private readonly ILoggingAdapter _log = Context.GetLogger(); private readonly ILoggingAdapter _log = Context.GetLogger();
/// <summary>Creates the props for this actor.</summary> /// <summary>Creates the props for this actor.</summary>
/// <param name="centralClient">
/// ClusterClient dialling central's receptionist, or <see langword="null"/> under
/// <c>MeshTransport:Mode = Dps</c>, where acks still go out over DistributedPubSub from
/// <c>DriverHostActor</c> and this actor never sees them — and under
/// <c>Mode = Grpc</c>, where no ClusterClient is created at all.
/// </param>
/// <param name="ackSender"> /// <param name="ackSender">
/// The gRPC ack transport under <c>MeshTransport:Mode = Grpc</c>, or <see langword="null"/> /// The gRPC ack transport under <c>MeshTransport:Mode = Grpc</c>, or <see langword="null"/>
/// under <c>Dps</c> and <c>ClusterClient</c>. /// under <c>Dps</c>, where acks still go out over DistributedPubSub from
/// <c>DriverHostActor</c> and this actor never sees them.
/// </param> /// </param>
/// <returns>The props.</returns> /// <returns>The props.</returns>
public static Props Props(IActorRef? centralClient, IApplyAckSender? ackSender = null) => public static Props Props(IApplyAckSender? ackSender = null) =>
Akka.Actor.Props.Create(() => new NodeCommunicationActor(centralClient, ackSender)); Akka.Actor.Props.Create(() => new NodeCommunicationActor(ackSender));
/// <summary>Initializes a new instance of the <see cref="NodeCommunicationActor"/> class.</summary> /// <summary>Initializes a new instance of the <see cref="NodeCommunicationActor"/> class.</summary>
/// <param name="centralClient">ClusterClient dialling central, or <see langword="null"/>.</param>
/// <param name="ackSender">The gRPC ack transport, or <see langword="null"/>.</param> /// <param name="ackSender">The gRPC ack transport, or <see langword="null"/>.</param>
public NodeCommunicationActor(IActorRef? centralClient, IApplyAckSender? ackSender = null) public NodeCommunicationActor(IApplyAckSender? ackSender = null)
{ {
_centralClient = centralClient;
_ackSender = ackSender; _ackSender = ackSender;
// Inbound from central. Each type is listed explicitly rather than caught by a ReceiveAny: // Inbound from central. Each type is listed explicitly rather than caught by a ReceiveAny:
@@ -85,14 +79,10 @@ public sealed class NodeCommunicationActor : ReceiveActor
{ {
if (_ackSender is not null) if (_ackSender is not null)
{ {
// Fire-and-forget by construction, exactly as the ClusterClient Tell below is. The sender // Fire-and-forget by construction. The sender never throws and never retries (see
// never throws and never retries (see IApplyAckSender), so there is nothing to await and // IApplyAckSender), so there is nothing to await and nothing to pipe back: awaiting would
// nothing to pipe back: awaiting would stall this mailbox behind a network call to central, // stall this mailbox behind a network call to central, and an Ask would invent a
// and an Ask would invent a request/response the one-way contract does not have. // request/response the one-way contract does not have.
//
// Checked BEFORE the ClusterClient. The registration arm builds exactly one transport, so
// both are never set — but were they, posting over both would deliver two acks for one
// apply and duplicate central's work for no benefit.
try try
{ {
_ = _ackSender.SendAsync(ack); _ = _ackSender.SendAsync(ack);
@@ -115,22 +105,12 @@ public sealed class NodeCommunicationActor : ReceiveActor
return; return;
} }
if (_centralClient is null) _log.Warning(
{ "No central ack transport — dropping ApplyAck for {DeploymentId}. The ack is not "
_log.Warning( + "buffered and will not be retried, so the deployment will time out at central "
"No central ack transport — dropping ApplyAck for {DeploymentId}. The ack is not " + "naming this node even though it applied successfully. Check "
+ "buffered and will not be retried, so the deployment will time out at central " + "MeshTransport:CentralGrpcEndpoints, and that MeshTransport:Mode is Grpc on a node "
+ "naming this node even though it applied successfully. Check " + "whose central lives in another mesh",
+ "MeshTransport:CentralContactPoints (ClusterClient mode) or " ack.DeploymentId);
+ "MeshTransport:CentralGrpcEndpoints (Grpc mode)",
ack.DeploymentId);
return;
}
// Send, not SendToAll: the deploy coordinator is a single singleton, and the central comm
// actor forwards to its proxy. Its NodeId-keyed upsert absorbs a duplicate idempotently,
// so single-target is defence in depth and a bound on wasted sends — mirroring the
// corrected Grpc-branch rationale above.
_centralClient.Tell(new ClusterClient.Send(MeshPaths.CentralCommunication, ack));
} }
} }
@@ -374,9 +374,8 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
/// <param name="dbFactory">Database context factory for configuration database access.</param> /// <param name="dbFactory">Database context factory for configuration database access.</param>
/// <param name="localNode">The local cluster node identifier.</param> /// <param name="localNode">The local cluster node identifier.</param>
/// <param name="ackRouter">Where ApplyAcks are sent. A direct coordinator handle in tests; under /// <param name="ackRouter">Where ApplyAcks are sent. A direct coordinator handle in tests; under
/// <c>MeshTransport:Mode = ClusterClient</c> <b>and</b> <c>= Grpc</c> the node's /// <c>MeshTransport:Mode = Grpc</c> the node's <c>NodeCommunicationActor</c>, which relays the ack
/// <c>NodeCommunicationActor</c>, which relays across the boundary over whichever transport it was /// across the mesh boundary. Null publishes on the <c>deployment-acks</c> DPS topic.</param>
/// built with. Null publishes on the <c>deployment-acks</c> DPS topic.</param>
/// <param name="driverFactory">Optional driver factory; defaults to null factory if not provided.</param> /// <param name="driverFactory">Optional driver factory; defaults to null factory if not provided.</param>
/// <param name="localRoles">Optional set of roles assigned to the local node.</param> /// <param name="localRoles">Optional set of roles assigned to the local node.</param>
/// <param name="dependencyMux">Optional actor reference for dependency multiplexing.</param> /// <param name="dependencyMux">Optional actor reference for dependency multiplexing.</param>
@@ -459,9 +458,8 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
/// <param name="dbFactory">Database context factory for configuration database access.</param> /// <param name="dbFactory">Database context factory for configuration database access.</param>
/// <param name="localNode">The local cluster node identifier.</param> /// <param name="localNode">The local cluster node identifier.</param>
/// <param name="ackRouter">Where ApplyAcks are sent. A direct coordinator handle in tests; under /// <param name="ackRouter">Where ApplyAcks are sent. A direct coordinator handle in tests; under
/// <c>MeshTransport:Mode = ClusterClient</c> <b>and</b> <c>= Grpc</c> the node's /// <c>MeshTransport:Mode = Grpc</c> the node's <c>NodeCommunicationActor</c>, which relays the ack
/// <c>NodeCommunicationActor</c>, which relays across the boundary over whichever transport it was /// across the mesh boundary. Null publishes on the <c>deployment-acks</c> DPS topic.</param>
/// built with. Null publishes on the <c>deployment-acks</c> DPS topic.</param>
/// <param name="driverFactory">Optional driver factory; defaults to null factory if not provided.</param> /// <param name="driverFactory">Optional driver factory; defaults to null factory if not provided.</param>
/// <param name="localRoles">Optional set of roles assigned to the local node.</param> /// <param name="localRoles">Optional set of roles assigned to the local node.</param>
/// <param name="dependencyMux">Optional actor reference for dependency multiplexing.</param> /// <param name="dependencyMux">Optional actor reference for dependency multiplexing.</param>
@@ -584,11 +582,11 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
_mediator.Tell( _mediator.Tell(
new Subscribe(ZB.MOM.WW.OtOpcUa.Runtime.OpcUa.OpcUaPublishActor.RedundancyStateTopic, Self)); new Subscribe(ZB.MOM.WW.OtOpcUa.Runtime.OpcUa.OpcUaPublishActor.RedundancyStateTopic, Self));
// Per-cluster mesh Phase 2 dark switch. Under MeshTransport:Mode = ClusterClient these same // Per-cluster mesh Phase 2. Under MeshTransport:Mode = Grpc these same commands arrive from
// commands arrive from NodeCommunicationActor on the node-local EventStream instead of the // NodeCommunicationActor on the node-local EventStream instead of the mesh-wide DPS topic.
// mesh-wide DPS topic. Subscribing to BOTH unconditionally is deliberate: exactly one of the // Subscribing to BOTH unconditionally is deliberate: exactly one of the two ever publishes, so
// two ever publishes, so the transport flag lives entirely on the publishing side and the // the transport choice lives entirely on the publishing side and the mode can be flipped (or
// switch can be flipped (or reverted) without touching any subscriber. // reverted) without touching any subscriber.
Context.System.EventStream.Subscribe(Self, typeof(DispatchDeployment)); Context.System.EventStream.Subscribe(Self, typeof(DispatchDeployment));
Context.System.EventStream.Subscribe(Self, typeof(RestartDriver)); Context.System.EventStream.Subscribe(Self, typeof(RestartDriver));
Context.System.EventStream.Subscribe(Self, typeof(ReconnectDriver)); Context.System.EventStream.Subscribe(Self, typeof(ReconnectDriver));
@@ -2492,11 +2490,11 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
var ack = new ApplyAck(deploymentId, _localNode, outcome, failureReason, correlation); var ack = new ApplyAck(deploymentId, _localNode, outcome, failureReason, correlation);
if (_ackRouter is not null) if (_ackRouter is not null)
{ {
// Either a direct coordinator handle (tests) or, under MeshTransport:Mode = ClusterClient // Either a direct coordinator handle (tests) or, under MeshTransport:Mode = Grpc, the
// and = Grpc alike, the NodeCommunicationActor that relays this across the boundary (it // NodeCommunicationActor that relays this across the mesh boundary (it owns the transport,
// owns the choice of transport, not this actor). Renamed from _coordinatorOverride in // not this actor). Renamed from _coordinatorOverride in per-cluster mesh Phase 2: under
// per-cluster mesh Phase 2: under those two modes this ref is emphatically NOT a // Grpc this ref is emphatically NOT a coordinator, and the old name would send the next
// coordinator, and the old name would send the next reader looking for one. // reader looking for one.
_ackRouter.Tell(ack); _ackRouter.Tell(ack);
} }
else else
@@ -627,7 +627,7 @@ public sealed class ScriptedAlarmHostActor : ReceiveActor
// Per-cluster mesh Phase 2 dark switch. An AdminUI-originated ack/shelve arrives from // Per-cluster mesh Phase 2 dark switch. An AdminUI-originated ack/shelve arrives from
// NodeCommunicationActor on the node-local EventStream under // NodeCommunicationActor on the node-local EventStream under
// MeshTransport:Mode = ClusterClient; the DPS subscribe above still carries the node-LOCAL // MeshTransport:Mode = Grpc; the DPS subscribe above still carries the node-LOCAL
// publisher (OtOpcUaServerHostedService, the OPC UA Part 9 method calls), which never // publisher (OtOpcUaServerHostedService, the OPC UA Part 9 method calls), which never
// crosses the boundary and stays on DPS in both modes. Subscribing to both is safe: exactly // crosses the boundary and stays on DPS in both modes. Subscribing to both is safe: exactly
// one publisher feeds each path. // one publisher feeds each path.
@@ -1,6 +1,4 @@
using System.Collections.Immutable;
using Akka.Actor; using Akka.Actor;
using Akka.Cluster.Tools.Client;
using Akka.Event; using Akka.Event;
using Akka.Hosting; using Akka.Hosting;
using Microsoft.EntityFrameworkCore; using Microsoft.EntityFrameworkCore;
@@ -47,7 +45,6 @@ public static class ServiceCollectionExtensions
public const string OpcUaPublishActorName = "opcua-publish"; public const string OpcUaPublishActorName = "opcua-publish";
public const string PeerProbeSupervisorName = "peer-probe-supervisor"; public const string PeerProbeSupervisorName = "peer-probe-supervisor";
public const string ContinuousHistorizationRecorderActorName = "continuous-historization-recorder"; public const string ContinuousHistorizationRecorderActorName = "continuous-historization-recorder";
public const string CentralClusterClientName = "central-cluster-client";
/// <summary> /// <summary>
/// Registers shared runtime services. Currently binds <see cref="IAlarmHistorianSink"/> /// Registers shared runtime services. Currently binds <see cref="IAlarmHistorianSink"/>
@@ -101,8 +98,8 @@ public static class ServiceCollectionExtensions
// with no gate at all, and both nodes would write every sample twice with nothing to say so. // with no gate at all, and both nodes would write every sample twice with nothing to say so.
services.TryAddSingleton<IHistorizationRoleView>(sp => sp.GetRequiredService<RedundancyRoleView>()); services.TryAddSingleton<IHistorizationRoleView>(sp => sp.GetRequiredService<RedundancyRoleView>());
// ClusterClient→gRPC migration Task 1B.2: the node→central ack transport under // The node→central ack transport under MeshTransport:Mode = Grpc (mesh gRPC command plane,
// MeshTransport:Mode = Grpc. Registered unconditionally but resolved only by the Grpc arm of // Task 1B.2). Registered unconditionally but resolved only by the Grpc arm of
// WithOtOpcUaRuntimeActors — the mode is not knowable here (this runs before configuration is // WithOtOpcUaRuntimeActors — the mode is not knowable here (this runs before configuration is
// read into options), and a singleton nobody resolves is never constructed, so a Dps node // read into options), and a singleton nobody resolves is never constructed, so a Dps node
// opens no channel. // opens no channel.
@@ -404,35 +401,15 @@ public static class ServiceCollectionExtensions
registry.Register<DependencyMuxActorKey>(mux); registry.Register<DependencyMuxActorKey>(mux);
// Per-cluster mesh Phase 2: the node end of the central↔node command boundary. Spawned // Per-cluster mesh Phase 2: the node end of the central↔node command boundary. Spawned
// BEFORE DriverHostActor because it is the host's ackRouter under ClusterClient mode. // BEFORE DriverHostActor because it is the host's ackRouter under Grpc mode. It has no
// It has no dependency of its own on the host — inbound commands travel via the node's // dependency of its own on the host — inbound commands travel via the node's
// EventStream — so this ordering costs nothing. // EventStream — so this ordering costs nothing.
var meshOptions = resolver.GetService<IOptions<MeshTransportOptions>>().Value; var meshOptions = resolver.GetService<IOptions<MeshTransportOptions>>().Value;
var useClusterClient = string.Equals(
meshOptions.Mode, MeshTransportOptions.ModeClusterClient, StringComparison.OrdinalIgnoreCase);
var useGrpc = string.Equals( var useGrpc = string.Equals(
meshOptions.Mode, MeshTransportOptions.ModeGrpc, StringComparison.OrdinalIgnoreCase); meshOptions.Mode, MeshTransportOptions.ModeGrpc, StringComparison.OrdinalIgnoreCase);
IActorRef? centralClient = null; // The node→central ack transport. Built ONLY under Grpc: under Dps the ack goes out on
if (useClusterClient) // the deployment-acks topic from DriverHostActor and never reaches this actor at all.
{
// Contacts come from appsettings, not the database — the deliberate asymmetry:
// central's node set changes as operators add and retire nodes, but central's own
// address is part of the deployment. MeshTransportOptionsValidator has already
// rejected an empty or malformed list at boot, so this cannot silently produce a
// client with no contacts.
var contacts = meshOptions.CentralContactPoints
.Select(cp => ActorPath.Parse($"{cp}/system/receptionist"))
.ToImmutableHashSet();
centralClient = system.ActorOf(
ClusterClient.Props(ClusterClientSettings.Create(system).WithInitialContacts(contacts)),
CentralClusterClientName);
}
// ClusterClient→gRPC migration Task 1B.2. Deliberately EXCLUSIVE with the branch above —
// under Grpc no ClusterClient is created at all, which is half the point of the migration
// (it is the cross-WAN Akka association this program removes). The two are never both
// handed to the actor, and it prefers the sender if they ever were.
IApplyAckSender? ackSender = null; IApplyAckSender? ackSender = null;
if (useGrpc) if (useGrpc)
{ {
@@ -451,13 +428,12 @@ public static class ServiceCollectionExtensions
} }
var nodeComm = system.ActorOf( var nodeComm = system.ActorOf(
NodeCommunicationActor.Props(centralClient, ackSender), NodeCommunicationActor.Props(ackSender),
MeshPaths.NodeCommunicationName); MeshPaths.NodeCommunicationName);
// Registered with the receptionist in ALL modes: under Dps and Grpc nothing dials it, and // The registry entry is the whole reachability story: NodeCommandGrpcService resolves
// an idle registration costs nothing — whereas registering only under ClusterClient would // this key to hand an inbound command over. Nothing registers with a receptionist any
// make flipping the flag require a node restart before central could reach it. // more — there is no Akka association from central into this mesh to register into.
ClusterClientReceptionist.Get(system).RegisterService(nodeComm);
registry.Register<NodeCommunicationActorKey>(nodeComm); registry.Register<NodeCommunicationActorKey>(nodeComm);
// Continuous-historization recorder — gated on ContinuousHistorization:Enabled AND the // Continuous-historization recorder — gated on ContinuousHistorization:Enabled AND the
@@ -567,12 +543,11 @@ public static class ServiceCollectionExtensions
: null; : null;
var driverHost = system.ActorOf( var driverHost = system.ActorOf(
// ackRouter: under ClusterClient AND Grpc the node comm actor relays ApplyAcks across // ackRouter: under Grpc the node comm actor relays ApplyAcks across the mesh
// the boundary (which transport it then uses is its own business — the ClusterClient // boundary through the gRPC sender it was built with; under Dps it stays null and
// ref or the gRPC sender it was built with); under Dps it stays null and the host // the host publishes on the deployment-acks topic exactly as before. Routed to DPS
// publishes on the deployment-acks topic exactly as before. Grpc MUST be included // instead, a Grpc node's ack would never reach the sender at all and every
// here: routed to DPS instead, a Grpc node's ack would never reach the sender at all // deployment to it would time out while it applied perfectly.
// and every deployment to it would time out while it applied perfectly.
// dbFactory is nullable (Phase 4): a driver-only node passes null here — ConfigDb is // dbFactory is nullable (Phase 4): a driver-only node passes null here — ConfigDb is
// gated on hasAdmin and driver-only nodes are forced to FetchAndCache. DriverHostActor // gated on hasAdmin and driver-only nodes are forced to FetchAndCache. DriverHostActor
// guards every ConfigDb-touching path on it (UpsertNodeDeploymentState no-ops; central // guards every ConfigDb-touching path on it (UpsertNodeDeploymentState no-ops; central
@@ -580,7 +555,7 @@ public static class ServiceCollectionExtensions
// alarmStateStore (there is no ConfigDb-backed EF fallback; the host skips the alarm // alarmStateStore (there is no ConfigDb-backed EF fallback; the host skips the alarm
// host when no store was wired). // host when no store was wired).
DriverHostActor.Props(dbFactory, roleInfo.LocalNode, DriverHostActor.Props(dbFactory, roleInfo.LocalNode,
ackRouter: useClusterClient || useGrpc ? nodeComm : null, ackRouter: useGrpc ? nodeComm : null,
driverFactory: driverFactory, localRoles: roleInfo.LocalRoles, driverFactory: driverFactory, localRoles: roleInfo.LocalRoles,
dependencyMux: mux, dependencyMux: mux,
opcUaPublishActor: publishActor, opcUaPublishActor: publishActor,
@@ -1,5 +1,4 @@
using Akka.Actor; using Akka.Actor;
using Akka.Cluster.Tools.Client;
using Akka.Configuration; using Akka.Configuration;
using Shouldly; using Shouldly;
using Xunit; using Xunit;
@@ -7,8 +6,8 @@ using Xunit;
namespace ZB.MOM.WW.OtOpcUa.Cluster.Tests; namespace ZB.MOM.WW.OtOpcUa.Cluster.Tests;
/// <summary> /// <summary>
/// Asserts the ClusterClient / receptionist / frame-size settings that are actually <b>in /// Asserts the remoting frame-size canary and the DistributedPubSub extension that are actually
/// force</b> on a real <see cref="ActorSystem"/> built from the shipped base config. /// <b>in force</b> on a real <see cref="ActorSystem"/> built from the shipped base config.
/// </summary> /// </summary>
/// <remarks> /// <remarks>
/// Asserting the <c>akka.conf</c> text instead would pass while a competing HOCON fragment /// Asserting the <c>akka.conf</c> text instead would pass while a competing HOCON fragment
@@ -29,38 +28,6 @@ public class MeshTransportHoconTests : IDisposable
_sys = ActorSystem.Create("mesh-hocon-probe", config); _sys = ActorSystem.Create("mesh-hocon-probe", config);
} }
[Fact]
public void Cluster_client_buffering_is_disabled()
{
// Asserted through ClusterClientSettings — the object the client is actually built from —
// NOT through Config.GetInt. GetInt returns 0 for a MISSING key, so a config assertion here
// passes identically whether the setting is explicitly 0 or absent with Akka's default of
// 1000 in force. It was green before this feature existed; that is the whole defect class.
//
// buffer-size = 0 means "drop now if the receptionist is unknown". Buffering would replay a
// DispatchDeployment on reconnect and apply a deployment the coordinator already sealed as
// TimedOut, leaving the node running a configuration the database says failed. This is a
// behaviour decision, not a tuning knob.
ClusterClientSettings.Create(_sys).BufferSize.ShouldBe(0);
}
[Fact]
public void Cluster_client_retries_forever_rather_than_self_stopping()
{
// reconnect-timeout = off surfaces as a null TimeSpan on the settings object. A central node
// down for an hour must not require a driver-node restart to become reachable again.
ClusterClientSettings.Create(_sys).ReconnectTimeout.ShouldBeNull();
}
[Fact]
public void Receptionist_serves_every_member_regardless_of_role()
{
// An empty role is what makes "registered per node, not as a singleton" work: every member
// hosts a receptionist, so ClusterClient contact rotation reaches whichever node answers.
// Scoping this to a role would make the boundary reachable through one node only.
ClusterReceptionistSettings.Create(_sys).Role.ShouldBeNull();
}
[Fact] [Fact]
public void Oversized_frames_are_logged_rather_than_dropped_silently() public void Oversized_frames_are_logged_rather_than_dropped_silently()
{ {
@@ -73,21 +40,11 @@ public class MeshTransportHoconTests : IDisposable
} }
[Fact] [Fact]
public void Receptionist_extension_is_registered_at_startup() public void Distributed_pub_sub_extension_is_registered_at_startup()
{ {
// Not auto-started. Without this entry the receptionist materialises only if some code path // DPS is PAIR-LOCAL and survives the cross-mesh transport's deletion: it carries
// happens to call ClusterClientReceptionist.Get(system) first, which makes "is the boundary // MeshTransport:Mode = Dps (the single-mesh default), the node-local alarm-command publishes
// listening?" depend on startup ordering rather than on configuration. // and the node<->node redundancy-state probe leg. Losing it would break far more than deploys.
_sys.Settings.Config.GetStringList("akka.extensions")
.ShouldContain(s => s.Contains("ClusterClientReceptionistExtensionProvider"));
}
[Fact]
public void Distributed_pub_sub_extension_survives_alongside_the_receptionist()
{
// The dark switch keeps both transports live for one phase, and DPS additionally carries
// traffic Phase 2 does not touch at all — node-local alarm-command publishes and the
// node<->node redundancy-state probe leg. Losing it here would break far more than deploys.
_sys.Settings.Config.GetStringList("akka.extensions") _sys.Settings.Config.GetStringList("akka.extensions")
.ShouldContain(s => s.Contains("DistributedPubSubExtensionProvider")); .ShouldContain(s => s.Contains("DistributedPubSubExtensionProvider"));
} }
@@ -64,33 +64,40 @@ public class MeshTransportOptionsValidatorTests
[Fact] [Fact]
public void Unknown_mode_fails() public void Unknown_mode_fails()
{ {
var result = Validate(new MeshTransportOptions { Mode = "receptionist" }); var result = Validate(new MeshTransportOptions { Mode = "not-a-transport" });
result.Failed.ShouldBeTrue(); result.Failed.ShouldBeTrue();
result.FailureMessage.ShouldContain("receptionist"); result.FailureMessage.ShouldContain("not-a-transport");
} }
[Fact] [Fact]
public void Unknown_mode_message_names_all_three_accepted_values() public void Unknown_mode_message_names_both_accepted_values()
{ {
// An operator who mistypes the mode gets the full menu, not a partial one — Grpc became a // An operator who mistypes the mode gets the full menu, not a partial one.
// legal value in Phase 0 and an error that omitted it would send them back to ClusterClient. var result = Validate(new MeshTransportOptions { Mode = "not-a-mode" });
var result = Validate(new MeshTransportOptions { Mode = "receptionist" });
result.Failed.ShouldBeTrue(); result.Failed.ShouldBeTrue();
result.FailureMessage.ShouldContain(MeshTransportOptions.ModeDps); result.FailureMessage.ShouldContain(MeshTransportOptions.ModeDps);
result.FailureMessage.ShouldContain(MeshTransportOptions.ModeClusterClient); result.FailureMessage.ShouldContain(MeshTransportOptions.ModeGrpc);
}
[Fact]
public void The_retired_ClusterClient_mode_fails_rather_than_falling_back_to_Dps()
{
// A node left carrying the deleted transport's mode value must be refused at boot. Silently
// treating it as Dps would leave a split-topology node running, listening, and delivering
// nothing across its mesh boundary — the exact absence this validator exists to prevent.
var result = Validate(new MeshTransportOptions { Mode = "ClusterClient" });
result.Failed.ShouldBeTrue();
result.FailureMessage.ShouldContain("MeshTransport:Mode");
result.FailureMessage.ShouldContain(MeshTransportOptions.ModeGrpc); result.FailureMessage.ShouldContain(MeshTransportOptions.ModeGrpc);
} }
[Fact] [Fact]
public void Mode_matching_is_case_insensitive() public void Mode_matching_is_case_insensitive()
{ {
Validate(new MeshTransportOptions Validate(new MeshTransportOptions { Mode = "dps" }).Succeeded.ShouldBeTrue();
{
Mode = "clusterclient",
CentralContactPoints = ["akka.tcp://otopcua@central-1:4053"],
}).Succeeded.ShouldBeTrue();
} }
[Fact] [Fact]
@@ -102,92 +109,7 @@ public class MeshTransportOptionsValidatorTests
result.FailureMessage.ShouldContain(nameof(MeshTransportOptions.ContactRefreshSeconds)); result.FailureMessage.ShouldContain(nameof(MeshTransportOptions.ContactRefreshSeconds));
} }
[Fact] // ---- Grpc mode (the cross-mesh command plane) -----------------------------------------------
public void ClusterClient_mode_requires_central_contact_points()
{
var result = Validate(new MeshTransportOptions
{
Mode = MeshTransportOptions.ModeClusterClient,
CentralContactPoints = [],
});
result.Failed.ShouldBeTrue();
result.FailureMessage.ShouldContain(nameof(MeshTransportOptions.CentralContactPoints));
}
[Fact]
public void Dps_mode_does_not_require_central_contact_points()
{
// An admin-only node has no reason to carry them, and requiring them under the default mode
// would make the section mandatory everywhere for a feature that is switched off.
Validate(new MeshTransportOptions
{
Mode = MeshTransportOptions.ModeDps,
CentralContactPoints = [],
}).Succeeded.ShouldBeTrue();
}
[Fact]
public void A_contact_point_that_is_not_an_akka_address_fails()
{
var result = Validate(new MeshTransportOptions
{
Mode = MeshTransportOptions.ModeClusterClient,
CentralContactPoints = ["central-1:4053"],
});
result.Failed.ShouldBeTrue();
result.FailureMessage.ShouldContain("akka.tcp://");
}
[Fact]
public void A_contact_point_carrying_an_actor_path_suffix_fails()
{
// The sister project's shipped site template listed a full receptionist path, and separately
// listed the site's OWN remoting port as the second contact — "a permanent failure in the
// initial-contact rotation". Both are invisible until a command goes missing.
var result = Validate(new MeshTransportOptions
{
Mode = MeshTransportOptions.ModeClusterClient,
CentralContactPoints = ["akka.tcp://otopcua@central-1:4053/system/receptionist"],
});
result.Failed.ShouldBeTrue();
result.FailureMessage.ShouldContain("/system/receptionist");
}
[Fact]
public void One_bad_contact_point_among_good_ones_still_fails()
{
var result = Validate(new MeshTransportOptions
{
Mode = MeshTransportOptions.ModeClusterClient,
CentralContactPoints =
[
"akka.tcp://otopcua@central-1:4053",
"central-2:4053",
],
});
result.Failed.ShouldBeTrue();
result.FailureMessage.ShouldContain("central-2:4053");
}
[Fact]
public void Well_formed_ClusterClient_options_succeed()
{
Validate(new MeshTransportOptions
{
Mode = MeshTransportOptions.ModeClusterClient,
CentralContactPoints =
[
"akka.tcp://otopcua@central-1:4053",
"akka.tcp://otopcua@central-2:4053",
],
}).Succeeded.ShouldBeTrue();
}
// ---- Grpc mode (ClusterClient→gRPC migration, Phase 0) --------------------------------------
[Fact] [Fact]
public void Grpc_is_an_accepted_mode() public void Grpc_is_an_accepted_mode()
@@ -382,20 +304,6 @@ public class MeshTransportOptionsValidatorTests
"admin", "cluster-MAIN").Succeeded.ShouldBeTrue(); "admin", "cluster-MAIN").Succeeded.ShouldBeTrue();
} }
[Fact]
public void ClusterClient_admin_node_needs_no_config_serve_listener()
{
// Contact points are required by the mode itself, independently of role — supplied so this
// test fails only if the ConfigServe rule wrongly fires outside Grpc mode.
Validate(
new MeshTransportOptions
{
Mode = MeshTransportOptions.ModeClusterClient,
CentralContactPoints = ["akka.tcp://otopcua@central-1:4053"],
},
"admin", "cluster-MAIN").Succeeded.ShouldBeTrue();
}
[Fact] [Fact]
public void The_admin_role_is_also_read_from_OTOPCUA_ROLES() public void The_admin_role_is_also_read_from_OTOPCUA_ROLES()
{ {
@@ -442,18 +350,6 @@ public class MeshTransportOptionsValidatorTests
"driver", "cluster-SITE-A").Succeeded.ShouldBeTrue(); "driver", "cluster-SITE-A").Succeeded.ShouldBeTrue();
} }
[Fact]
public void ClusterClient_mode_requires_none_of_the_new_Grpc_keys()
{
Validate(
new MeshTransportOptions
{
Mode = MeshTransportOptions.ModeClusterClient,
CentralContactPoints = ["akka.tcp://otopcua@central-1:4053"],
},
"driver", "cluster-SITE-A").Succeeded.ShouldBeTrue();
}
[Fact] [Fact]
public void Grpc_fused_admin_and_driver_node_must_satisfy_BOTH_surfaces() public void Grpc_fused_admin_and_driver_node_must_satisfy_BOTH_surfaces()
{ {
@@ -51,7 +51,6 @@ public class SplitTopologyTransportValidatorTests
result.Failed.ShouldBeTrue(); result.Failed.ShouldBeTrue();
result.FailureMessage.ShouldContain("MeshTransport:Mode"); result.FailureMessage.ShouldContain("MeshTransport:Mode");
result.FailureMessage.ShouldContain(MeshTransportOptions.ModeClusterClient);
result.FailureMessage.ShouldContain(MeshTransportOptions.ModeGrpc); result.FailureMessage.ShouldContain(MeshTransportOptions.ModeGrpc);
result.FailureMessage.ShouldContain("SITE-A"); result.FailureMessage.ShouldContain("SITE-A");
} }
@@ -59,9 +58,8 @@ public class SplitTopologyTransportValidatorTests
[Fact] [Fact]
public void Cluster_role_driver_node_on_Grpc_mesh_transport_passes() public void Cluster_role_driver_node_on_Grpc_mesh_transport_passes()
{ {
// The migration window: Grpc is the transport this rule exists to arrive at, so a rig node // Grpc is now the ONLY mode that satisfies this rule: it is the only transport that crosses
// flipped to it must boot. Without this relaxation the very first Grpc rollout step — setting // a mesh boundary.
// the mode on a split-topology node — is a host-start failure.
Validate( Validate(
meshMode: MeshTransportOptions.ModeGrpc, meshMode: MeshTransportOptions.ModeGrpc,
telemetryMode: TelemetryOptions.ModeGrpc, telemetryMode: TelemetryOptions.ModeGrpc,
@@ -92,10 +90,11 @@ public class SplitTopologyTransportValidatorTests
[Fact] [Fact]
public void Cluster_role_node_on_an_unrecognised_mesh_transport_still_fails() public void Cluster_role_node_on_an_unrecognised_mesh_transport_still_fails()
{ {
// Widening the rule from one accepted value to two must not widen it to "anything that is not // The rule is an explicit allow-list, not "anything that is not Dps" — a typo'd mode is
// Dps" — a typo'd mode is still a mode that delivers nothing across the boundary. // still a mode that delivers nothing across the boundary. The retired 'ClusterClient' value
// lands here too.
var result = Validate( var result = Validate(
meshMode: "receptionist", meshMode: "not-a-transport",
telemetryMode: TelemetryOptions.ModeGrpc, telemetryMode: TelemetryOptions.ModeGrpc,
telemetryDialMode: null, telemetryDialMode: null,
"driver", "cluster-SITE-A"); "driver", "cluster-SITE-A");
@@ -108,7 +107,7 @@ public class SplitTopologyTransportValidatorTests
public void Cluster_role_driver_node_fully_configured_passes() public void Cluster_role_driver_node_fully_configured_passes()
{ {
Validate( Validate(
meshMode: MeshTransportOptions.ModeClusterClient, meshMode: MeshTransportOptions.ModeGrpc,
telemetryMode: TelemetryOptions.ModeGrpc, telemetryMode: TelemetryOptions.ModeGrpc,
telemetryDialMode: null, telemetryDialMode: null,
"driver", "cluster-SITE-A").Succeeded.ShouldBeTrue(); "driver", "cluster-SITE-A").Succeeded.ShouldBeTrue();
@@ -118,7 +117,7 @@ public class SplitTopologyTransportValidatorTests
public void Cluster_role_admin_node_fully_configured_passes() public void Cluster_role_admin_node_fully_configured_passes()
{ {
Validate( Validate(
meshMode: MeshTransportOptions.ModeClusterClient, meshMode: MeshTransportOptions.ModeGrpc,
telemetryMode: null, telemetryMode: null,
telemetryDialMode: TelemetryDialOptions.ModeGrpc, telemetryDialMode: TelemetryDialOptions.ModeGrpc,
"admin", "cluster-SITE-A").Succeeded.ShouldBeTrue(); "admin", "cluster-SITE-A").Succeeded.ShouldBeTrue();
@@ -128,7 +127,7 @@ public class SplitTopologyTransportValidatorTests
public void Cluster_role_driver_node_on_Dps_telemetry_fails() public void Cluster_role_driver_node_on_Dps_telemetry_fails()
{ {
var result = Validate( var result = Validate(
meshMode: MeshTransportOptions.ModeClusterClient, meshMode: MeshTransportOptions.ModeGrpc,
telemetryMode: TelemetryOptions.ModeDps, telemetryMode: TelemetryOptions.ModeDps,
telemetryDialMode: null, telemetryDialMode: null,
"driver", "cluster-SITE-A"); "driver", "cluster-SITE-A");
@@ -142,7 +141,7 @@ public class SplitTopologyTransportValidatorTests
public void Cluster_role_admin_node_on_Dps_telemetry_dial_fails() public void Cluster_role_admin_node_on_Dps_telemetry_dial_fails()
{ {
var result = Validate( var result = Validate(
meshMode: MeshTransportOptions.ModeClusterClient, meshMode: MeshTransportOptions.ModeGrpc,
telemetryMode: null, telemetryMode: null,
telemetryDialMode: TelemetryDialOptions.ModeDps, telemetryDialMode: TelemetryDialOptions.ModeDps,
"admin", "cluster-SITE-A"); "admin", "cluster-SITE-A");
@@ -156,7 +155,7 @@ public class SplitTopologyTransportValidatorTests
public void Fused_cluster_role_node_needs_all_three_to_pass() public void Fused_cluster_role_node_needs_all_three_to_pass()
{ {
Validate( Validate(
meshMode: MeshTransportOptions.ModeClusterClient, meshMode: MeshTransportOptions.ModeGrpc,
telemetryMode: TelemetryOptions.ModeGrpc, telemetryMode: TelemetryOptions.ModeGrpc,
telemetryDialMode: TelemetryDialOptions.ModeGrpc, telemetryDialMode: TelemetryDialOptions.ModeGrpc,
"admin", "driver", "cluster-MAIN").Succeeded.ShouldBeTrue(); "admin", "driver", "cluster-MAIN").Succeeded.ShouldBeTrue();
@@ -166,7 +165,7 @@ public class SplitTopologyTransportValidatorTests
public void Fused_cluster_role_node_missing_telemetry_grpc_fails() public void Fused_cluster_role_node_missing_telemetry_grpc_fails()
{ {
var result = Validate( var result = Validate(
meshMode: MeshTransportOptions.ModeClusterClient, meshMode: MeshTransportOptions.ModeGrpc,
telemetryMode: TelemetryOptions.ModeDps, telemetryMode: TelemetryOptions.ModeDps,
telemetryDialMode: TelemetryDialOptions.ModeGrpc, telemetryDialMode: TelemetryDialOptions.ModeGrpc,
"admin", "driver", "cluster-MAIN"); "admin", "driver", "cluster-MAIN");
@@ -179,7 +178,7 @@ public class SplitTopologyTransportValidatorTests
public void Fused_cluster_role_node_missing_telemetry_dial_grpc_fails() public void Fused_cluster_role_node_missing_telemetry_dial_grpc_fails()
{ {
var result = Validate( var result = Validate(
meshMode: MeshTransportOptions.ModeClusterClient, meshMode: MeshTransportOptions.ModeGrpc,
telemetryMode: TelemetryOptions.ModeGrpc, telemetryMode: TelemetryOptions.ModeGrpc,
telemetryDialMode: TelemetryDialOptions.ModeDps, telemetryDialMode: TelemetryDialOptions.ModeDps,
"admin", "driver", "cluster-MAIN"); "admin", "driver", "cluster-MAIN");
@@ -229,7 +228,7 @@ public class SplitTopologyTransportValidatorTests
public void Mode_matching_is_case_insensitive() public void Mode_matching_is_case_insensitive()
{ {
Validate( Validate(
meshMode: "clusterclient", meshMode: "grpc",
telemetryMode: "grpc", telemetryMode: "grpc",
telemetryDialMode: "grpc", telemetryDialMode: "grpc",
"admin", "driver", "cluster-MAIN").Succeeded.ShouldBeTrue(); "admin", "driver", "cluster-MAIN").Succeeded.ShouldBeTrue();
@@ -287,13 +286,13 @@ public class SplitTopologyTransportValidatorTests
} }
[Fact] [Fact]
public void Cluster_role_node_with_neither_admin_nor_driver_only_needs_ClusterClient() public void Cluster_role_node_with_neither_admin_nor_driver_only_needs_the_mesh_mode()
{ {
// A cluster-role node that is neither admin nor driver has no telemetry surface to constrain — // A cluster-role node that is neither admin nor driver has no telemetry surface to constrain —
// only the always-on MeshTransport=ClusterClient rule applies. Telemetry/TelemetryDial left on // only the always-on MeshTransport=Grpc rule applies. Telemetry/TelemetryDial left on
// their Dps defaults must not fail it. // their Dps defaults must not fail it.
Validate( Validate(
meshMode: MeshTransportOptions.ModeClusterClient, meshMode: MeshTransportOptions.ModeGrpc,
telemetryMode: null, telemetryMode: null,
telemetryDialMode: null, telemetryDialMode: null,
"cluster-SITE-A").Succeeded.ShouldBeTrue(); "cluster-SITE-A").Succeeded.ShouldBeTrue();
@@ -12,7 +12,7 @@ using ZB.MOM.WW.OtOpcUa.Commons.Types;
namespace ZB.MOM.WW.OtOpcUa.Commons.Tests; namespace ZB.MOM.WW.OtOpcUa.Commons.Tests;
/// <summary> /// <summary>
/// Round-trip goldens for <see cref="MeshTransportDtoMapper"/> — the ClusterClient→gRPC command /// Round-trip goldens for <see cref="MeshTransportDtoMapper"/> — the mesh gRPC command
/// plane's only data contract. Every golden goes domain → proto → <b>serialized bytes</b> → proto → /// plane's only data contract. Every golden goes domain → proto → <b>serialized bytes</b> → proto →
/// domain, because the failure class this suite exists to catch is not a mis-assigned property but a /// domain, because the failure class this suite exists to catch is not a mis-assigned property but a
/// member that survives an in-memory copy and vanishes on the wire: proto3 explicit presence is a /// member that survives an in-memory copy and vanishes on the wire: proto3 explicit presence is a
@@ -6,7 +6,7 @@ using ZB.MOM.WW.OtOpcUa.Commons.Protos.MeshTransport.V1;
namespace ZB.MOM.WW.OtOpcUa.Commons.Tests; namespace ZB.MOM.WW.OtOpcUa.Commons.Tests;
/// <summary> /// <summary>
/// Contract-lock for the ClusterClient→gRPC command envelope's oneof. If a fifth command is added /// Contract-lock for the mesh command envelope's oneof. If a fifth command is added
/// to <c>mesh_transport.proto</c> without adding a matching entry to /// to <c>mesh_transport.proto</c> without adding a matching entry to
/// <see cref="MeshTransportProtoContract.HandledCases"/>, this test goes red — the single guard /// <see cref="MeshTransportProtoContract.HandledCases"/>, this test goes red — the single guard
/// that keeps the wire contract and the handled-case set from silently drifting apart. A drift /// that keeps the wire contract and the handled-case set from silently drifting apart. A drift
@@ -9,7 +9,7 @@ namespace ZB.MOM.WW.OtOpcUa.Configuration.Tests;
/// <summary> /// <summary>
/// Round-trips <see cref="ClusterNode.AkkaPort"/> / <see cref="ClusterNode.GrpcPort"/> through /// Round-trips <see cref="ClusterNode.AkkaPort"/> / <see cref="ClusterNode.GrpcPort"/> through
/// the real schema (per-cluster mesh Phase 1). These are <b>central's dial targets</b>: Phase 2 /// the real schema (per-cluster mesh Phase 1). These are <b>central's dial targets</b>: Phase 2
/// builds its ClusterClient contact points from <c>Host</c> + <c>AkkaPort</c>, and Phase 5 dials /// built its Akka contact points from <c>Host</c> + <c>AkkaPort</c>, and Phase 5 dials
/// <c>GrpcPort</c> for the telemetry stream. Central cannot see a site node's appsettings once /// <c>GrpcPort</c> for the telemetry stream. Central cannot see a site node's appsettings once
/// the meshes split, so the value has to live in a row central can read. /// the meshes split, so the value has to live in a row central can read.
/// </summary> /// </summary>
@@ -14,9 +14,9 @@ using ZB.MOM.WW.OtOpcUa.ControlPlane;
namespace ZB.MOM.WW.OtOpcUa.AdminUI.Tests.Grpc; namespace ZB.MOM.WW.OtOpcUa.AdminUI.Tests.Grpc;
/// <summary> /// <summary>
/// The central-side gRPC ingress for a node's deployment ack (ClusterClient→gRPC migration /// The central-side gRPC ingress for a node's deployment ack (mesh gRPC command plane
/// Phase 1B). It decodes the wire message and hands it to the registered /// Phase 1B). It decodes the wire message and hands it to the registered
/// <c>CentralCommunicationActor</c> — the same actor the ClusterClient receptionist delivered to, /// <c>CentralCommunicationActor</c> — the same actor the DistributedPubSub leg delivers to,
/// whose forward-to-the-coordinator path is untouched. /// whose forward-to-the-coordinator path is untouched.
/// </summary> /// </summary>
/// <remarks> /// <remarks>
@@ -1,9 +1,6 @@
using System.Collections.Immutable;
using System.Diagnostics.CodeAnalysis; using System.Diagnostics.CodeAnalysis;
using Akka.Actor; using Akka.Actor;
using Akka.Cluster.Tools.Client;
using Akka.Cluster.Tools.PublishSubscribe; using Akka.Cluster.Tools.PublishSubscribe;
using Akka.TestKit;
using Microsoft.EntityFrameworkCore; using Microsoft.EntityFrameworkCore;
using Shouldly; using Shouldly;
using Xunit; using Xunit;
@@ -25,40 +22,6 @@ public class CentralCommunicationActorTests : ControlPlaneActorTestBase
{ {
private const string DeploymentsTopic = "deployments"; private const string DeploymentsTopic = "deployments";
/// <summary>Records every client the actor asks for, and hands back a probe in its place.</summary>
private sealed class RecordingClientFactory : IMeshClusterClientFactory
{
private readonly Func<TestProbe> _newProbe;
public RecordingClientFactory(Func<TestProbe> newProbe) => _newProbe = newProbe;
public List<ImmutableHashSet<ActorPath>> Calls { get; } = [];
public List<string> ClusterIds { get; } = [];
public List<TestProbe> Probes { get; } = [];
/// <summary>Probe created for a given cluster id, for per-cluster fan-out assertions.</summary>
public Dictionary<string, TestProbe> ProbeByCluster { get; } = [];
public IActorRef Create(ActorSystem system, string clusterId, ImmutableHashSet<ActorPath> contacts)
{
Calls.Add(contacts);
ClusterIds.Add(clusterId);
var probe = _newProbe();
Probes.Add(probe);
ProbeByCluster[clusterId] = probe;
return probe.Ref;
}
}
private static MeshTransportOptions ClusterClientMode(int refreshSeconds = 60) => new()
{
Mode = MeshTransportOptions.ModeClusterClient,
CentralContactPoints = ["akka.tcp://otopcua@central-1:4053"],
ContactRefreshSeconds = refreshSeconds,
};
private static MeshTransportOptions GrpcMode(int refreshSeconds = 60) => new() private static MeshTransportOptions GrpcMode(int refreshSeconds = 60) => new()
{ {
Mode = MeshTransportOptions.ModeGrpc, Mode = MeshTransportOptions.ModeGrpc,
@@ -66,47 +29,7 @@ public class CentralCommunicationActorTests : ControlPlaneActorTestBase
DispatchTimeoutSeconds = 10, DispatchTimeoutSeconds = 10,
}; };
private static void SeedCluster( /// <summary>Seeds a cluster whose nodes carry a <c>GrpcPort</c> — the command plane's shape.</summary>
IDbContextFactory<OtOpcUaConfigDbContext> factory,
string clusterId,
params (string NodeId, string Host, int AkkaPort, bool Enabled, bool Maintenance)[] nodes)
{
using var db = factory.CreateDbContext();
db.ServerClusters.Add(new ServerCluster
{
ClusterId = clusterId,
Name = clusterId,
Enterprise = "ZB",
Site = "Site",
// Warm/Hot require NodeCount = 2 (CK_ServerCluster_RedundancyMode_NodeCount); these
// fixtures only need the row to exist, so use the standalone shape.
RedundancyMode = RedundancyMode.None,
NodeCount = 1,
CreatedBy = "test",
});
foreach (var n in nodes)
{
db.ClusterNodes.Add(new ClusterNode
{
NodeId = n.NodeId,
ClusterId = clusterId,
Host = n.Host,
AkkaPort = n.AkkaPort,
ApplicationUri = $"urn:{n.NodeId}",
Enabled = n.Enabled,
MaintenanceMode = n.Maintenance,
CreatedBy = "test",
});
}
db.SaveChanges();
}
/// <summary>
/// Seeds a cluster whose nodes carry a <c>GrpcPort</c> — the gRPC command plane's shape.
/// Kept separate from <see cref="SeedCluster"/> so the ClusterClient facts, which have no
/// opinion about the port, stay exactly as they were.
/// </summary>
private static void SeedGrpcCluster( private static void SeedGrpcCluster(
IDbContextFactory<OtOpcUaConfigDbContext> factory, IDbContextFactory<OtOpcUaConfigDbContext> factory,
string clusterId, string clusterId,
@@ -145,15 +68,16 @@ public class CentralCommunicationActorTests : ControlPlaneActorTestBase
private IActorRef Spawn( private IActorRef Spawn(
IDbContextFactory<OtOpcUaConfigDbContext> factory, IDbContextFactory<OtOpcUaConfigDbContext> factory,
MeshTransportOptions options, MeshTransportOptions options,
IMeshClusterClientFactory clientFactory,
IActorRef? coordinator = null) => IActorRef? coordinator = null) =>
Sys.ActorOf(CentralCommunicationActor.Props(factory, options, clientFactory, () => coordinator)); Sys.ActorOf(CentralCommunicationActor.Props(factory, options, () => coordinator));
[Fact] [Fact]
public void Dps_mode_publishes_on_the_carried_topic_and_creates_no_client() public void Dps_mode_publishes_on_the_carried_topic()
{ {
// Dps is the pair-local default and stays in the tree: it is the correct transport for a
// single-mesh topology, which every integration test and every profile that leaves
// MeshTransport:Mode unset still runs.
var factory = NewInMemoryDbFactory(); var factory = NewInMemoryDbFactory();
var clients = new RecordingClientFactory(() => CreateTestProbe());
var mediatorProbe = CreateTestProbe(); var mediatorProbe = CreateTestProbe();
// Subscribe on the probe's behalf: SubscribeAck goes to the SENDER, so telling the mediator // Subscribe on the probe's behalf: SubscribeAck goes to the SENDER, so telling the mediator
// without an explicit sender would ack to TestActor and this wait would hang. // without an explicit sender would ack to TestActor and this wait would hang.
@@ -161,150 +85,22 @@ public class CentralCommunicationActorTests : ControlPlaneActorTestBase
new Subscribe(DeploymentsTopic, mediatorProbe.Ref), mediatorProbe.Ref); new Subscribe(DeploymentsTopic, mediatorProbe.Ref), mediatorProbe.Ref);
mediatorProbe.ExpectMsg<SubscribeAck>(TimeSpan.FromSeconds(5)); mediatorProbe.ExpectMsg<SubscribeAck>(TimeSpan.FromSeconds(5));
var actor = Spawn(factory, new MeshTransportOptions(), clients); var actor = Spawn(factory, new MeshTransportOptions());
var dispatch = new DispatchDeployment( var dispatch = new DispatchDeployment(
new DeploymentId(Guid.NewGuid()), new RevisionHash("abc"), new CorrelationId(Guid.NewGuid())); new DeploymentId(Guid.NewGuid()), new RevisionHash("abc"), new CorrelationId(Guid.NewGuid()));
actor.Tell(new MeshCommand(DeploymentsTopic, dispatch)); actor.Tell(new MeshCommand(DeploymentsTopic, dispatch));
mediatorProbe.ExpectMsg<DispatchDeployment>(TimeSpan.FromSeconds(5)).ShouldBe(dispatch); mediatorProbe.ExpectMsg<DispatchDeployment>(TimeSpan.FromSeconds(5)).ShouldBe(dispatch);
clients.Calls.ShouldBeEmpty();
}
[Fact]
public void ClusterClient_mode_sends_to_all_not_to_one()
{
// THE assertion of this phase. ClusterClient.Send delivers to exactly ONE registered actor;
// today's DPS publish reaches EVERY DriverHostActor, and the node side has no ClusterId or
// node filter to compensate. Send would deploy to a single node while every other node
// silently kept its old configuration.
var factory = NewInMemoryDbFactory();
SeedCluster(factory, "C1", ("host-a:4053", "host-a", 4053, true, false));
var clients = new RecordingClientFactory(() => CreateTestProbe());
var actor = Spawn(factory, ClusterClientMode(), clients);
AwaitCondition(() => clients.Probes.Count == 1, TimeSpan.FromSeconds(5));
var dispatch = new DispatchDeployment(
new DeploymentId(Guid.NewGuid()), new RevisionHash("abc"), new CorrelationId(Guid.NewGuid()));
actor.Tell(new MeshCommand(DeploymentsTopic, dispatch));
var sent = clients.Probes[0].ExpectMsg<ClusterClient.SendToAll>(TimeSpan.FromSeconds(5));
sent.Path.ShouldBe(MeshPaths.NodeCommunication);
sent.Message.ShouldBe(dispatch);
}
[Fact]
public void One_client_per_cluster_is_created_for_a_multi_cluster_fleet()
{
// THE Phase 6 inversion. Each application Cluster is now its own Akka mesh, and a ClusterClient
// reaches only the mesh whose receptionist it dialled. Central must therefore build one client
// per cluster, each dialling only that cluster's nodes.
var factory = NewInMemoryDbFactory();
SeedCluster(factory, "C1", ("host-a:4053", "host-a", 4053, true, false));
SeedCluster(factory, "C2", ("host-b:4053", "host-b", 4053, true, false));
var clients = new RecordingClientFactory(() => CreateTestProbe());
Spawn(factory, ClusterClientMode(), clients);
AwaitCondition(() => clients.Calls.Count == 2, TimeSpan.FromSeconds(5));
// Give a second refresh a chance to (wrongly) create more.
ExpectNoMsg(TimeSpan.FromMilliseconds(300));
clients.Calls.Count.ShouldBe(2);
clients.ClusterIds.ShouldBe(new[] { "C1", "C2" }, ignoreOrder: true);
// Each client dials ONLY its own cluster's node.
clients.Calls[clients.ClusterIds.IndexOf("C1")].Single().ToString().ShouldContain("host-a");
clients.Calls[clients.ClusterIds.IndexOf("C2")].Single().ToString().ShouldContain("host-b");
}
[Fact]
public void A_command_fans_out_a_send_to_all_to_every_cluster_client()
{
// The other half of the inversion: one MeshCommand must reach EVERY cluster's mesh, so central
// fans SendToAll across all per-cluster clients. A single Send/one-client shape would leave
// every cluster but one silently on its old configuration.
var factory = NewInMemoryDbFactory();
SeedCluster(factory, "C1", ("host-a:4053", "host-a", 4053, true, false));
SeedCluster(factory, "C2", ("host-b:4053", "host-b", 4053, true, false));
var clients = new RecordingClientFactory(() => CreateTestProbe());
var actor = Spawn(factory, ClusterClientMode(), clients);
AwaitCondition(() => clients.Probes.Count == 2, TimeSpan.FromSeconds(5));
var dispatch = new DispatchDeployment(
new DeploymentId(Guid.NewGuid()), new RevisionHash("abc"), new CorrelationId(Guid.NewGuid()));
actor.Tell(new MeshCommand(DeploymentsTopic, dispatch));
foreach (var (clusterId, probe) in clients.ProbeByCluster)
{
var sent = probe.ExpectMsg<ClusterClient.SendToAll>(
TimeSpan.FromSeconds(5), $"cluster {clusterId} must receive the fanned SendToAll");
sent.Path.ShouldBe(MeshPaths.NodeCommunication);
sent.Message.ShouldBe(dispatch);
}
}
[Fact]
public void Disabled_and_maintenance_nodes_are_not_dialled()
{
var factory = NewInMemoryDbFactory();
SeedCluster(
factory,
"C1",
("host-a:4053", "host-a", 4053, true, false),
("host-b:4053", "host-b", 4053, false, false),
("host-c:4053", "host-c", 4053, true, true));
var clients = new RecordingClientFactory(() => CreateTestProbe());
Spawn(factory, ClusterClientMode(), clients);
AwaitCondition(() => clients.Calls.Count == 1, TimeSpan.FromSeconds(5));
clients.Calls[0].Count.ShouldBe(1);
clients.Calls[0].Single().ToString().ShouldContain("host-a");
}
[Fact]
public void A_malformed_node_row_does_not_abort_the_refresh()
{
// The bad row is ordered FIRST on purpose: the failure this guards against is an exception
// that aborts the loop and leaves the contact set half-built.
var factory = NewInMemoryDbFactory();
SeedCluster(
factory,
"C1",
("bad", "not a host!!", 4053, true, false),
("host-a:4053", "host-a", 4053, true, false));
var clients = new RecordingClientFactory(() => CreateTestProbe());
Spawn(factory, ClusterClientMode(), clients);
AwaitCondition(() => clients.Calls.Count == 1, TimeSpan.FromSeconds(5));
clients.Calls[0].ShouldContain(p => p.ToString().Contains("host-a"));
}
[Fact]
public void A_command_with_no_client_is_dropped_with_a_warning()
{
// No ClusterNode rows at all: nothing to dial, so the command must be dropped loudly rather
// than queued. buffer-size = 0 covers the case where a client exists but cannot connect;
// this covers the case where there is no client at all.
var factory = NewInMemoryDbFactory();
var clients = new RecordingClientFactory(() => CreateTestProbe());
var actor = Spawn(factory, ClusterClientMode(), clients);
EventFilter.Warning(contains: "dropping").ExpectOne(TimeSpan.FromSeconds(5), () =>
actor.Tell(new MeshCommand(DeploymentsTopic, "anything")));
} }
[Fact] [Fact]
public void ApplyAck_is_forwarded_to_the_coordinator_preserving_the_sender() public void ApplyAck_is_forwarded_to_the_coordinator_preserving_the_sender()
{ {
var factory = NewInMemoryDbFactory(); var factory = NewInMemoryDbFactory();
var clients = new RecordingClientFactory(() => CreateTestProbe());
var coordinator = CreateTestProbe(); var coordinator = CreateTestProbe();
var node = CreateTestProbe(); var node = CreateTestProbe();
var actor = Spawn(factory, ClusterClientMode(), clients, coordinator.Ref); var actor = Spawn(factory, GrpcMode(), coordinator.Ref);
var ack = new ApplyAck( var ack = new ApplyAck(
new DeploymentId(Guid.NewGuid()), new DeploymentId(Guid.NewGuid()),
@@ -324,8 +120,7 @@ public class CentralCommunicationActorTests : ControlPlaneActorTestBase
public void An_unresolvable_coordinator_logs_rather_than_losing_the_ack_silently() public void An_unresolvable_coordinator_logs_rather_than_losing_the_ack_silently()
{ {
var factory = NewInMemoryDbFactory(); var factory = NewInMemoryDbFactory();
var clients = new RecordingClientFactory(() => CreateTestProbe()); var actor = Spawn(factory, GrpcMode(), coordinator: null);
var actor = Spawn(factory, ClusterClientMode(), clients, coordinator: null);
var ack = new ApplyAck( var ack = new ApplyAck(
new DeploymentId(Guid.NewGuid()), new DeploymentId(Guid.NewGuid()),
@@ -342,12 +137,11 @@ public class CentralCommunicationActorTests : ControlPlaneActorTestBase
public void A_faulted_contact_load_warns_and_leaves_the_actor_alive() public void A_faulted_contact_load_warns_and_leaves_the_actor_alive()
{ {
var factory = NewInMemoryDbFactory(); var factory = NewInMemoryDbFactory();
var clients = new RecordingClientFactory(() => CreateTestProbe()); var actor = Spawn(factory, GrpcMode());
var actor = Spawn(factory, ClusterClientMode(), clients);
// Filter on a phrase unique to the DB-failure warning. EventFilter matches case-INSENSITIVELY, // Filter on a phrase unique to the DB-failure warning. EventFilter matches case-INSENSITIVELY,
// so a looser "was NOT" also caught this actor's other warning ("the mesh ClusterClient was // so a looser "was NOT" would also catch this actor's other warnings and the assertion would
// not created") and the assertion failed with two matches instead of one. // fail with two matches instead of one.
EventFilter.Warning(contains: "contact set was NOT refreshed").ExpectOne( EventFilter.Warning(contains: "contact set was NOT refreshed").ExpectOne(
TimeSpan.FromSeconds(5), TimeSpan.FromSeconds(5),
() => actor.Tell(new Status.Failure(new InvalidOperationException("db down")))); () => actor.Tell(new Status.Failure(new InvalidOperationException("db down"))));
@@ -359,16 +153,15 @@ public class CentralCommunicationActorTests : ControlPlaneActorTestBase
// ------------------------------------------------------------------ Mode = Grpc // ------------------------------------------------------------------ Mode = Grpc
// //
// The third transport. The fan-out must reach EXACTLY the recipient set SendToAll reaches today // The cross-mesh transport. The fan-out must reach EVERY enabled, non-maintenance node in every
// — every enabled, non-maintenance node in every cluster — because the node side still has no // cluster, because the node side has no ClusterId or node filter at all; scoping happens later,
// ClusterId or node filter; scoping happens later, inside the artifact. A narrower fan would // inside the artifact. A narrower fan would leave nodes silently on their old configuration
// leave nodes silently on their old configuration while the deployment sealed green. // while the deployment sealed green.
[Fact] [Fact]
public void Grpc_mode_fans_one_command_to_every_enabled_node_across_all_clusters() public void Grpc_mode_fans_one_command_to_every_enabled_node_across_all_clusters()
{ {
// THE assertion of this task, and the direct counterpart of // THE assertion of this transport: one command, one call per node, across every cluster.
// A_command_fans_out_a_send_to_all_to_every_cluster_client.
var factory = NewInMemoryDbFactory(); var factory = NewInMemoryDbFactory();
SeedGrpcCluster( SeedGrpcCluster(
factory, factory,
@@ -392,31 +185,32 @@ public class CentralCommunicationActorTests : ControlPlaneActorTestBase
} }
[Fact] [Fact]
public void Grpc_mode_creates_no_ClusterClient_and_publishes_nothing_on_pubsub() public void Grpc_mode_publishes_nothing_on_pubsub()
{ {
// DPS is mesh-wide and cannot cross an application-Cluster boundary, so a command that also
// went out on the topic would be delivered twice inside central's own mesh and not at all
// anywhere else.
var factory = NewInMemoryDbFactory(); var factory = NewInMemoryDbFactory();
SeedGrpcCluster(factory, "C1", ("host-a:4053", "host-a", 4056, true, false)); SeedGrpcCluster(factory, "C1", ("host-a:4053", "host-a", 4056, true, false));
var clients = new RecordingClientFactory(() => CreateTestProbe());
var mediatorProbe = CreateTestProbe(); var mediatorProbe = CreateTestProbe();
DistributedPubSub.Get(Sys).Mediator.Tell( DistributedPubSub.Get(Sys).Mediator.Tell(
new Subscribe(DeploymentsTopic, mediatorProbe.Ref), mediatorProbe.Ref); new Subscribe(DeploymentsTopic, mediatorProbe.Ref), mediatorProbe.Ref);
mediatorProbe.ExpectMsg<SubscribeAck>(TimeSpan.FromSeconds(5)); mediatorProbe.ExpectMsg<SubscribeAck>(TimeSpan.FromSeconds(5));
var rig = SpawnGrpcRig(factory, clientFactory: clients); var rig = SpawnGrpcRig(factory);
AwaitContactsLoaded(rig); AwaitContactsLoaded(rig);
rig.Actor.Tell(new MeshCommand(DeploymentsTopic, NewDispatch())); rig.Actor.Tell(new MeshCommand(DeploymentsTopic, NewDispatch()));
AwaitCondition(() => rig.Dialer.Calls.Count == 1, TimeSpan.FromSeconds(5)); AwaitCondition(() => rig.Dialer.Calls.Count == 1, TimeSpan.FromSeconds(5));
clients.Calls.ShouldBeEmpty();
mediatorProbe.ExpectNoMsg(TimeSpan.FromMilliseconds(300)); mediatorProbe.ExpectNoMsg(TimeSpan.FromMilliseconds(300));
} }
[Fact] [Fact]
public void Grpc_mode_does_not_dial_disabled_or_maintenance_nodes() public void Grpc_mode_does_not_dial_disabled_or_maintenance_nodes()
{ {
// Same recipient filter as the ClusterClient contact load, asserted independently so the two // Disabled and maintenance-mode rows are not nodes central talks to; the same filter the
// transports cannot drift on who counts as a node. // contact load applies, asserted here so the fan cannot drift from it.
var factory = NewInMemoryDbFactory(); var factory = NewInMemoryDbFactory();
SeedGrpcCluster( SeedGrpcCluster(
factory, factory,
@@ -677,7 +471,7 @@ public class CentralCommunicationActorTests : ControlPlaneActorTestBase
{ {
// The refresh is not decoration: operators add and retire nodes, and central must pick them // The refresh is not decoration: operators add and retire nodes, and central must pick them
// up without a restart. Under Grpc that is the ONLY thing keeping the dispatch set current — // up without a restart. Under Grpc that is the ONLY thing keeping the dispatch set current —
// there is no receptionist to notice a new registration on its own. // nothing on the node side announces itself; the DB row is the only source.
// //
// Driven by telling the actor its own RefreshContacts tick rather than by shortening // Driven by telling the actor its own RefreshContacts tick rather than by shortening
// ContactRefreshSeconds: a one-second timer would run for the whole fact, adding DB load to // ContactRefreshSeconds: a one-second timer would run for the whole fact, adding DB load to
@@ -721,26 +515,23 @@ public class CentralCommunicationActorTests : ControlPlaneActorTestBase
[Theory] [Theory]
[InlineData(MeshTransportOptions.ModeDps)] [InlineData(MeshTransportOptions.ModeDps)]
[InlineData(MeshTransportOptions.ModeClusterClient)]
[InlineData(MeshTransportOptions.ModeGrpc)] [InlineData(MeshTransportOptions.ModeGrpc)]
public void The_known_cluster_gate_is_published_in_every_transport_mode(string mode) public void The_known_cluster_gate_is_published_in_every_transport_mode(string mode)
{ {
// Replaces an earlier fact asserting the OPPOSITE, whose premise did not survive 1B: the // Replaces an earlier fact asserting the OPPOSITE, whose premise did not survive Phase 1B: the
// gate feeds CentralCommandAuthInterceptor, which is registered on every admin node, and the // gate feeds CentralCommandAuthInterceptor, which is registered on every admin node, and the
// ack service it protects is mapped by LISTENER PORT — neither consults MeshTransport:Mode. // ack service it protects is mapped by LISTENER PORT — neither consults MeshTransport:Mode.
// Gating the publication on Grpc meant a dark-switch rollback to ClusterClient or Dps left // Gating the publication on Grpc meant a rollback to Dps left that endpoint serving with a
// that endpoint serving with a permanently inactive gate, silently restoring both the timing // permanently inactive gate, silently restoring both the timing oracle and the
// oracle and the pre-authentication store/audit flood. The set is central's cluster // pre-authentication store/audit flood. The set is central's cluster inventory, not
// inventory, not transport state. // transport state.
var factory = NewInMemoryDbFactory(); var factory = NewInMemoryDbFactory();
SeedGrpcCluster(factory, "C1", ("host-a:4053", "host-a", 4056, true, false)); SeedGrpcCluster(factory, "C1", ("host-a:4053", "host-a", 4056, true, false));
var psk = new RecordingPskProvider(); var psk = new RecordingPskProvider();
var clients = new RecordingClientFactory(() => CreateTestProbe());
Sys.ActorOf(CentralCommunicationActor.Props( Sys.ActorOf(CentralCommunicationActor.Props(
factory, factory,
new MeshTransportOptions { Mode = mode, ContactRefreshSeconds = 60 }, new MeshTransportOptions { Mode = mode, ContactRefreshSeconds = 60 },
clients,
() => null, () => null,
() => new RecordingDialer(), () => new RecordingDialer(),
() => psk)); () => psk));
@@ -749,12 +540,6 @@ public class CentralCommunicationActorTests : ControlPlaneActorTestBase
() => psk.KnownClusterPublications.Count >= 1, TimeSpan.FromSeconds(5)); () => psk.KnownClusterPublications.Count >= 1, TimeSpan.FromSeconds(5));
psk.KnownClusterPublications[0].ShouldBe(new[] { "C1" }); psk.KnownClusterPublications[0].ShouldBe(new[] { "C1" });
psk.InvalidateAllCount.ShouldBeGreaterThanOrEqualTo(1); psk.InvalidateAllCount.ShouldBeGreaterThanOrEqualTo(1);
// The one load-bearing line the publication itself cannot witness: with the refresh
// timer now running in every mode, only the ClusterClient branch may construct clients.
// With rows seeded, deleting the !_useClusterClient guard turns this red for Dps/Grpc.
if (mode != MeshTransportOptions.ModeClusterClient)
clients.Calls.ShouldBeEmpty();
} }
[Fact] [Fact]
@@ -815,7 +600,6 @@ public class CentralCommunicationActorTests : ControlPlaneActorTestBase
var actor = Sys.ActorOf(CentralCommunicationActor.Props( var actor = Sys.ActorOf(CentralCommunicationActor.Props(
factory, factory,
GrpcMode(), GrpcMode(),
new RecordingClientFactory(() => CreateTestProbe()),
() => null, () => null,
dialer: null, dialer: null,
pskProvider: null)); pskProvider: null));
@@ -871,7 +655,6 @@ public class CentralCommunicationActorTests : ControlPlaneActorTestBase
actor = Sys.ActorOf(CentralCommunicationActor.Props( actor = Sys.ActorOf(CentralCommunicationActor.Props(
factory, factory,
GrpcMode(), GrpcMode(),
new RecordingClientFactory(() => CreateTestProbe()),
() => null, () => null,
DialerFactory, DialerFactory,
PskFactory)); PskFactory));
@@ -924,7 +707,6 @@ public class CentralCommunicationActorTests : ControlPlaneActorTestBase
actor = Sys.ActorOf(CentralCommunicationActor.Props( actor = Sys.ActorOf(CentralCommunicationActor.Props(
factory, factory,
GrpcMode(), GrpcMode(),
new RecordingClientFactory(() => CreateTestProbe()),
() => null, () => null,
() => new RecordingDialer(), () => new RecordingDialer(),
pskProvider: PskFactory)); pskProvider: PskFactory));
@@ -969,7 +751,6 @@ public class CentralCommunicationActorTests : ControlPlaneActorTestBase
actor = Sys.ActorOf(CentralCommunicationActor.Props( actor = Sys.ActorOf(CentralCommunicationActor.Props(
factory, factory,
GrpcMode(), GrpcMode(),
new RecordingClientFactory(() => CreateTestProbe()),
() => null, () => null,
dialer: () => throw new InvalidOperationException("permanently broken"), dialer: () => throw new InvalidOperationException("permanently broken"),
pskProvider: () => psk)); pskProvider: () => psk));
@@ -1006,7 +787,6 @@ public class CentralCommunicationActorTests : ControlPlaneActorTestBase
private GrpcRig SpawnGrpcRig( private GrpcRig SpawnGrpcRig(
IDbContextFactory<OtOpcUaConfigDbContext> factory, IDbContextFactory<OtOpcUaConfigDbContext> factory,
RecordingDialer? dialer = null, RecordingDialer? dialer = null,
IMeshClusterClientFactory? clientFactory = null,
IActorRef? coordinator = null) IActorRef? coordinator = null)
{ {
dialer ??= new RecordingDialer(); dialer ??= new RecordingDialer();
@@ -1014,7 +794,6 @@ public class CentralCommunicationActorTests : ControlPlaneActorTestBase
var actor = Sys.ActorOf(CentralCommunicationActor.Props( var actor = Sys.ActorOf(CentralCommunicationActor.Props(
factory, factory,
GrpcMode(), GrpcMode(),
clientFactory ?? new RecordingClientFactory(() => CreateTestProbe()),
() => coordinator, () => coordinator,
() => dialer, () => dialer,
() => psk)); () => psk));
@@ -55,11 +55,10 @@ public sealed class ControlPlaneStarterOrderTests
{ {
[Theory] [Theory]
[InlineData(MeshTransportOptions.ModeDps)] [InlineData(MeshTransportOptions.ModeDps)]
[InlineData(MeshTransportOptions.ModeClusterClient)]
[InlineData(MeshTransportOptions.ModeGrpc)] [InlineData(MeshTransportOptions.ModeGrpc)]
public async Task An_admin_node_boots_with_the_real_mesh_dependency_graph(string mode) public async Task An_admin_node_boots_with_the_real_mesh_dependency_graph(string mode)
{ {
// Every mode, because the eager resolve was mode-independent: the starter resolved both // Both modes, because the eager resolve was mode-independent: the starter resolved both
// dependencies before anything looked at MeshTransport:Mode, so Dps — the DEFAULT, and the // dependencies before anything looked at MeshTransport:Mode, so Dps — the DEFAULT, and the
// rollback target — died exactly like Grpc did. // rollback target — died exactly like Grpc did.
using var host = BuildAdminHost(mode); using var host = BuildAdminHost(mode);
@@ -82,7 +81,6 @@ public sealed class ControlPlaneStarterOrderTests
[Theory] [Theory]
[InlineData(MeshTransportOptions.ModeDps)] [InlineData(MeshTransportOptions.ModeDps)]
[InlineData(MeshTransportOptions.ModeClusterClient)]
[InlineData(MeshTransportOptions.ModeGrpc)] [InlineData(MeshTransportOptions.ModeGrpc)]
public async Task The_deferred_dependencies_are_resolved_and_used_once_the_host_is_up(string mode) public async Task The_deferred_dependencies_are_resolved_and_used_once_the_host_is_up(string mode)
{ {
@@ -90,7 +88,7 @@ public sealed class ControlPlaneStarterOrderTests
// silent one, with the known-cluster gate permanently unpublished and the dialer permanently // silent one, with the known-cluster gate permanently unpublished and the dialer permanently
// absent. So this asserts the far end of the deferral: the actor resolved the PSK provider // absent. So this asserts the far end of the deferral: the actor resolved the PSK provider
// FROM THE CONTAINER and drove it, through the ordinary contact-refresh path. A Theory over // FROM THE CONTAINER and drove it, through the ordinary contact-refresh path. A Theory over
// all three modes because publication is mode-independent (the gate follows the endpoint). // both modes because publication is mode-independent (the gate follows the endpoint).
// //
// The provider is registered as a TYPE with an ISecretResolver constructor, so the resolve // The provider is registered as a TYPE with an ISecretResolver constructor, so the resolve
// walks the same registry-bound chain the production MeshPskProvider does — which makes // walks the same registry-bound chain the production MeshPskProvider does — which makes
@@ -1,82 +0,0 @@
using System.Collections.Immutable;
using Akka.Actor;
using Akka.Cluster.Tools.Client;
using Akka.Configuration;
using Akka.TestKit.Xunit2;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Tests.Communication;
/// <summary>
/// Guards the one non-obvious part of client creation: recreating a client with the same name
/// inside a single message handler throws, because <c>Context.Stop</c> is asynchronous and the
/// name stays reserved until termination completes.
/// </summary>
public class MeshClusterClientFactoryTests : TestKit
{
private static readonly Config TestConfig = ConfigurationFactory.ParseString(@"
akka.actor.provider = cluster
akka.remote.dot-netty.tcp.port = 0
akka.remote.dot-netty.tcp.hostname = 127.0.0.1")
.WithFallback(ClusterClientReceptionist.DefaultConfig());
public MeshClusterClientFactoryTests()
: base(TestConfig)
{
}
// A real ClusterClient pointed at an unreachable contact is safe in TestKit — it simply retries
// the initial-contact rotation and never fails the test.
private static ImmutableHashSet<ActorPath> UnreachableContact => ImmutableHashSet.Create(
ActorPath.Parse("akka.tcp://otopcua@127.0.0.1:65501/system/receptionist"));
[Fact]
public void Recreating_a_client_in_one_pass_does_not_collide_on_the_actor_name()
{
var factory = new DefaultMeshClusterClientFactory();
var first = factory.Create(Sys, "C1", UnreachableContact);
Sys.Stop(first);
// Stop is asynchronous: `first`'s name is still reserved right here. Without the generation
// suffix this throws InvalidActorNameException — which is precisely what a contact-set
// change would do in production, where stop-then-recreate happens in one handler.
var second = factory.Create(Sys, "C1", UnreachableContact);
second.Path.Name.ShouldNotBe(first.Path.Name);
}
[Fact]
public void Every_incarnation_gets_a_distinct_name()
{
var factory = new DefaultMeshClusterClientFactory();
var names = Enumerable.Range(0, 5)
.Select(_ => factory.Create(Sys, "C1", UnreachableContact).Path.Name)
.ToList();
names.Distinct().Count().ShouldBe(5);
}
[Fact]
public void The_cluster_id_is_woven_into_the_client_actor_name()
{
// Per-cluster clients must be distinguishable in logs and the actor tree, and — more
// importantly — two clusters' clients created on one system must never collide on a name.
var factory = new DefaultMeshClusterClientFactory();
var a = factory.Create(Sys, "site-a", UnreachableContact);
var b = factory.Create(Sys, "site-b", UnreachableContact);
a.Path.Name.ShouldContain("site-a");
b.Path.Name.ShouldContain("site-b");
a.Path.Name.ShouldNotBe(b.Path.Name);
}
// Contact-set propagation is deliberately NOT asserted here: reading initial contacts back
// off a created ClusterClient is not possible, and asserting ClusterClientSettings directly
// would test Akka rather than this factory. It is covered where it can actually fail — the
// real two-ActorSystem boundary test, which only delivers if the contacts reached the client.
}
@@ -13,7 +13,7 @@ using ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Tests.Communication; namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Tests.Communication;
/// <summary> /// <summary>
/// ClusterClient→gRPC Phase 1A — central's outbound command dial. One node, one call, no retry. /// mesh gRPC command plane, Phase 1A — central's outbound command dial. One node, one call, no retry.
/// </summary> /// </summary>
/// <remarks> /// <remarks>
/// The facts that matter are the credential ones: the bearer must be the TARGET node's cluster's /// The facts that matter are the credential ones: the bearer must be the TARGET node's cluster's
@@ -11,7 +11,7 @@ using ZB.MOM.WW.Secrets.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Tests.Communication; namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Tests.Communication;
/// <summary> /// <summary>
/// ClusterClient→gRPC Phase 0 — central's per-cluster PSK lookup. Central talks to N site /// mesh gRPC command plane, Phase 0 — central's per-cluster PSK lookup. Central talks to N site
/// clusters, each with its own key, so every outbound dial and every inbound ack verification /// clusters, each with its own key, so every outbound dial and every inbound ack verification
/// goes through here. /// goes through here.
/// </summary> /// </summary>
@@ -1,446 +0,0 @@
using System.Collections.Immutable;
using Akka.Actor;
using Akka.Cluster;
using Akka.Cluster.Tools.Client;
using Akka.Configuration;
using Microsoft.EntityFrameworkCore;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Cluster;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Admin;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Deploy;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
using ZB.MOM.WW.OtOpcUa.Commons.Types;
using ZB.MOM.WW.OtOpcUa.Configuration;
using ZB.MOM.WW.OtOpcUa.Configuration.Entities;
using ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
using ZB.MOM.WW.OtOpcUa.Runtime.Communication;
using AkkaCluster = Akka.Cluster.Cluster;
namespace ZB.MOM.WW.OtOpcUa.Host.IntegrationTests;
/// <summary>
/// Drives a real <c>ClusterClient</c> across two genuinely separate <see cref="ActorSystem"/>s —
/// the only test in the repo where the mesh boundary carries a message over the wire.
/// </summary>
/// <remarks>
/// <para>
/// The sister project substitutes an in-process relay that unwraps
/// <see cref="ClusterClient.Send"/> and forwards. That tests actor <i>logic</i> and cannot
/// fail for any transport reason: not a missing receptionist extension, not a service that
/// was never registered, not a malformed contact path, not a serialization break. Those are
/// precisely the Phase 2 failure modes, so they are what this test exercises.
/// </para>
/// <para>
/// <b>Two clusters, one ActorSystem name, joined ONLY by ClusterClient.</b> The shared name
/// is mandatory — Akka.Remote matches on the full address, so a contact path naming
/// <c>otopcua</c> cannot reach a system called anything else. Each system seeds itself and
/// neither lists the other, and
/// <see cref="The_two_systems_are_separate_clusters_joined_only_by_the_client"/> asserts the
/// separation outright: if they ever gossiped into one mesh, every delivery below could be
/// plain local messaging and the whole class would be testing nothing.
/// </para>
/// <para>
/// <b>Scope limit.</b> With exactly one node per side, "registered per node" and "registered
/// as a cluster singleton" are indistinguishable by construction, so this class does NOT
/// cover the per-node property whose structural assertions <c>MeshCommActorPathTests</c>
/// dropped. That property's discriminating coverage is live-gate step 8: stop the OLDEST
/// central and the survivor's comm actor must still receive a node ack, which a singleton
/// hosted on the stopped node cannot.
/// </para>
/// </remarks>
public sealed class MeshClusterClientBoundaryTests : IAsyncLifetime
{
private const string SharedSystemName = "otopcua";
private static readonly TimeSpan Arrival = TimeSpan.FromSeconds(30);
/// <summary>
/// How long the falsifiability control waits before concluding nothing arrived. Deliberately
/// shorter than <see cref="Arrival"/> but far longer than a successful delivery takes in
/// practice, so a pass means "the boundary is shut", not "the clock ran out first".
/// </summary>
private static readonly TimeSpan Silence = TimeSpan.FromSeconds(10);
/// <summary>
/// One factory for every client this class creates, because the generation counter that keeps
/// client actor names unique is <b>per factory instance</b>. A second factory on the same
/// system restarts at generation 1 and collides with the first factory's client — which is
/// exactly how this test first failed. Production has one instance per
/// <c>CentralCommunicationActor</c>, so the real code path is safe.
/// </summary>
private readonly DefaultMeshClusterClientFactory _clientFactory = new();
private ActorSystem _central = null!;
private ActorSystem _node = null!;
private IActorRef _nodeComm = null!;
private IActorRef _centralClient = null!;
private TaskCompletionSource<object> _centralInbox = null!;
/// <inheritdoc />
public async ValueTask InitializeAsync()
{
_central = StartSelfSeededSystem(withReceptionist: true);
_node = StartSelfSeededSystem(withReceptionist: true);
await Task.WhenAll(AwaitSingleMemberUpAsync(_central), AwaitSingleMemberUpAsync(_node));
// Central: the ack landing pad, at the path the node's comm actor sends to.
_centralInbox = NewInbox();
var centralComm = _central.ActorOf(
CaptureActor.Props(_centralInbox), MeshPaths.CentralCommunicationName);
ClusterClientReceptionist.Get(_central).RegisterService(centralComm);
// Node: the REAL comm actor, dialling central through the REAL production client factory.
var nodeToCentral = _clientFactory.Create(_node, "central", ReceptionistOf(_central));
_nodeComm = _node.ActorOf(
NodeCommunicationActor.Props(nodeToCentral), MeshPaths.NodeCommunicationName);
ClusterClientReceptionist.Get(_node).RegisterService(_nodeComm);
_centralClient = _clientFactory.Create(_central, "site-a", ReceptionistOf(_node));
}
/// <inheritdoc />
public async ValueTask DisposeAsync()
{
await ShutdownAsync(_central);
await ShutdownAsync(_node);
}
[Fact]
public void The_two_systems_are_separate_clusters_joined_only_by_the_client()
{
// The load-bearing precondition of every other test here. Two systems sharing a name in one
// process are one gossip accident away from being one cluster, at which point ClusterClient
// is no longer the thing under test.
AkkaCluster.Get(_central).State.Members.Count.ShouldBe(1,
"central must be its own single-member cluster, or the boundary is not being crossed");
AkkaCluster.Get(_node).State.Members.Count.ShouldBe(1,
"the node must be its own single-member cluster, or the boundary is not being crossed");
}
[Fact]
public async Task Central_send_to_all_reaches_the_node_comm_actor_and_lands_on_its_event_stream()
{
var inbox = NewInbox();
_node.EventStream.Subscribe(_node.ActorOf(CaptureActor.Props(inbox)), typeof(RestartDriver));
var command = new RestartDriver("C1", "driver-1", "alice", Guid.NewGuid());
var delivered = await SendUntilDeliveredAsync(
_centralClient,
() => new ClusterClient.SendToAll(MeshPaths.NodeCommunication, command),
inbox,
Arrival);
delivered.ShouldNotBeNull(
"SendToAll must reach the node's registered comm actor and be re-emitted node-locally; "
+ "this is the whole central→node leg of the mesh transport");
delivered.ShouldBeOfType<RestartDriver>().DriverInstanceId.ShouldBe("driver-1");
}
[Fact]
public async Task A_node_ack_reaches_central_through_the_node_comm_actors_own_client()
{
// Told directly rather than routed from central, so a failure here can only be the outbound
// leg. The inbound leg has its own test above.
var ack = new ApplyAck(
new DeploymentId(Guid.NewGuid()),
new NodeId("node-a"),
ApplyAckOutcome.Applied,
null,
CorrelationId.NewId());
var delivered = await SendUntilDeliveredAsync(
_nodeComm, () => ack, _centralInbox, Arrival);
delivered.ShouldNotBeNull(
"the node's ApplyAck must reach central's comm actor, or every deployment times out at "
+ "the apply deadline naming nodes that applied successfully");
delivered.ShouldBeOfType<ApplyAck>().NodeId.Value.ShouldBe("node-a");
}
[Fact]
public async Task Falsifiability_control_a_node_without_the_receptionist_extension_receives_nothing()
{
// Without this, the two tests above cannot distinguish "the ClusterClient boundary works"
// from "something else in the process delivered the message". Same message, same client
// factory, same paths, same comm actor — and it must NOT arrive.
//
// HONEST SCOPE: this removes the receptionist extension AND, unavoidably, the RegisterService
// call — resolving ClusterClientReceptionist.Get to register would materialise the very
// extension whose absence is the control. So it falsifies "delivery happens without a
// receptionist boundary", not the extension line in isolation. Verified falsifiable: giving
// this node the extension and the registration turns the test red, which also proves the send
// below is live rather than silently misaddressed.
var deaf = StartSelfSeededSystem(withReceptionist: false);
try
{
await AwaitSingleMemberUpAsync(deaf);
var inbox = NewInbox();
deaf.EventStream.Subscribe(deaf.ActorOf(CaptureActor.Props(inbox)), typeof(RestartDriver));
// Deliberately NOT ClusterClientReceptionist.Get(deaf) — resolving the extension would
// materialise the very receptionist whose absence is the control.
var comm = deaf.ActorOf(NodeCommunicationActor.Props(null), MeshPaths.NodeCommunicationName);
comm.ShouldNotBeNull();
var client = _clientFactory.Create(_central, "deaf", ReceptionistOf(deaf));
var command = new RestartDriver("C1", "driver-1", "alice", Guid.NewGuid());
var deadline = DateTime.UtcNow + Silence;
while (DateTime.UtcNow < deadline && !inbox.Task.IsCompleted)
{
client.Tell(new ClusterClient.SendToAll(MeshPaths.NodeCommunication, command));
await Task.Delay(500, TestContext.Current.CancellationToken);
}
inbox.Task.IsCompleted.ShouldBeFalse(
"delivery survived removing the receptionist extension, so the two positive tests "
+ "above are not measuring the ClusterClient boundary at all");
}
finally
{
await ShutdownAsync(deaf);
}
}
[Fact]
public async Task One_dispatch_through_central_reaches_a_node_in_every_cluster_mesh()
{
// THE Phase 6 assertion at the transport level, driving the REAL CentralCommunicationActor.
// Two genuinely separate site meshes (site-A = _node, site-B created here), each its own
// single-member cluster with its own receptionist. Central must build ONE ClusterClient per
// cluster and fan SendToAll across them, so a single MeshCommand reaches BOTH meshes. A single
// fleet-wide client would reach only the one mesh whose receptionist answered — the exact
// failure this test is red-before-green against.
var siteB = StartSelfSeededSystem(withReceptionist: true);
try
{
await AwaitSingleMemberUpAsync(siteB);
RegisterNodeComm(siteB);
var inboxA = NewInbox();
_node.EventStream.Subscribe(_node.ActorOf(CaptureActor.Props(inboxA)), typeof(RestartDriver));
var inboxB = NewInbox();
siteB.EventStream.Subscribe(siteB.ActorOf(CaptureActor.Props(inboxB)), typeof(RestartDriver));
var dbFactory = SeedTwoClusterDb(SiteAddress(_node), SiteAddress(siteB));
var actor = _central.ActorOf(CentralCommunicationActor.Props(
dbFactory, ClusterClientOptions(), new DefaultMeshClusterClientFactory(), () => null));
var command = new RestartDriver("C1", "driver-1", "alice", Guid.NewGuid());
var deadline = DateTime.UtcNow + Arrival;
while (DateTime.UtcNow < deadline && !(inboxA.Task.IsCompleted && inboxB.Task.IsCompleted))
{
actor.Tell(new MeshCommand("deployments", command));
await Task.Delay(250, TestContext.Current.CancellationToken);
}
inboxA.Task.IsCompleted.ShouldBeTrue(
"site-A's mesh must receive the dispatch through its own per-cluster ClusterClient");
inboxB.Task.IsCompleted.ShouldBeTrue(
"site-B's mesh must receive the SAME dispatch through its own per-cluster ClusterClient; "
+ "a single fleet-wide client would have reached only one of the two meshes");
}
finally
{
await ShutdownAsync(siteB);
}
}
[Fact]
public async Task A_client_built_for_one_cluster_does_not_deliver_into_another_clusters_mesh()
{
// The partition proof for the fan-out: a client dialling ONLY site-A's contacts must land in
// site-A's mesh and NEVER cross into site-B's. This is why the per-cluster fan-out is correct
// rather than accidentally N x broadcasting — each client is scoped to its own mesh.
var siteB = StartSelfSeededSystem(withReceptionist: true);
try
{
await AwaitSingleMemberUpAsync(siteB);
RegisterNodeComm(siteB);
var inboxA = NewInbox();
_node.EventStream.Subscribe(_node.ActorOf(CaptureActor.Props(inboxA)), typeof(RestartDriver));
var inboxB = NewInbox();
siteB.EventStream.Subscribe(siteB.ActorOf(CaptureActor.Props(inboxB)), typeof(RestartDriver));
var clientA = _clientFactory.Create(_central, "site-a", ReceptionistOf(_node));
var command = new RestartDriver("C1", "driver-1", "alice", Guid.NewGuid());
// Keep sending for the full Silence window so any erroneous cross-delivery into site-B has
// every chance to arrive — a pass means the boundary is shut, not that the clock ran out.
var deadline = DateTime.UtcNow + Silence;
while (DateTime.UtcNow < deadline)
{
clientA.Tell(new ClusterClient.SendToAll(MeshPaths.NodeCommunication, command));
await Task.Delay(250, TestContext.Current.CancellationToken);
}
inboxA.Task.IsCompleted.ShouldBeTrue("site-A's client must deliver into site-A's mesh");
inboxB.Task.IsCompleted.ShouldBeFalse(
"a client built for site-A's contacts must NOT deliver into site-B's mesh — the "
+ "per-cluster fan-out must stay partitioned, one client per mesh");
}
finally
{
await ShutdownAsync(siteB);
}
}
/// <summary>Registers a real node-comm actor with a system's receptionist so central can reach it.</summary>
/// <param name="system">The site system to register into.</param>
private static void RegisterNodeComm(ActorSystem system)
{
var comm = system.ActorOf(NodeCommunicationActor.Props(null), MeshPaths.NodeCommunicationName);
ClusterClientReceptionist.Get(system).RegisterService(comm);
}
private static (string Host, int Port) SiteAddress(ActorSystem system)
{
var addr = AkkaCluster.Get(system).SelfAddress;
return (addr.Host!, addr.Port!.Value);
}
private static MeshTransportOptions ClusterClientOptions() => new()
{
Mode = MeshTransportOptions.ModeClusterClient,
CentralContactPoints = ["akka.tcp://otopcua@127.0.0.1:4053"],
ContactRefreshSeconds = 1,
};
private static IDbContextFactory<OtOpcUaConfigDbContext> SeedTwoClusterDb(
(string Host, int Port) siteA, (string Host, int Port) siteB)
{
var factory = new InMemoryConfigDbFactory(Guid.NewGuid().ToString("N"));
using var db = factory.CreateDbContext();
db.ClusterNodes.Add(NodeRow("SITE-A", "node-a", siteA));
db.ClusterNodes.Add(NodeRow("SITE-B", "node-b", siteB));
db.SaveChanges();
return factory;
}
private static ClusterNode NodeRow(string clusterId, string nodeId, (string Host, int Port) addr) => new()
{
NodeId = nodeId,
ClusterId = clusterId,
Host = addr.Host,
AkkaPort = addr.Port,
ApplicationUri = $"urn:{nodeId}",
Enabled = true,
MaintenanceMode = false,
CreatedBy = "test",
};
private sealed class InMemoryConfigDbFactory(string dbName)
: IDbContextFactory<OtOpcUaConfigDbContext>
{
public OtOpcUaConfigDbContext CreateDbContext()
{
var opts = new DbContextOptionsBuilder<OtOpcUaConfigDbContext>()
.UseInMemoryDatabase(dbName)
.Options;
return new OtOpcUaConfigDbContext(opts);
}
}
/// <summary>
/// Starts a single-member cluster on a dynamic port, optionally without the receptionist.
/// </summary>
/// <param name="withReceptionist">
/// <see langword="false"/> drops the receptionist provider from <c>akka.extensions</c>,
/// leaving DistributedPubSub in place so only the one variable changes.
/// </param>
/// <returns>The started system.</returns>
private static ActorSystem StartSelfSeededSystem(bool withReceptionist)
{
var extensions = withReceptionist
? string.Empty
: "akka.extensions = [\"Akka.Cluster.Tools.PublishSubscribe.DistributedPubSubExtensionProvider, "
+ "Akka.Cluster.Tools\"]\n";
// Port 0 + a post-start self-Join, rather than a seed-nodes list: the address is not known
// until the transport binds, and hard-coding ports makes the test collide with itself under
// parallel execution.
var config = ConfigurationFactory.ParseString(
"akka.remote.dot-netty.tcp.hostname = \"127.0.0.1\"\n"
+ "akka.remote.dot-netty.tcp.public-hostname = \"127.0.0.1\"\n"
+ "akka.remote.dot-netty.tcp.port = 0\n"
+ "akka.cluster.seed-nodes = []\n"
+ "akka.loglevel = \"WARNING\"\n"
+ extensions)
.WithFallback(ZB.MOM.WW.OtOpcUa.Cluster.HoconLoader.LoadBaseConfig());
var system = ActorSystem.Create(SharedSystemName, config);
var cluster = AkkaCluster.Get(system);
cluster.Join(cluster.SelfAddress);
return system;
}
private static async Task AwaitSingleMemberUpAsync(ActorSystem system)
{
var cluster = AkkaCluster.Get(system);
var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(30);
while (DateTime.UtcNow < deadline)
{
if (cluster.State.Members.Any(m => m.Status == MemberStatus.Up)) return;
await Task.Delay(100, TestContext.Current.CancellationToken);
}
cluster.State.Members.Any(m => m.Status == MemberStatus.Up).ShouldBeTrue(
$"{cluster.SelfAddress} never reached Up as its own seed");
}
/// <summary>
/// Re-sends until the inbox completes or the deadline passes.
/// </summary>
/// <remarks>
/// A ClusterClient establishes contact asynchronously and <c>buffer-size = 0</c> means anything
/// sent before that lands is dropped on the floor — by design. A single send would therefore
/// race the handshake and fail intermittently for a reason that is not a defect. Re-sending
/// weakens nothing: the control test re-sends on exactly the same schedule and must still
/// receive nothing.
/// </remarks>
/// <param name="target">Actor to send to.</param>
/// <param name="message">Builds the message for each attempt.</param>
/// <param name="inbox">Completion source the receiving capture actor resolves.</param>
/// <param name="timeout">How long to keep trying.</param>
/// <returns>The delivered message, or <see langword="null"/> if none arrived.</returns>
private static async Task<object?> SendUntilDeliveredAsync(
IActorRef target,
Func<object> message,
TaskCompletionSource<object> inbox,
TimeSpan timeout)
{
var deadline = DateTime.UtcNow + timeout;
while (DateTime.UtcNow < deadline)
{
if (inbox.Task.IsCompleted) return await inbox.Task;
target.Tell(message());
await Task.Delay(250, TestContext.Current.CancellationToken);
}
return inbox.Task.IsCompleted ? await inbox.Task : null;
}
private static TaskCompletionSource<object> NewInbox() =>
new(TaskCreationOptions.RunContinuationsAsynchronously);
private static ImmutableHashSet<ActorPath> ReceptionistOf(ActorSystem system) =>
ImmutableHashSet.Create<ActorPath>(
new RootActorPath(AkkaCluster.Get(system).SelfAddress) / "system" / "receptionist");
private static async Task ShutdownAsync(ActorSystem? system)
{
if (system is null) return;
await system.Terminate().WaitAsync(TimeSpan.FromSeconds(15));
}
/// <summary>Completes an inbox with the first message it receives.</summary>
private sealed class CaptureActor : ReceiveActor
{
public static Props Props(TaskCompletionSource<object> sink) =>
Akka.Actor.Props.Create(() => new CaptureActor(sink));
public CaptureActor(TaskCompletionSource<object> sink) => ReceiveAny(m => sink.TrySetResult(m));
}
}
@@ -6,44 +6,58 @@ using ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
namespace ZB.MOM.WW.OtOpcUa.Host.IntegrationTests; namespace ZB.MOM.WW.OtOpcUa.Host.IntegrationTests;
/// <summary> /// <summary>
/// Pins the two mesh comm-actor paths against a <b>really booted</b> host. /// Pins that both mesh comm actors are actually <b>spawned</b>, under their declared names, on a
/// really booted host.
/// </summary> /// </summary>
/// <remarks> /// <remarks>
/// <para> /// <para>
/// <c>/user/central-communication</c> and <c>/user/node-communication</c> are the wire /// <b>What this pins is existence, not a wire contract.</b> Since the cross-mesh transport
/// contract of the central↔node boundary: central sends to a path string, and the node /// became gRPC, both services resolve their comm actor through <c>ActorRegistry</c> rather
/// registers under one. Nothing else in the codebase asserts they agree. Renaming either /// than by path, so a rename is a compile error rather than a silent drop and
/// actor would compile, pass every unit test, deploy — and deliver nothing, with no error /// <see cref="MeshPaths"/> is only a local naming convention. What is still unpinned by
/// on either side, because a ClusterClient send to an unregistered path is simply dropped. /// anything else is whether the two registration sites
/// (<c>WithOtOpcUaControlPlaneSingletons</c> / <c>WithOtOpcUaRuntimeActors</c>) actually run
/// and spawn these actors on a real node — and if either did not, every inbound call would be
/// refused as <c>Unavailable</c> ("the node is still starting"), which reads as a boot-window
/// race rather than as an actor that will never appear.
/// </para> /// </para>
/// <para> /// <para>
/// Booting the real host is the point. A test that spawned the actors itself would pin the /// Booting the real host is the point. A test that spawned the actors itself would pin the
/// constant against itself and prove nothing about the wiring in /// constant against itself and prove nothing about that wiring.
/// <c>WithOtOpcUaControlPlaneSingletons</c> / <c>WithOtOpcUaRuntimeActors</c>.
/// </para> /// </para>
/// </remarks> /// </remarks>
public class MeshCommActorPathTests public class MeshCommActorPathTests
{ {
private static async Task AssertActorExistsAsync(ActorSystem system, string path) /// <summary>
/// Builds the <c>ActorSelection</c> string for a top-level actor name. Composed here rather
/// than held as a constant: the full path is a local addressing detail of this test, not
/// something the product declares — see <see cref="MeshPaths"/>.
/// </summary>
/// <param name="actorName">The declared actor name.</param>
/// <returns>The absolute user-guardian path.</returns>
private static string UserPath(string actorName) => "/user/" + actorName;
private static async Task AssertActorExistsAsync(ActorSystem system, string actorName)
{ {
var path = UserPath(actorName);
var identity = await system.ActorSelection(path) var identity = await system.ActorSelection(path)
.Ask<ActorIdentity>(new Identify(path), TimeSpan.FromSeconds(10)); .Ask<ActorIdentity>(new Identify(path), TimeSpan.FromSeconds(10));
identity.Subject.ShouldNotBeNull( identity.Subject.ShouldNotBeNull(
$"{path} is the ClusterClient wire contract for the central↔node boundary; nothing else " $"no actor exists at {path}; the registration that spawns it did not run, and every "
+ "pins it, and a mismatch delivers nothing without erroring on either side"); + "inbound mesh call to this node would be refused as a misleading readiness rejection");
} }
[Fact] [Fact]
public async Task Both_mesh_comm_actors_exist_at_their_contract_paths() public async Task Both_mesh_comm_actors_are_spawned_under_their_declared_names()
{ {
await using var harness = await TwoNodeClusterHarness.StartAsync(); await using var harness = await TwoNodeClusterHarness.StartAsync();
// Harness nodes carry admin+driver, so BOTH comm actors live on each of them. // Harness nodes carry admin+driver, so BOTH comm actors live on each of them.
await AssertActorExistsAsync(harness.NodeASystem, MeshPaths.CentralCommunication); await AssertActorExistsAsync(harness.NodeASystem, MeshPaths.CentralCommunicationName);
await AssertActorExistsAsync(harness.NodeASystem, MeshPaths.NodeCommunication); await AssertActorExistsAsync(harness.NodeASystem, MeshPaths.NodeCommunicationName);
await AssertActorExistsAsync(harness.NodeBSystem, MeshPaths.CentralCommunication); await AssertActorExistsAsync(harness.NodeBSystem, MeshPaths.CentralCommunicationName);
await AssertActorExistsAsync(harness.NodeBSystem, MeshPaths.NodeCommunication); await AssertActorExistsAsync(harness.NodeBSystem, MeshPaths.NodeCommunicationName);
} }
// NOT TESTED HERE: that the comm actors are registered per node rather than as cluster // NOT TESTED HERE: that the comm actors are registered per node rather than as cluster
@@ -60,7 +74,7 @@ public class MeshCommActorPathTests
// is left to live-gate step 8 (stop the OLDEST central; the survivor's comm actor must still // is left to live-gate step 8 (stop the OLDEST central; the survivor's comm actor must still
// receive a node ack, which a singleton hosted on the stopped node cannot). // receive a node ack, which a singleton hosted on the stopped node cannot).
// //
// NOT MeshClusterClientBoundaryTests: that class proves a send only arrives if the target is // NOT MeshGrpcBoundaryTests: that class proves a command only arrives if the target is
// registered with the receptionist, which is necessary but not discriminating — with one node // registered under its registry key, which is necessary but not discriminating — with one node
// per side, per-node and singleton registration look identical. // per side, per-node and singleton registration look identical.
} }
@@ -0,0 +1,422 @@
using System.Diagnostics.CodeAnalysis;
using System.Net;
using Akka.Actor;
using Akka.Cluster;
using Akka.Configuration;
using Akka.Hosting;
using Grpc.Core;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Server.Kestrel.Core;
using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Cluster;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Admin;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
using ZB.MOM.WW.OtOpcUa.Configuration;
using ZB.MOM.WW.OtOpcUa.Configuration.Entities;
using ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
using ZB.MOM.WW.OtOpcUa.Host.Configuration;
using ZB.MOM.WW.OtOpcUa.Host.Grpc;
using ZB.MOM.WW.OtOpcUa.Runtime;
using ZB.MOM.WW.OtOpcUa.Runtime.Communication;
using AkkaCluster = Akka.Cluster.Cluster;
namespace ZB.MOM.WW.OtOpcUa.Host.IntegrationTests;
/// <summary>
/// Drives the mesh command plane across two genuinely separate <see cref="ActorSystem"/>s over
/// a real gRPC wire — the only test in the repo where the central↔node boundary carries a
/// message off-process.
/// </summary>
/// <remarks>
/// <para>
/// Replaces <c>MeshClusterClientBoundaryTests</c>, which did the same job for the Akka
/// ClusterClient transport that this program deleted. The reason it exists is unchanged:
/// a fake dialer tests actor <i>logic</i> and cannot fail for any transport reason — not a
/// service that was never mapped, not an interceptor that refuses the call, not a
/// serialization break. Those are the failure modes this class exercises, end to end
/// through the REAL <c>CentralCommunicationActor</c>, <c>MeshGrpcCommandDialer</c>,
/// <c>MeshCommandAuthInterceptor</c>, <c>NodeCommandGrpcService</c> and
/// <c>NodeCommunicationActor</c>.
/// </para>
/// <para>
/// <b>Three clusters, one ActorSystem name, joined ONLY by gRPC.</b> The shared name is kept
/// from the predecessor so the shape is comparable; each system seeds itself and none lists
/// another, and
/// <see cref="No_cross_mesh_Akka_association_forms_between_central_and_either_site"/>
/// asserts the separation outright. That assertion is the whole point of the program — the
/// cross-WAN Akka association is gone — so it is kept as a test rather than left to a live
/// gate: if the three systems ever gossiped into one mesh, every delivery below could be
/// plain local messaging and the class would be testing nothing.
/// </para>
/// <para>
/// <b>Per-cluster keys, not one fleet key.</b> Each site node's interceptor is configured
/// with its OWN <c>MeshTransport:GrpcPsk</c> and central's provider hands the dialer the key
/// for the cluster it is dialling. A fan that presented the wrong cluster's key would be
/// refused at the far side, so the happy path below also pins that the dialer selects by
/// <c>ClusterId</c>.
/// </para>
/// </remarks>
public sealed class MeshGrpcBoundaryTests : IAsyncLifetime
{
private const string SharedSystemName = "otopcua";
private const string SiteAKey = "psk-site-a";
private const string SiteBKey = "psk-site-b";
private static readonly TimeSpan Arrival = TimeSpan.FromSeconds(30);
/// <summary>
/// How long the falsifiability control waits before concluding nothing arrived. Deliberately
/// shorter than <see cref="Arrival"/> but far longer than a successful delivery takes in
/// practice, so a pass means "the boundary refused it", not "the clock ran out first".
/// </summary>
private static readonly TimeSpan Silence = TimeSpan.FromSeconds(10);
private ActorSystem _central = null!;
private SiteNode _siteA = null!;
private SiteNode _siteB = null!;
/// <inheritdoc />
public async ValueTask InitializeAsync()
{
_central = StartSelfSeededSystem();
_siteA = await SiteNode.StartAsync("SITE-A", SiteAKey);
_siteB = await SiteNode.StartAsync("SITE-B", SiteBKey);
await Task.WhenAll(
AwaitSingleMemberUpAsync(_central),
AwaitSingleMemberUpAsync(_siteA.System),
AwaitSingleMemberUpAsync(_siteB.System));
}
/// <inheritdoc />
public async ValueTask DisposeAsync()
{
await _siteA.DisposeAsync();
await _siteB.DisposeAsync();
await ShutdownAsync(_central);
}
[Fact]
public void No_cross_mesh_Akka_association_forms_between_central_and_either_site()
{
// THE assertion of the ClusterClient→gRPC program, kept as a test rather than a live-gate
// note. Three systems sharing a name in one process are one gossip accident away from being
// one cluster, at which point the gRPC boundary is no longer the thing under test — and,
// more importantly, the cross-WAN Akka association this program removed would be back.
var central = AkkaCluster.Get(_central);
var siteA = AkkaCluster.Get(_siteA.System);
var siteB = AkkaCluster.Get(_siteB.System);
central.State.Members.Count.ShouldBe(1,
"central must be its own single-member cluster; commands cross to a site over gRPC only");
siteA.State.Members.Count.ShouldBe(1, "site-A must be its own single-member cluster");
siteB.State.Members.Count.ShouldBe(1, "site-B must be its own single-member cluster");
// Stated separately from the counts: a count of 1 on each side would also hold in the instant
// before a join completed, whereas naming the addresses says no side ever learned another's.
foreach (var (name, cluster) in new[]
{
("central", central), ("site-A", siteA), ("site-B", siteB),
})
{
var others = new[] { central.SelfAddress, siteA.SelfAddress, siteB.SelfAddress }
.Where(a => a != cluster.SelfAddress);
foreach (var other in others)
{
cluster.State.Members.ShouldNotContain(
m => m.Address == other,
$"{name} must not have {other} as a cluster member — the cross-mesh Akka "
+ "association is exactly what the gRPC command plane replaced");
}
}
}
[Fact]
public async Task One_dispatch_through_central_reaches_a_node_in_every_cluster_mesh()
{
// The direct counterpart of the deleted ClusterClient fan-out test, at the transport level and
// driving the REAL CentralCommunicationActor. Two genuinely separate site meshes, each with
// its own gRPC command endpoint and its own per-cluster key; one MeshCommand must reach BOTH.
var dbFactory = SeedTwoClusterDb(_siteA, _siteB);
var actor = _central.ActorOf(CentralCommunicationActor.Props(
dbFactory,
GrpcOptions(),
() => null,
() => Dialer(),
() => new FixedPskProvider()));
var command = new RestartDriver("C1", "driver-1", "alice", Guid.NewGuid());
var deadline = DateTime.UtcNow + Arrival;
while (DateTime.UtcNow < deadline
&& !(_siteA.Inbox.Task.IsCompleted && _siteB.Inbox.Task.IsCompleted))
{
// Re-sent rather than sent once: the actor's first contact load is a piped DB read, and a
// command told before it lands is dropped as "no targets" by design. Re-sending weakens
// nothing — the wrong-key control below re-sends on the same schedule and must still
// receive nothing.
actor.Tell(new MeshCommand("deployments", command));
await Task.Delay(250, TestContext.Current.CancellationToken);
}
_siteA.Inbox.Task.IsCompleted.ShouldBeTrue(
"site-A's mesh must receive the dispatch on its own gRPC command endpoint");
_siteB.Inbox.Task.IsCompleted.ShouldBeTrue(
"site-B's mesh must receive the SAME dispatch on its own endpoint, under its own "
+ "per-cluster key — a fan that stopped at the first cluster would leave site-B silently "
+ "on its old configuration while the deployment sealed green");
(await _siteA.Inbox.Task).ShouldBeOfType<RestartDriver>().DriverInstanceId.ShouldBe("driver-1");
(await _siteB.Inbox.Task).ShouldBeOfType<RestartDriver>().DriverInstanceId.ShouldBe("driver-1");
}
[Fact]
public async Task Falsifiability_control_a_dial_carrying_the_wrong_cluster_key_delivers_nothing()
{
// Without this, the test above cannot distinguish "the gRPC boundary carried the command"
// from "something else in the process delivered it". Same wire, same service, same comm
// actor — and with a key site-A's interceptor does not accept, it must NOT arrive.
var envelope = MeshTransportDtoMapper.ToEnvelope(
new RestartDriver("C1", "driver-1", "alice", Guid.NewGuid()));
var dialer = Dialer(new FixedPskProvider("not-the-key"));
var deadline = DateTime.UtcNow + Silence;
while (DateTime.UtcNow < deadline && !_siteA.Inbox.Task.IsCompleted)
{
try
{
await dialer.DispatchAsync(
_siteA.Endpoint, envelope, "SITE-A", TestContext.Current.CancellationToken);
}
catch (RpcException)
{
// Expected: the node's MeshCommandAuthInterceptor refuses the bearer. The assertion
// is what did NOT arrive, so the rejection itself is swallowed rather than asserted
// on here — MeshCommandAuthInterceptorTests owns the status-code contract.
}
await Task.Delay(500, TestContext.Current.CancellationToken);
}
_siteA.Inbox.Task.IsCompleted.ShouldBeFalse(
"delivery survived presenting the wrong per-cluster key, so the positive test above is "
+ "not measuring the authenticated gRPC boundary at all");
// Same node, same dialer shape, right key: proves the silence above was the auth gate rather
// than a dead endpoint or a misaddressed call.
await Dialer().DispatchAsync(
_siteA.Endpoint, envelope, "SITE-A", TestContext.Current.CancellationToken);
await _siteA.Inbox.Task.WaitAsync(Arrival, TestContext.Current.CancellationToken);
}
// ---- harness ---------------------------------------------------------------------------------
private static MeshGrpcCommandDialer Dialer(IMeshPskProvider? psk = null) =>
new(psk ?? new FixedPskProvider(), Options.Create(GrpcOptions()),
NullLogger<MeshGrpcCommandDialer>.Instance);
private static MeshTransportOptions GrpcOptions() => new()
{
Mode = MeshTransportOptions.ModeGrpc,
ContactRefreshSeconds = 1,
DispatchTimeoutSeconds = 10,
};
/// <summary>
/// Hands out the per-cluster key each site node's interceptor expects. Stands in for the real
/// <c>MeshPskProvider</c>, whose own subject (secret-store lookup, caching, the known-cluster
/// gate) is <c>MeshPskProviderTests</c>' and would add a KEK and a SQLite file to a class
/// whose subject is the wire.
/// </summary>
private sealed class FixedPskProvider : IMeshPskProvider
{
private readonly string? _override;
public FixedPskProvider(string? overrideKey = null) => _override = overrideKey;
public bool TryGetPsk(string clusterId, [NotNullWhen(true)] out string? psk)
{
psk = _override ?? KeyFor(clusterId);
return psk is not null;
}
public bool Matches(string clusterId, string presented) =>
string.Equals(presented, KeyFor(clusterId), StringComparison.Ordinal);
public void SetKnownClusters(IReadOnlyCollection<string> clusterIds) { }
public void Invalidate(string clusterId) { }
public void InvalidateAll() { }
private static string? KeyFor(string clusterId) => clusterId switch
{
"SITE-A" => SiteAKey,
"SITE-B" => SiteBKey,
_ => null,
};
}
private static IDbContextFactory<OtOpcUaConfigDbContext> SeedTwoClusterDb(SiteNode a, SiteNode b)
{
var factory = new InMemoryConfigDbFactory(Guid.NewGuid().ToString("N"));
using var db = factory.CreateDbContext();
db.ClusterNodes.Add(NodeRow("SITE-A", "node-a", a.Port));
db.ClusterNodes.Add(NodeRow("SITE-B", "node-b", b.Port));
db.SaveChanges();
return factory;
}
private static ClusterNode NodeRow(string clusterId, string nodeId, int grpcPort) => new()
{
NodeId = nodeId,
ClusterId = clusterId,
Host = "127.0.0.1",
AkkaPort = 4053,
GrpcPort = grpcPort,
ApplicationUri = $"urn:{nodeId}",
Enabled = true,
MaintenanceMode = false,
CreatedBy = "test",
};
private sealed class InMemoryConfigDbFactory(string dbName)
: IDbContextFactory<OtOpcUaConfigDbContext>
{
public OtOpcUaConfigDbContext CreateDbContext()
{
var opts = new DbContextOptionsBuilder<OtOpcUaConfigDbContext>()
.UseInMemoryDatabase(dbName)
.Options;
return new OtOpcUaConfigDbContext(opts);
}
}
/// <summary>
/// One site: its own single-member cluster, its own real <c>NodeCommunicationActor</c>, and a
/// real h2c Kestrel listener serving the real <c>NodeCommandGrpcService</c> behind the real
/// <c>MeshCommandAuthInterceptor</c> — the production shape of a driver node's command port.
/// </summary>
private sealed class SiteNode : IAsyncDisposable
{
private WebApplication _server = null!;
public ActorSystem System { get; private set; } = null!;
public int Port { get; private set; }
public string Endpoint => $"http://127.0.0.1:{Port}";
/// <summary>Completes with the first command re-emitted on this node's EventStream.</summary>
public TaskCompletionSource<object> Inbox { get; } =
new(TaskCreationOptions.RunContinuationsAsynchronously);
public static async Task<SiteNode> StartAsync(string clusterId, string psk)
{
var node = new SiteNode { System = StartSelfSeededSystem(), Port = GetFreePort() };
// The REAL comm actor, under the registry key the gRPC service resolves. Nothing else
// makes it reachable — there is no receptionist registration any more.
var comm = node.System.ActorOf(
NodeCommunicationActor.Props(), MeshPaths.NodeCommunicationName);
node.System.EventStream.Subscribe(
node.System.ActorOf(CaptureActor.Props(node.Inbox)), typeof(RestartDriver));
var registry = new ActorRegistry();
registry.Register<NodeCommunicationActorKey>(comm);
var builder = WebApplication.CreateBuilder();
builder.Environment.ApplicationName = typeof(MeshGrpcBoundaryTests).Assembly.GetName().Name!;
builder.Logging.ClearProviders();
builder.Services.AddSingleton(registry);
builder.Services.Configure<MeshTransportOptions>(o =>
{
o.Mode = MeshTransportOptions.ModeGrpc;
o.GrpcPsk = psk;
});
builder.Services.AddGrpc(o => o.Interceptors.Add<MeshCommandAuthInterceptor>());
builder.WebHost.ConfigureKestrel(k =>
k.ListenAnyIP(node.Port, o => o.Protocols = HttpProtocols.Http2));
node._server = builder.Build();
node._server.MapGrpcService<NodeCommandGrpcService>();
await node._server.StartAsync(TestContext.Current.CancellationToken);
_ = clusterId; // Named for readability at the call site; the key IS the cluster boundary.
return node;
}
public async ValueTask DisposeAsync()
{
await _server.StopAsync(TimeSpan.FromSeconds(10));
await _server.DisposeAsync();
await ShutdownAsync(System);
}
}
/// <summary>Starts a single-member cluster on a dynamic remoting port.</summary>
/// <remarks>
/// Port 0 + a post-start self-Join, rather than a seed-nodes list: the address is not known
/// until the transport binds, and hard-coding ports makes the test collide with itself under
/// parallel execution.
/// </remarks>
/// <returns>The started system.</returns>
private static ActorSystem StartSelfSeededSystem()
{
var config = ConfigurationFactory.ParseString(
"akka.remote.dot-netty.tcp.hostname = \"127.0.0.1\"\n"
+ "akka.remote.dot-netty.tcp.public-hostname = \"127.0.0.1\"\n"
+ "akka.remote.dot-netty.tcp.port = 0\n"
+ "akka.cluster.seed-nodes = []\n"
+ "akka.loglevel = \"WARNING\"\n")
.WithFallback(HoconLoader.LoadBaseConfig());
var system = ActorSystem.Create(SharedSystemName, config);
var cluster = AkkaCluster.Get(system);
cluster.Join(cluster.SelfAddress);
return system;
}
private static async Task AwaitSingleMemberUpAsync(ActorSystem system)
{
var cluster = AkkaCluster.Get(system);
var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(30);
while (DateTime.UtcNow < deadline)
{
if (cluster.State.Members.Any(m => m.Status == MemberStatus.Up)) return;
await Task.Delay(100, TestContext.Current.CancellationToken);
}
cluster.State.Members.Any(m => m.Status == MemberStatus.Up).ShouldBeTrue(
$"{cluster.SelfAddress} never reached Up as its own seed");
}
private static async Task ShutdownAsync(ActorSystem? system)
{
if (system is null) return;
await system.Terminate().WaitAsync(TimeSpan.FromSeconds(15));
}
private static int GetFreePort()
{
using var listener = new System.Net.Sockets.TcpListener(IPAddress.Loopback, 0);
listener.Start();
var port = ((IPEndPoint)listener.LocalEndpoint).Port;
listener.Stop();
return port;
}
/// <summary>Completes an inbox with the first message it receives.</summary>
private sealed class CaptureActor : ReceiveActor
{
public static Props Props(TaskCompletionSource<object> sink) =>
Akka.Actor.Props.Create(() => new CaptureActor(sink));
public CaptureActor(TaskCompletionSource<object> sink) => ReceiveAny(m => sink.TrySetResult(m));
}
}
@@ -9,7 +9,7 @@ using ZB.MOM.WW.OtOpcUa.Host.Configuration;
namespace ZB.MOM.WW.OtOpcUa.Host.Tests.Configuration; namespace ZB.MOM.WW.OtOpcUa.Host.Tests.Configuration;
/// <summary> /// <summary>
/// ClusterClient→gRPC Phase 0 — the central-side ack endpoint's inbound gate. Mapping /// mesh gRPC command plane, Phase 0 — the central-side ack endpoint's inbound gate. Mapping
/// <c>CentralCommandService</c> onto the pipeline is a Phase 1 task; this test covers only the /// <c>CentralCommandService</c> onto the pipeline is a Phase 1 task; this test covers only the
/// interceptor's own authorization behavior. /// interceptor's own authorization behavior.
/// </summary> /// </summary>
@@ -9,7 +9,7 @@ using ZB.MOM.WW.OtOpcUa.Host.Configuration;
namespace ZB.MOM.WW.OtOpcUa.Host.Tests.Configuration; namespace ZB.MOM.WW.OtOpcUa.Host.Tests.Configuration;
/// <summary> /// <summary>
/// ClusterClient→gRPC Phase 0 — the node-side command endpoint's inbound gate. Mapping /// mesh gRPC command plane, Phase 0 — the node-side command endpoint's inbound gate. Mapping
/// <c>NodeCommandService</c> onto the pipeline is a Phase 1 task; this test covers only the /// <c>NodeCommandService</c> onto the pipeline is a Phase 1 task; this test covers only the
/// interceptor's own authorization behavior. /// interceptor's own authorization behavior.
/// </summary> /// </summary>
@@ -16,10 +16,10 @@ using ZB.MOM.WW.OtOpcUa.Runtime;
namespace ZB.MOM.WW.OtOpcUa.Host.Tests.Grpc; namespace ZB.MOM.WW.OtOpcUa.Host.Tests.Grpc;
/// <summary> /// <summary>
/// Verifies the node-side command ingress (ClusterClient→gRPC migration Phase 1): each of the four /// Verifies the node-side command ingress (mesh gRPC command plane, Phase 1): each of the four
/// central→node commands is decoded off its <see cref="CommandEnvelope"/> and handed to the /// central→node commands is decoded off its <see cref="CommandEnvelope"/> and handed to the
/// registered <c>NodeCommunicationActor</c> as the same CLR record the ClusterClient transport /// registered <c>NodeCommunicationActor</c> as the same CLR record every other transport
/// delivered, an undecodable envelope is refused with <see cref="StatusCode.InvalidArgument"/>, and /// delivers, an undecodable envelope is refused with <see cref="StatusCode.InvalidArgument"/>, and
/// a call landing in the boot window before the actor is registered is refused with /// a call landing in the boot window before the actor is registered is refused with
/// <see cref="StatusCode.Unavailable"/>. /// <see cref="StatusCode.Unavailable"/>.
/// </summary> /// </summary>
@@ -50,7 +50,7 @@ public class EventStreamCommandDeliveryTests : RuntimeActorTestBase
var driverHost = Sys.ActorOf( var driverHost = Sys.ActorOf(
DriverHostActor.Props(NewInMemoryDbFactory(), new NodeId("127.0.0.1:4053"), ackRouter: ackRouter.Ref), DriverHostActor.Props(NewInMemoryDbFactory(), new NodeId("127.0.0.1:4053"), ackRouter: ackRouter.Ref),
"driver-host"); "driver-host");
var comm = Sys.ActorOf(NodeCommunicationActor.Props(centralClient: null), "node-communication"); var comm = Sys.ActorOf(NodeCommunicationActor.Props(), "node-communication");
// Warm-up: a direct Tell whose ack proves PreStart has completed, and with it the // Warm-up: a direct Tell whose ack proves PreStart has completed, and with it the
// EventStream subscription. Without this the relay below races the subscription. // EventStream subscription. Without this the relay below races the subscription.
@@ -141,7 +141,7 @@ akka {
ScriptedAlarmHostActor.Props( ScriptedAlarmHostActor.Props(
CreateTestProbe().Ref, CreateTestProbe().Ref, upstream, engine, new NodeId("node-A")), CreateTestProbe().Ref, CreateTestProbe().Ref, upstream, engine, new NodeId("node-A")),
"scripted-alarm-host"); "scripted-alarm-host");
var comm = Sys.ActorOf(NodeCommunicationActor.Props(centralClient: null), "node-communication"); var comm = Sys.ActorOf(NodeCommunicationActor.Props(), "node-communication");
var cmd = new AlarmCommand("alarm-unowned", "Acknowledge", "alice", "c", null); var cmd = new AlarmCommand("alarm-unowned", "Acknowledge", "alice", "c", null);
@@ -13,21 +13,21 @@ using ZB.MOM.WW.OtOpcUa.Runtime.Communication;
namespace ZB.MOM.WW.OtOpcUa.Runtime.Tests.Communication; namespace ZB.MOM.WW.OtOpcUa.Runtime.Tests.Communication;
/// <summary> /// <summary>
/// The node-side gRPC ack sender (ClusterClient→gRPC migration Task 1B.2). The client is a fake, /// The node-side gRPC ack sender (mesh gRPC command plane, Task 1B.2). The client is a fake,
/// so what is under test is the sender's own logic: the ordered single-pass over /// so what is under test is the sender's own logic: the ordered single-pass over
/// <c>CentralGrpcEndpoints</c>, the per-call credentials, and the drop that used to be silent. /// <c>CentralGrpcEndpoints</c>, the per-call credentials, and the drop that used to be silent.
/// </summary> /// </summary>
/// <remarks> /// <remarks>
/// <para> /// <para>
/// <b>The load-bearing fact is that ONE endpoint is dialled on success.</b> This replaces an /// <b>The load-bearing fact is that ONE endpoint is dialled on success.</b> This replaces an
/// Akka <c>ClusterClient.Send</c>, which is single-target at-most-once — not a /// one ack per node per dispatch, single-target at-most-once — not a
/// <c>SendToAll</c>. Both central nodes route an ack to the same /// <c>SendToAll</c>. Both central nodes route an ack to the same
/// <c>ConfigPublishCoordinator</c> singleton proxy, whose NodeId-keyed upsert absorbs a /// <c>ConfigPublishCoordinator</c> singleton proxy, whose NodeId-keyed upsert absorbs a
/// duplicate idempotently — so the single-endpoint rule is defence in depth and a bound /// duplicate idempotently — so the single-endpoint rule is defence in depth and a bound
/// on wasted dials, not what protects the seal. /// on wasted dials, not what protects the seal.
/// </para> /// </para>
/// <para> /// <para>
/// The failure side matters just as much: the receptionist dropped an undeliverable ack in /// The failure side matters just as much: an undeliverable ack is dropped in
/// silence, and the whole point of the migration is that the same drop is now a Warning naming /// silence, and the whole point of the migration is that the same drop is now a Warning naming
/// every endpoint and the status it answered with. /// every endpoint and the status it answered with.
/// </para> /// </para>
@@ -1,5 +1,4 @@
using Akka.Actor; using Akka.Actor;
using Akka.Cluster.Tools.Client;
using Shouldly; using Shouldly;
using Xunit; using Xunit;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Admin; using ZB.MOM.WW.OtOpcUa.Commons.Messages.Admin;
@@ -24,7 +23,7 @@ public class NodeCommunicationActorTests : RuntimeActorTestBase
{ {
var subscriber = CreateTestProbe(); var subscriber = CreateTestProbe();
Sys.EventStream.Subscribe(subscriber.Ref, typeof(DispatchDeployment)); Sys.EventStream.Subscribe(subscriber.Ref, typeof(DispatchDeployment));
var actor = Sys.ActorOf(NodeCommunicationActor.Props(centralClient: null)); var actor = Sys.ActorOf(NodeCommunicationActor.Props());
var dispatch = NewDispatch(); var dispatch = NewDispatch();
actor.Tell(dispatch); actor.Tell(dispatch);
@@ -40,7 +39,7 @@ public class NodeCommunicationActorTests : RuntimeActorTestBase
// subscriber. One publish per inbound command is the whole point. // subscriber. One publish per inbound command is the whole point.
var subscriber = CreateTestProbe(); var subscriber = CreateTestProbe();
Sys.EventStream.Subscribe(subscriber.Ref, typeof(DispatchDeployment)); Sys.EventStream.Subscribe(subscriber.Ref, typeof(DispatchDeployment));
var actor = Sys.ActorOf(NodeCommunicationActor.Props(centralClient: null)); var actor = Sys.ActorOf(NodeCommunicationActor.Props());
actor.Tell(NewDispatch()); actor.Tell(NewDispatch());
@@ -53,7 +52,7 @@ public class NodeCommunicationActorTests : RuntimeActorTestBase
{ {
var subscriber = CreateTestProbe(); var subscriber = CreateTestProbe();
Sys.EventStream.Subscribe(subscriber.Ref, typeof(RestartDriver)); Sys.EventStream.Subscribe(subscriber.Ref, typeof(RestartDriver));
var actor = Sys.ActorOf(NodeCommunicationActor.Props(centralClient: null)); var actor = Sys.ActorOf(NodeCommunicationActor.Props());
var msg = new RestartDriver("C1", "driver-1", "alice", Guid.NewGuid()); var msg = new RestartDriver("C1", "driver-1", "alice", Guid.NewGuid());
actor.Tell(msg); actor.Tell(msg);
@@ -66,7 +65,7 @@ public class NodeCommunicationActorTests : RuntimeActorTestBase
{ {
var subscriber = CreateTestProbe(); var subscriber = CreateTestProbe();
Sys.EventStream.Subscribe(subscriber.Ref, typeof(ReconnectDriver)); Sys.EventStream.Subscribe(subscriber.Ref, typeof(ReconnectDriver));
var actor = Sys.ActorOf(NodeCommunicationActor.Props(centralClient: null)); var actor = Sys.ActorOf(NodeCommunicationActor.Props());
var msg = new ReconnectDriver("C1", "driver-1", "alice", Guid.NewGuid()); var msg = new ReconnectDriver("C1", "driver-1", "alice", Guid.NewGuid());
actor.Tell(msg); actor.Tell(msg);
@@ -82,7 +81,7 @@ public class NodeCommunicationActorTests : RuntimeActorTestBase
// leg possible without a registration handshake. // leg possible without a registration handshake.
var subscriber = CreateTestProbe(); var subscriber = CreateTestProbe();
Sys.EventStream.Subscribe(subscriber.Ref, typeof(AlarmCommand)); Sys.EventStream.Subscribe(subscriber.Ref, typeof(AlarmCommand));
var actor = Sys.ActorOf(NodeCommunicationActor.Props(centralClient: null)); var actor = Sys.ActorOf(NodeCommunicationActor.Props());
var msg = new AlarmCommand("alarm-1", "Acknowledge", "alice", "comment", null); var msg = new AlarmCommand("alarm-1", "Acknowledge", "alice", "comment", null);
actor.Tell(msg); actor.Tell(msg);
@@ -91,32 +90,11 @@ public class NodeCommunicationActorTests : RuntimeActorTestBase
} }
[Fact] [Fact]
public void ApplyAck_goes_to_central_as_a_Send_not_a_SendToAll() public void An_ack_with_no_ack_sender_is_dropped_with_a_warning()
{ {
// Send, not SendToAll: the deploy coordinator is one singleton behind the central comm // Dps: DriverHostActor publishes the ack on the deployment-acks topic and never routes it
// actor's proxy. The coordinator's NodeId-keyed upsert absorbs a duplicate idempotently, // here, so an ack arriving with no sender wired is a mis-wiring and must say so.
// so the single-target rule is defence in depth and a bound on wasted work, not a var actor = Sys.ActorOf(NodeCommunicationActor.Props());
// correctness requirement.
var client = CreateTestProbe();
var actor = Sys.ActorOf(NodeCommunicationActor.Props(client.Ref));
var ack = new ApplyAck(
new DeploymentId(Guid.NewGuid()),
new NodeId("host-a:4053"),
ApplyAckOutcome.Applied,
null,
new CorrelationId(Guid.NewGuid()));
actor.Tell(ack);
var sent = client.ExpectMsg<ClusterClient.Send>(TimeSpan.FromSeconds(5));
sent.Path.ShouldBe(MeshPaths.CentralCommunication);
sent.Message.ShouldBe(ack);
}
[Fact]
public void An_ack_with_no_central_client_is_dropped_with_a_warning()
{
var actor = Sys.ActorOf(NodeCommunicationActor.Props(centralClient: null));
var ack = new ApplyAck( var ack = new ApplyAck(
new DeploymentId(Guid.NewGuid()), new DeploymentId(Guid.NewGuid()),
@@ -129,7 +107,7 @@ public class NodeCommunicationActorTests : RuntimeActorTestBase
TimeSpan.FromSeconds(5), () => actor.Tell(ack)); TimeSpan.FromSeconds(5), () => actor.Tell(ack));
} }
// ---- Mode=Grpc (ClusterClient→gRPC migration Task 1B.2) ------------------------------------- // ---- Mode=Grpc (the cross-mesh ack leg) -----------------------------------------------------
private static ApplyAck NewAck() => new( private static ApplyAck NewAck() => new(
new DeploymentId(Guid.NewGuid()), new DeploymentId(Guid.NewGuid()),
@@ -141,13 +119,12 @@ public class NodeCommunicationActorTests : RuntimeActorTestBase
[Fact] [Fact]
public void Under_Grpc_the_ack_goes_to_the_sender_exactly_once() public void Under_Grpc_the_ack_goes_to_the_sender_exactly_once()
{ {
// The gRPC counterpart of the ClusterClient.Send fact above, and it carries the same // One post per node per dispatch, because both central endpoints route to the SAME
// single-count obligation: one post per node per dispatch, because both central endpoints // coordinator singleton proxy. Where the ack goes among the endpoints is the sender's
// route to the SAME coordinator singleton proxy. Where the ack goes among the endpoints is the // business (GrpcApplyAckSenderTests); what this pins is that the actor hands it over once
// sender's business (GrpcApplyAckSenderTests); what this pins is that the actor hands it over // and does not also fan it out itself.
// once and does not also fan it out itself.
var sender = new RecordingAckSender(); var sender = new RecordingAckSender();
var actor = Sys.ActorOf(NodeCommunicationActor.Props(centralClient: null, ackSender: sender)); var actor = Sys.ActorOf(NodeCommunicationActor.Props(sender));
var ack = NewAck(); var ack = NewAck();
actor.Tell(ack); actor.Tell(ack);
@@ -158,37 +135,6 @@ public class NodeCommunicationActorTests : RuntimeActorTestBase
sender.Sent.Count.ShouldBe(1); sender.Sent.Count.ShouldBe(1);
} }
[Fact]
public void A_sender_wins_over_a_cluster_client_and_the_ack_is_not_sent_twice()
{
// Defence in depth. The registration arm builds exactly one transport — under Grpc no
// ClusterClient is created at all — and if both were ever present, posting over both would
// deliver two acks for one apply. The coordinator would absorb the duplicate (NodeId-keyed
// upsert), so this pins tidiness and bounded work, not seal correctness.
var client = CreateTestProbe();
var sender = new RecordingAckSender();
var actor = Sys.ActorOf(NodeCommunicationActor.Props(client.Ref, sender));
actor.Tell(NewAck());
AwaitAssert(() => sender.Sent.Count.ShouldBe(1), TimeSpan.FromSeconds(5));
client.ExpectNoMsg(TimeSpan.FromMilliseconds(300));
}
[Fact]
public void The_ClusterClient_branch_never_touches_the_sender()
{
// Null-safe by construction: under Dps and ClusterClient the sender is not built at all, so the
// ApplyAck path must not depend on one existing.
var client = CreateTestProbe();
var actor = Sys.ActorOf(NodeCommunicationActor.Props(client.Ref));
var ack = NewAck();
actor.Tell(ack);
client.ExpectMsg<ClusterClient.Send>(TimeSpan.FromSeconds(5)).Message.ShouldBe(ack);
}
[Fact] [Fact]
public void Inbound_commands_are_unaffected_by_the_presence_of_a_sender() public void Inbound_commands_are_unaffected_by_the_presence_of_a_sender()
{ {
@@ -197,7 +143,7 @@ public class NodeCommunicationActorTests : RuntimeActorTestBase
var subscriber = CreateTestProbe(); var subscriber = CreateTestProbe();
Sys.EventStream.Subscribe(subscriber.Ref, typeof(DispatchDeployment)); Sys.EventStream.Subscribe(subscriber.Ref, typeof(DispatchDeployment));
var sender = new RecordingAckSender(); var sender = new RecordingAckSender();
var actor = Sys.ActorOf(NodeCommunicationActor.Props(centralClient: null, ackSender: sender)); var actor = Sys.ActorOf(NodeCommunicationActor.Props(sender));
var dispatch = NewDispatch(); var dispatch = NewDispatch();
actor.Tell(dispatch); actor.Tell(dispatch);
@@ -127,23 +127,22 @@ public sealed class ServiceCollectionExtensionsTests
/// <summary> /// <summary>
/// Under <c>MeshTransport:Mode = Grpc</c> an ApplyAck produced by the driver host reaches the /// Under <c>MeshTransport:Mode = Grpc</c> an ApplyAck produced by the driver host reaches the
/// registered <see cref="IApplyAckSender"/>, and no ClusterClient actor exists /// registered <see cref="IApplyAckSender"/>.
/// (ClusterClient→gRPC migration Task 1B.2).
/// </summary> /// </summary>
/// <remarks> /// <remarks>
/// This is the fact that pins the <c>ackRouter</c> gate in <c>WithOtOpcUaRuntimeActors</c> /// This is the fact that pins the <c>ackRouter</c> gate in <c>WithOtOpcUaRuntimeActors</c>
/// (<c>useClusterClient || useGrpc ? nodeComm : null</c>). Narrowed back to ClusterClient-only /// (<c>useGrpc ? nodeComm : null</c>). Drop the <c>useGrpc</c> arm and that gate still compiles
/// that gate still compiles and every unit test still passes, while every ack on a Grpc node /// and every unit test still passes, while every ack on a Grpc node goes out on the
/// goes out on the <c>deployment-acks</c> DPS topic instead — where, in the split-mesh topology /// <c>deployment-acks</c> DPS topic instead — where, in the split-mesh topology Grpc exists to
/// Grpc exists to serve, central is not subscribed. The whole transport would be dead and /// serve, central is not subscribed. The whole transport would be dead and nothing below this
/// nothing below this level of the stack could tell. /// level of the stack could tell.
/// ///
/// The dispatch names a deployment that does not exist in the in-memory ConfigDb, so the apply /// The dispatch names a deployment that does not exist in the in-memory ConfigDb, so the apply
/// fails and the host emits a Failed ack — deterministic, synchronous, and it exercises the /// fails and the host emits a Failed ack — deterministic, synchronous, and it exercises the
/// real <c>SendAck</c> path rather than poking the router directly. /// real <c>SendAck</c> path rather than poking the router directly.
/// </remarks> /// </remarks>
[Fact] [Fact]
public async Task Under_Grpc_a_driver_host_ack_reaches_the_ack_sender_and_no_ClusterClient_exists() public async Task Under_Grpc_a_driver_host_ack_reaches_the_ack_sender()
{ {
var sender = new RecordingAckSender(); var sender = new RecordingAckSender();
using var host = BuildRuntimeActorHost(services => using var host = BuildRuntimeActorHost(services =>
@@ -160,7 +159,6 @@ public sealed class ServiceCollectionExtensionsTests
await host.StartAsync(); await host.StartAsync();
try try
{ {
var system = host.Services.GetRequiredService<ActorSystem>();
var driverHost = host.Services.GetRequiredService<IRequiredActor<DriverHostActorKey>>(); var driverHost = host.Services.GetRequiredService<IRequiredActor<DriverHostActorKey>>();
driverHost.ActorRef.Tell(new DispatchDeployment( driverHost.ActorRef.Tell(new DispatchDeployment(
@@ -168,18 +166,14 @@ public sealed class ServiceCollectionExtensionsTests
new RevisionHash("rev-that-does-not-exist"), new RevisionHash("rev-that-does-not-exist"),
new CorrelationId(Guid.NewGuid()))); new CorrelationId(Guid.NewGuid())));
// The ack travelled DriverHostActor -> NodeCommunicationActor -> IApplyAckSender. Under a // The ack travelled DriverHostActor -> NodeCommunicationActor -> IApplyAckSender. With the
// ClusterClient-only gate it would have gone to DPS and this never completes. // useGrpc arm dropped from the ackRouter gate it would have gone to DPS instead and this
// never completes.
var deadline = DateTime.UtcNow.AddSeconds(15); var deadline = DateTime.UtcNow.AddSeconds(15);
while (sender.Sent.Count == 0 && DateTime.UtcNow < deadline) while (sender.Sent.Count == 0 && DateTime.UtcNow < deadline)
await Task.Delay(50); await Task.Delay(50);
sender.Sent.Count.ShouldBe(1); sender.Sent.Count.ShouldBe(1);
// And the Akka cross-mesh association is genuinely gone, not merely unused.
await Should.ThrowAsync<ActorNotFoundException>(() => system
.ActorSelection($"/user/{ServiceCollectionExtensions.CentralClusterClientName}")
.ResolveOne(TimeSpan.FromMilliseconds(500)));
} }
finally finally
{ {