dohertj2/lmxopcua
1
0
Fork 0
Code Issues 8 Pull Requests Actions 1 Packages Projects Releases Wiki Activity

Merge branch 'feat/mesh-phase2' — per-cluster mesh Phase 2 (ClusterClient transport)

Browse Source 
Central↔node command transport behind the MeshTransport:Mode dark switch
(Dps default | ClusterClient). Both comm actors registered with the
receptionist in both modes; SendToAll fan-out, DB-sourced contacts, node-local
EventStream re-emit, ApplyAck relay. Live-gated on docker-dev end-to-end.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
2026-07-22 18:42:57 -04:00
parent 7654f24dab 32055c2238
commit 6eaa81cae7
38 changed files with 3917 additions and 46 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.dockerignore
+29
View File
@@ -0,0 +1,29 @@
# Build context exclusions for docker-dev/Dockerfile, whose `COPY . .` takes the whole repo.
#
# Without this file the context also carries every project's bin/obj AND .claude/worktrees — agent
# worktrees that are themselves full checkouts WITH their own build outputs. That reached ~19 GB and
# exhausted the Docker VM's disk mid-COPY, failing the rig build with "no space left on device"
# rather than anything a build log would attribute to the repo.
#
# Nothing excluded here is needed: the build restores and publishes inside the image, and the
# migrator stage runs `dotnet ef` against the same restored source.
# Agent worktrees — full checkouts, build outputs and all.
.claude/
# Local build outputs; the image builds its own.
**/bin/
**/obj/
artifacts/
# VCS + editor state.
.git/
.gitignore
.vs/
.vscode/
.idea/
# Docker's own state, and anything the image would never read.
**/.DS_Store
**/*.user
docker-dev/**/*.log
CLAUDE.md
+40 -1
View File
@@ -228,7 +228,46 @@ The server supports non-transparent warm/hot redundancy via the `Redundancy` sec
Assert the **effective** cluster config off a running `ActorSystem` (as `SplitBrainResolverActivationTests` does), never the typed options or the akka.conf text — several HOCON fragments compete and the losing one still reads correctly in the file.
**Still per-Akka-cluster, not per application `Cluster`.** One Primary is elected across the whole Akka mesh, so a fleet running several application clusters in one mesh (what docker-dev models) gets one Primary among all driver nodes. The fix is the per-cluster mesh workstream: `docs/plans/2026-07-21-per-cluster-mesh-design.md` (Phases 0a/0b/1 done; 2–7 not started).
**Still per-Akka-cluster, not per application `Cluster`.** One Primary is elected across the whole Akka mesh, so a fleet running several application clusters in one mesh (what docker-dev models) gets one Primary among all driver nodes. The fix is the per-cluster mesh workstream: `docs/plans/2026-07-21-per-cluster-mesh-design.md` (Phases 0a/0b/1 done; 2 code-complete behind a dark switch; 3–7 not started).
## Mesh command transport (`MeshTransport`)
Per-cluster mesh **Phase 2** added a second transport for the three central→node command channels
(`deployments`, `driver-control`, `alarm-commands`) and the `deployment-acks` reply, selected by
`MeshTransport:Mode`: `Dps` (the DistributedPubSub path this repo has always used, **the default**)
or `ClusterClient`, which does **not** require central and the node to share a gossip ring — the
prerequisite for splitting the fleet into one mesh per application `Cluster`. Both `CentralCommunicationActor`
(`/user/central-communication`) and `NodeCommunicationActor` (`/user/node-communication`) are spawned
and registered with the `ClusterClientReceptionist` in **both** modes, so flipping the flag is a config
change, not a redeploy. Things worth knowing before touching it:
- **`SendToAll`, never `Send`.** Today's DPS publish reaches every `DriverHostActor`; there is no
ClusterId or node filter on the node side at all (scoping happens later, inside the artifact via
`DeploymentArtifact.ResolveClusterScope`). `ClusterClient.Send` delivers to exactly **one** registered
actor — it would deploy to a single node while every other node silently kept its old configuration,
and the deployment could still seal green.
- **Exactly ONE fleet-wide ClusterClient while the fleet is one mesh.** A receptionist serves its whole
cluster, so `SendToAll` reaches every registered node-comm actor regardless of which node's address was
dialled. One client per application `Cluster` — the obviously-right-looking shape, and what Phase 6
wants — would fan each command out once per cluster (N× duplicate dispatch *and* N× duplicate ack).
Marked `TODO(Phase 6)` in `CentralCommunicationActor.RebuildClient`. **Corollary:** the contact set does
not scope delivery; excluding a `MaintenanceMode` node from the contacts does not stop it receiving.
- **Inbound commands re-emit on the node's `EventStream`, not on their DPS topic** — DPS is mesh-wide and
central `SendToAll`s to every node, so a DPS re-publish would deliver N copies to every subscriber.
Node-side subscribers (`DriverHostActor`, `ScriptedAlarmHostActor`) accept both.
- **Comm actors are per-node, NOT singletons.** A client rotates across contact points and must find a live
comm actor at whichever node answers; `akka.cluster.client.receptionist.role` is empty for the same reason.
- **`buffer-size = 0` is a behaviour decision, not a tuning knob.** Akka's default buffers 1000 and replays
on reconnect — silently, on the client's own timer, at an unbounded remove from the dispatch. Do not raise
it to quiet a flaky test. **But do not over-claim what it buys:** the Phase 2 live gate observed a node
applying a deployment **44 s after it was sealed `TimedOut`** anyway, via `DriverHostActor.Bootstrap()`
replaying the orphan `Applying` row on restart. Zero buffering removes the silent replay, not every replay.
Contact points are node **addresses only** (`akka.tcp://<system>@<host>:<port>`); `/system/receptionist` is
appended at construction and `MeshTransportOptionsValidator` refuses to start a node whose contact carries an
actor-path suffix. `redundancy-state` stays on DPS in both modes (bidirectional, and built from
`Cluster.State`). See `docs/Configuration.md` §`MeshTransport`, `docs/Redundancy.md` §"Command transport",
and `docs/plans/2026-07-22-mesh-phase2-clusterclient-transport.md`.
**Phase 1 landed 2026-07-22 and changed a deploy-path behaviour.** `ConfigPublishCoordinator` sources its expected-ack set from **enabled `ClusterNode` rows**, not from `Akka.Cluster.State.Members` filtered by the `driver` role — central must be able to name a deployment's nodes without sharing a gossip ring with them. Consequences worth knowing before touching deploys:
docker-dev/docker-compose.yml
+78
View File
@@ -190,6 +190,19 @@ services:
Cluster__SeedNodes__1: "akka.tcp://otopcua@central-2:4053"
Cluster__Roles__0: "admin"
Cluster__Roles__1: "driver"
# Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps",
# the transport this repo has always used, and every node keeps its ClusterClient wiring built
# and its comm actor registered with the receptionist either way. Flip the whole rig with
# OTOPCUA_MESH_MODE=ClusterClient at `docker compose up` — no compose edit, and no rebuild.
#
# Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
# time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
# actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
# silence. Both central nodes are listed so a driver node's client can rotate to whichever one
# is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Dps}"
MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
Security__Jwt__SigningKey: "docker-dev-signing-key-with-at-least-32-bytes-of-utf8-content-12345"
Security__Jwt__Issuer: "otopcua-dev"
Security__Jwt__Audience: "otopcua-dev"
@@ -263,6 +276,19 @@ services:
Cluster__SeedNodes__1: "akka.tcp://otopcua@central-1:4053"
Cluster__Roles__0: "admin"
Cluster__Roles__1: "driver"
# Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps",
# the transport this repo has always used, and every node keeps its ClusterClient wiring built
# and its comm actor registered with the receptionist either way. Flip the whole rig with
# OTOPCUA_MESH_MODE=ClusterClient at `docker compose up` — no compose edit, and no rebuild.
#
# Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
# time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
# actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
# silence. Both central nodes are listed so a driver node's client can rotate to whichever one
# is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Dps}"
MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
Security__Jwt__SigningKey: "docker-dev-signing-key-with-at-least-32-bytes-of-utf8-content-12345"
Security__Jwt__Issuer: "otopcua-dev"
Security__Jwt__Audience: "otopcua-dev"
@@ -329,6 +355,19 @@ services:
# is in its own seed list), so these configs are exempt rather than broken.
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
Cluster__Roles__0: "driver"
# Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps",
# the transport this repo has always used, and every node keeps its ClusterClient wiring built
# and its comm actor registered with the receptionist either way. Flip the whole rig with
# OTOPCUA_MESH_MODE=ClusterClient at `docker compose up` — no compose edit, and no rebuild.
#
# Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
# time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
# actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
# silence. Both central nodes are listed so a driver node's client can rotate to whichever one
# is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Dps}"
MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
<<: *secrets-env
# Quiet EF/AspNetCore SQL flood — see central-1 (Serilog override). mem_limit/
# mem_reservation are inherited from the *otopcua-host anchor.
@@ -395,6 +434,19 @@ services:
Cluster__PublicHostname: "site-a-2"
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
Cluster__Roles__0: "driver"
# Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps",
# the transport this repo has always used, and every node keeps its ClusterClient wiring built
# and its comm actor registered with the receptionist either way. Flip the whole rig with
# OTOPCUA_MESH_MODE=ClusterClient at `docker compose up` — no compose edit, and no rebuild.
#
# Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
# time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
# actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
# silence. Both central nodes are listed so a driver node's client can rotate to whichever one
# is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Dps}"
MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
<<: *secrets-env
Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
@@ -453,6 +505,19 @@ services:
Cluster__PublicHostname: "site-b-1"
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
Cluster__Roles__0: "driver"
# Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps",
# the transport this repo has always used, and every node keeps its ClusterClient wiring built
# and its comm actor registered with the receptionist either way. Flip the whole rig with
# OTOPCUA_MESH_MODE=ClusterClient at `docker compose up` — no compose edit, and no rebuild.
#
# Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
# time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
# actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
# silence. Both central nodes are listed so a driver node's client can rotate to whichever one
# is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Dps}"
MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
<<: *secrets-env
Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
@@ -494,6 +559,19 @@ services:
Cluster__PublicHostname: "site-b-2"
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
Cluster__Roles__0: "driver"
# Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps",
# the transport this repo has always used, and every node keeps its ClusterClient wiring built
# and its comm actor registered with the receptionist either way. Flip the whole rig with
# OTOPCUA_MESH_MODE=ClusterClient at `docker compose up` — no compose edit, and no rebuild.
#
# Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
# time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
# actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
# silence. Both central nodes are listed so a driver node's client can rotate to whichever one
# is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Dps}"
MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
<<: *secrets-env
Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
docs/Configuration.md
+23
View File
@@ -110,6 +110,29 @@ The checked-in `appsettings*.json` files are deliberately thin: they carry only
> **`Port` / `PublicHostname` are stored twice.** The node binds from these keys; the fleet's `ClusterNode` row records the same address as `AkkaPort` / `Host` so that **central can dial the node without sharing a gossip ring with it** — which is what [per-cluster mesh Phase 2](plans/2026-07-21-per-cluster-mesh-design.md) needs. Nothing in the schema makes the two agree, so `ClusterNodeAddressReconcilerActor` (admin-role singleton) compares them against live membership and logs an Error on mismatch. If you change `Cluster:Port` or `Cluster:PublicHostname` on a node, **update its `ClusterNode` row too** — a node binding 4054 with a row saying 4053 still gossips fine today, and fails silently in Phase 2. The row's `NodeId` is `host:port` and is also the deploy path's ack identity, so a drifted node's deployment acks stop matching as well. See [`config-db-schema.md` § `ClusterNode`](v2/config-db-schema.md#clusternode).
### `MeshTransport` (central↔node command transport)
- **Purpose:** selects how central's three command channels (`deployments`, `driver-control`, `alarm-commands`) and the node's `deployment-acks` reply reach the other side — over the mesh-wide DistributedPubSub, or over an Akka `ClusterClient` that does **not** require central and the node to share a gossip ring. The second option is what [per-cluster mesh](plans/2026-07-21-per-cluster-mesh-design.md) Phase 6 needs once the meshes split.
- **Options class:** `MeshTransportOptions` (`SectionName = "MeshTransport"`) — `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/MeshTransportOptions.cs`. Bound and validated by `AddOtOpcUaCluster(config)`; `MeshTransportOptionsValidator` runs at `ValidateOnStart`.
| Key | Type | Default | Meaning |
|---|---|---|---|
| `Mode` | string | `Dps` | `Dps` = DistributedPubSub (the transport this repo has always used). `ClusterClient` = the mesh boundary. Any other value **refuses to start**. |
| `CentralContactPoints` | string[] | `[]` | Node-side only: central node **addresses** (`akka.tcp://<system>@<host>:<port>`). Required under `ClusterClient`, ignored under `Dps`. |
| `ContactRefreshSeconds` | int | `60` | Central-side only: how often central rebuilds its ClusterClient contact set from the `ClusterNode` rows. |
**Addresses only — no actor-path suffix.** `/system/receptionist` is appended at construction time; a contact carrying `/user/` or `/system/` is rejected at startup, because a doubled suffix produces a path that resolves to nothing and drops every command in silence.
**It is a dark switch, and both sides are always wired.** The comm actors are spawned and registered with the `ClusterClientReceptionist` on every node in *both* modes, so flipping `Mode` is a config change rather than a redeploy. Under `Dps` the registration is simply idle.
**Central's contact set comes from the enabled, non-maintenance `ClusterNode` rows** — the same source Phase 1 gave the deploy ack set. A node in `MaintenanceMode` is not dialled, but note that **the contact set does not scope delivery**: a receptionist serves its whole cluster, so `SendToAll` still reaches every registered node exactly as today's DPS broadcast does. The filter is about which receptionists are worth dialling, not about who receives.
**Buffering is off by design** (`akka.cluster.client.buffer-size = 0`, set in the shipped `akka.conf`). If no receptionist is known, the command is dropped rather than queued. Akka's default of 1000 would replay a `DispatchDeployment` on reconnect — silently, on the client's own timer, at an unbounded remove from the dispatch. Do not "fix" it back to quiet a flaky test.
> **What this does *not* buy you.** It does not guarantee a node never applies a deployment the DB records as `TimedOut`. A node that missed a dispatch keeps the orphan `Applying` row the coordinator pre-created, and `DriverHostActor.Bootstrap()` replays that row on the node's next restart — live-observed applying a deployment 44 s after it was sealed `TimedOut` (see the [Phase 2 live gate](plans/2026-07-22-mesh-phase2-live-gate.md), finding B). The boot replay is the better-behaved of the two — deliberate, Warning-logged, and bounded to a restart — but it is still a replay. `buffer-size = 0` removes the *silent, timer-driven* one only.
On the docker-dev rig every node carries these keys already; flip the whole rig with `OTOPCUA_MESH_MODE=ClusterClient` at `docker compose up`.
### `ConnectionStrings` → `ConfigDb`
- **Purpose:** the central Config DB connection string. **Required for every role** — `Program.cs` calls `AddOtOpcUaConfigDb` unconditionally.
docs/Redundancy.md
+31
View File
@@ -407,6 +407,37 @@ Net effect: each alarm transition appears **once** on `/alerts` and would histor
See [ScriptedAlarms.md](ScriptedAlarms.md) and [AlarmTracking.md](AlarmTracking.md) for the scripted-alarm engine internals.
## Command transport (central↔node)
Everything above rides one Akka mesh: central publishes on DistributedPubSub, every node
subscribes, and the two ends must be members of the same cluster. Per-cluster mesh **Phase 2**
adds a second transport under `MeshTransport:Mode` (see
[Configuration.md § `MeshTransport`](Configuration.md#meshtransport-centralnode-command-transport))
that does not need shared membership — the prerequisite for splitting the fleet into one mesh per
application `Cluster`.
| | `Dps` (default) | `ClusterClient` |
|---|---|---|
| Central → node commands | `Publish` on `deployments` / `driver-control` / `alarm-commands` | `ClusterClient.SendToAll` to `/user/node-communication`, re-emitted on the node's **local** `EventStream` |
| Node → central acks | `Publish` on `deployment-acks` | `ClusterClient.Send` to `/user/central-communication`, forwarded to the deploy-coordinator singleton |
| Requires shared cluster membership | **yes** | no |
Two properties are load-bearing and easy to break:
- **Both comm actors are registered per node, NOT as cluster singletons.** A `ClusterClient` rotates
across its contact points and must find a live comm actor at whichever node answers. As a
singleton the boundary would be reachable through exactly one node, and rotation would fail
silently against the others — acks landing or not depending on where the client happened to
settle. `akka.cluster.client.receptionist.role` is deliberately empty for the same reason.
- **Inbound commands go to the node-local `EventStream`, never back onto a DPS topic.** DPS is
mesh-wide and central `SendToAll`s to *every* node, so a DPS re-publish would deliver N copies of
every command to every subscriber.
Phase 2 changes no redundancy semantics: the Primary gate, ServiceLevel, and the singleton layout
are untouched, and `redundancy-state` itself stays on DPS (it is bidirectional and built from
`Cluster.State`, which makes it genuinely mesh-bound — the hardest remaining dependency before the
meshes can split).
## Pair-local store (LocalDb — Phases 1 + 2)
Independently of the ServiceLevel machinery above, every **driver-role** node keeps a consolidated
docs/plans/2026-07-21-per-cluster-mesh-design.md
+8 -2
View File
@@ -68,8 +68,14 @@ Other load-bearing details:
caller's Ask times out. *"It keeps the central coordinator stateless with respect to site
availability."* A `ConnectionStateChanged` mechanism was built for this and **deleted as dead
code**.
- **Every unhandled message replies with a typed failure rather than dropping**, so a central Ask
can never stall on a missing handler.
- ~~**Every unhandled message replies with a typed failure rather than dropping**, so a central Ask
can never stall on a missing handler.~~ **CORRECTED 2026-07-22 (Phase 2 recon).** This overstates
what ScadaBridge does. Neither of their comm actors has a `ReceiveAny` or an `Unhandled` override:
the typed-failure idiom is **per message type**, and it fires only when a *registration* is missing
(no ClusterClient for the site yet, no handler registered for that type). A genuinely unknown
message type still dead-letters exactly as it would anywhere else. Do not build a catch-all failure
reply believing you are copying them. OtOpcUa's Phase 2 comm actors list each type explicitly and
deliberately let an unknown type dead-letter loudly rather than republish it blind.
- **Sender preservation is the whole Ask idiom**: `_client.Tell(new ClusterClient.Send(path, msg),
Sender)` routes the reply straight back to the waiting Ask, bypassing the comm actor.
docs/plans/2026-07-22-mesh-phase2-clusterclient-transport.md
+1018
View File
File diff suppressed because it is too large Load Diff
docs/plans/2026-07-22-mesh-phase2-clusterclient-transport.md.tasks.json
+141
View File
@@ -0,0 +1,141 @@
{
"planPath": "docs/plans/2026-07-22-mesh-phase2-clusterclient-transport.md",
"note": "Per-cluster mesh Phase 2. Decisions pre-made: config-flagged dark switch (MeshTransport:Mode), buffer-size=0, alarm-commands central leg in scope. Three design-doc corrections and the single-mesh duplicate-delivery trap are documented in the plan header \u2014 read them before Task 3.",
"tasks": [
{
"id": 0,
"subject": "Task 0: MeshTransportOptions + validator + registration",
"status": "completed",
"classification": "small",
"parallelizableWith": [
1
],
"commit": "5439f148"
},
{
"id": 1,
"subject": "Task 1: receptionist extension, buffer-size=0, frame-size logging (effective-config test)",
"status": "completed",
"classification": "small",
"parallelizableWith": [
0
],
"commit": "7b71a6a3",
"note": "First draft of the buffering test was VACUOUS -- Config.GetInt returns 0 for a missing key, so it passed before the feature existed. Rewritten to assert through ClusterClientSettings/ClusterReceptionistSettings."
},
{
"id": 2,
"subject": "Task 2: MeshCommand envelope + generation-suffixed ClusterClient factory",
"status": "completed",
"classification": "small",
"blockedBy": [
0
],
"commit": "0d6669d5",
"note": "Dropped a third test that asserted ClusterClientSettings directly and never touched the factory."
},
{
"id": 3,
"subject": "Task 3: CentralCommunicationActor \u2014 dark-switched SendToAll fan-out, DB-sourced contacts, ApplyAck forward",
"status": "completed",
"classification": "high-risk",
"blockedBy": [
1,
2
],
"parallelizableWith": [
4
],
"commit": "d5b5cb6e",
"note": "SendToAll sabotage-verified. Two test-harness bugs found: SubscribeAck goes to the SENDER, and EventFilter matches case-INSENSITIVELY (a 'was NOT' filter also caught 'was not created')."
},
{
"id": 4,
"subject": "Task 4: NodeCommunicationActor \u2014 inbound to node-local EventStream, outbound ApplyAck",
"status": "completed",
"classification": "standard",
"blockedBy": [
1,
2
],
"parallelizableWith": [
3
],
"commit": "915beec1",
"note": "No Ask crosses the boundary in Phase 2, so this actor is far smaller than the sister project's."
},
{
"id": 5,
"subject": "Task 5: node subscribers accept EventStream too; rename _coordinatorOverride -> _ackRouter",
"status": "completed",
"classification": "standard",
"blockedBy": [
4
],
"commit": "5cb0e721",
"note": "Two wiring tests added -- unit tests either side of the seam both pass with the subscription deleted. First draft raced (ActorOf returns before PreStart; EventStream publish to nobody is dropped silently); alarm version was vacuous until the warm-up moved outside the assertion window."
},
{
"id": 6,
"subject": "Task 6: wire both sides, receptionist registration, pin the wire-contract actor paths",
"status": "completed",
"classification": "standard",
"blockedBy": [
3,
5
],
"commit": "16d59856",
"note": "Dropped a planned 'not a singleton' assertion: two candidate discriminators were both invalid (a SingletonManager also sits at /user/<name>; /singleton is null even for the known singleton config-publish). A sabotage re-registering via WithSingleton left both green, which exposed them. Property deferred to Task 8."
},
{
"id": 7,
"subject": "Task 7: route the five publish sites through the mesh router (nullable, mediator fallback)",
"status": "completed",
"classification": "standard",
"blockedBy": [
6
],
"note": "Sabotage-verified by inverting the branch: all 6 tests red. Comm actor registration moved above the singletons so registry.Get works; an earlier ResolveOne().GetAwaiter().GetResult() blocked inside a props factory."
},
{
"id": 8,
"subject": "Task 8: real two-ActorSystem ClusterClient boundary test + falsifiability control",
"status": "completed",
"classification": "high-risk",
"blockedBy": [
7
],
"parallelizableWith": [
9
],
"commit": "5468b79d",
"note": "Not flaky (3/3 clean repeats; 4 tests, ~16s). Both legs sabotage-verified against production code, and the control was itself falsified by restoring the receptionist. The control necessarily removes RegisterService too -- registering would materialise the extension it is removing. Task 6's dropped per-node/singleton property does NOT land here (one node per side cannot discriminate); it is live-gate step 8, and the MeshCommActorPathTests comment was corrected to say so."
},
{
"id": 9,
"subject": "Task 9: docker-dev rig config + docs + design-doc correction",
"status": "completed",
"commit": "e4cb9a08",
"note": "Rig Mode is ${OTOPCUA_MESH_MODE:-Dps} rather than a hardcoded Dps, so Task 10 can flip all six nodes at `docker compose up` with no compose edit and no rebuild. Verified via `docker compose config` (6 services resolved).",
"classification": "small",
"blockedBy": [
7
],
"parallelizableWith": [
8
]
},
{
"id": 10,
"subject": "Task 10: live gate on docker-dev (11 steps; step 4 = deploy-seal semantics, step 8 = per-node/contact-rotation discriminator [replaces the dropped Task 6 assertion \u2014 do not skip], step 9 = both-centrals drop behavior)",
"status": "completed",
"note": "PASSED for the transport (steps 1-5, 8, 11 green; flip end-to-end via OTOPCUA_MESH_MODE=ClusterClient). Record: 2026-07-22-mesh-phase2-live-gate.md. Steps 6/7 NOT RUN (no UI surface for driver-control \u2014 finding D, DriverStatusPanel is orphaned; no alarms configured). Step 8's stated log-signal does not exist (HandleApplyAck is silent on success \u2014 finding E); proven instead by both centrals independently logging per-instance ClusterClient creation + rebuild, which a singleton cannot. FOUR findings: A) 10-30s post-restart delivery window is PRE-EXISTING (DPS control failed identically); B) buffer-size=0 rationale OVERSTATED \u2014 a node applies a TimedOut deploy anyway via Bootstrap() orphan-row replay at +44s, corrected in 3 docs; C) stopping BOTH centrals strands non-seed nodes at the membership layer (Phase 7 drill, transport-independent); D) driver Restart/Reconnect UI orphaned since v3 Batch 2. Rig restored to Dps default.",
"classification": "high-risk",
"blockedBy": [
8,
9
]
}
],
"lastUpdated": "2026-07-22T00:00:00Z"
}
docs/plans/2026-07-22-mesh-phase2-live-gate.md
+228
View File
@@ -0,0 +1,228 @@
# Per-cluster mesh Phase 2 — live gate record
**Date:** 2026-07-22 · **Rig:** `docker-dev` (six-node single mesh) · **Branch:** `feat/mesh-phase2`
**Plan:** [`2026-07-22-mesh-phase2-clusterclient-transport.md`](2026-07-22-mesh-phase2-clusterclient-transport.md)
**Result: PASSED for the transport**, with two gate steps not run (no surface exists to drive them),
one step whose stated pass-signal does not exist in the code, and **four findings** — one of which
corrects a claim this phase's own documentation makes.
The rig was flipped end-to-end with `OTOPCUA_MESH_MODE=ClusterClient` (no rebuild, no compose edit —
the env indirection added in Task 9 is what made the A/B and the DPS control cheap enough to run).
---
## Result table
| # | Step | Result |
|---|---|---|
| 1 | Deploy on `Mode=Dps` | **PASS** — `48d0c7c5` Sealed, 6/6 Applied |
| 2 | Flip all six to `ClusterClient`, redeploy | **PASS** — `5db8f6ad` Sealed, 6/6 Applied |
| 3 | Confirm the transport actually changed | **PASS** — positive log evidence, see below |
| 4 | Stop a node, deploy | **PASS** — `59b81248` TimedOut naming `site-b-2:4053`, 5/6 |
| 5 | `MaintenanceMode = 1`, redeploy | **PASS** — `c6cb9812` Sealed, 5 rows, contacts 6→5 |
| 6 | AdminUI Restart + Reconnect | **NOT RUN** — no UI surface exists (finding D) |
| 7 | AdminUI alarm ack | **NOT RUN** — no alarms configured on the rig |
| 8 | Per-node (not singleton) discriminator | **PASS** — via a better discriminator than the plan's |
| 9 | Both centrals down, restart a driver node | **PARTIAL** — node survives; the stated ack-drop cannot occur |
| 10 | Restart central, redeploy | **PASS only after operator action** (finding C) |
| 11 | Frame-size canary | **PASS** — 0 warnings across all six nodes |
## Step 3 — the transport really changed
Central, both nodes, at boot under the flag:
```
central-1 [22:18:16 INF] Mesh ClusterClient created with 6 contact point(s)
central-2 [22:18:17 INF] Mesh ClusterClient created with 6 contact point(s)
```
Node side, with `Serilog__MinimumLevel__Default=Debug` overlaid on `site-a-1` (the rig pins the
default above Debug, so this line is invisible on a stock node):
```
site-a-1 [22:10:34 DBG] Mesh: received DispatchDeployment from central; publishing node-locally
```
Under `Mode=Dps` the same node logs nothing of the sort and central logs
`Mesh transport is Dps: commands are published on DistributedPubSub and no ClusterClient is created`.
## Step 5 — the DB-sourced contact set is live
Setting `MaintenanceMode = 1` on `site-b-2` shrank both centrals' contact sets on the next refresh,
and clearing it restored them — independently, on each central's own 60 s timer:
```
central-1 [22:22:16 INF] Mesh contact set changed; replacing the ClusterClient
central-1 [22:22:16 INF] Mesh ClusterClient created with 5 contact point(s)
central-1 [22:23:16 INF] Mesh ClusterClient created with 6 contact point(s)
central-2 [22:22:16 INF] … identical, same timestamps
```
No `InvalidActorNameException` on any rebuild — the generation-suffixed client name (Task 2, ported
from the sister project's shipped bug) does its job.
## Step 8 — the per-node discriminator, replaced with a stronger one
**The plan's pass-signal does not exist.** It asked for the surviving central's comm actor to *log
receipt* of a node ack inside the downing window. `CentralCommunicationActor.HandleApplyAck` logs
**nothing** on the success path — it only warns when the coordinator is unresolvable. There is no
line to observe. (Recorded as finding E.)
The evidence above is a **cleaner** discriminator and needs no timing race. `Mesh ClusterClient
created` is emitted from `PreStart`, so it fires **once per live actor instance**. Both centrals
logged it, and both logged their own independent contact-set rebuild at step 5. A
`ClusterSingletonManager` runs the actor on the oldest node only — the other node hosts a manager
that never starts the actor and would log nothing. **Two nodes logging it proves two live instances,
i.e. per-node registration.** The node side is proven the same way: `SendToAll` collected 6/6 acks,
which requires six separately registered node comm actors.
The failover half was run anyway, and passed: `docker stop central-1` (the oldest, and the singleton
host) → the surviving central accepted a deployment **4 seconds later** and collected acks from all
five live nodes (`e3161c25`; it TimedOut only because `central-1` is itself a configured node that
was down — correct Phase 1 semantics). Graceful stop hands the singleton over immediately, so no
downing wait was involved.
## Step 9 — the stated failure mode cannot occur
The plan expected a restarting driver node to emit a boot-time ack that gets dropped with a Warning.
**Driver nodes do not ack on boot.** `DriverHostActor.SendAck` is called only from the apply paths —
a node with no dispatch to answer has nothing to send, so there is no drop to observe.
What was verified: with **both** centrals stopped, `site-a-1` restarted, came up clean, logged its
association failures against both central addresses, retried, and stayed up — no crash loop. The
node-side ClusterClient's forever-retry (`reconnect-timeout = off`) behaves as configured.
## Step 11 — the canary stayed quiet
`grep -ci "frame size|frame-size|maximum-frame"` = **0** on all six nodes, as designed: the deploy
notify carries only `DeploymentId` + `RevisionHash` + `CorrelationId`.
---
## Finding A — a 10–30 s post-restart delivery window, **pre-existing, not Phase 2**
A deploy issued shortly after a driver node restarts does not reach that node, and the deployment
fails at the 2-minute deadline naming it. Measured against `site-a-1`:
| Delay after the node logs readiness | Result |
|---|---|
| ~4 s | **missed** (`f738e1de` TimedOut, node row `Applying`) |
| 10 s | **missed** (`0c7a6296` TimedOut) |
| 30 s | applied, sealed |
**The DPS control fails identically.** Flipping the rig back to `Mode=Dps` and repeating the 10 s
case produced `e85f53c5` → TimedOut, `site-a-1` unacked. So the window is a property of cluster
membership convergence, not of the transport this phase introduces. Recorded rather than fixed.
> Running that control is the only reason this is a note and not a Phase 2 regression. The
> temptation to skip it — "obviously the new transport broke it" — would have produced a false
> attribution and a wasted investigation.
## Finding B — **`buffer-size = 0` does not deliver what this phase's docs claim it does**
The stated rationale, in the plan, `CLAUDE.md`, and `docs/Configuration.md`, is that buffering
*"would replay a `DispatchDeployment` on reconnect and apply a deployment the coordinator already
sealed as `TimedOut`, leaving the node running a configuration the database records as failed."*
**That outcome happens anyway, by a different route.** Deployment `0c7a6296` timed out at 22:14:30
with `site-a-1` unacked. The node's row:
```
NodeId Status StartedAtUtc AppliedAtUtc
site-a-1:4053 1 2026-07-22 22:12:30.145 2026-07-22 22:15:14.823
```
It applied **44 s after the deployment was sealed TimedOut**, on its next restart. The mechanism is
`DriverHostActor.Bootstrap()`, which reads the node's most recent `NodeDeploymentState`, finds the
orphan `Applying` row the coordinator pre-created at dispatch, and replays it:
```csharp
case NodeDeploymentStatus.Applying:
_log.Warning("DriverHost {Node}: found orphan Applying row for deployment {Id}; replaying", …);
ApplyAndAck(new DeploymentId(latest.DeploymentId), revision.Value, CorrelationId.NewId());
```
So `buffer-size = 0` removes the **transport-level, timer-driven** replay — a real difference: it is
silent, unbounded in time, and can fire without any node event. The **boot-time DB replay remains**,
and it is the better-behaved of the two (deliberate, Warning-logged, and bounded to a restart). The
decision stands; the *justification* was overstated and the docs are corrected in the same change as
this record.
## Finding C — stopping **both** centrals strands the non-seed nodes permanently
After `docker stop` of both centrals followed by `docker start`, the fleet did **not** recover.
Deployments reached 2/6 then 3/6 nodes. The cause is not the transport — the Akka cluster itself had
split:
```
Cluster hosts: MEMBERS 3 UP 3 UNREACHABLE 0
central-1:4053 UP admin,driver leader for admin+driver
central-2:4053 UP admin,driver
site-a-1:4053 UP driver
```
The three site nodes that never restarted were **not members at all**, and their own gossip showed
the stale inverse view — `central-2` as `Exiting`, `site-a-1` as `Leaving` — frozen since 22:29:
```
Gossip(members = [central-2 status = Exiting, site-a-1 status = Leaving,
site-a-2 Up, site-b-1 Up, site-b-2 Up])
```
A graceful stop puts the centrals into `Leaving`/`Exiting`; the processes died before the removal
handshake completed, leaving the survivors holding members that will never confirm. `auto-down`
does not help — these members are not *Unreachable*, they are stuck mid-`Exiting`. The restarted
centrals formed a fresh cluster (with `site-a-1`, which had also restarted and re-joined through its
seed), and the three stranded nodes had no path back: they believe they are already in a cluster, so
they never re-run the join process.
**Recovery required restarting the stranded nodes.** After that, the next deploy sealed 6/6
(`e20836ac`).
This is transport-independent — membership is below both DPS and ClusterClient, and the set of nodes
that acked matched the cluster view exactly. It is a **downing/rejoin gap of the current single-mesh
topology**, in the same family as the registered outage gap the design doc already records, and it
belongs to the Phase 7 drill work. Under the target per-pair topology (every node a seed of its own
pair) the shape changes, and this drill must be re-run there rather than assumed fixed.
## Finding D — the AdminUI's driver Restart/Reconnect buttons have no page
Step 6 could not be run, and not for a rig reason.
`Components/Shared/Drivers/DriverStatusPanel.razor` is the **only** component in the codebase
carrying the `DriverOperator`-gated Reconnect/Restart buttons, and **nothing renders it** — `grep -rn
"<DriverStatusPanel"` across `src/` returns nothing, and no `@page` route under `Components/Pages/`
hosts it. It was orphaned when v3 Batch 2 retired the routed `/clusters/{id}/drivers` flow.
So the `driver-control` channel has no operator surface **on either transport** — this is a
pre-existing v3 regression that Phase 2 merely surfaced. Filed as a follow-up; not fixed here.
**Residual risk for the channel:** central-side dispatch is sabotage-verified by
`MeshRouterDispatchTests` (all four `AdminOperationsActor` call sites go through one `Dispatch`
helper); node-side `EventStream` receipt by `DriverHostActor` is proven **live** by this gate, since
`DispatchDeployment` arrives through the identical `PublishLocally` path. What is unproven live is
only the composition for those two message types.
Step 7 (alarm ack) was likewise not run: the rig has no scripted alarms configured
(`ScriptedAlarmHost: applying (enabled=0/0, depRefs=0)`) and `/alerts` is empty, so there is nothing
to acknowledge. Authoring a scripted alarm end-to-end was judged out of proportion to the residual
risk above. `ScriptedAlarmHostActor`'s `AlarmCommand` EventStream subscription therefore rests on its
(sabotage-verified) unit test.
## Finding E — the central comm actor is silent on the success path
`HandleApplyAck` logs only on failure. During this gate that made "did the ack reach the surviving
central?" unanswerable from logs, which is exactly the question an operator asks during a failover.
A Debug line on ack receipt would have made step 8 directly observable. Not added during the gate —
adding code to make a gate step pass is the wrong order — but recorded as a follow-up.
---
## What this gate did not cover
- **Steps 6 and 7** — see findings D and E; no surface exists for driver-control, no alarms for ack.
- **The `alarm-commands` node-side leg** under a real transport, for the reason above.
- **Split meshes.** Everything here ran on the single mesh. The contact set was fleet-wide (6 points,
one client), which is the correct and only shape until Phase 6 splits the meshes.
- **A network partition** (as opposed to process stops). `auto-down`'s dual-active trade is untested
here and remains Phase 7's.
docs/plans/2026-07-22-per-cluster-mesh-program.md
+32 -4
View File
@@ -99,7 +99,35 @@ this removes the coordinator's one genuinely mesh-bound dependency).
set proven DB-sourced (test: a `ClusterNode` row present but node down → deploy reports that node
missing; a node up but row absent → its ack is not expected).
### Phase 2 — Comm actors + ClusterClient transport
### Phase 2 — Comm actors + ClusterClient transport — **DONE 2026-07-22 (live gate PASSED)**
Shipped per [`2026-07-22-mesh-phase2-clusterclient-transport.md`](2026-07-22-mesh-phase2-clusterclient-transport.md)
as a **dark switch**: `MeshTransport:Mode` defaults to `Dps`, so nothing changes on any deployment
until the flag flips. The live gate ([`2026-07-22-mesh-phase2-live-gate.md`](2026-07-22-mesh-phase2-live-gate.md))
flipped the whole rig to `ClusterClient` and passed for the transport, with four findings — most
notably that `buffer-size = 0`'s stated rationale was overstated (a node applies a `TimedOut`
deployment anyway via boot-time orphan-row replay) and that stopping both centrals strands non-seed
nodes at the membership layer (a Phase 7 drill item, transport-independent). Both comm actors are spawned and receptionist-registered in **both** modes, so
the flip is a config change rather than a redeploy. Three scope corrections landed against the
design doc (that plan's header carries them in full):
- the node path is `/user/node-communication`, not `/user/cluster-communication` (there is no
per-cluster scoping until Phase 6);
- the substitution for `Publish` is **`SendToAll`**, not `Send` — `Send` reaches exactly one node and
would deploy to one member of the fleet while the rest silently kept their old config;
- central creates **exactly one fleet-wide ClusterClient**. On a single mesh a receptionist serves
its whole cluster, so a client per application `Cluster` would fan every command out once per
cluster. Per-cluster clients become correct only when Phase 6 splits the meshes (`TODO(Phase 6)`
in `CentralCommunicationActor.RebuildClient`).
**Exit-gate deviation — "an Ask timing out cleanly against a stopped node" cannot be run as written.**
There is no cross-boundary Ask in Phase 2: every migrated command is fire-and-forget on the wire.
`DispatchDeployment` is a `Tell` whose ack returns later as an independent `ApplyAck`;
`RestartDriver` / `ReconnectDriver` / `AlarmCommand` reply `Ok = true` from the admin node the moment
they are dispatched (`Ok` has always meant *dispatched*, not *applied* — the AdminUI string is
literally "Restart dispatched"). The honest equivalents, both in the live gate: a **stopped node**
fails the deploy at the apply deadline naming it (Phase 1 semantics), and **no reachable contact**
drops the command with a Warning and never buffers it.
**Scope:** one receptionist-registered actor per side (`/user/central-communication`,
`/user/cluster-communication`), registered **per node, not as a singleton** (contact rotation);
ClusterClient central → cluster carrying deploy notify + acks + driver-control; the ScadaBridge
@@ -198,9 +226,9 @@ resource sizing on the site VMs should be checked once both products run the ful
|---|---|
| 0a downing strategy | DONE 2026-07-21 (live gate → Phase 7) |
| 0b oldest-Up election | DONE 2026-07-21 |
| Prereq: selfform-fallback + manual-failover plan | plan written 2026-07-22, not executed |
| 1 ClusterNode columns + DB ack set | **detailed plan ALREADY WRITTEN** (`2026-07-21-per-cluster-mesh-phase1.md`, from the design session — do NOT write a new one), not executed. ⚠ Carries a decision gate: DB-derived expected-ack set changes deploy-seal semantics (a stopped-but-enabled node now FAILS the deploy at the apply deadline instead of sealing green without it; escape hatch = `ClusterNode.Enabled=0`) — confirm with the owner before its Task 3. |
| 2 ClusterClient transport | not started |
| Prereq: selfform-fallback + manual-failover plan | **DONE 2026-07-22**, merged `a78425ea`. Shipped *not* as the planned `SelfFormAfter` watchdog — the live gate caught that islanding a manually-failed-over node — but as self-first seed ordering + `AkkaClusterOptionsValidator`. Phase 6's seed-ordering scope is therefore already discharged. |
| 1 ClusterNode columns + DB ack set | **DONE 2026-07-22**, merged `7654f24d`. Deploy-seal semantics changed as flagged, but the escape hatch is **`ClusterNode.MaintenanceMode`**, not `Enabled=0` — the live gate found `Enabled=0` is rejected by `DraftValidator.ValidateClusterTopology` on any 2-node pair. Gate record: `2026-07-22-mesh-phase1-live-gate.md`. |
| 2 ClusterClient transport | **DONE 2026-07-22** (live gate PASSED, shipped dark) — shipped dark (`MeshTransport:Mode=Dps` default), both comm actors registered in both modes. Corrections: `SendToAll` not `Send`, one fleet-wide client not one per cluster, `/user/node-communication` not `/user/cluster-communication`. The gate's "Ask timing out" item does not exist to test — no cross-boundary Ask in this phase; see the phase section. |
| 3 fetch-and-cache | not started |
| 4 cut driver ConfigDb | not started |
| 5 gRPC telemetry | not started |
docs/plans/2026-07-22-per-cluster-mesh-program.md.tasks.json
+3 -3
View File
@@ -2,9 +2,9 @@
"planPath": "docs/plans/2026-07-22-per-cluster-mesh-program.md",
"note": "PROGRAM plan: each task = write that phase's detailed plan (writing-plans), execute it (executing-plans), run its exit gate, update the tracking tables. Prereq task 0 is the separate selfform-fallback plan in this repo.",
"tasks": [
{"id": 0, "subject": "Prereq: execute 2026-07-22-selfform-fallback-and-manual-failover.md (7 tasks)", "status": "pending"},
{"id": 1, "subject": "Phase 1: ClusterNode Akka+gRPC address columns; ConfigPublishCoordinator ack set from DB", "status": "pending"},
{"id": 2, "subject": "Phase 2: comm actors + receptionist + ClusterClient transport (deploy notify/acks, driver-control)", "status": "pending", "blockedBy": [1]},
{"id": 0, "subject": "Prereq: execute 2026-07-22-selfform-fallback-and-manual-failover.md (7 tasks)", "status": "completed", "note": "Shipped as self-first seed ordering + AkkaClusterOptionsValidator, NOT the planned SelfFormAfter watchdog (live gate caught it islanding a failed-over node). Merged a78425ea."},
{"id": 1, "subject": "Phase 1: ClusterNode Akka+gRPC address columns; ConfigPublishCoordinator ack set from DB", "status": "completed", "note": "Merged 7654f24d. Escape hatch is ClusterNode.MaintenanceMode, not Enabled=0."},
{"id": 2, "subject": "Phase 2: comm actors + receptionist + ClusterClient transport (deploy notify/acks, driver-control)", "status": "in_progress", "blockedBy": [1]},
{"id": 3, "subject": "Phase 3: config fetch-and-cache from central; LocalDb steady-state config store (live gate)", "status": "pending", "blockedBy": [2]},
{"id": 4, "subject": "Phase 4: cut driver-side ConfigDb — EfAlarmConditionStateStore to LocalDb, DbHealthProbeActor ServiceLevel input, OpcUaPublishActor audit (live gate)", "status": "pending", "blockedBy": [3]},
{"id": 5, "subject": "Phase 5: gRPC oneof stream contract; migrate 7 observability topics; AdminUI reconnect story", "status": "pending", "blockedBy": [2]},
src/Core/ZB.MOM.WW.OtOpcUa.Cluster/MeshTransportOptions.cs
+61
View File
@@ -0,0 +1,61 @@
namespace ZB.MOM.WW.OtOpcUa.Cluster;
/// <summary>
/// Selects and configures the transport carrying central→node commands and node→central
/// deployment acks (per-cluster mesh Phase 2).
/// </summary>
/// <remarks>
/// <para>
/// <see cref="Mode"/> is a deliberate <b>dark switch</b>, not a permanent knob: both
/// transports live in the tree for one phase so the docker-dev rig can run the same
/// deployment over each and compare, and so a bad Phase 2 rolls back with a config change
/// rather than a revert. Phase 6 deletes the DistributedPubSub branch along with the single
/// mesh that made it viable.
/// </para>
/// <para>
/// <b>Discovery is deliberately asymmetric</b>, matching the sister project. Central
/// discovers nodes <i>from the database</i> — enabled, non-maintenance <c>ClusterNode</c>
/// rows, rebuilt every <see cref="ContactRefreshSeconds"/>. Nodes discover central <i>from
/// this section</i>: static, restart to change. The asymmetry follows the rate of change —
/// operators add and retire nodes, but central's own address is part of the deployment.
/// </para>
/// </remarks>
public sealed class MeshTransportOptions
{
/// <summary>Configuration section name.</summary>
public const string SectionName = "MeshTransport";
/// <summary>Every command stays on DistributedPubSub — the pre-Phase-2 behaviour.</summary>
public const string ModeDps = "Dps";
/// <summary>Commands cross the ClusterClient receptionist boundary.</summary>
public const string ModeClusterClient = "ClusterClient";
/// <summary>
/// <see cref="ModeDps"/> (default) or <see cref="ModeClusterClient"/>. Any other value fails
/// the host at startup rather than silently falling back — a transport that is neither of
/// these delivers nothing, and the symptom would be commands that vanish.
/// </summary>
public string Mode { get; set; } = ModeDps;
/// <summary>
/// Akka remoting addresses of the central nodes, e.g.
/// <c>akka.tcp://otopcua@central-1:4053</c>. <b>Node addresses only</b> — the
/// <c>/system/receptionist</c> path is appended at construction time.
/// </summary>
/// <remarks>
/// Each entry must be a <i>central</i> node's remoting endpoint, never this node's own port.
/// The sister project shipped a site template whose second contact was the site's own
/// remoting port; it is a permanent failure in the initial-contact rotation and stays
/// invisible until a command goes missing. <see cref="MeshTransportOptionsValidator"/>
/// rejects the malformed shapes at boot instead.
/// </remarks>
public string[] CentralContactPoints { get; set; } = [];
/// <summary>
/// How often central re-reads <c>ClusterNode</c> rows to rebuild its contact points.
/// Refreshing on a timer (rather than only on admin change) means a row edited by any
/// process — migration, direct SQL, a second admin node — is picked up without a restart.
/// </summary>
public int ContactRefreshSeconds { get; set; } = 60;
}
src/Core/ZB.MOM.WW.OtOpcUa.Cluster/MeshTransportOptionsValidator.cs
+83
View File
@@ -0,0 +1,83 @@
using Microsoft.Extensions.Options;
namespace ZB.MOM.WW.OtOpcUa.Cluster;
/// <summary>
/// Fails the host at startup on a <see cref="MeshTransportOptions"/> shape that would leave
/// commands silently undelivered.
/// </summary>
/// <remarks>
/// Every fault caught here otherwise surfaces as an <i>absence</i> — a deployment that never
/// arrives, an alarm acknowledgement that does nothing, a node whose ack is dropped without a
/// trace. An absence has no stack trace and no failing node to point at, which makes it the
/// most expensive class of misconfiguration to diagnose in the field. Refusing to start is
/// cheaper for everyone.
/// </remarks>
public sealed class MeshTransportOptionsValidator : IValidateOptions<MeshTransportOptions>
{
/// <inheritdoc />
public ValidateOptionsResult Validate(string? name, MeshTransportOptions options)
{
ArgumentNullException.ThrowIfNull(options);
var errors = new List<string>();
var isDps = string.Equals(options.Mode, MeshTransportOptions.ModeDps, StringComparison.OrdinalIgnoreCase);
var isClusterClient = string.Equals(
options.Mode, MeshTransportOptions.ModeClusterClient, StringComparison.OrdinalIgnoreCase);
if (!isDps && !isClusterClient)
{
errors.Add(
$"{MeshTransportOptions.SectionName}:{nameof(MeshTransportOptions.Mode)} is "
+ $"'{options.Mode}'. Expected '{MeshTransportOptions.ModeDps}' or "
+ $"'{MeshTransportOptions.ModeClusterClient}'.");
}
if (options.ContactRefreshSeconds <= 0)
{
errors.Add(
$"{MeshTransportOptions.SectionName}:{nameof(MeshTransportOptions.ContactRefreshSeconds)} "
+ $"must be greater than 0 (was {options.ContactRefreshSeconds}). Central rebuilds its "
+ "ClusterClient contact points on this interval; a non-positive value would leave the "
+ "contact set frozen at whatever it held when the node booted.");
}
// Contact points are only load-bearing under ClusterClient mode. Requiring them under the
// default would make the section mandatory everywhere for a feature that is switched off —
// including on admin-only nodes, which have no reason to carry them at all.
if (isClusterClient)
{
if (options.CentralContactPoints.Length == 0)
{
errors.Add(
$"{MeshTransportOptions.SectionName}:{nameof(MeshTransportOptions.CentralContactPoints)} "
+ "is empty. Under ClusterClient mode a driver node cannot reach central without at "
+ "least one contact point, and its deployment acks would be dropped silently — the "
+ "deployment would then time out naming a node that is running perfectly.");
}
foreach (var contactPoint in options.CentralContactPoints)
{
if (!contactPoint.StartsWith("akka.tcp://", StringComparison.Ordinal))
{
errors.Add(
$"Contact point '{contactPoint}' must start with 'akka.tcp://' — it is an Akka "
+ "remoting address (akka.tcp://<system>@<host>:<port>), not a bare host:port.");
}
else if (contactPoint.Contains("/user/", StringComparison.Ordinal)
|| contactPoint.Contains("/system/", StringComparison.Ordinal))
{
errors.Add(
$"Contact point '{contactPoint}' carries an actor-path suffix. Configure the "
+ "node address only — '/system/receptionist' is appended at construction time, "
+ "and a doubled suffix produces a path that never resolves.");
}
}
}
return errors.Count == 0
? ValidateOptionsResult.Success
: ValidateOptionsResult.Fail(string.Join(" ", errors));
}
}
src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf
+45
View File
@@ -15,6 +15,12 @@ akka {
extensions = [
"Akka.Cluster.Tools.PublishSubscribe.DistributedPubSubExtensionProvider, Akka.Cluster.Tools"
# Per-cluster mesh Phase 2 — the central<->node command boundary. NOT auto-started: without
# this entry the receptionist materialises only if some code path happens to call
# ClusterClientReceptionist.Get(system) first, which makes "is the boundary listening?"
# depend on startup ordering rather than on configuration.
"Akka.Cluster.Tools.Client.ClusterClientReceptionistExtensionProvider, Akka.Cluster.Tools"
]
actor {
@@ -25,6 +31,20 @@ akka {
dot-netty.tcp {
hostname = "0.0.0.0"
port = 4053
# FRAME SIZE. maximum-frame-size restates Akka's own default; what actually changes is
# log-frame-size-exceeding, which defaults to `off`. Over the limit the transport drops
# that ONE message without tearing down the association — heartbeats keep flowing, the
# node still reports healthy, and the only symptom is a command that never arrived. The
# sister project lost a deploy path to exactly this and could not attribute it; their
# postmortem also notes the default JSON serializer escapes an already-serialized
# payload a second time, so ~60-70 KB of real JSON is enough to blow 128 KB.
#
# OtOpcUa's deploy notify is payload-free by design (DispatchDeployment carries only
# DeploymentId + RevisionHash + CorrelationId), so this threshold should stay silent.
# It is a canary, and a quiet one is the point.
maximum-frame-size = 128000b
log-frame-size-exceeding = 32000b
}
transport-failure-detector {
heartbeat-interval = 2s
@@ -74,6 +94,31 @@ akka {
down-removal-margin = 15s
run-coordinated-shutdown-when-down = on
# CLUSTERCLIENT — the central<->node command boundary (per-cluster mesh Phase 2).
client {
# DELIBERATE, and NOT what the sister project runs: they never wrote this section, so
# they inherit buffer-size = 1000. Zero means "if the receptionist is unknown, drop the
# message now". Buffering would replay a DispatchDeployment on reconnect and apply a
# deployment ConfigPublishCoordinator had already sealed as TimedOut — the node would
# then be running a configuration the database says failed. Fail-fast keeps the DB the
# single record of truth, and reproduces today's DPS behaviour exactly: a node that is
# down misses the publish and self-heals from the DB on restart.
buffer-size = 0
# Retry forever rather than self-stopping the client. Restates Akka's default; stated
# explicitly because the behaviour is load-bearing — a central node down for an hour
# must not require a driver-node restart to become reachable again.
reconnect-timeout = off
receptionist {
# Empty = every member hosts a receptionist. This is what makes the comm actors
# "registered per node, not as a singleton" work: contact rotation reaches whichever
# node answers. Restates Akka's default; do NOT scope this to a role, or the
# boundary becomes reachable through one node only and rotation silently fails.
role = ""
}
}
singleton {
singleton-name = "singleton"
}
src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs
+6
View File
@@ -34,6 +34,12 @@ public static class ServiceCollectionExtensions
services.AddValidatedOptions<AkkaClusterOptions, AkkaClusterOptionsValidator>(
configuration, AkkaClusterOptions.SectionName);
// Per-cluster mesh Phase 2: which transport carries central→node commands. Validated at
// startup for the same reason the cluster options are — a contact point that cannot resolve
// produces silence, not an error, and silence is indistinguishable from "nothing to do".
services.AddValidatedOptions<MeshTransportOptions, MeshTransportOptionsValidator>(
configuration, MeshTransportOptions.SectionName);
services.AddSingleton<IClusterRoleInfo, ClusterRoleInfo>();
return services;
src/Core/ZB.MOM.WW.OtOpcUa.Commons/Messages/Mesh/MeshCommand.cs
+28
View File
@@ -0,0 +1,28 @@
namespace ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
/// <summary>
/// A central→node command, plus the DistributedPubSub topic it would have been published on.
/// </summary>
/// <param name="Topic">
/// The legacy DPS topic. Carried so the router can publish under
/// <c>MeshTransport:Mode = Dps</c> without a second dispatch table; ignored entirely under
/// <c>ClusterClient</c> mode, where the destination is an actor path rather than a topic.
/// </param>
/// <param name="Message">
/// The command itself, never wrapped. Node-side handlers see exactly the type they see today,
/// which is what lets the dark switch touch publishers only.
/// </param>
/// <remarks>
/// <para>
/// This envelope exists so the admin singletons stop knowing which transport is in force.
/// Before per-cluster mesh Phase 2 each publisher held its own
/// <c>DistributedPubSub.Get(Context.System).Mediator</c> call, so the switch would have had
/// to be threaded through five call sites across two actors, and each would have been free
/// to drift.
/// </para>
/// <para>
/// Central-side only — it never crosses the wire, so it carries no serialization contract
/// and needs no additive-evolution discipline.
/// </para>
/// </remarks>
public sealed record MeshCommand(string Topic, object Message);
src/Core/ZB.MOM.WW.OtOpcUa.Commons/Messages/Mesh/MeshPaths.cs
+33
View File
@@ -0,0 +1,33 @@
namespace ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
/// <summary>
/// Actor paths the two mesh comm actors register under. <b>These strings are the wire
/// contract</b> of the central↔node boundary.
/// </summary>
/// <remarks>
/// <para>
/// Declared once, here, rather than as literals on each side. The <c>deployments</c> /
/// <c>deployment-acks</c> topic names are declared twice — once in
/// <c>ConfigPublishCoordinator</c> and again in <c>DriverHostActor</c> — and agree only by
/// convention; renaming one would compile, deploy, and deliver nothing. A ClusterClient path
/// has the same failure mode and no compiler to catch it, so the constant is shared.
/// </para>
/// <para>
/// Nothing else pins these values, which is why <c>MeshCommActorPathTests</c> boots a real
/// host and <c>Identify</c>-probes both paths.
/// </para>
/// </remarks>
public static class MeshPaths
{
/// <summary>Central-side comm actor — where a driver node sends its deployment acks.</summary>
public const string CentralCommunication = "/user/central-communication";
/// <summary>Node-side comm actor — where central sends commands.</summary>
public const string NodeCommunication = "/user/node-communication";
/// <summary>Actor name of <see cref="CentralCommunication"/> (the path's final segment).</summary>
public const string CentralCommunicationName = "central-communication";
/// <summary>Actor name of <see cref="NodeCommunication"/> (the path's final segment).</summary>
public const string NodeCommunicationName = "node-communication";
}
src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/AdminOperations/AdminOperationsActor.cs
+41 -7
View File
@@ -4,6 +4,7 @@ using Akka.Event;
using Microsoft.EntityFrameworkCore;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Admin;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Deploy;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
using ZB.MOM.WW.OtOpcUa.Commons.Types;
using ZB.MOM.WW.OtOpcUa.Configuration;
@@ -26,6 +27,13 @@ public sealed class AdminOperationsActor : ReceiveActor
private readonly IActorRef _coordinator;
private readonly IReadOnlyDictionary<string, IDriverProbe> _probesByType;
private readonly TagConfigValidationMode _tagConfigValidationMode;
/// <summary>
/// Where central→node commands are handed off (per-cluster mesh Phase 2), or
/// <see langword="null"/> to publish on the mediator directly.
/// </summary>
private readonly IActorRef? _meshRouter;
private readonly ILoggingAdapter _log = Context.GetLogger();
/// <summary>Creates actor props for the admin operations actor.</summary>
@@ -33,29 +41,38 @@ public sealed class AdminOperationsActor : ReceiveActor
/// <param name="coordinator">Reference to the deployment coordinator actor.</param>
/// <param name="probes">Driver probes registered in DI; keyed by DriverType (case-insensitive).</param>
/// <param name="tagConfigValidationMode">Deploy-gate TagConfig strictness mode (Warn default | Error opt-in).</param>
/// <param name="meshRouter">
/// Central comm actor commands are handed to, or <see langword="null"/> to publish on the
/// DistributedPubSub mediator directly (the pre-Phase-2 behaviour).
/// </param>
/// <returns>Props configured to create an AdminOperationsActor.</returns>
public static Props Props(
IDbContextFactory<OtOpcUaConfigDbContext> dbFactory,
IActorRef coordinator,
IEnumerable<IDriverProbe> probes,
TagConfigValidationMode tagConfigValidationMode = TagConfigValidationMode.Warn) =>
Akka.Actor.Props.Create(() => new AdminOperationsActor(dbFactory, coordinator, probes, tagConfigValidationMode));
TagConfigValidationMode tagConfigValidationMode = TagConfigValidationMode.Warn,
IActorRef? meshRouter = null) =>
Akka.Actor.Props.Create(() =>
new AdminOperationsActor(dbFactory, coordinator, probes, tagConfigValidationMode, meshRouter));
/// <summary>Initializes a new instance of the AdminOperationsActor.</summary>
/// <param name="dbFactory">Factory for creating config database contexts.</param>
/// <param name="coordinator">Reference to the deployment coordinator actor.</param>
/// <param name="probes">Driver probes registered in DI; keyed by DriverType (case-insensitive).</param>
/// <param name="tagConfigValidationMode">Deploy-gate TagConfig strictness mode (Warn default | Error opt-in).</param>
/// <param name="meshRouter">Central comm actor, or <see langword="null"/> for the mediator.</param>
public AdminOperationsActor(
IDbContextFactory<OtOpcUaConfigDbContext> dbFactory,
IActorRef coordinator,
IEnumerable<IDriverProbe> probes,
TagConfigValidationMode tagConfigValidationMode = TagConfigValidationMode.Warn)
TagConfigValidationMode tagConfigValidationMode = TagConfigValidationMode.Warn,
IActorRef? meshRouter = null)
{
_dbFactory = dbFactory;
_coordinator = coordinator;
_probesByType = probes.ToDictionary(p => p.DriverType, StringComparer.OrdinalIgnoreCase);
_tagConfigValidationMode = tagConfigValidationMode;
_meshRouter = meshRouter;
ReceiveAsync<StartDeployment>(HandleStartDeploymentAsync);
ReceiveAsync<TestDriverConnect>(HandleTestDriverConnectAsync);
@@ -65,6 +82,23 @@ public sealed class AdminOperationsActor : ReceiveActor
Receive<ShelveAlarmCommand>(HandleShelveAlarm);
}
/// <summary>
/// Hands a central→node command to the mesh transport router, or publishes it directly when
/// no router is wired.
/// </summary>
/// <remarks>
/// The fallback keeps ~12 existing tests (and any node wired before per-cluster mesh Phase 2)
/// on exactly the pre-Phase-2 behaviour. It is not a permanent seam: Phase 6 deletes it along
/// with the DistributedPubSub branch and the single mesh.
/// </remarks>
/// <param name="topic">The legacy DistributedPubSub topic.</param>
/// <param name="message">The command.</param>
private void Dispatch(string topic, object message)
{
if (_meshRouter is not null) _meshRouter.Tell(new MeshCommand(topic, message));
else DistributedPubSub.Get(Context.System).Mediator.Tell(new Publish(topic, message));
}
/// <summary>
/// AdminUI Acknowledge path. Maps the control-plane command to a
/// <see cref="AlarmCommand"/> (<c>Operation = "Acknowledge"</c>) and publishes it onto the
@@ -85,7 +119,7 @@ public sealed class AdminOperationsActor : ReceiveActor
Comment: msg.Comment,
UnshelveAtUtc: null);
DistributedPubSub.Get(Context.System).Mediator.Tell(new Publish(AlarmCommandsTopic.Name, alarmCmd));
Dispatch(AlarmCommandsTopic.Name, alarmCmd);
_log.Info("AdminOps: Acknowledge published for alarm {AlarmId} by {User}", msg.AlarmId, msg.User);
replyTo.Tell(new AcknowledgeAlarmResult(true, null, msg.CorrelationId));
@@ -132,7 +166,7 @@ public sealed class AdminOperationsActor : ReceiveActor
Comment: msg.Comment,
UnshelveAtUtc: unshelveAt);
DistributedPubSub.Get(Context.System).Mediator.Tell(new Publish(AlarmCommandsTopic.Name, alarmCmd));
Dispatch(AlarmCommandsTopic.Name, alarmCmd);
_log.Info("AdminOps: {Operation} published for alarm {AlarmId} by {User}", operation, msg.AlarmId, msg.User);
replyTo.Tell(new ShelveAlarmResult(true, null, msg.CorrelationId));
@@ -394,7 +428,7 @@ public sealed class AdminOperationsActor : ReceiveActor
{
// Broadcast to every DriverHostActor on every node via the driver-control DPS topic.
// Only the host that owns the instance will act; others ignore it (id not found in _children).
DistributedPubSub.Get(Context.System).Mediator.Tell(new Publish(DriverControlTopic.Name, msg));
Dispatch(DriverControlTopic.Name, msg);
await using var db = await _dbFactory.CreateDbContextAsync();
db.ConfigEdits.Add(new ConfigEdit
@@ -424,7 +458,7 @@ public sealed class AdminOperationsActor : ReceiveActor
try
{
// Broadcast to every DriverHostActor; only the one owning the instance reacts.
DistributedPubSub.Get(Context.System).Mediator.Tell(new Publish(DriverControlTopic.Name, msg));
Dispatch(DriverControlTopic.Name, msg);
await using var db = await _dbFactory.CreateDbContextAsync();
db.ConfigEdits.Add(new ConfigEdit
src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Communication/CentralCommunicationActor.cs
+308
View File
@@ -0,0 +1,308 @@
using System.Collections.Immutable;
using Akka.Actor;
using Akka.Cluster.Tools.Client;
using Akka.Cluster.Tools.PublishSubscribe;
using Akka.Event;
using Microsoft.EntityFrameworkCore;
using ZB.MOM.WW.OtOpcUa.Cluster;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Deploy;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
using ZB.MOM.WW.OtOpcUa.Configuration;
namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
/// <summary>
/// Central-side end of the central↔node command boundary (per-cluster mesh Phase 2). Routes
/// every central→node command out — over DistributedPubSub or ClusterClient, per
/// <see cref="MeshTransportOptions.Mode"/> — and forwards inbound <see cref="ApplyAck"/>s to the
/// deploy coordinator singleton.
/// </summary>
/// <remarks>
/// <para>
/// <b>Registered per node, NOT as a cluster singleton.</b> A driver node's ClusterClient
/// rotates across contact points and must find a live comm actor at whichever central node
/// answers. A singleton would be reachable only through the one node hosting it, and the
/// rotation would fail silently against the other.
/// </para>
/// <para>
/// <b>Exactly one ClusterClient, fleet-wide.</b> See <see cref="RebuildClient"/> — this is
/// the single most important thing to understand before changing this actor.
/// </para>
/// </remarks>
public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
{
private const string RefreshTimerKey = "mesh-contact-refresh";
private readonly IDbContextFactory<OtOpcUaConfigDbContext> _dbFactory;
private readonly MeshTransportOptions _options;
private readonly IMeshClusterClientFactory _clientFactory;
private readonly Func<IActorRef?> _coordinator;
private readonly ILoggingAdapter _log = Context.GetLogger();
private readonly bool _useClusterClient;
private IActorRef? _client;
private ImmutableHashSet<string> _currentContacts = ImmutableHashSet<string>.Empty;
private CancellationTokenSource _lifecycle = new();
/// <summary>Gets the timer scheduler driving the periodic contact refresh.</summary>
public ITimerScheduler Timers { get; set; } = null!;
/// <summary>Creates the props for this actor.</summary>
/// <param name="dbFactory">Factory for the config database holding <c>ClusterNode</c> rows.</param>
/// <param name="options">The bound mesh-transport options.</param>
/// <param name="clientFactory">Creates the ClusterClient; substituted in tests.</param>
/// <param name="coordinator">
/// Resolves the deploy-coordinator singleton proxy. A <see cref="Func{TResult}"/> rather than
/// an <see cref="IActorRef"/> so this actor never depends on registration order at wiring
/// time — it resolves on first use and caches.
/// </param>
/// <returns>The props.</returns>
public static Props Props(
IDbContextFactory<OtOpcUaConfigDbContext> dbFactory,
MeshTransportOptions options,
IMeshClusterClientFactory clientFactory,
Func<IActorRef?> coordinator) =>
Akka.Actor.Props.Create(() =>
new CentralCommunicationActor(dbFactory, options, clientFactory, coordinator));
/// <summary>Initializes a new instance of the <see cref="CentralCommunicationActor"/> class.</summary>
/// <param name="dbFactory">Factory for the config database.</param>
/// <param name="options">The bound mesh-transport options.</param>
/// <param name="clientFactory">Creates the ClusterClient.</param>
/// <param name="coordinator">Lazily resolves the deploy-coordinator singleton proxy.</param>
public CentralCommunicationActor(
IDbContextFactory<OtOpcUaConfigDbContext> dbFactory,
MeshTransportOptions options,
IMeshClusterClientFactory clientFactory,
Func<IActorRef?> coordinator)
{
_dbFactory = dbFactory;
_options = options;
_clientFactory = clientFactory;
_coordinator = coordinator;
_useClusterClient = string.Equals(
options.Mode, MeshTransportOptions.ModeClusterClient, StringComparison.OrdinalIgnoreCase);
Receive<MeshCommand>(HandleMeshCommand);
Receive<ApplyAck>(HandleApplyAck);
Receive<RefreshContacts>(_ => LoadContactsFromDb());
Receive<ContactsLoaded>(HandleContactsLoaded);
// A faulted LoadContactsFromDb task is piped here as a Status.Failure. Without this handler
// the failure is an unhandled message (debug level only) and the refresh fails silently —
// an operator cannot distinguish "no nodes configured" from "the database is down", and the
// contact set silently freezes at whatever it last held.
Receive<Status.Failure>(f => _log.Warning(
f.Cause,
"Failed to load ClusterNode contact points; the mesh ClusterClient contact set was NOT "
+ "refreshed and may be stale or empty. Commands may be dropped until the next refresh"));
Receive<SubscribeAck>(_ => { /* DPS subscribe confirmation */ });
}
/// <inheritdoc />
protected override void PreStart()
{
if (!_useClusterClient)
{
_log.Info(
"Mesh transport is {Mode}: commands are published on DistributedPubSub and no "
+ "ClusterClient is created", _options.Mode);
return;
}
Timers.StartPeriodicTimer(
RefreshTimerKey,
new RefreshContacts(),
TimeSpan.Zero,
TimeSpan.FromSeconds(_options.ContactRefreshSeconds));
}
/// <inheritdoc />
protected override void PostStop()
{
_lifecycle.Cancel();
_lifecycle.Dispose();
}
private void HandleMeshCommand(MeshCommand cmd)
{
if (!_useClusterClient)
{
DistributedPubSub.Get(Context.System).Mediator.Tell(new Publish(cmd.Topic, cmd.Message));
return;
}
if (_client is null)
{
// App-level drop, matching the sister project's decision and this repo's buffer-size = 0.
// The command is NOT queued and will NOT be retried: a deploy fails at the coordinator's
// apply deadline naming the silent nodes, which is a better operator signal than a
// command arriving minutes late against a config the DB already recorded as failed.
_log.Warning(
"No mesh ClusterClient — dropping {MessageType}. The last contact refresh produced "
+ "no usable contact point from the enabled, non-maintenance ClusterNode rows; the "
+ "command is not buffered and will not be retried",
cmd.Message.GetType().Name);
return;
}
// SendToAll, NEVER Send. Today's DPS publish reaches EVERY DriverHostActor — there is no
// ClusterId or node filter on the node side at all; scoping happens later, inside the
// artifact (DeploymentArtifact.ResolveClusterScope). ClusterClient.Send delivers to exactly
// ONE registered actor, which would deploy to a single node of the fleet while every other
// node silently kept its old configuration — and the deployment would still seal green if
// the ack set happened to be satisfied.
_client.Tell(new ClusterClient.SendToAll(MeshPaths.NodeCommunication, cmd.Message));
}
private void HandleApplyAck(ApplyAck ack)
{
var coordinator = _coordinator();
if (coordinator is null)
{
_log.Warning(
"Received ApplyAck for {DeploymentId} from {NodeId} but the deploy coordinator is "
+ "not resolvable on this node; the ack is lost and the deployment will time out "
+ "naming a node that applied successfully",
ack.DeploymentId, ack.NodeId);
return;
}
// Forward, not Tell: preserves the original sender across the hop so the coordinator sees
// the node rather than this relay.
coordinator.Forward(ack);
}
private void LoadContactsFromDb()
{
var self = Self;
CancellationToken ct;
try
{
ct = _lifecycle.Token;
}
catch (ObjectDisposedException)
{
return; // Stopping.
}
var systemName = Context.System.Name;
var dbFactory = _dbFactory;
// Off the actor thread, result piped back as a message. A synchronous DB read here would
// block every command dispatch behind a slow or unreachable SQL Server.
Task.Run(async () =>
{
await using var db = await dbFactory.CreateDbContextAsync(ct).ConfigureAwait(false);
var rows = await db.ClusterNodes
.AsNoTracking()
.Where(n => n.Enabled && !n.MaintenanceMode)
.Select(n => new { n.NodeId, n.Host, n.AkkaPort })
.ToListAsync(ct)
.ConfigureAwait(false);
var contacts = new List<string>(rows.Count);
var malformed = new List<string>();
foreach (var row in rows)
{
var address = $"akka.tcp://{systemName}@{row.Host}:{row.AkkaPort}/system/receptionist";
// Parse up front, per row, inside the loop's own guard. A single malformed row must
// not abort the whole refresh and leave the contact set half-built — the sister
// project shipped that bug and its regression test deliberately orders the bad row
// first.
if (ActorPath.TryParse(address, out _)) contacts.Add(address);
else malformed.Add($"{row.NodeId} -> {address}");
}
return new ContactsLoaded(contacts, malformed);
}, ct).PipeTo(self);
}
private void HandleContactsLoaded(ContactsLoaded msg)
{
foreach (var bad in msg.Malformed)
{
_log.Warning(
"ClusterNode {Row} does not yield a usable receptionist address; skipping it in this "
+ "refresh (other nodes are unaffected). Check the row's Host and AkkaPort", bad);
}
var contacts = msg.Contacts.ToImmutableHashSet();
if (contacts.IsEmpty)
{
_log.Warning(
"No usable ClusterNode contact points; the mesh ClusterClient was not created and "
+ "every central→node command will be dropped until a refresh finds one");
return;
}
if (_client is not null && _currentContacts.SetEquals(contacts)) return;
RebuildClient(contacts);
}
/// <summary>Stops any existing client and creates a replacement for the new contact set.</summary>
/// <remarks>
/// <para>
/// <b>TODO(Phase 6): one client per application <c>Cluster</c>.</b> Today this actor
/// creates exactly <b>one</b> client whose contacts are the whole fleet, and that is not
/// a compromise — it is the only correct shape while the fleet is a single Akka mesh.
/// </para>
/// <para>
/// A <c>ClusterClientReceptionist</c> serves its entire cluster, so
/// <c>SendToAll</c> reaches every registered node-comm actor in the mesh <i>regardless of
/// which node's address was used as the contact point</i>. One client per Cluster would
/// therefore fan each command out once per cluster — N× duplicate
/// <c>DispatchDeployment</c> and N× duplicate <c>ApplyAck</c>. Phase 6 splits the meshes,
/// at which point per-cluster clients become both correct and necessary.
/// </para>
/// <para>
/// Corollary worth stating because it is easy to assume otherwise: <b>the contact set
/// does not scope delivery.</b> Excluding a maintenance-mode node from the contacts does
/// not stop it receiving commands — it still receives them, exactly as it does under
/// today's DPS broadcast. The filter is about which receptionists are worth <i>dialling</i>
/// (a node in maintenance may be switched off), not about who gets the message.
/// </para>
/// </remarks>
/// <param name="contacts">The receptionist addresses to dial.</param>
private void RebuildClient(ImmutableHashSet<string> contacts)
{
if (_client is not null)
{
_log.Info("Mesh contact set changed; replacing the ClusterClient");
Context.Stop(_client);
// Cleared now: if the create below throws, a stale ref would route commands into a
// stopping actor and they would vanish without a warning.
_client = null;
_currentContacts = ImmutableHashSet<string>.Empty;
}
try
{
var paths = contacts.Select(ActorPath.Parse).ToImmutableHashSet();
_client = _clientFactory.Create(Context.System, paths);
_currentContacts = contacts;
_log.Info("Mesh ClusterClient created with {Count} contact point(s)", contacts.Count);
}
catch (Exception ex)
{
_log.Error(ex,
"Failed to create the mesh ClusterClient; central→node commands are dropped until "
+ "the next contact refresh succeeds");
}
}
/// <summary>Internal tick asking for a contact-point refresh.</summary>
public sealed record RefreshContacts;
/// <summary>Result of a contact-point load, piped back from the DB read.</summary>
/// <param name="Contacts">Receptionist addresses that parsed cleanly.</param>
/// <param name="Malformed">Rows that did not, described for the log.</param>
public sealed record ContactsLoaded(
IReadOnlyList<string> Contacts,
IReadOnlyList<string> Malformed);
}
src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Communication/IMeshClusterClientFactory.cs
+47
View File
@@ -0,0 +1,47 @@
using System.Collections.Immutable;
using Akka.Actor;
using Akka.Cluster.Tools.Client;
namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
/// <summary>
/// Creates the <c>ClusterClient</c> central uses to reach driver nodes. Exists as a seam so
/// <c>CentralCommunicationActor</c> can be tested against a <c>TestProbe</c> without standing up
/// a second cluster.
/// </summary>
public interface IMeshClusterClientFactory
{
/// <summary>Creates a client for the given receptionist contact points.</summary>
/// <param name="system">The actor system to spawn the client in.</param>
/// <param name="contacts">Receptionist actor paths, one per reachable node.</param>
/// <returns>The new client's <see cref="IActorRef"/>.</returns>
IActorRef Create(ActorSystem system, ImmutableHashSet<ActorPath> contacts);
}
/// <inheritdoc />
public sealed class DefaultMeshClusterClientFactory : IMeshClusterClientFactory
{
/// <summary>
/// Per-incarnation generation counter, appended to the actor name.
/// </summary>
/// <remarks>
/// <c>Context.Stop</c> of the previous client is <b>asynchronous</b> — its actor name stays
/// reserved until termination completes — so recreating the same name while handling a single
/// message throws <see cref="InvalidActorNameException"/>. That is exactly what a contact-set
/// change does: stop the old client, create the replacement, both in one handler. A
/// generation suffix makes every incarnation's name unique by construction. Ported from the
/// sister project, where it was a shipped bug fix rather than foresight.
/// </remarks>
private long _generation;
/// <inheritdoc />
public IActorRef Create(ActorSystem system, ImmutableHashSet<ActorPath> contacts)
{
ArgumentNullException.ThrowIfNull(system);
ArgumentNullException.ThrowIfNull(contacts);
var settings = ClusterClientSettings.Create(system).WithInitialContacts(contacts);
var name = $"mesh-client-{Interlocked.Increment(ref _generation)}";
return system.ActorOf(ClusterClient.Props(settings), name);
}
}
src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Coordinators/ConfigPublishCoordinator.cs
+28 -4
View File
@@ -3,6 +3,7 @@ using Akka.Cluster.Tools.PublishSubscribe;
using Akka.Event;
using Microsoft.EntityFrameworkCore;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Deploy;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
using ZB.MOM.WW.OtOpcUa.Commons.Types;
using ZB.MOM.WW.OtOpcUa.Configuration;
using ZB.MOM.WW.OtOpcUa.Configuration.Entities;
@@ -60,6 +61,17 @@ public sealed class ConfigPublishCoordinator : ReceiveActor, IWithTimers
private readonly IDbContextFactory<OtOpcUaConfigDbContext> _dbFactory;
private readonly TimeSpan _applyDeadline;
/// <summary>
/// Where central→node commands are handed off (per-cluster mesh Phase 2), or
/// <see langword="null"/> to publish on the mediator directly.
/// </summary>
/// <remarks>
/// Null in tests and on any node wired before Phase 2, where the fallback reproduces the
/// pre-Phase-2 behaviour exactly. The fallback is not a permanent seam — Phase 6 deletes it
/// along with the DistributedPubSub branch and the single mesh.
/// </remarks>
private readonly IActorRef? _meshRouter;
private readonly ILoggingAdapter _log = Context.GetLogger();
// NodeId equality here is case-insensitive (by Value) to match the case-insensitive ClusterId/
// NodeId scoping in DeploymentArtifact.ResolveClusterScope — so an ack from a node whose address
@@ -78,22 +90,31 @@ public sealed class ConfigPublishCoordinator : ReceiveActor, IWithTimers
/// <param name="dbFactory">The database context factory for accessing configuration data.</param>
/// <param name="applyDeadline">The timeout for waiting for apply acknowledgments from cluster nodes.</param>
/// <returns>The Akka.NET <see cref="Akka.Actor.Props"/> used to spawn this actor.</returns>
/// <param name="meshRouter">
/// Central comm actor the dispatch is handed to, or <see langword="null"/> to publish on the
/// DistributedPubSub mediator directly (the pre-Phase-2 behaviour).
/// </param>
public static Props Props(
IDbContextFactory<OtOpcUaConfigDbContext> dbFactory,
TimeSpan? applyDeadline = null) =>
Akka.Actor.Props.Create(() => new ConfigPublishCoordinator(dbFactory, applyDeadline ?? DefaultApplyDeadline));
TimeSpan? applyDeadline = null,
IActorRef? meshRouter = null) =>
Akka.Actor.Props.Create(() =>
new ConfigPublishCoordinator(dbFactory, applyDeadline ?? DefaultApplyDeadline, meshRouter));
/// <summary>
/// Initializes a new instance of the <see cref="ConfigPublishCoordinator"/> class.
/// </summary>
/// <param name="dbFactory">The database context factory for persisting deployment state.</param>
/// <param name="applyDeadline">The timeout for waiting for per-node apply acknowledgments.</param>
/// <param name="meshRouter">Central comm actor, or <see langword="null"/> for the mediator.</param>
public ConfigPublishCoordinator(
IDbContextFactory<OtOpcUaConfigDbContext> dbFactory,
TimeSpan applyDeadline)
TimeSpan applyDeadline,
IActorRef? meshRouter = null)
{
_dbFactory = dbFactory;
_applyDeadline = applyDeadline;
_meshRouter = meshRouter;
Receive<DispatchDeployment>(HandleDispatch);
Receive<ApplyAck>(HandleAck);
Receive<DeadlineElapsed>(HandleDeadline);
@@ -172,7 +193,10 @@ public sealed class ConfigPublishCoordinator : ReceiveActor, IWithTimers
db.SaveChanges();
}
DistributedPubSub.Get(Context.System).Mediator.Tell(new Publish(DeploymentsTopic, msg));
// Per-cluster mesh Phase 2: hand off to the transport router rather than publishing
// directly, so this actor no longer knows which transport is in force.
if (_meshRouter is not null) _meshRouter.Tell(new MeshCommand(DeploymentsTopic, msg));
else DistributedPubSub.Get(Context.System).Mediator.Tell(new Publish(DeploymentsTopic, msg));
Timers.StartSingleTimer(DeadlineTimerKey, new DeadlineElapsed(msg.DeploymentId), _applyDeadline);
if (_expectedAcks.Count == 0)
src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/ServiceCollectionExtensions.cs
+46 -2
View File
@@ -1,12 +1,17 @@
using Akka.Actor;
using Akka.Cluster.Hosting;
using Akka.Cluster.Tools.Client;
using Akka.Hosting;
using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.OtOpcUa.Configuration;
using ZB.MOM.WW.OtOpcUa.Cluster;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
using ZB.MOM.WW.OtOpcUa.ControlPlane.AdminOperations;
using ZB.MOM.WW.OtOpcUa.ControlPlane.Audit;
using ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
using ZB.MOM.WW.OtOpcUa.ControlPlane.Coordinators;
using ZB.MOM.WW.OtOpcUa.ControlPlane.Fleet;
using ZB.MOM.WW.OtOpcUa.ControlPlane.Redundancy;
@@ -48,12 +53,49 @@ public static class ServiceCollectionExtensions
var singletonOptions = new ClusterSingletonOptions { Role = AdminRole };
var proxyOptions = new ClusterSingletonOptions { Role = AdminRole };
// Per-cluster mesh Phase 2: the central end of the central↔node command boundary.
//
// A plain WithActors, NOT a singleton — and that is the load-bearing part. A driver node's
// ClusterClient rotates across contact points and must find a live comm actor at whichever
// central node answers. As a singleton it would exist on one central node only, and the
// rotation would fail silently against the other: acks would land sometimes, depending on
// which contact the client happened to settle on.
builder.WithActors((system, registry, resolver) =>
{
var dbFactory = resolver.GetService<IDbContextFactory<OtOpcUaConfigDbContext>>();
var options = resolver.GetService<IOptions<MeshTransportOptions>>().Value;
var actor = system.ActorOf(
CentralCommunicationActor.Props(
dbFactory,
options,
new DefaultMeshClusterClientFactory(),
// Resolved lazily, per message, so this actor never depends on the order in
// which Akka.Hosting materialises the singleton proxy relative to this block.
() => registry.TryGet<ConfigPublishCoordinatorKey>(out var coordinator)
? coordinator
: null),
MeshPaths.CentralCommunicationName);
// Registered unconditionally, in both transport modes. Under Dps nothing dials it, and
// an idle registration costs nothing — whereas registering only under ClusterClient
// would mean flipping the flag on a running fleet requires a central restart before any
// node can ack, turning a config change back into a deployment.
ClusterClientReceptionist.Get(system).RegisterService(actor);
registry.Register<CentralCommunicationActorKey>(actor);
});
builder.WithSingleton<ConfigPublishCoordinatorKey>(
ConfigPublishSingletonName,
(system, registry, resolver) =>
{
var dbFactory = resolver.GetService<IDbContextFactory<OtOpcUaConfigDbContext>>();
return ConfigPublishCoordinator.Props(dbFactory);
// Registered above, so it is already in the registry — the same
// ordering-by-registration the AdminOperations singleton relies on to resolve the
// coordinator. Deliberately NOT an ActorSelection.ResolveOne here: that blocks
// inside a props factory and races the spawn it is waiting for.
var meshRouter = registry.Get<CentralCommunicationActorKey>();
return ConfigPublishCoordinator.Props(dbFactory, applyDeadline: null, meshRouter: meshRouter);
},
singletonOptions);
@@ -69,7 +111,8 @@ public static class ServiceCollectionExtensions
var mode = Enum.TryParse<TagConfigValidationMode>(
resolver.GetService<IConfiguration>()?["Deployment:TagConfigValidationMode"],
ignoreCase: true, out var m) ? m : TagConfigValidationMode.Warn;
return AdminOperationsActor.Props(dbFactory, coordinator, probes, mode);
var meshRouter = registry.Get<CentralCommunicationActorKey>();
return AdminOperationsActor.Props(dbFactory, coordinator, probes, mode, meshRouter);
},
singletonOptions);
@@ -123,3 +166,4 @@ public sealed class AuditWriterActorKey { }
public sealed class FleetStatusBroadcasterKey { }
public sealed class RedundancyStateActorKey { }
public sealed class ClusterNodeAddressReconcilerKey { }
public sealed class CentralCommunicationActorKey { }
src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Communication/NodeCommunicationActor.cs
+94
View File
@@ -0,0 +1,94 @@
using Akka.Actor;
using Akka.Cluster.Tools.Client;
using Akka.Event;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Admin;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Deploy;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
namespace ZB.MOM.WW.OtOpcUa.Runtime.Communication;
/// <summary>
/// Node-side end of the central↔node command boundary (per-cluster mesh Phase 2), registered
/// with the <c>ClusterClientReceptionist</c> at <see cref="MeshPaths.NodeCommunication"/>.
/// </summary>
/// <remarks>
/// <para>
/// <b>Registered per node, NOT as a cluster singleton</b> — central's ClusterClient rotates
/// across contact points and must find a live comm actor at whichever node answers.
/// </para>
/// <para>
/// <b>Inbound commands are re-emitted on the node's <see cref="EventStream"/>, not on their
/// DistributedPubSub topic.</b> DPS is mesh-wide, and central <c>SendToAll</c>s to every
/// node's comm actor — so a DPS re-publish would deliver N copies of every command to every
/// subscriber, where N is the node count. The EventStream is node-local by construction,
/// which is exactly the needed semantics. It also avoids a registration handshake with
/// actors spawned later: <c>ScriptedAlarmHostActor</c> is a <i>child of</i>
/// <c>DriverHostActor</c> and has no registry key, so there is no ref to hand in at wiring
/// time.
/// </para>
/// <para>
/// <b>Outbound is <see cref="ApplyAck"/> only.</b> There is no Ask across this boundary in
/// Phase 2 — every migrated command is fire-and-forget, and the deploy ack returns as an
/// independent message the coordinator already subscribes for. Nothing here needs sender
/// preservation, which is why this actor is far smaller than the sister project's.
/// </para>
/// </remarks>
public sealed class NodeCommunicationActor : ReceiveActor
{
private readonly IActorRef? _centralClient;
private readonly ILoggingAdapter _log = Context.GetLogger();
/// <summary>Creates the props for this actor.</summary>
/// <param name="centralClient">
/// ClusterClient dialling central's receptionist, or <see langword="null"/> under
/// <c>MeshTransport:Mode = Dps</c>, where acks still go out over DistributedPubSub from
/// <c>DriverHostActor</c> and this actor never sees them.
/// </param>
/// <returns>The props.</returns>
public static Props Props(IActorRef? centralClient) =>
Akka.Actor.Props.Create(() => new NodeCommunicationActor(centralClient));
/// <summary>Initializes a new instance of the <see cref="NodeCommunicationActor"/> class.</summary>
/// <param name="centralClient">ClusterClient dialling central, or <see langword="null"/>.</param>
public NodeCommunicationActor(IActorRef? centralClient)
{
_centralClient = centralClient;
// Inbound from central. Each type is listed explicitly rather than caught by a ReceiveAny:
// an unknown type must dead-letter loudly during development rather than be republished
// blind onto the node's EventStream, where it would be silently ignored by every subscriber.
Receive<DispatchDeployment>(PublishLocally);
Receive<RestartDriver>(PublishLocally);
Receive<ReconnectDriver>(PublishLocally);
Receive<AlarmCommand>(PublishLocally);
// Outbound to central.
Receive<ApplyAck>(HandleApplyAck);
}
private void PublishLocally(object msg)
{
_log.Debug("Mesh: received {MessageType} from central; publishing node-locally", msg.GetType().Name);
Context.System.EventStream.Publish(msg);
}
private void HandleApplyAck(ApplyAck ack)
{
if (_centralClient is null)
{
_log.Warning(
"No central ClusterClient — dropping ApplyAck for {DeploymentId}. The ack is not "
+ "buffered and will not be retried, so the deployment will time out at central "
+ "naming this node even though it applied successfully. Check "
+ "MeshTransport:CentralContactPoints",
ack.DeploymentId);
return;
}
// Send, not SendToAll: the deploy coordinator is a single singleton, and the central comm
// actor forwards to its proxy. SendToAll would deliver one ack per central node, and the
// coordinator would count the same node twice.
_centralClient.Tell(new ClusterClient.Send(MeshPaths.CentralCommunication, ack));
}
}
src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Drivers/DriverHostActor.cs
+31 -12
View File
@@ -86,7 +86,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
private readonly IDbContextFactory<OtOpcUaConfigDbContext> _dbFactory;
private readonly CommonsNodeId _localNode;
private readonly IActorRef? _coordinatorOverride;
private readonly IActorRef? _ackRouter;
private readonly IDriverFactory _driverFactory;
/// <summary>Builds the per-driver-instance <see cref="IDriverCapabilityInvoker"/> attached to each
@@ -328,7 +328,9 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
/// <summary>Creates props for a DriverHostActor with the specified dependencies.</summary>
/// <param name="dbFactory">Database context factory for configuration database access.</param>
/// <param name="localNode">The local cluster node identifier.</param>
/// <param name="coordinator">Optional coordinator actor reference for deployment coordination.</param>
/// <param name="ackRouter">Where ApplyAcks are sent. A direct coordinator handle in tests; under
/// <c>MeshTransport:Mode = ClusterClient</c> the node's <c>NodeCommunicationActor</c>, which relays
/// across the boundary. Null publishes on the <c>deployment-acks</c> DPS topic.</param>
/// <param name="driverFactory">Optional driver factory; defaults to null factory if not provided.</param>
/// <param name="localRoles">Optional set of roles assigned to the local node.</param>
/// <param name="dependencyMux">Optional actor reference for dependency multiplexing.</param>
@@ -364,7 +366,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
public static Props Props(
IDbContextFactory<OtOpcUaConfigDbContext> dbFactory,
CommonsNodeId localNode,
IActorRef? coordinator = null,
IActorRef? ackRouter = null,
IDriverFactory? driverFactory = null,
IReadOnlySet<string>? localRoles = null,
IActorRef? dependencyMux = null,
@@ -387,7 +389,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
// the wrong dependency at runtime. New parameters go LAST in all three places — this
// signature, the constructor's, and this call — and nothing else moves.
Akka.Actor.Props.Create(() => new DriverHostActor(
dbFactory, localNode, coordinator, driverFactory, localRoles, dependencyMux, opcUaPublishActor,
dbFactory, localNode, ackRouter, driverFactory, localRoles, dependencyMux, opcUaPublishActor,
healthPublisher, virtualTagEvaluator, historyWriter, virtualTagHostOverride,
loggerFactory, scriptRootLogger, scriptedAlarmHostOverride, invokerFactory, driverMemberCountProvider,
deploymentArtifactCache, redundancyRoleView, replicationPeerHost));
@@ -395,7 +397,9 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
/// <summary>Initializes a new DriverHostActor with the specified dependencies.</summary>
/// <param name="dbFactory">Database context factory for configuration database access.</param>
/// <param name="localNode">The local cluster node identifier.</param>
/// <param name="coordinator">Optional coordinator actor reference for deployment coordination.</param>
/// <param name="ackRouter">Where ApplyAcks are sent. A direct coordinator handle in tests; under
/// <c>MeshTransport:Mode = ClusterClient</c> the node's <c>NodeCommunicationActor</c>, which relays
/// across the boundary. Null publishes on the <c>deployment-acks</c> DPS topic.</param>
/// <param name="driverFactory">Optional driver factory; defaults to null factory if not provided.</param>
/// <param name="localRoles">Optional set of roles assigned to the local node.</param>
/// <param name="dependencyMux">Optional actor reference for dependency multiplexing.</param>
@@ -426,7 +430,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
public DriverHostActor(
IDbContextFactory<OtOpcUaConfigDbContext> dbFactory,
CommonsNodeId localNode,
IActorRef? coordinator,
IActorRef? ackRouter,
IDriverFactory? driverFactory = null,
IReadOnlySet<string>? localRoles = null,
IActorRef? dependencyMux = null,
@@ -449,7 +453,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
_replicationPeerHost = replicationPeerHost;
_dbFactory = dbFactory;
_localNode = localNode;
_coordinatorOverride = coordinator;
_ackRouter = ackRouter;
_driverFactory = driverFactory ?? NullDriverFactory.Instance;
_invokerFactory = invokerFactory ?? NullDriverCapabilityInvokerFactory.Instance;
_driverMemberCountProvider = driverMemberCountProvider;
@@ -483,6 +487,16 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
// uses so only the Primary services operator writes (the secondary keeps state warm for failover).
_mediator.Tell(
new Subscribe(ZB.MOM.WW.OtOpcUa.Runtime.OpcUa.OpcUaPublishActor.RedundancyStateTopic, Self));
// Per-cluster mesh Phase 2 dark switch. Under MeshTransport:Mode = ClusterClient these same
// commands arrive from NodeCommunicationActor on the node-local EventStream instead of the
// mesh-wide DPS topic. Subscribing to BOTH unconditionally is deliberate: exactly one of the
// two ever publishes, so the transport flag lives entirely on the publishing side and the
// switch can be flipped (or reverted) without touching any subscriber.
Context.System.EventStream.Subscribe(Self, typeof(DispatchDeployment));
Context.System.EventStream.Subscribe(Self, typeof(RestartDriver));
Context.System.EventStream.Subscribe(Self, typeof(ReconnectDriver));
// Spawn the VirtualTag host BEFORE Bootstrap so the bootstrap-restore path (which routes
// through PushDesiredSubscriptions and Tells ApplyVirtualTags) has a live host to target.
SpawnVirtualTagHost();
@@ -2410,15 +2424,20 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
private void SendAck(DeploymentId deploymentId, ApplyAckOutcome outcome, string? failureReason, CorrelationId correlation)
{
var ack = new ApplyAck(deploymentId, _localNode, outcome, failureReason, correlation);
if (_coordinatorOverride is not null)
if (_ackRouter is not null)
{
_coordinatorOverride.Tell(ack);
// Either a direct coordinator handle (tests) or, under
// MeshTransport:Mode = ClusterClient, the NodeCommunicationActor that relays this
// across the boundary. Renamed from _coordinatorOverride in per-cluster mesh Phase 2:
// under ClusterClient mode this ref is emphatically NOT a coordinator, and the old
// name would send the next reader looking for one.
_ackRouter.Tell(ack);
}
else
{
// No direct coordinator handle — publish on the dedicated ACK topic. The coordinator
// singleton subscribes there in PreStart so the ACK reaches whichever admin node hosts
// it without an actor-path lookup.
// No router — publish on the dedicated ACK topic. The coordinator singleton subscribes
// there in PreStart so the ACK reaches whichever admin node hosts it without an
// actor-path lookup.
DistributedPubSub.Get(Context.System).Mediator.Tell(new Publish(DeploymentAcksTopic, ack));
}
}
src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ScriptedAlarms/ScriptedAlarmHostActor.cs
+9
View File
@@ -528,6 +528,15 @@ public sealed class ScriptedAlarmHostActor : ReceiveActor
// RedundancyStateChanged and cache this node's role — OnEngineEmission gates the cluster-wide
// alerts publish to the Primary so a 2-node single-cluster deploy doesn't double-emit transitions.
_mediator.Tell(new Subscribe(OpcUaPublishActor.RedundancyStateTopic, Self));
// Per-cluster mesh Phase 2 dark switch. An AdminUI-originated ack/shelve arrives from
// NodeCommunicationActor on the node-local EventStream under
// MeshTransport:Mode = ClusterClient; the DPS subscribe above still carries the node-LOCAL
// publisher (OtOpcUaServerHostedService, the OPC UA Part 9 method calls), which never
// crosses the boundary and stays on DPS in both modes. Subscribing to both is safe: exactly
// one publisher feeds each path.
Context.System.EventStream.Subscribe(Self, typeof(AlarmCommand));
base.PreStart();
}
src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ServiceCollectionExtensions.cs
+47 -1
View File
@@ -1,4 +1,6 @@
using System.Collections.Immutable;
using Akka.Actor;
using Akka.Cluster.Tools.Client;
using Akka.Hosting;
using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.Configuration;
@@ -6,6 +8,10 @@ using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.DependencyInjection.Extensions;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.OtOpcUa.Cluster;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
using ZB.MOM.WW.OtOpcUa.Runtime.Communication;
using ZB.MOM.WW.OtOpcUa.Commons.Engines;
using ZB.MOM.WW.OtOpcUa.Commons.Interfaces;
using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
@@ -37,6 +43,7 @@ public static class ServiceCollectionExtensions
public const string OpcUaPublishActorName = "opcua-publish";
public const string PeerProbeSupervisorName = "peer-probe-supervisor";
public const string ContinuousHistorizationRecorderActorName = "continuous-historization-recorder";
public const string CentralClusterClientName = "central-cluster-client";
/// <summary>
/// Registers shared runtime services. Currently binds <see cref="IAlarmHistorianSink"/>
@@ -319,6 +326,40 @@ public static class ServiceCollectionExtensions
var mux = system.ActorOf(DependencyMuxActor.Props(), DependencyMuxActorName);
registry.Register<DependencyMuxActorKey>(mux);
// Per-cluster mesh Phase 2: the node end of the central↔node command boundary. Spawned
// BEFORE DriverHostActor because it is the host's ackRouter under ClusterClient mode.
// It has no dependency of its own on the host — inbound commands travel via the node's
// EventStream — so this ordering costs nothing.
var meshOptions = resolver.GetService<IOptions<MeshTransportOptions>>().Value;
var useClusterClient = string.Equals(
meshOptions.Mode, MeshTransportOptions.ModeClusterClient, StringComparison.OrdinalIgnoreCase);
IActorRef? centralClient = null;
if (useClusterClient)
{
// Contacts come from appsettings, not the database — the deliberate asymmetry:
// central's node set changes as operators add and retire nodes, but central's own
// address is part of the deployment. MeshTransportOptionsValidator has already
// rejected an empty or malformed list at boot, so this cannot silently produce a
// client with no contacts.
var contacts = meshOptions.CentralContactPoints
.Select(cp => ActorPath.Parse($"{cp}/system/receptionist"))
.ToImmutableHashSet();
centralClient = system.ActorOf(
ClusterClient.Props(ClusterClientSettings.Create(system).WithInitialContacts(contacts)),
CentralClusterClientName);
}
var nodeComm = system.ActorOf(
NodeCommunicationActor.Props(centralClient),
MeshPaths.NodeCommunicationName);
// Registered with the receptionist in BOTH modes: under Dps nothing dials it, and an
// idle registration costs nothing — whereas registering only under ClusterClient would
// make flipping the flag require a node restart before central could reach it.
ClusterClientReceptionist.Get(system).RegisterService(nodeComm);
registry.Register<NodeCommunicationActorKey>(nodeComm);
// Continuous-historization recorder — gated on ContinuousHistorization:Enabled AND the
// gateway-backed IHistorianValueWriter + the durable IHistorizationOutbox being registered
// (the Host registers both ONLY when historization is enabled and the ServerHistorian gateway
@@ -398,7 +439,11 @@ public static class ServiceCollectionExtensions
registry.Register<PeerProbeSupervisorKey>(peerProbes);
var driverHost = system.ActorOf(
DriverHostActor.Props(dbFactory, roleInfo.LocalNode, coordinator: null,
// ackRouter: under ClusterClient mode the node comm actor relays ApplyAcks across
// the boundary; under Dps it stays null and the host publishes on the
// deployment-acks topic exactly as before.
DriverHostActor.Props(dbFactory, roleInfo.LocalNode,
ackRouter: useClusterClient ? nodeComm : null,
driverFactory: driverFactory, localRoles: roleInfo.LocalRoles,
dependencyMux: mux,
opcUaPublishActor: publishActor,
@@ -430,6 +475,7 @@ public sealed class DbHealthProbeActorKey { }
public sealed class HistorianAdapterActorKey { }
public sealed class DependencyMuxActorKey { }
public sealed class OpcUaPublishActorKey { }
public sealed class NodeCommunicationActorKey { }
/// <summary>Marker key for the per-node ContinuousHistorizationRecorder (spawned only when
/// <c>ContinuousHistorization:Enabled=true</c> and the gateway value-writer + outbox are registered).</summary>
tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/MeshTransportHoconTests.cs
+100
View File
@@ -0,0 +1,100 @@
using Akka.Actor;
using Akka.Cluster.Tools.Client;
using Akka.Configuration;
using Shouldly;
using Xunit;
namespace ZB.MOM.WW.OtOpcUa.Cluster.Tests;
/// <summary>
/// Asserts the ClusterClient / receptionist / frame-size settings that are actually <b>in
/// force</b> on a real <see cref="ActorSystem"/> built from the shipped base config.
/// </summary>
/// <remarks>
/// Asserting the <c>akka.conf</c> text instead would pass while a competing HOCON fragment
/// silently overrode the value — the failure mode recorded at length in
/// <c>ServiceCollectionExtensions.BuildDowningHocon</c> and pinned the same way by
/// <c>SplitBrainResolverActivationTests</c>. In this codebase the effective value is the only
/// one worth asserting.
/// </remarks>
public class MeshTransportHoconTests : IDisposable
{
private readonly ActorSystem _sys;
public MeshTransportHoconTests()
{
var config = ConfigurationFactory
.ParseString("akka.remote.dot-netty.tcp.port = 0\nakka.cluster.seed-nodes = []")
.WithFallback(HoconLoader.LoadBaseConfig());
_sys = ActorSystem.Create("mesh-hocon-probe", config);
}
[Fact]
public void Cluster_client_buffering_is_disabled()
{
// Asserted through ClusterClientSettings — the object the client is actually built from —
// NOT through Config.GetInt. GetInt returns 0 for a MISSING key, so a config assertion here
// passes identically whether the setting is explicitly 0 or absent with Akka's default of
// 1000 in force. It was green before this feature existed; that is the whole defect class.
//
// buffer-size = 0 means "drop now if the receptionist is unknown". Buffering would replay a
// DispatchDeployment on reconnect and apply a deployment the coordinator already sealed as
// TimedOut, leaving the node running a configuration the database says failed. This is a
// behaviour decision, not a tuning knob.
ClusterClientSettings.Create(_sys).BufferSize.ShouldBe(0);
}
[Fact]
public void Cluster_client_retries_forever_rather_than_self_stopping()
{
// reconnect-timeout = off surfaces as a null TimeSpan on the settings object. A central node
// down for an hour must not require a driver-node restart to become reachable again.
ClusterClientSettings.Create(_sys).ReconnectTimeout.ShouldBeNull();
}
[Fact]
public void Receptionist_serves_every_member_regardless_of_role()
{
// An empty role is what makes "registered per node, not as a singleton" work: every member
// hosts a receptionist, so ClusterClient contact rotation reaches whichever node answers.
// Scoping this to a role would make the boundary reachable through one node only.
ClusterReceptionistSettings.Create(_sys).Role.ShouldBeNull();
}
[Fact]
public void Oversized_frames_are_logged_rather_than_dropped_silently()
{
// The sister project's most expensive incident. A message over the frame limit is dropped
// WITHOUT tearing down the association: heartbeats keep flowing, the node still reports
// healthy, and the only symptom is a command that never arrived. log-frame-size-exceeding
// converts that silence into a log line naming the offending size.
_sys.Settings.Config.GetByteSize("akka.remote.dot-netty.tcp.maximum-frame-size").ShouldBe(128000);
_sys.Settings.Config.GetByteSize("akka.remote.dot-netty.tcp.log-frame-size-exceeding").ShouldBe(32000);
}
[Fact]
public void Receptionist_extension_is_registered_at_startup()
{
// Not auto-started. Without this entry the receptionist materialises only if some code path
// happens to call ClusterClientReceptionist.Get(system) first, which makes "is the boundary
// listening?" depend on startup ordering rather than on configuration.
_sys.Settings.Config.GetStringList("akka.extensions")
.ShouldContain(s => s.Contains("ClusterClientReceptionistExtensionProvider"));
}
[Fact]
public void Distributed_pub_sub_extension_survives_alongside_the_receptionist()
{
// The dark switch keeps both transports live for one phase, and DPS additionally carries
// traffic Phase 2 does not touch at all — node-local alarm-command publishes and the
// node<->node redundancy-state probe leg. Losing it here would break far more than deploys.
_sys.Settings.Config.GetStringList("akka.extensions")
.ShouldContain(s => s.Contains("DistributedPubSubExtensionProvider"));
}
public void Dispose()
{
_sys.Terminate().Wait(TimeSpan.FromSeconds(5));
GC.SuppressFinalize(this);
}
}
tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/MeshTransportOptionsValidatorTests.cs
+138
View File
@@ -0,0 +1,138 @@
using Microsoft.Extensions.Options;
using Shouldly;
using Xunit;
namespace ZB.MOM.WW.OtOpcUa.Cluster.Tests;
/// <summary>
/// Every failure this validator catches otherwise surfaces as an <i>absence</i> — a deployment
/// that never arrives, an alarm ack that does nothing, a node whose ack is dropped without a
/// trace. Absences are the hardest class of fault to attribute in the field, so the whole point
/// of these tests is that the host refuses to start instead.
/// </summary>
public class MeshTransportOptionsValidatorTests
{
private static ValidateOptionsResult Validate(MeshTransportOptions o) =>
new MeshTransportOptionsValidator().Validate(MeshTransportOptions.SectionName, o);
[Fact]
public void Default_options_are_valid()
{
// The default is Dps — the pre-Phase-2 transport. A node that never sets this section must
// behave exactly as it did before, which is the whole premise of the dark switch.
Validate(new MeshTransportOptions()).Succeeded.ShouldBeTrue();
}
[Fact]
public void Unknown_mode_fails()
{
var result = Validate(new MeshTransportOptions { Mode = "grpc" });
result.Failed.ShouldBeTrue();
result.FailureMessage.ShouldContain("grpc");
}
[Fact]
public void Mode_matching_is_case_insensitive()
{
Validate(new MeshTransportOptions
{
Mode = "clusterclient",
CentralContactPoints = ["akka.tcp://otopcua@central-1:4053"],
}).Succeeded.ShouldBeTrue();
}
[Fact]
public void Non_positive_contact_refresh_fails()
{
var result = Validate(new MeshTransportOptions { ContactRefreshSeconds = 0 });
result.Failed.ShouldBeTrue();
result.FailureMessage.ShouldContain(nameof(MeshTransportOptions.ContactRefreshSeconds));
}
[Fact]
public void ClusterClient_mode_requires_central_contact_points()
{
var result = Validate(new MeshTransportOptions
{
Mode = MeshTransportOptions.ModeClusterClient,
CentralContactPoints = [],
});
result.Failed.ShouldBeTrue();
result.FailureMessage.ShouldContain(nameof(MeshTransportOptions.CentralContactPoints));
}
[Fact]
public void Dps_mode_does_not_require_central_contact_points()
{
// An admin-only node has no reason to carry them, and requiring them under the default mode
// would make the section mandatory everywhere for a feature that is switched off.
Validate(new MeshTransportOptions
{
Mode = MeshTransportOptions.ModeDps,
CentralContactPoints = [],
}).Succeeded.ShouldBeTrue();
}
[Fact]
public void A_contact_point_that_is_not_an_akka_address_fails()
{
var result = Validate(new MeshTransportOptions
{
Mode = MeshTransportOptions.ModeClusterClient,
CentralContactPoints = ["central-1:4053"],
});
result.Failed.ShouldBeTrue();
result.FailureMessage.ShouldContain("akka.tcp://");
}
[Fact]
public void A_contact_point_carrying_an_actor_path_suffix_fails()
{
// The sister project's shipped site template listed a full receptionist path, and separately
// listed the site's OWN remoting port as the second contact — "a permanent failure in the
// initial-contact rotation". Both are invisible until a command goes missing.
var result = Validate(new MeshTransportOptions
{
Mode = MeshTransportOptions.ModeClusterClient,
CentralContactPoints = ["akka.tcp://otopcua@central-1:4053/system/receptionist"],
});
result.Failed.ShouldBeTrue();
result.FailureMessage.ShouldContain("/system/receptionist");
}
[Fact]
public void One_bad_contact_point_among_good_ones_still_fails()
{
var result = Validate(new MeshTransportOptions
{
Mode = MeshTransportOptions.ModeClusterClient,
CentralContactPoints =
[
"akka.tcp://otopcua@central-1:4053",
"central-2:4053",
],
});
result.Failed.ShouldBeTrue();
result.FailureMessage.ShouldContain("central-2:4053");
}
[Fact]
public void Well_formed_ClusterClient_options_succeed()
{
Validate(new MeshTransportOptions
{
Mode = MeshTransportOptions.ModeClusterClient,
CentralContactPoints =
[
"akka.tcp://otopcua@central-1:4053",
"akka.tcp://otopcua@central-2:4053",
],
}).Succeeded.ShouldBeTrue();
}
}
tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/Communication/CentralCommunicationActorTests.cs
+271
View File
@@ -0,0 +1,271 @@
using System.Collections.Immutable;
using Akka.Actor;
using Akka.Cluster.Tools.Client;
using Akka.Cluster.Tools.PublishSubscribe;
using Akka.TestKit;
using Microsoft.EntityFrameworkCore;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Cluster;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Deploy;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
using ZB.MOM.WW.OtOpcUa.Commons.Types;
using ZB.MOM.WW.OtOpcUa.Configuration;
using ZB.MOM.WW.OtOpcUa.Configuration.Entities;
using ZB.MOM.WW.OtOpcUa.Configuration.Enums;
using ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
using ZB.MOM.WW.OtOpcUa.ControlPlane.Tests.Harness;
namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Tests.Communication;
public class CentralCommunicationActorTests : ControlPlaneActorTestBase
{
private const string DeploymentsTopic = "deployments";
/// <summary>Records every client the actor asks for, and hands back a probe in its place.</summary>
private sealed class RecordingClientFactory : IMeshClusterClientFactory
{
private readonly Func<TestProbe> _newProbe;
public RecordingClientFactory(Func<TestProbe> newProbe) => _newProbe = newProbe;
public List<ImmutableHashSet<ActorPath>> Calls { get; } = [];
public List<TestProbe> Probes { get; } = [];
public IActorRef Create(ActorSystem system, ImmutableHashSet<ActorPath> contacts)
{
Calls.Add(contacts);
var probe = _newProbe();
Probes.Add(probe);
return probe.Ref;
}
}
private static MeshTransportOptions ClusterClientMode(int refreshSeconds = 60) => new()
{
Mode = MeshTransportOptions.ModeClusterClient,
CentralContactPoints = ["akka.tcp://otopcua@central-1:4053"],
ContactRefreshSeconds = refreshSeconds,
};
private static void SeedCluster(
IDbContextFactory<OtOpcUaConfigDbContext> factory,
string clusterId,
params (string NodeId, string Host, int AkkaPort, bool Enabled, bool Maintenance)[] nodes)
{
using var db = factory.CreateDbContext();
db.ServerClusters.Add(new ServerCluster
{
ClusterId = clusterId,
Name = clusterId,
Enterprise = "ZB",
Site = "Site",
// Warm/Hot require NodeCount = 2 (CK_ServerCluster_RedundancyMode_NodeCount); these
// fixtures only need the row to exist, so use the standalone shape.
RedundancyMode = RedundancyMode.None,
NodeCount = 1,
CreatedBy = "test",
});
foreach (var n in nodes)
{
db.ClusterNodes.Add(new ClusterNode
{
NodeId = n.NodeId,
ClusterId = clusterId,
Host = n.Host,
AkkaPort = n.AkkaPort,
ApplicationUri = $"urn:{n.NodeId}",
Enabled = n.Enabled,
MaintenanceMode = n.Maintenance,
CreatedBy = "test",
});
}
db.SaveChanges();
}
private IActorRef Spawn(
IDbContextFactory<OtOpcUaConfigDbContext> factory,
MeshTransportOptions options,
IMeshClusterClientFactory clientFactory,
IActorRef? coordinator = null) =>
Sys.ActorOf(CentralCommunicationActor.Props(factory, options, clientFactory, () => coordinator));
[Fact]
public void Dps_mode_publishes_on_the_carried_topic_and_creates_no_client()
{
var factory = NewInMemoryDbFactory();
var clients = new RecordingClientFactory(() => CreateTestProbe());
var mediatorProbe = CreateTestProbe();
// Subscribe on the probe's behalf: SubscribeAck goes to the SENDER, so telling the mediator
// without an explicit sender would ack to TestActor and this wait would hang.
DistributedPubSub.Get(Sys).Mediator.Tell(
new Subscribe(DeploymentsTopic, mediatorProbe.Ref), mediatorProbe.Ref);
mediatorProbe.ExpectMsg<SubscribeAck>(TimeSpan.FromSeconds(5));
var actor = Spawn(factory, new MeshTransportOptions(), clients);
var dispatch = new DispatchDeployment(
new DeploymentId(Guid.NewGuid()), new RevisionHash("abc"), new CorrelationId(Guid.NewGuid()));
actor.Tell(new MeshCommand(DeploymentsTopic, dispatch));
mediatorProbe.ExpectMsg<DispatchDeployment>(TimeSpan.FromSeconds(5)).ShouldBe(dispatch);
clients.Calls.ShouldBeEmpty();
}
[Fact]
public void ClusterClient_mode_sends_to_all_not_to_one()
{
// THE assertion of this phase. ClusterClient.Send delivers to exactly ONE registered actor;
// today's DPS publish reaches EVERY DriverHostActor, and the node side has no ClusterId or
// node filter to compensate. Send would deploy to a single node while every other node
// silently kept its old configuration.
var factory = NewInMemoryDbFactory();
SeedCluster(factory, "C1", ("host-a:4053", "host-a", 4053, true, false));
var clients = new RecordingClientFactory(() => CreateTestProbe());
var actor = Spawn(factory, ClusterClientMode(), clients);
AwaitCondition(() => clients.Probes.Count == 1, TimeSpan.FromSeconds(5));
var dispatch = new DispatchDeployment(
new DeploymentId(Guid.NewGuid()), new RevisionHash("abc"), new CorrelationId(Guid.NewGuid()));
actor.Tell(new MeshCommand(DeploymentsTopic, dispatch));
var sent = clients.Probes[0].ExpectMsg<ClusterClient.SendToAll>(TimeSpan.FromSeconds(5));
sent.Path.ShouldBe(MeshPaths.NodeCommunication);
sent.Message.ShouldBe(dispatch);
}
[Fact]
public void Exactly_one_client_is_created_for_a_multi_cluster_fleet()
{
// A receptionist serves its whole mesh, so one client per Cluster would fan each command out
// once per cluster while the fleet is still a single mesh — N x duplicate deployments.
var factory = NewInMemoryDbFactory();
SeedCluster(factory, "C1", ("host-a:4053", "host-a", 4053, true, false));
SeedCluster(factory, "C2", ("host-b:4053", "host-b", 4053, true, false));
var clients = new RecordingClientFactory(() => CreateTestProbe());
Spawn(factory, ClusterClientMode(), clients);
AwaitCondition(() => clients.Calls.Count == 1, TimeSpan.FromSeconds(5));
// ...and that one client dials BOTH clusters' nodes.
clients.Calls[0].Count.ShouldBe(2);
// Give a second refresh a chance to (wrongly) create another.
ExpectNoMsg(TimeSpan.FromMilliseconds(300));
clients.Calls.Count.ShouldBe(1);
}
[Fact]
public void Disabled_and_maintenance_nodes_are_not_dialled()
{
var factory = NewInMemoryDbFactory();
SeedCluster(
factory,
"C1",
("host-a:4053", "host-a", 4053, true, false),
("host-b:4053", "host-b", 4053, false, false),
("host-c:4053", "host-c", 4053, true, true));
var clients = new RecordingClientFactory(() => CreateTestProbe());
Spawn(factory, ClusterClientMode(), clients);
AwaitCondition(() => clients.Calls.Count == 1, TimeSpan.FromSeconds(5));
clients.Calls[0].Count.ShouldBe(1);
clients.Calls[0].Single().ToString().ShouldContain("host-a");
}
[Fact]
public void A_malformed_node_row_does_not_abort_the_refresh()
{
// The bad row is ordered FIRST on purpose: the failure this guards against is an exception
// that aborts the loop and leaves the contact set half-built.
var factory = NewInMemoryDbFactory();
SeedCluster(
factory,
"C1",
("bad", "not a host!!", 4053, true, false),
("host-a:4053", "host-a", 4053, true, false));
var clients = new RecordingClientFactory(() => CreateTestProbe());
Spawn(factory, ClusterClientMode(), clients);
AwaitCondition(() => clients.Calls.Count == 1, TimeSpan.FromSeconds(5));
clients.Calls[0].ShouldContain(p => p.ToString().Contains("host-a"));
}
[Fact]
public void A_command_with_no_client_is_dropped_with_a_warning()
{
// No ClusterNode rows at all: nothing to dial, so the command must be dropped loudly rather
// than queued. buffer-size = 0 covers the case where a client exists but cannot connect;
// this covers the case where there is no client at all.
var factory = NewInMemoryDbFactory();
var clients = new RecordingClientFactory(() => CreateTestProbe());
var actor = Spawn(factory, ClusterClientMode(), clients);
EventFilter.Warning(contains: "dropping").ExpectOne(TimeSpan.FromSeconds(5), () =>
actor.Tell(new MeshCommand(DeploymentsTopic, "anything")));
}
[Fact]
public void ApplyAck_is_forwarded_to_the_coordinator_preserving_the_sender()
{
var factory = NewInMemoryDbFactory();
var clients = new RecordingClientFactory(() => CreateTestProbe());
var coordinator = CreateTestProbe();
var node = CreateTestProbe();
var actor = Spawn(factory, ClusterClientMode(), clients, coordinator.Ref);
var ack = new ApplyAck(
new DeploymentId(Guid.NewGuid()),
new NodeId("host-a:4053"),
ApplyAckOutcome.Applied,
null,
new CorrelationId(Guid.NewGuid()));
actor.Tell(ack, node.Ref);
coordinator.ExpectMsg<ApplyAck>(TimeSpan.FromSeconds(5)).ShouldBe(ack);
// Forward, not Tell: the coordinator must see the node, not this relay.
coordinator.LastSender.ShouldBe(node.Ref);
}
[Fact]
public void An_unresolvable_coordinator_logs_rather_than_losing_the_ack_silently()
{
var factory = NewInMemoryDbFactory();
var clients = new RecordingClientFactory(() => CreateTestProbe());
var actor = Spawn(factory, ClusterClientMode(), clients, coordinator: null);
var ack = new ApplyAck(
new DeploymentId(Guid.NewGuid()),
new NodeId("host-a:4053"),
ApplyAckOutcome.Applied,
null,
new CorrelationId(Guid.NewGuid()));
EventFilter.Warning(contains: "not resolvable").ExpectOne(TimeSpan.FromSeconds(5), () =>
actor.Tell(ack));
}
[Fact]
public void A_faulted_contact_load_warns_and_leaves_the_actor_alive()
{
var factory = NewInMemoryDbFactory();
var clients = new RecordingClientFactory(() => CreateTestProbe());
var actor = Spawn(factory, ClusterClientMode(), clients);
// Filter on a phrase unique to the DB-failure warning. EventFilter matches case-INSENSITIVELY,
// so a looser "was NOT" also caught this actor's other warning ("the mesh ClusterClient was
// not created") and the assertion failed with two matches instead of one.
EventFilter.Warning(contains: "contact set was NOT refreshed").ExpectOne(
TimeSpan.FromSeconds(5),
() => actor.Tell(new Status.Failure(new InvalidOperationException("db down"))));
// Still serving: a transient DB outage must not take the command boundary down with it.
actor.Tell(new MeshCommand(DeploymentsTopic, "anything"));
ExpectNoMsg(TimeSpan.FromMilliseconds(200));
}
}
tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/Communication/MeshClusterClientFactoryTests.cs
+67
View File
@@ -0,0 +1,67 @@
using System.Collections.Immutable;
using Akka.Actor;
using Akka.Cluster.Tools.Client;
using Akka.Configuration;
using Akka.TestKit.Xunit2;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Tests.Communication;
/// <summary>
/// Guards the one non-obvious part of client creation: recreating a client with the same name
/// inside a single message handler throws, because <c>Context.Stop</c> is asynchronous and the
/// name stays reserved until termination completes.
/// </summary>
public class MeshClusterClientFactoryTests : TestKit
{
private static readonly Config TestConfig = ConfigurationFactory.ParseString(@"
akka.actor.provider = cluster
akka.remote.dot-netty.tcp.port = 0
akka.remote.dot-netty.tcp.hostname = 127.0.0.1")
.WithFallback(ClusterClientReceptionist.DefaultConfig());
public MeshClusterClientFactoryTests()
: base(TestConfig)
{
}
// A real ClusterClient pointed at an unreachable contact is safe in TestKit — it simply retries
// the initial-contact rotation and never fails the test.
private static ImmutableHashSet<ActorPath> UnreachableContact => ImmutableHashSet.Create(
ActorPath.Parse("akka.tcp://otopcua@127.0.0.1:65501/system/receptionist"));
[Fact]
public void Recreating_a_client_in_one_pass_does_not_collide_on_the_actor_name()
{
var factory = new DefaultMeshClusterClientFactory();
var first = factory.Create(Sys, UnreachableContact);
Sys.Stop(first);
// Stop is asynchronous: `first`'s name is still reserved right here. Without the generation
// suffix this throws InvalidActorNameException — which is precisely what a contact-set
// change would do in production, where stop-then-recreate happens in one handler.
var second = factory.Create(Sys, UnreachableContact);
second.Path.Name.ShouldNotBe(first.Path.Name);
}
[Fact]
public void Every_incarnation_gets_a_distinct_name()
{
var factory = new DefaultMeshClusterClientFactory();
var names = Enumerable.Range(0, 5)
.Select(_ => factory.Create(Sys, UnreachableContact).Path.Name)
.ToList();
names.Distinct().Count().ShouldBe(5);
}
// Contact-set propagation is deliberately NOT asserted here: reading initial contacts back
// off a created ClusterClient is not possible, and asserting ClusterClientSettings directly
// would test Akka rather than this factory. It is covered where it can actually fail — the
// real two-ActorSystem boundary test, which only delivers if the contacts reached the client.
}
tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/Communication/MeshRouterDispatchTests.cs
+134
View File
@@ -0,0 +1,134 @@
using Akka.Actor;
using Akka.Cluster.Tools.PublishSubscribe;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Admin;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
using ZB.MOM.WW.OtOpcUa.Commons.Types;
using ZB.MOM.WW.OtOpcUa.ControlPlane.AdminOperations;
using ZB.MOM.WW.OtOpcUa.ControlPlane.Coordinators;
using ZB.MOM.WW.OtOpcUa.ControlPlane.Tests.Harness;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Tests.Communication;
/// <summary>
/// Proves the five central→node publish sites go through the transport router when one is
/// wired, and fall back to the mediator when it is not.
/// </summary>
/// <remarks>
/// The fallback is what keeps the ~12 pre-existing coordinator and admin-operations tests
/// meaningful: they construct these actors without a router and must still exercise the real
/// DistributedPubSub path, not a silently-disabled one.
/// </remarks>
public class MeshRouterDispatchTests : ControlPlaneActorTestBase
{
private const string DeploymentsTopic = "deployments";
[Fact]
public void Deploy_dispatch_goes_to_the_router_when_one_is_wired()
{
var router = CreateTestProbe();
var dbFactory = NewInMemoryDbFactory();
var coordinator = Sys.ActorOf(
ConfigPublishCoordinator.Props(dbFactory, applyDeadline: null, meshRouter: router.Ref));
var deploymentId = new DeploymentId(Guid.NewGuid());
coordinator.Tell(new Commons.Messages.Deploy.DispatchDeployment(
deploymentId, new RevisionHash("rev-1"), CorrelationId.NewId()));
var cmd = router.ExpectMsg<MeshCommand>(TimeSpan.FromSeconds(5));
cmd.Topic.ShouldBe(DeploymentsTopic);
cmd.Message.ShouldBeOfType<Commons.Messages.Deploy.DispatchDeployment>();
}
[Fact]
public void Deploy_dispatch_falls_back_to_the_mediator_when_no_router_is_wired()
{
var subscriber = CreateTestProbe();
DistributedPubSub.Get(Sys).Mediator.Tell(
new Subscribe(DeploymentsTopic, subscriber.Ref), subscriber.Ref);
subscriber.ExpectMsg<SubscribeAck>(TimeSpan.FromSeconds(5));
var coordinator = Sys.ActorOf(ConfigPublishCoordinator.Props(NewInMemoryDbFactory()));
var deploymentId = new DeploymentId(Guid.NewGuid());
coordinator.Tell(new Commons.Messages.Deploy.DispatchDeployment(
deploymentId, new RevisionHash("rev-1"), CorrelationId.NewId()));
subscriber.ExpectMsg<Commons.Messages.Deploy.DispatchDeployment>(TimeSpan.FromSeconds(5));
}
[Fact]
public void Restart_driver_goes_to_the_router_on_the_driver_control_topic()
{
var router = CreateTestProbe();
var admin = Sys.ActorOf(AdminOperationsActor.Props(
NewInMemoryDbFactory(),
CreateTestProbe().Ref,
Enumerable.Empty<IDriverProbe>(),
TagConfigValidationMode.Warn,
router.Ref));
admin.Tell(new RestartDriver("C1", "driver-1", "alice", Guid.NewGuid()));
var cmd = router.ExpectMsg<MeshCommand>(TimeSpan.FromSeconds(5));
cmd.Topic.ShouldBe(DriverControlTopic.Name);
cmd.Message.ShouldBeOfType<RestartDriver>();
}
[Fact]
public void Reconnect_driver_goes_to_the_router_on_the_driver_control_topic()
{
var router = CreateTestProbe();
var admin = Sys.ActorOf(AdminOperationsActor.Props(
NewInMemoryDbFactory(),
CreateTestProbe().Ref,
Enumerable.Empty<IDriverProbe>(),
TagConfigValidationMode.Warn,
router.Ref));
admin.Tell(new ReconnectDriver("C1", "driver-1", "alice", Guid.NewGuid()));
var cmd = router.ExpectMsg<MeshCommand>(TimeSpan.FromSeconds(5));
cmd.Topic.ShouldBe(DriverControlTopic.Name);
cmd.Message.ShouldBeOfType<ReconnectDriver>();
}
[Fact]
public void Alarm_acknowledge_goes_to_the_router_on_the_alarm_commands_topic()
{
var router = CreateTestProbe();
var admin = Sys.ActorOf(AdminOperationsActor.Props(
NewInMemoryDbFactory(),
CreateTestProbe().Ref,
Enumerable.Empty<IDriverProbe>(),
TagConfigValidationMode.Warn,
router.Ref));
admin.Tell(new AcknowledgeAlarmCommand("alarm-1", "alice", "c", CorrelationId.NewId()));
var cmd = router.ExpectMsg<MeshCommand>(TimeSpan.FromSeconds(5));
cmd.Topic.ShouldBe(AlarmCommandsTopic.Name);
cmd.Message.ShouldBeOfType<AlarmCommand>();
}
[Fact]
public void Alarm_shelve_goes_to_the_router_on_the_alarm_commands_topic()
{
var router = CreateTestProbe();
var admin = Sys.ActorOf(AdminOperationsActor.Props(
NewInMemoryDbFactory(),
CreateTestProbe().Ref,
Enumerable.Empty<IDriverProbe>(),
TagConfigValidationMode.Warn,
router.Ref));
admin.Tell(new ShelveAlarmCommand("alarm-1", "alice", ShelveKind.OneShot, null, "c", CorrelationId.NewId()));
var cmd = router.ExpectMsg<MeshCommand>(TimeSpan.FromSeconds(5));
cmd.Topic.ShouldBe(AlarmCommandsTopic.Name);
cmd.Message.ShouldBeOfType<AlarmCommand>();
}
}
tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/MeshClusterClientBoundaryTests.cs
+302
View File
@@ -0,0 +1,302 @@
using System.Collections.Immutable;
using Akka.Actor;
using Akka.Cluster;
using Akka.Cluster.Tools.Client;
using Akka.Configuration;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Admin;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Deploy;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
using ZB.MOM.WW.OtOpcUa.Commons.Types;
using ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
using ZB.MOM.WW.OtOpcUa.Runtime.Communication;
using AkkaCluster = Akka.Cluster.Cluster;
namespace ZB.MOM.WW.OtOpcUa.Host.IntegrationTests;
/// <summary>
/// Drives a real <c>ClusterClient</c> across two genuinely separate <see cref="ActorSystem"/>s —
/// the only test in the repo where the mesh boundary carries a message over the wire.
/// </summary>
/// <remarks>
/// <para>
/// The sister project substitutes an in-process relay that unwraps
/// <see cref="ClusterClient.Send"/> and forwards. That tests actor <i>logic</i> and cannot
/// fail for any transport reason: not a missing receptionist extension, not a service that
/// was never registered, not a malformed contact path, not a serialization break. Those are
/// precisely the Phase 2 failure modes, so they are what this test exercises.
/// </para>
/// <para>
/// <b>Two clusters, one ActorSystem name, joined ONLY by ClusterClient.</b> The shared name
/// is mandatory — Akka.Remote matches on the full address, so a contact path naming
/// <c>otopcua</c> cannot reach a system called anything else. Each system seeds itself and
/// neither lists the other, and
/// <see cref="The_two_systems_are_separate_clusters_joined_only_by_the_client"/> asserts the
/// separation outright: if they ever gossiped into one mesh, every delivery below could be
/// plain local messaging and the whole class would be testing nothing.
/// </para>
/// <para>
/// <b>Scope limit.</b> With exactly one node per side, "registered per node" and "registered
/// as a cluster singleton" are indistinguishable by construction, so this class does NOT
/// cover the per-node property whose structural assertions <c>MeshCommActorPathTests</c>
/// dropped. That property's discriminating coverage is live-gate step 8: stop the OLDEST
/// central and the survivor's comm actor must still receive a node ack, which a singleton
/// hosted on the stopped node cannot.
/// </para>
/// </remarks>
public sealed class MeshClusterClientBoundaryTests : IAsyncLifetime
{
private const string SharedSystemName = "otopcua";
private static readonly TimeSpan Arrival = TimeSpan.FromSeconds(30);
/// <summary>
/// How long the falsifiability control waits before concluding nothing arrived. Deliberately
/// shorter than <see cref="Arrival"/> but far longer than a successful delivery takes in
/// practice, so a pass means "the boundary is shut", not "the clock ran out first".
/// </summary>
private static readonly TimeSpan Silence = TimeSpan.FromSeconds(10);
/// <summary>
/// One factory for every client this class creates, because the generation counter that keeps
/// client actor names unique is <b>per factory instance</b>. A second factory on the same
/// system restarts at generation 1 and collides with the first factory's client — which is
/// exactly how this test first failed. Production has one instance per
/// <c>CentralCommunicationActor</c>, so the real code path is safe.
/// </summary>
private readonly DefaultMeshClusterClientFactory _clientFactory = new();
private ActorSystem _central = null!;
private ActorSystem _node = null!;
private IActorRef _nodeComm = null!;
private IActorRef _centralClient = null!;
private TaskCompletionSource<object> _centralInbox = null!;
/// <inheritdoc />
public async ValueTask InitializeAsync()
{
_central = StartSelfSeededSystem(withReceptionist: true);
_node = StartSelfSeededSystem(withReceptionist: true);
await Task.WhenAll(AwaitSingleMemberUpAsync(_central), AwaitSingleMemberUpAsync(_node));
// Central: the ack landing pad, at the path the node's comm actor sends to.
_centralInbox = NewInbox();
var centralComm = _central.ActorOf(
CaptureActor.Props(_centralInbox), MeshPaths.CentralCommunicationName);
ClusterClientReceptionist.Get(_central).RegisterService(centralComm);
// Node: the REAL comm actor, dialling central through the REAL production client factory.
var nodeToCentral = _clientFactory.Create(_node, ReceptionistOf(_central));
_nodeComm = _node.ActorOf(
NodeCommunicationActor.Props(nodeToCentral), MeshPaths.NodeCommunicationName);
ClusterClientReceptionist.Get(_node).RegisterService(_nodeComm);
_centralClient = _clientFactory.Create(_central, ReceptionistOf(_node));
}
/// <inheritdoc />
public async ValueTask DisposeAsync()
{
await ShutdownAsync(_central);
await ShutdownAsync(_node);
}
[Fact]
public void The_two_systems_are_separate_clusters_joined_only_by_the_client()
{
// The load-bearing precondition of every other test here. Two systems sharing a name in one
// process are one gossip accident away from being one cluster, at which point ClusterClient
// is no longer the thing under test.
AkkaCluster.Get(_central).State.Members.Count.ShouldBe(1,
"central must be its own single-member cluster, or the boundary is not being crossed");
AkkaCluster.Get(_node).State.Members.Count.ShouldBe(1,
"the node must be its own single-member cluster, or the boundary is not being crossed");
}
[Fact]
public async Task Central_send_to_all_reaches_the_node_comm_actor_and_lands_on_its_event_stream()
{
var inbox = NewInbox();
_node.EventStream.Subscribe(_node.ActorOf(CaptureActor.Props(inbox)), typeof(RestartDriver));
var command = new RestartDriver("C1", "driver-1", "alice", Guid.NewGuid());
var delivered = await SendUntilDeliveredAsync(
_centralClient,
() => new ClusterClient.SendToAll(MeshPaths.NodeCommunication, command),
inbox,
Arrival);
delivered.ShouldNotBeNull(
"SendToAll must reach the node's registered comm actor and be re-emitted node-locally; "
+ "this is the whole central→node leg of the mesh transport");
delivered.ShouldBeOfType<RestartDriver>().DriverInstanceId.ShouldBe("driver-1");
}
[Fact]
public async Task A_node_ack_reaches_central_through_the_node_comm_actors_own_client()
{
// Told directly rather than routed from central, so a failure here can only be the outbound
// leg. The inbound leg has its own test above.
var ack = new ApplyAck(
new DeploymentId(Guid.NewGuid()),
new NodeId("node-a"),
ApplyAckOutcome.Applied,
null,
CorrelationId.NewId());
var delivered = await SendUntilDeliveredAsync(
_nodeComm, () => ack, _centralInbox, Arrival);
delivered.ShouldNotBeNull(
"the node's ApplyAck must reach central's comm actor, or every deployment times out at "
+ "the apply deadline naming nodes that applied successfully");
delivered.ShouldBeOfType<ApplyAck>().NodeId.Value.ShouldBe("node-a");
}
[Fact]
public async Task Falsifiability_control_a_node_without_the_receptionist_extension_receives_nothing()
{
// Without this, the two tests above cannot distinguish "the ClusterClient boundary works"
// from "something else in the process delivered the message". Same message, same client
// factory, same paths, same comm actor — and it must NOT arrive.
//
// HONEST SCOPE: this removes the receptionist extension AND, unavoidably, the RegisterService
// call — resolving ClusterClientReceptionist.Get to register would materialise the very
// extension whose absence is the control. So it falsifies "delivery happens without a
// receptionist boundary", not the extension line in isolation. Verified falsifiable: giving
// this node the extension and the registration turns the test red, which also proves the send
// below is live rather than silently misaddressed.
var deaf = StartSelfSeededSystem(withReceptionist: false);
try
{
await AwaitSingleMemberUpAsync(deaf);
var inbox = NewInbox();
deaf.EventStream.Subscribe(deaf.ActorOf(CaptureActor.Props(inbox)), typeof(RestartDriver));
// Deliberately NOT ClusterClientReceptionist.Get(deaf) — resolving the extension would
// materialise the very receptionist whose absence is the control.
var comm = deaf.ActorOf(NodeCommunicationActor.Props(null), MeshPaths.NodeCommunicationName);
comm.ShouldNotBeNull();
var client = _clientFactory.Create(_central, ReceptionistOf(deaf));
var command = new RestartDriver("C1", "driver-1", "alice", Guid.NewGuid());
var deadline = DateTime.UtcNow + Silence;
while (DateTime.UtcNow < deadline && !inbox.Task.IsCompleted)
{
client.Tell(new ClusterClient.SendToAll(MeshPaths.NodeCommunication, command));
await Task.Delay(500, TestContext.Current.CancellationToken);
}
inbox.Task.IsCompleted.ShouldBeFalse(
"delivery survived removing the receptionist extension, so the two positive tests "
+ "above are not measuring the ClusterClient boundary at all");
}
finally
{
await ShutdownAsync(deaf);
}
}
/// <summary>
/// Starts a single-member cluster on a dynamic port, optionally without the receptionist.
/// </summary>
/// <param name="withReceptionist">
/// <see langword="false"/> drops the receptionist provider from <c>akka.extensions</c>,
/// leaving DistributedPubSub in place so only the one variable changes.
/// </param>
/// <returns>The started system.</returns>
private static ActorSystem StartSelfSeededSystem(bool withReceptionist)
{
var extensions = withReceptionist
? string.Empty
: "akka.extensions = [\"Akka.Cluster.Tools.PublishSubscribe.DistributedPubSubExtensionProvider, "
+ "Akka.Cluster.Tools\"]\n";
// Port 0 + a post-start self-Join, rather than a seed-nodes list: the address is not known
// until the transport binds, and hard-coding ports makes the test collide with itself under
// parallel execution.
var config = ConfigurationFactory.ParseString(
"akka.remote.dot-netty.tcp.hostname = \"127.0.0.1\"\n"
+ "akka.remote.dot-netty.tcp.public-hostname = \"127.0.0.1\"\n"
+ "akka.remote.dot-netty.tcp.port = 0\n"
+ "akka.cluster.seed-nodes = []\n"
+ "akka.loglevel = \"WARNING\"\n"
+ extensions)
.WithFallback(ZB.MOM.WW.OtOpcUa.Cluster.HoconLoader.LoadBaseConfig());
var system = ActorSystem.Create(SharedSystemName, config);
var cluster = AkkaCluster.Get(system);
cluster.Join(cluster.SelfAddress);
return system;
}
private static async Task AwaitSingleMemberUpAsync(ActorSystem system)
{
var cluster = AkkaCluster.Get(system);
var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(30);
while (DateTime.UtcNow < deadline)
{
if (cluster.State.Members.Any(m => m.Status == MemberStatus.Up)) return;
await Task.Delay(100, TestContext.Current.CancellationToken);
}
cluster.State.Members.Any(m => m.Status == MemberStatus.Up).ShouldBeTrue(
$"{cluster.SelfAddress} never reached Up as its own seed");
}
/// <summary>
/// Re-sends until the inbox completes or the deadline passes.
/// </summary>
/// <remarks>
/// A ClusterClient establishes contact asynchronously and <c>buffer-size = 0</c> means anything
/// sent before that lands is dropped on the floor — by design. A single send would therefore
/// race the handshake and fail intermittently for a reason that is not a defect. Re-sending
/// weakens nothing: the control test re-sends on exactly the same schedule and must still
/// receive nothing.
/// </remarks>
/// <param name="target">Actor to send to.</param>
/// <param name="message">Builds the message for each attempt.</param>
/// <param name="inbox">Completion source the receiving capture actor resolves.</param>
/// <param name="timeout">How long to keep trying.</param>
/// <returns>The delivered message, or <see langword="null"/> if none arrived.</returns>
private static async Task<object?> SendUntilDeliveredAsync(
IActorRef target,
Func<object> message,
TaskCompletionSource<object> inbox,
TimeSpan timeout)
{
var deadline = DateTime.UtcNow + timeout;
while (DateTime.UtcNow < deadline)
{
if (inbox.Task.IsCompleted) return await inbox.Task;
target.Tell(message());
await Task.Delay(250, TestContext.Current.CancellationToken);
}
return inbox.Task.IsCompleted ? await inbox.Task : null;
}
private static TaskCompletionSource<object> NewInbox() =>
new(TaskCreationOptions.RunContinuationsAsynchronously);
private static ImmutableHashSet<ActorPath> ReceptionistOf(ActorSystem system) =>
ImmutableHashSet.Create<ActorPath>(
new RootActorPath(AkkaCluster.Get(system).SelfAddress) / "system" / "receptionist");
private static async Task ShutdownAsync(ActorSystem? system)
{
if (system is null) return;
await system.Terminate().WaitAsync(TimeSpan.FromSeconds(15));
}
/// <summary>Completes an inbox with the first message it receives.</summary>
private sealed class CaptureActor : ReceiveActor
{
public static Props Props(TaskCompletionSource<object> sink) =>
Akka.Actor.Props.Create(() => new CaptureActor(sink));
public CaptureActor(TaskCompletionSource<object> sink) => ReceiveAny(m => sink.TrySetResult(m));
}
}
tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/MeshCommActorPathTests.cs
+66
View File
@@ -0,0 +1,66 @@
using Akka.Actor;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
namespace ZB.MOM.WW.OtOpcUa.Host.IntegrationTests;
/// <summary>
/// Pins the two mesh comm-actor paths against a <b>really booted</b> host.
/// </summary>
/// <remarks>
/// <para>
/// <c>/user/central-communication</c> and <c>/user/node-communication</c> are the wire
/// contract of the central↔node boundary: central sends to a path string, and the node
/// registers under one. Nothing else in the codebase asserts they agree. Renaming either
/// actor would compile, pass every unit test, deploy — and deliver nothing, with no error
/// on either side, because a ClusterClient send to an unregistered path is simply dropped.
/// </para>
/// <para>
/// Booting the real host is the point. A test that spawned the actors itself would pin the
/// constant against itself and prove nothing about the wiring in
/// <c>WithOtOpcUaControlPlaneSingletons</c> / <c>WithOtOpcUaRuntimeActors</c>.
/// </para>
/// </remarks>
public class MeshCommActorPathTests
{
private static async Task AssertActorExistsAsync(ActorSystem system, string path)
{
var identity = await system.ActorSelection(path)
.Ask<ActorIdentity>(new Identify(path), TimeSpan.FromSeconds(10));
identity.Subject.ShouldNotBeNull(
$"{path} is the ClusterClient wire contract for the central↔node boundary; nothing else "
+ "pins it, and a mismatch delivers nothing without erroring on either side");
}
[Fact]
public async Task Both_mesh_comm_actors_exist_at_their_contract_paths()
{
await using var harness = await TwoNodeClusterHarness.StartAsync();
// Harness nodes carry admin+driver, so BOTH comm actors live on each of them.
await AssertActorExistsAsync(harness.NodeASystem, MeshPaths.CentralCommunication);
await AssertActorExistsAsync(harness.NodeASystem, MeshPaths.NodeCommunication);
await AssertActorExistsAsync(harness.NodeBSystem, MeshPaths.CentralCommunication);
await AssertActorExistsAsync(harness.NodeBSystem, MeshPaths.NodeCommunication);
}
// NOT TESTED HERE: that the comm actors are registered per node rather than as cluster
// singletons. Two candidate discriminators were tried and BOTH were invalid:
//
// * "the path resolves on both nodes" — a ClusterSingletonManager is also created on every
// node in the role, so this passes under either shape.
// * "no /singleton child exists" — measured against the KNOWN singleton /user/config-publish,
// that child is null there too, so Akka.Hosting does not lay singletons out this way and
// the assertion proves nothing.
//
// A sabotage that re-registered CentralCommunicationActor via WithSingleton left both green,
// which is what exposed them. Rather than ship an assertion I could not make fail, the property
// is left to live-gate step 8 (stop the OLDEST central; the survivor's comm actor must still
// receive a node ack, which a singleton hosted on the stopped node cannot).
//
// NOT MeshClusterClientBoundaryTests: that class proves a send only arrives if the target is
// registered with the receptionist, which is necessary but not discriminating — with one node
// per side, per-node and singleton registration look identical.
}
tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Communication/EventStreamCommandDeliveryTests.cs
+161
View File
@@ -0,0 +1,161 @@
using Akka.Actor;
using Akka.TestKit.Xunit2;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Deploy;
using ZB.MOM.WW.OtOpcUa.Commons.Types;
using ZB.MOM.WW.OtOpcUa.Runtime.Communication;
using ZB.MOM.WW.OtOpcUa.Runtime.Drivers;
using Serilog;
using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
using ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms;
using ZB.MOM.WW.OtOpcUa.Runtime.ScriptedAlarms;
using ZB.MOM.WW.OtOpcUa.Runtime.Tests.Harness;
using ZB.MOM.WW.OtOpcUa.Core.Scripting;
namespace ZB.MOM.WW.OtOpcUa.Runtime.Tests.Communication;
/// <summary>
/// Proves the Phase 2 node-side leg joins up: a command handed to
/// <see cref="NodeCommunicationActor"/> reaches the <b>real</b> <see cref="DriverHostActor"/>,
/// not just a test probe.
/// </summary>
/// <remarks>
/// <para>
/// The unit tests either side of this seam both pass with the wiring broken.
/// <c>NodeCommunicationActorTests</c> asserts a probe subscribed to the EventStream receives
/// the message; the driver-host tests drive the actor by direct <c>Tell</c>. Neither notices
/// if <c>DriverHostActor.PreStart</c> never subscribes. That gap is why this file exists.
/// </para>
/// <para>
/// <b>Ordering matters, and it bit this test first time out.</b> <c>ActorOf</c> returns
/// before <c>PreStart</c> has run, and an <c>EventStream.Publish</c> with no subscriber is
/// dropped silently — there is no buffering and no dead-letter. Relaying immediately after
/// spawning therefore fails intermittently. The warm-up below is not ceremony: it is the
/// only available proof that <c>PreStart</c> has completed. (In production the driver host
/// is spawned at startup, long before any deployment, so the same race does not arise.)
/// </para>
/// </remarks>
public class EventStreamCommandDeliveryTests : RuntimeActorTestBase
{
private static DispatchDeployment NewDispatch() => new(
new DeploymentId(Guid.NewGuid()),
new RevisionHash("rev-does-not-exist"),
new CorrelationId(Guid.NewGuid()));
[Fact]
public void A_dispatch_relayed_by_the_comm_actor_reaches_the_real_driver_host()
{
var ackRouter = CreateTestProbe();
var driverHost = Sys.ActorOf(
DriverHostActor.Props(NewInMemoryDbFactory(), new NodeId("127.0.0.1:4053"), ackRouter: ackRouter.Ref),
"driver-host");
var comm = Sys.ActorOf(NodeCommunicationActor.Props(centralClient: null), "node-communication");
// Warm-up: a direct Tell whose ack proves PreStart has completed, and with it the
// EventStream subscription. Without this the relay below races the subscription.
var warmup = NewDispatch();
driverHost.Tell(warmup);
ackRouter.ExpectMsg<ApplyAck>(TimeSpan.FromSeconds(15)).DeploymentId.ShouldBe(warmup.DeploymentId);
// Now the real assertion — relayed exactly as central would send it. The comm actor only
// publishes node-locally, so the host must be subscribed for this to arrive at all.
var relayed = NewDispatch();
comm.Tell(relayed);
// The deployment id does not exist, so the host answers Failed. That is the stronger
// assertion, not a weaker one: an ack at all proves the command was received and handled,
// and only PreStart's EventStream subscription can have delivered it.
ackRouter.ExpectMsg<ApplyAck>(TimeSpan.FromSeconds(15)).DeploymentId.ShouldBe(relayed.DeploymentId);
}
[Fact]
public void CONTROL_the_driver_host_acks_a_direct_tell()
{
// Falsifiability control. If the test above fails, this one says which half broke: green
// here means the host is healthy and the EventStream relay is at fault; red here means the
// host would not have acked either way and the relay is not implicated.
var ackRouter = CreateTestProbe();
var driverHost = Sys.ActorOf(
DriverHostActor.Props(NewInMemoryDbFactory(), new NodeId("127.0.0.1:4053"), ackRouter: ackRouter.Ref),
"driver-host-control");
var dispatch = NewDispatch();
driverHost.Tell(dispatch);
ackRouter.ExpectMsg<ApplyAck>(TimeSpan.FromSeconds(15)).DeploymentId.ShouldBe(dispatch.DeploymentId);
}
}
/// <summary>
/// The alarm leg of the same wiring proof, in its own ActorSystem because the shared
/// <see cref="RuntimeActorTestBase"/> harness pins <c>loglevel = WARNING</c> and the only
/// observable signal <c>ScriptedAlarmHostActor</c> gives for an unowned alarm is a Debug line.
/// </summary>
public class AlarmCommandEventStreamDeliveryTests : TestKit
{
private static string DebugLevelHocon => @"
akka {
loglevel = ""DEBUG""
extensions = [
""Akka.Cluster.Tools.PublishSubscribe.DistributedPubSubExtensionProvider, Akka.Cluster.Tools""
]
actor {
provider = ""Akka.Cluster.ClusterActorRefProvider, Akka.Cluster""
}
remote.dot-netty.tcp {
hostname = ""127.0.0.1""
port = 0
}
cluster {
seed-nodes = []
roles = [""driver""]
min-nr-of-members = 1
run-coordinated-shutdown-when-down = off
}
}";
public AlarmCommandEventStreamDeliveryTests()
: base(DebugLevelHocon)
{
var cluster = Akka.Cluster.Cluster.Get(Sys);
cluster.Join(cluster.SelfAddress);
AwaitCondition(
() => cluster.State.Members.Any(m => m.Status == Akka.Cluster.MemberStatus.Up),
TimeSpan.FromSeconds(5));
}
[Fact]
public void An_alarm_command_relayed_by_the_comm_actor_reaches_the_real_scripted_alarm_host()
{
// Needed for the same reason as the deploy leg: the comm actor's unit test only proves a
// probe receives it, and the alarm-host tests drive the actor by direct Tell. Neither
// notices a missing PreStart subscription.
var upstream = new DependencyMuxTagUpstreamSource();
var logger = new LoggerConfiguration().CreateLogger();
var engine = new ScriptedAlarmEngine(
upstream, new InMemoryAlarmStateStore(), new ScriptLoggerFactory(logger), logger);
var host = Sys.ActorOf(
ScriptedAlarmHostActor.Props(
CreateTestProbe().Ref, CreateTestProbe().Ref, upstream, engine, new NodeId("node-A")),
"scripted-alarm-host");
var comm = Sys.ActorOf(NodeCommunicationActor.Props(centralClient: null), "node-communication");
var cmd = new AlarmCommand("alarm-unowned", "Acknowledge", "alice", "c", null);
// Warm-up, OUTSIDE the real assertion: a direct Tell whose debug line proves PreStart has
// completed and the EventStream subscription is registered. An earlier draft did the direct
// Tell INSIDE the assertion window alongside the relay, which made the test vacuous — the
// direct Tell alone produced the log the assertion was waiting for.
EventFilter.Debug(contains: "ignoring AlarmCommand")
.ExpectOne(TimeSpan.FromSeconds(15), () => host.Tell(cmd));
// The real assertion: relayed ONLY through the comm actor, which publishes node-locally.
// The host owns no alarms, so it logs and stops — still a positive delivery signal, because
// that line is only reached if the message arrived.
EventFilter.Debug(contains: "ignoring AlarmCommand")
.ExpectOne(TimeSpan.FromSeconds(15), () => comm.Tell(cmd));
}
}
tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Communication/NodeCommunicationActorTests.cs
+130
View File
@@ -0,0 +1,130 @@
using Akka.Actor;
using Akka.Cluster.Tools.Client;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Admin;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Deploy;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
using ZB.MOM.WW.OtOpcUa.Commons.Types;
using ZB.MOM.WW.OtOpcUa.Runtime.Communication;
using ZB.MOM.WW.OtOpcUa.Runtime.Tests.Harness;
namespace ZB.MOM.WW.OtOpcUa.Runtime.Tests.Communication;
public class NodeCommunicationActorTests : RuntimeActorTestBase
{
private static DispatchDeployment NewDispatch() => new(
new DeploymentId(Guid.NewGuid()),
new RevisionHash("rev-1"),
new CorrelationId(Guid.NewGuid()));
[Fact]
public void DispatchDeployment_from_central_lands_on_the_node_local_event_stream()
{
var subscriber = CreateTestProbe();
Sys.EventStream.Subscribe(subscriber.Ref, typeof(DispatchDeployment));
var actor = Sys.ActorOf(NodeCommunicationActor.Props(centralClient: null));
var dispatch = NewDispatch();
actor.Tell(dispatch);
subscriber.ExpectMsg<DispatchDeployment>(TimeSpan.FromSeconds(5)).ShouldBe(dispatch);
}
[Fact]
public void A_command_is_published_exactly_once()
{
// The reason inbound commands do NOT go back onto their DPS topic: central SendToAll-s to
// every node's comm actor, so a mesh-wide republish would deliver N copies to every
// subscriber. One publish per inbound command is the whole point.
var subscriber = CreateTestProbe();
Sys.EventStream.Subscribe(subscriber.Ref, typeof(DispatchDeployment));
var actor = Sys.ActorOf(NodeCommunicationActor.Props(centralClient: null));
actor.Tell(NewDispatch());
subscriber.ExpectMsg<DispatchDeployment>(TimeSpan.FromSeconds(5));
subscriber.ExpectNoMsg(TimeSpan.FromMilliseconds(300));
}
[Fact]
public void RestartDriver_from_central_lands_on_the_event_stream()
{
var subscriber = CreateTestProbe();
Sys.EventStream.Subscribe(subscriber.Ref, typeof(RestartDriver));
var actor = Sys.ActorOf(NodeCommunicationActor.Props(centralClient: null));
var msg = new RestartDriver("C1", "driver-1", "alice", Guid.NewGuid());
actor.Tell(msg);
subscriber.ExpectMsg<RestartDriver>(TimeSpan.FromSeconds(5)).ShouldBe(msg);
}
[Fact]
public void ReconnectDriver_from_central_lands_on_the_event_stream()
{
var subscriber = CreateTestProbe();
Sys.EventStream.Subscribe(subscriber.Ref, typeof(ReconnectDriver));
var actor = Sys.ActorOf(NodeCommunicationActor.Props(centralClient: null));
var msg = new ReconnectDriver("C1", "driver-1", "alice", Guid.NewGuid());
actor.Tell(msg);
subscriber.ExpectMsg<ReconnectDriver>(TimeSpan.FromSeconds(5)).ShouldBe(msg);
}
[Fact]
public void AlarmCommand_from_central_lands_on_the_event_stream()
{
// ScriptedAlarmHostActor is a CHILD of DriverHostActor with no registry key, so there is no
// ref to hand this actor at wiring time — the EventStream is what makes the alarm-command
// leg possible without a registration handshake.
var subscriber = CreateTestProbe();
Sys.EventStream.Subscribe(subscriber.Ref, typeof(AlarmCommand));
var actor = Sys.ActorOf(NodeCommunicationActor.Props(centralClient: null));
var msg = new AlarmCommand("alarm-1", "Acknowledge", "alice", "comment", null);
actor.Tell(msg);
subscriber.ExpectMsg<AlarmCommand>(TimeSpan.FromSeconds(5)).ShouldBe(msg);
}
[Fact]
public void ApplyAck_goes_to_central_as_a_Send_not_a_SendToAll()
{
// Send, not SendToAll: the deploy coordinator is one singleton behind the central comm
// actor's proxy. SendToAll would deliver one ack per central node and the coordinator would
// count this node twice.
var client = CreateTestProbe();
var actor = Sys.ActorOf(NodeCommunicationActor.Props(client.Ref));
var ack = new ApplyAck(
new DeploymentId(Guid.NewGuid()),
new NodeId("host-a:4053"),
ApplyAckOutcome.Applied,
null,
new CorrelationId(Guid.NewGuid()));
actor.Tell(ack);
var sent = client.ExpectMsg<ClusterClient.Send>(TimeSpan.FromSeconds(5));
sent.Path.ShouldBe(MeshPaths.CentralCommunication);
sent.Message.ShouldBe(ack);
}
[Fact]
public void An_ack_with_no_central_client_is_dropped_with_a_warning()
{
var actor = Sys.ActorOf(NodeCommunicationActor.Props(centralClient: null));
var ack = new ApplyAck(
new DeploymentId(Guid.NewGuid()),
new NodeId("host-a:4053"),
ApplyAckOutcome.Applied,
null,
new CorrelationId(Guid.NewGuid()));
EventFilter.Warning(contains: "dropping ApplyAck").ExpectOne(
TimeSpan.FromSeconds(5), () => actor.Tell(ack));
}
}
tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Drivers/DriverHostActorBootFromCacheTests.cs
+8 -8
View File
@@ -41,7 +41,7 @@ public sealed class DriverHostActorBootFromCacheTests : RuntimeActorTestBase
var cache = new StubArtifactCache(cached);
var actor = Sys.ActorOf(DriverHostActor.Props(
new ThrowingDbFactory(), TestNode, coordinator: null,
new ThrowingDbFactory(), TestNode, ackRouter: null,
localRoles: new HashSet<string> { "driver" },
deploymentArtifactCache: cache));
@@ -70,7 +70,7 @@ public sealed class DriverHostActorBootFromCacheTests : RuntimeActorTestBase
var publishProbe = CreateTestProbe();
Sys.ActorOf(DriverHostActor.Props(
new ThrowingDbFactory(), TestNode, coordinator: null,
new ThrowingDbFactory(), TestNode, ackRouter: null,
localRoles: new HashSet<string> { "driver" },
opcUaPublishActor: publishProbe.Ref,
deploymentArtifactCache: new StubArtifactCache(cached)));
@@ -85,7 +85,7 @@ public sealed class DriverHostActorBootFromCacheTests : RuntimeActorTestBase
{
// The pre-cache behaviour, unchanged. A cache miss must not invent a configuration.
var actor = Sys.ActorOf(DriverHostActor.Props(
new ThrowingDbFactory(), TestNode, coordinator: null,
new ThrowingDbFactory(), TestNode, ackRouter: null,
localRoles: new HashSet<string> { "driver" },
deploymentArtifactCache: new StubArtifactCache(current: null)));
@@ -100,7 +100,7 @@ public sealed class DriverHostActorBootFromCacheTests : RuntimeActorTestBase
{
// Admin-only graphs and older harnesses pass null; behaviour is identical to a cache miss.
var actor = Sys.ActorOf(DriverHostActor.Props(
new ThrowingDbFactory(), TestNode, coordinator: null,
new ThrowingDbFactory(), TestNode, ackRouter: null,
localRoles: new HashSet<string> { "driver" }));
var snapshot = AskDiagnostics(actor);
@@ -118,7 +118,7 @@ public sealed class DriverHostActorBootFromCacheTests : RuntimeActorTestBase
DeploymentId.NewId().ToString(), CachedRev.Value, EmptyArtifact(), DateTimeOffset.UtcNow));
var actor = Sys.ActorOf(DriverHostActor.Props(
NewInMemoryDbFactory(), TestNode, coordinator: null,
NewInMemoryDbFactory(), TestNode, ackRouter: null,
localRoles: new HashSet<string> { "driver" },
deploymentArtifactCache: cache));
@@ -140,7 +140,7 @@ public sealed class DriverHostActorBootFromCacheTests : RuntimeActorTestBase
deploymentId.ToString(), CachedRev.Value, EmptyArtifact(), DateTimeOffset.UtcNow));
var actor = Sys.ActorOf(DriverHostActor.Props(
db, TestNode, coordinator: null,
db, TestNode, ackRouter: null,
localRoles: new HashSet<string> { "driver" },
deploymentArtifactCache: cache));
@@ -156,7 +156,7 @@ public sealed class DriverHostActorBootFromCacheTests : RuntimeActorTestBase
{
// A cache fault must leave the node exactly where it would have been without a cache.
var actor = Sys.ActorOf(DriverHostActor.Props(
new ThrowingDbFactory(), TestNode, coordinator: null,
new ThrowingDbFactory(), TestNode, ackRouter: null,
localRoles: new HashSet<string> { "driver" },
deploymentArtifactCache: new ThrowingReadCache()));
@@ -183,7 +183,7 @@ public sealed class DriverHostActorBootFromCacheTests : RuntimeActorTestBase
var cache = new StubArtifactCache(current: null);
var actor = Sys.ActorOf(DriverHostActor.Props(
db, TestNode, coordinator: null,
db, TestNode, ackRouter: null,
localRoles: new HashSet<string> { "driver" },
deploymentArtifactCache: cache));
tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Drivers/DriverHostActorRoleViewTests.cs
+1 -1
View File
@@ -183,7 +183,7 @@ public sealed class DriverHostActorRoleViewTests : RuntimeActorTestBase
Sys.ActorOf(DriverHostActor.Props(
NewInMemoryDbFactory(),
TestNode,
coordinator: null,
ackRouter: null,
driverMemberCountProvider: () => driverMemberCount,
redundancyRoleView: view,
replicationPeerHost: PeerHost));
tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/VirtualTags/DependencyMuxActorTests.cs
+1 -1
View File
@@ -139,7 +139,7 @@ public sealed class DependencyMuxActorTests : RuntimeActorTestBase
var hostActor = Sys.ActorOf(DriverHostActor.Props(
dbFactory: NewInMemoryDbFactory(),
localNode: ZB.MOM.WW.OtOpcUa.Commons.Types.NodeId.Parse("host-1"),
coordinator: hostProbe.Ref,
ackRouter: hostProbe.Ref,
dependencyMux: mux));
// Tell the host an AttributeValuePublished — it should fan out to the mux + subscriber.