docs(mesh): MeshTransport config, Phase 2 as shipped, design-doc correction
Rig: all six docker-dev nodes carry MeshTransport__Mode plus both central
contact points. Mode is "${OTOPCUA_MESH_MODE:-Dps}", so the rig still comes up
on the transport it has always used and the live gate can flip the whole fleet
at `docker compose up` without a compose edit or a rebuild.
Docs: a MeshTransport section in Configuration.md (including why contact points
are addresses only, why the contact set does not scope delivery, and why
buffer-size=0 is a behaviour decision rather than a tuning knob), a "Command
transport" section in Redundancy.md contrasting the two modes, and a summary
beside the Redundancy notes in CLAUDE.md.
Corrects the design doc's claim that ScadaBridge replies with a typed failure to
every unhandled message. Neither of their comm actors has a ReceiveAny or an
Unhandled override; the idiom is per message type and fires on a missing
registration, and an unknown type dead-letters there as it does anywhere else.
Records the exit-gate deviation in the program plan: "an Ask timing out cleanly
against a stopped node" cannot be run, because no Ask crosses this boundary. The
honest equivalents are a stopped node failing the deploy at the apply deadline
and an unreachable contact dropping the command with a Warning.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
@@ -228,7 +228,44 @@ The server supports non-transparent warm/hot redundancy via the `Redundancy` sec
|
|||||||
|
|
||||||
Assert the **effective** cluster config off a running `ActorSystem` (as `SplitBrainResolverActivationTests` does), never the typed options or the akka.conf text — several HOCON fragments compete and the losing one still reads correctly in the file.
|
Assert the **effective** cluster config off a running `ActorSystem` (as `SplitBrainResolverActivationTests` does), never the typed options or the akka.conf text — several HOCON fragments compete and the losing one still reads correctly in the file.
|
||||||
|
|
||||||
**Still per-Akka-cluster, not per application `Cluster`.** One Primary is elected across the whole Akka mesh, so a fleet running several application clusters in one mesh (what docker-dev models) gets one Primary among all driver nodes. The fix is the per-cluster mesh workstream: `docs/plans/2026-07-21-per-cluster-mesh-design.md` (Phases 0a/0b/1 done; 2–7 not started).
|
**Still per-Akka-cluster, not per application `Cluster`.** One Primary is elected across the whole Akka mesh, so a fleet running several application clusters in one mesh (what docker-dev models) gets one Primary among all driver nodes. The fix is the per-cluster mesh workstream: `docs/plans/2026-07-21-per-cluster-mesh-design.md` (Phases 0a/0b/1 done; 2 code-complete behind a dark switch; 3–7 not started).
|
||||||
|
|
||||||
|
## Mesh command transport (`MeshTransport`)
|
||||||
|
|
||||||
|
Per-cluster mesh **Phase 2** added a second transport for the three central→node command channels
|
||||||
|
(`deployments`, `driver-control`, `alarm-commands`) and the `deployment-acks` reply, selected by
|
||||||
|
`MeshTransport:Mode`: `Dps` (the DistributedPubSub path this repo has always used, **the default**)
|
||||||
|
or `ClusterClient`, which does **not** require central and the node to share a gossip ring — the
|
||||||
|
prerequisite for splitting the fleet into one mesh per application `Cluster`. Both `CentralCommunicationActor`
|
||||||
|
(`/user/central-communication`) and `NodeCommunicationActor` (`/user/node-communication`) are spawned
|
||||||
|
and registered with the `ClusterClientReceptionist` in **both** modes, so flipping the flag is a config
|
||||||
|
change, not a redeploy. Things worth knowing before touching it:
|
||||||
|
|
||||||
|
- **`SendToAll`, never `Send`.** Today's DPS publish reaches every `DriverHostActor`; there is no
|
||||||
|
ClusterId or node filter on the node side at all (scoping happens later, inside the artifact via
|
||||||
|
`DeploymentArtifact.ResolveClusterScope`). `ClusterClient.Send` delivers to exactly **one** registered
|
||||||
|
actor — it would deploy to a single node while every other node silently kept its old configuration,
|
||||||
|
and the deployment could still seal green.
|
||||||
|
- **Exactly ONE fleet-wide ClusterClient while the fleet is one mesh.** A receptionist serves its whole
|
||||||
|
cluster, so `SendToAll` reaches every registered node-comm actor regardless of which node's address was
|
||||||
|
dialled. One client per application `Cluster` — the obviously-right-looking shape, and what Phase 6
|
||||||
|
wants — would fan each command out once per cluster (N× duplicate dispatch *and* N× duplicate ack).
|
||||||
|
Marked `TODO(Phase 6)` in `CentralCommunicationActor.RebuildClient`. **Corollary:** the contact set does
|
||||||
|
not scope delivery; excluding a `MaintenanceMode` node from the contacts does not stop it receiving.
|
||||||
|
- **Inbound commands re-emit on the node's `EventStream`, not on their DPS topic** — DPS is mesh-wide and
|
||||||
|
central `SendToAll`s to every node, so a DPS re-publish would deliver N copies to every subscriber.
|
||||||
|
Node-side subscribers (`DriverHostActor`, `ScriptedAlarmHostActor`) accept both.
|
||||||
|
- **Comm actors are per-node, NOT singletons.** A client rotates across contact points and must find a live
|
||||||
|
comm actor at whichever node answers; `akka.cluster.client.receptionist.role` is empty for the same reason.
|
||||||
|
- **`buffer-size = 0` is a behaviour decision, not a tuning knob.** Akka's default buffers 1000 and replays
|
||||||
|
on reconnect, which would apply a `DispatchDeployment` for a deployment the coordinator already sealed
|
||||||
|
`TimedOut` — the node would run a config the DB records as failed. Do not raise it to quiet a flaky test.
|
||||||
|
|
||||||
|
Contact points are node **addresses only** (`akka.tcp://<system>@<host>:<port>`); `/system/receptionist` is
|
||||||
|
appended at construction and `MeshTransportOptionsValidator` refuses to start a node whose contact carries an
|
||||||
|
actor-path suffix. `redundancy-state` stays on DPS in both modes (bidirectional, and built from
|
||||||
|
`Cluster.State`). See `docs/Configuration.md` §`MeshTransport`, `docs/Redundancy.md` §"Command transport",
|
||||||
|
and `docs/plans/2026-07-22-mesh-phase2-clusterclient-transport.md`.
|
||||||
|
|
||||||
**Phase 1 landed 2026-07-22 and changed a deploy-path behaviour.** `ConfigPublishCoordinator` sources its expected-ack set from **enabled `ClusterNode` rows**, not from `Akka.Cluster.State.Members` filtered by the `driver` role — central must be able to name a deployment's nodes without sharing a gossip ring with them. Consequences worth knowing before touching deploys:
|
**Phase 1 landed 2026-07-22 and changed a deploy-path behaviour.** `ConfigPublishCoordinator` sources its expected-ack set from **enabled `ClusterNode` rows**, not from `Akka.Cluster.State.Members` filtered by the `driver` role — central must be able to name a deployment's nodes without sharing a gossip ring with them. Consequences worth knowing before touching deploys:
|
||||||
|
|
||||||
|
|||||||
@@ -190,6 +190,19 @@ services:
|
|||||||
Cluster__SeedNodes__1: "akka.tcp://otopcua@central-2:4053"
|
Cluster__SeedNodes__1: "akka.tcp://otopcua@central-2:4053"
|
||||||
Cluster__Roles__0: "admin"
|
Cluster__Roles__0: "admin"
|
||||||
Cluster__Roles__1: "driver"
|
Cluster__Roles__1: "driver"
|
||||||
|
# Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps",
|
||||||
|
# the transport this repo has always used, and every node keeps its ClusterClient wiring built
|
||||||
|
# and its comm actor registered with the receptionist either way. Flip the whole rig with
|
||||||
|
# OTOPCUA_MESH_MODE=ClusterClient at `docker compose up` — no compose edit, and no rebuild.
|
||||||
|
#
|
||||||
|
# Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
|
||||||
|
# time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
|
||||||
|
# actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
|
||||||
|
# silence. Both central nodes are listed so a driver node's client can rotate to whichever one
|
||||||
|
# is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
|
||||||
|
MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Dps}"
|
||||||
|
MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
|
||||||
|
MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
|
||||||
Security__Jwt__SigningKey: "docker-dev-signing-key-with-at-least-32-bytes-of-utf8-content-12345"
|
Security__Jwt__SigningKey: "docker-dev-signing-key-with-at-least-32-bytes-of-utf8-content-12345"
|
||||||
Security__Jwt__Issuer: "otopcua-dev"
|
Security__Jwt__Issuer: "otopcua-dev"
|
||||||
Security__Jwt__Audience: "otopcua-dev"
|
Security__Jwt__Audience: "otopcua-dev"
|
||||||
@@ -263,6 +276,19 @@ services:
|
|||||||
Cluster__SeedNodes__1: "akka.tcp://otopcua@central-1:4053"
|
Cluster__SeedNodes__1: "akka.tcp://otopcua@central-1:4053"
|
||||||
Cluster__Roles__0: "admin"
|
Cluster__Roles__0: "admin"
|
||||||
Cluster__Roles__1: "driver"
|
Cluster__Roles__1: "driver"
|
||||||
|
# Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps",
|
||||||
|
# the transport this repo has always used, and every node keeps its ClusterClient wiring built
|
||||||
|
# and its comm actor registered with the receptionist either way. Flip the whole rig with
|
||||||
|
# OTOPCUA_MESH_MODE=ClusterClient at `docker compose up` — no compose edit, and no rebuild.
|
||||||
|
#
|
||||||
|
# Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
|
||||||
|
# time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
|
||||||
|
# actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
|
||||||
|
# silence. Both central nodes are listed so a driver node's client can rotate to whichever one
|
||||||
|
# is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
|
||||||
|
MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Dps}"
|
||||||
|
MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
|
||||||
|
MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
|
||||||
Security__Jwt__SigningKey: "docker-dev-signing-key-with-at-least-32-bytes-of-utf8-content-12345"
|
Security__Jwt__SigningKey: "docker-dev-signing-key-with-at-least-32-bytes-of-utf8-content-12345"
|
||||||
Security__Jwt__Issuer: "otopcua-dev"
|
Security__Jwt__Issuer: "otopcua-dev"
|
||||||
Security__Jwt__Audience: "otopcua-dev"
|
Security__Jwt__Audience: "otopcua-dev"
|
||||||
@@ -329,6 +355,19 @@ services:
|
|||||||
# is in its own seed list), so these configs are exempt rather than broken.
|
# is in its own seed list), so these configs are exempt rather than broken.
|
||||||
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
|
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
|
||||||
Cluster__Roles__0: "driver"
|
Cluster__Roles__0: "driver"
|
||||||
|
# Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps",
|
||||||
|
# the transport this repo has always used, and every node keeps its ClusterClient wiring built
|
||||||
|
# and its comm actor registered with the receptionist either way. Flip the whole rig with
|
||||||
|
# OTOPCUA_MESH_MODE=ClusterClient at `docker compose up` — no compose edit, and no rebuild.
|
||||||
|
#
|
||||||
|
# Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
|
||||||
|
# time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
|
||||||
|
# actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
|
||||||
|
# silence. Both central nodes are listed so a driver node's client can rotate to whichever one
|
||||||
|
# is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
|
||||||
|
MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Dps}"
|
||||||
|
MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
|
||||||
|
MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
|
||||||
<<: *secrets-env
|
<<: *secrets-env
|
||||||
# Quiet EF/AspNetCore SQL flood — see central-1 (Serilog override). mem_limit/
|
# Quiet EF/AspNetCore SQL flood — see central-1 (Serilog override). mem_limit/
|
||||||
# mem_reservation are inherited from the *otopcua-host anchor.
|
# mem_reservation are inherited from the *otopcua-host anchor.
|
||||||
@@ -395,6 +434,19 @@ services:
|
|||||||
Cluster__PublicHostname: "site-a-2"
|
Cluster__PublicHostname: "site-a-2"
|
||||||
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
|
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
|
||||||
Cluster__Roles__0: "driver"
|
Cluster__Roles__0: "driver"
|
||||||
|
# Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps",
|
||||||
|
# the transport this repo has always used, and every node keeps its ClusterClient wiring built
|
||||||
|
# and its comm actor registered with the receptionist either way. Flip the whole rig with
|
||||||
|
# OTOPCUA_MESH_MODE=ClusterClient at `docker compose up` — no compose edit, and no rebuild.
|
||||||
|
#
|
||||||
|
# Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
|
||||||
|
# time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
|
||||||
|
# actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
|
||||||
|
# silence. Both central nodes are listed so a driver node's client can rotate to whichever one
|
||||||
|
# is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
|
||||||
|
MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Dps}"
|
||||||
|
MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
|
||||||
|
MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
|
||||||
<<: *secrets-env
|
<<: *secrets-env
|
||||||
Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
|
Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
|
||||||
Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
|
Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
|
||||||
@@ -453,6 +505,19 @@ services:
|
|||||||
Cluster__PublicHostname: "site-b-1"
|
Cluster__PublicHostname: "site-b-1"
|
||||||
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
|
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
|
||||||
Cluster__Roles__0: "driver"
|
Cluster__Roles__0: "driver"
|
||||||
|
# Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps",
|
||||||
|
# the transport this repo has always used, and every node keeps its ClusterClient wiring built
|
||||||
|
# and its comm actor registered with the receptionist either way. Flip the whole rig with
|
||||||
|
# OTOPCUA_MESH_MODE=ClusterClient at `docker compose up` — no compose edit, and no rebuild.
|
||||||
|
#
|
||||||
|
# Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
|
||||||
|
# time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
|
||||||
|
# actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
|
||||||
|
# silence. Both central nodes are listed so a driver node's client can rotate to whichever one
|
||||||
|
# is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
|
||||||
|
MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Dps}"
|
||||||
|
MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
|
||||||
|
MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
|
||||||
<<: *secrets-env
|
<<: *secrets-env
|
||||||
Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
|
Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
|
||||||
Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
|
Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
|
||||||
@@ -494,6 +559,19 @@ services:
|
|||||||
Cluster__PublicHostname: "site-b-2"
|
Cluster__PublicHostname: "site-b-2"
|
||||||
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
|
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
|
||||||
Cluster__Roles__0: "driver"
|
Cluster__Roles__0: "driver"
|
||||||
|
# Mesh command transport (per-cluster mesh Phase 2). DARK SWITCH: the rig comes up on "Dps",
|
||||||
|
# the transport this repo has always used, and every node keeps its ClusterClient wiring built
|
||||||
|
# and its comm actor registered with the receptionist either way. Flip the whole rig with
|
||||||
|
# OTOPCUA_MESH_MODE=ClusterClient at `docker compose up` — no compose edit, and no rebuild.
|
||||||
|
#
|
||||||
|
# Contact points are node ADDRESSES ONLY. "/system/receptionist" is appended at construction
|
||||||
|
# time, and MeshTransportOptionsValidator refuses to start a node whose contact carries an
|
||||||
|
# actor-path suffix, because a doubled suffix resolves to nothing and drops every ack in
|
||||||
|
# silence. Both central nodes are listed so a driver node's client can rotate to whichever one
|
||||||
|
# is up — that rotation is why the comm actors are per-node and NOT cluster singletons.
|
||||||
|
MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Dps}"
|
||||||
|
MeshTransport__CentralContactPoints__0: "akka.tcp://otopcua@central-1:4053"
|
||||||
|
MeshTransport__CentralContactPoints__1: "akka.tcp://otopcua@central-2:4053"
|
||||||
<<: *secrets-env
|
<<: *secrets-env
|
||||||
Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
|
Serilog__MinimumLevel__Override__Microsoft.EntityFrameworkCore: "Warning"
|
||||||
Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
|
Serilog__MinimumLevel__Override__Microsoft.AspNetCore: "Warning"
|
||||||
|
|||||||
@@ -110,6 +110,27 @@ The checked-in `appsettings*.json` files are deliberately thin: they carry only
|
|||||||
|
|
||||||
> **`Port` / `PublicHostname` are stored twice.** The node binds from these keys; the fleet's `ClusterNode` row records the same address as `AkkaPort` / `Host` so that **central can dial the node without sharing a gossip ring with it** — which is what [per-cluster mesh Phase 2](plans/2026-07-21-per-cluster-mesh-design.md) needs. Nothing in the schema makes the two agree, so `ClusterNodeAddressReconcilerActor` (admin-role singleton) compares them against live membership and logs an Error on mismatch. If you change `Cluster:Port` or `Cluster:PublicHostname` on a node, **update its `ClusterNode` row too** — a node binding 4054 with a row saying 4053 still gossips fine today, and fails silently in Phase 2. The row's `NodeId` is `host:port` and is also the deploy path's ack identity, so a drifted node's deployment acks stop matching as well. See [`config-db-schema.md` § `ClusterNode`](v2/config-db-schema.md#clusternode).
|
> **`Port` / `PublicHostname` are stored twice.** The node binds from these keys; the fleet's `ClusterNode` row records the same address as `AkkaPort` / `Host` so that **central can dial the node without sharing a gossip ring with it** — which is what [per-cluster mesh Phase 2](plans/2026-07-21-per-cluster-mesh-design.md) needs. Nothing in the schema makes the two agree, so `ClusterNodeAddressReconcilerActor` (admin-role singleton) compares them against live membership and logs an Error on mismatch. If you change `Cluster:Port` or `Cluster:PublicHostname` on a node, **update its `ClusterNode` row too** — a node binding 4054 with a row saying 4053 still gossips fine today, and fails silently in Phase 2. The row's `NodeId` is `host:port` and is also the deploy path's ack identity, so a drifted node's deployment acks stop matching as well. See [`config-db-schema.md` § `ClusterNode`](v2/config-db-schema.md#clusternode).
|
||||||
|
|
||||||
|
### `MeshTransport` (central↔node command transport)
|
||||||
|
|
||||||
|
- **Purpose:** selects how central's three command channels (`deployments`, `driver-control`, `alarm-commands`) and the node's `deployment-acks` reply reach the other side — over the mesh-wide DistributedPubSub, or over an Akka `ClusterClient` that does **not** require central and the node to share a gossip ring. The second option is what [per-cluster mesh](plans/2026-07-21-per-cluster-mesh-design.md) Phase 6 needs once the meshes split.
|
||||||
|
- **Options class:** `MeshTransportOptions` (`SectionName = "MeshTransport"`) — `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/MeshTransportOptions.cs`. Bound and validated by `AddOtOpcUaCluster(config)`; `MeshTransportOptionsValidator` runs at `ValidateOnStart`.
|
||||||
|
|
||||||
|
| Key | Type | Default | Meaning |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `Mode` | string | `Dps` | `Dps` = DistributedPubSub (the transport this repo has always used). `ClusterClient` = the mesh boundary. Any other value **refuses to start**. |
|
||||||
|
| `CentralContactPoints` | string[] | `[]` | Node-side only: central node **addresses** (`akka.tcp://<system>@<host>:<port>`). Required under `ClusterClient`, ignored under `Dps`. |
|
||||||
|
| `ContactRefreshSeconds` | int | `60` | Central-side only: how often central rebuilds its ClusterClient contact set from the `ClusterNode` rows. |
|
||||||
|
|
||||||
|
**Addresses only — no actor-path suffix.** `/system/receptionist` is appended at construction time; a contact carrying `/user/` or `/system/` is rejected at startup, because a doubled suffix produces a path that resolves to nothing and drops every command in silence.
|
||||||
|
|
||||||
|
**It is a dark switch, and both sides are always wired.** The comm actors are spawned and registered with the `ClusterClientReceptionist` on every node in *both* modes, so flipping `Mode` is a config change rather than a redeploy. Under `Dps` the registration is simply idle.
|
||||||
|
|
||||||
|
**Central's contact set comes from the enabled, non-maintenance `ClusterNode` rows** — the same source Phase 1 gave the deploy ack set. A node in `MaintenanceMode` is not dialled, but note that **the contact set does not scope delivery**: a receptionist serves its whole cluster, so `SendToAll` still reaches every registered node exactly as today's DPS broadcast does. The filter is about which receptionists are worth dialling, not about who receives.
|
||||||
|
|
||||||
|
**Buffering is off by design** (`akka.cluster.client.buffer-size = 0`, set in the shipped `akka.conf`). If no receptionist is known, the command is dropped with a Warning rather than queued. Buffering would replay a `DispatchDeployment` on reconnect and apply a deployment the coordinator had already sealed as `TimedOut`, leaving the node running a configuration the database records as failed. Do not "fix" this back to Akka's default of 1000 to quiet a flaky test.
|
||||||
|
|
||||||
|
On the docker-dev rig every node carries these keys already; flip the whole rig with `OTOPCUA_MESH_MODE=ClusterClient` at `docker compose up`.
|
||||||
|
|
||||||
### `ConnectionStrings` → `ConfigDb`
|
### `ConnectionStrings` → `ConfigDb`
|
||||||
|
|
||||||
- **Purpose:** the central Config DB connection string. **Required for every role** — `Program.cs` calls `AddOtOpcUaConfigDb` unconditionally.
|
- **Purpose:** the central Config DB connection string. **Required for every role** — `Program.cs` calls `AddOtOpcUaConfigDb` unconditionally.
|
||||||
|
|||||||
@@ -407,6 +407,37 @@ Net effect: each alarm transition appears **once** on `/alerts` and would histor
|
|||||||
|
|
||||||
See [ScriptedAlarms.md](ScriptedAlarms.md) and [AlarmTracking.md](AlarmTracking.md) for the scripted-alarm engine internals.
|
See [ScriptedAlarms.md](ScriptedAlarms.md) and [AlarmTracking.md](AlarmTracking.md) for the scripted-alarm engine internals.
|
||||||
|
|
||||||
|
## Command transport (central↔node)
|
||||||
|
|
||||||
|
Everything above rides one Akka mesh: central publishes on DistributedPubSub, every node
|
||||||
|
subscribes, and the two ends must be members of the same cluster. Per-cluster mesh **Phase 2**
|
||||||
|
adds a second transport under `MeshTransport:Mode` (see
|
||||||
|
[Configuration.md § `MeshTransport`](Configuration.md#meshtransport-centralnode-command-transport))
|
||||||
|
that does not need shared membership — the prerequisite for splitting the fleet into one mesh per
|
||||||
|
application `Cluster`.
|
||||||
|
|
||||||
|
| | `Dps` (default) | `ClusterClient` |
|
||||||
|
|---|---|---|
|
||||||
|
| Central → node commands | `Publish` on `deployments` / `driver-control` / `alarm-commands` | `ClusterClient.SendToAll` to `/user/node-communication`, re-emitted on the node's **local** `EventStream` |
|
||||||
|
| Node → central acks | `Publish` on `deployment-acks` | `ClusterClient.Send` to `/user/central-communication`, forwarded to the deploy-coordinator singleton |
|
||||||
|
| Requires shared cluster membership | **yes** | no |
|
||||||
|
|
||||||
|
Two properties are load-bearing and easy to break:
|
||||||
|
|
||||||
|
- **Both comm actors are registered per node, NOT as cluster singletons.** A `ClusterClient` rotates
|
||||||
|
across its contact points and must find a live comm actor at whichever node answers. As a
|
||||||
|
singleton the boundary would be reachable through exactly one node, and rotation would fail
|
||||||
|
silently against the others — acks landing or not depending on where the client happened to
|
||||||
|
settle. `akka.cluster.client.receptionist.role` is deliberately empty for the same reason.
|
||||||
|
- **Inbound commands go to the node-local `EventStream`, never back onto a DPS topic.** DPS is
|
||||||
|
mesh-wide and central `SendToAll`s to *every* node, so a DPS re-publish would deliver N copies of
|
||||||
|
every command to every subscriber.
|
||||||
|
|
||||||
|
Phase 2 changes no redundancy semantics: the Primary gate, ServiceLevel, and the singleton layout
|
||||||
|
are untouched, and `redundancy-state` itself stays on DPS (it is bidirectional and built from
|
||||||
|
`Cluster.State`, which makes it genuinely mesh-bound — the hardest remaining dependency before the
|
||||||
|
meshes can split).
|
||||||
|
|
||||||
## Pair-local store (LocalDb — Phases 1 + 2)
|
## Pair-local store (LocalDb — Phases 1 + 2)
|
||||||
|
|
||||||
Independently of the ServiceLevel machinery above, every **driver-role** node keeps a consolidated
|
Independently of the ServiceLevel machinery above, every **driver-role** node keeps a consolidated
|
||||||
|
|||||||
@@ -68,8 +68,14 @@ Other load-bearing details:
|
|||||||
caller's Ask times out. *"It keeps the central coordinator stateless with respect to site
|
caller's Ask times out. *"It keeps the central coordinator stateless with respect to site
|
||||||
availability."* A `ConnectionStateChanged` mechanism was built for this and **deleted as dead
|
availability."* A `ConnectionStateChanged` mechanism was built for this and **deleted as dead
|
||||||
code**.
|
code**.
|
||||||
- **Every unhandled message replies with a typed failure rather than dropping**, so a central Ask
|
- ~~**Every unhandled message replies with a typed failure rather than dropping**, so a central Ask
|
||||||
can never stall on a missing handler.
|
can never stall on a missing handler.~~ **CORRECTED 2026-07-22 (Phase 2 recon).** This overstates
|
||||||
|
what ScadaBridge does. Neither of their comm actors has a `ReceiveAny` or an `Unhandled` override:
|
||||||
|
the typed-failure idiom is **per message type**, and it fires only when a *registration* is missing
|
||||||
|
(no ClusterClient for the site yet, no handler registered for that type). A genuinely unknown
|
||||||
|
message type still dead-letters exactly as it would anywhere else. Do not build a catch-all failure
|
||||||
|
reply believing you are copying them. OtOpcUa's Phase 2 comm actors list each type explicitly and
|
||||||
|
deliberately let an unknown type dead-letter loudly rather than republish it blind.
|
||||||
- **Sender preservation is the whole Ask idiom**: `_client.Tell(new ClusterClient.Send(path, msg),
|
- **Sender preservation is the whole Ask idiom**: `_client.Tell(new ClusterClient.Send(path, msg),
|
||||||
Sender)` routes the reply straight back to the waiting Ask, bypassing the comm actor.
|
Sender)` routes the reply straight back to the waiting Ask, bypassing the comm actor.
|
||||||
|
|
||||||
|
|||||||
@@ -99,7 +99,31 @@ this removes the coordinator's one genuinely mesh-bound dependency).
|
|||||||
set proven DB-sourced (test: a `ClusterNode` row present but node down → deploy reports that node
|
set proven DB-sourced (test: a `ClusterNode` row present but node down → deploy reports that node
|
||||||
missing; a node up but row absent → its ack is not expected).
|
missing; a node up but row absent → its ack is not expected).
|
||||||
|
|
||||||
### Phase 2 — Comm actors + ClusterClient transport
|
### Phase 2 — Comm actors + ClusterClient transport — **CODE COMPLETE 2026-07-22, live gate pending**
|
||||||
|
Shipped per [`2026-07-22-mesh-phase2-clusterclient-transport.md`](2026-07-22-mesh-phase2-clusterclient-transport.md)
|
||||||
|
as a **dark switch**: `MeshTransport:Mode` defaults to `Dps`, so nothing changes on any deployment
|
||||||
|
until the flag flips. Both comm actors are spawned and receptionist-registered in **both** modes, so
|
||||||
|
the flip is a config change rather than a redeploy. Three scope corrections landed against the
|
||||||
|
design doc (that plan's header carries them in full):
|
||||||
|
|
||||||
|
- the node path is `/user/node-communication`, not `/user/cluster-communication` (there is no
|
||||||
|
per-cluster scoping until Phase 6);
|
||||||
|
- the substitution for `Publish` is **`SendToAll`**, not `Send` — `Send` reaches exactly one node and
|
||||||
|
would deploy to one member of the fleet while the rest silently kept their old config;
|
||||||
|
- central creates **exactly one fleet-wide ClusterClient**. On a single mesh a receptionist serves
|
||||||
|
its whole cluster, so a client per application `Cluster` would fan every command out once per
|
||||||
|
cluster. Per-cluster clients become correct only when Phase 6 splits the meshes (`TODO(Phase 6)`
|
||||||
|
in `CentralCommunicationActor.RebuildClient`).
|
||||||
|
|
||||||
|
**Exit-gate deviation — "an Ask timing out cleanly against a stopped node" cannot be run as written.**
|
||||||
|
There is no cross-boundary Ask in Phase 2: every migrated command is fire-and-forget on the wire.
|
||||||
|
`DispatchDeployment` is a `Tell` whose ack returns later as an independent `ApplyAck`;
|
||||||
|
`RestartDriver` / `ReconnectDriver` / `AlarmCommand` reply `Ok = true` from the admin node the moment
|
||||||
|
they are dispatched (`Ok` has always meant *dispatched*, not *applied* — the AdminUI string is
|
||||||
|
literally "Restart dispatched"). The honest equivalents, both in the live gate: a **stopped node**
|
||||||
|
fails the deploy at the apply deadline naming it (Phase 1 semantics), and **no reachable contact**
|
||||||
|
drops the command with a Warning and never buffers it.
|
||||||
|
|
||||||
**Scope:** one receptionist-registered actor per side (`/user/central-communication`,
|
**Scope:** one receptionist-registered actor per side (`/user/central-communication`,
|
||||||
`/user/cluster-communication`), registered **per node, not as a singleton** (contact rotation);
|
`/user/cluster-communication`), registered **per node, not as a singleton** (contact rotation);
|
||||||
ClusterClient central → cluster carrying deploy notify + acks + driver-control; the ScadaBridge
|
ClusterClient central → cluster carrying deploy notify + acks + driver-control; the ScadaBridge
|
||||||
@@ -200,7 +224,7 @@ resource sizing on the site VMs should be checked once both products run the ful
|
|||||||
| 0b oldest-Up election | DONE 2026-07-21 |
|
| 0b oldest-Up election | DONE 2026-07-21 |
|
||||||
| Prereq: selfform-fallback + manual-failover plan | **DONE 2026-07-22**, merged `a78425ea`. Shipped *not* as the planned `SelfFormAfter` watchdog — the live gate caught that islanding a manually-failed-over node — but as self-first seed ordering + `AkkaClusterOptionsValidator`. Phase 6's seed-ordering scope is therefore already discharged. |
|
| Prereq: selfform-fallback + manual-failover plan | **DONE 2026-07-22**, merged `a78425ea`. Shipped *not* as the planned `SelfFormAfter` watchdog — the live gate caught that islanding a manually-failed-over node — but as self-first seed ordering + `AkkaClusterOptionsValidator`. Phase 6's seed-ordering scope is therefore already discharged. |
|
||||||
| 1 ClusterNode columns + DB ack set | **DONE 2026-07-22**, merged `7654f24d`. Deploy-seal semantics changed as flagged, but the escape hatch is **`ClusterNode.MaintenanceMode`**, not `Enabled=0` — the live gate found `Enabled=0` is rejected by `DraftValidator.ValidateClusterTopology` on any 2-node pair. Gate record: `2026-07-22-mesh-phase1-live-gate.md`. |
|
| 1 ClusterNode columns + DB ack set | **DONE 2026-07-22**, merged `7654f24d`. Deploy-seal semantics changed as flagged, but the escape hatch is **`ClusterNode.MaintenanceMode`**, not `Enabled=0` — the live gate found `Enabled=0` is rejected by `DraftValidator.ValidateClusterTopology` on any 2-node pair. Gate record: `2026-07-22-mesh-phase1-live-gate.md`. |
|
||||||
| 2 ClusterClient transport | **IN PROGRESS** — plan being written 2026-07-22 |
|
| 2 ClusterClient transport | **CODE COMPLETE 2026-07-22, live gate pending** — shipped dark (`MeshTransport:Mode=Dps` default), both comm actors registered in both modes. Corrections: `SendToAll` not `Send`, one fleet-wide client not one per cluster, `/user/node-communication` not `/user/cluster-communication`. The gate's "Ask timing out" item does not exist to test — no cross-boundary Ask in this phase; see the phase section. |
|
||||||
| 3 fetch-and-cache | not started |
|
| 3 fetch-and-cache | not started |
|
||||||
| 4 cut driver ConfigDb | not started |
|
| 4 cut driver ConfigDb | not started |
|
||||||
| 5 gRPC telemetry | not started |
|
| 5 gRPC telemetry | not started |
|
||||||
|
|||||||
Reference in New Issue
Block a user