docs(mesh): MeshTransport config, Phase 2 as shipped, design-doc correction

Rig: all six docker-dev nodes carry MeshTransport__Mode plus both central
contact points. Mode is "${OTOPCUA_MESH_MODE:-Dps}", so the rig still comes up
on the transport it has always used and the live gate can flip the whole fleet
at `docker compose up` without a compose edit or a rebuild.

Docs: a MeshTransport section in Configuration.md (including why contact points
are addresses only, why the contact set does not scope delivery, and why
buffer-size=0 is a behaviour decision rather than a tuning knob), a "Command
transport" section in Redundancy.md contrasting the two modes, and a summary
beside the Redundancy notes in CLAUDE.md.

Corrects the design doc's claim that ScadaBridge replies with a typed failure to
every unhandled message. Neither of their comm actors has a ReceiveAny or an
Unhandled override; the idiom is per message type and fires on a missing
registration, and an unknown type dead-letters there as it does anywhere else.

Records the exit-gate deviation in the program plan: "an Ask timing out cleanly
against a stopped node" cannot be run, because no Ask crosses this boundary. The
honest equivalents are a stopped node failing the deploy at the apply deadline
and an unreachable contact dropping the command with a Warning.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
2026-07-22 17:46:46 -04:00
parent 4eca87f04d
commit e4cb9a08e0
6 changed files with 202 additions and 5 deletions
+38 -1
View File
@@ -228,7 +228,44 @@ The server supports non-transparent warm/hot redundancy via the `Redundancy` sec
Assert the **effective** cluster config off a running `ActorSystem` (as `SplitBrainResolverActivationTests` does), never the typed options or the akka.conf text — several HOCON fragments compete and the losing one still reads correctly in the file.
**Still per-Akka-cluster, not per application `Cluster`.** One Primary is elected across the whole Akka mesh, so a fleet running several application clusters in one mesh (what docker-dev models) gets one Primary among all driver nodes. The fix is the per-cluster mesh workstream: `docs/plans/2026-07-21-per-cluster-mesh-design.md` (Phases 0a/0b/1 done; 27 not started).
**Still per-Akka-cluster, not per application `Cluster`.** One Primary is elected across the whole Akka mesh, so a fleet running several application clusters in one mesh (what docker-dev models) gets one Primary among all driver nodes. The fix is the per-cluster mesh workstream: `docs/plans/2026-07-21-per-cluster-mesh-design.md` (Phases 0a/0b/1 done; 2 code-complete behind a dark switch; 37 not started).
## Mesh command transport (`MeshTransport`)
Per-cluster mesh **Phase 2** added a second transport for the three central→node command channels
(`deployments`, `driver-control`, `alarm-commands`) and the `deployment-acks` reply, selected by
`MeshTransport:Mode`: `Dps` (the DistributedPubSub path this repo has always used, **the default**)
or `ClusterClient`, which does **not** require central and the node to share a gossip ring — the
prerequisite for splitting the fleet into one mesh per application `Cluster`. Both `CentralCommunicationActor`
(`/user/central-communication`) and `NodeCommunicationActor` (`/user/node-communication`) are spawned
and registered with the `ClusterClientReceptionist` in **both** modes, so flipping the flag is a config
change, not a redeploy. Things worth knowing before touching it:
- **`SendToAll`, never `Send`.** Today's DPS publish reaches every `DriverHostActor`; there is no
ClusterId or node filter on the node side at all (scoping happens later, inside the artifact via
`DeploymentArtifact.ResolveClusterScope`). `ClusterClient.Send` delivers to exactly **one** registered
actor — it would deploy to a single node while every other node silently kept its old configuration,
and the deployment could still seal green.
- **Exactly ONE fleet-wide ClusterClient while the fleet is one mesh.** A receptionist serves its whole
cluster, so `SendToAll` reaches every registered node-comm actor regardless of which node's address was
dialled. One client per application `Cluster` — the obviously-right-looking shape, and what Phase 6
wants — would fan each command out once per cluster (N× duplicate dispatch *and* N× duplicate ack).
Marked `TODO(Phase 6)` in `CentralCommunicationActor.RebuildClient`. **Corollary:** the contact set does
not scope delivery; excluding a `MaintenanceMode` node from the contacts does not stop it receiving.
- **Inbound commands re-emit on the node's `EventStream`, not on their DPS topic** — DPS is mesh-wide and
central `SendToAll`s to every node, so a DPS re-publish would deliver N copies to every subscriber.
Node-side subscribers (`DriverHostActor`, `ScriptedAlarmHostActor`) accept both.
- **Comm actors are per-node, NOT singletons.** A client rotates across contact points and must find a live
comm actor at whichever node answers; `akka.cluster.client.receptionist.role` is empty for the same reason.
- **`buffer-size = 0` is a behaviour decision, not a tuning knob.** Akka's default buffers 1000 and replays
on reconnect, which would apply a `DispatchDeployment` for a deployment the coordinator already sealed
`TimedOut` — the node would run a config the DB records as failed. Do not raise it to quiet a flaky test.
Contact points are node **addresses only** (`akka.tcp://<system>@<host>:<port>`); `/system/receptionist` is
appended at construction and `MeshTransportOptionsValidator` refuses to start a node whose contact carries an
actor-path suffix. `redundancy-state` stays on DPS in both modes (bidirectional, and built from
`Cluster.State`). See `docs/Configuration.md` §`MeshTransport`, `docs/Redundancy.md` §"Command transport",
and `docs/plans/2026-07-22-mesh-phase2-clusterclient-transport.md`.
**Phase 1 landed 2026-07-22 and changed a deploy-path behaviour.** `ConfigPublishCoordinator` sources its expected-ack set from **enabled `ClusterNode` rows**, not from `Akka.Cluster.State.Members` filtered by the `driver` role — central must be able to name a deployment's nodes without sharing a gossip ring with them. Consequences worth knowing before touching deploys: