docs(mesh): sweep for ClusterClient deletion — Dps|Grpc only, CentralContactPoints retired

Task 3.2 of the ClusterClient->gRPC migration. Sweeps the docs that still
described the Akka ClusterClient transport as live, following its Phase 3
deletion (this branch): MeshTransport:Mode is now Dps (pair-local DPS) or
Grpc (cross-mesh unary calls) only, ClusterClient fails validation loudly,
CentralContactPoints is deleted (not superseded), and the split-topology
validator requires exactly Grpc rather than ClusterClient.

- docs/Configuration.md: rewrite the MeshTransport section — table,
  dark-switch/PSK-seeding prose, listener sharing (:4056 driver-role,
  :4055 admin-role), remove the "shipped dark, cutover pending" framing.
- docs/Redundancy.md: replace the Dps/ClusterClient transport table with
  Dps/Grpc, fix the Phase 6 CentralCommunicationActor + validator bullets.
- docs/Telemetry.md: fix stale ClusterClient cross-refs, add a note that
  the driver-role telemetry listener also hosts NodeCommandService.
- docs/v2/config-db-schema.md: one-line "(retired: ...)" annotation on the
  historical AkkaPort/GrpcPort note — left as a historical record otherwise.
- CLAUDE.md: rewrite "## Mesh command transport (MeshTransport)" for the
  post-deletion world (no ClusterClient/receptionist, per-node gRPC dial,
  fire-and-forget semantics unchanged) and fix two Phase 6 bullets.

deferment.md has no mesh-transport mentions; left untouched.

docs-only change; `dotnet build ZB.MOM.WW.OtOpcUa.slnx` stays green (0
errors) after the edit.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
This commit is contained in:
Joseph Doherty
2026-08-06 16:23:23 -04:00
parent cd7135570c
commit 406b1ae5ac
5 changed files with 126 additions and 85 deletions
+65 -41
View File
@@ -371,8 +371,8 @@ Assert the **effective** cluster config off a running `ActorSystem` (as `SplitBr
- **The redundancy singleton is cluster-scoped and spawned per driver node.** `RedundancyStateActor` is now a `cluster-{ClusterId}`-scoped singleton (falls back to plain `driver` for a legacy node), so each pair elects and publishes its **own** Primary on its own mesh's DPS. **Two or more Primaries exist fleet-wide, one per pair — by design, not a bug.** ⚠️ **Wiring landmine (fixed `3a4ed8dd`, caught by the live gate):** the singleton's cluster-role *scope* is computed at `AddAkka`-configurator time, so it must come from `AkkaClusterOptions` (config) — **never** by resolving `IClusterRoleInfo` from the SP there. `ClusterRoleInfo` depends on the `ActorSystem`, and that lambda runs *while the ActorSystem is being built*, so resolving it recurses into ActorSystem construction and **stack-overflows every node at boot** (compiles clean; the rehome tests passed a hand-built fake straight into the extension, mocking around the composition-root line). Keep `WithOtOpcUaClusterRedundancySingleton`/`BuildClusterRedundancySingletonOptions` taking `AkkaClusterOptions`.
- `ClusterNodeAddressReconciler` is **own-cluster-scoped** — an admin node can only reconcile `ClusterNode` rows in its own cluster; it has no gossip visibility into another mesh.
- `CentralCommunicationActor` creates **one `ClusterClient` per application `Cluster`** and fans commands across them (closing the `TODO(Phase 6)` Phase 2 left behind), rather than one fleet-wide client.
- **`SplitTopologyTransportValidator` fails host start** on a node carrying a `cluster-{ClusterId}` role unless it also runs the split-safe transports: `MeshTransport:Mode=ClusterClient`, `Telemetry:Mode=Grpc` (driver), `TelemetryDial:Mode=Grpc` (admin). A DPS transport on a cluster-role node would try to gossip across a partition that no longer exists.
- `CentralCommunicationActor` dials each node directly over gRPC — one `NodeCommandService.Execute` call per enabled `ClusterNode` row carrying a `GrpcPort` — rather than fanning through a client/receptionist object. **Superseded by the [ClusterClient→gRPC migration](docs/plans/2026-08-06-clusterclient-to-grpc-design.md):** the Akka `ClusterClient` this bullet originally described (one per application `Cluster`, closing the `TODO(Phase 6)` Phase 2 left behind) is deleted; the per-node gRPC dial is its replacement, not an evolution of it.
- **`SplitTopologyTransportValidator` fails host start** on a node carrying a `cluster-{ClusterId}` role unless it also runs the split-safe transports: `MeshTransport:Mode=Grpc`, `Telemetry:Mode=Grpc` (driver), `TelemetryDial:Mode=Grpc` (admin). A DPS transport on a cluster-role node would try to gossip across a partition that no longer exists.
- **The compiled transport-mode defaults were deliberately NOT flipped** — `MeshTransport:Mode`/`Telemetry:Mode`/`TelemetryDial:Mode` all still default to `Dps`; a blast-radius assessment showed flipping the default breaks every integration test and appsettings profile that leaves the key unset. The validator + explicit per-deployment config is the guardrail, not a default flip.
- **Secrets replication is pair-local** — each pair replicates its own Akka DPS secrets within itself; there is no fleet-wide/cross-cluster sync after the split (sites have no SQL, so no SQL-hub, consistent with Phase 4).
@@ -380,51 +380,74 @@ See `docs/Redundancy.md` §"Per-cluster meshes (Phase 6)", `docs/Configuration.m
## Mesh command transport (`MeshTransport`)
Per-cluster mesh **Phase 2** added a second transport for the three central→node command channels
(`deployments`, `driver-control`, `alarm-commands`) and the `deployment-acks` reply, selected by
`MeshTransport:Mode`: `Dps` (the DistributedPubSub path this repo has always used, **the default**)
or `ClusterClient`, which does **not** require central and the node to share a gossip ring — the
prerequisite for splitting the fleet into one mesh per application `Cluster`. Both `CentralCommunicationActor`
(`/user/central-communication`) and `NodeCommunicationActor` (`/user/node-communication`) are spawned
and registered with the `ClusterClientReceptionist` in **both** modes, so flipping the flag is a config
change, not a redeploy. Things worth knowing before touching it:
**The Akka `ClusterClient` transport described in this section's earlier history is DELETED**
(branch `feat/grpc-phase3-deletion`, [ClusterClient→gRPC migration](docs/plans/2026-08-06-clusterclient-to-grpc-design.md),
Phase 3) — code, the `akka.cluster.client` HOCON block, the receptionist extension,
`MeshTransport:CentralContactPoints`, and the `ClusterClient` mode value itself, which now **fails
validation loudly** at boot (`MeshTransportOptionsValidator` names the key and the allowed values)
rather than silently falling back to `Dps`. What remains is exactly two transports, selected by
`MeshTransport:Mode`:
- **`SendToAll`, never `Send`.** Today's DPS publish reaches every `DriverHostActor`; there is no
ClusterId or node filter on the node side at all (scoping happens later, inside the artifact via
`DeploymentArtifact.ResolveClusterScope`). `ClusterClient.Send` delivers to exactly **one** registered
actor — it would deploy to a single node while every other node silently kept its old configuration,
and the deployment could still seal green.
- **RESOLVED by Phase 6 — one `ClusterClient` per application `Cluster`, not one fleet-wide.** When
this was written the fleet was still one mesh, so a receptionist served the whole cluster and
`SendToAll` reached every registered node-comm actor regardless of which node's address was dialled;
the `TODO(Phase 6)` in `CentralCommunicationActor.RebuildClient` asked for one client per application
`Cluster` instead. Phase 6 split the fleet into one mesh per `Cluster` and shipped exactly that — see
the [Redundancy](#redundancy) section's Phase 6 paragraph above. The `SendToAll`-within-a-mesh
behaviour below is otherwise unchanged; a `MaintenanceMode` node still isn't excluded from delivery
within its own mesh.
- **Inbound commands re-emit on the node's `EventStream`, not on their DPS topic** — DPS is mesh-wide and
central `SendToAll`s to every node, so a DPS re-publish would deliver N copies to every subscriber.
Node-side subscribers (`DriverHostActor`, `ScriptedAlarmHostActor`) accept both.
- **Comm actors are per-node, NOT singletons.** A client rotates across contact points and must find a live
comm actor at whichever node answers; `akka.cluster.client.receptionist.role` is empty for the same reason.
- **`buffer-size = 0` is a behaviour decision, not a tuning knob.** Akka's default buffers 1000 and replays
on reconnect — silently, on the client's own timer, at an unbounded remove from the dispatch. Do not raise
it to quiet a flaky test. **But do not over-claim what it buys:** the Phase 2 live gate observed a node
applying a deployment **44 s after it was sealed `TimedOut`** anyway, via `DriverHostActor.Bootstrap()`
replaying the orphan `Applying` row on restart. Zero buffering removes the silent replay, not every replay.
- **`Dps`** (default) — DistributedPubSub, the path this repo has always used. **Pair-local only**:
it works only when central and the node share a gossip ring, which after Phase 6 means within one
2-node mesh.
- **`Grpc`** — unary gRPC calls on the `mesh_transport.v1` contract. **Required** on any node
carrying a `cluster-{ClusterId}` role; `SplitTopologyTransportValidator` refuses to boot such a
node on anything else.
Contact points are node **addresses only** (`akka.tcp://<system>@<host>:<port>`); `/system/receptionist` is
appended at construction and `MeshTransportOptionsValidator` refuses to start a node whose contact carries an
actor-path suffix. `redundancy-state` stays on DPS in both modes (bidirectional, and built from
`Cluster.State`). See `docs/Configuration.md` §`MeshTransport`, `docs/Redundancy.md` §"Command transport",
and `docs/plans/2026-07-22-mesh-phase2-clusterclient-transport.md`.
Central→node commands (`deployments`, `driver-control`, `alarm-commands`) ride `NodeCommandService.Execute`,
served on each node's driver-role telemetry h2c listener (`Telemetry:GrpcListenPort`, the docker-dev
rig's `:4056` — it shares that listener with `TelemetryStreamGrpcService` rather than claiming a port
of its own). `CentralCommunicationActor` dials it **directly, one call per node** — the target set is
every enabled, non-maintenance `ClusterNode` row carrying a `GrpcPort`, dialled at `Host`+`GrpcPort`
and refreshed every `MeshTransport:ContactRefreshSeconds`. There is no client/receptionist object and
no rotation: each command goes to the specific node dialled, which is what makes per-node delivery
provable — the old `ClusterClient.SendToAll`/`Send` distinction this section used to explain no longer
applies to either direction.
Node→central deploy acks ride `CentralCommandService.ReportApply`, served on central's admin-role
config-serve h2c listener (`ConfigServe:GrpcListenPort`, the rig's `:4055` — again sharing rather than
claiming a new port). `GrpcApplyAckSender` walks `MeshTransport:CentralGrpcEndpoints` in order and
stops at the first success.
Auth is a **per-cluster PSK bearer**, both directions: `MeshTransport:GrpcPsk` (node-side, one key per
application `Cluster`) and `MeshTransport:ClusterPsks` (central-side map, consulted before the Secrets
store). `MeshCommandAuthInterceptor` (node) and `CentralCommandAuthInterceptor` (central) fail closed
on every unresolvable case — constant-time compare, no key material logged. Production keys are
`${secret:OT-MESH-PSK-<clusterId>}`; the docker-dev rig supplies three literal dev keys via env
(`dev-mesh-psk-MAIN`/`-SITE-A`/`-SITE-B`).
Things that carried over unchanged from the `ClusterClient` era, because they were never
`ClusterClient`-specific to begin with:
- **Fire-and-forget, no retry, on both transports.** A failed gRPC dispatch is logged (naming the node
and the gRPC status) and dropped in the dispatch second. No queue, no ordering guarantee across
nodes — the deploy coordinator's apply deadline and a node's boot-time replay are the only recovery.
The old `buffer-size = 0` decision this section used to describe is gone with the HOCON block that
set it; the gRPC transport has nothing to buffer in the first place — a dial either lands or doesn't.
- **Inbound commands re-emit on the node's `EventStream`, not back onto DPS.** DPS is mesh-wide, so a
DPS re-publish would deliver N copies to every subscriber. Node-side subscribers (`DriverHostActor`,
`ScriptedAlarmHostActor`) accept a command from either transport identically.
- **Do not over-claim what fire-and-forget buys.** The Phase 2 live gate observed a node applying a
deployment **44 s after it was sealed `TimedOut`** anyway, via `DriverHostActor.Bootstrap()`
replaying the orphan `Applying` row on restart. That backstop is unaffected by the ClusterClient
deletion — it was always about the deploy path, not the transport.
`redundancy-state` stays on DPS in both modes (bidirectional, and built from `Cluster.State`, which
makes it genuinely mesh-bound). See `docs/Configuration.md` §`MeshTransport`, `docs/Redundancy.md`
§"Command transport", and the migration's [design doc](docs/plans/2026-08-06-clusterclient-to-grpc-design.md) /
[plan](docs/plans/2026-08-06-clusterclient-to-grpc.md) / [tracking doc](docs/plans/2026-08-06-clusterclient-to-grpc-tracking.md) /
[live gate](docs/plans/2026-08-06-clusterclient-to-grpc-live-gate.md) for the phase-by-phase history —
those are historical records, not updated further by this sweep. `docs/plans/2026-07-22-mesh-phase2-clusterclient-transport.md`
is the original Phase 2 design doc and is likewise left as a historical record of the transport this
section now describes as deleted.
**Phase 1 landed 2026-07-22 and changed a deploy-path behaviour.** `ConfigPublishCoordinator` sources its expected-ack set from **enabled `ClusterNode` rows**, not from `Akka.Cluster.State.Members` filtered by the `driver` role — central must be able to name a deployment's nodes without sharing a gossip ring with them. Consequences worth knowing before touching deploys:
- **A configured node that is switched off now FAILS the deployment** at the apply deadline instead of letting it seal green without that node. The maintenance hatch is the **new `ClusterNode.MaintenanceMode` column**, NOT `Enabled = 0`: the live gate showed `Enabled = 0` is rejected outright by `DraftValidator.ValidateClusterTopology` (`ClusterEnabledNodeCountMismatch` — enabled count must equal `ServerCluster.NodeCount`), so on a 2-node pair — every cluster in the target topology — it can never be the hatch. `Enabled` = part of the declared topology; `MaintenanceMode` = do not expect it now. The deadline log names the silent nodes and points at the right flag.
- **Every `ClusterNode` row is now *defined* to be a driver node.** The DB has no per-node role column and deliberately did not gain one (it would drift from `Cluster:Roles`). Giving an admin-only node a `ClusterNode` row makes every deploy time out.
- **There is no such thing as a cluster-scoped deployment.** `Deployment` has no `ClusterId`, `ConfigComposer` always snapshots the whole DB, and `DeploymentArtifact.ResolveClusterScope` is *node-side* self-scoping of a fleet-wide artifact. The phase-1 plan asked for cluster-scope filtering of the ack set; it was dropped as describing a feature that does not exist.
- `ClusterNode` gained `AkkaPort` (NOT NULL, 4053) + `GrpcPort` (nullable) as **central's dial targets** for Phases 2/5. They duplicate the node's own `Cluster:Port` / `Cluster:PublicHostname` with no schema-level enforcement, so `ClusterNodeAddressReconcilerActor` (admin singleton) logs an Error on drift. **Phase 2 must revisit it** — once the meshes split, an admin node cannot see site members and every site row would report `EnabledRowNotInCluster` forever.
- `ClusterNode` gained `AkkaPort` (NOT NULL, 4053) + `GrpcPort` (nullable) as **central's dial targets** for Phases 2/5. They duplicate the node's own `Cluster:Port` / `Cluster:PublicHostname` with no schema-level enforcement, so `ClusterNodeAddressReconcilerActor` (admin singleton) logs an Error on drift. **Phase 2 must revisit it** — once the meshes split, an admin node cannot see site members and every site row would report `EnabledRowNotInCluster` forever. **Resolved, and not the way this bullet originally expected:** the [ClusterClient→gRPC migration](docs/plans/2026-08-06-clusterclient-to-grpc-design.md) deleted the Akka `ClusterClient` transport that would have dialled `Host`+`AkkaPort`; the Phase 2 command dial now uses `Host`+`GrpcPort`, the same column Phase 5 telemetry already dialled — `AkkaPort` is Akka remoting only again.
## Config source (`ConfigSource` / `ConfigServe`)
@@ -488,8 +511,9 @@ touching it:
`Cluster.State`, drives ServiceLevel + the Primary gate — genuinely mesh-bound by design, not an
observability panel); `fleet-status` is central-internal (`FleetStatusBroadcaster` builds it from
the admin node's own membership events, `Fleet.razor` polls the Config DB and ignores the feed —
never a node→central stream); `deployment-acks` already rides the Phase 2 `ClusterClient`
transport as a command-plane reply, not telemetry.
never a node→central stream); `deployment-acks` already rides the Phase 2 gRPC command transport
(`MeshTransport:Mode=Grpc`, `CentralCommandService.ReportApply`) as a command-plane reply, not
telemetry.
- **The node ALWAYS hosts and ALWAYS double-publishes, in both modes.** Every publish seam emits
into a node-local `ITelemetryLocalHub` **and** still publishes DPS, unconditionally; the gRPC
server binds whenever `Telemetry:GrpcListenPort > 0`, independent of `Telemetry:Mode`. Only