dohertj2/lmxopcua
1
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(mesh): sweep for ClusterClient deletion — Dps|Grpc only, CentralContactPoints retired

Browse Source 
Task 3.2 of the ClusterClient->gRPC migration. Sweeps the docs that still
described the Akka ClusterClient transport as live, following its Phase 3
deletion (this branch): MeshTransport:Mode is now Dps (pair-local DPS) or
Grpc (cross-mesh unary calls) only, ClusterClient fails validation loudly,
CentralContactPoints is deleted (not superseded), and the split-topology
validator requires exactly Grpc rather than ClusterClient.

- docs/Configuration.md: rewrite the MeshTransport section — table,
  dark-switch/PSK-seeding prose, listener sharing (:4056 driver-role,
  :4055 admin-role), remove the "shipped dark, cutover pending" framing.
- docs/Redundancy.md: replace the Dps/ClusterClient transport table with
  Dps/Grpc, fix the Phase 6 CentralCommunicationActor + validator bullets.
- docs/Telemetry.md: fix stale ClusterClient cross-refs, add a note that
  the driver-role telemetry listener also hosts NodeCommandService.
- docs/v2/config-db-schema.md: one-line "(retired: ...)" annotation on the
  historical AkkaPort/GrpcPort note — left as a historical record otherwise.
- CLAUDE.md: rewrite "## Mesh command transport (MeshTransport)" for the
  post-deletion world (no ClusterClient/receptionist, per-node gRPC dial,
  fire-and-forget semantics unchanged) and fix two Phase 6 bullets.

deferment.md has no mesh-transport mentions; left untouched.

docs-only change; `dotnet build ZB.MOM.WW.OtOpcUa.slnx` stays green (0
errors) after the edit.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
This commit is contained in:
Joseph Doherty
2026-08-06 16:23:23 -04:00
parent cd7135570c
commit 406b1ae5ac
5 changed files with 126 additions and 85 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/Configuration.md
+16 -23
View File
@@ -142,7 +142,7 @@ The checked-in `appsettings*.json` files are deliberately thin: they carry only
- **`SplitTopologyTransportValidator`** (`src/Core/ZB.MOM.WW.OtOpcUa.Cluster/`, wired at
`ValidateOnStart` alongside the other `Cluster`-family validators) fails host start on any node
carrying a `cluster-{ClusterId}` role unless it is also configured for the split-safe transports:
`MeshTransport:Mode = ClusterClient`, `Telemetry:Mode = Grpc` on a driver node, `TelemetryDial:Mode =
`MeshTransport:Mode = Grpc`, `Telemetry:Mode = Grpc` on a driver node, `TelemetryDial:Mode =
Grpc` on an admin node. A DPS-mode transport on a cluster-role node would try to gossip across a
partition boundary that no longer exists, so the validator fails closed rather than let commands or
telemetry silently vanish. A legacy node with no `cluster-{ClusterId}` role is exempt and may stay on
@@ -155,45 +155,38 @@ The checked-in `appsettings*.json` files are deliberately thin: they carry only
actually carries a `cluster-{ClusterId}` role must set the split-safe modes explicitly.
- See [`Redundancy.md` § Per-cluster meshes (Phase 6)](Redundancy.md#per-cluster-meshes-phase-6) for the
full runtime picture (one Primary per pair, the cluster-scoped `RedundancyStateActor` singleton,
`ClusterNodeAddressReconciler` own-cluster-scoping, and central's one-`ClusterClient`-per-`Cluster`
fan-out).
`ClusterNodeAddressReconciler` own-cluster-scoping, and central's per-node gRPC command dial —
one `NodeCommandService.Execute` call per enabled `ClusterNode` row carrying a `GrpcPort`).
### `MeshTransport` (central↔node command transport)
- **Purpose:** selects how central's three command channels (`deployments`, `driver-control`, `alarm-commands`) and the node's `deployment-acks` reply reach the other side — over the mesh-wide DistributedPubSub, or over an Akka `ClusterClient` that does **not** require central and the node to share a gossip ring. The second option is what [per-cluster mesh](plans/2026-07-21-per-cluster-mesh-design.md) Phase 6 needs once the meshes split.
- **Purpose:** selects how central's three command channels (`deployments`, `driver-control`, `alarm-commands`) and the node's `deployment-acks` reply reach the other side — over the mesh-wide DistributedPubSub (pair-local only), or over unary gRPC calls that do **not** require central and the node to share a gossip ring. The second option is what [per-cluster mesh](plans/2026-07-21-per-cluster-mesh-design.md) Phase 6 needs, now that the meshes are split. The Akka `ClusterClient` transport that used to be the second option here is **deleted** ([ClusterClient→gRPC migration](plans/2026-08-06-clusterclient-to-grpc-design.md), Phase 3) — `Grpc` is its replacement, not a third option alongside it.
- **Options class:** `MeshTransportOptions` (`SectionName = "MeshTransport"`) — `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/MeshTransportOptions.cs`. Bound and validated by `AddOtOpcUaCluster(config)`; `MeshTransportOptionsValidator` runs at `ValidateOnStart`.
| Key | Type | Default | Meaning |
|---|---|---|---|
| `Mode` | string | `Dps` | `Dps` = DistributedPubSub (the transport this repo has always used). `ClusterClient` = the mesh boundary. `Grpc` = the `mesh_transport.v1` gRPC contract — **shipped dark, cutover pending** (see below). Any other value **refuses to start**. |
| `CentralContactPoints` | string[] | `[]` | Node-side only: central node **addresses** (`akka.tcp://<system>@<host>:<port>`). Required under `ClusterClient`, ignored under `Dps`/`Grpc`. |
| `ContactRefreshSeconds` | int | `60` | Central-side only: how often central rebuilds its ClusterClient contact set from the `ClusterNode` rows. |
| `Mode` | string | `Dps` | `Dps` = DistributedPubSub, pair-local only — cannot cross an application-`Cluster` boundary. `Grpc` = the `mesh_transport.v1` gRPC contract (below), required on any node carrying a `cluster-{ClusterId}` role. Any other value **refuses to start**, including the retired `ClusterClient` value — a node still configured for it is refused at boot rather than silently demoted to `Dps`, which on a split-topology node would deliver nothing at all. |
| `ContactRefreshSeconds` | int | `60` | Central-side only: how often central rebuilds its node dispatch set — the gRPC dial targets and the known-cluster PSK gate — from the `ClusterNode` rows. Runs in every mode; under `Dps` it still keeps the inbound-ack endpoint's known-cluster gate current even though nothing dials out. |
**Addresses only — no actor-path suffix.** `/system/receptionist` is appended at construction time; a contact carrying `/user/` or `/system/` is rejected at startup, because a doubled suffix produces a path that resolves to nothing and drops every command in silence.
**It is a dark switch, and both sides are always wired.** The comm actors are spawned and registered with the `ClusterClientReceptionist` on every node in *both* modes, so flipping `Mode` is a config change rather than a redeploy. Under `Dps` the registration is simply idle.
**Central's contact set comes from the enabled, non-maintenance `ClusterNode` rows** — the same source Phase 1 gave the deploy ack set. A node in `MaintenanceMode` is not dialled, but note that **the contact set does not scope delivery**: a receptionist serves its whole cluster, so `SendToAll` still reaches every registered node exactly as today's DPS broadcast does. The filter is about which receptionists are worth dialling, not about who receives.
**Buffering is off by design** (`akka.cluster.client.buffer-size = 0`, set in the shipped `akka.conf`). If no receptionist is known, the command is dropped rather than queued. Akka's default of 1000 would replay a `DispatchDeployment` on reconnect — silently, on the client's own timer, at an unbounded remove from the dispatch. Do not "fix" it back to quiet a flaky test.
> **What this does *not* buy you.** It does not guarantee a node never applies a deployment the DB records as `TimedOut`. A node that missed a dispatch keeps the orphan `Applying` row the coordinator pre-created, and `DriverHostActor.Bootstrap()` replays that row on the node's next restart — live-observed applying a deployment 44 s after it was sealed `TimedOut` (see the [Phase 2 live gate](plans/2026-07-22-mesh-phase2-live-gate.md), finding B). The boot replay is the better-behaved of the two — deliberate, Warning-logged, and bounded to a restart — but it is still a replay. `buffer-size = 0` removes the *silent, timer-driven* one only.
On the docker-dev rig every node carries these keys already; flip the whole rig with `OTOPCUA_MESH_MODE=ClusterClient` at `docker compose up`.
**`Grpc` mode — shipped dark, cutover pending** ([ClusterClient→gRPC migration design](plans/2026-08-06-clusterclient-to-grpc-design.md)). A third `Mode` value replaces `ClusterClient` with unary gRPC calls on the `mesh_transport.v1` contract: central dials each node's `NodeCommandService`, a node posts its deployment ack to central's `CentralCommandService`. Unlike `ClusterClient`, the command plane is authenticated end to end (`GrpcPsk` below) rather than relying on the mesh boundary alone. As of Phase 0 only the contract, DTO mapper, options and the two auth interceptors exist — the compiled default and the docker-dev rig both stay on `ClusterClient`, and setting `Mode=Grpc` today validates the keys below but nothing yet dials or serves over the contract.
**`Grpc` mode: central dials each node, a node posts its ack to central.** `CentralCommunicationActor` fans one `NodeCommandService.Execute` call per node — the target set is every enabled, non-maintenance `ClusterNode` row that carries a `GrpcPort` (dialled at `Host`+`GrpcPort`), rebuilt every `ContactRefreshSeconds`. There is no receptionist and no single fleet-wide (or per-cluster) client object: it is a plain per-node dial. A node posts its `ApplyAck` with `GrpcApplyAckSender`, which walks `CentralGrpcEndpoints` in order and stops at the first success — an availability list, not a fan-out list, so central normally sees one ack per node per dispatch (the walk as a whole is at-least-once on a slow-then-retried endpoint; `ConfigPublishCoordinator.HandleAck` absorbs a duplicate idempotently). Both directions are **fire-and-forget, with no retry**: a failed dispatch is logged (naming the node and the gRPC status) and dropped in the dispatch second — the deploy path's 2-minute apply deadline and a node's boot-time replay (`DriverHostActor.Bootstrap()` re-applying an orphan `Applying` row) are the only recovery mechanisms, exactly as they were under `ClusterClient`.
| Key | Type | Default | Meaning |
|---|---|---|---|
| `GrpcPsk` | string? | `null` (unset) | Node-side only, required under `Grpc` on a driver-role node: this node's pre-shared key for its own `Cluster`, presented as the bearer on outbound acks and required on inbound `NodeCommandService` calls. Production form is `${secret:OT-MESH-PSK-<clusterId>}`; supply a literal only on a rig, via `MeshTransport__GrpcPsk` — never commit one. |
| `CentralGrpcEndpoints` | string[] | `[]` | Node-side only, required under `Grpc` on a driver-role node: full gRPC base URLs (`scheme://host:port`, e.g. `http://central-1:4055` for h2c) of the central pair's `CentralCommandService`, tried in order, first success wins. The `Grpc` counterpart of `CentralContactPoints`, and replaces it — a node on `Grpc` never reads the Akka contact points. |
| `CentralGrpcEndpoints` | string[] | `[]` | Node-side only, required under `Grpc` on a driver-role node: full gRPC base URLs (`scheme://host:port`, e.g. `http://central-1:4055` for h2c) of the central pair's `CentralCommandService`, tried in order, first success wins. There is no Akka-side counterpart any more — the old `MeshTransport:CentralContactPoints` (the `ClusterClient` contact-point list) is deleted along with `ClusterClient` itself, not merely superseded. |
| `DispatchTimeoutSeconds` | int | `10` | Central-side per-node deadline for one `NodeCommandService.Execute` call. Must be positive under `Grpc`; validated on driver-role nodes (covers the central pair, which is always fused). |
| `AckTimeoutSeconds` | int | `10` | Node-side per-endpoint deadline for one `CentralCommandService.ReportApply` call. Must be positive under `Grpc`. Applied per endpoint walked in `CentralGrpcEndpoints`, not per ack, so a dead first endpoint costs this deadline before the second is tried. |
| `ClusterPsks` | `Dictionary<string,string>` | `{}` (case-insensitive keys) | **Central-side only** override map, keyed by `Cluster` id: `IMeshPskProvider` consults this map *before* the Secrets store when central builds a channel to (or verifies an ack from) a given cluster. Production leaves this empty and resolves `OT-MESH-PSK-{clusterId}` from Secrets instead; the map exists so the docker-dev rig and integration tests can supply literals via the environment (`MeshTransport__ClusterPsks__SITE-A`) without standing up a KEK. |
**Auth is per-cluster PSK bearer, both directions, fail-closed.** `GrpcPsk` (node-side) and `ClusterPsks` (central-side) are two ends of the same key, one per application `Cluster` — not a fleet-wide shared bearer, because this channel carries `RestartDriver`, `AlarmCommand` and deployment dispatch, and a single fleet key would let one compromised site node command every other site or forge acks fleet-wide. Both `MeshCommandAuthInterceptor` (node-side, gates `NodeCommandService`) and `CentralCommandAuthInterceptor` (central-side, gates `CentralCommandService`) reject every call when no key is configured for the relevant cluster, compare in constant time, and never log key material. **PSK seeding:** production form is a secret reference, `${secret:OT-MESH-PSK-<clusterId>}` (e.g. `${secret:OT-MESH-PSK-SITE-A}`), resolved by the Secrets store so it works on a driver-only node with no ConfigDb; the docker-dev rig instead supplies three literal dev keys via environment — `dev-mesh-psk-MAIN`, `dev-mesh-psk-SITE-A`, `dev-mesh-psk-SITE-B` — one per cluster, node-side via `MeshTransport__GrpcPsk` and central-side via `MeshTransport__ClusterPsks__<clusterId>`. Never commit a literal key.
**`ClusterPsks` / `OT-MESH-PSK-<clusterId>` and `SecretName`'s allow-list.** `SecretName` (`ZB.MOM.WW.Secrets.Abstractions`) lowercases the secret name and restricts it to `[a-z0-9._/-]` — a `Cluster` id that carries any character outside that set cannot be expressed as `OT-MESH-PSK-<clusterId>` at all, so `MeshPskProvider.TryGetPsk` fails closed for that cluster on every call that reaches the store (`SecretName`'s constructor throws, `MeshPskProvider.ResolveFromStore` catches it, logs a warning, and returns no key) — `MeshTransport:ClusterPsks` is the only path that still works for such an id. The rig's three ids (`MAIN`, `SITE-A`, `SITE-B`) are all within the allow-list and resolve normally either way.
The `Grpc` mode's node-side comm actors and central-side services are not yet wired to `Mode` — Phase 0 lands only the contract, mapper, options and the two auth interceptors (`MeshCommandAuthInterceptor` node-side, `CentralCommandAuthInterceptor` central-side), both dark. See the [design doc](plans/2026-08-06-clusterclient-to-grpc-design.md) and [tracking doc](plans/2026-08-06-clusterclient-to-grpc-tracking.md) for the remaining phases.
**Both command-plane gRPC services ride an existing listener — neither claims a port of its own.** `NodeCommandService` is mapped alongside `TelemetryStreamGrpcService` on the driver-role node's `Telemetry:GrpcListenPort` listener (the docker-dev rig's `:4056`) whenever that port is bound, deliberately NOT gated on `MeshTransport:Mode` — mapping it under `Dps` costs nothing (nothing dials it), whereas gating it would mean a running fleet needs a central restart before any node could ack after flipping the mode. `CentralCommandService` (the node→central ack ingress) is mapped alongside `DeploymentArtifactService` on the admin-role node's `ConfigServe:GrpcListenPort` listener (the rig's `:4055`), for the same reason — see [`docs/Telemetry.md`](Telemetry.md) for the shared driver-role listener and [§ `ConfigSource`/`ConfigServe`](#configsource--configserve-config-fetch-and-cache) for the shared admin-role one.
On the docker-dev rig every node carries these keys already; the rig's compose default is `Grpc` (`OTOPCUA_MESH_MODE:-Grpc`, cutover commit `caf2eec6`) — the compiled class default in `MeshTransportOptions` stays `Dps`, so an unconfigured deployment outside the rig still needs `Mode=Grpc` set explicitly. Every rig node carries a `cluster-{ClusterId}` role (Phase 6), so `OTOPCUA_MESH_MODE=Dps` is not a usable override on this rig as shipped — `SplitTopologyTransportValidator` refuses to boot any of them on `Dps`.
See the [ClusterClient→gRPC migration design](plans/2026-08-06-clusterclient-to-grpc-design.md), [implementation plan](plans/2026-08-06-clusterclient-to-grpc.md), [tracking doc](plans/2026-08-06-clusterclient-to-grpc-tracking.md) and [live gate](plans/2026-08-06-clusterclient-to-grpc-live-gate.md) for the migration's phase-by-phase history — those are historical records of the cutover and are not updated further by this sweep.
### `ConfigSource` / `ConfigServe` (config fetch-and-cache)
@@ -284,7 +277,7 @@ non-empty `ApiKey`** — fail-closed, same as `ConfigServe`/`LocalDb:Replication
`redundancy-state` (pair-local, built from `Cluster.State`, stays on DPS in both modes — see
[`Redundancy.md`](Redundancy.md#command-transport-centralnode)), `fleet-status` (central-internal;
`Fleet.razor` polls the Config DB, not any stream), and `deployment-acks` (already rides the Phase 2
`ClusterClient` transport as a command-plane reply). See
gRPC command transport, `CentralCommandService.ReportApply`, as a command-plane reply). See
[`docs/Telemetry.md`](Telemetry.md) for the full architecture, the reconnect/generation-stamped
dialer story, and the rationale for the node-hosts/central-dials inversion.