docs(plans): ClusterClient→gRPC migration program — design, implementation plan, live-gate template, central tracking doc
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,405 @@
|
||||
# OtOpcUa: migrating cross-mesh ClusterClient traffic to gRPC-only
|
||||
|
||||
**Date:** 2026-08-06. Code-verified inventory of every ClusterClient touchpoint in
|
||||
`~/Desktop/OtOpcUa`, then the migration assessment. **Reference implementation:** ScadaBridge
|
||||
shipped the identical migration 2026-07-22→08-01 (design
|
||||
`~/Desktop/scadaproj/scadabridge_clusterclient_to_grpc.md`, plan + live gate under
|
||||
`~/Desktop/ScadaBridge/docs/plans/2026-07-22-clusterclient-to-grpc-*`) — live-gated end to end,
|
||||
ClusterClient deleted. OtOpcUa runs on the same cluster boxes, so this migration converges both
|
||||
apps on one WAN boundary posture: gRPC + PSK, no Akka remoting between site and central.
|
||||
|
||||
Companion docs: `docs/plans/2026-07-21-per-cluster-mesh-design.md` (why the boundary is
|
||||
per-cluster meshes), `2026-07-22-mesh-phase2-clusterclient-transport.md` (how ClusterClient got
|
||||
here), `2026-07-24-mesh-phase6-partition-and-colocation.md` (the mesh split this completes).
|
||||
|
||||
Everything below was code-verified 2026-08-06. If a cited line has drifted, re-locate by the
|
||||
quoted identifier, never by line number.
|
||||
|
||||
---
|
||||
|
||||
## 1. What ClusterClient actually carries today
|
||||
|
||||
Everything crossing the central↔site mesh boundary over Akka funnels through **two actors** —
|
||||
`CentralCommunicationActor` (central) and `NodeCommunicationActor` (node) — so the migration
|
||||
surface is even better-bounded than ScadaBridge's was.
|
||||
|
||||
**Receptionist registrations (2):**
|
||||
|
||||
| Role | Actor | Path | File:line | Note |
|
||||
|---|---|---|---|---|
|
||||
| admin | `CentralCommunicationActor` | `/user/central-communication` | `ControlPlane/ServiceCollectionExtensions.cs:85` | registered unconditionally in both modes (deliberate) |
|
||||
| driver | `NodeCommunicationActor` | `/user/node-communication` | `Runtime/ServiceCollectionExtensions.cs:416` | same |
|
||||
|
||||
Path constants: `Commons/Messages/Mesh/MeshPaths.cs`. Receptionist + DPS extensions are
|
||||
force-started via `Cluster/Resources/akka.conf:16-24`; receptionist role scoping deliberately
|
||||
empty (`akka.conf:113-119`).
|
||||
|
||||
**Client creation (2 sites, asymmetric discovery — documented `Cluster/MeshTransportOptions.cs:16-21`):**
|
||||
|
||||
- **Central:** one `ClusterClient` **per application Cluster**, contacts loaded **from the
|
||||
database** (`ClusterNode` rows `WHERE Enabled AND NOT MaintenanceMode`, grouped by
|
||||
`ClusterId`, `akka.tcp://…/system/receptionist` built from `Host`+`AkkaPort`) —
|
||||
`CentralCommunicationActor.cs:207-245`, refreshed every `MeshTransport:ContactRefreshSeconds`
|
||||
(default 60). Factory seam `IMeshClusterClientFactory` /
|
||||
`DefaultMeshClusterClientFactory` (`ControlPlane/Communication/IMeshClusterClientFactory.cs`).
|
||||
- **Node:** one `central-cluster-client`, contacts from **static config**
|
||||
`MeshTransport:CentralContactPoints[]` — `Runtime/ServiceCollectionExtensions.cs:394-407`,
|
||||
created only under `MeshTransport:Mode=ClusterClient`.
|
||||
|
||||
**Central → node (4 message types), all `ClusterClient.SendToAll("/user/node-communication", …)`
|
||||
fanned across every per-cluster client** (`CentralCommunicationActor.cs:167-168`; `SendToAll`
|
||||
not `Send` is load-bearing — `Send` would deploy to one node only). Node side re-emits on the
|
||||
local `EventStream` (`NodeCommunicationActor.cs:70-74`); subscribers (`DriverHostActor`,
|
||||
`ScriptedAlarmHostActor`) are transport-blind.
|
||||
|
||||
| Message | Purpose | Sender | Backstop layered on top |
|
||||
|---|---|---|---|
|
||||
| `DispatchDeployment` | "revision exists — fetch + apply" (payload-free: id + `RevisionHash` + correlation; artifact bytes ride gRPC config-serve) | `ConfigPublishCoordinator` (admin singleton), `:198` | 2-min apply deadline (`DefaultApplyDeadline`, `:60`, handling `:283-300`) + boot replay of orphan `Applying` rows in `DriverHostActor.Bootstrap()` |
|
||||
| `RestartDriver` | AdminUI restart of one driver instance | `AdminOperationsActor.cs:431` | none — local `RestartDriverResult` answers the UI *before/independent of* node delivery |
|
||||
| `ReconnectDriver` | AdminUI reconnect | `AdminOperationsActor.cs:461` | none |
|
||||
| `AlarmCommand` | operator Ack/Shelve/Unshelve into the scripted-alarm engine | `AdminOperationsActor.cs:122,169` | none |
|
||||
|
||||
**Node → central (1 message type), `ClusterClient.Send("/user/central-communication", …)`**
|
||||
(`Send` not `SendToAll` is load-bearing — `SendToAll` would double-count the node at the
|
||||
coordinator, `NodeCommunicationActor.cs:89-92`):
|
||||
|
||||
| Message | Purpose | Path |
|
||||
|---|---|---|
|
||||
| `ApplyAck` (`DeploymentId, NodeId, Outcome, FailureReason, CorrelationId`) | per-node deploy acknowledgement | `DriverHostActor.SendAck` (`:2488-2507`) → `_ackRouter` = `NodeCommunicationActor` → central `HandleApplyAck` (`:171-187`) → `Forward` to the `ConfigPublishCoordinator` singleton proxy → `PersistNodeAck` (`:247-260`) |
|
||||
|
||||
**The invariant that sizes this whole migration: there is NO Ask across this boundary.** Every
|
||||
message is a one-way Tell; no sender preservation, no reply futures, no Ask timeouts
|
||||
(`NodeCommunicationActor.cs:30-35` states it; code confirms it). The per-cluster mesh design
|
||||
doc's line about "sender preservation is the whole Ask idiom" describes ScadaBridge, not this
|
||||
codebase — do not size off it.
|
||||
|
||||
**Transport guarantees today: none.** `akka.cluster.client.buffer-size = 0` (no buffering, no
|
||||
replay), `reconnect-timeout = off` (`akka.conf:106,111`). No client for a cluster ⇒ the command
|
||||
is logged and dropped (`CentralCommunicationActor.cs:146-158`); no central client ⇒ the ack is
|
||||
dropped (`NodeCommunicationActor.cs:78-87`). Serialization is Akka's default Newtonsoft JSON —
|
||||
**no `serialization-bindings` exist for any mesh message** — under a 128 KB
|
||||
`maximum-frame-size` with drop-don't-teardown semantics (`akka.conf:35-47`). There is no proto
|
||||
contract and no serializer discipline to inherit; the wire surface must be authored fresh.
|
||||
|
||||
**Not ClusterClient (stays untouched):** gRPC config-serve (Phase 3), gRPC telemetry dial
|
||||
(Phase 5), LocalDb sync gRPC, Akka secrets replication (pair-local), the pair-local DPS
|
||||
channels (`alarm-commands`' OPC-UA-originated leg published by `OtOpcUaServerHostedService`,
|
||||
`redundancy-state`, `fleet-status`), and the node-local EventStream delivery leg.
|
||||
|
||||
---
|
||||
|
||||
## 2. Why gRPC-only is a natural completion, not a rewrite
|
||||
|
||||
1. **ClusterClient is the last Akka hop crossing the WAN.** Config artifacts (Phase 3),
|
||||
telemetry (Phase 5) and LocalDb sync already ride gRPC across the boundary. After this
|
||||
migration, Akka remoting is pair-internal only — the exact end state ScadaBridge reached.
|
||||
2. **The surface is tiny.** 5 one-way message types, 2 choke-point actors, no Ask/reply
|
||||
semantics, no ordering or delivery guarantees to preserve beyond "fan to all driver nodes"
|
||||
and "one ack per node." ScadaBridge migrated 35 message types with Ask semantics in ~5
|
||||
weeks; this is a fraction of that.
|
||||
3. **The playbook is proven on the same clusters.** ScadaBridge's design, PSK pattern,
|
||||
coexistence flags, deletion sequencing and live-gate format all shipped and survived their
|
||||
gates. OtOpcUa co-deploys on the same boxes; converging both apps on one boundary posture
|
||||
(gRPC + per-relationship PSK, no cross-WAN Akka) halves the firewall/audit story.
|
||||
4. **Auth is a real gap, not hygiene.** Cross-mesh commands (`RestartDriver`, `AlarmCommand`,
|
||||
`DispatchDeployment`) ride unauthenticated Akka remoting today — anyone who can reach a
|
||||
node's `:4053` and speak the protocol can reach `/user/node-communication`. The gRPC path
|
||||
inherits the family's fail-closed bearer/PSK discipline that already gates the other three
|
||||
channels.
|
||||
5. **Silent drop becomes visible failure.** `buffer-size = 0` means an undeliverable command
|
||||
vanishes with one central-side log line. A unary gRPC call fails *per node, with a status
|
||||
code* — the deploy path's 2-min deadline stops being the only witness.
|
||||
|
||||
---
|
||||
|
||||
## 3. What the migration takes
|
||||
|
||||
### 3.1 Topology — both directions already have a listener to hang off
|
||||
|
||||
This is the OtOpcUa-shaped economy (ScadaBridge had to build central's listener from scratch;
|
||||
OtOpcUa does not):
|
||||
|
||||
- **`NodeCommandService`** (central→node commands) is mapped on the **existing Telemetry
|
||||
listener** (`Telemetry:GrpcListenPort`, h2c, rig `:4056`) that every driver node already
|
||||
hosts. On a split-topology node, `SplitTopologyTransportValidator` already forces
|
||||
`Telemetry:Mode=Grpc`, whose validator forces `GrpcListenPort > 0` — the listener is
|
||||
guaranteed up.
|
||||
- **`CentralCommandService`** (node→central acks) is mapped on the **existing ConfigServe
|
||||
listener** (`ConfigServe:GrpcListenPort`, rig `:4055`) that admin nodes host. A new validator
|
||||
rule ties `MeshTransport:Mode=Grpc` + admin role to `ConfigServe:GrpcListenPort > 0`.
|
||||
|
||||
**Rejected: a dedicated mesh-command port.** It would need a new `ClusterNode` column + EF
|
||||
migration + the seed-backfill `UPDATE` dance (`docker-dev/seed/seed-clusters.sql:126-128`
|
||||
exists precisely because Phase 5's column landed NULL on persisted volumes) + compose churn on
|
||||
six services — for no isolation gain, because the interceptors path-scope shared listeners
|
||||
anyway (three services already coexist in one `AddGrpc` pipeline).
|
||||
|
||||
**Rejected: a node-held long-lived command stream** (node subscribes, central pushes). It
|
||||
re-implements the connection/reconnect management the receptionist gave for free, inverts the
|
||||
existing dial direction convention (central already dials nodes for telemetry), and buys
|
||||
nothing for 4 command types. Same reasoning ScadaBridge recorded for the mirror-image option.
|
||||
|
||||
**Either-pair-node delivery is already safe.** Acks land on whichever central answers; the
|
||||
receiving `CentralCommandService` Tells `CentralCommunicationActor`, which `Forward`s to the
|
||||
`ConfigPublishCoordinator` **singleton proxy** — same intra-pair routing that makes today's
|
||||
receptionist arrival-node-agnostic. Commands land on the *specific* node dialed, which is the
|
||||
required semantic (each node must apply its own deployment).
|
||||
|
||||
### 3.2 Contract
|
||||
|
||||
One new proto: `src/Core/ZB.MOM.WW.OtOpcUa.Commons/Protos/mesh_transport.proto`, package
|
||||
`mesh_transport.v1`, csharp_namespace `ZB.MOM.WW.OtOpcUa.Commons.Protos.MeshTransport.V1`
|
||||
(deliberately NOT `MeshCommand` — that C# name is taken by the existing envelope record
|
||||
`Commons.Messages.Mesh.MeshCommand`).
|
||||
|
||||
```proto
|
||||
service NodeCommandService {
|
||||
// Central → node. One-way semantics: the ack means "received and re-emitted on the
|
||||
// node's EventStream", nothing more. No retry is layered on a failure.
|
||||
rpc Execute (CommandEnvelope) returns (ExecuteAck);
|
||||
}
|
||||
service CentralCommandService {
|
||||
// Node → central. One-way semantics: the ack means "handed to the coordinator proxy".
|
||||
rpc ReportApply (ApplyAckMessage) returns (ReportAck);
|
||||
}
|
||||
message CommandEnvelope {
|
||||
string correlation_id = 1;
|
||||
oneof command {
|
||||
DispatchDeploymentCommand dispatch_deployment = 2;
|
||||
RestartDriverCommand restart_driver = 3;
|
||||
ReconnectDriverCommand reconnect_driver = 4;
|
||||
AlarmCommandMessage alarm_command = 5;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Codegen is **build-time `Grpc.Tools`** exactly like the two existing protos
|
||||
(`Commons.csproj:28-31` `<Protobuf … GrpcServices="Both"/>`; generated code is NOT checked in —
|
||||
OtOpcUa does not have ScadaBridge's linux_arm64 protoc segfault; the rig image build is the
|
||||
verification). Add a `MeshTransportProtoContract` contract-lock helper + tests mirroring
|
||||
`TelemetryProtoContract.cs` / `TelemetryProtoContractTests.cs` so a new oneof case cannot ship
|
||||
unhandled. Mapper: `MeshTransportDtoMapper` in Commons with **round-trip golden tests** for all
|
||||
five messages (the ScadaBridge T1A.1/T1B.1 lesson: nullable members, init-only props and
|
||||
enum-zero placement are where mappers silently drop data — reserve proto enum 0 as
|
||||
`UNSPECIFIED`, never a meaningful value).
|
||||
|
||||
### 3.3 Semantics to preserve (all transport-agnostic, all carry over)
|
||||
|
||||
- **Fire-and-forget stays fire-and-forget.** No retries, no queues, no new delivery
|
||||
guarantees. A failed `Execute` is logged per node with its status code and dropped — an
|
||||
honest upgrade over the silent drop, not a semantic change. The deploy path's backstops
|
||||
(2-min deadline, orphan-`Applying` boot replay) remain the only recovery mechanisms.
|
||||
- **`SendToAll` fan ⇒ dial every enabled, non-maintenance `ClusterNode`.** The recipient set is
|
||||
already DB-derived (`DiscoverDriverNodes` derives the expected-ack set from the same rows) —
|
||||
the gRPC dispatcher uses the same query shape, so dispatch set == expected-ack set by
|
||||
construction.
|
||||
- **`Send` singleton-counting ⇒ one `ReportApply` per node per dispatch.** Unary gives this
|
||||
naturally; the node tries central endpoints in order and stops at the first success.
|
||||
- **Node-side delivery stays the EventStream.** The gRPC service Tells
|
||||
`NodeCommunicationActor` the decoded message; the actor's existing re-emit (`:70-74`) is
|
||||
untouched, so `EventStreamCommandDeliveryTests` and every subscriber remain valid.
|
||||
- **Central-side ack handling stays `HandleApplyAck`.** The gRPC service Tells
|
||||
`CentralCommunicationActor`; the `Forward`-to-proxy and `PersistNodeAck` paths are untouched.
|
||||
- **Fused nodes work unchanged.** central-1/-2 are `admin,driver,cluster-MAIN`: they serve
|
||||
both services, dial their own pair with the MAIN key, and ack to themselves. Nothing special.
|
||||
|
||||
### 3.4 Work you must do anyway: auth — per-cluster PSK from the Secrets store (decided 2026-08-06)
|
||||
|
||||
The command plane is unauthenticated today. The gRPC migration adopts ScadaBridge's
|
||||
live-gate-proven shape, adapted to OtOpcUa's conventions:
|
||||
|
||||
- **Key scope: one PSK per application Cluster** — `OT-MESH-PSK-<clusterId>` (e.g.
|
||||
`OT-MESH-PSK-MAIN`, `OT-MESH-PSK-SITE-A`), covering both directions of that cluster's
|
||||
traffic. **Steelmanned alternative — one fleet-wide shared key** (what the other three
|
||||
OtOpcUa channels do): simpler seeding, but a compromised site node could then issue
|
||||
`RestartDriver`/`AlarmCommand` to *every other site* and forge acks fleet-wide. ScadaBridge
|
||||
rejected the shared key on exactly this blast-radius argument; the command plane is where it
|
||||
bites hardest, so we do too. (The three existing shared-bearer channels are read/telemetry
|
||||
planes; retrofitting them is out of scope.)
|
||||
- **Node side (one key):** new option `MeshTransport:GrpcPsk`, production value
|
||||
`${secret:OT-MESH-PSK-<clusterId>}`, resolved by the **existing pre-host
|
||||
`SecretReferenceExpander`** (`Host/Program.cs:93-112`) — zero new resolution code. Works on
|
||||
driver-only nodes (node-local SQLite secrets store + env KEK; no ConfigDb involved).
|
||||
- **Central dial + serve sides (many keys, cluster set is DB-dynamic):** new
|
||||
`IMeshPskProvider` (ControlPlane) over the runtime `ISecretResolver`, resolving
|
||||
`OT-MESH-PSK-{clusterId}` at channel-build/verify time, **cached per cluster, invalidated on
|
||||
contact refresh, fail-closed** (unresolvable ⇒ deny/skip, never proceed unauthenticated).
|
||||
A `MeshTransport:ClusterPsks:{clusterId}` config map is checked before the Secrets store —
|
||||
the rig/test override, mirroring ScadaBridge's `SitePsks__site-*` env pattern; production
|
||||
leaves it empty.
|
||||
- **Node-side server interceptor** `MeshCommandAuthInterceptor`, cloned from
|
||||
`TelemetryStreamAuthInterceptor`: path prefix `/mesh_transport.v1.NodeCommandService/`,
|
||||
`authorization: Bearer` extraction, `CryptographicOperations.FixedTimeEquals` over UTF-8,
|
||||
**fail-closed when `GrpcPsk` unset**, reject `PermissionDenied` (the 2-of-3 majority
|
||||
convention; `ConfigServeAuthInterceptor`'s `Unauthenticated` is the outlier — don't copy it).
|
||||
**Exactly ONE public constructor** — `Grpc.AspNetCore` silently stops invoking an
|
||||
interceptor with two (documented `TelemetryStreamAuthInterceptor.cs:38-42`, pinned by
|
||||
`HasExactlyOnePublicConstructor`; ScadaBridge lost a live-gate day to this).
|
||||
- **Central-side server interceptor** `CentralCommandAuthInterceptor`: gates
|
||||
`/mesh_transport.v1.CentralCommandService/`; callers send `x-otopcua-cluster: <clusterId>`
|
||||
so central picks which key to verify; missing/unknown cluster ⇒ `PermissionDenied`
|
||||
(fail-closed). **The header is a central-side routing hint only** — per-cluster isolation is
|
||||
proven by wrong-key rejection, not by the header; do not add a header requirement to the
|
||||
node-side interceptor (ScadaBridge's gate recorded that as theatre).
|
||||
- **Client attach:** raw `Metadata` `authorization: Bearer <psk>` per call, the exact pattern
|
||||
of `GrpcDeploymentArtifactFetcher.cs:122` / `TelemetryStreamClient.cs:134`.
|
||||
- **Boot validation:** `MeshTransportOptionsValidator` extended per the house rules (roles from
|
||||
`IConfiguration` via `RoleParser.ResolveEffectiveRoles` — never `IClusterRoleInfo`; require
|
||||
keys only where the mode makes them load-bearing; error messages name the exact key and env
|
||||
form, never echo the value). Under `Mode=Grpc`: driver role ⇒ `GrpcPsk` non-empty +
|
||||
`CentralGrpcEndpoints` non-empty and `http(s)://`-prefixed.
|
||||
- **Transport honesty:** bearer PSK over plaintext h2c is readable/replayable on-path — same
|
||||
accepted posture (network isolation) as the three existing channels, and strictly better
|
||||
than today's unauthenticated remoting. TLS on the internal listeners is follow-on hardening;
|
||||
the PSK design doesn't change if it lands.
|
||||
- **Seeding/rotation:** the `secret` CLI's *Reference audit & seed* flow per store; rotation =
|
||||
set new value both sides, restart the pair (pairs restart together by discipline). Rig uses
|
||||
committed dev literals per its zero-operator-step convention: `dev-mesh-psk-MAIN`,
|
||||
`dev-mesh-psk-SITE-A`, `dev-mesh-psk-SITE-B`.
|
||||
|
||||
### 3.5 Endpoint discovery — both patterns already exist
|
||||
|
||||
- **Central → node:** `ClusterNode.Host` + `GrpcPort` ⇒ `http://{host}:{port}` — the exact
|
||||
`TelemetryNodeSource` pattern, same `Enabled && !MaintenanceMode` filter, refreshed on the
|
||||
existing `MeshTransport:ContactRefreshSeconds` timer that today refreshes Akka contacts. **No
|
||||
new column, no migration, no seed change.**
|
||||
- **Node → central:** new `MeshTransport:CentralGrpcEndpoints[]` (e.g.
|
||||
`["http://central-1:4055", "http://central-2:4055"]`), tried in order per ack — the
|
||||
`GrpcDeploymentArtifactFetcher` endpoint-failover loop shape, channels cached per endpoint.
|
||||
It replaces `CentralContactPoints` (deleted in Phase 3) rather than reusing
|
||||
`ConfigSource:CentralFetchEndpoints`, because a `Direct`-mode fused node legitimately has no
|
||||
ConfigSource section but still acks deployments.
|
||||
- **Deadlines are net-new** (no Ask timeouts existed): `MeshTransport:DispatchTimeoutSeconds`
|
||||
(default 10) on `Execute`, `MeshTransport:AckTimeoutSeconds` (default 10) on `ReportApply`,
|
||||
enforced client-side via linked CTS like the fetcher. Never retry `DeadlineExceeded` — with
|
||||
one-way semantics a late-delivered command is worse than a dropped one only if we retry it
|
||||
into a duplicate.
|
||||
|
||||
### 3.6 The mode switch — third value in, one value out
|
||||
|
||||
`MeshTransport:Mode` gains **`Grpc`**. End state: `Dps` (compiled default; in-mesh/single-mesh
|
||||
topologies where admin and driver share one gossip ring) | `Grpc` (split topologies).
|
||||
`ClusterClient` is **deleted**. The DPS branch is deliberately **retained**: the
|
||||
"Phase 6 deletes the DistributedPubSub branch" comments
|
||||
(`MeshTransportOptions.cs:11-13`, `ConfigPublishCoordinator.cs:70-73`,
|
||||
`AdminOperationsActor.cs:90-92`) describe a deletion that never happened and contradict
|
||||
`docs/Configuration.md:150-156`, which records the deliberate reversal — the comments get
|
||||
corrected in Phase 0. `SplitTopologyTransportValidator` accepts `{ClusterClient, Grpc}` during
|
||||
the migration (Phase 0 relaxation — without it, flipping any rig node to `Grpc` fails boot) and
|
||||
requires `Grpc` from Phase 3.
|
||||
|
||||
The coexistence granularity falls out of where the switch is read: **central's mode governs the
|
||||
command direction, each node's mode governs its ack direction** — so a single site pair can be
|
||||
flipped to gRPC acks while everything else stays on ClusterClient, and vice versa. (Better than
|
||||
ScadaBridge's central-wide `SiteTransport`, whose per-site coexistence gate had to be recorded
|
||||
as unachievable.)
|
||||
|
||||
### 3.7 Deletions at the end
|
||||
|
||||
`DefaultMeshClusterClientFactory` + `IMeshClusterClientFactory`, both receptionist
|
||||
registrations, node client creation (`Runtime/ServiceCollectionExtensions.cs:394-407`) and
|
||||
central `RebuildClient`/per-cluster client cache (**keep `LoadContactsFromDb` — it now feeds
|
||||
the gRPC dialer's node list**), `akka.conf`'s `ClusterClientReceptionistExtensionProvider`
|
||||
line + the whole `akka.cluster.client` block (**keep the DPS extension** — pair-local channels
|
||||
live), `MeshTransport:CentralContactPoints` + its validator rules + rig env, the
|
||||
`ClusterClient` mode value, and the ClusterClient-shaped tests
|
||||
(`MeshClusterClientBoundaryTests` is *replaced* by a gRPC-boundary equivalent — the "no Akka
|
||||
association crosses the boundary" assertion is the point of the whole program and must survive
|
||||
as a test). `Akka.Cluster.Tools` package refs **stay** (ClusterSingleton + DPS).
|
||||
|
||||
### 3.8 Deliberately out of scope
|
||||
|
||||
- **`FleetStatusBroadcaster` is silently central-pair-blind post-Phase-6** (derives "fleet"
|
||||
from central's own gossip ring). Real, pre-existing, transport-independent — filed as its own
|
||||
issue in Phase 0, not folded in.
|
||||
- **Cross-mesh `redundancy-state` / `fleet-status` visibility** — deferred by Phase 5, stays
|
||||
deferred; this migration adds no new cross-mesh channels.
|
||||
- **`alarm-commands`' pair-local DPS leg** (OPC UA Part 9 method calls published node-locally
|
||||
by `OtOpcUaServerHostedService:158-159`) — untouched; it never crosses meshes.
|
||||
- **Delivery-guarantee upgrades** (queuing, retry, dedup) — explicitly not this program.
|
||||
- **Retrofitting per-cluster PSKs onto the three existing shared-bearer channels** — follow-on.
|
||||
|
||||
---
|
||||
|
||||
## 4. Code-verified findings this design is sized on (plan-vs-code corrections)
|
||||
|
||||
Baked in at authoring time rather than discovered mid-implementation (the ScadaBridge §7
|
||||
lesson, run as a three-track inventory before writing this doc):
|
||||
|
||||
1. **No Ask exists across the boundary** — the per-cluster mesh design doc's sender-preservation
|
||||
paragraph describes ScadaBridge. OtOpcUa's surface is 4+1 one-way Tells. (Also: its line
|
||||
listing "queries, subscribe handshakes" over ClusterClient — none exist.)
|
||||
2. **"Phase 6 deletes the DPS branch" never happened** and the compiled default is still `Dps`;
|
||||
three source comments assert the opposite of `docs/Configuration.md:150-156`. This design
|
||||
keeps DPS for in-mesh topologies and deletes only `ClusterClient`; the comments get fixed.
|
||||
3. **There is no `ICommandTransport` seam.** The seam is the `MeshCommand` envelope + the
|
||||
nullable `meshRouter`/`centralClient`/`ackRouter` ctor params. The gRPC branch lands inside
|
||||
the two choke-point actors (a third mode branch), NOT as a new polymorphic interface — the
|
||||
same "no interface extraction" economy ScadaBridge's §7.1 landed on.
|
||||
4. **`ConfigPublishCoordinator` subscribes to the `deployment-acks` DPS topic unconditionally**
|
||||
(`:132`) in both modes — it is the `Dps`-mode ack ingress and must survive; do not "clean it
|
||||
up" while deleting ClusterClient.
|
||||
5. **`FleetStatusBroadcaster` was left behind by Phase 6** — see §3.8; issue, not scope.
|
||||
6. **Interceptor multi-constructor trap and enum-zero mapping traps** are pre-known from
|
||||
ScadaBridge's live gate and OtOpcUa's own `TelemetryStreamAuthInterceptor.cs:38-42` — both
|
||||
are encoded as tests in the plan, not left to be rediscovered.
|
||||
|
||||
---
|
||||
|
||||
## 5. The other knob: keep ClusterClient (the do-nothing steelman)
|
||||
|
||||
ClusterClient works today: Phase 7 ran 6/6 failover drills through it, `SendToAll` fan-out and
|
||||
DB-driven contacts are debugged, and the 5-message surface is small enough that "it's a lot of
|
||||
transport" is not an argument. Keeping it costs nothing *now*.
|
||||
|
||||
What it leaves on the table: (1) the command plane stays **unauthenticated** across the WAN —
|
||||
the one boundary path with no auth story, in the app whose sister already closed the identical
|
||||
gap; (2) two transport postures on one set of boxes — every firewall rule, packet capture and
|
||||
incident runbook must reason about Akka remoting *and* gRPC crossing the same boundary;
|
||||
(3) `buffer-size=0` silent drops stay silent; (4) Akka.NET upstream has signalled the
|
||||
ClusterClient/receptionist pair as legacy — ScadaBridge's design recorded the same
|
||||
deprecation-direction argument and it has only aged one way.
|
||||
|
||||
**Verdict:** the auth gap alone justifies Phase 0-style hardening, and once auth + contract
|
||||
exist, the remaining migration is small. Do the whole thing.
|
||||
|
||||
---
|
||||
|
||||
## 6. Options
|
||||
|
||||
- **A — full migration (recommended):** contract + PSK + both directions behind
|
||||
`MeshTransport:Mode=Grpc`, coexistence-flagged, then delete ClusterClient. ~2 weeks of
|
||||
focused work given the 5-message surface and the existing listeners/patterns.
|
||||
- **B — auth-only:** bolt a PSK story onto ClusterClient (there isn't a natural one — Akka
|
||||
remoting auth means mTLS on the transport, a different and larger program) or accept the gap.
|
||||
Rejected: the "auth" half of the work has no cheap ClusterClient-shaped form.
|
||||
- **C — do nothing:** see §5.
|
||||
|
||||
**Recommendation: A.**
|
||||
|
||||
---
|
||||
|
||||
## 7. Phasing and effort
|
||||
|
||||
| Phase | Content | Est. | Parallel with |
|
||||
|---|---|---|---|
|
||||
| 0 | Contract (proto + mappers + goldens), auth interceptors + `IMeshPskProvider`, options/validators (+`Grpc` value accepted, SplitTopology relaxation), rig PSK seeding, stale-comment fixes, FleetStatus issue filed | 2–3 d | 1A/1B prep |
|
||||
| 1A | Central→node command plane behind central's `Mode=Grpc` (worktree A) | 3–4 d | 1B |
|
||||
| 1B | Node→central ack plane behind node's `Mode=Grpc` (worktree B) | 2–3 d | 1A |
|
||||
| 2 | Full-rig flip + soak + kill drills (live gate) | 1–2 d | — |
|
||||
| 3 | Deletion + validator tightening + docs sweep | 1–2 d | — |
|
||||
| 4 | Terminal live gate on the post-deletion build | 1 d | — |
|
||||
|
||||
Critical path ≈ 0 → 1A → 2 → 3 → 4: **~1.5–2.5 weeks**, one-third of ScadaBridge's, which is
|
||||
what a 5-message no-Ask surface with pre-built listeners should cost.
|
||||
|
||||
**Risks, honestly:** the mapper/enum-zero class of bug (mitigated by goldens + contract-lock
|
||||
tests, but ScadaBridge still found five of these in implementation); the Kestrel
|
||||
explicit-endpoints trap does NOT apply (no new listeners — but any future "just add a port"
|
||||
deviation re-imports it, see `Program.cs:564-583` before touching anything there); PSK cache
|
||||
invalidation on cluster add/remove is the one genuinely new stateful component
|
||||
(`IMeshPskProvider`) — its fail-closed behavior needs adversarial tests, not just happy-path.
|
||||
|
||||
Implementation plan: `2026-08-06-clusterclient-to-grpc.md`. Live gate:
|
||||
`2026-08-06-clusterclient-to-grpc-live-gate.md`. Central tracking:
|
||||
`2026-08-06-clusterclient-to-grpc-tracking.md`.
|
||||
@@ -0,0 +1,142 @@
|
||||
# ClusterClient → gRPC migration — live gate record
|
||||
|
||||
Rig: `docker-dev/` (central-1/-2 = `admin,driver,cluster-MAIN`; site-a-1/-2, site-b-1/-2;
|
||||
sql + traefik + migrator + cluster-seed), rebuilt from the branch under test via
|
||||
`docker compose -f docker-dev/docker-compose.yml up -d --build`. Recorded check-by-check in
|
||||
the family's live-gate format. Later phases' sections are appended as those phases land; every
|
||||
check row gets a **PASS**/**FAIL** verdict plus one quantitative justification clause, with
|
||||
fenced evidence (real commands, real output). A run invalidated by a concurrent rig rebuild is
|
||||
**void** — note it and re-run (the ScadaBridge void-run rule).
|
||||
|
||||
Provenance discipline: every phase section opens with branch + commit SHA + "image rebuilt,
|
||||
all N containers recreated", and Phase 4 additionally verifies the image build timestamp
|
||||
post-dates the deletion merge (a stale image invalidates the boundary checks).
|
||||
|
||||
Plan: `2026-08-06-clusterclient-to-grpc.md`. Tracking:
|
||||
`2026-08-06-clusterclient-to-grpc-tracking.md`.
|
||||
|
||||
---
|
||||
|
||||
## Baseline (run on the PRE-change build, before the Phase 2 flip)
|
||||
|
||||
The two gaps this program closes, **reproduced rather than assumed**:
|
||||
|
||||
**(a) Unauthenticated Akka remoting crosses the WAN boundary.** From the host:
|
||||
|
||||
```bash
|
||||
# A live TCP association from a site container to central's Akka port:
|
||||
docker exec otopcua-dev-site-a-1-1 bash -c 'exec 3<>/dev/tcp/central-1/4053 && echo OPEN'
|
||||
# And the receptionist actually accepting cross-mesh traffic (central log after any AdminUI
|
||||
# driver restart targeting site-a):
|
||||
docker logs otopcua-dev-central-1-1 2>&1 | grep -i "clusterclient\|receptionist" | tail -5
|
||||
```
|
||||
|
||||
Expected: `OPEN`; receptionist/client lines present. Nothing on this path presents any
|
||||
credential.
|
||||
|
||||
**(b) Silent drop on an unreachable cluster.** Stop the site-b pair, dispatch a deployment
|
||||
(`POST http://localhost:9200/api/deployments`, `X-Api-Key: docker-dev-deploy-key`), observe:
|
||||
no per-node dispatch error surfaces anywhere; site-b's `NodeDeploymentState` rows sit
|
||||
`Applying` until the 2-minute deadline marks the deployment `TimedOut`/partially failed. The
|
||||
deadline is the only witness. Restart the pair.
|
||||
|
||||
- [ ] Baseline (a) recorded with evidence — **PENDING**
|
||||
- [ ] Baseline (b) recorded with evidence — **PENDING**
|
||||
|
||||
---
|
||||
|
||||
## Phase 0 — contract + auth + config groundwork — **PENDING**
|
||||
|
||||
Branch `feat/grpc-phase0-contract-auth` @ `________`. Image rebuilt, all containers recreated.
|
||||
|
||||
| # | Check | Result |
|
||||
|---|---|---|
|
||||
| 1 | All six nodes boot on `ClusterClient` with the dark `Grpc` config present (PSKs, endpoints, ClusterPsks map); no validator fires | PENDING |
|
||||
| 2 | Deliberate misconfig: a scratch container with `MeshTransport__Mode=Grpc` and **no** `GrpcPsk` refuses to boot, error names `MeshTransport:GrpcPsk` + the env form | PENDING |
|
||||
| 3 | Same with empty `CentralGrpcEndpoints` — refuses, names the key | PENDING |
|
||||
| 4 | Full suite green, 0 warnings, on the branch | PENDING |
|
||||
|
||||
Note: the new gRPC services are not yet mapped in Phase 0, so no wire probes here — the
|
||||
interceptor negative/positive matrix is unit-tested and wire-probed in 1A/1B.
|
||||
|
||||
---
|
||||
|
||||
## Phase 1A — central→node command plane — **PENDING**
|
||||
|
||||
Branch `feat/grpc-node-command` @ `________`. Rig-only flip: central-1/-2 →
|
||||
`MeshTransport__Mode=Grpc`; **site nodes remain `ClusterClient`** (coexistence is the point).
|
||||
Flip reverted after the gate, never committed.
|
||||
|
||||
| # | Check | Result |
|
||||
|---|---|---|
|
||||
| 1 | Deploy dispatch from AdminUI reaches **all six** driver nodes over gRPC (per-node `Execute` log lines on every node; zero `SendToAll` lines on central) | PENDING |
|
||||
| 2 | Acks still return over ClusterClient (site mode unchanged) and the deployment **seals** — mixed-transport deploy proven end to end | PENDING |
|
||||
| 3 | `RestartDriver` + `ReconnectDriver` from AdminUI land on the target site node (EventStream delivery observed via driver restart in logs) | PENDING |
|
||||
| 4 | `AlarmCommand` (ack a scripted alarm from AdminUI) lands and the condition acknowledges | PENDING |
|
||||
| 5 | Unauthenticated `grpcurl` to a site node `:4056` `mesh_transport.v1.NodeCommandService/Execute` ⇒ `PermissionDenied`; correct dev PSK ⇒ accepted | PENDING |
|
||||
| 6 | **Wrong-key isolation:** site-b's PSK presented to site-a ⇒ `PermissionDenied` (per-cluster scoping is real, not decorative) | PENDING |
|
||||
| 7 | A node with NULL `GrpcPort` is skipped with the warn-once, fan continues to the others | PENDING |
|
||||
| 8 | Telemetry + config-serve + LocalDb sync unaffected: 0 auth failures on their interceptors across the run | PENDING |
|
||||
|
||||
---
|
||||
|
||||
## Phase 1B — node→central ack plane — **PENDING**
|
||||
|
||||
Branch `feat/grpc-central-ack` @ `________` (rebased on merged 1A). Rig-only flip:
|
||||
site-a-1/-2 → `Grpc`; site-b stays `ClusterClient`; central back on `ClusterClient` mode
|
||||
(its command direction) — acks are governed by the **node's** mode, so this isolates 1B.
|
||||
|
||||
| # | Check | Result |
|
||||
|---|---|---|
|
||||
| 1 | Deploy e2e: site-a acks arrive via `ReportApply` (central `CentralCommandService` log lines carry site-a node ids), `NodeDeploymentState` rows land, deployment seals | PENDING |
|
||||
| 2 | Site-b acks still arrive via ClusterClient in the same deployment — per-pair coexistence proven | PENDING |
|
||||
| 3 | Exactly **one** ack row per site-a node per dispatch (no double-count from endpoint failover) | PENDING |
|
||||
| 4 | Kill central-1 before the apply completes: site-a acks fail over to central-2's endpoint; deployment still seals | PENDING |
|
||||
| 5 | Unauthenticated / wrong-cluster-key / missing `x-otopcua-cluster` probes against central `:4055` `CentralCommandService/ReportApply` ⇒ `PermissionDenied` (missing-header rejection is central-side by design) | PENDING |
|
||||
| 6 | ConfigServe (`DeploymentArtifactService`) on the same listener unaffected: fetches succeed, 0 auth failures | PENDING |
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 — full-rig cutover + soak — **PENDING**
|
||||
|
||||
Rig default flipped and committed: `MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Grpc}"`.
|
||||
Branch/SHA `________`. **Run the Baseline section first on the pre-flip build.**
|
||||
|
||||
| # | Check | Result |
|
||||
|---|---|---|
|
||||
| 1 | Full rebuild all-`Grpc`: six nodes ready, deploy e2e seals, all acks via gRPC | PENDING |
|
||||
| 2 | Command matrix: Restart/Reconnect/AlarmCommand against site-a and site-b nodes | PENDING |
|
||||
| 3 | Kill site-a-1 mid-dispatch: per-node `Execute` failure **logged with node id + status** (the drop is visible now); no retry storm (exactly one attempt per node per dispatch); deployment outcome honest (deadline path) | PENDING |
|
||||
| 4 | Kill central-1: site acks fail over to central-2; restart central-1: rig converges, no manual action | PENDING |
|
||||
| 5 | Stop the site-b pair entirely, dispatch: per-node dispatch failures logged immediately (contrast with Baseline (b)'s silence); deadline backstop still marks the deployment; **no duplicate applies** after the pair returns (boot-replay path only) | PENDING |
|
||||
| 6 | **Zero ClusterClient activity under `Grpc`:** `docker logs <every node> \| grep -ci "clusterclient"` = 0 new lines post-boot | PENDING |
|
||||
| 7 | Soak ≥ 2 h under periodic deploys + command traffic: 0 `PermissionDenied`/`Unauthenticated` across all six nodes (PSK drift detector), 0 unexplained `Unavailable` | PENDING |
|
||||
| 8 | Full suite green on the cutover SHA | PENDING |
|
||||
|
||||
---
|
||||
|
||||
## Phase 4 — terminal gate on the post-deletion build — **PENDING**
|
||||
|
||||
`master` @ `________` (after the Phase 3 deletion merge). **Provenance guard:** image build
|
||||
timestamp verified LATER than the merge commit time — a pre-deletion image would invalidate
|
||||
checks 6–7.
|
||||
|
||||
| # | Check | Result |
|
||||
|---|---|---|
|
||||
| 1 | PSK negatives on both services (no key / wrong key / wrong cluster's key); **unset server key ⇒ all calls rejected** (fail-closed proven live) | PENDING |
|
||||
| 2 | Command matrix all six nodes (deploy dispatch + Restart + Reconnect + AlarmCommand) | PENDING |
|
||||
| 3 | Ack matrix: every node acks exactly once, deployment seals | PENDING |
|
||||
| 4 | Failover drills: central-kill ack failover; site-node-kill dispatch honesty (one logged failure, no retry) | PENDING |
|
||||
| 5 | Mid-deploy central kill + restart: deadline/boot-replay backstops hold, ack counts exact (no loss, no dupes across the restart) | PENDING |
|
||||
| 6 | **Boundary check: no Akka association between any site container and central** — the Baseline (a) probe now shows no cross-mesh `:4053` traffic (pair-internal remoting only) | PENDING |
|
||||
| 7 | Full-rig restart (pairs together): clean convergence; `grep -ri "receptionist\|clusterclient"` over all node logs = 0 lines | PENDING |
|
||||
| 8 | `MeshGrpcBoundaryTests` + full suite green on the merge SHA | PENDING |
|
||||
|
||||
### Carry-forwards
|
||||
|
||||
Anything a phase gate could not drive gets named here with why + what proves it instead, and
|
||||
is re-checked in the next gate (the ScadaBridge proportionate-gate rule). Currently: none.
|
||||
|
||||
### Defects the gates caught that the suite did not
|
||||
|
||||
Recorded per phase as they occur, with root cause + the pinned regression test.
|
||||
@@ -0,0 +1,81 @@
|
||||
# ClusterClient → gRPC migration — CENTRAL TRACKING DOCUMENT
|
||||
|
||||
**This is the single place to see program state.** Update it at every phase boundary, merge,
|
||||
defect, and decision. Everything else (design, plan, live gate) is referenced from here.
|
||||
|
||||
**Program goal:** all central↔site mesh traffic on gRPC with per-cluster PSK auth;
|
||||
`ClusterClient`/`ClusterClientReceptionist` deleted; Akka remoting never crosses the mesh
|
||||
boundary. Mirrors ScadaBridge's completed 2026-07-22 migration (same cluster boxes, same end
|
||||
posture).
|
||||
|
||||
## Document map
|
||||
|
||||
| Doc | Role |
|
||||
|---|---|
|
||||
| [`2026-08-06-clusterclient-to-grpc-design.md`](2026-08-06-clusterclient-to-grpc-design.md) | Design: inventory, rationale, decisions (§3), plan-vs-code corrections (§4), steelman (§5) |
|
||||
| [`2026-08-06-clusterclient-to-grpc.md`](2026-08-06-clusterclient-to-grpc.md) | Implementation plan: phases 0/1A/1B/2/3/4, task metadata, Opus-subagent execution rules, gotchas |
|
||||
| [`2026-08-06-clusterclient-to-grpc.md.tasks.json`](2026-08-06-clusterclient-to-grpc.md.tasks.json) | Machine task tracker (executing-plans resume point) |
|
||||
| [`2026-08-06-clusterclient-to-grpc-live-gate.md`](2026-08-06-clusterclient-to-grpc-live-gate.md) | Live gate record: baseline reproduction + per-phase check tables |
|
||||
| ScadaBridge reference | `~/Desktop/scadaproj/scadabridge_clusterclient_to_grpc.md` + `~/Desktop/ScadaBridge/docs/plans/2026-07-22-clusterclient-to-grpc-{plan,live-gate}.md` |
|
||||
|
||||
## Phase status board
|
||||
|
||||
| Phase | Scope | Branch | Status | PR / merge SHA | Live gate |
|
||||
|---|---|---|---|---|---|
|
||||
| 0 | Proto + mappers + goldens; both auth interceptors; `IMeshPskProvider`; options/validators (+`Grpc` value, SplitTopology relaxation); rig dark config; stale-comment fixes; FleetStatus issue filed | `feat/grpc-phase0-contract-auth` | **NOT STARTED** | — | PENDING |
|
||||
| 1A | Central→node command plane (`NodeCommandService` on the Telemetry listener; `MeshGrpcCommandDialer`; third mode branch in `CentralCommunicationActor`) — **worktree A** | `feat/grpc-node-command` | NOT STARTED | — | PENDING |
|
||||
| 1B | Node→central ack plane (`CentralCommandService` on the ConfigServe listener; `GrpcApplyAckSender`; third branch in `NodeCommunicationActor`) — **worktree B**, rebases on 1A | `feat/grpc-central-ack` | NOT STARTED | — | PENDING |
|
||||
| 2 | Baseline reproduction on pre-flip build, then full-rig cutover (`OTOPCUA_MESH_MODE` default → `Grpc`) + soak + kill drills | (rig commit) | NOT STARTED | — | PENDING |
|
||||
| 3 | Delete ClusterClient (code/HOCON/config/tests); SplitTopology requires `Grpc`; grep-gates; docs sweep incl. scadaproj index | `feat/grpc-phase3-deletion` | NOT STARTED | — | — |
|
||||
| 4 | Terminal live gate on the post-deletion image (8 checks, provenance-guarded) | (on `master`) | NOT STARTED | — | PENDING |
|
||||
|
||||
**Execution rules recap:** phase order 0 → (1A ∥ 1B) → 2 → 3 → 4; 1A∥1B only in separate git
|
||||
worktrees, one Opus subagent per worktree, sequential tasks within a worktree; 1A merges
|
||||
first, 1B rebases (union-resolve `Host/Program.cs` map block, compose env,
|
||||
`docs/Configuration.md`); rig mode flips before Phase 2 are never committed; **STOP at every
|
||||
phase DoD — PR + merge are user-owned.**
|
||||
|
||||
## Decision log
|
||||
|
||||
| Date | Decision | Where recorded |
|
||||
|---|---|---|
|
||||
| 2026-08-06 | Reuse existing listeners (Telemetry `:4056` node-side, ConfigServe `:4055` central-side); rejected dedicated port + `ClusterNode` column | design §3.1 |
|
||||
| 2026-08-06 | Per-cluster PSK `OT-MESH-PSK-<clusterId>` from Secrets; rejected fleet-wide shared key (blast radius); `MeshTransport:ClusterPsks` config map as rig/test override | design §3.4, plan T0.3/T0.5 |
|
||||
| 2026-08-06 | Fire-and-forget semantics preserved exactly — no retries/queues; failures become visible logs, backstops (2-min deadline, boot replay) unchanged | design §3.3 |
|
||||
| 2026-08-06 | `Dps` mode survives for single-mesh topologies; only `ClusterClient` is deleted; the three stale "Phase 6 deletes DPS" comments get corrected, not obeyed | design §3.6/§4.2 |
|
||||
| 2026-08-06 | No transport interface — third mode branch inside the two choke-point actors (ScadaBridge §7.1 economy) | design §4.3 |
|
||||
| 2026-08-06 | Coexistence granularity: central's mode governs commands, each node's mode governs its acks (per-pair flips possible — better than ScadaBridge's central-wide flag) | design §3.6 |
|
||||
|
||||
## Defect log
|
||||
|
||||
Defects found during execution (live-gate or review), with root cause + pinned test.
|
||||
|
||||
| # | Phase | Defect | Root cause | Fix / pin |
|
||||
|---|---|---|---|---|
|
||||
| — | — | *(none yet)* | | |
|
||||
|
||||
## Carry-forward register
|
||||
|
||||
Checks a gate could not drive, named + carried to the next gate (proportionate-gate rule).
|
||||
|
||||
| Item | Deferred from | Proven instead by | Closed in |
|
||||
|---|---|---|---|
|
||||
| — | — | *(none yet)* | |
|
||||
|
||||
## Issues filed by this program
|
||||
|
||||
| Issue | What | Status |
|
||||
|---|---|---|
|
||||
| *(Gitea # TBD in Phase 0, T0.6)* | `FleetStatusBroadcaster` derives "fleet" from central's own gossip ring — silently central-pair-blind since mesh Phase 6; pre-existing, transport-independent, out of this program's scope | to file |
|
||||
|
||||
## Known constraints carried from the research (do not rediscover)
|
||||
|
||||
- No Ask crosses the boundary — 4+1 one-way Tells is the whole surface (design §1).
|
||||
- `ConfigPublishCoordinator`'s unconditional `deployment-acks` DPS subscription is the
|
||||
`Dps`-mode ingress — keep it (design §4.4).
|
||||
- Interceptors: exactly one public constructor, reflection-pinned (gotcha 1).
|
||||
- Proto enum 0 = `UNSPECIFIED`; mapper goldens must cover every member incl. null variants
|
||||
(gotcha 2, plan T0.2).
|
||||
- Kestrel block untouchable — no new listeners in this program (gotcha 3).
|
||||
- `x-otopcua-cluster` is a central-side routing hint; isolation is proven by wrong-key
|
||||
rejection (gotcha 9).
|
||||
@@ -0,0 +1,480 @@
|
||||
# ClusterClient → gRPC-only cross-mesh transport — implementation plan
|
||||
|
||||
> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement
|
||||
> this plan task-by-task. Central tracking doc:
|
||||
> `docs/plans/2026-08-06-clusterclient-to-grpc-tracking.md` — **update it at every phase
|
||||
> boundary and defect.** Live gate results go in
|
||||
> `docs/plans/2026-08-06-clusterclient-to-grpc-live-gate.md`.
|
||||
|
||||
**Goal:** all central↔site mesh traffic rides gRPC with per-cluster PSK auth from
|
||||
`ZB.MOM.WW.Secrets`; `ClusterClient`/`ClusterClientReceptionist` are deleted; Akka remoting
|
||||
never crosses the mesh boundary. **Design:** `2026-08-06-clusterclient-to-grpc-design.md`
|
||||
(read it first — §4 holds the plan-vs-code corrections this plan builds on).
|
||||
|
||||
**Architecture:** one new proto (`mesh_transport.v1`) with two unary services;
|
||||
`NodeCommandService` hangs off the existing Telemetry listener on driver nodes,
|
||||
`CentralCommandService` off the existing ConfigServe listener on central. The gRPC path is a
|
||||
third `MeshTransport:Mode` branch inside the two existing choke-point actors
|
||||
(`CentralCommunicationActor` / `NodeCommunicationActor`) — no new transport interface, no new
|
||||
ports, no new DB columns.
|
||||
|
||||
**Tech Stack:** .NET 10, Grpc.Tools 2.76 build-time codegen, Akka.NET 1.5.62,
|
||||
`ZB.MOM.WW.Configuration` validators, `ZB.MOM.WW.Secrets`, xunit + Akka.TestKit.Xunit2.
|
||||
|
||||
Everything below was code-verified 2026-08-06. If a cited line has drifted, re-locate by the
|
||||
quoted identifier, never by line number.
|
||||
|
||||
---
|
||||
|
||||
## How to execute this plan (read before starting)
|
||||
|
||||
- **Phase order:** 0 → (1A ∥ 1B) → 2 → 3 → 4. Phases marked ∥ are parallel-safe **only in
|
||||
separate git worktrees** (`git worktree add ../OtOpcUa-1B feat/grpc-central-ack`) — never run
|
||||
two agents against one working tree (destructive git races are a known family incident).
|
||||
Designated merge order when tracks meet: **1A lands first, 1B rebases** (expected conflicts
|
||||
are confined to `Host/Program.cs` service-map blocks, `docker-dev/docker-compose.yml` env
|
||||
blocks, and `docs/Configuration.md` — resolve as union).
|
||||
- **Subagent model:** dispatch each task to an **Opus subagent** (`model: "opus"`). Within one
|
||||
phase/worktree, run tasks **sequentially** (they share files and git state). Across
|
||||
worktrees, run the 1A and 1B task streams **concurrently** — one Opus agent per worktree.
|
||||
Tasks whose `Parallelizable with:` names a task in the *same* worktree may be batched into a
|
||||
single agent dispatch, never into two agents on one tree. Review each task's diff before the
|
||||
next dispatch (superpowers-extended-cc:subagent-driven-development).
|
||||
- **Branches:** `feat/grpc-phase0-contract-auth`, `feat/grpc-node-command` (1A),
|
||||
`feat/grpc-central-ack` (1B), `feat/grpc-phase3-deletion`. One PR per phase to `master` with
|
||||
its DoD met; record PR number + merge SHA in the tracking doc. **Stop at each phase DoD** —
|
||||
PR + merge is user-owned unless told otherwise.
|
||||
- **Build/test:** `dotnet build ZB.MOM.WW.OtOpcUa.slnx` (0 warnings — TreatWarningsAsErrors),
|
||||
`dotnet test ZB.MOM.WW.OtOpcUa.slnx`. TestKit tests inherit the existing harnesses
|
||||
(`ControlPlaneTestHarness`, `RuntimeActorTestBase`) — never hand-roll
|
||||
`ActorSystem.Create`+join.
|
||||
- **Proto codegen is build-time** (`Grpc.Tools`, `<Protobuf>` items in
|
||||
`src/Core/ZB.MOM.WW.OtOpcUa.Commons/ZB.MOM.WW.OtOpcUa.Commons.csproj`) — no checked-in
|
||||
generated code. The docker image build is the cross-arch verification; if protoc ever
|
||||
misbehaves in the image, fall back to ScadaBridge's checked-in recipe and record it.
|
||||
- **Rig:** `docker compose -f docker-dev/docker-compose.yml up -d --build` (6 host nodes:
|
||||
central-1/-2 = `admin,driver,cluster-MAIN`; site-a-1/-2, site-b-1/-2 = driver + cluster
|
||||
role; plus sql/traefik/migrator/cluster-seed). AdminUI `http://localhost:9200` (login
|
||||
disabled), headless deploy `POST /api/deployments` with `X-Api-Key: docker-dev-deploy-key`.
|
||||
gRPC ports are container-internal: ConfigServe `:4055` (central), Telemetry `:4056` (all
|
||||
driver nodes). Mesh mode env: `MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-ClusterClient}"`
|
||||
on all six nodes.
|
||||
- **Coexistence rule:** every migrated path sits behind `MeshTransport:Mode` with
|
||||
`ClusterClient` remaining the rig default until Phase 2 and the compiled default staying
|
||||
`Dps` throughout (it is the single-mesh/in-process default, not this program's concern).
|
||||
Rollback at any point before Phase 3 = flip the mode back. DoD-test flag flips on the rig
|
||||
are **rig-only edits, reverted, never committed** until Phase 2.
|
||||
|
||||
## The two choke points (where all transport code changes)
|
||||
|
||||
- **Central→node:** `CentralCommunicationActor`
|
||||
(`src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Communication/CentralCommunicationActor.cs`) —
|
||||
`HandleMeshCommand` (`:140-168`) branches on mode; the gRPC branch fans unary `Execute`
|
||||
calls instead of `ClusterClient.SendToAll`.
|
||||
- **Node→central:** `NodeCommunicationActor`
|
||||
(`src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Communication/NodeCommunicationActor.cs`) —
|
||||
`HandleApplyAck` (`:78-92`) branches; the gRPC branch calls `ReportApply` instead of
|
||||
`ClusterClient.Send`. **Publishers and subscribers above these actors are untouched**
|
||||
(`ConfigPublishCoordinator`, `AdminOperationsActor`, `DriverHostActor`,
|
||||
`ScriptedAlarmHostActor`, the EventStream leg, the coordinator `Forward`).
|
||||
|
||||
---
|
||||
|
||||
## Phase 0 — Contract + auth + config groundwork (standalone; no behavior change)
|
||||
|
||||
Branch `feat/grpc-phase0-contract-auth`. Everything here is additive and dark.
|
||||
|
||||
### Task 0.1: `mesh_transport.proto` + contract lock
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~20 min
|
||||
**Parallelizable with:** Task 0.3 (same worktree — batch to one agent, do not split)
|
||||
|
||||
**Files:**
|
||||
- Create: `src/Core/ZB.MOM.WW.OtOpcUa.Commons/Protos/mesh_transport.proto`
|
||||
- Create: `src/Core/ZB.MOM.WW.OtOpcUa.Commons/Protos/MeshTransportProtoContract.cs`
|
||||
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Commons/ZB.MOM.WW.OtOpcUa.Commons.csproj` (third `<Protobuf Include="Protos\mesh_transport.proto" GrpcServices="Both"/>`)
|
||||
- Test: `tests/Core/ZB.MOM.WW.OtOpcUa.Commons.Tests/MeshTransportProtoContractTests.cs`
|
||||
|
||||
Package `mesh_transport.v1`, `csharp_namespace "ZB.MOM.WW.OtOpcUa.Commons.Protos.MeshTransport.V1"`
|
||||
(NOT `MeshCommand` — collides with `Commons.Messages.Mesh.MeshCommand`). Services + envelope per
|
||||
design §3.2: `NodeCommandService.Execute(CommandEnvelope) returns (ExecuteAck)`;
|
||||
`CentralCommandService.ReportApply(ApplyAckMessage) returns (ReportAck)`. `CommandEnvelope` has
|
||||
`correlation_id` + `oneof command` of 4 messages mirroring `DispatchDeployment`
|
||||
(`Commons/Messages/Deploy/DispatchDeployment.cs`), `RestartDriver`, `ReconnectDriver`
|
||||
(`Commons/Messages/Admin/`), `AlarmCommand` (`Commons/OpcUa/AlarmCommand.cs`).
|
||||
`ApplyAckMessage` mirrors `ApplyAck` (`DeploymentId, NodeId, Outcome, FailureReason,
|
||||
CorrelationId`). **Every enum reserves 0 as `*_UNSPECIFIED`** (`AlarmCommand` verb,
|
||||
`ApplyAck.Outcome`) — proto3 zero must never decode to a meaningful value.
|
||||
`MeshTransportProtoContract.HandledCases` mirrors `TelemetryProtoContract.cs:18-24`; the test
|
||||
mirrors `TelemetryProtoContractTests` (fails when a oneof case is added but unhandled). TDD:
|
||||
write the contract test red (proto absent), add proto + helper, green, commit
|
||||
`feat(mesh): mesh_transport.v1 proto contract`.
|
||||
|
||||
### Task 0.2: DTO mapper + round-trip goldens
|
||||
|
||||
**Classification:** high-risk (data contract — this is where ScadaBridge found five silent-drop bugs)
|
||||
**Estimated implement time:** ~30 min
|
||||
**Parallelizable with:** none (depends on 0.1)
|
||||
|
||||
**Files:**
|
||||
- Create: `src/Core/ZB.MOM.WW.OtOpcUa.Commons/Messages/Mesh/MeshTransportDtoMapper.cs`
|
||||
- Test: `tests/Core/ZB.MOM.WW.OtOpcUa.Commons.Tests/MeshTransportDtoMapperTests.cs`
|
||||
|
||||
Static mapper: `ToProto`/`FromProto` for all five messages + the envelope. Before writing it,
|
||||
**enumerate every member of the five records in the test first** (golden round-trips: construct
|
||||
each record with every property set to a non-default value, map to proto and back, assert
|
||||
record equality; add a null-variant golden for every nullable member — `FailureReason` at
|
||||
minimum). Guard against the known trap classes: init-only props (use object initializers in
|
||||
`FromProto`), nullable strings where `""` is not null (use proto3 `optional`), enum zero
|
||||
(decode unknown → a `Unknown` CLR value or throw, never a meaningful default). Unknown oneof
|
||||
case in `FromProto` ⇒ throw `InvalidOperationException` naming the case — the contract-lock
|
||||
test ensures it's unreachable. Commit `feat(mesh): DTO mapper with round-trip goldens`.
|
||||
|
||||
### Task 0.3: `MeshTransportOptions` + validators + SplitTopology relaxation
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~25 min
|
||||
**Parallelizable with:** Task 0.1 (same worktree — batch)
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/MeshTransportOptions.cs`
|
||||
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/MeshTransportOptionsValidator.cs`
|
||||
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/SplitTopologyTransportValidator.cs`
|
||||
- Test: `tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/MeshTransportOptionsValidatorTests.cs`, `.../SplitTopologyTransportValidatorTests.cs`
|
||||
|
||||
Add `ModeGrpc = "Grpc"`; new members `string[] CentralGrpcEndpoints`, `string? GrpcPsk`,
|
||||
`int DispatchTimeoutSeconds = 10`, `int AckTimeoutSeconds = 10`, and a config-map override
|
||||
`Dictionary<string,string> ClusterPsks` (central-side rig/test override; production leaves it
|
||||
empty and the runtime provider falls back to Secrets — mirrors ScadaBridge's
|
||||
`SitePsks__site-*` rig pattern). Validator, following the house rules (roles via
|
||||
`RoleParser.ResolveEffectiveRoles` from `IConfiguration`, keys required only where
|
||||
load-bearing, messages name the exact key + env form `MeshTransport__GrpcPsk`, never echo
|
||||
values): under `Grpc` + driver role ⇒ `GrpcPsk` non-empty, `CentralGrpcEndpoints` non-empty
|
||||
and each `http(s)://`-prefixed, both timeouts > 0. `SplitTopologyTransportValidator:101-109`:
|
||||
accept `ClusterClient` **or** `Grpc` (tighten to `Grpc`-only in Phase 3 — without this
|
||||
relaxation a rig node flipped to `Grpc` refuses to boot). Also correct the three stale
|
||||
"Phase 6 deletes the DPS branch" comments (`MeshTransportOptions.cs:11-13`,
|
||||
`ConfigPublishCoordinator.cs:70-73`, `AdminOperationsActor.cs:90-92`) to match
|
||||
`docs/Configuration.md:150-156`. TDD red-first on both validators. Commit
|
||||
`feat(mesh): Grpc mode value + validation`.
|
||||
|
||||
### Task 0.4: node-side `MeshCommandAuthInterceptor`
|
||||
|
||||
**Classification:** high-risk (security)
|
||||
**Estimated implement time:** ~20 min
|
||||
**Parallelizable with:** Task 0.5 (different files, same worktree — batch)
|
||||
|
||||
**Files:**
|
||||
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/MeshCommandAuthInterceptor.cs`
|
||||
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs:550-562` (add to the `hasDriver` interceptor block)
|
||||
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/Configuration/MeshCommandAuthInterceptorTests.cs`
|
||||
|
||||
Clone `TelemetryStreamAuthInterceptor.cs` byte-for-byte in structure: ServicePrefix
|
||||
`/mesh_transport.v1.NodeCommandService/`, all four `*ServerHandler` overrides → one
|
||||
`Authorize`, case-insensitive `authorization` metadata walk, `Bearer ` prefix,
|
||||
`CryptographicOperations.FixedTimeEquals` over UTF-8, expected token =
|
||||
`IOptions<MeshTransportOptions>.GrpcPsk`, **fail-closed when unset** (reject all with
|
||||
actionable Warning), deny `PermissionDenied`, non-matching paths pass through. **Exactly ONE
|
||||
public constructor** + a `HasExactlyOnePublicConstructor` reflection test (the
|
||||
`Grpc.AspNetCore` multi-ctor silent-skip trap, `TelemetryStreamAuthInterceptor.cs:38-42`; cost
|
||||
ScadaBridge a live-gate day). No `x-otopcua-cluster` requirement here — header-on-node-side is
|
||||
theatre (design §3.4). Tests: no-token / wrong-token / correct-token / unset-key /
|
||||
non-matching-path, mirroring `TelemetryStreamAuthInterceptorTests`. Commit
|
||||
`feat(mesh): node-side PSK interceptor`.
|
||||
|
||||
### Task 0.5: `IMeshPskProvider` + central-side `CentralCommandAuthInterceptor`
|
||||
|
||||
**Classification:** high-risk (security + the one new stateful component)
|
||||
**Estimated implement time:** ~35 min
|
||||
**Parallelizable with:** Task 0.4 (batch)
|
||||
|
||||
**Files:**
|
||||
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Communication/IMeshPskProvider.cs` (+ `MeshPskProvider`)
|
||||
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/CentralCommandAuthInterceptor.cs`
|
||||
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs` (DI registration; add interceptor to the `hasAdmin` block)
|
||||
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/Communication/MeshPskProviderTests.cs`, `tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/Configuration/CentralCommandAuthInterceptorTests.cs`
|
||||
|
||||
`IMeshPskProvider.TryGetPsk(string clusterId, out string psk)`: check
|
||||
`MeshTransportOptions.ClusterPsks` config map first, else resolve secret
|
||||
`OT-MESH-PSK-{clusterId}` via `ISecretResolver` (the fail-closed lazy pattern the Galaxy
|
||||
driver uses for `secret:` refs), **cache per cluster, `Invalidate(clusterId)` /
|
||||
`InvalidateAll()` hooks** (wired to contact refresh in 1A). Fail-closed: unresolvable ⇒
|
||||
`false`, caller denies/skips — never proceeds unauthenticated. Adversarial tests, not just
|
||||
happy-path: unknown cluster, resolver throw, cache serves stale until invalidated, empty
|
||||
secret value = failure. `CentralCommandAuthInterceptor`: ServicePrefix
|
||||
`/mesh_transport.v1.CentralCommandService/`; reads `x-otopcua-cluster`, missing/unknown ⇒
|
||||
`PermissionDenied`; verifies bearer against that cluster's PSK via the provider,
|
||||
`FixedTimeEquals`; one public ctor + reflection test. Commit
|
||||
`feat(mesh): per-cluster PSK provider + central interceptor`.
|
||||
|
||||
### Task 0.6: rig config, issue filing, docs stub
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** ~15 min
|
||||
**Parallelizable with:** none (last in phase; touches compose + docs)
|
||||
|
||||
**Files:**
|
||||
- Modify: `docker-dev/docker-compose.yml` (all six host nodes: `MeshTransport__GrpcPsk` dev literals `dev-mesh-psk-MAIN` / `-SITE-A` / `-SITE-B`; `MeshTransport__CentralGrpcEndpoints__0/1` = `http://central-1:4055`, `http://central-2:4055`; central pair additionally `MeshTransport__ClusterPsks__MAIN` / `SITE-A` / `SITE-B` config-map literals. **Mode stays `${OTOPCUA_MESH_MODE:-ClusterClient}`.**)
|
||||
- Modify: `docs/Configuration.md` (MeshTransport section: document `Grpc` mode + new keys as "shipped dark, cutover pending")
|
||||
- Modify: `docs/plans/2026-08-06-clusterclient-to-grpc-tracking.md` (Phase 0 row)
|
||||
|
||||
Also: file the Gitea issue for `FleetStatusBroadcaster`'s central-pair-blind fleet view
|
||||
(design §3.8 — pre-existing, transport-independent, NOT fixed here; record the number in the
|
||||
tracking doc). Full-suite run; rig rebuild; confirm all six nodes boot with the new env
|
||||
present and mode unchanged (validators must not fire — keys are only required under `Grpc`).
|
||||
Commit `chore(mesh): rig PSK config + docs stub`.
|
||||
|
||||
**Phase 0 DoD:** suite green, 0 warnings; rig boots all six nodes on `ClusterClient` with the
|
||||
dark config present; both interceptors + provider + mapper fully unit-tested; FleetStatus
|
||||
issue filed; tracking doc updated. Live-gate doc Phase 0 section recorded (small gate: boot +
|
||||
validator-firing checks, including one deliberate misconfig — `Mode=Grpc` with no `GrpcPsk` on
|
||||
a scratch container refuses boot with the exact-key message). **STOP — PR + merge are
|
||||
user-owned.**
|
||||
|
||||
---
|
||||
|
||||
## Phase 1A — Central→node command plane (worktree A, branch `feat/grpc-node-command`)
|
||||
|
||||
### Task 1A.1: `NodeCommandGrpcService`
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~25 min
|
||||
**Parallelizable with:** Task 1B.1 (OTHER worktree only)
|
||||
|
||||
**Files:**
|
||||
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Grpc/NodeCommandGrpcService.cs`
|
||||
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs:707-725` (map under `hasDriver && telemetryListenPort > 0`, beside `TelemetryStreamGrpcService`)
|
||||
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/Grpc/NodeCommandGrpcServiceTests.cs`
|
||||
|
||||
Follow the alias convention (`using GeneratedServiceBase = …NodeCommandService.NodeCommandServiceBase;`,
|
||||
class suffixed `GrpcService`). `Execute`: decode via `MeshTransportDtoMapper.FromProto`, Tell
|
||||
the registered `NodeCommunicationActor` (resolve via `NodeCommunicationActorKey` registry,
|
||||
`Runtime/ServiceCollectionExtensions.cs:583`), return `ExecuteAck`. Actor not yet registered
|
||||
(boot window) ⇒ `RpcException(Unavailable)` — the readiness convention. Decode failure ⇒
|
||||
`InvalidArgument`. Node side re-emits on EventStream exactly as under ClusterClient — the
|
||||
existing `NodeCommunicationActor` receive path handles the four types already; verify
|
||||
`EventStreamCommandDeliveryTests` pass **unmodified** (if they need edits, the seam is wrong).
|
||||
TDD with TestKit probe as the registered actor. Commit `feat(mesh): NodeCommandService on the
|
||||
telemetry listener`.
|
||||
|
||||
### Task 1A.2: `MeshGrpcCommandDialer` + the third branch in `CentralCommunicationActor`
|
||||
|
||||
**Classification:** high-risk (actor + concurrency)
|
||||
**Estimated implement time:** ~40 min
|
||||
**Parallelizable with:** Task 1B.2 (OTHER worktree only)
|
||||
|
||||
**Files:**
|
||||
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Communication/MeshGrpcCommandDialer.cs`
|
||||
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Communication/CentralCommunicationActor.cs`
|
||||
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/Communication/CentralCommunicationActorTests.cs` (new facts), `.../MeshGrpcCommandDialerTests.cs`
|
||||
|
||||
Dialer (plain class, injectable fake): `DispatchAsync(nodeEndpoint, CommandEnvelope, ct)` —
|
||||
`GrpcChannel` cached per endpoint (`ConcurrentDictionary`, `IDisposable`, the
|
||||
`GrpcDeploymentArtifactFetcher.cs:138-152` shape), bearer from `IMeshPskProvider` for the
|
||||
target node's cluster + `x-otopcua-cluster` header, deadline via linked CTS
|
||||
(`DispatchTimeoutSeconds`). Actor changes: extend the contact-load query to also select
|
||||
`GrpcPort` (rows already loaded — `LoadContactsFromDb`, `CentralCommunicationActor.cs:207-245`);
|
||||
under `Mode=Grpc`, `HandleMeshCommand` maps the message via `ToProto` and fans
|
||||
`DispatchAsync` to every enabled non-maintenance node across all clusters (the `SendToAll`
|
||||
recipient set), `PipeTo`-ing per-node results back for logging only — **a failure is logged
|
||||
with node id + status and dropped; no retry** (design §3.3). Skip-with-warn a node whose
|
||||
`GrpcPort` is NULL (mirror `TelemetryNodeSource:49-60`). Wire `IMeshPskProvider.Invalidate`
|
||||
to the contact-refresh tick. New facts mirror the existing `SendToAll` facts
|
||||
(`CentralCommunicationActorTests:142,191`) with a fake dialer: fan set, per-node failure
|
||||
isolation (one dead node doesn't stop the fan), NULL-port skip, PSK-miss skip (fail-closed).
|
||||
Commit `feat(mesh): gRPC command fan-out behind Mode=Grpc`.
|
||||
|
||||
### Task 1A.3: 1A live check (rig-only, nothing committed but the gate record)
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** ~20 min (rig wall-time)
|
||||
**Parallelizable with:** none
|
||||
|
||||
Rebuild the rig from the branch. Flip **central-1/-2 only** to `MeshTransport__Mode: "Grpc"`
|
||||
(rig-only edit, reverted after). Sites stay `ClusterClient`. Run live-gate Phase 1A checks
|
||||
(deploy dispatch reaches all four site nodes over gRPC while acks return over ClusterClient —
|
||||
coexistence proven; RestartDriver/AlarmCommand from AdminUI; unauthenticated + wrong-key
|
||||
`grpcurl` probes against a site node's `:4056` rejected). Record in the live-gate doc; revert
|
||||
the flip. **Phase 1A DoD:** suite green; gate section PASS; tracking updated. **STOP.**
|
||||
|
||||
---
|
||||
|
||||
## Phase 1B — Node→central ack plane (worktree B, branch `feat/grpc-central-ack`)
|
||||
|
||||
### Task 1B.1: `CentralCommandGrpcService` + admin validator rule
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~25 min
|
||||
**Parallelizable with:** Task 1A.1 (OTHER worktree only)
|
||||
|
||||
**Files:**
|
||||
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Grpc/CentralCommandGrpcService.cs` (beside `DeploymentArtifactService.cs`)
|
||||
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs:707-725` (map under `hasAdmin && configServeGrpcPort > 0`)
|
||||
- Modify: `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/MeshTransportOptionsValidator.cs` (admin role + `Mode=Grpc` ⇒ `ConfigServe:GrpcListenPort > 0`, cross-read from `IConfiguration` — the SplitTopology pattern)
|
||||
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.AdminUI.Tests/Grpc/CentralCommandGrpcServiceTests.cs`, validator tests
|
||||
|
||||
`ReportApply`: decode `ApplyAckMessage` → Tell `CentralCommunicationActor` (resolve via
|
||||
`CentralCommunicationActorKey` registry, `ControlPlane/ServiceCollectionExtensions.cs:246`) →
|
||||
its existing `HandleApplyAck` `Forward`s to the coordinator proxy untouched. Not-ready ⇒
|
||||
`Unavailable`; decode failure ⇒ `InvalidArgument`. Commit `feat(mesh): CentralCommandService
|
||||
on the ConfigServe listener`.
|
||||
|
||||
### Task 1B.2: `GrpcApplyAckSender` + node-side wiring
|
||||
|
||||
**Classification:** high-risk (actor wiring; the `Send`-not-`SendToAll` single-count semantic)
|
||||
**Estimated implement time:** ~35 min
|
||||
**Parallelizable with:** Task 1A.2 (OTHER worktree only)
|
||||
|
||||
**Files:**
|
||||
- Create: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Communication/GrpcApplyAckSender.cs`
|
||||
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Communication/NodeCommunicationActor.cs`
|
||||
- Modify: `src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ServiceCollectionExtensions.cs:389-416,536`
|
||||
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Communication/NodeCommunicationActorTests.cs` (new facts), `.../GrpcApplyAckSenderTests.cs`
|
||||
|
||||
Sender: try `CentralGrpcEndpoints` **in order, stop at first success** (one ack per node —
|
||||
the `Send` semantic; the endpoint loop + per-endpoint channel cache of
|
||||
`GrpcDeploymentArtifactFetcher:67-116`), bearer = own `GrpcPsk` + `x-otopcua-cluster` = own
|
||||
clusterId (from `RoleParser` cluster role), deadline `AckTimeoutSeconds`; all endpoints fail ⇒
|
||||
log Warning + drop (today's no-client semantic, now visible). `NodeCommunicationActor` gains
|
||||
an **optional** `ackSender` ctor param (null under `Dps`/`ClusterClient` — existing tests pass
|
||||
with ZERO edits; if they need edits, the seam is wrong); under `Mode=Grpc` the registration
|
||||
(`ServiceCollectionExtensions.cs:389-416`) creates no ClusterClient and passes the sender;
|
||||
`_ackRouter` wiring (`:536`) is unchanged (still the actor). New facts: gRPC branch sends
|
||||
exactly once on first-endpoint success, fails over to second, drops after both. Commit
|
||||
`feat(mesh): gRPC ApplyAck path behind Mode=Grpc`.
|
||||
|
||||
### Task 1B.3: 1B live check (rig-only)
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** ~20 min (rig wall-time)
|
||||
**Parallelizable with:** none
|
||||
|
||||
Rebuild from branch (after rebasing on merged 1A if it landed — expected union conflicts:
|
||||
`Program.cs` map block, compose, `Configuration.md`). Flip **site-a-1/-2 only** to `Grpc`.
|
||||
Deploy e2e: site-a acks arrive over gRPC (`PersistNodeAck` rows land, deployment seals),
|
||||
site-b still acks over ClusterClient — coexistence proven. Kill central-1 mid-deploy: site-a
|
||||
ack fails over to central-2's endpoint. Record; revert. **Phase 1B DoD:** suite green; gate
|
||||
PASS; tracking updated. **STOP.**
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 — Full-rig cutover + soak (sequential, after 1A+1B merged)
|
||||
|
||||
No feature branch; one commit flipping the rig default:
|
||||
`docker-dev/docker-compose.yml` `MeshTransport__Mode: "${OTOPCUA_MESH_MODE:-Grpc}"`.
|
||||
|
||||
**Before flipping, run the BASELINE on the pre-change build** (live-gate doc has the exact
|
||||
commands): reproduce the two gaps this program closes — (a) a site container holds a live Akka
|
||||
association to central's `:4053` (the unauthenticated cross-WAN surface), (b) stop the site-b
|
||||
pair, dispatch a deployment, observe the silent drop (no per-node error at dispatch; only the
|
||||
2-min deadline marks it). Reproduced, not assumed.
|
||||
|
||||
Then: full rebuild all-Grpc; run the Phase 2 check table (deploy e2e all six nodes;
|
||||
Restart/Reconnect/AlarmCommand matrix; kill-central-1 ack failover; kill-site-node
|
||||
mid-dispatch = logged per-node failure + deadline backstop, no retry storm; PSK negatives
|
||||
against both services on running containers; **zero ClusterClient log lines under `Grpc`**;
|
||||
soak ≥ 2 h with 0 `PermissionDenied` noise — PSK drift detector). **Phase 2 DoD:** every
|
||||
check PASS in the live-gate doc; rig default committed; tracking updated. **STOP.**
|
||||
|
||||
---
|
||||
|
||||
## Phase 3 — Deletion + tightening + docs (branch `feat/grpc-phase3-deletion`)
|
||||
|
||||
### Task 3.1: delete ClusterClient (code + HOCON + config)
|
||||
|
||||
**Classification:** high-risk (deletion ordering)
|
||||
**Estimated implement time:** ~40 min
|
||||
**Parallelizable with:** none
|
||||
|
||||
**Files:** `Runtime/ServiceCollectionExtensions.cs`, `ControlPlane/ServiceCollectionExtensions.cs`,
|
||||
`ControlPlane/Communication/{CentralCommunicationActor.cs,IMeshClusterClientFactory.cs}`,
|
||||
`Runtime/Communication/NodeCommunicationActor.cs`, `Cluster/Resources/akka.conf`,
|
||||
`Cluster/MeshTransportOptions.cs` + validator, `docker-dev/docker-compose.yml`, affected tests.
|
||||
|
||||
In order: (1) delete `IMeshClusterClientFactory`/`DefaultMeshClusterClientFactory` + central
|
||||
per-cluster client cache/`RebuildClient` (**keep `LoadContactsFromDb` — it feeds the gRPC
|
||||
dialer**); (2) delete node client creation + `centralClient` param; (3) delete both
|
||||
receptionist registrations; (4) `akka.conf`: remove `ClusterClientReceptionistExtensionProvider`
|
||||
from `akka.extensions` + the whole `akka.cluster.client` block (**keep DPS extension +
|
||||
`maximum-frame-size` canary — pair-local channels live**); (5) delete
|
||||
`CentralContactPoints` + validator rules + all six compose entries; (6) delete the
|
||||
`ClusterClient` mode value; `SplitTopologyTransportValidator` requires `Grpc`; (7) retire/
|
||||
replace tests: `MeshClusterClientBoundaryTests` → `MeshGrpcBoundaryTests` (two clusters, gRPC
|
||||
between them, **assert no cross-mesh Akka association forms** — the program's point, kept as a
|
||||
test), `MeshClusterClientFactoryTests` deleted, `MeshTransportHoconTests` drops the
|
||||
client-settings asserts (keeps frame-size + DPS pins), `CentralCommunicationActorTests` /
|
||||
`NodeCommunicationActorTests` / `MeshRouterDispatchTests` reshaped to Dps+Grpc. Grep-gates:
|
||||
`rg -i "clusterclient|receptionist" src tests docker-dev docs` → only historical
|
||||
plans/CHANGELOG; `rg "CentralContactPoints"` → empty. Commit sequence per step.
|
||||
|
||||
### Task 3.2: docs sweep
|
||||
|
||||
**Classification:** small
|
||||
**Estimated implement time:** ~20 min
|
||||
**Parallelizable with:** none
|
||||
|
||||
`docs/Configuration.md` (MeshTransport = Dps|Grpc, keys, PSK seeding), `docs/Redundancy.md`
|
||||
transport table, `docs/Telemetry.md` cross-refs, `OtOpcUa/CLAUDE.md` transport decisions,
|
||||
`deferment.md` if applicable — and the **scadaproj index entry** (`~/Desktop/scadaproj/CLAUDE.md`,
|
||||
committed on scadaproj `main` and pushed, per its index-edit rule). Update tracking doc.
|
||||
**Phase 3 DoD:** suite green; grep-gates clean; docs merged. **STOP.**
|
||||
|
||||
---
|
||||
|
||||
## Phase 4 — Terminal live gate (rig; sequential; every check PASS required)
|
||||
|
||||
Run on `master` AFTER Phase 3 merges, on the **post-deletion image** (verify image build
|
||||
timestamp > merge time — the ScadaBridge provenance guard). Eight checks, recorded in the
|
||||
live-gate doc: (1) PSK negatives both services incl. unset-key fail-closed; (2) command
|
||||
matrix all six nodes; (3) ack matrix + seal; (4) central-kill ack failover + site-node-kill
|
||||
dispatch honesty; (5) mid-deploy central kill → deadline/replay backstops hold, no dupes;
|
||||
(6) **boundary check: no Akka association between any site container and central** (was the
|
||||
baseline gap); (7) restart discipline: full-rig restart clean, **zero
|
||||
receptionist/ClusterClient log lines anywhere**; (8) `MeshGrpcBoundaryTests` + full suite
|
||||
green on the merge SHA. Close any carry-forwards named by earlier gates. Tracking doc final
|
||||
row; program CLOSED.
|
||||
|
||||
---
|
||||
|
||||
## Gotchas for the executor (will bite; read twice)
|
||||
|
||||
1. **One public constructor per interceptor** — `Grpc.AspNetCore` silently skips multi-ctor
|
||||
interceptors; correct/wrong/no key then fail identically (`Unknown`), which is
|
||||
indistinguishable from "working" in unit tests that register via DI. Reflection-pin it.
|
||||
2. **Enum zero is `UNSPECIFIED`, never a meaningful value** — proto3 decodes absent as 0.
|
||||
3. **Do NOT touch the Kestrel block** (`Program.cs:564-671`) — no new listeners exist in this
|
||||
plan; any deviation that adds one re-imports the explicit-endpoints-suppresses-URLS trap.
|
||||
4. **Fire-and-forget stays fire-and-forget** — no retry on `DeadlineExceeded` or anything
|
||||
else; the deploy deadline + boot replay are the only recovery. Adding a retry turns one
|
||||
`RestartDriver` into two.
|
||||
5. **`ConfigPublishCoordinator`'s `deployment-acks` DPS subscription stays** — it is the
|
||||
`Dps`-mode ingress, not dead code.
|
||||
6. **`SendToAll`/`Send` semantics:** fan = every enabled non-maintenance `ClusterNode`; ack =
|
||||
exactly once, first healthy endpoint. The coordinator counts acks — a double-send
|
||||
double-counts.
|
||||
7. **Existing tests are the seam oracle:** `EventStreamCommandDeliveryTests` and the
|
||||
pre-existing `NodeCommunicationActorTests`/`CentralCommunicationActorTests` construction
|
||||
paths must pass **unmodified** through 1A/1B (nullable ctor params, never signature
|
||||
breaks). If they need edits before Phase 3, the seam is wrong.
|
||||
8. **Rig flips before Phase 2 are never committed**; a redeploy from `master` must reset to
|
||||
all-ClusterClient until the Phase 2 commit.
|
||||
9. **`x-otopcua-cluster` is a central-side routing hint** — do not add it to the node-side
|
||||
interceptor's requirements (theatre), and do not prove isolation with it (wrong-key
|
||||
rejection is the proof).
|
||||
10. **Secrets on driver-only nodes work** (node-local SQLite + env KEK) — do not add a
|
||||
ConfigDb dependency anywhere on the node path; that constraint is the whole point of
|
||||
mesh Phases 3/4.
|
||||
|
||||
## Effort & parallelization
|
||||
|
||||
| Track | Est. | Parallel with |
|
||||
|---|---|---|
|
||||
| Phase 0 | 2–3 d | — (single worktree; tasks 0.1+0.3 and 0.4+0.5 batchable) |
|
||||
| 1A | 3–4 d | 1B (separate worktrees; 1A merges first, 1B rebases) |
|
||||
| 1B | 2–3 d | 1A |
|
||||
| 2 | 1–2 d | — |
|
||||
| 3 | 1–2 d | — |
|
||||
| 4 | 1 d | — |
|
||||
|
||||
Critical path ≈ 0 → 1A → 2 → 3 → 4: **~1.5–2.5 weeks.**
|
||||
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"planPath": "docs/plans/2026-08-06-clusterclient-to-grpc.md",
|
||||
"executionModel": "opus-subagents; sequential within a worktree, 1A and 1B concurrent in separate worktrees (git worktree add ../OtOpcUa-1B feat/grpc-central-ack); 1A merges first, 1B rebases; stop at each phase DoD (PR/merge user-owned)",
|
||||
"worktrees": {},
|
||||
"tasks": [
|
||||
{"id": 0, "subject": "Task 0.1: mesh_transport.proto + contract lock", "phase": "0", "branch": "feat/grpc-phase0-contract-auth", "classification": "standard", "parallelizableWith": ["0.3 (batch, same agent)"], "status": "pending"},
|
||||
{"id": 1, "subject": "Task 0.2: DTO mapper + round-trip goldens", "phase": "0", "branch": "feat/grpc-phase0-contract-auth", "classification": "high-risk", "parallelizableWith": [], "status": "pending", "blockedBy": [0]},
|
||||
{"id": 2, "subject": "Task 0.3: MeshTransportOptions + validators + SplitTopology relaxation", "phase": "0", "branch": "feat/grpc-phase0-contract-auth", "classification": "standard", "parallelizableWith": ["0.1 (batch, same agent)"], "status": "pending"},
|
||||
{"id": 3, "subject": "Task 0.4: node-side MeshCommandAuthInterceptor", "phase": "0", "branch": "feat/grpc-phase0-contract-auth", "classification": "high-risk", "parallelizableWith": ["0.5 (batch, same agent)"], "status": "pending", "blockedBy": [2]},
|
||||
{"id": 4, "subject": "Task 0.5: IMeshPskProvider + CentralCommandAuthInterceptor", "phase": "0", "branch": "feat/grpc-phase0-contract-auth", "classification": "high-risk", "parallelizableWith": ["0.4 (batch, same agent)"], "status": "pending", "blockedBy": [2]},
|
||||
{"id": 5, "subject": "Task 0.6: rig config + FleetStatus issue + docs stub + Phase 0 gate", "phase": "0", "branch": "feat/grpc-phase0-contract-auth", "classification": "small", "parallelizableWith": [], "status": "pending", "blockedBy": [0, 1, 2, 3, 4]},
|
||||
{"id": 6, "subject": "Task 1A.1: NodeCommandGrpcService on the telemetry listener", "phase": "1A", "branch": "feat/grpc-node-command", "classification": "standard", "parallelizableWith": ["1B.1 (other worktree)"], "status": "pending", "blockedBy": [5]},
|
||||
{"id": 7, "subject": "Task 1A.2: MeshGrpcCommandDialer + third branch in CentralCommunicationActor", "phase": "1A", "branch": "feat/grpc-node-command", "classification": "high-risk", "parallelizableWith": ["1B.2 (other worktree)"], "status": "pending", "blockedBy": [6]},
|
||||
{"id": 8, "subject": "Task 1A.3: 1A live gate (central pair flipped, rig-only, reverted)", "phase": "1A", "branch": "feat/grpc-node-command", "classification": "small", "parallelizableWith": [], "status": "pending", "blockedBy": [7]},
|
||||
{"id": 9, "subject": "Task 1B.1: CentralCommandGrpcService + admin validator rule", "phase": "1B", "branch": "feat/grpc-central-ack", "classification": "standard", "parallelizableWith": ["1A.1 (other worktree)"], "status": "pending", "blockedBy": [5]},
|
||||
{"id": 10, "subject": "Task 1B.2: GrpcApplyAckSender + node-side wiring", "phase": "1B", "branch": "feat/grpc-central-ack", "classification": "high-risk", "parallelizableWith": ["1A.2 (other worktree)"], "status": "pending", "blockedBy": [9]},
|
||||
{"id": 11, "subject": "Task 1B.3: 1B live gate (site-a flipped, rig-only, reverted; rebase on 1A first)", "phase": "1B", "branch": "feat/grpc-central-ack", "classification": "small", "parallelizableWith": [], "status": "pending", "blockedBy": [8, 10]},
|
||||
{"id": 12, "subject": "Phase 2: baseline reproduction + full-rig cutover + soak + kill drills", "phase": "2", "branch": "(rig commit)", "classification": "high-risk", "parallelizableWith": [], "status": "pending", "blockedBy": [8, 11]},
|
||||
{"id": 13, "subject": "Task 3.1: delete ClusterClient (code + HOCON + config + tests, ordered)", "phase": "3", "branch": "feat/grpc-phase3-deletion", "classification": "high-risk", "parallelizableWith": [], "status": "pending", "blockedBy": [12]},
|
||||
{"id": 14, "subject": "Task 3.2: docs sweep (Configuration/Redundancy/Telemetry/CLAUDE.md + scadaproj index)", "phase": "3", "branch": "feat/grpc-phase3-deletion", "classification": "small", "parallelizableWith": [], "status": "pending", "blockedBy": [13]},
|
||||
{"id": 15, "subject": "Phase 4: terminal live gate on post-deletion image (8 checks, provenance-guarded)", "phase": "4", "branch": "(master)", "classification": "high-risk", "parallelizableWith": [], "status": "pending", "blockedBy": [14]}
|
||||
],
|
||||
"lastUpdated": "2026-08-06"
|
||||
}
|
||||
Reference in New Issue
Block a user