dohertj2/lmxopcua
1
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases Wiki Activity

merge: ClusterClient→gRPC Phase 1A — central→node command plane over gRPC (NodeCommandService on the telemetry listener, MeshGrpcCommandDialer, third mode branch in CentralCommunicationActor; security-register items 1-2 closed, gate follows the endpoint; live gate PASS 8/8 on re-run after the starter-order boot defect was found live and fixed; #544/#545/#546 filed)

Browse Source
This commit is contained in:
Joseph Doherty
2026-08-06 08:41:30 -04:00
parent 3f96115cab 252618182f
commit b360239e5e
19 changed files with 3543 additions and 92 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/plans/2026-08-06-clusterclient-to-grpc-live-gate.md
+504 -15
View File
@@ -154,22 +154,498 @@ Rig left running on `ClusterClient` (its normal state) after the gate.
---
## Phase 1A — central→node command plane — **PENDING**
## Phase 1A — central→node command plane — **PASS (on the re-run at `e54883a8`)**
Branch `feat/grpc-node-command` @ `________`. Rig-only flip: central-1/-2 →
`MeshTransport__Mode=Grpc`; **site nodes remain `ClusterClient`** (coexistence is the point).
Flip reverted after the gate, never committed.
Two runs. **Run 1 @ `fbf7b708` FAILED** at the central boot boundary (defect 4) and could not
drive checks 1/2/7 or the central halves of 3/4; its checks 5 + 6 and the node-side halves of
3 + 4 stand as recorded and were not repeated. **Run 2 @ `e54883a8`** — after the boot fix
(`c0475f2d`) and its review-hardening follow-up (`e54883a8`) — drove every carried check. The
result table below is the **combined, current** verdict; run 1's failure record is preserved
underneath it because the defect it caught is the most valuable thing this gate produced.
| # | Check | Result |
|---|---|---|
| 1 | Deploy dispatch from AdminUI reaches **all six** driver nodes over gRPC (per-node `Execute` log lines on every node; zero `SendToAll` lines on central) | PENDING |
| 2 | Acks still return over ClusterClient (site mode unchanged) and the deployment **seals** — mixed-transport deploy proven end to end | PENDING |
| 3 | `RestartDriver` + `ReconnectDriver` from AdminUI land on the target site node (EventStream delivery observed via driver restart in logs) | PENDING |
| 4 | `AlarmCommand` (ack a scripted alarm from AdminUI) lands and the condition acknowledges | PENDING |
| 5 | Unauthenticated `grpcurl` to a site node `:4056` `mesh_transport.v1.NodeCommandService/Execute` ⇒ `PermissionDenied`; correct dev PSK ⇒ accepted | PENDING |
| 6 | **Wrong-key isolation:** site-b's PSK presented to site-a ⇒ `PermissionDenied` (per-cluster scoping is real, not decorative) | PENDING |
| 7 | A node with NULL `GrpcPort` is skipped with the warn-once, fan continues to the others | PENDING |
| 8 | Telemetry + config-serve + LocalDb sync unaffected: 0 auth failures on their interceptors across the run | PENDING |
| # | Check | Verdict | Run |
|---|---|---|---|
| — | **Boot:** all six nodes healthy INCLUDING the central pair, on the branch image | **PASS** — 6/6 `running restarts=0`, 0 `Hosting failed to start`, `/health/ready` 200 ×6, MAIN mesh `memberCount=2 unreachableCount=0`. **But the `Resolve` Error line the fix logs DID fire — see finding 5** | 2 |
| 1 | Deploy dispatch reaches **all six** driver nodes over gRPC; zero `SendToAll`/ClusterClient on central | **PASS** — 6/6 nodes accepted one `DispatchDeployment` under a single shared `correlationId=cf499b42…`; central's own per-node outcome log records 6 accepted / 0 failed; `grep -ci clusterclient` = **0** on both central nodes | 2 |
| 2 | Acks return over ClusterClient (site mode unchanged) and the deployment **seals** | **PASS** — deployment `8DF9AAE0…` `Status=2` (Sealed) with all six `NodeDeploymentState` rows `Status=1` (Applied) inside ~90 ms; all four site nodes hold live `Connected to [akka.tcp://…/system/receptionist]` associations, which is their only ack path in 1A | 2 |
| 3 | `RestartDriver` + `ReconnectDriver` land on the target site node | **PASS (composed)** — node-side leg driven live in run 1 (both commands acted on site-a-1). The AdminUI half **cannot be driven and this is now proven, not assumed**: `<DriverStatusPanel` has **0** usages repo-wide, seven rendered pages contain no Restart/Reconnect text or button, and the only non-script-analysis HTTP POST in the whole app is `/api/deployments`. Closed by composition — check 4 proves the *same* `AdminOperationsActor.Dispatch` → `DispatchOverGrpc` origin path live from central for an admin-originated command | 1 + 2 |
| 4 | `AlarmCommand` from the AdminUI lands and the condition acknowledges | **PASS** — Ack clicked on the real `/alerts` page against a live `gate481-alarm` ACTIVATED row; central fanned **one** `AlarmCommand` (`correlationId=ccafe375…`, minted once) to all six over gRPC, and the condition acknowledged — a new `ACKNOWLEDGED` row appeared 0.54 s later carrying the operator's identity (`multi-role-test`) | 2 |
| 5 | Unauthenticated ⇒ `PermissionDenied`; correct dev PSK ⇒ accepted | **PASS** — 3/3 probes as designed (run 1; not repeated — nothing in the fix touches the interceptor) | 1 |
| 6 | **Wrong-key isolation** | **PASS** — both directions rejected, with a positive control proving the rejection is the key and not reachability (run 1) | 1 |
| 7 | NULL `GrpcPort` node skipped with the warn-once; fan continues | **PASS** — with `site-b-2:4053` NULLed, both central nodes logged the warn **once** each and the next dispatch (`correlationId=88a9e0a2…`) was accepted by exactly **5** nodes, site-b-2 absent; the deployment then honestly ended `Status=4` (TimedOut) with site-b-2 alone at `Status=0`. Row restored; a later dispatch sealed 6/6 | 2 |
| 8 | Telemetry + config-serve + LocalDb sync unaffected: 0 auth failures | **PASS, and no longer vacuous** — 0 denials on all four interceptors across all six nodes, **and all three planes carried real traffic this time**: site nodes fetched + SHA-verified 28 KB artifacts from `http://central-1:4055`, `Telemetry stream opened` on every node with live driver health rendering on `/hosts`, LocalDb sync active on the site pairs | 2 |
### Run 2 provenance
`docker compose -f docker-dev/docker-compose.yml up -d --build` from the worktree, **exit 0**;
image `otopcua-host:dev` rebuilt to `1d8452a29018` at 12:15 UTC (run 1's preserved
`otopcua-host:grpc-1a` was stale and deliberately **not** reused). The six host nodes plus
`migrator` and `cluster-seed` were recreated (both one-shots exit 0); `sql` and its volume were
kept, and `traefik` was reused unchanged (its service definition did not change). Nothing else
touched the stack during the run, so it is not concurrent-rebuild-invalidated. Same rig-only
flip and same three Serilog Debug overrides as run 1; both reverted afterwards, never committed.
```
$ for c in central-1 central-2 site-a-1 site-b-1; do docker inspect otopcua-dev-$c-1 \
--format '{{range .Config.Env}}{{println .}}{{end}}' | grep '^MeshTransport__Mode='; done
central-1 MeshTransport__Mode=Grpc
central-2 MeshTransport__Mode=Grpc
site-a-1 MeshTransport__Mode=ClusterClient
site-b-1 MeshTransport__Mode=ClusterClient
```
### Run 2 evidence
**Boot — the fix works, and the race it was hardened against is real.**
```
$ for c in central-1 central-2 site-a-1 site-a-2 site-b-1 site-b-2; do … done
central-1 running restarts=0 mode=Grpc hostingFailed=0 resolveErr=2
central-2 running restarts=0 mode=Grpc hostingFailed=0 resolveErr=0
site-a-1 running restarts=0 mode=ClusterClient hostingFailed=0 resolveErr=0
site-a-2 running restarts=0 mode=ClusterClient hostingFailed=0 resolveErr=0
site-b-1 running restarts=0 mode=ClusterClient hostingFailed=0 resolveErr=0
site-b-2 running restarts=0 mode=ClusterClient hostingFailed=0 resolveErr=0
```
See **finding 5** below for the two `resolveErr` lines on central-1 — they are the PreStart
race losing on a real boot, and the memoize retry healing it was verified live rather than
argued.
**Check 1 — the fan, both ends.** One envelope, one correlation id, six recipients:
```
$ for c in …; do docker logs otopcua-dev-$c-1 | grep "Mesh: accepted DispatchDeployment" | tail -1; done
central-1 [12:16:49 DBG] Mesh: accepted DispatchDeployment (correlationId=cf499b42c6914a578035b2b5bc1020f7) over gRPC; handed to the node communication actor.
central-2 [12:16:49 DBG] …same correlationId…
site-a-1 [12:16:49 DBG] …same correlationId…
site-a-2 [12:16:49 DBG] …same correlationId…
site-b-1 [12:16:49 DBG] …same correlationId…
site-b-2 [12:16:49 DBG] …same correlationId…
$ docker logs otopcua-dev-central-2-1 | grep "accepted by" # the node hosting the coordinator
[12:16:49 DBG] Mesh: DispatchDeployment (correlationId=cf499b42…) accepted by central-2:4053 over gRPC
[12:16:49 DBG] … accepted by site-b-1:4053 … [12:16:49 DBG] … accepted by site-a-2:4053 …
[12:16:49 DBG] … accepted by site-a-1:4053 … [12:16:49 DBG] … accepted by central-1:4053 …
[12:16:49 DBG] … accepted by site-b-2:4053 …
$ docker logs … | grep -c "was NOT delivered to" → 0 on both central nodes
# Zero ClusterClient on the command side, both central nodes:
central-1 MeshClusterClientCreated=0 clusterclientAnyLine=0
central-2 MeshClusterClientCreated=0 clusterclientAnyLine=0
```
The `clusterclientAnyLine=0` count is over the whole log with
`Serilog:MinimumLevel:Override:Akka.Cluster.Tools.Client=Debug` in force, so it covers Akka's
own ClusterClient internals and not merely this repo's creation message.
**Check 2 — mixed transport, end to end.** Commands out over gRPC (above), acks back over
ClusterClient:
```
$ … SELECT Status FROM dbo.Deployment WHERE DeploymentId='8df9aae0-…'
2 -- Sealed
$ … SELECT NodeId, Status, AppliedAtUtc FROM dbo.NodeDeploymentState WHERE DeploymentId='8df9aae0-…'
central-1:4053|1|2026-08-06 12:16:49.939 site-a-1:4053|1|2026-08-06 12:16:49.977
central-2:4053|1|2026-08-06 12:16:49.915 site-a-2:4053|1|2026-08-06 12:16:49.950
site-b-1:4053|1|2026-08-06 12:16:49.957 site-b-2:4053|1|2026-08-06 12:16:49.967
$ for c in site-a-1 site-a-2 site-b-1 site-b-2; do docker logs … | grep "receptionist" | tail -1; done
site-a-1 [12:15:20 INF] Connected to [akka.tcp://otopcua@central-1:4053/system/receptionist]
site-a-2 [12:15:20 INF] Connected to [akka.tcp://otopcua@central-2:4053/system/receptionist]
site-b-1 [12:15:20 INF] Connected to [akka.tcp://otopcua@central-1:4053/system/receptionist]
site-b-2 [12:15:25 INF] Connected to [akka.tcp://otopcua@central-1:4053/system/receptionist]
$ grep -c "No central ClusterClient" → 0 on all six # no ack was dropped for want of a client
```
A site node has **no** gRPC ack path in 1A (that is 1B), so the receptionist association above
is not merely present, it is the only route those four Applied rows could have taken.
**Check 3 — why the AdminUI half is undrivable, established rather than asserted:**
```
$ grep -rn "<DriverStatusPanel" --include="*.razor" --include="*.cs" src/ | wc -l
0 # the component holding the buttons is referenced nowhere
$ grep -rn "DriverStatusPanel" --include="*.razor" src/ | grep -v "Shared/Drivers/DriverStatusPanel.razor:"
…5 hits, all of them code COMMENTS in Alerts.razor / Hosts.razor ("mirrors DriverStatusPanel…")
$ node scan-ui.js # every page a Designer/Administrator/Operator can reach
/ | reconnect/restart text: false | buttons: ["Sign out"]
/hosts | reconnect/restart text: false | buttons: ["Sign out","Refresh"]
/fleet | reconnect/restart text: false | buttons: ["Sign out","Refresh"]
/clusters | reconnect/restart text: false | buttons: ["Sign out"]
/clusters/SITE-A | reconnect/restart text: false | buttons: ["Sign out"]
/alerts | reconnect/restart text: false | buttons: ["Sign out","Clear"]
/deployments | reconnect/restart text: false | buttons: ["Sign out","Deploy current configuration"]
$ grep -rn "MapPost|MapPut|MapDelete" src/…/AdminUI src/…/Host
…/Api/DeployApiEndpoints.cs:43: app.MapPost("/api/deployments", …)
…/ScriptAnalysis/ScriptAnalysisEndpoints.cs:19-24: /diagnostics /completions /hover /signature-help /format /inlay-hints
```
There is no button and no endpoint. What closes the check is that `RestartDriver`,
`ReconnectDriver` and `AlarmCommand` all leave central through the **same three lines** —
`AdminOperationsActor.Dispatch` → `MeshCommand` → `CentralCommunicationActor.DispatchOverGrpc`
— and check 4 drove that path live from the AdminUI for `AlarmCommand`, while run 1 drove the
node-side decode-and-act for the two driver commands. The only per-type difference left is
`MeshTransportDtoMapper.ToEnvelope`, which the T0.2 goldens cover member-for-member.
**Check 4 — AdminUI ack, fanned by central, acknowledged by the owner.** The rig's one
scripted alarm sits Normal indefinitely (`hr200 > 70000`), so a transition was manufactured by
editing its predicate to an always-true form and deploying; the `/alerts` page was already open
when it fired, the Ack button was clicked, and the predicate + hash were restored afterwards
(verified byte-identical by `SELECT`) with a further deploy:
```
page open at 2026-08-06T12:20:59.036Z initial: empty window
ROWS: 12:21:00.017 gate481-alarm | PressHigh | EQ-6e09bcc2d907 | ACTIVATED | 750 | system | press high
ROW BUTTONS: ["Ack","Shelve","Shelve (timed)","Unshelve"]
clicking Acknowledge at 2026-08-06T12:21:00.362Z
after click: 12:21:00.557 | gate481-alarm | EQ-6e09bcc2d907 | ACKNOWLEDGED | 750 | multi-role-test | press high
$ docker logs otopcua-dev-central-2-1 | grep AlarmCommand
[12:21:00 DBG] Mesh: AlarmCommand (correlationId=ccafe3754aca40cfa8aa0a0602c86cbb) accepted by site-b-2:4053 over gRPC
[12:21:00 DBG] … accepted by central-1:4053 … [12:21:00 DBG] … accepted by central-2:4053 …
[12:21:00 DBG] … accepted by site-a-1:4053 … [12:21:00 DBG] … accepted by site-b-1:4053 …
[12:21:00 DBG] … accepted by site-a-2:4053 …
$ for c in …; do grep "Mesh: accepted AlarmCommand" …; done → all six, same correlationId
```
One correlation id across all six recipients is the "encode once, before the fan" rule holding
for the one command whose CLR record has no correlation member of its own — the case the design
called out as the reason encoding is hoisted out of the loop.
**Check 7 — the skip, the warn, and the honest deadline.**
```
$ … UPDATE dbo.ClusterNode SET GrpcPort=NULL WHERE NodeId='site-b-2:4053' # 12:23:14
# one warn per central node, once each, within one ContactRefreshSeconds:
central-2 [12:23:20 WRN] ClusterNode site-b-2:4053 has no GrpcPort; it exposes no command endpoint
and is skipped in this dispatch set (further skips of this node are silent). Commands
will not reach it until the row carries a port
central-1 [12:23:19 WRN] …identical…
# (each node also logs the TelemetryNodeSource twin — "exposes no telemetry stream … skipped in
# this dial refresh" — expected collateral: both target sets read the same column)
# dispatch at 12:23:36 — accepted by five, site-b-2 absent:
central-2 [12:23:37 DBG] … (correlationId=88a9e0a2…) accepted by central-2 / site-a-2 / site-b-1
/ site-a-1 / central-1 :4053 over gRPC
# cumulative per-node accept counts after this dispatch — site-b-2 one behind everyone:
central-1 4 central-2 4 site-a-1 4 site-a-2 4 site-b-1 4 site-b-2 3
$ … SELECT Status FROM dbo.Deployment WHERE DeploymentId='5c8bdc9e-…' → 4 (TimedOut)
$ … SELECT NodeId, Status FROM dbo.NodeDeploymentState …
central-1|1 central-2|1 site-a-1|1 site-a-2|1 site-b-1|1 site-b-2|0
```
Row restored to `4056` and a later dispatch sealed 6/6 (`080EBB0B…`, `Status=2`, every node
`Status=1`). One operational detail worth carrying: **the skip persists for up to a full
`ContactRefreshSeconds` after the row is repaired** — a deploy issued 5 s after the restore
(`DF1C0B74…`) was still skipped and still TimedOut, because the dispatch set is only rebuilt on
the refresh tick.
**Check 8 — denials, and proof the planes were actually used.**
```
$ for c in central-1 central-2 site-a-1 site-a-2 site-b-1 site-b-2; do … done
central-1 telemetry=0 configserve=0 localdbsync=0 mesh=0
central-2 telemetry=0 configserve=0 localdbsync=0 mesh=0
site-a-1 telemetry=0 configserve=0 localdbsync=0 mesh=0
site-a-2 telemetry=0 configserve=0 localdbsync=0 mesh=0
site-b-1 telemetry=0 configserve=0 localdbsync=0 mesh=0
site-b-2 telemetry=0 configserve=0 localdbsync=0 mesh=0
# config-serve carried real, authenticated traffic:
site-a-1 [12:20:59 INF] Fetched + verified artifact for 8d486c6b… from http://central-1:4055 (28313 bytes).
# telemetry likewise, end to end into the UI:
$ for c in …; do grep -c "Telemetry stream opened"; done → 3,4,3,3,3,5
/hosts → CLUSTER SITE-A … sa-modbus2 SITEA-modbus Modbus HEALTHY 1 host 1 tag 12:30:49 UTC 0 errors
```
The five telemetry `failed … reconnecting` warnings in the run are all accounted for: three at
12:15:20 (central dialing nodes that had not finished binding `:4056`) and two at 12:28:50 /
12:29:30 (the deliberate central-2 stop in finding 5). None is an auth failure.
### Phase 1A finding 5 — the PreStart race loses on real boots, and the memoize is what saves it
The hardening commit's premise is not hypothetical. On the run-2 boot, **central-1 lost the
race on both dependencies**:
```
$ docker logs otopcua-dev-central-1-1 | grep -A3 "Failed to resolve"
[12:15:19 ERR] Failed to resolve IMeshCommandDialer for the mesh command boundary; continuing
without it. The guards below report what that costs
Akka.Hosting.MissingActorRegistryEntryException: No actor registered for key
ZB.MOM.WW.OtOpcUa.ControlPlane.AuditWriterActorKey
at ZB.MOM.WW.OtOpcUa.AdminUI.Audit.ActorAuditWriter..ctor(ActorRegistry registry)
[12:15:19 ERR] Failed to resolve IMeshPskProvider for the mesh command boundary; …same cause…
```
It is not rare: across the three admin boots observed in run 2 (central-1 and central-2 at
12:15, central-2 again at 12:29 after the stop/start), **two of three lost it**. Had the fix
stopped at PreStart-only resolution, those nodes would have held a permanent silent `null` —
no dialer, and a never-published known-cluster gate — and the failure would only have surfaced
the first time the coordinator singleton moved onto such a node.
**The memoize heals it, verified on the node that actually lost the race.** Run 2's dispatches
were all fanned by central-2 (the singleton host), so central-1's dialer was never exercised
by them. Central-2 was therefore stopped to move the singleton, and a dispatch driven while
central-1 alone held it:
```
$ docker compose stop central-2 # 12:28:39
$ curl … /health/ready → self akka.tcp://otopcua@central-1:4053, members 1 # 12:28:57
$ curl -X POST /api/deployments -d '{"createdBy":"1a-gate-central1-heal"}' # 12:29:01
$ docker logs otopcua-dev-central-1-1 | grep -E "accepted by|no command dialer is wired"
[12:29:02 DBG] Mesh: DispatchDeployment (correlationId=ad97d386…) accepted by site-b-1:4053 over gRPC
[12:29:02 DBG] … accepted by site-a-2 … site-b-2 … site-a-1 … central-1 … # 5 live nodes
# no "no command dialer is wired" line, and no further "Failed to resolve"
$ docker compose start central-2 → members 2, unreachable 0, status Healthy # 12:29:51
```
So the boot-time Error is **cosmetically alarming but functionally healed**, and that is now an
observed property rather than a design claim. Two follow-ups for Phase 2 rather than for 1A:
the Error text says "continuing without it", which reads as permanent and should say that the
first use retries; and a `Failed to resolve` line on a healthy node is exactly the kind of
signal a soak's PSK-drift/error grep will trip over.
**One incidental observation, not a check.** The deployment `RevisionHash` is deterministic
(three separate no-change dispatches all produced `642b9d6f79b9…`), but restoring the alarm
predicate's two text columns did **not** restore the pre-gate hash `6178a8f302aa…` — so the
artifact covers a column the `UPDATE` bumped (`Script.RowVersion` is the obvious candidate).
The rig's configuration content is byte-identical to its pre-gate state; only the hash moved.
Not investigated further — out of this program's scope.
### Run 1 record — the defect that blocked it
Branch `feat/grpc-node-command` @ `fbf7b708`. Image rebuilt from this worktree
(`/Users/dohertj2/Desktop/OtOpcUa/.worktrees/grpc-1a`) so it carries this branch;
`docker compose -f docker-dev/docker-compose.yml up -d --build` exited 0 and **all eight
containers were recreated** — `migrator` and `cluster-seed` re-ran to completion (exit 0), the
six host nodes and traefik were recreated, the `sql` container and its named volume were kept.
Nothing else touched the stack afterwards, so this run is not a concurrent-rebuild-invalidated
one. Rig-only flip: central-1/-2 `MeshTransport__Mode: "Grpc"` (hard value); **site nodes left
on the compose default `ClusterClient`** (coexistence is the point). Three Serilog
`MinimumLevel:Override` Debug scopes were added on all six nodes for this run only
(`ZB.MOM.WW.OtOpcUa.Host.Grpc`, `ZB.MOM.WW.OtOpcUa.ControlPlane.Communication`,
`Akka.Cluster.Tools.Client`) — the accept/dispatch lines the checks read are `Debug` and the
rig's default Serilog level is `Information`, so without them checks 1/3 would have had no
observable signal. Flip and overrides reverted after the gate, never committed.
**Run 1 verdict: FAIL.** Neither central node booted, so nothing that originates at central
could be driven: checks 1, 2 and 7 were **BLOCKED** outright, 3 and 4 reached only their
node-side legs, and check 8's zero-denial result was vacuous for telemetry and config-serve
(central dials both and never booted). Checks 5 and 6 **PASSED** and are the run-1 evidence
carried into the combined table above. Root cause was defect 4 — resolving `IMeshCommandDialer`
/ `IMeshPskProvider` inside the first `WithActors` starter drags in `ISecretResolver` →
`IAuditWriter` → `ActorAuditWriter`, whose constructor reads `AuditWriterActorKey` out of the
`ActorRegistry` before the singleton that registers it has run. It was **mode-independent**
(reproduced on the same image with `MeshTransport__Mode=ClusterClient`), so it was a defect of
the branch, not of the gate's flip, and it would have failed a `ClusterClient` rig equally.
### Run 1 evidence
**Provenance** — flip in effect on central only, sites untouched:
```
$ for c in central-1 central-2 site-a-1 site-b-1; do docker inspect otopcua-dev-$c-1 \
--format '{{range .Config.Env}}{{println .}}{{end}}' | grep '^MeshTransport__Mode='; done
central-1 MeshTransport__Mode=Grpc
central-2 MeshTransport__Mode=Grpc
site-a-1 MeshTransport__Mode=ClusterClient
site-b-1 MeshTransport__Mode=ClusterClient
$ docker images --format '{{.Repository}}:{{.Tag}}|{{.ID}}|{{.CreatedAt}}' | grep otopcua-host
otopcua-host:dev|69360a173f1f|2026-08-06 07:20:14 -0400 EDT # rebuilt for this run
```
**Defect 4 — the boot failure (checks 1/2/7 blocked by it).** Both central nodes, identical:
```
$ docker logs otopcua-dev-central-1-1 2>&1 | sed -n '25,45p'
[11:21:27 INF] OPC UA server started on opc.tcp://0.0.0.0:4840
[11:21:27 FTL] Unable to start AkkaHostedService - shutting down application
Akka.Hosting.MissingActorRegistryEntryException: No actor registered for key
ZB.MOM.WW.OtOpcUa.ControlPlane.AuditWriterActorKey
at Akka.Hosting.ActorRegistry.Get[TKey]()
at ZB.MOM.WW.OtOpcUa.AdminUI.Audit.ActorAuditWriter..ctor(ActorRegistry registry)
in /src/src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/Audit/ActorAuditWriter.cs:line 24
...
at ZB.MOM.WW.Secrets.DependencyInjection.SecretsServiceCollectionExtensions.<>c.<AddZbSecrets>b__0_5(IServiceProvider sp)
...
at ZB.MOM.WW.Secrets.DependencyInjection.SecretsServiceCollectionExtensions.<>c.<AddZbSecrets>b__0_6(IServiceProvider sp)
...
at ZB.MOM.WW.OtOpcUa.ControlPlane.ServiceCollectionExtensions.<>c.<WithOtOpcUaControlPlaneSingletons>b__7_0(
ActorSystem system, IActorRegistry registry, IDependencyResolver resolver)
in /src/src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/ServiceCollectionExtensions.cs:line 74
at Akka.Hosting.AkkaConfigurationBuilder.StartAsync(ActorSystem sys)
at Akka.Hosting.AkkaHostedService.StartAsync(CancellationToken cancellationToken)
```
**Mode-independence, proven rather than argued** — same image, central-1's exact environment
with `MeshTransport__Mode=ClusterClient`, own hostname + self-seed so it cannot disturb the rig:
```
$ docker run -d --name scratch-cc-mode --network otopcua-dev_default --env-file c1-env.txt \
-e Cluster__PublicHostname=scratch-cc-mode \
-e Cluster__SeedNodes__0=akka.tcp://otopcua@scratch-cc-mode:4053 otopcua-host:dev
$ docker logs scratch-cc-mode 2>&1 | grep -m4 -E "MissingActorRegistry|Hosting failed|Host starting"
[11:28:45 INF] OtOpcUa.Host starting with roles=[admin,driver,cluster-main] (admin=True, driver=True)
Akka.Hosting.MissingActorRegistryEntryException: No actor registered for key
ZB.MOM.WW.OtOpcUa.ControlPlane.AuditWriterActorKey
[11:28:47 ERR] Hosting failed to start
```
Scratch container removed after capture (`docker rm -f scratch-cc-mode`); the six real nodes
were untouched by it.
**It presents as a healthy container.** Worth carrying into Phase 2/4 readings — the failed
process does not exit, so `docker ps` shows `Up`, `RestartCount=0`, and only an HTTP probe or
the log tells the truth:
```
$ docker inspect otopcua-dev-central-1-1 -f '{{.State.Status}} restarts={{.RestartCount}}'
running restarts=0
$ docker run --rm --network otopcua-dev_default curlimages/curl -s -o /dev/null \
-w '%{http_code}\n' --max-time 5 http://central-1:9000/health/ready
000
$ curl -s -o /dev/null -w '%{http_code}\n' http://localhost:9200/health/ready
503
```
**Checks 5 + 6 — the wire-probe auth matrix** (`-import-path` onto the in-repo proto; the node
has no reflection). An empty `CommandEnvelope` has no `oneof` arm set, so `FromEnvelope` throws
and the service answers `InvalidArgument` — **after** the interceptor ran, which is exactly why
it is the positive signal here: `PermissionDenied` means the key was refused, `InvalidArgument`
means the key was accepted and only the payload was rejected:
```
$ P=src/Core/ZB.MOM.WW.OtOpcUa.Commons/Protos
$ G="docker run --rm --network otopcua-dev_default -v $PWD/$P:/protos:ro fullstorydev/grpcurl \
-plaintext -import-path /protos -proto mesh_transport.proto"
# 5a no bearer -> site-a-1
Code: PermissionDenied Message: Node command authentication failed.
# 5b -H 'authorization: Bearer dev-mesh-psk-SITE-A' -> site-a-1
Code: InvalidArgument Message: Command envelope could not be decoded.
# 6a -H 'authorization: Bearer dev-mesh-psk-SITE-B' -> site-a-1
Code: PermissionDenied Message: Node command authentication failed.
# 6b -H 'authorization: Bearer dev-mesh-psk-SITE-A' -> site-b-1
Code: PermissionDenied Message: Node command authentication failed.
# 6c -H 'authorization: Bearer dev-mesh-psk-SITE-B' -> site-b-1 (positive control)
Code: InvalidArgument Message: Command envelope could not be decoded.
```
**Checks 3 + 4 — the node-side leg, driven with real envelopes** over the same authenticated
h2c path central would use. This is the first live proof that `MeshCommandAuthInterceptor` is
actually invoked on a real Kestrel pipeline (register item 4) and that
`NodeCommandGrpcService` → `NodeCommunicationActor` → EventStream → `DriverHostActor` works
end to end:
```
$ $G -H 'authorization: Bearer dev-mesh-psk-SITE-A' -H 'x-otopcua-cluster: SITE-A' \
-d '{"correlationId":"8461008e...","restartDriver":{"clusterId":"SITE-A",
"driverInstanceId":"SITEA-modbus","actorByUserName":"1a-gate",
"correlationId":"8461008e..."}}' \
site-a-1:4056 mesh_transport.v1.NodeCommandService/Execute
{}
$ docker logs otopcua-dev-site-a-1-1 2>&1 | tail
[11:31:01 DBG] Mesh: accepted RestartDriver (correlationId=8461008e37654724a292219202d78dfa) over gRPC; handed to the node communication actor.
[11:31:01 INF] DriverHost site-a-1:4053: restarting driver SITEA-modbus by request of 1a-gate
[11:31:01 INF] DriverHost site-a-1:4053: spawned Modbus driver SITEA-modbus (stub=False)
[11:31:55 DBG] Mesh: accepted ReconnectDriver (correlationId=29c5dc77283f4eb7869d0c7d6686cadd) over gRPC; handed to the node communication actor.
[11:31:55 INF] DriverHost site-a-1:4053: reconnecting driver SITEA-modbus by request of 1a-gate
[11:31:55 INF] DriverInstance SITEA-modbus: ForceReconnect requested by admin; re-entering Reconnecting
[11:31:56 DBG] Mesh: accepted AlarmCommand (correlationId=39ba451173954229a71b9dd1e00ac1b9) over gRPC; handed to the node communication actor.
```
**Check 8 — interceptor denial counts across every node, whole run:**
```
$ for c in central-1 central-2 site-a-1 site-a-2 site-b-1 site-b-2; do … done
central-1 telemetry=0 configserve=0 localdbsync=0 mesh=0
central-2 telemetry=0 configserve=0 localdbsync=0 mesh=0
site-a-1 telemetry=0 configserve=0 localdbsync=0 mesh=2 # the two deliberate negatives
site-a-2 telemetry=0 configserve=0 localdbsync=0 mesh=0
site-b-1 telemetry=0 configserve=0 localdbsync=0 mesh=1 # the one deliberate negative
site-b-2 telemetry=0 configserve=0 localdbsync=0 mesh=0
```
The `TwoNodeClusterHarness` I-1 null-provider Error (a known test-harness artifact) did **not**
appear in any rig log — `grep -c "IMeshPskProvider"` = 0 on all six nodes, as expected for
production wiring where every admin node registers the provider.
### Phase 1A defect — **CLOSED by `c0475f2d` + `e54883a8`, re-gated in run 2**
**Defect 4 — an admin node cannot boot: eager DI resolution inside the first Akka starter.**
`WithOtOpcUaControlPlaneSingletons`'s `WithActors` block resolves `IMeshCommandDialer` and
`IMeshPskProvider` from `IServiceProvider` to hand to `CentralCommunicationActor.Props`.
`MeshGrpcCommandDialer` takes `IMeshPskProvider`, which takes `ISecretResolver`, which
`AddZbSecrets` builds by resolving `IAuditWriter`, which on an admin node is
`ActorAuditWriter` — and its constructor does `registry.Get<AuditWriterActorKey>()`. That
singleton is registered ~60 lines later in the same builder, so at this point the key is
absent and `ActorRegistry.Get` throws `MissingActorRegistryEntryException`, failing
`AkkaHostedService.StartAsync` and the whole host.
Three things make this worth a pinned regression rather than a one-line fix note:
- **It is not mode-gated.** The two `GetService` calls run in every transport mode by design
(so a mode flip needs no restart), so the eager graph is walked on every admin boot. Proven
above with a `ClusterClient` scratch container on the same image.
- **It is invisible to the suite.** The 1A.2 container-activation test
(`It_activates_from_the_container_despite_the_optional_client_factory_parameter`) resolves
the dialer from a service collection that does not include `AddZbSecrets`, and
`TwoNodeClusterHarness` registers no `IMeshPskProvider` at all — so no existing test walks
the real graph in starter order. This is the same class as scadaproj#1 ("only visible with
the DI graph built in a real host") and as mesh Phase 6's `IClusterRoleInfo` boot crash.
- **The container lies about it.** The process does not exit, so `docker ps` reports `Up` with
`restarts=0` while the node serves no HTTP at all. Any Phase 2/4 readiness check that reads
container status rather than `/health/ready` would score this as healthy.
The shape of the fix is the one this file already uses two lines above for the coordinator
proxy — hand the actor a lazy accessor (`Func<IMeshCommandDialer?>` / `Func<IMeshPskProvider?>`,
or resolve on first use) so nothing is constructed while the `ActorSystem` is still building.
**That is what shipped.** `c0475f2d` moved both dependencies behind `Func<>` invoked in
`PreStart`, pinned by `ControlPlaneStarterOrderTests` (boots a real admin host through the
production starter order and asserts `StartAsync`; verified RED against the eager resolve in
3/3 modes). `e54883a8` then made the resolution memoize-on-first-use rather than PreStart-only,
because `PreStart` runs concurrently with the remaining starters and can still lose the race —
which **run 2 observed happening on 2 of 3 real admin boots**, so that hardening was not
theoretical. Both are re-gated above; see finding 5 for the live self-heal proof.
### Rig state after run 1 (superseded by the run-2 restoration below)
The compose flip was reverted and the six nodes recreated on the committed all-`ClusterClient`
compose. Because defect 4 was mode-independent, the branch image could not produce a healthy
central pair on *any* mode, so the rig was returned to its pre-gate image rather than left with
a dead central pair: the run-1 build was preserved as `otopcua-host:grpc-1a` (`69360a173f1f`)
and the pre-gate build (`8bd81d8aea9b`) re-tagged back to `otopcua-host:dev`.
### Rig state after run 2 — **restored, six healthy, on the branch image**
The flip was reverted (`git checkout docker-dev/docker-compose.yml`) and the six nodes
recreated on the committed all-`ClusterClient` compose. No image substitution was needed this
time: `otopcua-host:dev` is the run-2 build of the committed source (`1d8452a29018`), and it
boots a healthy central pair, so the rig runs the code under test. The run-1 workaround tag
`otopcua-host:grpc-1a` was deleted to stop anyone reusing a stale image.
```
central-1 running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0
central-2 running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0
site-a-1 running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0
site-a-2 running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0
site-b-1 running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0
site-b-2 running restarts=0 mode=ClusterClient image=1d8452a29018 hostingFailed=0
$ curl … /health/ready → status Healthy, members 2, unreachable 0
$ grep -c "Mesh ClusterClient for cluster" → 3 on each central node (MAIN + SITE-A + SITE-B)
```
The three ClusterClients reappearing on the rollback is an incidental but useful result: the
dark switch still rolls back cleanly on the fixed branch, which is what Phase 2 will depend on
if its cutover has to be reversed.
---
@@ -228,8 +704,21 @@ checks 6–7.
### Carry-forwards
Anything a phase gate could not drive gets named here with why + what proves it instead, and
is re-checked in the next gate (the ScadaBridge proportionate-gate rule). Currently: none.
is re-checked in the next gate (the ScadaBridge proportionate-gate rule).
| Item | Deferred from | Why it could not be driven | Proven instead by | Status |
|---|---|---|---|---|
| Checks 1, 2, 7 in full — deploy dispatch over gRPC, mixed-transport seal, NULL-`GrpcPort` skip | 1A run 1 | Both central nodes failed to boot (defect 4) | — | **CLOSED in 1A run 2** — all three PASS live |
| Check 4 acknowledge half — the condition actually acknowledging | 1A run 1 | The rig's one `ScriptedAlarm` sits Normal and its owning central node was down | — | **CLOSED in 1A run 2** — acked from the real `/alerts` page, `ACKNOWLEDGED` row observed |
| Check 8 for telemetry + config-serve | 1A run 1 | Central dials both planes and never booted, so neither carried a call — 0 denials was vacuous | — | **CLOSED in 1A run 2** — both planes carried authenticated traffic with 0 denials |
| Check 3 central half — `RestartDriver`/`ReconnectDriver` **originating at the AdminUI** | 1A runs 1 + 2 | **Structurally undrivable, not a rig problem:** `<DriverStatusPanel` has 0 usages repo-wide (the 5 name hits are comments), no rendered page carries a Restart/Reconnect control, and the only non-script-analysis HTTP POST is `/api/deployments`. Pre-existing on `master`, not caused by this program | Composition the controller accepted: run 1's live node-side leg for both commands + run 2's check 4, which drives the identical `AdminOperationsActor.Dispatch` → `DispatchOverGrpc` origin path from the AdminUI for another command type; per-type difference is `ToEnvelope`, covered by the T0.2 goldens | **CLOSED by composition.** The orphaned panel itself is out of this program's scope — file separately if the control is meant to exist |
| `Failed to resolve` Error text says "continuing without it", which reads permanent although first use retries; and it will trip a soak's error grep on a healthy node | 1A run 2 (finding 5) | Cosmetic/observability, not behaviour | The self-heal proven live (finding 5) | Phase 2 |
### Defects the gates caught that the suite did not
Recorded per phase as they occur, with root cause + the pinned regression test.
| # | Phase | Defect | Root cause | Status |
|---|---|---|---|---|
| 4 | 1A gate run 1 | **An admin node cannot boot at all** — `MissingActorRegistryEntryException: AuditWriterActorKey`, both central nodes, every transport mode | The first `WithActors` starter eagerly resolves `IMeshCommandDialer`/`IMeshPskProvider`, whose graph reaches `ActorAuditWriter`, whose ctor reads an `ActorRegistry` key registered ~60 lines later in the same builder | **CLOSED** — `c0475f2d` (`Func<>` + `PreStart`, pinned red-first by `ControlPlaneStarterOrderTests`) + `e54883a8` (memoize-on-first-use). Re-gated in run 2: 6/6 nodes boot, 0 `Hosting failed to start` |
| 5 | 1A gate run 2 | **The `PreStart` resolve loses its race on real boots** — 2 of 3 observed admin boots logged `Failed to resolve IMeshCommandDialer` / `IMeshPskProvider` with the same `AuditWriterActorKey` cause | `PreStart` runs on a dispatcher thread concurrently with the builder's remaining starters, so it can still precede `AuditWriterActorKey`'s registration | **Not a defect — it is the case `e54883a8` was written for, and the memoize demonstrably heals it.** Proven live by moving the coordinator singleton onto the node that lost the race (stop central-2) and watching it fan a dispatch to all five live nodes with no dialer error. Residual is cosmetic: the Error text and its effect on soak greps → Phase 2 |
docs/plans/2026-08-06-clusterclient-to-grpc-tracking.md
+46 -6
View File
@@ -23,8 +23,8 @@ posture).
| Phase | Scope | Branch | Status | PR / merge SHA | Live gate |
|---|---|---|---|---|---|
| 0 | Proto + mappers + goldens; both auth interceptors; `IMeshPskProvider`; options/validators (+`Grpc` value, SplitTopology relaxation); rig dark config; stale-comment fixes; FleetStatus issue filed | `feat/grpc-phase0-contract-auth` | **MERGED to `master`** @ `10b6d209` (2026-08-06) — all 6 tasks complete @ `23555ed0`; every task spec ✅ + code ✅ (Batch B on Opus with a closed fix loop); issues #542 + #543 filed | `10b6d209` | **PASS 4/4** |
| 1A | Central→node command plane (`NodeCommandService` on the Telemetry listener; `MeshGrpcCommandDialer`; third mode branch in `CentralCommunicationActor`) — **worktree A** | `feat/grpc-node-command` | NOT STARTED | — | PENDING |
| 1B | Node→central ack plane (`CentralCommandService` on the ConfigServe listener; `GrpcApplyAckSender`; third branch in `NodeCommunicationActor`) — **worktree B**, rebases on 1A | `feat/grpc-central-ack` | NOT STARTED | — | PENDING |
| 1A | Central→node command plane (`NodeCommandService` on the Telemetry listener; `MeshGrpcCommandDialer`; third mode branch in `CentralCommunicationActor`) — **worktree A** | `feat/grpc-node-command` | **COMPLETE @ `e54883a8` — gate re-run PASSED 2026-08-06.** Run 1 @ `fbf7b708` FAILED on defect 4 (no admin node could boot, mode-independent); fixed by `c0475f2d` + `e54883a8` and re-gated. 1A.1 ✅ `64774652` + 1A.2 ✅ `6a862011` (+fixes `48dc2120`/`86633f14`) all reviewed spec ✅ + code ✅ | — | **PASS 8/8** (check 3's AdminUI half closed by composition — the control is structurally unreachable, see carry-forwards) |
| 1B | Node→central ack plane (`CentralCommandService` on the ConfigServe listener; `GrpcApplyAckSender`; third branch in `NodeCommunicationActor`) — **worktree B**, rebases on 1A | `feat/grpc-central-ack` | **1B.1 ✅ `14dd07d2` + 1B.2 ✅ `a5a789d4` (+fixes `e54711a2`/`4d607575`)** — all reviews ✅ (1B.2 on Opus: 1 Critical found+fixed, see defect log); awaiting 1A merge → rebase → 1B.3 gate | — | PENDING |
| 2 | Baseline reproduction on pre-flip build, then full-rig cutover (`OTOPCUA_MESH_MODE` default → `Grpc`) + soak + kill drills | (rig commit) | NOT STARTED | — | PENDING |
| 3 | Delete ClusterClient (code/HOCON/config/tests); SplitTopology requires `Grpc`; grep-gates; docs sweep incl. scadaproj index | `feat/grpc-phase3-deletion` | NOT STARTED | — | — |
| 4 | Terminal live gate on the post-deletion image (8 checks, provenance-guarded) | (on `master`) | NOT STARTED | — | PENDING |
@@ -46,11 +46,14 @@ phase DoD — PR + merge are user-owned.**
| 2026-08-06 | No transport interface — third mode branch inside the two choke-point actors (ScadaBridge §7.1 economy) | design §4.3 |
| 2026-08-06 | Coexistence granularity: central's mode governs commands, each node's mode governs its acks (per-pair flips possible — better than ScadaBridge's central-wide flag) | design §3.6 |
| 2026-08-06 | `AlarmCommandMessage.operation` is a proto `string`, NOT an enum — the CLR `AlarmCommand.Operation` is an unconstrained string, and an enum would silently decode unknown operations to `UNSPECIFIED` (the drop class this migration removes). Plan T0.1's "AlarmCommand verb" enum wording was a plan defect; only `ApplyOutcome` carries the enum-0 rule. T0.2's goldens must pin pass-through of an unrecognized operation string | proto comment + this log (T0.1 review) |
| 2026-08-06 | **Gate follows the endpoint, not the mode** (1A.2 I-2): known-cluster set published on every contact load in all three modes, from ALL `ClusterNode` rows; fan set unchanged. Survives dark-switch rollback with nothing for 1B to remember | tracking §register refinement + actor docs |
| 2026-08-06 | `CentralCommandService` mapped by PORT, not mode-gated (1B.1): a mode flip on a running fleet must not need a central restart before acks flow; matches every sibling service | `Program.cs` comment + this log |
| 2026-08-06 | Ack deadline is PER ENDPOINT, not per walk; the walk is at-least-once on the deadline path, absorbed idempotently by the coordinator's NodeId-keyed upsert | `MeshTransportOptions` docs + defect log #1 |
| 2026-08-06 | `correlation_id` lives on the envelope AND as inner fields on the three commands whose CLR record carries one — each command round-trips independently in the T0.2 mapper; the envelope covers `AlarmCommand` (no CLR correlation member). For the three, the inner field is authoritative on decode | proto comment + this log (T0.1 review) |
## Phase 1A security register (from the Batch B Opus code review — HARD GATE: all four close BEFORE `CentralCommandService` is mapped)
1. **Known-cluster-set membership gate in `TryGetPsk`** — central rebuilds its cluster set from
1. **CLOSED (1A.2 @ `48dc2120`, refined — see below).** Known-cluster-set membership gate in `TryGetPsk` — central rebuilds its cluster set from
`ClusterNode` rows every `ContactRefreshSeconds`; refuse a cluster id outside that set in
O(1) with no store touch. Closes BOTH open review findings at once: (a) the timing oracle
that currently defeats the uniform-denial anti-enumeration text (unknown cluster pays a
@@ -61,13 +64,33 @@ phase DoD — PR + merge are user-owned.**
audit singleton, attacker-chosen `Target` values, before any credential check). Namespace
confinement (`OT-MESH-PSK-` prefix + `SecretName` allow-list) means no disclosure risk —
this is flood/oracle only.
2. **`bool Matches(string clusterId, string presented)` on `IMeshPskProvider`** — keep
2. **CLOSED (1A.2 @ `14d21496`).** `bool Matches(string clusterId, string presented)` on `IMeshPskProvider` — keep
`FixedTimeEquals` inside the provider so a future verify-side caller cannot write `==` and
have it compile. Do before 1B adds the dial-side caller.
3. **DI-shape pin test** — `IMeshPskProvider` is `hasAdmin`-only and its pairing with the
central interceptor is by convention; a driver-only node injecting it throws on first call,
not at boot. Add a Host.IntegrationTests check (admin host resolves it; driver-only host
does NOT) when Phase 1 maps the services.
**Refinement on item 1 (decision, 2026-08-06):** the gate follows the ENDPOINT, not the mode —
published on every contact load in every transport mode (the interceptor and service are
port-mapped, not mode-gated, so a dark-switch rollback must not deactivate the gate), and
built from ALL `ClusterNode` rows (gate answers "does central know this cluster"; the fan set
stays `Enabled && !MaintenanceMode`, so gate ⊇ fan — this also dissolved the
maintenance/ack-auth coupling). Pinned by the every-mode Theory + gate-⊋-fan fact + the Dps
no-ClusterClient guard pin (`86633f14`). Residuals accepted + visible: gate is fail-open until
the first successful contact load (and stays stale if central boots with SQL down — warned
every tick); gate derives from the node table, so a `ServerCluster` with zero rows is outside
it.
**Follow-ups from the 1A.2/1B.2 Opus reviews (tracked, not blocking):** per-tick
`InvalidateAll` puts a cold-cache 3 s store read on the inbound auth path — MEASURE on the
1B.3 gate (ack burst inside ~3 s after a refresh tick), then consider diff-based
invalidation; the command plane is h2c-only so the per-cluster PSK crosses the network in
cleartext on a write surface — precedent-consistent (all sibling gRPC planes), record as
accepted risk + TLS-option issue (queued below); dialer minors (Lazy channel wrapper,
retired-endpoint eviction, gRPC deadline vs token cancellation, pre-existing
`LoadContactsFromDb` Task.Run(ct) pattern).
4. **Integration gate over real Kestrel h2c** — both new interceptors have unit coverage only;
the one-public-ctor reflection test approximates but cannot prove the interceptor is
actually invoked. Wire-probe in the 1A/1B live gates (already in the live-gate tables).
@@ -104,7 +127,11 @@ Defects found during execution (live-gate or review), with root cause + pinned t
| # | Phase | Defect | Root cause | Fix / pin |
|---|---|---|---|---|
| — | — | *(none yet)* | | |
| 1 | 1B.2 | **Failover silently defeated on a hanging central** (CRITICAL, caught by Opus code review): the ack deadline was one absolute `DateTime` shared across the endpoint walk — a hung (not refused) central-1 burned the whole budget and central-2 was dialed with an expired deadline | deadline hoisted outside the `foreach`; a kill-style drill can't see it (refused = fast-fail), only a hang discriminates | `e54711a2`; pinned by `Each_endpoint_gets_a_FRESH_deadline` (mutation-verified); 1B.3 gains a `docker pause` hang drill |
| 2 | 1A.2 | **Dark-switch rollback deactivated the known-cluster gate** (Opus review, I-2): gate published only under `Mode=Grpc` while the interceptor + service are port-mapped in every mode — a rollback to ClusterClient/Dps would silently restore the timing oracle + pre-auth flood | gate activation coupled to transport mode instead of the endpoint it protects | `48dc2120` — gate follows the endpoint; pinned by the every-mode Theory + Dps guard pin `86633f14` |
| 5 | 1A.3 (live gate, run 2) | **The `PreStart` resolve loses its race on real boots** — 2 of 3 observed admin boots logged `Failed to resolve IMeshCommandDialer` / `IMeshPskProvider`, same `AuditWriterActorKey` cause. NOT a defect: it is exactly the case the review-prescribed hardening was written for | `PreStart` runs on a dispatcher thread concurrently with the builder's remaining starters, so it can still precede starter #4 | **Closed by `e54883a8`, and the heal is PROVEN LIVE** — the coordinator singleton was moved onto the node that lost the race (stop central-2) and it fanned a dispatch to all five live nodes with no dialer error. Residual is cosmetic (Error text reads permanent; will trip a soak error-grep on a healthy node) → Phase 2 |
| 4 | 1A.3 (live gate, run 1) | **An admin node cannot boot** (BLOCKED 1A): both central nodes die at startup with `MissingActorRegistryEntryException: ZB.MOM.WW.OtOpcUa.ControlPlane.AuditWriterActorKey`, taking `AkkaHostedService` and the whole host with them. Mode-INDEPENDENT — reproduced on the same image with `MeshTransport__Mode=ClusterClient`, so it is not the gate's flip. Driver-only nodes are unaffected (they register neither `IMeshPskProvider` nor `ActorAuditWriter`) | `WithOtOpcUaControlPlaneSingletons`'s first `WithActors` starter eagerly resolves `IMeshCommandDialer` + `IMeshPskProvider` to pass into `CentralCommunicationActor.Props`. That graph is dialer → `IMeshPskProvider` → `ISecretResolver` → (`AddZbSecrets`) → `IAuditWriter` → `ActorAuditWriter`, whose ctor calls `registry.Get<AuditWriterActorKey>()` — and `WithSingleton<AuditWriterActorKey>` is registered ~60 lines later in the same builder. Same class as scadaproj#1 and mesh Phase 6's `IClusterRoleInfo` crash: only visible with the DI graph built in a real host, which is why the container-activation test (no `AddZbSecrets` in its collection) and `TwoNodeClusterHarness` (no provider at all) both miss it. Aggravating: the failed process does not exit, so `docker ps` reports `Up restarts=0` while the node serves no HTTP | **OPEN.** Fix + red-first pin belong to 1A (the gate made no product change). Suggested shape: hand the actor lazy accessors, mirroring the `Func<>` the same block already uses for the coordinator proxy. Pin must build the graph in starter order (a Host.IntegrationTests boot of an admin-role host), not merely activate the dialer |
| 3 | 1A.2 | **Fan parked the actor mailbox** (self-caught by the implementer): dialing from the receive ran `TryGetPsk`'s cold-cache 3 s block pre-await (cold after EVERY refresh due to `InvalidateAll`), stalling deployment acks behind the fan | async methods run synchronously to the first await | `Task.Run` hop (token None — a canceled scheduling would read as a DB failure via PipeTo); pinned red-first by `A_slow_dial_does_not_park_the_actors_mailbox` |
## Carry-forward register
@@ -112,13 +139,26 @@ Checks a gate could not drive, named + carried to the next gate (proportionate-g
| Item | Deferred from | Proven instead by | Closed in |
|---|---|---|---|
| — | — | *(none yet)* | |
| Hang-style central failover drill (`docker pause`, not `docker stop` — only a hang discriminates the per-endpoint-deadline fix) | 1B.2 review | unit fact `Each_endpoint_gets_a_FRESH_deadline` | 1B.3 gate (MUST include) |
| I-4 measurement: ack burst within ~3 s after a contact-refresh tick (cold PSK cache on the inbound auth path) | 1A.2 review | bounded 3 s store read + `Lazy` stampede suppression | 1B.3 gate (observe) |
| `TwoNodeClusterHarness` logs the I-1 null-provider Error at every spawn (harness registers no `IMeshPskProvider`) — EXPECTED, not a rig fault | 1A.2 re-review | production pairing is `hasAdmin`-unconditional in one block | note for 1B.3/Phase 2 readers |
| Register item 3 (DI role-scoping pin: admin resolves provider+dialer, driver-only does NOT) — container-activation test exists, role-scoping pin outstanding | 1A.2 | `It_activates_from_the_container_despite_the_optional_client_factory_parameter`; starter-order half now covered by `ControlPlaneStarterOrderTests` | Phase 2 (or 1B.3) — note defect 4 shows the missing pin is not merely role-scoping but **starter-order**: the container-activation test resolves from a collection without `AddZbSecrets` |
| `Resolve`-failure Error wording says "continuing without it" (reads permanent — the memoize heals it at first use) and the line fires on real boots (2/3 in the gate) — reword or whitelist it before the Phase 2 soak error-grep, else a healthy central scores as failing the soak | 1A.3 re-run | heal proven live: singleton moved onto the race-losing node, fan clean, no dialer-missing error | Phase 2 (soak grep design) |
| A repaired `GrpcPort` takes effect only at the next contact refresh (≤60 s): a deploy 5 s post-restore was still skipped and honestly TimedOut — by design, operator-visible; document if it surprises in Phase 2 | 1A.3 re-run | check 7 evidence | Phase 2 (observe) |
| `RevisionHash` moved on byte-identical config after a direct SQL `UPDATE` bumped `Script.RowVersion` (hash covers a column content-identity does not); determinism itself holds (3 no-change dispatches → identical hash) | 1A.3 re-run | incidental, out of program scope | note only |
| 1A checks 1/2/7 (deploy fan over gRPC, mixed-transport seal, NULL-`GrpcPort` skip) and the central half of 4 | 1A.3 gate run 1 | — | **CLOSED in 1A.3 run 2** — all driven live and PASS |
| Check 8 is vacuous for telemetry + config-serve — central never booted, so neither plane carried a call | 1A.3 gate run 1 | — | **CLOSED in 1A.3 run 2** — both planes carried authenticated traffic (28 KB artifact fetches from `:4055`, telemetry streams open on all six) with 0 denials |
| **`DriverStatusPanel.razor` is referenced by no page or component** — the Reconnect/Restart buttons are unreachable in the running AdminUI (pre-existing on `master`, not caused by this program; re-verified in run 2: 0 `<DriverStatusPanel` usages, no Restart/Reconnect control on any rendered page, no such HTTP endpoint) | 1A.3 gate | Check 3 **closed by composition**: run 1's live node-side leg for both commands + run 2's check 4 driving the identical `AdminOperationsActor.Dispatch` → `DispatchOverGrpc` origin path from the AdminUI | Out of program scope — file separately if the control is meant to exist |
| `Failed to resolve` Error text reads as permanent though first use retries, and will trip a soak error-grep on a healthy node | 1A.3 gate run 2 (defect 5) | Self-heal proven live | Phase 2 |
## Issues filed by this program
| Issue | What | Status |
|---|---|---|
| [lmxopcua#542](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/542) | `FleetStatusBroadcaster` derives "fleet" from central's own gossip ring — silently central-pair-blind since mesh Phase 6; pre-existing, transport-independent, out of this program's scope | FILED 2026-08-06 |
| [lmxopcua#544](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/544) | `ConfigPublishCoordinatorTimeoutTests.DeadlineElapsed_for_current_deployment_marks_TimedOut` is racy on its own terms (150 ms wall-clock deadline; row inserted after spawn) — flaked 2/~11 full runs once 1A.2 grew the suite; green at base. Fix the test (inject clock / raise deadline) | FILED 2026-08-06 (1A.3) |
| [lmxopcua#545](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/545) | Mesh command/ack planes are h2c-only — the per-cluster PSK crosses the network in cleartext on a write surface (precedent: every sibling gRPC plane; trusted-site-link assumption). Decide: accepted risk, or a `UseTls`/scheme option | FILED 2026-08-06 (1A.3) |
| [lmxopcua#546](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/546) | AdminUI `DriverStatusPanel` (the only Restart/Reconnect control) is referenced by no page — the commands have no reachable UI; pre-existing, found by the 1A gate | FILED 2026-08-06 |
| [lmxopcua#543](https://gitea.dohertylan.com/dohertj2/lmxopcua/issues/543) | The three pre-existing auth interceptors (`TelemetryStream`/`ConfigServe`/`LocalDbSync`) treat a whitespace-only key as SET (`IsNullOrEmpty`), so a whitespace `Telemetry:ApiKey` etc. opens the endpoint to a whitespace bearer; the two new mesh interceptors use `IsNullOrWhiteSpace`. Backport is a behaviour change to shipped endpoints — own issue, out of this program's scope | FILED 2026-08-06 |
## Known constraints carried from the research (do not rediscover)
docs/plans/2026-08-06-clusterclient-to-grpc.md
+1 -1
View File
@@ -248,7 +248,7 @@ user-owned.**
- Test: `tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/Grpc/NodeCommandGrpcServiceTests.cs`
Follow the alias convention (`using GeneratedServiceBase = …NodeCommandService.NodeCommandServiceBase;`,
class suffixed `GrpcService`). `Execute`: decode via `MeshTransportDtoMapper.FromProto`, Tell
class suffixed `GrpcService`). `Execute`: decode via `MeshTransportDtoMapper.FromEnvelope`, Tell
the registered `NodeCommunicationActor` (resolve via `NodeCommunicationActorKey` registry,
`Runtime/ServiceCollectionExtensions.cs:583`), return `ExecuteAck`. Actor not yet registered
(boot window) ⇒ `RpcException(Unavailable)` — the readiness convention. Decode failure ⇒
docs/plans/2026-08-06-clusterclient-to-grpc.md.tasks.json
+227 -16
View File
@@ -3,22 +3,233 @@
"executionModel": "opus-subagents; sequential within a worktree, 1A and 1B concurrent in separate worktrees (git worktree add ../OtOpcUa-1B feat/grpc-central-ack); 1A merges first, 1B rebases; stop at each phase DoD (PR/merge user-owned)",
"worktrees": {},
"tasks": [
{"id": 0, "subject": "Task 0.1: mesh_transport.proto + contract lock", "phase": "0", "branch": "feat/grpc-phase0-contract-auth", "classification": "standard", "parallelizableWith": ["0.3 (batch, same agent)"], "status": "completed", "commit": "c0ccfa27", "review": "spec ✅ + code ✅"},
{"id": 1, "subject": "Task 0.2: DTO mapper + round-trip goldens", "phase": "0", "branch": "feat/grpc-phase0-contract-auth", "classification": "high-risk", "parallelizableWith": [], "status": "completed", "blockedBy": [0], "commit": "545413a8", "review": "spec ✅ + code ✅ (serial)"},
{"id": 2, "subject": "Task 0.3: MeshTransportOptions + validators + SplitTopology relaxation", "phase": "0", "branch": "feat/grpc-phase0-contract-auth", "classification": "standard", "parallelizableWith": ["0.1 (batch, same agent)"], "status": "completed", "commit": "aa21f61f", "review": "spec ✅ + code ✅"},
{"id": 3, "subject": "Task 0.4: node-side MeshCommandAuthInterceptor", "phase": "0", "branch": "feat/grpc-phase0-contract-auth", "classification": "high-risk", "parallelizableWith": ["0.5 (batch, same agent)"], "status": "completed", "blockedBy": [2], "commit": "fbda015c + fixes 24e55d33/a4631410", "review": "spec ✅ + code ✅ (opus, fix loop closed)"},
{"id": 4, "subject": "Task 0.5: IMeshPskProvider + CentralCommandAuthInterceptor", "phase": "0", "branch": "feat/grpc-phase0-contract-auth", "classification": "high-risk", "parallelizableWith": ["0.4 (batch, same agent)"], "status": "completed", "blockedBy": [2], "commit": "68a416e4 + fixes 24e55d33/a4631410", "review": "spec ✅ + code ✅ (opus, fix loop closed); 4 items gated on Phase 1A — see tracking §Phase 1A security register"},
{"id": 5, "subject": "Task 0.6: rig config + FleetStatus issue + docs stub + Phase 0 gate", "phase": "0", "branch": "feat/grpc-phase0-contract-auth", "classification": "small", "parallelizableWith": [], "status": "completed", "blockedBy": [0, 1, 2, 3, 4], "commit": "23555ed0", "review": "code ✅", "gate": "PASS 4/4; issues #542 #543 filed"},
{"id": 6, "subject": "Task 1A.1: NodeCommandGrpcService on the telemetry listener", "phase": "1A", "branch": "feat/grpc-node-command", "classification": "standard", "parallelizableWith": ["1B.1 (other worktree)"], "status": "pending", "blockedBy": [5]},
{"id": 7, "subject": "Task 1A.2: MeshGrpcCommandDialer + third branch in CentralCommunicationActor", "phase": "1A", "branch": "feat/grpc-node-command", "classification": "high-risk", "parallelizableWith": ["1B.2 (other worktree)"], "status": "pending", "blockedBy": [6]},
{"id": 8, "subject": "Task 1A.3: 1A live gate (central pair flipped, rig-only, reverted)", "phase": "1A", "branch": "feat/grpc-node-command", "classification": "small", "parallelizableWith": [], "status": "pending", "blockedBy": [7]},
{"id": 9, "subject": "Task 1B.1: CentralCommandGrpcService + admin validator rule", "phase": "1B", "branch": "feat/grpc-central-ack", "classification": "standard", "parallelizableWith": ["1A.1 (other worktree)"], "status": "pending", "blockedBy": [5]},
{"id": 10, "subject": "Task 1B.2: GrpcApplyAckSender + node-side wiring", "phase": "1B", "branch": "feat/grpc-central-ack", "classification": "high-risk", "parallelizableWith": ["1A.2 (other worktree)"], "status": "pending", "blockedBy": [9]},
{"id": 11, "subject": "Task 1B.3: 1B live gate (site-a flipped, rig-only, reverted; rebase on 1A first)", "phase": "1B", "branch": "feat/grpc-central-ack", "classification": "small", "parallelizableWith": [], "status": "pending", "blockedBy": [8, 10]},
{"id": 12, "subject": "Phase 2: baseline reproduction + full-rig cutover + soak + kill drills", "phase": "2", "branch": "(rig commit)", "classification": "high-risk", "parallelizableWith": [], "status": "pending", "blockedBy": [8, 11]},
{"id": 13, "subject": "Task 3.1: delete ClusterClient (code + HOCON + config + tests, ordered)", "phase": "3", "branch": "feat/grpc-phase3-deletion", "classification": "high-risk", "parallelizableWith": [], "status": "pending", "blockedBy": [12]},
{"id": 14, "subject": "Task 3.2: docs sweep (Configuration/Redundancy/Telemetry/CLAUDE.md + scadaproj index)", "phase": "3", "branch": "feat/grpc-phase3-deletion", "classification": "small", "parallelizableWith": [], "status": "pending", "blockedBy": [13]},
{"id": 15, "subject": "Phase 4: terminal live gate on post-deletion image (8 checks, provenance-guarded)", "phase": "4", "branch": "(master)", "classification": "high-risk", "parallelizableWith": [], "status": "pending", "blockedBy": [14]}
{
"id": 0,
"subject": "Task 0.1: mesh_transport.proto + contract lock",
"phase": "0",
"branch": "feat/grpc-phase0-contract-auth",
"classification": "standard",
"parallelizableWith": [
"0.3 (batch, same agent)"
],
"status": "completed",
"commit": "c0ccfa27",
"review": "spec ✅ + code ✅"
},
{
"id": 1,
"subject": "Task 0.2: DTO mapper + round-trip goldens",
"phase": "0",
"branch": "feat/grpc-phase0-contract-auth",
"classification": "high-risk",
"parallelizableWith": [],
"status": "completed",
"blockedBy": [
0
],
"commit": "545413a8",
"review": "spec ✅ + code ✅ (serial)"
},
{
"id": 2,
"subject": "Task 0.3: MeshTransportOptions + validators + SplitTopology relaxation",
"phase": "0",
"branch": "feat/grpc-phase0-contract-auth",
"classification": "standard",
"parallelizableWith": [
"0.1 (batch, same agent)"
],
"status": "completed",
"commit": "aa21f61f",
"review": "spec ✅ + code ✅"
},
{
"id": 3,
"subject": "Task 0.4: node-side MeshCommandAuthInterceptor",
"phase": "0",
"branch": "feat/grpc-phase0-contract-auth",
"classification": "high-risk",
"parallelizableWith": [
"0.5 (batch, same agent)"
],
"status": "completed",
"blockedBy": [
2
],
"commit": "fbda015c + fixes 24e55d33/a4631410",
"review": "spec ✅ + code ✅ (opus, fix loop closed)"
},
{
"id": 4,
"subject": "Task 0.5: IMeshPskProvider + CentralCommandAuthInterceptor",
"phase": "0",
"branch": "feat/grpc-phase0-contract-auth",
"classification": "high-risk",
"parallelizableWith": [
"0.4 (batch, same agent)"
],
"status": "completed",
"blockedBy": [
2
],
"commit": "68a416e4 + fixes 24e55d33/a4631410",
"review": "spec ✅ + code ✅ (opus, fix loop closed); 4 items gated on Phase 1A — see tracking §Phase 1A security register"
},
{
"id": 5,
"subject": "Task 0.6: rig config + FleetStatus issue + docs stub + Phase 0 gate",
"phase": "0",
"branch": "feat/grpc-phase0-contract-auth",
"classification": "small",
"parallelizableWith": [],
"status": "completed",
"blockedBy": [
0,
1,
2,
3,
4
],
"commit": "23555ed0",
"review": "code ✅",
"gate": "PASS 4/4; issues #542 #543 filed"
},
{
"id": 6,
"subject": "Task 1A.1: NodeCommandGrpcService on the telemetry listener",
"phase": "1A",
"branch": "feat/grpc-node-command",
"classification": "standard",
"parallelizableWith": [
"1B.1 (other worktree)"
],
"status": "completed",
"blockedBy": [
5
],
"commit": "64774652",
"review": "spec ✅ + code ✅"
},
{
"id": 7,
"subject": "Task 1A.2: MeshGrpcCommandDialer + third branch in CentralCommunicationActor (+ security register items 1-2)",
"phase": "1A",
"branch": "feat/grpc-node-command",
"classification": "high-risk",
"parallelizableWith": [
"1B.2 (other worktree)"
],
"status": "completed",
"blockedBy": [
6
],
"commit": "14d21496+6a862011 + fixes 48dc2120/86633f14",
"review": "spec ✅ + code ✅ (opus; I-2 fixed via gate-follows-endpoint decision)"
},
{
"id": 8,
"subject": "Task 1A.3: 1A live gate (central pair flipped, rig-only, reverted)",
"phase": "1A",
"branch": "feat/grpc-node-command",
"classification": "small",
"parallelizableWith": [],
"status": "completed",
"blockedBy": [
7
],
"commit": "gate records 410c2e29/e3a0ddbc; boot fixes c0475f2d/e54883a8",
"gate": "run 1 FAIL (defect 4) -> run 2 PASS 8/8; issues #544 #545 #546 filed"
},
{
"id": 9,
"subject": "Task 1B.1: CentralCommandGrpcService + admin validator rule",
"phase": "1B",
"branch": "feat/grpc-central-ack",
"classification": "standard",
"parallelizableWith": [
"1A.1 (other worktree)"
],
"status": "pending",
"blockedBy": [
5
]
},
{
"id": 10,
"subject": "Task 1B.2: GrpcApplyAckSender + node-side wiring",
"phase": "1B",
"branch": "feat/grpc-central-ack",
"classification": "high-risk",
"parallelizableWith": [
"1A.2 (other worktree)"
],
"status": "pending",
"blockedBy": [
9
]
},
{
"id": 11,
"subject": "Task 1B.3: 1B live gate (site-a flipped, rig-only, reverted; rebase on 1A first)",
"phase": "1B",
"branch": "feat/grpc-central-ack",
"classification": "small",
"parallelizableWith": [],
"status": "pending",
"blockedBy": [
8,
10
]
},
{
"id": 12,
"subject": "Phase 2: baseline reproduction + full-rig cutover + soak + kill drills",
"phase": "2",
"branch": "(rig commit)",
"classification": "high-risk",
"parallelizableWith": [],
"status": "pending",
"blockedBy": [
8,
11
]
},
{
"id": 13,
"subject": "Task 3.1: delete ClusterClient (code + HOCON + config + tests, ordered)",
"phase": "3",
"branch": "feat/grpc-phase3-deletion",
"classification": "high-risk",
"parallelizableWith": [],
"status": "pending",
"blockedBy": [
12
]
},
{
"id": 14,
"subject": "Task 3.2: docs sweep (Configuration/Redundancy/Telemetry/CLAUDE.md + scadaproj index)",
"phase": "3",
"branch": "feat/grpc-phase3-deletion",
"classification": "small",
"parallelizableWith": [],
"status": "pending",
"blockedBy": [
13
]
},
{
"id": 15,
"subject": "Phase 4: terminal live gate on post-deletion image (8 checks, provenance-guarded)",
"phase": "4",
"branch": "(master)",
"classification": "high-risk",
"parallelizableWith": [],
"status": "pending",
"blockedBy": [
14
]
}
],
"lastUpdated": "2026-08-06"
}
src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Communication/CentralCommunicationActor.cs
+571 -15
View File
@@ -3,17 +3,24 @@ using Akka.Actor;
using Akka.Cluster.Tools.Client;
using Akka.Cluster.Tools.PublishSubscribe;
using Akka.Event;
using Grpc.Core;
using Microsoft.EntityFrameworkCore;
using ZB.MOM.WW.OtOpcUa.Cluster;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Deploy;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
using ZB.MOM.WW.OtOpcUa.Commons.Protos.MeshTransport.V1;
using ZB.MOM.WW.OtOpcUa.Commons.Types;
using ZB.MOM.WW.OtOpcUa.Configuration;
// Grpc.Core (imported for RpcException on the dial path) declares a Status of its own. This file's
// bare Status is Akka's — the Status.Failure a faulted piped task arrives as; the gRPC one is only
// ever reached through an RpcException instance.
using Status = Akka.Actor.Status;
namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
/// <summary>
/// Central-side end of the central↔node command boundary (per-cluster mesh Phase 2). Routes
/// every central→node command out — over DistributedPubSub or ClusterClient, per
/// every central→node command out — over DistributedPubSub, ClusterClient or gRPC, per
/// <see cref="MeshTransportOptions.Mode"/> — and forwards inbound <see cref="ApplyAck"/>s to the
/// deploy coordinator singleton.
/// </summary>
@@ -31,6 +38,23 @@ namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
/// and fans <c>SendToAll</c> across all of them. See <see cref="RebuildClient"/> — this is the
/// single most important thing to understand before changing this actor.
/// </para>
/// <para>
/// <b>The gRPC branch fans over the same recipient set, node by node</b>
/// (ClusterClient→gRPC Phase 1A). <c>SendToAll</c> reaches every node registered on a mesh's
/// receptionist, and the contact load already selects exactly the enabled, non-maintenance
/// rows those nodes come from; the gRPC branch dials that same row set directly, one call per
/// node, so the recipient set is identical by construction rather than by coincidence. It is
/// deliberately NOT narrower: the node side has no <c>ClusterId</c> or node filter at all —
/// scoping happens later, inside the artifact — so a node left out here silently keeps its
/// old configuration while the deployment seals green.
/// </para>
/// <para>
/// <b>Fire-and-forget, on every transport.</b> The gRPC branch pipes per-node results back to
/// itself for logging ONLY: a failure names the node and the status and is dropped. No retry,
/// no queue, no ordering guarantee across nodes. That is not an omission — it is the
/// ClusterClient semantic preserved exactly, and the backstops are unchanged (the deploy
/// coordinator's apply deadline, and a node's boot-time replay).
/// </para>
/// </remarks>
public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
{
@@ -40,8 +64,30 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
private readonly MeshTransportOptions _options;
private readonly IMeshClusterClientFactory _clientFactory;
private readonly Func<IActorRef?> _coordinator;
// Resolved in PreStart, never at construction. See the Props doc: the DI graph behind these two
// reaches actor-registry entries that do not exist yet while the Akka builder's starters are
// still running, and touching it there is what stopped every central node from booting.
private readonly Func<IMeshCommandDialer?>? _dialerFactory;
private readonly Func<IMeshPskProvider?>? _pskProviderFactory;
private IMeshCommandDialer? _dialer;
private IMeshPskProvider? _pskProvider;
// Memoize-on-first-use, NOT PreStart-only: PreStart runs on a dispatcher thread concurrently
// with the builder's remaining starters, so its resolve can lose a millisecond-wide race
// against AuditWriterActorKey's registration (starter #4) and Resolve() would swallow that
// into a PERMANENT silent null — the known-cluster gate never published, fail-open. These
// properties retry on first use instead; the first RefreshContacts arrives as a message,
// long after every starter finished, so a lost boot race self-heals there. Actor-thread
// only, so ??= needs no synchronisation; both are container singletons, so a repeat resolve
// is a dictionary hit.
private IMeshPskProvider? PskProvider => _pskProvider ??= Resolve(_pskProviderFactory, nameof(IMeshPskProvider));
private IMeshCommandDialer? Dialer => _dialer ??= Resolve(_dialerFactory, nameof(IMeshCommandDialer));
private readonly ILoggingAdapter _log = Context.GetLogger();
private readonly bool _useClusterClient;
private readonly bool _useGrpc;
// One client per application Cluster, keyed by ClusterId. A ClusterClient reaches only the mesh
// whose receptionist it dialled, and after Phase 6 every cluster is its own mesh — so central must
@@ -49,6 +95,15 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
private readonly Dictionary<string, (IActorRef Client, ImmutableHashSet<string> Contacts)> _clients =
new();
// Node ids already warned about for a null GrpcPort. A persistently portless row would otherwise
// produce one warning per refresh forever; same warn-once shape as TelemetryNodeSource. Actor
// thread only.
private readonly HashSet<string> _warnedPortlessNodes = new(StringComparer.Ordinal);
// The gRPC dispatch set, replaced wholesale on every contact load. Actor thread only; the fan
// captures the list reference, so a refresh mid-fan cannot mutate a dispatch already under way.
private IReadOnlyList<MeshNodeTarget> _targets = [];
private CancellationTokenSource _lifecycle = new();
/// <summary>Gets the timer scheduler driving the periodic contact refresh.</summary>
@@ -63,46 +118,104 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
/// an <see cref="IActorRef"/> so this actor never depends on registration order at wiring
/// time — it resolves on first use and caches.
/// </param>
/// <param name="dialer">
/// Resolves the thing that places one command call on one node, under
/// <see cref="MeshTransportOptions.ModeGrpc"/>. Optional so the two older transports need
/// nothing new wired; a <see langword="null"/> dialer under <c>Grpc</c> is a mis-wiring and
/// is reported loudly rather than silently dropping every command.
/// </param>
/// <param name="pskProvider">
/// Resolves central's per-cluster key lookup, told on every contact refresh which clusters
/// exist (the known-cluster gate) and to drop its cache (so a rotated key takes effect within
/// one refresh). Optional for the same reason as <paramref name="dialer"/>.
/// </param>
/// <returns>The props.</returns>
/// <remarks>
/// <para>
/// <b>Both dependencies arrive as <see cref="Func{TResult}"/>, and that is not a style
/// choice.</b> Passing instances meant the caller resolved them while the Akka builder's
/// starters were still running, and the DI graph behind them reaches back into the actor
/// registry: dialer → <c>IMeshPskProvider</c> → <c>ISecretResolver</c> → <c>AddZbSecrets</c>
/// → <c>IAuditWriter</c> → <c>ActorAuditWriter</c>, whose constructor calls
/// <c>registry.Get&lt;AuditWriterActorKey&gt;()</c> — a singleton registered dozens of
/// lines LATER in the very same builder. Every central node failed to boot, in every
/// transport mode, on a <c>MissingActorRegistryEntryException</c>. Deferring to
/// <see cref="PreStart"/> moves the first resolve to a point where the host is fully
/// built, and matches the <see cref="Func{TResult}"/> the coordinator proxy already uses
/// one argument earlier for the same class of reason.
/// </para>
/// <para>
/// This is the third instance of the same failure mode in this family — eager resolution
/// inside an <c>AddAkka</c> configurator, previously <c>scadaproj#1</c> (a DI cycle that
/// deadlocked) and mesh Phase 6's <c>IClusterRoleInfo</c> boot <c>StackOverflow</c>. The
/// rule it keeps re-teaching: inside an Akka builder, resolve nothing you can defer.
/// </para>
/// </remarks>
public static Props Props(
IDbContextFactory<OtOpcUaConfigDbContext> dbFactory,
MeshTransportOptions options,
IMeshClusterClientFactory clientFactory,
Func<IActorRef?> coordinator) =>
Func<IActorRef?> coordinator,
Func<IMeshCommandDialer?>? dialer = null,
Func<IMeshPskProvider?>? pskProvider = null) =>
Akka.Actor.Props.Create(() =>
new CentralCommunicationActor(dbFactory, options, clientFactory, coordinator));
new CentralCommunicationActor(
dbFactory, options, clientFactory, coordinator, dialer, pskProvider));
/// <summary>Initializes a new instance of the <see cref="CentralCommunicationActor"/> class.</summary>
/// <param name="dbFactory">Factory for the config database.</param>
/// <param name="options">The bound mesh-transport options.</param>
/// <param name="clientFactory">Creates the ClusterClient.</param>
/// <param name="coordinator">Lazily resolves the deploy-coordinator singleton proxy.</param>
/// <param name="dialer">Resolves the gRPC command dialer; invoked in <see cref="PreStart"/>.</param>
/// <param name="pskProvider">Resolves the PSK provider; invoked in <see cref="PreStart"/>.</param>
public CentralCommunicationActor(
IDbContextFactory<OtOpcUaConfigDbContext> dbFactory,
MeshTransportOptions options,
IMeshClusterClientFactory clientFactory,
Func<IActorRef?> coordinator)
Func<IActorRef?> coordinator,
Func<IMeshCommandDialer?>? dialer = null,
Func<IMeshPskProvider?>? pskProvider = null)
{
_dbFactory = dbFactory;
_options = options;
_clientFactory = clientFactory;
_coordinator = coordinator;
_dialerFactory = dialer;
_pskProviderFactory = pskProvider;
_useClusterClient = string.Equals(
options.Mode, MeshTransportOptions.ModeClusterClient, StringComparison.OrdinalIgnoreCase);
_useGrpc = string.Equals(
options.Mode, MeshTransportOptions.ModeGrpc, StringComparison.OrdinalIgnoreCase);
Receive<MeshCommand>(HandleMeshCommand);
Receive<ApplyAck>(HandleApplyAck);
Receive<RefreshContacts>(_ => LoadContactsFromDb());
Receive<ContactsLoaded>(HandleContactsLoaded);
Receive<CommandDispatched>(HandleCommandDispatched);
// A faulted LoadContactsFromDb task is piped here as a Status.Failure. Without this handler
// the failure is an unhandled message (debug level only) and the refresh fails silently —
// an operator cannot distinguish "no nodes configured" from "the database is down", and the
// contact set silently freezes at whatever it last held.
Receive<Status.Failure>(f => _log.Warning(
f.Cause,
"Failed to load ClusterNode contact points; the mesh ClusterClient contact set was NOT "
+ "refreshed and may be stale or empty. Commands may be dropped until the next refresh"));
//
// The freeze extends to the known-cluster gate, deliberately: a failed load must neither
// widen the gate nor narrow it, and the previous set is the last thing central actually knew.
// The second line exists because that freeze is INVISIBLE otherwise, and it is fail-OPEN
// before the first successful load — a central that has never reached its ConfigDb refuses
// no cluster at all.
Receive<Status.Failure>(f =>
{
_log.Warning(
f.Cause,
"Failed to load ClusterNode contact points; the mesh contact set was NOT refreshed "
+ "and may be stale or empty. Commands may be dropped until the next refresh");
_log.Warning(
"The known-cluster gate was NOT refreshed either: it still holds the set from the "
+ "last successful load, and if none has succeeded yet (boot, or central SQL Server "
+ "unreachable since start) it is INACTIVE — every cluster id reaches the secret "
+ "store, which is the pre-authentication cost the gate exists to remove");
});
Receive<SubscribeAck>(_ => { /* DPS subscribe confirmation */ });
}
@@ -110,14 +223,51 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
/// <inheritdoc />
protected override void PreStart()
{
if (!_useClusterClient)
// First point at which the host is fully built — see the Props remark. Re-runs on every
// actor restart, which re-reads the container rather than caching a stale reference; both
// dependencies are container singletons, so a restart gets the same instances back.
_dialer = Resolve(_dialerFactory, nameof(IMeshCommandDialer));
_pskProvider = Resolve(_pskProviderFactory, nameof(IMeshPskProvider));
if (!_useClusterClient && !_useGrpc)
{
_log.Info(
"Mesh transport is {Mode}: commands are published on DistributedPubSub and no "
+ "ClusterClient is created", _options.Mode);
return;
+ "ClusterClient is created. The contact refresh still runs — it is what keeps the "
+ "known-cluster gate current, and that gate belongs to the inbound ack endpoint, "
+ "which is mapped by port and does not care about this mode", _options.Mode);
}
if (_useGrpc && _dialer is null)
{
// Not a fatal throw: the ack path (HandleApplyAck) is transport-independent and still
// works, so taking the actor down would turn a command-plane mis-wiring into an ack
// outage as well. Every dispatch logs its own error, so this cannot go unnoticed.
_log.Error(
"Mesh transport is Grpc but no command dialer was supplied to this actor; EVERY "
+ "central→node command will be dropped. Check the ControlPlane registration wires "
+ "IMeshCommandDialer");
}
if (_pskProvider is null)
{
// Mode-independent, like the gate itself. A silent absence here is the worst of the
// three outcomes in this method: nothing malfunctions, so nothing is ever investigated,
// while the gate stays permanently inactive and a rotated key is never picked up.
_log.Error(
"No IMeshPskProvider was supplied to this actor; the known-cluster gate will NEVER "
+ "be published and cached per-cluster keys will NEVER be dropped. Inbound mesh acks "
+ "stay exposed to the pre-authentication secret-store cost the gate removes, and a "
+ "rotated key needs a central restart. Check the ControlPlane registration");
}
// Started in EVERY mode. The refresh is not transport state: it is central's ClusterNode
// inventory, and the known-cluster gate it feeds protects an endpoint mapped by PORT, not by
// MeshTransport:Mode. Gating the timer on the mode would mean a dark-switch rollback to
// ClusterClient or Dps left that endpoint live with the gate permanently inactive — silently
// reopening both the timing oracle and the pre-auth store/audit flood. Under Dps this is a
// ConfigDb query every ContactRefreshSeconds that did not run before; central always has a
// ConfigDb, and one indexed read a minute is the cheaper half of the trade.
Timers.StartPeriodicTimer(
RefreshTimerKey,
new RefreshContacts(),
@@ -125,18 +275,64 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
TimeSpan.FromSeconds(_options.ContactRefreshSeconds));
}
/// <summary>
/// Invokes a dependency factory, converting a resolution failure into a logged absence.
/// </summary>
/// <typeparam name="T">The dependency's type.</typeparam>
/// <param name="factory">The factory, or <see langword="null"/> when nothing was supplied.</param>
/// <param name="name">The dependency's name, for the log.</param>
/// <returns>The resolved dependency, or <see langword="null"/>.</returns>
/// <remarks>
/// Catching is the point. A throw out of <see cref="PreStart"/> is an actor failure, and the
/// supervisor's answer to that is to restart — which calls <see cref="PreStart"/> again, and
/// a broken DI graph is not a transient condition. That is an unbounded restart loop on the
/// node's command boundary, taking the transport-independent ack path down with it. Logging
/// and continuing leaves the ack path serving and hands the two null guards below a genuine
/// absence to report, which is the same outcome as never having been wired.
/// </remarks>
private T? Resolve<T>(Func<T?>? factory, string name)
where T : class
{
if (factory is null)
return null;
try
{
return factory();
}
catch (Exception ex)
{
_log.Error(
ex,
"Failed to resolve {Dependency} for the mesh command boundary; continuing without "
+ "it. The guards below report what that costs", name);
return null;
}
}
/// <inheritdoc />
protected override void PostStop()
{
foreach (var (client, _) in _clients.Values) Context.Stop(client);
_clients.Clear();
// Cancelling the lifecycle token also cancels every gRPC dial in flight. The dialer itself is
// deliberately NOT disposed here: it is a container-owned singleton shared with anything else
// that may dial, and an actor restart must not tear its channel pool down.
_lifecycle.Cancel();
_lifecycle.Dispose();
}
private void HandleMeshCommand(MeshCommand cmd)
{
// Checked before the DPS arm: _useClusterClient is false under Grpc too, so the order is
// what keeps a Grpc-mode command off DistributedPubSub.
if (_useGrpc)
{
DispatchOverGrpc(cmd);
return;
}
if (!_useClusterClient)
{
DistributedPubSub.Get(Context.System).Mediator.Tell(new Publish(cmd.Topic, cmd.Message));
@@ -168,6 +364,199 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
client.Tell(new ClusterClient.SendToAll(MeshPaths.NodeCommunication, cmd.Message));
}
/// <summary>
/// Encodes one command once and fans it to every target node, one unary gRPC call each.
/// </summary>
/// <param name="cmd">The command envelope carrying the domain record.</param>
/// <remarks>
/// <para>
/// <b>Encoded once, before the fan.</b> Not per node — partly because the wire form is
/// identical for every recipient, but mostly because <c>AlarmCommand</c> has no
/// correlation member of its own and the envelope mints one: minting inside the loop
/// would give one operator action N unrelated correlation ids and make it untraceable.
/// </para>
/// <para>
/// <b>Nothing captured from the actor.</b> Each dial is a static call taking only locals,
/// so no per-node continuation can touch actor state off the actor's thread. Results come
/// back as ordinary messages via <c>PipeTo</c> and are handled on the actor thread like
/// anything else.
/// </para>
/// <para>
/// <b>Every dial starts on the thread pool, not here.</b> An <c>async</c> method runs
/// synchronously up to its first <c>await</c>, and the dialer's pre-await work includes
/// <c>IMeshPskProvider.TryGetPsk</c>, which blocks on the secret store (bounded, but
/// seconds) the first time a cluster is looked up — and again after every refresh, since
/// the refresh drops the cache to pick up rotations. Calling straight into it would park
/// this actor's mailbox, and deployment acks queue behind that mailbox.
/// </para>
/// </remarks>
private void DispatchOverGrpc(MeshCommand cmd)
{
var commandType = cmd.Message.GetType().Name;
var dialer = Dialer;
if (dialer is null)
{
_log.Error(
"Mesh transport is Grpc but no command dialer is wired — dropping {MessageType}. The "
+ "command is not buffered and will not be retried",
commandType);
return;
}
// Captured before the fan: a contact refresh between here and the last dial must not change
// the recipient set of a dispatch already under way.
var targets = _targets;
if (targets.Count == 0)
{
// App-level drop, exactly as the ClusterClient arm does with no client. The command is
// NOT queued and will NOT be retried: a deploy fails at the coordinator's apply deadline
// naming the silent nodes, which is a better operator signal than a command arriving
// minutes late against a config the DB already recorded as failed.
_log.Warning(
"No mesh gRPC targets — dropping {MessageType}. The last contact refresh produced no "
+ "enabled, non-maintenance ClusterNode row carrying a GrpcPort in any cluster; the "
+ "command is not buffered and will not be retried",
commandType);
return;
}
CommandEnvelope envelope;
try
{
envelope = MeshTransportDtoMapper.ToEnvelope(cmd.Message, new CorrelationId(Guid.NewGuid()));
}
catch (Exception ex)
{
// The mapper refuses a payload the far side could not decode. Dropping here is the same
// outcome the old transport gave for such a message (nothing on the node matched it), but
// loud instead of silent.
_log.Error(
ex,
"Cannot encode {MessageType} for the gRPC command plane; the command is dropped and "
+ "was NOT delivered to any node",
commandType);
return;
}
CancellationToken ct;
try
{
ct = _lifecycle.Token;
}
catch (ObjectDisposedException)
{
return; // Stopping.
}
var self = Self;
var correlationId = envelope.CorrelationId;
foreach (var target in targets)
{
var node = target;
// CancellationToken.None on Task.Run, deliberately: cancelling the SCHEDULING would make
// the task complete as canceled, which PipeTo converts into a Status.Failure — the one
// message this actor already reads as "the contact load failed". The lifecycle token is
// passed into the dial instead, where it cancels the call and comes back as an ordinary
// per-node outcome.
Task.Run(
() => DispatchOneAsync(dialer, node, envelope, commandType, correlationId, ct),
CancellationToken.None)
.PipeTo(self);
}
}
/// <summary>
/// Dials one node and turns every possible outcome into a <see cref="CommandDispatched"/>.
/// </summary>
/// <param name="dialer">The dialer seam.</param>
/// <param name="target">The node to dial.</param>
/// <param name="envelope">The already-encoded command.</param>
/// <param name="commandType">CLR type name of the command, for the log.</param>
/// <param name="correlationId">Transport correlation, for the log.</param>
/// <param name="ct">The actor's lifecycle token.</param>
/// <returns>The outcome, always — this method does not fault.</returns>
/// <remarks>
/// Catching everything is what keeps the result on ONE channel. A faulting task piped to an
/// actor arrives as <c>Status.Failure</c>, which this actor already handles as "the contact
/// load failed" — so an unhandled dial fault would log a database-outage warning about a node
/// that simply did not answer.
/// </remarks>
private static async Task<CommandDispatched> DispatchOneAsync(
IMeshCommandDialer dialer,
MeshNodeTarget target,
CommandEnvelope envelope,
string commandType,
string correlationId,
CancellationToken ct)
{
try
{
await dialer.DispatchAsync(target.Endpoint, envelope, target.ClusterId, ct)
.ConfigureAwait(false);
return new CommandDispatched(target.NodeId, commandType, correlationId, null, null);
}
catch (RpcException ex)
{
return new CommandDispatched(
target.NodeId,
commandType,
correlationId,
ex.StatusCode.ToString(),
Truncate(ex.Status.Detail));
}
catch (OperationCanceledException)
{
// Named rather than swept into the generic arm: this is the ordinary shape of a node that
// did not answer in time, and "Unknown / TaskCanceledException" would send an operator
// looking for a bug instead of at DispatchTimeoutSeconds.
return new CommandDispatched(
target.NodeId,
commandType,
correlationId,
"Cancelled",
"the dial did not complete within MeshTransport:DispatchTimeoutSeconds, or central "
+ "is shutting down");
}
catch (Exception ex)
{
// Type name only, never the message: this path also covers whatever the dialer's channel
// construction can throw, and a message is the one place a misconfigured endpoint string
// could carry something an operator did not mean to put in a log.
return new CommandDispatched(
target.NodeId, commandType, correlationId, "Unknown", ex.GetType().Name);
}
}
/// <summary>
/// Bounds a status detail before it reaches the log. The string comes from the far side, and
/// the far side is a node — trusted, but not a reason to let it choose how many lines of
/// central's log one failed command occupies.
/// </summary>
/// <param name="detail">The received status detail.</param>
/// <returns>The detail, at most 200 characters.</returns>
private static string Truncate(string? detail)
=> detail is null || detail.Length <= 200 ? detail ?? "" : detail[..200] + "…";
private void HandleCommandDispatched(CommandDispatched result)
{
if (result.FailureStatus is null)
{
_log.Debug(
"Mesh: {CommandType} (correlationId={CorrelationId}) accepted by {NodeId} over gRPC",
result.CommandType, result.CorrelationId, result.NodeId);
return;
}
_log.Warning(
"Mesh: {CommandType} (correlationId={CorrelationId}) was NOT delivered to {NodeId} "
+ "({Status}: {Detail}). It is dropped and will NOT be retried — the deployment apply "
+ "deadline and the node's boot-time replay are the backstops",
result.CommandType, result.CorrelationId, result.NodeId,
result.FailureStatus, result.FailureDetail);
}
private void HandleApplyAck(ApplyAck ack)
{
var coordinator = _coordinator();
@@ -207,10 +596,18 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
Task.Run(async () =>
{
await using var db = await dbFactory.CreateDbContextAsync(ct).ConfigureAwait(false);
// UNFILTERED, with the eligibility flags projected and applied per row below. The two
// outputs of this read want different row sets: the contact/dispatch sets are the
// enabled, non-maintenance nodes central talks TO, while the known-cluster gate answers
// "is this a cluster central has ever heard of" and must include a cluster whose every
// node is disabled or in maintenance — such a cluster can still send an ack, and denying
// it would make maintenance mode an authentication switch.
var rows = await db.ClusterNodes
.AsNoTracking()
.Where(n => n.Enabled && !n.MaintenanceMode)
.Select(n => new { n.NodeId, n.ClusterId, n.Host, n.AkkaPort })
.Select(n => new
{
n.NodeId, n.ClusterId, n.Host, n.AkkaPort, n.GrpcPort, n.Enabled, n.MaintenanceMode,
})
.ToListAsync(ct)
.ConfigureAwait(false);
@@ -219,8 +616,34 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
// client.
var byCluster = new Dictionary<string, List<string>>();
var malformed = new List<string>();
// The gRPC dispatch set, built from the SAME row set in the SAME pass. Sharing the read is
// what makes "the gRPC fan reaches exactly the ClusterClient recipient set" structural
// rather than a claim two separate queries would have to keep agreeing on.
var targets = new List<MeshNodeTarget>(rows.Count);
var portless = new List<string>();
// Every cluster with ANY ClusterNode row — the known-cluster gate's membership set, and
// deliberately a superset of the fan set. Independent of Enabled, MaintenanceMode and
// GrpcPort alike: none of those describe whether the cluster exists.
var clusterIds = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
foreach (var row in rows)
{
clusterIds.Add(row.ClusterId);
// The eligibility filter that used to be the query's WHERE clause. Everything below
// this line — receptionist contacts and gRPC targets — sees exactly the rows it saw
// before.
if (!row.Enabled || row.MaintenanceMode)
continue;
if (row.GrpcPort is { } grpcPort)
targets.Add(new MeshNodeTarget(
row.NodeId, row.ClusterId, $"http://{row.Host}:{grpcPort}"));
else
portless.Add(row.NodeId);
var address = $"akka.tcp://{systemName}@{row.Host}:{row.AkkaPort}/system/receptionist";
// Parse up front, per row, inside the loop's own guard. A single malformed row must
@@ -241,12 +664,24 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
var contactsByCluster = byCluster.ToDictionary(
kv => kv.Key, kv => (IReadOnlyList<string>)kv.Value);
return new ContactsLoaded(contactsByCluster, malformed);
return new ContactsLoaded(
contactsByCluster, malformed, targets, portless, clusterIds.ToList());
}, ct).PipeTo(self);
}
private void HandleContactsLoaded(ContactsLoaded msg)
{
// Order is load-bearing and is pinned by test: the dispatch set is adopted BEFORE the gate is
// published, so anything that observes the gate can rely on the targets already being current.
if (_useGrpc)
AdoptGrpcTargets(msg);
// EVERY mode. See PreStart — the gate protects an endpoint mapped by port, not by mode.
PublishKnownClusters(msg);
if (!_useClusterClient)
return; // Dps or Grpc: no ClusterClient to rebuild.
foreach (var bad in msg.Malformed)
{
_log.Warning(
@@ -305,6 +740,78 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
}
}
/// <summary>
/// Adopts a contact load's gRPC dispatch set. <see cref="MeshTransportOptions.ModeGrpc"/> only
/// — the other two transports have no use for it.
/// </summary>
/// <param name="msg">The load result.</param>
/// <remarks>
/// The malformed-receptionist-address warnings are deliberately NOT logged here: nothing on
/// this transport dials a receptionist, and reporting an unusable Akka address as a
/// command-plane problem would send an operator after the wrong column. A host that is
/// unusable as a gRPC endpoint surfaces instead as a named per-node dial failure on every
/// dispatch, which is a truer signal.
/// </remarks>
private void AdoptGrpcTargets(ContactsLoaded msg)
{
// Rebuilt against this refresh's skip list, not merely added to. A node that gained a port
// and later lost it again must warn a second time — the operator's mental model is "the last
// refresh told me", not "you were told once, in a log that has since rotated away".
_warnedPortlessNodes.IntersectWith(msg.PortlessNodes);
foreach (var nodeId in msg.PortlessNodes)
{
// Warn-once per node id per continuous skip run: a persistently portless row is
// re-skipped every refresh, and that is the repetition TelemetryNodeSource suppresses.
if (_warnedPortlessNodes.Add(nodeId))
{
_log.Warning(
"ClusterNode {NodeId} has no GrpcPort; it exposes no command endpoint and is "
+ "skipped in this dispatch set (further skips of this node are silent). Commands "
+ "will not reach it until the row carries a port", nodeId);
}
}
_targets = msg.Targets;
if (_targets.Count == 0)
{
_log.Warning(
"No enabled, non-maintenance ClusterNode row carries a GrpcPort in any cluster; there "
+ "are no gRPC command targets and every central→node command will be dropped until a "
+ "refresh finds one");
}
}
/// <summary>
/// Hands the PSK provider this refresh's cluster inventory and drops its key cache. Runs in
/// <b>every</b> transport mode.
/// </summary>
/// <param name="msg">The load result.</param>
/// <remarks>
/// <para>
/// <b>Not gated on <see cref="MeshTransportOptions.Mode"/>, on purpose.</b> The gate feeds
/// <c>CentralCommandAuthInterceptor</c>, which is registered on every admin node, and the
/// ack service it protects is mapped by listener port — neither consults the mode. Gating
/// this on <c>Grpc</c> would mean a dark-switch rollback to <c>ClusterClient</c> or
/// <c>Dps</c> left that endpoint serving with a permanently inactive gate: the timing
/// oracle and the pre-authentication store/audit flood would both come back, silently,
/// as a side effect of a rollback nobody would connect to authentication.
/// </para>
/// <para>
/// <b><c>InvalidateAll</c>, not per-cluster <c>Invalidate</c>.</b> The requirement is that
/// a rotated key takes effect within one refresh, and only a whole-cache drop delivers
/// that: a key can rotate without the cluster's membership changing at all, so
/// invalidating only departed clusters would serve the pre-rotation key forever. The cost
/// is one store read per cluster per refresh — the cache holds a single entry per cluster.
/// </para>
/// </remarks>
private void PublishKnownClusters(ContactsLoaded msg)
{
PskProvider?.SetKnownClusters(msg.ClusterIds);
PskProvider?.InvalidateAll();
}
/// <summary>Stops a cluster's existing client and creates a replacement for its new contact set.</summary>
/// <remarks>
/// <para>
@@ -371,7 +878,56 @@ public sealed class CentralCommunicationActor : ReceiveActor, IWithTimers
/// <c>Cluster</c>'s id. One <c>ClusterClient</c> is built per key.
/// </param>
/// <param name="Malformed">Rows that did not parse, described for the log.</param>
/// <param name="Targets">
/// The gRPC dispatch set: one entry per enabled, non-maintenance row that carries a
/// <c>GrpcPort</c>. Built from the same rows as <paramref name="ContactsByCluster"/>.
/// </param>
/// <param name="PortlessNodes">
/// Node ids skipped because their row has no <c>GrpcPort</c>. Carried rather than logged in
/// the load task so the warn-once bookkeeping stays on the actor's thread.
/// </param>
/// <param name="ClusterIds">
/// Every application <c>Cluster</c> with <b>any</b> <c>ClusterNode</c> row — the PSK
/// provider's known-cluster gate membership, and deliberately a superset of
/// <paramref name="Targets"/>: a cluster whose nodes are all disabled or in maintenance can
/// still send an ack, and refusing it would make maintenance mode an authentication switch.
/// </param>
public sealed record ContactsLoaded(
IReadOnlyDictionary<string, IReadOnlyList<string>> ContactsByCluster,
IReadOnlyList<string> Malformed);
IReadOnlyList<string> Malformed,
IReadOnlyList<MeshNodeTarget> Targets,
IReadOnlyList<string> PortlessNodes,
IReadOnlyList<string> ClusterIds);
/// <summary>One node the gRPC command plane can dial.</summary>
/// <param name="NodeId">The <c>ClusterNode</c> row's id — what a failure log names.</param>
/// <param name="ClusterId">
/// The node's application <c>Cluster</c>. Selects the pre-shared key presented to it: a node
/// dialled with another cluster's id is graded against the wrong key and refuses the command.
/// </param>
/// <param name="Endpoint">The h2c base address, <c>http://{Host}:{GrpcPort}</c>.</param>
public sealed record MeshNodeTarget(string NodeId, string ClusterId, string Endpoint);
/// <summary>
/// The outcome of one node's dial, piped back to the actor <b>for logging only</b>. Nothing
/// retries, counts or waits on these — a deployment's completion is judged by the acks nodes
/// send back, never by whether central's outbound call succeeded.
/// </summary>
/// <param name="NodeId">The node dialled.</param>
/// <param name="CommandType">CLR type name of the command.</param>
/// <param name="CorrelationId">The envelope's transport correlation.</param>
/// <param name="FailureStatus">
/// <see langword="null"/> on success; otherwise the gRPC status code's name, or
/// <c>Unknown</c> for a non-gRPC fault.
/// </param>
/// <param name="FailureDetail">
/// The gRPC status detail, or the exception's type name for a non-gRPC fault. Never a key,
/// and never a raw exception message.
/// </param>
public sealed record CommandDispatched(
string NodeId,
string CommandType,
string CorrelationId,
string? FailureStatus,
string? FailureDetail);
}
src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Communication/IMeshPskProvider.cs
+114 -4
View File
@@ -1,5 +1,7 @@
using System.Collections.Concurrent;
using System.Diagnostics.CodeAnalysis;
using System.Security.Cryptography;
using System.Text;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.OtOpcUa.Cluster;
@@ -21,10 +23,21 @@ namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
/// proceed unauthenticated; a missing key must cost delivery, not authentication.
/// </para>
/// <para>
/// The <c>Invalidate</c> hooks exist because central's cluster set is DB-dynamic. They are
/// deliberately unconsumed at Phase 0 — Phase 1A wires them to the contact refresh, which is
/// the point at which central learns a cluster's membership (and so possibly its key) has
/// changed.
/// The <c>Invalidate</c> hooks exist because central's cluster set is DB-dynamic. Phase 1A
/// wires them to <c>CentralCommunicationActor</c>'s contact refresh, which is the point at
/// which central learns a cluster's membership (and so possibly its key) has changed — so a
/// rotated key is picked up within one <c>ContactRefreshSeconds</c> without a restart.
/// </para>
/// <para>
/// <b>The known-cluster gate is a real security control, not an optimization</b> (Phase 1A
/// security register item 1). <see cref="SetKnownClusters"/> publishes the cluster ids
/// central just read from <c>ClusterNode</c>; <see cref="TryGetPsk"/> consults that set
/// first and refuses anything outside it in O(1), touching neither the config map nor the
/// secret store. Without it, an unauthenticated caller can pick an arbitrary
/// <c>x-otopcua-cluster</c> value and make central perform a secret-store read plus an audit
/// write per probe — and, because negatives are deliberately never cached, the cost
/// difference between a known and an unknown cluster is measurable from the wire, which
/// defeats the uniform denial text on the central interceptor.
/// </para>
/// <para>
/// <b><c>Invalidate</c> is not a revocation barrier.</b> It drops the cached entry; it does
@@ -43,6 +56,50 @@ public interface IMeshPskProvider
/// <returns><c>true</c> when a usable key was found; otherwise <c>false</c>.</returns>
bool TryGetPsk(string clusterId, [NotNullWhen(true)] out string? psk);
/// <summary>
/// Publishes the set of application <c>Cluster</c> ids central currently knows about.
/// <see cref="TryGetPsk"/> refuses anything outside it before consulting any source.
/// </summary>
/// <param name="clusterIds">
/// The known cluster ids, matched case-insensitively. <b>An empty collection deactivates the
/// gate</b> rather than refusing everything: central publishes this from its DB contact
/// refresh, and "the DB told me nothing" must not become a fleet-wide authentication outage.
/// Never having called this leaves the gate inactive for the same reason — the central
/// interceptor is live from the first request, before the first refresh has run.
/// </param>
/// <remarks>
/// Implementations must take a copy: the caller owns its collection and may reuse it. The
/// swap must also be atomic with respect to concurrent <see cref="TryGetPsk"/> callers,
/// which run on arbitrary gRPC threads.
/// </remarks>
void SetKnownClusters(IReadOnlyCollection<string> clusterIds);
/// <summary>
/// Verifies a presented bearer token against one cluster's key, in constant time with
/// respect to the key's content.
/// </summary>
/// <param name="clusterId">The application <c>Cluster</c> the caller claims to be.</param>
/// <param name="presented">The bearer token the caller supplied.</param>
/// <returns><c>true</c> only when a key exists for that cluster and the token equals it.</returns>
/// <remarks>
/// <para>
/// <b>This is the only comparison on the verify path</b> (Phase 1A security register
/// item 2). Handing the raw key back so a caller can compare it is how <c>==</c>
/// eventually appears in an interceptor and compiles; keeping the comparison behind this
/// member means the verify side never holds key material at all. <see cref="TryGetPsk"/>
/// survives for the <i>dial</i> side, which has to send the key rather than compare it.
/// </para>
/// <para>
/// Declared without a default implementation on purpose: a security member a new
/// implementer can inherit silently is a member a new implementer never considers.
/// </para>
/// <para>
/// Length differences remain observable through timing; that is inherent to comparing
/// byte arrays of different lengths and is not sensitive. Content is not.
/// </para>
/// </remarks>
bool Matches(string clusterId, string presented);
/// <summary>Drops any cached key for one cluster, so the next lookup re-reads the store.</summary>
/// <param name="clusterId">The application <c>Cluster</c> id, matched case-insensitively.</param>
void Invalidate(string clusterId);
@@ -87,6 +144,18 @@ public sealed class MeshPskProvider : IMeshPskProvider
private readonly ConcurrentDictionary<string, Lazy<string?>> _cache =
new(StringComparer.OrdinalIgnoreCase);
/// <summary>
/// The known-cluster gate's membership set, or <see langword="null"/> when the gate is
/// inactive (never published, or published empty — see <see cref="SetKnownClusters"/>).
/// </summary>
/// <remarks>
/// Swapped wholesale, never mutated in place, so a reader either sees the previous set or the
/// next one and never a half-built one. <c>volatile</c> is what makes the publish visible to
/// the gRPC threads doing the reading; no lock is needed because there is exactly one writer
/// (the contact-refresh handler, on the actor's own thread).
/// </remarks>
private volatile IReadOnlySet<string>? _knownClusters;
/// <summary>Creates the provider.</summary>
/// <param name="options">Mesh transport options; <c>ClusterPsks</c> is the override map.</param>
/// <param name="secretResolver">The shared secret store read seam.</param>
@@ -118,6 +187,19 @@ public sealed class MeshPskProvider : IMeshPskProvider
return false;
}
// The known-cluster gate, ahead of BOTH sources. Deliberately silent: the caller that
// rejected the call already logs the denial, and a second line here would be a log flood
// driven by exactly the unauthenticated traffic this gate exists to make cheap.
//
// Ahead of the config map too, not just the store. That has a visible consequence — a
// MeshTransport:ClusterPsks entry for a cluster with no enabled, non-maintenance ClusterNode
// row stops resolving once central publishes its set — and it is the right one: central
// would neither dial such a cluster nor accept an ack from it, so serving a key for it only
// widens what an unauthenticated caller can probe.
var known = _knownClusters;
if (known is not null && !known.Contains(clusterId))
return false;
// Config map first, on every call. Blank counts as absent: an empty
// MeshTransport__ClusterPsks__<id> is an unset environment variable, not an operator
// deciding this cluster has an empty key.
@@ -168,6 +250,34 @@ public sealed class MeshPskProvider : IMeshPskProvider
return true;
}
/// <inheritdoc />
public bool Matches(string clusterId, string presented)
{
// Blank is refused on its own terms rather than by relying on the fact that a blank key
// never survives resolution: this is an authentication gate, and "both sides happened to be
// empty" must never be a pass.
if (string.IsNullOrWhiteSpace(presented))
return false;
if (!TryGetPsk(clusterId, out var expected))
return false;
return CryptographicOperations.FixedTimeEquals(
Encoding.UTF8.GetBytes(presented), Encoding.UTF8.GetBytes(expected));
}
/// <inheritdoc />
public void SetKnownClusters(IReadOnlyCollection<string> clusterIds)
{
ArgumentNullException.ThrowIfNull(clusterIds);
// Copied, so a caller reusing its list cannot widen the gate after the fact; empty collapses
// to null so the gate has exactly one "inactive" representation to test for.
_knownClusters = clusterIds.Count == 0
? null
: clusterIds.ToHashSet(StringComparer.OrdinalIgnoreCase);
}
/// <inheritdoc />
public void Invalidate(string clusterId)
{
src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/Communication/MeshGrpcCommandDialer.cs
+200
View File
@@ -0,0 +1,200 @@
using System.Collections.Concurrent;
using Grpc.Core;
using Grpc.Net.Client;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.OtOpcUa.Cluster;
using ZB.MOM.WW.OtOpcUa.Commons.Protos.MeshTransport.V1;
namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
/// <summary>
/// Places one central→node command call on one node's <c>NodeCommandService</c>
/// (ClusterClient→gRPC migration Phase 1A). The seam <c>CentralCommunicationActor</c> fans
/// across its target set, kept as an interface so the actor's tests never open a socket.
/// </summary>
/// <remarks>
/// One dial, one node, no retry and no fan-out of its own — the fan and its per-node failure
/// logging belong to the actor, which is the component that knows what the recipient set is.
/// </remarks>
public interface IMeshCommandDialer
{
/// <summary>Delivers one command envelope to one node.</summary>
/// <param name="nodeEndpoint">The node's gRPC base address, e.g. <c>http://site-a-1:4056</c>.</param>
/// <param name="envelope">The wire envelope, already mapped from the domain record.</param>
/// <param name="clusterId">
/// The application <c>Cluster</c> the <i>target node</i> belongs to. Selects which pre-shared
/// key is presented, and is echoed to the node in <c>x-otopcua-cluster</c>.
/// </param>
/// <param name="ct">Cancels the dial; a per-call deadline is applied on top of it.</param>
/// <returns>A task completing when the node has accepted the command.</returns>
/// <exception cref="RpcException">
/// The node rejected the call, was unreachable, or the deadline expired — and also when no
/// key could be established for <paramref name="clusterId"/>, which is refused locally with
/// <see cref="StatusCode.Unauthenticated"/> rather than dialled unauthenticated.
/// </exception>
Task DispatchAsync(string nodeEndpoint, CommandEnvelope envelope, string clusterId, CancellationToken ct);
}
/// <inheritdoc />
/// <remarks>
/// <para>
/// <b>Fail-closed on a missing key, always.</b> There is no arm on which a command leaves
/// central without the target cluster's bearer. A node's <c>MeshCommandAuthInterceptor</c>
/// would reject an unauthenticated call anyway, but "the far side would have refused it" is
/// not a reason to make the call — an unauthenticated dial is a credential-free command
/// attempt against a node's control surface, and it should not exist even as a rejected one.
/// The refusal is reported as an <see cref="RpcException"/> so the actor's per-node failure
/// logging covers it on the same channel as a remote rejection; the operator response is
/// identical either way (seed the cluster's key), and a second channel would need a second
/// handler that could drift.
/// </para>
/// <para>
/// <b>One channel per endpoint, cached.</b> Same shape as
/// <c>GrpcDeploymentArtifactFetcher</c>: a <see cref="GrpcChannel"/> owns a connection pool
/// and creating one per call leaks sockets. The cache is keyed by the endpoint string
/// ordinally — endpoints come from <c>ClusterNode.Host</c> + <c>GrpcPort</c>, so two spellings
/// of one host would merely cost a second channel, never a wrong destination.
/// </para>
/// <para>
/// <b>The deadline is per call, not per fan.</b> A dispatch dials every node in the fleet;
/// an unbounded call to one unreachable node would hold a task and a socket indefinitely
/// while the deploy path's own 2-minute apply deadline ticked down.
/// <see cref="MeshTransportOptions.DispatchTimeoutSeconds"/> bounds it, linked to the
/// caller's token so actor shutdown still cancels everything in flight.
/// </para>
/// <para>
/// Registered as a singleton and disposed by the container. A dial started after disposal
/// throws, so no new work repopulates the cache — but a dial already past that check can
/// still add a channel after the cache was cleared. That leak is bounded to the dials in
/// flight at shutdown, which is the only time disposal happens, so it is accepted rather
/// than serialized against.
/// </para>
/// </remarks>
public sealed class MeshGrpcCommandDialer : IMeshCommandDialer, IDisposable
{
/// <summary>
/// Substituted for a non-positive <see cref="MeshTransportOptions.DispatchTimeoutSeconds"/>.
/// Deliberately the option's own default rather than a second value to reconcile.
/// </summary>
private const int FallbackDispatchTimeoutSeconds = 10;
/// <summary>
/// Header naming the cluster the presented bearer belongs to. Central's own inbound gate
/// reads the same header on the ack path; sending it outbound keeps one convention and tells
/// a node which key central graded itself against.
/// </summary>
private const string ClusterHeader = "x-otopcua-cluster";
private readonly IMeshPskProvider _pskProvider;
private readonly ILogger<MeshGrpcCommandDialer> _log;
private readonly int _dispatchTimeoutSeconds;
private readonly Func<string, NodeCommandService.NodeCommandServiceClient> _clientFactory;
private readonly ConcurrentDictionary<string, GrpcChannel> _channels = new(StringComparer.Ordinal);
private volatile bool _disposed;
/// <summary>Initializes a new instance of the <see cref="MeshGrpcCommandDialer"/> class.</summary>
/// <param name="pskProvider">Supplies the bearer for the target node's cluster.</param>
/// <param name="options">Mesh transport options; supplies the per-call deadline.</param>
/// <param name="log">Logger for the construction-time deadline check.</param>
/// <param name="clientFactory">
/// Seam for tests: builds the generated client for an endpoint. Defaults to a real h2c
/// <see cref="GrpcChannel"/>, one cached per endpoint.
/// </param>
/// <remarks>
/// A non-positive <c>DispatchTimeoutSeconds</c> is clamped, not honoured. Zero would mean an
/// unbounded call — the exact failure the deadline exists to prevent — and the validator that
/// rejects it is scoped to driver-role nodes, so the one deployment shape that reads this
/// value unvalidated is a pure-admin central. Clamping loudly is better than a fleet-wide fan
/// that can never time out.
/// </remarks>
public MeshGrpcCommandDialer(
IMeshPskProvider pskProvider,
IOptions<MeshTransportOptions> options,
ILogger<MeshGrpcCommandDialer> log,
Func<string, NodeCommandService.NodeCommandServiceClient>? clientFactory = null)
{
ArgumentNullException.ThrowIfNull(pskProvider);
ArgumentNullException.ThrowIfNull(options);
ArgumentNullException.ThrowIfNull(log);
_pskProvider = pskProvider;
_log = log;
var configured = options.Value.DispatchTimeoutSeconds;
if (configured <= 0)
{
_log.LogWarning(
"MeshTransport:DispatchTimeoutSeconds is {Configured}, which would leave every mesh "
+ "command dial unbounded; using {Fallback}s instead. Set a positive value — the "
+ "boot validator for this key covers driver-role nodes only, so a pure-admin central "
+ "reaches this line rather than failing at startup.",
configured, FallbackDispatchTimeoutSeconds);
}
_dispatchTimeoutSeconds = configured > 0 ? configured : FallbackDispatchTimeoutSeconds;
_clientFactory = clientFactory ?? DefaultClientFactory;
}
/// <summary>
/// Gets the number of endpoints holding a cached <see cref="GrpcChannel"/>. Exposed so the
/// cache — the whole defence against a socket per call across a fleet-wide fan — is
/// assertable rather than merely intended.
/// </summary>
public int CachedChannelCount => _channels.Count;
/// <inheritdoc />
public async Task DispatchAsync(
string nodeEndpoint, CommandEnvelope envelope, string clusterId, CancellationToken ct)
{
ArgumentException.ThrowIfNullOrWhiteSpace(nodeEndpoint);
ArgumentNullException.ThrowIfNull(envelope);
ArgumentException.ThrowIfNullOrWhiteSpace(clusterId);
ObjectDisposedException.ThrowIf(_disposed, this);
if (!_pskProvider.TryGetPsk(clusterId, out var psk))
{
// The cluster id is operator-authored configuration, not a secret, so naming it is what
// makes this actionable. The key never appears here or anywhere else in this class.
// Kept short on purpose: the actor truncates a status detail at 200 characters because a
// REMOTE node authors it, and this locally-authored one has to survive that cap with its
// remediation intact. It measures ~160 characters with a six-character cluster id, so an
// id would have to exceed ~40 characters to lose the trailing rig hint.
throw new RpcException(new Status(
StatusCode.Unauthenticated,
$"No mesh pre-shared key for cluster '{clusterId}'; dial refused locally. Seed secret "
+ "OT-MESH-PSK-<clusterId>, or set MeshTransport:ClusterPsks:<clusterId> on a rig."));
}
var headers = new Metadata
{
{ "authorization", $"Bearer {psk}" },
{ ClusterHeader, clusterId },
};
using var deadlineCts = CancellationTokenSource.CreateLinkedTokenSource(ct);
deadlineCts.CancelAfter(TimeSpan.FromSeconds(_dispatchTimeoutSeconds));
var client = _clientFactory(nodeEndpoint);
using var call = client.ExecuteAsync(envelope, headers, cancellationToken: deadlineCts.Token);
await call.ResponseAsync.ConfigureAwait(false);
}
/// <inheritdoc />
public void Dispose()
{
_disposed = true;
foreach (var channel in _channels.Values)
channel.Dispose();
_channels.Clear();
}
private NodeCommandService.NodeCommandServiceClient DefaultClientFactory(string endpoint)
{
// h2c: GrpcChannel over an http:// address negotiates prior-knowledge HTTP/2.
var channel = _channels.GetOrAdd(endpoint, GrpcChannel.ForAddress);
return new NodeCommandService.NodeCommandServiceClient(channel);
}
}
src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/ServiceCollectionExtensions.cs
+18 -1
View File
@@ -66,6 +66,21 @@ public static class ServiceCollectionExtensions
var dbFactory = resolver.GetService<IDbContextFactory<OtOpcUaConfigDbContext>>();
var options = resolver.GetService<IOptions<MeshTransportOptions>>().Value;
// Captured, NOT resolved. Resolving the mesh dialer or PSK provider here made every
// central node fail to boot in every transport mode: their graph is dialer →
// IMeshPskProvider → ISecretResolver → AddZbSecrets → IAuditWriter → ActorAuditWriter,
// whose constructor calls registry.Get<AuditWriterActorKey>() — and WithSingleton
// <AuditWriterActorKey> is registered further down THIS builder, so the entry does not
// exist while this starter runs. The actor invokes these in PreStart instead, by which
// point the host is built. Same shape, same reason, as the coordinator Func below.
//
// Through IServiceProvider rather than the Akka resolver, deliberately: these two are
// OPTIONAL (Grpc mode only, and admin-role only within that), and GetService's
// null-for-absent contract is the thing being relied on. The Akka resolver's
// GetService<T> makes no such promise, and this same block runs in test harnesses whose
// service collections register neither.
var serviceProvider = resolver.GetService<IServiceProvider>();
var actor = system.ActorOf(
CentralCommunicationActor.Props(
dbFactory,
@@ -75,7 +90,9 @@ public static class ServiceCollectionExtensions
// which Akka.Hosting materialises the singleton proxy relative to this block.
() => registry.TryGet<ConfigPublishCoordinatorKey>(out var coordinator)
? coordinator
: null),
: null,
() => serviceProvider.GetService<IMeshCommandDialer>(),
() => serviceProvider.GetService<IMeshPskProvider>()),
MeshPaths.CentralCommunicationName);
// Registered unconditionally, in both transport modes. Under Dps nothing dials it, and
src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/CentralCommandAuthInterceptor.cs
+36 -31
View File
@@ -1,5 +1,3 @@
using System.Security.Cryptography;
using System.Text;
using Grpc.Core;
using Grpc.Core.Interceptors;
using Microsoft.Extensions.Logging;
@@ -38,25 +36,31 @@ namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
/// key this design exists to avoid.
/// </para>
/// <para>
/// Comparison is <see cref="CryptographicOperations.FixedTimeEquals"/> over UTF-8 bytes, so a
/// wrong key cannot be recovered byte-by-byte from response timing. Length differences are
/// unavoidably observable and are not sensitive.
/// <b>This interceptor never handles a key.</b> Verification is
/// <see cref="IMeshPskProvider.Matches"/>, which owns the constant-time comparison, so a
/// wrong key cannot be recovered byte-by-byte from response timing and no future edit here
/// can replace that comparison with <c>==</c> and still compile (Phase 1A security register
/// item 2). Length differences are unavoidably observable and are not sensitive.
/// </para>
/// <para>
/// <b>All three denials return the same status text</b>, unlike the sibling interceptors,
/// which distinguish "no key configured" from "authentication failed". Here that distinction
/// would be a <i>content</i> oracle: the caller supplies the cluster id, so a reply that said
/// "no key for that cluster" would let an unauthenticated caller read central's cluster set
/// straight off the wire. The operator-facing detail goes to the log instead.
/// <b>All denials return the same status text</b>, unlike the sibling interceptors, which
/// distinguish "no key configured" from "authentication failed". Here that distinction would
/// be a <i>content</i> oracle: the caller supplies the cluster id, so a reply that said "no
/// key for that cluster" would let an unauthenticated caller read central's cluster set
/// straight off the wire. The operator-facing detail goes to the log instead — and for the
/// same reason the log no longer separates "no key for that cluster" from "wrong key": the
/// provider answers both as one <c>false</c>, and collapsing them costs an operator a hint
/// (restated in the one denial message) rather than costing a diagnosis.
/// </para>
/// <para>
/// <b>That is not yet full enumeration resistance, and should not be read as such.</b> A
/// timing oracle survives it: an unknown cluster misses the provider's cache and pays a store
/// read on every probe, while a known one is answered from memory after the first. Closing
/// that needs the lookup to be refused before it reaches the store for any cluster central
/// does not know — which is the Phase 1A known-cluster-set gate, where central learns its
/// cluster set from the DB. Until then, uniform text buys content-level indistinguishability
/// only.
/// <b>The timing oracle that survived Phase 0 is closed by the known-cluster gate.</b> An
/// unknown cluster used to miss the provider's cache and pay a secret-store read on every
/// probe, while a known one was answered from memory after the first — a stable, remotely
/// measurable distinguisher, and a pre-auth store/audit flood besides.
/// <c>CentralCommunicationActor</c> now publishes central's cluster set to the provider on
/// every contact refresh, and a cluster outside it is refused in O(1) with no store touch,
/// so the two cases cost the same. What remains is a known cluster's first-ever lookup,
/// which is one read per cluster per process and not attacker-triggerable.
/// </para>
/// <para>
/// <b>Exactly one public constructor.</b> <c>Grpc.AspNetCore</c> silently stops invoking an
@@ -151,13 +155,15 @@ public sealed class CentralCommandAuthInterceptor : Interceptor
"Central command authentication failed."));
}
if (!_pskProvider.TryGetPsk(clusterId, out var expected))
// Checked here rather than deferred to Matches so the log can still separate "the caller
// presented nothing" — an operator misconfiguration, and the one case the provider cannot
// see — from "the caller presented something that did not verify".
var presented = GrpcRequestHeaders.BearerToken(context.RequestHeaders);
if (presented is null)
{
_logger.LogWarning(
"Rejected a central command call to {Method} claiming cluster {ClusterId}: no "
+ "pre-shared key could be established for it. Seed the secret "
+ "OT-MESH-PSK-<clusterId> on this node, or set "
+ "MeshTransport:ClusterPsks:<clusterId> on a rig.",
"Rejected a central command call to {Method} claiming cluster {ClusterId}: no bearer "
+ "token presented.",
context.Method,
clusterId);
throw new RpcException(new Status(
@@ -165,21 +171,20 @@ public sealed class CentralCommandAuthInterceptor : Interceptor
"Central command authentication failed."));
}
var presented = GrpcRequestHeaders.BearerToken(context.RequestHeaders);
if (presented is null || !FixedTimeEquals(presented, expected))
if (!_pskProvider.Matches(clusterId, presented))
{
_logger.LogWarning(
"Rejected a central command call to {Method} claiming cluster {ClusterId}: {Reason}.",
"Rejected a central command call to {Method} claiming cluster {ClusterId}: the "
+ "presented bearer token did not match that cluster's pre-shared key, no key could "
+ "be established for it, or it is not a cluster this node knows. If the cluster is "
+ "legitimate, seed the secret OT-MESH-PSK-<clusterId> on this node, or set "
+ "MeshTransport:ClusterPsks:<clusterId> on a rig, and check it has an enabled, "
+ "non-maintenance ClusterNode row.",
context.Method,
clusterId,
presented is null ? "no bearer token presented" : "bearer token did not match");
clusterId);
throw new RpcException(new Status(
StatusCode.PermissionDenied,
"Central command authentication failed."));
}
}
private static bool FixedTimeEquals(string presented, string expected)
=> CryptographicOperations.FixedTimeEquals(
Encoding.UTF8.GetBytes(presented), Encoding.UTF8.GetBytes(expected));
}
src/Server/ZB.MOM.WW.OtOpcUa.Host/Grpc/NodeCommandGrpcService.cs
+110
View File
@@ -0,0 +1,110 @@
using Akka.Actor;
using Akka.Hosting;
using Grpc.Core;
using Microsoft.Extensions.Logging;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
using ZB.MOM.WW.OtOpcUa.Commons.Protos.MeshTransport.V1;
using ZB.MOM.WW.OtOpcUa.Runtime;
// The generated service container is itself named NodeCommandService; alias its nested server base so
// this impl can keep the natural name without colliding with the generated type.
using GeneratedServiceBase =
ZB.MOM.WW.OtOpcUa.Commons.Protos.MeshTransport.V1.NodeCommandService.NodeCommandServiceBase;
// Akka.Actor (imported for IActorRef.Tell) declares a Status of its own; only the gRPC one is used here.
using Status = Grpc.Core.Status;
namespace ZB.MOM.WW.OtOpcUa.Host.Grpc;
/// <summary>
/// Node-side command ingress (ClusterClient→gRPC migration Phase 1). Central dials a driver node
/// and calls <see cref="Execute"/>; the envelope is decoded back into the same CLR record the
/// ClusterClient receptionist used to deliver and handed to this node's
/// <c>NodeCommunicationActor</c>, which re-emits it on the node's <c>EventStream</c> exactly as
/// before. Nothing downstream of that actor can tell which transport carried the command.
/// </summary>
/// <remarks>
/// <para>
/// <b>One-way, verbatim from the transport it replaces.</b> <c>ExecuteAck</c> means "decoded and
/// handed to the comm actor" — not applied, and not acknowledged by any EventStream subscriber.
/// The <c>Tell</c> is deliberately not an <c>Ask</c>: no migrated command has a reply, and
/// waiting for one would invent a semantic central does not have.
/// </para>
/// <para>
/// <b>Decode runs BEFORE the registry lookup, and the two failures are graded differently.</b>
/// An undecodable envelope is <see cref="StatusCode.InvalidArgument"/> — permanent, and it will
/// never decode on a retry. A call arriving before <c>AddOtOpcUaRuntime</c> has spawned and
/// registered the comm actor is <see cref="StatusCode.Unavailable"/> — the house readiness
/// convention, transient, and correct to retry. Checking readiness first would report a
/// permanently malformed command as a readiness problem and invite an unbounded retry loop.
/// </para>
/// <para>
/// <b>Every log line carries the envelope's own <c>correlation_id</c>, read as a raw string.</b>
/// Three of the four commands also carry a domain correlation, but <c>AlarmCommand</c> has no
/// CLR member for one, so the envelope field is the only id every command shares — and it is
/// still readable when the inner fields are what failed to parse, which is precisely when an
/// operator needs it.
/// </para>
/// <para>
/// Inbound auth is <c>MeshCommandAuthInterceptor</c>, scoped to this service's method prefix and
/// registered at the <c>AddGrpc</c> level; it fail-closes with no <c>MeshTransport:GrpcPsk</c>
/// configured. Mapping this service is what makes that gate live.
/// </para>
/// </remarks>
public sealed class NodeCommandGrpcService : GeneratedServiceBase
{
private readonly ActorRegistry _registry;
private readonly ILogger<NodeCommandGrpcService> _log;
/// <summary>Initializes a new instance of the <see cref="NodeCommandGrpcService"/> class.</summary>
/// <param name="registry">Actor registry holding this node's <c>NodeCommunicationActor</c>.</param>
/// <param name="log">Logger.</param>
public NodeCommandGrpcService(ActorRegistry registry, ILogger<NodeCommandGrpcService> log)
{
_registry = registry;
_log = log;
}
/// <inheritdoc />
public override Task<ExecuteAck> Execute(CommandEnvelope request, ServerCallContext context)
{
ArgumentNullException.ThrowIfNull(request);
ArgumentNullException.ThrowIfNull(context);
object command;
try
{
command = MeshTransportDtoMapper.FromEnvelope(request);
}
catch (Exception ex) when (ex is InvalidOperationException or FormatException or ArgumentException)
{
// The envelope carries no credential material, so logging its correlation id is safe and is
// the only handle an operator has on a call whose inner fields did not parse.
_log.LogWarning(
ex,
"Rejected a mesh command (correlationId={CorrelationId}, oneof={CommandCase}): the "
+ "envelope could not be decoded. The command was NOT delivered.",
request.CorrelationId, request.CommandCase);
throw new RpcException(new Status(
StatusCode.InvalidArgument, "Command envelope could not be decoded."));
}
if (!_registry.TryGet<NodeCommunicationActorKey>(out var comm))
{
_log.LogWarning(
"Rejected a mesh command (correlationId={CorrelationId}, type={CommandType}): this node's "
+ "communication actor is not registered yet. The node is still starting; the command was "
+ "NOT delivered and central may retry.",
request.CorrelationId, command.GetType().Name);
throw new RpcException(new Status(
StatusCode.Unavailable, "Node is still starting: the command endpoint is not ready."));
}
comm.Tell(command);
_log.LogDebug(
"Mesh: accepted {CommandType} (correlationId={CorrelationId}) over gRPC; handed to the node "
+ "communication actor.",
command.GetType().Name, request.CorrelationId);
return Task.FromResult(new ExecuteAck());
}
}
src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs
+17
View File
@@ -534,6 +534,14 @@ if (hasAdmin)
// Depends on ISecretResolver, which AddOtOpcUaSecrets registers unconditionally below (DI
// registration order is irrelevant; resolution order is not).
builder.Services.AddSingleton<IMeshPskProvider, MeshPskProvider>();
// Central's outbound command dial (ClusterClient→gRPC Phase 1A). Admin-only for the same reason
// as the provider above — a node never dials another node — and singleton because it owns a
// GrpcChannel per node endpoint; per-call channels would leak sockets across a fleet-wide fan.
// Registered in every mode: CentralCommunicationActor only reaches it under Mode=Grpc, and an
// idle registration costs nothing, whereas gating it here would mean flipping the mode on a
// running fleet needs a central restart before any command lands — the same reasoning that
// registers the ClusterClient receptionist unconditionally.
builder.Services.AddSingleton<IMeshCommandDialer, MeshGrpcCommandDialer>();
}
// Registered unconditionally: driver-role nodes resolve Layer-B DriverConfig secrets and have no auth/DP/AdminUI.
@@ -740,9 +748,18 @@ if (hasAdmin && configServeGrpcPort > 0)
// Driver-node live-telemetry stream (per-cluster mesh Phase 5). Central dials in; gated by
// TelemetryStreamAuthInterceptor (fail-closed on empty Telemetry:ApiKey).
//
// The node-side command endpoint (ClusterClient→gRPC migration Phase 1) rides the SAME driver-role
// h2c listener rather than claiming a fourth port: central already dials this node here, both
// directions are per-cluster-keyed, and a separate port would be one more thing to open on every
// site firewall for no isolation gain. Mapping it is what makes MeshCommandAuthInterceptor live —
// it is registered above and is a pass-through until a call matches this service's method prefix.
// It fail-closes with no MeshTransport:GrpcPsk, so an operator who maps the service without setting
// a key gets a closed endpoint, not an open one.
if (hasDriver && telemetryListenPort > 0)
{
app.MapGrpcService<TelemetryStreamGrpcService>();
app.MapGrpcService<NodeCommandGrpcService>();
}
app.MapOtOpcUaHealth();
tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/Communication/CentralCommunicationActorTests.cs
+674
View File
@@ -1,4 +1,5 @@
using System.Collections.Immutable;
using System.Diagnostics.CodeAnalysis;
using Akka.Actor;
using Akka.Cluster.Tools.Client;
using Akka.Cluster.Tools.PublishSubscribe;
@@ -9,6 +10,8 @@ using Xunit;
using ZB.MOM.WW.OtOpcUa.Cluster;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Deploy;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
using ZB.MOM.WW.OtOpcUa.Commons.Protos.MeshTransport.V1;
using ZB.MOM.WW.OtOpcUa.Commons.Types;
using ZB.MOM.WW.OtOpcUa.Configuration;
using ZB.MOM.WW.OtOpcUa.Configuration.Entities;
@@ -56,6 +59,13 @@ public class CentralCommunicationActorTests : ControlPlaneActorTestBase
ContactRefreshSeconds = refreshSeconds,
};
private static MeshTransportOptions GrpcMode(int refreshSeconds = 60) => new()
{
Mode = MeshTransportOptions.ModeGrpc,
ContactRefreshSeconds = refreshSeconds,
DispatchTimeoutSeconds = 10,
};
private static void SeedCluster(
IDbContextFactory<OtOpcUaConfigDbContext> factory,
string clusterId,
@@ -92,6 +102,46 @@ public class CentralCommunicationActorTests : ControlPlaneActorTestBase
db.SaveChanges();
}
/// <summary>
/// Seeds a cluster whose nodes carry a <c>GrpcPort</c> — the gRPC command plane's shape.
/// Kept separate from <see cref="SeedCluster"/> so the ClusterClient facts, which have no
/// opinion about the port, stay exactly as they were.
/// </summary>
private static void SeedGrpcCluster(
IDbContextFactory<OtOpcUaConfigDbContext> factory,
string clusterId,
params (string NodeId, string Host, int? GrpcPort, bool Enabled, bool Maintenance)[] nodes)
{
using var db = factory.CreateDbContext();
db.ServerClusters.Add(new ServerCluster
{
ClusterId = clusterId,
Name = clusterId,
Enterprise = "ZB",
Site = "Site",
RedundancyMode = RedundancyMode.None,
NodeCount = 1,
CreatedBy = "test",
});
foreach (var n in nodes)
{
db.ClusterNodes.Add(new ClusterNode
{
NodeId = n.NodeId,
ClusterId = clusterId,
Host = n.Host,
AkkaPort = 4053,
GrpcPort = n.GrpcPort,
ApplicationUri = $"urn:{n.NodeId}",
Enabled = n.Enabled,
MaintenanceMode = n.Maintenance,
CreatedBy = "test",
});
}
db.SaveChanges();
}
private IActorRef Spawn(
IDbContextFactory<OtOpcUaConfigDbContext> factory,
MeshTransportOptions options,
@@ -306,4 +356,628 @@ public class CentralCommunicationActorTests : ControlPlaneActorTestBase
actor.Tell(new MeshCommand(DeploymentsTopic, "anything"));
ExpectNoMsg(TimeSpan.FromMilliseconds(200));
}
// ------------------------------------------------------------------ Mode = Grpc
//
// The third transport. The fan-out must reach EXACTLY the recipient set SendToAll reaches today
// — every enabled, non-maintenance node in every cluster — because the node side still has no
// ClusterId or node filter; scoping happens later, inside the artifact. A narrower fan would
// leave nodes silently on their old configuration while the deployment sealed green.
[Fact]
public void Grpc_mode_fans_one_command_to_every_enabled_node_across_all_clusters()
{
// THE assertion of this task, and the direct counterpart of
// A_command_fans_out_a_send_to_all_to_every_cluster_client.
var factory = NewInMemoryDbFactory();
SeedGrpcCluster(
factory,
"C1",
("host-a:4053", "host-a", 4056, true, false),
("host-b:4053", "host-b", 4056, true, false));
SeedGrpcCluster(factory, "C2", ("host-c:4053", "host-c", 4056, true, false));
var rig = SpawnGrpcRig(factory);
AwaitContactsLoaded(rig);
rig.Actor.Tell(new MeshCommand(DeploymentsTopic, NewDispatch()));
AwaitCondition(() => rig.Dialer.Calls.Count == 3, TimeSpan.FromSeconds(5));
rig.Dialer.Calls.Select(c => c.Endpoint).ShouldBe(
new[] { "http://host-a:4056", "http://host-b:4056", "http://host-c:4056" }, ignoreOrder: true);
// Each node is dialled with ITS OWN cluster's id — that is what selects the pre-shared key,
// so getting it wrong turns every command into a PermissionDenied at the node.
rig.Dialer.Calls.Single(c => c.Endpoint.Contains("host-c")).ClusterId.ShouldBe("C2");
rig.Dialer.Calls.Where(c => !c.Endpoint.Contains("host-c")).ShouldAllBe(c => c.ClusterId == "C1");
}
[Fact]
public void Grpc_mode_creates_no_ClusterClient_and_publishes_nothing_on_pubsub()
{
var factory = NewInMemoryDbFactory();
SeedGrpcCluster(factory, "C1", ("host-a:4053", "host-a", 4056, true, false));
var clients = new RecordingClientFactory(() => CreateTestProbe());
var mediatorProbe = CreateTestProbe();
DistributedPubSub.Get(Sys).Mediator.Tell(
new Subscribe(DeploymentsTopic, mediatorProbe.Ref), mediatorProbe.Ref);
mediatorProbe.ExpectMsg<SubscribeAck>(TimeSpan.FromSeconds(5));
var rig = SpawnGrpcRig(factory, clientFactory: clients);
AwaitContactsLoaded(rig);
rig.Actor.Tell(new MeshCommand(DeploymentsTopic, NewDispatch()));
AwaitCondition(() => rig.Dialer.Calls.Count == 1, TimeSpan.FromSeconds(5));
clients.Calls.ShouldBeEmpty();
mediatorProbe.ExpectNoMsg(TimeSpan.FromMilliseconds(300));
}
[Fact]
public void Grpc_mode_does_not_dial_disabled_or_maintenance_nodes()
{
// Same recipient filter as the ClusterClient contact load, asserted independently so the two
// transports cannot drift on who counts as a node.
var factory = NewInMemoryDbFactory();
SeedGrpcCluster(
factory,
"C1",
("host-a:4053", "host-a", 4056, true, false),
("host-b:4053", "host-b", 4056, false, false),
("host-c:4053", "host-c", 4056, true, true));
var rig = SpawnGrpcRig(factory);
AwaitContactsLoaded(rig);
rig.Actor.Tell(new MeshCommand(DeploymentsTopic, NewDispatch()));
AwaitCondition(() => rig.Dialer.Calls.Count == 1, TimeSpan.FromSeconds(5));
rig.Dialer.Calls[0].Endpoint.ShouldContain("host-a");
// And no late arrival from the two excluded rows.
ExpectNoMsg(TimeSpan.FromMilliseconds(300));
rig.Dialer.Calls.Count.ShouldBe(1);
}
[Fact]
public void A_node_with_no_GrpcPort_is_skipped_with_a_warning_and_the_others_are_still_dialled()
{
// Mirrors TelemetryNodeSource: a row with no port exposes no endpoint yet. It must cost that
// node its commands, never the whole fan.
var factory = NewInMemoryDbFactory();
SeedGrpcCluster(
factory,
"C1",
("host-a:4053", "host-a", null, true, false),
("host-b:4053", "host-b", 4056, true, false));
GrpcRig? rig = null;
EventFilter.Warning(contains: "has no GrpcPort").ExpectOne(
TimeSpan.FromSeconds(5), () => rig = SpawnGrpcRig(factory));
rig.ShouldNotBeNull();
AwaitContactsLoaded(rig);
rig.Actor.Tell(new MeshCommand(DeploymentsTopic, NewDispatch()));
AwaitCondition(() => rig.Dialer.Calls.Count == 1, TimeSpan.FromSeconds(5));
rig.Dialer.Calls[0].Endpoint.ShouldBe("http://host-b:4056");
}
[Fact]
public void A_node_that_regains_then_loses_its_GrpcPort_warns_again()
{
// Warn-once is per continuous skip run, not per process. The suppression exists so a
// permanently portless row does not repeat every minute; carried across a repair it would
// silence the SECOND time a node loses its endpoint — the occurrence an operator most needs
// to see, because it is a regression rather than a known gap.
var factory = NewInMemoryDbFactory();
SeedGrpcCluster(factory, "C1", ("host-a:4053", "host-a", null, true, false));
var rig = SpawnGrpcRig(factory);
AwaitContactsLoaded(rig);
SetGrpcPort(factory, "host-a:4053", 4056);
rig.Actor.Tell(new CentralCommunicationActor.RefreshContacts());
AwaitCondition(
() => rig.Psk.KnownClusterPublications.Count >= 2, TimeSpan.FromSeconds(5));
SetGrpcPort(factory, "host-a:4053", null);
EventFilter.Warning(contains: "has no GrpcPort").ExpectOne(TimeSpan.FromSeconds(5), () =>
rig.Actor.Tell(new CentralCommunicationActor.RefreshContacts()));
}
[Fact]
public void One_failing_node_does_not_stop_the_fan_and_is_logged_by_node_id()
{
// Fire-and-forget: a failure is logged and DROPPED. No retry, no queue, and above all no
// abort of the remaining nodes — the deploy path's apply deadline is the backstop.
var factory = NewInMemoryDbFactory();
SeedGrpcCluster(
factory,
"C1",
("host-a:4053", "host-a", 4056, true, false),
("host-b:4053", "host-b", 4056, true, false));
var dialer = new RecordingDialer
{
FailEndpointsContaining = "host-a",
Fault = new Grpc.Core.RpcException(
new Grpc.Core.Status(Grpc.Core.StatusCode.Unavailable, "node down")),
};
var rig = SpawnGrpcRig(factory, dialer);
AwaitContactsLoaded(rig);
// The node id, not the endpoint: an operator reading the log is looking at ClusterNode rows.
EventFilter.Warning(contains: "host-a:4053").ExpectOne(TimeSpan.FromSeconds(5), () =>
rig.Actor.Tell(new MeshCommand(DeploymentsTopic, NewDispatch())));
// Both were attempted; the healthy one is unaffected by its neighbour's failure.
AwaitCondition(() => rig.Dialer.Calls.Count == 2, TimeSpan.FromSeconds(5));
// Still serving after a per-node failure — no supervision restart, no state loss.
rig.Actor.Tell(new MeshCommand(DeploymentsTopic, NewDispatch()));
AwaitCondition(() => rig.Dialer.Calls.Count == 4, TimeSpan.FromSeconds(5));
}
[Fact]
public void A_psk_miss_costs_only_that_node_and_never_becomes_a_fan_level_abort()
{
// The dialer refuses a keyless cluster locally, so it arrives here as an ordinary per-node
// failure. The fan-level assertion is that the other cluster is still dialled.
var factory = NewInMemoryDbFactory();
SeedGrpcCluster(factory, "C1", ("host-a:4053", "host-a", 4056, true, false));
SeedGrpcCluster(factory, "C2", ("host-b:4053", "host-b", 4056, true, false));
var dialer = new RecordingDialer
{
FailClusters = ["C2"],
Fault = new Grpc.Core.RpcException(
new Grpc.Core.Status(Grpc.Core.StatusCode.Unauthenticated, "no key")),
};
var rig = SpawnGrpcRig(factory, dialer);
AwaitContactsLoaded(rig);
EventFilter.Warning(contains: "Unauthenticated").ExpectOne(TimeSpan.FromSeconds(5), () =>
rig.Actor.Tell(new MeshCommand(DeploymentsTopic, NewDispatch())));
AwaitCondition(() => rig.Dialer.Calls.Count == 2, TimeSpan.FromSeconds(5));
}
[Fact]
public void A_command_with_no_grpc_target_is_dropped_with_a_warning()
{
var factory = NewInMemoryDbFactory();
var rig = SpawnGrpcRig(factory);
AwaitContactsLoaded(rig);
EventFilter.Warning(contains: "dropping").ExpectOne(TimeSpan.FromSeconds(5), () =>
rig.Actor.Tell(new MeshCommand(DeploymentsTopic, NewDispatch())));
rig.Dialer.Calls.ShouldBeEmpty();
}
[Fact]
public void A_payload_that_is_not_a_mesh_command_record_is_dropped_with_an_error()
{
// The mapper refuses to encode an unknown payload rather than shipping something the node
// cannot decode. That refusal must cost the command, not the actor.
var factory = NewInMemoryDbFactory();
SeedGrpcCluster(factory, "C1", ("host-a:4053", "host-a", 4056, true, false));
var rig = SpawnGrpcRig(factory);
AwaitContactsLoaded(rig);
EventFilter.Error(contains: "Cannot encode").ExpectOne(TimeSpan.FromSeconds(5), () =>
rig.Actor.Tell(new MeshCommand(DeploymentsTopic, "not a command record")));
rig.Dialer.Calls.ShouldBeEmpty();
// Still serving.
rig.Actor.Tell(new MeshCommand(DeploymentsTopic, NewDispatch()));
AwaitCondition(() => rig.Dialer.Calls.Count == 1, TimeSpan.FromSeconds(5));
}
[Fact]
public void The_envelope_carries_the_commands_own_correlation_and_the_whole_fan_shares_it()
{
var factory = NewInMemoryDbFactory();
SeedGrpcCluster(
factory,
"C1",
("host-a:4053", "host-a", 4056, true, false),
("host-b:4053", "host-b", 4056, true, false));
var rig = SpawnGrpcRig(factory);
AwaitContactsLoaded(rig);
var correlationId = new CorrelationId(Guid.NewGuid());
rig.Actor.Tell(new MeshCommand(DeploymentsTopic, new DispatchDeployment(
new DeploymentId(Guid.NewGuid()), new RevisionHash("abc"), correlationId)));
AwaitCondition(() => rig.Dialer.Calls.Count == 2, TimeSpan.FromSeconds(5));
rig.Dialer.Calls.ShouldAllBe(c => c.Envelope.CorrelationId == correlationId.ToString());
// One envelope object, encoded once and fanned — not re-encoded per node.
rig.Dialer.Calls.Select(c => c.Envelope).Distinct().Count().ShouldBe(1);
}
[Fact]
public void An_alarm_command_gets_one_minted_transport_correlation_shared_by_the_whole_fan()
{
// AlarmCommand is the one record with no correlation member of its own, so the envelope's is
// the only handle an operator has on it. Minting per fan rather than per node is what makes
// "this alarm command" one traceable event instead of N unrelated ones.
var factory = NewInMemoryDbFactory();
SeedGrpcCluster(
factory,
"C1",
("host-a:4053", "host-a", 4056, true, false),
("host-b:4053", "host-b", 4056, true, false));
var rig = SpawnGrpcRig(factory);
AwaitContactsLoaded(rig);
rig.Actor.Tell(new MeshCommand(
"alarms", new AlarmCommand("alarm-1", "Acknowledge", "operator", null, null)));
AwaitCondition(() => rig.Dialer.Calls.Count == 2, TimeSpan.FromSeconds(5));
rig.Dialer.Calls.Select(c => c.Envelope.CorrelationId).Distinct().Count().ShouldBe(1);
rig.Dialer.Calls[0].Envelope.CorrelationId.ShouldNotBeNullOrWhiteSpace();
rig.Dialer.Calls[0].Envelope.AlarmCommand.AlarmId.ShouldBe("alarm-1");
}
[Fact]
public void The_contact_refresh_publishes_the_known_cluster_set_and_drops_cached_psks()
{
// Phase 1A security register item 1, wired: the gate is only a control if something keeps it
// current, and the refresh is the moment central's picture of the fleet is up to date. The
// InvalidateAll on the same tick is what makes a rotated key take effect within one refresh
// instead of requiring a central restart.
var factory = NewInMemoryDbFactory();
SeedGrpcCluster(factory, "C1", ("host-a:4053", "host-a", 4056, true, false));
SeedGrpcCluster(factory, "C2", ("host-b:4053", "host-b", null, true, false));
var rig = SpawnGrpcRig(factory);
AwaitContactsLoaded(rig);
// C2 is included even though its only node has no GrpcPort: the gate is about which clusters
// central KNOWS, not which it can currently dial. Excluding it would deny that cluster's acks.
rig.Psk.KnownClusterPublications[0].ShouldBe(new[] { "C1", "C2" }, ignoreOrder: true);
rig.Psk.InvalidateAllCount.ShouldBeGreaterThanOrEqualTo(1);
}
[Fact]
public void A_slow_dial_does_not_park_the_actors_mailbox()
{
// The dialer's first act is a PSK lookup, which blocks on the secret store — bounded, but
// seconds, and repeated after every refresh because the refresh drops the cache. An async
// method runs synchronously to its first await, so dialing straight from the receive would
// stall this actor for that long PER NODE, and deployment acks queue behind that mailbox.
var factory = NewInMemoryDbFactory();
SeedGrpcCluster(factory, "C1", ("host-a:4053", "host-a", 4056, true, false));
var coordinator = CreateTestProbe();
var dialer = new RecordingDialer { BlockBeforeFirstAwait = true };
var rig = SpawnGrpcRig(factory, dialer, coordinator: coordinator.Ref);
AwaitContactsLoaded(rig);
try
{
rig.Actor.Tell(new MeshCommand(DeploymentsTopic, NewDispatch()));
// The ack must be handled while the dial is still parked.
var ack = new ApplyAck(
new DeploymentId(Guid.NewGuid()),
new NodeId("host-a:4053"),
ApplyAckOutcome.Applied,
null,
new CorrelationId(Guid.NewGuid()));
rig.Actor.Tell(ack);
coordinator.ExpectMsg<ApplyAck>(TimeSpan.FromSeconds(5)).ShouldBe(ack);
}
finally
{
dialer.Release();
}
}
[Fact]
public void A_node_added_after_start_joins_the_fan_on_the_next_refresh()
{
// The refresh is not decoration: operators add and retire nodes, and central must pick them
// up without a restart. Under Grpc that is the ONLY thing keeping the dispatch set current —
// there is no receptionist to notice a new registration on its own.
//
// Driven by telling the actor its own RefreshContacts tick rather than by shortening
// ContactRefreshSeconds: a one-second timer would run for the whole fact, adding DB load to
// every test sharing the runner, and would make this assertion depend on wall-clock timing.
var factory = NewInMemoryDbFactory();
SeedGrpcCluster(factory, "C1", ("host-a:4053", "host-a", 4056, true, false));
var rig = SpawnGrpcRig(factory);
AwaitContactsLoaded(rig);
using (var db = factory.CreateDbContext())
{
db.ClusterNodes.Add(new ClusterNode
{
NodeId = "host-b:4053",
ClusterId = "C1",
Host = "host-b",
AkkaPort = 4053,
GrpcPort = 4056,
ApplicationUri = "urn:host-b",
Enabled = true,
CreatedBy = "test",
});
db.SaveChanges();
}
rig.Actor.Tell(new CentralCommunicationActor.RefreshContacts());
AwaitCondition(
() => rig.Psk.KnownClusterPublications.Count >= 2, TimeSpan.FromSeconds(5));
rig.Actor.Tell(new MeshCommand(DeploymentsTopic, NewDispatch()));
AwaitCondition(() => rig.Dialer.Calls.Count == 2, TimeSpan.FromSeconds(5));
rig.Dialer.Calls.Select(c => c.Endpoint).ShouldBe(
new[] { "http://host-a:4056", "http://host-b:4056" }, ignoreOrder: true);
// Once PER REFRESH, not once at startup. A key can rotate without any membership change, so
// an invalidation that only fired when the cluster set moved would serve the pre-rotation
// key forever and a rotation would need a central restart.
rig.Psk.InvalidateAllCount.ShouldBeGreaterThanOrEqualTo(2);
}
[Theory]
[InlineData(MeshTransportOptions.ModeDps)]
[InlineData(MeshTransportOptions.ModeClusterClient)]
[InlineData(MeshTransportOptions.ModeGrpc)]
public void The_known_cluster_gate_is_published_in_every_transport_mode(string mode)
{
// Replaces an earlier fact asserting the OPPOSITE, whose premise did not survive 1B: the
// gate feeds CentralCommandAuthInterceptor, which is registered on every admin node, and the
// ack service it protects is mapped by LISTENER PORT — neither consults MeshTransport:Mode.
// Gating the publication on Grpc meant a dark-switch rollback to ClusterClient or Dps left
// that endpoint serving with a permanently inactive gate, silently restoring both the timing
// oracle and the pre-authentication store/audit flood. The set is central's cluster
// inventory, not transport state.
var factory = NewInMemoryDbFactory();
SeedGrpcCluster(factory, "C1", ("host-a:4053", "host-a", 4056, true, false));
var psk = new RecordingPskProvider();
var clients = new RecordingClientFactory(() => CreateTestProbe());
Sys.ActorOf(CentralCommunicationActor.Props(
factory,
new MeshTransportOptions { Mode = mode, ContactRefreshSeconds = 60 },
clients,
() => null,
() => new RecordingDialer(),
() => psk));
AwaitCondition(
() => psk.KnownClusterPublications.Count >= 1, TimeSpan.FromSeconds(5));
psk.KnownClusterPublications[0].ShouldBe(new[] { "C1" });
psk.InvalidateAllCount.ShouldBeGreaterThanOrEqualTo(1);
// The one load-bearing line the publication itself cannot witness: with the refresh
// timer now running in every mode, only the ClusterClient branch may construct clients.
// With rows seeded, deleting the !_useClusterClient guard turns this red for Dps/Grpc.
if (mode != MeshTransportOptions.ModeClusterClient)
clients.Calls.ShouldBeEmpty();
}
[Fact]
public void The_gate_set_includes_a_cluster_the_fan_set_excludes()
{
// The two sets answer different questions. The FAN asks "who does central talk to" — enabled,
// non-maintenance, ported. The GATE asks "is this a cluster central has ever heard of", and
// must say yes for a cluster whose every node is disabled or in maintenance: such a cluster
// can still send a deployment ack, and refusing it would quietly turn maintenance mode into
// an authentication switch.
var factory = NewInMemoryDbFactory();
SeedGrpcCluster(factory, "C1", ("host-a:4053", "host-a", 4056, true, false));
SeedGrpcCluster(factory, "C-DISABLED", ("host-b:4053", "host-b", 4056, false, false));
SeedGrpcCluster(factory, "C-MAINT", ("host-c:4053", "host-c", 4056, true, true));
var rig = SpawnGrpcRig(factory);
AwaitContactsLoaded(rig);
rig.Psk.KnownClusterPublications[0].ShouldBe(
new[] { "C1", "C-DISABLED", "C-MAINT" }, ignoreOrder: true);
rig.Actor.Tell(new MeshCommand(DeploymentsTopic, NewDispatch()));
AwaitCondition(() => rig.Dialer.Calls.Count == 1, TimeSpan.FromSeconds(5));
ExpectNoMsg(TimeSpan.FromMilliseconds(300));
rig.Dialer.Calls.Count.ShouldBe(1);
rig.Dialer.Calls[0].ClusterId.ShouldBe("C1");
}
[Fact]
public void The_gate_is_published_after_the_dispatch_set_is_adopted_and_before_the_cache_is_dropped()
{
// Two orderings in one handler, both load-bearing. SetKnownClusters must precede
// InvalidateAll, or the first lookup after a refresh re-reads the store under the PREVIOUS
// membership — briefly resolving a cluster central has just retired. And the dispatch set is
// adopted first so a command processed immediately after the publication already fans over
// the current targets.
var factory = NewInMemoryDbFactory();
SeedGrpcCluster(factory, "C1", ("host-a:4053", "host-a", 4056, true, false));
var rig = SpawnGrpcRig(factory);
AwaitContactsLoaded(rig);
rig.Psk.Sequence.ShouldBe(new[] { "SetKnownClusters", "InvalidateAll" });
// Targets-before-gate, observed the only way it can be from outside the actor: the very next
// message finds the dispatch set already populated.
rig.Actor.Tell(new MeshCommand(DeploymentsTopic, NewDispatch()));
AwaitCondition(() => rig.Dialer.Calls.Count == 1, TimeSpan.FromSeconds(5));
}
[Fact]
public void Grpc_mode_with_no_dialer_wired_drops_the_command_with_an_error()
{
// Mis-wiring, not misconfiguration: Mode=Grpc validates fine but the DI graph did not supply
// a dialer. Silence here would be a fleet-wide command outage with no symptom until a deploy
// stalled at its apply deadline two minutes later.
var factory = NewInMemoryDbFactory();
SeedGrpcCluster(factory, "C1", ("host-a:4053", "host-a", 4056, true, false));
var actor = Sys.ActorOf(CentralCommunicationActor.Props(
factory,
GrpcMode(),
new RecordingClientFactory(() => CreateTestProbe()),
() => null,
dialer: null,
pskProvider: null));
// Filtered on the DISPATCH message's exact phrase: PreStart logs its own error about the same
// mis-wiring, and a looser filter races it into a second match.
EventFilter.Error(contains: "no command dialer is wired").ExpectOne(
TimeSpan.FromSeconds(5),
() => actor.Tell(new MeshCommand(DeploymentsTopic, NewDispatch())));
}
private static void SetGrpcPort(
IDbContextFactory<OtOpcUaConfigDbContext> factory, string nodeId, int? grpcPort)
{
using var db = factory.CreateDbContext();
var node = db.ClusterNodes.Single(n => n.NodeId == nodeId);
node.GrpcPort = grpcPort;
db.SaveChanges();
}
private static DispatchDeployment NewDispatch() => new(
new DeploymentId(Guid.NewGuid()), new RevisionHash("abc"), new CorrelationId(Guid.NewGuid()));
/// <summary>A spawned gRPC-mode actor plus the two seams the tests observe it through.</summary>
private sealed record GrpcRig(IActorRef Actor, RecordingDialer Dialer, RecordingPskProvider Psk);
private GrpcRig SpawnGrpcRig(
IDbContextFactory<OtOpcUaConfigDbContext> factory,
RecordingDialer? dialer = null,
IMeshClusterClientFactory? clientFactory = null,
IActorRef? coordinator = null)
{
dialer ??= new RecordingDialer();
var psk = new RecordingPskProvider();
var actor = Sys.ActorOf(CentralCommunicationActor.Props(
factory,
GrpcMode(),
clientFactory ?? new RecordingClientFactory(() => CreateTestProbe()),
() => coordinator,
() => dialer,
() => psk));
return new GrpcRig(actor, dialer, psk);
}
/// <summary>
/// Waits until the actor has applied a contact load. The known-cluster publication is the
/// signal because it happens in the same handler, immediately after the target set is
/// adopted — and unlike the target set itself it is observable from outside the actor. A
/// command told before this point would be dropped as "no targets" and never retried, which
/// is correct behaviour and a flaky test.
/// </summary>
private void AwaitContactsLoaded(GrpcRig rig) =>
AwaitCondition(
() => rig.Psk.KnownClusterPublications.Count >= 1, TimeSpan.FromSeconds(5));
/// <summary>Records every dial and answers however the test asks; never opens a socket.</summary>
private sealed class RecordingDialer : IMeshCommandDialer
{
private readonly object _gate = new();
private readonly List<Dial> _calls = [];
private readonly ManualResetEventSlim _released = new(false);
/// <summary>
/// When true, every dial blocks its calling thread synchronously until
/// <see cref="Release"/>. Models the dialer's pre-await PSK lookup.
/// </summary>
public bool BlockBeforeFirstAwait { get; set; }
/// <summary>Unblocks every dial parked by <see cref="BlockBeforeFirstAwait"/>.</summary>
public void Release() => _released.Set();
/// <summary>Every dial made. Copied under the lock — the fan runs on thread-pool threads.</summary>
public IReadOnlyList<Dial> Calls
{
get { lock (_gate) return _calls.ToList(); }
}
/// <summary>Endpoints whose substring match makes the dial fault.</summary>
public string? FailEndpointsContaining { get; set; }
/// <summary>Cluster ids whose dials fault.</summary>
public string[] FailClusters { get; set; } = [];
/// <summary>
/// The fault raised for a matching dial. Fully qualified because this file's bare
/// <c>Status</c> is Akka's — an existing fact drives the actor's <c>Status.Failure</c>
/// handler, and importing Grpc.Core here would make that reference ambiguous.
/// </summary>
public Grpc.Core.RpcException? Fault { get; set; }
public Task DispatchAsync(
string nodeEndpoint, CommandEnvelope envelope, string clusterId, CancellationToken ct)
{
lock (_gate) _calls.Add(new Dial(nodeEndpoint, envelope, clusterId));
if (BlockBeforeFirstAwait)
{
// Blocks the CALLING thread, exactly as IMeshPskProvider.TryGetPsk does on a cold
// cache: an async method runs synchronously to its first await, so a dial invoked
// straight from the actor would park the mailbox here.
_released.Wait(TimeSpan.FromSeconds(30));
}
var shouldFail =
(FailEndpointsContaining is not null && nodeEndpoint.Contains(FailEndpointsContaining))
|| FailClusters.Contains(clusterId, StringComparer.OrdinalIgnoreCase);
return shouldFail && Fault is not null
? Task.FromException(Fault)
: Task.CompletedTask;
}
/// <summary>One recorded dial.</summary>
public sealed record Dial(string Endpoint, CommandEnvelope Envelope, string ClusterId);
}
/// <summary>Records the gate and invalidation calls the contact refresh is supposed to make.</summary>
private sealed class RecordingPskProvider : IMeshPskProvider
{
private readonly object _gate = new();
private readonly List<IReadOnlyList<string>> _publications = [];
private readonly List<string> _sequence = [];
private int _invalidateAllCount;
/// <summary>Names of the mutating calls made, in the order they arrived.</summary>
public IReadOnlyList<string> Sequence
{
get { lock (_gate) return _sequence.ToList(); }
}
/// <summary>Every known-cluster set published, in order.</summary>
public IReadOnlyList<IReadOnlyList<string>> KnownClusterPublications
{
get { lock (_gate) return _publications.ToList(); }
}
/// <summary>How many times the whole cache was dropped.</summary>
public int InvalidateAllCount => Volatile.Read(ref _invalidateAllCount);
public bool TryGetPsk(string clusterId, [NotNullWhen(true)] out string? psk)
{
psk = "psk";
return true;
}
public bool Matches(string clusterId, string presented) => true;
public void SetKnownClusters(IReadOnlyCollection<string> clusterIds)
{
lock (_gate)
{
_publications.Add(clusterIds.ToList());
_sequence.Add(nameof(SetKnownClusters));
}
}
public void Invalidate(string clusterId) { }
public void InvalidateAll()
{
Interlocked.Increment(ref _invalidateAllCount);
lock (_gate) _sequence.Add(nameof(InvalidateAll));
}
}
}
tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/Communication/ControlPlaneStarterOrderTests.cs
+249
View File
@@ -0,0 +1,249 @@
using System.Diagnostics.CodeAnalysis;
using Akka.Actor;
using Akka.Cluster.Hosting;
using Akka.Hosting;
using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Options;
using Shouldly;
using Xunit;
using ZB.MOM.WW.Audit;
using ZB.MOM.WW.OtOpcUa.Cluster;
using ZB.MOM.WW.OtOpcUa.Commons.Interfaces;
using ZB.MOM.WW.OtOpcUa.Commons.Types;
using ZB.MOM.WW.OtOpcUa.Configuration;
using ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
using ZB.MOM.WW.Secrets.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Tests.Communication;
/// <summary>
/// Guards the one thing every other test in this project structurally cannot see: whether an
/// admin node <b>boots</b> with the real registration graph behind the mesh command boundary.
/// </summary>
/// <remarks>
/// <para>
/// The Phase 1A live gate failed here. <c>WithOtOpcUaControlPlaneSingletons</c>'s first
/// starter resolved <c>IMeshCommandDialer</c> and <c>IMeshPskProvider</c> eagerly, and their
/// graph reaches back into the actor registry — dialer → <c>IMeshPskProvider</c> →
/// <c>ISecretResolver</c> → <c>AddZbSecrets</c> → <c>IAuditWriter</c> →
/// <c>ActorAuditWriter</c>, whose constructor calls
/// <c>registry.Get&lt;AuditWriterActorKey&gt;()</c> for a singleton registered further down
/// the SAME builder. Every central node died on
/// <c>MissingActorRegistryEntryException</c>, in every transport mode, before serving
/// anything.
/// </para>
/// <para>
/// <b>Why the existing tests were blind.</b> <c>MeshGrpcCommandDialerTests</c> activates the
/// dialer from a <c>ServiceCollection</c> that has no secrets registration at all, so its
/// graph stops one hop in; the <c>CentralCommunicationActorTests</c> facts inject fakes
/// directly and never touch DI; and the existing whole-host boots register no mesh provider,
/// so the eager lookups returned <see langword="null"/> and resolved nothing. Every one of
/// them passed while no central node could start.
/// </para>
/// <para>
/// <b>So the assertion is <c>StartAsync</c> itself.</b> The failure happens while the Akka
/// builder's starters execute, which is inside <c>StartAsync</c> — building the
/// <c>ServiceProvider</c> and resolving from it by hand would not reproduce it. The graph
/// below is the production one from <c>IMeshCommandDialer</c> down to the registry-bound
/// audit writer; only the secret store itself is a stand-in, because a real one would add a
/// KEK and a SQLite file to a test whose subject is starter ORDER.
/// </para>
/// </remarks>
public sealed class ControlPlaneStarterOrderTests
{
[Theory]
[InlineData(MeshTransportOptions.ModeDps)]
[InlineData(MeshTransportOptions.ModeClusterClient)]
[InlineData(MeshTransportOptions.ModeGrpc)]
public async Task An_admin_node_boots_with_the_real_mesh_dependency_graph(string mode)
{
// Every mode, because the eager resolve was mode-independent: the starter resolved both
// dependencies before anything looked at MeshTransport:Mode, so Dps — the DEFAULT, and the
// rollback target — died exactly like Grpc did.
using var host = BuildAdminHost(mode);
await Should.NotThrowAsync(() => host.StartAsync());
try
{
var registry = host.Services.GetRequiredService<ActorRegistry>();
registry.TryGet<CentralCommunicationActorKey>(out _).ShouldBeTrue(
"the command boundary must be registered on a booted admin node");
registry.TryGet<AuditWriterActorKey>(out _).ShouldBeTrue(
"the audit singleton the eager resolve reached for too early must exist by now");
}
finally
{
await host.StopAsync();
}
}
[Theory]
[InlineData(MeshTransportOptions.ModeDps)]
[InlineData(MeshTransportOptions.ModeClusterClient)]
[InlineData(MeshTransportOptions.ModeGrpc)]
public async Task The_deferred_dependencies_are_resolved_and_used_once_the_host_is_up(string mode)
{
// Deferring must not become "never resolved" — that would trade a loud boot failure for a
// silent one, with the known-cluster gate permanently unpublished and the dialer permanently
// absent. So this asserts the far end of the deferral: the actor resolved the PSK provider
// FROM THE CONTAINER and drove it, through the ordinary contact-refresh path. A Theory over
// all three modes because publication is mode-independent (the gate follows the endpoint).
//
// The provider is registered as a TYPE with an ISecretResolver constructor, so the resolve
// walks the same registry-bound chain the production MeshPskProvider does — which makes
// this fact the race detector for the PreStart-vs-starter-#4 window: a lost race that
// stayed null forever would never publish, and the TaskCompletionSource below would time
// out red. (Memoize-on-first-use in the actor is what heals the lost race at the first
// RefreshContacts message.)
//
// The signal is a TaskCompletionSource rather than a counter, because PreStart runs on the
// actor's own thread and the contact load is a piped DB read: there is no instant after
// StartAsync at which a synchronous assertion is correct.
using var host = BuildAdminHost(mode, useSignallingProvider: true);
await host.StartAsync();
try
{
var signal = (SignallingPskProvider)host.Services.GetRequiredService<IMeshPskProvider>();
await signal.Published.WaitAsync(TimeSpan.FromSeconds(20));
}
finally
{
await host.StopAsync();
}
}
/// <summary>
/// Builds an admin host through the production cluster bootstrap and the production control-plane
/// starter registration, with the mesh dependency graph wired as it is on a real central node.
/// </summary>
/// <param name="mode">The mesh transport mode to boot under.</param>
/// <param name="useSignallingProvider">
/// Substitutes <see cref="SignallingPskProvider"/> for the real <see cref="MeshPskProvider"/>.
/// Only the resolution test sets it; the boot test keeps the production type so the graph it
/// walks is the production graph. The signalling provider is registered as a factory taking
/// <see cref="ISecretResolver"/> from the container, so its resolution walks the same
/// registry-bound chain either way.
/// </param>
/// <returns>The unstarted host.</returns>
private static IHost BuildAdminHost(string mode, bool useSignallingProvider = false)
{
var clusterOptions = new AkkaClusterOptions
{
Port = 0,
Hostname = "127.0.0.1",
PublicHostname = "127.0.0.1",
SeedNodes = [],
Roles = ["admin"],
};
return Host.CreateDefaultBuilder()
.ConfigureServices(services =>
{
services.AddSingleton<IOptions<AkkaClusterOptions>>(Options.Create(clusterOptions));
services.AddSingleton<IOptions<MeshTransportOptions>>(Options.Create(
new MeshTransportOptions { Mode = mode, DispatchTimeoutSeconds = 10 }));
services.AddDbContextFactory<OtOpcUaConfigDbContext>(
o => o.UseInMemoryDatabase($"starter-order-{Guid.NewGuid():N}"));
services.AddSingleton<IClusterRoleInfo>(new StubClusterRoleInfo());
// THE GRAPH. Production types from the dialer down to the registry-bound writer; the
// order of these registrations is deliberately irrelevant, because the defect was
// never about registration order — it was about WHEN the graph is first walked.
services.AddSingleton<IAuditWriter, RegistryBoundAuditWriter>();
services.AddSingleton<ISecretResolver, AuditingSecretResolver>();
if (useSignallingProvider)
services.AddSingleton<IMeshPskProvider>(sp =>
new SignallingPskProvider(sp.GetRequiredService<ISecretResolver>()));
else
services.AddSingleton<IMeshPskProvider, MeshPskProvider>();
services.AddSingleton<IMeshCommandDialer, MeshGrpcCommandDialer>();
services.AddAkka("otopcua-starter-order-test", (ab, sp) =>
{
ab.WithOtOpcUaClusterBootstrap(sp);
ab.WithOtOpcUaControlPlaneSingletons();
});
})
.Build();
}
/// <summary>
/// Stands in for <c>ActorAuditWriter</c>: an <see cref="IAuditWriter"/> that resolves a
/// cluster singleton from the <see cref="ActorRegistry"/> <b>in its constructor</b>. That
/// constructor is the tripwire — it throws
/// <c>MissingActorRegistryEntryException</c> for exactly as long as the audit singleton has
/// not been registered yet.
/// </summary>
private sealed class RegistryBoundAuditWriter : IAuditWriter
{
private readonly IActorRef _proxy;
public RegistryBoundAuditWriter(ActorRegistry registry) => _proxy = registry.Get<AuditWriterActorKey>();
public Task WriteAsync(AuditEvent evt, CancellationToken ct = default)
{
_proxy.Tell(evt);
return Task.CompletedTask;
}
}
/// <summary>
/// An <see cref="IMeshPskProvider"/> that completes a task the first time the actor publishes
/// the known-cluster set. Takes <see cref="ISecretResolver"/> it never uses, purely so that
/// resolving it still walks the same registry-bound chain the real provider does.
/// </summary>
private sealed class SignallingPskProvider : IMeshPskProvider
{
private readonly TaskCompletionSource _published =
new(TaskCreationOptions.RunContinuationsAsynchronously);
public SignallingPskProvider(ISecretResolver resolver) => ArgumentNullException.ThrowIfNull(resolver);
/// <summary>Completes once <see cref="SetKnownClusters"/> has been called.</summary>
public Task Published => _published.Task;
public bool TryGetPsk(string clusterId, [NotNullWhen(true)] out string? psk)
{
psk = null;
return false;
}
public bool Matches(string clusterId, string presented) => false;
public void SetKnownClusters(IReadOnlyCollection<string> clusterIds) => _published.TrySetResult();
public void Invalidate(string clusterId) { }
public void InvalidateAll() { }
}
/// <summary>
/// Stands in for the shared secret store's resolver. Its only load-bearing property is the
/// one that broke boot: it depends on <see cref="IAuditWriter"/>, exactly as
/// <c>AddZbSecrets</c>' resolver registration does.
/// </summary>
private sealed class AuditingSecretResolver : ISecretResolver
{
public AuditingSecretResolver(IAuditWriter auditWriter) => ArgumentNullException.ThrowIfNull(auditWriter);
public Task<string?> GetAsync(SecretName name, CancellationToken ct) => Task.FromResult<string?>(null);
}
/// <summary>Minimal <see cref="IClusterRoleInfo"/>; the admin singleton factories read it.</summary>
private sealed class StubClusterRoleInfo : IClusterRoleInfo
{
public string? ClusterRole => null;
public string? ClusterId => null;
public NodeId LocalNode => NodeId.Parse("127.0.0.1:0");
public IReadOnlySet<string> LocalRoles => new HashSet<string>();
public bool HasRole(string role) => false;
public IReadOnlyList<NodeId> MembersWithRole(string role) => [];
public NodeId? RoleLeader(string role) => null;
public event EventHandler<RoleLeaderChangedEventArgs>? RoleLeaderChanged { add { } remove { } }
}
}
tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/Communication/MeshGrpcCommandDialerTests.cs
+340
View File
@@ -0,0 +1,340 @@
using System.Diagnostics.CodeAnalysis;
using Grpc.Core;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Cluster;
using ZB.MOM.WW.OtOpcUa.Commons.Protos.MeshTransport.V1;
using ZB.MOM.WW.OtOpcUa.ControlPlane.Communication;
namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Tests.Communication;
/// <summary>
/// ClusterClient→gRPC Phase 1A — central's outbound command dial. One node, one call, no retry.
/// </summary>
/// <remarks>
/// The facts that matter are the credential ones: the bearer must be the TARGET node's cluster's
/// key (not central's, and not some other cluster's), and a cluster with no resolvable key must
/// never be dialled at all. Everything else is plumbing.
/// </remarks>
public sealed class MeshGrpcCommandDialerTests
{
private const string Endpoint = "http://site-a-1:4056";
private static CommandEnvelope Envelope() => new()
{
CorrelationId = Guid.NewGuid().ToString("N"),
RestartDriver = new RestartDriverCommand
{
ClusterId = "SITE-A",
DriverInstanceId = "drv-1",
ActorByUserName = "operator",
CorrelationId = Guid.NewGuid().ToString("N"),
},
};
private static MeshGrpcCommandDialer CreateDialer(
IMeshPskProvider psk,
RecordingClient client,
int dispatchTimeoutSeconds = 10)
=> new(
psk,
Options.Create(new MeshTransportOptions { DispatchTimeoutSeconds = dispatchTimeoutSeconds }),
NullLogger<MeshGrpcCommandDialer>.Instance,
_ => client);
[Fact]
public async Task Dial_presents_the_target_clusters_key_and_names_that_cluster()
{
// THE assertion. Central holds N keys; sending the wrong one turns a legitimate command into
// a PermissionDenied, and sending a fleet-wide one would be the blast radius this design
// exists to avoid.
var client = new RecordingClient();
var dialer = CreateDialer(
new FakePskProvider(("SITE-A", "psk-site-a"), ("SITE-B", "psk-site-b")), client);
await dialer.DispatchAsync(Endpoint, Envelope(), "SITE-B", CancellationToken.None);
client.LastHeaders.ShouldNotBeNull();
client.LastHeaders!.GetValue("authorization").ShouldBe("Bearer psk-site-b");
client.LastHeaders.GetValue("x-otopcua-cluster").ShouldBe("SITE-B");
}
[Fact]
public async Task A_cluster_with_no_key_is_refused_locally_and_never_dialled()
{
// Fail-closed: not "dial and let the node reject it". An unauthenticated command attempt
// against a node's control surface should not exist even as a rejected one.
var client = new RecordingClient();
var dialer = CreateDialer(new FakePskProvider(("SITE-A", "psk-site-a")), client);
var ex = await Should.ThrowAsync<RpcException>(
() => dialer.DispatchAsync(Endpoint, Envelope(), "SITE-Z", CancellationToken.None));
ex.StatusCode.ShouldBe(StatusCode.Unauthenticated);
client.Calls.ShouldBe(0);
}
[Fact]
public async Task The_envelope_reaches_the_node_unchanged()
{
var client = new RecordingClient();
var dialer = CreateDialer(new FakePskProvider(("SITE-A", "psk-site-a")), client);
var envelope = Envelope();
await dialer.DispatchAsync(Endpoint, envelope, "SITE-A", CancellationToken.None);
client.LastRequest.ShouldBe(envelope);
}
[Fact]
public async Task A_node_rejection_surfaces_as_its_own_RpcException()
{
// The actor grades per-node outcomes off the status code, so it must arrive intact rather
// than wrapped in an AggregateException or swallowed here.
var client = new RecordingClient
{
Fault = new RpcException(new Status(StatusCode.PermissionDenied, "nope")),
};
var dialer = CreateDialer(new FakePskProvider(("SITE-A", "psk-site-a")), client);
var ex = await Should.ThrowAsync<RpcException>(
() => dialer.DispatchAsync(Endpoint, Envelope(), "SITE-A", CancellationToken.None));
ex.StatusCode.ShouldBe(StatusCode.PermissionDenied);
}
[Fact]
public async Task The_per_call_deadline_bounds_a_node_that_never_answers()
{
// A dispatch dials every node in the fleet. Without this bound one unreachable node holds a
// task and a socket for as long as the process lives, while the deploy path's own apply
// deadline expires around it.
var client = new RecordingClient { HangUntilCancelled = true };
var dialer = CreateDialer(
new FakePskProvider(("SITE-A", "psk-site-a")), client, dispatchTimeoutSeconds: 1);
await Should.ThrowAsync<OperationCanceledException>(
() => dialer.DispatchAsync(Endpoint, Envelope(), "SITE-A", CancellationToken.None)
.WaitAsync(TimeSpan.FromSeconds(10)));
}
[Fact]
public async Task The_callers_token_still_cancels_a_dial_inside_the_deadline()
{
// Linked, not replaced: actor shutdown must cancel everything in flight even when the
// per-call deadline is nowhere near expiring.
var client = new RecordingClient { HangUntilCancelled = true };
var dialer = CreateDialer(
new FakePskProvider(("SITE-A", "psk-site-a")), client, dispatchTimeoutSeconds: 600);
using var cts = new CancellationTokenSource();
var dial = dialer.DispatchAsync(Endpoint, Envelope(), "SITE-A", cts.Token);
await cts.CancelAsync();
await Should.ThrowAsync<OperationCanceledException>(() => dial.WaitAsync(TimeSpan.FromSeconds(10)));
}
[Fact]
public async Task A_non_positive_dispatch_timeout_is_clamped_rather_than_honoured()
{
// Zero would mean an UNBOUNDED call — the exact failure the deadline exists to prevent — and
// MeshTransportOptionsValidator's rule for this key is scoped to driver-role nodes, so a
// pure-admin central reaches this code with the value unvalidated. Clamp loudly.
var client = new RecordingClient { HangUntilCancelled = true };
var logger = new RecordingLogger();
var dialer = new MeshGrpcCommandDialer(
new FakePskProvider(("SITE-A", "psk-site-a")),
Options.Create(new MeshTransportOptions { DispatchTimeoutSeconds = 0 }),
logger,
_ => client);
logger.Warnings.ShouldContain(w => w.Contains("DispatchTimeoutSeconds"));
// And the clamp is real, not just logged: the dial must still expire on its own.
await Should.ThrowAsync<OperationCanceledException>(
() => dialer.DispatchAsync(Endpoint, Envelope(), "SITE-A", CancellationToken.None)
.WaitAsync(TimeSpan.FromSeconds(20)));
}
[Fact]
public async Task Dialing_after_disposal_throws_rather_than_repopulating_the_channel_cache()
{
var client = new RecordingClient();
var dialer = CreateDialer(new FakePskProvider(("SITE-A", "psk-site-a")), client);
dialer.Dispose();
await Should.ThrowAsync<ObjectDisposedException>(
() => dialer.DispatchAsync(Endpoint, Envelope(), "SITE-A", CancellationToken.None));
}
[Fact]
public async Task The_real_channel_factory_dials_and_is_disposable()
{
// The fake client seam covers every other fact here, which means nothing else ever exercises
// GrpcChannel.ForAddress or the cache it fills. Dial an endpoint nothing is listening on:
// the failure must be an ordinary RpcException the actor can log, not a construction throw,
// and Dispose must then clean up the channel the dial created.
var dialer = new MeshGrpcCommandDialer(
new FakePskProvider(("SITE-A", "psk-site-a")),
Options.Create(new MeshTransportOptions { DispatchTimeoutSeconds = 5 }),
NullLogger<MeshGrpcCommandDialer>.Instance);
await Should.ThrowAsync<RpcException>(
() => dialer.DispatchAsync(
"http://127.0.0.1:1", Envelope(), "SITE-A", CancellationToken.None));
Should.NotThrow(dialer.Dispose);
}
[Fact]
public async Task Two_dials_to_one_endpoint_share_a_single_channel()
{
// The cache is the whole defence against a socket per call: a deploy dispatch fans across
// every node in the fleet, and it repeats on every redeploy. A channel per dial exhausts
// ports on central long before anything else notices.
var dialer = new MeshGrpcCommandDialer(
new FakePskProvider(("SITE-A", "psk-site-a")),
// 1 s, not 5: on a host that blackholes loopback instead of refusing, each dial costs
// the full deadline — keep the worst case bounded so this never reads as a hang.
Options.Create(new MeshTransportOptions { DispatchTimeoutSeconds = 1 }),
NullLogger<MeshGrpcCommandDialer>.Instance);
// Nothing is listening, so both dials fail — irrelevant here: the channel is constructed
// before the call is placed, which is exactly what is being counted.
for (var i = 0; i < 2; i++)
{
await Should.ThrowAsync<RpcException>(
() => dialer.DispatchAsync(
"http://127.0.0.1:1", Envelope(), "SITE-A", CancellationToken.None));
}
dialer.CachedChannelCount.ShouldBe(1);
// A second endpoint is a second channel — the cache keys on the endpoint, it does not
// collapse every node onto one connection.
await Should.ThrowAsync<RpcException>(
() => dialer.DispatchAsync(
"http://127.0.0.1:2", Envelope(), "SITE-A", CancellationToken.None));
dialer.CachedChannelCount.ShouldBe(2);
Should.NotThrow(dialer.Dispose);
}
[Fact]
public void It_activates_from_the_container_despite_the_optional_client_factory_parameter()
{
// The seam is an optional constructor parameter, which MS.DI honours only because it checks
// HasDefaultValue. Get that wrong — by making it non-optional, or by adding a second public
// constructor — and the failure is a startup crash on the admin node, invisible to every
// other test here because they all construct the dialer by hand.
var services = new ServiceCollection();
services.AddLogging();
services.AddSingleton<IMeshPskProvider>(new FakePskProvider(("SITE-A", "psk-site-a")));
services.AddSingleton<IOptions<MeshTransportOptions>>(
Options.Create(new MeshTransportOptions { DispatchTimeoutSeconds = 10 }));
services.AddSingleton<IMeshCommandDialer, MeshGrpcCommandDialer>();
using var provider = services.BuildServiceProvider();
provider.GetRequiredService<IMeshCommandDialer>().ShouldBeOfType<MeshGrpcCommandDialer>();
}
/// <summary>
/// A <see cref="NodeCommandService.NodeCommandServiceClient"/> that records what it was
/// handed and answers however the test asks. The generated client's protected parameterless
/// constructor exists precisely for this.
/// </summary>
private sealed class RecordingClient : NodeCommandService.NodeCommandServiceClient
{
/// <summary>Number of calls that reached the client.</summary>
public int Calls { get; private set; }
/// <summary>Headers of the most recent call.</summary>
public Metadata? LastHeaders { get; private set; }
/// <summary>Request of the most recent call.</summary>
public CommandEnvelope? LastRequest { get; private set; }
/// <summary>When set, the call faults with this instead of answering.</summary>
public RpcException? Fault { get; set; }
/// <summary>When true, the call never completes until its token is cancelled.</summary>
public bool HangUntilCancelled { get; set; }
public override AsyncUnaryCall<ExecuteAck> ExecuteAsync(
CommandEnvelope request,
Metadata? headers = null,
DateTime? deadline = null,
CancellationToken cancellationToken = default)
{
Calls++;
LastHeaders = headers;
LastRequest = request;
var response = Fault is not null
? Task.FromException<ExecuteAck>(Fault)
: HangUntilCancelled
? HangAsync(cancellationToken)
: Task.FromResult(new ExecuteAck());
return new AsyncUnaryCall<ExecuteAck>(
response,
Task.FromResult(new Metadata()),
() => Status.DefaultSuccess,
() => [],
() => { });
}
private static async Task<ExecuteAck> HangAsync(CancellationToken ct)
{
await Task.Delay(Timeout.InfiniteTimeSpan, ct).ConfigureAwait(false);
return new ExecuteAck();
}
}
/// <summary>Captures Warning-level messages so a construction-time check is assertable.</summary>
private sealed class RecordingLogger : ILogger<MeshGrpcCommandDialer>
{
/// <summary>Every warning logged, formatted.</summary>
public List<string> Warnings { get; } = [];
public IDisposable? BeginScope<TState>(TState state) where TState : notnull => null;
public bool IsEnabled(LogLevel logLevel) => true;
public void Log<TState>(
LogLevel logLevel,
EventId eventId,
TState state,
Exception? exception,
Func<TState, Exception?, string> formatter)
{
if (logLevel == LogLevel.Warning)
Warnings.Add(formatter(state, exception));
}
}
/// <summary>An <see cref="IMeshPskProvider"/> over a fixed key set; unknown clusters fail closed.</summary>
private sealed class FakePskProvider(params (string Cluster, string Psk)[] keys) : IMeshPskProvider
{
private readonly Dictionary<string, string> _keys =
keys.ToDictionary(k => k.Cluster, k => k.Psk, StringComparer.OrdinalIgnoreCase);
public bool TryGetPsk(string clusterId, [NotNullWhen(true)] out string? psk)
=> _keys.TryGetValue(clusterId, out psk);
public bool Matches(string clusterId, string presented)
=> _keys.TryGetValue(clusterId, out var expected)
&& string.Equals(expected, presented, StringComparison.Ordinal);
public void SetKnownClusters(IReadOnlyCollection<string> clusterIds) { }
public void Invalidate(string clusterId) { }
public void InvalidateAll() { }
}
}
tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/Communication/MeshPskProviderTests.cs
+199
View File
@@ -367,6 +367,205 @@ public sealed class MeshPskProviderTests
resolver.CallCount.ShouldBe(0);
}
// ------------------------------------------------------------------ known-cluster gate
//
// Phase 1A security register item 1. Central rebuilds its cluster set from ClusterNode rows on
// every contact refresh; a cluster id outside that set must be refused in O(1) BEFORE anything
// touches the secret store. Two things ride on that: an unauthenticated caller can no longer
// make central perform a store read (and an audit write) per forged x-otopcua-cluster header,
// and the timing difference between a known cluster (cached) and an unknown one (never cached,
// because negatives are deliberately not retained) stops being remotely measurable.
[Fact]
public void UnknownCluster_WithTheGatePopulated_IsRefused_WithoutTouchingTheStore()
{
// THE assertion of the register item. CallCount == 0 is the whole point: the refusal must
// cost nothing, or the gate is a comment rather than a control.
var resolver = new FakeSecretResolver { Value = "stored-psk" };
var provider = CreateProvider(resolver);
provider.SetKnownClusters(["SITE-A", "SITE-B"]);
provider.TryGetPsk("SITE-Z", out var psk).ShouldBeFalse();
psk.ShouldBeNull();
resolver.CallCount.ShouldBe(0);
}
[Fact]
public void KnownCluster_WithTheGatePopulated_StillResolves()
{
var resolver = new FakeSecretResolver { Value = "stored-psk" };
var provider = CreateProvider(resolver);
provider.SetKnownClusters(["SITE-A"]);
provider.TryGetPsk("SITE-A", out var psk).ShouldBeTrue();
psk.ShouldBe("stored-psk");
resolver.CallCount.ShouldBe(1);
}
[Fact]
public void TheKnownClusterSet_IsCaseInsensitive()
{
// ClusterId case is whatever the operator typed into the DB row; the header is whatever the
// node was configured with. A case-sensitive gate would refuse a legitimate cluster.
var resolver = new FakeSecretResolver { Value = "stored-psk" };
var provider = CreateProvider(resolver);
provider.SetKnownClusters(["SITE-A"]);
provider.TryGetPsk("site-a", out var psk).ShouldBeTrue();
psk.ShouldBe("stored-psk");
}
[Fact]
public void SwappingTheKnownClusterSet_AddsAndRemovesClusters()
{
// The set is rebuilt from the DB every refresh, so both directions happen in production: a
// new site appears, a retired one stops being accepted.
var resolver = new FakeSecretResolver { Value = "stored-psk" };
var provider = CreateProvider(resolver);
provider.SetKnownClusters(["SITE-A"]);
provider.TryGetPsk("SITE-A", out _).ShouldBeTrue();
provider.TryGetPsk("SITE-B", out _).ShouldBeFalse();
provider.SetKnownClusters(["SITE-B"]);
provider.TryGetPsk("SITE-B", out _).ShouldBeTrue();
provider.TryGetPsk("SITE-A", out _).ShouldBeFalse();
}
[Fact]
public void AnEmptyKnownClusterSet_LeavesTheGateInactive()
{
// Fail-OPEN, deliberately and narrowly: central publishes the set from its contact refresh,
// and an empty publish means "the DB told me nothing", not "no cluster is legitimate".
// Refusing everything there would make a fleet-wide ack outage the failure mode of an empty
// or unreadable ClusterNode table.
var resolver = new FakeSecretResolver { Value = "stored-psk" };
var provider = CreateProvider(resolver);
provider.SetKnownClusters(["SITE-A"]);
provider.SetKnownClusters([]);
provider.TryGetPsk("SITE-Z", out var psk).ShouldBeTrue();
psk.ShouldBe("stored-psk");
}
[Fact]
public void AnUnsetKnownClusterSet_LeavesTheGateInactive()
{
// Admin boot: the interceptor is live from the first request, but the first contact refresh
// has not run yet. Gating on a set nobody has published would deny every ack in that window.
var resolver = new FakeSecretResolver { Value = "stored-psk" };
var provider = CreateProvider(resolver);
provider.TryGetPsk("SITE-Z", out var psk).ShouldBeTrue();
psk.ShouldBe("stored-psk");
}
[Fact]
public void TheGate_IsConsultedBeforeTheConfigMap()
{
// Consequence worth pinning rather than discovering: a ClusterPsks entry for a cluster with
// no enabled ClusterNode row is refused once the gate is populated. That is correct — central
// would neither dial such a cluster nor accept an ack from it — but it is surprising enough
// on a rig to deserve a test that names it.
var resolver = new FakeSecretResolver();
var provider = CreateProvider(
resolver, new Dictionary<string, string> { ["SITE-X"] = "rig-psk-site-x" });
provider.SetKnownClusters(["SITE-A"]);
provider.TryGetPsk("SITE-X", out var psk).ShouldBeFalse();
psk.ShouldBeNull();
}
[Fact]
public void SetKnownClusters_TakesACopy_SoALaterMutationOfTheCallersList_CannotWidenTheGate()
{
var clusters = new List<string> { "SITE-A" };
var resolver = new FakeSecretResolver { Value = "stored-psk" };
var provider = CreateProvider(resolver);
provider.SetKnownClusters(clusters);
clusters.Add("SITE-Z");
provider.TryGetPsk("SITE-Z", out _).ShouldBeFalse();
}
// ------------------------------------------------------------------ Matches
[Fact]
public void Matches_AcceptsTheClustersOwnKey()
{
var resolver = new FakeSecretResolver();
var provider = CreateProvider(
resolver, new Dictionary<string, string> { ["SITE-A"] = "psk-site-a" });
provider.Matches("SITE-A", "psk-site-a").ShouldBeTrue();
}
[Fact]
public void Matches_RejectsAnotherClustersKey()
{
// The isolation property, asserted at the provider rather than only at the interceptor:
// choosing a cluster only changes which key you are graded against.
var resolver = new FakeSecretResolver();
var provider = CreateProvider(
resolver,
new Dictionary<string, string> { ["SITE-A"] = "psk-site-a", ["SITE-B"] = "psk-site-b" });
provider.Matches("SITE-A", "psk-site-b").ShouldBeFalse();
}
[Fact]
public void Matches_IsNotAPrefixMatch()
{
var resolver = new FakeSecretResolver();
var provider = CreateProvider(
resolver, new Dictionary<string, string> { ["SITE-A"] = "psk-site-a" });
provider.Matches("SITE-A", "psk-site-").ShouldBeFalse();
provider.Matches("SITE-A", "psk-site-a-and-more").ShouldBeFalse();
}
[Fact]
public void Matches_FailsClosed_WhenTheClusterHasNoKey()
{
var resolver = new FakeSecretResolver { Value = null };
var provider = CreateProvider(resolver);
provider.Matches("SITE-Z", "anything-at-all").ShouldBeFalse();
}
[Theory]
[InlineData("")]
[InlineData(" ")]
public void Matches_RejectsABlankPresentedToken(string presented)
{
// A blank key never reaches the cache (blank resolves fail closed), but a blank PRESENTED
// token must be refused on its own terms rather than relying on that.
var resolver = new FakeSecretResolver();
var provider = CreateProvider(
resolver, new Dictionary<string, string> { ["SITE-A"] = "psk-site-a" });
provider.Matches("SITE-A", presented).ShouldBeFalse();
}
[Fact]
public void Matches_IsRefusedByTheKnownClusterGate_WithoutTouchingTheStore()
{
var resolver = new FakeSecretResolver { Value = "stored-psk" };
var provider = CreateProvider(resolver);
provider.SetKnownClusters(["SITE-A"]);
provider.Matches("SITE-Z", "stored-psk").ShouldBeFalse();
resolver.CallCount.ShouldBe(0);
}
/// <summary>
/// An <see cref="ISecretResolver"/> whose answer the test controls between calls, so cache
/// staleness and invalidation are observable without a real store.
tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/AssemblyInfo.cs
+2 -1
View File
@@ -2,5 +2,6 @@ using Xunit;
// The TelemetryStreamGrpcService concurrency counter is process-wide (static). Run this assembly's
// tests serially so the cap test observes a deterministic active-stream count with no cross-test
// interference. This project holds only telemetry tests, so serial execution costs nothing.
// interference. Every other suite here is small and fast (the mesh-command tests spin a TestKit
// ActorSystem per class), so serial execution costs nothing.
[assembly: CollectionBehavior(DisableTestParallelization = true)]
tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/Configuration/CentralCommandAuthInterceptorTests.cs
+45 -1
View File
@@ -242,14 +242,58 @@ public sealed class CentralCommandAuthInterceptorTests
publicCtors.Length.ShouldBe(1);
}
[Fact]
public async Task Verification_GoesThroughMatches_AndTheInterceptorNeverHandlesTheKeyItself()
{
// Phase 1A security register item 2. The interceptor must not fetch the raw key and compare
// it — that is how a constant-time comparison eventually becomes '=='. Matches owns the
// comparison; TryGetPsk survives only for the DIAL side, which has to send the key.
// (The production provider's Matches calls TryGetPsk internally; this fake implements the two
// independently, so a zero TryGetPsk count here means the INTERCEPTOR never asked for the key.)
var provider = new FakeMeshPskProvider(("SITE-A", "psk-site-a"));
var interceptor = new CentralCommandAuthInterceptor(
provider, NullLogger<CentralCommandAuthInterceptor>.Instance);
await Invoke(interceptor, CreateContext(ReportApplyMethod, "Bearer psk-site-a", "SITE-A"));
provider.MatchesCalls.ShouldBe(1);
provider.TryGetPskCalls.ShouldBe(0);
}
/// <summary>An <see cref="IMeshPskProvider"/> over a fixed key set; unknown clusters fail closed.</summary>
private sealed class FakeMeshPskProvider(params (string Cluster, string Psk)[] keys) : IMeshPskProvider
{
private readonly Dictionary<string, string> _keys =
keys.ToDictionary(k => k.Cluster, k => k.Psk, StringComparer.OrdinalIgnoreCase);
/// <summary>How many times the raw key was asked for — must stay 0 on the verify path.</summary>
public int TryGetPskCalls { get; private set; }
/// <summary>How many times verification went through the provider's comparison.</summary>
public int MatchesCalls { get; private set; }
public bool TryGetPsk(string clusterId, [NotNullWhen(true)] out string? psk)
=> _keys.TryGetValue(clusterId, out psk);
{
TryGetPskCalls++;
return _keys.TryGetValue(clusterId, out psk);
}
/// <summary>
/// Implemented independently of <see cref="TryGetPsk"/> so the two calls are separately
/// countable — the interface declares this with no default implementation, deliberately.
/// Semantics match the production provider's: no key, or a key that is not exactly the
/// presented token.
/// </summary>
public bool Matches(string clusterId, string presented)
{
MatchesCalls++;
return !string.IsNullOrWhiteSpace(presented)
&& _keys.TryGetValue(clusterId, out var expected)
&& string.Equals(expected, presented, StringComparison.Ordinal);
}
/// <summary>No-op: the gate is the production provider's behaviour, pinned in its own tests.</summary>
public void SetKnownClusters(IReadOnlyCollection<string> clusterIds) { }
public void Invalidate(string clusterId) { }
tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/Grpc/NodeCommandGrpcServiceTests.cs
+186
View File
@@ -0,0 +1,186 @@
using Akka.Hosting;
using Akka.TestKit.Xunit2;
using Grpc.Core;
using Microsoft.Extensions.Logging.Abstractions;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Admin;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Deploy;
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Mesh;
using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
using ZB.MOM.WW.OtOpcUa.Commons.Protos.MeshTransport.V1;
using ZB.MOM.WW.OtOpcUa.Commons.Types;
using ZB.MOM.WW.OtOpcUa.Host.Grpc;
using ZB.MOM.WW.OtOpcUa.Runtime;
namespace ZB.MOM.WW.OtOpcUa.Host.Tests.Grpc;
/// <summary>
/// Verifies the node-side command ingress (ClusterClient→gRPC migration Phase 1): each of the four
/// central→node commands is decoded off its <see cref="CommandEnvelope"/> and handed to the
/// registered <c>NodeCommunicationActor</c> as the same CLR record the ClusterClient transport
/// delivered, an undecodable envelope is refused with <see cref="StatusCode.InvalidArgument"/>, and
/// a call landing in the boot window before the actor is registered is refused with
/// <see cref="StatusCode.Unavailable"/>.
/// </summary>
/// <remarks>
/// The actor is a <c>TestProbe</c> registered under <see cref="NodeCommunicationActorKey"/>: what
/// this service owes the rest of the system is the exact message the real actor already knows how to
/// re-emit, so asserting on what arrives at the registry key is the whole contract. The onward leg —
/// comm actor to EventStream to the real driver host — is pinned unchanged by
/// <c>EventStreamCommandDeliveryTests</c> in the Runtime test project.
/// </remarks>
public sealed class NodeCommandGrpcServiceTests : TestKit
{
private static readonly CorrelationId AlarmCorrelation = new(Guid.NewGuid());
private static ServerCallContext Ctx() => new FakeServerCallContext();
private static NodeCommandGrpcService NewService(ActorRegistry registry) =>
new(registry, NullLogger<NodeCommandGrpcService>.Instance);
[Fact]
public async Task Each_of_the_four_commands_reaches_the_comm_actor_as_its_domain_record()
{
var probe = CreateTestProbe();
var registry = new ActorRegistry();
registry.Register<NodeCommunicationActorKey>(probe.Ref);
var service = NewService(registry);
var dispatch = new DispatchDeployment(
new DeploymentId(Guid.NewGuid()), new RevisionHash("rev-1"), new CorrelationId(Guid.NewGuid()));
var restart = new RestartDriver("cluster-a", "drv-1", "alice", Guid.NewGuid());
var reconnect = new ReconnectDriver("cluster-a", "drv-2", "bob", Guid.NewGuid());
var alarm = new AlarmCommand("alarm-7", "Acknowledge", "carol", "looked at it", null);
(await service.Execute(MeshTransportDtoMapper.ToEnvelope(dispatch), Ctx())).ShouldNotBeNull();
probe.ExpectMsg<DispatchDeployment>().ShouldBe(dispatch);
await service.Execute(MeshTransportDtoMapper.ToEnvelope(restart), Ctx());
probe.ExpectMsg<RestartDriver>().ShouldBe(restart);
await service.Execute(MeshTransportDtoMapper.ToEnvelope(reconnect), Ctx());
probe.ExpectMsg<ReconnectDriver>().ShouldBe(reconnect);
await service.Execute(MeshTransportDtoMapper.ToEnvelope(alarm, AlarmCorrelation), Ctx());
probe.ExpectMsg<AlarmCommand>().ShouldBe(alarm);
}
[Fact]
public async Task A_call_in_the_boot_window_before_the_actor_is_registered_is_unavailable()
{
// Nothing registered: the ActorSystem exists but AddOtOpcUaRuntime has not spawned the comm
// actor yet. Unavailable (not Internal) because the condition is transient and retryable —
// central's dialer may try the same node again a moment later and succeed.
var service = NewService(new ActorRegistry());
var dispatch = new DispatchDeployment(
new DeploymentId(Guid.NewGuid()), new RevisionHash("rev-1"), new CorrelationId(Guid.NewGuid()));
var ex = await Should.ThrowAsync<RpcException>(
() => service.Execute(MeshTransportDtoMapper.ToEnvelope(dispatch), Ctx()));
ex.StatusCode.ShouldBe(StatusCode.Unavailable);
}
[Fact]
public async Task An_unset_oneof_is_invalid_argument()
{
var probe = CreateTestProbe();
var registry = new ActorRegistry();
registry.Register<NodeCommunicationActorKey>(probe.Ref);
var service = NewService(registry);
var ex = await Should.ThrowAsync<RpcException>(() => service.Execute(
new CommandEnvelope { CorrelationId = Guid.NewGuid().ToString("N") }, Ctx()));
ex.StatusCode.ShouldBe(StatusCode.InvalidArgument);
probe.ExpectNoMsg(TimeSpan.FromMilliseconds(200));
}
[Theory]
[MemberData(nameof(UndecodableEnvelopes))]
public async Task An_envelope_whose_fields_do_not_parse_is_invalid_argument(CommandEnvelope envelope)
{
var probe = CreateTestProbe();
var registry = new ActorRegistry();
registry.Register<NodeCommunicationActorKey>(probe.Ref);
var service = NewService(registry);
var ex = await Should.ThrowAsync<RpcException>(() => service.Execute(envelope, Ctx()));
ex.StatusCode.ShouldBe(StatusCode.InvalidArgument);
probe.ExpectNoMsg(TimeSpan.FromMilliseconds(200));
}
public static TheoryData<CommandEnvelope> UndecodableEnvelopes() =>
[
// Deployment id is not a hyphenless GUID -> FormatException out of the mapper.
new CommandEnvelope
{
CorrelationId = Guid.NewGuid().ToString("N"),
DispatchDeployment = new DispatchDeploymentCommand
{
DeploymentId = "not-a-guid",
RevisionHash = "rev-1",
CorrelationId = Guid.NewGuid().ToString("N"),
},
},
// Revision hash is blank -> ArgumentException out of RevisionHash.Parse.
new CommandEnvelope
{
CorrelationId = Guid.NewGuid().ToString("N"),
DispatchDeployment = new DispatchDeploymentCommand
{
DeploymentId = Guid.NewGuid().ToString("N"),
RevisionHash = "",
CorrelationId = Guid.NewGuid().ToString("N"),
},
},
// Restart command with an unparseable correlation id -> FormatException.
new CommandEnvelope
{
CorrelationId = Guid.NewGuid().ToString("N"),
RestartDriver = new RestartDriverCommand
{
ClusterId = "cluster-a",
DriverInstanceId = "drv-1",
ActorByUserName = "alice",
CorrelationId = "nope",
},
},
];
[Fact]
public async Task An_undecodable_envelope_in_the_boot_window_is_invalid_argument_not_unavailable()
{
// Ordering pin: decode runs BEFORE the registry lookup. Unavailable invites the caller to
// retry, and an envelope this build cannot decode will never decode — reporting it as a
// readiness problem would turn a permanent fault into an unbounded retry loop.
var service = NewService(new ActorRegistry());
var ex = await Should.ThrowAsync<RpcException>(() => service.Execute(
new CommandEnvelope { CorrelationId = Guid.NewGuid().ToString("N") }, Ctx()));
ex.StatusCode.ShouldBe(StatusCode.InvalidArgument);
}
/// <summary>Minimal <see cref="ServerCallContext"/>; this RPC reads nothing off the context but its method name.</summary>
private sealed class FakeServerCallContext : ServerCallContext
{
protected override string MethodCore => "/mesh_transport.v1.NodeCommandService/Execute";
protected override string HostCore => "localhost";
protected override string PeerCore => "ipv4:127.0.0.1:0";
protected override DateTime DeadlineCore => DateTime.MaxValue;
protected override Metadata RequestHeadersCore => [];
protected override CancellationToken CancellationTokenCore => CancellationToken.None;
protected override Metadata ResponseTrailersCore { get; } = [];
protected override Status StatusCore { get; set; }
protected override WriteOptions? WriteOptionsCore { get; set; }
protected override AuthContext AuthContextCore => new(null, new Dictionary<string, List<AuthProperty>>());
protected override ContextPropagationToken CreatePropagationTokenCore(ContextPropagationOptions? options) =>
throw new NotSupportedException();
protected override Task WriteResponseHeadersAsyncCore(Metadata responseHeaders) => Task.CompletedTask;
}
}
tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/ZB.MOM.WW.OtOpcUa.Host.Tests.csproj
+3
View File
@@ -10,6 +10,9 @@
<ItemGroup>
<PackageReference Include="xunit"/>
<PackageReference Include="Shouldly"/>
<!-- NodeCommandGrpcServiceTests registers a TestProbe under NodeCommunicationActorKey, so the
service is asserted against the same ActorRegistry seam the Host wires in production. -->
<PackageReference Include="Akka.TestKit.Xunit2"/>
<!-- ConfigDbApiKeyStoreTests drives the store against a real EF query, the same in-memory
provider convention the ControlPlane/Runtime test projects use. No central SQL Server. -->
<PackageReference Include="Microsoft.EntityFrameworkCore.InMemory"/>