Task 3.2 of the ClusterClient->gRPC migration. Sweeps the docs that still
described the Akka ClusterClient transport as live, following its Phase 3
deletion (this branch): MeshTransport:Mode is now Dps (pair-local DPS) or
Grpc (cross-mesh unary calls) only, ClusterClient fails validation loudly,
CentralContactPoints is deleted (not superseded), and the split-topology
validator requires exactly Grpc rather than ClusterClient.
- docs/Configuration.md: rewrite the MeshTransport section — table,
dark-switch/PSK-seeding prose, listener sharing (:4056 driver-role,
:4055 admin-role), remove the "shipped dark, cutover pending" framing.
- docs/Redundancy.md: replace the Dps/ClusterClient transport table with
Dps/Grpc, fix the Phase 6 CentralCommunicationActor + validator bullets.
- docs/Telemetry.md: fix stale ClusterClient cross-refs, add a note that
the driver-role telemetry listener also hosts NodeCommandService.
- docs/v2/config-db-schema.md: one-line "(retired: ...)" annotation on the
historical AkkaPort/GrpcPort note — left as a historical record otherwise.
- CLAUDE.md: rewrite "## Mesh command transport (MeshTransport)" for the
post-deletion world (no ClusterClient/receptionist, per-node gRPC dial,
fire-and-forget semantics unchanged) and fix two Phase 6 bullets.
deferment.md has no mesh-transport mentions; left untouched.
docs-only change; `dotnet build ZB.MOM.WW.OtOpcUa.slnx` stays green (0
errors) after the edit.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
MeshPaths.CentralCommunication / .NodeCommunication (the "/user/..." full-path constants) had
ZERO production consumers after the ClusterClient deletion: both gRPC services resolve their
comm actor through ActorRegistry by typed key, never by ActorSelection over a path. Only the
path-probe test still read them, while the class doc went on calling them "the wire contract
of the central↔node boundary" — a claim that stopped being true the moment nothing addressed
an actor across a process boundary by name.
- The two full-path constants are deleted; the two Name constants remain.
- MeshPaths' doc now says what it is (the local actor-naming convention the registry-keyed
services are spawned under) and records what it stopped being, so the next reader does not
re-derive a contract from the file's history.
- MeshCommActorPathTests composes "/user/" + Name itself — the absolute path is that test's
own addressing detail now, not something the product declares. Renamed
Both_mesh_comm_actors_exist_at_their_contract_paths ->
..._are_spawned_under_their_declared_names, because what it still pins is that the two
registration sites actually ran on a real booted node; a rename is now a compile error, so
"contract" was the wrong word for what would break.
- NodeCommunicationActor's summary no longer cross-references the deleted constant or calls
the name an address. CentralCommunicationActor was checked and makes no such claim.
MeshGrpcBoundaryTests replaces the MeshClusterClientBoundaryTests deleted in step 1. Three
self-seeded single-member clusters sharing the ActorSystem name, joined ONLY by gRPC, driving
the REAL CentralCommunicationActor -> MeshGrpcCommandDialer -> Kestrel h2c ->
MeshCommandAuthInterceptor -> NodeCommandGrpcService -> NodeCommunicationActor -> EventStream:
1. No_cross_mesh_Akka_association_forms_between_central_and_either_site — the program's own
point, kept as a test rather than a live-gate note: each cluster stays single-member and
none ever holds another's address as a member.
2. One_dispatch_through_central_reaches_a_node_in_every_cluster_mesh — the fan, with each
site holding its OWN per-cluster PSK, so a dial that selected the wrong cluster's key
would be refused at the far side.
3. Falsifiability_control_a_dial_carrying_the_wrong_cluster_key_delivers_nothing — without
it, (2) cannot distinguish "the gRPC boundary carried it" from "something else in the
process did". The same node then accepts the right key, so the silence is the auth gate
and not a dead endpoint.
All three run fully offline — no Docker fixture — and pass.
Comment sweep across every remaining clusterclient|receptionist hit in src/ and tests/:
the program label becomes "mesh gRPC command plane", and every comment that described the
deleted transport as live now describes what actually runs (Dps = pair-local default,
Grpc = cross-mesh). mesh_transport.proto's header/service comments are reworded; no message,
field or enum is touched — the contract is locked. MeshCommActorPathTests is reshaped rather
than retired: the two paths still exist and still matter, but what a mismatch costs is now a
misleading Unavailable from the gRPC ingress rather than a silently dropped receptionist send.
The only surviving mentions of the old names are the ones that must survive: the fail-closed
rules that name "ClusterClient" as a retired value refused at boot, their tests, and this
test class's own account of what it replaces.
MeshTransportOptions.ModeClusterClient is gone. MeshTransportOptionsValidator accepts Dps or
Grpc and nothing else, so a node left carrying the string "ClusterClient" is now REFUSED at
boot naming MeshTransport:Mode and both remaining values — deliberately fail-closed rather
than falling back to Dps, which on a split-topology node would deliver nothing at all. A new
fact pins exactly that.
SplitTopologyTransportValidator tightens from an allow-list of two to Grpc only, as its own
comment said it would once ClusterClient was deleted. Dps on a cluster-role node stays
refused; the rule is still an explicit allow-list, so a typo'd mode fails too.
Tests: the two ClusterClient-mode facts in MeshTransportOptionsValidatorTests are deleted
(their Dps siblings already assert the same rule for the one remaining non-Grpc mode), the
SplitTopology fixtures move to Grpc, and the three-mode Theories in
ControlPlaneStarterOrderTests / CentralCommunicationActorTests become two-mode.
The property, the four validator rules that graded it (empty / non-akka.tcp / actor-path
suffix / one-bad-among-good) and all twelve docker-compose entries (6 nodes x 2) are gone.
The compose comment blocks that explained "/system/receptionist is appended at construction
time" and the client's contact rotation go with them, and the surrounding prose now
describes the Grpc command plane it actually runs.
CentralGrpcEndpoints' doc no longer says it "replaces" the Akka contact points — there is
nothing left to replace — and ContactRefreshSeconds' failure message names the node
dispatch set rather than the ClusterClient contact points.
`rg CentralContactPoints src tests docker-dev` is now empty.
akka.extensions loses ClusterClientReceptionistExtensionProvider and the whole
akka.cluster.client block (buffer-size / reconnect-timeout / receptionist.role) goes with
it. The DistributedPubSub extension and the maximum-frame-size / log-frame-size-exceeding
canary both stay — DPS is the pair-local transport and the frame canary guards every
remoting message, neither of which the cross-mesh deletion touches.
MeshTransportHoconTests drops its three ClusterClientSettings/ClusterReceptionistSettings
asserts and keeps the frame-size pin and the DPS-extension pin, renamed since it no longer
has a receptionist to survive alongside.
Central's CentralCommunicationActor and a node's NodeCommunicationActor no longer register
with a receptionist; the ActorRegistry entry each already carried is now the whole
reachability story, and it is what CentralCommandGrpcService / NodeCommandGrpcService
resolve to hand an inbound message over. Both `using Akka.Cluster.Tools.Client` directives
are gone with them.
WithOtOpcUaRuntimeActors no longer builds a ClusterClient from
MeshTransport:CentralContactPoints, and NodeCommunicationActor.Props takes only the gRPC
IApplyAckSender. The ackRouter gate narrows from `useClusterClient || useGrpc` to `useGrpc`,
and the CentralClusterClientName constant is gone.
NodeCommunicationActor's no-transport warning now names MeshTransport:CentralGrpcEndpoints
and MeshTransport:Mode only — the old text pointed an operator at a key that no longer
exists.
Ripples in this step: the two ClusterClient-shaped facts in NodeCommunicationActorTests
(Send-not-SendToAll, and sender-wins-over-a-client) are deleted — the first has no subject
left, the second guarded a both-transports-present state that is now unconstructible. The
Runtime ServiceCollectionExtensions fact keeps its ackRouter pin and drops its
"no ClusterClient actor exists" half for the same reason.
IMeshClusterClientFactory/DefaultMeshClusterClientFactory are gone, along with
CentralCommunicationActor's _clients map, RebuildClient, the SendToAll fan and the
receptionist-address half of the contact load. LoadContactsFromDb SURVIVES — it feeds the
gRPC dialer's endpoint resolution and the known-cluster gate — but now projects only the
columns those two need (the AkkaPort/ActorPath.TryParse parse and its Malformed list went
with the receptionist addresses).
Ripples fixed in this step because the build must be green at every commit:
- ControlPlane/ServiceCollectionExtensions drops the factory argument.
- CentralCommunicationActorTests: the six ClusterClient-only facts are deleted (each has a
Grpc counterpart that already asserted the same property independently), the three
transport-agnostic facts move to Grpc mode, and the Dps fact keeps its own coverage.
- MeshClusterClientFactoryTests deleted (its subject is gone).
- MeshClusterClientBoundaryTests deleted here rather than in step 7: it drives
DefaultMeshClusterClientFactory directly, so it cannot compile past this commit. Its
replacement, MeshGrpcBoundaryTests, lands in step 7 and keeps the load-bearing assertion
(two separate meshes, no cross-mesh Akka association).
All six nodes now default to the gRPC command/ack transport; OTOPCUA_MESH_MODE
still overrides for rollback drills. Baseline reproduced pre-flip (1bb3a937):
cross-mesh :4053 open unauthenticated, unreachable-cluster dispatch suppressed
into sampled deadLetters INF with the deploy deadline as sole witness.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Follow-up to a94eb8a3, addressing CHANGES REQUESTED on code review. Production
logic was confirmed correct on all spec points; this commit fixes the review's
test/comment findings only.
IMPORTANT-1: A_permanently_unresolvable_pskProvider_is_reported_once_per_incarnation_not_once_per_refresh
did not prove the second refresh ran — its Expect(0) window gated on a dialer
dispatch landing, but LoadContactsFromDb is fire-and-forget and the first
block's AwaitCondition already re-Tells MeshCommand per poll, so the dialer
call count could reach the threshold before PublishKnownClusters was ever
re-invoked by the second load. Fixed by having the PSK factory count its own
invocations and gating both AwaitConditions on that count directly (pskCalls
>= 2 proves the first retry; pskCalls > pskCallsAfterFirstLoad, awaited INSIDE
the Expect(0) window, proves the second one). Bite-checked: with
_pskProviderAbsenceReported's guard temporarily removed, the test now
correctly fails ("Received 1 message too many... received 1 that matched
filter"); guard restored and reverified green.
MINOR-2: reworded four comments (CentralCommunicationActor.cs ~109, ~259,
~305-309, ~853) from narrating the change ("this fix removes", "already
existed, unchanged", "added by this fix", "the old two-call-site shape") to
stating the timeless invariant.
MINOR-3: added A_supplied_dialer_factory_that_always_throws_still_drops_each_command_loudly_while_PreStart_stays_quiet
— the symmetric case to the existing never-supplied-dialer test, exercising
Resolve's catch + firstUse:true path (a factory that WAS supplied and always
throws) rather than the null-factory path. Asserts PreStart plus the first
contact load stay silent and DispatchOverGrpc's existing per-dispatch Error
still fires on every dispatch, unguarded.
MINOR-4: "Check the ControlPlane registration wires IMeshPskProvider" ->
"Check that the ControlPlane registration wires up IMeshPskProvider"; no test
pinned the old substring, so no test update was needed for this one.
Skipped item 5 (restart-re-arms pin) per instruction — out of scope.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
PreStart's eager resolve of IMeshCommandDialer/IMeshPskProvider loses a
millisecond-wide race against the Akka builder's remaining starters on
essentially every admin boot (live gates: 1A 2/3 boots, 1B 4/4 boots) and
self-heals at the memoized Dialer/PskProvider properties' first use. Before
this change that produced two FALSE Errors per healthy boot, which made a
soak error-grep on a healthy rig impossible.
Resolve() now takes a `firstUse` flag and logs at Debug on BOTH call sites
(PreStart and the memoized properties' retry) — from inside Resolve a lost
race and a genuinely broken factory are indistinguishable, so severity is
decided by the CONSUMER of a still-null result after the retry, not by
Resolve itself:
- Dialer: DispatchOverGrpc's existing per-dispatch Error ("no command
dialer is wired") is unchanged and is now the ONLY loud signal for that
dependency.
- PskProvider: PublishKnownClusters gained a new Error carrying the old
PreStart guard's diagnostic content (gate never published, cached keys
never dropped, rotated key needs restart), guarded to fire ONCE per
actor incarnation via `_pskProviderAbsenceReported` — that handler runs
on the contact-refresh timer for the actor's whole life, so an
unguarded Error there would repeat every ContactRefreshSeconds forever.
Design choice for the "factory never supplied" case (Dps/ClusterClient test
constructions that pass no factories at all): kept silent at BOTH call
sites for both dependencies, same as before. For the dialer this was
already true (DispatchOverGrpc's check doesn't care why the dialer is
null). For the PSK provider the new once-guarded Error is gated on
`_pskProviderFactory is not null` specifically so the many existing
ClusterClient/Dps facts in CentralCommunicationActorTests that never wire a
PSK provider (using the bare Spawn helper) keep passing unmodified — a
never-supplied factory is a deliberate wiring/test choice, not a resolve
failure, and Resolve() already stayed silent about it for the same reason.
Two new tests pin the net behavior: a healthy boot (factories that throw
once then succeed) produces zero Warning/Error end-to-end through PreStart,
the first contact load and the first dispatch; a permanently-broken PSK
provider logs the absence exactly once across two contact loads, not once
per refresh.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Node→central ApplyAck over gRPC behind MeshTransport:Mode=Grpc: CentralCommandGrpcService
on the AdminUI listener, GrpcApplyAckSender with per-endpoint ack deadlines (Critical fix
e54711a2 proven live in the 4b hang drill), NodeCommunicationActor wiring, plus the
phases 2-4 plan update (targeted-test policy + Task 2.0 must-fix).
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Node→central ApplyAcks over gRPC, gated live on the docker-dev rig at
4a2a3d3a. All 7 checks PASS (the 6 planned plus the mandated 4b hang
drill).
The 4b drill is the one that earns its place: with central-1 PAUSED
rather than stopped, both site-a nodes burned a full AckTimeoutSeconds
on it as DeadlineExceeded and still reached central-2 with a live
deadline — the per-endpoint FRESH deadline from 667639e6, proven on the
failure shape a hoisted deadline could not survive. Observed failover
latency ~10 s, exactly one AckTimeoutSeconds.
It also produced the run's only duplicate ack, and that turned into
evidence rather than a defect: the frozen central had in fact received
both acks and replayed them on unpause, and the coordinator reported
2/6 acks landed, not 4/6 — the idempotent absorption the sender's
at-least-once remark asserts, observed instead of argued.
Coexistence is per pair in one deployment: site-a's rows are Applied
with matching ReportApply lines on central and zero receptionist
associations, site-b's are Applied with zero ReportApply lines and a
live receptionist association. Neither pair could have used the other's
transport.
Carry-forwards recorded: the I-4 cold-PSK-cache latency is structurally
unmeasurable on docker-dev (ClusterPsks is read ahead of the store, so
InvalidateAll clears an empty cache) and moves to a store-backed rig;
a restarted central's cached channel can stay in backoff ~2 min on the
node that watched it die, absorbed by the walk with no ack lost. The
1A finding-5 wording fix gains a second confirmation — 4 of 4 admin
boots lost the PreStart race, and every ack central-1 accepted proves
the provider it said would NEVER be published was.
Docs only; the rig flip and the four Serilog Debug overrides were
reverted and are not in this diff. Rig restored to six healthy,
all-ClusterClient, sealing 6/6.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
CRITICAL. The ack deadline was one absolute DateTime computed once for the whole
endpoint walk. Against the characteristic WAN failure — a central that HANGS
(SYN black-hole, frozen host) rather than refusing — endpoint 1 burned the whole
AckTimeoutSeconds and endpoint 2 was then dialled with a deadline already in the
past: instant DeadlineExceeded, no packet leaving the node, the surviving central
never really tried. Failover looked like it ran and did not. It also contradicted
MeshTransportOptions' own documented "applied per endpoint, not per ack". The
deadline is now computed fresh inside the loop; headers stay hoisted (read-only,
sequential walk).
Two test gaps that let that survive are closed: nothing pinned the ackRouter gate
(narrowing it back to ClusterClient-only compiled and passed every test while
dead-ending every Grpc ack to DPS), and the deadline fact's lower bound accepted
any positive timeout at all. Both new pins are mutation-verified.
Docs corrected where they overstated the case: duplicate acks CANNOT mis-seal a
deployment — ConfigPublishCoordinator.HandleAck keys _acks by NodeId behind an
expected-set guard, so a duplicate overwrites its own entry. The single-endpoint
rule is defence in depth and a bound on wasted work, not a correctness
dependency; and the walk is at-least-once on the deadline path, not at-most-once.
Task 1B.2 — the node→central half of the ClusterClient→gRPC migration.
GrpcApplyAckSender walks MeshTransport:CentralGrpcEndpoints in order and
STOPS AT THE FIRST SUCCESS. That single `return` is the whole single-count
guarantee: the transport it replaces was ClusterClient.Send, deliberately not
SendToAll, because both central nodes forward to the SAME
ConfigPublishCoordinator singleton proxy — an ack posted to both would have the
coordinator count one node twice and could seal a deployment on a doubled count.
The list is an availability list, not a fan-out list.
No retry beyond that one ordered pass, no queue: the receptionist buffered
nothing and retried nothing, and this is like-for-like, not an upgrade to
delivery guarantees. All endpoints failing is still a DROP — but a loud one,
naming the node, deployment, correlation id and the status each endpoint
answered with, which is what separates "central is down" from "central rejected
this node's key". The bearer never reaches a log; only endpoints and status
codes do. Per-endpoint GrpcChannel cache and the endpoint loop clone
GrpcDeploymentArtifactFetcher; the cluster id in x-otopcua-cluster comes from
this node's own cluster-<id> role via RoleParser, off the same bound
Cluster:Roles ClusterRoleInfo reads.
SendAsync never throws: the caller discards the task, so an escaping exception
would become an unobserved fault — a drop with no log line, the exact failure
mode this migration removes.
NodeCommunicationActor gains an OPTIONAL ackSender; every existing test compiles
and passes unedited. It is checked before the ClusterClient so the two can never
both fire. The registration arm creates no ClusterClient under Grpc.
NOTE — one wiring change beyond the actor: DriverHostActor's ackRouter is now
`useClusterClient || useGrpc ? nodeComm : null`. Left at ClusterClient-only, a
Grpc node's ack would have gone out on the DPS deployment-acks topic and never
reached the sender at all.
Two review items from Task 1B.1, both diagnostic:
1. CentralCommandGrpcService's "actor not ready" Warning omitted the peer and
the correlation id while the decode-failure Warning six lines above carried
both. These are the only two ways an ack fails to reach the coordinator, and
an operator has to correlate either against the node's own log. Decoding has
already succeeded on this path, so the DECODED correlation id is free.
2. A fused driver,admin,cluster-MAIN node under Mode=Grpc owes BOTH validator
surfaces at once (driver keys AND ConfigServe:GrpcListenPort). The two blocks
are independent `if`s; chained as `else if` a fused central node would be
graded on one surface only and boot with nowhere for any ack to land. Pinned
from both sides plus the fully-configured pass.
Re-ran the 1A gate on the docker-dev rig at e54883a8 after the defect-4 boot
fix. Image rebuilt (otopcua-host:dev -> 1d8452a29018); six host nodes plus
both one-shots recreated; run 1's preserved image was stale and deliberately
not reused. VERDICT: PASS.
Boot: 6/6 nodes healthy INCLUDING the central pair, 0 Hosting failed to start,
/health/ready 200 on every node, MAIN mesh 2/2.
Check 1 PASS: one DispatchDeployment, one correlation id, accepted by all six
nodes; central logs 6 accepted / 0 failed; zero ClusterClient lines on either
central node with Akka.Cluster.Tools.Client at Debug.
Check 2 PASS: deployment Sealed with all six NodeDeploymentState rows Applied
inside ~90 ms, while the four site nodes hold receptionist associations — the
only ack path they have in 1A. Mixed transport proven.
Check 4 PASS: Ack clicked on the real /alerts page against a live ACTIVATED
row; central fanned one AlarmCommand to all six over gRPC and the condition
acknowledged 0.54 s later carrying the operator identity.
Check 7 PASS: with site-b-2's GrpcPort NULLed, both central nodes warned once
and the next dispatch reached exactly five nodes; the deployment then honestly
ended TimedOut with site-b-2 alone unapplied. Row restored, 6/6 sealed after.
Check 8 PASS and no longer vacuous: 0 denials on all four interceptors, and
this time config-serve (28 KB verified artifact fetches from :4055) and
telemetry (streams open on all six, live driver health on /hosts) both carried
real traffic.
Check 3 closed by composition, with the reason now established rather than
assumed: <DriverStatusPanel has 0 usages repo-wide, no rendered page carries a
Restart/Reconnect control, and the only non-script-analysis POST is
/api/deployments. Run 1 drove the node-side leg for both commands; check 4
drives the identical AdminOperationsActor.Dispatch -> DispatchOverGrpc origin
path from the AdminUI.
Checks 5 + 6 stand from run 1 — nothing in the fix touches the interceptor.
New finding 5, recorded because it is load-bearing for the fix's design: the
PreStart resolve LOSES its race on real boots — 2 of 3 observed admin boots
logged Failed to resolve for both dependencies with the same
AuditWriterActorKey cause. The memoize-on-first-use hardening is therefore not
theoretical, and its heal was proven live by moving the coordinator singleton
onto the node that lost the race (stop central-2) and watching central-1 fan a
dispatch to all five live nodes with no dialer error. Residual is cosmetic and
carried to Phase 2: the Error text reads as permanent, and it will trip a soak
error-grep on a healthy node.
Defect 4 marked CLOSED; run 1's record kept in full underneath the combined
table. Rig restored: flip reverted, six nodes healthy on all-ClusterClient on
the committed image, three mesh ClusterClients back on each central node.
Phase 1A live gate defect 4. Neither central node booted on this branch,
in ANY transport mode:
Akka.Hosting.MissingActorRegistryEntryException: No actor registered
for key ...ControlPlane.AuditWriterActorKey
at AdminUI.Audit.ActorAuditWriter..ctor(ActorRegistry registry)
at ControlPlane.ServiceCollectionExtensions.<WithOtOpcUaControlPlaneSingletons>
WithOtOpcUaControlPlaneSingletons' first WithActors starter resolved
IMeshCommandDialer and IMeshPskProvider eagerly for the Props. Their
graph reaches back into the registry — dialer -> IMeshPskProvider ->
ISecretResolver -> AddZbSecrets -> IAuditWriter -> ActorAuditWriter,
whose ctor calls registry.Get<AuditWriterActorKey>() — and
WithSingleton<AuditWriterActorKey> is registered ~60 lines LATER in the
same builder. Boot died before the entry existed. Mode-independent: the
resolve happened before anything consulted MeshTransport:Mode, so Dps
(the default, and the rollback target) died exactly like Grpc.
Both dependencies now arrive as Func<> and are invoked in PreStart, by
which point the host is fully built — the same shape, for the same
class of reason, as the coordinator proxy Func one argument earlier.
PreStart re-resolves on restart rather than caching across one; both
are container singletons, so a restart gets the same instances. A
resolution throw is caught and logged rather than propagated: a throw
out of PreStart is an actor failure, the supervisor answers it with a
restart, and a broken DI graph is not transient — that is an unbounded
restart loop that would take the transport-independent ack path down
with it.
Regression pin: ControlPlaneStarterOrderTests boots a REAL admin host
through WithOtOpcUaClusterBootstrap + WithOtOpcUaControlPlaneSingletons
in production starter order, with the production graph from
IMeshCommandDialer down to a registry-bound IAuditWriter, and asserts
StartAsync itself — the failure occurs while the builder's starters
execute, so building a ServiceProvider and resolving by hand cannot
reproduce it. Verified RED against the eager resolve: 3/3 modes fail
with MissingActorRegistryEntryException. A second fact pins that the
deferral did not become 'never resolved' — the actor must resolve the
provider from the container and drive it through the contact refresh,
signalled by a TaskCompletionSource (PreStart runs on the actor thread
and the contact load is a piped DB read, so no synchronous assertion
after StartAsync is correct).
Why nothing caught this: the dialer's container-activation test resolves
from a collection with no secrets registration, so its graph stops one
hop in; the actor facts inject fakes and never touch DI; and the
existing whole-host boots register no mesh provider, so the eager
lookups returned null and walked nothing.
Two aggravators recorded by the gate, no code here: the failed process
did not exit (docker Up while /health/ready 503s — Phase 2 readiness
concern, tracked), and this is the third instance of the same
eager-resolve-during-AddAkka class after scadaproj#1 and mesh Phase 6's
IClusterRoleInfo boot crash.
ControlPlane.Tests 236 -> 240, Host.Tests 92, Cluster.Tests 184,
0 warnings in every touched project.
Ran the 1A live gate on the docker-dev rig at fbf7b708 (image rebuilt from
this worktree, all eight containers recreated). VERDICT: FAIL.
Both central nodes fail to boot on this branch with
MissingActorRegistryEntryException: AuditWriterActorKey — the first WithActors
starter eagerly resolves IMeshCommandDialer/IMeshPskProvider, whose graph
reaches ActorAuditWriter, whose ctor reads an ActorRegistry key registered ~60
lines later in the same builder. Reproduced on the same image with
MeshTransport__Mode=ClusterClient, so it is mode-independent and not caused by
the gate's rig-only flip. Recorded as defect 4; no product change made.
Checks 5 (auth negative/positive on real Kestrel h2c) and 6 (wrong-key
isolation, both directions, with a positive control) PASS. Checks 3 and 4 pass
their node-side leg only, driven with real RestartDriver/ReconnectDriver/
AlarmCommand envelopes that restarted and reconnected SITEA-modbus. Checks 1,
2 and 7 are blocked outright — no central, no dispatch, no contact refresh —
and check 8 is vacuous for telemetry + config-serve because central dials both
and never booted. All five gaps are named in the carry-forward register and
must be closed by a 1A gate re-run, not carried past the phase.
Issues filed by this task: lmxopcua#544 (ConfigPublishCoordinatorTimeoutTests
deflake) and lmxopcua#545 (mesh planes are h2c-only — accept the risk or add a
UseTls option); the tracking table's two placeholder rows now carry them.
Also recorded: DriverStatusPanel.razor is referenced by no page or component,
so the AdminUI Reconnect/Restart controls are unreachable on this branch and on
master — pre-existing, out of program scope, carried forward.
Rig restored: compose flip reverted, six nodes healthy on all-ClusterClient.
The rig runs the PRE-gate image (the 1A build is kept as otopcua-host:grpc-1a);
a re-run must rebuild.
The known-cluster gate published only under Mode=Grpc, but the
interceptor it feeds is registered on every admin node and 1B maps
CentralCommandService by LISTENER PORT — neither consults the mode. A
dark-switch rollback to ClusterClient or Dps would therefore have left
that endpoint serving with the gate permanently inactive, silently
restoring both the timing oracle and the pre-auth secret-store/audit
flood, as a side effect of a rollback nobody would connect to
authentication. The gate is central's cluster inventory, not transport
state, so it is now decoupled from the mode entirely:
- SetKnownClusters + InvalidateAll run on EVERY successful contact load
in EVERY mode, and the contact-refresh timer starts in all three.
Under Dps this is a ConfigDb query every ContactRefreshSeconds that
did not run before; central always has a ConfigDb, and one indexed
read a minute is the cheaper half of the trade.
- The GATE set is built from ALL ClusterNode rows. The eligibility
filter moved out of the query's WHERE and into the loop, so the FAN
set is byte-for-byte what it was (enabled && !maintenance) while the
gate widens. This also dissolves the maintenance coupling flagged in
1A.2: a cluster whose nodes are all disabled or in maintenance can
still ack, and refusing it would make maintenance mode an
authentication switch. Central still refuses clusters it has never
heard of.
- HandleContactsLoaded keeps a documented order — targets adopted, gate
published, cache dropped — now pinned by test.
Guards: PreStart mirrors the null-dialer error for a null PSK provider
(silent absence is the worst outcome there — nothing malfunctions, so
nothing is investigated, while the gate never publishes and keys never
rotate); a failed contact load now says out loud that the gate is stale
and, before any load has succeeded, inactive; MeshGrpcCommandDialer
clamps a non-positive DispatchTimeoutSeconds to 10 with a construction
Warning, because the validator for that key is driver-role-scoped and a
pure-admin central reads it unvalidated; the dialer's local PSK refusal
was shortened so its remediation survives the actor's 200-char
truncation of remote-authored status detail; the portless-node warn-once
set is rebuilt each refresh, so a node that regains then loses a port
warns again — the regression is the occurrence worth seeing.
Docs/pins: dialer disposal comment no longer claims one-way; the two
interceptor-test comments claiming Matches has a default implementation
corrected (it deliberately has none); new pins for per-refresh
invalidation, the SetKnownClusters-before-InvalidateAll order, and one
channel per endpoint via a new CachedChannelCount.
The_known_cluster_set_is_not_published_by_the_other_two_modes is
replaced — its premise is obsolete — by three facts: every mode
publishes, gate set is a strict superset of the fan set, and the handler
ordering.
ControlPlane.Tests 230 -> 236, Host.Tests 92, Cluster.Tests 184,
0 warnings.
MeshGrpcCommandDialer places one NodeCommandService.Execute call on one
node: a GrpcChannel cached per endpoint (the GrpcDeploymentArtifactFetcher
shape), the TARGET node's cluster's bearer plus x-otopcua-cluster naming
that cluster, and a per-call deadline linked to the caller's token. A
cluster with no resolvable key is refused locally with Unauthenticated —
there is no arm on which a command leaves central unauthenticated, even
one the far side would have rejected.
CentralCommunicationActor gains the third mode branch. The contact load
now also projects GrpcPort and, from the SAME rows in the SAME pass,
builds the dispatch set — which is what makes 'the gRPC fan reaches
exactly the SendToAll recipient set' structural rather than two queries
that have to keep agreeing. A row with a null GrpcPort is skipped
warn-once (the TelemetryNodeSource precedent). Per-node results are
piped back for LOGGING ONLY: named node, gRPC status, dropped. No retry,
no queue.
Every dial is scheduled on the thread pool rather than called from the
receive. Found by review and pinned red-first: an async method runs
synchronously to its first await, and the dialer's pre-await work is
TryGetPsk, which blocks on the secret store — bounded but seconds, and
repeated after every refresh since the refresh drops the cache. Dialing
inline parked the actor's mailbox, so deployment acks queued behind a
slow node; the regression test tells an ApplyAck while a dial is held
and fails without the hop.
The contact refresh also wires the Phase 1A register: it publishes the
loaded cluster ids as the PSK provider's known-cluster set and drops the
key cache, so a rotated key takes effect within one ContactRefreshSeconds.
Neither happens under Dps or ClusterClient.
Both existing mode branches are behaviourally untouched and every
existing fact passes unmodified. The one edit to shared text: the
Status.Failure warning drops the word 'ClusterClient', since that handler
now serves all three transports.
ControlPlane.Tests 205 -> 230, Host.Tests 92, Cluster.Tests 184, 0 warnings.
Phase 1A security register items 1 and 2, landed ahead of the command
fan-out because they are independently testable and the fan-out is the
first caller of TryGetPsk.
Item 1 — known-cluster gate. MeshPskProvider gains SetKnownClusters, an
atomic whole-set swap consulted by TryGetPsk BEFORE the config map and
before the secret store. A cluster id outside the set is refused in O(1)
with no store read and no audit write, which closes two things at once:
the pre-auth store/audit flood an unauthenticated caller could drive with
a forged x-otopcua-cluster header, and the timing oracle that defeated
CentralCommandAuthInterceptor's uniform denial text (unknown clusters are
never cached, so they paid a store read on every probe while known ones
answered from memory). Empty or never-published leaves the gate inactive
— central publishes the set from its DB contact refresh, and 'the DB told
me nothing' must not become a fleet-wide authentication outage during
admin boot.
Item 2 — bool Matches(clusterId, presented) on IMeshPskProvider, with
FixedTimeEquals inside the provider. CentralCommandAuthInterceptor now
calls it and never holds key material; its two provider-side denial logs
collapse into one (the provider answers both as false) and the operator
hint is restated there.
No production caller publishes a cluster set yet, so behaviour is
unchanged until the fan-out commit wires the contact refresh.
ControlPlane.Tests 190 -> 205, Host.Tests 91 -> 92.
Phase 0 task 0.6 of the ClusterClient->gRPC migration: wire the dark
MeshTransport Grpc-mode keys into the docker-dev rig (GrpcPsk +
CentralGrpcEndpoints on all six nodes, ClusterPsks on the central pair
only), document the new keys in docs/Configuration.md as shipped
dark/cutover pending, and record the Phase 0 live-gate evidence
(rig boot with no validator fires, two deliberate-misconfig refusals,
full suite green).
Files gitea.dohertylan.com/dohertj2/lmxopcua#542 (FleetStatusBroadcaster
mesh-blind post-Phase-6) and #543 (three pre-existing interceptors accept
a whitespace-only key as set) as follow-ups out of Phase 0's scope.
Joseph Doherty