MeshGrpcBoundaryTests replaces the MeshClusterClientBoundaryTests deleted in step 1. Three
self-seeded single-member clusters sharing the ActorSystem name, joined ONLY by gRPC, driving
the REAL CentralCommunicationActor -> MeshGrpcCommandDialer -> Kestrel h2c ->
MeshCommandAuthInterceptor -> NodeCommandGrpcService -> NodeCommunicationActor -> EventStream:
1. No_cross_mesh_Akka_association_forms_between_central_and_either_site — the program's own
point, kept as a test rather than a live-gate note: each cluster stays single-member and
none ever holds another's address as a member.
2. One_dispatch_through_central_reaches_a_node_in_every_cluster_mesh — the fan, with each
site holding its OWN per-cluster PSK, so a dial that selected the wrong cluster's key
would be refused at the far side.
3. Falsifiability_control_a_dial_carrying_the_wrong_cluster_key_delivers_nothing — without
it, (2) cannot distinguish "the gRPC boundary carried it" from "something else in the
process did". The same node then accepts the right key, so the silence is the auth gate
and not a dead endpoint.
All three run fully offline — no Docker fixture — and pass.
Comment sweep across every remaining clusterclient|receptionist hit in src/ and tests/:
the program label becomes "mesh gRPC command plane", and every comment that described the
deleted transport as live now describes what actually runs (Dps = pair-local default,
Grpc = cross-mesh). mesh_transport.proto's header/service comments are reworded; no message,
field or enum is touched — the contract is locked. MeshCommActorPathTests is reshaped rather
than retired: the two paths still exist and still matter, but what a mismatch costs is now a
misleading Unavailable from the gRPC ingress rather than a silently dropped receptionist send.
The only surviving mentions of the old names are the ones that must survive: the fail-closed
rules that name "ClusterClient" as a retired value refused at boot, their tests, and this
test class's own account of what it replaces.
MeshTransportOptions.ModeClusterClient is gone. MeshTransportOptionsValidator accepts Dps or
Grpc and nothing else, so a node left carrying the string "ClusterClient" is now REFUSED at
boot naming MeshTransport:Mode and both remaining values — deliberately fail-closed rather
than falling back to Dps, which on a split-topology node would deliver nothing at all. A new
fact pins exactly that.
SplitTopologyTransportValidator tightens from an allow-list of two to Grpc only, as its own
comment said it would once ClusterClient was deleted. Dps on a cluster-role node stays
refused; the rule is still an explicit allow-list, so a typo'd mode fails too.
Tests: the two ClusterClient-mode facts in MeshTransportOptionsValidatorTests are deleted
(their Dps siblings already assert the same rule for the one remaining non-Grpc mode), the
SplitTopology fixtures move to Grpc, and the three-mode Theories in
ControlPlaneStarterOrderTests / CentralCommunicationActorTests become two-mode.
IMeshClusterClientFactory/DefaultMeshClusterClientFactory are gone, along with
CentralCommunicationActor's _clients map, RebuildClient, the SendToAll fan and the
receptionist-address half of the contact load. LoadContactsFromDb SURVIVES — it feeds the
gRPC dialer's endpoint resolution and the known-cluster gate — but now projects only the
columns those two need (the AkkaPort/ActorPath.TryParse parse and its Malformed list went
with the receptionist addresses).
Ripples fixed in this step because the build must be green at every commit:
- ControlPlane/ServiceCollectionExtensions drops the factory argument.
- CentralCommunicationActorTests: the six ClusterClient-only facts are deleted (each has a
Grpc counterpart that already asserted the same property independently), the three
transport-agnostic facts move to Grpc mode, and the Dps fact keeps its own coverage.
- MeshClusterClientFactoryTests deleted (its subject is gone).
- MeshClusterClientBoundaryTests deleted here rather than in step 7: it drives
DefaultMeshClusterClientFactory directly, so it cannot compile past this commit. Its
replacement, MeshGrpcBoundaryTests, lands in step 7 and keeps the load-bearing assertion
(two separate meshes, no cross-mesh Akka association).
Follow-up to a94eb8a3, addressing CHANGES REQUESTED on code review. Production
logic was confirmed correct on all spec points; this commit fixes the review's
test/comment findings only.
IMPORTANT-1: A_permanently_unresolvable_pskProvider_is_reported_once_per_incarnation_not_once_per_refresh
did not prove the second refresh ran — its Expect(0) window gated on a dialer
dispatch landing, but LoadContactsFromDb is fire-and-forget and the first
block's AwaitCondition already re-Tells MeshCommand per poll, so the dialer
call count could reach the threshold before PublishKnownClusters was ever
re-invoked by the second load. Fixed by having the PSK factory count its own
invocations and gating both AwaitConditions on that count directly (pskCalls
>= 2 proves the first retry; pskCalls > pskCallsAfterFirstLoad, awaited INSIDE
the Expect(0) window, proves the second one). Bite-checked: with
_pskProviderAbsenceReported's guard temporarily removed, the test now
correctly fails ("Received 1 message too many... received 1 that matched
filter"); guard restored and reverified green.
MINOR-2: reworded four comments (CentralCommunicationActor.cs ~109, ~259,
~305-309, ~853) from narrating the change ("this fix removes", "already
existed, unchanged", "added by this fix", "the old two-call-site shape") to
stating the timeless invariant.
MINOR-3: added A_supplied_dialer_factory_that_always_throws_still_drops_each_command_loudly_while_PreStart_stays_quiet
— the symmetric case to the existing never-supplied-dialer test, exercising
Resolve's catch + firstUse:true path (a factory that WAS supplied and always
throws) rather than the null-factory path. Asserts PreStart plus the first
contact load stay silent and DispatchOverGrpc's existing per-dispatch Error
still fires on every dispatch, unguarded.
MINOR-4: "Check the ControlPlane registration wires IMeshPskProvider" ->
"Check that the ControlPlane registration wires up IMeshPskProvider"; no test
pinned the old substring, so no test update was needed for this one.
Skipped item 5 (restart-re-arms pin) per instruction — out of scope.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
PreStart's eager resolve of IMeshCommandDialer/IMeshPskProvider loses a
millisecond-wide race against the Akka builder's remaining starters on
essentially every admin boot (live gates: 1A 2/3 boots, 1B 4/4 boots) and
self-heals at the memoized Dialer/PskProvider properties' first use. Before
this change that produced two FALSE Errors per healthy boot, which made a
soak error-grep on a healthy rig impossible.
Resolve() now takes a `firstUse` flag and logs at Debug on BOTH call sites
(PreStart and the memoized properties' retry) — from inside Resolve a lost
race and a genuinely broken factory are indistinguishable, so severity is
decided by the CONSUMER of a still-null result after the retry, not by
Resolve itself:
- Dialer: DispatchOverGrpc's existing per-dispatch Error ("no command
dialer is wired") is unchanged and is now the ONLY loud signal for that
dependency.
- PskProvider: PublishKnownClusters gained a new Error carrying the old
PreStart guard's diagnostic content (gate never published, cached keys
never dropped, rotated key needs restart), guarded to fire ONCE per
actor incarnation via `_pskProviderAbsenceReported` — that handler runs
on the contact-refresh timer for the actor's whole life, so an
unguarded Error there would repeat every ContactRefreshSeconds forever.
Design choice for the "factory never supplied" case (Dps/ClusterClient test
constructions that pass no factories at all): kept silent at BOTH call
sites for both dependencies, same as before. For the dialer this was
already true (DispatchOverGrpc's check doesn't care why the dialer is
null). For the PSK provider the new once-guarded Error is gated on
`_pskProviderFactory is not null` specifically so the many existing
ClusterClient/Dps facts in CentralCommunicationActorTests that never wire a
PSK provider (using the bare Spawn helper) keep passing unmodified — a
never-supplied factory is a deliberate wiring/test choice, not a resolve
failure, and Resolve() already stayed silent about it for the same reason.
Two new tests pin the net behavior: a healthy boot (factories that throw
once then succeed) produces zero Warning/Error end-to-end through PreStart,
the first contact load and the first dispatch; a permanently-broken PSK
provider logs the absence exactly once across two contact loads, not once
per refresh.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Phase 1A live gate defect 4. Neither central node booted on this branch,
in ANY transport mode:
Akka.Hosting.MissingActorRegistryEntryException: No actor registered
for key ...ControlPlane.AuditWriterActorKey
at AdminUI.Audit.ActorAuditWriter..ctor(ActorRegistry registry)
at ControlPlane.ServiceCollectionExtensions.<WithOtOpcUaControlPlaneSingletons>
WithOtOpcUaControlPlaneSingletons' first WithActors starter resolved
IMeshCommandDialer and IMeshPskProvider eagerly for the Props. Their
graph reaches back into the registry — dialer -> IMeshPskProvider ->
ISecretResolver -> AddZbSecrets -> IAuditWriter -> ActorAuditWriter,
whose ctor calls registry.Get<AuditWriterActorKey>() — and
WithSingleton<AuditWriterActorKey> is registered ~60 lines LATER in the
same builder. Boot died before the entry existed. Mode-independent: the
resolve happened before anything consulted MeshTransport:Mode, so Dps
(the default, and the rollback target) died exactly like Grpc.
Both dependencies now arrive as Func<> and are invoked in PreStart, by
which point the host is fully built — the same shape, for the same
class of reason, as the coordinator proxy Func one argument earlier.
PreStart re-resolves on restart rather than caching across one; both
are container singletons, so a restart gets the same instances. A
resolution throw is caught and logged rather than propagated: a throw
out of PreStart is an actor failure, the supervisor answers it with a
restart, and a broken DI graph is not transient — that is an unbounded
restart loop that would take the transport-independent ack path down
with it.
Regression pin: ControlPlaneStarterOrderTests boots a REAL admin host
through WithOtOpcUaClusterBootstrap + WithOtOpcUaControlPlaneSingletons
in production starter order, with the production graph from
IMeshCommandDialer down to a registry-bound IAuditWriter, and asserts
StartAsync itself — the failure occurs while the builder's starters
execute, so building a ServiceProvider and resolving by hand cannot
reproduce it. Verified RED against the eager resolve: 3/3 modes fail
with MissingActorRegistryEntryException. A second fact pins that the
deferral did not become 'never resolved' — the actor must resolve the
provider from the container and drive it through the contact refresh,
signalled by a TaskCompletionSource (PreStart runs on the actor thread
and the contact load is a piped DB read, so no synchronous assertion
after StartAsync is correct).
Why nothing caught this: the dialer's container-activation test resolves
from a collection with no secrets registration, so its graph stops one
hop in; the actor facts inject fakes and never touch DI; and the
existing whole-host boots register no mesh provider, so the eager
lookups returned null and walked nothing.
Two aggravators recorded by the gate, no code here: the failed process
did not exit (docker Up while /health/ready 503s — Phase 2 readiness
concern, tracked), and this is the third instance of the same
eager-resolve-during-AddAkka class after scadaproj#1 and mesh Phase 6's
IClusterRoleInfo boot crash.
ControlPlane.Tests 236 -> 240, Host.Tests 92, Cluster.Tests 184,
0 warnings in every touched project.
The known-cluster gate published only under Mode=Grpc, but the
interceptor it feeds is registered on every admin node and 1B maps
CentralCommandService by LISTENER PORT — neither consults the mode. A
dark-switch rollback to ClusterClient or Dps would therefore have left
that endpoint serving with the gate permanently inactive, silently
restoring both the timing oracle and the pre-auth secret-store/audit
flood, as a side effect of a rollback nobody would connect to
authentication. The gate is central's cluster inventory, not transport
state, so it is now decoupled from the mode entirely:
- SetKnownClusters + InvalidateAll run on EVERY successful contact load
in EVERY mode, and the contact-refresh timer starts in all three.
Under Dps this is a ConfigDb query every ContactRefreshSeconds that
did not run before; central always has a ConfigDb, and one indexed
read a minute is the cheaper half of the trade.
- The GATE set is built from ALL ClusterNode rows. The eligibility
filter moved out of the query's WHERE and into the loop, so the FAN
set is byte-for-byte what it was (enabled && !maintenance) while the
gate widens. This also dissolves the maintenance coupling flagged in
1A.2: a cluster whose nodes are all disabled or in maintenance can
still ack, and refusing it would make maintenance mode an
authentication switch. Central still refuses clusters it has never
heard of.
- HandleContactsLoaded keeps a documented order — targets adopted, gate
published, cache dropped — now pinned by test.
Guards: PreStart mirrors the null-dialer error for a null PSK provider
(silent absence is the worst outcome there — nothing malfunctions, so
nothing is investigated, while the gate never publishes and keys never
rotate); a failed contact load now says out loud that the gate is stale
and, before any load has succeeded, inactive; MeshGrpcCommandDialer
clamps a non-positive DispatchTimeoutSeconds to 10 with a construction
Warning, because the validator for that key is driver-role-scoped and a
pure-admin central reads it unvalidated; the dialer's local PSK refusal
was shortened so its remediation survives the actor's 200-char
truncation of remote-authored status detail; the portless-node warn-once
set is rebuilt each refresh, so a node that regains then loses a port
warns again — the regression is the occurrence worth seeing.
Docs/pins: dialer disposal comment no longer claims one-way; the two
interceptor-test comments claiming Matches has a default implementation
corrected (it deliberately has none); new pins for per-refresh
invalidation, the SetKnownClusters-before-InvalidateAll order, and one
channel per endpoint via a new CachedChannelCount.
The_known_cluster_set_is_not_published_by_the_other_two_modes is
replaced — its premise is obsolete — by three facts: every mode
publishes, gate set is a strict superset of the fan set, and the handler
ordering.
ControlPlane.Tests 230 -> 236, Host.Tests 92, Cluster.Tests 184,
0 warnings.
MeshGrpcCommandDialer places one NodeCommandService.Execute call on one
node: a GrpcChannel cached per endpoint (the GrpcDeploymentArtifactFetcher
shape), the TARGET node's cluster's bearer plus x-otopcua-cluster naming
that cluster, and a per-call deadline linked to the caller's token. A
cluster with no resolvable key is refused locally with Unauthenticated —
there is no arm on which a command leaves central unauthenticated, even
one the far side would have rejected.
CentralCommunicationActor gains the third mode branch. The contact load
now also projects GrpcPort and, from the SAME rows in the SAME pass,
builds the dispatch set — which is what makes 'the gRPC fan reaches
exactly the SendToAll recipient set' structural rather than two queries
that have to keep agreeing. A row with a null GrpcPort is skipped
warn-once (the TelemetryNodeSource precedent). Per-node results are
piped back for LOGGING ONLY: named node, gRPC status, dropped. No retry,
no queue.
Every dial is scheduled on the thread pool rather than called from the
receive. Found by review and pinned red-first: an async method runs
synchronously to its first await, and the dialer's pre-await work is
TryGetPsk, which blocks on the secret store — bounded but seconds, and
repeated after every refresh since the refresh drops the cache. Dialing
inline parked the actor's mailbox, so deployment acks queued behind a
slow node; the regression test tells an ApplyAck while a dial is held
and fails without the hop.
The contact refresh also wires the Phase 1A register: it publishes the
loaded cluster ids as the PSK provider's known-cluster set and drops the
key cache, so a rotated key takes effect within one ContactRefreshSeconds.
Neither happens under Dps or ClusterClient.
Both existing mode branches are behaviourally untouched and every
existing fact passes unmodified. The one edit to shared text: the
Status.Failure warning drops the word 'ClusterClient', since that handler
now serves all three transports.
ControlPlane.Tests 205 -> 230, Host.Tests 92, Cluster.Tests 184, 0 warnings.
Phase 1A security register items 1 and 2, landed ahead of the command
fan-out because they are independently testable and the fan-out is the
first caller of TryGetPsk.
Item 1 — known-cluster gate. MeshPskProvider gains SetKnownClusters, an
atomic whole-set swap consulted by TryGetPsk BEFORE the config map and
before the secret store. A cluster id outside the set is refused in O(1)
with no store read and no audit write, which closes two things at once:
the pre-auth store/audit flood an unauthenticated caller could drive with
a forged x-otopcua-cluster header, and the timing oracle that defeated
CentralCommandAuthInterceptor's uniform denial text (unknown clusters are
never cached, so they paid a store read on every probe while known ones
answered from memory). Empty or never-published leaves the gate inactive
— central publishes the set from its DB contact refresh, and 'the DB told
me nothing' must not become a fleet-wide authentication outage during
admin boot.
Item 2 — bool Matches(clusterId, presented) on IMeshPskProvider, with
FixedTimeEquals inside the provider. CentralCommandAuthInterceptor now
calls it and never holds key material; its two provider-side denial logs
collapse into one (the provider answers both as false) and the operator
hint is restated there.
No production caller publishes a cluster set yet, so behaviour is
unchanged until the fan-out commit wires the contact refresh.
ControlPlane.Tests 190 -> 205, Host.Tests 91 -> 92.
After the fleet splits into one Akka mesh per application Cluster, a
ClusterClientReceptionist serves only its own mesh — so the single
fleet-wide client's SendToAll reached only the mesh whose receptionist
answered, leaving every other cluster silently on its old configuration.
Central now holds one ClusterClient per ClusterId and fans SendToAll
across all of them. LoadContactsFromDb selects ClusterId and groups the
receptionist contacts per cluster (keeping the per-row TryParse guard and
the enabled/non-maintenance filter); ContactsLoaded carries
ContactsByCluster; HandleContactsLoaded diffs per cluster (rebuild changed,
stop+drop vanished, warn-don't-create on empty); RebuildClient builds a
per-cluster client under a clusterId-derived actor name. IMeshClusterClientFactory.Create
gained a clusterId parameter so the per-cluster clients get distinct,
diagnosable names. ApplyAck handling and the DPS branch are unchanged; the
deploy path stays payload-free.
Tests: focused unit coverage of the grouping + fan-out (one-client-per-cluster,
fan-out SendToAll to every cluster client) via the recording factory double,
plus the real two-mesh boundary test extended to central + two separate site
meshes proving one dispatch reaches both and a client scoped to one cluster
never crosses into another. Red-before-green verified: crippling the fan-out
to a single client fails the reaches-both-meshes assertion.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 2 Task 7. The five central->node publish sites -- ConfigPublishCoordinator's
deploy dispatch, and AdminOperationsActor's restart, reconnect, acknowledge and
shelve -- now hand a MeshCommand to the transport router instead of telling the
DistributedPubSub mediator directly. Neither actor knows which transport is in
force any more.
_meshRouter is NULLABLE with a mediator fallback, and that is deliberate: about
a dozen existing coordinator and admin-operations tests construct these actors
without a router, and they must keep exercising the real DPS path rather than a
silently-disabled one. The fallback is not a permanent seam -- Phase 6 deletes
it with the DPS branch and the single mesh.
Wiring note: the comm actor's WithActors block moved ABOVE the singleton
registrations so both singletons can resolve it with registry.Get, which is the
ordering-by-registration idiom AdminOperationsActor already uses for the
coordinator. An earlier version used ActorSelection.ResolveOne().GetAwaiter()
.GetResult() inside the props factory -- that blocks inside actor construction
and races the very spawn it waits for.
Sabotage-verified by inverting the branch so the mediator always wins: all six
new tests go red, including the fallback test (which proves it is asserting the
mediator path rather than passing by accident).
Suites after: ControlPlane 118/118, Runtime 450/450, Host.IntegrationTests
196/202 -- sole failure AbCip_Green_AgainstSim, the fixture baseline that fails
on master too.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 2 Task 3. Central-side end of the boundary: routes every central->node
command out over DPS or ClusterClient per MeshTransport:Mode, and forwards
inbound ApplyAcks to the deploy coordinator singleton (Forward, not Tell, so
the coordinator sees the node rather than this relay).
Three things here are load-bearing and easy to get wrong later:
SendToAll, never Send. Today's DPS publish reaches EVERY DriverHostActor and
the node side has no ClusterId or node filter to compensate -- scoping happens
later, inside the artifact. ClusterClient.Send delivers to exactly ONE
registered actor, so it would deploy to a single node while every other node
silently kept its old config, and the deployment could still seal green.
Sabotage-verified: swapping to Send reddens exactly that one test.
Exactly ONE ClusterClient, fleet-wide. A receptionist serves its whole cluster,
so SendToAll reaches every registered node-comm actor in the mesh regardless of
which contact point was dialled. One client per application Cluster -- the
shape Phase 6 wants -- would fan each command out once per cluster while the
fleet is still one mesh: N x duplicate DispatchDeployment and ApplyAck. Marked
TODO(Phase 6).
The contact set does NOT scope delivery. Excluding a maintenance-mode node from
the contacts does not stop it receiving commands; it still receives them, as it
does under today's broadcast. The filter is about which receptionists are worth
dialling, not about who gets the message. Stated in the code because the
opposite is the natural assumption.
Also carries the sister project's shipped fixes: per-row address parsing so one
malformed ClusterNode cannot abort the refresh and leave the contact set
half-built (regression test orders the bad row FIRST), the cache entry cleared
before recreate so a failed create cannot leave commands routing into a
stopping actor, and a Status.Failure handler so a DB outage is a Warning rather
than a silent debug-level unhandled message.
Two test-harness bugs found and fixed while writing this, both mine: SubscribeAck
goes to the SENDER (so the mediator subscribe needed an explicit sender), and
EventFilter matches case-INSENSITIVELY, so a "was NOT" filter also caught this
actor's own "was not created" warning.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Phase 2 Task 2. Two small pieces the comm actors are built on.
MeshCommand carries a command plus the DPS topic it would have been published
on, so the admin singletons stop knowing which transport is in force. Without
it the dark switch would have to be threaded through five publish sites across
two actors, each free to drift. Central-side only -- it never crosses the wire,
so it carries no serialization contract.
DefaultMeshClusterClientFactory appends a generation counter to the actor name.
Context.Stop is asynchronous and the name stays reserved until termination
completes, so recreating the same name inside one message handler throws
InvalidActorNameException -- which is exactly what a contact-set change does
(stop the old client, create the replacement, one handler). Ported from the
sister project, where it was a shipped bug fix rather than foresight.
Sabotage-verified: freezing the name to a constant reddens both name tests with
InvalidActorNameException.
Dropped a third test I had written -- it asserted ClusterClientSettings
directly and never touched the factory, so it was coverage theatre. Replaced
with a comment saying where contact propagation IS covered (the Task 8 boundary
test, the only place it can actually fail).
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Joseph Doherty