b360239e5eba37b8c521541fcefb5917fbf56c48
6 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Joseph Doherty
|
c0475f2df4 |
fix(mesh): resolve dialer + PSK provider lazily — central boots again
Phase 1A live gate defect 4. Neither central node booted on this branch,
in ANY transport mode:
Akka.Hosting.MissingActorRegistryEntryException: No actor registered
for key ...ControlPlane.AuditWriterActorKey
at AdminUI.Audit.ActorAuditWriter..ctor(ActorRegistry registry)
at ControlPlane.ServiceCollectionExtensions.<WithOtOpcUaControlPlaneSingletons>
WithOtOpcUaControlPlaneSingletons' first WithActors starter resolved
IMeshCommandDialer and IMeshPskProvider eagerly for the Props. Their
graph reaches back into the registry — dialer -> IMeshPskProvider ->
ISecretResolver -> AddZbSecrets -> IAuditWriter -> ActorAuditWriter,
whose ctor calls registry.Get<AuditWriterActorKey>() — and
WithSingleton<AuditWriterActorKey> is registered ~60 lines LATER in the
same builder. Boot died before the entry existed. Mode-independent: the
resolve happened before anything consulted MeshTransport:Mode, so Dps
(the default, and the rollback target) died exactly like Grpc.
Both dependencies now arrive as Func<> and are invoked in PreStart, by
which point the host is fully built — the same shape, for the same
class of reason, as the coordinator proxy Func one argument earlier.
PreStart re-resolves on restart rather than caching across one; both
are container singletons, so a restart gets the same instances. A
resolution throw is caught and logged rather than propagated: a throw
out of PreStart is an actor failure, the supervisor answers it with a
restart, and a broken DI graph is not transient — that is an unbounded
restart loop that would take the transport-independent ack path down
with it.
Regression pin: ControlPlaneStarterOrderTests boots a REAL admin host
through WithOtOpcUaClusterBootstrap + WithOtOpcUaControlPlaneSingletons
in production starter order, with the production graph from
IMeshCommandDialer down to a registry-bound IAuditWriter, and asserts
StartAsync itself — the failure occurs while the builder's starters
execute, so building a ServiceProvider and resolving by hand cannot
reproduce it. Verified RED against the eager resolve: 3/3 modes fail
with MissingActorRegistryEntryException. A second fact pins that the
deferral did not become 'never resolved' — the actor must resolve the
provider from the container and drive it through the contact refresh,
signalled by a TaskCompletionSource (PreStart runs on the actor thread
and the contact load is a piped DB read, so no synchronous assertion
after StartAsync is correct).
Why nothing caught this: the dialer's container-activation test resolves
from a collection with no secrets registration, so its graph stops one
hop in; the actor facts inject fakes and never touch DI; and the
existing whole-host boots register no mesh provider, so the eager
lookups returned null and walked nothing.
Two aggravators recorded by the gate, no code here: the failed process
did not exit (docker Up while /health/ready 503s — Phase 2 readiness
concern, tracked), and this is the third instance of the same
eager-resolve-during-AddAkka class after scadaproj#1 and mesh Phase 6's
IClusterRoleInfo boot crash.
ControlPlane.Tests 236 -> 240, Host.Tests 92, Cluster.Tests 184,
0 warnings in every touched project.
|
||
|
Joseph Doherty
|
86633f140e | test(mesh): pin the Dps no-ClusterClient guard; bound the channel-share test's blackhole worst case | ||
|
Joseph Doherty
|
48dc2120c6 |
fix(mesh): gate follows the endpoint not the mode + review fixes
The known-cluster gate published only under Mode=Grpc, but the interceptor it feeds is registered on every admin node and 1B maps CentralCommandService by LISTENER PORT — neither consults the mode. A dark-switch rollback to ClusterClient or Dps would therefore have left that endpoint serving with the gate permanently inactive, silently restoring both the timing oracle and the pre-auth secret-store/audit flood, as a side effect of a rollback nobody would connect to authentication. The gate is central's cluster inventory, not transport state, so it is now decoupled from the mode entirely: - SetKnownClusters + InvalidateAll run on EVERY successful contact load in EVERY mode, and the contact-refresh timer starts in all three. Under Dps this is a ConfigDb query every ContactRefreshSeconds that did not run before; central always has a ConfigDb, and one indexed read a minute is the cheaper half of the trade. - The GATE set is built from ALL ClusterNode rows. The eligibility filter moved out of the query's WHERE and into the loop, so the FAN set is byte-for-byte what it was (enabled && !maintenance) while the gate widens. This also dissolves the maintenance coupling flagged in 1A.2: a cluster whose nodes are all disabled or in maintenance can still ack, and refusing it would make maintenance mode an authentication switch. Central still refuses clusters it has never heard of. - HandleContactsLoaded keeps a documented order — targets adopted, gate published, cache dropped — now pinned by test. Guards: PreStart mirrors the null-dialer error for a null PSK provider (silent absence is the worst outcome there — nothing malfunctions, so nothing is investigated, while the gate never publishes and keys never rotate); a failed contact load now says out loud that the gate is stale and, before any load has succeeded, inactive; MeshGrpcCommandDialer clamps a non-positive DispatchTimeoutSeconds to 10 with a construction Warning, because the validator for that key is driver-role-scoped and a pure-admin central reads it unvalidated; the dialer's local PSK refusal was shortened so its remediation survives the actor's 200-char truncation of remote-authored status detail; the portless-node warn-once set is rebuilt each refresh, so a node that regains then loses a port warns again — the regression is the occurrence worth seeing. Docs/pins: dialer disposal comment no longer claims one-way; the two interceptor-test comments claiming Matches has a default implementation corrected (it deliberately has none); new pins for per-refresh invalidation, the SetKnownClusters-before-InvalidateAll order, and one channel per endpoint via a new CachedChannelCount. The_known_cluster_set_is_not_published_by_the_other_two_modes is replaced — its premise is obsolete — by three facts: every mode publishes, gate set is a strict superset of the fan set, and the handler ordering. ControlPlane.Tests 230 -> 236, Host.Tests 92, Cluster.Tests 184, 0 warnings. |
||
|
Joseph Doherty
|
6a86201105 |
feat(mesh): gRPC command fan-out behind Mode=Grpc
MeshGrpcCommandDialer places one NodeCommandService.Execute call on one node: a GrpcChannel cached per endpoint (the GrpcDeploymentArtifactFetcher shape), the TARGET node's cluster's bearer plus x-otopcua-cluster naming that cluster, and a per-call deadline linked to the caller's token. A cluster with no resolvable key is refused locally with Unauthenticated — there is no arm on which a command leaves central unauthenticated, even one the far side would have rejected. CentralCommunicationActor gains the third mode branch. The contact load now also projects GrpcPort and, from the SAME rows in the SAME pass, builds the dispatch set — which is what makes 'the gRPC fan reaches exactly the SendToAll recipient set' structural rather than two queries that have to keep agreeing. A row with a null GrpcPort is skipped warn-once (the TelemetryNodeSource precedent). Per-node results are piped back for LOGGING ONLY: named node, gRPC status, dropped. No retry, no queue. Every dial is scheduled on the thread pool rather than called from the receive. Found by review and pinned red-first: an async method runs synchronously to its first await, and the dialer's pre-await work is TryGetPsk, which blocks on the secret store — bounded but seconds, and repeated after every refresh since the refresh drops the cache. Dialing inline parked the actor's mailbox, so deployment acks queued behind a slow node; the regression test tells an ApplyAck while a dial is held and fails without the hop. The contact refresh also wires the Phase 1A register: it publishes the loaded cluster ids as the PSK provider's known-cluster set and drops the key cache, so a rotated key takes effect within one ContactRefreshSeconds. Neither happens under Dps or ClusterClient. Both existing mode branches are behaviourally untouched and every existing fact passes unmodified. The one edit to shared text: the Status.Failure warning drops the word 'ClusterClient', since that handler now serves all three transports. ControlPlane.Tests 205 -> 230, Host.Tests 92, Cluster.Tests 184, 0 warnings. |
||
|
Joseph Doherty
|
6531ec1984 |
feat(mesh): one ClusterClient per Cluster in CentralCommunicationActor (Phase 6)
After the fleet splits into one Akka mesh per application Cluster, a ClusterClientReceptionist serves only its own mesh — so the single fleet-wide client's SendToAll reached only the mesh whose receptionist answered, leaving every other cluster silently on its old configuration. Central now holds one ClusterClient per ClusterId and fans SendToAll across all of them. LoadContactsFromDb selects ClusterId and groups the receptionist contacts per cluster (keeping the per-row TryParse guard and the enabled/non-maintenance filter); ContactsLoaded carries ContactsByCluster; HandleContactsLoaded diffs per cluster (rebuild changed, stop+drop vanished, warn-don't-create on empty); RebuildClient builds a per-cluster client under a clusterId-derived actor name. IMeshClusterClientFactory.Create gained a clusterId parameter so the per-cluster clients get distinct, diagnosable names. ApplyAck handling and the DPS branch are unchanged; the deploy path stays payload-free. Tests: focused unit coverage of the grouping + fan-out (one-client-per-cluster, fan-out SendToAll to every cluster client) via the recording factory double, plus the real two-mesh boundary test extended to central + two separate site meshes proving one dispatch reaches both and a client scoped to one cluster never crosses into another. Red-before-green verified: crippling the fan-out to a single client fails the reaches-both-meshes assertion. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |
||
|
Joseph Doherty
|
d5b5cb6ede |
feat(mesh): CentralCommunicationActor — dark-switched fan-out, DB-sourced contacts
Phase 2 Task 3. Central-side end of the boundary: routes every central->node command out over DPS or ClusterClient per MeshTransport:Mode, and forwards inbound ApplyAcks to the deploy coordinator singleton (Forward, not Tell, so the coordinator sees the node rather than this relay). Three things here are load-bearing and easy to get wrong later: SendToAll, never Send. Today's DPS publish reaches EVERY DriverHostActor and the node side has no ClusterId or node filter to compensate -- scoping happens later, inside the artifact. ClusterClient.Send delivers to exactly ONE registered actor, so it would deploy to a single node while every other node silently kept its old config, and the deployment could still seal green. Sabotage-verified: swapping to Send reddens exactly that one test. Exactly ONE ClusterClient, fleet-wide. A receptionist serves its whole cluster, so SendToAll reaches every registered node-comm actor in the mesh regardless of which contact point was dialled. One client per application Cluster -- the shape Phase 6 wants -- would fan each command out once per cluster while the fleet is still one mesh: N x duplicate DispatchDeployment and ApplyAck. Marked TODO(Phase 6). The contact set does NOT scope delivery. Excluding a maintenance-mode node from the contacts does not stop it receiving commands; it still receives them, as it does under today's broadcast. The filter is about which receptionists are worth dialling, not about who gets the message. Stated in the code because the opposite is the natural assumption. Also carries the sister project's shipped fixes: per-row address parsing so one malformed ClusterNode cannot abort the refresh and leave the contact set half-built (regression test orders the bad row FIRST), the cache entry cleared before recreate so a failed create cannot leave commands routing into a stopping actor, and a Status.Failure handler so a DB outage is a Warning rather than a silent debug-level unhandled message. Two test-harness bugs found and fixed while writing this, both mine: SubscribeAck goes to the SENDER (so the mediator subscribe needed an explicit sender), and EventFilter matches case-INSENSITIVELY, so a "was NOT" filter also caught this actor's own "was not created" warning. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW |