A CNC can report duplicate sibling names — the wonder Makino returns two
axes both called AA — which made the Axes/ subtree untrustworthy: the
browser offered two identical folders the commit gate had to refuse, and
the poll loop's name-keyed snapshot cache let the second axis silently
overwrite the first. Spindles and servo loads had the same collision in
their own keying, plus a resolve-to-first-match on the spindle read path.
Scheme: every member of a collision set gets a 1-based occurrence-ordinal
suffix in slot order (AA_1, AA_2); unique names stay bare. Chosen over the
alternatives deliberately:
- Names, not slot numbers, stay the primary identity — the segment map is
re-derived from the live device each session, so a name-keyed path
follows the physical axis across config changes, where a slot-qualified
path would silently rebind on any axis-list shift and rename every axis
on every machine for one pathological CNC.
- The WHOLE collision set is suffixed: bare AA ceases to exist when
ambiguous, so a tag authored against it reads BadNodeIdUnknown rather
than one arbitrary axis's data — the #534/#540 silent-wrongness class.
- Occurrence ordinal, not absolute slot: inserting/removing an unrelated
axis never renames an existing duplicate pair. If the collision-set
membership itself changes, segments change on the next browse and stale
authored segments read Bad, never another axis's data.
One pure helper (FixedTreeNames.Disambiguate, collision-proof against
genuine names by suffix escalation) feeds FocasFixedTreeCache.AxisSegments
/ SpindleSegments, and the emitter, axis snapshot keys, servo-load keys
(cnc_rdsvmeter answers in slot order, so the k-th AA load belongs to the
k-th AA axis) and the spindle read resolution all consume those — they can
no longer disagree. The nameless-spindle S{n} fallback applies before
disambiguation so a fallback name colliding with a real one is handled
like any other duplicate.
Tests are red-first on the pre-fix driver (all five driver-level tests
fail on master's code): browse emits AA_1/AA_2 and no bare AA, each
duplicate reads its own position and servo load, duplicate spindles
resolve per slot, bare AA reads Bad. The wire half runs live against
focas-mock (duplicate names survive the managed wire parse and publish as
suffixed segments); the mock's dynamic store is name-keyed so the
distinct-values half stays with the offline tests. FOCAS 327/327,
integration 11/11, FOCAS.Cli 52, AdminUI 1113, ControlPlane 231.
Closes#548.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
The summary list still described the pre-#520 world: NodeAcl/PermissionTrie
'built and unit-tested but never wired', reads/browse/subscribe/HistoryRead
carrying no authorization at all, and UserName tokens authenticated by LDAP
bind. The body had been updated by #520 stages 5-7 but the summary, the
Anonymous auth-table row, the LDAP bind-flow paragraph (still routing the OPC
UA UserName path through the deleted LdapOpcUaUserAuthenticator), the
'sourced from Security:Ldap:GroupToRole' role-source section, and the
'Dispatch gate (no such gate exists)' section all contradicted it — a reader
who stopped at any of them got the pre-#520 answer.
All now state the flag-gated reality: with Security:Acl:Enabled false (the
default) the only enforcement is the two coarse server-wide role gates and
reads are ungated; with it true, scoped NodeAcl grants decide every
data-plane operation via DataPlaneAuthorizer -> TriePermissionEvaluator.
Data-plane roles derive from cluster-scoped NodeAcl grants
(ApiKeyInterimRoleMapper); GroupToRole and LdapGroupRoleMapping feed AdminUI
login only. The acl-design.md pointer no longer calls the design 'unshipped
work'.
Closes#547.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
The two discovery-path libplctag tags (@tags symbol-list enumerator, @udt
template reader) still passed the bare Gateway host, so a browse against a
device authored on a non-default port silently browsed whatever answers on
44818 — #534's defect surviving in its discovery half. Both sites now consume
AbCipTagCreateParams.LibplctagGateway, with the Tag construction extracted
into internal static builders so the seam is unit-assertable (the #534 tests
pinned the property's composition but nothing pinned each site's consumption,
which is exactly how these two survived).
New guard: LibplctagGatewayConsumptionGuardTests scans both AB drivers'
sources and fails on any libplctag Tag built from the bare Gateway, so a
future call site cannot reintroduce the silent 44818 redirect.
Live-gated against the fixture host: browse of ab://10.100.0.35:44818
returns the nine-tag symbol table; browse of ab://10.100.0.35:44999 (nothing
listening — the issue's exact repro) now fails ErrorTimeout where it
previously returned the 44818 fixture's table.
Closes#539.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Ran on master @ 0b194003, post-deletion image d2a290fc5711 (provenance-guarded).
PSK matrix 7/7 incl. unset-key boot refusal; command matrix with Restart/Reconnect
driven from the real /hosts panel (#546 closed the 1A composition carry-forward by
direct evidence); exactly-once ack matrix sealed 6/6; central-kill + site-kill drills
honest (one loud WRN, +2:00 deadlines, boot-replay, no retro-ack); mid-deploy
coordinator kill: recovered coordinator fired the deadline at exactly +2:00, all
landed acks preserved, zero dupes; boundary: 3 established :4053 associations, all
pair-internal, zero site↔central; full-rig restart transport grep = 0 on all six
(one non-mesh observation: central pair reproduced the documented guard-off
cold-start split-brain, healed by single-node rejoin); MeshGrpcBoundaryTests 3/3;
terminal full suite RC=0 direct capture, 50/50 suites, 9,333/0.
I-4 cold-PSK-cache re-carried out of the program (unmeasurable on docker-dev).
No new defects.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
ConfigServeAuthInterceptor and LocalDbSyncAuthInterceptor gained HasExactlyOnePublicConstructor
pin tests in this branch but lacked the sibling-standard <remarks> paragraph naming them
(TelemetryStream/MeshCommand/CentralCommand all carry it). Cosmetic consistency item from the
#543 code review.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
The proxyOptions local in WithOtOpcUaControlPlaneSingletons was already unused on master
(pre-existing dead code, not orphaned by the deletion commit — every registration passes
singletonOptions); removed while the file is open. The phase-6-1 compliance script still
Assert-Deferred'd the FleetStatusHub /hosts follow-up — an item that is now deleted, not
deferred — replaced with a retirement comment.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
DriverStatusPanel decides whether to subscribe exactly once, in
OnInitializedAsync (`if (!Enabled) return;`), and exposes no OnParametersSet
hook. The detail row was keyed only by DriverInstanceId, so a live config edit
that flipped a row's Enabled true->false — surfaced on the page by the 5 s
LoadConfigAsync refresh — reused the same panel instance: the markup correctly
switched to "Disabled - not deployed" while the panel's two store SnapshotChanged
subscriptions and its 5 s staleness timer stayed wired underneath, until the
operator happened to collapse the row.
Including the flag in the detail row's key recycles the component instead. Both
directions verified against the panel's own lifecycle: true->false disposes the
old instance (unsubscribing both stores, draining both timers) and mounts one
that skips subscription setup entirely; false->true disposes an instance that
never subscribed (the unsubscribes are no-ops, both timers null) and mounts one
that subscribes and primes from the stores.
The data row's key is unchanged. No behaviour change when Enabled is stable.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
#542 filed FleetStatusBroadcaster as blind to site nodes since the Phase 6 per-cluster mesh
split: it derives "the fleet" from central's own Akka gossip ring, which after the split is
just central's own pair. Scoping the cross-mesh rewrite found the deeper truth — the whole
chain is dead end to end, so this deletes it rather than reviving it.
- Zero consumers. FleetStatusBroadcaster published FleetStatusChanged on the DPS
"fleet-status" topic; FleetStatusSignalRBridge forwarded it to FleetStatusHub
("/hubs/fleet-status"); nothing anywhere connected. The AdminUI bans self-connecting
HubConnections, no browser JS subscribes, and /fleet (Fleet.razor) reads
NodeDeploymentState from the ConfigDb plus IClusterRoleInfo on a 5 s poll instead.
- Its second input never existed. DriverHostStatusHeartbeat had no production sender —
only the broadcaster's own test told it — so CurrentRevision was always null and the
30 s staleness -> Degraded flip could never fire.
- Post-Phase-6 DPS is pair-local, so even the central-only view it did produce was an
accident of topology rather than a design.
The fleet visibility that actually works is unchanged and already cross-mesh-correct:
/fleet over NodeDeploymentState, and the four Phase 5 per-node telemetry dials.
Precedent for deleting inert machinery rather than activating it: #522 (driver stability
tiers), #507 (discovered-node injection), #525 (GetMemoryFootprint).
Deleted: FleetStatusBroadcaster (+ its nested DriverHostStatusHeartbeat), its admin
singleton registration and FleetStatusBroadcasterKey, FleetStatusSignalRBridge and its
key, FleetStatusHub and its route mapping, and the FleetStatusChanged / FleetNodeStatus /
FleetNodeHealth messages. Both Fleet/ directories survive — ControlPlane keeps
ClusterNodeAddressReconciler, Commons keeps GetDiagnostics. Docs swept where they claimed
a fleet-status channel exists (Telemetry.md's Phase 5 non-migrated channels,
Configuration.md, CLAUDE.md); completed-program plan docs under docs/plans and docs/v2 are
left as the historical record they are.
Build 0 errors, 868 warnings — byte-identical to master's rebuild baseline.
ControlPlane.Tests 231/231, AdminUI.Tests 1109/1109.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
DriverStatusPanel was orphaned — zero references repo-wide — so the Restart and
Reconnect driver commands had NO reachable UI at all, despite working end to end
over the new gRPC mesh transport. Nothing failed to build, test or deploy while
that was true, which is why it went unnoticed.
The panel is now reachable per driver row on /hosts, behind a per-row "Controls"
expander that is collapsed by default: the table stays scannable, and only the
opened panels hold snapshot-store subscriptions. Expansion state is a HashSet
keyed by DriverInstanceId held on the page, so it survives the 5 s refresh that
replaces every row record. Both rows are @key'd by instance id so the panel —
which binds its DriverInstanceId once in OnInitializedAsync — can never be diffed
onto a different driver.
The panel's Enabled parameter is fed from real config rather than defaulted:
HostsDriverInstanceInfo/HostsDriverRow now carry DriverInstance.Enabled, which
LoadConfigAsync already had in hand. A snapshot with no matching configured
instance is treated as enabled — the row exists only because live health arrived,
so "Disabled — not deployed" would contradict the same row.
The expander is visible to every AuthenticatedRead viewer; the panel self-gates
only its Reconnect/Restart buttons on DriverOperator.
A source-scan pin (HostsPanelWiringTests) asserts /hosts still renders the panel
and still binds it to the row's own driver, cluster and enabled state — removing
the tag would silently restore the orphaned state.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
OtOpcUa runs five internal gRPC planes between its own processes, all of
them cleartext HTTP/2 (h2c) with a shared-bearer interceptor: the live
telemetry stream, config-serve artifact fetch, LocalDb pair sync, mesh
commands and mesh deploy acks. The bearer therefore crosses the wire in
the clear on every one of them. That was true and undocumented; the two
mesh planes are command surfaces, so it is worth saying out loud rather
than leaving to be rediscovered per plane.
docs/security.md gains "Internal gRPC planes (h2c) — accepted cleartext
posture" under Transport Security (whose existing content is entirely
about the OPC UA endpoint). It carries a five-row table — service name,
listener config key, credential config key, interceptor — the honest
read-vs-write breakdown per plane, and the decision itself.
Decision (issue #545, option 1): risk ACCEPTED 2026-08-06, once, for all
five planes together, on the trusted-link assumption every plane already
makes (VPN / dedicated WAN). Per-cluster PSKs already bound blast radius
on the two command planes — a captured SITE-A key cannot drive SITE-B —
and production keys arrive as ${secret:...} references that must never be
reused across clusters. Recorded future work: a UseTls/scheme option,
which if built must cover all five planes together, since a mixed posture
is harder to reason about than either uniform answer.
docs/Configuration.md's MeshTransport section gains one cross-reference
paragraph pointing at the new subsection.
Docs only — no code, no behaviour change. Every service name, config key,
interceptor name and shared-listener claim was verified against the
source; ports 4056/4055 are phrased as the docker-dev rig's values of
Telemetry:GrpcListenPort / ConfigServe:GrpcListenPort, not as hardcoded
ports.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
DeadlineElapsed_for_current_deployment_marks_TimedOut carried two races,
both test-side. ConfigPublishCoordinator itself is untouched.
(a) The NodeDeploymentState row was inserted AFTER Sys.ActorOf. The
coordinator's PreStart reads NodeDeploymentStates to recover the
expected-ack set, and it runs as soon as the actor is scheduled — so if
PreStart won, the expected-ack set was empty and the coordinator could
short-circuit to Sealed instead of ever reaching TimedOut. The row is now
seeded before the spawn.
(b) The applyDeadline was 150 ms of real wall clock, and it arms a real
Akka timer. Under load that timer could fire on its own and race the
test's explicit DeadlineElapsed tell — so the test was intermittently
exercising the timer rather than the message handler it names. The
deadline is now TimeSpan.FromMinutes(5), matching the other two tests in
this class, which makes it un-hittable inside the test's own 3 s
AwaitAssert bound and leaves the explicit tell as the only deadline
signal the coordinator ever sees. That tell, and the AwaitAssert bound,
are unchanged.
Verified: the single test 5/5 green in a row, then the whole class 3/3.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
TelemetryStreamAuthInterceptor, ConfigServeAuthInterceptor and
LocalDbSyncAuthInterceptor all decided "is a key configured?" with
string.IsNullOrEmpty. A whitespace-only configured value therefore passed
that guard and fell through to the constant-time comparison — where it is
satisfied by a whitespace-only bearer token. The gate would be open to
anyone who guessed a blank credential.
This backports the check the two NEW mesh interceptors already use:
MeshCommandAuthInterceptor (and its central-side clone) validate with
string.IsNullOrWhiteSpace, so the four/five planes now answer the same way
to the same misconfiguration. Nothing else changes — the denial status
codes (PermissionDenied / Unauthenticated / PermissionDenied respectively)
and every denial message are untouched, and no key material is logged.
No deployment configures a whitespace key: the docker-dev rig and
production both set real keys, so this is a fail-closed hardening of an
unreachable-today misconfiguration rather than a live exposure.
Tests: each of the three test files gains a ..._WithBlankKeyConfigured_IsDenied
fact mirroring MeshCommandAuthInterceptorTests.ExecuteMethod_WithBlankPskConfigured_IsDenied
(configured key " ", presented "Bearer "), verified red before the fix.
ConfigServeAuthInterceptor and LocalDbSyncAuthInterceptor also gain the
HasExactlyOnePublicConstructor pin their two siblings already had —
Grpc.AspNetCore silently stops invoking an interceptor with more than one
public constructor, which would disable auth without failing anything.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
MeshTransport:Mode is now Dps | Grpc only; the retired ClusterClient value
fails validation loudly. CentralContactPoints, both receptionist
registrations, the akka.cluster.client HOCON block and the central/node
client machinery are deleted; SplitTopologyTransportValidator requires Grpc
on cluster-role nodes. MeshGrpcBoundaryTests replaces the old boundary test
and pins that no cross-mesh Akka association forms. Docs swept (incl. the
scadaproj index, pushed separately on its main).
Task 3.1 spec+code reviews closed (1 Important fixed: MeshPaths collapse);
Task 3.2 docs-accuracy review closed. DoD full suite: exit 0, 50/50 suites,
9326 passed / 0 failed on 8e25ed55.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Task 3.1: 7 ordered deletion steps + MeshPaths review fix, spec+code reviews closed.
Task 3.2: docs sweep + accuracy review, scadaproj index pushed on its main.
DoD full-suite: run 3 exit 0 (50 suites, 9326 passed); runs 1-2 each hit one
pre-existing unrelated timing flake (3 distinct tests, all untouched by the
branch, 9/9 green in isolation). Also records the exit-code-capture trap found
in run 1 and its retroactive caveat on Phase 2's check-8 wording.
Merge to master is user-owned — STOPPED at the Phase 3 DoD.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Review of the ClusterClient-deletion docs sweep (406b1ae5) flagged two
pre-existing staleness items in the swept files, unrelated to ClusterClient
but caught while touching the same passages:
- docs/Telemetry.md: the Phase 5 telemetry section's Status line still said
"code-complete ... live gate has not yet run" — the gate PASSED 2026-07-23
(merge 35552a96), already recorded in CLAUDE.md. Updated to match.
- docs/v2/config-db-schema.md: the AkkaPort/GrpcPort note asserted in the
present tense that "central shares a gossip ring with every node," which
Phase 6 (2026-07-24) ended — central now runs one 2-node Akka mesh per
application Cluster and reaches a site pair only over gRPC. Reworded the
premise to past tense so it no longer contradicts the ClusterClient-deleted
parenthetical sitting right after it.
docs-only; dotnet build ZB.MOM.WW.OtOpcUa.slnx stays green (0 errors).
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Task 3.2 of the ClusterClient->gRPC migration. Sweeps the docs that still
described the Akka ClusterClient transport as live, following its Phase 3
deletion (this branch): MeshTransport:Mode is now Dps (pair-local DPS) or
Grpc (cross-mesh unary calls) only, ClusterClient fails validation loudly,
CentralContactPoints is deleted (not superseded), and the split-topology
validator requires exactly Grpc rather than ClusterClient.
- docs/Configuration.md: rewrite the MeshTransport section — table,
dark-switch/PSK-seeding prose, listener sharing (:4056 driver-role,
:4055 admin-role), remove the "shipped dark, cutover pending" framing.
- docs/Redundancy.md: replace the Dps/ClusterClient transport table with
Dps/Grpc, fix the Phase 6 CentralCommunicationActor + validator bullets.
- docs/Telemetry.md: fix stale ClusterClient cross-refs, add a note that
the driver-role telemetry listener also hosts NodeCommandService.
- docs/v2/config-db-schema.md: one-line "(retired: ...)" annotation on the
historical AkkaPort/GrpcPort note — left as a historical record otherwise.
- CLAUDE.md: rewrite "## Mesh command transport (MeshTransport)" for the
post-deletion world (no ClusterClient/receptionist, per-node gRPC dial,
fire-and-forget semantics unchanged) and fix two Phase 6 bullets.
deferment.md has no mesh-transport mentions; left untouched.
docs-only change; `dotnet build ZB.MOM.WW.OtOpcUa.slnx` stays green (0
errors) after the edit.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
MeshPaths.CentralCommunication / .NodeCommunication (the "/user/..." full-path constants) had
ZERO production consumers after the ClusterClient deletion: both gRPC services resolve their
comm actor through ActorRegistry by typed key, never by ActorSelection over a path. Only the
path-probe test still read them, while the class doc went on calling them "the wire contract
of the central↔node boundary" — a claim that stopped being true the moment nothing addressed
an actor across a process boundary by name.
- The two full-path constants are deleted; the two Name constants remain.
- MeshPaths' doc now says what it is (the local actor-naming convention the registry-keyed
services are spawned under) and records what it stopped being, so the next reader does not
re-derive a contract from the file's history.
- MeshCommActorPathTests composes "/user/" + Name itself — the absolute path is that test's
own addressing detail now, not something the product declares. Renamed
Both_mesh_comm_actors_exist_at_their_contract_paths ->
..._are_spawned_under_their_declared_names, because what it still pins is that the two
registration sites actually ran on a real booted node; a rename is now a compile error, so
"contract" was the wrong word for what would break.
- NodeCommunicationActor's summary no longer cross-references the deleted constant or calls
the name an address. CentralCommunicationActor was checked and makes no such claim.
MeshGrpcBoundaryTests replaces the MeshClusterClientBoundaryTests deleted in step 1. Three
self-seeded single-member clusters sharing the ActorSystem name, joined ONLY by gRPC, driving
the REAL CentralCommunicationActor -> MeshGrpcCommandDialer -> Kestrel h2c ->
MeshCommandAuthInterceptor -> NodeCommandGrpcService -> NodeCommunicationActor -> EventStream:
1. No_cross_mesh_Akka_association_forms_between_central_and_either_site — the program's own
point, kept as a test rather than a live-gate note: each cluster stays single-member and
none ever holds another's address as a member.
2. One_dispatch_through_central_reaches_a_node_in_every_cluster_mesh — the fan, with each
site holding its OWN per-cluster PSK, so a dial that selected the wrong cluster's key
would be refused at the far side.
3. Falsifiability_control_a_dial_carrying_the_wrong_cluster_key_delivers_nothing — without
it, (2) cannot distinguish "the gRPC boundary carried it" from "something else in the
process did". The same node then accepts the right key, so the silence is the auth gate
and not a dead endpoint.
All three run fully offline — no Docker fixture — and pass.
Comment sweep across every remaining clusterclient|receptionist hit in src/ and tests/:
the program label becomes "mesh gRPC command plane", and every comment that described the
deleted transport as live now describes what actually runs (Dps = pair-local default,
Grpc = cross-mesh). mesh_transport.proto's header/service comments are reworded; no message,
field or enum is touched — the contract is locked. MeshCommActorPathTests is reshaped rather
than retired: the two paths still exist and still matter, but what a mismatch costs is now a
misleading Unavailable from the gRPC ingress rather than a silently dropped receptionist send.
The only surviving mentions of the old names are the ones that must survive: the fail-closed
rules that name "ClusterClient" as a retired value refused at boot, their tests, and this
test class's own account of what it replaces.
MeshTransportOptions.ModeClusterClient is gone. MeshTransportOptionsValidator accepts Dps or
Grpc and nothing else, so a node left carrying the string "ClusterClient" is now REFUSED at
boot naming MeshTransport:Mode and both remaining values — deliberately fail-closed rather
than falling back to Dps, which on a split-topology node would deliver nothing at all. A new
fact pins exactly that.
SplitTopologyTransportValidator tightens from an allow-list of two to Grpc only, as its own
comment said it would once ClusterClient was deleted. Dps on a cluster-role node stays
refused; the rule is still an explicit allow-list, so a typo'd mode fails too.
Tests: the two ClusterClient-mode facts in MeshTransportOptionsValidatorTests are deleted
(their Dps siblings already assert the same rule for the one remaining non-Grpc mode), the
SplitTopology fixtures move to Grpc, and the three-mode Theories in
ControlPlaneStarterOrderTests / CentralCommunicationActorTests become two-mode.
The property, the four validator rules that graded it (empty / non-akka.tcp / actor-path
suffix / one-bad-among-good) and all twelve docker-compose entries (6 nodes x 2) are gone.
The compose comment blocks that explained "/system/receptionist is appended at construction
time" and the client's contact rotation go with them, and the surrounding prose now
describes the Grpc command plane it actually runs.
CentralGrpcEndpoints' doc no longer says it "replaces" the Akka contact points — there is
nothing left to replace — and ContactRefreshSeconds' failure message names the node
dispatch set rather than the ClusterClient contact points.
`rg CentralContactPoints src tests docker-dev` is now empty.
akka.extensions loses ClusterClientReceptionistExtensionProvider and the whole
akka.cluster.client block (buffer-size / reconnect-timeout / receptionist.role) goes with
it. The DistributedPubSub extension and the maximum-frame-size / log-frame-size-exceeding
canary both stay — DPS is the pair-local transport and the frame canary guards every
remoting message, neither of which the cross-mesh deletion touches.
MeshTransportHoconTests drops its three ClusterClientSettings/ClusterReceptionistSettings
asserts and keeps the frame-size pin and the DPS-extension pin, renamed since it no longer
has a receptionist to survive alongside.
Central's CentralCommunicationActor and a node's NodeCommunicationActor no longer register
with a receptionist; the ActorRegistry entry each already carried is now the whole
reachability story, and it is what CentralCommandGrpcService / NodeCommandGrpcService
resolve to hand an inbound message over. Both `using Akka.Cluster.Tools.Client` directives
are gone with them.
WithOtOpcUaRuntimeActors no longer builds a ClusterClient from
MeshTransport:CentralContactPoints, and NodeCommunicationActor.Props takes only the gRPC
IApplyAckSender. The ackRouter gate narrows from `useClusterClient || useGrpc` to `useGrpc`,
and the CentralClusterClientName constant is gone.
NodeCommunicationActor's no-transport warning now names MeshTransport:CentralGrpcEndpoints
and MeshTransport:Mode only — the old text pointed an operator at a key that no longer
exists.
Ripples in this step: the two ClusterClient-shaped facts in NodeCommunicationActorTests
(Send-not-SendToAll, and sender-wins-over-a-client) are deleted — the first has no subject
left, the second guarded a both-transports-present state that is now unconstructible. The
Runtime ServiceCollectionExtensions fact keeps its ackRouter pin and drops its
"no ClusterClient actor exists" half for the same reason.
IMeshClusterClientFactory/DefaultMeshClusterClientFactory are gone, along with
CentralCommunicationActor's _clients map, RebuildClient, the SendToAll fan and the
receptionist-address half of the contact load. LoadContactsFromDb SURVIVES — it feeds the
gRPC dialer's endpoint resolution and the known-cluster gate — but now projects only the
columns those two need (the AkkaPort/ActorPath.TryParse parse and its Malformed list went
with the receptionist addresses).
Ripples fixed in this step because the build must be green at every commit:
- ControlPlane/ServiceCollectionExtensions drops the factory argument.
- CentralCommunicationActorTests: the six ClusterClient-only facts are deleted (each has a
Grpc counterpart that already asserted the same property independently), the three
transport-agnostic facts move to Grpc mode, and the Dps fact keeps its own coverage.
- MeshClusterClientFactoryTests deleted (its subject is gone).
- MeshClusterClientBoundaryTests deleted here rather than in step 7: it drives
DefaultMeshClusterClientFactory directly, so it cannot compile past this commit. Its
replacement, MeshGrpcBoundaryTests, lands in step 7 and keeps the load-bearing assertion
(two separate meshes, no cross-mesh Akka association).
All six nodes now default to the gRPC command/ack transport; OTOPCUA_MESH_MODE
still overrides for rollback drills. Baseline reproduced pre-flip (1bb3a937):
cross-mesh :4053 open unauthenticated, unreachable-cluster dispatch suppressed
into sampled deadLetters INF with the deploy deadline as sole witness.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Follow-up to a94eb8a3, addressing CHANGES REQUESTED on code review. Production
logic was confirmed correct on all spec points; this commit fixes the review's
test/comment findings only.
IMPORTANT-1: A_permanently_unresolvable_pskProvider_is_reported_once_per_incarnation_not_once_per_refresh
did not prove the second refresh ran — its Expect(0) window gated on a dialer
dispatch landing, but LoadContactsFromDb is fire-and-forget and the first
block's AwaitCondition already re-Tells MeshCommand per poll, so the dialer
call count could reach the threshold before PublishKnownClusters was ever
re-invoked by the second load. Fixed by having the PSK factory count its own
invocations and gating both AwaitConditions on that count directly (pskCalls
>= 2 proves the first retry; pskCalls > pskCallsAfterFirstLoad, awaited INSIDE
the Expect(0) window, proves the second one). Bite-checked: with
_pskProviderAbsenceReported's guard temporarily removed, the test now
correctly fails ("Received 1 message too many... received 1 that matched
filter"); guard restored and reverified green.
MINOR-2: reworded four comments (CentralCommunicationActor.cs ~109, ~259,
~305-309, ~853) from narrating the change ("this fix removes", "already
existed, unchanged", "added by this fix", "the old two-call-site shape") to
stating the timeless invariant.
MINOR-3: added A_supplied_dialer_factory_that_always_throws_still_drops_each_command_loudly_while_PreStart_stays_quiet
— the symmetric case to the existing never-supplied-dialer test, exercising
Resolve's catch + firstUse:true path (a factory that WAS supplied and always
throws) rather than the null-factory path. Asserts PreStart plus the first
contact load stay silent and DispatchOverGrpc's existing per-dispatch Error
still fires on every dispatch, unguarded.
MINOR-4: "Check the ControlPlane registration wires IMeshPskProvider" ->
"Check that the ControlPlane registration wires up IMeshPskProvider"; no test
pinned the old substring, so no test update was needed for this one.
Skipped item 5 (restart-re-arms pin) per instruction — out of scope.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
PreStart's eager resolve of IMeshCommandDialer/IMeshPskProvider loses a
millisecond-wide race against the Akka builder's remaining starters on
essentially every admin boot (live gates: 1A 2/3 boots, 1B 4/4 boots) and
self-heals at the memoized Dialer/PskProvider properties' first use. Before
this change that produced two FALSE Errors per healthy boot, which made a
soak error-grep on a healthy rig impossible.
Resolve() now takes a `firstUse` flag and logs at Debug on BOTH call sites
(PreStart and the memoized properties' retry) — from inside Resolve a lost
race and a genuinely broken factory are indistinguishable, so severity is
decided by the CONSUMER of a still-null result after the retry, not by
Resolve itself:
- Dialer: DispatchOverGrpc's existing per-dispatch Error ("no command
dialer is wired") is unchanged and is now the ONLY loud signal for that
dependency.
- PskProvider: PublishKnownClusters gained a new Error carrying the old
PreStart guard's diagnostic content (gate never published, cached keys
never dropped, rotated key needs restart), guarded to fire ONCE per
actor incarnation via `_pskProviderAbsenceReported` — that handler runs
on the contact-refresh timer for the actor's whole life, so an
unguarded Error there would repeat every ContactRefreshSeconds forever.
Design choice for the "factory never supplied" case (Dps/ClusterClient test
constructions that pass no factories at all): kept silent at BOTH call
sites for both dependencies, same as before. For the dialer this was
already true (DispatchOverGrpc's check doesn't care why the dialer is
null). For the PSK provider the new once-guarded Error is gated on
`_pskProviderFactory is not null` specifically so the many existing
ClusterClient/Dps facts in CentralCommunicationActorTests that never wire a
PSK provider (using the bare Spawn helper) keep passing unmodified — a
never-supplied factory is a deliberate wiring/test choice, not a resolve
failure, and Resolve() already stayed silent about it for the same reason.
Two new tests pin the net behavior: a healthy boot (factories that throw
once then succeed) produces zero Warning/Error end-to-end through PreStart,
the first contact load and the first dispatch; a permanently-broken PSK
provider logs the absence exactly once across two contact loads, not once
per refresh.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Node→central ApplyAck over gRPC behind MeshTransport:Mode=Grpc: CentralCommandGrpcService
on the AdminUI listener, GrpcApplyAckSender with per-endpoint ack deadlines (Critical fix
e54711a2 proven live in the 4b hang drill), NodeCommunicationActor wiring, plus the
phases 2-4 plan update (targeted-test policy + Task 2.0 must-fix).
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Node→central ApplyAcks over gRPC, gated live on the docker-dev rig at
4a2a3d3a. All 7 checks PASS (the 6 planned plus the mandated 4b hang
drill).
The 4b drill is the one that earns its place: with central-1 PAUSED
rather than stopped, both site-a nodes burned a full AckTimeoutSeconds
on it as DeadlineExceeded and still reached central-2 with a live
deadline — the per-endpoint FRESH deadline from 667639e6, proven on the
failure shape a hoisted deadline could not survive. Observed failover
latency ~10 s, exactly one AckTimeoutSeconds.
It also produced the run's only duplicate ack, and that turned into
evidence rather than a defect: the frozen central had in fact received
both acks and replayed them on unpause, and the coordinator reported
2/6 acks landed, not 4/6 — the idempotent absorption the sender's
at-least-once remark asserts, observed instead of argued.
Coexistence is per pair in one deployment: site-a's rows are Applied
with matching ReportApply lines on central and zero receptionist
associations, site-b's are Applied with zero ReportApply lines and a
live receptionist association. Neither pair could have used the other's
transport.
Carry-forwards recorded: the I-4 cold-PSK-cache latency is structurally
unmeasurable on docker-dev (ClusterPsks is read ahead of the store, so
InvalidateAll clears an empty cache) and moves to a store-backed rig;
a restarted central's cached channel can stay in backoff ~2 min on the
node that watched it die, absorbed by the walk with no ack lost. The
1A finding-5 wording fix gains a second confirmation — 4 of 4 admin
boots lost the PreStart race, and every ack central-1 accepted proves
the provider it said would NEVER be published was.
Docs only; the rig flip and the four Serilog Debug overrides were
reverted and are not in this diff. Rig restored to six healthy,
all-ClusterClient, sealing 6/6.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
CRITICAL. The ack deadline was one absolute DateTime computed once for the whole
endpoint walk. Against the characteristic WAN failure — a central that HANGS
(SYN black-hole, frozen host) rather than refusing — endpoint 1 burned the whole
AckTimeoutSeconds and endpoint 2 was then dialled with a deadline already in the
past: instant DeadlineExceeded, no packet leaving the node, the surviving central
never really tried. Failover looked like it ran and did not. It also contradicted
MeshTransportOptions' own documented "applied per endpoint, not per ack". The
deadline is now computed fresh inside the loop; headers stay hoisted (read-only,
sequential walk).
Two test gaps that let that survive are closed: nothing pinned the ackRouter gate
(narrowing it back to ClusterClient-only compiled and passed every test while
dead-ending every Grpc ack to DPS), and the deadline fact's lower bound accepted
any positive timeout at all. Both new pins are mutation-verified.
Docs corrected where they overstated the case: duplicate acks CANNOT mis-seal a
deployment — ConfigPublishCoordinator.HandleAck keys _acks by NodeId behind an
expected-set guard, so a duplicate overwrites its own entry. The single-endpoint
rule is defence in depth and a bound on wasted work, not a correctness
dependency; and the walk is at-least-once on the deadline path, not at-most-once.
Task 1B.2 — the node→central half of the ClusterClient→gRPC migration.
GrpcApplyAckSender walks MeshTransport:CentralGrpcEndpoints in order and
STOPS AT THE FIRST SUCCESS. That single `return` is the whole single-count
guarantee: the transport it replaces was ClusterClient.Send, deliberately not
SendToAll, because both central nodes forward to the SAME
ConfigPublishCoordinator singleton proxy — an ack posted to both would have the
coordinator count one node twice and could seal a deployment on a doubled count.
The list is an availability list, not a fan-out list.
No retry beyond that one ordered pass, no queue: the receptionist buffered
nothing and retried nothing, and this is like-for-like, not an upgrade to
delivery guarantees. All endpoints failing is still a DROP — but a loud one,
naming the node, deployment, correlation id and the status each endpoint
answered with, which is what separates "central is down" from "central rejected
this node's key". The bearer never reaches a log; only endpoints and status
codes do. Per-endpoint GrpcChannel cache and the endpoint loop clone
GrpcDeploymentArtifactFetcher; the cluster id in x-otopcua-cluster comes from
this node's own cluster-<id> role via RoleParser, off the same bound
Cluster:Roles ClusterRoleInfo reads.
SendAsync never throws: the caller discards the task, so an escaping exception
would become an unobserved fault — a drop with no log line, the exact failure
mode this migration removes.
NodeCommunicationActor gains an OPTIONAL ackSender; every existing test compiles
and passes unedited. It is checked before the ClusterClient so the two can never
both fire. The registration arm creates no ClusterClient under Grpc.
NOTE — one wiring change beyond the actor: DriverHostActor's ackRouter is now
`useClusterClient || useGrpc ? nodeComm : null`. Left at ClusterClient-only, a
Grpc node's ack would have gone out on the DPS deployment-acks topic and never
reached the sender at all.
Two review items from Task 1B.1, both diagnostic:
1. CentralCommandGrpcService's "actor not ready" Warning omitted the peer and
the correlation id while the decode-failure Warning six lines above carried
both. These are the only two ways an ack fails to reach the coordinator, and
an operator has to correlate either against the node's own log. Decoding has
already succeeded on this path, so the DECODED correlation id is free.
2. A fused driver,admin,cluster-MAIN node under Mode=Grpc owes BOTH validator
surfaces at once (driver keys AND ConfigServe:GrpcListenPort). The two blocks
are independent `if`s; chained as `else if` a fused central node would be
graded on one surface only and boot with nowhere for any ack to land. Pinned
from both sides plus the fully-configured pass.
Joseph Doherty