CRITICAL. The ack deadline was one absolute DateTime computed once for the whole
endpoint walk. Against the characteristic WAN failure — a central that HANGS
(SYN black-hole, frozen host) rather than refusing — endpoint 1 burned the whole
AckTimeoutSeconds and endpoint 2 was then dialled with a deadline already in the
past: instant DeadlineExceeded, no packet leaving the node, the surviving central
never really tried. Failover looked like it ran and did not. It also contradicted
MeshTransportOptions' own documented "applied per endpoint, not per ack". The
deadline is now computed fresh inside the loop; headers stay hoisted (read-only,
sequential walk).
Two test gaps that let that survive are closed: nothing pinned the ackRouter gate
(narrowing it back to ClusterClient-only compiled and passed every test while
dead-ending every Grpc ack to DPS), and the deadline fact's lower bound accepted
any positive timeout at all. Both new pins are mutation-verified.
Docs corrected where they overstated the case: duplicate acks CANNOT mis-seal a
deployment — ConfigPublishCoordinator.HandleAck keys _acks by NodeId behind an
expected-set guard, so a duplicate overwrites its own entry. The single-endpoint
rule is defence in depth and a bound on wasted work, not a correctness
dependency; and the walk is at-least-once on the deadline path, not at-most-once.
Task 1B.2 — the node→central half of the ClusterClient→gRPC migration.
GrpcApplyAckSender walks MeshTransport:CentralGrpcEndpoints in order and
STOPS AT THE FIRST SUCCESS. That single `return` is the whole single-count
guarantee: the transport it replaces was ClusterClient.Send, deliberately not
SendToAll, because both central nodes forward to the SAME
ConfigPublishCoordinator singleton proxy — an ack posted to both would have the
coordinator count one node twice and could seal a deployment on a doubled count.
The list is an availability list, not a fan-out list.
No retry beyond that one ordered pass, no queue: the receptionist buffered
nothing and retried nothing, and this is like-for-like, not an upgrade to
delivery guarantees. All endpoints failing is still a DROP — but a loud one,
naming the node, deployment, correlation id and the status each endpoint
answered with, which is what separates "central is down" from "central rejected
this node's key". The bearer never reaches a log; only endpoints and status
codes do. Per-endpoint GrpcChannel cache and the endpoint loop clone
GrpcDeploymentArtifactFetcher; the cluster id in x-otopcua-cluster comes from
this node's own cluster-<id> role via RoleParser, off the same bound
Cluster:Roles ClusterRoleInfo reads.
SendAsync never throws: the caller discards the task, so an escaping exception
would become an unobserved fault — a drop with no log line, the exact failure
mode this migration removes.
NodeCommunicationActor gains an OPTIONAL ackSender; every existing test compiles
and passes unedited. It is checked before the ClusterClient so the two can never
both fire. The registration arm creates no ClusterClient under Grpc.
NOTE — one wiring change beyond the actor: DriverHostActor's ackRouter is now
`useClusterClient || useGrpc ? nodeComm : null`. Left at ClusterClient-only, a
Grpc node's ack would have gone out on the DPS deployment-acks topic and never
reached the sender at all.
Phase 2 Task 4. Node-side end of the boundary, registered with the receptionist
per node rather than as a singleton so central's contact rotation reaches
whichever node answers.
Inbound commands are re-emitted on the node's EventStream, NOT back onto their
DistributedPubSub topic. DPS is mesh-wide and central SendToAll-s to every
node's comm actor, so a DPS republish would deliver N copies of every command
to every subscriber. The EventStream is node-local by construction. It also
avoids a registration handshake with actors spawned later: ScriptedAlarmHostActor
is a CHILD of DriverHostActor and has no registry key, so there is no ref to
hand in at wiring time.
Each inbound type is listed explicitly rather than caught by a ReceiveAny, so an
unknown type dead-letters loudly instead of being republished blind onto a
stream where every subscriber ignores it.
Outbound is ApplyAck only, and it uses Send rather than SendToAll -- the deploy
coordinator is one singleton behind central's proxy, so SendToAll would deliver
one ack per central node and the coordinator would count this node twice.
Notably there is NO Ask across this boundary in Phase 2: every migrated command
is fire-and-forget, which is why this actor is far smaller than the sister
project's equivalent and needs no sender preservation at all.
Sabotage-verified twice: deleting the AlarmCommand handler and flipping Send to
SendToAll each redden exactly their own test.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
Joseph Doherty