fix(mesh): per-endpoint deadline + review fixes — failover survives a hanging central

CRITICAL. The ack deadline was one absolute DateTime computed once for the whole endpoint walk. Against the characteristic WAN failure — a central that HANGS (SYN black-hole, frozen host) rather than refusing — endpoint 1 burned the whole AckTimeoutSeconds and endpoint 2 was then dialled with a deadline already in the past: instant DeadlineExceeded, no packet leaving the node, the surviving central never really tried. Failover looked like it ran and did not. It also contradicted MeshTransportOptions' own documented "applied per endpoint, not per ack". The deadline is now computed fresh inside the loop; headers stay hoisted (read-only, sequential walk). Two test gaps that let that survive are closed: nothing pinned the ackRouter gate (narrowing it back to ClusterClient-only compiled and passed every test while dead-ending every Grpc ack to DPS), and the deadline fact's lower bound accepted any positive timeout at all. Both new pins are mutation-verified. Docs corrected where they overstated the case: duplicate acks CANNOT mis-seal a deployment — ConfigPublishCoordinator.HandleAck keys _acks by NodeId behind an expected-set guard, so a duplicate overwrites its own entry. The single-endpoint rule is defence in depth and a bound on wasted work, not a correctness dependency; and the walk is at-least-once on the deadline path, not at-most-once.
2026-08-06 06:15:23 -04:00
parent 308c6aaa0f
commit 667639e6ab
7 changed files with 267 additions and 43 deletions
@@ -92,8 +92,26 @@ public sealed class NodeCommunicationActor : ReceiveActor
            //
            // Checked BEFORE the ClusterClient. The registration arm builds exactly one transport, so
            // both are never set — but were they, posting over both would deliver two acks for one
-            // apply and the coordinator would count this node twice.
-            _ = _ackSender.SendAsync(ack);
+            // apply and duplicate central's work for no benefit.
+            try
+            {
+                _ = _ackSender.SendAsync(ack);
+            }
+            catch (Exception ex)
+            {
+                // "Never throws" is a contract on a public seam, which makes it documentation until
+                // something enforces it. An implementation that threw synchronously — a guard clause
+                // ahead of the async state machine is the easy way to do it by accident — would fault
+                // this Receive, restart the actor, and take the node's whole inbound command path down
+                // with it. One dropped ack is a deployment that times out; a restarting comm actor is
+                // every command to this node going missing.
+                _log.Error(
+                    ex,
+                    "The ApplyAck sender threw synchronously for {DeploymentId}; dropping the ack. "
+                    + "IApplyAckSender must never throw — this is a defect in the registered sender.",
+                    ack.DeploymentId);
+            }
+
            return;
        }