fix(mesh): per-endpoint deadline + review fixes — failover survives a hanging central

CRITICAL. The ack deadline was one absolute DateTime computed once for the whole
endpoint walk. Against the characteristic WAN failure — a central that HANGS
(SYN black-hole, frozen host) rather than refusing — endpoint 1 burned the whole
AckTimeoutSeconds and endpoint 2 was then dialled with a deadline already in the
past: instant DeadlineExceeded, no packet leaving the node, the surviving central
never really tried. Failover looked like it ran and did not. It also contradicted
MeshTransportOptions' own documented "applied per endpoint, not per ack". The
deadline is now computed fresh inside the loop; headers stay hoisted (read-only,
sequential walk).

Two test gaps that let that survive are closed: nothing pinned the ackRouter gate
(narrowing it back to ClusterClient-only compiled and passed every test while
dead-ending every Grpc ack to DPS), and the deadline fact's lower bound accepted
any positive timeout at all. Both new pins are mutation-verified.

Docs corrected where they overstated the case: duplicate acks CANNOT mis-seal a
deployment — ConfigPublishCoordinator.HandleAck keys _acks by NodeId behind an
expected-set guard, so a duplicate overwrites its own entry. The single-endpoint
rule is defence in depth and a bound on wasted work, not a correctness
dependency; and the walk is at-least-once on the deadline path, not at-most-once.
This commit is contained in:
Joseph Doherty
2026-08-06 06:15:23 -04:00
parent 308c6aaa0f
commit 667639e6ab
7 changed files with 267 additions and 43 deletions
@@ -374,8 +374,9 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
/// <param name="dbFactory">Database context factory for configuration database access.</param>
/// <param name="localNode">The local cluster node identifier.</param>
/// <param name="ackRouter">Where ApplyAcks are sent. A direct coordinator handle in tests; under
/// <c>MeshTransport:Mode = ClusterClient</c> the node's <c>NodeCommunicationActor</c>, which relays
/// across the boundary. Null publishes on the <c>deployment-acks</c> DPS topic.</param>
/// <c>MeshTransport:Mode = ClusterClient</c> <b>and</b> <c>= Grpc</c> the node's
/// <c>NodeCommunicationActor</c>, which relays across the boundary over whichever transport it was
/// built with. Null publishes on the <c>deployment-acks</c> DPS topic.</param>
/// <param name="driverFactory">Optional driver factory; defaults to null factory if not provided.</param>
/// <param name="localRoles">Optional set of roles assigned to the local node.</param>
/// <param name="dependencyMux">Optional actor reference for dependency multiplexing.</param>
@@ -458,8 +459,9 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
/// <param name="dbFactory">Database context factory for configuration database access.</param>
/// <param name="localNode">The local cluster node identifier.</param>
/// <param name="ackRouter">Where ApplyAcks are sent. A direct coordinator handle in tests; under
/// <c>MeshTransport:Mode = ClusterClient</c> the node's <c>NodeCommunicationActor</c>, which relays
/// across the boundary. Null publishes on the <c>deployment-acks</c> DPS topic.</param>
/// <c>MeshTransport:Mode = ClusterClient</c> <b>and</b> <c>= Grpc</c> the node's
/// <c>NodeCommunicationActor</c>, which relays across the boundary over whichever transport it was
/// built with. Null publishes on the <c>deployment-acks</c> DPS topic.</param>
/// <param name="driverFactory">Optional driver factory; defaults to null factory if not provided.</param>
/// <param name="localRoles">Optional set of roles assigned to the local node.</param>
/// <param name="dependencyMux">Optional actor reference for dependency multiplexing.</param>
@@ -2490,11 +2492,11 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers
var ack = new ApplyAck(deploymentId, _localNode, outcome, failureReason, correlation);
if (_ackRouter is not null)
{
// Either a direct coordinator handle (tests) or, under
// MeshTransport:Mode = ClusterClient, the NodeCommunicationActor that relays this
// across the boundary. Renamed from _coordinatorOverride in per-cluster mesh Phase 2:
// under ClusterClient mode this ref is emphatically NOT a coordinator, and the old
// name would send the next reader looking for one.
// Either a direct coordinator handle (tests) or, under MeshTransport:Mode = ClusterClient
// and = Grpc alike, the NodeCommunicationActor that relays this across the boundary (it
// owns the choice of transport, not this actor). Renamed from _coordinatorOverride in
// per-cluster mesh Phase 2: under those two modes this ref is emphatically NOT a
// coordinator, and the old name would send the next reader looking for one.
_ackRouter.Tell(ack);
}
else