fix(mesh): per-endpoint deadline + review fixes — failover survives a hanging central

CRITICAL. The ack deadline was one absolute DateTime computed once for the whole
endpoint walk. Against the characteristic WAN failure — a central that HANGS
(SYN black-hole, frozen host) rather than refusing — endpoint 1 burned the whole
AckTimeoutSeconds and endpoint 2 was then dialled with a deadline already in the
past: instant DeadlineExceeded, no packet leaving the node, the surviving central
never really tried. Failover looked like it ran and did not. It also contradicted
MeshTransportOptions' own documented "applied per endpoint, not per ack". The
deadline is now computed fresh inside the loop; headers stay hoisted (read-only,
sequential walk).

Two test gaps that let that survive are closed: nothing pinned the ackRouter gate
(narrowing it back to ClusterClient-only compiled and passed every test while
dead-ending every Grpc ack to DPS), and the deadline fact's lower bound accepted
any positive timeout at all. Both new pins are mutation-verified.

Docs corrected where they overstated the case: duplicate acks CANNOT mis-seal a
deployment — ConfigPublishCoordinator.HandleAck keys _acks by NodeId behind an
expected-set guard, so a duplicate overwrites its own entry. The single-endpoint
rule is defence in depth and a bound on wasted work, not a correctness
dependency; and the walk is at-least-once on the deadline path, not at-most-once.
This commit is contained in:
Joseph Doherty
2026-08-06 06:15:23 -04:00
parent 308c6aaa0f
commit 667639e6ab
7 changed files with 267 additions and 43 deletions
@@ -28,14 +28,16 @@ namespace ZB.MOM.WW.OtOpcUa.AdminUI.Grpc;
/// <b>The actor is resolved per call, not in the constructor.</b> Kestrel answers before
/// <c>Akka.Hosting</c> has finished spawning the actor graph, so a ctor lookup would throw
/// during the boot window and surface as an opaque <c>Unknown</c>. Resolving per call turns
/// that window into <see cref="StatusCode.Unavailable"/> — the retryable status — so the
/// node's next attempt lands rather than the deploy recording a permanent failure against a
/// node that applied fine.
/// that window into <see cref="StatusCode.Unavailable"/> — the retryable status. The node's
/// sender does not in fact retry this endpoint; it advances to the next central endpoint in
/// its list, where the ack lands on a node whose actor graph is up. Either way the deploy does
/// not record a permanent failure against a node that applied fine.
/// </para>
/// <para>
/// <b>Decoding is graded before the registry.</b> A message that cannot decode is
/// <see cref="StatusCode.InvalidArgument"/> even when the actor is also missing: Unavailable
/// invites a retry, and retrying a message that can never decode is a loop. The decode itself
/// is the status that says "try elsewhere", and moving a message that can never decode to
/// another endpoint only spends the second one too. The decode itself
/// never guesses (see <see cref="MeshTransportDtoMapper"/>) — an unspecified outcome, an
/// unparseable id or a blank node name is refused rather than resolved to a plausible value,
/// because the deploy path seals a revision on <c>Applied</c> and a guessed ack seals a
@@ -105,8 +107,8 @@ public sealed class CentralCommandGrpcService : GeneratedServiceBase
_log.LogWarning(
"Received an ApplyAck for {DeploymentId} (correlation '{CorrelationId}') from {NodeId} "
+ "at {Peer} before the central communication actor was registered on this node; "
+ "refusing with Unavailable so the caller retries rather than recording a failure "
+ "against a node that applied",
+ "refusing with Unavailable so the caller moves on to another central endpoint "
+ "rather than recording a failure against a node that applied",
ack.DeploymentId, ack.CorrelationId, ack.NodeId, context.Peer);
throw new RpcException(new Status(StatusCode.Unavailable, "central command actor not ready"));
}