fix(mesh): per-endpoint deadline + review fixes — failover survives a hanging central
CRITICAL. The ack deadline was one absolute DateTime computed once for the whole endpoint walk. Against the characteristic WAN failure — a central that HANGS (SYN black-hole, frozen host) rather than refusing — endpoint 1 burned the whole AckTimeoutSeconds and endpoint 2 was then dialled with a deadline already in the past: instant DeadlineExceeded, no packet leaving the node, the surviving central never really tried. Failover looked like it ran and did not. It also contradicted MeshTransportOptions' own documented "applied per endpoint, not per ack". The deadline is now computed fresh inside the loop; headers stay hoisted (read-only, sequential walk). Two test gaps that let that survive are closed: nothing pinned the ackRouter gate (narrowing it back to ClusterClient-only compiled and passed every test while dead-ending every Grpc ack to DPS), and the deadline fact's lower bound accepted any positive timeout at all. Both new pins are mutation-verified. Docs corrected where they overstated the case: duplicate acks CANNOT mis-seal a deployment — ConfigPublishCoordinator.HandleAck keys _acks by NodeId behind an expected-set guard, so a duplicate overwrites its own entry. The single-endpoint rule is defence in depth and a bound on wasted work, not a correctness dependency; and the walk is at-least-once on the deadline path, not at-most-once.
This commit is contained in:
@@ -28,14 +28,16 @@ namespace ZB.MOM.WW.OtOpcUa.AdminUI.Grpc;
|
||||
/// <b>The actor is resolved per call, not in the constructor.</b> Kestrel answers before
|
||||
/// <c>Akka.Hosting</c> has finished spawning the actor graph, so a ctor lookup would throw
|
||||
/// during the boot window and surface as an opaque <c>Unknown</c>. Resolving per call turns
|
||||
/// that window into <see cref="StatusCode.Unavailable"/> — the retryable status — so the
|
||||
/// node's next attempt lands rather than the deploy recording a permanent failure against a
|
||||
/// node that applied fine.
|
||||
/// that window into <see cref="StatusCode.Unavailable"/> — the retryable status. The node's
|
||||
/// sender does not in fact retry this endpoint; it advances to the next central endpoint in
|
||||
/// its list, where the ack lands on a node whose actor graph is up. Either way the deploy does
|
||||
/// not record a permanent failure against a node that applied fine.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Decoding is graded before the registry.</b> A message that cannot decode is
|
||||
/// <see cref="StatusCode.InvalidArgument"/> even when the actor is also missing: Unavailable
|
||||
/// invites a retry, and retrying a message that can never decode is a loop. The decode itself
|
||||
/// is the status that says "try elsewhere", and moving a message that can never decode to
|
||||
/// another endpoint only spends the second one too. The decode itself
|
||||
/// never guesses (see <see cref="MeshTransportDtoMapper"/>) — an unspecified outcome, an
|
||||
/// unparseable id or a blank node name is refused rather than resolved to a plausible value,
|
||||
/// because the deploy path seals a revision on <c>Applied</c> and a guessed ack seals a
|
||||
@@ -105,8 +107,8 @@ public sealed class CentralCommandGrpcService : GeneratedServiceBase
|
||||
_log.LogWarning(
|
||||
"Received an ApplyAck for {DeploymentId} (correlation '{CorrelationId}') from {NodeId} "
|
||||
+ "at {Peer} before the central communication actor was registered on this node; "
|
||||
+ "refusing with Unavailable so the caller retries rather than recording a failure "
|
||||
+ "against a node that applied",
|
||||
+ "refusing with Unavailable so the caller moves on to another central endpoint "
|
||||
+ "rather than recording a failure against a node that applied",
|
||||
ack.DeploymentId, ack.CorrelationId, ack.NodeId, context.Peer);
|
||||
throw new RpcException(new Status(StatusCode.Unavailable, "central command actor not ready"));
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user