fix(deploy): persist which nodes never acked on a deploy timeout (lmxopcua#784)

In the normal timeout branch (some acks landed) the deployment row was set to
TimedOut with a null FailureReason, so GET /api/deployments/{id} could not say
which node to fix; only the central log named it. The branch now persists a
reason naming the silent nodes ("Timed out waiting for acks (n/m landed). No ack
from: ..."), bounded to 20 named nodes plus a count of the rest, and still
backstopped by the 2048-char Truncate.

Tests: the reason names the silent node and not the acked one; on a 25-node
silence it names 20, counts 5 and fits the column. Mutation-checked: dropping
the SetFailureReason fails both, dropping the bound fails the second.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-09 21:16:53 -05:00
parent 43f61eba2e
commit 80118a01c5
3 changed files with 97 additions and 1 deletions
@@ -470,11 +470,14 @@ public sealed class ConfigPublishCoordinator : ReceiveActor, IWithTimers
ResetForNext();
return;
}
db.SaveChanges();
// Name the silent nodes, not just the counts. Since the expected set became DB-derived this
// is the failure an operator meets after deploying while a configured node is switched off,
// so it has to say which node — "4/5 acks landed" leaves them reading logs on five machines.
var missing = _expectedAcks.Where(n => !_acks.ContainsKey(n)).Select(n => n.Value).Order().ToList();
// lmxopcua#784: persist the same answer on the deployment row, so GET /api/deployments/{id} says which
// node to fix instead of TimedOut + a null reason (only the no-expected-rows branch above set one).
SetFailureReason(db, TimeoutFailureReason(_acks.Count, _expectedAcks.Count, missing));
db.SaveChanges();
_log.Warning(
"Deployment {Id} timed out after {Deadline} ({Acked}/{Total} acks landed). " +
"No ack from: {MissingNodes}. Each is an enabled, non-maintenance ClusterNode row — start " +
@@ -483,6 +486,29 @@ public sealed class ConfigPublishCoordinator : ReceiveActor, IWithTimers
ResetForNext();
}
/// <summary>
/// The most silent nodes a timeout reason names before summarising the rest as a count. Bounds the
/// reason on a large fleet well inside <see cref="FailureReasonMaxLength"/>; <see cref="Truncate"/>
/// still backstops pathological node ids.
/// </summary>
private const int TimeoutReasonMaxNamedNodes = 20;
/// <summary>The persisted reason for a deployment that timed out with some acks outstanding.</summary>
/// <param name="acked">Acks that landed.</param>
/// <param name="expected">Acks that were expected.</param>
/// <param name="missing">The nodes that never acked, ordered.</param>
/// <returns>A reason naming the silent nodes, bounded in length.</returns>
private static string TimeoutFailureReason(int acked, int expected, IReadOnlyList<string> missing)
{
var named = string.Join(", ", missing.Take(TimeoutReasonMaxNamedNodes));
var rest = missing.Count > TimeoutReasonMaxNamedNodes
? $" (and {missing.Count - TimeoutReasonMaxNamedNodes} more)"
: string.Empty;
return Truncate(
$"Timed out waiting for acks ({acked}/{expected} landed). No ack from: {named}{rest}. Start the node, " +
"or set ClusterNode.MaintenanceMode = 1 if it is down for maintenance, then redeploy.");
}
private void ResetForNext()
{
Timers.Cancel(DeadlineTimerKey);