fix(controlplane,runtime): #868 third review — bounded failure reasons, poison ack writes, cache-booted revisions unconfirmed

1 PersistNodeAck wrote a Failed ack's FailureReason unbounded; the column is
  nvarchar(2048), so on SQL Server the write failed forever, the ack retried
  forever, the deadline deferred forever and every later deploy was refused.
  - One shared bound, Configuration.FailureReasons.Truncate (MaxLength 2048),
    used by the coordinator and DriverHostActor.UpsertNodeDeploymentState
    (exception messages and degraded reasons).
  - A non-transient DbUpdateException (not concurrency, no transient
    DbException / timeout / socket / IO underneath) is POISON: the ack is
    counted from memory with an Error, never retried.
  - Deadline deferral on unpersisted acks is capped at 10 x the retry ceiling;
    past it the outcome is decided from the acks held in memory.
2 A node restarting in the outcome window boots the cached revision (cache is
  written before the outcome) and never acks. Cache-booted revisions
  (BootstrapFromCache / TryBootFromCache) are now marked unconfirmed, so a
  re-dispatch re-applies and acks the real outcome instead of a clean
  short-circuit. docs/Redundancy.md corrected (replay only on the ConfigDb boot
  path).
3 A lost dispatch re-driven near its deadline now gets a FULL deadline (a node
  needs its whole outcome timeout + margin = ApplyDeadline, so any shorter
  remainder timed out nodes that applied). No dispatched-at column: a second
  restart recomputes from CreatedAtUtc and closes it instead (documented).
4 docs/Redundancy.md upgrade note: do not deploy while the central pair is mid
  rolling-upgrade across this change.

Tests (all fail on the previous code): oversized Failed reason truncated (on a
SaveChanges interceptor that enforces the column like SQL Server); rejected ack
write counted from memory; bounded deadline deferral; full deadline on a
near-deadline re-drive; cache-booted revision re-applies on re-dispatch;
oversized degraded reason truncated on the node's own row.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
This commit is contained in:
Joseph Doherty
2026-11-11 13:13:36 -05:00
parent 74496b4fec
commit 780c87eba3
8 changed files with 378 additions and 24 deletions
@@ -0,0 +1,20 @@
namespace ZB.MOM.WW.OtOpcUa.Configuration;
/// <summary>
/// The <c>FailureReason</c> column bound shared by <c>Deployment</c> and <c>NodeDeploymentState</c>
/// (<c>nvarchar(2048)</c>, <see cref="OtOpcUaConfigDbContext"/>). Every writer of either column goes through
/// <see cref="Truncate"/>: a reason built from free text (an exception message, a node's reported reason) that
/// overflows it makes SQL Server refuse the whole <c>SaveChanges</c>, and a write that can never succeed wedges
/// whatever is retrying it (lmxopcua#868).
/// </summary>
public static class FailureReasons
{
/// <summary>The column's maximum length.</summary>
public const int MaxLength = 2048;
/// <summary>Bounds a reason to <see cref="MaxLength"/> (ellipsis-terminated when cut); null stays null.</summary>
/// <param name="reason">The reason, or null.</param>
/// <returns>The reason, fitting the column.</returns>
public static string? Truncate(string? reason) =>
reason is null || reason.Length <= MaxLength ? reason : reason[..(MaxLength - 1)] + "…";
}