fix(controlplane,runtime): #868 fourth review — SQL poison allowlist, memory-counted acks survive restart, adopted revisions unconfirmed

M1 IsPoison classified every SQL Server transient as poison:
   Microsoft.Data.SqlClient never sets DbException.IsTransient, so a deadlock
   (1205), timeout (-2 / Win32 258), dropped connection or AG failover was
   counted from memory and the node's row left Applying. Poison is now an
   ALLOWLIST: a DbUpdateException (not concurrency) carrying a SqlException
   whose number is 8152, 2628, 547, 515, 2627 or 2601. Everything else
   retries, bounded by the deadline deferral cap. IsPoison is public for tests.
   Acks counted only from memory (poison branch and the deferral cap) are kept
   in _memoryOnlyAcks and re-sent by PreRestart alongside _unpersisted, so a
   restart before the terminal write does not lose them.
M2 Revisions adopted without an outcome-reporting apply are unconfirmed, so a
   same-revision dispatch re-applies instead of short-circuiting a clean
   Applied over an empty address space: TryRecoverFromStale (restores only the
   revision marker) and RestoreApplied when the artifact read fails (null
   blob, LoadArtifactBlob swallows the error) or the restore throws.
L1 The re-drive comment documents the CreatedAtUtc recompute after a second
   restart / failover.
L2 _currentRevisionDegraded renamed _currentRevisionUnconfirmed; the stale
   TryBootFromCache remark corrected.
L3 docs/Redundancy.md: a Direct node running from cache that still cannot
   reach the ConfigDb acks a same-revision re-dispatch Failed
   (PartiallyFailed, not Sealed); Stale recovery and failed restores are
   unconfirmed too.

Test harness: SqlExceptionFactory builds real SqlExceptions (reflection on
SqlError / SqlErrorCollection / SqlException.CreateException);
ConstraintEnforcingDbFactory now throws DbUpdateException over SQL 2628
(over-length), 547 (poison) and 1205 (transient), plus FailNextContexts.

Tests (all fail on the previous code): poison allowlist theory (1205, 40613,
10054, 4060 not poison) + timeout/concurrency/providerless not poison; a
deadlocked ack write is retried and recorded; a memory-counted ack survives a
coordinator restart; Stale-recovered and failed-restore revisions re-apply on
re-dispatch.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
This commit is contained in:
Joseph Doherty
2026-11-11 13:31:49 -05:00
parent ca6d131078
commit ec3b835e4a
7 changed files with 367 additions and 30 deletions
@@ -8,9 +8,10 @@ namespace ZB.MOM.WW.OtOpcUa.ControlPlane.Tests.Harness;
/// <summary>
/// An in-memory ConfigDb that behaves like SQL Server where the InMemory provider does not (lmxopcua#868 third
/// review): a <c>FailureReason</c> longer than the column refuses the whole <c>SaveChanges</c> with a
/// <see cref="DbUpdateException"/>. It can also reject (<see cref="PoisonNodeStateUpdates"/>, a non-transient
/// <see cref="DbUpdateException"/>) or fail transiently (<see cref="FailNodeStateUpdatesTransiently"/>) every
/// update of an existing <c>NodeDeploymentState</c> row.
/// <see cref="DbUpdateException"/> over SQL error 2628. It can also reject (<see cref="PoisonNodeStateUpdates"/>,
/// error 547) or fail transiently (<see cref="FailNodeStateUpdatesTransiently"/>, deadlock 1205) every update of
/// an existing <c>NodeDeploymentState</c> row — each a real <c>SqlException</c> inside EF's
/// <see cref="DbUpdateException"/>, the shape SQL Server produces (fourth review M1).
/// </summary>
public sealed class ConstraintEnforcingDbFactory(string dbName) : IDbContextFactory<OtOpcUaConfigDbContext>
{
@@ -22,8 +23,18 @@ public sealed class ConstraintEnforcingDbFactory(string dbName) : IDbContextFact
/// <summary>Every update of an existing node row fails as if the connection dropped.</summary>
public bool FailNodeStateUpdatesTransiently { get => _interceptor.Transient; set => _interceptor.Transient = value; }
private int _failNextContexts;
/// <summary>The next <paramref name="count"/> context creations throw (an unreachable server).</summary>
public void FailNextContexts(int count) => Volatile.Write(ref _failNextContexts, count);
/// <inheritdoc />
public OtOpcUaConfigDbContext CreateDbContext() => new(new DbContextOptionsBuilder<OtOpcUaConfigDbContext>()
public OtOpcUaConfigDbContext CreateDbContext() =>
Volatile.Read(ref _failNextContexts) > 0 && Interlocked.Decrement(ref _failNextContexts) >= 0
? throw new InvalidOperationException("ConfigDb unreachable (simulated)")
: Create();
private OtOpcUaConfigDbContext Create() => new(new DbContextOptionsBuilder<OtOpcUaConfigDbContext>()
.UseInMemoryDatabase(dbName)
.AddInterceptors(_interceptor)
.Options);
@@ -57,11 +68,16 @@ public sealed class ConstraintEnforcingDbFactory(string dbName) : IDbContextFact
_ => null,
};
if (reason is { Length: > FailureReasons.MaxLength })
throw new DbUpdateException("String or binary data would be truncated in column 'FailureReason'.");
throw new DbUpdateException("An error occurred while saving the entity changes.",
SqlExceptionFactory.Create(2628, "String or binary data would be truncated in column 'FailureReason'."));
if (e.Entity is NodeDeploymentState && e.State == EntityState.Modified)
{
if (Poison) throw new DbUpdateException("The UPDATE statement conflicted with a CHECK constraint.");
if (Transient) throw new InvalidOperationException("A network-related error occurred (simulated).");
if (Poison)
throw new DbUpdateException("An error occurred while saving the entity changes.",
SqlExceptionFactory.Create(547, "The UPDATE statement conflicted with a CHECK constraint."));
if (Transient)
throw new DbUpdateException("An error occurred while saving the entity changes.",
SqlExceptionFactory.Create(1205, "Transaction was deadlocked on lock resources and has been chosen as the deadlock victim."));
}
}
}