fix(controlplane,runtime): #868 fourth review — SQL poison allowlist, memory-counted acks survive restart, adopted revisions unconfirmed

M1 IsPoison classified every SQL Server transient as poison:
   Microsoft.Data.SqlClient never sets DbException.IsTransient, so a deadlock
   (1205), timeout (-2 / Win32 258), dropped connection or AG failover was
   counted from memory and the node's row left Applying. Poison is now an
   ALLOWLIST: a DbUpdateException (not concurrency) carrying a SqlException
   whose number is 8152, 2628, 547, 515, 2627 or 2601. Everything else
   retries, bounded by the deadline deferral cap. IsPoison is public for tests.
   Acks counted only from memory (poison branch and the deferral cap) are kept
   in _memoryOnlyAcks and re-sent by PreRestart alongside _unpersisted, so a
   restart before the terminal write does not lose them.
M2 Revisions adopted without an outcome-reporting apply are unconfirmed, so a
   same-revision dispatch re-applies instead of short-circuiting a clean
   Applied over an empty address space: TryRecoverFromStale (restores only the
   revision marker) and RestoreApplied when the artifact read fails (null
   blob, LoadArtifactBlob swallows the error) or the restore throws.
L1 The re-drive comment documents the CreatedAtUtc recompute after a second
   restart / failover.
L2 _currentRevisionDegraded renamed _currentRevisionUnconfirmed; the stale
   TryBootFromCache remark corrected.
L3 docs/Redundancy.md: a Direct node running from cache that still cannot
   reach the ConfigDb acks a same-revision re-dispatch Failed
   (PartiallyFailed, not Sealed); Stale recovery and failed restores are
   unconfirmed too.

Test harness: SqlExceptionFactory builds real SqlExceptions (reflection on
SqlError / SqlErrorCollection / SqlException.CreateException);
ConstraintEnforcingDbFactory now throws DbUpdateException over SQL 2628
(over-length), 547 (poison) and 1205 (transient), plus FailNextContexts.

Tests (all fail on the previous code): poison allowlist theory (1205, 40613,
10054, 4060 not poison) + timeout/concurrency/providerless not poison; a
deadlocked ack write is retried and recorded; a memory-counted ack survives a
coordinator restart; Stale-recovered and failed-restore revisions re-apply on
re-dispatch.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
This commit is contained in:
Joseph Doherty
2026-11-11 13:31:49 -05:00
parent ca6d131078
commit ec3b835e4a
7 changed files with 367 additions and 30 deletions
+7 -1
View File
@@ -687,7 +687,13 @@ deployment green fleet-wide. Now:
whose ConfigDb is unreachable) restores the cached revision — the cache is written before the outcome — and
sends **no ack**, so central times that deployment out even though the node runs it. The restored revision
is marked *unconfirmed*, so **re-deploying it re-applies** and acks the real outcome rather than
short-circuiting a clean `Applied`.
short-circuiting a clean `Applied`. ⚠️ A `Direct` node running from cache that **still cannot reach the
ConfigDb** cannot read the artifact to re-apply, so it acks that same-revision re-dispatch **`Failed`** and
the deployment reads **`PartiallyFailed`**, not `Sealed` — the honest answer; re-deploy once the node reaches
the ConfigDb again.
- The same *unconfirmed* marking applies to a revision adopted by **Stale recovery** (which restores only the
revision marker, not drivers or the address space) and to one whose **bootstrap restore failed**: a dispatch
of it re-applies instead of acking clean over an empty address space.
- **Upgrade note (lmxopcua#868): do not deploy while the central pair is mid rolling-upgrade across this
change.** An old-build fused central node still writes its own row `Applied` before the outcome, which a
new-build coordinator's recovery would count as a clean ack. Finish upgrading both central nodes, then deploy.