fix(controlplane,runtime): #868 third review — bounded failure reasons, poison ack writes, cache-booted revisions unconfirmed
1 PersistNodeAck wrote a Failed ack's FailureReason unbounded; the column is
nvarchar(2048), so on SQL Server the write failed forever, the ack retried
forever, the deadline deferred forever and every later deploy was refused.
- One shared bound, Configuration.FailureReasons.Truncate (MaxLength 2048),
used by the coordinator and DriverHostActor.UpsertNodeDeploymentState
(exception messages and degraded reasons).
- A non-transient DbUpdateException (not concurrency, no transient
DbException / timeout / socket / IO underneath) is POISON: the ack is
counted from memory with an Error, never retried.
- Deadline deferral on unpersisted acks is capped at 10 x the retry ceiling;
past it the outcome is decided from the acks held in memory.
2 A node restarting in the outcome window boots the cached revision (cache is
written before the outcome) and never acks. Cache-booted revisions
(BootstrapFromCache / TryBootFromCache) are now marked unconfirmed, so a
re-dispatch re-applies and acks the real outcome instead of a clean
short-circuit. docs/Redundancy.md corrected (replay only on the ConfigDb boot
path).
3 A lost dispatch re-driven near its deadline now gets a FULL deadline (a node
needs its whole outcome timeout + margin = ApplyDeadline, so any shorter
remainder timed out nodes that applied). No dispatched-at column: a second
restart recomputes from CreatedAtUtc and closes it instead (documented).
4 docs/Redundancy.md upgrade note: do not deploy while the central pair is mid
rolling-upgrade across this change.
Tests (all fail on the previous code): oversized Failed reason truncated (on a
SaveChanges interceptor that enforces the column like SQL Server); rejected ack
write counted from memory; bounded deadline deferral; full deadline on a
near-deadline re-drive; cache-booted revision re-applies on re-dispatch;
oversized degraded reason truncated on the node's own row.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
This commit is contained in:
+12
-2
@@ -679,8 +679,18 @@ deployment green fleet-wide. Now:
|
||||
**That row stays `Applying` until the outcome is known** (lmxopcua#868): it used to be written `Applied`/clean
|
||||
the moment the revision advanced, and `ConfigPublishCoordinator`'s recovery counts any non-`Applying` row as
|
||||
an ack, so a coordinator restart inside the ~105 s outcome window sealed the deployment clean while the node
|
||||
was about to report degraded. `/fleet` therefore shows the node *Applying* until its outcome lands; a restart
|
||||
of the node in that window replays the orphan `Applying` row (re-applies) instead of restoring it.
|
||||
was about to report degraded. `/fleet` therefore shows the node *Applying* until its outcome lands.
|
||||
A restart of the node inside that window depends on how it boots:
|
||||
- a `Direct` node that can read the ConfigDb finds the orphan `Applying` row and **replays** the apply, which
|
||||
acks the real outcome;
|
||||
- a node that boots **from its LocalDb cache** (every `FetchAndCache` driver-only node, or a `Direct` node
|
||||
whose ConfigDb is unreachable) restores the cached revision — the cache is written before the outcome — and
|
||||
sends **no ack**, so central times that deployment out even though the node runs it. The restored revision
|
||||
is marked *unconfirmed*, so **re-deploying it re-applies** and acks the real outcome rather than
|
||||
short-circuiting a clean `Applied`.
|
||||
- **Upgrade note (lmxopcua#868): do not deploy while the central pair is mid rolling-upgrade across this
|
||||
change.** An old-build fused central node still writes its own row `Applied` before the outcome, which a
|
||||
new-build coordinator's recovery would count as a clean ack. Finish upgrading both central nodes, then deploy.
|
||||
- **Mixed-version fleets.** An old node never sets the flag, so its acks are treated exactly as before
|
||||
(clean). A new node acking an old central: the old coordinator ignores the unknown field and seals as it
|
||||
always did — no worse than today.
|
||||
|
||||
Reference in New Issue
Block a user