fix(controlplane,runtime): #868 second review — final-outcome node rows, unpersisted acks survive deadline and restart

M1 DriverHostActor wrote its own NodeDeploymentState row Applied/null as soon
   as the revision advanced, up to ~105 s before the address-space outcome.
   The coordinator's recovery counts any non-Applying row as an ack, so a
   coordinator restart in that window sealed CLEAN while the node was about to
   report degraded (#601). The row now stays Applying until ReleasePendingAck
   (or the no-publish-actor path) writes the final Applied, clean or with the
   degraded reason. A node restart in the window replays the orphan Applying
   row; /fleet shows Applying until the outcome (docs/Redundancy.md).
M2 Acks whose persist failed are tracked in _unpersisted (newest per node).
   HandleDeadline persists that backlog first and defers its decision while
   any remain, so a node that acked is never timed out as silent; PreRestart
   re-sends them to Self so a restart (timers lost) does not drop them.
L3 A successful persist cancels the node's retry timer and drops its older
   pending ack; a retry only fires for the node's current pending ack.
L4 A dispatch queued during Loading is sent with a fresh deadline however long
   the outage lasted (never closed "Dispatch was lost"); the inline re-drive of
   a genuinely lost dispatch arms the REMAINING deadline.
L5 A recovered row with a blank RevisionHash (re-dispatch) or blank NodeId is
   closed PartiallyFailed with a reason instead of failing every load.
L6 Readiness carries PendingAckPersists + LastPersistError; the config-publish
   health check is Degraded while Ready with a backlog.
L7 Ack-persist retries back off (shared Backoff); stack trace on the first
   failure only.
L8 Terminal writes (Sealed / PartiallyFailed / TimedOut / failure reason) only
   apply to a row still Dispatching/AwaitingApplyAcks.

Tests (all fail on the previous code): ControlPlane DbOutage — deadline does
not time out an unpersisted ack; unpersisted ack survives a restart; stale
retry does not overwrite a newer ack; queued dispatch through an outage longer
than the deadline is sent; malformed row closed and coordinator carries on;
readiness reports the persist backlog. Runtime DegradedAck — own row stays
Applying until the outcome. Host — Ready + backlog is Degraded.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
This commit is contained in:
Joseph Doherty
2026-11-11 12:54:30 -05:00
parent ff85b8511a
commit 494dcb5b87
8 changed files with 475 additions and 53 deletions
+5
View File
@@ -676,6 +676,11 @@ deployment green fleet-wide. Now:
no second rebuild), reporting clean only if that pass is clean.
- **A node with a ConfigDb also writes the reason on its own row** (`ApplyAck.DegradedReasonPrefix`), because
central only records it when it *accepts* the ack — a timed-out deployment's late ack is discarded.
**That row stays `Applying` until the outcome is known** (lmxopcua#868): it used to be written `Applied`/clean
the moment the revision advanced, and `ConfigPublishCoordinator`'s recovery counts any non-`Applying` row as
an ack, so a coordinator restart inside the ~105 s outcome window sealed the deployment clean while the node
was about to report degraded. `/fleet` therefore shows the node *Applying* until its outcome lands; a restart
of the node in that window replays the orphan `Applying` row (re-applies) instead of restoring it.
- **Mixed-version fleets.** An old node never sets the flag, so its acks are treated exactly as before
(clean). A new node acking an old central: the old coordinator ignores the unknown field and seals as it
always did — no worse than today.