fix(controlplane,runtime): #868 second review — final-outcome node rows, unpersisted acks survive deadline and restart

M1 DriverHostActor wrote its own NodeDeploymentState row Applied/null as soon
   as the revision advanced, up to ~105 s before the address-space outcome.
   The coordinator's recovery counts any non-Applying row as an ack, so a
   coordinator restart in that window sealed CLEAN while the node was about to
   report degraded (#601). The row now stays Applying until ReleasePendingAck
   (or the no-publish-actor path) writes the final Applied, clean or with the
   degraded reason. A node restart in the window replays the orphan Applying
   row; /fleet shows Applying until the outcome (docs/Redundancy.md).
M2 Acks whose persist failed are tracked in _unpersisted (newest per node).
   HandleDeadline persists that backlog first and defers its decision while
   any remain, so a node that acked is never timed out as silent; PreRestart
   re-sends them to Self so a restart (timers lost) does not drop them.
L3 A successful persist cancels the node's retry timer and drops its older
   pending ack; a retry only fires for the node's current pending ack.
L4 A dispatch queued during Loading is sent with a fresh deadline however long
   the outage lasted (never closed "Dispatch was lost"); the inline re-drive of
   a genuinely lost dispatch arms the REMAINING deadline.
L5 A recovered row with a blank RevisionHash (re-dispatch) or blank NodeId is
   closed PartiallyFailed with a reason instead of failing every load.
L6 Readiness carries PendingAckPersists + LastPersistError; the config-publish
   health check is Degraded while Ready with a backlog.
L7 Ack-persist retries back off (shared Backoff); stack trace on the first
   failure only.
L8 Terminal writes (Sealed / PartiallyFailed / TimedOut / failure reason) only
   apply to a row still Dispatching/AwaitingApplyAcks.

Tests (all fail on the previous code): ControlPlane DbOutage — deadline does
not time out an unpersisted ack; unpersisted ack survives a restart; stale
retry does not overwrite a newer ack; queued dispatch through an outage longer
than the deadline is sent; malformed row closed and coordinator carries on;
readiness reports the persist backlog. Runtime DegradedAck — own row stays
Applying until the outcome. Host — Ready + backlog is Degraded.

Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
This commit is contained in:
Joseph Doherty
2026-11-11 12:54:30 -05:00
parent ff85b8511a
commit 494dcb5b87
8 changed files with 475 additions and 53 deletions
@@ -2984,7 +2984,13 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
}
_currentRevision = msg.Revision;
UpsertNodeDeploymentState(msg.DeploymentId, NodeDeploymentStatus.Applied, failureReason: null);
// lmxopcua#868 review M1: the own row stays APPLYING (written by BeginDirectApply) until the
// address-space outcome is known — AwaitAddressSpaceOutcomeThenAck / ReleasePendingAck write the
// final Applied (clean, or with the degraded reason). Writing Applied/null here, up to ~105 s before
// that outcome, made the row read as a CLEAN ack: a coordinator restart in that window recovered it
// as complete and sealed the deployment clean while this node was about to report degraded (#601).
// A restart of THIS node in the window now replays the orphan Applying row (re-applies), which is
// the honest answer to "outcome unknown".
// The Applied ack is NOT sent here any more. It used to go out right now, before the
// address-space apply below had even been dequeued, so a degraded apply (swallowed sink faults,
// _lastApplied not advanced) still sealed the deployment green fleet-wide. The ack is now held
@@ -3069,6 +3075,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
if (_opcUaPublishActor is null)
{
_currentRevisionDegraded = false;
UpsertNodeDeploymentState(deploymentId, NodeDeploymentStatus.Applied, failureReason: null);
SendAck(deploymentId, ApplyAckOutcome.Applied, failureReason: null, correlation);
return;
}
@@ -3126,13 +3133,14 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
if (_currentRevision is { } cur && cur == pending.Revision)
_currentRevisionDegraded = degraded;
// The own row has been Applying since the apply began (lmxopcua#868 review M1); the FINAL outcome is
// written here, clean or degraded. Central adds the reason from the ack only when it ACCEPTS the ack (a
// timed-out or superseded deployment's ack is discarded), so a node with a ConfigDb records it itself; a
// driver-only node no-ops here.
UpsertNodeDeploymentState(pending.DeploymentId, NodeDeploymentStatus.Applied,
degraded ? ApplyAck.DegradedReasonPrefix + (reason ?? "address-space apply degraded") : null);
if (degraded)
{
// The apply wrote this node's own row Applied/null before the outcome was known. Central
// adds the reason from the ack only when it ACCEPTS the ack (a timed-out or superseded deployment's ack
// is discarded), so a node with a ConfigDb records it itself; a driver-only node no-ops here.
UpsertNodeDeploymentState(pending.DeploymentId, NodeDeploymentStatus.Applied,
ApplyAck.DegradedReasonPrefix + (reason ?? "address-space apply degraded"));
OtOpcUaTelemetry.DeploymentApplied.Add(1, new KeyValuePair<string, object?>("outcome", "degraded"));
_log.Warning(
"DriverHost {Node}: deployment {Id} (rev {Rev}) applied DEGRADED — {Reason}. Acking Applied+degraded; re-dispatching the same revision re-applies it",