fix(controlplane,runtime): #868 second review — final-outcome node rows, unpersisted acks survive deadline and restart
M1 DriverHostActor wrote its own NodeDeploymentState row Applied/null as soon as the revision advanced, up to ~105 s before the address-space outcome. The coordinator's recovery counts any non-Applying row as an ack, so a coordinator restart in that window sealed CLEAN while the node was about to report degraded (#601). The row now stays Applying until ReleasePendingAck (or the no-publish-actor path) writes the final Applied, clean or with the degraded reason. A node restart in the window replays the orphan Applying row; /fleet shows Applying until the outcome (docs/Redundancy.md). M2 Acks whose persist failed are tracked in _unpersisted (newest per node). HandleDeadline persists that backlog first and defers its decision while any remain, so a node that acked is never timed out as silent; PreRestart re-sends them to Self so a restart (timers lost) does not drop them. L3 A successful persist cancels the node's retry timer and drops its older pending ack; a retry only fires for the node's current pending ack. L4 A dispatch queued during Loading is sent with a fresh deadline however long the outage lasted (never closed "Dispatch was lost"); the inline re-drive of a genuinely lost dispatch arms the REMAINING deadline. L5 A recovered row with a blank RevisionHash (re-dispatch) or blank NodeId is closed PartiallyFailed with a reason instead of failing every load. L6 Readiness carries PendingAckPersists + LastPersistError; the config-publish health check is Degraded while Ready with a backlog. L7 Ack-persist retries back off (shared Backoff); stack trace on the first failure only. L8 Terminal writes (Sealed / PartiallyFailed / TimedOut / failure reason) only apply to a row still Dispatching/AwaitingApplyAcks. Tests (all fail on the previous code): ControlPlane DbOutage — deadline does not time out an unpersisted ack; unpersisted ack survives a restart; stale retry does not overwrite a newer ack; queued dispatch through an outage longer than the deadline is sent; malformed row closed and coordinator carries on; readiness reports the persist backlog. Runtime DegradedAck — own row stays Applying until the outcome. Host — Ready + backlog is Degraded. Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
This commit is contained in:
@@ -2984,7 +2984,13 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
|
||||
}
|
||||
|
||||
_currentRevision = msg.Revision;
|
||||
UpsertNodeDeploymentState(msg.DeploymentId, NodeDeploymentStatus.Applied, failureReason: null);
|
||||
// lmxopcua#868 review M1: the own row stays APPLYING (written by BeginDirectApply) until the
|
||||
// address-space outcome is known — AwaitAddressSpaceOutcomeThenAck / ReleasePendingAck write the
|
||||
// final Applied (clean, or with the degraded reason). Writing Applied/null here, up to ~105 s before
|
||||
// that outcome, made the row read as a CLEAN ack: a coordinator restart in that window recovered it
|
||||
// as complete and sealed the deployment clean while this node was about to report degraded (#601).
|
||||
// A restart of THIS node in the window now replays the orphan Applying row (re-applies), which is
|
||||
// the honest answer to "outcome unknown".
|
||||
// The Applied ack is NOT sent here any more. It used to go out right now, before the
|
||||
// address-space apply below had even been dequeued, so a degraded apply (swallowed sink faults,
|
||||
// _lastApplied not advanced) still sealed the deployment green fleet-wide. The ack is now held
|
||||
@@ -3069,6 +3075,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
|
||||
if (_opcUaPublishActor is null)
|
||||
{
|
||||
_currentRevisionDegraded = false;
|
||||
UpsertNodeDeploymentState(deploymentId, NodeDeploymentStatus.Applied, failureReason: null);
|
||||
SendAck(deploymentId, ApplyAckOutcome.Applied, failureReason: null, correlation);
|
||||
return;
|
||||
}
|
||||
@@ -3126,13 +3133,14 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
|
||||
if (_currentRevision is { } cur && cur == pending.Revision)
|
||||
_currentRevisionDegraded = degraded;
|
||||
|
||||
// The own row has been Applying since the apply began (lmxopcua#868 review M1); the FINAL outcome is
|
||||
// written here, clean or degraded. Central adds the reason from the ack only when it ACCEPTS the ack (a
|
||||
// timed-out or superseded deployment's ack is discarded), so a node with a ConfigDb records it itself; a
|
||||
// driver-only node no-ops here.
|
||||
UpsertNodeDeploymentState(pending.DeploymentId, NodeDeploymentStatus.Applied,
|
||||
degraded ? ApplyAck.DegradedReasonPrefix + (reason ?? "address-space apply degraded") : null);
|
||||
if (degraded)
|
||||
{
|
||||
// The apply wrote this node's own row Applied/null before the outcome was known. Central
|
||||
// adds the reason from the ack only when it ACCEPTS the ack (a timed-out or superseded deployment's ack
|
||||
// is discarded), so a node with a ConfigDb records it itself; a driver-only node no-ops here.
|
||||
UpsertNodeDeploymentState(pending.DeploymentId, NodeDeploymentStatus.Applied,
|
||||
ApplyAck.DegradedReasonPrefix + (reason ?? "address-space apply degraded"));
|
||||
OtOpcUaTelemetry.DeploymentApplied.Add(1, new KeyValuePair<string, object?>("outcome", "degraded"));
|
||||
_log.Warning(
|
||||
"DriverHost {Node}: deployment {Id} (rev {Rev}) applied DEGRADED — {Reason}. Acking Applied+degraded; re-dispatching the same revision re-applies it",
|
||||
|
||||
Reference in New Issue
Block a user