fix(controlplane,runtime): #868 third review — bounded failure reasons, poison ack writes, cache-booted revisions unconfirmed
1 PersistNodeAck wrote a Failed ack's FailureReason unbounded; the column is
nvarchar(2048), so on SQL Server the write failed forever, the ack retried
forever, the deadline deferred forever and every later deploy was refused.
- One shared bound, Configuration.FailureReasons.Truncate (MaxLength 2048),
used by the coordinator and DriverHostActor.UpsertNodeDeploymentState
(exception messages and degraded reasons).
- A non-transient DbUpdateException (not concurrency, no transient
DbException / timeout / socket / IO underneath) is POISON: the ack is
counted from memory with an Error, never retried.
- Deadline deferral on unpersisted acks is capped at 10 x the retry ceiling;
past it the outcome is decided from the acks held in memory.
2 A node restarting in the outcome window boots the cached revision (cache is
written before the outcome) and never acks. Cache-booted revisions
(BootstrapFromCache / TryBootFromCache) are now marked unconfirmed, so a
re-dispatch re-applies and acks the real outcome instead of a clean
short-circuit. docs/Redundancy.md corrected (replay only on the ConfigDb boot
path).
3 A lost dispatch re-driven near its deadline now gets a FULL deadline (a node
needs its whole outcome timeout + margin = ApplyDeadline, so any shorter
remainder timed out nodes that applied). No dispatched-at column: a second
restart recomputes from CreatedAtUtc and closes it instead (documented).
4 docs/Redundancy.md upgrade note: do not deploy while the central pair is mid
rolling-upgrade across this change.
Tests (all fail on the previous code): oversized Failed reason truncated (on a
SaveChanges interceptor that enforces the column like SQL Server); rejected ack
write counted from memory; bounded deadline deferral; full deadline on a
near-deadline re-drive; cache-booted revision re-applies on re-dispatch;
oversized degraded reason truncated on the node's own row.
Claude-Session: https://claude.ai/code/session_01GT67AD99zQQqpLCVm3p7xk
This commit is contained in:
+25
@@ -56,6 +56,31 @@ public sealed class DriverHostActorBootFromCacheTests : RuntimeActorTestBase
|
||||
cache.UnkeyedReads.ShouldBe(1);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// lmxopcua#868 third review #2: the cache is written before the address-space outcome, so a cache-booted
|
||||
/// revision may never have been acked. A re-dispatch of it must RE-APPLY (here: fail honestly, since the
|
||||
/// ConfigDb is unreachable) — never short-circuit a clean Applied for an outcome nobody confirmed.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void A_cache_booted_revision_is_unconfirmed_so_a_redispatch_reapplies_instead_of_acking_clean()
|
||||
{
|
||||
var cached = new CachedDeploymentArtifact(
|
||||
DeploymentId.NewId().ToString(), CachedRev.Value, EmptyArtifact(), DateTimeOffset.UtcNow.AddMinutes(-1));
|
||||
var coordinator = CreateTestProbe();
|
||||
var actor = Sys.ActorOf(DriverHostActor.Props(
|
||||
new ThrowingDbFactory(), TestNode, ackRouter: coordinator.Ref,
|
||||
localRoles: new HashSet<string> { "driver" },
|
||||
deploymentArtifactCache: new StubArtifactCache(cached)));
|
||||
AskDiagnostics(actor).CurrentRevision.ShouldBe(CachedRev);
|
||||
|
||||
actor.Tell(new ZB.MOM.WW.OtOpcUa.Commons.Messages.Deploy.DispatchDeployment(
|
||||
DeploymentId.NewId(), CachedRev, CorrelationId.NewId()));
|
||||
|
||||
var ack = coordinator.ExpectMsg<ZB.MOM.WW.OtOpcUa.Commons.Messages.Deploy.ApplyAck>(
|
||||
PresenceBudget, cancellationToken: TestContext.Current.CancellationToken);
|
||||
ack.Outcome.ShouldNotBe(ZB.MOM.WW.OtOpcUa.Commons.Messages.Deploy.ApplyAckOutcome.Applied);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void BootFromCache_HandsTheCachedArtifactToTheAddressSpaceRebuild()
|
||||
{
|
||||
|
||||
Reference in New Issue
Block a user