Joseph Doherty
0793b0cba8
The default decider restarts a faulting child immediately and without limit, and the hook makes every restart do ShutdownAsync+InitializeAsync+SubscribeAsync, so a driver faulting on every value became a reconnect storm. Per-driver last-reprovision stamp + DefaultReprovisionBackoff (10s, matched to DefaultReconnectInterval); a held request is re-delivered by a single per-driver timer carrying the original incarnation ref, so the F2 identity check still applies. Seen RED with the backoff disabled: InitializeCount == 3 (second applied immediately) where the test requires 2-then-3. Also: the _children.Keys union-arm test (seeder gained taglessDrivers) and the _desiredByDriver xmldoc note that it lingers on last-known-good after a failed apply.
631 lines
34 KiB
C#
631 lines
34 KiB
C#
using Akka.Actor;
|
|
using Microsoft.EntityFrameworkCore;
|
|
using Shouldly;
|
|
using Xunit;
|
|
using ZB.MOM.WW.OtOpcUa.Commons.Messages.Admin;
|
|
using ZB.MOM.WW.OtOpcUa.Commons.Types;
|
|
using ZB.MOM.WW.OtOpcUa.Configuration;
|
|
using ZB.MOM.WW.OtOpcUa.Runtime.Drivers;
|
|
using ZB.MOM.WW.OtOpcUa.Runtime.Tests.Harness;
|
|
|
|
namespace ZB.MOM.WW.OtOpcUa.Runtime.Tests.Drivers;
|
|
|
|
/// <summary>
|
|
/// The repo's first driver-child <b>restart</b> suite (archreview 03/S16, 03/U6): what a child obtained
|
|
/// by each of the three incarnation routes is actually handed.
|
|
/// </summary>
|
|
/// <remarks>
|
|
/// <para>
|
|
/// <b>The falsification bar, and why this file exists at all.</b> An actor-restart test must assert a
|
|
/// FUNCTION the new incarnation performs — it subscribed, it re-initialised — never that the
|
|
/// incarnation EXISTS. Existence is exactly what
|
|
/// <c>DriverHostActorReconcileTests.RestartDriver_respawns_the_child_without_an_actor_name_collision</c>
|
|
/// asserts (no name collision, factory created a second driver, one child remains), and that test
|
|
/// passes green against the 03/S16 defect: a respawned child that connects and then reads nothing
|
|
/// forever. It is kept, because its own subject — the <c>InvalidActorNameException</c> — is real; it
|
|
/// is simply the weaker half of a pair, and this file is the other half.
|
|
/// </para>
|
|
/// <para>
|
|
/// <b>The three incarnation routes, and what re-provisions each.</b> (1) <i>Deploy reconcile</i> —
|
|
/// <c>ApplyDriverSpecs</c> → <c>SpawnChild</c>, then the apply's bulk desired-subscription push.
|
|
/// (2) <i>Operator Restart</i> — <c>HandleRestartDriver</c> stops the child and calls
|
|
/// <c>SpawnChild</c> with the cached spec; before 03/S16 that was the whole story, and the new child
|
|
/// started with an EMPTY desired set that nothing re-sent until the next deployment (the #488
|
|
/// periodic reconcile early-returns on an empty set, so it is structurally inert here).
|
|
/// (3) <i>Supervision restart</i> — Akka's default decider recreates the actor object around the
|
|
/// SAME driver instance without going through <c>SpawnChild</c> at all, so nothing re-sent
|
|
/// <c>InitializeRequested</c> either and the child sat in <c>Connecting</c> forever around a driver
|
|
/// its own <c>PostStop</c> had already shut down. A fourth route, <i>Force Reconnect</i>, re-uses the
|
|
/// live actor instance and needs nothing — a verified negative, pinned here so a future edit to the
|
|
/// restart path cannot break the path that was already correct.
|
|
/// </para>
|
|
/// <para>
|
|
/// <b>The supervision crash seam, and why the obvious ones do not work.</b> The seam used is the
|
|
/// driver's own <c>OnDataChange</c> carrying a NULL <c>Snapshot</c>
|
|
/// (<see cref="RecordingSubscribableDriver.FireMalformedDataChange"/>): <c>DataChangeEventArgs</c> is
|
|
/// a positional record with no null validation, and <c>DriverInstanceActor.OnDataChangeForward</c>
|
|
/// dereferences <c>msg.Snapshot</c> unguarded on the ACTOR thread, so it NREs inside the child and
|
|
/// the default decider Restarts it. Rejected alternatives, each for a specific reason: a throwing
|
|
/// <c>IDriverHealthPublisher</c> (<c>PublishHealthSnapshot</c> wraps the publish in a
|
|
/// <c>catch (Exception)</c>, so it never escapes); a throwing <c>SubscribeAsync</c> (also swallowed —
|
|
/// the stub already models it via <c>SubscribeFailuresBeforeSuccess</c>); <c>Kill.Instance</c> (the
|
|
/// default decider <b>Stops</b> on <c>ActorKilledException</c>, a different directive and therefore a
|
|
/// different code path); and an <c>internal</c> "throw this" test message (a production surface added
|
|
/// purely for a test). The fault is arbitrary by design — what is under test is the RECOVERY. If
|
|
/// <c>DataChangeEventArgs</c> ever gains null validation, the fallback is an explicit
|
|
/// <c>OneForOneStrategy(Directive.Restart)</c> on the host plus an internal test-only throw message.
|
|
/// </para>
|
|
/// </remarks>
|
|
public sealed class DriverHostActorRestartTests : RuntimeActorTestBase
|
|
{
|
|
private static readonly RevisionHash RevA = RevisionHash.Parse(new string('a', 64));
|
|
private static readonly RevisionHash RevB = RevisionHash.Parse(new string('b', 64));
|
|
private static readonly NodeId TestNode = NodeId.Parse("node-1:4053");
|
|
|
|
private const string DriverId = "DI-1";
|
|
private const string SpeedPath = "Plant/Modbus/dev1/speed";
|
|
private const string TempPath = "Plant/Modbus/dev1/temp";
|
|
private const string AlarmXPath = "Plant/Modbus/dev1/alarmX";
|
|
private const string AlarmYPath = "Plant/Modbus/dev1/alarmY";
|
|
|
|
/// <summary>TagConfig marking a raw tag as alarm-bearing (parsed by <c>TagConfigIntent</c>).</summary>
|
|
private const string AlarmConfig = """{"alarm":{"alarmType":"AlarmCondition","severity":500}}""";
|
|
|
|
/// <summary>Seeds + applies a v3 deployment whose single Modbus driver is ENABLED, so a real child is
|
|
/// spawned around a <see cref="RecordingSubscribableDriver"/>, and waits until that child has connected
|
|
/// and established its first subscription.</summary>
|
|
private (IActorRef Host, RecordingSubscribableDriverFactory Factory) DeployOneDriver(
|
|
IDbContextFactory<OtOpcUaConfigDbContext> db,
|
|
(string Driver, string DriverName, string Device, string Tag, string DataType, bool Writable)[] rawTags,
|
|
IReadOnlyDictionary<string, string>? tagConfigByTag = null)
|
|
{
|
|
var factory = new RecordingSubscribableDriverFactory("Modbus");
|
|
var deploymentId = V3ArtifactSeeder.SeedV3Deployment(
|
|
db, RevA, rawTags, Array.Empty<(string, string, string, string, string?)>(),
|
|
enabled: true, tagConfigByTag: tagConfigByTag);
|
|
var (host, _, _) = V3ArtifactSeeder.SpawnHostAndApply(this, db, TestNode, deploymentId, RevA, factory);
|
|
return (host, factory);
|
|
}
|
|
|
|
/// <summary>
|
|
/// <b>The 03/S16 falsification.</b> An operator's Restart is supposed to recover a sick driver; before
|
|
/// the fix it silently killed it — the respawned child came up with <c>_desiredRefs</c> empty, and the
|
|
/// only <c>SetDesiredSubscriptions</c> sender in the repo was the apply's bulk push, so the driver
|
|
/// subscribed to nothing until the next deployment dispatch.
|
|
/// <para>
|
|
/// Deliberately NOT the pass criterion: that a second driver was created, or that one child
|
|
/// remains. Both are true against the defect.
|
|
/// </para>
|
|
/// </summary>
|
|
[Fact]
|
|
public void Operator_restart_re_subscribes_the_respawned_child()
|
|
{
|
|
var db = NewInMemoryDbFactory();
|
|
var (host, factory) = DeployOneDriver(db, new[] { (DriverId, "Modbus", "dev1", "speed", "Float", false) });
|
|
|
|
AwaitAssert(
|
|
() =>
|
|
{
|
|
factory.Created.Count.ShouldBe(1);
|
|
factory.Created[0].SubscribeCount.ShouldBe(1);
|
|
factory.Created[0].LastSubscribedRefs.ShouldBe(new[] { SpeedPath });
|
|
},
|
|
duration: PresenceBudget);
|
|
|
|
host.Tell(new RestartDriver("c1", DriverId, "tester", Guid.NewGuid()));
|
|
|
|
AwaitAssert(
|
|
() =>
|
|
{
|
|
factory.Created.Count.ShouldBe(2, "the restart did not respawn the child around a new driver");
|
|
factory.Created[1].SubscribeCount.ShouldBeGreaterThanOrEqualTo(
|
|
1,
|
|
"the RESPAWNED child never subscribed — it is Connected and reading nothing, which is the " +
|
|
"03/S16 defect (an operator pressed Restart to recover a driver and silently killed it)");
|
|
factory.Created[1].LastSubscribedRefs.ShouldBe(new[] { SpeedPath });
|
|
},
|
|
duration: PresenceBudget);
|
|
}
|
|
|
|
/// <summary>
|
|
/// The value refs and the alarm refs ride in ONE <c>SetDesiredSubscriptions</c> but are applied by two
|
|
/// different child code paths (the value set self-Tells <c>Subscribe</c>; the alarm set runs
|
|
/// <c>SubscribeDesiredAlarms</c> against the <c>IAlarmSource</c> feed, which stays gated shut until it
|
|
/// is called). Sending the message is therefore not evidence that both halves landed, so the alarm
|
|
/// half is pinned separately.
|
|
/// </summary>
|
|
[Fact]
|
|
public void Operator_restart_re_subscribes_alarm_refs_too()
|
|
{
|
|
var db = NewInMemoryDbFactory();
|
|
var (host, factory) = DeployOneDriver(
|
|
db,
|
|
new[]
|
|
{
|
|
(DriverId, "Modbus", "dev1", "speed", "Float", false),
|
|
(DriverId, "Modbus", "dev1", "temp", "Float", false),
|
|
},
|
|
// An alarm-bearing raw tag is EXCLUDED from the value set and carried in the alarm set instead.
|
|
new Dictionary<string, string>
|
|
{
|
|
["temp"] = """{"alarm":{"alarmType":"AlarmCondition","severity":500}}""",
|
|
});
|
|
|
|
AwaitAssert(
|
|
() =>
|
|
{
|
|
factory.Created.Count.ShouldBe(1);
|
|
factory.Created[0].SubscribeAlarmsCount.ShouldBeGreaterThanOrEqualTo(1);
|
|
factory.Created[0].LastSubscribedAlarmRefs.ShouldBe(new[] { TempPath });
|
|
},
|
|
duration: PresenceBudget);
|
|
|
|
host.Tell(new RestartDriver("c1", DriverId, "tester", Guid.NewGuid()));
|
|
|
|
AwaitAssert(
|
|
() =>
|
|
{
|
|
factory.Created.Count.ShouldBe(2);
|
|
factory.Created[1].SubscribeAlarmsCount.ShouldBeGreaterThanOrEqualTo(
|
|
1,
|
|
"the respawned child never re-subscribed its alarm refs — the IAlarmSource feed stays " +
|
|
"gated shut, so no native alarm transition can ever reach the host again");
|
|
factory.Created[1].LastSubscribedAlarmRefs.ShouldBe(new[] { TempPath });
|
|
// The value half must survive the alarm half being present.
|
|
factory.Created[1].LastSubscribedRefs.ShouldBe(new[] { SpeedPath });
|
|
},
|
|
duration: PresenceBudget);
|
|
}
|
|
|
|
/// <summary>
|
|
/// The ordering guard for the spawn-time send. <c>ApplyDriverSpecs</c> (which spawns) runs BEFORE
|
|
/// <c>ApplyDesiredSubscriptions</c> (which rebuilds the authority), so a child spawned by a fresh
|
|
/// deploy is momentarily handed the PREVIOUS apply's set. That transient must never be acted on: it
|
|
/// is superseded by the bulk push on the same sender→receiver pair (Akka guarantees per-pair
|
|
/// ordering) while the child is still <c>Connecting</c>, where <c>SetDesiredSubscriptions</c> only
|
|
/// STORES; the subscribe happens on the Connected entry, from whatever was stored last.
|
|
/// <para>
|
|
/// Revision B swaps the driver's tag, so A's and B's sets are distinguishable. Note the raw tags
|
|
/// are folded into the merged <c>DriverConfig</c>, so a tag-set change DOES change the driver
|
|
/// spec and therefore respawns the child (#516) — which is precisely the arrangement under test:
|
|
/// a brand-new child, spawned while the authority still holds A, must end up subscribed to B.
|
|
/// </para>
|
|
/// <para>
|
|
/// <b>What is asserted is CONVERGENCE, not inertness</b>, and the difference was measured rather
|
|
/// than assumed. The child runs concurrently with the rest of the apply handler, so a driver
|
|
/// whose <c>InitializeAsync</c> completes instantly (this stub does) can reach Connected and
|
|
/// subscribe the transient stale set BEFORE the bulk push arrives — observed here as
|
|
/// <c>SubscribeCount == 2</c>. Akka's per-pair ordering still guarantees the final state is B's,
|
|
/// and <c>HandleSubscribeAsync</c> drops the prior subscription before establishing the new one,
|
|
/// so the cost is one extra round-trip on a respawning deploy. Pinning the count would pin a
|
|
/// race; the final refs are the invariant.
|
|
/// </para>
|
|
/// </summary>
|
|
[Fact]
|
|
public void A_child_spawned_by_a_fresh_deploy_ends_with_the_NEW_refs()
|
|
{
|
|
var db = NewInMemoryDbFactory();
|
|
var (host, factory) = DeployOneDriver(db, new[] { (DriverId, "Modbus", "dev1", "speed", "Float", false) });
|
|
|
|
AwaitAssert(
|
|
() => factory.Created[0].LastSubscribedRefs.ShouldBe(new[] { SpeedPath }),
|
|
duration: PresenceBudget);
|
|
|
|
var revB = V3ArtifactSeeder.SeedV3Deployment(
|
|
db, RevB,
|
|
new[] { (DriverId, "Modbus", "dev1", "temp", "Float", false) },
|
|
Array.Empty<(string, string, string, string, string?)>(),
|
|
enabled: true);
|
|
host.Tell(new Commons.Messages.Deploy.DispatchDeployment(revB, RevB, CorrelationId.NewId()));
|
|
|
|
AwaitAssert(
|
|
() =>
|
|
{
|
|
factory.Created.Count.ShouldBe(2, "the tag-set change re-spawns the child (#516 config respawn)");
|
|
factory.Created[1].LastSubscribedRefs.ShouldBe(
|
|
new[] { TempPath },
|
|
"the newly-spawned child is on the PREVIOUS revision's refs — the spawn-time set was not superseded");
|
|
},
|
|
duration: PresenceBudget);
|
|
|
|
// A Restart on top of that must hand the new child B's set, not A's — i.e. the authority really is
|
|
// repopulated WHOLE on every apply rather than accumulated.
|
|
host.Tell(new RestartDriver("c1", DriverId, "tester", Guid.NewGuid()));
|
|
AwaitAssert(
|
|
() =>
|
|
{
|
|
factory.Created.Count.ShouldBe(3);
|
|
factory.Created[2].LastSubscribedRefs.ShouldBe(new[] { TempPath });
|
|
},
|
|
duration: PresenceBudget);
|
|
}
|
|
|
|
/// <summary>
|
|
/// The empty-set semantics are load-bearing and must not degrade into "skip the send": an EMPTY value
|
|
/// set is what tells the child to <c>Unsubscribe</c> and drop a stale handle, so it must be SENT.
|
|
/// <para>
|
|
/// The subject is a driver whose only raw tag is alarm-bearing: alarm tags are excluded from the
|
|
/// value set and carried in the alarm one, so its value half is genuinely empty while its alarm
|
|
/// half is not — and both halves ride in ONE <c>SetDesiredSubscriptions</c>. That is what makes
|
|
/// the absence assertion non-vacuous: <c>SubscribeAlarmsCount >= 1</c> proves the message
|
|
/// ARRIVED, and <c>SubscribeCount == 0</c> then proves the empty value half was honoured rather
|
|
/// than the whole send being skipped. Asserting only "it did not subscribe" would pass equally
|
|
/// against a build that sent the respawned child nothing at all — the 03/S16 defect itself.
|
|
/// </para>
|
|
/// </summary>
|
|
[Fact]
|
|
public void Restart_of_a_driver_with_an_empty_value_set_still_sends_the_set()
|
|
{
|
|
var db = NewInMemoryDbFactory();
|
|
var (host, factory) = DeployOneDriver(
|
|
db,
|
|
new[] { (DriverId, "Modbus", "dev1", "temp", "Float", false) },
|
|
new Dictionary<string, string>
|
|
{
|
|
["temp"] = """{"alarm":{"alarmType":"AlarmCondition","severity":500}}""",
|
|
});
|
|
|
|
AwaitAssert(
|
|
() =>
|
|
{
|
|
factory.Created.Count.ShouldBe(1);
|
|
factory.Created[0].SubscribeAlarmsCount.ShouldBeGreaterThanOrEqualTo(1);
|
|
},
|
|
duration: PresenceBudget);
|
|
|
|
host.Tell(new RestartDriver("c1", DriverId, "tester", Guid.NewGuid()));
|
|
|
|
AwaitAssert(
|
|
() =>
|
|
{
|
|
factory.Created.Count.ShouldBe(2);
|
|
var respawned = factory.Created[1];
|
|
respawned.InitializeCount.ShouldBe(1, "the respawned child was never initialised");
|
|
// Positive control: the SetDesiredSubscriptions genuinely reached the respawned child.
|
|
respawned.SubscribeAlarmsCount.ShouldBeGreaterThanOrEqualTo(
|
|
1, "the respawned child received no desired set at all — the absence below would be vacuous");
|
|
respawned.LastSubscribedAlarmRefs.ShouldBe(new[] { TempPath });
|
|
respawned.SubscribeCount.ShouldBe(
|
|
0, "an EMPTY value set must route to Unsubscribe / no-op, never to a spurious subscribe");
|
|
},
|
|
duration: PresenceBudget);
|
|
}
|
|
|
|
/// <summary>
|
|
/// <b>Review finding F1 — the alarm subscription must FOLLOW a changed desired set, not latch.</b>
|
|
/// The value side converges by construction (<c>HandleSubscribeAsync</c> drops the prior subscription
|
|
/// before establishing the new one, and <c>StoreDesiredSubscriptions</c> is a full replace), but the
|
|
/// alarm side is guarded by <c>_alarmSubscriptionHandle is not null</c> in BOTH
|
|
/// <c>SubscribeDesiredAlarms</c> and <c>HandleSubscribeAlarmsAsync</c> — so once a handle exists it
|
|
/// was never re-established for the life of the connection.
|
|
/// <para>
|
|
/// That became reachable when every spawn started carrying a desired set: a tag-set edit forces a
|
|
/// respawn (raw tags fold into the merged <c>DriverConfig</c>, #516), and a fast-initialising
|
|
/// driver connects and subscribes the transient spawn-time set — the PREVIOUS revision's alarm
|
|
/// refs — before the bulk push lands. The bulk push then early-returns, latching the old set for
|
|
/// the connection's life. Drivers that FILTER on <c>sourceNodeIds</c> (OPC UA Client, AB CIP,
|
|
/// FOCAS) would silently never deliver the newly-added alarm.
|
|
/// </para>
|
|
/// </summary>
|
|
[Fact]
|
|
public void A_deploy_that_widens_the_alarm_set_re_subscribes_the_new_refs()
|
|
{
|
|
var db = NewInMemoryDbFactory();
|
|
var (host, factory) = DeployOneDriver(
|
|
db,
|
|
new[] { (DriverId, "Modbus", "dev1", "alarmX", "Float", false) },
|
|
new Dictionary<string, string> { ["alarmX"] = AlarmConfig });
|
|
|
|
AwaitAssert(
|
|
() =>
|
|
{
|
|
factory.Created.Count.ShouldBe(1);
|
|
factory.Created[0].LastSubscribedAlarmRefs.ShouldBe(new[] { AlarmXPath }, ignoreOrder: true);
|
|
},
|
|
duration: PresenceBudget);
|
|
|
|
// Revision B WIDENS the alarm set: X stays, Y is added.
|
|
var revB = V3ArtifactSeeder.SeedV3Deployment(
|
|
db, RevB,
|
|
new[]
|
|
{
|
|
(DriverId, "Modbus", "dev1", "alarmX", "Float", false),
|
|
(DriverId, "Modbus", "dev1", "alarmY", "Float", false),
|
|
},
|
|
Array.Empty<(string, string, string, string, string?)>(),
|
|
enabled: true,
|
|
tagConfigByTag: new Dictionary<string, string>
|
|
{
|
|
["alarmX"] = AlarmConfig,
|
|
["alarmY"] = AlarmConfig,
|
|
});
|
|
host.Tell(new Commons.Messages.Deploy.DispatchDeployment(revB, RevB, CorrelationId.NewId()));
|
|
|
|
AwaitAssert(
|
|
() =>
|
|
{
|
|
var current = factory.Created[^1];
|
|
current.LastSubscribedAlarmRefs.ShouldBe(
|
|
new[] { AlarmXPath, AlarmYPath },
|
|
ignoreOrder: true,
|
|
"the driver is still subscribed to the PREVIOUS revision's alarm refs — the alarm " +
|
|
"subscription latched on the transient spawn-time set and never followed the new one");
|
|
},
|
|
duration: PresenceBudget);
|
|
}
|
|
|
|
/// <summary>
|
|
/// <b>The repo's first genuine Akka SUPERVISION-restart test (03/U6).</b> Not a stop, not a stub: the
|
|
/// driver raises a malformed <c>OnDataChange</c>, the actor NREs on its own thread, and Akka's default
|
|
/// decider Restarts the child in place.
|
|
/// <para>
|
|
/// A restart is strictly worse than the operator-Restart case 03/S16 names. It recreates the actor
|
|
/// object around the SAME driver instance without going through <c>SpawnChild</c>, so nothing
|
|
/// re-sends <c>InitializeRequested</c> — while <c>PreRestart</c> has already run <c>PostStop</c>,
|
|
/// which called <c>ShutdownAsync</c> on that very driver. The child therefore sat in
|
|
/// <c>Connecting</c> forever around a shut-down driver, with its desired set gone too, and
|
|
/// silently: Akka logs the fault, nothing logs the non-recovery.
|
|
/// </para>
|
|
/// <para>
|
|
/// The assertions are FUNCTIONS of the new incarnation — it re-initialised, it re-subscribed the
|
|
/// deployed refs — never that it exists. <c>Created.Count == 1</c> is what distinguishes this from
|
|
/// the operator path: a restart must NOT produce a second driver object.
|
|
/// </para>
|
|
/// </summary>
|
|
[Fact]
|
|
public void Supervision_restart_reinitialises_and_re_subscribes_the_child()
|
|
{
|
|
var db = NewInMemoryDbFactory();
|
|
var (_, factory) = DeployOneDriver(db, new[] { (DriverId, "Modbus", "dev1", "speed", "Float", false) });
|
|
|
|
AwaitAssert(
|
|
() =>
|
|
{
|
|
factory.Created.Count.ShouldBe(1);
|
|
factory.Created[0].SubscribeCount.ShouldBe(1);
|
|
},
|
|
duration: PresenceBudget);
|
|
|
|
var driver = factory.Created[0];
|
|
driver.InitializeCount.ShouldBe(1);
|
|
|
|
// The crash. Subscribing is what attaches the actor's OnDataChange handler, so this only reaches the
|
|
// actor thread because the arrangement above got as far as a live subscription.
|
|
driver.FireMalformedDataChange(SpeedPath);
|
|
|
|
// FIRST: prove the crash seam actually tripped supervision, before asserting anything about the
|
|
// recovery. PreRestart runs PostStop, which calls ShutdownAsync on the driver — so a raised
|
|
// ShutdownCount is the evidence that a restart happened at all. Without this the assertions below
|
|
// would be ambiguous: "the child was not re-provisioned" and "the child never crashed" look
|
|
// identical from the outside, and a seam that silently stopped working would read as a passing
|
|
// regression test forever.
|
|
AwaitAssert(
|
|
() => driver.ShutdownCount.ShouldBeGreaterThanOrEqualTo(
|
|
1, "the crash seam did not trip supervision — nothing below is testing a restart"),
|
|
duration: PresenceBudget);
|
|
|
|
AwaitAssert(
|
|
() =>
|
|
{
|
|
factory.Created.Count.ShouldBe(
|
|
1, "a supervision restart must re-use the same driver object, not respawn the child");
|
|
driver.InitializeCount.ShouldBe(
|
|
2,
|
|
"the restarted child was never re-initialised — it is sitting in Connecting around a " +
|
|
"driver its own PostStop already shut down (03/S6, the worse half of 03/S16)");
|
|
driver.SubscribeCount.ShouldBeGreaterThanOrEqualTo(
|
|
2, "the restarted child never re-subscribed — its desired set died with the old instance");
|
|
driver.LastSubscribedRefs.ShouldBe(new[] { SpeedPath });
|
|
},
|
|
duration: PresenceBudget);
|
|
}
|
|
|
|
/// <summary>
|
|
/// <b>Review finding F2 — a reprovision must be routed by INCARNATION, not just by driver id.</b>
|
|
/// <c>ReprovisionRequested</c> is delivered asynchronously, so one can still be in flight from an old
|
|
/// incarnation when <c>HandleRestartDriver</c> has already stopped that child and swapped in a new
|
|
/// one under the same <c>DriverInstanceId</c>. Routing on the id alone delivers it to the NEW child,
|
|
/// which is in <c>Connecting</c> around a healthy driver: it triggers a second, concurrent
|
|
/// <c>InitializeAsync</c> that races the very dispose-before-overwrite the #527 teardown exists to
|
|
/// serialise (<c>ModbusDriver._transport</c>, <c>MqttDriver._connection</c>).
|
|
/// <para>
|
|
/// A stale sender is simulated directly — a probe standing in for the dead incarnation — because
|
|
/// provoking the real race would be timing-dependent. The assertion is an ABSENCE, so it carries
|
|
/// a positive control: the current child is proven alive and provisioned first, and its
|
|
/// <c>InitializeCount</c> must then not move.
|
|
/// </para>
|
|
/// </summary>
|
|
[Fact]
|
|
public void A_reprovision_from_a_stale_incarnation_is_ignored()
|
|
{
|
|
var db = NewInMemoryDbFactory();
|
|
// The child must be in CONNECTING for this defect to bite: InitializeRequested is unhandled in the
|
|
// Connected behaviour, so an already-connected child would swallow the stale reprovision for the
|
|
// wrong reason and the test would pass vacuously. The gated init holds it in Connecting.
|
|
var factory = new RecordingSubscribableDriverFactory("Modbus", gateInitialize: true);
|
|
var deploymentId = V3ArtifactSeeder.SeedV3Deployment(
|
|
db, RevA,
|
|
new[] { (DriverId, "Modbus", "dev1", "speed", "Float", false) },
|
|
Array.Empty<(string, string, string, string, string?)>(),
|
|
enabled: true);
|
|
var (host, _, _) = V3ArtifactSeeder.SpawnHostAndApply(this, db, TestNode, deploymentId, RevA, factory);
|
|
|
|
// Positive control: the current child is alive and has begun its one and only initialise.
|
|
AwaitAssert(
|
|
() =>
|
|
{
|
|
factory.Created.Count.ShouldBe(1);
|
|
factory.Created[0].InitializeCount.ShouldBe(1);
|
|
},
|
|
duration: PresenceBudget);
|
|
var driver = factory.Created[0];
|
|
|
|
// A ReprovisionRequested that did NOT come from the child the host currently holds.
|
|
var staleIncarnation = CreateTestProbe();
|
|
host.Tell(
|
|
new DriverInstanceActor.ReprovisionRequested(DriverId, "NullReferenceException"),
|
|
staleIncarnation.Ref);
|
|
|
|
// Round-trip a message the host must answer, so the reprovision above is provably processed before
|
|
// the absence is asserted — otherwise "nothing happened yet" would pass for "it was ignored".
|
|
var coordinator = CreateTestProbe();
|
|
host.Tell(new Commons.Messages.Fleet.GetDiagnostics(CorrelationId.NewId()), coordinator.Ref);
|
|
coordinator.ExpectMsg<Commons.Interfaces.NodeDiagnosticsSnapshot>(PresenceBudget);
|
|
|
|
driver.InitializeCount.ShouldBe(
|
|
1,
|
|
"a reprovision from a dead incarnation reached the CURRENT child and started a second, " +
|
|
"concurrent InitializeAsync on a driver that was already connected");
|
|
}
|
|
|
|
/// <summary>
|
|
/// <b>Review finding F3 - a crash-looping driver must not become a reconnect storm.</b> There is no
|
|
/// <c>SupervisorStrategy</c> anywhere under <c>src/Server</c>, so the default decider restarts a
|
|
/// faulting child <b>immediately and without limit</b>. That was survivable while a restart did
|
|
/// nothing; the reprovision hook now makes every one of them do
|
|
/// <c>ShutdownAsync</c> + <c>InitializeAsync</c> + <c>SubscribeAsync</c>, so a driver that faults on
|
|
/// every value would hammer its device as fast as it can crash.
|
|
/// <para>
|
|
/// The reprovision is therefore rate-limited per driver. Crucially it is <b>deferred, not
|
|
/// dropped</b> - the last request must still be honoured, or a child whose final crash fell
|
|
/// inside the window would be left un-provisioned forever, which is the very failure this whole
|
|
/// plan exists to close. Two genuine back-to-back supervision restarts: the first reprovision
|
|
/// applies immediately, the second is held and then lands.
|
|
/// </para>
|
|
/// </summary>
|
|
[Fact]
|
|
public void A_second_reprovision_inside_the_backoff_window_is_deferred_not_dropped()
|
|
{
|
|
var db = NewInMemoryDbFactory();
|
|
// Generous relative to the in-memory step time below, so "inside the window" is a decision rather
|
|
// than a race, while still far under the presence budget the deferred landing waits on.
|
|
var backoff = TimeSpan.FromSeconds(3);
|
|
var factory = new RecordingSubscribableDriverFactory("Modbus");
|
|
var deploymentId = V3ArtifactSeeder.SeedV3Deployment(
|
|
db, RevA,
|
|
new[] { (DriverId, "Modbus", "dev1", "speed", "Float", false) },
|
|
Array.Empty<(string, string, string, string, string?)>(),
|
|
enabled: true);
|
|
V3ArtifactSeeder.SpawnHostAndApply(
|
|
this, db, TestNode, deploymentId, RevA, factory, reprovisionBackoff: backoff);
|
|
|
|
AwaitAssert(
|
|
() =>
|
|
{
|
|
factory.Created.Count.ShouldBe(1);
|
|
factory.Created[0].SubscribeCount.ShouldBe(1);
|
|
},
|
|
duration: PresenceBudget);
|
|
var driver = factory.Created[0];
|
|
|
|
// Crash 1: the first reprovision has no prior stamp, so it applies immediately.
|
|
driver.FireMalformedDataChange(SpeedPath);
|
|
AwaitAssert(
|
|
() =>
|
|
{
|
|
driver.InitializeCount.ShouldBe(2);
|
|
driver.SubscribeCount.ShouldBe(2, "the re-subscribe must land before the second crash is armed");
|
|
},
|
|
duration: PresenceBudget);
|
|
|
|
// Crash 2, immediately after - well inside the window.
|
|
driver.FireMalformedDataChange(SpeedPath);
|
|
AwaitAssert(
|
|
() => driver.ShutdownCount.ShouldBeGreaterThanOrEqualTo(
|
|
2, "the second crash seam did not trip supervision - nothing below is testing a backoff"),
|
|
duration: PresenceBudget);
|
|
|
|
driver.InitializeCount.ShouldBe(
|
|
2, "the second reprovision was applied immediately - the backoff is not rate-limiting anything");
|
|
|
|
// ...but it must still land once the window elapses.
|
|
AwaitAssert(
|
|
() => driver.InitializeCount.ShouldBe(
|
|
3, "the deferred reprovision was DROPPED, not deferred"),
|
|
duration: PresenceBudget);
|
|
}
|
|
|
|
/// <summary>
|
|
/// The <c>_children.Keys</c> arm of the desired-set authority's union. A driver with NO authored tags
|
|
/// is absent from both the value- and alarm-ref maps, so keying the authority off those maps alone
|
|
/// would leave it with no entry at all — and "absent" must not be allowed to differ from "empty",
|
|
/// because an empty set is what tells a child to <c>Unsubscribe</c> and drop a stale handle.
|
|
/// <para>
|
|
/// Asserted the only way it is observable: the tagless child is initialised (so it is provably
|
|
/// alive and was provisioned) and never subscribes, while its tag-owning sibling in the same
|
|
/// deployment does — the sibling is the positive control proving the apply's push pass actually
|
|
/// ran, so "it did not subscribe" cannot pass for "nothing happened".
|
|
/// </para>
|
|
/// </summary>
|
|
[Fact]
|
|
public void A_running_driver_with_no_authored_tags_is_handed_an_empty_set()
|
|
{
|
|
var db = NewInMemoryDbFactory();
|
|
var factory = new RecordingSubscribableDriverFactory("Modbus");
|
|
var deploymentId = V3ArtifactSeeder.SeedV3Deployment(
|
|
db, RevA,
|
|
new[] { (DriverId, "Modbus", "dev1", "speed", "Float", false) },
|
|
Array.Empty<(string, string, string, string, string?)>(),
|
|
enabled: true,
|
|
taglessDrivers: new[] { "DI-TAGLESS" });
|
|
V3ArtifactSeeder.SpawnHostAndApply(this, db, TestNode, deploymentId, RevA, factory);
|
|
|
|
AwaitAssert(
|
|
() =>
|
|
{
|
|
factory.Created.Count.ShouldBe(2);
|
|
// Positive control: the sibling that DOES have a tag subscribed, so the push pass ran.
|
|
factory.Created.Single(d => d.DriverInstanceId == DriverId)
|
|
.SubscribeCount.ShouldBe(1);
|
|
factory.Created.Single(d => d.DriverInstanceId == "DI-TAGLESS")
|
|
.InitializeCount.ShouldBe(1);
|
|
},
|
|
duration: PresenceBudget);
|
|
|
|
factory.Created.Single(d => d.DriverInstanceId == "DI-TAGLESS")
|
|
.SubscribeCount.ShouldBe(
|
|
0, "a driver with no authored tags must be handed an EMPTY set, never a spurious subscribe");
|
|
}
|
|
|
|
/// <summary>
|
|
/// The verified negative, pinned so this plan's edits to the restart paths cannot break the one that
|
|
/// was already correct. Force Reconnect (<c>ReconnectDriver</c>) Tells <c>ForceReconnect</c> to the
|
|
/// SAME child actor: the instance survives, so <c>_desiredRefs</c>/<c>_desiredAlarmRefs</c> are intact
|
|
/// and the reconnect's Connected entry re-applies them by itself. Nothing needs re-provisioning, and
|
|
/// no new driver object is created.
|
|
/// </summary>
|
|
/// <remarks>
|
|
/// The budget here is deliberately wider than <see cref="RuntimeActorTestBase.PresenceBudget"/>:
|
|
/// <c>SpawnChild</c> gives the child the production <c>DefaultReconnectInterval</c> (10 s) and the
|
|
/// retry is a periodic timer, so the reconnect cannot land sooner than that. A budget is spent only
|
|
/// when the assertion fails, so widening it costs nothing on the happy path.
|
|
/// </remarks>
|
|
[Fact]
|
|
public void Force_reconnect_keeps_the_desired_set_and_re_subscribes()
|
|
{
|
|
var db = NewInMemoryDbFactory();
|
|
var (host, factory) = DeployOneDriver(db, new[] { (DriverId, "Modbus", "dev1", "speed", "Float", false) });
|
|
|
|
AwaitAssert(
|
|
() =>
|
|
{
|
|
factory.Created.Count.ShouldBe(1);
|
|
factory.Created[0].SubscribeCount.ShouldBe(1);
|
|
},
|
|
duration: PresenceBudget);
|
|
|
|
var driver = factory.Created[0];
|
|
host.Tell(new ReconnectDriver("c1", DriverId, "tester", Guid.NewGuid()));
|
|
|
|
AwaitAssert(
|
|
() =>
|
|
{
|
|
factory.Created.Count.ShouldBe(1, "Force Reconnect must NOT respawn the child");
|
|
driver.SubscribeCount.ShouldBeGreaterThanOrEqualTo(
|
|
2, "the reconnected driver never re-subscribed its (surviving) desired set");
|
|
driver.LastSubscribedRefs.ShouldBe(new[] { SpeedPath });
|
|
},
|
|
duration: TimeSpan.FromSeconds(30));
|
|
}
|
|
}
|