feat(dcl+siteruntime): reconfigure a LIVE data connection in place — register row 97
`DataConnectionManagerActor.HandleCreateConnection` is create-only (warns and
returns for a name it already holds), so `EnsureDclConnection`'s documented
"a changed config re-issues the create so the DCL adopts it" was never true on
ANY node: a corrected endpoint/credential/backup/failover-count took effect only
when the connection was next built from scratch (node restart, singleton
failover, remove-then-re-add), while the DM's hash cache recorded it as applied.
Rows 89/90 made that honest with a Warning; this closes it.
New `ReconfigureConnectionCommand` (Commons, same shape as the create):
- `DeploymentManagerActor.EnsureDclConnection` now has three outcomes off its one
per-name hash slot — unchanged -> nothing at all, new -> create, CHANGED ->
reconfigure — and logs "Reconfigured DCL connection {Connection}
(protocol={Protocol}) in place" at Information in place of the old Warning. The
winner-resolution / arbitration / tombstone logic is untouched; only what
happens after a change is detected changed.
- `DataConnectionManagerActor` FORWARDS the command to the owning connection
actor, falling back to a create for a name it does not hold, so the command is
safe whatever the manager's state.
- `DataConnectionActor` handles it in ALL THREE states and never stashes it — a
connection pointed at a wrong or dead endpoint lives in Connecting/Reconnecting,
which is exactly when a correction arrives; stashed it would wait on the broken
configuration forever. Handling replaces protocol/primary/backup/failover-count,
re-arms endpoint selection to Primary (the new config's primary is the intent,
whatever the old config had failed over to), and calls the new shared
`SwapAdapter` helper — now used by both failover sites too, so the swap exists
once: unhook + fire-and-forget dispose -> _factory.Create -> rewire -> bump
_adapterGeneration LAST. `BecomeReconnecting(afterReconfigure: true)` then
reconnects immediately (StartSingleTimer replaces any armed reconnect, so no
racing attempts) with the unstable-disconnect accounting and failover check
suppressed — the link was dropped on purpose, and counting it as an unstable
cycle could trip a failover onto the NEW backup before the new primary had been
tried once. ReSubscribeAll/ReSubscribeAllAlarms then re-issue everything from
the untouched subscription registries. Idempotent: a reconfigure carrying the
configuration already in force is a no-op.
Stop-and-recreate was rejected as destructive: subscriptions live inside
DataConnectionActor and InstanceActor cancels its subscribe-retry timer on first
success, so a rebuilt actor would have no subscribers and nothing would ever
re-subscribe.
Tests: new DataConnectionReconfigureTests (7; 5 fail / 2 pass against a
behavioural revert). DclConnectionArbitrationTests gains two row-97 tests and its
CDC-arrival test now asserts the second push is a reconfigure; its ping-pong
guard also asserts zero reconfigures, since a flip would now show up there.
SiteRuntime-010 renamed to ..._ReissuesReconfigureCommand. 3 fail against a
reverted EnsureDclConnection. DataConnectionLayer.Tests 327/327,
SiteRuntime.Tests 643/643, solution build 0 warnings.
Docs: Component-DataConnectionLayer.md gains an "In-place reconfiguration"
section under Endpoint Redundancy; Component-SiteRuntime.md, and the
docs/components/ mirrors, updated; the arbitration decision memo gains an
appended §9 postscript resolving its residual 1 (residuals 2 and 3 stay open).
This commit is contained in:
+351
@@ -0,0 +1,351 @@
|
||||
using Akka.Actor;
|
||||
using Akka.TestKit.Xunit2;
|
||||
using NSubstitute;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Protocol;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.DataConnection;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
|
||||
using ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Actors;
|
||||
using ZB.MOM.WW.ScadaBridge.HealthMonitoring;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests.Actors;
|
||||
|
||||
/// <summary>
|
||||
/// Deferred register row <b>97</b> — <b>in-place reconfiguration of a LIVE data connection</b>.
|
||||
///
|
||||
/// <para>
|
||||
/// <b>The gap these pin.</b> <c>DataConnectionManagerActor.HandleCreateConnection</c> is
|
||||
/// create-only: it warns and returns for a name it already holds. So the Deployment Manager's
|
||||
/// documented "a changed configuration re-issues the create so the DCL adopts it" was never true
|
||||
/// on ANY node — a corrected endpoint or credential took effect only when the connection was next
|
||||
/// built from scratch (node restart, singleton failover, or a remove-then-add), while the DM's
|
||||
/// hash cache recorded it as applied. Discovered while validating rows 89/90 and recorded as
|
||||
/// residual 1 of <c>docs/plans/2026-08-17-dcl-connection-arbitration-decision.md</c>.
|
||||
/// </para>
|
||||
///
|
||||
/// <para>
|
||||
/// <b>Why the fix is a swap and not a restart.</b> Stopping and re-creating the connection actor
|
||||
/// is destructive: the subscription registry lives inside <c>DataConnectionActor</c>, and
|
||||
/// <c>InstanceActor</c> cancels its per-connection subscribe-retry timer on the first success —
|
||||
/// so nothing would ever re-subscribe. That trades "stale endpoint, data flowing" for "correct
|
||||
/// endpoint, no data until redeploy". <see cref="ReconfigureConnectionCommand"/> instead reuses
|
||||
/// the primary→backup failover machinery: dispose the adapter, build one from the new config,
|
||||
/// bump the adapter generation, and let the reconnect cycle re-issue every subscription from the
|
||||
/// registry that was never touched.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
[Collection("DataConnectionManagerActor")]
|
||||
public class DataConnectionReconfigureTests : TestKit
|
||||
{
|
||||
private const string OldEndpoint = "opc.tcp://old-plc:4840";
|
||||
private const string NewEndpoint = "opc.tcp://new-plc:4840";
|
||||
|
||||
private readonly DataConnectionOptions _options;
|
||||
private readonly ISiteHealthCollector _healthCollector;
|
||||
private readonly IDataConnectionFactory _factory;
|
||||
|
||||
public DataConnectionReconfigureTests()
|
||||
: base(@"akka.loglevel = DEBUG")
|
||||
{
|
||||
_healthCollector = Substitute.For<ISiteHealthCollector>();
|
||||
_factory = Substitute.For<IDataConnectionFactory>();
|
||||
_options = new DataConnectionOptions
|
||||
{
|
||||
ReconnectInterval = TimeSpan.FromMilliseconds(100),
|
||||
TagResolutionRetryInterval = TimeSpan.FromMilliseconds(200),
|
||||
WriteTimeout = TimeSpan.FromSeconds(5),
|
||||
SeedReadMaxAttempts = 1,
|
||||
SeedReadRetryDelay = TimeSpan.FromMilliseconds(10)
|
||||
};
|
||||
}
|
||||
|
||||
private static Dictionary<string, string> Config(string endpoint) =>
|
||||
new() { ["Endpoint"] = endpoint };
|
||||
|
||||
private IActorRef CreateConnectionActor(
|
||||
string name, IDataConnection adapter, IDictionary<string, string> primaryConfig) =>
|
||||
Sys.ActorOf(Props.Create(() =>
|
||||
new DataConnectionActor(
|
||||
name, adapter, _options, _healthCollector, _factory, "OpcUa",
|
||||
primaryConfig, null, 3)), name);
|
||||
|
||||
/// <summary>An adapter that connects, subscribes and reads without incident.</summary>
|
||||
private static IDataConnection HealthyAdapter(string subscriptionId)
|
||||
{
|
||||
var adapter = Substitute.For<IDataConnection>();
|
||||
adapter.ConnectAsync(Arg.Any<IDictionary<string, string>>(), Arg.Any<CancellationToken>())
|
||||
.Returns(Task.CompletedTask);
|
||||
adapter.Status.Returns(ConnectionHealth.Connected);
|
||||
adapter.SubscribeAsync(Arg.Any<string>(), Arg.Any<SubscriptionCallback>(), Arg.Any<CancellationToken>())
|
||||
.Returns(subscriptionId);
|
||||
adapter.ReadAsync(Arg.Any<string>(), Arg.Any<CancellationToken>())
|
||||
.Returns(new ReadResult(false, null, null));
|
||||
return adapter;
|
||||
}
|
||||
|
||||
private static void RaiseDisconnected(IDataConnection adapter) =>
|
||||
adapter.Disconnected += Raise.Event<Action>();
|
||||
|
||||
private bool HasEnteredConnectedState(string connectionName) =>
|
||||
_healthCollector.ReceivedCalls().Any(c =>
|
||||
c.GetMethodInfo().Name == nameof(ISiteHealthCollector.UpdateConnectionHealth) &&
|
||||
c.GetArguments() is [string name, ConnectionHealth.Connected, ..] &&
|
||||
name == connectionName);
|
||||
|
||||
private bool CreatedAdapterFor(string endpoint) =>
|
||||
_factory.ReceivedCalls().Any(c =>
|
||||
c.GetMethodInfo().Name == nameof(IDataConnectionFactory.Create) &&
|
||||
c.GetArguments()[1] is IDictionary<string, string> d &&
|
||||
d.TryGetValue("Endpoint", out var e) && e == endpoint);
|
||||
|
||||
private static bool Subscribed(IDataConnection adapter) =>
|
||||
adapter.ReceivedCalls().Any(c => c.GetMethodInfo().Name == "SubscribeAsync");
|
||||
|
||||
private static bool Connected(IDataConnection adapter) =>
|
||||
adapter.ReceivedCalls().Any(c => c.GetMethodInfo().Name == "ConnectAsync");
|
||||
|
||||
// ── The connection actor ──
|
||||
|
||||
/// <summary>
|
||||
/// The headline behaviour: a live, CONNECTED connection told to reconfigure builds a new
|
||||
/// adapter from the new configuration and re-issues the subscription it already holds against
|
||||
/// it — without the subscriber doing anything, and without the connection actor being stopped.
|
||||
///
|
||||
/// <para>
|
||||
/// <b>Fails before the fix:</b> there was no reconfigure command at all, and the create the
|
||||
/// Deployment Manager re-issued was dropped by the manager's create-only guard, so the old
|
||||
/// adapter kept serving <c>old-plc</c> indefinitely.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void Reconfigure_WhileConnected_SwapsTheAdapter_AndReSubscribesExistingTags()
|
||||
{
|
||||
const string name = "reconf-connected";
|
||||
var oldAdapter = HealthyAdapter("sub-old");
|
||||
var newAdapter = HealthyAdapter("sub-new");
|
||||
_factory.Create("OpcUa", Arg.Is<IDictionary<string, string>>(d => d["Endpoint"] == NewEndpoint))
|
||||
.Returns(newAdapter);
|
||||
|
||||
var actor = CreateConnectionActor(name, oldAdapter, Config(OldEndpoint));
|
||||
AwaitCondition(() => HasEnteredConnectedState(name), TimeSpan.FromSeconds(5),
|
||||
TimeSpan.FromMilliseconds(10));
|
||||
|
||||
actor.Tell(new SubscribeTagsRequest(
|
||||
"c1", "inst1", name, ["sensor/temp"], DateTimeOffset.UtcNow));
|
||||
ExpectMsg<SubscribeTagsResponse>(TimeSpan.FromSeconds(5));
|
||||
AwaitCondition(() => Subscribed(oldAdapter), TimeSpan.FromSeconds(5));
|
||||
|
||||
actor.Tell(new ReconfigureConnectionCommand(
|
||||
name, "OpcUa", Config(NewEndpoint), null, 3));
|
||||
|
||||
AwaitCondition(() => CreatedAdapterFor(NewEndpoint), TimeSpan.FromSeconds(5),
|
||||
TimeSpan.FromMilliseconds(10));
|
||||
AwaitCondition(() => Connected(newAdapter), TimeSpan.FromSeconds(5));
|
||||
|
||||
// The whole point: the subscription survives the swap and is re-issued for free.
|
||||
AwaitCondition(() => Subscribed(newAdapter), TimeSpan.FromSeconds(5),
|
||||
TimeSpan.FromMilliseconds(10));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The adapter generation must be bumped by the swap, exactly as it is on failover: the
|
||||
/// outgoing adapter's subscription callbacks captured Self and keep firing, and a value from
|
||||
/// the endpoint the operator has just corrected AWAY from must never be delivered as if it
|
||||
/// came from the new one.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void Reconfigure_DropsValuesStillArrivingFromTheOldAdapter()
|
||||
{
|
||||
const string name = "reconf-generation";
|
||||
SubscriptionCallback? oldCallback = null;
|
||||
|
||||
var oldAdapter = HealthyAdapter("sub-old");
|
||||
oldAdapter.SubscribeAsync("sensor/temp", Arg.Any<SubscriptionCallback>(), Arg.Any<CancellationToken>())
|
||||
.Returns(ci =>
|
||||
{
|
||||
oldCallback = (SubscriptionCallback)ci[1];
|
||||
return Task.FromResult("sub-old");
|
||||
});
|
||||
|
||||
var newAdapter = HealthyAdapter("sub-new");
|
||||
_factory.Create("OpcUa", Arg.Is<IDictionary<string, string>>(d => d["Endpoint"] == NewEndpoint))
|
||||
.Returns(newAdapter);
|
||||
|
||||
var actor = CreateConnectionActor(name, oldAdapter, Config(OldEndpoint));
|
||||
AwaitCondition(() => HasEnteredConnectedState(name), TimeSpan.FromSeconds(5),
|
||||
TimeSpan.FromMilliseconds(10));
|
||||
|
||||
actor.Tell(new SubscribeTagsRequest(
|
||||
"c1", TestActor.Path.Name, name, ["sensor/temp"], DateTimeOffset.UtcNow));
|
||||
ExpectMsg<SubscribeTagsResponse>(TimeSpan.FromSeconds(5));
|
||||
AwaitCondition(() => oldCallback != null, TimeSpan.FromSeconds(5));
|
||||
|
||||
actor.Tell(new ReconfigureConnectionCommand(
|
||||
name, "OpcUa", Config(NewEndpoint), null, 3));
|
||||
|
||||
// The swap flips every subscribed tag to bad quality, same as a disconnect.
|
||||
ExpectMsg<ConnectionQualityChanged>(TimeSpan.FromSeconds(5));
|
||||
|
||||
AwaitCondition(() => Subscribed(newAdapter), TimeSpan.FromSeconds(5));
|
||||
ExpectNoMsg(TimeSpan.FromMilliseconds(300));
|
||||
|
||||
oldCallback!("sensor/temp", new TagValue(999, QualityCode.Good, DateTimeOffset.UtcNow));
|
||||
|
||||
ExpectNoMsg(TimeSpan.FromSeconds(1));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// <b>The state that matters most, and the one a stash would break.</b> A connection whose
|
||||
/// configured endpoint is wrong or dead never reaches Connected at all — it loops in the
|
||||
/// Connecting state retrying — which is EXACTLY when a corrected configuration is sent. If
|
||||
/// the reconfigure were stashed with the subscribe/write requests it would wait for the old
|
||||
/// (broken) configuration to succeed first, i.e. forever.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void Reconfigure_WhileStillConnecting_IsHonoured_NotStashed()
|
||||
{
|
||||
const string name = "reconf-connecting";
|
||||
var deadAdapter = Substitute.For<IDataConnection>();
|
||||
deadAdapter.ConnectAsync(Arg.Any<IDictionary<string, string>>(), Arg.Any<CancellationToken>())
|
||||
.Returns(Task.FromException(new Exception("Connection refused")));
|
||||
|
||||
var newAdapter = HealthyAdapter("sub-new");
|
||||
_factory.Create("OpcUa", Arg.Is<IDictionary<string, string>>(d => d["Endpoint"] == NewEndpoint))
|
||||
.Returns(newAdapter);
|
||||
|
||||
var actor = CreateConnectionActor(name, deadAdapter, Config(OldEndpoint));
|
||||
AwaitCondition(() => Connected(deadAdapter), TimeSpan.FromSeconds(5));
|
||||
Assert.False(HasEnteredConnectedState(name));
|
||||
|
||||
actor.Tell(new ReconfigureConnectionCommand(
|
||||
name, "OpcUa", Config(NewEndpoint), null, 3));
|
||||
|
||||
AwaitCondition(() => CreatedAdapterFor(NewEndpoint), TimeSpan.FromSeconds(5),
|
||||
TimeSpan.FromMilliseconds(10));
|
||||
AwaitCondition(() => HasEnteredConnectedState(name), TimeSpan.FromSeconds(5),
|
||||
TimeSpan.FromMilliseconds(10));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The same, one state over: a connection that HAD been up and is now looping in the
|
||||
/// Reconnecting state. Reconnecting stashes subscribes and writes too, so the reconfigure
|
||||
/// must be handled there as well — and the subscriptions it re-issues are the ones the
|
||||
/// registry has been holding across the outage.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void Reconfigure_WhileReconnecting_IsHonoured_AndReSubscribes()
|
||||
{
|
||||
const string name = "reconf-reconnecting";
|
||||
var connects = 0;
|
||||
var oldAdapter = HealthyAdapter("sub-old");
|
||||
oldAdapter.ConnectAsync(Arg.Any<IDictionary<string, string>>(), Arg.Any<CancellationToken>())
|
||||
.Returns(_ => Interlocked.Increment(ref connects) == 1
|
||||
? Task.CompletedTask
|
||||
: Task.FromException(new Exception("still down")));
|
||||
|
||||
var newAdapter = HealthyAdapter("sub-new");
|
||||
_factory.Create("OpcUa", Arg.Is<IDictionary<string, string>>(d => d["Endpoint"] == NewEndpoint))
|
||||
.Returns(newAdapter);
|
||||
|
||||
var actor = CreateConnectionActor(name, oldAdapter, Config(OldEndpoint));
|
||||
AwaitCondition(() => HasEnteredConnectedState(name), TimeSpan.FromSeconds(5),
|
||||
TimeSpan.FromMilliseconds(10));
|
||||
|
||||
actor.Tell(new SubscribeTagsRequest(
|
||||
"c1", "inst1", name, ["sensor/temp"], DateTimeOffset.UtcNow));
|
||||
ExpectMsg<SubscribeTagsResponse>(TimeSpan.FromSeconds(5));
|
||||
|
||||
// Drop the link; with no backup configured it retries the same dead endpoint forever.
|
||||
RaiseDisconnected(oldAdapter);
|
||||
AwaitCondition(() => connects >= 2, TimeSpan.FromSeconds(5));
|
||||
|
||||
actor.Tell(new ReconfigureConnectionCommand(
|
||||
name, "OpcUa", Config(NewEndpoint), null, 3));
|
||||
|
||||
AwaitCondition(() => CreatedAdapterFor(NewEndpoint), TimeSpan.FromSeconds(5),
|
||||
TimeSpan.FromMilliseconds(10));
|
||||
AwaitCondition(() => Subscribed(newAdapter), TimeSpan.FromSeconds(5),
|
||||
TimeSpan.FromMilliseconds(10));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// A reconfigure carrying the configuration already in force must cost nothing. The sender is
|
||||
/// hash-gated and does not send one, but an adapter swap drops the link and re-subscribes
|
||||
/// every tag, so a redundant command must never be able to do that.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void Reconfigure_WithTheConfigurationAlreadyInForce_IsANoOp()
|
||||
{
|
||||
const string name = "reconf-identical";
|
||||
var adapter = HealthyAdapter("sub-old");
|
||||
|
||||
var actor = CreateConnectionActor(name, adapter, Config(OldEndpoint));
|
||||
AwaitCondition(() => HasEnteredConnectedState(name), TimeSpan.FromSeconds(5),
|
||||
TimeSpan.FromMilliseconds(10));
|
||||
|
||||
actor.Tell(new ReconfigureConnectionCommand(
|
||||
name, "OpcUa", Config(OldEndpoint), null, 3));
|
||||
|
||||
// No adapter is ever built: the factory is only consulted by a swap.
|
||||
ExpectNoMsg(TimeSpan.FromMilliseconds(500));
|
||||
Assert.False(CreatedAdapterFor(OldEndpoint));
|
||||
Assert.DoesNotContain(_factory.ReceivedCalls(),
|
||||
c => c.GetMethodInfo().Name == nameof(IDataConnectionFactory.Create));
|
||||
}
|
||||
|
||||
// ── The manager's routing ──
|
||||
|
||||
/// <summary>
|
||||
/// The manager forwards a reconfigure to the connection actor that owns the name — which is
|
||||
/// the whole difference from <see cref="CreateConnectionCommand"/>, whose handler warns and
|
||||
/// returns for a name it already holds.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void Manager_Reconfigure_ForALiveConnection_ReachesTheConnectionActor()
|
||||
{
|
||||
var oldAdapter = HealthyAdapter("sub-old");
|
||||
var newAdapter = HealthyAdapter("sub-new");
|
||||
_factory.Create("OpcUa", Arg.Is<IDictionary<string, string>>(d => d["Endpoint"] == OldEndpoint))
|
||||
.Returns(oldAdapter);
|
||||
_factory.Create("OpcUa", Arg.Is<IDictionary<string, string>>(d => d["Endpoint"] == NewEndpoint))
|
||||
.Returns(newAdapter);
|
||||
|
||||
var manager = Sys.ActorOf(Props.Create(() =>
|
||||
new DataConnectionManagerActor(_factory, _options, _healthCollector)));
|
||||
|
||||
manager.Tell(new CreateConnectionCommand("conn1", "OpcUa", Config(OldEndpoint), null, 3));
|
||||
AwaitCondition(() => Connected(oldAdapter), TimeSpan.FromSeconds(10));
|
||||
|
||||
manager.Tell(new ReconfigureConnectionCommand("conn1", "OpcUa", Config(NewEndpoint), null, 3));
|
||||
|
||||
AwaitCondition(() => Connected(newAdapter), TimeSpan.FromSeconds(10),
|
||||
TimeSpan.FromMilliseconds(10));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// A reconfigure for a name the manager does not hold falls back to a create, so the command
|
||||
/// is safe whatever the manager's state — a sender whose view of which connections exist is
|
||||
/// stale still converges on the intended configuration in one message rather than leaving
|
||||
/// the connection missing until something else happens to create it.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void Manager_Reconfigure_ForAnUnknownConnection_FallsBackToCreate()
|
||||
{
|
||||
var adapter = HealthyAdapter("sub-new");
|
||||
_factory.Create("OpcUa", Arg.Is<IDictionary<string, string>>(d => d["Endpoint"] == NewEndpoint))
|
||||
.Returns(adapter);
|
||||
|
||||
var manager = Sys.ActorOf(Props.Create(() =>
|
||||
new DataConnectionManagerActor(_factory, _options, _healthCollector)));
|
||||
|
||||
manager.Tell(new ReconfigureConnectionCommand("never-created", "OpcUa", Config(NewEndpoint), null, 3));
|
||||
|
||||
AwaitCondition(() => Connected(adapter), TimeSpan.FromSeconds(10),
|
||||
TimeSpan.FromMilliseconds(10));
|
||||
|
||||
// And it is a real, routable connection afterwards — not an orphan.
|
||||
manager.Tell(new SubscribeTagsRequest(
|
||||
"c1", "inst1", "never-created", ["sensor/temp"], DateTimeOffset.UtcNow));
|
||||
var response = ExpectMsg<SubscribeTagsResponse>(TimeSpan.FromSeconds(10));
|
||||
Assert.True(response.Success);
|
||||
}
|
||||
}
|
||||
+117
-8
@@ -16,8 +16,8 @@ using RemoveConnectionCommand = ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Actors
|
||||
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors;
|
||||
|
||||
/// <summary>
|
||||
/// Regression tests for deferred register rows <b>89</b> and <b>90</b> — <b>which configuration
|
||||
/// source owns a live DCL connection</b>.
|
||||
/// Regression tests for deferred register rows <b>89</b>, <b>90</b> and <b>97</b> — <b>which
|
||||
/// configuration source owns a live DCL connection, and how a change reaches it</b>.
|
||||
///
|
||||
/// <para>
|
||||
/// <b>The arbitration rule under test.</b> The <c>data_connection_definitions</c> store wins
|
||||
@@ -39,6 +39,14 @@ namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors;
|
||||
/// flattened snapshots and never consults the newer definition rows. Design record:
|
||||
/// <c>docs/plans/2026-08-17-dcl-connection-arbitration-decision.md</c>.
|
||||
/// </para>
|
||||
///
|
||||
/// <para>
|
||||
/// <b>Row 97 — the third command.</b> Resolving the winner decides WHAT to push; it took a
|
||||
/// <see cref="ReconfigureConnectionCommand"/> to make a push against an ALREADY-LIVE connection
|
||||
/// mean anything at all, because the DCL manager's create handler warns and returns for a name it
|
||||
/// already holds. So a name the hash cache has never held gets a create, a changed one gets a
|
||||
/// reconfigure, and an unchanged one gets nothing.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public class DclConnectionArbitrationTests : TestKit, IDisposable
|
||||
{
|
||||
@@ -56,6 +64,7 @@ public class DclConnectionArbitrationTests : TestKit, IDisposable
|
||||
private readonly TestLocalDb _localDb;
|
||||
|
||||
private readonly ConcurrentQueue<CreateConnectionCommand> _creates = new();
|
||||
private readonly ConcurrentQueue<ReconfigureConnectionCommand> _reconfigures = new();
|
||||
private readonly ConcurrentQueue<RemoveConnectionCommand> _removes = new();
|
||||
|
||||
public DclConnectionArbitrationTests()
|
||||
@@ -93,7 +102,8 @@ public class DclConnectionArbitrationTests : TestKit, IDisposable
|
||||
/// </summary>
|
||||
private IActorRef CreateBootedDeploymentManager(int sweepSeconds = SweepSeconds)
|
||||
{
|
||||
var dclSpy = ActorOf(Props.Create(() => new DclConnectionSpy(_creates, _removes)));
|
||||
var dclSpy = ActorOf(Props.Create(() =>
|
||||
new DclConnectionSpy(_creates, _reconfigures, _removes)));
|
||||
var health = new DeploymentManagerStartupProbe();
|
||||
|
||||
var dm = ActorOf(Props.Create(() => new DeploymentManagerActor(
|
||||
@@ -150,12 +160,19 @@ public class DclConnectionArbitrationTests : TestKit, IDisposable
|
||||
private IReadOnlyList<CreateConnectionCommand> CreatesFor(string connectionName) =>
|
||||
_creates.Where(c => c.ConnectionName == connectionName).ToArray();
|
||||
|
||||
private IReadOnlyList<ReconfigureConnectionCommand> ReconfiguresFor(string connectionName) =>
|
||||
_reconfigures.Where(c => c.ConnectionName == connectionName).ToArray();
|
||||
|
||||
/// <summary>
|
||||
/// The endpoint the DCL was actually told to connect to — the whole point of the arbitration.
|
||||
/// </summary>
|
||||
private static string EndpointOf(CreateConnectionCommand command) =>
|
||||
command.PrimaryConnectionDetails["endpoint"];
|
||||
|
||||
/// <inheritdoc cref="EndpointOf(CreateConnectionCommand)"/>
|
||||
private static string EndpointOf(ReconfigureConnectionCommand command) =>
|
||||
command.PrimaryConnectionDetails["endpoint"];
|
||||
|
||||
private async Task AwaitCreateCountAsync(int count, string because) =>
|
||||
await AwaitAssertAsync(() =>
|
||||
{
|
||||
@@ -163,6 +180,13 @@ public class DclConnectionArbitrationTests : TestKit, IDisposable
|
||||
return Task.CompletedTask;
|
||||
}, TimeSpan.FromSeconds(20));
|
||||
|
||||
private async Task AwaitReconfigureCountAsync(int count, string because) =>
|
||||
await AwaitAssertAsync(() =>
|
||||
{
|
||||
Assert.True(ReconfiguresFor(ConnectionName).Count >= count, because);
|
||||
return Task.CompletedTask;
|
||||
}, TimeSpan.FromSeconds(20));
|
||||
|
||||
// ── Rule 1: a definition row outranks an older flattened snapshot ──
|
||||
|
||||
/// <summary>
|
||||
@@ -224,6 +248,10 @@ public class DclConnectionArbitrationTests : TestKit, IDisposable
|
||||
await Task.Delay(SeveralSweeps);
|
||||
|
||||
Assert.Single(CreatesFor(ConnectionName));
|
||||
// Row 97 moved the cost of a flip: a ping-pong now shows up as a stream of RECONFIGURES,
|
||||
// each of which drops the link and re-subscribes every tag. Assert on both, or the guard
|
||||
// silently stops guarding the thing that would actually happen.
|
||||
Assert.Empty(ReconfiguresFor(ConnectionName));
|
||||
Assert.Empty(_removes);
|
||||
}
|
||||
|
||||
@@ -283,8 +311,15 @@ public class DclConnectionArbitrationTests : TestKit, IDisposable
|
||||
/// pushes the new configuration for the connection the runtime already holds.
|
||||
///
|
||||
/// <para>
|
||||
/// <b>Fails before the fix:</b> exactly one create is ever issued, carrying <c>old-plc</c>,
|
||||
/// and no sweep tick ever looks at the definitions table.
|
||||
/// <b>Fails before the fix:</b> nothing is ever issued for the change at all, and no sweep
|
||||
/// tick ever looks at the definitions table.
|
||||
/// </para>
|
||||
///
|
||||
/// <para>
|
||||
/// <b>Register row 97 changed the SHAPE of the second push.</b> The connection already exists
|
||||
/// at this point, so the change goes out as a
|
||||
/// <see cref="ReconfigureConnectionCommand"/> — a second create would be dropped by the DCL
|
||||
/// manager's create-only guard and change nothing.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
[Fact]
|
||||
@@ -302,15 +337,87 @@ public class DclConnectionArbitrationTests : TestKit, IDisposable
|
||||
// Now the definition row lands — a bare storage write, the CDC applier's only side effect.
|
||||
await StoreDefinitionAsync(DefinitionEndpoint);
|
||||
|
||||
await AwaitCreateCountAsync(2,
|
||||
await AwaitReconfigureCountAsync(1,
|
||||
"a definition row that arrived by replication was never picked up: the live DCL kept "
|
||||
+ "the endpoint frozen into an instance's flattened config (register row 89)");
|
||||
|
||||
Assert.Equal(DefinitionEndpoint, EndpointOf(CreatesFor(ConnectionName).Last()));
|
||||
Assert.Equal(DefinitionEndpoint, EndpointOf(ReconfiguresFor(ConnectionName).Last()));
|
||||
|
||||
// And it settles: the overlay now owns the name, so further ticks are silent.
|
||||
await Task.Delay(SeveralSweeps);
|
||||
Assert.Equal(2, CreatesFor(ConnectionName).Count);
|
||||
Assert.Single(CreatesFor(ConnectionName));
|
||||
Assert.Single(ReconfiguresFor(ConnectionName));
|
||||
}
|
||||
|
||||
// ── Row 97: a changed configuration reaches the LIVE connection ──
|
||||
|
||||
/// <summary>
|
||||
/// <b>THE row-97 reproduction on the sender side.</b> A connection the DCL already holds gets
|
||||
/// a new endpoint. What must go out is a <see cref="ReconfigureConnectionCommand"/>, which the
|
||||
/// live connection actor applies in place by swapping its adapter and re-issuing its
|
||||
/// subscriptions.
|
||||
///
|
||||
/// <para>
|
||||
/// <b>Fails before the fix:</b> the Deployment Manager re-issued a
|
||||
/// <see cref="CreateConnectionCommand"/> and logged a Warning saying it would not take effect
|
||||
/// — which was accurate, because <c>DataConnectionManagerActor.HandleCreateConnection</c>
|
||||
/// warns and returns for a name it already holds. The hash cache recorded the change as
|
||||
/// applied regardless, so a corrected endpoint sat unapplied until a node restart, a singleton
|
||||
/// failover, or a remove-then-add.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task ChangedConfigurationForALiveConnection_IsSentAsAReconfigure_NotASecondCreate()
|
||||
{
|
||||
const string instanceName = "ReconfiguredPump";
|
||||
CreateBootedDeploymentManager();
|
||||
|
||||
await _storage.StoreDeployedConfigAsync(
|
||||
instanceName, MakeConfigJson(instanceName), "dep-rc1", "rev-rc1", true);
|
||||
await AwaitCreateCountAsync(1, "the sweep never created the connection");
|
||||
|
||||
// The connection-only artifact deploy that corrects the endpoint.
|
||||
await StoreDefinitionAsync(DefinitionEndpoint);
|
||||
|
||||
await AwaitReconfigureCountAsync(1,
|
||||
"a changed configuration for an ALREADY-LIVE connection was not sent as a "
|
||||
+ "ReconfigureConnectionCommand, so the live connection actor keeps serving the old "
|
||||
+ "endpoint until it is rebuilt from scratch (register row 97)");
|
||||
|
||||
Assert.Equal(DefinitionEndpoint, EndpointOf(ReconfiguresFor(ConnectionName).Single()));
|
||||
|
||||
// A create would have been dropped by the manager's create-only guard — so the change must
|
||||
// NOT have been shipped as one.
|
||||
Assert.Single(CreatesFor(ConnectionName));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The hash guard is unchanged by row 97: an UNCHANGED configuration still sends nothing at
|
||||
/// all. This is what keeps a healthy site silent across every sweep tick, and it is the reason
|
||||
/// a reconfigure — which drops the link and re-subscribes every tag — is safe to send on the
|
||||
/// change edge.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task UnchangedConfiguration_SendsNeitherACreateNorAReconfigure()
|
||||
{
|
||||
const string instanceName = "SteadyPump";
|
||||
|
||||
await StoreDefinitionAsync(DefinitionEndpoint);
|
||||
await _storage.StoreDeployedConfigAsync(
|
||||
instanceName, MakeConfigJson(instanceName), "dep-st", "rev-st", true);
|
||||
|
||||
CreateBootedDeploymentManager();
|
||||
await AwaitCreateCountAsync(1, "the boot load never created the connection");
|
||||
|
||||
// Re-write the SAME definition row several sweeps running, and start a second instance
|
||||
// naming the same connection: neither is a configuration change.
|
||||
await StoreDefinitionAsync(DefinitionEndpoint);
|
||||
await _storage.StoreDeployedConfigAsync(
|
||||
"SteadyPumpTwo", MakeConfigJson("SteadyPumpTwo"), "dep-st2", "rev-st2", true);
|
||||
await Task.Delay(SeveralSweeps);
|
||||
|
||||
Assert.Single(CreatesFor(ConnectionName));
|
||||
Assert.Empty(ReconfiguresFor(ConnectionName));
|
||||
}
|
||||
|
||||
// ── Row 90: removal, and not being resurrected by a flattened snapshot ──
|
||||
@@ -469,9 +576,11 @@ public class DclConnectionArbitrationTests : TestKit, IDisposable
|
||||
{
|
||||
public DclConnectionSpy(
|
||||
ConcurrentQueue<CreateConnectionCommand> creates,
|
||||
ConcurrentQueue<ReconfigureConnectionCommand> reconfigures,
|
||||
ConcurrentQueue<RemoveConnectionCommand> removes)
|
||||
{
|
||||
Receive<CreateConnectionCommand>(creates.Enqueue);
|
||||
Receive<ReconfigureConnectionCommand>(reconfigures.Enqueue);
|
||||
Receive<RemoveConnectionCommand>(removes.Enqueue);
|
||||
ReceiveAny(_ => { });
|
||||
}
|
||||
|
||||
+16
-8
@@ -166,12 +166,19 @@ public class DeploymentManagerMediumFindingsTests : TestKit, IDisposable
|
||||
|
||||
/// <summary>
|
||||
/// SiteRuntime-010: when a redeployment changes a connection's configuration
|
||||
/// (here the failover retry count and endpoint), the Deployment Manager must
|
||||
/// re-issue a <see cref="ZB.MOM.WW.ScadaBridge.Commons.Messages.DataConnection.CreateConnectionCommand"/>
|
||||
/// so the DCL adopts the new configuration rather than keeping the stale one.
|
||||
/// (here the failover retry count and endpoint), the Deployment Manager must push the new
|
||||
/// configuration to the DCL rather than leaving it serving the stale one.
|
||||
///
|
||||
/// <para>
|
||||
/// <b>Register row 97 changed WHICH command that is.</b> It used to be a re-issued
|
||||
/// <c>CreateConnectionCommand</c> — which the create-only manager silently dropped for a name
|
||||
/// it already held, so this test proved a message was sent and nothing more. It is now a
|
||||
/// <see cref="ZB.MOM.WW.ScadaBridge.Commons.Messages.DataConnection.ReconfigureConnectionCommand"/>,
|
||||
/// which the live connection actor actually applies in place.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task EnsureDclConnections_ConnectionConfigChanged_ReissuesCreateCommand()
|
||||
public async Task EnsureDclConnections_ConnectionConfigChanged_ReissuesReconfigureCommand()
|
||||
{
|
||||
var storage = NewStorage(_localDb.Db);
|
||||
await storage.InitializeAsync();
|
||||
@@ -198,11 +205,12 @@ public class DeploymentManagerMediumFindingsTests : TestKit, IDisposable
|
||||
MakeConfigJsonWithConnection("ConnPump", "opc.tcp://host-b:4840", 7),
|
||||
"admin", DateTimeOffset.UtcNow));
|
||||
|
||||
// The DCL must receive a fresh create command reflecting the new config.
|
||||
var secondCreate = dcl.ExpectMsg<ZB.MOM.WW.ScadaBridge.Commons.Messages.DataConnection.CreateConnectionCommand>(
|
||||
// The DCL must receive a RECONFIGURE reflecting the new config — not a second create,
|
||||
// which the create-only manager would drop for a name it already holds.
|
||||
var reconfigure = dcl.ExpectMsg<ZB.MOM.WW.ScadaBridge.Commons.Messages.DataConnection.ReconfigureConnectionCommand>(
|
||||
TimeSpan.FromSeconds(10));
|
||||
Assert.Equal("Conn1", secondCreate.ConnectionName);
|
||||
Assert.Equal(7, secondCreate.FailoverRetryCount);
|
||||
Assert.Equal("Conn1", reconfigure.ConnectionName);
|
||||
Assert.Equal(7, reconfigure.FailoverRetryCount);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
|
||||
Reference in New Issue
Block a user