dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 2 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

feat(localdb)!: replicate site config + sf_messages via CDC, delete the bespoke replicators

Browse Source 
Tasks 14, 15 and 16, landed as ONE commit.

PLAN DEFECT: these three tasks cannot compile separately. SiteReplicationActor
takes a ReplicationService and calls ReplaceAllAsync (Task 14 deletes both);
DeploymentManagerActor Tells message types declared in ReplicationMessages.cs
(Task 15 deletes it); AkkaHostedService constructs the actor (Task 16). Any
ordering leaves a broken intermediate. Combining them also strengthens the
invariant Task 14 already stated for itself — the two mechanisms never both
run, and never neither.

Registered 8 tables in SiteLocalDbSetup.OnReady: sf_messages plus the 7 site
config tables. notification_lists and smtp_configurations are deliberately NOT
registered — permanently empty by design, so registering them would open a
standing replication channel whose only historical payload was plaintext SMTP
passwords. Migrate stays the LAST call in OnReady, after all registrations, so
migrated rows enter the oplog through live capture triggers.

Deleted: SiteReplicationActor, ReplicationMessages.cs, ReplicationService,
StoreAndForwardStorage.ReplaceAllAsync, and 6 test files. ReplaceAllAsync is
not merely unused but unsafe to keep: a mass DELETE on a now-replicated table
would be captured and shipped to the peer.

Kept ActiveNodeEvaluator (delivery gate + heartbeat still need it) with its doc
corrected, and activeNodeCheck in AkkaHostedService (SiteCommunicationActor).

The positional-argument hazard the plan flagged was real: removing
DeploymentManagerActor's optional IActorRef? replicationActor shifted 6
trailing optionals, and 4 test call sites bound the wrong arguments with no
compile error at some positions. Converted them to named arguments where
possible — Props.Create builds an expression tree, which rejects out-of-position
named args, so the rest are padded positionally with a comment saying why.

The Task 7 'not yet registered' test was INVERTED rather than deleted, and is
exact in both directions: too few means a table silently stops replicating, too
many means the SMTP tables leak. Added a separate security-named test for those
two, and a composite-PK test (LWW keys on the full PK, so a truncated key set
would collapse distinct rows). The convergence suites now get their
registrations from the real OnReady — their temporary harness registration is
deleted, so they prove the cutover rather than agreeing with themselves.

Verified: build 0 warnings; SiteRuntime 512, StoreAndForward 130, Host 330,
AuditLog 355, ExternalSystemGateway 142, HealthMonitoring 97, LocalDb
integration 16 — all pass, 0 failures.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
This commit is contained in:
Joseph Doherty
2026-07-20 04:20:05 -04:00
parent df0c6031ba
commit 037798b367
26 changed files with 160 additions and 2513 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CompositionRootTests.cs
-1
View File
@@ -471,7 +471,6 @@ public class SiteCompositionRootTests : IDisposable
new object[] { typeof(IDataConnectionFactory) },
new object[] { typeof(StoreAndForwardStorage) },
new object[] { typeof(StoreAndForwardService) },
new object[] { typeof(ReplicationService) },
new object[] { typeof(ISiteEventLogger) },
new object[] { typeof(IEventLogQueryService) },
new object[] { typeof(ISiteIdentityProvider) },
tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbWiringTests.cs
+51 -8
View File
@@ -243,22 +243,65 @@ public class SiteLocalDbWiringTests : IDisposable
}
[Fact]
public void Site_LocalDb_DoesNotYetRegisterThePhase2Tables()
public void Site_LocalDb_RegistersExactlyTheTenReplicatedTables()
{
// The "not yet" is the assertion that matters, and it is why this is an EXACTLY
// check rather than a Contains. Until the Task 14 cutover, the bespoke
// SiteReplicationActor and the StoreAndForward ReplicationService still own these
// tables. Registering them early would run two replicators over the same rows and
// hide a defect in either one behind the other's writes.
// The Phase 2 cutover. This is an EXACTLY check rather than a Contains in both
// directions, and both directions are load-bearing.
//
// When Task 14 lands, this test does not get deleted — it gets inverted.
// Too few: the table silently stops replicating. Storage still works, every other
// test still passes, and the pair just quietly diverges — the failure mode Phase 1
// existed to end.
//
// Too many: notification_lists and smtp_configurations must NOT be here. They are
// permanently empty by design, and registering them would open a standing
// replication channel whose only historical payload was plaintext SMTP passwords.
// A Contains-based test would never catch that.
var db = _host.Services.GetRequiredService<ILocalDb>();
Assert.Equal(
["OperationTracking", "site_events"],
[
// Ordinal order: '_' (0x5F) sorts before 'b' (0x62), so
// data_connection_definitions precedes database_connections.
"OperationTracking", "data_connection_definitions", "database_connections",
"deployed_configurations", "external_systems", "native_alarm_state",
"sf_messages", "shared_scripts", "site_events", "static_attribute_overrides",
],
db.ReplicatedTables.Keys.OrderBy(k => k, StringComparer.Ordinal).ToArray());
}
[Fact]
public void Site_LocalDb_DoesNotReplicateTheCentralOnlyNotificationTables()
{
// Stated separately from the exact-set test above because this one is a security
// property, not a wiring property, and deserves to fail with its own name. The
// tables exist (SiteStorageSchema creates them) — they simply must never be
// captured. See SiteLocalDbSetup.OnReady for the rationale.
var db = _host.Services.GetRequiredService<ILocalDb>();
Assert.DoesNotContain("notification_lists", db.ReplicatedTables.Keys);
Assert.DoesNotContain("smtp_configurations", db.ReplicatedTables.Keys);
}
[Fact]
public void Site_LocalDb_ReplicatedPhase2TablesHaveTheirExpectedPrimaryKeys()
{
// RegisterReplicated refuses a table with no explicit PK, so reaching these
// assertions proves the DDL ran before registration. The composite keys are the
// interesting ones: LWW conflict resolution keys on the FULL PK, so a wrong or
// truncated key set would silently collapse distinct rows into one.
var db = _host.Services.GetRequiredService<ILocalDb>();
Assert.Equal(["id"], db.ReplicatedTables["sf_messages"].PkColumns);
Assert.Equal(
["instance_unique_name"], db.ReplicatedTables["deployed_configurations"].PkColumns);
Assert.Equal(
["instance_unique_name", "attribute_name"],
db.ReplicatedTables["static_attribute_overrides"].PkColumns);
Assert.Equal(
["instance_unique_name", "source_canonical_name", "source_reference"],
db.ReplicatedTables["native_alarm_state"].PkColumns);
}
private static HashSet<string> TableNames(ILocalDb db)
{
using var connection = db.CreateConnection();
tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/SfBufferResyncPredicateTests.cs
-122
View File
@@ -1,122 +0,0 @@
using Akka.Actor;
using Microsoft.Extensions.Logging.Abstractions;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
using ZB.MOM.WW.ScadaBridge.StoreAndForward;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
using ZB.MOM.WW.ScadaBridge.TestSupport;
namespace ZB.MOM.WW.ScadaBridge.IntegrationTests.Cluster;
/// <summary>
/// N1 regression (review 02 round 2, Critical): the resync authority must use the same
/// oldest-Up predicate as the S&F delivery gate. Divergence scenario = the delivering node
/// is OLDEST but not LEADER (leader = lowest address), the exact state a rolling restart of
/// the lower-address node produces. Pre-fix the delivering node requests a resync from the
/// stale peer and ReplaceAllAsync wipes its live buffer.
/// </summary>
public class SfBufferResyncPredicateTests
{
[Fact]
public async Task OldestButNotLeaderNode_KeepsItsBuffer_AndSeedsTheJoiner()
{
// Two explicit ports, deliberately assigned so the FIRST-started (oldest,
// delivering) node has the HIGHER address → the second node is cluster leader.
var p1 = TwoNodeClusterFixture.GetFreeTcpPort();
var p2 = TwoNodeClusterFixture.GetFreeTcpPort();
var (portHigh, portLow) = p1 > p2 ? (p1, p2) : (p2, p1);
var fixture = await TwoNodeClusterFixture.StartAsync(
role: "site-int", portA: portHigh, portB: portLow);
// The S&F stores AND SiteStorageService take an ILocalDb (LocalDb has no in-memory
// mode), so each node gets its own temp-file local databases. They are disposed
// AFTER the cluster is shut down — the actors hold connections while the systems
// are alive — and only then are the files (plus their WAL sidecars) deleted.
var localDbs = new List<TestLocalDb>();
try
{
// Real S&F storage + replication actor per node, production default predicate
// (no isActiveOverride) — the exact wiring under test.
var (storageOldest, _, sfDbOldest, siteDbOldest) =
await CreateReplicationActorAsync(fixture.NodeA, "oldest");
localDbs.Add(sfDbOldest);
localDbs.Add(siteDbOldest);
var (storageJoiner, _, sfDbJoiner, siteDbJoiner) =
await CreateReplicationActorAsync(fixture.NodeB, "joiner");
localDbs.Add(sfDbJoiner);
localDbs.Add(siteDbJoiner);
// The delivering (oldest) node has a live buffered row the standby never saw.
await storageOldest.EnqueueAsync(NewMessage("live-row"));
// Trigger peer (re)tracking on both sides: each actor got InitialStateAsSnapshot
// in PreStart, but the enqueue raced it — re-deliver via a fresh MemberUp is not
// needed; OnPeerTracked already fired on join. The resync exchange is async:
// wait until the JOINER holds the row (proves the snapshot flowed oldest→joiner,
// the correct direction). Pre-fix this times out (the joiner, as leader, never
// requests) AND the oldest node's row is deleted by the stale wipe.
await AwaitAsync(async () => await storageJoiner.GetMessageByIdAsync("live-row") != null,
TimeSpan.FromSeconds(20),
"joiner never received the resync snapshot (resync ran in the wrong direction)");
// And the delivering node's buffer is untouched — the N1 wipe assertion.
Assert.NotNull(await storageOldest.GetMessageByIdAsync("live-row"));
}
finally
{
await fixture.DisposeAsync();
foreach (var localDb in localDbs)
{
var path = localDb.Path;
localDb.Dispose();
TestLocalDb.DeleteFiles(path);
}
}
}
private static async Task<(
StoreAndForwardStorage Storage, IActorRef Actor, TestLocalDb SfLocalDb, TestLocalDb SiteLocalDb)>
CreateReplicationActorAsync(ActorSystem node, string tag)
{
var sfLocalDb = TestLocalDb.CreateTemp($"sf-resync-{tag}");
var sfStorage = new StoreAndForwardStorage(sfLocalDb.Db,
NullLogger<StoreAndForwardStorage>.Instance);
await sfStorage.InitializeAsync();
var siteLocalDb = TestLocalDb.CreateTemp($"site-resync-{tag}");
var siteStorage = new SiteStorageService(siteLocalDb.Db,
NullLogger<SiteStorageService>.Instance);
var replicationService = new ReplicationService(
new StoreAndForwardOptions(), NullLogger<ReplicationService>.Instance);
// Name MUST be "site-replication" — SendToPeer targets /user/site-replication.
var actor = node.ActorOf(Props.Create(() => new SiteReplicationActor(
siteStorage, sfStorage, replicationService, "site-int",
NullLogger<SiteReplicationActor>.Instance, null, null, null, null)),
"site-replication");
return (sfStorage, actor, sfLocalDb, siteLocalDb);
}
private static StoreAndForwardMessage NewMessage(string id) => new()
{
Id = id,
Category = StoreAndForwardCategory.Notification,
Target = "central",
PayloadJson = "{}",
CreatedAt = DateTimeOffset.UtcNow,
Status = StoreAndForwardMessageStatus.Pending,
MaxRetries = 0,
};
private static async Task AwaitAsync(Func<Task<bool>> condition, TimeSpan timeout, string why)
{
var deadline = DateTime.UtcNow + timeout;
while (DateTime.UtcNow < deadline)
{
if (await condition()) return;
await Task.Delay(250);
}
throw new TimeoutException(why);
}
}
tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbSitePairHarness.cs
+4 -30
View File
@@ -129,40 +129,14 @@ public abstract class LocalDbSitePairHarness : IAsyncLifetime
.Build();
return new ServiceCollection()
.AddZbLocalDb(config, db =>
{
SiteLocalDbSetup.OnReady(db, config);
RegisterPhase2TablesUntilCutover(db);
})
.AddZbLocalDb(config, db => SiteLocalDbSetup.OnReady(db, config))
.BuildServiceProvider();
}
/// <summary>
/// Registers the Phase 2 tables for capture, which production <c>OnReady</c> does not do
/// until the Task 14 cutover.
/// </summary>
/// <remarks>
/// <b>Delete this method at Task 14</b>, along with its call above. Until the cutover the
/// bespoke <c>SiteReplicationActor</c> and <c>ReplicationService</c> still own these
/// tables, so <c>OnReady</c> deliberately leaves them unregistered — but the convergence
/// specifications the cutover has to satisfy need capture triggers to mean anything. The
/// tables are empty at this point, so registering here captures nothing retroactively;
/// the ordering guarantee under test is unaffected.
/// <para>
/// After Task 14 these registrations come from <c>OnReady</c> itself, and leaving this
/// method behind would mask a cutover that forgot one — the whole point of these tests.
/// </para>
/// </remarks>
private static void RegisterPhase2TablesUntilCutover(ILocalDb db)
{
foreach (var table in Phase2ReplicatedTables)
db.RegisterReplicated(table);
}
/// <summary>
/// The eight tables Task 14 registers. Listed literally rather than derived from
/// production code, so that a cutover which registers the wrong set fails these tests
/// instead of agreeing with itself.
/// The eight tables the Phase 2 cutover registers. Listed literally rather than derived
/// from production code, so a registration that drifts fails these tests instead of
/// agreeing with itself.
/// </summary>
/// <remarks>
/// <c>notification_lists</c> and <c>smtp_configurations</c> are absent by design. They
tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs
+20 -14
View File
@@ -67,12 +67,14 @@ public class DeploymentManagerActorTests : TestKit, IDisposable
null, // no stream manager in tests
options,
NullLogger<DeploymentManagerActor>.Instance,
null,
null,
null,
serviceProvider,
null,
configFetcher)));
// Named from here on. These trailing parameters are all optional and several
// share a type, so a positional list silently binds the wrong argument when the
// signature changes — which is exactly what removing replicationActor did.
dclManager: null,
healthCollector: null,
serviceProvider: serviceProvider,
loggerFactory: null,
configFetcher: configFetcher)));
}
private static string MakeConfigJson(string instanceName)
@@ -301,8 +303,11 @@ public class DeploymentManagerActorTests : TestKit, IDisposable
var dm = ActorOf(Props.Create(() => new DeploymentManagerActor(
_storage, _compilationService, _sharedScriptLibrary, null,
new SiteRuntimeOptions(), NullLogger<DeploymentManagerActor>.Instance,
null, null, null, null, null, null,
TimeSpan.FromMilliseconds(200), loader)));
// dclManager, healthCollector, serviceProvider, loggerFactory, configFetcher.
// Props.Create builds an expression tree, which rejects named arguments that
// are out of position, so the optional tail has to be padded positionally.
null, null, null, null, null,
startupLoadRetryInterval: TimeSpan.FromMilliseconds(200), configLoader: loader)));
AwaitAssert(() =>
{
@@ -892,13 +897,14 @@ public class DeploymentManagerActorTests : TestKit, IDisposable
// Security cleanup. notification_lists and smtp_configurations can hold plaintext
// SMTP passwords written by a pre-2026-07-10 build, and the ACTIVE node's artifact
// apply is what clears them (DeploymentManagerActor.HandleDeployArtifacts). The
// standby's copy of this call lives in SiteReplicationActor and dies with it at
// Task 15, which makes this call site the ONLY remaining one.
// standby used to hold a second copy of this call in SiteReplicationActor; LocalDb
// Phase 2 deleted that actor, so this is now the ONLY call site that keeps the
// tables empty.
//
// Nothing pins it today: ArtifactStorageTests covers the storage method, not the
// actor's call to it, so Task 16's edits to this actor could drop the call and every
// suite would stay green. That is precisely the kind of silent security regression
// this test exists to prevent — verified red-first by commenting out the call.
// ArtifactStorageTests covers the storage method, not the actor's call to it, so
// without this test the call could be dropped and every suite would stay green.
// That is precisely the kind of silent security regression it exists to prevent —
// verified red-first by commenting out the call.
await SeedCentralOnlyRowsAsync();
Assert.Equal(1, await RowCountAsync("notification_lists"));
Assert.Equal(1, await RowCountAsync("smtp_configurations"));
tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerCertReconcileTests.cs
+1 -2
View File
@@ -80,8 +80,7 @@ akka {
private IActorRef CreateDeploymentManager() =>
ActorOf(Props.Create(() => new DeploymentManagerActor(
_storage, _compilationService, _sharedScriptLibrary,
null, new SiteRuntimeOptions(), NullLogger<DeploymentManagerActor>.Instance,
null, null, null, null, null, null, null, null)));
null, new SiteRuntimeOptions(), NullLogger<DeploymentManagerActor>.Instance)));
[Fact]
public void SiteNodeJoined_PushesLocalTrustedCertsToJoinedNode()
tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerLoggerFactoryTests.cs
+4 -5
View File
@@ -103,11 +103,10 @@ public class DeploymentManagerLoggerFactoryTests : TestKit, IDisposable
null,
new SiteRuntimeOptions { StartupBatchSize = 100, StartupBatchDelayMs = 5 },
NullLogger<DeploymentManagerActor>.Instance,
null,
null,
null,
null,
loggerFactory)));
// dclManager, healthCollector, serviceProvider — padded positionally because
// Props.Create is an expression tree and rejects out-of-position named args.
null, null, null,
loggerFactory: loggerFactory)));
// Allow async startup (load configs + staggered creation).
await Task.Delay(2000);
tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerRedeployTests.cs
+4 -3
View File
@@ -61,10 +61,11 @@ public class DeploymentManagerRedeployTests : TestKit, IDisposable
null,
new SiteRuntimeOptions(),
NullLogger<DeploymentManagerActor>.Instance,
// dclManager — padded positionally because Props.Create is an expression tree
// and rejects out-of-position named args.
null,
null,
healthCollector,
serviceProvider)));
healthCollector: healthCollector,
serviceProvider: serviceProvider)));
}
/// <summary>
tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/SiteReplicationActorTests.cs
-568
View File
@@ -1,568 +0,0 @@
using System.Collections.Concurrent;
using System.Diagnostics.Metrics;
using Akka.Actor;
using Akka.TestKit.Xunit2;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Logging.Abstractions;
using ZB.MOM.WW.ScadaBridge.SiteRuntime;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Deployment;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Messages;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
using ZB.MOM.WW.ScadaBridge.StoreAndForward;
using ZB.MOM.WW.ScadaBridge.Commons.Observability;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
using ZB.MOM.WW.ScadaBridge.TestSupport;
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors;
/// <summary>
/// Tests for <see cref="SiteReplicationActor"/>'s notify-and-fetch config replication:
/// the active node now replicates an id-only <see cref="ReplicateConfigDeploy"/> (no inline
/// config JSON — killing the intra-site 128 KB frame trap), and the standby fetches the
/// config from central over HTTP and writes it with the older-write guard.
/// </summary>
public class SiteReplicationActorTests : TestKit, IDisposable
{
// Cluster provider is required because SiteReplicationActor calls Cluster.Get in its ctor
// and subscribes to cluster events in PreStart. We use the in-memory TestTransport (not
// dot-netty) so no real socket is bound and no DNS lookup happens — the actor only needs
// the cluster extension to load; these tests never form a real two-node cluster.
private const string ClusterConfig = @"
akka {
actor { provider = cluster }
remote {
enabled-transports = [""akka.remote.test""]
test {
transport-class = ""Akka.Remote.Transport.TestTransport, Akka.Remote""
applied-adapters = []
registry-key = site-repl-test
local-address = ""test://site-repl@localhost:1""
maximum-payload-bytes = 128000b
scheme-identifier = test
}
}
cluster { roles = [""site-test""] }
loglevel = WARNING
}";
private const string SiteRole = "site-test";
private readonly SiteStorageService _storage;
private readonly TestLocalDb _siteLocalDb;
private readonly TestLocalDb _sfLocalDb;
private readonly StoreAndForwardStorage _sfStorage;
private readonly ReplicationService _replicationService;
private readonly string _sfDbFile;
public SiteReplicationActorTests() : base(ClusterConfig, "site-repl")
{
_sfDbFile = Path.Combine(Path.GetTempPath(), $"site-repl-sf-{Guid.NewGuid():N}.db");
// SiteStorageService takes an ILocalDb now; LocalDb has no in-memory mode, so the
// site store gets its own temp-file database alongside the S&F one.
_siteLocalDb = TestLocalDb.CreateTemp("site-repl-test");
_storage = new SiteStorageService(
_siteLocalDb.Db, NullLogger<SiteStorageService>.Instance);
_storage.InitializeAsync().GetAwaiter().GetResult();
_sfLocalDb = TestLocalDb.Create(_sfDbFile);
_sfStorage = new StoreAndForwardStorage(
_sfLocalDb.Db, NullLogger<StoreAndForwardStorage>.Instance);
_sfStorage.InitializeAsync().GetAwaiter().GetResult();
_replicationService = new ReplicationService(
new StoreAndForwardOptions(), NullLogger<ReplicationService>.Instance);
}
void IDisposable.Dispose()
{
Shutdown();
// The master connection anchors the WAL — dispose before deleting.
var siteDbPath = _siteLocalDb.Path;
_siteLocalDb.Dispose();
_sfLocalDb.Dispose();
TestLocalDb.DeleteFiles(siteDbPath);
TestLocalDb.DeleteFiles(_sfDbFile);
}
private IActorRef CreateReplicationActor(IDeploymentConfigFetcher fetcher) =>
ActorOf(Props.Create(() => new SiteReplicationActor(
_storage, _sfStorage, _replicationService, SiteRole,
NullLogger<SiteReplicationActor>.Instance, fetcher)));
private IActorRef CreateReplicationActor(
IDeploymentConfigFetcher fetcher, SiteRuntimeOptions options, TimeSpan retryDelay) =>
ActorOf(Props.Create(() => new SiteReplicationActor(
_storage, _sfStorage, _replicationService, SiteRole,
NullLogger<SiteReplicationActor>.Instance, fetcher, null, options, retryDelay)));
[Fact]
public async Task ReplicatedFetch_RetriesUpToConfigFetchRetryCount()
{
// The first two fetches fail transiently; the third succeeds. With
// ConfigFetchRetryCount = 3 the standby must retry to the third attempt and
// then guarded-write the fetched config (a short retry delay keeps the test fast).
var attempts = 0;
var fetcher = new FakeConfigFetcher(_ =>
Interlocked.Increment(ref attempts) < 3
? Task.FromException<string>(new InvalidOperationException("central hiccup"))
: Task.FromResult("{\"instanceUniqueName\":\"RetryPump\"}"));
var actor = CreateReplicationActor(
fetcher, new SiteRuntimeOptions { ConfigFetchRetryCount = 3 },
TimeSpan.FromMilliseconds(50));
actor.Tell(new ApplyConfigDeploy(
"RetryPump", "dep-r1", "sha256:r1", true,
"http://central:9000", "tok-r1"));
await AwaitAssertAsync(async () =>
{
Assert.Equal(3, Volatile.Read(ref attempts));
var configs = await _storage.GetAllDeployedConfigsAsync();
Assert.Single(configs, c => c.InstanceUniqueName == "RetryPump");
}, TimeSpan.FromSeconds(10));
}
[Fact]
public async Task ApplyConfigDeploy_StandbyFetchesConfigAndGuardedWrites()
{
// The standby receives an id-only ApplyConfigDeploy; it fetches the config from
// central using the message's coords, then guarded-writes the fetched config.
const string configJson = "{\"instanceUniqueName\":\"Pump1\"}";
var fetcher = new FakeConfigFetcher(_ => Task.FromResult(configJson));
var actor = CreateReplicationActor(fetcher);
actor.Tell(new ApplyConfigDeploy(
"Pump1", "dep-100", "sha256:abc", true,
"http://central:9000", "tok-xyz"));
// The continuation runs off-thread; await the guarded write landing.
await AwaitAssertAsync(async () =>
{
var configs = await _storage.GetAllDeployedConfigsAsync();
var row = Assert.Single(configs, c => c.InstanceUniqueName == "Pump1");
Assert.Equal(configJson, row.ConfigJson);
Assert.Equal("dep-100", row.DeploymentId);
Assert.Equal("sha256:abc", row.RevisionHash);
Assert.True(row.IsEnabled);
}, TimeSpan.FromSeconds(5));
// The fetcher was called with the message's coords.
var call = Assert.Single(fetcher.Calls);
Assert.Equal("http://central:9000", call.BaseUrl);
Assert.Equal("dep-100", call.DeploymentId);
Assert.Equal("tok-xyz", call.Token);
}
[Fact]
public async Task ApplyConfigDeploy_Superseded404_SkipsWriteAndActorSurvives()
{
// A 404 (superseded/expired) surfaces as DeploymentConfigFetchException{IsSuperseded}.
// The standby must skip the write, observe the exception (no crash), and stay alive.
var fetcher = new FakeConfigFetcher(_ =>
Task.FromException<string>(
new DeploymentConfigFetchException("expired", isSuperseded: true)));
var actor = CreateReplicationActor(fetcher);
actor.Tell(new ApplyConfigDeploy(
"GonePump", "dep-stale", "sha256:gone", true,
"http://central:9000", "tok-stale"));
// The fetch was attempted...
await AwaitAssertAsync(() =>
{
Assert.Single(fetcher.Calls);
return Task.CompletedTask;
}, TimeSpan.FromSeconds(5));
// ...the actor did not crash (no Terminated to its watcher within the window)...
Watch(actor);
ExpectNoMsg(TimeSpan.FromMilliseconds(500));
// ...and nothing was written for the superseded instance.
var configs = await _storage.GetAllDeployedConfigsAsync();
Assert.DoesNotContain(configs, c => c.InstanceUniqueName == "GonePump");
}
[Fact]
public async Task ApplyConfigDeploy_EmptyFetchCoords_SkipsFetchAndWrite()
{
// The direct DeployInstanceCommand cross-cluster wire path was retired in Task 14.
// This tests the defensive guard: if empty coords arrive, the actor must skip quietly
// — no FetchAsync("") call, no write — rather than erroring.
var fetcher = new FakeConfigFetcher(_ => Task.FromResult("never"));
var actor = CreateReplicationActor(fetcher);
actor.Tell(new ApplyConfigDeploy(
"NoCoordsPump", "dep-direct", "sha256:nc", true,
CentralFetchBaseUrl: "", FetchToken: ""));
// Give any (erroneous) async continuation time to run, then prove neither happened.
Watch(actor);
ExpectNoMsg(TimeSpan.FromMilliseconds(500));
Assert.Empty(fetcher.Calls);
var configs = await _storage.GetAllDeployedConfigsAsync();
Assert.DoesNotContain(configs, c => c.InstanceUniqueName == "NoCoordsPump");
}
[Fact]
public void ReplicateConfigDeploy_MapsToIdOnlyApplyConfigDeploy_ForPeer()
{
// The outbound mapping must forward an id-only ApplyConfigDeploy carrying the fetch
// coords (and NO inline config) to the peer.
var probe = CreateTestProbe();
var fetcher = new FakeConfigFetcher(_ => Task.FromResult("unused"));
var actor = ActorOf(Props.Create(() => new ProbeForwardingReplicationActor(
_storage, _sfStorage, _replicationService, SiteRole,
NullLogger<SiteReplicationActor>.Instance, fetcher, probe.Ref)));
actor.Tell(new ReplicateConfigDeploy(
"Pump2", "dep-200", "sha256:def", false,
"http://central:9000", "tok-abc"));
var applied = probe.ExpectMsg<ApplyConfigDeploy>(TimeSpan.FromSeconds(3));
Assert.Equal("Pump2", applied.InstanceName);
Assert.Equal("dep-200", applied.DeploymentId);
Assert.Equal("sha256:def", applied.RevisionHash);
Assert.False(applied.IsEnabled);
Assert.Equal("http://central:9000", applied.CentralFetchBaseUrl);
Assert.Equal("tok-abc", applied.FetchToken);
}
// ── Task 21: peer-join S&F buffer resync (anti-entropy) ──
[Fact]
public void StandbyTrackingPeer_SendsResyncRequest()
{
var probe = CreateTestProbe();
var actor = ActorOf(Props.Create(() => new ResyncTestActor(
_storage, _sfStorage, _replicationService, SiteRole,
NullLogger<SiteReplicationActor>.Instance, probe.Ref, () => false)));
actor.Tell(new TriggerPeerTracked()); // stands in for TryTrackPeer's MemberUp path
probe.ExpectMsg<RequestSfBufferResync>(TimeSpan.FromSeconds(3));
}
[Fact]
public void ActiveTrackingPeer_DoesNotRequestResync()
{
var probe = CreateTestProbe();
var actor = ActorOf(Props.Create(() => new ResyncTestActor(
_storage, _sfStorage, _replicationService, SiteRole,
NullLogger<SiteReplicationActor>.Instance, probe.Ref, () => true)));
actor.Tell(new TriggerPeerTracked());
probe.ExpectNoMsg(TimeSpan.FromMilliseconds(300)); // active node never requests a resync
}
[Fact]
public async Task ActiveNode_AnswersResyncRequest_WithChunkedSnapshot()
{
// Post-R2-T5 the active node answers with byte-budgeted SfBufferSnapshotChunk(s)
// (a single small row rides one chunk) rather than the monolithic SfBufferSnapshot.
await _sfStorage.EnqueueAsync(NewSfMessage("m1"));
var probe = CreateTestProbe();
var actor = ActorOf(Props.Create(() => new ResyncTestActor(
_storage, _sfStorage, _replicationService, SiteRole,
NullLogger<SiteReplicationActor>.Instance, probe.Ref, () => true)));
actor.Tell(new RequestSfBufferResync(), TestActor);
var chunk = ExpectMsg<SfBufferSnapshotChunk>(TimeSpan.FromSeconds(3));
Assert.Equal(1, chunk.TotalChunks);
Assert.Equal(1, chunk.Sequence);
Assert.Single(chunk.Messages);
Assert.False(chunk.Truncated);
}
[Fact]
public async Task StandbyNode_AppliesSnapshot_ReplacingItsBuffer()
{
await _sfStorage.EnqueueAsync(NewSfMessage("stale"));
var probe = CreateTestProbe();
var actor = ActorOf(Props.Create(() => new ResyncTestActor(
_storage, _sfStorage, _replicationService, SiteRole,
NullLogger<SiteReplicationActor>.Instance, probe.Ref, () => false)));
actor.Tell(new SfBufferSnapshot(new List<StoreAndForwardMessage> { NewSfMessage("fresh") }, false));
await AwaitAssertAsync(async () =>
{
Assert.Null(await _sfStorage.GetMessageByIdAsync("stale"));
Assert.NotNull(await _sfStorage.GetMessageByIdAsync("fresh"));
}, TimeSpan.FromSeconds(5));
}
// ── R2 T5: chunked resync answer ──
[Fact]
public void ChunkForRemoting_SplitsByByteBudget_PreservingOrderAndSequence()
{
var rows = Enumerable.Range(0, 10)
.Select(i => NewMessage($"m{i}", payloadJson: new string('x', 20_000)))
.ToList();
var chunks = SiteReplicationActor.ChunkForRemoting(rows, maxChunkBytes: 64_000, maxChunkRows: 200);
Assert.True(chunks.Count > 1); // 10 × 20 KB cannot ride one 64 KB chunk
Assert.Equal(rows.Select(r => r.Id), chunks.SelectMany(c => c).Select(r => r.Id)); // order preserved
Assert.All(chunks, c => Assert.True(
c.Sum(r => r.PayloadJson.Length) <= 64_000 || c.Count == 1)); // budget honored (oversized row isolated)
}
[Fact]
public void ChunkForRemoting_RowCapHonored_AndSingleOversizedRowIsolated()
{
var many = Enumerable.Range(0, 500).Select(i => NewMessage($"s{i}", payloadJson: "{}")).ToList();
Assert.All(SiteReplicationActor.ChunkForRemoting(many, 64_000, 200), c => Assert.True(c.Count <= 200));
var oversized = new List<StoreAndForwardMessage>
{ NewMessage("big", payloadJson: new string('y', 100_000)), NewMessage("small", payloadJson: "{}") };
var chunks = SiteReplicationActor.ChunkForRemoting(oversized, 64_000, 200);
Assert.Equal(2, chunks.Count); // the oversized row rides alone
}
[Fact]
public async Task ActiveNode_AnswersResyncRequest_WithSequencedChunks_SharingOneResyncId()
{
for (var i = 0; i < 3; i++)
await _sfStorage.EnqueueAsync(NewMessage($"c{i}", payloadJson: new string('z', 30_000)));
var actor = CreateResyncActor(isActive: () => true);
actor.Tell(new RequestSfBufferResync(), TestActor);
var first = ExpectMsg<SfBufferSnapshotChunk>(TimeSpan.FromSeconds(5));
var rest = Enumerable.Range(1, first.TotalChunks - 1)
.Select(_ => ExpectMsg<SfBufferSnapshotChunk>(TimeSpan.FromSeconds(5)))
.Prepend(first)
.ToList();
Assert.True(first.TotalChunks > 1);
Assert.All(rest, c => Assert.Equal(first.ResyncId, c.ResyncId));
Assert.Equal(Enumerable.Range(1, first.TotalChunks), rest.Select(c => c.Sequence));
Assert.Equal(3, rest.Sum(c => c.Messages.Count));
}
// ── R2 T6: standby chunk assembly + atomic apply + ack ──
[Fact]
public async Task StandbyNode_AssemblesChunks_AppliesOnce_AndAcks()
{
await _sfStorage.EnqueueAsync(NewMessage("stale"));
var actor = CreateResyncActor(isActive: () => false);
var resyncId = "r1";
actor.Tell(new SfBufferSnapshotChunk(resyncId, 1, 2,
new List<StoreAndForwardMessage> { NewMessage("f1") }, false), TestActor);
actor.Tell(new SfBufferSnapshotChunk(resyncId, 2, 2,
new List<StoreAndForwardMessage> { NewMessage("f2") }, false), TestActor);
var ack = ExpectMsg<SfBufferResyncAck>(TimeSpan.FromSeconds(5));
Assert.Equal(resyncId, ack.ResyncId);
Assert.Equal(2, ack.RowCount);
await AwaitAssertAsync(async () =>
{
Assert.Null(await _sfStorage.GetMessageByIdAsync("stale")); // replaced wholesale
Assert.NotNull(await _sfStorage.GetMessageByIdAsync("f1"));
Assert.NotNull(await _sfStorage.GetMessageByIdAsync("f2"));
});
}
[Fact]
public async Task StandbyNode_NewResyncId_DiscardsStalePartialAssembly()
{
var actor = CreateResyncActor(isActive: () => false);
actor.Tell(new SfBufferSnapshotChunk("old", 1, 2,
new List<StoreAndForwardMessage> { NewMessage("orphan") }, false), TestActor);
actor.Tell(new SfBufferSnapshotChunk("new", 1, 1,
new List<StoreAndForwardMessage> { NewMessage("fresh") }, false), TestActor);
ExpectMsg<SfBufferResyncAck>(TimeSpan.FromSeconds(5)); // "new" completed
await AwaitAssertAsync(async () =>
{
Assert.NotNull(await _sfStorage.GetMessageByIdAsync("fresh"));
Assert.Null(await _sfStorage.GetMessageByIdAsync("orphan")); // stale partial never applied
});
}
[Fact]
public void ActiveNode_IgnoresChunks_NeverAcks()
{
var actor = CreateResyncActor(isActive: () => true);
actor.Tell(new SfBufferSnapshotChunk("r", 1, 1,
new List<StoreAndForwardMessage> { NewMessage("x") }, false), TestActor);
ExpectNoMsg(TimeSpan.FromMilliseconds(300));
}
// ── R2 T7: active-side resync ack confirmation + telemetry ──
//
// NOTE (deviation from plan): the actor logs via Microsoft ILogger (NullLogger in
// tests), NOT Akka's EventStream, so the plan's EventFilter.Warning assertions can
// never observe these warnings. We observe the two OTel counters via a MeterListener
// instead — the equivalent, and stronger, observable signal.
[Fact]
public async Task ActiveNode_ReceivingAck_CountsResyncCompleted()
{
long completed = 0;
using var listener = ListenCounter("scadabridge.store_and_forward.resync.completed",
m => Interlocked.Add(ref completed, m));
await _sfStorage.EnqueueAsync(NewMessage("m1"));
var actor = CreateResyncActor(isActive: () => true, ackTimeout: TimeSpan.FromSeconds(30));
actor.Tell(new RequestSfBufferResync(), TestActor);
var chunk = ExpectMsg<SfBufferSnapshotChunk>(TimeSpan.FromSeconds(5));
actor.Tell(new SfBufferResyncAck(chunk.ResyncId, 1), TestActor);
await AwaitAssertAsync(() =>
{
Assert.True(Interlocked.Read(ref completed) >= 1); // ack recorded the completion
return Task.CompletedTask;
}, TimeSpan.FromSeconds(5));
}
[Fact]
public async Task ActiveNode_MissingAck_WarnsAfterAckTimeout()
{
long ackMissing = 0;
using var listener = ListenCounter("scadabridge.store_and_forward.resync.ack_missing",
m => Interlocked.Add(ref ackMissing, m));
await _sfStorage.EnqueueAsync(NewMessage("m1"));
var actor = CreateResyncActor(isActive: () => true, ackTimeout: TimeSpan.FromMilliseconds(200));
actor.Tell(new RequestSfBufferResync(), TestActor);
ExpectMsg<SfBufferSnapshotChunk>(TimeSpan.FromSeconds(5));
// No ack is sent → the ack window expires and the resync is counted unacknowledged.
await AwaitAssertAsync(() =>
{
Assert.True(Interlocked.Read(ref ackMissing) >= 1);
return Task.CompletedTask;
}, TimeSpan.FromSeconds(5));
}
/// <summary>Attaches a <see cref="MeterListener"/> to a single ScadaBridge counter by name,
/// forwarding each recorded increment to <paramref name="onMeasurement"/>.</summary>
private static MeterListener ListenCounter(string instrumentName, Action<long> onMeasurement)
{
var listener = new MeterListener();
listener.InstrumentPublished = (inst, l) =>
{
if (inst.Meter.Name == ScadaBridgeTelemetry.MeterName && inst.Name == instrumentName)
l.EnableMeasurementEvents(inst);
};
listener.SetMeasurementEventCallback<long>((_, m, _, _) => onMeasurement(m));
listener.Start();
return listener;
}
private static StoreAndForwardMessage NewSfMessage(string id) => new()
{
Id = id,
Category = StoreAndForwardCategory.ExternalSystem,
Target = "t",
PayloadJson = "{}",
RetryCount = 0,
MaxRetries = 50,
RetryIntervalMs = 30000,
CreatedAt = DateTimeOffset.UtcNow,
Status = StoreAndForwardMessageStatus.Pending,
};
/// <summary>
/// Builds a resync-test message with a settable payload (additive to
/// <see cref="NewSfMessage"/> — the chunker sizes on <c>PayloadJson</c> length).
/// </summary>
private static StoreAndForwardMessage NewMessage(string id, string payloadJson = "{}") => new()
{
Id = id,
Category = StoreAndForwardCategory.ExternalSystem,
Target = "t",
PayloadJson = payloadJson,
RetryCount = 0,
MaxRetries = 50,
RetryIntervalMs = 30000,
CreatedAt = DateTimeOffset.UtcNow,
Status = StoreAndForwardMessageStatus.Pending,
};
/// <summary>Constructs a <see cref="ResyncTestActor"/> with the given active-node check
/// (the resync chunk/ack tests Tell to and expect from <see cref="TestKit.TestActor"/>).
/// <paramref name="ackTimeout"/> is the active-side ack window seam (T7).</summary>
private IActorRef CreateResyncActor(Func<bool> isActive, TimeSpan? ackTimeout = null) =>
ActorOf(Props.Create(() => new ResyncTestActor(
_storage, _sfStorage, _replicationService, SiteRole,
NullLogger<SiteReplicationActor>.Instance, CreateTestProbe().Ref, isActive, ackTimeout)));
/// <summary>Test message: drives <see cref="SiteReplicationActor.OnPeerTracked"/> directly,
/// standing in for the MemberUp→TryTrackPeer path (a single-node TestKit cannot form a real peer).</summary>
private sealed record TriggerPeerTracked;
/// <summary>
/// Test subclass for the resync tests: captures peer sends to a probe, injects the
/// active-node check, and exposes <see cref="OnPeerTracked"/> via a test message.
/// </summary>
private sealed class ResyncTestActor : SiteReplicationActor
{
private readonly IActorRef _peerProbe;
public ResyncTestActor(
SiteStorageService storage, StoreAndForwardStorage sfStorage,
ReplicationService replicationService, string siteRole,
ILogger<SiteReplicationActor> logger, IActorRef peerProbe, Func<bool> isActive,
TimeSpan? ackTimeout = null)
: base(storage, sfStorage, replicationService, siteRole, logger,
configFetcher: null, isActiveOverride: isActive, resyncAckTimeout: ackTimeout)
{
_peerProbe = peerProbe;
Receive<TriggerPeerTracked>(_ => OnPeerTracked());
}
protected override void SendToPeer(object message) => _peerProbe.Tell(message, Self);
}
/// <summary>
/// Test subclass exposing the peer send: <see cref="SiteReplicationActor.SendToPeer"/> is
/// overridden to forward to a probe so the outbound mapping can be asserted without a real
/// two-node cluster (a single-node TestKit has no peer address, so the real send is dropped).
/// </summary>
private sealed class ProbeForwardingReplicationActor : SiteReplicationActor
{
private readonly IActorRef _peerProbe;
public ProbeForwardingReplicationActor(
SiteStorageService storage, StoreAndForwardStorage sfStorage,
ReplicationService replicationService, string siteRole,
ILogger<SiteReplicationActor> logger, IDeploymentConfigFetcher configFetcher,
IActorRef peerProbe)
: base(storage, sfStorage, replicationService, siteRole, logger, configFetcher)
=> _peerProbe = peerProbe;
protected override void SendToPeer(object message) => _peerProbe.Tell(message, Self);
}
/// <summary>
/// In-test fake <see cref="IDeploymentConfigFetcher"/>: runs a per-deploymentId behavior
/// (return config JSON or throw, as a Task — mirroring the real async HTTP fetcher) and
/// records every call's coords thread-safely (the continuation runs on a pool thread).
/// </summary>
private sealed class FakeConfigFetcher : IDeploymentConfigFetcher
{
private readonly Func<string, Task<string>> _behavior;
public ConcurrentQueue<(string BaseUrl, string DeploymentId, string Token)> Calls { get; } = new();
public FakeConfigFetcher(Func<string, Task<string>> behavior) => _behavior = behavior;
public async Task<string> FetchAsync(
string centralFetchBaseUrl, string deploymentId, string token, CancellationToken ct)
{
Calls.Enqueue((centralFetchBaseUrl, deploymentId, token));
await Task.Yield();
return await _behavior(deploymentId);
}
}
}
tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/ResyncWireSerializationPinTests.cs
-95
View File
@@ -1,95 +0,0 @@
using Akka.TestKit.Xunit2;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors;
using ZB.MOM.WW.ScadaBridge.StoreAndForward;
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests;
/// <summary>
/// Characterization pin for the chunked anti-entropy resync contract (review 02 round 2,
/// N2). <see cref="SfBufferSnapshotChunk"/> (active→standby) and <see cref="SfBufferResyncAck"/>
/// (standby→active) ride intra-site Akka remoting on the default reflective-JSON wire
/// format. A rename/move, dropped setter, or non-default-constructible message would
/// silently break resync across a rolling upgrade and only surface as a divergent buffer
/// after a failover. These pin round-trip fidelity and type identity. (These messages are
/// NOT ClusterClient traffic, so they are intentionally absent from ClusterClientContractLockTests.)
/// </summary>
public class ResyncWireSerializationPinTests : TestKit
{
private T RoundTrip<T>(T message)
{
var serialization = Sys.Serialization;
var serializer = serialization.FindSerializerFor(message);
var bytes = serializer.ToBinary(message);
return (T)serialization.Deserialize(bytes, serializer.Identifier, message!.GetType());
}
private static StoreAndForwardMessage FullMessage() => new()
{
Id = Guid.NewGuid().ToString("N"),
Category = StoreAndForwardCategory.Notification,
Target = "Operators",
PayloadJson = "{\"notificationId\":\"abc\"}",
RetryCount = 4,
MaxRetries = 0,
RetryIntervalMs = 30000,
CreatedAt = DateTimeOffset.UtcNow,
LastAttemptAt = DateTimeOffset.UtcNow,
Status = StoreAndForwardMessageStatus.Parked,
LastError = "central rejected",
OriginInstanceName = "Plant.Pump3",
ExecutionId = Guid.NewGuid(),
SourceScript = "ScriptActor:MonitorSpeed",
ParentExecutionId = Guid.NewGuid(),
};
[Fact]
public void SfBufferSnapshotChunk_WithFullMessage_RoundTripsOnTheWire()
{
var message = FullMessage();
var original = new SfBufferSnapshotChunk(
"resync-1", 2, 5, new List<StoreAndForwardMessage> { message }, Truncated: true);
var back = RoundTrip(original);
Assert.Equal(original.ResyncId, back.ResyncId);
Assert.Equal(original.Sequence, back.Sequence);
Assert.Equal(original.TotalChunks, back.TotalChunks);
Assert.Equal(original.Truncated, back.Truncated);
var m = Assert.Single(back.Messages);
Assert.Equal(message.Id, m.Id);
Assert.Equal(message.Category, m.Category);
Assert.Equal(message.Target, m.Target);
Assert.Equal(message.PayloadJson, m.PayloadJson);
Assert.Equal(message.RetryCount, m.RetryCount);
Assert.Equal(message.MaxRetries, m.MaxRetries);
Assert.Equal(message.RetryIntervalMs, m.RetryIntervalMs);
Assert.Equal(message.CreatedAt, m.CreatedAt);
Assert.Equal(message.LastAttemptAt, m.LastAttemptAt);
Assert.Equal(message.Status, m.Status);
Assert.Equal(message.LastError, m.LastError);
Assert.Equal(message.OriginInstanceName, m.OriginInstanceName);
Assert.Equal(message.ExecutionId, m.ExecutionId);
Assert.Equal(message.SourceScript, m.SourceScript);
Assert.Equal(message.ParentExecutionId, m.ParentExecutionId);
}
[Fact]
public void SfBufferResyncAck_RoundTripsOnTheWire()
{
var original = new SfBufferResyncAck("resync-1", 42);
var back = RoundTrip(original);
Assert.Equal(original.ResyncId, back.ResyncId);
Assert.Equal(original.RowCount, back.RowCount);
}
// Type-identity pins: the reflective-JSON wire embeds CLR type manifests, so a
// rename/move of either type silently breaks resync across a rolling upgrade.
[Theory]
[InlineData(typeof(SfBufferSnapshotChunk), "ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors.SfBufferSnapshotChunk")]
[InlineData(typeof(SfBufferResyncAck), "ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors.SfBufferResyncAck")]
public void ResyncContract_TypeIdentity_IsPinned(Type type, string expectedFullName) =>
Assert.Equal(expectedFullName, type.FullName);
}
tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/CachedCallAttemptEmissionTests.cs
-2
View File
@@ -45,7 +45,6 @@ public class CachedCallAttemptEmissionTests : IAsyncLifetime, IDisposable
_storage,
_options,
NullLogger<StoreAndForwardService>.Instance,
replication: null,
cachedCallObserver: _observer,
siteId: "site-77");
}
@@ -490,7 +489,6 @@ public class CachedCallAttemptEmissionTests : IAsyncLifetime, IDisposable
RetryTimerInterval = TimeSpan.FromHours(1), // timer never fires in-test
},
NullLogger<StoreAndForwardService>.Instance,
replication: null,
cachedCallObserver: observer,
siteId: "site-77");
tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ReplicationServiceTests.cs
-236
View File
@@ -1,236 +0,0 @@
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Logging.Abstractions;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
using ZB.MOM.WW.ScadaBridge.TestSupport;
namespace ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests;
/// <summary>
/// WP-11: Tests for async replication to standby.
/// </summary>
public class ReplicationServiceTests : IAsyncLifetime, IDisposable
{
private readonly TestLocalDb _localDb;
private readonly StoreAndForwardStorage _storage;
private readonly ReplicationService _replicationService;
public ReplicationServiceTests()
{
_localDb = TestLocalDb.CreateTemp("RepTests");
_storage = new StoreAndForwardStorage(_localDb.Db, NullLogger<StoreAndForwardStorage>.Instance);
var options = new StoreAndForwardOptions { ReplicationEnabled = true };
_replicationService = new ReplicationService(
options, NullLogger<ReplicationService>.Instance);
}
public async Task InitializeAsync() => await _storage.InitializeAsync();
public Task DisposeAsync() => Task.CompletedTask;
public void Dispose()
{
var path = _localDb.Path;
_localDb.Dispose();
TestLocalDb.DeleteFiles(path);
}
[Fact]
public void ReplicateEnqueue_NoHandler_DoesNotThrow()
{
var msg = CreateMessage("rep1");
_replicationService.ReplicateEnqueue(msg);
}
[Fact]
public async Task ReplicateEnqueue_WithHandler_ForwardsOperation()
{
ReplicationOperation? captured = null;
_replicationService.SetReplicationHandler(op =>
{
captured = op;
return Task.CompletedTask;
});
var msg = CreateMessage("rep2");
_replicationService.ReplicateEnqueue(msg);
await Task.Delay(200);
Assert.NotNull(captured);
Assert.Equal(ReplicationOperationType.Add, captured!.OperationType);
Assert.Equal("rep2", captured.MessageId);
}
[Fact]
public async Task ReplicateRemove_WithHandler_ForwardsRemoveOperation()
{
ReplicationOperation? captured = null;
_replicationService.SetReplicationHandler(op =>
{
captured = op;
return Task.CompletedTask;
});
_replicationService.ReplicateRemove("rep3");
await Task.Delay(200);
Assert.NotNull(captured);
Assert.Equal(ReplicationOperationType.Remove, captured!.OperationType);
Assert.Equal("rep3", captured.MessageId);
}
[Fact]
public async Task ReplicatePark_WithHandler_ForwardsParkOperation()
{
ReplicationOperation? captured = null;
_replicationService.SetReplicationHandler(op =>
{
captured = op;
return Task.CompletedTask;
});
var msg = CreateMessage("rep4");
_replicationService.ReplicatePark(msg);
await Task.Delay(200);
Assert.NotNull(captured);
Assert.Equal(ReplicationOperationType.Park, captured!.OperationType);
}
[Fact]
public async Task ApplyReplicatedOperationAsync_Add_EnqueuesMessage()
{
var msg = CreateMessage("apply1");
var operation = new ReplicationOperation(ReplicationOperationType.Add, "apply1", msg);
await _replicationService.ApplyReplicatedOperationAsync(operation, _storage);
var retrieved = await _storage.GetMessageByIdAsync("apply1");
Assert.NotNull(retrieved);
}
[Fact]
public async Task ApplyReplicatedOperationAsync_Remove_DeletesMessage()
{
var msg = CreateMessage("apply2");
await _storage.EnqueueAsync(msg);
var operation = new ReplicationOperation(ReplicationOperationType.Remove, "apply2", null);
await _replicationService.ApplyReplicatedOperationAsync(operation, _storage);
var retrieved = await _storage.GetMessageByIdAsync("apply2");
Assert.Null(retrieved);
}
[Fact]
public async Task ApplyReplicatedOperationAsync_Park_UpdatesStatus()
{
var msg = CreateMessage("apply3");
await _storage.EnqueueAsync(msg);
var operation = new ReplicationOperation(ReplicationOperationType.Park, "apply3", msg);
await _replicationService.ApplyReplicatedOperationAsync(operation, _storage);
var retrieved = await _storage.GetMessageByIdAsync("apply3");
Assert.NotNull(retrieved);
Assert.Equal(StoreAndForwardMessageStatus.Parked, retrieved!.Status);
}
[Fact]
public void ReplicateEnqueue_WhenReplicationDisabled_DoesNothing()
{
var options = new StoreAndForwardOptions { ReplicationEnabled = false };
var service = new ReplicationService(options, NullLogger<ReplicationService>.Instance);
bool handlerCalled = false;
service.SetReplicationHandler(_ => { handlerCalled = true; return Task.CompletedTask; });
service.ReplicateEnqueue(CreateMessage("disabled1"));
Assert.False(handlerCalled);
}
[Fact]
public async Task ReplicateEnqueue_HandlerThrows_DoesNotPropagateException()
{
_replicationService.SetReplicationHandler(_ =>
throw new InvalidOperationException("standby down"));
_replicationService.ReplicateEnqueue(CreateMessage("err1"));
await Task.Delay(200);
// No exception -- fire-and-forget, best-effort
}
// ── Task 10 (arch review 02): ordered, observable replication dispatch ──
[Fact]
public void ReplicationOperations_AreDispatchedInIssueOrder()
{
var seen = new List<(ReplicationOperationType, string)>();
_replicationService.SetReplicationHandler(op =>
{
seen.Add((op.OperationType, op.MessageId));
return Task.CompletedTask;
});
for (var i = 0; i < 200; i++)
{
_replicationService.ReplicateEnqueue(CreateMessage($"m{i}"));
_replicationService.ReplicateRemove($"m{i}");
}
// Inline dispatch: by the time the calls return, every op was handed to the
// handler, Add strictly before Remove per id. Pre-fix (Task.Run) this was
// racy in both count and order.
Assert.Equal(400, seen.Count);
for (var i = 0; i < 200; i++)
{
Assert.Equal((ReplicationOperationType.Add, $"m{i}"), seen[2 * i]);
Assert.Equal((ReplicationOperationType.Remove, $"m{i}"), seen[2 * i + 1]);
}
}
[Fact]
public void ReplicationHandlerThrow_IsSwallowed_AndLoggedAtWarning()
{
var logger = new CapturingLogger<ReplicationService>();
var service = new ReplicationService(new StoreAndForwardOptions(), logger);
service.SetReplicationHandler(_ => throw new InvalidOperationException("peer gone"));
service.ReplicateRemove("m1"); // must not throw
Assert.Contains(logger.Entries, e => e.Level == LogLevel.Warning);
}
private static StoreAndForwardMessage CreateMessage(string id)
{
return new StoreAndForwardMessage
{
Id = id,
Category = StoreAndForwardCategory.ExternalSystem,
Target = "target",
PayloadJson = "{}",
RetryCount = 0,
MaxRetries = 50,
RetryIntervalMs = 30000,
CreatedAt = DateTimeOffset.UtcNow,
Status = StoreAndForwardMessageStatus.Pending
};
}
/// <summary>Minimal in-memory logger that records level + rendered message.</summary>
private sealed class CapturingLogger<T> : ILogger<T>
{
public List<(LogLevel Level, string Message)> Entries { get; } = new();
public IDisposable? BeginScope<TState>(TState state) where TState : notnull => null;
public bool IsEnabled(LogLevel logLevel) => true;
public void Log<TState>(LogLevel logLevel, EventId eventId, TState state, Exception? exception,
Func<TState, Exception?, string> formatter)
=> Entries.Add((logLevel, formatter(state, exception)));
}
}
tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ReplicationWireSerializationPinTests.cs
-82
View File
@@ -1,82 +0,0 @@
using Akka.TestKit.Xunit2;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
namespace ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests;
/// <summary>
/// Characterization pin for the intra-cluster S&amp;F replication contract. A
/// <see cref="ReplicationOperation"/> (carrying a full <see cref="StoreAndForwardMessage"/>)
/// is Told from the active node to the standby node's replication actor and rides
/// the Akka default reflective-JSON wire format. Nothing tested that it round-trips
/// or that its type identity is stable — a rename/move, a dropped setter, or a
/// non-default-constructible message would silently break standby buffer sync and
/// only surface as a divergent buffer after a failover.
///
/// The serializer swap (proto / explicit bindings) is deferred; until then this pin
/// is the guardrail.
/// </summary>
public class ReplicationWireSerializationPinTests : TestKit
{
private T RoundTrip<T>(T message)
{
var serialization = Sys.Serialization;
var serializer = serialization.FindSerializerFor(message);
var bytes = serializer.ToBinary(message);
return (T)serialization.Deserialize(bytes, serializer.Identifier, message!.GetType());
}
[Fact]
public void ReplicationOperation_WithFullMessage_RoundTripsOnTheWire()
{
var message = new StoreAndForwardMessage
{
Id = Guid.NewGuid().ToString("N"),
Category = StoreAndForwardCategory.Notification,
Target = "Operators",
PayloadJson = "{\"notificationId\":\"abc\"}",
RetryCount = 4,
MaxRetries = 0,
RetryIntervalMs = 30000,
CreatedAt = DateTimeOffset.UtcNow,
LastAttemptAt = DateTimeOffset.UtcNow,
Status = StoreAndForwardMessageStatus.Parked,
LastError = "central rejected",
OriginInstanceName = "Plant.Pump3",
ExecutionId = Guid.NewGuid(),
SourceScript = "ScriptActor:MonitorSpeed",
ParentExecutionId = Guid.NewGuid(),
};
var original = new ReplicationOperation(ReplicationOperationType.Park, message.Id, message);
var back = RoundTrip(original);
Assert.Equal(original.OperationType, back.OperationType);
Assert.Equal(original.MessageId, back.MessageId);
Assert.NotNull(back.Message);
var m = back.Message!;
Assert.Equal(message.Id, m.Id);
Assert.Equal(message.Category, m.Category);
Assert.Equal(message.Target, m.Target);
Assert.Equal(message.PayloadJson, m.PayloadJson);
Assert.Equal(message.RetryCount, m.RetryCount);
Assert.Equal(message.MaxRetries, m.MaxRetries);
Assert.Equal(message.RetryIntervalMs, m.RetryIntervalMs);
Assert.Equal(message.CreatedAt, m.CreatedAt);
Assert.Equal(message.LastAttemptAt, m.LastAttemptAt);
Assert.Equal(message.Status, m.Status);
Assert.Equal(message.LastError, m.LastError);
Assert.Equal(message.OriginInstanceName, m.OriginInstanceName);
Assert.Equal(message.ExecutionId, m.ExecutionId);
Assert.Equal(message.SourceScript, m.SourceScript);
Assert.Equal(message.ParentExecutionId, m.ParentExecutionId);
}
// Type-identity pins: the reflective-JSON wire embeds CLR type manifests, so a
// rename/move of either type silently breaks standby replication across a
// rolling upgrade. If one fails, you are making a wire-breaking change.
[Theory]
[InlineData(typeof(ReplicationOperation), "ZB.MOM.WW.ScadaBridge.StoreAndForward.ReplicationOperation")]
[InlineData(typeof(StoreAndForwardMessage), "ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardMessage")]
public void ReplicationContract_TypeIdentity_IsPinned(Type type, string expectedFullName) =>
Assert.Equal(expectedFullName, type.FullName);
}
tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardReplicationTests.cs
-266
View File
@@ -1,266 +0,0 @@
using Microsoft.Extensions.Logging.Abstractions;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
using ZB.MOM.WW.ScadaBridge.TestSupport;
namespace ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests;
/// <summary>
/// StoreAndForward-001: the active node must forward every buffer operation
/// (add / remove / park) to the standby via the ReplicationService, so a
/// failover does not lose the buffer.
/// </summary>
public class StoreAndForwardReplicationTests : IAsyncLifetime, IDisposable
{
private readonly TestLocalDb _localDb;
private readonly StoreAndForwardStorage _storage;
private readonly StoreAndForwardService _service;
private readonly List<ReplicationOperation> _replicated = new();
public StoreAndForwardReplicationTests()
{
_localDb = TestLocalDb.CreateTemp("ReplTests");
_storage = new StoreAndForwardStorage(_localDb.Db, NullLogger<StoreAndForwardStorage>.Instance);
var options = new StoreAndForwardOptions
{
DefaultRetryInterval = TimeSpan.Zero,
DefaultMaxRetries = 1,
RetryTimerInterval = TimeSpan.FromMinutes(10),
ReplicationEnabled = true,
};
var replication = new ReplicationService(options, NullLogger<ReplicationService>.Instance);
replication.SetReplicationHandler(op =>
{
lock (_replicated) _replicated.Add(op);
return Task.CompletedTask;
});
_service = new StoreAndForwardService(
_storage, options, NullLogger<StoreAndForwardService>.Instance, replication);
}
public async Task InitializeAsync() => await _storage.InitializeAsync();
public Task DisposeAsync() => Task.CompletedTask;
public void Dispose()
{
var path = _localDb.Path;
_localDb.Dispose();
TestLocalDb.DeleteFiles(path);
}
/// <summary>Replication is fire-and-forget (Task.Run); poll until the expected ops arrive.</summary>
private async Task<List<ReplicationOperation>> WaitForReplicationAsync(int count)
{
for (var i = 0; i < 100; i++)
{
lock (_replicated)
if (_replicated.Count >= count) return _replicated.ToList();
await Task.Delay(20);
}
lock (_replicated) return _replicated.ToList();
}
[Fact]
public async Task BufferingAMessage_ReplicatesAnAddOperation()
{
// No handler registered → message is buffered → an Add is replicated.
var result = await _service.EnqueueAsync(
StoreAndForwardCategory.ExternalSystem, "api", """{}""");
Assert.True(result.WasBuffered);
var ops = await WaitForReplicationAsync(1);
Assert.Contains(ops, o =>
o.OperationType == ReplicationOperationType.Add && o.MessageId == result.MessageId);
}
[Fact]
public async Task SuccessfulRetry_ReplicatesARemoveOperation()
{
var calls = 0;
_service.RegisterDeliveryHandler(StoreAndForwardCategory.ExternalSystem,
_ => ++calls == 1
? throw new HttpRequestException("transient")
: Task.FromResult(true));
var result = await _service.EnqueueAsync(
StoreAndForwardCategory.ExternalSystem, "api", """{}""");
await _service.RetryPendingMessagesAsync();
var ops = await WaitForReplicationAsync(2);
Assert.Contains(ops, o => o.OperationType == ReplicationOperationType.Add);
Assert.Contains(ops, o =>
o.OperationType == ReplicationOperationType.Remove && o.MessageId == result.MessageId);
}
[Fact]
public async Task ParkedMessage_ReplicatesAParkOperation()
{
_service.RegisterDeliveryHandler(StoreAndForwardCategory.ExternalSystem,
_ => throw new HttpRequestException("always fails"));
var result = await _service.EnqueueAsync(
StoreAndForwardCategory.ExternalSystem, "api", """{}""", maxRetries: 1);
await _service.RetryPendingMessagesAsync();
var ops = await WaitForReplicationAsync(2);
Assert.Contains(ops, o =>
o.OperationType == ReplicationOperationType.Park && o.MessageId == result.MessageId);
}
/// <summary>
/// StoreAndForward-016: an operator discarding a parked message must replicate
/// a Remove so the standby's copy is also deleted (otherwise the discarded
/// message reappears in the parked list after a failover).
/// </summary>
[Fact]
public async Task DiscardingAParkedMessage_ReplicatesARemoveOperation()
{
_service.RegisterDeliveryHandler(StoreAndForwardCategory.ExternalSystem,
_ => throw new HttpRequestException("always fails"));
var result = await _service.EnqueueAsync(
StoreAndForwardCategory.ExternalSystem, "api", """{}""", maxRetries: 1);
await _service.RetryPendingMessagesAsync(); // -> parked
await WaitForReplicationAsync(2);
var discarded = await _service.DiscardParkedMessageAsync(result.MessageId);
Assert.True(discarded);
var ops = await WaitForReplicationAsync(3);
Assert.Contains(ops, o =>
o.OperationType == ReplicationOperationType.Remove && o.MessageId == result.MessageId);
}
/// <summary>
/// StoreAndForward-016: an operator retrying a parked message must replicate a
/// Requeue so the standby's copy moves back to Pending (otherwise it stays
/// Parked on the standby and the operator's retry is lost across a failover).
/// </summary>
[Fact]
public async Task RetryingAParkedMessage_ReplicatesARequeueOperation()
{
_service.RegisterDeliveryHandler(StoreAndForwardCategory.ExternalSystem,
_ => throw new HttpRequestException("always fails"));
var result = await _service.EnqueueAsync(
StoreAndForwardCategory.ExternalSystem, "api", """{}""", maxRetries: 1);
await _service.RetryPendingMessagesAsync(); // -> parked
await WaitForReplicationAsync(2);
var retried = await _service.RetryParkedMessageAsync(result.MessageId);
Assert.True(retried);
var ops = await WaitForReplicationAsync(3);
var requeue = ops.SingleOrDefault(o =>
o.OperationType == ReplicationOperationType.Requeue && o.MessageId == result.MessageId);
Assert.NotNull(requeue);
Assert.NotNull(requeue!.Message);
Assert.Equal(StoreAndForwardMessageStatus.Pending, requeue.Message!.Status);
}
/// <summary>
/// StoreAndForward-016: the standby applies a Requeue by moving its row back to
/// Pending with retry_count = 0, mirroring the active node's local state.
/// </summary>
[Fact]
public async Task ApplyReplicatedOperation_Requeue_MovesStandbyRowBackToPending()
{
var replication = new ReplicationService(
new StoreAndForwardOptions { ReplicationEnabled = true },
NullLogger<ReplicationService>.Instance);
var parked = new StoreAndForwardMessage
{
Id = "requeue1",
Category = StoreAndForwardCategory.ExternalSystem,
Target = "api",
PayloadJson = "{}",
RetryCount = 5,
MaxRetries = 1,
RetryIntervalMs = 0,
CreatedAt = DateTimeOffset.UtcNow,
Status = StoreAndForwardMessageStatus.Parked,
};
await _storage.EnqueueAsync(parked);
var requeued = new StoreAndForwardMessage
{
Id = parked.Id,
Category = parked.Category,
Target = parked.Target,
PayloadJson = parked.PayloadJson,
RetryCount = 0,
MaxRetries = parked.MaxRetries,
RetryIntervalMs = parked.RetryIntervalMs,
CreatedAt = parked.CreatedAt,
Status = StoreAndForwardMessageStatus.Pending,
};
await replication.ApplyReplicatedOperationAsync(
new ReplicationOperation(ReplicationOperationType.Requeue, parked.Id, requeued),
_storage);
var row = await _storage.GetMessageByIdAsync(parked.Id);
Assert.NotNull(row);
Assert.Equal(StoreAndForwardMessageStatus.Pending, row!.Status);
Assert.Equal(0, row.RetryCount);
}
private static ReplicationService NewReplicationService() =>
new(new StoreAndForwardOptions { ReplicationEnabled = true },
NullLogger<ReplicationService>.Instance);
private static StoreAndForwardMessage NewMessage(string id) => new()
{
Id = id,
Category = StoreAndForwardCategory.ExternalSystem,
Target = "api",
PayloadJson = "{}",
RetryCount = 0,
MaxRetries = 1,
RetryIntervalMs = 0,
CreatedAt = DateTimeOffset.UtcNow,
Status = StoreAndForwardMessageStatus.Pending,
};
/// <summary>
/// A Park (or Requeue) whose original Add was lost — e.g. the Add's
/// fire-and-forget replication dropped — must still materialise the row on the
/// standby. The full message rides in the operation, so an upsert self-heals;
/// a blind UPDATE would affect 0 rows and the row would be gone forever.
/// </summary>
[Fact]
public async Task ApplyReplicatedPark_WhenAddWasLost_MaterializesTheParkedRow()
{
var service = NewReplicationService();
var msg = NewMessage("lost-add-1"); // never Added on this (standby) storage
msg.Status = StoreAndForwardMessageStatus.Parked;
await service.ApplyReplicatedOperationAsync(
new ReplicationOperation(ReplicationOperationType.Park, msg.Id, msg), _storage);
var row = await _storage.GetMessageByIdAsync("lost-add-1");
Assert.NotNull(row); // pre-fix: blind UPDATE affected 0 rows, row is gone forever
Assert.Equal(StoreAndForwardMessageStatus.Parked, row!.Status);
}
/// <summary>
/// A duplicate Add (e.g. after Task 21's anti-entropy resync re-issues an Add
/// the standby already holds) must not violate the PK — the upsert applies
/// newest-wins instead of throwing.
/// </summary>
[Fact]
public async Task ApplyReplicatedAdd_Twice_IsIdempotent_NewestWins()
{
var service = NewReplicationService();
var msg = NewMessage("dup-add");
await service.ApplyReplicatedOperationAsync(
new ReplicationOperation(ReplicationOperationType.Add, msg.Id, msg), _storage);
msg.RetryCount = 3;
await service.ApplyReplicatedOperationAsync( // pre-fix: SqliteException PK violation
new ReplicationOperation(ReplicationOperationType.Add, msg.Id, msg), _storage);
Assert.Equal(3, (await _storage.GetMessageByIdAsync("dup-add"))!.RetryCount);
}
}
tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardSiteEventTests.cs
+1 -1
View File
@@ -54,7 +54,7 @@ public class StoreAndForwardSiteEventTests : IAsyncLifetime, IDisposable
_service = new StoreAndForwardService(
_storage, _options, NullLogger<StoreAndForwardService>.Instance,
replication: null, cachedCallObserver: null, siteId: "site-a",
cachedCallObserver: null, siteId: "site-a",
siteEventLogger: _siteLog);
}
tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardStorageTests.cs
+9 -10
View File
@@ -769,16 +769,15 @@ public class StoreAndForwardStorageTests : IAsyncLifetime, IDisposable
Assert.True(truncated); // a third row exists beyond the limit
}
[Fact]
public async Task ReplaceAll_SwapsTheEntireBuffer_Atomically()
{
await _storage.EnqueueAsync(NewMsg("stale"));
await _storage.ReplaceAllAsync(new[] { NewMsg("fresh-1"), NewMsg("fresh-2") });
Assert.Null(await _storage.GetMessageByIdAsync("stale"));
Assert.NotNull(await _storage.GetMessageByIdAsync("fresh-1"));
Assert.NotNull(await _storage.GetMessageByIdAsync("fresh-2"));
}
// ReplaceAll_SwapsTheEntireBuffer_Atomically was DELETED with ReplaceAllAsync in
// LocalDb Phase 2, and deliberately not replaced. It asserted a destructive
// delete-all-then-insert-all, which was the standby's anti-entropy apply: a resync
// overwrote the standby's copy with the active node's snapshot. sf_messages is now a
// replicated table, so a mass DELETE would be CAPTURED and shipped to the peer — the
// method is not merely unused, it is unsafe to keep. LocalDb's own snapshot resync
// merges per row under last-writer-wins and never deletes, which is what
// LocalDbConfigConvergenceTests.ANodeWithNewerLocalRows_KeepsThem_WhenAPeerSnapshotArrives
// now covers.
// ── Task 23: oldest-parked-age health signal ──