dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 2 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

feat(localdb)!: replicate site config + sf_messages via CDC, delete the bespoke replicators

Browse Source 
Tasks 14, 15 and 16, landed as ONE commit.

PLAN DEFECT: these three tasks cannot compile separately. SiteReplicationActor
takes a ReplicationService and calls ReplaceAllAsync (Task 14 deletes both);
DeploymentManagerActor Tells message types declared in ReplicationMessages.cs
(Task 15 deletes it); AkkaHostedService constructs the actor (Task 16). Any
ordering leaves a broken intermediate. Combining them also strengthens the
invariant Task 14 already stated for itself — the two mechanisms never both
run, and never neither.

Registered 8 tables in SiteLocalDbSetup.OnReady: sf_messages plus the 7 site
config tables. notification_lists and smtp_configurations are deliberately NOT
registered — permanently empty by design, so registering them would open a
standing replication channel whose only historical payload was plaintext SMTP
passwords. Migrate stays the LAST call in OnReady, after all registrations, so
migrated rows enter the oplog through live capture triggers.

Deleted: SiteReplicationActor, ReplicationMessages.cs, ReplicationService,
StoreAndForwardStorage.ReplaceAllAsync, and 6 test files. ReplaceAllAsync is
not merely unused but unsafe to keep: a mass DELETE on a now-replicated table
would be captured and shipped to the peer.

Kept ActiveNodeEvaluator (delivery gate + heartbeat still need it) with its doc
corrected, and activeNodeCheck in AkkaHostedService (SiteCommunicationActor).

The positional-argument hazard the plan flagged was real: removing
DeploymentManagerActor's optional IActorRef? replicationActor shifted 6
trailing optionals, and 4 test call sites bound the wrong arguments with no
compile error at some positions. Converted them to named arguments where
possible — Props.Create builds an expression tree, which rejects out-of-position
named args, so the rest are padded positionally with a comment saying why.

The Task 7 'not yet registered' test was INVERTED rather than deleted, and is
exact in both directions: too few means a table silently stops replicating, too
many means the SMTP tables leak. Added a separate security-named test for those
two, and a composite-PK test (LWW keys on the full PK, so a truncated key set
would collapse distinct rows). The convergence suites now get their
registrations from the real OnReady — their temporary harness registration is
deleted, so they prove the cutover rather than agreeing with themselves.

Verified: build 0 warnings; SiteRuntime 512, StoreAndForward 130, Host 330,
AuditLog 355, ExternalSystemGateway 142, HealthMonitoring 97, LocalDb
integration 16 — all pass, 0 failures.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
This commit is contained in:
Joseph Doherty
2026-07-20 04:20:05 -04:00
parent df0c6031ba
commit 037798b367
26 changed files with 160 additions and 2513 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/SfBufferResyncPredicateTests.cs
-122
View File
@@ -1,122 +0,0 @@
using Akka.Actor;
using Microsoft.Extensions.Logging.Abstractions;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
using ZB.MOM.WW.ScadaBridge.StoreAndForward;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
using ZB.MOM.WW.ScadaBridge.TestSupport;
namespace ZB.MOM.WW.ScadaBridge.IntegrationTests.Cluster;
/// <summary>
/// N1 regression (review 02 round 2, Critical): the resync authority must use the same
/// oldest-Up predicate as the S&F delivery gate. Divergence scenario = the delivering node
/// is OLDEST but not LEADER (leader = lowest address), the exact state a rolling restart of
/// the lower-address node produces. Pre-fix the delivering node requests a resync from the
/// stale peer and ReplaceAllAsync wipes its live buffer.
/// </summary>
public class SfBufferResyncPredicateTests
{
[Fact]
public async Task OldestButNotLeaderNode_KeepsItsBuffer_AndSeedsTheJoiner()
{
// Two explicit ports, deliberately assigned so the FIRST-started (oldest,
// delivering) node has the HIGHER address → the second node is cluster leader.
var p1 = TwoNodeClusterFixture.GetFreeTcpPort();
var p2 = TwoNodeClusterFixture.GetFreeTcpPort();
var (portHigh, portLow) = p1 > p2 ? (p1, p2) : (p2, p1);
var fixture = await TwoNodeClusterFixture.StartAsync(
role: "site-int", portA: portHigh, portB: portLow);
// The S&F stores AND SiteStorageService take an ILocalDb (LocalDb has no in-memory
// mode), so each node gets its own temp-file local databases. They are disposed
// AFTER the cluster is shut down — the actors hold connections while the systems
// are alive — and only then are the files (plus their WAL sidecars) deleted.
var localDbs = new List<TestLocalDb>();
try
{
// Real S&F storage + replication actor per node, production default predicate
// (no isActiveOverride) — the exact wiring under test.
var (storageOldest, _, sfDbOldest, siteDbOldest) =
await CreateReplicationActorAsync(fixture.NodeA, "oldest");
localDbs.Add(sfDbOldest);
localDbs.Add(siteDbOldest);
var (storageJoiner, _, sfDbJoiner, siteDbJoiner) =
await CreateReplicationActorAsync(fixture.NodeB, "joiner");
localDbs.Add(sfDbJoiner);
localDbs.Add(siteDbJoiner);
// The delivering (oldest) node has a live buffered row the standby never saw.
await storageOldest.EnqueueAsync(NewMessage("live-row"));
// Trigger peer (re)tracking on both sides: each actor got InitialStateAsSnapshot
// in PreStart, but the enqueue raced it — re-deliver via a fresh MemberUp is not
// needed; OnPeerTracked already fired on join. The resync exchange is async:
// wait until the JOINER holds the row (proves the snapshot flowed oldest→joiner,
// the correct direction). Pre-fix this times out (the joiner, as leader, never
// requests) AND the oldest node's row is deleted by the stale wipe.
await AwaitAsync(async () => await storageJoiner.GetMessageByIdAsync("live-row") != null,
TimeSpan.FromSeconds(20),
"joiner never received the resync snapshot (resync ran in the wrong direction)");
// And the delivering node's buffer is untouched — the N1 wipe assertion.
Assert.NotNull(await storageOldest.GetMessageByIdAsync("live-row"));
}
finally
{
await fixture.DisposeAsync();
foreach (var localDb in localDbs)
{
var path = localDb.Path;
localDb.Dispose();
TestLocalDb.DeleteFiles(path);
}
}
}
private static async Task<(
StoreAndForwardStorage Storage, IActorRef Actor, TestLocalDb SfLocalDb, TestLocalDb SiteLocalDb)>
CreateReplicationActorAsync(ActorSystem node, string tag)
{
var sfLocalDb = TestLocalDb.CreateTemp($"sf-resync-{tag}");
var sfStorage = new StoreAndForwardStorage(sfLocalDb.Db,
NullLogger<StoreAndForwardStorage>.Instance);
await sfStorage.InitializeAsync();
var siteLocalDb = TestLocalDb.CreateTemp($"site-resync-{tag}");
var siteStorage = new SiteStorageService(siteLocalDb.Db,
NullLogger<SiteStorageService>.Instance);
var replicationService = new ReplicationService(
new StoreAndForwardOptions(), NullLogger<ReplicationService>.Instance);
// Name MUST be "site-replication" — SendToPeer targets /user/site-replication.
var actor = node.ActorOf(Props.Create(() => new SiteReplicationActor(
siteStorage, sfStorage, replicationService, "site-int",
NullLogger<SiteReplicationActor>.Instance, null, null, null, null)),
"site-replication");
return (sfStorage, actor, sfLocalDb, siteLocalDb);
}
private static StoreAndForwardMessage NewMessage(string id) => new()
{
Id = id,
Category = StoreAndForwardCategory.Notification,
Target = "central",
PayloadJson = "{}",
CreatedAt = DateTimeOffset.UtcNow,
Status = StoreAndForwardMessageStatus.Pending,
MaxRetries = 0,
};
private static async Task AwaitAsync(Func<Task<bool>> condition, TimeSpan timeout, string why)
{
var deadline = DateTime.UtcNow + timeout;
while (DateTime.UtcNow < deadline)
{
if (await condition()) return;
await Task.Delay(250);
}
throw new TimeoutException(why);
}
}
tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbSitePairHarness.cs
+4 -30
View File
@@ -129,40 +129,14 @@ public abstract class LocalDbSitePairHarness : IAsyncLifetime
.Build();
return new ServiceCollection()
.AddZbLocalDb(config, db =>
{
SiteLocalDbSetup.OnReady(db, config);
RegisterPhase2TablesUntilCutover(db);
})
.AddZbLocalDb(config, db => SiteLocalDbSetup.OnReady(db, config))
.BuildServiceProvider();
}
/// <summary>
/// Registers the Phase 2 tables for capture, which production <c>OnReady</c> does not do
/// until the Task 14 cutover.
/// </summary>
/// <remarks>
/// <b>Delete this method at Task 14</b>, along with its call above. Until the cutover the
/// bespoke <c>SiteReplicationActor</c> and <c>ReplicationService</c> still own these
/// tables, so <c>OnReady</c> deliberately leaves them unregistered — but the convergence
/// specifications the cutover has to satisfy need capture triggers to mean anything. The
/// tables are empty at this point, so registering here captures nothing retroactively;
/// the ordering guarantee under test is unaffected.
/// <para>
/// After Task 14 these registrations come from <c>OnReady</c> itself, and leaving this
/// method behind would mask a cutover that forgot one — the whole point of these tests.
/// </para>
/// </remarks>
private static void RegisterPhase2TablesUntilCutover(ILocalDb db)
{
foreach (var table in Phase2ReplicatedTables)
db.RegisterReplicated(table);
}
/// <summary>
/// The eight tables Task 14 registers. Listed literally rather than derived from
/// production code, so that a cutover which registers the wrong set fails these tests
/// instead of agreeing with itself.
/// The eight tables the Phase 2 cutover registers. Listed literally rather than derived
/// from production code, so a registration that drifts fails these tests instead of
/// agreeing with itself.
/// </summary>
/// <remarks>
/// <c>notification_lists</c> and <c>smtp_configurations</c> are absent by design. They