feat(localdb)!: replicate site config + sf_messages via CDC, delete the bespoke replicators
Tasks 14, 15 and 16, landed as ONE commit. PLAN DEFECT: these three tasks cannot compile separately. SiteReplicationActor takes a ReplicationService and calls ReplaceAllAsync (Task 14 deletes both); DeploymentManagerActor Tells message types declared in ReplicationMessages.cs (Task 15 deletes it); AkkaHostedService constructs the actor (Task 16). Any ordering leaves a broken intermediate. Combining them also strengthens the invariant Task 14 already stated for itself — the two mechanisms never both run, and never neither. Registered 8 tables in SiteLocalDbSetup.OnReady: sf_messages plus the 7 site config tables. notification_lists and smtp_configurations are deliberately NOT registered — permanently empty by design, so registering them would open a standing replication channel whose only historical payload was plaintext SMTP passwords. Migrate stays the LAST call in OnReady, after all registrations, so migrated rows enter the oplog through live capture triggers. Deleted: SiteReplicationActor, ReplicationMessages.cs, ReplicationService, StoreAndForwardStorage.ReplaceAllAsync, and 6 test files. ReplaceAllAsync is not merely unused but unsafe to keep: a mass DELETE on a now-replicated table would be captured and shipped to the peer. Kept ActiveNodeEvaluator (delivery gate + heartbeat still need it) with its doc corrected, and activeNodeCheck in AkkaHostedService (SiteCommunicationActor). The positional-argument hazard the plan flagged was real: removing DeploymentManagerActor's optional IActorRef? replicationActor shifted 6 trailing optionals, and 4 test call sites bound the wrong arguments with no compile error at some positions. Converted them to named arguments where possible — Props.Create builds an expression tree, which rejects out-of-position named args, so the rest are padded positionally with a comment saying why. The Task 7 'not yet registered' test was INVERTED rather than deleted, and is exact in both directions: too few means a table silently stops replicating, too many means the SMTP tables leak. Added a separate security-named test for those two, and a composite-PK test (LWW keys on the full PK, so a truncated key set would collapse distinct rows). The convergence suites now get their registrations from the real OnReady — their temporary harness registration is deleted, so they prove the cutover rather than agreeing with themselves. Verified: build 0 warnings; SiteRuntime 512, StoreAndForward 130, Host 330, AuditLog 355, ExternalSystemGateway 142, HealthMonitoring 97, LocalDb integration 16 — all pass, 0 failures. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
This commit is contained in:
@@ -766,37 +766,18 @@ akka {{
|
||||
var deploymentConfigFetcher =
|
||||
_serviceProvider.GetService<ZB.MOM.WW.ScadaBridge.SiteRuntime.Deployment.IDeploymentConfigFetcher>();
|
||||
|
||||
// Create SiteReplicationActor on every node (not a singleton)
|
||||
var sfStorage = _serviceProvider.GetRequiredService<StoreAndForwardStorage>();
|
||||
var replicationService = _serviceProvider.GetRequiredService<ReplicationService>();
|
||||
var replicationLogger = _serviceProvider.GetRequiredService<ILoggerFactory>()
|
||||
.CreateLogger<SiteReplicationActor>();
|
||||
|
||||
// ONE active-node predicate instance governs the S&F delivery gate, the resync
|
||||
// authority checks (SiteReplicationActor), and the heartbeat IsActive stamp
|
||||
// (SiteCommunicationActor, wired below) — review 02 round 2, N1. Null in
|
||||
// non-clustered test hosts: the actors fall back to the shared oldest-Up
|
||||
// evaluator, never to a leader check.
|
||||
// ONE active-node predicate instance governs the S&F delivery gate and the
|
||||
// heartbeat IsActive stamp (SiteCommunicationActor, wired below) — review 02
|
||||
// round 2, N1. It also governed SiteReplicationActor's resync authority until
|
||||
// LocalDb Phase 2 deleted that actor: the library's snapshot resync merges per row
|
||||
// under last-writer-wins and never deletes, so there is no destructive apply left
|
||||
// to need an authority check. Null in non-clustered test hosts: the consumers fall
|
||||
// back to the shared oldest-Up evaluator, never to a leader check.
|
||||
var clusterNodeProvider = _serviceProvider.GetService<ZB.MOM.WW.ScadaBridge.HealthMonitoring.IClusterNodeProvider>();
|
||||
Func<bool>? activeNodeCheck = clusterNodeProvider != null
|
||||
? () => clusterNodeProvider.SelfIsPrimary
|
||||
: null;
|
||||
|
||||
var replicationActor = _actorSystem!.ActorOf(
|
||||
Props.Create(() => new SiteReplicationActor(
|
||||
storage, sfStorage, replicationService, siteRole, replicationLogger,
|
||||
deploymentConfigFetcher, activeNodeCheck, siteRuntimeOptionsValue, null)),
|
||||
"site-replication");
|
||||
|
||||
// Wire S&F replication handler to forward operations via the replication actor
|
||||
replicationService.SetReplicationHandler(op =>
|
||||
{
|
||||
replicationActor.Tell(new ReplicateStoreAndForward(op));
|
||||
return Task.CompletedTask;
|
||||
});
|
||||
|
||||
_logger.LogInformation("SiteReplicationActor created and S&F replication handler wired");
|
||||
|
||||
// Deployment Manager — role-scoped singleton via SingletonRegistrar
|
||||
// (review 01 round-2 N5): previously hand-rolled with bare PoisonPill
|
||||
// termination and NO PhaseClusterLeave drain, so in-flight SQLite
|
||||
@@ -807,7 +788,7 @@ akka {{
|
||||
_actorSystem!, "deployment-manager",
|
||||
Props.Create(() => new DeploymentManagerActor(
|
||||
storage, compilationService, sharedScriptLibrary, streamManager,
|
||||
siteRuntimeOptionsValue, dmLogger, dclManager, replicationActor,
|
||||
siteRuntimeOptionsValue, dmLogger, dclManager,
|
||||
siteHealthCollector, _serviceProvider, null, deploymentConfigFetcher)),
|
||||
_logger, role: siteRole);
|
||||
var dmProxy = dm.Proxy;
|
||||
@@ -1053,7 +1034,7 @@ akka {{
|
||||
// SetReady asserts a deliberately narrow contract. By this point the
|
||||
// actor system exists, SiteStreamManager.Initialize has run, and every
|
||||
// role actor (SiteCommunicationActor, deployment-manager singleton,
|
||||
// SiteReplicationActor, the ClusterClient) has been created with ActorOf —
|
||||
// the ClusterClient) has been created with ActorOf —
|
||||
// creation and the registration Tells are synchronous and strictly ordered.
|
||||
// What is NOT guaranteed is completion of each actor's PreStart or the
|
||||
// ClusterClient's initial-contact handshake with central: those are
|
||||
|
||||
@@ -48,8 +48,7 @@ public static class SiteLocalDbSetup
|
||||
SiteEventLogSchema.Apply(connection);
|
||||
|
||||
// Phase 2: the site's configuration tables and the store-and-forward buffer
|
||||
// now live in this file too. Created here, deliberately NOT registered — see
|
||||
// below.
|
||||
// now live in this file too.
|
||||
SiteStorageSchema.Apply(connection);
|
||||
StoreAndForwardSchema.Apply(connection);
|
||||
}
|
||||
@@ -60,12 +59,30 @@ public static class SiteLocalDbSetup
|
||||
db.RegisterReplicated("OperationTracking");
|
||||
db.RegisterReplicated("site_events");
|
||||
|
||||
// The Phase 2 tables are created above but NOT registered yet. The bespoke
|
||||
// SiteReplicationActor / StoreAndForward ReplicationService still own replicating
|
||||
// them until the Task 14 cutover deletes both and registers these in one commit.
|
||||
// Registering them now would mean two independent replicators writing the same
|
||||
// rows — harmless in principle (both upsert) but it would mask a defect in either
|
||||
// one, which is exactly what the cutover needs to be able to see.
|
||||
// Phase 2: the store-and-forward buffer and the seven site configuration tables.
|
||||
// These replaced the bespoke SiteReplicationActor and StoreAndForward
|
||||
// ReplicationService, which shipped hand-written Add/Remove/Park/Requeue operations
|
||||
// over Akka; both were deleted in the same commit that added these lines, so the
|
||||
// two mechanisms never ran at once.
|
||||
//
|
||||
// Both composite-PK tables are fine: RegisterReplicated orders multi-column PKs by
|
||||
// ordinal. No Phase 2 table has a BLOB column, which it would reject.
|
||||
db.RegisterReplicated("sf_messages");
|
||||
db.RegisterReplicated("deployed_configurations");
|
||||
db.RegisterReplicated("static_attribute_overrides");
|
||||
db.RegisterReplicated("shared_scripts");
|
||||
db.RegisterReplicated("external_systems");
|
||||
db.RegisterReplicated("database_connections");
|
||||
db.RegisterReplicated("data_connection_definitions");
|
||||
db.RegisterReplicated("native_alarm_state");
|
||||
|
||||
// notification_lists and smtp_configurations are created but deliberately NOT
|
||||
// registered. They are permanently empty by design — the site-side write paths were
|
||||
// removed on 2026-07-10, the legacy migrator skips them, and the active node's
|
||||
// artifact apply purges them on every deploy. Registering them would open a standing
|
||||
// replication channel whose only historical payload was plaintext SMTP passwords, in
|
||||
// exchange for replicating nothing. Anyone adding them here should first establish
|
||||
// that a site has a legitimate reason to hold SMTP credentials at all.
|
||||
|
||||
// AFTER registration, so migrated rows enter the oplog and reach the peer like
|
||||
// any other write. Before it, they would be invisible to replication forever.
|
||||
|
||||
Reference in New Issue
Block a user