dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 2 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

perf(host): install CDC capture only when replication is configured

Browse Source 
SiteLocalDbSetup.OnReady registered all ten replicated tables
unconditionally, so a deliberately unreplicated site node (site-b and
site-c on the rig) carried the full 30-trigger CDC set forever. Every
write to those tables paid two extra INSERTs plus a json_object
serialization of the whole row, inside the caller's own transaction, and
appended to an oplog nothing ever drains. Arch-review finding #5 (High),
repo half; the library half — trigger cleanup API and O(1) backlog — is
WP3.3.

The ten RegisterReplicated calls are now behind a guard on whether the
node has LocalDb:Replication:PeerAddress OR LocalDb:Replication:ApiKey.
Either key counts, and the OR is load-bearing rather than defensive:
replication is one bidirectional stream that exactly one side dials, so
only the initiator sets PeerAddress. Verified against the rig — site-a
node-a has PeerAddress + ApiKey, site-a node-b (passive) has ApiKey
alone, site-b/site-c have no Replication section at all. Keying on
PeerAddress alone would have stripped capture from every passive node and
silently made each pair converge in one direction only.

The load-bearing ordering documented in the file is preserved: DDL still
precedes registration, and the legacy migrator still runs unconditionally
after it — an unreplicated node must still absorb its pre-Phase-1 files,
and it has no peer for those rows to be invisible to.

Known residual, documented in-file and in the topology guide: a database
file first created by an older build keeps its stale __localdb_* triggers.
The guard decides whether triggers are installed, not whether existing
ones are removed, and the library has no removal API until WP3.3. Moot on
the docker rig, where a schema-change redeploy recreates the volumes.
The inverse is also now documented: enabling replication on a site that
has run without it does not baseline existing rows, since CDC never
recorded them in __localdb_row_version and the snapshot resync streams
from that ledger.

Tests: new SiteLocalDbCdcRegistrationTests asserts trigger presence and
absence via sqlite_master across all four config shapes (none, ApiKey
only, PeerAddress + ApiKey, and the notification-table exclusion), plus
DDL-still-runs and migrator-still-runs on the unreplicated branch.
SiteLocalDbWiringTests and the integration site-pair harness now
configure an ApiKey — mirroring the rig's passive node — so their
registration and convergence assertions still describe a replicating
node. 483/483 Host.Tests pass; the 20 offline LocalDb convergence tests
still pass.
This commit is contained in:
Joseph Doherty
2026-08-14 19:59:53 -04:00
parent ee193cd2bb
commit 7ebdcd370a
5 changed files with 358 additions and 33 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbSetup.cs
+96 -31
View File
@@ -21,6 +21,14 @@ namespace ZB.MOM.WW.ScadaBridge.Host;
/// with no error anywhere.
/// </para>
/// <para>
/// <b>Registration is conditional</b> on the node actually having a replication peer —
/// see <see cref="ReplicationIsConfigured"/>. An unreplicated node used to install the
/// full CDC trigger set anyway and then pay for it on every single write, forever, for an
/// oplog nothing ever drains. The ordering rule above is unaffected: when registration is
/// skipped there is no capture to be too late for, and the legacy migrator still runs
/// (its rows simply do not replicate, which is what "unreplicated node" means).
/// </para>
/// <para>
/// Runs inside the <c>AddZbLocalDb</c> onReady callback, once, before any caller
/// receives the <see cref="ILocalDb"/> singleton. onReady is a synchronous
/// <c>Action&lt;ILocalDb&gt;</c>, hence the direct <c>CreateConnection()</c> rather than
@@ -32,11 +40,14 @@ namespace ZB.MOM.WW.ScadaBridge.Host;
public static class SiteLocalDbSetup
{
/// <summary>
/// Creates the site node's replicated tables, opts them into change capture, and
/// migrates any pre-Phase-1 databases in.
/// Creates the site node's tables, opts them into change capture when this node
/// replicates, and migrates any pre-Phase-1 databases in.
/// </summary>
/// <param name="db">The LocalDb instance being initialized.</param>
/// <param name="config">Configuration, for the legacy migrator's old path keys and node name.</param>
/// <param name="config">
/// Configuration, for the replication predicate, the legacy migrator's old path keys,
/// and the node name.
/// </param>
public static void OnReady(ILocalDb db, IConfiguration config)
{
ArgumentNullException.ThrowIfNull(db);
@@ -53,39 +64,93 @@ public static class SiteLocalDbSetup
StoreAndForwardSchema.Apply(connection);
}
// Both tables qualify: each has an explicit primary key (RegisterReplicated
// rejects tables without one) and no BLOB columns (which json_object cannot
// capture). Registration is idempotent and installs the capture triggers.
db.RegisterReplicated("OperationTracking");
db.RegisterReplicated("site_events");
if (ReplicationIsConfigured(config))
{
// Both tables qualify: each has an explicit primary key (RegisterReplicated
// rejects tables without one) and no BLOB columns (which json_object cannot
// capture). Registration is idempotent and installs the capture triggers.
db.RegisterReplicated("OperationTracking");
db.RegisterReplicated("site_events");
// Phase 2: the store-and-forward buffer and the seven site configuration tables.
// These replaced the bespoke SiteReplicationActor and StoreAndForward
// ReplicationService, which shipped hand-written Add/Remove/Park/Requeue operations
// over Akka; both were deleted in the same commit that added these lines, so the
// two mechanisms never ran at once.
//
// Both composite-PK tables are fine: RegisterReplicated orders multi-column PKs by
// ordinal. No Phase 2 table has a BLOB column, which it would reject.
db.RegisterReplicated("sf_messages");
db.RegisterReplicated("deployed_configurations");
db.RegisterReplicated("static_attribute_overrides");
db.RegisterReplicated("shared_scripts");
db.RegisterReplicated("external_systems");
db.RegisterReplicated("database_connections");
db.RegisterReplicated("data_connection_definitions");
db.RegisterReplicated("native_alarm_state");
// Phase 2: the store-and-forward buffer and the seven site configuration tables.
// These replaced the bespoke SiteReplicationActor and StoreAndForward
// ReplicationService, which shipped hand-written Add/Remove/Park/Requeue operations
// over Akka; both were deleted in the same commit that added these lines, so the
// two mechanisms never ran at once.
//
// Both composite-PK tables are fine: RegisterReplicated orders multi-column PKs by
// ordinal. No Phase 2 table has a BLOB column, which it would reject.
db.RegisterReplicated("sf_messages");
db.RegisterReplicated("deployed_configurations");
db.RegisterReplicated("static_attribute_overrides");
db.RegisterReplicated("shared_scripts");
db.RegisterReplicated("external_systems");
db.RegisterReplicated("database_connections");
db.RegisterReplicated("data_connection_definitions");
db.RegisterReplicated("native_alarm_state");
// notification_lists and smtp_configurations are created but deliberately NOT
// registered. They are permanently empty by design — the site-side write paths were
// removed on 2026-07-10, the legacy migrator skips them, and the active node's
// artifact apply purges them on every deploy. Registering them would open a standing
// replication channel whose only historical payload was plaintext SMTP passwords, in
// exchange for replicating nothing. Anyone adding them here should first establish
// that a site has a legitimate reason to hold SMTP credentials at all.
// notification_lists and smtp_configurations are created but deliberately NOT
// registered. They are permanently empty by design — the site-side write paths were
// removed on 2026-07-10, the legacy migrator skips them, and the active node's
// artifact apply purges them on every deploy. Registering them would open a standing
// replication channel whose only historical payload was plaintext SMTP passwords, in
// exchange for replicating nothing. Anyone adding them here should first establish
// that a site has a legitimate reason to hold SMTP credentials at all.
}
// AFTER registration, so migrated rows enter the oplog and reach the peer like
// any other write. Before it, they would be invisible to replication forever.
// Deliberately OUTSIDE the guard: an unreplicated node still has to absorb its
// pre-Phase-1 files, and there is no peer for its rows to be invisible to.
SiteLocalDbLegacyMigrator.Migrate(db, config);
}
/// <summary>
/// Whether this node participates in LocalDb replication, and therefore needs the CDC
/// capture triggers at all.
/// </summary>
/// <remarks>
/// <para>
/// <b>Either key counts, and that is not redundancy — it is the only predicate that is
/// true on BOTH halves of a replicated pair.</b> Replication is one bidirectional
/// stream that exactly one side dials, so only the initiator sets
/// <c>LocalDb:Replication:PeerAddress</c>; the passive node has an <c>ApiKey</c> and
/// nothing else. (The rig is the reference: site-a node-a carries PeerAddress + ApiKey,
/// site-a node-b carries ApiKey alone, and site-b/site-c carry no <c>Replication</c>
/// section at all.) Keying on PeerAddress alone would strip capture from every passive
/// node — its local writes would stop reaching the initiator, and the pair would
/// silently converge in one direction only.
/// </para>
/// <para>
/// The keys are read straight from configuration rather than through
/// <c>IOptions&lt;ReplicationOptions&gt;</c> because this runs inside the
/// <c>AddZbLocalDb</c> factory, where resolving another option would nest a service
/// resolution inside a singleton construction. The section name matches what
/// <c>AddZbLocalDbReplication</c> binds, so the two cannot disagree about which node
/// replicates.
/// </para>
/// <para>
/// <b>Known residual:</b> a database file that was registered by an OLDER build keeps
/// its stale <c>__localdb_*</c> triggers — this only decides whether new ones are
/// installed, and the library has no removal API yet (it arrives with the WP3.3 library
/// work, which will also drop them on an unconfigured node). On the docker rig this is
/// moot: a schema change recreates the volumes. On a long-lived unreplicated node
/// upgraded in place, capture continues until that lands.
/// </para>
/// <para>
/// <b>The other direction has a consequence too:</b> turning replication ON for a site
/// that has been running without it does NOT baseline the rows already in the file.
/// Capture never recorded them in <c>__localdb_row_version</c>, and the snapshot resync
/// streams from that ledger, so the pair converges only on writes made after the
/// restart. Seed both nodes from one node's database if the existing rows matter. See
/// <c>docs/deployment/topology-guide.md</c>.
/// </para>
/// </remarks>
private static bool ReplicationIsConfigured(IConfiguration config)
{
var section = config.GetSection("LocalDb:Replication");
return !string.IsNullOrWhiteSpace(section["PeerAddress"])
|| !string.IsNullOrWhiteSpace(section["ApiKey"]);
}
}