chore(deps): LocalDb 0.2.0 — dereg cleanup, late-opt-in baselining, byte-budget replication

This commit is contained in:
Joseph Doherty
2026-08-14 22:22:18 -04:00
parent 312216ff2b
commit cca7f1786d
9 changed files with 288 additions and 69 deletions
+3 -2
View File
@@ -79,7 +79,7 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
- DCL write failures returned synchronously to calling script. - DCL write failures returned synchronously to calling script.
- Tag path resolution retried periodically for devices still booting. - Tag path resolution retried periodically for devices still booting.
- Static attribute writes persisted to local SQLite (survive restart/failover, reset on redeployment). - Static attribute writes persisted to local SQLite (survive restart/failover, reset on redeployment).
- **Consolidated site database (LocalDb Phase 1 + 2, complete 2026-07-20).** Ten tables now live in ONE `ZB.MOM.WW.LocalDb`-managed SQLite file (enumerate them with `grep -r RegisterReplicated src/`), configured by the **required** `LocalDb:Path` (`/app/data/site-localdb.db` on the rig; validated with `ValidateOnStart`, so a site config missing it fails to boot). Consequences worth knowing: - **Consolidated site database (LocalDb Phase 1 + 2, complete 2026-07-20).** Ten tables now live in ONE `ZB.MOM.WW.LocalDb`-managed SQLite file (the `SiteLocalDbSetup.ReplicatedTables` list), configured by the **required** `LocalDb:Path` (`/app/data/site-localdb.db` on the rig; validated with `ValidateOnStart`, so a site config missing it fails to boot). Consequences worth knowing:
- `site_events.id` changed from autoincrement INTEGER to an application-minted **GUID**. Last-writer-wins keys on the primary key, so two nodes independently minting `id=1,2,3…` would destroy each other's events rather than merge them. The event-log read path uses a composite `(timestamp, id)` keyset cursor with an **opaque string** continuation token; `EventLogEntry.Id` and both `ContinuationToken`s are `string`/`string?` on the site↔central Akka DTOs. - `site_events.id` changed from autoincrement INTEGER to an application-minted **GUID**. Last-writer-wins keys on the primary key, so two nodes independently minting `id=1,2,3…` would destroy each other's events rather than merge them. The event-log read path uses a composite `(timestamp, id)` keyset cursor with an **opaque string** continuation token; `EventLogEntry.Id` and both `ContinuationToken`s are `string`/`string?` on the site↔central Akka DTOs.
- `ScadaBridge:OperationTracking:ConnectionString`, `ScadaBridge:SiteEventLog:DatabasePath` and — as of Phase 2 — `ScadaBridge:StoreAndForward:SqliteDbPath` + `ScadaBridge:Database:SiteDbPath` are all **migration-only** — nothing reads them but `SiteLocalDbLegacyMigrator`, which copies a pre-Phase-1 file in once (deterministic `mig-{NodeName}-{legacyId}` event ids, `INSERT OR IGNORE`, runs AFTER `RegisterReplicated` so migrated rows replicate) and renames it `.migrated`. Delete the keys once a node has migrated. - `ScadaBridge:OperationTracking:ConnectionString`, `ScadaBridge:SiteEventLog:DatabasePath` and — as of Phase 2 — `ScadaBridge:StoreAndForward:SqliteDbPath` + `ScadaBridge:Database:SiteDbPath` are all **migration-only** — nothing reads them but `SiteLocalDbLegacyMigrator`, which copies a pre-Phase-1 file in once (deterministic `mig-{NodeName}-{legacyId}` event ids, `INSERT OR IGNORE`, runs AFTER `RegisterReplicated` so migrated rows replicate) and renames it `.migrated`. Delete the keys once a node has migrated.
- This incidentally fixes a data-loss bug: both legacy databases defaulted to CWD-relative paths **outside** the mounted volume and were discarded on every container recreate. - This incidentally fixes a data-loss bug: both legacy databases defaulted to CWD-relative paths **outside** the mounted volume and were discarded on every container recreate.
@@ -89,7 +89,8 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
- CDC replication does all three jobs now: config deploys reach the standby as ordinary row changes — **the standby makes no fetch at all** during a deploy (`SiteReconciliationActor`'s node-STARTUP fetch when central reports gaps is a different, surviving path) — and buffer mutations replicate via triggers on `sf_messages`. `ReplaceAllAsync` was a destructive delete-all-then-insert-all resync and is **unsafe to reintroduce**: a mass DELETE on a replicated table would be captured and shipped to the peer. LocalDb's snapshot resync merges per row under LWW and never deletes, which is also why the old N1 directional-authority guard is gone — there is no wipe left to gate. - CDC replication does all three jobs now: config deploys reach the standby as ordinary row changes — **the standby makes no fetch at all** during a deploy (`SiteReconciliationActor`'s node-STARTUP fetch when central reports gaps is a different, surviving path) — and buffer mutations replicate via triggers on `sf_messages`. `ReplaceAllAsync` was a destructive delete-all-then-insert-all resync and is **unsafe to reintroduce**: a mass DELETE on a replicated table would be captured and shipped to the peer. LocalDb's snapshot resync merges per row under LWW and never deletes, which is also why the old N1 directional-authority guard is gone — there is no wipe left to gate.
- **`notification_lists` and `smtp_configurations` are created but deliberately NOT registered.** They are permanently empty on a site (no writer since 2026-07-10, the migrator skips them, the active-node purge keeps them empty), and registering them would open a standing replication channel whose only historical payload was plaintext SMTP passwords. Pinned by a security-named test, and verified live: those two tables have **no CDC triggers** on either rig node. - **`notification_lists` and `smtp_configurations` are created but deliberately NOT registered.** They are permanently empty on a site (no writer since 2026-07-10, the migrator skips them, the active-node purge keeps them empty), and registering them would open a standing replication channel whose only historical payload was plaintext SMTP passwords. Pinned by a security-named test, and verified live: those two tables have **no CDC triggers** on either rig node.
- **Operational constraints (read before upgrading a site pair):** stop and start both nodes TOGETHER — rolling one at a time is no longer supported, since the legacy `SfBufferSnapshot` compatibility handler went with the replicator. And a node offline longer than `LocalDb:Replication:TombstoneRetention` (default 7 days) can resurrect deleted rows on rejoin. See `docs/deployment/topology-guide.md`. - **Operational constraints (read before upgrading a site pair):** stop and start both nodes TOGETHER — rolling one at a time is no longer supported, since the legacy `SfBufferSnapshot` compatibility handler went with the replicator. And a node offline longer than `LocalDb:Replication:TombstoneRetention` (default 7 days) can resurrect deleted rows on rejoin. See `docs/deployment/topology-guide.md`.
- `LocalDb:Replication:MaxBatchSize` batches by ROW COUNT, not bytes, against a 4 MB gRPC cap — the rig pins it to **16** (~70 KB worst-case `config_json` x 16 ~= 1.1 MB). The 500 default would allow ~35 MB. - **Batching is by BYTE BUDGET as of LocalDb 0.2.0** — `LocalDb:Replication:MaxBatchBytes` (default **2 MB**, sized under the 4 MB gRPC cap) bounds a delta/snapshot message by summed serialized size, with `MaxBatchSize` demoted to a secondary row cap; a single row over budget is sent alone rather than stalling the stream. **The rig's `MaxBatchSize = 16` pin is retired** (it was a hand-computed proxy: ~70 KB worst-case `config_json` x the 500 default is ~35 MB); both keys are now left unset on `docker/`.
- **CDC registration is conditional, and both directions self-heal at boot** (arch-review WP1.3 + WP3.3, `Host/SiteLocalDbSetup.cs`). Capture triggers are installed only when this node has replication configured — `PeerAddress` **or** `ApiKey`, an OR because only the dialling half sets `PeerAddress` while the passive half carries the key alone. An unreplicated node calls **`DeregisterReplicated`** on all ten tables at boot, dropping triggers an earlier build left behind and pruning their oplog/row-version rows (idempotent; logs once at Information when something was actually cleaned). A replicating node registers with **`baselineExistingRows: true`**, which seeds `__localdb_row_version` for pre-existing rows at the LWW floor (HLC `0`, this node's id) and flags a snapshot resync — so turning replication ON for a site that has been running without it now converges on the rows already in the file instead of only on writes made after the restart. Deregistration must be symmetric (the handshake compares registered-table digests fail-closed), so replication is a both-nodes-together change in either direction. LocalDb 0.2.0 also makes backlog depth O(1) and drops the unused `__localdb_oplog_hlc` index; the on-disk bookkeeping schema goes to v2, upgraded in place on open, with no wire change (0.1.x peers still sync).
- All timestamps are UTC throughout the system. - All timestamps are UTC throughout the system.
- Inter-cluster communication uses **three** transports, not two — **all cross-cluster command/control and data now rides gRPC** after the ClusterClient→gRPC migration's Phase 4 (`docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`) deleted Akka `ClusterClient`/`ClusterClientReceptionist`: (1) **gRPC command/control** — site→central over the central-hosted `CentralControlService` (`GrpcCentralTransport`, sticky central-a→central-b channel pair; deployments/notifications/health/heartbeat/audit-ingest/reconcile), and central→site over the site-hosted `SiteCommandService` (`GrpcSiteTransport`, per-site NodeA→NodeB channel pair; the 28 lifecycle/OPC-UA/query/parked/route/failover commands); (2) **gRPC** server-streaming for real-time data (attribute values, alarm states, `SiteStreamService`); and (3) **plain token-gated HTTP** for the deployment config itself — notify-and-fetch, the site pulls the config from `DeploymentConfigEndpoints` (`ManagementService/DeploymentConfigEndpoints.cs`) with an `X-Deployment-Token` header, `AllowAnonymous` with the per-deployment token as the entire security boundary. The gRPC boundary is per-site PSK-authenticated (`ControlPlaneAuthInterceptor`, unchanged). There is **no receptionist registration** — discovery is by dialling configured endpoints; central builds one `SitePairChannelProvider` per site (addresses from `Site.GrpcNodeAAddress`/`GrpcNodeBAddress`, refreshed from the DB every 60s and on admin changes), sites dial `ScadaBridge:Communication:CentralGrpcEndpoints` (both central nodes, h2c on `CentralGrpcPort` 8083, **NOT** via Traefik). **Discovery is asymmetric by design:** central discovers site gRPC addresses from the *database* (refreshable at runtime), sites discover central from *appsettings* (`CentralGrpcEndpoints`, static — restart required; `StartupValidator` requires a Site node to list at least one). `Akka.Cluster.Tools` stays for ClusterSingleton; only the ClusterClient part is gone. **Central never buffers for an unreachable site** — the send fails with the caller's Ask/deadline timing out; a `ConnectionStateChanged` mechanism built for this was deleted as dead code. - Inter-cluster communication uses **three** transports, not two — **all cross-cluster command/control and data now rides gRPC** after the ClusterClient→gRPC migration's Phase 4 (`docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`) deleted Akka `ClusterClient`/`ClusterClientReceptionist`: (1) **gRPC command/control** — site→central over the central-hosted `CentralControlService` (`GrpcCentralTransport`, sticky central-a→central-b channel pair; deployments/notifications/health/heartbeat/audit-ingest/reconcile), and central→site over the site-hosted `SiteCommandService` (`GrpcSiteTransport`, per-site NodeA→NodeB channel pair; the 28 lifecycle/OPC-UA/query/parked/route/failover commands); (2) **gRPC** server-streaming for real-time data (attribute values, alarm states, `SiteStreamService`); and (3) **plain token-gated HTTP** for the deployment config itself — notify-and-fetch, the site pulls the config from `DeploymentConfigEndpoints` (`ManagementService/DeploymentConfigEndpoints.cs`) with an `X-Deployment-Token` header, `AllowAnonymous` with the per-deployment token as the entire security boundary. The gRPC boundary is per-site PSK-authenticated (`ControlPlaneAuthInterceptor`, unchanged). There is **no receptionist registration** — discovery is by dialling configured endpoints; central builds one `SitePairChannelProvider` per site (addresses from `Site.GrpcNodeAAddress`/`GrpcNodeBAddress`, refreshed from the DB every 60s and on admin changes), sites dial `ScadaBridge:Communication:CentralGrpcEndpoints` (both central nodes, h2c on `CentralGrpcPort` 8083, **NOT** via Traefik). **Discovery is asymmetric by design:** central discovers site gRPC addresses from the *database* (refreshable at runtime), sites discover central from *appsettings* (`CentralGrpcEndpoints`, static — restart required; `StartupValidator` requires a Site node to list at least one). `Akka.Cluster.Tools` stays for ClusterSingleton; only the ClusterClient part is gone. **Central never buffers for an unreachable site** — the send fails with the caller's Ask/deadline timing out; a `ConnectionStateChanged` mechanism built for this was deleted as dead code.
- **All clusters share ONE ActorSystem name**, `"scadabridge"` — hardcoded in `AkkaHostedService` at the `ActorSystem.Create` call. Central and each site are separate clusters *only* by seed-node partitioning. The constraint originated with ClusterClient (Akka.Remote address matching meant it could not reach a differently-named system); whether it is still load-bearing after the gRPC migration has **not** been re-verified, so treat the name as fixed until someone checks. - **All clusters share ONE ActorSystem name**, `"scadabridge"` — hardcoded in `AkkaHostedService` at the `ActorSystem.Create` call. Central and each site are separate clusters *only* by seed-node partitioning. The constraint originated with ClusterClient (Akka.Remote address matching meant it could not reach a differently-named system); whether it is still load-bearing after the gRPC migration has **not** been re-verified, so treat the name as fixed until someone checks.
+3 -3
View File
@@ -109,9 +109,9 @@
<PackageVersion Include="ZB.MOM.WW.Secrets.Ui" Version="0.6.2" /> <PackageVersion Include="ZB.MOM.WW.Secrets.Ui" Version="0.6.2" />
<PackageVersion Include="ZB.MOM.WW.Secrets.Replicator.SqlServer" Version="0.6.2" /> <PackageVersion Include="ZB.MOM.WW.Secrets.Replicator.SqlServer" Version="0.6.2" />
<PackageVersion Include="ZB.MOM.WW.Secrets.Replicator.Grpc" Version="0.6.2" /> <PackageVersion Include="ZB.MOM.WW.Secrets.Replicator.Grpc" Version="0.6.2" />
<PackageVersion Include="ZB.MOM.WW.LocalDb" Version="0.1.3" /> <PackageVersion Include="ZB.MOM.WW.LocalDb" Version="0.2.0" />
<PackageVersion Include="ZB.MOM.WW.LocalDb.Replication" Version="0.1.3" /> <PackageVersion Include="ZB.MOM.WW.LocalDb.Replication" Version="0.2.0" />
<PackageVersion Include="ZB.MOM.WW.LocalDb.Contracts" Version="0.1.3" /> <PackageVersion Include="ZB.MOM.WW.LocalDb.Contracts" Version="0.2.0" />
</ItemGroup> </ItemGroup>
<!-- <!--
+9 -6
View File
@@ -104,12 +104,15 @@
"ApiKey": "dev-site-a-localdb-sync-key", "ApiKey": "dev-site-a-localdb-sync-key",
// ---- Phase 2 sizing, from the Task 1 rig soak (not from the defaults) ---- // ---- Phase 2 sizing, from the Task 1 rig soak (not from the defaults) ----
// //
// MaxBatchSize (default 500) is a ROW count, not a byte budget, so the batch // The MaxBatchSize = 16 pin is RETIRED as of LocalDb 0.2.0 (arch-review WP3.3).
// size in bytes is set by the widest replicated column. That is // It existed only as a hand-computed proxy for a byte budget: batching was
// deployed_configurations.config_json: ~721 B on this rig, but up to ~60-70 KB // row-count-only, and 70 KB of production config_json x the 500 default is
// in production (measured, Task 1) - and 70 KB x 500 is ~35 MB against gRPC's // ~35 MB against gRPC's 4 MB receive limit. The library now bounds a batch by
// 4 MB default receive limit. 16 keeps a worst-case batch near 1.1 MB. // MaxBatchBytes (default 2 MB of summed serialized size), with the row count
"MaxBatchSize": 16, // demoted to a secondary cap, so both are left at their defaults here - a
// deliberately unset MaxBatchBytes is the 2 MB default, and the widest row no
// longer has to be guessed at deploy time.
//
// Backlog caps bound the oplog while the peer is offline. Exceeding them is // Backlog caps bound the oplog while the peer is offline. Exceeding them is
// NOT data loss: the oplog is pruned to the ceiling and needs_snapshot is set, // NOT data loss: the oplog is pruned to the ceiling and needs_snapshot is set,
// so the peer catches up by snapshot resync instead of incrementally. That // so the peer catches up by snapshot resync instead of incrementally. That
+9 -6
View File
@@ -97,12 +97,15 @@
"ApiKey": "dev-site-a-localdb-sync-key", "ApiKey": "dev-site-a-localdb-sync-key",
// ---- Phase 2 sizing, from the Task 1 rig soak (not from the defaults) ---- // ---- Phase 2 sizing, from the Task 1 rig soak (not from the defaults) ----
// //
// MaxBatchSize (default 500) is a ROW count, not a byte budget, so the batch // The MaxBatchSize = 16 pin is RETIRED as of LocalDb 0.2.0 (arch-review WP3.3).
// size in bytes is set by the widest replicated column. That is // It existed only as a hand-computed proxy for a byte budget: batching was
// deployed_configurations.config_json: ~721 B on this rig, but up to ~60-70 KB // row-count-only, and 70 KB of production config_json x the 500 default is
// in production (measured, Task 1) - and 70 KB x 500 is ~35 MB against gRPC's // ~35 MB against gRPC's 4 MB receive limit. The library now bounds a batch by
// 4 MB default receive limit. 16 keeps a worst-case batch near 1.1 MB. // MaxBatchBytes (default 2 MB of summed serialized size), with the row count
"MaxBatchSize": 16, // demoted to a secondary cap, so both are left at their defaults here - a
// deliberately unset MaxBatchBytes is the 2 MB default, and the widest row no
// longer has to be guessed at deploy time.
//
// Backlog caps bound the oplog while the peer is offline. Exceeding them is // Backlog caps bound the oplog while the peer is offline. Exceeding them is
// NOT data loss: the oplog is pruned to the ceiling and needs_snapshot is set, // NOT data loss: the oplog is pruned to the ceiling and needs_snapshot is set,
// so the peer catches up by snapshot resync instead of incrementally. That // so the peer catches up by snapshot resync instead of incrementally. That
+31 -11
View File
@@ -162,21 +162,41 @@ Each site has its own two-node cluster:
only the initiating half of a pair sets `PeerAddress` (one bidirectional stream, dialled by one only the initiating half of a pair sets `PeerAddress` (one bidirectional stream, dialled by one
side); the passive half carries the key alone. A deliberately unreplicated node — site-b and side); the passive half carries the key alone. A deliberately unreplicated node — site-b and
site-c on the rig — runs with no triggers at all and stops paying the per-write capture cost. site-c on the rig — runs with no triggers at all and stops paying the per-write capture cost.
- **Stale-trigger cleanup is automatic** (LocalDb 0.2.0). A node with no replication configured does
not merely skip registration — at boot it calls `DeregisterReplicated` on all ten tables, dropping
any capture triggers an earlier build installed and pruning those tables' oplog and row-version
rows. It is idempotent, so a file that was never registered reports nothing to clean; when
something *was* cleaned the node logs it once at Information. Recreating the data volume is no
longer required to stop an in-place-upgraded node from capturing.
#### Turning replication ON for a site that has been running without it #### Turning replication ON for a site that has been running without it
Set the keys on **both** nodes and restart both. Two things to know before you do: **Supported as of LocalDb 0.2.0.** Set the keys on **both** nodes and restart both (see the
stop-and-start-together rule below — this is a pair-wide change, not a rolling one). Existing rows
are carried across:
- **Existing rows are not baselined.** Capture is change-data-capture: rows written while the node - **Pre-existing rows are baselined automatically.** ScadaBridge registers every replicated table
had no triggers were never recorded in `__localdb_row_version`, and LocalDb's snapshot resync with `baselineExistingRows: true`. Capture is change-data-capture, so rows written while the node
streams from that ledger, so it will not ship them. The pair converges on everything written had no triggers appear in neither the oplog nor `__localdb_row_version` — and LocalDb's snapshot
*after* the restart and stays silently divergent on everything before it. Start from a copy of one resync streams from that ledger. Baselining seeds the ledger for those rows at the LWW floor
node's database on both sides, or accept that only new writes converge. (HLC `0`, stamped with the node's own id) and flags a snapshot resync, so the peer actually
- **A node upgraded in place keeps stale triggers.** The guard decides whether triggers are receives them. Copying one node's database onto the other beforehand is no longer necessary.
*installed*, not whether existing ones are removed, and the library has no removal API yet. A - **What the floor means for conflicts.** Every genuine HLC is a UTC millisecond shifted left 16
database file first created by a build that always registered keeps capturing until that lands. bits, so it is strictly greater than `0`: a baselined row loses to any real remote write of the
Recreating the node's data volume clears it — which is what a schema-change redeploy does on the same key and wins only where the peer holds no version of that key at all. The one ambiguous case
docker rig, so the rig is unaffected. is **both** nodes baselining the same key (e.g. both were restored from the same legacy file) —
both hold HLC `0` and the node-id tie-break decides. That is convergent but arbitrary as to which
content survives, so if the two files may disagree on a key, start both nodes from one node's
database.
- **Seeding is idempotent** (`ON CONFLICT DO NOTHING`), so a row that already has a genuine version
keeps it and no snapshot is flagged. Booting with baselining on every start is free after the
first.
Turning replication back **OFF** is likewise a both-nodes change: deregistration must be symmetric,
because the sync handshake compares the two nodes' registered-table digests fail-closed — a node
that drops a table its peer still replicates stops syncing with a schema-mismatch error rather than
diverging silently. Turning it on again later re-baselines, which is what makes the ledger prune on
deregistration safe.
### Site Pair Upgrades — stop and start BOTH nodes together ### Site Pair Upgrades — stop and start BOTH nodes together
@@ -22,11 +22,17 @@ The central→site hop is unchanged — it still sends a small `RefreshDeploymen
still fetches over HTTP, so that half of the original fix stands. still fetches over HTTP, so that half of the original fix stands.
**The successor ceiling is different in kind.** The gRPC sync stream has a 4 MB default receive **The successor ceiling is different in kind.** The gRPC sync stream has a 4 MB default receive
limit, and LocalDb batches by ROW COUNT (`LocalDb:Replication:MaxBatchSize`, default 500), not by limit. Until LocalDb 0.2.0 the library batched by ROW COUNT only (`LocalDb:Replication:MaxBatchSize`,
bytes. A ~70 KB `config_json` — the largest measured in production — times 500 rows is ~35 MB, default 500): a ~70 KB `config_json` — the largest measured in production — times 500 rows is
which would exceed the limit. The rig therefore pins `MaxBatchSize` to **16** (~1.1 MB worst case). ~35 MB, which would exceed the limit, so the rig pinned `MaxBatchSize` to **16** (~1.1 MB worst
Any deployment replicating wide rows must size that key deliberately; see the Phase 2 plan (D6) and case). See the Phase 2 plan (D6) and `docs/plans/2026-07-19-localdb-phase2-live-gate.md`.
`docs/plans/2026-07-19-localdb-phase2-live-gate.md`.
**Closed by LocalDb 0.2.0** (arch-review WP3.3): batching is now bounded by
`LocalDb:Replication:MaxBatchBytes` — summed serialized bytes, default **2 MB**, sized under the
4 MB limit — with the row count demoted to a secondary cap, and a single row above the budget sent
alone rather than stalling the stream. The rig's `MaxBatchSize = 16` pin is retired and both keys
are left at their defaults; a deployment replicating wide rows no longer has to size a row count
against its widest column by hand.
Note the failure mode differs from the one documented below: an oversized gRPC message is Note the failure mode differs from the one documented below: an oversized gRPC message is
**rejected**, not silently dropped. **rejected**, not silently dropped.
@@ -29,6 +29,13 @@ namespace ZB.MOM.WW.ScadaBridge.Host;
/// (its rows simply do not replicate, which is what "unreplicated node" means). /// (its rows simply do not replicate, which is what "unreplicated node" means).
/// </para> /// </para>
/// <para> /// <para>
/// <b>The unreplicated branch actively cleans up</b> rather than merely abstaining:
/// <c>DeregisterReplicated</c> (LocalDb 0.2.0) drops any capture triggers a previous build
/// left in the file and prunes that table's oplog and row-version rows. Skipping
/// registration alone would have left a file registered under an older build capturing
/// forever, which is exactly the node that has no peer to capture for.
/// </para>
/// <para>
/// Runs inside the <c>AddZbLocalDb</c> onReady callback, once, before any caller /// Runs inside the <c>AddZbLocalDb</c> onReady callback, once, before any caller
/// receives the <see cref="ILocalDb"/> singleton. onReady is a synchronous /// receives the <see cref="ILocalDb"/> singleton. onReady is a synchronous
/// <c>Action&lt;ILocalDb&gt;</c>, hence the direct <c>CreateConnection()</c> rather than /// <c>Action&lt;ILocalDb&gt;</c>, hence the direct <c>CreateConnection()</c> rather than
@@ -39,6 +46,30 @@ namespace ZB.MOM.WW.ScadaBridge.Host;
/// </remarks> /// </remarks>
public static class SiteLocalDbSetup public static class SiteLocalDbSetup
{ {
/// <summary>
/// The ten tables that participate in replication — registered when this node has a peer,
/// and deregistered (stale triggers dropped) when it does not.
/// </summary>
/// <remarks>
/// <c>notification_lists</c> and <c>smtp_configurations</c> are created by the schema but
/// deliberately absent here; see the rationale in <see cref="OnReady"/>. The list is shared by
/// both branches on purpose: a table added to one and not the other would either register
/// without ever being cleaned up, or be cleaned up on a node that never registered it.
/// </remarks>
private static readonly string[] ReplicatedTables =
[
"OperationTracking",
"site_events",
"sf_messages",
"deployed_configurations",
"static_attribute_overrides",
"shared_scripts",
"external_systems",
"database_connections",
"data_connection_definitions",
"native_alarm_state",
];
/// <summary> /// <summary>
/// Creates the site node's tables, opts them into change capture when this node /// Creates the site node's tables, opts them into change capture when this node
/// replicates, and migrates any pre-Phase-1 databases in. /// replicates, and migrates any pre-Phase-1 databases in.
@@ -66,36 +97,70 @@ public static class SiteLocalDbSetup
if (ReplicationIsConfigured(config)) if (ReplicationIsConfigured(config))
{ {
// Both tables qualify: each has an explicit primary key (RegisterReplicated // Every table below qualifies: each has an explicit primary key (RegisterReplicated
// rejects tables without one) and no BLOB columns (which json_object cannot // rejects tables without one) and no BLOB columns (which json_object cannot
// capture). Registration is idempotent and installs the capture triggers. // capture); the two composite-PK tables are fine, since RegisterReplicated orders
db.RegisterReplicated("OperationTracking"); // multi-column PKs by ordinal. Registration is idempotent and installs the capture
db.RegisterReplicated("site_events"); // triggers.
//
// Phase 2: the store-and-forward buffer and the seven site configuration tables. // The Phase 2 members of the list — the store-and-forward buffer and the seven site
// These replaced the bespoke SiteReplicationActor and StoreAndForward // configuration tables — replaced the bespoke SiteReplicationActor and StoreAndForward
// ReplicationService, which shipped hand-written Add/Remove/Park/Requeue operations // ReplicationService, which shipped hand-written Add/Remove/Park/Requeue operations
// over Akka; both were deleted in the same commit that added these lines, so the // over Akka; both were deleted in the same commit that added these lines, so the
// two mechanisms never ran at once. // two mechanisms never ran at once.
// //
// Both composite-PK tables are fine: RegisterReplicated orders multi-column PKs by // baselineExistingRows: true is what makes turning replication ON for a site that has
// ordinal. No Phase 2 table has a BLOB column, which it would reject. // been running WITHOUT it actually converge. Capture is change-data-capture and the
db.RegisterReplicated("sf_messages"); // snapshot streamer pages from __localdb_row_version, so rows written before the first
db.RegisterReplicated("deployed_configurations"); // registration exist in neither — a late opt-in used to replicate only writes made
db.RegisterReplicated("static_attribute_overrides"); // after the restart, silently and forever. Baselining seeds the ledger at the LWW
db.RegisterReplicated("shared_scripts"); // floor (HLC 0, this node's id) and flags a snapshot resync, so a baselined row loses
db.RegisterReplicated("external_systems"); // to any genuine remote write of the same key and wins only where the peer has no
db.RegisterReplicated("database_connections"); // version at all. It is idempotent (ON CONFLICT DO NOTHING) and therefore free on
db.RegisterReplicated("data_connection_definitions"); // every boot after the first, which is why it is unconditional rather than a flag.
db.RegisterReplicated("native_alarm_state"); foreach (var table in ReplicatedTables)
db.RegisterReplicated(table, baselineExistingRows: true);
// notification_lists and smtp_configurations are created but deliberately NOT // notification_lists and smtp_configurations are created but deliberately NOT
// registered. They are permanently empty by design — the site-side write paths were // registered — see ReplicatedTables. They are permanently empty by design — the
// removed on 2026-07-10, the legacy migrator skips them, and the active node's // site-side write paths were removed on 2026-07-10, the legacy migrator skips them,
// artifact apply purges them on every deploy. Registering them would open a standing // and the active node's artifact apply purges them on every deploy. Registering them
// replication channel whose only historical payload was plaintext SMTP passwords, in // would open a standing replication channel whose only historical payload was
// exchange for replicating nothing. Anyone adding them here should first establish // plaintext SMTP passwords, in exchange for replicating nothing. Anyone adding them
// that a site has a legitimate reason to hold SMTP credentials at all. // here should first establish that a site has a legitimate reason to hold SMTP
// credentials at all.
}
else
{
// Not merely "do not register": actively remove capture this node must not pay for.
// The file may have been registered by an EARLIER build (every build before WP1.3
// registered unconditionally), in which case its triggers are still there, still
// running two INSERTs plus a json_object of the full row inside every write
// transaction, still appending to an oplog with no reader. DeregisterReplicated is
// idempotent and deliberately usable on a table this process never registered, so a
// never-replicated file simply reports nothing to clean.
//
// Symmetry is not a concern on this branch: this node has no peer, so there is no
// handshake digest for a one-sided deregistration to fail. And re-enabling later is
// safe despite the ledger prune, because the registration branch above baselines.
var cleaned = 0;
foreach (var table in ReplicatedTables)
{
if (db.DeregisterReplicated(table)) cleaned++;
}
if (cleaned > 0)
{
// Serilog's static logger, as in SecretsRegistration: this runs inside the
// AddZbLocalDb singleton factory, where no ILogger can be resolved without
// nesting a service resolution inside a singleton construction. Program.cs
// configures Log.Logger before the service graph is built.
Serilog.Log.Information(
"LocalDb: removed stale change-capture from {TableCount} table(s) — this node has no "
+ "replication peer configured, so the triggers a previous build installed have been "
+ "dropped and their oplog/row-version rows pruned.",
cleaned);
}
} }
// AFTER registration, so migrated rows enter the oplog and reach the peer like // AFTER registration, so migrated rows enter the oplog and reach the peer like
@@ -130,19 +195,16 @@ public static class SiteLocalDbSetup
/// replicates. /// replicates.
/// </para> /// </para>
/// <para> /// <para>
/// <b>Known residual:</b> a database file that was registered by an OLDER build keeps /// <b>Both directions of a change to this predicate are now handled at boot</b> (LocalDb
/// its stale <c>__localdb_*</c> triggers — this only decides whether new ones are /// 0.2.0). Flipping it to false deregisters, so a file registered by an older build stops
/// installed, and the library has no removal API yet (it arrives with the WP3.3 library /// capturing on the next start instead of paying for triggers forever; flipping it to true
/// work, which will also drop them on an unconfigured node). On the docker rig this is /// baselines, so a site that has been running unreplicated converges on the rows already in
/// moot: a schema change recreates the volumes. On a long-lived unreplicated node /// its file and not merely on writes made after the restart. Seeding one node from the
/// upgraded in place, capture continues until that lands. /// other's database beforehand is no longer required — only still advisable where BOTH
/// </para> /// files hold their own version of the same key, since two baselined rows both sit at HLC
/// <para> /// 0 and the node-id tie-break then decides arbitrarily which content survives. Either
/// <b>The other direction has a consequence too:</b> turning replication ON for a site /// way both nodes must be changed together: deregistration is only symmetric — and the
/// that has been running without it does NOT baseline the rows already in the file. /// handshake digest only agrees — if both sides do it. See
/// Capture never recorded them in <c>__localdb_row_version</c>, and the snapshot resync
/// streams from that ledger, so the pair converges only on writes made after the
/// restart. Seed both nodes from one node's database if the existing rows matter. See
/// <c>docs/deployment/topology-guide.md</c>. /// <c>docs/deployment/topology-guide.md</c>.
/// </para> /// </para>
/// </remarks> /// </remarks>
@@ -233,6 +233,12 @@ public class SiteHealthCheckTests : IDisposable
public ReplicatedTable RegisterReplicated(string tableName) => public ReplicatedTable RegisterReplicated(string tableName) =>
throw new InvalidOperationException("store unreachable"); throw new InvalidOperationException("store unreachable");
public ReplicatedTable RegisterReplicated(string tableName, bool baselineExistingRows) =>
throw new InvalidOperationException("store unreachable");
public bool DeregisterReplicated(string tableName) =>
throw new InvalidOperationException("store unreachable");
public IReadOnlyDictionary<string, ReplicatedTable> ReplicatedTables => public IReadOnlyDictionary<string, ReplicatedTable> ReplicatedTables =>
throw new InvalidOperationException("store unreachable"); throw new InvalidOperationException("store unreachable");
} }
@@ -104,6 +104,81 @@ public class SiteLocalDbCdcRegistrationTests : IDisposable
Assert.DoesNotContain(triggers, t => t.StartsWith("__localdb_smtp_configurations_", StringComparison.Ordinal)); Assert.DoesNotContain(triggers, t => t.StartsWith("__localdb_smtp_configurations_", StringComparison.Ordinal));
} }
[Fact]
public void UnreplicatedNode_DropsTriggersLeftByAPreviouslyReplicatedBuild()
{
// WP3.3, the residual WP1.3 could only document: skipping registration never removed
// what an EARLIER build installed, so a long-lived unreplicated node upgraded in place
// kept capturing forever. LocalDb 0.2.0's DeregisterReplicated closes it, and this is
// the exact upgrade shape — the same file, booted first WITH replication configured and
// then without. The first boot is what makes the assertion meaningful: without it an
// empty trigger set would prove nothing.
var replicated = BuildDatabase(Config(apiKey: "cdc-test-key"));
AssertCaptureTriggersCoverTheReplicatedTables(replicated);
DisposeProviders();
var unreplicated = BuildDatabase(Config());
Assert.Empty(CaptureTriggers(unreplicated));
}
[Fact]
public void UnreplicatedNode_PrunesTheOplogLeftByAPreviouslyReplicatedBuild()
{
// Deregistration is not only about the trigger cost: the rows already captured are an
// oplog nothing will ever drain, so they are pruned with the triggers. Written through
// the schema's own table so the capture path is the real one.
var replicated = BuildDatabase(Config(apiKey: "cdc-test-key"));
InsertSiteEvent(replicated, "evt-1");
Assert.NotEqual(0, OplogRowCount(replicated));
DisposeProviders();
var unreplicated = BuildDatabase(Config());
Assert.Equal(0, OplogRowCount(unreplicated));
// The table's own data is never touched — deregistration removes bookkeeping, not rows.
Assert.Equal(1, SiteEventCount(unreplicated));
}
[Fact]
public void UnreplicatedNode_OverAFileThatWasNeverRegistered_IsANoOp()
{
// DeregisterReplicated is idempotent and usable on a table this process never
// registered, so the clean-up branch must be harmless on the ordinary case: site-b and
// site-c boot after boot with nothing to clean.
_ = BuildDatabase(Config());
DisposeProviders();
var second = BuildDatabase(Config());
Assert.Empty(CaptureTriggers(second));
foreach (var table in AllSiteTables)
Assert.Contains(table, TableNames(second));
}
[Fact]
public void ReplicatedNode_BaselinesRowsThatPredateRegistration()
{
// The other half of the late-opt-in story: a site that has been running unreplicated
// and then turns replication ON. The snapshot streamer pages from __localdb_row_version,
// which only the capture triggers populate, so without baselining these rows would be
// invisible to the peer forever. RegisterReplicated(..., baselineExistingRows: true)
// seeds them at the LWW floor (HLC 0).
var unreplicated = BuildDatabase(Config());
Assert.Empty(CaptureTriggers(unreplicated));
InsertSiteEvent(unreplicated, "pre-existing");
Assert.Equal(0, RowVersionCount(unreplicated, "site_events"));
DisposeProviders();
var replicated = BuildDatabase(Config(apiKey: "cdc-test-key"));
Assert.Equal(1, RowVersionCount(replicated, "site_events"));
// At the floor, so the seeded version loses to any genuine remote write of the same key.
Assert.Equal(0L, RowVersionHlc(replicated, "site_events"));
}
[Fact] [Fact]
public void UnreplicatedNode_StillRunsTheLegacyMigrator() public void UnreplicatedNode_StillRunsTheLegacyMigrator()
{ {
@@ -149,6 +224,49 @@ public class SiteLocalDbCdcRegistrationTests : IDisposable
} }
} }
/// <summary>
/// Closes every database built so far and releases the pooled connections, so the NEXT
/// <see cref="BuildDatabase"/> on the same file opens it fresh — which is what a node
/// restarting with different replication configuration actually does.
/// </summary>
private void DisposeProviders()
{
foreach (var provider in _providers) provider.Dispose();
_providers.Clear();
Microsoft.Data.Sqlite.SqliteConnection.ClearAllPools();
}
private static void InsertSiteEvent(ILocalDb db, string id)
{
using var connection = db.CreateConnection();
using var cmd = connection.CreateCommand();
cmd.CommandText =
"""
INSERT INTO site_events (id, timestamp, event_type, severity, source, message)
VALUES ($id, '2026-08-14T00:00:00.0000000Z', 'Test', 'Info', 'cdc-test', 'row');
""";
cmd.Parameters.AddWithValue("$id", id);
cmd.ExecuteNonQuery();
}
private static long OplogRowCount(ILocalDb db) => Scalar(db, "SELECT COUNT(*) FROM __localdb_oplog");
private static long SiteEventCount(ILocalDb db) => Scalar(db, "SELECT COUNT(*) FROM site_events");
private static long RowVersionCount(ILocalDb db, string table) => Scalar(
db, $"SELECT COUNT(*) FROM __localdb_row_version WHERE table_name = '{table}'");
private static long RowVersionHlc(ILocalDb db, string table) => Scalar(
db, $"SELECT MAX(hlc) FROM __localdb_row_version WHERE table_name = '{table}'");
private static long Scalar(ILocalDb db, string sql)
{
using var connection = db.CreateConnection();
using var cmd = connection.CreateCommand();
cmd.CommandText = sql;
return Convert.ToInt64(cmd.ExecuteScalar());
}
private ILocalDb BuildDatabase(IConfiguration config) private ILocalDb BuildDatabase(IConfiguration config)
{ {
var provider = new ServiceCollection() var provider = new ServiceCollection()