diff --git a/CLAUDE.md b/CLAUDE.md index 7d993e6d..90fe3c5c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -120,6 +120,19 @@ Other peers in the `scadaproj` family (see `scadaproj/CLAUDE.md` for details): ` - DCL write failures returned synchronously to calling script. - Tag path resolution retried periodically for devices still booting. - Static attribute writes persisted to local SQLite (survive restart/failover, reset on redeployment). +- **Consolidated site database (LocalDb Phase 1 + 2, complete 2026-07-20).** **Ten** tables now live in ONE `ZB.MOM.WW.LocalDb`-managed SQLite file — the Phase 1 pair (`OperationTracking`, `site_events`) plus Phase 2's `sf_messages` and the seven site config tables (`deployed_configurations`, `static_attribute_overrides`, `shared_scripts`, `external_systems`, `database_connections`, `data_connection_definitions`, `native_alarm_state`), configured by the **required** `LocalDb:Path` (`/app/data/site-localdb.db` on the rig; validated with `ValidateOnStart`, so a site config missing it fails to boot). Both are `RegisterReplicated` tables. Consequences worth knowing: + - `site_events.id` changed from autoincrement INTEGER to an application-minted **GUID**. Last-writer-wins keys on the primary key, so two nodes independently minting `id=1,2,3…` would destroy each other's events rather than merge them. The event-log read path uses a composite `(timestamp, id)` keyset cursor with an **opaque string** continuation token; `EventLogEntry.Id` and both `ContinuationToken`s are `string`/`string?` on the site↔central Akka DTOs. + - `ScadaBridge:OperationTracking:ConnectionString`, `ScadaBridge:SiteEventLog:DatabasePath` and — as of Phase 2 — `ScadaBridge:StoreAndForward:SqliteDbPath` + `ScadaBridge:Database:SiteDbPath` are all **migration-only** — nothing reads them but `SiteLocalDbLegacyMigrator`, which copies a pre-Phase-1 file in once (deterministic `mig-{NodeName}-{legacyId}` event ids, `INSERT OR IGNORE`, runs AFTER `RegisterReplicated` so migrated rows replicate) and renames it `.migrated`. Delete the keys once a node has migrated. + - This incidentally fixes a data-loss bug: both legacy databases defaulted to CWD-relative paths **outside** the mounted volume and were discarded on every container recreate. + - **Replication is default-OFF and opt-in** via `LocalDb:Replication:PeerAddress` + a matching `ApiKey` on both nodes. `LocalDbSyncAuthInterceptor` is **fail-closed**: no configured key means no sync stream is accepted at all, so a key typo does not degrade to unauthenticated replication — the pair simply stops converging. The sync endpoint shares the existing site gRPC h2c listener (8083); no new port. Rig posture: **site-a replicated, site-b/site-c deliberately not**, so both states are proven side-by-side. Status surfaces on the site health report as `LocalDbReplicationConnected` / `LocalDbOplogBacklog` (both nullable — null means "no data", NOT "disconnected with an empty backlog") and as `localdb_*` Prometheus series. Note `ZbTelemetryOptions.Meters` is an **allowlist** (`SiteServiceRegistration.ObservedMeters`); an unlisted meter exports nothing, silently. + - Design: **scadaproj** `docs/plans/2026-07-19-scadabridge-localdb-design.md` (that doc lives in the umbrella repo, not here); Phase 1 plan + Phase 2 gate are here under `docs/plans/`. **Phase 2 is COMPLETE** (branch `feat/localdb-phase2`, live gate PASS 2026-07-20 — all 10 checks, evidence in `docs/plans/2026-07-19-localdb-phase2-live-gate.md`). It moved the config tables + `sf_messages` in and **deleted** `SiteReplicationActor`, its `ReplicationMessages`, StoreAndForward's `ReplicationService`, and `StoreAndForwardStorage.ReplaceAllAsync`. What those did, and why nothing replaced them: + + - `SiteReplicationActor` pushed config deploys to the peer and ran a **notify-and-fetch** exchange (tell the standby a deploy happened; it HTTP-fetches the config itself, with retries and a superseded-404 path). Config rows now simply replicate — **the standby makes no fetch at all** during a deploy (verified live). `SiteReconciliationActor` still fetches at node STARTUP when central reports gaps; that path survives and is a different thing. + - `ReplicationService` fanned each buffer mutation (add/remove/park/requeue) to the standby by hand. CDC triggers on `sf_messages` do it now. + - `ReplaceAllAsync` was a destructive delete-all-then-insert-all resync. It was not merely unused after the cutover but **unsafe to keep**: a mass DELETE on a now-replicated table would be captured and shipped to the peer. LocalDb's snapshot resync merges per row under LWW and never deletes, which is also why the old N1 directional-authority guard is gone — there is no wipe left to gate. + - **`notification_lists` and `smtp_configurations` are created but deliberately NOT registered.** They are permanently empty on a site (no writer since 2026-07-10, the migrator skips them, the active-node purge keeps them empty), and registering them would open a standing replication channel whose only historical payload was plaintext SMTP passwords. Pinned by a security-named test, and verified live: those two tables have **no CDC triggers** on either rig node. + - **Operational constraints (read before upgrading a site pair):** stop and start both nodes TOGETHER — rolling one at a time is no longer supported, since the legacy `SfBufferSnapshot` compatibility handler went with the replicator. And a node offline longer than `LocalDb:Replication:TombstoneRetention` (default 7 days) can resurrect deleted rows on rejoin. See `docs/deployment/topology-guide.md`. + - `LocalDb:Replication:MaxBatchSize` batches by ROW COUNT, not bytes, against a 4 MB gRPC cap — the rig pins it to **16** (~70 KB worst-case `config_json` x 16 ~= 1.1 MB). The 500 default would allow ~35 MB. - All timestamps are UTC throughout the system. - Inter-cluster communication uses two transports: ClusterClient for command/control (deployments, lifecycle, subscribe/unsubscribe handshake, snapshots) and gRPC server-streaming for real-time data (attribute values, alarm states). Both CentralCommunicationActor and SiteCommunicationActor registered with receptionist. Central creates one ClusterClient per site using NodeA/NodeB as contact points. Sites configure multiple central contact points for failover. Addresses cached in CentralCommunicationActor, refreshed periodically (60s) and on admin changes. Heartbeats serve health monitoring only. - gRPC streaming channel: SiteStreamGrpcServer on each site node (Kestrel HTTP/2, port 8083); central creates per-site SiteStreamGrpcClient via SiteStreamGrpcClientFactory. Site entity has GrpcNodeAAddress/GrpcNodeBAddress fields. Proto: sitestream.proto with SiteStreamService, SiteStreamEvent (oneof: AttributeValueUpdate, AlarmStateUpdate). DebugStreamEvent message removed (no longer flows through ClusterClient). diff --git a/Directory.Packages.props b/Directory.Packages.props index f3469127..98404995 100644 --- a/Directory.Packages.props +++ b/Directory.Packages.props @@ -15,10 +15,28 @@ - - - - + + + + + + @@ -90,6 +108,9 @@ + + + + + diff --git a/ZB.MOM.WW.ScadaBridge.slnx b/ZB.MOM.WW.ScadaBridge.slnx index 9f3c14d7..21748f21 100644 --- a/ZB.MOM.WW.ScadaBridge.slnx +++ b/ZB.MOM.WW.ScadaBridge.slnx @@ -36,6 +36,7 @@ + diff --git a/docker-env2/site-x-node-a/appsettings.Site.json b/docker-env2/site-x-node-a/appsettings.Site.json index 5c2ada30..b7ddf335 100644 --- a/docker-env2/site-x-node-a/appsettings.Site.json +++ b/docker-env2/site-x-node-a/appsettings.Site.json @@ -20,6 +20,10 @@ "MinNrOfMembers": 1 }, "Database": { + // Migration-only as of LocalDb Phase 2. The site config tables now live in the + // consolidated LocalDb database (LocalDb:Path). SiteDbPath is read once at boot to drain + // a pre-Phase-2 scadabridge.db, and is unused after that - keep it until this node has + // started once. "SiteDbPath": "/app/data/scadabridge.db" }, "DataConnection": { @@ -29,8 +33,11 @@ "SeedReadTimeout": "00:00:30" }, "StoreAndForward": { - "SqliteDbPath": "/app/data/store-and-forward.db", - "ReplicationEnabled": true + // Migration-only as of LocalDb Phase 2. The store-and-forward buffer now lives in the + // consolidated LocalDb database (LocalDb:Path) as the replicated sf_messages table. + // SqliteDbPath is read once at boot by SiteLocalDbLegacyMigrator to drain a pre-Phase-2 + // file, and is unused after that - keep it until this node has started once. + "SqliteDbPath": "/app/data/store-and-forward.db" }, "Communication": { "CentralContactPoints": [ @@ -56,5 +63,13 @@ "Logging": { "MinimumLevel": "Information" } + }, + // Consolidated site database (LocalDb Phase 1): OperationTracking + site_events. + // On the mounted /app/data volume so it survives container recreate - unlike the + // legacy site-tracking.db / site_events.db, which defaulted to CWD-relative paths + // outside the volume and were lost on every recreate. + // Replication is opt-in and configured separately; absent = local-only. + "LocalDb": { + "Path": "/app/data/site-localdb.db" } } diff --git a/docker-env2/site-x-node-b/appsettings.Site.json b/docker-env2/site-x-node-b/appsettings.Site.json index b53344a1..63abb3be 100644 --- a/docker-env2/site-x-node-b/appsettings.Site.json +++ b/docker-env2/site-x-node-b/appsettings.Site.json @@ -20,6 +20,10 @@ "MinNrOfMembers": 1 }, "Database": { + // Migration-only as of LocalDb Phase 2. The site config tables now live in the + // consolidated LocalDb database (LocalDb:Path). SiteDbPath is read once at boot to drain + // a pre-Phase-2 scadabridge.db, and is unused after that - keep it until this node has + // started once. "SiteDbPath": "/app/data/scadabridge.db" }, "DataConnection": { @@ -29,8 +33,11 @@ "SeedReadTimeout": "00:00:30" }, "StoreAndForward": { - "SqliteDbPath": "/app/data/store-and-forward.db", - "ReplicationEnabled": true + // Migration-only as of LocalDb Phase 2. The store-and-forward buffer now lives in the + // consolidated LocalDb database (LocalDb:Path) as the replicated sf_messages table. + // SqliteDbPath is read once at boot by SiteLocalDbLegacyMigrator to drain a pre-Phase-2 + // file, and is unused after that - keep it until this node has started once. + "SqliteDbPath": "/app/data/store-and-forward.db" }, "Communication": { "CentralContactPoints": [ @@ -56,5 +63,13 @@ "Logging": { "MinimumLevel": "Information" } + }, + // Consolidated site database (LocalDb Phase 1): OperationTracking + site_events. + // On the mounted /app/data volume so it survives container recreate - unlike the + // legacy site-tracking.db / site_events.db, which defaulted to CWD-relative paths + // outside the volume and were lost on every recreate. + // Replication is opt-in and configured separately; absent = local-only. + "LocalDb": { + "Path": "/app/data/site-localdb.db" } } diff --git a/docker/seed-sites.sh b/docker/seed-sites.sh index fa5c7102..2d2bc8eb 100755 --- a/docker/seed-sites.sh +++ b/docker/seed-sites.sh @@ -96,7 +96,7 @@ for ident in site-a site-b site-c; do done echo "" -echo "Seeding LDAP group mappings (Design + Deployment)..." +echo "Seeding LDAP group mappings (Designer + Deployer)..." # SecurityConfiguration.HasData declares 4 mappings but the InitialSchema # migration only inserts the Admin row, so a fresh ScadaBridgeConfig starts # with multi-role getting Admin only -- no Design and no Deployment access. @@ -106,11 +106,16 @@ docker exec -i scadabridge-mssql /opt/mssql-tools18/bin/sqlcmd \ -d ScadaBridgeConfig -Q " SET IDENTITY_INSERT LdapGroupMappings ON; IF NOT EXISTS (SELECT 1 FROM LdapGroupMappings WHERE Id = 2) - INSERT INTO LdapGroupMappings (Id, LdapGroupName, Role) VALUES (2, 'SCADA-Designers', 'Design'); + INSERT INTO LdapGroupMappings (Id, LdapGroupName, Role) VALUES (2, 'SCADA-Designers', 'Designer'); IF NOT EXISTS (SELECT 1 FROM LdapGroupMappings WHERE Id = 3) - INSERT INTO LdapGroupMappings (Id, LdapGroupName, Role) VALUES (3, 'SCADA-Deploy-All', 'Deployment'); + INSERT INTO LdapGroupMappings (Id, LdapGroupName, Role) VALUES (3, 'SCADA-Deploy-All', 'Deployer'); IF NOT EXISTS (SELECT 1 FROM LdapGroupMappings WHERE Id = 4) - INSERT INTO LdapGroupMappings (Id, LdapGroupName, Role) VALUES (4, 'SCADA-Deploy-SiteA', 'Deployment'); + INSERT INTO LdapGroupMappings (Id, LdapGroupName, Role) VALUES (4, 'SCADA-Deploy-SiteA', 'Deployer'); +-- Role strings MUST match the canonical vocabulary in +-- src/ZB.MOM.WW.ScadaBridge.Security/Roles.cs ('Designer' / 'Deployer'). +-- These rows previously carried the pre-rename 'Design' / 'Deployment', which +-- authorized nothing: every Designer/Deployer-gated management command failed +-- UNAUTHORIZED on a freshly reseeded rig. SET IDENTITY_INSERT LdapGroupMappings OFF; " diff --git a/docker/site-a-node-a/appsettings.Site.json b/docker/site-a-node-a/appsettings.Site.json index 905ff385..3e54acb0 100644 --- a/docker/site-a-node-a/appsettings.Site.json +++ b/docker/site-a-node-a/appsettings.Site.json @@ -21,6 +21,10 @@ "MinNrOfMembers": 1 }, "Database": { + // Migration-only as of LocalDb Phase 2. The site config tables now live in the + // consolidated LocalDb database (LocalDb:Path). SiteDbPath is read once at boot to drain + // a pre-Phase-2 scadabridge.db, and is unused after that - keep it until this node has + // started once. "SiteDbPath": "/app/data/scadabridge.db" }, "DataConnection": { @@ -30,8 +34,11 @@ "SeedReadTimeout": "00:00:30" }, "StoreAndForward": { - "SqliteDbPath": "/app/data/store-and-forward.db", - "ReplicationEnabled": true + // Migration-only as of LocalDb Phase 2. The store-and-forward buffer now lives in the + // consolidated LocalDb database (LocalDb:Path) as the replicated sf_messages table. + // SqliteDbPath is read once at boot by SiteLocalDbLegacyMigrator to drain a pre-Phase-2 + // file, and is unused after that - keep it until this node has started once. + "SqliteDbPath": "/app/data/store-and-forward.db" }, "Communication": { "CentralContactPoints": [ @@ -57,5 +64,47 @@ "Logging": { "MinimumLevel": "Information" } + }, + // Consolidated site database (LocalDb Phase 1): OperationTracking + site_events. + // On the mounted /app/data volume so it survives container recreate - unlike the + // legacy site-tracking.db / site_events.db, which defaulted to CWD-relative paths + // outside the volume and were lost on every recreate. + // Replication is opt-in and configured separately; absent = local-only. + "LocalDb": { + "Path": "/app/data/site-localdb.db", + // Site-a is the rig's REPLICATED pair; site-b and site-c deliberately stay + // unreplicated so the default-OFF posture is proven side-by-side on one rig. + // + // This node is the initiator: it dials the peer. Replication is still + // bidirectional - the passive node's writes flow back over the same stream - + // so only one side needs PeerAddress. Port 8083 is the existing site gRPC + // listener (h2c); the sync endpoint shares it, no new port. + // + // The ApiKey must be IDENTICAL on both nodes: the host's + // LocalDbSyncAuthInterceptor is fail-closed, so a mismatch (or a missing key) + // rejects every sync stream. A plain dev key here matches the rig's existing + // posture; PRODUCTION uses a "${secret:...}" reference resolved by the + // pre-host secret expander. + "Replication": { + "PeerAddress": "http://scadabridge-site-a-b:8083", + "ApiKey": "dev-site-a-localdb-sync-key", + // ---- Phase 2 sizing, from the Task 1 rig soak (not from the defaults) ---- + // + // MaxBatchSize (default 500) is a ROW count, not a byte budget, so the batch + // size in bytes is set by the widest replicated column. That is + // deployed_configurations.config_json: ~721 B on this rig, but up to ~60-70 KB + // in production (measured, Task 1) - and 70 KB x 500 is ~35 MB against gRPC's + // 4 MB default receive limit. 16 keeps a worst-case batch near 1.1 MB. + "MaxBatchSize": 16, + // Backlog caps bound the oplog while the peer is offline. Exceeding them is + // NOT data loss: the oplog is pruned to the ceiling and needs_snapshot is set, + // so the peer catches up by snapshot resync instead of incrementally. That + // makes tighter-than-default correct here - it trades a rare full resync for a + // bounded file. Sized from the soak's 0.80 sf_messages rows/sec (the only + // non-zero writer measured): ~69k rows/day, so 2 days is ~138k. 250,000 leaves + // room for burst without approaching the 1,000,000 default. + "MaxOplogRows": 250000, + "MaxOplogAge": "2.00:00:00" + } } } diff --git a/docker/site-a-node-b/appsettings.Site.json b/docker/site-a-node-b/appsettings.Site.json index f1db4f2a..376d2b44 100644 --- a/docker/site-a-node-b/appsettings.Site.json +++ b/docker/site-a-node-b/appsettings.Site.json @@ -21,6 +21,10 @@ "MinNrOfMembers": 1 }, "Database": { + // Migration-only as of LocalDb Phase 2. The site config tables now live in the + // consolidated LocalDb database (LocalDb:Path). SiteDbPath is read once at boot to drain + // a pre-Phase-2 scadabridge.db, and is unused after that - keep it until this node has + // started once. "SiteDbPath": "/app/data/scadabridge.db" }, "DataConnection": { @@ -30,8 +34,11 @@ "SeedReadTimeout": "00:00:30" }, "StoreAndForward": { - "SqliteDbPath": "/app/data/store-and-forward.db", - "ReplicationEnabled": true + // Migration-only as of LocalDb Phase 2. The store-and-forward buffer now lives in the + // consolidated LocalDb database (LocalDb:Path) as the replicated sf_messages table. + // SqliteDbPath is read once at boot by SiteLocalDbLegacyMigrator to drain a pre-Phase-2 + // file, and is unused after that - keep it until this node has started once. + "SqliteDbPath": "/app/data/store-and-forward.db" }, "Communication": { "CentralContactPoints": [ @@ -57,5 +64,40 @@ "Logging": { "MinimumLevel": "Information" } + }, + // Consolidated site database (LocalDb Phase 1): OperationTracking + site_events. + // On the mounted /app/data volume so it survives container recreate - unlike the + // legacy site-tracking.db / site_events.db, which defaulted to CWD-relative paths + // outside the volume and were lost on every recreate. + // Replication is opt-in and configured separately; absent = local-only. + "LocalDb": { + "Path": "/app/data/site-localdb.db", + // The PASSIVE half of site-a's replicated pair: no PeerAddress, so this node's + // sync initiator starts and idles while node-a dials in. Its own writes still + // reach node-a - the stream is bidirectional. + // + // The key MUST match node-a's exactly. LocalDbSyncAuthInterceptor is + // fail-closed, so a typo here does not degrade to unauthenticated replication; + // it rejects every stream and the pair silently stops converging. + "Replication": { + "ApiKey": "dev-site-a-localdb-sync-key", + // ---- Phase 2 sizing, from the Task 1 rig soak (not from the defaults) ---- + // + // MaxBatchSize (default 500) is a ROW count, not a byte budget, so the batch + // size in bytes is set by the widest replicated column. That is + // deployed_configurations.config_json: ~721 B on this rig, but up to ~60-70 KB + // in production (measured, Task 1) - and 70 KB x 500 is ~35 MB against gRPC's + // 4 MB default receive limit. 16 keeps a worst-case batch near 1.1 MB. + "MaxBatchSize": 16, + // Backlog caps bound the oplog while the peer is offline. Exceeding them is + // NOT data loss: the oplog is pruned to the ceiling and needs_snapshot is set, + // so the peer catches up by snapshot resync instead of incrementally. That + // makes tighter-than-default correct here - it trades a rare full resync for a + // bounded file. Sized from the soak's 0.80 sf_messages rows/sec (the only + // non-zero writer measured): ~69k rows/day, so 2 days is ~138k. 250,000 leaves + // room for burst without approaching the 1,000,000 default. + "MaxOplogRows": 250000, + "MaxOplogAge": "2.00:00:00" + } } } diff --git a/docker/site-b-node-a/appsettings.Site.json b/docker/site-b-node-a/appsettings.Site.json index 93018c13..e0351fef 100644 --- a/docker/site-b-node-a/appsettings.Site.json +++ b/docker/site-b-node-a/appsettings.Site.json @@ -21,6 +21,10 @@ "MinNrOfMembers": 1 }, "Database": { + // Migration-only as of LocalDb Phase 2. The site config tables now live in the + // consolidated LocalDb database (LocalDb:Path). SiteDbPath is read once at boot to drain + // a pre-Phase-2 scadabridge.db, and is unused after that - keep it until this node has + // started once. "SiteDbPath": "/app/data/scadabridge.db" }, "DataConnection": { @@ -30,8 +34,11 @@ "SeedReadTimeout": "00:00:30" }, "StoreAndForward": { - "SqliteDbPath": "/app/data/store-and-forward.db", - "ReplicationEnabled": true + // Migration-only as of LocalDb Phase 2. The store-and-forward buffer now lives in the + // consolidated LocalDb database (LocalDb:Path) as the replicated sf_messages table. + // SqliteDbPath is read once at boot by SiteLocalDbLegacyMigrator to drain a pre-Phase-2 + // file, and is unused after that - keep it until this node has started once. + "SqliteDbPath": "/app/data/store-and-forward.db" }, "Communication": { "CentralContactPoints": [ @@ -57,5 +64,13 @@ "Logging": { "MinimumLevel": "Information" } + }, + // Consolidated site database (LocalDb Phase 1): OperationTracking + site_events. + // On the mounted /app/data volume so it survives container recreate - unlike the + // legacy site-tracking.db / site_events.db, which defaulted to CWD-relative paths + // outside the volume and were lost on every recreate. + // Replication is opt-in and configured separately; absent = local-only. + "LocalDb": { + "Path": "/app/data/site-localdb.db" } } diff --git a/docker/site-b-node-b/appsettings.Site.json b/docker/site-b-node-b/appsettings.Site.json index 755b0b0a..92610887 100644 --- a/docker/site-b-node-b/appsettings.Site.json +++ b/docker/site-b-node-b/appsettings.Site.json @@ -21,6 +21,10 @@ "MinNrOfMembers": 1 }, "Database": { + // Migration-only as of LocalDb Phase 2. The site config tables now live in the + // consolidated LocalDb database (LocalDb:Path). SiteDbPath is read once at boot to drain + // a pre-Phase-2 scadabridge.db, and is unused after that - keep it until this node has + // started once. "SiteDbPath": "/app/data/scadabridge.db" }, "DataConnection": { @@ -30,8 +34,11 @@ "SeedReadTimeout": "00:00:30" }, "StoreAndForward": { - "SqliteDbPath": "/app/data/store-and-forward.db", - "ReplicationEnabled": true + // Migration-only as of LocalDb Phase 2. The store-and-forward buffer now lives in the + // consolidated LocalDb database (LocalDb:Path) as the replicated sf_messages table. + // SqliteDbPath is read once at boot by SiteLocalDbLegacyMigrator to drain a pre-Phase-2 + // file, and is unused after that - keep it until this node has started once. + "SqliteDbPath": "/app/data/store-and-forward.db" }, "Communication": { "CentralContactPoints": [ @@ -57,5 +64,13 @@ "Logging": { "MinimumLevel": "Information" } + }, + // Consolidated site database (LocalDb Phase 1): OperationTracking + site_events. + // On the mounted /app/data volume so it survives container recreate - unlike the + // legacy site-tracking.db / site_events.db, which defaulted to CWD-relative paths + // outside the volume and were lost on every recreate. + // Replication is opt-in and configured separately; absent = local-only. + "LocalDb": { + "Path": "/app/data/site-localdb.db" } } diff --git a/docker/site-c-node-a/appsettings.Site.json b/docker/site-c-node-a/appsettings.Site.json index 40d2269f..5cdb0125 100644 --- a/docker/site-c-node-a/appsettings.Site.json +++ b/docker/site-c-node-a/appsettings.Site.json @@ -21,6 +21,10 @@ "MinNrOfMembers": 1 }, "Database": { + // Migration-only as of LocalDb Phase 2. The site config tables now live in the + // consolidated LocalDb database (LocalDb:Path). SiteDbPath is read once at boot to drain + // a pre-Phase-2 scadabridge.db, and is unused after that - keep it until this node has + // started once. "SiteDbPath": "/app/data/scadabridge.db" }, "DataConnection": { @@ -30,8 +34,11 @@ "SeedReadTimeout": "00:00:30" }, "StoreAndForward": { - "SqliteDbPath": "/app/data/store-and-forward.db", - "ReplicationEnabled": true + // Migration-only as of LocalDb Phase 2. The store-and-forward buffer now lives in the + // consolidated LocalDb database (LocalDb:Path) as the replicated sf_messages table. + // SqliteDbPath is read once at boot by SiteLocalDbLegacyMigrator to drain a pre-Phase-2 + // file, and is unused after that - keep it until this node has started once. + "SqliteDbPath": "/app/data/store-and-forward.db" }, "Communication": { "CentralContactPoints": [ @@ -57,5 +64,13 @@ "Logging": { "MinimumLevel": "Information" } + }, + // Consolidated site database (LocalDb Phase 1): OperationTracking + site_events. + // On the mounted /app/data volume so it survives container recreate - unlike the + // legacy site-tracking.db / site_events.db, which defaulted to CWD-relative paths + // outside the volume and were lost on every recreate. + // Replication is opt-in and configured separately; absent = local-only. + "LocalDb": { + "Path": "/app/data/site-localdb.db" } } diff --git a/docker/site-c-node-b/appsettings.Site.json b/docker/site-c-node-b/appsettings.Site.json index 93a6b292..6460c296 100644 --- a/docker/site-c-node-b/appsettings.Site.json +++ b/docker/site-c-node-b/appsettings.Site.json @@ -21,6 +21,10 @@ "MinNrOfMembers": 1 }, "Database": { + // Migration-only as of LocalDb Phase 2. The site config tables now live in the + // consolidated LocalDb database (LocalDb:Path). SiteDbPath is read once at boot to drain + // a pre-Phase-2 scadabridge.db, and is unused after that - keep it until this node has + // started once. "SiteDbPath": "/app/data/scadabridge.db" }, "DataConnection": { @@ -30,8 +34,11 @@ "SeedReadTimeout": "00:00:30" }, "StoreAndForward": { - "SqliteDbPath": "/app/data/store-and-forward.db", - "ReplicationEnabled": true + // Migration-only as of LocalDb Phase 2. The store-and-forward buffer now lives in the + // consolidated LocalDb database (LocalDb:Path) as the replicated sf_messages table. + // SqliteDbPath is read once at boot by SiteLocalDbLegacyMigrator to drain a pre-Phase-2 + // file, and is unused after that - keep it until this node has started once. + "SqliteDbPath": "/app/data/store-and-forward.db" }, "Communication": { "CentralContactPoints": [ @@ -57,5 +64,13 @@ "Logging": { "MinimumLevel": "Information" } + }, + // Consolidated site database (LocalDb Phase 1): OperationTracking + site_events. + // On the mounted /app/data volume so it survives container recreate - unlike the + // legacy site-tracking.db / site_events.db, which defaulted to CWD-relative paths + // outside the volume and were lost on every recreate. + // Replication is opt-in and configured separately; absent = local-only. + "LocalDb": { + "Path": "/app/data/site-localdb.db" } } diff --git a/docs/components/Host.md b/docs/components/Host.md index c8c86447..09f80935 100644 --- a/docs/components/Host.md +++ b/docs/components/Host.md @@ -104,7 +104,6 @@ Before branching on role, `AkkaHostedService.StartAsync` creates one actor uncon `SiteServiceRegistration.Configure` registers the site-only components. `AkkaHostedService.RegisterSiteActorsAsync` creates: - `DeploymentManagerActor` — cluster singleton scoped to `"site-{SiteId}"`. - `SiteCommunicationActor` — registered with `ClusterClientReceptionist`; creates a `ClusterClient` to configured central contact points. -- `SiteReplicationActor` — one per node (not a singleton); handles best-effort S&F replication to the standby. - `EventLogHandlerActor` — cluster singleton scoped to `"site-{SiteId}"`. - `ParkedMessageHandlerActor` — bridges Akka to `StoreAndForwardService`. - `SiteAuditTelemetryActor` — created on a dedicated `audit-telemetry-dispatcher` (2-thread `ForkJoinDispatcher`) so SQLite reads and gRPC pushes never contend with hot-path actors. diff --git a/docs/components/SiteRuntime.md b/docs/components/SiteRuntime.md index a114056c..dcdfd465 100644 --- a/docs/components/SiteRuntime.md +++ b/docs/components/SiteRuntime.md @@ -8,7 +8,7 @@ Site Runtime (#3) operates exclusively on site clusters. Its entry point is the The component code lives in `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/`: -- `Actors/` — `DeploymentManagerActor`, `InstanceActor`, `ScriptActor`, `ScriptExecutionActor`, `AlarmActor`, `AlarmExecutionActor`, `NativeAlarmActor`, `SiteReplicationActor`. +- `Actors/` — `DeploymentManagerActor`, `InstanceActor`, `ScriptActor`, `ScriptExecutionActor`, `AlarmActor`, `AlarmExecutionActor`, `NativeAlarmActor`. - `Scripts/` — `ScriptCompilationService`, `ScriptExecutionScheduler`, `SharedScriptLibrary`, `ScriptRuntimeContext`, `ScopeAccessors`, `TriggerExpressionGlobals`. - `Streaming/` — `SiteStreamManager` (the site-wide Akka broadcast stream). - `Persistence/` — `SiteStorageService` (raw SQLite via `Microsoft.Data.Sqlite`), `SiteStorageInitializer`. @@ -79,7 +79,7 @@ Central sends a `DeployInstanceCommand` carrying a JSON `FlattenedConfiguration` 1. Calls `EnsureDclConnections` to push any new or changed connection definitions to the DCL manager (hash-guarded: unchanged configs are skipped). 2. Calls `CreateInstanceActor`, which does `Context.ActorOf(props, instanceName)`. -3. Runs an off-thread `Task` that calls `SiteStorageService.StoreDeployedConfigAsync`, clears static overrides and native alarm state, and — if `_replicationActor` is non-null (it is optional and null in isolated deployments/tests) — tells `SiteReplicationActor` to push to the peer node. +3. Runs an off-thread `Task` that calls `SiteStorageService.StoreDeployedConfigAsync` and clears static overrides and native alarm state. Nothing is pushed to the peer: as of LocalDb Phase 2 those three tables are replicated, so the writes themselves reach the standby. 4. Pipes back a `DeployPersistenceResult`; only on success does it tell the deployer `DeploymentStatus.Success`. If persistence fails, the optimistically-created actor is stopped and the error is returned to central (`SiteRuntime-005`). For redeployment (instance already running), the existing actor is stopped and watched: @@ -216,7 +216,7 @@ Both `AlarmActor` and `NativeAlarmActor` tell the `InstanceActor` an `AlarmState ### Standby replication -`SiteReplicationActor` runs on every site node (not a singleton). The active node's `DeploymentManagerActor` tells it `ReplicateConfigDeploy`, `ReplicateConfigRemove`, `ReplicateConfigSetEnabled`, `ReplicateArtifacts`, or `ReplicateStoreAndForward`. The replication actor tracks the peer node via Akka cluster membership events and forwards each command to `/user/site-replication` on the peer via `ActorSelection`. Replication is fire-and-forget (no ack wait per design), so a failed write to the standby is logged but does not fail the primary operation. +Config replication has no actor. `SiteReplicationActor` — which received `ReplicateConfigDeploy` / `ReplicateConfigRemove` / `ReplicateConfigSetEnabled` / `ReplicateArtifacts` / `ReplicateStoreAndForward` from the active node's `DeploymentManagerActor`, tracked the peer through cluster membership events, and forwarded each command to `/user/site-replication` via `ActorSelection` — was deleted in LocalDb Phase 2, together with its notify-and-fetch exchange (the standby was told a deploy had happened and then HTTP-fetched the config itself). The site's config tables are now replicated by LocalDb CDC, so a deploy on either node reaches the other as an ordinary row change. `SiteReconciliationActor` still fetches over HTTP at node startup when central reports gaps; that is a different path and it survives. ## Usage diff --git a/docs/components/StoreAndForward.md b/docs/components/StoreAndForward.md index fadc25a2..a770579f 100644 --- a/docs/components/StoreAndForward.md +++ b/docs/components/StoreAndForward.md @@ -4,13 +4,12 @@ The Store-and-Forward Engine buffers site-originated outbound messages when a ta ## Overview -The Store-and-Forward Engine (#6) is a site-only component. The central cluster has no equivalent buffer; it uses the Notification Outbox (#21) instead for its own queued delivery work. Every site node runs one `StoreAndForwardService` instance, backed by a `StoreAndForwardStorage` SQLite store and an optional `ReplicationService` that fans each buffer mutation to the standby. +The Store-and-Forward Engine (#6) is a site-only component. The central cluster has no equivalent buffer; it uses the Notification Outbox (#21) instead for its own queued delivery work. Every site node runs one `StoreAndForwardService` instance, backed by a `StoreAndForwardStorage` store. As of LocalDb Phase 2 that store writes to the consolidated LocalDb database, and the buffer reaches the peer as the replicated `sf_messages` table — the `ReplicationService` that used to fan each mutation to the standby by hand was deleted. The component code lives in `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/`: - `StoreAndForwardService` — the core buffer: enqueue, retry sweep, park/retry/discard, and the `ICachedCallLifecycleObserver` audit hook. - `StoreAndForwardStorage` — the SQLite layer; all reads and writes against `sf_messages`. -- `ReplicationService` — fire-and-forget buffer replication to the standby. - `ParkedMessageHandlerActor` — Akka actor bridge that exposes parked-message query/retry/discard to the `SiteCommunicationActor`. - `NotificationForwarder` — the delivery handler for the `Notification` category; forwards buffered notifications to central via the ClusterClient transport and interprets the ack. - `StoreAndForwardOptions` — options class bound from the `StoreAndForward` configuration section. @@ -129,7 +128,9 @@ else ### Async replication to standby -`ReplicationService` wraps each buffer mutation — add, remove, park, requeue — in a `Task.Run` fire-and-forget. The active node does not wait for standby acknowledgment. The standby applies each `ReplicationOperation` via `ApplyReplicatedOperationAsync`, which calls the same `StoreAndForwardStorage` methods. Replication failures are logged at Debug and discarded; the standby may be slightly behind the active at any moment, producing at-most a few duplicate deliveries or missed retries after a failover — an accepted trade-off for zero added latency on the enqueue path. +Replication is no longer the buffer's own concern. `sf_messages` is registered with LocalDb (`SiteLocalDbSetup.OnReady`), so every insert and status change is captured by a CDC trigger and shipped to the peer on the shared sync stream. The four hand-written operations — add, remove, park, requeue — and the `Task.Run` fan-out that carried them are gone, along with `ApplyReplicatedOperationAsync` and `ReplaceAllAsync`. + +The trade-off is unchanged in shape: replication is still asynchronous, so the peer may be slightly behind at any instant. What changed is the bound. Convergence is now per row under last-writer-wins with HLC-ordered tombstones, so a lagging peer converges rather than diverging, and duplicate delivery after a failover is limited to messages the old primary delivered whose status change had not yet replicated. See `Component-StoreAndForward.md` for the normative statement of that bound. The four `ReplicationOperationType` values are `Add`, `Remove`, `Park`, and `Requeue` (requeue was added to cover the operator-initiated `Parked→Pending` transition so the standby preserves retry intent after failover). diff --git a/docs/deployment/installation-guide.md b/docs/deployment/installation-guide.md index 1df8bf7c..33a1d0d2 100644 --- a/docs/deployment/installation-guide.md +++ b/docs/deployment/installation-guide.md @@ -187,6 +187,14 @@ ALTER ROLE db_owner ADD MEMBER scadabridge_svc; Ensure bidirectional TCP connectivity between all Akka.NET cluster peers. The remoting port (default 8081) must be open in both directions. +## Upgrading a Site Pair + +**Stop both nodes of a site pair, upgrade both, then start both.** Rolling one node at a time is +not supported as of LocalDb Phase 2 — the legacy snapshot-compatibility handler that made a +mixed-version pair converge was deleted with the bespoke replicator, and a mixed pair now diverges +silently. See `docs/deployment/topology-guide.md` for the reasoning and for the related +`TombstoneRetention` bound on how long one node may stay offline. + ## Post-Installation Verification 1. Start the service: `sc.exe start ScadaBridge-Central` diff --git a/docs/deployment/topology-guide.md b/docs/deployment/topology-guide.md index 532e0843..04dd9616 100644 --- a/docs/deployment/topology-guide.md +++ b/docs/deployment/topology-guide.md @@ -150,7 +150,27 @@ Each site has its own two-node cluster: - Same split-brain resolver as central (keep-oldest). - Singleton actors: Site Deployment Manager migrates on failover. - Staggered instance startup: 50ms delay between Instance Actor creation to prevent reconnection storms. -- SQLite persistence: Both nodes access the same SQLite files (or each has its own copy with async replication). +- SQLite persistence: each node owns its own consolidated LocalDb database, kept in step by + asynchronous CDC replication over a gRPC sync stream (LocalDb Phase 1 + 2). The nodes do NOT + share a SQLite file. + +### Site Pair Upgrades — stop and start BOTH nodes together + +**A rolling upgrade of a site pair, one node at a time, is no longer supported.** It worked while +the bespoke replicator kept a legacy `SfBufferSnapshot` compatibility handler so a new standby +could still apply an old active node's monolithic snapshot. LocalDb Phase 2 deleted that handler +along with the replicator, so a mixed-version pair has no common replication path: the two nodes +will run, but they will not converge, and the divergence is silent. + +Stop both nodes of a site pair, upgrade both, then start both. + +**Related bound — do not leave one node of a pair offline for long.** A node absent for longer than +`LocalDb:Replication:TombstoneRetention` (default **7 days**) can **resurrect deleted rows** when +it rejoins: deletes replicate as HLC-ordered tombstones, and once a tombstone is pruned there is +nothing left to suppress the stale row the returning node still holds. Within the retention window +a rejoin is safe and self-correcting (verified live: a node stopped and restarted mid-load rejoined +with both nodes byte-identical and zero duplicates). Beyond it, rebuild the returning node's +database from its peer rather than letting it rejoin. ### Central-Site Communication diff --git a/docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md b/docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md index 680273ae..a5c1574b 100644 --- a/docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md +++ b/docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md @@ -10,6 +10,27 @@ Fixed via the **notify-and-fetch** rework (the primary recommendation below), no - **Plan:** [`docs/plans/2026-06-26-deploy-config-notify-and-fetch.md`](../plans/2026-06-26-deploy-config-notify-and-fetch.md) - **Validated:** live docker-cluster smoke — a previously-hanging deploy now completes in ~0.11 s; reconciliation heals single-node and concurrent-both-missing gaps. +## Amendment (2026-07-20) — LocalDb Phase 2 removed the second hop entirely + +The resolution above fixed the intra-site hop by replacing it with notify-and-fetch. LocalDb +Phase 2 then deleted **notify-and-fetch itself**, along with `SiteReplicationActor`: the site's +`deployed_configurations` table is now replicated by CDC, so the config reaches the standby as an +ordinary row change over the gRPC sync stream. There is no intra-site Akka hop carrying config any +more, so the 128 000-byte frame constraint does not apply to it in any form. + +The central→site hop is unchanged — it still sends a small `RefreshDeploymentCommand` and the site +still fetches over HTTP, so that half of the original fix stands. + +**The successor ceiling is different in kind.** The gRPC sync stream has a 4 MB default receive +limit, and LocalDb batches by ROW COUNT (`LocalDb:Replication:MaxBatchSize`, default 500), not by +bytes. A ~70 KB `config_json` — the largest measured in production — times 500 rows is ~35 MB, +which would exceed the limit. The rig therefore pins `MaxBatchSize` to **16** (~1.1 MB worst case). +Any deployment replicating wide rows must size that key deliberately; see the Phase 2 plan (D6) and +`docs/plans/2026-07-19-localdb-phase2-live-gate.md`. + +Note the failure mode differs from the one documented below: an oversized gRPC message is +**rejected**, not silently dropped. + The diagnosis below is retained as the historical record of how the bug was found and reasoned about. ## Summary diff --git a/docs/known-issues/2026-07-20-cached-telemetry-drain-hot-loop.md b/docs/known-issues/2026-07-20-cached-telemetry-drain-hot-loop.md new file mode 100644 index 00000000..d44037a5 --- /dev/null +++ b/docs/known-issues/2026-07-20-cached-telemetry-drain-hot-loop.md @@ -0,0 +1,92 @@ +# Cached-telemetry drain hot-loops forever on a row whose tracking snapshot is gone + +**Date:** 2026-07-20 · **Status:** OPEN · **Severity:** Medium (log flood + wasted I/O; no data loss) +· **Area:** AuditLog / Site Telemetry + +## Summary + +`SiteAuditTelemetryActor`'s cached-telemetry drain reads Pending audit rows, looks up each row's +tracking snapshot by `CorrelationId`, and pushes the combined packet to central. When the lookup +returns `null` the row is **skipped and deliberately left Pending** +(`SiteAuditTelemetryActor.cs:307`), on the reasoning that "central reconciliation will pick it up". + +Nothing ever removes such a row from the local drain queue. The next tick re-reads it, fails the +same lookup, logs the same warning, and leaves it Pending again — **forever**. With a batch of +unresolvable rows the actor spins at its non-idle rate and emits one warning per row per pass. + +Measured on the docker rig: **~2 800 warnings/minute, sustained**, surviving both a process restart +and a container restart, until the audit database itself was discarded. + +``` +[09:57:41 WRN] [Site/scadabridge-site-a-a] Cached-telemetry drain: no tracking snapshot for + a5392796-291f-4f5b-9fbf-5817c1ec76c7 (TrackedOperationId 59bd4bf8-…); skipping. +``` + +## Why the rows became unresolvable + +Two independent stores must agree: + +- the **audit** rows live in `auditlog.db` (site-local, and on the docker rig **inside the + container at `/app/auditlog.db`**, not on the bind-mounted data volume); +- the **tracking** rows live in `OperationTracking`, which LocalDb Phase 1 moved into the + consolidated `LocalDb:Path` database (bind-mounted). + +Anything that resets one without the other strands every audit row that referenced it. The code +comment already anticipates the cause — *"possible if the audit row is older than the tracking +retention window, or the tracking store was reset"* — so this is a known-and-accepted input, not an +exotic one. + +**Two realistic production triggers, neither requiring operator error:** + +1. **Tracking retention expiry.** If the tracking retention window elapses before the audit drain + catches up — a long central outage, a large backlog — the snapshots are pruned out from under + still-Pending audit rows and every one of them becomes a permanent hot-loop entry. +2. **Restoring or resetting one store independently of the other**, e.g. rebuilding a node's + LocalDb file from its peer while its container-local `auditlog.db` survives. + +It was hit here by (2): the two site-a LocalDb databases were dropped during a rig cleanup while +`auditlog.db` — being inside the container — survived. + +## Why the current handling is not enough + +Skipping the row is correct; **leaving it Pending with no other state change is not**. The row is +now in a state it can never leave: + +- no attempt counter, so an unresolvable row is indistinguishable from a transiently-failing one; +- no backoff, so the actor runs at full non-idle rate against a queue that can never shrink; +- no terminal state, so it is retried for the life of the database; +- one Warning per row per pass, which buries every other log line on the node. + +The "central reconciliation will pick it up" comment is about the **audit half** reaching central by +another path. That may well be true — but it does not release the row from the local drain queue, +which is what actually loops. + +## Suggested fix + +Give an unresolvable row somewhere to go. Roughly, in increasing order of effort: + +1. **Bound the retries.** Add an attempt count; past a threshold mark the row terminal + (`TrackingUnavailable`) and stop re-reading it. Emit a single summary Warning with the count + rather than one per row per pass. +2. **Rate-limit the warning** to one per drain episode regardless of row count — the same pattern + `MaintenanceBackgroundService` already uses for the oplog caps-exceeded warning (`_snapshotFlagWarned`). +3. **Push the audit half alone** when the tracking snapshot is missing, rather than skipping the row + entirely, so the row can be marked emitted and leave the queue. Needs a decision on whether + central accepts a packet with no tracking half. + +(2) alone would remove the operational damage; (1) or (3) is needed to stop the wasted I/O. + +## Reproduction + +1. Run a site node until it has cached-call audit rows with tracking correlations. +2. Stop the node; delete its consolidated LocalDb database (which holds `OperationTracking`); + leave `auditlog.db` in place. +3. Start the node. The drain warning repeats indefinitely; the rate does not decay. + +## Notes + +- **No data loss.** The audit rows are intact and still reach central by the reconciliation path; + what is broken is the local drain's ability to ever finish. +- Discovered while cleaning the rig after the LocalDb Phase 2 live gate + (`docs/plans/2026-07-19-localdb-phase2-live-gate.md`), which is also where the related + "deleting an instance orphans its buffered messages" observation is recorded. diff --git a/docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md b/docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md new file mode 100644 index 00000000..6be51122 --- /dev/null +++ b/docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md @@ -0,0 +1,475 @@ +# LocalDb throws `SQLite Error 10: 'disk I/O error'` on the active site node under sustained write load + +**Date:** 2026-07-20 · **Status:** ROOT-CAUSED + FIX PASS COMPLETE 2026-07-20 (same day) — observer-induced, **not a LocalDb defect**; see §0 (cause) and §11a (fixes) · **Severity:** was High; resolved to an operational rule (now enforced in the tooling docs) + shipped hardening +**Area:** `ZB.MOM.WW.LocalDb` (library, `~/Desktop/scadaproj/ZB.MOM.WW.LocalDb/`) as consumed by ScadaBridge site nodes +**Found by:** the Phase 2 rig soak — [`docs/plans/2026-07-19-localdb-phase2-soak.md`](../plans/2026-07-19-localdb-phase2-soak.md) +**Branch:** `feat/localdb-phase2` (symptom observed on Phase 1 code) + +> **Update 2026-07-20:** the mechanism has been identified and reproduced on demand, both in a +> minimal SQLite-only repro and on the live rig, and the follow-up fixes have shipped. +> Sections §0, §11a and §11 below are authoritative; the original brief (§1–§10) is preserved +> as written, with corrections annotated where its conclusions did not survive +> (§4.2, §4.3, §7, §9) and fix notes where they did (§5.1, §8). + +--- + +## 0. ROOT CAUSE (verified) + +**A host-side (macOS) `sqlite3` read of a live, bind-mounted WAL database checkpoints and +resets the WAL out from under the container process, permanently poisoning that process's +connections.** LocalDb, its connection handling, its UDF, and its triggers are not involved — +the mechanism reproduces with plain Python `sqlite3` and no LocalDb code at all. + +Mechanism, step by step: + +1. POSIX advisory locks do **not** propagate across the Docker Desktop virtiofs bind-mount + boundary. A host `sqlite3` opening the database cannot see the container's locks (and vice + versa), so it believes it is the **only** connection. +2. On close, the "last" connection in WAL mode runs a full checkpoint and **resets the WAL to + 0 bytes**. Because the read happened while the container was idle (a standby node, or a gap + between write bursts), nothing blocks the checkpoint. This is exactly the file signature + found on both rig nodes: main DB mtime + 0-byte `-wal` stamped at the sampling minute. +3. The container process still holds the old WAL-index state (its per-inode `-shm` mapping, + kept alive forever by the held-open `_master` connection and the Microsoft.Data.Sqlite + connection pool). That index says the WAL contains N frames; the file now has none. Every + subsequent statement — reads and writes both consult the WAL index — fails with + **`SQLITE_IOERR_SHORT_READ` (extended code 522)**, surfaced as primary code 10 + `'disk I/O error'` (some paths surface `SQLITE_NOTADB` (26) instead). The poisoning is + **permanent until the process reopens the database** (restart). + +### Why the original brief's conclusions were wrong + +- **"It tracks the load, not the node" (§4.1)** — confounded. *Both* nodes were poisoned by + the 04:56 UTC host-side sampling (both nodes' `site-localdb.db` main files carry the 04:56 + mtime; node-b's WAL was left at 0 bytes). A poisoned **standby** shows zero errors only + because a standby issues ~zero LocalDb statements; the errors "followed the load" because + the load is what generates statements against an already-poisoned handle. Node-b's very + first write attempt after failover (04:59:37, `OperationTrackingStore.RecordAttemptAsync`) + failed — it had been poisoned for 3 minutes with nothing to say about it. +- **"It is LocalDb-specific" (§4.2)** — sampling-selection bias. The legacy WAL databases in + the same directory were healthy only because no host process ever read *them*. The minimal + repro poisons an arbitrary WAL database the same way. +- **"The observer has been ruled out" (§4.3)** — the exclusion assumed an error-free standby + was an unpoisoned standby. It wasn't; it was a poisoned node with no traffic. + +### Verification (2026-07-20, all on the live rig + minimal repro) + +1. **Load alone is harmless:** restarted the poisoned active node (site-a-b); freshly-reopened + site-a-a took the full soak load for **10+ minutes with zero errors** (the original model + predicted onset within ~2 min), WAL growing/checkpointing normally, DB 188 KiB → 476 KiB. +2. **One host read is sufficient and immediate:** a single + `sqlite3 docker/site-a-node-a/data/site-localdb.db "SELECT count(*) FROM site_events;"` + against the healthy loaded node reset its 4.6 MiB WAL to 0 bytes in place and produced the + first `disk I/O error` **one second later** (05:32:48 → 05:32:49), 203 errors in the next + 40 s — the same one-second onset correlation as the original 04:56:36 → 04:56:37 incident. +3. **Minimal repro (no LocalDb, no .NET):** a `python:3.12-alpine` container writing a + WAL-mode SQLite DB on a bind mount (held master connection + fresh connection per op, + `synchronous=NORMAL`, `busy_timeout=5000`). A host `sqlite3 "SELECT count(*)"`: + - against the **actively-writing** DB → immediate `SQLITE_IOERR_SHORT_READ` (522) + + `SQLITE_NOTADB` burst, then recovery (checkpoint could not fully reset a hot WAL); + - during an **idle window** (connections held open, WAL populated) → WAL reset + 1.2 MiB → 0 bytes, then **every fresh-connection write failed for the rest of the run + (200/200)** — the persistent variant, matching the rig. + +### Consequences + +1. **The operational rule in §5.3 ("do not query the databases with host-side `sqlite3`") is + the root cause, not a hygiene note.** One violation silently destroys the node's local + persistence until restart. This applies to *every* WAL SQLite file on the bind mount + (`scadabridge.db`, `store-and-forward.db` included), not just LocalDb. +2. **Safe inspection recipes:** copy the file triplet (`.db`, `-wal`, `-shm`) and open the + copy; or read from inside the container boundary (same kernel ⇒ locks visible), e.g. + `docker run --rm -v :/d alpine/sqlite3 sqlite3 /d/site-localdb.db "..."` — never the + macOS host against live files. +3. **LocalDb Phase 2 is unblocked** on this issue: the library sustained the full soak write + load indefinitely once nothing external touched its file. +4. Hardening follow-ups — **status as of the 2026-07-20 fix pass (see §11a):** + - **DONE — extended-code logging:** the LocalDb-adjacent catch sites (`SiteAuditTelemetryActor`, + `CachedCallTelemetryForwarder`, `SiteEventLogger`) now log + `SqliteException` primary/extended codes (`sqlite 10/522`-style) via + `SqliteErrorCodes.Describe` / `DescribeSqliteError`. + - **DONE — §8 async-context bug** (see §8). + - **DONE — §9.5 load regression test** (see §9). + - **NOT DONE (deliberately):** a detect-and-reopen self-heal in `SqliteLocalDb` for + persistent `SQLITE_IOERR`/`SQLITE_NOTADB`. This is a real library design change + (pool clear + master reopen + in-flight coordination) protecting against *external + interference only* — the trigger is operator/tooling action, now prevented at the + source, and on same-kernel production deployments external readers see the locks and + are safe. File as its own issue if production ever runs where a foreign-kernel reader + can touch the files. + +--- + +> **Original brief follows, preserved as written on 2026-07-20 before root-causing.** + +--- + +## 1. Summary + +On a ScadaBridge site node, once the node is **active** and under sustained concurrent write +load, effectively every write to the consolidated LocalDb database (`site-localdb.db`) fails +with: + +``` +Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 10: 'disk I/O error'. +``` + +Observed rate: **~1 000–1 500 failures per minute**, sustained, not transient. The node stays up +and reports healthy. Ordinary (non-LocalDb) SQLite databases in the same directory, in the same +process, under the same load, are completely unaffected. + +## 2. Why it matters + +1. **Silent data loss today.** `SiteEventLogger` fails its inserts and logs + `[ERR] Failed to record event: script from ScriptActor:…`. Site event logging is dropping + events on the floor on the active node whenever the site is busy. `OperationTracking` writes + fail too, which breaks cached-call status tracking (`Cached-telemetry drain: no tracking + snapshot for …; skipping`). +2. **It blocks LocalDb Phase 2.** Phase 2 registers eight further tables into this same + database — including `native_alarm_state` (highest-volume table on the node) and + `sf_messages` — **and deletes the bespoke mechanisms that currently carry that data** + (`SiteReplicationActor`, `StoreAndForward.ReplicationService`) in the same commit. Cutting + over onto this store while removing the fallback would convert a logging defect into config + and buffer loss. +3. Phase 1 was previously live-gated as PASS. That gate exercised correctness and convergence, + **not sustained write load** — which is why this was not caught. + +## 3. Exact symptom + +Two representative stacks, both from `docker logs scadabridge-site-a-b` while that node was +active and under load: + +``` +Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 10: 'disk I/O error'. + at Microsoft.Data.Sqlite.SqliteDataReader.NextResult() + at Microsoft.Data.Sqlite.SqliteCommand.ExecuteReader(CommandBehavior behavior) + at ZB.MOM.WW.ScadaBridge.SiteEventLogging.SiteEventLogger.<>c__DisplayClass15_0.b__0(SqliteConnection connection) + in /src/src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs:line 236 + at ZB.MOM.WW.ScadaBridge.SiteEventLogging.SiteEventLogger.ProcessWriteQueueAsync() + in /src/src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs:line 221 +``` + +``` +Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 10: 'disk I/O error'. + at Microsoft.Data.Sqlite.SqliteDataReader.NextResult() + at Microsoft.Data.Sqlite.SqliteCommand.ExecuteNonQuery() + at ZB.MOM.WW.ScadaBridge.SiteRuntime.Tracking.OperationTrackingStore.RecordEnqueueAsync(...) + in /src/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Tracking/OperationTrackingStore.cs:line 137 + at ZB.MOM.WW.ScadaBridge.AuditLog.Site.Telemetry.CachedCallTelemetryForwarder.TryEmitTrackingAsync(...) + in /src/src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/Telemetry/CachedCallTelemetryForwarder.cs:line 148 +``` + +Note both fail inside `SqliteDataReader.NextResult()` — i.e. at statement execution, not at +`Open()`. Connections are being acquired successfully; the failure is on the write itself. + +### Error-source distribution + +Error-stack frames counted over one 3-minute window on the loaded node: + +| Store | Backing file | Frames | +|---|---|---| +| `OperationTrackingStore` | `site-localdb.db` (**LocalDb**) | 13 044 | +| `SiteAuditTelemetryActor` | `site-localdb.db` (**LocalDb**) | 4 350 | +| `SiteEventLogger` | `site-localdb.db` (**LocalDb**) | 900 | +| `CachedCallTelemetryForwarder` | `site-localdb.db` (**LocalDb**) | 162 | +| `StoreAndForwardStorage` | `store-and-forward.db` (legacy) | **0** | +| `SiteStorageService` | `scadabridge.db` (legacy) | **0** | + +## 4. Evidence — what has been established + +### 4.1 It tracks the load, not the node + +The load was moved between the two site-a nodes by restarting the active one (the surviving node +becomes oldest-up and takes over): + +| Node | Role | Under load | `disk I/O error` / 4 min | +|---|---|---|---| +| site-a-a | active | yes | 2 175 | +| site-a-a | standby (after restart) | no | **0** | +| site-a-b | standby | no | **0** | +| site-a-b | active (after failover) | yes | **4 391** | + +### 4.2 It is LocalDb-specific, not the filesystem or the bind mount — **WRONG, see §0** + +> **Correction 2026-07-20:** sampling-selection bias — only the LocalDb file was ever read +> from the host. Any of these WAL databases is equally poisonable (minimal-repro-proven). + +This is the strongest signal. `store-and-forward.db` and `scadabridge.db` live in the **same +bind-mounted directory** (`/app/data`, host `docker/site-a-node-*/data/`), are opened by the +**same process**, are also **WAL-mode**, and are being written **concurrently under the same +load** — and they log zero errors. Only the LocalDb-managed file fails. + +### 4.3 The observer has been ruled out — **WRONG, see §0: the observer was the cause** + +> **Correction 2026-07-20:** the exclusion below assumed an error-free standby was an +> unpoisoned standby. Node-b's files carry the 04:56 sampling-time mtimes (WAL left at +> 0 bytes); it was poisoned then and merely silent until failover gave it write traffic. + +Onset (04:56:37) was **one second after** a host-side `sqlite3` read of the bind-mounted +database (04:56:36), making observer-induced `-shm` corruption the leading hypothesis. It is +excluded: + +- After node-a was restarted (fresh open, `-shm` recovered) and load failed over to node-b, + **node-b** — whose files no host process had touched since a single baseline read, and which + had been error-free for the entire preceding period — began erroring immediately at a *higher* + rate. +- **node-a**, whose files *had* been sampled, dropped to zero once it stopped carrying load. + +The variable that tracks the errors is load. (Host-side `sqlite3` against a live WAL database +over a bind mount is still unsafe and should be avoided — it is just not the cause here.) + +### 4.4 Not disk pressure + +Host had 215 GiB free throughout (`df -h`: 76 % used on the data volume). Files are small: +`site-localdb.db` 188 KiB, WAL peaked around 4.1 MiB then checkpointed to 0. + +## 5. Reproduction + +Fully reproducible in ~10 minutes on the local docker rig. + +### 5.1 Rig prerequisites + +Two rig-tooling bugs will block a fresh reseed; both are documented in the soak findings: + +- `docker/seed-sites.sh` role names — **already fixed** (commit `cf46e596`). +- **`infra/mssql/setup.sql` never executes** — **FIXED 2026-07-20**: `infra/reseed.sh` now + applies the three init scripts itself via `sqlcmd` once MSSQL accepts connections (the + `/docker-entrypoint-initdb.d/` compose mounts are informational only — the official + `mcr.microsoft.com/mssql/server` image does not implement that hook; noted in the compose + file). The manual workaround below is retained for historical context / older checkouts. + Original problem: after `infra/reseed.sh` dropped the volume, nothing created + `ScadaBridgeConfig` or the `scadabridge_app` login and `reseed.sh` hung forever on its + setup.sql poll. The by-hand equivalent: + + ```bash + cd ~/Desktop/ScadaBridge/infra + for f in mssql/setup.sql mssql/machinedata_seed.sql mssql/setup-env2.sql; do + docker exec -i scadabridge-mssql /opt/mssql-tools18/bin/sqlcmd \ + -S localhost -U sa -P 'ScadaBridge_Dev1#' -C -b < "$f" + done + ``` + Then restart the app containers so EF migrations run, and restart central again after + `seed-sites.sh` writes `LdapGroupMappings` (they are cached at startup). + +### 5.2 Build the load generator + +**The seeded `Motor Controller` template (id 4) cannot be used** — it fails pre-deployment +validation with 34 errors (30 `ConnectionBinding`, 4 `ScriptCompilation`). Build a minimal one. + +**Critical:** `ExternalSystem.Call` does **not** buffer to store-and-forward in practice. +`ExternalSystem.CachedCall` is the buffering surface. Using `Call` produces HTTP traffic and no +S&F rows, and will not reproduce this. + +```bash +cd ~/Desktop/ScadaBridge +SB=src/ZB.MOM.WW.ScadaBridge.CLI/bin/Debug/net10.0/scadabridge # dotnet build src/...CLI first +AUTH="--url http://localhost:9000 --username multi-role --password password" + +# 1. Point the seeded external system at a refusing address (discard port). +$SB $AUTH external-system update --id 1 --name "Test REST API" \ + --endpoint-url "http://127.0.0.1:9" --auth-type ApiKey --auth-config "scadabridge-test-key-1" + +# 2. Minimal template: no attributes, no compositions, no connection bindings. +$SB $AUTH --format json template create --name "SoakGenerator" # -> note the id + +$SB $AUTH --format json template script add --template-id --name "SoakCall" \ + --trigger-type Interval --trigger-config '{"intervalMs":5000}' \ + --code 'var parms = new Dictionary { ["a"] = 2, ["b"] = 3 }; await ExternalSystem.CachedCall("Test REST API", "Add", parms);' + +# 3. Four instances on site-a (site id 1), then deploy each. +for i in 1 2 3 4; do + $SB $AUTH --format json instance create --name "soakgen-$i" --template-id --site-id 1 +done +$SB $AUTH instance deploy --id +``` + +Note the CLI's `template script update` requires `--name` and `--trigger-type` even when only +changing `--code`. In zsh, do not put the auth flags in an unquoted variable — zsh does not +word-split, so pass them literally or use `${=AUTH}`. + +### 5.3 Observe + +```bash +# Identify the ACTIVE node — it is the one running the ScriptActors. +docker logs --since 4m scadabridge-site-a-a 2>&1 | grep -c "Connection refused" +docker logs --since 4m scadabridge-site-a-b 2>&1 | grep -c "Connection refused" + +# Errors appear on that node within ~2 minutes of load starting. +docker logs --since 4m scadabridge-site-a- 2>&1 | grep -c "disk I/O error" +``` + +Metrics (port 8084 is **not** published, and the `aspnet:10.0` image has **no `curl`**) — use a +sidecar in the container's network namespace: + +```bash +docker run --rm --network container:scadabridge-site-a-a curlimages/curl:latest \ + -s localhost:8084/metrics | grep '^localdb_' +``` + +Do **not** query the databases with host-side `sqlite3` while containers are writing them. + +## 6. Code map + +### Library — `~/Desktop/scadaproj/ZB.MOM.WW.LocalDb/src/ZB.MOM.WW.LocalDb/Internal/SqliteLocalDb.cs` + +Facts relevant to the failure: + +- **A `_master` connection is held open for the object's entire lifetime** (`:31`), explicitly to + "anchor the WAL journal". It is guarded by a `Lock _masterLock` because `SqliteConnection` is + not thread-safe. +- **`CreateConnection()` (`:83`) opens a brand-new `SqliteConnection` per call** — one per + operation, from many concurrent actors. Every call then runs + `PRAGMA synchronous=…; PRAGMA busy_timeout=…; PRAGMA foreign_keys=ON;` and registers a UDF: + ```csharp + conn.CreateFunction("zb_hlc_next", () => _clock.Next()); + ``` +- The connection string is **only** `DataSource=` (`:57`) — **connection pooling is left at + the Microsoft.Data.Sqlite default (enabled)**, and no `Cache=` or `Mode=` is set. +- Effective options on the rig are the defaults: `BusyTimeoutMs = 5000`, `Synchronous = NORMAL`. + ScadaBridge's rig config (`docker/site-a-node-*/appsettings.Site.json`, `LocalDb` section) sets + only `Path` and the replication block. +- `zb_hlc_next()` is invoked **from inside the capture triggers**, i.e. on the SQLite thread + during every INSERT/UPDATE/DELETE on a registered table, and it calls into the shared + `HybridLogicalClock` from arbitrary threads. + +### Failing call sites (ScadaBridge) + +- `src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs:221,236` — a channel-drained + single-writer loop (`ProcessWriteQueueAsync`) using a `WithConnection(...)` helper. +- `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Tracking/OperationTrackingStore.cs:137` (`RecordEnqueueAsync`), + `:260,266` (`GetStatusAsync`). +- `src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/Telemetry/CachedCallTelemetryForwarder.cs:148`. +- `src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/Telemetry/SiteAuditTelemetryActor.cs` — also see §8. + +## 7. Hypotheses, ranked + +> **Resolution 2026-07-20:** none of the four below is the cause. The mechanism is a variant +> of #2's territory (bind-mount `-shm`/WAL fragility) but triggered *only* by a host-side +> reader — LocalDb's concurrency, pooling, and UDF (hypotheses 1/3/4) are exonerated. The +> extended code, since captured, is `SQLITE_IOERR_SHORT_READ` (522). + +None verified. Ordered by how well they fit "LocalDb only, load-dependent, same directory as +healthy WAL databases". + +1. **Connection churn × pooling × per-connection UDF registration.** LocalDb opens a fresh + `SqliteConnection` per operation with pooling enabled, and calls `CreateFunction` on every + acquisition. Under high concurrency this drives far more open/close and `-shm` mapping churn + than the legacy stores (which reuse a small number of connections), and is the clearest + structural difference between the failing and healthy databases. Suspect the interaction of + the pool with the long-lived `_master` connection and WAL index growth. +2. **`-shm` / WAL-index growth over the bind mount, triggered only at LocalDb's concurrency.** + Would explain why the same mount is fine for lower-concurrency databases. `mmap` of the shared + WAL index across virtiofs is a known-fragile area. **Distinguishing test: run the same load + with `LocalDb:Path` pointed at a container-local path (a `tmpfs` or a plain volume rather than + the bind mount).** If the errors vanish, this is confirmed and the fix is environmental / + deployment-shaped rather than a library bug. **Run this test first — it is cheap and it + partitions the hypothesis space.** +3. **`zb_hlc_next` UDF failing inside a trigger.** An exception thrown out of the managed UDF + callback during trigger execution can surface as a generic SQLite error at the statement + level. Check `HybridLogicalClock.Next()` for thread-safety and for anything that can throw + under contention (e.g. a spin/overflow path when many callers request stamps in the same + millisecond). +4. **Busy-timeout exhaustion misreported.** `BusyTimeoutMs = 5000` with heavy multi-connection + write contention on one file. This would normally surface as `SQLITE_BUSY` (5), not + `SQLITE_IOERR` (10), so it is a weaker fit — but worth excluding. + +### The single highest-value next step + +**Capture the extended result code.** The logs only show the primary code (`10` = `SQLITE_IOERR`), +which is generic. `SqliteException.SqliteExtendedErrorCode` names the failing syscall and would +likely settle this outright: + +| Extended code | Meaning | Points at | +|---|---|---| +| `SQLITE_IOERR_SHMMAP` (6154) / `SQLITE_IOERR_SHMSIZE` (4874) | WAL index mmap/resize failed | hypothesis 2 | +| `SQLITE_IOERR_WRITE` (778) / `SQLITE_IOERR_FSYNC` (1034) | plain write/fsync failed | filesystem | +| `SQLITE_IOERR_LOCK` (3850) | file locking failed | bind mount locking | + +Add the extended code to the exception logging (or attach a debugger / run the repro against a +local non-container build) before pursuing any fix. + +## 8. Secondary defect in the same path — **FIXED 2026-07-20** + +``` +[ERROR][akka://scadabridge/user/site-audit-telemetry] There is no active ActorContext, +this is most likely due to use of async operations from within this actor. +Cause: System.NotSupportedException +``` + +`SiteAuditTelemetryActor` is touching `Context` (or `Self`/`Sender`) after an `await`. This is a +real bug independent of the I/O errors, though it sits in the same write path and may be +contributing. Note the family-wide rule already recorded for Akka work: never read `Self`/`Context` +after an `await` inside an actor. + +> **Fixed 2026-07-20.** Root cause: both drain handlers await with `ConfigureAwait(false)`, so +> their `finally`-block re-arm (`ScheduleNext`/`ScheduleNextCached`) runs on a pool thread with +> no active ActorContext. Investigation found the failure is **bimodal**, and the second mode is +> worse than the logged one: depending on what the pool thread's thread-static cell slot holds, +> `Context`/`Self` either **throw** `NotSupportedException` (the logged variant — actor crashes +> and restarts once per drain) or **silently resolve a STALE cell of whatever actor last ran on +> that thread**, re-arming the tick at the *wrong actor* so the drain loop just stops (observed +> under TestKit: the tick landed on the TestActor). Fix: capture `Context.System.Scheduler` and +> `Self` into fields at construction (both are thread-safe immutable handles) and use only those +> from the re-arm path. Regression test +> `SiteAuditTelemetryActorTests.Drains_Whose_Awaits_Complete_Off_The_Actor_Thread_Keep_Draining_Without_Crashing` +> forces the mock awaits to complete off the actor thread — which every pre-existing test +> avoided by returning already-completed tasks — and catches **both** variants (EventFilter for +> the throw, sustained-drain counts for the silent stall). AuditLog suite 355/355 green. + +## 9. What a fix must satisfy + +> **Resolution 2026-07-20:** criteria 1–4 are already satisfied by the unmodified code once no +> host process touches the live files — verified 10+ min of soak load with zero errors, events +> durably written, WAL checkpointing normally. Criterion 5 (a sustained concurrent-write load +> test) would **not** have caught this — the trigger is an external reader, not load — but is +> now in place anyway: `ConcurrentWriteLoadTests` in `ZB.MOM.WW.LocalDb.Tests` (8 concurrent +> writers × 250 inserts through fresh pooled connections against a registered/triggered table on +> a real file, with concurrent readers; asserts zero failures + exact row/oplog counts; suite +> 145/145). §8's `SiteAuditTelemetryActor` async-context bug is **fixed** — see §8. + +1. The §5 repro runs for **30 minutes under sustained load with zero `disk I/O error`** on the + active node. +2. No `Failed to record event` errors — site events are durably written under load. +3. `localdb_oplog_depth` rises under load and **drains** between bursts; zero dead letters. +4. Replication still converges across the site-a pair (Phase 1's existing convergence suite and + live gate still pass). +5. A regression test that would have caught this — i.e. a **concurrent-write load test** against + a real LocalDb file, not just the correctness/convergence tests Phase 1 shipped. Phase 1's + gate passed precisely because no test applied sustained concurrent write pressure. + +## 10. Rig state as left + +- Rig fully reseeded; central config volume dropped and replayed; site SQLite state wiped + (`reseed.sh` stage 2 does `rm -rf docker/site-*/data/*`). +- `ExternalSystemDefinitions` id 1 is **still repointed to `http://127.0.0.1:9`** — restore to + `http://scadabridge-restapi:5200` when done. +- Template `SoakGenerator` (id 2021) and instances `soakgen-1..4` (ids 5–8) are **still deployed + and still generating load** on site-a. +- `LdapGroupMappings` corrected in the live DB to the canonical `Designer`/`Deployer` names. + +## 11a. Fix pass (2026-07-20, same day — all verified) + +Everything actionable that this incident identified is now fixed (uncommitted on each repo's +current branch; ScadaBridge full solution builds clean, 0 warnings): + +| # | Issue | Fix | Verification | +|---|---|---|---| +| 1 | §8 `SiteAuditTelemetryActor` async-context bug (bimodal: crash-per-drain OR silent tick misroute) | Capture `Context.System.Scheduler` + `Self` at construction; re-arm path never reads thread-static context | New red→green regression test forcing off-actor-thread continuations; AuditLog suite 355/355 | +| 2 | Diagnostics gap: logs carried only the primary SQLite code | `SqliteErrorCodes.Describe` (AuditLog) + `DescribeSqliteError` (SiteEventLogging) — catch sites now log `sqlite /` | Builds clean; suites green (this gap cost the investigation a from-scratch repro to learn code 522) | +| 3 | §5.1 `reseed.sh` hangs forever waiting on the initdb hook the mssql image doesn't have | `reseed.sh` applies `setup.sql`/`machinedata_seed.sql`/`setup-env2.sql` itself via `sqlcmd`; compose mounts annotated as informational | `bash -n` clean; scripts verified idempotent (`IF NOT EXISTS` guards) | +| 4 | §9.5 missing concurrent-write load test | `ConcurrentWriteLoadTests` in `ZB.MOM.WW.LocalDb.Tests` (scadaproj) — 8 writers × 250 pooled-connection inserts on a registered table + concurrent readers, exact row/oplog count asserts | LocalDb suite 145/145 | +| 5 | Root cause itself (operator/tooling host reads) | Poisonous instructions removed from the Phase 2 plan + `.tasks.json` (safe `snap()` copy-based sampling); soak-doc verdict corrected; family-wide memory rule recorded | On-demand on/off reproduction, §0 | + +Deliberately **not** done: the `SqliteLocalDb` detect-and-reopen self-heal (see §0 +consequence 4 for the rationale and the condition under which to file it). + +## 11. Rig state after root-causing (2026-07-20 ~05:40 UTC) + +- Both site-a nodes restarted during verification, curing both poisonings. End state: + **site-a-b active** carrying the soak load, site-a-a standby, **zero `disk I/O error` on + both** under sustained load. +- The §10 items still stand: `ExternalSystemDefinitions` id 1 still points at + `http://127.0.0.1:9`, and `SoakGenerator` + `soakgen-1..4` are still deployed and + generating load — the Phase 2 soak can now proceed on a clean baseline. +- Minimal-repro scripts (`writer.py` burst variant, `writer2.py` idle-window variant) lived in + the session scratchpad; the recipe is fully described in §0 and takes ~2 minutes to rebuild. diff --git a/docs/plans/2026-07-19-localdb-adoption-phase1.md b/docs/plans/2026-07-19-localdb-adoption-phase1.md index a3a93254..c974d352 100644 --- a/docs/plans/2026-07-19-localdb-adoption-phase1.md +++ b/docs/plans/2026-07-19-localdb-adoption-phase1.md @@ -18,8 +18,8 @@ | 1 | Task 1 (packages) | | 2 | Task 2 (schema helpers) | | 3 | Task 3 (composition root) | -| 4 | Task 4 (tracking store) ∥ Task 5 (event logger) ∥ Task 7 (replication wiring) | -| 5 | Task 6 (migrator) ∥ Task 8 (auth interceptor) ∥ Task 9 (health signal) | +| 4 | Task 4 (tracking store) ∥ Task 5a (event writer) ∥ Task 7 (replication wiring) | +| 5 | Task 5b (event read path) ∥ Task 6 (migrator) ∥ Task 8 (auth interceptor) ∥ Task 9 (health signal) | | 6 | Task 10 (convergence test) ∥ Task 11 (rig config) | | 7 | Task 12 (live gate — serial, on the real rig) | | 8 | Task 13 (docs) ∥ Task 14 (Phase 2 planning) | @@ -181,8 +181,9 @@ services.AddZbLocalDb(config, SiteLocalDbSetup.OnReady); **Files:** - Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Tracking/OperationTrackingStore.cs` -- Modify: wherever `AddSiteRuntime` registers the store (follow `services.AddSiteRuntime` from `SiteServiceRegistration.cs:51` into the SiteRuntime SCE) +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/ServiceCollectionExtensions.cs:74` (the `IOperationTrackingStore` registration; `AddSiteRuntime`'s connection-string parameter becomes vestigial for this store — leave the parameter alone, it still feeds the site config DB) - Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Tracking/` (existing store tests) +- Test: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CompositionRootTests.cs:433` (`Site_Resolves_IOperationTrackingStore` — it overrides the connection string to a temp path because the store opens eagerly in its ctor; that override changes shape once the store takes `ILocalDb`) **Step 1:** Constructor gains `ILocalDb localDb`; `_writeConnection = localDb.CreateConnection()` replaces `new SqliteConnection(_connectionString)` + `Open()` (verified in the lib, `SqliteLocalDb.cs:83-98`: CreateConnection returns an **open** connection with per-connection pragmas — `synchronous`, `busy_timeout`, `foreign_keys=ON` — and the `zb_hlc_next` UDF registered; do NOT call `Open()` on it again). Reader paths that open ad-hoc connections from `_connectionString` switch to `localDb.CreateConnection()` too (reads don't fire triggers, but one connection source is one less config knob). `OperationTrackingOptions.ConnectionString` becomes unused by the store — keep the options class but drop the store's dependency; leave a validator relaxation only if the DI-graph test demands it. @@ -194,25 +195,70 @@ services.AddZbLocalDb(config, SiteLocalDbSetup.OnReady); --- -### Task 5: Rewire SiteEventLogger + EventLogQueryService (GUID ids) +### Task 5a: Rewire SiteEventLogger onto ILocalDb + mint GUID ids **Classification:** standard -**Estimated implement time:** ~5 min +**Estimated implement time:** ~4 min **Parallelizable with:** Task 4, Task 7 +> **Amended 2026-07-19 (execution review).** The original Task 5 said "any `ORDER BY id` +> becomes `ORDER BY timestamp`" and listed 3 files. Code recon found the integer `id` is +> load-bearing in three separate places beyond the writer — a keyset cursor, the +> storage-cap purge, and `long`-typed DTOs that cross the site↔central Akka boundary. +> Task 5 is therefore split: **5a = the writer** (this task, self-contained), **5b = the +> read/purge/contract change** (high-risk, its own task). Do not fold 5b back into 5a. + **Files:** - Modify: `src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs` -- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/EventLogQueryService.cs` - Modify: `src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/ServiceCollectionExtensions.cs` (AddSiteEventLogging registration) -- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/` (existing suite) +- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/` (writer-path tests only) -**Step 1:** `SiteEventLogger` constructor gains `ILocalDb localDb`; its single owned `_connection` becomes `localDb.CreateConnection()`. Keep the `connectionStringOverride` test seam by preferring the override when non-null (tests may still use raw files for non-replicated scenarios) — but note overridden connections lack the HLC UDF, so tests that write must use the ILocalDb path once tables are registered; simplest is to switch the test fixture to a real `ILocalDb` and delete the override parameter if nothing else uses it. +**Step 1:** `SiteEventLogger` constructor gains `ILocalDb localDb`; its single owned `_connection` (`SiteEventLogger.cs:34,62-65`) becomes `localDb.CreateConnection()` — already open, pragma-configured, HLC-UDF-registered; do **not** call `Open()` again. Delete the `connectionStringOverride` seam (`:50`) rather than preserving it: a raw connection lacks `zb_hlc_next()`, so any test writing through the override against a registered table now **fails closed**. Switch the test fixture to a real `ILocalDb` over a temp file. -**Step 2:** The insert statement gains an `id` column bound to `Guid.NewGuid().ToString("N")` (minted in the writer loop per event). `EventLogQueryService` (read-only) switches its connection source the same way; any `ORDER BY id` becomes `ORDER BY timestamp` (GUIDs don't sort chronologically — check the query paths and the UI paging contract). +**Step 2:** Preserve the `internal WithConnection` seam (`:104`, `:123`) — `EventLogQueryService` and `EventLogPurgeService` both depend on it deliberately (their ctors take the concrete `SiteEventLogger`, not the interface). Its body changes connection source only; the signature and locking semantics stay identical, so 5b's consumers keep compiling. -**Step 3:** Fix remaining writer-path tests from Task 2. Full `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests` — PASS. +**Step 3:** The INSERT (`:237-240`) gains an `id` column bound to `Guid.NewGuid().ToString("N")`, minted per event in the writer loop. `ISiteEventLogger.LogEventAsync`'s signature is unchanged — all 18 fire-and-forget call sites are untouched. -**Step 4:** Commit. +**Step 4:** Fix writer-path tests from Task 2. Run `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests` — writer tests PASS. Query/purge tests may be red here; 5b closes them. Note that in the commit message. + +**Step 5:** Commit. + +--- + +### Task 5b: Migrate the event-log read path off integer ids + +**Classification:** high-risk (cross-cluster data contract + silent-data-loss paging bug) +**Estimated implement time:** ~5 min +**Parallelizable with:** Task 6, Task 8, Task 9 +**Blocked by:** Task 5a + +The GUID PK breaks three integer-id assumptions. All three must move together — shipping any one alone is a live bug. + +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/EventLogQueryService.cs` (cursor `:68-71`, `ORDER BY id ASC` `:141`, `GetInt64(0)` `:153`, token mint `:181`) +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/EventLogPurgeService.cs` (storage-cap delete `:162-166`) +- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Messages/RemoteQuery/EventLogQueryResponse.cs` (`EventLogEntry.Id` `long`→`string`; `ContinuationToken` `long?`→`string?`) +- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Messages/RemoteQuery/EventLogQueryRequest.cs` (`ContinuationToken` `long?`→`string?`) +- Modify: `src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs:3127` (passes `null` today — verify it still compiles) +- Modify: `src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/EventLogs.razor:247,268,271` (`_continuationToken` type) +- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/` (query + purge) + +**Step 1: Write the failing tests first.** +- Paging: insert N events with known timestamps, page through with `PageSize` < N, assert **every** row is returned exactly once across pages and in chronological order. This test fails loudly against a naive `id > $afterId` GUID cursor — that is the point. +- Ordering: rows come back oldest-first regardless of GUID values. +- Purge: with the storage cap tripped, the **oldest** rows are deleted, not arbitrary ones. + +**Step 2:** Run — FAIL. + +**Step 3: Implement.** +- **Cursor** → composite keyset on `(timestamp, id)`, since `timestamp` alone is not unique: + `WHERE (timestamp > $afterTs) OR (timestamp = $afterTs AND id > $afterId)`, `ORDER BY timestamp ASC, id ASC`. The continuation token encodes both — use `"{timestamp}|{id}"` as the `string?` token; parse defensively and treat a malformed token as "start from the beginning" rather than throwing. +- **Purge** (`:162-166`) → `ORDER BY timestamp ASC` instead of `ORDER BY id ASC`. (The retention purge at `:128` already filters on `timestamp` and is unaffected.) +- **DTOs** → `Id` and both `ContinuationToken`s become `string`/`string?`; reader uses `GetString(0)`. + +**Step 4: Wire-compat check.** These DTOs cross the site↔central Akka boundary (`SiteCommunicationActor.cs:193` → `CommunicationService.cs:304` → ManagementActor / CentralUI). Both sides ship in the same deployable and the rig redeploys as a unit, so no rolling-upgrade shim is needed — **but state that explicitly in the commit message**, and confirm no serializer manifest/binding pins these types by name+type (grep the Akka serialization setup). If a binding exists, stop and surface it. + +**Step 5:** `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests` + full solution build (0 warnings) — PASS. Commit. --- @@ -228,7 +274,8 @@ services.AddZbLocalDb(config, SiteLocalDbSetup.OnReady); - Test: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbLegacyMigratorTests.cs` Semantics (write the tests first, one per bullet): -- Legacy files located from the OLD config keys (`ScadaBridge:OperationTracking:ConnectionString`, `ScadaBridge:SiteEventLog` DatabasePath); missing file ⇒ no-op. +- Legacy files located from the OLD config keys (`ScadaBridge:OperationTracking:ConnectionString`, `ScadaBridge:SiteEventLog:DatabasePath`); missing file ⇒ no-op. +- **Both keys are unset in every docker appsettings today**, so they fall back to the code defaults `"Data Source=site-tracking.db"` (`OperationTrackingOptions.cs:14`) and `"site_events.db"` (`SiteEventLogOptions.cs:10`) — **CWD-relative**, i.e. `/app/site-tracking.db` and `/app/site_events.db`, NOT `/app/data/`. Resolve relative paths against the process CWD; do not assume the data volume. (Consequence worth knowing: both DBs live outside the mounted volume today and are already lost on every container recreate — so on the rig the migrator will usually find nothing, and that is a legitimate no-op, not a failure. Phase 1 incidentally fixes that data-loss bug by consolidating into `/app/data/site-localdb.db`.) - Runs **after** `RegisterReplicated` so migrated rows enter the oplog and replicate. - Tracking rows copy with `INSERT OR IGNORE` (same TEXT PK ⇒ idempotent). - Legacy `site_events` rows get **deterministic** ids — `"mig-{NodeName}-{legacyId}"` — NOT fresh GUIDs, so a crashed-then-rerun migration cannot duplicate events (`INSERT OR IGNORE` on the deterministic key). NodeName from `ScadaBridge:Node:NodeName`. @@ -312,6 +359,7 @@ Bridge `ISyncStatus` (`Connected`, `OplogBacklog` — **nullable**, null must su **Classification:** high-risk **Estimated implement time:** ~5 min **Parallelizable with:** Task 11 +**Blocked by:** Task 6, Task 8, **Task 5b** (scenario 3 reads events back through the query path) **Files:** - Create: `tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbSitePairConvergenceTests.cs` diff --git a/docs/plans/2026-07-19-localdb-adoption-phase1.md.tasks.json b/docs/plans/2026-07-19-localdb-adoption-phase1.md.tasks.json index 5b857656..89584f89 100644 --- a/docs/plans/2026-07-19-localdb-adoption-phase1.md.tasks.json +++ b/docs/plans/2026-07-19-localdb-adoption-phase1.md.tasks.json @@ -4,23 +4,192 @@ "mode": "parallel-waves", "implementerModel": "opus", "isolation": "worktree", + "branch": "feat/localdb-phase1", "note": "Dispatch every unblocked task concurrently (waves in the plan header). Parallel implementers MUST use worktree isolation - concurrent git in one worktree races destructively." }, - "tasks": [ - {"id": 1, "subject": "Task 1: Add LocalDb packages to the build", "status": "pending"}, - {"id": 2, "subject": "Task 2: Shared schema helpers (tracking + events, GUID PK)", "status": "pending", "blockedBy": [1]}, - {"id": 3, "subject": "Task 3: Register LocalDb in the site composition root", "status": "pending", "blockedBy": [2]}, - {"id": 4, "subject": "Task 4: Rewire OperationTrackingStore onto ILocalDb connections", "status": "pending", "blockedBy": [3]}, - {"id": 5, "subject": "Task 5: Rewire SiteEventLogger + EventLogQueryService (GUID ids)", "status": "pending", "blockedBy": [3]}, - {"id": 6, "subject": "Task 6: One-time legacy data migrator", "status": "pending", "blockedBy": [4, 5]}, - {"id": 7, "subject": "Task 7: Replication registration + sync endpoint (default OFF)", "status": "pending", "blockedBy": [3]}, - {"id": 8, "subject": "Task 8: Bearer auth interceptor for the sync endpoint", "status": "pending", "blockedBy": [7]}, - {"id": 9, "subject": "Task 9: Surface ISyncStatus as site health signal", "status": "pending", "blockedBy": [7]}, - {"id": 10, "subject": "Task 10: Two-node in-process convergence test", "status": "pending", "blockedBy": [6, 8]}, - {"id": 11, "subject": "Task 11: Docker rig enablement (site-a pair)", "status": "pending", "blockedBy": [8]}, - {"id": 12, "subject": "Task 12: Live gate on the docker rig", "status": "pending", "blockedBy": [9, 10, 11]}, - {"id": 13, "subject": "Task 13: Documentation truth pass", "status": "pending", "blockedBy": [12]}, - {"id": 14, "subject": "Task 14: Phase 2 gate - plan the bespoke-replicator migration", "status": "pending", "blockedBy": [12]} + "amendments": [ + { + "date": "2026-07-19", + "by": "execution review", + "change": "Task 5 split into 5a (id 5, writer) and 5b (id 15, read path). Code recon found the integer site_events.id load-bearing in the keyset cursor (EventLogQueryService:70,141,153), the storage-cap purge (EventLogPurgeService:164), and long-typed DTOs crossing the site<->central Akka boundary (EventLogEntry.Id, both ContinuationTokens) - none of which were in the original Task 5 file set. 5b is high-risk. Task 10 gains a blockedBy on 15." + }, + { + "date": "2026-07-19", + "by": "execution", + "change": "Task 2 in isolation left the production writer broken (site_events.id NOT NULL, nothing minting it) - 40 tests red. Per the plan's own 'one PR-sized push' guidance, 5a's GUID minting and all of 5b landed together with Task 2 in commit 727fa48c. Task 5a remains OPEN for its other half: the ILocalDb.CreateConnection() swap, which still needs Task 3." + } ], - "lastUpdated": "2026-07-19" -} + "prerequisites": [ + { + "commit": "4715f2f9", + "what": "gRPC 2.71.0 -> 2.76.0 + Protobuf 3.34.1. Forced by LocalDb.Replication 0.1.0's nuspec floor; restore fails NU1605 below it. Directory.Packages.props had deferred this bump to 'its own reviewed commit' - honored." + }, + { + "commit": "f056b67e", + "what": "AngleSharp GHSA-pgww-w46g-26qg scoped suppression in CentralUI.Tests (supersedes the wrong pin in ecf6b628). Pre-existing: the solution build was RED on clean main, making every task's verification gate meaningless. No patched-and-working version exists upstream - 1.5.x breaks bunit at runtime, and bunit 2.7.2 still resolves vulnerable 1.4.0." + } + ], + "tasks": [ + { + "id": 1, + "subject": "Task 1: Add LocalDb packages to the build", + "status": "completed", + "commit": "7370b331" + }, + { + "id": 2, + "subject": "Task 2: Shared schema helpers (tracking + events, GUID PK)", + "status": "completed", + "blockedBy": [ + 1 + ], + "commit": "727fa48c" + }, + { + "id": 3, + "subject": "Task 3: Register LocalDb in the site composition root", + "status": "completed", + "blockedBy": [ + 2 + ], + "commit": "2977e6c4", + "note": "LocalDb:Path added to 10 configs (8 site nodes incl. docker-env2 site-x, dev template, wonder prod sample) - it is REQUIRED via ValidateOnStart, so any site config missing it fails startup." + }, + { + "id": 4, + "subject": "Task 4: Rewire OperationTrackingStore onto ILocalDb connections", + "status": "completed", + "blockedBy": [ + 3 + ], + "commit": "0b5e9b44", + "note": "Ctor takes ILocalDb, not IOptions; both ad-hoc reader paths switched too and their OpenAsync calls removed (CreateConnection returns an OPEN connection). Three DI fixtures needed LocalDb:Path added because resolving IOperationTrackingStore now forces ILocalDb construction." + }, + { + "id": 5, + "subject": "Task 5a: Rewire SiteEventLogger onto ILocalDb + mint GUID ids", + "status": "completed", + "blockedBy": [ + 3 + ], + "commit": "727fa48c + 0d8a80aa", + "note": "GUID minting landed early with Task 2 (727fa48c) to avoid committing a writer that violated site_events.id NOT NULL. The ILocalDb.CreateConnection() swap is 0d8a80aa. connectionStringOverride DELETED, not preserved: site_events is replicated, so a raw connection lacks zb_hlc_next() and fails closed." + }, + { + "id": 15, + "subject": "Task 5b: Migrate the event-log read path off integer ids", + "status": "completed", + "blockedBy": [ + 5 + ], + "commit": "727fa48c" + }, + { + "id": 6, + "subject": "Task 6: One-time legacy data migrator", + "status": "completed", + "blockedBy": [ + 4, + 5 + ], + "commit": "98b94771", + "note": "Runs AFTER RegisterReplicated (pinned by an oplog assertion). Deterministic mig-{NodeName}-{legacyId} event ids, not fresh GUIDs, so a crash between commit and rename cannot duplicate. DEVIATION: sync transaction on a CreateConnection() connection instead of BeginTransactionAsync - OnReady is a synchronous Action." + }, + { + "id": 7, + "subject": "Task 7: Replication registration + sync endpoint (default OFF)", + "status": "completed", + "blockedBy": [ + 3 + ], + "commit": "a560e9ea", + "note": "DEVIATION: AddZbLocalDbReplication moved from Program.cs (where the plan put it) into SiteServiceRegistration, because the composition-root tests build that graph and not Program.cs - in Program.cs a missing registration would still show green. MapZbLocalDbSync stays in Program.cs (needs the WebApplication). Endpoint is UNAUTHENTICATED until Task 8; no config in this repo sets a peer." + }, + { + "id": 8, + "subject": "Task 8: Bearer auth interceptor for the sync endpoint", + "status": "completed", + "blockedBy": [ + 7 + ], + "commit": "59c69519", + "note": "Fail-closed: no configured ApiKey rejects ALL sync streams. All four handler shapes gated, not just unary (the real RPC is a duplex stream). DEVIATION: Grpc.Core.Testing does not exist on grpc-dotnet; hand-rolled FakeServerCallContext instead of adding the retired native package." + }, + { + "id": 9, + "subject": "Task 9: Surface ISyncStatus as site health signal", + "status": "completed", + "blockedBy": [ + 7 + ], + "commit": "59c69519", + "note": "Additive init properties on SiteHealthReport so the Akka DTO ctor is untouched. Both fields nullable: null = no data, false/0 = a real reading (which is the healthy default-OFF state). Collector stores one tuple, read once, so a torn read cannot pair fresh Connected with stale backlog." + }, + { + "id": 10, + "subject": "Task 10: Two-node in-process convergence test", + "status": "completed", + "blockedBy": [ + 6, + 8, + 15 + ], + "commit": "3c395d79", + "note": "5 scenarios over real loopback gRPC + the real interceptor, initialized via SiteLocalDbSetup.OnReady. Verified NON-vacuous: mismatching the two ApiKeys turns all 5 red with 2m30s of timeouts." + }, + { + "id": 11, + "subject": "Task 11: Docker rig enablement (site-a pair)", + "status": "completed", + "blockedBy": [ + 8 + ], + "commit": "(with Task 11 commit)", + "note": "site-a replicated, site-b/site-c deliberately not, so default-OFF is proven side-by-side on one rig." + }, + { + "id": 12, + "subject": "Task 12: Live gate on the docker rig", + "status": "completed", + "blockedBy": [ + 9, + 10, + 11 + ], + "commit": "95c108f1", + "note": "PASS. Found one REAL defect: ZbTelemetryOptions.Meters is a silent allowlist, so localdb_* was absent from /metrics - the plan's 'flows through automatically' claim was wrong. Fixed + pinned. Evidence: identical row_version/HLC/originating node_id across the pair, 0 dead letters, oplog drained, site failover + rejoin catch-up, localdb_* scraped, site-b inert. NOTE: docker/failover-drill.sh targets CENTRAL nodes and does NOT exercise the site-pair claim; the site flip was run directly." + }, + { + "id": 13, + "subject": "Task 13: Documentation truth pass", + "status": "completed", + "blockedBy": [ + 12 + ], + "commit": "(docs commit)", + "note": "Both CLAUDE.md files + the false 'starts a fresh log' XML doc + the two now-migration-only option properties." + }, + { + "id": 14, + "subject": "Task 14: Phase 2 gate - plan the bespoke-replicator migration", + "status": "completed", + "blockedBy": [ + 12 + ], + "commit": "(gate commit)", + "note": "Gate document only, per the plan - Phase 2 gets its own implementation plan. docs/plans/2026-07-19-localdb-phase2-gate.md. KEY FINDING: the rig SOAK that Phase 2's oplog-cap tuning needs has NOT been run; the live gate proved correctness, not steady-state behaviour." + } + ], + "knownFlakes": [ + { + "test": "SiteRuntime.Tests InstanceActorChildAttributeRaceTests.ChildActors_AreSeededFromAnIsolatedCopy_NotTheLiveAttributesDictionary", + "note": "Intermittent ActorNotFoundException under full-suite load; passes in isolation and in most full runs. Pre-existing, unrelated to LocalDb. Worth its own issue." + }, + { + "test": "AuditLog.Tests ParentExecutionIdCorrelationTests.InboundRoutedRun_AllRoutedRows_CarryInboundExecutionId_AsParentExecutionId", + "note": "Cold-MSSQL-fixture timing: ~91s and AwaitAssert-times-out on a cold fixture, ~1s and passes warm. Baselined against a stashed tree to confirm it is not a LocalDb regression." + } + ], + "lastUpdated": "2026-07-19", + "phase1Status": "COMPLETE - all 15 tasks done, live gate PASS. Branch feat/localdb-phase1 NOT merged/pushed." +} \ No newline at end of file diff --git a/docs/plans/2026-07-19-localdb-adoption-phase2.md b/docs/plans/2026-07-19-localdb-adoption-phase2.md new file mode 100644 index 00000000..9198c8b6 --- /dev/null +++ b/docs/plans/2026-07-19-localdb-adoption-phase2.md @@ -0,0 +1,1207 @@ +# ScadaBridge LocalDb Phase 2 Implementation Plan + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task. + +**Goal:** Move `scadabridge.db`'s 9 config tables and StoreAndForward's `sf_messages` into the +consolidated `ZB.MOM.WW.LocalDb` database established by Phase 1, then delete `SiteReplicationActor` +and StoreAndForward's `ReplicationService` outright. + +**Architecture:** Both stores stop owning their own SQLite files and take `ILocalDb` instead, exactly +as `OperationTrackingStore` and `SiteEventLogger` did in Phase 1. Their DDL is extracted into +`*Schema.Apply(SqliteConnection)` helpers that `SiteLocalDbSetup.OnReady` runs before +`RegisterReplicated`. Trigger-based CDC then replicates every row mutation, replacing the bespoke +hand-shipped ops; the library's per-row-LWW snapshot resync replaces the chunked anti-entropy. + +**Tech Stack:** .NET 10, Akka.NET, Microsoft.Data.Sqlite, ZB.MOM.WW.LocalDb 0.1.0, gRPC, xunit. + +**Branch:** `feat/localdb-phase2`, cut from `feat/localdb-phase1`. + +> **Reviewed and corrected 2026-07-19** against the actual code (three verification sweeps over +> SiteRuntime, StoreAndForward, and Host/rig/docs, plus the LocalDb library source). Load-bearing +> corrections: D1 (the guarded write has a second surviving caller — `SiteReconciliationActor` — +> so it is NOT deleted), D3 (the active node already purges — Task 12 is now a pin, not a +> re-home), D6 (new: the 4 MB gRPC message cap × row-count-only batching), Task 1 (the Phase 2 +> tables aren't in the Phase 1 oplog — the soak measures legacy-DB write rates; port 8084; real +> metric names; no `sqlite3` in containers), and Task 14 (do NOT register the two +> notification/SMTP tables). Details inline at each site. + +--- + +## Read this before Task 1 + +Phase 1's record is the prerequisite context. Read, in order: + +1. `docs/plans/2026-07-19-localdb-adoption-phase1.md.tasks.json` — the `amendments`, + `prerequisites` and `knownFlakes` arrays. Several Phase 1 assumptions changed during execution. +2. `docs/plans/2026-07-19-localdb-phase2-gate.md` — the gate. §5's open questions are answered in + the Decisions section below; the gate itself is superseded by this plan. +3. `src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbSetup.cs` — the pattern every new table follows. + +### Two known flakes — do not chase these + +- `SiteRuntime.Tests InstanceActorChildAttributeRaceTests` — intermittent `ActorNotFoundException` + under full-suite load. Passes in isolation. Pre-existing. +- `AuditLog.Tests ParentExecutionIdCorrelationTests` — ~91 s and times out on a **cold** MSSQL + fixture, ~1 s warm. Re-run before investigating. + +--- + +## Decisions — the gate's open questions, answered + +These were resolved from code recon before this plan was written. Each names its evidence. **Task 2 +records them formally; the rest of the plan assumes them.** + +### D1. Config moves to CDC, and notify-and-fetch is deleted, not preserved + +`SiteReplicationActor` never sends config JSON over the replication hop — it sends a deployment id +plus fetch coordinates, and the standby HTTP-fetches from central itself +(`SiteReplicationActorTests` #2, #5). That design exists for exactly one reason: the config blob +exceeds Akka's 128 KB frame limit (`docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md`). + +Under CDC the row itself replicates over the LocalDb gRPC stream, which has no such limit. The +standby therefore never fetches at all. + +The version guard, however, does **not** die with it. `StoreDeployedConfigIfNewerAsync` +(`SiteStorageService.cs:301-336`) guards its upsert with: + +```sql +WHERE excluded.deployed_at > deployed_configurations.deployed_at +``` + +That predicate protects against **a stale fetch landing after a newer one**, and the method has +**two** production callers, not one: `SiteReplicationActor.HandleApplyConfigDeploy` +(`SiteReplicationActor.cs:375` — deleted in Task 15) *and* `SiteReconciliationActor` +(`SiteReconciliationActor.cs:166`) — the per-node startup self-heal that reports local inventory +to central and HTTP-fetches gap items. The reconciliation actor **survives Phase 2**, and its +stale-fetch race (a reconcile fetch completing after a newer deploy landed) still exists. **The +method and its guard stay.** Under CDC the reconcile write simply becomes a second CDC writer of +`deployed_configurations` — benign: a gap fetch only fires when the node genuinely lacks the row, +the content is deterministic per (deployment id, revision), and the guard blocks local downgrades. + +> **Do not** try to reproduce the `deployed_at` guard on top of LWW. They order by different +> clocks (central's deploy time vs. the writing node's HLC) and mixing them produces a +> non-convergent store. + +### D2. `ReplaceAllAsync` is deleted, and the N1 directional guard becomes unnecessary + +`StoreAndForwardStorage.ReplaceAllAsync` (`StoreAndForwardStorage.cs:284-306`) is an unqualified +`DELETE FROM sf_messages` followed by a full re-insert. Its own doc warns "Never call on an active +node — it discards every in-flight row." `SfBufferResyncPredicateTests` exists to enforce that only +a standby ever applies it — the N1 Critical regression test. + +The LocalDb library's snapshot resync **merges per row under LWW and never wipes**: + +- `SnapshotApplier.OnBeginAsync` (`SnapshotStreamer.cs:163-170`) resets counters only. There is no + wholesale `DELETE FROM` or `TRUNCATE` anywhere in the class. +- `OnBatchAsync` (`:172-186`) wraps each snapshot row as an ordinary oplog entry and pushes it + through the same `LwwApplier` as a delta. +- `LwwApplier.cs:69-78` discards the incoming row when the local row's HLC is higher. +- **Row-level deletes DO still replicate** — the delete trigger captures a tombstone + (`TriggerSqlGenerator`), `SnapshotStreamer` streams tombstone row-versions, and `LwwApplier` + applies an incoming tombstone as a real `DELETE`. Only the destructive whole-table replace is + gone. Caveat: tombstones are pruned after `LocalDb:Replication:TombstoneRetention` (default + 7 days) — a node offline longer than that can resurrect deleted rows on rejoin; Task 21 puts + this in the runbook. + +So a node with newer local rows keeps them. The failure the directional guard prevents — a stale +peer wiping a live buffer — is structurally impossible, not merely guarded against. **Port +`SfBufferResyncPredicateTests` as a convergence assertion (both nodes end with the union, newest +per id wins), not as a directional-authority assertion.** + +Consequence to state plainly: the standby is no longer guaranteed byte-identical to the active +node's buffer. It is guaranteed *convergent*. That is a real semantic change and Task 2 records it. + +### D3. The SMTP purge already runs on the active node — pin it, don't move it + +`PurgeCentralOnlyNotificationConfigAsync` (`SiteStorageService.cs:811-821`) deletes +`notification_lists` and `smtp_configurations` — including plaintext SMTP passwords left by +pre-fix deployments. The original recon placed its only caller on the standby's replication path; +**that was wrong.** It has two callers: `DeploymentManagerActor.HandleDeployArtifacts` +(`DeploymentManagerActor.cs:1921`) — the **active** node's artifact-apply — and +`SiteReplicationActor.HandleApplyArtifacts` (`SiteReplicationActor.cs:456`), the standby copy +that dies with the actor. The purge therefore never lapses: the active-node call already exists +and stays. Task 12 shrinks to **pinning that call with a regression test** (it sits inside a +method Task 16 edits, and nothing today fails if the call is dropped). + +Because no site code has written those two tables since 2026-07-10 (writers removed), the +migrator skips them (Task 9), and the active purge keeps them empty, they are **permanently empty +in the consolidated DB** — which is why Task 14 does **not** register them for replication (see +the rationale there). + +### D4. `native_alarm_state` volume is measured, not assumed + +`scadabridge.db` is not only config. `native_alarm_state` mirrors live A&C conditions from +`NativeAlarmActor.cs:504` via batched upserts, and is by a wide margin the highest-volume table in +either database. `sf_messages` worst case is ~50 row-writes/sec (`DefaultMaxRetries = 50`, 10 s +sweep, 4-way target parallelism). Neither number can be turned into an oplog cap from first +principles. + +**Task 1 is a rig soak that measures both.** Its output sets `MaxOplogRows` / `MaxOplogAge` in +Task 19. If the soak shows the shared oplog cannot absorb alarm-storm churn, the escape hatch is +keyed instances — named in the **adoption** design doc +(`~/Desktop/scadaproj/docs/plans/2026-07-19-scadabridge-localdb-design.md:139-141`, "Out of +scope"; not the 07-17 library design doc) — that is a library-level effort and would suspend +this plan, so **Task 1 gates everything after Task 2.** + +### D5. Cutover forecloses rolling site upgrades + +`SiteReplicationActor` retains a legacy monolithic `SfBufferSnapshot` handler specifically for +rolling upgrades. With no dual-mechanism period, a site pair cannot be upgraded one node at a time — +one node would speak a protocol the other no longer implements. **Both nodes of a site must be +stopped and started together.** Task 21 puts this in the deployment runbook. + +### D6. The 4 MB gRPC message cap replaces the 128 KB Akka frame as the size ceiling + +The Akka frame limit that motivated notify-and-fetch is gone, but LocalDb sync has its own +ceiling the plan must respect: **neither side configures gRPC message sizes** — ScadaBridge's +`AddGrpc` (`Program.cs:519-521`) sets only the auth interceptor, and the library's initiator +channel is a bare `GrpcChannel.ForAddress` — so the **4 MB default receive limit** applies in +both directions. Batching is **row-count-only** (`MaxBatchSize`, default 500 — +`SyncSession.cs:227`, `SnapshotStreamer.cs:55`; there is no byte-aware chunking). +`deployed_configurations.config_json` is documented to exceed 128 KB per row; a few dozen such +rows in one batch is over 4 MB, which fails the stream and wedges replication on a poison batch. + +**Task 1 measures the real max/typical `config_json` size** on the rig's legacy DB; **Task 19 +sets `LocalDb:Replication:MaxBatchSize`** so `max-row-bytes × MaxBatchSize` stays comfortably +under 4 MB. If any single row approaches 4 MB, **stop**: that needs a LocalDb library change +(byte-aware batching or configurable message sizes) — a `scadaproj` effort, same class as the +keyed-instances hatch. + +--- + +## Execution waves + +| Wave | Tasks | Note | +|---|---|---| +| 0 | 1, 2 | Soak + decision record. **Task 1 gates the rest.** | +| 1 | 3, 4 | Schema extraction — independent files, dispatch in parallel | +| 2 | 5, 6 | Store rewiring — independent files, dispatch in parallel | +| 3 | 7, 8, 9 | Setup + migrators | +| 4 | 10, 11 | Port test intents as specs — **before** any deletion | +| 5 | 12, 13 | Pin the active-node purge; notify-and-fetch scope check | +| 6 | 14, 15, 16, 17 | The cutover, serial | +| 7 | 18, 19 | Convergence suite + rig config | +| 8 | 20, 21 | Live gate + docs | + +--- + +### Task 1: Rig soak — measure oplog growth under real write rates + +**Classification:** high-risk +**Estimated implement time:** ~40 min wall-clock (mostly waiting) +**Parallelizable with:** none — this gates the plan + +This is the gate item from `2026-07-19-localdb-phase2-gate.md` §5. It is measurement, not code. + +**Files:** +- Create: `docs/plans/2026-07-19-localdb-phase2-soak.md` + +> **Method note (corrected):** on this rig the Phase 2 tables are still in the **legacy** DBs — +> `sf_messages` in `store-and-forward.db`, `native_alarm_state` in `scadabridge.db` — and are +> **not registered** with LocalDb, so driving S&F/alarm churn moves `__localdb_oplog` **not at +> all**. This soak therefore measures (a) **source write rates from the legacy DBs** and derives +> the oplog sizing arithmetically, and (b) that the Phase 1 oplog stays healthy alongside. The +> *empirical* post-cutover drain check lives in Task 20 (evidence item 10). +> +> Also: the containers have **no `sqlite3` binary** (bare `aspnet:10.0` image), and the data dirs +> are host bind mounts — but **NEVER run `sqlite3` on the macOS host against the live files**: +> host↔container POSIX locks do not propagate across the virtiofs boundary, and a single host-side +> read checkpoints + resets the WAL out from under the container, permanently poisoning its +> connections (`SQLITE_IOERR_SHORT_READ` → `disk I/O error` storm until restart). This is the +> **root cause of the 2026-07-20 "disk I/O error under load" incident** — see +> [`docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md`](../known-issues/2026-07-20-localdb-disk-io-error-under-load.md) §0. +> Safe sampling: copy the file triplet first (`cp x.db x.db-wal x.db-shm /tmp/…` — plain `cp` +> takes no SQLite locks and cannot harm the live DB), then query the copy. The helper used below: +> +> ```bash +> snap() { # snap "" — query a throwaway copy, never the live file +> local d; d=$(mktemp -d) +> cp "$1/$2" "$d/" && cp "$1/$2-wal" "$1/$2-shm" "$d/" 2>/dev/null +> sqlite3 "$d/$2" "$3"; rm -rf "$d" +> } +> ``` +> Metrics are on port **8084** inside the container (not published to the host). + +**Step 1: Bring up the rig with Phase 1 replication enabled** + +```bash +cd ~/Desktop/ScadaBridge # already on feat/localdb-phase2 (= phase1 + docs) +bash docker/deploy.sh +``` + +Site-a's pair replicates; site-b/site-c deliberately do not (the default-OFF pin). Confirm: + +```bash +docker exec scadabridge-site-a-a curl -s localhost:8084/metrics | grep '^localdb_' +``` + +Expected: `localdb_*` series present (`localdb_oplog_depth`, `localdb_sync_*` — the meter is +`ZB.MOM.WW.LocalDb.Replication`). If absent, the meter allowlist regressed — see +`SiteServiceRegistration.ObservedMeters`. + +**Step 2: Capture baselines (from throwaway copies — never the live files, see the `snap` helper above)** + +```bash +cd ~/Desktop/ScadaBridge/docker +snap site-a-node-a/data site-localdb.db "SELECT COUNT(*) FROM __localdb_oplog;" +snap site-a-node-a/data store-and-forward.db \ + "SELECT COUNT(*), COALESCE(SUM(retry_count),0) FROM sf_messages;" +snap site-a-node-a/data scadabridge.db "SELECT COUNT(*) FROM native_alarm_state;" +``` + +Record all values and the wall-clock time. (`COUNT + SUM(retry_count)` is the S&F row-write +proxy: every failed attempt is one `UPDATE` that increments `retry_count`.) + +**Step 3: Drive sustained load for 30 minutes** + +Two generators, run concurrently: + +- **Store-and-forward churn.** Point a store-and-forward target at an address that refuses + connections, then enqueue messages. Each failed attempt costs one `UPDATE`. With the default + 10 s sweep this is ~1 write/message/10 s per lane. +- **Alarm churn.** Drive A&C conditions on a deployed instance so `native_alarm_state` upserts + fire. This is the number that actually matters — it is unbounded by design. + +**Step 4: Sample write rates every 5 minutes** + +```bash +cd ~/Desktop/ScadaBridge/docker +for i in $(seq 1 6); do + echo "=== t+$((i*5))m ===" + snap site-a-node-a/data store-and-forward.db \ + "SELECT COUNT(*), COALESCE(SUM(retry_count),0) FROM sf_messages;" + snap site-a-node-a/data scadabridge.db \ + "SELECT COUNT(*), COUNT(CASE WHEN last_transition_at > datetime('now','-5 minutes') THEN 1 END) FROM native_alarm_state;" + snap site-a-node-a/data site-localdb.db "SELECT COUNT(*) FROM __localdb_oplog;" + docker exec scadabridge-site-a-a curl -s localhost:8084/metrics \ + | grep -E '^localdb_(oplog_depth|sync)' + sleep 300 +done +``` + +Per-interval deltas of `COUNT + SUM(retry_count)` give S&F row-writes/sec; the +`last_transition_at`-window count approximates alarm upserts/sec (it undercounts refresh-only +upserts — say so in the findings if alarm volume is near a threshold). + +**Step 5: Measure `config_json` sizes (D6)** + +```bash +snap site-a-node-a/data scadabridge.db \ + "SELECT COUNT(*), MAX(LENGTH(config_json)), AVG(LENGTH(config_json)) FROM deployed_configurations;" +``` + +If the rig's configs are trivially small, also run this against a production-representative +legacy DB (wonder) before trusting the number. + +**Step 6: Record findings** + +Write `docs/plans/2026-07-19-localdb-phase2-soak.md` with: +- rows/sec observed for `sf_messages` and for `native_alarm_state`, separately +- max/avg `config_json` bytes and the derived **`MaxBatchSize`** (`max-row-bytes × MaxBatchSize` + comfortably under 4 MB — D6) +- Phase 1 oplog depth over the run (should stay near zero — it is not driven by this load) +- dead-letter count (must be 0) +- **the recommended `MaxOplogRows` and `MaxOplogAge`**, with the arithmetic + (sustained rows/sec × retention window, with alarm-storm headroom) + +**Step 7: The decision gate** + +If the measured sustained write rate × any reasonable `MaxOplogAge` exceeds any reasonable +`MaxOplogRows` — i.e. the shared oplog cannot absorb this write profile even on paper — **stop. +Do not proceed to Task 3.** Report to the user: the keyed-instances escape hatch +(`~/Desktop/scadaproj/docs/plans/2026-07-19-scadabridge-localdb-design.md:139-141`) is required +first, and that is a library effort in `scadaproj`. Likewise stop if any single `config_json` +approaches 4 MB (D6 — needs byte-aware batching in the library). The empirical +drain-under-churn confirmation happens post-cutover in Task 20; monotonic growth there is the +same stop condition. + +**Step 8: Commit** + +```bash +git add docs/plans/2026-07-19-localdb-phase2-soak.md +git commit -m "docs(localdb): phase 2 rig soak findings — oplog sizing evidence" +``` + +(The `feat/localdb-phase2` branch already exists — it carries this plan — so commit to it; +do **not** `git checkout -b`.) + +--- + +### Task 2: Decision record + +**Classification:** trivial +**Estimated implement time:** ~3 min +**Parallelizable with:** none (needs Task 1's numbers) + +**Files:** +- Modify: `docs/plans/2026-07-19-localdb-phase2-gate.md` + +Update the gate's status header from `NOT STARTED` to point at this plan, and answer each §5 +question inline with a one-paragraph resolution referencing D1–D5 above plus Task 1's measured +numbers. Do not delete the questions — the reasoning is the value. + +**Commit:** `docs(localdb): close the phase 2 gate — questions answered, plan written` + +--- + +### Task 3: Extract `StoreAndForwardSchema.Apply` + +**Classification:** small +**Estimated implement time:** ~4 min +**Parallelizable with:** Task 4 + +Mirrors `OperationTrackingSchema` exactly. Read that file first — it is the reference. + +**Files:** +- Create: `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardSchema.cs` +- Modify: `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardStorage.cs:32-117` +- Test: `tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardSchemaTests.cs` + +**Step 1: Write the failing test** + +```csharp +[Fact] +public void Apply_IsIdempotent_AndCreatesEveryColumn() +{ + var path = Path.Combine(Path.GetTempPath(), $"sf-schema-{Guid.NewGuid():N}.db"); + try + { + using var connection = new SqliteConnection($"Data Source={path}"); + connection.Open(); + + StoreAndForwardSchema.Apply(connection); + StoreAndForwardSchema.Apply(connection); // second run must not throw + + using var cmd = connection.CreateCommand(); + cmd.CommandText = "SELECT name FROM pragma_table_info('sf_messages')"; + var columns = new List(); + using var reader = cmd.ExecuteReader(); + while (reader.Read()) columns.Add(reader.GetString(0)); + + // 12 original + 4 additive + Assert.Equal(16, columns.Count); + Assert.Contains("last_attempt_at_ms", columns); + Assert.Contains("parent_execution_id", columns); + } + finally + { + SqliteConnection.ClearAllPools(); + File.Delete(path); + } +} +``` + +**Step 2: Run it — expect FAIL** (`StoreAndForwardSchema` does not exist) + +```bash +dotnet test tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests.csproj --filter "FullyQualifiedName~StoreAndForwardSchemaTests" +``` + +**Step 3: Create the schema class** + +Move, verbatim: the DDL literal from `StoreAndForwardStorage.cs:50-68`, the four +`AddColumnIfMissingAsync` calls (`:79-92`) and their helper (`:125-142`), the `last_attempt_at_ms` +backfill (`:98-105`), and the due-index (`:109-114`). + +Convert every `*Async` to sync (`ExecuteNonQuery`, `ExecuteScalar`) — `OnReady` is a synchronous +`Action`. Depend only on `Microsoft.Data.Sqlite`, never on the LocalDb library. + +Carry this comment onto the class, because it is the whole reason the file exists: + +```csharp +/// +/// Deliberately depends only on Microsoft.Data.Sqlite, not on the LocalDb library. +/// The Host applies this DDL to a LocalDb-managed connection before +/// RegisterReplicated installs the capture triggers; nothing about the schema +/// itself is LocalDb-specific, and the store still calls it so a directly-constructed +/// store (tests, tooling) remains self-sufficient. +/// +``` + +**Step 4: Reduce `StoreAndForwardStorage.InitializeAsync` to a call** + +```csharp +public async Task InitializeAsync() +{ + EnsureDatabaseDirectoryExists(); + await using var connection = await OpenConnectionAsync(); + StoreAndForwardSchema.Apply(connection); +} +``` + +**Step 5: Run the S&F suite — expect PASS** + +```bash +dotnet test tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests.csproj +``` + +**Step 6: Commit** + +```bash +git add -A && git commit -m "refactor(sf): extract StoreAndForwardSchema.Apply from the storage class" +``` + +--- + +### Task 4: Extract `SiteStorageSchema.Apply` + +**Classification:** small +**Estimated implement time:** ~5 min +**Parallelizable with:** Task 3 + +**Files:** +- Create: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageSchema.cs` +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs:58-196` +- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/SiteStorageSchemaTests.cs` + +Same shape as Task 3. Move the DDL literal (`:79-158`, all 9 tables), `MigrateSchemaAsync` +(`:167-181`) and `TryAddColumnAsync` (`:183-196`), converting to sync. + +**Do not move** the `PRAGMA journal_mode=WAL` at `:68-76`. LocalDb owns the connection's pragmas. + +**Test assertion:** all 9 tables exist after `Apply`, and a second `Apply` does not throw. Name +every table explicitly — a loop that asserts "9 tables" would pass if one were renamed: + +```csharp +foreach (var table in new[] + { + "deployed_configurations", "static_attribute_overrides", "shared_scripts", + "external_systems", "database_connections", "notification_lists", + "data_connection_definitions", "smtp_configurations", "native_alarm_state", + }) +{ + Assert.True(TableExists(connection, table), $"missing table: {table}"); +} +``` + +**Commit:** `refactor(site): extract SiteStorageSchema.Apply from SiteStorageService` + +--- + +### Task 5: Rewire `StoreAndForwardStorage` onto `ILocalDb` + +**Classification:** high-risk +**Estimated implement time:** ~5 min +**Parallelizable with:** Task 6 + +Follow `OperationTrackingStore` (Phase 1, commit `0b5e9b44`) as the reference — it solved this +exact problem. + +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardStorage.cs:22-26,177-185` +- Modify: `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/ServiceCollectionExtensions.cs:18-25` +- Test: `tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/` (fixtures) + +**Step 1: Replace the constructor** + +```csharp +public StoreAndForwardStorage(ILocalDb localDb, ILogger logger) +{ + ArgumentNullException.ThrowIfNull(localDb); + ArgumentNullException.ThrowIfNull(logger); + _localDb = localDb; + _logger = logger; +} +``` + +**Step 2: Replace `OpenConnectionAsync`** + +```csharp +// CreateConnection returns an ALREADY-OPEN, pragma-configured connection with the +// zb_hlc_next() UDF registered. Calling OpenAsync on it throws, and a raw +// SqliteConnection would lack the UDF, making every capture trigger fail closed. +private SqliteConnection OpenConnection() => _localDb.CreateConnection(); +``` + +Update all 18 call sites (`:36, 238, 258, 286, 321, 378, 412, 449, 482, 503, 548, 572, 589, 619, +638, 659, 677, 694`) from `await using var connection = await OpenConnectionAsync();` to +`await using var connection = OpenConnection();`. + +**Step 3: Delete `EnsureDatabaseDirectoryExists` (`:152-168`)** — LocalDb owns the file. + +**Step 4: Update the DI registration** + +```csharp +services.AddSingleton(sp => new StoreAndForwardStorage( + sp.GetRequiredService(), + sp.GetRequiredService>())); +``` + +**Step 5: Fix the test fixtures** + +Every test constructing `StoreAndForwardStorage` with a connection string now needs an `ILocalDb`. +Phase 1 created `tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/TestLocalDb.cs` for exactly +this — read it and add an equivalent (or reference it) in the S&F test project. It must use a real +temp **file** (never `:memory:` — capture triggers need a durable file) and call +`SqliteConnection.ClearAllPools()` before deleting, including the `-wal` and `-shm` sidecars. + +**Step 6: Full S&F suite — expect PASS** + +```bash +dotnet test tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests.csproj +``` + +**Commit:** `refactor(sf): StoreAndForwardStorage takes ILocalDb instead of a connection string` + +--- + +### Task 6: Rewire `SiteStorageService` onto `ILocalDb` + +**Classification:** high-risk +**Estimated implement time:** ~5 min +**Parallelizable with:** Task 5 + +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs:13-51` +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/ServiceCollectionExtensions.cs:36-42` +- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs:70-71` + +Same shape as Task 5, but note the differences: + +- There are **21** inline `new SqliteConnection(_connectionString)` + `OpenAsync()` pairs, not a + helper (lines 60, 206, 248, 311, 345, 386, 414, 443, 470, 498, 538, 583, 608, 640, 663, 692, + 730, 769, 813, 837, 868 — counted before Task 4, which removes the :60 one inside + `InitializeAsync`). Add a `private SqliteConnection OpenConnection() => _localDb.CreateConnection();` + and convert all of them. +- `public SqliteConnection CreateConnection()` at `:51` is a public escape hatch used by + `SiteExternalSystemRepository` (the only repository consumer). Keep the member; change the body + to `=> _localDb.CreateConnection();`. +- The busy-timeout floor (`BusyTimeoutFloorSeconds`, `:36`) and its `SqliteConnectionStringBuilder` + normalization are **deleted** — LocalDb configures pragmas on every connection it hands out. +- `AddSiteRuntime(string siteDbConnectionString)` loses its parameter. Keep the no-arg overload at + `:23-28` (it already exists) and delete the string one, updating + `SiteServiceRegistration.cs:70-71` to `services.AddSiteRuntime();`. + +**Watch for:** transactions at `:348` (`RemoveDeployedConfigAsync`) and `:540` +(`UpsertNativeAlarmsAsync`) must keep using `connection.BeginTransaction()` on the LocalDb +connection, not `ILocalDb.BeginTransactionAsync()` — mixing the two would put the cascade's three +deletes in different transactions. + +**Step: Run both affected suites** + +```bash +dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.csproj +dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests/ZB.MOM.WW.ScadaBridge.Host.Tests.csproj +``` + +Host.Tests will surface DI fixtures that now need `LocalDb:Path` — Phase 1 hit the identical +failure (tasks.json Task 4 note). Add the config key to each failing fixture. + +**Commit:** `refactor(site): SiteStorageService takes ILocalDb instead of a connection string` + +--- + +### Task 7: Extend `SiteLocalDbSetup` with the new DDL (not yet registered) + +**Classification:** standard +**Estimated implement time:** ~3 min +**Parallelizable with:** none + +Deliberately applies DDL **without** calling `RegisterReplicated`. The tables live in the +consolidated file but are not yet captured, so the bespoke replicator keeps working unchanged and +the tree stays green. Task 14 flips them on and deletes the bespoke path in one commit. + +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbSetup.cs:43-57` +- Test: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbWiringTests.cs` + +**Step 1: Extend `OnReady`** + +```csharp +using (var connection = db.CreateConnection()) +{ + OperationTrackingSchema.Apply(connection); + SiteEventLogSchema.Apply(connection); + SiteStorageSchema.Apply(connection); + StoreAndForwardSchema.Apply(connection); +} + +db.RegisterReplicated("OperationTracking"); +db.RegisterReplicated("site_events"); + +// Phase 2 tables are created here but NOT yet registered — the bespoke replicator +// still owns them until the Task 14 cutover. Registering them now would double- +// replicate every row (harmless, both paths upsert, but it would mask a defect in +// either one). + +SiteLocalDbLegacyMigrator.Migrate(db, config); +``` + +**Step 2: Pin it** + +Add a test asserting all 4 Phase-1-and-2 table sets exist after `OnReady`, and that +`db.ReplicatedTables` contains **exactly** `OperationTracking` and `site_events` — the "not yet" +is the assertion that matters. + +**Commit:** `feat(localdb): create config + sf_messages tables in the consolidated DB` + +--- + +### Task 8: Extend the legacy migrator for `sf_messages` + +**Classification:** high-risk +**Estimated implement time:** ~5 min +**Parallelizable with:** Task 9 + +Read `SiteLocalDbLegacyMigrator.cs` in full first. `MigrateTracking` (`:116-161`) is the closest +reference — like `sf_messages`, it has a native TEXT PK and needs no id synthesis. + +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbLegacyMigrator.cs:57-66` +- Test: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbLegacyMigratorTests.cs` + +**Step 1: Write the failing tests** + +Three, minimum: + +```csharp +[Fact] public void SfMessages_AreCopiedFromTheLegacyFile() { } +[Fact] public void SfMessages_Migration_IsIdempotent_WhenRerunAfterACrashBeforeRename() { } +[Fact] public void MigratedSfMessages_EnterTheOplog_SoTheyActuallyReplicate() { } +``` + +The third is the one that catches the ordering defect. Phase 1's equivalent +(`SiteLocalDbLegacyMigratorTests.cs:244`, `MigratedRows_EnterTheOplog_SoTheyActuallyReplicate`) +asserts on `__localdb_oplog` directly — copy that assertion. **The oplog table is +`__localdb_oplog`** (`LocalDbSchema.cs:19`), not `zb_oplog`. + +> **Harness note:** production `OnReady` does not register `sf_messages` until the Task 14 +> cutover, so these tests must build their own `ILocalDb` (`TestLocalDb`), call +> `RegisterReplicated("sf_messages")` themselves, and then run the migrator — the oplog +> assertion needs live capture triggers. Correspondingly, Task 14 must keep +> `SiteLocalDbLegacyMigrator.Migrate` as the **last** call in `OnReady`, after all +> registrations. + +**Step 2: Add `ResolveStoreAndForwardPath` + `MigrateStoreAndForward`** + +```csharp +private const string DefaultStoreAndForwardPath = "./data/store-and-forward.db"; + +internal static string ResolveStoreAndForwardPath(IConfiguration config) => + config["ScadaBridge:StoreAndForward:SqliteDbPath"] ?? DefaultStoreAndForwardPath; +``` + +Migrate all 16 columns with `INSERT OR IGNORE`. The PK is native TEXT — **no id synthesis**, unlike +`MigrateEvents`. + +**Step 3: Wire into `Migrate`** — add the call alongside the existing two. + +**Commit:** `feat(localdb): migrate legacy store-and-forward.db into the consolidated DB` + +--- + +### Task 9: Extend the legacy migrator for the 9 config tables + +**Classification:** high-risk +**Estimated implement time:** ~5 min +**Parallelizable with:** Task 8 + +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbLegacyMigrator.cs` +- Test: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbLegacyMigratorTests.cs` + +All 9 tables migrate from one legacy file (`ScadaBridge:Database:SiteDbPath`), so this is one +`MigrateSiteStorage` that opens the legacy DB once and copies table by table inside a single +transaction, then renames once. + +**Two tables are deliberately skipped:** `notification_lists` and `smtp_configurations`. They are +purged on every deploy and are permanently empty by design (site write paths were removed +2026-07-10). Migrating them would resurrect plaintext SMTP passwords from a pre-fix legacy file into +a **replicated** table. Add this as a comment and as a test: + +```csharp +[Fact] +public void Migration_DoesNotCopyNotificationOrSmtpRows_EvenWhenTheLegacyFileHasThem() +{ + // These carry plaintext SMTP passwords in pre-2026-07-10 files. They are purged + // on every deploy and must never enter a replicated table. +} +``` + +**Step: Verify all 9 tables' columns match** between `SiteStorageSchema` and the migrator's +`INSERT` lists. A column-count mismatch here fails at runtime on a real deployment and nowhere else. + +**Commit:** `feat(localdb): migrate legacy scadabridge.db config tables into the consolidated DB` + +--- + +### Task 10: Port the S&F replication test intents as CDC specs + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** Task 11 + +**Specifications first, deletion second.** These tests encode behaviour the replacement must still +satisfy. The gate document named two files; the real specification is five. + +**Files:** +- Read (as spec): `tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ReplicationServiceTests.cs` +- Read (as spec): `tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardReplicationTests.cs` +- Create: `tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbStoreAndForwardConvergenceTests.cs` + +Port these **intents** (not the mechanics) as two-node convergence assertions, using Phase 1's +`LocalDbSitePairConvergenceTests.cs` as the harness reference: + +| Intent | Assertion under CDC | +|---|---| +| Add materialises on the peer | enqueue on A → row present on B | +| Remove deletes on the peer | deliver on A → row absent on B | +| Park sets status Parked on the peer | park on A → B's row has status Parked | +| Requeue resets status + RetryCount | requeue on A → B shows Pending, RetryCount 0 | +| Add-then-Remove never converges to present | **the portable form of the strict-ordering test** | +| Apply Add twice is idempotent, newest wins | LWW gives this | +| Park when Add was lost still materialises | upsert semantics | + +**Explicitly do not port:** `ReplicationServiceTests.ReplicationOperations_AreDispatchedInIssueOrder` +(`:169-195`; 200 interleaved ops dispatched synchronously in strict issue order, 400 observed). That asserts the *mechanism* — inline +fire-and-forget dispatch — not the outcome. CDC capture is asynchronous and batched by construction. +The portable intent is row 5 above. Record this in the test file as a comment so a future reader +does not think it was dropped by accident. + +**Commit:** `test(localdb): port store-and-forward replication intents as CDC convergence specs` + +--- + +### Task 11: Port the resync + directional-authority tests + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** Task 10 + +**Files:** +- Read (as spec): `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/SiteReplicationActorTests.cs` +- Read (as spec): `tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/SfBufferResyncPredicateTests.cs` +- Create: `tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbConfigConvergenceTests.cs` + +**`SfBufferResyncPredicateTests` is the most valuable test in the set** — the N1 Critical +regression. Per D2 it must be re-expressed, not ported literally: + +```csharp +[Fact] +public async Task ANodeWithNewerLocalRows_KeepsThem_WhenAPeerSnapshotArrives() +{ + // N1 Critical (was: SfBufferResyncPredicateTests). The bespoke replicator needed a + // directional guard because ReplaceAllAsync was a destructive DELETE-then-INSERT, + // so a wrong-direction resync wiped a live buffer. + // + // LocalDb's snapshot resync merges per row under LWW and never deletes + // (SnapshotApplier has no DELETE; LwwApplier.cs:69-78 discards an incoming row + // whose HLC is lower than the local row's). The failure mode is structurally + // impossible rather than guarded against — so this asserts convergence, NOT + // directional authority. There is no active/standby asymmetry left to enforce. +} +``` + +**Also do not port:** actor tests #1–#5 (notify-and-fetch config fetch/retry/supersede). Per D1 the +fetch path is deleted, so these describe code that will not exist. Replace with a single assertion +that a config deploy on A converges to B **without** B making any HTTP call — assert on a fetcher +test double that records zero invocations. That is the positive proof that notify-and-fetch is gone. + +> **Scope the zero-fetch assertion to the deploy-replication flow.** `SiteReconciliationActor` +> survives Phase 2 and legitimately HTTP-fetches at node startup when central reports gaps (D1). +> The two-node harness has no central, so its pass no-ops there — but do not write the assertion +> as "the standby never fetches, ever"; assert the fetcher double records zero calls **during +> deploy convergence**. + +**Commit:** `test(localdb): port resync + config replication intents as CDC convergence specs` + +--- + +### Task 12: Pin the active-node SMTP purge + +**Classification:** standard +**Estimated implement time:** ~3 min +**Parallelizable with:** none + +Per D3 (corrected): the active node's artifact-apply **already** calls +`PurgeCentralOnlyNotificationConfigAsync` — `DeploymentManagerActor.cs:1921`, inside +`HandleDeployArtifacts` (`:1864-1963`). Nothing is re-homed. What is missing is a **pin**: +Task 16 edits this actor's wiring, and no test today fails if the purge call is dropped +(`ArtifactStorageTests` covers the storage method, not the actor's call site). + +**Files:** +- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs` + +**Step 1: Write the pin test (red-first by commenting out the call locally, then restore)** + +```csharp +[Fact] +public async Task ApplyingArtifacts_PurgesCentralOnlyNotificationConfig() +{ + // Security cleanup: notification_lists and smtp_configurations can hold plaintext + // SMTP passwords from pre-2026-07-10 deployments. The ACTIVE node's artifact-apply + // purges them (DeploymentManagerActor.HandleDeployArtifacts); the standby's copy of + // this call dies with SiteReplicationActor in Task 15. This test pins the active-node + // call so Task 16's actor edits cannot silently drop it. + // Arrange: real temp-file SiteStorageService, seed one row into each table, + // drive HandleDeployArtifacts; assert both tables are empty afterwards. +} +``` + +**Step 2: No production change is expected.** If recon of the call site shows otherwise, stop +and report. + +**Commit:** `test(site): pin the active-node notification-config purge` + +--- + +### Task 13: Notify-and-fetch scope check — the guarded write stays + +**Classification:** standard +**Estimated implement time:** ~3 min +**Parallelizable with:** none + +Per D1 (corrected). The original task deleted `StoreDeployedConfigIfNewerAsync` here — that was +wrong twice over: (a) it has a **second surviving caller**, and (b) deleting it before Task 15 +would not even compile, since `SiteReplicationActor` still calls it until then. + +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs:301-336` (doc comment only) + +**Step 1: Keep `StoreDeployedConfigIfNewerAsync`.** Its callers are +`SiteReplicationActor.HandleApplyConfigDeploy` (`SiteReplicationActor.cs:375` — dies with the +actor in Task 15) **and `SiteReconciliationActor` (`SiteReconciliationActor.cs:166`)** — the +per-node startup self-heal against central, which survives Phase 2 and still needs the +`deployed_at` guard against a stale reconcile fetch. Update the method's doc comment to name +reconciliation as the remaining caller (and drop any mention of standby replication). The +unguarded `StoreDeployedConfigAsync` (`:241-271`) stays the active deploy path. + +**Step 2: `ConfigFetchRetryCount` — answered, but its removal moves to Task 17.** + +The grep was run during plan review: the only production reader is `SiteReplicationActor.cs:157` +(plus its validator rule and tests). `SiteReconciliationActor` does a single best-effort pass and +does not read it; `IDeploymentConfigFetcher` itself is kept (used by `DeploymentManagerActor`'s +refresh path, `SiteReconciliationActor`, and registered at `ServiceCollectionExtensions.cs:84`). +So the option (`SiteRuntimeOptions.cs:68`) and its validator rule +(`SiteRuntimeOptionsValidator.cs:56-58`) become dead **after Task 15 deletes the actor** — remove +them in Task 17 (config-key cleanup), not here, or the build breaks. Re-run +`grep -rn "ConfigFetchRetryCount" src/ tests/` there to confirm. + +**Commit:** `docs(site): StoreDeployedConfigIfNewerAsync serves reconciliation — guard stays under CDC` + +--- + +### Task 14: Register the Phase 2 tables and delete `ReplicationService` + +**Classification:** high-risk +**Estimated implement time:** ~5 min +**Parallelizable with:** none + +**The cutover.** Registration and bespoke-path deletion land in one commit so the two mechanisms +never both run. + +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbSetup.cs` +- Delete: `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/ReplicationService.cs` +- Modify: `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardService.cs` — field `:39`, + ctor param `:243`, and the 6 emission call sites `:654, 805, 836, 872, 1120, 1146` +- Modify: `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/ServiceCollectionExtensions.cs:63-68` + (the `ReplicationService` registration) **and `:32`** (`GetRequiredService()` + inside the `StoreAndForwardService` factory at `:27-61`) +- Modify: `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardStorage.cs:284-306` +- Delete: `tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ReplicationServiceTests.cs` +- Delete: `tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardReplicationTests.cs` +- Delete: `tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ReplicationWireSerializationPinTests.cs` + +**Step 1: Register 8 tables** in `SiteLocalDbSetup.OnReady`, replacing the Task 7 comment: the +7 config tables (`deployed_configurations`, `static_attribute_overrides`, `shared_scripts`, +`external_systems`, `database_connections`, `data_connection_definitions`, `native_alarm_state`) +plus `sf_messages`. **Do NOT register `notification_lists` or `smtp_configurations`** (this +reverses the plan's original instruction): they are permanently empty by design — no site writer +since 2026-07-10, the migrator skips them (Task 9), and the active-node purge +(`DeploymentManagerActor.cs:1921`) keeps them empty — so registering them would create a standing +replication channel whose only historical payload was plaintext SMTP passwords, for zero benefit. +Put that rationale in an `OnReady` comment so a future writer has to make an explicit decision. + +Keep `SiteLocalDbLegacyMigrator.Migrate` as the **last** call in `OnReady`, after all +registrations, so migrated rows enter the oplog through live capture triggers (the Phase 1 +ordering that is silently load-bearing). + +Note the two composite-PK tables are fine: LocalDb's `RegisterReplicated` supports multi-column +PKs (`SqliteLocalDb.RegisterReplicated` orders `pk` ordinals), and no Phase 2 table has a BLOB +column (which would be rejected). + +**Step 2: Delete `ReplicationService`** and remove the ctor parameter (`:243`), the +`_replication` field (`:39`), and all 6 emission call sites from `StoreAndForwardService`. + +**Step 3: Delete `ReplaceAllAsync`** (`:284-306`). Per D2 the library's resync merges; a destructive +delete-all must not survive into a replicated table, where CDC would capture the mass-delete and +ship it to the peer. + +**Step 4: Check `GetAllMessagesAsync`'s `Truncated` flag** — it exists only for the resync path. If +nothing else reads it, simplify the return type. + +**Step 5: `CompositionRootTests.cs:474`** asserts `ReplicationService` is a registered site +singleton. Delete that assertion. + +**Step 6: Full S&F + Host suites — expect PASS** + +**Commit:** `feat(localdb)!: replicate sf_messages via CDC, delete ReplicationService` + +--- + +### Task 15: Delete `SiteReplicationActor` and its messages + +**Classification:** high-risk +**Estimated implement time:** ~4 min +**Parallelizable with:** none + +**Files:** +- Delete: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/SiteReplicationActor.cs` +- Delete: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Messages/ReplicationMessages.cs` +- Delete: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/SiteReplicationActorTests.cs` +- Delete: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/ResyncWireSerializationPinTests.cs` +- Delete: `tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/SfBufferResyncPredicateTests.cs` + +The whole `ReplicationMessages.cs` file becomes dead — it contains exactly the 10 records +`ReplicateConfigDeploy/Remove/SetEnabled`, `ReplicateArtifacts`, `ReplicateStoreAndForward`, +`ApplyConfigDeploy/Remove/SetEnabled`, `ApplyArtifacts`, `ApplyStoreAndForward`, whose only +consumers are `SiteReplicationActor`, the 5 `DeploymentManagerActor` Tell sites (Task 16), the +`AkkaHostedService` wiring (Task 16), and the deleted tests. The four resync records +(`RequestSfBufferResync`, `SfBufferSnapshot`, `SfBufferSnapshotChunk`, `SfBufferResyncAck`) are +**not** in this file — they are declared at the bottom of `SiteReplicationActor.cs` (`:678-707`) +and die with the actor file. + +**Do not delete** `ActiveNodeEvaluator` (`Communication/ClusterState/ActiveNodeEvaluator.cs`) — the +S&F **delivery gate** still needs it (`AkkaHostedService.cs:866` → `SetDeliveryGate` → +`SelfIsPrimary` → `ActiveNodeEvaluator.SelfIsOldestUp`). Its doc comment at `:14` mentions +`SiteReplicationActor` and `:16` mentions the resync authority checks; the actor's own direct call +at `SiteReplicationActor.cs:288` goes away with the file. Update the comment, keep the type. + +Confirm the deletions are complete before committing: + +```bash +grep -rn "SiteReplicationActor\|ReplicateStoreAndForward\|SfBufferSnapshot" src/ tests/ +``` + +Expected: no matches. + +**Commit:** `feat(localdb)!: delete SiteReplicationActor — CDC replaces hand-shipped ops` + +--- + +### Task 16: Clean up `DeploymentManagerActor` and `AkkaHostedService` + +**Classification:** standard +**Estimated implement time:** ~4 min +**Parallelizable with:** none + +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs:55,161-190,794,970,1004,1102,1952` +- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs:770-796,806-812,1056` + +**Step 1:** Remove the `IActorRef? replicationActor = null` ctor parameter (**`:169`** — the ctor +signature spans `:161-175`; `:184` is the field *assignment*), the `_replicationActor` field +(`:55`), and all 5 `_replicationActor?.Tell(...)` sites (`:794, 970, 1004, 1102, 1952`). + +The parameter is optional and positional — removing it **silently shifts every argument after it** +(`healthCollector`, `serviceProvider`, `loggerFactory`, `configFetcher`, +`startupLoadRetryInterval`, `configLoader`). The sharpest hazard is the **same-typed `IActorRef?` +optional immediately before it** — `dclManager` at `:168`: a caller passing positionally could +silently feed the wrong actor ref with zero compile errors. Update the `Props.Create` at +`AkkaHostedService.cs:806-812` (which passes `replicationActor` positionally at `:810`) and every +test constructing this actor — check each call site by hand. + +**Step 2:** Delete `replicationService` / `replicationLogger` resolution (`:770-773`), the actor +creation (`:785-789`), the handler wiring (`:792-796`), and the actor-inventory comment at `:1056`. + +**Step 3:** `activeNodeCheck` (`:781-783`) is still used by `SiteCommunicationActor` (`:816-821`). +Keep it. + +**Step 4:** Full Host + SiteRuntime + IntegrationTests suites. + +**Commit:** `refactor(site): drop the replication actor from the deployment + host wiring` + +--- + +### Task 17: Config-key cleanup + +**Classification:** standard +**Estimated implement time:** ~4 min +**Parallelizable with:** none + +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/StartupValidator.cs:113-115` +- Modify: `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardOptions.cs:12` +- Modify: `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardOptionsValidator.cs:19-21` +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptions.cs:68` + `SiteRuntimeOptionsValidator.cs:56-58` (moved here from Task 13) +- Modify: **10** × `appsettings.Site.json` (Host template + 6 `docker/site-*` + 2 `docker-env2/site-x-*` + + **`deploy/wonder-app-vd03/appsettings.Site.json`** — the wonder single-node install, which the + original count missed; it sets both paths *and* `ReplicationEnabled: false`) + +| Key | Action | +|---|---| +| `ScadaBridge:Database:SiteDbPath` | → migration-only. **Relax the `Require` at `StartupValidator.cs:113`** or every site config keeps it mandatory forever. | +| `ScadaBridge:StoreAndForward:SqliteDbPath` | → migration-only. Relax the non-empty rule at `StoreAndForwardOptionsValidator.cs:19-21`. | +| `ScadaBridge:StoreAndForward:ReplicationEnabled` | → **fully dead.** Delete the property and every config entry (including wonder's). | +| `SiteRuntime … ConfigFetchRetryCount` | → **dead after Task 15** (only reader was `SiteReplicationActor.cs:157`; see Task 13). Delete the option, its validator rule, and any config entries — confirm with `grep -rn "ConfigFetchRetryCount" src/ tests/` first. | + +`ParkedOperationRelayTests.cs:39` and `ParkedMessageHandlerActorTests.cs:34` set +`ReplicationEnabled = false` — they only need the flag to exist, so remove those lines. +`StoreAndForwardOptionsTests.cs` touches it three times: `:14` asserts the default, `:26` sets and +`:33` asserts a customized value — remove all three references. + +**Leave the path keys present in configs** with a comment — they are read by the legacy migrator and +removing them would strand un-migrated data on a node that has not yet started once. + +**Commit:** `chore(config): retire ReplicationEnabled, make legacy db paths migration-only` + +--- + +### Task 18: Two-node convergence suite for the Phase 2 tables + +**Classification:** high-risk +**Estimated implement time:** ~5 min +**Parallelizable with:** Task 19 + +Extend Phase 1's `LocalDbSitePairConvergenceTests.cs` harness (real loopback Kestrel h2c + the real +`LocalDbSyncAuthInterceptor`, initialized via `SiteLocalDbSetup.OnReady`). + +**Files:** +- Modify: `tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbSitePairConvergenceTests.cs` + +Scenarios, beyond Tasks 10–11: + +1. A config deploy on A converges to B, and B's `deployed_configurations` row matches byte for byte. +2. `RemoveDeployedConfigAsync`'s **3-table cascade** converges — `static_attribute_overrides` and + `native_alarm_state` orphans do not survive on B. There are no foreign keys + (`SiteStorageService` DDL has none), so this is three independent delete streams that LWW may + reorder. **This is the most likely real defect in the whole plan.** +3. A burst of `native_alarm_state` upserts converges and the oplog drains. +4. Node B restarts and catches up via snapshot resync without losing rows written on B while down. + +**Verify the suite is non-vacuous.** (Corrected claim: Phase 1's convergence suite did **not** do +a mismatched-key run — it uses one shared key, `LocalDbSitePairConvergenceTests.cs:47`; wrong-key +*denial* is unit-covered separately in `Host.Tests/LocalDbSyncAuthInterceptorTests.cs`.) Prove it +directly for the new scenarios: run once with the new table registrations disabled on node B (or a +deliberately wrong key) and confirm every new scenario goes red, then restore and record the +result in the commit message. A convergence suite that cannot fail proves nothing. + +**Commit:** `test(localdb): two-node convergence for config tables + sf_messages` + +--- + +### Task 19: Rig configuration + +**Classification:** small +**Estimated implement time:** ~3 min +**Parallelizable with:** Task 18 + +**Files:** +- Modify: `docker/site-a-node-a/appsettings.Site.json`, `docker/site-a-node-b/appsettings.Site.json` + +Set `LocalDb:Replication:MaxOplogRows` and `LocalDb:Replication:MaxOplogAge` (exact key paths — +the lib binds the `LocalDb:Replication` section; defaults 1,000,000 rows / 7 days) from **Task 1's +measured numbers** — not from a guess. Per D6, also set `LocalDb:Replication:MaxBatchSize` +(default 500) so `max-config_json-bytes × MaxBatchSize` stays comfortably under the 4 MB gRPC +receive limit. Keep site-b/site-c unreplicated so default-OFF stays proven side by side on one rig. + +**Commit:** `chore(docker): size the site-a oplog caps from the phase 2 soak` + +--- + +### Task 20: Live gate on the docker rig + +**Classification:** high-risk +**Estimated implement time:** ~30 min wall-clock +**Parallelizable with:** none + +**Files:** +- Create: `docs/plans/2026-07-19-localdb-phase2-live-gate.md` + +```bash +cd ~/Desktop/ScadaBridge && bash docker/deploy.sh +``` + +Tooling: the containers have no `sqlite3` — run DB checks host-side against the bind mounts +(`docker/site-a-node-{a,b}/data/`); metrics via +`docker exec scadabridge-site-a-a curl -s localhost:8084/metrics` (port **8084**, not published +to the host). + +Evidence required — each must be captured, not asserted: + +1. **Migration ran.** `store-and-forward.db.migrated` and `scadabridge.db.migrated` exist on both + site-a nodes; row counts in the consolidated DB match the legacy files. (Both nodes migrate + their own legacy files independently; identical-content rows then LWW-converge — expected, + not an anomaly.) +2. **No SMTP/notification rows migrated** — `SELECT COUNT(*) FROM smtp_configurations` is 0. +3. **Config converges.** Deploy an instance to site-a; both nodes' `deployed_configurations` rows + are byte-identical with identical `__localdb_row_version` and originating node id. +4. **The standby made no config HTTP fetch during the deploy** (both nodes up) — grep the deploy + window of its log. This is the positive proof that notify-and-fetch is gone. A + `SiteReconciliationActor` fetch at node *startup* is legitimate (it survives Phase 2) and does + not fail this check. +5. **S&F converges.** Enqueue against a dead target; the parked row appears on the peer. +6. **Site failover.** Flip the site-a pair and confirm buffered messages survive and deliver + exactly once after the flip. `docker/failover-drill.sh` targets **CENTRAL** nodes and does not + exercise this — run the site flip directly. +7. **Cascade delete.** Remove a deployed config; confirm no orphan `static_attribute_overrides` or + `native_alarm_state` rows on the peer. +8. **Zero dead letters**, oplog drained, `localdb_*` scraped from `/metrics`. +9. **Both nodes stopped and started together** (D5) — confirm a clean rejoin. +10. **Post-cutover drain under churn** — the empirical half Task 1 could not measure pre-cutover: + re-run Task 1's two load generators against this build and watch `localdb_oplog_depth` + (and `SELECT COUNT(*) FROM __localdb_oplog`) rise **and drain between bursts**. Monotonic + growth that never drains is the same stop condition as Task 1 step 7 (keyed-instances hatch). + +**If any check fails, stop and report.** Do not proceed to Task 21. + +**Commit:** `docs(localdb): phase 2 live gate evidence` + +--- + +### Task 21: Documentation truth pass + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** none + +**Files:** +- Modify: `CLAUDE.md` (ScadaBridge), `~/Desktop/scadaproj/CLAUDE.md` (LocalDb component row) +- Modify: `docs/requirements/Component-StoreAndForward.md:83` +- Modify: `docs/components/StoreAndForward.md`, `SiteRuntime.md`, `Host.md` +- Modify: `docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md` +- Modify: `docs/deployment/installation-guide.md` + `topology-guide.md` (+ `production-checklist.md` + if it covers upgrades) — there is no file literally named "runbook" + +Specific corrections: + +- `Component-StoreAndForward.md:83` is a long **normative** paragraph specifying the whole chunked + resync protocol and the N5 race. Rewrite for CDC — state the new duplicate-delivery bound + explicitly rather than deleting the discussion. +- The frame-size known-issue is **already marked "Status: RESOLVED (2026-06-26)"** (resolved by the + notify-and-fetch rework). Amend it to record that Phase 2 deleted notify-and-fetch itself — CDC + ships the row over gRPC, so the frame constraint is gone entirely — and note the successor + ceiling: the 4 MB gRPC message cap managed via `MaxBatchSize` (D6). +- **Add to the deployment docs (D5):** a site pair must be stopped and started together. Rolling + one node at a time is no longer supported, because the legacy `SfBufferSnapshot` compatibility + handler is gone. Also record the D2 caveat: a node offline longer than + `LocalDb:Replication:TombstoneRetention` (default 7 days) can resurrect deleted rows on rejoin. +- Update the scadaproj LocalDb component row: Phase 2 complete, what replicates now, what + `SiteReplicationActor` used to do. + +**Commit:** `docs(localdb): phase 2 truth pass across both repos` + +--- + +## Definition of done + +All met, 2026-07-20. + +- [x] Solution builds with **0 warnings** (`TreatWarningsAsErrors` is on) +- [x] Every suite green — Host 330, CentralUI 925, Commons 684, SiteRuntime 512, AuditLog 355, + Communication 312, StoreAndForward 130, HealthMonitoring 97, SiteEventLogging 70, + IntegrationTests 94. 0 failures, 0 skips. +- [x] `grep -rn "SiteReplicationActor\|ReplicationService\|SfBufferSnapshot" src/` → **no code + references.** Four matches remain and are all deliberate: prose inside `///` and `//` comments + in `ActiveNodeEvaluator`, `SiteLocalDbSetup` and `AkkaHostedService` that records what was + deleted and why. **The check as written is too blunt** — a literal "no matches" would force + deleting the very explanations that stop someone re-introducing the old design. Read it as + "no references from code". +- [x] Live gate PASS — **10** evidence items, not the 9 this checklist says (Task 20 itself lists + ten). All captured in `docs/plans/2026-07-19-localdb-phase2-live-gate.md`, with two scope + caveats recorded there rather than glossed. +- [x] Convergence suite verified non-vacuous — with the eight `RegisterReplicated` calls commented + out, all four Task 18 scenarios go red (4 failed / 0 passed); restored, 20/20 across the three + LocalDb suites. +- [x] Both `CLAUDE.md` files updated diff --git a/docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json b/docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json new file mode 100644 index 00000000..e13c19c6 --- /dev/null +++ b/docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json @@ -0,0 +1,106 @@ +{ + "planPath": "docs/plans/2026-07-19-localdb-adoption-phase2.md", + "execution": { + "mode": "parallel-waves", + "implementerModel": "opus", + "isolation": "worktree", + "branch": "feat/localdb-phase2", + "baseBranch": "feat/localdb-phase1", + "note": "Phase 1's branch is NOT merged/pushed, so phase 2 branches from it. Dispatch every unblocked task concurrently per the wave table in the plan. Parallel implementers MUST use worktree isolation - concurrent git in one worktree races destructively." + }, + "scopeDecision": { + "date": "2026-07-19", + "by": "user", + "choice": "Full scope as designed", + "note": "User chose both surfaces (config tables + sf_messages) in one phase, deleting both bespoke mechanisms together, over the recommended split of S&F-first. The four open design questions are therefore resolved INSIDE this plan as D1-D6 rather than deferred." + }, + "reviewPass": { + "date": "2026-07-19", + "note": "Plan verified against actual code (3 verification sweeps + LocalDb library source) and corrected in place. Headline corrections: D1 (StoreDeployedConfigIfNewerAsync has a SECOND surviving caller, SiteReconciliationActor.cs:166 - the method and guard STAY; original Task 13 would also not have compiled, deleting a method whose caller dies only in Task 15), D3 (the active node ALREADY purges at DeploymentManagerActor.cs:1921 - Task 12 became a pin test, nothing is re-homed), D6 added (4 MB gRPC receive cap x row-count-only MaxBatchSize batching; config_json > 128 KB documented - measure in Task 1, size MaxBatchSize in Task 19, single row near 4 MB = stop/lib work), Task 1 rewritten (Phase 2 tables are NOT registered on the Phase 1 rig so driven churn never reaches __localdb_oplog - measure legacy-DB write rates + arithmetic; metrics port 8084 not 8080; real metric names are localdb_oplog_depth / localdb_sync_*, the plan's localdb_oplog_backlog/replication_dead_letters/sync_connected never existed; containers have no sqlite3 - sample via throwaway copies of the DB triplet, NEVER host-side sqlite3 against the live files [2026-07-20: that poisons the container's WAL state - see docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md]; branch already exists, no checkout -b), Task 14 (register 8 tables NOT 11 - notification_lists/smtp_configurations deliberately unregistered, reversing the original instruction; keep Migrate LAST in OnReady)." + }, + "decisions": [ + { + "id": "D1", + "subject": "Config moves to CDC; notify-and-fetch is DELETED - but the guarded write STAYS", + "evidence": "SiteReplicationActor sends id+fetch-coords only because the config blob exceeds Akka's 128KB frame (docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md, already marked RESOLVED by the notify-and-fetch rework). LocalDb sync is gRPC - no such limit (but see D6). CORRECTED: StoreDeployedConfigIfNewerAsync (SiteStorageService.cs:301-336, guard at :325) has TWO production callers - SiteReplicationActor.cs:375 (dies in Task 15) AND SiteReconciliationActor.cs:166 (per-node startup self-heal vs central, SURVIVES Phase 2, stale-fetch race still real). Method + guard stay; reconcile becomes a benign second CDC writer. Do NOT reproduce the guard on top of LWW - deployed_at and HLC are different clocks and mixing them is non-convergent." + }, + { + "id": "D2", + "subject": "ReplaceAllAsync deleted; the N1 directional guard becomes unnecessary", + "evidence": "LocalDb's snapshot resync MERGES per-row LWW and never WIPES: SnapshotApplier.OnBeginAsync (SnapshotStreamer.cs:163-170) resets counters only; OnBatchAsync (:172-186) routes snapshot rows through the same LwwApplier as deltas; LwwApplier.cs:69-78 discards an incoming row whose HLC is lower. Row-level deletes DO replicate (delete-trigger tombstones, streamed by SnapshotStreamer, applied as real DELETEs by LwwApplier) - only the destructive whole-table replace is gone. Caveat: tombstones pruned after TombstoneRetention (default 7d); a node offline longer can resurrect deleted rows (runbook, Task 21). SEMANTIC CHANGE: the standby is convergent, no longer byte-identical." + }, + { + "id": "D3", + "subject": "CORRECTED: the SMTP purge already runs on the active node - pin it, don't move it", + "evidence": "PurgeCentralOnlyNotificationConfigAsync (SiteStorageService.cs:811-821) has TWO callers: DeploymentManagerActor.cs:1921 (ACTIVE node's HandleDeployArtifacts, :1864-1963) and SiteReplicationActor.cs:456 (standby copy, dies with the actor). The purge never lapses; the original 're-home before any deletion' premise was false. Task 12 = pin test only (ArtifactStorageTests covers the storage method, not the actor call site Task 16 edits). No site writer to notification_lists/smtp_configurations since 2026-07-10 (verified: only test seeding inserts exist) + migrator skips them => permanently empty in the consolidated DB => Task 14 does NOT register them." + }, + { + "id": "D4", + "subject": "native_alarm_state volume is MEASURED, not assumed", + "evidence": "scadabridge.db is not only config - native_alarm_state mirrors live A&C conditions (NativeAlarmActor.cs:504) and is the highest-volume table in either DB. sf_messages worst case ~50 row-writes/sec. Task 1 measures both AT THE LEGACY-DB SOURCE (they are not in the Phase 1 oplog - see reviewPass) and sets MaxOplogRows/MaxOplogAge arithmetically; Task 20 evidence 10 does the empirical post-cutover drain check. If growth is monotonic, STOP: keyed-instances escape hatch = ~/Desktop/scadaproj/docs/plans/2026-07-19-scadabridge-localdb-design.md:139-141 (adoption design doc, NOT the 07-17 lib doc), a scadaproj library effort that would suspend this plan." + }, + { + "id": "D5", + "subject": "Cutover forecloses rolling site upgrades", + "evidence": "SiteReplicationActor retains a legacy monolithic SfBufferSnapshot handler for rolling upgrades. With no dual-mechanism period, one node would speak a protocol the other no longer implements. Both nodes of a site must be stopped and started together. Task 21 puts this in the deployment docs (installation-guide.md/topology-guide.md - no file literally named runbook)." + }, + { + "id": "D6", + "subject": "NEW (review pass): 4 MB gRPC message cap replaces the 128 KB Akka frame as the size ceiling", + "evidence": "Neither side configures gRPC message sizes (ScadaBridge AddGrpc at Program.cs:519-521 sets only the auth interceptor; the lib's initiator channel is bare GrpcChannel.ForAddress) => 4 MB default receive limit both directions. Batching is row-count-only (MaxBatchSize default 500; SyncSession.cs:227, SnapshotStreamer.cs:55; no byte-aware chunking). deployed_configurations.config_json documented >128 KB/row; a few dozen such rows in one batch exceeds 4 MB and wedges the stream on a poison batch. Task 1 measures max/avg config_json bytes; Task 19 sets LocalDb:Replication:MaxBatchSize so max-row-bytes x MaxBatchSize << 4 MB; any single row near 4 MB = STOP (needs byte-aware batching or size knobs in the LocalDb lib - scadaproj effort)." + } + ], + "reconFindings": [ + "The gate doc named 2 test files as the specification; the real spec is 5 - it missed StoreAndForwardReplicationTests.cs (incl. the only Requeue coverage), ReplicationWireSerializationPinTests.cs, ResyncWireSerializationPinTests.cs, and SfBufferResyncPredicateTests.cs (the N1 Critical regression test).", + "sf_messages has NO version column today - ON CONFLICT(id) DO UPDATE has no comparison predicate (StoreAndForwardStorage.cs:331-345). 'Newest wins' is bare arrival order. LWW-by-HLC is an IMPROVEMENT here, not a regression.", + "No autoincrement-integer PKs exist anywhere in Phase 2 scope - all 9 config tables use natural TEXT or composite TEXT keys, sf_messages is TEXT. LocalDb RegisterReplicated SUPPORTS composite PKs (ordered pk ordinals) and rejects BLOB columns - no Phase 2 table has one (verified). Phase 1's site_events GUID conversion has no Phase 2 analogue.", + "SiteStorageService has NO foreign keys. RemoveDeployedConfigAsync (:343-376) is a manual 3-statement cascade in one transaction. Under CDC these become three independent delete streams that LWW may reorder - the most likely real defect in the plan (Task 18 scenario 2).", + "DeploymentManagerActor's replicationActor is an OPTIONAL POSITIONAL parameter at :169 (ctor :161-175; :184 is the field ASSIGNMENT). The same-typed IActorRef? optional dclManager sits immediately BEFORE it at :168 - that's the real silent-shift hazard. Props.Create passes it positionally at AkkaHostedService.cs:810. Check every call site by hand (Task 16).", + "ActiveNodeEvaluator must NOT be deleted - the S&F delivery gate still uses it (AkkaHostedService.cs:866 SetDeliveryGate -> SelfIsPrimary -> SelfIsOldestUp). Doc comment :14/:16 mentions replication; SiteReplicationActor.cs:288 calls it directly (dies with the actor).", + "ConfigFetchRetryCount's ONLY production reader is SiteReplicationActor.cs:157 (verified) - dead after Task 15; removed in Task 17 (removing it in Task 13, before the actor deletion, would not compile). IDeploymentConfigFetcher is KEPT: DeploymentManagerActor refresh path + SiteReconciliationActor + DI at ServiceCollectionExtensions.cs:84.", + "notification_lists and smtp_configurations are deliberately NOT migrated (Task 9) AND NOT registered (Task 14, corrected) - migrating or replicating them would resurrect/ship plaintext SMTP passwords; they are permanently empty by design (writers removed 2026-07-10; verified only test-seeding inserts exist).", + "REVIEW-PASS ADDITIONS: SiteReconciliationActor (runs on EVERY node at startup, best-effort self-heal vs central) is the second caller of both StoreDeployedConfigIfNewerAsync and IDeploymentConfigFetcher - it survives Phase 2 and constrains Tasks 11/13/20 (a startup fetch on the standby is legitimate; zero-fetch assertions must scope to the deploy window).", + "SiteStorageService has 21 (not 22) inline connection+OpenAsync pairs: 60,206,248,311,345,386,414,443,470,498,538,583,608,640,663,692,730,769,813,837,868. CreateConnection():51's only repository consumer is SiteExternalSystemRepository.", + "StoreAndForwardService: :39 is the _replication FIELD, :243 the ctor param; exactly 6 emission sites (:654,805,836,872,1120,1146). ServiceCollectionExtensions.cs:32 also resolves ReplicationService inside the StoreAndForwardService factory - must go in Task 14.", + "ReplicationMessages.cs holds ONLY the 10 Replicate*/Apply* records; the four SfBuffer resync records live at the bottom of SiteReplicationActor.cs (:678-707) and die with the actor file.", + "10 (not 9) appsettings.Site.json files set the legacy paths - deploy/wonder-app-vd03/appsettings.Site.json was missed (also sets ReplicationEnabled:false).", + "Rig facts: site metrics on port 8084 (AnyIP in-container, NOT published to host; 8080 is Traefik); real metric names localdb_oplog_depth / localdb_sync_* (meter ZB.MOM.WW.LocalDb.Replication); containers (aspnet:10.0) have NO sqlite3 - data dirs are host bind mounts but NEVER run sqlite3 host-side against the live files (poisons the container's WAL state, root cause of the 2026-07-20 disk-I/O-error incident): cp the .db/-wal/-shm triplet and query the copy (see the snap() helper in the plan); ReplicationOptions bind at LocalDb:Replication:* (lib binds the section).", + "Phase 1's LocalDbSitePairConvergenceTests uses ONE shared API key (:47) - it never did a mismatched-key non-vacuity run; wrong-key denial is unit-covered in Host.Tests/LocalDbSyncAuthInterceptorTests.cs. Task 18's non-vacuity check must be done directly for the new scenarios.", + "MigrateEvents synthesizes deterministic 'mig-{node}-{legacyId}' ids (NOT fresh GUIDs - crash-rerun idempotency). Migration runs AFTER RegisterReplicated in OnReady (order is load-bearing); Task 8's oplog pin test must register sf_messages on its own TestLocalDb since production doesn't register it until Task 14." + ], + "tasks": [ + {"id": 1, "subject": "Task 1: Rig soak - measure legacy-DB write rates + config_json sizes (corrected method)", "status": "completed", "classification": "high-risk", "note": "GATE CLOSED - verdict PROCEED (2026-07-20). NEVER sample host-side sqlite3 on live bind-mounted WAL files (that method poisoned the first run - use the copy-based snap() helper). Clean re-run: sf_messages 0.80 rows/sec insert, 0/sec retry-UPDATE, oplog 0, alarms 0, max payload_json 76 B, max config_json 721 B (NOT representative), zero SQLite errors in 30 min, both nodes converged. No stop condition met. Binding output: MaxBatchSize 500 -> 16 (Task 19). Empirical drain check remains Task 20 evidence 10."}, + {"id": 2, "subject": "Task 2: Decision record - close the phase 2 gate", "status": "completed", "classification": "trivial", "blockedBy": [1], "note": "Gate doc status flipped NOT STARTED -> CLOSED; all 5 SS5 questions answered inline (questions kept, not deleted); SS2's N5 duplicate-bound requirement routed explicitly to Task 21."}, + {"id": 3, "subject": "Task 3: Extract StoreAndForwardSchema.Apply", "status": "completed", "classification": "small", "blockedBy": [2], "note": "DEVIATION: kept PRAGMA journal_mode=WAL in InitializeAsync - the plan Step 4 snippet drops it but Task 4 says not to move the equivalent pragma; contradictory, and dropping it would regress documented concurrent-writer support before Task 5 makes it moot. Added a legacy-upgrade test beyond the plan's (the specified test asserts against a FRESH table where CREATE TABLE already lists all 16 columns - it would pass with every ALTER deleted). That test found the last_attempt_at_ms backfill lands 1 ms low: julianday() double day-fraction rounding, pre-existing, carried verbatim, asserted with 1 ms tolerance. Suite 154/154."}, + {"id": 4, "subject": "Task 4: Extract SiteStorageSchema.Apply", "status": "completed", "classification": "small", "blockedBy": [2], "note": "DEVIATION: TryAddColumnAsync's catch-on-message-text ('duplicate column') became a PRAGMA table_info probe matching OperationTrackingSchema - the message form depends on a non-contractual error string and swallowed unrelated SqliteExceptions; also dropped the ILogger dep, which is what let the class be static. Cost: the per-column 'Migrated: added column' info log is gone (nothing consumes it). PRAGMA journal_mode=WAL stays in InitializeAsync per plan. Added a legacy-upgrade test for the same reason as Task 3. SiteRuntime 532/532, full solution build 0 warnings."}, + {"id": 5, "subject": "Task 5: Rewire StoreAndForwardStorage onto ILocalDb", "status": "completed", "classification": "high-risk", "blockedBy": [3], "note": "Test fallout was ~7x the plan estimate: 40 files across 7 test projects, not \"fixtures\" in one. Most used Mode=Memory;Cache=Shared and LocalDb has NO in-memory mode, so all moved to real temp files. Added shared tests/ZB.MOM.WW.ScadaBridge.TestSupport lib (TestLocalDb) instead of copying the Phase 1 fixture into 7 projects. WAL test retargeted to the LocalDb-backed store; directory-creation test moved to Host.Tests (different owner - see Task 6 note)."}, + {"id": 6, "subject": "Task 6: Rewire SiteStorageService onto ILocalDb", "status": "completed", "classification": "high-risk", "blockedBy": [4], "note": "LATENT PHASE-1 DEFECT FOUND+FIXED: LocalDb does NOT create the parent directory and SqliteLocalDb opens the file eagerly, so a missing dir is a HARD BOOT FAILURE (SQLite Error 14). Default site config is the RELATIVE ./data/site-localdb.db; docker escapes only because the volume mount creates /app/data. The plan wrongly assumed dir-creation moved to LocalDb with file ownership. Fixed via SiteLocalDbDirectory.Ensure(config) before AddZbLocalDb + Host.Tests/SiteLocalDbDirectoryTests (non-vacuity observed: 2 tests failed with exactly Error 14 pre-fix). ALSO a contract change: SiteStorageService.CreateConnection() used to return an UNOPENED connection; it now returns an ALREADY-OPEN one - SiteExternalSystemRepository dropped 5 OpenAsync calls. AddSiteRuntime(string) overload deleted."}, + {"id": 7, "subject": "Task 7: Extend SiteLocalDbSetup with the new DDL (not yet registered)", "status": "completed", "classification": "standard", "blockedBy": [5, 6], "commit": "f8aa02e2", "note": "As planned. Two pins in SiteLocalDbWiringTests through the REAL composition root: all 12 tables exist, and ReplicatedTables is EXACTLY the Phase 1 pair (an equality check, not Contains - the 'not yet' is the assertion that matters). Task 14 should INVERT that second test, not delete it. Non-vacuity verified by removing the DDL."}, + {"id": 8, "subject": "Task 8: Extend the legacy migrator for sf_messages", "status": "completed", "classification": "high-risk", "blockedBy": [7], "commit": "bdc0dffe", "note": "DEVIATION: the copy INTERSECTS the legacy column set with the current one instead of naming all 16 columns. A legacy file from an older build lacks execution_id/parent_execution_id/last_attempt_at_ms, and naming a missing column throws 'no such column' - which the existing ReadAll treats as an unrecognised shape and SILENTLY DISCARDS EVERY ROW. For undelivered messages that is real data loss. A required-column (PK) guard stops the tolerance degrading into NULL-keyed copies. Tests register sf_messages on their own harness (production OnReady does not until Task 14), so Task 14 MUST keep Migrate as the LAST call in OnReady, after all registrations. Non-vacuity verified: all 4 fail without the Migrate call."}, + {"id": 9, "subject": "Task 9: Extend the legacy migrator for the config tables (skip notification/smtp)", "status": "completed", "classification": "high-risk", "blockedBy": [7], "commit": "5ddc7eed", "note": "7 of 9 tables migrated; notification_lists + smtp_configurations skipped as planned (plaintext passwords must not enter a table Task 14 makes replicated) - the skip is pinned by a test that also greps __localdb_oplog.row_json for the secret. Task 8 MigrateTable generalized to MigrateFile(many tables, one transaction, one rename). ADDED beyond the plan: MigratorColumnLists_MatchTheLiveSchema, a column-parity test vs SiteStorageSchema - the plan called for a manual column check, but a mismatch is invisible at runtime in BOTH directions (a typo is silently dropped by the intersection; a missing column silently leaves data behind). SiteStorageTables + LegacyTable are internal so the test can read them. Non-vacuity verified twice: removing the call (3 fail) and wrongly adding notification/smtp to the map (skip test fails)."}, + {"id": 10, "subject": "Task 10: Port the S&F replication test intents as CDC specs", "status": "completed", "classification": "standard", "blockedBy": [8], "commit": "2bbe6631", "note": "DEVIATION: extracted the Phase 1 fixture into LocalDbSitePairHarness (abstract base) instead of duplicating ~150 lines; Phase 1 tests derive from it and pass unchanged. The harness registers the 8 Phase 2 tables itself (production OnReady does not until Task 14) - DELETE RegisterPhase2TablesUntilCutover at Task 14. The 8-table list is literal, not derived from production code, so a cutover registering the wrong set fails these tests. Non-vacuity: unregistering sf_messages failed 6 of 7 - the 7th (add-then-remove ordering) PASSED because an absent row is also what a non-replicating pair looks like; fixed with a control row that must converge in the same window."}, + {"id": 11, "subject": "Task 11: Port the resync + directional-authority tests", "status": "completed", "classification": "standard", "blockedBy": [9], "note": "N1 re-expressed per D2 as a no-rows-lost property (LWW merge, never deletes), not directional authority. DEVIATION on the zero-fetch assertion: the plan wanted a fetcher double recording zero calls, but this harness has no actor system / no central / no IDeploymentConfigFetcher in the graph, so the double could not fail either way. Replaced with the positive half (config reaches B over replication alone) plus an in-file comment recording that the negative half is proved by Task 15 deleting the code and the build passing. D1 scope note recorded in-file. Non-vacuity: unregistering deployed_configurations fails all 4.", "commit": "c56bf4ae"}, + {"id": 12, "subject": "Task 12: Pin the active-node SMTP purge (corrected: already exists at DeploymentManagerActor.cs:1921)", "status": "completed", "classification": "standard", "blockedBy": [10, 11], "note": "As planned - NO production change; D3 confirmed correct, the call already exists at DeploymentManagerActor.cs:1921. Pin verified RED-FIRST by commenting out the call. Test needed a using for Commons.Messages.Artifacts and polls (the apply runs on a Task.Run inside the actor).", "commit": "79ce5161"}, + {"id": 13, "subject": "Task 13: Notify-and-fetch scope check - guarded write STAYS for SiteReconciliationActor", "status": "completed", "classification": "standard", "blockedBy": [12], "note": "Doc-comment only, as planned. Re-verified both callers: SiteReplicationActor:375 (dies Task 15) + SiteReconciliationActor:166 (survives). Step 2 scope check re-run: ConfigFetchRetryCount's only production reader is still SiteReplicationActor:157, so option + validator rule stay until Task 17. Also fixed a stale 'guarded standby write' header in SiteStorageServiceTests.", "commit": "79ce5161"}, + {"id": 14, "subject": "Task 14: Register the 8 Phase 2 tables and delete ReplicationService", "status": "completed", "classification": "high-risk", "blockedBy": [13], "note": "LANDED AS ONE COMMIT with tasks 15+16 - PLAN DEFECT: they cannot compile separately (SiteReplicationActor takes a ReplicationService + calls ReplaceAllAsync; DeploymentManagerActor Tells ReplicationMessages types; AkkaHostedService constructs the actor). Combining also strengthens Task 14's own invariant: never both mechanisms, never neither. Registered 8 tables; notification_lists + smtp_configurations deliberately NOT registered. Deleted ReplaceAllAsync as UNSAFE (a mass DELETE on a replicated table would be captured and shipped), and its test replaced by a comment explaining that. The positional-arg hazard was REAL: 4 DeploymentManagerActor test call sites bound wrong args; converted to named where possible - Props.Create is an expression tree and rejects OUT-OF-POSITION named args, so the rest are padded positionally. Task 7's 'not yet' test INVERTED (exact in both directions) + new security-named test for the SMTP tables + composite-PK test. Harness's temporary registration deleted, so the convergence suites now prove the cutover. My expected table list was mis-sorted: ordinal puts '_' (0x5F) before 'b', so data_connection_definitions precedes database_connections.", "commit": "037798b3"}, + {"id": 15, "subject": "Task 15: Delete SiteReplicationActor and its messages", "status": "completed", "classification": "high-risk", "blockedBy": [14], "note": "LANDED AS ONE COMMIT with tasks 15+16 - PLAN DEFECT: they cannot compile separately (SiteReplicationActor takes a ReplicationService + calls ReplaceAllAsync; DeploymentManagerActor Tells ReplicationMessages types; AkkaHostedService constructs the actor). Combining also strengthens Task 14's own invariant: never both mechanisms, never neither. Registered 8 tables; notification_lists + smtp_configurations deliberately NOT registered. Deleted ReplaceAllAsync as UNSAFE (a mass DELETE on a replicated table would be captured and shipped), and its test replaced by a comment explaining that. The positional-arg hazard was REAL: 4 DeploymentManagerActor test call sites bound wrong args; converted to named where possible - Props.Create is an expression tree and rejects OUT-OF-POSITION named args, so the rest are padded positionally. Task 7's 'not yet' test INVERTED (exact in both directions) + new security-named test for the SMTP tables + composite-PK test. Harness's temporary registration deleted, so the convergence suites now prove the cutover. My expected table list was mis-sorted: ordinal puts '_' (0x5F) before 'b', so data_connection_definitions precedes database_connections.", "commit": "037798b3"}, + {"id": 16, "subject": "Task 16: Clean up DeploymentManagerActor and AkkaHostedService", "status": "completed", "classification": "standard", "blockedBy": [15], "note": "LANDED AS ONE COMMIT with tasks 15+16 - PLAN DEFECT: they cannot compile separately (SiteReplicationActor takes a ReplicationService + calls ReplaceAllAsync; DeploymentManagerActor Tells ReplicationMessages types; AkkaHostedService constructs the actor). Combining also strengthens Task 14's own invariant: never both mechanisms, never neither. Registered 8 tables; notification_lists + smtp_configurations deliberately NOT registered. Deleted ReplaceAllAsync as UNSAFE (a mass DELETE on a replicated table would be captured and shipped), and its test replaced by a comment explaining that. The positional-arg hazard was REAL: 4 DeploymentManagerActor test call sites bound wrong args; converted to named where possible - Props.Create is an expression tree and rejects OUT-OF-POSITION named args, so the rest are padded positionally. Task 7's 'not yet' test INVERTED (exact in both directions) + new security-named test for the SMTP tables + composite-PK test. Harness's temporary registration deleted, so the convergence suites now prove the cutover. My expected table list was mis-sorted: ordinal puts '_' (0x5F) before 'b', so data_connection_definitions precedes database_connections.", "commit": "037798b3"}, + {"id": 17, "subject": "Task 17: Config-key cleanup (incl. ConfigFetchRetryCount; 10 appsettings files)", "status": "completed", "classification": "standard", "blockedBy": [16], "note": "As planned. DEVIATION: deploy/wonder-app-vd03/appsettings.Site.json sits under a GITIGNORED deploy/ tree, so its edit is local-only and must be repeated on the box at deploy time. Comment style is // (JSONC) rather than \"_comment_\" keys: every one of these files already contains // comments and .NET's json config reader accepts them, and a _comment_ key inside a bound section is a phantom config entry. Both relaxations pinned by the INVERSE of the test they replace (Site_MissingSiteDbPath_IsAccepted..., EmptySqliteDbPath_IsAccepted...), each verified red with the old rule restored. DatabaseOptionsValidator needed no change - it was already null-tolerant/blank-rejecting, which is exactly migration-only semantics.", "commit": "605e5682"}, + {"id": 18, "subject": "Task 18: Two-node convergence suite for the Phase 2 tables", "status": "completed", "classification": "high-risk", "blockedBy": [17], "note": "DEVIATION: landed as a NEW file LocalDbPhase2ConvergenceTests.cs rather than extending Phase 1's LocalDbSitePairConvergenceTests.cs, and drives the REAL SiteStorageService instead of hand-written SQL - possible only post-cutover, and what makes the cascade scenario test the shipped transaction rather than a re-creation of it. Scenario 4 was retargeted onto shared_scripts/external_systems/static_attribute_overrides because the plan's version overlapped LocalDbConfigConvergenceTests' N1 scenario almost exactly; the union-survives property is per-table, so re-proving it on untouched tables is the non-redundant half. Cascade test carries a never-removed control instance (absence assertions otherwise cannot distinguish 'cascade converged' from 'node B lost these tables'). Non-vacuity PROVEN as mandated: with the 8 RegisterReplicated calls commented out, 4 failed / 0 passed; restored, 20/20 across the three LocalDb suites.", "commit": "15013156"}, + {"id": 19, "subject": "Task 19: Rig configuration (MaxOplogRows/MaxOplogAge + MaxBatchSize per D6)", "status": "completed", "classification": "small", "blockedBy": [17], "note": "MaxBatchSize 500->16 (D6: row-count batching x ~70 KB production config_json vs the 4 MB gRPC cap; 16 => ~1.1 MB worst case). MaxOplogRows 1M->250,000 and MaxOplogAge 7d->2d from the soak's 0.80 rows/sec (~69k/day). Tighter-than-default is SAFE because a cap breach prunes + sets needs_snapshot (graceful snapshot resync), not data loss - the Task 1 finding that the plan's stop condition was weaker than written. site-b/site-c left unreplicated so default-OFF stays proven side by side.", "commit": "921edab4"}, + {"id": 20, "subject": "Task 20: Live gate on the docker rig", "status": "completed", "classification": "high-risk", "blockedBy": [18, 19], "note": "ALL 10 CHECKS PASS. Evidence: docs/plans/2026-07-19-localdb-phase2-live-gate.md. Key blocker found and fixed mid-run: external systems reach a site ONLY via ArtifactDeploymentService, which `instance deploy` never invokes - `deploy artifacts` was needed both to deliver the probe harness and to propagate the owed ExternalSystemDefinitions restore. THREE METHOD CORRECTIONS to the plan: (1) its instruction to run DB checks host-side against the bind mounts is UNSAFE - host sqlite3 poisons the container WAL; copy the db/-wal/-shm triplet and query the copy. (2) `docker exec ... curl` cannot scrape metrics (no curl in aspnet:10.0) and with 2>/dev/null the failure is silent - it nearly became a false 'metrics missing' finding; use a network-sharing curl sidecar. (3) checks needing S&F load need CachedCall, not Call. CAVEATS recorded not glossed: check 2's zero-count is vacuous alone (legacy source was also empty) and rests on the ABSENCE of CDC triggers; check 7's native_alarm_state leg was empty live and is covered only offline; check 10's sampling is coarse and the real rise/drain evidence comes from check 6's 0->4->0.", "commit": "158e79bb"}, + {"id": 21, "subject": "Task 21: Documentation truth pass", "status": "completed", "classification": "standard", "blockedBy": [20], "note": "Both CLAUDE.md files + Component-StoreAndForward.md:83 (normative resync paragraph rewritten for CDC, stating the duplicate-delivery bound explicitly: limited to messages the OLD primary delivered whose status change had not yet replicated when the gate flipped - one flush interval plus in-flight ack, and it does NOT grow with backlog depth or absence duration) + components/{StoreAndForward,SiteRuntime,Host}.md + the frame-size known-issue (amended: Phase 2 deleted notify-and-fetch itself, so the 128KB Akka frame constraint is gone from the intra-site hop entirely; successor ceiling is the 4MB gRPC cap via MaxBatchSize, and note the failure mode differs - oversized gRPC is REJECTED, not silently dropped) + deployment topology-guide.md and installation-guide.md (D5 stop-both-together, D2 TombstoneRetention resurrection bound). DoD closed: build 0 warnings, all 10 suites green (3509 tests, 0 failures). DoD grep nuance recorded in the plan: 4 matches remain in src/ and are all deliberate COMMENT prose explaining what was deleted - a literal 'no matches' would delete the explanations that stop someone re-introducing the old design.", "commit": null} + ], + "knownFlakes": [ + { + "test": "SiteRuntime.Tests InstanceActorChildAttributeRaceTests.ChildActors_AreSeededFromAnIsolatedCopy_NotTheLiveAttributesDictionary", + "note": "Intermittent ActorNotFoundException under full-suite load; passes in isolation. Pre-existing, carried over from Phase 1." + }, + { + "test": "AuditLog.Tests ParentExecutionIdCorrelationTests.InboundRoutedRun_AllRoutedRows_CarryInboundExecutionId_AsParentExecutionId", + "note": "Cold-MSSQL-fixture timing: ~91s and AwaitAssert-times-out cold, ~1s warm. Re-run before investigating." + } + ], + "lastUpdated": "2026-07-20", + "phase2Status": "UNBLOCKED - Task 1 gate CLOSED (verdict PROCEED) and Task 2 DONE, 2026-07-20. Task 1's original STOP verdict is SUPERSEDED: the 'Phase 1 disk I/O defect' was OBSERVER-INDUCED (host-side sqlite3 against live bind-mounted WAL files resets the WAL across virtiofs and permanently poisons the container's connections) - NOT a product defect. See docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md. Both earlier isolation claims were confounded: one sampling pass had already poisoned BOTH nodes, and a poisoned standby looks healthy only because it issues almost no statements; 'LocalDb-specific' was sampling-selection bias (only the LocalDb file had ever been host-read). Clean re-run 2026-07-20 with the copy-based snap() helper on restarted nodes, 6 consecutive 60s intervals: sf_messages 48 rows/min = 0.80 rows/sec dead steady, retry-UPDATE rate 0/sec (SUM(retry_count) flat at 200), oplog 0, native_alarm_state 0, max payload_json 76 B, max config_json 721 B, ZERO SQLite errors across 30 min, both site-a nodes converged at 3564 rows. Honest gap: 0.80/s is ~1.6% of the 50/s ceiling and the retry-UPDATE path was never exercised - acceptable because that ceiling is structural (SweepBatchLimit/RetryTimerInterval), not empirical. Rig config rows (721 B) are NOT representative; D6 sizing rests on the documented ~60-70 KB production config_json. Plan-premise corrections stand: D6 (MaxBatchSize 500 -> 16, the one firmly evidence-backed number; Task 19 sets it), D4 (alarm writes bounded by per-SourceReference coalescing at a 100 ms flush, NOT unbounded), sf_messages hard ceiling 50 rows/sec, oplog cap overrun = graceful snapshot resync (needs_snapshot), not data loss. NEXT: Wave 1 = Tasks 3 + 4, dispatchable in parallel (disjoint Files blocks). Tasks 3-21 untouched; no plan code written yet. Rig cleanup still owed before Task 20: restore ExternalSystemDefinitions id 1 to http://scadabridge-restapi:5200 (currently http://127.0.0.1:9), and remove SoakGenerator template 2021 + instances soakgen-1..4 (ids 5-8), still deployed and generating load." +} diff --git a/docs/plans/2026-07-19-localdb-phase2-gate.md b/docs/plans/2026-07-19-localdb-phase2-gate.md new file mode 100644 index 00000000..0d966e4d --- /dev/null +++ b/docs/plans/2026-07-19-localdb-phase2-gate.md @@ -0,0 +1,218 @@ +# LocalDb Phase 2 — Gate Document + +> **Status: CLOSED (2026-07-20).** The implementation plan is +> [`2026-07-19-localdb-adoption-phase2.md`](2026-07-19-localdb-adoption-phase2.md); its +> decisions D1–D6 answer this gate and its Task 1 supplied the measurements §5 demanded +> (soak record: [`2026-07-19-localdb-phase2-soak.md`](2026-07-19-localdb-phase2-soak.md)). +> Each §5 question is answered inline below. **The questions are deliberately not deleted — +> the reasoning is the value, and several of the answers overturn a premise stated above.** + +**Phase 1 live gate: PASS** (2026-07-19, docker rig). See the Task 12 commit for evidence. + +**Phase 2 goal (from the design doc, scadaproj `docs/plans/2026-07-19-scadabridge-localdb-design.md` §1):** +move `scadabridge.db`'s config tables and StoreAndForward's `sf_messages` into the same +consolidated LocalDb file, then **delete** `SiteReplicationActor` and StoreAndForward's +`ReplicationService` outright. CDC triggers replace hand-shipped ops; the library's +snapshot resync replaces the bespoke chunked anti-entropy. The design explicitly chose +**replace + delete, no dual-mechanism period** — recoverable from git. + +--- + +## 1. What Phase 1 actually established + +Facts a Phase 2 plan can rely on, all verified rather than assumed: + +- One `ILocalDb` per site process at `LocalDb:Path`, **required** (`ValidateOnStart`). + A site config missing it fails to boot. +- `SiteLocalDbSetup.OnReady` is the single place tables are created and registered. + Ordering is load-bearing: **DDL → `RegisterReplicated` → writes**, then the legacy + migrator. Rows written before registration are invisible to the peer forever, silently. +- Replication is registered in `SiteServiceRegistration` (not `Program.cs`) so the + composition-root tests cover it; only `MapZbLocalDbSync` lives in `Program.cs`. +- Inbound auth is `LocalDbSyncAuthInterceptor`, fail-closed, scoped to + `/localdb_sync.v1.LocalDbSync/`, sharing the existing 8083 h2c listener. +- Default-OFF is real and proven side-by-side: rig site-a replicates, site-b/site-c do not. +- `ZbTelemetryOptions.Meters` is a **silent allowlist** (`SiteServiceRegistration.ObservedMeters`). +- Bidirectional sync works with only one node configured as initiator. + +## 2. What Phase 2 must port before deleting anything + +The bespoke replicator's tests encode behaviour that the design's CDC replacement must +still satisfy. **Port the test intents first, delete second** — the plan should treat the +existing tests as the specification, not as code to be removed alongside the implementation. + +- `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/SiteReplicationActorTests.cs` +- `tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ReplicationServiceTests.cs` + +Implementation under deletion: + +- `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/SiteReplicationActor.cs` +- `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/ReplicationService.cs` + +Known accepted behaviour to carry forward, not silently drop: the **N5 bounded-duplicate +race** the bespoke replicator documents and accepts. A Phase 2 plan must state whether CDC +inherits the same bound, a tighter one, or a different failure shape. + +> **Routed (2026-07-20).** The failure shape changes, so N5's bound does not simply carry +> over: CDC has no chunked anti-entropy hop to duplicate across, but LWW admits re-delivery +> of a row whose park loses to a later write (see the LWW answer in §5). **Task 21 states the +> new duplicate-delivery bound explicitly** and rewrites the N5 note rather than deleting it. + +## 3. Substantially harder than Phase 1 + +Phase 1 moved two tables that **nothing replicated before**, so the worst case was "no +worse than today". Phase 2 replaces a working mechanism, which makes it a different risk +class: + +- **`sf_messages` is an outbound buffer with side effects.** Phase 1's tables are + read-mostly records; a store-and-forward row that resurrects or double-delivers sends + real traffic to an external system. Park/requeue/remove becoming ordinary captured + row updates/deletes needs its own analysis of what LWW does to an in-flight send. +- **Config tables drive deployment.** A converged-but-wrong config row is a deployed + instance behaving incorrectly, not a missing history entry. +- **No dual-mechanism period.** The chosen posture means the cutover is the test. That + raises the bar on the offline convergence suite and the live gate considerably. +- **Migration is not a no-op this time.** Phase 1's migrator usually finds nothing, + because the legacy files sat outside the volume. `scadabridge.db` and + `store-and-forward.db` are real, populated, on the volume, and actively replicated + while the migration runs. + +## 4. Required inputs before writing the plan + +1. The design doc §1 Phase 2 and §2 schema/migration sections, re-read against + post-Phase-1 reality (several Phase 1 assumptions changed during execution — see the + `amendments` array in `2026-07-19-localdb-adoption-phase1.md.tasks.json`). +2. The two test files in §2, read as specifications. +3. **Rig soak observations that do not exist yet** — see §5. + +## 5. Open questions — answer these first + +- **Oplog growth and churn under real config/S&F write rates.** Phase 1's tables are + low-volume; `sf_messages` is not. `MaxOplogRows` / `MaxOplogAge` / snapshot-resync + thresholds cannot be chosen from first principles. *This needs a Phase-1 rig soak that + has not been run.* The live gate proved correctness, not steady-state behaviour over + time. + + **ANSWERED — the soak ran (Task 1); the caps are not the binding constraint, batch + *bytes* are.** Two of this question's own premises turned out to be wrong, and both + corrections relax it: + + 1. **The stop condition was much weaker than written.** Exceeding `MaxOplogRows` / + `MaxOplogAge` does not wedge or lose data — `OplogStore` prunes and flags the peer + `needs_snapshot` (`OplogStore.cs:109-138`, `MaintenanceBackgroundService.cs:57`), which + degrades to a **graceful snapshot resync**. Overrun is a performance event, not a + correctness event, so these caps do not need to be sized defensively. + 2. **`sf_messages` has a hard ceiling, not an estimate.** `SweepBatchLimit` (500) ÷ + `RetryTimerInterval` (10 s) = **≤50 row-writes/sec**, structurally. Measured rig rate + under the purpose-built `SoakGenerator` load was far below that (see the soak record). + Rows are small — max `payload_json` **76 bytes** on the rig. + + `native_alarm_state` is answered under the last question in this section, not here (and + the plan's D4, whose premise it corrects). The **real** ceiling this + question was groping toward is D6's 4 MB gRPC message cap, and it binds on + `MaxBatchSize × max-row-bytes`, not on oplog depth — **Task 19 sets + `LocalDb:Replication:MaxBatchSize = 16`**, the one firmly evidence-backed number the soak + produced. The keyed-instances escape hatch is **not** needed; this plan proceeds. + + Independently, the soak retired a scare: a `disk I/O error` storm initially read as a + Phase 1 library defect and STOP-gated this plan was root-caused as **observer-induced** + (host-side `sqlite3` against live bind-mounted WAL files) — see + [`docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md`](../known-issues/2026-07-20-localdb-disk-io-error-under-load.md) + §0. The unmodified library sustains the full soak load with **zero** SQLite errors. + +- **LWW semantics for in-flight store-and-forward sends.** What does last-writer-wins do + when one node parks a message the other is mid-delivery on? + + **ANSWERED — the scenario cannot arise on a correctly-behaving pair, and D2 covers what + happens if it does.** Only the **active** node sweeps and delivers; the standby holds a + convergent copy and sends nothing. So "one node mid-delivery while the other parks" is + a split-brain symptom, not steady-state behaviour, and store-and-forward is not where it + should be defended against. + + If it does happen, LWW resolves per row by HLC: the later write wins, and the losing + node's view converges to it. The concrete risk is **at-least-once delivery** — a park + that loses to a stale in-flight update can be re-swept and re-sent. That is not new; + store-and-forward is already at-least-once by construction (a send that succeeds but + whose ack is lost is retried). What **is** new is D2's semantic change, recorded here + because it is the honest cost of this gate: the standby is no longer guaranteed + **byte-identical** to the active buffer, only **convergent**. The bespoke replicator's + `ReplaceAllAsync` bought identity by wiping — and the N1 directional guard existed + precisely because that wipe was dangerous. The library's snapshot resync **merges per + row and never wipes** (`SnapshotApplier`, `LwwApplier.cs:69-78`), so the failure the + guard prevented is structurally impossible rather than merely tested against. Task 10 + ports `SfBufferResyncPredicateTests` as a **convergence** assertion, not a + directional-authority one. + +- **Migration-under-load.** How does the one-time copy of an actively-replicating + `scadabridge.db` interact with the bespoke replicator still running during cutover? + + **ANSWERED — the interaction is designed out, not managed.** The two mechanisms never run + concurrently. `SiteReplicationActor` and StoreAndForward's `ReplicationService` are + **deleted in the same commit** that registers the Phase 2 tables (Tasks 14/15), and D5 + forecloses rolling upgrades: **both nodes of a site stop and start together**. A process + that boots with the new code has no bespoke replicator to race, and one running the old + code has no CDC triggers. There is no window in which a row is written by one mechanism + and read by the other. + + Within a single booting process the ordering is the load-bearing part, and it is the same + invariant Phase 1 established: **DDL → `RegisterReplicated` → migrate → writes** + (`SiteLocalDbSetup.OnReady`). Rows written *before* registration are invisible to the peer + forever, silently — which is why Tasks 8/9 run the migrator strictly **after** + registration, so every migrated row is captured by the CDC triggers and replicates + normally. Both nodes migrating independently is fine: they migrate the same source rows, + and LWW converges them. + +- **Rollback story.** With no dual-mechanism period, what is the recovery path if the + cutover fails in production — beyond "revert the commit"? + + **ANSWERED — "revert the commit" is genuinely the path, and it is safe because the + migration is additive and the legacy files are left intact.** Tasks 8/9 **copy** rows out + of `scadabridge.db` and `store-and-forward.db` into the consolidated LocalDb file; they do + not drop, truncate, or delete the source databases. Rolling back is therefore: stop both + nodes of the site, deploy the previous build, start both together (D5). The old code + reopens the legacy files and finds them exactly as it left them. + + The bounded, honest cost of a rollback is **the delta** — rows written into the + consolidated DB after cutover do not flow back into the legacy files. For config tables + that self-heals: `SiteReconciliationActor` reports local inventory to central at startup + and fetches whatever it lacks, so a rolled-back node re-converges to central's truth + without operator action. For `sf_messages` the delta is **lost undelivered buffer** — the + practical mitigation is to drain the buffer before cutting over, which Task 20's live gate + and Task 21's runbook both call for. + + What has **no** rollback is a site pair split across versions — hence D5. That is a + deployment-procedure constraint, and Task 21 puts it in the runbook rather than leaving it + as tribal knowledge. + +- **Does the consolidated file stay appropriate?** One-DB-per-process was chosen partly + because Phase 1's tables were small. Adding config + S&F changes the size and write + profile of that single file. + + **ANSWERED — yes, and the soak is the evidence.** The write profile the consolidated file + must absorb is bounded on every axis: + + | Table | Rate bound | Row size | Basis | + |---|---|---|---| + | `sf_messages` | **≤50 writes/sec** (hard) | 76 B max on the rig | `SweepBatchLimit` ÷ `RetryTimerInterval` | + | `native_alarm_state` | `distinct_source_refs × 10/sec` | small | per-`SourceReference` coalescing on a 100 ms flush (`NativeAlarmActor.cs:473-502`) | + | config tables | deploy-driven, effectively idle | ~721 B max on the rig | Task 1 measurement | + + These are unremarkable for SQLite in WAL mode, and the soak ran the full generator load + against the Phase 1 consolidated file for 30 minutes with **zero** SQLite errors and no + oplog growth pathology. **One DB per process stays.** + + Two caveats carried into the plan rather than hidden here. First, the rig's config rows + are tiny (max 721 B) and **cannot** be treated as representative — the size ceiling that + matters comes from the documented ~60–70 KB production `config_json`, which is what + motivates D6 and `MaxBatchSize = 16`. Second, **D4's premise was wrong**: + `native_alarm_state` is *not* "unbounded by design" and *not* "by a wide margin the + highest-volume table" — the coalescing flush bounds it, and this rig has no alarm + generator at all (measured 0 rows), so its bound is analytic rather than observed. If a + production site ever shows alarm churn that swamps the shared oplog, the keyed-instances + hatch named in D4 remains the escape — it is simply not needed to start. + +## 6. Not in scope (unchanged from the design) + +`auditlog.db` (diverges per node by design — central pulls the union; replicating it would +double-forward), the secrets store (has its own answer in `Secrets.Replicator.SqlServer` +hub mode), and central nodes (SQL-Server-first, no LocalDb use case). diff --git a/docs/plans/2026-07-19-localdb-phase2-live-gate.md b/docs/plans/2026-07-19-localdb-phase2-live-gate.md new file mode 100644 index 00000000..1ee79c04 --- /dev/null +++ b/docs/plans/2026-07-19-localdb-phase2-live-gate.md @@ -0,0 +1,209 @@ +# LocalDb Phase 2 — live gate evidence (2026-07-20) + +Gate for [Task 20](2026-07-19-localdb-adoption-phase2.md). **Status: all 10 checks captured and +passing**, with two scope caveats recorded below (check 2's count is vacuous on its own; check 7's +third table was empty). + +Build under test: `feat/localdb-phase2` @ `166f07fa`, LocalDb `0.1.1` — confirmed live, not +assumed: the metrics scrape reports `otel_scope_name="ZB.MOM.WW.LocalDb.Replication" +otel_scope_version="0.1.1"`. Rig redeployed with `docker/deploy.sh`, exit 0, all 8 nodes up. + +## Method notes — two corrections to the plan + +**1. Do NOT run DB checks host-side against the bind mounts, as Task 20 instructs.** macOS↔container +locks do not cross virtiofs, so a host `sqlite3` open recovers the WAL out from under the container +and triggers a permanent `disk I/O error` (SQLITE_IOERR_SHORT_READ 522) storm — the root cause of +the 2026-07-20 incident. Every read below used a helper that copies the `db`/`-wal`/`-shm` triplet +and queries the **copy**; `cp` is a pure byte read with no SQLite involvement. + +**2. `docker exec … curl` cannot scrape the metrics** — the `aspnet:10.0` image has no `curl`, and +with `2>/dev/null` the failure is silent and looks exactly like "no metrics exported". An earlier +pass nearly recorded that as a finding. Use a network-sharing sidecar: + +```bash +docker run --rm --network container:scadabridge-site-a-a curlimages/curl:latest -s localhost:8084/metrics +``` + +**3. The probe harness must be delivered by `deploy artifacts`, not `instance deploy`.** External +systems reach a site only through `ArtifactDeploymentService` (`FetchGlobalArtifactsAsync` → +`GetAllExternalSystemsAsync`), which the per-instance deploy path does not invoke. A script that +names an external system that never arrived fails silently — no rows, no error. + +--- + +## 1. Migration ran — PASS + +| Node | `.migrated` markers | legacy `sf_messages` | consolidated `sf_messages` | +|---|---|---|---| +| site-a-node-a | `scadabridge.db.migrated`, `store-and-forward.db.migrated` | 11804 | 11804 | +| site-a-node-b | both | 11804 | 11804 | + +Both nodes migrated their own legacy files independently and the identical-content rows then +LWW-converged — expected per the plan, not an anomaly. + +**Caveat:** the migration itself ran on an earlier deploy of this build (markers dated 05:04), so +this is after-the-fact evidence. `deployed_configurations` was 0 in both legacy files, so that +table's migration path is untested here; check 3 exercises it fresh instead. + +## 2. No SMTP/notification rows migrated — PASS + +`smtp_configurations` and `notification_lists` are 0 on both nodes. + +**That count alone is vacuous** — the legacy source also held 0 SMTP rows, so a broken exclusion +would look identical. The non-vacuous evidence is structural: CDC capture triggers exist for +exactly these ten tables on both nodes — + +``` +OperationTracking, data_connection_definitions, database_connections, +deployed_configurations, external_systems, native_alarm_state, +sf_messages, shared_scripts, site_events, static_attribute_overrides +``` + +— and `smtp_configurations` / `notification_lists` have **no triggers at all**, so no replication +channel exists for them regardless of content. That is the property the security decision rests on. + +## 3. Config converges byte-identical — PASS + +Deployed `gate-sensor-1` (template 3) to site-a. Both nodes: + +- whole-row md5 identical: `0ec0f3cf790bfd65c45edf6c4afae50f` +- `deployment_id` `20b81607…`, `revision_hash` `sha256:253e1a56…`, + `deployed_at` `2026-07-20T09:06:11.2087221+00:00` +- `__localdb_row_version`: HLC `116951506695487488`, origin node + `cf61c688-a57f-4c5c-ba76-feff6e5679fe`, `is_tombstone` 0 — **identical on both** + +Identical HLC *and* origin id is the strong form: node B holds the row A wrote, not an +independently-derived copy of it. + +## 4. The standby made no config fetch — PASS + +Deploy window from `2026-07-20T09:05:52Z`: + +- `site-a-a` (active, deploy target): **1** fetch — `Fetching config for deployment 20b81607… + (notify-and-fetch)`. Legitimate: the active node pulling its own artifact. +- `site-a-b` (standby): **0** fetches. + +Under the old architecture `SiteReplicationActor` would have notified the standby and it would have +fetched independently. It now receives the config purely by CDC — corroborated from the other +direction by check 3's shared origin id. + +## 5. Store-and-forward converges — PASS + +Harness: external system `GateDeadTarget` → `http://127.0.0.1:9` with method `Ping`, plus an +interval script calling `ExternalSystem.CachedCall` (only `CachedCall` buffers; plain +`ExternalSystem.Call` does not). Delivered via `deploy artifacts`. + +- Buffered rows appear on **both** nodes with an identical rowset md5 `a484e369f6e0f00e7debff9b64464566`. +- `__localdb_row_version` for `sf_messages`: 2509 rows from **1 origin** on both nodes. +- The Pending→Parked transition of the pre-existing backlog also replicated identically + (10280 Pending / 1524 Parked on both), so **UPDATE** replicates, not just INSERT. + +## 6. Site failover — PASS + +Stopped the active node `site-a-a` at 09:18:37. Ten seconds later `site-a-b` logged +`InstanceActor started for gate-churn-1` and `ScriptActor GateBufferProbe started` — the standby +picked up the workload and kept buffering. While partitioned, node B's oplog rose to **4** unacked +entries. + +Restarted `site-a-a` at 09:18:57. After rejoin: + +| | node-a | node-b | +|---|---|---| +| probe rows | 48 | 48 | +| total `sf_messages` | 11852 | 11852 | +| oplog | 0 | 0 | +| dead letters | 0 | 0 | +| duplicate ids | 0 | 0 | +| rowset md5 | `54ded633c28134222bf034f3d0cec680` | `54ded633c28134222bf034f3d0cec680` | + +Buffered messages survived the flip, the backlog drained on rejoin, and **zero duplicate ids** on +either node is the exactly-once evidence. + +## 7. Cascade delete — PASS + +Seeded `gate-sensor-1` with a `static_attribute_overrides` row (via an `Instance.SetAttribute` +script — the attribute-override deploy path writes into the config JSON, not this table), confirmed +it replicated, then deleted the instance. + +| | pre-delete | post-delete | +|---|---|---| +| `deployed_configurations` | 1 / 1 | 0 / 0 | +| `static_attribute_overrides` | 1 / 1 | 0 / 0 | +| `native_alarm_state` | 0 / 0 | 0 / 0 | + +Crucially, **both nodes hold explicit tombstones** — `deployed_configurations=1`, +`static_attribute_overrides=1` in `__localdb_row_version WHERE is_tombstone=1`. The rows are not +merely absent on B; B applied the deletes. + +**Caveat:** `native_alarm_state` was empty for this instance, so the third leg of the cascade is +untested live. It is covered offline by +`LocalDbPhase2ConvergenceTests.RemovingAnInstance_ConvergesAllThreeCascadeTables`, which was +verified non-vacuous. + +## 8. Dead letters, oplog, metrics — PASS + +- `__localdb_oplog` = 0 and `__localdb_dead_letter` = 0 on both nodes at every sampling point. +- `localdb_oplog_depth` 0; `localdb_sync_applied_total` 11808 (a) / 224 (b); + `localdb_sync_reconnects_total` 1. + +## 9. Both nodes stopped and started together (D5) — PASS + +Stopped `site-a-a` and `site-a-b` together, started them together. + +| | node-a | node-b | +|---|---|---| +| `sf_messages` | 11845 | 11845 | +| probe rows | 41 | 41 | +| oplog | 0 | 0 | +| dead letters | 0 | 0 | + +Zero occurrences of `disk I/O error` / `database disk image` / `SQLITE_IOERR` / `corrupt` in either +node's log after restart. Clean rejoin. + +## 10. Drain under churn — PASS + +Sampled every 8 s under sustained probe load: + +``` +t=8s oplog_a=0 oplog_b=0 probe_a=23 probe_b=23 +t=16s oplog_a=0 oplog_b=0 probe_a=25 probe_b=25 +t=24s oplog_a=0 oplog_b=0 probe_a=26 probe_b=26 +t=32s oplog_a=0 oplog_b=0 probe_a=28 probe_b=28 +t=40s oplog_a=0 oplog_b=0 probe_a=30 probe_b=30 +t=48s oplog_a=0 oplog_b=0 probe_a=31 probe_b=31 +t=56s oplog_a=0 oplog_b=0 probe_a=33 probe_b=33 +t=64s oplog_a=0 oplog_b=0 probe_a=35 probe_b=35 +``` + +Writes climb steadily and the two nodes stay in lockstep at **every** sample, with no oplog +accumulation — flush plus ack outpaces the write rate, so no backlog forms. The stop condition the +plan cared about (monotonic growth that never drains) does not occur. + +**On its own this sampling is weak** — an 8 s interval is far coarser than the flush interval, so a +transient non-zero depth would be missed, and "always 0" is also what a dead pump looks like. The +rise-and-drain is proven instead by check 6, where the oplog demonstrably went 0 → 4 while the peer +was down and back to 0 after rejoin. + +--- + +## Observations outside the gate + +- **A central external-system delete does not remove the row from sites.** After deleting + `GateDeadTarget` centrally and re-running `deploy artifacts`, both site nodes still carry it. + Artifact application is an upsert with no reconciliation of removals. Pre-existing behaviour in + the artifact pipeline, unrelated to LocalDb — but it means site config tables accumulate orphans. +- **Deleting an instance orphans its buffered messages.** Removing the `soakgen-*` instances left + their 11,804 `sf_messages` with no tracking snapshot, producing a continuous + `Cached-telemetry drain: no tracking snapshot for …` warning flood. Also pre-existing, and worth + a cleanup path. + +## Rig state as left + +- Owed cleanup **done**: `SoakGenerator` (2021) and `soakgen-1..4` deleted; + `ExternalSystemDefinitions` id 1 restored to `http://scadabridge-restapi:5200` **and confirmed + propagated to both site nodes**. +- Gate harness removed: `gate-sensor-1`, `gate-churn-1`, template-3 scripts `GateBufferProbe` / + `GateSetStatic`, and the `GateDeadTarget` external system are deleted centrally. The site-side + `external_systems` row for `GateDeadTarget` remains, per the observation above. +- Remaining instances: `soak-motor-1..4` (template 4, not deployable) — untouched, as found. +- The ~11.8k orphaned `sf_messages` remain on both site-a nodes. diff --git a/docs/plans/2026-07-19-localdb-phase2-soak.md b/docs/plans/2026-07-19-localdb-phase2-soak.md new file mode 100644 index 00000000..58fa1460 --- /dev/null +++ b/docs/plans/2026-07-19-localdb-phase2-soak.md @@ -0,0 +1,348 @@ +# LocalDb Phase 2 — rig soak findings + +**Run date:** 2026-07-19 / 2026-07-20 (UTC) · **Rig:** local 8-node docker cluster, site-a pair +**Task:** Task 1 of [`2026-07-19-localdb-adoption-phase2.md`](2026-07-19-localdb-adoption-phase2.md) + +> ## GATE VERDICT: **STOP — do not proceed to Task 3.** *(superseded — see update)* +> +> Not for the reason the plan anticipated. Oplog sizing is fine and D6 is resolved. The soak +> instead surfaced what looked like a **pre-existing Phase 1 defect**: the consolidated LocalDb +> database (`site-localdb.db`) throws `SQLite Error 10: 'disk I/O error'` on essentially every +> write on the **active** node under sustained concurrent load. See +> [Finding 1](#finding-1-blocker). +> +> **Update 2026-07-20: Finding 1 is root-caused and is NOT a product defect.** The soak's own +> host-side `sqlite3` sampling poisoned both nodes (WAL reset across the virtiofs boundary); +> the unmodified code sustains the full soak load indefinitely with zero errors — +> reproduced/refuted on demand, see +> [`docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md`](../known-issues/2026-07-20-localdb-disk-io-error-under-load.md) §0. +> Finding 1 therefore no longer blocks Task 3. What still stands before proceeding: re-run the +> cut-short sampling (Findings 4/5 write-rate numbers) using the safe copy-based `snap` recipe +> now in the plan, on a rig where both site-a nodes have been restarted since any host-side read. +> +> ## GATE VERDICT (final, 2026-07-20): **PROCEED to Task 3.** +> +> The re-run is done — [§1b](#1b-clean-re-run-2026-07-20-post-root-cause). Safe copy-based +> sampling, both nodes restarted first, six clean 60-second intervals: **0.80 `sf_messages` +> rows/sec sustained, zero SQLite errors across the full window, both nodes converged.** No stop +> condition from any of D1–D6 is met. The one binding number Phase 2 must honour is +> **`LocalDb:Replication:MaxBatchSize = 16`** (Finding 2 / D6), which Task 19 sets. + +--- + +## 1. Method as actually executed + +The plan's method needed four corrections before it would run. Recorded here so the next run +does not rediscover them. + +| Plan said | Reality | +|---|---| +| `docker exec … curl -s localhost:8084/metrics` | **No `curl` in the `aspnet:10.0` image.** Use a sidecar sharing the container netns: `docker run --rm --network container:scadabridge-site-a-a curlimages/curl:latest -s localhost:8084/metrics` | +| Sample with host-side `sqlite3` against the bind mounts | **This IS the cause of Finding 1** (root-caused 2026-07-20 — the "ruled out" verdict below did not survive; see the known-issue doc §0). Host↔container POSIX locks don't propagate over virtiofs; a host-side read checkpoints + resets the WAL under the container and permanently poisons its connections. Copy the file triplet and query the copy, or read counters from `/metrics`. | +| Drive alarm churn on a deployed instance | **Not possible on this rig.** `infra/mssql/seed-config.sql` seeds zero `TemplateNativeAlarmSources`, opc-plc runs with no alarm flags, and no simulator or harness exists anywhere in the repo. `native_alarm_state` stayed at 0 rows throughout. Bounded analytically instead — see [Finding 3](#finding-3). | +| Drive S&F churn via template 4's `TestExternalSystem` script | Template 4 exists after a reseed but is **not deployable** (34 pre-deployment validation errors: 30 `ConnectionBinding` + 4 `ScriptCompilation`). A purpose-built `SoakGenerator` template was used instead — see below. | + +### The generator that worked + +`ExternalSystem.Call` does **not** buffer to store-and-forward in practice; `ExternalSystem.CachedCall` +is the buffering surface. This is the single most important operational detail for reproducing +S&F load. + +- Template `SoakGenerator` (id 2021), one `Interval` script at `{"intervalMs":5000}`: + ```csharp + var parms = new Dictionary { ["a"] = 2, ["b"] = 3 }; + await ExternalSystem.CachedCall("Test REST API", "Add", parms); + ``` +- No attributes, no compositions, no connection bindings — deliberately, so it deploys cleanly. +- `ExternalSystemDefinitions` id 1 repointed to `http://127.0.0.1:9` (discard port → connection + refused → classified transient → buffered). +- 4 instances (`soakgen-1..4`) deployed to site-a. + +Sustained rate observed: ~**2.9 HTTP attempts/sec** (688–864 connection-refused per 4 min). + +### Two rig-tooling bugs found and fixed en route + +1. **`docker/seed-sites.sh` seeded stale role names** — `Design`/`Deployment` instead of the + canonical `Designer`/`Deployer` (`src/ZB.MOM.WW.ScadaBridge.Security/Roles.cs:46-47`). Every + Designer/Deployer-gated management command failed `UNAUTHORIZED` on a freshly reseeded rig, + including `seed-sites.sh`'s own trailing `deploy artifacts` and `reseed.sh` stage 6d. + **Fixed** (commit `cf46e596`). +2. **`infra/mssql/setup.sql` never executes.** It is mounted into + `/docker-entrypoint-initdb.d/`, a convention the official `mcr.microsoft.com/mssql/server` + image does not implement. After `reseed.sh` drops the volume (`docker compose down -v`), + nothing recreates `ScadaBridgeConfig` or the `scadabridge_app` login, so `reseed.sh` hangs + forever on its "Waiting for setup.sql to create ScadaBridgeConfig" poll. Worked around by + applying the three init scripts by hand. **NOT yet fixed in the repo.** + +--- + +## Finding 1 (BLOCKER) — *root-caused 2026-07-20: observer-induced, not a product defect; see the gate-verdict update and the known-issue doc §0* + +### The Phase 1 consolidated LocalDb fails under sustained write load on the active node + +`site-localdb.db` throws `SQLite Error 10: 'disk I/O error'` on essentially every write once the +active node is under concurrent load. Both Phase 1 tables and the audit telemetry paths are +affected. + +Representative stacks (`docker logs scadabridge-site-a-b`): + +``` +Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 10: 'disk I/O error'. + at ZB.MOM.WW.ScadaBridge.SiteEventLogging.SiteEventLogger.ProcessWriteQueueAsync() + SiteEventLogger.cs:line 221/236 + +Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 10: 'disk I/O error'. + at ZB.MOM.WW.ScadaBridge.SiteRuntime.Tracking.OperationTrackingStore.RecordEnqueueAsync(...) + OperationTrackingStore.cs:line 137 + at ...CachedCallTelemetryForwarder.TryEmitTrackingAsync(...) line 148 +``` + +User-visible symptom: `[ERR] Failed to record event: script from ScriptActor:SoakCall` — **site +event logging is silently dropping events on the floor under load.** + +#### It follows the load, not the node, not the observer + +The failure was isolated by moving the load between nodes: + +| Node | Role | Under load | `disk I/O error` in 4 min | +|---|---|---|---| +| site-a-a | active | yes | 2 175 | +| site-a-a | standby (after restart) | no | **0** | +| site-a-b | standby | no | **0** | +| site-a-b | active (after failover) | yes | **4 391** | + +#### It is LocalDb-specific, not the filesystem — *wrong: sampling-selection bias; only the LocalDb file was ever host-read* + +The decisive control. Under identical load, on the same node, in the same bind-mounted +directory, counting error-stack frames over 3 minutes: + +| Store | Backing file | Errors | +|---|---|---| +| `OperationTrackingStore` | `site-localdb.db` (LocalDb) | 13 044 | +| `SiteAuditTelemetryActor` | `site-localdb.db` (LocalDb) | 4 350 | +| `SiteEventLogger` | `site-localdb.db` (LocalDb) | 900 | +| `CachedCallTelemetryForwarder` | `site-localdb.db` (LocalDb) | 162 | +| `StoreAndForwardStorage` | `store-and-forward.db` (legacy) | **0** | +| `SiteStorageService` | `scadabridge.db` (legacy) | **0** | + +Ordinary SQLite on the same bind mount is completely healthy. Only the LocalDb-managed +database fails. + +#### Ruling out the observer — **RETRACTED 2026-07-20: the observer was the cause** + +Onset (04:56:37) was one second after a host-side `sqlite3` sample (04:56:36), which made +observer-induced `-shm` corruption the leading hypothesis. The original run "excluded" it: +after node-a was restarted and the load failed over to node-b, node-b began erroring while no +host process touched its files, and sampled node-a went to zero once idle. + +**That exclusion was wrong.** The 04:56 sampling had poisoned *both* nodes' files (both carry +the 04:56 main-DB mtime; node-b's WAL was left at 0 bytes) — node-b was silent only because a +standby issues ~no LocalDb statements, and erupted on its first post-failover write. Verified +2026-07-20: 10+ min of full soak load on a freshly-reopened node with zero errors, then a +single host `sqlite3` read reset its 4.6 MiB WAL to 0 bytes and started the error storm one +second later (`SQLITE_IOERR_SHORT_READ`, 522). Full mechanism + minimal repro: +[`docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md`](../known-issues/2026-07-20-localdb-disk-io-error-under-load.md) §0. + +#### Secondary defect, same area + +``` +[ERROR][akka://scadabridge/user/site-audit-telemetry] There is no active ActorContext, +this is most likely due to use of async operations from within this actor. +Cause: System.NotSupportedException +``` + +`SiteAuditTelemetryActor` is closing over `ActorContext` across an `await`. Likely a +contributing cause rather than a separate issue — it is in the same write path — but it is a +real bug on its own terms. + +#### Why this blocks Phase 2 + +Phase 2 registers **eight further tables** into this database, including `native_alarm_state` +(the highest-volume table in either DB) and `sf_messages`. It also **deletes** the bespoke +mechanisms (`SiteReplicationActor`, `ReplicationService`) that currently carry that data +independently of LocalDb. Cutting over onto a store that cannot absorb the *current* write +load — and doing so in the same commit that removes the fallback — would convert a logging +defect into site-wide config and buffer loss. + +~~This is a Phase 1 defect. It must be root-caused and fixed before Phase 2's Task 3.~~ +**Root-caused 2026-07-20: not a Phase 1 defect** — observer-induced WAL reset across the +virtiofs bind-mount boundary; see the gate-verdict update at the top of this document. + +--- + +## Finding 2 — D6 resolved: no stop condition, but `MaxBatchSize` must be lowered + +The plan asserts `deployed_configurations.config_json` is "documented to exceed 128 KB per row." +That misreads the source. `docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md` +says the *escaped Akka envelope* exceeded the 128 KB frame, and that the default serializer +double-escapes the payload, so **"the raw flattened JSON only needs to be ~60-70 KB to blow the +128 KB frame."** + +So the largest known real production `config_json` is on the order of **60–70 KB**. + +Measured on the rig (4 deployed configs): **max 715 B, avg 714 B** — trivially small, as the plan +predicted, hence the production figure above is the one to size against. (A direct measurement +against wonder was attempted; `wonder-app-vd03.zmr.zimmer.com` resolves over the VPN but the +servecli SSH service on :2222 refuses connections, so the box was unreachable.) + +**Verdict: no stop condition.** Nothing approaches the 4 MB single-row ceiling. + +**But the batching risk is real.** Batching is row-count-only (`MaxBatchSize` default **500**), +and neither side configures gRPC message limits, so the 4 MB default receive cap applies: + +``` +500 rows × 70 KB ≈ 35 MB ≫ 4 MB → poison batch, stream wedged +``` + +Recommended for Task 19: + +``` +LocalDb:Replication:MaxBatchSize = 16 +``` + +`16 × 128 KB = 2 MB` — 2× headroom on row size over the known worst case, and 2× headroom +against the 4 MB cap. + +--- + +## Finding 3 — D4 corrected: alarm write rate is bounded, not unbounded + +The plan calls `native_alarm_state` "unbounded by design." It is not. +`NativeAlarmActor.MarkDirtyUpsert` (`NativeAlarmActor.cs:473-502`) coalesces into a dictionary +**keyed by `SourceReference`** and flushes on a timer (`_persistFlushInterval`, default +**100 ms**). One flush writes at most one row per distinct source reference, regardless of how +many transitions occurred in that window. + +``` +worst-case rows/sec = distinct_source_refs × 10 +``` + +An alarm storm on N sources costs N rows per 100 ms flush, not N × transition-rate. This makes +the table analytically sizeable without a generator — which matters, because this rig cannot +produce alarm load at all (see §1). + +**Not empirically validated.** `native_alarm_state` held 0 rows for the entire run. + +--- + +## Finding 4 — `sf_messages` has a hard structural ceiling of 50 rows/sec + +Confirmed by code rather than measurement, which is stronger here. The retry sweep takes at most +`SweepBatchLimit` messages every `RetryTimerInterval`, and each failed attempt is one `UPDATE` +incrementing `retry_count`: + +``` +SweepBatchLimit (500) ÷ RetryTimerInterval (10 s) = 50 row-writes/sec, hard ceiling +``` + +This confirms the plan's "~50 row-writes/sec worst case" — and it is a ceiling, not an estimate. +`DefaultMaxRetries = 50` at `DefaultRetryInterval = 30 s` bounds each message to 50 updates over +25 minutes. + +Observed during the run: ~2.9 attempts/sec, far below the ceiling. Row counts could not be +sampled reliably (see §1) and the run was cut short by Finding 1. + +--- + +## Finding 5 — exceeding the oplog caps is a graceful degradation, not a failure + +The plan's Task 1 step 7 treats "the shared oplog cannot absorb this write profile" as a hard +stop requiring the keyed-instances escape hatch. It is much weaker than that. + +`OplogStore.EnforceCapsAsync` (`OplogStore.cs:109-138`) prunes to the ceiling and sets +`needs_snapshot`; `MaintenanceBackgroundService.cs:57` logs *"Oplog backlog/age caps exceeded: +pruned to the ceiling and flagged needs_snapshot — the peer must snapshot-resync."* +`SyncSession.ComputeSnapshotRequiredAsync` (`:413-415`) then forces a snapshot resync. + +So overrunning the caps costs a **full snapshot resync**, not data loss and not a wedged stream. +The caps therefore express *"how long may a peer be absent before it needs a full resync"*, and +should be sized to the longest tolerable peer outage rather than treated as a correctness +boundary. In healthy two-node operation the oplog drains continuously — `localdb_oplog_depth` +read **0** throughout. + +### Provisional cap recommendation (Task 19) + +Sized for a ~3-hour peer outage at a conservative 100 rows/sec aggregate, pending re-measurement +after Finding 1 is fixed: + +``` +LocalDb:Replication:MaxOplogRows = 1000000 # default; ≈2.8 h at 100 rows/s +LocalDb:Replication:MaxOplogAge = 3.00:00:00 +LocalDb:Replication:MaxBatchSize = 16 # Finding 2 — this one is NOT optional +``` + +Only `MaxBatchSize` is firmly evidence-backed. The other two rest on an assumed aggregate write +rate that this run could not measure. + +--- + +## 1b. Clean re-run (2026-07-20, post-root-cause) + +The original sampling was cut short by Finding 1 and was itself the cause of it. Re-run after +both site-a nodes were restarted, using the plan's copy-based `snap()` helper — **no host-side +`sqlite3` ever touched a live file.** Six consecutive 60-second intervals, generator load +unchanged (`SoakGenerator` ×4 against a refusing endpoint): + +| UTC | `sf_messages` rows | `SUM(retry_count)` | status=0 | `__localdb_oplog` | `native_alarm_state` | +|---|---|---|---|---|---| +| 06:07:57 | 3384 | 200 | 3380 | 0 | 0 | +| 06:08:57 | 3432 | 200 | 3428 | 0 | 0 | +| 06:09:57 | 3480 | 200 | 3476 | 0 | 0 | +| 06:10:57 | 3528 | 200 | 3524 | 0 | 0 | +| 06:11:57 | 3576 | 200 | 3572 | 0 | 0 | +| 06:12:57 | 3624 | 200 | 3620 | 0 | 0 | +| 06:13:57 | 3672 | 200 | 3668 | 0 | 0 | + +**Measured:** + +- **`sf_messages` insert rate: exactly 48 rows/min = 0.80 rows/sec**, dead steady across all six + intervals (+288 rows over 360 s, zero variance). +- **Retry-UPDATE rate: 0/sec.** `SUM(retry_count)` never moved. Only 4 rows ever reached + `retry_count = 50` (status 2, dead-lettered); the other ~3.6 k sit at `retry_count = 0`, + status 0 — **enqueued but never swept**. So this generator exercises the *insert* path only. +- **Row sizes:** `sf_messages.payload_json` max **76 B**; `deployed_configurations.config_json` + max **721 B** over 4 rows (rig config is trivially small — see the caveat below). +- **`native_alarm_state`: 0 rows** — no alarm generator on this rig, as previously recorded. +- **`__localdb_oplog`: 0 throughout** — the Phase 1 tables are genuinely idle under this load, + which is the expected result and not a measurement failure (Phase 2's tables are not yet + registered, so this load cannot reach the oplog by construction). +- **Zero SQLite errors** in `docker logs scadabridge-site-a-a` across the full 30-minute window + (`grep -ciE "disk I/O|SQLITE_IOERR|NOTADB"` → **0**). This is the direct confirmation that + Finding 1 was observer-induced: identical load, identical library, nothing host-reading the + files, no errors. +- **Both site-a nodes converged identically** (3564 rows on each at a common sample point) under + the bespoke replicator — the pre-cutover baseline Task 20 should reproduce under CDC. + +**Honest limits of this measurement.** 0.80 rows/sec is **~1.6 % of the 50 rows/sec structural +ceiling**, and the retry path — the expensive one, one `UPDATE` per message per sweep — was never +exercised at all. This re-run therefore confirms *steady-state health and the absence of the +Finding 1 pathology*; it does **not** probe the ceiling. That is acceptable because the ceiling is +**structural rather than empirical** (Finding 4: `SweepBatchLimit` ÷ `RetryTimerInterval`), so +sizing does not depend on observing it. Likewise the rig's 721 B config rows are **not** +representative — D6's sizing rests on the documented ~60–70 KB production `config_json`, not on +this number. + +## 2. What still owes measurement + +Carry forward (items 1–2 partially discharged by §1b above): + +1. ~~Sustained `sf_messages` rows/sec~~ — **measured** (§1b: 0.80/s insert, 0/s retry). Still + unmeasured: the **retry-UPDATE** path under a saturating generator approaching the 50/s + ceiling. Not required for sizing (the ceiling is structural), but it is the honest gap. +2. Any `native_alarm_state` measurement at all — requires building alarm-source seeding plus an + A&C-capable server. The `OpcUaAlarmLiveSmokeTests` **passed**, so the rig's opc-plc *does* + answer ConditionRefresh with a `SnapshotComplete`; the missing piece is ongoing transitions + and a seeded `TemplateNativeAlarmSource`. (The test's own doc comment claiming the simulator + "does not reliably expose A&C" is stale.) +3. A production-representative `config_json` from wonder, to replace the ~60–70 KB inference. +4. The empirical oplog drain-under-churn check — Task 20 evidence item 10. + +## 3. Rig state left behind + +- Fully reseeded (central config volume dropped and replayed; site SQLite state wiped by + `reseed.sh` stage 2). +- `ExternalSystemDefinitions` id 1 is **repointed to `http://127.0.0.1:9`** — restore to + `http://scadabridge-restapi:5200` before using the rig for anything else. +- Template `SoakGenerator` (2021) and instances `soakgen-1..4` (ids 5–8) remain deployed on + site-a and are **still generating load**. Undeploy or delete them before the Task 20 live gate. +- `LdapGroupMappings` corrected in the live DB to the canonical role names. diff --git a/docs/requirements/Component-StoreAndForward.md b/docs/requirements/Component-StoreAndForward.md index f10f255f..3b831688 100644 --- a/docs/requirements/Component-StoreAndForward.md +++ b/docs/requirements/Component-StoreAndForward.md @@ -80,7 +80,7 @@ There is **no maximum buffer size**. Messages accumulate in the buffer until del - The standby node applies the same operations to its own local SQLite database but is **passive**: it never runs the delivery sweep. The retry sweep is **gated to the active node** (the oldest Up member / singleton host, re-evaluated every sweep tick), so only one node delivers at a time. The standby applies replicated operations purely to keep its copy warm for a future failover. - On failover, the new active node has a near-complete copy of the buffer. In rare cases, the most recent operations may not have been replicated (e.g., a message added or removed just before failover). This can result in a few **duplicate deliveries** (message delivered but its `Remove` not yet replicated) or a few **missed retries** (message added but not replicated). Duplicate deliveries are therefore confined to the **failover window** — an in-flight delivery whose `Remove` had not yet replicated — and never occur in steady-state operation (the standby's gate keeps it from delivering the same rows). Both are acceptable trade-offs for the latency benefit. - On failover, the new active node's gate flips to active within one sweep interval and it resumes delivery from its local copy. -- **Peer-join anti-entropy resync (chunked, ack-confirmed).** Asynchronous, no-ack replication keeps the standby warm in steady state, but a standby that was **down for an extended period** (a crash, a long maintenance window) misses every operation replicated while it was gone and would otherwise diverge from the active node's buffer forever. To close that gap, whenever a node **(re)tracks its peer**, a **standby** requests a full-buffer snapshot (`RequestSfBufferResync`); the **active** node loads up to `MaxResyncRows` (10 000) of its oldest rows and answers with a **sequence of byte-budgeted chunks** (`SfBufferSnapshotChunk`), each carrying a shared `ResyncId`, a 1-based `Sequence`, and the `TotalChunks` count. Chunking is mandatory because the monolithic snapshot exceeds Akka remoting's **default 128 000-byte frame** for any realistic backlog and `BuildHocon` sets no override — a single oversized message is silently undeliverable (review 02 round 2, **N2 High**). Rows accumulate into a chunk until the estimated payload budget (`MaxResyncChunkBytes` = 64 000, ≈50% frame headroom) or the row cap (`MaxResyncChunkRows` = 200) is hit; a single row whose payload alone exceeds the budget ships solo with a Warning. The standby **assembles all chunks of one `ResyncId`** (a new `ResyncId` discards any stale partial assembly; a partial that never completes is dropped after `resyncAssemblyTimeout`, default 30 s, and counted a replication failure), then **replaces its entire local buffer** with the assembled snapshot (`ReplaceAllAsync`, one transaction) and returns a delivery confirmation (`SfBufferResyncAck`). The active node arms an ack window (`resyncAckTimeout`, default 60 s): an acknowledged resync increments `scadabridge.store_and_forward.resync.completed`; an unacknowledged one (lost chunks / dead peer) logs a Warning and increments `scadabridge.store_and_forward.resync.ack_missing` — closing N2's silent-loss mode (previously nothing counted a lost snapshot and nothing retried until the next peer-track). Only the active node answers; only a standby applies, and both sides re-check at apply time (a mid-flight active-flip aborts the wipe) — each side checks the repo-standard **oldest-Up member** active-node predicate (singleton-host semantics via the shared `ActiveNodeEvaluator`, the **same predicate as the S&F delivery gate**; review 02 round 2, **N1 Critical** — using cluster *leadership* here let a rolling restart of the lower-address node make the delivering node wipe its own live buffer). Because replicated applies are **upserts** (see the replication apply path), any Add/Remove/Park that lands after the resync merges cleanly onto the resynced state — no primary-key conflict, no lost delta; the one accepted exception is the rare **N5** orphan-row race (a replicated `Remove` ordered before the snapshot chunks can re-add the removed row, re-delivered once and self-corrected at the next resync — inherent to no-ack replication). If the buffer exceeds the 10 000-row cap the snapshot is flagged `Truncated` and the standby logs a Warning; the residual divergence beyond the cap drains naturally as the active node delivers. The legacy monolithic `SfBufferSnapshot` message + standby handler are **retained** for rolling-upgrade compatibility (an old active node's monolithic snapshot is still applied by a new standby). +- **Peer-join anti-entropy resync (LocalDb CDC).** *(Rewritten for LocalDb Phase 2, 2026-07-20. The previous specification of a chunked, ack-confirmed `SfBufferSnapshotChunk` protocol described the bespoke `ReplicationService`, which Phase 2 deleted. The discussion is rewritten rather than removed, because the failure modes it reasoned about still exist — they are simply bounded differently now.)* The buffer lives in the consolidated LocalDb database as the replicated `sf_messages` table, and both nodes exchange changes over a gRPC sync stream rather than Akka remoting. A node that was down for an extended period no longer requests a full-buffer snapshot and **replaces** its local buffer; LocalDb's snapshot resync merges **per row under last-writer-wins and never deletes**. Several of the old hazards are therefore structurally gone rather than guarded against: **(a) the 128 000-byte Akka frame limit no longer applies** — the transport is gRPC, whose successor ceiling is the 4 MB default receive limit, managed by bounding `LocalDb:Replication:MaxBatchSize` (set to 16 on the rig, sized against a ~70 KB worst-case `config_json`; see the Phase 2 plan, D6). Chunking, `ResyncId` assembly, assembly timeouts, and the truncation flag are all retired with it. **(b) The N1 directional-authority hazard is gone.** That guard existed because the bespoke resync applied a destructive delete-all-then-insert-all, so running it in the wrong direction wiped a live buffer. With a non-destructive merge there is no wipe to gate, and replication is symmetric — either node may write. `ActiveNodeEvaluator` survives, but only for the **delivery** gate and the heartbeat, which still genuinely need a single active node. **(c) The N5 orphan-row race is gone.** A `Remove` ordered before a re-add can no longer resurrect a row: deletes are tombstoned with an HLC, and a tombstone beats any older write for the same key. **The duplicate-delivery bound, stated explicitly.** Delivery remains single-node: only the primary runs the sweep (`IClusterNodeProvider.SelfIsPrimary`). A message can therefore be delivered twice only when the OLD primary delivered it and the resulting status change had not yet replicated at the instant the gate flipped. The window is one replication flush interval plus the in-flight ack, not an unbounded divergence — and unlike the old model it does not grow with backlog depth or with how long a node was absent. **One new bound replaces the old ones:** a node offline longer than `LocalDb:Replication:TombstoneRetention` (default 7 days) may resurrect deleted rows on rejoin, because the tombstones that would have suppressed them have been pruned. Stop-and-start a site pair together, and do not leave one node of a pair offline across that horizon. ### Operation Tracking Table (lives in Site Runtime, not here) diff --git a/infra/docker-compose.yml b/infra/docker-compose.yml index cd32c86d..1d06f4e7 100644 --- a/infra/docker-compose.yml +++ b/infra/docker-compose.yml @@ -68,6 +68,9 @@ services: MSSQL_PID: "Developer" volumes: - scadabridge-mssql-data:/var/opt/mssql + # NOTE: the official mssql/server image does NOT run + # /docker-entrypoint-initdb.d — these mounts are informational only; + # infra/reseed.sh applies the scripts explicitly via sqlcmd. - ./mssql/setup.sql:/docker-entrypoint-initdb.d/setup.sql:ro - ./mssql/machinedata_seed.sql:/docker-entrypoint-initdb.d/machinedata_seed.sql:ro - ./mssql/setup-env2.sql:/docker-entrypoint-initdb.d/setup-env2.sql:ro diff --git a/infra/reseed.sh b/infra/reseed.sh index dfae57d7..50257d91 100755 --- a/infra/reseed.sh +++ b/infra/reseed.sh @@ -82,12 +82,15 @@ if ! $SKIP_TEARDOWN; then done echo " MSSQL ready." - echo " Waiting for setup.sql to create ScadaBridgeConfig..." - until docker exec scadabridge-mssql /opt/mssql-tools18/bin/sqlcmd \ - -S localhost -U sa -P 'ScadaBridge_Dev1#' -C \ - -Q "IF DB_ID('ScadaBridgeConfig') IS NULL THROW 50000, 'not ready', 1;" \ - >/dev/null 2>&1; do - sleep 2 + # The official mcr.microsoft.com/mssql/server image does NOT implement + # /docker-entrypoint-initdb.d, so the compose-mounted init scripts never run + # on their own — waiting for them here hangs forever on a fresh volume. + # Apply them explicitly instead (all are idempotent). + echo " Applying MSSQL init scripts (the mssql/server image has no initdb hook)..." + for f in mssql/setup.sql mssql/machinedata_seed.sql mssql/setup-env2.sql; do + echo " $f" + docker exec -i scadabridge-mssql /opt/mssql-tools18/bin/sqlcmd \ + -S localhost -U sa -P 'ScadaBridge_Dev1#' -C -b < "$SCRIPT_DIR/$f" done echo " ScadaBridgeConfig present." diff --git a/nuget.config b/nuget.config index e8ad4467..8b8ef126 100644 --- a/nuget.config +++ b/nuget.config @@ -27,6 +27,8 @@ + + - + + diff --git a/tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests.csproj b/tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests.csproj index 15dc7075..fb70a064 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests.csproj +++ b/tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests.csproj @@ -40,4 +40,28 @@ + + + + + \ No newline at end of file diff --git a/tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/DatabaseGatewayTests.cs b/tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/DatabaseGatewayTests.cs index 7ecba161..0971d130 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/DatabaseGatewayTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/DatabaseGatewayTests.cs @@ -3,18 +3,56 @@ using System.Data.Common; using System.Text.Json; using Microsoft.Extensions.Logging.Abstractions; using NSubstitute; +using ZB.MOM.WW.LocalDb; using ZB.MOM.WW.ScadaBridge.Commons.Entities.ExternalSystems; using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories; +using ZB.MOM.WW.ScadaBridge.TestSupport; namespace ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests; /// /// WP-9: Tests for Database access — connection resolution, cached writes. /// -public class DatabaseGatewayTests +public class DatabaseGatewayTests : IDisposable { private readonly IExternalSystemRepository _repository = Substitute.For(); + /// + /// Temp local databases opened by the Store-and-Forward tests, torn down in + /// . + /// + private readonly List _localDbs = new(); + + /// + /// Opens a real ILocalDb over a fresh temp file for a test that needs a live + /// . The + /// store takes an ILocalDb and LocalDb has no in-memory mode, so each test gets + /// its own file rather than a shared-cache in-memory database held open by a + /// keep-alive connection. + /// + private TestLocalDb CreateLocalDb(string prefix) + { + var localDb = TestLocalDb.CreateTemp(prefix); + _localDbs.Add(localDb); + return localDb; + } + + /// + /// Disposes every temp local database and then deletes its files — in that order, + /// because the master connection LocalDb holds anchors the WAL sidecars. + /// + public void Dispose() + { + foreach (var localDb in _localDbs) + { + var path = localDb.Path; + localDb.Dispose(); + TestLocalDb.DeleteFiles(path); + } + + GC.SuppressFinalize(this); + } + /// /// Configures the repository substitute for the name-keyed connection-resolution /// path used by DatabaseGateway (ExternalSystemGateway-011). A null @@ -84,12 +122,9 @@ public class DatabaseGatewayTests }; StubConnection(conn); - var dbName = $"EsgCachedWrite_{Guid.NewGuid():N}"; - var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared"; - using var keepAlive = new Microsoft.Data.Sqlite.SqliteConnection(connStr); - keepAlive.Open(); + var localDb = CreateLocalDb("EsgCachedWrite"); var storage = new ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardStorage( - connStr, NullLogger.Instance); + localDb.Db, NullLogger.Instance); await storage.InitializeAsync(); var sfOptions = new ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardOptions { @@ -121,7 +156,7 @@ public class DatabaseGatewayTests var depth = await storage.GetBufferDepthByCategoryAsync(); Assert.Equal(1, depth[ZB.MOM.WW.ScadaBridge.Commons.Types.Enums.StoreAndForwardCategory.CachedDbWrite]); - var buffered = ReadBufferedRetrySettings(connStr); + var buffered = ReadBufferedRetrySettings(localDb.Db); Assert.Equal(5, buffered.MaxRetries); Assert.Equal((long)TimeSpan.FromSeconds(12).TotalMilliseconds, buffered.RetryIntervalMs); @@ -148,12 +183,9 @@ public class DatabaseGatewayTests }; StubConnection(conn); - var dbName = $"EsgCachedWriteZero_{Guid.NewGuid():N}"; - var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared"; - using var keepAlive = new Microsoft.Data.Sqlite.SqliteConnection(connStr); - keepAlive.Open(); + var localDb = CreateLocalDb("EsgCachedWriteZero"); var storage = new ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardStorage( - connStr, NullLogger.Instance); + localDb.Db, NullLogger.Instance); await storage.InitializeAsync(); var sfOptions = new ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardOptions { @@ -172,7 +204,7 @@ public class DatabaseGatewayTests await gateway.CachedWriteAsync("testDb", "INSERT INTO t VALUES (1)"); - var (maxRetries, _, _, _) = ReadBufferedRetrySettings(connStr); + var (maxRetries, _, _, _) = ReadBufferedRetrySettings(localDb.Db); // Must be the bounded S&F default, never 0 — a stored 0 would mean retry-forever. Assert.Equal(99, maxRetries); Assert.NotEqual(0, maxRetries); @@ -182,19 +214,15 @@ public class DatabaseGatewayTests // cached-write attempt + the buffered retry path ── /// - /// Builds a real, initialised in-memory store-and-forward service plus a - /// keep-alive connection (the SQLite shared-cache DB lives only while a - /// connection is open). The caller disposes . + /// Builds a real, initialised store-and-forward service over a fresh temp local + /// database. The database is torn down by . /// - private static (ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardService Sf, string ConnStr, Microsoft.Data.Sqlite.SqliteConnection KeepAlive) + private (ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardService Sf, ILocalDb Db) NewStoreAndForward() { - var dbName = $"EsgCachedWriteClassify_{Guid.NewGuid():N}"; - var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared"; - var keepAlive = new Microsoft.Data.Sqlite.SqliteConnection(connStr); - keepAlive.Open(); + var localDb = CreateLocalDb("EsgCachedWriteClassify"); var storage = new ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardStorage( - connStr, NullLogger.Instance); + localDb.Db, NullLogger.Instance); storage.InitializeAsync().GetAwaiter().GetResult(); var sfOptions = new ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardOptions { @@ -204,7 +232,7 @@ public class DatabaseGatewayTests }; var sf = new ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardService( storage, sfOptions, NullLogger.Instance); - return (sf, connStr, keepAlive); + return (sf, localDb.Db); } [Fact] @@ -217,8 +245,7 @@ public class DatabaseGatewayTests var conn = new DatabaseConnectionDefinition("testDb", "Server=localhost;Database=test") { Id = 1 }; StubConnection(conn); - var (sf, connStr, keepAlive) = NewStoreAndForward(); - using var _ = keepAlive; + var (sf, sfDb) = NewStoreAndForward(); var gateway = new ExecuteStubGateway( _repository, @@ -233,7 +260,7 @@ public class DatabaseGatewayTests Assert.NotNull(result.ErrorMessage); // Nothing buffered — the permanent failure short-circuited S&F. - Assert.Equal(0, ReadBufferDepth(connStr)); + Assert.Equal(0, ReadBufferDepth(sfDb)); } [Fact] @@ -249,8 +276,7 @@ public class DatabaseGatewayTests }; StubConnection(conn); - var (sf, connStr, keepAlive) = NewStoreAndForward(); - using var _ = keepAlive; + var (sf, sfDb) = NewStoreAndForward(); var gateway = new ExecuteStubGateway( _repository, @@ -265,7 +291,7 @@ public class DatabaseGatewayTests Assert.True(result.WasBuffered); // handed to S&F, not synchronously failed Assert.Null(result.ErrorMessage); - Assert.Equal(1, ReadBufferDepth(connStr)); + Assert.Equal(1, ReadBufferDepth(sfDb)); } [Fact] @@ -277,8 +303,7 @@ public class DatabaseGatewayTests var conn = new DatabaseConnectionDefinition("testDb", "Server=localhost;Database=test") { Id = 1 }; StubConnection(conn); - var (sf, connStr, keepAlive) = NewStoreAndForward(); - using var _ = keepAlive; + var (sf, sfDb) = NewStoreAndForward(); var gateway = new ExecuteStubGateway(_repository, sf, onExecute: () => { /* succeeds */ }); @@ -288,7 +313,7 @@ public class DatabaseGatewayTests Assert.False(result.WasBuffered); Assert.Null(result.ErrorMessage); - Assert.Equal(0, ReadBufferDepth(connStr)); + Assert.Equal(0, ReadBufferDepth(sfDb)); } [Fact] @@ -381,8 +406,7 @@ public class DatabaseGatewayTests }; StubConnection(conn); - var (sf, connStr, keepAlive) = NewStoreAndForward(); - using var _ = keepAlive; + var (sf, sfDb) = NewStoreAndForward(); // RawExecuteStubGateway routes the raw throw through the PRODUCTION // ExecuteWriteAsync classification (the seam under test), unlike @@ -395,7 +419,7 @@ public class DatabaseGatewayTests Assert.True(result.WasBuffered); // handed to S&F as transient Assert.Null(result.ErrorMessage); // not a permanent Failed result - Assert.Equal(1, ReadBufferDepth(connStr)); + Assert.Equal(1, ReadBufferDepth(sfDb)); } [Fact] @@ -408,8 +432,7 @@ public class DatabaseGatewayTests var conn = new DatabaseConnectionDefinition("testDb", "Server=localhost;Database=test") { Id = 1 }; StubConnection(conn); - var (sf, connStr, keepAlive) = NewStoreAndForward(); - using var _ = keepAlive; + var (sf, sfDb) = NewStoreAndForward(); using var cts = new CancellationTokenSource(); cts.Cancel(); @@ -421,7 +444,7 @@ public class DatabaseGatewayTests () => gateway.CachedWriteAsync("testDb", "INSERT INTO t VALUES (1)", cancellationToken: cts.Token)); // Cancellation is not a transient failure — nothing must have been buffered. - Assert.Equal(0, ReadBufferDepth(connStr)); + Assert.Equal(0, ReadBufferDepth(sfDb)); } [Fact] @@ -434,8 +457,7 @@ public class DatabaseGatewayTests var conn = new DatabaseConnectionDefinition("testDb", "Server=localhost;Database=test") { Id = 1 }; StubConnection(conn); - var (sf, connStr, keepAlive) = NewStoreAndForward(); - using var _ = keepAlive; + var (sf, sfDb) = NewStoreAndForward(); var gateway = new RawExecuteStubGateway( _repository, sf, onRunSql: () => throw new ArgumentException("authoring bug")); @@ -443,7 +465,7 @@ public class DatabaseGatewayTests await Assert.ThrowsAsync( () => gateway.CachedWriteAsync("testDb", "INSERT INTO t VALUES (1)")); - Assert.Equal(0, ReadBufferDepth(connStr)); + Assert.Equal(0, ReadBufferDepth(sfDb)); } [Fact] @@ -486,8 +508,7 @@ public class DatabaseGatewayTests var conn = new DatabaseConnectionDefinition("testDb", "Server=localhost;Database=test") { Id = 1 }; StubConnection(conn); - var (sf, connStr, keepAlive) = NewStoreAndForward(); - using var _ = keepAlive; + var (sf, sfDb) = NewStoreAndForward(); using var cts = new CancellationTokenSource(); cts.Cancel(); @@ -505,7 +526,7 @@ public class DatabaseGatewayTests // The cancel won — it must NOT have been classified as transient (buffered) // nor returned as a permanent Failed result. - Assert.Equal(0, ReadBufferDepth(connStr)); + Assert.Equal(0, ReadBufferDepth(sfDb)); } /// @@ -542,10 +563,10 @@ public class DatabaseGatewayTests /// Reads the current buffered-message count off the S&F SQLite DB by /// counting sf_messages rows (the engine's persistence table). /// - private static int ReadBufferDepth(string connStr) + private static int ReadBufferDepth(ILocalDb localDb) { - using var conn = new Microsoft.Data.Sqlite.SqliteConnection(connStr); - conn.Open(); + // CreateConnection hands back an already-open, pragma-configured connection. + using var conn = localDb.CreateConnection(); using var cmd = conn.CreateCommand(); cmd.CommandText = "SELECT COUNT(*) FROM sf_messages"; return Convert.ToInt32(cmd.ExecuteScalar()); @@ -615,10 +636,10 @@ public class DatabaseGatewayTests } private static (int MaxRetries, long RetryIntervalMs, Guid? ExecutionId, string? SourceScript) - ReadBufferedRetrySettings(string connStr) + ReadBufferedRetrySettings(ILocalDb localDb) { - using var conn = new Microsoft.Data.Sqlite.SqliteConnection(connStr); - conn.Open(); + // CreateConnection hands back an already-open, pragma-configured connection. + using var conn = localDb.CreateConnection(); using var cmd = conn.CreateCommand(); cmd.CommandText = "SELECT max_retries, retry_interval_ms, execution_id, source_script FROM sf_messages"; diff --git a/tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/ExternalSystemClientTests.cs b/tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/ExternalSystemClientTests.cs index 206c592f..ebea6eca 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/ExternalSystemClientTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/ExternalSystemClientTests.cs @@ -1,24 +1,60 @@ using System.Net; using System.Net.Http.Headers; using System.Text; -using Microsoft.Data.Sqlite; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Logging.Abstractions; using NSubstitute; +using ZB.MOM.WW.LocalDb; using ZB.MOM.WW.ScadaBridge.Commons.Entities.ExternalSystems; using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories; using ZB.MOM.WW.ScadaBridge.StoreAndForward; +using ZB.MOM.WW.ScadaBridge.TestSupport; namespace ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests; /// /// WP-6/7: Tests for ExternalSystemClient — HTTP client, call modes, error handling. /// -public class ExternalSystemClientTests +public class ExternalSystemClientTests : IDisposable { private readonly IExternalSystemRepository _repository = Substitute.For(); private readonly IHttpClientFactory _httpClientFactory = Substitute.For(); + /// + /// Temp local databases opened by the Store-and-Forward tests, torn down in + /// . + /// + private readonly List _localDbs = new(); + + /// + /// Opens a real ILocalDb over a fresh temp file for a test that needs a live + /// . The store takes an ILocalDb and LocalDb + /// has no in-memory mode, so each test gets its own file rather than a shared-cache + /// in-memory database held open by a keep-alive connection. + /// + private TestLocalDb CreateLocalDb(string prefix) + { + var localDb = TestLocalDb.CreateTemp(prefix); + _localDbs.Add(localDb); + return localDb; + } + + /// + /// Disposes every temp local database and then deletes its files — in that order, + /// because the master connection LocalDb holds anchors the WAL sidecars. + /// + public void Dispose() + { + foreach (var localDb in _localDbs) + { + var path = localDb.Path; + localDb.Dispose(); + TestLocalDb.DeleteFiles(path); + } + + GC.SuppressFinalize(this); + } + /// /// Configures the repository substitute for the name-keyed resolution path used by /// ExternalSystemClient (ExternalSystemGateway-011). A null system or @@ -275,11 +311,8 @@ public class ExternalSystemClientTests _httpClientFactory.CreateClient(Arg.Any()).Returns(httpClient); // A real S&F service with a registered delivery handler that counts invocations. - var dbName = $"EsgDoubleDispatch_{Guid.NewGuid():N}"; - var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared"; - using var keepAlive = new SqliteConnection(connStr); - keepAlive.Open(); - var storage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + var localDb = CreateLocalDb("EsgDoubleDispatch"); + var storage = new StoreAndForwardStorage(localDb.Db, NullLogger.Instance); await storage.InitializeAsync(); var sfOptions = new StoreAndForwardOptions { @@ -422,11 +455,8 @@ public class ExternalSystemClientTests var httpClient = new HttpClient(new MockHttpMessageHandler(HttpStatusCode.InternalServerError, "boom")); _httpClientFactory.CreateClient(Arg.Any()).Returns(httpClient); - var dbName = $"EsgRetry_{Guid.NewGuid():N}"; - var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared"; - using var keepAlive = new SqliteConnection(connStr); - keepAlive.Open(); - var storage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + var localDb = CreateLocalDb("EsgRetry"); + var storage = new StoreAndForwardStorage(localDb.Db, NullLogger.Instance); await storage.InitializeAsync(); // S&F defaults deliberately different from the system's settings. var sfOptions = new StoreAndForwardOptions @@ -454,7 +484,7 @@ public class ExternalSystemClientTests var depth = await storage.GetBufferDepthByCategoryAsync(); Assert.Equal(1, depth[ZB.MOM.WW.ScadaBridge.Commons.Types.Enums.StoreAndForwardCategory.ExternalSystem]); - var buffered = ReadBufferedRetrySettings(connStr); + var buffered = ReadBufferedRetrySettings(localDb.Db); Assert.Equal(7, buffered.MaxRetries); Assert.Equal((long)TimeSpan.FromSeconds(42).TotalMilliseconds, buffered.RetryIntervalMs); @@ -466,10 +496,10 @@ public class ExternalSystemClientTests } private static (int MaxRetries, long RetryIntervalMs, Guid? ExecutionId, string? SourceScript) - ReadBufferedRetrySettings(string connStr) + ReadBufferedRetrySettings(ILocalDb localDb) { - using var conn = new SqliteConnection(connStr); - conn.Open(); + // CreateConnection hands back an already-open, pragma-configured connection. + using var conn = localDb.CreateConnection(); using var cmd = conn.CreateCommand(); cmd.CommandText = "SELECT max_retries, retry_interval_ms, execution_id, source_script FROM sf_messages"; @@ -505,11 +535,8 @@ public class ExternalSystemClientTests var httpClient = new HttpClient(new MockHttpMessageHandler(HttpStatusCode.InternalServerError, "boom")); _httpClientFactory.CreateClient(Arg.Any()).Returns(httpClient); - var dbName = $"EsgRetryZero_{Guid.NewGuid():N}"; - var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared"; - using var keepAlive = new SqliteConnection(connStr); - keepAlive.Open(); - var storage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + var localDb = CreateLocalDb("EsgRetryZero"); + var storage = new StoreAndForwardStorage(localDb.Db, NullLogger.Instance); await storage.InitializeAsync(); var sfOptions = new StoreAndForwardOptions { @@ -525,7 +552,7 @@ public class ExternalSystemClientTests await client.CachedCallAsync("TestAPI", "postData"); - var (maxRetries, _, _, _) = ReadBufferedRetrySettings(connStr); + var (maxRetries, _, _, _) = ReadBufferedRetrySettings(localDb.Db); // Must be the bounded S&F default, never 0 — a stored 0 would mean retry-forever. Assert.Equal(99, maxRetries); Assert.NotEqual(0, maxRetries); @@ -651,11 +678,8 @@ public class ExternalSystemClientTests var httpClient = new HttpClient(new HangingHttpMessageHandler(TimeSpan.FromMinutes(10))); _httpClientFactory.CreateClient(Arg.Any()).Returns(httpClient); - var dbName = $"EsgCancel_{Guid.NewGuid():N}"; - var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared"; - using var keepAlive = new SqliteConnection(connStr); - keepAlive.Open(); - var storage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + var localDb = CreateLocalDb("EsgCancel"); + var storage = new StoreAndForwardStorage(localDb.Db, NullLogger.Instance); await storage.InitializeAsync(); var sfOptions = new StoreAndForwardOptions { @@ -1060,11 +1084,8 @@ public class ExternalSystemClientTests .Returns(_ => new HttpClient(new MockHttpMessageHandler(HttpStatusCode.OK, hugeBody))); // A real S&F service so we can assert the oversized response is NOT buffered. - var dbName = $"EsgOversize_{Guid.NewGuid():N}"; - var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared"; - using var keepAlive = new SqliteConnection(connStr); - keepAlive.Open(); - var storage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + var localDb = CreateLocalDb("EsgOversize"); + var storage = new StoreAndForwardStorage(localDb.Db, NullLogger.Instance); await storage.InitializeAsync(); var sfOptions = new StoreAndForwardOptions { diff --git a/tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests.csproj b/tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests.csproj index d2b57969..abb801e5 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests.csproj +++ b/tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests.csproj @@ -24,6 +24,7 @@ - + + diff --git a/tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests/HealthReportSenderTests.cs b/tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests/HealthReportSenderTests.cs index 5ea6a969..55d8fa57 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests/HealthReportSenderTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests/HealthReportSenderTests.cs @@ -1,10 +1,10 @@ -using Microsoft.Data.Sqlite; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Logging.Abstractions; using Microsoft.Extensions.Options; using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health; using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; using ZB.MOM.WW.ScadaBridge.StoreAndForward; +using ZB.MOM.WW.ScadaBridge.TestSupport; namespace ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests; @@ -191,50 +191,55 @@ public class HealthReportSenderTests [Fact] public async Task ReportsIncludeStoreAndForwardBufferDepthsFromStorage() { - var dbName = $"HealthSfDepth_{Guid.NewGuid():N}"; - var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared"; - // Keep one connection alive so the in-memory DB persists for the test. - using var keepAlive = new SqliteConnection(connStr); - keepAlive.Open(); - - var storage = new StoreAndForwardStorage(connStr, NullLogger.Instance); - await storage.InitializeAsync(); - - // Two pending ExternalSystem messages and one pending Notification message. - await storage.EnqueueAsync(MakePendingMessage("m1", StoreAndForwardCategory.ExternalSystem)); - await storage.EnqueueAsync(MakePendingMessage("m2", StoreAndForwardCategory.ExternalSystem)); - await storage.EnqueueAsync(MakePendingMessage("m3", StoreAndForwardCategory.Notification)); - - var transport = new FakeTransport(); - var collector = new SiteHealthCollector(); - collector.SetActiveNode(true); - var options = Options.Create(new HealthMonitoringOptions - { - ReportInterval = TimeSpan.FromMilliseconds(50) - }); - - var sender = new HealthReportSender( - collector, - transport, - options, - NullLogger.Instance, - new FakeSiteIdentityProvider(), - sfStorage: storage); - - using var cts = new CancellationTokenSource(TimeSpan.FromMilliseconds(300)); + // LocalDb has no in-memory mode, so the store runs over a real temp file. + var localDb = TestLocalDb.CreateTemp("HealthSfDepth"); try { - await sender.StartAsync(cts.Token); - await Task.Delay(250, CancellationToken.None); - await sender.StopAsync(CancellationToken.None); - } - catch (OperationCanceledException) { } + var storage = new StoreAndForwardStorage(localDb.Db, NullLogger.Instance); + await storage.InitializeAsync(); - Assert.True(transport.SentReports.Count >= 1); - var depths = transport.SentReports[^1].StoreAndForwardBufferDepths; - Assert.Equal(2, depths[nameof(StoreAndForwardCategory.ExternalSystem)]); - Assert.Equal(1, depths[nameof(StoreAndForwardCategory.Notification)]); - Assert.False(depths.ContainsKey(nameof(StoreAndForwardCategory.CachedDbWrite))); + // Two pending ExternalSystem messages and one pending Notification message. + await storage.EnqueueAsync(MakePendingMessage("m1", StoreAndForwardCategory.ExternalSystem)); + await storage.EnqueueAsync(MakePendingMessage("m2", StoreAndForwardCategory.ExternalSystem)); + await storage.EnqueueAsync(MakePendingMessage("m3", StoreAndForwardCategory.Notification)); + + var transport = new FakeTransport(); + var collector = new SiteHealthCollector(); + collector.SetActiveNode(true); + var options = Options.Create(new HealthMonitoringOptions + { + ReportInterval = TimeSpan.FromMilliseconds(50) + }); + + var sender = new HealthReportSender( + collector, + transport, + options, + NullLogger.Instance, + new FakeSiteIdentityProvider(), + sfStorage: storage); + + using var cts = new CancellationTokenSource(TimeSpan.FromMilliseconds(300)); + try + { + await sender.StartAsync(cts.Token); + await Task.Delay(250, CancellationToken.None); + await sender.StopAsync(CancellationToken.None); + } + catch (OperationCanceledException) { } + + Assert.True(transport.SentReports.Count >= 1); + var depths = transport.SentReports[^1].StoreAndForwardBufferDepths; + Assert.Equal(2, depths[nameof(StoreAndForwardCategory.ExternalSystem)]); + Assert.Equal(1, depths[nameof(StoreAndForwardCategory.Notification)]); + Assert.False(depths.ContainsKey(nameof(StoreAndForwardCategory.CachedDbWrite))); + } + finally + { + // The master connection anchors the WAL — dispose before deleting. + localDb.Dispose(); + TestLocalDb.DeleteFiles(localDb.Path); + } } private static StoreAndForwardMessage MakePendingMessage(string id, StoreAndForwardCategory category) => diff --git a/tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests.csproj b/tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests.csproj index 2f7c9b4b..1cda3a29 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests.csproj +++ b/tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests.csproj @@ -28,6 +28,7 @@ - + + diff --git a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/ActorPathTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/ActorPathTests.cs index 5b0b7bcd..dbd8de93 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/ActorPathTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/ActorPathTests.cs @@ -163,10 +163,12 @@ public class SiteActorPathTests : IAsyncLifetime private IHost? _host; private ActorSystem? _actorSystem; private string _tempDbPath = null!; + private string _tempLocalDbPath = null!; public async Task InitializeAsync() { _tempDbPath = Path.Combine(Path.GetTempPath(), $"scadabridge_actor_test_{Guid.NewGuid()}.db"); + _tempLocalDbPath = Path.Combine(Path.GetTempPath(), $"scadabridge_actor_localdb_{Guid.NewGuid()}.db"); var builder = Microsoft.Extensions.Hosting.Host.CreateDefaultBuilder(); builder.ConfigureAppConfiguration(config => @@ -183,6 +185,10 @@ public class SiteActorPathTests : IAsyncLifetime ["ScadaBridge:Cluster:SeedNodes:1"] = "akka.tcp://scadabridge@localhost:25520", ["ScadaBridge:Cluster:MinNrOfMembers"] = "1", ["ScadaBridge:Database:SiteDbPath"] = _tempDbPath, + // Required: LocalDbOptions.Path is validated with ValidateOnStart, and this + // fixture actually starts the host — a site node with no configured + // consolidated-database path fails fast rather than booting half-working. + ["LocalDb:Path"] = _tempLocalDbPath, // Configure a dummy central contact point to trigger ClusterClient creation ["ScadaBridge:Communication:CentralContactPoints:0"] = "akka.tcp://scadabridge@localhost:25510", }); @@ -207,6 +213,7 @@ public class SiteActorPathTests : IAsyncLifetime _host.Dispose(); } try { File.Delete(_tempDbPath); } catch { /* best effort */ } + try { File.Delete(_tempLocalDbPath); } catch { /* best effort */ } } [Fact] diff --git a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/AkkaHostedServiceAuditWiringTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/AkkaHostedServiceAuditWiringTests.cs index d31f27a8..31e21242 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/AkkaHostedServiceAuditWiringTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/AkkaHostedServiceAuditWiringTests.cs @@ -290,10 +290,12 @@ public class SiteAuditWiringTests : IDisposable { private readonly WebApplication _host; private readonly string _tempDbPath; + private readonly string _tempLocalDbPath; public SiteAuditWiringTests() { _tempDbPath = Path.Combine(Path.GetTempPath(), $"scadabridge_audit_wiring_{Guid.NewGuid()}.db"); + _tempLocalDbPath = Path.Combine(Path.GetTempPath(), $"scadabridge_audit_wiring_localdb_{Guid.NewGuid()}.db"); var builder = WebApplication.CreateBuilder(); builder.Configuration.Sources.Clear(); @@ -312,6 +314,11 @@ public class SiteAuditWiringTests : IDisposable // resolved; point it at an in-memory connection so the test doesn't // pollute the working directory. ["AuditLog:SiteWriter:DatabasePath"] = ":memory:", + // The cached-call telemetry forwarder pulls in IOperationTrackingStore, + // which now resolves ILocalDb — so the consolidated site database must be + // configured or the whole graph fails to resolve. There is no in-memory + // mode: LocalDbOptions.Path is a filesystem path. + ["LocalDb:Path"] = _tempLocalDbPath, }); builder.Services.AddGrpc(); @@ -326,6 +333,7 @@ public class SiteAuditWiringTests : IDisposable { (_host as IDisposable)?.Dispose(); try { File.Delete(_tempDbPath); } catch { /* best effort */ } + try { File.Delete(_tempLocalDbPath); } catch { /* best effort */ } } [Fact] diff --git a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CompositionRootTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CompositionRootTests.cs index 5d89eb9d..59d4bf31 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CompositionRootTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CompositionRootTests.cs @@ -383,11 +383,13 @@ public class SiteCompositionRootTests : IDisposable private readonly WebApplication _host; private readonly string _tempDbPath; private readonly string _tempTrackingDbPath; + private readonly string _tempLocalDbPath; public SiteCompositionRootTests() { _tempDbPath = Path.Combine(Path.GetTempPath(), $"scadabridge_test_{Guid.NewGuid()}.db"); _tempTrackingDbPath = Path.Combine(Path.GetTempPath(), $"scadabridge_track_{Guid.NewGuid()}.db"); + _tempLocalDbPath = Path.Combine(Path.GetTempPath(), $"scadabridge_localdb_{Guid.NewGuid()}.db"); var builder = WebApplication.CreateBuilder(); builder.Configuration.Sources.Clear(); @@ -400,10 +402,13 @@ public class SiteCompositionRootTests : IDisposable ["ScadaBridge:Node:RemotingPort"] = "0", ["ScadaBridge:Node:GrpcPort"] = "0", ["ScadaBridge:Database:SiteDbPath"] = _tempDbPath, - // Point the site-local operation-tracking SQLite store (the ctor opens/creates - // the file eagerly) at a temp path so resolving IOperationTrackingStore in the - // composition-root test does not litter site-tracking.db in the test cwd. + // Retained for the site config DB; it no longer points the tracking store + // anywhere — that store now opens the consolidated database below. ["ScadaBridge:OperationTracking:ConnectionString"] = $"Data Source={_tempTrackingDbPath}", + // The consolidated site database. IOperationTrackingStore resolves ILocalDb + // and creates the file eagerly in its ctor, so this must be a temp path or + // the composition-root test litters the working directory. + ["LocalDb:Path"] = _tempLocalDbPath, // ClusterOptions requires at least one seed node (ClusterOptionsValidator). ["ScadaBridge:Cluster:SeedNodes:0"] = "akka.tcp://scadabridge@localhost:2551", ["ScadaBridge:Cluster:SeedNodes:1"] = "akka.tcp://scadabridge@localhost:2552", @@ -427,6 +432,7 @@ public class SiteCompositionRootTests : IDisposable (_host as IDisposable)?.Dispose(); try { File.Delete(_tempDbPath); } catch { /* best effort */ } try { File.Delete(_tempTrackingDbPath); } catch { /* best effort */ } + try { File.Delete(_tempLocalDbPath); } catch { /* best effort */ } } [Fact] @@ -465,7 +471,6 @@ public class SiteCompositionRootTests : IDisposable new object[] { typeof(IDataConnectionFactory) }, new object[] { typeof(StoreAndForwardStorage) }, new object[] { typeof(StoreAndForwardService) }, - new object[] { typeof(ReplicationService) }, new object[] { typeof(ISiteEventLogger) }, new object[] { typeof(IEventLogQueryService) }, new object[] { typeof(ISiteIdentityProvider) }, diff --git a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/LocalDbSyncAuthInterceptorTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/LocalDbSyncAuthInterceptorTests.cs new file mode 100644 index 00000000..bf0b3cac --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/LocalDbSyncAuthInterceptorTests.cs @@ -0,0 +1,171 @@ +using Grpc.Core; +using Microsoft.Extensions.Logging.Abstractions; +using Microsoft.Extensions.Options; +using ZB.MOM.WW.LocalDb.Replication; +using ZB.MOM.WW.ScadaBridge.Host; + +namespace ZB.MOM.WW.ScadaBridge.Host.Tests; + +/// +/// LocalDb Phase 1 (Task 8) — the passive sync endpoint's inbound gate. +/// +/// +/// The replication library's LocalDbSyncService verifies nothing; inbound auth is +/// explicitly the host's job. Without this interceptor, anything able to reach a site +/// node's gRPC port could stream arbitrary rows straight into the consolidated site +/// database — including into OperationTracking, which central reconciles from. +/// +public class LocalDbSyncAuthInterceptorTests +{ + private const string SyncMethod = "/localdb_sync.v1.LocalDbSync/Sync"; + private const string SiteStreamMethod = "/scadabridge.SiteStream/Connect"; + + private static LocalDbSyncAuthInterceptor CreateInterceptor(string? apiKey) + => new( + Options.Create(new ReplicationOptions { ApiKey = apiKey }), + NullLogger.Instance); + + private static ServerCallContext CreateContext(string method, string? authorizationHeader) + { + var headers = new Metadata(); + if (authorizationHeader is not null) + headers.Add("authorization", authorizationHeader); + + return new FakeServerCallContext(method, headers); + } + + /// + /// Minimal carrying just a method name and request + /// headers — the only two things the interceptor reads. + /// + /// + /// Hand-rolled rather than using Grpc.Core.Testing.TestServerCallContext: that + /// type ships in the retired native Grpc.Core package and does not exist on the + /// grpc-dotnet stack this solution runs on. Pulling in the dead package to get one + /// test helper would be a worse trade than these few overrides. + /// + private sealed class FakeServerCallContext(string method, Metadata requestHeaders) + : ServerCallContext + { + protected override string MethodCore => method; + protected override string HostCore => "localhost"; + protected override string PeerCore => "ipv4:127.0.0.1:12345"; + protected override DateTime DeadlineCore => DateTime.UtcNow.AddMinutes(1); + protected override Metadata RequestHeadersCore => requestHeaders; + protected override CancellationToken CancellationTokenCore => CancellationToken.None; + protected override Metadata ResponseTrailersCore { get; } = []; + protected override Status StatusCore { get; set; } + protected override WriteOptions? WriteOptionsCore { get; set; } + protected override AuthContext AuthContextCore { get; } = + new(null, new Dictionary>()); + + protected override ContextPropagationToken CreatePropagationTokenCore( + ContextPropagationOptions? options) + => throw new NotSupportedException(); + + protected override Task WriteResponseHeadersAsyncCore(Metadata responseHeaders) + => Task.CompletedTask; + } + + /// Invokes the interceptor's unary path with a trivial continuation. + private static Task Invoke( + LocalDbSyncAuthInterceptor interceptor, ServerCallContext context) + => interceptor.UnaryServerHandler( + "request", context, (_, _) => Task.FromResult("ok")); + + [Fact] + public async Task NonSyncMethod_PassesThrough_EvenWithNoKeyConfigured() + { + // The interceptor is registered on the shared site AddGrpc pipeline, so it sees + // every call. It must be scoped strictly to the sync service — gating SiteStream + // would break site↔central communication outright. + var interceptor = CreateInterceptor(apiKey: null); + var context = CreateContext(SiteStreamMethod, authorizationHeader: null); + + Assert.Equal("ok", await Invoke(interceptor, context)); + } + + [Fact] + public async Task SyncMethod_WithNoKeyConfigured_IsDenied_EvenWithABearerToken() + { + // Fail-closed. "No key configured" is the DEFAULT every site node ships with, so + // treating it as "no auth required" would silently expose the endpoint on exactly + // the configuration that is most common. Presenting a token must not help. + var interceptor = CreateInterceptor(apiKey: null); + var context = CreateContext(SyncMethod, "Bearer anything-at-all"); + + var ex = await Assert.ThrowsAsync(() => Invoke(interceptor, context)); + Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode); + } + + [Fact] + public async Task SyncMethod_WithNoBearerToken_IsDenied() + { + var interceptor = CreateInterceptor("the-shared-key"); + var context = CreateContext(SyncMethod, authorizationHeader: null); + + var ex = await Assert.ThrowsAsync(() => Invoke(interceptor, context)); + Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode); + } + + [Fact] + public async Task SyncMethod_WithWrongBearerToken_IsDenied() + { + var interceptor = CreateInterceptor("the-shared-key"); + var context = CreateContext(SyncMethod, "Bearer the-wrong-key"); + + var ex = await Assert.ThrowsAsync(() => Invoke(interceptor, context)); + Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode); + } + + [Fact] + public async Task SyncMethod_WithCorrectBearerToken_PassesThrough() + { + var interceptor = CreateInterceptor("the-shared-key"); + var context = CreateContext(SyncMethod, "Bearer the-shared-key"); + + Assert.Equal("ok", await Invoke(interceptor, context)); + } + + [Fact] + public async Task SyncMethod_WithCorrectKey_ButNoBearerScheme_IsDenied() + { + // A raw key with no "Bearer " prefix is not what SyncBackgroundService sends, and + // accepting it would widen the accepted credential shape for no reason. + var interceptor = CreateInterceptor("the-shared-key"); + var context = CreateContext(SyncMethod, "the-shared-key"); + + var ex = await Assert.ThrowsAsync(() => Invoke(interceptor, context)); + Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode); + } + + [Fact] + public async Task SyncMethod_TokenComparison_IsNotAPrefixMatch() + { + // A prefix/StartsWith comparison would accept a truncated key and make the secret + // recoverable one character at a time. + var interceptor = CreateInterceptor("the-shared-key"); + var context = CreateContext(SyncMethod, "Bearer the-shared-ke"); + + var ex = await Assert.ThrowsAsync(() => Invoke(interceptor, context)); + Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode); + } + + [Fact] + public async Task DuplexStreaming_IsGated_BecauseThatIsHowSyncActuallyRuns() + { + // The sync RPC is a bidirectional stream. Gating only the unary path would leave + // the real endpoint wide open while every unary test still passed. + var interceptor = CreateInterceptor("the-shared-key"); + var context = CreateContext(SyncMethod, "Bearer the-wrong-key"); + + var ex = await Assert.ThrowsAsync(() => + interceptor.DuplexStreamingServerHandler( + requestStream: null!, + responseStream: null!, + context, + (_, _, _) => Task.CompletedTask)); + + Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode); + } +} diff --git a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbDirectoryTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbDirectoryTests.cs new file mode 100644 index 00000000..78d6f1e6 --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbDirectoryTests.cs @@ -0,0 +1,72 @@ +using Microsoft.Extensions.Configuration; +using Microsoft.Extensions.DependencyInjection; +using ZB.MOM.WW.LocalDb; + +namespace ZB.MOM.WW.ScadaBridge.Host.Tests; + +/// +/// Pins that a site node creates the directory holding LocalDb:Path before the +/// database is opened. +/// +/// +/// +/// The guarantee has now had three owners: StoreAndForwardStorage created the +/// directory for its own SQLite file, LocalDb Phase 2 folded that file into the +/// consolidated database and the Host took it over as a SiteLocalDbDirectory shim, +/// and LocalDb 0.1.1 took it into the library itself. This test survived all three moves +/// unchanged in intent, because what a site node needs is the OUTCOME — resolving +/// ILocalDb from the site registration must not fail on a fresh machine — not any +/// particular implementer of it. It is deliberately written against the registration path +/// rather than the mechanism, so a future move costs nothing here. +/// +/// +/// This is not hypothetical: the default site configuration points at the relative +/// path ./data/site-localdb.db, so any site node whose working directory has no +/// data/ subdirectory hits it. The docker rig escapes only because its volume mount +/// creates /app/data — which is exactly the kind of coincidence that hides a defect +/// until a bare-metal or fresh deployment. +/// +/// +public class SiteLocalDbDirectoryTests +{ + [Fact] + public void SiteRegistration_CreatesTheLocalDbDirectory_WhenItDoesNotExist() + { + var root = Path.Combine(Path.GetTempPath(), "localdb-dir-test-" + Guid.NewGuid().ToString("N")); + var dbPath = Path.Combine(root, "nested", "site-localdb.db"); + Assert.False(Directory.Exists(root)); + + try + { + var config = new ConfigurationBuilder() + .AddInMemoryCollection(new Dictionary + { + ["LocalDb:Path"] = dbPath, + }) + .Build(); + + // The registration path under test. Resolving ILocalDb is what actually opens + // the file, so this fails with SQLite Error 14 if the directory step regresses + // — wherever it lives. + var services = new ServiceCollection(); + services.AddZbLocalDb(config); + + using var provider = services.BuildServiceProvider(); + using var scope = provider.CreateScope(); + var db = scope.ServiceProvider.GetRequiredService(); + + Assert.NotNull(db); + Assert.True(Directory.Exists(Path.GetDirectoryName(dbPath)!)); + Assert.True(File.Exists(dbPath)); + } + finally + { + SqliteConnectionPoolCleanup(); + if (Directory.Exists(root)) + Directory.Delete(root, recursive: true); + } + } + + private static void SqliteConnectionPoolCleanup() => + Microsoft.Data.Sqlite.SqliteConnection.ClearAllPools(); +} diff --git a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbLegacyMigratorTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbLegacyMigratorTests.cs new file mode 100644 index 00000000..329b1011 --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbLegacyMigratorTests.cs @@ -0,0 +1,614 @@ +using Microsoft.Data.Sqlite; +using Microsoft.Extensions.Configuration; +using Microsoft.Extensions.DependencyInjection; +using ZB.MOM.WW.LocalDb; +using ZB.MOM.WW.ScadaBridge.Host; + +namespace ZB.MOM.WW.ScadaBridge.Host.Tests; + +/// +/// LocalDb Phase 1 (Task 6) — the one-time copy of the pre-Phase-1 site databases into +/// the consolidated one. +/// +/// +/// This is a data migration that runs during host startup, so its failure modes are +/// expensive: a duplicate-on-rerun bug silently doubles a site's event history, and a +/// half-committed copy leaves an operator with no clean state to recover to. Every bullet +/// of the plan's semantics gets its own test. +/// +public class SiteLocalDbLegacyMigratorTests : IDisposable +{ + private readonly string _root; + private readonly List _providers = []; + + public SiteLocalDbLegacyMigratorTests() + { + _root = Path.Combine(Path.GetTempPath(), $"localdb-migrator-{Guid.NewGuid():N}"); + Directory.CreateDirectory(_root); + } + + public void Dispose() + { + foreach (var provider in _providers) + { + try { provider.Dispose(); } catch { /* best effort */ } + } + + try { Directory.Delete(_root, recursive: true); } catch { /* best effort */ } + GC.SuppressFinalize(this); + } + + private string Path_(string name) => System.IO.Path.Combine(_root, name); + + /// A consolidated database with the tables created and registered, as the host has it. + /// + /// sf_messages is registered here even though production OnReady does not + /// register it until the Task 14 cutover. The oplog assertion below needs live capture + /// triggers to mean anything, and the property under test — that the migrator runs after + /// registration — is the same either way. The corollary is a real constraint on Task 14: + /// Migrate must stay the LAST call in OnReady, after every registration. + /// + private ILocalDb CreateConsolidated(IConfiguration config) + { + var provider = new ServiceCollection() + .AddZbLocalDb(config, db => + { + using var connection = db.CreateConnection(); + ZB.MOM.WW.ScadaBridge.SiteRuntime.Tracking.OperationTrackingSchema.Apply(connection); + ZB.MOM.WW.ScadaBridge.SiteEventLogging.SiteEventLogSchema.Apply(connection); + ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardSchema.Apply(connection); + ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence.SiteStorageSchema.Apply(connection); + db.RegisterReplicated("OperationTracking"); + db.RegisterReplicated("site_events"); + db.RegisterReplicated("sf_messages"); + foreach (var table in SiteLocalDbLegacyMigrator.SiteStorageTables) + db.RegisterReplicated(table.Table); + }) + .BuildServiceProvider(); + _providers.Add(provider); + + return provider.GetRequiredService(); + } + + private IConfiguration Config( + string? trackingPath = null, + string? eventsPath = null, + string nodeName = "node-a", + string? storeAndForwardPath = null, + string? siteDbPath = null) + { + var values = new Dictionary + { + ["LocalDb:Path"] = Path_("consolidated.db"), + ["ScadaBridge:Node:NodeName"] = nodeName, + + // Always pinned inside the test's own directory, even when a test does not care + // about it. Left unset, the resolver falls back to the CWD-relative code default + // "./data/store-and-forward.db" — and the test run's CWD is the test binary's + // output directory, so an unlucky run could migrate (and RENAME) a real file. + ["ScadaBridge:StoreAndForward:SqliteDbPath"] = + storeAndForwardPath ?? Path_("absent-store-and-forward.db"), + + // Same reasoning: the site-storage default is "./data/scadabridge.db". + ["ScadaBridge:Database:SiteDbPath"] = siteDbPath ?? Path_("absent-scadabridge.db"), + }; + + if (trackingPath is not null) + values["ScadaBridge:OperationTracking:ConnectionString"] = $"Data Source={trackingPath}"; + if (eventsPath is not null) + values["ScadaBridge:SiteEventLog:DatabasePath"] = eventsPath; + + return new ConfigurationBuilder().AddInMemoryCollection(values).Build(); + } + + /// Seeds a legacy store-and-forward file with the CURRENT column set. + private static void SeedLegacyStoreAndForward(string path, params string[] ids) + { + using var connection = new SqliteConnection($"Data Source={path}"); + connection.Open(); + ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardSchema.Apply(connection); + + foreach (var id in ids) + { + using var cmd = connection.CreateCommand(); + cmd.CommandText = """ + INSERT INTO sf_messages ( + id, category, target, payload_json, retry_count, max_retries, + retry_interval_ms, created_at, status, execution_id) + VALUES ($id, 0, 'ERP', '{"order":1}', 0, 50, 30000, $now, 0, 'exec-1'); + """; + cmd.Parameters.AddWithValue("$id", id); + cmd.Parameters.AddWithValue("$now", DateTime.UtcNow.ToString("o")); + cmd.ExecuteNonQuery(); + } + } + + private static void SeedLegacyTracking(string path, params string[] ids) + { + using var connection = new SqliteConnection($"Data Source={path}"); + connection.Open(); + ZB.MOM.WW.ScadaBridge.SiteRuntime.Tracking.OperationTrackingSchema.Apply(connection); + + foreach (var id in ids) + { + using var cmd = connection.CreateCommand(); + cmd.CommandText = """ + INSERT INTO OperationTracking ( + TrackedOperationId, Kind, TargetSummary, Status, RetryCount, + CreatedAtUtc, UpdatedAtUtc) + VALUES ($id, 'ApiCallCached', 'ERP.GetOrder', 'Submitted', 0, $now, $now); + """; + cmd.Parameters.AddWithValue("$id", id); + cmd.Parameters.AddWithValue("$now", DateTime.UtcNow.ToString("o")); + cmd.ExecuteNonQuery(); + } + } + + /// Seeds the LEGACY event schema: an autoincrement integer id, not a GUID. + private static void SeedLegacyEvents(string path, int count) + { + using var connection = new SqliteConnection($"Data Source={path}"); + connection.Open(); + + using (var ddl = connection.CreateCommand()) + { + ddl.CommandText = """ + CREATE TABLE IF NOT EXISTS site_events ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + timestamp TEXT NOT NULL, event_type TEXT NOT NULL, severity TEXT NOT NULL, + instance_id TEXT, source TEXT NOT NULL, message TEXT NOT NULL, details TEXT + ); + """; + ddl.ExecuteNonQuery(); + } + + for (var i = 0; i < count; i++) + { + using var cmd = connection.CreateCommand(); + cmd.CommandText = """ + INSERT INTO site_events (timestamp, event_type, severity, source, message) + VALUES ($ts, 'script', 'Info', 'src', $msg); + """; + cmd.Parameters.AddWithValue("$ts", DateTimeOffset.UtcNow.AddMinutes(-i).ToString("o")); + cmd.Parameters.AddWithValue("$msg", $"legacy message {i}"); + cmd.ExecuteNonQuery(); + } + } + + private static long CountRows(ILocalDb db, string table) + { + using var connection = db.CreateConnection(); + using var cmd = connection.CreateCommand(); + cmd.CommandText = $"SELECT COUNT(*) FROM {table}"; + return (long)cmd.ExecuteScalar()!; + } + + private static List SelectIds(ILocalDb db, string table, string idColumn) + { + using var connection = db.CreateConnection(); + using var cmd = connection.CreateCommand(); + cmd.CommandText = $"SELECT {idColumn} FROM {table} ORDER BY {idColumn}"; + using var reader = cmd.ExecuteReader(); + + var ids = new List(); + while (reader.Read()) ids.Add(reader.GetString(0)); + return ids; + } + + [Fact] + public void MissingLegacyFiles_AreANoOp() + { + // The expected case on the docker rig: neither legacy key is set anywhere, and + // the CWD-relative defaults point outside the data volume, so there is usually + // nothing there at all. That must be a clean no-op, not a startup failure. + var config = Config(Path_("absent-tracking.db"), Path_("absent-events.db")); + var db = CreateConsolidated(config); + + SiteLocalDbLegacyMigrator.Migrate(db, config); + + Assert.Equal(0, CountRows(db, "OperationTracking")); + Assert.Equal(0, CountRows(db, "site_events")); + } + + [Fact] + public void LegacyTrackingRows_AreCopiedAndTheFileIsRenamed() + { + var trackingPath = Path_("site-tracking.db"); + SeedLegacyTracking(trackingPath, "op-1", "op-2", "op-3"); + var config = Config(trackingPath, Path_("absent-events.db")); + var db = CreateConsolidated(config); + + SiteLocalDbLegacyMigrator.Migrate(db, config); + + Assert.Equal(3, CountRows(db, "OperationTracking")); + Assert.Equal(["op-1", "op-2", "op-3"], SelectIds(db, "OperationTracking", "TrackedOperationId")); + + // Renamed, not deleted — the operator can still inspect the original. + Assert.False(File.Exists(trackingPath)); + Assert.True(File.Exists(trackingPath + ".migrated")); + } + + [Fact] + public void LegacyEvents_GetDeterministicIds_NotFreshGuids() + { + // The whole reason a rerun cannot duplicate. Fresh GUIDs would make + // INSERT OR IGNORE useless and double the history on every retry. + var eventsPath = Path_("site_events.db"); + SeedLegacyEvents(eventsPath, count: 3); + var config = Config(Path_("absent-tracking.db"), eventsPath, nodeName: "node-b"); + var db = CreateConsolidated(config); + + SiteLocalDbLegacyMigrator.Migrate(db, config); + + Assert.Equal( + ["mig-node-b-1", "mig-node-b-2", "mig-node-b-3"], + SelectIds(db, "site_events", "id")); + } + + [Fact] + public void SecondBoot_IsANoOp_BecauseTheFileIsAlreadyMarkedMigrated() + { + var trackingPath = Path_("site-tracking.db"); + var eventsPath = Path_("site_events.db"); + SeedLegacyTracking(trackingPath, "op-1", "op-2"); + SeedLegacyEvents(eventsPath, count: 2); + var config = Config(trackingPath, eventsPath); + var db = CreateConsolidated(config); + + SiteLocalDbLegacyMigrator.Migrate(db, config); + SiteLocalDbLegacyMigrator.Migrate(db, config); + + Assert.Equal(2, CountRows(db, "OperationTracking")); + Assert.Equal(2, CountRows(db, "site_events")); + } + + [Fact] + public void RerunAgainstAnUnrenamedFile_DoesNotDuplicate() + { + // Simulates the crash window: the copy committed but the rename did not. On the + // next boot the migrator sees the legacy file again and re-copies it. Both tables + // must absorb that with no duplicates — tracking via its natural TEXT key, events + // via the deterministic mig- id. + var trackingPath = Path_("site-tracking.db"); + var eventsPath = Path_("site_events.db"); + SeedLegacyTracking(trackingPath, "op-1", "op-2"); + SeedLegacyEvents(eventsPath, count: 2); + var config = Config(trackingPath, eventsPath); + var db = CreateConsolidated(config); + + SiteLocalDbLegacyMigrator.Migrate(db, config); + + // Undo the rename, exactly as a crash between commit and rename would leave it. + File.Move(trackingPath + ".migrated", trackingPath); + File.Move(eventsPath + ".migrated", eventsPath); + + SiteLocalDbLegacyMigrator.Migrate(db, config); + + Assert.Equal(2, CountRows(db, "OperationTracking")); + Assert.Equal(2, CountRows(db, "site_events")); + } + + [Fact] + public void MigratedRows_EnterTheOplog_SoTheyActuallyReplicate() + { + // The reason the migrator runs AFTER RegisterReplicated. Capture is trigger-based: + // migrate before registration and the rows are invisible to the peer forever, with + // no error anywhere. Asserting on the oplog is the only way to catch that ordering + // being reversed — every other test here would still pass. + var trackingPath = Path_("site-tracking.db"); + SeedLegacyTracking(trackingPath, "op-1", "op-2"); + var config = Config(trackingPath, Path_("absent-events.db")); + var db = CreateConsolidated(config); + + SiteLocalDbLegacyMigrator.Migrate(db, config); + + using var connection = db.CreateConnection(); + using var cmd = connection.CreateCommand(); + cmd.CommandText = + "SELECT COUNT(*) FROM __localdb_oplog WHERE table_name = 'OperationTracking'"; + Assert.Equal(2L, (long)cmd.ExecuteScalar()!); + } + + [Fact] + public void SfMessages_AreCopiedFromTheLegacyFile() + { + // Phase 2 (Task 8). Unlike the two Phase 1 files, the store-and-forward default path + // is inside the data volume, so on a real deployment this actually moves data: + // undelivered messages that a lost migration would silently discard. + var sfPath = Path_("store-and-forward.db"); + SeedLegacyStoreAndForward(sfPath, "msg-1", "msg-2", "msg-3"); + var config = Config(storeAndForwardPath: sfPath); + var db = CreateConsolidated(config); + + SiteLocalDbLegacyMigrator.Migrate(db, config); + + Assert.Equal(3, CountRows(db, "sf_messages")); + Assert.Equal(["msg-1", "msg-2", "msg-3"], SelectIds(db, "sf_messages", "id")); + + // Renamed, not deleted — same contract as the Phase 1 files. + Assert.False(File.Exists(sfPath)); + Assert.True(File.Exists(sfPath + ".migrated")); + } + + [Fact] + public void SfMessages_Migration_IsIdempotent_WhenRerunAfterACrashBeforeRename() + { + // The crash window: copy committed, rename did not. sf_messages.id is a natural + // TEXT key, so INSERT OR IGNORE absorbs the re-copy with no id synthesis needed. + var sfPath = Path_("store-and-forward.db"); + SeedLegacyStoreAndForward(sfPath, "msg-1", "msg-2"); + var config = Config(storeAndForwardPath: sfPath); + var db = CreateConsolidated(config); + + SiteLocalDbLegacyMigrator.Migrate(db, config); + File.Move(sfPath + ".migrated", sfPath); + SiteLocalDbLegacyMigrator.Migrate(db, config); + + Assert.Equal(2, CountRows(db, "sf_messages")); + } + + [Fact] + public void MigratedSfMessages_EnterTheOplog_SoTheyActuallyReplicate() + { + // Same ordering trap as the Phase 1 tables: migrate before RegisterReplicated and + // the rows never enter the oplog, never reach the peer, and nothing errors. Only an + // assertion on __localdb_oplog catches the ordering being reversed. + var sfPath = Path_("store-and-forward.db"); + SeedLegacyStoreAndForward(sfPath, "msg-1", "msg-2"); + var config = Config(storeAndForwardPath: sfPath); + var db = CreateConsolidated(config); + + SiteLocalDbLegacyMigrator.Migrate(db, config); + + using var connection = db.CreateConnection(); + using var cmd = connection.CreateCommand(); + cmd.CommandText = "SELECT COUNT(*) FROM __localdb_oplog WHERE table_name = 'sf_messages'"; + Assert.Equal(2L, (long)cmd.ExecuteScalar()!); + } + + [Fact] + public void SfMessages_FromAnOlderBuild_MigrateDespiteMissingColumns() + { + // A legacy file written before execution_id / parent_execution_id / last_attempt_at_ms + // existed. Naming a missing column in the SELECT throws "no such column", and the + // reader treats that as an unrecognised shape — which would silently discard every + // buffered message. The copy intersects the column sets instead. + var sfPath = Path_("store-and-forward.db"); + using (var legacy = new SqliteConnection($"Data Source={sfPath}")) + { + legacy.Open(); + using var ddl = legacy.CreateCommand(); + ddl.CommandText = """ + CREATE TABLE sf_messages ( + id TEXT PRIMARY KEY, category INTEGER NOT NULL, target TEXT NOT NULL, + payload_json TEXT NOT NULL, retry_count INTEGER NOT NULL DEFAULT 0, + max_retries INTEGER NOT NULL DEFAULT 50, + retry_interval_ms INTEGER NOT NULL DEFAULT 30000, + created_at TEXT NOT NULL, last_attempt_at TEXT, + status INTEGER NOT NULL DEFAULT 0, last_error TEXT, origin_instance TEXT + ); + INSERT INTO sf_messages (id, category, target, payload_json, created_at) + VALUES ('old-1', 0, 'ERP', '{}', '2026-01-01T00:00:00Z'); + """; + ddl.ExecuteNonQuery(); + } + + var config = Config(storeAndForwardPath: sfPath); + var db = CreateConsolidated(config); + + SiteLocalDbLegacyMigrator.Migrate(db, config); + + Assert.Equal(1, CountRows(db, "sf_messages")); + Assert.Equal(["old-1"], SelectIds(db, "sf_messages", "id")); + } + + /// Seeds a legacy scadabridge.db with the CURRENT config schema and one row per table. + private static void SeedLegacySiteStorage(string path) + { + using var connection = new SqliteConnection($"Data Source={path}"); + connection.Open(); + ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence.SiteStorageSchema.Apply(connection); + + using var cmd = connection.CreateCommand(); + cmd.CommandText = """ + INSERT INTO deployed_configurations + (instance_unique_name, config_json, deployment_id, revision_hash, is_enabled, deployed_at) + VALUES ('inst-1', '{"a":1}', 'dep-1', 'hash-1', 1, '2026-01-01T00:00:00Z'); + INSERT INTO static_attribute_overrides + (instance_unique_name, attribute_name, override_value, updated_at) + VALUES ('inst-1', 'Setpoint', '42', '2026-01-01T00:00:00Z'); + INSERT INTO shared_scripts (name, code, updated_at) + VALUES ('helper', 'return 1;', '2026-01-01T00:00:00Z'); + INSERT INTO external_systems (name, endpoint_url, auth_type, updated_at, timeout_seconds) + VALUES ('ERP', 'http://erp:5200', 'None', '2026-01-01T00:00:00Z', 30); + INSERT INTO database_connections (name, connection_string, updated_at) + VALUES ('BT', 'Server=sql;Database=BT', '2026-01-01T00:00:00Z'); + INSERT INTO data_connection_definitions (name, protocol, updated_at) + VALUES ('opc-1', 'OpcUa', '2026-01-01T00:00:00Z'); + INSERT INTO native_alarm_state + (instance_unique_name, source_canonical_name, source_reference, condition_json, last_transition_at) + VALUES ('inst-1', 'Area.Line', 'ref-1', '{"active":true}', '2026-01-01T00:00:00Z'); + + -- The two that must NOT be migrated. A pre-2026-07-10 file really can hold these. + INSERT INTO notification_lists (name, recipient_emails, updated_at) + VALUES ('ops', 'ops@example.com', '2026-01-01T00:00:00Z'); + INSERT INTO smtp_configurations + (name, server, port, auth_mode, from_address, username, password, updated_at) + VALUES ('smtp', 'smtp.example.com', 587, 'Basic', 'a@b.c', 'user', + 'PLAINTEXT-SECRET', '2026-01-01T00:00:00Z'); + """; + cmd.ExecuteNonQuery(); + } + + [Fact] + public void SiteConfigTables_AreCopiedFromTheLegacyFile() + { + // Phase 2 (Task 9). All seven migrated tables live in one legacy file and are copied + // in a single transaction: a partial config migration would leave a site node running + // against half its old configuration, which is worse than failing startup. + var sitePath = Path_("scadabridge.db"); + SeedLegacySiteStorage(sitePath); + var config = Config(siteDbPath: sitePath); + var db = CreateConsolidated(config); + + SiteLocalDbLegacyMigrator.Migrate(db, config); + + foreach (var table in SiteLocalDbLegacyMigrator.SiteStorageTables) + Assert.Equal(1, CountRows(db, table.Table)); + + Assert.False(File.Exists(sitePath)); + Assert.True(File.Exists(sitePath + ".migrated")); + } + + [Fact] + public void Migration_DoesNotCopyNotificationOrSmtpRows_EvenWhenTheLegacyFileHasThem() + { + // smtp_configurations.password is PLAINTEXT. Both tables are purged on every deploy + // and permanently empty by design since the site write paths were removed + // (2026-07-10), but a pre-fix legacy file can still hold rows. The cutover also + // declines to REGISTER these two for replication, for the same reason — so migrating + // them would leave plaintext SMTP passwords in the consolidated database, one future + // RegisterReplicated away from being shipped to a peer, for config nothing reads. + var sitePath = Path_("scadabridge.db"); + SeedLegacySiteStorage(sitePath); + var config = Config(siteDbPath: sitePath); + var db = CreateConsolidated(config); + + SiteLocalDbLegacyMigrator.Migrate(db, config); + + Assert.Equal(0, CountRows(db, "notification_lists")); + Assert.Equal(0, CountRows(db, "smtp_configurations")); + + // Belt and braces: the secret must not be anywhere in the consolidated file, + // including the oplog, whose row_json is a json_object copy of every captured row. + using var connection = db.CreateConnection(); + using var cmd = connection.CreateCommand(); + cmd.CommandText = "SELECT COUNT(*) FROM __localdb_oplog WHERE row_json LIKE '%PLAINTEXT-SECRET%'"; + Assert.Equal(0L, (long)cmd.ExecuteScalar()!); + } + + [Fact] + public void MigratedConfigRows_EnterTheOplog_SoTheyActuallyReplicate() + { + var sitePath = Path_("scadabridge.db"); + SeedLegacySiteStorage(sitePath); + var config = Config(siteDbPath: sitePath); + var db = CreateConsolidated(config); + + SiteLocalDbLegacyMigrator.Migrate(db, config); + + using var connection = db.CreateConnection(); + using var cmd = connection.CreateCommand(); + cmd.CommandText = + "SELECT COUNT(*) FROM __localdb_oplog WHERE table_name = 'deployed_configurations'"; + Assert.Equal(1L, (long)cmd.ExecuteScalar()!); + } + + [Fact] + public void SiteConfigMigration_IsIdempotent_WhenRerunAfterACrashBeforeRename() + { + var sitePath = Path_("scadabridge.db"); + SeedLegacySiteStorage(sitePath); + var config = Config(siteDbPath: sitePath); + var db = CreateConsolidated(config); + + SiteLocalDbLegacyMigrator.Migrate(db, config); + File.Move(sitePath + ".migrated", sitePath); + SiteLocalDbLegacyMigrator.Migrate(db, config); + + foreach (var table in SiteLocalDbLegacyMigrator.SiteStorageTables) + Assert.Equal(1, CountRows(db, table.Table)); + } + + [Fact] + public void MigratorColumnLists_MatchTheLiveSchema() + { + // A column named here but absent from the schema fails at runtime on a real + // deployment and NOWHERE else — the intersection logic hides a typo (the column is + // simply dropped from the copy) and every other test still passes. A column in the + // schema but missing here is worse: that data is silently left behind. + // + // So this asserts set equality against the schema the host actually applies. + using var connection = new SqliteConnection($"Data Source={Path_("schema-probe.db")}"); + connection.Open(); + ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence.SiteStorageSchema.Apply(connection); + + foreach (var table in SiteLocalDbLegacyMigrator.SiteStorageTables) + { + using var probe = connection.CreateCommand(); + probe.CommandText = $"SELECT name FROM pragma_table_info('{table.Table}')"; + using var reader = probe.ExecuteReader(); + + var actual = new List(); + while (reader.Read()) actual.Add(reader.GetString(0)); + + Assert.Equal( + actual.OrderBy(c => c, StringComparer.Ordinal).ToArray(), + table.Columns.OrderBy(c => c, StringComparer.Ordinal).ToArray()); + } + } + + [Fact] + public void LegacyFileWithoutTheExpectedTable_IsTreatedAsEmpty_NotAFailure() + { + // A file predating the table (or an unrecognised shape) must not fail host boot. + var trackingPath = Path_("site-tracking.db"); + using (var connection = new SqliteConnection($"Data Source={trackingPath}")) + { + connection.Open(); + using var ddl = connection.CreateCommand(); + ddl.CommandText = "CREATE TABLE something_else (x INTEGER);"; + ddl.ExecuteNonQuery(); + } + + var config = Config(trackingPath, Path_("absent-events.db")); + var db = CreateConsolidated(config); + + SiteLocalDbLegacyMigrator.Migrate(db, config); + + Assert.Equal(0, CountRows(db, "OperationTracking")); + Assert.True(File.Exists(trackingPath + ".migrated")); + } + + [Fact] + public void InMemoryLegacyConnectionStrings_AreSkipped() + { + // Test/dev configurations point the legacy stores at in-memory databases. There is + // nothing durable to migrate, and Path.GetFullPath on ":memory:" would be nonsense. + var config = new ConfigurationBuilder().AddInMemoryCollection(new Dictionary + { + ["LocalDb:Path"] = Path_("consolidated.db"), + ["ScadaBridge:Node:NodeName"] = "node-a", + ["ScadaBridge:OperationTracking:ConnectionString"] = + "Data Source=file:legacy?mode=memory&cache=shared", + ["ScadaBridge:SiteEventLog:DatabasePath"] = ":memory:", + }).Build(); + + Assert.Equal(string.Empty, SiteLocalDbLegacyMigrator.ResolveTrackingPath(config)); + Assert.Equal(string.Empty, SiteLocalDbLegacyMigrator.ResolveEventLogPath(config)); + + var db = CreateConsolidated(config); + SiteLocalDbLegacyMigrator.Migrate(db, config); + + Assert.Equal(0, CountRows(db, "site_events")); + } + + [Fact] + public void UnsetLegacyKeys_FallBackToTheCwdRelativeCodeDefaults() + { + // Neither key is set in any docker appsettings, so the OLD code used these + // CWD-relative defaults. Resolving them anywhere else (e.g. against the data + // volume) would look for the legacy data in a directory it was never written to. + var config = new ConfigurationBuilder().AddInMemoryCollection(new Dictionary + { + ["LocalDb:Path"] = Path_("consolidated.db"), + }).Build(); + + Assert.Equal( + System.IO.Path.GetFullPath("site-tracking.db"), + SiteLocalDbLegacyMigrator.ResolveTrackingPath(config)); + Assert.Equal( + System.IO.Path.GetFullPath("site_events.db"), + SiteLocalDbLegacyMigrator.ResolveEventLogPath(config)); + } +} diff --git a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbWiringTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbWiringTests.cs new file mode 100644 index 00000000..4340a148 --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbWiringTests.cs @@ -0,0 +1,327 @@ +using Microsoft.AspNetCore.Builder; +using Microsoft.Extensions.Configuration; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Logging.Abstractions; +using Microsoft.Extensions.Options; +using ZB.MOM.WW.ScadaBridge.HealthMonitoring; +using ZB.MOM.WW.Telemetry; +using ZB.MOM.WW.LocalDb; +using ZB.MOM.WW.LocalDb.Replication; +using ZB.MOM.WW.ScadaBridge.Host; +using ZB.MOM.WW.ScadaBridge.Host.Actors; + +namespace ZB.MOM.WW.ScadaBridge.Host.Tests; + +/// +/// LocalDb Phase 1 (Task 3) — locks the consolidated site database into the REAL site +/// composition root. +/// +/// These tests build the actual container from +/// rather than asserting on a +/// hand-assembled ServiceCollection. That is deliberate and load-bearing: this +/// family has now shipped three wiring defects that only a container-built graph would +/// have caught — the Secrets Akka replicator that registered into a no-op sink (0.2.0), +/// the singleton cycle that deadlocked a real host (0.2.2), and Secrets.Ui's missing +/// IAuditWriter that 500'd only behind a login wall (ScadaBridge#22). A test that +/// merely checks "AddZbLocalDb was called" would have passed in every one of those cases. +/// +/// +public class SiteLocalDbWiringTests : IDisposable +{ + private readonly WebApplication _host; + private readonly string _tempDbPath; + private readonly string _tempSiteDbPath; + private readonly string _tempTrackingDbPath; + + public SiteLocalDbWiringTests() + { + var stamp = Guid.NewGuid(); + _tempDbPath = Path.Combine(Path.GetTempPath(), $"scadabridge_localdb_{stamp}.db"); + _tempSiteDbPath = Path.Combine(Path.GetTempPath(), $"scadabridge_site_{stamp}.db"); + _tempTrackingDbPath = Path.Combine(Path.GetTempPath(), $"scadabridge_track_{stamp}.db"); + + var builder = WebApplication.CreateBuilder(); + builder.Configuration.Sources.Clear(); + builder.Configuration.AddInMemoryCollection(new Dictionary + { + ["ScadaBridge:Node:Role"] = "Site", + ["ScadaBridge:Node:NodeName"] = "node-a", + ["ScadaBridge:Node:NodeHostname"] = "test-site", + ["ScadaBridge:Node:SiteId"] = "TestSite", + ["ScadaBridge:Node:RemotingPort"] = "0", + ["ScadaBridge:Node:GrpcPort"] = "0", + ["ScadaBridge:Database:SiteDbPath"] = _tempSiteDbPath, + ["ScadaBridge:OperationTracking:ConnectionString"] = $"Data Source={_tempTrackingDbPath}", + ["ScadaBridge:Cluster:SeedNodes:0"] = "akka.tcp://scadabridge@localhost:2551", + ["ScadaBridge:Cluster:SeedNodes:1"] = "akka.tcp://scadabridge@localhost:2552", + + // The consolidated site database. No Replication section at all — this + // fixture is also the default-OFF pin: storage must work standalone. + ["LocalDb:Path"] = _tempDbPath, + }); + + builder.Services.AddGrpc(); + builder.Services.AddSingleton(); + + SiteServiceRegistration.Configure(builder.Services, builder.Configuration); + + AkkaHostedServiceRemover.RemoveAkkaHostedServiceOnly(builder.Services); + + _host = builder.Build(); + } + + public void Dispose() + { + (_host as IDisposable)?.Dispose(); + foreach (var path in new[] { _tempDbPath, _tempSiteDbPath, _tempTrackingDbPath }) + { + try { File.Delete(path); } catch { /* best effort */ } + } + + GC.SuppressFinalize(this); + } + + [Fact] + public void Site_Resolves_ILocalDb() + { + Assert.NotNull(_host.Services.GetService()); + } + + [Fact] + public void Site_LocalDb_RegistersBothReplicatedTables() + { + // The whole point of Phase 1. If onReady silently failed to register these, + // storage would still work and every other test would pass — the pair would + // just never replicate anything. Assert on the library's own view of what is + // registered, not on our belief that we called it. + var db = _host.Services.GetRequiredService(); + + Assert.Contains("OperationTracking", db.ReplicatedTables.Keys); + Assert.Contains("site_events", db.ReplicatedTables.Keys); + } + + [Fact] + public void Site_LocalDb_ReplicatedTablesHaveExplicitPrimaryKeys() + { + // RegisterReplicated refuses a table with no explicit PK, so reaching this + // assertion at all proves the DDL ran before registration. The PK names are + // pinned because they are what last-writer-wins conflict resolution keys on. + var db = _host.Services.GetRequiredService(); + + Assert.Equal( + ["TrackedOperationId"], db.ReplicatedTables["OperationTracking"].PkColumns); + Assert.Equal(["id"], db.ReplicatedTables["site_events"].PkColumns); + } + + [Fact] + public void Site_LocalDb_IsASingleton() + { + // AddZbLocalDb is one-DB-per-process and first-registration-wins. Two + // instances would mean two SQLite handles and two trigger installations + // against the same file. + var first = _host.Services.GetRequiredService(); + var second = _host.Services.GetRequiredService(); + + Assert.Same(first, second); + } + + [Fact] + public void Site_Replication_IsRegistered_AndInertWithNoPeer() + { + // Task 7's default-OFF pin. This fixture configures NO + // LocalDb:Replication:PeerAddress, which is the shipping default, so the + // engine must resolve cleanly and sit idle rather than throw, retry, or + // report a connection. "Default-off" here means inert, NOT unregistered — + // the services are always in the graph. + var status = _host.Services.GetService(); + + Assert.NotNull(status); + Assert.False(status!.Connected); + Assert.Null(status.PeerNodeId); + Assert.Null(status.LastSyncUtc); + } + + [Fact] + public void Site_Replication_OplogBacklog_IsZero_NotNull_OnAHealthyIdleNode() + { + // OplogBacklog is deliberately nullable: null means "unknown" (no provider + // wired, or the poll failed) and must never be reported as a healthy 0. On a + // node whose local database is present and readable, a real 0 proves the + // backlog provider is actually wired to the oplog — a null here would mean + // Task 9's health signal is about to publish "unknown" forever. + var status = _host.Services.GetRequiredService(); + + Assert.Equal(0L, status.OplogBacklog); + } + + [Fact] + public void Site_Replication_HealthBridge_IsRegisteredAsAHostedService() + { + // Task 9. Registered via a factory lambda, so ImplementationType is null and the + // only honest assertion is on the resolved instance. + var hosted = _host.Services.GetServices(); + + Assert.Contains(hosted, h => h is LocalDbReplicationStatusReporter); + } + + [Fact] + public void Site_Replication_HealthBridge_PublishesStatusOntoTheHealthReport() + { + // End-to-end through the REAL collector: probe once, then collect. This is what + // catches a bridge that is registered but wired to the wrong provider — the + // registration test above would pass either way. + var reporter = _host.Services.GetServices() + .OfType() + .Single(); + var collector = _host.Services.GetRequiredService(); + + reporter.Probe(); + var report = collector.CollectReport("TestSite"); + + // No peer configured: not connected, and a REAL 0 backlog rather than null. + Assert.False(report.LocalDbReplicationConnected); + Assert.Equal(0L, report.LocalDbOplogBacklog); + } + + [Fact] + public void Site_HealthReport_LocalDbFields_AreNull_BeforeTheReporterHasRun() + { + // Null means "no data yet", and must be distinguishable from a real + // "disconnected, zero backlog". A collector that defaulted these to false/0 would + // report an unwired node as a healthy connected one. + var collector = new SiteHealthCollector(); + + var report = collector.CollectReport("TestSite"); + + Assert.Null(report.LocalDbReplicationConnected); + Assert.Null(report.LocalDbOplogBacklog); + } + + [Fact] + public void Site_Telemetry_Allowlists_TheLocalDbReplicationMeter() + { + // ZbTelemetryOptions.Meters is an ALLOWLIST: an unlisted Meter's instruments are + // never observed, with no error and no missing-metric warning anywhere. The + // replication meter was silently absent from the rig's /metrics scrape until the + // live gate looked for it, so pin the allowlist rather than trusting the next + // reader to remember. + // + // This asserts on the allowlist constant, not on OTel's observed-meter set: + // AddZbTelemetry applies its options to a local instance and never registers + // IOptions, so there is nothing resolvable to interrogate. The end-to-end proof + // that localdb_* actually reaches /metrics is the Task 12 live gate; this test + // exists to stop the entry being dropped again. + Assert.Contains(LocalDbMetrics.MeterName, SiteServiceRegistration.ObservedMeters); + } + + [Fact] + public void Site_LocalDb_CreatesThePhase2Tables_InTheConsolidatedFile() + { + // Phase 2 (Task 7). The nine configuration tables and the store-and-forward buffer + // now live in the consolidated file rather than in scadabridge.db and + // store-and-forward.db. Asserting through the REAL composition root is what proves + // OnReady applies both schemas — the stores themselves also call Apply, so a test + // that went through a store would pass even if OnReady never touched them. + var db = _host.Services.GetRequiredService(); + + var tables = TableNames(db); + + // One from each Phase 1 schema, then every Phase 2 table. + Assert.Contains("OperationTracking", tables); + Assert.Contains("site_events", tables); + Assert.Contains("sf_messages", tables); + foreach (var table in new[] + { + "deployed_configurations", "static_attribute_overrides", "shared_scripts", + "external_systems", "database_connections", "notification_lists", + "data_connection_definitions", "smtp_configurations", "native_alarm_state", + }) + { + Assert.Contains(table, tables); + } + } + + [Fact] + public void Site_LocalDb_RegistersExactlyTheTenReplicatedTables() + { + // The Phase 2 cutover. This is an EXACTLY check rather than a Contains in both + // directions, and both directions are load-bearing. + // + // Too few: the table silently stops replicating. Storage still works, every other + // test still passes, and the pair just quietly diverges — the failure mode Phase 1 + // existed to end. + // + // Too many: notification_lists and smtp_configurations must NOT be here. They are + // permanently empty by design, and registering them would open a standing + // replication channel whose only historical payload was plaintext SMTP passwords. + // A Contains-based test would never catch that. + var db = _host.Services.GetRequiredService(); + + Assert.Equal( + [ + // Ordinal order: '_' (0x5F) sorts before 'b' (0x62), so + // data_connection_definitions precedes database_connections. + "OperationTracking", "data_connection_definitions", "database_connections", + "deployed_configurations", "external_systems", "native_alarm_state", + "sf_messages", "shared_scripts", "site_events", "static_attribute_overrides", + ], + db.ReplicatedTables.Keys.OrderBy(k => k, StringComparer.Ordinal).ToArray()); + } + + [Fact] + public void Site_LocalDb_DoesNotReplicateTheCentralOnlyNotificationTables() + { + // Stated separately from the exact-set test above because this one is a security + // property, not a wiring property, and deserves to fail with its own name. The + // tables exist (SiteStorageSchema creates them) — they simply must never be + // captured. See SiteLocalDbSetup.OnReady for the rationale. + var db = _host.Services.GetRequiredService(); + + Assert.DoesNotContain("notification_lists", db.ReplicatedTables.Keys); + Assert.DoesNotContain("smtp_configurations", db.ReplicatedTables.Keys); + } + + [Fact] + public void Site_LocalDb_ReplicatedPhase2TablesHaveTheirExpectedPrimaryKeys() + { + // RegisterReplicated refuses a table with no explicit PK, so reaching these + // assertions proves the DDL ran before registration. The composite keys are the + // interesting ones: LWW conflict resolution keys on the FULL PK, so a wrong or + // truncated key set would silently collapse distinct rows into one. + var db = _host.Services.GetRequiredService(); + + Assert.Equal(["id"], db.ReplicatedTables["sf_messages"].PkColumns); + Assert.Equal( + ["instance_unique_name"], db.ReplicatedTables["deployed_configurations"].PkColumns); + Assert.Equal( + ["instance_unique_name", "attribute_name"], + db.ReplicatedTables["static_attribute_overrides"].PkColumns); + Assert.Equal( + ["instance_unique_name", "source_canonical_name", "source_reference"], + db.ReplicatedTables["native_alarm_state"].PkColumns); + } + + private static HashSet TableNames(ILocalDb db) + { + using var connection = db.CreateConnection(); + using var cmd = connection.CreateCommand(); + cmd.CommandText = "SELECT name FROM sqlite_master WHERE type = 'table'"; + using var reader = cmd.ExecuteReader(); + + var names = new HashSet(StringComparer.Ordinal); + while (reader.Read()) names.Add(reader.GetString(0)); + return names; + } + + [Fact] + public void Site_LocalDb_CreatesTheConfiguredFile() + { + // Resolving is what triggers onReady; the file should exist afterwards at the + // configured path and not somewhere CWD-relative. + _ = _host.Services.GetRequiredService(); + + Assert.True(File.Exists(_tempDbPath), + $"Expected the consolidated site database at '{_tempDbPath}'."); + } +} diff --git a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/StartupValidatorTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/StartupValidatorTests.cs index de2f8375..3394b5dc 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/StartupValidatorTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/StartupValidatorTests.cs @@ -232,15 +232,22 @@ public class StartupValidatorTests Assert.Null(ex); } + /// + /// The inverse of the rule this replaces. SiteDbPath was mandatory for Site + /// nodes until LocalDb Phase 2 moved the site tables into the consolidated + /// LocalDb database; it now names only the legacy file the boot-time migrator + /// drains, so its absence means "nothing to migrate". Requiring it would make + /// every already-migrated node carry a dead key forever. + /// [Fact] - public void Site_MissingSiteDbPath_FailsValidation() + public void Site_MissingSiteDbPath_IsAccepted_BecauseThePathIsMigrationOnly() { var values = ValidSiteConfig(); values.Remove("ScadaBridge:Database:SiteDbPath"); var config = BuildConfig(values); - var ex = Assert.Throws(() => StartupValidator.Validate(config)); - Assert.Contains("SiteDbPath required for Site nodes", ex.Message); + var ex = Record.Exception(() => StartupValidator.Validate(config)); + Assert.Null(ex); } [Fact] diff --git a/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/SfBufferResyncPredicateTests.cs b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/SfBufferResyncPredicateTests.cs deleted file mode 100644 index 0dac6c10..00000000 --- a/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/SfBufferResyncPredicateTests.cs +++ /dev/null @@ -1,94 +0,0 @@ -using Akka.Actor; -using Microsoft.Extensions.Logging.Abstractions; -using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors; -using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; -using ZB.MOM.WW.ScadaBridge.StoreAndForward; -using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; - -namespace ZB.MOM.WW.ScadaBridge.IntegrationTests.Cluster; - -/// -/// N1 regression (review 02 round 2, Critical): the resync authority must use the same -/// oldest-Up predicate as the S&F delivery gate. Divergence scenario = the delivering node -/// is OLDEST but not LEADER (leader = lowest address), the exact state a rolling restart of -/// the lower-address node produces. Pre-fix the delivering node requests a resync from the -/// stale peer and ReplaceAllAsync wipes its live buffer. -/// -public class SfBufferResyncPredicateTests -{ - [Fact] - public async Task OldestButNotLeaderNode_KeepsItsBuffer_AndSeedsTheJoiner() - { - // Two explicit ports, deliberately assigned so the FIRST-started (oldest, - // delivering) node has the HIGHER address → the second node is cluster leader. - var p1 = TwoNodeClusterFixture.GetFreeTcpPort(); - var p2 = TwoNodeClusterFixture.GetFreeTcpPort(); - var (portHigh, portLow) = p1 > p2 ? (p1, p2) : (p2, p1); - - await using var fixture = await TwoNodeClusterFixture.StartAsync( - role: "site-int", portA: portHigh, portB: portLow); - - // Real S&F storage + replication actor per node, production default predicate - // (no isActiveOverride) — the exact wiring under test. - var (storageOldest, _) = await CreateReplicationActorAsync(fixture.NodeA, "oldest"); - var (storageJoiner, _) = await CreateReplicationActorAsync(fixture.NodeB, "joiner"); - - // The delivering (oldest) node has a live buffered row the standby never saw. - await storageOldest.EnqueueAsync(NewMessage("live-row")); - - // Trigger peer (re)tracking on both sides: each actor got InitialStateAsSnapshot - // in PreStart, but the enqueue raced it — re-deliver via a fresh MemberUp is not - // needed; OnPeerTracked already fired on join. The resync exchange is async: - // wait until the JOINER holds the row (proves the snapshot flowed oldest→joiner, - // the correct direction). Pre-fix this times out (the joiner, as leader, never - // requests) AND the oldest node's row is deleted by the stale wipe. - await AwaitAsync(async () => await storageJoiner.GetMessageByIdAsync("live-row") != null, - TimeSpan.FromSeconds(20), - "joiner never received the resync snapshot (resync ran in the wrong direction)"); - - // And the delivering node's buffer is untouched — the N1 wipe assertion. - Assert.NotNull(await storageOldest.GetMessageByIdAsync("live-row")); - } - - private static async Task<(StoreAndForwardStorage Storage, IActorRef Actor)> CreateReplicationActorAsync( - ActorSystem node, string tag) - { - var sfDb = Path.Combine(Path.GetTempPath(), $"sf-resync-{tag}-{Guid.NewGuid():N}.db"); - var siteDb = Path.Combine(Path.GetTempPath(), $"site-resync-{tag}-{Guid.NewGuid():N}.db"); - var sfStorage = new StoreAndForwardStorage($"Data Source={sfDb}", - NullLogger.Instance); - await sfStorage.InitializeAsync(); - var siteStorage = new SiteStorageService($"Data Source={siteDb}", - NullLogger.Instance); - var replicationService = new ReplicationService( - new StoreAndForwardOptions(), NullLogger.Instance); - // Name MUST be "site-replication" — SendToPeer targets /user/site-replication. - var actor = node.ActorOf(Props.Create(() => new SiteReplicationActor( - siteStorage, sfStorage, replicationService, "site-int", - NullLogger.Instance, null, null, null, null)), - "site-replication"); - return (sfStorage, actor); - } - - private static StoreAndForwardMessage NewMessage(string id) => new() - { - Id = id, - Category = StoreAndForwardCategory.Notification, - Target = "central", - PayloadJson = "{}", - CreatedAt = DateTimeOffset.UtcNow, - Status = StoreAndForwardMessageStatus.Pending, - MaxRetries = 0, - }; - - private static async Task AwaitAsync(Func> condition, TimeSpan timeout, string why) - { - var deadline = DateTime.UtcNow + timeout; - while (DateTime.UtcNow < deadline) - { - if (await condition()) return; - await Task.Delay(250); - } - throw new TimeoutException(why); - } -} diff --git a/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/DualNodeRecoveryTests.cs b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/DualNodeRecoveryTests.cs index 0cfc6485..87c3d89f 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/DualNodeRecoveryTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/DualNodeRecoveryTests.cs @@ -1,6 +1,7 @@ using Microsoft.Extensions.Logging.Abstractions; using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; using ZB.MOM.WW.ScadaBridge.StoreAndForward; +using ZB.MOM.WW.ScadaBridge.TestSupport; namespace ZB.MOM.WW.ScadaBridge.IntegrationTests; @@ -18,12 +19,17 @@ public class DualNodeRecoveryTests // Scenario: both site nodes crash. First node to restart opens the existing // SQLite database and finds all buffered S&F messages intact. var dbPath = Path.Combine(Path.GetTempPath(), $"sf_dual_{Guid.NewGuid():N}.db"); - var connStr = $"Data Source={dbPath}"; + + // The store takes an ILocalDb; the "restarted node" opens its own local database + // over the SAME file, which is how this test models recovery from disk. + TestLocalDb? crashedDb = null; + TestLocalDb? recoveryDb = null; try { // Setup: populate SQLite with messages (simulating pre-crash state) - var storage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + crashedDb = TestLocalDb.Create(dbPath); + var storage = new StoreAndForwardStorage(crashedDb.Db, NullLogger.Instance); await storage.InitializeAsync(); var messageIds = new List(); @@ -48,7 +54,8 @@ public class DualNodeRecoveryTests // Both nodes down — simulate by creating a fresh storage instance // (new process connecting to same SQLite file) - var recoveryStorage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + recoveryDb = TestLocalDb.Create(dbPath); + var recoveryStorage = new StoreAndForwardStorage(recoveryDb.Db, NullLogger.Instance); await recoveryStorage.InitializeAsync(); // Verify all messages are available for retry @@ -69,8 +76,10 @@ public class DualNodeRecoveryTests } finally { - if (File.Exists(dbPath)) - File.Delete(dbPath); + // Dispose before deleting — the master connections anchor the WAL sidecars. + recoveryDb?.Dispose(); + crashedDb?.Dispose(); + TestLocalDb.DeleteFiles(dbPath); } } @@ -79,11 +88,14 @@ public class DualNodeRecoveryTests public async Task SiteTopology_DualCrash_ParkedMessagesPreserved() { var dbPath = Path.Combine(Path.GetTempPath(), $"sf_dual_parked_{Guid.NewGuid():N}.db"); - var connStr = $"Data Source={dbPath}"; + + TestLocalDb? crashedDb = null; + TestLocalDb? recoveryDb = null; try { - var storage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + crashedDb = TestLocalDb.Create(dbPath); + var storage = new StoreAndForwardStorage(crashedDb.Db, NullLogger.Instance); await storage.InitializeAsync(); // Mix of pending and parked messages @@ -114,7 +126,8 @@ public class DualNodeRecoveryTests }); // Dual crash recovery - var recoveryStorage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + recoveryDb = TestLocalDb.Create(dbPath); + var recoveryStorage = new StoreAndForwardStorage(recoveryDb.Db, NullLogger.Instance); await recoveryStorage.InitializeAsync(); var pendingCount = await recoveryStorage.GetMessageCountByStatusAsync(StoreAndForwardMessageStatus.Pending); @@ -134,8 +147,10 @@ public class DualNodeRecoveryTests } finally { - if (File.Exists(dbPath)) - File.Delete(dbPath); + // Dispose before deleting — the master connections anchor the WAL sidecars. + recoveryDb?.Dispose(); + crashedDb?.Dispose(); + TestLocalDb.DeleteFiles(dbPath); } } @@ -176,11 +191,14 @@ public class DualNodeRecoveryTests { // CREATE TABLE IF NOT EXISTS is idempotent — safe to call on recovery var dbPath = Path.Combine(Path.GetTempPath(), $"sf_idempotent_{Guid.NewGuid():N}.db"); - var connStr = $"Data Source={dbPath}"; + + TestLocalDb? localDb1 = null; + TestLocalDb? localDb2 = null; try { - var storage1 = new StoreAndForwardStorage(connStr, NullLogger.Instance); + localDb1 = TestLocalDb.Create(dbPath); + var storage1 = new StoreAndForwardStorage(localDb1.Db, NullLogger.Instance); await storage1.InitializeAsync(); await storage1.EnqueueAsync(new StoreAndForwardMessage @@ -196,7 +214,8 @@ public class DualNodeRecoveryTests }); // Second InitializeAsync on same DB should be safe (no data loss) - var storage2 = new StoreAndForwardStorage(connStr, NullLogger.Instance); + localDb2 = TestLocalDb.Create(dbPath); + var storage2 = new StoreAndForwardStorage(localDb2.Db, NullLogger.Instance); await storage2.InitializeAsync(); var msg = await storage2.GetMessageByIdAsync("test-1"); @@ -205,8 +224,10 @@ public class DualNodeRecoveryTests } finally { - if (File.Exists(dbPath)) - File.Delete(dbPath); + // Dispose before deleting — the master connections anchor the WAL sidecars. + localDb2?.Dispose(); + localDb1?.Dispose(); + TestLocalDb.DeleteFiles(dbPath); } } } diff --git a/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/IntegrationSurfaceTests.cs b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/IntegrationSurfaceTests.cs index e52462db..e3276844 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/IntegrationSurfaceTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/IntegrationSurfaceTests.cs @@ -91,12 +91,12 @@ public class IntegrationSurfaceTests { // Notification Outbox: Notify.Send enqueues into the site Store-and-Forward // Engine and returns the NotificationId handle immediately. - var dbName = $"NotifyWired_{Guid.NewGuid():N}"; - var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared"; - using var keepAlive = new Microsoft.Data.Sqlite.SqliteConnection(connStr); - keepAlive.Open(); + // A real temp-file LocalDb, not the shared-cache in-memory database this used + // before: StoreAndForwardStorage takes ILocalDb now, and LocalDb has no + // in-memory mode. + using var localDb = ZB.MOM.WW.ScadaBridge.TestSupport.TestLocalDb.CreateTemp("NotifyWired"); var storage = new StoreAndForward.StoreAndForwardStorage( - connStr, Microsoft.Extensions.Logging.Abstractions.NullLogger.Instance); + localDb.Db, Microsoft.Extensions.Logging.Abstractions.NullLogger.Instance); await storage.InitializeAsync(); var saf = new StoreAndForward.StoreAndForwardService( storage, new StoreAndForward.StoreAndForwardOptions(), diff --git a/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbConfigConvergenceTests.cs b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbConfigConvergenceTests.cs new file mode 100644 index 00000000..418afe6b --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbConfigConvergenceTests.cs @@ -0,0 +1,177 @@ +using ZB.MOM.WW.LocalDb; + +namespace ZB.MOM.WW.ScadaBridge.IntegrationTests; + +/// +/// Phase 2 convergence: deployed configuration across a real site pair, plus the resync +/// behaviour that the bespoke replicator needed a directional guard for. +/// +/// +/// Companion to : same +/// specifications-before-deletion role, for the intents held by +/// SiteReplicationActorTests and SfBufferResyncPredicateTests rather than by +/// the store-and-forward ReplicationService. +/// +[Collection("LocalDbSitePairConvergence")] +public sealed class LocalDbConfigConvergenceTests : LocalDbSitePairHarness +{ + // ---- data helpers ----------------------------------------------------------------- + + private static Task DeployConfigAsync(ILocalDb db, string instance, string configJson, string deploymentId) + => db.ExecuteAsync( + """ + INSERT INTO deployed_configurations ( + instance_unique_name, config_json, deployment_id, revision_hash, + is_enabled, deployed_at) + VALUES (@instance, @configJson, @deploymentId, @deploymentId, 1, @now) + ON CONFLICT(instance_unique_name) DO UPDATE SET + config_json = excluded.config_json, + deployment_id = excluded.deployment_id, + revision_hash = excluded.revision_hash, + deployed_at = excluded.deployed_at; + """, + new { instance, configJson, deploymentId, now = DateTime.UtcNow.ToString("o") }); + + private static async Task ReadConfigAsync(ILocalDb db, string instance) + { + var rows = await db.QueryAsync( + "SELECT config_json FROM deployed_configurations WHERE instance_unique_name = @instance", + static r => r.GetString(0), new { instance }); + return rows.Count == 0 ? null : rows[0]; + } + + private static Task EnqueueAsync(ILocalDb db, string id) + => db.ExecuteAsync( + """ + INSERT INTO sf_messages ( + id, category, target, payload_json, retry_count, max_retries, + retry_interval_ms, created_at, status) + VALUES (@id, 0, 'central', '{}', 0, 50, 30000, @now, 0); + """, + new { id, now = DateTime.UtcNow.ToString("o") }); + + private static async Task MessageExistsAsync(ILocalDb db, string id) + { + var rows = await db.QueryAsync( + "SELECT 1 FROM sf_messages WHERE id = @id", static r => r.GetInt32(0), new { id }); + return rows.Count > 0; + } + + // ---- scenarios -------------------------------------------------------------------- + + [Fact] + public async Task ANodeWithNewerLocalRows_KeepsThem_WhenAPeerSnapshotArrives() + { + // N1 Critical (was: SfBufferResyncPredicateTests). That test existed because the + // bespoke replicator's ReplaceAllAsync was a destructive DELETE-then-INSERT: a resync + // running in the wrong direction WIPED a live buffer, so the code needed a + // same-oldest-Up predicate to decide who had authority. The divergence case was a + // rolling restart leaving the delivering node oldest but not leader. + // + // Under LocalDb that failure mode is structurally impossible rather than guarded + // against: snapshot resync merges per row under LWW and never deletes. So this is + // deliberately NOT a directional-authority test — there is no active/standby + // asymmetry left to enforce. It asserts the property the guard was protecting: no + // node loses rows to a peer's snapshot. + // + // The setup is the wipe scenario made concrete. Each node holds a row the other has + // never seen, and they disagree about a third; then they resync. + await DeployConfigAsync(A, "shared-instance", """{"v":1}""", "dep-1"); + await WaitUntilAsync( + async () => await ReadConfigAsync(B, "shared-instance") is not null, + "the shared row to exist on both nodes before they diverge"); + + await StopPassiveAsync(); + + // A keeps working while B is down: a brand-new buffered message and a newer version + // of the shared config. + await EnqueueAsync(A, "live-row-on-a"); + await DeployConfigAsync(A, "shared-instance", """{"v":2}""", "dep-2"); + + // B is offline but its database is still writable — it also has local state the + // snapshot exchange must not destroy. + await EnqueueAsync(B, "live-row-on-b"); + + await RestartPairAsync(); + + await WaitUntilAsync( + async () => + await MessageExistsAsync(A, "live-row-on-a") && + await MessageExistsAsync(A, "live-row-on-b") && + await MessageExistsAsync(B, "live-row-on-a") && + await MessageExistsAsync(B, "live-row-on-b"), + "both nodes to hold the UNION of the rows each wrote while partitioned"); + + // And the contended row converges to the newer write rather than either node's + // snapshot flattening the other. + await WaitUntilAsync( + async () => await ReadConfigAsync(A, "shared-instance") == """{"v":2}""" + && await ReadConfigAsync(B, "shared-instance") == """{"v":2}""", + "both nodes to converge on the newer config for the contended instance"); + } + + [Fact] + public async Task ConfigDeployedOnA_ConvergesToB_WithoutAnyFetch() + { + // Was: SiteReplicationActorTests #1-#5 (notify-and-fetch — the standby is told a + // deploy happened, then HTTP-fetches the config itself, with retries and a + // superseded-404 path). Under CDC the config row simply replicates, so that whole + // exchange is deleted at Task 15. + // + // The plan called for asserting a fetcher test double records zero invocations. That + // would be THEATRE here and is deliberately not done: this harness has no actor + // system, no central, and no IDeploymentConfigFetcher in the graph at all, so a + // double would record zero calls whether or not the fetch path still existed. An + // assertion that cannot fail is worse than none. + // + // What this test honestly proves is the positive half: the config reaches node B + // through replication ALONE, in a process where fetching is not merely unused but + // absent. The negative half — that no fetch path survives — is proved by Task 15 + // deleting the code and the build still passing, which is a stronger check than any + // runtime counter. + // + // Scope note (D1): SiteReconciliationActor survives Phase 2 and legitimately fetches + // over HTTP at node startup when central reports gaps. "The standby never fetches, + // ever" would be a FALSE claim; the claim is about the deploy path only. + await DeployConfigAsync(A, "inst-deploy", """{"setpoint":42}""", "dep-100"); + + await WaitUntilAsync( + async () => await ReadConfigAsync(B, "inst-deploy") == """{"setpoint":42}""", + "the deployed config to reach node B over replication alone"); + } + + [Fact] + public async Task RedeployingAnInstance_ConvergesToTheNewRevision_OnBothNodes() + { + // The steady-state deploy loop, and the reason the old path needed a deployed_at + // guard: a standby that applied a STALE fetch would pin an instance to superseded + // config until the next deploy. Under LWW the newer write wins on the primary key, + // so the guard is not what protects this — but the outcome still has to hold. + await DeployConfigAsync(A, "inst-redeploy", """{"rev":"a"}""", "dep-a"); + await WaitUntilAsync( + async () => await ReadConfigAsync(B, "inst-redeploy") == """{"rev":"a"}""", + "the first revision to reach node B"); + + await DeployConfigAsync(A, "inst-redeploy", """{"rev":"b"}""", "dep-b"); + + await WaitUntilAsync( + async () => await ReadConfigAsync(A, "inst-redeploy") == """{"rev":"b"}""" + && await ReadConfigAsync(B, "inst-redeploy") == """{"rev":"b"}""", + "both nodes to converge on the redeployed revision"); + } + + [Fact] + public async Task ConfigDeployedOnB_ConvergesToA() + { + // Deliberately the reverse direction. The bespoke replicator was asymmetric by + // design — the active node pushed, the standby fetched — so "which node deployed it" + // was a meaningful question. Under CDC it is not, and this test is what says so. + // After a failover the surviving node must be able to deploy without waiting to be + // promoted to some authority role. + await DeployConfigAsync(B, "inst-from-b", """{"origin":"b"}""", "dep-b1"); + + await WaitUntilAsync( + async () => await ReadConfigAsync(A, "inst-from-b") == """{"origin":"b"}""", + "a config deployed on node B to reach node A"); + } +} diff --git a/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbPhase2ConvergenceTests.cs b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbPhase2ConvergenceTests.cs new file mode 100644 index 00000000..610a9e4c --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbPhase2ConvergenceTests.cs @@ -0,0 +1,193 @@ +using Microsoft.Extensions.Logging.Abstractions; +using ZB.MOM.WW.LocalDb; +using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; + +namespace ZB.MOM.WW.ScadaBridge.IntegrationTests; + +/// +/// Phase 2 convergence driven through the REAL , over a real +/// site pair, rather than through hand-written SQL. +/// +/// +/// +/// The sibling suites (, +/// ) were written as specifications +/// BEFORE the cutover, so they necessarily use hand-written SQL — the production writers were +/// still going through the bespoke replicator at the time. This suite runs after the cutover +/// and therefore drives the production writer, which is what makes the cascade scenario below +/// meaningful: it is the shipped multi-statement transaction under test, not a re-creation +/// of it. +/// +/// +/// is constructed directly on the fixture's +/// — the same instance the host registers — so its writes flow through +/// the same CDC triggers. +/// +/// +[Collection("LocalDbSitePairConvergence")] +public sealed class LocalDbPhase2ConvergenceTests : LocalDbSitePairHarness +{ + private SiteStorageService Storage(ILocalDb db) => + new(db, NullLogger.Instance); + + // ---- read helpers ----------------------------------------------------------------- + + private static Task CountAsync(ILocalDb db, string table, string instance) => + ScalarAsync(db, $"SELECT COUNT(*) FROM {table} WHERE instance_unique_name = @instance", + new { instance }); + + private static async Task ScalarAsync(ILocalDb db, string sql, object? args = null) + { + var rows = await db.QueryAsync(sql, static r => r.GetInt64(0), args); + return rows[0]; + } + + /// + /// The whole deployed_configurations row as one comparable string, so a scenario + /// can assert every column replicated rather than just the payload it happened to check. + /// + private static async Task ReadWholeConfigRowAsync(ILocalDb db, string instance) + { + var rows = await db.QueryAsync( + """ + SELECT instance_unique_name || '|' || config_json || '|' || deployment_id || '|' || + revision_hash || '|' || is_enabled || '|' || deployed_at + FROM deployed_configurations WHERE instance_unique_name = @instance + """, + static r => r.GetString(0), new { instance }); + return rows.Count == 0 ? null : rows[0]; + } + + /// Unacked oplog backlog. Drains to zero once the peer acknowledges. + private static Task OplogDepthAsync(ILocalDb db) => + ScalarAsync(db, + "SELECT COUNT(*) FROM __localdb_oplog WHERE seq > " + + "(SELECT last_acked_seq FROM __localdb_peer_state WHERE id = 1)"); + + // ---- scenarios -------------------------------------------------------------------- + + [Fact] + public async Task DeployedConfigRow_ConvergesToB_ColumnForColumn() + { + // LocalDbConfigConvergenceTests asserts config_json converges. That is the payload + // only: a capture that dropped, defaulted or reordered any other column would still + // pass it. deployed_at is the one that matters most — SiteReconciliationActor's + // guarded write compares it, so a node whose replicated copy carried a different + // timestamp would make different staleness decisions from its peer. + await Storage(A).StoreDeployedConfigAsync( + "inst-columns", """{"setpoint":7}""", "dep-col-1", "hash-col-1", isEnabled: true); + + var onA = await ReadWholeConfigRowAsync(A, "inst-columns"); + Assert.NotNull(onA); + + await WaitUntilAsync( + async () => await ReadWholeConfigRowAsync(B, "inst-columns") == onA, + "node B's deployed_configurations row to match node A's in EVERY column"); + } + + [Fact] + public async Task RemovingAnInstance_ConvergesAllThreeCascadeTables() + { + // The plan flagged this as the most likely real defect in Phase 2, and the reasoning + // is sound: RemoveDeployedConfigAsync deletes from three tables in one transaction, + // the schema has NO foreign keys, and CDC captures the three deletes as independent + // per-table streams that LWW may apply to the peer in any order. Nothing in the + // system re-derives an orphan, so a dropped delete leaves a permanently stale + // static_attribute_overrides or native_alarm_state row on the standby — invisible + // until that instance name is redeployed and picks up ghost overrides. + var storageA = Storage(A); + + // A second instance that is never removed. Without it this test cannot distinguish + // "the cascade converged" from "node B lost these tables entirely" — a registration + // bug that wiped B would satisfy the absence assertions on its own. + foreach (var instance in new[] { "inst-cascade", "inst-survivor" }) + { + await storageA.StoreDeployedConfigAsync( + instance, """{"v":1}""", $"dep-{instance}", $"hash-{instance}", isEnabled: true); + await storageA.SetStaticOverrideAsync(instance, "Setpoint", "42"); + await storageA.SetStaticOverrideAsync(instance, "Mode", "Auto"); + await storageA.UpsertNativeAlarmAsync( + instance, "Area1.Line1.Alarm", "ref-1", """{"active":true}""", DateTimeOffset.UtcNow); + } + + await WaitUntilAsync( + async () => + await CountAsync(B, "deployed_configurations", "inst-cascade") == 1 && + await CountAsync(B, "static_attribute_overrides", "inst-cascade") == 2 && + await CountAsync(B, "native_alarm_state", "inst-cascade") == 1, + "all three tables to reach node B before the instance is removed"); + + await storageA.RemoveDeployedConfigAsync("inst-cascade"); + + await WaitUntilAsync( + async () => + await CountAsync(B, "deployed_configurations", "inst-cascade") == 0 && + await CountAsync(B, "static_attribute_overrides", "inst-cascade") == 0 && + await CountAsync(B, "native_alarm_state", "inst-cascade") == 0, + "the 3-table cascade delete to converge on node B with NO orphans left behind"); + + // The control instance is untouched on both nodes: the cascade is scoped, not a wipe. + Assert.Equal(1, await CountAsync(B, "deployed_configurations", "inst-survivor")); + Assert.Equal(2, await CountAsync(B, "static_attribute_overrides", "inst-survivor")); + Assert.Equal(1, await CountAsync(B, "native_alarm_state", "inst-survivor")); + } + + [Fact] + public async Task ANativeAlarmBurst_Converges_AndTheOplogDrains() + { + // native_alarm_state is the highest-rate replicated table (Task 1 found it bounded by + // per-SourceReference coalescing on a 100 ms flush, correcting the plan's D4). This + // asserts the burst converges AND that the oplog goes back to empty afterwards — + // convergence alone would still pass if entries replicated but were never acked, and + // an oplog that only grows eventually trips the caps and forces a snapshot resync. + const int burst = 200; + var storageA = Storage(A); + var at = DateTimeOffset.UtcNow; + + for (var i = 0; i < burst; i++) + { + await storageA.UpsertNativeAlarmAsync( + "inst-burst", "Area1.Line1.Alarm", $"ref-{i}", $$"""{"seq":{{i}}}""", at); + } + + await WaitUntilAsync( + async () => await CountAsync(B, "native_alarm_state", "inst-burst") == burst, + $"all {burst} alarm rows to reach node B"); + + await WaitUntilAsync( + async () => await OplogDepthAsync(A) == 0, + "node A's oplog to drain once node B acknowledges the burst"); + } + + [Fact] + public async Task RowsWrittenOnBWhileItsListenerIsDown_SurviveTheRejoin() + { + // Companion to LocalDbConfigConvergenceTests' N1 scenario, which covers the same + // rejoin for sf_messages and deployed_configurations. This one exercises the tables + // that scenario does not touch, because "the union survives" is a per-table property: + // it holds only if each table is actually registered, and an unregistered table is + // silently local-only rather than an error. + await Storage(A).StoreSharedScriptAsync("script-before", "return 1;", null, null); + await WaitUntilAsync( + async () => await ScalarAsync(B, + "SELECT COUNT(*) FROM shared_scripts WHERE name = 'script-before'") == 1, + "the pre-outage script to reach node B"); + + await StopPassiveAsync(); + + // Each node writes to a different Phase 2 table while they cannot see each other. + await Storage(A).StoreExternalSystemAsync( + "sys-from-a", "http://a.invalid", "None", null, null); + await Storage(B).SetStaticOverrideAsync("inst-offline", "WrittenWhileDown", "yes"); + + await RestartPairAsync(); + + await WaitUntilAsync( + async () => + await ScalarAsync(A, + "SELECT COUNT(*) FROM static_attribute_overrides WHERE attribute_name = 'WrittenWhileDown'") == 1 && + await ScalarAsync(B, + "SELECT COUNT(*) FROM external_systems WHERE name = 'sys-from-a'") == 1, + "both nodes to hold the union of what each wrote while partitioned"); + } +} diff --git a/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbSitePairConvergenceTests.cs b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbSitePairConvergenceTests.cs new file mode 100644 index 00000000..a8ee9ad1 --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbSitePairConvergenceTests.cs @@ -0,0 +1,152 @@ +using ZB.MOM.WW.LocalDb; + +namespace ZB.MOM.WW.ScadaBridge.IntegrationTests; + +/// +/// Phase 1 convergence: operation tracking and site events across a real site pair. +/// +/// +/// This is the test that answers the question Phase 1 exists to answer: does a site node +/// pair actually stop losing operation-tracking and site-event state? Everything upstream +/// of it — schema helpers, DI wiring, the interceptor — can be individually green while the +/// pair still fails to converge. +/// +/// The fixture lives in , shared with the Phase 2 +/// convergence suites. +/// +/// +[Collection("LocalDbSitePairConvergence")] +public sealed class LocalDbSitePairConvergenceTests : LocalDbSitePairHarness +{ + // ---- data helpers ----------------------------------------------------------------- + + private static Task WriteTrackingAsync(ILocalDb db, string id, string status, string target) + => db.ExecuteAsync( + """ + INSERT INTO OperationTracking ( + TrackedOperationId, Kind, TargetSummary, Status, RetryCount, + CreatedAtUtc, UpdatedAtUtc) + VALUES (@id, 'ApiCallCached', @target, @status, 0, @now, @now) + ON CONFLICT(TrackedOperationId) DO UPDATE SET + Status = excluded.Status, + TargetSummary = excluded.TargetSummary, + UpdatedAtUtc = excluded.UpdatedAtUtc; + """, + new { id, status, target, now = DateTime.UtcNow.ToString("o") }); + + private static Task WriteEventAsync(ILocalDb db, string id, string message) + => db.ExecuteAsync( + """ + INSERT INTO site_events (id, timestamp, event_type, severity, source, message) + VALUES (@id, @ts, 'script', 'Info', 'convergence-test', @message); + """, + new { id, ts = DateTimeOffset.UtcNow.ToString("o"), message }); + + private static async Task ReadTrackingStatusAsync(ILocalDb db, string id) + { + var rows = await db.QueryAsync( + "SELECT Status FROM OperationTracking WHERE TrackedOperationId = @id", + static r => r.GetString(0), new { id }); + return rows.Count == 0 ? null : rows[0]; + } + + private static Task> ReadEventIdsAsync(ILocalDb db) + => db.QueryAsync("SELECT id FROM site_events ORDER BY id", static r => r.GetString(0)); + + // ---- scenarios -------------------------------------------------------------------- + + [Fact] + public async Task TrackingRow_WrittenOnA_BecomesReadableOnB() + { + await WriteTrackingAsync(A, "op-a-1", "Submitted", "ERP.GetOrder"); + + await WaitUntilAsync( + async () => await ReadTrackingStatusAsync(B, "op-a-1") == "Submitted", + "the tracking row written on node A to appear on node B"); + } + + [Fact] + public async Task TrackingRow_WrittenOnB_BecomesReadableOnA() + { + // Replication is bidirectional even though only A dials: proving the passive node's + // writes flow back is what makes a failover in EITHER direction safe. + await WriteTrackingAsync(B, "op-b-1", "Submitted", "ERP.GetOrder"); + + await WaitUntilAsync( + async () => await ReadTrackingStatusAsync(A, "op-b-1") == "Submitted", + "the tracking row written on node B to appear on node A"); + } + + [Fact] + public async Task SameOperation_UpdatedOnBothNodes_ConvergesToOneWinner() + { + // Last-writer-wins on the primary key. The specific winner is not asserted — that is + // the HLC's business — but the two nodes MUST agree, and must agree on a value one of + // them actually wrote rather than a merge of both. + await WriteTrackingAsync(A, "op-conflict", "Submitted", "ERP.GetOrder"); + await WaitUntilAsync( + async () => await ReadTrackingStatusAsync(B, "op-conflict") is not null, + "the conflicting row to exist on both nodes before it is updated"); + + await WriteTrackingAsync(A, "op-conflict", "Delivered", "ERP.GetOrder"); + await WriteTrackingAsync(B, "op-conflict", "Parked", "ERP.GetOrder"); + + await WaitUntilAsync( + async () => + { + var a = await ReadTrackingStatusAsync(A, "op-conflict"); + var b = await ReadTrackingStatusAsync(B, "op-conflict"); + return a is not null && a == b; + }, + "both nodes to converge on one status for the contended operation"); + + var winner = await ReadTrackingStatusAsync(A, "op-conflict"); + Assert.Contains(winner, new[] { "Delivered", "Parked" }); + } + + [Fact] + public async Task EventsLoggedOnBothNodes_ConvergeToTheUnion_WithNoIdCollisions() + { + // The reason site_events moved to GUID ids. Under the old autoincrement scheme both + // nodes would independently mint id=1, id=2, ... and last-writer-wins on the primary + // key would silently DESTROY one node's events instead of merging them. The union + // count is the assertion that catches that. + var idsFromA = Enumerable.Range(0, 5).Select(_ => Guid.NewGuid().ToString("N")).ToList(); + var idsFromB = Enumerable.Range(0, 5).Select(_ => Guid.NewGuid().ToString("N")).ToList(); + + foreach (var id in idsFromA) await WriteEventAsync(A, id, "from A"); + foreach (var id in idsFromB) await WriteEventAsync(B, id, "from B"); + + var expected = idsFromA.Concat(idsFromB).OrderBy(x => x, StringComparer.Ordinal).ToList(); + + await WaitUntilAsync( + async () => (await ReadEventIdsAsync(A)).SequenceEqual(expected) + && (await ReadEventIdsAsync(B)).SequenceEqual(expected), + "both nodes to hold the union of all 10 events"); + } + + [Fact] + public async Task PeerOffline_ThenRejoins_CatchesUpOnEverythingItMissed() + { + // The failover case that motivates Phase 1: one node is down while the other keeps + // working, and nothing written during the outage may be lost. + await StopPassiveAsync(); + + var idsDuringOutage = Enumerable.Range(0, 5).Select(_ => Guid.NewGuid().ToString("N")).ToList(); + foreach (var id in idsDuringOutage) await WriteEventAsync(A, id, "written while B was down"); + await WriteTrackingAsync(A, "op-during-outage", "Delivered", "ERP.GetOrder"); + + // Node B's database survived the host teardown (pre-constructed instance), so this is + // a genuine rejoin rather than a fresh node. + await RestartPairAsync(); + + await WaitUntilAsync( + async () => + { + var events = await ReadEventIdsAsync(B); + return idsDuringOutage.All(events.Contains) + && await ReadTrackingStatusAsync(B, "op-during-outage") == "Delivered"; + }, + "node B to catch up on everything written while it was offline"); + } +} diff --git a/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbSitePairHarness.cs b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbSitePairHarness.cs new file mode 100644 index 00000000..0bc73e80 --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbSitePairHarness.cs @@ -0,0 +1,260 @@ +using System.Net; +using Microsoft.AspNetCore.Builder; +using Microsoft.AspNetCore.Hosting; +using Microsoft.AspNetCore.Hosting.Server; +using Microsoft.AspNetCore.Hosting.Server.Features; +using Microsoft.AspNetCore.Server.Kestrel.Core; +using Microsoft.Extensions.Configuration; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; +using ZB.MOM.WW.LocalDb; +using ZB.MOM.WW.LocalDb.Replication; +using ZB.MOM.WW.ScadaBridge.Host; + +namespace ZB.MOM.WW.ScadaBridge.IntegrationTests; + +/// +/// Serializes the site-pair convergence tests against each other: each one stands up a real +/// Kestrel listener plus two SQLite files, and running them concurrently under CI +/// contention is a flakiness risk. +/// +[CollectionDefinition("LocalDbSitePairConvergence")] +public sealed class LocalDbSitePairConvergenceCollection; + +/// +/// Two ScadaBridge site nodes replicating the consolidated site database over a REAL +/// loopback gRPC transport, through the REAL fail-closed auth interceptor. +/// +/// +/// +/// Extracted from the Phase 1 convergence tests once Phase 2 needed the same pair for the +/// store-and-forward buffer and the configuration tables. Everything here is fixture; the +/// derived classes hold only their own data helpers and scenarios. +/// +/// +/// It uses , not a hand-written schema, so the tables, +/// their primary keys, and the registration ORDER under test are the ones the host actually +/// runs. A separate schema here would prove only that the test agrees with itself. +/// +/// +/// Offline: no docker, no external services. Loopback Kestrel with h2c. +/// +/// +public abstract class LocalDbSitePairHarness : IAsyncLifetime +{ + private const string SharedApiKey = "site-pair-convergence-key"; + + /// How long a scenario waits for the pair to agree before failing. + protected static readonly TimeSpan ConvergeTimeout = TimeSpan.FromSeconds(30); + + private readonly string _pathA = Path.Combine(Path.GetTempPath(), $"sitepairA-{Guid.NewGuid():N}.db"); + private readonly string _pathB = Path.Combine(Path.GetTempPath(), $"sitepairB-{Guid.NewGuid():N}.db"); + + // The databases are owned by the fixture, in their own providers, and registered into the + // hosts as pre-constructed instances. MS.DI does not dispose instances it did not create, + // so tearing a host down (the offline-peer scenario) leaves the databases intact and + // writable — which is exactly what lets node A accumulate writes while B is down. + private ServiceProvider _dbProviderA = null!; + private ServiceProvider _dbProviderB = null!; + + private IHost? _serverHost; // node B — passive + private IHost? _initiatorHost; // node A — dials the peer + + static LocalDbSitePairHarness() => + // Grpc.Net.Client dials the loopback server over HTTP/2 cleartext. + AppContext.SetSwitch("System.Net.Http.SocketsHttpHandler.Http2UnencryptedSupport", true); + + /// Node A — the initiator, which dials the peer. + protected ILocalDb A => _dbProviderA.GetRequiredService(); + + /// Node B — the passive node, which listens. + protected ILocalDb B => _dbProviderB.GetRequiredService(); + + public async Task InitializeAsync() + { + _dbProviderA = BuildDatabaseProvider(_pathA, "node-a"); + _dbProviderB = BuildDatabaseProvider(_pathB, "node-b"); + + // Force construction (and therefore OnReady) before anything replicates. + _ = A; + _ = B; + + await StartPassiveAsync(); + await StartInitiatorAsync(); + } + + public async Task DisposeAsync() + { + await StopHostAsync(_initiatorHost); + await StopHostAsync(_serverHost); + await _dbProviderA.DisposeAsync(); + await _dbProviderB.DisposeAsync(); + + Microsoft.Data.Sqlite.SqliteConnection.ClearAllPools(); + foreach (var path in new[] { _pathA, _pathB }) + { + foreach (var suffix in new[] { "", "-wal", "-shm" }) + { + try { File.Delete(path + suffix); } catch { /* best effort */ } + } + } + } + + // ---- fixture internals ------------------------------------------------------------ + + /// + /// A provider owning one consolidated site database, initialized through the host's own + /// — same schema, same registration order. + /// + private static ServiceProvider BuildDatabaseProvider(string path, string nodeName) + { + var config = new ConfigurationBuilder() + .AddInMemoryCollection(new Dictionary + { + ["LocalDb:Path"] = path, + ["ScadaBridge:Node:NodeName"] = nodeName, + // Point the legacy migrators at paths that do not exist, so they no-op rather + // than picking up stray files from the test working directory. The two Phase 2 + // defaults matter most: unlike the Phase 1 pair they resolve inside ./data/, + // and a migration would also RENAME whatever it found. + ["ScadaBridge:OperationTracking:ConnectionString"] = + $"Data Source={Path.Combine(Path.GetTempPath(), $"absent-{Guid.NewGuid():N}.db")}", + ["ScadaBridge:SiteEventLog:DatabasePath"] = + Path.Combine(Path.GetTempPath(), $"absent-{Guid.NewGuid():N}.db"), + ["ScadaBridge:StoreAndForward:SqliteDbPath"] = + Path.Combine(Path.GetTempPath(), $"absent-{Guid.NewGuid():N}.db"), + ["ScadaBridge:Database:SiteDbPath"] = + Path.Combine(Path.GetTempPath(), $"absent-{Guid.NewGuid():N}.db"), + }) + .Build(); + + return new ServiceCollection() + .AddZbLocalDb(config, db => SiteLocalDbSetup.OnReady(db, config)) + .BuildServiceProvider(); + } + + /// + /// The eight tables the Phase 2 cutover registers. Listed literally rather than derived + /// from production code, so a registration that drifts fails these tests instead of + /// agreeing with itself. + /// + /// + /// notification_lists and smtp_configurations are absent by design. They + /// are permanently empty (no site writer since 2026-07-10, the migrator skips them, the + /// active-node purge keeps them empty), so registering them would open a standing + /// replication channel whose only historical payload was plaintext SMTP passwords. + /// + protected static readonly string[] Phase2ReplicatedTables = + [ + "sf_messages", + "deployed_configurations", "static_attribute_overrides", "shared_scripts", + "external_systems", "database_connections", "data_connection_definitions", + "native_alarm_state", + ]; + + private static IConfiguration ReplicationConfig(string? peerAddress) + { + var values = new Dictionary + { + // Tight flush + bounded reconnect backoff so convergence is observable well + // inside the poll deadline. The 60 s production default would let the doubling + // backoff overrun it after a peer outage. + ["LocalDb:Replication:FlushInterval"] = "00:00:00.050", + ["LocalDb:Replication:ReconnectBackoffMax"] = "00:00:02", + // Both nodes share one key — the interceptor is fail-closed, so a mismatch here + // turns every scenario below red (verified by deliberately breaking it). + ["LocalDb:Replication:ApiKey"] = SharedApiKey, + }; + + if (peerAddress is not null) + values["LocalDb:Replication:PeerAddress"] = peerAddress; + + return new ConfigurationBuilder().AddInMemoryCollection(values).Build(); + } + + /// Starts node B, the passive listener. + protected async Task StartPassiveAsync() + { + var config = ReplicationConfig(peerAddress: null); + + _serverHost = await new HostBuilder() + .ConfigureWebHost(web => + { + web.UseKestrel(o => + o.Listen(IPAddress.Loopback, 0, listen => listen.Protocols = HttpProtocols.Http2)); + web.ConfigureServices(services => + { + services.AddLogging(); + services.AddRouting(); + // The REAL interceptor, not a stand-in. If it rejected legitimate peer + // traffic, every scenario below would fail — which is the point. + services.AddGrpc(o => o.Interceptors.Add()); + services.AddSingleton(B); + services.AddZbLocalDbReplication(config); + }); + web.Configure(app => + { + app.UseRouting(); + app.UseEndpoints(e => e.MapZbLocalDbSync()); + }); + }) + .StartAsync(); + } + + /// Starts node A, which dials the passive node. + protected async Task StartInitiatorAsync() + { + var config = ReplicationConfig(PassiveAddress()); + + _initiatorHost = await new HostBuilder() + .ConfigureServices(services => + { + services.AddLogging(); + services.AddSingleton(A); + services.AddZbLocalDbReplication(config); + }) + .StartAsync(); + } + + /// Takes node B's listener down, leaving its database intact and writable. + protected async Task StopPassiveAsync() + { + await StopHostAsync(_serverHost); + _serverHost = null; + } + + /// + /// Brings node B back on a NEW loopback port and re-dials from A. The initiator's channel + /// factory re-reads the peer address on each reconnect, so this is a genuine rejoin. + /// + protected async Task RestartPairAsync() + { + await StartPassiveAsync(); + await StopHostAsync(_initiatorHost); + await StartInitiatorAsync(); + } + + private string PassiveAddress() + => _serverHost!.Services.GetRequiredService() + .Features.Get()!.Addresses.Single(); + + private static async Task StopHostAsync(IHost? host) + { + if (host is null) return; + try { await host.StopAsync(TimeSpan.FromSeconds(5)); } catch { /* teardown */ } + host.Dispose(); + } + + /// Polls until true or the deadline passes. + protected static async Task WaitUntilAsync(Func> condition, string because) + { + var deadline = DateTime.UtcNow + ConvergeTimeout; + while (DateTime.UtcNow < deadline) + { + if (await condition()) return; + await Task.Delay(50); + } + + Assert.Fail($"Timed out after {ConvergeTimeout.TotalSeconds:0}s waiting for: {because}"); + } +} diff --git a/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbStoreAndForwardConvergenceTests.cs b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbStoreAndForwardConvergenceTests.cs new file mode 100644 index 00000000..409f23d8 --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbStoreAndForwardConvergenceTests.cs @@ -0,0 +1,221 @@ +using ZB.MOM.WW.LocalDb; +using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; + +namespace ZB.MOM.WW.ScadaBridge.IntegrationTests; + +/// +/// Phase 2 convergence: the store-and-forward buffer across a real site pair. +/// +/// +/// +/// These are specifications, written before the deletion they justify. The bespoke +/// ReplicationService (an explicit Add/Remove/Park/Requeue operation stream over Akka) +/// is deleted at Task 14 and replaced by LocalDb's trigger-based change capture. Each test +/// here is one behaviour the old mechanism provided, restated as an outcome the replacement +/// must still deliver — so the cutover has something to be judged against other than "it +/// compiles". +/// +/// +/// They are written in terms of rows, not operations, which is the whole point: +/// under CDC there is no Add or Park message to observe, only a row that must end up in the +/// right state on both nodes. +/// +/// +/// One intent is deliberately not ported: +/// ReplicationServiceTests.ReplicationOperations_AreDispatchedInIssueOrder — 200 +/// interleaved operations dispatched synchronously, observed in strict issue order. That +/// asserts the mechanism (inline fire-and-forget dispatch), not an outcome, and CDC +/// capture is asynchronous and batched by construction, so no honest port exists. Its +/// portable content is the ordering outcome: an add followed by a remove must never +/// converge to present. That is +/// . This paragraph exists so a +/// future reader does not conclude the test was dropped by accident. +/// +/// +[Collection("LocalDbSitePairConvergence")] +public sealed class LocalDbStoreAndForwardConvergenceTests : LocalDbSitePairHarness +{ + // ---- data helpers ----------------------------------------------------------------- + + /// Buffers a message, as StoreAndForwardStorage.EnqueueAsync does. + private static Task EnqueueAsync(ILocalDb db, string id, string target = "ERP.GetOrder") + => db.ExecuteAsync( + """ + INSERT INTO sf_messages ( + id, category, target, payload_json, retry_count, max_retries, + retry_interval_ms, created_at, status) + VALUES (@id, 0, @target, '{"order":1}', 0, 50, 30000, @now, @pending) + ON CONFLICT(id) DO UPDATE SET + status = excluded.status, + retry_count = excluded.retry_count; + """, + new + { + id, + target, + now = DateTime.UtcNow.ToString("o"), + pending = (int)StoreAndForwardMessageStatus.Pending, + }); + + /// Deletes a delivered message, as the successful-retry path does. + private static Task DeleteAsync(ILocalDb db, string id) + => db.ExecuteAsync("DELETE FROM sf_messages WHERE id = @id;", new { id }); + + private static Task SetStatusAsync(ILocalDb db, string id, StoreAndForwardMessageStatus status, int retryCount) + => db.ExecuteAsync( + "UPDATE sf_messages SET status = @status, retry_count = @retryCount WHERE id = @id;", + new { id, status = (int)status, retryCount }); + + private static async Task<(int Status, int RetryCount)?> ReadAsync(ILocalDb db, string id) + { + var rows = await db.QueryAsync( + "SELECT status, retry_count FROM sf_messages WHERE id = @id", + static r => (r.GetInt32(0), r.GetInt32(1)), new { id }); + return rows.Count == 0 ? null : rows[0]; + } + + private static async Task ExistsAsync(ILocalDb db, string id) + => await ReadAsync(db, id) is not null; + + private static async Task HasStatusAsync( + ILocalDb db, string id, StoreAndForwardMessageStatus status) + => await ReadAsync(db, id) is { } row && row.Status == (int)status; + + // ---- scenarios -------------------------------------------------------------------- + + [Fact] + public async Task BufferedMessage_MaterialisesOnThePeer() + { + // Was: BufferingAMessage_ReplicatesAnAddOperation. + await EnqueueAsync(A, "msg-add"); + + await WaitUntilAsync( + () => ExistsAsync(B, "msg-add"), + "a message buffered on node A to appear on node B"); + } + + [Fact] + public async Task DeliveredMessage_DisappearsFromThePeer() + { + // Was: SuccessfulRetry_ReplicatesARemoveOperation. Row deletes propagate as + // tombstones under CDC — the peer must not keep re-delivering a message that node A + // already succeeded on. + await EnqueueAsync(A, "msg-remove"); + await WaitUntilAsync(() => ExistsAsync(B, "msg-remove"), "the message to arrive on B first"); + + await DeleteAsync(A, "msg-remove"); + + await WaitUntilAsync( + async () => !await ExistsAsync(B, "msg-remove"), + "the delivered message to be gone from node B"); + } + + [Fact] + public async Task ParkedMessage_ShowsAsParkedOnThePeer() + { + // Was: ParkedMessage_ReplicatesAParkOperation. + await EnqueueAsync(A, "msg-park"); + await WaitUntilAsync(() => ExistsAsync(B, "msg-park"), "the message to arrive on B first"); + + await SetStatusAsync(A, "msg-park", StoreAndForwardMessageStatus.Parked, retryCount: 50); + + await WaitUntilAsync( + () => HasStatusAsync(B, "msg-park", StoreAndForwardMessageStatus.Parked), + "the parked status to reach node B"); + } + + [Fact] + public async Task RequeuedMessage_ResetsStatusAndRetryCountOnThePeer() + { + // Was: RetryingAParkedMessage_ReplicatesARequeueOperation + + // ApplyReplicatedOperation_Requeue_MovesStandbyRowBackToPending. RetryCount matters + // as much as status: a peer that took Pending but kept retry_count at max would park + // the message again on its first attempt after a failover. + await EnqueueAsync(A, "msg-requeue"); + await SetStatusAsync(A, "msg-requeue", StoreAndForwardMessageStatus.Parked, retryCount: 50); + await WaitUntilAsync( + () => HasStatusAsync(B, "msg-requeue", StoreAndForwardMessageStatus.Parked), + "the message to be parked on B before it is requeued"); + + await SetStatusAsync(A, "msg-requeue", StoreAndForwardMessageStatus.Pending, retryCount: 0); + + await WaitUntilAsync( + async () => await ReadAsync(B, "msg-requeue") + is { Status: (int)StoreAndForwardMessageStatus.Pending, RetryCount: 0 }, + "node B to show the requeued message as Pending with retry_count 0"); + } + + [Fact] + public async Task MessageAddedThenRemoved_NeverConvergesToPresent() + { + // The portable intent of ReplicationOperations_AreDispatchedInIssueOrder (see the + // class remarks). Ordering only ever mattered because a remove overtaken by its own + // add would resurrect a delivered message and send it twice. Under LWW that + // reordering is not preventable by dispatch discipline — it is prevented because the + // tombstone carries the later HLC and therefore wins regardless of arrival order. + // + // Asserting the endpoint rather than the sequence is what makes this portable: the + // old test would fail on any async transport even when the outcome was correct. + await EnqueueAsync(A, "msg-ordering"); + await DeleteAsync(A, "msg-ordering"); + + // A control row written in the same window. Without it this test is VACUOUS: an + // absent row is also what a pair that replicates nothing at all looks like, so it + // would pass with capture switched off entirely (observed — it was the only one of + // these seven that survived unregistering sf_messages). The control converging is + // what makes the absence of msg-ordering evidence rather than silence. + await EnqueueAsync(A, "msg-ordering-control"); + + await WaitUntilAsync( + () => ExistsAsync(B, "msg-ordering-control"), + "the control message to reach node B, proving the pair was replicating"); + + await WaitUntilAsync( + async () => !await ExistsAsync(B, "msg-ordering") && !await ExistsAsync(A, "msg-ordering"), + "the add-then-remove pair to settle as absent on both nodes"); + + // Absence has to persist, not just occur once: a late-arriving add would flip the row + // back and the poll above could have observed a gap it never actually converged to. + await Task.Delay(TimeSpan.FromSeconds(2)); + Assert.False(await ExistsAsync(B, "msg-ordering")); + Assert.False(await ExistsAsync(A, "msg-ordering")); + Assert.True(await ExistsAsync(B, "msg-ordering-control")); + } + + [Fact] + public async Task SameMessage_BufferedTwice_ConvergesToTheNewerState() + { + // Was: ApplyReplicatedAdd_Twice_IsIdempotent_NewestWins. Under CDC this is not a + // property the application has to implement — LWW on the primary key gives it — but + // it is still a property the pair must HAVE, so it is asserted rather than assumed. + await EnqueueAsync(A, "msg-twice"); + await WaitUntilAsync(() => ExistsAsync(B, "msg-twice"), "the first write to reach B"); + + await SetStatusAsync(A, "msg-twice", StoreAndForwardMessageStatus.InFlight, retryCount: 3); + + await WaitUntilAsync( + async () => await ReadAsync(B, "msg-twice") + is { Status: (int)StoreAndForwardMessageStatus.InFlight, RetryCount: 3 }, + "the newer state to win on node B"); + } + + [Fact] + public async Task ParkArrivingWithoutItsAdd_StillMaterialisesTheRow() + { + // Was: ApplyReplicatedPark_WhenAddWasLost_MaterializesTheParkedRow. The bespoke + // replicator needed explicit upsert semantics so a lost Add did not leave a Park with + // nothing to update. The CDC equivalent is a peer that was offline for the add and + // only ever sees the row in its parked state — it must still end up with the row, + // not skip it as an update to something it never had. + await StopPassiveAsync(); + + await EnqueueAsync(A, "msg-park-no-add"); + await SetStatusAsync(A, "msg-park-no-add", StoreAndForwardMessageStatus.Parked, retryCount: 50); + + await RestartPairAsync(); + + await WaitUntilAsync( + () => HasStatusAsync(B, "msg-park-no-add", StoreAndForwardMessageStatus.Parked), + "node B to materialise a row it only ever saw in its parked state"); + } +} diff --git a/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/RecoveryDrillTests.cs b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/RecoveryDrillTests.cs index f1a3eeb3..b9b4cb7c 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/RecoveryDrillTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/RecoveryDrillTests.cs @@ -3,6 +3,7 @@ using Microsoft.Extensions.Logging.Abstractions; using ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment; using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; using ZB.MOM.WW.ScadaBridge.StoreAndForward; +using ZB.MOM.WW.ScadaBridge.TestSupport; namespace ZB.MOM.WW.ScadaBridge.IntegrationTests; @@ -59,11 +60,15 @@ public class RecoveryDrillTests // Scenario: Communication drops while deploying system-wide artifacts. // The deployment command is buffered by S&F and retried when connection restores. var dbPath = Path.Combine(Path.GetTempPath(), $"sf_commdrop_{Guid.NewGuid():N}.db"); - var connStr = $"Data Source={dbPath}"; + + // The store takes an ILocalDb (LocalDb has no in-memory mode), so the buffer + // lives in a real temp file for the duration of the drill. + TestLocalDb? localDb = null; try { - var storage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + localDb = TestLocalDb.Create(dbPath); + var storage = new StoreAndForwardStorage(localDb.Db, NullLogger.Instance); await storage.InitializeAsync(); var options = new StoreAndForwardOptions @@ -102,8 +107,9 @@ public class RecoveryDrillTests } finally { - if (File.Exists(dbPath)) - File.Delete(dbPath); + // Dispose before deleting — the master connection anchors the WAL sidecars. + localDb?.Dispose(); + TestLocalDb.DeleteFiles(dbPath); } } @@ -115,12 +121,17 @@ public class RecoveryDrillTests // On startup, the Deployment Manager Actor reads configs from SQLite and // recreates Instance Actors. var dbPath = Path.Combine(Path.GetTempPath(), $"sf_restart_{Guid.NewGuid():N}.db"); - var connStr = $"Data Source={dbPath}"; + + // The restarted "process" opens its own local database over the SAME file — that + // is what this drill verifies survives. + TestLocalDb? preRestartDb = null; + TestLocalDb? restartedDb = null; try { // Pre-restart: S&F messages in buffer - var storage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + preRestartDb = TestLocalDb.Create(dbPath); + var storage = new StoreAndForwardStorage(preRestartDb.Db, NullLogger.Instance); await storage.InitializeAsync(); for (var i = 0; i < 3; i++) @@ -140,7 +151,8 @@ public class RecoveryDrillTests } // Post-restart: new storage instance reads same DB - var restartedStorage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + restartedDb = TestLocalDb.Create(dbPath); + var restartedStorage = new StoreAndForwardStorage(restartedDb.Db, NullLogger.Instance); await restartedStorage.InitializeAsync(); var pending = await restartedStorage.GetMessagesForRetryAsync(); @@ -153,8 +165,10 @@ public class RecoveryDrillTests } finally { - if (File.Exists(dbPath)) - File.Delete(dbPath); + // Dispose before deleting — the master connections anchor the WAL sidecars. + restartedDb?.Dispose(); + preRestartDb?.Dispose(); + TestLocalDb.DeleteFiles(dbPath); } } diff --git a/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/SiteFailoverTests.cs b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/SiteFailoverTests.cs index 0eb2d110..762f64b1 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/SiteFailoverTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/SiteFailoverTests.cs @@ -4,6 +4,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health; using ZB.MOM.WW.ScadaBridge.Commons.Messages.Streaming; using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; using ZB.MOM.WW.ScadaBridge.StoreAndForward; +using ZB.MOM.WW.ScadaBridge.TestSupport; namespace ZB.MOM.WW.ScadaBridge.IntegrationTests; @@ -21,12 +22,18 @@ public class SiteFailoverTests // Simulates site failover: messages buffered in SQLite survive process restart. // The standby node picks up the same SQLite file and retries pending messages. var dbPath = Path.Combine(Path.GetTempPath(), $"sf_failover_{Guid.NewGuid():N}.db"); - var connStr = $"Data Source={dbPath}"; + + // The store takes an ILocalDb; each "node" opens its own local database over the + // SAME file, which is how this test models the standby picking up the primary's + // buffer after failover. + TestLocalDb? primaryDb = null; + TestLocalDb? standbyDb = null; try { // Phase 1: Buffer messages on "primary" node - var storage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + primaryDb = TestLocalDb.Create(dbPath); + var storage = new StoreAndForwardStorage(primaryDb.Db, NullLogger.Instance); await storage.InitializeAsync(); var message = new StoreAndForwardMessage @@ -46,7 +53,8 @@ public class SiteFailoverTests await storage.EnqueueAsync(message); // Phase 2: "Standby" node opens the same database (simulating failover) - var standbyStorage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + standbyDb = TestLocalDb.Create(dbPath); + var standbyStorage = new StoreAndForwardStorage(standbyDb.Db, NullLogger.Instance); await standbyStorage.InitializeAsync(); var pending = await standbyStorage.GetMessagesForRetryAsync(); @@ -57,8 +65,10 @@ public class SiteFailoverTests } finally { - if (File.Exists(dbPath)) - File.Delete(dbPath); + // Dispose before deleting — the master connections anchor the WAL sidecars. + standbyDb?.Dispose(); + primaryDb?.Dispose(); + TestLocalDb.DeleteFiles(dbPath); } } @@ -67,11 +77,14 @@ public class SiteFailoverTests public async Task StoreAndForward_ParkedMessages_SurviveFailover() { var dbPath = Path.Combine(Path.GetTempPath(), $"sf_parked_{Guid.NewGuid():N}.db"); - var connStr = $"Data Source={dbPath}"; + + TestLocalDb? primaryDb = null; + TestLocalDb? standbyDb = null; try { - var storage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + primaryDb = TestLocalDb.Create(dbPath); + var storage = new StoreAndForwardStorage(primaryDb.Db, NullLogger.Instance); await storage.InitializeAsync(); var parkedMsg = new StoreAndForwardMessage @@ -93,7 +106,8 @@ public class SiteFailoverTests await storage.EnqueueAsync(parkedMsg); // Standby opens same DB - var standbyStorage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + standbyDb = TestLocalDb.Create(dbPath); + var standbyStorage = new StoreAndForwardStorage(standbyDb.Db, NullLogger.Instance); await standbyStorage.InitializeAsync(); var (parked, count) = await standbyStorage.GetParkedMessagesAsync(); @@ -102,8 +116,10 @@ public class SiteFailoverTests } finally { - if (File.Exists(dbPath)) - File.Delete(dbPath); + // Dispose before deleting — the master connections anchor the WAL sidecars. + standbyDb?.Dispose(); + primaryDb?.Dispose(); + TestLocalDb.DeleteFiles(dbPath); } } @@ -160,11 +176,14 @@ public class SiteFailoverTests public async Task StoreAndForward_BufferDepth_ReportedAfterFailover() { var dbPath = Path.Combine(Path.GetTempPath(), $"sf_depth_{Guid.NewGuid():N}.db"); - var connStr = $"Data Source={dbPath}"; + + TestLocalDb? primaryDb = null; + TestLocalDb? standbyDb = null; try { - var storage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + primaryDb = TestLocalDb.Create(dbPath); + var storage = new StoreAndForwardStorage(primaryDb.Db, NullLogger.Instance); await storage.InitializeAsync(); // Enqueue messages in different categories @@ -199,7 +218,8 @@ public class SiteFailoverTests } // After failover, standby reads buffer depths - var standbyStorage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + standbyDb = TestLocalDb.Create(dbPath); + var standbyStorage = new StoreAndForwardStorage(standbyDb.Db, NullLogger.Instance); await standbyStorage.InitializeAsync(); var depths = await standbyStorage.GetBufferDepthByCategoryAsync(); @@ -208,8 +228,10 @@ public class SiteFailoverTests } finally { - if (File.Exists(dbPath)) - File.Delete(dbPath); + // Dispose before deleting — the master connections anchor the WAL sidecars. + standbyDb?.Dispose(); + primaryDb?.Dispose(); + TestLocalDb.DeleteFiles(dbPath); } } } diff --git a/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/ZB.MOM.WW.ScadaBridge.IntegrationTests.csproj b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/ZB.MOM.WW.ScadaBridge.IntegrationTests.csproj index 9eddb8c5..5090cd4d 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/ZB.MOM.WW.ScadaBridge.IntegrationTests.csproj +++ b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/ZB.MOM.WW.ScadaBridge.IntegrationTests.csproj @@ -42,6 +42,7 @@ - + + diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/EventLogCoverageTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/EventLogCoverageTests.cs index da267200..b2a6e9b1 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/EventLogCoverageTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/EventLogCoverageTests.cs @@ -11,20 +11,27 @@ namespace ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests; public class EventLogCoverageTests : IDisposable { private readonly string _dbPath; + private readonly TestLocalDb _localDb; public EventLogCoverageTests() { _dbPath = Path.Combine(Path.GetTempPath(), $"test_coverage_{Guid.NewGuid()}.db"); + _localDb = TestLocalDb.Create(_dbPath); } public void Dispose() { - if (File.Exists(_dbPath)) File.Delete(_dbPath); + _localDb.Dispose(); + TestLocalDb.DeleteFiles(_dbPath); + GC.SuppressFinalize(this); } + // Each call gets its own connection from the one shared local database, so the + // loggers these tests dispose mid-test do not tear down the database itself. private SiteEventLogger NewLogger() => new( Options.Create(new SiteEventLogOptions { DatabasePath = _dbPath }), - NullLogger.Instance); + NullLogger.Instance, + _localDb.Db); private static EventLogQueryRequest MakeRequest() => new( CorrelationId: "corr-err", diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/EventLogPurgeServiceTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/EventLogPurgeServiceTests.cs index dbd7522a..22240faa 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/EventLogPurgeServiceTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/EventLogPurgeServiceTests.cs @@ -6,6 +6,7 @@ namespace ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests; public class EventLogPurgeServiceTests : IDisposable { private readonly SiteEventLogger _eventLogger; + private readonly TestLocalDb _localDb; private readonly string _dbPath; private readonly SiteEventLogOptions _options; @@ -27,15 +28,19 @@ public class EventLogPurgeServiceTests : IDisposable RetentionDays = 30, MaxStorageMb = 1024 }; + _localDb = TestLocalDb.Create(_dbPath); _eventLogger = new SiteEventLogger( Options.Create(_options), - NullLogger.Instance); + NullLogger.Instance, + _localDb.Db); } public void Dispose() { _eventLogger.Dispose(); - if (File.Exists(_dbPath)) File.Delete(_dbPath); + _localDb.Dispose(); + TestLocalDb.DeleteFiles(_dbPath); + GC.SuppressFinalize(this); } private EventLogPurgeService CreatePurgeService( @@ -56,9 +61,10 @@ public class EventLogPurgeServiceTests : IDisposable { using var cmd = connection.CreateCommand(); cmd.CommandText = """ - INSERT INTO site_events (timestamp, event_type, severity, source, message) - VALUES ($ts, 'script', 'Info', 'Test', 'Test message') + INSERT INTO site_events (id, timestamp, event_type, severity, source, message) + VALUES ($id, $ts, 'script', 'Info', 'Test', 'Test message') """; + cmd.Parameters.AddWithValue("$id", Guid.NewGuid().ToString("N")); cmd.Parameters.AddWithValue("$ts", timestamp.ToString("o")); cmd.ExecuteNonQuery(); }); @@ -135,6 +141,29 @@ public class EventLogPurgeServiceTests : IDisposable Assert.True(size > 0); } + /// + /// Base instant for bulk-seeded events: one day ago, evaluated once so a run is + /// self-consistent. + /// + /// Must stay INSIDE the retention window (RetentionDays, default 30). + /// applies the retention purge before + /// the storage-cap purge, so a fixed calendar date in the past would be deleted + /// wholesale as stale and the storage-cap assertions would silently test an empty + /// table rather than cap-trimming behaviour. + /// + /// + private static readonly DateTimeOffset BulkSeedStart = + DateTimeOffset.UtcNow.AddDays(-1); + + /// + /// Timestamp of the i-th bulk-seeded event. Each row is one second newer than the + /// last, so "oldest" is unambiguous — the storage-cap purge orders by timestamp, + /// and previously every bulk row shared the same UtcNow, which left the + /// oldest-first guarantee untestable. + /// + private static DateTimeOffset BulkSeedTimestamp(int index) => + BulkSeedStart.AddSeconds(index); + private void InsertBulkEvents(int count) { // Each event carries a sizeable details payload so the database grows @@ -146,24 +175,30 @@ public class EventLogPurgeServiceTests : IDisposable { using var cmd = connection.CreateCommand(); cmd.CommandText = """ - INSERT INTO site_events (timestamp, event_type, severity, source, message, details) - VALUES ($ts, 'script', 'Info', 'Test', 'Bulk event', $details) + INSERT INTO site_events (id, timestamp, event_type, severity, source, message, details) + VALUES ($id, $ts, 'script', 'Info', 'Test', 'Bulk event', $details) """; - cmd.Parameters.AddWithValue("$ts", DateTimeOffset.UtcNow.ToString("o")); + cmd.Parameters.AddWithValue("$id", Guid.NewGuid().ToString("N")); + cmd.Parameters.AddWithValue("$ts", BulkSeedTimestamp(i).ToString("o")); cmd.Parameters.AddWithValue("$details", details); cmd.ExecuteNonQuery(); } }); } - private long MinEventId() + /// + /// Oldest surviving event timestamp. Replaces the former MIN(id) probe: + /// ids are GUIDs since LocalDb Phase 1 and carry no ordering, so age is measured + /// on the timestamp column the purge itself orders by. + /// + private string MinEventTimestamp() { return _eventLogger.WithConnection(connection => { using var cmd = connection.CreateCommand(); - cmd.CommandText = "SELECT MIN(id) FROM site_events"; + cmd.CommandText = "SELECT MIN(timestamp) FROM site_events"; var result = cmd.ExecuteScalar(); - return result is long l ? l : 0; + return result as string ?? ""; }); } @@ -204,9 +239,15 @@ public class EventLogPurgeServiceTests : IDisposable [Fact] public void PurgeByStorageCap_RemovesOldestEventsFirst() { - // Regression test for SiteEventLogging-002: only the oldest events - // (lowest ids) should be removed when trimming to the cap. - InsertBulkEvents(3000); + // Regression test for SiteEventLogging-002: only the oldest events should be + // removed when trimming to the cap. + // + // "Oldest" used to mean "lowest autoincrement id". Since LocalDb Phase 1 the id + // is a GUID with no ordering, so both the purge and this test key on timestamp. + // Ordering by id here would delete an arbitrary batch, which is precisely the + // regression this test now guards. + const int eventCount = 3000; + InsertBulkEvents(eventCount); var purge = CreatePurgeService(); var totalSize = purge.GetDatabaseSizeBytes(); @@ -218,20 +259,23 @@ public class EventLogPurgeServiceTests : IDisposable MaxStorageMb = (int)Math.Max(1, (totalSize / 2) / (1024 * 1024)) }; - var minIdBefore = MinEventId(); + var oldestBefore = MinEventTimestamp(); var cappedPurge = CreatePurgeService(capOptions); cappedPurge.RunPurge(); - var minIdAfter = MinEventId(); + var oldestAfter = MinEventTimestamp(); - // The surviving rows must be the newest ones — minimum id has advanced. - Assert.True(minIdAfter > minIdBefore, - "Oldest events (lowest ids) must be purged first."); + // The surviving rows must be the newest ones — the oldest timestamp advanced. + Assert.True( + string.CompareOrdinal(oldestAfter, oldestBefore) > 0, + $"Oldest events must be purged first; oldest went from '{oldestBefore}' to '{oldestAfter}'."); - // The newest event (highest id) must still be present. + // The newest event must still be present. + var newestTimestamp = BulkSeedTimestamp(eventCount - 1).ToString("o"); var newestPresent = _eventLogger.WithConnection(connection => { using var cmd = connection.CreateCommand(); - cmd.CommandText = "SELECT COUNT(*) FROM site_events WHERE id = 3000"; + cmd.CommandText = "SELECT COUNT(*) FROM site_events WHERE timestamp = $ts"; + cmd.Parameters.AddWithValue("$ts", newestTimestamp); return (long)cmd.ExecuteScalar()!; }); Assert.Equal(1L, newestPresent); diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/EventLogQueryServiceTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/EventLogQueryServiceTests.cs index b3622d99..a7b55f96 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/EventLogQueryServiceTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/EventLogQueryServiceTests.cs @@ -8,6 +8,7 @@ public class EventLogQueryServiceTests : IDisposable { private readonly SiteEventLogger _eventLogger; private readonly EventLogQueryService _queryService; + private readonly TestLocalDb _localDb; private readonly string _dbPath; public EventLogQueryServiceTests() @@ -18,7 +19,8 @@ public class EventLogQueryServiceTests : IDisposable DatabasePath = _dbPath, QueryPageSize = 500 }); - _eventLogger = new SiteEventLogger(options, NullLogger.Instance); + _localDb = TestLocalDb.Create(_dbPath); + _eventLogger = new SiteEventLogger(options, NullLogger.Instance, _localDb.Db); _queryService = new EventLogQueryService( _eventLogger, options, @@ -28,7 +30,9 @@ public class EventLogQueryServiceTests : IDisposable public void Dispose() { _eventLogger.Dispose(); - if (File.Exists(_dbPath)) File.Delete(_dbPath); + _localDb.Dispose(); + TestLocalDb.DeleteFiles(_dbPath); + GC.SuppressFinalize(this); } private async Task SeedEvents() @@ -45,7 +49,7 @@ public class EventLogQueryServiceTests : IDisposable string? severity = null, string? instanceId = null, string? keyword = null, - long? continuationToken = null, + string? continuationToken = null, int pageSize = 500, DateTimeOffset? from = null, DateTimeOffset? to = null) => @@ -281,7 +285,7 @@ public class EventLogQueryServiceTests : IDisposable Assert.Single(response.Entries); var entry = response.Entries[0]; - Assert.True(entry.Id > 0); + Assert.False(string.IsNullOrWhiteSpace(entry.Id)); Assert.Equal("script", entry.EventType); Assert.Equal("Error", entry.Severity); Assert.Equal("inst-1", entry.InstanceId); @@ -340,9 +344,10 @@ public class EventLogQueryServiceTests : IDisposable { using var cmd = connection.CreateCommand(); cmd.CommandText = """ - INSERT INTO site_events (timestamp, event_type, severity, instance_id, source, message) - VALUES ($ts, $et, $sev, $iid, $src, $msg) + INSERT INTO site_events (id, timestamp, event_type, severity, instance_id, source, message) + VALUES ($id, $ts, $et, $sev, $iid, $src, $msg) """; + cmd.Parameters.AddWithValue("$id", Guid.NewGuid().ToString("N")); cmd.Parameters.AddWithValue("$ts", timestamp.ToString("o")); cmd.Parameters.AddWithValue("$et", eventType); cmd.Parameters.AddWithValue("$sev", severity); @@ -366,7 +371,8 @@ public class EventLogQueryServiceTests : IDisposable QueryPageSize = 500, MaxQueryPageSize = 3, }); - using var logger = new SiteEventLogger(options, NullLogger.Instance); + using var localDb = TestLocalDb.Create(dbPath); + using var logger = new SiteEventLogger(options, NullLogger.Instance, localDb.Db); var query = new EventLogQueryService(logger, options, NullLogger.Instance); try @@ -390,7 +396,8 @@ public class EventLogQueryServiceTests : IDisposable finally { logger.Dispose(); - if (File.Exists(dbPath)) File.Delete(dbPath); + localDb.Dispose(); + TestLocalDb.DeleteFiles(dbPath); } } } diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/SchemaIndexTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/SchemaIndexTests.cs index 0038ad4c..7f70a4cb 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/SchemaIndexTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/SchemaIndexTests.cs @@ -13,12 +13,14 @@ public class SchemaIndexTests : IDisposable private readonly SiteEventLogger _logger; private readonly SqliteConnection _verifyConnection; private readonly string _dbPath; + private readonly TestLocalDb _localDb; public SchemaIndexTests() { _dbPath = Path.Combine(Path.GetTempPath(), $"test_index_{Guid.NewGuid()}.db"); var options = Options.Create(new SiteEventLogOptions { DatabasePath = _dbPath }); - _logger = new SiteEventLogger(options, NullLogger.Instance); + _localDb = TestLocalDb.Create(_dbPath); + _logger = new SiteEventLogger(options, NullLogger.Instance, _localDb.Db); _verifyConnection = new SqliteConnection($"Data Source={_dbPath}"); _verifyConnection.Open(); @@ -28,7 +30,8 @@ public class SchemaIndexTests : IDisposable { _verifyConnection.Dispose(); _logger.Dispose(); - if (File.Exists(_dbPath)) File.Delete(_dbPath); + _localDb.Dispose(); + TestLocalDb.DeleteFiles(_dbPath); } [Fact] diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/ServiceWiringTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/ServiceWiringTests.cs index 12546e64..9068edfb 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/ServiceWiringTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/ServiceWiringTests.cs @@ -1,6 +1,8 @@ +using Microsoft.Extensions.Configuration; using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Options; +using ZB.MOM.WW.LocalDb; namespace ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests; @@ -23,7 +25,7 @@ public class ServiceWiringTests : IDisposable public void Dispose() { _provider?.Dispose(); - if (File.Exists(_dbPath)) File.Delete(_dbPath); + TestLocalDb.DeleteFiles(_dbPath); } [Fact] @@ -32,6 +34,15 @@ public class ServiceWiringTests : IDisposable var services = new ServiceCollection(); services.AddLogging(); services.Configure(o => o.DatabasePath = _dbPath); + // SiteEventLogger now takes ILocalDb, so the consolidated site database has to + // be in the graph for AddSiteEventLogging to resolve at all. In the host this + // comes from SiteServiceRegistration's AddZbLocalDb. + services.AddZbLocalDb(new ConfigurationBuilder() + .AddInMemoryCollection(new Dictionary + { + ["LocalDb:Path"] = _dbPath, + }) + .Build()); services.AddSiteEventLogging(); _provider = services.BuildServiceProvider(); @@ -57,8 +68,9 @@ public class ServiceWiringTests : IDisposable // concrete SiteEventLogger. Constructing them directly must compile and work. var options = Options.Create(new SiteEventLogOptions { DatabasePath = _dbPath }); using var loggerFactory = LoggerFactory.Create(_ => { }); + using var localDb = TestLocalDb.Create(_dbPath); using var recorder = new SiteEventLogger( - options, loggerFactory.CreateLogger()); + options, loggerFactory.CreateLogger(), localDb.Db); var purge = new EventLogPurgeService( recorder, options, loggerFactory.CreateLogger()); diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/SiteEventLogSchemaTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/SiteEventLogSchemaTests.cs new file mode 100644 index 00000000..daed0b52 --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/SiteEventLogSchemaTests.cs @@ -0,0 +1,133 @@ +using Microsoft.Data.Sqlite; + +namespace ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests; + +/// +/// LocalDb Phase 1 (Task 2) — the site_events DDL is extracted out of +/// so the Host's AddZbLocalDb onReady callback can +/// create it in the consolidated site database. +/// +/// The extraction also carries the one deliberate schema CHANGE of Phase 1: the primary +/// key moves from INTEGER PRIMARY KEY AUTOINCREMENT to a TEXT GUID. Under the +/// library's last-writer-wins replication, two site nodes each minting autoincrement id 1 +/// would be treated as the same row and silently overwrite one another. GUID keys make +/// the event log a pure union across the pair instead. +/// +/// +public class SiteEventLogSchemaTests +{ + private static SqliteConnection OpenConnection() + { + var conn = new SqliteConnection("Data Source=:memory:"); + conn.Open(); + return conn; + } + + private static List<(string Name, string Type, bool NotNull, bool Pk)> TableInfo( + SqliteConnection conn, string table) + { + using var cmd = conn.CreateCommand(); + cmd.CommandText = $"PRAGMA table_info('{table}')"; + using var reader = cmd.ExecuteReader(); + var cols = new List<(string, string, bool, bool)>(); + while (reader.Read()) + { + cols.Add((reader.GetString(1), reader.GetString(2), reader.GetInt32(3) == 1, + reader.GetInt32(5) > 0)); + } + + return cols; + } + + [Fact] + public void Apply_CreatesTableWithExpectedColumns() + { + using var conn = OpenConnection(); + + SiteEventLogSchema.Apply(conn); + + Assert.Equal( + ["id", "timestamp", "event_type", "severity", "instance_id", "source", "message", "details"], + TableInfo(conn, "site_events").Select(c => c.Name)); + } + + [Fact] + public void Apply_IdIsTextPrimaryKey_NotAutoincrement() + { + using var conn = OpenConnection(); + + SiteEventLogSchema.Apply(conn); + + var pk = Assert.Single(TableInfo(conn, "site_events"), c => c.Pk); + Assert.Equal("id", pk.Name); + Assert.Equal("TEXT", pk.Type); + + // AUTOINCREMENT creates the sqlite_sequence bookkeeping table. Its absence + // is the load-bearing assertion: ids must be application-minted GUIDs, not + // server-minted integers that collide across the replicating pair. + using var cmd = conn.CreateCommand(); + cmd.CommandText = + "SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = 'sqlite_sequence'"; + Assert.Equal(0L, Convert.ToInt64(cmd.ExecuteScalar())); + } + + [Fact] + public void Apply_RejectsIntegerIdInserts() + { + using var conn = OpenConnection(); + + SiteEventLogSchema.Apply(conn); + + // A TEXT PK with no AUTOINCREMENT means an INSERT omitting id is an error, + // which is what forces the writer to mint one (Task 5a). + using var cmd = conn.CreateCommand(); + cmd.CommandText = """ + INSERT INTO site_events (timestamp, event_type, severity, source, message) + VALUES ('2026-07-19T00:00:00Z', 'Test', 'Info', 'test', 'no id supplied') + """; + Assert.Throws(() => cmd.ExecuteNonQuery()); + } + + [Fact] + public void Apply_HasNoBlobColumns() + { + using var conn = OpenConnection(); + + SiteEventLogSchema.Apply(conn); + + Assert.DoesNotContain( + TableInfo(conn, "site_events"), + c => c.Type.Contains("BLOB", StringComparison.OrdinalIgnoreCase)); + } + + [Fact] + public void Apply_CreatesExpectedIndexes() + { + using var conn = OpenConnection(); + + SiteEventLogSchema.Apply(conn); + + using var cmd = conn.CreateCommand(); + cmd.CommandText = + "SELECT name FROM sqlite_master WHERE type = 'index' AND tbl_name = 'site_events' " + + "AND name NOT LIKE 'sqlite_%' ORDER BY name"; + using var reader = cmd.ExecuteReader(); + var indexes = new List(); + while (reader.Read()) indexes.Add(reader.GetString(0)); + + Assert.Equal( + ["idx_events_instance", "idx_events_severity", "idx_events_timestamp", "idx_events_type"], + indexes); + } + + [Fact] + public void Apply_IsIdempotent() + { + using var conn = OpenConnection(); + + SiteEventLogSchema.Apply(conn); + SiteEventLogSchema.Apply(conn); + + Assert.Equal(8, TableInfo(conn, "site_events").Count); + } +} diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/SiteEventLoggerAsyncTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/SiteEventLoggerAsyncTests.cs index 4d617ff6..e469a4ec 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/SiteEventLoggerAsyncTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/SiteEventLoggerAsyncTests.cs @@ -13,18 +13,21 @@ public class SiteEventLoggerAsyncTests : IDisposable { private readonly SiteEventLogger _logger; private readonly string _dbPath; + private readonly TestLocalDb _localDb; public SiteEventLoggerAsyncTests() { _dbPath = Path.Combine(Path.GetTempPath(), $"test_async_{Guid.NewGuid()}.db"); var options = Options.Create(new SiteEventLogOptions { DatabasePath = _dbPath }); - _logger = new SiteEventLogger(options, NullLogger.Instance); + _localDb = TestLocalDb.Create(_dbPath); + _logger = new SiteEventLogger(options, NullLogger.Instance, _localDb.Db); } public void Dispose() { _logger.Dispose(); - if (File.Exists(_dbPath)) File.Delete(_dbPath); + _localDb.Dispose(); + TestLocalDb.DeleteFiles(_dbPath); } [Fact] @@ -91,11 +94,10 @@ public class SiteEventLoggerAsyncTests : IDisposable // Health Monitoring can surface a logging outage. using var failingConnection = new SqliteConnection("Data Source=:memory:"); failingConnection.Open(); - var options = Options.Create(new SiteEventLogOptions - { - DatabasePath = Path.Combine(Path.GetTempPath(), $"test_fail_{Guid.NewGuid()}.db") - }); - var logger = new SiteEventLogger(options, NullLogger.Instance); + var failDbPath = Path.Combine(Path.GetTempPath(), $"test_fail_{Guid.NewGuid()}.db"); + var options = Options.Create(new SiteEventLogOptions { DatabasePath = failDbPath }); + using var failLocalDb = TestLocalDb.Create(failDbPath); + var logger = new SiteEventLogger(options, NullLogger.Instance, failLocalDb.Db); // Drop the table so every write fails with "no such table". logger.WithConnection(connection => @@ -112,5 +114,7 @@ public class SiteEventLoggerAsyncTests : IDisposable "FailedWriteCount must increment when an event write fails."); logger.Dispose(); + failLocalDb.Dispose(); + TestLocalDb.DeleteFiles(failDbPath); } } diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/SiteEventLoggerTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/SiteEventLoggerTests.cs index 76dd1059..c2921663 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/SiteEventLoggerTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/SiteEventLoggerTests.cs @@ -9,12 +9,14 @@ public class SiteEventLoggerTests : IDisposable private readonly SiteEventLogger _logger; private readonly SqliteConnection _verifyConnection; private readonly string _dbPath; + private readonly TestLocalDb _localDb; public SiteEventLoggerTests() { _dbPath = Path.Combine(Path.GetTempPath(), $"test_events_{Guid.NewGuid()}.db"); var options = Options.Create(new SiteEventLogOptions { DatabasePath = _dbPath }); - _logger = new SiteEventLogger(options, NullLogger.Instance); + _localDb = TestLocalDb.Create(_dbPath); + _logger = new SiteEventLogger(options, NullLogger.Instance, _localDb.Db); // Separate connection for verification queries _verifyConnection = new SqliteConnection($"Data Source={_dbPath}"); @@ -25,7 +27,8 @@ public class SiteEventLoggerTests : IDisposable { _verifyConnection.Dispose(); _logger.Dispose(); - if (File.Exists(_dbPath)) File.Delete(_dbPath); + _localDb.Dispose(); + TestLocalDb.DeleteFiles(_dbPath); } [Fact] @@ -110,20 +113,30 @@ public class SiteEventLoggerTests : IDisposable } [Fact] - public async Task LogEventAsync_MultipleEvents_AutoIncrementIds() + public async Task LogEventAsync_MultipleEvents_GetDistinctApplicationMintedIds() { + // Replaces LogEventAsync_MultipleEvents_AutoIncrementIds. Ids are no longer + // server-minted autoincrement integers: under LocalDb replication the site pair + // converges by last-writer-wins on the primary key, so both nodes issuing 1, 2, + // 3... for unrelated events would silently overwrite each other on sync. + // + // The invariant is therefore uniqueness, NOT monotonicity — ordering is carried + // by the timestamp column, which is what the query and purge paths sort on. await _logger.LogEventAsync("script", "Info", null, "S1", "First"); await _logger.LogEventAsync("script", "Info", null, "S2", "Second"); await _logger.LogEventAsync("script", "Info", null, "S3", "Third"); using var cmd = _verifyConnection.CreateCommand(); - cmd.CommandText = "SELECT id FROM site_events ORDER BY id"; + cmd.CommandText = "SELECT id FROM site_events ORDER BY timestamp"; using var reader = cmd.ExecuteReader(); - var ids = new List(); - while (reader.Read()) ids.Add(reader.GetInt64(0)); + var ids = new List(); + while (reader.Read()) ids.Add(reader.GetString(0)); Assert.Equal(3, ids.Count); - Assert.True(ids[0] < ids[1] && ids[1] < ids[2]); + Assert.All(ids, id => Assert.False(string.IsNullOrWhiteSpace(id))); + Assert.Equal(3, ids.Distinct(StringComparer.Ordinal).Count()); + Assert.All(ids, id => Assert.True(Guid.TryParseExact(id, "N", out _), + $"Event id '{id}' is not a GUID in 'N' format.")); } [Fact] diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/TestLocalDb.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/TestLocalDb.cs new file mode 100644 index 00000000..a5df1b4a --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/TestLocalDb.cs @@ -0,0 +1,70 @@ +using Microsoft.Extensions.Configuration; +using Microsoft.Extensions.DependencyInjection; +using ZB.MOM.WW.LocalDb; + +namespace ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests; + +/// +/// A real over a temp file, for tests that construct a +/// directly. +/// +/// +/// +/// takes rather than a connection +/// string, and there is no in-memory mode — LocalDbOptions.Path is a filesystem +/// path. A real one is used rather than a stub on purpose: connections handed out by +/// ILocalDb carry the zb_hlc_next() UDF that site_events' capture +/// triggers call, so a stub returning a bare SqliteConnection would pass these +/// tests while failing closed the moment the table is registered for replication. +/// +/// +/// No onReady callback and no RegisterReplicated: the logger's own +/// InitializeSchema creates the table, which is what keeps a directly-constructed +/// logger self-sufficient. Registration is the host's job +/// (SiteLocalDbSetup.OnReady). +/// +/// +public sealed class TestLocalDb : IDisposable +{ + private readonly ServiceProvider _provider; + + private TestLocalDb(ServiceProvider provider, ILocalDb db) + { + _provider = provider; + Db = db; + } + + /// The live local database. + public ILocalDb Db { get; } + + /// Opens a local database at . + public static TestLocalDb Create(string databasePath) + { + var configuration = new ConfigurationBuilder() + .AddInMemoryCollection(new Dictionary + { + ["LocalDb:Path"] = databasePath, + }) + .Build(); + + var provider = new ServiceCollection() + .AddZbLocalDb(configuration) + .BuildServiceProvider(); + + return new TestLocalDb(provider, provider.GetRequiredService()); + } + + /// + /// Deletes and its WAL sidecars. Call only after the + /// owning is disposed — the master connection anchors the WAL. + /// + public static void DeleteFiles(string databasePath) + { + foreach (var suffix in new[] { "", "-wal", "-shm" }) + { + try { File.Delete(databasePath + suffix); } catch { /* best effort */ } + } + } + + public void Dispose() => _provider.Dispose(); +} diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs index 2f848369..4f448aa0 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs @@ -2,6 +2,7 @@ using Akka.Actor; using Akka.TestKit.Xunit2; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Logging.Abstractions; +using ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts; using ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment; using ZB.MOM.WW.ScadaBridge.Commons.Messages.DebugView; using ZB.MOM.WW.ScadaBridge.Commons.Messages.InboundApi; @@ -14,6 +15,7 @@ using ZB.MOM.WW.ScadaBridge.SiteRuntime.Deployment; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.TestSupport; +using ZB.MOM.WW.ScadaBridge.TestSupport; using System.Text.Json; using System.Threading; @@ -28,13 +30,13 @@ public class DeploymentManagerActorTests : TestKit, IDisposable private readonly SiteStorageService _storage; private readonly ScriptCompilationService _compilationService; private readonly SharedScriptLibrary _sharedScriptLibrary; - private readonly string _dbFile; + private readonly TestLocalDb _localDb; public DeploymentManagerActorTests() { - _dbFile = Path.Combine(Path.GetTempPath(), $"dm-test-{Guid.NewGuid():N}.db"); + _localDb = TestLocalDb.CreateTemp("dm-test"); _storage = new SiteStorageService( - $"Data Source={_dbFile}", + _localDb.Db, NullLogger.Instance); _storage.InitializeAsync().GetAwaiter().GetResult(); _compilationService = new ScriptCompilationService( @@ -45,8 +47,12 @@ public class DeploymentManagerActorTests : TestKit, IDisposable void IDisposable.Dispose() { + // TestKit teardown first, so no in-flight actor can reach a disposed ILocalDb; + // then dispose the database before deleting — the master connection anchors the WAL. Shutdown(); - try { File.Delete(_dbFile); } catch { /* cleanup */ } + var path = _localDb.Path; + _localDb.Dispose(); + TestLocalDb.DeleteFiles(path); } private IActorRef CreateDeploymentManager( @@ -61,12 +67,14 @@ public class DeploymentManagerActorTests : TestKit, IDisposable null, // no stream manager in tests options, NullLogger.Instance, - null, - null, - null, - serviceProvider, - null, - configFetcher))); + // Named from here on. These trailing parameters are all optional and several + // share a type, so a positional list silently binds the wrong argument when the + // signature changes — which is exactly what removing replicationActor did. + dclManager: null, + healthCollector: null, + serviceProvider: serviceProvider, + loggerFactory: null, + configFetcher: configFetcher))); } private static string MakeConfigJson(string instanceName) @@ -295,8 +303,11 @@ public class DeploymentManagerActorTests : TestKit, IDisposable var dm = ActorOf(Props.Create(() => new DeploymentManagerActor( _storage, _compilationService, _sharedScriptLibrary, null, new SiteRuntimeOptions(), NullLogger.Instance, - null, null, null, null, null, null, - TimeSpan.FromMilliseconds(200), loader))); + // dclManager, healthCollector, serviceProvider, loggerFactory, configFetcher. + // Props.Create builds an expression tree, which rejects named arguments that + // are out of position, so the optional tail has to be padded positionally. + null, null, null, null, null, + startupLoadRetryInterval: TimeSpan.FromMilliseconds(200), configLoader: loader))); AwaitAssert(() => { @@ -878,6 +889,88 @@ public class DeploymentManagerActorTests : TestKit, IDisposable Assert.Equal("SenderPump", response.InstanceUniqueName); } + // ── LocalDb Phase 2 / Task 12: the active-node central-only purge ── + + [Fact] + public async Task ApplyingArtifacts_PurgesCentralOnlyNotificationConfig() + { + // Security cleanup. notification_lists and smtp_configurations can hold plaintext + // SMTP passwords written by a pre-2026-07-10 build, and the ACTIVE node's artifact + // apply is what clears them (DeploymentManagerActor.HandleDeployArtifacts). The + // standby used to hold a second copy of this call in SiteReplicationActor; LocalDb + // Phase 2 deleted that actor, so this is now the ONLY call site that keeps the + // tables empty. + // + // ArtifactStorageTests covers the storage method, not the actor's call to it, so + // without this test the call could be dropped and every suite would stay green. + // That is precisely the kind of silent security regression it exists to prevent — + // verified red-first by commenting out the call. + await SeedCentralOnlyRowsAsync(); + Assert.Equal(1, await RowCountAsync("notification_lists")); + Assert.Equal(1, await RowCountAsync("smtp_configurations")); + + var manager = CreateDeploymentManager(); + manager.Tell(new DeployArtifactsCommand( + DeploymentId: "dep-purge-1", + SharedScripts: null, + ExternalSystems: null, + DatabaseConnections: null, + NotificationLists: null, + DataConnections: null, + SmtpConfigurations: null, + Timestamp: DateTimeOffset.UtcNow)); + + // The apply runs on a Task.Run inside the actor, so poll rather than assert once. + await AwaitPurgedAsync(); + } + + private async Task SeedCentralOnlyRowsAsync() + { + // Seeded through the service's own (already-open) LocalDb connection — a raw + // SqliteConnection would lack the pragmas and the zb_hlc_next() UDF the site + // tables' capture triggers call. Raw SQL is the only way these rows can exist at + // all now that the site-side write paths are gone. + await using var connection = _storage.CreateConnection(); + await using var command = connection.CreateCommand(); + command.CommandText = """ + INSERT INTO notification_lists (name, recipient_emails, updated_at) + VALUES ('Ops Team', '["ops@example.com"]', @u); + INSERT INTO smtp_configurations + (name, server, port, auth_mode, from_address, username, password, oauth_config, updated_at) + VALUES ('smtp.example.com:587', 'smtp.example.com', 587, 'BasicAuth', + 'noreply@example.com', 'smtpuser', 'PLAINTEXT-SECRET', NULL, @u); + """; + command.Parameters.AddWithValue("@u", DateTimeOffset.UtcNow.ToString("O")); + await command.ExecuteNonQueryAsync(); + } + + private async Task RowCountAsync(string table) + { + await using var connection = _storage.CreateConnection(); + await using var command = connection.CreateCommand(); + command.CommandText = $"SELECT COUNT(*) FROM {table}"; + return (long)(await command.ExecuteScalarAsync())!; + } + + private async Task AwaitPurgedAsync() + { + var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(10); + while (DateTime.UtcNow < deadline) + { + if (await RowCountAsync("notification_lists") == 0 && + await RowCountAsync("smtp_configurations") == 0) + { + return; + } + + await Task.Delay(50); + } + + Assert.Fail( + "Artifact apply did not purge the central-only notification/SMTP rows. " + + "The plaintext SMTP password is still on disk."); + } + /// /// In-test fake : returns a canned config JSON /// (notify-and-fetch success) or throws a canned exception (fetch failure), and records diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerCertReconcileTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerCertReconcileTests.cs index 20dbf736..11c6a0b4 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerCertReconcileTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerCertReconcileTests.cs @@ -6,6 +6,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Messages.Management; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts; +using ZB.MOM.WW.ScadaBridge.TestSupport; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors; @@ -46,13 +47,13 @@ akka { private readonly SiteStorageService _storage; private readonly ScriptCompilationService _compilationService; private readonly SharedScriptLibrary _sharedScriptLibrary; - private readonly string _dbFile; + private readonly TestLocalDb _localDb; public DeploymentManagerCertReconcileTests() : base(ClusterConfig, "dm-cert-reconcile") { - _dbFile = Path.Combine(Path.GetTempPath(), $"dm-cert-test-{Guid.NewGuid():N}.db"); + _localDb = TestLocalDb.CreateTemp("dm-cert-test"); _storage = new SiteStorageService( - $"Data Source={_dbFile}", NullLogger.Instance); + _localDb.Db, NullLogger.Instance); _storage.InitializeAsync().GetAwaiter().GetResult(); _compilationService = new ScriptCompilationService( NullLogger.Instance); @@ -62,8 +63,12 @@ akka { void IDisposable.Dispose() { + // TestKit teardown first, so no in-flight actor can reach a disposed ILocalDb; + // then dispose the database before deleting — the master connection anchors the WAL. Shutdown(); - try { File.Delete(_dbFile); } catch { /* cleanup */ } + var path = _localDb.Path; + _localDb.Dispose(); + TestLocalDb.DeleteFiles(path); } /// Forwards every message it receives to a probe, preserving the original sender. @@ -75,8 +80,7 @@ akka { private IActorRef CreateDeploymentManager() => ActorOf(Props.Create(() => new DeploymentManagerActor( _storage, _compilationService, _sharedScriptLibrary, - null, new SiteRuntimeOptions(), NullLogger.Instance, - null, null, null, null, null, null, null, null))); + null, new SiteRuntimeOptions(), NullLogger.Instance))); [Fact] public void SiteNodeJoined_PushesLocalTrustedCertsToJoinedNode() diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerLoggerFactoryTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerLoggerFactoryTests.cs index ad74f50b..fcf6da18 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerLoggerFactoryTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerLoggerFactoryTests.cs @@ -8,6 +8,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Types.Flattening; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts; +using ZB.MOM.WW.ScadaBridge.TestSupport; using System.Text.Json; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors; @@ -23,13 +24,13 @@ public class DeploymentManagerLoggerFactoryTests : TestKit, IDisposable private readonly SiteStorageService _storage; private readonly ScriptCompilationService _compilationService; private readonly SharedScriptLibrary _sharedScriptLibrary; - private readonly string _dbFile; + private readonly TestLocalDb _localDb; public DeploymentManagerLoggerFactoryTests() { - _dbFile = Path.Combine(Path.GetTempPath(), $"dm-loggerfactory-test-{Guid.NewGuid():N}.db"); + _localDb = TestLocalDb.CreateTemp("dm-loggerfactory-test"); _storage = new SiteStorageService( - $"Data Source={_dbFile}", + _localDb.Db, NullLogger.Instance); _storage.InitializeAsync().GetAwaiter().GetResult(); _compilationService = new ScriptCompilationService( @@ -40,8 +41,12 @@ public class DeploymentManagerLoggerFactoryTests : TestKit, IDisposable void IDisposable.Dispose() { + // TestKit teardown first, so no in-flight actor can reach a disposed ILocalDb; + // then dispose the database before deleting — the master connection anchors the WAL. Shutdown(); - try { File.Delete(_dbFile); } catch { /* cleanup */ } + var path = _localDb.Path; + _localDb.Dispose(); + TestLocalDb.DeleteFiles(path); } private static string MakeConfigJson(string instanceName) @@ -98,11 +103,10 @@ public class DeploymentManagerLoggerFactoryTests : TestKit, IDisposable null, new SiteRuntimeOptions { StartupBatchSize = 100, StartupBatchDelayMs = 5 }, NullLogger.Instance, - null, - null, - null, - null, - loggerFactory))); + // dclManager, healthCollector, serviceProvider — padded positionally because + // Props.Create is an expression tree and rejects out-of-position named args. + null, null, null, + loggerFactory: loggerFactory))); // Allow async startup (load configs + staggered creation). await Task.Delay(2000); diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerMediumFindingsTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerMediumFindingsTests.cs index a2351233..ab9adb9c 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerMediumFindingsTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerMediumFindingsTests.cs @@ -1,12 +1,14 @@ using Akka.Actor; using Akka.TestKit.Xunit2; using Microsoft.Extensions.Logging.Abstractions; +using ZB.MOM.WW.LocalDb; using ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment; using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; using ZB.MOM.WW.ScadaBridge.Commons.Types.Flattening; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts; +using ZB.MOM.WW.ScadaBridge.TestSupport; using System.Text.Json; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors; @@ -20,11 +22,11 @@ public class DeploymentManagerMediumFindingsTests : TestKit, IDisposable { private readonly ScriptCompilationService _compilationService; private readonly SharedScriptLibrary _sharedScriptLibrary; - private readonly string _dbFile; + private readonly TestLocalDb _localDb; public DeploymentManagerMediumFindingsTests() { - _dbFile = Path.Combine(Path.GetTempPath(), $"dm-medium-test-{Guid.NewGuid():N}.db"); + _localDb = TestLocalDb.CreateTemp("dm-medium-test"); _compilationService = new ScriptCompilationService( NullLogger.Instance); _sharedScriptLibrary = new SharedScriptLibrary( @@ -33,12 +35,16 @@ public class DeploymentManagerMediumFindingsTests : TestKit, IDisposable void IDisposable.Dispose() { + // TestKit teardown first, so no in-flight actor can reach a disposed ILocalDb; + // then dispose the database before deleting — the master connection anchors the WAL. Shutdown(); - try { File.Delete(_dbFile); } catch { /* cleanup */ } + var path = _localDb.Path; + _localDb.Dispose(); + TestLocalDb.DeleteFiles(path); } - private SiteStorageService NewStorage(string connectionString) - => new(connectionString, NullLogger.Instance); + private SiteStorageService NewStorage(ILocalDb localDb) + => new(localDb, NullLogger.Instance); private IActorRef CreateDeploymentManager(SiteStorageService storage, IActorRef? dclManager = null) { @@ -100,22 +106,37 @@ public class DeploymentManagerMediumFindingsTests : TestKit, IDisposable [Fact] public async Task Deploy_PersistenceFailure_ReportsFailedNotSuccess() { - // A connection string pointing at an unwritable path makes every storage - // write throw, so StoreDeployedConfigAsync fails. - var badPath = Path.Combine( - Path.GetTempPath(), $"no-such-dir-{Guid.NewGuid():N}", "site.db"); - var storage = NewStorage($"Data Source={badPath}"); + // A storage over a database whose site tables were never created makes every + // storage operation throw ("no such table"), so StoreDeployedConfigAsync fails. + // + // This replaces the old "connection string pointing at an unwritable path" trick: + // the service now takes an ILocalDb rather than a connection string, and LocalDb + // opens its file eagerly in its own constructor — so an unopenable path fails + // while building the fixture, before there is a storage to hand the actor at all. + // Skipping InitializeAsync is the equivalent lever on the new seam, and fails the + // same way the old one did: reads AND writes both throw. + var uninitialized = TestLocalDb.CreateTemp("dm-medium-persist-fail"); + try + { + var storage = NewStorage(uninitialized.Db); - var actor = CreateDeploymentManager(storage); - await Task.Delay(500); // empty startup + var actor = CreateDeploymentManager(storage); + await Task.Delay(500); // empty startup - actor.Tell(new DeployInstanceCommand( - "dep-fail", "FailPump", "h1", MakeConfigJson("FailPump"), "admin", DateTimeOffset.UtcNow)); + actor.Tell(new DeployInstanceCommand( + "dep-fail", "FailPump", "h1", MakeConfigJson("FailPump"), "admin", DateTimeOffset.UtcNow)); - var response = ExpectMsg(TimeSpan.FromSeconds(10)); - Assert.Equal("FailPump", response.InstanceUniqueName); - Assert.Equal(DeploymentStatus.Failed, response.Status); - Assert.False(string.IsNullOrEmpty(response.ErrorMessage)); + var response = ExpectMsg(TimeSpan.FromSeconds(10)); + Assert.Equal("FailPump", response.InstanceUniqueName); + Assert.Equal(DeploymentStatus.Failed, response.Status); + Assert.False(string.IsNullOrEmpty(response.ErrorMessage)); + } + finally + { + var path = uninitialized.Path; + uninitialized.Dispose(); + TestLocalDb.DeleteFiles(path); + } } /// @@ -126,7 +147,7 @@ public class DeploymentManagerMediumFindingsTests : TestKit, IDisposable [Fact] public async Task Deploy_Success_ReportsSuccessAndPersistsConfig() { - var storage = NewStorage($"Data Source={_dbFile}"); + var storage = NewStorage(_localDb.Db); await storage.InitializeAsync(); var actor = CreateDeploymentManager(storage); @@ -152,7 +173,7 @@ public class DeploymentManagerMediumFindingsTests : TestKit, IDisposable [Fact] public async Task EnsureDclConnections_ConnectionConfigChanged_ReissuesCreateCommand() { - var storage = NewStorage($"Data Source={_dbFile}"); + var storage = NewStorage(_localDb.Db); await storage.InitializeAsync(); var dcl = CreateTestProbe(); @@ -191,7 +212,7 @@ public class DeploymentManagerMediumFindingsTests : TestKit, IDisposable [Fact] public async Task EnsureDclConnections_UnchangedConfig_DoesNotReissueCreateCommand() { - var storage = NewStorage($"Data Source={_dbFile}"); + var storage = NewStorage(_localDb.Db); await storage.InitializeAsync(); var dcl = CreateTestProbe(); @@ -223,7 +244,7 @@ public class DeploymentManagerMediumFindingsTests : TestKit, IDisposable [Fact] public async Task Startup_WithSharedScripts_LoadsConfigsAndStaysResponsive() { - var storage = NewStorage($"Data Source={_dbFile}"); + var storage = NewStorage(_localDb.Db); await storage.InitializeAsync(); // Several shared scripts to compile during startup. diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerRedeployTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerRedeployTests.cs index e4e90fdf..663243a3 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerRedeployTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerRedeployTests.cs @@ -11,6 +11,7 @@ using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.TestSupport; +using ZB.MOM.WW.ScadaBridge.TestSupport; using System.Text.Json; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors; @@ -25,13 +26,13 @@ public class DeploymentManagerRedeployTests : TestKit, IDisposable private readonly SiteStorageService _storage; private readonly ScriptCompilationService _compilationService; private readonly SharedScriptLibrary _sharedScriptLibrary; - private readonly string _dbFile; + private readonly TestLocalDb _localDb; public DeploymentManagerRedeployTests() { - _dbFile = Path.Combine(Path.GetTempPath(), $"dm-redeploy-test-{Guid.NewGuid():N}.db"); + _localDb = TestLocalDb.CreateTemp("dm-redeploy-test"); _storage = new SiteStorageService( - $"Data Source={_dbFile}", + _localDb.Db, NullLogger.Instance); _storage.InitializeAsync().GetAwaiter().GetResult(); _compilationService = new ScriptCompilationService( @@ -42,8 +43,12 @@ public class DeploymentManagerRedeployTests : TestKit, IDisposable void IDisposable.Dispose() { + // TestKit teardown first, so no in-flight actor can reach a disposed ILocalDb; + // then dispose the database before deleting — the master connection anchors the WAL. Shutdown(); - try { File.Delete(_dbFile); } catch { /* cleanup */ } + var path = _localDb.Path; + _localDb.Dispose(); + TestLocalDb.DeleteFiles(path); } private IActorRef CreateDeploymentManager( @@ -56,10 +61,11 @@ public class DeploymentManagerRedeployTests : TestKit, IDisposable null, new SiteRuntimeOptions(), NullLogger.Instance, + // dclManager — padded positionally because Props.Create is an expression tree + // and rejects out-of-position named args. null, - null, - healthCollector, - serviceProvider))); + healthCollector: healthCollector, + serviceProvider: serviceProvider))); } /// diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorChildAttributeRaceTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorChildAttributeRaceTests.cs index f01cde4e..437b653b 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorChildAttributeRaceTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorChildAttributeRaceTests.cs @@ -7,6 +7,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Types.Flattening; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts; +using ZB.MOM.WW.ScadaBridge.TestSupport; using System.Reflection; using System.Text.Json; @@ -39,13 +40,13 @@ public class InstanceActorChildAttributeRaceTests : TestKit, IDisposable private readonly ScriptCompilationService _compilationService; private readonly SharedScriptLibrary _sharedScriptLibrary; private readonly SiteRuntimeOptions _options; - private readonly string _dbFile; + private readonly TestLocalDb _localDb; public InstanceActorChildAttributeRaceTests() { - _dbFile = Path.Combine(Path.GetTempPath(), $"instance-race-test-{Guid.NewGuid():N}.db"); + _localDb = TestLocalDb.CreateTemp("instance-race-test"); _storage = new SiteStorageService( - $"Data Source={_dbFile}", + _localDb.Db, NullLogger.Instance); _storage.InitializeAsync().GetAwaiter().GetResult(); _compilationService = new ScriptCompilationService( @@ -61,8 +62,12 @@ public class InstanceActorChildAttributeRaceTests : TestKit, IDisposable void IDisposable.Dispose() { + // TestKit teardown first, so no in-flight actor can reach a disposed ILocalDb; + // then dispose the database before deleting — the master connection anchors the WAL. Shutdown(); - try { File.Delete(_dbFile); } catch { /* cleanup */ } + var path = _localDb.Path; + _localDb.Dispose(); + TestLocalDb.DeleteFiles(path); } private static FlattenedConfiguration BuildConfig(string instanceName) diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorChildRoutingTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorChildRoutingTests.cs index 93b4b1ed..cb80b5a4 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorChildRoutingTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorChildRoutingTests.cs @@ -5,6 +5,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Types.Flattening; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts; +using ZB.MOM.WW.ScadaBridge.TestSupport; using System.Text.Json; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors; @@ -22,13 +23,13 @@ public class InstanceActorChildRoutingTests : TestKit, IDisposable private readonly ScriptCompilationService _compilationService; private readonly SharedScriptLibrary _sharedScriptLibrary; private readonly SiteRuntimeOptions _options; - private readonly string _dbFile; + private readonly TestLocalDb _localDb; public InstanceActorChildRoutingTests() { - _dbFile = Path.Combine(Path.GetTempPath(), $"instance-routing-test-{Guid.NewGuid():N}.db"); + _localDb = TestLocalDb.CreateTemp("instance-routing-test"); _storage = new SiteStorageService( - $"Data Source={_dbFile}", + _localDb.Db, NullLogger.Instance); _storage.InitializeAsync().GetAwaiter().GetResult(); _compilationService = new ScriptCompilationService( @@ -44,8 +45,12 @@ public class InstanceActorChildRoutingTests : TestKit, IDisposable void IDisposable.Dispose() { + // TestKit teardown first, so no in-flight actor can reach a disposed ILocalDb; + // then dispose the database before deleting — the master connection anchors the WAL. Shutdown(); - try { File.Delete(_dbFile); } catch { /* cleanup */ } + var path = _localDb.Path; + _localDb.Dispose(); + TestLocalDb.DeleteFiles(path); } private static FlattenedConfiguration ScriptsAB_and_Expression(string instanceName) diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorIntegrationTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorIntegrationTests.cs index 6933cef0..e2d0af31 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorIntegrationTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorIntegrationTests.cs @@ -9,6 +9,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Types.Flattening; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts; +using ZB.MOM.WW.ScadaBridge.TestSupport; using System.Text.Json; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors; @@ -22,13 +23,13 @@ public class InstanceActorIntegrationTests : TestKit, IDisposable private readonly ScriptCompilationService _compilationService; private readonly SharedScriptLibrary _sharedScriptLibrary; private readonly SiteRuntimeOptions _options; - private readonly string _dbFile; + private readonly TestLocalDb _localDb; public InstanceActorIntegrationTests() { - _dbFile = Path.Combine(Path.GetTempPath(), $"instance-int-test-{Guid.NewGuid():N}.db"); + _localDb = TestLocalDb.CreateTemp("instance-int-test"); _storage = new SiteStorageService( - $"Data Source={_dbFile}", + _localDb.Db, NullLogger.Instance); _storage.InitializeAsync().GetAwaiter().GetResult(); _compilationService = new ScriptCompilationService( @@ -45,8 +46,12 @@ public class InstanceActorIntegrationTests : TestKit, IDisposable void IDisposable.Dispose() { + // TestKit teardown first, so no in-flight actor can reach a disposed ILocalDb; + // then dispose the database before deleting — the master connection anchors the WAL. Shutdown(); - try { File.Delete(_dbFile); } catch { /* cleanup */ } + var path = _localDb.Path; + _localDb.Dispose(); + TestLocalDb.DeleteFiles(path); } private IActorRef CreateInstanceWithScripts( diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorNativeAlarmTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorNativeAlarmTests.cs index 33b7bd37..8cfcbf41 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorNativeAlarmTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorNativeAlarmTests.cs @@ -12,6 +12,7 @@ using ZB.MOM.WW.ScadaBridge.SiteRuntime; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts; +using ZB.MOM.WW.ScadaBridge.TestSupport; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors; @@ -25,12 +26,12 @@ public class InstanceActorNativeAlarmTests : TestKit, IDisposable private readonly ScriptCompilationService _compilationService; private readonly SharedScriptLibrary _sharedScriptLibrary; private readonly SiteRuntimeOptions _options = new(); - private readonly string _dbFile; + private readonly TestLocalDb _localDb; public InstanceActorNativeAlarmTests() { - _dbFile = Path.Combine(Path.GetTempPath(), $"instance-native-{Guid.NewGuid():N}.db"); - _storage = new SiteStorageService($"Data Source={_dbFile}", NullLogger.Instance); + _localDb = TestLocalDb.CreateTemp("instance-native"); + _storage = new SiteStorageService(_localDb.Db, NullLogger.Instance); _storage.InitializeAsync().GetAwaiter().GetResult(); _compilationService = new ScriptCompilationService(NullLogger.Instance); _sharedScriptLibrary = new SharedScriptLibrary(_compilationService, NullLogger.Instance); @@ -149,7 +150,11 @@ public class InstanceActorNativeAlarmTests : TestKit, IDisposable void IDisposable.Dispose() { + // TestKit teardown first, so no in-flight actor can reach a disposed ILocalDb; + // then dispose the database before deleting — the master connection anchors the WAL. Shutdown(); - try { File.Delete(_dbFile); } catch { /* cleanup */ } + var path = _localDb.Path; + _localDb.Dispose(); + TestLocalDb.DeleteFiles(path); } } diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorSetAttributeTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorSetAttributeTests.cs index 90256837..c1540b93 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorSetAttributeTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorSetAttributeTests.cs @@ -8,6 +8,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Types.Flattening; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts; +using ZB.MOM.WW.ScadaBridge.TestSupport; using System.Text.Json; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors; @@ -23,13 +24,13 @@ public class InstanceActorSetAttributeTests : TestKit, IDisposable private readonly ScriptCompilationService _compilationService; private readonly SharedScriptLibrary _sharedScriptLibrary; private readonly SiteRuntimeOptions _options; - private readonly string _dbFile; + private readonly TestLocalDb _localDb; public InstanceActorSetAttributeTests() { - _dbFile = Path.Combine(Path.GetTempPath(), $"instance-setattr-test-{Guid.NewGuid():N}.db"); + _localDb = TestLocalDb.CreateTemp("instance-setattr-test"); _storage = new SiteStorageService( - $"Data Source={_dbFile}", + _localDb.Db, NullLogger.Instance); _storage.InitializeAsync().GetAwaiter().GetResult(); _compilationService = new ScriptCompilationService( @@ -41,8 +42,12 @@ public class InstanceActorSetAttributeTests : TestKit, IDisposable void IDisposable.Dispose() { + // TestKit teardown first, so no in-flight actor can reach a disposed ILocalDb; + // then dispose the database before deleting — the master connection anchors the WAL. Shutdown(); - try { File.Delete(_dbFile); } catch { /* cleanup */ } + var path = _localDb.Path; + _localDb.Dispose(); + TestLocalDb.DeleteFiles(path); } private IActorRef CreateInstanceActor(string instanceName, FlattenedConfiguration config, IActorRef? dclManager) diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorTests.cs index bc8f3c31..5657c1a8 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorTests.cs @@ -12,6 +12,7 @@ using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.TestSupport; +using ZB.MOM.WW.ScadaBridge.TestSupport; using System.Text.Json; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors; @@ -25,13 +26,13 @@ public class InstanceActorTests : TestKit, IDisposable private readonly ScriptCompilationService _compilationService; private readonly SharedScriptLibrary _sharedScriptLibrary; private readonly SiteRuntimeOptions _options; - private readonly string _dbFile; + private readonly TestLocalDb _localDb; public InstanceActorTests() { - _dbFile = Path.Combine(Path.GetTempPath(), $"instance-actor-test-{Guid.NewGuid():N}.db"); + _localDb = TestLocalDb.CreateTemp("instance-actor-test"); _storage = new SiteStorageService( - $"Data Source={_dbFile}", + _localDb.Db, NullLogger.Instance); _storage.InitializeAsync().GetAwaiter().GetResult(); _compilationService = new ScriptCompilationService( @@ -56,8 +57,12 @@ public class InstanceActorTests : TestKit, IDisposable void IDisposable.Dispose() { + // TestKit teardown first, so no in-flight actor can reach a disposed ILocalDb; + // then dispose the database before deleting — the master connection anchors the WAL. Shutdown(); - try { File.Delete(_dbFile); } catch { /* cleanup */ } + var path = _localDb.Path; + _localDb.Dispose(); + TestLocalDb.DeleteFiles(path); } // ── M1.6: site event log `instance_lifecycle` category ────────────────── diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorWaitForAttributeTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorWaitForAttributeTests.cs index ca13817d..102947e9 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorWaitForAttributeTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorWaitForAttributeTests.cs @@ -10,6 +10,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Types.Flattening; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts; +using ZB.MOM.WW.ScadaBridge.TestSupport; using System.Text.Json; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors; @@ -26,13 +27,13 @@ public class InstanceActorWaitForAttributeTests : TestKit, IDisposable private readonly ScriptCompilationService _compilationService; private readonly SharedScriptLibrary _sharedScriptLibrary; private readonly SiteRuntimeOptions _options; - private readonly string _dbFile; + private readonly TestLocalDb _localDb; public InstanceActorWaitForAttributeTests() { - _dbFile = Path.Combine(Path.GetTempPath(), $"instance-waitfor-test-{Guid.NewGuid():N}.db"); + _localDb = TestLocalDb.CreateTemp("instance-waitfor-test"); _storage = new SiteStorageService( - $"Data Source={_dbFile}", + _localDb.Db, NullLogger.Instance); _storage.InitializeAsync().GetAwaiter().GetResult(); _compilationService = new ScriptCompilationService( @@ -57,8 +58,12 @@ public class InstanceActorWaitForAttributeTests : TestKit, IDisposable void IDisposable.Dispose() { + // TestKit teardown first, so no in-flight actor can reach a disposed ILocalDb; + // then dispose the database before deleting — the master connection anchors the WAL. Shutdown(); - try { File.Delete(_dbFile); } catch { /* cleanup */ } + var path = _localDb.Path; + _localDb.Dispose(); + TestLocalDb.DeleteFiles(path); } // ── 1. Fast-path: attribute already at target ──────────────────────────── diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/NativeAlarmActorTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/NativeAlarmActorTests.cs index f8dd29ec..c8874722 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/NativeAlarmActorTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/NativeAlarmActorTests.cs @@ -12,6 +12,7 @@ using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Messages; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.TestSupport; +using ZB.MOM.WW.ScadaBridge.TestSupport; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors; @@ -21,14 +22,16 @@ namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors; /// public class NativeAlarmActorTests : TestKit, IDisposable { - private readonly string _dbFile; + private readonly TestLocalDb _localDb; private readonly SiteStorageService _storage; private readonly SiteRuntimeOptions _options = new(); public NativeAlarmActorTests() { - _dbFile = Path.Combine(Path.GetTempPath(), $"naa-{Guid.NewGuid():N}.db"); - _storage = new SiteStorageService($"Data Source={_dbFile}", NullLogger.Instance); + // SiteStorageService takes an ILocalDb now; LocalDb has no in-memory mode, so the + // fixture is a real temp file (deleted in Dispose, after the TestKit shutdown). + _localDb = TestLocalDb.CreateTemp("naa"); + _storage = new SiteStorageService(_localDb.Db, NullLogger.Instance); _storage.InitializeAsync().GetAwaiter().GetResult(); } @@ -442,11 +445,14 @@ public class NativeAlarmActorTests : TestKit, IDisposable void IDisposable.Dispose() { + // Shut the actor system down FIRST: in-flight alarm actors still hold the + // ILocalDb, and their coalesced flush would hit a disposed database otherwise. Shutdown(); - if (File.Exists(_dbFile)) - { - File.Delete(_dbFile); - } + // Then dispose — the master connection anchors the WAL, so the sidecars cannot + // be removed while it is open. + var path = _localDb.Path; + _localDb.Dispose(); + TestLocalDb.DeleteFiles(path); } [Fact] diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/SiteReconciliationActorTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/SiteReconciliationActorTests.cs index 621cf6b4..7fa1825d 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/SiteReconciliationActorTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/SiteReconciliationActorTests.cs @@ -6,6 +6,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Deployment; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; +using ZB.MOM.WW.ScadaBridge.TestSupport; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors; @@ -22,20 +23,28 @@ public class SiteReconciliationActorTests : TestKit, IDisposable private const string NodeId = "node-a"; private readonly SiteStorageService _storage; - private readonly string _dbFile; + private readonly TestLocalDb _localDb; public SiteReconciliationActorTests() { - _dbFile = Path.Combine(Path.GetTempPath(), $"site-reconcile-test-{Guid.NewGuid():N}.db"); + // SiteStorageService takes an ILocalDb now; LocalDb has no in-memory mode, so the + // fixture is a real temp file (deleted in Dispose, after the TestKit shutdown). + _localDb = TestLocalDb.CreateTemp("site-reconcile-test"); _storage = new SiteStorageService( - $"Data Source={_dbFile}", Microsoft.Extensions.Logging.Abstractions.NullLogger.Instance); + _localDb.Db, Microsoft.Extensions.Logging.Abstractions.NullLogger.Instance); _storage.InitializeAsync().GetAwaiter().GetResult(); } void IDisposable.Dispose() { + // Shut the actor system down FIRST: a reconcile continuation may still be + // writing through the ILocalDb, which must outlive the actors. Shutdown(); - try { File.Delete(_dbFile); } catch { /* cleanup */ } + // Then dispose — the master connection anchors the WAL, so the sidecars cannot + // be removed while it is open. + var path = _localDb.Path; + _localDb.Dispose(); + TestLocalDb.DeleteFiles(path); } private IActorRef CreateReconciliationActor( diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/SiteReplicationActorTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/SiteReplicationActorTests.cs deleted file mode 100644 index eb67ee4f..00000000 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/SiteReplicationActorTests.cs +++ /dev/null @@ -1,559 +0,0 @@ -using System.Collections.Concurrent; -using System.Diagnostics.Metrics; -using Akka.Actor; -using Akka.TestKit.Xunit2; -using Microsoft.Extensions.Logging; -using Microsoft.Extensions.Logging.Abstractions; -using ZB.MOM.WW.ScadaBridge.SiteRuntime; -using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors; -using ZB.MOM.WW.ScadaBridge.SiteRuntime.Deployment; -using ZB.MOM.WW.ScadaBridge.SiteRuntime.Messages; -using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; -using ZB.MOM.WW.ScadaBridge.StoreAndForward; -using ZB.MOM.WW.ScadaBridge.Commons.Observability; -using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; - -namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors; - -/// -/// Tests for 's notify-and-fetch config replication: -/// the active node now replicates an id-only (no inline -/// config JSON — killing the intra-site 128 KB frame trap), and the standby fetches the -/// config from central over HTTP and writes it with the older-write guard. -/// -public class SiteReplicationActorTests : TestKit, IDisposable -{ - // Cluster provider is required because SiteReplicationActor calls Cluster.Get in its ctor - // and subscribes to cluster events in PreStart. We use the in-memory TestTransport (not - // dot-netty) so no real socket is bound and no DNS lookup happens — the actor only needs - // the cluster extension to load; these tests never form a real two-node cluster. - private const string ClusterConfig = @" -akka { - actor { provider = cluster } - remote { - enabled-transports = [""akka.remote.test""] - test { - transport-class = ""Akka.Remote.Transport.TestTransport, Akka.Remote"" - applied-adapters = [] - registry-key = site-repl-test - local-address = ""test://site-repl@localhost:1"" - maximum-payload-bytes = 128000b - scheme-identifier = test - } - } - cluster { roles = [""site-test""] } - loglevel = WARNING -}"; - - private const string SiteRole = "site-test"; - - private readonly SiteStorageService _storage; - private readonly StoreAndForwardStorage _sfStorage; - private readonly ReplicationService _replicationService; - private readonly string _dbFile; - private readonly string _sfDbFile; - - public SiteReplicationActorTests() : base(ClusterConfig, "site-repl") - { - _dbFile = Path.Combine(Path.GetTempPath(), $"site-repl-test-{Guid.NewGuid():N}.db"); - _sfDbFile = Path.Combine(Path.GetTempPath(), $"site-repl-sf-{Guid.NewGuid():N}.db"); - - _storage = new SiteStorageService( - $"Data Source={_dbFile}", NullLogger.Instance); - _storage.InitializeAsync().GetAwaiter().GetResult(); - - _sfStorage = new StoreAndForwardStorage( - $"Data Source={_sfDbFile}", NullLogger.Instance); - _sfStorage.InitializeAsync().GetAwaiter().GetResult(); - - _replicationService = new ReplicationService( - new StoreAndForwardOptions(), NullLogger.Instance); - } - - void IDisposable.Dispose() - { - Shutdown(); - try { File.Delete(_dbFile); } catch { /* cleanup */ } - try { File.Delete(_sfDbFile); } catch { /* cleanup */ } - } - - private IActorRef CreateReplicationActor(IDeploymentConfigFetcher fetcher) => - ActorOf(Props.Create(() => new SiteReplicationActor( - _storage, _sfStorage, _replicationService, SiteRole, - NullLogger.Instance, fetcher))); - - private IActorRef CreateReplicationActor( - IDeploymentConfigFetcher fetcher, SiteRuntimeOptions options, TimeSpan retryDelay) => - ActorOf(Props.Create(() => new SiteReplicationActor( - _storage, _sfStorage, _replicationService, SiteRole, - NullLogger.Instance, fetcher, null, options, retryDelay))); - - [Fact] - public async Task ReplicatedFetch_RetriesUpToConfigFetchRetryCount() - { - // The first two fetches fail transiently; the third succeeds. With - // ConfigFetchRetryCount = 3 the standby must retry to the third attempt and - // then guarded-write the fetched config (a short retry delay keeps the test fast). - var attempts = 0; - var fetcher = new FakeConfigFetcher(_ => - Interlocked.Increment(ref attempts) < 3 - ? Task.FromException(new InvalidOperationException("central hiccup")) - : Task.FromResult("{\"instanceUniqueName\":\"RetryPump\"}")); - var actor = CreateReplicationActor( - fetcher, new SiteRuntimeOptions { ConfigFetchRetryCount = 3 }, - TimeSpan.FromMilliseconds(50)); - - actor.Tell(new ApplyConfigDeploy( - "RetryPump", "dep-r1", "sha256:r1", true, - "http://central:9000", "tok-r1")); - - await AwaitAssertAsync(async () => - { - Assert.Equal(3, Volatile.Read(ref attempts)); - var configs = await _storage.GetAllDeployedConfigsAsync(); - Assert.Single(configs, c => c.InstanceUniqueName == "RetryPump"); - }, TimeSpan.FromSeconds(10)); - } - - [Fact] - public async Task ApplyConfigDeploy_StandbyFetchesConfigAndGuardedWrites() - { - // The standby receives an id-only ApplyConfigDeploy; it fetches the config from - // central using the message's coords, then guarded-writes the fetched config. - const string configJson = "{\"instanceUniqueName\":\"Pump1\"}"; - var fetcher = new FakeConfigFetcher(_ => Task.FromResult(configJson)); - var actor = CreateReplicationActor(fetcher); - - actor.Tell(new ApplyConfigDeploy( - "Pump1", "dep-100", "sha256:abc", true, - "http://central:9000", "tok-xyz")); - - // The continuation runs off-thread; await the guarded write landing. - await AwaitAssertAsync(async () => - { - var configs = await _storage.GetAllDeployedConfigsAsync(); - var row = Assert.Single(configs, c => c.InstanceUniqueName == "Pump1"); - Assert.Equal(configJson, row.ConfigJson); - Assert.Equal("dep-100", row.DeploymentId); - Assert.Equal("sha256:abc", row.RevisionHash); - Assert.True(row.IsEnabled); - }, TimeSpan.FromSeconds(5)); - - // The fetcher was called with the message's coords. - var call = Assert.Single(fetcher.Calls); - Assert.Equal("http://central:9000", call.BaseUrl); - Assert.Equal("dep-100", call.DeploymentId); - Assert.Equal("tok-xyz", call.Token); - } - - [Fact] - public async Task ApplyConfigDeploy_Superseded404_SkipsWriteAndActorSurvives() - { - // A 404 (superseded/expired) surfaces as DeploymentConfigFetchException{IsSuperseded}. - // The standby must skip the write, observe the exception (no crash), and stay alive. - var fetcher = new FakeConfigFetcher(_ => - Task.FromException( - new DeploymentConfigFetchException("expired", isSuperseded: true))); - var actor = CreateReplicationActor(fetcher); - - actor.Tell(new ApplyConfigDeploy( - "GonePump", "dep-stale", "sha256:gone", true, - "http://central:9000", "tok-stale")); - - // The fetch was attempted... - await AwaitAssertAsync(() => - { - Assert.Single(fetcher.Calls); - return Task.CompletedTask; - }, TimeSpan.FromSeconds(5)); - - // ...the actor did not crash (no Terminated to its watcher within the window)... - Watch(actor); - ExpectNoMsg(TimeSpan.FromMilliseconds(500)); - - // ...and nothing was written for the superseded instance. - var configs = await _storage.GetAllDeployedConfigsAsync(); - Assert.DoesNotContain(configs, c => c.InstanceUniqueName == "GonePump"); - } - - [Fact] - public async Task ApplyConfigDeploy_EmptyFetchCoords_SkipsFetchAndWrite() - { - // The direct DeployInstanceCommand cross-cluster wire path was retired in Task 14. - // This tests the defensive guard: if empty coords arrive, the actor must skip quietly - // — no FetchAsync("") call, no write — rather than erroring. - var fetcher = new FakeConfigFetcher(_ => Task.FromResult("never")); - var actor = CreateReplicationActor(fetcher); - - actor.Tell(new ApplyConfigDeploy( - "NoCoordsPump", "dep-direct", "sha256:nc", true, - CentralFetchBaseUrl: "", FetchToken: "")); - - // Give any (erroneous) async continuation time to run, then prove neither happened. - Watch(actor); - ExpectNoMsg(TimeSpan.FromMilliseconds(500)); - Assert.Empty(fetcher.Calls); - var configs = await _storage.GetAllDeployedConfigsAsync(); - Assert.DoesNotContain(configs, c => c.InstanceUniqueName == "NoCoordsPump"); - } - - [Fact] - public void ReplicateConfigDeploy_MapsToIdOnlyApplyConfigDeploy_ForPeer() - { - // The outbound mapping must forward an id-only ApplyConfigDeploy carrying the fetch - // coords (and NO inline config) to the peer. - var probe = CreateTestProbe(); - var fetcher = new FakeConfigFetcher(_ => Task.FromResult("unused")); - var actor = ActorOf(Props.Create(() => new ProbeForwardingReplicationActor( - _storage, _sfStorage, _replicationService, SiteRole, - NullLogger.Instance, fetcher, probe.Ref))); - - actor.Tell(new ReplicateConfigDeploy( - "Pump2", "dep-200", "sha256:def", false, - "http://central:9000", "tok-abc")); - - var applied = probe.ExpectMsg(TimeSpan.FromSeconds(3)); - Assert.Equal("Pump2", applied.InstanceName); - Assert.Equal("dep-200", applied.DeploymentId); - Assert.Equal("sha256:def", applied.RevisionHash); - Assert.False(applied.IsEnabled); - Assert.Equal("http://central:9000", applied.CentralFetchBaseUrl); - Assert.Equal("tok-abc", applied.FetchToken); - } - - // ── Task 21: peer-join S&F buffer resync (anti-entropy) ── - - [Fact] - public void StandbyTrackingPeer_SendsResyncRequest() - { - var probe = CreateTestProbe(); - var actor = ActorOf(Props.Create(() => new ResyncTestActor( - _storage, _sfStorage, _replicationService, SiteRole, - NullLogger.Instance, probe.Ref, () => false))); - - actor.Tell(new TriggerPeerTracked()); // stands in for TryTrackPeer's MemberUp path - - probe.ExpectMsg(TimeSpan.FromSeconds(3)); - } - - [Fact] - public void ActiveTrackingPeer_DoesNotRequestResync() - { - var probe = CreateTestProbe(); - var actor = ActorOf(Props.Create(() => new ResyncTestActor( - _storage, _sfStorage, _replicationService, SiteRole, - NullLogger.Instance, probe.Ref, () => true))); - - actor.Tell(new TriggerPeerTracked()); - - probe.ExpectNoMsg(TimeSpan.FromMilliseconds(300)); // active node never requests a resync - } - - [Fact] - public async Task ActiveNode_AnswersResyncRequest_WithChunkedSnapshot() - { - // Post-R2-T5 the active node answers with byte-budgeted SfBufferSnapshotChunk(s) - // (a single small row rides one chunk) rather than the monolithic SfBufferSnapshot. - await _sfStorage.EnqueueAsync(NewSfMessage("m1")); - var probe = CreateTestProbe(); - var actor = ActorOf(Props.Create(() => new ResyncTestActor( - _storage, _sfStorage, _replicationService, SiteRole, - NullLogger.Instance, probe.Ref, () => true))); - - actor.Tell(new RequestSfBufferResync(), TestActor); - - var chunk = ExpectMsg(TimeSpan.FromSeconds(3)); - Assert.Equal(1, chunk.TotalChunks); - Assert.Equal(1, chunk.Sequence); - Assert.Single(chunk.Messages); - Assert.False(chunk.Truncated); - } - - [Fact] - public async Task StandbyNode_AppliesSnapshot_ReplacingItsBuffer() - { - await _sfStorage.EnqueueAsync(NewSfMessage("stale")); - var probe = CreateTestProbe(); - var actor = ActorOf(Props.Create(() => new ResyncTestActor( - _storage, _sfStorage, _replicationService, SiteRole, - NullLogger.Instance, probe.Ref, () => false))); - - actor.Tell(new SfBufferSnapshot(new List { NewSfMessage("fresh") }, false)); - - await AwaitAssertAsync(async () => - { - Assert.Null(await _sfStorage.GetMessageByIdAsync("stale")); - Assert.NotNull(await _sfStorage.GetMessageByIdAsync("fresh")); - }, TimeSpan.FromSeconds(5)); - } - - // ── R2 T5: chunked resync answer ── - - [Fact] - public void ChunkForRemoting_SplitsByByteBudget_PreservingOrderAndSequence() - { - var rows = Enumerable.Range(0, 10) - .Select(i => NewMessage($"m{i}", payloadJson: new string('x', 20_000))) - .ToList(); - - var chunks = SiteReplicationActor.ChunkForRemoting(rows, maxChunkBytes: 64_000, maxChunkRows: 200); - - Assert.True(chunks.Count > 1); // 10 × 20 KB cannot ride one 64 KB chunk - Assert.Equal(rows.Select(r => r.Id), chunks.SelectMany(c => c).Select(r => r.Id)); // order preserved - Assert.All(chunks, c => Assert.True( - c.Sum(r => r.PayloadJson.Length) <= 64_000 || c.Count == 1)); // budget honored (oversized row isolated) - } - - [Fact] - public void ChunkForRemoting_RowCapHonored_AndSingleOversizedRowIsolated() - { - var many = Enumerable.Range(0, 500).Select(i => NewMessage($"s{i}", payloadJson: "{}")).ToList(); - Assert.All(SiteReplicationActor.ChunkForRemoting(many, 64_000, 200), c => Assert.True(c.Count <= 200)); - - var oversized = new List - { NewMessage("big", payloadJson: new string('y', 100_000)), NewMessage("small", payloadJson: "{}") }; - var chunks = SiteReplicationActor.ChunkForRemoting(oversized, 64_000, 200); - Assert.Equal(2, chunks.Count); // the oversized row rides alone - } - - [Fact] - public async Task ActiveNode_AnswersResyncRequest_WithSequencedChunks_SharingOneResyncId() - { - for (var i = 0; i < 3; i++) - await _sfStorage.EnqueueAsync(NewMessage($"c{i}", payloadJson: new string('z', 30_000))); - var actor = CreateResyncActor(isActive: () => true); - - actor.Tell(new RequestSfBufferResync(), TestActor); - - var first = ExpectMsg(TimeSpan.FromSeconds(5)); - var rest = Enumerable.Range(1, first.TotalChunks - 1) - .Select(_ => ExpectMsg(TimeSpan.FromSeconds(5))) - .Prepend(first) - .ToList(); - - Assert.True(first.TotalChunks > 1); - Assert.All(rest, c => Assert.Equal(first.ResyncId, c.ResyncId)); - Assert.Equal(Enumerable.Range(1, first.TotalChunks), rest.Select(c => c.Sequence)); - Assert.Equal(3, rest.Sum(c => c.Messages.Count)); - } - - // ── R2 T6: standby chunk assembly + atomic apply + ack ── - - [Fact] - public async Task StandbyNode_AssemblesChunks_AppliesOnce_AndAcks() - { - await _sfStorage.EnqueueAsync(NewMessage("stale")); - var actor = CreateResyncActor(isActive: () => false); - var resyncId = "r1"; - - actor.Tell(new SfBufferSnapshotChunk(resyncId, 1, 2, - new List { NewMessage("f1") }, false), TestActor); - actor.Tell(new SfBufferSnapshotChunk(resyncId, 2, 2, - new List { NewMessage("f2") }, false), TestActor); - - var ack = ExpectMsg(TimeSpan.FromSeconds(5)); - Assert.Equal(resyncId, ack.ResyncId); - Assert.Equal(2, ack.RowCount); - await AwaitAssertAsync(async () => - { - Assert.Null(await _sfStorage.GetMessageByIdAsync("stale")); // replaced wholesale - Assert.NotNull(await _sfStorage.GetMessageByIdAsync("f1")); - Assert.NotNull(await _sfStorage.GetMessageByIdAsync("f2")); - }); - } - - [Fact] - public async Task StandbyNode_NewResyncId_DiscardsStalePartialAssembly() - { - var actor = CreateResyncActor(isActive: () => false); - actor.Tell(new SfBufferSnapshotChunk("old", 1, 2, - new List { NewMessage("orphan") }, false), TestActor); - actor.Tell(new SfBufferSnapshotChunk("new", 1, 1, - new List { NewMessage("fresh") }, false), TestActor); - - ExpectMsg(TimeSpan.FromSeconds(5)); // "new" completed - await AwaitAssertAsync(async () => - { - Assert.NotNull(await _sfStorage.GetMessageByIdAsync("fresh")); - Assert.Null(await _sfStorage.GetMessageByIdAsync("orphan")); // stale partial never applied - }); - } - - [Fact] - public void ActiveNode_IgnoresChunks_NeverAcks() - { - var actor = CreateResyncActor(isActive: () => true); - actor.Tell(new SfBufferSnapshotChunk("r", 1, 1, - new List { NewMessage("x") }, false), TestActor); - ExpectNoMsg(TimeSpan.FromMilliseconds(300)); - } - - // ── R2 T7: active-side resync ack confirmation + telemetry ── - // - // NOTE (deviation from plan): the actor logs via Microsoft ILogger (NullLogger in - // tests), NOT Akka's EventStream, so the plan's EventFilter.Warning assertions can - // never observe these warnings. We observe the two OTel counters via a MeterListener - // instead — the equivalent, and stronger, observable signal. - - [Fact] - public async Task ActiveNode_ReceivingAck_CountsResyncCompleted() - { - long completed = 0; - using var listener = ListenCounter("scadabridge.store_and_forward.resync.completed", - m => Interlocked.Add(ref completed, m)); - - await _sfStorage.EnqueueAsync(NewMessage("m1")); - var actor = CreateResyncActor(isActive: () => true, ackTimeout: TimeSpan.FromSeconds(30)); - actor.Tell(new RequestSfBufferResync(), TestActor); - var chunk = ExpectMsg(TimeSpan.FromSeconds(5)); - - actor.Tell(new SfBufferResyncAck(chunk.ResyncId, 1), TestActor); - - await AwaitAssertAsync(() => - { - Assert.True(Interlocked.Read(ref completed) >= 1); // ack recorded the completion - return Task.CompletedTask; - }, TimeSpan.FromSeconds(5)); - } - - [Fact] - public async Task ActiveNode_MissingAck_WarnsAfterAckTimeout() - { - long ackMissing = 0; - using var listener = ListenCounter("scadabridge.store_and_forward.resync.ack_missing", - m => Interlocked.Add(ref ackMissing, m)); - - await _sfStorage.EnqueueAsync(NewMessage("m1")); - var actor = CreateResyncActor(isActive: () => true, ackTimeout: TimeSpan.FromMilliseconds(200)); - actor.Tell(new RequestSfBufferResync(), TestActor); - ExpectMsg(TimeSpan.FromSeconds(5)); - // No ack is sent → the ack window expires and the resync is counted unacknowledged. - - await AwaitAssertAsync(() => - { - Assert.True(Interlocked.Read(ref ackMissing) >= 1); - return Task.CompletedTask; - }, TimeSpan.FromSeconds(5)); - } - - /// Attaches a to a single ScadaBridge counter by name, - /// forwarding each recorded increment to . - private static MeterListener ListenCounter(string instrumentName, Action onMeasurement) - { - var listener = new MeterListener(); - listener.InstrumentPublished = (inst, l) => - { - if (inst.Meter.Name == ScadaBridgeTelemetry.MeterName && inst.Name == instrumentName) - l.EnableMeasurementEvents(inst); - }; - listener.SetMeasurementEventCallback((_, m, _, _) => onMeasurement(m)); - listener.Start(); - return listener; - } - - private static StoreAndForwardMessage NewSfMessage(string id) => new() - { - Id = id, - Category = StoreAndForwardCategory.ExternalSystem, - Target = "t", - PayloadJson = "{}", - RetryCount = 0, - MaxRetries = 50, - RetryIntervalMs = 30000, - CreatedAt = DateTimeOffset.UtcNow, - Status = StoreAndForwardMessageStatus.Pending, - }; - - /// - /// Builds a resync-test message with a settable payload (additive to - /// — the chunker sizes on PayloadJson length). - /// - private static StoreAndForwardMessage NewMessage(string id, string payloadJson = "{}") => new() - { - Id = id, - Category = StoreAndForwardCategory.ExternalSystem, - Target = "t", - PayloadJson = payloadJson, - RetryCount = 0, - MaxRetries = 50, - RetryIntervalMs = 30000, - CreatedAt = DateTimeOffset.UtcNow, - Status = StoreAndForwardMessageStatus.Pending, - }; - - /// Constructs a with the given active-node check - /// (the resync chunk/ack tests Tell to and expect from ). - /// is the active-side ack window seam (T7). - private IActorRef CreateResyncActor(Func isActive, TimeSpan? ackTimeout = null) => - ActorOf(Props.Create(() => new ResyncTestActor( - _storage, _sfStorage, _replicationService, SiteRole, - NullLogger.Instance, CreateTestProbe().Ref, isActive, ackTimeout))); - - /// Test message: drives directly, - /// standing in for the MemberUp→TryTrackPeer path (a single-node TestKit cannot form a real peer). - private sealed record TriggerPeerTracked; - - /// - /// Test subclass for the resync tests: captures peer sends to a probe, injects the - /// active-node check, and exposes via a test message. - /// - private sealed class ResyncTestActor : SiteReplicationActor - { - private readonly IActorRef _peerProbe; - - public ResyncTestActor( - SiteStorageService storage, StoreAndForwardStorage sfStorage, - ReplicationService replicationService, string siteRole, - ILogger logger, IActorRef peerProbe, Func isActive, - TimeSpan? ackTimeout = null) - : base(storage, sfStorage, replicationService, siteRole, logger, - configFetcher: null, isActiveOverride: isActive, resyncAckTimeout: ackTimeout) - { - _peerProbe = peerProbe; - Receive(_ => OnPeerTracked()); - } - - protected override void SendToPeer(object message) => _peerProbe.Tell(message, Self); - } - - /// - /// Test subclass exposing the peer send: is - /// overridden to forward to a probe so the outbound mapping can be asserted without a real - /// two-node cluster (a single-node TestKit has no peer address, so the real send is dropped). - /// - private sealed class ProbeForwardingReplicationActor : SiteReplicationActor - { - private readonly IActorRef _peerProbe; - - public ProbeForwardingReplicationActor( - SiteStorageService storage, StoreAndForwardStorage sfStorage, - ReplicationService replicationService, string siteRole, - ILogger logger, IDeploymentConfigFetcher configFetcher, - IActorRef peerProbe) - : base(storage, sfStorage, replicationService, siteRole, logger, configFetcher) - => _peerProbe = peerProbe; - - protected override void SendToPeer(object message) => _peerProbe.Tell(message, Self); - } - - /// - /// In-test fake : runs a per-deploymentId behavior - /// (return config JSON or throw, as a Task — mirroring the real async HTTP fetcher) and - /// records every call's coords thread-safely (the continuation runs on a pool thread). - /// - private sealed class FakeConfigFetcher : IDeploymentConfigFetcher - { - private readonly Func> _behavior; - public ConcurrentQueue<(string BaseUrl, string DeploymentId, string Token)> Calls { get; } = new(); - - public FakeConfigFetcher(Func> behavior) => _behavior = behavior; - - public async Task FetchAsync( - string centralFetchBaseUrl, string deploymentId, string token, CancellationToken ct) - { - Calls.Enqueue((centralFetchBaseUrl, deploymentId, token)); - await Task.Yield(); - return await _behavior(deploymentId); - } - } -} diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/NegativeTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/NegativeTests.cs index 133a1feb..f10835d5 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/NegativeTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/NegativeTests.cs @@ -1,6 +1,7 @@ using Microsoft.Data.Sqlite; using Microsoft.Extensions.Logging.Abstractions; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; +using ZB.MOM.WW.ScadaBridge.TestSupport; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests; @@ -14,18 +15,27 @@ public class NegativeTests { // Per design decision: no alarm state table in site SQLite schema. // The site SQLite stores only deployed configs and static attribute overrides. - var storage = new SiteStorageService( - "Data Source=:memory:", - NullLogger.Instance); - await storage.InitializeAsync(); + // + // SiteStorageService takes an ILocalDb now, and LocalDb has no in-memory mode + // (its Path is a filesystem path), so the service is initialized over a real + // temp file instead of the "Data Source=:memory:" it used before. The manually + // built schema subset below is still a plain in-memory SqliteConnection — it is + // not a LocalDb store, just a scratch database this test asserts against. + var localDb = TestLocalDb.CreateTemp("negative-schema"); + try + { + var storage = new SiteStorageService( + localDb.Db, + NullLogger.Instance); + await storage.InitializeAsync(); - // Try querying a non-existent alarm_states table — should throw - await using var connection = new SqliteConnection("Data Source=:memory:"); - await connection.OpenAsync(); + // Try querying a non-existent alarm_states table — should throw + await using var connection = new SqliteConnection("Data Source=:memory:"); + await connection.OpenAsync(); - // Re-initialize on this connection to get the schema - await using var initCmd = connection.CreateCommand(); - initCmd.CommandText = @" + // Re-initialize on this connection to get the schema + await using var initCmd = connection.CreateCommand(); + initCmd.CommandText = @" CREATE TABLE IF NOT EXISTS deployed_configurations ( instance_unique_name TEXT PRIMARY KEY, config_json TEXT NOT NULL, @@ -41,13 +51,22 @@ public class NegativeTests updated_at TEXT NOT NULL, PRIMARY KEY (instance_unique_name, attribute_name) );"; - await initCmd.ExecuteNonQueryAsync(); + await initCmd.ExecuteNonQueryAsync(); - // Verify alarm_states does NOT exist - await using var checkCmd = connection.CreateCommand(); - checkCmd.CommandText = "SELECT name FROM sqlite_master WHERE type='table' AND name='alarm_states'"; - var result = await checkCmd.ExecuteScalarAsync(); - Assert.Null(result); + // Verify alarm_states does NOT exist + await using var checkCmd = connection.CreateCommand(); + checkCmd.CommandText = "SELECT name FROM sqlite_master WHERE type='table' AND name='alarm_states'"; + var result = await checkCmd.ExecuteScalarAsync(); + Assert.Null(result); + } + finally + { + // Dispose first — the master connection anchors the WAL, so the sidecars + // cannot be removed while it is open. + var path = localDb.Path; + localDb.Dispose(); + TestLocalDb.DeleteFiles(path); + } } [Fact] diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/ArtifactStorageTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/ArtifactStorageTests.cs index 7d3e8423..997c244b 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/ArtifactStorageTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/ArtifactStorageTests.cs @@ -1,5 +1,6 @@ using Microsoft.Extensions.Logging.Abstractions; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; +using ZB.MOM.WW.ScadaBridge.TestSupport; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Persistence; @@ -7,20 +8,24 @@ namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Persistence; /// WP-33: Local Artifact Storage tests — shared scripts, external systems, /// database connections, notification lists. /// +/// +/// Backed by a real temp-file LocalDb: takes an +/// ILocalDb rather than a connection string, and LocalDb has no in-memory mode. +/// public class ArtifactStorageTests : IAsyncLifetime, IDisposable { - private readonly string _dbFile; + private readonly TestLocalDb _localDb; private SiteStorageService _storage = null!; public ArtifactStorageTests() { - _dbFile = Path.Combine(Path.GetTempPath(), $"artifact-test-{Guid.NewGuid():N}.db"); + _localDb = TestLocalDb.CreateTemp("artifact-test"); } public async Task InitializeAsync() { _storage = new SiteStorageService( - $"Data Source={_dbFile}", + _localDb.Db, NullLogger.Instance); await _storage.InitializeAsync(); } @@ -29,7 +34,11 @@ public class ArtifactStorageTests : IAsyncLifetime, IDisposable public void Dispose() { - try { File.Delete(_dbFile); } catch { /* cleanup */ } + // Dispose first — the master connection anchors the WAL, so the sidecars + // cannot be removed while it is open. + var path = _localDb.Path; + _localDb.Dispose(); + TestLocalDb.DeleteFiles(path); } // ── Shared Script Storage ── @@ -132,8 +141,10 @@ public class ArtifactStorageTests : IAsyncLifetime, IDisposable private async Task SeedNotificationRowAsync(string name, string emailsJson) { - await using var connection = new Microsoft.Data.Sqlite.SqliteConnection($"Data Source={_dbFile}"); - await connection.OpenAsync(); + // Seeded through the service's own (already-open) LocalDb connection — a raw + // SqliteConnection would lack the pragmas and the zb_hlc_next() UDF the site + // tables' capture triggers call. + await using var connection = _storage.CreateConnection(); await using var command = connection.CreateCommand(); command.CommandText = "INSERT INTO notification_lists (name, recipient_emails, updated_at) VALUES (@n, @e, @u)"; @@ -145,8 +156,7 @@ public class ArtifactStorageTests : IAsyncLifetime, IDisposable private async Task SeedSmtpRowAsync(string name, string password) { - await using var connection = new Microsoft.Data.Sqlite.SqliteConnection($"Data Source={_dbFile}"); - await connection.OpenAsync(); + await using var connection = _storage.CreateConnection(); await using var command = connection.CreateCommand(); command.CommandText = @"INSERT INTO smtp_configurations (name, server, port, auth_mode, from_address, username, password, oauth_config, updated_at) @@ -159,8 +169,7 @@ public class ArtifactStorageTests : IAsyncLifetime, IDisposable private async Task RowCountAsync(string table) { - await using var connection = new Microsoft.Data.Sqlite.SqliteConnection($"Data Source={_dbFile}"); - await connection.OpenAsync(); + await using var connection = _storage.CreateConnection(); await using var command = connection.CreateCommand(); command.CommandText = $"SELECT COUNT(*) FROM {table}"; return (long)(await command.ExecuteScalarAsync())!; diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/NativeAlarmStateStoreTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/NativeAlarmStateStoreTests.cs index 4063ab37..9a38bd45 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/NativeAlarmStateStoreTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/NativeAlarmStateStoreTests.cs @@ -1,5 +1,6 @@ using Microsoft.Extensions.Logging.Abstractions; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; +using ZB.MOM.WW.ScadaBridge.TestSupport; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Persistence; @@ -7,19 +8,23 @@ namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Persistence; /// Task 14: site-local SQLite native_alarm_state store — mirrored native alarm /// condition snapshots keyed by (instance, source canonical name, source reference). /// +/// +/// Backed by a real temp-file LocalDb: takes an +/// ILocalDb rather than a connection string, and LocalDb has no in-memory mode. +/// public class NativeAlarmStateStoreTests : IAsyncLifetime, IDisposable { - private readonly string _dbFile; + private readonly TestLocalDb _localDb; private SiteStorageService _storage = null!; public NativeAlarmStateStoreTests() { - _dbFile = Path.Combine(Path.GetTempPath(), $"nas-{Guid.NewGuid():N}.db"); + _localDb = TestLocalDb.CreateTemp("nas"); } public async Task InitializeAsync() { - _storage = new SiteStorageService($"Data Source={_dbFile}", NullLogger.Instance); + _storage = new SiteStorageService(_localDb.Db, NullLogger.Instance); await _storage.InitializeAsync(); } @@ -93,9 +98,10 @@ public class NativeAlarmStateStoreTests : IAsyncLifetime, IDisposable public void Dispose() { - if (File.Exists(_dbFile)) - { - File.Delete(_dbFile); - } + // Dispose first — the master connection anchors the WAL, so the sidecars + // cannot be removed while it is open. + var path = _localDb.Path; + _localDb.Dispose(); + TestLocalDb.DeleteFiles(path); } } diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/SiteStorageSchemaTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/SiteStorageSchemaTests.cs new file mode 100644 index 00000000..83b49ddf --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/SiteStorageSchemaTests.cs @@ -0,0 +1,128 @@ +using Microsoft.Data.Sqlite; +using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; + +namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Persistence; + +/// +/// Pins as the single owner of the site config DDL. +/// The Host applies this to a LocalDb-managed connection before +/// RegisterReplicated installs the capture triggers, so the shape it produces +/// is the shape that replicates. +/// +public class SiteStorageSchemaTests +{ + [Fact] + public void Apply_IsIdempotent_AndCreatesEveryTable() + { + var path = Path.Combine(Path.GetTempPath(), $"site-schema-{Guid.NewGuid():N}.db"); + try + { + using var connection = new SqliteConnection($"Data Source={path}"); + connection.Open(); + + SiteStorageSchema.Apply(connection); + SiteStorageSchema.Apply(connection); // second run must not throw + + // Every table is named explicitly on purpose — a loop asserting "9 tables" + // would still pass if one were renamed. + foreach (var table in new[] + { + "deployed_configurations", "static_attribute_overrides", "shared_scripts", + "external_systems", "database_connections", "notification_lists", + "data_connection_definitions", "smtp_configurations", "native_alarm_state", + }) + { + Assert.True(TableExists(connection, table), $"missing table: {table}"); + } + } + finally + { + SqliteConnection.ClearAllPools(); + File.Delete(path); + } + } + + /// + /// The migration columns exist to upgrade a database created by an older build in + /// place. Asserting only against a freshly-created table would pass even if every + /// ALTER were deleted, because CREATE TABLE already lists them — so + /// this starts from the pre-migration shape and proves the migration path runs. + /// + [Fact] + public void Apply_AddsMigrationColumns_ToALegacyDatabase() + { + var path = Path.Combine(Path.GetTempPath(), $"site-schema-legacy-{Guid.NewGuid():N}.db"); + try + { + using var connection = new SqliteConnection($"Data Source={path}"); + connection.Open(); + + // The shapes as they existed before backup_configuration / + // failover_retry_count / metadata_json / timeout_seconds were added. + using (var legacy = connection.CreateCommand()) + { + legacy.CommandText = """ + CREATE TABLE data_connection_definitions ( + name TEXT PRIMARY KEY, + protocol TEXT NOT NULL, + configuration TEXT, + updated_at TEXT NOT NULL + ); + CREATE TABLE native_alarm_state ( + instance_unique_name TEXT NOT NULL, + source_canonical_name TEXT NOT NULL, + source_reference TEXT NOT NULL, + condition_json TEXT NOT NULL, + last_transition_at TEXT NOT NULL, + PRIMARY KEY (instance_unique_name, source_canonical_name, source_reference) + ); + CREATE TABLE external_systems ( + name TEXT PRIMARY KEY, + endpoint_url TEXT NOT NULL, + auth_type TEXT NOT NULL, + auth_configuration TEXT, + method_definitions TEXT, + updated_at TEXT NOT NULL + ); + INSERT INTO external_systems (name, endpoint_url, auth_type, updated_at) + VALUES ('legacy-system', 'http://example.invalid', 'None', '2026-01-01T00:00:00Z'); + """; + legacy.ExecuteNonQuery(); + } + + SiteStorageSchema.Apply(connection); + + Assert.True(ColumnExists(connection, "data_connection_definitions", "backup_configuration")); + Assert.True(ColumnExists(connection, "data_connection_definitions", "failover_retry_count")); + Assert.True(ColumnExists(connection, "native_alarm_state", "metadata_json")); + Assert.True(ColumnExists(connection, "external_systems", "timeout_seconds")); + + // The pre-existing row survives the upgrade and reads back the NOT NULL + // column's default rather than failing. + using var check = connection.CreateCommand(); + check.CommandText = "SELECT timeout_seconds FROM external_systems WHERE name = 'legacy-system'"; + Assert.Equal(0, Convert.ToInt32(check.ExecuteScalar())); + } + finally + { + SqliteConnection.ClearAllPools(); + File.Delete(path); + } + } + + private static bool TableExists(SqliteConnection connection, string table) + { + using var cmd = connection.CreateCommand(); + cmd.CommandText = "SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = $name"; + cmd.Parameters.AddWithValue("$name", table); + return Convert.ToInt32(cmd.ExecuteScalar()) > 0; + } + + private static bool ColumnExists(SqliteConnection connection, string table, string column) + { + using var cmd = connection.CreateCommand(); + cmd.CommandText = $"SELECT COUNT(*) FROM pragma_table_info('{table}') WHERE name = $name"; + cmd.Parameters.AddWithValue("$name", column); + return Convert.ToInt32(cmd.ExecuteScalar()) > 0; + } +} diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/SiteStorageServiceTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/SiteStorageServiceTests.cs index 549a73ad..77dfcb78 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/SiteStorageServiceTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/SiteStorageServiceTests.cs @@ -1,7 +1,7 @@ -using Microsoft.Data.Sqlite; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Logging.Abstractions; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; +using ZB.MOM.WW.ScadaBridge.TestSupport; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Persistence; @@ -9,20 +9,26 @@ namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Persistence; /// Tests for SiteStorageService using file-based SQLite (temp files). /// Validates the schema, CRUD operations, and constraint behavior. /// +/// +/// The service now takes an ILocalDb rather than a connection string, so the fixture +/// is a real temp-file LocalDb. It stays a file (never in-memory): LocalDb has no in-memory +/// mode, and the connections it hands out carry the pragmas and the zb_hlc_next() UDF +/// the site tables' capture triggers depend on. +/// public class SiteStorageServiceTests : IAsyncLifetime, IDisposable { - private readonly string _dbFile; + private readonly TestLocalDb _localDb; private SiteStorageService _storage = null!; public SiteStorageServiceTests() { - _dbFile = Path.Combine(Path.GetTempPath(), $"site-storage-test-{Guid.NewGuid():N}.db"); + _localDb = TestLocalDb.CreateTemp("site-storage-test"); } public async Task InitializeAsync() { _storage = new SiteStorageService( - $"Data Source={_dbFile}", + _localDb.Db, NullLogger.Instance); await _storage.InitializeAsync(); } @@ -31,7 +37,11 @@ public class SiteStorageServiceTests : IAsyncLifetime, IDisposable public void Dispose() { - try { File.Delete(_dbFile); } catch { /* cleanup */ } + // Dispose first — the master connection anchors the WAL, so the sidecars + // cannot be removed while it is open. + var path = _localDb.Path; + _localDb.Dispose(); + TestLocalDb.DeleteFiles(path); } [Fact] @@ -45,10 +55,17 @@ public class SiteStorageServiceTests : IAsyncLifetime, IDisposable [Fact] public async Task Initialize_EnablesWalJournalMode() { - // WAL is set once at InitializeAsync (persistent, database-level). A file-backed DB - // is required — WAL is not available for :memory: databases. - await using var conn = _storage.CreateConnection(); - await conn.OpenAsync(); + // ── Invariant that moved owner ── + // WAL used to be SiteStorageService's own job (an explicit PRAGMA in + // InitializeAsync). LocalDb now owns the file and its pragmas, so the service no + // longer sets it. The guarantee production depends on has NOT moved: without WAL + // the site's concurrent readers and writers start serializing on "database is + // locked". So rather than deleting this test with the code that used to provide + // the pragma, it is retargeted to assert the same guarantee against the new, + // LocalDb-backed service. journal_mode is persistent and file-scoped, so any + // connection observes it. A file-backed DB is still required — WAL is not + // available for :memory: databases, which is also why LocalDb has no in-memory mode. + await using var conn = _storage.CreateConnection(); // already open — do NOT call OpenAsync await using var cmd = conn.CreateCommand(); cmd.CommandText = "PRAGMA journal_mode;"; var mode = (string)(await cmd.ExecuteScalarAsync())!; @@ -209,7 +226,12 @@ public class SiteStorageServiceTests : IAsyncLifetime, IDisposable Assert.Empty(overrides); } - // ── Task 13: StoreDeployedConfigIfNewerAsync (guarded standby write) ── + // ── StoreDeployedConfigIfNewerAsync (the deployed_at-guarded write) ── + // + // Originally the standby's notify-and-fetch write path. LocalDb Phase 2 replaced that + // with change-data-capture, so the surviving caller is SiteReconciliationActor's + // startup self-heal against central, where the guard still stops a slow reconcile + // response from overwriting a newer config that landed while it was in flight. /// /// Seeds a deployed_configurations row with an explicit deployed_at timestamp using the same @@ -219,8 +241,10 @@ public class SiteStorageServiceTests : IAsyncLifetime, IDisposable string instanceName, string configJson, string deploymentId, string revisionHash, bool isEnabled, DateTimeOffset deployedAt) { - await using var conn = new SqliteConnection($"Data Source={_dbFile}"); - await conn.OpenAsync(); + // Seeded through the service's own (already-open) LocalDb connection: a raw + // SqliteConnection would lack the pragmas and the zb_hlc_next() UDF the site + // tables' capture triggers call. + await using var conn = _storage.CreateConnection(); await using var cmd = conn.CreateCommand(); cmd.CommandText = @" INSERT INTO deployed_configurations diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Repositories/SiteRepositoryTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Repositories/SiteRepositoryTests.cs index a5b68ffb..366d5eb8 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Repositories/SiteRepositoryTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Repositories/SiteRepositoryTests.cs @@ -1,6 +1,7 @@ using Microsoft.Extensions.Logging.Abstractions; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Repositories; +using ZB.MOM.WW.ScadaBridge.TestSupport; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Repositories; @@ -17,21 +18,32 @@ namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Repositories; /// public class SiteRepositoryTests : IDisposable { - private readonly string _dbFile; + private readonly TestLocalDb _localDb; public SiteRepositoryTests() { - _dbFile = Path.Combine(Path.GetTempPath(), $"site-repo-test-{Guid.NewGuid():N}.db"); + _localDb = TestLocalDb.CreateTemp("site-repo-test"); } public void Dispose() { - try { File.Delete(_dbFile); } catch { /* cleanup */ } + // Dispose first — the master connection anchors the WAL, so the sidecars + // cannot be removed while it is open. + var path = _localDb.Path; + _localDb.Dispose(); + TestLocalDb.DeleteFiles(path); GC.SuppressFinalize(this); } + /// + /// A brand-new instance over the same site database. + /// The service now takes an ILocalDb instead of a connection string, so the + /// single fixture database is shared while each call still yields a fresh service + /// object — which is what the restart tests below actually vary (the synthetic IDs are + /// derived per service/repository instance, not per connection). + /// private SiteStorageService NewStorage() - => new($"Data Source={_dbFile}", NullLogger.Instance); + => new(_localDb.Db, NullLogger.Instance); /// /// SiteRuntime-006: an external system stored via diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/ResyncWireSerializationPinTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/ResyncWireSerializationPinTests.cs deleted file mode 100644 index 43560a14..00000000 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/ResyncWireSerializationPinTests.cs +++ /dev/null @@ -1,95 +0,0 @@ -using Akka.TestKit.Xunit2; -using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; -using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors; -using ZB.MOM.WW.ScadaBridge.StoreAndForward; - -namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests; - -/// -/// Characterization pin for the chunked anti-entropy resync contract (review 02 round 2, -/// N2). (active→standby) and -/// (standby→active) ride intra-site Akka remoting on the default reflective-JSON wire -/// format. A rename/move, dropped setter, or non-default-constructible message would -/// silently break resync across a rolling upgrade and only surface as a divergent buffer -/// after a failover. These pin round-trip fidelity and type identity. (These messages are -/// NOT ClusterClient traffic, so they are intentionally absent from ClusterClientContractLockTests.) -/// -public class ResyncWireSerializationPinTests : TestKit -{ - private T RoundTrip(T message) - { - var serialization = Sys.Serialization; - var serializer = serialization.FindSerializerFor(message); - var bytes = serializer.ToBinary(message); - return (T)serialization.Deserialize(bytes, serializer.Identifier, message!.GetType()); - } - - private static StoreAndForwardMessage FullMessage() => new() - { - Id = Guid.NewGuid().ToString("N"), - Category = StoreAndForwardCategory.Notification, - Target = "Operators", - PayloadJson = "{\"notificationId\":\"abc\"}", - RetryCount = 4, - MaxRetries = 0, - RetryIntervalMs = 30000, - CreatedAt = DateTimeOffset.UtcNow, - LastAttemptAt = DateTimeOffset.UtcNow, - Status = StoreAndForwardMessageStatus.Parked, - LastError = "central rejected", - OriginInstanceName = "Plant.Pump3", - ExecutionId = Guid.NewGuid(), - SourceScript = "ScriptActor:MonitorSpeed", - ParentExecutionId = Guid.NewGuid(), - }; - - [Fact] - public void SfBufferSnapshotChunk_WithFullMessage_RoundTripsOnTheWire() - { - var message = FullMessage(); - var original = new SfBufferSnapshotChunk( - "resync-1", 2, 5, new List { message }, Truncated: true); - - var back = RoundTrip(original); - - Assert.Equal(original.ResyncId, back.ResyncId); - Assert.Equal(original.Sequence, back.Sequence); - Assert.Equal(original.TotalChunks, back.TotalChunks); - Assert.Equal(original.Truncated, back.Truncated); - var m = Assert.Single(back.Messages); - Assert.Equal(message.Id, m.Id); - Assert.Equal(message.Category, m.Category); - Assert.Equal(message.Target, m.Target); - Assert.Equal(message.PayloadJson, m.PayloadJson); - Assert.Equal(message.RetryCount, m.RetryCount); - Assert.Equal(message.MaxRetries, m.MaxRetries); - Assert.Equal(message.RetryIntervalMs, m.RetryIntervalMs); - Assert.Equal(message.CreatedAt, m.CreatedAt); - Assert.Equal(message.LastAttemptAt, m.LastAttemptAt); - Assert.Equal(message.Status, m.Status); - Assert.Equal(message.LastError, m.LastError); - Assert.Equal(message.OriginInstanceName, m.OriginInstanceName); - Assert.Equal(message.ExecutionId, m.ExecutionId); - Assert.Equal(message.SourceScript, m.SourceScript); - Assert.Equal(message.ParentExecutionId, m.ParentExecutionId); - } - - [Fact] - public void SfBufferResyncAck_RoundTripsOnTheWire() - { - var original = new SfBufferResyncAck("resync-1", 42); - - var back = RoundTrip(original); - - Assert.Equal(original.ResyncId, back.ResyncId); - Assert.Equal(original.RowCount, back.RowCount); - } - - // Type-identity pins: the reflective-JSON wire embeds CLR type manifests, so a - // rename/move of either type silently breaks resync across a rolling upgrade. - [Theory] - [InlineData(typeof(SfBufferSnapshotChunk), "ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors.SfBufferSnapshotChunk")] - [InlineData(typeof(SfBufferResyncAck), "ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors.SfBufferResyncAck")] - public void ResyncContract_TypeIdentity_IsPinned(Type type, string expectedFullName) => - Assert.Equal(expectedFullName, type.FullName); -} diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/NotifyHelperTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/NotifyHelperTests.cs index 8b1f7f4f..b256e5a9 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/NotifyHelperTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/NotifyHelperTests.cs @@ -1,12 +1,12 @@ using System.Text.Json; using Akka.Actor; using Akka.TestKit.Xunit2; -using Microsoft.Data.Sqlite; using Microsoft.Extensions.Logging.Abstractions; using ZB.MOM.WW.ScadaBridge.Commons.Messages.Notification; using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts; using ZB.MOM.WW.ScadaBridge.StoreAndForward; +using ZB.MOM.WW.ScadaBridge.TestSupport; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Scripts; @@ -24,18 +24,16 @@ namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Scripts; /// public class NotifyHelperTests : TestKit, IAsyncLifetime, IDisposable { - private readonly SqliteConnection _keepAlive; + private readonly TestLocalDb _localDb; private readonly StoreAndForwardStorage _storage; private readonly StoreAndForwardService _saf; public NotifyHelperTests() { - var dbName = $"NotifyTests_{Guid.NewGuid():N}"; - var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared"; - _keepAlive = new SqliteConnection(connStr); - _keepAlive.Open(); + // LocalDb has no in-memory mode, so the store runs over a real temp file. + _localDb = TestLocalDb.CreateTemp("NotifyTests"); - _storage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + _storage = new StoreAndForwardStorage(_localDb.Db, NullLogger.Instance); var options = new StoreAndForwardOptions { DefaultRetryInterval = TimeSpan.Zero, @@ -53,7 +51,9 @@ public class NotifyHelperTests : TestKit, IAsyncLifetime, IDisposable { if (disposing) { - _keepAlive.Dispose(); + // The master connection anchors the WAL — dispose before deleting. + _localDb.Dispose(); + TestLocalDb.DeleteFiles(_localDb.Path); } base.Dispose(disposing); } diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/NotifySendAuditEmissionTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/NotifySendAuditEmissionTests.cs index 9bfa69c8..4c1f5d7c 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/NotifySendAuditEmissionTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/NotifySendAuditEmissionTests.cs @@ -1,7 +1,6 @@ using System.Text.Json; using Akka.Actor; using Akka.TestKit.Xunit2; -using Microsoft.Data.Sqlite; using Microsoft.Extensions.Logging.Abstractions; using ZB.MOM.WW.Audit; using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services; @@ -10,6 +9,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts; using IAuditWriter = ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services.IAuditWriter; using ZB.MOM.WW.ScadaBridge.StoreAndForward; +using ZB.MOM.WW.ScadaBridge.TestSupport; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Scripts; @@ -63,18 +63,16 @@ public class NotifySendAuditEmissionTests : TestKit, IAsyncLifetime, IDisposable /// private static readonly Guid TestExecutionId = Guid.NewGuid(); - private readonly SqliteConnection _keepAlive; + private readonly TestLocalDb _localDb; private readonly StoreAndForwardStorage _storage; private readonly StoreAndForwardService _saf; public NotifySendAuditEmissionTests() { - var dbName = $"NotifySendAudit_{Guid.NewGuid():N}"; - var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared"; - _keepAlive = new SqliteConnection(connStr); - _keepAlive.Open(); + // LocalDb has no in-memory mode, so the store runs over a real temp file. + _localDb = TestLocalDb.CreateTemp("NotifySendAudit"); - _storage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + _storage = new StoreAndForwardStorage(_localDb.Db, NullLogger.Instance); var options = new StoreAndForwardOptions { DefaultRetryInterval = TimeSpan.Zero, @@ -92,7 +90,9 @@ public class NotifySendAuditEmissionTests : TestKit, IAsyncLifetime, IDisposable { if (disposing) { - _keepAlive.Dispose(); + // The master connection anchors the WAL — dispose before deleting. + _localDb.Dispose(); + TestLocalDb.DeleteFiles(_localDb.Path); } base.Dispose(disposing); } diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Tracking/OperationTrackingSchemaTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Tracking/OperationTrackingSchemaTests.cs new file mode 100644 index 00000000..49ae95af --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Tracking/OperationTrackingSchemaTests.cs @@ -0,0 +1,147 @@ +using Microsoft.Data.Sqlite; +using ZB.MOM.WW.ScadaBridge.SiteRuntime.Tracking; + +namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Tracking; + +/// +/// LocalDb Phase 1 (Task 2) — the OperationTracking DDL is extracted out of +/// so it can also be applied from the Host's +/// AddZbLocalDb onReady callback, which owns table creation for the consolidated +/// site database. These tests pin the extracted schema against the shape the store has +/// always created: the move must be behaviour-preserving. +/// +public class OperationTrackingSchemaTests +{ + private static SqliteConnection OpenConnection() + { + var conn = new SqliteConnection("Data Source=:memory:"); + conn.Open(); + return conn; + } + + private static List<(string Name, string Type, bool NotNull, bool Pk)> TableInfo( + SqliteConnection conn, string table) + { + using var cmd = conn.CreateCommand(); + cmd.CommandText = $"PRAGMA table_info('{table}')"; + using var reader = cmd.ExecuteReader(); + var cols = new List<(string, string, bool, bool)>(); + while (reader.Read()) + { + cols.Add((reader.GetString(1), reader.GetString(2), reader.GetInt32(3) == 1, + reader.GetInt32(5) > 0)); + } + + return cols; + } + + [Fact] + public void Apply_CreatesTableWithExpectedColumns() + { + using var conn = OpenConnection(); + + OperationTrackingSchema.Apply(conn); + + var cols = TableInfo(conn, "OperationTracking"); + Assert.Equal( + [ + "TrackedOperationId", "Kind", "TargetSummary", "Status", "RetryCount", + "LastError", "HttpStatus", "CreatedAtUtc", "UpdatedAtUtc", "TerminalAtUtc", + "SourceInstanceId", "SourceScript", "SourceNode" + ], + cols.Select(c => c.Name)); + } + + [Fact] + public void Apply_UsesTextPrimaryKey() + { + using var conn = OpenConnection(); + + OperationTrackingSchema.Apply(conn); + + // RegisterReplicated requires an explicit primary key and rejects BLOB + // columns; a TEXT PK is what makes this table replicate as-is. + var pk = Assert.Single(TableInfo(conn, "OperationTracking"), c => c.Pk); + Assert.Equal("TrackedOperationId", pk.Name); + Assert.Equal("TEXT", pk.Type); + } + + [Fact] + public void Apply_HasNoBlobColumns() + { + using var conn = OpenConnection(); + + OperationTrackingSchema.Apply(conn); + + // ILocalDb.RegisterReplicated throws on any column whose declared type + // contains BLOB (json_object cannot capture them). + Assert.DoesNotContain( + TableInfo(conn, "OperationTracking"), + c => c.Type.Contains("BLOB", StringComparison.OrdinalIgnoreCase)); + } + + [Fact] + public void Apply_CreatesExpectedIndexes() + { + using var conn = OpenConnection(); + + OperationTrackingSchema.Apply(conn); + + using var cmd = conn.CreateCommand(); + cmd.CommandText = + "SELECT name FROM sqlite_master WHERE type = 'index' AND tbl_name = 'OperationTracking' " + + "AND name NOT LIKE 'sqlite_%' ORDER BY name"; + using var reader = cmd.ExecuteReader(); + var indexes = new List(); + while (reader.Read()) indexes.Add(reader.GetString(0)); + + Assert.Equal( + ["IX_OperationTracking_Status_Updated", "IX_OperationTracking_UpdatedAt"], + indexes); + } + + [Fact] + public void Apply_IsIdempotent() + { + using var conn = OpenConnection(); + + OperationTrackingSchema.Apply(conn); + OperationTrackingSchema.Apply(conn); + + Assert.Equal(13, TableInfo(conn, "OperationTracking").Count); + } + + [Fact] + public void Apply_AddsSourceNodeToLegacyTableMissingIt() + { + using var conn = OpenConnection(); + + // A tracking DB created by a pre-SourceNode build. CREATE TABLE IF NOT + // EXISTS will not add the column, so Apply must ALTER it in — the + // additive-migration behaviour the store has today must survive extraction. + using (var legacy = conn.CreateCommand()) + { + legacy.CommandText = """ + CREATE TABLE OperationTracking ( + TrackedOperationId TEXT NOT NULL PRIMARY KEY, + Kind TEXT NOT NULL, + TargetSummary TEXT NULL, + Status TEXT NOT NULL, + RetryCount INTEGER NOT NULL DEFAULT 0, + LastError TEXT NULL, + HttpStatus INTEGER NULL, + CreatedAtUtc TEXT NOT NULL, + UpdatedAtUtc TEXT NOT NULL, + TerminalAtUtc TEXT NULL, + SourceInstanceId TEXT NULL, + SourceScript TEXT NULL + ); + """; + legacy.ExecuteNonQuery(); + } + + OperationTrackingSchema.Apply(conn); + + Assert.Contains(TableInfo(conn, "OperationTracking"), c => c.Name == "SourceNode"); + } +} diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Tracking/OperationTrackingStoreTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Tracking/OperationTrackingStoreTests.cs index 8ae7ebb4..c327324b 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Tracking/OperationTrackingStoreTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Tracking/OperationTrackingStoreTests.cs @@ -1,6 +1,8 @@ using Microsoft.Data.Sqlite; +using Microsoft.Extensions.Configuration; +using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.Logging.Abstractions; -using Microsoft.Extensions.Options; +using ZB.MOM.WW.LocalDb; using ZB.MOM.WW.ScadaBridge.Commons.Types; using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Tracking; @@ -9,30 +11,93 @@ namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Tracking; /// /// Audit Log #23 (M3 Bundle A — Task A2) — schema + behaviour tests for the -/// site-local . Each test uses a unique -/// shared-cache in-memory SQLite database so the store and the verifier share -/// the same store without touching disk. +/// site-local . /// -public class OperationTrackingStoreTests +/// +/// The store now takes rather than a connection string, so each +/// test gets a real ILocalDb over a unique temp FILE. The previous +/// mode=memory&cache=shared fixture is gone and cannot come back: +/// LocalDbOptions.Path is a filesystem path, and — more importantly — a raw +/// connection to a shared-cache in-memory database would not carry the +/// zb_hlc_next() UDF the capture triggers call, so testing through one would no +/// longer resemble how the store runs in the host. +/// +/// Verifier connections are plain s over the same file. +/// That is deliberate and safe: they only read, or write columns on an unregistered +/// table (this fixture never calls RegisterReplicated), so no trigger fires and +/// the missing UDF is never reached. +/// +/// +public class OperationTrackingStoreTests : IDisposable { - private static (OperationTrackingStore store, string dataSource) CreateStore( - string testName) + // Every ILocalDb and temp file created by this fixture, torn down together. The + // ILocalDb holds an open master connection anchoring the WAL, so it must be + // disposed before the file can be deleted. + private readonly List _providers = []; + private readonly List _tempFiles = []; + + public void Dispose() { - var dataSource = $"file:{testName}-{Guid.NewGuid():N}?mode=memory&cache=shared"; - var connectionString = $"Data Source={dataSource};Cache=Shared"; - var options = new OperationTrackingOptions + foreach (var provider in _providers) { - ConnectionString = connectionString, - }; - var store = new OperationTrackingStore( - Options.Create(options), - NullLogger.Instance); - return (store, dataSource); + try { provider.Dispose(); } catch { /* best effort */ } + } + + foreach (var path in _tempFiles) + { + foreach (var suffix in new[] { "", "-wal", "-shm" }) + { + try { File.Delete(path + suffix); } catch { /* best effort */ } + } + } + + GC.SuppressFinalize(this); } - private static SqliteConnection OpenVerifierConnection(string dataSource) + /// Allocates a unique temp database path, registered for cleanup. + private string NewDatabasePath(string testName) { - var connection = new SqliteConnection($"Data Source={dataSource};Cache=Shared"); + var path = Path.Combine( + Path.GetTempPath(), $"optracking-{testName}-{Guid.NewGuid():N}.db"); + _tempFiles.Add(path); + return path; + } + + /// + /// Builds a real over . No + /// onReady callback and no RegisterReplicated — the store's own + /// InitializeSchema creates the table, which is what keeps a + /// directly-constructed store self-sufficient. + /// + private ILocalDb CreateLocalDb(string databasePath) + { + var configuration = new ConfigurationBuilder() + .AddInMemoryCollection(new Dictionary + { + ["LocalDb:Path"] = databasePath, + }) + .Build(); + + var provider = new ServiceCollection() + .AddZbLocalDb(configuration) + .BuildServiceProvider(); + _providers.Add(provider); + + return provider.GetRequiredService(); + } + + private (OperationTrackingStore store, string dataSource) CreateStore(string testName) + { + var databasePath = NewDatabasePath(testName); + var store = new OperationTrackingStore( + CreateLocalDb(databasePath), + NullLogger.Instance); + return (store, databasePath); + } + + private static SqliteConnection OpenVerifierConnection(string databasePath) + { + var connection = new SqliteConnection($"Data Source={databasePath}"); connection.Open(); return connection; } @@ -111,14 +176,19 @@ public class OperationTrackingStoreTests ON OperationTracking (Status, UpdatedAtUtc); """; - private static SqliteConnection SeedPreSourceNodeSchemaDatabase(string dataSource) + /// + /// Writes the pre-SourceNode schema into and closes + /// the connection again. Unlike the old shared-cache in-memory fixture — where the + /// seeding connection had to stay open or the database evaporated — a file database + /// persists on its own, so nothing is held across the store's construction. + /// + private static void SeedPreSourceNodeSchemaDatabase(string databasePath) { - var connection = new SqliteConnection($"Data Source={dataSource};Cache=Shared"); + using var connection = new SqliteConnection($"Data Source={databasePath}"); connection.Open(); using var cmd = connection.CreateCommand(); cmd.CommandText = OldPreSourceNodeSchema; cmd.ExecuteNonQuery(); - return connection; } private static bool ColumnExists(SqliteConnection connection, string columnName) @@ -129,36 +199,36 @@ public class OperationTrackingStoreTests return Convert.ToInt32(cmd.ExecuteScalar()) > 0; } - private static OperationTrackingStore CreateStoreOver(string dataSource) - { - var connectionString = $"Data Source={dataSource};Cache=Shared"; - var options = new OperationTrackingOptions { ConnectionString = connectionString }; - return new OperationTrackingStore( - Options.Create(options), - NullLogger.Instance); - } + private OperationTrackingStore CreateStoreOver(string databasePath) + => new(CreateLocalDb(databasePath), NullLogger.Instance); [Fact] public async Task Initialize_adds_SourceNode_to_pre_existing_schema() { - var dataSource = $"file:{nameof(Initialize_adds_SourceNode_to_pre_existing_schema)}-{Guid.NewGuid():N}?mode=memory&cache=shared"; + var databasePath = NewDatabasePath(nameof(Initialize_adds_SourceNode_to_pre_existing_schema)); - // A pre-SourceNode deployment: tracking.db already exists with the - // 12-column schema and NO SourceNode column. - using var seedConnection = SeedPreSourceNodeSchemaDatabase(dataSource); - Assert.True(ColumnExists(seedConnection, "SourceInstanceId")); - Assert.True(ColumnExists(seedConnection, "SourceScript")); - Assert.False(ColumnExists(seedConnection, "SourceNode")); + // A pre-SourceNode deployment: the tracking database already exists with + // the 12-column schema and NO SourceNode column. + SeedPreSourceNodeSchemaDatabase(databasePath); + using (var seeded = OpenVerifierConnection(databasePath)) + { + Assert.True(ColumnExists(seeded, "SourceInstanceId")); + Assert.True(ColumnExists(seeded, "SourceScript")); + Assert.False(ColumnExists(seeded, "SourceNode")); + } // Upgrade: a post-branch OperationTrackingStore opens the same database. // Its InitializeSchema must ALTER the missing SourceNode column in — // the CREATE TABLE IF NOT EXISTS alone is a no-op against the existing // table. - await using (var store = CreateStoreOver(dataSource)) + await using (var store = CreateStoreOver(databasePath)) { - Assert.True( - ColumnExists(seedConnection, "SourceNode"), - "OperationTrackingStore must ALTER the SourceNode column into a pre-existing OperationTracking table."); + using (var verifier = OpenVerifierConnection(databasePath)) + { + Assert.True( + ColumnExists(verifier, "SourceNode"), + "OperationTrackingStore must ALTER the SourceNode column into a pre-existing OperationTracking table."); + } // A RecordEnqueueAsync binding $sourceNode must now succeed; without // the ALTER it would fail with "no such column: SourceNode". @@ -178,9 +248,10 @@ public class OperationTrackingStoreTests // Idempotency: a second store over the now-upgraded DB must not error // (the probe sees SourceNode already present and skips the ALTER). - await using (var storeAgain = CreateStoreOver(dataSource)) + await using (var storeAgain = CreateStoreOver(databasePath)) { - Assert.True(ColumnExists(seedConnection, "SourceNode")); + using var verifier = OpenVerifierConnection(databasePath); + Assert.True(ColumnExists(verifier, "SourceNode")); } } diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.csproj b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.csproj index 0f1db5ff..581a977d 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.csproj +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.csproj @@ -28,6 +28,7 @@ - + + diff --git a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/CachedCallAttemptEmissionTests.cs b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/CachedCallAttemptEmissionTests.cs index d04e8bb9..f01746fa 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/CachedCallAttemptEmissionTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/CachedCallAttemptEmissionTests.cs @@ -1,8 +1,8 @@ -using Microsoft.Data.Sqlite; using Microsoft.Extensions.Logging.Abstractions; using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services; using ZB.MOM.WW.ScadaBridge.Commons.Types; using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; +using ZB.MOM.WW.ScadaBridge.TestSupport; namespace ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests; @@ -20,7 +20,7 @@ namespace ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests; /// public class CachedCallAttemptEmissionTests : IAsyncLifetime, IDisposable { - private readonly SqliteConnection _keepAlive; + private readonly TestLocalDb _localDb; private readonly StoreAndForwardStorage _storage; private readonly StoreAndForwardService _service; private readonly StoreAndForwardOptions _options; @@ -28,12 +28,9 @@ public class CachedCallAttemptEmissionTests : IAsyncLifetime, IDisposable public CachedCallAttemptEmissionTests() { - var dbName = $"E4Tests_{Guid.NewGuid():N}"; - var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared"; - _keepAlive = new SqliteConnection(connStr); - _keepAlive.Open(); + _localDb = TestLocalDb.CreateTemp("E4Tests"); - _storage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + _storage = new StoreAndForwardStorage(_localDb.Db, NullLogger.Instance); _options = new StoreAndForwardOptions { @@ -48,14 +45,18 @@ public class CachedCallAttemptEmissionTests : IAsyncLifetime, IDisposable _storage, _options, NullLogger.Instance, - replication: null, cachedCallObserver: _observer, siteId: "site-77"); } public async Task InitializeAsync() => await _storage.InitializeAsync(); public Task DisposeAsync() => Task.CompletedTask; - public void Dispose() => _keepAlive.Dispose(); + public void Dispose() + { + var path = _localDb.Path; + _localDb.Dispose(); + TestLocalDb.DeleteFiles(path); + } /// /// Captures every observer notification so tests can assert on the @@ -476,10 +477,8 @@ public class CachedCallAttemptEmissionTests : IAsyncLifetime, IDisposable // Fresh service over its own storage so StartAsync's pump/timer is isolated // from the shared _service. - var connStr = $"Data Source=SlowObs_{Guid.NewGuid():N};Mode=Memory;Cache=Shared"; - using var keepAlive = new SqliteConnection(connStr); - keepAlive.Open(); - var storage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + var localDb = TestLocalDb.CreateTemp("SlowObs"); + var storage = new StoreAndForwardStorage(localDb.Db, NullLogger.Instance); await storage.InitializeAsync(); var service = new StoreAndForwardService( storage, @@ -490,7 +489,6 @@ public class CachedCallAttemptEmissionTests : IAsyncLifetime, IDisposable RetryTimerInterval = TimeSpan.FromHours(1), // timer never fires in-test }, NullLogger.Instance, - replication: null, cachedCallObserver: observer, siteId: "site-77"); @@ -515,6 +513,12 @@ public class CachedCallAttemptEmissionTests : IAsyncLifetime, IDisposable } lock (observed) Assert.Single(observed); } - finally { await service.StopAsync(); } + finally + { + await service.StopAsync(); + var path = localDb.Path; + localDb.Dispose(); + TestLocalDb.DeleteFiles(path); + } } } diff --git a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ParkedMessageHandlerActorTests.cs b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ParkedMessageHandlerActorTests.cs index 93240a69..3692eb01 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ParkedMessageHandlerActorTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ParkedMessageHandlerActorTests.cs @@ -1,9 +1,9 @@ using Akka.Actor; using Akka.TestKit.Xunit2; -using Microsoft.Data.Sqlite; using Microsoft.Extensions.Logging.Abstractions; using ZB.MOM.WW.ScadaBridge.Commons.Messages.RemoteQuery; using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; +using ZB.MOM.WW.ScadaBridge.TestSupport; namespace ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests; @@ -14,24 +14,21 @@ namespace ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests; /// public class ParkedMessageHandlerActorTests : TestKit, IAsyncLifetime, IDisposable { - private readonly SqliteConnection _keepAlive; + private readonly TestLocalDb _localDb; private readonly StoreAndForwardStorage _storage; private readonly StoreAndForwardService _service; public ParkedMessageHandlerActorTests() { - var connStr = $"Data Source=ActorTests_{Guid.NewGuid():N};Mode=Memory;Cache=Shared"; - _keepAlive = new SqliteConnection(connStr); - _keepAlive.Open(); + _localDb = TestLocalDb.CreateTemp("ActorTests"); - _storage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + _storage = new StoreAndForwardStorage(_localDb.Db, NullLogger.Instance); var options = new StoreAndForwardOptions { DefaultRetryInterval = TimeSpan.Zero, DefaultMaxRetries = 1, RetryTimerInterval = TimeSpan.FromMinutes(10), - ReplicationEnabled = false, }; _service = new StoreAndForwardService( @@ -44,7 +41,13 @@ public class ParkedMessageHandlerActorTests : TestKit, IAsyncLifetime, IDisposab protected override void Dispose(bool disposing) { - if (disposing) _keepAlive.Dispose(); + if (disposing) + { + var path = _localDb.Path; + _localDb.Dispose(); + TestLocalDb.DeleteFiles(path); + } + base.Dispose(disposing); } diff --git a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ParkedOperationRelayTests.cs b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ParkedOperationRelayTests.cs index 91861693..3f32e3ec 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ParkedOperationRelayTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ParkedOperationRelayTests.cs @@ -1,10 +1,10 @@ using Akka.Actor; using Akka.TestKit.Xunit2; -using Microsoft.Data.Sqlite; using Microsoft.Extensions.Logging.Abstractions; using ZB.MOM.WW.ScadaBridge.Commons.Messages.RemoteQuery; using ZB.MOM.WW.ScadaBridge.Commons.Types; using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; +using ZB.MOM.WW.ScadaBridge.TestSupport; namespace ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests; @@ -19,24 +19,21 @@ namespace ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests; /// public class ParkedOperationRelayTests : TestKit, IAsyncLifetime, IDisposable { - private readonly SqliteConnection _keepAlive; + private readonly TestLocalDb _localDb; private readonly StoreAndForwardStorage _storage; private readonly StoreAndForwardService _service; public ParkedOperationRelayTests() { - var connStr = $"Data Source=RelayTests_{Guid.NewGuid():N};Mode=Memory;Cache=Shared"; - _keepAlive = new SqliteConnection(connStr); - _keepAlive.Open(); + _localDb = TestLocalDb.CreateTemp("RelayTests"); - _storage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + _storage = new StoreAndForwardStorage(_localDb.Db, NullLogger.Instance); var options = new StoreAndForwardOptions { DefaultRetryInterval = TimeSpan.Zero, DefaultMaxRetries = 1, RetryTimerInterval = TimeSpan.FromMinutes(10), - ReplicationEnabled = false, }; _service = new StoreAndForwardService( @@ -49,7 +46,13 @@ public class ParkedOperationRelayTests : TestKit, IAsyncLifetime, IDisposable protected override void Dispose(bool disposing) { - if (disposing) _keepAlive.Dispose(); + if (disposing) + { + var path = _localDb.Path; + _localDb.Dispose(); + TestLocalDb.DeleteFiles(path); + } + base.Dispose(disposing); } diff --git a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/QueueDepthGaugeTests.cs b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/QueueDepthGaugeTests.cs index 57dcea6c..6b435f5f 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/QueueDepthGaugeTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/QueueDepthGaugeTests.cs @@ -1,8 +1,8 @@ using System.Diagnostics.Metrics; -using Microsoft.Data.Sqlite; using Microsoft.Extensions.Logging.Abstractions; using ZB.MOM.WW.ScadaBridge.Commons.Observability; using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; +using ZB.MOM.WW.ScadaBridge.TestSupport; namespace ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests; @@ -20,18 +20,15 @@ namespace ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests; /// public class QueueDepthGaugeTests : IAsyncLifetime, IDisposable { - private readonly SqliteConnection _keepAlive; + private readonly TestLocalDb _localDb; private readonly StoreAndForwardStorage _storage; private readonly StoreAndForwardService _service; public QueueDepthGaugeTests() { - var dbName = $"QueueDepthTests_{Guid.NewGuid():N}"; - var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared"; - _keepAlive = new SqliteConnection(connStr); - _keepAlive.Open(); + _localDb = TestLocalDb.CreateTemp("QueueDepthTests"); - _storage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + _storage = new StoreAndForwardStorage(_localDb.Db, NullLogger.Instance); var options = new StoreAndForwardOptions { @@ -56,7 +53,12 @@ public class QueueDepthGaugeTests : IAsyncLifetime, IDisposable public async Task DisposeAsync() => await _service.StopAsync(); - public void Dispose() => _keepAlive.Dispose(); + public void Dispose() + { + var path = _localDb.Path; + _localDb.Dispose(); + TestLocalDb.DeleteFiles(path); + } /// /// Reads the current value of the scadabridge.store_and_forward.queue.depth diff --git a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ReplicationServiceTests.cs b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ReplicationServiceTests.cs deleted file mode 100644 index 7df0d8e8..00000000 --- a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ReplicationServiceTests.cs +++ /dev/null @@ -1,234 +0,0 @@ -using Microsoft.Data.Sqlite; -using Microsoft.Extensions.Logging; -using Microsoft.Extensions.Logging.Abstractions; -using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; - -namespace ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests; - -/// -/// WP-11: Tests for async replication to standby. -/// -public class ReplicationServiceTests : IAsyncLifetime, IDisposable -{ - private readonly SqliteConnection _keepAlive; - private readonly StoreAndForwardStorage _storage; - private readonly ReplicationService _replicationService; - - public ReplicationServiceTests() - { - var dbName = $"RepTests_{Guid.NewGuid():N}"; - var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared"; - _keepAlive = new SqliteConnection(connStr); - _keepAlive.Open(); - - _storage = new StoreAndForwardStorage(connStr, NullLogger.Instance); - - var options = new StoreAndForwardOptions { ReplicationEnabled = true }; - _replicationService = new ReplicationService( - options, NullLogger.Instance); - } - - public async Task InitializeAsync() => await _storage.InitializeAsync(); - - public Task DisposeAsync() => Task.CompletedTask; - - public void Dispose() => _keepAlive.Dispose(); - - [Fact] - public void ReplicateEnqueue_NoHandler_DoesNotThrow() - { - var msg = CreateMessage("rep1"); - _replicationService.ReplicateEnqueue(msg); - } - - [Fact] - public async Task ReplicateEnqueue_WithHandler_ForwardsOperation() - { - ReplicationOperation? captured = null; - _replicationService.SetReplicationHandler(op => - { - captured = op; - return Task.CompletedTask; - }); - - var msg = CreateMessage("rep2"); - _replicationService.ReplicateEnqueue(msg); - - await Task.Delay(200); - - Assert.NotNull(captured); - Assert.Equal(ReplicationOperationType.Add, captured!.OperationType); - Assert.Equal("rep2", captured.MessageId); - } - - [Fact] - public async Task ReplicateRemove_WithHandler_ForwardsRemoveOperation() - { - ReplicationOperation? captured = null; - _replicationService.SetReplicationHandler(op => - { - captured = op; - return Task.CompletedTask; - }); - - _replicationService.ReplicateRemove("rep3"); - - await Task.Delay(200); - - Assert.NotNull(captured); - Assert.Equal(ReplicationOperationType.Remove, captured!.OperationType); - Assert.Equal("rep3", captured.MessageId); - } - - [Fact] - public async Task ReplicatePark_WithHandler_ForwardsParkOperation() - { - ReplicationOperation? captured = null; - _replicationService.SetReplicationHandler(op => - { - captured = op; - return Task.CompletedTask; - }); - - var msg = CreateMessage("rep4"); - _replicationService.ReplicatePark(msg); - - await Task.Delay(200); - - Assert.NotNull(captured); - Assert.Equal(ReplicationOperationType.Park, captured!.OperationType); - } - - [Fact] - public async Task ApplyReplicatedOperationAsync_Add_EnqueuesMessage() - { - var msg = CreateMessage("apply1"); - var operation = new ReplicationOperation(ReplicationOperationType.Add, "apply1", msg); - - await _replicationService.ApplyReplicatedOperationAsync(operation, _storage); - - var retrieved = await _storage.GetMessageByIdAsync("apply1"); - Assert.NotNull(retrieved); - } - - [Fact] - public async Task ApplyReplicatedOperationAsync_Remove_DeletesMessage() - { - var msg = CreateMessage("apply2"); - await _storage.EnqueueAsync(msg); - - var operation = new ReplicationOperation(ReplicationOperationType.Remove, "apply2", null); - await _replicationService.ApplyReplicatedOperationAsync(operation, _storage); - - var retrieved = await _storage.GetMessageByIdAsync("apply2"); - Assert.Null(retrieved); - } - - [Fact] - public async Task ApplyReplicatedOperationAsync_Park_UpdatesStatus() - { - var msg = CreateMessage("apply3"); - await _storage.EnqueueAsync(msg); - - var operation = new ReplicationOperation(ReplicationOperationType.Park, "apply3", msg); - await _replicationService.ApplyReplicatedOperationAsync(operation, _storage); - - var retrieved = await _storage.GetMessageByIdAsync("apply3"); - Assert.NotNull(retrieved); - Assert.Equal(StoreAndForwardMessageStatus.Parked, retrieved!.Status); - } - - [Fact] - public void ReplicateEnqueue_WhenReplicationDisabled_DoesNothing() - { - var options = new StoreAndForwardOptions { ReplicationEnabled = false }; - var service = new ReplicationService(options, NullLogger.Instance); - - bool handlerCalled = false; - service.SetReplicationHandler(_ => { handlerCalled = true; return Task.CompletedTask; }); - - service.ReplicateEnqueue(CreateMessage("disabled1")); - - Assert.False(handlerCalled); - } - - [Fact] - public async Task ReplicateEnqueue_HandlerThrows_DoesNotPropagateException() - { - _replicationService.SetReplicationHandler(_ => - throw new InvalidOperationException("standby down")); - - _replicationService.ReplicateEnqueue(CreateMessage("err1")); - - await Task.Delay(200); - // No exception -- fire-and-forget, best-effort - } - - // ── Task 10 (arch review 02): ordered, observable replication dispatch ── - - [Fact] - public void ReplicationOperations_AreDispatchedInIssueOrder() - { - var seen = new List<(ReplicationOperationType, string)>(); - _replicationService.SetReplicationHandler(op => - { - seen.Add((op.OperationType, op.MessageId)); - return Task.CompletedTask; - }); - - for (var i = 0; i < 200; i++) - { - _replicationService.ReplicateEnqueue(CreateMessage($"m{i}")); - _replicationService.ReplicateRemove($"m{i}"); - } - - // Inline dispatch: by the time the calls return, every op was handed to the - // handler, Add strictly before Remove per id. Pre-fix (Task.Run) this was - // racy in both count and order. - Assert.Equal(400, seen.Count); - for (var i = 0; i < 200; i++) - { - Assert.Equal((ReplicationOperationType.Add, $"m{i}"), seen[2 * i]); - Assert.Equal((ReplicationOperationType.Remove, $"m{i}"), seen[2 * i + 1]); - } - } - - [Fact] - public void ReplicationHandlerThrow_IsSwallowed_AndLoggedAtWarning() - { - var logger = new CapturingLogger(); - var service = new ReplicationService(new StoreAndForwardOptions(), logger); - service.SetReplicationHandler(_ => throw new InvalidOperationException("peer gone")); - - service.ReplicateRemove("m1"); // must not throw - - Assert.Contains(logger.Entries, e => e.Level == LogLevel.Warning); - } - - private static StoreAndForwardMessage CreateMessage(string id) - { - return new StoreAndForwardMessage - { - Id = id, - Category = StoreAndForwardCategory.ExternalSystem, - Target = "target", - PayloadJson = "{}", - RetryCount = 0, - MaxRetries = 50, - RetryIntervalMs = 30000, - CreatedAt = DateTimeOffset.UtcNow, - Status = StoreAndForwardMessageStatus.Pending - }; - } - - /// Minimal in-memory logger that records level + rendered message. - private sealed class CapturingLogger : ILogger - { - public List<(LogLevel Level, string Message)> Entries { get; } = new(); - public IDisposable? BeginScope(TState state) where TState : notnull => null; - public bool IsEnabled(LogLevel logLevel) => true; - public void Log(LogLevel logLevel, EventId eventId, TState state, Exception? exception, - Func formatter) - => Entries.Add((logLevel, formatter(state, exception))); - } -} diff --git a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ReplicationWireSerializationPinTests.cs b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ReplicationWireSerializationPinTests.cs deleted file mode 100644 index f1d194f0..00000000 --- a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ReplicationWireSerializationPinTests.cs +++ /dev/null @@ -1,82 +0,0 @@ -using Akka.TestKit.Xunit2; -using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; - -namespace ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests; - -/// -/// Characterization pin for the intra-cluster S&F replication contract. A -/// (carrying a full ) -/// is Told from the active node to the standby node's replication actor and rides -/// the Akka default reflective-JSON wire format. Nothing tested that it round-trips -/// or that its type identity is stable — a rename/move, a dropped setter, or a -/// non-default-constructible message would silently break standby buffer sync and -/// only surface as a divergent buffer after a failover. -/// -/// The serializer swap (proto / explicit bindings) is deferred; until then this pin -/// is the guardrail. -/// -public class ReplicationWireSerializationPinTests : TestKit -{ - private T RoundTrip(T message) - { - var serialization = Sys.Serialization; - var serializer = serialization.FindSerializerFor(message); - var bytes = serializer.ToBinary(message); - return (T)serialization.Deserialize(bytes, serializer.Identifier, message!.GetType()); - } - - [Fact] - public void ReplicationOperation_WithFullMessage_RoundTripsOnTheWire() - { - var message = new StoreAndForwardMessage - { - Id = Guid.NewGuid().ToString("N"), - Category = StoreAndForwardCategory.Notification, - Target = "Operators", - PayloadJson = "{\"notificationId\":\"abc\"}", - RetryCount = 4, - MaxRetries = 0, - RetryIntervalMs = 30000, - CreatedAt = DateTimeOffset.UtcNow, - LastAttemptAt = DateTimeOffset.UtcNow, - Status = StoreAndForwardMessageStatus.Parked, - LastError = "central rejected", - OriginInstanceName = "Plant.Pump3", - ExecutionId = Guid.NewGuid(), - SourceScript = "ScriptActor:MonitorSpeed", - ParentExecutionId = Guid.NewGuid(), - }; - var original = new ReplicationOperation(ReplicationOperationType.Park, message.Id, message); - - var back = RoundTrip(original); - - Assert.Equal(original.OperationType, back.OperationType); - Assert.Equal(original.MessageId, back.MessageId); - Assert.NotNull(back.Message); - var m = back.Message!; - Assert.Equal(message.Id, m.Id); - Assert.Equal(message.Category, m.Category); - Assert.Equal(message.Target, m.Target); - Assert.Equal(message.PayloadJson, m.PayloadJson); - Assert.Equal(message.RetryCount, m.RetryCount); - Assert.Equal(message.MaxRetries, m.MaxRetries); - Assert.Equal(message.RetryIntervalMs, m.RetryIntervalMs); - Assert.Equal(message.CreatedAt, m.CreatedAt); - Assert.Equal(message.LastAttemptAt, m.LastAttemptAt); - Assert.Equal(message.Status, m.Status); - Assert.Equal(message.LastError, m.LastError); - Assert.Equal(message.OriginInstanceName, m.OriginInstanceName); - Assert.Equal(message.ExecutionId, m.ExecutionId); - Assert.Equal(message.SourceScript, m.SourceScript); - Assert.Equal(message.ParentExecutionId, m.ParentExecutionId); - } - - // Type-identity pins: the reflective-JSON wire embeds CLR type manifests, so a - // rename/move of either type silently breaks standby replication across a - // rolling upgrade. If one fails, you are making a wire-breaking change. - [Theory] - [InlineData(typeof(ReplicationOperation), "ZB.MOM.WW.ScadaBridge.StoreAndForward.ReplicationOperation")] - [InlineData(typeof(StoreAndForwardMessage), "ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardMessage")] - public void ReplicationContract_TypeIdentity_IsPinned(Type type, string expectedFullName) => - Assert.Equal(expectedFullName, type.FullName); -} diff --git a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardOptionsTests.cs b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardOptionsTests.cs index 1da1c996..e8b9d0f1 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardOptionsTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardOptionsTests.cs @@ -11,7 +11,6 @@ public class StoreAndForwardOptionsTests var options = new StoreAndForwardOptions(); Assert.Equal("./data/store-and-forward.db", options.SqliteDbPath); - Assert.True(options.ReplicationEnabled); Assert.Equal(TimeSpan.FromSeconds(30), options.DefaultRetryInterval); Assert.Equal(50, options.DefaultMaxRetries); Assert.Equal(TimeSpan.FromSeconds(10), options.RetryTimerInterval); @@ -23,14 +22,12 @@ public class StoreAndForwardOptionsTests var options = new StoreAndForwardOptions { SqliteDbPath = "/custom/path.db", - ReplicationEnabled = false, DefaultRetryInterval = TimeSpan.FromMinutes(5), DefaultMaxRetries = 100, RetryTimerInterval = TimeSpan.FromSeconds(30) }; Assert.Equal("/custom/path.db", options.SqliteDbPath); - Assert.False(options.ReplicationEnabled); Assert.Equal(TimeSpan.FromMinutes(5), options.DefaultRetryInterval); Assert.Equal(100, options.DefaultMaxRetries); } diff --git a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardOptionsValidatorTests.cs b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardOptionsValidatorTests.cs index 346516fa..2a7da0e5 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardOptionsValidatorTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardOptionsValidatorTests.cs @@ -39,13 +39,20 @@ public class StoreAndForwardOptionsValidatorTests Assert.Contains("RetryTimerInterval", result.FailureMessage); } + /// + /// The inverse of the rule this replaces. Before LocalDb Phase 2 an empty + /// SqliteDbPath was rejected, because it was the live buffer file and an empty + /// value failed opaquely at first enqueue. The buffer now lives in the + /// consolidated LocalDb database, so the key is only the legacy migration + /// source and an empty value legitimately means "nothing to migrate" — a node + /// that has already migrated must be able to drop it. + /// [Fact] - public void EmptySqliteDbPath_IsRejected() + public void EmptySqliteDbPath_IsAccepted_BecauseThePathIsMigrationOnly() { var result = Validate(new StoreAndForwardOptions { SqliteDbPath = "" }); - Assert.True(result.Failed); - Assert.Contains("SqliteDbPath", result.FailureMessage); + Assert.False(result.Failed); } // ── R2 T9: sweep-tuning eager validation (N4) ── diff --git a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardReplicationTests.cs b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardReplicationTests.cs deleted file mode 100644 index 64e584fb..00000000 --- a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardReplicationTests.cs +++ /dev/null @@ -1,263 +0,0 @@ -using Microsoft.Data.Sqlite; -using Microsoft.Extensions.Logging.Abstractions; -using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; - -namespace ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests; - -/// -/// StoreAndForward-001: the active node must forward every buffer operation -/// (add / remove / park) to the standby via the ReplicationService, so a -/// failover does not lose the buffer. -/// -public class StoreAndForwardReplicationTests : IAsyncLifetime, IDisposable -{ - private readonly SqliteConnection _keepAlive; - private readonly StoreAndForwardStorage _storage; - private readonly StoreAndForwardService _service; - private readonly List _replicated = new(); - - public StoreAndForwardReplicationTests() - { - var connStr = $"Data Source=ReplTests_{Guid.NewGuid():N};Mode=Memory;Cache=Shared"; - _keepAlive = new SqliteConnection(connStr); - _keepAlive.Open(); - - _storage = new StoreAndForwardStorage(connStr, NullLogger.Instance); - - var options = new StoreAndForwardOptions - { - DefaultRetryInterval = TimeSpan.Zero, - DefaultMaxRetries = 1, - RetryTimerInterval = TimeSpan.FromMinutes(10), - ReplicationEnabled = true, - }; - - var replication = new ReplicationService(options, NullLogger.Instance); - replication.SetReplicationHandler(op => - { - lock (_replicated) _replicated.Add(op); - return Task.CompletedTask; - }); - - _service = new StoreAndForwardService( - _storage, options, NullLogger.Instance, replication); - } - - public async Task InitializeAsync() => await _storage.InitializeAsync(); - public Task DisposeAsync() => Task.CompletedTask; - public void Dispose() => _keepAlive.Dispose(); - - /// Replication is fire-and-forget (Task.Run); poll until the expected ops arrive. - private async Task> WaitForReplicationAsync(int count) - { - for (var i = 0; i < 100; i++) - { - lock (_replicated) - if (_replicated.Count >= count) return _replicated.ToList(); - await Task.Delay(20); - } - lock (_replicated) return _replicated.ToList(); - } - - [Fact] - public async Task BufferingAMessage_ReplicatesAnAddOperation() - { - // No handler registered → message is buffered → an Add is replicated. - var result = await _service.EnqueueAsync( - StoreAndForwardCategory.ExternalSystem, "api", """{}"""); - Assert.True(result.WasBuffered); - - var ops = await WaitForReplicationAsync(1); - Assert.Contains(ops, o => - o.OperationType == ReplicationOperationType.Add && o.MessageId == result.MessageId); - } - - [Fact] - public async Task SuccessfulRetry_ReplicatesARemoveOperation() - { - var calls = 0; - _service.RegisterDeliveryHandler(StoreAndForwardCategory.ExternalSystem, - _ => ++calls == 1 - ? throw new HttpRequestException("transient") - : Task.FromResult(true)); - - var result = await _service.EnqueueAsync( - StoreAndForwardCategory.ExternalSystem, "api", """{}"""); - await _service.RetryPendingMessagesAsync(); - - var ops = await WaitForReplicationAsync(2); - Assert.Contains(ops, o => o.OperationType == ReplicationOperationType.Add); - Assert.Contains(ops, o => - o.OperationType == ReplicationOperationType.Remove && o.MessageId == result.MessageId); - } - - [Fact] - public async Task ParkedMessage_ReplicatesAParkOperation() - { - _service.RegisterDeliveryHandler(StoreAndForwardCategory.ExternalSystem, - _ => throw new HttpRequestException("always fails")); - - var result = await _service.EnqueueAsync( - StoreAndForwardCategory.ExternalSystem, "api", """{}""", maxRetries: 1); - await _service.RetryPendingMessagesAsync(); - - var ops = await WaitForReplicationAsync(2); - Assert.Contains(ops, o => - o.OperationType == ReplicationOperationType.Park && o.MessageId == result.MessageId); - } - - /// - /// StoreAndForward-016: an operator discarding a parked message must replicate - /// a Remove so the standby's copy is also deleted (otherwise the discarded - /// message reappears in the parked list after a failover). - /// - [Fact] - public async Task DiscardingAParkedMessage_ReplicatesARemoveOperation() - { - _service.RegisterDeliveryHandler(StoreAndForwardCategory.ExternalSystem, - _ => throw new HttpRequestException("always fails")); - - var result = await _service.EnqueueAsync( - StoreAndForwardCategory.ExternalSystem, "api", """{}""", maxRetries: 1); - await _service.RetryPendingMessagesAsync(); // -> parked - await WaitForReplicationAsync(2); - - var discarded = await _service.DiscardParkedMessageAsync(result.MessageId); - Assert.True(discarded); - - var ops = await WaitForReplicationAsync(3); - Assert.Contains(ops, o => - o.OperationType == ReplicationOperationType.Remove && o.MessageId == result.MessageId); - } - - /// - /// StoreAndForward-016: an operator retrying a parked message must replicate a - /// Requeue so the standby's copy moves back to Pending (otherwise it stays - /// Parked on the standby and the operator's retry is lost across a failover). - /// - [Fact] - public async Task RetryingAParkedMessage_ReplicatesARequeueOperation() - { - _service.RegisterDeliveryHandler(StoreAndForwardCategory.ExternalSystem, - _ => throw new HttpRequestException("always fails")); - - var result = await _service.EnqueueAsync( - StoreAndForwardCategory.ExternalSystem, "api", """{}""", maxRetries: 1); - await _service.RetryPendingMessagesAsync(); // -> parked - await WaitForReplicationAsync(2); - - var retried = await _service.RetryParkedMessageAsync(result.MessageId); - Assert.True(retried); - - var ops = await WaitForReplicationAsync(3); - var requeue = ops.SingleOrDefault(o => - o.OperationType == ReplicationOperationType.Requeue && o.MessageId == result.MessageId); - Assert.NotNull(requeue); - Assert.NotNull(requeue!.Message); - Assert.Equal(StoreAndForwardMessageStatus.Pending, requeue.Message!.Status); - } - - /// - /// StoreAndForward-016: the standby applies a Requeue by moving its row back to - /// Pending with retry_count = 0, mirroring the active node's local state. - /// - [Fact] - public async Task ApplyReplicatedOperation_Requeue_MovesStandbyRowBackToPending() - { - var replication = new ReplicationService( - new StoreAndForwardOptions { ReplicationEnabled = true }, - NullLogger.Instance); - - var parked = new StoreAndForwardMessage - { - Id = "requeue1", - Category = StoreAndForwardCategory.ExternalSystem, - Target = "api", - PayloadJson = "{}", - RetryCount = 5, - MaxRetries = 1, - RetryIntervalMs = 0, - CreatedAt = DateTimeOffset.UtcNow, - Status = StoreAndForwardMessageStatus.Parked, - }; - await _storage.EnqueueAsync(parked); - - var requeued = new StoreAndForwardMessage - { - Id = parked.Id, - Category = parked.Category, - Target = parked.Target, - PayloadJson = parked.PayloadJson, - RetryCount = 0, - MaxRetries = parked.MaxRetries, - RetryIntervalMs = parked.RetryIntervalMs, - CreatedAt = parked.CreatedAt, - Status = StoreAndForwardMessageStatus.Pending, - }; - await replication.ApplyReplicatedOperationAsync( - new ReplicationOperation(ReplicationOperationType.Requeue, parked.Id, requeued), - _storage); - - var row = await _storage.GetMessageByIdAsync(parked.Id); - Assert.NotNull(row); - Assert.Equal(StoreAndForwardMessageStatus.Pending, row!.Status); - Assert.Equal(0, row.RetryCount); - } - - private static ReplicationService NewReplicationService() => - new(new StoreAndForwardOptions { ReplicationEnabled = true }, - NullLogger.Instance); - - private static StoreAndForwardMessage NewMessage(string id) => new() - { - Id = id, - Category = StoreAndForwardCategory.ExternalSystem, - Target = "api", - PayloadJson = "{}", - RetryCount = 0, - MaxRetries = 1, - RetryIntervalMs = 0, - CreatedAt = DateTimeOffset.UtcNow, - Status = StoreAndForwardMessageStatus.Pending, - }; - - /// - /// A Park (or Requeue) whose original Add was lost — e.g. the Add's - /// fire-and-forget replication dropped — must still materialise the row on the - /// standby. The full message rides in the operation, so an upsert self-heals; - /// a blind UPDATE would affect 0 rows and the row would be gone forever. - /// - [Fact] - public async Task ApplyReplicatedPark_WhenAddWasLost_MaterializesTheParkedRow() - { - var service = NewReplicationService(); - var msg = NewMessage("lost-add-1"); // never Added on this (standby) storage - msg.Status = StoreAndForwardMessageStatus.Parked; - - await service.ApplyReplicatedOperationAsync( - new ReplicationOperation(ReplicationOperationType.Park, msg.Id, msg), _storage); - - var row = await _storage.GetMessageByIdAsync("lost-add-1"); - Assert.NotNull(row); // pre-fix: blind UPDATE affected 0 rows, row is gone forever - Assert.Equal(StoreAndForwardMessageStatus.Parked, row!.Status); - } - - /// - /// A duplicate Add (e.g. after Task 21's anti-entropy resync re-issues an Add - /// the standby already holds) must not violate the PK — the upsert applies - /// newest-wins instead of throwing. - /// - [Fact] - public async Task ApplyReplicatedAdd_Twice_IsIdempotent_NewestWins() - { - var service = NewReplicationService(); - var msg = NewMessage("dup-add"); - await service.ApplyReplicatedOperationAsync( - new ReplicationOperation(ReplicationOperationType.Add, msg.Id, msg), _storage); - msg.RetryCount = 3; - await service.ApplyReplicatedOperationAsync( // pre-fix: SqliteException PK violation - new ReplicationOperation(ReplicationOperationType.Add, msg.Id, msg), _storage); - - Assert.Equal(3, (await _storage.GetMessageByIdAsync("dup-add"))!.RetryCount); - } -} diff --git a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardSchemaTests.cs b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardSchemaTests.cs new file mode 100644 index 00000000..b4279e5a --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardSchemaTests.cs @@ -0,0 +1,113 @@ +using Microsoft.Data.Sqlite; + +namespace ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests; + +/// +/// Pins as the single owner of the +/// sf_messages DDL. The Host applies this to a LocalDb-managed connection +/// before RegisterReplicated installs the capture triggers, so the shape it +/// produces is the shape that replicates. +/// +public class StoreAndForwardSchemaTests +{ + [Fact] + public void Apply_IsIdempotent_AndCreatesEveryColumn() + { + var path = Path.Combine(Path.GetTempPath(), $"sf-schema-{Guid.NewGuid():N}.db"); + try + { + using var connection = new SqliteConnection($"Data Source={path}"); + connection.Open(); + + StoreAndForwardSchema.Apply(connection); + StoreAndForwardSchema.Apply(connection); // second run must not throw + + using var cmd = connection.CreateCommand(); + cmd.CommandText = "SELECT name FROM pragma_table_info('sf_messages')"; + var columns = new List(); + using var reader = cmd.ExecuteReader(); + while (reader.Read()) columns.Add(reader.GetString(0)); + + // 12 original + 4 additive + Assert.Equal(16, columns.Count); + Assert.Contains("last_attempt_at_ms", columns); + Assert.Contains("parent_execution_id", columns); + } + finally + { + SqliteConnection.ClearAllPools(); + File.Delete(path); + } + } + + /// + /// The additive columns exist to upgrade a database created by an older build in + /// place. Asserting only against a freshly-created table would pass even if every + /// ALTER were deleted, because CREATE TABLE already lists them — so + /// this starts from the pre-upgrade shape and proves the ALTER path runs. + /// + [Fact] + public void Apply_UpgradesALegacyTable_WithoutLosingRows() + { + var path = Path.Combine(Path.GetTempPath(), $"sf-schema-legacy-{Guid.NewGuid():N}.db"); + try + { + using var connection = new SqliteConnection($"Data Source={path}"); + connection.Open(); + + // The 12-column shape as it existed before execution_id / source_script / + // parent_execution_id / last_attempt_at_ms were added. + using (var legacy = connection.CreateCommand()) + { + legacy.CommandText = """ + CREATE TABLE sf_messages ( + id TEXT PRIMARY KEY, + category INTEGER NOT NULL, + target TEXT NOT NULL, + payload_json TEXT NOT NULL, + retry_count INTEGER NOT NULL DEFAULT 0, + max_retries INTEGER NOT NULL DEFAULT 50, + retry_interval_ms INTEGER NOT NULL DEFAULT 30000, + created_at TEXT NOT NULL, + last_attempt_at TEXT, + status INTEGER NOT NULL DEFAULT 0, + last_error TEXT, + origin_instance TEXT + ); + INSERT INTO sf_messages (id, category, target, payload_json, created_at, last_attempt_at) + VALUES ('legacy-1', 0, 'target-a', '{}', '2026-01-01T00:00:00Z', '2026-01-02T03:04:05Z'); + """; + legacy.ExecuteNonQuery(); + } + + StoreAndForwardSchema.Apply(connection); + + using var check = connection.CreateCommand(); + check.CommandText = + "SELECT COUNT(*) FROM pragma_table_info('sf_messages') WHERE name = 'last_attempt_at_ms'"; + Assert.Equal(1, Convert.ToInt32(check.ExecuteScalar())); + + // The pre-existing row survives and its epoch-ms sibling is backfilled from + // the authoritative text timestamp. + using var backfilled = connection.CreateCommand(); + backfilled.CommandText = + "SELECT last_attempt_at_ms FROM sf_messages WHERE id = 'legacy-1'"; + var ms = Convert.ToInt64(backfilled.ExecuteScalar()); + var expected = DateTimeOffset.Parse("2026-01-02T03:04:05Z").ToUnixTimeMilliseconds(); + + // Tolerance of 1 ms, not sloppiness: the backfill derives epoch-ms from + // julianday(), whose double-precision day-fraction cannot represent every + // millisecond exactly, so it lands 1 ms low here. That rounding is + // pre-existing production behaviour carried over verbatim from + // StoreAndForwardStorage — the column is a sweep-ordering hint, with the + // ISO-8601 text column remaining authoritative for reads. Asserting exact + // equality would pin a precision the implementation never had. + Assert.InRange(ms, expected - 1, expected + 1); + } + finally + { + SqliteConnection.ClearAllPools(); + File.Delete(path); + } + } +} diff --git a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardServiceTests.cs b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardServiceTests.cs index 3c7a4c94..434b3d60 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardServiceTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardServiceTests.cs @@ -1,6 +1,6 @@ -using Microsoft.Data.Sqlite; using Microsoft.Extensions.Logging.Abstractions; using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; +using ZB.MOM.WW.ScadaBridge.TestSupport; namespace ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests; @@ -9,20 +9,17 @@ namespace ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests; /// public class StoreAndForwardServiceTests : IAsyncLifetime, IDisposable { - private readonly SqliteConnection _keepAlive; + private readonly TestLocalDb _localDb; private readonly StoreAndForwardStorage _storage; private readonly StoreAndForwardService _service; private readonly StoreAndForwardOptions _options; - private readonly List _extraKeepAlives = new(); + private readonly List _extraLocalDbs = new(); public StoreAndForwardServiceTests() { - var dbName = $"SvcTests_{Guid.NewGuid():N}"; - var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared"; - _keepAlive = new SqliteConnection(connStr); - _keepAlive.Open(); + _localDb = TestLocalDb.CreateTemp("SvcTests"); - _storage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + _storage = new StoreAndForwardStorage(_localDb.Db, NullLogger.Instance); _options = new StoreAndForwardOptions { @@ -41,23 +38,29 @@ public class StoreAndForwardServiceTests : IAsyncLifetime, IDisposable public void Dispose() { - _keepAlive.Dispose(); - foreach (var c in _extraKeepAlives) c.Dispose(); + DisposeLocalDb(_localDb); + foreach (var db in _extraLocalDbs) DisposeLocalDb(db); + } + + /// Disposes a local database, then removes its file and WAL sidecars. + private static void DisposeLocalDb(TestLocalDb localDb) + { + var path = localDb.Path; + localDb.Dispose(); + TestLocalDb.DeleteFiles(path); } /// - /// Builds a fresh service over its own in-memory (shared-cache) SQLite so a test - /// can call StartAsync without racing the shared _service's timer. The - /// keep-alive connection is tracked for disposal. + /// Builds a fresh service over its own local database so a test can call + /// StartAsync without racing the shared _service's timer. The database + /// is tracked for disposal. /// private StoreAndForwardService CreateService(TimeSpan? retryTimerInterval = null) { - var connStr = $"Data Source=DeferTests_{Guid.NewGuid():N};Mode=Memory;Cache=Shared"; - var keepAlive = new SqliteConnection(connStr); - keepAlive.Open(); - _extraKeepAlives.Add(keepAlive); + var localDb = TestLocalDb.CreateTemp("DeferTests"); + _extraLocalDbs.Add(localDb); - var storage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + var storage = new StoreAndForwardStorage(localDb.Db, NullLogger.Instance); storage.InitializeAsync().GetAwaiter().GetResult(); var options = new StoreAndForwardOptions @@ -557,12 +560,10 @@ public class StoreAndForwardServiceTests : IAsyncLifetime, IDisposable // Build a service whose timer fires almost immediately, with a handler // that pauses in the middle of delivery so we can observe StopAsync's // wait behaviour. - var dbName = $"StopWait_{Guid.NewGuid():N}"; - var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared"; - using var keepAlive = new SqliteConnection(connStr); - keepAlive.Open(); + var localDb = TestLocalDb.CreateTemp("StopWait"); + _extraLocalDbs.Add(localDb); - var storage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + var storage = new StoreAndForwardStorage(localDb.Db, NullLogger.Instance); await storage.InitializeAsync(); var options = new StoreAndForwardOptions diff --git a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardSiteEventTests.cs b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardSiteEventTests.cs index 2c3dcea7..b21acfd8 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardSiteEventTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardSiteEventTests.cs @@ -1,8 +1,8 @@ using System.Collections.Concurrent; -using Microsoft.Data.Sqlite; using Microsoft.Extensions.Logging.Abstractions; using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; using ZB.MOM.WW.ScadaBridge.SiteEventLogging; +using ZB.MOM.WW.ScadaBridge.TestSupport; namespace ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests; @@ -34,7 +34,7 @@ public class StoreAndForwardSiteEventTests : IAsyncLifetime, IDisposable public long FailedWriteCount => 0; } - private readonly SqliteConnection _keepAlive; + private readonly TestLocalDb _localDb; private readonly StoreAndForwardStorage _storage; private readonly StoreAndForwardOptions _options; private readonly FakeSiteEventLogger _siteLog = new(); @@ -42,12 +42,9 @@ public class StoreAndForwardSiteEventTests : IAsyncLifetime, IDisposable public StoreAndForwardSiteEventTests() { - var dbName = $"SiteEvt_{Guid.NewGuid():N}"; - var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared"; - _keepAlive = new SqliteConnection(connStr); - _keepAlive.Open(); + _localDb = TestLocalDb.CreateTemp("SiteEvt"); - _storage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + _storage = new StoreAndForwardStorage(_localDb.Db, NullLogger.Instance); _options = new StoreAndForwardOptions { DefaultRetryInterval = TimeSpan.Zero, @@ -57,13 +54,18 @@ public class StoreAndForwardSiteEventTests : IAsyncLifetime, IDisposable _service = new StoreAndForwardService( _storage, _options, NullLogger.Instance, - replication: null, cachedCallObserver: null, siteId: "site-a", + cachedCallObserver: null, siteId: "site-a", siteEventLogger: _siteLog); } public async Task InitializeAsync() => await _storage.InitializeAsync(); public Task DisposeAsync() => Task.CompletedTask; - public void Dispose() => _keepAlive.Dispose(); + public void Dispose() + { + var path = _localDb.Path; + _localDb.Dispose(); + TestLocalDb.DeleteFiles(path); + } [Fact] public async Task BufferForRetry_ExternalSystem_EmitsStoreAndForwardSiteEvent() diff --git a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardStorageTests.cs b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardStorageTests.cs index ad6057fb..8a2f725a 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardStorageTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardStorageTests.cs @@ -1,27 +1,29 @@ using Microsoft.Data.Sqlite; using Microsoft.Extensions.Logging.Abstractions; using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; +using ZB.MOM.WW.ScadaBridge.TestSupport; namespace ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests; /// /// WP-9: Tests for SQLite persistence layer. -/// Uses in-memory SQLite with a kept-alive connection for test isolation. /// +/// +/// Backed by a real temp-file LocalDb rather than the shared-cache in-memory database +/// this class used before: now takes +/// ILocalDb, and LocalDb has no in-memory mode (LocalDb:Path is a +/// filesystem path). Isolation still comes from a fresh database per test — xUnit +/// constructs one instance per test — it is just a file now. +/// public class StoreAndForwardStorageTests : IAsyncLifetime, IDisposable { - private readonly SqliteConnection _keepAlive; + private readonly TestLocalDb _localDb; private readonly StoreAndForwardStorage _storage; - private readonly string _dbName; public StoreAndForwardStorageTests() { - _dbName = $"StorageTests_{Guid.NewGuid():N}"; - var connStr = $"Data Source={_dbName};Mode=Memory;Cache=Shared"; - // Keep one connection alive so the in-memory DB persists - _keepAlive = new SqliteConnection(connStr); - _keepAlive.Open(); - _storage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + _localDb = TestLocalDb.CreateTemp("StorageTests"); + _storage = new StoreAndForwardStorage(_localDb.Db, NullLogger.Instance); } public async Task InitializeAsync() => await _storage.InitializeAsync(); @@ -30,7 +32,11 @@ public class StoreAndForwardStorageTests : IAsyncLifetime, IDisposable public void Dispose() { - _keepAlive.Dispose(); + // Dispose first — the master connection anchors the WAL, so the sidecars + // cannot be removed while it is open. + var path = _localDb.Path; + _localDb.Dispose(); + TestLocalDb.DeleteFiles(path); } [Fact] @@ -360,9 +366,8 @@ public class StoreAndForwardStorageTests : IAsyncLifetime, IDisposable // schema by dropping the table and recreating it without the columns, // inserting directly, then running InitializeAsync (which ALTER-adds // the columns) and reading the row back. - await using (var setup = new SqliteConnection($"Data Source={_dbName};Mode=Memory;Cache=Shared")) + await using (var setup = _localDb.Db.CreateConnection()) { - await setup.OpenAsync(); await using var drop = setup.CreateCommand(); drop.CommandText = @" DROP TABLE IF EXISTS sf_messages; @@ -415,9 +420,8 @@ public class StoreAndForwardStorageTests : IAsyncLifetime, IDisposable bad.LastAttemptAt = null; // due immediately await _storage.EnqueueAsync(bad); - await using (var conn = new SqliteConnection($"Data Source={_dbName};Mode=Memory;Cache=Shared")) + await using (var conn = _localDb.Db.CreateConnection()) { - await conn.OpenAsync(); await using var corrupt = conn.CreateCommand(); corrupt.CommandText = "UPDATE sf_messages SET execution_id = 'not-a-guid' WHERE id = 'bad1';"; @@ -514,9 +518,8 @@ public class StoreAndForwardStorageTests : IAsyncLifetime, IDisposable // ExecutionId rollout) by recreating the table without // parent_execution_id, inserting directly, then running InitializeAsync // which ALTER-adds the column. - await using (var setup = new SqliteConnection($"Data Source={_dbName};Mode=Memory;Cache=Shared")) + await using (var setup = _localDb.Db.CreateConnection()) { - await setup.OpenAsync(); await using var drop = setup.CreateCommand(); drop.CommandText = @" DROP TABLE IF EXISTS sf_messages; @@ -569,9 +572,8 @@ public class StoreAndForwardStorageTests : IAsyncLifetime, IDisposable bad.LastAttemptAt = null; // due immediately await _storage.EnqueueAsync(bad); - await using (var conn = new SqliteConnection($"Data Source={_dbName};Mode=Memory;Cache=Shared")) + await using (var conn = _localDb.Db.CreateConnection()) { - await conn.OpenAsync(); await using var corrupt = conn.CreateCommand(); corrupt.CommandText = "UPDATE sf_messages SET parent_execution_id = 'not-a-guid' WHERE id = 'pbad1';"; @@ -630,57 +632,46 @@ public class StoreAndForwardStorageTests : IAsyncLifetime, IDisposable }; } - [Fact] - public async Task InitializeAsync_FileInMissingDirectory_CreatesDirectory() - { - // SQLite creates the database file on demand but not its parent directory; - // the storage must create the directory itself or OpenAsync fails with - // "unable to open database file" (the cause of the SiteActorPathTests failures). - var directory = Path.Combine(Path.GetTempPath(), "sf-storage-test-" + Guid.NewGuid().ToString("N")); - var dbPath = Path.Combine(directory, "store-and-forward.db"); - Assert.False(Directory.Exists(directory)); - - try - { - var storage = new StoreAndForwardStorage( - $"Data Source={dbPath}", NullLogger.Instance); - - await storage.InitializeAsync(); - - Assert.True(Directory.Exists(directory)); - Assert.True(File.Exists(dbPath)); - } - finally - { - if (Directory.Exists(directory)) - Directory.Delete(directory, recursive: true); - } - } + // ── Invariants that moved owner ── + // + // Directory creation and WAL journal mode used to be this storage class's own job + // (EnsureDatabaseDirectoryExists / an explicit PRAGMA in InitializeAsync). Both moved + // out when the store stopped owning its file. They landed in DIFFERENT places, so the + // coverage split rather than moving wholesale: + // + // * WAL is genuinely LocalDb's — it sets the journal mode on the database it owns. + // The test below still asserts it, now against the LocalDb-backed store. + // + // * Directory creation was NOT LocalDb's when Phase 2 landed: the library did not + // create the parent directory yet opened the file eagerly, so a missing directory + // was a hard boot failure, and the Host re-established the guarantee with a + // SiteLocalDbDirectory shim. LocalDb 0.1.1 took the guarantee back into the + // library, so the shim is gone — but either way it is not this class's to assert, + // and SiteLocalDbDirectoryTests pins the outcome for the site host. [Fact] - public async Task Initialize_EnablesWalJournalMode_OnFileDatabase() + public async Task Initialize_LeavesTheDatabaseInWalJournalMode() { // WAL lets the retry-sweep lanes, script enqueues, and standby replication - // applies read/write concurrently without "database is locked". journal_mode - // is persistent + file-scoped, so a fresh connection observes it. - var directory = Path.Combine(Path.GetTempPath(), "sf-wal-test-" + Guid.NewGuid().ToString("N")); - var path = Path.Combine(directory, "wal-test.db"); + // applies read/write concurrently without "database is locked". + using var localDb = TestLocalDb.CreateTemp("sf-wal-test"); + var path = localDb.Path; try { var storage = new StoreAndForwardStorage( - $"Data Source={path}", NullLogger.Instance); + localDb.Db, NullLogger.Instance); await storage.InitializeAsync(); - await using var conn = new SqliteConnection($"Data Source={path}"); - await conn.OpenAsync(); + // journal_mode is persistent + file-scoped, so any connection observes it. + await using var conn = localDb.Db.CreateConnection(); await using var cmd = conn.CreateCommand(); cmd.CommandText = "PRAGMA journal_mode"; Assert.Equal("wal", (string)(await cmd.ExecuteScalarAsync())!); } finally { - if (Directory.Exists(directory)) - Directory.Delete(directory, recursive: true); + localDb.Dispose(); + TestLocalDb.DeleteFiles(path); } } @@ -731,8 +722,7 @@ public class StoreAndForwardStorageTests : IAsyncLifetime, IDisposable private async Task ExecRawAsync(string sql, params (string, object)[] parameters) { - await using var conn = new SqliteConnection($"Data Source={_dbName};Mode=Memory;Cache=Shared"); - await conn.OpenAsync(); + await using var conn = _localDb.Db.CreateConnection(); await using var cmd = conn.CreateCommand(); cmd.CommandText = sql; foreach (var (name, value) in parameters) cmd.Parameters.AddWithValue(name, value); @@ -741,8 +731,7 @@ public class StoreAndForwardStorageTests : IAsyncLifetime, IDisposable private async Task ScalarRawAsync(string sql) { - await using var conn = new SqliteConnection($"Data Source={_dbName};Mode=Memory;Cache=Shared"); - await conn.OpenAsync(); + await using var conn = _localDb.Db.CreateConnection(); await using var cmd = conn.CreateCommand(); cmd.CommandText = sql; var result = await cmd.ExecuteScalarAsync(); @@ -780,16 +769,15 @@ public class StoreAndForwardStorageTests : IAsyncLifetime, IDisposable Assert.True(truncated); // a third row exists beyond the limit } - [Fact] - public async Task ReplaceAll_SwapsTheEntireBuffer_Atomically() - { - await _storage.EnqueueAsync(NewMsg("stale")); - await _storage.ReplaceAllAsync(new[] { NewMsg("fresh-1"), NewMsg("fresh-2") }); - - Assert.Null(await _storage.GetMessageByIdAsync("stale")); - Assert.NotNull(await _storage.GetMessageByIdAsync("fresh-1")); - Assert.NotNull(await _storage.GetMessageByIdAsync("fresh-2")); - } + // ReplaceAll_SwapsTheEntireBuffer_Atomically was DELETED with ReplaceAllAsync in + // LocalDb Phase 2, and deliberately not replaced. It asserted a destructive + // delete-all-then-insert-all, which was the standby's anti-entropy apply: a resync + // overwrote the standby's copy with the active node's snapshot. sf_messages is now a + // replicated table, so a mass DELETE would be CAPTURED and shipped to the peer — the + // method is not merely unused, it is unsafe to keep. LocalDb's own snapshot resync + // merges per row under last-writer-wins and never deletes, which is what + // LocalDbConfigConvergenceTests.ANodeWithNewerLocalRows_KeepsThem_WhenAPeerSnapshotArrives + // now covers. // ── Task 23: oldest-parked-age health signal ── diff --git a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests.csproj b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests.csproj index ad27b629..e6ebb6b8 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests.csproj +++ b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests.csproj @@ -27,6 +27,7 @@ - + + diff --git a/tests/ZB.MOM.WW.ScadaBridge.TestSupport/TestLocalDb.cs b/tests/ZB.MOM.WW.ScadaBridge.TestSupport/TestLocalDb.cs new file mode 100644 index 00000000..46a68f1f --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.TestSupport/TestLocalDb.cs @@ -0,0 +1,91 @@ +using Microsoft.Extensions.Configuration; +using Microsoft.Extensions.DependencyInjection; +using ZB.MOM.WW.LocalDb; + +namespace ZB.MOM.WW.ScadaBridge.TestSupport; + +/// +/// A real over a temp file, for tests that construct a +/// LocalDb-backed store directly. +/// +/// +/// +/// The stores take rather than a connection string, and there is +/// no in-memory mode — LocalDbOptions.Path is a filesystem path. A real one is +/// used rather than a stub on purpose: connections handed out by ILocalDb carry +/// the zb_hlc_next() UDF that the capture triggers call, so a stub returning a +/// bare SqliteConnection would pass these tests while failing closed the moment +/// the table is registered for replication. +/// +/// +/// No onReady callback and no RegisterReplicated: each store's own schema +/// initialization creates its table, which is what keeps a directly-constructed store +/// self-sufficient. Registration is the host's job (SiteLocalDbSetup.OnReady). +/// +/// +/// This lives in a shared support project rather than being copied per test project: +/// seven test projects construct stores that now take , and +/// duplicating the fixture would mean seven places to keep the WAL-sidecar cleanup and +/// the "real, not stubbed" rationale correct. +/// +/// +public sealed class TestLocalDb : IDisposable +{ + private readonly ServiceProvider _provider; + + private TestLocalDb(ServiceProvider provider, ILocalDb db) + { + _provider = provider; + Db = db; + } + + /// The live local database. + public ILocalDb Db { get; } + + /// Opens a local database at . + public static TestLocalDb Create(string databasePath) + { + var configuration = new ConfigurationBuilder() + .AddInMemoryCollection(new Dictionary + { + ["LocalDb:Path"] = databasePath, + }) + .Build(); + + var provider = new ServiceCollection() + .AddZbLocalDb(configuration) + .BuildServiceProvider(); + + return new TestLocalDb(provider, provider.GetRequiredService()); + } + + /// + /// Creates a local database at a fresh temp path. Dispose the instance, then call + /// with to clean up. + /// + public static TestLocalDb CreateTemp(string prefix = "localdb") + { + var path = System.IO.Path.Combine( + System.IO.Path.GetTempPath(), $"{prefix}-{Guid.NewGuid():N}.db"); + var db = Create(path); + db.Path = path; + return db; + } + + /// The filesystem path, when created via . + public string Path { get; private set; } = string.Empty; + + /// + /// Deletes and its WAL sidecars. Call only after the + /// owning is disposed — the master connection anchors the WAL. + /// + public static void DeleteFiles(string databasePath) + { + foreach (var suffix in new[] { "", "-wal", "-shm" }) + { + try { File.Delete(databasePath + suffix); } catch { /* best effort */ } + } + } + + public void Dispose() => _provider.Dispose(); +} diff --git a/tests/ZB.MOM.WW.ScadaBridge.TestSupport/ZB.MOM.WW.ScadaBridge.TestSupport.csproj b/tests/ZB.MOM.WW.ScadaBridge.TestSupport/ZB.MOM.WW.ScadaBridge.TestSupport.csproj new file mode 100644 index 00000000..dc5e2997 --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.TestSupport/ZB.MOM.WW.ScadaBridge.TestSupport.csproj @@ -0,0 +1,29 @@ + + + + + net10.0 + enable + enable + true + false + + + + + + + + + + + +