From 1556ae04e4982ed8a04bbaa0113c3c1d55fc5b3f Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sun, 19 Jul 2026 02:34:38 -0400 Subject: [PATCH 01/55] docs(localdb): amend Phase 1 plan - split Task 5 (event-log id change) Execution review against the code found the integer site_events.id is load-bearing in three places the original Task 5 did not list: - keyset cursor EventLogQueryService.cs:70,141,153 (id > $afterId) - storage-cap purge EventLogPurgeService.cs:164 (ORDER BY id ASC) - wire contracts EventLogEntry.Id, both ContinuationTokens (long) cross the site<->central Akka boundary A GUID PK against the existing cursor silently drops rows, and against the purge deletes arbitrary rows instead of the oldest. Task 5 is therefore split into 5a (writer, standard) and 5b (read path + DTOs, high-risk), with 5b's full file set spelled out and Task 10 gated on it. Also recorded: Task 4's real registration site, and that both legacy DB paths are CWD-relative and unset in docker today (so they live outside the data volume and are already lost on container recreate - Phase 1 fixes that). Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts --- .../2026-07-19-localdb-adoption-phase1.md | 72 +++++++++++++++---- ...7-19-localdb-adoption-phase1.md.tasks.json | 13 +++- 2 files changed, 71 insertions(+), 14 deletions(-) diff --git a/docs/plans/2026-07-19-localdb-adoption-phase1.md b/docs/plans/2026-07-19-localdb-adoption-phase1.md index a3a93254..c974d352 100644 --- a/docs/plans/2026-07-19-localdb-adoption-phase1.md +++ b/docs/plans/2026-07-19-localdb-adoption-phase1.md @@ -18,8 +18,8 @@ | 1 | Task 1 (packages) | | 2 | Task 2 (schema helpers) | | 3 | Task 3 (composition root) | -| 4 | Task 4 (tracking store) ∥ Task 5 (event logger) ∥ Task 7 (replication wiring) | -| 5 | Task 6 (migrator) ∥ Task 8 (auth interceptor) ∥ Task 9 (health signal) | +| 4 | Task 4 (tracking store) ∥ Task 5a (event writer) ∥ Task 7 (replication wiring) | +| 5 | Task 5b (event read path) ∥ Task 6 (migrator) ∥ Task 8 (auth interceptor) ∥ Task 9 (health signal) | | 6 | Task 10 (convergence test) ∥ Task 11 (rig config) | | 7 | Task 12 (live gate — serial, on the real rig) | | 8 | Task 13 (docs) ∥ Task 14 (Phase 2 planning) | @@ -181,8 +181,9 @@ services.AddZbLocalDb(config, SiteLocalDbSetup.OnReady); **Files:** - Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Tracking/OperationTrackingStore.cs` -- Modify: wherever `AddSiteRuntime` registers the store (follow `services.AddSiteRuntime` from `SiteServiceRegistration.cs:51` into the SiteRuntime SCE) +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/ServiceCollectionExtensions.cs:74` (the `IOperationTrackingStore` registration; `AddSiteRuntime`'s connection-string parameter becomes vestigial for this store — leave the parameter alone, it still feeds the site config DB) - Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Tracking/` (existing store tests) +- Test: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CompositionRootTests.cs:433` (`Site_Resolves_IOperationTrackingStore` — it overrides the connection string to a temp path because the store opens eagerly in its ctor; that override changes shape once the store takes `ILocalDb`) **Step 1:** Constructor gains `ILocalDb localDb`; `_writeConnection = localDb.CreateConnection()` replaces `new SqliteConnection(_connectionString)` + `Open()` (verified in the lib, `SqliteLocalDb.cs:83-98`: CreateConnection returns an **open** connection with per-connection pragmas — `synchronous`, `busy_timeout`, `foreign_keys=ON` — and the `zb_hlc_next` UDF registered; do NOT call `Open()` on it again). Reader paths that open ad-hoc connections from `_connectionString` switch to `localDb.CreateConnection()` too (reads don't fire triggers, but one connection source is one less config knob). `OperationTrackingOptions.ConnectionString` becomes unused by the store — keep the options class but drop the store's dependency; leave a validator relaxation only if the DI-graph test demands it. @@ -194,25 +195,70 @@ services.AddZbLocalDb(config, SiteLocalDbSetup.OnReady); --- -### Task 5: Rewire SiteEventLogger + EventLogQueryService (GUID ids) +### Task 5a: Rewire SiteEventLogger onto ILocalDb + mint GUID ids **Classification:** standard -**Estimated implement time:** ~5 min +**Estimated implement time:** ~4 min **Parallelizable with:** Task 4, Task 7 +> **Amended 2026-07-19 (execution review).** The original Task 5 said "any `ORDER BY id` +> becomes `ORDER BY timestamp`" and listed 3 files. Code recon found the integer `id` is +> load-bearing in three separate places beyond the writer — a keyset cursor, the +> storage-cap purge, and `long`-typed DTOs that cross the site↔central Akka boundary. +> Task 5 is therefore split: **5a = the writer** (this task, self-contained), **5b = the +> read/purge/contract change** (high-risk, its own task). Do not fold 5b back into 5a. + **Files:** - Modify: `src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs` -- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/EventLogQueryService.cs` - Modify: `src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/ServiceCollectionExtensions.cs` (AddSiteEventLogging registration) -- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/` (existing suite) +- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/` (writer-path tests only) -**Step 1:** `SiteEventLogger` constructor gains `ILocalDb localDb`; its single owned `_connection` becomes `localDb.CreateConnection()`. Keep the `connectionStringOverride` test seam by preferring the override when non-null (tests may still use raw files for non-replicated scenarios) — but note overridden connections lack the HLC UDF, so tests that write must use the ILocalDb path once tables are registered; simplest is to switch the test fixture to a real `ILocalDb` and delete the override parameter if nothing else uses it. +**Step 1:** `SiteEventLogger` constructor gains `ILocalDb localDb`; its single owned `_connection` (`SiteEventLogger.cs:34,62-65`) becomes `localDb.CreateConnection()` — already open, pragma-configured, HLC-UDF-registered; do **not** call `Open()` again. Delete the `connectionStringOverride` seam (`:50`) rather than preserving it: a raw connection lacks `zb_hlc_next()`, so any test writing through the override against a registered table now **fails closed**. Switch the test fixture to a real `ILocalDb` over a temp file. -**Step 2:** The insert statement gains an `id` column bound to `Guid.NewGuid().ToString("N")` (minted in the writer loop per event). `EventLogQueryService` (read-only) switches its connection source the same way; any `ORDER BY id` becomes `ORDER BY timestamp` (GUIDs don't sort chronologically — check the query paths and the UI paging contract). +**Step 2:** Preserve the `internal WithConnection` seam (`:104`, `:123`) — `EventLogQueryService` and `EventLogPurgeService` both depend on it deliberately (their ctors take the concrete `SiteEventLogger`, not the interface). Its body changes connection source only; the signature and locking semantics stay identical, so 5b's consumers keep compiling. -**Step 3:** Fix remaining writer-path tests from Task 2. Full `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests` — PASS. +**Step 3:** The INSERT (`:237-240`) gains an `id` column bound to `Guid.NewGuid().ToString("N")`, minted per event in the writer loop. `ISiteEventLogger.LogEventAsync`'s signature is unchanged — all 18 fire-and-forget call sites are untouched. -**Step 4:** Commit. +**Step 4:** Fix writer-path tests from Task 2. Run `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests` — writer tests PASS. Query/purge tests may be red here; 5b closes them. Note that in the commit message. + +**Step 5:** Commit. + +--- + +### Task 5b: Migrate the event-log read path off integer ids + +**Classification:** high-risk (cross-cluster data contract + silent-data-loss paging bug) +**Estimated implement time:** ~5 min +**Parallelizable with:** Task 6, Task 8, Task 9 +**Blocked by:** Task 5a + +The GUID PK breaks three integer-id assumptions. All three must move together — shipping any one alone is a live bug. + +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/EventLogQueryService.cs` (cursor `:68-71`, `ORDER BY id ASC` `:141`, `GetInt64(0)` `:153`, token mint `:181`) +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/EventLogPurgeService.cs` (storage-cap delete `:162-166`) +- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Messages/RemoteQuery/EventLogQueryResponse.cs` (`EventLogEntry.Id` `long`→`string`; `ContinuationToken` `long?`→`string?`) +- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Messages/RemoteQuery/EventLogQueryRequest.cs` (`ContinuationToken` `long?`→`string?`) +- Modify: `src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs:3127` (passes `null` today — verify it still compiles) +- Modify: `src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/EventLogs.razor:247,268,271` (`_continuationToken` type) +- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/` (query + purge) + +**Step 1: Write the failing tests first.** +- Paging: insert N events with known timestamps, page through with `PageSize` < N, assert **every** row is returned exactly once across pages and in chronological order. This test fails loudly against a naive `id > $afterId` GUID cursor — that is the point. +- Ordering: rows come back oldest-first regardless of GUID values. +- Purge: with the storage cap tripped, the **oldest** rows are deleted, not arbitrary ones. + +**Step 2:** Run — FAIL. + +**Step 3: Implement.** +- **Cursor** → composite keyset on `(timestamp, id)`, since `timestamp` alone is not unique: + `WHERE (timestamp > $afterTs) OR (timestamp = $afterTs AND id > $afterId)`, `ORDER BY timestamp ASC, id ASC`. The continuation token encodes both — use `"{timestamp}|{id}"` as the `string?` token; parse defensively and treat a malformed token as "start from the beginning" rather than throwing. +- **Purge** (`:162-166`) → `ORDER BY timestamp ASC` instead of `ORDER BY id ASC`. (The retention purge at `:128` already filters on `timestamp` and is unaffected.) +- **DTOs** → `Id` and both `ContinuationToken`s become `string`/`string?`; reader uses `GetString(0)`. + +**Step 4: Wire-compat check.** These DTOs cross the site↔central Akka boundary (`SiteCommunicationActor.cs:193` → `CommunicationService.cs:304` → ManagementActor / CentralUI). Both sides ship in the same deployable and the rig redeploys as a unit, so no rolling-upgrade shim is needed — **but state that explicitly in the commit message**, and confirm no serializer manifest/binding pins these types by name+type (grep the Akka serialization setup). If a binding exists, stop and surface it. + +**Step 5:** `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests` + full solution build (0 warnings) — PASS. Commit. --- @@ -228,7 +274,8 @@ services.AddZbLocalDb(config, SiteLocalDbSetup.OnReady); - Test: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbLegacyMigratorTests.cs` Semantics (write the tests first, one per bullet): -- Legacy files located from the OLD config keys (`ScadaBridge:OperationTracking:ConnectionString`, `ScadaBridge:SiteEventLog` DatabasePath); missing file ⇒ no-op. +- Legacy files located from the OLD config keys (`ScadaBridge:OperationTracking:ConnectionString`, `ScadaBridge:SiteEventLog:DatabasePath`); missing file ⇒ no-op. +- **Both keys are unset in every docker appsettings today**, so they fall back to the code defaults `"Data Source=site-tracking.db"` (`OperationTrackingOptions.cs:14`) and `"site_events.db"` (`SiteEventLogOptions.cs:10`) — **CWD-relative**, i.e. `/app/site-tracking.db` and `/app/site_events.db`, NOT `/app/data/`. Resolve relative paths against the process CWD; do not assume the data volume. (Consequence worth knowing: both DBs live outside the mounted volume today and are already lost on every container recreate — so on the rig the migrator will usually find nothing, and that is a legitimate no-op, not a failure. Phase 1 incidentally fixes that data-loss bug by consolidating into `/app/data/site-localdb.db`.) - Runs **after** `RegisterReplicated` so migrated rows enter the oplog and replicate. - Tracking rows copy with `INSERT OR IGNORE` (same TEXT PK ⇒ idempotent). - Legacy `site_events` rows get **deterministic** ids — `"mig-{NodeName}-{legacyId}"` — NOT fresh GUIDs, so a crashed-then-rerun migration cannot duplicate events (`INSERT OR IGNORE` on the deterministic key). NodeName from `ScadaBridge:Node:NodeName`. @@ -312,6 +359,7 @@ Bridge `ISyncStatus` (`Connected`, `OplogBacklog` — **nullable**, null must su **Classification:** high-risk **Estimated implement time:** ~5 min **Parallelizable with:** Task 11 +**Blocked by:** Task 6, Task 8, **Task 5b** (scenario 3 reads events back through the query path) **Files:** - Create: `tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbSitePairConvergenceTests.cs` diff --git a/docs/plans/2026-07-19-localdb-adoption-phase1.md.tasks.json b/docs/plans/2026-07-19-localdb-adoption-phase1.md.tasks.json index 5b857656..239a6171 100644 --- a/docs/plans/2026-07-19-localdb-adoption-phase1.md.tasks.json +++ b/docs/plans/2026-07-19-localdb-adoption-phase1.md.tasks.json @@ -4,19 +4,28 @@ "mode": "parallel-waves", "implementerModel": "opus", "isolation": "worktree", + "branch": "feat/localdb-phase1", "note": "Dispatch every unblocked task concurrently (waves in the plan header). Parallel implementers MUST use worktree isolation - concurrent git in one worktree races destructively." }, + "amendments": [ + { + "date": "2026-07-19", + "by": "execution review", + "change": "Task 5 split into 5a (id 5, writer) and 5b (id 15, read path). Code recon found the integer site_events.id load-bearing in the keyset cursor (EventLogQueryService:70,141,153), the storage-cap purge (EventLogPurgeService:164), and long-typed DTOs crossing the site<->central Akka boundary (EventLogEntry.Id, both ContinuationTokens) - none of which were in the original Task 5 file set. 5b is high-risk. Task 10 gains a blockedBy on 15." + } + ], "tasks": [ {"id": 1, "subject": "Task 1: Add LocalDb packages to the build", "status": "pending"}, {"id": 2, "subject": "Task 2: Shared schema helpers (tracking + events, GUID PK)", "status": "pending", "blockedBy": [1]}, {"id": 3, "subject": "Task 3: Register LocalDb in the site composition root", "status": "pending", "blockedBy": [2]}, {"id": 4, "subject": "Task 4: Rewire OperationTrackingStore onto ILocalDb connections", "status": "pending", "blockedBy": [3]}, - {"id": 5, "subject": "Task 5: Rewire SiteEventLogger + EventLogQueryService (GUID ids)", "status": "pending", "blockedBy": [3]}, + {"id": 5, "subject": "Task 5a: Rewire SiteEventLogger onto ILocalDb + mint GUID ids", "status": "pending", "blockedBy": [3]}, + {"id": 15, "subject": "Task 5b: Migrate the event-log read path off integer ids", "status": "pending", "blockedBy": [5]}, {"id": 6, "subject": "Task 6: One-time legacy data migrator", "status": "pending", "blockedBy": [4, 5]}, {"id": 7, "subject": "Task 7: Replication registration + sync endpoint (default OFF)", "status": "pending", "blockedBy": [3]}, {"id": 8, "subject": "Task 8: Bearer auth interceptor for the sync endpoint", "status": "pending", "blockedBy": [7]}, {"id": 9, "subject": "Task 9: Surface ISyncStatus as site health signal", "status": "pending", "blockedBy": [7]}, - {"id": 10, "subject": "Task 10: Two-node in-process convergence test", "status": "pending", "blockedBy": [6, 8]}, + {"id": 10, "subject": "Task 10: Two-node in-process convergence test", "status": "pending", "blockedBy": [6, 8, 15]}, {"id": 11, "subject": "Task 11: Docker rig enablement (site-a pair)", "status": "pending", "blockedBy": [8]}, {"id": 12, "subject": "Task 12: Live gate on the docker rig", "status": "pending", "blockedBy": [9, 10, 11]}, {"id": 13, "subject": "Task 13: Documentation truth pass", "status": "pending", "blockedBy": [12]}, From ecf6b628500de0513306b5e80bc25f0efcec6b06 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sun, 19 Jul 2026 02:37:50 -0400 Subject: [PATCH 02/55] fix(build): pin AngleSharp 1.5.2 to unbreak the solution build GHSA-pgww-w46g-26qg (NU1902, moderate) on AngleSharp 1.1.1, pulled in transitively by bunit into CentralUI.Tests. Under TreatWarningsAsErrors this made `dotnet build ZB.MOM.WW.ScadaBridge.slnx` fail on a CLEAN main - the whole suite was unbuildable and every "0 warnings / suite green" gate unverifiable. Pre-existing, not introduced here; surfaced while starting LocalDb Phase 1, whose per-task verification depends on a green baseline. Same fix as the identical break in HistorianGateway (historiangw @ 6bc005d): pin the leaf, test-only. AngleSharp is a bunit HTML-parsing dependency and reaches no production project. Bumping bunit does not help - 2.0.33-preview is the current preview line and still resolves the vulnerable AngleSharp. Verified: dotnet build ZB.MOM.WW.ScadaBridge.slnx -> 0 Error(s), 0 Warning(s) (was 1 Error before this commit). Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts --- Directory.Packages.props | 17 +++++++++++++++++ ...ZB.MOM.WW.ScadaBridge.CentralUI.Tests.csproj | 3 +++ 2 files changed, 20 insertions(+) diff --git a/Directory.Packages.props b/Directory.Packages.props index f3469127..7da10e4f 100644 --- a/Directory.Packages.props +++ b/Directory.Packages.props @@ -124,4 +124,21 @@ + + + + + diff --git a/tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests.csproj b/tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests.csproj index 15dc7075..faedc98c 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests.csproj +++ b/tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests.csproj @@ -10,6 +10,9 @@ + + From 4715f2f921f0b9dde2fc0a88c3d76e1455baf3e9 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sun, 19 Jul 2026 02:39:11 -0400 Subject: [PATCH 03/55] chore(deps): raise gRPC stack 2.71.0 -> 2.76.0 (Protobuf 3.34.1) Forced by ZB.MOM.WW.LocalDb.Replication 0.1.0, whose nuspec floors Grpc.AspNetCore / Grpc.Net.Client / Grpc.Core.Api / Grpc.Tools at 2.76.0 and Google.Protobuf at 3.34.1. Restore fails NU1605 (package downgrade) below that floor, so LocalDb adoption cannot proceed without it. Landed as its own commit deliberately. The SQLitePCLRaw note in Directory.Packages.props deferred precisely this bump as belonging in "their own reviewed commit - not smuggled in under a SQLite security fix"; this honors that. Scope kept minimal: - Grpc.Core.Api pinned explicitly so the family stays on one version instead of floating in transitively. - Microsoft.Data.SqlClient and Newtonsoft.Json, also named in that note, are NOT bumped - nothing forces them and they carry their own risk profile. Direct consumer surface is two projects: Grpc.AspNetCore in Host, Grpc.Net.Client in Communication. Verified: dotnet build ZB.MOM.WW.ScadaBridge.slnx -> 0 Error(s), 0 Warning(s) Communication.Tests -> 312 passed, 0 failed Transport.IntegrationTests -> 103 passed, 0 failed Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts --- Directory.Packages.props | 26 ++++++++++++++++++++++---- 1 file changed, 22 insertions(+), 4 deletions(-) diff --git a/Directory.Packages.props b/Directory.Packages.props index 7da10e4f..62d64872 100644 --- a/Directory.Packages.props +++ b/Directory.Packages.props @@ -15,10 +15,28 @@ - - - - + + + + + + From 7370b3318646cdc6f9cfe925c9a600f9f9954249 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sun, 19 Jul 2026 02:39:52 -0400 Subject: [PATCH 04/55] feat(localdb): reference ZB.MOM.WW.LocalDb 0.1.0 packages Task 1 of the Phase 1 adoption plan. Wiring only - no behavior change yet. - Directory.Packages.props: LocalDb / .Replication / .Contracts @ 0.1.0 - NuGet.config: ZB.MOM.WW.LocalDb + ZB.MOM.WW.LocalDb.* mapped to the dohertj2-gitea feed (central package management requires explicit mapping) - SiteRuntime + SiteEventLogging: PackageReference ZB.MOM.WW.LocalDb - Host: ZB.MOM.WW.LocalDb + ZB.MOM.WW.LocalDb.Replication Required two prerequisite commits, both landed ahead of this one: ecf6b628 - AngleSharp pin (pre-existing break; the solution would not build) 4715f2f9 - gRPC 2.71.0 -> 2.76.0 (LocalDb.Replication's nuspec floor) SQLitePCLRaw stays pinned at 2.1.12 (GHSA-2m69-gcr7-jv3q) - verified intact. Verified: dotnet build ZB.MOM.WW.ScadaBridge.slnx -> 0 Error(s), 0 Warning(s) all three LocalDb packages restored from the Gitea feed Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts --- Directory.Packages.props | 3 +++ nuget.config | 2 ++ .../ZB.MOM.WW.ScadaBridge.Host.csproj | 2 ++ .../ZB.MOM.WW.ScadaBridge.SiteEventLogging.csproj | 1 + .../ZB.MOM.WW.ScadaBridge.SiteRuntime.csproj | 1 + 5 files changed, 9 insertions(+) diff --git a/Directory.Packages.props b/Directory.Packages.props index 62d64872..b49857f0 100644 --- a/Directory.Packages.props +++ b/Directory.Packages.props @@ -108,6 +108,9 @@ + + + - - - + diff --git a/tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests.csproj b/tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests.csproj index faedc98c..fb70a064 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests.csproj +++ b/tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests.csproj @@ -10,9 +10,6 @@ - - @@ -43,4 +40,28 @@ + + + + + \ No newline at end of file From 727fa48cbaf249c1a45339beb91e501fd7af2452 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sun, 19 Jul 2026 03:15:17 -0400 Subject: [PATCH 06/55] feat(localdb): move site_events to application-minted GUID ids Tasks 2 + 5a-writer + 5b of the LocalDb Phase 1 plan, landed together because they are one indivisible change: the schema, the writer that fills it, and every consumer that assumed the old id semantics. WHY the id changes. Site pairs will replicate site_events with last-writer-wins on the primary key. With INTEGER PRIMARY KEY AUTOINCREMENT both nodes independently mint 1, 2, 3... for unrelated events, so sync would treat them as the same row and silently overwrite. A GUID makes the event log a pure union across the pair. Task 2 - schema extracted so the Host's AddZbLocalDb onReady can create the tables before RegisterReplicated installs capture triggers (pre-registration rows are never captured): - new OperationTrackingSchema.Apply / SiteEventLogSchema.Apply, plain Microsoft.Data.Sqlite, no LocalDb dependency - both stores delegate their InitializeSchema to them (idempotent, so a directly-constructed store still works) - OperationTracking is unchanged - it already had a TEXT PK and replicates as-is Task 5a - SiteEventLogger mints Guid.NewGuid("N") per event and inserts it. Task 5b - three consumers assumed a monotonic integer id. All three move to timestamp ordering; leaving any one behind would be a live bug: - EventLogQueryService: "id > $afterId" would return an ARBITRARY subset of a GUID-keyed table and SILENTLY DROP ROWS from page-through. Now a composite (timestamp, id) keyset cursor with an opaque string token; timestamps are not unique, so id is the tie-break that guarantees exactly-once paging. - EventLogPurgeService: "ORDER BY id ASC LIMIT 1000" would delete a RANDOM batch instead of the oldest. Now orders by timestamp. - EventLogEntry.Id and both ContinuationTokens: long -> string. WIRE COMPATIBILITY. Those DTOs cross the site<->central Akka boundary (SiteCommunicationActor -> CommunicationService -> ManagementActor / CentralUI). No rolling-upgrade shim is needed because both sides ship in the same deployable and the rig redeploys as a unit. Checked: no Akka serializer binding pins these types by name. A stale numeric token degrades to "start from the beginning" (a visible repeat) rather than throwing or losing rows. Tests: the two that encoded the old semantics were rewritten to guard the new invariant rather than deleted - uniqueness instead of monotonicity, and oldest-purged-first keyed on timestamp. That second test also exposed a latent weakness: the bulk seed stamped every row with the same UtcNow, so "oldest" was never actually well-defined; rows now get distinct increasing timestamps, kept inside the retention window so the retention purge does not eat them first. Verified: dotnet build ZB.MOM.WW.ScadaBridge.slnx -> 0 Error(s), 0 Warning(s) SiteEventLogging.Tests -> 70 passed (12 new schema tests, red-first) CentralUI.Tests -> 925 passed Commons.Tests -> 684 passed SiteRuntime.Tests -> 529 passed, 1 pre-existing flaky failure (InstanceActorChildAttributeRaceTests - passes 3/3 in isolation with and without this change) Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts --- .../Pages/Monitoring/EventLogs.razor | 5 +- .../RemoteQuery/EventLogQueryRequest.cs | 10 +- .../RemoteQuery/EventLogQueryResponse.cs | 21 ++- .../EventLogPurgeService.cs | 9 +- .../EventLogQueryService.cs | 74 ++++++++- .../SiteEventLogSchema.cs | 71 +++++++++ .../SiteEventLogger.cs | 51 ++---- .../Tracking/OperationTrackingSchema.cs | 105 +++++++++++++ .../Tracking/OperationTrackingStore.cs | 76 +-------- .../EventLogPurgeServiceTests.cs | 75 ++++++--- .../EventLogQueryServiceTests.cs | 9 +- .../SiteEventLogSchemaTests.cs | 133 ++++++++++++++++ .../SiteEventLoggerTests.cs | 20 ++- .../Tracking/OperationTrackingSchemaTests.cs | 147 ++++++++++++++++++ 14 files changed, 659 insertions(+), 147 deletions(-) create mode 100644 src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogSchema.cs create mode 100644 src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Tracking/OperationTrackingSchema.cs create mode 100644 tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/SiteEventLogSchemaTests.cs create mode 100644 tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Tracking/OperationTrackingSchemaTests.cs diff --git a/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/EventLogs.razor b/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/EventLogs.razor index 83adeefb..357f4d80 100644 --- a/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/EventLogs.razor +++ b/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/EventLogs.razor @@ -178,7 +178,10 @@ private List? _entries; private bool _hasMore; - private long? _continuationToken; + // Opaque keyset cursor from the previous response — passed back verbatim, never + // parsed. It became a string in LocalDb Phase 1 when site_events ids turned into + // GUIDs and the cursor had to carry the timestamp too. + private string? _continuationToken; private bool _searching; private string? _errorMessage; private ToastNotification _toast = default!; diff --git a/src/ZB.MOM.WW.ScadaBridge.Commons/Messages/RemoteQuery/EventLogQueryRequest.cs b/src/ZB.MOM.WW.ScadaBridge.Commons/Messages/RemoteQuery/EventLogQueryRequest.cs index a7493f6e..628cb4bf 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Commons/Messages/RemoteQuery/EventLogQueryRequest.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Commons/Messages/RemoteQuery/EventLogQueryRequest.cs @@ -3,13 +3,19 @@ namespace ZB.MOM.WW.ScadaBridge.Commons.Messages.RemoteQuery; /// /// Request to query site event logs from central. /// Supports filtering by event type, severity, instance, time range, and keyword search. -/// Uses keyset pagination via continuation token (last event ID). +/// Uses keyset pagination via an opaque continuation token. /// /// /// Instance filter matched against the site event log's instance_id column, /// which stores the instance UniqueName (InstanceActor.LogLifecycleEvent passes /// _instanceUniqueName; EventLogQueryService matches instance_id = $instanceId). /// +/// +/// Opaque cursor from the previous response's ContinuationToken, or +/// to start from the oldest matching event. Treat as opaque — +/// it encodes timestamp and id together and its format is not part of the contract. +/// An unparseable token is treated as "start from the beginning" rather than an error. +/// public record EventLogQueryRequest( string CorrelationId, string SiteId, @@ -19,6 +25,6 @@ public record EventLogQueryRequest( string? Severity, string? InstanceId, string? KeywordFilter, - long? ContinuationToken, + string? ContinuationToken, int PageSize, DateTimeOffset Timestamp); diff --git a/src/ZB.MOM.WW.ScadaBridge.Commons/Messages/RemoteQuery/EventLogQueryResponse.cs b/src/ZB.MOM.WW.ScadaBridge.Commons/Messages/RemoteQuery/EventLogQueryResponse.cs index f807d373..8e4a8c78 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Commons/Messages/RemoteQuery/EventLogQueryResponse.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Commons/Messages/RemoteQuery/EventLogQueryResponse.cs @@ -3,8 +3,18 @@ namespace ZB.MOM.WW.ScadaBridge.Commons.Messages.RemoteQuery; /// /// A single event log entry returned from a site query. /// +/// +/// The event's primary key: a GUID string minted by the recording site node. +/// +/// This was a long autoincrement id until LocalDb Phase 1. Site pairs now +/// replicate site_events with last-writer-wins on this key, so a +/// server-minted integer would have both nodes issuing the same ids for unrelated +/// events and silently overwriting each other on sync. Consumers must treat it as +/// an opaque identifier — it carries no ordering. +/// +/// public record EventLogEntry( - long Id, + string Id, DateTimeOffset Timestamp, string EventType, string Severity, @@ -15,13 +25,18 @@ public record EventLogEntry( /// /// Response containing paginated event log entries from a site. -/// Uses keyset pagination: ContinuationToken is the last event ID in the result set. /// +/// +/// Opaque keyset-pagination cursor: pass it back verbatim on the next request to +/// continue after the last returned row, or to start from the +/// beginning. Encodes timestamp and id together, because GUID ids do not +/// sort chronologically and timestamps alone are not unique. Do not parse it. +/// public record EventLogQueryResponse( string CorrelationId, string SiteId, IReadOnlyList Entries, - long? ContinuationToken, + string? ContinuationToken, bool HasMore, bool Success, string? ErrorMessage, diff --git a/src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/EventLogPurgeService.cs b/src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/EventLogPurgeService.cs index 1bc3431b..5668d75f 100644 --- a/src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/EventLogPurgeService.cs +++ b/src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/EventLogPurgeService.cs @@ -159,9 +159,16 @@ public class EventLogPurgeService : BackgroundService var deleted = _eventLogger.WithConnection(connection => { using var cmd = connection.CreateCommand(); + // Order by timestamp, NOT by id. The id was a monotonic autoincrement + // when this was written, so "lowest id" meant "oldest"; since LocalDb + // Phase 1 it is a GUID and ordering by it would delete an ARBITRARY + // batch of events rather than the oldest ones. id remains in the ORDER + // BY only as a tie-break so the batch is deterministic. cmd.CommandText = $""" DELETE FROM site_events WHERE id IN ( - SELECT id FROM site_events ORDER BY id ASC LIMIT {CapPurgeBatchSize} + SELECT id FROM site_events + ORDER BY timestamp ASC, id ASC + LIMIT {CapPurgeBatchSize} ) """; var rows = cmd.ExecuteNonQuery(); diff --git a/src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/EventLogQueryService.cs b/src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/EventLogQueryService.cs index c2685dd5..e7f0f0a4 100644 --- a/src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/EventLogQueryService.cs +++ b/src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/EventLogQueryService.cs @@ -48,6 +48,52 @@ public class EventLogQueryService : IEventLogQueryService .Replace("_", "\\_"); } + /// + /// Separator between the timestamp and id halves of a continuation token. Chosen + /// because it cannot occur in an ISO 8601 "o" timestamp or in a GUID "N" string, + /// so the split is unambiguous. + /// + private const char TokenSeparator = '|'; + + /// + /// Builds the opaque keyset cursor for the last row of a page. The token carries + /// the exact stored timestamp string (not a re-formatted ) + /// so it compares byte-for-byte against the column under SQLite's BINARY collation. + /// + private static string FormatContinuationToken(EventLogEntry last) => + string.Concat( + last.Timestamp.ToUniversalTime().ToString("o", CultureInfo.InvariantCulture), + TokenSeparator, + last.Id); + + /// + /// Parses a continuation token back into its timestamp and id halves. + /// Returns for null, empty, or malformed tokens — including + /// a legacy numeric token from a client that predates the GUID id change — in which + /// case the caller starts from the beginning instead of failing the query. + /// + private static bool TryParseContinuationToken( + string? token, out string timestamp, out string id) + { + timestamp = ""; + id = ""; + + if (string.IsNullOrWhiteSpace(token)) + { + return false; + } + + var separator = token.IndexOf(TokenSeparator); + if (separator <= 0 || separator == token.Length - 1) + { + return false; + } + + timestamp = token[..separator]; + id = token[(separator + 1)..]; + return true; + } + /// public EventLogQueryResponse ExecuteQuery(EventLogQueryRequest request) { @@ -64,11 +110,23 @@ public class EventLogQueryService : IEventLogQueryService var whereClauses = new List(); var parameters = new List(); - // Keyset pagination: only return events with id > continuation token - if (request.ContinuationToken.HasValue) + // Keyset pagination on the composite (timestamp, id) key. + // + // This used to be a plain "id > $afterId" against a monotonic autoincrement + // id. Since LocalDb Phase 1 the id is a GUID, which does not sort + // chronologically — a naive "id > x" would return an arbitrary subset and + // SILENTLY DROP ROWS from the page-through. Timestamps alone are not unique + // (two events can share a tick), so the tie is broken on id to guarantee a + // total order and exactly-once paging. + // + // A malformed token is ignored rather than throwing: an old long-typed token + // from a client mid-upgrade degrades to "start from the beginning", which is + // a visible repeat rather than silent data loss. + if (TryParseContinuationToken(request.ContinuationToken, out var afterTs, out var afterId)) { - whereClauses.Add("id > $afterId"); - parameters.Add(new SqliteParameter("$afterId", request.ContinuationToken.Value)); + whereClauses.Add("(timestamp > $afterTs OR (timestamp = $afterTs AND id > $afterId))"); + parameters.Add(new SqliteParameter("$afterTs", afterTs)); + parameters.Add(new SqliteParameter("$afterId", afterId)); } if (request.From.HasValue) @@ -138,7 +196,7 @@ public class EventLogQueryService : IEventLogQueryService SELECT id, timestamp, event_type, severity, instance_id, source, message, details FROM site_events {whereClause} - ORDER BY id ASC + ORDER BY timestamp ASC, id ASC LIMIT $limit """; cmd.Parameters.AddWithValue("$limit", pageSize + 1); @@ -150,7 +208,7 @@ public class EventLogQueryService : IEventLogQueryService while (reader.Read()) { rows.Add(new EventLogEntry( - Id: reader.GetInt64(0), + Id: reader.GetString(0), // Parse with explicit invariant culture and round-trip style. // Stored values are ISO 8601 "o" UTC // (see SiteEventLogger.LogEventAsync), and the recorder's @@ -178,7 +236,9 @@ public class EventLogQueryService : IEventLogQueryService entries.RemoveAt(entries.Count - 1); } - var continuationToken = entries.Count > 0 ? entries[^1].Id : (long?)null; + var continuationToken = entries.Count > 0 + ? FormatContinuationToken(entries[^1]) + : null; return new EventLogQueryResponse( CorrelationId: request.CorrelationId, diff --git a/src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogSchema.cs b/src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogSchema.cs new file mode 100644 index 00000000..cde099b0 --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogSchema.cs @@ -0,0 +1,71 @@ +using Microsoft.Data.Sqlite; + +namespace ZB.MOM.WW.ScadaBridge.SiteEventLogging; + +/// +/// DDL for the site_events table, extracted from so +/// it can be applied by whoever owns the database file. +/// +/// Under LocalDb Phase 1 that owner is the Host's AddZbLocalDb onReady callback: +/// the table must exist before ILocalDb.RegisterReplicated installs its capture +/// triggers, and rows written before registration are never captured or snapshotted. +/// +/// +/// Schema change. The primary key is a TEXT GUID minted by the writer, replacing +/// the historical INTEGER PRIMARY KEY AUTOINCREMENT. This is required, not +/// cosmetic: replication resolves conflicts by last-writer-wins on the primary key, so +/// two site nodes each minting autoincrement id 1 for unrelated events would be treated +/// as one row and silently overwrite each other. GUID keys make the event log a pure +/// union across the pair. Consumers that relied on the id being monotonic — the keyset +/// paging cursor and the storage-cap purge — order by timestamp instead. +/// +/// +/// Deliberately depends only on Microsoft.Data.Sqlite, not on the LocalDb +/// library — the Host needs to apply this DDL to a LocalDb-managed connection, but +/// nothing about the schema itself is LocalDb-specific. +/// +/// +public static class SiteEventLogSchema +{ + /// + /// Creates the site_events table and its indexes when absent. Idempotent — + /// safe to run on every startup. + /// + /// An open connection to the database that should hold the table. + public static void Apply(SqliteConnection connection) + { + ArgumentNullException.ThrowIfNull(connection); + + // auto_vacuum must be set before any table is created for it to take effect + // on a fresh database. With INCREMENTAL mode, PRAGMA incremental_vacuum can + // later reclaim free pages so the storage-cap purge can shrink the file. + using (var pragmaCmd = connection.CreateCommand()) + { + pragmaCmd.CommandText = "PRAGMA auto_vacuum = INCREMENTAL"; + pragmaCmd.ExecuteNonQuery(); + } + + using var cmd = connection.CreateCommand(); + cmd.CommandText = """ + CREATE TABLE IF NOT EXISTS site_events ( + id TEXT NOT NULL PRIMARY KEY, + timestamp TEXT NOT NULL, + event_type TEXT NOT NULL, + severity TEXT NOT NULL, + instance_id TEXT, + source TEXT NOT NULL, + message TEXT NOT NULL, + details TEXT + ); + CREATE INDEX IF NOT EXISTS idx_events_timestamp ON site_events(timestamp); + CREATE INDEX IF NOT EXISTS idx_events_type ON site_events(event_type); + CREATE INDEX IF NOT EXISTS idx_events_instance ON site_events(instance_id); + CREATE INDEX IF NOT EXISTS idx_events_severity ON site_events(severity); + """; + // The query service also supports keyword search via leading-wildcard + // LIKE on message/source. A leading-wildcard LIKE cannot use a B-tree + // index, so that path intentionally full-scans; severity/event_type/ + // instance_id/timestamp filters above are all covered. + cmd.ExecuteNonQuery(); + } +} diff --git a/src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs b/src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs index 23ce86e5..53ff5add 100644 --- a/src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs +++ b/src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs @@ -130,40 +130,11 @@ public class SiteEventLogger : ISiteEventLogger, IDisposable } } - private void InitializeSchema() - { - // auto_vacuum must be set before any table is created for it to take effect - // on a fresh database. With INCREMENTAL mode, PRAGMA incremental_vacuum can - // later reclaim free pages so the storage-cap purge can shrink the file. - using (var pragmaCmd = _connection.CreateCommand()) - { - pragmaCmd.CommandText = "PRAGMA auto_vacuum = INCREMENTAL"; - pragmaCmd.ExecuteNonQuery(); - } - - using var cmd = _connection.CreateCommand(); - cmd.CommandText = """ - CREATE TABLE IF NOT EXISTS site_events ( - id INTEGER PRIMARY KEY AUTOINCREMENT, - timestamp TEXT NOT NULL, - event_type TEXT NOT NULL, - severity TEXT NOT NULL, - instance_id TEXT, - source TEXT NOT NULL, - message TEXT NOT NULL, - details TEXT - ); - CREATE INDEX IF NOT EXISTS idx_events_timestamp ON site_events(timestamp); - CREATE INDEX IF NOT EXISTS idx_events_type ON site_events(event_type); - CREATE INDEX IF NOT EXISTS idx_events_instance ON site_events(instance_id); - CREATE INDEX IF NOT EXISTS idx_events_severity ON site_events(severity); - """; - // The query service also supports keyword search via leading-wildcard - // LIKE on message/source. A leading-wildcard LIKE cannot use a B-tree - // index, so that path intentionally full-scans; severity/event_type/ - // instance_id/timestamp filters above are all covered. - cmd.ExecuteNonQuery(); - } + // Schema lives in SiteEventLogSchema so the Host's AddZbLocalDb onReady callback + // can create this table in the consolidated site database before RegisterReplicated + // installs its capture triggers. Applying it here too keeps a directly-constructed + // logger (tests, tooling) self-sufficient; the DDL is idempotent. + private void InitializeSchema() => SiteEventLogSchema.Apply(_connection); /// /// Closed set of allowed severities. Case-sensitive to @@ -198,7 +169,13 @@ public class SiteEventLogger : ISiteEventLogger, IDisposable nameof(severity)); } + // The id is minted here, by the application, rather than by SQLite. Under + // LocalDb replication the site pair converges by last-writer-wins on the + // primary key, so a server-minted AUTOINCREMENT would have both nodes + // independently issuing id 1, 2, 3... for unrelated events and silently + // overwriting each other on sync. A GUID makes the event log a pure union. var pending = new PendingEvent( + Guid.NewGuid().ToString("N"), DateTimeOffset.UtcNow.ToString("o"), eventType, severity, @@ -235,9 +212,10 @@ public class SiteEventLogger : ISiteEventLogger, IDisposable { using var cmd = connection.CreateCommand(); cmd.CommandText = """ - INSERT INTO site_events (timestamp, event_type, severity, instance_id, source, message, details) - VALUES ($timestamp, $event_type, $severity, $instance_id, $source, $message, $details) + INSERT INTO site_events (id, timestamp, event_type, severity, instance_id, source, message, details) + VALUES ($id, $timestamp, $event_type, $severity, $instance_id, $source, $message, $details) """; + cmd.Parameters.AddWithValue("$id", pending.Id); cmd.Parameters.AddWithValue("$timestamp", pending.Timestamp); cmd.Parameters.AddWithValue("$event_type", pending.EventType); cmd.Parameters.AddWithValue("$severity", pending.Severity); @@ -311,6 +289,7 @@ public class SiteEventLogger : ISiteEventLogger, IDisposable /// An event awaiting persistence by the background writer. private sealed record PendingEvent( + string Id, string Timestamp, string EventType, string Severity, diff --git a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Tracking/OperationTrackingSchema.cs b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Tracking/OperationTrackingSchema.cs new file mode 100644 index 00000000..3cbd65e1 --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Tracking/OperationTrackingSchema.cs @@ -0,0 +1,105 @@ +using Microsoft.Data.Sqlite; + +namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tracking; + +/// +/// DDL for the OperationTracking table, extracted from +/// so it can be applied by whoever owns the +/// database file. +/// +/// Under LocalDb Phase 1 that owner is the Host's AddZbLocalDb onReady callback: +/// the table must exist before ILocalDb.RegisterReplicated installs its capture +/// triggers, and rows written before registration are never captured or snapshotted. +/// The store still calls this during its own initialization, so a store constructed +/// directly (tests, tooling) remains self-sufficient. +/// +/// +/// Deliberately depends only on Microsoft.Data.Sqlite, not on the LocalDb +/// library — the Host needs to apply this DDL to a LocalDb-managed connection, but +/// nothing about the schema itself is LocalDb-specific. +/// +/// +public static class OperationTrackingSchema +{ + /// + /// Creates the OperationTracking table and its indexes when absent, and + /// additively upgrades a table created by an older build. Idempotent — safe to run + /// on every startup. + /// + /// An open connection to the database that should hold the table. + public static void Apply(SqliteConnection connection) + { + ArgumentNullException.ThrowIfNull(connection); + + using (var cmd = connection.CreateCommand()) + { + // TrackedOperationId is a TEXT GUID, which is also what lets this table + // replicate as-is: RegisterReplicated requires an explicit primary key and + // rejects BLOB columns, and there are none here. + cmd.CommandText = """ + CREATE TABLE IF NOT EXISTS OperationTracking ( + TrackedOperationId TEXT NOT NULL PRIMARY KEY, + Kind TEXT NOT NULL, + TargetSummary TEXT NULL, + Status TEXT NOT NULL, + RetryCount INTEGER NOT NULL DEFAULT 0, + LastError TEXT NULL, + HttpStatus INTEGER NULL, + CreatedAtUtc TEXT NOT NULL, + UpdatedAtUtc TEXT NOT NULL, + TerminalAtUtc TEXT NULL, + SourceInstanceId TEXT NULL, + SourceScript TEXT NULL, + SourceNode TEXT NULL + ); + CREATE INDEX IF NOT EXISTS IX_OperationTracking_Status_Updated + ON OperationTracking (Status, UpdatedAtUtc); + CREATE INDEX IF NOT EXISTS IX_OperationTracking_UpdatedAt + ON OperationTracking (UpdatedAtUtc); + """; + cmd.ExecuteNonQuery(); + } + + // SourceNode stamping: additively add the SourceNode column. + // CREATE TABLE IF NOT EXISTS above does NOT add columns to an + // OperationTracking table that already exists from a pre-SourceNode + // build, so a tracking.db created by an older build needs the column + // ALTER-ed in. The file is durable across restart/failover by design + // (retention window default 7 days), so without this step every + // RecordEnqueueAsync on an upgraded deployment would bind $sourceNode + // against a missing column and the write would fail. + // SQLite has no "ADD COLUMN IF NOT EXISTS"; the column presence is + // probed first and the ALTER skipped when already there. The column is + // nullable with no default, so any row written before this migration + // reads back SourceNode = null (back-compat). + AddColumnIfMissing(connection, "SourceNode", "TEXT NULL"); + } + + /// + /// Additively adds a column to OperationTracking only when it is not + /// already present. SQLite lacks ADD COLUMN IF NOT EXISTS, so the + /// schema is probed via PRAGMA table_info first. Mirrors the + /// SqliteAuditWriter.AddColumnIfMissing precedent. + /// + private static void AddColumnIfMissing( + SqliteConnection connection, string columnName, string columnDefinition) + { + using (var probe = connection.CreateCommand()) + { + probe.CommandText = + "SELECT COUNT(*) FROM pragma_table_info('OperationTracking') WHERE name = $name"; + probe.Parameters.AddWithValue("$name", columnName); + if (Convert.ToInt32(probe.ExecuteScalar()) > 0) + { + return; + } + } + + using var alter = connection.CreateCommand(); + // Column name + definition are caller-controlled constants, never user + // input — safe to interpolate (parameters are not permitted in DDL). + alter.CommandText = + $"ALTER TABLE OperationTracking ADD COLUMN {columnName} {columnDefinition}"; + alter.ExecuteNonQuery(); + } +} diff --git a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Tracking/OperationTrackingStore.cs b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Tracking/OperationTrackingStore.cs index db2d4462..1257ea10 100644 --- a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Tracking/OperationTrackingStore.cs +++ b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Tracking/OperationTrackingStore.cs @@ -69,76 +69,12 @@ public class OperationTrackingStore : IOperationTrackingStore, IAsyncDisposable, InitializeSchema(); } - private void InitializeSchema() - { - using var cmd = _writeConnection.CreateCommand(); - cmd.CommandText = """ - CREATE TABLE IF NOT EXISTS OperationTracking ( - TrackedOperationId TEXT NOT NULL PRIMARY KEY, - Kind TEXT NOT NULL, - TargetSummary TEXT NULL, - Status TEXT NOT NULL, - RetryCount INTEGER NOT NULL DEFAULT 0, - LastError TEXT NULL, - HttpStatus INTEGER NULL, - CreatedAtUtc TEXT NOT NULL, - UpdatedAtUtc TEXT NOT NULL, - TerminalAtUtc TEXT NULL, - SourceInstanceId TEXT NULL, - SourceScript TEXT NULL, - SourceNode TEXT NULL - ); - CREATE INDEX IF NOT EXISTS IX_OperationTracking_Status_Updated - ON OperationTracking (Status, UpdatedAtUtc); - CREATE INDEX IF NOT EXISTS IX_OperationTracking_UpdatedAt - ON OperationTracking (UpdatedAtUtc); - """; - cmd.ExecuteNonQuery(); - - // SourceNode stamping: additively add the SourceNode column. - // CREATE TABLE IF NOT EXISTS above does NOT add columns to an - // OperationTracking table that already exists from a pre-SourceNode - // build, so a tracking.db created by an older build needs the column - // ALTER-ed in. The file is durable across restart/failover by design - // (retention window default 7 days), so without this step every - // RecordEnqueueAsync on an upgraded deployment would bind $sourceNode - // against a missing column and the write would fail. - // SQLite has no "ADD COLUMN IF NOT EXISTS"; the column presence is - // probed first and the ALTER skipped when already there. The column is - // nullable with no default, so any row written before this migration - // reads back SourceNode = null (back-compat). - // - // NOTE: This is the FIRST idempotent column-upgrade in - // OperationTrackingStore — prior schema changes pre-dated any - // production rollout and relied solely on CREATE TABLE IF NOT EXISTS. - // The helper mirrors the SqliteAuditWriter precedent. - AddColumnIfMissing("SourceNode", "TEXT NULL"); - } - - /// - /// Additively adds a column to OperationTracking only when it is not - /// already present. SQLite lacks ADD COLUMN IF NOT EXISTS, so the - /// schema is probed via PRAGMA table_info first. Idempotent — safe - /// to run on every . Mirrors the - /// SqliteAuditWriter.AddColumnIfMissing precedent. - /// - private void AddColumnIfMissing(string columnName, string columnDefinition) - { - using var probe = _writeConnection.CreateCommand(); - probe.CommandText = "SELECT COUNT(*) FROM pragma_table_info('OperationTracking') WHERE name = $name"; - probe.Parameters.AddWithValue("$name", columnName); - var exists = Convert.ToInt32(probe.ExecuteScalar()) > 0; - if (exists) - { - return; - } - - using var alter = _writeConnection.CreateCommand(); - // Column name + definition are caller-controlled constants, never user - // input — safe to interpolate (parameters are not permitted in DDL). - alter.CommandText = $"ALTER TABLE OperationTracking ADD COLUMN {columnName} {columnDefinition}"; - alter.ExecuteNonQuery(); - } + // Schema lives in OperationTrackingSchema so the Host's AddZbLocalDb onReady + // callback can create this table in the consolidated site database before + // RegisterReplicated installs its capture triggers. Applying it here too keeps a + // directly-constructed store (tests, tooling) self-sufficient; the DDL is + // idempotent, so running it from both places is harmless. + private void InitializeSchema() => OperationTrackingSchema.Apply(_writeConnection); /// public async Task RecordEnqueueAsync( diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/EventLogPurgeServiceTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/EventLogPurgeServiceTests.cs index dbd7522a..8725d83a 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/EventLogPurgeServiceTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/EventLogPurgeServiceTests.cs @@ -56,9 +56,10 @@ public class EventLogPurgeServiceTests : IDisposable { using var cmd = connection.CreateCommand(); cmd.CommandText = """ - INSERT INTO site_events (timestamp, event_type, severity, source, message) - VALUES ($ts, 'script', 'Info', 'Test', 'Test message') + INSERT INTO site_events (id, timestamp, event_type, severity, source, message) + VALUES ($id, $ts, 'script', 'Info', 'Test', 'Test message') """; + cmd.Parameters.AddWithValue("$id", Guid.NewGuid().ToString("N")); cmd.Parameters.AddWithValue("$ts", timestamp.ToString("o")); cmd.ExecuteNonQuery(); }); @@ -135,6 +136,29 @@ public class EventLogPurgeServiceTests : IDisposable Assert.True(size > 0); } + /// + /// Base instant for bulk-seeded events: one day ago, evaluated once so a run is + /// self-consistent. + /// + /// Must stay INSIDE the retention window (RetentionDays, default 30). + /// applies the retention purge before + /// the storage-cap purge, so a fixed calendar date in the past would be deleted + /// wholesale as stale and the storage-cap assertions would silently test an empty + /// table rather than cap-trimming behaviour. + /// + /// + private static readonly DateTimeOffset BulkSeedStart = + DateTimeOffset.UtcNow.AddDays(-1); + + /// + /// Timestamp of the i-th bulk-seeded event. Each row is one second newer than the + /// last, so "oldest" is unambiguous — the storage-cap purge orders by timestamp, + /// and previously every bulk row shared the same UtcNow, which left the + /// oldest-first guarantee untestable. + /// + private static DateTimeOffset BulkSeedTimestamp(int index) => + BulkSeedStart.AddSeconds(index); + private void InsertBulkEvents(int count) { // Each event carries a sizeable details payload so the database grows @@ -146,24 +170,30 @@ public class EventLogPurgeServiceTests : IDisposable { using var cmd = connection.CreateCommand(); cmd.CommandText = """ - INSERT INTO site_events (timestamp, event_type, severity, source, message, details) - VALUES ($ts, 'script', 'Info', 'Test', 'Bulk event', $details) + INSERT INTO site_events (id, timestamp, event_type, severity, source, message, details) + VALUES ($id, $ts, 'script', 'Info', 'Test', 'Bulk event', $details) """; - cmd.Parameters.AddWithValue("$ts", DateTimeOffset.UtcNow.ToString("o")); + cmd.Parameters.AddWithValue("$id", Guid.NewGuid().ToString("N")); + cmd.Parameters.AddWithValue("$ts", BulkSeedTimestamp(i).ToString("o")); cmd.Parameters.AddWithValue("$details", details); cmd.ExecuteNonQuery(); } }); } - private long MinEventId() + /// + /// Oldest surviving event timestamp. Replaces the former MIN(id) probe: + /// ids are GUIDs since LocalDb Phase 1 and carry no ordering, so age is measured + /// on the timestamp column the purge itself orders by. + /// + private string MinEventTimestamp() { return _eventLogger.WithConnection(connection => { using var cmd = connection.CreateCommand(); - cmd.CommandText = "SELECT MIN(id) FROM site_events"; + cmd.CommandText = "SELECT MIN(timestamp) FROM site_events"; var result = cmd.ExecuteScalar(); - return result is long l ? l : 0; + return result as string ?? ""; }); } @@ -204,9 +234,15 @@ public class EventLogPurgeServiceTests : IDisposable [Fact] public void PurgeByStorageCap_RemovesOldestEventsFirst() { - // Regression test for SiteEventLogging-002: only the oldest events - // (lowest ids) should be removed when trimming to the cap. - InsertBulkEvents(3000); + // Regression test for SiteEventLogging-002: only the oldest events should be + // removed when trimming to the cap. + // + // "Oldest" used to mean "lowest autoincrement id". Since LocalDb Phase 1 the id + // is a GUID with no ordering, so both the purge and this test key on timestamp. + // Ordering by id here would delete an arbitrary batch, which is precisely the + // regression this test now guards. + const int eventCount = 3000; + InsertBulkEvents(eventCount); var purge = CreatePurgeService(); var totalSize = purge.GetDatabaseSizeBytes(); @@ -218,20 +254,23 @@ public class EventLogPurgeServiceTests : IDisposable MaxStorageMb = (int)Math.Max(1, (totalSize / 2) / (1024 * 1024)) }; - var minIdBefore = MinEventId(); + var oldestBefore = MinEventTimestamp(); var cappedPurge = CreatePurgeService(capOptions); cappedPurge.RunPurge(); - var minIdAfter = MinEventId(); + var oldestAfter = MinEventTimestamp(); - // The surviving rows must be the newest ones — minimum id has advanced. - Assert.True(minIdAfter > minIdBefore, - "Oldest events (lowest ids) must be purged first."); + // The surviving rows must be the newest ones — the oldest timestamp advanced. + Assert.True( + string.CompareOrdinal(oldestAfter, oldestBefore) > 0, + $"Oldest events must be purged first; oldest went from '{oldestBefore}' to '{oldestAfter}'."); - // The newest event (highest id) must still be present. + // The newest event must still be present. + var newestTimestamp = BulkSeedTimestamp(eventCount - 1).ToString("o"); var newestPresent = _eventLogger.WithConnection(connection => { using var cmd = connection.CreateCommand(); - cmd.CommandText = "SELECT COUNT(*) FROM site_events WHERE id = 3000"; + cmd.CommandText = "SELECT COUNT(*) FROM site_events WHERE timestamp = $ts"; + cmd.Parameters.AddWithValue("$ts", newestTimestamp); return (long)cmd.ExecuteScalar()!; }); Assert.Equal(1L, newestPresent); diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/EventLogQueryServiceTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/EventLogQueryServiceTests.cs index b3622d99..82cac861 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/EventLogQueryServiceTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/EventLogQueryServiceTests.cs @@ -45,7 +45,7 @@ public class EventLogQueryServiceTests : IDisposable string? severity = null, string? instanceId = null, string? keyword = null, - long? continuationToken = null, + string? continuationToken = null, int pageSize = 500, DateTimeOffset? from = null, DateTimeOffset? to = null) => @@ -281,7 +281,7 @@ public class EventLogQueryServiceTests : IDisposable Assert.Single(response.Entries); var entry = response.Entries[0]; - Assert.True(entry.Id > 0); + Assert.False(string.IsNullOrWhiteSpace(entry.Id)); Assert.Equal("script", entry.EventType); Assert.Equal("Error", entry.Severity); Assert.Equal("inst-1", entry.InstanceId); @@ -340,9 +340,10 @@ public class EventLogQueryServiceTests : IDisposable { using var cmd = connection.CreateCommand(); cmd.CommandText = """ - INSERT INTO site_events (timestamp, event_type, severity, instance_id, source, message) - VALUES ($ts, $et, $sev, $iid, $src, $msg) + INSERT INTO site_events (id, timestamp, event_type, severity, instance_id, source, message) + VALUES ($id, $ts, $et, $sev, $iid, $src, $msg) """; + cmd.Parameters.AddWithValue("$id", Guid.NewGuid().ToString("N")); cmd.Parameters.AddWithValue("$ts", timestamp.ToString("o")); cmd.Parameters.AddWithValue("$et", eventType); cmd.Parameters.AddWithValue("$sev", severity); diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/SiteEventLogSchemaTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/SiteEventLogSchemaTests.cs new file mode 100644 index 00000000..daed0b52 --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/SiteEventLogSchemaTests.cs @@ -0,0 +1,133 @@ +using Microsoft.Data.Sqlite; + +namespace ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests; + +/// +/// LocalDb Phase 1 (Task 2) — the site_events DDL is extracted out of +/// so the Host's AddZbLocalDb onReady callback can +/// create it in the consolidated site database. +/// +/// The extraction also carries the one deliberate schema CHANGE of Phase 1: the primary +/// key moves from INTEGER PRIMARY KEY AUTOINCREMENT to a TEXT GUID. Under the +/// library's last-writer-wins replication, two site nodes each minting autoincrement id 1 +/// would be treated as the same row and silently overwrite one another. GUID keys make +/// the event log a pure union across the pair instead. +/// +/// +public class SiteEventLogSchemaTests +{ + private static SqliteConnection OpenConnection() + { + var conn = new SqliteConnection("Data Source=:memory:"); + conn.Open(); + return conn; + } + + private static List<(string Name, string Type, bool NotNull, bool Pk)> TableInfo( + SqliteConnection conn, string table) + { + using var cmd = conn.CreateCommand(); + cmd.CommandText = $"PRAGMA table_info('{table}')"; + using var reader = cmd.ExecuteReader(); + var cols = new List<(string, string, bool, bool)>(); + while (reader.Read()) + { + cols.Add((reader.GetString(1), reader.GetString(2), reader.GetInt32(3) == 1, + reader.GetInt32(5) > 0)); + } + + return cols; + } + + [Fact] + public void Apply_CreatesTableWithExpectedColumns() + { + using var conn = OpenConnection(); + + SiteEventLogSchema.Apply(conn); + + Assert.Equal( + ["id", "timestamp", "event_type", "severity", "instance_id", "source", "message", "details"], + TableInfo(conn, "site_events").Select(c => c.Name)); + } + + [Fact] + public void Apply_IdIsTextPrimaryKey_NotAutoincrement() + { + using var conn = OpenConnection(); + + SiteEventLogSchema.Apply(conn); + + var pk = Assert.Single(TableInfo(conn, "site_events"), c => c.Pk); + Assert.Equal("id", pk.Name); + Assert.Equal("TEXT", pk.Type); + + // AUTOINCREMENT creates the sqlite_sequence bookkeeping table. Its absence + // is the load-bearing assertion: ids must be application-minted GUIDs, not + // server-minted integers that collide across the replicating pair. + using var cmd = conn.CreateCommand(); + cmd.CommandText = + "SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = 'sqlite_sequence'"; + Assert.Equal(0L, Convert.ToInt64(cmd.ExecuteScalar())); + } + + [Fact] + public void Apply_RejectsIntegerIdInserts() + { + using var conn = OpenConnection(); + + SiteEventLogSchema.Apply(conn); + + // A TEXT PK with no AUTOINCREMENT means an INSERT omitting id is an error, + // which is what forces the writer to mint one (Task 5a). + using var cmd = conn.CreateCommand(); + cmd.CommandText = """ + INSERT INTO site_events (timestamp, event_type, severity, source, message) + VALUES ('2026-07-19T00:00:00Z', 'Test', 'Info', 'test', 'no id supplied') + """; + Assert.Throws(() => cmd.ExecuteNonQuery()); + } + + [Fact] + public void Apply_HasNoBlobColumns() + { + using var conn = OpenConnection(); + + SiteEventLogSchema.Apply(conn); + + Assert.DoesNotContain( + TableInfo(conn, "site_events"), + c => c.Type.Contains("BLOB", StringComparison.OrdinalIgnoreCase)); + } + + [Fact] + public void Apply_CreatesExpectedIndexes() + { + using var conn = OpenConnection(); + + SiteEventLogSchema.Apply(conn); + + using var cmd = conn.CreateCommand(); + cmd.CommandText = + "SELECT name FROM sqlite_master WHERE type = 'index' AND tbl_name = 'site_events' " + + "AND name NOT LIKE 'sqlite_%' ORDER BY name"; + using var reader = cmd.ExecuteReader(); + var indexes = new List(); + while (reader.Read()) indexes.Add(reader.GetString(0)); + + Assert.Equal( + ["idx_events_instance", "idx_events_severity", "idx_events_timestamp", "idx_events_type"], + indexes); + } + + [Fact] + public void Apply_IsIdempotent() + { + using var conn = OpenConnection(); + + SiteEventLogSchema.Apply(conn); + SiteEventLogSchema.Apply(conn); + + Assert.Equal(8, TableInfo(conn, "site_events").Count); + } +} diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/SiteEventLoggerTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/SiteEventLoggerTests.cs index 76dd1059..81ae9137 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/SiteEventLoggerTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/SiteEventLoggerTests.cs @@ -110,20 +110,30 @@ public class SiteEventLoggerTests : IDisposable } [Fact] - public async Task LogEventAsync_MultipleEvents_AutoIncrementIds() + public async Task LogEventAsync_MultipleEvents_GetDistinctApplicationMintedIds() { + // Replaces LogEventAsync_MultipleEvents_AutoIncrementIds. Ids are no longer + // server-minted autoincrement integers: under LocalDb replication the site pair + // converges by last-writer-wins on the primary key, so both nodes issuing 1, 2, + // 3... for unrelated events would silently overwrite each other on sync. + // + // The invariant is therefore uniqueness, NOT monotonicity — ordering is carried + // by the timestamp column, which is what the query and purge paths sort on. await _logger.LogEventAsync("script", "Info", null, "S1", "First"); await _logger.LogEventAsync("script", "Info", null, "S2", "Second"); await _logger.LogEventAsync("script", "Info", null, "S3", "Third"); using var cmd = _verifyConnection.CreateCommand(); - cmd.CommandText = "SELECT id FROM site_events ORDER BY id"; + cmd.CommandText = "SELECT id FROM site_events ORDER BY timestamp"; using var reader = cmd.ExecuteReader(); - var ids = new List(); - while (reader.Read()) ids.Add(reader.GetInt64(0)); + var ids = new List(); + while (reader.Read()) ids.Add(reader.GetString(0)); Assert.Equal(3, ids.Count); - Assert.True(ids[0] < ids[1] && ids[1] < ids[2]); + Assert.All(ids, id => Assert.False(string.IsNullOrWhiteSpace(id))); + Assert.Equal(3, ids.Distinct(StringComparer.Ordinal).Count()); + Assert.All(ids, id => Assert.True(Guid.TryParseExact(id, "N", out _), + $"Event id '{id}' is not a GUID in 'N' format.")); } [Fact] diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Tracking/OperationTrackingSchemaTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Tracking/OperationTrackingSchemaTests.cs new file mode 100644 index 00000000..49ae95af --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Tracking/OperationTrackingSchemaTests.cs @@ -0,0 +1,147 @@ +using Microsoft.Data.Sqlite; +using ZB.MOM.WW.ScadaBridge.SiteRuntime.Tracking; + +namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Tracking; + +/// +/// LocalDb Phase 1 (Task 2) — the OperationTracking DDL is extracted out of +/// so it can also be applied from the Host's +/// AddZbLocalDb onReady callback, which owns table creation for the consolidated +/// site database. These tests pin the extracted schema against the shape the store has +/// always created: the move must be behaviour-preserving. +/// +public class OperationTrackingSchemaTests +{ + private static SqliteConnection OpenConnection() + { + var conn = new SqliteConnection("Data Source=:memory:"); + conn.Open(); + return conn; + } + + private static List<(string Name, string Type, bool NotNull, bool Pk)> TableInfo( + SqliteConnection conn, string table) + { + using var cmd = conn.CreateCommand(); + cmd.CommandText = $"PRAGMA table_info('{table}')"; + using var reader = cmd.ExecuteReader(); + var cols = new List<(string, string, bool, bool)>(); + while (reader.Read()) + { + cols.Add((reader.GetString(1), reader.GetString(2), reader.GetInt32(3) == 1, + reader.GetInt32(5) > 0)); + } + + return cols; + } + + [Fact] + public void Apply_CreatesTableWithExpectedColumns() + { + using var conn = OpenConnection(); + + OperationTrackingSchema.Apply(conn); + + var cols = TableInfo(conn, "OperationTracking"); + Assert.Equal( + [ + "TrackedOperationId", "Kind", "TargetSummary", "Status", "RetryCount", + "LastError", "HttpStatus", "CreatedAtUtc", "UpdatedAtUtc", "TerminalAtUtc", + "SourceInstanceId", "SourceScript", "SourceNode" + ], + cols.Select(c => c.Name)); + } + + [Fact] + public void Apply_UsesTextPrimaryKey() + { + using var conn = OpenConnection(); + + OperationTrackingSchema.Apply(conn); + + // RegisterReplicated requires an explicit primary key and rejects BLOB + // columns; a TEXT PK is what makes this table replicate as-is. + var pk = Assert.Single(TableInfo(conn, "OperationTracking"), c => c.Pk); + Assert.Equal("TrackedOperationId", pk.Name); + Assert.Equal("TEXT", pk.Type); + } + + [Fact] + public void Apply_HasNoBlobColumns() + { + using var conn = OpenConnection(); + + OperationTrackingSchema.Apply(conn); + + // ILocalDb.RegisterReplicated throws on any column whose declared type + // contains BLOB (json_object cannot capture them). + Assert.DoesNotContain( + TableInfo(conn, "OperationTracking"), + c => c.Type.Contains("BLOB", StringComparison.OrdinalIgnoreCase)); + } + + [Fact] + public void Apply_CreatesExpectedIndexes() + { + using var conn = OpenConnection(); + + OperationTrackingSchema.Apply(conn); + + using var cmd = conn.CreateCommand(); + cmd.CommandText = + "SELECT name FROM sqlite_master WHERE type = 'index' AND tbl_name = 'OperationTracking' " + + "AND name NOT LIKE 'sqlite_%' ORDER BY name"; + using var reader = cmd.ExecuteReader(); + var indexes = new List(); + while (reader.Read()) indexes.Add(reader.GetString(0)); + + Assert.Equal( + ["IX_OperationTracking_Status_Updated", "IX_OperationTracking_UpdatedAt"], + indexes); + } + + [Fact] + public void Apply_IsIdempotent() + { + using var conn = OpenConnection(); + + OperationTrackingSchema.Apply(conn); + OperationTrackingSchema.Apply(conn); + + Assert.Equal(13, TableInfo(conn, "OperationTracking").Count); + } + + [Fact] + public void Apply_AddsSourceNodeToLegacyTableMissingIt() + { + using var conn = OpenConnection(); + + // A tracking DB created by a pre-SourceNode build. CREATE TABLE IF NOT + // EXISTS will not add the column, so Apply must ALTER it in — the + // additive-migration behaviour the store has today must survive extraction. + using (var legacy = conn.CreateCommand()) + { + legacy.CommandText = """ + CREATE TABLE OperationTracking ( + TrackedOperationId TEXT NOT NULL PRIMARY KEY, + Kind TEXT NOT NULL, + TargetSummary TEXT NULL, + Status TEXT NOT NULL, + RetryCount INTEGER NOT NULL DEFAULT 0, + LastError TEXT NULL, + HttpStatus INTEGER NULL, + CreatedAtUtc TEXT NOT NULL, + UpdatedAtUtc TEXT NOT NULL, + TerminalAtUtc TEXT NULL, + SourceInstanceId TEXT NULL, + SourceScript TEXT NULL + ); + """; + legacy.ExecuteNonQuery(); + } + + OperationTrackingSchema.Apply(conn); + + Assert.Contains(TableInfo(conn, "OperationTracking"), c => c.Name == "SourceNode"); + } +} From 2742d54c9f703c467ac8f7ff1277625a475f5d11 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sun, 19 Jul 2026 03:15:50 -0400 Subject: [PATCH 07/55] docs(localdb): record Task 1/2/5b completion + prerequisite commits Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts --- ...7-19-localdb-adoption-phase1.md.tasks.json | 23 +++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-) diff --git a/docs/plans/2026-07-19-localdb-adoption-phase1.md.tasks.json b/docs/plans/2026-07-19-localdb-adoption-phase1.md.tasks.json index 239a6171..8bbe3a4f 100644 --- a/docs/plans/2026-07-19-localdb-adoption-phase1.md.tasks.json +++ b/docs/plans/2026-07-19-localdb-adoption-phase1.md.tasks.json @@ -12,15 +12,30 @@ "date": "2026-07-19", "by": "execution review", "change": "Task 5 split into 5a (id 5, writer) and 5b (id 15, read path). Code recon found the integer site_events.id load-bearing in the keyset cursor (EventLogQueryService:70,141,153), the storage-cap purge (EventLogPurgeService:164), and long-typed DTOs crossing the site<->central Akka boundary (EventLogEntry.Id, both ContinuationTokens) - none of which were in the original Task 5 file set. 5b is high-risk. Task 10 gains a blockedBy on 15." + }, + { + "date": "2026-07-19", + "by": "execution", + "change": "Task 2 in isolation left the production writer broken (site_events.id NOT NULL, nothing minting it) - 40 tests red. Per the plan's own 'one PR-sized push' guidance, 5a's GUID minting and all of 5b landed together with Task 2 in commit 727fa48c. Task 5a remains OPEN for its other half: the ILocalDb.CreateConnection() swap, which still needs Task 3." + } + ], + "prerequisites": [ + { + "commit": "4715f2f9", + "what": "gRPC 2.71.0 -> 2.76.0 + Protobuf 3.34.1. Forced by LocalDb.Replication 0.1.0's nuspec floor; restore fails NU1605 below it. Directory.Packages.props had deferred this bump to 'its own reviewed commit' - honored." + }, + { + "commit": "f056b67e", + "what": "AngleSharp GHSA-pgww-w46g-26qg scoped suppression in CentralUI.Tests (supersedes the wrong pin in ecf6b628). Pre-existing: the solution build was RED on clean main, making every task's verification gate meaningless. No patched-and-working version exists upstream - 1.5.x breaks bunit at runtime, and bunit 2.7.2 still resolves vulnerable 1.4.0." } ], "tasks": [ - {"id": 1, "subject": "Task 1: Add LocalDb packages to the build", "status": "pending"}, - {"id": 2, "subject": "Task 2: Shared schema helpers (tracking + events, GUID PK)", "status": "pending", "blockedBy": [1]}, + {"id": 1, "subject": "Task 1: Add LocalDb packages to the build", "status": "completed", "commit": "7370b331"}, + {"id": 2, "subject": "Task 2: Shared schema helpers (tracking + events, GUID PK)", "status": "completed", "blockedBy": [1], "commit": "727fa48c"}, {"id": 3, "subject": "Task 3: Register LocalDb in the site composition root", "status": "pending", "blockedBy": [2]}, {"id": 4, "subject": "Task 4: Rewire OperationTrackingStore onto ILocalDb connections", "status": "pending", "blockedBy": [3]}, - {"id": 5, "subject": "Task 5a: Rewire SiteEventLogger onto ILocalDb + mint GUID ids", "status": "pending", "blockedBy": [3]}, - {"id": 15, "subject": "Task 5b: Migrate the event-log read path off integer ids", "status": "pending", "blockedBy": [5]}, + {"id": 5, "subject": "Task 5a: Rewire SiteEventLogger onto ILocalDb + mint GUID ids", "status": "partial", "blockedBy": [3], "commit": "727fa48c", "note": "GUID minting DONE (landed with Task 2 to avoid committing a broken writer). REMAINING: swap the owned SqliteConnection to ILocalDb.CreateConnection() - still blocked on Task 3."}, + {"id": 15, "subject": "Task 5b: Migrate the event-log read path off integer ids", "status": "completed", "blockedBy": [5], "commit": "727fa48c"}, {"id": 6, "subject": "Task 6: One-time legacy data migrator", "status": "pending", "blockedBy": [4, 5]}, {"id": 7, "subject": "Task 7: Replication registration + sync endpoint (default OFF)", "status": "pending", "blockedBy": [3]}, {"id": 8, "subject": "Task 8: Bearer auth interceptor for the sync endpoint", "status": "pending", "blockedBy": [7]}, From 2977e6c45cd0e05bc8937da8e45db8edc71aa306 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sun, 19 Jul 2026 03:36:54 -0400 Subject: [PATCH 08/55] feat(localdb): register the consolidated site database in the composition root Task 3. Site nodes now open one ZB.MOM.WW.LocalDb-managed database holding OperationTracking and site_events as replicated tables. SiteLocalDbSetup.OnReady applies both schemas and then RegisterReplicated's them. That order is load-bearing and easy to get silently wrong: capture is trigger-based CDC, so rows written before registration are never captured or snapshotted - they would be invisible to the peer forever, with no error anywhere. Storage ships UNCONDITIONALLY, no feature flag. With no peer configured LocalDb is just a local SQLite file, so this is behaviour-equivalent to what site nodes do today. Replication is opt-in and lands in Program.cs (Task 7), which owns the gRPC host the sync endpoint needs. Config: LocalDb:Path added to all EIGHT site-node appsettings (the plan said six - docker-env2's site-x pair exists too), plus the dev template and the wonder-app-vd03 production sample. All ten are required, not optional: LocalDbOptions.Path is validated with ValidateOnStart, so a site node without it fails fast at startup. That is the desired posture, and it is what caught ActorPathTests - the one Host fixture that actually starts a host - whose config now sets the path too. Note the path choice fixes an existing data-loss bug in passing: site-tracking.db and site_events.db both defaulted to CWD-relative paths (/app/...), outside the mounted volume, so they were silently lost on every container recreate. The consolidated database lives on /app/data. Tests are container-built against the REAL SiteServiceRegistration.Configure, not a hand-assembled ServiceCollection - this family has shipped three wiring defects that only a real graph would catch (inert Secrets replicator 0.2.0, singleton deadlock 0.2.2, Secrets.Ui missing IAuditWriter). They assert the library's own view of what is registered rather than that we called the right method. Verified (all after the change): dotnet build ZB.MOM.WW.ScadaBridge.slnx -> 0 Error(s), 0 Warning(s) Host.Tests -> 294 passed (5 new, red-first) SiteRuntime.Tests -> 530 passed CentralUI.Tests -> 925 passed Commons.Tests -> 684 passed Communication.Tests -> 312 passed SiteEventLogging.Tests-> 70 passed Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts --- .../site-x-node-a/appsettings.Site.json | 8 ++ .../site-x-node-b/appsettings.Site.json | 8 ++ docker/site-a-node-a/appsettings.Site.json | 8 ++ docker/site-a-node-b/appsettings.Site.json | 8 ++ docker/site-b-node-a/appsettings.Site.json | 8 ++ docker/site-b-node-b/appsettings.Site.json | 8 ++ docker/site-c-node-a/appsettings.Site.json | 8 ++ docker/site-c-node-b/appsettings.Site.json | 8 ++ .../SiteLocalDbSetup.cs | 51 +++++++ .../SiteServiceRegistration.cs | 13 ++ .../appsettings.Site.json | 6 + .../ActorPathTests.cs | 7 + .../SiteLocalDbWiringTests.cs | 133 ++++++++++++++++++ 13 files changed, 274 insertions(+) create mode 100644 src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbSetup.cs create mode 100644 tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbWiringTests.cs diff --git a/docker-env2/site-x-node-a/appsettings.Site.json b/docker-env2/site-x-node-a/appsettings.Site.json index 5c2ada30..edcd4240 100644 --- a/docker-env2/site-x-node-a/appsettings.Site.json +++ b/docker-env2/site-x-node-a/appsettings.Site.json @@ -56,5 +56,13 @@ "Logging": { "MinimumLevel": "Information" } + }, + // Consolidated site database (LocalDb Phase 1): OperationTracking + site_events. + // On the mounted /app/data volume so it survives container recreate - unlike the + // legacy site-tracking.db / site_events.db, which defaulted to CWD-relative paths + // outside the volume and were lost on every recreate. + // Replication is opt-in and configured separately; absent = local-only. + "LocalDb": { + "Path": "/app/data/site-localdb.db" } } diff --git a/docker-env2/site-x-node-b/appsettings.Site.json b/docker-env2/site-x-node-b/appsettings.Site.json index b53344a1..459dbf04 100644 --- a/docker-env2/site-x-node-b/appsettings.Site.json +++ b/docker-env2/site-x-node-b/appsettings.Site.json @@ -56,5 +56,13 @@ "Logging": { "MinimumLevel": "Information" } + }, + // Consolidated site database (LocalDb Phase 1): OperationTracking + site_events. + // On the mounted /app/data volume so it survives container recreate - unlike the + // legacy site-tracking.db / site_events.db, which defaulted to CWD-relative paths + // outside the volume and were lost on every recreate. + // Replication is opt-in and configured separately; absent = local-only. + "LocalDb": { + "Path": "/app/data/site-localdb.db" } } diff --git a/docker/site-a-node-a/appsettings.Site.json b/docker/site-a-node-a/appsettings.Site.json index 905ff385..7c841b41 100644 --- a/docker/site-a-node-a/appsettings.Site.json +++ b/docker/site-a-node-a/appsettings.Site.json @@ -57,5 +57,13 @@ "Logging": { "MinimumLevel": "Information" } + }, + // Consolidated site database (LocalDb Phase 1): OperationTracking + site_events. + // On the mounted /app/data volume so it survives container recreate - unlike the + // legacy site-tracking.db / site_events.db, which defaulted to CWD-relative paths + // outside the volume and were lost on every recreate. + // Replication is opt-in and configured separately; absent = local-only. + "LocalDb": { + "Path": "/app/data/site-localdb.db" } } diff --git a/docker/site-a-node-b/appsettings.Site.json b/docker/site-a-node-b/appsettings.Site.json index f1db4f2a..299055cc 100644 --- a/docker/site-a-node-b/appsettings.Site.json +++ b/docker/site-a-node-b/appsettings.Site.json @@ -57,5 +57,13 @@ "Logging": { "MinimumLevel": "Information" } + }, + // Consolidated site database (LocalDb Phase 1): OperationTracking + site_events. + // On the mounted /app/data volume so it survives container recreate - unlike the + // legacy site-tracking.db / site_events.db, which defaulted to CWD-relative paths + // outside the volume and were lost on every recreate. + // Replication is opt-in and configured separately; absent = local-only. + "LocalDb": { + "Path": "/app/data/site-localdb.db" } } diff --git a/docker/site-b-node-a/appsettings.Site.json b/docker/site-b-node-a/appsettings.Site.json index 93018c13..cd13388a 100644 --- a/docker/site-b-node-a/appsettings.Site.json +++ b/docker/site-b-node-a/appsettings.Site.json @@ -57,5 +57,13 @@ "Logging": { "MinimumLevel": "Information" } + }, + // Consolidated site database (LocalDb Phase 1): OperationTracking + site_events. + // On the mounted /app/data volume so it survives container recreate - unlike the + // legacy site-tracking.db / site_events.db, which defaulted to CWD-relative paths + // outside the volume and were lost on every recreate. + // Replication is opt-in and configured separately; absent = local-only. + "LocalDb": { + "Path": "/app/data/site-localdb.db" } } diff --git a/docker/site-b-node-b/appsettings.Site.json b/docker/site-b-node-b/appsettings.Site.json index 755b0b0a..9b373702 100644 --- a/docker/site-b-node-b/appsettings.Site.json +++ b/docker/site-b-node-b/appsettings.Site.json @@ -57,5 +57,13 @@ "Logging": { "MinimumLevel": "Information" } + }, + // Consolidated site database (LocalDb Phase 1): OperationTracking + site_events. + // On the mounted /app/data volume so it survives container recreate - unlike the + // legacy site-tracking.db / site_events.db, which defaulted to CWD-relative paths + // outside the volume and were lost on every recreate. + // Replication is opt-in and configured separately; absent = local-only. + "LocalDb": { + "Path": "/app/data/site-localdb.db" } } diff --git a/docker/site-c-node-a/appsettings.Site.json b/docker/site-c-node-a/appsettings.Site.json index 40d2269f..759cc46d 100644 --- a/docker/site-c-node-a/appsettings.Site.json +++ b/docker/site-c-node-a/appsettings.Site.json @@ -57,5 +57,13 @@ "Logging": { "MinimumLevel": "Information" } + }, + // Consolidated site database (LocalDb Phase 1): OperationTracking + site_events. + // On the mounted /app/data volume so it survives container recreate - unlike the + // legacy site-tracking.db / site_events.db, which defaulted to CWD-relative paths + // outside the volume and were lost on every recreate. + // Replication is opt-in and configured separately; absent = local-only. + "LocalDb": { + "Path": "/app/data/site-localdb.db" } } diff --git a/docker/site-c-node-b/appsettings.Site.json b/docker/site-c-node-b/appsettings.Site.json index 93a6b292..0aeba3aa 100644 --- a/docker/site-c-node-b/appsettings.Site.json +++ b/docker/site-c-node-b/appsettings.Site.json @@ -57,5 +57,13 @@ "Logging": { "MinimumLevel": "Information" } + }, + // Consolidated site database (LocalDb Phase 1): OperationTracking + site_events. + // On the mounted /app/data volume so it survives container recreate - unlike the + // legacy site-tracking.db / site_events.db, which defaulted to CWD-relative paths + // outside the volume and were lost on every recreate. + // Replication is opt-in and configured separately; absent = local-only. + "LocalDb": { + "Path": "/app/data/site-localdb.db" } } diff --git a/src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbSetup.cs b/src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbSetup.cs new file mode 100644 index 00000000..070dcd1e --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbSetup.cs @@ -0,0 +1,51 @@ +using ZB.MOM.WW.LocalDb; +using ZB.MOM.WW.ScadaBridge.SiteEventLogging; +using ZB.MOM.WW.ScadaBridge.SiteRuntime.Tracking; + +namespace ZB.MOM.WW.ScadaBridge.Host; + +/// +/// Initializes the consolidated site database — the single ZB.MOM.WW.LocalDb-managed +/// file that holds the site node's replicated state. +/// +/// +/// +/// Ordering is load-bearing. Table DDL must run BEFORE +/// RegisterReplicated, and every row that should replicate must be written +/// AFTER it. Capture is trigger-based change-data-capture: the library neither +/// captures nor snapshots rows that already existed when the triggers were installed, +/// so a row written before registration is invisible to the peer forever — silently, +/// with no error anywhere. +/// +/// +/// Runs inside the AddZbLocalDb onReady callback, once, before any caller +/// receives the singleton. onReady is a synchronous +/// Action<ILocalDb>, hence the direct CreateConnection() rather than +/// blocking on the async execute API. A throw here propagates out of the first +/// GetRequiredService<ILocalDb>() and fails host startup, which is the +/// intent: a site node that cannot establish its schema must not come up half-working. +/// +/// +public static class SiteLocalDbSetup +{ + /// + /// Creates the site node's replicated tables and opts them into change capture. + /// + /// The LocalDb instance being initialized. + public static void OnReady(ILocalDb db) + { + ArgumentNullException.ThrowIfNull(db); + + using (var connection = db.CreateConnection()) + { + OperationTrackingSchema.Apply(connection); + SiteEventLogSchema.Apply(connection); + } + + // Both tables qualify: each has an explicit primary key (RegisterReplicated + // rejects tables without one) and no BLOB columns (which json_object cannot + // capture). Registration is idempotent and installs the capture triggers. + db.RegisterReplicated("OperationTracking"); + db.RegisterReplicated("site_events"); + } +} diff --git a/src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs b/src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs index 6d282c33..3c5f691c 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs @@ -14,6 +14,7 @@ using ZB.MOM.WW.ScadaBridge.NotificationService; using ZB.MOM.WW.ScadaBridge.SiteEventLogging; using ZB.MOM.WW.ScadaBridge.SiteRuntime; using ZB.MOM.WW.ScadaBridge.StoreAndForward; +using ZB.MOM.WW.LocalDb; using ZB.MOM.WW.Secrets.DependencyInjection; using ZB.MOM.WW.Telemetry; @@ -49,6 +50,18 @@ public static class SiteServiceRegistration // and site-local repository implementations (IExternalSystemRepository, INotificationRepository) var siteDbPath = config["ScadaBridge:Database:SiteDbPath"] ?? "site.db"; services.AddSiteRuntime($"Data Source={siteDbPath}"); + + // Consolidated site database (LocalDb Phase 1). Holds OperationTracking and + // site_events as replicated tables so the pair stops losing them on failover. + // + // Storage is registered UNCONDITIONALLY and needs no flag: with no peer + // configured, LocalDb is simply a local SQLite file and the replication + // initiator idles. Replication itself is opt-in via LocalDb:Replication:PeerAddress + // and is wired in Program.cs, which owns the gRPC host the sync endpoint needs. + // + // Design: scadaproj docs/plans/2026-07-19-scadabridge-localdb-design.md + services.AddZbLocalDb(config, SiteLocalDbSetup.OnReady); + services.AddDataConnectionLayer(); // Local SQLite store by default; a local store replicating against a shared SQL-Server hub // only when Secrets:Replication:Enabled is true AND a hub connection string is present. diff --git a/src/ZB.MOM.WW.ScadaBridge.Host/appsettings.Site.json b/src/ZB.MOM.WW.ScadaBridge.Host/appsettings.Site.json index cf8d3fed..ca1c8315 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Host/appsettings.Site.json +++ b/src/ZB.MOM.WW.ScadaBridge.Host/appsettings.Site.json @@ -58,5 +58,11 @@ "Logging": { "MinimumLevel": "Information" } + }, + // Consolidated site database (LocalDb Phase 1): OperationTracking + site_events. + // LocalDb:Path is REQUIRED and validated on start - a site node with no value here + // fails to boot, so every site config must set it. + "LocalDb": { + "Path": "./data/site-localdb.db" } } diff --git a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/ActorPathTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/ActorPathTests.cs index 5b0b7bcd..dbd8de93 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/ActorPathTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/ActorPathTests.cs @@ -163,10 +163,12 @@ public class SiteActorPathTests : IAsyncLifetime private IHost? _host; private ActorSystem? _actorSystem; private string _tempDbPath = null!; + private string _tempLocalDbPath = null!; public async Task InitializeAsync() { _tempDbPath = Path.Combine(Path.GetTempPath(), $"scadabridge_actor_test_{Guid.NewGuid()}.db"); + _tempLocalDbPath = Path.Combine(Path.GetTempPath(), $"scadabridge_actor_localdb_{Guid.NewGuid()}.db"); var builder = Microsoft.Extensions.Hosting.Host.CreateDefaultBuilder(); builder.ConfigureAppConfiguration(config => @@ -183,6 +185,10 @@ public class SiteActorPathTests : IAsyncLifetime ["ScadaBridge:Cluster:SeedNodes:1"] = "akka.tcp://scadabridge@localhost:25520", ["ScadaBridge:Cluster:MinNrOfMembers"] = "1", ["ScadaBridge:Database:SiteDbPath"] = _tempDbPath, + // Required: LocalDbOptions.Path is validated with ValidateOnStart, and this + // fixture actually starts the host — a site node with no configured + // consolidated-database path fails fast rather than booting half-working. + ["LocalDb:Path"] = _tempLocalDbPath, // Configure a dummy central contact point to trigger ClusterClient creation ["ScadaBridge:Communication:CentralContactPoints:0"] = "akka.tcp://scadabridge@localhost:25510", }); @@ -207,6 +213,7 @@ public class SiteActorPathTests : IAsyncLifetime _host.Dispose(); } try { File.Delete(_tempDbPath); } catch { /* best effort */ } + try { File.Delete(_tempLocalDbPath); } catch { /* best effort */ } } [Fact] diff --git a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbWiringTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbWiringTests.cs new file mode 100644 index 00000000..28a6dc57 --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbWiringTests.cs @@ -0,0 +1,133 @@ +using Microsoft.AspNetCore.Builder; +using Microsoft.Extensions.Configuration; +using Microsoft.Extensions.DependencyInjection; +using ZB.MOM.WW.LocalDb; +using ZB.MOM.WW.ScadaBridge.Host; +using ZB.MOM.WW.ScadaBridge.Host.Actors; + +namespace ZB.MOM.WW.ScadaBridge.Host.Tests; + +/// +/// LocalDb Phase 1 (Task 3) — locks the consolidated site database into the REAL site +/// composition root. +/// +/// These tests build the actual container from +/// rather than asserting on a +/// hand-assembled ServiceCollection. That is deliberate and load-bearing: this +/// family has now shipped three wiring defects that only a container-built graph would +/// have caught — the Secrets Akka replicator that registered into a no-op sink (0.2.0), +/// the singleton cycle that deadlocked a real host (0.2.2), and Secrets.Ui's missing +/// IAuditWriter that 500'd only behind a login wall (ScadaBridge#22). A test that +/// merely checks "AddZbLocalDb was called" would have passed in every one of those cases. +/// +/// +public class SiteLocalDbWiringTests : IDisposable +{ + private readonly WebApplication _host; + private readonly string _tempDbPath; + private readonly string _tempSiteDbPath; + private readonly string _tempTrackingDbPath; + + public SiteLocalDbWiringTests() + { + var stamp = Guid.NewGuid(); + _tempDbPath = Path.Combine(Path.GetTempPath(), $"scadabridge_localdb_{stamp}.db"); + _tempSiteDbPath = Path.Combine(Path.GetTempPath(), $"scadabridge_site_{stamp}.db"); + _tempTrackingDbPath = Path.Combine(Path.GetTempPath(), $"scadabridge_track_{stamp}.db"); + + var builder = WebApplication.CreateBuilder(); + builder.Configuration.Sources.Clear(); + builder.Configuration.AddInMemoryCollection(new Dictionary + { + ["ScadaBridge:Node:Role"] = "Site", + ["ScadaBridge:Node:NodeName"] = "node-a", + ["ScadaBridge:Node:NodeHostname"] = "test-site", + ["ScadaBridge:Node:SiteId"] = "TestSite", + ["ScadaBridge:Node:RemotingPort"] = "0", + ["ScadaBridge:Node:GrpcPort"] = "0", + ["ScadaBridge:Database:SiteDbPath"] = _tempSiteDbPath, + ["ScadaBridge:OperationTracking:ConnectionString"] = $"Data Source={_tempTrackingDbPath}", + ["ScadaBridge:Cluster:SeedNodes:0"] = "akka.tcp://scadabridge@localhost:2551", + ["ScadaBridge:Cluster:SeedNodes:1"] = "akka.tcp://scadabridge@localhost:2552", + + // The consolidated site database. No Replication section at all — this + // fixture is also the default-OFF pin: storage must work standalone. + ["LocalDb:Path"] = _tempDbPath, + }); + + builder.Services.AddGrpc(); + builder.Services.AddSingleton(); + + SiteServiceRegistration.Configure(builder.Services, builder.Configuration); + + AkkaHostedServiceRemover.RemoveAkkaHostedServiceOnly(builder.Services); + + _host = builder.Build(); + } + + public void Dispose() + { + (_host as IDisposable)?.Dispose(); + foreach (var path in new[] { _tempDbPath, _tempSiteDbPath, _tempTrackingDbPath }) + { + try { File.Delete(path); } catch { /* best effort */ } + } + + GC.SuppressFinalize(this); + } + + [Fact] + public void Site_Resolves_ILocalDb() + { + Assert.NotNull(_host.Services.GetService()); + } + + [Fact] + public void Site_LocalDb_RegistersBothReplicatedTables() + { + // The whole point of Phase 1. If onReady silently failed to register these, + // storage would still work and every other test would pass — the pair would + // just never replicate anything. Assert on the library's own view of what is + // registered, not on our belief that we called it. + var db = _host.Services.GetRequiredService(); + + Assert.Contains("OperationTracking", db.ReplicatedTables.Keys); + Assert.Contains("site_events", db.ReplicatedTables.Keys); + } + + [Fact] + public void Site_LocalDb_ReplicatedTablesHaveExplicitPrimaryKeys() + { + // RegisterReplicated refuses a table with no explicit PK, so reaching this + // assertion at all proves the DDL ran before registration. The PK names are + // pinned because they are what last-writer-wins conflict resolution keys on. + var db = _host.Services.GetRequiredService(); + + Assert.Equal( + ["TrackedOperationId"], db.ReplicatedTables["OperationTracking"].PkColumns); + Assert.Equal(["id"], db.ReplicatedTables["site_events"].PkColumns); + } + + [Fact] + public void Site_LocalDb_IsASingleton() + { + // AddZbLocalDb is one-DB-per-process and first-registration-wins. Two + // instances would mean two SQLite handles and two trigger installations + // against the same file. + var first = _host.Services.GetRequiredService(); + var second = _host.Services.GetRequiredService(); + + Assert.Same(first, second); + } + + [Fact] + public void Site_LocalDb_CreatesTheConfiguredFile() + { + // Resolving is what triggers onReady; the file should exist afterwards at the + // configured path and not somewhere CWD-relative. + _ = _host.Services.GetRequiredService(); + + Assert.True(File.Exists(_tempDbPath), + $"Expected the consolidated site database at '{_tempDbPath}'."); + } +} From 2bff52724771214b8bb015ef4ae1e17bc97411e2 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sun, 19 Jul 2026 03:37:03 -0400 Subject: [PATCH 09/55] docs(localdb): record Task 3 completion Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts --- docs/plans/2026-07-19-localdb-adoption-phase1.md.tasks.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/plans/2026-07-19-localdb-adoption-phase1.md.tasks.json b/docs/plans/2026-07-19-localdb-adoption-phase1.md.tasks.json index 8bbe3a4f..19ff438b 100644 --- a/docs/plans/2026-07-19-localdb-adoption-phase1.md.tasks.json +++ b/docs/plans/2026-07-19-localdb-adoption-phase1.md.tasks.json @@ -32,7 +32,7 @@ "tasks": [ {"id": 1, "subject": "Task 1: Add LocalDb packages to the build", "status": "completed", "commit": "7370b331"}, {"id": 2, "subject": "Task 2: Shared schema helpers (tracking + events, GUID PK)", "status": "completed", "blockedBy": [1], "commit": "727fa48c"}, - {"id": 3, "subject": "Task 3: Register LocalDb in the site composition root", "status": "pending", "blockedBy": [2]}, + {"id": 3, "subject": "Task 3: Register LocalDb in the site composition root", "status": "completed", "blockedBy": [2], "commit": "2977e6c4", "note": "LocalDb:Path added to 10 configs (8 site nodes incl. docker-env2 site-x, dev template, wonder prod sample) - it is REQUIRED via ValidateOnStart, so any site config missing it fails startup."}, {"id": 4, "subject": "Task 4: Rewire OperationTrackingStore onto ILocalDb connections", "status": "pending", "blockedBy": [3]}, {"id": 5, "subject": "Task 5a: Rewire SiteEventLogger onto ILocalDb + mint GUID ids", "status": "partial", "blockedBy": [3], "commit": "727fa48c", "note": "GUID minting DONE (landed with Task 2 to avoid committing a broken writer). REMAINING: swap the owned SqliteConnection to ILocalDb.CreateConnection() - still blocked on Task 3."}, {"id": 15, "subject": "Task 5b: Migrate the event-log read path off integer ids", "status": "completed", "blockedBy": [5], "commit": "727fa48c"}, From 0b5e9b44f68e40573623957756472b8e5b3e076d Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sun, 19 Jul 2026 08:59:38 -0400 Subject: [PATCH 10/55] feat(localdb): rewire OperationTrackingStore onto the consolidated site database Task 4 of the LocalDb Phase 1 adoption plan. OperationTrackingStore took IOptions and opened its own SqliteConnection from ConnectionString. It now takes ILocalDb and gets every connection - writer and the two ad-hoc reader paths - from CreateConnection(). This is not cosmetic. OperationTracking is a RegisterReplicated table, so its capture triggers call zb_hlc_next(). That UDF is registered per connection by ILocalDb and by nothing else, so a raw SqliteConnection would fail closed on every write. Connections from CreateConnection() also arrive already open - calling Open()/OpenAsync() on one throws - hence the removed OpenAsync calls on the reader paths. OperationTrackingOptions.ConnectionString is now vestigial for this store; the database location is LocalDb:Path. The options class stays (retention settings) and the config key stays bound for the site config DB. InitializeSchema is kept but is now always a no-op in the host: onReady runs while ILocalDb is being constructed, strictly before this constructor can receive it. It remains so a directly-constructed store (tests, tooling) is self-sufficient. Tests: the store fixture moves off mode=memory&cache=shared onto a real ILocalDb over a temp file. There is no in-memory mode - LocalDbOptions.Path is a filesystem path - and testing through a raw in-memory connection would no longer resemble the host. Verifier connections stay raw on purpose: this fixture never registers the table, so no trigger fires and the UDF is never reached. Three DI fixtures needed LocalDb:Path added, because resolving IOperationTrackingStore now forces ILocalDb construction: SiteCompositionRootTests, SiteAuditWiringTests, and the AuditLog CombinedTelemetryHarness. Verified: build 0 warnings; Host 294/294, SiteRuntime 530/530, AuditLog 354/354. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts --- .../ServiceCollectionExtensions.cs | 6 +- .../Tracking/OperationTrackingStore.cs | 56 ++++--- .../CombinedTelemetryHarness.cs | 35 +++- .../AkkaHostedServiceAuditWiringTests.cs | 8 + .../CompositionRootTests.cs | 12 +- .../Tracking/OperationTrackingStoreTests.cs | 155 +++++++++++++----- 6 files changed, 198 insertions(+), 74 deletions(-) diff --git a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/ServiceCollectionExtensions.cs b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/ServiceCollectionExtensions.cs index 7352e990..3d499a2e 100644 --- a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/ServiceCollectionExtensions.cs +++ b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/ServiceCollectionExtensions.cs @@ -70,7 +70,11 @@ public static class ServiceCollectionExtensions // reconciliation, Tracking.Status audit-only). Central roots never call // AddSiteRuntime, so this stays absent on central — matching the SITE-ONLY // contract. OperationTrackingOptions is bound + validated by the Host's - // SiteServiceRegistration site-options block. + // SiteServiceRegistration site-options block — but it no longer supplies the + // store's database: the store takes ILocalDb (the consolidated site database + // at LocalDb:Path), so OperationTrackingOptions.ConnectionString is now only + // read by the site config DB. Resolving this store therefore forces ILocalDb + // construction, which is what runs SiteLocalDbSetup.OnReady. services.AddSingleton(); // Site-local repository implementations backed by SQLite diff --git a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Tracking/OperationTrackingStore.cs b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Tracking/OperationTrackingStore.cs index 1257ea10..feb1464a 100644 --- a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Tracking/OperationTrackingStore.cs +++ b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Tracking/OperationTrackingStore.cs @@ -1,7 +1,7 @@ using System.Globalization; using Microsoft.Data.Sqlite; using Microsoft.Extensions.Logging; -using Microsoft.Extensions.Options; +using ZB.MOM.WW.LocalDb; using ZB.MOM.WW.ScadaBridge.Commons.Interfaces; using ZB.MOM.WW.ScadaBridge.Commons.Types; @@ -39,7 +39,7 @@ public class OperationTrackingStore : IOperationTrackingStore, IAsyncDisposable, // _writeGate. Readers do NOT share this connection or gate; see GetStatusAsync. private readonly SqliteConnection _writeConnection; private readonly SemaphoreSlim _writeGate = new(1, 1); - private readonly string _connectionString; + private readonly ILocalDb _localDb; private readonly ILogger _logger; // Dispose-once state shared by the sync Dispose and async @@ -51,29 +51,43 @@ public class OperationTrackingStore : IOperationTrackingStore, IAsyncDisposable, private int _disposeState; /// - /// Initializes the tracking store, opens the SQLite connection, and applies the schema. + /// Initializes the tracking store over the consolidated site database and applies the schema. /// - /// Tracking store configuration (connection string, retention window). + /// + /// The consolidated site database. Every connection it hands out is already open and carries + /// the per-connection pragmas plus the zb_hlc_next() UDF that + /// OperationTracking's capture triggers call — which is exactly why the store no longer + /// opens its own from a connection string. A raw connection + /// would lack the UDF and every write to the replicated table would fail closed. + /// /// Logger for diagnostics. + /// + /// no longer feeds this store — the + /// database location is LocalDb:Path. The option remains bound for the purge/retention + /// settings that share the class. + /// public OperationTrackingStore( - IOptions options, + ILocalDb localDb, ILogger logger) { - ArgumentNullException.ThrowIfNull(options); + ArgumentNullException.ThrowIfNull(localDb); ArgumentNullException.ThrowIfNull(logger); _logger = logger; - _connectionString = options.Value.ConnectionString; - _writeConnection = new SqliteConnection(_connectionString); - _writeConnection.Open(); + _localDb = localDb; + // Already open — CreateConnection returns a live, pragma-configured, + // UDF-registered connection. Calling Open() on it again would throw. + _writeConnection = localDb.CreateConnection(); InitializeSchema(); } // Schema lives in OperationTrackingSchema so the Host's AddZbLocalDb onReady // callback can create this table in the consolidated site database before - // RegisterReplicated installs its capture triggers. Applying it here too keeps a - // directly-constructed store (tests, tooling) self-sufficient; the DDL is - // idempotent, so running it from both places is harmless. + // RegisterReplicated installs its capture triggers. In the host this call is + // therefore always a no-op — onReady runs while ILocalDb is being constructed, + // which is strictly before this constructor can receive it. It stays because it + // keeps a directly-constructed store (tests, tooling) self-sufficient, and the + // DDL is idempotent. private void InitializeSchema() => OperationTrackingSchema.Apply(_writeConnection); /// @@ -225,14 +239,12 @@ public class OperationTrackingStore : IOperationTrackingStore, IAsyncDisposable, { ObjectDisposedException.ThrowIf(Volatile.Read(ref _disposeState) != 0, this); - // Reads open a fresh, ungated SqliteConnection so a - // long-running write doesn't block status queries. The connection - // string is shared with the writer; SQLite handles cross-connection - // isolation natively (a reader sees a consistent snapshot via the - // shared cache lock for in-memory DBs, or a WAL snapshot for file DBs). - // Mirrors the SiteStorageService precedent. - await using var readConnection = new SqliteConnection(_connectionString); - await readConnection.OpenAsync(ct).ConfigureAwait(false); + // Reads open a fresh, ungated connection so a long-running write doesn't + // block status queries. It comes from the same ILocalDb as the writer; + // SQLite handles cross-connection isolation natively (readers see a WAL + // snapshot). Mirrors the SiteStorageService precedent. Already open — do + // not call OpenAsync on it. + await using var readConnection = _localDb.CreateConnection(); await using var cmd = readConnection.CreateCommand(); cmd.CommandText = """ @@ -312,8 +324,8 @@ public class OperationTrackingStore : IOperationTrackingStore, IAsyncDisposable, // the standalone IX_OperationTracking_UpdatedAt index — UpdatedAtUtc is // the cursor. (The composite (Status, UpdatedAtUtc) index cannot satisfy a // status-less UpdatedAtUtc range scan; this dedicated index does.) - await using var readConnection = new SqliteConnection(_connectionString); - await readConnection.OpenAsync(ct).ConfigureAwait(false); + // Already open — do not call OpenAsync on it. + await using var readConnection = _localDb.CreateConnection(); await using var cmd = readConnection.CreateCommand(); diff --git a/tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Integration/Infrastructure/CombinedTelemetryHarness.cs b/tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Integration/Infrastructure/CombinedTelemetryHarness.cs index 3c83dd97..c7bdeff9 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Integration/Infrastructure/CombinedTelemetryHarness.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Integration/Infrastructure/CombinedTelemetryHarness.cs @@ -1,8 +1,10 @@ using Akka.Actor; using Microsoft.EntityFrameworkCore; +using Microsoft.Extensions.Configuration; using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.Logging.Abstractions; using Microsoft.Extensions.Options; +using ZB.MOM.WW.LocalDb; using ZB.MOM.WW.ScadaBridge.AuditLog.Central; using ZB.MOM.WW.ScadaBridge.AuditLog.Site; using ZB.MOM.WW.ScadaBridge.AuditLog.Site.Telemetry; @@ -58,6 +60,8 @@ public sealed class CombinedTelemetryHarness : IAsyncDisposable public IServiceProvider ServiceProvider { get; } private readonly MsSqlMigrationFixture _fixture; + private readonly string _trackingDbPath; + private readonly ServiceProvider _trackingLocalDbProvider; private bool _disposed; public CombinedTelemetryHarness( @@ -88,13 +92,23 @@ public sealed class CombinedTelemetryHarness : IAsyncDisposable SqliteWriter, Ring, new NoOpAuditWriteFailureCounter(), NullLogger.Instance); + // The tracking store now runs on the consolidated ILocalDb rather than its own + // connection string, so it needs a real database FILE — LocalDbOptions.Path is + // a filesystem path and there is no in-memory mode. Unique per harness, torn + // down in DisposeAsync alongside its WAL sidecars. + _trackingDbPath = Path.Combine( + Path.GetTempPath(), $"tracking-g-{Guid.NewGuid():N}.db"); + _trackingLocalDbProvider = new ServiceCollection() + .AddZbLocalDb(new ConfigurationBuilder() + .AddInMemoryCollection(new Dictionary + { + ["LocalDb:Path"] = _trackingDbPath, + }) + .Build()) + .BuildServiceProvider(); + TrackingStore = new OperationTrackingStore( - Options.Create(new OperationTrackingOptions - { - // Same shared-in-memory pattern as the audit writer. - ConnectionString = - $"Data Source=file:tracking-g-{Guid.NewGuid():N}?mode=memory&cache=shared", - }), + _trackingLocalDbProvider.GetRequiredService(), NullLogger.Instance); // Central wiring: real repositories backed by the MSSQL fixture's DB. @@ -165,6 +179,15 @@ public sealed class CombinedTelemetryHarness : IAsyncDisposable _disposed = true; await SqliteWriter.DisposeAsync().ConfigureAwait(false); await TrackingStore.DisposeAsync().ConfigureAwait(false); + + // Dispose the ILocalDb before deleting the file — it holds an open master + // connection anchoring the WAL. + await _trackingLocalDbProvider.DisposeAsync().ConfigureAwait(false); + foreach (var suffix in new[] { "", "-wal", "-shm" }) + { + try { File.Delete(_trackingDbPath + suffix); } catch { /* best effort */ } + } + if (ServiceProvider is IAsyncDisposable asyncSp) { await asyncSp.DisposeAsync().ConfigureAwait(false); diff --git a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/AkkaHostedServiceAuditWiringTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/AkkaHostedServiceAuditWiringTests.cs index d31f27a8..31e21242 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/AkkaHostedServiceAuditWiringTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/AkkaHostedServiceAuditWiringTests.cs @@ -290,10 +290,12 @@ public class SiteAuditWiringTests : IDisposable { private readonly WebApplication _host; private readonly string _tempDbPath; + private readonly string _tempLocalDbPath; public SiteAuditWiringTests() { _tempDbPath = Path.Combine(Path.GetTempPath(), $"scadabridge_audit_wiring_{Guid.NewGuid()}.db"); + _tempLocalDbPath = Path.Combine(Path.GetTempPath(), $"scadabridge_audit_wiring_localdb_{Guid.NewGuid()}.db"); var builder = WebApplication.CreateBuilder(); builder.Configuration.Sources.Clear(); @@ -312,6 +314,11 @@ public class SiteAuditWiringTests : IDisposable // resolved; point it at an in-memory connection so the test doesn't // pollute the working directory. ["AuditLog:SiteWriter:DatabasePath"] = ":memory:", + // The cached-call telemetry forwarder pulls in IOperationTrackingStore, + // which now resolves ILocalDb — so the consolidated site database must be + // configured or the whole graph fails to resolve. There is no in-memory + // mode: LocalDbOptions.Path is a filesystem path. + ["LocalDb:Path"] = _tempLocalDbPath, }); builder.Services.AddGrpc(); @@ -326,6 +333,7 @@ public class SiteAuditWiringTests : IDisposable { (_host as IDisposable)?.Dispose(); try { File.Delete(_tempDbPath); } catch { /* best effort */ } + try { File.Delete(_tempLocalDbPath); } catch { /* best effort */ } } [Fact] diff --git a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CompositionRootTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CompositionRootTests.cs index 5d89eb9d..488f186f 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CompositionRootTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CompositionRootTests.cs @@ -383,11 +383,13 @@ public class SiteCompositionRootTests : IDisposable private readonly WebApplication _host; private readonly string _tempDbPath; private readonly string _tempTrackingDbPath; + private readonly string _tempLocalDbPath; public SiteCompositionRootTests() { _tempDbPath = Path.Combine(Path.GetTempPath(), $"scadabridge_test_{Guid.NewGuid()}.db"); _tempTrackingDbPath = Path.Combine(Path.GetTempPath(), $"scadabridge_track_{Guid.NewGuid()}.db"); + _tempLocalDbPath = Path.Combine(Path.GetTempPath(), $"scadabridge_localdb_{Guid.NewGuid()}.db"); var builder = WebApplication.CreateBuilder(); builder.Configuration.Sources.Clear(); @@ -400,10 +402,13 @@ public class SiteCompositionRootTests : IDisposable ["ScadaBridge:Node:RemotingPort"] = "0", ["ScadaBridge:Node:GrpcPort"] = "0", ["ScadaBridge:Database:SiteDbPath"] = _tempDbPath, - // Point the site-local operation-tracking SQLite store (the ctor opens/creates - // the file eagerly) at a temp path so resolving IOperationTrackingStore in the - // composition-root test does not litter site-tracking.db in the test cwd. + // Retained for the site config DB; it no longer points the tracking store + // anywhere — that store now opens the consolidated database below. ["ScadaBridge:OperationTracking:ConnectionString"] = $"Data Source={_tempTrackingDbPath}", + // The consolidated site database. IOperationTrackingStore resolves ILocalDb + // and creates the file eagerly in its ctor, so this must be a temp path or + // the composition-root test litters the working directory. + ["LocalDb:Path"] = _tempLocalDbPath, // ClusterOptions requires at least one seed node (ClusterOptionsValidator). ["ScadaBridge:Cluster:SeedNodes:0"] = "akka.tcp://scadabridge@localhost:2551", ["ScadaBridge:Cluster:SeedNodes:1"] = "akka.tcp://scadabridge@localhost:2552", @@ -427,6 +432,7 @@ public class SiteCompositionRootTests : IDisposable (_host as IDisposable)?.Dispose(); try { File.Delete(_tempDbPath); } catch { /* best effort */ } try { File.Delete(_tempTrackingDbPath); } catch { /* best effort */ } + try { File.Delete(_tempLocalDbPath); } catch { /* best effort */ } } [Fact] diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Tracking/OperationTrackingStoreTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Tracking/OperationTrackingStoreTests.cs index 8ae7ebb4..c327324b 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Tracking/OperationTrackingStoreTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Tracking/OperationTrackingStoreTests.cs @@ -1,6 +1,8 @@ using Microsoft.Data.Sqlite; +using Microsoft.Extensions.Configuration; +using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.Logging.Abstractions; -using Microsoft.Extensions.Options; +using ZB.MOM.WW.LocalDb; using ZB.MOM.WW.ScadaBridge.Commons.Types; using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Tracking; @@ -9,30 +11,93 @@ namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Tracking; /// /// Audit Log #23 (M3 Bundle A — Task A2) — schema + behaviour tests for the -/// site-local . Each test uses a unique -/// shared-cache in-memory SQLite database so the store and the verifier share -/// the same store without touching disk. +/// site-local . /// -public class OperationTrackingStoreTests +/// +/// The store now takes rather than a connection string, so each +/// test gets a real ILocalDb over a unique temp FILE. The previous +/// mode=memory&cache=shared fixture is gone and cannot come back: +/// LocalDbOptions.Path is a filesystem path, and — more importantly — a raw +/// connection to a shared-cache in-memory database would not carry the +/// zb_hlc_next() UDF the capture triggers call, so testing through one would no +/// longer resemble how the store runs in the host. +/// +/// Verifier connections are plain s over the same file. +/// That is deliberate and safe: they only read, or write columns on an unregistered +/// table (this fixture never calls RegisterReplicated), so no trigger fires and +/// the missing UDF is never reached. +/// +/// +public class OperationTrackingStoreTests : IDisposable { - private static (OperationTrackingStore store, string dataSource) CreateStore( - string testName) + // Every ILocalDb and temp file created by this fixture, torn down together. The + // ILocalDb holds an open master connection anchoring the WAL, so it must be + // disposed before the file can be deleted. + private readonly List _providers = []; + private readonly List _tempFiles = []; + + public void Dispose() { - var dataSource = $"file:{testName}-{Guid.NewGuid():N}?mode=memory&cache=shared"; - var connectionString = $"Data Source={dataSource};Cache=Shared"; - var options = new OperationTrackingOptions + foreach (var provider in _providers) { - ConnectionString = connectionString, - }; - var store = new OperationTrackingStore( - Options.Create(options), - NullLogger.Instance); - return (store, dataSource); + try { provider.Dispose(); } catch { /* best effort */ } + } + + foreach (var path in _tempFiles) + { + foreach (var suffix in new[] { "", "-wal", "-shm" }) + { + try { File.Delete(path + suffix); } catch { /* best effort */ } + } + } + + GC.SuppressFinalize(this); } - private static SqliteConnection OpenVerifierConnection(string dataSource) + /// Allocates a unique temp database path, registered for cleanup. + private string NewDatabasePath(string testName) { - var connection = new SqliteConnection($"Data Source={dataSource};Cache=Shared"); + var path = Path.Combine( + Path.GetTempPath(), $"optracking-{testName}-{Guid.NewGuid():N}.db"); + _tempFiles.Add(path); + return path; + } + + /// + /// Builds a real over . No + /// onReady callback and no RegisterReplicated — the store's own + /// InitializeSchema creates the table, which is what keeps a + /// directly-constructed store self-sufficient. + /// + private ILocalDb CreateLocalDb(string databasePath) + { + var configuration = new ConfigurationBuilder() + .AddInMemoryCollection(new Dictionary + { + ["LocalDb:Path"] = databasePath, + }) + .Build(); + + var provider = new ServiceCollection() + .AddZbLocalDb(configuration) + .BuildServiceProvider(); + _providers.Add(provider); + + return provider.GetRequiredService(); + } + + private (OperationTrackingStore store, string dataSource) CreateStore(string testName) + { + var databasePath = NewDatabasePath(testName); + var store = new OperationTrackingStore( + CreateLocalDb(databasePath), + NullLogger.Instance); + return (store, databasePath); + } + + private static SqliteConnection OpenVerifierConnection(string databasePath) + { + var connection = new SqliteConnection($"Data Source={databasePath}"); connection.Open(); return connection; } @@ -111,14 +176,19 @@ public class OperationTrackingStoreTests ON OperationTracking (Status, UpdatedAtUtc); """; - private static SqliteConnection SeedPreSourceNodeSchemaDatabase(string dataSource) + /// + /// Writes the pre-SourceNode schema into and closes + /// the connection again. Unlike the old shared-cache in-memory fixture — where the + /// seeding connection had to stay open or the database evaporated — a file database + /// persists on its own, so nothing is held across the store's construction. + /// + private static void SeedPreSourceNodeSchemaDatabase(string databasePath) { - var connection = new SqliteConnection($"Data Source={dataSource};Cache=Shared"); + using var connection = new SqliteConnection($"Data Source={databasePath}"); connection.Open(); using var cmd = connection.CreateCommand(); cmd.CommandText = OldPreSourceNodeSchema; cmd.ExecuteNonQuery(); - return connection; } private static bool ColumnExists(SqliteConnection connection, string columnName) @@ -129,36 +199,36 @@ public class OperationTrackingStoreTests return Convert.ToInt32(cmd.ExecuteScalar()) > 0; } - private static OperationTrackingStore CreateStoreOver(string dataSource) - { - var connectionString = $"Data Source={dataSource};Cache=Shared"; - var options = new OperationTrackingOptions { ConnectionString = connectionString }; - return new OperationTrackingStore( - Options.Create(options), - NullLogger.Instance); - } + private OperationTrackingStore CreateStoreOver(string databasePath) + => new(CreateLocalDb(databasePath), NullLogger.Instance); [Fact] public async Task Initialize_adds_SourceNode_to_pre_existing_schema() { - var dataSource = $"file:{nameof(Initialize_adds_SourceNode_to_pre_existing_schema)}-{Guid.NewGuid():N}?mode=memory&cache=shared"; + var databasePath = NewDatabasePath(nameof(Initialize_adds_SourceNode_to_pre_existing_schema)); - // A pre-SourceNode deployment: tracking.db already exists with the - // 12-column schema and NO SourceNode column. - using var seedConnection = SeedPreSourceNodeSchemaDatabase(dataSource); - Assert.True(ColumnExists(seedConnection, "SourceInstanceId")); - Assert.True(ColumnExists(seedConnection, "SourceScript")); - Assert.False(ColumnExists(seedConnection, "SourceNode")); + // A pre-SourceNode deployment: the tracking database already exists with + // the 12-column schema and NO SourceNode column. + SeedPreSourceNodeSchemaDatabase(databasePath); + using (var seeded = OpenVerifierConnection(databasePath)) + { + Assert.True(ColumnExists(seeded, "SourceInstanceId")); + Assert.True(ColumnExists(seeded, "SourceScript")); + Assert.False(ColumnExists(seeded, "SourceNode")); + } // Upgrade: a post-branch OperationTrackingStore opens the same database. // Its InitializeSchema must ALTER the missing SourceNode column in — // the CREATE TABLE IF NOT EXISTS alone is a no-op against the existing // table. - await using (var store = CreateStoreOver(dataSource)) + await using (var store = CreateStoreOver(databasePath)) { - Assert.True( - ColumnExists(seedConnection, "SourceNode"), - "OperationTrackingStore must ALTER the SourceNode column into a pre-existing OperationTracking table."); + using (var verifier = OpenVerifierConnection(databasePath)) + { + Assert.True( + ColumnExists(verifier, "SourceNode"), + "OperationTrackingStore must ALTER the SourceNode column into a pre-existing OperationTracking table."); + } // A RecordEnqueueAsync binding $sourceNode must now succeed; without // the ALTER it would fail with "no such column: SourceNode". @@ -178,9 +248,10 @@ public class OperationTrackingStoreTests // Idempotency: a second store over the now-upgraded DB must not error // (the probe sees SourceNode already present and skips the ALTER). - await using (var storeAgain = CreateStoreOver(dataSource)) + await using (var storeAgain = CreateStoreOver(databasePath)) { - Assert.True(ColumnExists(seedConnection, "SourceNode")); + using var verifier = OpenVerifierConnection(databasePath); + Assert.True(ColumnExists(verifier, "SourceNode")); } } From 0d8a80aa27f5f13642292f80e691c8a30079f509 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sun, 19 Jul 2026 09:07:49 -0400 Subject: [PATCH 11/55] feat(localdb): rewire SiteEventLogger onto the consolidated site database Completes Task 5a of the LocalDb Phase 1 adoption plan. The GUID-id half landed with Task 2 (727fa48c) because leaving it out would have committed a writer that could not satisfy site_events.id NOT NULL; this is the connection half, which was blocked on Task 3. SiteEventLogger owned a SqliteConnection built from SiteEventLogOptions .DatabasePath. It now takes ILocalDb and gets that connection from CreateConnection() - already open, pragma-configured, and carrying the zb_hlc_next() UDF that site_events' capture triggers call. It still owns and disposes the connection; WithConnection's signature and locking semantics are untouched, so EventLogQueryService and EventLogPurgeService are unaffected. The connectionStringOverride seam is deleted rather than preserved. site_events is a replicated table, so a raw connection lacks the UDF and fails closed on every insert - an override could only produce a logger that cannot write. It had no callers outside this class (the connectionStringOverride hits elsewhere in the tree belong to SqliteAuditWriter, a different type). Tests: a shared TestLocalDb helper gives each fixture a real ILocalDb over a temp file. Real rather than stubbed on purpose - a stub handing back a bare SqliteConnection would pass today and fail closed the moment the table is registered, which is exactly the silent-wiring failure mode this family has hit three times. ServiceWiringTests' DI path also needed AddZbLocalDb, mirroring SiteServiceRegistration. Verified: build 0 warnings; SiteEventLogging 70/70, Host 294/294, SiteRuntime 530/530. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts --- .../SiteEventLogger.cs | 38 ++++++---- .../EventLogCoverageTests.cs | 11 ++- .../EventLogPurgeServiceTests.cs | 9 ++- .../EventLogQueryServiceTests.cs | 14 ++-- .../SchemaIndexTests.cs | 7 +- .../ServiceWiringTests.cs | 16 ++++- .../SiteEventLoggerAsyncTests.cs | 18 +++-- .../SiteEventLoggerTests.cs | 7 +- .../TestLocalDb.cs | 70 +++++++++++++++++++ 9 files changed, 154 insertions(+), 36 deletions(-) create mode 100644 tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/TestLocalDb.cs diff --git a/src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs b/src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs index 53ff5add..a5c6e868 100644 --- a/src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs +++ b/src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs @@ -2,6 +2,7 @@ using System.Threading.Channels; using Microsoft.Data.Sqlite; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Options; +using ZB.MOM.WW.LocalDb; namespace ZB.MOM.WW.ScadaBridge.SiteEventLogging; @@ -40,29 +41,36 @@ public class SiteEventLogger : ISiteEventLogger, IDisposable private bool _disposed; /// - /// Initializes the event logger, opens the SQLite connection, and starts the background writer loop. + /// Initializes the event logger over the consolidated site database and starts the + /// background writer loop. /// - /// Site event log configuration (database path, retention settings). + /// Site event log configuration (retention settings, queue capacity). /// Logger for write-failure diagnostics. - /// Optional connection string override; uses the configured path when null. + /// + /// The consolidated site database. The connection it hands out is already open and + /// carries the zb_hlc_next() UDF that site_events' capture triggers + /// call. This logger owns the connection for its lifetime and disposes it. + /// + /// + /// The former connectionStringOverride seam is deliberately gone rather than + /// preserved. site_events is a replicated table, so a raw connection lacking + /// the UDF would fail closed on every insert — an override would only let a caller + /// build a logger that cannot write. SiteEventLogOptions.DatabasePath is + /// likewise no longer read: the location is LocalDb:Path. + /// public SiteEventLogger( IOptions options, ILogger logger, - string? connectionStringOverride = null) + ILocalDb localDb) { + ArgumentNullException.ThrowIfNull(options); + ArgumentNullException.ThrowIfNull(localDb); + _logger = logger; - // Cache=Shared is a cross-connection optimisation - // that lets multiple SqliteConnections share an in-process page cache. - // This logger owns exactly one SqliteConnection and serialises all - // access through _writeLock, so the mode is dormant — at best dead - // configuration, at worst a small future foot-gun for any second - // connection opened to the same file. A test path that genuinely - // needs Cache=Shared can still inject it via connectionStringOverride. - var connectionString = connectionStringOverride - ?? $"Data Source={options.Value.DatabasePath}"; - _connection = new SqliteConnection(connectionString); - _connection.Open(); + // Already open — CreateConnection returns a live, pragma-configured, + // UDF-registered connection. Calling Open() on it again would throw. + _connection = localDb.CreateConnection(); InitializeSchema(); diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/EventLogCoverageTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/EventLogCoverageTests.cs index da267200..b2a6e9b1 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/EventLogCoverageTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/EventLogCoverageTests.cs @@ -11,20 +11,27 @@ namespace ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests; public class EventLogCoverageTests : IDisposable { private readonly string _dbPath; + private readonly TestLocalDb _localDb; public EventLogCoverageTests() { _dbPath = Path.Combine(Path.GetTempPath(), $"test_coverage_{Guid.NewGuid()}.db"); + _localDb = TestLocalDb.Create(_dbPath); } public void Dispose() { - if (File.Exists(_dbPath)) File.Delete(_dbPath); + _localDb.Dispose(); + TestLocalDb.DeleteFiles(_dbPath); + GC.SuppressFinalize(this); } + // Each call gets its own connection from the one shared local database, so the + // loggers these tests dispose mid-test do not tear down the database itself. private SiteEventLogger NewLogger() => new( Options.Create(new SiteEventLogOptions { DatabasePath = _dbPath }), - NullLogger.Instance); + NullLogger.Instance, + _localDb.Db); private static EventLogQueryRequest MakeRequest() => new( CorrelationId: "corr-err", diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/EventLogPurgeServiceTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/EventLogPurgeServiceTests.cs index 8725d83a..22240faa 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/EventLogPurgeServiceTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/EventLogPurgeServiceTests.cs @@ -6,6 +6,7 @@ namespace ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests; public class EventLogPurgeServiceTests : IDisposable { private readonly SiteEventLogger _eventLogger; + private readonly TestLocalDb _localDb; private readonly string _dbPath; private readonly SiteEventLogOptions _options; @@ -27,15 +28,19 @@ public class EventLogPurgeServiceTests : IDisposable RetentionDays = 30, MaxStorageMb = 1024 }; + _localDb = TestLocalDb.Create(_dbPath); _eventLogger = new SiteEventLogger( Options.Create(_options), - NullLogger.Instance); + NullLogger.Instance, + _localDb.Db); } public void Dispose() { _eventLogger.Dispose(); - if (File.Exists(_dbPath)) File.Delete(_dbPath); + _localDb.Dispose(); + TestLocalDb.DeleteFiles(_dbPath); + GC.SuppressFinalize(this); } private EventLogPurgeService CreatePurgeService( diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/EventLogQueryServiceTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/EventLogQueryServiceTests.cs index 82cac861..a7b55f96 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/EventLogQueryServiceTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/EventLogQueryServiceTests.cs @@ -8,6 +8,7 @@ public class EventLogQueryServiceTests : IDisposable { private readonly SiteEventLogger _eventLogger; private readonly EventLogQueryService _queryService; + private readonly TestLocalDb _localDb; private readonly string _dbPath; public EventLogQueryServiceTests() @@ -18,7 +19,8 @@ public class EventLogQueryServiceTests : IDisposable DatabasePath = _dbPath, QueryPageSize = 500 }); - _eventLogger = new SiteEventLogger(options, NullLogger.Instance); + _localDb = TestLocalDb.Create(_dbPath); + _eventLogger = new SiteEventLogger(options, NullLogger.Instance, _localDb.Db); _queryService = new EventLogQueryService( _eventLogger, options, @@ -28,7 +30,9 @@ public class EventLogQueryServiceTests : IDisposable public void Dispose() { _eventLogger.Dispose(); - if (File.Exists(_dbPath)) File.Delete(_dbPath); + _localDb.Dispose(); + TestLocalDb.DeleteFiles(_dbPath); + GC.SuppressFinalize(this); } private async Task SeedEvents() @@ -367,7 +371,8 @@ public class EventLogQueryServiceTests : IDisposable QueryPageSize = 500, MaxQueryPageSize = 3, }); - using var logger = new SiteEventLogger(options, NullLogger.Instance); + using var localDb = TestLocalDb.Create(dbPath); + using var logger = new SiteEventLogger(options, NullLogger.Instance, localDb.Db); var query = new EventLogQueryService(logger, options, NullLogger.Instance); try @@ -391,7 +396,8 @@ public class EventLogQueryServiceTests : IDisposable finally { logger.Dispose(); - if (File.Exists(dbPath)) File.Delete(dbPath); + localDb.Dispose(); + TestLocalDb.DeleteFiles(dbPath); } } } diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/SchemaIndexTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/SchemaIndexTests.cs index 0038ad4c..7f70a4cb 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/SchemaIndexTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/SchemaIndexTests.cs @@ -13,12 +13,14 @@ public class SchemaIndexTests : IDisposable private readonly SiteEventLogger _logger; private readonly SqliteConnection _verifyConnection; private readonly string _dbPath; + private readonly TestLocalDb _localDb; public SchemaIndexTests() { _dbPath = Path.Combine(Path.GetTempPath(), $"test_index_{Guid.NewGuid()}.db"); var options = Options.Create(new SiteEventLogOptions { DatabasePath = _dbPath }); - _logger = new SiteEventLogger(options, NullLogger.Instance); + _localDb = TestLocalDb.Create(_dbPath); + _logger = new SiteEventLogger(options, NullLogger.Instance, _localDb.Db); _verifyConnection = new SqliteConnection($"Data Source={_dbPath}"); _verifyConnection.Open(); @@ -28,7 +30,8 @@ public class SchemaIndexTests : IDisposable { _verifyConnection.Dispose(); _logger.Dispose(); - if (File.Exists(_dbPath)) File.Delete(_dbPath); + _localDb.Dispose(); + TestLocalDb.DeleteFiles(_dbPath); } [Fact] diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/ServiceWiringTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/ServiceWiringTests.cs index 12546e64..9068edfb 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/ServiceWiringTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/ServiceWiringTests.cs @@ -1,6 +1,8 @@ +using Microsoft.Extensions.Configuration; using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Options; +using ZB.MOM.WW.LocalDb; namespace ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests; @@ -23,7 +25,7 @@ public class ServiceWiringTests : IDisposable public void Dispose() { _provider?.Dispose(); - if (File.Exists(_dbPath)) File.Delete(_dbPath); + TestLocalDb.DeleteFiles(_dbPath); } [Fact] @@ -32,6 +34,15 @@ public class ServiceWiringTests : IDisposable var services = new ServiceCollection(); services.AddLogging(); services.Configure(o => o.DatabasePath = _dbPath); + // SiteEventLogger now takes ILocalDb, so the consolidated site database has to + // be in the graph for AddSiteEventLogging to resolve at all. In the host this + // comes from SiteServiceRegistration's AddZbLocalDb. + services.AddZbLocalDb(new ConfigurationBuilder() + .AddInMemoryCollection(new Dictionary + { + ["LocalDb:Path"] = _dbPath, + }) + .Build()); services.AddSiteEventLogging(); _provider = services.BuildServiceProvider(); @@ -57,8 +68,9 @@ public class ServiceWiringTests : IDisposable // concrete SiteEventLogger. Constructing them directly must compile and work. var options = Options.Create(new SiteEventLogOptions { DatabasePath = _dbPath }); using var loggerFactory = LoggerFactory.Create(_ => { }); + using var localDb = TestLocalDb.Create(_dbPath); using var recorder = new SiteEventLogger( - options, loggerFactory.CreateLogger()); + options, loggerFactory.CreateLogger(), localDb.Db); var purge = new EventLogPurgeService( recorder, options, loggerFactory.CreateLogger()); diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/SiteEventLoggerAsyncTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/SiteEventLoggerAsyncTests.cs index 4d617ff6..e469a4ec 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/SiteEventLoggerAsyncTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/SiteEventLoggerAsyncTests.cs @@ -13,18 +13,21 @@ public class SiteEventLoggerAsyncTests : IDisposable { private readonly SiteEventLogger _logger; private readonly string _dbPath; + private readonly TestLocalDb _localDb; public SiteEventLoggerAsyncTests() { _dbPath = Path.Combine(Path.GetTempPath(), $"test_async_{Guid.NewGuid()}.db"); var options = Options.Create(new SiteEventLogOptions { DatabasePath = _dbPath }); - _logger = new SiteEventLogger(options, NullLogger.Instance); + _localDb = TestLocalDb.Create(_dbPath); + _logger = new SiteEventLogger(options, NullLogger.Instance, _localDb.Db); } public void Dispose() { _logger.Dispose(); - if (File.Exists(_dbPath)) File.Delete(_dbPath); + _localDb.Dispose(); + TestLocalDb.DeleteFiles(_dbPath); } [Fact] @@ -91,11 +94,10 @@ public class SiteEventLoggerAsyncTests : IDisposable // Health Monitoring can surface a logging outage. using var failingConnection = new SqliteConnection("Data Source=:memory:"); failingConnection.Open(); - var options = Options.Create(new SiteEventLogOptions - { - DatabasePath = Path.Combine(Path.GetTempPath(), $"test_fail_{Guid.NewGuid()}.db") - }); - var logger = new SiteEventLogger(options, NullLogger.Instance); + var failDbPath = Path.Combine(Path.GetTempPath(), $"test_fail_{Guid.NewGuid()}.db"); + var options = Options.Create(new SiteEventLogOptions { DatabasePath = failDbPath }); + using var failLocalDb = TestLocalDb.Create(failDbPath); + var logger = new SiteEventLogger(options, NullLogger.Instance, failLocalDb.Db); // Drop the table so every write fails with "no such table". logger.WithConnection(connection => @@ -112,5 +114,7 @@ public class SiteEventLoggerAsyncTests : IDisposable "FailedWriteCount must increment when an event write fails."); logger.Dispose(); + failLocalDb.Dispose(); + TestLocalDb.DeleteFiles(failDbPath); } } diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/SiteEventLoggerTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/SiteEventLoggerTests.cs index 81ae9137..c2921663 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/SiteEventLoggerTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/SiteEventLoggerTests.cs @@ -9,12 +9,14 @@ public class SiteEventLoggerTests : IDisposable private readonly SiteEventLogger _logger; private readonly SqliteConnection _verifyConnection; private readonly string _dbPath; + private readonly TestLocalDb _localDb; public SiteEventLoggerTests() { _dbPath = Path.Combine(Path.GetTempPath(), $"test_events_{Guid.NewGuid()}.db"); var options = Options.Create(new SiteEventLogOptions { DatabasePath = _dbPath }); - _logger = new SiteEventLogger(options, NullLogger.Instance); + _localDb = TestLocalDb.Create(_dbPath); + _logger = new SiteEventLogger(options, NullLogger.Instance, _localDb.Db); // Separate connection for verification queries _verifyConnection = new SqliteConnection($"Data Source={_dbPath}"); @@ -25,7 +27,8 @@ public class SiteEventLoggerTests : IDisposable { _verifyConnection.Dispose(); _logger.Dispose(); - if (File.Exists(_dbPath)) File.Delete(_dbPath); + _localDb.Dispose(); + TestLocalDb.DeleteFiles(_dbPath); } [Fact] diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/TestLocalDb.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/TestLocalDb.cs new file mode 100644 index 00000000..a5df1b4a --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/TestLocalDb.cs @@ -0,0 +1,70 @@ +using Microsoft.Extensions.Configuration; +using Microsoft.Extensions.DependencyInjection; +using ZB.MOM.WW.LocalDb; + +namespace ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests; + +/// +/// A real over a temp file, for tests that construct a +/// directly. +/// +/// +/// +/// takes rather than a connection +/// string, and there is no in-memory mode — LocalDbOptions.Path is a filesystem +/// path. A real one is used rather than a stub on purpose: connections handed out by +/// ILocalDb carry the zb_hlc_next() UDF that site_events' capture +/// triggers call, so a stub returning a bare SqliteConnection would pass these +/// tests while failing closed the moment the table is registered for replication. +/// +/// +/// No onReady callback and no RegisterReplicated: the logger's own +/// InitializeSchema creates the table, which is what keeps a directly-constructed +/// logger self-sufficient. Registration is the host's job +/// (SiteLocalDbSetup.OnReady). +/// +/// +public sealed class TestLocalDb : IDisposable +{ + private readonly ServiceProvider _provider; + + private TestLocalDb(ServiceProvider provider, ILocalDb db) + { + _provider = provider; + Db = db; + } + + /// The live local database. + public ILocalDb Db { get; } + + /// Opens a local database at . + public static TestLocalDb Create(string databasePath) + { + var configuration = new ConfigurationBuilder() + .AddInMemoryCollection(new Dictionary + { + ["LocalDb:Path"] = databasePath, + }) + .Build(); + + var provider = new ServiceCollection() + .AddZbLocalDb(configuration) + .BuildServiceProvider(); + + return new TestLocalDb(provider, provider.GetRequiredService()); + } + + /// + /// Deletes and its WAL sidecars. Call only after the + /// owning is disposed — the master connection anchors the WAL. + /// + public static void DeleteFiles(string databasePath) + { + foreach (var suffix in new[] { "", "-wal", "-shm" }) + { + try { File.Delete(databasePath + suffix); } catch { /* best effort */ } + } + } + + public void Dispose() => _provider.Dispose(); +} From a560e9eaf949639444beae38b6af1115d412d9b6 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sun, 19 Jul 2026 09:15:26 -0400 Subject: [PATCH 12/55] feat(localdb): wire 2-node replication for the site database, default OFF Task 7 of the LocalDb Phase 1 adoption plan. AddZbLocalDbReplication registers the engine; MapZbLocalDbSync exposes the passive endpoint the peer node dials. Both are unconditional but INERT by default: with no LocalDb:Replication:PeerAddress the initiating SyncBackgroundService starts and immediately idles, and nothing dials the endpoint. A site pair replicates only once an operator sets a peer on BOTH nodes. The endpoint shares the Kestrel h2c listener the site gRPC server already uses, so there are no listener or port changes. Deviation from the plan: it put AddZbLocalDbReplication in Program.cs. Doing so would have left it untested - the composition-root tests build SiteServiceRegistration's graph, not Program.cs - so a registration that silently went missing would still show green. It now sits next to AddZbLocalDb in SiteServiceRegistration, where the DI-graph tests actually cover it. MapZbLocalDbSync stays in Program.cs because it needs the WebApplication. Two pins added to SiteLocalDbWiringTests, which configures no peer: - the engine resolves and is idle (not connected, no peer, never synced) - default-off means inert, not unregistered; - OplogBacklog is 0, not null. It is deliberately nullable so a failed poll reads as "unknown" instead of a healthy zero; a real 0 proves the provider is wired to the oplog, which Task 9's health signal depends on. Inbound auth on the sync endpoint is Task 8. Until that lands the endpoint is unauthenticated - which is precisely why replication ships default-OFF and no config in this repo sets a peer. Verified: build 0 warnings; Host 296/296, SiteEventLogging 70/70, SiteRuntime 530/530, Commons 684/684, Communication 312/312. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts --- src/ZB.MOM.WW.ScadaBridge.Host/Program.cs | 12 +++++++- .../SiteServiceRegistration.cs | 15 ++++++++-- .../SiteLocalDbWiringTests.cs | 30 +++++++++++++++++++ 3 files changed, 54 insertions(+), 3 deletions(-) diff --git a/src/ZB.MOM.WW.ScadaBridge.Host/Program.cs b/src/ZB.MOM.WW.ScadaBridge.Host/Program.cs index 3ef98a9a..e8ea49ab 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Host/Program.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Host/Program.cs @@ -7,6 +7,7 @@ using ZB.MOM.WW.Auth.AspNetCore; using ZB.MOM.WW.Health; using ZB.MOM.WW.Health.Akka; using ZB.MOM.WW.Health.EntityFrameworkCore; +using ZB.MOM.WW.LocalDb.Replication; using ZB.MOM.WW.ScadaBridge.AuditLog; using ZB.MOM.WW.ScadaBridge.CentralUI; using ZB.MOM.WW.ScadaBridge.ClusterInfrastructure; @@ -516,7 +517,8 @@ try builder.Services.AddGrpc(); builder.Services.AddSingleton(); - // Existing site service registrations + // Existing site service registrations (this is also where LocalDb and its + // replication engine are registered — see SiteServiceRegistration) SiteServiceRegistration.Configure(builder.Services, builder.Configuration); var app = builder.Build(); @@ -535,6 +537,14 @@ try // Map gRPC service — resolves the singleton SiteStreamGrpcServer from DI app.MapGrpcService(); + // The passive half of LocalDb replication: the peer node dials THIS endpoint. + // It shares the Kestrel h2c listener the site gRPC server already uses, so no + // listener or port changes are needed. Mapping it is harmless with no peer + // configured — nothing dials it. Inbound auth is the host's job and lands in + // Task 8; until then this endpoint is unauthenticated, which is why + // replication ships default-OFF. + app.MapZbLocalDbSync(); + // Site-shutdown ordering. ApplicationStopping // fires BEFORE IHostedService.StopAsync runs, so the gRPC server // refuses new streams (Unavailable) and cancels every active stream diff --git a/src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs b/src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs index 3c5f691c..3abee357 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs @@ -15,6 +15,7 @@ using ZB.MOM.WW.ScadaBridge.SiteEventLogging; using ZB.MOM.WW.ScadaBridge.SiteRuntime; using ZB.MOM.WW.ScadaBridge.StoreAndForward; using ZB.MOM.WW.LocalDb; +using ZB.MOM.WW.LocalDb.Replication; using ZB.MOM.WW.Secrets.DependencyInjection; using ZB.MOM.WW.Telemetry; @@ -56,12 +57,22 @@ public static class SiteServiceRegistration // // Storage is registered UNCONDITIONALLY and needs no flag: with no peer // configured, LocalDb is simply a local SQLite file and the replication - // initiator idles. Replication itself is opt-in via LocalDb:Replication:PeerAddress - // and is wired in Program.cs, which owns the gRPC host the sync endpoint needs. + // initiator idles. // // Design: scadaproj docs/plans/2026-07-19-scadabridge-localdb-design.md services.AddZbLocalDb(config, SiteLocalDbSetup.OnReady); + // The replication engine, likewise unconditional but INERT by default: with no + // LocalDb:Replication:PeerAddress the initiating SyncBackgroundService starts and + // immediately idles, and nothing dials the passive endpoint. A site pair + // replicates only once an operator sets a peer on both nodes. + // + // Registered HERE rather than in Program.cs (where the plan first put it) so the + // composition-root tests, which build this graph and not Program.cs, actually + // cover it. The endpoint half — MapZbLocalDbSync — has to stay in Program.cs + // because it needs the WebApplication. + services.AddZbLocalDbReplication(config); + services.AddDataConnectionLayer(); // Local SQLite store by default; a local store replicating against a shared SQL-Server hub // only when Secrets:Replication:Enabled is true AND a hub connection string is present. diff --git a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbWiringTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbWiringTests.cs index 28a6dc57..db34fa6b 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbWiringTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbWiringTests.cs @@ -2,6 +2,7 @@ using Microsoft.AspNetCore.Builder; using Microsoft.Extensions.Configuration; using Microsoft.Extensions.DependencyInjection; using ZB.MOM.WW.LocalDb; +using ZB.MOM.WW.LocalDb.Replication; using ZB.MOM.WW.ScadaBridge.Host; using ZB.MOM.WW.ScadaBridge.Host.Actors; @@ -120,6 +121,35 @@ public class SiteLocalDbWiringTests : IDisposable Assert.Same(first, second); } + [Fact] + public void Site_Replication_IsRegistered_AndInertWithNoPeer() + { + // Task 7's default-OFF pin. This fixture configures NO + // LocalDb:Replication:PeerAddress, which is the shipping default, so the + // engine must resolve cleanly and sit idle rather than throw, retry, or + // report a connection. "Default-off" here means inert, NOT unregistered — + // the services are always in the graph. + var status = _host.Services.GetService(); + + Assert.NotNull(status); + Assert.False(status!.Connected); + Assert.Null(status.PeerNodeId); + Assert.Null(status.LastSyncUtc); + } + + [Fact] + public void Site_Replication_OplogBacklog_IsZero_NotNull_OnAHealthyIdleNode() + { + // OplogBacklog is deliberately nullable: null means "unknown" (no provider + // wired, or the poll failed) and must never be reported as a healthy 0. On a + // node whose local database is present and readable, a real 0 proves the + // backlog provider is actually wired to the oplog — a null here would mean + // Task 9's health signal is about to publish "unknown" forever. + var status = _host.Services.GetRequiredService(); + + Assert.Equal(0L, status.OplogBacklog); + } + [Fact] public void Site_LocalDb_CreatesTheConfiguredFile() { From e62b076f2e3b1e5375ef3d745e308ea820fcf3c6 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sun, 19 Jul 2026 09:15:52 -0400 Subject: [PATCH 13/55] docs(localdb): record wave 4 completion (Tasks 4, 5a, 7) + known flakes Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts --- ...6-07-19-localdb-adoption-phase1.md.tasks.json | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/docs/plans/2026-07-19-localdb-adoption-phase1.md.tasks.json b/docs/plans/2026-07-19-localdb-adoption-phase1.md.tasks.json index 19ff438b..98d191c1 100644 --- a/docs/plans/2026-07-19-localdb-adoption-phase1.md.tasks.json +++ b/docs/plans/2026-07-19-localdb-adoption-phase1.md.tasks.json @@ -33,11 +33,11 @@ {"id": 1, "subject": "Task 1: Add LocalDb packages to the build", "status": "completed", "commit": "7370b331"}, {"id": 2, "subject": "Task 2: Shared schema helpers (tracking + events, GUID PK)", "status": "completed", "blockedBy": [1], "commit": "727fa48c"}, {"id": 3, "subject": "Task 3: Register LocalDb in the site composition root", "status": "completed", "blockedBy": [2], "commit": "2977e6c4", "note": "LocalDb:Path added to 10 configs (8 site nodes incl. docker-env2 site-x, dev template, wonder prod sample) - it is REQUIRED via ValidateOnStart, so any site config missing it fails startup."}, - {"id": 4, "subject": "Task 4: Rewire OperationTrackingStore onto ILocalDb connections", "status": "pending", "blockedBy": [3]}, - {"id": 5, "subject": "Task 5a: Rewire SiteEventLogger onto ILocalDb + mint GUID ids", "status": "partial", "blockedBy": [3], "commit": "727fa48c", "note": "GUID minting DONE (landed with Task 2 to avoid committing a broken writer). REMAINING: swap the owned SqliteConnection to ILocalDb.CreateConnection() - still blocked on Task 3."}, + {"id": 4, "subject": "Task 4: Rewire OperationTrackingStore onto ILocalDb connections", "status": "completed", "blockedBy": [3], "commit": "0b5e9b44", "note": "Ctor takes ILocalDb, not IOptions; both ad-hoc reader paths switched too and their OpenAsync calls removed (CreateConnection returns an OPEN connection). Three DI fixtures needed LocalDb:Path added because resolving IOperationTrackingStore now forces ILocalDb construction."}, + {"id": 5, "subject": "Task 5a: Rewire SiteEventLogger onto ILocalDb + mint GUID ids", "status": "completed", "blockedBy": [3], "commit": "727fa48c + 0d8a80aa", "note": "GUID minting landed early with Task 2 (727fa48c) to avoid committing a writer that violated site_events.id NOT NULL. The ILocalDb.CreateConnection() swap is 0d8a80aa. connectionStringOverride DELETED, not preserved: site_events is replicated, so a raw connection lacks zb_hlc_next() and fails closed."}, {"id": 15, "subject": "Task 5b: Migrate the event-log read path off integer ids", "status": "completed", "blockedBy": [5], "commit": "727fa48c"}, {"id": 6, "subject": "Task 6: One-time legacy data migrator", "status": "pending", "blockedBy": [4, 5]}, - {"id": 7, "subject": "Task 7: Replication registration + sync endpoint (default OFF)", "status": "pending", "blockedBy": [3]}, + {"id": 7, "subject": "Task 7: Replication registration + sync endpoint (default OFF)", "status": "completed", "blockedBy": [3], "commit": "a560e9ea", "note": "DEVIATION: AddZbLocalDbReplication moved from Program.cs (where the plan put it) into SiteServiceRegistration, because the composition-root tests build that graph and not Program.cs - in Program.cs a missing registration would still show green. MapZbLocalDbSync stays in Program.cs (needs the WebApplication). Endpoint is UNAUTHENTICATED until Task 8; no config in this repo sets a peer."}, {"id": 8, "subject": "Task 8: Bearer auth interceptor for the sync endpoint", "status": "pending", "blockedBy": [7]}, {"id": 9, "subject": "Task 9: Surface ISyncStatus as site health signal", "status": "pending", "blockedBy": [7]}, {"id": 10, "subject": "Task 10: Two-node in-process convergence test", "status": "pending", "blockedBy": [6, 8, 15]}, @@ -46,5 +46,15 @@ {"id": 13, "subject": "Task 13: Documentation truth pass", "status": "pending", "blockedBy": [12]}, {"id": 14, "subject": "Task 14: Phase 2 gate - plan the bespoke-replicator migration", "status": "pending", "blockedBy": [12]} ], + "knownFlakes": [ + { + "test": "SiteRuntime.Tests InstanceActorChildAttributeRaceTests.ChildActors_AreSeededFromAnIsolatedCopy_NotTheLiveAttributesDictionary", + "note": "Intermittent ActorNotFoundException under full-suite load; passes in isolation and in most full runs. Pre-existing, unrelated to LocalDb. Worth its own issue." + }, + { + "test": "AuditLog.Tests ParentExecutionIdCorrelationTests.InboundRoutedRun_AllRoutedRows_CarryInboundExecutionId_AsParentExecutionId", + "note": "Cold-MSSQL-fixture timing: ~91s and AwaitAssert-times-out on a cold fixture, ~1s and passes warm. Baselined against a stashed tree to confirm it is not a LocalDb regression." + } + ], "lastUpdated": "2026-07-19" } From 59c695191cd9d980daf302d6932f9051273f624a Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sun, 19 Jul 2026 09:29:22 -0400 Subject: [PATCH 14/55] feat(localdb): fail-closed auth on the sync endpoint + replication health signal Tasks 8 and 9 of the LocalDb Phase 1 adoption plan. Task 8 - LocalDbSyncAuthInterceptor. The replication library's LocalDbSyncService verifies nothing; inbound auth is explicitly the host's job. Without this, anything able to reach a site node's gRPC port could stream arbitrary rows into the consolidated site database - including OperationTracking, which central reconciles from. Scoped strictly to /localdb_sync.v1.LocalDbSync/; SiteStream shares the same AddGrpc pipeline and passes through untouched. Fail-closed: with no LocalDb:Replication:ApiKey configured NO sync stream is accepted, authenticated or not. That is deliberate - "no key" is the default every site node ships with, so treating it as "no auth required" would expose the endpoint on precisely the most common configuration. Comparison is FixedTimeEquals over UTF-8 bytes. All four server handler shapes are gated, not just unary: the sync RPC is a bidirectional stream, so gating only the unary path would leave the real endpoint open while every unary test still passed. There is a test for that. Deviation from the plan: it specified Grpc.Core.Testing for the fake ServerCallContext. That type ships in the retired native Grpc.Core package and does not exist on the grpc-dotnet stack this solution uses; a minimal FakeServerCallContext in the test file was the better trade than adding a dead dependency. Task 9 - ISyncStatus onto the site health report as LocalDbReplicationConnected and LocalDbOplogBacklog, via a delegate-seam hosted service following the AddSiteEventLogHealthMetricsBridge precedent (HealthMonitoring takes no reference on the replication library). Both are additive init properties, so the Akka-remoted SiteHealthReport constructor signature is untouched. Both fields are nullable and the distinction is load-bearing: - null = the reporter has not run / replication is not wired ("no data"); - false/0 = a real reading. On a node with no peer that IS the healthy default-OFF state, not an outage. OplogBacklog is passed through nullable end-to-end because ISyncStatus returns null when the poll fails - flattening it to 0 would report a replication pair that cannot read its own oplog as perfectly healthy. The collector stores both values as one tuple and CollectReport reads it once, so a torn read cannot pair a fresh Connected with a stale backlog. Verified: build 0 warnings; Host 307/307 (8 interceptor + 3 health tests new), HealthMonitoring 97/97, Commons 684/684. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts --- .../Messages/Health/SiteHealthReport.cs | 31 ++++ .../ISiteHealthCollector.cs | 19 ++ .../LocalDbReplicationStatusReporter.cs | 130 +++++++++++++ .../ServiceCollectionExtensions.cs | 46 +++++ .../SiteHealthCollector.cs | 20 +- .../LocalDbSyncAuthInterceptor.cs | 158 ++++++++++++++++ src/ZB.MOM.WW.ScadaBridge.Host/Program.cs | 11 +- .../SiteServiceRegistration.cs | 10 + .../LocalDbSyncAuthInterceptorTests.cs | 171 ++++++++++++++++++ .../SiteLocalDbWiringTests.cs | 46 +++++ 10 files changed, 637 insertions(+), 5 deletions(-) create mode 100644 src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/LocalDbReplicationStatusReporter.cs create mode 100644 src/ZB.MOM.WW.ScadaBridge.Host/LocalDbSyncAuthInterceptor.cs create mode 100644 tests/ZB.MOM.WW.ScadaBridge.Host.Tests/LocalDbSyncAuthInterceptorTests.cs diff --git a/src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Health/SiteHealthReport.cs b/src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Health/SiteHealthReport.cs index 216bd656..5e498b78 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Health/SiteHealthReport.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Commons/Messages/Health/SiteHealthReport.cs @@ -73,6 +73,37 @@ public record SiteHealthReport( /// indicates a stuck/blocked script holding a dedicated thread. /// public double? ScriptOldestBusyAgeSeconds { get; init; } + + // LocalDb 2-node replication of the consolidated site database (Phase 1). + // Additive init properties for the same reason as the scheduler gauges above: + // the positional constructor stays untouched. Refreshed on the site by + // LocalDbReplicationStatusReporter. + + /// + /// Whether a replication sync session is currently running with the peer site node, + /// or when replication is not wired on this node. + /// + /// + /// Nullable on purpose, and is NOT the same as null. + /// Replication ships default-OFF, so a site node with no peer configured reports + /// — that is a healthy, expected state, not an outage. Only + /// a node that was configured with a peer and is reporting + /// is degraded, and distinguishing those two cases is the operator's job, not this + /// field's. + /// + public bool? LocalDbReplicationConnected { get; init; } + + /// + /// Unacked replication oplog entries waiting to reach the peer, or + /// when unknown. + /// + /// + /// Null means unknown, never zero. The underlying provider returns null when + /// no backlog source is wired or the poll failed, and that must survive to the + /// operator: a failed read rendered as "0 backlog" would report a broken replication + /// pair as perfectly healthy. Collapsing null to 0 anywhere on this path is a bug. + /// + public long? LocalDbOplogBacklog { get; init; } } /// diff --git a/src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/ISiteHealthCollector.cs b/src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/ISiteHealthCollector.cs index 3f9ab84d..8781553a 100644 --- a/src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/ISiteHealthCollector.cs +++ b/src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/ISiteHealthCollector.cs @@ -141,6 +141,25 @@ public interface ISiteHealthCollector // SiteHealthCollector overrides this with the Interlocked.Exchange store. } + /// + /// Replace the latest LocalDb replication status (peer-session connectivity and + /// unacked oplog backlog) used by the next call. + /// Refreshed periodically by the LocalDbReplicationStatusReporter hosted + /// service. Point-in-time: values are NOT reset on . + /// + /// + /// Whether a sync session is currently running. is the + /// normal state on a node with no peer configured — replication ships default-OFF. + /// + /// + /// Unacked oplog entries, or when unknown. Pass null through + /// unchanged: a failed poll rendered as 0 would report a broken pair as healthy. + /// + void SetLocalDbReplicationStatus(bool connected, long? oplogBacklog) + { + // Default no-op so test fakes do not need to be updated. + } + /// /// Replace the latest script-execution-scheduler gauges (queue depth, busy /// thread count, and age in seconds of the oldest in-flight script) used by diff --git a/src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/LocalDbReplicationStatusReporter.cs b/src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/LocalDbReplicationStatusReporter.cs new file mode 100644 index 00000000..39f14ff1 --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/LocalDbReplicationStatusReporter.cs @@ -0,0 +1,130 @@ +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Logging; + +namespace ZB.MOM.WW.ScadaBridge.HealthMonitoring; + +/// +/// Site-side hosted service that periodically reads LocalDb replication status and pushes +/// it into , so the next +/// emits fresh +/// LocalDbReplicationConnected / LocalDbOplogBacklog fields. +/// +/// +/// +/// Why delegates and not ISyncStatus directly. Same reasoning as +/// : HealthMonitoring does not take a +/// reference on the replication library. The Host site wiring captures the two reads as +/// lambdas at registration time; this service only moves numbers. +/// +/// +/// Null backlog is preserved, never coerced to zero. ISyncStatus.OplogBacklog +/// is nullable precisely so a failed poll reads as "unknown" — rendering it as 0 would +/// report a replication pair that cannot read its own oplog as perfectly healthy. +/// +/// +/// Cadence. 30 s, matching and +/// SiteAuditBacklogReporter. Any exception during a probe is logged and swallowed; +/// the next tick retries. +/// +/// +public sealed class LocalDbReplicationStatusReporter : IHostedService, IDisposable +{ + /// Default poll cadence, matching the other site health bridges. + internal static readonly TimeSpan DefaultRefreshInterval = TimeSpan.FromSeconds(30); + + private readonly Func _connectedProvider; + private readonly Func _oplogBacklogProvider; + private readonly ISiteHealthCollector _collector; + private readonly ILogger _logger; + private readonly TimeSpan _refreshInterval; + private CancellationTokenSource? _cts; + private Task? _loop; + + /// Initializes a new instance of . + /// Reads whether a peer sync session is currently running. + /// Reads the unacked oplog backlog, or null when unknown. + /// The site health collector receiving the snapshot. + /// Logger instance. + /// Poll interval override; defaults to 30 s. + public LocalDbReplicationStatusReporter( + Func connectedProvider, + Func oplogBacklogProvider, + ISiteHealthCollector collector, + ILogger logger, + TimeSpan? refreshInterval = null) + { + _connectedProvider = connectedProvider ?? throw new ArgumentNullException(nameof(connectedProvider)); + _oplogBacklogProvider = oplogBacklogProvider ?? throw new ArgumentNullException(nameof(oplogBacklogProvider)); + _collector = collector ?? throw new ArgumentNullException(nameof(collector)); + _logger = logger ?? throw new ArgumentNullException(nameof(logger)); + _refreshInterval = refreshInterval ?? DefaultRefreshInterval; + } + + /// Starts the polling loop, probing once immediately before entering the timed cycle. + /// Cancellation token signalling host shutdown. + /// A task that represents the asynchronous operation. + public Task StartAsync(CancellationToken ct) + { + var cts = CancellationTokenSource.CreateLinkedTokenSource(ct); + _cts = cts; + _loop = RunAsync(cts.Token); + return Task.CompletedTask; + } + + /// Stops the polling loop. + /// Cancellation token for the stop operation. + /// A task that represents the asynchronous operation. + public async Task StopAsync(CancellationToken ct) + { + if (_cts is null) return; + + await _cts.CancelAsync().ConfigureAwait(false); + if (_loop is not null) + { + // Await the loop so the reporter is quiescent before the host disposes the + // collector out from under it. + try { await _loop.ConfigureAwait(false); } + catch (OperationCanceledException) { /* expected on shutdown */ } + } + } + + private async Task RunAsync(CancellationToken ct) + { + while (!ct.IsCancellationRequested) + { + Probe(); + + try + { + await Task.Delay(_refreshInterval, ct).ConfigureAwait(false); + } + catch (OperationCanceledException) + { + return; + } + } + } + + /// + /// Reads both providers and pushes one snapshot onto the collector; never throws. + /// Public so a test can drive a single deterministic probe instead of starting the + /// service and waiting out the 30 s cadence. + /// + public void Probe() + { + try + { + _collector.SetLocalDbReplicationStatus( + _connectedProvider(), _oplogBacklogProvider()); + } + catch (Exception ex) + { + // Health reporting must never take the site node down. The previous snapshot + // stays on the collector and the next tick retries. + _logger.LogWarning(ex, "Failed to read LocalDb replication status for the site health report."); + } + } + + /// + public void Dispose() => _cts?.Dispose(); +} diff --git a/src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/ServiceCollectionExtensions.cs b/src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/ServiceCollectionExtensions.cs index e4587f24..d8d3b457 100644 --- a/src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/ServiceCollectionExtensions.cs +++ b/src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/ServiceCollectionExtensions.cs @@ -21,6 +21,12 @@ public static class ServiceCollectionExtensions /// private sealed class SiteEventLogHealthMetricsBridgeMarker { } + /// + /// Sentinel marker for 's idempotency + /// guard — same rationale as . + /// + private sealed class LocalDbReplicationHealthBridgeMarker { } + /// /// Register site-side health monitoring services (metric collection + periodic reporting). /// Call this on site nodes only. For central, call AddCentralHealthAggregation() instead. @@ -153,6 +159,46 @@ public static class ServiceCollectionExtensions return services; } + /// + /// Bridge LocalDb replication status (peer connectivity + unacked oplog backlog) onto + /// the site health report. Must be called AFTER AddSiteHealthMonitoring + /// (registers ) and after the replication engine is + /// registered. Idempotent via a marker sentinel. + /// + /// + /// Delegates rather than an ISyncStatus parameter keep HealthMonitoring free of + /// a reference on the replication library, matching + /// . Pass the backlog through as + /// nullable — coercing null to 0 would report an unreadable oplog as a healthy one. + /// + /// The service collection to register into. + /// Given the root provider, returns a reader for "a sync session is running". + /// Given the root provider, returns a reader for the unacked backlog (null = unknown). + /// The same for chaining. + public static IServiceCollection AddLocalDbReplicationHealthBridge( + this IServiceCollection services, + Func> connectedProvider, + Func> oplogBacklogProvider) + { + ArgumentNullException.ThrowIfNull(services); + ArgumentNullException.ThrowIfNull(connectedProvider); + ArgumentNullException.ThrowIfNull(oplogBacklogProvider); + + if (services.Any(d => d.ServiceType == typeof(LocalDbReplicationHealthBridgeMarker))) + { + return services; + } + + services.AddSingleton(); + services.AddHostedService(sp => new LocalDbReplicationStatusReporter( + connectedProvider(sp), + oplogBacklogProvider(sp), + sp.GetRequiredService(), + sp.GetRequiredService>())); + + return services; + } + /// /// Register the /// so a misconfigured ScadaBridge:HealthMonitoring section (zero/negative diff --git a/src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/SiteHealthCollector.cs b/src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/SiteHealthCollector.cs index 51f22a7e..be008182 100644 --- a/src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/SiteHealthCollector.cs +++ b/src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/SiteHealthCollector.cs @@ -18,6 +18,10 @@ public class SiteHealthCollector : ISiteHealthCollector private int _auditRedactionFailures; private volatile SiteAuditBacklogSnapshot? _siteAuditBacklog; private long _siteEventLogWriteFailures; + // One volatile tuple rather than two independent fields: a torn read that paired a + // fresh Connected with a stale backlog would be indistinguishable from a real state. + // Null = the reporter has not yet run (or replication is not wired). + private volatile Tuple? _localDbReplicationStatus; private readonly ConcurrentDictionary _connectionStatuses = new(); private readonly ConcurrentDictionary _tagResolutionCounts = new(); private readonly ConcurrentDictionary _connectionEndpoints = new(); @@ -94,6 +98,12 @@ public class SiteHealthCollector : ISiteHealthCollector Interlocked.Exchange(ref _siteEventLogWriteFailures, count); } + /// + public void SetLocalDbReplicationStatus(bool connected, long? oplogBacklog) + { + _localDbReplicationStatus = Tuple.Create(connected, oplogBacklog); + } + /// public void UpdateConnectionHealth(string connectionName, ConnectionHealth health) { @@ -219,6 +229,9 @@ public class SiteHealthCollector : ISiteHealthCollector var deadLetters = Interlocked.Exchange(ref _deadLetterCount, 0); var siteAuditWriteFailures = Interlocked.Exchange(ref _siteAuditWriteFailures, 0); var auditRedactionFailures = Interlocked.Exchange(ref _auditRedactionFailures, 0); + // Single read of the volatile tuple — two reads could straddle a reporter tick and + // pair a fresh Connected with a stale backlog. + var localDbReplication = _localDbReplicationStatus; // Snapshot current connection and tag resolution state var connectionStatuses = new Dictionary(_connectionStatuses); @@ -259,7 +272,12 @@ public class SiteHealthCollector : ISiteHealthCollector { ScriptQueueDepth = Interlocked.CompareExchange(ref _scriptQueueDepth, 0, 0), ScriptBusyThreads = Interlocked.CompareExchange(ref _scriptBusyThreads, 0, 0), - ScriptOldestBusyAgeSeconds = ReadScriptOldestBusyAgeSeconds() + ScriptOldestBusyAgeSeconds = ReadScriptOldestBusyAgeSeconds(), + // Both fields come from the ONE snapshot read above. Null (the reporter has + // not run) leaves both report fields null — "no data", not "disconnected + // with an empty backlog". + LocalDbReplicationConnected = localDbReplication?.Item1, + LocalDbOplogBacklog = localDbReplication?.Item2 }; } } diff --git a/src/ZB.MOM.WW.ScadaBridge.Host/LocalDbSyncAuthInterceptor.cs b/src/ZB.MOM.WW.ScadaBridge.Host/LocalDbSyncAuthInterceptor.cs new file mode 100644 index 00000000..d7b21d07 --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.Host/LocalDbSyncAuthInterceptor.cs @@ -0,0 +1,158 @@ +using System.Security.Cryptography; +using System.Text; +using Grpc.Core; +using Grpc.Core.Interceptors; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Options; +using ZB.MOM.WW.LocalDb.Replication; + +namespace ZB.MOM.WW.ScadaBridge.Host; + +/// +/// Gates the LocalDb passive sync endpoint. The replication library deliberately leaves +/// inbound authentication to the host — its LocalDbSyncService verifies nothing — +/// so without this interceptor anything that can reach the site node's gRPC port could +/// stream arbitrary rows into the consolidated site database. +/// +/// +/// +/// Scoped by method path. Only calls under +/// /localdb_sync.v1.LocalDbSync/ are gated; every other method — notably the +/// existing SiteStream service sharing this listener — passes through untouched. +/// +/// +/// Fail-closed. With no LocalDb:Replication:ApiKey configured, NO sync +/// stream is accepted, authenticated or not. That is the deliberate choice: the +/// alternative — treating "no key" as "no auth required" — would silently expose the +/// endpoint on exactly the default configuration every site node ships with. An operator +/// enabling replication must set the same key on both nodes, which is already required +/// for the initiator to dial out (SyncBackgroundService sends +/// Authorization: Bearer <key>). +/// +/// +/// Comparison is over UTF-8 bytes, +/// so a wrong key cannot be recovered byte-by-byte from response timing. Length differences +/// are unavoidably observable and are not sensitive. +/// +/// +public sealed class LocalDbSyncAuthInterceptor : Interceptor +{ + private const string ServicePrefix = "/localdb_sync.v1.LocalDbSync/"; + private const string AuthorizationHeader = "authorization"; + private const string BearerPrefix = "Bearer "; + + private readonly IOptions _options; + private readonly ILogger _logger; + + /// Creates the interceptor. + /// Replication options; ApiKey is the expected bearer token. + /// Logger for denial diagnostics. + public LocalDbSyncAuthInterceptor( + IOptions options, + ILogger logger) + { + ArgumentNullException.ThrowIfNull(options); + ArgumentNullException.ThrowIfNull(logger); + + _options = options; + _logger = logger; + } + + /// + public override Task UnaryServerHandler( + TRequest request, + ServerCallContext context, + UnaryServerMethod continuation) + { + Authorize(context); + return continuation(request, context); + } + + /// + public override Task DuplexStreamingServerHandler( + IAsyncStreamReader requestStream, + IServerStreamWriter responseStream, + ServerCallContext context, + DuplexStreamingServerMethod continuation) + { + Authorize(context); + return continuation(requestStream, responseStream, context); + } + + /// + public override Task ClientStreamingServerHandler( + IAsyncStreamReader requestStream, + ServerCallContext context, + ClientStreamingServerMethod continuation) + { + Authorize(context); + return continuation(requestStream, context); + } + + /// + public override Task ServerStreamingServerHandler( + TRequest request, + IServerStreamWriter responseStream, + ServerCallContext context, + ServerStreamingServerMethod continuation) + { + Authorize(context); + return continuation(request, responseStream, context); + } + + /// + /// Throws with if + /// this is a sync call that does not carry the configured bearer token. Non-sync calls + /// return immediately. + /// + private void Authorize(ServerCallContext context) + { + if (!context.Method.StartsWith(ServicePrefix, StringComparison.Ordinal)) + return; + + var expected = _options.Value.ApiKey; + if (string.IsNullOrEmpty(expected)) + { + _logger.LogWarning( + "Rejected a LocalDb sync call to {Method}: no LocalDb:Replication:ApiKey is configured, " + + "so the passive sync endpoint is closed. Configure the same key on both nodes of the pair.", + context.Method); + throw new RpcException(new Status( + StatusCode.PermissionDenied, + "LocalDb sync is not accepting connections: no API key is configured on this node.")); + } + + var presented = ExtractBearerToken(context.RequestHeaders); + if (presented is null || !FixedTimeEquals(presented, expected)) + { + _logger.LogWarning( + "Rejected a LocalDb sync call to {Method}: {Reason}.", + context.Method, + presented is null ? "no bearer token presented" : "bearer token did not match"); + throw new RpcException(new Status( + StatusCode.PermissionDenied, + "LocalDb sync authentication failed.")); + } + } + + private static string? ExtractBearerToken(Metadata headers) + { + // gRPC lowercases header keys on the wire; compare case-insensitively anyway so a + // hand-built Metadata in a test behaves the same as a real request. + foreach (var entry in headers) + { + if (!string.Equals(entry.Key, AuthorizationHeader, StringComparison.OrdinalIgnoreCase)) + continue; + + var value = entry.Value; + if (value is not null && value.StartsWith(BearerPrefix, StringComparison.OrdinalIgnoreCase)) + return value[BearerPrefix.Length..]; + } + + return null; + } + + private static bool FixedTimeEquals(string presented, string expected) + => CryptographicOperations.FixedTimeEquals( + Encoding.UTF8.GetBytes(presented), Encoding.UTF8.GetBytes(expected)); +} diff --git a/src/ZB.MOM.WW.ScadaBridge.Host/Program.cs b/src/ZB.MOM.WW.ScadaBridge.Host/Program.cs index e8ea49ab..5df284bc 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Host/Program.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Host/Program.cs @@ -514,7 +514,11 @@ try }); // gRPC server registration - builder.Services.AddGrpc(); + // The interceptor gates ONLY /localdb_sync.v1.LocalDbSync/ — SiteStream calls on + // this same pipeline pass through untouched. It is fail-closed: with no + // LocalDb:Replication:ApiKey configured, no sync stream is accepted at all. + builder.Services.AddGrpc(options => + options.Interceptors.Add()); builder.Services.AddSingleton(); // Existing site service registrations (this is also where LocalDb and its @@ -540,9 +544,8 @@ try // The passive half of LocalDb replication: the peer node dials THIS endpoint. // It shares the Kestrel h2c listener the site gRPC server already uses, so no // listener or port changes are needed. Mapping it is harmless with no peer - // configured — nothing dials it. Inbound auth is the host's job and lands in - // Task 8; until then this endpoint is unauthenticated, which is why - // replication ships default-OFF. + // configured — nothing dials it, and LocalDbSyncAuthInterceptor (registered on + // AddGrpc above) rejects anything that tries until an ApiKey is configured. app.MapZbLocalDbSync(); // Site-shutdown ordering. ApplicationStopping diff --git a/src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs b/src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs index 3abee357..3b2122d7 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs @@ -100,6 +100,16 @@ public static class SiteServiceRegistration services.AddSiteEventLogHealthMetricsBridge( sp => () => sp.GetRequiredService().FailedWriteCount); + // LocalDb replication — bridge ISyncStatus onto the site health report. Registered + // unconditionally alongside the engine: on a node with no peer this reports + // Connected=false with a real 0 backlog, which is the healthy default-OFF state. + // OplogBacklog is passed through NULLABLE on purpose — null means the poll failed + // or no provider is wired, and flattening it to 0 would render a replication pair + // that cannot read its own oplog as perfectly healthy. + services.AddLocalDbReplicationHealthBridge( + sp => () => sp.GetRequiredService().Connected, + sp => () => sp.GetRequiredService().OplogBacklog); + // Audit Log — site-side hot-path writer + telemetry collaborators. // The SiteAuditTelemetryActor itself is registered by AkkaHostedService // in the site-role block; this call wires every DI dependency it (and diff --git a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/LocalDbSyncAuthInterceptorTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/LocalDbSyncAuthInterceptorTests.cs new file mode 100644 index 00000000..bf0b3cac --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/LocalDbSyncAuthInterceptorTests.cs @@ -0,0 +1,171 @@ +using Grpc.Core; +using Microsoft.Extensions.Logging.Abstractions; +using Microsoft.Extensions.Options; +using ZB.MOM.WW.LocalDb.Replication; +using ZB.MOM.WW.ScadaBridge.Host; + +namespace ZB.MOM.WW.ScadaBridge.Host.Tests; + +/// +/// LocalDb Phase 1 (Task 8) — the passive sync endpoint's inbound gate. +/// +/// +/// The replication library's LocalDbSyncService verifies nothing; inbound auth is +/// explicitly the host's job. Without this interceptor, anything able to reach a site +/// node's gRPC port could stream arbitrary rows straight into the consolidated site +/// database — including into OperationTracking, which central reconciles from. +/// +public class LocalDbSyncAuthInterceptorTests +{ + private const string SyncMethod = "/localdb_sync.v1.LocalDbSync/Sync"; + private const string SiteStreamMethod = "/scadabridge.SiteStream/Connect"; + + private static LocalDbSyncAuthInterceptor CreateInterceptor(string? apiKey) + => new( + Options.Create(new ReplicationOptions { ApiKey = apiKey }), + NullLogger.Instance); + + private static ServerCallContext CreateContext(string method, string? authorizationHeader) + { + var headers = new Metadata(); + if (authorizationHeader is not null) + headers.Add("authorization", authorizationHeader); + + return new FakeServerCallContext(method, headers); + } + + /// + /// Minimal carrying just a method name and request + /// headers — the only two things the interceptor reads. + /// + /// + /// Hand-rolled rather than using Grpc.Core.Testing.TestServerCallContext: that + /// type ships in the retired native Grpc.Core package and does not exist on the + /// grpc-dotnet stack this solution runs on. Pulling in the dead package to get one + /// test helper would be a worse trade than these few overrides. + /// + private sealed class FakeServerCallContext(string method, Metadata requestHeaders) + : ServerCallContext + { + protected override string MethodCore => method; + protected override string HostCore => "localhost"; + protected override string PeerCore => "ipv4:127.0.0.1:12345"; + protected override DateTime DeadlineCore => DateTime.UtcNow.AddMinutes(1); + protected override Metadata RequestHeadersCore => requestHeaders; + protected override CancellationToken CancellationTokenCore => CancellationToken.None; + protected override Metadata ResponseTrailersCore { get; } = []; + protected override Status StatusCore { get; set; } + protected override WriteOptions? WriteOptionsCore { get; set; } + protected override AuthContext AuthContextCore { get; } = + new(null, new Dictionary>()); + + protected override ContextPropagationToken CreatePropagationTokenCore( + ContextPropagationOptions? options) + => throw new NotSupportedException(); + + protected override Task WriteResponseHeadersAsyncCore(Metadata responseHeaders) + => Task.CompletedTask; + } + + /// Invokes the interceptor's unary path with a trivial continuation. + private static Task Invoke( + LocalDbSyncAuthInterceptor interceptor, ServerCallContext context) + => interceptor.UnaryServerHandler( + "request", context, (_, _) => Task.FromResult("ok")); + + [Fact] + public async Task NonSyncMethod_PassesThrough_EvenWithNoKeyConfigured() + { + // The interceptor is registered on the shared site AddGrpc pipeline, so it sees + // every call. It must be scoped strictly to the sync service — gating SiteStream + // would break site↔central communication outright. + var interceptor = CreateInterceptor(apiKey: null); + var context = CreateContext(SiteStreamMethod, authorizationHeader: null); + + Assert.Equal("ok", await Invoke(interceptor, context)); + } + + [Fact] + public async Task SyncMethod_WithNoKeyConfigured_IsDenied_EvenWithABearerToken() + { + // Fail-closed. "No key configured" is the DEFAULT every site node ships with, so + // treating it as "no auth required" would silently expose the endpoint on exactly + // the configuration that is most common. Presenting a token must not help. + var interceptor = CreateInterceptor(apiKey: null); + var context = CreateContext(SyncMethod, "Bearer anything-at-all"); + + var ex = await Assert.ThrowsAsync(() => Invoke(interceptor, context)); + Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode); + } + + [Fact] + public async Task SyncMethod_WithNoBearerToken_IsDenied() + { + var interceptor = CreateInterceptor("the-shared-key"); + var context = CreateContext(SyncMethod, authorizationHeader: null); + + var ex = await Assert.ThrowsAsync(() => Invoke(interceptor, context)); + Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode); + } + + [Fact] + public async Task SyncMethod_WithWrongBearerToken_IsDenied() + { + var interceptor = CreateInterceptor("the-shared-key"); + var context = CreateContext(SyncMethod, "Bearer the-wrong-key"); + + var ex = await Assert.ThrowsAsync(() => Invoke(interceptor, context)); + Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode); + } + + [Fact] + public async Task SyncMethod_WithCorrectBearerToken_PassesThrough() + { + var interceptor = CreateInterceptor("the-shared-key"); + var context = CreateContext(SyncMethod, "Bearer the-shared-key"); + + Assert.Equal("ok", await Invoke(interceptor, context)); + } + + [Fact] + public async Task SyncMethod_WithCorrectKey_ButNoBearerScheme_IsDenied() + { + // A raw key with no "Bearer " prefix is not what SyncBackgroundService sends, and + // accepting it would widen the accepted credential shape for no reason. + var interceptor = CreateInterceptor("the-shared-key"); + var context = CreateContext(SyncMethod, "the-shared-key"); + + var ex = await Assert.ThrowsAsync(() => Invoke(interceptor, context)); + Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode); + } + + [Fact] + public async Task SyncMethod_TokenComparison_IsNotAPrefixMatch() + { + // A prefix/StartsWith comparison would accept a truncated key and make the secret + // recoverable one character at a time. + var interceptor = CreateInterceptor("the-shared-key"); + var context = CreateContext(SyncMethod, "Bearer the-shared-ke"); + + var ex = await Assert.ThrowsAsync(() => Invoke(interceptor, context)); + Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode); + } + + [Fact] + public async Task DuplexStreaming_IsGated_BecauseThatIsHowSyncActuallyRuns() + { + // The sync RPC is a bidirectional stream. Gating only the unary path would leave + // the real endpoint wide open while every unary test still passed. + var interceptor = CreateInterceptor("the-shared-key"); + var context = CreateContext(SyncMethod, "Bearer the-wrong-key"); + + var ex = await Assert.ThrowsAsync(() => + interceptor.DuplexStreamingServerHandler( + requestStream: null!, + responseStream: null!, + context, + (_, _, _) => Task.CompletedTask)); + + Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode); + } +} diff --git a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbWiringTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbWiringTests.cs index db34fa6b..cff13488 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbWiringTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbWiringTests.cs @@ -1,6 +1,9 @@ using Microsoft.AspNetCore.Builder; using Microsoft.Extensions.Configuration; using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Logging.Abstractions; +using ZB.MOM.WW.ScadaBridge.HealthMonitoring; using ZB.MOM.WW.LocalDb; using ZB.MOM.WW.LocalDb.Replication; using ZB.MOM.WW.ScadaBridge.Host; @@ -150,6 +153,49 @@ public class SiteLocalDbWiringTests : IDisposable Assert.Equal(0L, status.OplogBacklog); } + [Fact] + public void Site_Replication_HealthBridge_IsRegisteredAsAHostedService() + { + // Task 9. Registered via a factory lambda, so ImplementationType is null and the + // only honest assertion is on the resolved instance. + var hosted = _host.Services.GetServices(); + + Assert.Contains(hosted, h => h is LocalDbReplicationStatusReporter); + } + + [Fact] + public void Site_Replication_HealthBridge_PublishesStatusOntoTheHealthReport() + { + // End-to-end through the REAL collector: probe once, then collect. This is what + // catches a bridge that is registered but wired to the wrong provider — the + // registration test above would pass either way. + var reporter = _host.Services.GetServices() + .OfType() + .Single(); + var collector = _host.Services.GetRequiredService(); + + reporter.Probe(); + var report = collector.CollectReport("TestSite"); + + // No peer configured: not connected, and a REAL 0 backlog rather than null. + Assert.False(report.LocalDbReplicationConnected); + Assert.Equal(0L, report.LocalDbOplogBacklog); + } + + [Fact] + public void Site_HealthReport_LocalDbFields_AreNull_BeforeTheReporterHasRun() + { + // Null means "no data yet", and must be distinguishable from a real + // "disconnected, zero backlog". A collector that defaulted these to false/0 would + // report an unwired node as a healthy connected one. + var collector = new SiteHealthCollector(); + + var report = collector.CollectReport("TestSite"); + + Assert.Null(report.LocalDbReplicationConnected); + Assert.Null(report.LocalDbOplogBacklog); + } + [Fact] public void Site_LocalDb_CreatesTheConfiguredFile() { From 98b94771ae5fcd25ba54f1dec2e5498d3381e69a Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sun, 19 Jul 2026 09:36:38 -0400 Subject: [PATCH 15/55] feat(localdb): one-time migrator for the pre-Phase-1 site databases Task 6 of the LocalDb Phase 1 adoption plan. Copies site-tracking.db and site_events.db into the consolidated database, then renames each source to .migrated. Runs AFTER RegisterReplicated, deliberately: capture is trigger-based, so rows inserted before registration would never enter the oplog and would never reach the peer - silently. There is a test asserting migrated rows land in __localdb_oplog, because every other test in the file would still pass with that ordering reversed. Idempotent twice over. Tracking rows keep their existing TEXT primary key, so INSERT OR IGNORE dedupes naturally. Legacy events had autoincrement integer ids, which the consolidated schema replaced with GUIDs; they get DETERMINISTIC ids of the form mig-{NodeName}-{legacyId} rather than fresh GUIDs, so a migration that crashes between commit and rename cannot duplicate history on the next boot. The NodeName prefix keeps the two nodes of a pair from colliding on their independent legacy id sequences. Both properties are covered, including an explicit crash-window test that undoes the rename and re-runs. All-or-nothing: the copy commits in one transaction and the rename happens only afterwards, so a failure throws out of OnReady, fails host startup, and leaves the legacy files untouched. Deviation from the plan: it specified BeginTransactionAsync, but OnReady is a synchronous Action. A raw transaction on a CreateConnection() connection gives identical atomicity and identical trigger capture without blocking on async in a startup callback. Legacy paths come from the OLD config keys and fall back to the CWD-relative code defaults, because that is where the old code actually wrote them. Note this means both legacy databases have always lived outside the mounted data volume and were already discarded on every container recreate - so on the rig this migrator will usually find nothing, which is a legitimate no-op rather than a failure. Phase 1 incidentally fixes that data-loss bug. Non-failure cases are covered too: missing files, a legacy file with no such table, and in-memory legacy connection strings from test/dev configs. Verified: build 0 warnings; Host 316/316 (9 migrator tests new). Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts --- .../SiteLocalDbLegacyMigrator.cs | 264 ++++++++++++++ .../SiteLocalDbSetup.cs | 12 +- .../SiteServiceRegistration.cs | 2 +- .../SiteLocalDbLegacyMigratorTests.cs | 327 ++++++++++++++++++ 4 files changed, 602 insertions(+), 3 deletions(-) create mode 100644 src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbLegacyMigrator.cs create mode 100644 tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbLegacyMigratorTests.cs diff --git a/src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbLegacyMigrator.cs b/src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbLegacyMigrator.cs new file mode 100644 index 00000000..3e52ab8e --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbLegacyMigrator.cs @@ -0,0 +1,264 @@ +using Microsoft.Data.Sqlite; +using Microsoft.Extensions.Configuration; +using ZB.MOM.WW.LocalDb; + +namespace ZB.MOM.WW.ScadaBridge.Host; + +/// +/// One-time copy of the pre-Phase-1 site databases (site-tracking.db and +/// site_events.db) into the consolidated ZB.MOM.WW.LocalDb database. +/// +/// +/// +/// Runs after RegisterReplicated, deliberately. Capture is trigger-based, so +/// rows inserted before registration would never enter the oplog and would never reach the +/// peer. Migrating after registration means the migrated history replicates like any other +/// write — which is the entire point of Phase 1. +/// +/// +/// Idempotent twice over. Tracking rows carry the same TEXT primary key they always +/// had, so INSERT OR IGNORE is naturally idempotent. Legacy events had autoincrement +/// integer ids, which the consolidated schema replaced with GUIDs; they are given +/// deterministic ids of the form mig-{NodeName}-{legacyId} rather than fresh +/// GUIDs, so a migration that crashes and re-runs cannot duplicate events. The NodeName +/// prefix keeps the two nodes of a pair from colliding on their independent legacy id +/// sequences. +/// +/// +/// All-or-nothing. The copy runs in one transaction and the legacy file is renamed +/// to <name>.migrated only after commit. A failure throws out of +/// OnReady, which fails host startup with the legacy files untouched — no +/// half-migrated state to reason about. A second boot sees the renamed file and no-ops. +/// +/// +/// Finding nothing is the expected case on the docker rig. Neither legacy config key +/// is set in any rig appsettings, so both fall back to CWD-relative code defaults +/// (/app/site-tracking.db, /app/site_events.db) that sit OUTSIDE the mounted +/// data volume — meaning they were already being discarded on every container recreate. +/// Phase 1 incidentally fixes that data-loss bug by consolidating into +/// /app/data/site-localdb.db. A no-op here is a legitimate result, not a failure. +/// +/// +public static class SiteLocalDbLegacyMigrator +{ + private const string MigratedSuffix = ".migrated"; + + /// Default legacy tracking connection string (OperationTrackingOptions.ConnectionString). + private const string DefaultTrackingConnectionString = "Data Source=site-tracking.db"; + + /// Default legacy event-log path (SiteEventLogOptions.DatabasePath). + private const string DefaultEventLogPath = "site_events.db"; + + /// + /// Copies any legacy site databases into , then renames them. + /// + /// The consolidated site database, with both tables already registered. + /// Configuration supplying the legacy paths and the node name. + public static void Migrate(ILocalDb db, IConfiguration config) + { + ArgumentNullException.ThrowIfNull(db); + ArgumentNullException.ThrowIfNull(config); + + var nodeName = config["ScadaBridge:Node:NodeName"] ?? "unknown-node"; + + MigrateTracking(db, ResolveTrackingPath(config)); + MigrateEvents(db, ResolveEventLogPath(config), nodeName); + } + + /// + /// Resolves the legacy tracking database path from the OLD connection-string key, + /// falling back to the code default. Relative paths resolve against the process working + /// directory — NOT the data volume — because that is where the old code actually put them. + /// + internal static string ResolveTrackingPath(IConfiguration config) + { + var connectionString = + config["ScadaBridge:OperationTracking:ConnectionString"] ?? DefaultTrackingConnectionString; + + string dataSource; + try + { + dataSource = new SqliteConnectionStringBuilder(connectionString).DataSource; + } + catch (ArgumentException) + { + // An unparseable legacy connection string means there is nothing to migrate + // from. Do not fail the host over a stale key we are about to stop reading. + return string.Empty; + } + + // In-memory legacy databases (test/dev configurations) have nothing durable to + // migrate, and Path.GetFullPath on them would produce nonsense. + if (string.IsNullOrWhiteSpace(dataSource) || + dataSource.Equals(":memory:", StringComparison.OrdinalIgnoreCase) || + dataSource.StartsWith("file:", StringComparison.OrdinalIgnoreCase)) + { + return string.Empty; + } + + return Path.GetFullPath(dataSource); + } + + /// Resolves the legacy event-log path from the OLD key, falling back to the code default. + internal static string ResolveEventLogPath(IConfiguration config) + { + var path = config["ScadaBridge:SiteEventLog:DatabasePath"] ?? DefaultEventLogPath; + + if (string.IsNullOrWhiteSpace(path) || + path.Equals(":memory:", StringComparison.OrdinalIgnoreCase)) + { + return string.Empty; + } + + return Path.GetFullPath(path); + } + + private static void MigrateTracking(ILocalDb db, string legacyPath) + { + if (!ShouldMigrate(legacyPath)) return; + + var rows = ReadAll( + legacyPath, + """ + SELECT TrackedOperationId, Kind, TargetSummary, Status, + RetryCount, LastError, HttpStatus, + CreatedAtUtc, UpdatedAtUtc, TerminalAtUtc, + SourceInstanceId, SourceScript, SourceNode + FROM OperationTracking; + """, + expectedColumns: 13); + + // A legacy file with no OperationTracking table at all reads as "nothing to do"; + // it still gets renamed so the probe does not repeat on every boot. + if (rows is not null) + { + using var connection = db.CreateConnection(); + using var transaction = connection.BeginTransaction(); + + foreach (var row in rows) + { + using var cmd = connection.CreateCommand(); + cmd.Transaction = transaction; + cmd.CommandText = """ + INSERT OR IGNORE INTO OperationTracking ( + TrackedOperationId, Kind, TargetSummary, Status, + RetryCount, LastError, HttpStatus, + CreatedAtUtc, UpdatedAtUtc, TerminalAtUtc, + SourceInstanceId, SourceScript, SourceNode + ) VALUES ( + $p0, $p1, $p2, $p3, $p4, $p5, $p6, + $p7, $p8, $p9, $p10, $p11, $p12 + ); + """; + Bind(cmd, row); + cmd.ExecuteNonQuery(); + } + + transaction.Commit(); + } + + MarkMigrated(legacyPath); + } + + private static void MigrateEvents(ILocalDb db, string legacyPath, string nodeName) + { + if (!ShouldMigrate(legacyPath)) return; + + var rows = ReadAll( + legacyPath, + """ + SELECT id, timestamp, event_type, severity, instance_id, source, message, details + FROM site_events; + """, + expectedColumns: 8); + + if (rows is not null) + { + using var connection = db.CreateConnection(); + using var transaction = connection.BeginTransaction(); + + foreach (var row in rows) + { + using var cmd = connection.CreateCommand(); + cmd.Transaction = transaction; + cmd.CommandText = """ + INSERT OR IGNORE INTO site_events ( + id, timestamp, event_type, severity, instance_id, source, message, details + ) VALUES ( + $id, $p1, $p2, $p3, $p4, $p5, $p6, $p7 + ); + """; + + // Deterministic id, NOT a fresh GUID: a crashed-then-rerun migration must + // not duplicate events, and INSERT OR IGNORE can only dedupe on a stable key. + cmd.Parameters.AddWithValue("$id", $"mig-{nodeName}-{row[0] ?? "null"}"); + for (var i = 1; i < row.Length; i++) + cmd.Parameters.AddWithValue($"$p{i}", row[i] ?? (object)DBNull.Value); + + cmd.ExecuteNonQuery(); + } + + transaction.Commit(); + } + + MarkMigrated(legacyPath); + } + + /// True when there is a legacy file present that has not already been migrated. + private static bool ShouldMigrate(string legacyPath) + => !string.IsNullOrEmpty(legacyPath) + && File.Exists(legacyPath) + && !File.Exists(legacyPath + MigratedSuffix); + + /// + /// Reads every row of from the legacy file, or null when the + /// table does not exist (an old file predating the table is not an error). + /// + private static List? ReadAll(string legacyPath, string sql, int expectedColumns) + { + using var connection = new SqliteConnection($"Data Source={legacyPath};Mode=ReadOnly"); + connection.Open(); + + using var cmd = connection.CreateCommand(); + cmd.CommandText = sql; + + SqliteDataReader reader; + try + { + reader = cmd.ExecuteReader(); + } + catch (SqliteException) + { + // "no such table" / "no such column" — a legacy file from before this table + // existed, or a shape we do not recognise. Nothing to copy. + return null; + } + + var rows = new List(); + using (reader) + { + while (reader.Read()) + { + var row = new object?[expectedColumns]; + for (var i = 0; i < expectedColumns; i++) + row[i] = reader.IsDBNull(i) ? null : reader.GetValue(i); + rows.Add(row); + } + } + + return rows; + } + + private static void Bind(SqliteCommand cmd, object?[] row) + { + for (var i = 0; i < row.Length; i++) + cmd.Parameters.AddWithValue($"$p{i}", row[i] ?? (object)DBNull.Value); + } + + /// + /// Renames the legacy file so a later boot skips it. Done only after the copy has + /// committed; the file is kept rather than deleted so an operator can still inspect it. + /// + private static void MarkMigrated(string legacyPath) + => File.Move(legacyPath, legacyPath + MigratedSuffix); +} diff --git a/src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbSetup.cs b/src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbSetup.cs index 070dcd1e..13b08ae4 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbSetup.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbSetup.cs @@ -1,3 +1,4 @@ +using Microsoft.Extensions.Configuration; using ZB.MOM.WW.LocalDb; using ZB.MOM.WW.ScadaBridge.SiteEventLogging; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Tracking; @@ -29,12 +30,15 @@ namespace ZB.MOM.WW.ScadaBridge.Host; public static class SiteLocalDbSetup { /// - /// Creates the site node's replicated tables and opts them into change capture. + /// Creates the site node's replicated tables, opts them into change capture, and + /// migrates any pre-Phase-1 databases in. /// /// The LocalDb instance being initialized. - public static void OnReady(ILocalDb db) + /// Configuration, for the legacy migrator's old path keys and node name. + public static void OnReady(ILocalDb db, IConfiguration config) { ArgumentNullException.ThrowIfNull(db); + ArgumentNullException.ThrowIfNull(config); using (var connection = db.CreateConnection()) { @@ -47,5 +51,9 @@ public static class SiteLocalDbSetup // capture). Registration is idempotent and installs the capture triggers. db.RegisterReplicated("OperationTracking"); db.RegisterReplicated("site_events"); + + // AFTER registration, so migrated rows enter the oplog and reach the peer like + // any other write. Before it, they would be invisible to replication forever. + SiteLocalDbLegacyMigrator.Migrate(db, config); } } diff --git a/src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs b/src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs index 3b2122d7..e943b705 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs @@ -60,7 +60,7 @@ public static class SiteServiceRegistration // initiator idles. // // Design: scadaproj docs/plans/2026-07-19-scadabridge-localdb-design.md - services.AddZbLocalDb(config, SiteLocalDbSetup.OnReady); + services.AddZbLocalDb(config, db => SiteLocalDbSetup.OnReady(db, config)); // The replication engine, likewise unconditional but INERT by default: with no // LocalDb:Replication:PeerAddress the initiating SyncBackgroundService starts and diff --git a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbLegacyMigratorTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbLegacyMigratorTests.cs new file mode 100644 index 00000000..d3065f3c --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbLegacyMigratorTests.cs @@ -0,0 +1,327 @@ +using Microsoft.Data.Sqlite; +using Microsoft.Extensions.Configuration; +using Microsoft.Extensions.DependencyInjection; +using ZB.MOM.WW.LocalDb; +using ZB.MOM.WW.ScadaBridge.Host; + +namespace ZB.MOM.WW.ScadaBridge.Host.Tests; + +/// +/// LocalDb Phase 1 (Task 6) — the one-time copy of the pre-Phase-1 site databases into +/// the consolidated one. +/// +/// +/// This is a data migration that runs during host startup, so its failure modes are +/// expensive: a duplicate-on-rerun bug silently doubles a site's event history, and a +/// half-committed copy leaves an operator with no clean state to recover to. Every bullet +/// of the plan's semantics gets its own test. +/// +public class SiteLocalDbLegacyMigratorTests : IDisposable +{ + private readonly string _root; + private readonly List _providers = []; + + public SiteLocalDbLegacyMigratorTests() + { + _root = Path.Combine(Path.GetTempPath(), $"localdb-migrator-{Guid.NewGuid():N}"); + Directory.CreateDirectory(_root); + } + + public void Dispose() + { + foreach (var provider in _providers) + { + try { provider.Dispose(); } catch { /* best effort */ } + } + + try { Directory.Delete(_root, recursive: true); } catch { /* best effort */ } + GC.SuppressFinalize(this); + } + + private string Path_(string name) => System.IO.Path.Combine(_root, name); + + /// A consolidated database with both tables created and registered, as the host has it. + private ILocalDb CreateConsolidated(IConfiguration config) + { + var provider = new ServiceCollection() + .AddZbLocalDb(config, db => + { + using var connection = db.CreateConnection(); + ZB.MOM.WW.ScadaBridge.SiteRuntime.Tracking.OperationTrackingSchema.Apply(connection); + ZB.MOM.WW.ScadaBridge.SiteEventLogging.SiteEventLogSchema.Apply(connection); + db.RegisterReplicated("OperationTracking"); + db.RegisterReplicated("site_events"); + }) + .BuildServiceProvider(); + _providers.Add(provider); + + return provider.GetRequiredService(); + } + + private IConfiguration Config( + string? trackingPath = null, string? eventsPath = null, string nodeName = "node-a") + { + var values = new Dictionary + { + ["LocalDb:Path"] = Path_("consolidated.db"), + ["ScadaBridge:Node:NodeName"] = nodeName, + }; + + if (trackingPath is not null) + values["ScadaBridge:OperationTracking:ConnectionString"] = $"Data Source={trackingPath}"; + if (eventsPath is not null) + values["ScadaBridge:SiteEventLog:DatabasePath"] = eventsPath; + + return new ConfigurationBuilder().AddInMemoryCollection(values).Build(); + } + + private static void SeedLegacyTracking(string path, params string[] ids) + { + using var connection = new SqliteConnection($"Data Source={path}"); + connection.Open(); + ZB.MOM.WW.ScadaBridge.SiteRuntime.Tracking.OperationTrackingSchema.Apply(connection); + + foreach (var id in ids) + { + using var cmd = connection.CreateCommand(); + cmd.CommandText = """ + INSERT INTO OperationTracking ( + TrackedOperationId, Kind, TargetSummary, Status, RetryCount, + CreatedAtUtc, UpdatedAtUtc) + VALUES ($id, 'ApiCallCached', 'ERP.GetOrder', 'Submitted', 0, $now, $now); + """; + cmd.Parameters.AddWithValue("$id", id); + cmd.Parameters.AddWithValue("$now", DateTime.UtcNow.ToString("o")); + cmd.ExecuteNonQuery(); + } + } + + /// Seeds the LEGACY event schema: an autoincrement integer id, not a GUID. + private static void SeedLegacyEvents(string path, int count) + { + using var connection = new SqliteConnection($"Data Source={path}"); + connection.Open(); + + using (var ddl = connection.CreateCommand()) + { + ddl.CommandText = """ + CREATE TABLE IF NOT EXISTS site_events ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + timestamp TEXT NOT NULL, event_type TEXT NOT NULL, severity TEXT NOT NULL, + instance_id TEXT, source TEXT NOT NULL, message TEXT NOT NULL, details TEXT + ); + """; + ddl.ExecuteNonQuery(); + } + + for (var i = 0; i < count; i++) + { + using var cmd = connection.CreateCommand(); + cmd.CommandText = """ + INSERT INTO site_events (timestamp, event_type, severity, source, message) + VALUES ($ts, 'script', 'Info', 'src', $msg); + """; + cmd.Parameters.AddWithValue("$ts", DateTimeOffset.UtcNow.AddMinutes(-i).ToString("o")); + cmd.Parameters.AddWithValue("$msg", $"legacy message {i}"); + cmd.ExecuteNonQuery(); + } + } + + private static long CountRows(ILocalDb db, string table) + { + using var connection = db.CreateConnection(); + using var cmd = connection.CreateCommand(); + cmd.CommandText = $"SELECT COUNT(*) FROM {table}"; + return (long)cmd.ExecuteScalar()!; + } + + private static List SelectIds(ILocalDb db, string table, string idColumn) + { + using var connection = db.CreateConnection(); + using var cmd = connection.CreateCommand(); + cmd.CommandText = $"SELECT {idColumn} FROM {table} ORDER BY {idColumn}"; + using var reader = cmd.ExecuteReader(); + + var ids = new List(); + while (reader.Read()) ids.Add(reader.GetString(0)); + return ids; + } + + [Fact] + public void MissingLegacyFiles_AreANoOp() + { + // The expected case on the docker rig: neither legacy key is set anywhere, and + // the CWD-relative defaults point outside the data volume, so there is usually + // nothing there at all. That must be a clean no-op, not a startup failure. + var config = Config(Path_("absent-tracking.db"), Path_("absent-events.db")); + var db = CreateConsolidated(config); + + SiteLocalDbLegacyMigrator.Migrate(db, config); + + Assert.Equal(0, CountRows(db, "OperationTracking")); + Assert.Equal(0, CountRows(db, "site_events")); + } + + [Fact] + public void LegacyTrackingRows_AreCopiedAndTheFileIsRenamed() + { + var trackingPath = Path_("site-tracking.db"); + SeedLegacyTracking(trackingPath, "op-1", "op-2", "op-3"); + var config = Config(trackingPath, Path_("absent-events.db")); + var db = CreateConsolidated(config); + + SiteLocalDbLegacyMigrator.Migrate(db, config); + + Assert.Equal(3, CountRows(db, "OperationTracking")); + Assert.Equal(["op-1", "op-2", "op-3"], SelectIds(db, "OperationTracking", "TrackedOperationId")); + + // Renamed, not deleted — the operator can still inspect the original. + Assert.False(File.Exists(trackingPath)); + Assert.True(File.Exists(trackingPath + ".migrated")); + } + + [Fact] + public void LegacyEvents_GetDeterministicIds_NotFreshGuids() + { + // The whole reason a rerun cannot duplicate. Fresh GUIDs would make + // INSERT OR IGNORE useless and double the history on every retry. + var eventsPath = Path_("site_events.db"); + SeedLegacyEvents(eventsPath, count: 3); + var config = Config(Path_("absent-tracking.db"), eventsPath, nodeName: "node-b"); + var db = CreateConsolidated(config); + + SiteLocalDbLegacyMigrator.Migrate(db, config); + + Assert.Equal( + ["mig-node-b-1", "mig-node-b-2", "mig-node-b-3"], + SelectIds(db, "site_events", "id")); + } + + [Fact] + public void SecondBoot_IsANoOp_BecauseTheFileIsAlreadyMarkedMigrated() + { + var trackingPath = Path_("site-tracking.db"); + var eventsPath = Path_("site_events.db"); + SeedLegacyTracking(trackingPath, "op-1", "op-2"); + SeedLegacyEvents(eventsPath, count: 2); + var config = Config(trackingPath, eventsPath); + var db = CreateConsolidated(config); + + SiteLocalDbLegacyMigrator.Migrate(db, config); + SiteLocalDbLegacyMigrator.Migrate(db, config); + + Assert.Equal(2, CountRows(db, "OperationTracking")); + Assert.Equal(2, CountRows(db, "site_events")); + } + + [Fact] + public void RerunAgainstAnUnrenamedFile_DoesNotDuplicate() + { + // Simulates the crash window: the copy committed but the rename did not. On the + // next boot the migrator sees the legacy file again and re-copies it. Both tables + // must absorb that with no duplicates — tracking via its natural TEXT key, events + // via the deterministic mig- id. + var trackingPath = Path_("site-tracking.db"); + var eventsPath = Path_("site_events.db"); + SeedLegacyTracking(trackingPath, "op-1", "op-2"); + SeedLegacyEvents(eventsPath, count: 2); + var config = Config(trackingPath, eventsPath); + var db = CreateConsolidated(config); + + SiteLocalDbLegacyMigrator.Migrate(db, config); + + // Undo the rename, exactly as a crash between commit and rename would leave it. + File.Move(trackingPath + ".migrated", trackingPath); + File.Move(eventsPath + ".migrated", eventsPath); + + SiteLocalDbLegacyMigrator.Migrate(db, config); + + Assert.Equal(2, CountRows(db, "OperationTracking")); + Assert.Equal(2, CountRows(db, "site_events")); + } + + [Fact] + public void MigratedRows_EnterTheOplog_SoTheyActuallyReplicate() + { + // The reason the migrator runs AFTER RegisterReplicated. Capture is trigger-based: + // migrate before registration and the rows are invisible to the peer forever, with + // no error anywhere. Asserting on the oplog is the only way to catch that ordering + // being reversed — every other test here would still pass. + var trackingPath = Path_("site-tracking.db"); + SeedLegacyTracking(trackingPath, "op-1", "op-2"); + var config = Config(trackingPath, Path_("absent-events.db")); + var db = CreateConsolidated(config); + + SiteLocalDbLegacyMigrator.Migrate(db, config); + + using var connection = db.CreateConnection(); + using var cmd = connection.CreateCommand(); + cmd.CommandText = + "SELECT COUNT(*) FROM __localdb_oplog WHERE table_name = 'OperationTracking'"; + Assert.Equal(2L, (long)cmd.ExecuteScalar()!); + } + + [Fact] + public void LegacyFileWithoutTheExpectedTable_IsTreatedAsEmpty_NotAFailure() + { + // A file predating the table (or an unrecognised shape) must not fail host boot. + var trackingPath = Path_("site-tracking.db"); + using (var connection = new SqliteConnection($"Data Source={trackingPath}")) + { + connection.Open(); + using var ddl = connection.CreateCommand(); + ddl.CommandText = "CREATE TABLE something_else (x INTEGER);"; + ddl.ExecuteNonQuery(); + } + + var config = Config(trackingPath, Path_("absent-events.db")); + var db = CreateConsolidated(config); + + SiteLocalDbLegacyMigrator.Migrate(db, config); + + Assert.Equal(0, CountRows(db, "OperationTracking")); + Assert.True(File.Exists(trackingPath + ".migrated")); + } + + [Fact] + public void InMemoryLegacyConnectionStrings_AreSkipped() + { + // Test/dev configurations point the legacy stores at in-memory databases. There is + // nothing durable to migrate, and Path.GetFullPath on ":memory:" would be nonsense. + var config = new ConfigurationBuilder().AddInMemoryCollection(new Dictionary + { + ["LocalDb:Path"] = Path_("consolidated.db"), + ["ScadaBridge:Node:NodeName"] = "node-a", + ["ScadaBridge:OperationTracking:ConnectionString"] = + "Data Source=file:legacy?mode=memory&cache=shared", + ["ScadaBridge:SiteEventLog:DatabasePath"] = ":memory:", + }).Build(); + + Assert.Equal(string.Empty, SiteLocalDbLegacyMigrator.ResolveTrackingPath(config)); + Assert.Equal(string.Empty, SiteLocalDbLegacyMigrator.ResolveEventLogPath(config)); + + var db = CreateConsolidated(config); + SiteLocalDbLegacyMigrator.Migrate(db, config); + + Assert.Equal(0, CountRows(db, "site_events")); + } + + [Fact] + public void UnsetLegacyKeys_FallBackToTheCwdRelativeCodeDefaults() + { + // Neither key is set in any docker appsettings, so the OLD code used these + // CWD-relative defaults. Resolving them anywhere else (e.g. against the data + // volume) would look for the legacy data in a directory it was never written to. + var config = new ConfigurationBuilder().AddInMemoryCollection(new Dictionary + { + ["LocalDb:Path"] = Path_("consolidated.db"), + }).Build(); + + Assert.Equal( + System.IO.Path.GetFullPath("site-tracking.db"), + SiteLocalDbLegacyMigrator.ResolveTrackingPath(config)); + Assert.Equal( + System.IO.Path.GetFullPath("site_events.db"), + SiteLocalDbLegacyMigrator.ResolveEventLogPath(config)); + } +} From 3c395d794ac48e4cb872cb4a925d87370e161b81 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sun, 19 Jul 2026 09:42:21 -0400 Subject: [PATCH 16/55] test(localdb): two-node site-pair convergence over a real gRPC transport Task 10 of the LocalDb Phase 1 adoption plan. Two ScadaBridge site nodes replicating the consolidated site database over loopback Kestrel h2c, through the REAL fail-closed auth interceptor - not a stand-in. This is the test that answers the question Phase 1 exists to answer. Every piece upstream of it - schema helpers, DI wiring, the interceptor - can be individually green while the pair still fails to converge. It initializes both databases through SiteLocalDbSetup.OnReady rather than a hand-written schema, so the tables, primary keys and registration ORDER under test are the ones the host actually runs. A local schema would prove only that the test agrees with itself. Scenarios: A->B, B->A (bidirectional even though only A dials, which is what makes failover safe in either direction), LWW convergence on a contended operation, the event union across both nodes, and a peer-offline-then-rejoin catch-up. The event-union scenario is the one that pins the GUID id change: under the old autoincrement scheme both nodes independently mint id=1,2,3..., and last-writer-wins on the primary key would silently DESTROY one node's events rather than merge them. Asserting the union count is what catches that. The whole suite passes in ~0.6s, which is fast enough to be suspicious of a vacuous pass, so it was verified red-first the hard way: deliberately mismatching the two nodes' ApiKey turns all 5 red with 2m30s of timeouts. The tests really do run through the interceptor and the real transport. Node B's database survives its host teardown (registered as a pre-constructed instance, which MS.DI does not dispose), so the rejoin scenario is a genuine rejoin rather than a fresh node. Offline - no docker, no external services. Verified: build 0 warnings; IntegrationTests 80/80. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts --- .../LocalDbSitePairConvergenceTests.cs | 348 ++++++++++++++++++ 1 file changed, 348 insertions(+) create mode 100644 tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbSitePairConvergenceTests.cs diff --git a/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbSitePairConvergenceTests.cs b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbSitePairConvergenceTests.cs new file mode 100644 index 00000000..b9c13ced --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbSitePairConvergenceTests.cs @@ -0,0 +1,348 @@ +using System.Net; +using Microsoft.AspNetCore.Builder; +using Microsoft.AspNetCore.Hosting; +using Microsoft.AspNetCore.Hosting.Server; +using Microsoft.AspNetCore.Hosting.Server.Features; +using Microsoft.AspNetCore.Server.Kestrel.Core; +using Microsoft.Extensions.Configuration; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; +using ZB.MOM.WW.LocalDb; +using ZB.MOM.WW.LocalDb.Replication; +using ZB.MOM.WW.ScadaBridge.Host; + +namespace ZB.MOM.WW.ScadaBridge.IntegrationTests; + +/// +/// Serializes the site-pair convergence tests against each other: each one stands up a real +/// Kestrel listener plus two SQLite files, and running them concurrently under CI +/// contention is a flakiness risk. +/// +[CollectionDefinition("LocalDbSitePairConvergence")] +public sealed class LocalDbSitePairConvergenceCollection; + +/// +/// Two ScadaBridge site nodes replicating the consolidated site database over a REAL +/// loopback gRPC transport, through the REAL fail-closed auth interceptor. +/// +/// +/// +/// This is the test that answers the question Phase 1 exists to answer: does a site node +/// pair actually stop losing operation-tracking and site-event state? Everything upstream +/// of it — schema helpers, DI wiring, the interceptor — can be individually green while the +/// pair still fails to converge. +/// +/// +/// It uses , not a hand-written schema, so the tables, +/// their primary keys, and the registration ORDER under test are the ones the host actually +/// runs. A separate schema here would prove only that the test agrees with itself. +/// +/// +/// Offline: no docker, no external services. Loopback Kestrel with h2c. +/// +/// +[Collection("LocalDbSitePairConvergence")] +public sealed class LocalDbSitePairConvergenceTests : IAsyncLifetime +{ + private const string SharedApiKey = "site-pair-convergence-key"; + private static readonly TimeSpan ConvergeTimeout = TimeSpan.FromSeconds(30); + + private readonly string _pathA = Path.Combine(Path.GetTempPath(), $"sitepairA-{Guid.NewGuid():N}.db"); + private readonly string _pathB = Path.Combine(Path.GetTempPath(), $"sitepairB-{Guid.NewGuid():N}.db"); + + // The databases are owned by the fixture, in their own providers, and registered into the + // hosts as pre-constructed instances. MS.DI does not dispose instances it did not create, + // so tearing a host down (the offline-peer scenario) leaves the databases intact and + // writable — which is exactly what lets node A accumulate writes while B is down. + private ServiceProvider _dbProviderA = null!; + private ServiceProvider _dbProviderB = null!; + + private IHost? _serverHost; // node B — passive + private IHost? _initiatorHost; // node A — dials the peer + + static LocalDbSitePairConvergenceTests() => + // Grpc.Net.Client dials the loopback server over HTTP/2 cleartext. + AppContext.SetSwitch("System.Net.Http.SocketsHttpHandler.Http2UnencryptedSupport", true); + + private ILocalDb A => _dbProviderA.GetRequiredService(); + private ILocalDb B => _dbProviderB.GetRequiredService(); + + public async Task InitializeAsync() + { + _dbProviderA = BuildDatabaseProvider(_pathA, "node-a"); + _dbProviderB = BuildDatabaseProvider(_pathB, "node-b"); + + // Force construction (and therefore OnReady) before anything replicates. + _ = A; + _ = B; + + await StartPassiveAsync(); + await StartInitiatorAsync(); + } + + public async Task DisposeAsync() + { + await StopHostAsync(_initiatorHost); + await StopHostAsync(_serverHost); + await _dbProviderA.DisposeAsync(); + await _dbProviderB.DisposeAsync(); + + Microsoft.Data.Sqlite.SqliteConnection.ClearAllPools(); + foreach (var path in new[] { _pathA, _pathB }) + { + foreach (var suffix in new[] { "", "-wal", "-shm" }) + { + try { File.Delete(path + suffix); } catch { /* best effort */ } + } + } + } + + // ---- fixture internals ------------------------------------------------------------ + + /// + /// A provider owning one consolidated site database, initialized through the host's own + /// — same schema, same registration order. + /// + private static ServiceProvider BuildDatabaseProvider(string path, string nodeName) + { + var config = new ConfigurationBuilder() + .AddInMemoryCollection(new Dictionary + { + ["LocalDb:Path"] = path, + ["ScadaBridge:Node:NodeName"] = nodeName, + // Point the legacy migrator at paths that do not exist, so it no-ops rather + // than picking up stray files from the test working directory. + ["ScadaBridge:OperationTracking:ConnectionString"] = + $"Data Source={Path.Combine(Path.GetTempPath(), $"absent-{Guid.NewGuid():N}.db")}", + ["ScadaBridge:SiteEventLog:DatabasePath"] = + Path.Combine(Path.GetTempPath(), $"absent-{Guid.NewGuid():N}.db"), + }) + .Build(); + + return new ServiceCollection() + .AddZbLocalDb(config, db => SiteLocalDbSetup.OnReady(db, config)) + .BuildServiceProvider(); + } + + private static IConfiguration ReplicationConfig(string? peerAddress) + { + var values = new Dictionary + { + // Tight flush + bounded reconnect backoff so convergence is observable well + // inside the poll deadline. The 60 s production default would let the doubling + // backoff overrun it after a peer outage. + ["LocalDb:Replication:FlushInterval"] = "00:00:00.050", + ["LocalDb:Replication:ReconnectBackoffMax"] = "00:00:02", + // Both nodes share one key — the interceptor is fail-closed, so a mismatch here + // turns every scenario below red (verified by deliberately breaking it). + ["LocalDb:Replication:ApiKey"] = SharedApiKey, + }; + + if (peerAddress is not null) + values["LocalDb:Replication:PeerAddress"] = peerAddress; + + return new ConfigurationBuilder().AddInMemoryCollection(values).Build(); + } + + private async Task StartPassiveAsync() + { + var config = ReplicationConfig(peerAddress: null); + + _serverHost = await new HostBuilder() + .ConfigureWebHost(web => + { + web.UseKestrel(o => + o.Listen(IPAddress.Loopback, 0, listen => listen.Protocols = HttpProtocols.Http2)); + web.ConfigureServices(services => + { + services.AddLogging(); + services.AddRouting(); + // The REAL interceptor, not a stand-in. If it rejected legitimate peer + // traffic, every scenario below would fail — which is the point. + services.AddGrpc(o => o.Interceptors.Add()); + services.AddSingleton(B); + services.AddZbLocalDbReplication(config); + }); + web.Configure(app => + { + app.UseRouting(); + app.UseEndpoints(e => e.MapZbLocalDbSync()); + }); + }) + .StartAsync(); + } + + private async Task StartInitiatorAsync() + { + var config = ReplicationConfig(PassiveAddress()); + + _initiatorHost = await new HostBuilder() + .ConfigureServices(services => + { + services.AddLogging(); + services.AddSingleton(A); + services.AddZbLocalDbReplication(config); + }) + .StartAsync(); + } + + private string PassiveAddress() + => _serverHost!.Services.GetRequiredService() + .Features.Get()!.Addresses.Single(); + + private static async Task StopHostAsync(IHost? host) + { + if (host is null) return; + try { await host.StopAsync(TimeSpan.FromSeconds(5)); } catch { /* teardown */ } + host.Dispose(); + } + + // ---- data helpers ----------------------------------------------------------------- + + private static Task WriteTrackingAsync(ILocalDb db, string id, string status, string target) + => db.ExecuteAsync( + """ + INSERT INTO OperationTracking ( + TrackedOperationId, Kind, TargetSummary, Status, RetryCount, + CreatedAtUtc, UpdatedAtUtc) + VALUES (@id, 'ApiCallCached', @target, @status, 0, @now, @now) + ON CONFLICT(TrackedOperationId) DO UPDATE SET + Status = excluded.Status, + TargetSummary = excluded.TargetSummary, + UpdatedAtUtc = excluded.UpdatedAtUtc; + """, + new { id, status, target, now = DateTime.UtcNow.ToString("o") }); + + private static Task WriteEventAsync(ILocalDb db, string id, string message) + => db.ExecuteAsync( + """ + INSERT INTO site_events (id, timestamp, event_type, severity, source, message) + VALUES (@id, @ts, 'script', 'Info', 'convergence-test', @message); + """, + new { id, ts = DateTimeOffset.UtcNow.ToString("o"), message }); + + private static async Task ReadTrackingStatusAsync(ILocalDb db, string id) + { + var rows = await db.QueryAsync( + "SELECT Status FROM OperationTracking WHERE TrackedOperationId = @id", + static r => r.GetString(0), new { id }); + return rows.Count == 0 ? null : rows[0]; + } + + private static Task> ReadEventIdsAsync(ILocalDb db) + => db.QueryAsync("SELECT id FROM site_events ORDER BY id", static r => r.GetString(0)); + + /// Polls until true or the deadline passes. + private static async Task WaitUntilAsync(Func> condition, string because) + { + var deadline = DateTime.UtcNow + ConvergeTimeout; + while (DateTime.UtcNow < deadline) + { + if (await condition()) return; + await Task.Delay(50); + } + + Assert.Fail($"Timed out after {ConvergeTimeout.TotalSeconds:0}s waiting for: {because}"); + } + + // ---- scenarios -------------------------------------------------------------------- + + [Fact] + public async Task TrackingRow_WrittenOnA_BecomesReadableOnB() + { + await WriteTrackingAsync(A, "op-a-1", "Submitted", "ERP.GetOrder"); + + await WaitUntilAsync( + async () => await ReadTrackingStatusAsync(B, "op-a-1") == "Submitted", + "the tracking row written on node A to appear on node B"); + } + + [Fact] + public async Task TrackingRow_WrittenOnB_BecomesReadableOnA() + { + // Replication is bidirectional even though only A dials: proving the passive node's + // writes flow back is what makes a failover in EITHER direction safe. + await WriteTrackingAsync(B, "op-b-1", "Submitted", "ERP.GetOrder"); + + await WaitUntilAsync( + async () => await ReadTrackingStatusAsync(A, "op-b-1") == "Submitted", + "the tracking row written on node B to appear on node A"); + } + + [Fact] + public async Task SameOperation_UpdatedOnBothNodes_ConvergesToOneWinner() + { + // Last-writer-wins on the primary key. The specific winner is not asserted — that is + // the HLC's business — but the two nodes MUST agree, and must agree on a value one of + // them actually wrote rather than a merge of both. + await WriteTrackingAsync(A, "op-conflict", "Submitted", "ERP.GetOrder"); + await WaitUntilAsync( + async () => await ReadTrackingStatusAsync(B, "op-conflict") is not null, + "the conflicting row to exist on both nodes before it is updated"); + + await WriteTrackingAsync(A, "op-conflict", "Delivered", "ERP.GetOrder"); + await WriteTrackingAsync(B, "op-conflict", "Parked", "ERP.GetOrder"); + + await WaitUntilAsync( + async () => + { + var a = await ReadTrackingStatusAsync(A, "op-conflict"); + var b = await ReadTrackingStatusAsync(B, "op-conflict"); + return a is not null && a == b; + }, + "both nodes to converge on one status for the contended operation"); + + var winner = await ReadTrackingStatusAsync(A, "op-conflict"); + Assert.Contains(winner, new[] { "Delivered", "Parked" }); + } + + [Fact] + public async Task EventsLoggedOnBothNodes_ConvergeToTheUnion_WithNoIdCollisions() + { + // The reason site_events moved to GUID ids. Under the old autoincrement scheme both + // nodes would independently mint id=1, id=2, ... and last-writer-wins on the primary + // key would silently DESTROY one node's events instead of merging them. The union + // count is the assertion that catches that. + var idsFromA = Enumerable.Range(0, 5).Select(_ => Guid.NewGuid().ToString("N")).ToList(); + var idsFromB = Enumerable.Range(0, 5).Select(_ => Guid.NewGuid().ToString("N")).ToList(); + + foreach (var id in idsFromA) await WriteEventAsync(A, id, "from A"); + foreach (var id in idsFromB) await WriteEventAsync(B, id, "from B"); + + var expected = idsFromA.Concat(idsFromB).OrderBy(x => x, StringComparer.Ordinal).ToList(); + + await WaitUntilAsync( + async () => (await ReadEventIdsAsync(A)).SequenceEqual(expected) + && (await ReadEventIdsAsync(B)).SequenceEqual(expected), + "both nodes to hold the union of all 10 events"); + } + + [Fact] + public async Task PeerOffline_ThenRejoins_CatchesUpOnEverythingItMissed() + { + // The failover case that motivates Phase 1: one node is down while the other keeps + // working, and nothing written during the outage may be lost. + await StopHostAsync(_serverHost); + _serverHost = null; + + var idsDuringOutage = Enumerable.Range(0, 5).Select(_ => Guid.NewGuid().ToString("N")).ToList(); + foreach (var id in idsDuringOutage) await WriteEventAsync(A, id, "written while B was down"); + await WriteTrackingAsync(A, "op-during-outage", "Delivered", "ERP.GetOrder"); + + // Node B's database survived the host teardown (pre-constructed instance), so this is + // a genuine rejoin rather than a fresh node. It comes back on a NEW loopback port; + // the initiator's channel factory re-reads the peer address on each reconnect. + await StartPassiveAsync(); + await StopHostAsync(_initiatorHost); + await StartInitiatorAsync(); + + await WaitUntilAsync( + async () => + { + var events = await ReadEventIdsAsync(B); + return idsDuringOutage.All(events.Contains) + && await ReadTrackingStatusAsync(B, "op-during-outage") == "Delivered"; + }, + "node B to catch up on everything written while it was offline"); + } +} From 9cf7c0f0073619433fb737590875270550d8dbde Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sun, 19 Jul 2026 09:43:04 -0400 Subject: [PATCH 17/55] feat(localdb): enable replication on the site-a rig pair Task 11. site-a-a is the initiator (dials scadabridge-site-a-b:8083, the existing site gRPC h2c listener - no new port); site-a-b is passive. Both carry the SAME dev ApiKey, which is load-bearing: LocalDbSyncAuthInterceptor is fail-closed, so a mismatch does not degrade to unauthenticated replication, it rejects every stream and the pair silently stops converging. site-b and site-c stay unreplicated on purpose, so the default-OFF posture is proven side-by-side on one rig rather than asserted. Plain dev key matches the rig's existing posture; production uses a ${secret:...} reference through the pre-host expander. Noted in both files. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts --- docker/site-a-node-a/appsettings.Site.json | 19 ++++++++++++++++++- docker/site-a-node-b/appsettings.Site.json | 12 +++++++++++- 2 files changed, 29 insertions(+), 2 deletions(-) diff --git a/docker/site-a-node-a/appsettings.Site.json b/docker/site-a-node-a/appsettings.Site.json index 7c841b41..53be3432 100644 --- a/docker/site-a-node-a/appsettings.Site.json +++ b/docker/site-a-node-a/appsettings.Site.json @@ -64,6 +64,23 @@ // outside the volume and were lost on every recreate. // Replication is opt-in and configured separately; absent = local-only. "LocalDb": { - "Path": "/app/data/site-localdb.db" + "Path": "/app/data/site-localdb.db", + // Site-a is the rig's REPLICATED pair; site-b and site-c deliberately stay + // unreplicated so the default-OFF posture is proven side-by-side on one rig. + // + // This node is the initiator: it dials the peer. Replication is still + // bidirectional - the passive node's writes flow back over the same stream - + // so only one side needs PeerAddress. Port 8083 is the existing site gRPC + // listener (h2c); the sync endpoint shares it, no new port. + // + // The ApiKey must be IDENTICAL on both nodes: the host's + // LocalDbSyncAuthInterceptor is fail-closed, so a mismatch (or a missing key) + // rejects every sync stream. A plain dev key here matches the rig's existing + // posture; PRODUCTION uses a "${secret:...}" reference resolved by the + // pre-host secret expander. + "Replication": { + "PeerAddress": "http://scadabridge-site-a-b:8083", + "ApiKey": "dev-site-a-localdb-sync-key" + } } } diff --git a/docker/site-a-node-b/appsettings.Site.json b/docker/site-a-node-b/appsettings.Site.json index 299055cc..8e7bd55c 100644 --- a/docker/site-a-node-b/appsettings.Site.json +++ b/docker/site-a-node-b/appsettings.Site.json @@ -64,6 +64,16 @@ // outside the volume and were lost on every recreate. // Replication is opt-in and configured separately; absent = local-only. "LocalDb": { - "Path": "/app/data/site-localdb.db" + "Path": "/app/data/site-localdb.db", + // The PASSIVE half of site-a's replicated pair: no PeerAddress, so this node's + // sync initiator starts and idles while node-a dials in. Its own writes still + // reach node-a - the stream is bidirectional. + // + // The key MUST match node-a's exactly. LocalDbSyncAuthInterceptor is + // fail-closed, so a typo here does not degrade to unauthenticated replication; + // it rejects every stream and the pair silently stops converging. + "Replication": { + "ApiKey": "dev-site-a-localdb-sync-key" + } } } From 95c108f1d78cbb219edbf5f751e8a3a1107acf52 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sun, 19 Jul 2026 09:50:11 -0400 Subject: [PATCH 18/55] fix(localdb): allowlist the replication meter + record the Task 12 live gate Task 12 (live gate on the docker rig) found one real defect, which this fixes. THE DEFECT. The plan asserted LocalDb metrics "need no work - the ZB.MOM.WW.LocalDb.Replication meter flows through the existing AddZbTelemetry Prometheus export". It does not. ZbTelemetryOptions.Meters is an ALLOWLIST: an unlisted meter's instruments are never observed, with no error, no warning, and no missing-metric signal anywhere. The rig's /metrics scrape returned 301 lines and zero localdb_* series. Only looking for them found it. Fixed by hoisting the allowlist to SiteServiceRegistration.ObservedMeters and adding LocalDbMetrics.MeterName. The pin test asserts on that constant rather than on OTel's observed set, because AddZbTelemetry applies its options to a local instance and never registers IOptions - there is nothing resolvable to interrogate. The end-to-end proof is the live scrape below; the test exists to stop the entry being dropped again. Called out as such in the test. LIVE GATE EVIDENCE (docker rig, 8 nodes, rebuilt via docker/deploy.sh): 1. Consolidated DB created on every site node at /app/data/site-localdb.db - inside the mounted volume, unlike the legacy CWD-relative databases. 2. Replication proven REAL, not two independent local writes. Both site-a nodes hold the identical row and, decisively, the identical originating node_id and HLC in __localdb_row_version: site_events|{"id":"8b06b37c..."}|116946936661147648|65b00319-...|0 __localdb_peer_state shows a completed handshake in both directions. 3. Convergence: 5/5 events on both nodes, __localdb_row_version byte-identical across the pair, oplog drained to 0, dead_letter 0 on both. 4. SITE failover drill. Note the repo's failover-drill.sh targets CENTRAL nodes and does not exercise this claim, so the site-pair flip was run directly: stopped site-a-a (the node holding the history); site-a-b survived with the complete pre-flip history (3/3 events) plus its own takeover event. Restarted site-a-a; it caught up on what it missed and both nodes reconverged to the same 5 ids. This is the failure Phase 1 exists to prevent. 5. localdb_* now exported after the fix: localdb_sync_reconnects_total{...} 1 localdb_oplog_depth{...} 0 6. DEFAULT-OFF pin, live and side-by-side: site-b (no Replication section) boots identically, creates its consolidated DB, registers the engine - and stays inert. Zero replication log lines, no reconnect counter at all, so it never dialled anything. One transient WRN at startup on site-a-a ("Connection refused", reconnecting) is node A dialling before node B's listener was up; it reconnected within a second. Expected for a pair that starts together. Verified: build 0 warnings; Host wiring tests 11/11. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts --- .../SiteServiceRegistration.cs | 20 ++++++++++++++++++- .../SiteLocalDbWiringTests.cs | 19 ++++++++++++++++++ 2 files changed, 38 insertions(+), 1 deletion(-) diff --git a/src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs b/src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs index e943b705..afc500f0 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs @@ -27,6 +27,24 @@ namespace ZB.MOM.WW.ScadaBridge.Host; /// public static class SiteServiceRegistration { + /// + /// Every OpenTelemetry is told to observe. + /// + /// + /// ZbTelemetryOptions.Meters is an ALLOWLIST, and an unlisted meter's instruments + /// are simply never observed — no error, no warning, no missing-metric signal anywhere. + /// The LocalDb replication meter was silently absent from the rig's /metrics + /// scrape until the Phase 1 live gate went looking for it (the plan had assumed it + /// flowed through automatically). Hoisted to a named constant so the list is + /// assertable; adding a meter anywhere in the product means adding it here. + /// + public static readonly string[] ObservedMeters = + [ + ScadaBridgeTelemetry.MeterName, + // Emitted only on site nodes; harmless on central, which never records against it. + LocalDbMetrics.MeterName, + ]; + /// Registers all DI services required for the site role. /// The service collection to register into. /// Application configuration for options binding. @@ -262,7 +280,7 @@ public static class SiteServiceRegistration o.ServiceName = "scadabridge"; o.SiteId = config["ScadaBridge:Node:SiteId"] ?? "central"; o.NodeRole = config["ScadaBridge:Node:Role"]; - o.Meters = [ScadaBridgeTelemetry.MeterName]; + o.Meters = ObservedMeters; if (Enum.TryParse(config["ScadaBridge:Telemetry:Exporter"], ignoreCase: true, out var exporter)) o.Exporter = exporter; var otlp = config["ScadaBridge:Telemetry:OtlpEndpoint"]; diff --git a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbWiringTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbWiringTests.cs index cff13488..c2d3fc91 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbWiringTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbWiringTests.cs @@ -3,7 +3,9 @@ using Microsoft.Extensions.Configuration; using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.Hosting; using Microsoft.Extensions.Logging.Abstractions; +using Microsoft.Extensions.Options; using ZB.MOM.WW.ScadaBridge.HealthMonitoring; +using ZB.MOM.WW.Telemetry; using ZB.MOM.WW.LocalDb; using ZB.MOM.WW.LocalDb.Replication; using ZB.MOM.WW.ScadaBridge.Host; @@ -196,6 +198,23 @@ public class SiteLocalDbWiringTests : IDisposable Assert.Null(report.LocalDbOplogBacklog); } + [Fact] + public void Site_Telemetry_Allowlists_TheLocalDbReplicationMeter() + { + // ZbTelemetryOptions.Meters is an ALLOWLIST: an unlisted Meter's instruments are + // never observed, with no error and no missing-metric warning anywhere. The + // replication meter was silently absent from the rig's /metrics scrape until the + // live gate looked for it, so pin the allowlist rather than trusting the next + // reader to remember. + // + // This asserts on the allowlist constant, not on OTel's observed-meter set: + // AddZbTelemetry applies its options to a local instance and never registers + // IOptions, so there is nothing resolvable to interrogate. The end-to-end proof + // that localdb_* actually reaches /metrics is the Task 12 live gate; this test + // exists to stop the entry being dropped again. + Assert.Contains(LocalDbMetrics.MeterName, SiteServiceRegistration.ObservedMeters); + } + [Fact] public void Site_LocalDb_CreatesTheConfiguredFile() { From 568735a41ef54a3710a2d6844006917e6f6758df Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sun, 19 Jul 2026 09:51:55 -0400 Subject: [PATCH 19/55] docs(localdb): Phase 1 truth pass Task 13. SiteEventLogger's XML doc said 'Not replicated to standby. On failover, the new active node starts a fresh log.' That is now false and was the exact behaviour Phase 1 set out to fix. OperationTrackingOptions.ConnectionString and SiteEventLogOptions.DatabasePath marked MIGRATION-ONLY: nothing reads them but SiteLocalDbLegacyMigrator, and they are safe to delete from a config once a node has migrated. ScadaBridge CLAUDE.md gains a consolidated-site-database entry covering the GUID/opaque-token contract change, the migration-only keys, the incidental CWD-outside-the-volume data-loss fix, the fail-closed default-OFF replication posture, and the fact that ZbTelemetryOptions.Meters is a silent allowlist. scadaproj CLAUDE.md's LocalDb row goes from 'no app adoption yet' to the honest current state, per the component-status-honesty convention: ScadaBridge Phase 1 only, on an UNMERGED branch, replication live-proven on the rig's site-a pair ONLY, default-OFF everywhere else, Phase 2 not started, no other app adopted, no production deployment replicating. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts --- CLAUDE.md | 6 ++++++ .../SiteEventLogOptions.cs | 9 ++++++++- .../SiteEventLogger.cs | 8 +++++--- .../Tracking/OperationTrackingOptions.cs | 9 ++++++--- 4 files changed, 25 insertions(+), 7 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 7d993e6d..a94e758b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -120,6 +120,12 @@ Other peers in the `scadaproj` family (see `scadaproj/CLAUDE.md` for details): ` - DCL write failures returned synchronously to calling script. - Tag path resolution retried periodically for devices still booting. - Static attribute writes persisted to local SQLite (survive restart/failover, reset on redeployment). +- **Consolidated site database (LocalDb Phase 1, 2026-07-19).** `OperationTracking` and `site_events` now live in ONE `ZB.MOM.WW.LocalDb`-managed SQLite file, configured by the **required** `LocalDb:Path` (`/app/data/site-localdb.db` on the rig; validated with `ValidateOnStart`, so a site config missing it fails to boot). Both are `RegisterReplicated` tables. Consequences worth knowing: + - `site_events.id` changed from autoincrement INTEGER to an application-minted **GUID**. Last-writer-wins keys on the primary key, so two nodes independently minting `id=1,2,3…` would destroy each other's events rather than merge them. The event-log read path uses a composite `(timestamp, id)` keyset cursor with an **opaque string** continuation token; `EventLogEntry.Id` and both `ContinuationToken`s are `string`/`string?` on the site↔central Akka DTOs. + - `ScadaBridge:OperationTracking:ConnectionString` and `ScadaBridge:SiteEventLog:DatabasePath` are **migration-only** — nothing reads them but `SiteLocalDbLegacyMigrator`, which copies a pre-Phase-1 file in once (deterministic `mig-{NodeName}-{legacyId}` event ids, `INSERT OR IGNORE`, runs AFTER `RegisterReplicated` so migrated rows replicate) and renames it `.migrated`. Delete the keys once a node has migrated. + - This incidentally fixes a data-loss bug: both legacy databases defaulted to CWD-relative paths **outside** the mounted volume and were discarded on every container recreate. + - **Replication is default-OFF and opt-in** via `LocalDb:Replication:PeerAddress` + a matching `ApiKey` on both nodes. `LocalDbSyncAuthInterceptor` is **fail-closed**: no configured key means no sync stream is accepted at all, so a key typo does not degrade to unauthenticated replication — the pair simply stops converging. The sync endpoint shares the existing site gRPC h2c listener (8083); no new port. Rig posture: **site-a replicated, site-b/site-c deliberately not**, so both states are proven side-by-side. Status surfaces on the site health report as `LocalDbReplicationConnected` / `LocalDbOplogBacklog` (both nullable — null means "no data", NOT "disconnected with an empty backlog") and as `localdb_*` Prometheus series. Note `ZbTelemetryOptions.Meters` is an **allowlist** (`SiteServiceRegistration.ObservedMeters`); an unlisted meter exports nothing, silently. + - Design: `docs/plans/2026-07-19-scadabridge-localdb-design.md`; Phase 1 plan `docs/plans/2026-07-19-localdb-adoption-phase1.md`. Phase 2 (config tables + `sf_messages`, deleting `SiteReplicationActor` + StoreAndForward `ReplicationService`) is NOT started and needs its own plan. - All timestamps are UTC throughout the system. - Inter-cluster communication uses two transports: ClusterClient for command/control (deployments, lifecycle, subscribe/unsubscribe handshake, snapshots) and gRPC server-streaming for real-time data (attribute values, alarm states). Both CentralCommunicationActor and SiteCommunicationActor registered with receptionist. Central creates one ClusterClient per site using NodeA/NodeB as contact points. Sites configure multiple central contact points for failover. Addresses cached in CentralCommunicationActor, refreshed periodically (60s) and on admin changes. Heartbeats serve health monitoring only. - gRPC streaming channel: SiteStreamGrpcServer on each site node (Kestrel HTTP/2, port 8083); central creates per-site SiteStreamGrpcClient via SiteStreamGrpcClientFactory. Site entity has GrpcNodeAAddress/GrpcNodeBAddress fields. Proto: sitestream.proto with SiteStreamService, SiteStreamEvent (oneof: AttributeValueUpdate, AlarmStateUpdate). DebugStreamEvent message removed (no longer flows through ClusterClient). diff --git a/src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogOptions.cs b/src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogOptions.cs index 78a9a3d3..483ac83e 100644 --- a/src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogOptions.cs +++ b/src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogOptions.cs @@ -6,7 +6,14 @@ public class SiteEventLogOptions public int RetentionDays { get; set; } = 30; /// Maximum SQLite database size in megabytes before old entries are purged; default 1024 MB. public int MaxStorageMb { get; set; } = 1024; - /// File path for the site event log SQLite database. + /// + /// MIGRATION-ONLY as of LocalDb Phase 1. no longer reads + /// this: site_events lives in the consolidated site database at + /// LocalDb:Path. The only remaining consumer is + /// SiteLocalDbLegacyMigrator, which uses it to locate a pre-Phase-1 + /// site_events.db and copy it in once. Safe to delete from a config once that + /// node's migration has run. + /// public string DatabasePath { get; set; } = "site_events.db"; /// Maximum number of rows returned per paginated query; default 500. public int QueryPageSize { get; set; } = 500; diff --git a/src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs b/src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs index a5c6e868..0b95b684 100644 --- a/src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs +++ b/src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs @@ -7,9 +7,11 @@ using ZB.MOM.WW.LocalDb; namespace ZB.MOM.WW.ScadaBridge.SiteEventLogging; /// -/// Records operational events to a local SQLite database. -/// Only the active node generates events. Not replicated to standby. -/// On failover, the new active node starts a fresh log. +/// Records operational events into the consolidated site database (LocalDb Phase 1). +/// Only the active node generates events, but site_events is a replicated table: +/// when the pair has a peer configured, the standby holds the same history and a failover +/// no longer starts a fresh log. With no peer configured the database is simply local — +/// replication is opt-in via LocalDb:Replication:PeerAddress. /// /// /// diff --git a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Tracking/OperationTrackingOptions.cs b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Tracking/OperationTrackingOptions.cs index 6ca4c2cc..76979eeb 100644 --- a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Tracking/OperationTrackingOptions.cs +++ b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Tracking/OperationTrackingOptions.cs @@ -7,9 +7,12 @@ namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tracking; public class OperationTrackingOptions { /// - /// Full ADO.NET connection string for the SQLite database (e.g. - /// "Data Source=site-tracking.db"). Tests use the - /// Mode=Memory;Cache=Shared form to keep the database in-memory. + /// MIGRATION-ONLY as of LocalDb Phase 1. no longer + /// reads this: the tracking table lives in the consolidated site database at + /// LocalDb:Path. The only remaining consumer is + /// SiteLocalDbLegacyMigrator, which uses it to locate a pre-Phase-1 + /// site-tracking.db and copy it in once. Safe to delete from a config once that + /// node's migration has run. /// public string ConnectionString { get; set; } = "Data Source=site-tracking.db"; From b64857c8d6016d019177e0eca87f0dc9d467b5af Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sun, 19 Jul 2026 10:00:50 -0400 Subject: [PATCH 20/55] docs(localdb): Phase 2 gate + final Phase 1 task record Task 14. A GATE document, not an implementation plan - the plan explicitly wants Phase 2 planned against post-Phase-1 reality, which means after this record exists rather than from the original design alone. Records what Phase 1 actually established (ordering constraints, where registration lives and why, the fail-closed interceptor, the silent meter allowlist), what must be ported before anything is deleted (both bespoke replicator test files, treated as specifications, plus the accepted N5 bounded-duplicate race), and why Phase 2 is a harder risk class than Phase 1: it REPLACES a working mechanism rather than adding one where none existed, sf_messages has external side effects, config tables drive deployment, there is no dual-mechanism period, and the migration is real this time rather than the usual no-op. The most important line is an open question, not an answer: choosing MaxOplogRows / MaxOplogAge / snapshot-resync thresholds for sf_messages write rates needs a Phase-1 rig SOAK that has not been run. The live gate proved correctness, not steady-state behaviour over time. Phase 2 should not be planned without it. tasks.json finalized: all 15 tasks completed, every deviation from the plan recorded with its reason. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts --- CLAUDE.md | 2 +- ...7-19-localdb-adoption-phase1.md.tasks.json | 169 ++++++++++++++++-- docs/plans/2026-07-19-localdb-phase2-gate.md | 101 +++++++++++ 3 files changed, 254 insertions(+), 18 deletions(-) create mode 100644 docs/plans/2026-07-19-localdb-phase2-gate.md diff --git a/CLAUDE.md b/CLAUDE.md index a94e758b..28e7bc35 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -125,7 +125,7 @@ Other peers in the `scadaproj` family (see `scadaproj/CLAUDE.md` for details): ` - `ScadaBridge:OperationTracking:ConnectionString` and `ScadaBridge:SiteEventLog:DatabasePath` are **migration-only** — nothing reads them but `SiteLocalDbLegacyMigrator`, which copies a pre-Phase-1 file in once (deterministic `mig-{NodeName}-{legacyId}` event ids, `INSERT OR IGNORE`, runs AFTER `RegisterReplicated` so migrated rows replicate) and renames it `.migrated`. Delete the keys once a node has migrated. - This incidentally fixes a data-loss bug: both legacy databases defaulted to CWD-relative paths **outside** the mounted volume and were discarded on every container recreate. - **Replication is default-OFF and opt-in** via `LocalDb:Replication:PeerAddress` + a matching `ApiKey` on both nodes. `LocalDbSyncAuthInterceptor` is **fail-closed**: no configured key means no sync stream is accepted at all, so a key typo does not degrade to unauthenticated replication — the pair simply stops converging. The sync endpoint shares the existing site gRPC h2c listener (8083); no new port. Rig posture: **site-a replicated, site-b/site-c deliberately not**, so both states are proven side-by-side. Status surfaces on the site health report as `LocalDbReplicationConnected` / `LocalDbOplogBacklog` (both nullable — null means "no data", NOT "disconnected with an empty backlog") and as `localdb_*` Prometheus series. Note `ZbTelemetryOptions.Meters` is an **allowlist** (`SiteServiceRegistration.ObservedMeters`); an unlisted meter exports nothing, silently. - - Design: `docs/plans/2026-07-19-scadabridge-localdb-design.md`; Phase 1 plan `docs/plans/2026-07-19-localdb-adoption-phase1.md`. Phase 2 (config tables + `sf_messages`, deleting `SiteReplicationActor` + StoreAndForward `ReplicationService`) is NOT started and needs its own plan. + - Design: **scadaproj** `docs/plans/2026-07-19-scadabridge-localdb-design.md` (that doc lives in the umbrella repo, not here); Phase 1 plan + Phase 2 gate are here under `docs/plans/`. Phase 2 (config tables + `sf_messages`, deleting `SiteReplicationActor` + StoreAndForward `ReplicationService`) is NOT started — see `docs/plans/2026-07-19-localdb-phase2-gate.md`. - All timestamps are UTC throughout the system. - Inter-cluster communication uses two transports: ClusterClient for command/control (deployments, lifecycle, subscribe/unsubscribe handshake, snapshots) and gRPC server-streaming for real-time data (attribute values, alarm states). Both CentralCommunicationActor and SiteCommunicationActor registered with receptionist. Central creates one ClusterClient per site using NodeA/NodeB as contact points. Sites configure multiple central contact points for failover. Addresses cached in CentralCommunicationActor, refreshed periodically (60s) and on admin changes. Heartbeats serve health monitoring only. - gRPC streaming channel: SiteStreamGrpcServer on each site node (Kestrel HTTP/2, port 8083); central creates per-site SiteStreamGrpcClient via SiteStreamGrpcClientFactory. Site entity has GrpcNodeAAddress/GrpcNodeBAddress fields. Proto: sitestream.proto with SiteStreamService, SiteStreamEvent (oneof: AttributeValueUpdate, AlarmStateUpdate). DebugStreamEvent message removed (no longer flows through ClusterClient). diff --git a/docs/plans/2026-07-19-localdb-adoption-phase1.md.tasks.json b/docs/plans/2026-07-19-localdb-adoption-phase1.md.tasks.json index 98d191c1..89584f89 100644 --- a/docs/plans/2026-07-19-localdb-adoption-phase1.md.tasks.json +++ b/docs/plans/2026-07-19-localdb-adoption-phase1.md.tasks.json @@ -30,21 +30,155 @@ } ], "tasks": [ - {"id": 1, "subject": "Task 1: Add LocalDb packages to the build", "status": "completed", "commit": "7370b331"}, - {"id": 2, "subject": "Task 2: Shared schema helpers (tracking + events, GUID PK)", "status": "completed", "blockedBy": [1], "commit": "727fa48c"}, - {"id": 3, "subject": "Task 3: Register LocalDb in the site composition root", "status": "completed", "blockedBy": [2], "commit": "2977e6c4", "note": "LocalDb:Path added to 10 configs (8 site nodes incl. docker-env2 site-x, dev template, wonder prod sample) - it is REQUIRED via ValidateOnStart, so any site config missing it fails startup."}, - {"id": 4, "subject": "Task 4: Rewire OperationTrackingStore onto ILocalDb connections", "status": "completed", "blockedBy": [3], "commit": "0b5e9b44", "note": "Ctor takes ILocalDb, not IOptions; both ad-hoc reader paths switched too and their OpenAsync calls removed (CreateConnection returns an OPEN connection). Three DI fixtures needed LocalDb:Path added because resolving IOperationTrackingStore now forces ILocalDb construction."}, - {"id": 5, "subject": "Task 5a: Rewire SiteEventLogger onto ILocalDb + mint GUID ids", "status": "completed", "blockedBy": [3], "commit": "727fa48c + 0d8a80aa", "note": "GUID minting landed early with Task 2 (727fa48c) to avoid committing a writer that violated site_events.id NOT NULL. The ILocalDb.CreateConnection() swap is 0d8a80aa. connectionStringOverride DELETED, not preserved: site_events is replicated, so a raw connection lacks zb_hlc_next() and fails closed."}, - {"id": 15, "subject": "Task 5b: Migrate the event-log read path off integer ids", "status": "completed", "blockedBy": [5], "commit": "727fa48c"}, - {"id": 6, "subject": "Task 6: One-time legacy data migrator", "status": "pending", "blockedBy": [4, 5]}, - {"id": 7, "subject": "Task 7: Replication registration + sync endpoint (default OFF)", "status": "completed", "blockedBy": [3], "commit": "a560e9ea", "note": "DEVIATION: AddZbLocalDbReplication moved from Program.cs (where the plan put it) into SiteServiceRegistration, because the composition-root tests build that graph and not Program.cs - in Program.cs a missing registration would still show green. MapZbLocalDbSync stays in Program.cs (needs the WebApplication). Endpoint is UNAUTHENTICATED until Task 8; no config in this repo sets a peer."}, - {"id": 8, "subject": "Task 8: Bearer auth interceptor for the sync endpoint", "status": "pending", "blockedBy": [7]}, - {"id": 9, "subject": "Task 9: Surface ISyncStatus as site health signal", "status": "pending", "blockedBy": [7]}, - {"id": 10, "subject": "Task 10: Two-node in-process convergence test", "status": "pending", "blockedBy": [6, 8, 15]}, - {"id": 11, "subject": "Task 11: Docker rig enablement (site-a pair)", "status": "pending", "blockedBy": [8]}, - {"id": 12, "subject": "Task 12: Live gate on the docker rig", "status": "pending", "blockedBy": [9, 10, 11]}, - {"id": 13, "subject": "Task 13: Documentation truth pass", "status": "pending", "blockedBy": [12]}, - {"id": 14, "subject": "Task 14: Phase 2 gate - plan the bespoke-replicator migration", "status": "pending", "blockedBy": [12]} + { + "id": 1, + "subject": "Task 1: Add LocalDb packages to the build", + "status": "completed", + "commit": "7370b331" + }, + { + "id": 2, + "subject": "Task 2: Shared schema helpers (tracking + events, GUID PK)", + "status": "completed", + "blockedBy": [ + 1 + ], + "commit": "727fa48c" + }, + { + "id": 3, + "subject": "Task 3: Register LocalDb in the site composition root", + "status": "completed", + "blockedBy": [ + 2 + ], + "commit": "2977e6c4", + "note": "LocalDb:Path added to 10 configs (8 site nodes incl. docker-env2 site-x, dev template, wonder prod sample) - it is REQUIRED via ValidateOnStart, so any site config missing it fails startup." + }, + { + "id": 4, + "subject": "Task 4: Rewire OperationTrackingStore onto ILocalDb connections", + "status": "completed", + "blockedBy": [ + 3 + ], + "commit": "0b5e9b44", + "note": "Ctor takes ILocalDb, not IOptions; both ad-hoc reader paths switched too and their OpenAsync calls removed (CreateConnection returns an OPEN connection). Three DI fixtures needed LocalDb:Path added because resolving IOperationTrackingStore now forces ILocalDb construction." + }, + { + "id": 5, + "subject": "Task 5a: Rewire SiteEventLogger onto ILocalDb + mint GUID ids", + "status": "completed", + "blockedBy": [ + 3 + ], + "commit": "727fa48c + 0d8a80aa", + "note": "GUID minting landed early with Task 2 (727fa48c) to avoid committing a writer that violated site_events.id NOT NULL. The ILocalDb.CreateConnection() swap is 0d8a80aa. connectionStringOverride DELETED, not preserved: site_events is replicated, so a raw connection lacks zb_hlc_next() and fails closed." + }, + { + "id": 15, + "subject": "Task 5b: Migrate the event-log read path off integer ids", + "status": "completed", + "blockedBy": [ + 5 + ], + "commit": "727fa48c" + }, + { + "id": 6, + "subject": "Task 6: One-time legacy data migrator", + "status": "completed", + "blockedBy": [ + 4, + 5 + ], + "commit": "98b94771", + "note": "Runs AFTER RegisterReplicated (pinned by an oplog assertion). Deterministic mig-{NodeName}-{legacyId} event ids, not fresh GUIDs, so a crash between commit and rename cannot duplicate. DEVIATION: sync transaction on a CreateConnection() connection instead of BeginTransactionAsync - OnReady is a synchronous Action." + }, + { + "id": 7, + "subject": "Task 7: Replication registration + sync endpoint (default OFF)", + "status": "completed", + "blockedBy": [ + 3 + ], + "commit": "a560e9ea", + "note": "DEVIATION: AddZbLocalDbReplication moved from Program.cs (where the plan put it) into SiteServiceRegistration, because the composition-root tests build that graph and not Program.cs - in Program.cs a missing registration would still show green. MapZbLocalDbSync stays in Program.cs (needs the WebApplication). Endpoint is UNAUTHENTICATED until Task 8; no config in this repo sets a peer." + }, + { + "id": 8, + "subject": "Task 8: Bearer auth interceptor for the sync endpoint", + "status": "completed", + "blockedBy": [ + 7 + ], + "commit": "59c69519", + "note": "Fail-closed: no configured ApiKey rejects ALL sync streams. All four handler shapes gated, not just unary (the real RPC is a duplex stream). DEVIATION: Grpc.Core.Testing does not exist on grpc-dotnet; hand-rolled FakeServerCallContext instead of adding the retired native package." + }, + { + "id": 9, + "subject": "Task 9: Surface ISyncStatus as site health signal", + "status": "completed", + "blockedBy": [ + 7 + ], + "commit": "59c69519", + "note": "Additive init properties on SiteHealthReport so the Akka DTO ctor is untouched. Both fields nullable: null = no data, false/0 = a real reading (which is the healthy default-OFF state). Collector stores one tuple, read once, so a torn read cannot pair fresh Connected with stale backlog." + }, + { + "id": 10, + "subject": "Task 10: Two-node in-process convergence test", + "status": "completed", + "blockedBy": [ + 6, + 8, + 15 + ], + "commit": "3c395d79", + "note": "5 scenarios over real loopback gRPC + the real interceptor, initialized via SiteLocalDbSetup.OnReady. Verified NON-vacuous: mismatching the two ApiKeys turns all 5 red with 2m30s of timeouts." + }, + { + "id": 11, + "subject": "Task 11: Docker rig enablement (site-a pair)", + "status": "completed", + "blockedBy": [ + 8 + ], + "commit": "(with Task 11 commit)", + "note": "site-a replicated, site-b/site-c deliberately not, so default-OFF is proven side-by-side on one rig." + }, + { + "id": 12, + "subject": "Task 12: Live gate on the docker rig", + "status": "completed", + "blockedBy": [ + 9, + 10, + 11 + ], + "commit": "95c108f1", + "note": "PASS. Found one REAL defect: ZbTelemetryOptions.Meters is a silent allowlist, so localdb_* was absent from /metrics - the plan's 'flows through automatically' claim was wrong. Fixed + pinned. Evidence: identical row_version/HLC/originating node_id across the pair, 0 dead letters, oplog drained, site failover + rejoin catch-up, localdb_* scraped, site-b inert. NOTE: docker/failover-drill.sh targets CENTRAL nodes and does NOT exercise the site-pair claim; the site flip was run directly." + }, + { + "id": 13, + "subject": "Task 13: Documentation truth pass", + "status": "completed", + "blockedBy": [ + 12 + ], + "commit": "(docs commit)", + "note": "Both CLAUDE.md files + the false 'starts a fresh log' XML doc + the two now-migration-only option properties." + }, + { + "id": 14, + "subject": "Task 14: Phase 2 gate - plan the bespoke-replicator migration", + "status": "completed", + "blockedBy": [ + 12 + ], + "commit": "(gate commit)", + "note": "Gate document only, per the plan - Phase 2 gets its own implementation plan. docs/plans/2026-07-19-localdb-phase2-gate.md. KEY FINDING: the rig SOAK that Phase 2's oplog-cap tuning needs has NOT been run; the live gate proved correctness, not steady-state behaviour." + } ], "knownFlakes": [ { @@ -56,5 +190,6 @@ "note": "Cold-MSSQL-fixture timing: ~91s and AwaitAssert-times-out on a cold fixture, ~1s and passes warm. Baselined against a stashed tree to confirm it is not a LocalDb regression." } ], - "lastUpdated": "2026-07-19" -} + "lastUpdated": "2026-07-19", + "phase1Status": "COMPLETE - all 15 tasks done, live gate PASS. Branch feat/localdb-phase1 NOT merged/pushed." +} \ No newline at end of file diff --git a/docs/plans/2026-07-19-localdb-phase2-gate.md b/docs/plans/2026-07-19-localdb-phase2-gate.md new file mode 100644 index 00000000..f68c1aae --- /dev/null +++ b/docs/plans/2026-07-19-localdb-phase2-gate.md @@ -0,0 +1,101 @@ +# LocalDb Phase 2 — Gate Document + +> **Status: NOT STARTED.** This is the gate, not the plan. Phase 2 work must not begin +> until an implementation plan is written against the facts below and the open questions +> in §5 are answered. + +**Phase 1 live gate: PASS** (2026-07-19, docker rig). See the Task 12 commit for evidence. + +**Phase 2 goal (from the design doc, scadaproj `docs/plans/2026-07-19-scadabridge-localdb-design.md` §1):** +move `scadabridge.db`'s config tables and StoreAndForward's `sf_messages` into the same +consolidated LocalDb file, then **delete** `SiteReplicationActor` and StoreAndForward's +`ReplicationService` outright. CDC triggers replace hand-shipped ops; the library's +snapshot resync replaces the bespoke chunked anti-entropy. The design explicitly chose +**replace + delete, no dual-mechanism period** — recoverable from git. + +--- + +## 1. What Phase 1 actually established + +Facts a Phase 2 plan can rely on, all verified rather than assumed: + +- One `ILocalDb` per site process at `LocalDb:Path`, **required** (`ValidateOnStart`). + A site config missing it fails to boot. +- `SiteLocalDbSetup.OnReady` is the single place tables are created and registered. + Ordering is load-bearing: **DDL → `RegisterReplicated` → writes**, then the legacy + migrator. Rows written before registration are invisible to the peer forever, silently. +- Replication is registered in `SiteServiceRegistration` (not `Program.cs`) so the + composition-root tests cover it; only `MapZbLocalDbSync` lives in `Program.cs`. +- Inbound auth is `LocalDbSyncAuthInterceptor`, fail-closed, scoped to + `/localdb_sync.v1.LocalDbSync/`, sharing the existing 8083 h2c listener. +- Default-OFF is real and proven side-by-side: rig site-a replicates, site-b/site-c do not. +- `ZbTelemetryOptions.Meters` is a **silent allowlist** (`SiteServiceRegistration.ObservedMeters`). +- Bidirectional sync works with only one node configured as initiator. + +## 2. What Phase 2 must port before deleting anything + +The bespoke replicator's tests encode behaviour that the design's CDC replacement must +still satisfy. **Port the test intents first, delete second** — the plan should treat the +existing tests as the specification, not as code to be removed alongside the implementation. + +- `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/SiteReplicationActorTests.cs` +- `tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ReplicationServiceTests.cs` + +Implementation under deletion: + +- `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/SiteReplicationActor.cs` +- `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/ReplicationService.cs` + +Known accepted behaviour to carry forward, not silently drop: the **N5 bounded-duplicate +race** the bespoke replicator documents and accepts. A Phase 2 plan must state whether CDC +inherits the same bound, a tighter one, or a different failure shape. + +## 3. Substantially harder than Phase 1 + +Phase 1 moved two tables that **nothing replicated before**, so the worst case was "no +worse than today". Phase 2 replaces a working mechanism, which makes it a different risk +class: + +- **`sf_messages` is an outbound buffer with side effects.** Phase 1's tables are + read-mostly records; a store-and-forward row that resurrects or double-delivers sends + real traffic to an external system. Park/requeue/remove becoming ordinary captured + row updates/deletes needs its own analysis of what LWW does to an in-flight send. +- **Config tables drive deployment.** A converged-but-wrong config row is a deployed + instance behaving incorrectly, not a missing history entry. +- **No dual-mechanism period.** The chosen posture means the cutover is the test. That + raises the bar on the offline convergence suite and the live gate considerably. +- **Migration is not a no-op this time.** Phase 1's migrator usually finds nothing, + because the legacy files sat outside the volume. `scadabridge.db` and + `store-and-forward.db` are real, populated, on the volume, and actively replicated + while the migration runs. + +## 4. Required inputs before writing the plan + +1. The design doc §1 Phase 2 and §2 schema/migration sections, re-read against + post-Phase-1 reality (several Phase 1 assumptions changed during execution — see the + `amendments` array in `2026-07-19-localdb-adoption-phase1.md.tasks.json`). +2. The two test files in §2, read as specifications. +3. **Rig soak observations that do not exist yet** — see §5. + +## 5. Open questions — answer these first + +- **Oplog growth and churn under real config/S&F write rates.** Phase 1's tables are + low-volume; `sf_messages` is not. `MaxOplogRows` / `MaxOplogAge` / snapshot-resync + thresholds cannot be chosen from first principles. *This needs a Phase-1 rig soak that + has not been run.* The live gate proved correctness, not steady-state behaviour over + time. +- **LWW semantics for in-flight store-and-forward sends.** What does last-writer-wins do + when one node parks a message the other is mid-delivery on? +- **Migration-under-load.** How does the one-time copy of an actively-replicating + `scadabridge.db` interact with the bespoke replicator still running during cutover? +- **Rollback story.** With no dual-mechanism period, what is the recovery path if the + cutover fails in production — beyond "revert the commit"? +- **Does the consolidated file stay appropriate?** One-DB-per-process was chosen partly + because Phase 1's tables were small. Adding config + S&F changes the size and write + profile of that single file. + +## 6. Not in scope (unchanged from the design) + +`auditlog.db` (diverges per node by design — central pulls the union; replicating it would +double-forward), the secrets store (has its own answer in `Secrets.Replicator.SqlServer` +hub mode), and central nodes (SQL-Server-first, no LocalDb use case). From f6ca82a9e2b7476b3d7dcd7182d8de363cb39949 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sun, 19 Jul 2026 15:04:37 -0400 Subject: [PATCH 21/55] docs(localdb): phase 2 implementation plan (21 tasks, full scope) Moves scadabridge.db's 9 config tables + sf_messages into the consolidated LocalDb file and deletes SiteReplicationActor + StoreAndForward's ReplicationService. Resolves the phase 2 gate's five open questions from code recon (D1-D5): D1 notify-and-fetch is DELETED, not preserved. It exists only because the config blob exceeds Akka's 128KB frame; LocalDb sync is gRPC. The deployed_at version guard protected against a stale fetch racing, so it dies with the fetch. Do not reproduce it on LWW - different clocks. D2 ReplaceAllAsync is deleted and the N1 directional guard becomes unnecessary: LocalDb's snapshot resync merges per-row LWW and never deletes (SnapshotApplier has no DELETE; LwwApplier.cs:69-78 discards a lower-HLC incoming row). Semantic change - the standby is convergent, no longer byte-identical. D3 The SMTP purge (plaintext passwords) rides the replication path and is re-homed to the active node BEFORE any deletion. D4 native_alarm_state volume is measured by a rig soak, not assumed. Task 1 gates the plan and stops it if the oplog cannot absorb the churn. D5 No dual-mechanism period forecloses rolling site upgrades - both nodes must stop and start together. Recon also found the gate doc's "two test files are the spec" undercounts: the real specification is five files, including the N1 Critical regression test and the only Requeue coverage. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts --- .../2026-07-19-localdb-adoption-phase2.md | 1019 +++++++++++++++++ ...7-19-localdb-adoption-phase2.md.tasks.json | 89 ++ 2 files changed, 1108 insertions(+) create mode 100644 docs/plans/2026-07-19-localdb-adoption-phase2.md create mode 100644 docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json diff --git a/docs/plans/2026-07-19-localdb-adoption-phase2.md b/docs/plans/2026-07-19-localdb-adoption-phase2.md new file mode 100644 index 00000000..a4315d6a --- /dev/null +++ b/docs/plans/2026-07-19-localdb-adoption-phase2.md @@ -0,0 +1,1019 @@ +# ScadaBridge LocalDb Phase 2 Implementation Plan + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers-extended-cc:executing-plans to implement this plan task-by-task. + +**Goal:** Move `scadabridge.db`'s 9 config tables and StoreAndForward's `sf_messages` into the +consolidated `ZB.MOM.WW.LocalDb` database established by Phase 1, then delete `SiteReplicationActor` +and StoreAndForward's `ReplicationService` outright. + +**Architecture:** Both stores stop owning their own SQLite files and take `ILocalDb` instead, exactly +as `OperationTrackingStore` and `SiteEventLogger` did in Phase 1. Their DDL is extracted into +`*Schema.Apply(SqliteConnection)` helpers that `SiteLocalDbSetup.OnReady` runs before +`RegisterReplicated`. Trigger-based CDC then replicates every row mutation, replacing the bespoke +hand-shipped ops; the library's per-row-LWW snapshot resync replaces the chunked anti-entropy. + +**Tech Stack:** .NET 10, Akka.NET, Microsoft.Data.Sqlite, ZB.MOM.WW.LocalDb 0.1.0, gRPC, xunit. + +**Branch:** `feat/localdb-phase2`, cut from `feat/localdb-phase1`. + +--- + +## Read this before Task 1 + +Phase 1's record is the prerequisite context. Read, in order: + +1. `docs/plans/2026-07-19-localdb-adoption-phase1.md.tasks.json` — the `amendments`, + `prerequisites` and `knownFlakes` arrays. Several Phase 1 assumptions changed during execution. +2. `docs/plans/2026-07-19-localdb-phase2-gate.md` — the gate. §5's open questions are answered in + the Decisions section below; the gate itself is superseded by this plan. +3. `src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbSetup.cs` — the pattern every new table follows. + +### Two known flakes — do not chase these + +- `SiteRuntime.Tests InstanceActorChildAttributeRaceTests` — intermittent `ActorNotFoundException` + under full-suite load. Passes in isolation. Pre-existing. +- `AuditLog.Tests ParentExecutionIdCorrelationTests` — ~91 s and times out on a **cold** MSSQL + fixture, ~1 s warm. Re-run before investigating. + +--- + +## Decisions — the gate's open questions, answered + +These were resolved from code recon before this plan was written. Each names its evidence. **Task 2 +records them formally; the rest of the plan assumes them.** + +### D1. Config moves to CDC, and notify-and-fetch is deleted, not preserved + +`SiteReplicationActor` never sends config JSON over the replication hop — it sends a deployment id +plus fetch coordinates, and the standby HTTP-fetches from central itself +(`SiteReplicationActorTests` #2, #5). That design exists for exactly one reason: the config blob +exceeds Akka's 128 KB frame limit (`docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md`). + +Under CDC the row itself replicates over the LocalDb gRPC stream, which has no such limit. The +standby therefore never fetches at all. + +This also disposes of the version guard. `StoreDeployedConfigIfNewerAsync` +(`SiteStorageService.cs:301-336`) guards its upsert with: + +```sql +WHERE excluded.deployed_at > deployed_configurations.deployed_at +``` + +That predicate protects against **a stale fetch landing after a newer one** — a race that only +exists because the standby was independently fetching. With the fetch gone, the only writer is the +active node's deploy path, and HLC ordering on a single writer is monotonic. The guard becomes +dead code. + +> **Do not** try to reproduce the `deployed_at` guard on top of LWW. They order by different +> clocks (central's deploy time vs. the writing node's HLC) and mixing them produces a +> non-convergent store. + +### D2. `ReplaceAllAsync` is deleted, and the N1 directional guard becomes unnecessary + +`StoreAndForwardStorage.ReplaceAllAsync` (`StoreAndForwardStorage.cs:284-306`) is an unqualified +`DELETE FROM sf_messages` followed by a full re-insert. Its own doc warns "Never call on an active +node — it discards every in-flight row." `SfBufferResyncPredicateTests` exists to enforce that only +a standby ever applies it — the N1 Critical regression test. + +The LocalDb library's snapshot resync **merges per row under LWW and never deletes**: + +- `SnapshotApplier.OnBeginAsync` (`SnapshotStreamer.cs:163-170`) resets counters only. There is no + `DELETE` or `TRUNCATE` anywhere in the class. +- `OnBatchAsync` (`:172-186`) wraps each snapshot row as an ordinary oplog entry and pushes it + through the same `LwwApplier` as a delta. +- `LwwApplier.cs:69-78` discards the incoming row when the local row's HLC is higher. + +So a node with newer local rows keeps them. The failure the directional guard prevents — a stale +peer wiping a live buffer — is structurally impossible, not merely guarded against. **Port +`SfBufferResyncPredicateTests` as a convergence assertion (both nodes end with the union, newest +per id wins), not as a directional-authority assertion.** + +Consequence to state plainly: the standby is no longer guaranteed byte-identical to the active +node's buffer. It is guaranteed *convergent*. That is a real semantic change and Task 2 records it. + +### D3. The SMTP purge is re-homed before anything is deleted + +`HandleApplyArtifacts` calls `PurgeCentralOnlyNotificationConfigAsync` +(`SiteStorageService.cs:811-821`), which deletes `notification_lists` and `smtp_configurations` — +including plaintext SMTP passwords left by pre-fix deployments. This is a **security cleanup riding +the replication path**, not a table sync. CDC will not reproduce it. + +Under CDC, if the active node purges, the deletes replicate as ordinary tombstones. So the fix is +to ensure the *active* node's deploy path performs the purge. Task 12 does this **before** any +deletion, so the purge never stops happening even for one commit. + +### D4. `native_alarm_state` volume is measured, not assumed + +`scadabridge.db` is not only config. `native_alarm_state` mirrors live A&C conditions from +`NativeAlarmActor.cs:504` via batched upserts, and is by a wide margin the highest-volume table in +either database. `sf_messages` worst case is ~50 row-writes/sec (`DefaultMaxRetries = 50`, 10 s +sweep, 4-way target parallelism). Neither number can be turned into an oplog cap from first +principles. + +**Task 1 is a rig soak that measures both.** Its output sets `MaxOplogRows` / `MaxOplogAge` in +Task 19. If the soak shows the shared oplog cannot absorb alarm-storm churn, the escape hatch named +in the design doc (line 140) is keyed instances — that is a library-level effort and would suspend +this plan, so **Task 1 gates everything after Task 2.** + +### D5. Cutover forecloses rolling site upgrades + +`SiteReplicationActor` retains a legacy monolithic `SfBufferSnapshot` handler specifically for +rolling upgrades. With no dual-mechanism period, a site pair cannot be upgraded one node at a time — +one node would speak a protocol the other no longer implements. **Both nodes of a site must be +stopped and started together.** Task 21 puts this in the deployment runbook. + +--- + +## Execution waves + +| Wave | Tasks | Note | +|---|---|---| +| 0 | 1, 2 | Soak + decision record. **Task 1 gates the rest.** | +| 1 | 3, 4 | Schema extraction — independent files, dispatch in parallel | +| 2 | 5, 6 | Store rewiring — independent files, dispatch in parallel | +| 3 | 7, 8, 9 | Setup + migrators | +| 4 | 10, 11 | Port test intents as specs — **before** any deletion | +| 5 | 12, 13 | Re-home the purge, delete notify-and-fetch | +| 6 | 14, 15, 16, 17 | The cutover, serial | +| 7 | 18, 19 | Convergence suite + rig config | +| 8 | 20, 21 | Live gate + docs | + +--- + +### Task 1: Rig soak — measure oplog growth under real write rates + +**Classification:** high-risk +**Estimated implement time:** ~40 min wall-clock (mostly waiting) +**Parallelizable with:** none — this gates the plan + +This is the gate item from `2026-07-19-localdb-phase2-gate.md` §5. It is measurement, not code. + +**Files:** +- Create: `docs/plans/2026-07-19-localdb-phase2-soak.md` + +**Step 1: Bring up the rig with Phase 1 replication enabled** + +```bash +cd ~/Desktop/ScadaBridge +git checkout feat/localdb-phase1 +bash docker/deploy.sh +``` + +Site-a's pair replicates; site-b/site-c deliberately do not (the default-OFF pin). Confirm: + +```bash +docker exec scadabridge-site-a-a curl -s localhost:8080/metrics | grep '^localdb_' +``` + +Expected: `localdb_*` series present. If absent, the meter allowlist regressed — see +`SiteServiceRegistration.ObservedMeters`. + +**Step 2: Capture a baseline** + +```bash +docker exec scadabridge-site-a-a sqlite3 /app/data/site-localdb.db \ + "SELECT COUNT(*) FROM __localdb_oplog;" +``` + +Record the count and the wall-clock time. + +**Step 3: Drive sustained load for 30 minutes** + +Two generators, run concurrently: + +- **Store-and-forward churn.** Point a store-and-forward target at an address that refuses + connections, then enqueue messages. Each failed attempt costs one `UPDATE`. With the default + 10 s sweep this is ~1 write/message/10 s per lane. +- **Alarm churn.** Drive A&C conditions on a deployed instance so `native_alarm_state` upserts + fire. This is the number that actually matters — it is unbounded by design. + +**Step 4: Sample oplog depth every 5 minutes** + +```bash +for i in $(seq 1 6); do + echo "=== t+$((i*5))m ===" + docker exec scadabridge-site-a-a sqlite3 /app/data/site-localdb.db \ + "SELECT COUNT(*) FROM __localdb_oplog;" + docker exec scadabridge-site-a-a curl -s localhost:8080/metrics \ + | grep -E '^localdb_(oplog_backlog|replication_dead_letters|sync_connected)' + sleep 300 +done +``` + +**Step 5: Record findings** + +Write `docs/plans/2026-07-19-localdb-phase2-soak.md` with: +- rows/sec observed for `sf_messages` and for `native_alarm_state`, separately +- peak `localdb_oplog_backlog` +- whether backlog drained between bursts or grew monotonically +- dead-letter count (must be 0) +- **the recommended `MaxOplogRows` and `MaxOplogAge`**, with the arithmetic + +**Step 6: The decision gate** + +If backlog grew monotonically and never drained, the shared oplog cannot absorb this write profile. +**Stop. Do not proceed to Task 3.** Report to the user: the design doc's keyed-instances escape +hatch (design doc line 140) is required first, and that is a library effort in `scadaproj`. + +**Step 7: Commit** + +```bash +git checkout -b feat/localdb-phase2 +git add docs/plans/2026-07-19-localdb-phase2-soak.md +git commit -m "docs(localdb): phase 2 rig soak findings — oplog sizing evidence" +``` + +--- + +### Task 2: Decision record + +**Classification:** trivial +**Estimated implement time:** ~3 min +**Parallelizable with:** none (needs Task 1's numbers) + +**Files:** +- Modify: `docs/plans/2026-07-19-localdb-phase2-gate.md` + +Update the gate's status header from `NOT STARTED` to point at this plan, and answer each §5 +question inline with a one-paragraph resolution referencing D1–D5 above plus Task 1's measured +numbers. Do not delete the questions — the reasoning is the value. + +**Commit:** `docs(localdb): close the phase 2 gate — questions answered, plan written` + +--- + +### Task 3: Extract `StoreAndForwardSchema.Apply` + +**Classification:** small +**Estimated implement time:** ~4 min +**Parallelizable with:** Task 4 + +Mirrors `OperationTrackingSchema` exactly. Read that file first — it is the reference. + +**Files:** +- Create: `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardSchema.cs` +- Modify: `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardStorage.cs:32-117` +- Test: `tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardSchemaTests.cs` + +**Step 1: Write the failing test** + +```csharp +[Fact] +public void Apply_IsIdempotent_AndCreatesEveryColumn() +{ + var path = Path.Combine(Path.GetTempPath(), $"sf-schema-{Guid.NewGuid():N}.db"); + try + { + using var connection = new SqliteConnection($"Data Source={path}"); + connection.Open(); + + StoreAndForwardSchema.Apply(connection); + StoreAndForwardSchema.Apply(connection); // second run must not throw + + using var cmd = connection.CreateCommand(); + cmd.CommandText = "SELECT name FROM pragma_table_info('sf_messages')"; + var columns = new List(); + using var reader = cmd.ExecuteReader(); + while (reader.Read()) columns.Add(reader.GetString(0)); + + // 12 original + 4 additive + Assert.Equal(16, columns.Count); + Assert.Contains("last_attempt_at_ms", columns); + Assert.Contains("parent_execution_id", columns); + } + finally + { + SqliteConnection.ClearAllPools(); + File.Delete(path); + } +} +``` + +**Step 2: Run it — expect FAIL** (`StoreAndForwardSchema` does not exist) + +```bash +dotnet test tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests.csproj --filter "FullyQualifiedName~StoreAndForwardSchemaTests" +``` + +**Step 3: Create the schema class** + +Move, verbatim: the DDL literal from `StoreAndForwardStorage.cs:50-68`, the four +`AddColumnIfMissingAsync` calls (`:79-92`) and their helper (`:125-142`), the `last_attempt_at_ms` +backfill (`:98-105`), and the due-index (`:109-114`). + +Convert every `*Async` to sync (`ExecuteNonQuery`, `ExecuteScalar`) — `OnReady` is a synchronous +`Action`. Depend only on `Microsoft.Data.Sqlite`, never on the LocalDb library. + +Carry this comment onto the class, because it is the whole reason the file exists: + +```csharp +/// +/// Deliberately depends only on Microsoft.Data.Sqlite, not on the LocalDb library. +/// The Host applies this DDL to a LocalDb-managed connection before +/// RegisterReplicated installs the capture triggers; nothing about the schema +/// itself is LocalDb-specific, and the store still calls it so a directly-constructed +/// store (tests, tooling) remains self-sufficient. +/// +``` + +**Step 4: Reduce `StoreAndForwardStorage.InitializeAsync` to a call** + +```csharp +public async Task InitializeAsync() +{ + EnsureDatabaseDirectoryExists(); + await using var connection = await OpenConnectionAsync(); + StoreAndForwardSchema.Apply(connection); +} +``` + +**Step 5: Run the S&F suite — expect PASS** + +```bash +dotnet test tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests.csproj +``` + +**Step 6: Commit** + +```bash +git add -A && git commit -m "refactor(sf): extract StoreAndForwardSchema.Apply from the storage class" +``` + +--- + +### Task 4: Extract `SiteStorageSchema.Apply` + +**Classification:** small +**Estimated implement time:** ~5 min +**Parallelizable with:** Task 3 + +**Files:** +- Create: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageSchema.cs` +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs:58-196` +- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/SiteStorageSchemaTests.cs` + +Same shape as Task 3. Move the DDL literal (`:79-158`, all 9 tables), `MigrateSchemaAsync` +(`:167-181`) and `TryAddColumnAsync` (`:183-196`), converting to sync. + +**Do not move** the `PRAGMA journal_mode=WAL` at `:68-76`. LocalDb owns the connection's pragmas. + +**Test assertion:** all 9 tables exist after `Apply`, and a second `Apply` does not throw. Name +every table explicitly — a loop that asserts "9 tables" would pass if one were renamed: + +```csharp +foreach (var table in new[] + { + "deployed_configurations", "static_attribute_overrides", "shared_scripts", + "external_systems", "database_connections", "notification_lists", + "data_connection_definitions", "smtp_configurations", "native_alarm_state", + }) +{ + Assert.True(TableExists(connection, table), $"missing table: {table}"); +} +``` + +**Commit:** `refactor(site): extract SiteStorageSchema.Apply from SiteStorageService` + +--- + +### Task 5: Rewire `StoreAndForwardStorage` onto `ILocalDb` + +**Classification:** high-risk +**Estimated implement time:** ~5 min +**Parallelizable with:** Task 6 + +Follow `OperationTrackingStore` (Phase 1, commit `0b5e9b44`) as the reference — it solved this +exact problem. + +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardStorage.cs:22-26,177-185` +- Modify: `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/ServiceCollectionExtensions.cs:18-25` +- Test: `tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/` (fixtures) + +**Step 1: Replace the constructor** + +```csharp +public StoreAndForwardStorage(ILocalDb localDb, ILogger logger) +{ + ArgumentNullException.ThrowIfNull(localDb); + ArgumentNullException.ThrowIfNull(logger); + _localDb = localDb; + _logger = logger; +} +``` + +**Step 2: Replace `OpenConnectionAsync`** + +```csharp +// CreateConnection returns an ALREADY-OPEN, pragma-configured connection with the +// zb_hlc_next() UDF registered. Calling OpenAsync on it throws, and a raw +// SqliteConnection would lack the UDF, making every capture trigger fail closed. +private SqliteConnection OpenConnection() => _localDb.CreateConnection(); +``` + +Update all 18 call sites (`:36, 238, 258, 286, 321, 378, 412, 449, 482, 503, 548, 572, 589, 619, +638, 659, 677, 694`) from `await using var connection = await OpenConnectionAsync();` to +`await using var connection = OpenConnection();`. + +**Step 3: Delete `EnsureDatabaseDirectoryExists` (`:152-168`)** — LocalDb owns the file. + +**Step 4: Update the DI registration** + +```csharp +services.AddSingleton(sp => new StoreAndForwardStorage( + sp.GetRequiredService(), + sp.GetRequiredService>())); +``` + +**Step 5: Fix the test fixtures** + +Every test constructing `StoreAndForwardStorage` with a connection string now needs an `ILocalDb`. +Phase 1 created `tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/TestLocalDb.cs` for exactly +this — read it and add an equivalent (or reference it) in the S&F test project. It must use a real +temp **file** (never `:memory:` — capture triggers need a durable file) and call +`SqliteConnection.ClearAllPools()` before deleting, including the `-wal` and `-shm` sidecars. + +**Step 6: Full S&F suite — expect PASS** + +```bash +dotnet test tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests.csproj +``` + +**Commit:** `refactor(sf): StoreAndForwardStorage takes ILocalDb instead of a connection string` + +--- + +### Task 6: Rewire `SiteStorageService` onto `ILocalDb` + +**Classification:** high-risk +**Estimated implement time:** ~5 min +**Parallelizable with:** Task 5 + +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs:13-51` +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/ServiceCollectionExtensions.cs:36-42` +- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs:70-71` + +Same shape as Task 5, but note the differences: + +- There are **22** inline `new SqliteConnection(_connectionString)` + `OpenAsync()` pairs, not a + helper. Add a `private SqliteConnection OpenConnection() => _localDb.CreateConnection();` and + convert all of them. +- `public SqliteConnection CreateConnection()` at `:51` is a public escape hatch used by site + repositories. Keep the member; change the body to `=> _localDb.CreateConnection();`. +- The busy-timeout floor (`BusyTimeoutFloorSeconds`, `:36`) and its `SqliteConnectionStringBuilder` + normalization are **deleted** — LocalDb configures pragmas on every connection it hands out. +- `AddSiteRuntime(string siteDbConnectionString)` loses its parameter. Keep the no-arg overload at + `:23-28` (it already exists) and delete the string one, updating + `SiteServiceRegistration.cs:70-71` to `services.AddSiteRuntime();`. + +**Watch for:** transactions at `:348` (`RemoveDeployedConfigAsync`) and `:540` +(`UpsertNativeAlarmsAsync`) must keep using `connection.BeginTransaction()` on the LocalDb +connection, not `ILocalDb.BeginTransactionAsync()` — mixing the two would put the cascade's three +deletes in different transactions. + +**Step: Run both affected suites** + +```bash +dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.csproj +dotnet test tests/ZB.MOM.WW.ScadaBridge.Host.Tests/ZB.MOM.WW.ScadaBridge.Host.Tests.csproj +``` + +Host.Tests will surface DI fixtures that now need `LocalDb:Path` — Phase 1 hit the identical +failure (tasks.json Task 4 note). Add the config key to each failing fixture. + +**Commit:** `refactor(site): SiteStorageService takes ILocalDb instead of a connection string` + +--- + +### Task 7: Extend `SiteLocalDbSetup` with the new DDL (not yet registered) + +**Classification:** standard +**Estimated implement time:** ~3 min +**Parallelizable with:** none + +Deliberately applies DDL **without** calling `RegisterReplicated`. The tables live in the +consolidated file but are not yet captured, so the bespoke replicator keeps working unchanged and +the tree stays green. Task 14 flips them on and deletes the bespoke path in one commit. + +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbSetup.cs:43-57` +- Test: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbWiringTests.cs` + +**Step 1: Extend `OnReady`** + +```csharp +using (var connection = db.CreateConnection()) +{ + OperationTrackingSchema.Apply(connection); + SiteEventLogSchema.Apply(connection); + SiteStorageSchema.Apply(connection); + StoreAndForwardSchema.Apply(connection); +} + +db.RegisterReplicated("OperationTracking"); +db.RegisterReplicated("site_events"); + +// Phase 2 tables are created here but NOT yet registered — the bespoke replicator +// still owns them until the Task 14 cutover. Registering them now would double- +// replicate every row (harmless, both paths upsert, but it would mask a defect in +// either one). + +SiteLocalDbLegacyMigrator.Migrate(db, config); +``` + +**Step 2: Pin it** + +Add a test asserting all 4 Phase-1-and-2 table sets exist after `OnReady`, and that +`db.ReplicatedTables` contains **exactly** `OperationTracking` and `site_events` — the "not yet" +is the assertion that matters. + +**Commit:** `feat(localdb): create config + sf_messages tables in the consolidated DB` + +--- + +### Task 8: Extend the legacy migrator for `sf_messages` + +**Classification:** high-risk +**Estimated implement time:** ~5 min +**Parallelizable with:** Task 9 + +Read `SiteLocalDbLegacyMigrator.cs` in full first. `MigrateTracking` (`:116-161`) is the closest +reference — like `sf_messages`, it has a native TEXT PK and needs no id synthesis. + +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbLegacyMigrator.cs:57-66` +- Test: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbLegacyMigratorTests.cs` + +**Step 1: Write the failing tests** + +Three, minimum: + +```csharp +[Fact] public void SfMessages_AreCopiedFromTheLegacyFile() { } +[Fact] public void SfMessages_Migration_IsIdempotent_WhenRerunAfterACrashBeforeRename() { } +[Fact] public void MigratedSfMessages_EnterTheOplog_SoTheyActuallyReplicate() { } +``` + +The third is the one that catches the ordering defect. Phase 1's equivalent asserts on +`__localdb_oplog` directly — copy that assertion. **The oplog table is `__localdb_oplog`** +(`LocalDbSchema.cs:19`), not `zb_oplog`. + +**Step 2: Add `ResolveStoreAndForwardPath` + `MigrateStoreAndForward`** + +```csharp +private const string DefaultStoreAndForwardPath = "./data/store-and-forward.db"; + +internal static string ResolveStoreAndForwardPath(IConfiguration config) => + config["ScadaBridge:StoreAndForward:SqliteDbPath"] ?? DefaultStoreAndForwardPath; +``` + +Migrate all 16 columns with `INSERT OR IGNORE`. The PK is native TEXT — **no id synthesis**, unlike +`MigrateEvents`. + +**Step 3: Wire into `Migrate`** — add the call alongside the existing two. + +**Commit:** `feat(localdb): migrate legacy store-and-forward.db into the consolidated DB` + +--- + +### Task 9: Extend the legacy migrator for the 9 config tables + +**Classification:** high-risk +**Estimated implement time:** ~5 min +**Parallelizable with:** Task 8 + +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbLegacyMigrator.cs` +- Test: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbLegacyMigratorTests.cs` + +All 9 tables migrate from one legacy file (`ScadaBridge:Database:SiteDbPath`), so this is one +`MigrateSiteStorage` that opens the legacy DB once and copies table by table inside a single +transaction, then renames once. + +**Two tables are deliberately skipped:** `notification_lists` and `smtp_configurations`. They are +purged on every deploy and are permanently empty by design (site write paths were removed +2026-07-10). Migrating them would resurrect plaintext SMTP passwords from a pre-fix legacy file into +a **replicated** table. Add this as a comment and as a test: + +```csharp +[Fact] +public void Migration_DoesNotCopyNotificationOrSmtpRows_EvenWhenTheLegacyFileHasThem() +{ + // These carry plaintext SMTP passwords in pre-2026-07-10 files. They are purged + // on every deploy and must never enter a replicated table. +} +``` + +**Step: Verify all 9 tables' columns match** between `SiteStorageSchema` and the migrator's +`INSERT` lists. A column-count mismatch here fails at runtime on a real deployment and nowhere else. + +**Commit:** `feat(localdb): migrate legacy scadabridge.db config tables into the consolidated DB` + +--- + +### Task 10: Port the S&F replication test intents as CDC specs + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** Task 11 + +**Specifications first, deletion second.** These tests encode behaviour the replacement must still +satisfy. The gate document named two files; the real specification is five. + +**Files:** +- Read (as spec): `tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ReplicationServiceTests.cs` +- Read (as spec): `tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardReplicationTests.cs` +- Create: `tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbStoreAndForwardConvergenceTests.cs` + +Port these **intents** (not the mechanics) as two-node convergence assertions, using Phase 1's +`LocalDbSitePairConvergenceTests.cs` as the harness reference: + +| Intent | Assertion under CDC | +|---|---| +| Add materialises on the peer | enqueue on A → row present on B | +| Remove deletes on the peer | deliver on A → row absent on B | +| Park sets status Parked on the peer | park on A → B's row has status Parked | +| Requeue resets status + RetryCount | requeue on A → B shows Pending, RetryCount 0 | +| Add-then-Remove never converges to present | **the portable form of the strict-ordering test** | +| Apply Add twice is idempotent, newest wins | LWW gives this | +| Park when Add was lost still materialises | upsert semantics | + +**Explicitly do not port:** `ReplicationServiceTests` #10 (200 interleaved ops dispatched +synchronously in strict issue order, 400 observed). That asserts the *mechanism* — inline +fire-and-forget dispatch — not the outcome. CDC capture is asynchronous and batched by construction. +The portable intent is row 5 above. Record this in the test file as a comment so a future reader +does not think it was dropped by accident. + +**Commit:** `test(localdb): port store-and-forward replication intents as CDC convergence specs` + +--- + +### Task 11: Port the resync + directional-authority tests + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** Task 10 + +**Files:** +- Read (as spec): `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/SiteReplicationActorTests.cs` +- Read (as spec): `tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/SfBufferResyncPredicateTests.cs` +- Create: `tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbConfigConvergenceTests.cs` + +**`SfBufferResyncPredicateTests` is the most valuable test in the set** — the N1 Critical +regression. Per D2 it must be re-expressed, not ported literally: + +```csharp +[Fact] +public async Task ANodeWithNewerLocalRows_KeepsThem_WhenAPeerSnapshotArrives() +{ + // N1 Critical (was: SfBufferResyncPredicateTests). The bespoke replicator needed a + // directional guard because ReplaceAllAsync was a destructive DELETE-then-INSERT, + // so a wrong-direction resync wiped a live buffer. + // + // LocalDb's snapshot resync merges per row under LWW and never deletes + // (SnapshotApplier has no DELETE; LwwApplier.cs:69-78 discards an incoming row + // whose HLC is lower than the local row's). The failure mode is structurally + // impossible rather than guarded against — so this asserts convergence, NOT + // directional authority. There is no active/standby asymmetry left to enforce. +} +``` + +**Also do not port:** actor tests #1–#5 (notify-and-fetch config fetch/retry/supersede). Per D1 the +fetch path is deleted, so these describe code that will not exist. Replace with a single assertion +that a config deploy on A converges to B **without** B making any HTTP call — assert on a fetcher +test double that records zero invocations. That is the positive proof that notify-and-fetch is gone. + +**Commit:** `test(localdb): port resync + config replication intents as CDC convergence specs` + +--- + +### Task 12: Re-home the SMTP purge off the replication path + +**Classification:** high-risk +**Estimated implement time:** ~4 min +**Parallelizable with:** none + +Per D3. **This must land before any deletion** so the purge never lapses, even for one commit. + +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs` (artifact-apply path) +- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs` + +**Step 1: Write the failing test** + +```csharp +[Fact] +public async Task ApplyingArtifacts_PurgesCentralOnlyNotificationConfig_OnTheActiveNode() +{ + // Security cleanup: notification_lists and smtp_configurations can hold plaintext + // SMTP passwords from pre-2026-07-10 deployments. This ran on the standby via + // SiteReplicationActor.HandleApplyArtifacts; with that actor deleted the ACTIVE + // node must do it, and CDC replicates the deletes as ordinary tombstones. +} +``` + +**Step 2: Call `PurgeCentralOnlyNotificationConfigAsync` from the active node's artifact-apply path** + +The existing call site is `SiteReplicationActor.HandleApplyArtifacts`. Find where +`DeploymentManagerActor` applies artifacts locally (the same path that reaches +`_replicationActor?.Tell(new ReplicateArtifacts(command))` at `:1952`) and call it there. + +**Step 3: Verify the deletes replicate** — extend the Task 11 convergence suite with a case where +A purges a seeded row and B's copy disappears. + +**Commit:** `fix(site): purge central-only notification config on the active node` + +--- + +### Task 13: Delete the notify-and-fetch config path + +**Classification:** high-risk +**Estimated implement time:** ~4 min +**Parallelizable with:** none + +Per D1. + +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs:301-336` +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptions.cs:66-68` +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptionsValidator.cs:56-58` + +**Step 1: Delete `StoreDeployedConfigIfNewerAsync`** — its only caller is +`SiteReplicationActor.HandleApplyConfigDeploy`, deleted in Task 15. The unguarded +`StoreDeployedConfigAsync` (`:241-271`) is the active node's path and stays. + +**Step 2: Check `ConfigFetchRetryCount` before removing it** + +`IDeploymentConfigFetcher` is used by **both** the standby replication path *and* the active +singleton's `RefreshDeploymentCommand` path. Only the first is being deleted. + +```bash +grep -rn "ConfigFetchRetryCount" src/ tests/ +``` + +If the active path reads it, **keep the option and its validator rule** and note that in the commit +message. If nothing outside the deleted actor reads it, remove both. + +**Commit:** `refactor(site): delete notify-and-fetch config replication — CDC ships the row` + +--- + +### Task 14: Register the Phase 2 tables and delete `ReplicationService` + +**Classification:** high-risk +**Estimated implement time:** ~5 min +**Parallelizable with:** none + +**The cutover.** Registration and bespoke-path deletion land in one commit so the two mechanisms +never both run. + +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbSetup.cs` +- Delete: `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/ReplicationService.cs` +- Modify: `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardService.cs:39,243,654,805,836,872,1120,1146` +- Modify: `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/ServiceCollectionExtensions.cs:63-68` +- Modify: `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardStorage.cs:284-306` +- Delete: `tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ReplicationServiceTests.cs` +- Delete: `tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardReplicationTests.cs` +- Delete: `tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ReplicationWireSerializationPinTests.cs` + +**Step 1: Register all 11 tables** in `SiteLocalDbSetup.OnReady`, replacing the Task 7 comment. +Skip `notification_lists` and `smtp_configurations`? **No — register them.** They must replicate +their *deletes* (Task 12). Register all 9 config tables plus `sf_messages`. + +**Step 2: Delete `ReplicationService`** and remove the ctor parameter + all 6 emission call sites +from `StoreAndForwardService`. + +**Step 3: Delete `ReplaceAllAsync`** (`:284-306`). Per D2 the library's resync merges; a destructive +delete-all must not survive into a replicated table, where CDC would capture the mass-delete and +ship it to the peer. + +**Step 4: Check `GetAllMessagesAsync`'s `Truncated` flag** — it exists only for the resync path. If +nothing else reads it, simplify the return type. + +**Step 5: `CompositionRootTests.cs:474`** asserts `ReplicationService` is a registered site +singleton. Delete that assertion. + +**Step 6: Full S&F + Host suites — expect PASS** + +**Commit:** `feat(localdb)!: replicate sf_messages via CDC, delete ReplicationService` + +--- + +### Task 15: Delete `SiteReplicationActor` and its messages + +**Classification:** high-risk +**Estimated implement time:** ~4 min +**Parallelizable with:** none + +**Files:** +- Delete: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/SiteReplicationActor.cs` +- Delete: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Messages/ReplicationMessages.cs` +- Delete: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/SiteReplicationActorTests.cs` +- Delete: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/ResyncWireSerializationPinTests.cs` +- Delete: `tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/SfBufferResyncPredicateTests.cs` + +The whole `ReplicationMessages.cs` file becomes dead: `ReplicateConfigDeploy/Remove/SetEnabled`, +`ReplicateArtifacts`, `ReplicateStoreAndForward`, `ApplyConfigDeploy/Remove/SetEnabled`, +`ApplyArtifacts`, `ApplyStoreAndForward`, plus the resync records `RequestSfBufferResync`, +`SfBufferSnapshot`, `SfBufferSnapshotChunk`, `SfBufferResyncAck`. + +**Do not delete** `ActiveNodeEvaluator` (`Communication/ClusterState/ActiveNodeEvaluator.cs`) — the +S&F **delivery gate** still needs it. Only its doc comment at `:14` mentions replication; update +the comment, keep the type. + +Confirm the deletions are complete before committing: + +```bash +grep -rn "SiteReplicationActor\|ReplicateStoreAndForward\|SfBufferSnapshot" src/ tests/ +``` + +Expected: no matches. + +**Commit:** `feat(localdb)!: delete SiteReplicationActor — CDC replaces hand-shipped ops` + +--- + +### Task 16: Clean up `DeploymentManagerActor` and `AkkaHostedService` + +**Classification:** standard +**Estimated implement time:** ~4 min +**Parallelizable with:** none + +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs:55,161-190,794,970,1004,1102,1952` +- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs:770-796,806-812,1056` + +**Step 1:** Remove the `IActorRef? replicationActor = null` ctor parameter (`:184`), the +`_replicationActor` field (`:55`), and all 5 `_replicationActor?.Tell(...)` sites. + +The parameter is optional and positional — removing it **silently shifts every argument after it** +(`healthCollector`, `serviceProvider`, `loggerFactory`, `configFetcher`, ...). Update the +`Props.Create` at `AkkaHostedService.cs:806-812` and every test constructing this actor. Compile +errors will not catch a positional shift between two same-typed optional parameters — check each +call site by hand. + +**Step 2:** Delete `replicationService` / `replicationLogger` resolution (`:770-772`), the actor +creation (`:785-789`), the handler wiring (`:792-796`), and the actor-inventory comment at `:1056`. + +**Step 3:** `activeNodeCheck` (`:781-783`) is still used by `SiteCommunicationActor` (`:816-821`). +Keep it. + +**Step 4:** Full Host + SiteRuntime + IntegrationTests suites. + +**Commit:** `refactor(site): drop the replication actor from the deployment + host wiring` + +--- + +### Task 17: Config-key cleanup + +**Classification:** standard +**Estimated implement time:** ~4 min +**Parallelizable with:** none + +**Files:** +- Modify: `src/ZB.MOM.WW.ScadaBridge.Host/StartupValidator.cs:113-115` +- Modify: `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardOptions.cs:12` +- Modify: `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardOptionsValidator.cs:20` +- Modify: 9 × `appsettings.Site.json` (Host template + 6 `docker/site-*` + 2 `docker-env2/site-x-*`) + +| Key | Action | +|---|---| +| `ScadaBridge:Database:SiteDbPath` | → migration-only. **Relax the `Require` at `StartupValidator.cs:113`** or every site config keeps it mandatory forever. | +| `ScadaBridge:StoreAndForward:SqliteDbPath` | → migration-only. Relax the non-empty rule at `StoreAndForwardOptionsValidator.cs:20`. | +| `ScadaBridge:StoreAndForward:ReplicationEnabled` | → **fully dead.** Delete the property and all 9 config entries. | + +`ParkedOperationRelayTests.cs:39` and `ParkedMessageHandlerActorTests.cs:34` set +`ReplicationEnabled = false` — they only need the flag to exist, so remove those lines. +`StoreAndForwardOptionsTests.cs:14,26,33` assert its default; delete those assertions. + +**Leave the path keys present in configs** with a comment — they are read by the legacy migrator and +removing them would strand un-migrated data on a node that has not yet started once. + +**Commit:** `chore(config): retire ReplicationEnabled, make legacy db paths migration-only` + +--- + +### Task 18: Two-node convergence suite for the Phase 2 tables + +**Classification:** high-risk +**Estimated implement time:** ~5 min +**Parallelizable with:** Task 19 + +Extend Phase 1's `LocalDbSitePairConvergenceTests.cs` harness (real loopback Kestrel h2c + the real +`LocalDbSyncAuthInterceptor`, initialized via `SiteLocalDbSetup.OnReady`). + +**Files:** +- Modify: `tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbSitePairConvergenceTests.cs` + +Scenarios, beyond Tasks 10–11: + +1. A config deploy on A converges to B, and B's `deployed_configurations` row matches byte for byte. +2. `RemoveDeployedConfigAsync`'s **3-table cascade** converges — `static_attribute_overrides` and + `native_alarm_state` orphans do not survive on B. There are no foreign keys + (`SiteStorageService` DDL has none), so this is three independent delete streams that LWW may + reorder. **This is the most likely real defect in the whole plan.** +3. A burst of `native_alarm_state` upserts converges and the oplog drains. +4. Node B restarts and catches up via snapshot resync without losing rows written on B while down. + +**Verify the suite is non-vacuous.** Phase 1 did this by mismatching the two API keys and confirming +all scenarios went red. Do the same and record the result in the commit message. A convergence suite +that cannot fail proves nothing. + +**Commit:** `test(localdb): two-node convergence for config tables + sf_messages` + +--- + +### Task 19: Rig configuration + +**Classification:** small +**Estimated implement time:** ~3 min +**Parallelizable with:** Task 18 + +**Files:** +- Modify: `docker/site-a-node-a/appsettings.Site.json`, `docker/site-a-node-b/appsettings.Site.json` + +Set `MaxOplogRows` and `MaxOplogAge` from **Task 1's measured numbers** — not from a guess. Keep +site-b/site-c unreplicated so default-OFF stays proven side by side on one rig. + +**Commit:** `chore(docker): size the site-a oplog caps from the phase 2 soak` + +--- + +### Task 20: Live gate on the docker rig + +**Classification:** high-risk +**Estimated implement time:** ~30 min wall-clock +**Parallelizable with:** none + +**Files:** +- Create: `docs/plans/2026-07-19-localdb-phase2-live-gate.md` + +```bash +cd ~/Desktop/ScadaBridge && bash docker/deploy.sh +``` + +Evidence required — each must be captured, not asserted: + +1. **Migration ran.** `store-and-forward.db.migrated` and `scadabridge.db.migrated` exist on both + site-a nodes; row counts in the consolidated DB match the legacy files. +2. **No SMTP/notification rows migrated** — `SELECT COUNT(*) FROM smtp_configurations` is 0. +3. **Config converges.** Deploy an instance to site-a; both nodes' `deployed_configurations` rows + are byte-identical with identical `__localdb_row_version` and originating node id. +4. **The standby made no HTTP fetch** — nothing in its log resembling a config fetch. This is the + positive proof that notify-and-fetch is gone. +5. **S&F converges.** Enqueue against a dead target; the parked row appears on the peer. +6. **Site failover.** Flip the site-a pair and confirm buffered messages survive and deliver + exactly once after the flip. `docker/failover-drill.sh` targets **CENTRAL** nodes and does not + exercise this — run the site flip directly. +7. **Cascade delete.** Remove a deployed config; confirm no orphan `static_attribute_overrides` or + `native_alarm_state` rows on the peer. +8. **Zero dead letters**, oplog drained, `localdb_*` scraped from `/metrics`. +9. **Both nodes stopped and started together** (D5) — confirm a clean rejoin. + +**If any check fails, stop and report.** Do not proceed to Task 21. + +**Commit:** `docs(localdb): phase 2 live gate evidence` + +--- + +### Task 21: Documentation truth pass + +**Classification:** standard +**Estimated implement time:** ~5 min +**Parallelizable with:** none + +**Files:** +- Modify: `CLAUDE.md` (ScadaBridge), `~/Desktop/scadaproj/CLAUDE.md` (LocalDb component row) +- Modify: `docs/requirements/Component-StoreAndForward.md:83` +- Modify: `docs/components/StoreAndForward.md`, `SiteRuntime.md`, `Host.md` +- Modify: `docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md` +- Modify: `docs/deployment/` runbook + +Specific corrections: + +- `Component-StoreAndForward.md:83` is a long **normative** paragraph specifying the whole chunked + resync protocol and the N5 race. Rewrite for CDC — state the new duplicate-delivery bound + explicitly rather than deleting the discussion. +- The frame-size known-issue is **resolved by** this change (D1). Mark it resolved and say why, + rather than leaving a live known-issue describing deleted code. +- **Add to the deployment runbook (D5):** a site pair must be stopped and started together. Rolling + one node at a time is no longer supported, because the legacy `SfBufferSnapshot` compatibility + handler is gone. +- Update the scadaproj LocalDb component row: Phase 2 complete, what replicates now, what + `SiteReplicationActor` used to do. + +**Commit:** `docs(localdb): phase 2 truth pass across both repos` + +--- + +## Definition of done + +- [ ] Solution builds with **0 warnings** (`TreatWarningsAsErrors` is on) +- [ ] Every suite green: Host, CentralUI, Commons, SiteRuntime, AuditLog, Communication, + StoreAndForward, HealthMonitoring, IntegrationTests, SiteEventLogging +- [ ] `grep -rn "SiteReplicationActor\|ReplicationService\|SfBufferSnapshot" src/` → no matches +- [ ] Live gate PASS with all 9 evidence items captured +- [ ] Convergence suite verified non-vacuous +- [ ] Both `CLAUDE.md` files updated diff --git a/docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json b/docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json new file mode 100644 index 00000000..41bd42b7 --- /dev/null +++ b/docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json @@ -0,0 +1,89 @@ +{ + "planPath": "docs/plans/2026-07-19-localdb-adoption-phase2.md", + "execution": { + "mode": "parallel-waves", + "implementerModel": "opus", + "isolation": "worktree", + "branch": "feat/localdb-phase2", + "baseBranch": "feat/localdb-phase1", + "note": "Phase 1's branch is NOT merged/pushed, so phase 2 branches from it. Dispatch every unblocked task concurrently per the wave table in the plan. Parallel implementers MUST use worktree isolation - concurrent git in one worktree races destructively." + }, + "scopeDecision": { + "date": "2026-07-19", + "by": "user", + "choice": "Full scope as designed", + "note": "User chose both surfaces (config tables + sf_messages) in one phase, deleting both bespoke mechanisms together, over the recommended split of S&F-first. The four open design questions are therefore resolved INSIDE this plan as D1-D5 rather than deferred." + }, + "decisions": [ + { + "id": "D1", + "subject": "Config moves to CDC; notify-and-fetch is DELETED, not preserved", + "evidence": "SiteReplicationActor sends id+fetch-coords only because the config blob exceeds Akka's 128KB frame (docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md). LocalDb sync is gRPC - no such limit. StoreDeployedConfigIfNewerAsync's guard (SiteStorageService.cs:325, `WHERE excluded.deployed_at > deployed_configurations.deployed_at`) protects only against a stale FETCH racing; with no fetch there is no race. Do NOT reproduce the guard on top of LWW - deployed_at and HLC are different clocks and mixing them is non-convergent." + }, + { + "id": "D2", + "subject": "ReplaceAllAsync deleted; the N1 directional guard becomes unnecessary", + "evidence": "LocalDb's snapshot resync MERGES per-row LWW and never deletes: SnapshotApplier.OnBeginAsync (SnapshotStreamer.cs:163-170) resets counters only, no DELETE anywhere in the class; OnBatchAsync (:172-186) routes snapshot rows through the same LwwApplier as deltas; LwwApplier.cs:69-78 discards an incoming row whose HLC is lower. So a stale peer cannot wipe a live buffer - structurally impossible, not guarded. SEMANTIC CHANGE: the standby is convergent, no longer byte-identical." + }, + { + "id": "D3", + "subject": "The SMTP purge is re-homed BEFORE anything is deleted", + "evidence": "SiteReplicationActor.HandleApplyArtifacts calls PurgeCentralOnlyNotificationConfigAsync (SiteStorageService.cs:811-821), deleting notification_lists + smtp_configurations incl. plaintext SMTP passwords from pre-2026-07-10 deployments. A security cleanup riding the replication path; CDC will not reproduce it. Task 12 moves it to the active node's deploy path before any deletion, so it never lapses." + }, + { + "id": "D4", + "subject": "native_alarm_state volume is MEASURED, not assumed", + "evidence": "scadabridge.db is not only config - native_alarm_state mirrors live A&C conditions (NativeAlarmActor.cs:504) and is the highest-volume table in either DB. sf_messages worst case ~50 row-writes/sec. Task 1 is a rig soak that measures both and sets MaxOplogRows/MaxOplogAge. If backlog grows monotonically, STOP: the design doc's keyed-instances escape hatch (line 140) is a scadaproj library effort that would suspend this plan." + }, + { + "id": "D5", + "subject": "Cutover forecloses rolling site upgrades", + "evidence": "SiteReplicationActor retains a legacy monolithic SfBufferSnapshot handler for rolling upgrades. With no dual-mechanism period, one node would speak a protocol the other no longer implements. Both nodes of a site must be stopped and started together. Task 21 puts this in the deployment runbook." + } + ], + "reconFindings": [ + "The gate doc named 2 test files as the specification; the real spec is 5 - it missed StoreAndForwardReplicationTests.cs (incl. the only Requeue coverage), ReplicationWireSerializationPinTests.cs, ResyncWireSerializationPinTests.cs, and SfBufferResyncPredicateTests.cs (the N1 Critical regression test).", + "sf_messages has NO version column today - ON CONFLICT(id) DO UPDATE has no comparison predicate (StoreAndForwardStorage.cs:331-345). 'Newest wins' is bare arrival order. LWW-by-HLC is an IMPROVEMENT here, not a regression.", + "No autoincrement-integer PKs exist anywhere in Phase 2 scope - all 9 config tables use natural TEXT or composite TEXT keys, sf_messages is TEXT. Phase 1's site_events GUID conversion has no Phase 2 analogue.", + "SiteStorageService has NO foreign keys. RemoveDeployedConfigAsync (:343-376) is a manual 3-statement cascade in one transaction. Under CDC these become three independent delete streams that LWW may reorder - the most likely real defect in the plan (Task 18 scenario 2).", + "DeploymentManagerActor's replicationActor is an OPTIONAL POSITIONAL parameter (:184). Removing it silently shifts every argument after it. Compilers will not catch a shift between two same-typed optional params - check every call site by hand (Task 16).", + "ActiveNodeEvaluator must NOT be deleted - the S&F delivery gate still uses it. Only its doc comment mentions replication.", + "ConfigFetchRetryCount may not be fully dead: IDeploymentConfigFetcher serves BOTH the standby replication path and the active singleton's RefreshDeploymentCommand path. Verify before removing (Task 13 step 2).", + "notification_lists and smtp_configurations are deliberately NOT migrated (Task 9) - migrating them would resurrect plaintext SMTP passwords from a pre-fix legacy file into a replicated table." + ], + "tasks": [ + {"id": 1, "subject": "Task 1: Rig soak - measure oplog growth under real write rates", "status": "pending", "classification": "high-risk", "note": "GATES EVERYTHING AFTER TASK 2. Stop the plan if oplog backlog grows monotonically."}, + {"id": 2, "subject": "Task 2: Decision record - close the phase 2 gate", "status": "pending", "classification": "trivial", "blockedBy": [1]}, + {"id": 3, "subject": "Task 3: Extract StoreAndForwardSchema.Apply", "status": "pending", "classification": "small", "blockedBy": [2]}, + {"id": 4, "subject": "Task 4: Extract SiteStorageSchema.Apply", "status": "pending", "classification": "small", "blockedBy": [2]}, + {"id": 5, "subject": "Task 5: Rewire StoreAndForwardStorage onto ILocalDb", "status": "pending", "classification": "high-risk", "blockedBy": [3]}, + {"id": 6, "subject": "Task 6: Rewire SiteStorageService onto ILocalDb", "status": "pending", "classification": "high-risk", "blockedBy": [4]}, + {"id": 7, "subject": "Task 7: Extend SiteLocalDbSetup with the new DDL (not yet registered)", "status": "pending", "classification": "standard", "blockedBy": [5, 6]}, + {"id": 8, "subject": "Task 8: Extend the legacy migrator for sf_messages", "status": "pending", "classification": "high-risk", "blockedBy": [7]}, + {"id": 9, "subject": "Task 9: Extend the legacy migrator for the 9 config tables", "status": "pending", "classification": "high-risk", "blockedBy": [7]}, + {"id": 10, "subject": "Task 10: Port the S&F replication test intents as CDC specs", "status": "pending", "classification": "standard", "blockedBy": [8]}, + {"id": 11, "subject": "Task 11: Port the resync + directional-authority tests", "status": "pending", "classification": "standard", "blockedBy": [9]}, + {"id": 12, "subject": "Task 12: Re-home the SMTP purge off the replication path", "status": "pending", "classification": "high-risk", "blockedBy": [10, 11], "note": "MUST land before any deletion so the security purge never lapses."}, + {"id": 13, "subject": "Task 13: Delete the notify-and-fetch config path", "status": "pending", "classification": "high-risk", "blockedBy": [12]}, + {"id": 14, "subject": "Task 14: Register the Phase 2 tables and delete ReplicationService", "status": "pending", "classification": "high-risk", "blockedBy": [13], "note": "THE CUTOVER. Registration + bespoke deletion in ONE commit so both mechanisms never run together."}, + {"id": 15, "subject": "Task 15: Delete SiteReplicationActor and its messages", "status": "pending", "classification": "high-risk", "blockedBy": [14]}, + {"id": 16, "subject": "Task 16: Clean up DeploymentManagerActor and AkkaHostedService", "status": "pending", "classification": "standard", "blockedBy": [15]}, + {"id": 17, "subject": "Task 17: Config-key cleanup", "status": "pending", "classification": "standard", "blockedBy": [16]}, + {"id": 18, "subject": "Task 18: Two-node convergence suite for the Phase 2 tables", "status": "pending", "classification": "high-risk", "blockedBy": [17]}, + {"id": 19, "subject": "Task 19: Rig configuration", "status": "pending", "classification": "small", "blockedBy": [17]}, + {"id": 20, "subject": "Task 20: Live gate on the docker rig", "status": "pending", "classification": "high-risk", "blockedBy": [18, 19]}, + {"id": 21, "subject": "Task 21: Documentation truth pass", "status": "pending", "classification": "standard", "blockedBy": [20]} + ], + "knownFlakes": [ + { + "test": "SiteRuntime.Tests InstanceActorChildAttributeRaceTests.ChildActors_AreSeededFromAnIsolatedCopy_NotTheLiveAttributesDictionary", + "note": "Intermittent ActorNotFoundException under full-suite load; passes in isolation. Pre-existing, carried over from Phase 1." + }, + { + "test": "AuditLog.Tests ParentExecutionIdCorrelationTests.InboundRoutedRun_AllRoutedRows_CarryInboundExecutionId_AsParentExecutionId", + "note": "Cold-MSSQL-fixture timing: ~91s and AwaitAssert-times-out cold, ~1s warm. Re-run before investigating." + } + ], + "lastUpdated": "2026-07-19", + "phase2Status": "PLANNED - not started. Task 1 (rig soak) gates the rest." +} From 25463d522f226bf51d4b7f89e30a5806326dc34f Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sun, 19 Jul 2026 17:27:11 -0400 Subject: [PATCH 22/55] =?UTF-8?q?docs(localdb):=20phase=202=20plan=20revie?= =?UTF-8?q?w=20pass=20=E2=80=94=20corrections=20from=20code=20verification?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three verification sweeps over SiteRuntime, StoreAndForward, and Host/rig/docs plus the LocalDb library source. Load-bearing corrections: D1 (the guarded write has a second surviving caller, SiteReconciliationActor), D3 (the active node already purges — Task 12 becomes a pin), D6 (new: the 4 MB gRPC cap vs row-count-only batching), Task 1 (rewritten method — the Phase 2 tables are not in the Phase 1 oplog), and Task 14 (do not register the notification/SMTP tables). Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts --- .../2026-07-19-localdb-adoption-phase2.md | 415 ++++++++++++------ ...7-19-localdb-adoption-phase2.md.tasks.json | 71 +-- 2 files changed, 332 insertions(+), 154 deletions(-) diff --git a/docs/plans/2026-07-19-localdb-adoption-phase2.md b/docs/plans/2026-07-19-localdb-adoption-phase2.md index a4315d6a..ea5fd918 100644 --- a/docs/plans/2026-07-19-localdb-adoption-phase2.md +++ b/docs/plans/2026-07-19-localdb-adoption-phase2.md @@ -16,6 +16,15 @@ hand-shipped ops; the library's per-row-LWW snapshot resync replaces the chunked **Branch:** `feat/localdb-phase2`, cut from `feat/localdb-phase1`. +> **Reviewed and corrected 2026-07-19** against the actual code (three verification sweeps over +> SiteRuntime, StoreAndForward, and Host/rig/docs, plus the LocalDb library source). Load-bearing +> corrections: D1 (the guarded write has a second surviving caller — `SiteReconciliationActor` — +> so it is NOT deleted), D3 (the active node already purges — Task 12 is now a pin, not a +> re-home), D6 (new: the 4 MB gRPC message cap × row-count-only batching), Task 1 (the Phase 2 +> tables aren't in the Phase 1 oplog — the soak measures legacy-DB write rates; port 8084; real +> metric names; no `sqlite3` in containers), and Task 14 (do NOT register the two +> notification/SMTP tables). Details inline at each site. + --- ## Read this before Task 1 @@ -52,17 +61,22 @@ exceeds Akka's 128 KB frame limit (`docs/known-issues/2026-06-26-deploy-config-e Under CDC the row itself replicates over the LocalDb gRPC stream, which has no such limit. The standby therefore never fetches at all. -This also disposes of the version guard. `StoreDeployedConfigIfNewerAsync` +The version guard, however, does **not** die with it. `StoreDeployedConfigIfNewerAsync` (`SiteStorageService.cs:301-336`) guards its upsert with: ```sql WHERE excluded.deployed_at > deployed_configurations.deployed_at ``` -That predicate protects against **a stale fetch landing after a newer one** — a race that only -exists because the standby was independently fetching. With the fetch gone, the only writer is the -active node's deploy path, and HLC ordering on a single writer is monotonic. The guard becomes -dead code. +That predicate protects against **a stale fetch landing after a newer one**, and the method has +**two** production callers, not one: `SiteReplicationActor.HandleApplyConfigDeploy` +(`SiteReplicationActor.cs:375` — deleted in Task 15) *and* `SiteReconciliationActor` +(`SiteReconciliationActor.cs:166`) — the per-node startup self-heal that reports local inventory +to central and HTTP-fetches gap items. The reconciliation actor **survives Phase 2**, and its +stale-fetch race (a reconcile fetch completing after a newer deploy landed) still exists. **The +method and its guard stay.** Under CDC the reconcile write simply becomes a second CDC writer of +`deployed_configurations` — benign: a gap fetch only fires when the node genuinely lacks the row, +the content is deterministic per (deployment id, revision), and the guard blocks local downgrades. > **Do not** try to reproduce the `deployed_at` guard on top of LWW. They order by different > clocks (central's deploy time vs. the writing node's HLC) and mixing them produces a @@ -75,13 +89,19 @@ dead code. node — it discards every in-flight row." `SfBufferResyncPredicateTests` exists to enforce that only a standby ever applies it — the N1 Critical regression test. -The LocalDb library's snapshot resync **merges per row under LWW and never deletes**: +The LocalDb library's snapshot resync **merges per row under LWW and never wipes**: - `SnapshotApplier.OnBeginAsync` (`SnapshotStreamer.cs:163-170`) resets counters only. There is no - `DELETE` or `TRUNCATE` anywhere in the class. + wholesale `DELETE FROM` or `TRUNCATE` anywhere in the class. - `OnBatchAsync` (`:172-186`) wraps each snapshot row as an ordinary oplog entry and pushes it through the same `LwwApplier` as a delta. - `LwwApplier.cs:69-78` discards the incoming row when the local row's HLC is higher. +- **Row-level deletes DO still replicate** — the delete trigger captures a tombstone + (`TriggerSqlGenerator`), `SnapshotStreamer` streams tombstone row-versions, and `LwwApplier` + applies an incoming tombstone as a real `DELETE`. Only the destructive whole-table replace is + gone. Caveat: tombstones are pruned after `LocalDb:Replication:TombstoneRetention` (default + 7 days) — a node offline longer than that can resurrect deleted rows on rejoin; Task 21 puts + this in the runbook. So a node with newer local rows keeps them. The failure the directional guard prevents — a stale peer wiping a live buffer — is structurally impossible, not merely guarded against. **Port @@ -91,16 +111,22 @@ per id wins), not as a directional-authority assertion.** Consequence to state plainly: the standby is no longer guaranteed byte-identical to the active node's buffer. It is guaranteed *convergent*. That is a real semantic change and Task 2 records it. -### D3. The SMTP purge is re-homed before anything is deleted +### D3. The SMTP purge already runs on the active node — pin it, don't move it -`HandleApplyArtifacts` calls `PurgeCentralOnlyNotificationConfigAsync` -(`SiteStorageService.cs:811-821`), which deletes `notification_lists` and `smtp_configurations` — -including plaintext SMTP passwords left by pre-fix deployments. This is a **security cleanup riding -the replication path**, not a table sync. CDC will not reproduce it. +`PurgeCentralOnlyNotificationConfigAsync` (`SiteStorageService.cs:811-821`) deletes +`notification_lists` and `smtp_configurations` — including plaintext SMTP passwords left by +pre-fix deployments. The original recon placed its only caller on the standby's replication path; +**that was wrong.** It has two callers: `DeploymentManagerActor.HandleDeployArtifacts` +(`DeploymentManagerActor.cs:1921`) — the **active** node's artifact-apply — and +`SiteReplicationActor.HandleApplyArtifacts` (`SiteReplicationActor.cs:456`), the standby copy +that dies with the actor. The purge therefore never lapses: the active-node call already exists +and stays. Task 12 shrinks to **pinning that call with a regression test** (it sits inside a +method Task 16 edits, and nothing today fails if the call is dropped). -Under CDC, if the active node purges, the deletes replicate as ordinary tombstones. So the fix is -to ensure the *active* node's deploy path performs the purge. Task 12 does this **before** any -deletion, so the purge never stops happening even for one commit. +Because no site code has written those two tables since 2026-07-10 (writers removed), the +migrator skips them (Task 9), and the active purge keeps them empty, they are **permanently empty +in the consolidated DB** — which is why Task 14 does **not** register them for replication (see +the rationale there). ### D4. `native_alarm_state` volume is measured, not assumed @@ -111,8 +137,10 @@ sweep, 4-way target parallelism). Neither number can be turned into an oplog cap principles. **Task 1 is a rig soak that measures both.** Its output sets `MaxOplogRows` / `MaxOplogAge` in -Task 19. If the soak shows the shared oplog cannot absorb alarm-storm churn, the escape hatch named -in the design doc (line 140) is keyed instances — that is a library-level effort and would suspend +Task 19. If the soak shows the shared oplog cannot absorb alarm-storm churn, the escape hatch is +keyed instances — named in the **adoption** design doc +(`~/Desktop/scadaproj/docs/plans/2026-07-19-scadabridge-localdb-design.md:139-141`, "Out of +scope"; not the 07-17 library design doc) — that is a library-level effort and would suspend this plan, so **Task 1 gates everything after Task 2.** ### D5. Cutover forecloses rolling site upgrades @@ -122,6 +150,23 @@ rolling upgrades. With no dual-mechanism period, a site pair cannot be upgraded one node would speak a protocol the other no longer implements. **Both nodes of a site must be stopped and started together.** Task 21 puts this in the deployment runbook. +### D6. The 4 MB gRPC message cap replaces the 128 KB Akka frame as the size ceiling + +The Akka frame limit that motivated notify-and-fetch is gone, but LocalDb sync has its own +ceiling the plan must respect: **neither side configures gRPC message sizes** — ScadaBridge's +`AddGrpc` (`Program.cs:519-521`) sets only the auth interceptor, and the library's initiator +channel is a bare `GrpcChannel.ForAddress` — so the **4 MB default receive limit** applies in +both directions. Batching is **row-count-only** (`MaxBatchSize`, default 500 — +`SyncSession.cs:227`, `SnapshotStreamer.cs:55`; there is no byte-aware chunking). +`deployed_configurations.config_json` is documented to exceed 128 KB per row; a few dozen such +rows in one batch is over 4 MB, which fails the stream and wedges replication on a poison batch. + +**Task 1 measures the real max/typical `config_json` size** on the rig's legacy DB; **Task 19 +sets `LocalDb:Replication:MaxBatchSize`** so `max-row-bytes × MaxBatchSize` stays comfortably +under 4 MB. If any single row approaches 4 MB, **stop**: that needs a LocalDb library change +(byte-aware batching or configurable message sizes) — a `scadaproj` effort, same class as the +keyed-instances hatch. + --- ## Execution waves @@ -133,7 +178,7 @@ stopped and started together.** Task 21 puts this in the deployment runbook. | 2 | 5, 6 | Store rewiring — independent files, dispatch in parallel | | 3 | 7, 8, 9 | Setup + migrators | | 4 | 10, 11 | Port test intents as specs — **before** any deletion | -| 5 | 12, 13 | Re-home the purge, delete notify-and-fetch | +| 5 | 12, 13 | Pin the active-node purge; notify-and-fetch scope check | | 6 | 14, 15, 16, 17 | The cutover, serial | | 7 | 18, 19 | Convergence suite + rig config | | 8 | 20, 21 | Live gate + docs | @@ -151,31 +196,46 @@ This is the gate item from `2026-07-19-localdb-phase2-gate.md` §5. It is measur **Files:** - Create: `docs/plans/2026-07-19-localdb-phase2-soak.md` +> **Method note (corrected):** on this rig the Phase 2 tables are still in the **legacy** DBs — +> `sf_messages` in `store-and-forward.db`, `native_alarm_state` in `scadabridge.db` — and are +> **not registered** with LocalDb, so driving S&F/alarm churn moves `__localdb_oplog` **not at +> all**. This soak therefore measures (a) **source write rates from the legacy DBs** and derives +> the oplog sizing arithmetically, and (b) that the Phase 1 oplog stays healthy alongside. The +> *empirical* post-cutover drain check lives in Task 20 (evidence item 10). +> +> Also: the containers have **no `sqlite3` binary** (bare `aspnet:10.0` image), but the data dirs +> are **host bind mounts** — run `sqlite3` on the host against `docker/site-a-node-*/data/`. +> Metrics are on port **8084** inside the container (not published to the host). + **Step 1: Bring up the rig with Phase 1 replication enabled** ```bash -cd ~/Desktop/ScadaBridge -git checkout feat/localdb-phase1 +cd ~/Desktop/ScadaBridge # already on feat/localdb-phase2 (= phase1 + docs) bash docker/deploy.sh ``` Site-a's pair replicates; site-b/site-c deliberately do not (the default-OFF pin). Confirm: ```bash -docker exec scadabridge-site-a-a curl -s localhost:8080/metrics | grep '^localdb_' +docker exec scadabridge-site-a-a curl -s localhost:8084/metrics | grep '^localdb_' ``` -Expected: `localdb_*` series present. If absent, the meter allowlist regressed — see +Expected: `localdb_*` series present (`localdb_oplog_depth`, `localdb_sync_*` — the meter is +`ZB.MOM.WW.LocalDb.Replication`). If absent, the meter allowlist regressed — see `SiteServiceRegistration.ObservedMeters`. -**Step 2: Capture a baseline** +**Step 2: Capture baselines (host-side, against the bind mounts)** ```bash -docker exec scadabridge-site-a-a sqlite3 /app/data/site-localdb.db \ - "SELECT COUNT(*) FROM __localdb_oplog;" +cd ~/Desktop/ScadaBridge/docker +sqlite3 site-a-node-a/data/site-localdb.db "SELECT COUNT(*) FROM __localdb_oplog;" +sqlite3 site-a-node-a/data/store-and-forward.db \ + "SELECT COUNT(*), COALESCE(SUM(retry_count),0) FROM sf_messages;" +sqlite3 site-a-node-a/data/scadabridge.db "SELECT COUNT(*) FROM native_alarm_state;" ``` -Record the count and the wall-clock time. +Record all values and the wall-clock time. (`COUNT + SUM(retry_count)` is the S&F row-write +proxy: every failed attempt is one `UPDATE` that increments `retry_count`.) **Step 3: Drive sustained load for 30 minutes** @@ -187,42 +247,69 @@ Two generators, run concurrently: - **Alarm churn.** Drive A&C conditions on a deployed instance so `native_alarm_state` upserts fire. This is the number that actually matters — it is unbounded by design. -**Step 4: Sample oplog depth every 5 minutes** +**Step 4: Sample write rates every 5 minutes** ```bash +cd ~/Desktop/ScadaBridge/docker for i in $(seq 1 6); do echo "=== t+$((i*5))m ===" - docker exec scadabridge-site-a-a sqlite3 /app/data/site-localdb.db \ - "SELECT COUNT(*) FROM __localdb_oplog;" - docker exec scadabridge-site-a-a curl -s localhost:8080/metrics \ - | grep -E '^localdb_(oplog_backlog|replication_dead_letters|sync_connected)' + sqlite3 site-a-node-a/data/store-and-forward.db \ + "SELECT COUNT(*), COALESCE(SUM(retry_count),0) FROM sf_messages;" + sqlite3 site-a-node-a/data/scadabridge.db \ + "SELECT COUNT(*), COUNT(CASE WHEN last_transition_at > datetime('now','-5 minutes') THEN 1 END) FROM native_alarm_state;" + sqlite3 site-a-node-a/data/site-localdb.db "SELECT COUNT(*) FROM __localdb_oplog;" + docker exec scadabridge-site-a-a curl -s localhost:8084/metrics \ + | grep -E '^localdb_(oplog_depth|sync)' sleep 300 done ``` -**Step 5: Record findings** +Per-interval deltas of `COUNT + SUM(retry_count)` give S&F row-writes/sec; the +`last_transition_at`-window count approximates alarm upserts/sec (it undercounts refresh-only +upserts — say so in the findings if alarm volume is near a threshold). + +**Step 5: Measure `config_json` sizes (D6)** + +```bash +sqlite3 site-a-node-a/data/scadabridge.db \ + "SELECT COUNT(*), MAX(LENGTH(config_json)), AVG(LENGTH(config_json)) FROM deployed_configurations;" +``` + +If the rig's configs are trivially small, also run this against a production-representative +legacy DB (wonder) before trusting the number. + +**Step 6: Record findings** Write `docs/plans/2026-07-19-localdb-phase2-soak.md` with: - rows/sec observed for `sf_messages` and for `native_alarm_state`, separately -- peak `localdb_oplog_backlog` -- whether backlog drained between bursts or grew monotonically +- max/avg `config_json` bytes and the derived **`MaxBatchSize`** (`max-row-bytes × MaxBatchSize` + comfortably under 4 MB — D6) +- Phase 1 oplog depth over the run (should stay near zero — it is not driven by this load) - dead-letter count (must be 0) - **the recommended `MaxOplogRows` and `MaxOplogAge`**, with the arithmetic + (sustained rows/sec × retention window, with alarm-storm headroom) -**Step 6: The decision gate** +**Step 7: The decision gate** -If backlog grew monotonically and never drained, the shared oplog cannot absorb this write profile. -**Stop. Do not proceed to Task 3.** Report to the user: the design doc's keyed-instances escape -hatch (design doc line 140) is required first, and that is a library effort in `scadaproj`. +If the measured sustained write rate × any reasonable `MaxOplogAge` exceeds any reasonable +`MaxOplogRows` — i.e. the shared oplog cannot absorb this write profile even on paper — **stop. +Do not proceed to Task 3.** Report to the user: the keyed-instances escape hatch +(`~/Desktop/scadaproj/docs/plans/2026-07-19-scadabridge-localdb-design.md:139-141`) is required +first, and that is a library effort in `scadaproj`. Likewise stop if any single `config_json` +approaches 4 MB (D6 — needs byte-aware batching in the library). The empirical +drain-under-churn confirmation happens post-cutover in Task 20; monotonic growth there is the +same stop condition. -**Step 7: Commit** +**Step 8: Commit** ```bash -git checkout -b feat/localdb-phase2 git add docs/plans/2026-07-19-localdb-phase2-soak.md git commit -m "docs(localdb): phase 2 rig soak findings — oplog sizing evidence" ``` +(The `feat/localdb-phase2` branch already exists — it carries this plan — so commit to it; +do **not** `git checkout -b`.) + --- ### Task 2: Decision record @@ -456,11 +543,14 @@ dotnet test tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ZB.MOM.WW.ScadaBri Same shape as Task 5, but note the differences: -- There are **22** inline `new SqliteConnection(_connectionString)` + `OpenAsync()` pairs, not a - helper. Add a `private SqliteConnection OpenConnection() => _localDb.CreateConnection();` and - convert all of them. -- `public SqliteConnection CreateConnection()` at `:51` is a public escape hatch used by site - repositories. Keep the member; change the body to `=> _localDb.CreateConnection();`. +- There are **21** inline `new SqliteConnection(_connectionString)` + `OpenAsync()` pairs, not a + helper (lines 60, 206, 248, 311, 345, 386, 414, 443, 470, 498, 538, 583, 608, 640, 663, 692, + 730, 769, 813, 837, 868 — counted before Task 4, which removes the :60 one inside + `InitializeAsync`). Add a `private SqliteConnection OpenConnection() => _localDb.CreateConnection();` + and convert all of them. +- `public SqliteConnection CreateConnection()` at `:51` is a public escape hatch used by + `SiteExternalSystemRepository` (the only repository consumer). Keep the member; change the body + to `=> _localDb.CreateConnection();`. - The busy-timeout floor (`BusyTimeoutFloorSeconds`, `:36`) and its `SqliteConnectionStringBuilder` normalization are **deleted** — LocalDb configures pragmas on every connection it hands out. - `AddSiteRuntime(string siteDbConnectionString)` loses its parameter. Keep the no-arg overload at @@ -555,9 +645,17 @@ Three, minimum: [Fact] public void MigratedSfMessages_EnterTheOplog_SoTheyActuallyReplicate() { } ``` -The third is the one that catches the ordering defect. Phase 1's equivalent asserts on -`__localdb_oplog` directly — copy that assertion. **The oplog table is `__localdb_oplog`** -(`LocalDbSchema.cs:19`), not `zb_oplog`. +The third is the one that catches the ordering defect. Phase 1's equivalent +(`SiteLocalDbLegacyMigratorTests.cs:244`, `MigratedRows_EnterTheOplog_SoTheyActuallyReplicate`) +asserts on `__localdb_oplog` directly — copy that assertion. **The oplog table is +`__localdb_oplog`** (`LocalDbSchema.cs:19`), not `zb_oplog`. + +> **Harness note:** production `OnReady` does not register `sf_messages` until the Task 14 +> cutover, so these tests must build their own `ILocalDb` (`TestLocalDb`), call +> `RegisterReplicated("sf_messages")` themselves, and then run the migrator — the oplog +> assertion needs live capture triggers. Correspondingly, Task 14 must keep +> `SiteLocalDbLegacyMigrator.Migrate` as the **last** call in `OnReady`, after all +> registrations. **Step 2: Add `ResolveStoreAndForwardPath` + `MigrateStoreAndForward`** @@ -639,8 +737,8 @@ Port these **intents** (not the mechanics) as two-node convergence assertions, u | Apply Add twice is idempotent, newest wins | LWW gives this | | Park when Add was lost still materialises | upsert semantics | -**Explicitly do not port:** `ReplicationServiceTests` #10 (200 interleaved ops dispatched -synchronously in strict issue order, 400 observed). That asserts the *mechanism* — inline +**Explicitly do not port:** `ReplicationServiceTests.ReplicationOperations_AreDispatchedInIssueOrder` +(`:169-195`; 200 interleaved ops dispatched synchronously in strict issue order, 400 observed). That asserts the *mechanism* — inline fire-and-forget dispatch — not the outcome. CDC capture is asynchronous and batched by construction. The portable intent is row 5 above. Record this in the test file as a comment so a future reader does not think it was dropped by accident. @@ -684,78 +782,87 @@ fetch path is deleted, so these describe code that will not exist. Replace with that a config deploy on A converges to B **without** B making any HTTP call — assert on a fetcher test double that records zero invocations. That is the positive proof that notify-and-fetch is gone. +> **Scope the zero-fetch assertion to the deploy-replication flow.** `SiteReconciliationActor` +> survives Phase 2 and legitimately HTTP-fetches at node startup when central reports gaps (D1). +> The two-node harness has no central, so its pass no-ops there — but do not write the assertion +> as "the standby never fetches, ever"; assert the fetcher double records zero calls **during +> deploy convergence**. + **Commit:** `test(localdb): port resync + config replication intents as CDC convergence specs` --- -### Task 12: Re-home the SMTP purge off the replication path +### Task 12: Pin the active-node SMTP purge -**Classification:** high-risk -**Estimated implement time:** ~4 min +**Classification:** standard +**Estimated implement time:** ~3 min **Parallelizable with:** none -Per D3. **This must land before any deletion** so the purge never lapses, even for one commit. +Per D3 (corrected): the active node's artifact-apply **already** calls +`PurgeCentralOnlyNotificationConfigAsync` — `DeploymentManagerActor.cs:1921`, inside +`HandleDeployArtifacts` (`:1864-1963`). Nothing is re-homed. What is missing is a **pin**: +Task 16 edits this actor's wiring, and no test today fails if the purge call is dropped +(`ArtifactStorageTests` covers the storage method, not the actor's call site). **Files:** -- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs` (artifact-apply path) - Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs` -**Step 1: Write the failing test** +**Step 1: Write the pin test (red-first by commenting out the call locally, then restore)** ```csharp [Fact] -public async Task ApplyingArtifacts_PurgesCentralOnlyNotificationConfig_OnTheActiveNode() +public async Task ApplyingArtifacts_PurgesCentralOnlyNotificationConfig() { // Security cleanup: notification_lists and smtp_configurations can hold plaintext - // SMTP passwords from pre-2026-07-10 deployments. This ran on the standby via - // SiteReplicationActor.HandleApplyArtifacts; with that actor deleted the ACTIVE - // node must do it, and CDC replicates the deletes as ordinary tombstones. + // SMTP passwords from pre-2026-07-10 deployments. The ACTIVE node's artifact-apply + // purges them (DeploymentManagerActor.HandleDeployArtifacts); the standby's copy of + // this call dies with SiteReplicationActor in Task 15. This test pins the active-node + // call so Task 16's actor edits cannot silently drop it. + // Arrange: real temp-file SiteStorageService, seed one row into each table, + // drive HandleDeployArtifacts; assert both tables are empty afterwards. } ``` -**Step 2: Call `PurgeCentralOnlyNotificationConfigAsync` from the active node's artifact-apply path** +**Step 2: No production change is expected.** If recon of the call site shows otherwise, stop +and report. -The existing call site is `SiteReplicationActor.HandleApplyArtifacts`. Find where -`DeploymentManagerActor` applies artifacts locally (the same path that reaches -`_replicationActor?.Tell(new ReplicateArtifacts(command))` at `:1952`) and call it there. - -**Step 3: Verify the deletes replicate** — extend the Task 11 convergence suite with a case where -A purges a seeded row and B's copy disappears. - -**Commit:** `fix(site): purge central-only notification config on the active node` +**Commit:** `test(site): pin the active-node notification-config purge` --- -### Task 13: Delete the notify-and-fetch config path +### Task 13: Notify-and-fetch scope check — the guarded write stays -**Classification:** high-risk -**Estimated implement time:** ~4 min +**Classification:** standard +**Estimated implement time:** ~3 min **Parallelizable with:** none -Per D1. +Per D1 (corrected). The original task deleted `StoreDeployedConfigIfNewerAsync` here — that was +wrong twice over: (a) it has a **second surviving caller**, and (b) deleting it before Task 15 +would not even compile, since `SiteReplicationActor` still calls it until then. **Files:** -- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs:301-336` -- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptions.cs:66-68` -- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptionsValidator.cs:56-58` +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs:301-336` (doc comment only) -**Step 1: Delete `StoreDeployedConfigIfNewerAsync`** — its only caller is -`SiteReplicationActor.HandleApplyConfigDeploy`, deleted in Task 15. The unguarded -`StoreDeployedConfigAsync` (`:241-271`) is the active node's path and stays. +**Step 1: Keep `StoreDeployedConfigIfNewerAsync`.** Its callers are +`SiteReplicationActor.HandleApplyConfigDeploy` (`SiteReplicationActor.cs:375` — dies with the +actor in Task 15) **and `SiteReconciliationActor` (`SiteReconciliationActor.cs:166`)** — the +per-node startup self-heal against central, which survives Phase 2 and still needs the +`deployed_at` guard against a stale reconcile fetch. Update the method's doc comment to name +reconciliation as the remaining caller (and drop any mention of standby replication). The +unguarded `StoreDeployedConfigAsync` (`:241-271`) stays the active deploy path. -**Step 2: Check `ConfigFetchRetryCount` before removing it** +**Step 2: `ConfigFetchRetryCount` — answered, but its removal moves to Task 17.** -`IDeploymentConfigFetcher` is used by **both** the standby replication path *and* the active -singleton's `RefreshDeploymentCommand` path. Only the first is being deleted. +The grep was run during plan review: the only production reader is `SiteReplicationActor.cs:157` +(plus its validator rule and tests). `SiteReconciliationActor` does a single best-effort pass and +does not read it; `IDeploymentConfigFetcher` itself is kept (used by `DeploymentManagerActor`'s +refresh path, `SiteReconciliationActor`, and registered at `ServiceCollectionExtensions.cs:84`). +So the option (`SiteRuntimeOptions.cs:68`) and its validator rule +(`SiteRuntimeOptionsValidator.cs:56-58`) become dead **after Task 15 deletes the actor** — remove +them in Task 17 (config-key cleanup), not here, or the build breaks. Re-run +`grep -rn "ConfigFetchRetryCount" src/ tests/` there to confirm. -```bash -grep -rn "ConfigFetchRetryCount" src/ tests/ -``` - -If the active path reads it, **keep the option and its validator rule** and note that in the commit -message. If nothing outside the deleted actor reads it, remove both. - -**Commit:** `refactor(site): delete notify-and-fetch config replication — CDC ships the row` +**Commit:** `docs(site): StoreDeployedConfigIfNewerAsync serves reconciliation — guard stays under CDC` --- @@ -771,19 +878,36 @@ never both run. **Files:** - Modify: `src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbSetup.cs` - Delete: `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/ReplicationService.cs` -- Modify: `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardService.cs:39,243,654,805,836,872,1120,1146` +- Modify: `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardService.cs` — field `:39`, + ctor param `:243`, and the 6 emission call sites `:654, 805, 836, 872, 1120, 1146` - Modify: `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/ServiceCollectionExtensions.cs:63-68` + (the `ReplicationService` registration) **and `:32`** (`GetRequiredService()` + inside the `StoreAndForwardService` factory at `:27-61`) - Modify: `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardStorage.cs:284-306` - Delete: `tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ReplicationServiceTests.cs` - Delete: `tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardReplicationTests.cs` - Delete: `tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ReplicationWireSerializationPinTests.cs` -**Step 1: Register all 11 tables** in `SiteLocalDbSetup.OnReady`, replacing the Task 7 comment. -Skip `notification_lists` and `smtp_configurations`? **No — register them.** They must replicate -their *deletes* (Task 12). Register all 9 config tables plus `sf_messages`. +**Step 1: Register 8 tables** in `SiteLocalDbSetup.OnReady`, replacing the Task 7 comment: the +7 config tables (`deployed_configurations`, `static_attribute_overrides`, `shared_scripts`, +`external_systems`, `database_connections`, `data_connection_definitions`, `native_alarm_state`) +plus `sf_messages`. **Do NOT register `notification_lists` or `smtp_configurations`** (this +reverses the plan's original instruction): they are permanently empty by design — no site writer +since 2026-07-10, the migrator skips them (Task 9), and the active-node purge +(`DeploymentManagerActor.cs:1921`) keeps them empty — so registering them would create a standing +replication channel whose only historical payload was plaintext SMTP passwords, for zero benefit. +Put that rationale in an `OnReady` comment so a future writer has to make an explicit decision. -**Step 2: Delete `ReplicationService`** and remove the ctor parameter + all 6 emission call sites -from `StoreAndForwardService`. +Keep `SiteLocalDbLegacyMigrator.Migrate` as the **last** call in `OnReady`, after all +registrations, so migrated rows enter the oplog through live capture triggers (the Phase 1 +ordering that is silently load-bearing). + +Note the two composite-PK tables are fine: LocalDb's `RegisterReplicated` supports multi-column +PKs (`SqliteLocalDb.RegisterReplicated` orders `pk` ordinals), and no Phase 2 table has a BLOB +column (which would be rejected). + +**Step 2: Delete `ReplicationService`** and remove the ctor parameter (`:243`), the +`_replication` field (`:39`), and all 6 emission call sites from `StoreAndForwardService`. **Step 3: Delete `ReplaceAllAsync`** (`:284-306`). Per D2 the library's resync merges; a destructive delete-all must not survive into a replicated table, where CDC would capture the mass-delete and @@ -814,14 +938,20 @@ singleton. Delete that assertion. - Delete: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/ResyncWireSerializationPinTests.cs` - Delete: `tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/SfBufferResyncPredicateTests.cs` -The whole `ReplicationMessages.cs` file becomes dead: `ReplicateConfigDeploy/Remove/SetEnabled`, -`ReplicateArtifacts`, `ReplicateStoreAndForward`, `ApplyConfigDeploy/Remove/SetEnabled`, -`ApplyArtifacts`, `ApplyStoreAndForward`, plus the resync records `RequestSfBufferResync`, -`SfBufferSnapshot`, `SfBufferSnapshotChunk`, `SfBufferResyncAck`. +The whole `ReplicationMessages.cs` file becomes dead — it contains exactly the 10 records +`ReplicateConfigDeploy/Remove/SetEnabled`, `ReplicateArtifacts`, `ReplicateStoreAndForward`, +`ApplyConfigDeploy/Remove/SetEnabled`, `ApplyArtifacts`, `ApplyStoreAndForward`, whose only +consumers are `SiteReplicationActor`, the 5 `DeploymentManagerActor` Tell sites (Task 16), the +`AkkaHostedService` wiring (Task 16), and the deleted tests. The four resync records +(`RequestSfBufferResync`, `SfBufferSnapshot`, `SfBufferSnapshotChunk`, `SfBufferResyncAck`) are +**not** in this file — they are declared at the bottom of `SiteReplicationActor.cs` (`:678-707`) +and die with the actor file. **Do not delete** `ActiveNodeEvaluator` (`Communication/ClusterState/ActiveNodeEvaluator.cs`) — the -S&F **delivery gate** still needs it. Only its doc comment at `:14` mentions replication; update -the comment, keep the type. +S&F **delivery gate** still needs it (`AkkaHostedService.cs:866` → `SetDeliveryGate` → +`SelfIsPrimary` → `ActiveNodeEvaluator.SelfIsOldestUp`). Its doc comment at `:14` mentions +`SiteReplicationActor` and `:16` mentions the resync authority checks; the actor's own direct call +at `SiteReplicationActor.cs:288` goes away with the file. Update the comment, keep the type. Confirm the deletions are complete before committing: @@ -845,16 +975,19 @@ Expected: no matches. - Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs:55,161-190,794,970,1004,1102,1952` - Modify: `src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs:770-796,806-812,1056` -**Step 1:** Remove the `IActorRef? replicationActor = null` ctor parameter (`:184`), the -`_replicationActor` field (`:55`), and all 5 `_replicationActor?.Tell(...)` sites. +**Step 1:** Remove the `IActorRef? replicationActor = null` ctor parameter (**`:169`** — the ctor +signature spans `:161-175`; `:184` is the field *assignment*), the `_replicationActor` field +(`:55`), and all 5 `_replicationActor?.Tell(...)` sites (`:794, 970, 1004, 1102, 1952`). The parameter is optional and positional — removing it **silently shifts every argument after it** -(`healthCollector`, `serviceProvider`, `loggerFactory`, `configFetcher`, ...). Update the -`Props.Create` at `AkkaHostedService.cs:806-812` and every test constructing this actor. Compile -errors will not catch a positional shift between two same-typed optional parameters — check each -call site by hand. +(`healthCollector`, `serviceProvider`, `loggerFactory`, `configFetcher`, +`startupLoadRetryInterval`, `configLoader`). The sharpest hazard is the **same-typed `IActorRef?` +optional immediately before it** — `dclManager` at `:168`: a caller passing positionally could +silently feed the wrong actor ref with zero compile errors. Update the `Props.Create` at +`AkkaHostedService.cs:806-812` (which passes `replicationActor` positionally at `:810`) and every +test constructing this actor — check each call site by hand. -**Step 2:** Delete `replicationService` / `replicationLogger` resolution (`:770-772`), the actor +**Step 2:** Delete `replicationService` / `replicationLogger` resolution (`:770-773`), the actor creation (`:785-789`), the handler wiring (`:792-796`), and the actor-inventory comment at `:1056`. **Step 3:** `activeNodeCheck` (`:781-783`) is still used by `SiteCommunicationActor` (`:816-821`). @@ -875,18 +1008,23 @@ Keep it. **Files:** - Modify: `src/ZB.MOM.WW.ScadaBridge.Host/StartupValidator.cs:113-115` - Modify: `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardOptions.cs:12` -- Modify: `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardOptionsValidator.cs:20` -- Modify: 9 × `appsettings.Site.json` (Host template + 6 `docker/site-*` + 2 `docker-env2/site-x-*`) +- Modify: `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardOptionsValidator.cs:19-21` +- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptions.cs:68` + `SiteRuntimeOptionsValidator.cs:56-58` (moved here from Task 13) +- Modify: **10** × `appsettings.Site.json` (Host template + 6 `docker/site-*` + 2 `docker-env2/site-x-*` + + **`deploy/wonder-app-vd03/appsettings.Site.json`** — the wonder single-node install, which the + original count missed; it sets both paths *and* `ReplicationEnabled: false`) | Key | Action | |---|---| | `ScadaBridge:Database:SiteDbPath` | → migration-only. **Relax the `Require` at `StartupValidator.cs:113`** or every site config keeps it mandatory forever. | -| `ScadaBridge:StoreAndForward:SqliteDbPath` | → migration-only. Relax the non-empty rule at `StoreAndForwardOptionsValidator.cs:20`. | -| `ScadaBridge:StoreAndForward:ReplicationEnabled` | → **fully dead.** Delete the property and all 9 config entries. | +| `ScadaBridge:StoreAndForward:SqliteDbPath` | → migration-only. Relax the non-empty rule at `StoreAndForwardOptionsValidator.cs:19-21`. | +| `ScadaBridge:StoreAndForward:ReplicationEnabled` | → **fully dead.** Delete the property and every config entry (including wonder's). | +| `SiteRuntime … ConfigFetchRetryCount` | → **dead after Task 15** (only reader was `SiteReplicationActor.cs:157`; see Task 13). Delete the option, its validator rule, and any config entries — confirm with `grep -rn "ConfigFetchRetryCount" src/ tests/` first. | `ParkedOperationRelayTests.cs:39` and `ParkedMessageHandlerActorTests.cs:34` set `ReplicationEnabled = false` — they only need the flag to exist, so remove those lines. -`StoreAndForwardOptionsTests.cs:14,26,33` assert its default; delete those assertions. +`StoreAndForwardOptionsTests.cs` touches it three times: `:14` asserts the default, `:26` sets and +`:33` asserts a customized value — remove all three references. **Leave the path keys present in configs** with a comment — they are read by the legacy migrator and removing them would strand un-migrated data on a node that has not yet started once. @@ -917,9 +1055,12 @@ Scenarios, beyond Tasks 10–11: 3. A burst of `native_alarm_state` upserts converges and the oplog drains. 4. Node B restarts and catches up via snapshot resync without losing rows written on B while down. -**Verify the suite is non-vacuous.** Phase 1 did this by mismatching the two API keys and confirming -all scenarios went red. Do the same and record the result in the commit message. A convergence suite -that cannot fail proves nothing. +**Verify the suite is non-vacuous.** (Corrected claim: Phase 1's convergence suite did **not** do +a mismatched-key run — it uses one shared key, `LocalDbSitePairConvergenceTests.cs:47`; wrong-key +*denial* is unit-covered separately in `Host.Tests/LocalDbSyncAuthInterceptorTests.cs`.) Prove it +directly for the new scenarios: run once with the new table registrations disabled on node B (or a +deliberately wrong key) and confirm every new scenario goes red, then restore and record the +result in the commit message. A convergence suite that cannot fail proves nothing. **Commit:** `test(localdb): two-node convergence for config tables + sf_messages` @@ -934,8 +1075,11 @@ that cannot fail proves nothing. **Files:** - Modify: `docker/site-a-node-a/appsettings.Site.json`, `docker/site-a-node-b/appsettings.Site.json` -Set `MaxOplogRows` and `MaxOplogAge` from **Task 1's measured numbers** — not from a guess. Keep -site-b/site-c unreplicated so default-OFF stays proven side by side on one rig. +Set `LocalDb:Replication:MaxOplogRows` and `LocalDb:Replication:MaxOplogAge` (exact key paths — +the lib binds the `LocalDb:Replication` section; defaults 1,000,000 rows / 7 days) from **Task 1's +measured numbers** — not from a guess. Per D6, also set `LocalDb:Replication:MaxBatchSize` +(default 500) so `max-config_json-bytes × MaxBatchSize` stays comfortably under the 4 MB gRPC +receive limit. Keep site-b/site-c unreplicated so default-OFF stays proven side by side on one rig. **Commit:** `chore(docker): size the site-a oplog caps from the phase 2 soak` @@ -954,15 +1098,24 @@ site-b/site-c unreplicated so default-OFF stays proven side by side on one rig. cd ~/Desktop/ScadaBridge && bash docker/deploy.sh ``` +Tooling: the containers have no `sqlite3` — run DB checks host-side against the bind mounts +(`docker/site-a-node-{a,b}/data/`); metrics via +`docker exec scadabridge-site-a-a curl -s localhost:8084/metrics` (port **8084**, not published +to the host). + Evidence required — each must be captured, not asserted: 1. **Migration ran.** `store-and-forward.db.migrated` and `scadabridge.db.migrated` exist on both - site-a nodes; row counts in the consolidated DB match the legacy files. + site-a nodes; row counts in the consolidated DB match the legacy files. (Both nodes migrate + their own legacy files independently; identical-content rows then LWW-converge — expected, + not an anomaly.) 2. **No SMTP/notification rows migrated** — `SELECT COUNT(*) FROM smtp_configurations` is 0. 3. **Config converges.** Deploy an instance to site-a; both nodes' `deployed_configurations` rows are byte-identical with identical `__localdb_row_version` and originating node id. -4. **The standby made no HTTP fetch** — nothing in its log resembling a config fetch. This is the - positive proof that notify-and-fetch is gone. +4. **The standby made no config HTTP fetch during the deploy** (both nodes up) — grep the deploy + window of its log. This is the positive proof that notify-and-fetch is gone. A + `SiteReconciliationActor` fetch at node *startup* is legitimate (it survives Phase 2) and does + not fail this check. 5. **S&F converges.** Enqueue against a dead target; the parked row appears on the peer. 6. **Site failover.** Flip the site-a pair and confirm buffered messages survive and deliver exactly once after the flip. `docker/failover-drill.sh` targets **CENTRAL** nodes and does not @@ -971,6 +1124,10 @@ Evidence required — each must be captured, not asserted: `native_alarm_state` rows on the peer. 8. **Zero dead letters**, oplog drained, `localdb_*` scraped from `/metrics`. 9. **Both nodes stopped and started together** (D5) — confirm a clean rejoin. +10. **Post-cutover drain under churn** — the empirical half Task 1 could not measure pre-cutover: + re-run Task 1's two load generators against this build and watch `localdb_oplog_depth` + (and `SELECT COUNT(*) FROM __localdb_oplog`) rise **and drain between bursts**. Monotonic + growth that never drains is the same stop condition as Task 1 step 7 (keyed-instances hatch). **If any check fails, stop and report.** Do not proceed to Task 21. @@ -989,18 +1146,22 @@ Evidence required — each must be captured, not asserted: - Modify: `docs/requirements/Component-StoreAndForward.md:83` - Modify: `docs/components/StoreAndForward.md`, `SiteRuntime.md`, `Host.md` - Modify: `docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md` -- Modify: `docs/deployment/` runbook +- Modify: `docs/deployment/installation-guide.md` + `topology-guide.md` (+ `production-checklist.md` + if it covers upgrades) — there is no file literally named "runbook" Specific corrections: - `Component-StoreAndForward.md:83` is a long **normative** paragraph specifying the whole chunked resync protocol and the N5 race. Rewrite for CDC — state the new duplicate-delivery bound explicitly rather than deleting the discussion. -- The frame-size known-issue is **resolved by** this change (D1). Mark it resolved and say why, - rather than leaving a live known-issue describing deleted code. -- **Add to the deployment runbook (D5):** a site pair must be stopped and started together. Rolling +- The frame-size known-issue is **already marked "Status: RESOLVED (2026-06-26)"** (resolved by the + notify-and-fetch rework). Amend it to record that Phase 2 deleted notify-and-fetch itself — CDC + ships the row over gRPC, so the frame constraint is gone entirely — and note the successor + ceiling: the 4 MB gRPC message cap managed via `MaxBatchSize` (D6). +- **Add to the deployment docs (D5):** a site pair must be stopped and started together. Rolling one node at a time is no longer supported, because the legacy `SfBufferSnapshot` compatibility - handler is gone. + handler is gone. Also record the D2 caveat: a node offline longer than + `LocalDb:Replication:TombstoneRetention` (default 7 days) can resurrect deleted rows on rejoin. - Update the scadaproj LocalDb component row: Phase 2 complete, what replicates now, what `SiteReplicationActor` used to do. diff --git a/docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json b/docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json index 41bd42b7..36401179 100644 --- a/docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json +++ b/docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json @@ -12,66 +12,83 @@ "date": "2026-07-19", "by": "user", "choice": "Full scope as designed", - "note": "User chose both surfaces (config tables + sf_messages) in one phase, deleting both bespoke mechanisms together, over the recommended split of S&F-first. The four open design questions are therefore resolved INSIDE this plan as D1-D5 rather than deferred." + "note": "User chose both surfaces (config tables + sf_messages) in one phase, deleting both bespoke mechanisms together, over the recommended split of S&F-first. The four open design questions are therefore resolved INSIDE this plan as D1-D6 rather than deferred." + }, + "reviewPass": { + "date": "2026-07-19", + "note": "Plan verified against actual code (3 verification sweeps + LocalDb library source) and corrected in place. Headline corrections: D1 (StoreDeployedConfigIfNewerAsync has a SECOND surviving caller, SiteReconciliationActor.cs:166 - the method and guard STAY; original Task 13 would also not have compiled, deleting a method whose caller dies only in Task 15), D3 (the active node ALREADY purges at DeploymentManagerActor.cs:1921 - Task 12 became a pin test, nothing is re-homed), D6 added (4 MB gRPC receive cap x row-count-only MaxBatchSize batching; config_json > 128 KB documented - measure in Task 1, size MaxBatchSize in Task 19, single row near 4 MB = stop/lib work), Task 1 rewritten (Phase 2 tables are NOT registered on the Phase 1 rig so driven churn never reaches __localdb_oplog - measure legacy-DB write rates + arithmetic; metrics port 8084 not 8080; real metric names are localdb_oplog_depth / localdb_sync_*, the plan's localdb_oplog_backlog/replication_dead_letters/sync_connected never existed; containers have no sqlite3 - use host-side sqlite3 on the bind mounts; branch already exists, no checkout -b), Task 14 (register 8 tables NOT 11 - notification_lists/smtp_configurations deliberately unregistered, reversing the original instruction; keep Migrate LAST in OnReady)." }, "decisions": [ { "id": "D1", - "subject": "Config moves to CDC; notify-and-fetch is DELETED, not preserved", - "evidence": "SiteReplicationActor sends id+fetch-coords only because the config blob exceeds Akka's 128KB frame (docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md). LocalDb sync is gRPC - no such limit. StoreDeployedConfigIfNewerAsync's guard (SiteStorageService.cs:325, `WHERE excluded.deployed_at > deployed_configurations.deployed_at`) protects only against a stale FETCH racing; with no fetch there is no race. Do NOT reproduce the guard on top of LWW - deployed_at and HLC are different clocks and mixing them is non-convergent." + "subject": "Config moves to CDC; notify-and-fetch is DELETED - but the guarded write STAYS", + "evidence": "SiteReplicationActor sends id+fetch-coords only because the config blob exceeds Akka's 128KB frame (docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md, already marked RESOLVED by the notify-and-fetch rework). LocalDb sync is gRPC - no such limit (but see D6). CORRECTED: StoreDeployedConfigIfNewerAsync (SiteStorageService.cs:301-336, guard at :325) has TWO production callers - SiteReplicationActor.cs:375 (dies in Task 15) AND SiteReconciliationActor.cs:166 (per-node startup self-heal vs central, SURVIVES Phase 2, stale-fetch race still real). Method + guard stay; reconcile becomes a benign second CDC writer. Do NOT reproduce the guard on top of LWW - deployed_at and HLC are different clocks and mixing them is non-convergent." }, { "id": "D2", "subject": "ReplaceAllAsync deleted; the N1 directional guard becomes unnecessary", - "evidence": "LocalDb's snapshot resync MERGES per-row LWW and never deletes: SnapshotApplier.OnBeginAsync (SnapshotStreamer.cs:163-170) resets counters only, no DELETE anywhere in the class; OnBatchAsync (:172-186) routes snapshot rows through the same LwwApplier as deltas; LwwApplier.cs:69-78 discards an incoming row whose HLC is lower. So a stale peer cannot wipe a live buffer - structurally impossible, not guarded. SEMANTIC CHANGE: the standby is convergent, no longer byte-identical." + "evidence": "LocalDb's snapshot resync MERGES per-row LWW and never WIPES: SnapshotApplier.OnBeginAsync (SnapshotStreamer.cs:163-170) resets counters only; OnBatchAsync (:172-186) routes snapshot rows through the same LwwApplier as deltas; LwwApplier.cs:69-78 discards an incoming row whose HLC is lower. Row-level deletes DO replicate (delete-trigger tombstones, streamed by SnapshotStreamer, applied as real DELETEs by LwwApplier) - only the destructive whole-table replace is gone. Caveat: tombstones pruned after TombstoneRetention (default 7d); a node offline longer can resurrect deleted rows (runbook, Task 21). SEMANTIC CHANGE: the standby is convergent, no longer byte-identical." }, { "id": "D3", - "subject": "The SMTP purge is re-homed BEFORE anything is deleted", - "evidence": "SiteReplicationActor.HandleApplyArtifacts calls PurgeCentralOnlyNotificationConfigAsync (SiteStorageService.cs:811-821), deleting notification_lists + smtp_configurations incl. plaintext SMTP passwords from pre-2026-07-10 deployments. A security cleanup riding the replication path; CDC will not reproduce it. Task 12 moves it to the active node's deploy path before any deletion, so it never lapses." + "subject": "CORRECTED: the SMTP purge already runs on the active node - pin it, don't move it", + "evidence": "PurgeCentralOnlyNotificationConfigAsync (SiteStorageService.cs:811-821) has TWO callers: DeploymentManagerActor.cs:1921 (ACTIVE node's HandleDeployArtifacts, :1864-1963) and SiteReplicationActor.cs:456 (standby copy, dies with the actor). The purge never lapses; the original 're-home before any deletion' premise was false. Task 12 = pin test only (ArtifactStorageTests covers the storage method, not the actor call site Task 16 edits). No site writer to notification_lists/smtp_configurations since 2026-07-10 (verified: only test seeding inserts exist) + migrator skips them => permanently empty in the consolidated DB => Task 14 does NOT register them." }, { "id": "D4", "subject": "native_alarm_state volume is MEASURED, not assumed", - "evidence": "scadabridge.db is not only config - native_alarm_state mirrors live A&C conditions (NativeAlarmActor.cs:504) and is the highest-volume table in either DB. sf_messages worst case ~50 row-writes/sec. Task 1 is a rig soak that measures both and sets MaxOplogRows/MaxOplogAge. If backlog grows monotonically, STOP: the design doc's keyed-instances escape hatch (line 140) is a scadaproj library effort that would suspend this plan." + "evidence": "scadabridge.db is not only config - native_alarm_state mirrors live A&C conditions (NativeAlarmActor.cs:504) and is the highest-volume table in either DB. sf_messages worst case ~50 row-writes/sec. Task 1 measures both AT THE LEGACY-DB SOURCE (they are not in the Phase 1 oplog - see reviewPass) and sets MaxOplogRows/MaxOplogAge arithmetically; Task 20 evidence 10 does the empirical post-cutover drain check. If growth is monotonic, STOP: keyed-instances escape hatch = ~/Desktop/scadaproj/docs/plans/2026-07-19-scadabridge-localdb-design.md:139-141 (adoption design doc, NOT the 07-17 lib doc), a scadaproj library effort that would suspend this plan." }, { "id": "D5", "subject": "Cutover forecloses rolling site upgrades", - "evidence": "SiteReplicationActor retains a legacy monolithic SfBufferSnapshot handler for rolling upgrades. With no dual-mechanism period, one node would speak a protocol the other no longer implements. Both nodes of a site must be stopped and started together. Task 21 puts this in the deployment runbook." + "evidence": "SiteReplicationActor retains a legacy monolithic SfBufferSnapshot handler for rolling upgrades. With no dual-mechanism period, one node would speak a protocol the other no longer implements. Both nodes of a site must be stopped and started together. Task 21 puts this in the deployment docs (installation-guide.md/topology-guide.md - no file literally named runbook)." + }, + { + "id": "D6", + "subject": "NEW (review pass): 4 MB gRPC message cap replaces the 128 KB Akka frame as the size ceiling", + "evidence": "Neither side configures gRPC message sizes (ScadaBridge AddGrpc at Program.cs:519-521 sets only the auth interceptor; the lib's initiator channel is bare GrpcChannel.ForAddress) => 4 MB default receive limit both directions. Batching is row-count-only (MaxBatchSize default 500; SyncSession.cs:227, SnapshotStreamer.cs:55; no byte-aware chunking). deployed_configurations.config_json documented >128 KB/row; a few dozen such rows in one batch exceeds 4 MB and wedges the stream on a poison batch. Task 1 measures max/avg config_json bytes; Task 19 sets LocalDb:Replication:MaxBatchSize so max-row-bytes x MaxBatchSize << 4 MB; any single row near 4 MB = STOP (needs byte-aware batching or size knobs in the LocalDb lib - scadaproj effort)." } ], "reconFindings": [ "The gate doc named 2 test files as the specification; the real spec is 5 - it missed StoreAndForwardReplicationTests.cs (incl. the only Requeue coverage), ReplicationWireSerializationPinTests.cs, ResyncWireSerializationPinTests.cs, and SfBufferResyncPredicateTests.cs (the N1 Critical regression test).", "sf_messages has NO version column today - ON CONFLICT(id) DO UPDATE has no comparison predicate (StoreAndForwardStorage.cs:331-345). 'Newest wins' is bare arrival order. LWW-by-HLC is an IMPROVEMENT here, not a regression.", - "No autoincrement-integer PKs exist anywhere in Phase 2 scope - all 9 config tables use natural TEXT or composite TEXT keys, sf_messages is TEXT. Phase 1's site_events GUID conversion has no Phase 2 analogue.", + "No autoincrement-integer PKs exist anywhere in Phase 2 scope - all 9 config tables use natural TEXT or composite TEXT keys, sf_messages is TEXT. LocalDb RegisterReplicated SUPPORTS composite PKs (ordered pk ordinals) and rejects BLOB columns - no Phase 2 table has one (verified). Phase 1's site_events GUID conversion has no Phase 2 analogue.", "SiteStorageService has NO foreign keys. RemoveDeployedConfigAsync (:343-376) is a manual 3-statement cascade in one transaction. Under CDC these become three independent delete streams that LWW may reorder - the most likely real defect in the plan (Task 18 scenario 2).", - "DeploymentManagerActor's replicationActor is an OPTIONAL POSITIONAL parameter (:184). Removing it silently shifts every argument after it. Compilers will not catch a shift between two same-typed optional params - check every call site by hand (Task 16).", - "ActiveNodeEvaluator must NOT be deleted - the S&F delivery gate still uses it. Only its doc comment mentions replication.", - "ConfigFetchRetryCount may not be fully dead: IDeploymentConfigFetcher serves BOTH the standby replication path and the active singleton's RefreshDeploymentCommand path. Verify before removing (Task 13 step 2).", - "notification_lists and smtp_configurations are deliberately NOT migrated (Task 9) - migrating them would resurrect plaintext SMTP passwords from a pre-fix legacy file into a replicated table." + "DeploymentManagerActor's replicationActor is an OPTIONAL POSITIONAL parameter at :169 (ctor :161-175; :184 is the field ASSIGNMENT). The same-typed IActorRef? optional dclManager sits immediately BEFORE it at :168 - that's the real silent-shift hazard. Props.Create passes it positionally at AkkaHostedService.cs:810. Check every call site by hand (Task 16).", + "ActiveNodeEvaluator must NOT be deleted - the S&F delivery gate still uses it (AkkaHostedService.cs:866 SetDeliveryGate -> SelfIsPrimary -> SelfIsOldestUp). Doc comment :14/:16 mentions replication; SiteReplicationActor.cs:288 calls it directly (dies with the actor).", + "ConfigFetchRetryCount's ONLY production reader is SiteReplicationActor.cs:157 (verified) - dead after Task 15; removed in Task 17 (removing it in Task 13, before the actor deletion, would not compile). IDeploymentConfigFetcher is KEPT: DeploymentManagerActor refresh path + SiteReconciliationActor + DI at ServiceCollectionExtensions.cs:84.", + "notification_lists and smtp_configurations are deliberately NOT migrated (Task 9) AND NOT registered (Task 14, corrected) - migrating or replicating them would resurrect/ship plaintext SMTP passwords; they are permanently empty by design (writers removed 2026-07-10; verified only test-seeding inserts exist).", + "REVIEW-PASS ADDITIONS: SiteReconciliationActor (runs on EVERY node at startup, best-effort self-heal vs central) is the second caller of both StoreDeployedConfigIfNewerAsync and IDeploymentConfigFetcher - it survives Phase 2 and constrains Tasks 11/13/20 (a startup fetch on the standby is legitimate; zero-fetch assertions must scope to the deploy window).", + "SiteStorageService has 21 (not 22) inline connection+OpenAsync pairs: 60,206,248,311,345,386,414,443,470,498,538,583,608,640,663,692,730,769,813,837,868. CreateConnection():51's only repository consumer is SiteExternalSystemRepository.", + "StoreAndForwardService: :39 is the _replication FIELD, :243 the ctor param; exactly 6 emission sites (:654,805,836,872,1120,1146). ServiceCollectionExtensions.cs:32 also resolves ReplicationService inside the StoreAndForwardService factory - must go in Task 14.", + "ReplicationMessages.cs holds ONLY the 10 Replicate*/Apply* records; the four SfBuffer resync records live at the bottom of SiteReplicationActor.cs (:678-707) and die with the actor file.", + "10 (not 9) appsettings.Site.json files set the legacy paths - deploy/wonder-app-vd03/appsettings.Site.json was missed (also sets ReplicationEnabled:false).", + "Rig facts: site metrics on port 8084 (AnyIP in-container, NOT published to host; 8080 is Traefik); real metric names localdb_oplog_depth / localdb_sync_* (meter ZB.MOM.WW.LocalDb.Replication); containers (aspnet:10.0) have NO sqlite3 - data dirs are host bind mounts, run sqlite3 host-side; ReplicationOptions bind at LocalDb:Replication:* (lib binds the section).", + "Phase 1's LocalDbSitePairConvergenceTests uses ONE shared API key (:47) - it never did a mismatched-key non-vacuity run; wrong-key denial is unit-covered in Host.Tests/LocalDbSyncAuthInterceptorTests.cs. Task 18's non-vacuity check must be done directly for the new scenarios.", + "MigrateEvents synthesizes deterministic 'mig-{node}-{legacyId}' ids (NOT fresh GUIDs - crash-rerun idempotency). Migration runs AFTER RegisterReplicated in OnReady (order is load-bearing); Task 8's oplog pin test must register sf_messages on its own TestLocalDb since production doesn't register it until Task 14." ], "tasks": [ - {"id": 1, "subject": "Task 1: Rig soak - measure oplog growth under real write rates", "status": "pending", "classification": "high-risk", "note": "GATES EVERYTHING AFTER TASK 2. Stop the plan if oplog backlog grows monotonically."}, + {"id": 1, "subject": "Task 1: Rig soak - measure legacy-DB write rates + config_json sizes (corrected method)", "status": "pending", "classification": "high-risk", "note": "GATES EVERYTHING AFTER TASK 2. Phase 2 tables are NOT in the Phase 1 oplog - measure at the legacy DBs host-side, size caps arithmetically; empirical drain check moves to Task 20 evidence 10. Stop conditions: on-paper oplog overflow OR any config_json near 4 MB (D6)."}, {"id": 2, "subject": "Task 2: Decision record - close the phase 2 gate", "status": "pending", "classification": "trivial", "blockedBy": [1]}, {"id": 3, "subject": "Task 3: Extract StoreAndForwardSchema.Apply", "status": "pending", "classification": "small", "blockedBy": [2]}, {"id": 4, "subject": "Task 4: Extract SiteStorageSchema.Apply", "status": "pending", "classification": "small", "blockedBy": [2]}, {"id": 5, "subject": "Task 5: Rewire StoreAndForwardStorage onto ILocalDb", "status": "pending", "classification": "high-risk", "blockedBy": [3]}, {"id": 6, "subject": "Task 6: Rewire SiteStorageService onto ILocalDb", "status": "pending", "classification": "high-risk", "blockedBy": [4]}, {"id": 7, "subject": "Task 7: Extend SiteLocalDbSetup with the new DDL (not yet registered)", "status": "pending", "classification": "standard", "blockedBy": [5, 6]}, - {"id": 8, "subject": "Task 8: Extend the legacy migrator for sf_messages", "status": "pending", "classification": "high-risk", "blockedBy": [7]}, - {"id": 9, "subject": "Task 9: Extend the legacy migrator for the 9 config tables", "status": "pending", "classification": "high-risk", "blockedBy": [7]}, + {"id": 8, "subject": "Task 8: Extend the legacy migrator for sf_messages", "status": "pending", "classification": "high-risk", "blockedBy": [7], "note": "Oplog pin test registers sf_messages on its own TestLocalDb - production OnReady doesn't register it until Task 14."}, + {"id": 9, "subject": "Task 9: Extend the legacy migrator for the config tables (skip notification/smtp)", "status": "pending", "classification": "high-risk", "blockedBy": [7]}, {"id": 10, "subject": "Task 10: Port the S&F replication test intents as CDC specs", "status": "pending", "classification": "standard", "blockedBy": [8]}, - {"id": 11, "subject": "Task 11: Port the resync + directional-authority tests", "status": "pending", "classification": "standard", "blockedBy": [9]}, - {"id": 12, "subject": "Task 12: Re-home the SMTP purge off the replication path", "status": "pending", "classification": "high-risk", "blockedBy": [10, 11], "note": "MUST land before any deletion so the security purge never lapses."}, - {"id": 13, "subject": "Task 13: Delete the notify-and-fetch config path", "status": "pending", "classification": "high-risk", "blockedBy": [12]}, - {"id": 14, "subject": "Task 14: Register the Phase 2 tables and delete ReplicationService", "status": "pending", "classification": "high-risk", "blockedBy": [13], "note": "THE CUTOVER. Registration + bespoke deletion in ONE commit so both mechanisms never run together."}, - {"id": 15, "subject": "Task 15: Delete SiteReplicationActor and its messages", "status": "pending", "classification": "high-risk", "blockedBy": [14]}, - {"id": 16, "subject": "Task 16: Clean up DeploymentManagerActor and AkkaHostedService", "status": "pending", "classification": "standard", "blockedBy": [15]}, - {"id": 17, "subject": "Task 17: Config-key cleanup", "status": "pending", "classification": "standard", "blockedBy": [16]}, - {"id": 18, "subject": "Task 18: Two-node convergence suite for the Phase 2 tables", "status": "pending", "classification": "high-risk", "blockedBy": [17]}, - {"id": 19, "subject": "Task 19: Rig configuration", "status": "pending", "classification": "small", "blockedBy": [17]}, - {"id": 20, "subject": "Task 20: Live gate on the docker rig", "status": "pending", "classification": "high-risk", "blockedBy": [18, 19]}, + {"id": 11, "subject": "Task 11: Port the resync + directional-authority tests", "status": "pending", "classification": "standard", "blockedBy": [9], "note": "Zero-fetch assertion scoped to deploy convergence - SiteReconciliationActor's startup fetch is legitimate and survives."}, + {"id": 12, "subject": "Task 12: Pin the active-node SMTP purge (corrected: already exists at DeploymentManagerActor.cs:1921)", "status": "pending", "classification": "standard", "blockedBy": [10, 11], "note": "Pin test only - no production change. Guards Task 16's actor edits from silently dropping the purge call."}, + {"id": 13, "subject": "Task 13: Notify-and-fetch scope check - guarded write STAYS for SiteReconciliationActor", "status": "pending", "classification": "standard", "blockedBy": [12], "note": "CORRECTED: do NOT delete StoreDeployedConfigIfNewerAsync (second caller SiteReconciliationActor.cs:166 survives; deleting here wouldn't compile anyway). Doc-comment update only; ConfigFetchRetryCount removal moved to Task 17."}, + {"id": 14, "subject": "Task 14: Register the 8 Phase 2 tables and delete ReplicationService", "status": "pending", "classification": "high-risk", "blockedBy": [13], "note": "THE CUTOVER. Register 7 config tables + sf_messages - NOT notification_lists/smtp_configurations (reversed from original; see D3). Keep Migrate LAST in OnReady. Registration + bespoke deletion in ONE commit."}, + {"id": 15, "subject": "Task 15: Delete SiteReplicationActor and its messages", "status": "pending", "classification": "high-risk", "blockedBy": [14], "note": "Resync records live at SiteReplicationActor.cs:678-707, not ReplicationMessages.cs."}, + {"id": 16, "subject": "Task 16: Clean up DeploymentManagerActor and AkkaHostedService", "status": "pending", "classification": "standard", "blockedBy": [15], "note": "Ctor param at :169; same-typed IActorRef? dclManager at :168 BEFORE it is the silent-shift hazard; Props.Create positional at AkkaHostedService.cs:810."}, + {"id": 17, "subject": "Task 17: Config-key cleanup (incl. ConfigFetchRetryCount; 10 appsettings files)", "status": "pending", "classification": "standard", "blockedBy": [16]}, + {"id": 18, "subject": "Task 18: Two-node convergence suite for the Phase 2 tables", "status": "pending", "classification": "high-risk", "blockedBy": [17], "note": "Non-vacuity must be proven directly (Phase 1 never did the mismatched-key run the original plan claimed)."}, + {"id": 19, "subject": "Task 19: Rig configuration (MaxOplogRows/MaxOplogAge + MaxBatchSize per D6)", "status": "pending", "classification": "small", "blockedBy": [17]}, + {"id": 20, "subject": "Task 20: Live gate on the docker rig (10 evidence items)", "status": "pending", "classification": "high-risk", "blockedBy": [18, 19], "note": "Includes the post-cutover oplog drain-under-churn check (evidence 10) that Task 1 cannot measure pre-cutover. Host-side sqlite3 on bind mounts; metrics on :8084 in-container."}, {"id": 21, "subject": "Task 21: Documentation truth pass", "status": "pending", "classification": "standard", "blockedBy": [20]} ], "knownFlakes": [ @@ -85,5 +102,5 @@ } ], "lastUpdated": "2026-07-19", - "phase2Status": "PLANNED - not started. Task 1 (rig soak) gates the rest." + "phase2Status": "PLANNED - not started. Reviewed + corrected 2026-07-19. Task 1 (rig soak) gates the rest." } From cf46e596805177a9b40de4ce42d8732647b0b78c Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sun, 19 Jul 2026 17:55:04 -0400 Subject: [PATCH 23/55] fix(rig): seed LDAP group mappings with the canonical role names docker/seed-sites.sh inserted the pre-rename 'Design' / 'Deployment' role strings, but the canonical vocabulary in Roles.cs is 'Designer' / 'Deployer'. The mismatch authorized nothing: on a freshly reseeded rig every Designer/Deployer-gated management command failed UNAUTHORIZED, including seed-sites.sh's own trailing `deploy artifacts` and reseed.sh's stage 6d encrypted-secret restore. Found while standing up the Phase 2 soak rig. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts --- docker/seed-sites.sh | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/docker/seed-sites.sh b/docker/seed-sites.sh index fa5c7102..2d2bc8eb 100755 --- a/docker/seed-sites.sh +++ b/docker/seed-sites.sh @@ -96,7 +96,7 @@ for ident in site-a site-b site-c; do done echo "" -echo "Seeding LDAP group mappings (Design + Deployment)..." +echo "Seeding LDAP group mappings (Designer + Deployer)..." # SecurityConfiguration.HasData declares 4 mappings but the InitialSchema # migration only inserts the Admin row, so a fresh ScadaBridgeConfig starts # with multi-role getting Admin only -- no Design and no Deployment access. @@ -106,11 +106,16 @@ docker exec -i scadabridge-mssql /opt/mssql-tools18/bin/sqlcmd \ -d ScadaBridgeConfig -Q " SET IDENTITY_INSERT LdapGroupMappings ON; IF NOT EXISTS (SELECT 1 FROM LdapGroupMappings WHERE Id = 2) - INSERT INTO LdapGroupMappings (Id, LdapGroupName, Role) VALUES (2, 'SCADA-Designers', 'Design'); + INSERT INTO LdapGroupMappings (Id, LdapGroupName, Role) VALUES (2, 'SCADA-Designers', 'Designer'); IF NOT EXISTS (SELECT 1 FROM LdapGroupMappings WHERE Id = 3) - INSERT INTO LdapGroupMappings (Id, LdapGroupName, Role) VALUES (3, 'SCADA-Deploy-All', 'Deployment'); + INSERT INTO LdapGroupMappings (Id, LdapGroupName, Role) VALUES (3, 'SCADA-Deploy-All', 'Deployer'); IF NOT EXISTS (SELECT 1 FROM LdapGroupMappings WHERE Id = 4) - INSERT INTO LdapGroupMappings (Id, LdapGroupName, Role) VALUES (4, 'SCADA-Deploy-SiteA', 'Deployment'); + INSERT INTO LdapGroupMappings (Id, LdapGroupName, Role) VALUES (4, 'SCADA-Deploy-SiteA', 'Deployer'); +-- Role strings MUST match the canonical vocabulary in +-- src/ZB.MOM.WW.ScadaBridge.Security/Roles.cs ('Designer' / 'Deployer'). +-- These rows previously carried the pre-rename 'Design' / 'Deployment', which +-- authorized nothing: every Designer/Deployer-gated management command failed +-- UNAUTHORIZED on a freshly reseeded rig. SET IDENTITY_INSERT LdapGroupMappings OFF; " From fc553bd9bab24e700de897edd8a671254b96587b Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Mon, 20 Jul 2026 01:06:59 -0400 Subject: [PATCH 24/55] =?UTF-8?q?docs(localdb):=20phase=202=20rig=20soak?= =?UTF-8?q?=20findings=20=E2=80=94=20GATE=20FAILS=20on=20a=20phase=201=20d?= =?UTF-8?q?efect?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Task 1 of the Phase 2 plan. The gate stops the plan, but not for the reason it anticipated: oplog sizing is fine and D6 is resolved. BLOCKER: the Phase 1 consolidated LocalDb (site-localdb.db) throws SQLite Error 10 'disk I/O error' on essentially every write on the ACTIVE node under sustained load — site_events, OperationTracking and the audit telemetry paths all fail, and site event logging silently drops events. Isolated to the load (not the node, not host-side observation) by failing over between nodes, and to LocalDb specifically (legacy store-and-forward.db / scadabridge.db in the same bind-mounted directory take zero errors under identical load). Phase 2 would register 8 more tables into that database — including the two highest-volume ones — while deleting the bespoke mechanisms that currently carry them. Must be root-caused first. Also recorded: D6's premise corrected (largest known production config_json is ~60-70 KB, not >128 KB — but MaxBatchSize 500 is still unsafe, use 16); D4's premise corrected (alarm writes are bounded by per-SourceReference coalescing at a 100 ms flush); sf_messages has a hard 50 rows/sec structural ceiling; and exceeding the oplog caps is a graceful snapshot-resync, not a failure. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts --- docs/plans/2026-07-19-localdb-phase2-soak.md | 277 +++++++++++++++++++ 1 file changed, 277 insertions(+) create mode 100644 docs/plans/2026-07-19-localdb-phase2-soak.md diff --git a/docs/plans/2026-07-19-localdb-phase2-soak.md b/docs/plans/2026-07-19-localdb-phase2-soak.md new file mode 100644 index 00000000..99c69c86 --- /dev/null +++ b/docs/plans/2026-07-19-localdb-phase2-soak.md @@ -0,0 +1,277 @@ +# LocalDb Phase 2 — rig soak findings + +**Run date:** 2026-07-19 / 2026-07-20 (UTC) · **Rig:** local 8-node docker cluster, site-a pair +**Task:** Task 1 of [`2026-07-19-localdb-adoption-phase2.md`](2026-07-19-localdb-adoption-phase2.md) + +> ## GATE VERDICT: **STOP — do not proceed to Task 3.** +> +> Not for the reason the plan anticipated. Oplog sizing is fine and D6 is resolved. The soak +> instead surfaced a **pre-existing Phase 1 defect**: the consolidated LocalDb database +> (`site-localdb.db`) throws `SQLite Error 10: 'disk I/O error'` on essentially every write on +> the **active** node under sustained concurrent load. Phase 2 moves eight more tables — +> including the two highest-volume ones — into exactly that database. See +> [Finding 1](#finding-1-blocker). + +--- + +## 1. Method as actually executed + +The plan's method needed four corrections before it would run. Recorded here so the next run +does not rediscover them. + +| Plan said | Reality | +|---|---| +| `docker exec … curl -s localhost:8084/metrics` | **No `curl` in the `aspnet:10.0` image.** Use a sidecar sharing the container netns: `docker run --rm --network container:scadabridge-site-a-a curlimages/curl:latest -s localhost:8084/metrics` | +| Sample with host-side `sqlite3` against the bind mounts | **Unsafe under live writes** — SQLite's `-shm` shared-memory index is not reliable across the Docker VM boundary. Ultimately *not* the cause of Finding 1 (proven below), but it should not be done against a database the container is actively writing. Copy the files first, or read counters from `/metrics`. | +| Drive alarm churn on a deployed instance | **Not possible on this rig.** `infra/mssql/seed-config.sql` seeds zero `TemplateNativeAlarmSources`, opc-plc runs with no alarm flags, and no simulator or harness exists anywhere in the repo. `native_alarm_state` stayed at 0 rows throughout. Bounded analytically instead — see [Finding 3](#finding-3). | +| Drive S&F churn via template 4's `TestExternalSystem` script | Template 4 exists after a reseed but is **not deployable** (34 pre-deployment validation errors: 30 `ConnectionBinding` + 4 `ScriptCompilation`). A purpose-built `SoakGenerator` template was used instead — see below. | + +### The generator that worked + +`ExternalSystem.Call` does **not** buffer to store-and-forward in practice; `ExternalSystem.CachedCall` +is the buffering surface. This is the single most important operational detail for reproducing +S&F load. + +- Template `SoakGenerator` (id 2021), one `Interval` script at `{"intervalMs":5000}`: + ```csharp + var parms = new Dictionary { ["a"] = 2, ["b"] = 3 }; + await ExternalSystem.CachedCall("Test REST API", "Add", parms); + ``` +- No attributes, no compositions, no connection bindings — deliberately, so it deploys cleanly. +- `ExternalSystemDefinitions` id 1 repointed to `http://127.0.0.1:9` (discard port → connection + refused → classified transient → buffered). +- 4 instances (`soakgen-1..4`) deployed to site-a. + +Sustained rate observed: ~**2.9 HTTP attempts/sec** (688–864 connection-refused per 4 min). + +### Two rig-tooling bugs found and fixed en route + +1. **`docker/seed-sites.sh` seeded stale role names** — `Design`/`Deployment` instead of the + canonical `Designer`/`Deployer` (`src/ZB.MOM.WW.ScadaBridge.Security/Roles.cs:46-47`). Every + Designer/Deployer-gated management command failed `UNAUTHORIZED` on a freshly reseeded rig, + including `seed-sites.sh`'s own trailing `deploy artifacts` and `reseed.sh` stage 6d. + **Fixed** (commit `cf46e596`). +2. **`infra/mssql/setup.sql` never executes.** It is mounted into + `/docker-entrypoint-initdb.d/`, a convention the official `mcr.microsoft.com/mssql/server` + image does not implement. After `reseed.sh` drops the volume (`docker compose down -v`), + nothing recreates `ScadaBridgeConfig` or the `scadabridge_app` login, so `reseed.sh` hangs + forever on its "Waiting for setup.sql to create ScadaBridgeConfig" poll. Worked around by + applying the three init scripts by hand. **NOT yet fixed in the repo.** + +--- + +## Finding 1 (BLOCKER) + +### The Phase 1 consolidated LocalDb fails under sustained write load on the active node + +`site-localdb.db` throws `SQLite Error 10: 'disk I/O error'` on essentially every write once the +active node is under concurrent load. Both Phase 1 tables and the audit telemetry paths are +affected. + +Representative stacks (`docker logs scadabridge-site-a-b`): + +``` +Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 10: 'disk I/O error'. + at ZB.MOM.WW.ScadaBridge.SiteEventLogging.SiteEventLogger.ProcessWriteQueueAsync() + SiteEventLogger.cs:line 221/236 + +Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 10: 'disk I/O error'. + at ZB.MOM.WW.ScadaBridge.SiteRuntime.Tracking.OperationTrackingStore.RecordEnqueueAsync(...) + OperationTrackingStore.cs:line 137 + at ...CachedCallTelemetryForwarder.TryEmitTrackingAsync(...) line 148 +``` + +User-visible symptom: `[ERR] Failed to record event: script from ScriptActor:SoakCall` — **site +event logging is silently dropping events on the floor under load.** + +#### It follows the load, not the node, not the observer + +The failure was isolated by moving the load between nodes: + +| Node | Role | Under load | `disk I/O error` in 4 min | +|---|---|---|---| +| site-a-a | active | yes | 2 175 | +| site-a-a | standby (after restart) | no | **0** | +| site-a-b | standby | no | **0** | +| site-a-b | active (after failover) | yes | **4 391** | + +#### It is LocalDb-specific, not the filesystem + +The decisive control. Under identical load, on the same node, in the same bind-mounted +directory, counting error-stack frames over 3 minutes: + +| Store | Backing file | Errors | +|---|---|---| +| `OperationTrackingStore` | `site-localdb.db` (LocalDb) | 13 044 | +| `SiteAuditTelemetryActor` | `site-localdb.db` (LocalDb) | 4 350 | +| `SiteEventLogger` | `site-localdb.db` (LocalDb) | 900 | +| `CachedCallTelemetryForwarder` | `site-localdb.db` (LocalDb) | 162 | +| `StoreAndForwardStorage` | `store-and-forward.db` (legacy) | **0** | +| `SiteStorageService` | `scadabridge.db` (legacy) | **0** | + +Ordinary SQLite on the same bind mount is completely healthy. Only the LocalDb-managed +database fails. + +#### Ruling out the observer + +Onset (04:56:37) was one second after a host-side `sqlite3` sample (04:56:36), which made +observer-induced `-shm` corruption the leading hypothesis. It is **excluded**: after node-a was +restarted (fresh open, recovered shm) and the load failed over to node-b, node-b began erroring +at a *higher* rate while no host process touched its files, and node-a — whose files *had* been +sampled — went to zero once it stopped carrying load. The variable that tracks the errors is +**load**, not host access. + +#### Secondary defect, same area + +``` +[ERROR][akka://scadabridge/user/site-audit-telemetry] There is no active ActorContext, +this is most likely due to use of async operations from within this actor. +Cause: System.NotSupportedException +``` + +`SiteAuditTelemetryActor` is closing over `ActorContext` across an `await`. Likely a +contributing cause rather than a separate issue — it is in the same write path — but it is a +real bug on its own terms. + +#### Why this blocks Phase 2 + +Phase 2 registers **eight further tables** into this database, including `native_alarm_state` +(the highest-volume table in either DB) and `sf_messages`. It also **deletes** the bespoke +mechanisms (`SiteReplicationActor`, `ReplicationService`) that currently carry that data +independently of LocalDb. Cutting over onto a store that cannot absorb the *current* write +load — and doing so in the same commit that removes the fallback — would convert a logging +defect into site-wide config and buffer loss. + +This is a Phase 1 defect. It must be root-caused and fixed before Phase 2's Task 3. + +--- + +## Finding 2 — D6 resolved: no stop condition, but `MaxBatchSize` must be lowered + +The plan asserts `deployed_configurations.config_json` is "documented to exceed 128 KB per row." +That misreads the source. `docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md` +says the *escaped Akka envelope* exceeded the 128 KB frame, and that the default serializer +double-escapes the payload, so **"the raw flattened JSON only needs to be ~60-70 KB to blow the +128 KB frame."** + +So the largest known real production `config_json` is on the order of **60–70 KB**. + +Measured on the rig (4 deployed configs): **max 715 B, avg 714 B** — trivially small, as the plan +predicted, hence the production figure above is the one to size against. (A direct measurement +against wonder was attempted; `wonder-app-vd03.zmr.zimmer.com` resolves over the VPN but the +servecli SSH service on :2222 refuses connections, so the box was unreachable.) + +**Verdict: no stop condition.** Nothing approaches the 4 MB single-row ceiling. + +**But the batching risk is real.** Batching is row-count-only (`MaxBatchSize` default **500**), +and neither side configures gRPC message limits, so the 4 MB default receive cap applies: + +``` +500 rows × 70 KB ≈ 35 MB ≫ 4 MB → poison batch, stream wedged +``` + +Recommended for Task 19: + +``` +LocalDb:Replication:MaxBatchSize = 16 +``` + +`16 × 128 KB = 2 MB` — 2× headroom on row size over the known worst case, and 2× headroom +against the 4 MB cap. + +--- + +## Finding 3 — D4 corrected: alarm write rate is bounded, not unbounded + +The plan calls `native_alarm_state` "unbounded by design." It is not. +`NativeAlarmActor.MarkDirtyUpsert` (`NativeAlarmActor.cs:473-502`) coalesces into a dictionary +**keyed by `SourceReference`** and flushes on a timer (`_persistFlushInterval`, default +**100 ms**). One flush writes at most one row per distinct source reference, regardless of how +many transitions occurred in that window. + +``` +worst-case rows/sec = distinct_source_refs × 10 +``` + +An alarm storm on N sources costs N rows per 100 ms flush, not N × transition-rate. This makes +the table analytically sizeable without a generator — which matters, because this rig cannot +produce alarm load at all (see §1). + +**Not empirically validated.** `native_alarm_state` held 0 rows for the entire run. + +--- + +## Finding 4 — `sf_messages` has a hard structural ceiling of 50 rows/sec + +Confirmed by code rather than measurement, which is stronger here. The retry sweep takes at most +`SweepBatchLimit` messages every `RetryTimerInterval`, and each failed attempt is one `UPDATE` +incrementing `retry_count`: + +``` +SweepBatchLimit (500) ÷ RetryTimerInterval (10 s) = 50 row-writes/sec, hard ceiling +``` + +This confirms the plan's "~50 row-writes/sec worst case" — and it is a ceiling, not an estimate. +`DefaultMaxRetries = 50` at `DefaultRetryInterval = 30 s` bounds each message to 50 updates over +25 minutes. + +Observed during the run: ~2.9 attempts/sec, far below the ceiling. Row counts could not be +sampled reliably (see §1) and the run was cut short by Finding 1. + +--- + +## Finding 5 — exceeding the oplog caps is a graceful degradation, not a failure + +The plan's Task 1 step 7 treats "the shared oplog cannot absorb this write profile" as a hard +stop requiring the keyed-instances escape hatch. It is much weaker than that. + +`OplogStore.EnforceCapsAsync` (`OplogStore.cs:109-138`) prunes to the ceiling and sets +`needs_snapshot`; `MaintenanceBackgroundService.cs:57` logs *"Oplog backlog/age caps exceeded: +pruned to the ceiling and flagged needs_snapshot — the peer must snapshot-resync."* +`SyncSession.ComputeSnapshotRequiredAsync` (`:413-415`) then forces a snapshot resync. + +So overrunning the caps costs a **full snapshot resync**, not data loss and not a wedged stream. +The caps therefore express *"how long may a peer be absent before it needs a full resync"*, and +should be sized to the longest tolerable peer outage rather than treated as a correctness +boundary. In healthy two-node operation the oplog drains continuously — `localdb_oplog_depth` +read **0** throughout. + +### Provisional cap recommendation (Task 19) + +Sized for a ~3-hour peer outage at a conservative 100 rows/sec aggregate, pending re-measurement +after Finding 1 is fixed: + +``` +LocalDb:Replication:MaxOplogRows = 1000000 # default; ≈2.8 h at 100 rows/s +LocalDb:Replication:MaxOplogAge = 3.00:00:00 +LocalDb:Replication:MaxBatchSize = 16 # Finding 2 — this one is NOT optional +``` + +Only `MaxBatchSize` is firmly evidence-backed. The other two rest on an assumed aggregate write +rate that this run could not measure. + +--- + +## 2. What still owes measurement + +Carry into the re-run once Finding 1 is fixed: + +1. Sustained `sf_messages` rows/sec under a saturating generator (needs many more instances or a + shorter interval to approach the 50/s ceiling). +2. Any `native_alarm_state` measurement at all — requires building alarm-source seeding plus an + A&C-capable server. The `OpcUaAlarmLiveSmokeTests` **passed**, so the rig's opc-plc *does* + answer ConditionRefresh with a `SnapshotComplete`; the missing piece is ongoing transitions + and a seeded `TemplateNativeAlarmSource`. (The test's own doc comment claiming the simulator + "does not reliably expose A&C" is stale.) +3. A production-representative `config_json` from wonder, to replace the ~60–70 KB inference. +4. The empirical oplog drain-under-churn check — Task 20 evidence item 10. + +## 3. Rig state left behind + +- Fully reseeded (central config volume dropped and replayed; site SQLite state wiped by + `reseed.sh` stage 2). +- `ExternalSystemDefinitions` id 1 is **repointed to `http://127.0.0.1:9`** — restore to + `http://scadabridge-restapi:5200` before using the rig for anything else. +- Template `SoakGenerator` (2021) and instances `soakgen-1..4` (ids 5–8) remain deployed on + site-a and are **still generating load**. Undeploy or delete them before the Task 20 live gate. +- `LdapGroupMappings` corrected in the live DB to the canonical role names. From 82c869a1df9b1777feefcc6696be1e81feb75f52 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Mon, 20 Jul 2026 01:07:18 -0400 Subject: [PATCH 25/55] docs(localdb): record the phase 2 task-1 gate verdict in the task record Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts --- docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json b/docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json index 36401179..de5b004b 100644 --- a/docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json +++ b/docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json @@ -102,5 +102,5 @@ } ], "lastUpdated": "2026-07-19", - "phase2Status": "PLANNED - not started. Reviewed + corrected 2026-07-19. Task 1 (rig soak) gates the rest." + "phase2Status": "BLOCKED at the Task 1 gate (2026-07-20). Task 1 ran and its verdict is STOP - see docs/plans/2026-07-19-localdb-phase2-soak.md. The blocker is NOT oplog sizing (fine) or D6 (resolved, no stop condition): the Phase 1 consolidated LocalDb (site-localdb.db) throws SQLite Error 10 'disk I/O error' on essentially every write on the ACTIVE node under sustained load, affecting site_events + OperationTracking + audit telemetry, and silently dropping site events. Isolated to load (not node, not host-side observation) by failing over between nodes, and to LocalDb specifically (legacy store-and-forward.db/scadabridge.db in the same bind mount take ZERO errors under identical load). Phase 2 would register 8 more tables into that DB - incl. the two highest-volume - while deleting the bespoke fallbacks, so this Phase 1 defect must be root-caused and fixed before Task 3. Tasks 2-21 remain pending and untouched; no plan code has been written. Secondary defect in the same path: SiteAuditTelemetryActor closes over ActorContext across an await (NotSupportedException). Plan-premise corrections now recorded in the soak doc: D6 (largest known production config_json ~60-70 KB not >128 KB, but MaxBatchSize must drop 500 -> 16), D4 (alarm writes bounded by per-SourceReference coalescing at a 100 ms flush), sf_messages hard ceiling 50 rows/sec (SweepBatchLimit/RetryTimerInterval), and oplog cap overrun = graceful snapshot resync, not failure. Rig caveats: ExternalSystemDefinitions id 1 still repointed to http://127.0.0.1:9 and SoakGenerator instances 5-8 still generating load - clean up before Task 20." } From e9e11d635eba93e543ab273e3da30390f596e4da Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Mon, 20 Jul 2026 01:11:08 -0400 Subject: [PATCH 26/55] docs(known-issues): root-cause brief for the LocalDb disk-I/O-error-under-load defect Standalone handoff for a follow-up investigation. Records the symptom, a full reproduction (including the two rig-tooling blockers and the CachedCall-vs-Call detail needed to generate load at all), the isolation evidence, a code map of the LocalDb connection model, four ranked hypotheses, and acceptance criteria. Not root-caused. Highest-value next step identified: capture the SQLite EXTENDED result code (logs only carry the generic primary code 10), and run the load with LocalDb:Path off the bind mount to partition environmental vs library causes. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts --- ...-07-20-localdb-disk-io-error-under-load.md | 315 ++++++++++++++++++ 1 file changed, 315 insertions(+) create mode 100644 docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md diff --git a/docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md b/docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md new file mode 100644 index 00000000..a5ff583a --- /dev/null +++ b/docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md @@ -0,0 +1,315 @@ +# LocalDb throws `SQLite Error 10: 'disk I/O error'` on the active site node under sustained write load + +**Date:** 2026-07-20 · **Status:** OPEN — not root-caused · **Severity:** High (silent event loss; blocks LocalDb Phase 2) +**Area:** `ZB.MOM.WW.LocalDb` (library, `~/Desktop/scadaproj/ZB.MOM.WW.LocalDb/`) as consumed by ScadaBridge site nodes +**Found by:** the Phase 2 rig soak — [`docs/plans/2026-07-19-localdb-phase2-soak.md`](../plans/2026-07-19-localdb-phase2-soak.md) +**Branch:** `feat/localdb-phase2` (defect is in Phase 1 code, present on `feat/localdb-phase1`) + +> **This document is a root-cause brief.** The investigation below establishes *what* fails and +> rules out several explanations. It does **not** identify the mechanism. Nothing has been fixed. + +--- + +## 1. Summary + +On a ScadaBridge site node, once the node is **active** and under sustained concurrent write +load, effectively every write to the consolidated LocalDb database (`site-localdb.db`) fails +with: + +``` +Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 10: 'disk I/O error'. +``` + +Observed rate: **~1 000–1 500 failures per minute**, sustained, not transient. The node stays up +and reports healthy. Ordinary (non-LocalDb) SQLite databases in the same directory, in the same +process, under the same load, are completely unaffected. + +## 2. Why it matters + +1. **Silent data loss today.** `SiteEventLogger` fails its inserts and logs + `[ERR] Failed to record event: script from ScriptActor:…`. Site event logging is dropping + events on the floor on the active node whenever the site is busy. `OperationTracking` writes + fail too, which breaks cached-call status tracking (`Cached-telemetry drain: no tracking + snapshot for …; skipping`). +2. **It blocks LocalDb Phase 2.** Phase 2 registers eight further tables into this same + database — including `native_alarm_state` (highest-volume table on the node) and + `sf_messages` — **and deletes the bespoke mechanisms that currently carry that data** + (`SiteReplicationActor`, `StoreAndForward.ReplicationService`) in the same commit. Cutting + over onto this store while removing the fallback would convert a logging defect into config + and buffer loss. +3. Phase 1 was previously live-gated as PASS. That gate exercised correctness and convergence, + **not sustained write load** — which is why this was not caught. + +## 3. Exact symptom + +Two representative stacks, both from `docker logs scadabridge-site-a-b` while that node was +active and under load: + +``` +Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 10: 'disk I/O error'. + at Microsoft.Data.Sqlite.SqliteDataReader.NextResult() + at Microsoft.Data.Sqlite.SqliteCommand.ExecuteReader(CommandBehavior behavior) + at ZB.MOM.WW.ScadaBridge.SiteEventLogging.SiteEventLogger.<>c__DisplayClass15_0.b__0(SqliteConnection connection) + in /src/src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs:line 236 + at ZB.MOM.WW.ScadaBridge.SiteEventLogging.SiteEventLogger.ProcessWriteQueueAsync() + in /src/src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs:line 221 +``` + +``` +Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 10: 'disk I/O error'. + at Microsoft.Data.Sqlite.SqliteDataReader.NextResult() + at Microsoft.Data.Sqlite.SqliteCommand.ExecuteNonQuery() + at ZB.MOM.WW.ScadaBridge.SiteRuntime.Tracking.OperationTrackingStore.RecordEnqueueAsync(...) + in /src/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Tracking/OperationTrackingStore.cs:line 137 + at ZB.MOM.WW.ScadaBridge.AuditLog.Site.Telemetry.CachedCallTelemetryForwarder.TryEmitTrackingAsync(...) + in /src/src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/Telemetry/CachedCallTelemetryForwarder.cs:line 148 +``` + +Note both fail inside `SqliteDataReader.NextResult()` — i.e. at statement execution, not at +`Open()`. Connections are being acquired successfully; the failure is on the write itself. + +### Error-source distribution + +Error-stack frames counted over one 3-minute window on the loaded node: + +| Store | Backing file | Frames | +|---|---|---| +| `OperationTrackingStore` | `site-localdb.db` (**LocalDb**) | 13 044 | +| `SiteAuditTelemetryActor` | `site-localdb.db` (**LocalDb**) | 4 350 | +| `SiteEventLogger` | `site-localdb.db` (**LocalDb**) | 900 | +| `CachedCallTelemetryForwarder` | `site-localdb.db` (**LocalDb**) | 162 | +| `StoreAndForwardStorage` | `store-and-forward.db` (legacy) | **0** | +| `SiteStorageService` | `scadabridge.db` (legacy) | **0** | + +## 4. Evidence — what has been established + +### 4.1 It tracks the load, not the node + +The load was moved between the two site-a nodes by restarting the active one (the surviving node +becomes oldest-up and takes over): + +| Node | Role | Under load | `disk I/O error` / 4 min | +|---|---|---|---| +| site-a-a | active | yes | 2 175 | +| site-a-a | standby (after restart) | no | **0** | +| site-a-b | standby | no | **0** | +| site-a-b | active (after failover) | yes | **4 391** | + +### 4.2 It is LocalDb-specific, not the filesystem or the bind mount + +This is the strongest signal. `store-and-forward.db` and `scadabridge.db` live in the **same +bind-mounted directory** (`/app/data`, host `docker/site-a-node-*/data/`), are opened by the +**same process**, are also **WAL-mode**, and are being written **concurrently under the same +load** — and they log zero errors. Only the LocalDb-managed file fails. + +### 4.3 The observer has been ruled out + +Onset (04:56:37) was **one second after** a host-side `sqlite3` read of the bind-mounted +database (04:56:36), making observer-induced `-shm` corruption the leading hypothesis. It is +excluded: + +- After node-a was restarted (fresh open, `-shm` recovered) and load failed over to node-b, + **node-b** — whose files no host process had touched since a single baseline read, and which + had been error-free for the entire preceding period — began erroring immediately at a *higher* + rate. +- **node-a**, whose files *had* been sampled, dropped to zero once it stopped carrying load. + +The variable that tracks the errors is load. (Host-side `sqlite3` against a live WAL database +over a bind mount is still unsafe and should be avoided — it is just not the cause here.) + +### 4.4 Not disk pressure + +Host had 215 GiB free throughout (`df -h`: 76 % used on the data volume). Files are small: +`site-localdb.db` 188 KiB, WAL peaked around 4.1 MiB then checkpointed to 0. + +## 5. Reproduction + +Fully reproducible in ~10 minutes on the local docker rig. + +### 5.1 Rig prerequisites + +Two rig-tooling bugs will block a fresh reseed; both are documented in the soak findings: + +- `docker/seed-sites.sh` role names — **already fixed** (commit `cf46e596`). +- **`infra/mssql/setup.sql` never executes** — still broken. It is mounted into + `/docker-entrypoint-initdb.d/`, which the official `mcr.microsoft.com/mssql/server` image does + not implement. After `infra/reseed.sh` drops the volume, nothing creates `ScadaBridgeConfig` or + the `scadabridge_app` login and `reseed.sh` hangs forever on its setup.sql poll. Work around + by applying the init scripts by hand once MSSQL is accepting connections: + + ```bash + cd ~/Desktop/ScadaBridge/infra + for f in mssql/setup.sql mssql/machinedata_seed.sql mssql/setup-env2.sql; do + docker exec -i scadabridge-mssql /opt/mssql-tools18/bin/sqlcmd \ + -S localhost -U sa -P 'ScadaBridge_Dev1#' -C -b < "$f" + done + ``` + Then restart the app containers so EF migrations run, and restart central again after + `seed-sites.sh` writes `LdapGroupMappings` (they are cached at startup). + +### 5.2 Build the load generator + +**The seeded `Motor Controller` template (id 4) cannot be used** — it fails pre-deployment +validation with 34 errors (30 `ConnectionBinding`, 4 `ScriptCompilation`). Build a minimal one. + +**Critical:** `ExternalSystem.Call` does **not** buffer to store-and-forward in practice. +`ExternalSystem.CachedCall` is the buffering surface. Using `Call` produces HTTP traffic and no +S&F rows, and will not reproduce this. + +```bash +cd ~/Desktop/ScadaBridge +SB=src/ZB.MOM.WW.ScadaBridge.CLI/bin/Debug/net10.0/scadabridge # dotnet build src/...CLI first +AUTH="--url http://localhost:9000 --username multi-role --password password" + +# 1. Point the seeded external system at a refusing address (discard port). +$SB $AUTH external-system update --id 1 --name "Test REST API" \ + --endpoint-url "http://127.0.0.1:9" --auth-type ApiKey --auth-config "scadabridge-test-key-1" + +# 2. Minimal template: no attributes, no compositions, no connection bindings. +$SB $AUTH --format json template create --name "SoakGenerator" # -> note the id + +$SB $AUTH --format json template script add --template-id --name "SoakCall" \ + --trigger-type Interval --trigger-config '{"intervalMs":5000}' \ + --code 'var parms = new Dictionary { ["a"] = 2, ["b"] = 3 }; await ExternalSystem.CachedCall("Test REST API", "Add", parms);' + +# 3. Four instances on site-a (site id 1), then deploy each. +for i in 1 2 3 4; do + $SB $AUTH --format json instance create --name "soakgen-$i" --template-id --site-id 1 +done +$SB $AUTH instance deploy --id +``` + +Note the CLI's `template script update` requires `--name` and `--trigger-type` even when only +changing `--code`. In zsh, do not put the auth flags in an unquoted variable — zsh does not +word-split, so pass them literally or use `${=AUTH}`. + +### 5.3 Observe + +```bash +# Identify the ACTIVE node — it is the one running the ScriptActors. +docker logs --since 4m scadabridge-site-a-a 2>&1 | grep -c "Connection refused" +docker logs --since 4m scadabridge-site-a-b 2>&1 | grep -c "Connection refused" + +# Errors appear on that node within ~2 minutes of load starting. +docker logs --since 4m scadabridge-site-a- 2>&1 | grep -c "disk I/O error" +``` + +Metrics (port 8084 is **not** published, and the `aspnet:10.0` image has **no `curl`**) — use a +sidecar in the container's network namespace: + +```bash +docker run --rm --network container:scadabridge-site-a-a curlimages/curl:latest \ + -s localhost:8084/metrics | grep '^localdb_' +``` + +Do **not** query the databases with host-side `sqlite3` while containers are writing them. + +## 6. Code map + +### Library — `~/Desktop/scadaproj/ZB.MOM.WW.LocalDb/src/ZB.MOM.WW.LocalDb/Internal/SqliteLocalDb.cs` + +Facts relevant to the failure: + +- **A `_master` connection is held open for the object's entire lifetime** (`:31`), explicitly to + "anchor the WAL journal". It is guarded by a `Lock _masterLock` because `SqliteConnection` is + not thread-safe. +- **`CreateConnection()` (`:83`) opens a brand-new `SqliteConnection` per call** — one per + operation, from many concurrent actors. Every call then runs + `PRAGMA synchronous=…; PRAGMA busy_timeout=…; PRAGMA foreign_keys=ON;` and registers a UDF: + ```csharp + conn.CreateFunction("zb_hlc_next", () => _clock.Next()); + ``` +- The connection string is **only** `DataSource=` (`:57`) — **connection pooling is left at + the Microsoft.Data.Sqlite default (enabled)**, and no `Cache=` or `Mode=` is set. +- Effective options on the rig are the defaults: `BusyTimeoutMs = 5000`, `Synchronous = NORMAL`. + ScadaBridge's rig config (`docker/site-a-node-*/appsettings.Site.json`, `LocalDb` section) sets + only `Path` and the replication block. +- `zb_hlc_next()` is invoked **from inside the capture triggers**, i.e. on the SQLite thread + during every INSERT/UPDATE/DELETE on a registered table, and it calls into the shared + `HybridLogicalClock` from arbitrary threads. + +### Failing call sites (ScadaBridge) + +- `src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs:221,236` — a channel-drained + single-writer loop (`ProcessWriteQueueAsync`) using a `WithConnection(...)` helper. +- `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Tracking/OperationTrackingStore.cs:137` (`RecordEnqueueAsync`), + `:260,266` (`GetStatusAsync`). +- `src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/Telemetry/CachedCallTelemetryForwarder.cs:148`. +- `src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/Telemetry/SiteAuditTelemetryActor.cs` — also see §8. + +## 7. Hypotheses, ranked + +None verified. Ordered by how well they fit "LocalDb only, load-dependent, same directory as +healthy WAL databases". + +1. **Connection churn × pooling × per-connection UDF registration.** LocalDb opens a fresh + `SqliteConnection` per operation with pooling enabled, and calls `CreateFunction` on every + acquisition. Under high concurrency this drives far more open/close and `-shm` mapping churn + than the legacy stores (which reuse a small number of connections), and is the clearest + structural difference between the failing and healthy databases. Suspect the interaction of + the pool with the long-lived `_master` connection and WAL index growth. +2. **`-shm` / WAL-index growth over the bind mount, triggered only at LocalDb's concurrency.** + Would explain why the same mount is fine for lower-concurrency databases. `mmap` of the shared + WAL index across virtiofs is a known-fragile area. **Distinguishing test: run the same load + with `LocalDb:Path` pointed at a container-local path (a `tmpfs` or a plain volume rather than + the bind mount).** If the errors vanish, this is confirmed and the fix is environmental / + deployment-shaped rather than a library bug. **Run this test first — it is cheap and it + partitions the hypothesis space.** +3. **`zb_hlc_next` UDF failing inside a trigger.** An exception thrown out of the managed UDF + callback during trigger execution can surface as a generic SQLite error at the statement + level. Check `HybridLogicalClock.Next()` for thread-safety and for anything that can throw + under contention (e.g. a spin/overflow path when many callers request stamps in the same + millisecond). +4. **Busy-timeout exhaustion misreported.** `BusyTimeoutMs = 5000` with heavy multi-connection + write contention on one file. This would normally surface as `SQLITE_BUSY` (5), not + `SQLITE_IOERR` (10), so it is a weaker fit — but worth excluding. + +### The single highest-value next step + +**Capture the extended result code.** The logs only show the primary code (`10` = `SQLITE_IOERR`), +which is generic. `SqliteException.SqliteExtendedErrorCode` names the failing syscall and would +likely settle this outright: + +| Extended code | Meaning | Points at | +|---|---|---| +| `SQLITE_IOERR_SHMMAP` (6154) / `SQLITE_IOERR_SHMSIZE` (4874) | WAL index mmap/resize failed | hypothesis 2 | +| `SQLITE_IOERR_WRITE` (778) / `SQLITE_IOERR_FSYNC` (1034) | plain write/fsync failed | filesystem | +| `SQLITE_IOERR_LOCK` (3850) | file locking failed | bind mount locking | + +Add the extended code to the exception logging (or attach a debugger / run the repro against a +local non-container build) before pursuing any fix. + +## 8. Secondary defect in the same path + +``` +[ERROR][akka://scadabridge/user/site-audit-telemetry] There is no active ActorContext, +this is most likely due to use of async operations from within this actor. +Cause: System.NotSupportedException +``` + +`SiteAuditTelemetryActor` is touching `Context` (or `Self`/`Sender`) after an `await`. This is a +real bug independent of the I/O errors, though it sits in the same write path and may be +contributing. Note the family-wide rule already recorded for Akka work: never read `Self`/`Context` +after an `await` inside an actor. + +## 9. What a fix must satisfy + +1. The §5 repro runs for **30 minutes under sustained load with zero `disk I/O error`** on the + active node. +2. No `Failed to record event` errors — site events are durably written under load. +3. `localdb_oplog_depth` rises under load and **drains** between bursts; zero dead letters. +4. Replication still converges across the site-a pair (Phase 1's existing convergence suite and + live gate still pass). +5. A regression test that would have caught this — i.e. a **concurrent-write load test** against + a real LocalDb file, not just the correctness/convergence tests Phase 1 shipped. Phase 1's + gate passed precisely because no test applied sustained concurrent write pressure. + +## 10. Rig state as left + +- Rig fully reseeded; central config volume dropped and replayed; site SQLite state wiped + (`reseed.sh` stage 2 does `rm -rf docker/site-*/data/*`). +- `ExternalSystemDefinitions` id 1 is **still repointed to `http://127.0.0.1:9`** — restore to + `http://scadabridge-restapi:5200` when done. +- Template `SoakGenerator` (id 2021) and instances `soakgen-1..4` (ids 5–8) are **still deployed + and still generating load** on site-a. +- `LdapGroupMappings` corrected in the live DB to the canonical `Designer`/`Deployer` names. From 8652eab98e933fc670bf13105e38c5c46958ab0a Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Mon, 20 Jul 2026 02:01:00 -0400 Subject: [PATCH 27/55] fix(localdb): root-cause the soak disk-I/O failure + ship the hardening follow-ups The Phase 2 soak's "LocalDb fails under load" blocker is NOT a product defect. Root cause: host-side (macOS) sqlite3 reads of the live, bind-mounted WAL databases. POSIX advisory locks do not propagate across the virtiofs boundary, so the host reader believes it is the only connection, checkpoints on close and resets the WAL to 0 bytes under the container. The container's still-mapped WAL index then references frames that no longer exist and every subsequent statement fails SQLITE_IOERR_SHORT_READ (522) -> primary code 10, permanently until the process reopens the database. Reproduced on demand both on the rig (one sqlite3 SELECT reset a 4.6 MiB WAL and produced the first error one second later) and in a minimal python:3.12-alpine repro with no LocalDb or .NET involved. Refuted the converse: a freshly-reopened node sustained the full soak load 10+ minutes with zero errors. The original brief's isolation was confounded - both nodes had been poisoned by the same sampling pass, and a poisoned standby looks healthy only because it issues almost no statements. Corrections annotated in place. Hardening shipped alongside: - SqliteErrorCodes.Describe: log SQLite primary AND extended codes at the LocalDb-adjacent catch sites (the missing extended code is what made the original diagnosis so slow). - SiteAuditTelemetryActor: stop touching ActorContext across an await (NotSupportedException), with regression coverage. - infra/reseed.sh: apply the MSSQL init scripts explicitly. The official mssql/server image does not implement /docker-entrypoint-initdb.d, so a fresh volume hung the reseed forever; compose mounts annotated as informational. Deliberately NOT done: detect-and-reopen self-heal in SqliteLocalDb. It defends only against external interference, which is now prevented at the source, and same-kernel production readers see the locks correctly. Build 0 warnings; SiteAuditTelemetryActorTests 9/9, SiteEventLogging 70/70. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts --- ...-07-20-localdb-disk-io-error-under-load.md | 184 ++++++++++++++++-- .../2026-07-19-localdb-adoption-phase2.md | 35 +++- ...7-19-localdb-adoption-phase2.md.tasks.json | 6 +- docs/plans/2026-07-19-localdb-phase2-soak.md | 46 +++-- infra/docker-compose.yml | 3 + infra/reseed.sh | 15 +- .../Telemetry/CachedCallTelemetryForwarder.cs | 8 +- .../Site/Telemetry/SiteAuditTelemetryActor.cs | 37 +++- .../Site/Telemetry/SqliteErrorCodes.cs | 31 +++ .../SiteEventLogger.cs | 23 ++- .../Telemetry/SiteAuditTelemetryActorTests.cs | 52 +++++ 11 files changed, 378 insertions(+), 62 deletions(-) create mode 100644 src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/Telemetry/SqliteErrorCodes.cs diff --git a/docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md b/docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md index a5ff583a..6be51122 100644 --- a/docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md +++ b/docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md @@ -1,12 +1,106 @@ # LocalDb throws `SQLite Error 10: 'disk I/O error'` on the active site node under sustained write load -**Date:** 2026-07-20 · **Status:** OPEN — not root-caused · **Severity:** High (silent event loss; blocks LocalDb Phase 2) +**Date:** 2026-07-20 · **Status:** ROOT-CAUSED + FIX PASS COMPLETE 2026-07-20 (same day) — observer-induced, **not a LocalDb defect**; see §0 (cause) and §11a (fixes) · **Severity:** was High; resolved to an operational rule (now enforced in the tooling docs) + shipped hardening **Area:** `ZB.MOM.WW.LocalDb` (library, `~/Desktop/scadaproj/ZB.MOM.WW.LocalDb/`) as consumed by ScadaBridge site nodes **Found by:** the Phase 2 rig soak — [`docs/plans/2026-07-19-localdb-phase2-soak.md`](../plans/2026-07-19-localdb-phase2-soak.md) -**Branch:** `feat/localdb-phase2` (defect is in Phase 1 code, present on `feat/localdb-phase1`) +**Branch:** `feat/localdb-phase2` (symptom observed on Phase 1 code) -> **This document is a root-cause brief.** The investigation below establishes *what* fails and -> rules out several explanations. It does **not** identify the mechanism. Nothing has been fixed. +> **Update 2026-07-20:** the mechanism has been identified and reproduced on demand, both in a +> minimal SQLite-only repro and on the live rig, and the follow-up fixes have shipped. +> Sections §0, §11a and §11 below are authoritative; the original brief (§1–§10) is preserved +> as written, with corrections annotated where its conclusions did not survive +> (§4.2, §4.3, §7, §9) and fix notes where they did (§5.1, §8). + +--- + +## 0. ROOT CAUSE (verified) + +**A host-side (macOS) `sqlite3` read of a live, bind-mounted WAL database checkpoints and +resets the WAL out from under the container process, permanently poisoning that process's +connections.** LocalDb, its connection handling, its UDF, and its triggers are not involved — +the mechanism reproduces with plain Python `sqlite3` and no LocalDb code at all. + +Mechanism, step by step: + +1. POSIX advisory locks do **not** propagate across the Docker Desktop virtiofs bind-mount + boundary. A host `sqlite3` opening the database cannot see the container's locks (and vice + versa), so it believes it is the **only** connection. +2. On close, the "last" connection in WAL mode runs a full checkpoint and **resets the WAL to + 0 bytes**. Because the read happened while the container was idle (a standby node, or a gap + between write bursts), nothing blocks the checkpoint. This is exactly the file signature + found on both rig nodes: main DB mtime + 0-byte `-wal` stamped at the sampling minute. +3. The container process still holds the old WAL-index state (its per-inode `-shm` mapping, + kept alive forever by the held-open `_master` connection and the Microsoft.Data.Sqlite + connection pool). That index says the WAL contains N frames; the file now has none. Every + subsequent statement — reads and writes both consult the WAL index — fails with + **`SQLITE_IOERR_SHORT_READ` (extended code 522)**, surfaced as primary code 10 + `'disk I/O error'` (some paths surface `SQLITE_NOTADB` (26) instead). The poisoning is + **permanent until the process reopens the database** (restart). + +### Why the original brief's conclusions were wrong + +- **"It tracks the load, not the node" (§4.1)** — confounded. *Both* nodes were poisoned by + the 04:56 UTC host-side sampling (both nodes' `site-localdb.db` main files carry the 04:56 + mtime; node-b's WAL was left at 0 bytes). A poisoned **standby** shows zero errors only + because a standby issues ~zero LocalDb statements; the errors "followed the load" because + the load is what generates statements against an already-poisoned handle. Node-b's very + first write attempt after failover (04:59:37, `OperationTrackingStore.RecordAttemptAsync`) + failed — it had been poisoned for 3 minutes with nothing to say about it. +- **"It is LocalDb-specific" (§4.2)** — sampling-selection bias. The legacy WAL databases in + the same directory were healthy only because no host process ever read *them*. The minimal + repro poisons an arbitrary WAL database the same way. +- **"The observer has been ruled out" (§4.3)** — the exclusion assumed an error-free standby + was an unpoisoned standby. It wasn't; it was a poisoned node with no traffic. + +### Verification (2026-07-20, all on the live rig + minimal repro) + +1. **Load alone is harmless:** restarted the poisoned active node (site-a-b); freshly-reopened + site-a-a took the full soak load for **10+ minutes with zero errors** (the original model + predicted onset within ~2 min), WAL growing/checkpointing normally, DB 188 KiB → 476 KiB. +2. **One host read is sufficient and immediate:** a single + `sqlite3 docker/site-a-node-a/data/site-localdb.db "SELECT count(*) FROM site_events;"` + against the healthy loaded node reset its 4.6 MiB WAL to 0 bytes in place and produced the + first `disk I/O error` **one second later** (05:32:48 → 05:32:49), 203 errors in the next + 40 s — the same one-second onset correlation as the original 04:56:36 → 04:56:37 incident. +3. **Minimal repro (no LocalDb, no .NET):** a `python:3.12-alpine` container writing a + WAL-mode SQLite DB on a bind mount (held master connection + fresh connection per op, + `synchronous=NORMAL`, `busy_timeout=5000`). A host `sqlite3 "SELECT count(*)"`: + - against the **actively-writing** DB → immediate `SQLITE_IOERR_SHORT_READ` (522) + + `SQLITE_NOTADB` burst, then recovery (checkpoint could not fully reset a hot WAL); + - during an **idle window** (connections held open, WAL populated) → WAL reset + 1.2 MiB → 0 bytes, then **every fresh-connection write failed for the rest of the run + (200/200)** — the persistent variant, matching the rig. + +### Consequences + +1. **The operational rule in §5.3 ("do not query the databases with host-side `sqlite3`") is + the root cause, not a hygiene note.** One violation silently destroys the node's local + persistence until restart. This applies to *every* WAL SQLite file on the bind mount + (`scadabridge.db`, `store-and-forward.db` included), not just LocalDb. +2. **Safe inspection recipes:** copy the file triplet (`.db`, `-wal`, `-shm`) and open the + copy; or read from inside the container boundary (same kernel ⇒ locks visible), e.g. + `docker run --rm -v :/d alpine/sqlite3 sqlite3 /d/site-localdb.db "..."` — never the + macOS host against live files. +3. **LocalDb Phase 2 is unblocked** on this issue: the library sustained the full soak write + load indefinitely once nothing external touched its file. +4. Hardening follow-ups — **status as of the 2026-07-20 fix pass (see §11a):** + - **DONE — extended-code logging:** the LocalDb-adjacent catch sites (`SiteAuditTelemetryActor`, + `CachedCallTelemetryForwarder`, `SiteEventLogger`) now log + `SqliteException` primary/extended codes (`sqlite 10/522`-style) via + `SqliteErrorCodes.Describe` / `DescribeSqliteError`. + - **DONE — §8 async-context bug** (see §8). + - **DONE — §9.5 load regression test** (see §9). + - **NOT DONE (deliberately):** a detect-and-reopen self-heal in `SqliteLocalDb` for + persistent `SQLITE_IOERR`/`SQLITE_NOTADB`. This is a real library design change + (pool clear + master reopen + in-flight coordination) protecting against *external + interference only* — the trigger is operator/tooling action, now prevented at the + source, and on same-kernel production deployments external readers see the locks and + are safe. File as its own issue if production ever runs where a foreign-kernel reader + can touch the files. + +--- + +> **Original brief follows, preserved as written on 2026-07-20 before root-causing.** --- @@ -95,14 +189,21 @@ becomes oldest-up and takes over): | site-a-b | standby | no | **0** | | site-a-b | active (after failover) | yes | **4 391** | -### 4.2 It is LocalDb-specific, not the filesystem or the bind mount +### 4.2 It is LocalDb-specific, not the filesystem or the bind mount — **WRONG, see §0** + +> **Correction 2026-07-20:** sampling-selection bias — only the LocalDb file was ever read +> from the host. Any of these WAL databases is equally poisonable (minimal-repro-proven). This is the strongest signal. `store-and-forward.db` and `scadabridge.db` live in the **same bind-mounted directory** (`/app/data`, host `docker/site-a-node-*/data/`), are opened by the **same process**, are also **WAL-mode**, and are being written **concurrently under the same load** — and they log zero errors. Only the LocalDb-managed file fails. -### 4.3 The observer has been ruled out +### 4.3 The observer has been ruled out — **WRONG, see §0: the observer was the cause** + +> **Correction 2026-07-20:** the exclusion below assumed an error-free standby was an +> unpoisoned standby. Node-b's files carry the 04:56 sampling-time mtimes (WAL left at +> 0 bytes); it was poisoned then and merely silent until failover gave it write traffic. Onset (04:56:37) was **one second after** a host-side `sqlite3` read of the bind-mounted database (04:56:36), making observer-induced `-shm` corruption the leading hypothesis. It is @@ -131,11 +232,14 @@ Fully reproducible in ~10 minutes on the local docker rig. Two rig-tooling bugs will block a fresh reseed; both are documented in the soak findings: - `docker/seed-sites.sh` role names — **already fixed** (commit `cf46e596`). -- **`infra/mssql/setup.sql` never executes** — still broken. It is mounted into - `/docker-entrypoint-initdb.d/`, which the official `mcr.microsoft.com/mssql/server` image does - not implement. After `infra/reseed.sh` drops the volume, nothing creates `ScadaBridgeConfig` or - the `scadabridge_app` login and `reseed.sh` hangs forever on its setup.sql poll. Work around - by applying the init scripts by hand once MSSQL is accepting connections: +- **`infra/mssql/setup.sql` never executes** — **FIXED 2026-07-20**: `infra/reseed.sh` now + applies the three init scripts itself via `sqlcmd` once MSSQL accepts connections (the + `/docker-entrypoint-initdb.d/` compose mounts are informational only — the official + `mcr.microsoft.com/mssql/server` image does not implement that hook; noted in the compose + file). The manual workaround below is retained for historical context / older checkouts. + Original problem: after `infra/reseed.sh` dropped the volume, nothing created + `ScadaBridgeConfig` or the `scadabridge_app` login and `reseed.sh` hung forever on its + setup.sql poll. The by-hand equivalent: ```bash cd ~/Desktop/ScadaBridge/infra @@ -239,6 +343,11 @@ Facts relevant to the failure: ## 7. Hypotheses, ranked +> **Resolution 2026-07-20:** none of the four below is the cause. The mechanism is a variant +> of #2's territory (bind-mount `-shm`/WAL fragility) but triggered *only* by a host-side +> reader — LocalDb's concurrency, pooling, and UDF (hypotheses 1/3/4) are exonerated. The +> extended code, since captured, is `SQLITE_IOERR_SHORT_READ` (522). + None verified. Ordered by how well they fit "LocalDb only, load-dependent, same directory as healthy WAL databases". @@ -279,7 +388,7 @@ likely settle this outright: Add the extended code to the exception logging (or attach a debugger / run the repro against a local non-container build) before pursuing any fix. -## 8. Secondary defect in the same path +## 8. Secondary defect in the same path — **FIXED 2026-07-20** ``` [ERROR][akka://scadabridge/user/site-audit-telemetry] There is no active ActorContext, @@ -292,8 +401,32 @@ real bug independent of the I/O errors, though it sits in the same write path an contributing. Note the family-wide rule already recorded for Akka work: never read `Self`/`Context` after an `await` inside an actor. +> **Fixed 2026-07-20.** Root cause: both drain handlers await with `ConfigureAwait(false)`, so +> their `finally`-block re-arm (`ScheduleNext`/`ScheduleNextCached`) runs on a pool thread with +> no active ActorContext. Investigation found the failure is **bimodal**, and the second mode is +> worse than the logged one: depending on what the pool thread's thread-static cell slot holds, +> `Context`/`Self` either **throw** `NotSupportedException` (the logged variant — actor crashes +> and restarts once per drain) or **silently resolve a STALE cell of whatever actor last ran on +> that thread**, re-arming the tick at the *wrong actor* so the drain loop just stops (observed +> under TestKit: the tick landed on the TestActor). Fix: capture `Context.System.Scheduler` and +> `Self` into fields at construction (both are thread-safe immutable handles) and use only those +> from the re-arm path. Regression test +> `SiteAuditTelemetryActorTests.Drains_Whose_Awaits_Complete_Off_The_Actor_Thread_Keep_Draining_Without_Crashing` +> forces the mock awaits to complete off the actor thread — which every pre-existing test +> avoided by returning already-completed tasks — and catches **both** variants (EventFilter for +> the throw, sustained-drain counts for the silent stall). AuditLog suite 355/355 green. + ## 9. What a fix must satisfy +> **Resolution 2026-07-20:** criteria 1–4 are already satisfied by the unmodified code once no +> host process touches the live files — verified 10+ min of soak load with zero errors, events +> durably written, WAL checkpointing normally. Criterion 5 (a sustained concurrent-write load +> test) would **not** have caught this — the trigger is an external reader, not load — but is +> now in place anyway: `ConcurrentWriteLoadTests` in `ZB.MOM.WW.LocalDb.Tests` (8 concurrent +> writers × 250 inserts through fresh pooled connections against a registered/triggered table on +> a real file, with concurrent readers; asserts zero failures + exact row/oplog counts; suite +> 145/145). §8's `SiteAuditTelemetryActor` async-context bug is **fixed** — see §8. + 1. The §5 repro runs for **30 minutes under sustained load with zero `disk I/O error`** on the active node. 2. No `Failed to record event` errors — site events are durably written under load. @@ -313,3 +446,30 @@ after an `await` inside an actor. - Template `SoakGenerator` (id 2021) and instances `soakgen-1..4` (ids 5–8) are **still deployed and still generating load** on site-a. - `LdapGroupMappings` corrected in the live DB to the canonical `Designer`/`Deployer` names. + +## 11a. Fix pass (2026-07-20, same day — all verified) + +Everything actionable that this incident identified is now fixed (uncommitted on each repo's +current branch; ScadaBridge full solution builds clean, 0 warnings): + +| # | Issue | Fix | Verification | +|---|---|---|---| +| 1 | §8 `SiteAuditTelemetryActor` async-context bug (bimodal: crash-per-drain OR silent tick misroute) | Capture `Context.System.Scheduler` + `Self` at construction; re-arm path never reads thread-static context | New red→green regression test forcing off-actor-thread continuations; AuditLog suite 355/355 | +| 2 | Diagnostics gap: logs carried only the primary SQLite code | `SqliteErrorCodes.Describe` (AuditLog) + `DescribeSqliteError` (SiteEventLogging) — catch sites now log `sqlite /` | Builds clean; suites green (this gap cost the investigation a from-scratch repro to learn code 522) | +| 3 | §5.1 `reseed.sh` hangs forever waiting on the initdb hook the mssql image doesn't have | `reseed.sh` applies `setup.sql`/`machinedata_seed.sql`/`setup-env2.sql` itself via `sqlcmd`; compose mounts annotated as informational | `bash -n` clean; scripts verified idempotent (`IF NOT EXISTS` guards) | +| 4 | §9.5 missing concurrent-write load test | `ConcurrentWriteLoadTests` in `ZB.MOM.WW.LocalDb.Tests` (scadaproj) — 8 writers × 250 pooled-connection inserts on a registered table + concurrent readers, exact row/oplog count asserts | LocalDb suite 145/145 | +| 5 | Root cause itself (operator/tooling host reads) | Poisonous instructions removed from the Phase 2 plan + `.tasks.json` (safe `snap()` copy-based sampling); soak-doc verdict corrected; family-wide memory rule recorded | On-demand on/off reproduction, §0 | + +Deliberately **not** done: the `SqliteLocalDb` detect-and-reopen self-heal (see §0 +consequence 4 for the rationale and the condition under which to file it). + +## 11. Rig state after root-causing (2026-07-20 ~05:40 UTC) + +- Both site-a nodes restarted during verification, curing both poisonings. End state: + **site-a-b active** carrying the soak load, site-a-a standby, **zero `disk I/O error` on + both** under sustained load. +- The §10 items still stand: `ExternalSystemDefinitions` id 1 still points at + `http://127.0.0.1:9`, and `SoakGenerator` + `soakgen-1..4` are still deployed and + generating load — the Phase 2 soak can now proceed on a clean baseline. +- Minimal-repro scripts (`writer.py` burst variant, `writer2.py` idle-window variant) lived in + the session scratchpad; the recipe is fully described in §0 and takes ~2 minutes to rebuild. diff --git a/docs/plans/2026-07-19-localdb-adoption-phase2.md b/docs/plans/2026-07-19-localdb-adoption-phase2.md index ea5fd918..30fe5dc3 100644 --- a/docs/plans/2026-07-19-localdb-adoption-phase2.md +++ b/docs/plans/2026-07-19-localdb-adoption-phase2.md @@ -203,8 +203,23 @@ This is the gate item from `2026-07-19-localdb-phase2-gate.md` §5. It is measur > the oplog sizing arithmetically, and (b) that the Phase 1 oplog stays healthy alongside. The > *empirical* post-cutover drain check lives in Task 20 (evidence item 10). > -> Also: the containers have **no `sqlite3` binary** (bare `aspnet:10.0` image), but the data dirs -> are **host bind mounts** — run `sqlite3` on the host against `docker/site-a-node-*/data/`. +> Also: the containers have **no `sqlite3` binary** (bare `aspnet:10.0` image), and the data dirs +> are host bind mounts — but **NEVER run `sqlite3` on the macOS host against the live files**: +> host↔container POSIX locks do not propagate across the virtiofs boundary, and a single host-side +> read checkpoints + resets the WAL out from under the container, permanently poisoning its +> connections (`SQLITE_IOERR_SHORT_READ` → `disk I/O error` storm until restart). This is the +> **root cause of the 2026-07-20 "disk I/O error under load" incident** — see +> [`docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md`](../known-issues/2026-07-20-localdb-disk-io-error-under-load.md) §0. +> Safe sampling: copy the file triplet first (`cp x.db x.db-wal x.db-shm /tmp/…` — plain `cp` +> takes no SQLite locks and cannot harm the live DB), then query the copy. The helper used below: +> +> ```bash +> snap() { # snap "" — query a throwaway copy, never the live file +> local d; d=$(mktemp -d) +> cp "$1/$2" "$d/" && cp "$1/$2-wal" "$1/$2-shm" "$d/" 2>/dev/null +> sqlite3 "$d/$2" "$3"; rm -rf "$d" +> } +> ``` > Metrics are on port **8084** inside the container (not published to the host). **Step 1: Bring up the rig with Phase 1 replication enabled** @@ -224,14 +239,14 @@ Expected: `localdb_*` series present (`localdb_oplog_depth`, `localdb_sync_*` `ZB.MOM.WW.LocalDb.Replication`). If absent, the meter allowlist regressed — see `SiteServiceRegistration.ObservedMeters`. -**Step 2: Capture baselines (host-side, against the bind mounts)** +**Step 2: Capture baselines (from throwaway copies — never the live files, see the `snap` helper above)** ```bash cd ~/Desktop/ScadaBridge/docker -sqlite3 site-a-node-a/data/site-localdb.db "SELECT COUNT(*) FROM __localdb_oplog;" -sqlite3 site-a-node-a/data/store-and-forward.db \ +snap site-a-node-a/data site-localdb.db "SELECT COUNT(*) FROM __localdb_oplog;" +snap site-a-node-a/data store-and-forward.db \ "SELECT COUNT(*), COALESCE(SUM(retry_count),0) FROM sf_messages;" -sqlite3 site-a-node-a/data/scadabridge.db "SELECT COUNT(*) FROM native_alarm_state;" +snap site-a-node-a/data scadabridge.db "SELECT COUNT(*) FROM native_alarm_state;" ``` Record all values and the wall-clock time. (`COUNT + SUM(retry_count)` is the S&F row-write @@ -253,11 +268,11 @@ Two generators, run concurrently: cd ~/Desktop/ScadaBridge/docker for i in $(seq 1 6); do echo "=== t+$((i*5))m ===" - sqlite3 site-a-node-a/data/store-and-forward.db \ + snap site-a-node-a/data store-and-forward.db \ "SELECT COUNT(*), COALESCE(SUM(retry_count),0) FROM sf_messages;" - sqlite3 site-a-node-a/data/scadabridge.db \ + snap site-a-node-a/data scadabridge.db \ "SELECT COUNT(*), COUNT(CASE WHEN last_transition_at > datetime('now','-5 minutes') THEN 1 END) FROM native_alarm_state;" - sqlite3 site-a-node-a/data/site-localdb.db "SELECT COUNT(*) FROM __localdb_oplog;" + snap site-a-node-a/data site-localdb.db "SELECT COUNT(*) FROM __localdb_oplog;" docker exec scadabridge-site-a-a curl -s localhost:8084/metrics \ | grep -E '^localdb_(oplog_depth|sync)' sleep 300 @@ -271,7 +286,7 @@ upserts — say so in the findings if alarm volume is near a threshold). **Step 5: Measure `config_json` sizes (D6)** ```bash -sqlite3 site-a-node-a/data/scadabridge.db \ +snap site-a-node-a/data scadabridge.db \ "SELECT COUNT(*), MAX(LENGTH(config_json)), AVG(LENGTH(config_json)) FROM deployed_configurations;" ``` diff --git a/docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json b/docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json index de5b004b..64b301b9 100644 --- a/docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json +++ b/docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json @@ -16,7 +16,7 @@ }, "reviewPass": { "date": "2026-07-19", - "note": "Plan verified against actual code (3 verification sweeps + LocalDb library source) and corrected in place. Headline corrections: D1 (StoreDeployedConfigIfNewerAsync has a SECOND surviving caller, SiteReconciliationActor.cs:166 - the method and guard STAY; original Task 13 would also not have compiled, deleting a method whose caller dies only in Task 15), D3 (the active node ALREADY purges at DeploymentManagerActor.cs:1921 - Task 12 became a pin test, nothing is re-homed), D6 added (4 MB gRPC receive cap x row-count-only MaxBatchSize batching; config_json > 128 KB documented - measure in Task 1, size MaxBatchSize in Task 19, single row near 4 MB = stop/lib work), Task 1 rewritten (Phase 2 tables are NOT registered on the Phase 1 rig so driven churn never reaches __localdb_oplog - measure legacy-DB write rates + arithmetic; metrics port 8084 not 8080; real metric names are localdb_oplog_depth / localdb_sync_*, the plan's localdb_oplog_backlog/replication_dead_letters/sync_connected never existed; containers have no sqlite3 - use host-side sqlite3 on the bind mounts; branch already exists, no checkout -b), Task 14 (register 8 tables NOT 11 - notification_lists/smtp_configurations deliberately unregistered, reversing the original instruction; keep Migrate LAST in OnReady)." + "note": "Plan verified against actual code (3 verification sweeps + LocalDb library source) and corrected in place. Headline corrections: D1 (StoreDeployedConfigIfNewerAsync has a SECOND surviving caller, SiteReconciliationActor.cs:166 - the method and guard STAY; original Task 13 would also not have compiled, deleting a method whose caller dies only in Task 15), D3 (the active node ALREADY purges at DeploymentManagerActor.cs:1921 - Task 12 became a pin test, nothing is re-homed), D6 added (4 MB gRPC receive cap x row-count-only MaxBatchSize batching; config_json > 128 KB documented - measure in Task 1, size MaxBatchSize in Task 19, single row near 4 MB = stop/lib work), Task 1 rewritten (Phase 2 tables are NOT registered on the Phase 1 rig so driven churn never reaches __localdb_oplog - measure legacy-DB write rates + arithmetic; metrics port 8084 not 8080; real metric names are localdb_oplog_depth / localdb_sync_*, the plan's localdb_oplog_backlog/replication_dead_letters/sync_connected never existed; containers have no sqlite3 - sample via throwaway copies of the DB triplet, NEVER host-side sqlite3 against the live files [2026-07-20: that poisons the container's WAL state - see docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md]; branch already exists, no checkout -b), Task 14 (register 8 tables NOT 11 - notification_lists/smtp_configurations deliberately unregistered, reversing the original instruction; keep Migrate LAST in OnReady)." }, "decisions": [ { @@ -64,7 +64,7 @@ "StoreAndForwardService: :39 is the _replication FIELD, :243 the ctor param; exactly 6 emission sites (:654,805,836,872,1120,1146). ServiceCollectionExtensions.cs:32 also resolves ReplicationService inside the StoreAndForwardService factory - must go in Task 14.", "ReplicationMessages.cs holds ONLY the 10 Replicate*/Apply* records; the four SfBuffer resync records live at the bottom of SiteReplicationActor.cs (:678-707) and die with the actor file.", "10 (not 9) appsettings.Site.json files set the legacy paths - deploy/wonder-app-vd03/appsettings.Site.json was missed (also sets ReplicationEnabled:false).", - "Rig facts: site metrics on port 8084 (AnyIP in-container, NOT published to host; 8080 is Traefik); real metric names localdb_oplog_depth / localdb_sync_* (meter ZB.MOM.WW.LocalDb.Replication); containers (aspnet:10.0) have NO sqlite3 - data dirs are host bind mounts, run sqlite3 host-side; ReplicationOptions bind at LocalDb:Replication:* (lib binds the section).", + "Rig facts: site metrics on port 8084 (AnyIP in-container, NOT published to host; 8080 is Traefik); real metric names localdb_oplog_depth / localdb_sync_* (meter ZB.MOM.WW.LocalDb.Replication); containers (aspnet:10.0) have NO sqlite3 - data dirs are host bind mounts but NEVER run sqlite3 host-side against the live files (poisons the container's WAL state, root cause of the 2026-07-20 disk-I/O-error incident): cp the .db/-wal/-shm triplet and query the copy (see the snap() helper in the plan); ReplicationOptions bind at LocalDb:Replication:* (lib binds the section).", "Phase 1's LocalDbSitePairConvergenceTests uses ONE shared API key (:47) - it never did a mismatched-key non-vacuity run; wrong-key denial is unit-covered in Host.Tests/LocalDbSyncAuthInterceptorTests.cs. Task 18's non-vacuity check must be done directly for the new scenarios.", "MigrateEvents synthesizes deterministic 'mig-{node}-{legacyId}' ids (NOT fresh GUIDs - crash-rerun idempotency). Migration runs AFTER RegisterReplicated in OnReady (order is load-bearing); Task 8's oplog pin test must register sf_messages on its own TestLocalDb since production doesn't register it until Task 14." ], @@ -88,7 +88,7 @@ {"id": 17, "subject": "Task 17: Config-key cleanup (incl. ConfigFetchRetryCount; 10 appsettings files)", "status": "pending", "classification": "standard", "blockedBy": [16]}, {"id": 18, "subject": "Task 18: Two-node convergence suite for the Phase 2 tables", "status": "pending", "classification": "high-risk", "blockedBy": [17], "note": "Non-vacuity must be proven directly (Phase 1 never did the mismatched-key run the original plan claimed)."}, {"id": 19, "subject": "Task 19: Rig configuration (MaxOplogRows/MaxOplogAge + MaxBatchSize per D6)", "status": "pending", "classification": "small", "blockedBy": [17]}, - {"id": 20, "subject": "Task 20: Live gate on the docker rig (10 evidence items)", "status": "pending", "classification": "high-risk", "blockedBy": [18, 19], "note": "Includes the post-cutover oplog drain-under-churn check (evidence 10) that Task 1 cannot measure pre-cutover. Host-side sqlite3 on bind mounts; metrics on :8084 in-container."}, + {"id": 20, "subject": "Task 20: Live gate on the docker rig (10 evidence items)", "status": "pending", "classification": "high-risk", "blockedBy": [18, 19], "note": "Includes the post-cutover oplog drain-under-churn check (evidence 10) that Task 1 cannot measure pre-cutover. Sample DBs via throwaway copies (snap() helper), never host-side sqlite3 on live files; metrics on :8084 in-container."}, {"id": 21, "subject": "Task 21: Documentation truth pass", "status": "pending", "classification": "standard", "blockedBy": [20]} ], "knownFlakes": [ diff --git a/docs/plans/2026-07-19-localdb-phase2-soak.md b/docs/plans/2026-07-19-localdb-phase2-soak.md index 99c69c86..d80c2433 100644 --- a/docs/plans/2026-07-19-localdb-phase2-soak.md +++ b/docs/plans/2026-07-19-localdb-phase2-soak.md @@ -3,14 +3,22 @@ **Run date:** 2026-07-19 / 2026-07-20 (UTC) · **Rig:** local 8-node docker cluster, site-a pair **Task:** Task 1 of [`2026-07-19-localdb-adoption-phase2.md`](2026-07-19-localdb-adoption-phase2.md) -> ## GATE VERDICT: **STOP — do not proceed to Task 3.** +> ## GATE VERDICT: **STOP — do not proceed to Task 3.** *(superseded — see update)* > > Not for the reason the plan anticipated. Oplog sizing is fine and D6 is resolved. The soak -> instead surfaced a **pre-existing Phase 1 defect**: the consolidated LocalDb database -> (`site-localdb.db`) throws `SQLite Error 10: 'disk I/O error'` on essentially every write on -> the **active** node under sustained concurrent load. Phase 2 moves eight more tables — -> including the two highest-volume ones — into exactly that database. See +> instead surfaced what looked like a **pre-existing Phase 1 defect**: the consolidated LocalDb +> database (`site-localdb.db`) throws `SQLite Error 10: 'disk I/O error'` on essentially every +> write on the **active** node under sustained concurrent load. See > [Finding 1](#finding-1-blocker). +> +> **Update 2026-07-20: Finding 1 is root-caused and is NOT a product defect.** The soak's own +> host-side `sqlite3` sampling poisoned both nodes (WAL reset across the virtiofs boundary); +> the unmodified code sustains the full soak load indefinitely with zero errors — +> reproduced/refuted on demand, see +> [`docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md`](../known-issues/2026-07-20-localdb-disk-io-error-under-load.md) §0. +> Finding 1 therefore no longer blocks Task 3. What still stands before proceeding: re-run the +> cut-short sampling (Findings 4/5 write-rate numbers) using the safe copy-based `snap` recipe +> now in the plan, on a rig where both site-a nodes have been restarted since any host-side read. --- @@ -22,7 +30,7 @@ does not rediscover them. | Plan said | Reality | |---|---| | `docker exec … curl -s localhost:8084/metrics` | **No `curl` in the `aspnet:10.0` image.** Use a sidecar sharing the container netns: `docker run --rm --network container:scadabridge-site-a-a curlimages/curl:latest -s localhost:8084/metrics` | -| Sample with host-side `sqlite3` against the bind mounts | **Unsafe under live writes** — SQLite's `-shm` shared-memory index is not reliable across the Docker VM boundary. Ultimately *not* the cause of Finding 1 (proven below), but it should not be done against a database the container is actively writing. Copy the files first, or read counters from `/metrics`. | +| Sample with host-side `sqlite3` against the bind mounts | **This IS the cause of Finding 1** (root-caused 2026-07-20 — the "ruled out" verdict below did not survive; see the known-issue doc §0). Host↔container POSIX locks don't propagate over virtiofs; a host-side read checkpoints + resets the WAL under the container and permanently poisons its connections. Copy the file triplet and query the copy, or read counters from `/metrics`. | | Drive alarm churn on a deployed instance | **Not possible on this rig.** `infra/mssql/seed-config.sql` seeds zero `TemplateNativeAlarmSources`, opc-plc runs with no alarm flags, and no simulator or harness exists anywhere in the repo. `native_alarm_state` stayed at 0 rows throughout. Bounded analytically instead — see [Finding 3](#finding-3). | | Drive S&F churn via template 4's `TestExternalSystem` script | Template 4 exists after a reseed but is **not deployable** (34 pre-deployment validation errors: 30 `ConnectionBinding` + 4 `ScriptCompilation`). A purpose-built `SoakGenerator` template was used instead — see below. | @@ -60,7 +68,7 @@ Sustained rate observed: ~**2.9 HTTP attempts/sec** (688–864 connection-refuse --- -## Finding 1 (BLOCKER) +## Finding 1 (BLOCKER) — *root-caused 2026-07-20: observer-induced, not a product defect; see the gate-verdict update and the known-issue doc §0* ### The Phase 1 consolidated LocalDb fails under sustained write load on the active node @@ -95,7 +103,7 @@ The failure was isolated by moving the load between nodes: | site-a-b | standby | no | **0** | | site-a-b | active (after failover) | yes | **4 391** | -#### It is LocalDb-specific, not the filesystem +#### It is LocalDb-specific, not the filesystem — *wrong: sampling-selection bias; only the LocalDb file was ever host-read* The decisive control. Under identical load, on the same node, in the same bind-mounted directory, counting error-stack frames over 3 minutes: @@ -112,14 +120,20 @@ directory, counting error-stack frames over 3 minutes: Ordinary SQLite on the same bind mount is completely healthy. Only the LocalDb-managed database fails. -#### Ruling out the observer +#### Ruling out the observer — **RETRACTED 2026-07-20: the observer was the cause** Onset (04:56:37) was one second after a host-side `sqlite3` sample (04:56:36), which made -observer-induced `-shm` corruption the leading hypothesis. It is **excluded**: after node-a was -restarted (fresh open, recovered shm) and the load failed over to node-b, node-b began erroring -at a *higher* rate while no host process touched its files, and node-a — whose files *had* been -sampled — went to zero once it stopped carrying load. The variable that tracks the errors is -**load**, not host access. +observer-induced `-shm` corruption the leading hypothesis. The original run "excluded" it: +after node-a was restarted and the load failed over to node-b, node-b began erroring while no +host process touched its files, and sampled node-a went to zero once idle. + +**That exclusion was wrong.** The 04:56 sampling had poisoned *both* nodes' files (both carry +the 04:56 main-DB mtime; node-b's WAL was left at 0 bytes) — node-b was silent only because a +standby issues ~no LocalDb statements, and erupted on its first post-failover write. Verified +2026-07-20: 10+ min of full soak load on a freshly-reopened node with zero errors, then a +single host `sqlite3` read reset its 4.6 MiB WAL to 0 bytes and started the error storm one +second later (`SQLITE_IOERR_SHORT_READ`, 522). Full mechanism + minimal repro: +[`docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md`](../known-issues/2026-07-20-localdb-disk-io-error-under-load.md) §0. #### Secondary defect, same area @@ -142,7 +156,9 @@ independently of LocalDb. Cutting over onto a store that cannot absorb the *curr load — and doing so in the same commit that removes the fallback — would convert a logging defect into site-wide config and buffer loss. -This is a Phase 1 defect. It must be root-caused and fixed before Phase 2's Task 3. +~~This is a Phase 1 defect. It must be root-caused and fixed before Phase 2's Task 3.~~ +**Root-caused 2026-07-20: not a Phase 1 defect** — observer-induced WAL reset across the +virtiofs bind-mount boundary; see the gate-verdict update at the top of this document. --- diff --git a/infra/docker-compose.yml b/infra/docker-compose.yml index cd32c86d..1d06f4e7 100644 --- a/infra/docker-compose.yml +++ b/infra/docker-compose.yml @@ -68,6 +68,9 @@ services: MSSQL_PID: "Developer" volumes: - scadabridge-mssql-data:/var/opt/mssql + # NOTE: the official mssql/server image does NOT run + # /docker-entrypoint-initdb.d — these mounts are informational only; + # infra/reseed.sh applies the scripts explicitly via sqlcmd. - ./mssql/setup.sql:/docker-entrypoint-initdb.d/setup.sql:ro - ./mssql/machinedata_seed.sql:/docker-entrypoint-initdb.d/machinedata_seed.sql:ro - ./mssql/setup-env2.sql:/docker-entrypoint-initdb.d/setup-env2.sql:ro diff --git a/infra/reseed.sh b/infra/reseed.sh index dfae57d7..50257d91 100755 --- a/infra/reseed.sh +++ b/infra/reseed.sh @@ -82,12 +82,15 @@ if ! $SKIP_TEARDOWN; then done echo " MSSQL ready." - echo " Waiting for setup.sql to create ScadaBridgeConfig..." - until docker exec scadabridge-mssql /opt/mssql-tools18/bin/sqlcmd \ - -S localhost -U sa -P 'ScadaBridge_Dev1#' -C \ - -Q "IF DB_ID('ScadaBridgeConfig') IS NULL THROW 50000, 'not ready', 1;" \ - >/dev/null 2>&1; do - sleep 2 + # The official mcr.microsoft.com/mssql/server image does NOT implement + # /docker-entrypoint-initdb.d, so the compose-mounted init scripts never run + # on their own — waiting for them here hangs forever on a fresh volume. + # Apply them explicitly instead (all are idempotent). + echo " Applying MSSQL init scripts (the mssql/server image has no initdb hook)..." + for f in mssql/setup.sql mssql/machinedata_seed.sql mssql/setup-env2.sql; do + echo " $f" + docker exec -i scadabridge-mssql /opt/mssql-tools18/bin/sqlcmd \ + -S localhost -U sa -P 'ScadaBridge_Dev1#' -C -b < "$SCRIPT_DIR/$f" done echo " ScadaBridgeConfig present." diff --git a/src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/Telemetry/CachedCallTelemetryForwarder.cs b/src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/Telemetry/CachedCallTelemetryForwarder.cs index b376da08..7df5588c 100644 --- a/src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/Telemetry/CachedCallTelemetryForwarder.cs +++ b/src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/Telemetry/CachedCallTelemetryForwarder.cs @@ -114,8 +114,8 @@ public sealed class CachedCallTelemetryForwarder : ICachedCallTelemetryForwarder // Kind/Status are domain fields carried in DetailsJson — decompose to log them. var d = AuditRowProjection.Decompose(telemetry.Audit); _logger.LogWarning(ex, - "CachedCallTelemetryForwarder: audit emission threw for EventId {EventId} (Kind {Kind}, Status {Status})", - d.EventId, d.Kind, d.Status); + "CachedCallTelemetryForwarder: audit emission threw for EventId {EventId} (Kind {Kind}, Status {Status}, sqlite {SqliteError})", + d.EventId, d.Kind, d.Status, SqliteErrorCodes.Describe(ex)); } } @@ -192,8 +192,8 @@ public sealed class CachedCallTelemetryForwarder : ICachedCallTelemetryForwarder catch (Exception ex) { _logger.LogWarning(ex, - "CachedCallTelemetryForwarder: tracking-store emission threw for TrackedOperationId {Id} (Status {Status})", - telemetry.Operational.TrackedOperationId, telemetry.Operational.Status); + "CachedCallTelemetryForwarder: tracking-store emission threw for TrackedOperationId {Id} (Status {Status}, sqlite {SqliteError})", + telemetry.Operational.TrackedOperationId, telemetry.Operational.Status, SqliteErrorCodes.Describe(ex)); } } } diff --git a/src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/Telemetry/SiteAuditTelemetryActor.cs b/src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/Telemetry/SiteAuditTelemetryActor.cs index 88169bce..dbc9e463 100644 --- a/src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/Telemetry/SiteAuditTelemetryActor.cs +++ b/src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/Telemetry/SiteAuditTelemetryActor.cs @@ -68,6 +68,15 @@ public class SiteAuditTelemetryActor : ReceiveActor private readonly IOperationTrackingStore? _trackingStore; private readonly SiteAuditTelemetryOptions _options; private readonly ILogger _logger; + // Captured at construction (both are thread-safe immutable handles) because + // ScheduleNext/ScheduleNextCached run from the drains' finally blocks, whose + // ConfigureAwait(false) continuations complete on pool threads with no + // active ActorContext — reading Context/Self there either throws + // NotSupportedException or, worse, silently resolves a STALE cell left in + // the thread-static slot and re-arms the tick at the wrong actor + // (known-issues/2026-07-20-localdb-disk-io-error-under-load.md §8). + private readonly IScheduler _scheduler; + private readonly IActorRef _self; private ICancelable? _pendingTick; private ICancelable? _pendingCachedTick; // Per-actor lifecycle CTS so an in-flight drain (queue read, @@ -108,6 +117,8 @@ public class SiteAuditTelemetryActor : ReceiveActor _options = options.Value; _logger = logger; _trackingStore = trackingStore; + _scheduler = Context.System.Scheduler; + _self = Self; ReceiveAsync(_ => OnDrainAsync()); ReceiveAsync(_ => OnCachedDrainAsync()); @@ -197,7 +208,9 @@ public class SiteAuditTelemetryActor : ReceiveActor { // Catch-all so a SQLite hiccup or mapper bug never crashes the // actor. The next tick is still scheduled in the finally block. - _logger.LogError(ex, "Unexpected error during audit-log telemetry drain."); + _logger.LogError(ex, + "Unexpected error during audit-log telemetry drain (sqlite {SqliteError}).", + SqliteErrorCodes.Describe(ex)); } finally { @@ -278,8 +291,8 @@ public class SiteAuditTelemetryActor : ReceiveActor // batch — the audit half is best-effort. Log and skip // this row; it stays Pending for the next drain. _logger.LogWarning(ex, - "Cached-telemetry drain: tracking lookup threw for {EventId} (TrackedOperationId {Tid}); skipping.", - auditRow.EventId, auditRow.CorrelationId); + "Cached-telemetry drain: tracking lookup threw for {EventId} (TrackedOperationId {Tid}, sqlite {SqliteError}); skipping.", + auditRow.EventId, auditRow.CorrelationId, SqliteErrorCodes.Describe(ex)); continue; } @@ -332,7 +345,9 @@ public class SiteAuditTelemetryActor : ReceiveActor } catch (Exception ex) { - _logger.LogError(ex, "Unexpected error during cached-telemetry drain."); + _logger.LogError(ex, + "Unexpected error during cached-telemetry drain (sqlite {SqliteError}).", + SqliteErrorCodes.Describe(ex)); } finally { @@ -428,24 +443,26 @@ public class SiteAuditTelemetryActor : ReceiveActor return list; } + // Must stay off Context/Self: called from off-context continuations — see + // the _scheduler/_self field comment. private void ScheduleNext(TimeSpan delay) { _pendingTick?.Cancel(); - _pendingTick = Context.System.Scheduler.ScheduleTellOnceCancelable( + _pendingTick = _scheduler.ScheduleTellOnceCancelable( delay, - Self, + _self, Drain.Instance, - Self); + _self); } private void ScheduleNextCached(TimeSpan delay) { _pendingCachedTick?.Cancel(); - _pendingCachedTick = Context.System.Scheduler.ScheduleTellOnceCancelable( + _pendingCachedTick = _scheduler.ScheduleTellOnceCancelable( delay, - Self, + _self, CachedDrain.Instance, - Self); + _self); } /// Self-tick message that triggers an audit-only drain cycle. diff --git a/src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/Telemetry/SqliteErrorCodes.cs b/src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/Telemetry/SqliteErrorCodes.cs new file mode 100644 index 00000000..83bc351a --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/Telemetry/SqliteErrorCodes.cs @@ -0,0 +1,31 @@ +using Microsoft.Data.Sqlite; + +namespace ZB.MOM.WW.ScadaBridge.AuditLog.Site.Telemetry; + +/// +/// Renders the primary/extended SQLite result codes of a +/// for log messages. The exception's own message +/// carries only the primary code ("SQLite Error 10: 'disk I/O error'"), which +/// is too generic to act on — the 2026-07-20 disk-I/O incident +/// (known-issues/2026-07-20-localdb-disk-io-error-under-load.md) had to be +/// reproduced from scratch to learn the extended code (522 = +/// SQLITE_IOERR_SHORT_READ) that names the failing operation. +/// +internal static class SqliteErrorCodes +{ + /// + /// "primary/extended" (e.g. "10/522") for a + /// anywhere in the exception chain; "n/a" for non-SQLite failures. + /// + public static string Describe(Exception ex) + { + for (Exception? e = ex; e is not null; e = e.InnerException) + { + if (e is SqliteException se) + { + return $"{se.SqliteErrorCode}/{se.SqliteExtendedErrorCode}"; + } + } + return "n/a"; + } +} diff --git a/src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs b/src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs index 0b95b684..95f2361c 100644 --- a/src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs +++ b/src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs @@ -257,8 +257,8 @@ public class SiteEventLogger : ISiteEventLogger, IDisposable // (Health Monitoring reads FailedWriteCount) and fault the caller's // Task instead of silently discarding the exception. Interlocked.Increment(ref _failedWriteCount); - _logger.LogError(ex, "Failed to record event: {EventType} from {Source}", - pending.EventType, pending.Source); + _logger.LogError(ex, "Failed to record event: {EventType} from {Source} (sqlite {SqliteError})", + pending.EventType, pending.Source, DescribeSqliteError(ex)); pending.Completion.TrySetException(ex); } } @@ -297,6 +297,25 @@ public class SiteEventLogger : ISiteEventLogger, IDisposable } } + /// + /// "primary/extended" SQLite result codes (e.g. "10/522") for a + /// anywhere in the chain; "n/a" otherwise. + /// The exception message alone carries only the primary code, which proved + /// too generic to diagnose the 2026-07-20 disk-I/O incident + /// (known-issues/2026-07-20-localdb-disk-io-error-under-load.md). + /// + private static string DescribeSqliteError(Exception ex) + { + for (Exception? e = ex; e is not null; e = e.InnerException) + { + if (e is SqliteException se) + { + return $"{se.SqliteErrorCode}/{se.SqliteExtendedErrorCode}"; + } + } + return "n/a"; + } + /// An event awaiting persistence by the background writer. private sealed record PendingEvent( string Id, diff --git a/tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Site/Telemetry/SiteAuditTelemetryActorTests.cs b/tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Site/Telemetry/SiteAuditTelemetryActorTests.cs index 1aad60dd..c19ed203 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Site/Telemetry/SiteAuditTelemetryActorTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Site/Telemetry/SiteAuditTelemetryActorTests.cs @@ -5,6 +5,7 @@ using Microsoft.Extensions.Logging.Abstractions; using Microsoft.Extensions.Options; using NSubstitute; using NSubstitute.ExceptionExtensions; +using NSubstitute.ReceivedExtensions; using ZB.MOM.WW.ScadaBridge.AuditLog.Site.Telemetry; using ZB.MOM.WW.Audit; using ZB.MOM.WW.ScadaBridge.Commons.Types.Audit; @@ -451,4 +452,55 @@ public class SiteAuditTelemetryActorTests : TestKit await _client.DidNotReceiveWithAnyArgs().IngestCachedTelemetryAsync(default!, default); await _queue.DidNotReceiveWithAnyArgs().ReadPendingCachedTelemetryAsync(default, default); } + + /// + /// Regression for the 2026-07-20 rig finding (known-issues/2026-07-20-localdb-disk-io-error-under-load.md §8): + /// both drain handlers await with ConfigureAwait(false), so when a + /// dependency call completes off the actor thread (as real SQLite/gRPC + /// always do) the finally-block re-arm runs on a pool thread with no + /// active ActorContext. Depending on what that thread's thread-static cell + /// slot holds, Context/Self either throw + /// NotSupportedException: There is no active ActorContext (crashing + /// the actor once per drain — the rig's logged variant) or silently resolve + /// to a STALE cell of some other actor, misrouting the tick so the drain + /// loop stops. The two assertions below catch one variant each. Every other + /// test in this class masks the bug by returning already-completed tasks + /// from the mocks, which keeps the continuations on the actor thread. + /// + [Fact] + public async Task Drains_Whose_Awaits_Complete_Off_The_Actor_Thread_Keep_Draining_Without_Crashing() + { + // Task.Delay completes on a timer thread; with ConfigureAwait(false) + // everything after the await — including the finally-block re-arm — + // stays off the actor context. + _queue.ReadPendingAsync(Arg.Any(), Arg.Any()) + .Returns(async _ => + { + await Task.Delay(25).ConfigureAwait(false); + return (IReadOnlyList)Array.Empty(); + }); + _queue.ReadPendingCachedTelemetryAsync(Arg.Any(), Arg.Any()) + .Returns(async _ => + { + await Task.Delay(25).ConfigureAwait(false); + return (IReadOnlyList)Array.Empty(); + }); + + // Variant 1 (throw → restart storm): any NotSupportedException logged + // during the run fails the filter. Variant 2 (silent misroute → the + // drain loop stalls after the first tick): the sustained-drain + // assertion inside fails because reads stop at 1. + await EventFilter.Exception().ExpectAsync(0, async () => + { + CreateActorWithCachedDrain(Opts(busySeconds: 1, idleSeconds: 1)); + + // Three full cycles prove the finally-block re-arm works from an + // off-context thread: tick → drain → re-arm → tick → … + await AwaitAssertAsync(async () => + { + await _queue.Received(Quantity.Within(3, int.MaxValue)).ReadPendingAsync(Arg.Any(), Arg.Any()); + await _queue.Received(Quantity.Within(3, int.MaxValue)).ReadPendingCachedTelemetryAsync(Arg.Any(), Arg.Any()); + }, TimeSpan.FromSeconds(10)); + }); + } } From d512572dac1e87553bf927a6bde1c32c16ecca11 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Mon, 20 Jul 2026 02:01:42 -0400 Subject: [PATCH 28/55] =?UTF-8?q?docs(localdb):=20phase=202=20resume=20sta?= =?UTF-8?q?te=20=E2=80=94=20where=20the=20plan=20stands=20and=20what=20car?= =?UTF-8?q?ries=20forward?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Scratch handoff so the plan can be picked back up cleanly: Task 1 done and its STOP verdict superseded, what must happen before Task 3, the four plan-premise corrections, the rig gotchas that are not discoverable from the repo, and the rig cleanup the Task 20 live gate needs. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts --- docs/plans/2026-07-20-phase2-resume-state.md | 106 +++++++++++++++++++ 1 file changed, 106 insertions(+) create mode 100644 docs/plans/2026-07-20-phase2-resume-state.md diff --git a/docs/plans/2026-07-20-phase2-resume-state.md b/docs/plans/2026-07-20-phase2-resume-state.md new file mode 100644 index 00000000..f7360e46 --- /dev/null +++ b/docs/plans/2026-07-20-phase2-resume-state.md @@ -0,0 +1,106 @@ +# LocalDb Phase 2 — resume state (written 2026-07-20, pre-compaction) + +Scratch handoff for picking the plan back up. Delete once Phase 2 lands. + +**Plan:** [`2026-07-19-localdb-adoption-phase2.md`](2026-07-19-localdb-adoption-phase2.md) +(execute with `superpowers-extended-cc:executing-plans`; task state in +`…-phase2.md.tasks.json`) +**Branch:** `feat/localdb-phase2` (from `feat/localdb-phase1`, which is **not** merged/pushed) +**Tree:** clean @ `8652eab9`. Build 0 warnings. + +--- + +## Where we are + +**Task 1 (the gate) is DONE. Its STOP verdict has been superseded — Phase 2 is unblocked.** + +Task 1 appeared to fail on a Phase 1 blocker (LocalDb throwing `disk I/O error` under load). +That was **root-caused the same day as observer-induced and not a product defect**: host-side +`sqlite3` reads of the live bind-mounted WAL files reset the WAL under the container. The +unmodified library sustains the full soak load indefinitely. Full write-up in +[`docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md`](../known-issues/2026-07-20-localdb-disk-io-error-under-load.md) +§0 (cause) and §11a (fixes shipped). + +**Tasks 2–21 are untouched. No plan code has been written.** + +### Before starting Task 3 + +1. **Task 2** (trivial, ~3 min) — the decision record. Blocked on Task 1's numbers, which are + partly missing (below). +2. **Re-run the cut-short sampling.** The soak died before producing sustained + `sf_messages` / `native_alarm_state` write rates. Use the **`snap` helper now in the plan** + (Task 1 step 2, ~line 217) — copies the `.db`/`-wal`/`-shm` triplet and queries the copy. + **Never** run host-side `sqlite3` against a live file. Both site-a nodes must have been + restarted since any host-side read (they have been — see rig state). +3. Then Wave 1 (Tasks 3 + 4) can be dispatched in parallel. + +--- + +## Findings that change the plan (carry these forward) + +From [`2026-07-19-localdb-phase2-soak.md`](2026-07-19-localdb-phase2-soak.md): + +- **D6's premise is wrong, its risk is real.** The plan says `config_json` is "documented to + exceed 128 KB per row." The source actually says the *escaped Akka envelope* blew the 128 KB + frame and the raw JSON "only needs to be ~60-70 KB". So the largest known production row is + ~60–70 KB → **no stop condition**. But `MaxBatchSize=500` is unsafe (500 × 70 KB ≈ 35 MB vs a + 4 MB gRPC cap) → **Task 19 must set `LocalDb:Replication:MaxBatchSize = 16`.** This is the one + firmly evidence-backed number. +- **D4 is wrong.** `native_alarm_state` is not "unbounded by design" — `NativeAlarmActor` + coalesces per `SourceReference` on a 100 ms flush (`NativeAlarmActor.cs:473-502`), so + worst case is `distinct_source_refs × 10/sec`. +- **`sf_messages` has a hard 50 rows/sec ceiling** — `SweepBatchLimit` (500) ÷ + `RetryTimerInterval` (10 s). A ceiling, not an estimate. +- **Task 1 step 7's stop condition is much weaker than written.** Exceeding the oplog caps + prunes and flags `needs_snapshot` → graceful snapshot resync (`OplogStore.cs:109-138`, + `MaintenanceBackgroundService.cs:57`). Not data loss, not a wedge. + +The plan's own Task-1 method also needed four corrections (metrics sidecar, safe sampling, +no alarm generator on this rig, template 4 undeployable) — all recorded in the soak doc §1. + +--- + +## Rig operational gotchas (hard-won; not obvious from the repo) + +- **`ExternalSystem.Call` does NOT buffer to store-and-forward — only `ExternalSystem.CachedCall` + does.** Using `Call` gives HTTP traffic and zero `sf_messages` rows. This is the single most + important detail for generating S&F load. +- **Never** query the bind-mounted SQLite files from the macOS host while containers run. Use + the plan's `snap` helper, or `docker run --rm -v :/d alpine/sqlite3 …`. +- **No `curl` in the `aspnet:10.0` image**, and metrics port 8084 is not published. Use a + sidecar: `docker run --rm --network container:scadabridge-site-a-a curlimages/curl:latest -s localhost:8084/metrics` +- **Seeded template 4 (`Motor Controller`) is not deployable** — 34 pre-deployment validation + errors. The soak used a purpose-built `SoakGenerator` template instead. +- CLI: `template script update` requires `--name` and `--trigger-type` even to change only + `--code`. In zsh, don't put auth flags in an unquoted variable (no word-splitting). +- Rig auth: `--url http://localhost:9000 --username multi-role --password password`. + `LdapGroupMappings` role strings must be canonical `Designer`/`Deployer`; central caches them + at startup, so restart central after seeding them. + +--- + +## Rig state as left + +Reseeded and healthy; both site-a nodes restarted after the poisoning incident. + +**Needs cleanup before the Task 20 live gate:** +- `ExternalSystemDefinitions` id 1 is still repointed to `http://127.0.0.1:9` — restore to + `http://scadabridge-restapi:5200`. +- Template `SoakGenerator` (id 2021) and instances `soakgen-1..4` (ids 5–8) are still deployed + on site-a and **still generating load**. + +--- + +## Commits this session + +| Commit | What | +|---|---| +| `25463d52` | plan review pass — corrections from code verification (D1/D3/D6, Task 1 method, Task 14) | +| `cf46e596` | fix(rig): `seed-sites.sh` canonical `Designer`/`Deployer` role names | +| `fc553bd9` | soak findings (gate verdict, since superseded) | +| `82c869a1` | task record: gate verdict | +| `e9e11d63` | root-cause brief for the disk-I/O issue | +| `8652eab9` | root cause + hardening: extended SQLite error codes, `SiteAuditTelemetryActor` async-context fix, `reseed.sh` init-script fix | + +Two known flakes, do not chase: `InstanceActorChildAttributeRaceTests` (intermittent under full-suite +load) and `ParentExecutionIdCorrelationTests` (~91 s / times out on a cold MSSQL fixture, ~1 s warm). From 12eb97e07a992330e80d49f413c38965962082b7 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Mon, 20 Jul 2026 02:15:53 -0400 Subject: [PATCH 29/55] =?UTF-8?q?docs(localdb):=20close=20the=20phase=202?= =?UTF-8?q?=20gate=20=E2=80=94=20questions=20answered,=20plan=20written?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Task 2 of the Phase 2 plan, plus the clean re-run of Task 1's cut-short sampling that Task 2 was waiting on. Gate doc: status NOT STARTED -> CLOSED. All five §5 open questions answered inline. The questions are kept, not deleted — several of the answers overturn a premise the gate itself stated, and that reasoning is the value: - Oplog sizing is not the binding constraint. Cap overrun prunes and flags needs_snapshot (graceful resync), so the caps need no defensive sizing; the real ceiling is D6's 4 MB gRPC cap, binding on MaxBatchSize x row-bytes. - LWW-vs-in-flight-send cannot arise on a correct pair (only the active node sweeps). The honest cost is D2's semantic change: the standby is convergent, no longer byte-identical. - Migration-under-load is designed out, not managed — both mechanisms are deleted in the same commit and D5 forecloses rolling upgrades. - Rollback is genuinely "revert the commit": the migration is additive and leaves the legacy files intact. Bounded cost is the post-cutover delta. - One DB per process stays; every write axis is bounded. §2's requirement to state CDC's duplicate-delivery bound is routed explicitly to Task 21 rather than left implicit. Task 1 re-run (safe copy-based snap(), both nodes restarted first, six 60s intervals): sf_messages 0.80 rows/sec insert, 0/sec retry-UPDATE, oplog 0, alarms 0, max payload_json 76 B, zero SQLite errors across 30 min, both site-a nodes converged at 3564 rows. Independently confirms the disk-I/O storm was observer-induced. Recorded with its limits stated: 0.80/s is ~1.6% of the 50/s ceiling and the retry path was never exercised — acceptable only because that ceiling is structural rather than empirical, and the rig's 721 B config rows are explicitly NOT representative. GATE VERDICT: PROCEED to Task 3. Binding output is MaxBatchSize 500 -> 16. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts --- ...7-19-localdb-adoption-phase2.md.tasks.json | 8 +- docs/plans/2026-07-19-localdb-phase2-gate.md | 123 +++++++++++++++++- docs/plans/2026-07-19-localdb-phase2-soak.md | 61 ++++++++- 3 files changed, 182 insertions(+), 10 deletions(-) diff --git a/docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json b/docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json index 64b301b9..c3006ce1 100644 --- a/docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json +++ b/docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json @@ -69,8 +69,8 @@ "MigrateEvents synthesizes deterministic 'mig-{node}-{legacyId}' ids (NOT fresh GUIDs - crash-rerun idempotency). Migration runs AFTER RegisterReplicated in OnReady (order is load-bearing); Task 8's oplog pin test must register sf_messages on its own TestLocalDb since production doesn't register it until Task 14." ], "tasks": [ - {"id": 1, "subject": "Task 1: Rig soak - measure legacy-DB write rates + config_json sizes (corrected method)", "status": "pending", "classification": "high-risk", "note": "GATES EVERYTHING AFTER TASK 2. Phase 2 tables are NOT in the Phase 1 oplog - measure at the legacy DBs host-side, size caps arithmetically; empirical drain check moves to Task 20 evidence 10. Stop conditions: on-paper oplog overflow OR any config_json near 4 MB (D6)."}, - {"id": 2, "subject": "Task 2: Decision record - close the phase 2 gate", "status": "pending", "classification": "trivial", "blockedBy": [1]}, + {"id": 1, "subject": "Task 1: Rig soak - measure legacy-DB write rates + config_json sizes (corrected method)", "status": "completed", "classification": "high-risk", "note": "GATE CLOSED - verdict PROCEED (2026-07-20). NEVER sample host-side sqlite3 on live bind-mounted WAL files (that method poisoned the first run - use the copy-based snap() helper). Clean re-run: sf_messages 0.80 rows/sec insert, 0/sec retry-UPDATE, oplog 0, alarms 0, max payload_json 76 B, max config_json 721 B (NOT representative), zero SQLite errors in 30 min, both nodes converged. No stop condition met. Binding output: MaxBatchSize 500 -> 16 (Task 19). Empirical drain check remains Task 20 evidence 10."}, + {"id": 2, "subject": "Task 2: Decision record - close the phase 2 gate", "status": "completed", "classification": "trivial", "blockedBy": [1], "note": "Gate doc status flipped NOT STARTED -> CLOSED; all 5 SS5 questions answered inline (questions kept, not deleted); SS2's N5 duplicate-bound requirement routed explicitly to Task 21."}, {"id": 3, "subject": "Task 3: Extract StoreAndForwardSchema.Apply", "status": "pending", "classification": "small", "blockedBy": [2]}, {"id": 4, "subject": "Task 4: Extract SiteStorageSchema.Apply", "status": "pending", "classification": "small", "blockedBy": [2]}, {"id": 5, "subject": "Task 5: Rewire StoreAndForwardStorage onto ILocalDb", "status": "pending", "classification": "high-risk", "blockedBy": [3]}, @@ -101,6 +101,6 @@ "note": "Cold-MSSQL-fixture timing: ~91s and AwaitAssert-times-out cold, ~1s warm. Re-run before investigating." } ], - "lastUpdated": "2026-07-19", - "phase2Status": "BLOCKED at the Task 1 gate (2026-07-20). Task 1 ran and its verdict is STOP - see docs/plans/2026-07-19-localdb-phase2-soak.md. The blocker is NOT oplog sizing (fine) or D6 (resolved, no stop condition): the Phase 1 consolidated LocalDb (site-localdb.db) throws SQLite Error 10 'disk I/O error' on essentially every write on the ACTIVE node under sustained load, affecting site_events + OperationTracking + audit telemetry, and silently dropping site events. Isolated to load (not node, not host-side observation) by failing over between nodes, and to LocalDb specifically (legacy store-and-forward.db/scadabridge.db in the same bind mount take ZERO errors under identical load). Phase 2 would register 8 more tables into that DB - incl. the two highest-volume - while deleting the bespoke fallbacks, so this Phase 1 defect must be root-caused and fixed before Task 3. Tasks 2-21 remain pending and untouched; no plan code has been written. Secondary defect in the same path: SiteAuditTelemetryActor closes over ActorContext across an await (NotSupportedException). Plan-premise corrections now recorded in the soak doc: D6 (largest known production config_json ~60-70 KB not >128 KB, but MaxBatchSize must drop 500 -> 16), D4 (alarm writes bounded by per-SourceReference coalescing at a 100 ms flush), sf_messages hard ceiling 50 rows/sec (SweepBatchLimit/RetryTimerInterval), and oplog cap overrun = graceful snapshot resync, not failure. Rig caveats: ExternalSystemDefinitions id 1 still repointed to http://127.0.0.1:9 and SoakGenerator instances 5-8 still generating load - clean up before Task 20." + "lastUpdated": "2026-07-20", + "phase2Status": "UNBLOCKED - Task 1 gate CLOSED (verdict PROCEED) and Task 2 DONE, 2026-07-20. Task 1's original STOP verdict is SUPERSEDED: the 'Phase 1 disk I/O defect' was OBSERVER-INDUCED (host-side sqlite3 against live bind-mounted WAL files resets the WAL across virtiofs and permanently poisons the container's connections) - NOT a product defect. See docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md. Both earlier isolation claims were confounded: one sampling pass had already poisoned BOTH nodes, and a poisoned standby looks healthy only because it issues almost no statements; 'LocalDb-specific' was sampling-selection bias (only the LocalDb file had ever been host-read). Clean re-run 2026-07-20 with the copy-based snap() helper on restarted nodes, 6 consecutive 60s intervals: sf_messages 48 rows/min = 0.80 rows/sec dead steady, retry-UPDATE rate 0/sec (SUM(retry_count) flat at 200), oplog 0, native_alarm_state 0, max payload_json 76 B, max config_json 721 B, ZERO SQLite errors across 30 min, both site-a nodes converged at 3564 rows. Honest gap: 0.80/s is ~1.6% of the 50/s ceiling and the retry-UPDATE path was never exercised - acceptable because that ceiling is structural (SweepBatchLimit/RetryTimerInterval), not empirical. Rig config rows (721 B) are NOT representative; D6 sizing rests on the documented ~60-70 KB production config_json. Plan-premise corrections stand: D6 (MaxBatchSize 500 -> 16, the one firmly evidence-backed number; Task 19 sets it), D4 (alarm writes bounded by per-SourceReference coalescing at a 100 ms flush, NOT unbounded), sf_messages hard ceiling 50 rows/sec, oplog cap overrun = graceful snapshot resync (needs_snapshot), not data loss. NEXT: Wave 1 = Tasks 3 + 4, dispatchable in parallel (disjoint Files blocks). Tasks 3-21 untouched; no plan code written yet. Rig cleanup still owed before Task 20: restore ExternalSystemDefinitions id 1 to http://scadabridge-restapi:5200 (currently http://127.0.0.1:9), and remove SoakGenerator template 2021 + instances soakgen-1..4 (ids 5-8), still deployed and generating load." } diff --git a/docs/plans/2026-07-19-localdb-phase2-gate.md b/docs/plans/2026-07-19-localdb-phase2-gate.md index f68c1aae..0d966e4d 100644 --- a/docs/plans/2026-07-19-localdb-phase2-gate.md +++ b/docs/plans/2026-07-19-localdb-phase2-gate.md @@ -1,8 +1,11 @@ # LocalDb Phase 2 — Gate Document -> **Status: NOT STARTED.** This is the gate, not the plan. Phase 2 work must not begin -> until an implementation plan is written against the facts below and the open questions -> in §5 are answered. +> **Status: CLOSED (2026-07-20).** The implementation plan is +> [`2026-07-19-localdb-adoption-phase2.md`](2026-07-19-localdb-adoption-phase2.md); its +> decisions D1–D6 answer this gate and its Task 1 supplied the measurements §5 demanded +> (soak record: [`2026-07-19-localdb-phase2-soak.md`](2026-07-19-localdb-phase2-soak.md)). +> Each §5 question is answered inline below. **The questions are deliberately not deleted — +> the reasoning is the value, and several of the answers overturn a premise stated above.** **Phase 1 live gate: PASS** (2026-07-19, docker rig). See the Task 12 commit for evidence. @@ -50,6 +53,11 @@ Known accepted behaviour to carry forward, not silently drop: the **N5 bounded-d race** the bespoke replicator documents and accepts. A Phase 2 plan must state whether CDC inherits the same bound, a tighter one, or a different failure shape. +> **Routed (2026-07-20).** The failure shape changes, so N5's bound does not simply carry +> over: CDC has no chunked anti-entropy hop to duplicate across, but LWW admits re-delivery +> of a row whose park loses to a later write (see the LWW answer in §5). **Task 21 states the +> new duplicate-delivery bound explicitly** and rewrites the N5 note rather than deleting it. + ## 3. Substantially harder than Phase 1 Phase 1 moved two tables that **nothing replicated before**, so the worst case was "no @@ -84,16 +92,125 @@ class: thresholds cannot be chosen from first principles. *This needs a Phase-1 rig soak that has not been run.* The live gate proved correctness, not steady-state behaviour over time. + + **ANSWERED — the soak ran (Task 1); the caps are not the binding constraint, batch + *bytes* are.** Two of this question's own premises turned out to be wrong, and both + corrections relax it: + + 1. **The stop condition was much weaker than written.** Exceeding `MaxOplogRows` / + `MaxOplogAge` does not wedge or lose data — `OplogStore` prunes and flags the peer + `needs_snapshot` (`OplogStore.cs:109-138`, `MaintenanceBackgroundService.cs:57`), which + degrades to a **graceful snapshot resync**. Overrun is a performance event, not a + correctness event, so these caps do not need to be sized defensively. + 2. **`sf_messages` has a hard ceiling, not an estimate.** `SweepBatchLimit` (500) ÷ + `RetryTimerInterval` (10 s) = **≤50 row-writes/sec**, structurally. Measured rig rate + under the purpose-built `SoakGenerator` load was far below that (see the soak record). + Rows are small — max `payload_json` **76 bytes** on the rig. + + `native_alarm_state` is answered under the last question in this section, not here (and + the plan's D4, whose premise it corrects). The **real** ceiling this + question was groping toward is D6's 4 MB gRPC message cap, and it binds on + `MaxBatchSize × max-row-bytes`, not on oplog depth — **Task 19 sets + `LocalDb:Replication:MaxBatchSize = 16`**, the one firmly evidence-backed number the soak + produced. The keyed-instances escape hatch is **not** needed; this plan proceeds. + + Independently, the soak retired a scare: a `disk I/O error` storm initially read as a + Phase 1 library defect and STOP-gated this plan was root-caused as **observer-induced** + (host-side `sqlite3` against live bind-mounted WAL files) — see + [`docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md`](../known-issues/2026-07-20-localdb-disk-io-error-under-load.md) + §0. The unmodified library sustains the full soak load with **zero** SQLite errors. + - **LWW semantics for in-flight store-and-forward sends.** What does last-writer-wins do when one node parks a message the other is mid-delivery on? + + **ANSWERED — the scenario cannot arise on a correctly-behaving pair, and D2 covers what + happens if it does.** Only the **active** node sweeps and delivers; the standby holds a + convergent copy and sends nothing. So "one node mid-delivery while the other parks" is + a split-brain symptom, not steady-state behaviour, and store-and-forward is not where it + should be defended against. + + If it does happen, LWW resolves per row by HLC: the later write wins, and the losing + node's view converges to it. The concrete risk is **at-least-once delivery** — a park + that loses to a stale in-flight update can be re-swept and re-sent. That is not new; + store-and-forward is already at-least-once by construction (a send that succeeds but + whose ack is lost is retried). What **is** new is D2's semantic change, recorded here + because it is the honest cost of this gate: the standby is no longer guaranteed + **byte-identical** to the active buffer, only **convergent**. The bespoke replicator's + `ReplaceAllAsync` bought identity by wiping — and the N1 directional guard existed + precisely because that wipe was dangerous. The library's snapshot resync **merges per + row and never wipes** (`SnapshotApplier`, `LwwApplier.cs:69-78`), so the failure the + guard prevented is structurally impossible rather than merely tested against. Task 10 + ports `SfBufferResyncPredicateTests` as a **convergence** assertion, not a + directional-authority one. + - **Migration-under-load.** How does the one-time copy of an actively-replicating `scadabridge.db` interact with the bespoke replicator still running during cutover? + + **ANSWERED — the interaction is designed out, not managed.** The two mechanisms never run + concurrently. `SiteReplicationActor` and StoreAndForward's `ReplicationService` are + **deleted in the same commit** that registers the Phase 2 tables (Tasks 14/15), and D5 + forecloses rolling upgrades: **both nodes of a site stop and start together**. A process + that boots with the new code has no bespoke replicator to race, and one running the old + code has no CDC triggers. There is no window in which a row is written by one mechanism + and read by the other. + + Within a single booting process the ordering is the load-bearing part, and it is the same + invariant Phase 1 established: **DDL → `RegisterReplicated` → migrate → writes** + (`SiteLocalDbSetup.OnReady`). Rows written *before* registration are invisible to the peer + forever, silently — which is why Tasks 8/9 run the migrator strictly **after** + registration, so every migrated row is captured by the CDC triggers and replicates + normally. Both nodes migrating independently is fine: they migrate the same source rows, + and LWW converges them. + - **Rollback story.** With no dual-mechanism period, what is the recovery path if the cutover fails in production — beyond "revert the commit"? + + **ANSWERED — "revert the commit" is genuinely the path, and it is safe because the + migration is additive and the legacy files are left intact.** Tasks 8/9 **copy** rows out + of `scadabridge.db` and `store-and-forward.db` into the consolidated LocalDb file; they do + not drop, truncate, or delete the source databases. Rolling back is therefore: stop both + nodes of the site, deploy the previous build, start both together (D5). The old code + reopens the legacy files and finds them exactly as it left them. + + The bounded, honest cost of a rollback is **the delta** — rows written into the + consolidated DB after cutover do not flow back into the legacy files. For config tables + that self-heals: `SiteReconciliationActor` reports local inventory to central at startup + and fetches whatever it lacks, so a rolled-back node re-converges to central's truth + without operator action. For `sf_messages` the delta is **lost undelivered buffer** — the + practical mitigation is to drain the buffer before cutting over, which Task 20's live gate + and Task 21's runbook both call for. + + What has **no** rollback is a site pair split across versions — hence D5. That is a + deployment-procedure constraint, and Task 21 puts it in the runbook rather than leaving it + as tribal knowledge. + - **Does the consolidated file stay appropriate?** One-DB-per-process was chosen partly because Phase 1's tables were small. Adding config + S&F changes the size and write profile of that single file. + **ANSWERED — yes, and the soak is the evidence.** The write profile the consolidated file + must absorb is bounded on every axis: + + | Table | Rate bound | Row size | Basis | + |---|---|---|---| + | `sf_messages` | **≤50 writes/sec** (hard) | 76 B max on the rig | `SweepBatchLimit` ÷ `RetryTimerInterval` | + | `native_alarm_state` | `distinct_source_refs × 10/sec` | small | per-`SourceReference` coalescing on a 100 ms flush (`NativeAlarmActor.cs:473-502`) | + | config tables | deploy-driven, effectively idle | ~721 B max on the rig | Task 1 measurement | + + These are unremarkable for SQLite in WAL mode, and the soak ran the full generator load + against the Phase 1 consolidated file for 30 minutes with **zero** SQLite errors and no + oplog growth pathology. **One DB per process stays.** + + Two caveats carried into the plan rather than hidden here. First, the rig's config rows + are tiny (max 721 B) and **cannot** be treated as representative — the size ceiling that + matters comes from the documented ~60–70 KB production `config_json`, which is what + motivates D6 and `MaxBatchSize = 16`. Second, **D4's premise was wrong**: + `native_alarm_state` is *not* "unbounded by design" and *not* "by a wide margin the + highest-volume table" — the coalescing flush bounds it, and this rig has no alarm + generator at all (measured 0 rows), so its bound is analytic rather than observed. If a + production site ever shows alarm churn that swamps the shared oplog, the keyed-instances + hatch named in D4 remains the escape — it is simply not needed to start. + ## 6. Not in scope (unchanged from the design) `auditlog.db` (diverges per node by design — central pulls the union; replicating it would diff --git a/docs/plans/2026-07-19-localdb-phase2-soak.md b/docs/plans/2026-07-19-localdb-phase2-soak.md index d80c2433..58fa1460 100644 --- a/docs/plans/2026-07-19-localdb-phase2-soak.md +++ b/docs/plans/2026-07-19-localdb-phase2-soak.md @@ -19,6 +19,14 @@ > Finding 1 therefore no longer blocks Task 3. What still stands before proceeding: re-run the > cut-short sampling (Findings 4/5 write-rate numbers) using the safe copy-based `snap` recipe > now in the plan, on a rig where both site-a nodes have been restarted since any host-side read. +> +> ## GATE VERDICT (final, 2026-07-20): **PROCEED to Task 3.** +> +> The re-run is done — [§1b](#1b-clean-re-run-2026-07-20-post-root-cause). Safe copy-based +> sampling, both nodes restarted first, six clean 60-second intervals: **0.80 `sf_messages` +> rows/sec sustained, zero SQLite errors across the full window, both nodes converged.** No stop +> condition from any of D1–D6 is met. The one binding number Phase 2 must honour is +> **`LocalDb:Replication:MaxBatchSize = 16`** (Finding 2 / D6), which Task 19 sets. --- @@ -268,12 +276,59 @@ rate that this run could not measure. --- +## 1b. Clean re-run (2026-07-20, post-root-cause) + +The original sampling was cut short by Finding 1 and was itself the cause of it. Re-run after +both site-a nodes were restarted, using the plan's copy-based `snap()` helper — **no host-side +`sqlite3` ever touched a live file.** Six consecutive 60-second intervals, generator load +unchanged (`SoakGenerator` ×4 against a refusing endpoint): + +| UTC | `sf_messages` rows | `SUM(retry_count)` | status=0 | `__localdb_oplog` | `native_alarm_state` | +|---|---|---|---|---|---| +| 06:07:57 | 3384 | 200 | 3380 | 0 | 0 | +| 06:08:57 | 3432 | 200 | 3428 | 0 | 0 | +| 06:09:57 | 3480 | 200 | 3476 | 0 | 0 | +| 06:10:57 | 3528 | 200 | 3524 | 0 | 0 | +| 06:11:57 | 3576 | 200 | 3572 | 0 | 0 | +| 06:12:57 | 3624 | 200 | 3620 | 0 | 0 | +| 06:13:57 | 3672 | 200 | 3668 | 0 | 0 | + +**Measured:** + +- **`sf_messages` insert rate: exactly 48 rows/min = 0.80 rows/sec**, dead steady across all six + intervals (+288 rows over 360 s, zero variance). +- **Retry-UPDATE rate: 0/sec.** `SUM(retry_count)` never moved. Only 4 rows ever reached + `retry_count = 50` (status 2, dead-lettered); the other ~3.6 k sit at `retry_count = 0`, + status 0 — **enqueued but never swept**. So this generator exercises the *insert* path only. +- **Row sizes:** `sf_messages.payload_json` max **76 B**; `deployed_configurations.config_json` + max **721 B** over 4 rows (rig config is trivially small — see the caveat below). +- **`native_alarm_state`: 0 rows** — no alarm generator on this rig, as previously recorded. +- **`__localdb_oplog`: 0 throughout** — the Phase 1 tables are genuinely idle under this load, + which is the expected result and not a measurement failure (Phase 2's tables are not yet + registered, so this load cannot reach the oplog by construction). +- **Zero SQLite errors** in `docker logs scadabridge-site-a-a` across the full 30-minute window + (`grep -ciE "disk I/O|SQLITE_IOERR|NOTADB"` → **0**). This is the direct confirmation that + Finding 1 was observer-induced: identical load, identical library, nothing host-reading the + files, no errors. +- **Both site-a nodes converged identically** (3564 rows on each at a common sample point) under + the bespoke replicator — the pre-cutover baseline Task 20 should reproduce under CDC. + +**Honest limits of this measurement.** 0.80 rows/sec is **~1.6 % of the 50 rows/sec structural +ceiling**, and the retry path — the expensive one, one `UPDATE` per message per sweep — was never +exercised at all. This re-run therefore confirms *steady-state health and the absence of the +Finding 1 pathology*; it does **not** probe the ceiling. That is acceptable because the ceiling is +**structural rather than empirical** (Finding 4: `SweepBatchLimit` ÷ `RetryTimerInterval`), so +sizing does not depend on observing it. Likewise the rig's 721 B config rows are **not** +representative — D6's sizing rests on the documented ~60–70 KB production `config_json`, not on +this number. + ## 2. What still owes measurement -Carry into the re-run once Finding 1 is fixed: +Carry forward (items 1–2 partially discharged by §1b above): -1. Sustained `sf_messages` rows/sec under a saturating generator (needs many more instances or a - shorter interval to approach the 50/s ceiling). +1. ~~Sustained `sf_messages` rows/sec~~ — **measured** (§1b: 0.80/s insert, 0/s retry). Still + unmeasured: the **retry-UPDATE** path under a saturating generator approaching the 50/s + ceiling. Not required for sizing (the ceiling is structural), but it is the honest gap. 2. Any `native_alarm_state` measurement at all — requires building alarm-source seeding plus an A&C-capable server. The `OpcUaAlarmLiveSmokeTests` **passed**, so the rig's opc-plc *does* answer ConditionRefresh with a `SnapshotComplete`; the missing piece is ongoing transitions From 9e3239c5d933bb76aea6c18a6e93269b8fb2f2ee Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Mon, 20 Jul 2026 02:18:05 -0400 Subject: [PATCH 30/55] refactor(sf): extract StoreAndForwardSchema.Apply from the storage class MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Task 3 of the Phase 2 plan. Mirrors OperationTrackingSchema: the sf_messages DDL, the four additive ALTERs, the last_attempt_at_ms backfill and the due-index move verbatim into a static sync Apply(SqliteConnection), depending only on Microsoft.Data.Sqlite. The Host needs to apply this to a LocalDb-managed connection before RegisterReplicated installs the capture triggers; the store still calls it so a directly-constructed store stays self-sufficient. Two deviations from the plan as written, both deliberate: 1. The PRAGMA journal_mode=WAL in InitializeAsync STAYS. The plan's Step 4 snippet drops it, but Task 4 explicitly says not to move the equivalent pragma ("LocalDb owns the connection's pragmas") — the two tasks contradict each other. Keeping it preserves behaviour for the intermediate commits and for directly-constructed stores; it becomes moot in Task 5 when the store stops opening its own connections. Dropping it now would quietly regress the concurrent-writer support the pragma's own comment documents. 2. Added a second test beyond the plan's. The specified test asserts only against a freshly-created table, where CREATE TABLE already lists all 16 columns — it would pass with every ALTER deleted. The added test starts from the pre-upgrade 12-column shape with a row in it and proves the upgrade path runs and preserves data. That second test also surfaced that the backfill lands 1 ms low: julianday()'s double day-fraction cannot represent every millisecond. Pre-existing behaviour, carried over verbatim, asserted with a 1 ms tolerance rather than pinning a precision the implementation never had. Suite: 154 passed, 0 failed. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts --- .../StoreAndForwardSchema.cs | 124 ++++++++++++++++++ .../StoreAndForwardStorage.cs | 96 +------------- .../StoreAndForwardSchemaTests.cs | 113 ++++++++++++++++ 3 files changed, 242 insertions(+), 91 deletions(-) create mode 100644 src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardSchema.cs create mode 100644 tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardSchemaTests.cs diff --git a/src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardSchema.cs b/src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardSchema.cs new file mode 100644 index 00000000..70a8c270 --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardSchema.cs @@ -0,0 +1,124 @@ +using Microsoft.Data.Sqlite; + +namespace ZB.MOM.WW.ScadaBridge.StoreAndForward; + +/// +/// DDL for the sf_messages table, extracted from +/// so it can be applied by whoever owns the +/// database file. +/// +/// Deliberately depends only on Microsoft.Data.Sqlite, not on the LocalDb library. +/// The Host applies this DDL to a LocalDb-managed connection before +/// RegisterReplicated installs the capture triggers; nothing about the schema +/// itself is LocalDb-specific, and the store still calls it so a directly-constructed +/// store (tests, tooling) remains self-sufficient. +/// +/// +public static class StoreAndForwardSchema +{ + /// + /// Creates the sf_messages table and its indexes when absent, and additively + /// upgrades a table created by an older build. Idempotent — safe to run on every + /// startup. + /// + /// An open connection to the database that should hold the table. + public static void Apply(SqliteConnection connection) + { + ArgumentNullException.ThrowIfNull(connection); + + using (var command = connection.CreateCommand()) + { + command.CommandText = @" + CREATE TABLE IF NOT EXISTS sf_messages ( + id TEXT PRIMARY KEY, + category INTEGER NOT NULL, + target TEXT NOT NULL, + payload_json TEXT NOT NULL, + retry_count INTEGER NOT NULL DEFAULT 0, + max_retries INTEGER NOT NULL DEFAULT 50, + retry_interval_ms INTEGER NOT NULL DEFAULT 30000, + created_at TEXT NOT NULL, + last_attempt_at TEXT, + status INTEGER NOT NULL DEFAULT 0, + last_error TEXT, + origin_instance TEXT + ); + + CREATE INDEX IF NOT EXISTS idx_sf_messages_status ON sf_messages(status); + CREATE INDEX IF NOT EXISTS idx_sf_messages_category ON sf_messages(category); + "; + command.ExecuteNonQuery(); + } + + // Additively add the execution_id / + // source_script columns. CREATE TABLE IF NOT EXISTS above does NOT add + // columns to a table that already exists from before these fields, so a + // databases created by an older build needs the columns ALTER-ed in. + // SQLite has no "ADD COLUMN IF NOT EXISTS"; the column presence is + // probed first and the ALTER skipped when already there. Both columns + // are nullable with no default, so any row buffered before this + // migration reads back ExecutionId/SourceScript = null (back-compat). + AddColumnIfMissing(connection, "execution_id", "TEXT"); + AddColumnIfMissing(connection, "source_script", "TEXT"); + + // Additively add the + // parent_execution_id column the same way — a sibling to execution_id. + // Nullable with no default, so any row buffered before this migration + // reads back ParentExecutionId = null (back-compat). + AddColumnIfMissing(connection, "parent_execution_id", "TEXT"); + + // Additively add the epoch-ms sibling of last_attempt_at. The + // ISO-8601 text column stays authoritative for reads / back-compat; this + // INTEGER column drives the due predicate so the sweep no longer parses + // julianday() per row. + AddColumnIfMissing(connection, "last_attempt_at_ms", "INTEGER"); + + // One-time backfill for rows persisted before the ms column existed: derive + // epoch-ms from the text timestamp. The "... IS NULL" guard makes this run + // once per legacy DB and never match again — this is the only remaining + // julianday() use in the store. + using (var backfill = connection.CreateCommand()) + { + backfill.CommandText = @" + UPDATE sf_messages + SET last_attempt_at_ms = CAST((julianday(last_attempt_at) - 2440587.5) * 86400000 AS INTEGER) + WHERE last_attempt_at IS NOT NULL AND last_attempt_at_ms IS NULL"; + backfill.ExecuteNonQuery(); + } + + // Covering index for the due query (status filter + ms ordering column). + // Created after the ALTER above so the column exists. + using (var dueIndex = connection.CreateCommand()) + { + dueIndex.CommandText = + "CREATE INDEX IF NOT EXISTS idx_sf_messages_status_due ON sf_messages(status, last_attempt_at_ms)"; + dueIndex.ExecuteNonQuery(); + } + } + + /// + /// Adds a column to sf_messages + /// only when it is not already present. SQLite lacks ADD COLUMN IF NOT + /// EXISTS, so the schema is probed via PRAGMA table_info first. + /// Idempotent — safe to run on every . + /// + private static void AddColumnIfMissing( + SqliteConnection connection, string columnName, string columnType) + { + using (var probe = connection.CreateCommand()) + { + probe.CommandText = "SELECT COUNT(*) FROM pragma_table_info('sf_messages') WHERE name = @name"; + probe.Parameters.AddWithValue("@name", columnName); + if (Convert.ToInt32(probe.ExecuteScalar()) > 0) + { + return; + } + } + + using var alter = connection.CreateCommand(); + // Column name + type are caller-controlled constants, never user input — + // safe to interpolate (parameters are not permitted in DDL). + alter.CommandText = $"ALTER TABLE sf_messages ADD COLUMN {columnName} {columnType}"; + alter.ExecuteNonQuery(); + } +} diff --git a/src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardStorage.cs b/src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardStorage.cs index 65f1162d..187d4436 100644 --- a/src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardStorage.cs +++ b/src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardStorage.cs @@ -46,101 +46,15 @@ public class StoreAndForwardStorage await walCmd.ExecuteNonQueryAsync(); } - await using var command = connection.CreateCommand(); - command.CommandText = @" - CREATE TABLE IF NOT EXISTS sf_messages ( - id TEXT PRIMARY KEY, - category INTEGER NOT NULL, - target TEXT NOT NULL, - payload_json TEXT NOT NULL, - retry_count INTEGER NOT NULL DEFAULT 0, - max_retries INTEGER NOT NULL DEFAULT 50, - retry_interval_ms INTEGER NOT NULL DEFAULT 30000, - created_at TEXT NOT NULL, - last_attempt_at TEXT, - status INTEGER NOT NULL DEFAULT 0, - last_error TEXT, - origin_instance TEXT - ); - - CREATE INDEX IF NOT EXISTS idx_sf_messages_status ON sf_messages(status); - CREATE INDEX IF NOT EXISTS idx_sf_messages_category ON sf_messages(category); - "; - await command.ExecuteNonQueryAsync(); - - // Additively add the execution_id / - // source_script columns. CREATE TABLE IF NOT EXISTS above does NOT add - // columns to a table that already exists from before these fields, so a - // databases created by an older build needs the columns ALTER-ed in. - // SQLite has no "ADD COLUMN IF NOT EXISTS"; the column presence is - // probed first and the ALTER skipped when already there. Both columns - // are nullable with no default, so any row buffered before this - // migration reads back ExecutionId/SourceScript = null (back-compat). - await AddColumnIfMissingAsync(connection, "execution_id", "TEXT"); - await AddColumnIfMissingAsync(connection, "source_script", "TEXT"); - - // Additively add the - // parent_execution_id column the same way — a sibling to execution_id. - // Nullable with no default, so any row buffered before this migration - // reads back ParentExecutionId = null (back-compat). - await AddColumnIfMissingAsync(connection, "parent_execution_id", "TEXT"); - - // Additively add the epoch-ms sibling of last_attempt_at. The - // ISO-8601 text column stays authoritative for reads / back-compat; this - // INTEGER column drives the due predicate so the sweep no longer parses - // julianday() per row. - await AddColumnIfMissingAsync(connection, "last_attempt_at_ms", "INTEGER"); - - // One-time backfill for rows persisted before the ms column existed: derive - // epoch-ms from the text timestamp. The "... IS NULL" guard makes this run - // once per legacy DB and never match again — this is the only remaining - // julianday() use in the store. - await using (var backfill = connection.CreateCommand()) - { - backfill.CommandText = @" - UPDATE sf_messages - SET last_attempt_at_ms = CAST((julianday(last_attempt_at) - 2440587.5) * 86400000 AS INTEGER) - WHERE last_attempt_at IS NOT NULL AND last_attempt_at_ms IS NULL"; - await backfill.ExecuteNonQueryAsync(); - } - - // Covering index for the due query (status filter + ms ordering column). - // Created after the ALTER above so the column exists. - await using (var dueIndex = connection.CreateCommand()) - { - dueIndex.CommandText = - "CREATE INDEX IF NOT EXISTS idx_sf_messages_status_due ON sf_messages(status, last_attempt_at_ms)"; - await dueIndex.ExecuteNonQueryAsync(); - } + // The DDL itself lives in StoreAndForwardSchema so the Host can apply it to a + // LocalDb-managed connection before RegisterReplicated installs the capture + // triggers. The store still calls it, so a directly-constructed store (tests, + // tooling) remains self-sufficient. + StoreAndForwardSchema.Apply(connection); _logger.LogInformation("Store-and-forward SQLite storage initialized"); } - /// - /// Adds a column to sf_messages - /// only when it is not already present. SQLite lacks ADD COLUMN IF NOT - /// EXISTS, so the schema is probed via PRAGMA table_info first. - /// Idempotent — safe to run on every . - /// - private static async Task AddColumnIfMissingAsync( - SqliteConnection connection, string columnName, string columnType) - { - await using var probe = connection.CreateCommand(); - probe.CommandText = "SELECT COUNT(*) FROM pragma_table_info('sf_messages') WHERE name = @name"; - probe.Parameters.AddWithValue("@name", columnName); - var exists = Convert.ToInt32(await probe.ExecuteScalarAsync()) > 0; - if (exists) - { - return; - } - - await using var alter = connection.CreateCommand(); - // Column name + type are caller-controlled constants, never user input — - // safe to interpolate (parameters are not permitted in DDL). - alter.CommandText = $"ALTER TABLE sf_messages ADD COLUMN {columnName} {columnType}"; - await alter.ExecuteNonQueryAsync(); - } - /// /// Ensures the directory for a file-backed SQLite database exists. SQLite creates /// the database file on demand but not its parent directory, so a configured path diff --git a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardSchemaTests.cs b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardSchemaTests.cs new file mode 100644 index 00000000..b4279e5a --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardSchemaTests.cs @@ -0,0 +1,113 @@ +using Microsoft.Data.Sqlite; + +namespace ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests; + +/// +/// Pins as the single owner of the +/// sf_messages DDL. The Host applies this to a LocalDb-managed connection +/// before RegisterReplicated installs the capture triggers, so the shape it +/// produces is the shape that replicates. +/// +public class StoreAndForwardSchemaTests +{ + [Fact] + public void Apply_IsIdempotent_AndCreatesEveryColumn() + { + var path = Path.Combine(Path.GetTempPath(), $"sf-schema-{Guid.NewGuid():N}.db"); + try + { + using var connection = new SqliteConnection($"Data Source={path}"); + connection.Open(); + + StoreAndForwardSchema.Apply(connection); + StoreAndForwardSchema.Apply(connection); // second run must not throw + + using var cmd = connection.CreateCommand(); + cmd.CommandText = "SELECT name FROM pragma_table_info('sf_messages')"; + var columns = new List(); + using var reader = cmd.ExecuteReader(); + while (reader.Read()) columns.Add(reader.GetString(0)); + + // 12 original + 4 additive + Assert.Equal(16, columns.Count); + Assert.Contains("last_attempt_at_ms", columns); + Assert.Contains("parent_execution_id", columns); + } + finally + { + SqliteConnection.ClearAllPools(); + File.Delete(path); + } + } + + /// + /// The additive columns exist to upgrade a database created by an older build in + /// place. Asserting only against a freshly-created table would pass even if every + /// ALTER were deleted, because CREATE TABLE already lists them — so + /// this starts from the pre-upgrade shape and proves the ALTER path runs. + /// + [Fact] + public void Apply_UpgradesALegacyTable_WithoutLosingRows() + { + var path = Path.Combine(Path.GetTempPath(), $"sf-schema-legacy-{Guid.NewGuid():N}.db"); + try + { + using var connection = new SqliteConnection($"Data Source={path}"); + connection.Open(); + + // The 12-column shape as it existed before execution_id / source_script / + // parent_execution_id / last_attempt_at_ms were added. + using (var legacy = connection.CreateCommand()) + { + legacy.CommandText = """ + CREATE TABLE sf_messages ( + id TEXT PRIMARY KEY, + category INTEGER NOT NULL, + target TEXT NOT NULL, + payload_json TEXT NOT NULL, + retry_count INTEGER NOT NULL DEFAULT 0, + max_retries INTEGER NOT NULL DEFAULT 50, + retry_interval_ms INTEGER NOT NULL DEFAULT 30000, + created_at TEXT NOT NULL, + last_attempt_at TEXT, + status INTEGER NOT NULL DEFAULT 0, + last_error TEXT, + origin_instance TEXT + ); + INSERT INTO sf_messages (id, category, target, payload_json, created_at, last_attempt_at) + VALUES ('legacy-1', 0, 'target-a', '{}', '2026-01-01T00:00:00Z', '2026-01-02T03:04:05Z'); + """; + legacy.ExecuteNonQuery(); + } + + StoreAndForwardSchema.Apply(connection); + + using var check = connection.CreateCommand(); + check.CommandText = + "SELECT COUNT(*) FROM pragma_table_info('sf_messages') WHERE name = 'last_attempt_at_ms'"; + Assert.Equal(1, Convert.ToInt32(check.ExecuteScalar())); + + // The pre-existing row survives and its epoch-ms sibling is backfilled from + // the authoritative text timestamp. + using var backfilled = connection.CreateCommand(); + backfilled.CommandText = + "SELECT last_attempt_at_ms FROM sf_messages WHERE id = 'legacy-1'"; + var ms = Convert.ToInt64(backfilled.ExecuteScalar()); + var expected = DateTimeOffset.Parse("2026-01-02T03:04:05Z").ToUnixTimeMilliseconds(); + + // Tolerance of 1 ms, not sloppiness: the backfill derives epoch-ms from + // julianday(), whose double-precision day-fraction cannot represent every + // millisecond exactly, so it lands 1 ms low here. That rounding is + // pre-existing production behaviour carried over verbatim from + // StoreAndForwardStorage — the column is a sweep-ordering hint, with the + // ISO-8601 text column remaining authoritative for reads. Asserting exact + // equality would pin a precision the implementation never had. + Assert.InRange(ms, expected - 1, expected + 1); + } + finally + { + SqliteConnection.ClearAllPools(); + File.Delete(path); + } + } +} From ac5eb12cce254da999f1f9c671b63065af186d31 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Mon, 20 Jul 2026 02:20:50 -0400 Subject: [PATCH 31/55] refactor(site): extract SiteStorageSchema.Apply from SiteStorageService MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Task 4 of the Phase 2 plan. All nine table definitions plus MigrateSchemaAsync and TryAddColumnAsync move into a static sync Apply(SqliteConnection) depending only on Microsoft.Data.Sqlite, so the Host can apply the DDL to a LocalDb-managed connection before RegisterReplicated installs the capture triggers. Per the plan, PRAGMA journal_mode=WAL stays in InitializeAsync — LocalDb owns the connection's pragmas, and the service still opens its own connections until Task 6. One deviation: TryAddColumnAsync's `catch (SqliteException) when (ex.Message.Contains("duplicate column"))` becomes a PRAGMA table_info probe, matching OperationTrackingSchema. The message-matching form depends on an error string that is not part of SQLite's contract, and it swallowed every SqliteException whose message happened to contain that substring. The probe also drops the ILogger dependency, which is what let the class become static. Cost: the per-column "Migrated: added column" info log is gone — it fired once per column per legacy database and nothing consumes it. Also added a test beyond the plan's specified one, for the same reason as Task 3: the specified test asserts against a freshly-created database, where CREATE TABLE already lists the migration columns, so it would pass with every ALTER deleted. The added test starts from the pre-migration shapes with a row present and proves the migration path runs and preserves data across a NOT NULL DEFAULT add. SiteRuntime suite 532 passed / 0 failed; full solution build 0 warnings. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts --- .../Persistence/SiteStorageSchema.cs | 169 ++++++++++++++++++ .../Persistence/SiteStorageService.cs | 121 +------------ .../Persistence/SiteStorageSchemaTests.cs | 128 +++++++++++++ 3 files changed, 302 insertions(+), 116 deletions(-) create mode 100644 src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageSchema.cs create mode 100644 tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/SiteStorageSchemaTests.cs diff --git a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageSchema.cs b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageSchema.cs new file mode 100644 index 00000000..b54711b8 --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageSchema.cs @@ -0,0 +1,169 @@ +using Microsoft.Data.Sqlite; + +namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; + +/// +/// DDL for the site's nine configuration tables, extracted from +/// so it can be applied by whoever owns the database +/// file. +/// +/// Deliberately depends only on Microsoft.Data.Sqlite, not on the LocalDb library. +/// The Host applies this DDL to a LocalDb-managed connection before +/// RegisterReplicated installs the capture triggers; nothing about the schema +/// itself is LocalDb-specific, and the service still calls it so a directly-constructed +/// service (tests, tooling) remains self-sufficient. +/// +/// +/// Journal-mode and other connection pragmas are deliberately NOT set here — LocalDb owns +/// the connection's pragmas, and keeps its own WAL set-up +/// for the databases it opens itself. +/// +/// +public static class SiteStorageSchema +{ + /// + /// Creates the nine site configuration tables when absent, and additively upgrades + /// tables created by an older build. Idempotent — safe to run on every startup. + /// + /// An open connection to the database that should hold the tables. + public static void Apply(SqliteConnection connection) + { + ArgumentNullException.ThrowIfNull(connection); + + using (var command = connection.CreateCommand()) + { + command.CommandText = @" + CREATE TABLE IF NOT EXISTS deployed_configurations ( + instance_unique_name TEXT PRIMARY KEY, + config_json TEXT NOT NULL, + deployment_id TEXT NOT NULL, + revision_hash TEXT NOT NULL, + is_enabled INTEGER NOT NULL DEFAULT 1, + deployed_at TEXT NOT NULL + ); + + CREATE TABLE IF NOT EXISTS static_attribute_overrides ( + instance_unique_name TEXT NOT NULL, + attribute_name TEXT NOT NULL, + override_value TEXT NOT NULL, + updated_at TEXT NOT NULL, + PRIMARY KEY (instance_unique_name, attribute_name) + ); + + CREATE TABLE IF NOT EXISTS shared_scripts ( + name TEXT PRIMARY KEY, + code TEXT NOT NULL, + parameter_definitions TEXT, + return_definition TEXT, + updated_at TEXT NOT NULL + ); + + CREATE TABLE IF NOT EXISTS external_systems ( + name TEXT PRIMARY KEY, + endpoint_url TEXT NOT NULL, + auth_type TEXT NOT NULL, + auth_configuration TEXT, + method_definitions TEXT, + updated_at TEXT NOT NULL + ); + + CREATE TABLE IF NOT EXISTS database_connections ( + name TEXT PRIMARY KEY, + connection_string TEXT NOT NULL, + max_retries INTEGER NOT NULL DEFAULT 3, + retry_delay_ms INTEGER NOT NULL DEFAULT 1000, + updated_at TEXT NOT NULL + ); + + CREATE TABLE IF NOT EXISTS notification_lists ( + name TEXT PRIMARY KEY, + recipient_emails TEXT NOT NULL, + updated_at TEXT NOT NULL + ); + + CREATE TABLE IF NOT EXISTS data_connection_definitions ( + name TEXT PRIMARY KEY, + protocol TEXT NOT NULL, + configuration TEXT, + backup_configuration TEXT, + failover_retry_count INTEGER NOT NULL DEFAULT 3, + updated_at TEXT NOT NULL + ); + + CREATE TABLE IF NOT EXISTS smtp_configurations ( + name TEXT PRIMARY KEY, + server TEXT NOT NULL, + port INTEGER NOT NULL, + auth_mode TEXT NOT NULL, + from_address TEXT NOT NULL, + username TEXT, + password TEXT, + oauth_config TEXT, + updated_at TEXT NOT NULL + ); + + CREATE TABLE IF NOT EXISTS native_alarm_state ( + instance_unique_name TEXT NOT NULL, + source_canonical_name TEXT NOT NULL, + source_reference TEXT NOT NULL, + condition_json TEXT NOT NULL, + last_transition_at TEXT NOT NULL, + metadata_json TEXT, + PRIMARY KEY (instance_unique_name, source_canonical_name, source_reference) + ); + "; + command.ExecuteNonQuery(); + } + + // Schema migrations — add columns that may not exist on older databases + MigrateSchema(connection); + } + + private static void MigrateSchema(SqliteConnection connection) + { + // Add backup_configuration and failover_retry_count to data_connection_definitions + // (added in primary/backup data connections feature) + AddColumnIfMissing(connection, "data_connection_definitions", "backup_configuration", "TEXT"); + AddColumnIfMissing(connection, "data_connection_definitions", "failover_retry_count", "INTEGER NOT NULL DEFAULT 3"); + + // Native-alarm display metadata (UA4) — restored on rehydration so a persisted condition + // renders fully (type/category/message/values) before the first source snapshot arrives. + AddColumnIfMissing(connection, "native_alarm_state", "metadata_json", "TEXT"); + + // Per-external-system call timeout (ExternalSystemGateway Timeout) — carried through the + // artifact pipeline so site-side calls honor it; 0 = use the site default. + AddColumnIfMissing(connection, "external_systems", "timeout_seconds", "INTEGER NOT NULL DEFAULT 0"); + } + + /// + /// Additively adds a column only when it is not already present. SQLite lacks + /// ADD COLUMN IF NOT EXISTS, so the schema is probed via + /// PRAGMA table_info first. Mirrors OperationTrackingSchema. + /// + /// This replaces the previous try/catch on SqliteException message text + /// containing "duplicate column": probing is the same precedent the other schema + /// classes use, does not depend on an error string that is not part of SQLite's + /// contract, and does not swallow unrelated failures of the same exception type. + /// + /// + private static void AddColumnIfMissing( + SqliteConnection connection, string table, string columnName, string columnDefinition) + { + using (var probe = connection.CreateCommand()) + { + // Table + column names are caller-controlled constants, never user input — + // safe to interpolate (parameters are not permitted in DDL or in a + // pragma-function argument). + probe.CommandText = $"SELECT COUNT(*) FROM pragma_table_info('{table}') WHERE name = $name"; + probe.Parameters.AddWithValue("$name", columnName); + if (Convert.ToInt32(probe.ExecuteScalar()) > 0) + { + return; + } + } + + using var alter = connection.CreateCommand(); + alter.CommandText = $"ALTER TABLE {table} ADD COLUMN {columnName} {columnDefinition}"; + alter.ExecuteNonQuery(); + } +} diff --git a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs index e02515b2..4ccb0c0b 100644 --- a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs +++ b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs @@ -75,126 +75,15 @@ public class SiteStorageService resultingMode); } - await using var command = connection.CreateCommand(); - command.CommandText = @" - CREATE TABLE IF NOT EXISTS deployed_configurations ( - instance_unique_name TEXT PRIMARY KEY, - config_json TEXT NOT NULL, - deployment_id TEXT NOT NULL, - revision_hash TEXT NOT NULL, - is_enabled INTEGER NOT NULL DEFAULT 1, - deployed_at TEXT NOT NULL - ); - - CREATE TABLE IF NOT EXISTS static_attribute_overrides ( - instance_unique_name TEXT NOT NULL, - attribute_name TEXT NOT NULL, - override_value TEXT NOT NULL, - updated_at TEXT NOT NULL, - PRIMARY KEY (instance_unique_name, attribute_name) - ); - - CREATE TABLE IF NOT EXISTS shared_scripts ( - name TEXT PRIMARY KEY, - code TEXT NOT NULL, - parameter_definitions TEXT, - return_definition TEXT, - updated_at TEXT NOT NULL - ); - - CREATE TABLE IF NOT EXISTS external_systems ( - name TEXT PRIMARY KEY, - endpoint_url TEXT NOT NULL, - auth_type TEXT NOT NULL, - auth_configuration TEXT, - method_definitions TEXT, - updated_at TEXT NOT NULL - ); - - CREATE TABLE IF NOT EXISTS database_connections ( - name TEXT PRIMARY KEY, - connection_string TEXT NOT NULL, - max_retries INTEGER NOT NULL DEFAULT 3, - retry_delay_ms INTEGER NOT NULL DEFAULT 1000, - updated_at TEXT NOT NULL - ); - - CREATE TABLE IF NOT EXISTS notification_lists ( - name TEXT PRIMARY KEY, - recipient_emails TEXT NOT NULL, - updated_at TEXT NOT NULL - ); - - CREATE TABLE IF NOT EXISTS data_connection_definitions ( - name TEXT PRIMARY KEY, - protocol TEXT NOT NULL, - configuration TEXT, - backup_configuration TEXT, - failover_retry_count INTEGER NOT NULL DEFAULT 3, - updated_at TEXT NOT NULL - ); - - CREATE TABLE IF NOT EXISTS smtp_configurations ( - name TEXT PRIMARY KEY, - server TEXT NOT NULL, - port INTEGER NOT NULL, - auth_mode TEXT NOT NULL, - from_address TEXT NOT NULL, - username TEXT, - password TEXT, - oauth_config TEXT, - updated_at TEXT NOT NULL - ); - - CREATE TABLE IF NOT EXISTS native_alarm_state ( - instance_unique_name TEXT NOT NULL, - source_canonical_name TEXT NOT NULL, - source_reference TEXT NOT NULL, - condition_json TEXT NOT NULL, - last_transition_at TEXT NOT NULL, - metadata_json TEXT, - PRIMARY KEY (instance_unique_name, source_canonical_name, source_reference) - ); - "; - await command.ExecuteNonQueryAsync(); - - // Schema migrations — add columns that may not exist on older databases - await MigrateSchemaAsync(connection); + // The DDL itself lives in SiteStorageSchema so the Host can apply it to a + // LocalDb-managed connection before RegisterReplicated installs the capture + // triggers. The service still calls it, so a directly-constructed service + // (tests, tooling) remains self-sufficient. + SiteStorageSchema.Apply(connection); _logger.LogInformation("Site SQLite storage initialized at {ConnectionString}", _connectionString); } - private async Task MigrateSchemaAsync(SqliteConnection connection) - { - // Add backup_configuration and failover_retry_count to data_connection_definitions - // (added in primary/backup data connections feature) - await TryAddColumnAsync(connection, "data_connection_definitions", "backup_configuration", "TEXT"); - await TryAddColumnAsync(connection, "data_connection_definitions", "failover_retry_count", "INTEGER NOT NULL DEFAULT 3"); - - // Native-alarm display metadata (UA4) — restored on rehydration so a persisted condition - // renders fully (type/category/message/values) before the first source snapshot arrives. - await TryAddColumnAsync(connection, "native_alarm_state", "metadata_json", "TEXT"); - - // Per-external-system call timeout (ExternalSystemGateway Timeout) — carried through the - // artifact pipeline so site-side calls honor it; 0 = use the site default. - await TryAddColumnAsync(connection, "external_systems", "timeout_seconds", "INTEGER NOT NULL DEFAULT 0"); - } - - private async Task TryAddColumnAsync(SqliteConnection connection, string table, string column, string type) - { - try - { - await using var cmd = connection.CreateCommand(); - cmd.CommandText = $"ALTER TABLE {table} ADD COLUMN {column} {type}"; - await cmd.ExecuteNonQueryAsync(); - _logger.LogInformation("Migrated: added column {Column} to {Table}", column, table); - } - catch (SqliteException ex) when (ex.Message.Contains("duplicate column")) - { - // Column already exists — no action needed - } - } - // ── Deployed Configuration CRUD ── /// diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/SiteStorageSchemaTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/SiteStorageSchemaTests.cs new file mode 100644 index 00000000..83b49ddf --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/SiteStorageSchemaTests.cs @@ -0,0 +1,128 @@ +using Microsoft.Data.Sqlite; +using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; + +namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Persistence; + +/// +/// Pins as the single owner of the site config DDL. +/// The Host applies this to a LocalDb-managed connection before +/// RegisterReplicated installs the capture triggers, so the shape it produces +/// is the shape that replicates. +/// +public class SiteStorageSchemaTests +{ + [Fact] + public void Apply_IsIdempotent_AndCreatesEveryTable() + { + var path = Path.Combine(Path.GetTempPath(), $"site-schema-{Guid.NewGuid():N}.db"); + try + { + using var connection = new SqliteConnection($"Data Source={path}"); + connection.Open(); + + SiteStorageSchema.Apply(connection); + SiteStorageSchema.Apply(connection); // second run must not throw + + // Every table is named explicitly on purpose — a loop asserting "9 tables" + // would still pass if one were renamed. + foreach (var table in new[] + { + "deployed_configurations", "static_attribute_overrides", "shared_scripts", + "external_systems", "database_connections", "notification_lists", + "data_connection_definitions", "smtp_configurations", "native_alarm_state", + }) + { + Assert.True(TableExists(connection, table), $"missing table: {table}"); + } + } + finally + { + SqliteConnection.ClearAllPools(); + File.Delete(path); + } + } + + /// + /// The migration columns exist to upgrade a database created by an older build in + /// place. Asserting only against a freshly-created table would pass even if every + /// ALTER were deleted, because CREATE TABLE already lists them — so + /// this starts from the pre-migration shape and proves the migration path runs. + /// + [Fact] + public void Apply_AddsMigrationColumns_ToALegacyDatabase() + { + var path = Path.Combine(Path.GetTempPath(), $"site-schema-legacy-{Guid.NewGuid():N}.db"); + try + { + using var connection = new SqliteConnection($"Data Source={path}"); + connection.Open(); + + // The shapes as they existed before backup_configuration / + // failover_retry_count / metadata_json / timeout_seconds were added. + using (var legacy = connection.CreateCommand()) + { + legacy.CommandText = """ + CREATE TABLE data_connection_definitions ( + name TEXT PRIMARY KEY, + protocol TEXT NOT NULL, + configuration TEXT, + updated_at TEXT NOT NULL + ); + CREATE TABLE native_alarm_state ( + instance_unique_name TEXT NOT NULL, + source_canonical_name TEXT NOT NULL, + source_reference TEXT NOT NULL, + condition_json TEXT NOT NULL, + last_transition_at TEXT NOT NULL, + PRIMARY KEY (instance_unique_name, source_canonical_name, source_reference) + ); + CREATE TABLE external_systems ( + name TEXT PRIMARY KEY, + endpoint_url TEXT NOT NULL, + auth_type TEXT NOT NULL, + auth_configuration TEXT, + method_definitions TEXT, + updated_at TEXT NOT NULL + ); + INSERT INTO external_systems (name, endpoint_url, auth_type, updated_at) + VALUES ('legacy-system', 'http://example.invalid', 'None', '2026-01-01T00:00:00Z'); + """; + legacy.ExecuteNonQuery(); + } + + SiteStorageSchema.Apply(connection); + + Assert.True(ColumnExists(connection, "data_connection_definitions", "backup_configuration")); + Assert.True(ColumnExists(connection, "data_connection_definitions", "failover_retry_count")); + Assert.True(ColumnExists(connection, "native_alarm_state", "metadata_json")); + Assert.True(ColumnExists(connection, "external_systems", "timeout_seconds")); + + // The pre-existing row survives the upgrade and reads back the NOT NULL + // column's default rather than failing. + using var check = connection.CreateCommand(); + check.CommandText = "SELECT timeout_seconds FROM external_systems WHERE name = 'legacy-system'"; + Assert.Equal(0, Convert.ToInt32(check.ExecuteScalar())); + } + finally + { + SqliteConnection.ClearAllPools(); + File.Delete(path); + } + } + + private static bool TableExists(SqliteConnection connection, string table) + { + using var cmd = connection.CreateCommand(); + cmd.CommandText = "SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = $name"; + cmd.Parameters.AddWithValue("$name", table); + return Convert.ToInt32(cmd.ExecuteScalar()) > 0; + } + + private static bool ColumnExists(SqliteConnection connection, string table, string column) + { + using var cmd = connection.CreateCommand(); + cmd.CommandText = $"SELECT COUNT(*) FROM pragma_table_info('{table}') WHERE name = $name"; + cmd.Parameters.AddWithValue("$name", column); + return Convert.ToInt32(cmd.ExecuteScalar()) > 0; + } +} From 3dfb288b7487f4f18b1421d9c537c03ee6bb10e8 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Mon, 20 Jul 2026 02:21:09 -0400 Subject: [PATCH 32/55] chore(localdb): record tasks 3-4 completion and their deviations in the task state Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts --- docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json b/docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json index c3006ce1..69cc6c29 100644 --- a/docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json +++ b/docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json @@ -71,8 +71,8 @@ "tasks": [ {"id": 1, "subject": "Task 1: Rig soak - measure legacy-DB write rates + config_json sizes (corrected method)", "status": "completed", "classification": "high-risk", "note": "GATE CLOSED - verdict PROCEED (2026-07-20). NEVER sample host-side sqlite3 on live bind-mounted WAL files (that method poisoned the first run - use the copy-based snap() helper). Clean re-run: sf_messages 0.80 rows/sec insert, 0/sec retry-UPDATE, oplog 0, alarms 0, max payload_json 76 B, max config_json 721 B (NOT representative), zero SQLite errors in 30 min, both nodes converged. No stop condition met. Binding output: MaxBatchSize 500 -> 16 (Task 19). Empirical drain check remains Task 20 evidence 10."}, {"id": 2, "subject": "Task 2: Decision record - close the phase 2 gate", "status": "completed", "classification": "trivial", "blockedBy": [1], "note": "Gate doc status flipped NOT STARTED -> CLOSED; all 5 SS5 questions answered inline (questions kept, not deleted); SS2's N5 duplicate-bound requirement routed explicitly to Task 21."}, - {"id": 3, "subject": "Task 3: Extract StoreAndForwardSchema.Apply", "status": "pending", "classification": "small", "blockedBy": [2]}, - {"id": 4, "subject": "Task 4: Extract SiteStorageSchema.Apply", "status": "pending", "classification": "small", "blockedBy": [2]}, + {"id": 3, "subject": "Task 3: Extract StoreAndForwardSchema.Apply", "status": "completed", "classification": "small", "blockedBy": [2], "note": "DEVIATION: kept PRAGMA journal_mode=WAL in InitializeAsync - the plan Step 4 snippet drops it but Task 4 says not to move the equivalent pragma; contradictory, and dropping it would regress documented concurrent-writer support before Task 5 makes it moot. Added a legacy-upgrade test beyond the plan's (the specified test asserts against a FRESH table where CREATE TABLE already lists all 16 columns - it would pass with every ALTER deleted). That test found the last_attempt_at_ms backfill lands 1 ms low: julianday() double day-fraction rounding, pre-existing, carried verbatim, asserted with 1 ms tolerance. Suite 154/154."}, + {"id": 4, "subject": "Task 4: Extract SiteStorageSchema.Apply", "status": "completed", "classification": "small", "blockedBy": [2], "note": "DEVIATION: TryAddColumnAsync's catch-on-message-text ('duplicate column') became a PRAGMA table_info probe matching OperationTrackingSchema - the message form depends on a non-contractual error string and swallowed unrelated SqliteExceptions; also dropped the ILogger dep, which is what let the class be static. Cost: the per-column 'Migrated: added column' info log is gone (nothing consumes it). PRAGMA journal_mode=WAL stays in InitializeAsync per plan. Added a legacy-upgrade test for the same reason as Task 3. SiteRuntime 532/532, full solution build 0 warnings."}, {"id": 5, "subject": "Task 5: Rewire StoreAndForwardStorage onto ILocalDb", "status": "pending", "classification": "high-risk", "blockedBy": [3]}, {"id": 6, "subject": "Task 6: Rewire SiteStorageService onto ILocalDb", "status": "pending", "classification": "high-risk", "blockedBy": [4]}, {"id": 7, "subject": "Task 7: Extend SiteLocalDbSetup with the new DDL (not yet registered)", "status": "pending", "classification": "standard", "blockedBy": [5, 6]}, From f2efeb37b73762aee672ac6eb068a192ff0856ee Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Mon, 20 Jul 2026 03:05:45 -0400 Subject: [PATCH 33/55] refactor(sf,site): both stores take ILocalDb instead of a connection string MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Tasks 5 and 6 of the Phase 2 plan, committed together because their test fallout is entangled — several fixtures construct both stores. StoreAndForwardStorage and SiteStorageService now take ILocalDb. Connections come from ILocalDb.CreateConnection(), which hands out an already-open, pragma-configured connection carrying the zb_hlc_next() UDF the capture triggers call; a raw connection would lack the UDF and every write to a replicated table would fail closed. Deleted with the connection strings: S&F's EnsureDatabaseDirectoryExists and its per-open busy_timeout pragma, and the site service's BusyTimeoutFloorSeconds normalization — LocalDb owns all of it now. DI: AddSiteRuntime's string overload is gone (nothing left to supply), so the Host calls the no-arg form. ScadaBridge:Database:SiteDbPath and StoreAndForwardOptions.SqliteDbPath survive only as the migrator's source locations in Tasks 8/9. Two things the plan did not anticipate, both worth reading: 1. FOUND A REAL LATENT DEFECT, from Phase 1, now fixed. The plan assumed directory creation simply moved to LocalDb along with file ownership. It did not: the LocalDb library never creates the parent directory, and SqliteLocalDb opens the file eagerly in its constructor — so a missing directory is a hard boot failure ("SQLite Error 14: unable to open database file"), not a degraded start. The default site config points at the RELATIVE path ./data/site-localdb.db, so any site node without a pre-existing data/ directory fails to boot. The docker rig escapes only because its volume mount happens to create /app/data — a coincidence that would have hidden this until a bare-metal or fresh deployment. This has been latent since Phase 1 made LocalDb:Path required; deleting S&F's EnsureDatabaseDirectoryExists here would have widened it. Re-established the guarantee at the layer that now owns the path (SiteLocalDbDirectory.Ensure, called before AddZbLocalDb) and pinned it with SiteLocalDbDirectoryTests. Non-vacuity is not assumed: two tests written against the wrong assumption failed with exactly this SQLite Error 14 before the fix existed. 2. Test fallout was ~7x the plan's estimate. The plan named "fixtures" in one project; the constructor change actually reaches 40 files across 7 test projects, and most used Mode=Memory;Cache=Shared — which LocalDb has no equivalent for, so every one had to move to a real temp file. Rather than copy the Phase 1 TestLocalDb fixture into 7 projects, added a shared tests/ZB.MOM.WW.ScadaBridge.TestSupport library (not a test project) so the WAL-sidecar cleanup and the "real, not stubbed" rationale live in one place. Retargeted rather than deleted, in both directions: the S&F WAL test now asserts against the LocalDb-backed store (WAL genuinely is LocalDb's job), while the directory-creation test moved to Host.Tests (that guarantee is NOT LocalDb's). SiteStorageServiceTests.Initialize_EnablesWalJournalMode got the same treatment. DeploymentManagerMediumFindingsTests induced a persistence failure via an unopenable path, which no longer reaches the assertion since the fixture now throws first; it induces the same failure shape via an uninitialized store. Verified: full solution build 0 warnings; SiteRuntime 532, Host 318, AuditLog 355, ExternalSystemGateway 142, HealthMonitoring 97, StoreAndForward 153 — 1597 passed, 0 failed. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts --- ZB.MOM.WW.ScadaBridge.slnx | 1 + .../SiteLocalDbDirectory.cs | 59 +++ .../SiteServiceRegistration.cs | 23 +- .../Persistence/SiteStorageService.cs | 143 +++----- .../SiteExternalSystemRepository.cs | 15 +- .../ServiceCollectionExtensions.cs | 30 +- .../ServiceCollectionExtensions.cs | 15 +- .../StoreAndForwardStorage.cs | 119 +++---- .../ParentExecutionIdCorrelationTests.cs | 335 +++++++++--------- ...B.MOM.WW.ScadaBridge.AuditLog.Tests.csproj | 3 +- .../DatabaseGatewayTests.cs | 121 ++++--- .../ExternalSystemClientTests.cs | 85 +++-- ...aBridge.ExternalSystemGateway.Tests.csproj | 3 +- .../HealthReportSenderTests.cs | 89 ++--- ....ScadaBridge.HealthMonitoring.Tests.csproj | 3 +- .../SiteLocalDbDirectoryTests.cs | 70 ++++ .../Cluster/SfBufferResyncPredicateTests.cs | 78 ++-- .../DualNodeRecoveryTests.cs | 51 ++- .../IntegrationSurfaceTests.cs | 10 +- .../RecoveryDrillTests.cs | 32 +- .../SiteFailoverTests.cs | 52 ++- ...MOM.WW.ScadaBridge.IntegrationTests.csproj | 3 +- .../Actors/DeploymentManagerActorTests.cs | 13 +- .../DeploymentManagerCertReconcileTests.cs | 13 +- .../DeploymentManagerLoggerFactoryTests.cs | 13 +- .../DeploymentManagerMediumFindingsTests.cs | 65 ++-- .../Actors/DeploymentManagerRedeployTests.cs | 13 +- .../InstanceActorChildAttributeRaceTests.cs | 13 +- .../Actors/InstanceActorChildRoutingTests.cs | 13 +- .../Actors/InstanceActorIntegrationTests.cs | 13 +- .../Actors/InstanceActorNativeAlarmTests.cs | 13 +- .../Actors/InstanceActorSetAttributeTests.cs | 13 +- .../Actors/InstanceActorTests.cs | 13 +- .../InstanceActorWaitForAttributeTests.cs | 13 +- .../Actors/NativeAlarmActorTests.cs | 20 +- .../Actors/SiteReconciliationActorTests.cs | 17 +- .../Actors/SiteReplicationActorTests.cs | 21 +- .../NegativeTests.cs | 51 ++- .../Persistence/ArtifactStorageTests.cs | 29 +- .../Persistence/NativeAlarmStateStoreTests.cs | 20 +- .../Persistence/SiteStorageServiceTests.cs | 41 ++- .../Repositories/SiteRepositoryTests.cs | 20 +- .../Scripts/NotifyHelperTests.cs | 16 +- .../Scripts/NotifySendAuditEmissionTests.cs | 16 +- ...OM.WW.ScadaBridge.SiteRuntime.Tests.csproj | 3 +- .../CachedCallAttemptEmissionTests.cs | 32 +- .../ParkedMessageHandlerActorTests.cs | 18 +- .../ParkedOperationRelayTests.cs | 18 +- .../QueueDepthGaugeTests.cs | 18 +- .../ReplicationServiceTests.cs | 18 +- .../StoreAndForwardReplicationTests.cs | 17 +- .../StoreAndForwardServiceTests.cs | 47 +-- .../StoreAndForwardSiteEventTests.cs | 18 +- .../StoreAndForwardStorageTests.cs | 105 +++--- ...W.ScadaBridge.StoreAndForward.Tests.csproj | 3 +- .../TestLocalDb.cs | 91 +++++ .../ZB.MOM.WW.ScadaBridge.TestSupport.csproj | 29 ++ 57 files changed, 1368 insertions(+), 848 deletions(-) create mode 100644 src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbDirectory.cs create mode 100644 tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbDirectoryTests.cs create mode 100644 tests/ZB.MOM.WW.ScadaBridge.TestSupport/TestLocalDb.cs create mode 100644 tests/ZB.MOM.WW.ScadaBridge.TestSupport/ZB.MOM.WW.ScadaBridge.TestSupport.csproj diff --git a/ZB.MOM.WW.ScadaBridge.slnx b/ZB.MOM.WW.ScadaBridge.slnx index 9f3c14d7..21748f21 100644 --- a/ZB.MOM.WW.ScadaBridge.slnx +++ b/ZB.MOM.WW.ScadaBridge.slnx @@ -36,6 +36,7 @@ + diff --git a/src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbDirectory.cs b/src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbDirectory.cs new file mode 100644 index 00000000..c850ea3b --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbDirectory.cs @@ -0,0 +1,59 @@ +using Microsoft.Extensions.Configuration; + +namespace ZB.MOM.WW.ScadaBridge.Host; + +/// +/// Ensures the directory holding the consolidated site database (LocalDb:Path) +/// exists before LocalDb opens the file. +/// +/// +/// +/// SQLite creates the database file on demand but never its parent directory, and +/// SqliteLocalDb opens the file eagerly in its constructor — so a missing directory +/// is a hard boot failure ("SQLite Error 14: unable to open database file"), not a degraded +/// start. Neither the LocalDb library nor its options validation does this. +/// +/// +/// The guarantee previously lived in StoreAndForwardStorage.EnsureDatabaseDirectoryExists, +/// which created the directory for the store's own SQLite file. Once that file was folded +/// into the consolidated database the store stopped owning a path, so the guarantee had to +/// be re-established at the layer that now configures the path: the site host. +/// +/// +/// It matters because the default site configuration uses the relative path +/// ./data/site-localdb.db. The docker rig never noticed the gap only because its +/// volume mount happens to create /app/data. +/// +/// +public static class SiteLocalDbDirectory +{ + /// + /// Creates the directory containing LocalDb:Path when it does not already exist. + /// No-op when the key is unset. + /// + /// + /// Best-effort by design: if the directory cannot be created (permissions, read-only + /// mount) this stays silent and lets LocalDb raise the real error, which names the + /// actual path and is a better diagnostic than anything thrown from here. + /// + /// Configuration carrying LocalDb:Path. + public static void Ensure(IConfiguration config) + { + ArgumentNullException.ThrowIfNull(config); + + var path = config["LocalDb:Path"]; + if (string.IsNullOrWhiteSpace(path)) + return; + + try + { + var directory = Path.GetDirectoryName(Path.GetFullPath(path)); + if (!string.IsNullOrEmpty(directory) && !Directory.Exists(directory)) + Directory.CreateDirectory(directory); + } + catch (Exception ex) when (ex is IOException or UnauthorizedAccessException or ArgumentException) + { + // Deliberately swallowed — see the remarks above. + } + } +} diff --git a/src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs b/src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs index afc500f0..3e696238 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs @@ -65,10 +65,13 @@ public static class SiteServiceRegistration services.AddSingleton(); services.AddSingleton(); - // Site-only components — AddSiteRuntime registers SiteStorageService with SQLite path - // and site-local repository implementations (IExternalSystemRepository, INotificationRepository) - var siteDbPath = config["ScadaBridge:Database:SiteDbPath"] ?? "site.db"; - services.AddSiteRuntime($"Data Source={siteDbPath}"); + // Site-only components — AddSiteRuntime registers SiteStorageService and the + // site-local repository implementations (IExternalSystemRepository, + // INotificationRepository). It takes no connection string any more: + // SiteStorageService persists to the consolidated LocalDb database registered + // just below (LocalDb:Path). ScadaBridge:Database:SiteDbPath survives only as + // the legacy migrator's source location. + services.AddSiteRuntime(); // Consolidated site database (LocalDb Phase 1). Holds OperationTracking and // site_events as replicated tables so the pair stops losing them on failover. @@ -78,6 +81,18 @@ public static class SiteServiceRegistration // initiator idles. // // Design: scadaproj docs/plans/2026-07-19-scadabridge-localdb-design.md + // + // Create the parent directory FIRST. SQLite creates the database file on demand + // but not its directory, and neither the LocalDb library nor its options + // validation does this — SqliteLocalDb's constructor opens the file eagerly, so a + // missing directory is a hard boot failure ("SQLite Error 14: unable to open + // database file"), not a degraded start. The default site path is the relative + // "./data/site-localdb.db", so this bites any site node whose data directory has + // not been pre-created; the docker rig only escapes it because the volume mount + // creates /app/data. StoreAndForwardStorage used to do this for its own file + // (EnsureDatabaseDirectoryExists) and that guarantee has to keep existing + // somewhere now that LocalDb owns the file. + SiteLocalDbDirectory.Ensure(config); services.AddZbLocalDb(config, db => SiteLocalDbSetup.OnReady(db, config)); // The replication engine, likewise unconditional but INERT by default: with no diff --git a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs index 4ccb0c0b..746e8547 100644 --- a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs +++ b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs @@ -1,5 +1,6 @@ using Microsoft.Data.Sqlite; using Microsoft.Extensions.Logging; +using ZB.MOM.WW.LocalDb; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; @@ -10,70 +11,59 @@ namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; /// public class SiteStorageService { - private readonly string _connectionString; + private readonly ILocalDb _localDb; private readonly ILogger _logger; /// - /// Initializes a new instance of the SiteStorageService with the specified SQLite connection string and logger. + /// Initializes the service over the consolidated site database. /// - /// SQLite connection string for the site database. + /// + /// The consolidated site database. Every connection it hands out is already open and carries + /// the per-connection pragmas (including the busy timeout that this class used to pin itself) + /// plus the zb_hlc_next() UDF that the config tables' capture triggers call — which is + /// exactly why the service no longer builds its own connection string. A raw connection would + /// lack the UDF and every write to a replicated table would fail closed. + /// /// Logger instance for diagnostic messages. - public SiteStorageService(string connectionString, ILogger logger) + public SiteStorageService(ILocalDb localDb, ILogger logger) { - // Normalize the connection string and pin a busy-timeout floor (S8). WAL lets a reader - // and a writer proceed concurrently, but two writers still contend; Microsoft.Data.Sqlite - // drives its busy handler off the command timeout, so a floor of 5 s means a briefly - // busy database is waited-on rather than failing fast with SQLITE_BUSY. Only raise a - // caller-supplied value that is lower than the floor (the library default of 30 s stays). - var builder = new SqliteConnectionStringBuilder(connectionString); - if (builder.DefaultTimeout < BusyTimeoutFloorSeconds) - builder.DefaultTimeout = BusyTimeoutFloorSeconds; - _connectionString = builder.ToString(); + ArgumentNullException.ThrowIfNull(localDb); + ArgumentNullException.ThrowIfNull(logger); + _localDb = localDb; _logger = logger; } - /// Busy-timeout floor in seconds applied to the site connection string (S8). - private const int BusyTimeoutFloorSeconds = 5; + /// + /// Returns an already-open connection against the site database. + /// Exposed so site-local repositories can get their own connections without reaching + /// into private state via reflection. The caller owns the connection and is + /// responsible for disposing it. + /// + /// + /// Contract change: this used to return an unopened connection that the + /// caller opened. LocalDb hands out connections already open, pragma-configured, and + /// carrying the zb_hlc_next() UDF — calling Open/OpenAsync on one + /// throws. Callers must not open it. + /// + /// An open against the site database. + public SqliteConnection CreateConnection() => _localDb.CreateConnection(); /// - /// Creates a new (unopened) SQLite connection against the site database. - /// Exposed so site-local repositories can open their own connections without - /// reaching into private state via reflection. The caller owns - /// the connection and is responsible for opening and disposing it. - /// - /// The database runs in WAL journal mode (set once in ) with a - /// busy-timeout floor (see the constructor), so connections handed out here inherit - /// concurrent-reader/writer behavior and wait out a briefly-busy database instead of - /// failing with SQLITE_BUSY. - /// + /// Convenience alias used internally, mirroring the other LocalDb-backed stores. /// - /// A new, unopened against the site database. - public SqliteConnection CreateConnection() => new(_connectionString); + private SqliteConnection OpenConnection() => _localDb.CreateConnection(); /// /// Creates the SQLite tables if they do not exist. /// Called once on site startup. /// /// A task that completes when all tables have been created or verified. - public async Task InitializeAsync() + public Task InitializeAsync() { - await using var connection = new SqliteConnection(_connectionString); - await connection.OpenAsync(); - - // Switch to WAL journal mode (S8). WAL is persistent (database-level, survives across - // connections) so this one-time set is enough. It lets readers and a writer proceed - // concurrently instead of every access serializing on the rollback journal. A - // :memory: database or a filesystem that cannot support WAL falls back to its prior - // mode — we log the result rather than throwing. - await using (var walCommand = connection.CreateCommand()) - { - walCommand.CommandText = "PRAGMA journal_mode=WAL;"; - var resultingMode = (await walCommand.ExecuteScalarAsync()) as string ?? "unknown"; - if (!string.Equals(resultingMode, "wal", StringComparison.OrdinalIgnoreCase)) - _logger.LogWarning( - "Site SQLite could not enable WAL journal mode (got '{Mode}') — concurrent access may serialize", - resultingMode); - } + // No journal-mode pragma and no busy-timeout normalization here any more: + // LocalDb owns the file and sets WAL plus the per-connection pragmas on every + // connection it hands out. + using var connection = OpenConnection(); // The DDL itself lives in SiteStorageSchema so the Host can apply it to a // LocalDb-managed connection before RegisterReplicated installs the capture @@ -81,7 +71,8 @@ public class SiteStorageService // (tests, tooling) remains self-sufficient. SiteStorageSchema.Apply(connection); - _logger.LogInformation("Site SQLite storage initialized at {ConnectionString}", _connectionString); + _logger.LogInformation("Site SQLite storage initialized"); + return Task.CompletedTask; } // ── Deployed Configuration CRUD ── @@ -92,8 +83,7 @@ public class SiteStorageService /// A task that resolves to the list of all deployed instance configurations. public async Task> GetAllDeployedConfigsAsync() { - await using var connection = new SqliteConnection(_connectionString); - await connection.OpenAsync(); + await using var connection = OpenConnection(); await using var command = connection.CreateCommand(); command.CommandText = @" @@ -134,8 +124,7 @@ public class SiteStorageService string revisionHash, bool isEnabled) { - await using var connection = new SqliteConnection(_connectionString); - await connection.OpenAsync(); + await using var connection = OpenConnection(); await using var command = connection.CreateCommand(); command.CommandText = @" @@ -197,8 +186,7 @@ public class SiteStorageService { var deployedAt = (deployedAtOverride ?? DateTimeOffset.UtcNow).ToString("O"); - await using var connection = new SqliteConnection(_connectionString); - await connection.OpenAsync(); + await using var connection = OpenConnection(); await using var command = connection.CreateCommand(); command.CommandText = @" @@ -231,8 +219,7 @@ public class SiteStorageService /// A task that completes when the configuration and its overrides have been removed. public async Task RemoveDeployedConfigAsync(string instanceName) { - await using var connection = new SqliteConnection(_connectionString); - await connection.OpenAsync(); + await using var connection = OpenConnection(); await using var transaction = await connection.BeginTransactionAsync(); @@ -272,8 +259,7 @@ public class SiteStorageService /// A task that completes when the enabled flag has been updated. public async Task SetInstanceEnabledAsync(string instanceName, bool isEnabled) { - await using var connection = new SqliteConnection(_connectionString); - await connection.OpenAsync(); + await using var connection = OpenConnection(); await using var command = connection.CreateCommand(); command.CommandText = @" @@ -300,8 +286,7 @@ public class SiteStorageService /// A task that resolves to a dictionary mapping attribute names to their override values. public async Task> GetStaticOverridesAsync(string instanceName) { - await using var connection = new SqliteConnection(_connectionString); - await connection.OpenAsync(); + await using var connection = OpenConnection(); await using var command = connection.CreateCommand(); command.CommandText = @" @@ -329,8 +314,7 @@ public class SiteStorageService /// A task that completes when the override has been saved. public async Task SetStaticOverrideAsync(string instanceName, string attributeName, string value) { - await using var connection = new SqliteConnection(_connectionString); - await connection.OpenAsync(); + await using var connection = OpenConnection(); await using var command = connection.CreateCommand(); command.CommandText = @" @@ -356,8 +340,7 @@ public class SiteStorageService /// A task that completes when all overrides for the instance have been deleted. public async Task ClearStaticOverridesAsync(string instanceName) { - await using var connection = new SqliteConnection(_connectionString); - await connection.OpenAsync(); + await using var connection = OpenConnection(); await using var command = connection.CreateCommand(); command.CommandText = "DELETE FROM static_attribute_overrides WHERE instance_unique_name = @name"; @@ -384,8 +367,7 @@ public class SiteStorageService string instanceName, string sourceCanonicalName, string sourceReference, string conditionJson, DateTimeOffset lastTransitionAt, string? metadataJson = null) { - await using var connection = new SqliteConnection(_connectionString); - await connection.OpenAsync(); + await using var connection = OpenConnection(); await using var command = connection.CreateCommand(); command.CommandText = @" @@ -424,8 +406,7 @@ public class SiteStorageService if (rows.Count == 0) return; - await using var connection = new SqliteConnection(_connectionString); - await connection.OpenAsync(); + await using var connection = OpenConnection(); await using var transaction = (SqliteTransaction)await connection.BeginTransactionAsync(); await using var command = connection.CreateCommand(); @@ -469,8 +450,7 @@ public class SiteStorageService /// A task that completes when the alarm condition row has been deleted. public async Task DeleteNativeAlarmAsync(string instanceName, string sourceCanonicalName, string sourceReference) { - await using var connection = new SqliteConnection(_connectionString); - await connection.OpenAsync(); + await using var connection = OpenConnection(); await using var command = connection.CreateCommand(); command.CommandText = @" @@ -494,8 +474,7 @@ public class SiteStorageService /// A task that resolves to the list of stored native alarm condition rows for the binding. public async Task> GetNativeAlarmsAsync(string instanceName, string sourceCanonicalName) { - await using var connection = new SqliteConnection(_connectionString); - await connection.OpenAsync(); + await using var connection = OpenConnection(); await using var command = connection.CreateCommand(); command.CommandText = @" @@ -526,8 +505,7 @@ public class SiteStorageService /// A task that completes when all native alarm rows for the instance have been deleted. public async Task ClearNativeAlarmsForInstanceAsync(string instanceName) { - await using var connection = new SqliteConnection(_connectionString); - await connection.OpenAsync(); + await using var connection = OpenConnection(); await using var command = connection.CreateCommand(); command.CommandText = "DELETE FROM native_alarm_state WHERE instance_unique_name = @name"; @@ -549,8 +527,7 @@ public class SiteStorageService /// A task that completes when the shared script has been stored or updated. public async Task StoreSharedScriptAsync(string name, string code, string? parameterDefs, string? returnDef) { - await using var connection = new SqliteConnection(_connectionString); - await connection.OpenAsync(); + await using var connection = OpenConnection(); await using var command = connection.CreateCommand(); command.CommandText = @" @@ -578,8 +555,7 @@ public class SiteStorageService /// A task that resolves to the list of all stored shared scripts. public async Task> GetAllSharedScriptsAsync() { - await using var connection = new SqliteConnection(_connectionString); - await connection.OpenAsync(); + await using var connection = OpenConnection(); await using var command = connection.CreateCommand(); command.CommandText = "SELECT name, code, parameter_definitions, return_definition FROM shared_scripts"; @@ -616,8 +592,7 @@ public class SiteStorageService string name, string endpointUrl, string authType, string? authConfig, string? methodDefs, int timeoutSeconds = 0) { - await using var connection = new SqliteConnection(_connectionString); - await connection.OpenAsync(); + await using var connection = OpenConnection(); await using var command = connection.CreateCommand(); command.CommandText = @" @@ -655,8 +630,7 @@ public class SiteStorageService public async Task StoreDatabaseConnectionAsync( string name, string connectionString, int maxRetries, TimeSpan retryDelay) { - await using var connection = new SqliteConnection(_connectionString); - await connection.OpenAsync(); + await using var connection = OpenConnection(); await using var command = connection.CreateCommand(); command.CommandText = @" @@ -699,8 +673,7 @@ public class SiteStorageService /// A task that completes when both tables have been emptied. public async Task PurgeCentralOnlyNotificationConfigAsync() { - await using var connection = new SqliteConnection(_connectionString); - await connection.OpenAsync(); + await using var connection = OpenConnection(); await using var command = connection.CreateCommand(); command.CommandText = @" @@ -723,8 +696,7 @@ public class SiteStorageService public async Task StoreDataConnectionDefinitionAsync( string name, string protocol, string? configJson, string? backupConfigJson = null, int failoverRetryCount = 3) { - await using var connection = new SqliteConnection(_connectionString); - await connection.OpenAsync(); + await using var connection = OpenConnection(); await using var command = connection.CreateCommand(); command.CommandText = @" @@ -754,8 +726,7 @@ public class SiteStorageService /// A task that resolves to the list of all stored data connection definitions. public async Task> GetAllDataConnectionDefinitionsAsync() { - await using var connection = new SqliteConnection(_connectionString); - await connection.OpenAsync(); + await using var connection = OpenConnection(); await using var command = connection.CreateCommand(); command.CommandText = "SELECT name, protocol, configuration, backup_configuration, failover_retry_count FROM data_connection_definitions"; diff --git a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Repositories/SiteExternalSystemRepository.cs b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Repositories/SiteExternalSystemRepository.cs index efb8a48e..1fe373f4 100644 --- a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Repositories/SiteExternalSystemRepository.cs +++ b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Repositories/SiteExternalSystemRepository.cs @@ -31,8 +31,9 @@ public class SiteExternalSystemRepository : IExternalSystemRepository public async Task> GetAllExternalSystemsAsync( CancellationToken cancellationToken = default) { + // Already open — SiteStorageService.CreateConnection now hands out a + // LocalDb-managed connection. Opening it again would throw. await using var connection = CreateConnection(); - await connection.OpenAsync(cancellationToken); await using var command = connection.CreateCommand(); command.CommandText = @" @@ -63,8 +64,9 @@ public class SiteExternalSystemRepository : IExternalSystemRepository public async Task GetExternalSystemByNameAsync( string name, CancellationToken cancellationToken = default) { + // Already open — SiteStorageService.CreateConnection now hands out a + // LocalDb-managed connection. Opening it again would throw. await using var connection = CreateConnection(); - await connection.OpenAsync(cancellationToken); await using var command = connection.CreateCommand(); command.CommandText = @" @@ -91,8 +93,9 @@ public class SiteExternalSystemRepository : IExternalSystemRepository if (system is null) return Array.Empty(); + // Already open — SiteStorageService.CreateConnection now hands out a + // LocalDb-managed connection. Opening it again would throw. await using var connection = CreateConnection(); - await connection.OpenAsync(cancellationToken); await using var command = connection.CreateCommand(); command.CommandText = @" @@ -140,8 +143,9 @@ public class SiteExternalSystemRepository : IExternalSystemRepository public async Task> GetAllDatabaseConnectionsAsync( CancellationToken cancellationToken = default) { + // Already open — SiteStorageService.CreateConnection now hands out a + // LocalDb-managed connection. Opening it again would throw. await using var connection = CreateConnection(); - await connection.OpenAsync(cancellationToken); await using var command = connection.CreateCommand(); command.CommandText = @" @@ -178,8 +182,9 @@ public class SiteExternalSystemRepository : IExternalSystemRepository public async Task GetDatabaseConnectionByNameAsync( string name, CancellationToken cancellationToken = default) { + // Already open — SiteStorageService.CreateConnection now hands out a + // LocalDb-managed connection. Opening it again would throw. await using var connection = CreateConnection(); - await connection.OpenAsync(cancellationToken); await using var command = connection.CreateCommand(); command.CommandText = @" diff --git a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/ServiceCollectionExtensions.cs b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/ServiceCollectionExtensions.cs index 3d499a2e..6b018167 100644 --- a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/ServiceCollectionExtensions.cs +++ b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/ServiceCollectionExtensions.cs @@ -1,6 +1,7 @@ using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Options; +using ZB.MOM.WW.LocalDb; using ZB.MOM.WW.ScadaBridge.Communication.Grpc; using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Deployment; @@ -14,32 +15,19 @@ namespace ZB.MOM.WW.ScadaBridge.SiteRuntime; public static class ServiceCollectionExtensions { /// - /// Registers Site Runtime services including SiteStorageService for SQLite persistence. - /// The caller must register an or call the - /// overload with an explicit connection string. + /// Registers Site Runtime services including SiteStorageService, which persists to the + /// consolidated site database resolved from ILocalDb (LocalDb:Path). /// /// The DI service collection to register services into. /// The same to allow chaining. public static IServiceCollection AddSiteRuntime(this IServiceCollection services) { - // SiteStorageService is registered by the Host using AddSiteRuntime(connectionString) - // This overload is for backward compatibility / skeleton placeholder - return services; - } - - /// - /// Registers Site Runtime services with an explicit SQLite connection string. - /// - /// The DI service collection to register services into. - /// The SQLite connection string for the site local storage database. - /// The same to allow chaining. - public static IServiceCollection AddSiteRuntime(this IServiceCollection services, string siteDbConnectionString) - { - services.AddSingleton(sp => - { - var logger = sp.GetRequiredService>(); - return new SiteStorageService(siteDbConnectionString, logger); - }); + // SiteStorageService takes ILocalDb (the consolidated site database at + // LocalDb:Path) rather than a connection string, so there is nothing left for a + // caller to supply — the string overload is gone and this is the only entry point. + services.AddSingleton(sp => new SiteStorageService( + sp.GetRequiredService(), + sp.GetRequiredService>())); services.AddHostedService(); diff --git a/src/ZB.MOM.WW.ScadaBridge.StoreAndForward/ServiceCollectionExtensions.cs b/src/ZB.MOM.WW.ScadaBridge.StoreAndForward/ServiceCollectionExtensions.cs index 0f9082f1..a5c2e513 100644 --- a/src/ZB.MOM.WW.ScadaBridge.StoreAndForward/ServiceCollectionExtensions.cs +++ b/src/ZB.MOM.WW.ScadaBridge.StoreAndForward/ServiceCollectionExtensions.cs @@ -1,6 +1,7 @@ using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Options; +using ZB.MOM.WW.LocalDb; using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services; using ZB.MOM.WW.ScadaBridge.SiteEventLogging; @@ -15,14 +16,12 @@ public static class ServiceCollectionExtensions /// The same collection, for chaining. public static IServiceCollection AddStoreAndForward(this IServiceCollection services) { - services.AddSingleton(sp => - { - var options = sp.GetRequiredService>().Value; - var logger = sp.GetRequiredService>(); - return new StoreAndForwardStorage( - $"Data Source={options.SqliteDbPath}", - logger); - }); + // The buffer now lives in the consolidated LocalDb database at LocalDb:Path, not + // at StoreAndForwardOptions.SqliteDbPath — that option survives only as the + // migrator's source location (Task 8). + services.AddSingleton(sp => new StoreAndForwardStorage( + sp.GetRequiredService(), + sp.GetRequiredService>())); services.AddSingleton(sp => { diff --git a/src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardStorage.cs b/src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardStorage.cs index 187d4436..fbe87cc6 100644 --- a/src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardStorage.cs +++ b/src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardStorage.cs @@ -1,5 +1,6 @@ using Microsoft.Data.Sqlite; using Microsoft.Extensions.Logging; +using ZB.MOM.WW.LocalDb; using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; namespace ZB.MOM.WW.ScadaBridge.StoreAndForward; @@ -11,17 +12,25 @@ namespace ZB.MOM.WW.ScadaBridge.StoreAndForward; /// public class StoreAndForwardStorage { - private readonly string _connectionString; + private readonly ILocalDb _localDb; private readonly ILogger _logger; /// - /// Initializes a new instance of with the given SQLite connection string. + /// Initializes the store over the consolidated site database and applies the schema. /// - /// SQLite connection string for the store-and-forward database. + /// + /// The consolidated site database. Every connection it hands out is already open and carries + /// the per-connection pragmas (including busy_timeout) plus the zb_hlc_next() + /// UDF that sf_messages' capture triggers call — which is exactly why the store no + /// longer opens its own from a connection string. A raw + /// connection would lack the UDF and every write to the replicated table would fail closed. + /// /// Logger for diagnostics. - public StoreAndForwardStorage(string connectionString, ILogger logger) + public StoreAndForwardStorage(ILocalDb localDb, ILogger logger) { - _connectionString = connectionString; + ArgumentNullException.ThrowIfNull(localDb); + ArgumentNullException.ThrowIfNull(logger); + _localDb = localDb; _logger = logger; } @@ -29,22 +38,12 @@ public class StoreAndForwardStorage /// Creates the sf_messages table if it does not exist. /// /// A task that represents the asynchronous operation. - public async Task InitializeAsync() + public Task InitializeAsync() { - EnsureDatabaseDirectoryExists(); - - await using var connection = await OpenConnectionAsync(); - - // Enable WAL so the concurrent writers this store has by design (script - // enqueues, the retry sweep's per-target lanes, standby replication applies, - // central pull queries) read/write without "database is locked". WAL is - // persistent + file-scoped; in-memory DBs report "memory" instead — harmless, - // so it is not asserted here. - await using (var walCmd = connection.CreateCommand()) - { - walCmd.CommandText = "PRAGMA journal_mode=WAL"; - await walCmd.ExecuteNonQueryAsync(); - } + // No directory creation and no journal-mode pragma here any more: LocalDb owns + // the file (it creates the directory) and sets WAL plus the per-connection + // pragmas on every connection it hands out. + using var connection = OpenConnection(); // The DDL itself lives in StoreAndForwardSchema so the Host can apply it to a // LocalDb-managed connection before RegisterReplicated installs the capture @@ -53,50 +52,16 @@ public class StoreAndForwardStorage StoreAndForwardSchema.Apply(connection); _logger.LogInformation("Store-and-forward SQLite storage initialized"); + return Task.CompletedTask; } /// - /// Ensures the directory for a file-backed SQLite database exists. SQLite creates - /// the database file on demand but not its parent directory, so a configured path - /// such as "./data/store-and-forward.db" fails to open ("unable to open database - /// file") when the "data" directory does not yet exist. In-memory databases and - /// bare filenames in the working directory have no directory to create and are - /// skipped. + /// CreateConnection returns an ALREADY-OPEN, pragma-configured connection with the + /// zb_hlc_next() UDF registered. Calling OpenAsync on it throws, and a raw + /// would lack the UDF, making every capture trigger fail + /// closed. /// - private void EnsureDatabaseDirectoryExists() - { - var builder = new SqliteConnectionStringBuilder(_connectionString); - if (builder.Mode == SqliteOpenMode.Memory) - return; - - var dataSource = builder.DataSource; - if (string.IsNullOrEmpty(dataSource) || dataSource == ":memory:") - return; - - var directory = System.IO.Path.GetDirectoryName(System.IO.Path.GetFullPath(dataSource)); - if (!string.IsNullOrEmpty(directory) && !System.IO.Directory.Exists(directory)) - { - System.IO.Directory.CreateDirectory(directory); - _logger.LogInformation("Created store-and-forward database directory: {Directory}", directory); - } - } - - /// - /// Opens a connection with a 5s busy_timeout. Concurrent writers exist by - /// design (script enqueues, the sweep's lanes, standby replication applies, - /// central pull queries); with connection-per-operation the pragma must be - /// set per open (it is per-connection, and Microsoft.Data.Sqlite pooling - /// makes the extra statement cheap on a pooled physical connection). - /// - private async Task OpenConnectionAsync() - { - var connection = new SqliteConnection(_connectionString); - await connection.OpenAsync(); - await using var pragma = connection.CreateCommand(); - pragma.CommandText = "PRAGMA busy_timeout = 5000"; - await pragma.ExecuteNonQueryAsync(); - return connection; - } + private SqliteConnection OpenConnection() => _localDb.CreateConnection(); /// /// INSERT statement for a full message row. Shared by @@ -149,7 +114,7 @@ public class StoreAndForwardStorage /// A task that represents the asynchronous operation. public async Task EnqueueAsync(StoreAndForwardMessage message) { - await using var connection = await OpenConnectionAsync(); + await using var connection = OpenConnection(); await using var cmd = connection.CreateCommand(); cmd.CommandText = InsertMessageSql; @@ -169,7 +134,7 @@ public class StoreAndForwardStorage /// The oldest-first page (at most rows) and whether more rows exist beyond it. public async Task<(List Messages, bool Truncated)> GetAllMessagesAsync(int limit) { - await using var connection = await OpenConnectionAsync(); + await using var connection = OpenConnection(); await using var cmd = connection.CreateCommand(); cmd.CommandText = @" @@ -197,7 +162,7 @@ public class StoreAndForwardStorage /// A task that represents the asynchronous operation. public async Task ReplaceAllAsync(IReadOnlyList messages) { - await using var connection = await OpenConnectionAsync(); + await using var connection = OpenConnection(); await using var transaction = (SqliteTransaction)await connection.BeginTransactionAsync(); await using (var deleteCmd = connection.CreateCommand()) @@ -232,7 +197,7 @@ public class StoreAndForwardStorage /// A task that represents the asynchronous operation. public async Task UpsertMessageAsync(StoreAndForwardMessage message) { - await using var connection = await OpenConnectionAsync(); + await using var connection = OpenConnection(); await using var cmd = connection.CreateCommand(); cmd.CommandText = @" @@ -289,7 +254,7 @@ public class StoreAndForwardStorage /// A task that resolves to the list of messages due for retry, ordered by creation time ascending. public async Task> GetMessagesForRetryAsync(int limit = 0) { - await using var connection = await OpenConnectionAsync(); + await using var connection = OpenConnection(); await using var cmd = connection.CreateCommand(); cmd.CommandText = @" @@ -323,7 +288,7 @@ public class StoreAndForwardStorage /// A task that represents the asynchronous operation. public async Task UpdateMessageAsync(StoreAndForwardMessage message) { - await using var connection = await OpenConnectionAsync(); + await using var connection = OpenConnection(); await using var cmd = connection.CreateCommand(); cmd.CommandText = @" @@ -360,7 +325,7 @@ public class StoreAndForwardStorage StoreAndForwardMessage message, StoreAndForwardMessageStatus expectedStatus) { - await using var connection = await OpenConnectionAsync(); + await using var connection = OpenConnection(); await using var cmd = connection.CreateCommand(); cmd.CommandText = @" @@ -393,7 +358,7 @@ public class StoreAndForwardStorage /// A task that represents the asynchronous operation. public async Task RemoveMessageAsync(string messageId) { - await using var connection = await OpenConnectionAsync(); + await using var connection = OpenConnection(); await using var cmd = connection.CreateCommand(); cmd.CommandText = "DELETE FROM sf_messages WHERE id = @id"; @@ -414,7 +379,7 @@ public class StoreAndForwardStorage int pageNumber = 1, int pageSize = 50) { - await using var connection = await OpenConnectionAsync(); + await using var connection = OpenConnection(); await using var transaction = (SqliteTransaction)await connection.BeginTransactionAsync(); @@ -459,7 +424,7 @@ public class StoreAndForwardStorage /// A task that resolves to true if the message was found and reset to Pending; false if not found or not in Parked status. public async Task RetryParkedMessageAsync(string messageId) { - await using var connection = await OpenConnectionAsync(); + await using var connection = OpenConnection(); await using var cmd = connection.CreateCommand(); cmd.CommandText = @" @@ -483,7 +448,7 @@ public class StoreAndForwardStorage /// A task that resolves to true if the message was found and deleted; false if not found or not in Parked status. public async Task DiscardParkedMessageAsync(string messageId) { - await using var connection = await OpenConnectionAsync(); + await using var connection = OpenConnection(); await using var cmd = connection.CreateCommand(); cmd.CommandText = "DELETE FROM sf_messages WHERE id = @id AND status = @parked"; @@ -500,7 +465,7 @@ public class StoreAndForwardStorage /// A task that resolves to a dictionary mapping each category to its pending message count. public async Task> GetBufferDepthByCategoryAsync() { - await using var connection = await OpenConnectionAsync(); + await using var connection = OpenConnection(); await using var cmd = connection.CreateCommand(); cmd.CommandText = @" @@ -530,7 +495,7 @@ public class StoreAndForwardStorage /// A task that resolves to the number of messages whose origin instance matches . public async Task GetMessageCountByOriginInstanceAsync(string instanceName) { - await using var connection = await OpenConnectionAsync(); + await using var connection = OpenConnection(); await using var cmd = connection.CreateCommand(); cmd.CommandText = @" @@ -549,7 +514,7 @@ public class StoreAndForwardStorage /// A task that resolves to the matching message, or null if not found. public async Task GetMessageByIdAsync(string messageId) { - await using var connection = await OpenConnectionAsync(); + await using var connection = OpenConnection(); await using var cmd = connection.CreateCommand(); cmd.CommandText = @" @@ -570,7 +535,7 @@ public class StoreAndForwardStorage /// A task that resolves to the number of messages currently in Parked status. public async Task GetParkedMessageCountAsync() { - await using var conn = await OpenConnectionAsync(); + await using var conn = OpenConnection(); await using var cmd = conn.CreateCommand(); cmd.CommandText = "SELECT COUNT(*) FROM sf_messages WHERE status = @parked"; cmd.Parameters.AddWithValue("@parked", (int)StoreAndForwardMessageStatus.Parked); @@ -588,7 +553,7 @@ public class StoreAndForwardStorage /// A task that resolves to the oldest parked row's creation time, or null if none are parked. public async Task GetOldestParkedCreatedAtAsync() { - await using var conn = await OpenConnectionAsync(); + await using var conn = OpenConnection(); await using var cmd = conn.CreateCommand(); cmd.CommandText = "SELECT MIN(created_at) FROM sf_messages WHERE status = @parked"; cmd.Parameters.AddWithValue("@parked", (int)StoreAndForwardMessageStatus.Parked); @@ -605,7 +570,7 @@ public class StoreAndForwardStorage /// A task that resolves to the count of messages with the specified status. public async Task GetMessageCountByStatusAsync(StoreAndForwardMessageStatus status) { - await using var connection = await OpenConnectionAsync(); + await using var connection = OpenConnection(); await using var cmd = connection.CreateCommand(); cmd.CommandText = "SELECT COUNT(*) FROM sf_messages WHERE status = @status"; diff --git a/tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Integration/ParentExecutionIdCorrelationTests.cs b/tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Integration/ParentExecutionIdCorrelationTests.cs index 1b21ffec..ac72f09f 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Integration/ParentExecutionIdCorrelationTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Integration/ParentExecutionIdCorrelationTests.cs @@ -35,6 +35,7 @@ using ZB.MOM.WW.ScadaBridge.NotificationOutbox.Delivery; using ZB.MOM.WW.ScadaBridge.NotificationOutbox.Messages; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts; using ZB.MOM.WW.ScadaBridge.StoreAndForward; +using ZB.MOM.WW.ScadaBridge.TestSupport; namespace ZB.MOM.WW.ScadaBridge.AuditLog.Tests.Integration; @@ -197,174 +198,184 @@ public class ParentExecutionIdCorrelationTests : TestKit, IClassFixture.Instance); - await safStorage.InitializeAsync(); - var storeAndForward = new StoreAndForwardService( - safStorage, - new StoreAndForwardOptions + // The storage takes an ILocalDb and LocalDb has no in-memory mode, so this is a + // real temp file, disposed (the master connection anchors the WAL) and deleted in + // the finally below. + var safLocalDb = TestLocalDb.CreateTemp("parentexec-saf"); + var safDbPath = safLocalDb.Path; + try + { + var safStorage = new StoreAndForwardStorage( + safLocalDb.Db, NullLogger.Instance); + await safStorage.InitializeAsync(); + var storeAndForward = new StoreAndForwardService( + safStorage, + new StoreAndForwardOptions + { + DefaultRetryInterval = TimeSpan.Zero, + DefaultMaxRetries = 3, + RetryTimerInterval = TimeSpan.FromMinutes(10), + }, + NullLogger.Instance); + + // ── Outbound external-system client (routed run): sync Call succeeds, + // CachedCall completes immediately (WasBuffered=false) so the script + // helper emits the Submit + Attempted + CachedResolve lifecycle. ── + var externalClient = Substitute.For(); + externalClient + .CallAsync(ExternalSystemName, ExternalMethodName, + Arg.Any?>(), Arg.Any()) + .Returns(new ExternalCallResult(true, "{\"ok\":true}", null)); + externalClient + .CachedCallAsync(ExternalSystemName, ExternalMethodName, + Arg.Any?>(), + Arg.Any(), Arg.Any(), + Arg.Any(), + Arg.Any(), Arg.Any(), Arg.Any()) + .Returns(new ExternalCallResult(true, "{\"ok\":true}", null, WasBuffered: false)); + + // ── The routing transport stand-in: builds the routed ScriptRuntimeContext + // carrying RouteToCallRequest.ParentExecutionId — exactly what the + // production site handler (DeploymentManagerActor) does. ── + var router = new BridgingInstanceRouter( + siteId, + externalClient, + siteAuditWriter, + cachedForwarder, + storeAndForward); + + // ── The inbound API method script: it calls Route.Call into the site + // instance. The real InboundScriptExecutor binds the inbound request's + // ExecutionId onto the RouteHelper, so the routed call carries it as + // ParentExecutionId. ── + var inboundMethod = new ZB.MOM.WW.ScadaBridge.Commons.Entities.InboundApi.ApiMethod( + "submitOrder", + $"return await Route.To(\"{RoutedInstanceCode}\").Call(\"{RoutedScriptName}\", new {{ order = 7 }});"); + var locator = Substitute.For(); + locator.GetSiteIdForInstanceAsync(RoutedInstanceCode, Arg.Any()) + .Returns(siteId); + var scriptExecutor = new InboundScriptExecutor( + NullLogger.Instance, + new ServiceCollection().BuildServiceProvider()); + Assert.True(scriptExecutor.CompileAndRegister(inboundMethod)); + + // ── Act — issue the inbound HTTP request through a TestHost pipeline + // fronted by the real AuditWriteMiddleware. The endpoint handler reads + // the middleware-stashed inbound ExecutionId and runs the inbound + // method script with it as parentExecutionId. ── + using var host = await BuildInboundHostAsync(centralAuditWriter, async ctx => { - DefaultRetryInterval = TimeSpan.Zero, - DefaultMaxRetries = 3, - RetryTimerInterval = TimeSpan.FromMinutes(10), - }, - NullLogger.Instance); + var inboundExecutionId = (Guid)ctx.Items[AuditWriteMiddleware.InboundExecutionIdItemKey]!; + var route = new RouteHelper(locator, router); + var result = await scriptExecutor.ExecuteAsync( + inboundMethod, + new Dictionary(), + route, + TimeSpan.FromSeconds(30), + ctx.RequestAborted, + parentExecutionId: inboundExecutionId); - // ── Outbound external-system client (routed run): sync Call succeeds, - // CachedCall completes immediately (WasBuffered=false) so the script - // helper emits the Submit + Attempted + CachedResolve lifecycle. ── - var externalClient = Substitute.For(); - externalClient - .CallAsync(ExternalSystemName, ExternalMethodName, - Arg.Any?>(), Arg.Any()) - .Returns(new ExternalCallResult(true, "{\"ok\":true}", null)); - externalClient - .CachedCallAsync(ExternalSystemName, ExternalMethodName, - Arg.Any?>(), - Arg.Any(), Arg.Any(), - Arg.Any(), - Arg.Any(), Arg.Any(), Arg.Any()) - .Returns(new ExternalCallResult(true, "{\"ok\":true}", null, WasBuffered: false)); + ctx.Response.StatusCode = result.Success ? 200 : 500; + await ctx.Response.WriteAsync(result.Success ? "ok" : "fail"); + }); - // ── The routing transport stand-in: builds the routed ScriptRuntimeContext - // carrying RouteToCallRequest.ParentExecutionId — exactly what the - // production site handler (DeploymentManagerActor) does. ── - var router = new BridgingInstanceRouter( - siteId, - externalClient, - siteAuditWriter, - cachedForwarder, - storeAndForward); + var client = host.GetTestClient(); + var response = await client.PostAsync( + "/api/submitOrder", + new StringContent("{}", Encoding.UTF8, "application/json")); + Assert.Equal(System.Net.HttpStatusCode.OK, response.StatusCode); - // ── The inbound API method script: it calls Route.Call into the site - // instance. The real InboundScriptExecutor binds the inbound request's - // ExecutionId onto the RouteHelper, so the routed call carries it as - // ParentExecutionId. ── - var inboundMethod = new ZB.MOM.WW.ScadaBridge.Commons.Entities.InboundApi.ApiMethod( - "submitOrder", - $"return await Route.To(\"{RoutedInstanceCode}\").Call(\"{RoutedScriptName}\", new {{ order = 7 }});"); - var locator = Substitute.For(); - locator.GetSiteIdForInstanceAsync(RoutedInstanceCode, Arg.Any()) - .Returns(siteId); - var scriptExecutor = new InboundScriptExecutor( - NullLogger.Instance, - new ServiceCollection().BuildServiceProvider()); - Assert.True(scriptExecutor.CompileAndRegister(inboundMethod)); + // The routed run emits its sync-ApiCall and NotifySend audit rows on a + // deliberately fire-and-forget path (alog.md §7 — an audit write must + // never block the user-facing script call). `Notify.Send` therefore + // returns — and the routed `RouteToCallAsync` completes — BEFORE the + // SqliteAuditWriter background loop has flushed the NotifySend row into + // the site hot-path. Wait for all five site rows to be durably present + // in SQLite before the central assertion: this is the production + // durability point (the row IS in SQLite before it is considered + // audited), and pinning it removes the emit-vs-drain race that + // otherwise let the SiteAuditTelemetryADrain forward only four rows on + // its first tick and leave NotifySend stranded for a full drain + // interval under heavy parallel load. + await WaitForSiteRowsPersistedAsync(sqliteWriter); - // ── Act — issue the inbound HTTP request through a TestHost pipeline - // fronted by the real AuditWriteMiddleware. The endpoint handler reads - // the middleware-stashed inbound ExecutionId and runs the inbound - // method script with it as parentExecutionId. ── - using var host = await BuildInboundHostAsync(centralAuditWriter, async ctx => + // The routed run produced a NotifySend that buffered a NotificationSubmit + // into S&F. Drain that genuine site-produced submission to the central + // NotificationOutboxActor so the NotifyDeliver dispatch rows materialise. + await ForwardBufferedNotificationToCentralAsync( + storeAndForward, router.NotificationId!, centralProvider, centralAuditWriter); + + // ── Assert ────────────────────────────────────────────────────────── + await AwaitAssertAsync(async () => + { + await using var readContext = CreateContext(); + var repo = new AuditLogRepository(readContext); + + // Every audit row this site produced (sync ApiCall + cached lifecycle + // + NotifySend) plus the central NotifyDeliver rows. + var siteRows = await repo.QueryAsync( + new AuditLogQueryFilter(SourceSiteIds: new[] { siteId }), + new AuditLogPaging(PageSize: 100)); + + // sync ApiCall (1) + cached Submit/Attempted/Resolve (3) + NotifySend (1) + // + NotifyDeliver Attempted/Delivered (2) = 7 rows for the routed run. + Assert.True(siteRows.Count == 7, + "Expected 7 routed-run audit rows; saw: " + + string.Join(", ", siteRows.Select(r => $"{r.AsRow().Channel}/{r.AsRow().Kind}/{r.AsRow().Status}"))); + Assert.Single(siteRows, r => r.AsRow().Channel == AuditChannel.ApiOutbound && r.AsRow().Kind == AuditKind.ApiCall); + Assert.Single(siteRows, r => r.AsRow().Kind == AuditKind.CachedSubmit); + Assert.Single(siteRows, r => r.AsRow().Kind == AuditKind.CachedResolve); + Assert.Single(siteRows, r => r.AsRow().Kind == AuditKind.NotifySend); + Assert.Equal(2, siteRows.Count(r => r.AsRow().Kind == AuditKind.NotifyDeliver)); + + // CORE PROMISE: every routed-run row carries the SAME non-null + // ParentExecutionId — the inbound request's ExecutionId. + var parentIds = siteRows.Select(r => r.AsRow().ParentExecutionId).Distinct().ToList(); + Assert.Single(parentIds); + Assert.NotNull(parentIds[0]); + var inboundExecutionId = parentIds[0]!.Value; + + // The routed run has its OWN distinct ExecutionId — not the parent's. + var routedExecutionIds = siteRows + .Select(r => r.AsRow().ExecutionId) + .Distinct() + .ToList(); + Assert.Single(routedExecutionIds); + Assert.NotNull(routedExecutionIds[0]); + var routedExecutionId = routedExecutionIds[0]!.Value; + Assert.NotEqual(inboundExecutionId, routedExecutionId); + + // The inbound request's own InboundRequest row is TOP-LEVEL — + // ExecutionId = the propagated id, ParentExecutionId = NULL. + var inboundRows = await repo.QueryAsync( + new AuditLogQueryFilter(ExecutionId: inboundExecutionId), + new AuditLogPaging(PageSize: 10)); + var inboundRow = Assert.Single(inboundRows, + r => r.AsRow().Channel == AuditChannel.ApiInbound && r.AsRow().Kind == AuditKind.InboundRequest); + Assert.Equal(AuditStatus.Delivered, inboundRow.AsRow().Status); + Assert.Null(inboundRow.AsRow().ParentExecutionId); + + // The parentExecutionId filter pulls the routed run's complete + // trust-boundary footprint (all 7 routed rows, none of the inbound). + var byParent = await repo.QueryAsync( + new AuditLogQueryFilter(ParentExecutionId: inboundExecutionId), + new AuditLogPaging(PageSize: 100)); + Assert.Equal(7, byParent.Count); + Assert.All(byParent, r => Assert.Equal(routedExecutionId, r.AsRow().ExecutionId)); + + // GetExecutionTreeAsync returns BOTH executions in one chain — + // inbound (root) and routed (child), regardless of entry point. + var treeFromChild = await repo.GetExecutionTreeAsync(routedExecutionId); + AssertChain(treeFromChild, inboundExecutionId, routedExecutionId); + var treeFromRoot = await repo.GetExecutionTreeAsync(inboundExecutionId); + AssertChain(treeFromRoot, inboundExecutionId, routedExecutionId); + }, TimeSpan.FromSeconds(90)); + } + finally { - var inboundExecutionId = (Guid)ctx.Items[AuditWriteMiddleware.InboundExecutionIdItemKey]!; - var route = new RouteHelper(locator, router); - var result = await scriptExecutor.ExecuteAsync( - inboundMethod, - new Dictionary(), - route, - TimeSpan.FromSeconds(30), - ctx.RequestAborted, - parentExecutionId: inboundExecutionId); - - ctx.Response.StatusCode = result.Success ? 200 : 500; - await ctx.Response.WriteAsync(result.Success ? "ok" : "fail"); - }); - - var client = host.GetTestClient(); - var response = await client.PostAsync( - "/api/submitOrder", - new StringContent("{}", Encoding.UTF8, "application/json")); - Assert.Equal(System.Net.HttpStatusCode.OK, response.StatusCode); - - // The routed run emits its sync-ApiCall and NotifySend audit rows on a - // deliberately fire-and-forget path (alog.md §7 — an audit write must - // never block the user-facing script call). `Notify.Send` therefore - // returns — and the routed `RouteToCallAsync` completes — BEFORE the - // SqliteAuditWriter background loop has flushed the NotifySend row into - // the site hot-path. Wait for all five site rows to be durably present - // in SQLite before the central assertion: this is the production - // durability point (the row IS in SQLite before it is considered - // audited), and pinning it removes the emit-vs-drain race that - // otherwise let the SiteAuditTelemetryADrain forward only four rows on - // its first tick and leave NotifySend stranded for a full drain - // interval under heavy parallel load. - await WaitForSiteRowsPersistedAsync(sqliteWriter); - - // The routed run produced a NotifySend that buffered a NotificationSubmit - // into S&F. Drain that genuine site-produced submission to the central - // NotificationOutboxActor so the NotifyDeliver dispatch rows materialise. - await ForwardBufferedNotificationToCentralAsync( - storeAndForward, router.NotificationId!, centralProvider, centralAuditWriter); - - // ── Assert ────────────────────────────────────────────────────────── - await AwaitAssertAsync(async () => - { - await using var readContext = CreateContext(); - var repo = new AuditLogRepository(readContext); - - // Every audit row this site produced (sync ApiCall + cached lifecycle - // + NotifySend) plus the central NotifyDeliver rows. - var siteRows = await repo.QueryAsync( - new AuditLogQueryFilter(SourceSiteIds: new[] { siteId }), - new AuditLogPaging(PageSize: 100)); - - // sync ApiCall (1) + cached Submit/Attempted/Resolve (3) + NotifySend (1) - // + NotifyDeliver Attempted/Delivered (2) = 7 rows for the routed run. - Assert.True(siteRows.Count == 7, - "Expected 7 routed-run audit rows; saw: " - + string.Join(", ", siteRows.Select(r => $"{r.AsRow().Channel}/{r.AsRow().Kind}/{r.AsRow().Status}"))); - Assert.Single(siteRows, r => r.AsRow().Channel == AuditChannel.ApiOutbound && r.AsRow().Kind == AuditKind.ApiCall); - Assert.Single(siteRows, r => r.AsRow().Kind == AuditKind.CachedSubmit); - Assert.Single(siteRows, r => r.AsRow().Kind == AuditKind.CachedResolve); - Assert.Single(siteRows, r => r.AsRow().Kind == AuditKind.NotifySend); - Assert.Equal(2, siteRows.Count(r => r.AsRow().Kind == AuditKind.NotifyDeliver)); - - // CORE PROMISE: every routed-run row carries the SAME non-null - // ParentExecutionId — the inbound request's ExecutionId. - var parentIds = siteRows.Select(r => r.AsRow().ParentExecutionId).Distinct().ToList(); - Assert.Single(parentIds); - Assert.NotNull(parentIds[0]); - var inboundExecutionId = parentIds[0]!.Value; - - // The routed run has its OWN distinct ExecutionId — not the parent's. - var routedExecutionIds = siteRows - .Select(r => r.AsRow().ExecutionId) - .Distinct() - .ToList(); - Assert.Single(routedExecutionIds); - Assert.NotNull(routedExecutionIds[0]); - var routedExecutionId = routedExecutionIds[0]!.Value; - Assert.NotEqual(inboundExecutionId, routedExecutionId); - - // The inbound request's own InboundRequest row is TOP-LEVEL — - // ExecutionId = the propagated id, ParentExecutionId = NULL. - var inboundRows = await repo.QueryAsync( - new AuditLogQueryFilter(ExecutionId: inboundExecutionId), - new AuditLogPaging(PageSize: 10)); - var inboundRow = Assert.Single(inboundRows, - r => r.AsRow().Channel == AuditChannel.ApiInbound && r.AsRow().Kind == AuditKind.InboundRequest); - Assert.Equal(AuditStatus.Delivered, inboundRow.AsRow().Status); - Assert.Null(inboundRow.AsRow().ParentExecutionId); - - // The parentExecutionId filter pulls the routed run's complete - // trust-boundary footprint (all 7 routed rows, none of the inbound). - var byParent = await repo.QueryAsync( - new AuditLogQueryFilter(ParentExecutionId: inboundExecutionId), - new AuditLogPaging(PageSize: 100)); - Assert.Equal(7, byParent.Count); - Assert.All(byParent, r => Assert.Equal(routedExecutionId, r.AsRow().ExecutionId)); - - // GetExecutionTreeAsync returns BOTH executions in one chain — - // inbound (root) and routed (child), regardless of entry point. - var treeFromChild = await repo.GetExecutionTreeAsync(routedExecutionId); - AssertChain(treeFromChild, inboundExecutionId, routedExecutionId); - var treeFromRoot = await repo.GetExecutionTreeAsync(inboundExecutionId); - AssertChain(treeFromRoot, inboundExecutionId, routedExecutionId); - }, TimeSpan.FromSeconds(90)); + safLocalDb.Dispose(); + TestLocalDb.DeleteFiles(safDbPath); + } } /// diff --git a/tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests.csproj b/tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests.csproj index 9af02ae7..337d93b2 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests.csproj +++ b/tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests.csproj @@ -76,7 +76,8 @@ central MSSQL AuditLog. --> - + + diff --git a/tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/DatabaseGatewayTests.cs b/tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/DatabaseGatewayTests.cs index 7ecba161..0971d130 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/DatabaseGatewayTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/DatabaseGatewayTests.cs @@ -3,18 +3,56 @@ using System.Data.Common; using System.Text.Json; using Microsoft.Extensions.Logging.Abstractions; using NSubstitute; +using ZB.MOM.WW.LocalDb; using ZB.MOM.WW.ScadaBridge.Commons.Entities.ExternalSystems; using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories; +using ZB.MOM.WW.ScadaBridge.TestSupport; namespace ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests; /// /// WP-9: Tests for Database access — connection resolution, cached writes. /// -public class DatabaseGatewayTests +public class DatabaseGatewayTests : IDisposable { private readonly IExternalSystemRepository _repository = Substitute.For(); + /// + /// Temp local databases opened by the Store-and-Forward tests, torn down in + /// . + /// + private readonly List _localDbs = new(); + + /// + /// Opens a real ILocalDb over a fresh temp file for a test that needs a live + /// . The + /// store takes an ILocalDb and LocalDb has no in-memory mode, so each test gets + /// its own file rather than a shared-cache in-memory database held open by a + /// keep-alive connection. + /// + private TestLocalDb CreateLocalDb(string prefix) + { + var localDb = TestLocalDb.CreateTemp(prefix); + _localDbs.Add(localDb); + return localDb; + } + + /// + /// Disposes every temp local database and then deletes its files — in that order, + /// because the master connection LocalDb holds anchors the WAL sidecars. + /// + public void Dispose() + { + foreach (var localDb in _localDbs) + { + var path = localDb.Path; + localDb.Dispose(); + TestLocalDb.DeleteFiles(path); + } + + GC.SuppressFinalize(this); + } + /// /// Configures the repository substitute for the name-keyed connection-resolution /// path used by DatabaseGateway (ExternalSystemGateway-011). A null @@ -84,12 +122,9 @@ public class DatabaseGatewayTests }; StubConnection(conn); - var dbName = $"EsgCachedWrite_{Guid.NewGuid():N}"; - var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared"; - using var keepAlive = new Microsoft.Data.Sqlite.SqliteConnection(connStr); - keepAlive.Open(); + var localDb = CreateLocalDb("EsgCachedWrite"); var storage = new ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardStorage( - connStr, NullLogger.Instance); + localDb.Db, NullLogger.Instance); await storage.InitializeAsync(); var sfOptions = new ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardOptions { @@ -121,7 +156,7 @@ public class DatabaseGatewayTests var depth = await storage.GetBufferDepthByCategoryAsync(); Assert.Equal(1, depth[ZB.MOM.WW.ScadaBridge.Commons.Types.Enums.StoreAndForwardCategory.CachedDbWrite]); - var buffered = ReadBufferedRetrySettings(connStr); + var buffered = ReadBufferedRetrySettings(localDb.Db); Assert.Equal(5, buffered.MaxRetries); Assert.Equal((long)TimeSpan.FromSeconds(12).TotalMilliseconds, buffered.RetryIntervalMs); @@ -148,12 +183,9 @@ public class DatabaseGatewayTests }; StubConnection(conn); - var dbName = $"EsgCachedWriteZero_{Guid.NewGuid():N}"; - var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared"; - using var keepAlive = new Microsoft.Data.Sqlite.SqliteConnection(connStr); - keepAlive.Open(); + var localDb = CreateLocalDb("EsgCachedWriteZero"); var storage = new ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardStorage( - connStr, NullLogger.Instance); + localDb.Db, NullLogger.Instance); await storage.InitializeAsync(); var sfOptions = new ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardOptions { @@ -172,7 +204,7 @@ public class DatabaseGatewayTests await gateway.CachedWriteAsync("testDb", "INSERT INTO t VALUES (1)"); - var (maxRetries, _, _, _) = ReadBufferedRetrySettings(connStr); + var (maxRetries, _, _, _) = ReadBufferedRetrySettings(localDb.Db); // Must be the bounded S&F default, never 0 — a stored 0 would mean retry-forever. Assert.Equal(99, maxRetries); Assert.NotEqual(0, maxRetries); @@ -182,19 +214,15 @@ public class DatabaseGatewayTests // cached-write attempt + the buffered retry path ── /// - /// Builds a real, initialised in-memory store-and-forward service plus a - /// keep-alive connection (the SQLite shared-cache DB lives only while a - /// connection is open). The caller disposes . + /// Builds a real, initialised store-and-forward service over a fresh temp local + /// database. The database is torn down by . /// - private static (ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardService Sf, string ConnStr, Microsoft.Data.Sqlite.SqliteConnection KeepAlive) + private (ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardService Sf, ILocalDb Db) NewStoreAndForward() { - var dbName = $"EsgCachedWriteClassify_{Guid.NewGuid():N}"; - var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared"; - var keepAlive = new Microsoft.Data.Sqlite.SqliteConnection(connStr); - keepAlive.Open(); + var localDb = CreateLocalDb("EsgCachedWriteClassify"); var storage = new ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardStorage( - connStr, NullLogger.Instance); + localDb.Db, NullLogger.Instance); storage.InitializeAsync().GetAwaiter().GetResult(); var sfOptions = new ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardOptions { @@ -204,7 +232,7 @@ public class DatabaseGatewayTests }; var sf = new ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardService( storage, sfOptions, NullLogger.Instance); - return (sf, connStr, keepAlive); + return (sf, localDb.Db); } [Fact] @@ -217,8 +245,7 @@ public class DatabaseGatewayTests var conn = new DatabaseConnectionDefinition("testDb", "Server=localhost;Database=test") { Id = 1 }; StubConnection(conn); - var (sf, connStr, keepAlive) = NewStoreAndForward(); - using var _ = keepAlive; + var (sf, sfDb) = NewStoreAndForward(); var gateway = new ExecuteStubGateway( _repository, @@ -233,7 +260,7 @@ public class DatabaseGatewayTests Assert.NotNull(result.ErrorMessage); // Nothing buffered — the permanent failure short-circuited S&F. - Assert.Equal(0, ReadBufferDepth(connStr)); + Assert.Equal(0, ReadBufferDepth(sfDb)); } [Fact] @@ -249,8 +276,7 @@ public class DatabaseGatewayTests }; StubConnection(conn); - var (sf, connStr, keepAlive) = NewStoreAndForward(); - using var _ = keepAlive; + var (sf, sfDb) = NewStoreAndForward(); var gateway = new ExecuteStubGateway( _repository, @@ -265,7 +291,7 @@ public class DatabaseGatewayTests Assert.True(result.WasBuffered); // handed to S&F, not synchronously failed Assert.Null(result.ErrorMessage); - Assert.Equal(1, ReadBufferDepth(connStr)); + Assert.Equal(1, ReadBufferDepth(sfDb)); } [Fact] @@ -277,8 +303,7 @@ public class DatabaseGatewayTests var conn = new DatabaseConnectionDefinition("testDb", "Server=localhost;Database=test") { Id = 1 }; StubConnection(conn); - var (sf, connStr, keepAlive) = NewStoreAndForward(); - using var _ = keepAlive; + var (sf, sfDb) = NewStoreAndForward(); var gateway = new ExecuteStubGateway(_repository, sf, onExecute: () => { /* succeeds */ }); @@ -288,7 +313,7 @@ public class DatabaseGatewayTests Assert.False(result.WasBuffered); Assert.Null(result.ErrorMessage); - Assert.Equal(0, ReadBufferDepth(connStr)); + Assert.Equal(0, ReadBufferDepth(sfDb)); } [Fact] @@ -381,8 +406,7 @@ public class DatabaseGatewayTests }; StubConnection(conn); - var (sf, connStr, keepAlive) = NewStoreAndForward(); - using var _ = keepAlive; + var (sf, sfDb) = NewStoreAndForward(); // RawExecuteStubGateway routes the raw throw through the PRODUCTION // ExecuteWriteAsync classification (the seam under test), unlike @@ -395,7 +419,7 @@ public class DatabaseGatewayTests Assert.True(result.WasBuffered); // handed to S&F as transient Assert.Null(result.ErrorMessage); // not a permanent Failed result - Assert.Equal(1, ReadBufferDepth(connStr)); + Assert.Equal(1, ReadBufferDepth(sfDb)); } [Fact] @@ -408,8 +432,7 @@ public class DatabaseGatewayTests var conn = new DatabaseConnectionDefinition("testDb", "Server=localhost;Database=test") { Id = 1 }; StubConnection(conn); - var (sf, connStr, keepAlive) = NewStoreAndForward(); - using var _ = keepAlive; + var (sf, sfDb) = NewStoreAndForward(); using var cts = new CancellationTokenSource(); cts.Cancel(); @@ -421,7 +444,7 @@ public class DatabaseGatewayTests () => gateway.CachedWriteAsync("testDb", "INSERT INTO t VALUES (1)", cancellationToken: cts.Token)); // Cancellation is not a transient failure — nothing must have been buffered. - Assert.Equal(0, ReadBufferDepth(connStr)); + Assert.Equal(0, ReadBufferDepth(sfDb)); } [Fact] @@ -434,8 +457,7 @@ public class DatabaseGatewayTests var conn = new DatabaseConnectionDefinition("testDb", "Server=localhost;Database=test") { Id = 1 }; StubConnection(conn); - var (sf, connStr, keepAlive) = NewStoreAndForward(); - using var _ = keepAlive; + var (sf, sfDb) = NewStoreAndForward(); var gateway = new RawExecuteStubGateway( _repository, sf, onRunSql: () => throw new ArgumentException("authoring bug")); @@ -443,7 +465,7 @@ public class DatabaseGatewayTests await Assert.ThrowsAsync( () => gateway.CachedWriteAsync("testDb", "INSERT INTO t VALUES (1)")); - Assert.Equal(0, ReadBufferDepth(connStr)); + Assert.Equal(0, ReadBufferDepth(sfDb)); } [Fact] @@ -486,8 +508,7 @@ public class DatabaseGatewayTests var conn = new DatabaseConnectionDefinition("testDb", "Server=localhost;Database=test") { Id = 1 }; StubConnection(conn); - var (sf, connStr, keepAlive) = NewStoreAndForward(); - using var _ = keepAlive; + var (sf, sfDb) = NewStoreAndForward(); using var cts = new CancellationTokenSource(); cts.Cancel(); @@ -505,7 +526,7 @@ public class DatabaseGatewayTests // The cancel won — it must NOT have been classified as transient (buffered) // nor returned as a permanent Failed result. - Assert.Equal(0, ReadBufferDepth(connStr)); + Assert.Equal(0, ReadBufferDepth(sfDb)); } /// @@ -542,10 +563,10 @@ public class DatabaseGatewayTests /// Reads the current buffered-message count off the S&F SQLite DB by /// counting sf_messages rows (the engine's persistence table). /// - private static int ReadBufferDepth(string connStr) + private static int ReadBufferDepth(ILocalDb localDb) { - using var conn = new Microsoft.Data.Sqlite.SqliteConnection(connStr); - conn.Open(); + // CreateConnection hands back an already-open, pragma-configured connection. + using var conn = localDb.CreateConnection(); using var cmd = conn.CreateCommand(); cmd.CommandText = "SELECT COUNT(*) FROM sf_messages"; return Convert.ToInt32(cmd.ExecuteScalar()); @@ -615,10 +636,10 @@ public class DatabaseGatewayTests } private static (int MaxRetries, long RetryIntervalMs, Guid? ExecutionId, string? SourceScript) - ReadBufferedRetrySettings(string connStr) + ReadBufferedRetrySettings(ILocalDb localDb) { - using var conn = new Microsoft.Data.Sqlite.SqliteConnection(connStr); - conn.Open(); + // CreateConnection hands back an already-open, pragma-configured connection. + using var conn = localDb.CreateConnection(); using var cmd = conn.CreateCommand(); cmd.CommandText = "SELECT max_retries, retry_interval_ms, execution_id, source_script FROM sf_messages"; diff --git a/tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/ExternalSystemClientTests.cs b/tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/ExternalSystemClientTests.cs index 206c592f..ebea6eca 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/ExternalSystemClientTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/ExternalSystemClientTests.cs @@ -1,24 +1,60 @@ using System.Net; using System.Net.Http.Headers; using System.Text; -using Microsoft.Data.Sqlite; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Logging.Abstractions; using NSubstitute; +using ZB.MOM.WW.LocalDb; using ZB.MOM.WW.ScadaBridge.Commons.Entities.ExternalSystems; using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories; using ZB.MOM.WW.ScadaBridge.StoreAndForward; +using ZB.MOM.WW.ScadaBridge.TestSupport; namespace ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests; /// /// WP-6/7: Tests for ExternalSystemClient — HTTP client, call modes, error handling. /// -public class ExternalSystemClientTests +public class ExternalSystemClientTests : IDisposable { private readonly IExternalSystemRepository _repository = Substitute.For(); private readonly IHttpClientFactory _httpClientFactory = Substitute.For(); + /// + /// Temp local databases opened by the Store-and-Forward tests, torn down in + /// . + /// + private readonly List _localDbs = new(); + + /// + /// Opens a real ILocalDb over a fresh temp file for a test that needs a live + /// . The store takes an ILocalDb and LocalDb + /// has no in-memory mode, so each test gets its own file rather than a shared-cache + /// in-memory database held open by a keep-alive connection. + /// + private TestLocalDb CreateLocalDb(string prefix) + { + var localDb = TestLocalDb.CreateTemp(prefix); + _localDbs.Add(localDb); + return localDb; + } + + /// + /// Disposes every temp local database and then deletes its files — in that order, + /// because the master connection LocalDb holds anchors the WAL sidecars. + /// + public void Dispose() + { + foreach (var localDb in _localDbs) + { + var path = localDb.Path; + localDb.Dispose(); + TestLocalDb.DeleteFiles(path); + } + + GC.SuppressFinalize(this); + } + /// /// Configures the repository substitute for the name-keyed resolution path used by /// ExternalSystemClient (ExternalSystemGateway-011). A null system or @@ -275,11 +311,8 @@ public class ExternalSystemClientTests _httpClientFactory.CreateClient(Arg.Any()).Returns(httpClient); // A real S&F service with a registered delivery handler that counts invocations. - var dbName = $"EsgDoubleDispatch_{Guid.NewGuid():N}"; - var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared"; - using var keepAlive = new SqliteConnection(connStr); - keepAlive.Open(); - var storage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + var localDb = CreateLocalDb("EsgDoubleDispatch"); + var storage = new StoreAndForwardStorage(localDb.Db, NullLogger.Instance); await storage.InitializeAsync(); var sfOptions = new StoreAndForwardOptions { @@ -422,11 +455,8 @@ public class ExternalSystemClientTests var httpClient = new HttpClient(new MockHttpMessageHandler(HttpStatusCode.InternalServerError, "boom")); _httpClientFactory.CreateClient(Arg.Any()).Returns(httpClient); - var dbName = $"EsgRetry_{Guid.NewGuid():N}"; - var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared"; - using var keepAlive = new SqliteConnection(connStr); - keepAlive.Open(); - var storage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + var localDb = CreateLocalDb("EsgRetry"); + var storage = new StoreAndForwardStorage(localDb.Db, NullLogger.Instance); await storage.InitializeAsync(); // S&F defaults deliberately different from the system's settings. var sfOptions = new StoreAndForwardOptions @@ -454,7 +484,7 @@ public class ExternalSystemClientTests var depth = await storage.GetBufferDepthByCategoryAsync(); Assert.Equal(1, depth[ZB.MOM.WW.ScadaBridge.Commons.Types.Enums.StoreAndForwardCategory.ExternalSystem]); - var buffered = ReadBufferedRetrySettings(connStr); + var buffered = ReadBufferedRetrySettings(localDb.Db); Assert.Equal(7, buffered.MaxRetries); Assert.Equal((long)TimeSpan.FromSeconds(42).TotalMilliseconds, buffered.RetryIntervalMs); @@ -466,10 +496,10 @@ public class ExternalSystemClientTests } private static (int MaxRetries, long RetryIntervalMs, Guid? ExecutionId, string? SourceScript) - ReadBufferedRetrySettings(string connStr) + ReadBufferedRetrySettings(ILocalDb localDb) { - using var conn = new SqliteConnection(connStr); - conn.Open(); + // CreateConnection hands back an already-open, pragma-configured connection. + using var conn = localDb.CreateConnection(); using var cmd = conn.CreateCommand(); cmd.CommandText = "SELECT max_retries, retry_interval_ms, execution_id, source_script FROM sf_messages"; @@ -505,11 +535,8 @@ public class ExternalSystemClientTests var httpClient = new HttpClient(new MockHttpMessageHandler(HttpStatusCode.InternalServerError, "boom")); _httpClientFactory.CreateClient(Arg.Any()).Returns(httpClient); - var dbName = $"EsgRetryZero_{Guid.NewGuid():N}"; - var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared"; - using var keepAlive = new SqliteConnection(connStr); - keepAlive.Open(); - var storage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + var localDb = CreateLocalDb("EsgRetryZero"); + var storage = new StoreAndForwardStorage(localDb.Db, NullLogger.Instance); await storage.InitializeAsync(); var sfOptions = new StoreAndForwardOptions { @@ -525,7 +552,7 @@ public class ExternalSystemClientTests await client.CachedCallAsync("TestAPI", "postData"); - var (maxRetries, _, _, _) = ReadBufferedRetrySettings(connStr); + var (maxRetries, _, _, _) = ReadBufferedRetrySettings(localDb.Db); // Must be the bounded S&F default, never 0 — a stored 0 would mean retry-forever. Assert.Equal(99, maxRetries); Assert.NotEqual(0, maxRetries); @@ -651,11 +678,8 @@ public class ExternalSystemClientTests var httpClient = new HttpClient(new HangingHttpMessageHandler(TimeSpan.FromMinutes(10))); _httpClientFactory.CreateClient(Arg.Any()).Returns(httpClient); - var dbName = $"EsgCancel_{Guid.NewGuid():N}"; - var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared"; - using var keepAlive = new SqliteConnection(connStr); - keepAlive.Open(); - var storage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + var localDb = CreateLocalDb("EsgCancel"); + var storage = new StoreAndForwardStorage(localDb.Db, NullLogger.Instance); await storage.InitializeAsync(); var sfOptions = new StoreAndForwardOptions { @@ -1060,11 +1084,8 @@ public class ExternalSystemClientTests .Returns(_ => new HttpClient(new MockHttpMessageHandler(HttpStatusCode.OK, hugeBody))); // A real S&F service so we can assert the oversized response is NOT buffered. - var dbName = $"EsgOversize_{Guid.NewGuid():N}"; - var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared"; - using var keepAlive = new SqliteConnection(connStr); - keepAlive.Open(); - var storage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + var localDb = CreateLocalDb("EsgOversize"); + var storage = new StoreAndForwardStorage(localDb.Db, NullLogger.Instance); await storage.InitializeAsync(); var sfOptions = new StoreAndForwardOptions { diff --git a/tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests.csproj b/tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests.csproj index d2b57969..abb801e5 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests.csproj +++ b/tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests.csproj @@ -24,6 +24,7 @@ - + + diff --git a/tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests/HealthReportSenderTests.cs b/tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests/HealthReportSenderTests.cs index 5ea6a969..55d8fa57 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests/HealthReportSenderTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests/HealthReportSenderTests.cs @@ -1,10 +1,10 @@ -using Microsoft.Data.Sqlite; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Logging.Abstractions; using Microsoft.Extensions.Options; using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health; using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; using ZB.MOM.WW.ScadaBridge.StoreAndForward; +using ZB.MOM.WW.ScadaBridge.TestSupport; namespace ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests; @@ -191,50 +191,55 @@ public class HealthReportSenderTests [Fact] public async Task ReportsIncludeStoreAndForwardBufferDepthsFromStorage() { - var dbName = $"HealthSfDepth_{Guid.NewGuid():N}"; - var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared"; - // Keep one connection alive so the in-memory DB persists for the test. - using var keepAlive = new SqliteConnection(connStr); - keepAlive.Open(); - - var storage = new StoreAndForwardStorage(connStr, NullLogger.Instance); - await storage.InitializeAsync(); - - // Two pending ExternalSystem messages and one pending Notification message. - await storage.EnqueueAsync(MakePendingMessage("m1", StoreAndForwardCategory.ExternalSystem)); - await storage.EnqueueAsync(MakePendingMessage("m2", StoreAndForwardCategory.ExternalSystem)); - await storage.EnqueueAsync(MakePendingMessage("m3", StoreAndForwardCategory.Notification)); - - var transport = new FakeTransport(); - var collector = new SiteHealthCollector(); - collector.SetActiveNode(true); - var options = Options.Create(new HealthMonitoringOptions - { - ReportInterval = TimeSpan.FromMilliseconds(50) - }); - - var sender = new HealthReportSender( - collector, - transport, - options, - NullLogger.Instance, - new FakeSiteIdentityProvider(), - sfStorage: storage); - - using var cts = new CancellationTokenSource(TimeSpan.FromMilliseconds(300)); + // LocalDb has no in-memory mode, so the store runs over a real temp file. + var localDb = TestLocalDb.CreateTemp("HealthSfDepth"); try { - await sender.StartAsync(cts.Token); - await Task.Delay(250, CancellationToken.None); - await sender.StopAsync(CancellationToken.None); - } - catch (OperationCanceledException) { } + var storage = new StoreAndForwardStorage(localDb.Db, NullLogger.Instance); + await storage.InitializeAsync(); - Assert.True(transport.SentReports.Count >= 1); - var depths = transport.SentReports[^1].StoreAndForwardBufferDepths; - Assert.Equal(2, depths[nameof(StoreAndForwardCategory.ExternalSystem)]); - Assert.Equal(1, depths[nameof(StoreAndForwardCategory.Notification)]); - Assert.False(depths.ContainsKey(nameof(StoreAndForwardCategory.CachedDbWrite))); + // Two pending ExternalSystem messages and one pending Notification message. + await storage.EnqueueAsync(MakePendingMessage("m1", StoreAndForwardCategory.ExternalSystem)); + await storage.EnqueueAsync(MakePendingMessage("m2", StoreAndForwardCategory.ExternalSystem)); + await storage.EnqueueAsync(MakePendingMessage("m3", StoreAndForwardCategory.Notification)); + + var transport = new FakeTransport(); + var collector = new SiteHealthCollector(); + collector.SetActiveNode(true); + var options = Options.Create(new HealthMonitoringOptions + { + ReportInterval = TimeSpan.FromMilliseconds(50) + }); + + var sender = new HealthReportSender( + collector, + transport, + options, + NullLogger.Instance, + new FakeSiteIdentityProvider(), + sfStorage: storage); + + using var cts = new CancellationTokenSource(TimeSpan.FromMilliseconds(300)); + try + { + await sender.StartAsync(cts.Token); + await Task.Delay(250, CancellationToken.None); + await sender.StopAsync(CancellationToken.None); + } + catch (OperationCanceledException) { } + + Assert.True(transport.SentReports.Count >= 1); + var depths = transport.SentReports[^1].StoreAndForwardBufferDepths; + Assert.Equal(2, depths[nameof(StoreAndForwardCategory.ExternalSystem)]); + Assert.Equal(1, depths[nameof(StoreAndForwardCategory.Notification)]); + Assert.False(depths.ContainsKey(nameof(StoreAndForwardCategory.CachedDbWrite))); + } + finally + { + // The master connection anchors the WAL — dispose before deleting. + localDb.Dispose(); + TestLocalDb.DeleteFiles(localDb.Path); + } } private static StoreAndForwardMessage MakePendingMessage(string id, StoreAndForwardCategory category) => diff --git a/tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests.csproj b/tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests.csproj index 2f7c9b4b..1cda3a29 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests.csproj +++ b/tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests.csproj @@ -28,6 +28,7 @@ - + + diff --git a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbDirectoryTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbDirectoryTests.cs new file mode 100644 index 00000000..74b9c2f2 --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbDirectoryTests.cs @@ -0,0 +1,70 @@ +using Microsoft.Extensions.Configuration; +using Microsoft.Extensions.DependencyInjection; +using ZB.MOM.WW.LocalDb; + +namespace ZB.MOM.WW.ScadaBridge.Host.Tests; + +/// +/// Pins that a site node creates the directory holding LocalDb:Path before the +/// database is opened. +/// +/// +/// +/// This guarantee used to live in StoreAndForwardStorage.EnsureDatabaseDirectoryExists, +/// which created the parent directory for its own SQLite file. LocalDb Phase 2 moved that +/// file into the consolidated database — but the LocalDb library does not create the +/// directory, and SqliteLocalDb opens the file eagerly in its constructor, so a +/// missing directory is a hard boot failure ("SQLite Error 14: unable to open database +/// file") rather than a degraded start. +/// +/// +/// This is not hypothetical: the default site configuration points at the relative +/// path ./data/site-localdb.db, so any site node whose working directory has no +/// data/ subdirectory hits it. The docker rig escapes only because its volume mount +/// creates /app/data — which is exactly the kind of coincidence that hides a defect +/// until a bare-metal or fresh deployment. +/// +/// +public class SiteLocalDbDirectoryTests +{ + [Fact] + public void SiteRegistration_CreatesTheLocalDbDirectory_WhenItDoesNotExist() + { + var root = Path.Combine(Path.GetTempPath(), "localdb-dir-test-" + Guid.NewGuid().ToString("N")); + var dbPath = Path.Combine(root, "nested", "site-localdb.db"); + Assert.False(Directory.Exists(root)); + + try + { + var config = new ConfigurationBuilder() + .AddInMemoryCollection(new Dictionary + { + ["LocalDb:Path"] = dbPath, + }) + .Build(); + + // The registration path under test. Resolving ILocalDb is what actually opens + // the file, so this fails with SQLite Error 14 if the directory step regresses. + var services = new ServiceCollection(); + SiteLocalDbDirectory.Ensure(config); + services.AddZbLocalDb(config); + + using var provider = services.BuildServiceProvider(); + using var scope = provider.CreateScope(); + var db = scope.ServiceProvider.GetRequiredService(); + + Assert.NotNull(db); + Assert.True(Directory.Exists(Path.GetDirectoryName(dbPath)!)); + Assert.True(File.Exists(dbPath)); + } + finally + { + SqliteConnectionPoolCleanup(); + if (Directory.Exists(root)) + Directory.Delete(root, recursive: true); + } + } + + private static void SqliteConnectionPoolCleanup() => + Microsoft.Data.Sqlite.SqliteConnection.ClearAllPools(); +} diff --git a/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/SfBufferResyncPredicateTests.cs b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/SfBufferResyncPredicateTests.cs index 0dac6c10..9e9f3cb2 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/SfBufferResyncPredicateTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/SfBufferResyncPredicateTests.cs @@ -4,6 +4,7 @@ using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; using ZB.MOM.WW.ScadaBridge.StoreAndForward; using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; +using ZB.MOM.WW.ScadaBridge.TestSupport; namespace ZB.MOM.WW.ScadaBridge.IntegrationTests.Cluster; @@ -25,40 +26,67 @@ public class SfBufferResyncPredicateTests var p2 = TwoNodeClusterFixture.GetFreeTcpPort(); var (portHigh, portLow) = p1 > p2 ? (p1, p2) : (p2, p1); - await using var fixture = await TwoNodeClusterFixture.StartAsync( + var fixture = await TwoNodeClusterFixture.StartAsync( role: "site-int", portA: portHigh, portB: portLow); - // Real S&F storage + replication actor per node, production default predicate - // (no isActiveOverride) — the exact wiring under test. - var (storageOldest, _) = await CreateReplicationActorAsync(fixture.NodeA, "oldest"); - var (storageJoiner, _) = await CreateReplicationActorAsync(fixture.NodeB, "joiner"); + // The S&F stores AND SiteStorageService take an ILocalDb (LocalDb has no in-memory + // mode), so each node gets its own temp-file local databases. They are disposed + // AFTER the cluster is shut down — the actors hold connections while the systems + // are alive — and only then are the files (plus their WAL sidecars) deleted. + var localDbs = new List(); - // The delivering (oldest) node has a live buffered row the standby never saw. - await storageOldest.EnqueueAsync(NewMessage("live-row")); + try + { + // Real S&F storage + replication actor per node, production default predicate + // (no isActiveOverride) — the exact wiring under test. + var (storageOldest, _, sfDbOldest, siteDbOldest) = + await CreateReplicationActorAsync(fixture.NodeA, "oldest"); + localDbs.Add(sfDbOldest); + localDbs.Add(siteDbOldest); + var (storageJoiner, _, sfDbJoiner, siteDbJoiner) = + await CreateReplicationActorAsync(fixture.NodeB, "joiner"); + localDbs.Add(sfDbJoiner); + localDbs.Add(siteDbJoiner); - // Trigger peer (re)tracking on both sides: each actor got InitialStateAsSnapshot - // in PreStart, but the enqueue raced it — re-deliver via a fresh MemberUp is not - // needed; OnPeerTracked already fired on join. The resync exchange is async: - // wait until the JOINER holds the row (proves the snapshot flowed oldest→joiner, - // the correct direction). Pre-fix this times out (the joiner, as leader, never - // requests) AND the oldest node's row is deleted by the stale wipe. - await AwaitAsync(async () => await storageJoiner.GetMessageByIdAsync("live-row") != null, - TimeSpan.FromSeconds(20), - "joiner never received the resync snapshot (resync ran in the wrong direction)"); + // The delivering (oldest) node has a live buffered row the standby never saw. + await storageOldest.EnqueueAsync(NewMessage("live-row")); - // And the delivering node's buffer is untouched — the N1 wipe assertion. - Assert.NotNull(await storageOldest.GetMessageByIdAsync("live-row")); + // Trigger peer (re)tracking on both sides: each actor got InitialStateAsSnapshot + // in PreStart, but the enqueue raced it — re-deliver via a fresh MemberUp is not + // needed; OnPeerTracked already fired on join. The resync exchange is async: + // wait until the JOINER holds the row (proves the snapshot flowed oldest→joiner, + // the correct direction). Pre-fix this times out (the joiner, as leader, never + // requests) AND the oldest node's row is deleted by the stale wipe. + await AwaitAsync(async () => await storageJoiner.GetMessageByIdAsync("live-row") != null, + TimeSpan.FromSeconds(20), + "joiner never received the resync snapshot (resync ran in the wrong direction)"); + + // And the delivering node's buffer is untouched — the N1 wipe assertion. + Assert.NotNull(await storageOldest.GetMessageByIdAsync("live-row")); + } + finally + { + await fixture.DisposeAsync(); + + foreach (var localDb in localDbs) + { + var path = localDb.Path; + localDb.Dispose(); + TestLocalDb.DeleteFiles(path); + } + } } - private static async Task<(StoreAndForwardStorage Storage, IActorRef Actor)> CreateReplicationActorAsync( - ActorSystem node, string tag) + private static async Task<( + StoreAndForwardStorage Storage, IActorRef Actor, TestLocalDb SfLocalDb, TestLocalDb SiteLocalDb)> + CreateReplicationActorAsync(ActorSystem node, string tag) { - var sfDb = Path.Combine(Path.GetTempPath(), $"sf-resync-{tag}-{Guid.NewGuid():N}.db"); - var siteDb = Path.Combine(Path.GetTempPath(), $"site-resync-{tag}-{Guid.NewGuid():N}.db"); - var sfStorage = new StoreAndForwardStorage($"Data Source={sfDb}", + var sfLocalDb = TestLocalDb.CreateTemp($"sf-resync-{tag}"); + var sfStorage = new StoreAndForwardStorage(sfLocalDb.Db, NullLogger.Instance); await sfStorage.InitializeAsync(); - var siteStorage = new SiteStorageService($"Data Source={siteDb}", + var siteLocalDb = TestLocalDb.CreateTemp($"site-resync-{tag}"); + var siteStorage = new SiteStorageService(siteLocalDb.Db, NullLogger.Instance); var replicationService = new ReplicationService( new StoreAndForwardOptions(), NullLogger.Instance); @@ -67,7 +95,7 @@ public class SfBufferResyncPredicateTests siteStorage, sfStorage, replicationService, "site-int", NullLogger.Instance, null, null, null, null)), "site-replication"); - return (sfStorage, actor); + return (sfStorage, actor, sfLocalDb, siteLocalDb); } private static StoreAndForwardMessage NewMessage(string id) => new() diff --git a/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/DualNodeRecoveryTests.cs b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/DualNodeRecoveryTests.cs index 0cfc6485..87c3d89f 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/DualNodeRecoveryTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/DualNodeRecoveryTests.cs @@ -1,6 +1,7 @@ using Microsoft.Extensions.Logging.Abstractions; using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; using ZB.MOM.WW.ScadaBridge.StoreAndForward; +using ZB.MOM.WW.ScadaBridge.TestSupport; namespace ZB.MOM.WW.ScadaBridge.IntegrationTests; @@ -18,12 +19,17 @@ public class DualNodeRecoveryTests // Scenario: both site nodes crash. First node to restart opens the existing // SQLite database and finds all buffered S&F messages intact. var dbPath = Path.Combine(Path.GetTempPath(), $"sf_dual_{Guid.NewGuid():N}.db"); - var connStr = $"Data Source={dbPath}"; + + // The store takes an ILocalDb; the "restarted node" opens its own local database + // over the SAME file, which is how this test models recovery from disk. + TestLocalDb? crashedDb = null; + TestLocalDb? recoveryDb = null; try { // Setup: populate SQLite with messages (simulating pre-crash state) - var storage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + crashedDb = TestLocalDb.Create(dbPath); + var storage = new StoreAndForwardStorage(crashedDb.Db, NullLogger.Instance); await storage.InitializeAsync(); var messageIds = new List(); @@ -48,7 +54,8 @@ public class DualNodeRecoveryTests // Both nodes down — simulate by creating a fresh storage instance // (new process connecting to same SQLite file) - var recoveryStorage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + recoveryDb = TestLocalDb.Create(dbPath); + var recoveryStorage = new StoreAndForwardStorage(recoveryDb.Db, NullLogger.Instance); await recoveryStorage.InitializeAsync(); // Verify all messages are available for retry @@ -69,8 +76,10 @@ public class DualNodeRecoveryTests } finally { - if (File.Exists(dbPath)) - File.Delete(dbPath); + // Dispose before deleting — the master connections anchor the WAL sidecars. + recoveryDb?.Dispose(); + crashedDb?.Dispose(); + TestLocalDb.DeleteFiles(dbPath); } } @@ -79,11 +88,14 @@ public class DualNodeRecoveryTests public async Task SiteTopology_DualCrash_ParkedMessagesPreserved() { var dbPath = Path.Combine(Path.GetTempPath(), $"sf_dual_parked_{Guid.NewGuid():N}.db"); - var connStr = $"Data Source={dbPath}"; + + TestLocalDb? crashedDb = null; + TestLocalDb? recoveryDb = null; try { - var storage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + crashedDb = TestLocalDb.Create(dbPath); + var storage = new StoreAndForwardStorage(crashedDb.Db, NullLogger.Instance); await storage.InitializeAsync(); // Mix of pending and parked messages @@ -114,7 +126,8 @@ public class DualNodeRecoveryTests }); // Dual crash recovery - var recoveryStorage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + recoveryDb = TestLocalDb.Create(dbPath); + var recoveryStorage = new StoreAndForwardStorage(recoveryDb.Db, NullLogger.Instance); await recoveryStorage.InitializeAsync(); var pendingCount = await recoveryStorage.GetMessageCountByStatusAsync(StoreAndForwardMessageStatus.Pending); @@ -134,8 +147,10 @@ public class DualNodeRecoveryTests } finally { - if (File.Exists(dbPath)) - File.Delete(dbPath); + // Dispose before deleting — the master connections anchor the WAL sidecars. + recoveryDb?.Dispose(); + crashedDb?.Dispose(); + TestLocalDb.DeleteFiles(dbPath); } } @@ -176,11 +191,14 @@ public class DualNodeRecoveryTests { // CREATE TABLE IF NOT EXISTS is idempotent — safe to call on recovery var dbPath = Path.Combine(Path.GetTempPath(), $"sf_idempotent_{Guid.NewGuid():N}.db"); - var connStr = $"Data Source={dbPath}"; + + TestLocalDb? localDb1 = null; + TestLocalDb? localDb2 = null; try { - var storage1 = new StoreAndForwardStorage(connStr, NullLogger.Instance); + localDb1 = TestLocalDb.Create(dbPath); + var storage1 = new StoreAndForwardStorage(localDb1.Db, NullLogger.Instance); await storage1.InitializeAsync(); await storage1.EnqueueAsync(new StoreAndForwardMessage @@ -196,7 +214,8 @@ public class DualNodeRecoveryTests }); // Second InitializeAsync on same DB should be safe (no data loss) - var storage2 = new StoreAndForwardStorage(connStr, NullLogger.Instance); + localDb2 = TestLocalDb.Create(dbPath); + var storage2 = new StoreAndForwardStorage(localDb2.Db, NullLogger.Instance); await storage2.InitializeAsync(); var msg = await storage2.GetMessageByIdAsync("test-1"); @@ -205,8 +224,10 @@ public class DualNodeRecoveryTests } finally { - if (File.Exists(dbPath)) - File.Delete(dbPath); + // Dispose before deleting — the master connections anchor the WAL sidecars. + localDb2?.Dispose(); + localDb1?.Dispose(); + TestLocalDb.DeleteFiles(dbPath); } } } diff --git a/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/IntegrationSurfaceTests.cs b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/IntegrationSurfaceTests.cs index e52462db..e3276844 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/IntegrationSurfaceTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/IntegrationSurfaceTests.cs @@ -91,12 +91,12 @@ public class IntegrationSurfaceTests { // Notification Outbox: Notify.Send enqueues into the site Store-and-Forward // Engine and returns the NotificationId handle immediately. - var dbName = $"NotifyWired_{Guid.NewGuid():N}"; - var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared"; - using var keepAlive = new Microsoft.Data.Sqlite.SqliteConnection(connStr); - keepAlive.Open(); + // A real temp-file LocalDb, not the shared-cache in-memory database this used + // before: StoreAndForwardStorage takes ILocalDb now, and LocalDb has no + // in-memory mode. + using var localDb = ZB.MOM.WW.ScadaBridge.TestSupport.TestLocalDb.CreateTemp("NotifyWired"); var storage = new StoreAndForward.StoreAndForwardStorage( - connStr, Microsoft.Extensions.Logging.Abstractions.NullLogger.Instance); + localDb.Db, Microsoft.Extensions.Logging.Abstractions.NullLogger.Instance); await storage.InitializeAsync(); var saf = new StoreAndForward.StoreAndForwardService( storage, new StoreAndForward.StoreAndForwardOptions(), diff --git a/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/RecoveryDrillTests.cs b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/RecoveryDrillTests.cs index f1a3eeb3..b9b4cb7c 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/RecoveryDrillTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/RecoveryDrillTests.cs @@ -3,6 +3,7 @@ using Microsoft.Extensions.Logging.Abstractions; using ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment; using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; using ZB.MOM.WW.ScadaBridge.StoreAndForward; +using ZB.MOM.WW.ScadaBridge.TestSupport; namespace ZB.MOM.WW.ScadaBridge.IntegrationTests; @@ -59,11 +60,15 @@ public class RecoveryDrillTests // Scenario: Communication drops while deploying system-wide artifacts. // The deployment command is buffered by S&F and retried when connection restores. var dbPath = Path.Combine(Path.GetTempPath(), $"sf_commdrop_{Guid.NewGuid():N}.db"); - var connStr = $"Data Source={dbPath}"; + + // The store takes an ILocalDb (LocalDb has no in-memory mode), so the buffer + // lives in a real temp file for the duration of the drill. + TestLocalDb? localDb = null; try { - var storage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + localDb = TestLocalDb.Create(dbPath); + var storage = new StoreAndForwardStorage(localDb.Db, NullLogger.Instance); await storage.InitializeAsync(); var options = new StoreAndForwardOptions @@ -102,8 +107,9 @@ public class RecoveryDrillTests } finally { - if (File.Exists(dbPath)) - File.Delete(dbPath); + // Dispose before deleting — the master connection anchors the WAL sidecars. + localDb?.Dispose(); + TestLocalDb.DeleteFiles(dbPath); } } @@ -115,12 +121,17 @@ public class RecoveryDrillTests // On startup, the Deployment Manager Actor reads configs from SQLite and // recreates Instance Actors. var dbPath = Path.Combine(Path.GetTempPath(), $"sf_restart_{Guid.NewGuid():N}.db"); - var connStr = $"Data Source={dbPath}"; + + // The restarted "process" opens its own local database over the SAME file — that + // is what this drill verifies survives. + TestLocalDb? preRestartDb = null; + TestLocalDb? restartedDb = null; try { // Pre-restart: S&F messages in buffer - var storage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + preRestartDb = TestLocalDb.Create(dbPath); + var storage = new StoreAndForwardStorage(preRestartDb.Db, NullLogger.Instance); await storage.InitializeAsync(); for (var i = 0; i < 3; i++) @@ -140,7 +151,8 @@ public class RecoveryDrillTests } // Post-restart: new storage instance reads same DB - var restartedStorage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + restartedDb = TestLocalDb.Create(dbPath); + var restartedStorage = new StoreAndForwardStorage(restartedDb.Db, NullLogger.Instance); await restartedStorage.InitializeAsync(); var pending = await restartedStorage.GetMessagesForRetryAsync(); @@ -153,8 +165,10 @@ public class RecoveryDrillTests } finally { - if (File.Exists(dbPath)) - File.Delete(dbPath); + // Dispose before deleting — the master connections anchor the WAL sidecars. + restartedDb?.Dispose(); + preRestartDb?.Dispose(); + TestLocalDb.DeleteFiles(dbPath); } } diff --git a/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/SiteFailoverTests.cs b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/SiteFailoverTests.cs index 0eb2d110..762f64b1 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/SiteFailoverTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/SiteFailoverTests.cs @@ -4,6 +4,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health; using ZB.MOM.WW.ScadaBridge.Commons.Messages.Streaming; using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; using ZB.MOM.WW.ScadaBridge.StoreAndForward; +using ZB.MOM.WW.ScadaBridge.TestSupport; namespace ZB.MOM.WW.ScadaBridge.IntegrationTests; @@ -21,12 +22,18 @@ public class SiteFailoverTests // Simulates site failover: messages buffered in SQLite survive process restart. // The standby node picks up the same SQLite file and retries pending messages. var dbPath = Path.Combine(Path.GetTempPath(), $"sf_failover_{Guid.NewGuid():N}.db"); - var connStr = $"Data Source={dbPath}"; + + // The store takes an ILocalDb; each "node" opens its own local database over the + // SAME file, which is how this test models the standby picking up the primary's + // buffer after failover. + TestLocalDb? primaryDb = null; + TestLocalDb? standbyDb = null; try { // Phase 1: Buffer messages on "primary" node - var storage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + primaryDb = TestLocalDb.Create(dbPath); + var storage = new StoreAndForwardStorage(primaryDb.Db, NullLogger.Instance); await storage.InitializeAsync(); var message = new StoreAndForwardMessage @@ -46,7 +53,8 @@ public class SiteFailoverTests await storage.EnqueueAsync(message); // Phase 2: "Standby" node opens the same database (simulating failover) - var standbyStorage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + standbyDb = TestLocalDb.Create(dbPath); + var standbyStorage = new StoreAndForwardStorage(standbyDb.Db, NullLogger.Instance); await standbyStorage.InitializeAsync(); var pending = await standbyStorage.GetMessagesForRetryAsync(); @@ -57,8 +65,10 @@ public class SiteFailoverTests } finally { - if (File.Exists(dbPath)) - File.Delete(dbPath); + // Dispose before deleting — the master connections anchor the WAL sidecars. + standbyDb?.Dispose(); + primaryDb?.Dispose(); + TestLocalDb.DeleteFiles(dbPath); } } @@ -67,11 +77,14 @@ public class SiteFailoverTests public async Task StoreAndForward_ParkedMessages_SurviveFailover() { var dbPath = Path.Combine(Path.GetTempPath(), $"sf_parked_{Guid.NewGuid():N}.db"); - var connStr = $"Data Source={dbPath}"; + + TestLocalDb? primaryDb = null; + TestLocalDb? standbyDb = null; try { - var storage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + primaryDb = TestLocalDb.Create(dbPath); + var storage = new StoreAndForwardStorage(primaryDb.Db, NullLogger.Instance); await storage.InitializeAsync(); var parkedMsg = new StoreAndForwardMessage @@ -93,7 +106,8 @@ public class SiteFailoverTests await storage.EnqueueAsync(parkedMsg); // Standby opens same DB - var standbyStorage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + standbyDb = TestLocalDb.Create(dbPath); + var standbyStorage = new StoreAndForwardStorage(standbyDb.Db, NullLogger.Instance); await standbyStorage.InitializeAsync(); var (parked, count) = await standbyStorage.GetParkedMessagesAsync(); @@ -102,8 +116,10 @@ public class SiteFailoverTests } finally { - if (File.Exists(dbPath)) - File.Delete(dbPath); + // Dispose before deleting — the master connections anchor the WAL sidecars. + standbyDb?.Dispose(); + primaryDb?.Dispose(); + TestLocalDb.DeleteFiles(dbPath); } } @@ -160,11 +176,14 @@ public class SiteFailoverTests public async Task StoreAndForward_BufferDepth_ReportedAfterFailover() { var dbPath = Path.Combine(Path.GetTempPath(), $"sf_depth_{Guid.NewGuid():N}.db"); - var connStr = $"Data Source={dbPath}"; + + TestLocalDb? primaryDb = null; + TestLocalDb? standbyDb = null; try { - var storage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + primaryDb = TestLocalDb.Create(dbPath); + var storage = new StoreAndForwardStorage(primaryDb.Db, NullLogger.Instance); await storage.InitializeAsync(); // Enqueue messages in different categories @@ -199,7 +218,8 @@ public class SiteFailoverTests } // After failover, standby reads buffer depths - var standbyStorage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + standbyDb = TestLocalDb.Create(dbPath); + var standbyStorage = new StoreAndForwardStorage(standbyDb.Db, NullLogger.Instance); await standbyStorage.InitializeAsync(); var depths = await standbyStorage.GetBufferDepthByCategoryAsync(); @@ -208,8 +228,10 @@ public class SiteFailoverTests } finally { - if (File.Exists(dbPath)) - File.Delete(dbPath); + // Dispose before deleting — the master connections anchor the WAL sidecars. + standbyDb?.Dispose(); + primaryDb?.Dispose(); + TestLocalDb.DeleteFiles(dbPath); } } } diff --git a/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/ZB.MOM.WW.ScadaBridge.IntegrationTests.csproj b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/ZB.MOM.WW.ScadaBridge.IntegrationTests.csproj index 9eddb8c5..5090cd4d 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/ZB.MOM.WW.ScadaBridge.IntegrationTests.csproj +++ b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/ZB.MOM.WW.ScadaBridge.IntegrationTests.csproj @@ -42,6 +42,7 @@ - + + diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs index 2f848369..8852d3fc 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs @@ -14,6 +14,7 @@ using ZB.MOM.WW.ScadaBridge.SiteRuntime.Deployment; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.TestSupport; +using ZB.MOM.WW.ScadaBridge.TestSupport; using System.Text.Json; using System.Threading; @@ -28,13 +29,13 @@ public class DeploymentManagerActorTests : TestKit, IDisposable private readonly SiteStorageService _storage; private readonly ScriptCompilationService _compilationService; private readonly SharedScriptLibrary _sharedScriptLibrary; - private readonly string _dbFile; + private readonly TestLocalDb _localDb; public DeploymentManagerActorTests() { - _dbFile = Path.Combine(Path.GetTempPath(), $"dm-test-{Guid.NewGuid():N}.db"); + _localDb = TestLocalDb.CreateTemp("dm-test"); _storage = new SiteStorageService( - $"Data Source={_dbFile}", + _localDb.Db, NullLogger.Instance); _storage.InitializeAsync().GetAwaiter().GetResult(); _compilationService = new ScriptCompilationService( @@ -45,8 +46,12 @@ public class DeploymentManagerActorTests : TestKit, IDisposable void IDisposable.Dispose() { + // TestKit teardown first, so no in-flight actor can reach a disposed ILocalDb; + // then dispose the database before deleting — the master connection anchors the WAL. Shutdown(); - try { File.Delete(_dbFile); } catch { /* cleanup */ } + var path = _localDb.Path; + _localDb.Dispose(); + TestLocalDb.DeleteFiles(path); } private IActorRef CreateDeploymentManager( diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerCertReconcileTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerCertReconcileTests.cs index 20dbf736..7ea67d09 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerCertReconcileTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerCertReconcileTests.cs @@ -6,6 +6,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Messages.Management; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts; +using ZB.MOM.WW.ScadaBridge.TestSupport; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors; @@ -46,13 +47,13 @@ akka { private readonly SiteStorageService _storage; private readonly ScriptCompilationService _compilationService; private readonly SharedScriptLibrary _sharedScriptLibrary; - private readonly string _dbFile; + private readonly TestLocalDb _localDb; public DeploymentManagerCertReconcileTests() : base(ClusterConfig, "dm-cert-reconcile") { - _dbFile = Path.Combine(Path.GetTempPath(), $"dm-cert-test-{Guid.NewGuid():N}.db"); + _localDb = TestLocalDb.CreateTemp("dm-cert-test"); _storage = new SiteStorageService( - $"Data Source={_dbFile}", NullLogger.Instance); + _localDb.Db, NullLogger.Instance); _storage.InitializeAsync().GetAwaiter().GetResult(); _compilationService = new ScriptCompilationService( NullLogger.Instance); @@ -62,8 +63,12 @@ akka { void IDisposable.Dispose() { + // TestKit teardown first, so no in-flight actor can reach a disposed ILocalDb; + // then dispose the database before deleting — the master connection anchors the WAL. Shutdown(); - try { File.Delete(_dbFile); } catch { /* cleanup */ } + var path = _localDb.Path; + _localDb.Dispose(); + TestLocalDb.DeleteFiles(path); } /// Forwards every message it receives to a probe, preserving the original sender. diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerLoggerFactoryTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerLoggerFactoryTests.cs index ad74f50b..087b7f0d 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerLoggerFactoryTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerLoggerFactoryTests.cs @@ -8,6 +8,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Types.Flattening; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts; +using ZB.MOM.WW.ScadaBridge.TestSupport; using System.Text.Json; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors; @@ -23,13 +24,13 @@ public class DeploymentManagerLoggerFactoryTests : TestKit, IDisposable private readonly SiteStorageService _storage; private readonly ScriptCompilationService _compilationService; private readonly SharedScriptLibrary _sharedScriptLibrary; - private readonly string _dbFile; + private readonly TestLocalDb _localDb; public DeploymentManagerLoggerFactoryTests() { - _dbFile = Path.Combine(Path.GetTempPath(), $"dm-loggerfactory-test-{Guid.NewGuid():N}.db"); + _localDb = TestLocalDb.CreateTemp("dm-loggerfactory-test"); _storage = new SiteStorageService( - $"Data Source={_dbFile}", + _localDb.Db, NullLogger.Instance); _storage.InitializeAsync().GetAwaiter().GetResult(); _compilationService = new ScriptCompilationService( @@ -40,8 +41,12 @@ public class DeploymentManagerLoggerFactoryTests : TestKit, IDisposable void IDisposable.Dispose() { + // TestKit teardown first, so no in-flight actor can reach a disposed ILocalDb; + // then dispose the database before deleting — the master connection anchors the WAL. Shutdown(); - try { File.Delete(_dbFile); } catch { /* cleanup */ } + var path = _localDb.Path; + _localDb.Dispose(); + TestLocalDb.DeleteFiles(path); } private static string MakeConfigJson(string instanceName) diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerMediumFindingsTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerMediumFindingsTests.cs index a2351233..ab9adb9c 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerMediumFindingsTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerMediumFindingsTests.cs @@ -1,12 +1,14 @@ using Akka.Actor; using Akka.TestKit.Xunit2; using Microsoft.Extensions.Logging.Abstractions; +using ZB.MOM.WW.LocalDb; using ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment; using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; using ZB.MOM.WW.ScadaBridge.Commons.Types.Flattening; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts; +using ZB.MOM.WW.ScadaBridge.TestSupport; using System.Text.Json; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors; @@ -20,11 +22,11 @@ public class DeploymentManagerMediumFindingsTests : TestKit, IDisposable { private readonly ScriptCompilationService _compilationService; private readonly SharedScriptLibrary _sharedScriptLibrary; - private readonly string _dbFile; + private readonly TestLocalDb _localDb; public DeploymentManagerMediumFindingsTests() { - _dbFile = Path.Combine(Path.GetTempPath(), $"dm-medium-test-{Guid.NewGuid():N}.db"); + _localDb = TestLocalDb.CreateTemp("dm-medium-test"); _compilationService = new ScriptCompilationService( NullLogger.Instance); _sharedScriptLibrary = new SharedScriptLibrary( @@ -33,12 +35,16 @@ public class DeploymentManagerMediumFindingsTests : TestKit, IDisposable void IDisposable.Dispose() { + // TestKit teardown first, so no in-flight actor can reach a disposed ILocalDb; + // then dispose the database before deleting — the master connection anchors the WAL. Shutdown(); - try { File.Delete(_dbFile); } catch { /* cleanup */ } + var path = _localDb.Path; + _localDb.Dispose(); + TestLocalDb.DeleteFiles(path); } - private SiteStorageService NewStorage(string connectionString) - => new(connectionString, NullLogger.Instance); + private SiteStorageService NewStorage(ILocalDb localDb) + => new(localDb, NullLogger.Instance); private IActorRef CreateDeploymentManager(SiteStorageService storage, IActorRef? dclManager = null) { @@ -100,22 +106,37 @@ public class DeploymentManagerMediumFindingsTests : TestKit, IDisposable [Fact] public async Task Deploy_PersistenceFailure_ReportsFailedNotSuccess() { - // A connection string pointing at an unwritable path makes every storage - // write throw, so StoreDeployedConfigAsync fails. - var badPath = Path.Combine( - Path.GetTempPath(), $"no-such-dir-{Guid.NewGuid():N}", "site.db"); - var storage = NewStorage($"Data Source={badPath}"); + // A storage over a database whose site tables were never created makes every + // storage operation throw ("no such table"), so StoreDeployedConfigAsync fails. + // + // This replaces the old "connection string pointing at an unwritable path" trick: + // the service now takes an ILocalDb rather than a connection string, and LocalDb + // opens its file eagerly in its own constructor — so an unopenable path fails + // while building the fixture, before there is a storage to hand the actor at all. + // Skipping InitializeAsync is the equivalent lever on the new seam, and fails the + // same way the old one did: reads AND writes both throw. + var uninitialized = TestLocalDb.CreateTemp("dm-medium-persist-fail"); + try + { + var storage = NewStorage(uninitialized.Db); - var actor = CreateDeploymentManager(storage); - await Task.Delay(500); // empty startup + var actor = CreateDeploymentManager(storage); + await Task.Delay(500); // empty startup - actor.Tell(new DeployInstanceCommand( - "dep-fail", "FailPump", "h1", MakeConfigJson("FailPump"), "admin", DateTimeOffset.UtcNow)); + actor.Tell(new DeployInstanceCommand( + "dep-fail", "FailPump", "h1", MakeConfigJson("FailPump"), "admin", DateTimeOffset.UtcNow)); - var response = ExpectMsg(TimeSpan.FromSeconds(10)); - Assert.Equal("FailPump", response.InstanceUniqueName); - Assert.Equal(DeploymentStatus.Failed, response.Status); - Assert.False(string.IsNullOrEmpty(response.ErrorMessage)); + var response = ExpectMsg(TimeSpan.FromSeconds(10)); + Assert.Equal("FailPump", response.InstanceUniqueName); + Assert.Equal(DeploymentStatus.Failed, response.Status); + Assert.False(string.IsNullOrEmpty(response.ErrorMessage)); + } + finally + { + var path = uninitialized.Path; + uninitialized.Dispose(); + TestLocalDb.DeleteFiles(path); + } } /// @@ -126,7 +147,7 @@ public class DeploymentManagerMediumFindingsTests : TestKit, IDisposable [Fact] public async Task Deploy_Success_ReportsSuccessAndPersistsConfig() { - var storage = NewStorage($"Data Source={_dbFile}"); + var storage = NewStorage(_localDb.Db); await storage.InitializeAsync(); var actor = CreateDeploymentManager(storage); @@ -152,7 +173,7 @@ public class DeploymentManagerMediumFindingsTests : TestKit, IDisposable [Fact] public async Task EnsureDclConnections_ConnectionConfigChanged_ReissuesCreateCommand() { - var storage = NewStorage($"Data Source={_dbFile}"); + var storage = NewStorage(_localDb.Db); await storage.InitializeAsync(); var dcl = CreateTestProbe(); @@ -191,7 +212,7 @@ public class DeploymentManagerMediumFindingsTests : TestKit, IDisposable [Fact] public async Task EnsureDclConnections_UnchangedConfig_DoesNotReissueCreateCommand() { - var storage = NewStorage($"Data Source={_dbFile}"); + var storage = NewStorage(_localDb.Db); await storage.InitializeAsync(); var dcl = CreateTestProbe(); @@ -223,7 +244,7 @@ public class DeploymentManagerMediumFindingsTests : TestKit, IDisposable [Fact] public async Task Startup_WithSharedScripts_LoadsConfigsAndStaysResponsive() { - var storage = NewStorage($"Data Source={_dbFile}"); + var storage = NewStorage(_localDb.Db); await storage.InitializeAsync(); // Several shared scripts to compile during startup. diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerRedeployTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerRedeployTests.cs index e4e90fdf..a853d061 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerRedeployTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerRedeployTests.cs @@ -11,6 +11,7 @@ using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.TestSupport; +using ZB.MOM.WW.ScadaBridge.TestSupport; using System.Text.Json; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors; @@ -25,13 +26,13 @@ public class DeploymentManagerRedeployTests : TestKit, IDisposable private readonly SiteStorageService _storage; private readonly ScriptCompilationService _compilationService; private readonly SharedScriptLibrary _sharedScriptLibrary; - private readonly string _dbFile; + private readonly TestLocalDb _localDb; public DeploymentManagerRedeployTests() { - _dbFile = Path.Combine(Path.GetTempPath(), $"dm-redeploy-test-{Guid.NewGuid():N}.db"); + _localDb = TestLocalDb.CreateTemp("dm-redeploy-test"); _storage = new SiteStorageService( - $"Data Source={_dbFile}", + _localDb.Db, NullLogger.Instance); _storage.InitializeAsync().GetAwaiter().GetResult(); _compilationService = new ScriptCompilationService( @@ -42,8 +43,12 @@ public class DeploymentManagerRedeployTests : TestKit, IDisposable void IDisposable.Dispose() { + // TestKit teardown first, so no in-flight actor can reach a disposed ILocalDb; + // then dispose the database before deleting — the master connection anchors the WAL. Shutdown(); - try { File.Delete(_dbFile); } catch { /* cleanup */ } + var path = _localDb.Path; + _localDb.Dispose(); + TestLocalDb.DeleteFiles(path); } private IActorRef CreateDeploymentManager( diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorChildAttributeRaceTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorChildAttributeRaceTests.cs index f01cde4e..437b653b 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorChildAttributeRaceTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorChildAttributeRaceTests.cs @@ -7,6 +7,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Types.Flattening; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts; +using ZB.MOM.WW.ScadaBridge.TestSupport; using System.Reflection; using System.Text.Json; @@ -39,13 +40,13 @@ public class InstanceActorChildAttributeRaceTests : TestKit, IDisposable private readonly ScriptCompilationService _compilationService; private readonly SharedScriptLibrary _sharedScriptLibrary; private readonly SiteRuntimeOptions _options; - private readonly string _dbFile; + private readonly TestLocalDb _localDb; public InstanceActorChildAttributeRaceTests() { - _dbFile = Path.Combine(Path.GetTempPath(), $"instance-race-test-{Guid.NewGuid():N}.db"); + _localDb = TestLocalDb.CreateTemp("instance-race-test"); _storage = new SiteStorageService( - $"Data Source={_dbFile}", + _localDb.Db, NullLogger.Instance); _storage.InitializeAsync().GetAwaiter().GetResult(); _compilationService = new ScriptCompilationService( @@ -61,8 +62,12 @@ public class InstanceActorChildAttributeRaceTests : TestKit, IDisposable void IDisposable.Dispose() { + // TestKit teardown first, so no in-flight actor can reach a disposed ILocalDb; + // then dispose the database before deleting — the master connection anchors the WAL. Shutdown(); - try { File.Delete(_dbFile); } catch { /* cleanup */ } + var path = _localDb.Path; + _localDb.Dispose(); + TestLocalDb.DeleteFiles(path); } private static FlattenedConfiguration BuildConfig(string instanceName) diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorChildRoutingTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorChildRoutingTests.cs index 93b4b1ed..cb80b5a4 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorChildRoutingTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorChildRoutingTests.cs @@ -5,6 +5,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Types.Flattening; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts; +using ZB.MOM.WW.ScadaBridge.TestSupport; using System.Text.Json; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors; @@ -22,13 +23,13 @@ public class InstanceActorChildRoutingTests : TestKit, IDisposable private readonly ScriptCompilationService _compilationService; private readonly SharedScriptLibrary _sharedScriptLibrary; private readonly SiteRuntimeOptions _options; - private readonly string _dbFile; + private readonly TestLocalDb _localDb; public InstanceActorChildRoutingTests() { - _dbFile = Path.Combine(Path.GetTempPath(), $"instance-routing-test-{Guid.NewGuid():N}.db"); + _localDb = TestLocalDb.CreateTemp("instance-routing-test"); _storage = new SiteStorageService( - $"Data Source={_dbFile}", + _localDb.Db, NullLogger.Instance); _storage.InitializeAsync().GetAwaiter().GetResult(); _compilationService = new ScriptCompilationService( @@ -44,8 +45,12 @@ public class InstanceActorChildRoutingTests : TestKit, IDisposable void IDisposable.Dispose() { + // TestKit teardown first, so no in-flight actor can reach a disposed ILocalDb; + // then dispose the database before deleting — the master connection anchors the WAL. Shutdown(); - try { File.Delete(_dbFile); } catch { /* cleanup */ } + var path = _localDb.Path; + _localDb.Dispose(); + TestLocalDb.DeleteFiles(path); } private static FlattenedConfiguration ScriptsAB_and_Expression(string instanceName) diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorIntegrationTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorIntegrationTests.cs index 6933cef0..e2d0af31 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorIntegrationTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorIntegrationTests.cs @@ -9,6 +9,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Types.Flattening; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts; +using ZB.MOM.WW.ScadaBridge.TestSupport; using System.Text.Json; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors; @@ -22,13 +23,13 @@ public class InstanceActorIntegrationTests : TestKit, IDisposable private readonly ScriptCompilationService _compilationService; private readonly SharedScriptLibrary _sharedScriptLibrary; private readonly SiteRuntimeOptions _options; - private readonly string _dbFile; + private readonly TestLocalDb _localDb; public InstanceActorIntegrationTests() { - _dbFile = Path.Combine(Path.GetTempPath(), $"instance-int-test-{Guid.NewGuid():N}.db"); + _localDb = TestLocalDb.CreateTemp("instance-int-test"); _storage = new SiteStorageService( - $"Data Source={_dbFile}", + _localDb.Db, NullLogger.Instance); _storage.InitializeAsync().GetAwaiter().GetResult(); _compilationService = new ScriptCompilationService( @@ -45,8 +46,12 @@ public class InstanceActorIntegrationTests : TestKit, IDisposable void IDisposable.Dispose() { + // TestKit teardown first, so no in-flight actor can reach a disposed ILocalDb; + // then dispose the database before deleting — the master connection anchors the WAL. Shutdown(); - try { File.Delete(_dbFile); } catch { /* cleanup */ } + var path = _localDb.Path; + _localDb.Dispose(); + TestLocalDb.DeleteFiles(path); } private IActorRef CreateInstanceWithScripts( diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorNativeAlarmTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorNativeAlarmTests.cs index 33b7bd37..8cfcbf41 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorNativeAlarmTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorNativeAlarmTests.cs @@ -12,6 +12,7 @@ using ZB.MOM.WW.ScadaBridge.SiteRuntime; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts; +using ZB.MOM.WW.ScadaBridge.TestSupport; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors; @@ -25,12 +26,12 @@ public class InstanceActorNativeAlarmTests : TestKit, IDisposable private readonly ScriptCompilationService _compilationService; private readonly SharedScriptLibrary _sharedScriptLibrary; private readonly SiteRuntimeOptions _options = new(); - private readonly string _dbFile; + private readonly TestLocalDb _localDb; public InstanceActorNativeAlarmTests() { - _dbFile = Path.Combine(Path.GetTempPath(), $"instance-native-{Guid.NewGuid():N}.db"); - _storage = new SiteStorageService($"Data Source={_dbFile}", NullLogger.Instance); + _localDb = TestLocalDb.CreateTemp("instance-native"); + _storage = new SiteStorageService(_localDb.Db, NullLogger.Instance); _storage.InitializeAsync().GetAwaiter().GetResult(); _compilationService = new ScriptCompilationService(NullLogger.Instance); _sharedScriptLibrary = new SharedScriptLibrary(_compilationService, NullLogger.Instance); @@ -149,7 +150,11 @@ public class InstanceActorNativeAlarmTests : TestKit, IDisposable void IDisposable.Dispose() { + // TestKit teardown first, so no in-flight actor can reach a disposed ILocalDb; + // then dispose the database before deleting — the master connection anchors the WAL. Shutdown(); - try { File.Delete(_dbFile); } catch { /* cleanup */ } + var path = _localDb.Path; + _localDb.Dispose(); + TestLocalDb.DeleteFiles(path); } } diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorSetAttributeTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorSetAttributeTests.cs index 90256837..c1540b93 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorSetAttributeTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorSetAttributeTests.cs @@ -8,6 +8,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Types.Flattening; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts; +using ZB.MOM.WW.ScadaBridge.TestSupport; using System.Text.Json; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors; @@ -23,13 +24,13 @@ public class InstanceActorSetAttributeTests : TestKit, IDisposable private readonly ScriptCompilationService _compilationService; private readonly SharedScriptLibrary _sharedScriptLibrary; private readonly SiteRuntimeOptions _options; - private readonly string _dbFile; + private readonly TestLocalDb _localDb; public InstanceActorSetAttributeTests() { - _dbFile = Path.Combine(Path.GetTempPath(), $"instance-setattr-test-{Guid.NewGuid():N}.db"); + _localDb = TestLocalDb.CreateTemp("instance-setattr-test"); _storage = new SiteStorageService( - $"Data Source={_dbFile}", + _localDb.Db, NullLogger.Instance); _storage.InitializeAsync().GetAwaiter().GetResult(); _compilationService = new ScriptCompilationService( @@ -41,8 +42,12 @@ public class InstanceActorSetAttributeTests : TestKit, IDisposable void IDisposable.Dispose() { + // TestKit teardown first, so no in-flight actor can reach a disposed ILocalDb; + // then dispose the database before deleting — the master connection anchors the WAL. Shutdown(); - try { File.Delete(_dbFile); } catch { /* cleanup */ } + var path = _localDb.Path; + _localDb.Dispose(); + TestLocalDb.DeleteFiles(path); } private IActorRef CreateInstanceActor(string instanceName, FlattenedConfiguration config, IActorRef? dclManager) diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorTests.cs index bc8f3c31..5657c1a8 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorTests.cs @@ -12,6 +12,7 @@ using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.TestSupport; +using ZB.MOM.WW.ScadaBridge.TestSupport; using System.Text.Json; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors; @@ -25,13 +26,13 @@ public class InstanceActorTests : TestKit, IDisposable private readonly ScriptCompilationService _compilationService; private readonly SharedScriptLibrary _sharedScriptLibrary; private readonly SiteRuntimeOptions _options; - private readonly string _dbFile; + private readonly TestLocalDb _localDb; public InstanceActorTests() { - _dbFile = Path.Combine(Path.GetTempPath(), $"instance-actor-test-{Guid.NewGuid():N}.db"); + _localDb = TestLocalDb.CreateTemp("instance-actor-test"); _storage = new SiteStorageService( - $"Data Source={_dbFile}", + _localDb.Db, NullLogger.Instance); _storage.InitializeAsync().GetAwaiter().GetResult(); _compilationService = new ScriptCompilationService( @@ -56,8 +57,12 @@ public class InstanceActorTests : TestKit, IDisposable void IDisposable.Dispose() { + // TestKit teardown first, so no in-flight actor can reach a disposed ILocalDb; + // then dispose the database before deleting — the master connection anchors the WAL. Shutdown(); - try { File.Delete(_dbFile); } catch { /* cleanup */ } + var path = _localDb.Path; + _localDb.Dispose(); + TestLocalDb.DeleteFiles(path); } // ── M1.6: site event log `instance_lifecycle` category ────────────────── diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorWaitForAttributeTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorWaitForAttributeTests.cs index ca13817d..102947e9 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorWaitForAttributeTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorWaitForAttributeTests.cs @@ -10,6 +10,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Types.Flattening; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts; +using ZB.MOM.WW.ScadaBridge.TestSupport; using System.Text.Json; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors; @@ -26,13 +27,13 @@ public class InstanceActorWaitForAttributeTests : TestKit, IDisposable private readonly ScriptCompilationService _compilationService; private readonly SharedScriptLibrary _sharedScriptLibrary; private readonly SiteRuntimeOptions _options; - private readonly string _dbFile; + private readonly TestLocalDb _localDb; public InstanceActorWaitForAttributeTests() { - _dbFile = Path.Combine(Path.GetTempPath(), $"instance-waitfor-test-{Guid.NewGuid():N}.db"); + _localDb = TestLocalDb.CreateTemp("instance-waitfor-test"); _storage = new SiteStorageService( - $"Data Source={_dbFile}", + _localDb.Db, NullLogger.Instance); _storage.InitializeAsync().GetAwaiter().GetResult(); _compilationService = new ScriptCompilationService( @@ -57,8 +58,12 @@ public class InstanceActorWaitForAttributeTests : TestKit, IDisposable void IDisposable.Dispose() { + // TestKit teardown first, so no in-flight actor can reach a disposed ILocalDb; + // then dispose the database before deleting — the master connection anchors the WAL. Shutdown(); - try { File.Delete(_dbFile); } catch { /* cleanup */ } + var path = _localDb.Path; + _localDb.Dispose(); + TestLocalDb.DeleteFiles(path); } // ── 1. Fast-path: attribute already at target ──────────────────────────── diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/NativeAlarmActorTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/NativeAlarmActorTests.cs index f8dd29ec..c8874722 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/NativeAlarmActorTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/NativeAlarmActorTests.cs @@ -12,6 +12,7 @@ using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Messages; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.TestSupport; +using ZB.MOM.WW.ScadaBridge.TestSupport; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors; @@ -21,14 +22,16 @@ namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors; /// public class NativeAlarmActorTests : TestKit, IDisposable { - private readonly string _dbFile; + private readonly TestLocalDb _localDb; private readonly SiteStorageService _storage; private readonly SiteRuntimeOptions _options = new(); public NativeAlarmActorTests() { - _dbFile = Path.Combine(Path.GetTempPath(), $"naa-{Guid.NewGuid():N}.db"); - _storage = new SiteStorageService($"Data Source={_dbFile}", NullLogger.Instance); + // SiteStorageService takes an ILocalDb now; LocalDb has no in-memory mode, so the + // fixture is a real temp file (deleted in Dispose, after the TestKit shutdown). + _localDb = TestLocalDb.CreateTemp("naa"); + _storage = new SiteStorageService(_localDb.Db, NullLogger.Instance); _storage.InitializeAsync().GetAwaiter().GetResult(); } @@ -442,11 +445,14 @@ public class NativeAlarmActorTests : TestKit, IDisposable void IDisposable.Dispose() { + // Shut the actor system down FIRST: in-flight alarm actors still hold the + // ILocalDb, and their coalesced flush would hit a disposed database otherwise. Shutdown(); - if (File.Exists(_dbFile)) - { - File.Delete(_dbFile); - } + // Then dispose — the master connection anchors the WAL, so the sidecars cannot + // be removed while it is open. + var path = _localDb.Path; + _localDb.Dispose(); + TestLocalDb.DeleteFiles(path); } [Fact] diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/SiteReconciliationActorTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/SiteReconciliationActorTests.cs index 621cf6b4..7fa1825d 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/SiteReconciliationActorTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/SiteReconciliationActorTests.cs @@ -6,6 +6,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Deployment; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; +using ZB.MOM.WW.ScadaBridge.TestSupport; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors; @@ -22,20 +23,28 @@ public class SiteReconciliationActorTests : TestKit, IDisposable private const string NodeId = "node-a"; private readonly SiteStorageService _storage; - private readonly string _dbFile; + private readonly TestLocalDb _localDb; public SiteReconciliationActorTests() { - _dbFile = Path.Combine(Path.GetTempPath(), $"site-reconcile-test-{Guid.NewGuid():N}.db"); + // SiteStorageService takes an ILocalDb now; LocalDb has no in-memory mode, so the + // fixture is a real temp file (deleted in Dispose, after the TestKit shutdown). + _localDb = TestLocalDb.CreateTemp("site-reconcile-test"); _storage = new SiteStorageService( - $"Data Source={_dbFile}", Microsoft.Extensions.Logging.Abstractions.NullLogger.Instance); + _localDb.Db, Microsoft.Extensions.Logging.Abstractions.NullLogger.Instance); _storage.InitializeAsync().GetAwaiter().GetResult(); } void IDisposable.Dispose() { + // Shut the actor system down FIRST: a reconcile continuation may still be + // writing through the ILocalDb, which must outlive the actors. Shutdown(); - try { File.Delete(_dbFile); } catch { /* cleanup */ } + // Then dispose — the master connection anchors the WAL, so the sidecars cannot + // be removed while it is open. + var path = _localDb.Path; + _localDb.Dispose(); + TestLocalDb.DeleteFiles(path); } private IActorRef CreateReconciliationActor( diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/SiteReplicationActorTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/SiteReplicationActorTests.cs index eb67ee4f..0dae7d19 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/SiteReplicationActorTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/SiteReplicationActorTests.cs @@ -12,6 +12,7 @@ using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; using ZB.MOM.WW.ScadaBridge.StoreAndForward; using ZB.MOM.WW.ScadaBridge.Commons.Observability; using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; +using ZB.MOM.WW.ScadaBridge.TestSupport; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors; @@ -48,22 +49,26 @@ akka { private const string SiteRole = "site-test"; private readonly SiteStorageService _storage; + private readonly TestLocalDb _siteLocalDb; + private readonly TestLocalDb _sfLocalDb; private readonly StoreAndForwardStorage _sfStorage; private readonly ReplicationService _replicationService; - private readonly string _dbFile; private readonly string _sfDbFile; public SiteReplicationActorTests() : base(ClusterConfig, "site-repl") { - _dbFile = Path.Combine(Path.GetTempPath(), $"site-repl-test-{Guid.NewGuid():N}.db"); _sfDbFile = Path.Combine(Path.GetTempPath(), $"site-repl-sf-{Guid.NewGuid():N}.db"); + // SiteStorageService takes an ILocalDb now; LocalDb has no in-memory mode, so the + // site store gets its own temp-file database alongside the S&F one. + _siteLocalDb = TestLocalDb.CreateTemp("site-repl-test"); _storage = new SiteStorageService( - $"Data Source={_dbFile}", NullLogger.Instance); + _siteLocalDb.Db, NullLogger.Instance); _storage.InitializeAsync().GetAwaiter().GetResult(); + _sfLocalDb = TestLocalDb.Create(_sfDbFile); _sfStorage = new StoreAndForwardStorage( - $"Data Source={_sfDbFile}", NullLogger.Instance); + _sfLocalDb.Db, NullLogger.Instance); _sfStorage.InitializeAsync().GetAwaiter().GetResult(); _replicationService = new ReplicationService( @@ -73,8 +78,12 @@ akka { void IDisposable.Dispose() { Shutdown(); - try { File.Delete(_dbFile); } catch { /* cleanup */ } - try { File.Delete(_sfDbFile); } catch { /* cleanup */ } + // The master connection anchors the WAL — dispose before deleting. + var siteDbPath = _siteLocalDb.Path; + _siteLocalDb.Dispose(); + _sfLocalDb.Dispose(); + TestLocalDb.DeleteFiles(siteDbPath); + TestLocalDb.DeleteFiles(_sfDbFile); } private IActorRef CreateReplicationActor(IDeploymentConfigFetcher fetcher) => diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/NegativeTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/NegativeTests.cs index 133a1feb..f10835d5 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/NegativeTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/NegativeTests.cs @@ -1,6 +1,7 @@ using Microsoft.Data.Sqlite; using Microsoft.Extensions.Logging.Abstractions; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; +using ZB.MOM.WW.ScadaBridge.TestSupport; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests; @@ -14,18 +15,27 @@ public class NegativeTests { // Per design decision: no alarm state table in site SQLite schema. // The site SQLite stores only deployed configs and static attribute overrides. - var storage = new SiteStorageService( - "Data Source=:memory:", - NullLogger.Instance); - await storage.InitializeAsync(); + // + // SiteStorageService takes an ILocalDb now, and LocalDb has no in-memory mode + // (its Path is a filesystem path), so the service is initialized over a real + // temp file instead of the "Data Source=:memory:" it used before. The manually + // built schema subset below is still a plain in-memory SqliteConnection — it is + // not a LocalDb store, just a scratch database this test asserts against. + var localDb = TestLocalDb.CreateTemp("negative-schema"); + try + { + var storage = new SiteStorageService( + localDb.Db, + NullLogger.Instance); + await storage.InitializeAsync(); - // Try querying a non-existent alarm_states table — should throw - await using var connection = new SqliteConnection("Data Source=:memory:"); - await connection.OpenAsync(); + // Try querying a non-existent alarm_states table — should throw + await using var connection = new SqliteConnection("Data Source=:memory:"); + await connection.OpenAsync(); - // Re-initialize on this connection to get the schema - await using var initCmd = connection.CreateCommand(); - initCmd.CommandText = @" + // Re-initialize on this connection to get the schema + await using var initCmd = connection.CreateCommand(); + initCmd.CommandText = @" CREATE TABLE IF NOT EXISTS deployed_configurations ( instance_unique_name TEXT PRIMARY KEY, config_json TEXT NOT NULL, @@ -41,13 +51,22 @@ public class NegativeTests updated_at TEXT NOT NULL, PRIMARY KEY (instance_unique_name, attribute_name) );"; - await initCmd.ExecuteNonQueryAsync(); + await initCmd.ExecuteNonQueryAsync(); - // Verify alarm_states does NOT exist - await using var checkCmd = connection.CreateCommand(); - checkCmd.CommandText = "SELECT name FROM sqlite_master WHERE type='table' AND name='alarm_states'"; - var result = await checkCmd.ExecuteScalarAsync(); - Assert.Null(result); + // Verify alarm_states does NOT exist + await using var checkCmd = connection.CreateCommand(); + checkCmd.CommandText = "SELECT name FROM sqlite_master WHERE type='table' AND name='alarm_states'"; + var result = await checkCmd.ExecuteScalarAsync(); + Assert.Null(result); + } + finally + { + // Dispose first — the master connection anchors the WAL, so the sidecars + // cannot be removed while it is open. + var path = localDb.Path; + localDb.Dispose(); + TestLocalDb.DeleteFiles(path); + } } [Fact] diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/ArtifactStorageTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/ArtifactStorageTests.cs index 7d3e8423..997c244b 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/ArtifactStorageTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/ArtifactStorageTests.cs @@ -1,5 +1,6 @@ using Microsoft.Extensions.Logging.Abstractions; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; +using ZB.MOM.WW.ScadaBridge.TestSupport; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Persistence; @@ -7,20 +8,24 @@ namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Persistence; /// WP-33: Local Artifact Storage tests — shared scripts, external systems, /// database connections, notification lists. /// +/// +/// Backed by a real temp-file LocalDb: takes an +/// ILocalDb rather than a connection string, and LocalDb has no in-memory mode. +/// public class ArtifactStorageTests : IAsyncLifetime, IDisposable { - private readonly string _dbFile; + private readonly TestLocalDb _localDb; private SiteStorageService _storage = null!; public ArtifactStorageTests() { - _dbFile = Path.Combine(Path.GetTempPath(), $"artifact-test-{Guid.NewGuid():N}.db"); + _localDb = TestLocalDb.CreateTemp("artifact-test"); } public async Task InitializeAsync() { _storage = new SiteStorageService( - $"Data Source={_dbFile}", + _localDb.Db, NullLogger.Instance); await _storage.InitializeAsync(); } @@ -29,7 +34,11 @@ public class ArtifactStorageTests : IAsyncLifetime, IDisposable public void Dispose() { - try { File.Delete(_dbFile); } catch { /* cleanup */ } + // Dispose first — the master connection anchors the WAL, so the sidecars + // cannot be removed while it is open. + var path = _localDb.Path; + _localDb.Dispose(); + TestLocalDb.DeleteFiles(path); } // ── Shared Script Storage ── @@ -132,8 +141,10 @@ public class ArtifactStorageTests : IAsyncLifetime, IDisposable private async Task SeedNotificationRowAsync(string name, string emailsJson) { - await using var connection = new Microsoft.Data.Sqlite.SqliteConnection($"Data Source={_dbFile}"); - await connection.OpenAsync(); + // Seeded through the service's own (already-open) LocalDb connection — a raw + // SqliteConnection would lack the pragmas and the zb_hlc_next() UDF the site + // tables' capture triggers call. + await using var connection = _storage.CreateConnection(); await using var command = connection.CreateCommand(); command.CommandText = "INSERT INTO notification_lists (name, recipient_emails, updated_at) VALUES (@n, @e, @u)"; @@ -145,8 +156,7 @@ public class ArtifactStorageTests : IAsyncLifetime, IDisposable private async Task SeedSmtpRowAsync(string name, string password) { - await using var connection = new Microsoft.Data.Sqlite.SqliteConnection($"Data Source={_dbFile}"); - await connection.OpenAsync(); + await using var connection = _storage.CreateConnection(); await using var command = connection.CreateCommand(); command.CommandText = @"INSERT INTO smtp_configurations (name, server, port, auth_mode, from_address, username, password, oauth_config, updated_at) @@ -159,8 +169,7 @@ public class ArtifactStorageTests : IAsyncLifetime, IDisposable private async Task RowCountAsync(string table) { - await using var connection = new Microsoft.Data.Sqlite.SqliteConnection($"Data Source={_dbFile}"); - await connection.OpenAsync(); + await using var connection = _storage.CreateConnection(); await using var command = connection.CreateCommand(); command.CommandText = $"SELECT COUNT(*) FROM {table}"; return (long)(await command.ExecuteScalarAsync())!; diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/NativeAlarmStateStoreTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/NativeAlarmStateStoreTests.cs index 4063ab37..9a38bd45 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/NativeAlarmStateStoreTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/NativeAlarmStateStoreTests.cs @@ -1,5 +1,6 @@ using Microsoft.Extensions.Logging.Abstractions; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; +using ZB.MOM.WW.ScadaBridge.TestSupport; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Persistence; @@ -7,19 +8,23 @@ namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Persistence; /// Task 14: site-local SQLite native_alarm_state store — mirrored native alarm /// condition snapshots keyed by (instance, source canonical name, source reference). /// +/// +/// Backed by a real temp-file LocalDb: takes an +/// ILocalDb rather than a connection string, and LocalDb has no in-memory mode. +/// public class NativeAlarmStateStoreTests : IAsyncLifetime, IDisposable { - private readonly string _dbFile; + private readonly TestLocalDb _localDb; private SiteStorageService _storage = null!; public NativeAlarmStateStoreTests() { - _dbFile = Path.Combine(Path.GetTempPath(), $"nas-{Guid.NewGuid():N}.db"); + _localDb = TestLocalDb.CreateTemp("nas"); } public async Task InitializeAsync() { - _storage = new SiteStorageService($"Data Source={_dbFile}", NullLogger.Instance); + _storage = new SiteStorageService(_localDb.Db, NullLogger.Instance); await _storage.InitializeAsync(); } @@ -93,9 +98,10 @@ public class NativeAlarmStateStoreTests : IAsyncLifetime, IDisposable public void Dispose() { - if (File.Exists(_dbFile)) - { - File.Delete(_dbFile); - } + // Dispose first — the master connection anchors the WAL, so the sidecars + // cannot be removed while it is open. + var path = _localDb.Path; + _localDb.Dispose(); + TestLocalDb.DeleteFiles(path); } } diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/SiteStorageServiceTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/SiteStorageServiceTests.cs index 549a73ad..585e09c3 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/SiteStorageServiceTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/SiteStorageServiceTests.cs @@ -1,7 +1,7 @@ -using Microsoft.Data.Sqlite; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Logging.Abstractions; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; +using ZB.MOM.WW.ScadaBridge.TestSupport; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Persistence; @@ -9,20 +9,26 @@ namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Persistence; /// Tests for SiteStorageService using file-based SQLite (temp files). /// Validates the schema, CRUD operations, and constraint behavior. /// +/// +/// The service now takes an ILocalDb rather than a connection string, so the fixture +/// is a real temp-file LocalDb. It stays a file (never in-memory): LocalDb has no in-memory +/// mode, and the connections it hands out carry the pragmas and the zb_hlc_next() UDF +/// the site tables' capture triggers depend on. +/// public class SiteStorageServiceTests : IAsyncLifetime, IDisposable { - private readonly string _dbFile; + private readonly TestLocalDb _localDb; private SiteStorageService _storage = null!; public SiteStorageServiceTests() { - _dbFile = Path.Combine(Path.GetTempPath(), $"site-storage-test-{Guid.NewGuid():N}.db"); + _localDb = TestLocalDb.CreateTemp("site-storage-test"); } public async Task InitializeAsync() { _storage = new SiteStorageService( - $"Data Source={_dbFile}", + _localDb.Db, NullLogger.Instance); await _storage.InitializeAsync(); } @@ -31,7 +37,11 @@ public class SiteStorageServiceTests : IAsyncLifetime, IDisposable public void Dispose() { - try { File.Delete(_dbFile); } catch { /* cleanup */ } + // Dispose first — the master connection anchors the WAL, so the sidecars + // cannot be removed while it is open. + var path = _localDb.Path; + _localDb.Dispose(); + TestLocalDb.DeleteFiles(path); } [Fact] @@ -45,10 +55,17 @@ public class SiteStorageServiceTests : IAsyncLifetime, IDisposable [Fact] public async Task Initialize_EnablesWalJournalMode() { - // WAL is set once at InitializeAsync (persistent, database-level). A file-backed DB - // is required — WAL is not available for :memory: databases. - await using var conn = _storage.CreateConnection(); - await conn.OpenAsync(); + // ── Invariant that moved owner ── + // WAL used to be SiteStorageService's own job (an explicit PRAGMA in + // InitializeAsync). LocalDb now owns the file and its pragmas, so the service no + // longer sets it. The guarantee production depends on has NOT moved: without WAL + // the site's concurrent readers and writers start serializing on "database is + // locked". So rather than deleting this test with the code that used to provide + // the pragma, it is retargeted to assert the same guarantee against the new, + // LocalDb-backed service. journal_mode is persistent and file-scoped, so any + // connection observes it. A file-backed DB is still required — WAL is not + // available for :memory: databases, which is also why LocalDb has no in-memory mode. + await using var conn = _storage.CreateConnection(); // already open — do NOT call OpenAsync await using var cmd = conn.CreateCommand(); cmd.CommandText = "PRAGMA journal_mode;"; var mode = (string)(await cmd.ExecuteScalarAsync())!; @@ -219,8 +236,10 @@ public class SiteStorageServiceTests : IAsyncLifetime, IDisposable string instanceName, string configJson, string deploymentId, string revisionHash, bool isEnabled, DateTimeOffset deployedAt) { - await using var conn = new SqliteConnection($"Data Source={_dbFile}"); - await conn.OpenAsync(); + // Seeded through the service's own (already-open) LocalDb connection: a raw + // SqliteConnection would lack the pragmas and the zb_hlc_next() UDF the site + // tables' capture triggers call. + await using var conn = _storage.CreateConnection(); await using var cmd = conn.CreateCommand(); cmd.CommandText = @" INSERT INTO deployed_configurations diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Repositories/SiteRepositoryTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Repositories/SiteRepositoryTests.cs index a5b68ffb..366d5eb8 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Repositories/SiteRepositoryTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Repositories/SiteRepositoryTests.cs @@ -1,6 +1,7 @@ using Microsoft.Extensions.Logging.Abstractions; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Repositories; +using ZB.MOM.WW.ScadaBridge.TestSupport; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Repositories; @@ -17,21 +18,32 @@ namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Repositories; /// public class SiteRepositoryTests : IDisposable { - private readonly string _dbFile; + private readonly TestLocalDb _localDb; public SiteRepositoryTests() { - _dbFile = Path.Combine(Path.GetTempPath(), $"site-repo-test-{Guid.NewGuid():N}.db"); + _localDb = TestLocalDb.CreateTemp("site-repo-test"); } public void Dispose() { - try { File.Delete(_dbFile); } catch { /* cleanup */ } + // Dispose first — the master connection anchors the WAL, so the sidecars + // cannot be removed while it is open. + var path = _localDb.Path; + _localDb.Dispose(); + TestLocalDb.DeleteFiles(path); GC.SuppressFinalize(this); } + /// + /// A brand-new instance over the same site database. + /// The service now takes an ILocalDb instead of a connection string, so the + /// single fixture database is shared while each call still yields a fresh service + /// object — which is what the restart tests below actually vary (the synthetic IDs are + /// derived per service/repository instance, not per connection). + /// private SiteStorageService NewStorage() - => new($"Data Source={_dbFile}", NullLogger.Instance); + => new(_localDb.Db, NullLogger.Instance); /// /// SiteRuntime-006: an external system stored via diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/NotifyHelperTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/NotifyHelperTests.cs index 8b1f7f4f..b256e5a9 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/NotifyHelperTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/NotifyHelperTests.cs @@ -1,12 +1,12 @@ using System.Text.Json; using Akka.Actor; using Akka.TestKit.Xunit2; -using Microsoft.Data.Sqlite; using Microsoft.Extensions.Logging.Abstractions; using ZB.MOM.WW.ScadaBridge.Commons.Messages.Notification; using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts; using ZB.MOM.WW.ScadaBridge.StoreAndForward; +using ZB.MOM.WW.ScadaBridge.TestSupport; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Scripts; @@ -24,18 +24,16 @@ namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Scripts; /// public class NotifyHelperTests : TestKit, IAsyncLifetime, IDisposable { - private readonly SqliteConnection _keepAlive; + private readonly TestLocalDb _localDb; private readonly StoreAndForwardStorage _storage; private readonly StoreAndForwardService _saf; public NotifyHelperTests() { - var dbName = $"NotifyTests_{Guid.NewGuid():N}"; - var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared"; - _keepAlive = new SqliteConnection(connStr); - _keepAlive.Open(); + // LocalDb has no in-memory mode, so the store runs over a real temp file. + _localDb = TestLocalDb.CreateTemp("NotifyTests"); - _storage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + _storage = new StoreAndForwardStorage(_localDb.Db, NullLogger.Instance); var options = new StoreAndForwardOptions { DefaultRetryInterval = TimeSpan.Zero, @@ -53,7 +51,9 @@ public class NotifyHelperTests : TestKit, IAsyncLifetime, IDisposable { if (disposing) { - _keepAlive.Dispose(); + // The master connection anchors the WAL — dispose before deleting. + _localDb.Dispose(); + TestLocalDb.DeleteFiles(_localDb.Path); } base.Dispose(disposing); } diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/NotifySendAuditEmissionTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/NotifySendAuditEmissionTests.cs index 9bfa69c8..4c1f5d7c 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/NotifySendAuditEmissionTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/NotifySendAuditEmissionTests.cs @@ -1,7 +1,6 @@ using System.Text.Json; using Akka.Actor; using Akka.TestKit.Xunit2; -using Microsoft.Data.Sqlite; using Microsoft.Extensions.Logging.Abstractions; using ZB.MOM.WW.Audit; using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services; @@ -10,6 +9,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts; using IAuditWriter = ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services.IAuditWriter; using ZB.MOM.WW.ScadaBridge.StoreAndForward; +using ZB.MOM.WW.ScadaBridge.TestSupport; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Scripts; @@ -63,18 +63,16 @@ public class NotifySendAuditEmissionTests : TestKit, IAsyncLifetime, IDisposable /// private static readonly Guid TestExecutionId = Guid.NewGuid(); - private readonly SqliteConnection _keepAlive; + private readonly TestLocalDb _localDb; private readonly StoreAndForwardStorage _storage; private readonly StoreAndForwardService _saf; public NotifySendAuditEmissionTests() { - var dbName = $"NotifySendAudit_{Guid.NewGuid():N}"; - var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared"; - _keepAlive = new SqliteConnection(connStr); - _keepAlive.Open(); + // LocalDb has no in-memory mode, so the store runs over a real temp file. + _localDb = TestLocalDb.CreateTemp("NotifySendAudit"); - _storage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + _storage = new StoreAndForwardStorage(_localDb.Db, NullLogger.Instance); var options = new StoreAndForwardOptions { DefaultRetryInterval = TimeSpan.Zero, @@ -92,7 +90,9 @@ public class NotifySendAuditEmissionTests : TestKit, IAsyncLifetime, IDisposable { if (disposing) { - _keepAlive.Dispose(); + // The master connection anchors the WAL — dispose before deleting. + _localDb.Dispose(); + TestLocalDb.DeleteFiles(_localDb.Path); } base.Dispose(disposing); } diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.csproj b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.csproj index 0f1db5ff..581a977d 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.csproj +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.csproj @@ -28,6 +28,7 @@ - + + diff --git a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/CachedCallAttemptEmissionTests.cs b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/CachedCallAttemptEmissionTests.cs index d04e8bb9..4bf217e7 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/CachedCallAttemptEmissionTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/CachedCallAttemptEmissionTests.cs @@ -1,8 +1,8 @@ -using Microsoft.Data.Sqlite; using Microsoft.Extensions.Logging.Abstractions; using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services; using ZB.MOM.WW.ScadaBridge.Commons.Types; using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; +using ZB.MOM.WW.ScadaBridge.TestSupport; namespace ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests; @@ -20,7 +20,7 @@ namespace ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests; /// public class CachedCallAttemptEmissionTests : IAsyncLifetime, IDisposable { - private readonly SqliteConnection _keepAlive; + private readonly TestLocalDb _localDb; private readonly StoreAndForwardStorage _storage; private readonly StoreAndForwardService _service; private readonly StoreAndForwardOptions _options; @@ -28,12 +28,9 @@ public class CachedCallAttemptEmissionTests : IAsyncLifetime, IDisposable public CachedCallAttemptEmissionTests() { - var dbName = $"E4Tests_{Guid.NewGuid():N}"; - var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared"; - _keepAlive = new SqliteConnection(connStr); - _keepAlive.Open(); + _localDb = TestLocalDb.CreateTemp("E4Tests"); - _storage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + _storage = new StoreAndForwardStorage(_localDb.Db, NullLogger.Instance); _options = new StoreAndForwardOptions { @@ -55,7 +52,12 @@ public class CachedCallAttemptEmissionTests : IAsyncLifetime, IDisposable public async Task InitializeAsync() => await _storage.InitializeAsync(); public Task DisposeAsync() => Task.CompletedTask; - public void Dispose() => _keepAlive.Dispose(); + public void Dispose() + { + var path = _localDb.Path; + _localDb.Dispose(); + TestLocalDb.DeleteFiles(path); + } /// /// Captures every observer notification so tests can assert on the @@ -476,10 +478,8 @@ public class CachedCallAttemptEmissionTests : IAsyncLifetime, IDisposable // Fresh service over its own storage so StartAsync's pump/timer is isolated // from the shared _service. - var connStr = $"Data Source=SlowObs_{Guid.NewGuid():N};Mode=Memory;Cache=Shared"; - using var keepAlive = new SqliteConnection(connStr); - keepAlive.Open(); - var storage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + var localDb = TestLocalDb.CreateTemp("SlowObs"); + var storage = new StoreAndForwardStorage(localDb.Db, NullLogger.Instance); await storage.InitializeAsync(); var service = new StoreAndForwardService( storage, @@ -515,6 +515,12 @@ public class CachedCallAttemptEmissionTests : IAsyncLifetime, IDisposable } lock (observed) Assert.Single(observed); } - finally { await service.StopAsync(); } + finally + { + await service.StopAsync(); + var path = localDb.Path; + localDb.Dispose(); + TestLocalDb.DeleteFiles(path); + } } } diff --git a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ParkedMessageHandlerActorTests.cs b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ParkedMessageHandlerActorTests.cs index 93240a69..717636a2 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ParkedMessageHandlerActorTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ParkedMessageHandlerActorTests.cs @@ -1,9 +1,9 @@ using Akka.Actor; using Akka.TestKit.Xunit2; -using Microsoft.Data.Sqlite; using Microsoft.Extensions.Logging.Abstractions; using ZB.MOM.WW.ScadaBridge.Commons.Messages.RemoteQuery; using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; +using ZB.MOM.WW.ScadaBridge.TestSupport; namespace ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests; @@ -14,17 +14,15 @@ namespace ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests; /// public class ParkedMessageHandlerActorTests : TestKit, IAsyncLifetime, IDisposable { - private readonly SqliteConnection _keepAlive; + private readonly TestLocalDb _localDb; private readonly StoreAndForwardStorage _storage; private readonly StoreAndForwardService _service; public ParkedMessageHandlerActorTests() { - var connStr = $"Data Source=ActorTests_{Guid.NewGuid():N};Mode=Memory;Cache=Shared"; - _keepAlive = new SqliteConnection(connStr); - _keepAlive.Open(); + _localDb = TestLocalDb.CreateTemp("ActorTests"); - _storage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + _storage = new StoreAndForwardStorage(_localDb.Db, NullLogger.Instance); var options = new StoreAndForwardOptions { @@ -44,7 +42,13 @@ public class ParkedMessageHandlerActorTests : TestKit, IAsyncLifetime, IDisposab protected override void Dispose(bool disposing) { - if (disposing) _keepAlive.Dispose(); + if (disposing) + { + var path = _localDb.Path; + _localDb.Dispose(); + TestLocalDb.DeleteFiles(path); + } + base.Dispose(disposing); } diff --git a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ParkedOperationRelayTests.cs b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ParkedOperationRelayTests.cs index 91861693..426bf681 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ParkedOperationRelayTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ParkedOperationRelayTests.cs @@ -1,10 +1,10 @@ using Akka.Actor; using Akka.TestKit.Xunit2; -using Microsoft.Data.Sqlite; using Microsoft.Extensions.Logging.Abstractions; using ZB.MOM.WW.ScadaBridge.Commons.Messages.RemoteQuery; using ZB.MOM.WW.ScadaBridge.Commons.Types; using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; +using ZB.MOM.WW.ScadaBridge.TestSupport; namespace ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests; @@ -19,17 +19,15 @@ namespace ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests; /// public class ParkedOperationRelayTests : TestKit, IAsyncLifetime, IDisposable { - private readonly SqliteConnection _keepAlive; + private readonly TestLocalDb _localDb; private readonly StoreAndForwardStorage _storage; private readonly StoreAndForwardService _service; public ParkedOperationRelayTests() { - var connStr = $"Data Source=RelayTests_{Guid.NewGuid():N};Mode=Memory;Cache=Shared"; - _keepAlive = new SqliteConnection(connStr); - _keepAlive.Open(); + _localDb = TestLocalDb.CreateTemp("RelayTests"); - _storage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + _storage = new StoreAndForwardStorage(_localDb.Db, NullLogger.Instance); var options = new StoreAndForwardOptions { @@ -49,7 +47,13 @@ public class ParkedOperationRelayTests : TestKit, IAsyncLifetime, IDisposable protected override void Dispose(bool disposing) { - if (disposing) _keepAlive.Dispose(); + if (disposing) + { + var path = _localDb.Path; + _localDb.Dispose(); + TestLocalDb.DeleteFiles(path); + } + base.Dispose(disposing); } diff --git a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/QueueDepthGaugeTests.cs b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/QueueDepthGaugeTests.cs index 57dcea6c..6b435f5f 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/QueueDepthGaugeTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/QueueDepthGaugeTests.cs @@ -1,8 +1,8 @@ using System.Diagnostics.Metrics; -using Microsoft.Data.Sqlite; using Microsoft.Extensions.Logging.Abstractions; using ZB.MOM.WW.ScadaBridge.Commons.Observability; using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; +using ZB.MOM.WW.ScadaBridge.TestSupport; namespace ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests; @@ -20,18 +20,15 @@ namespace ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests; /// public class QueueDepthGaugeTests : IAsyncLifetime, IDisposable { - private readonly SqliteConnection _keepAlive; + private readonly TestLocalDb _localDb; private readonly StoreAndForwardStorage _storage; private readonly StoreAndForwardService _service; public QueueDepthGaugeTests() { - var dbName = $"QueueDepthTests_{Guid.NewGuid():N}"; - var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared"; - _keepAlive = new SqliteConnection(connStr); - _keepAlive.Open(); + _localDb = TestLocalDb.CreateTemp("QueueDepthTests"); - _storage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + _storage = new StoreAndForwardStorage(_localDb.Db, NullLogger.Instance); var options = new StoreAndForwardOptions { @@ -56,7 +53,12 @@ public class QueueDepthGaugeTests : IAsyncLifetime, IDisposable public async Task DisposeAsync() => await _service.StopAsync(); - public void Dispose() => _keepAlive.Dispose(); + public void Dispose() + { + var path = _localDb.Path; + _localDb.Dispose(); + TestLocalDb.DeleteFiles(path); + } /// /// Reads the current value of the scadabridge.store_and_forward.queue.depth diff --git a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ReplicationServiceTests.cs b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ReplicationServiceTests.cs index 7df0d8e8..19fa4b2b 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ReplicationServiceTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ReplicationServiceTests.cs @@ -1,7 +1,7 @@ -using Microsoft.Data.Sqlite; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Logging.Abstractions; using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; +using ZB.MOM.WW.ScadaBridge.TestSupport; namespace ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests; @@ -10,18 +10,15 @@ namespace ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests; /// public class ReplicationServiceTests : IAsyncLifetime, IDisposable { - private readonly SqliteConnection _keepAlive; + private readonly TestLocalDb _localDb; private readonly StoreAndForwardStorage _storage; private readonly ReplicationService _replicationService; public ReplicationServiceTests() { - var dbName = $"RepTests_{Guid.NewGuid():N}"; - var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared"; - _keepAlive = new SqliteConnection(connStr); - _keepAlive.Open(); + _localDb = TestLocalDb.CreateTemp("RepTests"); - _storage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + _storage = new StoreAndForwardStorage(_localDb.Db, NullLogger.Instance); var options = new StoreAndForwardOptions { ReplicationEnabled = true }; _replicationService = new ReplicationService( @@ -32,7 +29,12 @@ public class ReplicationServiceTests : IAsyncLifetime, IDisposable public Task DisposeAsync() => Task.CompletedTask; - public void Dispose() => _keepAlive.Dispose(); + public void Dispose() + { + var path = _localDb.Path; + _localDb.Dispose(); + TestLocalDb.DeleteFiles(path); + } [Fact] public void ReplicateEnqueue_NoHandler_DoesNotThrow() diff --git a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardReplicationTests.cs b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardReplicationTests.cs index 64e584fb..8080ce78 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardReplicationTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardReplicationTests.cs @@ -1,6 +1,6 @@ -using Microsoft.Data.Sqlite; using Microsoft.Extensions.Logging.Abstractions; using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; +using ZB.MOM.WW.ScadaBridge.TestSupport; namespace ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests; @@ -11,18 +11,16 @@ namespace ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests; /// public class StoreAndForwardReplicationTests : IAsyncLifetime, IDisposable { - private readonly SqliteConnection _keepAlive; + private readonly TestLocalDb _localDb; private readonly StoreAndForwardStorage _storage; private readonly StoreAndForwardService _service; private readonly List _replicated = new(); public StoreAndForwardReplicationTests() { - var connStr = $"Data Source=ReplTests_{Guid.NewGuid():N};Mode=Memory;Cache=Shared"; - _keepAlive = new SqliteConnection(connStr); - _keepAlive.Open(); + _localDb = TestLocalDb.CreateTemp("ReplTests"); - _storage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + _storage = new StoreAndForwardStorage(_localDb.Db, NullLogger.Instance); var options = new StoreAndForwardOptions { @@ -45,7 +43,12 @@ public class StoreAndForwardReplicationTests : IAsyncLifetime, IDisposable public async Task InitializeAsync() => await _storage.InitializeAsync(); public Task DisposeAsync() => Task.CompletedTask; - public void Dispose() => _keepAlive.Dispose(); + public void Dispose() + { + var path = _localDb.Path; + _localDb.Dispose(); + TestLocalDb.DeleteFiles(path); + } /// Replication is fire-and-forget (Task.Run); poll until the expected ops arrive. private async Task> WaitForReplicationAsync(int count) diff --git a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardServiceTests.cs b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardServiceTests.cs index 3c7a4c94..434b3d60 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardServiceTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardServiceTests.cs @@ -1,6 +1,6 @@ -using Microsoft.Data.Sqlite; using Microsoft.Extensions.Logging.Abstractions; using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; +using ZB.MOM.WW.ScadaBridge.TestSupport; namespace ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests; @@ -9,20 +9,17 @@ namespace ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests; /// public class StoreAndForwardServiceTests : IAsyncLifetime, IDisposable { - private readonly SqliteConnection _keepAlive; + private readonly TestLocalDb _localDb; private readonly StoreAndForwardStorage _storage; private readonly StoreAndForwardService _service; private readonly StoreAndForwardOptions _options; - private readonly List _extraKeepAlives = new(); + private readonly List _extraLocalDbs = new(); public StoreAndForwardServiceTests() { - var dbName = $"SvcTests_{Guid.NewGuid():N}"; - var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared"; - _keepAlive = new SqliteConnection(connStr); - _keepAlive.Open(); + _localDb = TestLocalDb.CreateTemp("SvcTests"); - _storage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + _storage = new StoreAndForwardStorage(_localDb.Db, NullLogger.Instance); _options = new StoreAndForwardOptions { @@ -41,23 +38,29 @@ public class StoreAndForwardServiceTests : IAsyncLifetime, IDisposable public void Dispose() { - _keepAlive.Dispose(); - foreach (var c in _extraKeepAlives) c.Dispose(); + DisposeLocalDb(_localDb); + foreach (var db in _extraLocalDbs) DisposeLocalDb(db); + } + + /// Disposes a local database, then removes its file and WAL sidecars. + private static void DisposeLocalDb(TestLocalDb localDb) + { + var path = localDb.Path; + localDb.Dispose(); + TestLocalDb.DeleteFiles(path); } /// - /// Builds a fresh service over its own in-memory (shared-cache) SQLite so a test - /// can call StartAsync without racing the shared _service's timer. The - /// keep-alive connection is tracked for disposal. + /// Builds a fresh service over its own local database so a test can call + /// StartAsync without racing the shared _service's timer. The database + /// is tracked for disposal. /// private StoreAndForwardService CreateService(TimeSpan? retryTimerInterval = null) { - var connStr = $"Data Source=DeferTests_{Guid.NewGuid():N};Mode=Memory;Cache=Shared"; - var keepAlive = new SqliteConnection(connStr); - keepAlive.Open(); - _extraKeepAlives.Add(keepAlive); + var localDb = TestLocalDb.CreateTemp("DeferTests"); + _extraLocalDbs.Add(localDb); - var storage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + var storage = new StoreAndForwardStorage(localDb.Db, NullLogger.Instance); storage.InitializeAsync().GetAwaiter().GetResult(); var options = new StoreAndForwardOptions @@ -557,12 +560,10 @@ public class StoreAndForwardServiceTests : IAsyncLifetime, IDisposable // Build a service whose timer fires almost immediately, with a handler // that pauses in the middle of delivery so we can observe StopAsync's // wait behaviour. - var dbName = $"StopWait_{Guid.NewGuid():N}"; - var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared"; - using var keepAlive = new SqliteConnection(connStr); - keepAlive.Open(); + var localDb = TestLocalDb.CreateTemp("StopWait"); + _extraLocalDbs.Add(localDb); - var storage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + var storage = new StoreAndForwardStorage(localDb.Db, NullLogger.Instance); await storage.InitializeAsync(); var options = new StoreAndForwardOptions diff --git a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardSiteEventTests.cs b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardSiteEventTests.cs index 2c3dcea7..52144ee7 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardSiteEventTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardSiteEventTests.cs @@ -1,8 +1,8 @@ using System.Collections.Concurrent; -using Microsoft.Data.Sqlite; using Microsoft.Extensions.Logging.Abstractions; using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; using ZB.MOM.WW.ScadaBridge.SiteEventLogging; +using ZB.MOM.WW.ScadaBridge.TestSupport; namespace ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests; @@ -34,7 +34,7 @@ public class StoreAndForwardSiteEventTests : IAsyncLifetime, IDisposable public long FailedWriteCount => 0; } - private readonly SqliteConnection _keepAlive; + private readonly TestLocalDb _localDb; private readonly StoreAndForwardStorage _storage; private readonly StoreAndForwardOptions _options; private readonly FakeSiteEventLogger _siteLog = new(); @@ -42,12 +42,9 @@ public class StoreAndForwardSiteEventTests : IAsyncLifetime, IDisposable public StoreAndForwardSiteEventTests() { - var dbName = $"SiteEvt_{Guid.NewGuid():N}"; - var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared"; - _keepAlive = new SqliteConnection(connStr); - _keepAlive.Open(); + _localDb = TestLocalDb.CreateTemp("SiteEvt"); - _storage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + _storage = new StoreAndForwardStorage(_localDb.Db, NullLogger.Instance); _options = new StoreAndForwardOptions { DefaultRetryInterval = TimeSpan.Zero, @@ -63,7 +60,12 @@ public class StoreAndForwardSiteEventTests : IAsyncLifetime, IDisposable public async Task InitializeAsync() => await _storage.InitializeAsync(); public Task DisposeAsync() => Task.CompletedTask; - public void Dispose() => _keepAlive.Dispose(); + public void Dispose() + { + var path = _localDb.Path; + _localDb.Dispose(); + TestLocalDb.DeleteFiles(path); + } [Fact] public async Task BufferForRetry_ExternalSystem_EmitsStoreAndForwardSiteEvent() diff --git a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardStorageTests.cs b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardStorageTests.cs index ad6057fb..668a3007 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardStorageTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardStorageTests.cs @@ -1,27 +1,29 @@ using Microsoft.Data.Sqlite; using Microsoft.Extensions.Logging.Abstractions; using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; +using ZB.MOM.WW.ScadaBridge.TestSupport; namespace ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests; /// /// WP-9: Tests for SQLite persistence layer. -/// Uses in-memory SQLite with a kept-alive connection for test isolation. /// +/// +/// Backed by a real temp-file LocalDb rather than the shared-cache in-memory database +/// this class used before: now takes +/// ILocalDb, and LocalDb has no in-memory mode (LocalDb:Path is a +/// filesystem path). Isolation still comes from a fresh database per test — xUnit +/// constructs one instance per test — it is just a file now. +/// public class StoreAndForwardStorageTests : IAsyncLifetime, IDisposable { - private readonly SqliteConnection _keepAlive; + private readonly TestLocalDb _localDb; private readonly StoreAndForwardStorage _storage; - private readonly string _dbName; public StoreAndForwardStorageTests() { - _dbName = $"StorageTests_{Guid.NewGuid():N}"; - var connStr = $"Data Source={_dbName};Mode=Memory;Cache=Shared"; - // Keep one connection alive so the in-memory DB persists - _keepAlive = new SqliteConnection(connStr); - _keepAlive.Open(); - _storage = new StoreAndForwardStorage(connStr, NullLogger.Instance); + _localDb = TestLocalDb.CreateTemp("StorageTests"); + _storage = new StoreAndForwardStorage(_localDb.Db, NullLogger.Instance); } public async Task InitializeAsync() => await _storage.InitializeAsync(); @@ -30,7 +32,11 @@ public class StoreAndForwardStorageTests : IAsyncLifetime, IDisposable public void Dispose() { - _keepAlive.Dispose(); + // Dispose first — the master connection anchors the WAL, so the sidecars + // cannot be removed while it is open. + var path = _localDb.Path; + _localDb.Dispose(); + TestLocalDb.DeleteFiles(path); } [Fact] @@ -360,9 +366,8 @@ public class StoreAndForwardStorageTests : IAsyncLifetime, IDisposable // schema by dropping the table and recreating it without the columns, // inserting directly, then running InitializeAsync (which ALTER-adds // the columns) and reading the row back. - await using (var setup = new SqliteConnection($"Data Source={_dbName};Mode=Memory;Cache=Shared")) + await using (var setup = _localDb.Db.CreateConnection()) { - await setup.OpenAsync(); await using var drop = setup.CreateCommand(); drop.CommandText = @" DROP TABLE IF EXISTS sf_messages; @@ -415,9 +420,8 @@ public class StoreAndForwardStorageTests : IAsyncLifetime, IDisposable bad.LastAttemptAt = null; // due immediately await _storage.EnqueueAsync(bad); - await using (var conn = new SqliteConnection($"Data Source={_dbName};Mode=Memory;Cache=Shared")) + await using (var conn = _localDb.Db.CreateConnection()) { - await conn.OpenAsync(); await using var corrupt = conn.CreateCommand(); corrupt.CommandText = "UPDATE sf_messages SET execution_id = 'not-a-guid' WHERE id = 'bad1';"; @@ -514,9 +518,8 @@ public class StoreAndForwardStorageTests : IAsyncLifetime, IDisposable // ExecutionId rollout) by recreating the table without // parent_execution_id, inserting directly, then running InitializeAsync // which ALTER-adds the column. - await using (var setup = new SqliteConnection($"Data Source={_dbName};Mode=Memory;Cache=Shared")) + await using (var setup = _localDb.Db.CreateConnection()) { - await setup.OpenAsync(); await using var drop = setup.CreateCommand(); drop.CommandText = @" DROP TABLE IF EXISTS sf_messages; @@ -569,9 +572,8 @@ public class StoreAndForwardStorageTests : IAsyncLifetime, IDisposable bad.LastAttemptAt = null; // due immediately await _storage.EnqueueAsync(bad); - await using (var conn = new SqliteConnection($"Data Source={_dbName};Mode=Memory;Cache=Shared")) + await using (var conn = _localDb.Db.CreateConnection()) { - await conn.OpenAsync(); await using var corrupt = conn.CreateCommand(); corrupt.CommandText = "UPDATE sf_messages SET parent_execution_id = 'not-a-guid' WHERE id = 'pbad1';"; @@ -630,57 +632,46 @@ public class StoreAndForwardStorageTests : IAsyncLifetime, IDisposable }; } - [Fact] - public async Task InitializeAsync_FileInMissingDirectory_CreatesDirectory() - { - // SQLite creates the database file on demand but not its parent directory; - // the storage must create the directory itself or OpenAsync fails with - // "unable to open database file" (the cause of the SiteActorPathTests failures). - var directory = Path.Combine(Path.GetTempPath(), "sf-storage-test-" + Guid.NewGuid().ToString("N")); - var dbPath = Path.Combine(directory, "store-and-forward.db"); - Assert.False(Directory.Exists(directory)); - - try - { - var storage = new StoreAndForwardStorage( - $"Data Source={dbPath}", NullLogger.Instance); - - await storage.InitializeAsync(); - - Assert.True(Directory.Exists(directory)); - Assert.True(File.Exists(dbPath)); - } - finally - { - if (Directory.Exists(directory)) - Directory.Delete(directory, recursive: true); - } - } + // ── Invariants that moved owner ── + // + // Directory creation and WAL journal mode used to be this storage class's own job + // (EnsureDatabaseDirectoryExists / an explicit PRAGMA in InitializeAsync). Both moved + // out when the store stopped owning its file. They landed in DIFFERENT places, so the + // coverage split rather than moving wholesale: + // + // * WAL is genuinely LocalDb's — it sets the journal mode on the database it owns. + // The test below still asserts it, now against the LocalDb-backed store. + // + // * Directory creation is NOT LocalDb's. The library does not create the parent + // directory and opens the file eagerly, so a missing directory is a hard boot + // failure. The guarantee had to be re-established explicitly in the Host + // (SiteServiceRegistration.EnsureLocalDbDirectoryExists), and it is pinned there + // by SiteLocalDbDirectoryTests — the layer that now owns it. Asserting it here + // would be asserting a guarantee this class no longer provides. [Fact] - public async Task Initialize_EnablesWalJournalMode_OnFileDatabase() + public async Task Initialize_LeavesTheDatabaseInWalJournalMode() { // WAL lets the retry-sweep lanes, script enqueues, and standby replication - // applies read/write concurrently without "database is locked". journal_mode - // is persistent + file-scoped, so a fresh connection observes it. - var directory = Path.Combine(Path.GetTempPath(), "sf-wal-test-" + Guid.NewGuid().ToString("N")); - var path = Path.Combine(directory, "wal-test.db"); + // applies read/write concurrently without "database is locked". + using var localDb = TestLocalDb.CreateTemp("sf-wal-test"); + var path = localDb.Path; try { var storage = new StoreAndForwardStorage( - $"Data Source={path}", NullLogger.Instance); + localDb.Db, NullLogger.Instance); await storage.InitializeAsync(); - await using var conn = new SqliteConnection($"Data Source={path}"); - await conn.OpenAsync(); + // journal_mode is persistent + file-scoped, so any connection observes it. + await using var conn = localDb.Db.CreateConnection(); await using var cmd = conn.CreateCommand(); cmd.CommandText = "PRAGMA journal_mode"; Assert.Equal("wal", (string)(await cmd.ExecuteScalarAsync())!); } finally { - if (Directory.Exists(directory)) - Directory.Delete(directory, recursive: true); + localDb.Dispose(); + TestLocalDb.DeleteFiles(path); } } @@ -731,8 +722,7 @@ public class StoreAndForwardStorageTests : IAsyncLifetime, IDisposable private async Task ExecRawAsync(string sql, params (string, object)[] parameters) { - await using var conn = new SqliteConnection($"Data Source={_dbName};Mode=Memory;Cache=Shared"); - await conn.OpenAsync(); + await using var conn = _localDb.Db.CreateConnection(); await using var cmd = conn.CreateCommand(); cmd.CommandText = sql; foreach (var (name, value) in parameters) cmd.Parameters.AddWithValue(name, value); @@ -741,8 +731,7 @@ public class StoreAndForwardStorageTests : IAsyncLifetime, IDisposable private async Task ScalarRawAsync(string sql) { - await using var conn = new SqliteConnection($"Data Source={_dbName};Mode=Memory;Cache=Shared"); - await conn.OpenAsync(); + await using var conn = _localDb.Db.CreateConnection(); await using var cmd = conn.CreateCommand(); cmd.CommandText = sql; var result = await cmd.ExecuteScalarAsync(); diff --git a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests.csproj b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests.csproj index ad27b629..e6ebb6b8 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests.csproj +++ b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests.csproj @@ -27,6 +27,7 @@ - + + diff --git a/tests/ZB.MOM.WW.ScadaBridge.TestSupport/TestLocalDb.cs b/tests/ZB.MOM.WW.ScadaBridge.TestSupport/TestLocalDb.cs new file mode 100644 index 00000000..46a68f1f --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.TestSupport/TestLocalDb.cs @@ -0,0 +1,91 @@ +using Microsoft.Extensions.Configuration; +using Microsoft.Extensions.DependencyInjection; +using ZB.MOM.WW.LocalDb; + +namespace ZB.MOM.WW.ScadaBridge.TestSupport; + +/// +/// A real over a temp file, for tests that construct a +/// LocalDb-backed store directly. +/// +/// +/// +/// The stores take rather than a connection string, and there is +/// no in-memory mode — LocalDbOptions.Path is a filesystem path. A real one is +/// used rather than a stub on purpose: connections handed out by ILocalDb carry +/// the zb_hlc_next() UDF that the capture triggers call, so a stub returning a +/// bare SqliteConnection would pass these tests while failing closed the moment +/// the table is registered for replication. +/// +/// +/// No onReady callback and no RegisterReplicated: each store's own schema +/// initialization creates its table, which is what keeps a directly-constructed store +/// self-sufficient. Registration is the host's job (SiteLocalDbSetup.OnReady). +/// +/// +/// This lives in a shared support project rather than being copied per test project: +/// seven test projects construct stores that now take , and +/// duplicating the fixture would mean seven places to keep the WAL-sidecar cleanup and +/// the "real, not stubbed" rationale correct. +/// +/// +public sealed class TestLocalDb : IDisposable +{ + private readonly ServiceProvider _provider; + + private TestLocalDb(ServiceProvider provider, ILocalDb db) + { + _provider = provider; + Db = db; + } + + /// The live local database. + public ILocalDb Db { get; } + + /// Opens a local database at . + public static TestLocalDb Create(string databasePath) + { + var configuration = new ConfigurationBuilder() + .AddInMemoryCollection(new Dictionary + { + ["LocalDb:Path"] = databasePath, + }) + .Build(); + + var provider = new ServiceCollection() + .AddZbLocalDb(configuration) + .BuildServiceProvider(); + + return new TestLocalDb(provider, provider.GetRequiredService()); + } + + /// + /// Creates a local database at a fresh temp path. Dispose the instance, then call + /// with to clean up. + /// + public static TestLocalDb CreateTemp(string prefix = "localdb") + { + var path = System.IO.Path.Combine( + System.IO.Path.GetTempPath(), $"{prefix}-{Guid.NewGuid():N}.db"); + var db = Create(path); + db.Path = path; + return db; + } + + /// The filesystem path, when created via . + public string Path { get; private set; } = string.Empty; + + /// + /// Deletes and its WAL sidecars. Call only after the + /// owning is disposed — the master connection anchors the WAL. + /// + public static void DeleteFiles(string databasePath) + { + foreach (var suffix in new[] { "", "-wal", "-shm" }) + { + try { File.Delete(databasePath + suffix); } catch { /* best effort */ } + } + } + + public void Dispose() => _provider.Dispose(); +} diff --git a/tests/ZB.MOM.WW.ScadaBridge.TestSupport/ZB.MOM.WW.ScadaBridge.TestSupport.csproj b/tests/ZB.MOM.WW.ScadaBridge.TestSupport/ZB.MOM.WW.ScadaBridge.TestSupport.csproj new file mode 100644 index 00000000..dc5e2997 --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.TestSupport/ZB.MOM.WW.ScadaBridge.TestSupport.csproj @@ -0,0 +1,29 @@ + + + + + net10.0 + enable + enable + true + false + + + + + + + + + + + + From 19ab0ac913f8fe5f617b66bb5db9ec69ba5ae2c4 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Mon, 20 Jul 2026 03:06:06 -0400 Subject: [PATCH 34/55] chore(localdb): record tasks 5-6 completion, the latent directory defect, and the CreateConnection contract change Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts --- docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json b/docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json index 69cc6c29..f775ad28 100644 --- a/docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json +++ b/docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json @@ -73,8 +73,8 @@ {"id": 2, "subject": "Task 2: Decision record - close the phase 2 gate", "status": "completed", "classification": "trivial", "blockedBy": [1], "note": "Gate doc status flipped NOT STARTED -> CLOSED; all 5 SS5 questions answered inline (questions kept, not deleted); SS2's N5 duplicate-bound requirement routed explicitly to Task 21."}, {"id": 3, "subject": "Task 3: Extract StoreAndForwardSchema.Apply", "status": "completed", "classification": "small", "blockedBy": [2], "note": "DEVIATION: kept PRAGMA journal_mode=WAL in InitializeAsync - the plan Step 4 snippet drops it but Task 4 says not to move the equivalent pragma; contradictory, and dropping it would regress documented concurrent-writer support before Task 5 makes it moot. Added a legacy-upgrade test beyond the plan's (the specified test asserts against a FRESH table where CREATE TABLE already lists all 16 columns - it would pass with every ALTER deleted). That test found the last_attempt_at_ms backfill lands 1 ms low: julianday() double day-fraction rounding, pre-existing, carried verbatim, asserted with 1 ms tolerance. Suite 154/154."}, {"id": 4, "subject": "Task 4: Extract SiteStorageSchema.Apply", "status": "completed", "classification": "small", "blockedBy": [2], "note": "DEVIATION: TryAddColumnAsync's catch-on-message-text ('duplicate column') became a PRAGMA table_info probe matching OperationTrackingSchema - the message form depends on a non-contractual error string and swallowed unrelated SqliteExceptions; also dropped the ILogger dep, which is what let the class be static. Cost: the per-column 'Migrated: added column' info log is gone (nothing consumes it). PRAGMA journal_mode=WAL stays in InitializeAsync per plan. Added a legacy-upgrade test for the same reason as Task 3. SiteRuntime 532/532, full solution build 0 warnings."}, - {"id": 5, "subject": "Task 5: Rewire StoreAndForwardStorage onto ILocalDb", "status": "pending", "classification": "high-risk", "blockedBy": [3]}, - {"id": 6, "subject": "Task 6: Rewire SiteStorageService onto ILocalDb", "status": "pending", "classification": "high-risk", "blockedBy": [4]}, + {"id": 5, "subject": "Task 5: Rewire StoreAndForwardStorage onto ILocalDb", "status": "completed", "classification": "high-risk", "blockedBy": [3], "note": "Test fallout was ~7x the plan estimate: 40 files across 7 test projects, not \"fixtures\" in one. Most used Mode=Memory;Cache=Shared and LocalDb has NO in-memory mode, so all moved to real temp files. Added shared tests/ZB.MOM.WW.ScadaBridge.TestSupport lib (TestLocalDb) instead of copying the Phase 1 fixture into 7 projects. WAL test retargeted to the LocalDb-backed store; directory-creation test moved to Host.Tests (different owner - see Task 6 note)."}, + {"id": 6, "subject": "Task 6: Rewire SiteStorageService onto ILocalDb", "status": "completed", "classification": "high-risk", "blockedBy": [4], "note": "LATENT PHASE-1 DEFECT FOUND+FIXED: LocalDb does NOT create the parent directory and SqliteLocalDb opens the file eagerly, so a missing dir is a HARD BOOT FAILURE (SQLite Error 14). Default site config is the RELATIVE ./data/site-localdb.db; docker escapes only because the volume mount creates /app/data. The plan wrongly assumed dir-creation moved to LocalDb with file ownership. Fixed via SiteLocalDbDirectory.Ensure(config) before AddZbLocalDb + Host.Tests/SiteLocalDbDirectoryTests (non-vacuity observed: 2 tests failed with exactly Error 14 pre-fix). ALSO a contract change: SiteStorageService.CreateConnection() used to return an UNOPENED connection; it now returns an ALREADY-OPEN one - SiteExternalSystemRepository dropped 5 OpenAsync calls. AddSiteRuntime(string) overload deleted."}, {"id": 7, "subject": "Task 7: Extend SiteLocalDbSetup with the new DDL (not yet registered)", "status": "pending", "classification": "standard", "blockedBy": [5, 6]}, {"id": 8, "subject": "Task 8: Extend the legacy migrator for sf_messages", "status": "pending", "classification": "high-risk", "blockedBy": [7], "note": "Oplog pin test registers sf_messages on its own TestLocalDb - production OnReady doesn't register it until Task 14."}, {"id": 9, "subject": "Task 9: Extend the legacy migrator for the config tables (skip notification/smtp)", "status": "pending", "classification": "high-risk", "blockedBy": [7]}, From fefbbb31da47ff33db4af4cba6120bc826216006 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Mon, 20 Jul 2026 03:14:18 -0400 Subject: [PATCH 35/55] docs(localdb): resume state through wave 2 (tasks 1-6 done, wave 3 next) Records the latent Phase 1 directory defect and its open library-vs-app decision, the CreateConnection contract change, the new TestSupport library, and the verification numbers at the wave-2 boundary. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts --- docs/plans/2026-07-20-phase2-resume-state.md | 100 +++++++++++++++---- 1 file changed, 83 insertions(+), 17 deletions(-) diff --git a/docs/plans/2026-07-20-phase2-resume-state.md b/docs/plans/2026-07-20-phase2-resume-state.md index f7360e46..2786f585 100644 --- a/docs/plans/2026-07-20-phase2-resume-state.md +++ b/docs/plans/2026-07-20-phase2-resume-state.md @@ -6,33 +6,49 @@ Scratch handoff for picking the plan back up. Delete once Phase 2 lands. (execute with `superpowers-extended-cc:executing-plans`; task state in `…-phase2.md.tasks.json`) **Branch:** `feat/localdb-phase2` (from `feat/localdb-phase1`, which is **not** merged/pushed) -**Tree:** clean @ `8652eab9`. Build 0 warnings. +**Tree:** clean @ `19ab0ac9`. Build 0 warnings. **Nothing pushed to origin yet.** --- ## Where we are -**Task 1 (the gate) is DONE. Its STOP verdict has been superseded — Phase 2 is unblocked.** +**Tasks 1–6 are DONE (waves 0, 1, 2). Next up: Wave 3 = Tasks 7, 8, 9.** -Task 1 appeared to fail on a Phase 1 blocker (LocalDb throwing `disk I/O error` under load). -That was **root-caused the same day as observer-induced and not a product defect**: host-side -`sqlite3` reads of the live bind-mounted WAL files reset the WAL under the container. The -unmodified library sustains the full soak load indefinitely. Full write-up in +Task 1's gate is closed with verdict **PROCEED**; its earlier STOP was superseded (the +`disk I/O error` storm was **observer-induced** — host-side `sqlite3` reads of live +bind-mounted WAL files reset the WAL under the container, not a product defect; write-up in [`docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md`](../known-issues/2026-07-20-localdb-disk-io-error-under-load.md) -§0 (cause) and §11a (fixes shipped). +§0/§11a). -**Tasks 2–21 are untouched. No plan code has been written.** +| Task | State | +|---|---| +| 1 Rig soak | done — gate PROCEED | +| 2 Decision record | done — gate doc CLOSED, all §5 questions answered inline | +| 3 `StoreAndForwardSchema.Apply` | done | +| 4 `SiteStorageSchema.Apply` | done | +| 5 S&F store → `ILocalDb` | done | +| 6 `SiteStorageService` → `ILocalDb` | done | +| **7–21** | **not started, no code written** | -### Before starting Task 3 +**Verification at the wave-2 boundary:** full solution build 0 warnings; SiteRuntime 532, +Host 318, AuditLog 355, StoreAndForward 153, ExternalSystemGateway 142, HealthMonitoring 97 +— **1597 passed / 0 failed**. -1. **Task 2** (trivial, ~3 min) — the decision record. Blocked on Task 1's numbers, which are - partly missing (below). -2. **Re-run the cut-short sampling.** The soak died before producing sustained - `sf_messages` / `native_alarm_state` write rates. Use the **`snap` helper now in the plan** - (Task 1 step 2, ~line 217) — copies the `.db`/`-wal`/`-shm` triplet and queries the copy. - **Never** run host-side `sqlite3` against a live file. Both site-a nodes must have been - restarted since any host-side read (they have been — see rig state). -3. Then Wave 1 (Tasks 3 + 4) can be dispatched in parallel. +### Task 1's measured numbers (clean re-run, 2026-07-20) + +Six consecutive 60 s intervals, copy-based `snap()` sampling on restarted nodes: +`sf_messages` **0.80 rows/sec** insert, **0/sec** retry-UPDATE, oplog 0, `native_alarm_state` 0, +max `payload_json` **76 B**, max `config_json` **721 B** (rig — *not* representative), +**zero** SQLite errors in 30 min, both site-a nodes converged at 3564 rows. Honest gap: 0.80/s +is ~1.6 % of the 50/s ceiling and the retry path was never exercised — fine only because that +ceiling is structural, not empirical. + +### Next: Wave 3 (Tasks 7, 8, 9) + +Task 7 extends `SiteLocalDbSetup.OnReady` with the new DDL (**not** registered yet — registration +is Task 14, the cutover). Tasks 8 and 9 are the legacy migrators and are both **high-risk**. +Ordering inside `OnReady` is load-bearing: **DDL → `RegisterReplicated` → migrate**. Rows written +before registration are invisible to the peer forever, silently. --- @@ -58,6 +74,38 @@ From [`2026-07-19-localdb-phase2-soak.md`](2026-07-19-localdb-phase2-soak.md): The plan's own Task-1 method also needed four corrections (metrics sidecar, safe sampling, no alarm generator on this rig, template 4 undeployable) — all recorded in the soak doc §1. +### Found during waves 1–2 (new) + +- **LATENT PHASE-1 DEFECT, found and fixed in Task 6.** The plan assumed directory creation + moved to LocalDb along with file ownership. **It did not.** The LocalDb library never creates + the parent directory, and `SqliteLocalDb` opens the file **eagerly in its constructor**, so a + missing directory is a **hard boot failure** (`SQLite Error 14: unable to open database file`), + not a degraded start. The default site config uses the **relative** path + `./data/site-localdb.db`, so any site node without a pre-existing `data/` fails to boot; the + docker rig escapes only because its volume mount creates `/app/data`. Latent since Phase 1 made + `LocalDb:Path` required. Fixed in ScadaBridge via `SiteLocalDbDirectory.Ensure(config)` called + before `AddZbLocalDb`, pinned by `Host.Tests/SiteLocalDbDirectoryTests`. + **OPEN DECISION FOR THE USER:** the real fix arguably belongs in the `ZB.MOM.WW.LocalDb` + library (it owns the file) — **every other LocalDb consumer still has this gap.** Not done + here because a library change means a version bump and is outside this plan's scope. +- **Silent contract change on `SiteStorageService.CreateConnection()`.** It used to return an + **unopened** connection that callers opened; it now returns an **already-open** LocalDb + connection, and calling `Open`/`OpenAsync` on one **throws**. `SiteExternalSystemRepository` + dropped 5 `OpenAsync` calls. Anything new that consumes this member must not open it. +- **LocalDb has NO in-memory mode.** `LocalDbOptions.Path` is a filesystem path, so every + fixture that used `Mode=Memory;Cache=Shared` had to move to a real temp file. Expect this for + any remaining store that gets rewired. +- **Test fallout from a store constructor change is ~7× what the plan estimates.** Task 5's plan + named "fixtures" in one project; the real blast radius was **40 files across 7 test projects**. + Budget accordingly for Tasks 14/15. +- **`tests/ZB.MOM.WW.ScadaBridge.TestSupport`** is a NEW shared, non-test library holding + `TestLocalDb` (`Create`/`CreateTemp`/`DeleteFiles`; dispose **before** deleting — the master + connection anchors the WAL). Use it rather than copying fixtures. Wired into AuditLog, + ExternalSystemGateway, HealthMonitoring, IntegrationTests, SiteRuntime, StoreAndForward tests. +- **Where behaviour moved owner, retarget tests — do not delete them with the code.** WAL genuinely + is LocalDb's job (test retargeted in place); directory creation is **not** (test moved to + `Host.Tests`). Both directions occurred, so check which owner actually holds the guarantee. + --- ## Rig operational gotchas (hard-won; not obvious from the repo) @@ -101,6 +149,24 @@ Reseeded and healthy; both site-a nodes restarted after the poisoning incident. | `82c869a1` | task record: gate verdict | | `e9e11d63` | root-cause brief for the disk-I/O issue | | `8652eab9` | root cause + hardening: extended SQLite error codes, `SiteAuditTelemetryActor` async-context fix, `reseed.sh` init-script fix | +| `d512572d` | this resume-state doc (first version) | +| `12eb97e0` | **Task 2** — gate CLOSED, all §5 questions answered + Task 1's clean re-run | +| `9e3239c5` | **Task 3** — extract `StoreAndForwardSchema.Apply` | +| `ac5eb12c` | **Task 4** — extract `SiteStorageSchema.Apply` | +| `3dfb288b` | task state: tasks 3–4 + deviations | +| `f2efeb37` | **Tasks 5+6** — both stores take `ILocalDb`; latent directory defect fixed; `TestSupport` lib added | +| `19ab0ac9` | task state: tasks 5–6, the directory defect, the `CreateConnection` contract change | Two known flakes, do not chase: `InstanceActorChildAttributeRaceTests` (intermittent under full-suite load) and `ParentExecutionIdCorrelationTests` (~91 s / times out on a cold MSSQL fixture, ~1 s warm). +Neither reproduced during waves 1–2. + +## Execution notes + +- Waves 1–2 were run **sequentially in the main worktree**, not as the plan's parallel + worktree agents — the tasks were small enough that worktree setup + merge cost more than it + saved, and sequential execution removes the git race the isolation rule exists to prevent. +- Subagents *were* used for the bulk mechanical fixture rewiring (4 of them, disjoint file sets, + explicitly forbidden from running git). **Their reported results were re-verified + independently** — one agent's partial run had missed 2 real failures that a full-suite run + caught. Re-run the suites yourself; do not take an agent's pass at face value. From f8aa02e2a99c7dec4309c8616f4cce4885355cba Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Mon, 20 Jul 2026 03:28:05 -0400 Subject: [PATCH 36/55] feat(localdb): create config + sf_messages tables in the consolidated DB Task 7. SiteLocalDbSetup.OnReady now applies SiteStorageSchema and StoreAndForwardSchema alongside the Phase 1 schemas, so the nine site configuration tables and the store-and-forward buffer live in the consolidated LocalDb file. Deliberately NOT registered for replication. The bespoke SiteReplicationActor and the StoreAndForward ReplicationService still own these tables until the Task 14 cutover deletes both and registers them in one commit; registering early would run two replicators over the same rows and let either one's defects hide behind the other's writes. Pinned by two tests through the real composition root: one asserting all twelve tables exist, one asserting ReplicatedTables is EXACTLY the Phase 1 pair. Non-vacuity verified by removing the DDL and observing the failure. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts --- .../SiteLocalDbSetup.cs | 15 +++++ .../SiteLocalDbWiringTests.cs | 56 +++++++++++++++++++ 2 files changed, 71 insertions(+) diff --git a/src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbSetup.cs b/src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbSetup.cs index 13b08ae4..626c6d5d 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbSetup.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbSetup.cs @@ -1,7 +1,9 @@ using Microsoft.Extensions.Configuration; using ZB.MOM.WW.LocalDb; using ZB.MOM.WW.ScadaBridge.SiteEventLogging; +using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Tracking; +using ZB.MOM.WW.ScadaBridge.StoreAndForward; namespace ZB.MOM.WW.ScadaBridge.Host; @@ -44,6 +46,12 @@ public static class SiteLocalDbSetup { OperationTrackingSchema.Apply(connection); SiteEventLogSchema.Apply(connection); + + // Phase 2: the site's configuration tables and the store-and-forward buffer + // now live in this file too. Created here, deliberately NOT registered — see + // below. + SiteStorageSchema.Apply(connection); + StoreAndForwardSchema.Apply(connection); } // Both tables qualify: each has an explicit primary key (RegisterReplicated @@ -52,6 +60,13 @@ public static class SiteLocalDbSetup db.RegisterReplicated("OperationTracking"); db.RegisterReplicated("site_events"); + // The Phase 2 tables are created above but NOT registered yet. The bespoke + // SiteReplicationActor / StoreAndForward ReplicationService still own replicating + // them until the Task 14 cutover deletes both and registers these in one commit. + // Registering them now would mean two independent replicators writing the same + // rows — harmless in principle (both upsert) but it would mask a defect in either + // one, which is exactly what the cutover needs to be able to see. + // AFTER registration, so migrated rows enter the oplog and reach the peer like // any other write. Before it, they would be invisible to replication forever. SiteLocalDbLegacyMigrator.Migrate(db, config); diff --git a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbWiringTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbWiringTests.cs index c2d3fc91..f9e8e3eb 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbWiringTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbWiringTests.cs @@ -215,6 +215,62 @@ public class SiteLocalDbWiringTests : IDisposable Assert.Contains(LocalDbMetrics.MeterName, SiteServiceRegistration.ObservedMeters); } + [Fact] + public void Site_LocalDb_CreatesThePhase2Tables_InTheConsolidatedFile() + { + // Phase 2 (Task 7). The nine configuration tables and the store-and-forward buffer + // now live in the consolidated file rather than in scadabridge.db and + // store-and-forward.db. Asserting through the REAL composition root is what proves + // OnReady applies both schemas — the stores themselves also call Apply, so a test + // that went through a store would pass even if OnReady never touched them. + var db = _host.Services.GetRequiredService(); + + var tables = TableNames(db); + + // One from each Phase 1 schema, then every Phase 2 table. + Assert.Contains("OperationTracking", tables); + Assert.Contains("site_events", tables); + Assert.Contains("sf_messages", tables); + foreach (var table in new[] + { + "deployed_configurations", "static_attribute_overrides", "shared_scripts", + "external_systems", "database_connections", "notification_lists", + "data_connection_definitions", "smtp_configurations", "native_alarm_state", + }) + { + Assert.Contains(table, tables); + } + } + + [Fact] + public void Site_LocalDb_DoesNotYetRegisterThePhase2Tables() + { + // The "not yet" is the assertion that matters, and it is why this is an EXACTLY + // check rather than a Contains. Until the Task 14 cutover, the bespoke + // SiteReplicationActor and the StoreAndForward ReplicationService still own these + // tables. Registering them early would run two replicators over the same rows and + // hide a defect in either one behind the other's writes. + // + // When Task 14 lands, this test does not get deleted — it gets inverted. + var db = _host.Services.GetRequiredService(); + + Assert.Equal( + ["OperationTracking", "site_events"], + db.ReplicatedTables.Keys.OrderBy(k => k, StringComparer.Ordinal).ToArray()); + } + + private static HashSet TableNames(ILocalDb db) + { + using var connection = db.CreateConnection(); + using var cmd = connection.CreateCommand(); + cmd.CommandText = "SELECT name FROM sqlite_master WHERE type = 'table'"; + using var reader = cmd.ExecuteReader(); + + var names = new HashSet(StringComparer.Ordinal); + while (reader.Read()) names.Add(reader.GetString(0)); + return names; + } + [Fact] public void Site_LocalDb_CreatesTheConfiguredFile() { From bdc0dffea23cd36093d69f83c3cfcd887a5db580 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Mon, 20 Jul 2026 03:32:10 -0400 Subject: [PATCH 37/55] feat(localdb): migrate legacy store-and-forward.db into the consolidated DB MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Task 8. Adds ResolveStoreAndForwardPath + MigrateStoreAndForward, wired as a third call in Migrate alongside the two Phase 1 files. Unlike the Phase 1 paths, the store-and-forward default sits INSIDE the data volume, so a real deployment has a real file here holding undelivered messages. This migration genuinely moves data; losing it would discard exactly the buffered calls store-and-forward exists to protect. No id synthesis: sf_messages.id is already a caller-assigned TEXT primary key, so INSERT OR IGNORE is idempotent across a crash-then-rerun. The copy intersects the legacy column set with the current one rather than naming all 16 columns outright. A file from an older build predates execution_id / parent_execution_id / last_attempt_at_ms, and naming a missing column throws 'no such column' — which the existing reader treats as an unrecognised shape and silently discards every row. A required-column (PK) guard keeps that tolerance from degrading into copying NULL-keyed rows. Four tests: copy+rename, crash-before-rename idempotence, the __localdb_oplog assertion that catches migrate-before-register, and the older-column-set case. All four verified to fail without the Migrate call. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts --- .../SiteLocalDbLegacyMigrator.cs | 168 +++++++++++++++++- .../SiteLocalDbLegacyMigratorTests.cs | 138 +++++++++++++- 2 files changed, 301 insertions(+), 5 deletions(-) diff --git a/src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbLegacyMigrator.cs b/src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbLegacyMigrator.cs index 3e52ab8e..1abb67fc 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbLegacyMigrator.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbLegacyMigrator.cs @@ -5,8 +5,9 @@ using ZB.MOM.WW.LocalDb; namespace ZB.MOM.WW.ScadaBridge.Host; /// -/// One-time copy of the pre-Phase-1 site databases (site-tracking.db and -/// site_events.db) into the consolidated ZB.MOM.WW.LocalDb database. +/// One-time copy of the pre-consolidation site databases — Phase 1's +/// site-tracking.db and site_events.db, and Phase 2's +/// store-and-forward.db — into the consolidated ZB.MOM.WW.LocalDb database. /// /// /// @@ -31,13 +32,20 @@ namespace ZB.MOM.WW.ScadaBridge.Host; /// half-migrated state to reason about. A second boot sees the renamed file and no-ops. /// /// -/// Finding nothing is the expected case on the docker rig. Neither legacy config key +/// Finding nothing is the expected case on the docker rig for the two Phase 1 files. +/// Neither legacy config key /// is set in any rig appsettings, so both fall back to CWD-relative code defaults /// (/app/site-tracking.db, /app/site_events.db) that sit OUTSIDE the mounted /// data volume — meaning they were already being discarded on every container recreate. /// Phase 1 incidentally fixes that data-loss bug by consolidating into /// /app/data/site-localdb.db. A no-op here is a legitimate result, not a failure. /// +/// +/// Store-and-forward is the exception. Its default path is inside the data +/// volume (./data/store-and-forward.db), so a real deployment has a real file there +/// holding undelivered messages. That migration genuinely moves data, and dropping it would +/// silently discard exactly the buffered calls store-and-forward exists to protect. +/// /// public static class SiteLocalDbLegacyMigrator { @@ -49,6 +57,22 @@ public static class SiteLocalDbLegacyMigrator /// Default legacy event-log path (SiteEventLogOptions.DatabasePath). private const string DefaultEventLogPath = "site_events.db"; + /// Default legacy store-and-forward path (StoreAndForwardOptions.SqliteDbPath). + private const string DefaultStoreAndForwardPath = "./data/store-and-forward.db"; + + /// + /// Every column of the current sf_messages schema, in a fixed order. Columns + /// absent from an older legacy file are dropped from the copy rather than failing it — + /// see . + /// + private static readonly string[] StoreAndForwardColumns = + [ + "id", "category", "target", "payload_json", + "retry_count", "max_retries", "retry_interval_ms", + "created_at", "last_attempt_at", "status", "last_error", "origin_instance", + "execution_id", "source_script", "parent_execution_id", "last_attempt_at_ms", + ]; + /// /// Copies any legacy site databases into , then renames them. /// @@ -63,6 +87,7 @@ public static class SiteLocalDbLegacyMigrator MigrateTracking(db, ResolveTrackingPath(config)); MigrateEvents(db, ResolveEventLogPath(config), nodeName); + MigrateStoreAndForward(db, ResolveStoreAndForwardPath(config)); } /// @@ -113,6 +138,86 @@ public static class SiteLocalDbLegacyMigrator return Path.GetFullPath(path); } + /// + /// Resolves the legacy store-and-forward database path from the OLD key, falling back + /// to the code default. Unlike the two Phase 1 paths, this default sits INSIDE the + /// mounted data volume (./data/), so on the docker rig there is a real file here + /// with real buffered messages — this migration is not the usual no-op. + /// + internal static string ResolveStoreAndForwardPath(IConfiguration config) + { + var path = config["ScadaBridge:StoreAndForward:SqliteDbPath"] ?? DefaultStoreAndForwardPath; + + if (string.IsNullOrWhiteSpace(path) || + path.Equals(":memory:", StringComparison.OrdinalIgnoreCase) || + path.StartsWith("file:", StringComparison.OrdinalIgnoreCase)) + { + return string.Empty; + } + + return Path.GetFullPath(path); + } + + /// + /// Copies buffered store-and-forward messages out of the legacy file. + /// + /// + /// No id synthesis, unlike : sf_messages.id is already + /// a caller-assigned TEXT primary key, so INSERT OR IGNORE is naturally idempotent + /// across a crash-then-rerun. + /// + /// These are undelivered messages, so dropping them is real data loss — a buffered call + /// that never reaches its external system is exactly what store-and-forward exists to + /// prevent. That is why the copy tolerates an older column set rather than bailing. + /// + /// + private static void MigrateStoreAndForward(ILocalDb db, string legacyPath) + => MigrateTable(db, legacyPath, "sf_messages", "id", StoreAndForwardColumns); + + /// + /// Copies one table from a legacy file into the consolidated database, then renames the + /// legacy file. + /// + /// + /// The copy is restricted to the columns the legacy table actually has. A file + /// written by an older build predates some columns, and naming a missing column in the + /// SELECT would throw "no such column" — which the reader treats as an unrecognised + /// shape, silently discarding every row in the table. Intersecting first means an old + /// file migrates its data and simply leaves the newer columns at their schema defaults. + /// + /// The consolidated database. + /// The legacy file, which may not exist. + /// Table name, identical on both sides. + /// + /// A column without which the table is not the one we mean — normally the primary key. + /// Copying rows with a NULL PK would be worse than copying nothing. + /// + /// The current schema's full column list, in a fixed order. + private static void MigrateTable( + ILocalDb db, string legacyPath, string table, string requiredColumn, IReadOnlyList columns) + { + if (!ShouldMigrate(legacyPath)) return; + + using (var legacy = OpenLegacyReadOnly(legacyPath)) + { + var present = PresentColumns(legacy, table, columns); + + // An absent table probes as zero columns, so this one guard covers both "old + // file predating the table" and "file we do not recognise". + if (present.Contains(requiredColumn)) + { + using var connection = db.CreateConnection(); + using var transaction = connection.BeginTransaction(); + + CopyRows(legacy, connection, transaction, table, present); + + transaction.Commit(); + } + } + + MarkMigrated(legacyPath); + } + private static void MigrateTracking(ILocalDb db, string legacyPath) { if (!ShouldMigrate(legacyPath)) return; @@ -249,6 +354,63 @@ public static class SiteLocalDbLegacyMigrator return rows; } + private static SqliteConnection OpenLegacyReadOnly(string legacyPath) + { + var connection = new SqliteConnection($"Data Source={legacyPath};Mode=ReadOnly"); + connection.Open(); + return connection; + } + + /// + /// Returns the subset of that the legacy table actually has, + /// in the caller's order. An absent table yields an empty list rather than throwing. + /// + private static List PresentColumns( + SqliteConnection legacy, string table, IReadOnlyList wanted) + { + var present = new HashSet(StringComparer.OrdinalIgnoreCase); + + using (var probe = legacy.CreateCommand()) + { + // Table name is a caller-controlled constant, never user input — safe to + // interpolate (parameters are not permitted as a pragma-function argument). + probe.CommandText = $"SELECT name FROM pragma_table_info('{table}')"; + using var reader = probe.ExecuteReader(); + while (reader.Read()) present.Add(reader.GetString(0)); + } + + return [.. wanted.Where(present.Contains)]; + } + + /// Streams every row of from the legacy table into the target. + private static void CopyRows( + SqliteConnection legacy, + SqliteConnection target, + SqliteTransaction transaction, + string table, + IReadOnlyList columns) + { + var columnList = string.Join(", ", columns); + var parameterList = string.Join(", ", columns.Select((_, i) => $"$p{i}")); + + using var read = legacy.CreateCommand(); + read.CommandText = $"SELECT {columnList} FROM {table};"; + using var reader = read.ExecuteReader(); + + while (reader.Read()) + { + using var write = target.CreateCommand(); + write.Transaction = transaction; + write.CommandText = + $"INSERT OR IGNORE INTO {table} ({columnList}) VALUES ({parameterList});"; + + for (var i = 0; i < columns.Count; i++) + write.Parameters.AddWithValue($"$p{i}", reader.IsDBNull(i) ? DBNull.Value : reader.GetValue(i)); + + write.ExecuteNonQuery(); + } + } + private static void Bind(SqliteCommand cmd, object?[] row) { for (var i = 0; i < row.Length; i++) diff --git a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbLegacyMigratorTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbLegacyMigratorTests.cs index d3065f3c..e12e6be8 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbLegacyMigratorTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbLegacyMigratorTests.cs @@ -40,7 +40,14 @@ public class SiteLocalDbLegacyMigratorTests : IDisposable private string Path_(string name) => System.IO.Path.Combine(_root, name); - /// A consolidated database with both tables created and registered, as the host has it. + /// A consolidated database with the tables created and registered, as the host has it. + /// + /// sf_messages is registered here even though production OnReady does not + /// register it until the Task 14 cutover. The oplog assertion below needs live capture + /// triggers to mean anything, and the property under test — that the migrator runs after + /// registration — is the same either way. The corollary is a real constraint on Task 14: + /// Migrate must stay the LAST call in OnReady, after every registration. + /// private ILocalDb CreateConsolidated(IConfiguration config) { var provider = new ServiceCollection() @@ -49,8 +56,10 @@ public class SiteLocalDbLegacyMigratorTests : IDisposable using var connection = db.CreateConnection(); ZB.MOM.WW.ScadaBridge.SiteRuntime.Tracking.OperationTrackingSchema.Apply(connection); ZB.MOM.WW.ScadaBridge.SiteEventLogging.SiteEventLogSchema.Apply(connection); + ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardSchema.Apply(connection); db.RegisterReplicated("OperationTracking"); db.RegisterReplicated("site_events"); + db.RegisterReplicated("sf_messages"); }) .BuildServiceProvider(); _providers.Add(provider); @@ -59,12 +68,22 @@ public class SiteLocalDbLegacyMigratorTests : IDisposable } private IConfiguration Config( - string? trackingPath = null, string? eventsPath = null, string nodeName = "node-a") + string? trackingPath = null, + string? eventsPath = null, + string nodeName = "node-a", + string? storeAndForwardPath = null) { var values = new Dictionary { ["LocalDb:Path"] = Path_("consolidated.db"), ["ScadaBridge:Node:NodeName"] = nodeName, + + // Always pinned inside the test's own directory, even when a test does not care + // about it. Left unset, the resolver falls back to the CWD-relative code default + // "./data/store-and-forward.db" — and the test run's CWD is the test binary's + // output directory, so an unlucky run could migrate (and RENAME) a real file. + ["ScadaBridge:StoreAndForward:SqliteDbPath"] = + storeAndForwardPath ?? Path_("absent-store-and-forward.db"), }; if (trackingPath is not null) @@ -75,6 +94,28 @@ public class SiteLocalDbLegacyMigratorTests : IDisposable return new ConfigurationBuilder().AddInMemoryCollection(values).Build(); } + /// Seeds a legacy store-and-forward file with the CURRENT column set. + private static void SeedLegacyStoreAndForward(string path, params string[] ids) + { + using var connection = new SqliteConnection($"Data Source={path}"); + connection.Open(); + ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardSchema.Apply(connection); + + foreach (var id in ids) + { + using var cmd = connection.CreateCommand(); + cmd.CommandText = """ + INSERT INTO sf_messages ( + id, category, target, payload_json, retry_count, max_retries, + retry_interval_ms, created_at, status, execution_id) + VALUES ($id, 0, 'ERP', '{"order":1}', 0, 50, 30000, $now, 0, 'exec-1'); + """; + cmd.Parameters.AddWithValue("$id", id); + cmd.Parameters.AddWithValue("$now", DateTime.UtcNow.ToString("o")); + cmd.ExecuteNonQuery(); + } + } + private static void SeedLegacyTracking(string path, params string[] ids) { using var connection = new SqliteConnection($"Data Source={path}"); @@ -261,6 +302,99 @@ public class SiteLocalDbLegacyMigratorTests : IDisposable Assert.Equal(2L, (long)cmd.ExecuteScalar()!); } + [Fact] + public void SfMessages_AreCopiedFromTheLegacyFile() + { + // Phase 2 (Task 8). Unlike the two Phase 1 files, the store-and-forward default path + // is inside the data volume, so on a real deployment this actually moves data: + // undelivered messages that a lost migration would silently discard. + var sfPath = Path_("store-and-forward.db"); + SeedLegacyStoreAndForward(sfPath, "msg-1", "msg-2", "msg-3"); + var config = Config(storeAndForwardPath: sfPath); + var db = CreateConsolidated(config); + + SiteLocalDbLegacyMigrator.Migrate(db, config); + + Assert.Equal(3, CountRows(db, "sf_messages")); + Assert.Equal(["msg-1", "msg-2", "msg-3"], SelectIds(db, "sf_messages", "id")); + + // Renamed, not deleted — same contract as the Phase 1 files. + Assert.False(File.Exists(sfPath)); + Assert.True(File.Exists(sfPath + ".migrated")); + } + + [Fact] + public void SfMessages_Migration_IsIdempotent_WhenRerunAfterACrashBeforeRename() + { + // The crash window: copy committed, rename did not. sf_messages.id is a natural + // TEXT key, so INSERT OR IGNORE absorbs the re-copy with no id synthesis needed. + var sfPath = Path_("store-and-forward.db"); + SeedLegacyStoreAndForward(sfPath, "msg-1", "msg-2"); + var config = Config(storeAndForwardPath: sfPath); + var db = CreateConsolidated(config); + + SiteLocalDbLegacyMigrator.Migrate(db, config); + File.Move(sfPath + ".migrated", sfPath); + SiteLocalDbLegacyMigrator.Migrate(db, config); + + Assert.Equal(2, CountRows(db, "sf_messages")); + } + + [Fact] + public void MigratedSfMessages_EnterTheOplog_SoTheyActuallyReplicate() + { + // Same ordering trap as the Phase 1 tables: migrate before RegisterReplicated and + // the rows never enter the oplog, never reach the peer, and nothing errors. Only an + // assertion on __localdb_oplog catches the ordering being reversed. + var sfPath = Path_("store-and-forward.db"); + SeedLegacyStoreAndForward(sfPath, "msg-1", "msg-2"); + var config = Config(storeAndForwardPath: sfPath); + var db = CreateConsolidated(config); + + SiteLocalDbLegacyMigrator.Migrate(db, config); + + using var connection = db.CreateConnection(); + using var cmd = connection.CreateCommand(); + cmd.CommandText = "SELECT COUNT(*) FROM __localdb_oplog WHERE table_name = 'sf_messages'"; + Assert.Equal(2L, (long)cmd.ExecuteScalar()!); + } + + [Fact] + public void SfMessages_FromAnOlderBuild_MigrateDespiteMissingColumns() + { + // A legacy file written before execution_id / parent_execution_id / last_attempt_at_ms + // existed. Naming a missing column in the SELECT throws "no such column", and the + // reader treats that as an unrecognised shape — which would silently discard every + // buffered message. The copy intersects the column sets instead. + var sfPath = Path_("store-and-forward.db"); + using (var legacy = new SqliteConnection($"Data Source={sfPath}")) + { + legacy.Open(); + using var ddl = legacy.CreateCommand(); + ddl.CommandText = """ + CREATE TABLE sf_messages ( + id TEXT PRIMARY KEY, category INTEGER NOT NULL, target TEXT NOT NULL, + payload_json TEXT NOT NULL, retry_count INTEGER NOT NULL DEFAULT 0, + max_retries INTEGER NOT NULL DEFAULT 50, + retry_interval_ms INTEGER NOT NULL DEFAULT 30000, + created_at TEXT NOT NULL, last_attempt_at TEXT, + status INTEGER NOT NULL DEFAULT 0, last_error TEXT, origin_instance TEXT + ); + INSERT INTO sf_messages (id, category, target, payload_json, created_at) + VALUES ('old-1', 0, 'ERP', '{}', '2026-01-01T00:00:00Z'); + """; + ddl.ExecuteNonQuery(); + } + + var config = Config(storeAndForwardPath: sfPath); + var db = CreateConsolidated(config); + + SiteLocalDbLegacyMigrator.Migrate(db, config); + + Assert.Equal(1, CountRows(db, "sf_messages")); + Assert.Equal(["old-1"], SelectIds(db, "sf_messages", "id")); + } + [Fact] public void LegacyFileWithoutTheExpectedTable_IsTreatedAsEmpty_NotAFailure() { From 5ddc7eed6d6d879104e458439add55d0d7834feb Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Mon, 20 Jul 2026 03:40:31 -0400 Subject: [PATCH 38/55] feat(localdb): migrate legacy scadabridge.db config tables into the consolidated DB MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Task 9. Seven of the nine site configuration tables are copied out of the legacy scadabridge.db in a single transaction, with one rename at the end: a partial config migration would leave a site node running against half its old configuration, which is worse than failing startup outright. notification_lists and smtp_configurations are deliberately NOT migrated. Both are purged on every deploy and permanently empty by design since the site write paths were removed (2026-07-10), but a pre-fix legacy file can still hold rows — and smtp_configurations.password is plaintext. Task 14 makes these tables replicated, so migrating them would push plaintext SMTP passwords across a channel whose only historical payload was exactly that. The tables are still created; only their historical contents stay behind. Generalizes Task 8's MigrateTable into MigrateFile(many tables, one transaction, one rename), keeping the legacy/current column intersection. Five tests, including a column-parity test asserting each declared column list equals the live SiteStorageSchema's. That mismatch is otherwise invisible: a typo'd column is silently dropped by the intersection, and a missing one silently leaves data behind. Non-vacuity verified twice — once by removing the Migrate call (3 fail), once by wrongly adding notification_lists and smtp_configurations to the table map (the skip test fails). Verified: solution build 0 warnings; Host 329, SiteRuntime 532, StoreAndForward 153 — all pass. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts --- .../SiteLocalDbLegacyMigrator.cs | 124 ++++++++++++-- .../SiteLocalDbLegacyMigratorTests.cs | 154 +++++++++++++++++- 2 files changed, 259 insertions(+), 19 deletions(-) diff --git a/src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbLegacyMigrator.cs b/src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbLegacyMigrator.cs index 1abb67fc..42bf6e42 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbLegacyMigrator.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbLegacyMigrator.cs @@ -60,6 +60,9 @@ public static class SiteLocalDbLegacyMigrator /// Default legacy store-and-forward path (StoreAndForwardOptions.SqliteDbPath). private const string DefaultStoreAndForwardPath = "./data/store-and-forward.db"; + /// Default legacy site configuration path (appsettings.Site.json). + private const string DefaultSiteStoragePath = "./data/scadabridge.db"; + /// /// Every column of the current sf_messages schema, in a fixed order. Columns /// absent from an older legacy file are dropped from the copy rather than failing it — @@ -73,6 +76,55 @@ public static class SiteLocalDbLegacyMigrator "execution_id", "source_script", "parent_execution_id", "last_attempt_at_ms", ]; + /// + /// The site configuration tables copied out of the legacy scadabridge.db, with + /// the current schema's columns for each. + /// + /// + /// notification_lists and smtp_configurations are deliberately absent. + /// Both are purged on every deploy and are permanently empty by design — the site-side + /// write paths were removed on 2026-07-10. A pre-fix legacy file can still hold rows, + /// and smtp_configurations.password is plaintext, so migrating them would + /// resurrect plaintext SMTP passwords into a table that Task 14 makes REPLICATED. The + /// tables themselves are still created (see SiteStorageSchema); only their + /// historical contents are left behind. + /// + internal static readonly LegacyTable[] SiteStorageTables = + [ + new("deployed_configurations", "instance_unique_name", + [ + "instance_unique_name", "config_json", "deployment_id", "revision_hash", + "is_enabled", "deployed_at", + ]), + new("static_attribute_overrides", "instance_unique_name", + [ + "instance_unique_name", "attribute_name", "override_value", "updated_at", + ]), + new("shared_scripts", "name", + [ + "name", "code", "parameter_definitions", "return_definition", "updated_at", + ]), + new("external_systems", "name", + [ + "name", "endpoint_url", "auth_type", "auth_configuration", "method_definitions", + "updated_at", "timeout_seconds", + ]), + new("database_connections", "name", + [ + "name", "connection_string", "max_retries", "retry_delay_ms", "updated_at", + ]), + new("data_connection_definitions", "name", + [ + "name", "protocol", "configuration", "backup_configuration", + "failover_retry_count", "updated_at", + ]), + new("native_alarm_state", "instance_unique_name", + [ + "instance_unique_name", "source_canonical_name", "source_reference", + "condition_json", "last_transition_at", "metadata_json", + ]), + ]; + /// /// Copies any legacy site databases into , then renames them. /// @@ -88,6 +140,7 @@ public static class SiteLocalDbLegacyMigrator MigrateTracking(db, ResolveTrackingPath(config)); MigrateEvents(db, ResolveEventLogPath(config), nodeName); MigrateStoreAndForward(db, ResolveStoreAndForwardPath(config)); + MigrateSiteStorage(db, ResolveSiteStoragePath(config)); } /// @@ -158,6 +211,25 @@ public static class SiteLocalDbLegacyMigrator return Path.GetFullPath(path); } + /// + /// Resolves the legacy site configuration database path from the OLD key, falling back + /// to the code default. Like store-and-forward — and unlike the two Phase 1 paths — this + /// default is inside the mounted data volume, so a real deployment has real config here. + /// + internal static string ResolveSiteStoragePath(IConfiguration config) + { + var path = config["ScadaBridge:Database:SiteDbPath"] ?? DefaultSiteStoragePath; + + if (string.IsNullOrWhiteSpace(path) || + path.Equals(":memory:", StringComparison.OrdinalIgnoreCase) || + path.StartsWith("file:", StringComparison.OrdinalIgnoreCase)) + { + return string.Empty; + } + + return Path.GetFullPath(path); + } + /// /// Copies buffered store-and-forward messages out of the legacy file. /// @@ -172,7 +244,19 @@ public static class SiteLocalDbLegacyMigrator /// /// private static void MigrateStoreAndForward(ILocalDb db, string legacyPath) - => MigrateTable(db, legacyPath, "sf_messages", "id", StoreAndForwardColumns); + => MigrateFile(db, legacyPath, [new LegacyTable("sf_messages", "id", StoreAndForwardColumns)]); + + /// + /// Copies the site's configuration tables out of the legacy scadabridge.db. + /// + /// + /// All seven migrated tables live in one file, so they are copied inside a single + /// transaction and the file is renamed once: a partial config migration would leave a + /// site node running against half its old configuration, which is worse than failing + /// startup outright. + /// + private static void MigrateSiteStorage(ILocalDb db, string legacyPath) + => MigrateFile(db, legacyPath, SiteStorageTables); /// /// Copies one table from a legacy file into the consolidated database, then renames the @@ -187,37 +271,41 @@ public static class SiteLocalDbLegacyMigrator /// /// The consolidated database. /// The legacy file, which may not exist. - /// Table name, identical on both sides. - /// - /// A column without which the table is not the one we mean — normally the primary key. - /// Copying rows with a NULL PK would be worse than copying nothing. - /// - /// The current schema's full column list, in a fixed order. - private static void MigrateTable( - ILocalDb db, string legacyPath, string table, string requiredColumn, IReadOnlyList columns) + /// Every table to copy out of this file, in order. + private static void MigrateFile(ILocalDb db, string legacyPath, IReadOnlyList tables) { if (!ShouldMigrate(legacyPath)) return; using (var legacy = OpenLegacyReadOnly(legacyPath)) { - var present = PresentColumns(legacy, table, columns); + using var connection = db.CreateConnection(); + using var transaction = connection.BeginTransaction(); - // An absent table probes as zero columns, so this one guard covers both "old - // file predating the table" and "file we do not recognise". - if (present.Contains(requiredColumn)) + foreach (var table in tables) { - using var connection = db.CreateConnection(); - using var transaction = connection.BeginTransaction(); + var present = PresentColumns(legacy, table.Table, table.Columns); - CopyRows(legacy, connection, transaction, table, present); - - transaction.Commit(); + // An absent table probes as zero columns, so this one guard covers both + // "old file predating the table" and "file we do not recognise". + if (present.Contains(table.RequiredColumn)) + CopyRows(legacy, connection, transaction, table.Table, present); } + + transaction.Commit(); } MarkMigrated(legacyPath); } + /// One table to copy out of a legacy file. + /// Table name, identical on both sides. + /// + /// A column without which the table is not the one we mean — normally the primary key. + /// Copying rows with a NULL PK would be worse than copying nothing. + /// + /// The current schema's full column list, in a fixed order. + internal sealed record LegacyTable(string Table, string RequiredColumn, string[] Columns); + private static void MigrateTracking(ILocalDb db, string legacyPath) { if (!ShouldMigrate(legacyPath)) return; diff --git a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbLegacyMigratorTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbLegacyMigratorTests.cs index e12e6be8..834aa36a 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbLegacyMigratorTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbLegacyMigratorTests.cs @@ -57,9 +57,12 @@ public class SiteLocalDbLegacyMigratorTests : IDisposable ZB.MOM.WW.ScadaBridge.SiteRuntime.Tracking.OperationTrackingSchema.Apply(connection); ZB.MOM.WW.ScadaBridge.SiteEventLogging.SiteEventLogSchema.Apply(connection); ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardSchema.Apply(connection); + ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence.SiteStorageSchema.Apply(connection); db.RegisterReplicated("OperationTracking"); db.RegisterReplicated("site_events"); db.RegisterReplicated("sf_messages"); + foreach (var table in SiteLocalDbLegacyMigrator.SiteStorageTables) + db.RegisterReplicated(table.Table); }) .BuildServiceProvider(); _providers.Add(provider); @@ -71,7 +74,8 @@ public class SiteLocalDbLegacyMigratorTests : IDisposable string? trackingPath = null, string? eventsPath = null, string nodeName = "node-a", - string? storeAndForwardPath = null) + string? storeAndForwardPath = null, + string? siteDbPath = null) { var values = new Dictionary { @@ -84,6 +88,9 @@ public class SiteLocalDbLegacyMigratorTests : IDisposable // output directory, so an unlucky run could migrate (and RENAME) a real file. ["ScadaBridge:StoreAndForward:SqliteDbPath"] = storeAndForwardPath ?? Path_("absent-store-and-forward.db"), + + // Same reasoning: the site-storage default is "./data/scadabridge.db". + ["ScadaBridge:Database:SiteDbPath"] = siteDbPath ?? Path_("absent-scadabridge.db"), }; if (trackingPath is not null) @@ -395,6 +402,151 @@ public class SiteLocalDbLegacyMigratorTests : IDisposable Assert.Equal(["old-1"], SelectIds(db, "sf_messages", "id")); } + /// Seeds a legacy scadabridge.db with the CURRENT config schema and one row per table. + private static void SeedLegacySiteStorage(string path) + { + using var connection = new SqliteConnection($"Data Source={path}"); + connection.Open(); + ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence.SiteStorageSchema.Apply(connection); + + using var cmd = connection.CreateCommand(); + cmd.CommandText = """ + INSERT INTO deployed_configurations + (instance_unique_name, config_json, deployment_id, revision_hash, is_enabled, deployed_at) + VALUES ('inst-1', '{"a":1}', 'dep-1', 'hash-1', 1, '2026-01-01T00:00:00Z'); + INSERT INTO static_attribute_overrides + (instance_unique_name, attribute_name, override_value, updated_at) + VALUES ('inst-1', 'Setpoint', '42', '2026-01-01T00:00:00Z'); + INSERT INTO shared_scripts (name, code, updated_at) + VALUES ('helper', 'return 1;', '2026-01-01T00:00:00Z'); + INSERT INTO external_systems (name, endpoint_url, auth_type, updated_at, timeout_seconds) + VALUES ('ERP', 'http://erp:5200', 'None', '2026-01-01T00:00:00Z', 30); + INSERT INTO database_connections (name, connection_string, updated_at) + VALUES ('BT', 'Server=sql;Database=BT', '2026-01-01T00:00:00Z'); + INSERT INTO data_connection_definitions (name, protocol, updated_at) + VALUES ('opc-1', 'OpcUa', '2026-01-01T00:00:00Z'); + INSERT INTO native_alarm_state + (instance_unique_name, source_canonical_name, source_reference, condition_json, last_transition_at) + VALUES ('inst-1', 'Area.Line', 'ref-1', '{"active":true}', '2026-01-01T00:00:00Z'); + + -- The two that must NOT be migrated. A pre-2026-07-10 file really can hold these. + INSERT INTO notification_lists (name, recipient_emails, updated_at) + VALUES ('ops', 'ops@example.com', '2026-01-01T00:00:00Z'); + INSERT INTO smtp_configurations + (name, server, port, auth_mode, from_address, username, password, updated_at) + VALUES ('smtp', 'smtp.example.com', 587, 'Basic', 'a@b.c', 'user', + 'PLAINTEXT-SECRET', '2026-01-01T00:00:00Z'); + """; + cmd.ExecuteNonQuery(); + } + + [Fact] + public void SiteConfigTables_AreCopiedFromTheLegacyFile() + { + // Phase 2 (Task 9). All seven migrated tables live in one legacy file and are copied + // in a single transaction: a partial config migration would leave a site node running + // against half its old configuration, which is worse than failing startup. + var sitePath = Path_("scadabridge.db"); + SeedLegacySiteStorage(sitePath); + var config = Config(siteDbPath: sitePath); + var db = CreateConsolidated(config); + + SiteLocalDbLegacyMigrator.Migrate(db, config); + + foreach (var table in SiteLocalDbLegacyMigrator.SiteStorageTables) + Assert.Equal(1, CountRows(db, table.Table)); + + Assert.False(File.Exists(sitePath)); + Assert.True(File.Exists(sitePath + ".migrated")); + } + + [Fact] + public void Migration_DoesNotCopyNotificationOrSmtpRows_EvenWhenTheLegacyFileHasThem() + { + // smtp_configurations.password is PLAINTEXT. Both tables are purged on every deploy + // and permanently empty by design since the site write paths were removed + // (2026-07-10), but a pre-fix legacy file can still hold rows — and Task 14 makes + // these tables REPLICATED. Migrating them would push plaintext SMTP passwords across + // a replication channel whose only historical payload was exactly that. + var sitePath = Path_("scadabridge.db"); + SeedLegacySiteStorage(sitePath); + var config = Config(siteDbPath: sitePath); + var db = CreateConsolidated(config); + + SiteLocalDbLegacyMigrator.Migrate(db, config); + + Assert.Equal(0, CountRows(db, "notification_lists")); + Assert.Equal(0, CountRows(db, "smtp_configurations")); + + // Belt and braces: the secret must not be anywhere in the consolidated file, + // including the oplog, whose row_json is a json_object copy of every captured row. + using var connection = db.CreateConnection(); + using var cmd = connection.CreateCommand(); + cmd.CommandText = "SELECT COUNT(*) FROM __localdb_oplog WHERE row_json LIKE '%PLAINTEXT-SECRET%'"; + Assert.Equal(0L, (long)cmd.ExecuteScalar()!); + } + + [Fact] + public void MigratedConfigRows_EnterTheOplog_SoTheyActuallyReplicate() + { + var sitePath = Path_("scadabridge.db"); + SeedLegacySiteStorage(sitePath); + var config = Config(siteDbPath: sitePath); + var db = CreateConsolidated(config); + + SiteLocalDbLegacyMigrator.Migrate(db, config); + + using var connection = db.CreateConnection(); + using var cmd = connection.CreateCommand(); + cmd.CommandText = + "SELECT COUNT(*) FROM __localdb_oplog WHERE table_name = 'deployed_configurations'"; + Assert.Equal(1L, (long)cmd.ExecuteScalar()!); + } + + [Fact] + public void SiteConfigMigration_IsIdempotent_WhenRerunAfterACrashBeforeRename() + { + var sitePath = Path_("scadabridge.db"); + SeedLegacySiteStorage(sitePath); + var config = Config(siteDbPath: sitePath); + var db = CreateConsolidated(config); + + SiteLocalDbLegacyMigrator.Migrate(db, config); + File.Move(sitePath + ".migrated", sitePath); + SiteLocalDbLegacyMigrator.Migrate(db, config); + + foreach (var table in SiteLocalDbLegacyMigrator.SiteStorageTables) + Assert.Equal(1, CountRows(db, table.Table)); + } + + [Fact] + public void MigratorColumnLists_MatchTheLiveSchema() + { + // A column named here but absent from the schema fails at runtime on a real + // deployment and NOWHERE else — the intersection logic hides a typo (the column is + // simply dropped from the copy) and every other test still passes. A column in the + // schema but missing here is worse: that data is silently left behind. + // + // So this asserts set equality against the schema the host actually applies. + using var connection = new SqliteConnection($"Data Source={Path_("schema-probe.db")}"); + connection.Open(); + ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence.SiteStorageSchema.Apply(connection); + + foreach (var table in SiteLocalDbLegacyMigrator.SiteStorageTables) + { + using var probe = connection.CreateCommand(); + probe.CommandText = $"SELECT name FROM pragma_table_info('{table.Table}')"; + using var reader = probe.ExecuteReader(); + + var actual = new List(); + while (reader.Read()) actual.Add(reader.GetString(0)); + + Assert.Equal( + actual.OrderBy(c => c, StringComparer.Ordinal).ToArray(), + table.Columns.OrderBy(c => c, StringComparer.Ordinal).ToArray()); + } + } + [Fact] public void LegacyFileWithoutTheExpectedTable_IsTreatedAsEmpty_NotAFailure() { From 0dbfefba627490056b4a2183eccaa5549a64470c Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Mon, 20 Jul 2026 03:40:59 -0400 Subject: [PATCH 39/55] docs(localdb): task state for tasks 7-9 + deviations Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts --- docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json b/docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json index f775ad28..19d4b185 100644 --- a/docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json +++ b/docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json @@ -75,9 +75,9 @@ {"id": 4, "subject": "Task 4: Extract SiteStorageSchema.Apply", "status": "completed", "classification": "small", "blockedBy": [2], "note": "DEVIATION: TryAddColumnAsync's catch-on-message-text ('duplicate column') became a PRAGMA table_info probe matching OperationTrackingSchema - the message form depends on a non-contractual error string and swallowed unrelated SqliteExceptions; also dropped the ILogger dep, which is what let the class be static. Cost: the per-column 'Migrated: added column' info log is gone (nothing consumes it). PRAGMA journal_mode=WAL stays in InitializeAsync per plan. Added a legacy-upgrade test for the same reason as Task 3. SiteRuntime 532/532, full solution build 0 warnings."}, {"id": 5, "subject": "Task 5: Rewire StoreAndForwardStorage onto ILocalDb", "status": "completed", "classification": "high-risk", "blockedBy": [3], "note": "Test fallout was ~7x the plan estimate: 40 files across 7 test projects, not \"fixtures\" in one. Most used Mode=Memory;Cache=Shared and LocalDb has NO in-memory mode, so all moved to real temp files. Added shared tests/ZB.MOM.WW.ScadaBridge.TestSupport lib (TestLocalDb) instead of copying the Phase 1 fixture into 7 projects. WAL test retargeted to the LocalDb-backed store; directory-creation test moved to Host.Tests (different owner - see Task 6 note)."}, {"id": 6, "subject": "Task 6: Rewire SiteStorageService onto ILocalDb", "status": "completed", "classification": "high-risk", "blockedBy": [4], "note": "LATENT PHASE-1 DEFECT FOUND+FIXED: LocalDb does NOT create the parent directory and SqliteLocalDb opens the file eagerly, so a missing dir is a HARD BOOT FAILURE (SQLite Error 14). Default site config is the RELATIVE ./data/site-localdb.db; docker escapes only because the volume mount creates /app/data. The plan wrongly assumed dir-creation moved to LocalDb with file ownership. Fixed via SiteLocalDbDirectory.Ensure(config) before AddZbLocalDb + Host.Tests/SiteLocalDbDirectoryTests (non-vacuity observed: 2 tests failed with exactly Error 14 pre-fix). ALSO a contract change: SiteStorageService.CreateConnection() used to return an UNOPENED connection; it now returns an ALREADY-OPEN one - SiteExternalSystemRepository dropped 5 OpenAsync calls. AddSiteRuntime(string) overload deleted."}, - {"id": 7, "subject": "Task 7: Extend SiteLocalDbSetup with the new DDL (not yet registered)", "status": "pending", "classification": "standard", "blockedBy": [5, 6]}, - {"id": 8, "subject": "Task 8: Extend the legacy migrator for sf_messages", "status": "pending", "classification": "high-risk", "blockedBy": [7], "note": "Oplog pin test registers sf_messages on its own TestLocalDb - production OnReady doesn't register it until Task 14."}, - {"id": 9, "subject": "Task 9: Extend the legacy migrator for the config tables (skip notification/smtp)", "status": "pending", "classification": "high-risk", "blockedBy": [7]}, + {"id": 7, "subject": "Task 7: Extend SiteLocalDbSetup with the new DDL (not yet registered)", "status": "completed", "classification": "standard", "blockedBy": [5, 6], "commit": "f8aa02e2", "note": "As planned. Two pins in SiteLocalDbWiringTests through the REAL composition root: all 12 tables exist, and ReplicatedTables is EXACTLY the Phase 1 pair (an equality check, not Contains - the 'not yet' is the assertion that matters). Task 14 should INVERT that second test, not delete it. Non-vacuity verified by removing the DDL."}, + {"id": 8, "subject": "Task 8: Extend the legacy migrator for sf_messages", "status": "completed", "classification": "high-risk", "blockedBy": [7], "commit": "bdc0dffe", "note": "DEVIATION: the copy INTERSECTS the legacy column set with the current one instead of naming all 16 columns. A legacy file from an older build lacks execution_id/parent_execution_id/last_attempt_at_ms, and naming a missing column throws 'no such column' - which the existing ReadAll treats as an unrecognised shape and SILENTLY DISCARDS EVERY ROW. For undelivered messages that is real data loss. A required-column (PK) guard stops the tolerance degrading into NULL-keyed copies. Tests register sf_messages on their own harness (production OnReady does not until Task 14), so Task 14 MUST keep Migrate as the LAST call in OnReady, after all registrations. Non-vacuity verified: all 4 fail without the Migrate call."}, + {"id": 9, "subject": "Task 9: Extend the legacy migrator for the config tables (skip notification/smtp)", "status": "completed", "classification": "high-risk", "blockedBy": [7], "commit": "5ddc7eed", "note": "7 of 9 tables migrated; notification_lists + smtp_configurations skipped as planned (plaintext passwords must not enter a table Task 14 makes replicated) - the skip is pinned by a test that also greps __localdb_oplog.row_json for the secret. Task 8 MigrateTable generalized to MigrateFile(many tables, one transaction, one rename). ADDED beyond the plan: MigratorColumnLists_MatchTheLiveSchema, a column-parity test vs SiteStorageSchema - the plan called for a manual column check, but a mismatch is invisible at runtime in BOTH directions (a typo is silently dropped by the intersection; a missing column silently leaves data behind). SiteStorageTables + LegacyTable are internal so the test can read them. Non-vacuity verified twice: removing the call (3 fail) and wrongly adding notification/smtp to the map (skip test fails)."}, {"id": 10, "subject": "Task 10: Port the S&F replication test intents as CDC specs", "status": "pending", "classification": "standard", "blockedBy": [8]}, {"id": 11, "subject": "Task 11: Port the resync + directional-authority tests", "status": "pending", "classification": "standard", "blockedBy": [9], "note": "Zero-fetch assertion scoped to deploy convergence - SiteReconciliationActor's startup fetch is legitimate and survives."}, {"id": 12, "subject": "Task 12: Pin the active-node SMTP purge (corrected: already exists at DeploymentManagerActor.cs:1921)", "status": "pending", "classification": "standard", "blockedBy": [10, 11], "note": "Pin test only - no production change. Guards Task 16's actor edits from silently dropping the purge call."}, From 2bbe66311dc456eba5347b43f9a7e0690a9cba31 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Mon, 20 Jul 2026 03:50:48 -0400 Subject: [PATCH 40/55] test(localdb): port store-and-forward replication intents as CDC convergence specs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Task 10. Specifications first, deletion second: the bespoke ReplicationService (explicit Add/Remove/Park/Requeue over Akka) dies at Task 14, so its behaviour is restated here as outcomes the CDC replacement must still deliver. Written in terms of ROWS, not operations — under CDC there is no Add or Park message to observe, only a row that must end up right on both nodes. Not ported: ReplicationOperations_AreDispatchedInIssueOrder. It asserts the mechanism (inline fire-and-forget dispatch), and CDC capture is asynchronous and batched by construction. Its portable content is the ordering OUTCOME — add-then-remove must never converge to present — which is a test here, with that reasoning recorded in the file so it does not read as an accidental drop. DEVIATION: extracted the Phase 1 fixture into LocalDbSitePairHarness rather than duplicating ~150 lines. Phase 1's tests now derive from it and still pass unchanged. The harness registers the Phase 2 tables itself, since production OnReady does not until Task 14; that method is marked for deletion at the cutover, and the 8-table list is written literally so a cutover registering the wrong set fails these tests instead of agreeing with itself. Non-vacuity verified by unregistering sf_messages: 6 of 7 failed. The 7th — the ordering test — PASSED, because an absent row is also what a pair that replicates nothing looks like. Fixed with a control row that must converge in the same window, so the absence is evidence rather than silence. Also corrected two comments from Task 9 that claimed Task 14 makes notification_lists/smtp_configurations replicated. It explicitly does not register them, for the same reason the migrator skips them. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts --- .../SiteLocalDbLegacyMigrator.cs | 14 +- .../SiteLocalDbLegacyMigratorTests.cs | 7 +- .../LocalDbSitePairConvergenceTests.cs | 210 +------------ .../LocalDbSitePairHarness.cs | 286 ++++++++++++++++++ .../LocalDbStoreAndForwardConvergenceTests.cs | 221 ++++++++++++++ 5 files changed, 528 insertions(+), 210 deletions(-) create mode 100644 tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbSitePairHarness.cs create mode 100644 tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbStoreAndForwardConvergenceTests.cs diff --git a/src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbLegacyMigrator.cs b/src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbLegacyMigrator.cs index 42bf6e42..29f0eddd 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbLegacyMigrator.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbLegacyMigrator.cs @@ -83,10 +83,16 @@ public static class SiteLocalDbLegacyMigrator /// /// notification_lists and smtp_configurations are deliberately absent. /// Both are purged on every deploy and are permanently empty by design — the site-side - /// write paths were removed on 2026-07-10. A pre-fix legacy file can still hold rows, - /// and smtp_configurations.password is plaintext, so migrating them would - /// resurrect plaintext SMTP passwords into a table that Task 14 makes REPLICATED. The - /// tables themselves are still created (see SiteStorageSchema); only their + /// write paths were removed on 2026-07-10. A pre-fix legacy file can still hold rows, and + /// smtp_configurations.password is plaintext. + /// + /// Skipping them here is one half of a pair: the cutover also declines to register them + /// for replication, for the same reason. Migrating them would leave plaintext SMTP + /// passwords sitting in the consolidated database — one future RegisterReplicated + /// away from being shipped to a peer — in exchange for resurrecting config that nothing + /// reads. Keeping the tables permanently empty is what makes both decisions safe. + /// + /// The tables themselves are still created (see SiteStorageSchema); only their /// historical contents are left behind. /// internal static readonly LegacyTable[] SiteStorageTables = diff --git a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbLegacyMigratorTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbLegacyMigratorTests.cs index 834aa36a..329b1011 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbLegacyMigratorTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbLegacyMigratorTests.cs @@ -465,9 +465,10 @@ public class SiteLocalDbLegacyMigratorTests : IDisposable { // smtp_configurations.password is PLAINTEXT. Both tables are purged on every deploy // and permanently empty by design since the site write paths were removed - // (2026-07-10), but a pre-fix legacy file can still hold rows — and Task 14 makes - // these tables REPLICATED. Migrating them would push plaintext SMTP passwords across - // a replication channel whose only historical payload was exactly that. + // (2026-07-10), but a pre-fix legacy file can still hold rows. The cutover also + // declines to REGISTER these two for replication, for the same reason — so migrating + // them would leave plaintext SMTP passwords in the consolidated database, one future + // RegisterReplicated away from being shipped to a peer, for config nothing reads. var sitePath = Path_("scadabridge.db"); SeedLegacySiteStorage(sitePath); var config = Config(siteDbPath: sitePath); diff --git a/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbSitePairConvergenceTests.cs b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbSitePairConvergenceTests.cs index b9c13ced..a8ee9ad1 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbSitePairConvergenceTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbSitePairConvergenceTests.cs @@ -1,202 +1,23 @@ -using System.Net; -using Microsoft.AspNetCore.Builder; -using Microsoft.AspNetCore.Hosting; -using Microsoft.AspNetCore.Hosting.Server; -using Microsoft.AspNetCore.Hosting.Server.Features; -using Microsoft.AspNetCore.Server.Kestrel.Core; -using Microsoft.Extensions.Configuration; -using Microsoft.Extensions.DependencyInjection; -using Microsoft.Extensions.Hosting; using ZB.MOM.WW.LocalDb; -using ZB.MOM.WW.LocalDb.Replication; -using ZB.MOM.WW.ScadaBridge.Host; namespace ZB.MOM.WW.ScadaBridge.IntegrationTests; /// -/// Serializes the site-pair convergence tests against each other: each one stands up a real -/// Kestrel listener plus two SQLite files, and running them concurrently under CI -/// contention is a flakiness risk. -/// -[CollectionDefinition("LocalDbSitePairConvergence")] -public sealed class LocalDbSitePairConvergenceCollection; - -/// -/// Two ScadaBridge site nodes replicating the consolidated site database over a REAL -/// loopback gRPC transport, through the REAL fail-closed auth interceptor. +/// Phase 1 convergence: operation tracking and site events across a real site pair. /// /// -/// /// This is the test that answers the question Phase 1 exists to answer: does a site node /// pair actually stop losing operation-tracking and site-event state? Everything upstream /// of it — schema helpers, DI wiring, the interceptor — can be individually green while the /// pair still fails to converge. -/// /// -/// It uses , not a hand-written schema, so the tables, -/// their primary keys, and the registration ORDER under test are the ones the host actually -/// runs. A separate schema here would prove only that the test agrees with itself. -/// -/// -/// Offline: no docker, no external services. Loopback Kestrel with h2c. +/// The fixture lives in , shared with the Phase 2 +/// convergence suites. /// /// [Collection("LocalDbSitePairConvergence")] -public sealed class LocalDbSitePairConvergenceTests : IAsyncLifetime +public sealed class LocalDbSitePairConvergenceTests : LocalDbSitePairHarness { - private const string SharedApiKey = "site-pair-convergence-key"; - private static readonly TimeSpan ConvergeTimeout = TimeSpan.FromSeconds(30); - - private readonly string _pathA = Path.Combine(Path.GetTempPath(), $"sitepairA-{Guid.NewGuid():N}.db"); - private readonly string _pathB = Path.Combine(Path.GetTempPath(), $"sitepairB-{Guid.NewGuid():N}.db"); - - // The databases are owned by the fixture, in their own providers, and registered into the - // hosts as pre-constructed instances. MS.DI does not dispose instances it did not create, - // so tearing a host down (the offline-peer scenario) leaves the databases intact and - // writable — which is exactly what lets node A accumulate writes while B is down. - private ServiceProvider _dbProviderA = null!; - private ServiceProvider _dbProviderB = null!; - - private IHost? _serverHost; // node B — passive - private IHost? _initiatorHost; // node A — dials the peer - - static LocalDbSitePairConvergenceTests() => - // Grpc.Net.Client dials the loopback server over HTTP/2 cleartext. - AppContext.SetSwitch("System.Net.Http.SocketsHttpHandler.Http2UnencryptedSupport", true); - - private ILocalDb A => _dbProviderA.GetRequiredService(); - private ILocalDb B => _dbProviderB.GetRequiredService(); - - public async Task InitializeAsync() - { - _dbProviderA = BuildDatabaseProvider(_pathA, "node-a"); - _dbProviderB = BuildDatabaseProvider(_pathB, "node-b"); - - // Force construction (and therefore OnReady) before anything replicates. - _ = A; - _ = B; - - await StartPassiveAsync(); - await StartInitiatorAsync(); - } - - public async Task DisposeAsync() - { - await StopHostAsync(_initiatorHost); - await StopHostAsync(_serverHost); - await _dbProviderA.DisposeAsync(); - await _dbProviderB.DisposeAsync(); - - Microsoft.Data.Sqlite.SqliteConnection.ClearAllPools(); - foreach (var path in new[] { _pathA, _pathB }) - { - foreach (var suffix in new[] { "", "-wal", "-shm" }) - { - try { File.Delete(path + suffix); } catch { /* best effort */ } - } - } - } - - // ---- fixture internals ------------------------------------------------------------ - - /// - /// A provider owning one consolidated site database, initialized through the host's own - /// — same schema, same registration order. - /// - private static ServiceProvider BuildDatabaseProvider(string path, string nodeName) - { - var config = new ConfigurationBuilder() - .AddInMemoryCollection(new Dictionary - { - ["LocalDb:Path"] = path, - ["ScadaBridge:Node:NodeName"] = nodeName, - // Point the legacy migrator at paths that do not exist, so it no-ops rather - // than picking up stray files from the test working directory. - ["ScadaBridge:OperationTracking:ConnectionString"] = - $"Data Source={Path.Combine(Path.GetTempPath(), $"absent-{Guid.NewGuid():N}.db")}", - ["ScadaBridge:SiteEventLog:DatabasePath"] = - Path.Combine(Path.GetTempPath(), $"absent-{Guid.NewGuid():N}.db"), - }) - .Build(); - - return new ServiceCollection() - .AddZbLocalDb(config, db => SiteLocalDbSetup.OnReady(db, config)) - .BuildServiceProvider(); - } - - private static IConfiguration ReplicationConfig(string? peerAddress) - { - var values = new Dictionary - { - // Tight flush + bounded reconnect backoff so convergence is observable well - // inside the poll deadline. The 60 s production default would let the doubling - // backoff overrun it after a peer outage. - ["LocalDb:Replication:FlushInterval"] = "00:00:00.050", - ["LocalDb:Replication:ReconnectBackoffMax"] = "00:00:02", - // Both nodes share one key — the interceptor is fail-closed, so a mismatch here - // turns every scenario below red (verified by deliberately breaking it). - ["LocalDb:Replication:ApiKey"] = SharedApiKey, - }; - - if (peerAddress is not null) - values["LocalDb:Replication:PeerAddress"] = peerAddress; - - return new ConfigurationBuilder().AddInMemoryCollection(values).Build(); - } - - private async Task StartPassiveAsync() - { - var config = ReplicationConfig(peerAddress: null); - - _serverHost = await new HostBuilder() - .ConfigureWebHost(web => - { - web.UseKestrel(o => - o.Listen(IPAddress.Loopback, 0, listen => listen.Protocols = HttpProtocols.Http2)); - web.ConfigureServices(services => - { - services.AddLogging(); - services.AddRouting(); - // The REAL interceptor, not a stand-in. If it rejected legitimate peer - // traffic, every scenario below would fail — which is the point. - services.AddGrpc(o => o.Interceptors.Add()); - services.AddSingleton(B); - services.AddZbLocalDbReplication(config); - }); - web.Configure(app => - { - app.UseRouting(); - app.UseEndpoints(e => e.MapZbLocalDbSync()); - }); - }) - .StartAsync(); - } - - private async Task StartInitiatorAsync() - { - var config = ReplicationConfig(PassiveAddress()); - - _initiatorHost = await new HostBuilder() - .ConfigureServices(services => - { - services.AddLogging(); - services.AddSingleton(A); - services.AddZbLocalDbReplication(config); - }) - .StartAsync(); - } - - private string PassiveAddress() - => _serverHost!.Services.GetRequiredService() - .Features.Get()!.Addresses.Single(); - - private static async Task StopHostAsync(IHost? host) - { - if (host is null) return; - try { await host.StopAsync(TimeSpan.FromSeconds(5)); } catch { /* teardown */ } - host.Dispose(); - } - // ---- data helpers ----------------------------------------------------------------- private static Task WriteTrackingAsync(ILocalDb db, string id, string status, string target) @@ -232,19 +53,6 @@ public sealed class LocalDbSitePairConvergenceTests : IAsyncLifetime private static Task> ReadEventIdsAsync(ILocalDb db) => db.QueryAsync("SELECT id FROM site_events ORDER BY id", static r => r.GetString(0)); - /// Polls until true or the deadline passes. - private static async Task WaitUntilAsync(Func> condition, string because) - { - var deadline = DateTime.UtcNow + ConvergeTimeout; - while (DateTime.UtcNow < deadline) - { - if (await condition()) return; - await Task.Delay(50); - } - - Assert.Fail($"Timed out after {ConvergeTimeout.TotalSeconds:0}s waiting for: {because}"); - } - // ---- scenarios -------------------------------------------------------------------- [Fact] @@ -322,19 +130,15 @@ public sealed class LocalDbSitePairConvergenceTests : IAsyncLifetime { // The failover case that motivates Phase 1: one node is down while the other keeps // working, and nothing written during the outage may be lost. - await StopHostAsync(_serverHost); - _serverHost = null; + await StopPassiveAsync(); var idsDuringOutage = Enumerable.Range(0, 5).Select(_ => Guid.NewGuid().ToString("N")).ToList(); foreach (var id in idsDuringOutage) await WriteEventAsync(A, id, "written while B was down"); await WriteTrackingAsync(A, "op-during-outage", "Delivered", "ERP.GetOrder"); // Node B's database survived the host teardown (pre-constructed instance), so this is - // a genuine rejoin rather than a fresh node. It comes back on a NEW loopback port; - // the initiator's channel factory re-reads the peer address on each reconnect. - await StartPassiveAsync(); - await StopHostAsync(_initiatorHost); - await StartInitiatorAsync(); + // a genuine rejoin rather than a fresh node. + await RestartPairAsync(); await WaitUntilAsync( async () => diff --git a/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbSitePairHarness.cs b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbSitePairHarness.cs new file mode 100644 index 00000000..dc37eefb --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbSitePairHarness.cs @@ -0,0 +1,286 @@ +using System.Net; +using Microsoft.AspNetCore.Builder; +using Microsoft.AspNetCore.Hosting; +using Microsoft.AspNetCore.Hosting.Server; +using Microsoft.AspNetCore.Hosting.Server.Features; +using Microsoft.AspNetCore.Server.Kestrel.Core; +using Microsoft.Extensions.Configuration; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; +using ZB.MOM.WW.LocalDb; +using ZB.MOM.WW.LocalDb.Replication; +using ZB.MOM.WW.ScadaBridge.Host; + +namespace ZB.MOM.WW.ScadaBridge.IntegrationTests; + +/// +/// Serializes the site-pair convergence tests against each other: each one stands up a real +/// Kestrel listener plus two SQLite files, and running them concurrently under CI +/// contention is a flakiness risk. +/// +[CollectionDefinition("LocalDbSitePairConvergence")] +public sealed class LocalDbSitePairConvergenceCollection; + +/// +/// Two ScadaBridge site nodes replicating the consolidated site database over a REAL +/// loopback gRPC transport, through the REAL fail-closed auth interceptor. +/// +/// +/// +/// Extracted from the Phase 1 convergence tests once Phase 2 needed the same pair for the +/// store-and-forward buffer and the configuration tables. Everything here is fixture; the +/// derived classes hold only their own data helpers and scenarios. +/// +/// +/// It uses , not a hand-written schema, so the tables, +/// their primary keys, and the registration ORDER under test are the ones the host actually +/// runs. A separate schema here would prove only that the test agrees with itself. +/// +/// +/// Offline: no docker, no external services. Loopback Kestrel with h2c. +/// +/// +public abstract class LocalDbSitePairHarness : IAsyncLifetime +{ + private const string SharedApiKey = "site-pair-convergence-key"; + + /// How long a scenario waits for the pair to agree before failing. + protected static readonly TimeSpan ConvergeTimeout = TimeSpan.FromSeconds(30); + + private readonly string _pathA = Path.Combine(Path.GetTempPath(), $"sitepairA-{Guid.NewGuid():N}.db"); + private readonly string _pathB = Path.Combine(Path.GetTempPath(), $"sitepairB-{Guid.NewGuid():N}.db"); + + // The databases are owned by the fixture, in their own providers, and registered into the + // hosts as pre-constructed instances. MS.DI does not dispose instances it did not create, + // so tearing a host down (the offline-peer scenario) leaves the databases intact and + // writable — which is exactly what lets node A accumulate writes while B is down. + private ServiceProvider _dbProviderA = null!; + private ServiceProvider _dbProviderB = null!; + + private IHost? _serverHost; // node B — passive + private IHost? _initiatorHost; // node A — dials the peer + + static LocalDbSitePairHarness() => + // Grpc.Net.Client dials the loopback server over HTTP/2 cleartext. + AppContext.SetSwitch("System.Net.Http.SocketsHttpHandler.Http2UnencryptedSupport", true); + + /// Node A — the initiator, which dials the peer. + protected ILocalDb A => _dbProviderA.GetRequiredService(); + + /// Node B — the passive node, which listens. + protected ILocalDb B => _dbProviderB.GetRequiredService(); + + public async Task InitializeAsync() + { + _dbProviderA = BuildDatabaseProvider(_pathA, "node-a"); + _dbProviderB = BuildDatabaseProvider(_pathB, "node-b"); + + // Force construction (and therefore OnReady) before anything replicates. + _ = A; + _ = B; + + await StartPassiveAsync(); + await StartInitiatorAsync(); + } + + public async Task DisposeAsync() + { + await StopHostAsync(_initiatorHost); + await StopHostAsync(_serverHost); + await _dbProviderA.DisposeAsync(); + await _dbProviderB.DisposeAsync(); + + Microsoft.Data.Sqlite.SqliteConnection.ClearAllPools(); + foreach (var path in new[] { _pathA, _pathB }) + { + foreach (var suffix in new[] { "", "-wal", "-shm" }) + { + try { File.Delete(path + suffix); } catch { /* best effort */ } + } + } + } + + // ---- fixture internals ------------------------------------------------------------ + + /// + /// A provider owning one consolidated site database, initialized through the host's own + /// — same schema, same registration order. + /// + private static ServiceProvider BuildDatabaseProvider(string path, string nodeName) + { + var config = new ConfigurationBuilder() + .AddInMemoryCollection(new Dictionary + { + ["LocalDb:Path"] = path, + ["ScadaBridge:Node:NodeName"] = nodeName, + // Point the legacy migrators at paths that do not exist, so they no-op rather + // than picking up stray files from the test working directory. The two Phase 2 + // defaults matter most: unlike the Phase 1 pair they resolve inside ./data/, + // and a migration would also RENAME whatever it found. + ["ScadaBridge:OperationTracking:ConnectionString"] = + $"Data Source={Path.Combine(Path.GetTempPath(), $"absent-{Guid.NewGuid():N}.db")}", + ["ScadaBridge:SiteEventLog:DatabasePath"] = + Path.Combine(Path.GetTempPath(), $"absent-{Guid.NewGuid():N}.db"), + ["ScadaBridge:StoreAndForward:SqliteDbPath"] = + Path.Combine(Path.GetTempPath(), $"absent-{Guid.NewGuid():N}.db"), + ["ScadaBridge:Database:SiteDbPath"] = + Path.Combine(Path.GetTempPath(), $"absent-{Guid.NewGuid():N}.db"), + }) + .Build(); + + return new ServiceCollection() + .AddZbLocalDb(config, db => + { + SiteLocalDbSetup.OnReady(db, config); + RegisterPhase2TablesUntilCutover(db); + }) + .BuildServiceProvider(); + } + + /// + /// Registers the Phase 2 tables for capture, which production OnReady does not do + /// until the Task 14 cutover. + /// + /// + /// Delete this method at Task 14, along with its call above. Until the cutover the + /// bespoke SiteReplicationActor and ReplicationService still own these + /// tables, so OnReady deliberately leaves them unregistered — but the convergence + /// specifications the cutover has to satisfy need capture triggers to mean anything. The + /// tables are empty at this point, so registering here captures nothing retroactively; + /// the ordering guarantee under test is unaffected. + /// + /// After Task 14 these registrations come from OnReady itself, and leaving this + /// method behind would mask a cutover that forgot one — the whole point of these tests. + /// + /// + private static void RegisterPhase2TablesUntilCutover(ILocalDb db) + { + foreach (var table in Phase2ReplicatedTables) + db.RegisterReplicated(table); + } + + /// + /// The eight tables Task 14 registers. Listed literally rather than derived from + /// production code, so that a cutover which registers the wrong set fails these tests + /// instead of agreeing with itself. + /// + /// + /// notification_lists and smtp_configurations are absent by design. They + /// are permanently empty (no site writer since 2026-07-10, the migrator skips them, the + /// active-node purge keeps them empty), so registering them would open a standing + /// replication channel whose only historical payload was plaintext SMTP passwords. + /// + protected static readonly string[] Phase2ReplicatedTables = + [ + "sf_messages", + "deployed_configurations", "static_attribute_overrides", "shared_scripts", + "external_systems", "database_connections", "data_connection_definitions", + "native_alarm_state", + ]; + + private static IConfiguration ReplicationConfig(string? peerAddress) + { + var values = new Dictionary + { + // Tight flush + bounded reconnect backoff so convergence is observable well + // inside the poll deadline. The 60 s production default would let the doubling + // backoff overrun it after a peer outage. + ["LocalDb:Replication:FlushInterval"] = "00:00:00.050", + ["LocalDb:Replication:ReconnectBackoffMax"] = "00:00:02", + // Both nodes share one key — the interceptor is fail-closed, so a mismatch here + // turns every scenario below red (verified by deliberately breaking it). + ["LocalDb:Replication:ApiKey"] = SharedApiKey, + }; + + if (peerAddress is not null) + values["LocalDb:Replication:PeerAddress"] = peerAddress; + + return new ConfigurationBuilder().AddInMemoryCollection(values).Build(); + } + + /// Starts node B, the passive listener. + protected async Task StartPassiveAsync() + { + var config = ReplicationConfig(peerAddress: null); + + _serverHost = await new HostBuilder() + .ConfigureWebHost(web => + { + web.UseKestrel(o => + o.Listen(IPAddress.Loopback, 0, listen => listen.Protocols = HttpProtocols.Http2)); + web.ConfigureServices(services => + { + services.AddLogging(); + services.AddRouting(); + // The REAL interceptor, not a stand-in. If it rejected legitimate peer + // traffic, every scenario below would fail — which is the point. + services.AddGrpc(o => o.Interceptors.Add()); + services.AddSingleton(B); + services.AddZbLocalDbReplication(config); + }); + web.Configure(app => + { + app.UseRouting(); + app.UseEndpoints(e => e.MapZbLocalDbSync()); + }); + }) + .StartAsync(); + } + + /// Starts node A, which dials the passive node. + protected async Task StartInitiatorAsync() + { + var config = ReplicationConfig(PassiveAddress()); + + _initiatorHost = await new HostBuilder() + .ConfigureServices(services => + { + services.AddLogging(); + services.AddSingleton(A); + services.AddZbLocalDbReplication(config); + }) + .StartAsync(); + } + + /// Takes node B's listener down, leaving its database intact and writable. + protected async Task StopPassiveAsync() + { + await StopHostAsync(_serverHost); + _serverHost = null; + } + + /// + /// Brings node B back on a NEW loopback port and re-dials from A. The initiator's channel + /// factory re-reads the peer address on each reconnect, so this is a genuine rejoin. + /// + protected async Task RestartPairAsync() + { + await StartPassiveAsync(); + await StopHostAsync(_initiatorHost); + await StartInitiatorAsync(); + } + + private string PassiveAddress() + => _serverHost!.Services.GetRequiredService() + .Features.Get()!.Addresses.Single(); + + private static async Task StopHostAsync(IHost? host) + { + if (host is null) return; + try { await host.StopAsync(TimeSpan.FromSeconds(5)); } catch { /* teardown */ } + host.Dispose(); + } + + /// Polls until true or the deadline passes. + protected static async Task WaitUntilAsync(Func> condition, string because) + { + var deadline = DateTime.UtcNow + ConvergeTimeout; + while (DateTime.UtcNow < deadline) + { + if (await condition()) return; + await Task.Delay(50); + } + + Assert.Fail($"Timed out after {ConvergeTimeout.TotalSeconds:0}s waiting for: {because}"); + } +} diff --git a/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbStoreAndForwardConvergenceTests.cs b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbStoreAndForwardConvergenceTests.cs new file mode 100644 index 00000000..409f23d8 --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbStoreAndForwardConvergenceTests.cs @@ -0,0 +1,221 @@ +using ZB.MOM.WW.LocalDb; +using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; + +namespace ZB.MOM.WW.ScadaBridge.IntegrationTests; + +/// +/// Phase 2 convergence: the store-and-forward buffer across a real site pair. +/// +/// +/// +/// These are specifications, written before the deletion they justify. The bespoke +/// ReplicationService (an explicit Add/Remove/Park/Requeue operation stream over Akka) +/// is deleted at Task 14 and replaced by LocalDb's trigger-based change capture. Each test +/// here is one behaviour the old mechanism provided, restated as an outcome the replacement +/// must still deliver — so the cutover has something to be judged against other than "it +/// compiles". +/// +/// +/// They are written in terms of rows, not operations, which is the whole point: +/// under CDC there is no Add or Park message to observe, only a row that must end up in the +/// right state on both nodes. +/// +/// +/// One intent is deliberately not ported: +/// ReplicationServiceTests.ReplicationOperations_AreDispatchedInIssueOrder — 200 +/// interleaved operations dispatched synchronously, observed in strict issue order. That +/// asserts the mechanism (inline fire-and-forget dispatch), not an outcome, and CDC +/// capture is asynchronous and batched by construction, so no honest port exists. Its +/// portable content is the ordering outcome: an add followed by a remove must never +/// converge to present. That is +/// . This paragraph exists so a +/// future reader does not conclude the test was dropped by accident. +/// +/// +[Collection("LocalDbSitePairConvergence")] +public sealed class LocalDbStoreAndForwardConvergenceTests : LocalDbSitePairHarness +{ + // ---- data helpers ----------------------------------------------------------------- + + /// Buffers a message, as StoreAndForwardStorage.EnqueueAsync does. + private static Task EnqueueAsync(ILocalDb db, string id, string target = "ERP.GetOrder") + => db.ExecuteAsync( + """ + INSERT INTO sf_messages ( + id, category, target, payload_json, retry_count, max_retries, + retry_interval_ms, created_at, status) + VALUES (@id, 0, @target, '{"order":1}', 0, 50, 30000, @now, @pending) + ON CONFLICT(id) DO UPDATE SET + status = excluded.status, + retry_count = excluded.retry_count; + """, + new + { + id, + target, + now = DateTime.UtcNow.ToString("o"), + pending = (int)StoreAndForwardMessageStatus.Pending, + }); + + /// Deletes a delivered message, as the successful-retry path does. + private static Task DeleteAsync(ILocalDb db, string id) + => db.ExecuteAsync("DELETE FROM sf_messages WHERE id = @id;", new { id }); + + private static Task SetStatusAsync(ILocalDb db, string id, StoreAndForwardMessageStatus status, int retryCount) + => db.ExecuteAsync( + "UPDATE sf_messages SET status = @status, retry_count = @retryCount WHERE id = @id;", + new { id, status = (int)status, retryCount }); + + private static async Task<(int Status, int RetryCount)?> ReadAsync(ILocalDb db, string id) + { + var rows = await db.QueryAsync( + "SELECT status, retry_count FROM sf_messages WHERE id = @id", + static r => (r.GetInt32(0), r.GetInt32(1)), new { id }); + return rows.Count == 0 ? null : rows[0]; + } + + private static async Task ExistsAsync(ILocalDb db, string id) + => await ReadAsync(db, id) is not null; + + private static async Task HasStatusAsync( + ILocalDb db, string id, StoreAndForwardMessageStatus status) + => await ReadAsync(db, id) is { } row && row.Status == (int)status; + + // ---- scenarios -------------------------------------------------------------------- + + [Fact] + public async Task BufferedMessage_MaterialisesOnThePeer() + { + // Was: BufferingAMessage_ReplicatesAnAddOperation. + await EnqueueAsync(A, "msg-add"); + + await WaitUntilAsync( + () => ExistsAsync(B, "msg-add"), + "a message buffered on node A to appear on node B"); + } + + [Fact] + public async Task DeliveredMessage_DisappearsFromThePeer() + { + // Was: SuccessfulRetry_ReplicatesARemoveOperation. Row deletes propagate as + // tombstones under CDC — the peer must not keep re-delivering a message that node A + // already succeeded on. + await EnqueueAsync(A, "msg-remove"); + await WaitUntilAsync(() => ExistsAsync(B, "msg-remove"), "the message to arrive on B first"); + + await DeleteAsync(A, "msg-remove"); + + await WaitUntilAsync( + async () => !await ExistsAsync(B, "msg-remove"), + "the delivered message to be gone from node B"); + } + + [Fact] + public async Task ParkedMessage_ShowsAsParkedOnThePeer() + { + // Was: ParkedMessage_ReplicatesAParkOperation. + await EnqueueAsync(A, "msg-park"); + await WaitUntilAsync(() => ExistsAsync(B, "msg-park"), "the message to arrive on B first"); + + await SetStatusAsync(A, "msg-park", StoreAndForwardMessageStatus.Parked, retryCount: 50); + + await WaitUntilAsync( + () => HasStatusAsync(B, "msg-park", StoreAndForwardMessageStatus.Parked), + "the parked status to reach node B"); + } + + [Fact] + public async Task RequeuedMessage_ResetsStatusAndRetryCountOnThePeer() + { + // Was: RetryingAParkedMessage_ReplicatesARequeueOperation + + // ApplyReplicatedOperation_Requeue_MovesStandbyRowBackToPending. RetryCount matters + // as much as status: a peer that took Pending but kept retry_count at max would park + // the message again on its first attempt after a failover. + await EnqueueAsync(A, "msg-requeue"); + await SetStatusAsync(A, "msg-requeue", StoreAndForwardMessageStatus.Parked, retryCount: 50); + await WaitUntilAsync( + () => HasStatusAsync(B, "msg-requeue", StoreAndForwardMessageStatus.Parked), + "the message to be parked on B before it is requeued"); + + await SetStatusAsync(A, "msg-requeue", StoreAndForwardMessageStatus.Pending, retryCount: 0); + + await WaitUntilAsync( + async () => await ReadAsync(B, "msg-requeue") + is { Status: (int)StoreAndForwardMessageStatus.Pending, RetryCount: 0 }, + "node B to show the requeued message as Pending with retry_count 0"); + } + + [Fact] + public async Task MessageAddedThenRemoved_NeverConvergesToPresent() + { + // The portable intent of ReplicationOperations_AreDispatchedInIssueOrder (see the + // class remarks). Ordering only ever mattered because a remove overtaken by its own + // add would resurrect a delivered message and send it twice. Under LWW that + // reordering is not preventable by dispatch discipline — it is prevented because the + // tombstone carries the later HLC and therefore wins regardless of arrival order. + // + // Asserting the endpoint rather than the sequence is what makes this portable: the + // old test would fail on any async transport even when the outcome was correct. + await EnqueueAsync(A, "msg-ordering"); + await DeleteAsync(A, "msg-ordering"); + + // A control row written in the same window. Without it this test is VACUOUS: an + // absent row is also what a pair that replicates nothing at all looks like, so it + // would pass with capture switched off entirely (observed — it was the only one of + // these seven that survived unregistering sf_messages). The control converging is + // what makes the absence of msg-ordering evidence rather than silence. + await EnqueueAsync(A, "msg-ordering-control"); + + await WaitUntilAsync( + () => ExistsAsync(B, "msg-ordering-control"), + "the control message to reach node B, proving the pair was replicating"); + + await WaitUntilAsync( + async () => !await ExistsAsync(B, "msg-ordering") && !await ExistsAsync(A, "msg-ordering"), + "the add-then-remove pair to settle as absent on both nodes"); + + // Absence has to persist, not just occur once: a late-arriving add would flip the row + // back and the poll above could have observed a gap it never actually converged to. + await Task.Delay(TimeSpan.FromSeconds(2)); + Assert.False(await ExistsAsync(B, "msg-ordering")); + Assert.False(await ExistsAsync(A, "msg-ordering")); + Assert.True(await ExistsAsync(B, "msg-ordering-control")); + } + + [Fact] + public async Task SameMessage_BufferedTwice_ConvergesToTheNewerState() + { + // Was: ApplyReplicatedAdd_Twice_IsIdempotent_NewestWins. Under CDC this is not a + // property the application has to implement — LWW on the primary key gives it — but + // it is still a property the pair must HAVE, so it is asserted rather than assumed. + await EnqueueAsync(A, "msg-twice"); + await WaitUntilAsync(() => ExistsAsync(B, "msg-twice"), "the first write to reach B"); + + await SetStatusAsync(A, "msg-twice", StoreAndForwardMessageStatus.InFlight, retryCount: 3); + + await WaitUntilAsync( + async () => await ReadAsync(B, "msg-twice") + is { Status: (int)StoreAndForwardMessageStatus.InFlight, RetryCount: 3 }, + "the newer state to win on node B"); + } + + [Fact] + public async Task ParkArrivingWithoutItsAdd_StillMaterialisesTheRow() + { + // Was: ApplyReplicatedPark_WhenAddWasLost_MaterializesTheParkedRow. The bespoke + // replicator needed explicit upsert semantics so a lost Add did not leave a Park with + // nothing to update. The CDC equivalent is a peer that was offline for the add and + // only ever sees the row in its parked state — it must still end up with the row, + // not skip it as an update to something it never had. + await StopPassiveAsync(); + + await EnqueueAsync(A, "msg-park-no-add"); + await SetStatusAsync(A, "msg-park-no-add", StoreAndForwardMessageStatus.Parked, retryCount: 50); + + await RestartPairAsync(); + + await WaitUntilAsync( + () => HasStatusAsync(B, "msg-park-no-add", StoreAndForwardMessageStatus.Parked), + "node B to materialise a row it only ever saw in its parked state"); + } +} From c56bf4ae651c91e2c1195b53f0c3f70109bf3dcf Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Mon, 20 Jul 2026 03:54:42 -0400 Subject: [PATCH 41/55] test(localdb): port resync + config replication intents as CDC convergence specs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Task 11. Companion to Task 10, covering the intents held by SiteReplicationActorTests and SfBufferResyncPredicateTests. N1 Critical is re-expressed, not ported literally (D2). SfBufferResyncPredicate existed because ReplaceAllAsync was a destructive DELETE-then-INSERT, so a wrong-direction resync wiped a live buffer and the code needed an oldest-Up predicate to decide authority. LocalDb's snapshot resync merges per row under LWW and never deletes, so this asserts the property the guard protected — no node loses rows to a peer's snapshot — with no directional-authority assertion, because there is no active/standby asymmetry left to enforce. The setup is the wipe scenario made concrete: each node writes rows the other never sees while partitioned, plus a contended row, then they resync. DEVIATION on the zero-fetch assertion. The plan asked for a fetcher test double recording zero invocations; this harness has no actor system, no central and no IDeploymentConfigFetcher in the graph, so the double would record zero calls whether or not the fetch path still existed. An assertion that cannot fail is worse than none, so the test proves the positive half — config reaches B over replication alone — and the file records that the negative half is proved by Task 15 deleting the code and the build still passing. It also records the D1 scope note: SiteReconciliationActor survives and legitimately fetches at startup, so 'the standby never fetches, ever' would be false. Non-vacuity verified by unregistering deployed_configurations: all 4 fail. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts --- .../LocalDbConfigConvergenceTests.cs | 177 ++++++++++++++++++ 1 file changed, 177 insertions(+) create mode 100644 tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbConfigConvergenceTests.cs diff --git a/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbConfigConvergenceTests.cs b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbConfigConvergenceTests.cs new file mode 100644 index 00000000..418afe6b --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbConfigConvergenceTests.cs @@ -0,0 +1,177 @@ +using ZB.MOM.WW.LocalDb; + +namespace ZB.MOM.WW.ScadaBridge.IntegrationTests; + +/// +/// Phase 2 convergence: deployed configuration across a real site pair, plus the resync +/// behaviour that the bespoke replicator needed a directional guard for. +/// +/// +/// Companion to : same +/// specifications-before-deletion role, for the intents held by +/// SiteReplicationActorTests and SfBufferResyncPredicateTests rather than by +/// the store-and-forward ReplicationService. +/// +[Collection("LocalDbSitePairConvergence")] +public sealed class LocalDbConfigConvergenceTests : LocalDbSitePairHarness +{ + // ---- data helpers ----------------------------------------------------------------- + + private static Task DeployConfigAsync(ILocalDb db, string instance, string configJson, string deploymentId) + => db.ExecuteAsync( + """ + INSERT INTO deployed_configurations ( + instance_unique_name, config_json, deployment_id, revision_hash, + is_enabled, deployed_at) + VALUES (@instance, @configJson, @deploymentId, @deploymentId, 1, @now) + ON CONFLICT(instance_unique_name) DO UPDATE SET + config_json = excluded.config_json, + deployment_id = excluded.deployment_id, + revision_hash = excluded.revision_hash, + deployed_at = excluded.deployed_at; + """, + new { instance, configJson, deploymentId, now = DateTime.UtcNow.ToString("o") }); + + private static async Task ReadConfigAsync(ILocalDb db, string instance) + { + var rows = await db.QueryAsync( + "SELECT config_json FROM deployed_configurations WHERE instance_unique_name = @instance", + static r => r.GetString(0), new { instance }); + return rows.Count == 0 ? null : rows[0]; + } + + private static Task EnqueueAsync(ILocalDb db, string id) + => db.ExecuteAsync( + """ + INSERT INTO sf_messages ( + id, category, target, payload_json, retry_count, max_retries, + retry_interval_ms, created_at, status) + VALUES (@id, 0, 'central', '{}', 0, 50, 30000, @now, 0); + """, + new { id, now = DateTime.UtcNow.ToString("o") }); + + private static async Task MessageExistsAsync(ILocalDb db, string id) + { + var rows = await db.QueryAsync( + "SELECT 1 FROM sf_messages WHERE id = @id", static r => r.GetInt32(0), new { id }); + return rows.Count > 0; + } + + // ---- scenarios -------------------------------------------------------------------- + + [Fact] + public async Task ANodeWithNewerLocalRows_KeepsThem_WhenAPeerSnapshotArrives() + { + // N1 Critical (was: SfBufferResyncPredicateTests). That test existed because the + // bespoke replicator's ReplaceAllAsync was a destructive DELETE-then-INSERT: a resync + // running in the wrong direction WIPED a live buffer, so the code needed a + // same-oldest-Up predicate to decide who had authority. The divergence case was a + // rolling restart leaving the delivering node oldest but not leader. + // + // Under LocalDb that failure mode is structurally impossible rather than guarded + // against: snapshot resync merges per row under LWW and never deletes. So this is + // deliberately NOT a directional-authority test — there is no active/standby + // asymmetry left to enforce. It asserts the property the guard was protecting: no + // node loses rows to a peer's snapshot. + // + // The setup is the wipe scenario made concrete. Each node holds a row the other has + // never seen, and they disagree about a third; then they resync. + await DeployConfigAsync(A, "shared-instance", """{"v":1}""", "dep-1"); + await WaitUntilAsync( + async () => await ReadConfigAsync(B, "shared-instance") is not null, + "the shared row to exist on both nodes before they diverge"); + + await StopPassiveAsync(); + + // A keeps working while B is down: a brand-new buffered message and a newer version + // of the shared config. + await EnqueueAsync(A, "live-row-on-a"); + await DeployConfigAsync(A, "shared-instance", """{"v":2}""", "dep-2"); + + // B is offline but its database is still writable — it also has local state the + // snapshot exchange must not destroy. + await EnqueueAsync(B, "live-row-on-b"); + + await RestartPairAsync(); + + await WaitUntilAsync( + async () => + await MessageExistsAsync(A, "live-row-on-a") && + await MessageExistsAsync(A, "live-row-on-b") && + await MessageExistsAsync(B, "live-row-on-a") && + await MessageExistsAsync(B, "live-row-on-b"), + "both nodes to hold the UNION of the rows each wrote while partitioned"); + + // And the contended row converges to the newer write rather than either node's + // snapshot flattening the other. + await WaitUntilAsync( + async () => await ReadConfigAsync(A, "shared-instance") == """{"v":2}""" + && await ReadConfigAsync(B, "shared-instance") == """{"v":2}""", + "both nodes to converge on the newer config for the contended instance"); + } + + [Fact] + public async Task ConfigDeployedOnA_ConvergesToB_WithoutAnyFetch() + { + // Was: SiteReplicationActorTests #1-#5 (notify-and-fetch — the standby is told a + // deploy happened, then HTTP-fetches the config itself, with retries and a + // superseded-404 path). Under CDC the config row simply replicates, so that whole + // exchange is deleted at Task 15. + // + // The plan called for asserting a fetcher test double records zero invocations. That + // would be THEATRE here and is deliberately not done: this harness has no actor + // system, no central, and no IDeploymentConfigFetcher in the graph at all, so a + // double would record zero calls whether or not the fetch path still existed. An + // assertion that cannot fail is worse than none. + // + // What this test honestly proves is the positive half: the config reaches node B + // through replication ALONE, in a process where fetching is not merely unused but + // absent. The negative half — that no fetch path survives — is proved by Task 15 + // deleting the code and the build still passing, which is a stronger check than any + // runtime counter. + // + // Scope note (D1): SiteReconciliationActor survives Phase 2 and legitimately fetches + // over HTTP at node startup when central reports gaps. "The standby never fetches, + // ever" would be a FALSE claim; the claim is about the deploy path only. + await DeployConfigAsync(A, "inst-deploy", """{"setpoint":42}""", "dep-100"); + + await WaitUntilAsync( + async () => await ReadConfigAsync(B, "inst-deploy") == """{"setpoint":42}""", + "the deployed config to reach node B over replication alone"); + } + + [Fact] + public async Task RedeployingAnInstance_ConvergesToTheNewRevision_OnBothNodes() + { + // The steady-state deploy loop, and the reason the old path needed a deployed_at + // guard: a standby that applied a STALE fetch would pin an instance to superseded + // config until the next deploy. Under LWW the newer write wins on the primary key, + // so the guard is not what protects this — but the outcome still has to hold. + await DeployConfigAsync(A, "inst-redeploy", """{"rev":"a"}""", "dep-a"); + await WaitUntilAsync( + async () => await ReadConfigAsync(B, "inst-redeploy") == """{"rev":"a"}""", + "the first revision to reach node B"); + + await DeployConfigAsync(A, "inst-redeploy", """{"rev":"b"}""", "dep-b"); + + await WaitUntilAsync( + async () => await ReadConfigAsync(A, "inst-redeploy") == """{"rev":"b"}""" + && await ReadConfigAsync(B, "inst-redeploy") == """{"rev":"b"}""", + "both nodes to converge on the redeployed revision"); + } + + [Fact] + public async Task ConfigDeployedOnB_ConvergesToA() + { + // Deliberately the reverse direction. The bespoke replicator was asymmetric by + // design — the active node pushed, the standby fetched — so "which node deployed it" + // was a meaningful question. Under CDC it is not, and this test is what says so. + // After a failover the surviving node must be able to deploy without waiting to be + // promoted to some authority role. + await DeployConfigAsync(B, "inst-from-b", """{"origin":"b"}""", "dep-b1"); + + await WaitUntilAsync( + async () => await ReadConfigAsync(A, "inst-from-b") == """{"origin":"b"}""", + "a config deployed on node B to reach node A"); + } +} From 79ce51612e4f0d517fdba21cb294ff18fd30f14c Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Mon, 20 Jul 2026 04:03:00 -0400 Subject: [PATCH 42/55] test(site): pin the active-node notification-config purge; scope the guarded write MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Task 12 + Task 13. No production behaviour change in either. Task 12: DeploymentManagerActor.HandleDeployArtifacts already purges notification_lists and smtp_configurations on every artifact apply, but nothing pinned the actor's CALL to it — ArtifactStorageTests covers the storage method only. Task 15 deletes SiteReplicationActor's copy, making this the sole remaining call site, and Task 16 edits this actor's wiring; dropping the call would leave plaintext SMTP passwords on disk with every suite still green. Verified red-first by commenting the call out: the pin fails with that message. Task 13: StoreDeployedConfigIfNewerAsync STAYS. Re-verified both callers — SiteReplicationActor:375 (dies at Task 15) and SiteReconciliationActor:166 (survives). Reconciliation is a per-node startup self-heal against central whose fetch races real deploys, so the deployed_at guard still does real work there. Doc comment rewritten to say so, and to warn against porting the guard onto the replication path where it would fight the HLC rather than help it. Also corrected a stale 'guarded standby write' section header in the tests. Re-ran the Task 13 step 2 scope check: ConfigFetchRetryCount's only production reader remains SiteReplicationActor:157, so its option + validator rule stay until Task 17, after Task 15 deletes the actor. Verified: build 0 warnings; SiteRuntime 533, Host 329, StoreAndForward 153, LocalDb integration 16 — all pass. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts --- .../Persistence/SiteStorageService.cs | 19 ++++- .../Actors/DeploymentManagerActorTests.cs | 82 +++++++++++++++++++ .../Persistence/SiteStorageServiceTests.cs | 7 +- 3 files changed, 104 insertions(+), 4 deletions(-) diff --git a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs index 746e8547..8dd80e06 100644 --- a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs +++ b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs @@ -156,9 +156,22 @@ public class SiteStorageService /// clause so the guard is atomic with no application-level read-modify-write. /// /// - /// This is the standby-node write path for replicated configs. The active-node - /// apply path () remains unguarded and always - /// overwrites, because the active node's write is always authoritative. + /// This is the reconciliation write path. SiteReconciliationActor runs a + /// per-node startup self-heal against central: it asks central what this node should be + /// running and fetches anything missing. That fetch races real deploys, so the + /// deployed_at guard is what stops a slow reconcile response from overwriting a + /// newer config that landed while it was in flight. The active-node apply path + /// () remains unguarded and always overwrites, + /// because a deploy is always authoritative. + /// + /// It was originally the standby write path as well, under notify-and-fetch: the + /// standby was told a deploy had happened and fetched the config itself. LocalDb Phase 2 + /// replaced that with change-data-capture — the config row simply replicates — so the + /// standby no longer writes here at all, and last-writer-wins on the primary key (not + /// this guard) is what orders concurrent writes between the two nodes. Reconciliation is + /// the reason the method survives; do not port the deployed_at guard onto the + /// replication path, where it would fight the HLC rather than help it. + /// /// /// is exposed for testing so that the exact /// deployed_at value can be controlled without sleeping between calls. diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs index 8852d3fc..f898add5 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs @@ -2,6 +2,7 @@ using Akka.Actor; using Akka.TestKit.Xunit2; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Logging.Abstractions; +using ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts; using ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment; using ZB.MOM.WW.ScadaBridge.Commons.Messages.DebugView; using ZB.MOM.WW.ScadaBridge.Commons.Messages.InboundApi; @@ -883,6 +884,87 @@ public class DeploymentManagerActorTests : TestKit, IDisposable Assert.Equal("SenderPump", response.InstanceUniqueName); } + // ── LocalDb Phase 2 / Task 12: the active-node central-only purge ── + + [Fact] + public async Task ApplyingArtifacts_PurgesCentralOnlyNotificationConfig() + { + // Security cleanup. notification_lists and smtp_configurations can hold plaintext + // SMTP passwords written by a pre-2026-07-10 build, and the ACTIVE node's artifact + // apply is what clears them (DeploymentManagerActor.HandleDeployArtifacts). The + // standby's copy of this call lives in SiteReplicationActor and dies with it at + // Task 15, which makes this call site the ONLY remaining one. + // + // Nothing pins it today: ArtifactStorageTests covers the storage method, not the + // actor's call to it, so Task 16's edits to this actor could drop the call and every + // suite would stay green. That is precisely the kind of silent security regression + // this test exists to prevent — verified red-first by commenting out the call. + await SeedCentralOnlyRowsAsync(); + Assert.Equal(1, await RowCountAsync("notification_lists")); + Assert.Equal(1, await RowCountAsync("smtp_configurations")); + + var manager = CreateDeploymentManager(); + manager.Tell(new DeployArtifactsCommand( + DeploymentId: "dep-purge-1", + SharedScripts: null, + ExternalSystems: null, + DatabaseConnections: null, + NotificationLists: null, + DataConnections: null, + SmtpConfigurations: null, + Timestamp: DateTimeOffset.UtcNow)); + + // The apply runs on a Task.Run inside the actor, so poll rather than assert once. + await AwaitPurgedAsync(); + } + + private async Task SeedCentralOnlyRowsAsync() + { + // Seeded through the service's own (already-open) LocalDb connection — a raw + // SqliteConnection would lack the pragmas and the zb_hlc_next() UDF the site + // tables' capture triggers call. Raw SQL is the only way these rows can exist at + // all now that the site-side write paths are gone. + await using var connection = _storage.CreateConnection(); + await using var command = connection.CreateCommand(); + command.CommandText = """ + INSERT INTO notification_lists (name, recipient_emails, updated_at) + VALUES ('Ops Team', '["ops@example.com"]', @u); + INSERT INTO smtp_configurations + (name, server, port, auth_mode, from_address, username, password, oauth_config, updated_at) + VALUES ('smtp.example.com:587', 'smtp.example.com', 587, 'BasicAuth', + 'noreply@example.com', 'smtpuser', 'PLAINTEXT-SECRET', NULL, @u); + """; + command.Parameters.AddWithValue("@u", DateTimeOffset.UtcNow.ToString("O")); + await command.ExecuteNonQueryAsync(); + } + + private async Task RowCountAsync(string table) + { + await using var connection = _storage.CreateConnection(); + await using var command = connection.CreateCommand(); + command.CommandText = $"SELECT COUNT(*) FROM {table}"; + return (long)(await command.ExecuteScalarAsync())!; + } + + private async Task AwaitPurgedAsync() + { + var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(10); + while (DateTime.UtcNow < deadline) + { + if (await RowCountAsync("notification_lists") == 0 && + await RowCountAsync("smtp_configurations") == 0) + { + return; + } + + await Task.Delay(50); + } + + Assert.Fail( + "Artifact apply did not purge the central-only notification/SMTP rows. " + + "The plaintext SMTP password is still on disk."); + } + /// /// In-test fake : returns a canned config JSON /// (notify-and-fetch success) or throws a canned exception (fetch failure), and records diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/SiteStorageServiceTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/SiteStorageServiceTests.cs index 585e09c3..77dfcb78 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/SiteStorageServiceTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/SiteStorageServiceTests.cs @@ -226,7 +226,12 @@ public class SiteStorageServiceTests : IAsyncLifetime, IDisposable Assert.Empty(overrides); } - // ── Task 13: StoreDeployedConfigIfNewerAsync (guarded standby write) ── + // ── StoreDeployedConfigIfNewerAsync (the deployed_at-guarded write) ── + // + // Originally the standby's notify-and-fetch write path. LocalDb Phase 2 replaced that + // with change-data-capture, so the surviving caller is SiteReconciliationActor's + // startup self-heal against central, where the guard still stops a slow reconcile + // response from overwriting a newer config that landed while it was in flight. /// /// Seeds a deployed_configurations row with an explicit deployed_at timestamp using the same From df0c6031ba586a369dd32bc72c6358f5d8fe8558 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Mon, 20 Jul 2026 04:03:48 -0400 Subject: [PATCH 43/55] docs(localdb): task state for tasks 10-13 + deviations Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts --- .../2026-07-19-localdb-adoption-phase2.md.tasks.json | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json b/docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json index 19d4b185..c1b96e57 100644 --- a/docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json +++ b/docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json @@ -78,10 +78,10 @@ {"id": 7, "subject": "Task 7: Extend SiteLocalDbSetup with the new DDL (not yet registered)", "status": "completed", "classification": "standard", "blockedBy": [5, 6], "commit": "f8aa02e2", "note": "As planned. Two pins in SiteLocalDbWiringTests through the REAL composition root: all 12 tables exist, and ReplicatedTables is EXACTLY the Phase 1 pair (an equality check, not Contains - the 'not yet' is the assertion that matters). Task 14 should INVERT that second test, not delete it. Non-vacuity verified by removing the DDL."}, {"id": 8, "subject": "Task 8: Extend the legacy migrator for sf_messages", "status": "completed", "classification": "high-risk", "blockedBy": [7], "commit": "bdc0dffe", "note": "DEVIATION: the copy INTERSECTS the legacy column set with the current one instead of naming all 16 columns. A legacy file from an older build lacks execution_id/parent_execution_id/last_attempt_at_ms, and naming a missing column throws 'no such column' - which the existing ReadAll treats as an unrecognised shape and SILENTLY DISCARDS EVERY ROW. For undelivered messages that is real data loss. A required-column (PK) guard stops the tolerance degrading into NULL-keyed copies. Tests register sf_messages on their own harness (production OnReady does not until Task 14), so Task 14 MUST keep Migrate as the LAST call in OnReady, after all registrations. Non-vacuity verified: all 4 fail without the Migrate call."}, {"id": 9, "subject": "Task 9: Extend the legacy migrator for the config tables (skip notification/smtp)", "status": "completed", "classification": "high-risk", "blockedBy": [7], "commit": "5ddc7eed", "note": "7 of 9 tables migrated; notification_lists + smtp_configurations skipped as planned (plaintext passwords must not enter a table Task 14 makes replicated) - the skip is pinned by a test that also greps __localdb_oplog.row_json for the secret. Task 8 MigrateTable generalized to MigrateFile(many tables, one transaction, one rename). ADDED beyond the plan: MigratorColumnLists_MatchTheLiveSchema, a column-parity test vs SiteStorageSchema - the plan called for a manual column check, but a mismatch is invisible at runtime in BOTH directions (a typo is silently dropped by the intersection; a missing column silently leaves data behind). SiteStorageTables + LegacyTable are internal so the test can read them. Non-vacuity verified twice: removing the call (3 fail) and wrongly adding notification/smtp to the map (skip test fails)."}, - {"id": 10, "subject": "Task 10: Port the S&F replication test intents as CDC specs", "status": "pending", "classification": "standard", "blockedBy": [8]}, - {"id": 11, "subject": "Task 11: Port the resync + directional-authority tests", "status": "pending", "classification": "standard", "blockedBy": [9], "note": "Zero-fetch assertion scoped to deploy convergence - SiteReconciliationActor's startup fetch is legitimate and survives."}, - {"id": 12, "subject": "Task 12: Pin the active-node SMTP purge (corrected: already exists at DeploymentManagerActor.cs:1921)", "status": "pending", "classification": "standard", "blockedBy": [10, 11], "note": "Pin test only - no production change. Guards Task 16's actor edits from silently dropping the purge call."}, - {"id": 13, "subject": "Task 13: Notify-and-fetch scope check - guarded write STAYS for SiteReconciliationActor", "status": "pending", "classification": "standard", "blockedBy": [12], "note": "CORRECTED: do NOT delete StoreDeployedConfigIfNewerAsync (second caller SiteReconciliationActor.cs:166 survives; deleting here wouldn't compile anyway). Doc-comment update only; ConfigFetchRetryCount removal moved to Task 17."}, + {"id": 10, "subject": "Task 10: Port the S&F replication test intents as CDC specs", "status": "completed", "classification": "standard", "blockedBy": [8], "commit": "2bbe6631", "note": "DEVIATION: extracted the Phase 1 fixture into LocalDbSitePairHarness (abstract base) instead of duplicating ~150 lines; Phase 1 tests derive from it and pass unchanged. The harness registers the 8 Phase 2 tables itself (production OnReady does not until Task 14) - DELETE RegisterPhase2TablesUntilCutover at Task 14. The 8-table list is literal, not derived from production code, so a cutover registering the wrong set fails these tests. Non-vacuity: unregistering sf_messages failed 6 of 7 - the 7th (add-then-remove ordering) PASSED because an absent row is also what a non-replicating pair looks like; fixed with a control row that must converge in the same window."}, + {"id": 11, "subject": "Task 11: Port the resync + directional-authority tests", "status": "completed", "classification": "standard", "blockedBy": [9], "note": "N1 re-expressed per D2 as a no-rows-lost property (LWW merge, never deletes), not directional authority. DEVIATION on the zero-fetch assertion: the plan wanted a fetcher double recording zero calls, but this harness has no actor system / no central / no IDeploymentConfigFetcher in the graph, so the double could not fail either way. Replaced with the positive half (config reaches B over replication alone) plus an in-file comment recording that the negative half is proved by Task 15 deleting the code and the build passing. D1 scope note recorded in-file. Non-vacuity: unregistering deployed_configurations fails all 4.", "commit": "c56bf4ae"}, + {"id": 12, "subject": "Task 12: Pin the active-node SMTP purge (corrected: already exists at DeploymentManagerActor.cs:1921)", "status": "completed", "classification": "standard", "blockedBy": [10, 11], "note": "As planned - NO production change; D3 confirmed correct, the call already exists at DeploymentManagerActor.cs:1921. Pin verified RED-FIRST by commenting out the call. Test needed a using for Commons.Messages.Artifacts and polls (the apply runs on a Task.Run inside the actor).", "commit": "79ce5161"}, + {"id": 13, "subject": "Task 13: Notify-and-fetch scope check - guarded write STAYS for SiteReconciliationActor", "status": "completed", "classification": "standard", "blockedBy": [12], "note": "Doc-comment only, as planned. Re-verified both callers: SiteReplicationActor:375 (dies Task 15) + SiteReconciliationActor:166 (survives). Step 2 scope check re-run: ConfigFetchRetryCount's only production reader is still SiteReplicationActor:157, so option + validator rule stay until Task 17. Also fixed a stale 'guarded standby write' header in SiteStorageServiceTests.", "commit": "79ce5161"}, {"id": 14, "subject": "Task 14: Register the 8 Phase 2 tables and delete ReplicationService", "status": "pending", "classification": "high-risk", "blockedBy": [13], "note": "THE CUTOVER. Register 7 config tables + sf_messages - NOT notification_lists/smtp_configurations (reversed from original; see D3). Keep Migrate LAST in OnReady. Registration + bespoke deletion in ONE commit."}, {"id": 15, "subject": "Task 15: Delete SiteReplicationActor and its messages", "status": "pending", "classification": "high-risk", "blockedBy": [14], "note": "Resync records live at SiteReplicationActor.cs:678-707, not ReplicationMessages.cs."}, {"id": 16, "subject": "Task 16: Clean up DeploymentManagerActor and AkkaHostedService", "status": "pending", "classification": "standard", "blockedBy": [15], "note": "Ctor param at :169; same-typed IActorRef? dclManager at :168 BEFORE it is the silent-shift hazard; Props.Create positional at AkkaHostedService.cs:810."}, From 037798b3679bbf693170fdbc1f0b5a2fb068bf77 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Mon, 20 Jul 2026 04:20:05 -0400 Subject: [PATCH 44/55] feat(localdb)!: replicate site config + sf_messages via CDC, delete the bespoke replicators MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Tasks 14, 15 and 16, landed as ONE commit. PLAN DEFECT: these three tasks cannot compile separately. SiteReplicationActor takes a ReplicationService and calls ReplaceAllAsync (Task 14 deletes both); DeploymentManagerActor Tells message types declared in ReplicationMessages.cs (Task 15 deletes it); AkkaHostedService constructs the actor (Task 16). Any ordering leaves a broken intermediate. Combining them also strengthens the invariant Task 14 already stated for itself — the two mechanisms never both run, and never neither. Registered 8 tables in SiteLocalDbSetup.OnReady: sf_messages plus the 7 site config tables. notification_lists and smtp_configurations are deliberately NOT registered — permanently empty by design, so registering them would open a standing replication channel whose only historical payload was plaintext SMTP passwords. Migrate stays the LAST call in OnReady, after all registrations, so migrated rows enter the oplog through live capture triggers. Deleted: SiteReplicationActor, ReplicationMessages.cs, ReplicationService, StoreAndForwardStorage.ReplaceAllAsync, and 6 test files. ReplaceAllAsync is not merely unused but unsafe to keep: a mass DELETE on a now-replicated table would be captured and shipped to the peer. Kept ActiveNodeEvaluator (delivery gate + heartbeat still need it) with its doc corrected, and activeNodeCheck in AkkaHostedService (SiteCommunicationActor). The positional-argument hazard the plan flagged was real: removing DeploymentManagerActor's optional IActorRef? replicationActor shifted 6 trailing optionals, and 4 test call sites bound the wrong arguments with no compile error at some positions. Converted them to named arguments where possible — Props.Create builds an expression tree, which rejects out-of-position named args, so the rest are padded positionally with a comment saying why. The Task 7 'not yet registered' test was INVERTED rather than deleted, and is exact in both directions: too few means a table silently stops replicating, too many means the SMTP tables leak. Added a separate security-named test for those two, and a composite-PK test (LWW keys on the full PK, so a truncated key set would collapse distinct rows). The convergence suites now get their registrations from the real OnReady — their temporary harness registration is deleted, so they prove the cutover rather than agreeing with themselves. Verified: build 0 warnings; SiteRuntime 512, StoreAndForward 130, Host 330, AuditLog 355, ExternalSystemGateway 142, HealthMonitoring 97, LocalDb integration 16 — all pass, 0 failures. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts --- .../ClusterState/ActiveNodeEvaluator.cs | 19 +- .../Actors/AkkaHostedService.cs | 37 +- .../SiteLocalDbSetup.cs | 33 +- .../Actors/DeploymentManagerActor.cs | 24 +- .../Actors/SiteReplicationActor.cs | 707 ------------------ .../Messages/ReplicationMessages.cs | 44 -- .../ReplicationService.cs | 194 ----- .../ServiceCollectionExtensions.cs | 9 - .../StoreAndForwardService.cs | 32 +- .../StoreAndForwardStorage.cs | 35 +- .../CompositionRootTests.cs | 1 - .../SiteLocalDbWiringTests.cs | 59 +- .../Cluster/SfBufferResyncPredicateTests.cs | 122 --- .../LocalDbSitePairHarness.cs | 34 +- .../Actors/DeploymentManagerActorTests.cs | 34 +- .../DeploymentManagerCertReconcileTests.cs | 3 +- .../DeploymentManagerLoggerFactoryTests.cs | 9 +- .../Actors/DeploymentManagerRedeployTests.cs | 7 +- .../Actors/SiteReplicationActorTests.cs | 568 -------------- .../ResyncWireSerializationPinTests.cs | 95 --- .../CachedCallAttemptEmissionTests.cs | 2 - .../ReplicationServiceTests.cs | 236 ------ .../ReplicationWireSerializationPinTests.cs | 82 -- .../StoreAndForwardReplicationTests.cs | 266 ------- .../StoreAndForwardSiteEventTests.cs | 2 +- .../StoreAndForwardStorageTests.cs | 19 +- 26 files changed, 160 insertions(+), 2513 deletions(-) delete mode 100644 src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/SiteReplicationActor.cs delete mode 100644 src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Messages/ReplicationMessages.cs delete mode 100644 src/ZB.MOM.WW.ScadaBridge.StoreAndForward/ReplicationService.cs delete mode 100644 tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/SfBufferResyncPredicateTests.cs delete mode 100644 tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/SiteReplicationActorTests.cs delete mode 100644 tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/ResyncWireSerializationPinTests.cs delete mode 100644 tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ReplicationServiceTests.cs delete mode 100644 tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ReplicationWireSerializationPinTests.cs delete mode 100644 tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardReplicationTests.cs diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/ClusterState/ActiveNodeEvaluator.cs b/src/ZB.MOM.WW.ScadaBridge.Communication/ClusterState/ActiveNodeEvaluator.cs index 57d7b425..43ba93a7 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/ClusterState/ActiveNodeEvaluator.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/ClusterState/ActiveNodeEvaluator.cs @@ -10,11 +10,20 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.ClusterState; /// once the original first node restarts and rejoins; every product-level active/standby /// decision must use this evaluator, never cluster.State.Leader. /// -/// Lives in Communication (not Host) so BOTH SiteCommunicationActor and -/// SiteReplicationActor can default to it — Host cannot be referenced from either. -/// The Host's ClusterActivityEvaluator.SelfIsOldest delegates here, so the S&F -/// delivery gate (IClusterNodeProvider.SelfIsPrimary), the resync authority checks, -/// and the heartbeat IsActive stamp all share one implementation. +/// Lives in Communication (not Host) so SiteCommunicationActor can default to it — +/// Host cannot be referenced from there. The Host's +/// ClusterActivityEvaluator.SelfIsOldest delegates here, so the S&F delivery gate +/// (IClusterNodeProvider.SelfIsPrimary) and the heartbeat IsActive stamp share one +/// implementation. +/// +/// It also backed SiteReplicationActor's resync authority checks until LocalDb +/// Phase 2 deleted that actor. Those checks existed because the bespoke resync applied a +/// destructive delete-all-then-insert-all, so running it in the wrong direction wiped a +/// live store-and-forward buffer. LocalDb's snapshot resync merges per row under +/// last-writer-wins and never deletes, so there is no destructive apply left to gate — the +/// evaluator survives for the delivery gate and the heartbeat, which still genuinely need +/// a single active node. +/// /// /// public static class ActiveNodeEvaluator diff --git a/src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs b/src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs index 2e29f03a..16b8b41c 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs @@ -766,37 +766,18 @@ akka {{ var deploymentConfigFetcher = _serviceProvider.GetService(); - // Create SiteReplicationActor on every node (not a singleton) - var sfStorage = _serviceProvider.GetRequiredService(); - var replicationService = _serviceProvider.GetRequiredService(); - var replicationLogger = _serviceProvider.GetRequiredService() - .CreateLogger(); - - // ONE active-node predicate instance governs the S&F delivery gate, the resync - // authority checks (SiteReplicationActor), and the heartbeat IsActive stamp - // (SiteCommunicationActor, wired below) — review 02 round 2, N1. Null in - // non-clustered test hosts: the actors fall back to the shared oldest-Up - // evaluator, never to a leader check. + // ONE active-node predicate instance governs the S&F delivery gate and the + // heartbeat IsActive stamp (SiteCommunicationActor, wired below) — review 02 + // round 2, N1. It also governed SiteReplicationActor's resync authority until + // LocalDb Phase 2 deleted that actor: the library's snapshot resync merges per row + // under last-writer-wins and never deletes, so there is no destructive apply left + // to need an authority check. Null in non-clustered test hosts: the consumers fall + // back to the shared oldest-Up evaluator, never to a leader check. var clusterNodeProvider = _serviceProvider.GetService(); Func? activeNodeCheck = clusterNodeProvider != null ? () => clusterNodeProvider.SelfIsPrimary : null; - var replicationActor = _actorSystem!.ActorOf( - Props.Create(() => new SiteReplicationActor( - storage, sfStorage, replicationService, siteRole, replicationLogger, - deploymentConfigFetcher, activeNodeCheck, siteRuntimeOptionsValue, null)), - "site-replication"); - - // Wire S&F replication handler to forward operations via the replication actor - replicationService.SetReplicationHandler(op => - { - replicationActor.Tell(new ReplicateStoreAndForward(op)); - return Task.CompletedTask; - }); - - _logger.LogInformation("SiteReplicationActor created and S&F replication handler wired"); - // Deployment Manager — role-scoped singleton via SingletonRegistrar // (review 01 round-2 N5): previously hand-rolled with bare PoisonPill // termination and NO PhaseClusterLeave drain, so in-flight SQLite @@ -807,7 +788,7 @@ akka {{ _actorSystem!, "deployment-manager", Props.Create(() => new DeploymentManagerActor( storage, compilationService, sharedScriptLibrary, streamManager, - siteRuntimeOptionsValue, dmLogger, dclManager, replicationActor, + siteRuntimeOptionsValue, dmLogger, dclManager, siteHealthCollector, _serviceProvider, null, deploymentConfigFetcher)), _logger, role: siteRole); var dmProxy = dm.Proxy; @@ -1053,7 +1034,7 @@ akka {{ // SetReady asserts a deliberately narrow contract. By this point the // actor system exists, SiteStreamManager.Initialize has run, and every // role actor (SiteCommunicationActor, deployment-manager singleton, - // SiteReplicationActor, the ClusterClient) has been created with ActorOf — + // the ClusterClient) has been created with ActorOf — // creation and the registration Tells are synchronous and strictly ordered. // What is NOT guaranteed is completion of each actor's PreStart or the // ClusterClient's initial-contact handshake with central: those are diff --git a/src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbSetup.cs b/src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbSetup.cs index 626c6d5d..beb7f311 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbSetup.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbSetup.cs @@ -48,8 +48,7 @@ public static class SiteLocalDbSetup SiteEventLogSchema.Apply(connection); // Phase 2: the site's configuration tables and the store-and-forward buffer - // now live in this file too. Created here, deliberately NOT registered — see - // below. + // now live in this file too. SiteStorageSchema.Apply(connection); StoreAndForwardSchema.Apply(connection); } @@ -60,12 +59,30 @@ public static class SiteLocalDbSetup db.RegisterReplicated("OperationTracking"); db.RegisterReplicated("site_events"); - // The Phase 2 tables are created above but NOT registered yet. The bespoke - // SiteReplicationActor / StoreAndForward ReplicationService still own replicating - // them until the Task 14 cutover deletes both and registers these in one commit. - // Registering them now would mean two independent replicators writing the same - // rows — harmless in principle (both upsert) but it would mask a defect in either - // one, which is exactly what the cutover needs to be able to see. + // Phase 2: the store-and-forward buffer and the seven site configuration tables. + // These replaced the bespoke SiteReplicationActor and StoreAndForward + // ReplicationService, which shipped hand-written Add/Remove/Park/Requeue operations + // over Akka; both were deleted in the same commit that added these lines, so the + // two mechanisms never ran at once. + // + // Both composite-PK tables are fine: RegisterReplicated orders multi-column PKs by + // ordinal. No Phase 2 table has a BLOB column, which it would reject. + db.RegisterReplicated("sf_messages"); + db.RegisterReplicated("deployed_configurations"); + db.RegisterReplicated("static_attribute_overrides"); + db.RegisterReplicated("shared_scripts"); + db.RegisterReplicated("external_systems"); + db.RegisterReplicated("database_connections"); + db.RegisterReplicated("data_connection_definitions"); + db.RegisterReplicated("native_alarm_state"); + + // notification_lists and smtp_configurations are created but deliberately NOT + // registered. They are permanently empty by design — the site-side write paths were + // removed on 2026-07-10, the legacy migrator skips them, and the active node's + // artifact apply purges them on every deploy. Registering them would open a standing + // replication channel whose only historical payload was plaintext SMTP passwords, in + // exchange for replicating nothing. Anyone adding them here should first establish + // that a site has a legitimate reason to hold SMTP credentials at all. // AFTER registration, so migrated rows enter the oplog and reach the peer like // any other write. Before it, they would be invisible to replication forever. diff --git a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs index e360dffc..56152dc2 100644 --- a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs +++ b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs @@ -52,7 +52,6 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers /// private readonly ILoggerFactory _loggerFactory; private readonly IActorRef? _dclManager; - private readonly IActorRef? _replicationActor; private readonly ISiteHealthCollector? _healthCollector; private readonly IServiceProvider? _serviceProvider; /// @@ -166,7 +165,6 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers SiteRuntimeOptions options, ILogger logger, IActorRef? dclManager = null, - IActorRef? replicationActor = null, ISiteHealthCollector? healthCollector = null, IServiceProvider? serviceProvider = null, ILoggerFactory? loggerFactory = null, @@ -181,7 +179,6 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers _streamManager = streamManager; _options = options; _dclManager = dclManager; - _replicationActor = replicationActor; _healthCollector = healthCollector; _serviceProvider = serviceProvider; _configFetcher = configFetcher; @@ -785,15 +782,14 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers await _storage.ClearStaticOverridesAsync(instanceName); await _storage.ClearNativeAlarmsForInstanceAsync(instanceName); - // Replicate to standby node — notify-and-fetch: send only the deployment id + - // central fetch coordinates (NOT the config JSON). The standby fetches the - // config over HTTP itself, so a large config never crosses the intra-site Akka - // hop (which would silently drop on the 128 KB frame trap). When the coords are - // absent (deploy paths other than RefreshDeployment), the standby fetch is a - // no-op miss and reconciliation is the durable backstop. - _replicationActor?.Tell(new ReplicateConfigDeploy( - instanceName, command.DeploymentId, command.RevisionHash, true, - command.CentralFetchBaseUrl ?? "", command.FetchToken ?? "")); + // No explicit replication step: deployed_configurations is a replicated table, + // so the write above is captured and shipped to the peer like any other row. + // This used to be a notify-and-fetch Tell — the standby was sent the deployment + // id plus central fetch coordinates and pulled the config over HTTP itself, + // because a large config would silently drop on the intra-site Akka hop's + // 128 KB frame trap. Replication carries the row directly and has no such limit, + // so the standby no longer fetches on deploy. (SiteReconciliationActor still + // fetches at node startup when central reports gaps — a different path.) return new DeployPersistenceResult( command.DeploymentId, instanceName, true, null, sender, isRedeploy); @@ -967,7 +963,6 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers { if (t.IsCompletedSuccessfully) { - _replicationActor?.Tell(new ReplicateConfigSetEnabled(instanceName, false)); // Operational `deployment` event — disable succeeded. LogDeploymentEvent("Info", instanceName, $"Instance {instanceName} disabled"); } @@ -1001,7 +996,6 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers Task.Run(async () => { await _storage.SetInstanceEnabledAsync(instanceName, true); - _replicationActor?.Tell(new ReplicateConfigSetEnabled(instanceName, true)); var configs = await _storage.GetAllDeployedConfigsAsync(); var config = configs.FirstOrDefault(c => c.InstanceUniqueName == instanceName); return new EnableResult(command, config, null, sender); @@ -1099,7 +1093,6 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers { if (t.IsCompletedSuccessfully) { - _replicationActor?.Tell(new ReplicateConfigRemove(instanceName)); // Operational `deployment` event — delete succeeded. LogDeploymentEvent("Info", instanceName, $"Instance {instanceName} deleted"); } @@ -1949,7 +1942,6 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers // central-only and is never stored on a site (see the purge above). // Replicate artifacts to standby node - _replicationActor?.Tell(new ReplicateArtifacts(command)); return new ArtifactDeploymentResponse( command.DeploymentId, "", true, null, DateTimeOffset.UtcNow); diff --git a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/SiteReplicationActor.cs b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/SiteReplicationActor.cs deleted file mode 100644 index 63066d2d..00000000 --- a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/SiteReplicationActor.cs +++ /dev/null @@ -1,707 +0,0 @@ -using Akka.Actor; -using Akka.Cluster; -using Akka.Event; -using Microsoft.Extensions.Logging; -using ZB.MOM.WW.ScadaBridge.Commons.Observability; -using ZB.MOM.WW.ScadaBridge.SiteRuntime.Deployment; -using ZB.MOM.WW.ScadaBridge.SiteRuntime.Messages; -using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; -using ZB.MOM.WW.ScadaBridge.StoreAndForward; - -namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors; - -/// -/// Runs on every site node (not a singleton). Handles both config and S&F replication -/// between site cluster peers. -/// -/// Outbound: receives local replication requests and forwards to peer via ActorSelection. -/// Inbound: receives replicated operations from peer and applies to local SQLite. -/// Uses fire-and-forget (Tell) — no ack wait per design. -/// -public class SiteReplicationActor : ReceiveActor, IWithTimers -{ - private readonly SiteStorageService _storage; - private readonly StoreAndForwardStorage _sfStorage; - private readonly ReplicationService _replicationService; - private readonly IDeploymentConfigFetcher? _configFetcher; - private readonly string _siteRole; - private readonly ILogger _logger; - private readonly Cluster _cluster; - private readonly Func _isActive; - private readonly int _configFetchRetryCount; - private readonly TimeSpan _configFetchRetryDelay; - private Address? _peerAddress; - - /// Akka timer scheduler injected by the framework via . - public ITimerScheduler Timers { get; set; } = null!; - - // ── Chunked-resync assembly (standby side; actor-thread only) ── - private string? _assemblingResyncId; - private int _assemblingTotalChunks; - private bool _assemblingTruncated; - private readonly Dictionary> _assemblingChunks = new(); - private const string ResyncAssemblyTimerKey = "sf-resync-assembly-timeout"; - - /// How long a partial chunk assembly may wait for its missing chunks before - /// being discarded (a lost chunk = lost resync; the next peer-track retries). Ctor - /// test seam; production default 30 s. - private readonly TimeSpan _resyncAssemblyTimeout; - - // ── Resync delivery confirmation (active side; actor-thread only) ── - private string? _pendingAckResyncId; - private const string ResyncAckTimerKey = "sf-resync-ack-timeout"; - - /// How long the active node waits for the standby's - /// before warning + counting the resync as unacknowledged (lost chunks / dead peer). Ctor - /// test seam; production default 60 s. - private readonly TimeSpan _resyncAckTimeout; - - /// - /// Maximum rows an active node returns in a single anti-entropy resync snapshot. - /// A standby whose buffer exceeded this (Truncated snapshot) resyncs the oldest - /// 10 000 rows; further divergence drains naturally as the active node delivers. - /// - private const int MaxResyncRows = 10_000; - - /// - /// Estimated per-chunk payload budget for a resync snapshot. Akka remoting's default - /// maximum-frame-size is 128 000 bytes and BuildHocon sets no override, - /// so the monolithic is silently undeliverable for any - /// realistic backlog (review 02 round 2, N2). 64 000 bytes leaves ≈50% headroom for - /// the JSON envelope, CLR type manifests, and the non-payload columns. - /// - internal const int MaxResyncChunkBytes = 64_000; - - /// Row cap per resync chunk (bounds a chunk even when every row is tiny). - internal const int MaxResyncChunkRows = 200; - - /// - /// Splits a resync snapshot into chunks that fit Akka remoting's default - /// 128 000-byte frame (review 02 round 2, N2): rows accumulate until the estimated - /// payload budget or the row cap is hit. Estimation is payload-dominated - /// (payload_json length + 512 bytes fixed overhead per row); a single row whose - /// payload exceeds the budget ships alone (Warning at the call site). Order is - /// preserved (oldest-first, matching GetAllMessagesAsync). - /// - internal static List> ChunkForRemoting( - IReadOnlyList rows, int maxChunkBytes, int maxChunkRows) - { - var chunks = new List>(); - var current = new List(); - var currentBytes = 0; - foreach (var row in rows) - { - var estimate = (row.PayloadJson?.Length ?? 0) + 512; - if (current.Count > 0 && (currentBytes + estimate > maxChunkBytes || current.Count >= maxChunkRows)) - { - chunks.Add(current); - current = new List(); - currentBytes = 0; - } - current.Add(row); - currentBytes += estimate; - } - if (current.Count > 0) chunks.Add(current); - return chunks; - } - - /// - /// Initializes a new and registers Akka message handlers. - /// - /// Service for accessing local site storage. - /// Store-and-forward SQLite storage for replication of buffered messages. - /// Service providing replication transport logic. - /// Akka cluster role used to identify peer nodes to replicate to. - /// Logger instance. - /// - /// Fetches a deployed instance's config JSON from central over HTTP. Used by the - /// notify-and-fetch standby apply path (): the peer - /// replicates only the deployment id, and the standby fetches the config itself so a large - /// config never crosses the intra-site Akka hop. Null on nodes/tests without a fetcher. - /// - /// - /// Active-node check that gates the buffer-resync roles (a standby requests a - /// resync, the active node answers). Production wiring passes the Host's - /// IClusterNodeProvider.SelfIsPrimary delegate (the same instance gating the - /// S&F delivery sweep); null falls back to the shared oldest-Up evaluator - /// (). - /// - /// Site runtime options, including the config-fetch retry count; production defaults apply when null. - /// Delay between config-fetch retry attempts; defaults to 2 seconds when null. - public SiteReplicationActor( - SiteStorageService storage, - StoreAndForwardStorage sfStorage, - ReplicationService replicationService, - string siteRole, - ILogger logger, - IDeploymentConfigFetcher? configFetcher = null, - Func? isActiveOverride = null, - SiteRuntimeOptions? options = null, - TimeSpan? configFetchRetryDelay = null, - TimeSpan? resyncAssemblyTimeout = null, - TimeSpan? resyncAckTimeout = null) - { - _storage = storage; - _sfStorage = sfStorage; - _replicationService = replicationService; - _configFetcher = configFetcher; - _siteRole = siteRole; - _logger = logger; - _cluster = Cluster.Get(Context.System); - _isActive = isActiveOverride ?? DefaultIsActive; - _resyncAssemblyTimeout = resyncAssemblyTimeout ?? TimeSpan.FromSeconds(30); - _resyncAckTimeout = resyncAckTimeout ?? TimeSpan.FromSeconds(60); - // UA2: bound the standby's replicated-config fetch retries. At least one - // attempt always runs; the fixed inter-attempt delay is a test seam - // (production default 2 s). - _configFetchRetryCount = Math.Max(1, options?.ConfigFetchRetryCount ?? 1); - _configFetchRetryDelay = configFetchRetryDelay ?? TimeSpan.FromSeconds(2); - - // Cluster member events - Receive(HandleMemberUp); - Receive(HandleMemberRemoved); - Receive(HandleCurrentClusterState); - - // Outbound — forward to peer - Receive(msg => SendToPeer(new ApplyConfigDeploy( - msg.InstanceName, msg.DeploymentId, msg.RevisionHash, msg.IsEnabled, - msg.CentralFetchBaseUrl, msg.FetchToken))); - Receive(msg => SendToPeer(new ApplyConfigRemove(msg.InstanceName))); - Receive(msg => SendToPeer(new ApplyConfigSetEnabled( - msg.InstanceName, msg.IsEnabled))); - Receive(msg => SendToPeer(new ApplyArtifacts(msg.Command))); - Receive(msg => SendToPeer(new ApplyStoreAndForward(msg.Operation))); - - // Inbound — apply from peer - Receive(HandleApplyConfigDeploy); - Receive(HandleApplyConfigRemove); - Receive(HandleApplyConfigSetEnabled); - Receive(HandleApplyArtifacts); - Receive(HandleApplyStoreAndForward); - - // Anti-entropy — full S&F buffer resync on peer (re)join - Receive(HandleRequestSfBufferResync); - Receive(HandleSfResyncSnapshotLoaded); - Receive(HandleSfBufferSnapshotChunk); - Receive(HandleResyncAssemblyTimedOut); - Receive(msg => - { - if (msg.ResyncId == _pendingAckResyncId) - { - _pendingAckResyncId = null; - Timers.Cancel(ResyncAckTimerKey); - } - ScadaBridgeTelemetry.RecordSfResyncCompleted(); - _logger.LogInformation("S&F resync {ResyncId} acknowledged by standby: {Rows} row(s) applied", - msg.ResyncId, msg.RowCount); - }); - Receive(msg => - { - if (msg.ResyncId != _pendingAckResyncId) return; - _pendingAckResyncId = null; - ScadaBridgeTelemetry.RecordSfResyncAckMissing(); - _logger.LogWarning( - "S&F resync {ResyncId} was never acknowledged within {Window} — snapshot chunks may have been lost (frame drop / dead peer); the next peer-track retries", - msg.ResyncId, _resyncAckTimeout); - }); - Receive(HandleSfBufferSnapshot); // legacy monolithic handler — retained for rolling compat - } - - /// - protected override void PreStart() - { - base.PreStart(); - _cluster.Subscribe(Self, ClusterEvent.SubscriptionInitialStateMode.InitialStateAsSnapshot, - typeof(ClusterEvent.MemberUp), - typeof(ClusterEvent.MemberRemoved)); - _logger.LogInformation("SiteReplicationActor started, subscribing to cluster events for role {Role}", _siteRole); - } - - /// - protected override void PostStop() - { - _cluster.Unsubscribe(Self); - base.PostStop(); - } - - private void HandleCurrentClusterState(ClusterEvent.CurrentClusterState state) - { - foreach (var member in state.Members) - { - if (member.Status == MemberStatus.Up) - TryTrackPeer(member); - } - } - - private void HandleMemberUp(ClusterEvent.MemberUp evt) - { - TryTrackPeer(evt.Member); - } - - private void HandleMemberRemoved(ClusterEvent.MemberRemoved evt) - { - if (evt.Member.Address.Equals(_peerAddress)) - { - _logger.LogInformation("Peer node removed: {Address}", _peerAddress); - _peerAddress = null; - } - } - - private void TryTrackPeer(Member member) - { - // Must have our site role, and must not be self - if (member.HasRole(_siteRole) && !member.Address.Equals(_cluster.SelfAddress)) - { - _peerAddress = member.Address; - _logger.LogInformation("Peer node tracked: {Address}", _peerAddress); - OnPeerTracked(); - } - } - - /// - /// Side-effect run whenever a peer is (re)tracked. A standby requests a - /// full S&F buffer snapshot for anti-entropy resync — this closes the "a - /// standby down for an hour rejoins and diverges forever" gap: it may have missed - /// replicated Add/Remove/Park ops while it was gone. The active node never - /// requests. so tests can drive it without a - /// real two-node cluster. - /// - protected virtual void OnPeerTracked() - { - if (!SafeIsActive()) - { - SendToPeer(new RequestSfBufferResync()); - } - } - - /// - /// Repo-standard active-node check: this node is active when it is the OLDEST Up - /// member carrying the site role — the same oldest-Up semantics as the S&F delivery - /// gate (IClusterNodeProvider.SelfIsPrimary → ClusterActivityEvaluator → shared - /// ActiveNodeEvaluator). NEVER the cluster leader: leadership is lowest-address and - /// diverges from singleton/delivery placement permanently after the lower-address - /// node restarts — the divergence that made the delivering node wipe its own live - /// buffer via a wrong-direction resync (review 02 round 2, N1 Critical). Any other - /// state reports standby — safe-by-default. - /// - private bool DefaultIsActive() => - Communication.ClusterState.ActiveNodeEvaluator.SelfIsOldestUp(_cluster, _siteRole); - - /// - /// Evaluates the active-node check, treating a throwing check as standby - /// (safe-by-default: a standby never delivers or answers resyncs). - /// - private bool SafeIsActive() - { - try - { - return _isActive(); - } - catch (Exception ex) - { - _logger.LogWarning(ex, "Active-node check threw; treating node as standby"); - return false; - } - } - - /// - /// Forwards a replication message to the tracked peer node's site-replication actor - /// (fire-and-forget, dropped when no peer is tracked). - /// so tests can intercept the peer send without standing up a real two-node cluster. - /// - /// The replication message to forward to the peer. - protected virtual void SendToPeer(object message) - { - if (_peerAddress == null) - { - // A dropped op is a lost delta — surface it at Warning + a metric so a - // never-tracked peer (a dead standby) is visible, not silent (arch - // review 02). In single-node dev the peer is legitimately absent; the - // per-op warning is rate-tolerable (accepted per review). - ScadaBridgeTelemetry.RecordReplicationFailure(); - _logger.LogWarning("No peer available, dropping replication message {Type}", message.GetType().Name); - return; - } - - var path = new RootActorPath(_peerAddress) / "user" / "site-replication"; - Context.ActorSelection(path).Tell(message); - } - - // ── Inbound handlers ── - - private void HandleApplyConfigDeploy(ApplyConfigDeploy msg) - { - if (string.IsNullOrEmpty(msg.CentralFetchBaseUrl)) - { - // The direct DeployInstanceCommand cross-cluster wire path was retired. - // This guard is a defensive fallback: skip quietly rather than calling FetchAsync("") - // and logging a spurious error. Reconciliation backstops any missed writes. - _logger.LogDebug( - "No fetch coords for {Instance} (deployment {DeploymentId}) — skipping replicated fetch; T18 reconciliation is the backstop", - msg.InstanceName, msg.DeploymentId); - return; - } - - if (_configFetcher is null) - { - _logger.LogWarning( - "No config fetcher available; cannot apply replicated config for {Instance} (deployment {DeploymentId}) — reconciliation will backstop", - msg.InstanceName, msg.DeploymentId); - return; - } - - _logger.LogInformation( - "Replicating config for {Instance} (deployment {DeploymentId}) — fetching from central", - msg.InstanceName, msg.DeploymentId); - - // Notify-and-fetch: the peer sent only the id, so the standby fetches the config - // itself (off-thread; best-effort fire-and-forget, matching the no-ack replication - // model). The guarded write only overwrites a strictly-older local row. The fetch - // is retried up to ConfigFetchRetryCount times with a fixed delay (UA2) — a transient - // central hiccup no longer defers to the slower reconciliation backstop, which still - // covers a total failure after the last attempt. - _ = FetchWithRetryAsync(); - return; - - async Task FetchWithRetryAsync() - { - for (var attempt = 1; attempt <= _configFetchRetryCount; attempt++) - { - try - { - // Non-null: the outer method returns early when _configFetcher is null. - var json = await _configFetcher!.FetchAsync( - msg.CentralFetchBaseUrl, msg.DeploymentId, msg.FetchToken, CancellationToken.None); - await _storage.StoreDeployedConfigIfNewerAsync( - msg.InstanceName, json, msg.DeploymentId, msg.RevisionHash, msg.IsEnabled); - return; - } - catch (DeploymentConfigFetchException fex) when (fex.IsSuperseded) - { - // A superseded/expired fetch never heals by retrying — a newer deploy - // will replicate its own id. - _logger.LogInformation( - "Skip replicated config for {Instance}: superseded/expired (a newer deploy will replicate)", - msg.InstanceName); - return; - } - catch (Exception ex) - { - if (attempt < _configFetchRetryCount) - { - _logger.LogWarning(ex, - "Replicated config fetch attempt {Attempt}/{Max} failed for {Instance} (deployment {DeploymentId}) — retrying in {Delay}", - attempt, _configFetchRetryCount, msg.InstanceName, msg.DeploymentId, _configFetchRetryDelay); - await Task.Delay(_configFetchRetryDelay); - } - else - { - _logger.LogError(ex, - "Replicated config fetch failed after {Attempts} attempt(s) for {Instance} (deployment {DeploymentId}) — reconciliation will backstop", - _configFetchRetryCount, msg.InstanceName, msg.DeploymentId); - } - } - } - } - } - - private void HandleApplyConfigRemove(ApplyConfigRemove msg) - { - _logger.LogInformation("Applying replicated config remove for {Instance}", msg.InstanceName); - _storage.RemoveDeployedConfigAsync(msg.InstanceName) - .ContinueWith(t => - { - if (t.IsFaulted) - _logger.LogError(t.Exception, "Failed to apply replicated remove for {Instance}", msg.InstanceName); - }); - } - - private void HandleApplyConfigSetEnabled(ApplyConfigSetEnabled msg) - { - _logger.LogInformation("Applying replicated set-enabled={Enabled} for {Instance}", msg.IsEnabled, msg.InstanceName); - _storage.SetInstanceEnabledAsync(msg.InstanceName, msg.IsEnabled) - .ContinueWith(t => - { - if (t.IsFaulted) - _logger.LogError(t.Exception, "Failed to apply replicated set-enabled for {Instance}", msg.InstanceName); - }); - } - - private void HandleApplyArtifacts(ApplyArtifacts msg) - { - var command = msg.Command; - _logger.LogInformation("Applying replicated artifacts, deploymentId={DeploymentId}", command.DeploymentId); - - Task.Run(async () => - { - try - { - if (command.SharedScripts != null) - foreach (var s in command.SharedScripts) - await _storage.StoreSharedScriptAsync(s.Name, s.Code, s.ParameterDefinitions, s.ReturnDefinition); - - if (command.ExternalSystems != null) - foreach (var es in command.ExternalSystems) - await _storage.StoreExternalSystemAsync(es.Name, es.EndpointUrl, es.AuthType, es.AuthConfiguration, es.MethodDefinitionsJson, es.TimeoutSeconds); - - if (command.DatabaseConnections != null) - foreach (var db in command.DatabaseConnections) - await _storage.StoreDatabaseConnectionAsync(db.Name, db.ConnectionString, db.MaxRetries, db.RetryDelay); - - // Notification lists and SMTP - // configuration are central-only and are never persisted on a site. - // Mirror the primary apply path: purge any pre-fix rows (including the - // plaintext SMTP password) instead of writing the command's - // (now-always-null) NotificationLists/SmtpConfigurations. - await _storage.PurgeCentralOnlyNotificationConfigAsync(); - - if (command.DataConnections != null) - foreach (var dc in command.DataConnections) - await _storage.StoreDataConnectionDefinitionAsync(dc.Name, dc.Protocol, dc.PrimaryConfigurationJson, dc.BackupConfigurationJson, dc.FailoverRetryCount); - } - catch (Exception ex) - { - _logger.LogError(ex, "Failed to apply replicated artifacts"); - } - }); - } - - private void HandleApplyStoreAndForward(ApplyStoreAndForward msg) - { - _logger.LogDebug("Applying replicated S&F operation {OpType} for message {Id}", - msg.Operation.OperationType, msg.Operation.MessageId); - - _replicationService.ApplyReplicatedOperationAsync(msg.Operation, _sfStorage) - .ContinueWith(t => - { - if (t.IsFaulted) - _logger.LogError(t.Exception, "Failed to apply replicated S&F operation {Id}", msg.Operation.MessageId); - }); - } - - /// - /// Active-node side of the anti-entropy resync: answers a standby's - /// with a sequence of byte-budgeted - /// s (up to oldest rows). - /// A non-active node ignores the request — only the authoritative node may answer. - /// The snapshot is piped back to Self so chunking + ack bookkeeping stays actor-safe. - /// - private void HandleRequestSfBufferResync(RequestSfBufferResync msg) - { - if (!SafeIsActive()) - { - _logger.LogDebug("Ignoring S&F buffer resync request — this node is not active"); - return; - } - - var replyTo = Sender; - _sfStorage.GetAllMessagesAsync(MaxResyncRows).PipeTo( - Self, - failure: ex => new Status.Failure(ex), - success: result => new SfResyncSnapshotLoaded(replyTo, result.Messages, result.Truncated)); - } - - /// - /// Active-node continuation: the resync snapshot finished loading; chunk it to fit the - /// remoting frame and send the sequenced chunks to the requester (all sharing one - /// resyncId). Task 7 arms the ack-timeout here. - /// - private void HandleSfResyncSnapshotLoaded(SfResyncSnapshotLoaded msg) - { - var chunks = ChunkForRemoting(msg.Messages, MaxResyncChunkBytes, MaxResyncChunkRows); - if (chunks.Count == 0) chunks.Add(new List()); // empty buffer still resyncs (clears the standby) - var resyncId = Guid.NewGuid().ToString("N"); - for (var i = 0; i < chunks.Count; i++) - { - if (chunks[i].Count == 1 && (chunks[i][0].PayloadJson?.Length ?? 0) + 512 > MaxResyncChunkBytes) - _logger.LogWarning( - "Resync row {Id} alone exceeds the chunk budget ({Bytes}B payload); sending solo — it may exceed the remoting frame", - chunks[i][0].Id, chunks[i][0].PayloadJson?.Length ?? 0); - msg.ReplyTo.Tell(new SfBufferSnapshotChunk(resyncId, i + 1, chunks.Count, chunks[i], msg.Truncated), Self); - } - _logger.LogInformation( - "Answered S&F resync request with {Rows} row(s) in {Chunks} chunk(s), resyncId={ResyncId}", - msg.Messages.Count, chunks.Count, resyncId); - // Arm the ack window: absence of an SfBufferResyncAck within it surfaces as a - // Warning + counter (the silent-loss mode N2 flagged). Single-outstanding-resync - // bookkeeping: a new request supersedes by overwriting the id and restarting the timer. - _pendingAckResyncId = resyncId; - Timers.StartSingleTimer(ResyncAckTimerKey, new ResyncAckTimedOut(resyncId), _resyncAckTimeout); - } - - /// - /// Standby-node side of the anti-entropy resync: replaces the local buffer - /// wholesale with the active node's snapshot. Combined with the upsert-based - /// replicated applies (arch review 02), any replicated op that lands - /// after this resync merges cleanly onto the resynced state. An active node - /// ignores a snapshot — it is the source of truth, never a resync target. - /// - private void HandleSfBufferSnapshot(SfBufferSnapshot msg) - { - if (SafeIsActive()) - { - _logger.LogDebug("Ignoring S&F buffer snapshot — this node is active"); - return; - } - - if (msg.Truncated) - { - _logger.LogWarning( - "S&F buffer resync snapshot truncated at {Cap} rows; divergence beyond the cap drains naturally", - MaxResyncRows); - } - - _logger.LogInformation( - "Applying S&F buffer resync snapshot ({Count} rows), replacing local buffer", msg.Messages.Count); - - Task.Run(async () => - { - // Belt-and-braces (N1): re-check at apply time. ReplaceAllAsync discards - // every in-flight row (StoreAndForwardStorage.cs "Never call on an active - // node"); a flip between message receipt and this point must abort. - if (SafeIsActive()) - { - _logger.LogWarning( - "Discarding S&F buffer resync snapshot: this node became active before apply"); - return; - } - await _sfStorage.ReplaceAllAsync(msg.Messages); - }) - .ContinueWith(t => - { - if (t.IsFaulted) - _logger.LogError(t.Exception, "Failed to apply S&F buffer resync snapshot"); - }); - } - - /// - /// Standby-node side of the chunked anti-entropy resync: accumulates the chunks of one - /// ResyncId, and once all have arrived, assembles them in sequence order and - /// replaces the local buffer atomically, then acks. A new ResyncId discards any - /// stale partial assembly (review 02 round 2, N2). An active node ignores chunks. - /// - private void HandleSfBufferSnapshotChunk(SfBufferSnapshotChunk msg) - { - if (SafeIsActive()) - { - _logger.LogDebug("Ignoring S&F resync chunk — this node is active"); - return; - } - - if (_assemblingResyncId != msg.ResyncId) - { - if (_assemblingResyncId != null) - _logger.LogWarning( - "Discarding partial S&F resync assembly {Old} ({Have}/{Want} chunks): a new resync {New} superseded it", - _assemblingResyncId, _assemblingChunks.Count, _assemblingTotalChunks, msg.ResyncId); - _assemblingResyncId = msg.ResyncId; - _assemblingTotalChunks = msg.TotalChunks; - _assemblingTruncated = msg.Truncated; - _assemblingChunks.Clear(); - } - - _assemblingChunks[msg.Sequence] = msg.Messages; - Timers.StartSingleTimer(ResyncAssemblyTimerKey, new ResyncAssemblyTimedOut(msg.ResyncId), _resyncAssemblyTimeout); - - if (_assemblingChunks.Count < _assemblingTotalChunks) - return; - - // Complete: assemble in sequence order and apply atomically. - var assembled = Enumerable.Range(1, _assemblingTotalChunks) - .SelectMany(seq => _assemblingChunks[seq]) - .ToList(); - var resyncId = _assemblingResyncId!; - var truncated = _assemblingTruncated; - _assemblingResyncId = null; - _assemblingChunks.Clear(); - Timers.Cancel(ResyncAssemblyTimerKey); - - if (truncated) - _logger.LogWarning( - "S&F buffer resync snapshot truncated at {Cap} rows; divergence beyond the cap drains naturally", - MaxResyncRows); - _logger.LogInformation( - "Applying chunked S&F resync {ResyncId} ({Count} rows), replacing local buffer", resyncId, assembled.Count); - - // KNOWN, ACCEPTED race (review 02 round 2, N5 — do NOT "fix" this into something - // worse): a replicated Remove sent after the active node read its snapshot but - // before the snapshot's chunks is ordered BEFORE them on the wire (same - // sender/receiver pair), so this apply can re-add the removed row → an orphan - // Pending row that a later failover re-delivers ONCE. Bounded, self-correcting at - // the next resync, and inherent to no-ack replication; a delivered-side dedup or - // op-sequencing scheme would cost far more than one rare duplicate. - var replyTo = Sender; - Task.Run(async () => - { - if (SafeIsActive()) // belt-and-braces, mirrors the monolithic path (T3) - { - _logger.LogWarning("Discarding chunked S&F resync {ResyncId}: node became active before apply", resyncId); - return; - } - await _sfStorage.ReplaceAllAsync(assembled); - replyTo.Tell(new SfBufferResyncAck(resyncId, assembled.Count)); - }) - .ContinueWith(t => - { - if (t.IsFaulted) - _logger.LogError(t.Exception, "Failed to apply chunked S&F resync {ResyncId}", resyncId); - }); - } - - private void HandleResyncAssemblyTimedOut(ResyncAssemblyTimedOut msg) - { - if (_assemblingResyncId != msg.ResyncId) return; // superseded already - _logger.LogWarning( - "S&F resync assembly {ResyncId} timed out with {Have}/{Want} chunks — discarding partial (a lost chunk; the next peer-track retries)", - msg.ResyncId, _assemblingChunks.Count, _assemblingTotalChunks); - ScadaBridgeTelemetry.RecordReplicationFailure(); - _assemblingResyncId = null; - _assemblingChunks.Clear(); - } - - /// Internal: the resync snapshot finished loading; chunk and send to the requester. - internal sealed record SfResyncSnapshotLoaded( - IActorRef ReplyTo, List Messages, bool Truncated); - - /// Internal: a partial chunk assembly for exceeded its window. - internal sealed record ResyncAssemblyTimedOut(string ResyncId); - - /// Internal: the active node's ack window for expired. - internal sealed record ResyncAckTimedOut(string ResyncId); -} - -/// -/// Standby→active: request a full S&F buffer snapshot for anti-entropy resync -/// (sent when a standby (re)tracks a peer). Crosses Akka remoting between the two -/// site nodes; the POCO rides the default serializer. -/// -public sealed record RequestSfBufferResync; - -/// -/// Active→standby: full-buffer snapshot. is true when -/// the active node's buffer exceeded MaxResyncRows (the standby logs a Warning — -/// divergence beyond the cap drains naturally as the active node delivers). Crosses -/// Akka remoting; the message list rides the default serializer. -/// -public sealed record SfBufferSnapshot(List Messages, bool Truncated); - -/// -/// Active→standby: one sequenced chunk of a full-buffer anti-entropy snapshot -/// (review 02 round 2, N2 — the monolithic exceeds Akka -/// remoting's default 128 000-byte frame for any realistic backlog). All chunks of one -/// resync share ; is 1-based up to -/// . Additive message — the legacy monolithic snapshot -/// handler is retained for rolling upgrades. Crosses intra-site Akka remoting (NOT -/// ClusterClient — ClusterClientContractLockTests is intentionally not involved). -/// -public sealed record SfBufferSnapshotChunk( - string ResyncId, int Sequence, int TotalChunks, - List Messages, bool Truncated); - -/// -/// Standby→active: delivery confirmation — the standby assembled all chunks of -/// and applied them atomically ( -/// rows installed). Absence within the ack window is surfaced by the active node -/// (Warning + counter) — the silent-loss mode N2 flagged. -/// -public sealed record SfBufferResyncAck(string ResyncId, int RowCount); diff --git a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Messages/ReplicationMessages.cs b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Messages/ReplicationMessages.cs deleted file mode 100644 index b937cc0d..00000000 --- a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Messages/ReplicationMessages.cs +++ /dev/null @@ -1,44 +0,0 @@ -using ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts; -using ZB.MOM.WW.ScadaBridge.StoreAndForward; - -namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Messages; - -// Outbound messages — sent by local DeploymentManagerActor/S&F service -// to the local SiteReplicationActor for forwarding to the peer node. - -/// Outbound: tell the peer to fetch+apply a deployed instance config by id (notify-and-fetch; no inline config). -public record ReplicateConfigDeploy( - string InstanceName, string DeploymentId, string RevisionHash, bool IsEnabled, - string CentralFetchBaseUrl, string FetchToken); - -/// Outbound: replicate removal of a deployed instance config to the peer node. -public record ReplicateConfigRemove(string InstanceName); - -/// Outbound: replicate an instance enabled/disabled flag change to the peer node. -public record ReplicateConfigSetEnabled(string InstanceName, bool IsEnabled); - -/// Outbound: replicate a system-wide artifact deployment (shared scripts, external systems, etc.) to the peer node. -public record ReplicateArtifacts(DeployArtifactsCommand Command); - -/// Outbound: replicate a store-and-forward buffer mutation (enqueue/dequeue/park/etc.) to the peer node. -public record ReplicateStoreAndForward(ReplicationOperation Operation); - -// Inbound messages — received from the peer's SiteReplicationActor -// and applied to local SQLite storage. - -/// Inbound: peer-replicated config deploy — the standby fetches the config by id and writes it (guarded). -public record ApplyConfigDeploy( - string InstanceName, string DeploymentId, string RevisionHash, bool IsEnabled, - string CentralFetchBaseUrl, string FetchToken); - -/// Inbound: apply peer-replicated removal of a deployed instance config to local SQLite. -public record ApplyConfigRemove(string InstanceName); - -/// Inbound: apply a peer-replicated instance enabled/disabled flag change to local SQLite. -public record ApplyConfigSetEnabled(string InstanceName, bool IsEnabled); - -/// Inbound: apply a peer-replicated system-wide artifact deployment to local SQLite. -public record ApplyArtifacts(DeployArtifactsCommand Command); - -/// Inbound: apply a peer-replicated store-and-forward buffer mutation to the local buffer. -public record ApplyStoreAndForward(ReplicationOperation Operation); diff --git a/src/ZB.MOM.WW.ScadaBridge.StoreAndForward/ReplicationService.cs b/src/ZB.MOM.WW.ScadaBridge.StoreAndForward/ReplicationService.cs deleted file mode 100644 index 60f2b162..00000000 --- a/src/ZB.MOM.WW.ScadaBridge.StoreAndForward/ReplicationService.cs +++ /dev/null @@ -1,194 +0,0 @@ -using Microsoft.Extensions.Logging; -using ZB.MOM.WW.ScadaBridge.Commons.Observability; -using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; - -namespace ZB.MOM.WW.ScadaBridge.StoreAndForward; - -/// -/// Async replication of buffer operations to standby node. -/// -/// - Forwards add/remove/park operations to standby via a replication handler. -/// - No ack wait (fire-and-forget per design). -/// - Standby applies operations to its own SQLite. -/// - On failover, standby resumes delivery from its replicated state. -/// -public class ReplicationService -{ - private readonly StoreAndForwardOptions _options; - private readonly ILogger _logger; - private Func? _replicationHandler; - - /// Initializes a new instance of . - /// Store-and-forward configuration options. - /// Logger instance. - public ReplicationService( - StoreAndForwardOptions options, - ILogger logger) - { - _options = options; - _logger = logger; - } - - /// - /// Sets the handler for forwarding replication operations to the standby node. - /// Typically wraps Akka Tell to the standby's replication actor. - /// - /// The async delegate that forwards each replication operation to the standby. - public void SetReplicationHandler(Func handler) - { - _replicationHandler = handler; - } - - /// - /// Replicates an enqueue operation to standby (fire-and-forget). - /// - /// The message that was enqueued on the active node. - public void ReplicateEnqueue(StoreAndForwardMessage message) - { - if (!_options.ReplicationEnabled || _replicationHandler == null) return; - - FireAndForget(new ReplicationOperation( - ReplicationOperationType.Add, - message.Id, - message)); - } - - /// - /// Replicates a remove operation to standby (fire-and-forget). - /// - /// The identifier of the message to remove from the standby buffer. - public void ReplicateRemove(string messageId) - { - if (!_options.ReplicationEnabled || _replicationHandler == null) return; - - FireAndForget(new ReplicationOperation( - ReplicationOperationType.Remove, - messageId, - null)); - } - - /// - /// Replicates a park operation to standby (fire-and-forget). - /// - /// The message that was parked on the active node. - public void ReplicatePark(StoreAndForwardMessage message) - { - if (!_options.ReplicationEnabled || _replicationHandler == null) return; - - FireAndForget(new ReplicationOperation( - ReplicationOperationType.Park, - message.Id, - message)); - } - - /// - /// Replicates an operator-initiated requeue (a parked - /// message moved back to the pending queue) to standby (fire-and-forget). The - /// carried message reflects the active node's post-requeue state (Pending, - /// retry_count = 0) so the standby's copy can be brought into sync. - /// - /// The message in its post-requeue (Pending, retry_count=0) state. - public void ReplicateRequeue(StoreAndForwardMessage message) - { - if (!_options.ReplicationEnabled || _replicationHandler == null) return; - - FireAndForget(new ReplicationOperation( - ReplicationOperationType.Requeue, - message.Id, - message)); - } - - /// - /// Applies a replicated operation received from the active node. - /// Used by the standby node to keep its SQLite in sync. - /// - /// Add/Park/Requeue are applied as upserts (), - /// not blind INSERT/UPDATE: the full message rides in every one of those operations, - /// so a Park/Requeue whose original Add was lost (fire-and-forget replication is - /// best-effort) self-heals by materialising the row, and a duplicate Add (e.g. - /// re-issued by an anti-entropy resync) applies newest-wins instead of throwing a - /// primary-key violation. Remove is a plain delete. - /// - /// The replication operation to apply. - /// The standby node's store-and-forward storage to update. - /// A task representing the asynchronous apply operation. - public async Task ApplyReplicatedOperationAsync( - ReplicationOperation operation, - StoreAndForwardStorage storage) - { - switch (operation.OperationType) - { - case ReplicationOperationType.Add when operation.Message != null: - await storage.UpsertMessageAsync(operation.Message); - break; - - case ReplicationOperationType.Remove: - await storage.RemoveMessageAsync(operation.MessageId); - break; - - case ReplicationOperationType.Park when operation.Message != null: - operation.Message.Status = StoreAndForwardMessageStatus.Parked; - await storage.UpsertMessageAsync(operation.Message); - break; - - case ReplicationOperationType.Requeue when operation.Message != null: - operation.Message.Status = StoreAndForwardMessageStatus.Pending; - operation.Message.RetryCount = 0; - await storage.UpsertMessageAsync(operation.Message); - break; - } - } - - private void FireAndForget(ReplicationOperation operation) - { - // Invoked inline, NOT via Task.Run: the handler is a non-blocking Akka - // Tell, and thread-pool hand-off destroyed Add/Remove ordering for the - // same message id (arch review 02). Inline invocation preserves issue - // order; Akka's per-sender/receiver guarantee preserves it on the wire. - try - { - var task = _replicationHandler!.Invoke(operation); - if (!task.IsCompletedSuccessfully) - { - task.ContinueWith(t => - { - ScadaBridgeTelemetry.RecordReplicationFailure(); - _logger.LogWarning(t.Exception, - "Replication of {OpType} for message {MessageId} failed (best-effort); standby buffer may be diverging", - operation.OperationType, operation.MessageId); - }, - TaskContinuationOptions.OnlyOnFaulted); - } - } - catch (Exception ex) - { - ScadaBridgeTelemetry.RecordReplicationFailure(); - _logger.LogWarning(ex, - "Replication of {OpType} for message {MessageId} failed (best-effort); standby buffer may be diverging", - operation.OperationType, operation.MessageId); - } - } -} - -/// -/// Represents a buffer operation to be replicated to standby. -/// -public record ReplicationOperation( - ReplicationOperationType OperationType, - string MessageId, - StoreAndForwardMessage? Message); - -/// -/// Types of buffer operations that are replicated. -/// -public enum ReplicationOperationType -{ - Add, - Remove, - Park, - /// - /// An operator moved a parked message back to the pending - /// queue. The standby resets its matching row to Pending with retry_count = 0. - /// - Requeue -} diff --git a/src/ZB.MOM.WW.ScadaBridge.StoreAndForward/ServiceCollectionExtensions.cs b/src/ZB.MOM.WW.ScadaBridge.StoreAndForward/ServiceCollectionExtensions.cs index a5c2e513..314b5522 100644 --- a/src/ZB.MOM.WW.ScadaBridge.StoreAndForward/ServiceCollectionExtensions.cs +++ b/src/ZB.MOM.WW.ScadaBridge.StoreAndForward/ServiceCollectionExtensions.cs @@ -28,7 +28,6 @@ public static class ServiceCollectionExtensions var storage = sp.GetRequiredService(); var options = sp.GetRequiredService>().Value; var logger = sp.GetRequiredService>(); - var replication = sp.GetRequiredService(); // Wire the cached-call lifecycle // observer + site identity through DI so the S&F retry loop emits // per-attempt + terminal telemetry under the same TrackedOperationId @@ -53,19 +52,11 @@ public static class ServiceCollectionExtensions storage, options, logger, - replication, cachedCallObserver, siteId, siteEventLogger); }); - services.AddSingleton(sp => - { - var options = sp.GetRequiredService>().Value; - var logger = sp.GetRequiredService>(); - return new ReplicationService(options, logger); - }); - return services; } diff --git a/src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardService.cs b/src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardService.cs index 2b8dda17..53a6ad2f 100644 --- a/src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardService.cs +++ b/src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardService.cs @@ -36,7 +36,6 @@ public class StoreAndForwardService { private readonly StoreAndForwardStorage _storage; private readonly StoreAndForwardOptions _options; - private readonly ReplicationService? _replication; private readonly ILogger _logger; /// /// Site-side observer notified @@ -107,7 +106,7 @@ public class StoreAndForwardService /// null when no sweep is currently running. Captured when the timer /// callback starts a sweep so can wait for it to /// finish before the host disposes downstream dependencies - /// (, ) that the sweep is + /// () that the sweep is /// still touching. Written from the timer thread and from /// , so reads are synchronised via the /// APIs. @@ -227,7 +226,6 @@ public class StoreAndForwardService /// The storage backend for buffered messages. /// Configuration options. /// Logger instance. - /// Optional replication service for standby synchronization. /// Optional observer for cached call lifecycle events. /// The site identifier this service belongs to. /// @@ -240,7 +238,6 @@ public class StoreAndForwardService StoreAndForwardStorage storage, StoreAndForwardOptions options, ILogger logger, - ReplicationService? replication = null, ICachedCallLifecycleObserver? cachedCallObserver = null, string siteId = "", ISiteEventLogger? siteEventLogger = null) @@ -248,7 +245,6 @@ public class StoreAndForwardService _storage = storage; _options = options; _logger = logger; - _replication = replication; _cachedCallObserver = cachedCallObserver; _siteId = string.IsNullOrWhiteSpace(siteId) ? UnknownSiteSentinel : siteId; _siteEventLogger = siteEventLogger; @@ -651,7 +647,6 @@ public class StoreAndForwardService private async Task BufferAsync(StoreAndForwardMessage message) { await _storage.EnqueueAsync(message); - _replication?.ReplicateEnqueue(message); Interlocked.Increment(ref _bufferedCount); } @@ -802,7 +797,6 @@ public class StoreAndForwardService if (success) { await _storage.RemoveMessageAsync(message.Id); - _replication?.ReplicateRemove(message.Id); Interlocked.Decrement(ref _bufferedCount); RaiseActivity("Delivered", message.Category, $"Delivered to {message.Target} after {message.RetryCount} retries"); @@ -833,7 +827,6 @@ public class StoreAndForwardService return RetryOutcome.Skipped; } Interlocked.Decrement(ref _bufferedCount); - _replication?.ReplicatePark(message); RaiseActivity("Parked", message.Category, $"Permanent failure for {message.Target}: handler returned false"); @@ -869,7 +862,6 @@ public class StoreAndForwardService return RetryOutcome.Skipped; } Interlocked.Decrement(ref _bufferedCount); - _replication?.ReplicatePark(message); RaiseActivity("Parked", message.Category, $"Max retries ({message.MaxRetries}) reached for {message.Target}"); _logger.LogWarning( @@ -1077,17 +1069,12 @@ public class StoreAndForwardService /// /// Retries a parked message (moves back to pending queue). /// - /// An operator requeue is a buffer state change and is - /// replicated to the standby (as a ) - /// so a failover preserves the operator's retry intent. + /// An operator requeue is a buffer state change, and the peer picks it up because + /// sf_messages is a replicated table: the row update is captured and shipped + /// like any other write, so a failover preserves the operator's retry intent. This + /// used to be an explicit Requeue operation sent to the standby. /// The activity-log entry carries the message's true /// category rather than a hard-coded one. - /// The parked row is captured before the local - /// requeue write rather than re-read after it, so a concurrent - /// RemoveMessageAsync or DiscardParkedMessageAsync running - /// between the two storage calls cannot leave the standby in Parked - /// while the active node has already requeued — we always have the row in - /// hand for the Requeue replication. /// /// The identifier of the message to retry. /// True if successfully retried, false otherwise. @@ -1117,7 +1104,6 @@ public class StoreAndForwardService captured.RetryCount = 0; captured.LastError = null; captured.LastAttemptAt = null; - _replication?.ReplicateRequeue(captured); RaiseActivity("Retry", captured.Category, $"Parked message {messageId} moved back to queue"); @@ -1127,9 +1113,10 @@ public class StoreAndForwardService /// /// Permanently discards a parked message. /// - /// An operator discard is a buffer removal and is replicated - /// to the standby (as a ) so the - /// discarded message does not reappear after a failover. + /// An operator discard is a buffer removal. The delete is captured as a tombstone on + /// the replicated sf_messages table and carries the later HLC, so the discarded + /// message cannot reappear after a failover regardless of arrival order. This used to + /// be an explicit Remove operation sent to the standby. /// The activity-log entry carries the message's true /// category rather than a hard-coded one. /// @@ -1143,7 +1130,6 @@ public class StoreAndForwardService var success = await _storage.DiscardParkedMessageAsync(messageId); if (success) { - _replication?.ReplicateRemove(messageId); RaiseActivity("Discard", message?.Category ?? StoreAndForwardCategory.ExternalSystem, $"Parked message {messageId} discarded"); } diff --git a/src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardStorage.cs b/src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardStorage.cs index fbe87cc6..54859796 100644 --- a/src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardStorage.cs +++ b/src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardStorage.cs @@ -65,7 +65,7 @@ public class StoreAndForwardStorage /// /// INSERT statement for a full message row. Shared by - /// and ; bind with + /// and ; bind with /// so the column list and the parameters never drift apart. /// private const string InsertMessageSql = @" @@ -151,39 +151,6 @@ public class StoreAndForwardStorage return (rows.Take(limit).ToList(), truncated); } - /// - /// Atomically replaces the entire buffer with in a - /// single transaction (delete-all then insert-all). Standby-side anti-entropy - /// apply: a peer-join resync overwrites the standby's divergent copy with the - /// active node's authoritative snapshot. Never call on an active node — - /// it discards every in-flight row. - /// - /// The full buffer snapshot to install. - /// A task that represents the asynchronous operation. - public async Task ReplaceAllAsync(IReadOnlyList messages) - { - await using var connection = OpenConnection(); - await using var transaction = (SqliteTransaction)await connection.BeginTransactionAsync(); - - await using (var deleteCmd = connection.CreateCommand()) - { - deleteCmd.Transaction = transaction; - deleteCmd.CommandText = "DELETE FROM sf_messages"; - await deleteCmd.ExecuteNonQueryAsync(); - } - - foreach (var message in messages) - { - await using var insertCmd = connection.CreateCommand(); - insertCmd.Transaction = transaction; - insertCmd.CommandText = InsertMessageSql; - BindMessageParameters(insertCmd, message); - await insertCmd.ExecuteNonQueryAsync(); - } - - await transaction.CommitAsync(); - } - /// /// Inserts a message, or updates every mutable column in place if a row with the /// same id already exists (ON CONFLICT(id) DO UPDATE). Used by the standby diff --git a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CompositionRootTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CompositionRootTests.cs index 488f186f..59d4bf31 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CompositionRootTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CompositionRootTests.cs @@ -471,7 +471,6 @@ public class SiteCompositionRootTests : IDisposable new object[] { typeof(IDataConnectionFactory) }, new object[] { typeof(StoreAndForwardStorage) }, new object[] { typeof(StoreAndForwardService) }, - new object[] { typeof(ReplicationService) }, new object[] { typeof(ISiteEventLogger) }, new object[] { typeof(IEventLogQueryService) }, new object[] { typeof(ISiteIdentityProvider) }, diff --git a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbWiringTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbWiringTests.cs index f9e8e3eb..4340a148 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbWiringTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbWiringTests.cs @@ -243,22 +243,65 @@ public class SiteLocalDbWiringTests : IDisposable } [Fact] - public void Site_LocalDb_DoesNotYetRegisterThePhase2Tables() + public void Site_LocalDb_RegistersExactlyTheTenReplicatedTables() { - // The "not yet" is the assertion that matters, and it is why this is an EXACTLY - // check rather than a Contains. Until the Task 14 cutover, the bespoke - // SiteReplicationActor and the StoreAndForward ReplicationService still own these - // tables. Registering them early would run two replicators over the same rows and - // hide a defect in either one behind the other's writes. + // The Phase 2 cutover. This is an EXACTLY check rather than a Contains in both + // directions, and both directions are load-bearing. // - // When Task 14 lands, this test does not get deleted — it gets inverted. + // Too few: the table silently stops replicating. Storage still works, every other + // test still passes, and the pair just quietly diverges — the failure mode Phase 1 + // existed to end. + // + // Too many: notification_lists and smtp_configurations must NOT be here. They are + // permanently empty by design, and registering them would open a standing + // replication channel whose only historical payload was plaintext SMTP passwords. + // A Contains-based test would never catch that. var db = _host.Services.GetRequiredService(); Assert.Equal( - ["OperationTracking", "site_events"], + [ + // Ordinal order: '_' (0x5F) sorts before 'b' (0x62), so + // data_connection_definitions precedes database_connections. + "OperationTracking", "data_connection_definitions", "database_connections", + "deployed_configurations", "external_systems", "native_alarm_state", + "sf_messages", "shared_scripts", "site_events", "static_attribute_overrides", + ], db.ReplicatedTables.Keys.OrderBy(k => k, StringComparer.Ordinal).ToArray()); } + [Fact] + public void Site_LocalDb_DoesNotReplicateTheCentralOnlyNotificationTables() + { + // Stated separately from the exact-set test above because this one is a security + // property, not a wiring property, and deserves to fail with its own name. The + // tables exist (SiteStorageSchema creates them) — they simply must never be + // captured. See SiteLocalDbSetup.OnReady for the rationale. + var db = _host.Services.GetRequiredService(); + + Assert.DoesNotContain("notification_lists", db.ReplicatedTables.Keys); + Assert.DoesNotContain("smtp_configurations", db.ReplicatedTables.Keys); + } + + [Fact] + public void Site_LocalDb_ReplicatedPhase2TablesHaveTheirExpectedPrimaryKeys() + { + // RegisterReplicated refuses a table with no explicit PK, so reaching these + // assertions proves the DDL ran before registration. The composite keys are the + // interesting ones: LWW conflict resolution keys on the FULL PK, so a wrong or + // truncated key set would silently collapse distinct rows into one. + var db = _host.Services.GetRequiredService(); + + Assert.Equal(["id"], db.ReplicatedTables["sf_messages"].PkColumns); + Assert.Equal( + ["instance_unique_name"], db.ReplicatedTables["deployed_configurations"].PkColumns); + Assert.Equal( + ["instance_unique_name", "attribute_name"], + db.ReplicatedTables["static_attribute_overrides"].PkColumns); + Assert.Equal( + ["instance_unique_name", "source_canonical_name", "source_reference"], + db.ReplicatedTables["native_alarm_state"].PkColumns); + } + private static HashSet TableNames(ILocalDb db) { using var connection = db.CreateConnection(); diff --git a/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/SfBufferResyncPredicateTests.cs b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/SfBufferResyncPredicateTests.cs deleted file mode 100644 index 9e9f3cb2..00000000 --- a/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/SfBufferResyncPredicateTests.cs +++ /dev/null @@ -1,122 +0,0 @@ -using Akka.Actor; -using Microsoft.Extensions.Logging.Abstractions; -using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors; -using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; -using ZB.MOM.WW.ScadaBridge.StoreAndForward; -using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; -using ZB.MOM.WW.ScadaBridge.TestSupport; - -namespace ZB.MOM.WW.ScadaBridge.IntegrationTests.Cluster; - -/// -/// N1 regression (review 02 round 2, Critical): the resync authority must use the same -/// oldest-Up predicate as the S&F delivery gate. Divergence scenario = the delivering node -/// is OLDEST but not LEADER (leader = lowest address), the exact state a rolling restart of -/// the lower-address node produces. Pre-fix the delivering node requests a resync from the -/// stale peer and ReplaceAllAsync wipes its live buffer. -/// -public class SfBufferResyncPredicateTests -{ - [Fact] - public async Task OldestButNotLeaderNode_KeepsItsBuffer_AndSeedsTheJoiner() - { - // Two explicit ports, deliberately assigned so the FIRST-started (oldest, - // delivering) node has the HIGHER address → the second node is cluster leader. - var p1 = TwoNodeClusterFixture.GetFreeTcpPort(); - var p2 = TwoNodeClusterFixture.GetFreeTcpPort(); - var (portHigh, portLow) = p1 > p2 ? (p1, p2) : (p2, p1); - - var fixture = await TwoNodeClusterFixture.StartAsync( - role: "site-int", portA: portHigh, portB: portLow); - - // The S&F stores AND SiteStorageService take an ILocalDb (LocalDb has no in-memory - // mode), so each node gets its own temp-file local databases. They are disposed - // AFTER the cluster is shut down — the actors hold connections while the systems - // are alive — and only then are the files (plus their WAL sidecars) deleted. - var localDbs = new List(); - - try - { - // Real S&F storage + replication actor per node, production default predicate - // (no isActiveOverride) — the exact wiring under test. - var (storageOldest, _, sfDbOldest, siteDbOldest) = - await CreateReplicationActorAsync(fixture.NodeA, "oldest"); - localDbs.Add(sfDbOldest); - localDbs.Add(siteDbOldest); - var (storageJoiner, _, sfDbJoiner, siteDbJoiner) = - await CreateReplicationActorAsync(fixture.NodeB, "joiner"); - localDbs.Add(sfDbJoiner); - localDbs.Add(siteDbJoiner); - - // The delivering (oldest) node has a live buffered row the standby never saw. - await storageOldest.EnqueueAsync(NewMessage("live-row")); - - // Trigger peer (re)tracking on both sides: each actor got InitialStateAsSnapshot - // in PreStart, but the enqueue raced it — re-deliver via a fresh MemberUp is not - // needed; OnPeerTracked already fired on join. The resync exchange is async: - // wait until the JOINER holds the row (proves the snapshot flowed oldest→joiner, - // the correct direction). Pre-fix this times out (the joiner, as leader, never - // requests) AND the oldest node's row is deleted by the stale wipe. - await AwaitAsync(async () => await storageJoiner.GetMessageByIdAsync("live-row") != null, - TimeSpan.FromSeconds(20), - "joiner never received the resync snapshot (resync ran in the wrong direction)"); - - // And the delivering node's buffer is untouched — the N1 wipe assertion. - Assert.NotNull(await storageOldest.GetMessageByIdAsync("live-row")); - } - finally - { - await fixture.DisposeAsync(); - - foreach (var localDb in localDbs) - { - var path = localDb.Path; - localDb.Dispose(); - TestLocalDb.DeleteFiles(path); - } - } - } - - private static async Task<( - StoreAndForwardStorage Storage, IActorRef Actor, TestLocalDb SfLocalDb, TestLocalDb SiteLocalDb)> - CreateReplicationActorAsync(ActorSystem node, string tag) - { - var sfLocalDb = TestLocalDb.CreateTemp($"sf-resync-{tag}"); - var sfStorage = new StoreAndForwardStorage(sfLocalDb.Db, - NullLogger.Instance); - await sfStorage.InitializeAsync(); - var siteLocalDb = TestLocalDb.CreateTemp($"site-resync-{tag}"); - var siteStorage = new SiteStorageService(siteLocalDb.Db, - NullLogger.Instance); - var replicationService = new ReplicationService( - new StoreAndForwardOptions(), NullLogger.Instance); - // Name MUST be "site-replication" — SendToPeer targets /user/site-replication. - var actor = node.ActorOf(Props.Create(() => new SiteReplicationActor( - siteStorage, sfStorage, replicationService, "site-int", - NullLogger.Instance, null, null, null, null)), - "site-replication"); - return (sfStorage, actor, sfLocalDb, siteLocalDb); - } - - private static StoreAndForwardMessage NewMessage(string id) => new() - { - Id = id, - Category = StoreAndForwardCategory.Notification, - Target = "central", - PayloadJson = "{}", - CreatedAt = DateTimeOffset.UtcNow, - Status = StoreAndForwardMessageStatus.Pending, - MaxRetries = 0, - }; - - private static async Task AwaitAsync(Func> condition, TimeSpan timeout, string why) - { - var deadline = DateTime.UtcNow + timeout; - while (DateTime.UtcNow < deadline) - { - if (await condition()) return; - await Task.Delay(250); - } - throw new TimeoutException(why); - } -} diff --git a/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbSitePairHarness.cs b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbSitePairHarness.cs index dc37eefb..0bc73e80 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbSitePairHarness.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbSitePairHarness.cs @@ -129,40 +129,14 @@ public abstract class LocalDbSitePairHarness : IAsyncLifetime .Build(); return new ServiceCollection() - .AddZbLocalDb(config, db => - { - SiteLocalDbSetup.OnReady(db, config); - RegisterPhase2TablesUntilCutover(db); - }) + .AddZbLocalDb(config, db => SiteLocalDbSetup.OnReady(db, config)) .BuildServiceProvider(); } /// - /// Registers the Phase 2 tables for capture, which production OnReady does not do - /// until the Task 14 cutover. - /// - /// - /// Delete this method at Task 14, along with its call above. Until the cutover the - /// bespoke SiteReplicationActor and ReplicationService still own these - /// tables, so OnReady deliberately leaves them unregistered — but the convergence - /// specifications the cutover has to satisfy need capture triggers to mean anything. The - /// tables are empty at this point, so registering here captures nothing retroactively; - /// the ordering guarantee under test is unaffected. - /// - /// After Task 14 these registrations come from OnReady itself, and leaving this - /// method behind would mask a cutover that forgot one — the whole point of these tests. - /// - /// - private static void RegisterPhase2TablesUntilCutover(ILocalDb db) - { - foreach (var table in Phase2ReplicatedTables) - db.RegisterReplicated(table); - } - - /// - /// The eight tables Task 14 registers. Listed literally rather than derived from - /// production code, so that a cutover which registers the wrong set fails these tests - /// instead of agreeing with itself. + /// The eight tables the Phase 2 cutover registers. Listed literally rather than derived + /// from production code, so a registration that drifts fails these tests instead of + /// agreeing with itself. /// /// /// notification_lists and smtp_configurations are absent by design. They diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs index f898add5..4f448aa0 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs @@ -67,12 +67,14 @@ public class DeploymentManagerActorTests : TestKit, IDisposable null, // no stream manager in tests options, NullLogger.Instance, - null, - null, - null, - serviceProvider, - null, - configFetcher))); + // Named from here on. These trailing parameters are all optional and several + // share a type, so a positional list silently binds the wrong argument when the + // signature changes — which is exactly what removing replicationActor did. + dclManager: null, + healthCollector: null, + serviceProvider: serviceProvider, + loggerFactory: null, + configFetcher: configFetcher))); } private static string MakeConfigJson(string instanceName) @@ -301,8 +303,11 @@ public class DeploymentManagerActorTests : TestKit, IDisposable var dm = ActorOf(Props.Create(() => new DeploymentManagerActor( _storage, _compilationService, _sharedScriptLibrary, null, new SiteRuntimeOptions(), NullLogger.Instance, - null, null, null, null, null, null, - TimeSpan.FromMilliseconds(200), loader))); + // dclManager, healthCollector, serviceProvider, loggerFactory, configFetcher. + // Props.Create builds an expression tree, which rejects named arguments that + // are out of position, so the optional tail has to be padded positionally. + null, null, null, null, null, + startupLoadRetryInterval: TimeSpan.FromMilliseconds(200), configLoader: loader))); AwaitAssert(() => { @@ -892,13 +897,14 @@ public class DeploymentManagerActorTests : TestKit, IDisposable // Security cleanup. notification_lists and smtp_configurations can hold plaintext // SMTP passwords written by a pre-2026-07-10 build, and the ACTIVE node's artifact // apply is what clears them (DeploymentManagerActor.HandleDeployArtifacts). The - // standby's copy of this call lives in SiteReplicationActor and dies with it at - // Task 15, which makes this call site the ONLY remaining one. + // standby used to hold a second copy of this call in SiteReplicationActor; LocalDb + // Phase 2 deleted that actor, so this is now the ONLY call site that keeps the + // tables empty. // - // Nothing pins it today: ArtifactStorageTests covers the storage method, not the - // actor's call to it, so Task 16's edits to this actor could drop the call and every - // suite would stay green. That is precisely the kind of silent security regression - // this test exists to prevent — verified red-first by commenting out the call. + // ArtifactStorageTests covers the storage method, not the actor's call to it, so + // without this test the call could be dropped and every suite would stay green. + // That is precisely the kind of silent security regression it exists to prevent — + // verified red-first by commenting out the call. await SeedCentralOnlyRowsAsync(); Assert.Equal(1, await RowCountAsync("notification_lists")); Assert.Equal(1, await RowCountAsync("smtp_configurations")); diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerCertReconcileTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerCertReconcileTests.cs index 7ea67d09..11c6a0b4 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerCertReconcileTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerCertReconcileTests.cs @@ -80,8 +80,7 @@ akka { private IActorRef CreateDeploymentManager() => ActorOf(Props.Create(() => new DeploymentManagerActor( _storage, _compilationService, _sharedScriptLibrary, - null, new SiteRuntimeOptions(), NullLogger.Instance, - null, null, null, null, null, null, null, null))); + null, new SiteRuntimeOptions(), NullLogger.Instance))); [Fact] public void SiteNodeJoined_PushesLocalTrustedCertsToJoinedNode() diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerLoggerFactoryTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerLoggerFactoryTests.cs index 087b7f0d..fcf6da18 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerLoggerFactoryTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerLoggerFactoryTests.cs @@ -103,11 +103,10 @@ public class DeploymentManagerLoggerFactoryTests : TestKit, IDisposable null, new SiteRuntimeOptions { StartupBatchSize = 100, StartupBatchDelayMs = 5 }, NullLogger.Instance, - null, - null, - null, - null, - loggerFactory))); + // dclManager, healthCollector, serviceProvider — padded positionally because + // Props.Create is an expression tree and rejects out-of-position named args. + null, null, null, + loggerFactory: loggerFactory))); // Allow async startup (load configs + staggered creation). await Task.Delay(2000); diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerRedeployTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerRedeployTests.cs index a853d061..663243a3 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerRedeployTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerRedeployTests.cs @@ -61,10 +61,11 @@ public class DeploymentManagerRedeployTests : TestKit, IDisposable null, new SiteRuntimeOptions(), NullLogger.Instance, + // dclManager — padded positionally because Props.Create is an expression tree + // and rejects out-of-position named args. null, - null, - healthCollector, - serviceProvider))); + healthCollector: healthCollector, + serviceProvider: serviceProvider))); } /// diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/SiteReplicationActorTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/SiteReplicationActorTests.cs deleted file mode 100644 index 0dae7d19..00000000 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/SiteReplicationActorTests.cs +++ /dev/null @@ -1,568 +0,0 @@ -using System.Collections.Concurrent; -using System.Diagnostics.Metrics; -using Akka.Actor; -using Akka.TestKit.Xunit2; -using Microsoft.Extensions.Logging; -using Microsoft.Extensions.Logging.Abstractions; -using ZB.MOM.WW.ScadaBridge.SiteRuntime; -using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors; -using ZB.MOM.WW.ScadaBridge.SiteRuntime.Deployment; -using ZB.MOM.WW.ScadaBridge.SiteRuntime.Messages; -using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; -using ZB.MOM.WW.ScadaBridge.StoreAndForward; -using ZB.MOM.WW.ScadaBridge.Commons.Observability; -using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; -using ZB.MOM.WW.ScadaBridge.TestSupport; - -namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors; - -/// -/// Tests for 's notify-and-fetch config replication: -/// the active node now replicates an id-only (no inline -/// config JSON — killing the intra-site 128 KB frame trap), and the standby fetches the -/// config from central over HTTP and writes it with the older-write guard. -/// -public class SiteReplicationActorTests : TestKit, IDisposable -{ - // Cluster provider is required because SiteReplicationActor calls Cluster.Get in its ctor - // and subscribes to cluster events in PreStart. We use the in-memory TestTransport (not - // dot-netty) so no real socket is bound and no DNS lookup happens — the actor only needs - // the cluster extension to load; these tests never form a real two-node cluster. - private const string ClusterConfig = @" -akka { - actor { provider = cluster } - remote { - enabled-transports = [""akka.remote.test""] - test { - transport-class = ""Akka.Remote.Transport.TestTransport, Akka.Remote"" - applied-adapters = [] - registry-key = site-repl-test - local-address = ""test://site-repl@localhost:1"" - maximum-payload-bytes = 128000b - scheme-identifier = test - } - } - cluster { roles = [""site-test""] } - loglevel = WARNING -}"; - - private const string SiteRole = "site-test"; - - private readonly SiteStorageService _storage; - private readonly TestLocalDb _siteLocalDb; - private readonly TestLocalDb _sfLocalDb; - private readonly StoreAndForwardStorage _sfStorage; - private readonly ReplicationService _replicationService; - private readonly string _sfDbFile; - - public SiteReplicationActorTests() : base(ClusterConfig, "site-repl") - { - _sfDbFile = Path.Combine(Path.GetTempPath(), $"site-repl-sf-{Guid.NewGuid():N}.db"); - - // SiteStorageService takes an ILocalDb now; LocalDb has no in-memory mode, so the - // site store gets its own temp-file database alongside the S&F one. - _siteLocalDb = TestLocalDb.CreateTemp("site-repl-test"); - _storage = new SiteStorageService( - _siteLocalDb.Db, NullLogger.Instance); - _storage.InitializeAsync().GetAwaiter().GetResult(); - - _sfLocalDb = TestLocalDb.Create(_sfDbFile); - _sfStorage = new StoreAndForwardStorage( - _sfLocalDb.Db, NullLogger.Instance); - _sfStorage.InitializeAsync().GetAwaiter().GetResult(); - - _replicationService = new ReplicationService( - new StoreAndForwardOptions(), NullLogger.Instance); - } - - void IDisposable.Dispose() - { - Shutdown(); - // The master connection anchors the WAL — dispose before deleting. - var siteDbPath = _siteLocalDb.Path; - _siteLocalDb.Dispose(); - _sfLocalDb.Dispose(); - TestLocalDb.DeleteFiles(siteDbPath); - TestLocalDb.DeleteFiles(_sfDbFile); - } - - private IActorRef CreateReplicationActor(IDeploymentConfigFetcher fetcher) => - ActorOf(Props.Create(() => new SiteReplicationActor( - _storage, _sfStorage, _replicationService, SiteRole, - NullLogger.Instance, fetcher))); - - private IActorRef CreateReplicationActor( - IDeploymentConfigFetcher fetcher, SiteRuntimeOptions options, TimeSpan retryDelay) => - ActorOf(Props.Create(() => new SiteReplicationActor( - _storage, _sfStorage, _replicationService, SiteRole, - NullLogger.Instance, fetcher, null, options, retryDelay))); - - [Fact] - public async Task ReplicatedFetch_RetriesUpToConfigFetchRetryCount() - { - // The first two fetches fail transiently; the third succeeds. With - // ConfigFetchRetryCount = 3 the standby must retry to the third attempt and - // then guarded-write the fetched config (a short retry delay keeps the test fast). - var attempts = 0; - var fetcher = new FakeConfigFetcher(_ => - Interlocked.Increment(ref attempts) < 3 - ? Task.FromException(new InvalidOperationException("central hiccup")) - : Task.FromResult("{\"instanceUniqueName\":\"RetryPump\"}")); - var actor = CreateReplicationActor( - fetcher, new SiteRuntimeOptions { ConfigFetchRetryCount = 3 }, - TimeSpan.FromMilliseconds(50)); - - actor.Tell(new ApplyConfigDeploy( - "RetryPump", "dep-r1", "sha256:r1", true, - "http://central:9000", "tok-r1")); - - await AwaitAssertAsync(async () => - { - Assert.Equal(3, Volatile.Read(ref attempts)); - var configs = await _storage.GetAllDeployedConfigsAsync(); - Assert.Single(configs, c => c.InstanceUniqueName == "RetryPump"); - }, TimeSpan.FromSeconds(10)); - } - - [Fact] - public async Task ApplyConfigDeploy_StandbyFetchesConfigAndGuardedWrites() - { - // The standby receives an id-only ApplyConfigDeploy; it fetches the config from - // central using the message's coords, then guarded-writes the fetched config. - const string configJson = "{\"instanceUniqueName\":\"Pump1\"}"; - var fetcher = new FakeConfigFetcher(_ => Task.FromResult(configJson)); - var actor = CreateReplicationActor(fetcher); - - actor.Tell(new ApplyConfigDeploy( - "Pump1", "dep-100", "sha256:abc", true, - "http://central:9000", "tok-xyz")); - - // The continuation runs off-thread; await the guarded write landing. - await AwaitAssertAsync(async () => - { - var configs = await _storage.GetAllDeployedConfigsAsync(); - var row = Assert.Single(configs, c => c.InstanceUniqueName == "Pump1"); - Assert.Equal(configJson, row.ConfigJson); - Assert.Equal("dep-100", row.DeploymentId); - Assert.Equal("sha256:abc", row.RevisionHash); - Assert.True(row.IsEnabled); - }, TimeSpan.FromSeconds(5)); - - // The fetcher was called with the message's coords. - var call = Assert.Single(fetcher.Calls); - Assert.Equal("http://central:9000", call.BaseUrl); - Assert.Equal("dep-100", call.DeploymentId); - Assert.Equal("tok-xyz", call.Token); - } - - [Fact] - public async Task ApplyConfigDeploy_Superseded404_SkipsWriteAndActorSurvives() - { - // A 404 (superseded/expired) surfaces as DeploymentConfigFetchException{IsSuperseded}. - // The standby must skip the write, observe the exception (no crash), and stay alive. - var fetcher = new FakeConfigFetcher(_ => - Task.FromException( - new DeploymentConfigFetchException("expired", isSuperseded: true))); - var actor = CreateReplicationActor(fetcher); - - actor.Tell(new ApplyConfigDeploy( - "GonePump", "dep-stale", "sha256:gone", true, - "http://central:9000", "tok-stale")); - - // The fetch was attempted... - await AwaitAssertAsync(() => - { - Assert.Single(fetcher.Calls); - return Task.CompletedTask; - }, TimeSpan.FromSeconds(5)); - - // ...the actor did not crash (no Terminated to its watcher within the window)... - Watch(actor); - ExpectNoMsg(TimeSpan.FromMilliseconds(500)); - - // ...and nothing was written for the superseded instance. - var configs = await _storage.GetAllDeployedConfigsAsync(); - Assert.DoesNotContain(configs, c => c.InstanceUniqueName == "GonePump"); - } - - [Fact] - public async Task ApplyConfigDeploy_EmptyFetchCoords_SkipsFetchAndWrite() - { - // The direct DeployInstanceCommand cross-cluster wire path was retired in Task 14. - // This tests the defensive guard: if empty coords arrive, the actor must skip quietly - // — no FetchAsync("") call, no write — rather than erroring. - var fetcher = new FakeConfigFetcher(_ => Task.FromResult("never")); - var actor = CreateReplicationActor(fetcher); - - actor.Tell(new ApplyConfigDeploy( - "NoCoordsPump", "dep-direct", "sha256:nc", true, - CentralFetchBaseUrl: "", FetchToken: "")); - - // Give any (erroneous) async continuation time to run, then prove neither happened. - Watch(actor); - ExpectNoMsg(TimeSpan.FromMilliseconds(500)); - Assert.Empty(fetcher.Calls); - var configs = await _storage.GetAllDeployedConfigsAsync(); - Assert.DoesNotContain(configs, c => c.InstanceUniqueName == "NoCoordsPump"); - } - - [Fact] - public void ReplicateConfigDeploy_MapsToIdOnlyApplyConfigDeploy_ForPeer() - { - // The outbound mapping must forward an id-only ApplyConfigDeploy carrying the fetch - // coords (and NO inline config) to the peer. - var probe = CreateTestProbe(); - var fetcher = new FakeConfigFetcher(_ => Task.FromResult("unused")); - var actor = ActorOf(Props.Create(() => new ProbeForwardingReplicationActor( - _storage, _sfStorage, _replicationService, SiteRole, - NullLogger.Instance, fetcher, probe.Ref))); - - actor.Tell(new ReplicateConfigDeploy( - "Pump2", "dep-200", "sha256:def", false, - "http://central:9000", "tok-abc")); - - var applied = probe.ExpectMsg(TimeSpan.FromSeconds(3)); - Assert.Equal("Pump2", applied.InstanceName); - Assert.Equal("dep-200", applied.DeploymentId); - Assert.Equal("sha256:def", applied.RevisionHash); - Assert.False(applied.IsEnabled); - Assert.Equal("http://central:9000", applied.CentralFetchBaseUrl); - Assert.Equal("tok-abc", applied.FetchToken); - } - - // ── Task 21: peer-join S&F buffer resync (anti-entropy) ── - - [Fact] - public void StandbyTrackingPeer_SendsResyncRequest() - { - var probe = CreateTestProbe(); - var actor = ActorOf(Props.Create(() => new ResyncTestActor( - _storage, _sfStorage, _replicationService, SiteRole, - NullLogger.Instance, probe.Ref, () => false))); - - actor.Tell(new TriggerPeerTracked()); // stands in for TryTrackPeer's MemberUp path - - probe.ExpectMsg(TimeSpan.FromSeconds(3)); - } - - [Fact] - public void ActiveTrackingPeer_DoesNotRequestResync() - { - var probe = CreateTestProbe(); - var actor = ActorOf(Props.Create(() => new ResyncTestActor( - _storage, _sfStorage, _replicationService, SiteRole, - NullLogger.Instance, probe.Ref, () => true))); - - actor.Tell(new TriggerPeerTracked()); - - probe.ExpectNoMsg(TimeSpan.FromMilliseconds(300)); // active node never requests a resync - } - - [Fact] - public async Task ActiveNode_AnswersResyncRequest_WithChunkedSnapshot() - { - // Post-R2-T5 the active node answers with byte-budgeted SfBufferSnapshotChunk(s) - // (a single small row rides one chunk) rather than the monolithic SfBufferSnapshot. - await _sfStorage.EnqueueAsync(NewSfMessage("m1")); - var probe = CreateTestProbe(); - var actor = ActorOf(Props.Create(() => new ResyncTestActor( - _storage, _sfStorage, _replicationService, SiteRole, - NullLogger.Instance, probe.Ref, () => true))); - - actor.Tell(new RequestSfBufferResync(), TestActor); - - var chunk = ExpectMsg(TimeSpan.FromSeconds(3)); - Assert.Equal(1, chunk.TotalChunks); - Assert.Equal(1, chunk.Sequence); - Assert.Single(chunk.Messages); - Assert.False(chunk.Truncated); - } - - [Fact] - public async Task StandbyNode_AppliesSnapshot_ReplacingItsBuffer() - { - await _sfStorage.EnqueueAsync(NewSfMessage("stale")); - var probe = CreateTestProbe(); - var actor = ActorOf(Props.Create(() => new ResyncTestActor( - _storage, _sfStorage, _replicationService, SiteRole, - NullLogger.Instance, probe.Ref, () => false))); - - actor.Tell(new SfBufferSnapshot(new List { NewSfMessage("fresh") }, false)); - - await AwaitAssertAsync(async () => - { - Assert.Null(await _sfStorage.GetMessageByIdAsync("stale")); - Assert.NotNull(await _sfStorage.GetMessageByIdAsync("fresh")); - }, TimeSpan.FromSeconds(5)); - } - - // ── R2 T5: chunked resync answer ── - - [Fact] - public void ChunkForRemoting_SplitsByByteBudget_PreservingOrderAndSequence() - { - var rows = Enumerable.Range(0, 10) - .Select(i => NewMessage($"m{i}", payloadJson: new string('x', 20_000))) - .ToList(); - - var chunks = SiteReplicationActor.ChunkForRemoting(rows, maxChunkBytes: 64_000, maxChunkRows: 200); - - Assert.True(chunks.Count > 1); // 10 × 20 KB cannot ride one 64 KB chunk - Assert.Equal(rows.Select(r => r.Id), chunks.SelectMany(c => c).Select(r => r.Id)); // order preserved - Assert.All(chunks, c => Assert.True( - c.Sum(r => r.PayloadJson.Length) <= 64_000 || c.Count == 1)); // budget honored (oversized row isolated) - } - - [Fact] - public void ChunkForRemoting_RowCapHonored_AndSingleOversizedRowIsolated() - { - var many = Enumerable.Range(0, 500).Select(i => NewMessage($"s{i}", payloadJson: "{}")).ToList(); - Assert.All(SiteReplicationActor.ChunkForRemoting(many, 64_000, 200), c => Assert.True(c.Count <= 200)); - - var oversized = new List - { NewMessage("big", payloadJson: new string('y', 100_000)), NewMessage("small", payloadJson: "{}") }; - var chunks = SiteReplicationActor.ChunkForRemoting(oversized, 64_000, 200); - Assert.Equal(2, chunks.Count); // the oversized row rides alone - } - - [Fact] - public async Task ActiveNode_AnswersResyncRequest_WithSequencedChunks_SharingOneResyncId() - { - for (var i = 0; i < 3; i++) - await _sfStorage.EnqueueAsync(NewMessage($"c{i}", payloadJson: new string('z', 30_000))); - var actor = CreateResyncActor(isActive: () => true); - - actor.Tell(new RequestSfBufferResync(), TestActor); - - var first = ExpectMsg(TimeSpan.FromSeconds(5)); - var rest = Enumerable.Range(1, first.TotalChunks - 1) - .Select(_ => ExpectMsg(TimeSpan.FromSeconds(5))) - .Prepend(first) - .ToList(); - - Assert.True(first.TotalChunks > 1); - Assert.All(rest, c => Assert.Equal(first.ResyncId, c.ResyncId)); - Assert.Equal(Enumerable.Range(1, first.TotalChunks), rest.Select(c => c.Sequence)); - Assert.Equal(3, rest.Sum(c => c.Messages.Count)); - } - - // ── R2 T6: standby chunk assembly + atomic apply + ack ── - - [Fact] - public async Task StandbyNode_AssemblesChunks_AppliesOnce_AndAcks() - { - await _sfStorage.EnqueueAsync(NewMessage("stale")); - var actor = CreateResyncActor(isActive: () => false); - var resyncId = "r1"; - - actor.Tell(new SfBufferSnapshotChunk(resyncId, 1, 2, - new List { NewMessage("f1") }, false), TestActor); - actor.Tell(new SfBufferSnapshotChunk(resyncId, 2, 2, - new List { NewMessage("f2") }, false), TestActor); - - var ack = ExpectMsg(TimeSpan.FromSeconds(5)); - Assert.Equal(resyncId, ack.ResyncId); - Assert.Equal(2, ack.RowCount); - await AwaitAssertAsync(async () => - { - Assert.Null(await _sfStorage.GetMessageByIdAsync("stale")); // replaced wholesale - Assert.NotNull(await _sfStorage.GetMessageByIdAsync("f1")); - Assert.NotNull(await _sfStorage.GetMessageByIdAsync("f2")); - }); - } - - [Fact] - public async Task StandbyNode_NewResyncId_DiscardsStalePartialAssembly() - { - var actor = CreateResyncActor(isActive: () => false); - actor.Tell(new SfBufferSnapshotChunk("old", 1, 2, - new List { NewMessage("orphan") }, false), TestActor); - actor.Tell(new SfBufferSnapshotChunk("new", 1, 1, - new List { NewMessage("fresh") }, false), TestActor); - - ExpectMsg(TimeSpan.FromSeconds(5)); // "new" completed - await AwaitAssertAsync(async () => - { - Assert.NotNull(await _sfStorage.GetMessageByIdAsync("fresh")); - Assert.Null(await _sfStorage.GetMessageByIdAsync("orphan")); // stale partial never applied - }); - } - - [Fact] - public void ActiveNode_IgnoresChunks_NeverAcks() - { - var actor = CreateResyncActor(isActive: () => true); - actor.Tell(new SfBufferSnapshotChunk("r", 1, 1, - new List { NewMessage("x") }, false), TestActor); - ExpectNoMsg(TimeSpan.FromMilliseconds(300)); - } - - // ── R2 T7: active-side resync ack confirmation + telemetry ── - // - // NOTE (deviation from plan): the actor logs via Microsoft ILogger (NullLogger in - // tests), NOT Akka's EventStream, so the plan's EventFilter.Warning assertions can - // never observe these warnings. We observe the two OTel counters via a MeterListener - // instead — the equivalent, and stronger, observable signal. - - [Fact] - public async Task ActiveNode_ReceivingAck_CountsResyncCompleted() - { - long completed = 0; - using var listener = ListenCounter("scadabridge.store_and_forward.resync.completed", - m => Interlocked.Add(ref completed, m)); - - await _sfStorage.EnqueueAsync(NewMessage("m1")); - var actor = CreateResyncActor(isActive: () => true, ackTimeout: TimeSpan.FromSeconds(30)); - actor.Tell(new RequestSfBufferResync(), TestActor); - var chunk = ExpectMsg(TimeSpan.FromSeconds(5)); - - actor.Tell(new SfBufferResyncAck(chunk.ResyncId, 1), TestActor); - - await AwaitAssertAsync(() => - { - Assert.True(Interlocked.Read(ref completed) >= 1); // ack recorded the completion - return Task.CompletedTask; - }, TimeSpan.FromSeconds(5)); - } - - [Fact] - public async Task ActiveNode_MissingAck_WarnsAfterAckTimeout() - { - long ackMissing = 0; - using var listener = ListenCounter("scadabridge.store_and_forward.resync.ack_missing", - m => Interlocked.Add(ref ackMissing, m)); - - await _sfStorage.EnqueueAsync(NewMessage("m1")); - var actor = CreateResyncActor(isActive: () => true, ackTimeout: TimeSpan.FromMilliseconds(200)); - actor.Tell(new RequestSfBufferResync(), TestActor); - ExpectMsg(TimeSpan.FromSeconds(5)); - // No ack is sent → the ack window expires and the resync is counted unacknowledged. - - await AwaitAssertAsync(() => - { - Assert.True(Interlocked.Read(ref ackMissing) >= 1); - return Task.CompletedTask; - }, TimeSpan.FromSeconds(5)); - } - - /// Attaches a to a single ScadaBridge counter by name, - /// forwarding each recorded increment to . - private static MeterListener ListenCounter(string instrumentName, Action onMeasurement) - { - var listener = new MeterListener(); - listener.InstrumentPublished = (inst, l) => - { - if (inst.Meter.Name == ScadaBridgeTelemetry.MeterName && inst.Name == instrumentName) - l.EnableMeasurementEvents(inst); - }; - listener.SetMeasurementEventCallback((_, m, _, _) => onMeasurement(m)); - listener.Start(); - return listener; - } - - private static StoreAndForwardMessage NewSfMessage(string id) => new() - { - Id = id, - Category = StoreAndForwardCategory.ExternalSystem, - Target = "t", - PayloadJson = "{}", - RetryCount = 0, - MaxRetries = 50, - RetryIntervalMs = 30000, - CreatedAt = DateTimeOffset.UtcNow, - Status = StoreAndForwardMessageStatus.Pending, - }; - - /// - /// Builds a resync-test message with a settable payload (additive to - /// — the chunker sizes on PayloadJson length). - /// - private static StoreAndForwardMessage NewMessage(string id, string payloadJson = "{}") => new() - { - Id = id, - Category = StoreAndForwardCategory.ExternalSystem, - Target = "t", - PayloadJson = payloadJson, - RetryCount = 0, - MaxRetries = 50, - RetryIntervalMs = 30000, - CreatedAt = DateTimeOffset.UtcNow, - Status = StoreAndForwardMessageStatus.Pending, - }; - - /// Constructs a with the given active-node check - /// (the resync chunk/ack tests Tell to and expect from ). - /// is the active-side ack window seam (T7). - private IActorRef CreateResyncActor(Func isActive, TimeSpan? ackTimeout = null) => - ActorOf(Props.Create(() => new ResyncTestActor( - _storage, _sfStorage, _replicationService, SiteRole, - NullLogger.Instance, CreateTestProbe().Ref, isActive, ackTimeout))); - - /// Test message: drives directly, - /// standing in for the MemberUp→TryTrackPeer path (a single-node TestKit cannot form a real peer). - private sealed record TriggerPeerTracked; - - /// - /// Test subclass for the resync tests: captures peer sends to a probe, injects the - /// active-node check, and exposes via a test message. - /// - private sealed class ResyncTestActor : SiteReplicationActor - { - private readonly IActorRef _peerProbe; - - public ResyncTestActor( - SiteStorageService storage, StoreAndForwardStorage sfStorage, - ReplicationService replicationService, string siteRole, - ILogger logger, IActorRef peerProbe, Func isActive, - TimeSpan? ackTimeout = null) - : base(storage, sfStorage, replicationService, siteRole, logger, - configFetcher: null, isActiveOverride: isActive, resyncAckTimeout: ackTimeout) - { - _peerProbe = peerProbe; - Receive(_ => OnPeerTracked()); - } - - protected override void SendToPeer(object message) => _peerProbe.Tell(message, Self); - } - - /// - /// Test subclass exposing the peer send: is - /// overridden to forward to a probe so the outbound mapping can be asserted without a real - /// two-node cluster (a single-node TestKit has no peer address, so the real send is dropped). - /// - private sealed class ProbeForwardingReplicationActor : SiteReplicationActor - { - private readonly IActorRef _peerProbe; - - public ProbeForwardingReplicationActor( - SiteStorageService storage, StoreAndForwardStorage sfStorage, - ReplicationService replicationService, string siteRole, - ILogger logger, IDeploymentConfigFetcher configFetcher, - IActorRef peerProbe) - : base(storage, sfStorage, replicationService, siteRole, logger, configFetcher) - => _peerProbe = peerProbe; - - protected override void SendToPeer(object message) => _peerProbe.Tell(message, Self); - } - - /// - /// In-test fake : runs a per-deploymentId behavior - /// (return config JSON or throw, as a Task — mirroring the real async HTTP fetcher) and - /// records every call's coords thread-safely (the continuation runs on a pool thread). - /// - private sealed class FakeConfigFetcher : IDeploymentConfigFetcher - { - private readonly Func> _behavior; - public ConcurrentQueue<(string BaseUrl, string DeploymentId, string Token)> Calls { get; } = new(); - - public FakeConfigFetcher(Func> behavior) => _behavior = behavior; - - public async Task FetchAsync( - string centralFetchBaseUrl, string deploymentId, string token, CancellationToken ct) - { - Calls.Enqueue((centralFetchBaseUrl, deploymentId, token)); - await Task.Yield(); - return await _behavior(deploymentId); - } - } -} diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/ResyncWireSerializationPinTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/ResyncWireSerializationPinTests.cs deleted file mode 100644 index 43560a14..00000000 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/ResyncWireSerializationPinTests.cs +++ /dev/null @@ -1,95 +0,0 @@ -using Akka.TestKit.Xunit2; -using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; -using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors; -using ZB.MOM.WW.ScadaBridge.StoreAndForward; - -namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests; - -/// -/// Characterization pin for the chunked anti-entropy resync contract (review 02 round 2, -/// N2). (active→standby) and -/// (standby→active) ride intra-site Akka remoting on the default reflective-JSON wire -/// format. A rename/move, dropped setter, or non-default-constructible message would -/// silently break resync across a rolling upgrade and only surface as a divergent buffer -/// after a failover. These pin round-trip fidelity and type identity. (These messages are -/// NOT ClusterClient traffic, so they are intentionally absent from ClusterClientContractLockTests.) -/// -public class ResyncWireSerializationPinTests : TestKit -{ - private T RoundTrip(T message) - { - var serialization = Sys.Serialization; - var serializer = serialization.FindSerializerFor(message); - var bytes = serializer.ToBinary(message); - return (T)serialization.Deserialize(bytes, serializer.Identifier, message!.GetType()); - } - - private static StoreAndForwardMessage FullMessage() => new() - { - Id = Guid.NewGuid().ToString("N"), - Category = StoreAndForwardCategory.Notification, - Target = "Operators", - PayloadJson = "{\"notificationId\":\"abc\"}", - RetryCount = 4, - MaxRetries = 0, - RetryIntervalMs = 30000, - CreatedAt = DateTimeOffset.UtcNow, - LastAttemptAt = DateTimeOffset.UtcNow, - Status = StoreAndForwardMessageStatus.Parked, - LastError = "central rejected", - OriginInstanceName = "Plant.Pump3", - ExecutionId = Guid.NewGuid(), - SourceScript = "ScriptActor:MonitorSpeed", - ParentExecutionId = Guid.NewGuid(), - }; - - [Fact] - public void SfBufferSnapshotChunk_WithFullMessage_RoundTripsOnTheWire() - { - var message = FullMessage(); - var original = new SfBufferSnapshotChunk( - "resync-1", 2, 5, new List { message }, Truncated: true); - - var back = RoundTrip(original); - - Assert.Equal(original.ResyncId, back.ResyncId); - Assert.Equal(original.Sequence, back.Sequence); - Assert.Equal(original.TotalChunks, back.TotalChunks); - Assert.Equal(original.Truncated, back.Truncated); - var m = Assert.Single(back.Messages); - Assert.Equal(message.Id, m.Id); - Assert.Equal(message.Category, m.Category); - Assert.Equal(message.Target, m.Target); - Assert.Equal(message.PayloadJson, m.PayloadJson); - Assert.Equal(message.RetryCount, m.RetryCount); - Assert.Equal(message.MaxRetries, m.MaxRetries); - Assert.Equal(message.RetryIntervalMs, m.RetryIntervalMs); - Assert.Equal(message.CreatedAt, m.CreatedAt); - Assert.Equal(message.LastAttemptAt, m.LastAttemptAt); - Assert.Equal(message.Status, m.Status); - Assert.Equal(message.LastError, m.LastError); - Assert.Equal(message.OriginInstanceName, m.OriginInstanceName); - Assert.Equal(message.ExecutionId, m.ExecutionId); - Assert.Equal(message.SourceScript, m.SourceScript); - Assert.Equal(message.ParentExecutionId, m.ParentExecutionId); - } - - [Fact] - public void SfBufferResyncAck_RoundTripsOnTheWire() - { - var original = new SfBufferResyncAck("resync-1", 42); - - var back = RoundTrip(original); - - Assert.Equal(original.ResyncId, back.ResyncId); - Assert.Equal(original.RowCount, back.RowCount); - } - - // Type-identity pins: the reflective-JSON wire embeds CLR type manifests, so a - // rename/move of either type silently breaks resync across a rolling upgrade. - [Theory] - [InlineData(typeof(SfBufferSnapshotChunk), "ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors.SfBufferSnapshotChunk")] - [InlineData(typeof(SfBufferResyncAck), "ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors.SfBufferResyncAck")] - public void ResyncContract_TypeIdentity_IsPinned(Type type, string expectedFullName) => - Assert.Equal(expectedFullName, type.FullName); -} diff --git a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/CachedCallAttemptEmissionTests.cs b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/CachedCallAttemptEmissionTests.cs index 4bf217e7..f01746fa 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/CachedCallAttemptEmissionTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/CachedCallAttemptEmissionTests.cs @@ -45,7 +45,6 @@ public class CachedCallAttemptEmissionTests : IAsyncLifetime, IDisposable _storage, _options, NullLogger.Instance, - replication: null, cachedCallObserver: _observer, siteId: "site-77"); } @@ -490,7 +489,6 @@ public class CachedCallAttemptEmissionTests : IAsyncLifetime, IDisposable RetryTimerInterval = TimeSpan.FromHours(1), // timer never fires in-test }, NullLogger.Instance, - replication: null, cachedCallObserver: observer, siteId: "site-77"); diff --git a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ReplicationServiceTests.cs b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ReplicationServiceTests.cs deleted file mode 100644 index 19fa4b2b..00000000 --- a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ReplicationServiceTests.cs +++ /dev/null @@ -1,236 +0,0 @@ -using Microsoft.Extensions.Logging; -using Microsoft.Extensions.Logging.Abstractions; -using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; -using ZB.MOM.WW.ScadaBridge.TestSupport; - -namespace ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests; - -/// -/// WP-11: Tests for async replication to standby. -/// -public class ReplicationServiceTests : IAsyncLifetime, IDisposable -{ - private readonly TestLocalDb _localDb; - private readonly StoreAndForwardStorage _storage; - private readonly ReplicationService _replicationService; - - public ReplicationServiceTests() - { - _localDb = TestLocalDb.CreateTemp("RepTests"); - - _storage = new StoreAndForwardStorage(_localDb.Db, NullLogger.Instance); - - var options = new StoreAndForwardOptions { ReplicationEnabled = true }; - _replicationService = new ReplicationService( - options, NullLogger.Instance); - } - - public async Task InitializeAsync() => await _storage.InitializeAsync(); - - public Task DisposeAsync() => Task.CompletedTask; - - public void Dispose() - { - var path = _localDb.Path; - _localDb.Dispose(); - TestLocalDb.DeleteFiles(path); - } - - [Fact] - public void ReplicateEnqueue_NoHandler_DoesNotThrow() - { - var msg = CreateMessage("rep1"); - _replicationService.ReplicateEnqueue(msg); - } - - [Fact] - public async Task ReplicateEnqueue_WithHandler_ForwardsOperation() - { - ReplicationOperation? captured = null; - _replicationService.SetReplicationHandler(op => - { - captured = op; - return Task.CompletedTask; - }); - - var msg = CreateMessage("rep2"); - _replicationService.ReplicateEnqueue(msg); - - await Task.Delay(200); - - Assert.NotNull(captured); - Assert.Equal(ReplicationOperationType.Add, captured!.OperationType); - Assert.Equal("rep2", captured.MessageId); - } - - [Fact] - public async Task ReplicateRemove_WithHandler_ForwardsRemoveOperation() - { - ReplicationOperation? captured = null; - _replicationService.SetReplicationHandler(op => - { - captured = op; - return Task.CompletedTask; - }); - - _replicationService.ReplicateRemove("rep3"); - - await Task.Delay(200); - - Assert.NotNull(captured); - Assert.Equal(ReplicationOperationType.Remove, captured!.OperationType); - Assert.Equal("rep3", captured.MessageId); - } - - [Fact] - public async Task ReplicatePark_WithHandler_ForwardsParkOperation() - { - ReplicationOperation? captured = null; - _replicationService.SetReplicationHandler(op => - { - captured = op; - return Task.CompletedTask; - }); - - var msg = CreateMessage("rep4"); - _replicationService.ReplicatePark(msg); - - await Task.Delay(200); - - Assert.NotNull(captured); - Assert.Equal(ReplicationOperationType.Park, captured!.OperationType); - } - - [Fact] - public async Task ApplyReplicatedOperationAsync_Add_EnqueuesMessage() - { - var msg = CreateMessage("apply1"); - var operation = new ReplicationOperation(ReplicationOperationType.Add, "apply1", msg); - - await _replicationService.ApplyReplicatedOperationAsync(operation, _storage); - - var retrieved = await _storage.GetMessageByIdAsync("apply1"); - Assert.NotNull(retrieved); - } - - [Fact] - public async Task ApplyReplicatedOperationAsync_Remove_DeletesMessage() - { - var msg = CreateMessage("apply2"); - await _storage.EnqueueAsync(msg); - - var operation = new ReplicationOperation(ReplicationOperationType.Remove, "apply2", null); - await _replicationService.ApplyReplicatedOperationAsync(operation, _storage); - - var retrieved = await _storage.GetMessageByIdAsync("apply2"); - Assert.Null(retrieved); - } - - [Fact] - public async Task ApplyReplicatedOperationAsync_Park_UpdatesStatus() - { - var msg = CreateMessage("apply3"); - await _storage.EnqueueAsync(msg); - - var operation = new ReplicationOperation(ReplicationOperationType.Park, "apply3", msg); - await _replicationService.ApplyReplicatedOperationAsync(operation, _storage); - - var retrieved = await _storage.GetMessageByIdAsync("apply3"); - Assert.NotNull(retrieved); - Assert.Equal(StoreAndForwardMessageStatus.Parked, retrieved!.Status); - } - - [Fact] - public void ReplicateEnqueue_WhenReplicationDisabled_DoesNothing() - { - var options = new StoreAndForwardOptions { ReplicationEnabled = false }; - var service = new ReplicationService(options, NullLogger.Instance); - - bool handlerCalled = false; - service.SetReplicationHandler(_ => { handlerCalled = true; return Task.CompletedTask; }); - - service.ReplicateEnqueue(CreateMessage("disabled1")); - - Assert.False(handlerCalled); - } - - [Fact] - public async Task ReplicateEnqueue_HandlerThrows_DoesNotPropagateException() - { - _replicationService.SetReplicationHandler(_ => - throw new InvalidOperationException("standby down")); - - _replicationService.ReplicateEnqueue(CreateMessage("err1")); - - await Task.Delay(200); - // No exception -- fire-and-forget, best-effort - } - - // ── Task 10 (arch review 02): ordered, observable replication dispatch ── - - [Fact] - public void ReplicationOperations_AreDispatchedInIssueOrder() - { - var seen = new List<(ReplicationOperationType, string)>(); - _replicationService.SetReplicationHandler(op => - { - seen.Add((op.OperationType, op.MessageId)); - return Task.CompletedTask; - }); - - for (var i = 0; i < 200; i++) - { - _replicationService.ReplicateEnqueue(CreateMessage($"m{i}")); - _replicationService.ReplicateRemove($"m{i}"); - } - - // Inline dispatch: by the time the calls return, every op was handed to the - // handler, Add strictly before Remove per id. Pre-fix (Task.Run) this was - // racy in both count and order. - Assert.Equal(400, seen.Count); - for (var i = 0; i < 200; i++) - { - Assert.Equal((ReplicationOperationType.Add, $"m{i}"), seen[2 * i]); - Assert.Equal((ReplicationOperationType.Remove, $"m{i}"), seen[2 * i + 1]); - } - } - - [Fact] - public void ReplicationHandlerThrow_IsSwallowed_AndLoggedAtWarning() - { - var logger = new CapturingLogger(); - var service = new ReplicationService(new StoreAndForwardOptions(), logger); - service.SetReplicationHandler(_ => throw new InvalidOperationException("peer gone")); - - service.ReplicateRemove("m1"); // must not throw - - Assert.Contains(logger.Entries, e => e.Level == LogLevel.Warning); - } - - private static StoreAndForwardMessage CreateMessage(string id) - { - return new StoreAndForwardMessage - { - Id = id, - Category = StoreAndForwardCategory.ExternalSystem, - Target = "target", - PayloadJson = "{}", - RetryCount = 0, - MaxRetries = 50, - RetryIntervalMs = 30000, - CreatedAt = DateTimeOffset.UtcNow, - Status = StoreAndForwardMessageStatus.Pending - }; - } - - /// Minimal in-memory logger that records level + rendered message. - private sealed class CapturingLogger : ILogger - { - public List<(LogLevel Level, string Message)> Entries { get; } = new(); - public IDisposable? BeginScope(TState state) where TState : notnull => null; - public bool IsEnabled(LogLevel logLevel) => true; - public void Log(LogLevel logLevel, EventId eventId, TState state, Exception? exception, - Func formatter) - => Entries.Add((logLevel, formatter(state, exception))); - } -} diff --git a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ReplicationWireSerializationPinTests.cs b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ReplicationWireSerializationPinTests.cs deleted file mode 100644 index f1d194f0..00000000 --- a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ReplicationWireSerializationPinTests.cs +++ /dev/null @@ -1,82 +0,0 @@ -using Akka.TestKit.Xunit2; -using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; - -namespace ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests; - -/// -/// Characterization pin for the intra-cluster S&F replication contract. A -/// (carrying a full ) -/// is Told from the active node to the standby node's replication actor and rides -/// the Akka default reflective-JSON wire format. Nothing tested that it round-trips -/// or that its type identity is stable — a rename/move, a dropped setter, or a -/// non-default-constructible message would silently break standby buffer sync and -/// only surface as a divergent buffer after a failover. -/// -/// The serializer swap (proto / explicit bindings) is deferred; until then this pin -/// is the guardrail. -/// -public class ReplicationWireSerializationPinTests : TestKit -{ - private T RoundTrip(T message) - { - var serialization = Sys.Serialization; - var serializer = serialization.FindSerializerFor(message); - var bytes = serializer.ToBinary(message); - return (T)serialization.Deserialize(bytes, serializer.Identifier, message!.GetType()); - } - - [Fact] - public void ReplicationOperation_WithFullMessage_RoundTripsOnTheWire() - { - var message = new StoreAndForwardMessage - { - Id = Guid.NewGuid().ToString("N"), - Category = StoreAndForwardCategory.Notification, - Target = "Operators", - PayloadJson = "{\"notificationId\":\"abc\"}", - RetryCount = 4, - MaxRetries = 0, - RetryIntervalMs = 30000, - CreatedAt = DateTimeOffset.UtcNow, - LastAttemptAt = DateTimeOffset.UtcNow, - Status = StoreAndForwardMessageStatus.Parked, - LastError = "central rejected", - OriginInstanceName = "Plant.Pump3", - ExecutionId = Guid.NewGuid(), - SourceScript = "ScriptActor:MonitorSpeed", - ParentExecutionId = Guid.NewGuid(), - }; - var original = new ReplicationOperation(ReplicationOperationType.Park, message.Id, message); - - var back = RoundTrip(original); - - Assert.Equal(original.OperationType, back.OperationType); - Assert.Equal(original.MessageId, back.MessageId); - Assert.NotNull(back.Message); - var m = back.Message!; - Assert.Equal(message.Id, m.Id); - Assert.Equal(message.Category, m.Category); - Assert.Equal(message.Target, m.Target); - Assert.Equal(message.PayloadJson, m.PayloadJson); - Assert.Equal(message.RetryCount, m.RetryCount); - Assert.Equal(message.MaxRetries, m.MaxRetries); - Assert.Equal(message.RetryIntervalMs, m.RetryIntervalMs); - Assert.Equal(message.CreatedAt, m.CreatedAt); - Assert.Equal(message.LastAttemptAt, m.LastAttemptAt); - Assert.Equal(message.Status, m.Status); - Assert.Equal(message.LastError, m.LastError); - Assert.Equal(message.OriginInstanceName, m.OriginInstanceName); - Assert.Equal(message.ExecutionId, m.ExecutionId); - Assert.Equal(message.SourceScript, m.SourceScript); - Assert.Equal(message.ParentExecutionId, m.ParentExecutionId); - } - - // Type-identity pins: the reflective-JSON wire embeds CLR type manifests, so a - // rename/move of either type silently breaks standby replication across a - // rolling upgrade. If one fails, you are making a wire-breaking change. - [Theory] - [InlineData(typeof(ReplicationOperation), "ZB.MOM.WW.ScadaBridge.StoreAndForward.ReplicationOperation")] - [InlineData(typeof(StoreAndForwardMessage), "ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardMessage")] - public void ReplicationContract_TypeIdentity_IsPinned(Type type, string expectedFullName) => - Assert.Equal(expectedFullName, type.FullName); -} diff --git a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardReplicationTests.cs b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardReplicationTests.cs deleted file mode 100644 index 8080ce78..00000000 --- a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardReplicationTests.cs +++ /dev/null @@ -1,266 +0,0 @@ -using Microsoft.Extensions.Logging.Abstractions; -using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; -using ZB.MOM.WW.ScadaBridge.TestSupport; - -namespace ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests; - -/// -/// StoreAndForward-001: the active node must forward every buffer operation -/// (add / remove / park) to the standby via the ReplicationService, so a -/// failover does not lose the buffer. -/// -public class StoreAndForwardReplicationTests : IAsyncLifetime, IDisposable -{ - private readonly TestLocalDb _localDb; - private readonly StoreAndForwardStorage _storage; - private readonly StoreAndForwardService _service; - private readonly List _replicated = new(); - - public StoreAndForwardReplicationTests() - { - _localDb = TestLocalDb.CreateTemp("ReplTests"); - - _storage = new StoreAndForwardStorage(_localDb.Db, NullLogger.Instance); - - var options = new StoreAndForwardOptions - { - DefaultRetryInterval = TimeSpan.Zero, - DefaultMaxRetries = 1, - RetryTimerInterval = TimeSpan.FromMinutes(10), - ReplicationEnabled = true, - }; - - var replication = new ReplicationService(options, NullLogger.Instance); - replication.SetReplicationHandler(op => - { - lock (_replicated) _replicated.Add(op); - return Task.CompletedTask; - }); - - _service = new StoreAndForwardService( - _storage, options, NullLogger.Instance, replication); - } - - public async Task InitializeAsync() => await _storage.InitializeAsync(); - public Task DisposeAsync() => Task.CompletedTask; - public void Dispose() - { - var path = _localDb.Path; - _localDb.Dispose(); - TestLocalDb.DeleteFiles(path); - } - - /// Replication is fire-and-forget (Task.Run); poll until the expected ops arrive. - private async Task> WaitForReplicationAsync(int count) - { - for (var i = 0; i < 100; i++) - { - lock (_replicated) - if (_replicated.Count >= count) return _replicated.ToList(); - await Task.Delay(20); - } - lock (_replicated) return _replicated.ToList(); - } - - [Fact] - public async Task BufferingAMessage_ReplicatesAnAddOperation() - { - // No handler registered → message is buffered → an Add is replicated. - var result = await _service.EnqueueAsync( - StoreAndForwardCategory.ExternalSystem, "api", """{}"""); - Assert.True(result.WasBuffered); - - var ops = await WaitForReplicationAsync(1); - Assert.Contains(ops, o => - o.OperationType == ReplicationOperationType.Add && o.MessageId == result.MessageId); - } - - [Fact] - public async Task SuccessfulRetry_ReplicatesARemoveOperation() - { - var calls = 0; - _service.RegisterDeliveryHandler(StoreAndForwardCategory.ExternalSystem, - _ => ++calls == 1 - ? throw new HttpRequestException("transient") - : Task.FromResult(true)); - - var result = await _service.EnqueueAsync( - StoreAndForwardCategory.ExternalSystem, "api", """{}"""); - await _service.RetryPendingMessagesAsync(); - - var ops = await WaitForReplicationAsync(2); - Assert.Contains(ops, o => o.OperationType == ReplicationOperationType.Add); - Assert.Contains(ops, o => - o.OperationType == ReplicationOperationType.Remove && o.MessageId == result.MessageId); - } - - [Fact] - public async Task ParkedMessage_ReplicatesAParkOperation() - { - _service.RegisterDeliveryHandler(StoreAndForwardCategory.ExternalSystem, - _ => throw new HttpRequestException("always fails")); - - var result = await _service.EnqueueAsync( - StoreAndForwardCategory.ExternalSystem, "api", """{}""", maxRetries: 1); - await _service.RetryPendingMessagesAsync(); - - var ops = await WaitForReplicationAsync(2); - Assert.Contains(ops, o => - o.OperationType == ReplicationOperationType.Park && o.MessageId == result.MessageId); - } - - /// - /// StoreAndForward-016: an operator discarding a parked message must replicate - /// a Remove so the standby's copy is also deleted (otherwise the discarded - /// message reappears in the parked list after a failover). - /// - [Fact] - public async Task DiscardingAParkedMessage_ReplicatesARemoveOperation() - { - _service.RegisterDeliveryHandler(StoreAndForwardCategory.ExternalSystem, - _ => throw new HttpRequestException("always fails")); - - var result = await _service.EnqueueAsync( - StoreAndForwardCategory.ExternalSystem, "api", """{}""", maxRetries: 1); - await _service.RetryPendingMessagesAsync(); // -> parked - await WaitForReplicationAsync(2); - - var discarded = await _service.DiscardParkedMessageAsync(result.MessageId); - Assert.True(discarded); - - var ops = await WaitForReplicationAsync(3); - Assert.Contains(ops, o => - o.OperationType == ReplicationOperationType.Remove && o.MessageId == result.MessageId); - } - - /// - /// StoreAndForward-016: an operator retrying a parked message must replicate a - /// Requeue so the standby's copy moves back to Pending (otherwise it stays - /// Parked on the standby and the operator's retry is lost across a failover). - /// - [Fact] - public async Task RetryingAParkedMessage_ReplicatesARequeueOperation() - { - _service.RegisterDeliveryHandler(StoreAndForwardCategory.ExternalSystem, - _ => throw new HttpRequestException("always fails")); - - var result = await _service.EnqueueAsync( - StoreAndForwardCategory.ExternalSystem, "api", """{}""", maxRetries: 1); - await _service.RetryPendingMessagesAsync(); // -> parked - await WaitForReplicationAsync(2); - - var retried = await _service.RetryParkedMessageAsync(result.MessageId); - Assert.True(retried); - - var ops = await WaitForReplicationAsync(3); - var requeue = ops.SingleOrDefault(o => - o.OperationType == ReplicationOperationType.Requeue && o.MessageId == result.MessageId); - Assert.NotNull(requeue); - Assert.NotNull(requeue!.Message); - Assert.Equal(StoreAndForwardMessageStatus.Pending, requeue.Message!.Status); - } - - /// - /// StoreAndForward-016: the standby applies a Requeue by moving its row back to - /// Pending with retry_count = 0, mirroring the active node's local state. - /// - [Fact] - public async Task ApplyReplicatedOperation_Requeue_MovesStandbyRowBackToPending() - { - var replication = new ReplicationService( - new StoreAndForwardOptions { ReplicationEnabled = true }, - NullLogger.Instance); - - var parked = new StoreAndForwardMessage - { - Id = "requeue1", - Category = StoreAndForwardCategory.ExternalSystem, - Target = "api", - PayloadJson = "{}", - RetryCount = 5, - MaxRetries = 1, - RetryIntervalMs = 0, - CreatedAt = DateTimeOffset.UtcNow, - Status = StoreAndForwardMessageStatus.Parked, - }; - await _storage.EnqueueAsync(parked); - - var requeued = new StoreAndForwardMessage - { - Id = parked.Id, - Category = parked.Category, - Target = parked.Target, - PayloadJson = parked.PayloadJson, - RetryCount = 0, - MaxRetries = parked.MaxRetries, - RetryIntervalMs = parked.RetryIntervalMs, - CreatedAt = parked.CreatedAt, - Status = StoreAndForwardMessageStatus.Pending, - }; - await replication.ApplyReplicatedOperationAsync( - new ReplicationOperation(ReplicationOperationType.Requeue, parked.Id, requeued), - _storage); - - var row = await _storage.GetMessageByIdAsync(parked.Id); - Assert.NotNull(row); - Assert.Equal(StoreAndForwardMessageStatus.Pending, row!.Status); - Assert.Equal(0, row.RetryCount); - } - - private static ReplicationService NewReplicationService() => - new(new StoreAndForwardOptions { ReplicationEnabled = true }, - NullLogger.Instance); - - private static StoreAndForwardMessage NewMessage(string id) => new() - { - Id = id, - Category = StoreAndForwardCategory.ExternalSystem, - Target = "api", - PayloadJson = "{}", - RetryCount = 0, - MaxRetries = 1, - RetryIntervalMs = 0, - CreatedAt = DateTimeOffset.UtcNow, - Status = StoreAndForwardMessageStatus.Pending, - }; - - /// - /// A Park (or Requeue) whose original Add was lost — e.g. the Add's - /// fire-and-forget replication dropped — must still materialise the row on the - /// standby. The full message rides in the operation, so an upsert self-heals; - /// a blind UPDATE would affect 0 rows and the row would be gone forever. - /// - [Fact] - public async Task ApplyReplicatedPark_WhenAddWasLost_MaterializesTheParkedRow() - { - var service = NewReplicationService(); - var msg = NewMessage("lost-add-1"); // never Added on this (standby) storage - msg.Status = StoreAndForwardMessageStatus.Parked; - - await service.ApplyReplicatedOperationAsync( - new ReplicationOperation(ReplicationOperationType.Park, msg.Id, msg), _storage); - - var row = await _storage.GetMessageByIdAsync("lost-add-1"); - Assert.NotNull(row); // pre-fix: blind UPDATE affected 0 rows, row is gone forever - Assert.Equal(StoreAndForwardMessageStatus.Parked, row!.Status); - } - - /// - /// A duplicate Add (e.g. after Task 21's anti-entropy resync re-issues an Add - /// the standby already holds) must not violate the PK — the upsert applies - /// newest-wins instead of throwing. - /// - [Fact] - public async Task ApplyReplicatedAdd_Twice_IsIdempotent_NewestWins() - { - var service = NewReplicationService(); - var msg = NewMessage("dup-add"); - await service.ApplyReplicatedOperationAsync( - new ReplicationOperation(ReplicationOperationType.Add, msg.Id, msg), _storage); - msg.RetryCount = 3; - await service.ApplyReplicatedOperationAsync( // pre-fix: SqliteException PK violation - new ReplicationOperation(ReplicationOperationType.Add, msg.Id, msg), _storage); - - Assert.Equal(3, (await _storage.GetMessageByIdAsync("dup-add"))!.RetryCount); - } -} diff --git a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardSiteEventTests.cs b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardSiteEventTests.cs index 52144ee7..b21acfd8 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardSiteEventTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardSiteEventTests.cs @@ -54,7 +54,7 @@ public class StoreAndForwardSiteEventTests : IAsyncLifetime, IDisposable _service = new StoreAndForwardService( _storage, _options, NullLogger.Instance, - replication: null, cachedCallObserver: null, siteId: "site-a", + cachedCallObserver: null, siteId: "site-a", siteEventLogger: _siteLog); } diff --git a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardStorageTests.cs b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardStorageTests.cs index 668a3007..18ce27c2 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardStorageTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardStorageTests.cs @@ -769,16 +769,15 @@ public class StoreAndForwardStorageTests : IAsyncLifetime, IDisposable Assert.True(truncated); // a third row exists beyond the limit } - [Fact] - public async Task ReplaceAll_SwapsTheEntireBuffer_Atomically() - { - await _storage.EnqueueAsync(NewMsg("stale")); - await _storage.ReplaceAllAsync(new[] { NewMsg("fresh-1"), NewMsg("fresh-2") }); - - Assert.Null(await _storage.GetMessageByIdAsync("stale")); - Assert.NotNull(await _storage.GetMessageByIdAsync("fresh-1")); - Assert.NotNull(await _storage.GetMessageByIdAsync("fresh-2")); - } + // ReplaceAll_SwapsTheEntireBuffer_Atomically was DELETED with ReplaceAllAsync in + // LocalDb Phase 2, and deliberately not replaced. It asserted a destructive + // delete-all-then-insert-all, which was the standby's anti-entropy apply: a resync + // overwrote the standby's copy with the active node's snapshot. sf_messages is now a + // replicated table, so a mass DELETE would be CAPTURED and shipped to the peer — the + // method is not merely unused, it is unsafe to keep. LocalDb's own snapshot resync + // merges per row under last-writer-wins and never deletes, which is what + // LocalDbConfigConvergenceTests.ANodeWithNewerLocalRows_KeepsThem_WhenAPeerSnapshotArrives + // now covers. // ── Task 23: oldest-parked-age health signal ── From 0ad11d6b550138448e13387ddd424a4ad7ddd1e7 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Mon, 20 Jul 2026 04:20:17 -0400 Subject: [PATCH 45/55] docs(localdb): task state for the 14/15/16 cutover Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts --- docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json b/docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json index c1b96e57..e9718bf1 100644 --- a/docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json +++ b/docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json @@ -82,9 +82,9 @@ {"id": 11, "subject": "Task 11: Port the resync + directional-authority tests", "status": "completed", "classification": "standard", "blockedBy": [9], "note": "N1 re-expressed per D2 as a no-rows-lost property (LWW merge, never deletes), not directional authority. DEVIATION on the zero-fetch assertion: the plan wanted a fetcher double recording zero calls, but this harness has no actor system / no central / no IDeploymentConfigFetcher in the graph, so the double could not fail either way. Replaced with the positive half (config reaches B over replication alone) plus an in-file comment recording that the negative half is proved by Task 15 deleting the code and the build passing. D1 scope note recorded in-file. Non-vacuity: unregistering deployed_configurations fails all 4.", "commit": "c56bf4ae"}, {"id": 12, "subject": "Task 12: Pin the active-node SMTP purge (corrected: already exists at DeploymentManagerActor.cs:1921)", "status": "completed", "classification": "standard", "blockedBy": [10, 11], "note": "As planned - NO production change; D3 confirmed correct, the call already exists at DeploymentManagerActor.cs:1921. Pin verified RED-FIRST by commenting out the call. Test needed a using for Commons.Messages.Artifacts and polls (the apply runs on a Task.Run inside the actor).", "commit": "79ce5161"}, {"id": 13, "subject": "Task 13: Notify-and-fetch scope check - guarded write STAYS for SiteReconciliationActor", "status": "completed", "classification": "standard", "blockedBy": [12], "note": "Doc-comment only, as planned. Re-verified both callers: SiteReplicationActor:375 (dies Task 15) + SiteReconciliationActor:166 (survives). Step 2 scope check re-run: ConfigFetchRetryCount's only production reader is still SiteReplicationActor:157, so option + validator rule stay until Task 17. Also fixed a stale 'guarded standby write' header in SiteStorageServiceTests.", "commit": "79ce5161"}, - {"id": 14, "subject": "Task 14: Register the 8 Phase 2 tables and delete ReplicationService", "status": "pending", "classification": "high-risk", "blockedBy": [13], "note": "THE CUTOVER. Register 7 config tables + sf_messages - NOT notification_lists/smtp_configurations (reversed from original; see D3). Keep Migrate LAST in OnReady. Registration + bespoke deletion in ONE commit."}, - {"id": 15, "subject": "Task 15: Delete SiteReplicationActor and its messages", "status": "pending", "classification": "high-risk", "blockedBy": [14], "note": "Resync records live at SiteReplicationActor.cs:678-707, not ReplicationMessages.cs."}, - {"id": 16, "subject": "Task 16: Clean up DeploymentManagerActor and AkkaHostedService", "status": "pending", "classification": "standard", "blockedBy": [15], "note": "Ctor param at :169; same-typed IActorRef? dclManager at :168 BEFORE it is the silent-shift hazard; Props.Create positional at AkkaHostedService.cs:810."}, + {"id": 14, "subject": "Task 14: Register the 8 Phase 2 tables and delete ReplicationService", "status": "completed", "classification": "high-risk", "blockedBy": [13], "note": "LANDED AS ONE COMMIT with tasks 15+16 - PLAN DEFECT: they cannot compile separately (SiteReplicationActor takes a ReplicationService + calls ReplaceAllAsync; DeploymentManagerActor Tells ReplicationMessages types; AkkaHostedService constructs the actor). Combining also strengthens Task 14's own invariant: never both mechanisms, never neither. Registered 8 tables; notification_lists + smtp_configurations deliberately NOT registered. Deleted ReplaceAllAsync as UNSAFE (a mass DELETE on a replicated table would be captured and shipped), and its test replaced by a comment explaining that. The positional-arg hazard was REAL: 4 DeploymentManagerActor test call sites bound wrong args; converted to named where possible - Props.Create is an expression tree and rejects OUT-OF-POSITION named args, so the rest are padded positionally. Task 7's 'not yet' test INVERTED (exact in both directions) + new security-named test for the SMTP tables + composite-PK test. Harness's temporary registration deleted, so the convergence suites now prove the cutover. My expected table list was mis-sorted: ordinal puts '_' (0x5F) before 'b', so data_connection_definitions precedes database_connections.", "commit": "037798b3"}, + {"id": 15, "subject": "Task 15: Delete SiteReplicationActor and its messages", "status": "completed", "classification": "high-risk", "blockedBy": [14], "note": "LANDED AS ONE COMMIT with tasks 15+16 - PLAN DEFECT: they cannot compile separately (SiteReplicationActor takes a ReplicationService + calls ReplaceAllAsync; DeploymentManagerActor Tells ReplicationMessages types; AkkaHostedService constructs the actor). Combining also strengthens Task 14's own invariant: never both mechanisms, never neither. Registered 8 tables; notification_lists + smtp_configurations deliberately NOT registered. Deleted ReplaceAllAsync as UNSAFE (a mass DELETE on a replicated table would be captured and shipped), and its test replaced by a comment explaining that. The positional-arg hazard was REAL: 4 DeploymentManagerActor test call sites bound wrong args; converted to named where possible - Props.Create is an expression tree and rejects OUT-OF-POSITION named args, so the rest are padded positionally. Task 7's 'not yet' test INVERTED (exact in both directions) + new security-named test for the SMTP tables + composite-PK test. Harness's temporary registration deleted, so the convergence suites now prove the cutover. My expected table list was mis-sorted: ordinal puts '_' (0x5F) before 'b', so data_connection_definitions precedes database_connections.", "commit": "037798b3"}, + {"id": 16, "subject": "Task 16: Clean up DeploymentManagerActor and AkkaHostedService", "status": "completed", "classification": "standard", "blockedBy": [15], "note": "LANDED AS ONE COMMIT with tasks 15+16 - PLAN DEFECT: they cannot compile separately (SiteReplicationActor takes a ReplicationService + calls ReplaceAllAsync; DeploymentManagerActor Tells ReplicationMessages types; AkkaHostedService constructs the actor). Combining also strengthens Task 14's own invariant: never both mechanisms, never neither. Registered 8 tables; notification_lists + smtp_configurations deliberately NOT registered. Deleted ReplaceAllAsync as UNSAFE (a mass DELETE on a replicated table would be captured and shipped), and its test replaced by a comment explaining that. The positional-arg hazard was REAL: 4 DeploymentManagerActor test call sites bound wrong args; converted to named where possible - Props.Create is an expression tree and rejects OUT-OF-POSITION named args, so the rest are padded positionally. Task 7's 'not yet' test INVERTED (exact in both directions) + new security-named test for the SMTP tables + composite-PK test. Harness's temporary registration deleted, so the convergence suites now prove the cutover. My expected table list was mis-sorted: ordinal puts '_' (0x5F) before 'b', so data_connection_definitions precedes database_connections.", "commit": "037798b3"}, {"id": 17, "subject": "Task 17: Config-key cleanup (incl. ConfigFetchRetryCount; 10 appsettings files)", "status": "pending", "classification": "standard", "blockedBy": [16]}, {"id": 18, "subject": "Task 18: Two-node convergence suite for the Phase 2 tables", "status": "pending", "classification": "high-risk", "blockedBy": [17], "note": "Non-vacuity must be proven directly (Phase 1 never did the mismatched-key run the original plan claimed)."}, {"id": 19, "subject": "Task 19: Rig configuration (MaxOplogRows/MaxOplogAge + MaxBatchSize per D6)", "status": "pending", "classification": "small", "blockedBy": [17]}, From 3364145d63ef2d69bdf95da7e3d2727df5410927 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Mon, 20 Jul 2026 04:22:30 -0400 Subject: [PATCH 46/55] docs(localdb): resume state through the cutover (tasks 1-16 done, task 17 next) Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts --- docs/plans/2026-07-20-phase2-resume-state.md | 266 ++++++++++--------- 1 file changed, 146 insertions(+), 120 deletions(-) diff --git a/docs/plans/2026-07-20-phase2-resume-state.md b/docs/plans/2026-07-20-phase2-resume-state.md index 2786f585..05ab64e1 100644 --- a/docs/plans/2026-07-20-phase2-resume-state.md +++ b/docs/plans/2026-07-20-phase2-resume-state.md @@ -1,131 +1,154 @@ -# LocalDb Phase 2 — resume state (written 2026-07-20, pre-compaction) +# LocalDb Phase 2 — resume state (updated 2026-07-20, pre-compaction #2) Scratch handoff for picking the plan back up. Delete once Phase 2 lands. **Plan:** [`2026-07-19-localdb-adoption-phase2.md`](2026-07-19-localdb-adoption-phase2.md) (execute with `superpowers-extended-cc:executing-plans`; task state in -`…-phase2.md.tasks.json`) -**Branch:** `feat/localdb-phase2` (from `feat/localdb-phase1`, which is **not** merged/pushed) -**Tree:** clean @ `19ab0ac9`. Build 0 warnings. **Nothing pushed to origin yet.** +`…-phase2.md.tasks.json`, which carries a per-task deviation note) +**Branch:** `feat/localdb-phase2` (from `feat/localdb-phase1`, which is **not** merged) +**Tree:** clean @ `0ad11d6b`. Build 0 warnings. 25 commits ahead of `origin/feat/localdb-phase2`. --- ## Where we are -**Tasks 1–6 are DONE (waves 0, 1, 2). Next up: Wave 3 = Tasks 7, 8, 9.** - -Task 1's gate is closed with verdict **PROCEED**; its earlier STOP was superseded (the -`disk I/O error` storm was **observer-induced** — host-side `sqlite3` reads of live -bind-mounted WAL files reset the WAL under the container, not a product defect; write-up in -[`docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md`](../known-issues/2026-07-20-localdb-disk-io-error-under-load.md) -§0/§11a). +**Tasks 1–16 are DONE. The cutover has landed. Next up: Task 17.** | Task | State | |---|---| -| 1 Rig soak | done — gate PROCEED | -| 2 Decision record | done — gate doc CLOSED, all §5 questions answered inline | -| 3 `StoreAndForwardSchema.Apply` | done | -| 4 `SiteStorageSchema.Apply` | done | -| 5 S&F store → `ILocalDb` | done | -| 6 `SiteStorageService` → `ILocalDb` | done | -| **7–21** | **not started, no code written** | +| 1–6 | done (waves 0–2) — see the git log | +| 7 `SiteLocalDbSetup` DDL | done | +| 8 `sf_messages` migrator | done | +| 9 config-table migrator | done | +| 10 S&F CDC convergence specs | done | +| 11 resync + config convergence specs | done | +| 12 active-node purge pin | done | +| 13 guarded-write scope check | done | +| **14 + 15 + 16 the CUTOVER** | **done — landed as ONE commit `037798b3`** | +| **17–21** | **not started** | -**Verification at the wave-2 boundary:** full solution build 0 warnings; SiteRuntime 532, -Host 318, AuditLog 355, StoreAndForward 153, ExternalSystemGateway 142, HealthMonitoring 97 -— **1597 passed / 0 failed**. +**Verification at the cutover boundary:** build 0 warnings; SiteRuntime 512, StoreAndForward 130, +Host 330, AuditLog 355, ExternalSystemGateway 142, HealthMonitoring 97, LocalDb integration 16 — +**all pass, 0 failures.** (SiteRuntime/StoreAndForward fell from 533/153 as 6 obsolete test files +were deleted.) -### Task 1's measured numbers (clean re-run, 2026-07-20) +### The bespoke replicators are gone -Six consecutive 60 s intervals, copy-based `snap()` sampling on restarted nodes: -`sf_messages` **0.80 rows/sec** insert, **0/sec** retry-UPDATE, oplog 0, `native_alarm_state` 0, -max `payload_json` **76 B**, max `config_json` **721 B** (rig — *not* representative), -**zero** SQLite errors in 30 min, both site-a nodes converged at 3564 rows. Honest gap: 0.80/s -is ~1.6 % of the 50/s ceiling and the retry path was never exercised — fine only because that -ceiling is structural, not empirical. +`SiteReplicationActor`, `ReplicationMessages.cs`, `ReplicationService` and +`StoreAndForwardStorage.ReplaceAllAsync` are deleted. Ten tables now replicate via LocalDb CDC: +the Phase 1 pair plus `sf_messages` and the 7 site config tables. -### Next: Wave 3 (Tasks 7, 8, 9) +`notification_lists` and `smtp_configurations` are created but **never registered** — permanently +empty by design, and registering them would open a standing replication channel whose only +historical payload was plaintext SMTP passwords. That decision is pinned by its own +security-named test and stated in `OnReady`. -Task 7 extends `SiteLocalDbSetup.OnReady` with the new DDL (**not** registered yet — registration -is Task 14, the cutover). Tasks 8 and 9 are the legacy migrators and are both **high-risk**. -Ordering inside `OnReady` is load-bearing: **DDL → `RegisterReplicated` → migrate**. Rows written -before registration are invisible to the peer forever, silently. +--- + +## Next: Task 17 (config-key cleanup) + +Partly **required**, not optional: `SiteRuntimeOptions.cs:66` still documents +`ConfigFetchRetryCount` as "Consumed by `SiteReplicationActor`", which no longer exists. + +Per the plan: relax `StartupValidator.cs:113` (`SiteDbPath`) and +`StoreAndForwardOptionsValidator.cs:19-21` (`SqliteDbPath`) to migration-only; delete +`ReplicationEnabled` and `ConfigFetchRetryCount` outright with their validator rules and config +entries; **leave the two path keys present in configs** with a comment — the legacy migrator reads +them, and removing them would strand un-migrated data on a node that has not yet started once. +**10** `appsettings.Site.json` files, including `deploy/wonder-app-vd03/`. + +Then 18 (two-node convergence suite), 19 (rig config — `MaxBatchSize = 16`), 20 (live gate), +21 (docs truth pass). --- ## Findings that change the plan (carry these forward) -From [`2026-07-19-localdb-phase2-soak.md`](2026-07-19-localdb-phase2-soak.md): +### From Task 1 (unchanged) -- **D6's premise is wrong, its risk is real.** The plan says `config_json` is "documented to - exceed 128 KB per row." The source actually says the *escaped Akka envelope* blew the 128 KB - frame and the raw JSON "only needs to be ~60-70 KB". So the largest known production row is - ~60–70 KB → **no stop condition**. But `MaxBatchSize=500` is unsafe (500 × 70 KB ≈ 35 MB vs a - 4 MB gRPC cap) → **Task 19 must set `LocalDb:Replication:MaxBatchSize = 16`.** This is the one - firmly evidence-backed number. -- **D4 is wrong.** `native_alarm_state` is not "unbounded by design" — `NativeAlarmActor` - coalesces per `SourceReference` on a 100 ms flush (`NativeAlarmActor.cs:473-502`), so - worst case is `distinct_source_refs × 10/sec`. -- **`sf_messages` has a hard 50 rows/sec ceiling** — `SweepBatchLimit` (500) ÷ - `RetryTimerInterval` (10 s). A ceiling, not an estimate. -- **Task 1 step 7's stop condition is much weaker than written.** Exceeding the oplog caps - prunes and flags `needs_snapshot` → graceful snapshot resync (`OplogStore.cs:109-138`, - `MaintenanceBackgroundService.cs:57`). Not data loss, not a wedge. +- **D6's premise is wrong, its risk is real.** Largest known production `config_json` is ~60–70 KB, + not >128 KB → no stop condition. But `MaxBatchSize=500` × 70 KB ≈ 35 MB vs a 4 MB gRPC cap → + **Task 19 must set `LocalDb:Replication:MaxBatchSize = 16`.** The one firmly evidence-backed number. +- **D4 is wrong.** `native_alarm_state` is bounded by per-`SourceReference` coalescing on a 100 ms + flush (`NativeAlarmActor.cs:473-502`). +- **`sf_messages` has a hard 50 rows/sec ceiling** — `SweepBatchLimit` (500) ÷ `RetryTimerInterval` (10 s). +- **Task 1 step 7's stop condition is weaker than written.** Exceeding the oplog caps prunes and + flags `needs_snapshot` → graceful resync, not data loss. -The plan's own Task-1 method also needed four corrections (metrics sidecar, safe sampling, -no alarm generator on this rig, template 4 undeployable) — all recorded in the soak doc §1. +Measured (six 60 s intervals, copy-based `snap()`): `sf_messages` **0.80 rows/sec** insert, **0/sec** +retry-UPDATE, oplog 0, max `payload_json` 76 B, max `config_json` 721 B (rig — *not* representative), +zero SQLite errors in 30 min. -### Found during waves 1–2 (new) +### From waves 1–2 -- **LATENT PHASE-1 DEFECT, found and fixed in Task 6.** The plan assumed directory creation - moved to LocalDb along with file ownership. **It did not.** The LocalDb library never creates - the parent directory, and `SqliteLocalDb` opens the file **eagerly in its constructor**, so a - missing directory is a **hard boot failure** (`SQLite Error 14: unable to open database file`), - not a degraded start. The default site config uses the **relative** path - `./data/site-localdb.db`, so any site node without a pre-existing `data/` fails to boot; the - docker rig escapes only because its volume mount creates `/app/data`. Latent since Phase 1 made - `LocalDb:Path` required. Fixed in ScadaBridge via `SiteLocalDbDirectory.Ensure(config)` called - before `AddZbLocalDb`, pinned by `Host.Tests/SiteLocalDbDirectoryTests`. - **OPEN DECISION FOR THE USER:** the real fix arguably belongs in the `ZB.MOM.WW.LocalDb` - library (it owns the file) — **every other LocalDb consumer still has this gap.** Not done - here because a library change means a version bump and is outside this plan's scope. -- **Silent contract change on `SiteStorageService.CreateConnection()`.** It used to return an - **unopened** connection that callers opened; it now returns an **already-open** LocalDb - connection, and calling `Open`/`OpenAsync` on one **throws**. `SiteExternalSystemRepository` - dropped 5 `OpenAsync` calls. Anything new that consumes this member must not open it. -- **LocalDb has NO in-memory mode.** `LocalDbOptions.Path` is a filesystem path, so every - fixture that used `Mode=Memory;Cache=Shared` had to move to a real temp file. Expect this for - any remaining store that gets rewired. -- **Test fallout from a store constructor change is ~7× what the plan estimates.** Task 5's plan - named "fixtures" in one project; the real blast radius was **40 files across 7 test projects**. - Budget accordingly for Tasks 14/15. -- **`tests/ZB.MOM.WW.ScadaBridge.TestSupport`** is a NEW shared, non-test library holding - `TestLocalDb` (`Create`/`CreateTemp`/`DeleteFiles`; dispose **before** deleting — the master - connection anchors the WAL). Use it rather than copying fixtures. Wired into AuditLog, - ExternalSystemGateway, HealthMonitoring, IntegrationTests, SiteRuntime, StoreAndForward tests. -- **Where behaviour moved owner, retarget tests — do not delete them with the code.** WAL genuinely - is LocalDb's job (test retargeted in place); directory creation is **not** (test moved to - `Host.Tests`). Both directions occurred, so check which owner actually holds the guarantee. +- **LATENT PHASE-1 DEFECT, fixed in-app only.** The LocalDb library never creates the parent + directory of `LocalDb:Path`, and `SqliteLocalDb` opens the file **eagerly in its constructor**, so + a missing directory is a **hard boot failure** (`SQLite Error 14`). The default site config uses + the **relative** `./data/site-localdb.db`; the docker rig escapes only because its volume mount + creates `/app/data`. Fixed via `SiteLocalDbDirectory.Ensure(config)` before `AddZbLocalDb`, pinned + by `Host.Tests/SiteLocalDbDirectoryTests`. + **OPEN DECISION FOR THE USER:** the real fix arguably belongs in the `ZB.MOM.WW.LocalDb` library + — **every other consumer still has this gap.** Not done here: a library change means a version + bump and is outside this plan's scope. +- **`SiteStorageService.CreateConnection()` contract flipped** — returns an **already-open** + connection; calling `Open`/`OpenAsync` on one **throws**. +- **LocalDb has NO in-memory mode.** Every `Mode=Memory;Cache=Shared` fixture became a real temp file. +- **A store constructor change's test blast radius is ~7× the plan's estimate** (Task 5: 40 files, + 7 projects). +- **`tests/ZB.MOM.WW.ScadaBridge.TestSupport`** holds `TestLocalDb` (dispose **before** deleting — + the master connection anchors the WAL). Use it; don't copy fixtures. +- **Where behaviour moved owner, retarget tests — don't delete them with the code.** WAL is genuinely + LocalDb's; directory creation is **not**. Both directions occurred. + +### From waves 3–5 (new) + +- **PLAN DEFECT — Tasks 14/15/16 cannot compile separately.** `SiteReplicationActor` takes a + `ReplicationService` and calls `ReplaceAllAsync`; `DeploymentManagerActor` Tells types from + `ReplicationMessages.cs`; `AkkaHostedService` constructs the actor. Landed as one commit, which + also strengthens Task 14's own invariant (never both mechanisms, never neither). +- **The legacy-copy column intersection.** The plan said "migrate all 16 columns"; that would be a + data-loss bug. An older legacy file lacks `execution_id`/`parent_execution_id`/ + `last_attempt_at_ms`, naming a missing column throws `no such column`, and the existing reader + treats that as an unrecognised shape and **silently discards every row**. The copy intersects the + legacy and current column sets, with a required-column (PK) guard so the tolerance cannot degrade + into copying NULL-keyed rows. +- **`MigratorColumnLists_MatchTheLiveSchema`** (added beyond the plan). A column-list mismatch is + invisible at runtime in BOTH directions — a typo is silently dropped by the intersection, a + missing column silently leaves data behind. `SiteStorageTables`/`LegacyTable` are `internal` for it. +- **Absence assertions need a positive control.** `MessageAddedThenRemoved_NeverConvergesToPresent` + PASSED with `sf_messages` unregistered entirely — an absent row is also what a pair replicating + nothing looks like. Fixed with a control row that must converge in the same window. +- **The positional-argument hazard was REAL.** Removing `DeploymentManagerActor`'s optional + `IActorRef? replicationActor` shifted 6 trailing optionals and 4 test call sites bound the wrong + arguments, some with no compile error. **`Props.Create` builds an expression tree, which rejects + OUT-OF-POSITION named arguments** — so named args work only if in position, and the rest must be + padded positionally. +- **`ReplaceAllAsync` was unsafe to keep, not merely unused.** A mass DELETE on a now-replicated + table would be captured and shipped to the peer. +- **`LocalDbSitePairHarness`** is the shared two-node fixture (Phase 1 + both Phase 2 suites; Task 18 + makes four). Its `Phase2ReplicatedTables` list is **literal, not derived from production code**, so + a registration that drifts fails the tests instead of agreeing with itself. +- **Ordinal sort gotcha:** `_` (0x5F) sorts before `b` (0x62), so `data_connection_definitions` + precedes `database_connections`. +- **Never `git checkout ` to undo a non-vacuity experiment** — it reverts to HEAD and takes + in-progress work with it (cost one redo of Task 8). Use `cp` backups. --- ## Rig operational gotchas (hard-won; not obvious from the repo) - **`ExternalSystem.Call` does NOT buffer to store-and-forward — only `ExternalSystem.CachedCall` - does.** Using `Call` gives HTTP traffic and zero `sf_messages` rows. This is the single most - important detail for generating S&F load. -- **Never** query the bind-mounted SQLite files from the macOS host while containers run. Use - the plan's `snap` helper, or `docker run --rm -v :/d alpine/sqlite3 …`. -- **No `curl` in the `aspnet:10.0` image**, and metrics port 8084 is not published. Use a - sidecar: `docker run --rm --network container:scadabridge-site-a-a curlimages/curl:latest -s localhost:8084/metrics` -- **Seeded template 4 (`Motor Controller`) is not deployable** — 34 pre-deployment validation - errors. The soak used a purpose-built `SoakGenerator` template instead. -- CLI: `template script update` requires `--name` and `--trigger-type` even to change only - `--code`. In zsh, don't put auth flags in an unquoted variable (no word-splitting). + does.** The single most important detail for generating S&F load. +- **Never** query the bind-mounted SQLite files from the macOS host while containers run. Use the + plan's `snap` helper, or `docker run --rm -v :/d alpine/sqlite3 …`. +- **No `curl` in the `aspnet:10.0` image**, and metrics port 8084 is not published. Use a sidecar: + `docker run --rm --network container:scadabridge-site-a-a curlimages/curl:latest -s localhost:8084/metrics` +- **Seeded template 4 (`Motor Controller`) is not deployable** — 34 validation errors. The soak used + a purpose-built `SoakGenerator` template. +- CLI: `template script update` requires `--name` and `--trigger-type` even to change only `--code`. + In zsh, don't put auth flags in an unquoted variable. - Rig auth: `--url http://localhost:9000 --username multi-role --password password`. - `LdapGroupMappings` role strings must be canonical `Designer`/`Deployer`; central caches them - at startup, so restart central after seeding them. - ---- + `LdapGroupMappings` roles must be canonical `Designer`/`Deployer`; central caches them at startup. ## Rig state as left @@ -134,39 +157,42 @@ Reseeded and healthy; both site-a nodes restarted after the poisoning incident. **Needs cleanup before the Task 20 live gate:** - `ExternalSystemDefinitions` id 1 is still repointed to `http://127.0.0.1:9` — restore to `http://scadabridge-restapi:5200`. -- Template `SoakGenerator` (id 2021) and instances `soakgen-1..4` (ids 5–8) are still deployed - on site-a and **still generating load**. +- Template `SoakGenerator` (id 2021) and instances `soakgen-1..4` (ids 5–8) are still deployed on + site-a and **still generating load**. --- -## Commits this session +## Commits (this branch, newest last) | Commit | What | |---|---| -| `25463d52` | plan review pass — corrections from code verification (D1/D3/D6, Task 1 method, Task 14) | -| `cf46e596` | fix(rig): `seed-sites.sh` canonical `Designer`/`Deployer` role names | -| `fc553bd9` | soak findings (gate verdict, since superseded) | -| `82c869a1` | task record: gate verdict | -| `e9e11d63` | root-cause brief for the disk-I/O issue | -| `8652eab9` | root cause + hardening: extended SQLite error codes, `SiteAuditTelemetryActor` async-context fix, `reseed.sh` init-script fix | -| `d512572d` | this resume-state doc (first version) | -| `12eb97e0` | **Task 2** — gate CLOSED, all §5 questions answered + Task 1's clean re-run | -| `9e3239c5` | **Task 3** — extract `StoreAndForwardSchema.Apply` | -| `ac5eb12c` | **Task 4** — extract `SiteStorageSchema.Apply` | -| `3dfb288b` | task state: tasks 3–4 + deviations | -| `f2efeb37` | **Tasks 5+6** — both stores take `ILocalDb`; latent directory defect fixed; `TestSupport` lib added | -| `19ab0ac9` | task state: tasks 5–6, the directory defect, the `CreateConnection` contract change | +| `25463d52` … `d512572d` | plan review, soak, root-cause + hardening, first resume doc | +| `12eb97e0` | **Task 2** — gate CLOSED | +| `9e3239c5` | **Task 3** — `StoreAndForwardSchema.Apply` | +| `ac5eb12c` | **Task 4** — `SiteStorageSchema.Apply` | +| `f2efeb37` | **Tasks 5+6** — both stores take `ILocalDb`; latent directory defect fixed; `TestSupport` added | +| `fefbbb31` | resume state through wave 2 | +| `f8aa02e2` | **Task 7** — Phase 2 DDL in the consolidated DB (not yet registered) | +| `bdc0dffe` | **Task 8** — migrate legacy `store-and-forward.db` | +| `5ddc7eed` | **Task 9** — migrate legacy `scadabridge.db` config tables | +| `2bbe6631` | **Task 10** — S&F CDC convergence specs (+ `LocalDbSitePairHarness` extraction) | +| `c56bf4ae` | **Task 11** — resync + config convergence specs | +| `79ce5161` | **Tasks 12+13** — purge pin; guarded-write scope | +| `037798b3` | **Tasks 14+15+16 — THE CUTOVER** | +| `0ad11d6b` | task state for the cutover | -Two known flakes, do not chase: `InstanceActorChildAttributeRaceTests` (intermittent under full-suite -load) and `ParentExecutionIdCorrelationTests` (~91 s / times out on a cold MSSQL fixture, ~1 s warm). -Neither reproduced during waves 1–2. +Two known flakes, do not chase: `InstanceActorChildAttributeRaceTests` (intermittent under +full-suite load) and `ParentExecutionIdCorrelationTests` (~91 s / times out on a cold MSSQL +fixture, ~1 s warm). Neither reproduced during waves 1–5. ## Execution notes -- Waves 1–2 were run **sequentially in the main worktree**, not as the plan's parallel - worktree agents — the tasks were small enough that worktree setup + merge cost more than it - saved, and sequential execution removes the git race the isolation rule exists to prevent. -- Subagents *were* used for the bulk mechanical fixture rewiring (4 of them, disjoint file sets, - explicitly forbidden from running git). **Their reported results were re-verified - independently** — one agent's partial run had missed 2 real failures that a full-suite run - caught. Re-run the suites yourself; do not take an agent's pass at face value. +- Waves ran **sequentially in the main worktree**, not as the plan's parallel worktree agents — the + tasks were small enough that worktree setup + merge cost more than it saved, and sequential + execution removes the git race the isolation rule exists to prevent. +- Subagents were used only in waves 1–2, for bulk mechanical fixture rewiring. **Their reported + results were re-verified independently** — one agent's partial run had missed 2 real failures that + a full-suite run caught. Re-run the suites yourself. +- Every new test was checked for **non-vacuity** by breaking the thing it claims to pin (removing + DDL, removing the `Migrate` call, unregistering a table, commenting out the purge). This caught one + genuinely vacuous test. Keep doing it. From 605e56829e5f524afcf13917f2cfa6463a9fc6e2 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Mon, 20 Jul 2026 04:35:11 -0400 Subject: [PATCH 47/55] chore(config): retire ReplicationEnabled, make legacy db paths migration-only MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit LocalDb Phase 2 deleted the bespoke replicators, so three config keys changed meaning or died outright: - ScadaBridge:StoreAndForward:ReplicationEnabled is fully dead. Deleted the property, its 10 config entries, and the 5 test references. - SqliteDbPath / SiteDbPath are now migration-only: they name the legacy files SiteLocalDbLegacyMigrator drains at boot, not live databases. Both mandatory rules are relaxed accordingly (StartupValidator's Site-only Require, and the S&F validator's non-empty rule) — an absent value now means "nothing to migrate", so an already-migrated node can drop the key. DatabaseOptions- Validator still rejects a present-but-blank value. - SiteRuntime:ConfigFetchRetryCount's only reader was SiteReplicationActor. Deleted with its validator rule. The two path keys stay present in every config, now with a comment explaining why: removing them would strand un-migrated data on a node that has not yet started once. Both relaxations are pinned by the inverse of the test they replace (Site_MissingSiteDbPath_IsAccepted..., EmptySqliteDbPath_IsAccepted...), each verified to fail with the old rule restored. Note: deploy/wonder-app-vd03/appsettings.Site.json is under a gitignored deploy/ tree, so its edit is local-only and must be repeated on the box. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts --- .../site-x-node-a/appsettings.Site.json | 11 ++++++++-- .../site-x-node-b/appsettings.Site.json | 11 ++++++++-- docker/site-a-node-a/appsettings.Site.json | 11 ++++++++-- docker/site-a-node-b/appsettings.Site.json | 11 ++++++++-- docker/site-b-node-a/appsettings.Site.json | 11 ++++++++-- docker/site-b-node-b/appsettings.Site.json | 11 ++++++++-- docker/site-c-node-a/appsettings.Site.json | 11 ++++++++-- docker/site-c-node-b/appsettings.Site.json | 11 ++++++++-- .../StartupValidator.cs | 11 ++++++---- .../appsettings.Site.json | 11 ++++++++-- .../SiteRuntimeOptions.cs | 7 ------- .../SiteRuntimeOptionsValidator.cs | 4 ---- .../StoreAndForwardOptions.cs | 11 ++++++---- .../StoreAndForwardOptionsValidator.cs | 20 ++++++++++--------- .../StartupValidatorTests.cs | 13 +++++++++--- .../ParkedMessageHandlerActorTests.cs | 1 - .../ParkedOperationRelayTests.cs | 1 - .../StoreAndForwardOptionsTests.cs | 3 --- .../StoreAndForwardOptionsValidatorTests.cs | 13 +++++++++--- 19 files changed, 126 insertions(+), 57 deletions(-) diff --git a/docker-env2/site-x-node-a/appsettings.Site.json b/docker-env2/site-x-node-a/appsettings.Site.json index edcd4240..b7ddf335 100644 --- a/docker-env2/site-x-node-a/appsettings.Site.json +++ b/docker-env2/site-x-node-a/appsettings.Site.json @@ -20,6 +20,10 @@ "MinNrOfMembers": 1 }, "Database": { + // Migration-only as of LocalDb Phase 2. The site config tables now live in the + // consolidated LocalDb database (LocalDb:Path). SiteDbPath is read once at boot to drain + // a pre-Phase-2 scadabridge.db, and is unused after that - keep it until this node has + // started once. "SiteDbPath": "/app/data/scadabridge.db" }, "DataConnection": { @@ -29,8 +33,11 @@ "SeedReadTimeout": "00:00:30" }, "StoreAndForward": { - "SqliteDbPath": "/app/data/store-and-forward.db", - "ReplicationEnabled": true + // Migration-only as of LocalDb Phase 2. The store-and-forward buffer now lives in the + // consolidated LocalDb database (LocalDb:Path) as the replicated sf_messages table. + // SqliteDbPath is read once at boot by SiteLocalDbLegacyMigrator to drain a pre-Phase-2 + // file, and is unused after that - keep it until this node has started once. + "SqliteDbPath": "/app/data/store-and-forward.db" }, "Communication": { "CentralContactPoints": [ diff --git a/docker-env2/site-x-node-b/appsettings.Site.json b/docker-env2/site-x-node-b/appsettings.Site.json index 459dbf04..63abb3be 100644 --- a/docker-env2/site-x-node-b/appsettings.Site.json +++ b/docker-env2/site-x-node-b/appsettings.Site.json @@ -20,6 +20,10 @@ "MinNrOfMembers": 1 }, "Database": { + // Migration-only as of LocalDb Phase 2. The site config tables now live in the + // consolidated LocalDb database (LocalDb:Path). SiteDbPath is read once at boot to drain + // a pre-Phase-2 scadabridge.db, and is unused after that - keep it until this node has + // started once. "SiteDbPath": "/app/data/scadabridge.db" }, "DataConnection": { @@ -29,8 +33,11 @@ "SeedReadTimeout": "00:00:30" }, "StoreAndForward": { - "SqliteDbPath": "/app/data/store-and-forward.db", - "ReplicationEnabled": true + // Migration-only as of LocalDb Phase 2. The store-and-forward buffer now lives in the + // consolidated LocalDb database (LocalDb:Path) as the replicated sf_messages table. + // SqliteDbPath is read once at boot by SiteLocalDbLegacyMigrator to drain a pre-Phase-2 + // file, and is unused after that - keep it until this node has started once. + "SqliteDbPath": "/app/data/store-and-forward.db" }, "Communication": { "CentralContactPoints": [ diff --git a/docker/site-a-node-a/appsettings.Site.json b/docker/site-a-node-a/appsettings.Site.json index 53be3432..f33407c3 100644 --- a/docker/site-a-node-a/appsettings.Site.json +++ b/docker/site-a-node-a/appsettings.Site.json @@ -21,6 +21,10 @@ "MinNrOfMembers": 1 }, "Database": { + // Migration-only as of LocalDb Phase 2. The site config tables now live in the + // consolidated LocalDb database (LocalDb:Path). SiteDbPath is read once at boot to drain + // a pre-Phase-2 scadabridge.db, and is unused after that - keep it until this node has + // started once. "SiteDbPath": "/app/data/scadabridge.db" }, "DataConnection": { @@ -30,8 +34,11 @@ "SeedReadTimeout": "00:00:30" }, "StoreAndForward": { - "SqliteDbPath": "/app/data/store-and-forward.db", - "ReplicationEnabled": true + // Migration-only as of LocalDb Phase 2. The store-and-forward buffer now lives in the + // consolidated LocalDb database (LocalDb:Path) as the replicated sf_messages table. + // SqliteDbPath is read once at boot by SiteLocalDbLegacyMigrator to drain a pre-Phase-2 + // file, and is unused after that - keep it until this node has started once. + "SqliteDbPath": "/app/data/store-and-forward.db" }, "Communication": { "CentralContactPoints": [ diff --git a/docker/site-a-node-b/appsettings.Site.json b/docker/site-a-node-b/appsettings.Site.json index 8e7bd55c..78062879 100644 --- a/docker/site-a-node-b/appsettings.Site.json +++ b/docker/site-a-node-b/appsettings.Site.json @@ -21,6 +21,10 @@ "MinNrOfMembers": 1 }, "Database": { + // Migration-only as of LocalDb Phase 2. The site config tables now live in the + // consolidated LocalDb database (LocalDb:Path). SiteDbPath is read once at boot to drain + // a pre-Phase-2 scadabridge.db, and is unused after that - keep it until this node has + // started once. "SiteDbPath": "/app/data/scadabridge.db" }, "DataConnection": { @@ -30,8 +34,11 @@ "SeedReadTimeout": "00:00:30" }, "StoreAndForward": { - "SqliteDbPath": "/app/data/store-and-forward.db", - "ReplicationEnabled": true + // Migration-only as of LocalDb Phase 2. The store-and-forward buffer now lives in the + // consolidated LocalDb database (LocalDb:Path) as the replicated sf_messages table. + // SqliteDbPath is read once at boot by SiteLocalDbLegacyMigrator to drain a pre-Phase-2 + // file, and is unused after that - keep it until this node has started once. + "SqliteDbPath": "/app/data/store-and-forward.db" }, "Communication": { "CentralContactPoints": [ diff --git a/docker/site-b-node-a/appsettings.Site.json b/docker/site-b-node-a/appsettings.Site.json index cd13388a..e0351fef 100644 --- a/docker/site-b-node-a/appsettings.Site.json +++ b/docker/site-b-node-a/appsettings.Site.json @@ -21,6 +21,10 @@ "MinNrOfMembers": 1 }, "Database": { + // Migration-only as of LocalDb Phase 2. The site config tables now live in the + // consolidated LocalDb database (LocalDb:Path). SiteDbPath is read once at boot to drain + // a pre-Phase-2 scadabridge.db, and is unused after that - keep it until this node has + // started once. "SiteDbPath": "/app/data/scadabridge.db" }, "DataConnection": { @@ -30,8 +34,11 @@ "SeedReadTimeout": "00:00:30" }, "StoreAndForward": { - "SqliteDbPath": "/app/data/store-and-forward.db", - "ReplicationEnabled": true + // Migration-only as of LocalDb Phase 2. The store-and-forward buffer now lives in the + // consolidated LocalDb database (LocalDb:Path) as the replicated sf_messages table. + // SqliteDbPath is read once at boot by SiteLocalDbLegacyMigrator to drain a pre-Phase-2 + // file, and is unused after that - keep it until this node has started once. + "SqliteDbPath": "/app/data/store-and-forward.db" }, "Communication": { "CentralContactPoints": [ diff --git a/docker/site-b-node-b/appsettings.Site.json b/docker/site-b-node-b/appsettings.Site.json index 9b373702..92610887 100644 --- a/docker/site-b-node-b/appsettings.Site.json +++ b/docker/site-b-node-b/appsettings.Site.json @@ -21,6 +21,10 @@ "MinNrOfMembers": 1 }, "Database": { + // Migration-only as of LocalDb Phase 2. The site config tables now live in the + // consolidated LocalDb database (LocalDb:Path). SiteDbPath is read once at boot to drain + // a pre-Phase-2 scadabridge.db, and is unused after that - keep it until this node has + // started once. "SiteDbPath": "/app/data/scadabridge.db" }, "DataConnection": { @@ -30,8 +34,11 @@ "SeedReadTimeout": "00:00:30" }, "StoreAndForward": { - "SqliteDbPath": "/app/data/store-and-forward.db", - "ReplicationEnabled": true + // Migration-only as of LocalDb Phase 2. The store-and-forward buffer now lives in the + // consolidated LocalDb database (LocalDb:Path) as the replicated sf_messages table. + // SqliteDbPath is read once at boot by SiteLocalDbLegacyMigrator to drain a pre-Phase-2 + // file, and is unused after that - keep it until this node has started once. + "SqliteDbPath": "/app/data/store-and-forward.db" }, "Communication": { "CentralContactPoints": [ diff --git a/docker/site-c-node-a/appsettings.Site.json b/docker/site-c-node-a/appsettings.Site.json index 759cc46d..5cdb0125 100644 --- a/docker/site-c-node-a/appsettings.Site.json +++ b/docker/site-c-node-a/appsettings.Site.json @@ -21,6 +21,10 @@ "MinNrOfMembers": 1 }, "Database": { + // Migration-only as of LocalDb Phase 2. The site config tables now live in the + // consolidated LocalDb database (LocalDb:Path). SiteDbPath is read once at boot to drain + // a pre-Phase-2 scadabridge.db, and is unused after that - keep it until this node has + // started once. "SiteDbPath": "/app/data/scadabridge.db" }, "DataConnection": { @@ -30,8 +34,11 @@ "SeedReadTimeout": "00:00:30" }, "StoreAndForward": { - "SqliteDbPath": "/app/data/store-and-forward.db", - "ReplicationEnabled": true + // Migration-only as of LocalDb Phase 2. The store-and-forward buffer now lives in the + // consolidated LocalDb database (LocalDb:Path) as the replicated sf_messages table. + // SqliteDbPath is read once at boot by SiteLocalDbLegacyMigrator to drain a pre-Phase-2 + // file, and is unused after that - keep it until this node has started once. + "SqliteDbPath": "/app/data/store-and-forward.db" }, "Communication": { "CentralContactPoints": [ diff --git a/docker/site-c-node-b/appsettings.Site.json b/docker/site-c-node-b/appsettings.Site.json index 0aeba3aa..6460c296 100644 --- a/docker/site-c-node-b/appsettings.Site.json +++ b/docker/site-c-node-b/appsettings.Site.json @@ -21,6 +21,10 @@ "MinNrOfMembers": 1 }, "Database": { + // Migration-only as of LocalDb Phase 2. The site config tables now live in the + // consolidated LocalDb database (LocalDb:Path). SiteDbPath is read once at boot to drain + // a pre-Phase-2 scadabridge.db, and is unused after that - keep it until this node has + // started once. "SiteDbPath": "/app/data/scadabridge.db" }, "DataConnection": { @@ -30,8 +34,11 @@ "SeedReadTimeout": "00:00:30" }, "StoreAndForward": { - "SqliteDbPath": "/app/data/store-and-forward.db", - "ReplicationEnabled": true + // Migration-only as of LocalDb Phase 2. The store-and-forward buffer now lives in the + // consolidated LocalDb database (LocalDb:Path) as the replicated sf_messages table. + // SqliteDbPath is read once at boot by SiteLocalDbLegacyMigrator to drain a pre-Phase-2 + // file, and is unused after that - keep it until this node has started once. + "SqliteDbPath": "/app/data/store-and-forward.db" }, "Communication": { "CentralContactPoints": [ diff --git a/src/ZB.MOM.WW.ScadaBridge.Host/StartupValidator.cs b/src/ZB.MOM.WW.ScadaBridge.Host/StartupValidator.cs index 71f40fa0..7dc63d94 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Host/StartupValidator.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Host/StartupValidator.cs @@ -94,7 +94,7 @@ public static class StartupValidator _ => seedNodes != null && seedNodes.Count >= 2, "must have at least 2 entries") // The big Site-only block: GrpcPort/MetricsPort validity + cross-field - // collisions + SiteDbPath + seed-node-port loop, in the original order. + // collisions + seed-node-port loop, in the original order. .When(role == "Site", p => { // GrpcPort range, then GrpcPort vs RemotingPort. @@ -110,9 +110,12 @@ public static class StartupValidator p.Require("ScadaBridge:Node:MetricsPort", _ => metricsPort != port, "must differ from RemotingPort"); p.Require("ScadaBridge:Node:MetricsPort", _ => metricsPort != grpcPort, "must differ from GrpcPort"); - p.Require("ScadaBridge:Database:SiteDbPath", - _ => !string.IsNullOrEmpty(configuration.GetSection("ScadaBridge:Database")["SiteDbPath"]), - "required for Site nodes"); + // ScadaBridge:Database:SiteDbPath was required here until LocalDb + // Phase 2. The site's tables now live in the consolidated LocalDb + // database (LocalDb:Path, which SiteServiceRegistration requires), + // and SiteDbPath survives only as the legacy migration source — so + // its absence means "nothing to migrate", not a misconfiguration. + // DatabaseOptionsValidator still rejects a present-but-blank value. // A seed node must reference an Akka.Remote endpoint, never the // Kestrel HTTP/2 gRPC port. A seed entry whose port equals this node's diff --git a/src/ZB.MOM.WW.ScadaBridge.Host/appsettings.Site.json b/src/ZB.MOM.WW.ScadaBridge.Host/appsettings.Site.json index ca1c8315..9e53a827 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Host/appsettings.Site.json +++ b/src/ZB.MOM.WW.ScadaBridge.Host/appsettings.Site.json @@ -23,6 +23,10 @@ "MinNrOfMembers": 1 }, "Database": { + // Migration-only as of LocalDb Phase 2. The site config tables now live in the + // consolidated LocalDb database (LocalDb:Path). SiteDbPath is read once at boot to drain + // a pre-Phase-2 scadabridge.db, and is unused after that - keep it until this node has + // started once. "SiteDbPath": "./data/scadabridge.db" }, "DataConnection": { @@ -31,8 +35,11 @@ "WriteTimeout": "00:00:30" }, "StoreAndForward": { - "SqliteDbPath": "./data/store-and-forward.db", - "ReplicationEnabled": true + // Migration-only as of LocalDb Phase 2. The store-and-forward buffer now lives in the + // consolidated LocalDb database (LocalDb:Path) as the replicated sf_messages table. + // SqliteDbPath is read once at boot by SiteLocalDbLegacyMigrator to drain a pre-Phase-2 + // file, and is unused after that - keep it until this node has started once. + "SqliteDbPath": "./data/store-and-forward.db" }, "Communication": { "_centralContactPoints": "Host-016: each entry MUST be a central node's remoting endpoint, NOT this site's own remoting port. The single dev-loopback default below points only at central-a (localhost:8081). In a multi-central deployment add the second central node here (e.g. 'akka.tcp://scadabridge@central-b-host:8081') so ClusterClient can fail over when central-a is down. The previous template listed localhost:8082 as the second contact — that is THIS site's own RemotingPort and is a permanent failure in the initial-contact rotation.", diff --git a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptions.cs b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptions.cs index 8606fea5..42503ca5 100644 --- a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptions.cs +++ b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptions.cs @@ -60,13 +60,6 @@ public class SiteRuntimeOptions /// HTTP timeout (seconds) for fetching a deployment config from central (notify-and-fetch). public int ConfigFetchTimeoutSeconds { get; set; } = 30; - /// - /// Bounded attempt count (including the first) for the standby's replicated-config - /// fetch; a 2 s fixed delay separates attempts and superseded fetches never retry. - /// Consumed by SiteReplicationActor.HandleApplyConfigDeploy (UA2). Default: 3. - /// - public int ConfigFetchRetryCount { get; set; } = 3; - /// /// Fixed interval (ms) at which an Instance Actor re-sends a tag-subscribe /// request that either failed or whose response was lost (S4/UA6). The retry is diff --git a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptionsValidator.cs b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptionsValidator.cs index e34ef736..f3ed4789 100644 --- a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptionsValidator.cs +++ b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptionsValidator.cs @@ -53,10 +53,6 @@ public sealed class SiteRuntimeOptionsValidator : OptionsValidatorBase= 0, - $"ScadaBridge:SiteRuntime:ConfigFetchRetryCount must be >= 0 " + - $"(was {options.ConfigFetchRetryCount})."); - builder.RequireThat(options.TagSubscribeRetryIntervalMs > 0, $"ScadaBridge:SiteRuntime:TagSubscribeRetryIntervalMs must be greater than 0 " + $"(was {options.TagSubscribeRetryIntervalMs}); a zero interval hot-loops the tag-subscribe " + diff --git a/src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardOptions.cs b/src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardOptions.cs index 39aaa321..5425c10b 100644 --- a/src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardOptions.cs +++ b/src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardOptions.cs @@ -5,12 +5,15 @@ namespace ZB.MOM.WW.ScadaBridge.StoreAndForward; /// public class StoreAndForwardOptions { - /// Path to the SQLite database for S&F message persistence. + /// + /// Path to the legacy standalone store-and-forward SQLite file. Migration-only. + /// The buffer itself lives in the consolidated LocalDb database (LocalDb:Path) + /// as of LocalDb Phase 2; this path is read once at boot by + /// SiteLocalDbLegacyMigrator to drain a pre-Phase-2 file, and is otherwise + /// unused. A node that has already migrated may leave it set or unset. + /// public string SqliteDbPath { get; set; } = "./data/store-and-forward.db"; - /// Whether to replicate buffer operations to standby node. - public bool ReplicationEnabled { get; set; } = true; - /// Default retry interval for messages without per-source settings. public TimeSpan DefaultRetryInterval { get; set; } = TimeSpan.FromSeconds(30); diff --git a/src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardOptionsValidator.cs b/src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardOptionsValidator.cs index b1a16d33..41bef58b 100644 --- a/src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardOptionsValidator.cs +++ b/src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardOptionsValidator.cs @@ -5,21 +5,23 @@ namespace ZB.MOM.WW.ScadaBridge.StoreAndForward; /// /// Validates at startup. The retry intervals /// feed the background sweep timer (a zero/negative period trips -/// in the timer constructor) and the -/// SQLite path is opened for the S&F buffer; an empty path yields an opaque -/// connection failure at first enqueue. Registered with ValidateOnStart() -/// so a bad ScadaBridge:StoreAndForward section fails fast at boot with a -/// clear, key-naming message. +/// in the timer constructor). Registered +/// with ValidateOnStart() so a bad ScadaBridge:StoreAndForward section +/// fails fast at boot with a clear, key-naming message. +/// +/// is deliberately NOT validated. +/// Before LocalDb Phase 2 it was the live buffer file, so an empty value produced an +/// opaque connection failure at first enqueue; the buffer now lives in the +/// consolidated LocalDb database and the key survives only as the legacy migration +/// source. An empty value there is a legitimate "nothing to migrate", so requiring +/// it would make every already-migrated node carry a dead key forever. +/// /// public sealed class StoreAndForwardOptionsValidator : OptionsValidatorBase { /// protected override void Validate(ValidationBuilder builder, StoreAndForwardOptions options) { - builder.RequireThat(!string.IsNullOrWhiteSpace(options.SqliteDbPath), - "ScadaBridge:StoreAndForward:SqliteDbPath must be a non-empty path; " + - "it is the SQLite file backing the store-and-forward buffer."); - builder.RequireThat(options.DefaultRetryInterval > TimeSpan.Zero, $"ScadaBridge:StoreAndForward:DefaultRetryInterval must be a positive duration " + $"(was {options.DefaultRetryInterval}); it is the default per-message retry interval."); diff --git a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/StartupValidatorTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/StartupValidatorTests.cs index de2f8375..3394b5dc 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/StartupValidatorTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/StartupValidatorTests.cs @@ -232,15 +232,22 @@ public class StartupValidatorTests Assert.Null(ex); } + /// + /// The inverse of the rule this replaces. SiteDbPath was mandatory for Site + /// nodes until LocalDb Phase 2 moved the site tables into the consolidated + /// LocalDb database; it now names only the legacy file the boot-time migrator + /// drains, so its absence means "nothing to migrate". Requiring it would make + /// every already-migrated node carry a dead key forever. + /// [Fact] - public void Site_MissingSiteDbPath_FailsValidation() + public void Site_MissingSiteDbPath_IsAccepted_BecauseThePathIsMigrationOnly() { var values = ValidSiteConfig(); values.Remove("ScadaBridge:Database:SiteDbPath"); var config = BuildConfig(values); - var ex = Assert.Throws(() => StartupValidator.Validate(config)); - Assert.Contains("SiteDbPath required for Site nodes", ex.Message); + var ex = Record.Exception(() => StartupValidator.Validate(config)); + Assert.Null(ex); } [Fact] diff --git a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ParkedMessageHandlerActorTests.cs b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ParkedMessageHandlerActorTests.cs index 717636a2..3692eb01 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ParkedMessageHandlerActorTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ParkedMessageHandlerActorTests.cs @@ -29,7 +29,6 @@ public class ParkedMessageHandlerActorTests : TestKit, IAsyncLifetime, IDisposab DefaultRetryInterval = TimeSpan.Zero, DefaultMaxRetries = 1, RetryTimerInterval = TimeSpan.FromMinutes(10), - ReplicationEnabled = false, }; _service = new StoreAndForwardService( diff --git a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ParkedOperationRelayTests.cs b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ParkedOperationRelayTests.cs index 426bf681..3f32e3ec 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ParkedOperationRelayTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ParkedOperationRelayTests.cs @@ -34,7 +34,6 @@ public class ParkedOperationRelayTests : TestKit, IAsyncLifetime, IDisposable DefaultRetryInterval = TimeSpan.Zero, DefaultMaxRetries = 1, RetryTimerInterval = TimeSpan.FromMinutes(10), - ReplicationEnabled = false, }; _service = new StoreAndForwardService( diff --git a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardOptionsTests.cs b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardOptionsTests.cs index 1da1c996..e8b9d0f1 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardOptionsTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardOptionsTests.cs @@ -11,7 +11,6 @@ public class StoreAndForwardOptionsTests var options = new StoreAndForwardOptions(); Assert.Equal("./data/store-and-forward.db", options.SqliteDbPath); - Assert.True(options.ReplicationEnabled); Assert.Equal(TimeSpan.FromSeconds(30), options.DefaultRetryInterval); Assert.Equal(50, options.DefaultMaxRetries); Assert.Equal(TimeSpan.FromSeconds(10), options.RetryTimerInterval); @@ -23,14 +22,12 @@ public class StoreAndForwardOptionsTests var options = new StoreAndForwardOptions { SqliteDbPath = "/custom/path.db", - ReplicationEnabled = false, DefaultRetryInterval = TimeSpan.FromMinutes(5), DefaultMaxRetries = 100, RetryTimerInterval = TimeSpan.FromSeconds(30) }; Assert.Equal("/custom/path.db", options.SqliteDbPath); - Assert.False(options.ReplicationEnabled); Assert.Equal(TimeSpan.FromMinutes(5), options.DefaultRetryInterval); Assert.Equal(100, options.DefaultMaxRetries); } diff --git a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardOptionsValidatorTests.cs b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardOptionsValidatorTests.cs index 346516fa..2a7da0e5 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardOptionsValidatorTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/StoreAndForwardOptionsValidatorTests.cs @@ -39,13 +39,20 @@ public class StoreAndForwardOptionsValidatorTests Assert.Contains("RetryTimerInterval", result.FailureMessage); } + /// + /// The inverse of the rule this replaces. Before LocalDb Phase 2 an empty + /// SqliteDbPath was rejected, because it was the live buffer file and an empty + /// value failed opaquely at first enqueue. The buffer now lives in the + /// consolidated LocalDb database, so the key is only the legacy migration + /// source and an empty value legitimately means "nothing to migrate" — a node + /// that has already migrated must be able to drop it. + /// [Fact] - public void EmptySqliteDbPath_IsRejected() + public void EmptySqliteDbPath_IsAccepted_BecauseThePathIsMigrationOnly() { var result = Validate(new StoreAndForwardOptions { SqliteDbPath = "" }); - Assert.True(result.Failed); - Assert.Contains("SqliteDbPath", result.FailureMessage); + Assert.False(result.Failed); } // ── R2 T9: sweep-tuning eager validation (N4) ── From 15013156bf3d672006b607801726868454c3c9b6 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Mon, 20 Jul 2026 04:41:01 -0400 Subject: [PATCH 48/55] test(localdb): two-node convergence for config tables + sf_messages MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four scenarios over the real site pair, driven through the REAL SiteStorageService rather than hand-written SQL. The sibling suites had to hand-write SQL because they were specifications written BEFORE the cutover; this suite runs after it, so it can drive the shipped writers — which is what makes the cascade scenario meaningful. - DeployedConfigRow_ConvergesToB_ColumnForColumn: the existing suite asserts config_json only, so a capture that dropped or defaulted any other column would still pass. deployed_at matters most — SiteReconciliationActor's guarded write compares it. - RemovingAnInstance_ConvergesAllThreeCascadeTables: the plan's flagged highest-risk case. RemoveDeployedConfigAsync deletes from three tables in one transaction, the schema has no foreign keys, and CDC ships three independent per-table streams. A dropped delete leaves a permanently stale override or alarm row on the standby, invisible until the instance name is redeployed. Carries a never-removed control instance, without which "the cascade converged" is indistinguishable from "node B lost these tables entirely". - ANativeAlarmBurst_Converges_AndTheOplogDrains: convergence alone would pass if entries replicated but were never acked, and an oplog that only grows trips the caps into a snapshot resync. - RowsWrittenOnBWhileItsListenerIsDown_SurviveTheRejoin: the union-survives property is per-table, and an unregistered table is silently local-only rather than an error, so it is re-proved on the tables the N1 scenario does not touch. Non-vacuity verified as the plan requires: with the eight Phase 2 RegisterReplicated calls commented out in SiteLocalDbSetup, all four go red (4 failed / 0 passed); restored, 20/20 pass across the three LocalDb suites. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts --- .../LocalDbPhase2ConvergenceTests.cs | 193 ++++++++++++++++++ 1 file changed, 193 insertions(+) create mode 100644 tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbPhase2ConvergenceTests.cs diff --git a/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbPhase2ConvergenceTests.cs b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbPhase2ConvergenceTests.cs new file mode 100644 index 00000000..610a9e4c --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbPhase2ConvergenceTests.cs @@ -0,0 +1,193 @@ +using Microsoft.Extensions.Logging.Abstractions; +using ZB.MOM.WW.LocalDb; +using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; + +namespace ZB.MOM.WW.ScadaBridge.IntegrationTests; + +/// +/// Phase 2 convergence driven through the REAL , over a real +/// site pair, rather than through hand-written SQL. +/// +/// +/// +/// The sibling suites (, +/// ) were written as specifications +/// BEFORE the cutover, so they necessarily use hand-written SQL — the production writers were +/// still going through the bespoke replicator at the time. This suite runs after the cutover +/// and therefore drives the production writer, which is what makes the cascade scenario below +/// meaningful: it is the shipped multi-statement transaction under test, not a re-creation +/// of it. +/// +/// +/// is constructed directly on the fixture's +/// — the same instance the host registers — so its writes flow through +/// the same CDC triggers. +/// +/// +[Collection("LocalDbSitePairConvergence")] +public sealed class LocalDbPhase2ConvergenceTests : LocalDbSitePairHarness +{ + private SiteStorageService Storage(ILocalDb db) => + new(db, NullLogger.Instance); + + // ---- read helpers ----------------------------------------------------------------- + + private static Task CountAsync(ILocalDb db, string table, string instance) => + ScalarAsync(db, $"SELECT COUNT(*) FROM {table} WHERE instance_unique_name = @instance", + new { instance }); + + private static async Task ScalarAsync(ILocalDb db, string sql, object? args = null) + { + var rows = await db.QueryAsync(sql, static r => r.GetInt64(0), args); + return rows[0]; + } + + /// + /// The whole deployed_configurations row as one comparable string, so a scenario + /// can assert every column replicated rather than just the payload it happened to check. + /// + private static async Task ReadWholeConfigRowAsync(ILocalDb db, string instance) + { + var rows = await db.QueryAsync( + """ + SELECT instance_unique_name || '|' || config_json || '|' || deployment_id || '|' || + revision_hash || '|' || is_enabled || '|' || deployed_at + FROM deployed_configurations WHERE instance_unique_name = @instance + """, + static r => r.GetString(0), new { instance }); + return rows.Count == 0 ? null : rows[0]; + } + + /// Unacked oplog backlog. Drains to zero once the peer acknowledges. + private static Task OplogDepthAsync(ILocalDb db) => + ScalarAsync(db, + "SELECT COUNT(*) FROM __localdb_oplog WHERE seq > " + + "(SELECT last_acked_seq FROM __localdb_peer_state WHERE id = 1)"); + + // ---- scenarios -------------------------------------------------------------------- + + [Fact] + public async Task DeployedConfigRow_ConvergesToB_ColumnForColumn() + { + // LocalDbConfigConvergenceTests asserts config_json converges. That is the payload + // only: a capture that dropped, defaulted or reordered any other column would still + // pass it. deployed_at is the one that matters most — SiteReconciliationActor's + // guarded write compares it, so a node whose replicated copy carried a different + // timestamp would make different staleness decisions from its peer. + await Storage(A).StoreDeployedConfigAsync( + "inst-columns", """{"setpoint":7}""", "dep-col-1", "hash-col-1", isEnabled: true); + + var onA = await ReadWholeConfigRowAsync(A, "inst-columns"); + Assert.NotNull(onA); + + await WaitUntilAsync( + async () => await ReadWholeConfigRowAsync(B, "inst-columns") == onA, + "node B's deployed_configurations row to match node A's in EVERY column"); + } + + [Fact] + public async Task RemovingAnInstance_ConvergesAllThreeCascadeTables() + { + // The plan flagged this as the most likely real defect in Phase 2, and the reasoning + // is sound: RemoveDeployedConfigAsync deletes from three tables in one transaction, + // the schema has NO foreign keys, and CDC captures the three deletes as independent + // per-table streams that LWW may apply to the peer in any order. Nothing in the + // system re-derives an orphan, so a dropped delete leaves a permanently stale + // static_attribute_overrides or native_alarm_state row on the standby — invisible + // until that instance name is redeployed and picks up ghost overrides. + var storageA = Storage(A); + + // A second instance that is never removed. Without it this test cannot distinguish + // "the cascade converged" from "node B lost these tables entirely" — a registration + // bug that wiped B would satisfy the absence assertions on its own. + foreach (var instance in new[] { "inst-cascade", "inst-survivor" }) + { + await storageA.StoreDeployedConfigAsync( + instance, """{"v":1}""", $"dep-{instance}", $"hash-{instance}", isEnabled: true); + await storageA.SetStaticOverrideAsync(instance, "Setpoint", "42"); + await storageA.SetStaticOverrideAsync(instance, "Mode", "Auto"); + await storageA.UpsertNativeAlarmAsync( + instance, "Area1.Line1.Alarm", "ref-1", """{"active":true}""", DateTimeOffset.UtcNow); + } + + await WaitUntilAsync( + async () => + await CountAsync(B, "deployed_configurations", "inst-cascade") == 1 && + await CountAsync(B, "static_attribute_overrides", "inst-cascade") == 2 && + await CountAsync(B, "native_alarm_state", "inst-cascade") == 1, + "all three tables to reach node B before the instance is removed"); + + await storageA.RemoveDeployedConfigAsync("inst-cascade"); + + await WaitUntilAsync( + async () => + await CountAsync(B, "deployed_configurations", "inst-cascade") == 0 && + await CountAsync(B, "static_attribute_overrides", "inst-cascade") == 0 && + await CountAsync(B, "native_alarm_state", "inst-cascade") == 0, + "the 3-table cascade delete to converge on node B with NO orphans left behind"); + + // The control instance is untouched on both nodes: the cascade is scoped, not a wipe. + Assert.Equal(1, await CountAsync(B, "deployed_configurations", "inst-survivor")); + Assert.Equal(2, await CountAsync(B, "static_attribute_overrides", "inst-survivor")); + Assert.Equal(1, await CountAsync(B, "native_alarm_state", "inst-survivor")); + } + + [Fact] + public async Task ANativeAlarmBurst_Converges_AndTheOplogDrains() + { + // native_alarm_state is the highest-rate replicated table (Task 1 found it bounded by + // per-SourceReference coalescing on a 100 ms flush, correcting the plan's D4). This + // asserts the burst converges AND that the oplog goes back to empty afterwards — + // convergence alone would still pass if entries replicated but were never acked, and + // an oplog that only grows eventually trips the caps and forces a snapshot resync. + const int burst = 200; + var storageA = Storage(A); + var at = DateTimeOffset.UtcNow; + + for (var i = 0; i < burst; i++) + { + await storageA.UpsertNativeAlarmAsync( + "inst-burst", "Area1.Line1.Alarm", $"ref-{i}", $$"""{"seq":{{i}}}""", at); + } + + await WaitUntilAsync( + async () => await CountAsync(B, "native_alarm_state", "inst-burst") == burst, + $"all {burst} alarm rows to reach node B"); + + await WaitUntilAsync( + async () => await OplogDepthAsync(A) == 0, + "node A's oplog to drain once node B acknowledges the burst"); + } + + [Fact] + public async Task RowsWrittenOnBWhileItsListenerIsDown_SurviveTheRejoin() + { + // Companion to LocalDbConfigConvergenceTests' N1 scenario, which covers the same + // rejoin for sf_messages and deployed_configurations. This one exercises the tables + // that scenario does not touch, because "the union survives" is a per-table property: + // it holds only if each table is actually registered, and an unregistered table is + // silently local-only rather than an error. + await Storage(A).StoreSharedScriptAsync("script-before", "return 1;", null, null); + await WaitUntilAsync( + async () => await ScalarAsync(B, + "SELECT COUNT(*) FROM shared_scripts WHERE name = 'script-before'") == 1, + "the pre-outage script to reach node B"); + + await StopPassiveAsync(); + + // Each node writes to a different Phase 2 table while they cannot see each other. + await Storage(A).StoreExternalSystemAsync( + "sys-from-a", "http://a.invalid", "None", null, null); + await Storage(B).SetStaticOverrideAsync("inst-offline", "WrittenWhileDown", "yes"); + + await RestartPairAsync(); + + await WaitUntilAsync( + async () => + await ScalarAsync(A, + "SELECT COUNT(*) FROM static_attribute_overrides WHERE attribute_name = 'WrittenWhileDown'") == 1 && + await ScalarAsync(B, + "SELECT COUNT(*) FROM external_systems WHERE name = 'sys-from-a'") == 1, + "both nodes to hold the union of what each wrote while partitioned"); + } +} From 921edab45434b0b61b7b9ddb0885e1695f5c938b Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Mon, 20 Jul 2026 04:41:47 -0400 Subject: [PATCH 49/55] chore(docker): size the site-a oplog caps from the phase 2 soak MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit MaxBatchSize 500 -> 16. The default is a ROW count, not a byte budget, so the batch size in bytes is set by the widest replicated column — config_json, which Task 1 measured at up to ~60-70 KB in production. 70 KB x 500 is ~35 MB against gRPC's 4 MB default receive limit; 16 keeps a worst-case batch near 1.1 MB. MaxOplogRows 1,000,000 -> 250,000 and MaxOplogAge 7d -> 2d, sized from the soak's 0.80 sf_messages rows/sec (~69k rows/day). Tighter than default is correct here because exceeding a cap is not data loss: the oplog prunes to the ceiling and sets needs_snapshot, so the peer catches up by snapshot resync instead of incrementally. That trades a rare full resync for a bounded file. site-b and site-c stay unreplicated, so the default-OFF posture is still proven side by side on one rig. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts --- docker/site-a-node-a/appsettings.Site.json | 19 ++++++++++++++++++- docker/site-a-node-b/appsettings.Site.json | 19 ++++++++++++++++++- 2 files changed, 36 insertions(+), 2 deletions(-) diff --git a/docker/site-a-node-a/appsettings.Site.json b/docker/site-a-node-a/appsettings.Site.json index f33407c3..3e54acb0 100644 --- a/docker/site-a-node-a/appsettings.Site.json +++ b/docker/site-a-node-a/appsettings.Site.json @@ -87,7 +87,24 @@ // pre-host secret expander. "Replication": { "PeerAddress": "http://scadabridge-site-a-b:8083", - "ApiKey": "dev-site-a-localdb-sync-key" + "ApiKey": "dev-site-a-localdb-sync-key", + // ---- Phase 2 sizing, from the Task 1 rig soak (not from the defaults) ---- + // + // MaxBatchSize (default 500) is a ROW count, not a byte budget, so the batch + // size in bytes is set by the widest replicated column. That is + // deployed_configurations.config_json: ~721 B on this rig, but up to ~60-70 KB + // in production (measured, Task 1) - and 70 KB x 500 is ~35 MB against gRPC's + // 4 MB default receive limit. 16 keeps a worst-case batch near 1.1 MB. + "MaxBatchSize": 16, + // Backlog caps bound the oplog while the peer is offline. Exceeding them is + // NOT data loss: the oplog is pruned to the ceiling and needs_snapshot is set, + // so the peer catches up by snapshot resync instead of incrementally. That + // makes tighter-than-default correct here - it trades a rare full resync for a + // bounded file. Sized from the soak's 0.80 sf_messages rows/sec (the only + // non-zero writer measured): ~69k rows/day, so 2 days is ~138k. 250,000 leaves + // room for burst without approaching the 1,000,000 default. + "MaxOplogRows": 250000, + "MaxOplogAge": "2.00:00:00" } } } diff --git a/docker/site-a-node-b/appsettings.Site.json b/docker/site-a-node-b/appsettings.Site.json index 78062879..376d2b44 100644 --- a/docker/site-a-node-b/appsettings.Site.json +++ b/docker/site-a-node-b/appsettings.Site.json @@ -80,7 +80,24 @@ // fail-closed, so a typo here does not degrade to unauthenticated replication; // it rejects every stream and the pair silently stops converging. "Replication": { - "ApiKey": "dev-site-a-localdb-sync-key" + "ApiKey": "dev-site-a-localdb-sync-key", + // ---- Phase 2 sizing, from the Task 1 rig soak (not from the defaults) ---- + // + // MaxBatchSize (default 500) is a ROW count, not a byte budget, so the batch + // size in bytes is set by the widest replicated column. That is + // deployed_configurations.config_json: ~721 B on this rig, but up to ~60-70 KB + // in production (measured, Task 1) - and 70 KB x 500 is ~35 MB against gRPC's + // 4 MB default receive limit. 16 keeps a worst-case batch near 1.1 MB. + "MaxBatchSize": 16, + // Backlog caps bound the oplog while the peer is offline. Exceeding them is + // NOT data loss: the oplog is pruned to the ceiling and needs_snapshot is set, + // so the peer catches up by snapshot resync instead of incrementally. That + // makes tighter-than-default correct here - it trades a rare full resync for a + // bounded file. Sized from the soak's 0.80 sf_messages rows/sec (the only + // non-zero writer measured): ~69k rows/day, so 2 days is ~138k. 250,000 leaves + // room for burst without approaching the 1,000,000 default. + "MaxOplogRows": 250000, + "MaxOplogAge": "2.00:00:00" } } } From 9ec7966dacad16c77316704782fbffbc31b6a002 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Mon, 20 Jul 2026 04:42:16 -0400 Subject: [PATCH 50/55] docs(localdb): task state for tasks 17-19 Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts --- docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json b/docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json index e9718bf1..135e4297 100644 --- a/docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json +++ b/docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json @@ -85,9 +85,9 @@ {"id": 14, "subject": "Task 14: Register the 8 Phase 2 tables and delete ReplicationService", "status": "completed", "classification": "high-risk", "blockedBy": [13], "note": "LANDED AS ONE COMMIT with tasks 15+16 - PLAN DEFECT: they cannot compile separately (SiteReplicationActor takes a ReplicationService + calls ReplaceAllAsync; DeploymentManagerActor Tells ReplicationMessages types; AkkaHostedService constructs the actor). Combining also strengthens Task 14's own invariant: never both mechanisms, never neither. Registered 8 tables; notification_lists + smtp_configurations deliberately NOT registered. Deleted ReplaceAllAsync as UNSAFE (a mass DELETE on a replicated table would be captured and shipped), and its test replaced by a comment explaining that. The positional-arg hazard was REAL: 4 DeploymentManagerActor test call sites bound wrong args; converted to named where possible - Props.Create is an expression tree and rejects OUT-OF-POSITION named args, so the rest are padded positionally. Task 7's 'not yet' test INVERTED (exact in both directions) + new security-named test for the SMTP tables + composite-PK test. Harness's temporary registration deleted, so the convergence suites now prove the cutover. My expected table list was mis-sorted: ordinal puts '_' (0x5F) before 'b', so data_connection_definitions precedes database_connections.", "commit": "037798b3"}, {"id": 15, "subject": "Task 15: Delete SiteReplicationActor and its messages", "status": "completed", "classification": "high-risk", "blockedBy": [14], "note": "LANDED AS ONE COMMIT with tasks 15+16 - PLAN DEFECT: they cannot compile separately (SiteReplicationActor takes a ReplicationService + calls ReplaceAllAsync; DeploymentManagerActor Tells ReplicationMessages types; AkkaHostedService constructs the actor). Combining also strengthens Task 14's own invariant: never both mechanisms, never neither. Registered 8 tables; notification_lists + smtp_configurations deliberately NOT registered. Deleted ReplaceAllAsync as UNSAFE (a mass DELETE on a replicated table would be captured and shipped), and its test replaced by a comment explaining that. The positional-arg hazard was REAL: 4 DeploymentManagerActor test call sites bound wrong args; converted to named where possible - Props.Create is an expression tree and rejects OUT-OF-POSITION named args, so the rest are padded positionally. Task 7's 'not yet' test INVERTED (exact in both directions) + new security-named test for the SMTP tables + composite-PK test. Harness's temporary registration deleted, so the convergence suites now prove the cutover. My expected table list was mis-sorted: ordinal puts '_' (0x5F) before 'b', so data_connection_definitions precedes database_connections.", "commit": "037798b3"}, {"id": 16, "subject": "Task 16: Clean up DeploymentManagerActor and AkkaHostedService", "status": "completed", "classification": "standard", "blockedBy": [15], "note": "LANDED AS ONE COMMIT with tasks 15+16 - PLAN DEFECT: they cannot compile separately (SiteReplicationActor takes a ReplicationService + calls ReplaceAllAsync; DeploymentManagerActor Tells ReplicationMessages types; AkkaHostedService constructs the actor). Combining also strengthens Task 14's own invariant: never both mechanisms, never neither. Registered 8 tables; notification_lists + smtp_configurations deliberately NOT registered. Deleted ReplaceAllAsync as UNSAFE (a mass DELETE on a replicated table would be captured and shipped), and its test replaced by a comment explaining that. The positional-arg hazard was REAL: 4 DeploymentManagerActor test call sites bound wrong args; converted to named where possible - Props.Create is an expression tree and rejects OUT-OF-POSITION named args, so the rest are padded positionally. Task 7's 'not yet' test INVERTED (exact in both directions) + new security-named test for the SMTP tables + composite-PK test. Harness's temporary registration deleted, so the convergence suites now prove the cutover. My expected table list was mis-sorted: ordinal puts '_' (0x5F) before 'b', so data_connection_definitions precedes database_connections.", "commit": "037798b3"}, - {"id": 17, "subject": "Task 17: Config-key cleanup (incl. ConfigFetchRetryCount; 10 appsettings files)", "status": "pending", "classification": "standard", "blockedBy": [16]}, - {"id": 18, "subject": "Task 18: Two-node convergence suite for the Phase 2 tables", "status": "pending", "classification": "high-risk", "blockedBy": [17], "note": "Non-vacuity must be proven directly (Phase 1 never did the mismatched-key run the original plan claimed)."}, - {"id": 19, "subject": "Task 19: Rig configuration (MaxOplogRows/MaxOplogAge + MaxBatchSize per D6)", "status": "pending", "classification": "small", "blockedBy": [17]}, + {"id": 17, "subject": "Task 17: Config-key cleanup (incl. ConfigFetchRetryCount; 10 appsettings files)", "status": "completed", "classification": "standard", "blockedBy": [16], "note": "As planned. DEVIATION: deploy/wonder-app-vd03/appsettings.Site.json sits under a GITIGNORED deploy/ tree, so its edit is local-only and must be repeated on the box at deploy time. Comment style is // (JSONC) rather than \"_comment_\" keys: every one of these files already contains // comments and .NET's json config reader accepts them, and a _comment_ key inside a bound section is a phantom config entry. Both relaxations pinned by the INVERSE of the test they replace (Site_MissingSiteDbPath_IsAccepted..., EmptySqliteDbPath_IsAccepted...), each verified red with the old rule restored. DatabaseOptionsValidator needed no change - it was already null-tolerant/blank-rejecting, which is exactly migration-only semantics.", "commit": "605e5682"}, + {"id": 18, "subject": "Task 18: Two-node convergence suite for the Phase 2 tables", "status": "completed", "classification": "high-risk", "blockedBy": [17], "note": "DEVIATION: landed as a NEW file LocalDbPhase2ConvergenceTests.cs rather than extending Phase 1's LocalDbSitePairConvergenceTests.cs, and drives the REAL SiteStorageService instead of hand-written SQL - possible only post-cutover, and what makes the cascade scenario test the shipped transaction rather than a re-creation of it. Scenario 4 was retargeted onto shared_scripts/external_systems/static_attribute_overrides because the plan's version overlapped LocalDbConfigConvergenceTests' N1 scenario almost exactly; the union-survives property is per-table, so re-proving it on untouched tables is the non-redundant half. Cascade test carries a never-removed control instance (absence assertions otherwise cannot distinguish 'cascade converged' from 'node B lost these tables'). Non-vacuity PROVEN as mandated: with the 8 RegisterReplicated calls commented out, 4 failed / 0 passed; restored, 20/20 across the three LocalDb suites.", "commit": "15013156"}, + {"id": 19, "subject": "Task 19: Rig configuration (MaxOplogRows/MaxOplogAge + MaxBatchSize per D6)", "status": "completed", "classification": "small", "blockedBy": [17], "note": "MaxBatchSize 500->16 (D6: row-count batching x ~70 KB production config_json vs the 4 MB gRPC cap; 16 => ~1.1 MB worst case). MaxOplogRows 1M->250,000 and MaxOplogAge 7d->2d from the soak's 0.80 rows/sec (~69k/day). Tighter-than-default is SAFE because a cap breach prunes + sets needs_snapshot (graceful snapshot resync), not data loss - the Task 1 finding that the plan's stop condition was weaker than written. site-b/site-c left unreplicated so default-OFF stays proven side by side.", "commit": "921edab4"}, {"id": 20, "subject": "Task 20: Live gate on the docker rig (10 evidence items)", "status": "pending", "classification": "high-risk", "blockedBy": [18, 19], "note": "Includes the post-cutover oplog drain-under-churn check (evidence 10) that Task 1 cannot measure pre-cutover. Sample DBs via throwaway copies (snap() helper), never host-side sqlite3 on live files; metrics on :8084 in-container."}, {"id": 21, "subject": "Task 21: Documentation truth pass", "status": "pending", "classification": "standard", "blockedBy": [20]} ], From 166f07fa68ccd1b963185a41f4fa7d1d0f2d2929 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Mon, 20 Jul 2026 04:58:20 -0400 Subject: [PATCH 51/55] chore(localdb): adopt LocalDb 0.1.1 and drop the directory shim MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit LocalDb 0.1.1 creates the parent directory of LocalDb:Path itself, so the SiteLocalDbDirectory shim this repo carried through Phase 2 is deleted along with its call site. The gap was found here but was never ScadaBridge's alone — every LocalDb consumer had it — so the fix moved to the library. SiteLocalDbDirectoryTests is RETAINED and retargeted rather than deleted with the shim. It was already written against the site registration path, not the mechanism, so it needed only its Ensure() call removed: what a site node requires is that resolving ILocalDb not fail on a fresh machine, regardless of who provides that. Verified it still earns its place — pinned back to LocalDb 0.1.0 it fails inside SqliteLocalDb..ctor -> SqliteConnection.Open(), so it genuinely depends on the library behaviour and not on a coincidence. Also corrects the coverage-split note in StoreAndForwardStorageTests, which asserted that directory creation is "NOT LocalDb's" — true when written, wrong as of 0.1.1. Build 0 warnings; Host 330, StoreAndForward 130, LocalDb integration 20 pass. Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts --- Directory.Packages.props | 6 +- .../SiteLocalDbDirectory.cs | 59 ------------------- .../SiteServiceRegistration.cs | 18 +++--- .../SiteLocalDbDirectoryTests.cs | 18 +++--- .../StoreAndForwardStorageTests.cs | 12 ++-- 5 files changed, 26 insertions(+), 87 deletions(-) delete mode 100644 src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbDirectory.cs diff --git a/Directory.Packages.props b/Directory.Packages.props index 68cbef4f..98404995 100644 --- a/Directory.Packages.props +++ b/Directory.Packages.props @@ -108,9 +108,9 @@ - - - + + +