LocalDb adoption Phase 1 + 2: consolidate the site database, delete the bespoke replicators (#23)
10-check live gate PASS; 3,509 tests green; replication remains default-OFF.
This commit was merged in pull request #23.
This commit is contained in:
@@ -120,6 +120,19 @@ Other peers in the `scadaproj` family (see `scadaproj/CLAUDE.md` for details): `
|
||||
- DCL write failures returned synchronously to calling script.
|
||||
- Tag path resolution retried periodically for devices still booting.
|
||||
- Static attribute writes persisted to local SQLite (survive restart/failover, reset on redeployment).
|
||||
- **Consolidated site database (LocalDb Phase 1 + 2, complete 2026-07-20).** **Ten** tables now live in ONE `ZB.MOM.WW.LocalDb`-managed SQLite file — the Phase 1 pair (`OperationTracking`, `site_events`) plus Phase 2's `sf_messages` and the seven site config tables (`deployed_configurations`, `static_attribute_overrides`, `shared_scripts`, `external_systems`, `database_connections`, `data_connection_definitions`, `native_alarm_state`), configured by the **required** `LocalDb:Path` (`/app/data/site-localdb.db` on the rig; validated with `ValidateOnStart`, so a site config missing it fails to boot). Both are `RegisterReplicated` tables. Consequences worth knowing:
|
||||
- `site_events.id` changed from autoincrement INTEGER to an application-minted **GUID**. Last-writer-wins keys on the primary key, so two nodes independently minting `id=1,2,3…` would destroy each other's events rather than merge them. The event-log read path uses a composite `(timestamp, id)` keyset cursor with an **opaque string** continuation token; `EventLogEntry.Id` and both `ContinuationToken`s are `string`/`string?` on the site↔central Akka DTOs.
|
||||
- `ScadaBridge:OperationTracking:ConnectionString`, `ScadaBridge:SiteEventLog:DatabasePath` and — as of Phase 2 — `ScadaBridge:StoreAndForward:SqliteDbPath` + `ScadaBridge:Database:SiteDbPath` are all **migration-only** — nothing reads them but `SiteLocalDbLegacyMigrator`, which copies a pre-Phase-1 file in once (deterministic `mig-{NodeName}-{legacyId}` event ids, `INSERT OR IGNORE`, runs AFTER `RegisterReplicated` so migrated rows replicate) and renames it `.migrated`. Delete the keys once a node has migrated.
|
||||
- This incidentally fixes a data-loss bug: both legacy databases defaulted to CWD-relative paths **outside** the mounted volume and were discarded on every container recreate.
|
||||
- **Replication is default-OFF and opt-in** via `LocalDb:Replication:PeerAddress` + a matching `ApiKey` on both nodes. `LocalDbSyncAuthInterceptor` is **fail-closed**: no configured key means no sync stream is accepted at all, so a key typo does not degrade to unauthenticated replication — the pair simply stops converging. The sync endpoint shares the existing site gRPC h2c listener (8083); no new port. Rig posture: **site-a replicated, site-b/site-c deliberately not**, so both states are proven side-by-side. Status surfaces on the site health report as `LocalDbReplicationConnected` / `LocalDbOplogBacklog` (both nullable — null means "no data", NOT "disconnected with an empty backlog") and as `localdb_*` Prometheus series. Note `ZbTelemetryOptions.Meters` is an **allowlist** (`SiteServiceRegistration.ObservedMeters`); an unlisted meter exports nothing, silently.
|
||||
- Design: **scadaproj** `docs/plans/2026-07-19-scadabridge-localdb-design.md` (that doc lives in the umbrella repo, not here); Phase 1 plan + Phase 2 gate are here under `docs/plans/`. **Phase 2 is COMPLETE** (branch `feat/localdb-phase2`, live gate PASS 2026-07-20 — all 10 checks, evidence in `docs/plans/2026-07-19-localdb-phase2-live-gate.md`). It moved the config tables + `sf_messages` in and **deleted** `SiteReplicationActor`, its `ReplicationMessages`, StoreAndForward's `ReplicationService`, and `StoreAndForwardStorage.ReplaceAllAsync`. What those did, and why nothing replaced them:
|
||||
|
||||
- `SiteReplicationActor` pushed config deploys to the peer and ran a **notify-and-fetch** exchange (tell the standby a deploy happened; it HTTP-fetches the config itself, with retries and a superseded-404 path). Config rows now simply replicate — **the standby makes no fetch at all** during a deploy (verified live). `SiteReconciliationActor` still fetches at node STARTUP when central reports gaps; that path survives and is a different thing.
|
||||
- `ReplicationService` fanned each buffer mutation (add/remove/park/requeue) to the standby by hand. CDC triggers on `sf_messages` do it now.
|
||||
- `ReplaceAllAsync` was a destructive delete-all-then-insert-all resync. It was not merely unused after the cutover but **unsafe to keep**: a mass DELETE on a now-replicated table would be captured and shipped to the peer. LocalDb's snapshot resync merges per row under LWW and never deletes, which is also why the old N1 directional-authority guard is gone — there is no wipe left to gate.
|
||||
- **`notification_lists` and `smtp_configurations` are created but deliberately NOT registered.** They are permanently empty on a site (no writer since 2026-07-10, the migrator skips them, the active-node purge keeps them empty), and registering them would open a standing replication channel whose only historical payload was plaintext SMTP passwords. Pinned by a security-named test, and verified live: those two tables have **no CDC triggers** on either rig node.
|
||||
- **Operational constraints (read before upgrading a site pair):** stop and start both nodes TOGETHER — rolling one at a time is no longer supported, since the legacy `SfBufferSnapshot` compatibility handler went with the replicator. And a node offline longer than `LocalDb:Replication:TombstoneRetention` (default 7 days) can resurrect deleted rows on rejoin. See `docs/deployment/topology-guide.md`.
|
||||
- `LocalDb:Replication:MaxBatchSize` batches by ROW COUNT, not bytes, against a 4 MB gRPC cap — the rig pins it to **16** (~70 KB worst-case `config_json` x 16 ~= 1.1 MB). The 500 default would allow ~35 MB.
|
||||
- All timestamps are UTC throughout the system.
|
||||
- Inter-cluster communication uses two transports: ClusterClient for command/control (deployments, lifecycle, subscribe/unsubscribe handshake, snapshots) and gRPC server-streaming for real-time data (attribute values, alarm states). Both CentralCommunicationActor and SiteCommunicationActor registered with receptionist. Central creates one ClusterClient per site using NodeA/NodeB as contact points. Sites configure multiple central contact points for failover. Addresses cached in CentralCommunicationActor, refreshed periodically (60s) and on admin changes. Heartbeats serve health monitoring only.
|
||||
- gRPC streaming channel: SiteStreamGrpcServer on each site node (Kestrel HTTP/2, port 8083); central creates per-site SiteStreamGrpcClient via SiteStreamGrpcClientFactory. Site entity has GrpcNodeAAddress/GrpcNodeBAddress fields. Proto: sitestream.proto with SiteStreamService, SiteStreamEvent (oneof: AttributeValueUpdate, AlarmStateUpdate). DebugStreamEvent message removed (no longer flows through ClusterClient).
|
||||
|
||||
@@ -15,10 +15,28 @@
|
||||
<PackageVersion Include="bunit" Version="2.0.33-preview" />
|
||||
<PackageVersion Include="coverlet.collector" Version="6.0.4" />
|
||||
<PackageVersion Include="FluentAssertions" Version="8.3.0" />
|
||||
<PackageVersion Include="Google.Protobuf" Version="3.29.3" />
|
||||
<PackageVersion Include="Grpc.AspNetCore" Version="2.71.0" />
|
||||
<PackageVersion Include="Grpc.Net.Client" Version="2.71.0" />
|
||||
<PackageVersion Include="Grpc.Tools" Version="2.71.0" />
|
||||
<!--
|
||||
gRPC stack raised 2.71.0 -> 2.76.0 (Protobuf 3.29.3 -> 3.34.1) on 2026-07-19.
|
||||
|
||||
Forced by ZB.MOM.WW.LocalDb.Replication 0.1.0, whose nuspec floors Grpc.AspNetCore,
|
||||
Grpc.Net.Client, Grpc.Core.Api and Grpc.Tools at 2.76.0 and Google.Protobuf at 3.34.1.
|
||||
Below that floor restore fails NU1605 (package downgrade), so this is not optional for
|
||||
LocalDb adoption.
|
||||
|
||||
The SQLitePCLRaw note below deferred exactly this bump as belonging in "their own
|
||||
reviewed commit" rather than smuggled under a security fix - so it IS its own commit.
|
||||
Direct consumer surface is narrow: Grpc.AspNetCore only in Host, Grpc.Net.Client only
|
||||
in Communication. Grpc.Core.Api is pinned explicitly here to keep the whole family on
|
||||
one version rather than letting it float in transitively.
|
||||
|
||||
Not bumped here: Microsoft.Data.SqlClient and Newtonsoft.Json, also named in that note.
|
||||
Nothing forces them, and they are a data-access change with its own risk profile.
|
||||
-->
|
||||
<PackageVersion Include="Google.Protobuf" Version="3.34.1" />
|
||||
<PackageVersion Include="Grpc.AspNetCore" Version="2.76.0" />
|
||||
<PackageVersion Include="Grpc.Net.Client" Version="2.76.0" />
|
||||
<PackageVersion Include="Grpc.Core.Api" Version="2.76.0" />
|
||||
<PackageVersion Include="Grpc.Tools" Version="2.76.0" />
|
||||
<PackageVersion Include="MailKit" Version="4.16.0" />
|
||||
<PackageVersion Include="Microsoft.AspNetCore.Authentication.JwtBearer" Version="10.0.7" />
|
||||
<PackageVersion Include="Microsoft.AspNetCore.Authorization" Version="10.0.7" />
|
||||
@@ -90,6 +108,9 @@
|
||||
<PackageVersion Include="ZB.MOM.WW.Secrets.Abstractions" Version="0.2.3" />
|
||||
<PackageVersion Include="ZB.MOM.WW.Secrets.Ui" Version="0.2.3" />
|
||||
<PackageVersion Include="ZB.MOM.WW.Secrets.Replicator.SqlServer" Version="0.2.3" />
|
||||
<PackageVersion Include="ZB.MOM.WW.LocalDb" Version="0.1.1" />
|
||||
<PackageVersion Include="ZB.MOM.WW.LocalDb.Replication" Version="0.1.1" />
|
||||
<PackageVersion Include="ZB.MOM.WW.LocalDb.Contracts" Version="0.1.1" />
|
||||
</ItemGroup>
|
||||
|
||||
<!--
|
||||
@@ -124,4 +145,19 @@
|
||||
<PackageVersion Include="SQLitePCLRaw.lib.e_sqlite3" Version="2.1.12" />
|
||||
</ItemGroup>
|
||||
|
||||
<!--
|
||||
GHSA-pgww-w46g-26qg (NU1902, moderate) on AngleSharp, reached only transitively via bunit
|
||||
in ZB.MOM.WW.ScadaBridge.CentralUI.Tests. With TreatWarningsAsErrors it made the WHOLE
|
||||
SOLUTION BUILD RED, so no suite could run and every "0 warnings / green" gate was
|
||||
unverifiable.
|
||||
|
||||
Resolved with a scoped NuGetAuditSuppress in that ONE test project (see its csproj for the
|
||||
full rationale) — deliberately NOT a version pin here. A pin was tried first and reverted:
|
||||
AngleSharp >= 1.5.0 is the only patched line, and it changes IHtmlCollection<T>'s indexer,
|
||||
which makes every bunit build throw MissingMethodException at runtime (33 render tests
|
||||
failed). Bumping bunit does not help either — even 2.7.2 resolves AngleSharp 1.4.0, which
|
||||
is still vulnerable. There is no working patched combination upstream today.
|
||||
-->
|
||||
|
||||
|
||||
</Project>
|
||||
|
||||
@@ -36,6 +36,7 @@
|
||||
<Project Path="tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.csproj" />
|
||||
<Project Path="tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests.csproj" />
|
||||
<Project Path="tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/ZB.MOM.WW.ScadaBridge.Communication.Tests.csproj" />
|
||||
<Project Path="tests/ZB.MOM.WW.ScadaBridge.TestSupport/ZB.MOM.WW.ScadaBridge.TestSupport.csproj" />
|
||||
<Project Path="tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests.csproj" />
|
||||
<Project Path="tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests.csproj" />
|
||||
<Project Path="tests/ZB.MOM.WW.ScadaBridge.NotificationService.Tests/ZB.MOM.WW.ScadaBridge.NotificationService.Tests.csproj" />
|
||||
|
||||
@@ -20,6 +20,10 @@
|
||||
"MinNrOfMembers": 1
|
||||
},
|
||||
"Database": {
|
||||
// Migration-only as of LocalDb Phase 2. The site config tables now live in the
|
||||
// consolidated LocalDb database (LocalDb:Path). SiteDbPath is read once at boot to drain
|
||||
// a pre-Phase-2 scadabridge.db, and is unused after that - keep it until this node has
|
||||
// started once.
|
||||
"SiteDbPath": "/app/data/scadabridge.db"
|
||||
},
|
||||
"DataConnection": {
|
||||
@@ -29,8 +33,11 @@
|
||||
"SeedReadTimeout": "00:00:30"
|
||||
},
|
||||
"StoreAndForward": {
|
||||
"SqliteDbPath": "/app/data/store-and-forward.db",
|
||||
"ReplicationEnabled": true
|
||||
// Migration-only as of LocalDb Phase 2. The store-and-forward buffer now lives in the
|
||||
// consolidated LocalDb database (LocalDb:Path) as the replicated sf_messages table.
|
||||
// SqliteDbPath is read once at boot by SiteLocalDbLegacyMigrator to drain a pre-Phase-2
|
||||
// file, and is unused after that - keep it until this node has started once.
|
||||
"SqliteDbPath": "/app/data/store-and-forward.db"
|
||||
},
|
||||
"Communication": {
|
||||
"CentralContactPoints": [
|
||||
@@ -56,5 +63,13 @@
|
||||
"Logging": {
|
||||
"MinimumLevel": "Information"
|
||||
}
|
||||
},
|
||||
// Consolidated site database (LocalDb Phase 1): OperationTracking + site_events.
|
||||
// On the mounted /app/data volume so it survives container recreate - unlike the
|
||||
// legacy site-tracking.db / site_events.db, which defaulted to CWD-relative paths
|
||||
// outside the volume and were lost on every recreate.
|
||||
// Replication is opt-in and configured separately; absent = local-only.
|
||||
"LocalDb": {
|
||||
"Path": "/app/data/site-localdb.db"
|
||||
}
|
||||
}
|
||||
|
||||
@@ -20,6 +20,10 @@
|
||||
"MinNrOfMembers": 1
|
||||
},
|
||||
"Database": {
|
||||
// Migration-only as of LocalDb Phase 2. The site config tables now live in the
|
||||
// consolidated LocalDb database (LocalDb:Path). SiteDbPath is read once at boot to drain
|
||||
// a pre-Phase-2 scadabridge.db, and is unused after that - keep it until this node has
|
||||
// started once.
|
||||
"SiteDbPath": "/app/data/scadabridge.db"
|
||||
},
|
||||
"DataConnection": {
|
||||
@@ -29,8 +33,11 @@
|
||||
"SeedReadTimeout": "00:00:30"
|
||||
},
|
||||
"StoreAndForward": {
|
||||
"SqliteDbPath": "/app/data/store-and-forward.db",
|
||||
"ReplicationEnabled": true
|
||||
// Migration-only as of LocalDb Phase 2. The store-and-forward buffer now lives in the
|
||||
// consolidated LocalDb database (LocalDb:Path) as the replicated sf_messages table.
|
||||
// SqliteDbPath is read once at boot by SiteLocalDbLegacyMigrator to drain a pre-Phase-2
|
||||
// file, and is unused after that - keep it until this node has started once.
|
||||
"SqliteDbPath": "/app/data/store-and-forward.db"
|
||||
},
|
||||
"Communication": {
|
||||
"CentralContactPoints": [
|
||||
@@ -56,5 +63,13 @@
|
||||
"Logging": {
|
||||
"MinimumLevel": "Information"
|
||||
}
|
||||
},
|
||||
// Consolidated site database (LocalDb Phase 1): OperationTracking + site_events.
|
||||
// On the mounted /app/data volume so it survives container recreate - unlike the
|
||||
// legacy site-tracking.db / site_events.db, which defaulted to CWD-relative paths
|
||||
// outside the volume and were lost on every recreate.
|
||||
// Replication is opt-in and configured separately; absent = local-only.
|
||||
"LocalDb": {
|
||||
"Path": "/app/data/site-localdb.db"
|
||||
}
|
||||
}
|
||||
|
||||
@@ -96,7 +96,7 @@ for ident in site-a site-b site-c; do
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo "Seeding LDAP group mappings (Design + Deployment)..."
|
||||
echo "Seeding LDAP group mappings (Designer + Deployer)..."
|
||||
# SecurityConfiguration.HasData declares 4 mappings but the InitialSchema
|
||||
# migration only inserts the Admin row, so a fresh ScadaBridgeConfig starts
|
||||
# with multi-role getting Admin only -- no Design and no Deployment access.
|
||||
@@ -106,11 +106,16 @@ docker exec -i scadabridge-mssql /opt/mssql-tools18/bin/sqlcmd \
|
||||
-d ScadaBridgeConfig -Q "
|
||||
SET IDENTITY_INSERT LdapGroupMappings ON;
|
||||
IF NOT EXISTS (SELECT 1 FROM LdapGroupMappings WHERE Id = 2)
|
||||
INSERT INTO LdapGroupMappings (Id, LdapGroupName, Role) VALUES (2, 'SCADA-Designers', 'Design');
|
||||
INSERT INTO LdapGroupMappings (Id, LdapGroupName, Role) VALUES (2, 'SCADA-Designers', 'Designer');
|
||||
IF NOT EXISTS (SELECT 1 FROM LdapGroupMappings WHERE Id = 3)
|
||||
INSERT INTO LdapGroupMappings (Id, LdapGroupName, Role) VALUES (3, 'SCADA-Deploy-All', 'Deployment');
|
||||
INSERT INTO LdapGroupMappings (Id, LdapGroupName, Role) VALUES (3, 'SCADA-Deploy-All', 'Deployer');
|
||||
IF NOT EXISTS (SELECT 1 FROM LdapGroupMappings WHERE Id = 4)
|
||||
INSERT INTO LdapGroupMappings (Id, LdapGroupName, Role) VALUES (4, 'SCADA-Deploy-SiteA', 'Deployment');
|
||||
INSERT INTO LdapGroupMappings (Id, LdapGroupName, Role) VALUES (4, 'SCADA-Deploy-SiteA', 'Deployer');
|
||||
-- Role strings MUST match the canonical vocabulary in
|
||||
-- src/ZB.MOM.WW.ScadaBridge.Security/Roles.cs ('Designer' / 'Deployer').
|
||||
-- These rows previously carried the pre-rename 'Design' / 'Deployment', which
|
||||
-- authorized nothing: every Designer/Deployer-gated management command failed
|
||||
-- UNAUTHORIZED on a freshly reseeded rig.
|
||||
SET IDENTITY_INSERT LdapGroupMappings OFF;
|
||||
"
|
||||
|
||||
|
||||
@@ -21,6 +21,10 @@
|
||||
"MinNrOfMembers": 1
|
||||
},
|
||||
"Database": {
|
||||
// Migration-only as of LocalDb Phase 2. The site config tables now live in the
|
||||
// consolidated LocalDb database (LocalDb:Path). SiteDbPath is read once at boot to drain
|
||||
// a pre-Phase-2 scadabridge.db, and is unused after that - keep it until this node has
|
||||
// started once.
|
||||
"SiteDbPath": "/app/data/scadabridge.db"
|
||||
},
|
||||
"DataConnection": {
|
||||
@@ -30,8 +34,11 @@
|
||||
"SeedReadTimeout": "00:00:30"
|
||||
},
|
||||
"StoreAndForward": {
|
||||
"SqliteDbPath": "/app/data/store-and-forward.db",
|
||||
"ReplicationEnabled": true
|
||||
// Migration-only as of LocalDb Phase 2. The store-and-forward buffer now lives in the
|
||||
// consolidated LocalDb database (LocalDb:Path) as the replicated sf_messages table.
|
||||
// SqliteDbPath is read once at boot by SiteLocalDbLegacyMigrator to drain a pre-Phase-2
|
||||
// file, and is unused after that - keep it until this node has started once.
|
||||
"SqliteDbPath": "/app/data/store-and-forward.db"
|
||||
},
|
||||
"Communication": {
|
||||
"CentralContactPoints": [
|
||||
@@ -57,5 +64,47 @@
|
||||
"Logging": {
|
||||
"MinimumLevel": "Information"
|
||||
}
|
||||
},
|
||||
// Consolidated site database (LocalDb Phase 1): OperationTracking + site_events.
|
||||
// On the mounted /app/data volume so it survives container recreate - unlike the
|
||||
// legacy site-tracking.db / site_events.db, which defaulted to CWD-relative paths
|
||||
// outside the volume and were lost on every recreate.
|
||||
// Replication is opt-in and configured separately; absent = local-only.
|
||||
"LocalDb": {
|
||||
"Path": "/app/data/site-localdb.db",
|
||||
// Site-a is the rig's REPLICATED pair; site-b and site-c deliberately stay
|
||||
// unreplicated so the default-OFF posture is proven side-by-side on one rig.
|
||||
//
|
||||
// This node is the initiator: it dials the peer. Replication is still
|
||||
// bidirectional - the passive node's writes flow back over the same stream -
|
||||
// so only one side needs PeerAddress. Port 8083 is the existing site gRPC
|
||||
// listener (h2c); the sync endpoint shares it, no new port.
|
||||
//
|
||||
// The ApiKey must be IDENTICAL on both nodes: the host's
|
||||
// LocalDbSyncAuthInterceptor is fail-closed, so a mismatch (or a missing key)
|
||||
// rejects every sync stream. A plain dev key here matches the rig's existing
|
||||
// posture; PRODUCTION uses a "${secret:...}" reference resolved by the
|
||||
// pre-host secret expander.
|
||||
"Replication": {
|
||||
"PeerAddress": "http://scadabridge-site-a-b:8083",
|
||||
"ApiKey": "dev-site-a-localdb-sync-key",
|
||||
// ---- Phase 2 sizing, from the Task 1 rig soak (not from the defaults) ----
|
||||
//
|
||||
// MaxBatchSize (default 500) is a ROW count, not a byte budget, so the batch
|
||||
// size in bytes is set by the widest replicated column. That is
|
||||
// deployed_configurations.config_json: ~721 B on this rig, but up to ~60-70 KB
|
||||
// in production (measured, Task 1) - and 70 KB x 500 is ~35 MB against gRPC's
|
||||
// 4 MB default receive limit. 16 keeps a worst-case batch near 1.1 MB.
|
||||
"MaxBatchSize": 16,
|
||||
// Backlog caps bound the oplog while the peer is offline. Exceeding them is
|
||||
// NOT data loss: the oplog is pruned to the ceiling and needs_snapshot is set,
|
||||
// so the peer catches up by snapshot resync instead of incrementally. That
|
||||
// makes tighter-than-default correct here - it trades a rare full resync for a
|
||||
// bounded file. Sized from the soak's 0.80 sf_messages rows/sec (the only
|
||||
// non-zero writer measured): ~69k rows/day, so 2 days is ~138k. 250,000 leaves
|
||||
// room for burst without approaching the 1,000,000 default.
|
||||
"MaxOplogRows": 250000,
|
||||
"MaxOplogAge": "2.00:00:00"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -21,6 +21,10 @@
|
||||
"MinNrOfMembers": 1
|
||||
},
|
||||
"Database": {
|
||||
// Migration-only as of LocalDb Phase 2. The site config tables now live in the
|
||||
// consolidated LocalDb database (LocalDb:Path). SiteDbPath is read once at boot to drain
|
||||
// a pre-Phase-2 scadabridge.db, and is unused after that - keep it until this node has
|
||||
// started once.
|
||||
"SiteDbPath": "/app/data/scadabridge.db"
|
||||
},
|
||||
"DataConnection": {
|
||||
@@ -30,8 +34,11 @@
|
||||
"SeedReadTimeout": "00:00:30"
|
||||
},
|
||||
"StoreAndForward": {
|
||||
"SqliteDbPath": "/app/data/store-and-forward.db",
|
||||
"ReplicationEnabled": true
|
||||
// Migration-only as of LocalDb Phase 2. The store-and-forward buffer now lives in the
|
||||
// consolidated LocalDb database (LocalDb:Path) as the replicated sf_messages table.
|
||||
// SqliteDbPath is read once at boot by SiteLocalDbLegacyMigrator to drain a pre-Phase-2
|
||||
// file, and is unused after that - keep it until this node has started once.
|
||||
"SqliteDbPath": "/app/data/store-and-forward.db"
|
||||
},
|
||||
"Communication": {
|
||||
"CentralContactPoints": [
|
||||
@@ -57,5 +64,40 @@
|
||||
"Logging": {
|
||||
"MinimumLevel": "Information"
|
||||
}
|
||||
},
|
||||
// Consolidated site database (LocalDb Phase 1): OperationTracking + site_events.
|
||||
// On the mounted /app/data volume so it survives container recreate - unlike the
|
||||
// legacy site-tracking.db / site_events.db, which defaulted to CWD-relative paths
|
||||
// outside the volume and were lost on every recreate.
|
||||
// Replication is opt-in and configured separately; absent = local-only.
|
||||
"LocalDb": {
|
||||
"Path": "/app/data/site-localdb.db",
|
||||
// The PASSIVE half of site-a's replicated pair: no PeerAddress, so this node's
|
||||
// sync initiator starts and idles while node-a dials in. Its own writes still
|
||||
// reach node-a - the stream is bidirectional.
|
||||
//
|
||||
// The key MUST match node-a's exactly. LocalDbSyncAuthInterceptor is
|
||||
// fail-closed, so a typo here does not degrade to unauthenticated replication;
|
||||
// it rejects every stream and the pair silently stops converging.
|
||||
"Replication": {
|
||||
"ApiKey": "dev-site-a-localdb-sync-key",
|
||||
// ---- Phase 2 sizing, from the Task 1 rig soak (not from the defaults) ----
|
||||
//
|
||||
// MaxBatchSize (default 500) is a ROW count, not a byte budget, so the batch
|
||||
// size in bytes is set by the widest replicated column. That is
|
||||
// deployed_configurations.config_json: ~721 B on this rig, but up to ~60-70 KB
|
||||
// in production (measured, Task 1) - and 70 KB x 500 is ~35 MB against gRPC's
|
||||
// 4 MB default receive limit. 16 keeps a worst-case batch near 1.1 MB.
|
||||
"MaxBatchSize": 16,
|
||||
// Backlog caps bound the oplog while the peer is offline. Exceeding them is
|
||||
// NOT data loss: the oplog is pruned to the ceiling and needs_snapshot is set,
|
||||
// so the peer catches up by snapshot resync instead of incrementally. That
|
||||
// makes tighter-than-default correct here - it trades a rare full resync for a
|
||||
// bounded file. Sized from the soak's 0.80 sf_messages rows/sec (the only
|
||||
// non-zero writer measured): ~69k rows/day, so 2 days is ~138k. 250,000 leaves
|
||||
// room for burst without approaching the 1,000,000 default.
|
||||
"MaxOplogRows": 250000,
|
||||
"MaxOplogAge": "2.00:00:00"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -21,6 +21,10 @@
|
||||
"MinNrOfMembers": 1
|
||||
},
|
||||
"Database": {
|
||||
// Migration-only as of LocalDb Phase 2. The site config tables now live in the
|
||||
// consolidated LocalDb database (LocalDb:Path). SiteDbPath is read once at boot to drain
|
||||
// a pre-Phase-2 scadabridge.db, and is unused after that - keep it until this node has
|
||||
// started once.
|
||||
"SiteDbPath": "/app/data/scadabridge.db"
|
||||
},
|
||||
"DataConnection": {
|
||||
@@ -30,8 +34,11 @@
|
||||
"SeedReadTimeout": "00:00:30"
|
||||
},
|
||||
"StoreAndForward": {
|
||||
"SqliteDbPath": "/app/data/store-and-forward.db",
|
||||
"ReplicationEnabled": true
|
||||
// Migration-only as of LocalDb Phase 2. The store-and-forward buffer now lives in the
|
||||
// consolidated LocalDb database (LocalDb:Path) as the replicated sf_messages table.
|
||||
// SqliteDbPath is read once at boot by SiteLocalDbLegacyMigrator to drain a pre-Phase-2
|
||||
// file, and is unused after that - keep it until this node has started once.
|
||||
"SqliteDbPath": "/app/data/store-and-forward.db"
|
||||
},
|
||||
"Communication": {
|
||||
"CentralContactPoints": [
|
||||
@@ -57,5 +64,13 @@
|
||||
"Logging": {
|
||||
"MinimumLevel": "Information"
|
||||
}
|
||||
},
|
||||
// Consolidated site database (LocalDb Phase 1): OperationTracking + site_events.
|
||||
// On the mounted /app/data volume so it survives container recreate - unlike the
|
||||
// legacy site-tracking.db / site_events.db, which defaulted to CWD-relative paths
|
||||
// outside the volume and were lost on every recreate.
|
||||
// Replication is opt-in and configured separately; absent = local-only.
|
||||
"LocalDb": {
|
||||
"Path": "/app/data/site-localdb.db"
|
||||
}
|
||||
}
|
||||
|
||||
@@ -21,6 +21,10 @@
|
||||
"MinNrOfMembers": 1
|
||||
},
|
||||
"Database": {
|
||||
// Migration-only as of LocalDb Phase 2. The site config tables now live in the
|
||||
// consolidated LocalDb database (LocalDb:Path). SiteDbPath is read once at boot to drain
|
||||
// a pre-Phase-2 scadabridge.db, and is unused after that - keep it until this node has
|
||||
// started once.
|
||||
"SiteDbPath": "/app/data/scadabridge.db"
|
||||
},
|
||||
"DataConnection": {
|
||||
@@ -30,8 +34,11 @@
|
||||
"SeedReadTimeout": "00:00:30"
|
||||
},
|
||||
"StoreAndForward": {
|
||||
"SqliteDbPath": "/app/data/store-and-forward.db",
|
||||
"ReplicationEnabled": true
|
||||
// Migration-only as of LocalDb Phase 2. The store-and-forward buffer now lives in the
|
||||
// consolidated LocalDb database (LocalDb:Path) as the replicated sf_messages table.
|
||||
// SqliteDbPath is read once at boot by SiteLocalDbLegacyMigrator to drain a pre-Phase-2
|
||||
// file, and is unused after that - keep it until this node has started once.
|
||||
"SqliteDbPath": "/app/data/store-and-forward.db"
|
||||
},
|
||||
"Communication": {
|
||||
"CentralContactPoints": [
|
||||
@@ -57,5 +64,13 @@
|
||||
"Logging": {
|
||||
"MinimumLevel": "Information"
|
||||
}
|
||||
},
|
||||
// Consolidated site database (LocalDb Phase 1): OperationTracking + site_events.
|
||||
// On the mounted /app/data volume so it survives container recreate - unlike the
|
||||
// legacy site-tracking.db / site_events.db, which defaulted to CWD-relative paths
|
||||
// outside the volume and were lost on every recreate.
|
||||
// Replication is opt-in and configured separately; absent = local-only.
|
||||
"LocalDb": {
|
||||
"Path": "/app/data/site-localdb.db"
|
||||
}
|
||||
}
|
||||
|
||||
@@ -21,6 +21,10 @@
|
||||
"MinNrOfMembers": 1
|
||||
},
|
||||
"Database": {
|
||||
// Migration-only as of LocalDb Phase 2. The site config tables now live in the
|
||||
// consolidated LocalDb database (LocalDb:Path). SiteDbPath is read once at boot to drain
|
||||
// a pre-Phase-2 scadabridge.db, and is unused after that - keep it until this node has
|
||||
// started once.
|
||||
"SiteDbPath": "/app/data/scadabridge.db"
|
||||
},
|
||||
"DataConnection": {
|
||||
@@ -30,8 +34,11 @@
|
||||
"SeedReadTimeout": "00:00:30"
|
||||
},
|
||||
"StoreAndForward": {
|
||||
"SqliteDbPath": "/app/data/store-and-forward.db",
|
||||
"ReplicationEnabled": true
|
||||
// Migration-only as of LocalDb Phase 2. The store-and-forward buffer now lives in the
|
||||
// consolidated LocalDb database (LocalDb:Path) as the replicated sf_messages table.
|
||||
// SqliteDbPath is read once at boot by SiteLocalDbLegacyMigrator to drain a pre-Phase-2
|
||||
// file, and is unused after that - keep it until this node has started once.
|
||||
"SqliteDbPath": "/app/data/store-and-forward.db"
|
||||
},
|
||||
"Communication": {
|
||||
"CentralContactPoints": [
|
||||
@@ -57,5 +64,13 @@
|
||||
"Logging": {
|
||||
"MinimumLevel": "Information"
|
||||
}
|
||||
},
|
||||
// Consolidated site database (LocalDb Phase 1): OperationTracking + site_events.
|
||||
// On the mounted /app/data volume so it survives container recreate - unlike the
|
||||
// legacy site-tracking.db / site_events.db, which defaulted to CWD-relative paths
|
||||
// outside the volume and were lost on every recreate.
|
||||
// Replication is opt-in and configured separately; absent = local-only.
|
||||
"LocalDb": {
|
||||
"Path": "/app/data/site-localdb.db"
|
||||
}
|
||||
}
|
||||
|
||||
@@ -21,6 +21,10 @@
|
||||
"MinNrOfMembers": 1
|
||||
},
|
||||
"Database": {
|
||||
// Migration-only as of LocalDb Phase 2. The site config tables now live in the
|
||||
// consolidated LocalDb database (LocalDb:Path). SiteDbPath is read once at boot to drain
|
||||
// a pre-Phase-2 scadabridge.db, and is unused after that - keep it until this node has
|
||||
// started once.
|
||||
"SiteDbPath": "/app/data/scadabridge.db"
|
||||
},
|
||||
"DataConnection": {
|
||||
@@ -30,8 +34,11 @@
|
||||
"SeedReadTimeout": "00:00:30"
|
||||
},
|
||||
"StoreAndForward": {
|
||||
"SqliteDbPath": "/app/data/store-and-forward.db",
|
||||
"ReplicationEnabled": true
|
||||
// Migration-only as of LocalDb Phase 2. The store-and-forward buffer now lives in the
|
||||
// consolidated LocalDb database (LocalDb:Path) as the replicated sf_messages table.
|
||||
// SqliteDbPath is read once at boot by SiteLocalDbLegacyMigrator to drain a pre-Phase-2
|
||||
// file, and is unused after that - keep it until this node has started once.
|
||||
"SqliteDbPath": "/app/data/store-and-forward.db"
|
||||
},
|
||||
"Communication": {
|
||||
"CentralContactPoints": [
|
||||
@@ -57,5 +64,13 @@
|
||||
"Logging": {
|
||||
"MinimumLevel": "Information"
|
||||
}
|
||||
},
|
||||
// Consolidated site database (LocalDb Phase 1): OperationTracking + site_events.
|
||||
// On the mounted /app/data volume so it survives container recreate - unlike the
|
||||
// legacy site-tracking.db / site_events.db, which defaulted to CWD-relative paths
|
||||
// outside the volume and were lost on every recreate.
|
||||
// Replication is opt-in and configured separately; absent = local-only.
|
||||
"LocalDb": {
|
||||
"Path": "/app/data/site-localdb.db"
|
||||
}
|
||||
}
|
||||
|
||||
@@ -104,7 +104,6 @@ Before branching on role, `AkkaHostedService.StartAsync` creates one actor uncon
|
||||
`SiteServiceRegistration.Configure` registers the site-only components. `AkkaHostedService.RegisterSiteActorsAsync` creates:
|
||||
- `DeploymentManagerActor` — cluster singleton scoped to `"site-{SiteId}"`.
|
||||
- `SiteCommunicationActor` — registered with `ClusterClientReceptionist`; creates a `ClusterClient` to configured central contact points.
|
||||
- `SiteReplicationActor` — one per node (not a singleton); handles best-effort S&F replication to the standby.
|
||||
- `EventLogHandlerActor` — cluster singleton scoped to `"site-{SiteId}"`.
|
||||
- `ParkedMessageHandlerActor` — bridges Akka to `StoreAndForwardService`.
|
||||
- `SiteAuditTelemetryActor` — created on a dedicated `audit-telemetry-dispatcher` (2-thread `ForkJoinDispatcher`) so SQLite reads and gRPC pushes never contend with hot-path actors.
|
||||
|
||||
@@ -8,7 +8,7 @@ Site Runtime (#3) operates exclusively on site clusters. Its entry point is the
|
||||
|
||||
The component code lives in `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/`:
|
||||
|
||||
- `Actors/` — `DeploymentManagerActor`, `InstanceActor`, `ScriptActor`, `ScriptExecutionActor`, `AlarmActor`, `AlarmExecutionActor`, `NativeAlarmActor`, `SiteReplicationActor`.
|
||||
- `Actors/` — `DeploymentManagerActor`, `InstanceActor`, `ScriptActor`, `ScriptExecutionActor`, `AlarmActor`, `AlarmExecutionActor`, `NativeAlarmActor`.
|
||||
- `Scripts/` — `ScriptCompilationService`, `ScriptExecutionScheduler`, `SharedScriptLibrary`, `ScriptRuntimeContext`, `ScopeAccessors`, `TriggerExpressionGlobals`.
|
||||
- `Streaming/` — `SiteStreamManager` (the site-wide Akka broadcast stream).
|
||||
- `Persistence/` — `SiteStorageService` (raw SQLite via `Microsoft.Data.Sqlite`), `SiteStorageInitializer`.
|
||||
@@ -79,7 +79,7 @@ Central sends a `DeployInstanceCommand` carrying a JSON `FlattenedConfiguration`
|
||||
|
||||
1. Calls `EnsureDclConnections` to push any new or changed connection definitions to the DCL manager (hash-guarded: unchanged configs are skipped).
|
||||
2. Calls `CreateInstanceActor`, which does `Context.ActorOf(props, instanceName)`.
|
||||
3. Runs an off-thread `Task` that calls `SiteStorageService.StoreDeployedConfigAsync`, clears static overrides and native alarm state, and — if `_replicationActor` is non-null (it is optional and null in isolated deployments/tests) — tells `SiteReplicationActor` to push to the peer node.
|
||||
3. Runs an off-thread `Task` that calls `SiteStorageService.StoreDeployedConfigAsync` and clears static overrides and native alarm state. Nothing is pushed to the peer: as of LocalDb Phase 2 those three tables are replicated, so the writes themselves reach the standby.
|
||||
4. Pipes back a `DeployPersistenceResult`; only on success does it tell the deployer `DeploymentStatus.Success`. If persistence fails, the optimistically-created actor is stopped and the error is returned to central (`SiteRuntime-005`).
|
||||
|
||||
For redeployment (instance already running), the existing actor is stopped and watched:
|
||||
@@ -216,7 +216,7 @@ Both `AlarmActor` and `NativeAlarmActor` tell the `InstanceActor` an `AlarmState
|
||||
|
||||
### Standby replication
|
||||
|
||||
`SiteReplicationActor` runs on every site node (not a singleton). The active node's `DeploymentManagerActor` tells it `ReplicateConfigDeploy`, `ReplicateConfigRemove`, `ReplicateConfigSetEnabled`, `ReplicateArtifacts`, or `ReplicateStoreAndForward`. The replication actor tracks the peer node via Akka cluster membership events and forwards each command to `/user/site-replication` on the peer via `ActorSelection`. Replication is fire-and-forget (no ack wait per design), so a failed write to the standby is logged but does not fail the primary operation.
|
||||
Config replication has no actor. `SiteReplicationActor` — which received `ReplicateConfigDeploy` / `ReplicateConfigRemove` / `ReplicateConfigSetEnabled` / `ReplicateArtifacts` / `ReplicateStoreAndForward` from the active node's `DeploymentManagerActor`, tracked the peer through cluster membership events, and forwarded each command to `/user/site-replication` via `ActorSelection` — was deleted in LocalDb Phase 2, together with its notify-and-fetch exchange (the standby was told a deploy had happened and then HTTP-fetched the config itself). The site's config tables are now replicated by LocalDb CDC, so a deploy on either node reaches the other as an ordinary row change. `SiteReconciliationActor` still fetches over HTTP at node startup when central reports gaps; that is a different path and it survives.
|
||||
|
||||
## Usage
|
||||
|
||||
|
||||
@@ -4,13 +4,12 @@ The Store-and-Forward Engine buffers site-originated outbound messages when a ta
|
||||
|
||||
## Overview
|
||||
|
||||
The Store-and-Forward Engine (#6) is a site-only component. The central cluster has no equivalent buffer; it uses the Notification Outbox (#21) instead for its own queued delivery work. Every site node runs one `StoreAndForwardService` instance, backed by a `StoreAndForwardStorage` SQLite store and an optional `ReplicationService` that fans each buffer mutation to the standby.
|
||||
The Store-and-Forward Engine (#6) is a site-only component. The central cluster has no equivalent buffer; it uses the Notification Outbox (#21) instead for its own queued delivery work. Every site node runs one `StoreAndForwardService` instance, backed by a `StoreAndForwardStorage` store. As of LocalDb Phase 2 that store writes to the consolidated LocalDb database, and the buffer reaches the peer as the replicated `sf_messages` table — the `ReplicationService` that used to fan each mutation to the standby by hand was deleted.
|
||||
|
||||
The component code lives in `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/`:
|
||||
|
||||
- `StoreAndForwardService` — the core buffer: enqueue, retry sweep, park/retry/discard, and the `ICachedCallLifecycleObserver` audit hook.
|
||||
- `StoreAndForwardStorage` — the SQLite layer; all reads and writes against `sf_messages`.
|
||||
- `ReplicationService` — fire-and-forget buffer replication to the standby.
|
||||
- `ParkedMessageHandlerActor` — Akka actor bridge that exposes parked-message query/retry/discard to the `SiteCommunicationActor`.
|
||||
- `NotificationForwarder` — the delivery handler for the `Notification` category; forwards buffered notifications to central via the ClusterClient transport and interprets the ack.
|
||||
- `StoreAndForwardOptions` — options class bound from the `StoreAndForward` configuration section.
|
||||
@@ -129,7 +128,9 @@ else
|
||||
|
||||
### Async replication to standby
|
||||
|
||||
`ReplicationService` wraps each buffer mutation — add, remove, park, requeue — in a `Task.Run` fire-and-forget. The active node does not wait for standby acknowledgment. The standby applies each `ReplicationOperation` via `ApplyReplicatedOperationAsync`, which calls the same `StoreAndForwardStorage` methods. Replication failures are logged at Debug and discarded; the standby may be slightly behind the active at any moment, producing at-most a few duplicate deliveries or missed retries after a failover — an accepted trade-off for zero added latency on the enqueue path.
|
||||
Replication is no longer the buffer's own concern. `sf_messages` is registered with LocalDb (`SiteLocalDbSetup.OnReady`), so every insert and status change is captured by a CDC trigger and shipped to the peer on the shared sync stream. The four hand-written operations — add, remove, park, requeue — and the `Task.Run` fan-out that carried them are gone, along with `ApplyReplicatedOperationAsync` and `ReplaceAllAsync`.
|
||||
|
||||
The trade-off is unchanged in shape: replication is still asynchronous, so the peer may be slightly behind at any instant. What changed is the bound. Convergence is now per row under last-writer-wins with HLC-ordered tombstones, so a lagging peer converges rather than diverging, and duplicate delivery after a failover is limited to messages the old primary delivered whose status change had not yet replicated. See `Component-StoreAndForward.md` for the normative statement of that bound.
|
||||
|
||||
The four `ReplicationOperationType` values are `Add`, `Remove`, `Park`, and `Requeue` (requeue was added to cover the operator-initiated `Parked→Pending` transition so the standby preserves retry intent after failover).
|
||||
|
||||
|
||||
@@ -187,6 +187,14 @@ ALTER ROLE db_owner ADD MEMBER scadabridge_svc;
|
||||
|
||||
Ensure bidirectional TCP connectivity between all Akka.NET cluster peers. The remoting port (default 8081) must be open in both directions.
|
||||
|
||||
## Upgrading a Site Pair
|
||||
|
||||
**Stop both nodes of a site pair, upgrade both, then start both.** Rolling one node at a time is
|
||||
not supported as of LocalDb Phase 2 — the legacy snapshot-compatibility handler that made a
|
||||
mixed-version pair converge was deleted with the bespoke replicator, and a mixed pair now diverges
|
||||
silently. See `docs/deployment/topology-guide.md` for the reasoning and for the related
|
||||
`TombstoneRetention` bound on how long one node may stay offline.
|
||||
|
||||
## Post-Installation Verification
|
||||
|
||||
1. Start the service: `sc.exe start ScadaBridge-Central`
|
||||
|
||||
@@ -150,7 +150,27 @@ Each site has its own two-node cluster:
|
||||
- Same split-brain resolver as central (keep-oldest).
|
||||
- Singleton actors: Site Deployment Manager migrates on failover.
|
||||
- Staggered instance startup: 50ms delay between Instance Actor creation to prevent reconnection storms.
|
||||
- SQLite persistence: Both nodes access the same SQLite files (or each has its own copy with async replication).
|
||||
- SQLite persistence: each node owns its own consolidated LocalDb database, kept in step by
|
||||
asynchronous CDC replication over a gRPC sync stream (LocalDb Phase 1 + 2). The nodes do NOT
|
||||
share a SQLite file.
|
||||
|
||||
### Site Pair Upgrades — stop and start BOTH nodes together
|
||||
|
||||
**A rolling upgrade of a site pair, one node at a time, is no longer supported.** It worked while
|
||||
the bespoke replicator kept a legacy `SfBufferSnapshot` compatibility handler so a new standby
|
||||
could still apply an old active node's monolithic snapshot. LocalDb Phase 2 deleted that handler
|
||||
along with the replicator, so a mixed-version pair has no common replication path: the two nodes
|
||||
will run, but they will not converge, and the divergence is silent.
|
||||
|
||||
Stop both nodes of a site pair, upgrade both, then start both.
|
||||
|
||||
**Related bound — do not leave one node of a pair offline for long.** A node absent for longer than
|
||||
`LocalDb:Replication:TombstoneRetention` (default **7 days**) can **resurrect deleted rows** when
|
||||
it rejoins: deletes replicate as HLC-ordered tombstones, and once a tombstone is pruned there is
|
||||
nothing left to suppress the stale row the returning node still holds. Within the retention window
|
||||
a rejoin is safe and self-correcting (verified live: a node stopped and restarted mid-load rejoined
|
||||
with both nodes byte-identical and zero duplicates). Beyond it, rebuild the returning node's
|
||||
database from its peer rather than letting it rejoin.
|
||||
|
||||
### Central-Site Communication
|
||||
|
||||
|
||||
@@ -10,6 +10,27 @@ Fixed via the **notify-and-fetch** rework (the primary recommendation below), no
|
||||
- **Plan:** [`docs/plans/2026-06-26-deploy-config-notify-and-fetch.md`](../plans/2026-06-26-deploy-config-notify-and-fetch.md)
|
||||
- **Validated:** live docker-cluster smoke — a previously-hanging deploy now completes in ~0.11 s; reconciliation heals single-node and concurrent-both-missing gaps.
|
||||
|
||||
## Amendment (2026-07-20) — LocalDb Phase 2 removed the second hop entirely
|
||||
|
||||
The resolution above fixed the intra-site hop by replacing it with notify-and-fetch. LocalDb
|
||||
Phase 2 then deleted **notify-and-fetch itself**, along with `SiteReplicationActor`: the site's
|
||||
`deployed_configurations` table is now replicated by CDC, so the config reaches the standby as an
|
||||
ordinary row change over the gRPC sync stream. There is no intra-site Akka hop carrying config any
|
||||
more, so the 128 000-byte frame constraint does not apply to it in any form.
|
||||
|
||||
The central→site hop is unchanged — it still sends a small `RefreshDeploymentCommand` and the site
|
||||
still fetches over HTTP, so that half of the original fix stands.
|
||||
|
||||
**The successor ceiling is different in kind.** The gRPC sync stream has a 4 MB default receive
|
||||
limit, and LocalDb batches by ROW COUNT (`LocalDb:Replication:MaxBatchSize`, default 500), not by
|
||||
bytes. A ~70 KB `config_json` — the largest measured in production — times 500 rows is ~35 MB,
|
||||
which would exceed the limit. The rig therefore pins `MaxBatchSize` to **16** (~1.1 MB worst case).
|
||||
Any deployment replicating wide rows must size that key deliberately; see the Phase 2 plan (D6) and
|
||||
`docs/plans/2026-07-19-localdb-phase2-live-gate.md`.
|
||||
|
||||
Note the failure mode differs from the one documented below: an oversized gRPC message is
|
||||
**rejected**, not silently dropped.
|
||||
|
||||
The diagnosis below is retained as the historical record of how the bug was found and reasoned about.
|
||||
|
||||
## Summary
|
||||
|
||||
@@ -0,0 +1,92 @@
|
||||
# Cached-telemetry drain hot-loops forever on a row whose tracking snapshot is gone
|
||||
|
||||
**Date:** 2026-07-20 · **Status:** OPEN · **Severity:** Medium (log flood + wasted I/O; no data loss)
|
||||
· **Area:** AuditLog / Site Telemetry
|
||||
|
||||
## Summary
|
||||
|
||||
`SiteAuditTelemetryActor`'s cached-telemetry drain reads Pending audit rows, looks up each row's
|
||||
tracking snapshot by `CorrelationId`, and pushes the combined packet to central. When the lookup
|
||||
returns `null` the row is **skipped and deliberately left Pending**
|
||||
(`SiteAuditTelemetryActor.cs:307`), on the reasoning that "central reconciliation will pick it up".
|
||||
|
||||
Nothing ever removes such a row from the local drain queue. The next tick re-reads it, fails the
|
||||
same lookup, logs the same warning, and leaves it Pending again — **forever**. With a batch of
|
||||
unresolvable rows the actor spins at its non-idle rate and emits one warning per row per pass.
|
||||
|
||||
Measured on the docker rig: **~2 800 warnings/minute, sustained**, surviving both a process restart
|
||||
and a container restart, until the audit database itself was discarded.
|
||||
|
||||
```
|
||||
[09:57:41 WRN] [Site/scadabridge-site-a-a] Cached-telemetry drain: no tracking snapshot for
|
||||
a5392796-291f-4f5b-9fbf-5817c1ec76c7 (TrackedOperationId 59bd4bf8-…); skipping.
|
||||
```
|
||||
|
||||
## Why the rows became unresolvable
|
||||
|
||||
Two independent stores must agree:
|
||||
|
||||
- the **audit** rows live in `auditlog.db` (site-local, and on the docker rig **inside the
|
||||
container at `/app/auditlog.db`**, not on the bind-mounted data volume);
|
||||
- the **tracking** rows live in `OperationTracking`, which LocalDb Phase 1 moved into the
|
||||
consolidated `LocalDb:Path` database (bind-mounted).
|
||||
|
||||
Anything that resets one without the other strands every audit row that referenced it. The code
|
||||
comment already anticipates the cause — *"possible if the audit row is older than the tracking
|
||||
retention window, or the tracking store was reset"* — so this is a known-and-accepted input, not an
|
||||
exotic one.
|
||||
|
||||
**Two realistic production triggers, neither requiring operator error:**
|
||||
|
||||
1. **Tracking retention expiry.** If the tracking retention window elapses before the audit drain
|
||||
catches up — a long central outage, a large backlog — the snapshots are pruned out from under
|
||||
still-Pending audit rows and every one of them becomes a permanent hot-loop entry.
|
||||
2. **Restoring or resetting one store independently of the other**, e.g. rebuilding a node's
|
||||
LocalDb file from its peer while its container-local `auditlog.db` survives.
|
||||
|
||||
It was hit here by (2): the two site-a LocalDb databases were dropped during a rig cleanup while
|
||||
`auditlog.db` — being inside the container — survived.
|
||||
|
||||
## Why the current handling is not enough
|
||||
|
||||
Skipping the row is correct; **leaving it Pending with no other state change is not**. The row is
|
||||
now in a state it can never leave:
|
||||
|
||||
- no attempt counter, so an unresolvable row is indistinguishable from a transiently-failing one;
|
||||
- no backoff, so the actor runs at full non-idle rate against a queue that can never shrink;
|
||||
- no terminal state, so it is retried for the life of the database;
|
||||
- one Warning per row per pass, which buries every other log line on the node.
|
||||
|
||||
The "central reconciliation will pick it up" comment is about the **audit half** reaching central by
|
||||
another path. That may well be true — but it does not release the row from the local drain queue,
|
||||
which is what actually loops.
|
||||
|
||||
## Suggested fix
|
||||
|
||||
Give an unresolvable row somewhere to go. Roughly, in increasing order of effort:
|
||||
|
||||
1. **Bound the retries.** Add an attempt count; past a threshold mark the row terminal
|
||||
(`TrackingUnavailable`) and stop re-reading it. Emit a single summary Warning with the count
|
||||
rather than one per row per pass.
|
||||
2. **Rate-limit the warning** to one per drain episode regardless of row count — the same pattern
|
||||
`MaintenanceBackgroundService` already uses for the oplog caps-exceeded warning (`_snapshotFlagWarned`).
|
||||
3. **Push the audit half alone** when the tracking snapshot is missing, rather than skipping the row
|
||||
entirely, so the row can be marked emitted and leave the queue. Needs a decision on whether
|
||||
central accepts a packet with no tracking half.
|
||||
|
||||
(2) alone would remove the operational damage; (1) or (3) is needed to stop the wasted I/O.
|
||||
|
||||
## Reproduction
|
||||
|
||||
1. Run a site node until it has cached-call audit rows with tracking correlations.
|
||||
2. Stop the node; delete its consolidated LocalDb database (which holds `OperationTracking`);
|
||||
leave `auditlog.db` in place.
|
||||
3. Start the node. The drain warning repeats indefinitely; the rate does not decay.
|
||||
|
||||
## Notes
|
||||
|
||||
- **No data loss.** The audit rows are intact and still reach central by the reconciliation path;
|
||||
what is broken is the local drain's ability to ever finish.
|
||||
- Discovered while cleaning the rig after the LocalDb Phase 2 live gate
|
||||
(`docs/plans/2026-07-19-localdb-phase2-live-gate.md`), which is also where the related
|
||||
"deleting an instance orphans its buffered messages" observation is recorded.
|
||||
@@ -0,0 +1,475 @@
|
||||
# LocalDb throws `SQLite Error 10: 'disk I/O error'` on the active site node under sustained write load
|
||||
|
||||
**Date:** 2026-07-20 · **Status:** ROOT-CAUSED + FIX PASS COMPLETE 2026-07-20 (same day) — observer-induced, **not a LocalDb defect**; see §0 (cause) and §11a (fixes) · **Severity:** was High; resolved to an operational rule (now enforced in the tooling docs) + shipped hardening
|
||||
**Area:** `ZB.MOM.WW.LocalDb` (library, `~/Desktop/scadaproj/ZB.MOM.WW.LocalDb/`) as consumed by ScadaBridge site nodes
|
||||
**Found by:** the Phase 2 rig soak — [`docs/plans/2026-07-19-localdb-phase2-soak.md`](../plans/2026-07-19-localdb-phase2-soak.md)
|
||||
**Branch:** `feat/localdb-phase2` (symptom observed on Phase 1 code)
|
||||
|
||||
> **Update 2026-07-20:** the mechanism has been identified and reproduced on demand, both in a
|
||||
> minimal SQLite-only repro and on the live rig, and the follow-up fixes have shipped.
|
||||
> Sections §0, §11a and §11 below are authoritative; the original brief (§1–§10) is preserved
|
||||
> as written, with corrections annotated where its conclusions did not survive
|
||||
> (§4.2, §4.3, §7, §9) and fix notes where they did (§5.1, §8).
|
||||
|
||||
---
|
||||
|
||||
## 0. ROOT CAUSE (verified)
|
||||
|
||||
**A host-side (macOS) `sqlite3` read of a live, bind-mounted WAL database checkpoints and
|
||||
resets the WAL out from under the container process, permanently poisoning that process's
|
||||
connections.** LocalDb, its connection handling, its UDF, and its triggers are not involved —
|
||||
the mechanism reproduces with plain Python `sqlite3` and no LocalDb code at all.
|
||||
|
||||
Mechanism, step by step:
|
||||
|
||||
1. POSIX advisory locks do **not** propagate across the Docker Desktop virtiofs bind-mount
|
||||
boundary. A host `sqlite3` opening the database cannot see the container's locks (and vice
|
||||
versa), so it believes it is the **only** connection.
|
||||
2. On close, the "last" connection in WAL mode runs a full checkpoint and **resets the WAL to
|
||||
0 bytes**. Because the read happened while the container was idle (a standby node, or a gap
|
||||
between write bursts), nothing blocks the checkpoint. This is exactly the file signature
|
||||
found on both rig nodes: main DB mtime + 0-byte `-wal` stamped at the sampling minute.
|
||||
3. The container process still holds the old WAL-index state (its per-inode `-shm` mapping,
|
||||
kept alive forever by the held-open `_master` connection and the Microsoft.Data.Sqlite
|
||||
connection pool). That index says the WAL contains N frames; the file now has none. Every
|
||||
subsequent statement — reads and writes both consult the WAL index — fails with
|
||||
**`SQLITE_IOERR_SHORT_READ` (extended code 522)**, surfaced as primary code 10
|
||||
`'disk I/O error'` (some paths surface `SQLITE_NOTADB` (26) instead). The poisoning is
|
||||
**permanent until the process reopens the database** (restart).
|
||||
|
||||
### Why the original brief's conclusions were wrong
|
||||
|
||||
- **"It tracks the load, not the node" (§4.1)** — confounded. *Both* nodes were poisoned by
|
||||
the 04:56 UTC host-side sampling (both nodes' `site-localdb.db` main files carry the 04:56
|
||||
mtime; node-b's WAL was left at 0 bytes). A poisoned **standby** shows zero errors only
|
||||
because a standby issues ~zero LocalDb statements; the errors "followed the load" because
|
||||
the load is what generates statements against an already-poisoned handle. Node-b's very
|
||||
first write attempt after failover (04:59:37, `OperationTrackingStore.RecordAttemptAsync`)
|
||||
failed — it had been poisoned for 3 minutes with nothing to say about it.
|
||||
- **"It is LocalDb-specific" (§4.2)** — sampling-selection bias. The legacy WAL databases in
|
||||
the same directory were healthy only because no host process ever read *them*. The minimal
|
||||
repro poisons an arbitrary WAL database the same way.
|
||||
- **"The observer has been ruled out" (§4.3)** — the exclusion assumed an error-free standby
|
||||
was an unpoisoned standby. It wasn't; it was a poisoned node with no traffic.
|
||||
|
||||
### Verification (2026-07-20, all on the live rig + minimal repro)
|
||||
|
||||
1. **Load alone is harmless:** restarted the poisoned active node (site-a-b); freshly-reopened
|
||||
site-a-a took the full soak load for **10+ minutes with zero errors** (the original model
|
||||
predicted onset within ~2 min), WAL growing/checkpointing normally, DB 188 KiB → 476 KiB.
|
||||
2. **One host read is sufficient and immediate:** a single
|
||||
`sqlite3 docker/site-a-node-a/data/site-localdb.db "SELECT count(*) FROM site_events;"`
|
||||
against the healthy loaded node reset its 4.6 MiB WAL to 0 bytes in place and produced the
|
||||
first `disk I/O error` **one second later** (05:32:48 → 05:32:49), 203 errors in the next
|
||||
40 s — the same one-second onset correlation as the original 04:56:36 → 04:56:37 incident.
|
||||
3. **Minimal repro (no LocalDb, no .NET):** a `python:3.12-alpine` container writing a
|
||||
WAL-mode SQLite DB on a bind mount (held master connection + fresh connection per op,
|
||||
`synchronous=NORMAL`, `busy_timeout=5000`). A host `sqlite3 "SELECT count(*)"`:
|
||||
- against the **actively-writing** DB → immediate `SQLITE_IOERR_SHORT_READ` (522) +
|
||||
`SQLITE_NOTADB` burst, then recovery (checkpoint could not fully reset a hot WAL);
|
||||
- during an **idle window** (connections held open, WAL populated) → WAL reset
|
||||
1.2 MiB → 0 bytes, then **every fresh-connection write failed for the rest of the run
|
||||
(200/200)** — the persistent variant, matching the rig.
|
||||
|
||||
### Consequences
|
||||
|
||||
1. **The operational rule in §5.3 ("do not query the databases with host-side `sqlite3`") is
|
||||
the root cause, not a hygiene note.** One violation silently destroys the node's local
|
||||
persistence until restart. This applies to *every* WAL SQLite file on the bind mount
|
||||
(`scadabridge.db`, `store-and-forward.db` included), not just LocalDb.
|
||||
2. **Safe inspection recipes:** copy the file triplet (`.db`, `-wal`, `-shm`) and open the
|
||||
copy; or read from inside the container boundary (same kernel ⇒ locks visible), e.g.
|
||||
`docker run --rm -v <dir>:/d alpine/sqlite3 sqlite3 /d/site-localdb.db "..."` — never the
|
||||
macOS host against live files.
|
||||
3. **LocalDb Phase 2 is unblocked** on this issue: the library sustained the full soak write
|
||||
load indefinitely once nothing external touched its file.
|
||||
4. Hardening follow-ups — **status as of the 2026-07-20 fix pass (see §11a):**
|
||||
- **DONE — extended-code logging:** the LocalDb-adjacent catch sites (`SiteAuditTelemetryActor`,
|
||||
`CachedCallTelemetryForwarder`, `SiteEventLogger`) now log
|
||||
`SqliteException` primary/extended codes (`sqlite 10/522`-style) via
|
||||
`SqliteErrorCodes.Describe` / `DescribeSqliteError`.
|
||||
- **DONE — §8 async-context bug** (see §8).
|
||||
- **DONE — §9.5 load regression test** (see §9).
|
||||
- **NOT DONE (deliberately):** a detect-and-reopen self-heal in `SqliteLocalDb` for
|
||||
persistent `SQLITE_IOERR`/`SQLITE_NOTADB`. This is a real library design change
|
||||
(pool clear + master reopen + in-flight coordination) protecting against *external
|
||||
interference only* — the trigger is operator/tooling action, now prevented at the
|
||||
source, and on same-kernel production deployments external readers see the locks and
|
||||
are safe. File as its own issue if production ever runs where a foreign-kernel reader
|
||||
can touch the files.
|
||||
|
||||
---
|
||||
|
||||
> **Original brief follows, preserved as written on 2026-07-20 before root-causing.**
|
||||
|
||||
---
|
||||
|
||||
## 1. Summary
|
||||
|
||||
On a ScadaBridge site node, once the node is **active** and under sustained concurrent write
|
||||
load, effectively every write to the consolidated LocalDb database (`site-localdb.db`) fails
|
||||
with:
|
||||
|
||||
```
|
||||
Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 10: 'disk I/O error'.
|
||||
```
|
||||
|
||||
Observed rate: **~1 000–1 500 failures per minute**, sustained, not transient. The node stays up
|
||||
and reports healthy. Ordinary (non-LocalDb) SQLite databases in the same directory, in the same
|
||||
process, under the same load, are completely unaffected.
|
||||
|
||||
## 2. Why it matters
|
||||
|
||||
1. **Silent data loss today.** `SiteEventLogger` fails its inserts and logs
|
||||
`[ERR] Failed to record event: script from ScriptActor:…`. Site event logging is dropping
|
||||
events on the floor on the active node whenever the site is busy. `OperationTracking` writes
|
||||
fail too, which breaks cached-call status tracking (`Cached-telemetry drain: no tracking
|
||||
snapshot for …; skipping`).
|
||||
2. **It blocks LocalDb Phase 2.** Phase 2 registers eight further tables into this same
|
||||
database — including `native_alarm_state` (highest-volume table on the node) and
|
||||
`sf_messages` — **and deletes the bespoke mechanisms that currently carry that data**
|
||||
(`SiteReplicationActor`, `StoreAndForward.ReplicationService`) in the same commit. Cutting
|
||||
over onto this store while removing the fallback would convert a logging defect into config
|
||||
and buffer loss.
|
||||
3. Phase 1 was previously live-gated as PASS. That gate exercised correctness and convergence,
|
||||
**not sustained write load** — which is why this was not caught.
|
||||
|
||||
## 3. Exact symptom
|
||||
|
||||
Two representative stacks, both from `docker logs scadabridge-site-a-b` while that node was
|
||||
active and under load:
|
||||
|
||||
```
|
||||
Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 10: 'disk I/O error'.
|
||||
at Microsoft.Data.Sqlite.SqliteDataReader.NextResult()
|
||||
at Microsoft.Data.Sqlite.SqliteCommand.ExecuteReader(CommandBehavior behavior)
|
||||
at ZB.MOM.WW.ScadaBridge.SiteEventLogging.SiteEventLogger.<>c__DisplayClass15_0.<ProcessWriteQueueAsync>b__0(SqliteConnection connection)
|
||||
in /src/src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs:line 236
|
||||
at ZB.MOM.WW.ScadaBridge.SiteEventLogging.SiteEventLogger.ProcessWriteQueueAsync()
|
||||
in /src/src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs:line 221
|
||||
```
|
||||
|
||||
```
|
||||
Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 10: 'disk I/O error'.
|
||||
at Microsoft.Data.Sqlite.SqliteDataReader.NextResult()
|
||||
at Microsoft.Data.Sqlite.SqliteCommand.ExecuteNonQuery()
|
||||
at ZB.MOM.WW.ScadaBridge.SiteRuntime.Tracking.OperationTrackingStore.RecordEnqueueAsync(...)
|
||||
in /src/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Tracking/OperationTrackingStore.cs:line 137
|
||||
at ZB.MOM.WW.ScadaBridge.AuditLog.Site.Telemetry.CachedCallTelemetryForwarder.TryEmitTrackingAsync(...)
|
||||
in /src/src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/Telemetry/CachedCallTelemetryForwarder.cs:line 148
|
||||
```
|
||||
|
||||
Note both fail inside `SqliteDataReader.NextResult()` — i.e. at statement execution, not at
|
||||
`Open()`. Connections are being acquired successfully; the failure is on the write itself.
|
||||
|
||||
### Error-source distribution
|
||||
|
||||
Error-stack frames counted over one 3-minute window on the loaded node:
|
||||
|
||||
| Store | Backing file | Frames |
|
||||
|---|---|---|
|
||||
| `OperationTrackingStore` | `site-localdb.db` (**LocalDb**) | 13 044 |
|
||||
| `SiteAuditTelemetryActor` | `site-localdb.db` (**LocalDb**) | 4 350 |
|
||||
| `SiteEventLogger` | `site-localdb.db` (**LocalDb**) | 900 |
|
||||
| `CachedCallTelemetryForwarder` | `site-localdb.db` (**LocalDb**) | 162 |
|
||||
| `StoreAndForwardStorage` | `store-and-forward.db` (legacy) | **0** |
|
||||
| `SiteStorageService` | `scadabridge.db` (legacy) | **0** |
|
||||
|
||||
## 4. Evidence — what has been established
|
||||
|
||||
### 4.1 It tracks the load, not the node
|
||||
|
||||
The load was moved between the two site-a nodes by restarting the active one (the surviving node
|
||||
becomes oldest-up and takes over):
|
||||
|
||||
| Node | Role | Under load | `disk I/O error` / 4 min |
|
||||
|---|---|---|---|
|
||||
| site-a-a | active | yes | 2 175 |
|
||||
| site-a-a | standby (after restart) | no | **0** |
|
||||
| site-a-b | standby | no | **0** |
|
||||
| site-a-b | active (after failover) | yes | **4 391** |
|
||||
|
||||
### 4.2 It is LocalDb-specific, not the filesystem or the bind mount — **WRONG, see §0**
|
||||
|
||||
> **Correction 2026-07-20:** sampling-selection bias — only the LocalDb file was ever read
|
||||
> from the host. Any of these WAL databases is equally poisonable (minimal-repro-proven).
|
||||
|
||||
This is the strongest signal. `store-and-forward.db` and `scadabridge.db` live in the **same
|
||||
bind-mounted directory** (`/app/data`, host `docker/site-a-node-*/data/`), are opened by the
|
||||
**same process**, are also **WAL-mode**, and are being written **concurrently under the same
|
||||
load** — and they log zero errors. Only the LocalDb-managed file fails.
|
||||
|
||||
### 4.3 The observer has been ruled out — **WRONG, see §0: the observer was the cause**
|
||||
|
||||
> **Correction 2026-07-20:** the exclusion below assumed an error-free standby was an
|
||||
> unpoisoned standby. Node-b's files carry the 04:56 sampling-time mtimes (WAL left at
|
||||
> 0 bytes); it was poisoned then and merely silent until failover gave it write traffic.
|
||||
|
||||
Onset (04:56:37) was **one second after** a host-side `sqlite3` read of the bind-mounted
|
||||
database (04:56:36), making observer-induced `-shm` corruption the leading hypothesis. It is
|
||||
excluded:
|
||||
|
||||
- After node-a was restarted (fresh open, `-shm` recovered) and load failed over to node-b,
|
||||
**node-b** — whose files no host process had touched since a single baseline read, and which
|
||||
had been error-free for the entire preceding period — began erroring immediately at a *higher*
|
||||
rate.
|
||||
- **node-a**, whose files *had* been sampled, dropped to zero once it stopped carrying load.
|
||||
|
||||
The variable that tracks the errors is load. (Host-side `sqlite3` against a live WAL database
|
||||
over a bind mount is still unsafe and should be avoided — it is just not the cause here.)
|
||||
|
||||
### 4.4 Not disk pressure
|
||||
|
||||
Host had 215 GiB free throughout (`df -h`: 76 % used on the data volume). Files are small:
|
||||
`site-localdb.db` 188 KiB, WAL peaked around 4.1 MiB then checkpointed to 0.
|
||||
|
||||
## 5. Reproduction
|
||||
|
||||
Fully reproducible in ~10 minutes on the local docker rig.
|
||||
|
||||
### 5.1 Rig prerequisites
|
||||
|
||||
Two rig-tooling bugs will block a fresh reseed; both are documented in the soak findings:
|
||||
|
||||
- `docker/seed-sites.sh` role names — **already fixed** (commit `cf46e596`).
|
||||
- **`infra/mssql/setup.sql` never executes** — **FIXED 2026-07-20**: `infra/reseed.sh` now
|
||||
applies the three init scripts itself via `sqlcmd` once MSSQL accepts connections (the
|
||||
`/docker-entrypoint-initdb.d/` compose mounts are informational only — the official
|
||||
`mcr.microsoft.com/mssql/server` image does not implement that hook; noted in the compose
|
||||
file). The manual workaround below is retained for historical context / older checkouts.
|
||||
Original problem: after `infra/reseed.sh` dropped the volume, nothing created
|
||||
`ScadaBridgeConfig` or the `scadabridge_app` login and `reseed.sh` hung forever on its
|
||||
setup.sql poll. The by-hand equivalent:
|
||||
|
||||
```bash
|
||||
cd ~/Desktop/ScadaBridge/infra
|
||||
for f in mssql/setup.sql mssql/machinedata_seed.sql mssql/setup-env2.sql; do
|
||||
docker exec -i scadabridge-mssql /opt/mssql-tools18/bin/sqlcmd \
|
||||
-S localhost -U sa -P 'ScadaBridge_Dev1#' -C -b < "$f"
|
||||
done
|
||||
```
|
||||
Then restart the app containers so EF migrations run, and restart central again after
|
||||
`seed-sites.sh` writes `LdapGroupMappings` (they are cached at startup).
|
||||
|
||||
### 5.2 Build the load generator
|
||||
|
||||
**The seeded `Motor Controller` template (id 4) cannot be used** — it fails pre-deployment
|
||||
validation with 34 errors (30 `ConnectionBinding`, 4 `ScriptCompilation`). Build a minimal one.
|
||||
|
||||
**Critical:** `ExternalSystem.Call` does **not** buffer to store-and-forward in practice.
|
||||
`ExternalSystem.CachedCall` is the buffering surface. Using `Call` produces HTTP traffic and no
|
||||
S&F rows, and will not reproduce this.
|
||||
|
||||
```bash
|
||||
cd ~/Desktop/ScadaBridge
|
||||
SB=src/ZB.MOM.WW.ScadaBridge.CLI/bin/Debug/net10.0/scadabridge # dotnet build src/...CLI first
|
||||
AUTH="--url http://localhost:9000 --username multi-role --password password"
|
||||
|
||||
# 1. Point the seeded external system at a refusing address (discard port).
|
||||
$SB $AUTH external-system update --id 1 --name "Test REST API" \
|
||||
--endpoint-url "http://127.0.0.1:9" --auth-type ApiKey --auth-config "scadabridge-test-key-1"
|
||||
|
||||
# 2. Minimal template: no attributes, no compositions, no connection bindings.
|
||||
$SB $AUTH --format json template create --name "SoakGenerator" # -> note the id
|
||||
|
||||
$SB $AUTH --format json template script add --template-id <TID> --name "SoakCall" \
|
||||
--trigger-type Interval --trigger-config '{"intervalMs":5000}' \
|
||||
--code 'var parms = new Dictionary<string, object?> { ["a"] = 2, ["b"] = 3 }; await ExternalSystem.CachedCall("Test REST API", "Add", parms);'
|
||||
|
||||
# 3. Four instances on site-a (site id 1), then deploy each.
|
||||
for i in 1 2 3 4; do
|
||||
$SB $AUTH --format json instance create --name "soakgen-$i" --template-id <TID> --site-id 1
|
||||
done
|
||||
$SB $AUTH instance deploy --id <each instance id>
|
||||
```
|
||||
|
||||
Note the CLI's `template script update` requires `--name` and `--trigger-type` even when only
|
||||
changing `--code`. In zsh, do not put the auth flags in an unquoted variable — zsh does not
|
||||
word-split, so pass them literally or use `${=AUTH}`.
|
||||
|
||||
### 5.3 Observe
|
||||
|
||||
```bash
|
||||
# Identify the ACTIVE node — it is the one running the ScriptActors.
|
||||
docker logs --since 4m scadabridge-site-a-a 2>&1 | grep -c "Connection refused"
|
||||
docker logs --since 4m scadabridge-site-a-b 2>&1 | grep -c "Connection refused"
|
||||
|
||||
# Errors appear on that node within ~2 minutes of load starting.
|
||||
docker logs --since 4m scadabridge-site-a-<active> 2>&1 | grep -c "disk I/O error"
|
||||
```
|
||||
|
||||
Metrics (port 8084 is **not** published, and the `aspnet:10.0` image has **no `curl`**) — use a
|
||||
sidecar in the container's network namespace:
|
||||
|
||||
```bash
|
||||
docker run --rm --network container:scadabridge-site-a-a curlimages/curl:latest \
|
||||
-s localhost:8084/metrics | grep '^localdb_'
|
||||
```
|
||||
|
||||
Do **not** query the databases with host-side `sqlite3` while containers are writing them.
|
||||
|
||||
## 6. Code map
|
||||
|
||||
### Library — `~/Desktop/scadaproj/ZB.MOM.WW.LocalDb/src/ZB.MOM.WW.LocalDb/Internal/SqliteLocalDb.cs`
|
||||
|
||||
Facts relevant to the failure:
|
||||
|
||||
- **A `_master` connection is held open for the object's entire lifetime** (`:31`), explicitly to
|
||||
"anchor the WAL journal". It is guarded by a `Lock _masterLock` because `SqliteConnection` is
|
||||
not thread-safe.
|
||||
- **`CreateConnection()` (`:83`) opens a brand-new `SqliteConnection` per call** — one per
|
||||
operation, from many concurrent actors. Every call then runs
|
||||
`PRAGMA synchronous=…; PRAGMA busy_timeout=…; PRAGMA foreign_keys=ON;` and registers a UDF:
|
||||
```csharp
|
||||
conn.CreateFunction("zb_hlc_next", () => _clock.Next());
|
||||
```
|
||||
- The connection string is **only** `DataSource=<path>` (`:57`) — **connection pooling is left at
|
||||
the Microsoft.Data.Sqlite default (enabled)**, and no `Cache=` or `Mode=` is set.
|
||||
- Effective options on the rig are the defaults: `BusyTimeoutMs = 5000`, `Synchronous = NORMAL`.
|
||||
ScadaBridge's rig config (`docker/site-a-node-*/appsettings.Site.json`, `LocalDb` section) sets
|
||||
only `Path` and the replication block.
|
||||
- `zb_hlc_next()` is invoked **from inside the capture triggers**, i.e. on the SQLite thread
|
||||
during every INSERT/UPDATE/DELETE on a registered table, and it calls into the shared
|
||||
`HybridLogicalClock` from arbitrary threads.
|
||||
|
||||
### Failing call sites (ScadaBridge)
|
||||
|
||||
- `src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs:221,236` — a channel-drained
|
||||
single-writer loop (`ProcessWriteQueueAsync`) using a `WithConnection(...)` helper.
|
||||
- `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Tracking/OperationTrackingStore.cs:137` (`RecordEnqueueAsync`),
|
||||
`:260,266` (`GetStatusAsync`).
|
||||
- `src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/Telemetry/CachedCallTelemetryForwarder.cs:148`.
|
||||
- `src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/Telemetry/SiteAuditTelemetryActor.cs` — also see §8.
|
||||
|
||||
## 7. Hypotheses, ranked
|
||||
|
||||
> **Resolution 2026-07-20:** none of the four below is the cause. The mechanism is a variant
|
||||
> of #2's territory (bind-mount `-shm`/WAL fragility) but triggered *only* by a host-side
|
||||
> reader — LocalDb's concurrency, pooling, and UDF (hypotheses 1/3/4) are exonerated. The
|
||||
> extended code, since captured, is `SQLITE_IOERR_SHORT_READ` (522).
|
||||
|
||||
None verified. Ordered by how well they fit "LocalDb only, load-dependent, same directory as
|
||||
healthy WAL databases".
|
||||
|
||||
1. **Connection churn × pooling × per-connection UDF registration.** LocalDb opens a fresh
|
||||
`SqliteConnection` per operation with pooling enabled, and calls `CreateFunction` on every
|
||||
acquisition. Under high concurrency this drives far more open/close and `-shm` mapping churn
|
||||
than the legacy stores (which reuse a small number of connections), and is the clearest
|
||||
structural difference between the failing and healthy databases. Suspect the interaction of
|
||||
the pool with the long-lived `_master` connection and WAL index growth.
|
||||
2. **`-shm` / WAL-index growth over the bind mount, triggered only at LocalDb's concurrency.**
|
||||
Would explain why the same mount is fine for lower-concurrency databases. `mmap` of the shared
|
||||
WAL index across virtiofs is a known-fragile area. **Distinguishing test: run the same load
|
||||
with `LocalDb:Path` pointed at a container-local path (a `tmpfs` or a plain volume rather than
|
||||
the bind mount).** If the errors vanish, this is confirmed and the fix is environmental /
|
||||
deployment-shaped rather than a library bug. **Run this test first — it is cheap and it
|
||||
partitions the hypothesis space.**
|
||||
3. **`zb_hlc_next` UDF failing inside a trigger.** An exception thrown out of the managed UDF
|
||||
callback during trigger execution can surface as a generic SQLite error at the statement
|
||||
level. Check `HybridLogicalClock.Next()` for thread-safety and for anything that can throw
|
||||
under contention (e.g. a spin/overflow path when many callers request stamps in the same
|
||||
millisecond).
|
||||
4. **Busy-timeout exhaustion misreported.** `BusyTimeoutMs = 5000` with heavy multi-connection
|
||||
write contention on one file. This would normally surface as `SQLITE_BUSY` (5), not
|
||||
`SQLITE_IOERR` (10), so it is a weaker fit — but worth excluding.
|
||||
|
||||
### The single highest-value next step
|
||||
|
||||
**Capture the extended result code.** The logs only show the primary code (`10` = `SQLITE_IOERR`),
|
||||
which is generic. `SqliteException.SqliteExtendedErrorCode` names the failing syscall and would
|
||||
likely settle this outright:
|
||||
|
||||
| Extended code | Meaning | Points at |
|
||||
|---|---|---|
|
||||
| `SQLITE_IOERR_SHMMAP` (6154) / `SQLITE_IOERR_SHMSIZE` (4874) | WAL index mmap/resize failed | hypothesis 2 |
|
||||
| `SQLITE_IOERR_WRITE` (778) / `SQLITE_IOERR_FSYNC` (1034) | plain write/fsync failed | filesystem |
|
||||
| `SQLITE_IOERR_LOCK` (3850) | file locking failed | bind mount locking |
|
||||
|
||||
Add the extended code to the exception logging (or attach a debugger / run the repro against a
|
||||
local non-container build) before pursuing any fix.
|
||||
|
||||
## 8. Secondary defect in the same path — **FIXED 2026-07-20**
|
||||
|
||||
```
|
||||
[ERROR][akka://scadabridge/user/site-audit-telemetry] There is no active ActorContext,
|
||||
this is most likely due to use of async operations from within this actor.
|
||||
Cause: System.NotSupportedException
|
||||
```
|
||||
|
||||
`SiteAuditTelemetryActor` is touching `Context` (or `Self`/`Sender`) after an `await`. This is a
|
||||
real bug independent of the I/O errors, though it sits in the same write path and may be
|
||||
contributing. Note the family-wide rule already recorded for Akka work: never read `Self`/`Context`
|
||||
after an `await` inside an actor.
|
||||
|
||||
> **Fixed 2026-07-20.** Root cause: both drain handlers await with `ConfigureAwait(false)`, so
|
||||
> their `finally`-block re-arm (`ScheduleNext`/`ScheduleNextCached`) runs on a pool thread with
|
||||
> no active ActorContext. Investigation found the failure is **bimodal**, and the second mode is
|
||||
> worse than the logged one: depending on what the pool thread's thread-static cell slot holds,
|
||||
> `Context`/`Self` either **throw** `NotSupportedException` (the logged variant — actor crashes
|
||||
> and restarts once per drain) or **silently resolve a STALE cell of whatever actor last ran on
|
||||
> that thread**, re-arming the tick at the *wrong actor* so the drain loop just stops (observed
|
||||
> under TestKit: the tick landed on the TestActor). Fix: capture `Context.System.Scheduler` and
|
||||
> `Self` into fields at construction (both are thread-safe immutable handles) and use only those
|
||||
> from the re-arm path. Regression test
|
||||
> `SiteAuditTelemetryActorTests.Drains_Whose_Awaits_Complete_Off_The_Actor_Thread_Keep_Draining_Without_Crashing`
|
||||
> forces the mock awaits to complete off the actor thread — which every pre-existing test
|
||||
> avoided by returning already-completed tasks — and catches **both** variants (EventFilter for
|
||||
> the throw, sustained-drain counts for the silent stall). AuditLog suite 355/355 green.
|
||||
|
||||
## 9. What a fix must satisfy
|
||||
|
||||
> **Resolution 2026-07-20:** criteria 1–4 are already satisfied by the unmodified code once no
|
||||
> host process touches the live files — verified 10+ min of soak load with zero errors, events
|
||||
> durably written, WAL checkpointing normally. Criterion 5 (a sustained concurrent-write load
|
||||
> test) would **not** have caught this — the trigger is an external reader, not load — but is
|
||||
> now in place anyway: `ConcurrentWriteLoadTests` in `ZB.MOM.WW.LocalDb.Tests` (8 concurrent
|
||||
> writers × 250 inserts through fresh pooled connections against a registered/triggered table on
|
||||
> a real file, with concurrent readers; asserts zero failures + exact row/oplog counts; suite
|
||||
> 145/145). §8's `SiteAuditTelemetryActor` async-context bug is **fixed** — see §8.
|
||||
|
||||
1. The §5 repro runs for **30 minutes under sustained load with zero `disk I/O error`** on the
|
||||
active node.
|
||||
2. No `Failed to record event` errors — site events are durably written under load.
|
||||
3. `localdb_oplog_depth` rises under load and **drains** between bursts; zero dead letters.
|
||||
4. Replication still converges across the site-a pair (Phase 1's existing convergence suite and
|
||||
live gate still pass).
|
||||
5. A regression test that would have caught this — i.e. a **concurrent-write load test** against
|
||||
a real LocalDb file, not just the correctness/convergence tests Phase 1 shipped. Phase 1's
|
||||
gate passed precisely because no test applied sustained concurrent write pressure.
|
||||
|
||||
## 10. Rig state as left
|
||||
|
||||
- Rig fully reseeded; central config volume dropped and replayed; site SQLite state wiped
|
||||
(`reseed.sh` stage 2 does `rm -rf docker/site-*/data/*`).
|
||||
- `ExternalSystemDefinitions` id 1 is **still repointed to `http://127.0.0.1:9`** — restore to
|
||||
`http://scadabridge-restapi:5200` when done.
|
||||
- Template `SoakGenerator` (id 2021) and instances `soakgen-1..4` (ids 5–8) are **still deployed
|
||||
and still generating load** on site-a.
|
||||
- `LdapGroupMappings` corrected in the live DB to the canonical `Designer`/`Deployer` names.
|
||||
|
||||
## 11a. Fix pass (2026-07-20, same day — all verified)
|
||||
|
||||
Everything actionable that this incident identified is now fixed (uncommitted on each repo's
|
||||
current branch; ScadaBridge full solution builds clean, 0 warnings):
|
||||
|
||||
| # | Issue | Fix | Verification |
|
||||
|---|---|---|---|
|
||||
| 1 | §8 `SiteAuditTelemetryActor` async-context bug (bimodal: crash-per-drain OR silent tick misroute) | Capture `Context.System.Scheduler` + `Self` at construction; re-arm path never reads thread-static context | New red→green regression test forcing off-actor-thread continuations; AuditLog suite 355/355 |
|
||||
| 2 | Diagnostics gap: logs carried only the primary SQLite code | `SqliteErrorCodes.Describe` (AuditLog) + `DescribeSqliteError` (SiteEventLogging) — catch sites now log `sqlite <primary>/<extended>` | Builds clean; suites green (this gap cost the investigation a from-scratch repro to learn code 522) |
|
||||
| 3 | §5.1 `reseed.sh` hangs forever waiting on the initdb hook the mssql image doesn't have | `reseed.sh` applies `setup.sql`/`machinedata_seed.sql`/`setup-env2.sql` itself via `sqlcmd`; compose mounts annotated as informational | `bash -n` clean; scripts verified idempotent (`IF NOT EXISTS` guards) |
|
||||
| 4 | §9.5 missing concurrent-write load test | `ConcurrentWriteLoadTests` in `ZB.MOM.WW.LocalDb.Tests` (scadaproj) — 8 writers × 250 pooled-connection inserts on a registered table + concurrent readers, exact row/oplog count asserts | LocalDb suite 145/145 |
|
||||
| 5 | Root cause itself (operator/tooling host reads) | Poisonous instructions removed from the Phase 2 plan + `.tasks.json` (safe `snap()` copy-based sampling); soak-doc verdict corrected; family-wide memory rule recorded | On-demand on/off reproduction, §0 |
|
||||
|
||||
Deliberately **not** done: the `SqliteLocalDb` detect-and-reopen self-heal (see §0
|
||||
consequence 4 for the rationale and the condition under which to file it).
|
||||
|
||||
## 11. Rig state after root-causing (2026-07-20 ~05:40 UTC)
|
||||
|
||||
- Both site-a nodes restarted during verification, curing both poisonings. End state:
|
||||
**site-a-b active** carrying the soak load, site-a-a standby, **zero `disk I/O error` on
|
||||
both** under sustained load.
|
||||
- The §10 items still stand: `ExternalSystemDefinitions` id 1 still points at
|
||||
`http://127.0.0.1:9`, and `SoakGenerator` + `soakgen-1..4` are still deployed and
|
||||
generating load — the Phase 2 soak can now proceed on a clean baseline.
|
||||
- Minimal-repro scripts (`writer.py` burst variant, `writer2.py` idle-window variant) lived in
|
||||
the session scratchpad; the recipe is fully described in §0 and takes ~2 minutes to rebuild.
|
||||
@@ -18,8 +18,8 @@
|
||||
| 1 | Task 1 (packages) |
|
||||
| 2 | Task 2 (schema helpers) |
|
||||
| 3 | Task 3 (composition root) |
|
||||
| 4 | Task 4 (tracking store) ∥ Task 5 (event logger) ∥ Task 7 (replication wiring) |
|
||||
| 5 | Task 6 (migrator) ∥ Task 8 (auth interceptor) ∥ Task 9 (health signal) |
|
||||
| 4 | Task 4 (tracking store) ∥ Task 5a (event writer) ∥ Task 7 (replication wiring) |
|
||||
| 5 | Task 5b (event read path) ∥ Task 6 (migrator) ∥ Task 8 (auth interceptor) ∥ Task 9 (health signal) |
|
||||
| 6 | Task 10 (convergence test) ∥ Task 11 (rig config) |
|
||||
| 7 | Task 12 (live gate — serial, on the real rig) |
|
||||
| 8 | Task 13 (docs) ∥ Task 14 (Phase 2 planning) |
|
||||
@@ -181,8 +181,9 @@ services.AddZbLocalDb(config, SiteLocalDbSetup.OnReady);
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Tracking/OperationTrackingStore.cs`
|
||||
- Modify: wherever `AddSiteRuntime` registers the store (follow `services.AddSiteRuntime` from `SiteServiceRegistration.cs:51` into the SiteRuntime SCE)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/ServiceCollectionExtensions.cs:74` (the `IOperationTrackingStore` registration; `AddSiteRuntime`'s connection-string parameter becomes vestigial for this store — leave the parameter alone, it still feeds the site config DB)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Tracking/` (existing store tests)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CompositionRootTests.cs:433` (`Site_Resolves_IOperationTrackingStore` — it overrides the connection string to a temp path because the store opens eagerly in its ctor; that override changes shape once the store takes `ILocalDb`)
|
||||
|
||||
**Step 1:** Constructor gains `ILocalDb localDb`; `_writeConnection = localDb.CreateConnection()` replaces `new SqliteConnection(_connectionString)` + `Open()` (verified in the lib, `SqliteLocalDb.cs:83-98`: CreateConnection returns an **open** connection with per-connection pragmas — `synchronous`, `busy_timeout`, `foreign_keys=ON` — and the `zb_hlc_next` UDF registered; do NOT call `Open()` on it again). Reader paths that open ad-hoc connections from `_connectionString` switch to `localDb.CreateConnection()` too (reads don't fire triggers, but one connection source is one less config knob). `OperationTrackingOptions.ConnectionString` becomes unused by the store — keep the options class but drop the store's dependency; leave a validator relaxation only if the DI-graph test demands it.
|
||||
|
||||
@@ -194,25 +195,70 @@ services.AddZbLocalDb(config, SiteLocalDbSetup.OnReady);
|
||||
|
||||
---
|
||||
|
||||
### Task 5: Rewire SiteEventLogger + EventLogQueryService (GUID ids)
|
||||
### Task 5a: Rewire SiteEventLogger onto ILocalDb + mint GUID ids
|
||||
|
||||
**Classification:** standard
|
||||
**Estimated implement time:** ~5 min
|
||||
**Estimated implement time:** ~4 min
|
||||
**Parallelizable with:** Task 4, Task 7
|
||||
|
||||
> **Amended 2026-07-19 (execution review).** The original Task 5 said "any `ORDER BY id`
|
||||
> becomes `ORDER BY timestamp`" and listed 3 files. Code recon found the integer `id` is
|
||||
> load-bearing in three separate places beyond the writer — a keyset cursor, the
|
||||
> storage-cap purge, and `long`-typed DTOs that cross the site↔central Akka boundary.
|
||||
> Task 5 is therefore split: **5a = the writer** (this task, self-contained), **5b = the
|
||||
> read/purge/contract change** (high-risk, its own task). Do not fold 5b back into 5a.
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs`
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/EventLogQueryService.cs`
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/ServiceCollectionExtensions.cs` (AddSiteEventLogging registration)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/` (existing suite)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/` (writer-path tests only)
|
||||
|
||||
**Step 1:** `SiteEventLogger` constructor gains `ILocalDb localDb`; its single owned `_connection` becomes `localDb.CreateConnection()`. Keep the `connectionStringOverride` test seam by preferring the override when non-null (tests may still use raw files for non-replicated scenarios) — but note overridden connections lack the HLC UDF, so tests that write must use the ILocalDb path once tables are registered; simplest is to switch the test fixture to a real `ILocalDb` and delete the override parameter if nothing else uses it.
|
||||
**Step 1:** `SiteEventLogger` constructor gains `ILocalDb localDb`; its single owned `_connection` (`SiteEventLogger.cs:34,62-65`) becomes `localDb.CreateConnection()` — already open, pragma-configured, HLC-UDF-registered; do **not** call `Open()` again. Delete the `connectionStringOverride` seam (`:50`) rather than preserving it: a raw connection lacks `zb_hlc_next()`, so any test writing through the override against a registered table now **fails closed**. Switch the test fixture to a real `ILocalDb` over a temp file.
|
||||
|
||||
**Step 2:** The insert statement gains an `id` column bound to `Guid.NewGuid().ToString("N")` (minted in the writer loop per event). `EventLogQueryService` (read-only) switches its connection source the same way; any `ORDER BY id` becomes `ORDER BY timestamp` (GUIDs don't sort chronologically — check the query paths and the UI paging contract).
|
||||
**Step 2:** Preserve the `internal WithConnection` seam (`:104`, `:123`) — `EventLogQueryService` and `EventLogPurgeService` both depend on it deliberately (their ctors take the concrete `SiteEventLogger`, not the interface). Its body changes connection source only; the signature and locking semantics stay identical, so 5b's consumers keep compiling.
|
||||
|
||||
**Step 3:** Fix remaining writer-path tests from Task 2. Full `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests` — PASS.
|
||||
**Step 3:** The INSERT (`:237-240`) gains an `id` column bound to `Guid.NewGuid().ToString("N")`, minted per event in the writer loop. `ISiteEventLogger.LogEventAsync`'s signature is unchanged — all 18 fire-and-forget call sites are untouched.
|
||||
|
||||
**Step 4:** Commit.
|
||||
**Step 4:** Fix writer-path tests from Task 2. Run `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests` — writer tests PASS. Query/purge tests may be red here; 5b closes them. Note that in the commit message.
|
||||
|
||||
**Step 5:** Commit.
|
||||
|
||||
---
|
||||
|
||||
### Task 5b: Migrate the event-log read path off integer ids
|
||||
|
||||
**Classification:** high-risk (cross-cluster data contract + silent-data-loss paging bug)
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** Task 6, Task 8, Task 9
|
||||
**Blocked by:** Task 5a
|
||||
|
||||
The GUID PK breaks three integer-id assumptions. All three must move together — shipping any one alone is a live bug.
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/EventLogQueryService.cs` (cursor `:68-71`, `ORDER BY id ASC` `:141`, `GetInt64(0)` `:153`, token mint `:181`)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/EventLogPurgeService.cs` (storage-cap delete `:162-166`)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Messages/RemoteQuery/EventLogQueryResponse.cs` (`EventLogEntry.Id` `long`→`string`; `ContinuationToken` `long?`→`string?`)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.Commons/Messages/RemoteQuery/EventLogQueryRequest.cs` (`ContinuationToken` `long?`→`string?`)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.ManagementService/ManagementActor.cs:3127` (passes `null` today — verify it still compiles)
|
||||
- Modify: `src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/EventLogs.razor:247,268,271` (`_continuationToken` type)
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests/` (query + purge)
|
||||
|
||||
**Step 1: Write the failing tests first.**
|
||||
- Paging: insert N events with known timestamps, page through with `PageSize` < N, assert **every** row is returned exactly once across pages and in chronological order. This test fails loudly against a naive `id > $afterId` GUID cursor — that is the point.
|
||||
- Ordering: rows come back oldest-first regardless of GUID values.
|
||||
- Purge: with the storage cap tripped, the **oldest** rows are deleted, not arbitrary ones.
|
||||
|
||||
**Step 2:** Run — FAIL.
|
||||
|
||||
**Step 3: Implement.**
|
||||
- **Cursor** → composite keyset on `(timestamp, id)`, since `timestamp` alone is not unique:
|
||||
`WHERE (timestamp > $afterTs) OR (timestamp = $afterTs AND id > $afterId)`, `ORDER BY timestamp ASC, id ASC`. The continuation token encodes both — use `"{timestamp}|{id}"` as the `string?` token; parse defensively and treat a malformed token as "start from the beginning" rather than throwing.
|
||||
- **Purge** (`:162-166`) → `ORDER BY timestamp ASC` instead of `ORDER BY id ASC`. (The retention purge at `:128` already filters on `timestamp` and is unaffected.)
|
||||
- **DTOs** → `Id` and both `ContinuationToken`s become `string`/`string?`; reader uses `GetString(0)`.
|
||||
|
||||
**Step 4: Wire-compat check.** These DTOs cross the site↔central Akka boundary (`SiteCommunicationActor.cs:193` → `CommunicationService.cs:304` → ManagementActor / CentralUI). Both sides ship in the same deployable and the rig redeploys as a unit, so no rolling-upgrade shim is needed — **but state that explicitly in the commit message**, and confirm no serializer manifest/binding pins these types by name+type (grep the Akka serialization setup). If a binding exists, stop and surface it.
|
||||
|
||||
**Step 5:** `dotnet test tests/ZB.MOM.WW.ScadaBridge.SiteEventLogging.Tests` + full solution build (0 warnings) — PASS. Commit.
|
||||
|
||||
---
|
||||
|
||||
@@ -228,7 +274,8 @@ services.AddZbLocalDb(config, SiteLocalDbSetup.OnReady);
|
||||
- Test: `tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbLegacyMigratorTests.cs`
|
||||
|
||||
Semantics (write the tests first, one per bullet):
|
||||
- Legacy files located from the OLD config keys (`ScadaBridge:OperationTracking:ConnectionString`, `ScadaBridge:SiteEventLog` DatabasePath); missing file ⇒ no-op.
|
||||
- Legacy files located from the OLD config keys (`ScadaBridge:OperationTracking:ConnectionString`, `ScadaBridge:SiteEventLog:DatabasePath`); missing file ⇒ no-op.
|
||||
- **Both keys are unset in every docker appsettings today**, so they fall back to the code defaults `"Data Source=site-tracking.db"` (`OperationTrackingOptions.cs:14`) and `"site_events.db"` (`SiteEventLogOptions.cs:10`) — **CWD-relative**, i.e. `/app/site-tracking.db` and `/app/site_events.db`, NOT `/app/data/`. Resolve relative paths against the process CWD; do not assume the data volume. (Consequence worth knowing: both DBs live outside the mounted volume today and are already lost on every container recreate — so on the rig the migrator will usually find nothing, and that is a legitimate no-op, not a failure. Phase 1 incidentally fixes that data-loss bug by consolidating into `/app/data/site-localdb.db`.)
|
||||
- Runs **after** `RegisterReplicated` so migrated rows enter the oplog and replicate.
|
||||
- Tracking rows copy with `INSERT OR IGNORE` (same TEXT PK ⇒ idempotent).
|
||||
- Legacy `site_events` rows get **deterministic** ids — `"mig-{NodeName}-{legacyId}"` — NOT fresh GUIDs, so a crashed-then-rerun migration cannot duplicate events (`INSERT OR IGNORE` on the deterministic key). NodeName from `ScadaBridge:Node:NodeName`.
|
||||
@@ -312,6 +359,7 @@ Bridge `ISyncStatus` (`Connected`, `OplogBacklog` — **nullable**, null must su
|
||||
**Classification:** high-risk
|
||||
**Estimated implement time:** ~5 min
|
||||
**Parallelizable with:** Task 11
|
||||
**Blocked by:** Task 6, Task 8, **Task 5b** (scenario 3 reads events back through the query path)
|
||||
|
||||
**Files:**
|
||||
- Create: `tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbSitePairConvergenceTests.cs`
|
||||
|
||||
@@ -4,23 +4,192 @@
|
||||
"mode": "parallel-waves",
|
||||
"implementerModel": "opus",
|
||||
"isolation": "worktree",
|
||||
"branch": "feat/localdb-phase1",
|
||||
"note": "Dispatch every unblocked task concurrently (waves in the plan header). Parallel implementers MUST use worktree isolation - concurrent git in one worktree races destructively."
|
||||
},
|
||||
"tasks": [
|
||||
{"id": 1, "subject": "Task 1: Add LocalDb packages to the build", "status": "pending"},
|
||||
{"id": 2, "subject": "Task 2: Shared schema helpers (tracking + events, GUID PK)", "status": "pending", "blockedBy": [1]},
|
||||
{"id": 3, "subject": "Task 3: Register LocalDb in the site composition root", "status": "pending", "blockedBy": [2]},
|
||||
{"id": 4, "subject": "Task 4: Rewire OperationTrackingStore onto ILocalDb connections", "status": "pending", "blockedBy": [3]},
|
||||
{"id": 5, "subject": "Task 5: Rewire SiteEventLogger + EventLogQueryService (GUID ids)", "status": "pending", "blockedBy": [3]},
|
||||
{"id": 6, "subject": "Task 6: One-time legacy data migrator", "status": "pending", "blockedBy": [4, 5]},
|
||||
{"id": 7, "subject": "Task 7: Replication registration + sync endpoint (default OFF)", "status": "pending", "blockedBy": [3]},
|
||||
{"id": 8, "subject": "Task 8: Bearer auth interceptor for the sync endpoint", "status": "pending", "blockedBy": [7]},
|
||||
{"id": 9, "subject": "Task 9: Surface ISyncStatus as site health signal", "status": "pending", "blockedBy": [7]},
|
||||
{"id": 10, "subject": "Task 10: Two-node in-process convergence test", "status": "pending", "blockedBy": [6, 8]},
|
||||
{"id": 11, "subject": "Task 11: Docker rig enablement (site-a pair)", "status": "pending", "blockedBy": [8]},
|
||||
{"id": 12, "subject": "Task 12: Live gate on the docker rig", "status": "pending", "blockedBy": [9, 10, 11]},
|
||||
{"id": 13, "subject": "Task 13: Documentation truth pass", "status": "pending", "blockedBy": [12]},
|
||||
{"id": 14, "subject": "Task 14: Phase 2 gate - plan the bespoke-replicator migration", "status": "pending", "blockedBy": [12]}
|
||||
"amendments": [
|
||||
{
|
||||
"date": "2026-07-19",
|
||||
"by": "execution review",
|
||||
"change": "Task 5 split into 5a (id 5, writer) and 5b (id 15, read path). Code recon found the integer site_events.id load-bearing in the keyset cursor (EventLogQueryService:70,141,153), the storage-cap purge (EventLogPurgeService:164), and long-typed DTOs crossing the site<->central Akka boundary (EventLogEntry.Id, both ContinuationTokens) - none of which were in the original Task 5 file set. 5b is high-risk. Task 10 gains a blockedBy on 15."
|
||||
},
|
||||
{
|
||||
"date": "2026-07-19",
|
||||
"by": "execution",
|
||||
"change": "Task 2 in isolation left the production writer broken (site_events.id NOT NULL, nothing minting it) - 40 tests red. Per the plan's own 'one PR-sized push' guidance, 5a's GUID minting and all of 5b landed together with Task 2 in commit 727fa48c. Task 5a remains OPEN for its other half: the ILocalDb.CreateConnection() swap, which still needs Task 3."
|
||||
}
|
||||
],
|
||||
"lastUpdated": "2026-07-19"
|
||||
}
|
||||
"prerequisites": [
|
||||
{
|
||||
"commit": "4715f2f9",
|
||||
"what": "gRPC 2.71.0 -> 2.76.0 + Protobuf 3.34.1. Forced by LocalDb.Replication 0.1.0's nuspec floor; restore fails NU1605 below it. Directory.Packages.props had deferred this bump to 'its own reviewed commit' - honored."
|
||||
},
|
||||
{
|
||||
"commit": "f056b67e",
|
||||
"what": "AngleSharp GHSA-pgww-w46g-26qg scoped suppression in CentralUI.Tests (supersedes the wrong pin in ecf6b628). Pre-existing: the solution build was RED on clean main, making every task's verification gate meaningless. No patched-and-working version exists upstream - 1.5.x breaks bunit at runtime, and bunit 2.7.2 still resolves vulnerable 1.4.0."
|
||||
}
|
||||
],
|
||||
"tasks": [
|
||||
{
|
||||
"id": 1,
|
||||
"subject": "Task 1: Add LocalDb packages to the build",
|
||||
"status": "completed",
|
||||
"commit": "7370b331"
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"subject": "Task 2: Shared schema helpers (tracking + events, GUID PK)",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
1
|
||||
],
|
||||
"commit": "727fa48c"
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"subject": "Task 3: Register LocalDb in the site composition root",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
2
|
||||
],
|
||||
"commit": "2977e6c4",
|
||||
"note": "LocalDb:Path added to 10 configs (8 site nodes incl. docker-env2 site-x, dev template, wonder prod sample) - it is REQUIRED via ValidateOnStart, so any site config missing it fails startup."
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
"subject": "Task 4: Rewire OperationTrackingStore onto ILocalDb connections",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
3
|
||||
],
|
||||
"commit": "0b5e9b44",
|
||||
"note": "Ctor takes ILocalDb, not IOptions<OperationTrackingOptions>; both ad-hoc reader paths switched too and their OpenAsync calls removed (CreateConnection returns an OPEN connection). Three DI fixtures needed LocalDb:Path added because resolving IOperationTrackingStore now forces ILocalDb construction."
|
||||
},
|
||||
{
|
||||
"id": 5,
|
||||
"subject": "Task 5a: Rewire SiteEventLogger onto ILocalDb + mint GUID ids",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
3
|
||||
],
|
||||
"commit": "727fa48c + 0d8a80aa",
|
||||
"note": "GUID minting landed early with Task 2 (727fa48c) to avoid committing a writer that violated site_events.id NOT NULL. The ILocalDb.CreateConnection() swap is 0d8a80aa. connectionStringOverride DELETED, not preserved: site_events is replicated, so a raw connection lacks zb_hlc_next() and fails closed."
|
||||
},
|
||||
{
|
||||
"id": 15,
|
||||
"subject": "Task 5b: Migrate the event-log read path off integer ids",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
5
|
||||
],
|
||||
"commit": "727fa48c"
|
||||
},
|
||||
{
|
||||
"id": 6,
|
||||
"subject": "Task 6: One-time legacy data migrator",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
4,
|
||||
5
|
||||
],
|
||||
"commit": "98b94771",
|
||||
"note": "Runs AFTER RegisterReplicated (pinned by an oplog assertion). Deterministic mig-{NodeName}-{legacyId} event ids, not fresh GUIDs, so a crash between commit and rename cannot duplicate. DEVIATION: sync transaction on a CreateConnection() connection instead of BeginTransactionAsync - OnReady is a synchronous Action<ILocalDb>."
|
||||
},
|
||||
{
|
||||
"id": 7,
|
||||
"subject": "Task 7: Replication registration + sync endpoint (default OFF)",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
3
|
||||
],
|
||||
"commit": "a560e9ea",
|
||||
"note": "DEVIATION: AddZbLocalDbReplication moved from Program.cs (where the plan put it) into SiteServiceRegistration, because the composition-root tests build that graph and not Program.cs - in Program.cs a missing registration would still show green. MapZbLocalDbSync stays in Program.cs (needs the WebApplication). Endpoint is UNAUTHENTICATED until Task 8; no config in this repo sets a peer."
|
||||
},
|
||||
{
|
||||
"id": 8,
|
||||
"subject": "Task 8: Bearer auth interceptor for the sync endpoint",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
7
|
||||
],
|
||||
"commit": "59c69519",
|
||||
"note": "Fail-closed: no configured ApiKey rejects ALL sync streams. All four handler shapes gated, not just unary (the real RPC is a duplex stream). DEVIATION: Grpc.Core.Testing does not exist on grpc-dotnet; hand-rolled FakeServerCallContext instead of adding the retired native package."
|
||||
},
|
||||
{
|
||||
"id": 9,
|
||||
"subject": "Task 9: Surface ISyncStatus as site health signal",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
7
|
||||
],
|
||||
"commit": "59c69519",
|
||||
"note": "Additive init properties on SiteHealthReport so the Akka DTO ctor is untouched. Both fields nullable: null = no data, false/0 = a real reading (which is the healthy default-OFF state). Collector stores one tuple, read once, so a torn read cannot pair fresh Connected with stale backlog."
|
||||
},
|
||||
{
|
||||
"id": 10,
|
||||
"subject": "Task 10: Two-node in-process convergence test",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
6,
|
||||
8,
|
||||
15
|
||||
],
|
||||
"commit": "3c395d79",
|
||||
"note": "5 scenarios over real loopback gRPC + the real interceptor, initialized via SiteLocalDbSetup.OnReady. Verified NON-vacuous: mismatching the two ApiKeys turns all 5 red with 2m30s of timeouts."
|
||||
},
|
||||
{
|
||||
"id": 11,
|
||||
"subject": "Task 11: Docker rig enablement (site-a pair)",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
8
|
||||
],
|
||||
"commit": "(with Task 11 commit)",
|
||||
"note": "site-a replicated, site-b/site-c deliberately not, so default-OFF is proven side-by-side on one rig."
|
||||
},
|
||||
{
|
||||
"id": 12,
|
||||
"subject": "Task 12: Live gate on the docker rig",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
9,
|
||||
10,
|
||||
11
|
||||
],
|
||||
"commit": "95c108f1",
|
||||
"note": "PASS. Found one REAL defect: ZbTelemetryOptions.Meters is a silent allowlist, so localdb_* was absent from /metrics - the plan's 'flows through automatically' claim was wrong. Fixed + pinned. Evidence: identical row_version/HLC/originating node_id across the pair, 0 dead letters, oplog drained, site failover + rejoin catch-up, localdb_* scraped, site-b inert. NOTE: docker/failover-drill.sh targets CENTRAL nodes and does NOT exercise the site-pair claim; the site flip was run directly."
|
||||
},
|
||||
{
|
||||
"id": 13,
|
||||
"subject": "Task 13: Documentation truth pass",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
12
|
||||
],
|
||||
"commit": "(docs commit)",
|
||||
"note": "Both CLAUDE.md files + the false 'starts a fresh log' XML doc + the two now-migration-only option properties."
|
||||
},
|
||||
{
|
||||
"id": 14,
|
||||
"subject": "Task 14: Phase 2 gate - plan the bespoke-replicator migration",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
12
|
||||
],
|
||||
"commit": "(gate commit)",
|
||||
"note": "Gate document only, per the plan - Phase 2 gets its own implementation plan. docs/plans/2026-07-19-localdb-phase2-gate.md. KEY FINDING: the rig SOAK that Phase 2's oplog-cap tuning needs has NOT been run; the live gate proved correctness, not steady-state behaviour."
|
||||
}
|
||||
],
|
||||
"knownFlakes": [
|
||||
{
|
||||
"test": "SiteRuntime.Tests InstanceActorChildAttributeRaceTests.ChildActors_AreSeededFromAnIsolatedCopy_NotTheLiveAttributesDictionary",
|
||||
"note": "Intermittent ActorNotFoundException under full-suite load; passes in isolation and in most full runs. Pre-existing, unrelated to LocalDb. Worth its own issue."
|
||||
},
|
||||
{
|
||||
"test": "AuditLog.Tests ParentExecutionIdCorrelationTests.InboundRoutedRun_AllRoutedRows_CarryInboundExecutionId_AsParentExecutionId",
|
||||
"note": "Cold-MSSQL-fixture timing: ~91s and AwaitAssert-times-out on a cold fixture, ~1s and passes warm. Baselined against a stashed tree to confirm it is not a LocalDb regression."
|
||||
}
|
||||
],
|
||||
"lastUpdated": "2026-07-19",
|
||||
"phase1Status": "COMPLETE - all 15 tasks done, live gate PASS. Branch feat/localdb-phase1 NOT merged/pushed."
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,106 @@
|
||||
{
|
||||
"planPath": "docs/plans/2026-07-19-localdb-adoption-phase2.md",
|
||||
"execution": {
|
||||
"mode": "parallel-waves",
|
||||
"implementerModel": "opus",
|
||||
"isolation": "worktree",
|
||||
"branch": "feat/localdb-phase2",
|
||||
"baseBranch": "feat/localdb-phase1",
|
||||
"note": "Phase 1's branch is NOT merged/pushed, so phase 2 branches from it. Dispatch every unblocked task concurrently per the wave table in the plan. Parallel implementers MUST use worktree isolation - concurrent git in one worktree races destructively."
|
||||
},
|
||||
"scopeDecision": {
|
||||
"date": "2026-07-19",
|
||||
"by": "user",
|
||||
"choice": "Full scope as designed",
|
||||
"note": "User chose both surfaces (config tables + sf_messages) in one phase, deleting both bespoke mechanisms together, over the recommended split of S&F-first. The four open design questions are therefore resolved INSIDE this plan as D1-D6 rather than deferred."
|
||||
},
|
||||
"reviewPass": {
|
||||
"date": "2026-07-19",
|
||||
"note": "Plan verified against actual code (3 verification sweeps + LocalDb library source) and corrected in place. Headline corrections: D1 (StoreDeployedConfigIfNewerAsync has a SECOND surviving caller, SiteReconciliationActor.cs:166 - the method and guard STAY; original Task 13 would also not have compiled, deleting a method whose caller dies only in Task 15), D3 (the active node ALREADY purges at DeploymentManagerActor.cs:1921 - Task 12 became a pin test, nothing is re-homed), D6 added (4 MB gRPC receive cap x row-count-only MaxBatchSize batching; config_json > 128 KB documented - measure in Task 1, size MaxBatchSize in Task 19, single row near 4 MB = stop/lib work), Task 1 rewritten (Phase 2 tables are NOT registered on the Phase 1 rig so driven churn never reaches __localdb_oplog - measure legacy-DB write rates + arithmetic; metrics port 8084 not 8080; real metric names are localdb_oplog_depth / localdb_sync_*, the plan's localdb_oplog_backlog/replication_dead_letters/sync_connected never existed; containers have no sqlite3 - sample via throwaway copies of the DB triplet, NEVER host-side sqlite3 against the live files [2026-07-20: that poisons the container's WAL state - see docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md]; branch already exists, no checkout -b), Task 14 (register 8 tables NOT 11 - notification_lists/smtp_configurations deliberately unregistered, reversing the original instruction; keep Migrate LAST in OnReady)."
|
||||
},
|
||||
"decisions": [
|
||||
{
|
||||
"id": "D1",
|
||||
"subject": "Config moves to CDC; notify-and-fetch is DELETED - but the guarded write STAYS",
|
||||
"evidence": "SiteReplicationActor sends id+fetch-coords only because the config blob exceeds Akka's 128KB frame (docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md, already marked RESOLVED by the notify-and-fetch rework). LocalDb sync is gRPC - no such limit (but see D6). CORRECTED: StoreDeployedConfigIfNewerAsync (SiteStorageService.cs:301-336, guard at :325) has TWO production callers - SiteReplicationActor.cs:375 (dies in Task 15) AND SiteReconciliationActor.cs:166 (per-node startup self-heal vs central, SURVIVES Phase 2, stale-fetch race still real). Method + guard stay; reconcile becomes a benign second CDC writer. Do NOT reproduce the guard on top of LWW - deployed_at and HLC are different clocks and mixing them is non-convergent."
|
||||
},
|
||||
{
|
||||
"id": "D2",
|
||||
"subject": "ReplaceAllAsync deleted; the N1 directional guard becomes unnecessary",
|
||||
"evidence": "LocalDb's snapshot resync MERGES per-row LWW and never WIPES: SnapshotApplier.OnBeginAsync (SnapshotStreamer.cs:163-170) resets counters only; OnBatchAsync (:172-186) routes snapshot rows through the same LwwApplier as deltas; LwwApplier.cs:69-78 discards an incoming row whose HLC is lower. Row-level deletes DO replicate (delete-trigger tombstones, streamed by SnapshotStreamer, applied as real DELETEs by LwwApplier) - only the destructive whole-table replace is gone. Caveat: tombstones pruned after TombstoneRetention (default 7d); a node offline longer can resurrect deleted rows (runbook, Task 21). SEMANTIC CHANGE: the standby is convergent, no longer byte-identical."
|
||||
},
|
||||
{
|
||||
"id": "D3",
|
||||
"subject": "CORRECTED: the SMTP purge already runs on the active node - pin it, don't move it",
|
||||
"evidence": "PurgeCentralOnlyNotificationConfigAsync (SiteStorageService.cs:811-821) has TWO callers: DeploymentManagerActor.cs:1921 (ACTIVE node's HandleDeployArtifacts, :1864-1963) and SiteReplicationActor.cs:456 (standby copy, dies with the actor). The purge never lapses; the original 're-home before any deletion' premise was false. Task 12 = pin test only (ArtifactStorageTests covers the storage method, not the actor call site Task 16 edits). No site writer to notification_lists/smtp_configurations since 2026-07-10 (verified: only test seeding inserts exist) + migrator skips them => permanently empty in the consolidated DB => Task 14 does NOT register them."
|
||||
},
|
||||
{
|
||||
"id": "D4",
|
||||
"subject": "native_alarm_state volume is MEASURED, not assumed",
|
||||
"evidence": "scadabridge.db is not only config - native_alarm_state mirrors live A&C conditions (NativeAlarmActor.cs:504) and is the highest-volume table in either DB. sf_messages worst case ~50 row-writes/sec. Task 1 measures both AT THE LEGACY-DB SOURCE (they are not in the Phase 1 oplog - see reviewPass) and sets MaxOplogRows/MaxOplogAge arithmetically; Task 20 evidence 10 does the empirical post-cutover drain check. If growth is monotonic, STOP: keyed-instances escape hatch = ~/Desktop/scadaproj/docs/plans/2026-07-19-scadabridge-localdb-design.md:139-141 (adoption design doc, NOT the 07-17 lib doc), a scadaproj library effort that would suspend this plan."
|
||||
},
|
||||
{
|
||||
"id": "D5",
|
||||
"subject": "Cutover forecloses rolling site upgrades",
|
||||
"evidence": "SiteReplicationActor retains a legacy monolithic SfBufferSnapshot handler for rolling upgrades. With no dual-mechanism period, one node would speak a protocol the other no longer implements. Both nodes of a site must be stopped and started together. Task 21 puts this in the deployment docs (installation-guide.md/topology-guide.md - no file literally named runbook)."
|
||||
},
|
||||
{
|
||||
"id": "D6",
|
||||
"subject": "NEW (review pass): 4 MB gRPC message cap replaces the 128 KB Akka frame as the size ceiling",
|
||||
"evidence": "Neither side configures gRPC message sizes (ScadaBridge AddGrpc at Program.cs:519-521 sets only the auth interceptor; the lib's initiator channel is bare GrpcChannel.ForAddress) => 4 MB default receive limit both directions. Batching is row-count-only (MaxBatchSize default 500; SyncSession.cs:227, SnapshotStreamer.cs:55; no byte-aware chunking). deployed_configurations.config_json documented >128 KB/row; a few dozen such rows in one batch exceeds 4 MB and wedges the stream on a poison batch. Task 1 measures max/avg config_json bytes; Task 19 sets LocalDb:Replication:MaxBatchSize so max-row-bytes x MaxBatchSize << 4 MB; any single row near 4 MB = STOP (needs byte-aware batching or size knobs in the LocalDb lib - scadaproj effort)."
|
||||
}
|
||||
],
|
||||
"reconFindings": [
|
||||
"The gate doc named 2 test files as the specification; the real spec is 5 - it missed StoreAndForwardReplicationTests.cs (incl. the only Requeue coverage), ReplicationWireSerializationPinTests.cs, ResyncWireSerializationPinTests.cs, and SfBufferResyncPredicateTests.cs (the N1 Critical regression test).",
|
||||
"sf_messages has NO version column today - ON CONFLICT(id) DO UPDATE has no comparison predicate (StoreAndForwardStorage.cs:331-345). 'Newest wins' is bare arrival order. LWW-by-HLC is an IMPROVEMENT here, not a regression.",
|
||||
"No autoincrement-integer PKs exist anywhere in Phase 2 scope - all 9 config tables use natural TEXT or composite TEXT keys, sf_messages is TEXT. LocalDb RegisterReplicated SUPPORTS composite PKs (ordered pk ordinals) and rejects BLOB columns - no Phase 2 table has one (verified). Phase 1's site_events GUID conversion has no Phase 2 analogue.",
|
||||
"SiteStorageService has NO foreign keys. RemoveDeployedConfigAsync (:343-376) is a manual 3-statement cascade in one transaction. Under CDC these become three independent delete streams that LWW may reorder - the most likely real defect in the plan (Task 18 scenario 2).",
|
||||
"DeploymentManagerActor's replicationActor is an OPTIONAL POSITIONAL parameter at :169 (ctor :161-175; :184 is the field ASSIGNMENT). The same-typed IActorRef? optional dclManager sits immediately BEFORE it at :168 - that's the real silent-shift hazard. Props.Create passes it positionally at AkkaHostedService.cs:810. Check every call site by hand (Task 16).",
|
||||
"ActiveNodeEvaluator must NOT be deleted - the S&F delivery gate still uses it (AkkaHostedService.cs:866 SetDeliveryGate -> SelfIsPrimary -> SelfIsOldestUp). Doc comment :14/:16 mentions replication; SiteReplicationActor.cs:288 calls it directly (dies with the actor).",
|
||||
"ConfigFetchRetryCount's ONLY production reader is SiteReplicationActor.cs:157 (verified) - dead after Task 15; removed in Task 17 (removing it in Task 13, before the actor deletion, would not compile). IDeploymentConfigFetcher is KEPT: DeploymentManagerActor refresh path + SiteReconciliationActor + DI at ServiceCollectionExtensions.cs:84.",
|
||||
"notification_lists and smtp_configurations are deliberately NOT migrated (Task 9) AND NOT registered (Task 14, corrected) - migrating or replicating them would resurrect/ship plaintext SMTP passwords; they are permanently empty by design (writers removed 2026-07-10; verified only test-seeding inserts exist).",
|
||||
"REVIEW-PASS ADDITIONS: SiteReconciliationActor (runs on EVERY node at startup, best-effort self-heal vs central) is the second caller of both StoreDeployedConfigIfNewerAsync and IDeploymentConfigFetcher - it survives Phase 2 and constrains Tasks 11/13/20 (a startup fetch on the standby is legitimate; zero-fetch assertions must scope to the deploy window).",
|
||||
"SiteStorageService has 21 (not 22) inline connection+OpenAsync pairs: 60,206,248,311,345,386,414,443,470,498,538,583,608,640,663,692,730,769,813,837,868. CreateConnection():51's only repository consumer is SiteExternalSystemRepository.",
|
||||
"StoreAndForwardService: :39 is the _replication FIELD, :243 the ctor param; exactly 6 emission sites (:654,805,836,872,1120,1146). ServiceCollectionExtensions.cs:32 also resolves ReplicationService inside the StoreAndForwardService factory - must go in Task 14.",
|
||||
"ReplicationMessages.cs holds ONLY the 10 Replicate*/Apply* records; the four SfBuffer resync records live at the bottom of SiteReplicationActor.cs (:678-707) and die with the actor file.",
|
||||
"10 (not 9) appsettings.Site.json files set the legacy paths - deploy/wonder-app-vd03/appsettings.Site.json was missed (also sets ReplicationEnabled:false).",
|
||||
"Rig facts: site metrics on port 8084 (AnyIP in-container, NOT published to host; 8080 is Traefik); real metric names localdb_oplog_depth / localdb_sync_* (meter ZB.MOM.WW.LocalDb.Replication); containers (aspnet:10.0) have NO sqlite3 - data dirs are host bind mounts but NEVER run sqlite3 host-side against the live files (poisons the container's WAL state, root cause of the 2026-07-20 disk-I/O-error incident): cp the .db/-wal/-shm triplet and query the copy (see the snap() helper in the plan); ReplicationOptions bind at LocalDb:Replication:* (lib binds the section).",
|
||||
"Phase 1's LocalDbSitePairConvergenceTests uses ONE shared API key (:47) - it never did a mismatched-key non-vacuity run; wrong-key denial is unit-covered in Host.Tests/LocalDbSyncAuthInterceptorTests.cs. Task 18's non-vacuity check must be done directly for the new scenarios.",
|
||||
"MigrateEvents synthesizes deterministic 'mig-{node}-{legacyId}' ids (NOT fresh GUIDs - crash-rerun idempotency). Migration runs AFTER RegisterReplicated in OnReady (order is load-bearing); Task 8's oplog pin test must register sf_messages on its own TestLocalDb since production doesn't register it until Task 14."
|
||||
],
|
||||
"tasks": [
|
||||
{"id": 1, "subject": "Task 1: Rig soak - measure legacy-DB write rates + config_json sizes (corrected method)", "status": "completed", "classification": "high-risk", "note": "GATE CLOSED - verdict PROCEED (2026-07-20). NEVER sample host-side sqlite3 on live bind-mounted WAL files (that method poisoned the first run - use the copy-based snap() helper). Clean re-run: sf_messages 0.80 rows/sec insert, 0/sec retry-UPDATE, oplog 0, alarms 0, max payload_json 76 B, max config_json 721 B (NOT representative), zero SQLite errors in 30 min, both nodes converged. No stop condition met. Binding output: MaxBatchSize 500 -> 16 (Task 19). Empirical drain check remains Task 20 evidence 10."},
|
||||
{"id": 2, "subject": "Task 2: Decision record - close the phase 2 gate", "status": "completed", "classification": "trivial", "blockedBy": [1], "note": "Gate doc status flipped NOT STARTED -> CLOSED; all 5 SS5 questions answered inline (questions kept, not deleted); SS2's N5 duplicate-bound requirement routed explicitly to Task 21."},
|
||||
{"id": 3, "subject": "Task 3: Extract StoreAndForwardSchema.Apply", "status": "completed", "classification": "small", "blockedBy": [2], "note": "DEVIATION: kept PRAGMA journal_mode=WAL in InitializeAsync - the plan Step 4 snippet drops it but Task 4 says not to move the equivalent pragma; contradictory, and dropping it would regress documented concurrent-writer support before Task 5 makes it moot. Added a legacy-upgrade test beyond the plan's (the specified test asserts against a FRESH table where CREATE TABLE already lists all 16 columns - it would pass with every ALTER deleted). That test found the last_attempt_at_ms backfill lands 1 ms low: julianday() double day-fraction rounding, pre-existing, carried verbatim, asserted with 1 ms tolerance. Suite 154/154."},
|
||||
{"id": 4, "subject": "Task 4: Extract SiteStorageSchema.Apply", "status": "completed", "classification": "small", "blockedBy": [2], "note": "DEVIATION: TryAddColumnAsync's catch-on-message-text ('duplicate column') became a PRAGMA table_info probe matching OperationTrackingSchema - the message form depends on a non-contractual error string and swallowed unrelated SqliteExceptions; also dropped the ILogger dep, which is what let the class be static. Cost: the per-column 'Migrated: added column' info log is gone (nothing consumes it). PRAGMA journal_mode=WAL stays in InitializeAsync per plan. Added a legacy-upgrade test for the same reason as Task 3. SiteRuntime 532/532, full solution build 0 warnings."},
|
||||
{"id": 5, "subject": "Task 5: Rewire StoreAndForwardStorage onto ILocalDb", "status": "completed", "classification": "high-risk", "blockedBy": [3], "note": "Test fallout was ~7x the plan estimate: 40 files across 7 test projects, not \"fixtures\" in one. Most used Mode=Memory;Cache=Shared and LocalDb has NO in-memory mode, so all moved to real temp files. Added shared tests/ZB.MOM.WW.ScadaBridge.TestSupport lib (TestLocalDb) instead of copying the Phase 1 fixture into 7 projects. WAL test retargeted to the LocalDb-backed store; directory-creation test moved to Host.Tests (different owner - see Task 6 note)."},
|
||||
{"id": 6, "subject": "Task 6: Rewire SiteStorageService onto ILocalDb", "status": "completed", "classification": "high-risk", "blockedBy": [4], "note": "LATENT PHASE-1 DEFECT FOUND+FIXED: LocalDb does NOT create the parent directory and SqliteLocalDb opens the file eagerly, so a missing dir is a HARD BOOT FAILURE (SQLite Error 14). Default site config is the RELATIVE ./data/site-localdb.db; docker escapes only because the volume mount creates /app/data. The plan wrongly assumed dir-creation moved to LocalDb with file ownership. Fixed via SiteLocalDbDirectory.Ensure(config) before AddZbLocalDb + Host.Tests/SiteLocalDbDirectoryTests (non-vacuity observed: 2 tests failed with exactly Error 14 pre-fix). ALSO a contract change: SiteStorageService.CreateConnection() used to return an UNOPENED connection; it now returns an ALREADY-OPEN one - SiteExternalSystemRepository dropped 5 OpenAsync calls. AddSiteRuntime(string) overload deleted."},
|
||||
{"id": 7, "subject": "Task 7: Extend SiteLocalDbSetup with the new DDL (not yet registered)", "status": "completed", "classification": "standard", "blockedBy": [5, 6], "commit": "f8aa02e2", "note": "As planned. Two pins in SiteLocalDbWiringTests through the REAL composition root: all 12 tables exist, and ReplicatedTables is EXACTLY the Phase 1 pair (an equality check, not Contains - the 'not yet' is the assertion that matters). Task 14 should INVERT that second test, not delete it. Non-vacuity verified by removing the DDL."},
|
||||
{"id": 8, "subject": "Task 8: Extend the legacy migrator for sf_messages", "status": "completed", "classification": "high-risk", "blockedBy": [7], "commit": "bdc0dffe", "note": "DEVIATION: the copy INTERSECTS the legacy column set with the current one instead of naming all 16 columns. A legacy file from an older build lacks execution_id/parent_execution_id/last_attempt_at_ms, and naming a missing column throws 'no such column' - which the existing ReadAll treats as an unrecognised shape and SILENTLY DISCARDS EVERY ROW. For undelivered messages that is real data loss. A required-column (PK) guard stops the tolerance degrading into NULL-keyed copies. Tests register sf_messages on their own harness (production OnReady does not until Task 14), so Task 14 MUST keep Migrate as the LAST call in OnReady, after all registrations. Non-vacuity verified: all 4 fail without the Migrate call."},
|
||||
{"id": 9, "subject": "Task 9: Extend the legacy migrator for the config tables (skip notification/smtp)", "status": "completed", "classification": "high-risk", "blockedBy": [7], "commit": "5ddc7eed", "note": "7 of 9 tables migrated; notification_lists + smtp_configurations skipped as planned (plaintext passwords must not enter a table Task 14 makes replicated) - the skip is pinned by a test that also greps __localdb_oplog.row_json for the secret. Task 8 MigrateTable generalized to MigrateFile(many tables, one transaction, one rename). ADDED beyond the plan: MigratorColumnLists_MatchTheLiveSchema, a column-parity test vs SiteStorageSchema - the plan called for a manual column check, but a mismatch is invisible at runtime in BOTH directions (a typo is silently dropped by the intersection; a missing column silently leaves data behind). SiteStorageTables + LegacyTable are internal so the test can read them. Non-vacuity verified twice: removing the call (3 fail) and wrongly adding notification/smtp to the map (skip test fails)."},
|
||||
{"id": 10, "subject": "Task 10: Port the S&F replication test intents as CDC specs", "status": "completed", "classification": "standard", "blockedBy": [8], "commit": "2bbe6631", "note": "DEVIATION: extracted the Phase 1 fixture into LocalDbSitePairHarness (abstract base) instead of duplicating ~150 lines; Phase 1 tests derive from it and pass unchanged. The harness registers the 8 Phase 2 tables itself (production OnReady does not until Task 14) - DELETE RegisterPhase2TablesUntilCutover at Task 14. The 8-table list is literal, not derived from production code, so a cutover registering the wrong set fails these tests. Non-vacuity: unregistering sf_messages failed 6 of 7 - the 7th (add-then-remove ordering) PASSED because an absent row is also what a non-replicating pair looks like; fixed with a control row that must converge in the same window."},
|
||||
{"id": 11, "subject": "Task 11: Port the resync + directional-authority tests", "status": "completed", "classification": "standard", "blockedBy": [9], "note": "N1 re-expressed per D2 as a no-rows-lost property (LWW merge, never deletes), not directional authority. DEVIATION on the zero-fetch assertion: the plan wanted a fetcher double recording zero calls, but this harness has no actor system / no central / no IDeploymentConfigFetcher in the graph, so the double could not fail either way. Replaced with the positive half (config reaches B over replication alone) plus an in-file comment recording that the negative half is proved by Task 15 deleting the code and the build passing. D1 scope note recorded in-file. Non-vacuity: unregistering deployed_configurations fails all 4.", "commit": "c56bf4ae"},
|
||||
{"id": 12, "subject": "Task 12: Pin the active-node SMTP purge (corrected: already exists at DeploymentManagerActor.cs:1921)", "status": "completed", "classification": "standard", "blockedBy": [10, 11], "note": "As planned - NO production change; D3 confirmed correct, the call already exists at DeploymentManagerActor.cs:1921. Pin verified RED-FIRST by commenting out the call. Test needed a using for Commons.Messages.Artifacts and polls (the apply runs on a Task.Run inside the actor).", "commit": "79ce5161"},
|
||||
{"id": 13, "subject": "Task 13: Notify-and-fetch scope check - guarded write STAYS for SiteReconciliationActor", "status": "completed", "classification": "standard", "blockedBy": [12], "note": "Doc-comment only, as planned. Re-verified both callers: SiteReplicationActor:375 (dies Task 15) + SiteReconciliationActor:166 (survives). Step 2 scope check re-run: ConfigFetchRetryCount's only production reader is still SiteReplicationActor:157, so option + validator rule stay until Task 17. Also fixed a stale 'guarded standby write' header in SiteStorageServiceTests.", "commit": "79ce5161"},
|
||||
{"id": 14, "subject": "Task 14: Register the 8 Phase 2 tables and delete ReplicationService", "status": "completed", "classification": "high-risk", "blockedBy": [13], "note": "LANDED AS ONE COMMIT with tasks 15+16 - PLAN DEFECT: they cannot compile separately (SiteReplicationActor takes a ReplicationService + calls ReplaceAllAsync; DeploymentManagerActor Tells ReplicationMessages types; AkkaHostedService constructs the actor). Combining also strengthens Task 14's own invariant: never both mechanisms, never neither. Registered 8 tables; notification_lists + smtp_configurations deliberately NOT registered. Deleted ReplaceAllAsync as UNSAFE (a mass DELETE on a replicated table would be captured and shipped), and its test replaced by a comment explaining that. The positional-arg hazard was REAL: 4 DeploymentManagerActor test call sites bound wrong args; converted to named where possible - Props.Create is an expression tree and rejects OUT-OF-POSITION named args, so the rest are padded positionally. Task 7's 'not yet' test INVERTED (exact in both directions) + new security-named test for the SMTP tables + composite-PK test. Harness's temporary registration deleted, so the convergence suites now prove the cutover. My expected table list was mis-sorted: ordinal puts '_' (0x5F) before 'b', so data_connection_definitions precedes database_connections.", "commit": "037798b3"},
|
||||
{"id": 15, "subject": "Task 15: Delete SiteReplicationActor and its messages", "status": "completed", "classification": "high-risk", "blockedBy": [14], "note": "LANDED AS ONE COMMIT with tasks 15+16 - PLAN DEFECT: they cannot compile separately (SiteReplicationActor takes a ReplicationService + calls ReplaceAllAsync; DeploymentManagerActor Tells ReplicationMessages types; AkkaHostedService constructs the actor). Combining also strengthens Task 14's own invariant: never both mechanisms, never neither. Registered 8 tables; notification_lists + smtp_configurations deliberately NOT registered. Deleted ReplaceAllAsync as UNSAFE (a mass DELETE on a replicated table would be captured and shipped), and its test replaced by a comment explaining that. The positional-arg hazard was REAL: 4 DeploymentManagerActor test call sites bound wrong args; converted to named where possible - Props.Create is an expression tree and rejects OUT-OF-POSITION named args, so the rest are padded positionally. Task 7's 'not yet' test INVERTED (exact in both directions) + new security-named test for the SMTP tables + composite-PK test. Harness's temporary registration deleted, so the convergence suites now prove the cutover. My expected table list was mis-sorted: ordinal puts '_' (0x5F) before 'b', so data_connection_definitions precedes database_connections.", "commit": "037798b3"},
|
||||
{"id": 16, "subject": "Task 16: Clean up DeploymentManagerActor and AkkaHostedService", "status": "completed", "classification": "standard", "blockedBy": [15], "note": "LANDED AS ONE COMMIT with tasks 15+16 - PLAN DEFECT: they cannot compile separately (SiteReplicationActor takes a ReplicationService + calls ReplaceAllAsync; DeploymentManagerActor Tells ReplicationMessages types; AkkaHostedService constructs the actor). Combining also strengthens Task 14's own invariant: never both mechanisms, never neither. Registered 8 tables; notification_lists + smtp_configurations deliberately NOT registered. Deleted ReplaceAllAsync as UNSAFE (a mass DELETE on a replicated table would be captured and shipped), and its test replaced by a comment explaining that. The positional-arg hazard was REAL: 4 DeploymentManagerActor test call sites bound wrong args; converted to named where possible - Props.Create is an expression tree and rejects OUT-OF-POSITION named args, so the rest are padded positionally. Task 7's 'not yet' test INVERTED (exact in both directions) + new security-named test for the SMTP tables + composite-PK test. Harness's temporary registration deleted, so the convergence suites now prove the cutover. My expected table list was mis-sorted: ordinal puts '_' (0x5F) before 'b', so data_connection_definitions precedes database_connections.", "commit": "037798b3"},
|
||||
{"id": 17, "subject": "Task 17: Config-key cleanup (incl. ConfigFetchRetryCount; 10 appsettings files)", "status": "completed", "classification": "standard", "blockedBy": [16], "note": "As planned. DEVIATION: deploy/wonder-app-vd03/appsettings.Site.json sits under a GITIGNORED deploy/ tree, so its edit is local-only and must be repeated on the box at deploy time. Comment style is // (JSONC) rather than \"_comment_\" keys: every one of these files already contains // comments and .NET's json config reader accepts them, and a _comment_ key inside a bound section is a phantom config entry. Both relaxations pinned by the INVERSE of the test they replace (Site_MissingSiteDbPath_IsAccepted..., EmptySqliteDbPath_IsAccepted...), each verified red with the old rule restored. DatabaseOptionsValidator needed no change - it was already null-tolerant/blank-rejecting, which is exactly migration-only semantics.", "commit": "605e5682"},
|
||||
{"id": 18, "subject": "Task 18: Two-node convergence suite for the Phase 2 tables", "status": "completed", "classification": "high-risk", "blockedBy": [17], "note": "DEVIATION: landed as a NEW file LocalDbPhase2ConvergenceTests.cs rather than extending Phase 1's LocalDbSitePairConvergenceTests.cs, and drives the REAL SiteStorageService instead of hand-written SQL - possible only post-cutover, and what makes the cascade scenario test the shipped transaction rather than a re-creation of it. Scenario 4 was retargeted onto shared_scripts/external_systems/static_attribute_overrides because the plan's version overlapped LocalDbConfigConvergenceTests' N1 scenario almost exactly; the union-survives property is per-table, so re-proving it on untouched tables is the non-redundant half. Cascade test carries a never-removed control instance (absence assertions otherwise cannot distinguish 'cascade converged' from 'node B lost these tables'). Non-vacuity PROVEN as mandated: with the 8 RegisterReplicated calls commented out, 4 failed / 0 passed; restored, 20/20 across the three LocalDb suites.", "commit": "15013156"},
|
||||
{"id": 19, "subject": "Task 19: Rig configuration (MaxOplogRows/MaxOplogAge + MaxBatchSize per D6)", "status": "completed", "classification": "small", "blockedBy": [17], "note": "MaxBatchSize 500->16 (D6: row-count batching x ~70 KB production config_json vs the 4 MB gRPC cap; 16 => ~1.1 MB worst case). MaxOplogRows 1M->250,000 and MaxOplogAge 7d->2d from the soak's 0.80 rows/sec (~69k/day). Tighter-than-default is SAFE because a cap breach prunes + sets needs_snapshot (graceful snapshot resync), not data loss - the Task 1 finding that the plan's stop condition was weaker than written. site-b/site-c left unreplicated so default-OFF stays proven side by side.", "commit": "921edab4"},
|
||||
{"id": 20, "subject": "Task 20: Live gate on the docker rig", "status": "completed", "classification": "high-risk", "blockedBy": [18, 19], "note": "ALL 10 CHECKS PASS. Evidence: docs/plans/2026-07-19-localdb-phase2-live-gate.md. Key blocker found and fixed mid-run: external systems reach a site ONLY via ArtifactDeploymentService, which `instance deploy` never invokes - `deploy artifacts` was needed both to deliver the probe harness and to propagate the owed ExternalSystemDefinitions restore. THREE METHOD CORRECTIONS to the plan: (1) its instruction to run DB checks host-side against the bind mounts is UNSAFE - host sqlite3 poisons the container WAL; copy the db/-wal/-shm triplet and query the copy. (2) `docker exec ... curl` cannot scrape metrics (no curl in aspnet:10.0) and with 2>/dev/null the failure is silent - it nearly became a false 'metrics missing' finding; use a network-sharing curl sidecar. (3) checks needing S&F load need CachedCall, not Call. CAVEATS recorded not glossed: check 2's zero-count is vacuous alone (legacy source was also empty) and rests on the ABSENCE of CDC triggers; check 7's native_alarm_state leg was empty live and is covered only offline; check 10's sampling is coarse and the real rise/drain evidence comes from check 6's 0->4->0.", "commit": "158e79bb"},
|
||||
{"id": 21, "subject": "Task 21: Documentation truth pass", "status": "completed", "classification": "standard", "blockedBy": [20], "note": "Both CLAUDE.md files + Component-StoreAndForward.md:83 (normative resync paragraph rewritten for CDC, stating the duplicate-delivery bound explicitly: limited to messages the OLD primary delivered whose status change had not yet replicated when the gate flipped - one flush interval plus in-flight ack, and it does NOT grow with backlog depth or absence duration) + components/{StoreAndForward,SiteRuntime,Host}.md + the frame-size known-issue (amended: Phase 2 deleted notify-and-fetch itself, so the 128KB Akka frame constraint is gone from the intra-site hop entirely; successor ceiling is the 4MB gRPC cap via MaxBatchSize, and note the failure mode differs - oversized gRPC is REJECTED, not silently dropped) + deployment topology-guide.md and installation-guide.md (D5 stop-both-together, D2 TombstoneRetention resurrection bound). DoD closed: build 0 warnings, all 10 suites green (3509 tests, 0 failures). DoD grep nuance recorded in the plan: 4 matches remain in src/ and are all deliberate COMMENT prose explaining what was deleted - a literal 'no matches' would delete the explanations that stop someone re-introducing the old design.", "commit": null}
|
||||
],
|
||||
"knownFlakes": [
|
||||
{
|
||||
"test": "SiteRuntime.Tests InstanceActorChildAttributeRaceTests.ChildActors_AreSeededFromAnIsolatedCopy_NotTheLiveAttributesDictionary",
|
||||
"note": "Intermittent ActorNotFoundException under full-suite load; passes in isolation. Pre-existing, carried over from Phase 1."
|
||||
},
|
||||
{
|
||||
"test": "AuditLog.Tests ParentExecutionIdCorrelationTests.InboundRoutedRun_AllRoutedRows_CarryInboundExecutionId_AsParentExecutionId",
|
||||
"note": "Cold-MSSQL-fixture timing: ~91s and AwaitAssert-times-out cold, ~1s warm. Re-run before investigating."
|
||||
}
|
||||
],
|
||||
"lastUpdated": "2026-07-20",
|
||||
"phase2Status": "UNBLOCKED - Task 1 gate CLOSED (verdict PROCEED) and Task 2 DONE, 2026-07-20. Task 1's original STOP verdict is SUPERSEDED: the 'Phase 1 disk I/O defect' was OBSERVER-INDUCED (host-side sqlite3 against live bind-mounted WAL files resets the WAL across virtiofs and permanently poisons the container's connections) - NOT a product defect. See docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md. Both earlier isolation claims were confounded: one sampling pass had already poisoned BOTH nodes, and a poisoned standby looks healthy only because it issues almost no statements; 'LocalDb-specific' was sampling-selection bias (only the LocalDb file had ever been host-read). Clean re-run 2026-07-20 with the copy-based snap() helper on restarted nodes, 6 consecutive 60s intervals: sf_messages 48 rows/min = 0.80 rows/sec dead steady, retry-UPDATE rate 0/sec (SUM(retry_count) flat at 200), oplog 0, native_alarm_state 0, max payload_json 76 B, max config_json 721 B, ZERO SQLite errors across 30 min, both site-a nodes converged at 3564 rows. Honest gap: 0.80/s is ~1.6% of the 50/s ceiling and the retry-UPDATE path was never exercised - acceptable because that ceiling is structural (SweepBatchLimit/RetryTimerInterval), not empirical. Rig config rows (721 B) are NOT representative; D6 sizing rests on the documented ~60-70 KB production config_json. Plan-premise corrections stand: D6 (MaxBatchSize 500 -> 16, the one firmly evidence-backed number; Task 19 sets it), D4 (alarm writes bounded by per-SourceReference coalescing at a 100 ms flush, NOT unbounded), sf_messages hard ceiling 50 rows/sec, oplog cap overrun = graceful snapshot resync (needs_snapshot), not data loss. NEXT: Wave 1 = Tasks 3 + 4, dispatchable in parallel (disjoint Files blocks). Tasks 3-21 untouched; no plan code written yet. Rig cleanup still owed before Task 20: restore ExternalSystemDefinitions id 1 to http://scadabridge-restapi:5200 (currently http://127.0.0.1:9), and remove SoakGenerator template 2021 + instances soakgen-1..4 (ids 5-8), still deployed and generating load."
|
||||
}
|
||||
@@ -0,0 +1,218 @@
|
||||
# LocalDb Phase 2 — Gate Document
|
||||
|
||||
> **Status: CLOSED (2026-07-20).** The implementation plan is
|
||||
> [`2026-07-19-localdb-adoption-phase2.md`](2026-07-19-localdb-adoption-phase2.md); its
|
||||
> decisions D1–D6 answer this gate and its Task 1 supplied the measurements §5 demanded
|
||||
> (soak record: [`2026-07-19-localdb-phase2-soak.md`](2026-07-19-localdb-phase2-soak.md)).
|
||||
> Each §5 question is answered inline below. **The questions are deliberately not deleted —
|
||||
> the reasoning is the value, and several of the answers overturn a premise stated above.**
|
||||
|
||||
**Phase 1 live gate: PASS** (2026-07-19, docker rig). See the Task 12 commit for evidence.
|
||||
|
||||
**Phase 2 goal (from the design doc, scadaproj `docs/plans/2026-07-19-scadabridge-localdb-design.md` §1):**
|
||||
move `scadabridge.db`'s config tables and StoreAndForward's `sf_messages` into the same
|
||||
consolidated LocalDb file, then **delete** `SiteReplicationActor` and StoreAndForward's
|
||||
`ReplicationService` outright. CDC triggers replace hand-shipped ops; the library's
|
||||
snapshot resync replaces the bespoke chunked anti-entropy. The design explicitly chose
|
||||
**replace + delete, no dual-mechanism period** — recoverable from git.
|
||||
|
||||
---
|
||||
|
||||
## 1. What Phase 1 actually established
|
||||
|
||||
Facts a Phase 2 plan can rely on, all verified rather than assumed:
|
||||
|
||||
- One `ILocalDb` per site process at `LocalDb:Path`, **required** (`ValidateOnStart`).
|
||||
A site config missing it fails to boot.
|
||||
- `SiteLocalDbSetup.OnReady` is the single place tables are created and registered.
|
||||
Ordering is load-bearing: **DDL → `RegisterReplicated` → writes**, then the legacy
|
||||
migrator. Rows written before registration are invisible to the peer forever, silently.
|
||||
- Replication is registered in `SiteServiceRegistration` (not `Program.cs`) so the
|
||||
composition-root tests cover it; only `MapZbLocalDbSync` lives in `Program.cs`.
|
||||
- Inbound auth is `LocalDbSyncAuthInterceptor`, fail-closed, scoped to
|
||||
`/localdb_sync.v1.LocalDbSync/`, sharing the existing 8083 h2c listener.
|
||||
- Default-OFF is real and proven side-by-side: rig site-a replicates, site-b/site-c do not.
|
||||
- `ZbTelemetryOptions.Meters` is a **silent allowlist** (`SiteServiceRegistration.ObservedMeters`).
|
||||
- Bidirectional sync works with only one node configured as initiator.
|
||||
|
||||
## 2. What Phase 2 must port before deleting anything
|
||||
|
||||
The bespoke replicator's tests encode behaviour that the design's CDC replacement must
|
||||
still satisfy. **Port the test intents first, delete second** — the plan should treat the
|
||||
existing tests as the specification, not as code to be removed alongside the implementation.
|
||||
|
||||
- `tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/SiteReplicationActorTests.cs`
|
||||
- `tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ReplicationServiceTests.cs`
|
||||
|
||||
Implementation under deletion:
|
||||
|
||||
- `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/SiteReplicationActor.cs`
|
||||
- `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/ReplicationService.cs`
|
||||
|
||||
Known accepted behaviour to carry forward, not silently drop: the **N5 bounded-duplicate
|
||||
race** the bespoke replicator documents and accepts. A Phase 2 plan must state whether CDC
|
||||
inherits the same bound, a tighter one, or a different failure shape.
|
||||
|
||||
> **Routed (2026-07-20).** The failure shape changes, so N5's bound does not simply carry
|
||||
> over: CDC has no chunked anti-entropy hop to duplicate across, but LWW admits re-delivery
|
||||
> of a row whose park loses to a later write (see the LWW answer in §5). **Task 21 states the
|
||||
> new duplicate-delivery bound explicitly** and rewrites the N5 note rather than deleting it.
|
||||
|
||||
## 3. Substantially harder than Phase 1
|
||||
|
||||
Phase 1 moved two tables that **nothing replicated before**, so the worst case was "no
|
||||
worse than today". Phase 2 replaces a working mechanism, which makes it a different risk
|
||||
class:
|
||||
|
||||
- **`sf_messages` is an outbound buffer with side effects.** Phase 1's tables are
|
||||
read-mostly records; a store-and-forward row that resurrects or double-delivers sends
|
||||
real traffic to an external system. Park/requeue/remove becoming ordinary captured
|
||||
row updates/deletes needs its own analysis of what LWW does to an in-flight send.
|
||||
- **Config tables drive deployment.** A converged-but-wrong config row is a deployed
|
||||
instance behaving incorrectly, not a missing history entry.
|
||||
- **No dual-mechanism period.** The chosen posture means the cutover is the test. That
|
||||
raises the bar on the offline convergence suite and the live gate considerably.
|
||||
- **Migration is not a no-op this time.** Phase 1's migrator usually finds nothing,
|
||||
because the legacy files sat outside the volume. `scadabridge.db` and
|
||||
`store-and-forward.db` are real, populated, on the volume, and actively replicated
|
||||
while the migration runs.
|
||||
|
||||
## 4. Required inputs before writing the plan
|
||||
|
||||
1. The design doc §1 Phase 2 and §2 schema/migration sections, re-read against
|
||||
post-Phase-1 reality (several Phase 1 assumptions changed during execution — see the
|
||||
`amendments` array in `2026-07-19-localdb-adoption-phase1.md.tasks.json`).
|
||||
2. The two test files in §2, read as specifications.
|
||||
3. **Rig soak observations that do not exist yet** — see §5.
|
||||
|
||||
## 5. Open questions — answer these first
|
||||
|
||||
- **Oplog growth and churn under real config/S&F write rates.** Phase 1's tables are
|
||||
low-volume; `sf_messages` is not. `MaxOplogRows` / `MaxOplogAge` / snapshot-resync
|
||||
thresholds cannot be chosen from first principles. *This needs a Phase-1 rig soak that
|
||||
has not been run.* The live gate proved correctness, not steady-state behaviour over
|
||||
time.
|
||||
|
||||
**ANSWERED — the soak ran (Task 1); the caps are not the binding constraint, batch
|
||||
*bytes* are.** Two of this question's own premises turned out to be wrong, and both
|
||||
corrections relax it:
|
||||
|
||||
1. **The stop condition was much weaker than written.** Exceeding `MaxOplogRows` /
|
||||
`MaxOplogAge` does not wedge or lose data — `OplogStore` prunes and flags the peer
|
||||
`needs_snapshot` (`OplogStore.cs:109-138`, `MaintenanceBackgroundService.cs:57`), which
|
||||
degrades to a **graceful snapshot resync**. Overrun is a performance event, not a
|
||||
correctness event, so these caps do not need to be sized defensively.
|
||||
2. **`sf_messages` has a hard ceiling, not an estimate.** `SweepBatchLimit` (500) ÷
|
||||
`RetryTimerInterval` (10 s) = **≤50 row-writes/sec**, structurally. Measured rig rate
|
||||
under the purpose-built `SoakGenerator` load was far below that (see the soak record).
|
||||
Rows are small — max `payload_json` **76 bytes** on the rig.
|
||||
|
||||
`native_alarm_state` is answered under the last question in this section, not here (and
|
||||
the plan's D4, whose premise it corrects). The **real** ceiling this
|
||||
question was groping toward is D6's 4 MB gRPC message cap, and it binds on
|
||||
`MaxBatchSize × max-row-bytes`, not on oplog depth — **Task 19 sets
|
||||
`LocalDb:Replication:MaxBatchSize = 16`**, the one firmly evidence-backed number the soak
|
||||
produced. The keyed-instances escape hatch is **not** needed; this plan proceeds.
|
||||
|
||||
Independently, the soak retired a scare: a `disk I/O error` storm initially read as a
|
||||
Phase 1 library defect and STOP-gated this plan was root-caused as **observer-induced**
|
||||
(host-side `sqlite3` against live bind-mounted WAL files) — see
|
||||
[`docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md`](../known-issues/2026-07-20-localdb-disk-io-error-under-load.md)
|
||||
§0. The unmodified library sustains the full soak load with **zero** SQLite errors.
|
||||
|
||||
- **LWW semantics for in-flight store-and-forward sends.** What does last-writer-wins do
|
||||
when one node parks a message the other is mid-delivery on?
|
||||
|
||||
**ANSWERED — the scenario cannot arise on a correctly-behaving pair, and D2 covers what
|
||||
happens if it does.** Only the **active** node sweeps and delivers; the standby holds a
|
||||
convergent copy and sends nothing. So "one node mid-delivery while the other parks" is
|
||||
a split-brain symptom, not steady-state behaviour, and store-and-forward is not where it
|
||||
should be defended against.
|
||||
|
||||
If it does happen, LWW resolves per row by HLC: the later write wins, and the losing
|
||||
node's view converges to it. The concrete risk is **at-least-once delivery** — a park
|
||||
that loses to a stale in-flight update can be re-swept and re-sent. That is not new;
|
||||
store-and-forward is already at-least-once by construction (a send that succeeds but
|
||||
whose ack is lost is retried). What **is** new is D2's semantic change, recorded here
|
||||
because it is the honest cost of this gate: the standby is no longer guaranteed
|
||||
**byte-identical** to the active buffer, only **convergent**. The bespoke replicator's
|
||||
`ReplaceAllAsync` bought identity by wiping — and the N1 directional guard existed
|
||||
precisely because that wipe was dangerous. The library's snapshot resync **merges per
|
||||
row and never wipes** (`SnapshotApplier`, `LwwApplier.cs:69-78`), so the failure the
|
||||
guard prevented is structurally impossible rather than merely tested against. Task 10
|
||||
ports `SfBufferResyncPredicateTests` as a **convergence** assertion, not a
|
||||
directional-authority one.
|
||||
|
||||
- **Migration-under-load.** How does the one-time copy of an actively-replicating
|
||||
`scadabridge.db` interact with the bespoke replicator still running during cutover?
|
||||
|
||||
**ANSWERED — the interaction is designed out, not managed.** The two mechanisms never run
|
||||
concurrently. `SiteReplicationActor` and StoreAndForward's `ReplicationService` are
|
||||
**deleted in the same commit** that registers the Phase 2 tables (Tasks 14/15), and D5
|
||||
forecloses rolling upgrades: **both nodes of a site stop and start together**. A process
|
||||
that boots with the new code has no bespoke replicator to race, and one running the old
|
||||
code has no CDC triggers. There is no window in which a row is written by one mechanism
|
||||
and read by the other.
|
||||
|
||||
Within a single booting process the ordering is the load-bearing part, and it is the same
|
||||
invariant Phase 1 established: **DDL → `RegisterReplicated` → migrate → writes**
|
||||
(`SiteLocalDbSetup.OnReady`). Rows written *before* registration are invisible to the peer
|
||||
forever, silently — which is why Tasks 8/9 run the migrator strictly **after**
|
||||
registration, so every migrated row is captured by the CDC triggers and replicates
|
||||
normally. Both nodes migrating independently is fine: they migrate the same source rows,
|
||||
and LWW converges them.
|
||||
|
||||
- **Rollback story.** With no dual-mechanism period, what is the recovery path if the
|
||||
cutover fails in production — beyond "revert the commit"?
|
||||
|
||||
**ANSWERED — "revert the commit" is genuinely the path, and it is safe because the
|
||||
migration is additive and the legacy files are left intact.** Tasks 8/9 **copy** rows out
|
||||
of `scadabridge.db` and `store-and-forward.db` into the consolidated LocalDb file; they do
|
||||
not drop, truncate, or delete the source databases. Rolling back is therefore: stop both
|
||||
nodes of the site, deploy the previous build, start both together (D5). The old code
|
||||
reopens the legacy files and finds them exactly as it left them.
|
||||
|
||||
The bounded, honest cost of a rollback is **the delta** — rows written into the
|
||||
consolidated DB after cutover do not flow back into the legacy files. For config tables
|
||||
that self-heals: `SiteReconciliationActor` reports local inventory to central at startup
|
||||
and fetches whatever it lacks, so a rolled-back node re-converges to central's truth
|
||||
without operator action. For `sf_messages` the delta is **lost undelivered buffer** — the
|
||||
practical mitigation is to drain the buffer before cutting over, which Task 20's live gate
|
||||
and Task 21's runbook both call for.
|
||||
|
||||
What has **no** rollback is a site pair split across versions — hence D5. That is a
|
||||
deployment-procedure constraint, and Task 21 puts it in the runbook rather than leaving it
|
||||
as tribal knowledge.
|
||||
|
||||
- **Does the consolidated file stay appropriate?** One-DB-per-process was chosen partly
|
||||
because Phase 1's tables were small. Adding config + S&F changes the size and write
|
||||
profile of that single file.
|
||||
|
||||
**ANSWERED — yes, and the soak is the evidence.** The write profile the consolidated file
|
||||
must absorb is bounded on every axis:
|
||||
|
||||
| Table | Rate bound | Row size | Basis |
|
||||
|---|---|---|---|
|
||||
| `sf_messages` | **≤50 writes/sec** (hard) | 76 B max on the rig | `SweepBatchLimit` ÷ `RetryTimerInterval` |
|
||||
| `native_alarm_state` | `distinct_source_refs × 10/sec` | small | per-`SourceReference` coalescing on a 100 ms flush (`NativeAlarmActor.cs:473-502`) |
|
||||
| config tables | deploy-driven, effectively idle | ~721 B max on the rig | Task 1 measurement |
|
||||
|
||||
These are unremarkable for SQLite in WAL mode, and the soak ran the full generator load
|
||||
against the Phase 1 consolidated file for 30 minutes with **zero** SQLite errors and no
|
||||
oplog growth pathology. **One DB per process stays.**
|
||||
|
||||
Two caveats carried into the plan rather than hidden here. First, the rig's config rows
|
||||
are tiny (max 721 B) and **cannot** be treated as representative — the size ceiling that
|
||||
matters comes from the documented ~60–70 KB production `config_json`, which is what
|
||||
motivates D6 and `MaxBatchSize = 16`. Second, **D4's premise was wrong**:
|
||||
`native_alarm_state` is *not* "unbounded by design" and *not* "by a wide margin the
|
||||
highest-volume table" — the coalescing flush bounds it, and this rig has no alarm
|
||||
generator at all (measured 0 rows), so its bound is analytic rather than observed. If a
|
||||
production site ever shows alarm churn that swamps the shared oplog, the keyed-instances
|
||||
hatch named in D4 remains the escape — it is simply not needed to start.
|
||||
|
||||
## 6. Not in scope (unchanged from the design)
|
||||
|
||||
`auditlog.db` (diverges per node by design — central pulls the union; replicating it would
|
||||
double-forward), the secrets store (has its own answer in `Secrets.Replicator.SqlServer`
|
||||
hub mode), and central nodes (SQL-Server-first, no LocalDb use case).
|
||||
@@ -0,0 +1,209 @@
|
||||
# LocalDb Phase 2 — live gate evidence (2026-07-20)
|
||||
|
||||
Gate for [Task 20](2026-07-19-localdb-adoption-phase2.md). **Status: all 10 checks captured and
|
||||
passing**, with two scope caveats recorded below (check 2's count is vacuous on its own; check 7's
|
||||
third table was empty).
|
||||
|
||||
Build under test: `feat/localdb-phase2` @ `166f07fa`, LocalDb `0.1.1` — confirmed live, not
|
||||
assumed: the metrics scrape reports `otel_scope_name="ZB.MOM.WW.LocalDb.Replication"
|
||||
otel_scope_version="0.1.1"`. Rig redeployed with `docker/deploy.sh`, exit 0, all 8 nodes up.
|
||||
|
||||
## Method notes — two corrections to the plan
|
||||
|
||||
**1. Do NOT run DB checks host-side against the bind mounts, as Task 20 instructs.** macOS↔container
|
||||
locks do not cross virtiofs, so a host `sqlite3` open recovers the WAL out from under the container
|
||||
and triggers a permanent `disk I/O error` (SQLITE_IOERR_SHORT_READ 522) storm — the root cause of
|
||||
the 2026-07-20 incident. Every read below used a helper that copies the `db`/`-wal`/`-shm` triplet
|
||||
and queries the **copy**; `cp` is a pure byte read with no SQLite involvement.
|
||||
|
||||
**2. `docker exec … curl` cannot scrape the metrics** — the `aspnet:10.0` image has no `curl`, and
|
||||
with `2>/dev/null` the failure is silent and looks exactly like "no metrics exported". An earlier
|
||||
pass nearly recorded that as a finding. Use a network-sharing sidecar:
|
||||
|
||||
```bash
|
||||
docker run --rm --network container:scadabridge-site-a-a curlimages/curl:latest -s localhost:8084/metrics
|
||||
```
|
||||
|
||||
**3. The probe harness must be delivered by `deploy artifacts`, not `instance deploy`.** External
|
||||
systems reach a site only through `ArtifactDeploymentService` (`FetchGlobalArtifactsAsync` →
|
||||
`GetAllExternalSystemsAsync`), which the per-instance deploy path does not invoke. A script that
|
||||
names an external system that never arrived fails silently — no rows, no error.
|
||||
|
||||
---
|
||||
|
||||
## 1. Migration ran — PASS
|
||||
|
||||
| Node | `.migrated` markers | legacy `sf_messages` | consolidated `sf_messages` |
|
||||
|---|---|---|---|
|
||||
| site-a-node-a | `scadabridge.db.migrated`, `store-and-forward.db.migrated` | 11804 | 11804 |
|
||||
| site-a-node-b | both | 11804 | 11804 |
|
||||
|
||||
Both nodes migrated their own legacy files independently and the identical-content rows then
|
||||
LWW-converged — expected per the plan, not an anomaly.
|
||||
|
||||
**Caveat:** the migration itself ran on an earlier deploy of this build (markers dated 05:04), so
|
||||
this is after-the-fact evidence. `deployed_configurations` was 0 in both legacy files, so that
|
||||
table's migration path is untested here; check 3 exercises it fresh instead.
|
||||
|
||||
## 2. No SMTP/notification rows migrated — PASS
|
||||
|
||||
`smtp_configurations` and `notification_lists` are 0 on both nodes.
|
||||
|
||||
**That count alone is vacuous** — the legacy source also held 0 SMTP rows, so a broken exclusion
|
||||
would look identical. The non-vacuous evidence is structural: CDC capture triggers exist for
|
||||
exactly these ten tables on both nodes —
|
||||
|
||||
```
|
||||
OperationTracking, data_connection_definitions, database_connections,
|
||||
deployed_configurations, external_systems, native_alarm_state,
|
||||
sf_messages, shared_scripts, site_events, static_attribute_overrides
|
||||
```
|
||||
|
||||
— and `smtp_configurations` / `notification_lists` have **no triggers at all**, so no replication
|
||||
channel exists for them regardless of content. That is the property the security decision rests on.
|
||||
|
||||
## 3. Config converges byte-identical — PASS
|
||||
|
||||
Deployed `gate-sensor-1` (template 3) to site-a. Both nodes:
|
||||
|
||||
- whole-row md5 identical: `0ec0f3cf790bfd65c45edf6c4afae50f`
|
||||
- `deployment_id` `20b81607…`, `revision_hash` `sha256:253e1a56…`,
|
||||
`deployed_at` `2026-07-20T09:06:11.2087221+00:00`
|
||||
- `__localdb_row_version`: HLC `116951506695487488`, origin node
|
||||
`cf61c688-a57f-4c5c-ba76-feff6e5679fe`, `is_tombstone` 0 — **identical on both**
|
||||
|
||||
Identical HLC *and* origin id is the strong form: node B holds the row A wrote, not an
|
||||
independently-derived copy of it.
|
||||
|
||||
## 4. The standby made no config fetch — PASS
|
||||
|
||||
Deploy window from `2026-07-20T09:05:52Z`:
|
||||
|
||||
- `site-a-a` (active, deploy target): **1** fetch — `Fetching config for deployment 20b81607…
|
||||
(notify-and-fetch)`. Legitimate: the active node pulling its own artifact.
|
||||
- `site-a-b` (standby): **0** fetches.
|
||||
|
||||
Under the old architecture `SiteReplicationActor` would have notified the standby and it would have
|
||||
fetched independently. It now receives the config purely by CDC — corroborated from the other
|
||||
direction by check 3's shared origin id.
|
||||
|
||||
## 5. Store-and-forward converges — PASS
|
||||
|
||||
Harness: external system `GateDeadTarget` → `http://127.0.0.1:9` with method `Ping`, plus an
|
||||
interval script calling `ExternalSystem.CachedCall` (only `CachedCall` buffers; plain
|
||||
`ExternalSystem.Call` does not). Delivered via `deploy artifacts`.
|
||||
|
||||
- Buffered rows appear on **both** nodes with an identical rowset md5 `a484e369f6e0f00e7debff9b64464566`.
|
||||
- `__localdb_row_version` for `sf_messages`: 2509 rows from **1 origin** on both nodes.
|
||||
- The Pending→Parked transition of the pre-existing backlog also replicated identically
|
||||
(10280 Pending / 1524 Parked on both), so **UPDATE** replicates, not just INSERT.
|
||||
|
||||
## 6. Site failover — PASS
|
||||
|
||||
Stopped the active node `site-a-a` at 09:18:37. Ten seconds later `site-a-b` logged
|
||||
`InstanceActor started for gate-churn-1` and `ScriptActor GateBufferProbe started` — the standby
|
||||
picked up the workload and kept buffering. While partitioned, node B's oplog rose to **4** unacked
|
||||
entries.
|
||||
|
||||
Restarted `site-a-a` at 09:18:57. After rejoin:
|
||||
|
||||
| | node-a | node-b |
|
||||
|---|---|---|
|
||||
| probe rows | 48 | 48 |
|
||||
| total `sf_messages` | 11852 | 11852 |
|
||||
| oplog | 0 | 0 |
|
||||
| dead letters | 0 | 0 |
|
||||
| duplicate ids | 0 | 0 |
|
||||
| rowset md5 | `54ded633c28134222bf034f3d0cec680` | `54ded633c28134222bf034f3d0cec680` |
|
||||
|
||||
Buffered messages survived the flip, the backlog drained on rejoin, and **zero duplicate ids** on
|
||||
either node is the exactly-once evidence.
|
||||
|
||||
## 7. Cascade delete — PASS
|
||||
|
||||
Seeded `gate-sensor-1` with a `static_attribute_overrides` row (via an `Instance.SetAttribute`
|
||||
script — the attribute-override deploy path writes into the config JSON, not this table), confirmed
|
||||
it replicated, then deleted the instance.
|
||||
|
||||
| | pre-delete | post-delete |
|
||||
|---|---|---|
|
||||
| `deployed_configurations` | 1 / 1 | 0 / 0 |
|
||||
| `static_attribute_overrides` | 1 / 1 | 0 / 0 |
|
||||
| `native_alarm_state` | 0 / 0 | 0 / 0 |
|
||||
|
||||
Crucially, **both nodes hold explicit tombstones** — `deployed_configurations=1`,
|
||||
`static_attribute_overrides=1` in `__localdb_row_version WHERE is_tombstone=1`. The rows are not
|
||||
merely absent on B; B applied the deletes.
|
||||
|
||||
**Caveat:** `native_alarm_state` was empty for this instance, so the third leg of the cascade is
|
||||
untested live. It is covered offline by
|
||||
`LocalDbPhase2ConvergenceTests.RemovingAnInstance_ConvergesAllThreeCascadeTables`, which was
|
||||
verified non-vacuous.
|
||||
|
||||
## 8. Dead letters, oplog, metrics — PASS
|
||||
|
||||
- `__localdb_oplog` = 0 and `__localdb_dead_letter` = 0 on both nodes at every sampling point.
|
||||
- `localdb_oplog_depth` 0; `localdb_sync_applied_total` 11808 (a) / 224 (b);
|
||||
`localdb_sync_reconnects_total` 1.
|
||||
|
||||
## 9. Both nodes stopped and started together (D5) — PASS
|
||||
|
||||
Stopped `site-a-a` and `site-a-b` together, started them together.
|
||||
|
||||
| | node-a | node-b |
|
||||
|---|---|---|
|
||||
| `sf_messages` | 11845 | 11845 |
|
||||
| probe rows | 41 | 41 |
|
||||
| oplog | 0 | 0 |
|
||||
| dead letters | 0 | 0 |
|
||||
|
||||
Zero occurrences of `disk I/O error` / `database disk image` / `SQLITE_IOERR` / `corrupt` in either
|
||||
node's log after restart. Clean rejoin.
|
||||
|
||||
## 10. Drain under churn — PASS
|
||||
|
||||
Sampled every 8 s under sustained probe load:
|
||||
|
||||
```
|
||||
t=8s oplog_a=0 oplog_b=0 probe_a=23 probe_b=23
|
||||
t=16s oplog_a=0 oplog_b=0 probe_a=25 probe_b=25
|
||||
t=24s oplog_a=0 oplog_b=0 probe_a=26 probe_b=26
|
||||
t=32s oplog_a=0 oplog_b=0 probe_a=28 probe_b=28
|
||||
t=40s oplog_a=0 oplog_b=0 probe_a=30 probe_b=30
|
||||
t=48s oplog_a=0 oplog_b=0 probe_a=31 probe_b=31
|
||||
t=56s oplog_a=0 oplog_b=0 probe_a=33 probe_b=33
|
||||
t=64s oplog_a=0 oplog_b=0 probe_a=35 probe_b=35
|
||||
```
|
||||
|
||||
Writes climb steadily and the two nodes stay in lockstep at **every** sample, with no oplog
|
||||
accumulation — flush plus ack outpaces the write rate, so no backlog forms. The stop condition the
|
||||
plan cared about (monotonic growth that never drains) does not occur.
|
||||
|
||||
**On its own this sampling is weak** — an 8 s interval is far coarser than the flush interval, so a
|
||||
transient non-zero depth would be missed, and "always 0" is also what a dead pump looks like. The
|
||||
rise-and-drain is proven instead by check 6, where the oplog demonstrably went 0 → 4 while the peer
|
||||
was down and back to 0 after rejoin.
|
||||
|
||||
---
|
||||
|
||||
## Observations outside the gate
|
||||
|
||||
- **A central external-system delete does not remove the row from sites.** After deleting
|
||||
`GateDeadTarget` centrally and re-running `deploy artifacts`, both site nodes still carry it.
|
||||
Artifact application is an upsert with no reconciliation of removals. Pre-existing behaviour in
|
||||
the artifact pipeline, unrelated to LocalDb — but it means site config tables accumulate orphans.
|
||||
- **Deleting an instance orphans its buffered messages.** Removing the `soakgen-*` instances left
|
||||
their 11,804 `sf_messages` with no tracking snapshot, producing a continuous
|
||||
`Cached-telemetry drain: no tracking snapshot for …` warning flood. Also pre-existing, and worth
|
||||
a cleanup path.
|
||||
|
||||
## Rig state as left
|
||||
|
||||
- Owed cleanup **done**: `SoakGenerator` (2021) and `soakgen-1..4` deleted;
|
||||
`ExternalSystemDefinitions` id 1 restored to `http://scadabridge-restapi:5200` **and confirmed
|
||||
propagated to both site nodes**.
|
||||
- Gate harness removed: `gate-sensor-1`, `gate-churn-1`, template-3 scripts `GateBufferProbe` /
|
||||
`GateSetStatic`, and the `GateDeadTarget` external system are deleted centrally. The site-side
|
||||
`external_systems` row for `GateDeadTarget` remains, per the observation above.
|
||||
- Remaining instances: `soak-motor-1..4` (template 4, not deployable) — untouched, as found.
|
||||
- The ~11.8k orphaned `sf_messages` remain on both site-a nodes.
|
||||
@@ -0,0 +1,348 @@
|
||||
# LocalDb Phase 2 — rig soak findings
|
||||
|
||||
**Run date:** 2026-07-19 / 2026-07-20 (UTC) · **Rig:** local 8-node docker cluster, site-a pair
|
||||
**Task:** Task 1 of [`2026-07-19-localdb-adoption-phase2.md`](2026-07-19-localdb-adoption-phase2.md)
|
||||
|
||||
> ## GATE VERDICT: **STOP — do not proceed to Task 3.** *(superseded — see update)*
|
||||
>
|
||||
> Not for the reason the plan anticipated. Oplog sizing is fine and D6 is resolved. The soak
|
||||
> instead surfaced what looked like a **pre-existing Phase 1 defect**: the consolidated LocalDb
|
||||
> database (`site-localdb.db`) throws `SQLite Error 10: 'disk I/O error'` on essentially every
|
||||
> write on the **active** node under sustained concurrent load. See
|
||||
> [Finding 1](#finding-1-blocker).
|
||||
>
|
||||
> **Update 2026-07-20: Finding 1 is root-caused and is NOT a product defect.** The soak's own
|
||||
> host-side `sqlite3` sampling poisoned both nodes (WAL reset across the virtiofs boundary);
|
||||
> the unmodified code sustains the full soak load indefinitely with zero errors —
|
||||
> reproduced/refuted on demand, see
|
||||
> [`docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md`](../known-issues/2026-07-20-localdb-disk-io-error-under-load.md) §0.
|
||||
> Finding 1 therefore no longer blocks Task 3. What still stands before proceeding: re-run the
|
||||
> cut-short sampling (Findings 4/5 write-rate numbers) using the safe copy-based `snap` recipe
|
||||
> now in the plan, on a rig where both site-a nodes have been restarted since any host-side read.
|
||||
>
|
||||
> ## GATE VERDICT (final, 2026-07-20): **PROCEED to Task 3.**
|
||||
>
|
||||
> The re-run is done — [§1b](#1b-clean-re-run-2026-07-20-post-root-cause). Safe copy-based
|
||||
> sampling, both nodes restarted first, six clean 60-second intervals: **0.80 `sf_messages`
|
||||
> rows/sec sustained, zero SQLite errors across the full window, both nodes converged.** No stop
|
||||
> condition from any of D1–D6 is met. The one binding number Phase 2 must honour is
|
||||
> **`LocalDb:Replication:MaxBatchSize = 16`** (Finding 2 / D6), which Task 19 sets.
|
||||
|
||||
---
|
||||
|
||||
## 1. Method as actually executed
|
||||
|
||||
The plan's method needed four corrections before it would run. Recorded here so the next run
|
||||
does not rediscover them.
|
||||
|
||||
| Plan said | Reality |
|
||||
|---|---|
|
||||
| `docker exec … curl -s localhost:8084/metrics` | **No `curl` in the `aspnet:10.0` image.** Use a sidecar sharing the container netns: `docker run --rm --network container:scadabridge-site-a-a curlimages/curl:latest -s localhost:8084/metrics` |
|
||||
| Sample with host-side `sqlite3` against the bind mounts | **This IS the cause of Finding 1** (root-caused 2026-07-20 — the "ruled out" verdict below did not survive; see the known-issue doc §0). Host↔container POSIX locks don't propagate over virtiofs; a host-side read checkpoints + resets the WAL under the container and permanently poisons its connections. Copy the file triplet and query the copy, or read counters from `/metrics`. |
|
||||
| Drive alarm churn on a deployed instance | **Not possible on this rig.** `infra/mssql/seed-config.sql` seeds zero `TemplateNativeAlarmSources`, opc-plc runs with no alarm flags, and no simulator or harness exists anywhere in the repo. `native_alarm_state` stayed at 0 rows throughout. Bounded analytically instead — see [Finding 3](#finding-3). |
|
||||
| Drive S&F churn via template 4's `TestExternalSystem` script | Template 4 exists after a reseed but is **not deployable** (34 pre-deployment validation errors: 30 `ConnectionBinding` + 4 `ScriptCompilation`). A purpose-built `SoakGenerator` template was used instead — see below. |
|
||||
|
||||
### The generator that worked
|
||||
|
||||
`ExternalSystem.Call` does **not** buffer to store-and-forward in practice; `ExternalSystem.CachedCall`
|
||||
is the buffering surface. This is the single most important operational detail for reproducing
|
||||
S&F load.
|
||||
|
||||
- Template `SoakGenerator` (id 2021), one `Interval` script at `{"intervalMs":5000}`:
|
||||
```csharp
|
||||
var parms = new Dictionary<string, object?> { ["a"] = 2, ["b"] = 3 };
|
||||
await ExternalSystem.CachedCall("Test REST API", "Add", parms);
|
||||
```
|
||||
- No attributes, no compositions, no connection bindings — deliberately, so it deploys cleanly.
|
||||
- `ExternalSystemDefinitions` id 1 repointed to `http://127.0.0.1:9` (discard port → connection
|
||||
refused → classified transient → buffered).
|
||||
- 4 instances (`soakgen-1..4`) deployed to site-a.
|
||||
|
||||
Sustained rate observed: ~**2.9 HTTP attempts/sec** (688–864 connection-refused per 4 min).
|
||||
|
||||
### Two rig-tooling bugs found and fixed en route
|
||||
|
||||
1. **`docker/seed-sites.sh` seeded stale role names** — `Design`/`Deployment` instead of the
|
||||
canonical `Designer`/`Deployer` (`src/ZB.MOM.WW.ScadaBridge.Security/Roles.cs:46-47`). Every
|
||||
Designer/Deployer-gated management command failed `UNAUTHORIZED` on a freshly reseeded rig,
|
||||
including `seed-sites.sh`'s own trailing `deploy artifacts` and `reseed.sh` stage 6d.
|
||||
**Fixed** (commit `cf46e596`).
|
||||
2. **`infra/mssql/setup.sql` never executes.** It is mounted into
|
||||
`/docker-entrypoint-initdb.d/`, a convention the official `mcr.microsoft.com/mssql/server`
|
||||
image does not implement. After `reseed.sh` drops the volume (`docker compose down -v`),
|
||||
nothing recreates `ScadaBridgeConfig` or the `scadabridge_app` login, so `reseed.sh` hangs
|
||||
forever on its "Waiting for setup.sql to create ScadaBridgeConfig" poll. Worked around by
|
||||
applying the three init scripts by hand. **NOT yet fixed in the repo.**
|
||||
|
||||
---
|
||||
|
||||
## Finding 1 (BLOCKER) — *root-caused 2026-07-20: observer-induced, not a product defect; see the gate-verdict update and the known-issue doc §0*
|
||||
|
||||
### The Phase 1 consolidated LocalDb fails under sustained write load on the active node
|
||||
|
||||
`site-localdb.db` throws `SQLite Error 10: 'disk I/O error'` on essentially every write once the
|
||||
active node is under concurrent load. Both Phase 1 tables and the audit telemetry paths are
|
||||
affected.
|
||||
|
||||
Representative stacks (`docker logs scadabridge-site-a-b`):
|
||||
|
||||
```
|
||||
Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 10: 'disk I/O error'.
|
||||
at ZB.MOM.WW.ScadaBridge.SiteEventLogging.SiteEventLogger.ProcessWriteQueueAsync()
|
||||
SiteEventLogger.cs:line 221/236
|
||||
|
||||
Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 10: 'disk I/O error'.
|
||||
at ZB.MOM.WW.ScadaBridge.SiteRuntime.Tracking.OperationTrackingStore.RecordEnqueueAsync(...)
|
||||
OperationTrackingStore.cs:line 137
|
||||
at ...CachedCallTelemetryForwarder.TryEmitTrackingAsync(...) line 148
|
||||
```
|
||||
|
||||
User-visible symptom: `[ERR] Failed to record event: script from ScriptActor:SoakCall` — **site
|
||||
event logging is silently dropping events on the floor under load.**
|
||||
|
||||
#### It follows the load, not the node, not the observer
|
||||
|
||||
The failure was isolated by moving the load between nodes:
|
||||
|
||||
| Node | Role | Under load | `disk I/O error` in 4 min |
|
||||
|---|---|---|---|
|
||||
| site-a-a | active | yes | 2 175 |
|
||||
| site-a-a | standby (after restart) | no | **0** |
|
||||
| site-a-b | standby | no | **0** |
|
||||
| site-a-b | active (after failover) | yes | **4 391** |
|
||||
|
||||
#### It is LocalDb-specific, not the filesystem — *wrong: sampling-selection bias; only the LocalDb file was ever host-read*
|
||||
|
||||
The decisive control. Under identical load, on the same node, in the same bind-mounted
|
||||
directory, counting error-stack frames over 3 minutes:
|
||||
|
||||
| Store | Backing file | Errors |
|
||||
|---|---|---|
|
||||
| `OperationTrackingStore` | `site-localdb.db` (LocalDb) | 13 044 |
|
||||
| `SiteAuditTelemetryActor` | `site-localdb.db` (LocalDb) | 4 350 |
|
||||
| `SiteEventLogger` | `site-localdb.db` (LocalDb) | 900 |
|
||||
| `CachedCallTelemetryForwarder` | `site-localdb.db` (LocalDb) | 162 |
|
||||
| `StoreAndForwardStorage` | `store-and-forward.db` (legacy) | **0** |
|
||||
| `SiteStorageService` | `scadabridge.db` (legacy) | **0** |
|
||||
|
||||
Ordinary SQLite on the same bind mount is completely healthy. Only the LocalDb-managed
|
||||
database fails.
|
||||
|
||||
#### Ruling out the observer — **RETRACTED 2026-07-20: the observer was the cause**
|
||||
|
||||
Onset (04:56:37) was one second after a host-side `sqlite3` sample (04:56:36), which made
|
||||
observer-induced `-shm` corruption the leading hypothesis. The original run "excluded" it:
|
||||
after node-a was restarted and the load failed over to node-b, node-b began erroring while no
|
||||
host process touched its files, and sampled node-a went to zero once idle.
|
||||
|
||||
**That exclusion was wrong.** The 04:56 sampling had poisoned *both* nodes' files (both carry
|
||||
the 04:56 main-DB mtime; node-b's WAL was left at 0 bytes) — node-b was silent only because a
|
||||
standby issues ~no LocalDb statements, and erupted on its first post-failover write. Verified
|
||||
2026-07-20: 10+ min of full soak load on a freshly-reopened node with zero errors, then a
|
||||
single host `sqlite3` read reset its 4.6 MiB WAL to 0 bytes and started the error storm one
|
||||
second later (`SQLITE_IOERR_SHORT_READ`, 522). Full mechanism + minimal repro:
|
||||
[`docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md`](../known-issues/2026-07-20-localdb-disk-io-error-under-load.md) §0.
|
||||
|
||||
#### Secondary defect, same area
|
||||
|
||||
```
|
||||
[ERROR][akka://scadabridge/user/site-audit-telemetry] There is no active ActorContext,
|
||||
this is most likely due to use of async operations from within this actor.
|
||||
Cause: System.NotSupportedException
|
||||
```
|
||||
|
||||
`SiteAuditTelemetryActor` is closing over `ActorContext` across an `await`. Likely a
|
||||
contributing cause rather than a separate issue — it is in the same write path — but it is a
|
||||
real bug on its own terms.
|
||||
|
||||
#### Why this blocks Phase 2
|
||||
|
||||
Phase 2 registers **eight further tables** into this database, including `native_alarm_state`
|
||||
(the highest-volume table in either DB) and `sf_messages`. It also **deletes** the bespoke
|
||||
mechanisms (`SiteReplicationActor`, `ReplicationService`) that currently carry that data
|
||||
independently of LocalDb. Cutting over onto a store that cannot absorb the *current* write
|
||||
load — and doing so in the same commit that removes the fallback — would convert a logging
|
||||
defect into site-wide config and buffer loss.
|
||||
|
||||
~~This is a Phase 1 defect. It must be root-caused and fixed before Phase 2's Task 3.~~
|
||||
**Root-caused 2026-07-20: not a Phase 1 defect** — observer-induced WAL reset across the
|
||||
virtiofs bind-mount boundary; see the gate-verdict update at the top of this document.
|
||||
|
||||
---
|
||||
|
||||
## Finding 2 — D6 resolved: no stop condition, but `MaxBatchSize` must be lowered
|
||||
|
||||
The plan asserts `deployed_configurations.config_json` is "documented to exceed 128 KB per row."
|
||||
That misreads the source. `docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md`
|
||||
says the *escaped Akka envelope* exceeded the 128 KB frame, and that the default serializer
|
||||
double-escapes the payload, so **"the raw flattened JSON only needs to be ~60-70 KB to blow the
|
||||
128 KB frame."**
|
||||
|
||||
So the largest known real production `config_json` is on the order of **60–70 KB**.
|
||||
|
||||
Measured on the rig (4 deployed configs): **max 715 B, avg 714 B** — trivially small, as the plan
|
||||
predicted, hence the production figure above is the one to size against. (A direct measurement
|
||||
against wonder was attempted; `wonder-app-vd03.zmr.zimmer.com` resolves over the VPN but the
|
||||
servecli SSH service on :2222 refuses connections, so the box was unreachable.)
|
||||
|
||||
**Verdict: no stop condition.** Nothing approaches the 4 MB single-row ceiling.
|
||||
|
||||
**But the batching risk is real.** Batching is row-count-only (`MaxBatchSize` default **500**),
|
||||
and neither side configures gRPC message limits, so the 4 MB default receive cap applies:
|
||||
|
||||
```
|
||||
500 rows × 70 KB ≈ 35 MB ≫ 4 MB → poison batch, stream wedged
|
||||
```
|
||||
|
||||
Recommended for Task 19:
|
||||
|
||||
```
|
||||
LocalDb:Replication:MaxBatchSize = 16
|
||||
```
|
||||
|
||||
`16 × 128 KB = 2 MB` — 2× headroom on row size over the known worst case, and 2× headroom
|
||||
against the 4 MB cap.
|
||||
|
||||
---
|
||||
|
||||
## Finding 3 — D4 corrected: alarm write rate is bounded, not unbounded
|
||||
|
||||
The plan calls `native_alarm_state` "unbounded by design." It is not.
|
||||
`NativeAlarmActor.MarkDirtyUpsert` (`NativeAlarmActor.cs:473-502`) coalesces into a dictionary
|
||||
**keyed by `SourceReference`** and flushes on a timer (`_persistFlushInterval`, default
|
||||
**100 ms**). One flush writes at most one row per distinct source reference, regardless of how
|
||||
many transitions occurred in that window.
|
||||
|
||||
```
|
||||
worst-case rows/sec = distinct_source_refs × 10
|
||||
```
|
||||
|
||||
An alarm storm on N sources costs N rows per 100 ms flush, not N × transition-rate. This makes
|
||||
the table analytically sizeable without a generator — which matters, because this rig cannot
|
||||
produce alarm load at all (see §1).
|
||||
|
||||
**Not empirically validated.** `native_alarm_state` held 0 rows for the entire run.
|
||||
|
||||
---
|
||||
|
||||
## Finding 4 — `sf_messages` has a hard structural ceiling of 50 rows/sec
|
||||
|
||||
Confirmed by code rather than measurement, which is stronger here. The retry sweep takes at most
|
||||
`SweepBatchLimit` messages every `RetryTimerInterval`, and each failed attempt is one `UPDATE`
|
||||
incrementing `retry_count`:
|
||||
|
||||
```
|
||||
SweepBatchLimit (500) ÷ RetryTimerInterval (10 s) = 50 row-writes/sec, hard ceiling
|
||||
```
|
||||
|
||||
This confirms the plan's "~50 row-writes/sec worst case" — and it is a ceiling, not an estimate.
|
||||
`DefaultMaxRetries = 50` at `DefaultRetryInterval = 30 s` bounds each message to 50 updates over
|
||||
25 minutes.
|
||||
|
||||
Observed during the run: ~2.9 attempts/sec, far below the ceiling. Row counts could not be
|
||||
sampled reliably (see §1) and the run was cut short by Finding 1.
|
||||
|
||||
---
|
||||
|
||||
## Finding 5 — exceeding the oplog caps is a graceful degradation, not a failure
|
||||
|
||||
The plan's Task 1 step 7 treats "the shared oplog cannot absorb this write profile" as a hard
|
||||
stop requiring the keyed-instances escape hatch. It is much weaker than that.
|
||||
|
||||
`OplogStore.EnforceCapsAsync` (`OplogStore.cs:109-138`) prunes to the ceiling and sets
|
||||
`needs_snapshot`; `MaintenanceBackgroundService.cs:57` logs *"Oplog backlog/age caps exceeded:
|
||||
pruned to the ceiling and flagged needs_snapshot — the peer must snapshot-resync."*
|
||||
`SyncSession.ComputeSnapshotRequiredAsync` (`:413-415`) then forces a snapshot resync.
|
||||
|
||||
So overrunning the caps costs a **full snapshot resync**, not data loss and not a wedged stream.
|
||||
The caps therefore express *"how long may a peer be absent before it needs a full resync"*, and
|
||||
should be sized to the longest tolerable peer outage rather than treated as a correctness
|
||||
boundary. In healthy two-node operation the oplog drains continuously — `localdb_oplog_depth`
|
||||
read **0** throughout.
|
||||
|
||||
### Provisional cap recommendation (Task 19)
|
||||
|
||||
Sized for a ~3-hour peer outage at a conservative 100 rows/sec aggregate, pending re-measurement
|
||||
after Finding 1 is fixed:
|
||||
|
||||
```
|
||||
LocalDb:Replication:MaxOplogRows = 1000000 # default; ≈2.8 h at 100 rows/s
|
||||
LocalDb:Replication:MaxOplogAge = 3.00:00:00
|
||||
LocalDb:Replication:MaxBatchSize = 16 # Finding 2 — this one is NOT optional
|
||||
```
|
||||
|
||||
Only `MaxBatchSize` is firmly evidence-backed. The other two rest on an assumed aggregate write
|
||||
rate that this run could not measure.
|
||||
|
||||
---
|
||||
|
||||
## 1b. Clean re-run (2026-07-20, post-root-cause)
|
||||
|
||||
The original sampling was cut short by Finding 1 and was itself the cause of it. Re-run after
|
||||
both site-a nodes were restarted, using the plan's copy-based `snap()` helper — **no host-side
|
||||
`sqlite3` ever touched a live file.** Six consecutive 60-second intervals, generator load
|
||||
unchanged (`SoakGenerator` ×4 against a refusing endpoint):
|
||||
|
||||
| UTC | `sf_messages` rows | `SUM(retry_count)` | status=0 | `__localdb_oplog` | `native_alarm_state` |
|
||||
|---|---|---|---|---|---|
|
||||
| 06:07:57 | 3384 | 200 | 3380 | 0 | 0 |
|
||||
| 06:08:57 | 3432 | 200 | 3428 | 0 | 0 |
|
||||
| 06:09:57 | 3480 | 200 | 3476 | 0 | 0 |
|
||||
| 06:10:57 | 3528 | 200 | 3524 | 0 | 0 |
|
||||
| 06:11:57 | 3576 | 200 | 3572 | 0 | 0 |
|
||||
| 06:12:57 | 3624 | 200 | 3620 | 0 | 0 |
|
||||
| 06:13:57 | 3672 | 200 | 3668 | 0 | 0 |
|
||||
|
||||
**Measured:**
|
||||
|
||||
- **`sf_messages` insert rate: exactly 48 rows/min = 0.80 rows/sec**, dead steady across all six
|
||||
intervals (+288 rows over 360 s, zero variance).
|
||||
- **Retry-UPDATE rate: 0/sec.** `SUM(retry_count)` never moved. Only 4 rows ever reached
|
||||
`retry_count = 50` (status 2, dead-lettered); the other ~3.6 k sit at `retry_count = 0`,
|
||||
status 0 — **enqueued but never swept**. So this generator exercises the *insert* path only.
|
||||
- **Row sizes:** `sf_messages.payload_json` max **76 B**; `deployed_configurations.config_json`
|
||||
max **721 B** over 4 rows (rig config is trivially small — see the caveat below).
|
||||
- **`native_alarm_state`: 0 rows** — no alarm generator on this rig, as previously recorded.
|
||||
- **`__localdb_oplog`: 0 throughout** — the Phase 1 tables are genuinely idle under this load,
|
||||
which is the expected result and not a measurement failure (Phase 2's tables are not yet
|
||||
registered, so this load cannot reach the oplog by construction).
|
||||
- **Zero SQLite errors** in `docker logs scadabridge-site-a-a` across the full 30-minute window
|
||||
(`grep -ciE "disk I/O|SQLITE_IOERR|NOTADB"` → **0**). This is the direct confirmation that
|
||||
Finding 1 was observer-induced: identical load, identical library, nothing host-reading the
|
||||
files, no errors.
|
||||
- **Both site-a nodes converged identically** (3564 rows on each at a common sample point) under
|
||||
the bespoke replicator — the pre-cutover baseline Task 20 should reproduce under CDC.
|
||||
|
||||
**Honest limits of this measurement.** 0.80 rows/sec is **~1.6 % of the 50 rows/sec structural
|
||||
ceiling**, and the retry path — the expensive one, one `UPDATE` per message per sweep — was never
|
||||
exercised at all. This re-run therefore confirms *steady-state health and the absence of the
|
||||
Finding 1 pathology*; it does **not** probe the ceiling. That is acceptable because the ceiling is
|
||||
**structural rather than empirical** (Finding 4: `SweepBatchLimit` ÷ `RetryTimerInterval`), so
|
||||
sizing does not depend on observing it. Likewise the rig's 721 B config rows are **not**
|
||||
representative — D6's sizing rests on the documented ~60–70 KB production `config_json`, not on
|
||||
this number.
|
||||
|
||||
## 2. What still owes measurement
|
||||
|
||||
Carry forward (items 1–2 partially discharged by §1b above):
|
||||
|
||||
1. ~~Sustained `sf_messages` rows/sec~~ — **measured** (§1b: 0.80/s insert, 0/s retry). Still
|
||||
unmeasured: the **retry-UPDATE** path under a saturating generator approaching the 50/s
|
||||
ceiling. Not required for sizing (the ceiling is structural), but it is the honest gap.
|
||||
2. Any `native_alarm_state` measurement at all — requires building alarm-source seeding plus an
|
||||
A&C-capable server. The `OpcUaAlarmLiveSmokeTests` **passed**, so the rig's opc-plc *does*
|
||||
answer ConditionRefresh with a `SnapshotComplete`; the missing piece is ongoing transitions
|
||||
and a seeded `TemplateNativeAlarmSource`. (The test's own doc comment claiming the simulator
|
||||
"does not reliably expose A&C" is stale.)
|
||||
3. A production-representative `config_json` from wonder, to replace the ~60–70 KB inference.
|
||||
4. The empirical oplog drain-under-churn check — Task 20 evidence item 10.
|
||||
|
||||
## 3. Rig state left behind
|
||||
|
||||
- Fully reseeded (central config volume dropped and replayed; site SQLite state wiped by
|
||||
`reseed.sh` stage 2).
|
||||
- `ExternalSystemDefinitions` id 1 is **repointed to `http://127.0.0.1:9`** — restore to
|
||||
`http://scadabridge-restapi:5200` before using the rig for anything else.
|
||||
- Template `SoakGenerator` (2021) and instances `soakgen-1..4` (ids 5–8) remain deployed on
|
||||
site-a and are **still generating load**. Undeploy or delete them before the Task 20 live gate.
|
||||
- `LdapGroupMappings` corrected in the live DB to the canonical role names.
|
||||
@@ -80,7 +80,7 @@ There is **no maximum buffer size**. Messages accumulate in the buffer until del
|
||||
- The standby node applies the same operations to its own local SQLite database but is **passive**: it never runs the delivery sweep. The retry sweep is **gated to the active node** (the oldest Up member / singleton host, re-evaluated every sweep tick), so only one node delivers at a time. The standby applies replicated operations purely to keep its copy warm for a future failover.
|
||||
- On failover, the new active node has a near-complete copy of the buffer. In rare cases, the most recent operations may not have been replicated (e.g., a message added or removed just before failover). This can result in a few **duplicate deliveries** (message delivered but its `Remove` not yet replicated) or a few **missed retries** (message added but not replicated). Duplicate deliveries are therefore confined to the **failover window** — an in-flight delivery whose `Remove` had not yet replicated — and never occur in steady-state operation (the standby's gate keeps it from delivering the same rows). Both are acceptable trade-offs for the latency benefit.
|
||||
- On failover, the new active node's gate flips to active within one sweep interval and it resumes delivery from its local copy.
|
||||
- **Peer-join anti-entropy resync (chunked, ack-confirmed).** Asynchronous, no-ack replication keeps the standby warm in steady state, but a standby that was **down for an extended period** (a crash, a long maintenance window) misses every operation replicated while it was gone and would otherwise diverge from the active node's buffer forever. To close that gap, whenever a node **(re)tracks its peer**, a **standby** requests a full-buffer snapshot (`RequestSfBufferResync`); the **active** node loads up to `MaxResyncRows` (10 000) of its oldest rows and answers with a **sequence of byte-budgeted chunks** (`SfBufferSnapshotChunk`), each carrying a shared `ResyncId`, a 1-based `Sequence`, and the `TotalChunks` count. Chunking is mandatory because the monolithic snapshot exceeds Akka remoting's **default 128 000-byte frame** for any realistic backlog and `BuildHocon` sets no override — a single oversized message is silently undeliverable (review 02 round 2, **N2 High**). Rows accumulate into a chunk until the estimated payload budget (`MaxResyncChunkBytes` = 64 000, ≈50% frame headroom) or the row cap (`MaxResyncChunkRows` = 200) is hit; a single row whose payload alone exceeds the budget ships solo with a Warning. The standby **assembles all chunks of one `ResyncId`** (a new `ResyncId` discards any stale partial assembly; a partial that never completes is dropped after `resyncAssemblyTimeout`, default 30 s, and counted a replication failure), then **replaces its entire local buffer** with the assembled snapshot (`ReplaceAllAsync`, one transaction) and returns a delivery confirmation (`SfBufferResyncAck`). The active node arms an ack window (`resyncAckTimeout`, default 60 s): an acknowledged resync increments `scadabridge.store_and_forward.resync.completed`; an unacknowledged one (lost chunks / dead peer) logs a Warning and increments `scadabridge.store_and_forward.resync.ack_missing` — closing N2's silent-loss mode (previously nothing counted a lost snapshot and nothing retried until the next peer-track). Only the active node answers; only a standby applies, and both sides re-check at apply time (a mid-flight active-flip aborts the wipe) — each side checks the repo-standard **oldest-Up member** active-node predicate (singleton-host semantics via the shared `ActiveNodeEvaluator`, the **same predicate as the S&F delivery gate**; review 02 round 2, **N1 Critical** — using cluster *leadership* here let a rolling restart of the lower-address node make the delivering node wipe its own live buffer). Because replicated applies are **upserts** (see the replication apply path), any Add/Remove/Park that lands after the resync merges cleanly onto the resynced state — no primary-key conflict, no lost delta; the one accepted exception is the rare **N5** orphan-row race (a replicated `Remove` ordered before the snapshot chunks can re-add the removed row, re-delivered once and self-corrected at the next resync — inherent to no-ack replication). If the buffer exceeds the 10 000-row cap the snapshot is flagged `Truncated` and the standby logs a Warning; the residual divergence beyond the cap drains naturally as the active node delivers. The legacy monolithic `SfBufferSnapshot` message + standby handler are **retained** for rolling-upgrade compatibility (an old active node's monolithic snapshot is still applied by a new standby).
|
||||
- **Peer-join anti-entropy resync (LocalDb CDC).** *(Rewritten for LocalDb Phase 2, 2026-07-20. The previous specification of a chunked, ack-confirmed `SfBufferSnapshotChunk` protocol described the bespoke `ReplicationService`, which Phase 2 deleted. The discussion is rewritten rather than removed, because the failure modes it reasoned about still exist — they are simply bounded differently now.)* The buffer lives in the consolidated LocalDb database as the replicated `sf_messages` table, and both nodes exchange changes over a gRPC sync stream rather than Akka remoting. A node that was down for an extended period no longer requests a full-buffer snapshot and **replaces** its local buffer; LocalDb's snapshot resync merges **per row under last-writer-wins and never deletes**. Several of the old hazards are therefore structurally gone rather than guarded against: **(a) the 128 000-byte Akka frame limit no longer applies** — the transport is gRPC, whose successor ceiling is the 4 MB default receive limit, managed by bounding `LocalDb:Replication:MaxBatchSize` (set to 16 on the rig, sized against a ~70 KB worst-case `config_json`; see the Phase 2 plan, D6). Chunking, `ResyncId` assembly, assembly timeouts, and the truncation flag are all retired with it. **(b) The N1 directional-authority hazard is gone.** That guard existed because the bespoke resync applied a destructive delete-all-then-insert-all, so running it in the wrong direction wiped a live buffer. With a non-destructive merge there is no wipe to gate, and replication is symmetric — either node may write. `ActiveNodeEvaluator` survives, but only for the **delivery** gate and the heartbeat, which still genuinely need a single active node. **(c) The N5 orphan-row race is gone.** A `Remove` ordered before a re-add can no longer resurrect a row: deletes are tombstoned with an HLC, and a tombstone beats any older write for the same key. **The duplicate-delivery bound, stated explicitly.** Delivery remains single-node: only the primary runs the sweep (`IClusterNodeProvider.SelfIsPrimary`). A message can therefore be delivered twice only when the OLD primary delivered it and the resulting status change had not yet replicated at the instant the gate flipped. The window is one replication flush interval plus the in-flight ack, not an unbounded divergence — and unlike the old model it does not grow with backlog depth or with how long a node was absent. **One new bound replaces the old ones:** a node offline longer than `LocalDb:Replication:TombstoneRetention` (default 7 days) may resurrect deleted rows on rejoin, because the tombstones that would have suppressed them have been pruned. Stop-and-start a site pair together, and do not leave one node of a pair offline across that horizon.
|
||||
|
||||
### Operation Tracking Table (lives in Site Runtime, not here)
|
||||
|
||||
|
||||
@@ -68,6 +68,9 @@ services:
|
||||
MSSQL_PID: "Developer"
|
||||
volumes:
|
||||
- scadabridge-mssql-data:/var/opt/mssql
|
||||
# NOTE: the official mssql/server image does NOT run
|
||||
# /docker-entrypoint-initdb.d — these mounts are informational only;
|
||||
# infra/reseed.sh applies the scripts explicitly via sqlcmd.
|
||||
- ./mssql/setup.sql:/docker-entrypoint-initdb.d/setup.sql:ro
|
||||
- ./mssql/machinedata_seed.sql:/docker-entrypoint-initdb.d/machinedata_seed.sql:ro
|
||||
- ./mssql/setup-env2.sql:/docker-entrypoint-initdb.d/setup-env2.sql:ro
|
||||
|
||||
+9
-6
@@ -82,12 +82,15 @@ if ! $SKIP_TEARDOWN; then
|
||||
done
|
||||
echo " MSSQL ready."
|
||||
|
||||
echo " Waiting for setup.sql to create ScadaBridgeConfig..."
|
||||
until docker exec scadabridge-mssql /opt/mssql-tools18/bin/sqlcmd \
|
||||
-S localhost -U sa -P 'ScadaBridge_Dev1#' -C \
|
||||
-Q "IF DB_ID('ScadaBridgeConfig') IS NULL THROW 50000, 'not ready', 1;" \
|
||||
>/dev/null 2>&1; do
|
||||
sleep 2
|
||||
# The official mcr.microsoft.com/mssql/server image does NOT implement
|
||||
# /docker-entrypoint-initdb.d, so the compose-mounted init scripts never run
|
||||
# on their own — waiting for them here hangs forever on a fresh volume.
|
||||
# Apply them explicitly instead (all are idempotent).
|
||||
echo " Applying MSSQL init scripts (the mssql/server image has no initdb hook)..."
|
||||
for f in mssql/setup.sql mssql/machinedata_seed.sql mssql/setup-env2.sql; do
|
||||
echo " $f"
|
||||
docker exec -i scadabridge-mssql /opt/mssql-tools18/bin/sqlcmd \
|
||||
-S localhost -U sa -P 'ScadaBridge_Dev1#' -C -b < "$SCRIPT_DIR/$f"
|
||||
done
|
||||
echo " ScadaBridgeConfig present."
|
||||
|
||||
|
||||
@@ -27,6 +27,8 @@
|
||||
<package pattern="ZB.MOM.WW.Theme" />
|
||||
<package pattern="ZB.MOM.WW.Secrets" />
|
||||
<package pattern="ZB.MOM.WW.Secrets.*" />
|
||||
<package pattern="ZB.MOM.WW.LocalDb" />
|
||||
<package pattern="ZB.MOM.WW.LocalDb.*" />
|
||||
</packageSource>
|
||||
</packageSourceMapping>
|
||||
<!--
|
||||
|
||||
@@ -114,8 +114,8 @@ public sealed class CachedCallTelemetryForwarder : ICachedCallTelemetryForwarder
|
||||
// Kind/Status are domain fields carried in DetailsJson — decompose to log them.
|
||||
var d = AuditRowProjection.Decompose(telemetry.Audit);
|
||||
_logger.LogWarning(ex,
|
||||
"CachedCallTelemetryForwarder: audit emission threw for EventId {EventId} (Kind {Kind}, Status {Status})",
|
||||
d.EventId, d.Kind, d.Status);
|
||||
"CachedCallTelemetryForwarder: audit emission threw for EventId {EventId} (Kind {Kind}, Status {Status}, sqlite {SqliteError})",
|
||||
d.EventId, d.Kind, d.Status, SqliteErrorCodes.Describe(ex));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -192,8 +192,8 @@ public sealed class CachedCallTelemetryForwarder : ICachedCallTelemetryForwarder
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex,
|
||||
"CachedCallTelemetryForwarder: tracking-store emission threw for TrackedOperationId {Id} (Status {Status})",
|
||||
telemetry.Operational.TrackedOperationId, telemetry.Operational.Status);
|
||||
"CachedCallTelemetryForwarder: tracking-store emission threw for TrackedOperationId {Id} (Status {Status}, sqlite {SqliteError})",
|
||||
telemetry.Operational.TrackedOperationId, telemetry.Operational.Status, SqliteErrorCodes.Describe(ex));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -68,6 +68,15 @@ public class SiteAuditTelemetryActor : ReceiveActor
|
||||
private readonly IOperationTrackingStore? _trackingStore;
|
||||
private readonly SiteAuditTelemetryOptions _options;
|
||||
private readonly ILogger<SiteAuditTelemetryActor> _logger;
|
||||
// Captured at construction (both are thread-safe immutable handles) because
|
||||
// ScheduleNext/ScheduleNextCached run from the drains' finally blocks, whose
|
||||
// ConfigureAwait(false) continuations complete on pool threads with no
|
||||
// active ActorContext — reading Context/Self there either throws
|
||||
// NotSupportedException or, worse, silently resolves a STALE cell left in
|
||||
// the thread-static slot and re-arms the tick at the wrong actor
|
||||
// (known-issues/2026-07-20-localdb-disk-io-error-under-load.md §8).
|
||||
private readonly IScheduler _scheduler;
|
||||
private readonly IActorRef _self;
|
||||
private ICancelable? _pendingTick;
|
||||
private ICancelable? _pendingCachedTick;
|
||||
// Per-actor lifecycle CTS so an in-flight drain (queue read,
|
||||
@@ -108,6 +117,8 @@ public class SiteAuditTelemetryActor : ReceiveActor
|
||||
_options = options.Value;
|
||||
_logger = logger;
|
||||
_trackingStore = trackingStore;
|
||||
_scheduler = Context.System.Scheduler;
|
||||
_self = Self;
|
||||
|
||||
ReceiveAsync<Drain>(_ => OnDrainAsync());
|
||||
ReceiveAsync<CachedDrain>(_ => OnCachedDrainAsync());
|
||||
@@ -197,7 +208,9 @@ public class SiteAuditTelemetryActor : ReceiveActor
|
||||
{
|
||||
// Catch-all so a SQLite hiccup or mapper bug never crashes the
|
||||
// actor. The next tick is still scheduled in the finally block.
|
||||
_logger.LogError(ex, "Unexpected error during audit-log telemetry drain.");
|
||||
_logger.LogError(ex,
|
||||
"Unexpected error during audit-log telemetry drain (sqlite {SqliteError}).",
|
||||
SqliteErrorCodes.Describe(ex));
|
||||
}
|
||||
finally
|
||||
{
|
||||
@@ -278,8 +291,8 @@ public class SiteAuditTelemetryActor : ReceiveActor
|
||||
// batch — the audit half is best-effort. Log and skip
|
||||
// this row; it stays Pending for the next drain.
|
||||
_logger.LogWarning(ex,
|
||||
"Cached-telemetry drain: tracking lookup threw for {EventId} (TrackedOperationId {Tid}); skipping.",
|
||||
auditRow.EventId, auditRow.CorrelationId);
|
||||
"Cached-telemetry drain: tracking lookup threw for {EventId} (TrackedOperationId {Tid}, sqlite {SqliteError}); skipping.",
|
||||
auditRow.EventId, auditRow.CorrelationId, SqliteErrorCodes.Describe(ex));
|
||||
continue;
|
||||
}
|
||||
|
||||
@@ -332,7 +345,9 @@ public class SiteAuditTelemetryActor : ReceiveActor
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogError(ex, "Unexpected error during cached-telemetry drain.");
|
||||
_logger.LogError(ex,
|
||||
"Unexpected error during cached-telemetry drain (sqlite {SqliteError}).",
|
||||
SqliteErrorCodes.Describe(ex));
|
||||
}
|
||||
finally
|
||||
{
|
||||
@@ -428,24 +443,26 @@ public class SiteAuditTelemetryActor : ReceiveActor
|
||||
return list;
|
||||
}
|
||||
|
||||
// Must stay off Context/Self: called from off-context continuations — see
|
||||
// the _scheduler/_self field comment.
|
||||
private void ScheduleNext(TimeSpan delay)
|
||||
{
|
||||
_pendingTick?.Cancel();
|
||||
_pendingTick = Context.System.Scheduler.ScheduleTellOnceCancelable(
|
||||
_pendingTick = _scheduler.ScheduleTellOnceCancelable(
|
||||
delay,
|
||||
Self,
|
||||
_self,
|
||||
Drain.Instance,
|
||||
Self);
|
||||
_self);
|
||||
}
|
||||
|
||||
private void ScheduleNextCached(TimeSpan delay)
|
||||
{
|
||||
_pendingCachedTick?.Cancel();
|
||||
_pendingCachedTick = Context.System.Scheduler.ScheduleTellOnceCancelable(
|
||||
_pendingCachedTick = _scheduler.ScheduleTellOnceCancelable(
|
||||
delay,
|
||||
Self,
|
||||
_self,
|
||||
CachedDrain.Instance,
|
||||
Self);
|
||||
_self);
|
||||
}
|
||||
|
||||
/// <summary>Self-tick message that triggers an audit-only drain cycle.</summary>
|
||||
|
||||
@@ -0,0 +1,31 @@
|
||||
using Microsoft.Data.Sqlite;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.AuditLog.Site.Telemetry;
|
||||
|
||||
/// <summary>
|
||||
/// Renders the primary/extended SQLite result codes of a
|
||||
/// <see cref="SqliteException"/> for log messages. The exception's own message
|
||||
/// carries only the primary code ("SQLite Error 10: 'disk I/O error'"), which
|
||||
/// is too generic to act on — the 2026-07-20 disk-I/O incident
|
||||
/// (known-issues/2026-07-20-localdb-disk-io-error-under-load.md) had to be
|
||||
/// reproduced from scratch to learn the extended code (522 =
|
||||
/// SQLITE_IOERR_SHORT_READ) that names the failing operation.
|
||||
/// </summary>
|
||||
internal static class SqliteErrorCodes
|
||||
{
|
||||
/// <summary>
|
||||
/// "primary/extended" (e.g. "10/522") for a <see cref="SqliteException"/>
|
||||
/// anywhere in the exception chain; "n/a" for non-SQLite failures.
|
||||
/// </summary>
|
||||
public static string Describe(Exception ex)
|
||||
{
|
||||
for (Exception? e = ex; e is not null; e = e.InnerException)
|
||||
{
|
||||
if (e is SqliteException se)
|
||||
{
|
||||
return $"{se.SqliteErrorCode}/{se.SqliteExtendedErrorCode}";
|
||||
}
|
||||
}
|
||||
return "n/a";
|
||||
}
|
||||
}
|
||||
@@ -178,7 +178,10 @@
|
||||
|
||||
private List<EventLogEntry>? _entries;
|
||||
private bool _hasMore;
|
||||
private long? _continuationToken;
|
||||
// Opaque keyset cursor from the previous response — passed back verbatim, never
|
||||
// parsed. It became a string in LocalDb Phase 1 when site_events ids turned into
|
||||
// GUIDs and the cursor had to carry the timestamp too.
|
||||
private string? _continuationToken;
|
||||
private bool _searching;
|
||||
private string? _errorMessage;
|
||||
private ToastNotification _toast = default!;
|
||||
|
||||
@@ -73,6 +73,37 @@ public record SiteHealthReport(
|
||||
/// indicates a stuck/blocked script holding a dedicated thread.
|
||||
/// </summary>
|
||||
public double? ScriptOldestBusyAgeSeconds { get; init; }
|
||||
|
||||
// LocalDb 2-node replication of the consolidated site database (Phase 1).
|
||||
// Additive init properties for the same reason as the scheduler gauges above:
|
||||
// the positional constructor stays untouched. Refreshed on the site by
|
||||
// LocalDbReplicationStatusReporter.
|
||||
|
||||
/// <summary>
|
||||
/// Whether a replication sync session is currently running with the peer site node,
|
||||
/// or <see langword="null"/> when replication is not wired on this node.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Nullable on purpose, and <see langword="false"/> is NOT the same as null.
|
||||
/// Replication ships default-OFF, so a site node with no peer configured reports
|
||||
/// <see langword="false"/> — that is a healthy, expected state, not an outage. Only
|
||||
/// a node that was configured with a peer and is reporting <see langword="false"/>
|
||||
/// is degraded, and distinguishing those two cases is the operator's job, not this
|
||||
/// field's.
|
||||
/// </remarks>
|
||||
public bool? LocalDbReplicationConnected { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// Unacked replication oplog entries waiting to reach the peer, or
|
||||
/// <see langword="null"/> when unknown.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <b>Null means unknown, never zero.</b> The underlying provider returns null when
|
||||
/// no backlog source is wired or the poll failed, and that must survive to the
|
||||
/// operator: a failed read rendered as "0 backlog" would report a broken replication
|
||||
/// pair as perfectly healthy. Collapsing null to 0 anywhere on this path is a bug.
|
||||
/// </remarks>
|
||||
public long? LocalDbOplogBacklog { get; init; }
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
|
||||
@@ -3,13 +3,19 @@ namespace ZB.MOM.WW.ScadaBridge.Commons.Messages.RemoteQuery;
|
||||
/// <summary>
|
||||
/// Request to query site event logs from central.
|
||||
/// Supports filtering by event type, severity, instance, time range, and keyword search.
|
||||
/// Uses keyset pagination via continuation token (last event ID).
|
||||
/// Uses keyset pagination via an opaque continuation token.
|
||||
/// </summary>
|
||||
/// <param name="InstanceId">
|
||||
/// Instance filter matched against the site event log's <c>instance_id</c> column,
|
||||
/// which stores the instance <b>UniqueName</b> (InstanceActor.LogLifecycleEvent passes
|
||||
/// <c>_instanceUniqueName</c>; EventLogQueryService matches <c>instance_id = $instanceId</c>).
|
||||
/// </param>
|
||||
/// <param name="ContinuationToken">
|
||||
/// Opaque cursor from the previous response's <c>ContinuationToken</c>, or
|
||||
/// <see langword="null"/> to start from the oldest matching event. Treat as opaque —
|
||||
/// it encodes timestamp and id together and its format is not part of the contract.
|
||||
/// An unparseable token is treated as "start from the beginning" rather than an error.
|
||||
/// </param>
|
||||
public record EventLogQueryRequest(
|
||||
string CorrelationId,
|
||||
string SiteId,
|
||||
@@ -19,6 +25,6 @@ public record EventLogQueryRequest(
|
||||
string? Severity,
|
||||
string? InstanceId,
|
||||
string? KeywordFilter,
|
||||
long? ContinuationToken,
|
||||
string? ContinuationToken,
|
||||
int PageSize,
|
||||
DateTimeOffset Timestamp);
|
||||
|
||||
@@ -3,8 +3,18 @@ namespace ZB.MOM.WW.ScadaBridge.Commons.Messages.RemoteQuery;
|
||||
/// <summary>
|
||||
/// A single event log entry returned from a site query.
|
||||
/// </summary>
|
||||
/// <param name="Id">
|
||||
/// The event's primary key: a GUID string minted by the recording site node.
|
||||
/// <para>
|
||||
/// This was a <c>long</c> autoincrement id until LocalDb Phase 1. Site pairs now
|
||||
/// replicate <c>site_events</c> with last-writer-wins on this key, so a
|
||||
/// server-minted integer would have both nodes issuing the same ids for unrelated
|
||||
/// events and silently overwriting each other on sync. Consumers must treat it as
|
||||
/// an opaque identifier — it carries no ordering.
|
||||
/// </para>
|
||||
/// </param>
|
||||
public record EventLogEntry(
|
||||
long Id,
|
||||
string Id,
|
||||
DateTimeOffset Timestamp,
|
||||
string EventType,
|
||||
string Severity,
|
||||
@@ -15,13 +25,18 @@ public record EventLogEntry(
|
||||
|
||||
/// <summary>
|
||||
/// Response containing paginated event log entries from a site.
|
||||
/// Uses keyset pagination: ContinuationToken is the last event ID in the result set.
|
||||
/// </summary>
|
||||
/// <param name="ContinuationToken">
|
||||
/// Opaque keyset-pagination cursor: pass it back verbatim on the next request to
|
||||
/// continue after the last returned row, or <see langword="null"/> to start from the
|
||||
/// beginning. Encodes <c>timestamp</c> and <c>id</c> together, because GUID ids do not
|
||||
/// sort chronologically and timestamps alone are not unique. Do not parse it.
|
||||
/// </param>
|
||||
public record EventLogQueryResponse(
|
||||
string CorrelationId,
|
||||
string SiteId,
|
||||
IReadOnlyList<EventLogEntry> Entries,
|
||||
long? ContinuationToken,
|
||||
string? ContinuationToken,
|
||||
bool HasMore,
|
||||
bool Success,
|
||||
string? ErrorMessage,
|
||||
|
||||
@@ -10,11 +10,20 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.ClusterState;
|
||||
/// once the original first node restarts and rejoins; every product-level active/standby
|
||||
/// decision must use this evaluator, never <c>cluster.State.Leader</c>.
|
||||
/// <para>
|
||||
/// Lives in Communication (not Host) so BOTH <c>SiteCommunicationActor</c> and
|
||||
/// <c>SiteReplicationActor</c> can default to it — Host cannot be referenced from either.
|
||||
/// The Host's <c>ClusterActivityEvaluator.SelfIsOldest</c> delegates here, so the S&F
|
||||
/// delivery gate (<c>IClusterNodeProvider.SelfIsPrimary</c>), the resync authority checks,
|
||||
/// and the heartbeat IsActive stamp all share one implementation.
|
||||
/// Lives in Communication (not Host) so <c>SiteCommunicationActor</c> can default to it —
|
||||
/// Host cannot be referenced from there. The Host's
|
||||
/// <c>ClusterActivityEvaluator.SelfIsOldest</c> delegates here, so the S&F delivery gate
|
||||
/// (<c>IClusterNodeProvider.SelfIsPrimary</c>) and the heartbeat IsActive stamp share one
|
||||
/// implementation.
|
||||
/// <para>
|
||||
/// It also backed <c>SiteReplicationActor</c>'s resync authority checks until LocalDb
|
||||
/// Phase 2 deleted that actor. Those checks existed because the bespoke resync applied a
|
||||
/// destructive delete-all-then-insert-all, so running it in the wrong direction wiped a
|
||||
/// live store-and-forward buffer. LocalDb's snapshot resync merges per row under
|
||||
/// last-writer-wins and never deletes, so there is no destructive apply left to gate — the
|
||||
/// evaluator survives for the delivery gate and the heartbeat, which still genuinely need
|
||||
/// a single active node.
|
||||
/// </para>
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public static class ActiveNodeEvaluator
|
||||
|
||||
@@ -141,6 +141,25 @@ public interface ISiteHealthCollector
|
||||
// SiteHealthCollector overrides this with the Interlocked.Exchange store.
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Replace the latest LocalDb replication status (peer-session connectivity and
|
||||
/// unacked oplog backlog) used by the next <see cref="CollectReport"/> call.
|
||||
/// Refreshed periodically by the <c>LocalDbReplicationStatusReporter</c> hosted
|
||||
/// service. Point-in-time: values are NOT reset on <see cref="CollectReport"/>.
|
||||
/// </summary>
|
||||
/// <param name="connected">
|
||||
/// Whether a sync session is currently running. <see langword="false"/> is the
|
||||
/// normal state on a node with no peer configured — replication ships default-OFF.
|
||||
/// </param>
|
||||
/// <param name="oplogBacklog">
|
||||
/// Unacked oplog entries, or <see langword="null"/> when unknown. Pass null through
|
||||
/// unchanged: a failed poll rendered as 0 would report a broken pair as healthy.
|
||||
/// </param>
|
||||
void SetLocalDbReplicationStatus(bool connected, long? oplogBacklog)
|
||||
{
|
||||
// Default no-op so test fakes do not need to be updated.
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Replace the latest script-execution-scheduler gauges (queue depth, busy
|
||||
/// thread count, and age in seconds of the oldest in-flight script) used by
|
||||
|
||||
@@ -0,0 +1,130 @@
|
||||
using Microsoft.Extensions.Hosting;
|
||||
using Microsoft.Extensions.Logging;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.HealthMonitoring;
|
||||
|
||||
/// <summary>
|
||||
/// Site-side hosted service that periodically reads LocalDb replication status and pushes
|
||||
/// it into <see cref="ISiteHealthCollector"/>, so the next
|
||||
/// <see cref="ISiteHealthCollector.CollectReport"/> emits fresh
|
||||
/// <c>LocalDbReplicationConnected</c> / <c>LocalDbOplogBacklog</c> fields.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>Why delegates and not <c>ISyncStatus</c> directly.</b> Same reasoning as
|
||||
/// <see cref="SiteEventLogFailureCountReporter"/>: HealthMonitoring does not take a
|
||||
/// reference on the replication library. The Host site wiring captures the two reads as
|
||||
/// lambdas at registration time; this service only moves numbers.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Null backlog is preserved, never coerced to zero.</b> <c>ISyncStatus.OplogBacklog</c>
|
||||
/// is nullable precisely so a failed poll reads as "unknown" — rendering it as 0 would
|
||||
/// report a replication pair that cannot read its own oplog as perfectly healthy.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Cadence.</b> 30 s, matching <see cref="SiteEventLogFailureCountReporter"/> and
|
||||
/// <c>SiteAuditBacklogReporter</c>. Any exception during a probe is logged and swallowed;
|
||||
/// the next tick retries.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed class LocalDbReplicationStatusReporter : IHostedService, IDisposable
|
||||
{
|
||||
/// <summary>Default poll cadence, matching the other site health bridges.</summary>
|
||||
internal static readonly TimeSpan DefaultRefreshInterval = TimeSpan.FromSeconds(30);
|
||||
|
||||
private readonly Func<bool> _connectedProvider;
|
||||
private readonly Func<long?> _oplogBacklogProvider;
|
||||
private readonly ISiteHealthCollector _collector;
|
||||
private readonly ILogger<LocalDbReplicationStatusReporter> _logger;
|
||||
private readonly TimeSpan _refreshInterval;
|
||||
private CancellationTokenSource? _cts;
|
||||
private Task? _loop;
|
||||
|
||||
/// <summary>Initializes a new instance of <see cref="LocalDbReplicationStatusReporter"/>.</summary>
|
||||
/// <param name="connectedProvider">Reads whether a peer sync session is currently running.</param>
|
||||
/// <param name="oplogBacklogProvider">Reads the unacked oplog backlog, or null when unknown.</param>
|
||||
/// <param name="collector">The site health collector receiving the snapshot.</param>
|
||||
/// <param name="logger">Logger instance.</param>
|
||||
/// <param name="refreshInterval">Poll interval override; defaults to 30 s.</param>
|
||||
public LocalDbReplicationStatusReporter(
|
||||
Func<bool> connectedProvider,
|
||||
Func<long?> oplogBacklogProvider,
|
||||
ISiteHealthCollector collector,
|
||||
ILogger<LocalDbReplicationStatusReporter> logger,
|
||||
TimeSpan? refreshInterval = null)
|
||||
{
|
||||
_connectedProvider = connectedProvider ?? throw new ArgumentNullException(nameof(connectedProvider));
|
||||
_oplogBacklogProvider = oplogBacklogProvider ?? throw new ArgumentNullException(nameof(oplogBacklogProvider));
|
||||
_collector = collector ?? throw new ArgumentNullException(nameof(collector));
|
||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||
_refreshInterval = refreshInterval ?? DefaultRefreshInterval;
|
||||
}
|
||||
|
||||
/// <summary>Starts the polling loop, probing once immediately before entering the timed cycle.</summary>
|
||||
/// <param name="ct">Cancellation token signalling host shutdown.</param>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
public Task StartAsync(CancellationToken ct)
|
||||
{
|
||||
var cts = CancellationTokenSource.CreateLinkedTokenSource(ct);
|
||||
_cts = cts;
|
||||
_loop = RunAsync(cts.Token);
|
||||
return Task.CompletedTask;
|
||||
}
|
||||
|
||||
/// <summary>Stops the polling loop.</summary>
|
||||
/// <param name="ct">Cancellation token for the stop operation.</param>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
public async Task StopAsync(CancellationToken ct)
|
||||
{
|
||||
if (_cts is null) return;
|
||||
|
||||
await _cts.CancelAsync().ConfigureAwait(false);
|
||||
if (_loop is not null)
|
||||
{
|
||||
// Await the loop so the reporter is quiescent before the host disposes the
|
||||
// collector out from under it.
|
||||
try { await _loop.ConfigureAwait(false); }
|
||||
catch (OperationCanceledException) { /* expected on shutdown */ }
|
||||
}
|
||||
}
|
||||
|
||||
private async Task RunAsync(CancellationToken ct)
|
||||
{
|
||||
while (!ct.IsCancellationRequested)
|
||||
{
|
||||
Probe();
|
||||
|
||||
try
|
||||
{
|
||||
await Task.Delay(_refreshInterval, ct).ConfigureAwait(false);
|
||||
}
|
||||
catch (OperationCanceledException)
|
||||
{
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Reads both providers and pushes one snapshot onto the collector; never throws.
|
||||
/// Public so a test can drive a single deterministic probe instead of starting the
|
||||
/// service and waiting out the 30 s cadence.
|
||||
/// </summary>
|
||||
public void Probe()
|
||||
{
|
||||
try
|
||||
{
|
||||
_collector.SetLocalDbReplicationStatus(
|
||||
_connectedProvider(), _oplogBacklogProvider());
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
// Health reporting must never take the site node down. The previous snapshot
|
||||
// stays on the collector and the next tick retries.
|
||||
_logger.LogWarning(ex, "Failed to read LocalDb replication status for the site health report.");
|
||||
}
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public void Dispose() => _cts?.Dispose();
|
||||
}
|
||||
@@ -21,6 +21,12 @@ public static class ServiceCollectionExtensions
|
||||
/// </summary>
|
||||
private sealed class SiteEventLogHealthMetricsBridgeMarker { }
|
||||
|
||||
/// <summary>
|
||||
/// Sentinel marker for <see cref="AddLocalDbReplicationHealthBridge"/>'s idempotency
|
||||
/// guard — same rationale as <see cref="SiteEventLogHealthMetricsBridgeMarker"/>.
|
||||
/// </summary>
|
||||
private sealed class LocalDbReplicationHealthBridgeMarker { }
|
||||
|
||||
/// <summary>
|
||||
/// Register site-side health monitoring services (metric collection + periodic reporting).
|
||||
/// Call this on site nodes only. For central, call AddCentralHealthAggregation() instead.
|
||||
@@ -153,6 +159,46 @@ public static class ServiceCollectionExtensions
|
||||
return services;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Bridge LocalDb replication status (peer connectivity + unacked oplog backlog) onto
|
||||
/// the site health report. Must be called AFTER <c>AddSiteHealthMonitoring</c>
|
||||
/// (registers <see cref="ISiteHealthCollector"/>) and after the replication engine is
|
||||
/// registered. Idempotent via a marker sentinel.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Delegates rather than an <c>ISyncStatus</c> parameter keep HealthMonitoring free of
|
||||
/// a reference on the replication library, matching
|
||||
/// <see cref="AddSiteEventLogHealthMetricsBridge"/>. Pass the backlog through as
|
||||
/// nullable — coercing null to 0 would report an unreadable oplog as a healthy one.
|
||||
/// </remarks>
|
||||
/// <param name="services">The service collection to register into.</param>
|
||||
/// <param name="connectedProvider">Given the root provider, returns a reader for "a sync session is running".</param>
|
||||
/// <param name="oplogBacklogProvider">Given the root provider, returns a reader for the unacked backlog (null = unknown).</param>
|
||||
/// <returns>The same <see cref="IServiceCollection"/> for chaining.</returns>
|
||||
public static IServiceCollection AddLocalDbReplicationHealthBridge(
|
||||
this IServiceCollection services,
|
||||
Func<IServiceProvider, Func<bool>> connectedProvider,
|
||||
Func<IServiceProvider, Func<long?>> oplogBacklogProvider)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(services);
|
||||
ArgumentNullException.ThrowIfNull(connectedProvider);
|
||||
ArgumentNullException.ThrowIfNull(oplogBacklogProvider);
|
||||
|
||||
if (services.Any(d => d.ServiceType == typeof(LocalDbReplicationHealthBridgeMarker)))
|
||||
{
|
||||
return services;
|
||||
}
|
||||
|
||||
services.AddSingleton<LocalDbReplicationHealthBridgeMarker>();
|
||||
services.AddHostedService(sp => new LocalDbReplicationStatusReporter(
|
||||
connectedProvider(sp),
|
||||
oplogBacklogProvider(sp),
|
||||
sp.GetRequiredService<ISiteHealthCollector>(),
|
||||
sp.GetRequiredService<ILogger<LocalDbReplicationStatusReporter>>()));
|
||||
|
||||
return services;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Register the <see cref="HealthMonitoringOptionsValidator"/>
|
||||
/// so a misconfigured <c>ScadaBridge:HealthMonitoring</c> section (zero/negative
|
||||
|
||||
@@ -18,6 +18,10 @@ public class SiteHealthCollector : ISiteHealthCollector
|
||||
private int _auditRedactionFailures;
|
||||
private volatile SiteAuditBacklogSnapshot? _siteAuditBacklog;
|
||||
private long _siteEventLogWriteFailures;
|
||||
// One volatile tuple rather than two independent fields: a torn read that paired a
|
||||
// fresh Connected with a stale backlog would be indistinguishable from a real state.
|
||||
// Null = the reporter has not yet run (or replication is not wired).
|
||||
private volatile Tuple<bool, long?>? _localDbReplicationStatus;
|
||||
private readonly ConcurrentDictionary<string, ConnectionHealth> _connectionStatuses = new();
|
||||
private readonly ConcurrentDictionary<string, TagResolutionStatus> _tagResolutionCounts = new();
|
||||
private readonly ConcurrentDictionary<string, string> _connectionEndpoints = new();
|
||||
@@ -94,6 +98,12 @@ public class SiteHealthCollector : ISiteHealthCollector
|
||||
Interlocked.Exchange(ref _siteEventLogWriteFailures, count);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public void SetLocalDbReplicationStatus(bool connected, long? oplogBacklog)
|
||||
{
|
||||
_localDbReplicationStatus = Tuple.Create(connected, oplogBacklog);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public void UpdateConnectionHealth(string connectionName, ConnectionHealth health)
|
||||
{
|
||||
@@ -219,6 +229,9 @@ public class SiteHealthCollector : ISiteHealthCollector
|
||||
var deadLetters = Interlocked.Exchange(ref _deadLetterCount, 0);
|
||||
var siteAuditWriteFailures = Interlocked.Exchange(ref _siteAuditWriteFailures, 0);
|
||||
var auditRedactionFailures = Interlocked.Exchange(ref _auditRedactionFailures, 0);
|
||||
// Single read of the volatile tuple — two reads could straddle a reporter tick and
|
||||
// pair a fresh Connected with a stale backlog.
|
||||
var localDbReplication = _localDbReplicationStatus;
|
||||
|
||||
// Snapshot current connection and tag resolution state
|
||||
var connectionStatuses = new Dictionary<string, ConnectionHealth>(_connectionStatuses);
|
||||
@@ -259,7 +272,12 @@ public class SiteHealthCollector : ISiteHealthCollector
|
||||
{
|
||||
ScriptQueueDepth = Interlocked.CompareExchange(ref _scriptQueueDepth, 0, 0),
|
||||
ScriptBusyThreads = Interlocked.CompareExchange(ref _scriptBusyThreads, 0, 0),
|
||||
ScriptOldestBusyAgeSeconds = ReadScriptOldestBusyAgeSeconds()
|
||||
ScriptOldestBusyAgeSeconds = ReadScriptOldestBusyAgeSeconds(),
|
||||
// Both fields come from the ONE snapshot read above. Null (the reporter has
|
||||
// not run) leaves both report fields null — "no data", not "disconnected
|
||||
// with an empty backlog".
|
||||
LocalDbReplicationConnected = localDbReplication?.Item1,
|
||||
LocalDbOplogBacklog = localDbReplication?.Item2
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
@@ -766,37 +766,18 @@ akka {{
|
||||
var deploymentConfigFetcher =
|
||||
_serviceProvider.GetService<ZB.MOM.WW.ScadaBridge.SiteRuntime.Deployment.IDeploymentConfigFetcher>();
|
||||
|
||||
// Create SiteReplicationActor on every node (not a singleton)
|
||||
var sfStorage = _serviceProvider.GetRequiredService<StoreAndForwardStorage>();
|
||||
var replicationService = _serviceProvider.GetRequiredService<ReplicationService>();
|
||||
var replicationLogger = _serviceProvider.GetRequiredService<ILoggerFactory>()
|
||||
.CreateLogger<SiteReplicationActor>();
|
||||
|
||||
// ONE active-node predicate instance governs the S&F delivery gate, the resync
|
||||
// authority checks (SiteReplicationActor), and the heartbeat IsActive stamp
|
||||
// (SiteCommunicationActor, wired below) — review 02 round 2, N1. Null in
|
||||
// non-clustered test hosts: the actors fall back to the shared oldest-Up
|
||||
// evaluator, never to a leader check.
|
||||
// ONE active-node predicate instance governs the S&F delivery gate and the
|
||||
// heartbeat IsActive stamp (SiteCommunicationActor, wired below) — review 02
|
||||
// round 2, N1. It also governed SiteReplicationActor's resync authority until
|
||||
// LocalDb Phase 2 deleted that actor: the library's snapshot resync merges per row
|
||||
// under last-writer-wins and never deletes, so there is no destructive apply left
|
||||
// to need an authority check. Null in non-clustered test hosts: the consumers fall
|
||||
// back to the shared oldest-Up evaluator, never to a leader check.
|
||||
var clusterNodeProvider = _serviceProvider.GetService<ZB.MOM.WW.ScadaBridge.HealthMonitoring.IClusterNodeProvider>();
|
||||
Func<bool>? activeNodeCheck = clusterNodeProvider != null
|
||||
? () => clusterNodeProvider.SelfIsPrimary
|
||||
: null;
|
||||
|
||||
var replicationActor = _actorSystem!.ActorOf(
|
||||
Props.Create(() => new SiteReplicationActor(
|
||||
storage, sfStorage, replicationService, siteRole, replicationLogger,
|
||||
deploymentConfigFetcher, activeNodeCheck, siteRuntimeOptionsValue, null)),
|
||||
"site-replication");
|
||||
|
||||
// Wire S&F replication handler to forward operations via the replication actor
|
||||
replicationService.SetReplicationHandler(op =>
|
||||
{
|
||||
replicationActor.Tell(new ReplicateStoreAndForward(op));
|
||||
return Task.CompletedTask;
|
||||
});
|
||||
|
||||
_logger.LogInformation("SiteReplicationActor created and S&F replication handler wired");
|
||||
|
||||
// Deployment Manager — role-scoped singleton via SingletonRegistrar
|
||||
// (review 01 round-2 N5): previously hand-rolled with bare PoisonPill
|
||||
// termination and NO PhaseClusterLeave drain, so in-flight SQLite
|
||||
@@ -807,7 +788,7 @@ akka {{
|
||||
_actorSystem!, "deployment-manager",
|
||||
Props.Create(() => new DeploymentManagerActor(
|
||||
storage, compilationService, sharedScriptLibrary, streamManager,
|
||||
siteRuntimeOptionsValue, dmLogger, dclManager, replicationActor,
|
||||
siteRuntimeOptionsValue, dmLogger, dclManager,
|
||||
siteHealthCollector, _serviceProvider, null, deploymentConfigFetcher)),
|
||||
_logger, role: siteRole);
|
||||
var dmProxy = dm.Proxy;
|
||||
@@ -1053,7 +1034,7 @@ akka {{
|
||||
// SetReady asserts a deliberately narrow contract. By this point the
|
||||
// actor system exists, SiteStreamManager.Initialize has run, and every
|
||||
// role actor (SiteCommunicationActor, deployment-manager singleton,
|
||||
// SiteReplicationActor, the ClusterClient) has been created with ActorOf —
|
||||
// the ClusterClient) has been created with ActorOf —
|
||||
// creation and the registration Tells are synchronous and strictly ordered.
|
||||
// What is NOT guaranteed is completion of each actor's PreStart or the
|
||||
// ClusterClient's initial-contact handshake with central: those are
|
||||
|
||||
@@ -0,0 +1,158 @@
|
||||
using System.Security.Cryptography;
|
||||
using System.Text;
|
||||
using Grpc.Core;
|
||||
using Grpc.Core.Interceptors;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.LocalDb.Replication;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.Host;
|
||||
|
||||
/// <summary>
|
||||
/// Gates the LocalDb passive sync endpoint. The replication library deliberately leaves
|
||||
/// inbound authentication to the host — its <c>LocalDbSyncService</c> verifies nothing —
|
||||
/// so without this interceptor anything that can reach the site node's gRPC port could
|
||||
/// stream arbitrary rows into the consolidated site database.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>Scoped by method path.</b> Only calls under
|
||||
/// <c>/localdb_sync.v1.LocalDbSync/</c> are gated; every other method — notably the
|
||||
/// existing <c>SiteStream</c> service sharing this listener — passes through untouched.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Fail-closed.</b> With no <c>LocalDb:Replication:ApiKey</c> configured, NO sync
|
||||
/// stream is accepted, authenticated or not. That is the deliberate choice: the
|
||||
/// alternative — treating "no key" as "no auth required" — would silently expose the
|
||||
/// endpoint on exactly the default configuration every site node ships with. An operator
|
||||
/// enabling replication must set the same key on both nodes, which is already required
|
||||
/// for the initiator to dial out (<c>SyncBackgroundService</c> sends
|
||||
/// <c>Authorization: Bearer <key></c>).
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Comparison is <see cref="CryptographicOperations.FixedTimeEquals"/> over UTF-8 bytes,
|
||||
/// so a wrong key cannot be recovered byte-by-byte from response timing. Length differences
|
||||
/// are unavoidably observable and are not sensitive.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed class LocalDbSyncAuthInterceptor : Interceptor
|
||||
{
|
||||
private const string ServicePrefix = "/localdb_sync.v1.LocalDbSync/";
|
||||
private const string AuthorizationHeader = "authorization";
|
||||
private const string BearerPrefix = "Bearer ";
|
||||
|
||||
private readonly IOptions<ReplicationOptions> _options;
|
||||
private readonly ILogger<LocalDbSyncAuthInterceptor> _logger;
|
||||
|
||||
/// <summary>Creates the interceptor.</summary>
|
||||
/// <param name="options">Replication options; <c>ApiKey</c> is the expected bearer token.</param>
|
||||
/// <param name="logger">Logger for denial diagnostics.</param>
|
||||
public LocalDbSyncAuthInterceptor(
|
||||
IOptions<ReplicationOptions> options,
|
||||
ILogger<LocalDbSyncAuthInterceptor> logger)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
ArgumentNullException.ThrowIfNull(logger);
|
||||
|
||||
_options = options;
|
||||
_logger = logger;
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public override Task<TResponse> UnaryServerHandler<TRequest, TResponse>(
|
||||
TRequest request,
|
||||
ServerCallContext context,
|
||||
UnaryServerMethod<TRequest, TResponse> continuation)
|
||||
{
|
||||
Authorize(context);
|
||||
return continuation(request, context);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public override Task DuplexStreamingServerHandler<TRequest, TResponse>(
|
||||
IAsyncStreamReader<TRequest> requestStream,
|
||||
IServerStreamWriter<TResponse> responseStream,
|
||||
ServerCallContext context,
|
||||
DuplexStreamingServerMethod<TRequest, TResponse> continuation)
|
||||
{
|
||||
Authorize(context);
|
||||
return continuation(requestStream, responseStream, context);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public override Task<TResponse> ClientStreamingServerHandler<TRequest, TResponse>(
|
||||
IAsyncStreamReader<TRequest> requestStream,
|
||||
ServerCallContext context,
|
||||
ClientStreamingServerMethod<TRequest, TResponse> continuation)
|
||||
{
|
||||
Authorize(context);
|
||||
return continuation(requestStream, context);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public override Task ServerStreamingServerHandler<TRequest, TResponse>(
|
||||
TRequest request,
|
||||
IServerStreamWriter<TResponse> responseStream,
|
||||
ServerCallContext context,
|
||||
ServerStreamingServerMethod<TRequest, TResponse> continuation)
|
||||
{
|
||||
Authorize(context);
|
||||
return continuation(request, responseStream, context);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Throws <see cref="RpcException"/> with <see cref="StatusCode.PermissionDenied"/> if
|
||||
/// this is a sync call that does not carry the configured bearer token. Non-sync calls
|
||||
/// return immediately.
|
||||
/// </summary>
|
||||
private void Authorize(ServerCallContext context)
|
||||
{
|
||||
if (!context.Method.StartsWith(ServicePrefix, StringComparison.Ordinal))
|
||||
return;
|
||||
|
||||
var expected = _options.Value.ApiKey;
|
||||
if (string.IsNullOrEmpty(expected))
|
||||
{
|
||||
_logger.LogWarning(
|
||||
"Rejected a LocalDb sync call to {Method}: no LocalDb:Replication:ApiKey is configured, " +
|
||||
"so the passive sync endpoint is closed. Configure the same key on both nodes of the pair.",
|
||||
context.Method);
|
||||
throw new RpcException(new Status(
|
||||
StatusCode.PermissionDenied,
|
||||
"LocalDb sync is not accepting connections: no API key is configured on this node."));
|
||||
}
|
||||
|
||||
var presented = ExtractBearerToken(context.RequestHeaders);
|
||||
if (presented is null || !FixedTimeEquals(presented, expected))
|
||||
{
|
||||
_logger.LogWarning(
|
||||
"Rejected a LocalDb sync call to {Method}: {Reason}.",
|
||||
context.Method,
|
||||
presented is null ? "no bearer token presented" : "bearer token did not match");
|
||||
throw new RpcException(new Status(
|
||||
StatusCode.PermissionDenied,
|
||||
"LocalDb sync authentication failed."));
|
||||
}
|
||||
}
|
||||
|
||||
private static string? ExtractBearerToken(Metadata headers)
|
||||
{
|
||||
// gRPC lowercases header keys on the wire; compare case-insensitively anyway so a
|
||||
// hand-built Metadata in a test behaves the same as a real request.
|
||||
foreach (var entry in headers)
|
||||
{
|
||||
if (!string.Equals(entry.Key, AuthorizationHeader, StringComparison.OrdinalIgnoreCase))
|
||||
continue;
|
||||
|
||||
var value = entry.Value;
|
||||
if (value is not null && value.StartsWith(BearerPrefix, StringComparison.OrdinalIgnoreCase))
|
||||
return value[BearerPrefix.Length..];
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
private static bool FixedTimeEquals(string presented, string expected)
|
||||
=> CryptographicOperations.FixedTimeEquals(
|
||||
Encoding.UTF8.GetBytes(presented), Encoding.UTF8.GetBytes(expected));
|
||||
}
|
||||
@@ -7,6 +7,7 @@ using ZB.MOM.WW.Auth.AspNetCore;
|
||||
using ZB.MOM.WW.Health;
|
||||
using ZB.MOM.WW.Health.Akka;
|
||||
using ZB.MOM.WW.Health.EntityFrameworkCore;
|
||||
using ZB.MOM.WW.LocalDb.Replication;
|
||||
using ZB.MOM.WW.ScadaBridge.AuditLog;
|
||||
using ZB.MOM.WW.ScadaBridge.CentralUI;
|
||||
using ZB.MOM.WW.ScadaBridge.ClusterInfrastructure;
|
||||
@@ -513,10 +514,15 @@ try
|
||||
});
|
||||
|
||||
// gRPC server registration
|
||||
builder.Services.AddGrpc();
|
||||
// The interceptor gates ONLY /localdb_sync.v1.LocalDbSync/ — SiteStream calls on
|
||||
// this same pipeline pass through untouched. It is fail-closed: with no
|
||||
// LocalDb:Replication:ApiKey configured, no sync stream is accepted at all.
|
||||
builder.Services.AddGrpc(options =>
|
||||
options.Interceptors.Add<LocalDbSyncAuthInterceptor>());
|
||||
builder.Services.AddSingleton<ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamGrpcServer>();
|
||||
|
||||
// Existing site service registrations
|
||||
// Existing site service registrations (this is also where LocalDb and its
|
||||
// replication engine are registered — see SiteServiceRegistration)
|
||||
SiteServiceRegistration.Configure(builder.Services, builder.Configuration);
|
||||
|
||||
var app = builder.Build();
|
||||
@@ -535,6 +541,13 @@ try
|
||||
// Map gRPC service — resolves the singleton SiteStreamGrpcServer from DI
|
||||
app.MapGrpcService<ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamGrpcServer>();
|
||||
|
||||
// The passive half of LocalDb replication: the peer node dials THIS endpoint.
|
||||
// It shares the Kestrel h2c listener the site gRPC server already uses, so no
|
||||
// listener or port changes are needed. Mapping it is harmless with no peer
|
||||
// configured — nothing dials it, and LocalDbSyncAuthInterceptor (registered on
|
||||
// AddGrpc above) rejects anything that tries until an ApiKey is configured.
|
||||
app.MapZbLocalDbSync();
|
||||
|
||||
// Site-shutdown ordering. ApplicationStopping
|
||||
// fires BEFORE IHostedService.StopAsync runs, so the gRPC server
|
||||
// refuses new streams (Unavailable) and cancels every active stream
|
||||
|
||||
@@ -0,0 +1,520 @@
|
||||
using Microsoft.Data.Sqlite;
|
||||
using Microsoft.Extensions.Configuration;
|
||||
using ZB.MOM.WW.LocalDb;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.Host;
|
||||
|
||||
/// <summary>
|
||||
/// One-time copy of the pre-consolidation site databases — Phase 1's
|
||||
/// <c>site-tracking.db</c> and <c>site_events.db</c>, and Phase 2's
|
||||
/// <c>store-and-forward.db</c> — into the consolidated <c>ZB.MOM.WW.LocalDb</c> database.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>Runs after <c>RegisterReplicated</c>, deliberately.</b> Capture is trigger-based, so
|
||||
/// rows inserted before registration would never enter the oplog and would never reach the
|
||||
/// peer. Migrating after registration means the migrated history replicates like any other
|
||||
/// write — which is the entire point of Phase 1.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Idempotent twice over.</b> Tracking rows carry the same TEXT primary key they always
|
||||
/// had, so <c>INSERT OR IGNORE</c> is naturally idempotent. Legacy events had autoincrement
|
||||
/// integer ids, which the consolidated schema replaced with GUIDs; they are given
|
||||
/// <i>deterministic</i> ids of the form <c>mig-{NodeName}-{legacyId}</c> rather than fresh
|
||||
/// GUIDs, so a migration that crashes and re-runs cannot duplicate events. The NodeName
|
||||
/// prefix keeps the two nodes of a pair from colliding on their independent legacy id
|
||||
/// sequences.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>All-or-nothing.</b> The copy runs in one transaction and the legacy file is renamed
|
||||
/// to <c><name>.migrated</c> only after commit. A failure throws out of
|
||||
/// <c>OnReady</c>, which fails host startup with the legacy files untouched — no
|
||||
/// half-migrated state to reason about. A second boot sees the renamed file and no-ops.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Finding nothing is the expected case on the docker rig for the two Phase 1 files.</b>
|
||||
/// Neither legacy config key
|
||||
/// is set in any rig appsettings, so both fall back to CWD-relative code defaults
|
||||
/// (<c>/app/site-tracking.db</c>, <c>/app/site_events.db</c>) that sit OUTSIDE the mounted
|
||||
/// data volume — meaning they were already being discarded on every container recreate.
|
||||
/// Phase 1 incidentally fixes that data-loss bug by consolidating into
|
||||
/// <c>/app/data/site-localdb.db</c>. A no-op here is a legitimate result, not a failure.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Store-and-forward is the exception.</b> Its default path <i>is</i> inside the data
|
||||
/// volume (<c>./data/store-and-forward.db</c>), so a real deployment has a real file there
|
||||
/// holding undelivered messages. That migration genuinely moves data, and dropping it would
|
||||
/// silently discard exactly the buffered calls store-and-forward exists to protect.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public static class SiteLocalDbLegacyMigrator
|
||||
{
|
||||
private const string MigratedSuffix = ".migrated";
|
||||
|
||||
/// <summary>Default legacy tracking connection string (<c>OperationTrackingOptions.ConnectionString</c>).</summary>
|
||||
private const string DefaultTrackingConnectionString = "Data Source=site-tracking.db";
|
||||
|
||||
/// <summary>Default legacy event-log path (<c>SiteEventLogOptions.DatabasePath</c>).</summary>
|
||||
private const string DefaultEventLogPath = "site_events.db";
|
||||
|
||||
/// <summary>Default legacy store-and-forward path (<c>StoreAndForwardOptions.SqliteDbPath</c>).</summary>
|
||||
private const string DefaultStoreAndForwardPath = "./data/store-and-forward.db";
|
||||
|
||||
/// <summary>Default legacy site configuration path (<c>appsettings.Site.json</c>).</summary>
|
||||
private const string DefaultSiteStoragePath = "./data/scadabridge.db";
|
||||
|
||||
/// <summary>
|
||||
/// Every column of the current <c>sf_messages</c> schema, in a fixed order. Columns
|
||||
/// absent from an older legacy file are dropped from the copy rather than failing it —
|
||||
/// see <see cref="PresentColumns"/>.
|
||||
/// </summary>
|
||||
private static readonly string[] StoreAndForwardColumns =
|
||||
[
|
||||
"id", "category", "target", "payload_json",
|
||||
"retry_count", "max_retries", "retry_interval_ms",
|
||||
"created_at", "last_attempt_at", "status", "last_error", "origin_instance",
|
||||
"execution_id", "source_script", "parent_execution_id", "last_attempt_at_ms",
|
||||
];
|
||||
|
||||
/// <summary>
|
||||
/// The site configuration tables copied out of the legacy <c>scadabridge.db</c>, with
|
||||
/// the current schema's columns for each.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <b><c>notification_lists</c> and <c>smtp_configurations</c> are deliberately absent.</b>
|
||||
/// Both are purged on every deploy and are permanently empty by design — the site-side
|
||||
/// write paths were removed on 2026-07-10. A pre-fix legacy file can still hold rows, and
|
||||
/// <c>smtp_configurations.password</c> is plaintext.
|
||||
/// <para>
|
||||
/// Skipping them here is one half of a pair: the cutover also declines to register them
|
||||
/// for replication, for the same reason. Migrating them would leave plaintext SMTP
|
||||
/// passwords sitting in the consolidated database — one future <c>RegisterReplicated</c>
|
||||
/// away from being shipped to a peer — in exchange for resurrecting config that nothing
|
||||
/// reads. Keeping the tables permanently empty is what makes both decisions safe.
|
||||
/// </para>
|
||||
/// The tables themselves are still created (see <c>SiteStorageSchema</c>); only their
|
||||
/// historical contents are left behind.
|
||||
/// </remarks>
|
||||
internal static readonly LegacyTable[] SiteStorageTables =
|
||||
[
|
||||
new("deployed_configurations", "instance_unique_name",
|
||||
[
|
||||
"instance_unique_name", "config_json", "deployment_id", "revision_hash",
|
||||
"is_enabled", "deployed_at",
|
||||
]),
|
||||
new("static_attribute_overrides", "instance_unique_name",
|
||||
[
|
||||
"instance_unique_name", "attribute_name", "override_value", "updated_at",
|
||||
]),
|
||||
new("shared_scripts", "name",
|
||||
[
|
||||
"name", "code", "parameter_definitions", "return_definition", "updated_at",
|
||||
]),
|
||||
new("external_systems", "name",
|
||||
[
|
||||
"name", "endpoint_url", "auth_type", "auth_configuration", "method_definitions",
|
||||
"updated_at", "timeout_seconds",
|
||||
]),
|
||||
new("database_connections", "name",
|
||||
[
|
||||
"name", "connection_string", "max_retries", "retry_delay_ms", "updated_at",
|
||||
]),
|
||||
new("data_connection_definitions", "name",
|
||||
[
|
||||
"name", "protocol", "configuration", "backup_configuration",
|
||||
"failover_retry_count", "updated_at",
|
||||
]),
|
||||
new("native_alarm_state", "instance_unique_name",
|
||||
[
|
||||
"instance_unique_name", "source_canonical_name", "source_reference",
|
||||
"condition_json", "last_transition_at", "metadata_json",
|
||||
]),
|
||||
];
|
||||
|
||||
/// <summary>
|
||||
/// Copies any legacy site databases into <paramref name="db"/>, then renames them.
|
||||
/// </summary>
|
||||
/// <param name="db">The consolidated site database, with both tables already registered.</param>
|
||||
/// <param name="config">Configuration supplying the legacy paths and the node name.</param>
|
||||
public static void Migrate(ILocalDb db, IConfiguration config)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(db);
|
||||
ArgumentNullException.ThrowIfNull(config);
|
||||
|
||||
var nodeName = config["ScadaBridge:Node:NodeName"] ?? "unknown-node";
|
||||
|
||||
MigrateTracking(db, ResolveTrackingPath(config));
|
||||
MigrateEvents(db, ResolveEventLogPath(config), nodeName);
|
||||
MigrateStoreAndForward(db, ResolveStoreAndForwardPath(config));
|
||||
MigrateSiteStorage(db, ResolveSiteStoragePath(config));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Resolves the legacy tracking database path from the OLD connection-string key,
|
||||
/// falling back to the code default. Relative paths resolve against the process working
|
||||
/// directory — NOT the data volume — because that is where the old code actually put them.
|
||||
/// </summary>
|
||||
internal static string ResolveTrackingPath(IConfiguration config)
|
||||
{
|
||||
var connectionString =
|
||||
config["ScadaBridge:OperationTracking:ConnectionString"] ?? DefaultTrackingConnectionString;
|
||||
|
||||
string dataSource;
|
||||
try
|
||||
{
|
||||
dataSource = new SqliteConnectionStringBuilder(connectionString).DataSource;
|
||||
}
|
||||
catch (ArgumentException)
|
||||
{
|
||||
// An unparseable legacy connection string means there is nothing to migrate
|
||||
// from. Do not fail the host over a stale key we are about to stop reading.
|
||||
return string.Empty;
|
||||
}
|
||||
|
||||
// In-memory legacy databases (test/dev configurations) have nothing durable to
|
||||
// migrate, and Path.GetFullPath on them would produce nonsense.
|
||||
if (string.IsNullOrWhiteSpace(dataSource) ||
|
||||
dataSource.Equals(":memory:", StringComparison.OrdinalIgnoreCase) ||
|
||||
dataSource.StartsWith("file:", StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
return string.Empty;
|
||||
}
|
||||
|
||||
return Path.GetFullPath(dataSource);
|
||||
}
|
||||
|
||||
/// <summary>Resolves the legacy event-log path from the OLD key, falling back to the code default.</summary>
|
||||
internal static string ResolveEventLogPath(IConfiguration config)
|
||||
{
|
||||
var path = config["ScadaBridge:SiteEventLog:DatabasePath"] ?? DefaultEventLogPath;
|
||||
|
||||
if (string.IsNullOrWhiteSpace(path) ||
|
||||
path.Equals(":memory:", StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
return string.Empty;
|
||||
}
|
||||
|
||||
return Path.GetFullPath(path);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Resolves the legacy store-and-forward database path from the OLD key, falling back
|
||||
/// to the code default. Unlike the two Phase 1 paths, this default sits INSIDE the
|
||||
/// mounted data volume (<c>./data/</c>), so on the docker rig there is a real file here
|
||||
/// with real buffered messages — this migration is not the usual no-op.
|
||||
/// </summary>
|
||||
internal static string ResolveStoreAndForwardPath(IConfiguration config)
|
||||
{
|
||||
var path = config["ScadaBridge:StoreAndForward:SqliteDbPath"] ?? DefaultStoreAndForwardPath;
|
||||
|
||||
if (string.IsNullOrWhiteSpace(path) ||
|
||||
path.Equals(":memory:", StringComparison.OrdinalIgnoreCase) ||
|
||||
path.StartsWith("file:", StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
return string.Empty;
|
||||
}
|
||||
|
||||
return Path.GetFullPath(path);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Resolves the legacy site configuration database path from the OLD key, falling back
|
||||
/// to the code default. Like store-and-forward — and unlike the two Phase 1 paths — this
|
||||
/// default is inside the mounted data volume, so a real deployment has real config here.
|
||||
/// </summary>
|
||||
internal static string ResolveSiteStoragePath(IConfiguration config)
|
||||
{
|
||||
var path = config["ScadaBridge:Database:SiteDbPath"] ?? DefaultSiteStoragePath;
|
||||
|
||||
if (string.IsNullOrWhiteSpace(path) ||
|
||||
path.Equals(":memory:", StringComparison.OrdinalIgnoreCase) ||
|
||||
path.StartsWith("file:", StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
return string.Empty;
|
||||
}
|
||||
|
||||
return Path.GetFullPath(path);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Copies buffered store-and-forward messages out of the legacy file.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// No id synthesis, unlike <see cref="MigrateEvents"/>: <c>sf_messages.id</c> is already
|
||||
/// a caller-assigned TEXT primary key, so <c>INSERT OR IGNORE</c> is naturally idempotent
|
||||
/// across a crash-then-rerun.
|
||||
/// <para>
|
||||
/// These are undelivered messages, so dropping them is real data loss — a buffered call
|
||||
/// that never reaches its external system is exactly what store-and-forward exists to
|
||||
/// prevent. That is why the copy tolerates an older column set rather than bailing.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
private static void MigrateStoreAndForward(ILocalDb db, string legacyPath)
|
||||
=> MigrateFile(db, legacyPath, [new LegacyTable("sf_messages", "id", StoreAndForwardColumns)]);
|
||||
|
||||
/// <summary>
|
||||
/// Copies the site's configuration tables out of the legacy <c>scadabridge.db</c>.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// All seven migrated tables live in one file, so they are copied inside a single
|
||||
/// transaction and the file is renamed once: a partial config migration would leave a
|
||||
/// site node running against half its old configuration, which is worse than failing
|
||||
/// startup outright.
|
||||
/// </remarks>
|
||||
private static void MigrateSiteStorage(ILocalDb db, string legacyPath)
|
||||
=> MigrateFile(db, legacyPath, SiteStorageTables);
|
||||
|
||||
/// <summary>
|
||||
/// Copies one table from a legacy file into the consolidated database, then renames the
|
||||
/// legacy file.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The copy is restricted to the columns the legacy table <i>actually has</i>. A file
|
||||
/// written by an older build predates some columns, and naming a missing column in the
|
||||
/// SELECT would throw "no such column" — which the reader treats as an unrecognised
|
||||
/// shape, silently discarding every row in the table. Intersecting first means an old
|
||||
/// file migrates its data and simply leaves the newer columns at their schema defaults.
|
||||
/// </remarks>
|
||||
/// <param name="db">The consolidated database.</param>
|
||||
/// <param name="legacyPath">The legacy file, which may not exist.</param>
|
||||
/// <param name="tables">Every table to copy out of this file, in order.</param>
|
||||
private static void MigrateFile(ILocalDb db, string legacyPath, IReadOnlyList<LegacyTable> tables)
|
||||
{
|
||||
if (!ShouldMigrate(legacyPath)) return;
|
||||
|
||||
using (var legacy = OpenLegacyReadOnly(legacyPath))
|
||||
{
|
||||
using var connection = db.CreateConnection();
|
||||
using var transaction = connection.BeginTransaction();
|
||||
|
||||
foreach (var table in tables)
|
||||
{
|
||||
var present = PresentColumns(legacy, table.Table, table.Columns);
|
||||
|
||||
// An absent table probes as zero columns, so this one guard covers both
|
||||
// "old file predating the table" and "file we do not recognise".
|
||||
if (present.Contains(table.RequiredColumn))
|
||||
CopyRows(legacy, connection, transaction, table.Table, present);
|
||||
}
|
||||
|
||||
transaction.Commit();
|
||||
}
|
||||
|
||||
MarkMigrated(legacyPath);
|
||||
}
|
||||
|
||||
/// <summary>One table to copy out of a legacy file.</summary>
|
||||
/// <param name="Table">Table name, identical on both sides.</param>
|
||||
/// <param name="RequiredColumn">
|
||||
/// A column without which the table is not the one we mean — normally the primary key.
|
||||
/// Copying rows with a NULL PK would be worse than copying nothing.
|
||||
/// </param>
|
||||
/// <param name="Columns">The current schema's full column list, in a fixed order.</param>
|
||||
internal sealed record LegacyTable(string Table, string RequiredColumn, string[] Columns);
|
||||
|
||||
private static void MigrateTracking(ILocalDb db, string legacyPath)
|
||||
{
|
||||
if (!ShouldMigrate(legacyPath)) return;
|
||||
|
||||
var rows = ReadAll(
|
||||
legacyPath,
|
||||
"""
|
||||
SELECT TrackedOperationId, Kind, TargetSummary, Status,
|
||||
RetryCount, LastError, HttpStatus,
|
||||
CreatedAtUtc, UpdatedAtUtc, TerminalAtUtc,
|
||||
SourceInstanceId, SourceScript, SourceNode
|
||||
FROM OperationTracking;
|
||||
""",
|
||||
expectedColumns: 13);
|
||||
|
||||
// A legacy file with no OperationTracking table at all reads as "nothing to do";
|
||||
// it still gets renamed so the probe does not repeat on every boot.
|
||||
if (rows is not null)
|
||||
{
|
||||
using var connection = db.CreateConnection();
|
||||
using var transaction = connection.BeginTransaction();
|
||||
|
||||
foreach (var row in rows)
|
||||
{
|
||||
using var cmd = connection.CreateCommand();
|
||||
cmd.Transaction = transaction;
|
||||
cmd.CommandText = """
|
||||
INSERT OR IGNORE INTO OperationTracking (
|
||||
TrackedOperationId, Kind, TargetSummary, Status,
|
||||
RetryCount, LastError, HttpStatus,
|
||||
CreatedAtUtc, UpdatedAtUtc, TerminalAtUtc,
|
||||
SourceInstanceId, SourceScript, SourceNode
|
||||
) VALUES (
|
||||
$p0, $p1, $p2, $p3, $p4, $p5, $p6,
|
||||
$p7, $p8, $p9, $p10, $p11, $p12
|
||||
);
|
||||
""";
|
||||
Bind(cmd, row);
|
||||
cmd.ExecuteNonQuery();
|
||||
}
|
||||
|
||||
transaction.Commit();
|
||||
}
|
||||
|
||||
MarkMigrated(legacyPath);
|
||||
}
|
||||
|
||||
private static void MigrateEvents(ILocalDb db, string legacyPath, string nodeName)
|
||||
{
|
||||
if (!ShouldMigrate(legacyPath)) return;
|
||||
|
||||
var rows = ReadAll(
|
||||
legacyPath,
|
||||
"""
|
||||
SELECT id, timestamp, event_type, severity, instance_id, source, message, details
|
||||
FROM site_events;
|
||||
""",
|
||||
expectedColumns: 8);
|
||||
|
||||
if (rows is not null)
|
||||
{
|
||||
using var connection = db.CreateConnection();
|
||||
using var transaction = connection.BeginTransaction();
|
||||
|
||||
foreach (var row in rows)
|
||||
{
|
||||
using var cmd = connection.CreateCommand();
|
||||
cmd.Transaction = transaction;
|
||||
cmd.CommandText = """
|
||||
INSERT OR IGNORE INTO site_events (
|
||||
id, timestamp, event_type, severity, instance_id, source, message, details
|
||||
) VALUES (
|
||||
$id, $p1, $p2, $p3, $p4, $p5, $p6, $p7
|
||||
);
|
||||
""";
|
||||
|
||||
// Deterministic id, NOT a fresh GUID: a crashed-then-rerun migration must
|
||||
// not duplicate events, and INSERT OR IGNORE can only dedupe on a stable key.
|
||||
cmd.Parameters.AddWithValue("$id", $"mig-{nodeName}-{row[0] ?? "null"}");
|
||||
for (var i = 1; i < row.Length; i++)
|
||||
cmd.Parameters.AddWithValue($"$p{i}", row[i] ?? (object)DBNull.Value);
|
||||
|
||||
cmd.ExecuteNonQuery();
|
||||
}
|
||||
|
||||
transaction.Commit();
|
||||
}
|
||||
|
||||
MarkMigrated(legacyPath);
|
||||
}
|
||||
|
||||
/// <summary>True when there is a legacy file present that has not already been migrated.</summary>
|
||||
private static bool ShouldMigrate(string legacyPath)
|
||||
=> !string.IsNullOrEmpty(legacyPath)
|
||||
&& File.Exists(legacyPath)
|
||||
&& !File.Exists(legacyPath + MigratedSuffix);
|
||||
|
||||
/// <summary>
|
||||
/// Reads every row of <paramref name="sql"/> from the legacy file, or null when the
|
||||
/// table does not exist (an old file predating the table is not an error).
|
||||
/// </summary>
|
||||
private static List<object?[]>? ReadAll(string legacyPath, string sql, int expectedColumns)
|
||||
{
|
||||
using var connection = new SqliteConnection($"Data Source={legacyPath};Mode=ReadOnly");
|
||||
connection.Open();
|
||||
|
||||
using var cmd = connection.CreateCommand();
|
||||
cmd.CommandText = sql;
|
||||
|
||||
SqliteDataReader reader;
|
||||
try
|
||||
{
|
||||
reader = cmd.ExecuteReader();
|
||||
}
|
||||
catch (SqliteException)
|
||||
{
|
||||
// "no such table" / "no such column" — a legacy file from before this table
|
||||
// existed, or a shape we do not recognise. Nothing to copy.
|
||||
return null;
|
||||
}
|
||||
|
||||
var rows = new List<object?[]>();
|
||||
using (reader)
|
||||
{
|
||||
while (reader.Read())
|
||||
{
|
||||
var row = new object?[expectedColumns];
|
||||
for (var i = 0; i < expectedColumns; i++)
|
||||
row[i] = reader.IsDBNull(i) ? null : reader.GetValue(i);
|
||||
rows.Add(row);
|
||||
}
|
||||
}
|
||||
|
||||
return rows;
|
||||
}
|
||||
|
||||
private static SqliteConnection OpenLegacyReadOnly(string legacyPath)
|
||||
{
|
||||
var connection = new SqliteConnection($"Data Source={legacyPath};Mode=ReadOnly");
|
||||
connection.Open();
|
||||
return connection;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Returns the subset of <paramref name="wanted"/> that the legacy table actually has,
|
||||
/// in the caller's order. An absent table yields an empty list rather than throwing.
|
||||
/// </summary>
|
||||
private static List<string> PresentColumns(
|
||||
SqliteConnection legacy, string table, IReadOnlyList<string> wanted)
|
||||
{
|
||||
var present = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
|
||||
|
||||
using (var probe = legacy.CreateCommand())
|
||||
{
|
||||
// Table name is a caller-controlled constant, never user input — safe to
|
||||
// interpolate (parameters are not permitted as a pragma-function argument).
|
||||
probe.CommandText = $"SELECT name FROM pragma_table_info('{table}')";
|
||||
using var reader = probe.ExecuteReader();
|
||||
while (reader.Read()) present.Add(reader.GetString(0));
|
||||
}
|
||||
|
||||
return [.. wanted.Where(present.Contains)];
|
||||
}
|
||||
|
||||
/// <summary>Streams every row of <paramref name="columns"/> from the legacy table into the target.</summary>
|
||||
private static void CopyRows(
|
||||
SqliteConnection legacy,
|
||||
SqliteConnection target,
|
||||
SqliteTransaction transaction,
|
||||
string table,
|
||||
IReadOnlyList<string> columns)
|
||||
{
|
||||
var columnList = string.Join(", ", columns);
|
||||
var parameterList = string.Join(", ", columns.Select((_, i) => $"$p{i}"));
|
||||
|
||||
using var read = legacy.CreateCommand();
|
||||
read.CommandText = $"SELECT {columnList} FROM {table};";
|
||||
using var reader = read.ExecuteReader();
|
||||
|
||||
while (reader.Read())
|
||||
{
|
||||
using var write = target.CreateCommand();
|
||||
write.Transaction = transaction;
|
||||
write.CommandText =
|
||||
$"INSERT OR IGNORE INTO {table} ({columnList}) VALUES ({parameterList});";
|
||||
|
||||
for (var i = 0; i < columns.Count; i++)
|
||||
write.Parameters.AddWithValue($"$p{i}", reader.IsDBNull(i) ? DBNull.Value : reader.GetValue(i));
|
||||
|
||||
write.ExecuteNonQuery();
|
||||
}
|
||||
}
|
||||
|
||||
private static void Bind(SqliteCommand cmd, object?[] row)
|
||||
{
|
||||
for (var i = 0; i < row.Length; i++)
|
||||
cmd.Parameters.AddWithValue($"$p{i}", row[i] ?? (object)DBNull.Value);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Renames the legacy file so a later boot skips it. Done only after the copy has
|
||||
/// committed; the file is kept rather than deleted so an operator can still inspect it.
|
||||
/// </summary>
|
||||
private static void MarkMigrated(string legacyPath)
|
||||
=> File.Move(legacyPath, legacyPath + MigratedSuffix);
|
||||
}
|
||||
@@ -0,0 +1,91 @@
|
||||
using Microsoft.Extensions.Configuration;
|
||||
using ZB.MOM.WW.LocalDb;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteEventLogging;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Tracking;
|
||||
using ZB.MOM.WW.ScadaBridge.StoreAndForward;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.Host;
|
||||
|
||||
/// <summary>
|
||||
/// Initializes the consolidated site database — the single <c>ZB.MOM.WW.LocalDb</c>-managed
|
||||
/// file that holds the site node's replicated state.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>Ordering is load-bearing.</b> Table DDL must run BEFORE
|
||||
/// <c>RegisterReplicated</c>, and every row that should replicate must be written
|
||||
/// AFTER it. Capture is trigger-based change-data-capture: the library neither
|
||||
/// captures nor snapshots rows that already existed when the triggers were installed,
|
||||
/// so a row written before registration is invisible to the peer forever — silently,
|
||||
/// with no error anywhere.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Runs inside the <c>AddZbLocalDb</c> onReady callback, once, before any caller
|
||||
/// receives the <see cref="ILocalDb"/> singleton. onReady is a synchronous
|
||||
/// <c>Action<ILocalDb></c>, hence the direct <c>CreateConnection()</c> rather than
|
||||
/// blocking on the async execute API. A throw here propagates out of the first
|
||||
/// <c>GetRequiredService<ILocalDb>()</c> and fails host startup, which is the
|
||||
/// intent: a site node that cannot establish its schema must not come up half-working.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public static class SiteLocalDbSetup
|
||||
{
|
||||
/// <summary>
|
||||
/// Creates the site node's replicated tables, opts them into change capture, and
|
||||
/// migrates any pre-Phase-1 databases in.
|
||||
/// </summary>
|
||||
/// <param name="db">The LocalDb instance being initialized.</param>
|
||||
/// <param name="config">Configuration, for the legacy migrator's old path keys and node name.</param>
|
||||
public static void OnReady(ILocalDb db, IConfiguration config)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(db);
|
||||
ArgumentNullException.ThrowIfNull(config);
|
||||
|
||||
using (var connection = db.CreateConnection())
|
||||
{
|
||||
OperationTrackingSchema.Apply(connection);
|
||||
SiteEventLogSchema.Apply(connection);
|
||||
|
||||
// Phase 2: the site's configuration tables and the store-and-forward buffer
|
||||
// now live in this file too.
|
||||
SiteStorageSchema.Apply(connection);
|
||||
StoreAndForwardSchema.Apply(connection);
|
||||
}
|
||||
|
||||
// Both tables qualify: each has an explicit primary key (RegisterReplicated
|
||||
// rejects tables without one) and no BLOB columns (which json_object cannot
|
||||
// capture). Registration is idempotent and installs the capture triggers.
|
||||
db.RegisterReplicated("OperationTracking");
|
||||
db.RegisterReplicated("site_events");
|
||||
|
||||
// Phase 2: the store-and-forward buffer and the seven site configuration tables.
|
||||
// These replaced the bespoke SiteReplicationActor and StoreAndForward
|
||||
// ReplicationService, which shipped hand-written Add/Remove/Park/Requeue operations
|
||||
// over Akka; both were deleted in the same commit that added these lines, so the
|
||||
// two mechanisms never ran at once.
|
||||
//
|
||||
// Both composite-PK tables are fine: RegisterReplicated orders multi-column PKs by
|
||||
// ordinal. No Phase 2 table has a BLOB column, which it would reject.
|
||||
db.RegisterReplicated("sf_messages");
|
||||
db.RegisterReplicated("deployed_configurations");
|
||||
db.RegisterReplicated("static_attribute_overrides");
|
||||
db.RegisterReplicated("shared_scripts");
|
||||
db.RegisterReplicated("external_systems");
|
||||
db.RegisterReplicated("database_connections");
|
||||
db.RegisterReplicated("data_connection_definitions");
|
||||
db.RegisterReplicated("native_alarm_state");
|
||||
|
||||
// notification_lists and smtp_configurations are created but deliberately NOT
|
||||
// registered. They are permanently empty by design — the site-side write paths were
|
||||
// removed on 2026-07-10, the legacy migrator skips them, and the active node's
|
||||
// artifact apply purges them on every deploy. Registering them would open a standing
|
||||
// replication channel whose only historical payload was plaintext SMTP passwords, in
|
||||
// exchange for replicating nothing. Anyone adding them here should first establish
|
||||
// that a site has a legitimate reason to hold SMTP credentials at all.
|
||||
|
||||
// AFTER registration, so migrated rows enter the oplog and reach the peer like
|
||||
// any other write. Before it, they would be invisible to replication forever.
|
||||
SiteLocalDbLegacyMigrator.Migrate(db, config);
|
||||
}
|
||||
}
|
||||
@@ -14,6 +14,8 @@ using ZB.MOM.WW.ScadaBridge.NotificationService;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteEventLogging;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteRuntime;
|
||||
using ZB.MOM.WW.ScadaBridge.StoreAndForward;
|
||||
using ZB.MOM.WW.LocalDb;
|
||||
using ZB.MOM.WW.LocalDb.Replication;
|
||||
using ZB.MOM.WW.Secrets.DependencyInjection;
|
||||
using ZB.MOM.WW.Telemetry;
|
||||
|
||||
@@ -25,6 +27,24 @@ namespace ZB.MOM.WW.ScadaBridge.Host;
|
||||
/// </summary>
|
||||
public static class SiteServiceRegistration
|
||||
{
|
||||
/// <summary>
|
||||
/// Every <see cref="System.Diagnostics.Metrics.Meter"/> OpenTelemetry is told to observe.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <c>ZbTelemetryOptions.Meters</c> is an ALLOWLIST, and an unlisted meter's instruments
|
||||
/// are simply never observed — no error, no warning, no missing-metric signal anywhere.
|
||||
/// The LocalDb replication meter was silently absent from the rig's <c>/metrics</c>
|
||||
/// scrape until the Phase 1 live gate went looking for it (the plan had assumed it
|
||||
/// flowed through automatically). Hoisted to a named constant so the list is
|
||||
/// assertable; adding a meter anywhere in the product means adding it here.
|
||||
/// </remarks>
|
||||
public static readonly string[] ObservedMeters =
|
||||
[
|
||||
ScadaBridgeTelemetry.MeterName,
|
||||
// Emitted only on site nodes; harmless on central, which never records against it.
|
||||
LocalDbMetrics.MeterName,
|
||||
];
|
||||
|
||||
/// <summary>Registers all DI services required for the site role.</summary>
|
||||
/// <param name="services">The service collection to register into.</param>
|
||||
/// <param name="config">Application configuration for options binding.</param>
|
||||
@@ -45,10 +65,43 @@ public static class SiteServiceRegistration
|
||||
services.AddSingleton<ISiteIdentityProvider, SiteIdentityProvider>();
|
||||
services.AddSingleton<IHealthReportTransport, AkkaHealthReportTransport>();
|
||||
|
||||
// Site-only components — AddSiteRuntime registers SiteStorageService with SQLite path
|
||||
// and site-local repository implementations (IExternalSystemRepository, INotificationRepository)
|
||||
var siteDbPath = config["ScadaBridge:Database:SiteDbPath"] ?? "site.db";
|
||||
services.AddSiteRuntime($"Data Source={siteDbPath}");
|
||||
// Site-only components — AddSiteRuntime registers SiteStorageService and the
|
||||
// site-local repository implementations (IExternalSystemRepository,
|
||||
// INotificationRepository). It takes no connection string any more:
|
||||
// SiteStorageService persists to the consolidated LocalDb database registered
|
||||
// just below (LocalDb:Path). ScadaBridge:Database:SiteDbPath survives only as
|
||||
// the legacy migrator's source location.
|
||||
services.AddSiteRuntime();
|
||||
|
||||
// Consolidated site database (LocalDb Phase 1). Holds OperationTracking and
|
||||
// site_events as replicated tables so the pair stops losing them on failover.
|
||||
//
|
||||
// Storage is registered UNCONDITIONALLY and needs no flag: with no peer
|
||||
// configured, LocalDb is simply a local SQLite file and the replication
|
||||
// initiator idles.
|
||||
//
|
||||
// Design: scadaproj docs/plans/2026-07-19-scadabridge-localdb-design.md
|
||||
//
|
||||
// The parent directory of LocalDb:Path is created by the library as of 0.1.1.
|
||||
// ScadaBridge carried a SiteLocalDbDirectory shim for it during Phase 2, because
|
||||
// SQLite creates the database file on demand but not its directory and
|
||||
// SqliteLocalDb opens the file eagerly — a missing directory was a hard boot
|
||||
// failure, and the default site path is the relative "./data/site-localdb.db".
|
||||
// The shim is gone; SiteLocalDbDirectoryTests still pins the outcome here, since
|
||||
// what this host needs is the guarantee, not any particular owner of it.
|
||||
services.AddZbLocalDb(config, db => SiteLocalDbSetup.OnReady(db, config));
|
||||
|
||||
// The replication engine, likewise unconditional but INERT by default: with no
|
||||
// LocalDb:Replication:PeerAddress the initiating SyncBackgroundService starts and
|
||||
// immediately idles, and nothing dials the passive endpoint. A site pair
|
||||
// replicates only once an operator sets a peer on both nodes.
|
||||
//
|
||||
// Registered HERE rather than in Program.cs (where the plan first put it) so the
|
||||
// composition-root tests, which build this graph and not Program.cs, actually
|
||||
// cover it. The endpoint half — MapZbLocalDbSync — has to stay in Program.cs
|
||||
// because it needs the WebApplication.
|
||||
services.AddZbLocalDbReplication(config);
|
||||
|
||||
services.AddDataConnectionLayer();
|
||||
// Local SQLite store by default; a local store replicating against a shared SQL-Server hub
|
||||
// only when Secrets:Replication:Enabled is true AND a hub connection string is present.
|
||||
@@ -76,6 +129,16 @@ public static class SiteServiceRegistration
|
||||
services.AddSiteEventLogHealthMetricsBridge(
|
||||
sp => () => sp.GetRequiredService<ISiteEventLogger>().FailedWriteCount);
|
||||
|
||||
// LocalDb replication — bridge ISyncStatus onto the site health report. Registered
|
||||
// unconditionally alongside the engine: on a node with no peer this reports
|
||||
// Connected=false with a real 0 backlog, which is the healthy default-OFF state.
|
||||
// OplogBacklog is passed through NULLABLE on purpose — null means the poll failed
|
||||
// or no provider is wired, and flattening it to 0 would render a replication pair
|
||||
// that cannot read its own oplog as perfectly healthy.
|
||||
services.AddLocalDbReplicationHealthBridge(
|
||||
sp => () => sp.GetRequiredService<ISyncStatus>().Connected,
|
||||
sp => () => sp.GetRequiredService<ISyncStatus>().OplogBacklog);
|
||||
|
||||
// Audit Log — site-side hot-path writer + telemetry collaborators.
|
||||
// The SiteAuditTelemetryActor itself is registered by AkkaHostedService
|
||||
// in the site-role block; this call wires every DI dependency it (and
|
||||
@@ -228,7 +291,7 @@ public static class SiteServiceRegistration
|
||||
o.ServiceName = "scadabridge";
|
||||
o.SiteId = config["ScadaBridge:Node:SiteId"] ?? "central";
|
||||
o.NodeRole = config["ScadaBridge:Node:Role"];
|
||||
o.Meters = [ScadaBridgeTelemetry.MeterName];
|
||||
o.Meters = ObservedMeters;
|
||||
if (Enum.TryParse<ZbExporter>(config["ScadaBridge:Telemetry:Exporter"], ignoreCase: true, out var exporter))
|
||||
o.Exporter = exporter;
|
||||
var otlp = config["ScadaBridge:Telemetry:OtlpEndpoint"];
|
||||
|
||||
@@ -94,7 +94,7 @@ public static class StartupValidator
|
||||
_ => seedNodes != null && seedNodes.Count >= 2,
|
||||
"must have at least 2 entries")
|
||||
// The big Site-only block: GrpcPort/MetricsPort validity + cross-field
|
||||
// collisions + SiteDbPath + seed-node-port loop, in the original order.
|
||||
// collisions + seed-node-port loop, in the original order.
|
||||
.When(role == "Site", p =>
|
||||
{
|
||||
// GrpcPort range, then GrpcPort vs RemotingPort.
|
||||
@@ -110,9 +110,12 @@ public static class StartupValidator
|
||||
p.Require("ScadaBridge:Node:MetricsPort", _ => metricsPort != port, "must differ from RemotingPort");
|
||||
p.Require("ScadaBridge:Node:MetricsPort", _ => metricsPort != grpcPort, "must differ from GrpcPort");
|
||||
|
||||
p.Require("ScadaBridge:Database:SiteDbPath",
|
||||
_ => !string.IsNullOrEmpty(configuration.GetSection("ScadaBridge:Database")["SiteDbPath"]),
|
||||
"required for Site nodes");
|
||||
// ScadaBridge:Database:SiteDbPath was required here until LocalDb
|
||||
// Phase 2. The site's tables now live in the consolidated LocalDb
|
||||
// database (LocalDb:Path, which SiteServiceRegistration requires),
|
||||
// and SiteDbPath survives only as the legacy migration source — so
|
||||
// its absence means "nothing to migrate", not a misconfiguration.
|
||||
// DatabaseOptionsValidator still rejects a present-but-blank value.
|
||||
|
||||
// A seed node must reference an Akka.Remote endpoint, never the
|
||||
// Kestrel HTTP/2 gRPC port. A seed entry whose port equals this node's
|
||||
|
||||
@@ -41,6 +41,8 @@
|
||||
<PackageReference Include="ZB.MOM.WW.Secrets.Abstractions" />
|
||||
<PackageReference Include="ZB.MOM.WW.Secrets.Ui" />
|
||||
<PackageReference Include="ZB.MOM.WW.Secrets.Replicator.SqlServer" />
|
||||
<PackageReference Include="ZB.MOM.WW.LocalDb" />
|
||||
<PackageReference Include="ZB.MOM.WW.LocalDb.Replication" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
|
||||
@@ -23,6 +23,10 @@
|
||||
"MinNrOfMembers": 1
|
||||
},
|
||||
"Database": {
|
||||
// Migration-only as of LocalDb Phase 2. The site config tables now live in the
|
||||
// consolidated LocalDb database (LocalDb:Path). SiteDbPath is read once at boot to drain
|
||||
// a pre-Phase-2 scadabridge.db, and is unused after that - keep it until this node has
|
||||
// started once.
|
||||
"SiteDbPath": "./data/scadabridge.db"
|
||||
},
|
||||
"DataConnection": {
|
||||
@@ -31,8 +35,11 @@
|
||||
"WriteTimeout": "00:00:30"
|
||||
},
|
||||
"StoreAndForward": {
|
||||
"SqliteDbPath": "./data/store-and-forward.db",
|
||||
"ReplicationEnabled": true
|
||||
// Migration-only as of LocalDb Phase 2. The store-and-forward buffer now lives in the
|
||||
// consolidated LocalDb database (LocalDb:Path) as the replicated sf_messages table.
|
||||
// SqliteDbPath is read once at boot by SiteLocalDbLegacyMigrator to drain a pre-Phase-2
|
||||
// file, and is unused after that - keep it until this node has started once.
|
||||
"SqliteDbPath": "./data/store-and-forward.db"
|
||||
},
|
||||
"Communication": {
|
||||
"_centralContactPoints": "Host-016: each entry MUST be a central node's remoting endpoint, NOT this site's own remoting port. The single dev-loopback default below points only at central-a (localhost:8081). In a multi-central deployment add the second central node here (e.g. 'akka.tcp://scadabridge@central-b-host:8081') so ClusterClient can fail over when central-a is down. The previous template listed localhost:8082 as the second contact — that is THIS site's own RemotingPort and is a permanent failure in the initial-contact rotation.",
|
||||
@@ -58,5 +65,11 @@
|
||||
"Logging": {
|
||||
"MinimumLevel": "Information"
|
||||
}
|
||||
},
|
||||
// Consolidated site database (LocalDb Phase 1): OperationTracking + site_events.
|
||||
// LocalDb:Path is REQUIRED and validated on start - a site node with no value here
|
||||
// fails to boot, so every site config must set it.
|
||||
"LocalDb": {
|
||||
"Path": "./data/site-localdb.db"
|
||||
}
|
||||
}
|
||||
|
||||
@@ -159,9 +159,16 @@ public class EventLogPurgeService : BackgroundService
|
||||
var deleted = _eventLogger.WithConnection(connection =>
|
||||
{
|
||||
using var cmd = connection.CreateCommand();
|
||||
// Order by timestamp, NOT by id. The id was a monotonic autoincrement
|
||||
// when this was written, so "lowest id" meant "oldest"; since LocalDb
|
||||
// Phase 1 it is a GUID and ordering by it would delete an ARBITRARY
|
||||
// batch of events rather than the oldest ones. id remains in the ORDER
|
||||
// BY only as a tie-break so the batch is deterministic.
|
||||
cmd.CommandText = $"""
|
||||
DELETE FROM site_events WHERE id IN (
|
||||
SELECT id FROM site_events ORDER BY id ASC LIMIT {CapPurgeBatchSize}
|
||||
SELECT id FROM site_events
|
||||
ORDER BY timestamp ASC, id ASC
|
||||
LIMIT {CapPurgeBatchSize}
|
||||
)
|
||||
""";
|
||||
var rows = cmd.ExecuteNonQuery();
|
||||
|
||||
@@ -48,6 +48,52 @@ public class EventLogQueryService : IEventLogQueryService
|
||||
.Replace("_", "\\_");
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Separator between the timestamp and id halves of a continuation token. Chosen
|
||||
/// because it cannot occur in an ISO 8601 "o" timestamp or in a GUID "N" string,
|
||||
/// so the split is unambiguous.
|
||||
/// </summary>
|
||||
private const char TokenSeparator = '|';
|
||||
|
||||
/// <summary>
|
||||
/// Builds the opaque keyset cursor for the last row of a page. The token carries
|
||||
/// the exact stored timestamp string (not a re-formatted <see cref="DateTimeOffset"/>)
|
||||
/// so it compares byte-for-byte against the column under SQLite's BINARY collation.
|
||||
/// </summary>
|
||||
private static string FormatContinuationToken(EventLogEntry last) =>
|
||||
string.Concat(
|
||||
last.Timestamp.ToUniversalTime().ToString("o", CultureInfo.InvariantCulture),
|
||||
TokenSeparator,
|
||||
last.Id);
|
||||
|
||||
/// <summary>
|
||||
/// Parses a continuation token back into its timestamp and id halves.
|
||||
/// Returns <see langword="false"/> for null, empty, or malformed tokens — including
|
||||
/// a legacy numeric token from a client that predates the GUID id change — in which
|
||||
/// case the caller starts from the beginning instead of failing the query.
|
||||
/// </summary>
|
||||
private static bool TryParseContinuationToken(
|
||||
string? token, out string timestamp, out string id)
|
||||
{
|
||||
timestamp = "";
|
||||
id = "";
|
||||
|
||||
if (string.IsNullOrWhiteSpace(token))
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
var separator = token.IndexOf(TokenSeparator);
|
||||
if (separator <= 0 || separator == token.Length - 1)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
timestamp = token[..separator];
|
||||
id = token[(separator + 1)..];
|
||||
return true;
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public EventLogQueryResponse ExecuteQuery(EventLogQueryRequest request)
|
||||
{
|
||||
@@ -64,11 +110,23 @@ public class EventLogQueryService : IEventLogQueryService
|
||||
var whereClauses = new List<string>();
|
||||
var parameters = new List<SqliteParameter>();
|
||||
|
||||
// Keyset pagination: only return events with id > continuation token
|
||||
if (request.ContinuationToken.HasValue)
|
||||
// Keyset pagination on the composite (timestamp, id) key.
|
||||
//
|
||||
// This used to be a plain "id > $afterId" against a monotonic autoincrement
|
||||
// id. Since LocalDb Phase 1 the id is a GUID, which does not sort
|
||||
// chronologically — a naive "id > x" would return an arbitrary subset and
|
||||
// SILENTLY DROP ROWS from the page-through. Timestamps alone are not unique
|
||||
// (two events can share a tick), so the tie is broken on id to guarantee a
|
||||
// total order and exactly-once paging.
|
||||
//
|
||||
// A malformed token is ignored rather than throwing: an old long-typed token
|
||||
// from a client mid-upgrade degrades to "start from the beginning", which is
|
||||
// a visible repeat rather than silent data loss.
|
||||
if (TryParseContinuationToken(request.ContinuationToken, out var afterTs, out var afterId))
|
||||
{
|
||||
whereClauses.Add("id > $afterId");
|
||||
parameters.Add(new SqliteParameter("$afterId", request.ContinuationToken.Value));
|
||||
whereClauses.Add("(timestamp > $afterTs OR (timestamp = $afterTs AND id > $afterId))");
|
||||
parameters.Add(new SqliteParameter("$afterTs", afterTs));
|
||||
parameters.Add(new SqliteParameter("$afterId", afterId));
|
||||
}
|
||||
|
||||
if (request.From.HasValue)
|
||||
@@ -138,7 +196,7 @@ public class EventLogQueryService : IEventLogQueryService
|
||||
SELECT id, timestamp, event_type, severity, instance_id, source, message, details
|
||||
FROM site_events
|
||||
{whereClause}
|
||||
ORDER BY id ASC
|
||||
ORDER BY timestamp ASC, id ASC
|
||||
LIMIT $limit
|
||||
""";
|
||||
cmd.Parameters.AddWithValue("$limit", pageSize + 1);
|
||||
@@ -150,7 +208,7 @@ public class EventLogQueryService : IEventLogQueryService
|
||||
while (reader.Read())
|
||||
{
|
||||
rows.Add(new EventLogEntry(
|
||||
Id: reader.GetInt64(0),
|
||||
Id: reader.GetString(0),
|
||||
// Parse with explicit invariant culture and round-trip style.
|
||||
// Stored values are ISO 8601 "o" UTC
|
||||
// (see SiteEventLogger.LogEventAsync), and the recorder's
|
||||
@@ -178,7 +236,9 @@ public class EventLogQueryService : IEventLogQueryService
|
||||
entries.RemoveAt(entries.Count - 1);
|
||||
}
|
||||
|
||||
var continuationToken = entries.Count > 0 ? entries[^1].Id : (long?)null;
|
||||
var continuationToken = entries.Count > 0
|
||||
? FormatContinuationToken(entries[^1])
|
||||
: null;
|
||||
|
||||
return new EventLogQueryResponse(
|
||||
CorrelationId: request.CorrelationId,
|
||||
|
||||
@@ -6,7 +6,14 @@ public class SiteEventLogOptions
|
||||
public int RetentionDays { get; set; } = 30;
|
||||
/// <summary>Maximum SQLite database size in megabytes before old entries are purged; default 1024 MB.</summary>
|
||||
public int MaxStorageMb { get; set; } = 1024;
|
||||
/// <summary>File path for the site event log SQLite database.</summary>
|
||||
/// <summary>
|
||||
/// MIGRATION-ONLY as of LocalDb Phase 1. <see cref="SiteEventLogger"/> no longer reads
|
||||
/// this: <c>site_events</c> lives in the consolidated site database at
|
||||
/// <c>LocalDb:Path</c>. The only remaining consumer is
|
||||
/// <c>SiteLocalDbLegacyMigrator</c>, which uses it to locate a pre-Phase-1
|
||||
/// <c>site_events.db</c> and copy it in once. Safe to delete from a config once that
|
||||
/// node's migration has run.
|
||||
/// </summary>
|
||||
public string DatabasePath { get; set; } = "site_events.db";
|
||||
/// <summary>Maximum number of rows returned per paginated query; default 500.</summary>
|
||||
public int QueryPageSize { get; set; } = 500;
|
||||
|
||||
@@ -0,0 +1,71 @@
|
||||
using Microsoft.Data.Sqlite;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.SiteEventLogging;
|
||||
|
||||
/// <summary>
|
||||
/// DDL for the <c>site_events</c> table, extracted from <see cref="SiteEventLogger"/> so
|
||||
/// it can be applied by whoever owns the database file.
|
||||
/// <para>
|
||||
/// Under LocalDb Phase 1 that owner is the Host's <c>AddZbLocalDb</c> onReady callback:
|
||||
/// the table must exist before <c>ILocalDb.RegisterReplicated</c> installs its capture
|
||||
/// triggers, and rows written before registration are never captured or snapshotted.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Schema change.</b> The primary key is a TEXT GUID minted by the writer, replacing
|
||||
/// the historical <c>INTEGER PRIMARY KEY AUTOINCREMENT</c>. This is required, not
|
||||
/// cosmetic: replication resolves conflicts by last-writer-wins on the primary key, so
|
||||
/// two site nodes each minting autoincrement id 1 for unrelated events would be treated
|
||||
/// as one row and silently overwrite each other. GUID keys make the event log a pure
|
||||
/// union across the pair. Consumers that relied on the id being monotonic — the keyset
|
||||
/// paging cursor and the storage-cap purge — order by <c>timestamp</c> instead.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Deliberately depends only on <c>Microsoft.Data.Sqlite</c>, not on the LocalDb
|
||||
/// library — the Host needs to apply this DDL to a LocalDb-managed connection, but
|
||||
/// nothing about the schema itself is LocalDb-specific.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public static class SiteEventLogSchema
|
||||
{
|
||||
/// <summary>
|
||||
/// Creates the <c>site_events</c> table and its indexes when absent. Idempotent —
|
||||
/// safe to run on every startup.
|
||||
/// </summary>
|
||||
/// <param name="connection">An open connection to the database that should hold the table.</param>
|
||||
public static void Apply(SqliteConnection connection)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(connection);
|
||||
|
||||
// auto_vacuum must be set before any table is created for it to take effect
|
||||
// on a fresh database. With INCREMENTAL mode, PRAGMA incremental_vacuum can
|
||||
// later reclaim free pages so the storage-cap purge can shrink the file.
|
||||
using (var pragmaCmd = connection.CreateCommand())
|
||||
{
|
||||
pragmaCmd.CommandText = "PRAGMA auto_vacuum = INCREMENTAL";
|
||||
pragmaCmd.ExecuteNonQuery();
|
||||
}
|
||||
|
||||
using var cmd = connection.CreateCommand();
|
||||
cmd.CommandText = """
|
||||
CREATE TABLE IF NOT EXISTS site_events (
|
||||
id TEXT NOT NULL PRIMARY KEY,
|
||||
timestamp TEXT NOT NULL,
|
||||
event_type TEXT NOT NULL,
|
||||
severity TEXT NOT NULL,
|
||||
instance_id TEXT,
|
||||
source TEXT NOT NULL,
|
||||
message TEXT NOT NULL,
|
||||
details TEXT
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_events_timestamp ON site_events(timestamp);
|
||||
CREATE INDEX IF NOT EXISTS idx_events_type ON site_events(event_type);
|
||||
CREATE INDEX IF NOT EXISTS idx_events_instance ON site_events(instance_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_events_severity ON site_events(severity);
|
||||
""";
|
||||
// The query service also supports keyword search via leading-wildcard
|
||||
// LIKE on message/source. A leading-wildcard LIKE cannot use a B-tree
|
||||
// index, so that path intentionally full-scans; severity/event_type/
|
||||
// instance_id/timestamp filters above are all covered.
|
||||
cmd.ExecuteNonQuery();
|
||||
}
|
||||
}
|
||||
@@ -2,13 +2,16 @@ using System.Threading.Channels;
|
||||
using Microsoft.Data.Sqlite;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.LocalDb;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.SiteEventLogging;
|
||||
|
||||
/// <summary>
|
||||
/// Records operational events to a local SQLite database.
|
||||
/// Only the active node generates events. Not replicated to standby.
|
||||
/// On failover, the new active node starts a fresh log.
|
||||
/// Records operational events into the consolidated site database (LocalDb Phase 1).
|
||||
/// Only the active node generates events, but <c>site_events</c> is a replicated table:
|
||||
/// when the pair has a peer configured, the standby holds the same history and a failover
|
||||
/// no longer starts a fresh log. With no peer configured the database is simply local —
|
||||
/// replication is opt-in via <c>LocalDb:Replication:PeerAddress</c>.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
@@ -40,29 +43,36 @@ public class SiteEventLogger : ISiteEventLogger, IDisposable
|
||||
private bool _disposed;
|
||||
|
||||
/// <summary>
|
||||
/// Initializes the event logger, opens the SQLite connection, and starts the background writer loop.
|
||||
/// Initializes the event logger over the consolidated site database and starts the
|
||||
/// background writer loop.
|
||||
/// </summary>
|
||||
/// <param name="options">Site event log configuration (database path, retention settings).</param>
|
||||
/// <param name="options">Site event log configuration (retention settings, queue capacity).</param>
|
||||
/// <param name="logger">Logger for write-failure diagnostics.</param>
|
||||
/// <param name="connectionStringOverride">Optional connection string override; uses the configured path when null.</param>
|
||||
/// <param name="localDb">
|
||||
/// The consolidated site database. The connection it hands out is already open and
|
||||
/// carries the <c>zb_hlc_next()</c> UDF that <c>site_events</c>' capture triggers
|
||||
/// call. This logger owns the connection for its lifetime and disposes it.
|
||||
/// </param>
|
||||
/// <remarks>
|
||||
/// The former <c>connectionStringOverride</c> seam is deliberately gone rather than
|
||||
/// preserved. <c>site_events</c> is a replicated table, so a raw connection lacking
|
||||
/// the UDF would fail closed on every insert — an override would only let a caller
|
||||
/// build a logger that cannot write. <c>SiteEventLogOptions.DatabasePath</c> is
|
||||
/// likewise no longer read: the location is <c>LocalDb:Path</c>.
|
||||
/// </remarks>
|
||||
public SiteEventLogger(
|
||||
IOptions<SiteEventLogOptions> options,
|
||||
ILogger<SiteEventLogger> logger,
|
||||
string? connectionStringOverride = null)
|
||||
ILocalDb localDb)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
ArgumentNullException.ThrowIfNull(localDb);
|
||||
|
||||
_logger = logger;
|
||||
|
||||
// Cache=Shared is a cross-connection optimisation
|
||||
// that lets multiple SqliteConnections share an in-process page cache.
|
||||
// This logger owns exactly one SqliteConnection and serialises all
|
||||
// access through _writeLock, so the mode is dormant — at best dead
|
||||
// configuration, at worst a small future foot-gun for any second
|
||||
// connection opened to the same file. A test path that genuinely
|
||||
// needs Cache=Shared can still inject it via connectionStringOverride.
|
||||
var connectionString = connectionStringOverride
|
||||
?? $"Data Source={options.Value.DatabasePath}";
|
||||
_connection = new SqliteConnection(connectionString);
|
||||
_connection.Open();
|
||||
// Already open — CreateConnection returns a live, pragma-configured,
|
||||
// UDF-registered connection. Calling Open() on it again would throw.
|
||||
_connection = localDb.CreateConnection();
|
||||
|
||||
InitializeSchema();
|
||||
|
||||
@@ -130,40 +140,11 @@ public class SiteEventLogger : ISiteEventLogger, IDisposable
|
||||
}
|
||||
}
|
||||
|
||||
private void InitializeSchema()
|
||||
{
|
||||
// auto_vacuum must be set before any table is created for it to take effect
|
||||
// on a fresh database. With INCREMENTAL mode, PRAGMA incremental_vacuum can
|
||||
// later reclaim free pages so the storage-cap purge can shrink the file.
|
||||
using (var pragmaCmd = _connection.CreateCommand())
|
||||
{
|
||||
pragmaCmd.CommandText = "PRAGMA auto_vacuum = INCREMENTAL";
|
||||
pragmaCmd.ExecuteNonQuery();
|
||||
}
|
||||
|
||||
using var cmd = _connection.CreateCommand();
|
||||
cmd.CommandText = """
|
||||
CREATE TABLE IF NOT EXISTS site_events (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
timestamp TEXT NOT NULL,
|
||||
event_type TEXT NOT NULL,
|
||||
severity TEXT NOT NULL,
|
||||
instance_id TEXT,
|
||||
source TEXT NOT NULL,
|
||||
message TEXT NOT NULL,
|
||||
details TEXT
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_events_timestamp ON site_events(timestamp);
|
||||
CREATE INDEX IF NOT EXISTS idx_events_type ON site_events(event_type);
|
||||
CREATE INDEX IF NOT EXISTS idx_events_instance ON site_events(instance_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_events_severity ON site_events(severity);
|
||||
""";
|
||||
// The query service also supports keyword search via leading-wildcard
|
||||
// LIKE on message/source. A leading-wildcard LIKE cannot use a B-tree
|
||||
// index, so that path intentionally full-scans; severity/event_type/
|
||||
// instance_id/timestamp filters above are all covered.
|
||||
cmd.ExecuteNonQuery();
|
||||
}
|
||||
// Schema lives in SiteEventLogSchema so the Host's AddZbLocalDb onReady callback
|
||||
// can create this table in the consolidated site database before RegisterReplicated
|
||||
// installs its capture triggers. Applying it here too keeps a directly-constructed
|
||||
// logger (tests, tooling) self-sufficient; the DDL is idempotent.
|
||||
private void InitializeSchema() => SiteEventLogSchema.Apply(_connection);
|
||||
|
||||
/// <summary>
|
||||
/// Closed set of allowed severities. Case-sensitive to
|
||||
@@ -198,7 +179,13 @@ public class SiteEventLogger : ISiteEventLogger, IDisposable
|
||||
nameof(severity));
|
||||
}
|
||||
|
||||
// The id is minted here, by the application, rather than by SQLite. Under
|
||||
// LocalDb replication the site pair converges by last-writer-wins on the
|
||||
// primary key, so a server-minted AUTOINCREMENT would have both nodes
|
||||
// independently issuing id 1, 2, 3... for unrelated events and silently
|
||||
// overwriting each other on sync. A GUID makes the event log a pure union.
|
||||
var pending = new PendingEvent(
|
||||
Guid.NewGuid().ToString("N"),
|
||||
DateTimeOffset.UtcNow.ToString("o"),
|
||||
eventType,
|
||||
severity,
|
||||
@@ -235,9 +222,10 @@ public class SiteEventLogger : ISiteEventLogger, IDisposable
|
||||
{
|
||||
using var cmd = connection.CreateCommand();
|
||||
cmd.CommandText = """
|
||||
INSERT INTO site_events (timestamp, event_type, severity, instance_id, source, message, details)
|
||||
VALUES ($timestamp, $event_type, $severity, $instance_id, $source, $message, $details)
|
||||
INSERT INTO site_events (id, timestamp, event_type, severity, instance_id, source, message, details)
|
||||
VALUES ($id, $timestamp, $event_type, $severity, $instance_id, $source, $message, $details)
|
||||
""";
|
||||
cmd.Parameters.AddWithValue("$id", pending.Id);
|
||||
cmd.Parameters.AddWithValue("$timestamp", pending.Timestamp);
|
||||
cmd.Parameters.AddWithValue("$event_type", pending.EventType);
|
||||
cmd.Parameters.AddWithValue("$severity", pending.Severity);
|
||||
@@ -269,8 +257,8 @@ public class SiteEventLogger : ISiteEventLogger, IDisposable
|
||||
// (Health Monitoring reads FailedWriteCount) and fault the caller's
|
||||
// Task instead of silently discarding the exception.
|
||||
Interlocked.Increment(ref _failedWriteCount);
|
||||
_logger.LogError(ex, "Failed to record event: {EventType} from {Source}",
|
||||
pending.EventType, pending.Source);
|
||||
_logger.LogError(ex, "Failed to record event: {EventType} from {Source} (sqlite {SqliteError})",
|
||||
pending.EventType, pending.Source, DescribeSqliteError(ex));
|
||||
pending.Completion.TrySetException(ex);
|
||||
}
|
||||
}
|
||||
@@ -309,8 +297,28 @@ public class SiteEventLogger : ISiteEventLogger, IDisposable
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// "primary/extended" SQLite result codes (e.g. "10/522") for a
|
||||
/// <see cref="SqliteException"/> anywhere in the chain; "n/a" otherwise.
|
||||
/// The exception message alone carries only the primary code, which proved
|
||||
/// too generic to diagnose the 2026-07-20 disk-I/O incident
|
||||
/// (known-issues/2026-07-20-localdb-disk-io-error-under-load.md).
|
||||
/// </summary>
|
||||
private static string DescribeSqliteError(Exception ex)
|
||||
{
|
||||
for (Exception? e = ex; e is not null; e = e.InnerException)
|
||||
{
|
||||
if (e is SqliteException se)
|
||||
{
|
||||
return $"{se.SqliteErrorCode}/{se.SqliteExtendedErrorCode}";
|
||||
}
|
||||
}
|
||||
return "n/a";
|
||||
}
|
||||
|
||||
/// <summary>An event awaiting persistence by the background writer.</summary>
|
||||
private sealed record PendingEvent(
|
||||
string Id,
|
||||
string Timestamp,
|
||||
string EventType,
|
||||
string Severity,
|
||||
|
||||
+1
@@ -18,6 +18,7 @@
|
||||
<PackageReference Include="Microsoft.Extensions.Logging.Abstractions" />
|
||||
<PackageReference Include="Microsoft.Extensions.Options" />
|
||||
<PackageReference Include="ZB.MOM.WW.Configuration" />
|
||||
<PackageReference Include="ZB.MOM.WW.LocalDb" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
|
||||
@@ -52,7 +52,6 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
|
||||
/// </summary>
|
||||
private readonly ILoggerFactory _loggerFactory;
|
||||
private readonly IActorRef? _dclManager;
|
||||
private readonly IActorRef? _replicationActor;
|
||||
private readonly ISiteHealthCollector? _healthCollector;
|
||||
private readonly IServiceProvider? _serviceProvider;
|
||||
/// <summary>
|
||||
@@ -166,7 +165,6 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
|
||||
SiteRuntimeOptions options,
|
||||
ILogger<DeploymentManagerActor> logger,
|
||||
IActorRef? dclManager = null,
|
||||
IActorRef? replicationActor = null,
|
||||
ISiteHealthCollector? healthCollector = null,
|
||||
IServiceProvider? serviceProvider = null,
|
||||
ILoggerFactory? loggerFactory = null,
|
||||
@@ -181,7 +179,6 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
|
||||
_streamManager = streamManager;
|
||||
_options = options;
|
||||
_dclManager = dclManager;
|
||||
_replicationActor = replicationActor;
|
||||
_healthCollector = healthCollector;
|
||||
_serviceProvider = serviceProvider;
|
||||
_configFetcher = configFetcher;
|
||||
@@ -785,15 +782,14 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
|
||||
await _storage.ClearStaticOverridesAsync(instanceName);
|
||||
await _storage.ClearNativeAlarmsForInstanceAsync(instanceName);
|
||||
|
||||
// Replicate to standby node — notify-and-fetch: send only the deployment id +
|
||||
// central fetch coordinates (NOT the config JSON). The standby fetches the
|
||||
// config over HTTP itself, so a large config never crosses the intra-site Akka
|
||||
// hop (which would silently drop on the 128 KB frame trap). When the coords are
|
||||
// absent (deploy paths other than RefreshDeployment), the standby fetch is a
|
||||
// no-op miss and reconciliation is the durable backstop.
|
||||
_replicationActor?.Tell(new ReplicateConfigDeploy(
|
||||
instanceName, command.DeploymentId, command.RevisionHash, true,
|
||||
command.CentralFetchBaseUrl ?? "", command.FetchToken ?? ""));
|
||||
// No explicit replication step: deployed_configurations is a replicated table,
|
||||
// so the write above is captured and shipped to the peer like any other row.
|
||||
// This used to be a notify-and-fetch Tell — the standby was sent the deployment
|
||||
// id plus central fetch coordinates and pulled the config over HTTP itself,
|
||||
// because a large config would silently drop on the intra-site Akka hop's
|
||||
// 128 KB frame trap. Replication carries the row directly and has no such limit,
|
||||
// so the standby no longer fetches on deploy. (SiteReconciliationActor still
|
||||
// fetches at node startup when central reports gaps — a different path.)
|
||||
|
||||
return new DeployPersistenceResult(
|
||||
command.DeploymentId, instanceName, true, null, sender, isRedeploy);
|
||||
@@ -967,7 +963,6 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
|
||||
{
|
||||
if (t.IsCompletedSuccessfully)
|
||||
{
|
||||
_replicationActor?.Tell(new ReplicateConfigSetEnabled(instanceName, false));
|
||||
// Operational `deployment` event — disable succeeded.
|
||||
LogDeploymentEvent("Info", instanceName, $"Instance {instanceName} disabled");
|
||||
}
|
||||
@@ -1001,7 +996,6 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
|
||||
Task.Run(async () =>
|
||||
{
|
||||
await _storage.SetInstanceEnabledAsync(instanceName, true);
|
||||
_replicationActor?.Tell(new ReplicateConfigSetEnabled(instanceName, true));
|
||||
var configs = await _storage.GetAllDeployedConfigsAsync();
|
||||
var config = configs.FirstOrDefault(c => c.InstanceUniqueName == instanceName);
|
||||
return new EnableResult(command, config, null, sender);
|
||||
@@ -1099,7 +1093,6 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
|
||||
{
|
||||
if (t.IsCompletedSuccessfully)
|
||||
{
|
||||
_replicationActor?.Tell(new ReplicateConfigRemove(instanceName));
|
||||
// Operational `deployment` event — delete succeeded.
|
||||
LogDeploymentEvent("Info", instanceName, $"Instance {instanceName} deleted");
|
||||
}
|
||||
@@ -1949,7 +1942,6 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
|
||||
// central-only and is never stored on a site (see the purge above).
|
||||
|
||||
// Replicate artifacts to standby node
|
||||
_replicationActor?.Tell(new ReplicateArtifacts(command));
|
||||
|
||||
return new ArtifactDeploymentResponse(
|
||||
command.DeploymentId, "", true, null, DateTimeOffset.UtcNow);
|
||||
|
||||
@@ -1,707 +0,0 @@
|
||||
using Akka.Actor;
|
||||
using Akka.Cluster;
|
||||
using Akka.Event;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Observability;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Deployment;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Messages;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
|
||||
using ZB.MOM.WW.ScadaBridge.StoreAndForward;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors;
|
||||
|
||||
/// <summary>
|
||||
/// Runs on every site node (not a singleton). Handles both config and S&F replication
|
||||
/// between site cluster peers.
|
||||
///
|
||||
/// Outbound: receives local replication requests and forwards to peer via ActorSelection.
|
||||
/// Inbound: receives replicated operations from peer and applies to local SQLite.
|
||||
/// Uses fire-and-forget (Tell) — no ack wait per design.
|
||||
/// </summary>
|
||||
public class SiteReplicationActor : ReceiveActor, IWithTimers
|
||||
{
|
||||
private readonly SiteStorageService _storage;
|
||||
private readonly StoreAndForwardStorage _sfStorage;
|
||||
private readonly ReplicationService _replicationService;
|
||||
private readonly IDeploymentConfigFetcher? _configFetcher;
|
||||
private readonly string _siteRole;
|
||||
private readonly ILogger<SiteReplicationActor> _logger;
|
||||
private readonly Cluster _cluster;
|
||||
private readonly Func<bool> _isActive;
|
||||
private readonly int _configFetchRetryCount;
|
||||
private readonly TimeSpan _configFetchRetryDelay;
|
||||
private Address? _peerAddress;
|
||||
|
||||
/// <summary>Akka timer scheduler injected by the framework via <see cref="IWithTimers"/>.</summary>
|
||||
public ITimerScheduler Timers { get; set; } = null!;
|
||||
|
||||
// ── Chunked-resync assembly (standby side; actor-thread only) ──
|
||||
private string? _assemblingResyncId;
|
||||
private int _assemblingTotalChunks;
|
||||
private bool _assemblingTruncated;
|
||||
private readonly Dictionary<int, List<StoreAndForwardMessage>> _assemblingChunks = new();
|
||||
private const string ResyncAssemblyTimerKey = "sf-resync-assembly-timeout";
|
||||
|
||||
/// <summary>How long a partial chunk assembly may wait for its missing chunks before
|
||||
/// being discarded (a lost chunk = lost resync; the next peer-track retries). Ctor
|
||||
/// test seam; production default 30 s.</summary>
|
||||
private readonly TimeSpan _resyncAssemblyTimeout;
|
||||
|
||||
// ── Resync delivery confirmation (active side; actor-thread only) ──
|
||||
private string? _pendingAckResyncId;
|
||||
private const string ResyncAckTimerKey = "sf-resync-ack-timeout";
|
||||
|
||||
/// <summary>How long the active node waits for the standby's <see cref="SfBufferResyncAck"/>
|
||||
/// before warning + counting the resync as unacknowledged (lost chunks / dead peer). Ctor
|
||||
/// test seam; production default 60 s.</summary>
|
||||
private readonly TimeSpan _resyncAckTimeout;
|
||||
|
||||
/// <summary>
|
||||
/// Maximum rows an active node returns in a single anti-entropy resync snapshot.
|
||||
/// A standby whose buffer exceeded this (Truncated snapshot) resyncs the oldest
|
||||
/// 10 000 rows; further divergence drains naturally as the active node delivers.
|
||||
/// </summary>
|
||||
private const int MaxResyncRows = 10_000;
|
||||
|
||||
/// <summary>
|
||||
/// Estimated per-chunk payload budget for a resync snapshot. Akka remoting's default
|
||||
/// <c>maximum-frame-size</c> is 128 000 bytes and <c>BuildHocon</c> sets no override,
|
||||
/// so the monolithic <see cref="SfBufferSnapshot"/> is silently undeliverable for any
|
||||
/// realistic backlog (review 02 round 2, N2). 64 000 bytes leaves ≈50% headroom for
|
||||
/// the JSON envelope, CLR type manifests, and the non-payload columns.
|
||||
/// </summary>
|
||||
internal const int MaxResyncChunkBytes = 64_000;
|
||||
|
||||
/// <summary>Row cap per resync chunk (bounds a chunk even when every row is tiny).</summary>
|
||||
internal const int MaxResyncChunkRows = 200;
|
||||
|
||||
/// <summary>
|
||||
/// Splits a resync snapshot into chunks that fit Akka remoting's default
|
||||
/// 128 000-byte frame (review 02 round 2, N2): rows accumulate until the estimated
|
||||
/// payload budget or the row cap is hit. Estimation is payload-dominated
|
||||
/// (payload_json length + 512 bytes fixed overhead per row); a single row whose
|
||||
/// payload exceeds the budget ships alone (Warning at the call site). Order is
|
||||
/// preserved (oldest-first, matching GetAllMessagesAsync).
|
||||
/// </summary>
|
||||
internal static List<List<StoreAndForwardMessage>> ChunkForRemoting(
|
||||
IReadOnlyList<StoreAndForwardMessage> rows, int maxChunkBytes, int maxChunkRows)
|
||||
{
|
||||
var chunks = new List<List<StoreAndForwardMessage>>();
|
||||
var current = new List<StoreAndForwardMessage>();
|
||||
var currentBytes = 0;
|
||||
foreach (var row in rows)
|
||||
{
|
||||
var estimate = (row.PayloadJson?.Length ?? 0) + 512;
|
||||
if (current.Count > 0 && (currentBytes + estimate > maxChunkBytes || current.Count >= maxChunkRows))
|
||||
{
|
||||
chunks.Add(current);
|
||||
current = new List<StoreAndForwardMessage>();
|
||||
currentBytes = 0;
|
||||
}
|
||||
current.Add(row);
|
||||
currentBytes += estimate;
|
||||
}
|
||||
if (current.Count > 0) chunks.Add(current);
|
||||
return chunks;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Initializes a new <see cref="SiteReplicationActor"/> and registers Akka message handlers.
|
||||
/// </summary>
|
||||
/// <param name="storage">Service for accessing local site storage.</param>
|
||||
/// <param name="sfStorage">Store-and-forward SQLite storage for replication of buffered messages.</param>
|
||||
/// <param name="replicationService">Service providing replication transport logic.</param>
|
||||
/// <param name="siteRole">Akka cluster role used to identify peer nodes to replicate to.</param>
|
||||
/// <param name="logger">Logger instance.</param>
|
||||
/// <param name="configFetcher">
|
||||
/// Fetches a deployed instance's config JSON from central over HTTP. Used by the
|
||||
/// notify-and-fetch standby apply path (<see cref="HandleApplyConfigDeploy"/>): the peer
|
||||
/// replicates only the deployment id, and the standby fetches the config itself so a large
|
||||
/// config never crosses the intra-site Akka hop. Null on nodes/tests without a fetcher.
|
||||
/// </param>
|
||||
/// <param name="isActiveOverride">
|
||||
/// Active-node check that gates the buffer-resync roles (a standby requests a
|
||||
/// resync, the active node answers). Production wiring passes the Host's
|
||||
/// <c>IClusterNodeProvider.SelfIsPrimary</c> delegate (the same instance gating the
|
||||
/// S&F delivery sweep); null falls back to the shared oldest-Up evaluator
|
||||
/// (<see cref="Communication.ClusterState.ActiveNodeEvaluator"/>).
|
||||
/// </param>
|
||||
/// <param name="options">Site runtime options, including the config-fetch retry count; production defaults apply when null.</param>
|
||||
/// <param name="configFetchRetryDelay">Delay between config-fetch retry attempts; defaults to 2 seconds when null.</param>
|
||||
public SiteReplicationActor(
|
||||
SiteStorageService storage,
|
||||
StoreAndForwardStorage sfStorage,
|
||||
ReplicationService replicationService,
|
||||
string siteRole,
|
||||
ILogger<SiteReplicationActor> logger,
|
||||
IDeploymentConfigFetcher? configFetcher = null,
|
||||
Func<bool>? isActiveOverride = null,
|
||||
SiteRuntimeOptions? options = null,
|
||||
TimeSpan? configFetchRetryDelay = null,
|
||||
TimeSpan? resyncAssemblyTimeout = null,
|
||||
TimeSpan? resyncAckTimeout = null)
|
||||
{
|
||||
_storage = storage;
|
||||
_sfStorage = sfStorage;
|
||||
_replicationService = replicationService;
|
||||
_configFetcher = configFetcher;
|
||||
_siteRole = siteRole;
|
||||
_logger = logger;
|
||||
_cluster = Cluster.Get(Context.System);
|
||||
_isActive = isActiveOverride ?? DefaultIsActive;
|
||||
_resyncAssemblyTimeout = resyncAssemblyTimeout ?? TimeSpan.FromSeconds(30);
|
||||
_resyncAckTimeout = resyncAckTimeout ?? TimeSpan.FromSeconds(60);
|
||||
// UA2: bound the standby's replicated-config fetch retries. At least one
|
||||
// attempt always runs; the fixed inter-attempt delay is a test seam
|
||||
// (production default 2 s).
|
||||
_configFetchRetryCount = Math.Max(1, options?.ConfigFetchRetryCount ?? 1);
|
||||
_configFetchRetryDelay = configFetchRetryDelay ?? TimeSpan.FromSeconds(2);
|
||||
|
||||
// Cluster member events
|
||||
Receive<ClusterEvent.MemberUp>(HandleMemberUp);
|
||||
Receive<ClusterEvent.MemberRemoved>(HandleMemberRemoved);
|
||||
Receive<ClusterEvent.CurrentClusterState>(HandleCurrentClusterState);
|
||||
|
||||
// Outbound — forward to peer
|
||||
Receive<ReplicateConfigDeploy>(msg => SendToPeer(new ApplyConfigDeploy(
|
||||
msg.InstanceName, msg.DeploymentId, msg.RevisionHash, msg.IsEnabled,
|
||||
msg.CentralFetchBaseUrl, msg.FetchToken)));
|
||||
Receive<ReplicateConfigRemove>(msg => SendToPeer(new ApplyConfigRemove(msg.InstanceName)));
|
||||
Receive<ReplicateConfigSetEnabled>(msg => SendToPeer(new ApplyConfigSetEnabled(
|
||||
msg.InstanceName, msg.IsEnabled)));
|
||||
Receive<ReplicateArtifacts>(msg => SendToPeer(new ApplyArtifacts(msg.Command)));
|
||||
Receive<ReplicateStoreAndForward>(msg => SendToPeer(new ApplyStoreAndForward(msg.Operation)));
|
||||
|
||||
// Inbound — apply from peer
|
||||
Receive<ApplyConfigDeploy>(HandleApplyConfigDeploy);
|
||||
Receive<ApplyConfigRemove>(HandleApplyConfigRemove);
|
||||
Receive<ApplyConfigSetEnabled>(HandleApplyConfigSetEnabled);
|
||||
Receive<ApplyArtifacts>(HandleApplyArtifacts);
|
||||
Receive<ApplyStoreAndForward>(HandleApplyStoreAndForward);
|
||||
|
||||
// Anti-entropy — full S&F buffer resync on peer (re)join
|
||||
Receive<RequestSfBufferResync>(HandleRequestSfBufferResync);
|
||||
Receive<SfResyncSnapshotLoaded>(HandleSfResyncSnapshotLoaded);
|
||||
Receive<SfBufferSnapshotChunk>(HandleSfBufferSnapshotChunk);
|
||||
Receive<ResyncAssemblyTimedOut>(HandleResyncAssemblyTimedOut);
|
||||
Receive<SfBufferResyncAck>(msg =>
|
||||
{
|
||||
if (msg.ResyncId == _pendingAckResyncId)
|
||||
{
|
||||
_pendingAckResyncId = null;
|
||||
Timers.Cancel(ResyncAckTimerKey);
|
||||
}
|
||||
ScadaBridgeTelemetry.RecordSfResyncCompleted();
|
||||
_logger.LogInformation("S&F resync {ResyncId} acknowledged by standby: {Rows} row(s) applied",
|
||||
msg.ResyncId, msg.RowCount);
|
||||
});
|
||||
Receive<ResyncAckTimedOut>(msg =>
|
||||
{
|
||||
if (msg.ResyncId != _pendingAckResyncId) return;
|
||||
_pendingAckResyncId = null;
|
||||
ScadaBridgeTelemetry.RecordSfResyncAckMissing();
|
||||
_logger.LogWarning(
|
||||
"S&F resync {ResyncId} was never acknowledged within {Window} — snapshot chunks may have been lost (frame drop / dead peer); the next peer-track retries",
|
||||
msg.ResyncId, _resyncAckTimeout);
|
||||
});
|
||||
Receive<SfBufferSnapshot>(HandleSfBufferSnapshot); // legacy monolithic handler — retained for rolling compat
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override void PreStart()
|
||||
{
|
||||
base.PreStart();
|
||||
_cluster.Subscribe(Self, ClusterEvent.SubscriptionInitialStateMode.InitialStateAsSnapshot,
|
||||
typeof(ClusterEvent.MemberUp),
|
||||
typeof(ClusterEvent.MemberRemoved));
|
||||
_logger.LogInformation("SiteReplicationActor started, subscribing to cluster events for role {Role}", _siteRole);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override void PostStop()
|
||||
{
|
||||
_cluster.Unsubscribe(Self);
|
||||
base.PostStop();
|
||||
}
|
||||
|
||||
private void HandleCurrentClusterState(ClusterEvent.CurrentClusterState state)
|
||||
{
|
||||
foreach (var member in state.Members)
|
||||
{
|
||||
if (member.Status == MemberStatus.Up)
|
||||
TryTrackPeer(member);
|
||||
}
|
||||
}
|
||||
|
||||
private void HandleMemberUp(ClusterEvent.MemberUp evt)
|
||||
{
|
||||
TryTrackPeer(evt.Member);
|
||||
}
|
||||
|
||||
private void HandleMemberRemoved(ClusterEvent.MemberRemoved evt)
|
||||
{
|
||||
if (evt.Member.Address.Equals(_peerAddress))
|
||||
{
|
||||
_logger.LogInformation("Peer node removed: {Address}", _peerAddress);
|
||||
_peerAddress = null;
|
||||
}
|
||||
}
|
||||
|
||||
private void TryTrackPeer(Member member)
|
||||
{
|
||||
// Must have our site role, and must not be self
|
||||
if (member.HasRole(_siteRole) && !member.Address.Equals(_cluster.SelfAddress))
|
||||
{
|
||||
_peerAddress = member.Address;
|
||||
_logger.LogInformation("Peer node tracked: {Address}", _peerAddress);
|
||||
OnPeerTracked();
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Side-effect run whenever a peer is (re)tracked. A <b>standby</b> requests a
|
||||
/// full S&F buffer snapshot for anti-entropy resync — this closes the "a
|
||||
/// standby down for an hour rejoins and diverges forever" gap: it may have missed
|
||||
/// replicated Add/Remove/Park ops while it was gone. The active node never
|
||||
/// requests. <see langword="protected virtual"/> so tests can drive it without a
|
||||
/// real two-node cluster.
|
||||
/// </summary>
|
||||
protected virtual void OnPeerTracked()
|
||||
{
|
||||
if (!SafeIsActive())
|
||||
{
|
||||
SendToPeer(new RequestSfBufferResync());
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Repo-standard active-node check: this node is active when it is the OLDEST Up
|
||||
/// member carrying the site role — the same oldest-Up semantics as the S&F delivery
|
||||
/// gate (IClusterNodeProvider.SelfIsPrimary → ClusterActivityEvaluator → shared
|
||||
/// ActiveNodeEvaluator). NEVER the cluster leader: leadership is lowest-address and
|
||||
/// diverges from singleton/delivery placement permanently after the lower-address
|
||||
/// node restarts — the divergence that made the delivering node wipe its own live
|
||||
/// buffer via a wrong-direction resync (review 02 round 2, N1 Critical). Any other
|
||||
/// state reports standby — safe-by-default.
|
||||
/// </summary>
|
||||
private bool DefaultIsActive() =>
|
||||
Communication.ClusterState.ActiveNodeEvaluator.SelfIsOldestUp(_cluster, _siteRole);
|
||||
|
||||
/// <summary>
|
||||
/// Evaluates the active-node check, treating a throwing check as standby
|
||||
/// (safe-by-default: a standby never delivers or answers resyncs).
|
||||
/// </summary>
|
||||
private bool SafeIsActive()
|
||||
{
|
||||
try
|
||||
{
|
||||
return _isActive();
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex, "Active-node check threw; treating node as standby");
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Forwards a replication message to the tracked peer node's <c>site-replication</c> actor
|
||||
/// (fire-and-forget, dropped when no peer is tracked). <see langword="protected virtual"/>
|
||||
/// so tests can intercept the peer send without standing up a real two-node cluster.
|
||||
/// </summary>
|
||||
/// <param name="message">The replication message to forward to the peer.</param>
|
||||
protected virtual void SendToPeer(object message)
|
||||
{
|
||||
if (_peerAddress == null)
|
||||
{
|
||||
// A dropped op is a lost delta — surface it at Warning + a metric so a
|
||||
// never-tracked peer (a dead standby) is visible, not silent (arch
|
||||
// review 02). In single-node dev the peer is legitimately absent; the
|
||||
// per-op warning is rate-tolerable (accepted per review).
|
||||
ScadaBridgeTelemetry.RecordReplicationFailure();
|
||||
_logger.LogWarning("No peer available, dropping replication message {Type}", message.GetType().Name);
|
||||
return;
|
||||
}
|
||||
|
||||
var path = new RootActorPath(_peerAddress) / "user" / "site-replication";
|
||||
Context.ActorSelection(path).Tell(message);
|
||||
}
|
||||
|
||||
// ── Inbound handlers ──
|
||||
|
||||
private void HandleApplyConfigDeploy(ApplyConfigDeploy msg)
|
||||
{
|
||||
if (string.IsNullOrEmpty(msg.CentralFetchBaseUrl))
|
||||
{
|
||||
// The direct DeployInstanceCommand cross-cluster wire path was retired.
|
||||
// This guard is a defensive fallback: skip quietly rather than calling FetchAsync("")
|
||||
// and logging a spurious error. Reconciliation backstops any missed writes.
|
||||
_logger.LogDebug(
|
||||
"No fetch coords for {Instance} (deployment {DeploymentId}) — skipping replicated fetch; T18 reconciliation is the backstop",
|
||||
msg.InstanceName, msg.DeploymentId);
|
||||
return;
|
||||
}
|
||||
|
||||
if (_configFetcher is null)
|
||||
{
|
||||
_logger.LogWarning(
|
||||
"No config fetcher available; cannot apply replicated config for {Instance} (deployment {DeploymentId}) — reconciliation will backstop",
|
||||
msg.InstanceName, msg.DeploymentId);
|
||||
return;
|
||||
}
|
||||
|
||||
_logger.LogInformation(
|
||||
"Replicating config for {Instance} (deployment {DeploymentId}) — fetching from central",
|
||||
msg.InstanceName, msg.DeploymentId);
|
||||
|
||||
// Notify-and-fetch: the peer sent only the id, so the standby fetches the config
|
||||
// itself (off-thread; best-effort fire-and-forget, matching the no-ack replication
|
||||
// model). The guarded write only overwrites a strictly-older local row. The fetch
|
||||
// is retried up to ConfigFetchRetryCount times with a fixed delay (UA2) — a transient
|
||||
// central hiccup no longer defers to the slower reconciliation backstop, which still
|
||||
// covers a total failure after the last attempt.
|
||||
_ = FetchWithRetryAsync();
|
||||
return;
|
||||
|
||||
async Task FetchWithRetryAsync()
|
||||
{
|
||||
for (var attempt = 1; attempt <= _configFetchRetryCount; attempt++)
|
||||
{
|
||||
try
|
||||
{
|
||||
// Non-null: the outer method returns early when _configFetcher is null.
|
||||
var json = await _configFetcher!.FetchAsync(
|
||||
msg.CentralFetchBaseUrl, msg.DeploymentId, msg.FetchToken, CancellationToken.None);
|
||||
await _storage.StoreDeployedConfigIfNewerAsync(
|
||||
msg.InstanceName, json, msg.DeploymentId, msg.RevisionHash, msg.IsEnabled);
|
||||
return;
|
||||
}
|
||||
catch (DeploymentConfigFetchException fex) when (fex.IsSuperseded)
|
||||
{
|
||||
// A superseded/expired fetch never heals by retrying — a newer deploy
|
||||
// will replicate its own id.
|
||||
_logger.LogInformation(
|
||||
"Skip replicated config for {Instance}: superseded/expired (a newer deploy will replicate)",
|
||||
msg.InstanceName);
|
||||
return;
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
if (attempt < _configFetchRetryCount)
|
||||
{
|
||||
_logger.LogWarning(ex,
|
||||
"Replicated config fetch attempt {Attempt}/{Max} failed for {Instance} (deployment {DeploymentId}) — retrying in {Delay}",
|
||||
attempt, _configFetchRetryCount, msg.InstanceName, msg.DeploymentId, _configFetchRetryDelay);
|
||||
await Task.Delay(_configFetchRetryDelay);
|
||||
}
|
||||
else
|
||||
{
|
||||
_logger.LogError(ex,
|
||||
"Replicated config fetch failed after {Attempts} attempt(s) for {Instance} (deployment {DeploymentId}) — reconciliation will backstop",
|
||||
_configFetchRetryCount, msg.InstanceName, msg.DeploymentId);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private void HandleApplyConfigRemove(ApplyConfigRemove msg)
|
||||
{
|
||||
_logger.LogInformation("Applying replicated config remove for {Instance}", msg.InstanceName);
|
||||
_storage.RemoveDeployedConfigAsync(msg.InstanceName)
|
||||
.ContinueWith(t =>
|
||||
{
|
||||
if (t.IsFaulted)
|
||||
_logger.LogError(t.Exception, "Failed to apply replicated remove for {Instance}", msg.InstanceName);
|
||||
});
|
||||
}
|
||||
|
||||
private void HandleApplyConfigSetEnabled(ApplyConfigSetEnabled msg)
|
||||
{
|
||||
_logger.LogInformation("Applying replicated set-enabled={Enabled} for {Instance}", msg.IsEnabled, msg.InstanceName);
|
||||
_storage.SetInstanceEnabledAsync(msg.InstanceName, msg.IsEnabled)
|
||||
.ContinueWith(t =>
|
||||
{
|
||||
if (t.IsFaulted)
|
||||
_logger.LogError(t.Exception, "Failed to apply replicated set-enabled for {Instance}", msg.InstanceName);
|
||||
});
|
||||
}
|
||||
|
||||
private void HandleApplyArtifacts(ApplyArtifacts msg)
|
||||
{
|
||||
var command = msg.Command;
|
||||
_logger.LogInformation("Applying replicated artifacts, deploymentId={DeploymentId}", command.DeploymentId);
|
||||
|
||||
Task.Run(async () =>
|
||||
{
|
||||
try
|
||||
{
|
||||
if (command.SharedScripts != null)
|
||||
foreach (var s in command.SharedScripts)
|
||||
await _storage.StoreSharedScriptAsync(s.Name, s.Code, s.ParameterDefinitions, s.ReturnDefinition);
|
||||
|
||||
if (command.ExternalSystems != null)
|
||||
foreach (var es in command.ExternalSystems)
|
||||
await _storage.StoreExternalSystemAsync(es.Name, es.EndpointUrl, es.AuthType, es.AuthConfiguration, es.MethodDefinitionsJson, es.TimeoutSeconds);
|
||||
|
||||
if (command.DatabaseConnections != null)
|
||||
foreach (var db in command.DatabaseConnections)
|
||||
await _storage.StoreDatabaseConnectionAsync(db.Name, db.ConnectionString, db.MaxRetries, db.RetryDelay);
|
||||
|
||||
// Notification lists and SMTP
|
||||
// configuration are central-only and are never persisted on a site.
|
||||
// Mirror the primary apply path: purge any pre-fix rows (including the
|
||||
// plaintext SMTP password) instead of writing the command's
|
||||
// (now-always-null) NotificationLists/SmtpConfigurations.
|
||||
await _storage.PurgeCentralOnlyNotificationConfigAsync();
|
||||
|
||||
if (command.DataConnections != null)
|
||||
foreach (var dc in command.DataConnections)
|
||||
await _storage.StoreDataConnectionDefinitionAsync(dc.Name, dc.Protocol, dc.PrimaryConfigurationJson, dc.BackupConfigurationJson, dc.FailoverRetryCount);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogError(ex, "Failed to apply replicated artifacts");
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
private void HandleApplyStoreAndForward(ApplyStoreAndForward msg)
|
||||
{
|
||||
_logger.LogDebug("Applying replicated S&F operation {OpType} for message {Id}",
|
||||
msg.Operation.OperationType, msg.Operation.MessageId);
|
||||
|
||||
_replicationService.ApplyReplicatedOperationAsync(msg.Operation, _sfStorage)
|
||||
.ContinueWith(t =>
|
||||
{
|
||||
if (t.IsFaulted)
|
||||
_logger.LogError(t.Exception, "Failed to apply replicated S&F operation {Id}", msg.Operation.MessageId);
|
||||
});
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Active-node side of the anti-entropy resync: answers a standby's
|
||||
/// <see cref="RequestSfBufferResync"/> with a sequence of byte-budgeted
|
||||
/// <see cref="SfBufferSnapshotChunk"/>s (up to <see cref="MaxResyncRows"/> oldest rows).
|
||||
/// A non-active node ignores the request — only the authoritative node may answer.
|
||||
/// The snapshot is piped back to Self so chunking + ack bookkeeping stays actor-safe.
|
||||
/// </summary>
|
||||
private void HandleRequestSfBufferResync(RequestSfBufferResync msg)
|
||||
{
|
||||
if (!SafeIsActive())
|
||||
{
|
||||
_logger.LogDebug("Ignoring S&F buffer resync request — this node is not active");
|
||||
return;
|
||||
}
|
||||
|
||||
var replyTo = Sender;
|
||||
_sfStorage.GetAllMessagesAsync(MaxResyncRows).PipeTo(
|
||||
Self,
|
||||
failure: ex => new Status.Failure(ex),
|
||||
success: result => new SfResyncSnapshotLoaded(replyTo, result.Messages, result.Truncated));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Active-node continuation: the resync snapshot finished loading; chunk it to fit the
|
||||
/// remoting frame and send the sequenced chunks to the requester (all sharing one
|
||||
/// resyncId). Task 7 arms the ack-timeout here.
|
||||
/// </summary>
|
||||
private void HandleSfResyncSnapshotLoaded(SfResyncSnapshotLoaded msg)
|
||||
{
|
||||
var chunks = ChunkForRemoting(msg.Messages, MaxResyncChunkBytes, MaxResyncChunkRows);
|
||||
if (chunks.Count == 0) chunks.Add(new List<StoreAndForwardMessage>()); // empty buffer still resyncs (clears the standby)
|
||||
var resyncId = Guid.NewGuid().ToString("N");
|
||||
for (var i = 0; i < chunks.Count; i++)
|
||||
{
|
||||
if (chunks[i].Count == 1 && (chunks[i][0].PayloadJson?.Length ?? 0) + 512 > MaxResyncChunkBytes)
|
||||
_logger.LogWarning(
|
||||
"Resync row {Id} alone exceeds the chunk budget ({Bytes}B payload); sending solo — it may exceed the remoting frame",
|
||||
chunks[i][0].Id, chunks[i][0].PayloadJson?.Length ?? 0);
|
||||
msg.ReplyTo.Tell(new SfBufferSnapshotChunk(resyncId, i + 1, chunks.Count, chunks[i], msg.Truncated), Self);
|
||||
}
|
||||
_logger.LogInformation(
|
||||
"Answered S&F resync request with {Rows} row(s) in {Chunks} chunk(s), resyncId={ResyncId}",
|
||||
msg.Messages.Count, chunks.Count, resyncId);
|
||||
// Arm the ack window: absence of an SfBufferResyncAck within it surfaces as a
|
||||
// Warning + counter (the silent-loss mode N2 flagged). Single-outstanding-resync
|
||||
// bookkeeping: a new request supersedes by overwriting the id and restarting the timer.
|
||||
_pendingAckResyncId = resyncId;
|
||||
Timers.StartSingleTimer(ResyncAckTimerKey, new ResyncAckTimedOut(resyncId), _resyncAckTimeout);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Standby-node side of the anti-entropy resync: replaces the local buffer
|
||||
/// wholesale with the active node's snapshot. Combined with the upsert-based
|
||||
/// replicated applies (arch review 02), any replicated op that lands
|
||||
/// after this resync merges cleanly onto the resynced state. An active node
|
||||
/// ignores a snapshot — it is the source of truth, never a resync target.
|
||||
/// </summary>
|
||||
private void HandleSfBufferSnapshot(SfBufferSnapshot msg)
|
||||
{
|
||||
if (SafeIsActive())
|
||||
{
|
||||
_logger.LogDebug("Ignoring S&F buffer snapshot — this node is active");
|
||||
return;
|
||||
}
|
||||
|
||||
if (msg.Truncated)
|
||||
{
|
||||
_logger.LogWarning(
|
||||
"S&F buffer resync snapshot truncated at {Cap} rows; divergence beyond the cap drains naturally",
|
||||
MaxResyncRows);
|
||||
}
|
||||
|
||||
_logger.LogInformation(
|
||||
"Applying S&F buffer resync snapshot ({Count} rows), replacing local buffer", msg.Messages.Count);
|
||||
|
||||
Task.Run(async () =>
|
||||
{
|
||||
// Belt-and-braces (N1): re-check at apply time. ReplaceAllAsync discards
|
||||
// every in-flight row (StoreAndForwardStorage.cs "Never call on an active
|
||||
// node"); a flip between message receipt and this point must abort.
|
||||
if (SafeIsActive())
|
||||
{
|
||||
_logger.LogWarning(
|
||||
"Discarding S&F buffer resync snapshot: this node became active before apply");
|
||||
return;
|
||||
}
|
||||
await _sfStorage.ReplaceAllAsync(msg.Messages);
|
||||
})
|
||||
.ContinueWith(t =>
|
||||
{
|
||||
if (t.IsFaulted)
|
||||
_logger.LogError(t.Exception, "Failed to apply S&F buffer resync snapshot");
|
||||
});
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Standby-node side of the chunked anti-entropy resync: accumulates the chunks of one
|
||||
/// <c>ResyncId</c>, and once all have arrived, assembles them in sequence order and
|
||||
/// replaces the local buffer atomically, then acks. A new <c>ResyncId</c> discards any
|
||||
/// stale partial assembly (review 02 round 2, N2). An active node ignores chunks.
|
||||
/// </summary>
|
||||
private void HandleSfBufferSnapshotChunk(SfBufferSnapshotChunk msg)
|
||||
{
|
||||
if (SafeIsActive())
|
||||
{
|
||||
_logger.LogDebug("Ignoring S&F resync chunk — this node is active");
|
||||
return;
|
||||
}
|
||||
|
||||
if (_assemblingResyncId != msg.ResyncId)
|
||||
{
|
||||
if (_assemblingResyncId != null)
|
||||
_logger.LogWarning(
|
||||
"Discarding partial S&F resync assembly {Old} ({Have}/{Want} chunks): a new resync {New} superseded it",
|
||||
_assemblingResyncId, _assemblingChunks.Count, _assemblingTotalChunks, msg.ResyncId);
|
||||
_assemblingResyncId = msg.ResyncId;
|
||||
_assemblingTotalChunks = msg.TotalChunks;
|
||||
_assemblingTruncated = msg.Truncated;
|
||||
_assemblingChunks.Clear();
|
||||
}
|
||||
|
||||
_assemblingChunks[msg.Sequence] = msg.Messages;
|
||||
Timers.StartSingleTimer(ResyncAssemblyTimerKey, new ResyncAssemblyTimedOut(msg.ResyncId), _resyncAssemblyTimeout);
|
||||
|
||||
if (_assemblingChunks.Count < _assemblingTotalChunks)
|
||||
return;
|
||||
|
||||
// Complete: assemble in sequence order and apply atomically.
|
||||
var assembled = Enumerable.Range(1, _assemblingTotalChunks)
|
||||
.SelectMany(seq => _assemblingChunks[seq])
|
||||
.ToList();
|
||||
var resyncId = _assemblingResyncId!;
|
||||
var truncated = _assemblingTruncated;
|
||||
_assemblingResyncId = null;
|
||||
_assemblingChunks.Clear();
|
||||
Timers.Cancel(ResyncAssemblyTimerKey);
|
||||
|
||||
if (truncated)
|
||||
_logger.LogWarning(
|
||||
"S&F buffer resync snapshot truncated at {Cap} rows; divergence beyond the cap drains naturally",
|
||||
MaxResyncRows);
|
||||
_logger.LogInformation(
|
||||
"Applying chunked S&F resync {ResyncId} ({Count} rows), replacing local buffer", resyncId, assembled.Count);
|
||||
|
||||
// KNOWN, ACCEPTED race (review 02 round 2, N5 — do NOT "fix" this into something
|
||||
// worse): a replicated Remove sent after the active node read its snapshot but
|
||||
// before the snapshot's chunks is ordered BEFORE them on the wire (same
|
||||
// sender/receiver pair), so this apply can re-add the removed row → an orphan
|
||||
// Pending row that a later failover re-delivers ONCE. Bounded, self-correcting at
|
||||
// the next resync, and inherent to no-ack replication; a delivered-side dedup or
|
||||
// op-sequencing scheme would cost far more than one rare duplicate.
|
||||
var replyTo = Sender;
|
||||
Task.Run(async () =>
|
||||
{
|
||||
if (SafeIsActive()) // belt-and-braces, mirrors the monolithic path (T3)
|
||||
{
|
||||
_logger.LogWarning("Discarding chunked S&F resync {ResyncId}: node became active before apply", resyncId);
|
||||
return;
|
||||
}
|
||||
await _sfStorage.ReplaceAllAsync(assembled);
|
||||
replyTo.Tell(new SfBufferResyncAck(resyncId, assembled.Count));
|
||||
})
|
||||
.ContinueWith(t =>
|
||||
{
|
||||
if (t.IsFaulted)
|
||||
_logger.LogError(t.Exception, "Failed to apply chunked S&F resync {ResyncId}", resyncId);
|
||||
});
|
||||
}
|
||||
|
||||
private void HandleResyncAssemblyTimedOut(ResyncAssemblyTimedOut msg)
|
||||
{
|
||||
if (_assemblingResyncId != msg.ResyncId) return; // superseded already
|
||||
_logger.LogWarning(
|
||||
"S&F resync assembly {ResyncId} timed out with {Have}/{Want} chunks — discarding partial (a lost chunk; the next peer-track retries)",
|
||||
msg.ResyncId, _assemblingChunks.Count, _assemblingTotalChunks);
|
||||
ScadaBridgeTelemetry.RecordReplicationFailure();
|
||||
_assemblingResyncId = null;
|
||||
_assemblingChunks.Clear();
|
||||
}
|
||||
|
||||
/// <summary>Internal: the resync snapshot finished loading; chunk and send to the requester.</summary>
|
||||
internal sealed record SfResyncSnapshotLoaded(
|
||||
IActorRef ReplyTo, List<StoreAndForwardMessage> Messages, bool Truncated);
|
||||
|
||||
/// <summary>Internal: a partial chunk assembly for <paramref name="ResyncId"/> exceeded its window.</summary>
|
||||
internal sealed record ResyncAssemblyTimedOut(string ResyncId);
|
||||
|
||||
/// <summary>Internal: the active node's ack window for <paramref name="ResyncId"/> expired.</summary>
|
||||
internal sealed record ResyncAckTimedOut(string ResyncId);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Standby→active: request a full S&F buffer snapshot for anti-entropy resync
|
||||
/// (sent when a standby (re)tracks a peer). Crosses Akka remoting between the two
|
||||
/// site nodes; the POCO rides the default serializer.
|
||||
/// </summary>
|
||||
public sealed record RequestSfBufferResync;
|
||||
|
||||
/// <summary>
|
||||
/// Active→standby: full-buffer snapshot. <paramref name="Truncated"/> is true when
|
||||
/// the active node's buffer exceeded <c>MaxResyncRows</c> (the standby logs a Warning —
|
||||
/// divergence beyond the cap drains naturally as the active node delivers). Crosses
|
||||
/// Akka remoting; the message list rides the default serializer.
|
||||
/// </summary>
|
||||
public sealed record SfBufferSnapshot(List<StoreAndForwardMessage> Messages, bool Truncated);
|
||||
|
||||
/// <summary>
|
||||
/// Active→standby: one sequenced chunk of a full-buffer anti-entropy snapshot
|
||||
/// (review 02 round 2, N2 — the monolithic <see cref="SfBufferSnapshot"/> exceeds Akka
|
||||
/// remoting's default 128 000-byte frame for any realistic backlog). All chunks of one
|
||||
/// resync share <paramref name="ResyncId"/>; <paramref name="Sequence"/> is 1-based up to
|
||||
/// <paramref name="TotalChunks"/>. Additive message — the legacy monolithic snapshot
|
||||
/// handler is retained for rolling upgrades. Crosses intra-site Akka remoting (NOT
|
||||
/// ClusterClient — ClusterClientContractLockTests is intentionally not involved).
|
||||
/// </summary>
|
||||
public sealed record SfBufferSnapshotChunk(
|
||||
string ResyncId, int Sequence, int TotalChunks,
|
||||
List<StoreAndForwardMessage> Messages, bool Truncated);
|
||||
|
||||
/// <summary>
|
||||
/// Standby→active: delivery confirmation — the standby assembled all chunks of
|
||||
/// <paramref name="ResyncId"/> and applied them atomically (<paramref name="RowCount"/>
|
||||
/// rows installed). Absence within the ack window is surfaced by the active node
|
||||
/// (Warning + counter) — the silent-loss mode N2 flagged.
|
||||
/// </summary>
|
||||
public sealed record SfBufferResyncAck(string ResyncId, int RowCount);
|
||||
@@ -1,44 +0,0 @@
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts;
|
||||
using ZB.MOM.WW.ScadaBridge.StoreAndForward;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Messages;
|
||||
|
||||
// Outbound messages — sent by local DeploymentManagerActor/S&F service
|
||||
// to the local SiteReplicationActor for forwarding to the peer node.
|
||||
|
||||
/// <summary>Outbound: tell the peer to fetch+apply a deployed instance config by id (notify-and-fetch; no inline config).</summary>
|
||||
public record ReplicateConfigDeploy(
|
||||
string InstanceName, string DeploymentId, string RevisionHash, bool IsEnabled,
|
||||
string CentralFetchBaseUrl, string FetchToken);
|
||||
|
||||
/// <summary>Outbound: replicate removal of a deployed instance config to the peer node.</summary>
|
||||
public record ReplicateConfigRemove(string InstanceName);
|
||||
|
||||
/// <summary>Outbound: replicate an instance enabled/disabled flag change to the peer node.</summary>
|
||||
public record ReplicateConfigSetEnabled(string InstanceName, bool IsEnabled);
|
||||
|
||||
/// <summary>Outbound: replicate a system-wide artifact deployment (shared scripts, external systems, etc.) to the peer node.</summary>
|
||||
public record ReplicateArtifacts(DeployArtifactsCommand Command);
|
||||
|
||||
/// <summary>Outbound: replicate a store-and-forward buffer mutation (enqueue/dequeue/park/etc.) to the peer node.</summary>
|
||||
public record ReplicateStoreAndForward(ReplicationOperation Operation);
|
||||
|
||||
// Inbound messages — received from the peer's SiteReplicationActor
|
||||
// and applied to local SQLite storage.
|
||||
|
||||
/// <summary>Inbound: peer-replicated config deploy — the standby fetches the config by id and writes it (guarded).</summary>
|
||||
public record ApplyConfigDeploy(
|
||||
string InstanceName, string DeploymentId, string RevisionHash, bool IsEnabled,
|
||||
string CentralFetchBaseUrl, string FetchToken);
|
||||
|
||||
/// <summary>Inbound: apply peer-replicated removal of a deployed instance config to local SQLite.</summary>
|
||||
public record ApplyConfigRemove(string InstanceName);
|
||||
|
||||
/// <summary>Inbound: apply a peer-replicated instance enabled/disabled flag change to local SQLite.</summary>
|
||||
public record ApplyConfigSetEnabled(string InstanceName, bool IsEnabled);
|
||||
|
||||
/// <summary>Inbound: apply a peer-replicated system-wide artifact deployment to local SQLite.</summary>
|
||||
public record ApplyArtifacts(DeployArtifactsCommand Command);
|
||||
|
||||
/// <summary>Inbound: apply a peer-replicated store-and-forward buffer mutation to the local buffer.</summary>
|
||||
public record ApplyStoreAndForward(ReplicationOperation Operation);
|
||||
@@ -0,0 +1,169 @@
|
||||
using Microsoft.Data.Sqlite;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
|
||||
|
||||
/// <summary>
|
||||
/// DDL for the site's nine configuration tables, extracted from
|
||||
/// <see cref="SiteStorageService"/> so it can be applied by whoever owns the database
|
||||
/// file.
|
||||
/// <para>
|
||||
/// Deliberately depends only on <c>Microsoft.Data.Sqlite</c>, not on the LocalDb library.
|
||||
/// The Host applies this DDL to a LocalDb-managed connection before
|
||||
/// <c>RegisterReplicated</c> installs the capture triggers; nothing about the schema
|
||||
/// itself is LocalDb-specific, and the service still calls it so a directly-constructed
|
||||
/// service (tests, tooling) remains self-sufficient.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Journal-mode and other connection pragmas are deliberately NOT set here — LocalDb owns
|
||||
/// the connection's pragmas, and <see cref="SiteStorageService"/> keeps its own WAL set-up
|
||||
/// for the databases it opens itself.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public static class SiteStorageSchema
|
||||
{
|
||||
/// <summary>
|
||||
/// Creates the nine site configuration tables when absent, and additively upgrades
|
||||
/// tables created by an older build. Idempotent — safe to run on every startup.
|
||||
/// </summary>
|
||||
/// <param name="connection">An open connection to the database that should hold the tables.</param>
|
||||
public static void Apply(SqliteConnection connection)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(connection);
|
||||
|
||||
using (var command = connection.CreateCommand())
|
||||
{
|
||||
command.CommandText = @"
|
||||
CREATE TABLE IF NOT EXISTS deployed_configurations (
|
||||
instance_unique_name TEXT PRIMARY KEY,
|
||||
config_json TEXT NOT NULL,
|
||||
deployment_id TEXT NOT NULL,
|
||||
revision_hash TEXT NOT NULL,
|
||||
is_enabled INTEGER NOT NULL DEFAULT 1,
|
||||
deployed_at TEXT NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS static_attribute_overrides (
|
||||
instance_unique_name TEXT NOT NULL,
|
||||
attribute_name TEXT NOT NULL,
|
||||
override_value TEXT NOT NULL,
|
||||
updated_at TEXT NOT NULL,
|
||||
PRIMARY KEY (instance_unique_name, attribute_name)
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS shared_scripts (
|
||||
name TEXT PRIMARY KEY,
|
||||
code TEXT NOT NULL,
|
||||
parameter_definitions TEXT,
|
||||
return_definition TEXT,
|
||||
updated_at TEXT NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS external_systems (
|
||||
name TEXT PRIMARY KEY,
|
||||
endpoint_url TEXT NOT NULL,
|
||||
auth_type TEXT NOT NULL,
|
||||
auth_configuration TEXT,
|
||||
method_definitions TEXT,
|
||||
updated_at TEXT NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS database_connections (
|
||||
name TEXT PRIMARY KEY,
|
||||
connection_string TEXT NOT NULL,
|
||||
max_retries INTEGER NOT NULL DEFAULT 3,
|
||||
retry_delay_ms INTEGER NOT NULL DEFAULT 1000,
|
||||
updated_at TEXT NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS notification_lists (
|
||||
name TEXT PRIMARY KEY,
|
||||
recipient_emails TEXT NOT NULL,
|
||||
updated_at TEXT NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS data_connection_definitions (
|
||||
name TEXT PRIMARY KEY,
|
||||
protocol TEXT NOT NULL,
|
||||
configuration TEXT,
|
||||
backup_configuration TEXT,
|
||||
failover_retry_count INTEGER NOT NULL DEFAULT 3,
|
||||
updated_at TEXT NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS smtp_configurations (
|
||||
name TEXT PRIMARY KEY,
|
||||
server TEXT NOT NULL,
|
||||
port INTEGER NOT NULL,
|
||||
auth_mode TEXT NOT NULL,
|
||||
from_address TEXT NOT NULL,
|
||||
username TEXT,
|
||||
password TEXT,
|
||||
oauth_config TEXT,
|
||||
updated_at TEXT NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS native_alarm_state (
|
||||
instance_unique_name TEXT NOT NULL,
|
||||
source_canonical_name TEXT NOT NULL,
|
||||
source_reference TEXT NOT NULL,
|
||||
condition_json TEXT NOT NULL,
|
||||
last_transition_at TEXT NOT NULL,
|
||||
metadata_json TEXT,
|
||||
PRIMARY KEY (instance_unique_name, source_canonical_name, source_reference)
|
||||
);
|
||||
";
|
||||
command.ExecuteNonQuery();
|
||||
}
|
||||
|
||||
// Schema migrations — add columns that may not exist on older databases
|
||||
MigrateSchema(connection);
|
||||
}
|
||||
|
||||
private static void MigrateSchema(SqliteConnection connection)
|
||||
{
|
||||
// Add backup_configuration and failover_retry_count to data_connection_definitions
|
||||
// (added in primary/backup data connections feature)
|
||||
AddColumnIfMissing(connection, "data_connection_definitions", "backup_configuration", "TEXT");
|
||||
AddColumnIfMissing(connection, "data_connection_definitions", "failover_retry_count", "INTEGER NOT NULL DEFAULT 3");
|
||||
|
||||
// Native-alarm display metadata (UA4) — restored on rehydration so a persisted condition
|
||||
// renders fully (type/category/message/values) before the first source snapshot arrives.
|
||||
AddColumnIfMissing(connection, "native_alarm_state", "metadata_json", "TEXT");
|
||||
|
||||
// Per-external-system call timeout (ExternalSystemGateway Timeout) — carried through the
|
||||
// artifact pipeline so site-side calls honor it; 0 = use the site default.
|
||||
AddColumnIfMissing(connection, "external_systems", "timeout_seconds", "INTEGER NOT NULL DEFAULT 0");
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Additively adds a column only when it is not already present. SQLite lacks
|
||||
/// <c>ADD COLUMN IF NOT EXISTS</c>, so the schema is probed via
|
||||
/// <c>PRAGMA table_info</c> first. Mirrors <c>OperationTrackingSchema</c>.
|
||||
/// <para>
|
||||
/// This replaces the previous try/catch on <c>SqliteException</c> message text
|
||||
/// containing "duplicate column": probing is the same precedent the other schema
|
||||
/// classes use, does not depend on an error string that is not part of SQLite's
|
||||
/// contract, and does not swallow unrelated failures of the same exception type.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
private static void AddColumnIfMissing(
|
||||
SqliteConnection connection, string table, string columnName, string columnDefinition)
|
||||
{
|
||||
using (var probe = connection.CreateCommand())
|
||||
{
|
||||
// Table + column names are caller-controlled constants, never user input —
|
||||
// safe to interpolate (parameters are not permitted in DDL or in a
|
||||
// pragma-function argument).
|
||||
probe.CommandText = $"SELECT COUNT(*) FROM pragma_table_info('{table}') WHERE name = $name";
|
||||
probe.Parameters.AddWithValue("$name", columnName);
|
||||
if (Convert.ToInt32(probe.ExecuteScalar()) > 0)
|
||||
{
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
using var alter = connection.CreateCommand();
|
||||
alter.CommandText = $"ALTER TABLE {table} ADD COLUMN {columnName} {columnDefinition}";
|
||||
alter.ExecuteNonQuery();
|
||||
}
|
||||
}
|
||||
@@ -1,5 +1,6 @@
|
||||
using Microsoft.Data.Sqlite;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using ZB.MOM.WW.LocalDb;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
|
||||
|
||||
@@ -10,189 +11,68 @@ namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
|
||||
/// </summary>
|
||||
public class SiteStorageService
|
||||
{
|
||||
private readonly string _connectionString;
|
||||
private readonly ILocalDb _localDb;
|
||||
private readonly ILogger<SiteStorageService> _logger;
|
||||
|
||||
/// <summary>
|
||||
/// Initializes a new instance of the SiteStorageService with the specified SQLite connection string and logger.
|
||||
/// Initializes the service over the consolidated site database.
|
||||
/// </summary>
|
||||
/// <param name="connectionString">SQLite connection string for the site database.</param>
|
||||
/// <param name="localDb">
|
||||
/// The consolidated site database. Every connection it hands out is already open and carries
|
||||
/// the per-connection pragmas (including the busy timeout that this class used to pin itself)
|
||||
/// plus the <c>zb_hlc_next()</c> UDF that the config tables' capture triggers call — which is
|
||||
/// exactly why the service no longer builds its own connection string. A raw connection would
|
||||
/// lack the UDF and every write to a replicated table would fail closed.
|
||||
/// </param>
|
||||
/// <param name="logger">Logger instance for diagnostic messages.</param>
|
||||
public SiteStorageService(string connectionString, ILogger<SiteStorageService> logger)
|
||||
public SiteStorageService(ILocalDb localDb, ILogger<SiteStorageService> logger)
|
||||
{
|
||||
// Normalize the connection string and pin a busy-timeout floor (S8). WAL lets a reader
|
||||
// and a writer proceed concurrently, but two writers still contend; Microsoft.Data.Sqlite
|
||||
// drives its busy handler off the command timeout, so a floor of 5 s means a briefly
|
||||
// busy database is waited-on rather than failing fast with SQLITE_BUSY. Only raise a
|
||||
// caller-supplied value that is lower than the floor (the library default of 30 s stays).
|
||||
var builder = new SqliteConnectionStringBuilder(connectionString);
|
||||
if (builder.DefaultTimeout < BusyTimeoutFloorSeconds)
|
||||
builder.DefaultTimeout = BusyTimeoutFloorSeconds;
|
||||
_connectionString = builder.ToString();
|
||||
ArgumentNullException.ThrowIfNull(localDb);
|
||||
ArgumentNullException.ThrowIfNull(logger);
|
||||
_localDb = localDb;
|
||||
_logger = logger;
|
||||
}
|
||||
|
||||
/// <summary>Busy-timeout floor in seconds applied to the site connection string (S8).</summary>
|
||||
private const int BusyTimeoutFloorSeconds = 5;
|
||||
/// <summary>
|
||||
/// Returns an <b>already-open</b> connection against the site database.
|
||||
/// Exposed so site-local repositories can get their own connections without reaching
|
||||
/// into private state via reflection. The caller owns the connection and is
|
||||
/// responsible for disposing it.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <b>Contract change:</b> this used to return an <i>unopened</i> connection that the
|
||||
/// caller opened. LocalDb hands out connections already open, pragma-configured, and
|
||||
/// carrying the <c>zb_hlc_next()</c> UDF — calling <c>Open</c>/<c>OpenAsync</c> on one
|
||||
/// throws. Callers must not open it.
|
||||
/// </remarks>
|
||||
/// <returns>An open <see cref="SqliteConnection"/> against the site database.</returns>
|
||||
public SqliteConnection CreateConnection() => _localDb.CreateConnection();
|
||||
|
||||
/// <summary>
|
||||
/// Creates a new (unopened) SQLite connection against the site database.
|
||||
/// Exposed so site-local repositories can open their own connections without
|
||||
/// reaching into private state via reflection. The caller owns
|
||||
/// the connection and is responsible for opening and disposing it.
|
||||
/// <para>
|
||||
/// The database runs in WAL journal mode (set once in <see cref="InitializeAsync"/>) with a
|
||||
/// busy-timeout floor (see the constructor), so connections handed out here inherit
|
||||
/// concurrent-reader/writer behavior and wait out a briefly-busy database instead of
|
||||
/// failing with SQLITE_BUSY.
|
||||
/// </para>
|
||||
/// Convenience alias used internally, mirroring the other LocalDb-backed stores.
|
||||
/// </summary>
|
||||
/// <returns>A new, unopened <see cref="SqliteConnection"/> against the site database.</returns>
|
||||
public SqliteConnection CreateConnection() => new(_connectionString);
|
||||
private SqliteConnection OpenConnection() => _localDb.CreateConnection();
|
||||
|
||||
/// <summary>
|
||||
/// Creates the SQLite tables if they do not exist.
|
||||
/// Called once on site startup.
|
||||
/// </summary>
|
||||
/// <returns>A task that completes when all tables have been created or verified.</returns>
|
||||
public async Task InitializeAsync()
|
||||
public Task InitializeAsync()
|
||||
{
|
||||
await using var connection = new SqliteConnection(_connectionString);
|
||||
await connection.OpenAsync();
|
||||
// No journal-mode pragma and no busy-timeout normalization here any more:
|
||||
// LocalDb owns the file and sets WAL plus the per-connection pragmas on every
|
||||
// connection it hands out.
|
||||
using var connection = OpenConnection();
|
||||
|
||||
// Switch to WAL journal mode (S8). WAL is persistent (database-level, survives across
|
||||
// connections) so this one-time set is enough. It lets readers and a writer proceed
|
||||
// concurrently instead of every access serializing on the rollback journal. A
|
||||
// :memory: database or a filesystem that cannot support WAL falls back to its prior
|
||||
// mode — we log the result rather than throwing.
|
||||
await using (var walCommand = connection.CreateCommand())
|
||||
{
|
||||
walCommand.CommandText = "PRAGMA journal_mode=WAL;";
|
||||
var resultingMode = (await walCommand.ExecuteScalarAsync()) as string ?? "unknown";
|
||||
if (!string.Equals(resultingMode, "wal", StringComparison.OrdinalIgnoreCase))
|
||||
_logger.LogWarning(
|
||||
"Site SQLite could not enable WAL journal mode (got '{Mode}') — concurrent access may serialize",
|
||||
resultingMode);
|
||||
}
|
||||
// The DDL itself lives in SiteStorageSchema so the Host can apply it to a
|
||||
// LocalDb-managed connection before RegisterReplicated installs the capture
|
||||
// triggers. The service still calls it, so a directly-constructed service
|
||||
// (tests, tooling) remains self-sufficient.
|
||||
SiteStorageSchema.Apply(connection);
|
||||
|
||||
await using var command = connection.CreateCommand();
|
||||
command.CommandText = @"
|
||||
CREATE TABLE IF NOT EXISTS deployed_configurations (
|
||||
instance_unique_name TEXT PRIMARY KEY,
|
||||
config_json TEXT NOT NULL,
|
||||
deployment_id TEXT NOT NULL,
|
||||
revision_hash TEXT NOT NULL,
|
||||
is_enabled INTEGER NOT NULL DEFAULT 1,
|
||||
deployed_at TEXT NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS static_attribute_overrides (
|
||||
instance_unique_name TEXT NOT NULL,
|
||||
attribute_name TEXT NOT NULL,
|
||||
override_value TEXT NOT NULL,
|
||||
updated_at TEXT NOT NULL,
|
||||
PRIMARY KEY (instance_unique_name, attribute_name)
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS shared_scripts (
|
||||
name TEXT PRIMARY KEY,
|
||||
code TEXT NOT NULL,
|
||||
parameter_definitions TEXT,
|
||||
return_definition TEXT,
|
||||
updated_at TEXT NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS external_systems (
|
||||
name TEXT PRIMARY KEY,
|
||||
endpoint_url TEXT NOT NULL,
|
||||
auth_type TEXT NOT NULL,
|
||||
auth_configuration TEXT,
|
||||
method_definitions TEXT,
|
||||
updated_at TEXT NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS database_connections (
|
||||
name TEXT PRIMARY KEY,
|
||||
connection_string TEXT NOT NULL,
|
||||
max_retries INTEGER NOT NULL DEFAULT 3,
|
||||
retry_delay_ms INTEGER NOT NULL DEFAULT 1000,
|
||||
updated_at TEXT NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS notification_lists (
|
||||
name TEXT PRIMARY KEY,
|
||||
recipient_emails TEXT NOT NULL,
|
||||
updated_at TEXT NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS data_connection_definitions (
|
||||
name TEXT PRIMARY KEY,
|
||||
protocol TEXT NOT NULL,
|
||||
configuration TEXT,
|
||||
backup_configuration TEXT,
|
||||
failover_retry_count INTEGER NOT NULL DEFAULT 3,
|
||||
updated_at TEXT NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS smtp_configurations (
|
||||
name TEXT PRIMARY KEY,
|
||||
server TEXT NOT NULL,
|
||||
port INTEGER NOT NULL,
|
||||
auth_mode TEXT NOT NULL,
|
||||
from_address TEXT NOT NULL,
|
||||
username TEXT,
|
||||
password TEXT,
|
||||
oauth_config TEXT,
|
||||
updated_at TEXT NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS native_alarm_state (
|
||||
instance_unique_name TEXT NOT NULL,
|
||||
source_canonical_name TEXT NOT NULL,
|
||||
source_reference TEXT NOT NULL,
|
||||
condition_json TEXT NOT NULL,
|
||||
last_transition_at TEXT NOT NULL,
|
||||
metadata_json TEXT,
|
||||
PRIMARY KEY (instance_unique_name, source_canonical_name, source_reference)
|
||||
);
|
||||
";
|
||||
await command.ExecuteNonQueryAsync();
|
||||
|
||||
// Schema migrations — add columns that may not exist on older databases
|
||||
await MigrateSchemaAsync(connection);
|
||||
|
||||
_logger.LogInformation("Site SQLite storage initialized at {ConnectionString}", _connectionString);
|
||||
}
|
||||
|
||||
private async Task MigrateSchemaAsync(SqliteConnection connection)
|
||||
{
|
||||
// Add backup_configuration and failover_retry_count to data_connection_definitions
|
||||
// (added in primary/backup data connections feature)
|
||||
await TryAddColumnAsync(connection, "data_connection_definitions", "backup_configuration", "TEXT");
|
||||
await TryAddColumnAsync(connection, "data_connection_definitions", "failover_retry_count", "INTEGER NOT NULL DEFAULT 3");
|
||||
|
||||
// Native-alarm display metadata (UA4) — restored on rehydration so a persisted condition
|
||||
// renders fully (type/category/message/values) before the first source snapshot arrives.
|
||||
await TryAddColumnAsync(connection, "native_alarm_state", "metadata_json", "TEXT");
|
||||
|
||||
// Per-external-system call timeout (ExternalSystemGateway Timeout) — carried through the
|
||||
// artifact pipeline so site-side calls honor it; 0 = use the site default.
|
||||
await TryAddColumnAsync(connection, "external_systems", "timeout_seconds", "INTEGER NOT NULL DEFAULT 0");
|
||||
}
|
||||
|
||||
private async Task TryAddColumnAsync(SqliteConnection connection, string table, string column, string type)
|
||||
{
|
||||
try
|
||||
{
|
||||
await using var cmd = connection.CreateCommand();
|
||||
cmd.CommandText = $"ALTER TABLE {table} ADD COLUMN {column} {type}";
|
||||
await cmd.ExecuteNonQueryAsync();
|
||||
_logger.LogInformation("Migrated: added column {Column} to {Table}", column, table);
|
||||
}
|
||||
catch (SqliteException ex) when (ex.Message.Contains("duplicate column"))
|
||||
{
|
||||
// Column already exists — no action needed
|
||||
}
|
||||
_logger.LogInformation("Site SQLite storage initialized");
|
||||
return Task.CompletedTask;
|
||||
}
|
||||
|
||||
// ── Deployed Configuration CRUD ──
|
||||
@@ -203,8 +83,7 @@ public class SiteStorageService
|
||||
/// <returns>A task that resolves to the list of all deployed instance configurations.</returns>
|
||||
public async Task<List<DeployedInstance>> GetAllDeployedConfigsAsync()
|
||||
{
|
||||
await using var connection = new SqliteConnection(_connectionString);
|
||||
await connection.OpenAsync();
|
||||
await using var connection = OpenConnection();
|
||||
|
||||
await using var command = connection.CreateCommand();
|
||||
command.CommandText = @"
|
||||
@@ -245,8 +124,7 @@ public class SiteStorageService
|
||||
string revisionHash,
|
||||
bool isEnabled)
|
||||
{
|
||||
await using var connection = new SqliteConnection(_connectionString);
|
||||
await connection.OpenAsync();
|
||||
await using var connection = OpenConnection();
|
||||
|
||||
await using var command = connection.CreateCommand();
|
||||
command.CommandText = @"
|
||||
@@ -278,9 +156,22 @@ public class SiteStorageService
|
||||
/// clause so the guard is atomic with no application-level read-modify-write.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// This is the standby-node write path for replicated configs. The active-node
|
||||
/// apply path (<see cref="StoreDeployedConfigAsync"/>) remains unguarded and always
|
||||
/// overwrites, because the active node's write is always authoritative.
|
||||
/// <b>This is the reconciliation write path.</b> <c>SiteReconciliationActor</c> runs a
|
||||
/// per-node startup self-heal against central: it asks central what this node should be
|
||||
/// running and fetches anything missing. That fetch races real deploys, so the
|
||||
/// <c>deployed_at</c> guard is what stops a slow reconcile response from overwriting a
|
||||
/// newer config that landed while it was in flight. The active-node apply path
|
||||
/// (<see cref="StoreDeployedConfigAsync"/>) remains unguarded and always overwrites,
|
||||
/// because a deploy is always authoritative.
|
||||
/// <para>
|
||||
/// It was originally the <i>standby</i> write path as well, under notify-and-fetch: the
|
||||
/// standby was told a deploy had happened and fetched the config itself. LocalDb Phase 2
|
||||
/// replaced that with change-data-capture — the config row simply replicates — so the
|
||||
/// standby no longer writes here at all, and last-writer-wins on the primary key (not
|
||||
/// this guard) is what orders concurrent writes between the two nodes. Reconciliation is
|
||||
/// the reason the method survives; do not port the <c>deployed_at</c> guard onto the
|
||||
/// replication path, where it would fight the HLC rather than help it.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <paramref name="deployedAtOverride"/> is exposed for testing so that the exact
|
||||
/// <c>deployed_at</c> value can be controlled without sleeping between calls.
|
||||
@@ -308,8 +199,7 @@ public class SiteStorageService
|
||||
{
|
||||
var deployedAt = (deployedAtOverride ?? DateTimeOffset.UtcNow).ToString("O");
|
||||
|
||||
await using var connection = new SqliteConnection(_connectionString);
|
||||
await connection.OpenAsync();
|
||||
await using var connection = OpenConnection();
|
||||
|
||||
await using var command = connection.CreateCommand();
|
||||
command.CommandText = @"
|
||||
@@ -342,8 +232,7 @@ public class SiteStorageService
|
||||
/// <returns>A task that completes when the configuration and its overrides have been removed.</returns>
|
||||
public async Task RemoveDeployedConfigAsync(string instanceName)
|
||||
{
|
||||
await using var connection = new SqliteConnection(_connectionString);
|
||||
await connection.OpenAsync();
|
||||
await using var connection = OpenConnection();
|
||||
|
||||
await using var transaction = await connection.BeginTransactionAsync();
|
||||
|
||||
@@ -383,8 +272,7 @@ public class SiteStorageService
|
||||
/// <returns>A task that completes when the enabled flag has been updated.</returns>
|
||||
public async Task SetInstanceEnabledAsync(string instanceName, bool isEnabled)
|
||||
{
|
||||
await using var connection = new SqliteConnection(_connectionString);
|
||||
await connection.OpenAsync();
|
||||
await using var connection = OpenConnection();
|
||||
|
||||
await using var command = connection.CreateCommand();
|
||||
command.CommandText = @"
|
||||
@@ -411,8 +299,7 @@ public class SiteStorageService
|
||||
/// <returns>A task that resolves to a dictionary mapping attribute names to their override values.</returns>
|
||||
public async Task<Dictionary<string, string>> GetStaticOverridesAsync(string instanceName)
|
||||
{
|
||||
await using var connection = new SqliteConnection(_connectionString);
|
||||
await connection.OpenAsync();
|
||||
await using var connection = OpenConnection();
|
||||
|
||||
await using var command = connection.CreateCommand();
|
||||
command.CommandText = @"
|
||||
@@ -440,8 +327,7 @@ public class SiteStorageService
|
||||
/// <returns>A task that completes when the override has been saved.</returns>
|
||||
public async Task SetStaticOverrideAsync(string instanceName, string attributeName, string value)
|
||||
{
|
||||
await using var connection = new SqliteConnection(_connectionString);
|
||||
await connection.OpenAsync();
|
||||
await using var connection = OpenConnection();
|
||||
|
||||
await using var command = connection.CreateCommand();
|
||||
command.CommandText = @"
|
||||
@@ -467,8 +353,7 @@ public class SiteStorageService
|
||||
/// <returns>A task that completes when all overrides for the instance have been deleted.</returns>
|
||||
public async Task ClearStaticOverridesAsync(string instanceName)
|
||||
{
|
||||
await using var connection = new SqliteConnection(_connectionString);
|
||||
await connection.OpenAsync();
|
||||
await using var connection = OpenConnection();
|
||||
|
||||
await using var command = connection.CreateCommand();
|
||||
command.CommandText = "DELETE FROM static_attribute_overrides WHERE instance_unique_name = @name";
|
||||
@@ -495,8 +380,7 @@ public class SiteStorageService
|
||||
string instanceName, string sourceCanonicalName, string sourceReference,
|
||||
string conditionJson, DateTimeOffset lastTransitionAt, string? metadataJson = null)
|
||||
{
|
||||
await using var connection = new SqliteConnection(_connectionString);
|
||||
await connection.OpenAsync();
|
||||
await using var connection = OpenConnection();
|
||||
|
||||
await using var command = connection.CreateCommand();
|
||||
command.CommandText = @"
|
||||
@@ -535,8 +419,7 @@ public class SiteStorageService
|
||||
if (rows.Count == 0)
|
||||
return;
|
||||
|
||||
await using var connection = new SqliteConnection(_connectionString);
|
||||
await connection.OpenAsync();
|
||||
await using var connection = OpenConnection();
|
||||
await using var transaction = (SqliteTransaction)await connection.BeginTransactionAsync();
|
||||
|
||||
await using var command = connection.CreateCommand();
|
||||
@@ -580,8 +463,7 @@ public class SiteStorageService
|
||||
/// <returns>A task that completes when the alarm condition row has been deleted.</returns>
|
||||
public async Task DeleteNativeAlarmAsync(string instanceName, string sourceCanonicalName, string sourceReference)
|
||||
{
|
||||
await using var connection = new SqliteConnection(_connectionString);
|
||||
await connection.OpenAsync();
|
||||
await using var connection = OpenConnection();
|
||||
|
||||
await using var command = connection.CreateCommand();
|
||||
command.CommandText = @"
|
||||
@@ -605,8 +487,7 @@ public class SiteStorageService
|
||||
/// <returns>A task that resolves to the list of stored native alarm condition rows for the binding.</returns>
|
||||
public async Task<IReadOnlyList<NativeAlarmRow>> GetNativeAlarmsAsync(string instanceName, string sourceCanonicalName)
|
||||
{
|
||||
await using var connection = new SqliteConnection(_connectionString);
|
||||
await connection.OpenAsync();
|
||||
await using var connection = OpenConnection();
|
||||
|
||||
await using var command = connection.CreateCommand();
|
||||
command.CommandText = @"
|
||||
@@ -637,8 +518,7 @@ public class SiteStorageService
|
||||
/// <returns>A task that completes when all native alarm rows for the instance have been deleted.</returns>
|
||||
public async Task ClearNativeAlarmsForInstanceAsync(string instanceName)
|
||||
{
|
||||
await using var connection = new SqliteConnection(_connectionString);
|
||||
await connection.OpenAsync();
|
||||
await using var connection = OpenConnection();
|
||||
|
||||
await using var command = connection.CreateCommand();
|
||||
command.CommandText = "DELETE FROM native_alarm_state WHERE instance_unique_name = @name";
|
||||
@@ -660,8 +540,7 @@ public class SiteStorageService
|
||||
/// <returns>A task that completes when the shared script has been stored or updated.</returns>
|
||||
public async Task StoreSharedScriptAsync(string name, string code, string? parameterDefs, string? returnDef)
|
||||
{
|
||||
await using var connection = new SqliteConnection(_connectionString);
|
||||
await connection.OpenAsync();
|
||||
await using var connection = OpenConnection();
|
||||
|
||||
await using var command = connection.CreateCommand();
|
||||
command.CommandText = @"
|
||||
@@ -689,8 +568,7 @@ public class SiteStorageService
|
||||
/// <returns>A task that resolves to the list of all stored shared scripts.</returns>
|
||||
public async Task<List<StoredSharedScript>> GetAllSharedScriptsAsync()
|
||||
{
|
||||
await using var connection = new SqliteConnection(_connectionString);
|
||||
await connection.OpenAsync();
|
||||
await using var connection = OpenConnection();
|
||||
|
||||
await using var command = connection.CreateCommand();
|
||||
command.CommandText = "SELECT name, code, parameter_definitions, return_definition FROM shared_scripts";
|
||||
@@ -727,8 +605,7 @@ public class SiteStorageService
|
||||
string name, string endpointUrl, string authType, string? authConfig, string? methodDefs,
|
||||
int timeoutSeconds = 0)
|
||||
{
|
||||
await using var connection = new SqliteConnection(_connectionString);
|
||||
await connection.OpenAsync();
|
||||
await using var connection = OpenConnection();
|
||||
|
||||
await using var command = connection.CreateCommand();
|
||||
command.CommandText = @"
|
||||
@@ -766,8 +643,7 @@ public class SiteStorageService
|
||||
public async Task StoreDatabaseConnectionAsync(
|
||||
string name, string connectionString, int maxRetries, TimeSpan retryDelay)
|
||||
{
|
||||
await using var connection = new SqliteConnection(_connectionString);
|
||||
await connection.OpenAsync();
|
||||
await using var connection = OpenConnection();
|
||||
|
||||
await using var command = connection.CreateCommand();
|
||||
command.CommandText = @"
|
||||
@@ -810,8 +686,7 @@ public class SiteStorageService
|
||||
/// <returns>A task that completes when both tables have been emptied.</returns>
|
||||
public async Task PurgeCentralOnlyNotificationConfigAsync()
|
||||
{
|
||||
await using var connection = new SqliteConnection(_connectionString);
|
||||
await connection.OpenAsync();
|
||||
await using var connection = OpenConnection();
|
||||
|
||||
await using var command = connection.CreateCommand();
|
||||
command.CommandText = @"
|
||||
@@ -834,8 +709,7 @@ public class SiteStorageService
|
||||
public async Task StoreDataConnectionDefinitionAsync(
|
||||
string name, string protocol, string? configJson, string? backupConfigJson = null, int failoverRetryCount = 3)
|
||||
{
|
||||
await using var connection = new SqliteConnection(_connectionString);
|
||||
await connection.OpenAsync();
|
||||
await using var connection = OpenConnection();
|
||||
|
||||
await using var command = connection.CreateCommand();
|
||||
command.CommandText = @"
|
||||
@@ -865,8 +739,7 @@ public class SiteStorageService
|
||||
/// <returns>A task that resolves to the list of all stored data connection definitions.</returns>
|
||||
public async Task<List<StoredDataConnectionDefinition>> GetAllDataConnectionDefinitionsAsync()
|
||||
{
|
||||
await using var connection = new SqliteConnection(_connectionString);
|
||||
await connection.OpenAsync();
|
||||
await using var connection = OpenConnection();
|
||||
|
||||
await using var command = connection.CreateCommand();
|
||||
command.CommandText = "SELECT name, protocol, configuration, backup_configuration, failover_retry_count FROM data_connection_definitions";
|
||||
|
||||
@@ -31,8 +31,9 @@ public class SiteExternalSystemRepository : IExternalSystemRepository
|
||||
public async Task<IReadOnlyList<ExternalSystemDefinition>> GetAllExternalSystemsAsync(
|
||||
CancellationToken cancellationToken = default)
|
||||
{
|
||||
// Already open — SiteStorageService.CreateConnection now hands out a
|
||||
// LocalDb-managed connection. Opening it again would throw.
|
||||
await using var connection = CreateConnection();
|
||||
await connection.OpenAsync(cancellationToken);
|
||||
|
||||
await using var command = connection.CreateCommand();
|
||||
command.CommandText = @"
|
||||
@@ -63,8 +64,9 @@ public class SiteExternalSystemRepository : IExternalSystemRepository
|
||||
public async Task<ExternalSystemDefinition?> GetExternalSystemByNameAsync(
|
||||
string name, CancellationToken cancellationToken = default)
|
||||
{
|
||||
// Already open — SiteStorageService.CreateConnection now hands out a
|
||||
// LocalDb-managed connection. Opening it again would throw.
|
||||
await using var connection = CreateConnection();
|
||||
await connection.OpenAsync(cancellationToken);
|
||||
|
||||
await using var command = connection.CreateCommand();
|
||||
command.CommandText = @"
|
||||
@@ -91,8 +93,9 @@ public class SiteExternalSystemRepository : IExternalSystemRepository
|
||||
if (system is null)
|
||||
return Array.Empty<ExternalSystemMethod>();
|
||||
|
||||
// Already open — SiteStorageService.CreateConnection now hands out a
|
||||
// LocalDb-managed connection. Opening it again would throw.
|
||||
await using var connection = CreateConnection();
|
||||
await connection.OpenAsync(cancellationToken);
|
||||
|
||||
await using var command = connection.CreateCommand();
|
||||
command.CommandText = @"
|
||||
@@ -140,8 +143,9 @@ public class SiteExternalSystemRepository : IExternalSystemRepository
|
||||
public async Task<IReadOnlyList<DatabaseConnectionDefinition>> GetAllDatabaseConnectionsAsync(
|
||||
CancellationToken cancellationToken = default)
|
||||
{
|
||||
// Already open — SiteStorageService.CreateConnection now hands out a
|
||||
// LocalDb-managed connection. Opening it again would throw.
|
||||
await using var connection = CreateConnection();
|
||||
await connection.OpenAsync(cancellationToken);
|
||||
|
||||
await using var command = connection.CreateCommand();
|
||||
command.CommandText = @"
|
||||
@@ -178,8 +182,9 @@ public class SiteExternalSystemRepository : IExternalSystemRepository
|
||||
public async Task<DatabaseConnectionDefinition?> GetDatabaseConnectionByNameAsync(
|
||||
string name, CancellationToken cancellationToken = default)
|
||||
{
|
||||
// Already open — SiteStorageService.CreateConnection now hands out a
|
||||
// LocalDb-managed connection. Opening it again would throw.
|
||||
await using var connection = CreateConnection();
|
||||
await connection.OpenAsync(cancellationToken);
|
||||
|
||||
await using var command = connection.CreateCommand();
|
||||
command.CommandText = @"
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.LocalDb;
|
||||
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Deployment;
|
||||
@@ -14,32 +15,19 @@ namespace ZB.MOM.WW.ScadaBridge.SiteRuntime;
|
||||
public static class ServiceCollectionExtensions
|
||||
{
|
||||
/// <summary>
|
||||
/// Registers Site Runtime services including SiteStorageService for SQLite persistence.
|
||||
/// The caller must register an <see cref="ISiteStorageConnectionProvider"/> or call the
|
||||
/// overload with an explicit connection string.
|
||||
/// Registers Site Runtime services including SiteStorageService, which persists to the
|
||||
/// consolidated site database resolved from <c>ILocalDb</c> (<c>LocalDb:Path</c>).
|
||||
/// </summary>
|
||||
/// <param name="services">The DI service collection to register services into.</param>
|
||||
/// <returns>The same <see cref="IServiceCollection"/> to allow chaining.</returns>
|
||||
public static IServiceCollection AddSiteRuntime(this IServiceCollection services)
|
||||
{
|
||||
// SiteStorageService is registered by the Host using AddSiteRuntime(connectionString)
|
||||
// This overload is for backward compatibility / skeleton placeholder
|
||||
return services;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Registers Site Runtime services with an explicit SQLite connection string.
|
||||
/// </summary>
|
||||
/// <param name="services">The DI service collection to register services into.</param>
|
||||
/// <param name="siteDbConnectionString">The SQLite connection string for the site local storage database.</param>
|
||||
/// <returns>The same <see cref="IServiceCollection"/> to allow chaining.</returns>
|
||||
public static IServiceCollection AddSiteRuntime(this IServiceCollection services, string siteDbConnectionString)
|
||||
{
|
||||
services.AddSingleton(sp =>
|
||||
{
|
||||
var logger = sp.GetRequiredService<ILogger<SiteStorageService>>();
|
||||
return new SiteStorageService(siteDbConnectionString, logger);
|
||||
});
|
||||
// SiteStorageService takes ILocalDb (the consolidated site database at
|
||||
// LocalDb:Path) rather than a connection string, so there is nothing left for a
|
||||
// caller to supply — the string overload is gone and this is the only entry point.
|
||||
services.AddSingleton(sp => new SiteStorageService(
|
||||
sp.GetRequiredService<ILocalDb>(),
|
||||
sp.GetRequiredService<ILogger<SiteStorageService>>()));
|
||||
|
||||
services.AddHostedService<SiteStorageInitializer>();
|
||||
|
||||
@@ -70,7 +58,11 @@ public static class ServiceCollectionExtensions
|
||||
// reconciliation, Tracking.Status audit-only). Central roots never call
|
||||
// AddSiteRuntime, so this stays absent on central — matching the SITE-ONLY
|
||||
// contract. OperationTrackingOptions is bound + validated by the Host's
|
||||
// SiteServiceRegistration site-options block.
|
||||
// SiteServiceRegistration site-options block — but it no longer supplies the
|
||||
// store's database: the store takes ILocalDb (the consolidated site database
|
||||
// at LocalDb:Path), so OperationTrackingOptions.ConnectionString is now only
|
||||
// read by the site config DB. Resolving this store therefore forces ILocalDb
|
||||
// construction, which is what runs SiteLocalDbSetup.OnReady.
|
||||
services.AddSingleton<Commons.Interfaces.IOperationTrackingStore, Tracking.OperationTrackingStore>();
|
||||
|
||||
// Site-local repository implementations backed by SQLite
|
||||
|
||||
@@ -60,13 +60,6 @@ public class SiteRuntimeOptions
|
||||
/// <summary>HTTP timeout (seconds) for fetching a deployment config from central (notify-and-fetch).</summary>
|
||||
public int ConfigFetchTimeoutSeconds { get; set; } = 30;
|
||||
|
||||
/// <summary>
|
||||
/// Bounded attempt count (including the first) for the standby's replicated-config
|
||||
/// fetch; a 2 s fixed delay separates attempts and superseded fetches never retry.
|
||||
/// Consumed by <c>SiteReplicationActor.HandleApplyConfigDeploy</c> (UA2). Default: 3.
|
||||
/// </summary>
|
||||
public int ConfigFetchRetryCount { get; set; } = 3;
|
||||
|
||||
/// <summary>
|
||||
/// Fixed interval (ms) at which an Instance Actor re-sends a tag-subscribe
|
||||
/// request that either failed or whose response was lost (S4/UA6). The retry is
|
||||
|
||||
@@ -53,10 +53,6 @@ public sealed class SiteRuntimeOptionsValidator : OptionsValidatorBase<SiteRunti
|
||||
$"ScadaBridge:SiteRuntime:ConfigFetchTimeoutSeconds must be greater than 0 " +
|
||||
$"(was {options.ConfigFetchTimeoutSeconds}).");
|
||||
|
||||
builder.RequireThat(options.ConfigFetchRetryCount >= 0,
|
||||
$"ScadaBridge:SiteRuntime:ConfigFetchRetryCount must be >= 0 " +
|
||||
$"(was {options.ConfigFetchRetryCount}).");
|
||||
|
||||
builder.RequireThat(options.TagSubscribeRetryIntervalMs > 0,
|
||||
$"ScadaBridge:SiteRuntime:TagSubscribeRetryIntervalMs must be greater than 0 " +
|
||||
$"(was {options.TagSubscribeRetryIntervalMs}); a zero interval hot-loops the tag-subscribe " +
|
||||
|
||||
@@ -7,9 +7,12 @@ namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tracking;
|
||||
public class OperationTrackingOptions
|
||||
{
|
||||
/// <summary>
|
||||
/// Full ADO.NET connection string for the SQLite database (e.g.
|
||||
/// <c>"Data Source=site-tracking.db"</c>). Tests use the
|
||||
/// <c>Mode=Memory;Cache=Shared</c> form to keep the database in-memory.
|
||||
/// MIGRATION-ONLY as of LocalDb Phase 1. <see cref="OperationTrackingStore"/> no longer
|
||||
/// reads this: the tracking table lives in the consolidated site database at
|
||||
/// <c>LocalDb:Path</c>. The only remaining consumer is
|
||||
/// <c>SiteLocalDbLegacyMigrator</c>, which uses it to locate a pre-Phase-1
|
||||
/// <c>site-tracking.db</c> and copy it in once. Safe to delete from a config once that
|
||||
/// node's migration has run.
|
||||
/// </summary>
|
||||
public string ConnectionString { get; set; } = "Data Source=site-tracking.db";
|
||||
|
||||
|
||||
@@ -0,0 +1,105 @@
|
||||
using Microsoft.Data.Sqlite;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tracking;
|
||||
|
||||
/// <summary>
|
||||
/// DDL for the <c>OperationTracking</c> table, extracted from
|
||||
/// <see cref="OperationTrackingStore"/> so it can be applied by whoever owns the
|
||||
/// database file.
|
||||
/// <para>
|
||||
/// Under LocalDb Phase 1 that owner is the Host's <c>AddZbLocalDb</c> onReady callback:
|
||||
/// the table must exist before <c>ILocalDb.RegisterReplicated</c> installs its capture
|
||||
/// triggers, and rows written before registration are never captured or snapshotted.
|
||||
/// The store still calls this during its own initialization, so a store constructed
|
||||
/// directly (tests, tooling) remains self-sufficient.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Deliberately depends only on <c>Microsoft.Data.Sqlite</c>, not on the LocalDb
|
||||
/// library — the Host needs to apply this DDL to a LocalDb-managed connection, but
|
||||
/// nothing about the schema itself is LocalDb-specific.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public static class OperationTrackingSchema
|
||||
{
|
||||
/// <summary>
|
||||
/// Creates the <c>OperationTracking</c> table and its indexes when absent, and
|
||||
/// additively upgrades a table created by an older build. Idempotent — safe to run
|
||||
/// on every startup.
|
||||
/// </summary>
|
||||
/// <param name="connection">An open connection to the database that should hold the table.</param>
|
||||
public static void Apply(SqliteConnection connection)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(connection);
|
||||
|
||||
using (var cmd = connection.CreateCommand())
|
||||
{
|
||||
// TrackedOperationId is a TEXT GUID, which is also what lets this table
|
||||
// replicate as-is: RegisterReplicated requires an explicit primary key and
|
||||
// rejects BLOB columns, and there are none here.
|
||||
cmd.CommandText = """
|
||||
CREATE TABLE IF NOT EXISTS OperationTracking (
|
||||
TrackedOperationId TEXT NOT NULL PRIMARY KEY,
|
||||
Kind TEXT NOT NULL,
|
||||
TargetSummary TEXT NULL,
|
||||
Status TEXT NOT NULL,
|
||||
RetryCount INTEGER NOT NULL DEFAULT 0,
|
||||
LastError TEXT NULL,
|
||||
HttpStatus INTEGER NULL,
|
||||
CreatedAtUtc TEXT NOT NULL,
|
||||
UpdatedAtUtc TEXT NOT NULL,
|
||||
TerminalAtUtc TEXT NULL,
|
||||
SourceInstanceId TEXT NULL,
|
||||
SourceScript TEXT NULL,
|
||||
SourceNode TEXT NULL
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS IX_OperationTracking_Status_Updated
|
||||
ON OperationTracking (Status, UpdatedAtUtc);
|
||||
CREATE INDEX IF NOT EXISTS IX_OperationTracking_UpdatedAt
|
||||
ON OperationTracking (UpdatedAtUtc);
|
||||
""";
|
||||
cmd.ExecuteNonQuery();
|
||||
}
|
||||
|
||||
// SourceNode stamping: additively add the SourceNode column.
|
||||
// CREATE TABLE IF NOT EXISTS above does NOT add columns to an
|
||||
// OperationTracking table that already exists from a pre-SourceNode
|
||||
// build, so a tracking.db created by an older build needs the column
|
||||
// ALTER-ed in. The file is durable across restart/failover by design
|
||||
// (retention window default 7 days), so without this step every
|
||||
// RecordEnqueueAsync on an upgraded deployment would bind $sourceNode
|
||||
// against a missing column and the write would fail.
|
||||
// SQLite has no "ADD COLUMN IF NOT EXISTS"; the column presence is
|
||||
// probed first and the ALTER skipped when already there. The column is
|
||||
// nullable with no default, so any row written before this migration
|
||||
// reads back SourceNode = null (back-compat).
|
||||
AddColumnIfMissing(connection, "SourceNode", "TEXT NULL");
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Additively adds a column to <c>OperationTracking</c> only when it is not
|
||||
/// already present. SQLite lacks <c>ADD COLUMN IF NOT EXISTS</c>, so the
|
||||
/// schema is probed via <c>PRAGMA table_info</c> first. Mirrors the
|
||||
/// <c>SqliteAuditWriter.AddColumnIfMissing</c> precedent.
|
||||
/// </summary>
|
||||
private static void AddColumnIfMissing(
|
||||
SqliteConnection connection, string columnName, string columnDefinition)
|
||||
{
|
||||
using (var probe = connection.CreateCommand())
|
||||
{
|
||||
probe.CommandText =
|
||||
"SELECT COUNT(*) FROM pragma_table_info('OperationTracking') WHERE name = $name";
|
||||
probe.Parameters.AddWithValue("$name", columnName);
|
||||
if (Convert.ToInt32(probe.ExecuteScalar()) > 0)
|
||||
{
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
using var alter = connection.CreateCommand();
|
||||
// Column name + definition are caller-controlled constants, never user
|
||||
// input — safe to interpolate (parameters are not permitted in DDL).
|
||||
alter.CommandText =
|
||||
$"ALTER TABLE OperationTracking ADD COLUMN {columnName} {columnDefinition}";
|
||||
alter.ExecuteNonQuery();
|
||||
}
|
||||
}
|
||||
@@ -1,7 +1,7 @@
|
||||
using System.Globalization;
|
||||
using Microsoft.Data.Sqlite;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.LocalDb;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types;
|
||||
|
||||
@@ -39,7 +39,7 @@ public class OperationTrackingStore : IOperationTrackingStore, IAsyncDisposable,
|
||||
// _writeGate. Readers do NOT share this connection or gate; see GetStatusAsync.
|
||||
private readonly SqliteConnection _writeConnection;
|
||||
private readonly SemaphoreSlim _writeGate = new(1, 1);
|
||||
private readonly string _connectionString;
|
||||
private readonly ILocalDb _localDb;
|
||||
private readonly ILogger<OperationTrackingStore> _logger;
|
||||
|
||||
// Dispose-once state shared by the sync Dispose and async
|
||||
@@ -51,94 +51,44 @@ public class OperationTrackingStore : IOperationTrackingStore, IAsyncDisposable,
|
||||
private int _disposeState;
|
||||
|
||||
/// <summary>
|
||||
/// Initializes the tracking store, opens the SQLite connection, and applies the schema.
|
||||
/// Initializes the tracking store over the consolidated site database and applies the schema.
|
||||
/// </summary>
|
||||
/// <param name="options">Tracking store configuration (connection string, retention window).</param>
|
||||
/// <param name="localDb">
|
||||
/// The consolidated site database. Every connection it hands out is already open and carries
|
||||
/// the per-connection pragmas plus the <c>zb_hlc_next()</c> UDF that
|
||||
/// <c>OperationTracking</c>'s capture triggers call — which is exactly why the store no longer
|
||||
/// opens its own <see cref="SqliteConnection"/> from a connection string. A raw connection
|
||||
/// would lack the UDF and every write to the replicated table would fail closed.
|
||||
/// </param>
|
||||
/// <param name="logger">Logger for diagnostics.</param>
|
||||
/// <remarks>
|
||||
/// <see cref="OperationTrackingOptions.ConnectionString"/> no longer feeds this store — the
|
||||
/// database location is <c>LocalDb:Path</c>. The option remains bound for the purge/retention
|
||||
/// settings that share the class.
|
||||
/// </remarks>
|
||||
public OperationTrackingStore(
|
||||
IOptions<OperationTrackingOptions> options,
|
||||
ILocalDb localDb,
|
||||
ILogger<OperationTrackingStore> logger)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
ArgumentNullException.ThrowIfNull(localDb);
|
||||
ArgumentNullException.ThrowIfNull(logger);
|
||||
|
||||
_logger = logger;
|
||||
_connectionString = options.Value.ConnectionString;
|
||||
_writeConnection = new SqliteConnection(_connectionString);
|
||||
_writeConnection.Open();
|
||||
_localDb = localDb;
|
||||
// Already open — CreateConnection returns a live, pragma-configured,
|
||||
// UDF-registered connection. Calling Open() on it again would throw.
|
||||
_writeConnection = localDb.CreateConnection();
|
||||
InitializeSchema();
|
||||
}
|
||||
|
||||
private void InitializeSchema()
|
||||
{
|
||||
using var cmd = _writeConnection.CreateCommand();
|
||||
cmd.CommandText = """
|
||||
CREATE TABLE IF NOT EXISTS OperationTracking (
|
||||
TrackedOperationId TEXT NOT NULL PRIMARY KEY,
|
||||
Kind TEXT NOT NULL,
|
||||
TargetSummary TEXT NULL,
|
||||
Status TEXT NOT NULL,
|
||||
RetryCount INTEGER NOT NULL DEFAULT 0,
|
||||
LastError TEXT NULL,
|
||||
HttpStatus INTEGER NULL,
|
||||
CreatedAtUtc TEXT NOT NULL,
|
||||
UpdatedAtUtc TEXT NOT NULL,
|
||||
TerminalAtUtc TEXT NULL,
|
||||
SourceInstanceId TEXT NULL,
|
||||
SourceScript TEXT NULL,
|
||||
SourceNode TEXT NULL
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS IX_OperationTracking_Status_Updated
|
||||
ON OperationTracking (Status, UpdatedAtUtc);
|
||||
CREATE INDEX IF NOT EXISTS IX_OperationTracking_UpdatedAt
|
||||
ON OperationTracking (UpdatedAtUtc);
|
||||
""";
|
||||
cmd.ExecuteNonQuery();
|
||||
|
||||
// SourceNode stamping: additively add the SourceNode column.
|
||||
// CREATE TABLE IF NOT EXISTS above does NOT add columns to an
|
||||
// OperationTracking table that already exists from a pre-SourceNode
|
||||
// build, so a tracking.db created by an older build needs the column
|
||||
// ALTER-ed in. The file is durable across restart/failover by design
|
||||
// (retention window default 7 days), so without this step every
|
||||
// RecordEnqueueAsync on an upgraded deployment would bind $sourceNode
|
||||
// against a missing column and the write would fail.
|
||||
// SQLite has no "ADD COLUMN IF NOT EXISTS"; the column presence is
|
||||
// probed first and the ALTER skipped when already there. The column is
|
||||
// nullable with no default, so any row written before this migration
|
||||
// reads back SourceNode = null (back-compat).
|
||||
//
|
||||
// NOTE: This is the FIRST idempotent column-upgrade in
|
||||
// OperationTrackingStore — prior schema changes pre-dated any
|
||||
// production rollout and relied solely on CREATE TABLE IF NOT EXISTS.
|
||||
// The helper mirrors the SqliteAuditWriter precedent.
|
||||
AddColumnIfMissing("SourceNode", "TEXT NULL");
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Additively adds a column to <c>OperationTracking</c> only when it is not
|
||||
/// already present. SQLite lacks <c>ADD COLUMN IF NOT EXISTS</c>, so the
|
||||
/// schema is probed via <c>PRAGMA table_info</c> first. Idempotent — safe
|
||||
/// to run on every <see cref="InitializeSchema"/>. Mirrors the
|
||||
/// <c>SqliteAuditWriter.AddColumnIfMissing</c> precedent.
|
||||
/// </summary>
|
||||
private void AddColumnIfMissing(string columnName, string columnDefinition)
|
||||
{
|
||||
using var probe = _writeConnection.CreateCommand();
|
||||
probe.CommandText = "SELECT COUNT(*) FROM pragma_table_info('OperationTracking') WHERE name = $name";
|
||||
probe.Parameters.AddWithValue("$name", columnName);
|
||||
var exists = Convert.ToInt32(probe.ExecuteScalar()) > 0;
|
||||
if (exists)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
using var alter = _writeConnection.CreateCommand();
|
||||
// Column name + definition are caller-controlled constants, never user
|
||||
// input — safe to interpolate (parameters are not permitted in DDL).
|
||||
alter.CommandText = $"ALTER TABLE OperationTracking ADD COLUMN {columnName} {columnDefinition}";
|
||||
alter.ExecuteNonQuery();
|
||||
}
|
||||
// Schema lives in OperationTrackingSchema so the Host's AddZbLocalDb onReady
|
||||
// callback can create this table in the consolidated site database before
|
||||
// RegisterReplicated installs its capture triggers. In the host this call is
|
||||
// therefore always a no-op — onReady runs while ILocalDb is being constructed,
|
||||
// which is strictly before this constructor can receive it. It stays because it
|
||||
// keeps a directly-constructed store (tests, tooling) self-sufficient, and the
|
||||
// DDL is idempotent.
|
||||
private void InitializeSchema() => OperationTrackingSchema.Apply(_writeConnection);
|
||||
|
||||
/// <inheritdoc/>
|
||||
public async Task RecordEnqueueAsync(
|
||||
@@ -289,14 +239,12 @@ public class OperationTrackingStore : IOperationTrackingStore, IAsyncDisposable,
|
||||
{
|
||||
ObjectDisposedException.ThrowIf(Volatile.Read(ref _disposeState) != 0, this);
|
||||
|
||||
// Reads open a fresh, ungated SqliteConnection so a
|
||||
// long-running write doesn't block status queries. The connection
|
||||
// string is shared with the writer; SQLite handles cross-connection
|
||||
// isolation natively (a reader sees a consistent snapshot via the
|
||||
// shared cache lock for in-memory DBs, or a WAL snapshot for file DBs).
|
||||
// Mirrors the SiteStorageService precedent.
|
||||
await using var readConnection = new SqliteConnection(_connectionString);
|
||||
await readConnection.OpenAsync(ct).ConfigureAwait(false);
|
||||
// Reads open a fresh, ungated connection so a long-running write doesn't
|
||||
// block status queries. It comes from the same ILocalDb as the writer;
|
||||
// SQLite handles cross-connection isolation natively (readers see a WAL
|
||||
// snapshot). Mirrors the SiteStorageService precedent. Already open — do
|
||||
// not call OpenAsync on it.
|
||||
await using var readConnection = _localDb.CreateConnection();
|
||||
|
||||
await using var cmd = readConnection.CreateCommand();
|
||||
cmd.CommandText = """
|
||||
@@ -376,8 +324,8 @@ public class OperationTrackingStore : IOperationTrackingStore, IAsyncDisposable,
|
||||
// the standalone IX_OperationTracking_UpdatedAt index — UpdatedAtUtc is
|
||||
// the cursor. (The composite (Status, UpdatedAtUtc) index cannot satisfy a
|
||||
// status-less UpdatedAtUtc range scan; this dedicated index does.)
|
||||
await using var readConnection = new SqliteConnection(_connectionString);
|
||||
await readConnection.OpenAsync(ct).ConfigureAwait(false);
|
||||
// Already open — do not call OpenAsync on it.
|
||||
await using var readConnection = _localDb.CreateConnection();
|
||||
|
||||
await using var cmd = readConnection.CreateCommand();
|
||||
|
||||
|
||||
@@ -23,6 +23,7 @@
|
||||
<PackageReference Include="Microsoft.Extensions.Logging.Abstractions" />
|
||||
<PackageReference Include="Microsoft.Extensions.Options" />
|
||||
<PackageReference Include="ZB.MOM.WW.Configuration" />
|
||||
<PackageReference Include="ZB.MOM.WW.LocalDb" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
|
||||
@@ -1,194 +0,0 @@
|
||||
using Microsoft.Extensions.Logging;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Observability;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.StoreAndForward;
|
||||
|
||||
/// <summary>
|
||||
/// Async replication of buffer operations to standby node.
|
||||
///
|
||||
/// - Forwards add/remove/park operations to standby via a replication handler.
|
||||
/// - No ack wait (fire-and-forget per design).
|
||||
/// - Standby applies operations to its own SQLite.
|
||||
/// - On failover, standby resumes delivery from its replicated state.
|
||||
/// </summary>
|
||||
public class ReplicationService
|
||||
{
|
||||
private readonly StoreAndForwardOptions _options;
|
||||
private readonly ILogger<ReplicationService> _logger;
|
||||
private Func<ReplicationOperation, Task>? _replicationHandler;
|
||||
|
||||
/// <summary>Initializes a new instance of <see cref="ReplicationService"/>.</summary>
|
||||
/// <param name="options">Store-and-forward configuration options.</param>
|
||||
/// <param name="logger">Logger instance.</param>
|
||||
public ReplicationService(
|
||||
StoreAndForwardOptions options,
|
||||
ILogger<ReplicationService> logger)
|
||||
{
|
||||
_options = options;
|
||||
_logger = logger;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Sets the handler for forwarding replication operations to the standby node.
|
||||
/// Typically wraps Akka Tell to the standby's replication actor.
|
||||
/// </summary>
|
||||
/// <param name="handler">The async delegate that forwards each replication operation to the standby.</param>
|
||||
public void SetReplicationHandler(Func<ReplicationOperation, Task> handler)
|
||||
{
|
||||
_replicationHandler = handler;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Replicates an enqueue operation to standby (fire-and-forget).
|
||||
/// </summary>
|
||||
/// <param name="message">The message that was enqueued on the active node.</param>
|
||||
public void ReplicateEnqueue(StoreAndForwardMessage message)
|
||||
{
|
||||
if (!_options.ReplicationEnabled || _replicationHandler == null) return;
|
||||
|
||||
FireAndForget(new ReplicationOperation(
|
||||
ReplicationOperationType.Add,
|
||||
message.Id,
|
||||
message));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Replicates a remove operation to standby (fire-and-forget).
|
||||
/// </summary>
|
||||
/// <param name="messageId">The identifier of the message to remove from the standby buffer.</param>
|
||||
public void ReplicateRemove(string messageId)
|
||||
{
|
||||
if (!_options.ReplicationEnabled || _replicationHandler == null) return;
|
||||
|
||||
FireAndForget(new ReplicationOperation(
|
||||
ReplicationOperationType.Remove,
|
||||
messageId,
|
||||
null));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Replicates a park operation to standby (fire-and-forget).
|
||||
/// </summary>
|
||||
/// <param name="message">The message that was parked on the active node.</param>
|
||||
public void ReplicatePark(StoreAndForwardMessage message)
|
||||
{
|
||||
if (!_options.ReplicationEnabled || _replicationHandler == null) return;
|
||||
|
||||
FireAndForget(new ReplicationOperation(
|
||||
ReplicationOperationType.Park,
|
||||
message.Id,
|
||||
message));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Replicates an operator-initiated requeue (a parked
|
||||
/// message moved back to the pending queue) to standby (fire-and-forget). The
|
||||
/// carried message reflects the active node's post-requeue state (Pending,
|
||||
/// retry_count = 0) so the standby's copy can be brought into sync.
|
||||
/// </summary>
|
||||
/// <param name="message">The message in its post-requeue (Pending, retry_count=0) state.</param>
|
||||
public void ReplicateRequeue(StoreAndForwardMessage message)
|
||||
{
|
||||
if (!_options.ReplicationEnabled || _replicationHandler == null) return;
|
||||
|
||||
FireAndForget(new ReplicationOperation(
|
||||
ReplicationOperationType.Requeue,
|
||||
message.Id,
|
||||
message));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Applies a replicated operation received from the active node.
|
||||
/// Used by the standby node to keep its SQLite in sync.
|
||||
///
|
||||
/// Add/Park/Requeue are applied as <b>upserts</b> (<see cref="StoreAndForwardStorage.UpsertMessageAsync"/>),
|
||||
/// not blind INSERT/UPDATE: the full message rides in every one of those operations,
|
||||
/// so a Park/Requeue whose original Add was lost (fire-and-forget replication is
|
||||
/// best-effort) self-heals by materialising the row, and a duplicate Add (e.g.
|
||||
/// re-issued by an anti-entropy resync) applies newest-wins instead of throwing a
|
||||
/// primary-key violation. Remove is a plain delete.
|
||||
/// </summary>
|
||||
/// <param name="operation">The replication operation to apply.</param>
|
||||
/// <param name="storage">The standby node's store-and-forward storage to update.</param>
|
||||
/// <returns>A task representing the asynchronous apply operation.</returns>
|
||||
public async Task ApplyReplicatedOperationAsync(
|
||||
ReplicationOperation operation,
|
||||
StoreAndForwardStorage storage)
|
||||
{
|
||||
switch (operation.OperationType)
|
||||
{
|
||||
case ReplicationOperationType.Add when operation.Message != null:
|
||||
await storage.UpsertMessageAsync(operation.Message);
|
||||
break;
|
||||
|
||||
case ReplicationOperationType.Remove:
|
||||
await storage.RemoveMessageAsync(operation.MessageId);
|
||||
break;
|
||||
|
||||
case ReplicationOperationType.Park when operation.Message != null:
|
||||
operation.Message.Status = StoreAndForwardMessageStatus.Parked;
|
||||
await storage.UpsertMessageAsync(operation.Message);
|
||||
break;
|
||||
|
||||
case ReplicationOperationType.Requeue when operation.Message != null:
|
||||
operation.Message.Status = StoreAndForwardMessageStatus.Pending;
|
||||
operation.Message.RetryCount = 0;
|
||||
await storage.UpsertMessageAsync(operation.Message);
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
private void FireAndForget(ReplicationOperation operation)
|
||||
{
|
||||
// Invoked inline, NOT via Task.Run: the handler is a non-blocking Akka
|
||||
// Tell, and thread-pool hand-off destroyed Add/Remove ordering for the
|
||||
// same message id (arch review 02). Inline invocation preserves issue
|
||||
// order; Akka's per-sender/receiver guarantee preserves it on the wire.
|
||||
try
|
||||
{
|
||||
var task = _replicationHandler!.Invoke(operation);
|
||||
if (!task.IsCompletedSuccessfully)
|
||||
{
|
||||
task.ContinueWith(t =>
|
||||
{
|
||||
ScadaBridgeTelemetry.RecordReplicationFailure();
|
||||
_logger.LogWarning(t.Exception,
|
||||
"Replication of {OpType} for message {MessageId} failed (best-effort); standby buffer may be diverging",
|
||||
operation.OperationType, operation.MessageId);
|
||||
},
|
||||
TaskContinuationOptions.OnlyOnFaulted);
|
||||
}
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
ScadaBridgeTelemetry.RecordReplicationFailure();
|
||||
_logger.LogWarning(ex,
|
||||
"Replication of {OpType} for message {MessageId} failed (best-effort); standby buffer may be diverging",
|
||||
operation.OperationType, operation.MessageId);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Represents a buffer operation to be replicated to standby.
|
||||
/// </summary>
|
||||
public record ReplicationOperation(
|
||||
ReplicationOperationType OperationType,
|
||||
string MessageId,
|
||||
StoreAndForwardMessage? Message);
|
||||
|
||||
/// <summary>
|
||||
/// Types of buffer operations that are replicated.
|
||||
/// </summary>
|
||||
public enum ReplicationOperationType
|
||||
{
|
||||
Add,
|
||||
Remove,
|
||||
Park,
|
||||
/// <summary>
|
||||
/// An operator moved a parked message back to the pending
|
||||
/// queue. The standby resets its matching row to Pending with retry_count = 0.
|
||||
/// </summary>
|
||||
Requeue
|
||||
}
|
||||
@@ -1,6 +1,7 @@
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.LocalDb;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteEventLogging;
|
||||
|
||||
@@ -15,21 +16,18 @@ public static class ServiceCollectionExtensions
|
||||
/// <returns>The same <paramref name="services"/> collection, for chaining.</returns>
|
||||
public static IServiceCollection AddStoreAndForward(this IServiceCollection services)
|
||||
{
|
||||
services.AddSingleton<StoreAndForwardStorage>(sp =>
|
||||
{
|
||||
var options = sp.GetRequiredService<IOptions<StoreAndForwardOptions>>().Value;
|
||||
var logger = sp.GetRequiredService<ILogger<StoreAndForwardStorage>>();
|
||||
return new StoreAndForwardStorage(
|
||||
$"Data Source={options.SqliteDbPath}",
|
||||
logger);
|
||||
});
|
||||
// The buffer now lives in the consolidated LocalDb database at LocalDb:Path, not
|
||||
// at StoreAndForwardOptions.SqliteDbPath — that option survives only as the
|
||||
// migrator's source location (Task 8).
|
||||
services.AddSingleton<StoreAndForwardStorage>(sp => new StoreAndForwardStorage(
|
||||
sp.GetRequiredService<ILocalDb>(),
|
||||
sp.GetRequiredService<ILogger<StoreAndForwardStorage>>()));
|
||||
|
||||
services.AddSingleton<StoreAndForwardService>(sp =>
|
||||
{
|
||||
var storage = sp.GetRequiredService<StoreAndForwardStorage>();
|
||||
var options = sp.GetRequiredService<IOptions<StoreAndForwardOptions>>().Value;
|
||||
var logger = sp.GetRequiredService<ILogger<StoreAndForwardService>>();
|
||||
var replication = sp.GetRequiredService<ReplicationService>();
|
||||
// Wire the cached-call lifecycle
|
||||
// observer + site identity through DI so the S&F retry loop emits
|
||||
// per-attempt + terminal telemetry under the same TrackedOperationId
|
||||
@@ -54,19 +52,11 @@ public static class ServiceCollectionExtensions
|
||||
storage,
|
||||
options,
|
||||
logger,
|
||||
replication,
|
||||
cachedCallObserver,
|
||||
siteId,
|
||||
siteEventLogger);
|
||||
});
|
||||
|
||||
services.AddSingleton<ReplicationService>(sp =>
|
||||
{
|
||||
var options = sp.GetRequiredService<IOptions<StoreAndForwardOptions>>().Value;
|
||||
var logger = sp.GetRequiredService<ILogger<ReplicationService>>();
|
||||
return new ReplicationService(options, logger);
|
||||
});
|
||||
|
||||
return services;
|
||||
}
|
||||
|
||||
|
||||
@@ -5,12 +5,15 @@ namespace ZB.MOM.WW.ScadaBridge.StoreAndForward;
|
||||
/// </summary>
|
||||
public class StoreAndForwardOptions
|
||||
{
|
||||
/// <summary>Path to the SQLite database for S&F message persistence.</summary>
|
||||
/// <summary>
|
||||
/// Path to the legacy standalone store-and-forward SQLite file. <b>Migration-only.</b>
|
||||
/// The buffer itself lives in the consolidated LocalDb database (<c>LocalDb:Path</c>)
|
||||
/// as of LocalDb Phase 2; this path is read once at boot by
|
||||
/// <c>SiteLocalDbLegacyMigrator</c> to drain a pre-Phase-2 file, and is otherwise
|
||||
/// unused. A node that has already migrated may leave it set or unset.
|
||||
/// </summary>
|
||||
public string SqliteDbPath { get; set; } = "./data/store-and-forward.db";
|
||||
|
||||
/// <summary>Whether to replicate buffer operations to standby node.</summary>
|
||||
public bool ReplicationEnabled { get; set; } = true;
|
||||
|
||||
/// <summary>Default retry interval for messages without per-source settings.</summary>
|
||||
public TimeSpan DefaultRetryInterval { get; set; } = TimeSpan.FromSeconds(30);
|
||||
|
||||
|
||||
@@ -5,21 +5,23 @@ namespace ZB.MOM.WW.ScadaBridge.StoreAndForward;
|
||||
/// <summary>
|
||||
/// Validates <see cref="StoreAndForwardOptions"/> at startup. The retry intervals
|
||||
/// feed the background sweep timer (a zero/negative period trips
|
||||
/// <see cref="ArgumentOutOfRangeException"/> in the timer constructor) and the
|
||||
/// SQLite path is opened for the S&F buffer; an empty path yields an opaque
|
||||
/// connection failure at first enqueue. Registered with <c>ValidateOnStart()</c>
|
||||
/// so a bad <c>ScadaBridge:StoreAndForward</c> section fails fast at boot with a
|
||||
/// clear, key-naming message.
|
||||
/// <see cref="ArgumentOutOfRangeException"/> in the timer constructor). Registered
|
||||
/// with <c>ValidateOnStart()</c> so a bad <c>ScadaBridge:StoreAndForward</c> section
|
||||
/// fails fast at boot with a clear, key-naming message.
|
||||
/// <para>
|
||||
/// <see cref="StoreAndForwardOptions.SqliteDbPath"/> is deliberately NOT validated.
|
||||
/// Before LocalDb Phase 2 it was the live buffer file, so an empty value produced an
|
||||
/// opaque connection failure at first enqueue; the buffer now lives in the
|
||||
/// consolidated LocalDb database and the key survives only as the legacy migration
|
||||
/// source. An empty value there is a legitimate "nothing to migrate", so requiring
|
||||
/// it would make every already-migrated node carry a dead key forever.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public sealed class StoreAndForwardOptionsValidator : OptionsValidatorBase<StoreAndForwardOptions>
|
||||
{
|
||||
/// <inheritdoc />
|
||||
protected override void Validate(ValidationBuilder builder, StoreAndForwardOptions options)
|
||||
{
|
||||
builder.RequireThat(!string.IsNullOrWhiteSpace(options.SqliteDbPath),
|
||||
"ScadaBridge:StoreAndForward:SqliteDbPath must be a non-empty path; " +
|
||||
"it is the SQLite file backing the store-and-forward buffer.");
|
||||
|
||||
builder.RequireThat(options.DefaultRetryInterval > TimeSpan.Zero,
|
||||
$"ScadaBridge:StoreAndForward:DefaultRetryInterval must be a positive duration " +
|
||||
$"(was {options.DefaultRetryInterval}); it is the default per-message retry interval.");
|
||||
|
||||
@@ -0,0 +1,124 @@
|
||||
using Microsoft.Data.Sqlite;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.StoreAndForward;
|
||||
|
||||
/// <summary>
|
||||
/// DDL for the <c>sf_messages</c> table, extracted from
|
||||
/// <see cref="StoreAndForwardStorage"/> so it can be applied by whoever owns the
|
||||
/// database file.
|
||||
/// <para>
|
||||
/// Deliberately depends only on <c>Microsoft.Data.Sqlite</c>, not on the LocalDb library.
|
||||
/// The Host applies this DDL to a LocalDb-managed connection before
|
||||
/// <c>RegisterReplicated</c> installs the capture triggers; nothing about the schema
|
||||
/// itself is LocalDb-specific, and the store still calls it so a directly-constructed
|
||||
/// store (tests, tooling) remains self-sufficient.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public static class StoreAndForwardSchema
|
||||
{
|
||||
/// <summary>
|
||||
/// Creates the <c>sf_messages</c> table and its indexes when absent, and additively
|
||||
/// upgrades a table created by an older build. Idempotent — safe to run on every
|
||||
/// startup.
|
||||
/// </summary>
|
||||
/// <param name="connection">An open connection to the database that should hold the table.</param>
|
||||
public static void Apply(SqliteConnection connection)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(connection);
|
||||
|
||||
using (var command = connection.CreateCommand())
|
||||
{
|
||||
command.CommandText = @"
|
||||
CREATE TABLE IF NOT EXISTS sf_messages (
|
||||
id TEXT PRIMARY KEY,
|
||||
category INTEGER NOT NULL,
|
||||
target TEXT NOT NULL,
|
||||
payload_json TEXT NOT NULL,
|
||||
retry_count INTEGER NOT NULL DEFAULT 0,
|
||||
max_retries INTEGER NOT NULL DEFAULT 50,
|
||||
retry_interval_ms INTEGER NOT NULL DEFAULT 30000,
|
||||
created_at TEXT NOT NULL,
|
||||
last_attempt_at TEXT,
|
||||
status INTEGER NOT NULL DEFAULT 0,
|
||||
last_error TEXT,
|
||||
origin_instance TEXT
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_sf_messages_status ON sf_messages(status);
|
||||
CREATE INDEX IF NOT EXISTS idx_sf_messages_category ON sf_messages(category);
|
||||
";
|
||||
command.ExecuteNonQuery();
|
||||
}
|
||||
|
||||
// Additively add the execution_id /
|
||||
// source_script columns. CREATE TABLE IF NOT EXISTS above does NOT add
|
||||
// columns to a table that already exists from before these fields, so a
|
||||
// databases created by an older build needs the columns ALTER-ed in.
|
||||
// SQLite has no "ADD COLUMN IF NOT EXISTS"; the column presence is
|
||||
// probed first and the ALTER skipped when already there. Both columns
|
||||
// are nullable with no default, so any row buffered before this
|
||||
// migration reads back ExecutionId/SourceScript = null (back-compat).
|
||||
AddColumnIfMissing(connection, "execution_id", "TEXT");
|
||||
AddColumnIfMissing(connection, "source_script", "TEXT");
|
||||
|
||||
// Additively add the
|
||||
// parent_execution_id column the same way — a sibling to execution_id.
|
||||
// Nullable with no default, so any row buffered before this migration
|
||||
// reads back ParentExecutionId = null (back-compat).
|
||||
AddColumnIfMissing(connection, "parent_execution_id", "TEXT");
|
||||
|
||||
// Additively add the epoch-ms sibling of last_attempt_at. The
|
||||
// ISO-8601 text column stays authoritative for reads / back-compat; this
|
||||
// INTEGER column drives the due predicate so the sweep no longer parses
|
||||
// julianday() per row.
|
||||
AddColumnIfMissing(connection, "last_attempt_at_ms", "INTEGER");
|
||||
|
||||
// One-time backfill for rows persisted before the ms column existed: derive
|
||||
// epoch-ms from the text timestamp. The "... IS NULL" guard makes this run
|
||||
// once per legacy DB and never match again — this is the only remaining
|
||||
// julianday() use in the store.
|
||||
using (var backfill = connection.CreateCommand())
|
||||
{
|
||||
backfill.CommandText = @"
|
||||
UPDATE sf_messages
|
||||
SET last_attempt_at_ms = CAST((julianday(last_attempt_at) - 2440587.5) * 86400000 AS INTEGER)
|
||||
WHERE last_attempt_at IS NOT NULL AND last_attempt_at_ms IS NULL";
|
||||
backfill.ExecuteNonQuery();
|
||||
}
|
||||
|
||||
// Covering index for the due query (status filter + ms ordering column).
|
||||
// Created after the ALTER above so the column exists.
|
||||
using (var dueIndex = connection.CreateCommand())
|
||||
{
|
||||
dueIndex.CommandText =
|
||||
"CREATE INDEX IF NOT EXISTS idx_sf_messages_status_due ON sf_messages(status, last_attempt_at_ms)";
|
||||
dueIndex.ExecuteNonQuery();
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Adds a column to <c>sf_messages</c>
|
||||
/// only when it is not already present. SQLite lacks <c>ADD COLUMN IF NOT
|
||||
/// EXISTS</c>, so the schema is probed via <c>PRAGMA table_info</c> first.
|
||||
/// Idempotent — safe to run on every <see cref="Apply"/>.
|
||||
/// </summary>
|
||||
private static void AddColumnIfMissing(
|
||||
SqliteConnection connection, string columnName, string columnType)
|
||||
{
|
||||
using (var probe = connection.CreateCommand())
|
||||
{
|
||||
probe.CommandText = "SELECT COUNT(*) FROM pragma_table_info('sf_messages') WHERE name = @name";
|
||||
probe.Parameters.AddWithValue("@name", columnName);
|
||||
if (Convert.ToInt32(probe.ExecuteScalar()) > 0)
|
||||
{
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
using var alter = connection.CreateCommand();
|
||||
// Column name + type are caller-controlled constants, never user input —
|
||||
// safe to interpolate (parameters are not permitted in DDL).
|
||||
alter.CommandText = $"ALTER TABLE sf_messages ADD COLUMN {columnName} {columnType}";
|
||||
alter.ExecuteNonQuery();
|
||||
}
|
||||
}
|
||||
@@ -36,7 +36,6 @@ public class StoreAndForwardService
|
||||
{
|
||||
private readonly StoreAndForwardStorage _storage;
|
||||
private readonly StoreAndForwardOptions _options;
|
||||
private readonly ReplicationService? _replication;
|
||||
private readonly ILogger<StoreAndForwardService> _logger;
|
||||
/// <summary>
|
||||
/// Site-side observer notified
|
||||
@@ -107,7 +106,7 @@ public class StoreAndForwardService
|
||||
/// <c>null</c> when no sweep is currently running. Captured when the timer
|
||||
/// callback starts a sweep so <see cref="StopAsync"/> can wait for it to
|
||||
/// finish before the host disposes downstream dependencies
|
||||
/// (<see cref="_storage"/>, <see cref="_replication"/>) that the sweep is
|
||||
/// (<see cref="_storage"/>) that the sweep is
|
||||
/// still touching. Written from the timer thread and from
|
||||
/// <see cref="StopAsync"/>, so reads are synchronised via the
|
||||
/// <see cref="Volatile"/> APIs.
|
||||
@@ -227,7 +226,6 @@ public class StoreAndForwardService
|
||||
/// <param name="storage">The storage backend for buffered messages.</param>
|
||||
/// <param name="options">Configuration options.</param>
|
||||
/// <param name="logger">Logger instance.</param>
|
||||
/// <param name="replication">Optional replication service for standby synchronization.</param>
|
||||
/// <param name="cachedCallObserver">Optional observer for cached call lifecycle events.</param>
|
||||
/// <param name="siteId">The site identifier this service belongs to.</param>
|
||||
/// <param name="siteEventLogger">
|
||||
@@ -240,7 +238,6 @@ public class StoreAndForwardService
|
||||
StoreAndForwardStorage storage,
|
||||
StoreAndForwardOptions options,
|
||||
ILogger<StoreAndForwardService> logger,
|
||||
ReplicationService? replication = null,
|
||||
ICachedCallLifecycleObserver? cachedCallObserver = null,
|
||||
string siteId = "",
|
||||
ISiteEventLogger? siteEventLogger = null)
|
||||
@@ -248,7 +245,6 @@ public class StoreAndForwardService
|
||||
_storage = storage;
|
||||
_options = options;
|
||||
_logger = logger;
|
||||
_replication = replication;
|
||||
_cachedCallObserver = cachedCallObserver;
|
||||
_siteId = string.IsNullOrWhiteSpace(siteId) ? UnknownSiteSentinel : siteId;
|
||||
_siteEventLogger = siteEventLogger;
|
||||
@@ -651,7 +647,6 @@ public class StoreAndForwardService
|
||||
private async Task BufferAsync(StoreAndForwardMessage message)
|
||||
{
|
||||
await _storage.EnqueueAsync(message);
|
||||
_replication?.ReplicateEnqueue(message);
|
||||
Interlocked.Increment(ref _bufferedCount);
|
||||
}
|
||||
|
||||
@@ -802,7 +797,6 @@ public class StoreAndForwardService
|
||||
if (success)
|
||||
{
|
||||
await _storage.RemoveMessageAsync(message.Id);
|
||||
_replication?.ReplicateRemove(message.Id);
|
||||
Interlocked.Decrement(ref _bufferedCount);
|
||||
RaiseActivity("Delivered", message.Category,
|
||||
$"Delivered to {message.Target} after {message.RetryCount} retries");
|
||||
@@ -833,7 +827,6 @@ public class StoreAndForwardService
|
||||
return RetryOutcome.Skipped;
|
||||
}
|
||||
Interlocked.Decrement(ref _bufferedCount);
|
||||
_replication?.ReplicatePark(message);
|
||||
RaiseActivity("Parked", message.Category,
|
||||
$"Permanent failure for {message.Target}: handler returned false");
|
||||
|
||||
@@ -869,7 +862,6 @@ public class StoreAndForwardService
|
||||
return RetryOutcome.Skipped;
|
||||
}
|
||||
Interlocked.Decrement(ref _bufferedCount);
|
||||
_replication?.ReplicatePark(message);
|
||||
RaiseActivity("Parked", message.Category,
|
||||
$"Max retries ({message.MaxRetries}) reached for {message.Target}");
|
||||
_logger.LogWarning(
|
||||
@@ -1077,17 +1069,12 @@ public class StoreAndForwardService
|
||||
/// <summary>
|
||||
/// Retries a parked message (moves back to pending queue).
|
||||
///
|
||||
/// An operator requeue is a buffer state change and is
|
||||
/// replicated to the standby (as a <see cref="ReplicationOperationType.Requeue"/>)
|
||||
/// so a failover preserves the operator's retry intent.
|
||||
/// An operator requeue is a buffer state change, and the peer picks it up because
|
||||
/// <c>sf_messages</c> is a replicated table: the row update is captured and shipped
|
||||
/// like any other write, so a failover preserves the operator's retry intent. This
|
||||
/// used to be an explicit Requeue operation sent to the standby.
|
||||
/// The activity-log entry carries the message's true
|
||||
/// category rather than a hard-coded one.
|
||||
/// The parked row is captured <i>before</i> the local
|
||||
/// requeue write rather than re-read after it, so a concurrent
|
||||
/// <c>RemoveMessageAsync</c> or <c>DiscardParkedMessageAsync</c> running
|
||||
/// between the two storage calls cannot leave the standby in <c>Parked</c>
|
||||
/// while the active node has already requeued — we always have the row in
|
||||
/// hand for the <c>Requeue</c> replication.
|
||||
/// </summary>
|
||||
/// <param name="messageId">The identifier of the message to retry.</param>
|
||||
/// <returns>True if successfully retried, false otherwise.</returns>
|
||||
@@ -1117,7 +1104,6 @@ public class StoreAndForwardService
|
||||
captured.RetryCount = 0;
|
||||
captured.LastError = null;
|
||||
captured.LastAttemptAt = null;
|
||||
_replication?.ReplicateRequeue(captured);
|
||||
|
||||
RaiseActivity("Retry", captured.Category,
|
||||
$"Parked message {messageId} moved back to queue");
|
||||
@@ -1127,9 +1113,10 @@ public class StoreAndForwardService
|
||||
/// <summary>
|
||||
/// Permanently discards a parked message.
|
||||
///
|
||||
/// An operator discard is a buffer removal and is replicated
|
||||
/// to the standby (as a <see cref="ReplicationOperationType.Remove"/>) so the
|
||||
/// discarded message does not reappear after a failover.
|
||||
/// An operator discard is a buffer removal. The delete is captured as a tombstone on
|
||||
/// the replicated <c>sf_messages</c> table and carries the later HLC, so the discarded
|
||||
/// message cannot reappear after a failover regardless of arrival order. This used to
|
||||
/// be an explicit Remove operation sent to the standby.
|
||||
/// The activity-log entry carries the message's true
|
||||
/// category rather than a hard-coded one.
|
||||
/// </summary>
|
||||
@@ -1143,7 +1130,6 @@ public class StoreAndForwardService
|
||||
var success = await _storage.DiscardParkedMessageAsync(messageId);
|
||||
if (success)
|
||||
{
|
||||
_replication?.ReplicateRemove(messageId);
|
||||
RaiseActivity("Discard", message?.Category ?? StoreAndForwardCategory.ExternalSystem,
|
||||
$"Parked message {messageId} discarded");
|
||||
}
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
using Microsoft.Data.Sqlite;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using ZB.MOM.WW.LocalDb;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.StoreAndForward;
|
||||
@@ -11,17 +12,25 @@ namespace ZB.MOM.WW.ScadaBridge.StoreAndForward;
|
||||
/// </summary>
|
||||
public class StoreAndForwardStorage
|
||||
{
|
||||
private readonly string _connectionString;
|
||||
private readonly ILocalDb _localDb;
|
||||
private readonly ILogger<StoreAndForwardStorage> _logger;
|
||||
|
||||
/// <summary>
|
||||
/// Initializes a new instance of <see cref="StoreAndForwardStorage"/> with the given SQLite connection string.
|
||||
/// Initializes the store over the consolidated site database and applies the schema.
|
||||
/// </summary>
|
||||
/// <param name="connectionString">SQLite connection string for the store-and-forward database.</param>
|
||||
/// <param name="localDb">
|
||||
/// The consolidated site database. Every connection it hands out is already open and carries
|
||||
/// the per-connection pragmas (including <c>busy_timeout</c>) plus the <c>zb_hlc_next()</c>
|
||||
/// UDF that <c>sf_messages</c>' capture triggers call — which is exactly why the store no
|
||||
/// longer opens its own <see cref="SqliteConnection"/> from a connection string. A raw
|
||||
/// connection would lack the UDF and every write to the replicated table would fail closed.
|
||||
/// </param>
|
||||
/// <param name="logger">Logger for diagnostics.</param>
|
||||
public StoreAndForwardStorage(string connectionString, ILogger<StoreAndForwardStorage> logger)
|
||||
public StoreAndForwardStorage(ILocalDb localDb, ILogger<StoreAndForwardStorage> logger)
|
||||
{
|
||||
_connectionString = connectionString;
|
||||
ArgumentNullException.ThrowIfNull(localDb);
|
||||
ArgumentNullException.ThrowIfNull(logger);
|
||||
_localDb = localDb;
|
||||
_logger = logger;
|
||||
}
|
||||
|
||||
@@ -29,164 +38,34 @@ public class StoreAndForwardStorage
|
||||
/// Creates the sf_messages table if it does not exist.
|
||||
/// </summary>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
public async Task InitializeAsync()
|
||||
public Task InitializeAsync()
|
||||
{
|
||||
EnsureDatabaseDirectoryExists();
|
||||
// No directory creation and no journal-mode pragma here any more: LocalDb owns
|
||||
// the file (it creates the directory) and sets WAL plus the per-connection
|
||||
// pragmas on every connection it hands out.
|
||||
using var connection = OpenConnection();
|
||||
|
||||
await using var connection = await OpenConnectionAsync();
|
||||
|
||||
// Enable WAL so the concurrent writers this store has by design (script
|
||||
// enqueues, the retry sweep's per-target lanes, standby replication applies,
|
||||
// central pull queries) read/write without "database is locked". WAL is
|
||||
// persistent + file-scoped; in-memory DBs report "memory" instead — harmless,
|
||||
// so it is not asserted here.
|
||||
await using (var walCmd = connection.CreateCommand())
|
||||
{
|
||||
walCmd.CommandText = "PRAGMA journal_mode=WAL";
|
||||
await walCmd.ExecuteNonQueryAsync();
|
||||
}
|
||||
|
||||
await using var command = connection.CreateCommand();
|
||||
command.CommandText = @"
|
||||
CREATE TABLE IF NOT EXISTS sf_messages (
|
||||
id TEXT PRIMARY KEY,
|
||||
category INTEGER NOT NULL,
|
||||
target TEXT NOT NULL,
|
||||
payload_json TEXT NOT NULL,
|
||||
retry_count INTEGER NOT NULL DEFAULT 0,
|
||||
max_retries INTEGER NOT NULL DEFAULT 50,
|
||||
retry_interval_ms INTEGER NOT NULL DEFAULT 30000,
|
||||
created_at TEXT NOT NULL,
|
||||
last_attempt_at TEXT,
|
||||
status INTEGER NOT NULL DEFAULT 0,
|
||||
last_error TEXT,
|
||||
origin_instance TEXT
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_sf_messages_status ON sf_messages(status);
|
||||
CREATE INDEX IF NOT EXISTS idx_sf_messages_category ON sf_messages(category);
|
||||
";
|
||||
await command.ExecuteNonQueryAsync();
|
||||
|
||||
// Additively add the execution_id /
|
||||
// source_script columns. CREATE TABLE IF NOT EXISTS above does NOT add
|
||||
// columns to a table that already exists from before these fields, so a
|
||||
// databases created by an older build needs the columns ALTER-ed in.
|
||||
// SQLite has no "ADD COLUMN IF NOT EXISTS"; the column presence is
|
||||
// probed first and the ALTER skipped when already there. Both columns
|
||||
// are nullable with no default, so any row buffered before this
|
||||
// migration reads back ExecutionId/SourceScript = null (back-compat).
|
||||
await AddColumnIfMissingAsync(connection, "execution_id", "TEXT");
|
||||
await AddColumnIfMissingAsync(connection, "source_script", "TEXT");
|
||||
|
||||
// Additively add the
|
||||
// parent_execution_id column the same way — a sibling to execution_id.
|
||||
// Nullable with no default, so any row buffered before this migration
|
||||
// reads back ParentExecutionId = null (back-compat).
|
||||
await AddColumnIfMissingAsync(connection, "parent_execution_id", "TEXT");
|
||||
|
||||
// Additively add the epoch-ms sibling of last_attempt_at. The
|
||||
// ISO-8601 text column stays authoritative for reads / back-compat; this
|
||||
// INTEGER column drives the due predicate so the sweep no longer parses
|
||||
// julianday() per row.
|
||||
await AddColumnIfMissingAsync(connection, "last_attempt_at_ms", "INTEGER");
|
||||
|
||||
// One-time backfill for rows persisted before the ms column existed: derive
|
||||
// epoch-ms from the text timestamp. The "... IS NULL" guard makes this run
|
||||
// once per legacy DB and never match again — this is the only remaining
|
||||
// julianday() use in the store.
|
||||
await using (var backfill = connection.CreateCommand())
|
||||
{
|
||||
backfill.CommandText = @"
|
||||
UPDATE sf_messages
|
||||
SET last_attempt_at_ms = CAST((julianday(last_attempt_at) - 2440587.5) * 86400000 AS INTEGER)
|
||||
WHERE last_attempt_at IS NOT NULL AND last_attempt_at_ms IS NULL";
|
||||
await backfill.ExecuteNonQueryAsync();
|
||||
}
|
||||
|
||||
// Covering index for the due query (status filter + ms ordering column).
|
||||
// Created after the ALTER above so the column exists.
|
||||
await using (var dueIndex = connection.CreateCommand())
|
||||
{
|
||||
dueIndex.CommandText =
|
||||
"CREATE INDEX IF NOT EXISTS idx_sf_messages_status_due ON sf_messages(status, last_attempt_at_ms)";
|
||||
await dueIndex.ExecuteNonQueryAsync();
|
||||
}
|
||||
// The DDL itself lives in StoreAndForwardSchema so the Host can apply it to a
|
||||
// LocalDb-managed connection before RegisterReplicated installs the capture
|
||||
// triggers. The store still calls it, so a directly-constructed store (tests,
|
||||
// tooling) remains self-sufficient.
|
||||
StoreAndForwardSchema.Apply(connection);
|
||||
|
||||
_logger.LogInformation("Store-and-forward SQLite storage initialized");
|
||||
return Task.CompletedTask;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Adds a column to <c>sf_messages</c>
|
||||
/// only when it is not already present. SQLite lacks <c>ADD COLUMN IF NOT
|
||||
/// EXISTS</c>, so the schema is probed via <c>PRAGMA table_info</c> first.
|
||||
/// Idempotent — safe to run on every <see cref="InitializeAsync"/>.
|
||||
/// CreateConnection returns an ALREADY-OPEN, pragma-configured connection with the
|
||||
/// <c>zb_hlc_next()</c> UDF registered. Calling <c>OpenAsync</c> on it throws, and a raw
|
||||
/// <see cref="SqliteConnection"/> would lack the UDF, making every capture trigger fail
|
||||
/// closed.
|
||||
/// </summary>
|
||||
private static async Task AddColumnIfMissingAsync(
|
||||
SqliteConnection connection, string columnName, string columnType)
|
||||
{
|
||||
await using var probe = connection.CreateCommand();
|
||||
probe.CommandText = "SELECT COUNT(*) FROM pragma_table_info('sf_messages') WHERE name = @name";
|
||||
probe.Parameters.AddWithValue("@name", columnName);
|
||||
var exists = Convert.ToInt32(await probe.ExecuteScalarAsync()) > 0;
|
||||
if (exists)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
await using var alter = connection.CreateCommand();
|
||||
// Column name + type are caller-controlled constants, never user input —
|
||||
// safe to interpolate (parameters are not permitted in DDL).
|
||||
alter.CommandText = $"ALTER TABLE sf_messages ADD COLUMN {columnName} {columnType}";
|
||||
await alter.ExecuteNonQueryAsync();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Ensures the directory for a file-backed SQLite database exists. SQLite creates
|
||||
/// the database file on demand but not its parent directory, so a configured path
|
||||
/// such as "./data/store-and-forward.db" fails to open ("unable to open database
|
||||
/// file") when the "data" directory does not yet exist. In-memory databases and
|
||||
/// bare filenames in the working directory have no directory to create and are
|
||||
/// skipped.
|
||||
/// </summary>
|
||||
private void EnsureDatabaseDirectoryExists()
|
||||
{
|
||||
var builder = new SqliteConnectionStringBuilder(_connectionString);
|
||||
if (builder.Mode == SqliteOpenMode.Memory)
|
||||
return;
|
||||
|
||||
var dataSource = builder.DataSource;
|
||||
if (string.IsNullOrEmpty(dataSource) || dataSource == ":memory:")
|
||||
return;
|
||||
|
||||
var directory = System.IO.Path.GetDirectoryName(System.IO.Path.GetFullPath(dataSource));
|
||||
if (!string.IsNullOrEmpty(directory) && !System.IO.Directory.Exists(directory))
|
||||
{
|
||||
System.IO.Directory.CreateDirectory(directory);
|
||||
_logger.LogInformation("Created store-and-forward database directory: {Directory}", directory);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Opens a connection with a 5s busy_timeout. Concurrent writers exist by
|
||||
/// design (script enqueues, the sweep's lanes, standby replication applies,
|
||||
/// central pull queries); with connection-per-operation the pragma must be
|
||||
/// set per open (it is per-connection, and Microsoft.Data.Sqlite pooling
|
||||
/// makes the extra statement cheap on a pooled physical connection).
|
||||
/// </summary>
|
||||
private async Task<SqliteConnection> OpenConnectionAsync()
|
||||
{
|
||||
var connection = new SqliteConnection(_connectionString);
|
||||
await connection.OpenAsync();
|
||||
await using var pragma = connection.CreateCommand();
|
||||
pragma.CommandText = "PRAGMA busy_timeout = 5000";
|
||||
await pragma.ExecuteNonQueryAsync();
|
||||
return connection;
|
||||
}
|
||||
private SqliteConnection OpenConnection() => _localDb.CreateConnection();
|
||||
|
||||
/// <summary>
|
||||
/// INSERT statement for a full message row. Shared by <see cref="EnqueueAsync"/>
|
||||
/// and <see cref="ReplaceAllAsync"/>; bind with <see cref="BindMessageParameters"/>
|
||||
/// and <see cref="UpsertMessageAsync"/>; bind with <see cref="BindMessageParameters"/>
|
||||
/// so the column list and the parameters never drift apart.
|
||||
/// </summary>
|
||||
private const string InsertMessageSql = @"
|
||||
@@ -235,7 +114,7 @@ public class StoreAndForwardStorage
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
public async Task EnqueueAsync(StoreAndForwardMessage message)
|
||||
{
|
||||
await using var connection = await OpenConnectionAsync();
|
||||
await using var connection = OpenConnection();
|
||||
|
||||
await using var cmd = connection.CreateCommand();
|
||||
cmd.CommandText = InsertMessageSql;
|
||||
@@ -255,7 +134,7 @@ public class StoreAndForwardStorage
|
||||
/// <returns>The oldest-first page (at most <paramref name="limit"/> rows) and whether more rows exist beyond it.</returns>
|
||||
public async Task<(List<StoreAndForwardMessage> Messages, bool Truncated)> GetAllMessagesAsync(int limit)
|
||||
{
|
||||
await using var connection = await OpenConnectionAsync();
|
||||
await using var connection = OpenConnection();
|
||||
|
||||
await using var cmd = connection.CreateCommand();
|
||||
cmd.CommandText = @"
|
||||
@@ -272,39 +151,6 @@ public class StoreAndForwardStorage
|
||||
return (rows.Take(limit).ToList(), truncated);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Atomically replaces the entire buffer with <paramref name="messages"/> in a
|
||||
/// single transaction (delete-all then insert-all). Standby-side anti-entropy
|
||||
/// apply: a peer-join resync overwrites the standby's divergent copy with the
|
||||
/// active node's authoritative snapshot. <b>Never call on an active node</b> —
|
||||
/// it discards every in-flight row.
|
||||
/// </summary>
|
||||
/// <param name="messages">The full buffer snapshot to install.</param>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
public async Task ReplaceAllAsync(IReadOnlyList<StoreAndForwardMessage> messages)
|
||||
{
|
||||
await using var connection = await OpenConnectionAsync();
|
||||
await using var transaction = (SqliteTransaction)await connection.BeginTransactionAsync();
|
||||
|
||||
await using (var deleteCmd = connection.CreateCommand())
|
||||
{
|
||||
deleteCmd.Transaction = transaction;
|
||||
deleteCmd.CommandText = "DELETE FROM sf_messages";
|
||||
await deleteCmd.ExecuteNonQueryAsync();
|
||||
}
|
||||
|
||||
foreach (var message in messages)
|
||||
{
|
||||
await using var insertCmd = connection.CreateCommand();
|
||||
insertCmd.Transaction = transaction;
|
||||
insertCmd.CommandText = InsertMessageSql;
|
||||
BindMessageParameters(insertCmd, message);
|
||||
await insertCmd.ExecuteNonQueryAsync();
|
||||
}
|
||||
|
||||
await transaction.CommitAsync();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Inserts a message, or updates every mutable column in place if a row with the
|
||||
/// same id already exists (<c>ON CONFLICT(id) DO UPDATE</c>). Used by the standby
|
||||
@@ -318,7 +164,7 @@ public class StoreAndForwardStorage
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
public async Task UpsertMessageAsync(StoreAndForwardMessage message)
|
||||
{
|
||||
await using var connection = await OpenConnectionAsync();
|
||||
await using var connection = OpenConnection();
|
||||
|
||||
await using var cmd = connection.CreateCommand();
|
||||
cmd.CommandText = @"
|
||||
@@ -375,7 +221,7 @@ public class StoreAndForwardStorage
|
||||
/// <returns>A task that resolves to the list of messages due for retry, ordered by creation time ascending.</returns>
|
||||
public async Task<List<StoreAndForwardMessage>> GetMessagesForRetryAsync(int limit = 0)
|
||||
{
|
||||
await using var connection = await OpenConnectionAsync();
|
||||
await using var connection = OpenConnection();
|
||||
|
||||
await using var cmd = connection.CreateCommand();
|
||||
cmd.CommandText = @"
|
||||
@@ -409,7 +255,7 @@ public class StoreAndForwardStorage
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
public async Task UpdateMessageAsync(StoreAndForwardMessage message)
|
||||
{
|
||||
await using var connection = await OpenConnectionAsync();
|
||||
await using var connection = OpenConnection();
|
||||
|
||||
await using var cmd = connection.CreateCommand();
|
||||
cmd.CommandText = @"
|
||||
@@ -446,7 +292,7 @@ public class StoreAndForwardStorage
|
||||
StoreAndForwardMessage message,
|
||||
StoreAndForwardMessageStatus expectedStatus)
|
||||
{
|
||||
await using var connection = await OpenConnectionAsync();
|
||||
await using var connection = OpenConnection();
|
||||
|
||||
await using var cmd = connection.CreateCommand();
|
||||
cmd.CommandText = @"
|
||||
@@ -479,7 +325,7 @@ public class StoreAndForwardStorage
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
public async Task RemoveMessageAsync(string messageId)
|
||||
{
|
||||
await using var connection = await OpenConnectionAsync();
|
||||
await using var connection = OpenConnection();
|
||||
|
||||
await using var cmd = connection.CreateCommand();
|
||||
cmd.CommandText = "DELETE FROM sf_messages WHERE id = @id";
|
||||
@@ -500,7 +346,7 @@ public class StoreAndForwardStorage
|
||||
int pageNumber = 1,
|
||||
int pageSize = 50)
|
||||
{
|
||||
await using var connection = await OpenConnectionAsync();
|
||||
await using var connection = OpenConnection();
|
||||
|
||||
await using var transaction = (SqliteTransaction)await connection.BeginTransactionAsync();
|
||||
|
||||
@@ -545,7 +391,7 @@ public class StoreAndForwardStorage
|
||||
/// <returns>A task that resolves to <c>true</c> if the message was found and reset to Pending; <c>false</c> if not found or not in Parked status.</returns>
|
||||
public async Task<bool> RetryParkedMessageAsync(string messageId)
|
||||
{
|
||||
await using var connection = await OpenConnectionAsync();
|
||||
await using var connection = OpenConnection();
|
||||
|
||||
await using var cmd = connection.CreateCommand();
|
||||
cmd.CommandText = @"
|
||||
@@ -569,7 +415,7 @@ public class StoreAndForwardStorage
|
||||
/// <returns>A task that resolves to <c>true</c> if the message was found and deleted; <c>false</c> if not found or not in Parked status.</returns>
|
||||
public async Task<bool> DiscardParkedMessageAsync(string messageId)
|
||||
{
|
||||
await using var connection = await OpenConnectionAsync();
|
||||
await using var connection = OpenConnection();
|
||||
|
||||
await using var cmd = connection.CreateCommand();
|
||||
cmd.CommandText = "DELETE FROM sf_messages WHERE id = @id AND status = @parked";
|
||||
@@ -586,7 +432,7 @@ public class StoreAndForwardStorage
|
||||
/// <returns>A task that resolves to a dictionary mapping each category to its pending message count.</returns>
|
||||
public async Task<Dictionary<StoreAndForwardCategory, int>> GetBufferDepthByCategoryAsync()
|
||||
{
|
||||
await using var connection = await OpenConnectionAsync();
|
||||
await using var connection = OpenConnection();
|
||||
|
||||
await using var cmd = connection.CreateCommand();
|
||||
cmd.CommandText = @"
|
||||
@@ -616,7 +462,7 @@ public class StoreAndForwardStorage
|
||||
/// <returns>A task that resolves to the number of messages whose origin instance matches <paramref name="instanceName"/>.</returns>
|
||||
public async Task<int> GetMessageCountByOriginInstanceAsync(string instanceName)
|
||||
{
|
||||
await using var connection = await OpenConnectionAsync();
|
||||
await using var connection = OpenConnection();
|
||||
|
||||
await using var cmd = connection.CreateCommand();
|
||||
cmd.CommandText = @"
|
||||
@@ -635,7 +481,7 @@ public class StoreAndForwardStorage
|
||||
/// <returns>A task that resolves to the matching message, or <c>null</c> if not found.</returns>
|
||||
public async Task<StoreAndForwardMessage?> GetMessageByIdAsync(string messageId)
|
||||
{
|
||||
await using var connection = await OpenConnectionAsync();
|
||||
await using var connection = OpenConnection();
|
||||
|
||||
await using var cmd = connection.CreateCommand();
|
||||
cmd.CommandText = @"
|
||||
@@ -656,7 +502,7 @@ public class StoreAndForwardStorage
|
||||
/// <returns>A task that resolves to the number of messages currently in Parked status.</returns>
|
||||
public async Task<int> GetParkedMessageCountAsync()
|
||||
{
|
||||
await using var conn = await OpenConnectionAsync();
|
||||
await using var conn = OpenConnection();
|
||||
await using var cmd = conn.CreateCommand();
|
||||
cmd.CommandText = "SELECT COUNT(*) FROM sf_messages WHERE status = @parked";
|
||||
cmd.Parameters.AddWithValue("@parked", (int)StoreAndForwardMessageStatus.Parked);
|
||||
@@ -674,7 +520,7 @@ public class StoreAndForwardStorage
|
||||
/// <returns>A task that resolves to the oldest parked row's creation time, or <c>null</c> if none are parked.</returns>
|
||||
public async Task<DateTimeOffset?> GetOldestParkedCreatedAtAsync()
|
||||
{
|
||||
await using var conn = await OpenConnectionAsync();
|
||||
await using var conn = OpenConnection();
|
||||
await using var cmd = conn.CreateCommand();
|
||||
cmd.CommandText = "SELECT MIN(created_at) FROM sf_messages WHERE status = @parked";
|
||||
cmd.Parameters.AddWithValue("@parked", (int)StoreAndForwardMessageStatus.Parked);
|
||||
@@ -691,7 +537,7 @@ public class StoreAndForwardStorage
|
||||
/// <returns>A task that resolves to the count of messages with the specified status.</returns>
|
||||
public async Task<int> GetMessageCountByStatusAsync(StoreAndForwardMessageStatus status)
|
||||
{
|
||||
await using var connection = await OpenConnectionAsync();
|
||||
await using var connection = OpenConnection();
|
||||
|
||||
await using var cmd = connection.CreateCommand();
|
||||
cmd.CommandText = "SELECT COUNT(*) FROM sf_messages WHERE status = @status";
|
||||
|
||||
+29
-6
@@ -1,8 +1,10 @@
|
||||
using Akka.Actor;
|
||||
using Microsoft.EntityFrameworkCore;
|
||||
using Microsoft.Extensions.Configuration;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.LocalDb;
|
||||
using ZB.MOM.WW.ScadaBridge.AuditLog.Central;
|
||||
using ZB.MOM.WW.ScadaBridge.AuditLog.Site;
|
||||
using ZB.MOM.WW.ScadaBridge.AuditLog.Site.Telemetry;
|
||||
@@ -58,6 +60,8 @@ public sealed class CombinedTelemetryHarness : IAsyncDisposable
|
||||
public IServiceProvider ServiceProvider { get; }
|
||||
|
||||
private readonly MsSqlMigrationFixture _fixture;
|
||||
private readonly string _trackingDbPath;
|
||||
private readonly ServiceProvider _trackingLocalDbProvider;
|
||||
private bool _disposed;
|
||||
|
||||
public CombinedTelemetryHarness(
|
||||
@@ -88,13 +92,23 @@ public sealed class CombinedTelemetryHarness : IAsyncDisposable
|
||||
SqliteWriter, Ring, new NoOpAuditWriteFailureCounter(),
|
||||
NullLogger<FallbackAuditWriter>.Instance);
|
||||
|
||||
// The tracking store now runs on the consolidated ILocalDb rather than its own
|
||||
// connection string, so it needs a real database FILE — LocalDbOptions.Path is
|
||||
// a filesystem path and there is no in-memory mode. Unique per harness, torn
|
||||
// down in DisposeAsync alongside its WAL sidecars.
|
||||
_trackingDbPath = Path.Combine(
|
||||
Path.GetTempPath(), $"tracking-g-{Guid.NewGuid():N}.db");
|
||||
_trackingLocalDbProvider = new ServiceCollection()
|
||||
.AddZbLocalDb(new ConfigurationBuilder()
|
||||
.AddInMemoryCollection(new Dictionary<string, string?>
|
||||
{
|
||||
["LocalDb:Path"] = _trackingDbPath,
|
||||
})
|
||||
.Build())
|
||||
.BuildServiceProvider();
|
||||
|
||||
TrackingStore = new OperationTrackingStore(
|
||||
Options.Create(new OperationTrackingOptions
|
||||
{
|
||||
// Same shared-in-memory pattern as the audit writer.
|
||||
ConnectionString =
|
||||
$"Data Source=file:tracking-g-{Guid.NewGuid():N}?mode=memory&cache=shared",
|
||||
}),
|
||||
_trackingLocalDbProvider.GetRequiredService<ILocalDb>(),
|
||||
NullLogger<OperationTrackingStore>.Instance);
|
||||
|
||||
// Central wiring: real repositories backed by the MSSQL fixture's DB.
|
||||
@@ -165,6 +179,15 @@ public sealed class CombinedTelemetryHarness : IAsyncDisposable
|
||||
_disposed = true;
|
||||
await SqliteWriter.DisposeAsync().ConfigureAwait(false);
|
||||
await TrackingStore.DisposeAsync().ConfigureAwait(false);
|
||||
|
||||
// Dispose the ILocalDb before deleting the file — it holds an open master
|
||||
// connection anchoring the WAL.
|
||||
await _trackingLocalDbProvider.DisposeAsync().ConfigureAwait(false);
|
||||
foreach (var suffix in new[] { "", "-wal", "-shm" })
|
||||
{
|
||||
try { File.Delete(_trackingDbPath + suffix); } catch { /* best effort */ }
|
||||
}
|
||||
|
||||
if (ServiceProvider is IAsyncDisposable asyncSp)
|
||||
{
|
||||
await asyncSp.DisposeAsync().ConfigureAwait(false);
|
||||
|
||||
+173
-162
@@ -35,6 +35,7 @@ using ZB.MOM.WW.ScadaBridge.NotificationOutbox.Delivery;
|
||||
using ZB.MOM.WW.ScadaBridge.NotificationOutbox.Messages;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts;
|
||||
using ZB.MOM.WW.ScadaBridge.StoreAndForward;
|
||||
using ZB.MOM.WW.ScadaBridge.TestSupport;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.AuditLog.Tests.Integration;
|
||||
|
||||
@@ -197,174 +198,184 @@ public class ParentExecutionIdCorrelationTests : TestKit, IClassFixture<MsSqlMig
|
||||
stubClient);
|
||||
|
||||
// Site Store-and-Forward — Notify.Send buffers a NotificationSubmit here.
|
||||
using var safKeepAlive = new Microsoft.Data.Sqlite.SqliteConnection(
|
||||
$"Data Source=parentexec-saf-{Guid.NewGuid():N};Mode=Memory;Cache=Shared");
|
||||
safKeepAlive.Open();
|
||||
var safStorage = new StoreAndForwardStorage(
|
||||
safKeepAlive.ConnectionString, NullLogger<StoreAndForwardStorage>.Instance);
|
||||
await safStorage.InitializeAsync();
|
||||
var storeAndForward = new StoreAndForwardService(
|
||||
safStorage,
|
||||
new StoreAndForwardOptions
|
||||
// The storage takes an ILocalDb and LocalDb has no in-memory mode, so this is a
|
||||
// real temp file, disposed (the master connection anchors the WAL) and deleted in
|
||||
// the finally below.
|
||||
var safLocalDb = TestLocalDb.CreateTemp("parentexec-saf");
|
||||
var safDbPath = safLocalDb.Path;
|
||||
try
|
||||
{
|
||||
var safStorage = new StoreAndForwardStorage(
|
||||
safLocalDb.Db, NullLogger<StoreAndForwardStorage>.Instance);
|
||||
await safStorage.InitializeAsync();
|
||||
var storeAndForward = new StoreAndForwardService(
|
||||
safStorage,
|
||||
new StoreAndForwardOptions
|
||||
{
|
||||
DefaultRetryInterval = TimeSpan.Zero,
|
||||
DefaultMaxRetries = 3,
|
||||
RetryTimerInterval = TimeSpan.FromMinutes(10),
|
||||
},
|
||||
NullLogger<StoreAndForwardService>.Instance);
|
||||
|
||||
// ── Outbound external-system client (routed run): sync Call succeeds,
|
||||
// CachedCall completes immediately (WasBuffered=false) so the script
|
||||
// helper emits the Submit + Attempted + CachedResolve lifecycle. ──
|
||||
var externalClient = Substitute.For<IExternalSystemClient>();
|
||||
externalClient
|
||||
.CallAsync(ExternalSystemName, ExternalMethodName,
|
||||
Arg.Any<IReadOnlyDictionary<string, object?>?>(), Arg.Any<CancellationToken>())
|
||||
.Returns(new ExternalCallResult(true, "{\"ok\":true}", null));
|
||||
externalClient
|
||||
.CachedCallAsync(ExternalSystemName, ExternalMethodName,
|
||||
Arg.Any<IReadOnlyDictionary<string, object?>?>(),
|
||||
Arg.Any<string?>(), Arg.Any<CancellationToken>(),
|
||||
Arg.Any<ZB.MOM.WW.ScadaBridge.Commons.Types.TrackedOperationId?>(),
|
||||
Arg.Any<Guid?>(), Arg.Any<string?>(), Arg.Any<Guid?>())
|
||||
.Returns(new ExternalCallResult(true, "{\"ok\":true}", null, WasBuffered: false));
|
||||
|
||||
// ── The routing transport stand-in: builds the routed ScriptRuntimeContext
|
||||
// carrying RouteToCallRequest.ParentExecutionId — exactly what the
|
||||
// production site handler (DeploymentManagerActor) does. ──
|
||||
var router = new BridgingInstanceRouter(
|
||||
siteId,
|
||||
externalClient,
|
||||
siteAuditWriter,
|
||||
cachedForwarder,
|
||||
storeAndForward);
|
||||
|
||||
// ── The inbound API method script: it calls Route.Call into the site
|
||||
// instance. The real InboundScriptExecutor binds the inbound request's
|
||||
// ExecutionId onto the RouteHelper, so the routed call carries it as
|
||||
// ParentExecutionId. ──
|
||||
var inboundMethod = new ZB.MOM.WW.ScadaBridge.Commons.Entities.InboundApi.ApiMethod(
|
||||
"submitOrder",
|
||||
$"return await Route.To(\"{RoutedInstanceCode}\").Call(\"{RoutedScriptName}\", new {{ order = 7 }});");
|
||||
var locator = Substitute.For<IInstanceLocator>();
|
||||
locator.GetSiteIdForInstanceAsync(RoutedInstanceCode, Arg.Any<CancellationToken>())
|
||||
.Returns(siteId);
|
||||
var scriptExecutor = new InboundScriptExecutor(
|
||||
NullLogger<InboundScriptExecutor>.Instance,
|
||||
new ServiceCollection().BuildServiceProvider());
|
||||
Assert.True(scriptExecutor.CompileAndRegister(inboundMethod));
|
||||
|
||||
// ── Act — issue the inbound HTTP request through a TestHost pipeline
|
||||
// fronted by the real AuditWriteMiddleware. The endpoint handler reads
|
||||
// the middleware-stashed inbound ExecutionId and runs the inbound
|
||||
// method script with it as parentExecutionId. ──
|
||||
using var host = await BuildInboundHostAsync(centralAuditWriter, async ctx =>
|
||||
{
|
||||
DefaultRetryInterval = TimeSpan.Zero,
|
||||
DefaultMaxRetries = 3,
|
||||
RetryTimerInterval = TimeSpan.FromMinutes(10),
|
||||
},
|
||||
NullLogger<StoreAndForwardService>.Instance);
|
||||
var inboundExecutionId = (Guid)ctx.Items[AuditWriteMiddleware.InboundExecutionIdItemKey]!;
|
||||
var route = new RouteHelper(locator, router);
|
||||
var result = await scriptExecutor.ExecuteAsync(
|
||||
inboundMethod,
|
||||
new Dictionary<string, object?>(),
|
||||
route,
|
||||
TimeSpan.FromSeconds(30),
|
||||
ctx.RequestAborted,
|
||||
parentExecutionId: inboundExecutionId);
|
||||
|
||||
// ── Outbound external-system client (routed run): sync Call succeeds,
|
||||
// CachedCall completes immediately (WasBuffered=false) so the script
|
||||
// helper emits the Submit + Attempted + CachedResolve lifecycle. ──
|
||||
var externalClient = Substitute.For<IExternalSystemClient>();
|
||||
externalClient
|
||||
.CallAsync(ExternalSystemName, ExternalMethodName,
|
||||
Arg.Any<IReadOnlyDictionary<string, object?>?>(), Arg.Any<CancellationToken>())
|
||||
.Returns(new ExternalCallResult(true, "{\"ok\":true}", null));
|
||||
externalClient
|
||||
.CachedCallAsync(ExternalSystemName, ExternalMethodName,
|
||||
Arg.Any<IReadOnlyDictionary<string, object?>?>(),
|
||||
Arg.Any<string?>(), Arg.Any<CancellationToken>(),
|
||||
Arg.Any<ZB.MOM.WW.ScadaBridge.Commons.Types.TrackedOperationId?>(),
|
||||
Arg.Any<Guid?>(), Arg.Any<string?>(), Arg.Any<Guid?>())
|
||||
.Returns(new ExternalCallResult(true, "{\"ok\":true}", null, WasBuffered: false));
|
||||
ctx.Response.StatusCode = result.Success ? 200 : 500;
|
||||
await ctx.Response.WriteAsync(result.Success ? "ok" : "fail");
|
||||
});
|
||||
|
||||
// ── The routing transport stand-in: builds the routed ScriptRuntimeContext
|
||||
// carrying RouteToCallRequest.ParentExecutionId — exactly what the
|
||||
// production site handler (DeploymentManagerActor) does. ──
|
||||
var router = new BridgingInstanceRouter(
|
||||
siteId,
|
||||
externalClient,
|
||||
siteAuditWriter,
|
||||
cachedForwarder,
|
||||
storeAndForward);
|
||||
var client = host.GetTestClient();
|
||||
var response = await client.PostAsync(
|
||||
"/api/submitOrder",
|
||||
new StringContent("{}", Encoding.UTF8, "application/json"));
|
||||
Assert.Equal(System.Net.HttpStatusCode.OK, response.StatusCode);
|
||||
|
||||
// ── The inbound API method script: it calls Route.Call into the site
|
||||
// instance. The real InboundScriptExecutor binds the inbound request's
|
||||
// ExecutionId onto the RouteHelper, so the routed call carries it as
|
||||
// ParentExecutionId. ──
|
||||
var inboundMethod = new ZB.MOM.WW.ScadaBridge.Commons.Entities.InboundApi.ApiMethod(
|
||||
"submitOrder",
|
||||
$"return await Route.To(\"{RoutedInstanceCode}\").Call(\"{RoutedScriptName}\", new {{ order = 7 }});");
|
||||
var locator = Substitute.For<IInstanceLocator>();
|
||||
locator.GetSiteIdForInstanceAsync(RoutedInstanceCode, Arg.Any<CancellationToken>())
|
||||
.Returns(siteId);
|
||||
var scriptExecutor = new InboundScriptExecutor(
|
||||
NullLogger<InboundScriptExecutor>.Instance,
|
||||
new ServiceCollection().BuildServiceProvider());
|
||||
Assert.True(scriptExecutor.CompileAndRegister(inboundMethod));
|
||||
// The routed run emits its sync-ApiCall and NotifySend audit rows on a
|
||||
// deliberately fire-and-forget path (alog.md §7 — an audit write must
|
||||
// never block the user-facing script call). `Notify.Send` therefore
|
||||
// returns — and the routed `RouteToCallAsync` completes — BEFORE the
|
||||
// SqliteAuditWriter background loop has flushed the NotifySend row into
|
||||
// the site hot-path. Wait for all five site rows to be durably present
|
||||
// in SQLite before the central assertion: this is the production
|
||||
// durability point (the row IS in SQLite before it is considered
|
||||
// audited), and pinning it removes the emit-vs-drain race that
|
||||
// otherwise let the SiteAuditTelemetryADrain forward only four rows on
|
||||
// its first tick and leave NotifySend stranded for a full drain
|
||||
// interval under heavy parallel load.
|
||||
await WaitForSiteRowsPersistedAsync(sqliteWriter);
|
||||
|
||||
// ── Act — issue the inbound HTTP request through a TestHost pipeline
|
||||
// fronted by the real AuditWriteMiddleware. The endpoint handler reads
|
||||
// the middleware-stashed inbound ExecutionId and runs the inbound
|
||||
// method script with it as parentExecutionId. ──
|
||||
using var host = await BuildInboundHostAsync(centralAuditWriter, async ctx =>
|
||||
// The routed run produced a NotifySend that buffered a NotificationSubmit
|
||||
// into S&F. Drain that genuine site-produced submission to the central
|
||||
// NotificationOutboxActor so the NotifyDeliver dispatch rows materialise.
|
||||
await ForwardBufferedNotificationToCentralAsync(
|
||||
storeAndForward, router.NotificationId!, centralProvider, centralAuditWriter);
|
||||
|
||||
// ── Assert ──────────────────────────────────────────────────────────
|
||||
await AwaitAssertAsync(async () =>
|
||||
{
|
||||
await using var readContext = CreateContext();
|
||||
var repo = new AuditLogRepository(readContext);
|
||||
|
||||
// Every audit row this site produced (sync ApiCall + cached lifecycle
|
||||
// + NotifySend) plus the central NotifyDeliver rows.
|
||||
var siteRows = await repo.QueryAsync(
|
||||
new AuditLogQueryFilter(SourceSiteIds: new[] { siteId }),
|
||||
new AuditLogPaging(PageSize: 100));
|
||||
|
||||
// sync ApiCall (1) + cached Submit/Attempted/Resolve (3) + NotifySend (1)
|
||||
// + NotifyDeliver Attempted/Delivered (2) = 7 rows for the routed run.
|
||||
Assert.True(siteRows.Count == 7,
|
||||
"Expected 7 routed-run audit rows; saw: "
|
||||
+ string.Join(", ", siteRows.Select(r => $"{r.AsRow().Channel}/{r.AsRow().Kind}/{r.AsRow().Status}")));
|
||||
Assert.Single(siteRows, r => r.AsRow().Channel == AuditChannel.ApiOutbound && r.AsRow().Kind == AuditKind.ApiCall);
|
||||
Assert.Single(siteRows, r => r.AsRow().Kind == AuditKind.CachedSubmit);
|
||||
Assert.Single(siteRows, r => r.AsRow().Kind == AuditKind.CachedResolve);
|
||||
Assert.Single(siteRows, r => r.AsRow().Kind == AuditKind.NotifySend);
|
||||
Assert.Equal(2, siteRows.Count(r => r.AsRow().Kind == AuditKind.NotifyDeliver));
|
||||
|
||||
// CORE PROMISE: every routed-run row carries the SAME non-null
|
||||
// ParentExecutionId — the inbound request's ExecutionId.
|
||||
var parentIds = siteRows.Select(r => r.AsRow().ParentExecutionId).Distinct().ToList();
|
||||
Assert.Single(parentIds);
|
||||
Assert.NotNull(parentIds[0]);
|
||||
var inboundExecutionId = parentIds[0]!.Value;
|
||||
|
||||
// The routed run has its OWN distinct ExecutionId — not the parent's.
|
||||
var routedExecutionIds = siteRows
|
||||
.Select(r => r.AsRow().ExecutionId)
|
||||
.Distinct()
|
||||
.ToList();
|
||||
Assert.Single(routedExecutionIds);
|
||||
Assert.NotNull(routedExecutionIds[0]);
|
||||
var routedExecutionId = routedExecutionIds[0]!.Value;
|
||||
Assert.NotEqual(inboundExecutionId, routedExecutionId);
|
||||
|
||||
// The inbound request's own InboundRequest row is TOP-LEVEL —
|
||||
// ExecutionId = the propagated id, ParentExecutionId = NULL.
|
||||
var inboundRows = await repo.QueryAsync(
|
||||
new AuditLogQueryFilter(ExecutionId: inboundExecutionId),
|
||||
new AuditLogPaging(PageSize: 10));
|
||||
var inboundRow = Assert.Single(inboundRows,
|
||||
r => r.AsRow().Channel == AuditChannel.ApiInbound && r.AsRow().Kind == AuditKind.InboundRequest);
|
||||
Assert.Equal(AuditStatus.Delivered, inboundRow.AsRow().Status);
|
||||
Assert.Null(inboundRow.AsRow().ParentExecutionId);
|
||||
|
||||
// The parentExecutionId filter pulls the routed run's complete
|
||||
// trust-boundary footprint (all 7 routed rows, none of the inbound).
|
||||
var byParent = await repo.QueryAsync(
|
||||
new AuditLogQueryFilter(ParentExecutionId: inboundExecutionId),
|
||||
new AuditLogPaging(PageSize: 100));
|
||||
Assert.Equal(7, byParent.Count);
|
||||
Assert.All(byParent, r => Assert.Equal(routedExecutionId, r.AsRow().ExecutionId));
|
||||
|
||||
// GetExecutionTreeAsync returns BOTH executions in one chain —
|
||||
// inbound (root) and routed (child), regardless of entry point.
|
||||
var treeFromChild = await repo.GetExecutionTreeAsync(routedExecutionId);
|
||||
AssertChain(treeFromChild, inboundExecutionId, routedExecutionId);
|
||||
var treeFromRoot = await repo.GetExecutionTreeAsync(inboundExecutionId);
|
||||
AssertChain(treeFromRoot, inboundExecutionId, routedExecutionId);
|
||||
}, TimeSpan.FromSeconds(90));
|
||||
}
|
||||
finally
|
||||
{
|
||||
var inboundExecutionId = (Guid)ctx.Items[AuditWriteMiddleware.InboundExecutionIdItemKey]!;
|
||||
var route = new RouteHelper(locator, router);
|
||||
var result = await scriptExecutor.ExecuteAsync(
|
||||
inboundMethod,
|
||||
new Dictionary<string, object?>(),
|
||||
route,
|
||||
TimeSpan.FromSeconds(30),
|
||||
ctx.RequestAborted,
|
||||
parentExecutionId: inboundExecutionId);
|
||||
|
||||
ctx.Response.StatusCode = result.Success ? 200 : 500;
|
||||
await ctx.Response.WriteAsync(result.Success ? "ok" : "fail");
|
||||
});
|
||||
|
||||
var client = host.GetTestClient();
|
||||
var response = await client.PostAsync(
|
||||
"/api/submitOrder",
|
||||
new StringContent("{}", Encoding.UTF8, "application/json"));
|
||||
Assert.Equal(System.Net.HttpStatusCode.OK, response.StatusCode);
|
||||
|
||||
// The routed run emits its sync-ApiCall and NotifySend audit rows on a
|
||||
// deliberately fire-and-forget path (alog.md §7 — an audit write must
|
||||
// never block the user-facing script call). `Notify.Send` therefore
|
||||
// returns — and the routed `RouteToCallAsync` completes — BEFORE the
|
||||
// SqliteAuditWriter background loop has flushed the NotifySend row into
|
||||
// the site hot-path. Wait for all five site rows to be durably present
|
||||
// in SQLite before the central assertion: this is the production
|
||||
// durability point (the row IS in SQLite before it is considered
|
||||
// audited), and pinning it removes the emit-vs-drain race that
|
||||
// otherwise let the SiteAuditTelemetryADrain forward only four rows on
|
||||
// its first tick and leave NotifySend stranded for a full drain
|
||||
// interval under heavy parallel load.
|
||||
await WaitForSiteRowsPersistedAsync(sqliteWriter);
|
||||
|
||||
// The routed run produced a NotifySend that buffered a NotificationSubmit
|
||||
// into S&F. Drain that genuine site-produced submission to the central
|
||||
// NotificationOutboxActor so the NotifyDeliver dispatch rows materialise.
|
||||
await ForwardBufferedNotificationToCentralAsync(
|
||||
storeAndForward, router.NotificationId!, centralProvider, centralAuditWriter);
|
||||
|
||||
// ── Assert ──────────────────────────────────────────────────────────
|
||||
await AwaitAssertAsync(async () =>
|
||||
{
|
||||
await using var readContext = CreateContext();
|
||||
var repo = new AuditLogRepository(readContext);
|
||||
|
||||
// Every audit row this site produced (sync ApiCall + cached lifecycle
|
||||
// + NotifySend) plus the central NotifyDeliver rows.
|
||||
var siteRows = await repo.QueryAsync(
|
||||
new AuditLogQueryFilter(SourceSiteIds: new[] { siteId }),
|
||||
new AuditLogPaging(PageSize: 100));
|
||||
|
||||
// sync ApiCall (1) + cached Submit/Attempted/Resolve (3) + NotifySend (1)
|
||||
// + NotifyDeliver Attempted/Delivered (2) = 7 rows for the routed run.
|
||||
Assert.True(siteRows.Count == 7,
|
||||
"Expected 7 routed-run audit rows; saw: "
|
||||
+ string.Join(", ", siteRows.Select(r => $"{r.AsRow().Channel}/{r.AsRow().Kind}/{r.AsRow().Status}")));
|
||||
Assert.Single(siteRows, r => r.AsRow().Channel == AuditChannel.ApiOutbound && r.AsRow().Kind == AuditKind.ApiCall);
|
||||
Assert.Single(siteRows, r => r.AsRow().Kind == AuditKind.CachedSubmit);
|
||||
Assert.Single(siteRows, r => r.AsRow().Kind == AuditKind.CachedResolve);
|
||||
Assert.Single(siteRows, r => r.AsRow().Kind == AuditKind.NotifySend);
|
||||
Assert.Equal(2, siteRows.Count(r => r.AsRow().Kind == AuditKind.NotifyDeliver));
|
||||
|
||||
// CORE PROMISE: every routed-run row carries the SAME non-null
|
||||
// ParentExecutionId — the inbound request's ExecutionId.
|
||||
var parentIds = siteRows.Select(r => r.AsRow().ParentExecutionId).Distinct().ToList();
|
||||
Assert.Single(parentIds);
|
||||
Assert.NotNull(parentIds[0]);
|
||||
var inboundExecutionId = parentIds[0]!.Value;
|
||||
|
||||
// The routed run has its OWN distinct ExecutionId — not the parent's.
|
||||
var routedExecutionIds = siteRows
|
||||
.Select(r => r.AsRow().ExecutionId)
|
||||
.Distinct()
|
||||
.ToList();
|
||||
Assert.Single(routedExecutionIds);
|
||||
Assert.NotNull(routedExecutionIds[0]);
|
||||
var routedExecutionId = routedExecutionIds[0]!.Value;
|
||||
Assert.NotEqual(inboundExecutionId, routedExecutionId);
|
||||
|
||||
// The inbound request's own InboundRequest row is TOP-LEVEL —
|
||||
// ExecutionId = the propagated id, ParentExecutionId = NULL.
|
||||
var inboundRows = await repo.QueryAsync(
|
||||
new AuditLogQueryFilter(ExecutionId: inboundExecutionId),
|
||||
new AuditLogPaging(PageSize: 10));
|
||||
var inboundRow = Assert.Single(inboundRows,
|
||||
r => r.AsRow().Channel == AuditChannel.ApiInbound && r.AsRow().Kind == AuditKind.InboundRequest);
|
||||
Assert.Equal(AuditStatus.Delivered, inboundRow.AsRow().Status);
|
||||
Assert.Null(inboundRow.AsRow().ParentExecutionId);
|
||||
|
||||
// The parentExecutionId filter pulls the routed run's complete
|
||||
// trust-boundary footprint (all 7 routed rows, none of the inbound).
|
||||
var byParent = await repo.QueryAsync(
|
||||
new AuditLogQueryFilter(ParentExecutionId: inboundExecutionId),
|
||||
new AuditLogPaging(PageSize: 100));
|
||||
Assert.Equal(7, byParent.Count);
|
||||
Assert.All(byParent, r => Assert.Equal(routedExecutionId, r.AsRow().ExecutionId));
|
||||
|
||||
// GetExecutionTreeAsync returns BOTH executions in one chain —
|
||||
// inbound (root) and routed (child), regardless of entry point.
|
||||
var treeFromChild = await repo.GetExecutionTreeAsync(routedExecutionId);
|
||||
AssertChain(treeFromChild, inboundExecutionId, routedExecutionId);
|
||||
var treeFromRoot = await repo.GetExecutionTreeAsync(inboundExecutionId);
|
||||
AssertChain(treeFromRoot, inboundExecutionId, routedExecutionId);
|
||||
}, TimeSpan.FromSeconds(90));
|
||||
safLocalDb.Dispose();
|
||||
TestLocalDb.DeleteFiles(safDbPath);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
|
||||
+52
@@ -5,6 +5,7 @@ using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Microsoft.Extensions.Options;
|
||||
using NSubstitute;
|
||||
using NSubstitute.ExceptionExtensions;
|
||||
using NSubstitute.ReceivedExtensions;
|
||||
using ZB.MOM.WW.ScadaBridge.AuditLog.Site.Telemetry;
|
||||
using ZB.MOM.WW.Audit;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types.Audit;
|
||||
@@ -451,4 +452,55 @@ public class SiteAuditTelemetryActorTests : TestKit
|
||||
await _client.DidNotReceiveWithAnyArgs().IngestCachedTelemetryAsync(default!, default);
|
||||
await _queue.DidNotReceiveWithAnyArgs().ReadPendingCachedTelemetryAsync(default, default);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Regression for the 2026-07-20 rig finding (known-issues/2026-07-20-localdb-disk-io-error-under-load.md §8):
|
||||
/// both drain handlers await with <c>ConfigureAwait(false)</c>, so when a
|
||||
/// dependency call completes off the actor thread (as real SQLite/gRPC
|
||||
/// always do) the <c>finally</c>-block re-arm runs on a pool thread with no
|
||||
/// active ActorContext. Depending on what that thread's thread-static cell
|
||||
/// slot holds, <c>Context</c>/<c>Self</c> either throw
|
||||
/// <c>NotSupportedException: There is no active ActorContext</c> (crashing
|
||||
/// the actor once per drain — the rig's logged variant) or silently resolve
|
||||
/// to a STALE cell of some other actor, misrouting the tick so the drain
|
||||
/// loop stops. The two assertions below catch one variant each. Every other
|
||||
/// test in this class masks the bug by returning already-completed tasks
|
||||
/// from the mocks, which keeps the continuations on the actor thread.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task Drains_Whose_Awaits_Complete_Off_The_Actor_Thread_Keep_Draining_Without_Crashing()
|
||||
{
|
||||
// Task.Delay completes on a timer thread; with ConfigureAwait(false)
|
||||
// everything after the await — including the finally-block re-arm —
|
||||
// stays off the actor context.
|
||||
_queue.ReadPendingAsync(Arg.Any<int>(), Arg.Any<CancellationToken>())
|
||||
.Returns(async _ =>
|
||||
{
|
||||
await Task.Delay(25).ConfigureAwait(false);
|
||||
return (IReadOnlyList<AuditEvent>)Array.Empty<AuditEvent>();
|
||||
});
|
||||
_queue.ReadPendingCachedTelemetryAsync(Arg.Any<int>(), Arg.Any<CancellationToken>())
|
||||
.Returns(async _ =>
|
||||
{
|
||||
await Task.Delay(25).ConfigureAwait(false);
|
||||
return (IReadOnlyList<AuditEvent>)Array.Empty<AuditEvent>();
|
||||
});
|
||||
|
||||
// Variant 1 (throw → restart storm): any NotSupportedException logged
|
||||
// during the run fails the filter. Variant 2 (silent misroute → the
|
||||
// drain loop stalls after the first tick): the sustained-drain
|
||||
// assertion inside fails because reads stop at 1.
|
||||
await EventFilter.Exception<NotSupportedException>().ExpectAsync(0, async () =>
|
||||
{
|
||||
CreateActorWithCachedDrain(Opts(busySeconds: 1, idleSeconds: 1));
|
||||
|
||||
// Three full cycles prove the finally-block re-arm works from an
|
||||
// off-context thread: tick → drain → re-arm → tick → …
|
||||
await AwaitAssertAsync(async () =>
|
||||
{
|
||||
await _queue.Received(Quantity.Within(3, int.MaxValue)).ReadPendingAsync(Arg.Any<int>(), Arg.Any<CancellationToken>());
|
||||
await _queue.Received(Quantity.Within(3, int.MaxValue)).ReadPendingCachedTelemetryAsync(Arg.Any<int>(), Arg.Any<CancellationToken>());
|
||||
}, TimeSpan.FromSeconds(10));
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
+2
-1
@@ -76,7 +76,8 @@
|
||||
central MSSQL AuditLog.
|
||||
-->
|
||||
<ProjectReference Include="../../src/ZB.MOM.WW.ScadaBridge.InboundAPI/ZB.MOM.WW.ScadaBridge.InboundAPI.csproj" />
|
||||
</ItemGroup>
|
||||
<ProjectReference Include="../ZB.MOM.WW.ScadaBridge.TestSupport/ZB.MOM.WW.ScadaBridge.TestSupport.csproj" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<!-- M4 Bundle E (Task E3): need ASP.NET Core for the TestHost-based middleware E2E. -->
|
||||
|
||||
+24
@@ -40,4 +40,28 @@
|
||||
<ProjectReference Include="../../src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.csproj" />
|
||||
</ItemGroup>
|
||||
|
||||
<!--
|
||||
GHSA-pgww-w46g-26qg (NU1902, moderate) on AngleSharp, reached ONLY transitively via
|
||||
bunit. Suppressed here, scoped to this one test project, because there is verifiably
|
||||
no version combination that resolves it:
|
||||
|
||||
- AngleSharp < 1.5.0 is vulnerable (1.4.0 is what even the newest bunit resolves).
|
||||
- AngleSharp >= 1.5.0 changed IHtmlCollection<T>'s indexer, so every bunit build
|
||||
throws MissingMethodException at runtime - 33 CentralUI render tests fail.
|
||||
- Bumping bunit does not help: checked up to 2.7.2, still AngleSharp 1.4.0.
|
||||
|
||||
So the choice is a suppressed advisory or an unbuildable/failing test suite. Scoped
|
||||
suppression wins because the reach is genuinely nil: AngleSharp is an HTML parser used
|
||||
by bunit to assert on rendered markup in unit tests. It ships in no production project
|
||||
and processes no untrusted input - the "documents" it parses are our own components'
|
||||
output. Revisit when bunit ships against AngleSharp 1.5+.
|
||||
|
||||
NOT the same call as GHSA-2m69-gcr7-jv3q (SQLitePCLRaw): that suppression was removed
|
||||
in 2026-07 precisely because it was masking a vulnerability in PRODUCTION code paths.
|
||||
This one is test-only, and a patched-but-working version does not exist upstream.
|
||||
-->
|
||||
<ItemGroup>
|
||||
<NuGetAuditSuppress Include="https://github.com/advisories/GHSA-pgww-w46g-26qg" />
|
||||
</ItemGroup>
|
||||
|
||||
</Project>
|
||||
@@ -3,18 +3,56 @@ using System.Data.Common;
|
||||
using System.Text.Json;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using NSubstitute;
|
||||
using ZB.MOM.WW.LocalDb;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Entities.ExternalSystems;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
|
||||
using ZB.MOM.WW.ScadaBridge.TestSupport;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// WP-9: Tests for Database access — connection resolution, cached writes.
|
||||
/// </summary>
|
||||
public class DatabaseGatewayTests
|
||||
public class DatabaseGatewayTests : IDisposable
|
||||
{
|
||||
private readonly IExternalSystemRepository _repository = Substitute.For<IExternalSystemRepository>();
|
||||
|
||||
/// <summary>
|
||||
/// Temp local databases opened by the Store-and-Forward tests, torn down in
|
||||
/// <see cref="Dispose"/>.
|
||||
/// </summary>
|
||||
private readonly List<TestLocalDb> _localDbs = new();
|
||||
|
||||
/// <summary>
|
||||
/// Opens a real <c>ILocalDb</c> over a fresh temp file for a test that needs a live
|
||||
/// <see cref="ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardStorage"/>. The
|
||||
/// store takes an <c>ILocalDb</c> and LocalDb has no in-memory mode, so each test gets
|
||||
/// its own file rather than a shared-cache in-memory database held open by a
|
||||
/// keep-alive connection.
|
||||
/// </summary>
|
||||
private TestLocalDb CreateLocalDb(string prefix)
|
||||
{
|
||||
var localDb = TestLocalDb.CreateTemp(prefix);
|
||||
_localDbs.Add(localDb);
|
||||
return localDb;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Disposes every temp local database and then deletes its files — in that order,
|
||||
/// because the master connection LocalDb holds anchors the WAL sidecars.
|
||||
/// </summary>
|
||||
public void Dispose()
|
||||
{
|
||||
foreach (var localDb in _localDbs)
|
||||
{
|
||||
var path = localDb.Path;
|
||||
localDb.Dispose();
|
||||
TestLocalDb.DeleteFiles(path);
|
||||
}
|
||||
|
||||
GC.SuppressFinalize(this);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Configures the repository substitute for the name-keyed connection-resolution
|
||||
/// path used by <c>DatabaseGateway</c> (ExternalSystemGateway-011). A <c>null</c>
|
||||
@@ -84,12 +122,9 @@ public class DatabaseGatewayTests
|
||||
};
|
||||
StubConnection(conn);
|
||||
|
||||
var dbName = $"EsgCachedWrite_{Guid.NewGuid():N}";
|
||||
var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared";
|
||||
using var keepAlive = new Microsoft.Data.Sqlite.SqliteConnection(connStr);
|
||||
keepAlive.Open();
|
||||
var localDb = CreateLocalDb("EsgCachedWrite");
|
||||
var storage = new ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardStorage(
|
||||
connStr, NullLogger<ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardStorage>.Instance);
|
||||
localDb.Db, NullLogger<ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardStorage>.Instance);
|
||||
await storage.InitializeAsync();
|
||||
var sfOptions = new ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardOptions
|
||||
{
|
||||
@@ -121,7 +156,7 @@ public class DatabaseGatewayTests
|
||||
var depth = await storage.GetBufferDepthByCategoryAsync();
|
||||
Assert.Equal(1, depth[ZB.MOM.WW.ScadaBridge.Commons.Types.Enums.StoreAndForwardCategory.CachedDbWrite]);
|
||||
|
||||
var buffered = ReadBufferedRetrySettings(connStr);
|
||||
var buffered = ReadBufferedRetrySettings(localDb.Db);
|
||||
Assert.Equal(5, buffered.MaxRetries);
|
||||
Assert.Equal((long)TimeSpan.FromSeconds(12).TotalMilliseconds, buffered.RetryIntervalMs);
|
||||
|
||||
@@ -148,12 +183,9 @@ public class DatabaseGatewayTests
|
||||
};
|
||||
StubConnection(conn);
|
||||
|
||||
var dbName = $"EsgCachedWriteZero_{Guid.NewGuid():N}";
|
||||
var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared";
|
||||
using var keepAlive = new Microsoft.Data.Sqlite.SqliteConnection(connStr);
|
||||
keepAlive.Open();
|
||||
var localDb = CreateLocalDb("EsgCachedWriteZero");
|
||||
var storage = new ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardStorage(
|
||||
connStr, NullLogger<ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardStorage>.Instance);
|
||||
localDb.Db, NullLogger<ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardStorage>.Instance);
|
||||
await storage.InitializeAsync();
|
||||
var sfOptions = new ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardOptions
|
||||
{
|
||||
@@ -172,7 +204,7 @@ public class DatabaseGatewayTests
|
||||
|
||||
await gateway.CachedWriteAsync("testDb", "INSERT INTO t VALUES (1)");
|
||||
|
||||
var (maxRetries, _, _, _) = ReadBufferedRetrySettings(connStr);
|
||||
var (maxRetries, _, _, _) = ReadBufferedRetrySettings(localDb.Db);
|
||||
// Must be the bounded S&F default, never 0 — a stored 0 would mean retry-forever.
|
||||
Assert.Equal(99, maxRetries);
|
||||
Assert.NotEqual(0, maxRetries);
|
||||
@@ -182,19 +214,15 @@ public class DatabaseGatewayTests
|
||||
// cached-write attempt + the buffered retry path ──
|
||||
|
||||
/// <summary>
|
||||
/// Builds a real, initialised in-memory store-and-forward service plus a
|
||||
/// keep-alive connection (the SQLite shared-cache DB lives only while a
|
||||
/// connection is open). The caller disposes <paramref name="keepAlive"/>.
|
||||
/// Builds a real, initialised store-and-forward service over a fresh temp local
|
||||
/// database. The database is torn down by <see cref="Dispose"/>.
|
||||
/// </summary>
|
||||
private static (ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardService Sf, string ConnStr, Microsoft.Data.Sqlite.SqliteConnection KeepAlive)
|
||||
private (ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardService Sf, ILocalDb Db)
|
||||
NewStoreAndForward()
|
||||
{
|
||||
var dbName = $"EsgCachedWriteClassify_{Guid.NewGuid():N}";
|
||||
var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared";
|
||||
var keepAlive = new Microsoft.Data.Sqlite.SqliteConnection(connStr);
|
||||
keepAlive.Open();
|
||||
var localDb = CreateLocalDb("EsgCachedWriteClassify");
|
||||
var storage = new ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardStorage(
|
||||
connStr, NullLogger<ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardStorage>.Instance);
|
||||
localDb.Db, NullLogger<ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardStorage>.Instance);
|
||||
storage.InitializeAsync().GetAwaiter().GetResult();
|
||||
var sfOptions = new ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardOptions
|
||||
{
|
||||
@@ -204,7 +232,7 @@ public class DatabaseGatewayTests
|
||||
};
|
||||
var sf = new ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardService(
|
||||
storage, sfOptions, NullLogger<ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardService>.Instance);
|
||||
return (sf, connStr, keepAlive);
|
||||
return (sf, localDb.Db);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
@@ -217,8 +245,7 @@ public class DatabaseGatewayTests
|
||||
var conn = new DatabaseConnectionDefinition("testDb", "Server=localhost;Database=test") { Id = 1 };
|
||||
StubConnection(conn);
|
||||
|
||||
var (sf, connStr, keepAlive) = NewStoreAndForward();
|
||||
using var _ = keepAlive;
|
||||
var (sf, sfDb) = NewStoreAndForward();
|
||||
|
||||
var gateway = new ExecuteStubGateway(
|
||||
_repository,
|
||||
@@ -233,7 +260,7 @@ public class DatabaseGatewayTests
|
||||
Assert.NotNull(result.ErrorMessage);
|
||||
|
||||
// Nothing buffered — the permanent failure short-circuited S&F.
|
||||
Assert.Equal(0, ReadBufferDepth(connStr));
|
||||
Assert.Equal(0, ReadBufferDepth(sfDb));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
@@ -249,8 +276,7 @@ public class DatabaseGatewayTests
|
||||
};
|
||||
StubConnection(conn);
|
||||
|
||||
var (sf, connStr, keepAlive) = NewStoreAndForward();
|
||||
using var _ = keepAlive;
|
||||
var (sf, sfDb) = NewStoreAndForward();
|
||||
|
||||
var gateway = new ExecuteStubGateway(
|
||||
_repository,
|
||||
@@ -265,7 +291,7 @@ public class DatabaseGatewayTests
|
||||
Assert.True(result.WasBuffered); // handed to S&F, not synchronously failed
|
||||
Assert.Null(result.ErrorMessage);
|
||||
|
||||
Assert.Equal(1, ReadBufferDepth(connStr));
|
||||
Assert.Equal(1, ReadBufferDepth(sfDb));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
@@ -277,8 +303,7 @@ public class DatabaseGatewayTests
|
||||
var conn = new DatabaseConnectionDefinition("testDb", "Server=localhost;Database=test") { Id = 1 };
|
||||
StubConnection(conn);
|
||||
|
||||
var (sf, connStr, keepAlive) = NewStoreAndForward();
|
||||
using var _ = keepAlive;
|
||||
var (sf, sfDb) = NewStoreAndForward();
|
||||
|
||||
var gateway = new ExecuteStubGateway(_repository, sf, onExecute: () => { /* succeeds */ });
|
||||
|
||||
@@ -288,7 +313,7 @@ public class DatabaseGatewayTests
|
||||
Assert.False(result.WasBuffered);
|
||||
Assert.Null(result.ErrorMessage);
|
||||
|
||||
Assert.Equal(0, ReadBufferDepth(connStr));
|
||||
Assert.Equal(0, ReadBufferDepth(sfDb));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
@@ -381,8 +406,7 @@ public class DatabaseGatewayTests
|
||||
};
|
||||
StubConnection(conn);
|
||||
|
||||
var (sf, connStr, keepAlive) = NewStoreAndForward();
|
||||
using var _ = keepAlive;
|
||||
var (sf, sfDb) = NewStoreAndForward();
|
||||
|
||||
// RawExecuteStubGateway routes the raw throw through the PRODUCTION
|
||||
// ExecuteWriteAsync classification (the seam under test), unlike
|
||||
@@ -395,7 +419,7 @@ public class DatabaseGatewayTests
|
||||
Assert.True(result.WasBuffered); // handed to S&F as transient
|
||||
Assert.Null(result.ErrorMessage); // not a permanent Failed result
|
||||
|
||||
Assert.Equal(1, ReadBufferDepth(connStr));
|
||||
Assert.Equal(1, ReadBufferDepth(sfDb));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
@@ -408,8 +432,7 @@ public class DatabaseGatewayTests
|
||||
var conn = new DatabaseConnectionDefinition("testDb", "Server=localhost;Database=test") { Id = 1 };
|
||||
StubConnection(conn);
|
||||
|
||||
var (sf, connStr, keepAlive) = NewStoreAndForward();
|
||||
using var _ = keepAlive;
|
||||
var (sf, sfDb) = NewStoreAndForward();
|
||||
|
||||
using var cts = new CancellationTokenSource();
|
||||
cts.Cancel();
|
||||
@@ -421,7 +444,7 @@ public class DatabaseGatewayTests
|
||||
() => gateway.CachedWriteAsync("testDb", "INSERT INTO t VALUES (1)", cancellationToken: cts.Token));
|
||||
|
||||
// Cancellation is not a transient failure — nothing must have been buffered.
|
||||
Assert.Equal(0, ReadBufferDepth(connStr));
|
||||
Assert.Equal(0, ReadBufferDepth(sfDb));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
@@ -434,8 +457,7 @@ public class DatabaseGatewayTests
|
||||
var conn = new DatabaseConnectionDefinition("testDb", "Server=localhost;Database=test") { Id = 1 };
|
||||
StubConnection(conn);
|
||||
|
||||
var (sf, connStr, keepAlive) = NewStoreAndForward();
|
||||
using var _ = keepAlive;
|
||||
var (sf, sfDb) = NewStoreAndForward();
|
||||
|
||||
var gateway = new RawExecuteStubGateway(
|
||||
_repository, sf, onRunSql: () => throw new ArgumentException("authoring bug"));
|
||||
@@ -443,7 +465,7 @@ public class DatabaseGatewayTests
|
||||
await Assert.ThrowsAsync<ArgumentException>(
|
||||
() => gateway.CachedWriteAsync("testDb", "INSERT INTO t VALUES (1)"));
|
||||
|
||||
Assert.Equal(0, ReadBufferDepth(connStr));
|
||||
Assert.Equal(0, ReadBufferDepth(sfDb));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
@@ -486,8 +508,7 @@ public class DatabaseGatewayTests
|
||||
var conn = new DatabaseConnectionDefinition("testDb", "Server=localhost;Database=test") { Id = 1 };
|
||||
StubConnection(conn);
|
||||
|
||||
var (sf, connStr, keepAlive) = NewStoreAndForward();
|
||||
using var _ = keepAlive;
|
||||
var (sf, sfDb) = NewStoreAndForward();
|
||||
|
||||
using var cts = new CancellationTokenSource();
|
||||
cts.Cancel();
|
||||
@@ -505,7 +526,7 @@ public class DatabaseGatewayTests
|
||||
|
||||
// The cancel won — it must NOT have been classified as transient (buffered)
|
||||
// nor returned as a permanent Failed result.
|
||||
Assert.Equal(0, ReadBufferDepth(connStr));
|
||||
Assert.Equal(0, ReadBufferDepth(sfDb));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
@@ -542,10 +563,10 @@ public class DatabaseGatewayTests
|
||||
/// Reads the current buffered-message count off the S&F SQLite DB by
|
||||
/// counting <c>sf_messages</c> rows (the engine's persistence table).
|
||||
/// </summary>
|
||||
private static int ReadBufferDepth(string connStr)
|
||||
private static int ReadBufferDepth(ILocalDb localDb)
|
||||
{
|
||||
using var conn = new Microsoft.Data.Sqlite.SqliteConnection(connStr);
|
||||
conn.Open();
|
||||
// CreateConnection hands back an already-open, pragma-configured connection.
|
||||
using var conn = localDb.CreateConnection();
|
||||
using var cmd = conn.CreateCommand();
|
||||
cmd.CommandText = "SELECT COUNT(*) FROM sf_messages";
|
||||
return Convert.ToInt32(cmd.ExecuteScalar());
|
||||
@@ -615,10 +636,10 @@ public class DatabaseGatewayTests
|
||||
}
|
||||
|
||||
private static (int MaxRetries, long RetryIntervalMs, Guid? ExecutionId, string? SourceScript)
|
||||
ReadBufferedRetrySettings(string connStr)
|
||||
ReadBufferedRetrySettings(ILocalDb localDb)
|
||||
{
|
||||
using var conn = new Microsoft.Data.Sqlite.SqliteConnection(connStr);
|
||||
conn.Open();
|
||||
// CreateConnection hands back an already-open, pragma-configured connection.
|
||||
using var conn = localDb.CreateConnection();
|
||||
using var cmd = conn.CreateCommand();
|
||||
cmd.CommandText =
|
||||
"SELECT max_retries, retry_interval_ms, execution_id, source_script FROM sf_messages";
|
||||
|
||||
+53
-32
@@ -1,24 +1,60 @@
|
||||
using System.Net;
|
||||
using System.Net.Http.Headers;
|
||||
using System.Text;
|
||||
using Microsoft.Data.Sqlite;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using NSubstitute;
|
||||
using ZB.MOM.WW.LocalDb;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Entities.ExternalSystems;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
|
||||
using ZB.MOM.WW.ScadaBridge.StoreAndForward;
|
||||
using ZB.MOM.WW.ScadaBridge.TestSupport;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// WP-6/7: Tests for ExternalSystemClient — HTTP client, call modes, error handling.
|
||||
/// </summary>
|
||||
public class ExternalSystemClientTests
|
||||
public class ExternalSystemClientTests : IDisposable
|
||||
{
|
||||
private readonly IExternalSystemRepository _repository = Substitute.For<IExternalSystemRepository>();
|
||||
private readonly IHttpClientFactory _httpClientFactory = Substitute.For<IHttpClientFactory>();
|
||||
|
||||
/// <summary>
|
||||
/// Temp local databases opened by the Store-and-Forward tests, torn down in
|
||||
/// <see cref="Dispose"/>.
|
||||
/// </summary>
|
||||
private readonly List<TestLocalDb> _localDbs = new();
|
||||
|
||||
/// <summary>
|
||||
/// Opens a real <c>ILocalDb</c> over a fresh temp file for a test that needs a live
|
||||
/// <see cref="StoreAndForwardStorage"/>. The store takes an <c>ILocalDb</c> and LocalDb
|
||||
/// has no in-memory mode, so each test gets its own file rather than a shared-cache
|
||||
/// in-memory database held open by a keep-alive connection.
|
||||
/// </summary>
|
||||
private TestLocalDb CreateLocalDb(string prefix)
|
||||
{
|
||||
var localDb = TestLocalDb.CreateTemp(prefix);
|
||||
_localDbs.Add(localDb);
|
||||
return localDb;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Disposes every temp local database and then deletes its files — in that order,
|
||||
/// because the master connection LocalDb holds anchors the WAL sidecars.
|
||||
/// </summary>
|
||||
public void Dispose()
|
||||
{
|
||||
foreach (var localDb in _localDbs)
|
||||
{
|
||||
var path = localDb.Path;
|
||||
localDb.Dispose();
|
||||
TestLocalDb.DeleteFiles(path);
|
||||
}
|
||||
|
||||
GC.SuppressFinalize(this);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Configures the repository substitute for the name-keyed resolution path used by
|
||||
/// <c>ExternalSystemClient</c> (ExternalSystemGateway-011). A <c>null</c> system or
|
||||
@@ -275,11 +311,8 @@ public class ExternalSystemClientTests
|
||||
_httpClientFactory.CreateClient(Arg.Any<string>()).Returns(httpClient);
|
||||
|
||||
// A real S&F service with a registered delivery handler that counts invocations.
|
||||
var dbName = $"EsgDoubleDispatch_{Guid.NewGuid():N}";
|
||||
var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared";
|
||||
using var keepAlive = new SqliteConnection(connStr);
|
||||
keepAlive.Open();
|
||||
var storage = new StoreAndForwardStorage(connStr, NullLogger<StoreAndForwardStorage>.Instance);
|
||||
var localDb = CreateLocalDb("EsgDoubleDispatch");
|
||||
var storage = new StoreAndForwardStorage(localDb.Db, NullLogger<StoreAndForwardStorage>.Instance);
|
||||
await storage.InitializeAsync();
|
||||
var sfOptions = new StoreAndForwardOptions
|
||||
{
|
||||
@@ -422,11 +455,8 @@ public class ExternalSystemClientTests
|
||||
var httpClient = new HttpClient(new MockHttpMessageHandler(HttpStatusCode.InternalServerError, "boom"));
|
||||
_httpClientFactory.CreateClient(Arg.Any<string>()).Returns(httpClient);
|
||||
|
||||
var dbName = $"EsgRetry_{Guid.NewGuid():N}";
|
||||
var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared";
|
||||
using var keepAlive = new SqliteConnection(connStr);
|
||||
keepAlive.Open();
|
||||
var storage = new StoreAndForwardStorage(connStr, NullLogger<StoreAndForwardStorage>.Instance);
|
||||
var localDb = CreateLocalDb("EsgRetry");
|
||||
var storage = new StoreAndForwardStorage(localDb.Db, NullLogger<StoreAndForwardStorage>.Instance);
|
||||
await storage.InitializeAsync();
|
||||
// S&F defaults deliberately different from the system's settings.
|
||||
var sfOptions = new StoreAndForwardOptions
|
||||
@@ -454,7 +484,7 @@ public class ExternalSystemClientTests
|
||||
var depth = await storage.GetBufferDepthByCategoryAsync();
|
||||
Assert.Equal(1, depth[ZB.MOM.WW.ScadaBridge.Commons.Types.Enums.StoreAndForwardCategory.ExternalSystem]);
|
||||
|
||||
var buffered = ReadBufferedRetrySettings(connStr);
|
||||
var buffered = ReadBufferedRetrySettings(localDb.Db);
|
||||
Assert.Equal(7, buffered.MaxRetries);
|
||||
Assert.Equal((long)TimeSpan.FromSeconds(42).TotalMilliseconds, buffered.RetryIntervalMs);
|
||||
|
||||
@@ -466,10 +496,10 @@ public class ExternalSystemClientTests
|
||||
}
|
||||
|
||||
private static (int MaxRetries, long RetryIntervalMs, Guid? ExecutionId, string? SourceScript)
|
||||
ReadBufferedRetrySettings(string connStr)
|
||||
ReadBufferedRetrySettings(ILocalDb localDb)
|
||||
{
|
||||
using var conn = new SqliteConnection(connStr);
|
||||
conn.Open();
|
||||
// CreateConnection hands back an already-open, pragma-configured connection.
|
||||
using var conn = localDb.CreateConnection();
|
||||
using var cmd = conn.CreateCommand();
|
||||
cmd.CommandText =
|
||||
"SELECT max_retries, retry_interval_ms, execution_id, source_script FROM sf_messages";
|
||||
@@ -505,11 +535,8 @@ public class ExternalSystemClientTests
|
||||
var httpClient = new HttpClient(new MockHttpMessageHandler(HttpStatusCode.InternalServerError, "boom"));
|
||||
_httpClientFactory.CreateClient(Arg.Any<string>()).Returns(httpClient);
|
||||
|
||||
var dbName = $"EsgRetryZero_{Guid.NewGuid():N}";
|
||||
var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared";
|
||||
using var keepAlive = new SqliteConnection(connStr);
|
||||
keepAlive.Open();
|
||||
var storage = new StoreAndForwardStorage(connStr, NullLogger<StoreAndForwardStorage>.Instance);
|
||||
var localDb = CreateLocalDb("EsgRetryZero");
|
||||
var storage = new StoreAndForwardStorage(localDb.Db, NullLogger<StoreAndForwardStorage>.Instance);
|
||||
await storage.InitializeAsync();
|
||||
var sfOptions = new StoreAndForwardOptions
|
||||
{
|
||||
@@ -525,7 +552,7 @@ public class ExternalSystemClientTests
|
||||
|
||||
await client.CachedCallAsync("TestAPI", "postData");
|
||||
|
||||
var (maxRetries, _, _, _) = ReadBufferedRetrySettings(connStr);
|
||||
var (maxRetries, _, _, _) = ReadBufferedRetrySettings(localDb.Db);
|
||||
// Must be the bounded S&F default, never 0 — a stored 0 would mean retry-forever.
|
||||
Assert.Equal(99, maxRetries);
|
||||
Assert.NotEqual(0, maxRetries);
|
||||
@@ -651,11 +678,8 @@ public class ExternalSystemClientTests
|
||||
var httpClient = new HttpClient(new HangingHttpMessageHandler(TimeSpan.FromMinutes(10)));
|
||||
_httpClientFactory.CreateClient(Arg.Any<string>()).Returns(httpClient);
|
||||
|
||||
var dbName = $"EsgCancel_{Guid.NewGuid():N}";
|
||||
var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared";
|
||||
using var keepAlive = new SqliteConnection(connStr);
|
||||
keepAlive.Open();
|
||||
var storage = new StoreAndForwardStorage(connStr, NullLogger<StoreAndForwardStorage>.Instance);
|
||||
var localDb = CreateLocalDb("EsgCancel");
|
||||
var storage = new StoreAndForwardStorage(localDb.Db, NullLogger<StoreAndForwardStorage>.Instance);
|
||||
await storage.InitializeAsync();
|
||||
var sfOptions = new StoreAndForwardOptions
|
||||
{
|
||||
@@ -1060,11 +1084,8 @@ public class ExternalSystemClientTests
|
||||
.Returns(_ => new HttpClient(new MockHttpMessageHandler(HttpStatusCode.OK, hugeBody)));
|
||||
|
||||
// A real S&F service so we can assert the oversized response is NOT buffered.
|
||||
var dbName = $"EsgOversize_{Guid.NewGuid():N}";
|
||||
var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared";
|
||||
using var keepAlive = new SqliteConnection(connStr);
|
||||
keepAlive.Open();
|
||||
var storage = new StoreAndForwardStorage(connStr, NullLogger<StoreAndForwardStorage>.Instance);
|
||||
var localDb = CreateLocalDb("EsgOversize");
|
||||
var storage = new StoreAndForwardStorage(localDb.Db, NullLogger<StoreAndForwardStorage>.Instance);
|
||||
await storage.InitializeAsync();
|
||||
var sfOptions = new StoreAndForwardOptions
|
||||
{
|
||||
|
||||
+2
-1
@@ -24,6 +24,7 @@
|
||||
<ProjectReference Include="../../src/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.csproj" />
|
||||
<ProjectReference Include="../../src/ZB.MOM.WW.ScadaBridge.Commons/ZB.MOM.WW.ScadaBridge.Commons.csproj" />
|
||||
<ProjectReference Include="../../src/ZB.MOM.WW.ScadaBridge.StoreAndForward/ZB.MOM.WW.ScadaBridge.StoreAndForward.csproj" />
|
||||
</ItemGroup>
|
||||
<ProjectReference Include="../ZB.MOM.WW.ScadaBridge.TestSupport/ZB.MOM.WW.ScadaBridge.TestSupport.csproj" />
|
||||
</ItemGroup>
|
||||
|
||||
</Project>
|
||||
|
||||
@@ -1,10 +1,10 @@
|
||||
using Microsoft.Data.Sqlite;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
|
||||
using ZB.MOM.WW.ScadaBridge.StoreAndForward;
|
||||
using ZB.MOM.WW.ScadaBridge.TestSupport;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests;
|
||||
|
||||
@@ -191,50 +191,55 @@ public class HealthReportSenderTests
|
||||
[Fact]
|
||||
public async Task ReportsIncludeStoreAndForwardBufferDepthsFromStorage()
|
||||
{
|
||||
var dbName = $"HealthSfDepth_{Guid.NewGuid():N}";
|
||||
var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared";
|
||||
// Keep one connection alive so the in-memory DB persists for the test.
|
||||
using var keepAlive = new SqliteConnection(connStr);
|
||||
keepAlive.Open();
|
||||
|
||||
var storage = new StoreAndForwardStorage(connStr, NullLogger<StoreAndForwardStorage>.Instance);
|
||||
await storage.InitializeAsync();
|
||||
|
||||
// Two pending ExternalSystem messages and one pending Notification message.
|
||||
await storage.EnqueueAsync(MakePendingMessage("m1", StoreAndForwardCategory.ExternalSystem));
|
||||
await storage.EnqueueAsync(MakePendingMessage("m2", StoreAndForwardCategory.ExternalSystem));
|
||||
await storage.EnqueueAsync(MakePendingMessage("m3", StoreAndForwardCategory.Notification));
|
||||
|
||||
var transport = new FakeTransport();
|
||||
var collector = new SiteHealthCollector();
|
||||
collector.SetActiveNode(true);
|
||||
var options = Options.Create(new HealthMonitoringOptions
|
||||
{
|
||||
ReportInterval = TimeSpan.FromMilliseconds(50)
|
||||
});
|
||||
|
||||
var sender = new HealthReportSender(
|
||||
collector,
|
||||
transport,
|
||||
options,
|
||||
NullLogger<HealthReportSender>.Instance,
|
||||
new FakeSiteIdentityProvider(),
|
||||
sfStorage: storage);
|
||||
|
||||
using var cts = new CancellationTokenSource(TimeSpan.FromMilliseconds(300));
|
||||
// LocalDb has no in-memory mode, so the store runs over a real temp file.
|
||||
var localDb = TestLocalDb.CreateTemp("HealthSfDepth");
|
||||
try
|
||||
{
|
||||
await sender.StartAsync(cts.Token);
|
||||
await Task.Delay(250, CancellationToken.None);
|
||||
await sender.StopAsync(CancellationToken.None);
|
||||
}
|
||||
catch (OperationCanceledException) { }
|
||||
var storage = new StoreAndForwardStorage(localDb.Db, NullLogger<StoreAndForwardStorage>.Instance);
|
||||
await storage.InitializeAsync();
|
||||
|
||||
Assert.True(transport.SentReports.Count >= 1);
|
||||
var depths = transport.SentReports[^1].StoreAndForwardBufferDepths;
|
||||
Assert.Equal(2, depths[nameof(StoreAndForwardCategory.ExternalSystem)]);
|
||||
Assert.Equal(1, depths[nameof(StoreAndForwardCategory.Notification)]);
|
||||
Assert.False(depths.ContainsKey(nameof(StoreAndForwardCategory.CachedDbWrite)));
|
||||
// Two pending ExternalSystem messages and one pending Notification message.
|
||||
await storage.EnqueueAsync(MakePendingMessage("m1", StoreAndForwardCategory.ExternalSystem));
|
||||
await storage.EnqueueAsync(MakePendingMessage("m2", StoreAndForwardCategory.ExternalSystem));
|
||||
await storage.EnqueueAsync(MakePendingMessage("m3", StoreAndForwardCategory.Notification));
|
||||
|
||||
var transport = new FakeTransport();
|
||||
var collector = new SiteHealthCollector();
|
||||
collector.SetActiveNode(true);
|
||||
var options = Options.Create(new HealthMonitoringOptions
|
||||
{
|
||||
ReportInterval = TimeSpan.FromMilliseconds(50)
|
||||
});
|
||||
|
||||
var sender = new HealthReportSender(
|
||||
collector,
|
||||
transport,
|
||||
options,
|
||||
NullLogger<HealthReportSender>.Instance,
|
||||
new FakeSiteIdentityProvider(),
|
||||
sfStorage: storage);
|
||||
|
||||
using var cts = new CancellationTokenSource(TimeSpan.FromMilliseconds(300));
|
||||
try
|
||||
{
|
||||
await sender.StartAsync(cts.Token);
|
||||
await Task.Delay(250, CancellationToken.None);
|
||||
await sender.StopAsync(CancellationToken.None);
|
||||
}
|
||||
catch (OperationCanceledException) { }
|
||||
|
||||
Assert.True(transport.SentReports.Count >= 1);
|
||||
var depths = transport.SentReports[^1].StoreAndForwardBufferDepths;
|
||||
Assert.Equal(2, depths[nameof(StoreAndForwardCategory.ExternalSystem)]);
|
||||
Assert.Equal(1, depths[nameof(StoreAndForwardCategory.Notification)]);
|
||||
Assert.False(depths.ContainsKey(nameof(StoreAndForwardCategory.CachedDbWrite)));
|
||||
}
|
||||
finally
|
||||
{
|
||||
// The master connection anchors the WAL — dispose before deleting.
|
||||
localDb.Dispose();
|
||||
TestLocalDb.DeleteFiles(localDb.Path);
|
||||
}
|
||||
}
|
||||
|
||||
private static StoreAndForwardMessage MakePendingMessage(string id, StoreAndForwardCategory category) =>
|
||||
|
||||
+2
-1
@@ -28,6 +28,7 @@
|
||||
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="../../src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/ZB.MOM.WW.ScadaBridge.HealthMonitoring.csproj" />
|
||||
</ItemGroup>
|
||||
<ProjectReference Include="../ZB.MOM.WW.ScadaBridge.TestSupport/ZB.MOM.WW.ScadaBridge.TestSupport.csproj" />
|
||||
</ItemGroup>
|
||||
|
||||
</Project>
|
||||
|
||||
@@ -163,10 +163,12 @@ public class SiteActorPathTests : IAsyncLifetime
|
||||
private IHost? _host;
|
||||
private ActorSystem? _actorSystem;
|
||||
private string _tempDbPath = null!;
|
||||
private string _tempLocalDbPath = null!;
|
||||
|
||||
public async Task InitializeAsync()
|
||||
{
|
||||
_tempDbPath = Path.Combine(Path.GetTempPath(), $"scadabridge_actor_test_{Guid.NewGuid()}.db");
|
||||
_tempLocalDbPath = Path.Combine(Path.GetTempPath(), $"scadabridge_actor_localdb_{Guid.NewGuid()}.db");
|
||||
|
||||
var builder = Microsoft.Extensions.Hosting.Host.CreateDefaultBuilder();
|
||||
builder.ConfigureAppConfiguration(config =>
|
||||
@@ -183,6 +185,10 @@ public class SiteActorPathTests : IAsyncLifetime
|
||||
["ScadaBridge:Cluster:SeedNodes:1"] = "akka.tcp://scadabridge@localhost:25520",
|
||||
["ScadaBridge:Cluster:MinNrOfMembers"] = "1",
|
||||
["ScadaBridge:Database:SiteDbPath"] = _tempDbPath,
|
||||
// Required: LocalDbOptions.Path is validated with ValidateOnStart, and this
|
||||
// fixture actually starts the host — a site node with no configured
|
||||
// consolidated-database path fails fast rather than booting half-working.
|
||||
["LocalDb:Path"] = _tempLocalDbPath,
|
||||
// Configure a dummy central contact point to trigger ClusterClient creation
|
||||
["ScadaBridge:Communication:CentralContactPoints:0"] = "akka.tcp://scadabridge@localhost:25510",
|
||||
});
|
||||
@@ -207,6 +213,7 @@ public class SiteActorPathTests : IAsyncLifetime
|
||||
_host.Dispose();
|
||||
}
|
||||
try { File.Delete(_tempDbPath); } catch { /* best effort */ }
|
||||
try { File.Delete(_tempLocalDbPath); } catch { /* best effort */ }
|
||||
}
|
||||
|
||||
[Fact]
|
||||
|
||||
@@ -290,10 +290,12 @@ public class SiteAuditWiringTests : IDisposable
|
||||
{
|
||||
private readonly WebApplication _host;
|
||||
private readonly string _tempDbPath;
|
||||
private readonly string _tempLocalDbPath;
|
||||
|
||||
public SiteAuditWiringTests()
|
||||
{
|
||||
_tempDbPath = Path.Combine(Path.GetTempPath(), $"scadabridge_audit_wiring_{Guid.NewGuid()}.db");
|
||||
_tempLocalDbPath = Path.Combine(Path.GetTempPath(), $"scadabridge_audit_wiring_localdb_{Guid.NewGuid()}.db");
|
||||
|
||||
var builder = WebApplication.CreateBuilder();
|
||||
builder.Configuration.Sources.Clear();
|
||||
@@ -312,6 +314,11 @@ public class SiteAuditWiringTests : IDisposable
|
||||
// resolved; point it at an in-memory connection so the test doesn't
|
||||
// pollute the working directory.
|
||||
["AuditLog:SiteWriter:DatabasePath"] = ":memory:",
|
||||
// The cached-call telemetry forwarder pulls in IOperationTrackingStore,
|
||||
// which now resolves ILocalDb — so the consolidated site database must be
|
||||
// configured or the whole graph fails to resolve. There is no in-memory
|
||||
// mode: LocalDbOptions.Path is a filesystem path.
|
||||
["LocalDb:Path"] = _tempLocalDbPath,
|
||||
});
|
||||
|
||||
builder.Services.AddGrpc();
|
||||
@@ -326,6 +333,7 @@ public class SiteAuditWiringTests : IDisposable
|
||||
{
|
||||
(_host as IDisposable)?.Dispose();
|
||||
try { File.Delete(_tempDbPath); } catch { /* best effort */ }
|
||||
try { File.Delete(_tempLocalDbPath); } catch { /* best effort */ }
|
||||
}
|
||||
|
||||
[Fact]
|
||||
|
||||
@@ -383,11 +383,13 @@ public class SiteCompositionRootTests : IDisposable
|
||||
private readonly WebApplication _host;
|
||||
private readonly string _tempDbPath;
|
||||
private readonly string _tempTrackingDbPath;
|
||||
private readonly string _tempLocalDbPath;
|
||||
|
||||
public SiteCompositionRootTests()
|
||||
{
|
||||
_tempDbPath = Path.Combine(Path.GetTempPath(), $"scadabridge_test_{Guid.NewGuid()}.db");
|
||||
_tempTrackingDbPath = Path.Combine(Path.GetTempPath(), $"scadabridge_track_{Guid.NewGuid()}.db");
|
||||
_tempLocalDbPath = Path.Combine(Path.GetTempPath(), $"scadabridge_localdb_{Guid.NewGuid()}.db");
|
||||
|
||||
var builder = WebApplication.CreateBuilder();
|
||||
builder.Configuration.Sources.Clear();
|
||||
@@ -400,10 +402,13 @@ public class SiteCompositionRootTests : IDisposable
|
||||
["ScadaBridge:Node:RemotingPort"] = "0",
|
||||
["ScadaBridge:Node:GrpcPort"] = "0",
|
||||
["ScadaBridge:Database:SiteDbPath"] = _tempDbPath,
|
||||
// Point the site-local operation-tracking SQLite store (the ctor opens/creates
|
||||
// the file eagerly) at a temp path so resolving IOperationTrackingStore in the
|
||||
// composition-root test does not litter site-tracking.db in the test cwd.
|
||||
// Retained for the site config DB; it no longer points the tracking store
|
||||
// anywhere — that store now opens the consolidated database below.
|
||||
["ScadaBridge:OperationTracking:ConnectionString"] = $"Data Source={_tempTrackingDbPath}",
|
||||
// The consolidated site database. IOperationTrackingStore resolves ILocalDb
|
||||
// and creates the file eagerly in its ctor, so this must be a temp path or
|
||||
// the composition-root test litters the working directory.
|
||||
["LocalDb:Path"] = _tempLocalDbPath,
|
||||
// ClusterOptions requires at least one seed node (ClusterOptionsValidator).
|
||||
["ScadaBridge:Cluster:SeedNodes:0"] = "akka.tcp://scadabridge@localhost:2551",
|
||||
["ScadaBridge:Cluster:SeedNodes:1"] = "akka.tcp://scadabridge@localhost:2552",
|
||||
@@ -427,6 +432,7 @@ public class SiteCompositionRootTests : IDisposable
|
||||
(_host as IDisposable)?.Dispose();
|
||||
try { File.Delete(_tempDbPath); } catch { /* best effort */ }
|
||||
try { File.Delete(_tempTrackingDbPath); } catch { /* best effort */ }
|
||||
try { File.Delete(_tempLocalDbPath); } catch { /* best effort */ }
|
||||
}
|
||||
|
||||
[Fact]
|
||||
@@ -465,7 +471,6 @@ public class SiteCompositionRootTests : IDisposable
|
||||
new object[] { typeof(IDataConnectionFactory) },
|
||||
new object[] { typeof(StoreAndForwardStorage) },
|
||||
new object[] { typeof(StoreAndForwardService) },
|
||||
new object[] { typeof(ReplicationService) },
|
||||
new object[] { typeof(ISiteEventLogger) },
|
||||
new object[] { typeof(IEventLogQueryService) },
|
||||
new object[] { typeof(ISiteIdentityProvider) },
|
||||
|
||||
@@ -0,0 +1,171 @@
|
||||
using Grpc.Core;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.LocalDb.Replication;
|
||||
using ZB.MOM.WW.ScadaBridge.Host;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// LocalDb Phase 1 (Task 8) — the passive sync endpoint's inbound gate.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The replication library's <c>LocalDbSyncService</c> verifies nothing; inbound auth is
|
||||
/// explicitly the host's job. Without this interceptor, anything able to reach a site
|
||||
/// node's gRPC port could stream arbitrary rows straight into the consolidated site
|
||||
/// database — including into <c>OperationTracking</c>, which central reconciles from.
|
||||
/// </remarks>
|
||||
public class LocalDbSyncAuthInterceptorTests
|
||||
{
|
||||
private const string SyncMethod = "/localdb_sync.v1.LocalDbSync/Sync";
|
||||
private const string SiteStreamMethod = "/scadabridge.SiteStream/Connect";
|
||||
|
||||
private static LocalDbSyncAuthInterceptor CreateInterceptor(string? apiKey)
|
||||
=> new(
|
||||
Options.Create(new ReplicationOptions { ApiKey = apiKey }),
|
||||
NullLogger<LocalDbSyncAuthInterceptor>.Instance);
|
||||
|
||||
private static ServerCallContext CreateContext(string method, string? authorizationHeader)
|
||||
{
|
||||
var headers = new Metadata();
|
||||
if (authorizationHeader is not null)
|
||||
headers.Add("authorization", authorizationHeader);
|
||||
|
||||
return new FakeServerCallContext(method, headers);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Minimal <see cref="ServerCallContext"/> carrying just a method name and request
|
||||
/// headers — the only two things the interceptor reads.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Hand-rolled rather than using <c>Grpc.Core.Testing.TestServerCallContext</c>: that
|
||||
/// type ships in the retired native <c>Grpc.Core</c> package and does not exist on the
|
||||
/// grpc-dotnet stack this solution runs on. Pulling in the dead package to get one
|
||||
/// test helper would be a worse trade than these few overrides.
|
||||
/// </remarks>
|
||||
private sealed class FakeServerCallContext(string method, Metadata requestHeaders)
|
||||
: ServerCallContext
|
||||
{
|
||||
protected override string MethodCore => method;
|
||||
protected override string HostCore => "localhost";
|
||||
protected override string PeerCore => "ipv4:127.0.0.1:12345";
|
||||
protected override DateTime DeadlineCore => DateTime.UtcNow.AddMinutes(1);
|
||||
protected override Metadata RequestHeadersCore => requestHeaders;
|
||||
protected override CancellationToken CancellationTokenCore => CancellationToken.None;
|
||||
protected override Metadata ResponseTrailersCore { get; } = [];
|
||||
protected override Status StatusCore { get; set; }
|
||||
protected override WriteOptions? WriteOptionsCore { get; set; }
|
||||
protected override AuthContext AuthContextCore { get; } =
|
||||
new(null, new Dictionary<string, List<AuthProperty>>());
|
||||
|
||||
protected override ContextPropagationToken CreatePropagationTokenCore(
|
||||
ContextPropagationOptions? options)
|
||||
=> throw new NotSupportedException();
|
||||
|
||||
protected override Task WriteResponseHeadersAsyncCore(Metadata responseHeaders)
|
||||
=> Task.CompletedTask;
|
||||
}
|
||||
|
||||
/// <summary>Invokes the interceptor's unary path with a trivial continuation.</summary>
|
||||
private static Task<string> Invoke(
|
||||
LocalDbSyncAuthInterceptor interceptor, ServerCallContext context)
|
||||
=> interceptor.UnaryServerHandler<string, string>(
|
||||
"request", context, (_, _) => Task.FromResult("ok"));
|
||||
|
||||
[Fact]
|
||||
public async Task NonSyncMethod_PassesThrough_EvenWithNoKeyConfigured()
|
||||
{
|
||||
// The interceptor is registered on the shared site AddGrpc pipeline, so it sees
|
||||
// every call. It must be scoped strictly to the sync service — gating SiteStream
|
||||
// would break site↔central communication outright.
|
||||
var interceptor = CreateInterceptor(apiKey: null);
|
||||
var context = CreateContext(SiteStreamMethod, authorizationHeader: null);
|
||||
|
||||
Assert.Equal("ok", await Invoke(interceptor, context));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task SyncMethod_WithNoKeyConfigured_IsDenied_EvenWithABearerToken()
|
||||
{
|
||||
// Fail-closed. "No key configured" is the DEFAULT every site node ships with, so
|
||||
// treating it as "no auth required" would silently expose the endpoint on exactly
|
||||
// the configuration that is most common. Presenting a token must not help.
|
||||
var interceptor = CreateInterceptor(apiKey: null);
|
||||
var context = CreateContext(SyncMethod, "Bearer anything-at-all");
|
||||
|
||||
var ex = await Assert.ThrowsAsync<RpcException>(() => Invoke(interceptor, context));
|
||||
Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task SyncMethod_WithNoBearerToken_IsDenied()
|
||||
{
|
||||
var interceptor = CreateInterceptor("the-shared-key");
|
||||
var context = CreateContext(SyncMethod, authorizationHeader: null);
|
||||
|
||||
var ex = await Assert.ThrowsAsync<RpcException>(() => Invoke(interceptor, context));
|
||||
Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task SyncMethod_WithWrongBearerToken_IsDenied()
|
||||
{
|
||||
var interceptor = CreateInterceptor("the-shared-key");
|
||||
var context = CreateContext(SyncMethod, "Bearer the-wrong-key");
|
||||
|
||||
var ex = await Assert.ThrowsAsync<RpcException>(() => Invoke(interceptor, context));
|
||||
Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task SyncMethod_WithCorrectBearerToken_PassesThrough()
|
||||
{
|
||||
var interceptor = CreateInterceptor("the-shared-key");
|
||||
var context = CreateContext(SyncMethod, "Bearer the-shared-key");
|
||||
|
||||
Assert.Equal("ok", await Invoke(interceptor, context));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task SyncMethod_WithCorrectKey_ButNoBearerScheme_IsDenied()
|
||||
{
|
||||
// A raw key with no "Bearer " prefix is not what SyncBackgroundService sends, and
|
||||
// accepting it would widen the accepted credential shape for no reason.
|
||||
var interceptor = CreateInterceptor("the-shared-key");
|
||||
var context = CreateContext(SyncMethod, "the-shared-key");
|
||||
|
||||
var ex = await Assert.ThrowsAsync<RpcException>(() => Invoke(interceptor, context));
|
||||
Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task SyncMethod_TokenComparison_IsNotAPrefixMatch()
|
||||
{
|
||||
// A prefix/StartsWith comparison would accept a truncated key and make the secret
|
||||
// recoverable one character at a time.
|
||||
var interceptor = CreateInterceptor("the-shared-key");
|
||||
var context = CreateContext(SyncMethod, "Bearer the-shared-ke");
|
||||
|
||||
var ex = await Assert.ThrowsAsync<RpcException>(() => Invoke(interceptor, context));
|
||||
Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task DuplexStreaming_IsGated_BecauseThatIsHowSyncActuallyRuns()
|
||||
{
|
||||
// The sync RPC is a bidirectional stream. Gating only the unary path would leave
|
||||
// the real endpoint wide open while every unary test still passed.
|
||||
var interceptor = CreateInterceptor("the-shared-key");
|
||||
var context = CreateContext(SyncMethod, "Bearer the-wrong-key");
|
||||
|
||||
var ex = await Assert.ThrowsAsync<RpcException>(() =>
|
||||
interceptor.DuplexStreamingServerHandler<string, string>(
|
||||
requestStream: null!,
|
||||
responseStream: null!,
|
||||
context,
|
||||
(_, _, _) => Task.CompletedTask));
|
||||
|
||||
Assert.Equal(StatusCode.PermissionDenied, ex.StatusCode);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,72 @@
|
||||
using Microsoft.Extensions.Configuration;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using ZB.MOM.WW.LocalDb;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Pins that a site node creates the directory holding <c>LocalDb:Path</c> before the
|
||||
/// database is opened.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// The guarantee has now had three owners: <c>StoreAndForwardStorage</c> created the
|
||||
/// directory for its own SQLite file, LocalDb Phase 2 folded that file into the
|
||||
/// consolidated database and the Host took it over as a <c>SiteLocalDbDirectory</c> shim,
|
||||
/// and LocalDb 0.1.1 took it into the library itself. This test survived all three moves
|
||||
/// unchanged in intent, because what a site node needs is the OUTCOME — resolving
|
||||
/// <c>ILocalDb</c> from the site registration must not fail on a fresh machine — not any
|
||||
/// particular implementer of it. It is deliberately written against the registration path
|
||||
/// rather than the mechanism, so a future move costs nothing here.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// This is not hypothetical: the default site configuration points at the <i>relative</i>
|
||||
/// path <c>./data/site-localdb.db</c>, so any site node whose working directory has no
|
||||
/// <c>data/</c> subdirectory hits it. The docker rig escapes only because its volume mount
|
||||
/// creates <c>/app/data</c> — which is exactly the kind of coincidence that hides a defect
|
||||
/// until a bare-metal or fresh deployment.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public class SiteLocalDbDirectoryTests
|
||||
{
|
||||
[Fact]
|
||||
public void SiteRegistration_CreatesTheLocalDbDirectory_WhenItDoesNotExist()
|
||||
{
|
||||
var root = Path.Combine(Path.GetTempPath(), "localdb-dir-test-" + Guid.NewGuid().ToString("N"));
|
||||
var dbPath = Path.Combine(root, "nested", "site-localdb.db");
|
||||
Assert.False(Directory.Exists(root));
|
||||
|
||||
try
|
||||
{
|
||||
var config = new ConfigurationBuilder()
|
||||
.AddInMemoryCollection(new Dictionary<string, string?>
|
||||
{
|
||||
["LocalDb:Path"] = dbPath,
|
||||
})
|
||||
.Build();
|
||||
|
||||
// The registration path under test. Resolving ILocalDb is what actually opens
|
||||
// the file, so this fails with SQLite Error 14 if the directory step regresses
|
||||
// — wherever it lives.
|
||||
var services = new ServiceCollection();
|
||||
services.AddZbLocalDb(config);
|
||||
|
||||
using var provider = services.BuildServiceProvider();
|
||||
using var scope = provider.CreateScope();
|
||||
var db = scope.ServiceProvider.GetRequiredService<ILocalDb>();
|
||||
|
||||
Assert.NotNull(db);
|
||||
Assert.True(Directory.Exists(Path.GetDirectoryName(dbPath)!));
|
||||
Assert.True(File.Exists(dbPath));
|
||||
}
|
||||
finally
|
||||
{
|
||||
SqliteConnectionPoolCleanup();
|
||||
if (Directory.Exists(root))
|
||||
Directory.Delete(root, recursive: true);
|
||||
}
|
||||
}
|
||||
|
||||
private static void SqliteConnectionPoolCleanup() =>
|
||||
Microsoft.Data.Sqlite.SqliteConnection.ClearAllPools();
|
||||
}
|
||||
@@ -0,0 +1,614 @@
|
||||
using Microsoft.Data.Sqlite;
|
||||
using Microsoft.Extensions.Configuration;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using ZB.MOM.WW.LocalDb;
|
||||
using ZB.MOM.WW.ScadaBridge.Host;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// LocalDb Phase 1 (Task 6) — the one-time copy of the pre-Phase-1 site databases into
|
||||
/// the consolidated one.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// This is a data migration that runs during host startup, so its failure modes are
|
||||
/// expensive: a duplicate-on-rerun bug silently doubles a site's event history, and a
|
||||
/// half-committed copy leaves an operator with no clean state to recover to. Every bullet
|
||||
/// of the plan's semantics gets its own test.
|
||||
/// </remarks>
|
||||
public class SiteLocalDbLegacyMigratorTests : IDisposable
|
||||
{
|
||||
private readonly string _root;
|
||||
private readonly List<ServiceProvider> _providers = [];
|
||||
|
||||
public SiteLocalDbLegacyMigratorTests()
|
||||
{
|
||||
_root = Path.Combine(Path.GetTempPath(), $"localdb-migrator-{Guid.NewGuid():N}");
|
||||
Directory.CreateDirectory(_root);
|
||||
}
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
foreach (var provider in _providers)
|
||||
{
|
||||
try { provider.Dispose(); } catch { /* best effort */ }
|
||||
}
|
||||
|
||||
try { Directory.Delete(_root, recursive: true); } catch { /* best effort */ }
|
||||
GC.SuppressFinalize(this);
|
||||
}
|
||||
|
||||
private string Path_(string name) => System.IO.Path.Combine(_root, name);
|
||||
|
||||
/// <summary>A consolidated database with the tables created and registered, as the host has it.</summary>
|
||||
/// <remarks>
|
||||
/// <c>sf_messages</c> is registered here even though production <c>OnReady</c> does not
|
||||
/// register it until the Task 14 cutover. The oplog assertion below needs live capture
|
||||
/// triggers to mean anything, and the property under test — that the migrator runs after
|
||||
/// registration — is the same either way. The corollary is a real constraint on Task 14:
|
||||
/// <c>Migrate</c> must stay the LAST call in <c>OnReady</c>, after every registration.
|
||||
/// </remarks>
|
||||
private ILocalDb CreateConsolidated(IConfiguration config)
|
||||
{
|
||||
var provider = new ServiceCollection()
|
||||
.AddZbLocalDb(config, db =>
|
||||
{
|
||||
using var connection = db.CreateConnection();
|
||||
ZB.MOM.WW.ScadaBridge.SiteRuntime.Tracking.OperationTrackingSchema.Apply(connection);
|
||||
ZB.MOM.WW.ScadaBridge.SiteEventLogging.SiteEventLogSchema.Apply(connection);
|
||||
ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardSchema.Apply(connection);
|
||||
ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence.SiteStorageSchema.Apply(connection);
|
||||
db.RegisterReplicated("OperationTracking");
|
||||
db.RegisterReplicated("site_events");
|
||||
db.RegisterReplicated("sf_messages");
|
||||
foreach (var table in SiteLocalDbLegacyMigrator.SiteStorageTables)
|
||||
db.RegisterReplicated(table.Table);
|
||||
})
|
||||
.BuildServiceProvider();
|
||||
_providers.Add(provider);
|
||||
|
||||
return provider.GetRequiredService<ILocalDb>();
|
||||
}
|
||||
|
||||
private IConfiguration Config(
|
||||
string? trackingPath = null,
|
||||
string? eventsPath = null,
|
||||
string nodeName = "node-a",
|
||||
string? storeAndForwardPath = null,
|
||||
string? siteDbPath = null)
|
||||
{
|
||||
var values = new Dictionary<string, string?>
|
||||
{
|
||||
["LocalDb:Path"] = Path_("consolidated.db"),
|
||||
["ScadaBridge:Node:NodeName"] = nodeName,
|
||||
|
||||
// Always pinned inside the test's own directory, even when a test does not care
|
||||
// about it. Left unset, the resolver falls back to the CWD-relative code default
|
||||
// "./data/store-and-forward.db" — and the test run's CWD is the test binary's
|
||||
// output directory, so an unlucky run could migrate (and RENAME) a real file.
|
||||
["ScadaBridge:StoreAndForward:SqliteDbPath"] =
|
||||
storeAndForwardPath ?? Path_("absent-store-and-forward.db"),
|
||||
|
||||
// Same reasoning: the site-storage default is "./data/scadabridge.db".
|
||||
["ScadaBridge:Database:SiteDbPath"] = siteDbPath ?? Path_("absent-scadabridge.db"),
|
||||
};
|
||||
|
||||
if (trackingPath is not null)
|
||||
values["ScadaBridge:OperationTracking:ConnectionString"] = $"Data Source={trackingPath}";
|
||||
if (eventsPath is not null)
|
||||
values["ScadaBridge:SiteEventLog:DatabasePath"] = eventsPath;
|
||||
|
||||
return new ConfigurationBuilder().AddInMemoryCollection(values).Build();
|
||||
}
|
||||
|
||||
/// <summary>Seeds a legacy store-and-forward file with the CURRENT column set.</summary>
|
||||
private static void SeedLegacyStoreAndForward(string path, params string[] ids)
|
||||
{
|
||||
using var connection = new SqliteConnection($"Data Source={path}");
|
||||
connection.Open();
|
||||
ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardSchema.Apply(connection);
|
||||
|
||||
foreach (var id in ids)
|
||||
{
|
||||
using var cmd = connection.CreateCommand();
|
||||
cmd.CommandText = """
|
||||
INSERT INTO sf_messages (
|
||||
id, category, target, payload_json, retry_count, max_retries,
|
||||
retry_interval_ms, created_at, status, execution_id)
|
||||
VALUES ($id, 0, 'ERP', '{"order":1}', 0, 50, 30000, $now, 0, 'exec-1');
|
||||
""";
|
||||
cmd.Parameters.AddWithValue("$id", id);
|
||||
cmd.Parameters.AddWithValue("$now", DateTime.UtcNow.ToString("o"));
|
||||
cmd.ExecuteNonQuery();
|
||||
}
|
||||
}
|
||||
|
||||
private static void SeedLegacyTracking(string path, params string[] ids)
|
||||
{
|
||||
using var connection = new SqliteConnection($"Data Source={path}");
|
||||
connection.Open();
|
||||
ZB.MOM.WW.ScadaBridge.SiteRuntime.Tracking.OperationTrackingSchema.Apply(connection);
|
||||
|
||||
foreach (var id in ids)
|
||||
{
|
||||
using var cmd = connection.CreateCommand();
|
||||
cmd.CommandText = """
|
||||
INSERT INTO OperationTracking (
|
||||
TrackedOperationId, Kind, TargetSummary, Status, RetryCount,
|
||||
CreatedAtUtc, UpdatedAtUtc)
|
||||
VALUES ($id, 'ApiCallCached', 'ERP.GetOrder', 'Submitted', 0, $now, $now);
|
||||
""";
|
||||
cmd.Parameters.AddWithValue("$id", id);
|
||||
cmd.Parameters.AddWithValue("$now", DateTime.UtcNow.ToString("o"));
|
||||
cmd.ExecuteNonQuery();
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Seeds the LEGACY event schema: an autoincrement integer id, not a GUID.</summary>
|
||||
private static void SeedLegacyEvents(string path, int count)
|
||||
{
|
||||
using var connection = new SqliteConnection($"Data Source={path}");
|
||||
connection.Open();
|
||||
|
||||
using (var ddl = connection.CreateCommand())
|
||||
{
|
||||
ddl.CommandText = """
|
||||
CREATE TABLE IF NOT EXISTS site_events (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
timestamp TEXT NOT NULL, event_type TEXT NOT NULL, severity TEXT NOT NULL,
|
||||
instance_id TEXT, source TEXT NOT NULL, message TEXT NOT NULL, details TEXT
|
||||
);
|
||||
""";
|
||||
ddl.ExecuteNonQuery();
|
||||
}
|
||||
|
||||
for (var i = 0; i < count; i++)
|
||||
{
|
||||
using var cmd = connection.CreateCommand();
|
||||
cmd.CommandText = """
|
||||
INSERT INTO site_events (timestamp, event_type, severity, source, message)
|
||||
VALUES ($ts, 'script', 'Info', 'src', $msg);
|
||||
""";
|
||||
cmd.Parameters.AddWithValue("$ts", DateTimeOffset.UtcNow.AddMinutes(-i).ToString("o"));
|
||||
cmd.Parameters.AddWithValue("$msg", $"legacy message {i}");
|
||||
cmd.ExecuteNonQuery();
|
||||
}
|
||||
}
|
||||
|
||||
private static long CountRows(ILocalDb db, string table)
|
||||
{
|
||||
using var connection = db.CreateConnection();
|
||||
using var cmd = connection.CreateCommand();
|
||||
cmd.CommandText = $"SELECT COUNT(*) FROM {table}";
|
||||
return (long)cmd.ExecuteScalar()!;
|
||||
}
|
||||
|
||||
private static List<string> SelectIds(ILocalDb db, string table, string idColumn)
|
||||
{
|
||||
using var connection = db.CreateConnection();
|
||||
using var cmd = connection.CreateCommand();
|
||||
cmd.CommandText = $"SELECT {idColumn} FROM {table} ORDER BY {idColumn}";
|
||||
using var reader = cmd.ExecuteReader();
|
||||
|
||||
var ids = new List<string>();
|
||||
while (reader.Read()) ids.Add(reader.GetString(0));
|
||||
return ids;
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void MissingLegacyFiles_AreANoOp()
|
||||
{
|
||||
// The expected case on the docker rig: neither legacy key is set anywhere, and
|
||||
// the CWD-relative defaults point outside the data volume, so there is usually
|
||||
// nothing there at all. That must be a clean no-op, not a startup failure.
|
||||
var config = Config(Path_("absent-tracking.db"), Path_("absent-events.db"));
|
||||
var db = CreateConsolidated(config);
|
||||
|
||||
SiteLocalDbLegacyMigrator.Migrate(db, config);
|
||||
|
||||
Assert.Equal(0, CountRows(db, "OperationTracking"));
|
||||
Assert.Equal(0, CountRows(db, "site_events"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void LegacyTrackingRows_AreCopiedAndTheFileIsRenamed()
|
||||
{
|
||||
var trackingPath = Path_("site-tracking.db");
|
||||
SeedLegacyTracking(trackingPath, "op-1", "op-2", "op-3");
|
||||
var config = Config(trackingPath, Path_("absent-events.db"));
|
||||
var db = CreateConsolidated(config);
|
||||
|
||||
SiteLocalDbLegacyMigrator.Migrate(db, config);
|
||||
|
||||
Assert.Equal(3, CountRows(db, "OperationTracking"));
|
||||
Assert.Equal(["op-1", "op-2", "op-3"], SelectIds(db, "OperationTracking", "TrackedOperationId"));
|
||||
|
||||
// Renamed, not deleted — the operator can still inspect the original.
|
||||
Assert.False(File.Exists(trackingPath));
|
||||
Assert.True(File.Exists(trackingPath + ".migrated"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void LegacyEvents_GetDeterministicIds_NotFreshGuids()
|
||||
{
|
||||
// The whole reason a rerun cannot duplicate. Fresh GUIDs would make
|
||||
// INSERT OR IGNORE useless and double the history on every retry.
|
||||
var eventsPath = Path_("site_events.db");
|
||||
SeedLegacyEvents(eventsPath, count: 3);
|
||||
var config = Config(Path_("absent-tracking.db"), eventsPath, nodeName: "node-b");
|
||||
var db = CreateConsolidated(config);
|
||||
|
||||
SiteLocalDbLegacyMigrator.Migrate(db, config);
|
||||
|
||||
Assert.Equal(
|
||||
["mig-node-b-1", "mig-node-b-2", "mig-node-b-3"],
|
||||
SelectIds(db, "site_events", "id"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void SecondBoot_IsANoOp_BecauseTheFileIsAlreadyMarkedMigrated()
|
||||
{
|
||||
var trackingPath = Path_("site-tracking.db");
|
||||
var eventsPath = Path_("site_events.db");
|
||||
SeedLegacyTracking(trackingPath, "op-1", "op-2");
|
||||
SeedLegacyEvents(eventsPath, count: 2);
|
||||
var config = Config(trackingPath, eventsPath);
|
||||
var db = CreateConsolidated(config);
|
||||
|
||||
SiteLocalDbLegacyMigrator.Migrate(db, config);
|
||||
SiteLocalDbLegacyMigrator.Migrate(db, config);
|
||||
|
||||
Assert.Equal(2, CountRows(db, "OperationTracking"));
|
||||
Assert.Equal(2, CountRows(db, "site_events"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void RerunAgainstAnUnrenamedFile_DoesNotDuplicate()
|
||||
{
|
||||
// Simulates the crash window: the copy committed but the rename did not. On the
|
||||
// next boot the migrator sees the legacy file again and re-copies it. Both tables
|
||||
// must absorb that with no duplicates — tracking via its natural TEXT key, events
|
||||
// via the deterministic mig- id.
|
||||
var trackingPath = Path_("site-tracking.db");
|
||||
var eventsPath = Path_("site_events.db");
|
||||
SeedLegacyTracking(trackingPath, "op-1", "op-2");
|
||||
SeedLegacyEvents(eventsPath, count: 2);
|
||||
var config = Config(trackingPath, eventsPath);
|
||||
var db = CreateConsolidated(config);
|
||||
|
||||
SiteLocalDbLegacyMigrator.Migrate(db, config);
|
||||
|
||||
// Undo the rename, exactly as a crash between commit and rename would leave it.
|
||||
File.Move(trackingPath + ".migrated", trackingPath);
|
||||
File.Move(eventsPath + ".migrated", eventsPath);
|
||||
|
||||
SiteLocalDbLegacyMigrator.Migrate(db, config);
|
||||
|
||||
Assert.Equal(2, CountRows(db, "OperationTracking"));
|
||||
Assert.Equal(2, CountRows(db, "site_events"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void MigratedRows_EnterTheOplog_SoTheyActuallyReplicate()
|
||||
{
|
||||
// The reason the migrator runs AFTER RegisterReplicated. Capture is trigger-based:
|
||||
// migrate before registration and the rows are invisible to the peer forever, with
|
||||
// no error anywhere. Asserting on the oplog is the only way to catch that ordering
|
||||
// being reversed — every other test here would still pass.
|
||||
var trackingPath = Path_("site-tracking.db");
|
||||
SeedLegacyTracking(trackingPath, "op-1", "op-2");
|
||||
var config = Config(trackingPath, Path_("absent-events.db"));
|
||||
var db = CreateConsolidated(config);
|
||||
|
||||
SiteLocalDbLegacyMigrator.Migrate(db, config);
|
||||
|
||||
using var connection = db.CreateConnection();
|
||||
using var cmd = connection.CreateCommand();
|
||||
cmd.CommandText =
|
||||
"SELECT COUNT(*) FROM __localdb_oplog WHERE table_name = 'OperationTracking'";
|
||||
Assert.Equal(2L, (long)cmd.ExecuteScalar()!);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void SfMessages_AreCopiedFromTheLegacyFile()
|
||||
{
|
||||
// Phase 2 (Task 8). Unlike the two Phase 1 files, the store-and-forward default path
|
||||
// is inside the data volume, so on a real deployment this actually moves data:
|
||||
// undelivered messages that a lost migration would silently discard.
|
||||
var sfPath = Path_("store-and-forward.db");
|
||||
SeedLegacyStoreAndForward(sfPath, "msg-1", "msg-2", "msg-3");
|
||||
var config = Config(storeAndForwardPath: sfPath);
|
||||
var db = CreateConsolidated(config);
|
||||
|
||||
SiteLocalDbLegacyMigrator.Migrate(db, config);
|
||||
|
||||
Assert.Equal(3, CountRows(db, "sf_messages"));
|
||||
Assert.Equal(["msg-1", "msg-2", "msg-3"], SelectIds(db, "sf_messages", "id"));
|
||||
|
||||
// Renamed, not deleted — same contract as the Phase 1 files.
|
||||
Assert.False(File.Exists(sfPath));
|
||||
Assert.True(File.Exists(sfPath + ".migrated"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void SfMessages_Migration_IsIdempotent_WhenRerunAfterACrashBeforeRename()
|
||||
{
|
||||
// The crash window: copy committed, rename did not. sf_messages.id is a natural
|
||||
// TEXT key, so INSERT OR IGNORE absorbs the re-copy with no id synthesis needed.
|
||||
var sfPath = Path_("store-and-forward.db");
|
||||
SeedLegacyStoreAndForward(sfPath, "msg-1", "msg-2");
|
||||
var config = Config(storeAndForwardPath: sfPath);
|
||||
var db = CreateConsolidated(config);
|
||||
|
||||
SiteLocalDbLegacyMigrator.Migrate(db, config);
|
||||
File.Move(sfPath + ".migrated", sfPath);
|
||||
SiteLocalDbLegacyMigrator.Migrate(db, config);
|
||||
|
||||
Assert.Equal(2, CountRows(db, "sf_messages"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void MigratedSfMessages_EnterTheOplog_SoTheyActuallyReplicate()
|
||||
{
|
||||
// Same ordering trap as the Phase 1 tables: migrate before RegisterReplicated and
|
||||
// the rows never enter the oplog, never reach the peer, and nothing errors. Only an
|
||||
// assertion on __localdb_oplog catches the ordering being reversed.
|
||||
var sfPath = Path_("store-and-forward.db");
|
||||
SeedLegacyStoreAndForward(sfPath, "msg-1", "msg-2");
|
||||
var config = Config(storeAndForwardPath: sfPath);
|
||||
var db = CreateConsolidated(config);
|
||||
|
||||
SiteLocalDbLegacyMigrator.Migrate(db, config);
|
||||
|
||||
using var connection = db.CreateConnection();
|
||||
using var cmd = connection.CreateCommand();
|
||||
cmd.CommandText = "SELECT COUNT(*) FROM __localdb_oplog WHERE table_name = 'sf_messages'";
|
||||
Assert.Equal(2L, (long)cmd.ExecuteScalar()!);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void SfMessages_FromAnOlderBuild_MigrateDespiteMissingColumns()
|
||||
{
|
||||
// A legacy file written before execution_id / parent_execution_id / last_attempt_at_ms
|
||||
// existed. Naming a missing column in the SELECT throws "no such column", and the
|
||||
// reader treats that as an unrecognised shape — which would silently discard every
|
||||
// buffered message. The copy intersects the column sets instead.
|
||||
var sfPath = Path_("store-and-forward.db");
|
||||
using (var legacy = new SqliteConnection($"Data Source={sfPath}"))
|
||||
{
|
||||
legacy.Open();
|
||||
using var ddl = legacy.CreateCommand();
|
||||
ddl.CommandText = """
|
||||
CREATE TABLE sf_messages (
|
||||
id TEXT PRIMARY KEY, category INTEGER NOT NULL, target TEXT NOT NULL,
|
||||
payload_json TEXT NOT NULL, retry_count INTEGER NOT NULL DEFAULT 0,
|
||||
max_retries INTEGER NOT NULL DEFAULT 50,
|
||||
retry_interval_ms INTEGER NOT NULL DEFAULT 30000,
|
||||
created_at TEXT NOT NULL, last_attempt_at TEXT,
|
||||
status INTEGER NOT NULL DEFAULT 0, last_error TEXT, origin_instance TEXT
|
||||
);
|
||||
INSERT INTO sf_messages (id, category, target, payload_json, created_at)
|
||||
VALUES ('old-1', 0, 'ERP', '{}', '2026-01-01T00:00:00Z');
|
||||
""";
|
||||
ddl.ExecuteNonQuery();
|
||||
}
|
||||
|
||||
var config = Config(storeAndForwardPath: sfPath);
|
||||
var db = CreateConsolidated(config);
|
||||
|
||||
SiteLocalDbLegacyMigrator.Migrate(db, config);
|
||||
|
||||
Assert.Equal(1, CountRows(db, "sf_messages"));
|
||||
Assert.Equal(["old-1"], SelectIds(db, "sf_messages", "id"));
|
||||
}
|
||||
|
||||
/// <summary>Seeds a legacy scadabridge.db with the CURRENT config schema and one row per table.</summary>
|
||||
private static void SeedLegacySiteStorage(string path)
|
||||
{
|
||||
using var connection = new SqliteConnection($"Data Source={path}");
|
||||
connection.Open();
|
||||
ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence.SiteStorageSchema.Apply(connection);
|
||||
|
||||
using var cmd = connection.CreateCommand();
|
||||
cmd.CommandText = """
|
||||
INSERT INTO deployed_configurations
|
||||
(instance_unique_name, config_json, deployment_id, revision_hash, is_enabled, deployed_at)
|
||||
VALUES ('inst-1', '{"a":1}', 'dep-1', 'hash-1', 1, '2026-01-01T00:00:00Z');
|
||||
INSERT INTO static_attribute_overrides
|
||||
(instance_unique_name, attribute_name, override_value, updated_at)
|
||||
VALUES ('inst-1', 'Setpoint', '42', '2026-01-01T00:00:00Z');
|
||||
INSERT INTO shared_scripts (name, code, updated_at)
|
||||
VALUES ('helper', 'return 1;', '2026-01-01T00:00:00Z');
|
||||
INSERT INTO external_systems (name, endpoint_url, auth_type, updated_at, timeout_seconds)
|
||||
VALUES ('ERP', 'http://erp:5200', 'None', '2026-01-01T00:00:00Z', 30);
|
||||
INSERT INTO database_connections (name, connection_string, updated_at)
|
||||
VALUES ('BT', 'Server=sql;Database=BT', '2026-01-01T00:00:00Z');
|
||||
INSERT INTO data_connection_definitions (name, protocol, updated_at)
|
||||
VALUES ('opc-1', 'OpcUa', '2026-01-01T00:00:00Z');
|
||||
INSERT INTO native_alarm_state
|
||||
(instance_unique_name, source_canonical_name, source_reference, condition_json, last_transition_at)
|
||||
VALUES ('inst-1', 'Area.Line', 'ref-1', '{"active":true}', '2026-01-01T00:00:00Z');
|
||||
|
||||
-- The two that must NOT be migrated. A pre-2026-07-10 file really can hold these.
|
||||
INSERT INTO notification_lists (name, recipient_emails, updated_at)
|
||||
VALUES ('ops', 'ops@example.com', '2026-01-01T00:00:00Z');
|
||||
INSERT INTO smtp_configurations
|
||||
(name, server, port, auth_mode, from_address, username, password, updated_at)
|
||||
VALUES ('smtp', 'smtp.example.com', 587, 'Basic', 'a@b.c', 'user',
|
||||
'PLAINTEXT-SECRET', '2026-01-01T00:00:00Z');
|
||||
""";
|
||||
cmd.ExecuteNonQuery();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void SiteConfigTables_AreCopiedFromTheLegacyFile()
|
||||
{
|
||||
// Phase 2 (Task 9). All seven migrated tables live in one legacy file and are copied
|
||||
// in a single transaction: a partial config migration would leave a site node running
|
||||
// against half its old configuration, which is worse than failing startup.
|
||||
var sitePath = Path_("scadabridge.db");
|
||||
SeedLegacySiteStorage(sitePath);
|
||||
var config = Config(siteDbPath: sitePath);
|
||||
var db = CreateConsolidated(config);
|
||||
|
||||
SiteLocalDbLegacyMigrator.Migrate(db, config);
|
||||
|
||||
foreach (var table in SiteLocalDbLegacyMigrator.SiteStorageTables)
|
||||
Assert.Equal(1, CountRows(db, table.Table));
|
||||
|
||||
Assert.False(File.Exists(sitePath));
|
||||
Assert.True(File.Exists(sitePath + ".migrated"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Migration_DoesNotCopyNotificationOrSmtpRows_EvenWhenTheLegacyFileHasThem()
|
||||
{
|
||||
// smtp_configurations.password is PLAINTEXT. Both tables are purged on every deploy
|
||||
// and permanently empty by design since the site write paths were removed
|
||||
// (2026-07-10), but a pre-fix legacy file can still hold rows. The cutover also
|
||||
// declines to REGISTER these two for replication, for the same reason — so migrating
|
||||
// them would leave plaintext SMTP passwords in the consolidated database, one future
|
||||
// RegisterReplicated away from being shipped to a peer, for config nothing reads.
|
||||
var sitePath = Path_("scadabridge.db");
|
||||
SeedLegacySiteStorage(sitePath);
|
||||
var config = Config(siteDbPath: sitePath);
|
||||
var db = CreateConsolidated(config);
|
||||
|
||||
SiteLocalDbLegacyMigrator.Migrate(db, config);
|
||||
|
||||
Assert.Equal(0, CountRows(db, "notification_lists"));
|
||||
Assert.Equal(0, CountRows(db, "smtp_configurations"));
|
||||
|
||||
// Belt and braces: the secret must not be anywhere in the consolidated file,
|
||||
// including the oplog, whose row_json is a json_object copy of every captured row.
|
||||
using var connection = db.CreateConnection();
|
||||
using var cmd = connection.CreateCommand();
|
||||
cmd.CommandText = "SELECT COUNT(*) FROM __localdb_oplog WHERE row_json LIKE '%PLAINTEXT-SECRET%'";
|
||||
Assert.Equal(0L, (long)cmd.ExecuteScalar()!);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void MigratedConfigRows_EnterTheOplog_SoTheyActuallyReplicate()
|
||||
{
|
||||
var sitePath = Path_("scadabridge.db");
|
||||
SeedLegacySiteStorage(sitePath);
|
||||
var config = Config(siteDbPath: sitePath);
|
||||
var db = CreateConsolidated(config);
|
||||
|
||||
SiteLocalDbLegacyMigrator.Migrate(db, config);
|
||||
|
||||
using var connection = db.CreateConnection();
|
||||
using var cmd = connection.CreateCommand();
|
||||
cmd.CommandText =
|
||||
"SELECT COUNT(*) FROM __localdb_oplog WHERE table_name = 'deployed_configurations'";
|
||||
Assert.Equal(1L, (long)cmd.ExecuteScalar()!);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void SiteConfigMigration_IsIdempotent_WhenRerunAfterACrashBeforeRename()
|
||||
{
|
||||
var sitePath = Path_("scadabridge.db");
|
||||
SeedLegacySiteStorage(sitePath);
|
||||
var config = Config(siteDbPath: sitePath);
|
||||
var db = CreateConsolidated(config);
|
||||
|
||||
SiteLocalDbLegacyMigrator.Migrate(db, config);
|
||||
File.Move(sitePath + ".migrated", sitePath);
|
||||
SiteLocalDbLegacyMigrator.Migrate(db, config);
|
||||
|
||||
foreach (var table in SiteLocalDbLegacyMigrator.SiteStorageTables)
|
||||
Assert.Equal(1, CountRows(db, table.Table));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void MigratorColumnLists_MatchTheLiveSchema()
|
||||
{
|
||||
// A column named here but absent from the schema fails at runtime on a real
|
||||
// deployment and NOWHERE else — the intersection logic hides a typo (the column is
|
||||
// simply dropped from the copy) and every other test still passes. A column in the
|
||||
// schema but missing here is worse: that data is silently left behind.
|
||||
//
|
||||
// So this asserts set equality against the schema the host actually applies.
|
||||
using var connection = new SqliteConnection($"Data Source={Path_("schema-probe.db")}");
|
||||
connection.Open();
|
||||
ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence.SiteStorageSchema.Apply(connection);
|
||||
|
||||
foreach (var table in SiteLocalDbLegacyMigrator.SiteStorageTables)
|
||||
{
|
||||
using var probe = connection.CreateCommand();
|
||||
probe.CommandText = $"SELECT name FROM pragma_table_info('{table.Table}')";
|
||||
using var reader = probe.ExecuteReader();
|
||||
|
||||
var actual = new List<string>();
|
||||
while (reader.Read()) actual.Add(reader.GetString(0));
|
||||
|
||||
Assert.Equal(
|
||||
actual.OrderBy(c => c, StringComparer.Ordinal).ToArray(),
|
||||
table.Columns.OrderBy(c => c, StringComparer.Ordinal).ToArray());
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void LegacyFileWithoutTheExpectedTable_IsTreatedAsEmpty_NotAFailure()
|
||||
{
|
||||
// A file predating the table (or an unrecognised shape) must not fail host boot.
|
||||
var trackingPath = Path_("site-tracking.db");
|
||||
using (var connection = new SqliteConnection($"Data Source={trackingPath}"))
|
||||
{
|
||||
connection.Open();
|
||||
using var ddl = connection.CreateCommand();
|
||||
ddl.CommandText = "CREATE TABLE something_else (x INTEGER);";
|
||||
ddl.ExecuteNonQuery();
|
||||
}
|
||||
|
||||
var config = Config(trackingPath, Path_("absent-events.db"));
|
||||
var db = CreateConsolidated(config);
|
||||
|
||||
SiteLocalDbLegacyMigrator.Migrate(db, config);
|
||||
|
||||
Assert.Equal(0, CountRows(db, "OperationTracking"));
|
||||
Assert.True(File.Exists(trackingPath + ".migrated"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void InMemoryLegacyConnectionStrings_AreSkipped()
|
||||
{
|
||||
// Test/dev configurations point the legacy stores at in-memory databases. There is
|
||||
// nothing durable to migrate, and Path.GetFullPath on ":memory:" would be nonsense.
|
||||
var config = new ConfigurationBuilder().AddInMemoryCollection(new Dictionary<string, string?>
|
||||
{
|
||||
["LocalDb:Path"] = Path_("consolidated.db"),
|
||||
["ScadaBridge:Node:NodeName"] = "node-a",
|
||||
["ScadaBridge:OperationTracking:ConnectionString"] =
|
||||
"Data Source=file:legacy?mode=memory&cache=shared",
|
||||
["ScadaBridge:SiteEventLog:DatabasePath"] = ":memory:",
|
||||
}).Build();
|
||||
|
||||
Assert.Equal(string.Empty, SiteLocalDbLegacyMigrator.ResolveTrackingPath(config));
|
||||
Assert.Equal(string.Empty, SiteLocalDbLegacyMigrator.ResolveEventLogPath(config));
|
||||
|
||||
var db = CreateConsolidated(config);
|
||||
SiteLocalDbLegacyMigrator.Migrate(db, config);
|
||||
|
||||
Assert.Equal(0, CountRows(db, "site_events"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void UnsetLegacyKeys_FallBackToTheCwdRelativeCodeDefaults()
|
||||
{
|
||||
// Neither key is set in any docker appsettings, so the OLD code used these
|
||||
// CWD-relative defaults. Resolving them anywhere else (e.g. against the data
|
||||
// volume) would look for the legacy data in a directory it was never written to.
|
||||
var config = new ConfigurationBuilder().AddInMemoryCollection(new Dictionary<string, string?>
|
||||
{
|
||||
["LocalDb:Path"] = Path_("consolidated.db"),
|
||||
}).Build();
|
||||
|
||||
Assert.Equal(
|
||||
System.IO.Path.GetFullPath("site-tracking.db"),
|
||||
SiteLocalDbLegacyMigrator.ResolveTrackingPath(config));
|
||||
Assert.Equal(
|
||||
System.IO.Path.GetFullPath("site_events.db"),
|
||||
SiteLocalDbLegacyMigrator.ResolveEventLogPath(config));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,327 @@
|
||||
using Microsoft.AspNetCore.Builder;
|
||||
using Microsoft.Extensions.Configuration;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.Hosting;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.ScadaBridge.HealthMonitoring;
|
||||
using ZB.MOM.WW.Telemetry;
|
||||
using ZB.MOM.WW.LocalDb;
|
||||
using ZB.MOM.WW.LocalDb.Replication;
|
||||
using ZB.MOM.WW.ScadaBridge.Host;
|
||||
using ZB.MOM.WW.ScadaBridge.Host.Actors;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// LocalDb Phase 1 (Task 3) — locks the consolidated site database into the REAL site
|
||||
/// composition root.
|
||||
/// <para>
|
||||
/// These tests build the actual container from
|
||||
/// <see cref="SiteServiceRegistration.Configure"/> rather than asserting on a
|
||||
/// hand-assembled <c>ServiceCollection</c>. That is deliberate and load-bearing: this
|
||||
/// family has now shipped three wiring defects that only a container-built graph would
|
||||
/// have caught — the Secrets Akka replicator that registered into a no-op sink (0.2.0),
|
||||
/// the singleton cycle that deadlocked a real host (0.2.2), and Secrets.Ui's missing
|
||||
/// <c>IAuditWriter</c> that 500'd only behind a login wall (ScadaBridge#22). A test that
|
||||
/// merely checks "AddZbLocalDb was called" would have passed in every one of those cases.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public class SiteLocalDbWiringTests : IDisposable
|
||||
{
|
||||
private readonly WebApplication _host;
|
||||
private readonly string _tempDbPath;
|
||||
private readonly string _tempSiteDbPath;
|
||||
private readonly string _tempTrackingDbPath;
|
||||
|
||||
public SiteLocalDbWiringTests()
|
||||
{
|
||||
var stamp = Guid.NewGuid();
|
||||
_tempDbPath = Path.Combine(Path.GetTempPath(), $"scadabridge_localdb_{stamp}.db");
|
||||
_tempSiteDbPath = Path.Combine(Path.GetTempPath(), $"scadabridge_site_{stamp}.db");
|
||||
_tempTrackingDbPath = Path.Combine(Path.GetTempPath(), $"scadabridge_track_{stamp}.db");
|
||||
|
||||
var builder = WebApplication.CreateBuilder();
|
||||
builder.Configuration.Sources.Clear();
|
||||
builder.Configuration.AddInMemoryCollection(new Dictionary<string, string?>
|
||||
{
|
||||
["ScadaBridge:Node:Role"] = "Site",
|
||||
["ScadaBridge:Node:NodeName"] = "node-a",
|
||||
["ScadaBridge:Node:NodeHostname"] = "test-site",
|
||||
["ScadaBridge:Node:SiteId"] = "TestSite",
|
||||
["ScadaBridge:Node:RemotingPort"] = "0",
|
||||
["ScadaBridge:Node:GrpcPort"] = "0",
|
||||
["ScadaBridge:Database:SiteDbPath"] = _tempSiteDbPath,
|
||||
["ScadaBridge:OperationTracking:ConnectionString"] = $"Data Source={_tempTrackingDbPath}",
|
||||
["ScadaBridge:Cluster:SeedNodes:0"] = "akka.tcp://scadabridge@localhost:2551",
|
||||
["ScadaBridge:Cluster:SeedNodes:1"] = "akka.tcp://scadabridge@localhost:2552",
|
||||
|
||||
// The consolidated site database. No Replication section at all — this
|
||||
// fixture is also the default-OFF pin: storage must work standalone.
|
||||
["LocalDb:Path"] = _tempDbPath,
|
||||
});
|
||||
|
||||
builder.Services.AddGrpc();
|
||||
builder.Services.AddSingleton<ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamGrpcServer>();
|
||||
|
||||
SiteServiceRegistration.Configure(builder.Services, builder.Configuration);
|
||||
|
||||
AkkaHostedServiceRemover.RemoveAkkaHostedServiceOnly(builder.Services);
|
||||
|
||||
_host = builder.Build();
|
||||
}
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
(_host as IDisposable)?.Dispose();
|
||||
foreach (var path in new[] { _tempDbPath, _tempSiteDbPath, _tempTrackingDbPath })
|
||||
{
|
||||
try { File.Delete(path); } catch { /* best effort */ }
|
||||
}
|
||||
|
||||
GC.SuppressFinalize(this);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Site_Resolves_ILocalDb()
|
||||
{
|
||||
Assert.NotNull(_host.Services.GetService<ILocalDb>());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Site_LocalDb_RegistersBothReplicatedTables()
|
||||
{
|
||||
// The whole point of Phase 1. If onReady silently failed to register these,
|
||||
// storage would still work and every other test would pass — the pair would
|
||||
// just never replicate anything. Assert on the library's own view of what is
|
||||
// registered, not on our belief that we called it.
|
||||
var db = _host.Services.GetRequiredService<ILocalDb>();
|
||||
|
||||
Assert.Contains("OperationTracking", db.ReplicatedTables.Keys);
|
||||
Assert.Contains("site_events", db.ReplicatedTables.Keys);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Site_LocalDb_ReplicatedTablesHaveExplicitPrimaryKeys()
|
||||
{
|
||||
// RegisterReplicated refuses a table with no explicit PK, so reaching this
|
||||
// assertion at all proves the DDL ran before registration. The PK names are
|
||||
// pinned because they are what last-writer-wins conflict resolution keys on.
|
||||
var db = _host.Services.GetRequiredService<ILocalDb>();
|
||||
|
||||
Assert.Equal(
|
||||
["TrackedOperationId"], db.ReplicatedTables["OperationTracking"].PkColumns);
|
||||
Assert.Equal(["id"], db.ReplicatedTables["site_events"].PkColumns);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Site_LocalDb_IsASingleton()
|
||||
{
|
||||
// AddZbLocalDb is one-DB-per-process and first-registration-wins. Two
|
||||
// instances would mean two SQLite handles and two trigger installations
|
||||
// against the same file.
|
||||
var first = _host.Services.GetRequiredService<ILocalDb>();
|
||||
var second = _host.Services.GetRequiredService<ILocalDb>();
|
||||
|
||||
Assert.Same(first, second);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Site_Replication_IsRegistered_AndInertWithNoPeer()
|
||||
{
|
||||
// Task 7's default-OFF pin. This fixture configures NO
|
||||
// LocalDb:Replication:PeerAddress, which is the shipping default, so the
|
||||
// engine must resolve cleanly and sit idle rather than throw, retry, or
|
||||
// report a connection. "Default-off" here means inert, NOT unregistered —
|
||||
// the services are always in the graph.
|
||||
var status = _host.Services.GetService<ISyncStatus>();
|
||||
|
||||
Assert.NotNull(status);
|
||||
Assert.False(status!.Connected);
|
||||
Assert.Null(status.PeerNodeId);
|
||||
Assert.Null(status.LastSyncUtc);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Site_Replication_OplogBacklog_IsZero_NotNull_OnAHealthyIdleNode()
|
||||
{
|
||||
// OplogBacklog is deliberately nullable: null means "unknown" (no provider
|
||||
// wired, or the poll failed) and must never be reported as a healthy 0. On a
|
||||
// node whose local database is present and readable, a real 0 proves the
|
||||
// backlog provider is actually wired to the oplog — a null here would mean
|
||||
// Task 9's health signal is about to publish "unknown" forever.
|
||||
var status = _host.Services.GetRequiredService<ISyncStatus>();
|
||||
|
||||
Assert.Equal(0L, status.OplogBacklog);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Site_Replication_HealthBridge_IsRegisteredAsAHostedService()
|
||||
{
|
||||
// Task 9. Registered via a factory lambda, so ImplementationType is null and the
|
||||
// only honest assertion is on the resolved instance.
|
||||
var hosted = _host.Services.GetServices<IHostedService>();
|
||||
|
||||
Assert.Contains(hosted, h => h is LocalDbReplicationStatusReporter);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Site_Replication_HealthBridge_PublishesStatusOntoTheHealthReport()
|
||||
{
|
||||
// End-to-end through the REAL collector: probe once, then collect. This is what
|
||||
// catches a bridge that is registered but wired to the wrong provider — the
|
||||
// registration test above would pass either way.
|
||||
var reporter = _host.Services.GetServices<IHostedService>()
|
||||
.OfType<LocalDbReplicationStatusReporter>()
|
||||
.Single();
|
||||
var collector = _host.Services.GetRequiredService<ISiteHealthCollector>();
|
||||
|
||||
reporter.Probe();
|
||||
var report = collector.CollectReport("TestSite");
|
||||
|
||||
// No peer configured: not connected, and a REAL 0 backlog rather than null.
|
||||
Assert.False(report.LocalDbReplicationConnected);
|
||||
Assert.Equal(0L, report.LocalDbOplogBacklog);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Site_HealthReport_LocalDbFields_AreNull_BeforeTheReporterHasRun()
|
||||
{
|
||||
// Null means "no data yet", and must be distinguishable from a real
|
||||
// "disconnected, zero backlog". A collector that defaulted these to false/0 would
|
||||
// report an unwired node as a healthy connected one.
|
||||
var collector = new SiteHealthCollector();
|
||||
|
||||
var report = collector.CollectReport("TestSite");
|
||||
|
||||
Assert.Null(report.LocalDbReplicationConnected);
|
||||
Assert.Null(report.LocalDbOplogBacklog);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Site_Telemetry_Allowlists_TheLocalDbReplicationMeter()
|
||||
{
|
||||
// ZbTelemetryOptions.Meters is an ALLOWLIST: an unlisted Meter's instruments are
|
||||
// never observed, with no error and no missing-metric warning anywhere. The
|
||||
// replication meter was silently absent from the rig's /metrics scrape until the
|
||||
// live gate looked for it, so pin the allowlist rather than trusting the next
|
||||
// reader to remember.
|
||||
//
|
||||
// This asserts on the allowlist constant, not on OTel's observed-meter set:
|
||||
// AddZbTelemetry applies its options to a local instance and never registers
|
||||
// IOptions, so there is nothing resolvable to interrogate. The end-to-end proof
|
||||
// that localdb_* actually reaches /metrics is the Task 12 live gate; this test
|
||||
// exists to stop the entry being dropped again.
|
||||
Assert.Contains(LocalDbMetrics.MeterName, SiteServiceRegistration.ObservedMeters);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Site_LocalDb_CreatesThePhase2Tables_InTheConsolidatedFile()
|
||||
{
|
||||
// Phase 2 (Task 7). The nine configuration tables and the store-and-forward buffer
|
||||
// now live in the consolidated file rather than in scadabridge.db and
|
||||
// store-and-forward.db. Asserting through the REAL composition root is what proves
|
||||
// OnReady applies both schemas — the stores themselves also call Apply, so a test
|
||||
// that went through a store would pass even if OnReady never touched them.
|
||||
var db = _host.Services.GetRequiredService<ILocalDb>();
|
||||
|
||||
var tables = TableNames(db);
|
||||
|
||||
// One from each Phase 1 schema, then every Phase 2 table.
|
||||
Assert.Contains("OperationTracking", tables);
|
||||
Assert.Contains("site_events", tables);
|
||||
Assert.Contains("sf_messages", tables);
|
||||
foreach (var table in new[]
|
||||
{
|
||||
"deployed_configurations", "static_attribute_overrides", "shared_scripts",
|
||||
"external_systems", "database_connections", "notification_lists",
|
||||
"data_connection_definitions", "smtp_configurations", "native_alarm_state",
|
||||
})
|
||||
{
|
||||
Assert.Contains(table, tables);
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Site_LocalDb_RegistersExactlyTheTenReplicatedTables()
|
||||
{
|
||||
// The Phase 2 cutover. This is an EXACTLY check rather than a Contains in both
|
||||
// directions, and both directions are load-bearing.
|
||||
//
|
||||
// Too few: the table silently stops replicating. Storage still works, every other
|
||||
// test still passes, and the pair just quietly diverges — the failure mode Phase 1
|
||||
// existed to end.
|
||||
//
|
||||
// Too many: notification_lists and smtp_configurations must NOT be here. They are
|
||||
// permanently empty by design, and registering them would open a standing
|
||||
// replication channel whose only historical payload was plaintext SMTP passwords.
|
||||
// A Contains-based test would never catch that.
|
||||
var db = _host.Services.GetRequiredService<ILocalDb>();
|
||||
|
||||
Assert.Equal(
|
||||
[
|
||||
// Ordinal order: '_' (0x5F) sorts before 'b' (0x62), so
|
||||
// data_connection_definitions precedes database_connections.
|
||||
"OperationTracking", "data_connection_definitions", "database_connections",
|
||||
"deployed_configurations", "external_systems", "native_alarm_state",
|
||||
"sf_messages", "shared_scripts", "site_events", "static_attribute_overrides",
|
||||
],
|
||||
db.ReplicatedTables.Keys.OrderBy(k => k, StringComparer.Ordinal).ToArray());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Site_LocalDb_DoesNotReplicateTheCentralOnlyNotificationTables()
|
||||
{
|
||||
// Stated separately from the exact-set test above because this one is a security
|
||||
// property, not a wiring property, and deserves to fail with its own name. The
|
||||
// tables exist (SiteStorageSchema creates them) — they simply must never be
|
||||
// captured. See SiteLocalDbSetup.OnReady for the rationale.
|
||||
var db = _host.Services.GetRequiredService<ILocalDb>();
|
||||
|
||||
Assert.DoesNotContain("notification_lists", db.ReplicatedTables.Keys);
|
||||
Assert.DoesNotContain("smtp_configurations", db.ReplicatedTables.Keys);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Site_LocalDb_ReplicatedPhase2TablesHaveTheirExpectedPrimaryKeys()
|
||||
{
|
||||
// RegisterReplicated refuses a table with no explicit PK, so reaching these
|
||||
// assertions proves the DDL ran before registration. The composite keys are the
|
||||
// interesting ones: LWW conflict resolution keys on the FULL PK, so a wrong or
|
||||
// truncated key set would silently collapse distinct rows into one.
|
||||
var db = _host.Services.GetRequiredService<ILocalDb>();
|
||||
|
||||
Assert.Equal(["id"], db.ReplicatedTables["sf_messages"].PkColumns);
|
||||
Assert.Equal(
|
||||
["instance_unique_name"], db.ReplicatedTables["deployed_configurations"].PkColumns);
|
||||
Assert.Equal(
|
||||
["instance_unique_name", "attribute_name"],
|
||||
db.ReplicatedTables["static_attribute_overrides"].PkColumns);
|
||||
Assert.Equal(
|
||||
["instance_unique_name", "source_canonical_name", "source_reference"],
|
||||
db.ReplicatedTables["native_alarm_state"].PkColumns);
|
||||
}
|
||||
|
||||
private static HashSet<string> TableNames(ILocalDb db)
|
||||
{
|
||||
using var connection = db.CreateConnection();
|
||||
using var cmd = connection.CreateCommand();
|
||||
cmd.CommandText = "SELECT name FROM sqlite_master WHERE type = 'table'";
|
||||
using var reader = cmd.ExecuteReader();
|
||||
|
||||
var names = new HashSet<string>(StringComparer.Ordinal);
|
||||
while (reader.Read()) names.Add(reader.GetString(0));
|
||||
return names;
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Site_LocalDb_CreatesTheConfiguredFile()
|
||||
{
|
||||
// Resolving is what triggers onReady; the file should exist afterwards at the
|
||||
// configured path and not somewhere CWD-relative.
|
||||
_ = _host.Services.GetRequiredService<ILocalDb>();
|
||||
|
||||
Assert.True(File.Exists(_tempDbPath),
|
||||
$"Expected the consolidated site database at '{_tempDbPath}'.");
|
||||
}
|
||||
}
|
||||
@@ -232,15 +232,22 @@ public class StartupValidatorTests
|
||||
Assert.Null(ex);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The inverse of the rule this replaces. SiteDbPath was mandatory for Site
|
||||
/// nodes until LocalDb Phase 2 moved the site tables into the consolidated
|
||||
/// LocalDb database; it now names only the legacy file the boot-time migrator
|
||||
/// drains, so its absence means "nothing to migrate". Requiring it would make
|
||||
/// every already-migrated node carry a dead key forever.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void Site_MissingSiteDbPath_FailsValidation()
|
||||
public void Site_MissingSiteDbPath_IsAccepted_BecauseThePathIsMigrationOnly()
|
||||
{
|
||||
var values = ValidSiteConfig();
|
||||
values.Remove("ScadaBridge:Database:SiteDbPath");
|
||||
var config = BuildConfig(values);
|
||||
|
||||
var ex = Assert.Throws<InvalidOperationException>(() => StartupValidator.Validate(config));
|
||||
Assert.Contains("SiteDbPath required for Site nodes", ex.Message);
|
||||
var ex = Record.Exception(() => StartupValidator.Validate(config));
|
||||
Assert.Null(ex);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
|
||||
@@ -1,94 +0,0 @@
|
||||
using Akka.Actor;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
|
||||
using ZB.MOM.WW.ScadaBridge.StoreAndForward;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.IntegrationTests.Cluster;
|
||||
|
||||
/// <summary>
|
||||
/// N1 regression (review 02 round 2, Critical): the resync authority must use the same
|
||||
/// oldest-Up predicate as the S&F delivery gate. Divergence scenario = the delivering node
|
||||
/// is OLDEST but not LEADER (leader = lowest address), the exact state a rolling restart of
|
||||
/// the lower-address node produces. Pre-fix the delivering node requests a resync from the
|
||||
/// stale peer and ReplaceAllAsync wipes its live buffer.
|
||||
/// </summary>
|
||||
public class SfBufferResyncPredicateTests
|
||||
{
|
||||
[Fact]
|
||||
public async Task OldestButNotLeaderNode_KeepsItsBuffer_AndSeedsTheJoiner()
|
||||
{
|
||||
// Two explicit ports, deliberately assigned so the FIRST-started (oldest,
|
||||
// delivering) node has the HIGHER address → the second node is cluster leader.
|
||||
var p1 = TwoNodeClusterFixture.GetFreeTcpPort();
|
||||
var p2 = TwoNodeClusterFixture.GetFreeTcpPort();
|
||||
var (portHigh, portLow) = p1 > p2 ? (p1, p2) : (p2, p1);
|
||||
|
||||
await using var fixture = await TwoNodeClusterFixture.StartAsync(
|
||||
role: "site-int", portA: portHigh, portB: portLow);
|
||||
|
||||
// Real S&F storage + replication actor per node, production default predicate
|
||||
// (no isActiveOverride) — the exact wiring under test.
|
||||
var (storageOldest, _) = await CreateReplicationActorAsync(fixture.NodeA, "oldest");
|
||||
var (storageJoiner, _) = await CreateReplicationActorAsync(fixture.NodeB, "joiner");
|
||||
|
||||
// The delivering (oldest) node has a live buffered row the standby never saw.
|
||||
await storageOldest.EnqueueAsync(NewMessage("live-row"));
|
||||
|
||||
// Trigger peer (re)tracking on both sides: each actor got InitialStateAsSnapshot
|
||||
// in PreStart, but the enqueue raced it — re-deliver via a fresh MemberUp is not
|
||||
// needed; OnPeerTracked already fired on join. The resync exchange is async:
|
||||
// wait until the JOINER holds the row (proves the snapshot flowed oldest→joiner,
|
||||
// the correct direction). Pre-fix this times out (the joiner, as leader, never
|
||||
// requests) AND the oldest node's row is deleted by the stale wipe.
|
||||
await AwaitAsync(async () => await storageJoiner.GetMessageByIdAsync("live-row") != null,
|
||||
TimeSpan.FromSeconds(20),
|
||||
"joiner never received the resync snapshot (resync ran in the wrong direction)");
|
||||
|
||||
// And the delivering node's buffer is untouched — the N1 wipe assertion.
|
||||
Assert.NotNull(await storageOldest.GetMessageByIdAsync("live-row"));
|
||||
}
|
||||
|
||||
private static async Task<(StoreAndForwardStorage Storage, IActorRef Actor)> CreateReplicationActorAsync(
|
||||
ActorSystem node, string tag)
|
||||
{
|
||||
var sfDb = Path.Combine(Path.GetTempPath(), $"sf-resync-{tag}-{Guid.NewGuid():N}.db");
|
||||
var siteDb = Path.Combine(Path.GetTempPath(), $"site-resync-{tag}-{Guid.NewGuid():N}.db");
|
||||
var sfStorage = new StoreAndForwardStorage($"Data Source={sfDb}",
|
||||
NullLogger<StoreAndForwardStorage>.Instance);
|
||||
await sfStorage.InitializeAsync();
|
||||
var siteStorage = new SiteStorageService($"Data Source={siteDb}",
|
||||
NullLogger<SiteStorageService>.Instance);
|
||||
var replicationService = new ReplicationService(
|
||||
new StoreAndForwardOptions(), NullLogger<ReplicationService>.Instance);
|
||||
// Name MUST be "site-replication" — SendToPeer targets /user/site-replication.
|
||||
var actor = node.ActorOf(Props.Create(() => new SiteReplicationActor(
|
||||
siteStorage, sfStorage, replicationService, "site-int",
|
||||
NullLogger<SiteReplicationActor>.Instance, null, null, null, null)),
|
||||
"site-replication");
|
||||
return (sfStorage, actor);
|
||||
}
|
||||
|
||||
private static StoreAndForwardMessage NewMessage(string id) => new()
|
||||
{
|
||||
Id = id,
|
||||
Category = StoreAndForwardCategory.Notification,
|
||||
Target = "central",
|
||||
PayloadJson = "{}",
|
||||
CreatedAt = DateTimeOffset.UtcNow,
|
||||
Status = StoreAndForwardMessageStatus.Pending,
|
||||
MaxRetries = 0,
|
||||
};
|
||||
|
||||
private static async Task AwaitAsync(Func<Task<bool>> condition, TimeSpan timeout, string why)
|
||||
{
|
||||
var deadline = DateTime.UtcNow + timeout;
|
||||
while (DateTime.UtcNow < deadline)
|
||||
{
|
||||
if (await condition()) return;
|
||||
await Task.Delay(250);
|
||||
}
|
||||
throw new TimeoutException(why);
|
||||
}
|
||||
}
|
||||
@@ -1,6 +1,7 @@
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
|
||||
using ZB.MOM.WW.ScadaBridge.StoreAndForward;
|
||||
using ZB.MOM.WW.ScadaBridge.TestSupport;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.IntegrationTests;
|
||||
|
||||
@@ -18,12 +19,17 @@ public class DualNodeRecoveryTests
|
||||
// Scenario: both site nodes crash. First node to restart opens the existing
|
||||
// SQLite database and finds all buffered S&F messages intact.
|
||||
var dbPath = Path.Combine(Path.GetTempPath(), $"sf_dual_{Guid.NewGuid():N}.db");
|
||||
var connStr = $"Data Source={dbPath}";
|
||||
|
||||
// The store takes an ILocalDb; the "restarted node" opens its own local database
|
||||
// over the SAME file, which is how this test models recovery from disk.
|
||||
TestLocalDb? crashedDb = null;
|
||||
TestLocalDb? recoveryDb = null;
|
||||
|
||||
try
|
||||
{
|
||||
// Setup: populate SQLite with messages (simulating pre-crash state)
|
||||
var storage = new StoreAndForwardStorage(connStr, NullLogger<StoreAndForwardStorage>.Instance);
|
||||
crashedDb = TestLocalDb.Create(dbPath);
|
||||
var storage = new StoreAndForwardStorage(crashedDb.Db, NullLogger<StoreAndForwardStorage>.Instance);
|
||||
await storage.InitializeAsync();
|
||||
|
||||
var messageIds = new List<string>();
|
||||
@@ -48,7 +54,8 @@ public class DualNodeRecoveryTests
|
||||
|
||||
// Both nodes down — simulate by creating a fresh storage instance
|
||||
// (new process connecting to same SQLite file)
|
||||
var recoveryStorage = new StoreAndForwardStorage(connStr, NullLogger<StoreAndForwardStorage>.Instance);
|
||||
recoveryDb = TestLocalDb.Create(dbPath);
|
||||
var recoveryStorage = new StoreAndForwardStorage(recoveryDb.Db, NullLogger<StoreAndForwardStorage>.Instance);
|
||||
await recoveryStorage.InitializeAsync();
|
||||
|
||||
// Verify all messages are available for retry
|
||||
@@ -69,8 +76,10 @@ public class DualNodeRecoveryTests
|
||||
}
|
||||
finally
|
||||
{
|
||||
if (File.Exists(dbPath))
|
||||
File.Delete(dbPath);
|
||||
// Dispose before deleting — the master connections anchor the WAL sidecars.
|
||||
recoveryDb?.Dispose();
|
||||
crashedDb?.Dispose();
|
||||
TestLocalDb.DeleteFiles(dbPath);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -79,11 +88,14 @@ public class DualNodeRecoveryTests
|
||||
public async Task SiteTopology_DualCrash_ParkedMessagesPreserved()
|
||||
{
|
||||
var dbPath = Path.Combine(Path.GetTempPath(), $"sf_dual_parked_{Guid.NewGuid():N}.db");
|
||||
var connStr = $"Data Source={dbPath}";
|
||||
|
||||
TestLocalDb? crashedDb = null;
|
||||
TestLocalDb? recoveryDb = null;
|
||||
|
||||
try
|
||||
{
|
||||
var storage = new StoreAndForwardStorage(connStr, NullLogger<StoreAndForwardStorage>.Instance);
|
||||
crashedDb = TestLocalDb.Create(dbPath);
|
||||
var storage = new StoreAndForwardStorage(crashedDb.Db, NullLogger<StoreAndForwardStorage>.Instance);
|
||||
await storage.InitializeAsync();
|
||||
|
||||
// Mix of pending and parked messages
|
||||
@@ -114,7 +126,8 @@ public class DualNodeRecoveryTests
|
||||
});
|
||||
|
||||
// Dual crash recovery
|
||||
var recoveryStorage = new StoreAndForwardStorage(connStr, NullLogger<StoreAndForwardStorage>.Instance);
|
||||
recoveryDb = TestLocalDb.Create(dbPath);
|
||||
var recoveryStorage = new StoreAndForwardStorage(recoveryDb.Db, NullLogger<StoreAndForwardStorage>.Instance);
|
||||
await recoveryStorage.InitializeAsync();
|
||||
|
||||
var pendingCount = await recoveryStorage.GetMessageCountByStatusAsync(StoreAndForwardMessageStatus.Pending);
|
||||
@@ -134,8 +147,10 @@ public class DualNodeRecoveryTests
|
||||
}
|
||||
finally
|
||||
{
|
||||
if (File.Exists(dbPath))
|
||||
File.Delete(dbPath);
|
||||
// Dispose before deleting — the master connections anchor the WAL sidecars.
|
||||
recoveryDb?.Dispose();
|
||||
crashedDb?.Dispose();
|
||||
TestLocalDb.DeleteFiles(dbPath);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -176,11 +191,14 @@ public class DualNodeRecoveryTests
|
||||
{
|
||||
// CREATE TABLE IF NOT EXISTS is idempotent — safe to call on recovery
|
||||
var dbPath = Path.Combine(Path.GetTempPath(), $"sf_idempotent_{Guid.NewGuid():N}.db");
|
||||
var connStr = $"Data Source={dbPath}";
|
||||
|
||||
TestLocalDb? localDb1 = null;
|
||||
TestLocalDb? localDb2 = null;
|
||||
|
||||
try
|
||||
{
|
||||
var storage1 = new StoreAndForwardStorage(connStr, NullLogger<StoreAndForwardStorage>.Instance);
|
||||
localDb1 = TestLocalDb.Create(dbPath);
|
||||
var storage1 = new StoreAndForwardStorage(localDb1.Db, NullLogger<StoreAndForwardStorage>.Instance);
|
||||
await storage1.InitializeAsync();
|
||||
|
||||
await storage1.EnqueueAsync(new StoreAndForwardMessage
|
||||
@@ -196,7 +214,8 @@ public class DualNodeRecoveryTests
|
||||
});
|
||||
|
||||
// Second InitializeAsync on same DB should be safe (no data loss)
|
||||
var storage2 = new StoreAndForwardStorage(connStr, NullLogger<StoreAndForwardStorage>.Instance);
|
||||
localDb2 = TestLocalDb.Create(dbPath);
|
||||
var storage2 = new StoreAndForwardStorage(localDb2.Db, NullLogger<StoreAndForwardStorage>.Instance);
|
||||
await storage2.InitializeAsync();
|
||||
|
||||
var msg = await storage2.GetMessageByIdAsync("test-1");
|
||||
@@ -205,8 +224,10 @@ public class DualNodeRecoveryTests
|
||||
}
|
||||
finally
|
||||
{
|
||||
if (File.Exists(dbPath))
|
||||
File.Delete(dbPath);
|
||||
// Dispose before deleting — the master connections anchor the WAL sidecars.
|
||||
localDb2?.Dispose();
|
||||
localDb1?.Dispose();
|
||||
TestLocalDb.DeleteFiles(dbPath);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -91,12 +91,12 @@ public class IntegrationSurfaceTests
|
||||
{
|
||||
// Notification Outbox: Notify.Send enqueues into the site Store-and-Forward
|
||||
// Engine and returns the NotificationId handle immediately.
|
||||
var dbName = $"NotifyWired_{Guid.NewGuid():N}";
|
||||
var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared";
|
||||
using var keepAlive = new Microsoft.Data.Sqlite.SqliteConnection(connStr);
|
||||
keepAlive.Open();
|
||||
// A real temp-file LocalDb, not the shared-cache in-memory database this used
|
||||
// before: StoreAndForwardStorage takes ILocalDb now, and LocalDb has no
|
||||
// in-memory mode.
|
||||
using var localDb = ZB.MOM.WW.ScadaBridge.TestSupport.TestLocalDb.CreateTemp("NotifyWired");
|
||||
var storage = new StoreAndForward.StoreAndForwardStorage(
|
||||
connStr, Microsoft.Extensions.Logging.Abstractions.NullLogger<StoreAndForward.StoreAndForwardStorage>.Instance);
|
||||
localDb.Db, Microsoft.Extensions.Logging.Abstractions.NullLogger<StoreAndForward.StoreAndForwardStorage>.Instance);
|
||||
await storage.InitializeAsync();
|
||||
var saf = new StoreAndForward.StoreAndForwardService(
|
||||
storage, new StoreAndForward.StoreAndForwardOptions(),
|
||||
|
||||
@@ -0,0 +1,177 @@
|
||||
using ZB.MOM.WW.LocalDb;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.IntegrationTests;
|
||||
|
||||
/// <summary>
|
||||
/// Phase 2 convergence: deployed configuration across a real site pair, plus the resync
|
||||
/// behaviour that the bespoke replicator needed a directional guard for.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Companion to <see cref="LocalDbStoreAndForwardConvergenceTests"/>: same
|
||||
/// specifications-before-deletion role, for the intents held by
|
||||
/// <c>SiteReplicationActorTests</c> and <c>SfBufferResyncPredicateTests</c> rather than by
|
||||
/// the store-and-forward <c>ReplicationService</c>.
|
||||
/// </remarks>
|
||||
[Collection("LocalDbSitePairConvergence")]
|
||||
public sealed class LocalDbConfigConvergenceTests : LocalDbSitePairHarness
|
||||
{
|
||||
// ---- data helpers -----------------------------------------------------------------
|
||||
|
||||
private static Task DeployConfigAsync(ILocalDb db, string instance, string configJson, string deploymentId)
|
||||
=> db.ExecuteAsync(
|
||||
"""
|
||||
INSERT INTO deployed_configurations (
|
||||
instance_unique_name, config_json, deployment_id, revision_hash,
|
||||
is_enabled, deployed_at)
|
||||
VALUES (@instance, @configJson, @deploymentId, @deploymentId, 1, @now)
|
||||
ON CONFLICT(instance_unique_name) DO UPDATE SET
|
||||
config_json = excluded.config_json,
|
||||
deployment_id = excluded.deployment_id,
|
||||
revision_hash = excluded.revision_hash,
|
||||
deployed_at = excluded.deployed_at;
|
||||
""",
|
||||
new { instance, configJson, deploymentId, now = DateTime.UtcNow.ToString("o") });
|
||||
|
||||
private static async Task<string?> ReadConfigAsync(ILocalDb db, string instance)
|
||||
{
|
||||
var rows = await db.QueryAsync(
|
||||
"SELECT config_json FROM deployed_configurations WHERE instance_unique_name = @instance",
|
||||
static r => r.GetString(0), new { instance });
|
||||
return rows.Count == 0 ? null : rows[0];
|
||||
}
|
||||
|
||||
private static Task EnqueueAsync(ILocalDb db, string id)
|
||||
=> db.ExecuteAsync(
|
||||
"""
|
||||
INSERT INTO sf_messages (
|
||||
id, category, target, payload_json, retry_count, max_retries,
|
||||
retry_interval_ms, created_at, status)
|
||||
VALUES (@id, 0, 'central', '{}', 0, 50, 30000, @now, 0);
|
||||
""",
|
||||
new { id, now = DateTime.UtcNow.ToString("o") });
|
||||
|
||||
private static async Task<bool> MessageExistsAsync(ILocalDb db, string id)
|
||||
{
|
||||
var rows = await db.QueryAsync(
|
||||
"SELECT 1 FROM sf_messages WHERE id = @id", static r => r.GetInt32(0), new { id });
|
||||
return rows.Count > 0;
|
||||
}
|
||||
|
||||
// ---- scenarios --------------------------------------------------------------------
|
||||
|
||||
[Fact]
|
||||
public async Task ANodeWithNewerLocalRows_KeepsThem_WhenAPeerSnapshotArrives()
|
||||
{
|
||||
// N1 Critical (was: SfBufferResyncPredicateTests). That test existed because the
|
||||
// bespoke replicator's ReplaceAllAsync was a destructive DELETE-then-INSERT: a resync
|
||||
// running in the wrong direction WIPED a live buffer, so the code needed a
|
||||
// same-oldest-Up predicate to decide who had authority. The divergence case was a
|
||||
// rolling restart leaving the delivering node oldest but not leader.
|
||||
//
|
||||
// Under LocalDb that failure mode is structurally impossible rather than guarded
|
||||
// against: snapshot resync merges per row under LWW and never deletes. So this is
|
||||
// deliberately NOT a directional-authority test — there is no active/standby
|
||||
// asymmetry left to enforce. It asserts the property the guard was protecting: no
|
||||
// node loses rows to a peer's snapshot.
|
||||
//
|
||||
// The setup is the wipe scenario made concrete. Each node holds a row the other has
|
||||
// never seen, and they disagree about a third; then they resync.
|
||||
await DeployConfigAsync(A, "shared-instance", """{"v":1}""", "dep-1");
|
||||
await WaitUntilAsync(
|
||||
async () => await ReadConfigAsync(B, "shared-instance") is not null,
|
||||
"the shared row to exist on both nodes before they diverge");
|
||||
|
||||
await StopPassiveAsync();
|
||||
|
||||
// A keeps working while B is down: a brand-new buffered message and a newer version
|
||||
// of the shared config.
|
||||
await EnqueueAsync(A, "live-row-on-a");
|
||||
await DeployConfigAsync(A, "shared-instance", """{"v":2}""", "dep-2");
|
||||
|
||||
// B is offline but its database is still writable — it also has local state the
|
||||
// snapshot exchange must not destroy.
|
||||
await EnqueueAsync(B, "live-row-on-b");
|
||||
|
||||
await RestartPairAsync();
|
||||
|
||||
await WaitUntilAsync(
|
||||
async () =>
|
||||
await MessageExistsAsync(A, "live-row-on-a") &&
|
||||
await MessageExistsAsync(A, "live-row-on-b") &&
|
||||
await MessageExistsAsync(B, "live-row-on-a") &&
|
||||
await MessageExistsAsync(B, "live-row-on-b"),
|
||||
"both nodes to hold the UNION of the rows each wrote while partitioned");
|
||||
|
||||
// And the contended row converges to the newer write rather than either node's
|
||||
// snapshot flattening the other.
|
||||
await WaitUntilAsync(
|
||||
async () => await ReadConfigAsync(A, "shared-instance") == """{"v":2}"""
|
||||
&& await ReadConfigAsync(B, "shared-instance") == """{"v":2}""",
|
||||
"both nodes to converge on the newer config for the contended instance");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ConfigDeployedOnA_ConvergesToB_WithoutAnyFetch()
|
||||
{
|
||||
// Was: SiteReplicationActorTests #1-#5 (notify-and-fetch — the standby is told a
|
||||
// deploy happened, then HTTP-fetches the config itself, with retries and a
|
||||
// superseded-404 path). Under CDC the config row simply replicates, so that whole
|
||||
// exchange is deleted at Task 15.
|
||||
//
|
||||
// The plan called for asserting a fetcher test double records zero invocations. That
|
||||
// would be THEATRE here and is deliberately not done: this harness has no actor
|
||||
// system, no central, and no IDeploymentConfigFetcher in the graph at all, so a
|
||||
// double would record zero calls whether or not the fetch path still existed. An
|
||||
// assertion that cannot fail is worse than none.
|
||||
//
|
||||
// What this test honestly proves is the positive half: the config reaches node B
|
||||
// through replication ALONE, in a process where fetching is not merely unused but
|
||||
// absent. The negative half — that no fetch path survives — is proved by Task 15
|
||||
// deleting the code and the build still passing, which is a stronger check than any
|
||||
// runtime counter.
|
||||
//
|
||||
// Scope note (D1): SiteReconciliationActor survives Phase 2 and legitimately fetches
|
||||
// over HTTP at node startup when central reports gaps. "The standby never fetches,
|
||||
// ever" would be a FALSE claim; the claim is about the deploy path only.
|
||||
await DeployConfigAsync(A, "inst-deploy", """{"setpoint":42}""", "dep-100");
|
||||
|
||||
await WaitUntilAsync(
|
||||
async () => await ReadConfigAsync(B, "inst-deploy") == """{"setpoint":42}""",
|
||||
"the deployed config to reach node B over replication alone");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task RedeployingAnInstance_ConvergesToTheNewRevision_OnBothNodes()
|
||||
{
|
||||
// The steady-state deploy loop, and the reason the old path needed a deployed_at
|
||||
// guard: a standby that applied a STALE fetch would pin an instance to superseded
|
||||
// config until the next deploy. Under LWW the newer write wins on the primary key,
|
||||
// so the guard is not what protects this — but the outcome still has to hold.
|
||||
await DeployConfigAsync(A, "inst-redeploy", """{"rev":"a"}""", "dep-a");
|
||||
await WaitUntilAsync(
|
||||
async () => await ReadConfigAsync(B, "inst-redeploy") == """{"rev":"a"}""",
|
||||
"the first revision to reach node B");
|
||||
|
||||
await DeployConfigAsync(A, "inst-redeploy", """{"rev":"b"}""", "dep-b");
|
||||
|
||||
await WaitUntilAsync(
|
||||
async () => await ReadConfigAsync(A, "inst-redeploy") == """{"rev":"b"}"""
|
||||
&& await ReadConfigAsync(B, "inst-redeploy") == """{"rev":"b"}""",
|
||||
"both nodes to converge on the redeployed revision");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ConfigDeployedOnB_ConvergesToA()
|
||||
{
|
||||
// Deliberately the reverse direction. The bespoke replicator was asymmetric by
|
||||
// design — the active node pushed, the standby fetched — so "which node deployed it"
|
||||
// was a meaningful question. Under CDC it is not, and this test is what says so.
|
||||
// After a failover the surviving node must be able to deploy without waiting to be
|
||||
// promoted to some authority role.
|
||||
await DeployConfigAsync(B, "inst-from-b", """{"origin":"b"}""", "dep-b1");
|
||||
|
||||
await WaitUntilAsync(
|
||||
async () => await ReadConfigAsync(A, "inst-from-b") == """{"origin":"b"}""",
|
||||
"a config deployed on node B to reach node A");
|
||||
}
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user