dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 2 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Compare commits

merge into: dohertj2/ScadaBridge:feat/localdb-phase1
Branches Tags
dohertj2/ScadaBridge:main dohertj2/ScadaBridge:feat/grpc-sitecommand dohertj2/ScadaBridge:feat/grpc-central-control dohertj2/ScadaBridge:feat/grpc-phase0-psk dohertj2/ScadaBridge:fix/cached-telemetry-drain-hot-loop dohertj2/ScadaBridge:feat/localdb-phase2 dohertj2/ScadaBridge:feat/localdb-phase1 dohertj2/ScadaBridge:feat/secrets-0.2.0
...
pull from: dohertj2/ScadaBridge:feat/localdb-phase2
Branches Tags
dohertj2/ScadaBridge:main dohertj2/ScadaBridge:feat/grpc-sitecommand dohertj2/ScadaBridge:feat/grpc-central-control dohertj2/ScadaBridge:feat/grpc-phase0-psk dohertj2/ScadaBridge:fix/cached-telemetry-drain-hot-loop dohertj2/ScadaBridge:feat/localdb-phase2 dohertj2/ScadaBridge:feat/localdb-phase1 dohertj2/ScadaBridge:feat/secrets-0.2.0

35 Commits
feat/localdb-phase1 ... feat/localdb-phase2

Author SHA1 Message Date
Joseph Doherty 7621b48925 docs(known-issues): cached-telemetry drain hot-loops on a missing tracking snapshot 
An audit row whose tracking snapshot cannot be resolved is skipped and left
Pending, on the reasoning that central reconciliation will pick it up. Nothing
removes it from the local drain queue, so the next tick re-reads it, fails
identically, and logs again — forever. Measured on the rig at ~2,800
warnings/minute, sustained across both a process restart and a container
restart, until the audit database itself was discarded.

The code comment already names the cause ("the tracking store was reset"), so
this is an anticipated input, not an exotic one. Two production triggers need
no operator error: the tracking retention window elapsing before the drain
catches up (long central outage, large backlog), or restoring one store
independently of the other. The two stores are easy to desync because they live
in different places — audit in auditlog.db, which on the docker rig is INSIDE
the container, and tracking in the bind-mounted LocalDb database.

Skipping the row is right; leaving it Pending with no other state change is
not. There is no attempt counter, no backoff, no terminal state, and one
Warning per row per pass. Suggested fixes ranked by effort, the cheapest being
to rate-limit the warning the way MaintenanceBackgroundService already does for
oplog caps.

No data loss — the audit rows are intact and still reach central by the
reconciliation path. Found while cleaning the rig after the Phase 2 live gate.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 05:59:35 -04:00
Joseph Doherty 7b5a5a6f34 docs(localdb): phase 2 truth pass across both repos 
Normative first: Component-StoreAndForward.md:83 specified the whole chunked,
ack-confirmed SfBufferSnapshotChunk resync protocol, which Phase 2 deleted. It
is rewritten rather than removed — the failure modes it reasoned about still
exist, they are just bounded differently — and it now states the
duplicate-delivery bound explicitly: a message can be delivered twice only when
the OLD primary delivered it and the status change had not yet replicated when
the gate flipped. One flush interval plus the in-flight ack, and unlike the old
model it does NOT grow with backlog depth or with how long a node was absent.
The N1 directional-authority and N5 orphan-row hazards are recorded as
structurally gone, not merely unguarded.

Frame-size known-issue amended: its 2026-06-26 resolution replaced the
intra-site hop with notify-and-fetch; Phase 2 then deleted notify-and-fetch
itself, so the 128 KB Akka frame constraint no longer applies to that hop in
any form. Successor ceiling recorded (4 MB gRPC cap via MaxBatchSize, which
batches by ROW COUNT), including that the failure mode differs — an oversized
gRPC message is rejected, not silently dropped.

Deployment docs gain the two operational constraints that have no home in code:
a site pair must be stopped and started TOGETHER (the SfBufferSnapshot compat
handler that made a mixed-version pair converge went with the replicator, and a
mixed pair now diverges silently), and a node offline beyond TombstoneRetention
can resurrect deleted rows on rejoin.

Both CLAUDE.md files corrected — each still said Phase 2 was NOT started.

Definition of done closed: build 0 warnings, all 10 suites green (3509 tests,
0 failures, 0 skips). Two DoD items needed amending rather than ticking: the
stale-symbol grep still matches 4 lines, all deliberate comment prose recording
what was deleted (a literal zero would delete the explanations that stop the
old design coming back), and the live gate has 10 evidence items, not the 9 the
checklist claimed.

Deletes the phase2 resume-state scratch doc, which said to delete it on landing.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 05:37:48 -04:00
Joseph Doherty 158e79bb50 docs(localdb): phase 2 live gate evidence — all 10 checks pass 
The blocker in the previous (incomplete) run was diagnosed: external systems
reach a site only through ArtifactDeploymentService, which `instance deploy`
never invokes. `deploy artifacts` delivered the probe harness AND propagated
the owed ExternalSystemDefinitions restore to both site nodes, unblocking
checks 5 and 10.

Highlights:
- 3+4: the deployed row is byte-identical on both nodes with the SAME
  __localdb_row_version HLC and origin node id, and the standby logged zero
  config fetches in the deploy window. B holds the row A wrote, by CDC alone.
- 6: stopped the active node; the standby took over in 10s and kept buffering,
  oplog rose to 4 unacked while partitioned, drained to 0 on rejoin, and both
  nodes ended byte-identical with ZERO duplicate ids (exactly-once).
- 7: the 3-table cascade converged with explicit TOMBSTONES on both nodes —
  the rows are not merely absent on B, B applied the deletes.
- 9: both nodes stopped and started together; clean rejoin, zero SQLite I/O
  errors.

Two caveats recorded rather than glossed: check 2's zero-count is vacuous on
its own (the legacy source was also empty) and rests instead on the ABSENCE of
CDC triggers for smtp_configurations/notification_lists; check 7's
native_alarm_state leg was empty live and is covered only offline.

Also records three method corrections — the plan's host-side sqlite3
instruction is unsafe, `docker exec curl` cannot scrape metrics from an image
with no curl (a silent failure that nearly became a false finding), and the
artifact-vs-instance deploy distinction above.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 05:20:54 -04:00
Joseph Doherty 3c87b11bcf docs(localdb): phase 2 live gate evidence (INCOMPLETE — 5 of 10) 
Checks 1, 2, 3, 4 and the oplog/dead-letter/metrics half of 8 captured and
passing. Check 5 blocked on rig state, checks 6, 7, 9, 10 not run. Task 21 must
NOT proceed on this evidence.

Strongest result is check 3+4 together: the deployed config row is byte-
identical on both site-a nodes with the SAME __localdb_row_version HLC and
originating node id, and the standby logged zero config fetches in the deploy
window. The standby holds the row node A wrote, obtained purely by CDC — which
is the whole point of deleting notify-and-fetch.

Check 5 is blocked by a rig-shaping problem, not a product fault: removing the
owed soakgen instances orphaned their 11,804 buffered sf_messages, and a
replacement probe's external system never reached the site nodes (site config
tables come from deployment artifacts, and an external system referenced only
by name inside script code is not carried). Same blocker stops check 10.

Two method corrections recorded in the doc:
- The plan says to run DB checks host-side against the bind mounts. That is
  WRONG — a host sqlite3 open poisons the container's WAL. All reads here copy
  the db/-wal/-shm triplet and query the copy.
- An apparent "localdb_* metrics missing" finding was an artifact of the
  aspnet:10.0 image having no curl, with the error swallowed by 2>/dev/null.
  Re-scraped via a network-sharing sidecar: the metrics are present.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 05:10:43 -04:00
Joseph Doherty 166f07fa68 chore(localdb): adopt LocalDb 0.1.1 and drop the directory shim 
LocalDb 0.1.1 creates the parent directory of LocalDb:Path itself, so the
SiteLocalDbDirectory shim this repo carried through Phase 2 is deleted along
with its call site. The gap was found here but was never ScadaBridge's alone —
every LocalDb consumer had it — so the fix moved to the library.

SiteLocalDbDirectoryTests is RETAINED and retargeted rather than deleted with
the shim. It was already written against the site registration path, not the
mechanism, so it needed only its Ensure() call removed: what a site node
requires is that resolving ILocalDb not fail on a fresh machine, regardless of
who provides that. Verified it still earns its place — pinned back to LocalDb
0.1.0 it fails inside SqliteLocalDb..ctor -> SqliteConnection.Open(), so it
genuinely depends on the library behaviour and not on a coincidence.

Also corrects the coverage-split note in StoreAndForwardStorageTests, which
asserted that directory creation is "NOT LocalDb's" — true when written, wrong
as of 0.1.1.

Build 0 warnings; Host 330, StoreAndForward 130, LocalDb integration 20 pass.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 04:58:20 -04:00
Joseph Doherty 9ec7966dac docs(localdb): task state for tasks 17-19 
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 04:42:16 -04:00
Joseph Doherty 921edab454 chore(docker): size the site-a oplog caps from the phase 2 soak 
MaxBatchSize 500 -> 16. The default is a ROW count, not a byte budget, so the
batch size in bytes is set by the widest replicated column — config_json, which
Task 1 measured at up to ~60-70 KB in production. 70 KB x 500 is ~35 MB against
gRPC's 4 MB default receive limit; 16 keeps a worst-case batch near 1.1 MB.

MaxOplogRows 1,000,000 -> 250,000 and MaxOplogAge 7d -> 2d, sized from the
soak's 0.80 sf_messages rows/sec (~69k rows/day). Tighter than default is
correct here because exceeding a cap is not data loss: the oplog prunes to the
ceiling and sets needs_snapshot, so the peer catches up by snapshot resync
instead of incrementally. That trades a rare full resync for a bounded file.

site-b and site-c stay unreplicated, so the default-OFF posture is still proven
side by side on one rig.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 04:41:47 -04:00
Joseph Doherty 15013156bf test(localdb): two-node convergence for config tables + sf_messages 
Four scenarios over the real site pair, driven through the REAL
SiteStorageService rather than hand-written SQL. The sibling suites had to
hand-write SQL because they were specifications written BEFORE the cutover;
this suite runs after it, so it can drive the shipped writers — which is what
makes the cascade scenario meaningful.

- DeployedConfigRow_ConvergesToB_ColumnForColumn: the existing suite asserts
  config_json only, so a capture that dropped or defaulted any other column
  would still pass. deployed_at matters most — SiteReconciliationActor's
  guarded write compares it.
- RemovingAnInstance_ConvergesAllThreeCascadeTables: the plan's flagged
  highest-risk case. RemoveDeployedConfigAsync deletes from three tables in one
  transaction, the schema has no foreign keys, and CDC ships three independent
  per-table streams. A dropped delete leaves a permanently stale override or
  alarm row on the standby, invisible until the instance name is redeployed.
  Carries a never-removed control instance, without which "the cascade
  converged" is indistinguishable from "node B lost these tables entirely".
- ANativeAlarmBurst_Converges_AndTheOplogDrains: convergence alone would pass
  if entries replicated but were never acked, and an oplog that only grows
  trips the caps into a snapshot resync.
- RowsWrittenOnBWhileItsListenerIsDown_SurviveTheRejoin: the union-survives
  property is per-table, and an unregistered table is silently local-only
  rather than an error, so it is re-proved on the tables the N1 scenario does
  not touch.

Non-vacuity verified as the plan requires: with the eight Phase 2
RegisterReplicated calls commented out in SiteLocalDbSetup, all four go red
(4 failed / 0 passed); restored, 20/20 pass across the three LocalDb suites.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 04:41:01 -04:00
Joseph Doherty 605e56829e chore(config): retire ReplicationEnabled, make legacy db paths migration-only 
LocalDb Phase 2 deleted the bespoke replicators, so three config keys changed
meaning or died outright:

- ScadaBridge:StoreAndForward:ReplicationEnabled is fully dead. Deleted the
  property, its 10 config entries, and the 5 test references.
- SqliteDbPath / SiteDbPath are now migration-only: they name the legacy files
  SiteLocalDbLegacyMigrator drains at boot, not live databases. Both mandatory
  rules are relaxed accordingly (StartupValidator's Site-only Require, and the
  S&F validator's non-empty rule) — an absent value now means "nothing to
  migrate", so an already-migrated node can drop the key. DatabaseOptions-
  Validator still rejects a present-but-blank value.
- SiteRuntime:ConfigFetchRetryCount's only reader was SiteReplicationActor.
  Deleted with its validator rule.

The two path keys stay present in every config, now with a comment explaining
why: removing them would strand un-migrated data on a node that has not yet
started once.

Both relaxations are pinned by the inverse of the test they replace
(Site_MissingSiteDbPath_IsAccepted..., EmptySqliteDbPath_IsAccepted...), each
verified to fail with the old rule restored.

Note: deploy/wonder-app-vd03/appsettings.Site.json is under a gitignored
deploy/ tree, so its edit is local-only and must be repeated on the box.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 04:35:11 -04:00
Joseph Doherty 3364145d63 docs(localdb): resume state through the cutover (tasks 1-16 done, task 17 next) 
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 04:22:30 -04:00
Joseph Doherty 0ad11d6b55 docs(localdb): task state for the 14/15/16 cutover 
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 04:20:17 -04:00
Joseph Doherty 037798b367 feat(localdb)!: replicate site config + sf_messages via CDC, delete the bespoke replicators 
Tasks 14, 15 and 16, landed as ONE commit.

PLAN DEFECT: these three tasks cannot compile separately. SiteReplicationActor
takes a ReplicationService and calls ReplaceAllAsync (Task 14 deletes both);
DeploymentManagerActor Tells message types declared in ReplicationMessages.cs
(Task 15 deletes it); AkkaHostedService constructs the actor (Task 16). Any
ordering leaves a broken intermediate. Combining them also strengthens the
invariant Task 14 already stated for itself — the two mechanisms never both
run, and never neither.

Registered 8 tables in SiteLocalDbSetup.OnReady: sf_messages plus the 7 site
config tables. notification_lists and smtp_configurations are deliberately NOT
registered — permanently empty by design, so registering them would open a
standing replication channel whose only historical payload was plaintext SMTP
passwords. Migrate stays the LAST call in OnReady, after all registrations, so
migrated rows enter the oplog through live capture triggers.

Deleted: SiteReplicationActor, ReplicationMessages.cs, ReplicationService,
StoreAndForwardStorage.ReplaceAllAsync, and 6 test files. ReplaceAllAsync is
not merely unused but unsafe to keep: a mass DELETE on a now-replicated table
would be captured and shipped to the peer.

Kept ActiveNodeEvaluator (delivery gate + heartbeat still need it) with its doc
corrected, and activeNodeCheck in AkkaHostedService (SiteCommunicationActor).

The positional-argument hazard the plan flagged was real: removing
DeploymentManagerActor's optional IActorRef? replicationActor shifted 6
trailing optionals, and 4 test call sites bound the wrong arguments with no
compile error at some positions. Converted them to named arguments where
possible — Props.Create builds an expression tree, which rejects out-of-position
named args, so the rest are padded positionally with a comment saying why.

The Task 7 'not yet registered' test was INVERTED rather than deleted, and is
exact in both directions: too few means a table silently stops replicating, too
many means the SMTP tables leak. Added a separate security-named test for those
two, and a composite-PK test (LWW keys on the full PK, so a truncated key set
would collapse distinct rows). The convergence suites now get their
registrations from the real OnReady — their temporary harness registration is
deleted, so they prove the cutover rather than agreeing with themselves.

Verified: build 0 warnings; SiteRuntime 512, StoreAndForward 130, Host 330,
AuditLog 355, ExternalSystemGateway 142, HealthMonitoring 97, LocalDb
integration 16 — all pass, 0 failures.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 04:20:05 -04:00
Joseph Doherty df0c6031ba docs(localdb): task state for tasks 10-13 + deviations 
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 04:03:48 -04:00
Joseph Doherty 79ce51612e test(site): pin the active-node notification-config purge; scope the guarded write 
Task 12 + Task 13. No production behaviour change in either.

Task 12: DeploymentManagerActor.HandleDeployArtifacts already purges
notification_lists and smtp_configurations on every artifact apply, but nothing
pinned the actor's CALL to it — ArtifactStorageTests covers the storage method
only. Task 15 deletes SiteReplicationActor's copy, making this the sole
remaining call site, and Task 16 edits this actor's wiring; dropping the call
would leave plaintext SMTP passwords on disk with every suite still green.
Verified red-first by commenting the call out: the pin fails with that message.

Task 13: StoreDeployedConfigIfNewerAsync STAYS. Re-verified both callers —
SiteReplicationActor:375 (dies at Task 15) and SiteReconciliationActor:166
(survives). Reconciliation is a per-node startup self-heal against central
whose fetch races real deploys, so the deployed_at guard still does real work
there. Doc comment rewritten to say so, and to warn against porting the guard
onto the replication path where it would fight the HLC rather than help it.
Also corrected a stale 'guarded standby write' section header in the tests.

Re-ran the Task 13 step 2 scope check: ConfigFetchRetryCount's only production
reader remains SiteReplicationActor:157, so its option + validator rule stay
until Task 17, after Task 15 deletes the actor.

Verified: build 0 warnings; SiteRuntime 533, Host 329, StoreAndForward 153,
LocalDb integration 16 — all pass.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 04:03:00 -04:00
Joseph Doherty c56bf4ae65 test(localdb): port resync + config replication intents as CDC convergence specs 
Task 11. Companion to Task 10, covering the intents held by
SiteReplicationActorTests and SfBufferResyncPredicateTests.

N1 Critical is re-expressed, not ported literally (D2). SfBufferResyncPredicate
existed because ReplaceAllAsync was a destructive DELETE-then-INSERT, so a
wrong-direction resync wiped a live buffer and the code needed an oldest-Up
predicate to decide authority. LocalDb's snapshot resync merges per row under
LWW and never deletes, so this asserts the property the guard protected — no
node loses rows to a peer's snapshot — with no directional-authority assertion,
because there is no active/standby asymmetry left to enforce. The setup is the
wipe scenario made concrete: each node writes rows the other never sees while
partitioned, plus a contended row, then they resync.

DEVIATION on the zero-fetch assertion. The plan asked for a fetcher test double
recording zero invocations; this harness has no actor system, no central and no
IDeploymentConfigFetcher in the graph, so the double would record zero calls
whether or not the fetch path still existed. An assertion that cannot fail is
worse than none, so the test proves the positive half — config reaches B over
replication alone — and the file records that the negative half is proved by
Task 15 deleting the code and the build still passing. It also records the D1
scope note: SiteReconciliationActor survives and legitimately fetches at
startup, so 'the standby never fetches, ever' would be false.

Non-vacuity verified by unregistering deployed_configurations: all 4 fail.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 03:54:42 -04:00
Joseph Doherty 2bbe66311d test(localdb): port store-and-forward replication intents as CDC convergence specs 
Task 10. Specifications first, deletion second: the bespoke ReplicationService
(explicit Add/Remove/Park/Requeue over Akka) dies at Task 14, so its behaviour
is restated here as outcomes the CDC replacement must still deliver. Written
in terms of ROWS, not operations — under CDC there is no Add or Park message to
observe, only a row that must end up right on both nodes.

Not ported: ReplicationOperations_AreDispatchedInIssueOrder. It asserts the
mechanism (inline fire-and-forget dispatch), and CDC capture is asynchronous
and batched by construction. Its portable content is the ordering OUTCOME —
add-then-remove must never converge to present — which is a test here, with
that reasoning recorded in the file so it does not read as an accidental drop.

DEVIATION: extracted the Phase 1 fixture into LocalDbSitePairHarness rather
than duplicating ~150 lines. Phase 1's tests now derive from it and still pass
unchanged. The harness registers the Phase 2 tables itself, since production
OnReady does not until Task 14; that method is marked for deletion at the
cutover, and the 8-table list is written literally so a cutover registering the
wrong set fails these tests instead of agreeing with itself.

Non-vacuity verified by unregistering sf_messages: 6 of 7 failed. The 7th —
the ordering test — PASSED, because an absent row is also what a pair that
replicates nothing looks like. Fixed with a control row that must converge in
the same window, so the absence is evidence rather than silence.

Also corrected two comments from Task 9 that claimed Task 14 makes
notification_lists/smtp_configurations replicated. It explicitly does not
register them, for the same reason the migrator skips them.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 03:50:48 -04:00
Joseph Doherty 0dbfefba62 docs(localdb): task state for tasks 7-9 + deviations 
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 03:40:59 -04:00
Joseph Doherty 5ddc7eed6d feat(localdb): migrate legacy scadabridge.db config tables into the consolidated DB 
Task 9. Seven of the nine site configuration tables are copied out of the
legacy scadabridge.db in a single transaction, with one rename at the end: a
partial config migration would leave a site node running against half its old
configuration, which is worse than failing startup outright.

notification_lists and smtp_configurations are deliberately NOT migrated.
Both are purged on every deploy and permanently empty by design since the
site write paths were removed (2026-07-10), but a pre-fix legacy file can
still hold rows — and smtp_configurations.password is plaintext. Task 14
makes these tables replicated, so migrating them would push plaintext SMTP
passwords across a channel whose only historical payload was exactly that.
The tables are still created; only their historical contents stay behind.

Generalizes Task 8's MigrateTable into MigrateFile(many tables, one
transaction, one rename), keeping the legacy/current column intersection.

Five tests, including a column-parity test asserting each declared column
list equals the live SiteStorageSchema's. That mismatch is otherwise
invisible: a typo'd column is silently dropped by the intersection, and a
missing one silently leaves data behind. Non-vacuity verified twice — once by
removing the Migrate call (3 fail), once by wrongly adding notification_lists
and smtp_configurations to the table map (the skip test fails).

Verified: solution build 0 warnings; Host 329, SiteRuntime 532,
StoreAndForward 153 — all pass.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 03:40:31 -04:00
Joseph Doherty bdc0dffea2 feat(localdb): migrate legacy store-and-forward.db into the consolidated DB 
Task 8. Adds ResolveStoreAndForwardPath + MigrateStoreAndForward, wired as a
third call in Migrate alongside the two Phase 1 files.

Unlike the Phase 1 paths, the store-and-forward default sits INSIDE the data
volume, so a real deployment has a real file here holding undelivered
messages. This migration genuinely moves data; losing it would discard exactly
the buffered calls store-and-forward exists to protect.

No id synthesis: sf_messages.id is already a caller-assigned TEXT primary key,
so INSERT OR IGNORE is idempotent across a crash-then-rerun.

The copy intersects the legacy column set with the current one rather than
naming all 16 columns outright. A file from an older build predates
execution_id / parent_execution_id / last_attempt_at_ms, and naming a missing
column throws 'no such column' — which the existing reader treats as an
unrecognised shape and silently discards every row. A required-column (PK)
guard keeps that tolerance from degrading into copying NULL-keyed rows.

Four tests: copy+rename, crash-before-rename idempotence, the __localdb_oplog
assertion that catches migrate-before-register, and the older-column-set case.
All four verified to fail without the Migrate call.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 03:32:10 -04:00
Joseph Doherty f8aa02e2a9 feat(localdb): create config + sf_messages tables in the consolidated DB 
Task 7. SiteLocalDbSetup.OnReady now applies SiteStorageSchema and
StoreAndForwardSchema alongside the Phase 1 schemas, so the nine site
configuration tables and the store-and-forward buffer live in the
consolidated LocalDb file.

Deliberately NOT registered for replication. The bespoke SiteReplicationActor
and the StoreAndForward ReplicationService still own these tables until the
Task 14 cutover deletes both and registers them in one commit; registering
early would run two replicators over the same rows and let either one's
defects hide behind the other's writes.

Pinned by two tests through the real composition root: one asserting all
twelve tables exist, one asserting ReplicatedTables is EXACTLY the Phase 1
pair. Non-vacuity verified by removing the DDL and observing the failure.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 03:28:05 -04:00
Joseph Doherty fefbbb31da docs(localdb): resume state through wave 2 (tasks 1-6 done, wave 3 next) 
Records the latent Phase 1 directory defect and its open library-vs-app decision,
the CreateConnection contract change, the new TestSupport library, and the
verification numbers at the wave-2 boundary.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 03:14:18 -04:00
Joseph Doherty 19ab0ac913 chore(localdb): record tasks 5-6 completion, the latent directory defect, and the CreateConnection contract change 
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 03:06:06 -04:00
Joseph Doherty f2efeb37b7 refactor(sf,site): both stores take ILocalDb instead of a connection string 
Tasks 5 and 6 of the Phase 2 plan, committed together because their test
fallout is entangled — several fixtures construct both stores.

StoreAndForwardStorage and SiteStorageService now take ILocalDb. Connections
come from ILocalDb.CreateConnection(), which hands out an already-open,
pragma-configured connection carrying the zb_hlc_next() UDF the capture triggers
call; a raw connection would lack the UDF and every write to a replicated table
would fail closed. Deleted with the connection strings: S&F's
EnsureDatabaseDirectoryExists and its per-open busy_timeout pragma, and the site
service's BusyTimeoutFloorSeconds normalization — LocalDb owns all of it now.

DI: AddSiteRuntime's string overload is gone (nothing left to supply), so the
Host calls the no-arg form. ScadaBridge:Database:SiteDbPath and
StoreAndForwardOptions.SqliteDbPath survive only as the migrator's source
locations in Tasks 8/9.

Two things the plan did not anticipate, both worth reading:

1. FOUND A REAL LATENT DEFECT, from Phase 1, now fixed. The plan assumed
   directory creation simply moved to LocalDb along with file ownership. It did
   not: the LocalDb library never creates the parent directory, and
   SqliteLocalDb opens the file eagerly in its constructor — so a missing
   directory is a hard boot failure ("SQLite Error 14: unable to open database
   file"), not a degraded start. The default site config points at the RELATIVE
   path ./data/site-localdb.db, so any site node without a pre-existing data/
   directory fails to boot. The docker rig escapes only because its volume mount
   happens to create /app/data — a coincidence that would have hidden this until
   a bare-metal or fresh deployment. This has been latent since Phase 1 made
   LocalDb:Path required; deleting S&F's EnsureDatabaseDirectoryExists here
   would have widened it. Re-established the guarantee at the layer that now
   owns the path (SiteLocalDbDirectory.Ensure, called before AddZbLocalDb) and
   pinned it with SiteLocalDbDirectoryTests. Non-vacuity is not assumed: two
   tests written against the wrong assumption failed with exactly this
   SQLite Error 14 before the fix existed.

2. Test fallout was ~7x the plan's estimate. The plan named "fixtures" in one
   project; the constructor change actually reaches 40 files across 7 test
   projects, and most used Mode=Memory;Cache=Shared — which LocalDb has no
   equivalent for, so every one had to move to a real temp file. Rather than
   copy the Phase 1 TestLocalDb fixture into 7 projects, added a shared
   tests/ZB.MOM.WW.ScadaBridge.TestSupport library (not a test project) so the
   WAL-sidecar cleanup and the "real, not stubbed" rationale live in one place.

Retargeted rather than deleted, in both directions: the S&F WAL test now asserts
against the LocalDb-backed store (WAL genuinely is LocalDb's job), while the
directory-creation test moved to Host.Tests (that guarantee is NOT LocalDb's).
SiteStorageServiceTests.Initialize_EnablesWalJournalMode got the same treatment.
DeploymentManagerMediumFindingsTests induced a persistence failure via an
unopenable path, which no longer reaches the assertion since the fixture now
throws first; it induces the same failure shape via an uninitialized store.

Verified: full solution build 0 warnings; SiteRuntime 532, Host 318,
AuditLog 355, ExternalSystemGateway 142, HealthMonitoring 97,
StoreAndForward 153 — 1597 passed, 0 failed.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 03:05:45 -04:00
Joseph Doherty 3dfb288b74 chore(localdb): record tasks 3-4 completion and their deviations in the task state 
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 02:21:09 -04:00
Joseph Doherty ac5eb12cce refactor(site): extract SiteStorageSchema.Apply from SiteStorageService 
Task 4 of the Phase 2 plan. All nine table definitions plus MigrateSchemaAsync
and TryAddColumnAsync move into a static sync Apply(SqliteConnection) depending
only on Microsoft.Data.Sqlite, so the Host can apply the DDL to a LocalDb-managed
connection before RegisterReplicated installs the capture triggers.

Per the plan, PRAGMA journal_mode=WAL stays in InitializeAsync — LocalDb owns
the connection's pragmas, and the service still opens its own connections until
Task 6.

One deviation: TryAddColumnAsync's `catch (SqliteException) when
(ex.Message.Contains("duplicate column"))` becomes a PRAGMA table_info probe,
matching OperationTrackingSchema. The message-matching form depends on an error
string that is not part of SQLite's contract, and it swallowed every
SqliteException whose message happened to contain that substring. The probe also
drops the ILogger dependency, which is what let the class become static. Cost:
the per-column "Migrated: added column" info log is gone — it fired once per
column per legacy database and nothing consumes it.

Also added a test beyond the plan's specified one, for the same reason as Task 3:
the specified test asserts against a freshly-created database, where CREATE TABLE
already lists the migration columns, so it would pass with every ALTER deleted.
The added test starts from the pre-migration shapes with a row present and proves
the migration path runs and preserves data across a NOT NULL DEFAULT add.

SiteRuntime suite 532 passed / 0 failed; full solution build 0 warnings.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 02:20:50 -04:00
Joseph Doherty 9e3239c5d9 refactor(sf): extract StoreAndForwardSchema.Apply from the storage class 
Task 3 of the Phase 2 plan. Mirrors OperationTrackingSchema: the sf_messages
DDL, the four additive ALTERs, the last_attempt_at_ms backfill and the due-index
move verbatim into a static sync Apply(SqliteConnection), depending only on
Microsoft.Data.Sqlite. The Host needs to apply this to a LocalDb-managed
connection before RegisterReplicated installs the capture triggers; the store
still calls it so a directly-constructed store stays self-sufficient.

Two deviations from the plan as written, both deliberate:

1. The PRAGMA journal_mode=WAL in InitializeAsync STAYS. The plan's Step 4
   snippet drops it, but Task 4 explicitly says not to move the equivalent
   pragma ("LocalDb owns the connection's pragmas") — the two tasks contradict
   each other. Keeping it preserves behaviour for the intermediate commits and
   for directly-constructed stores; it becomes moot in Task 5 when the store
   stops opening its own connections. Dropping it now would quietly regress the
   concurrent-writer support the pragma's own comment documents.

2. Added a second test beyond the plan's. The specified test asserts only
   against a freshly-created table, where CREATE TABLE already lists all 16
   columns — it would pass with every ALTER deleted. The added test starts from
   the pre-upgrade 12-column shape with a row in it and proves the upgrade path
   runs and preserves data.

That second test also surfaced that the backfill lands 1 ms low: julianday()'s
double day-fraction cannot represent every millisecond. Pre-existing behaviour,
carried over verbatim, asserted with a 1 ms tolerance rather than pinning a
precision the implementation never had.

Suite: 154 passed, 0 failed.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 02:18:05 -04:00
Joseph Doherty 12eb97e07a docs(localdb): close the phase 2 gate — questions answered, plan written 
Task 2 of the Phase 2 plan, plus the clean re-run of Task 1's cut-short
sampling that Task 2 was waiting on.

Gate doc: status NOT STARTED -> CLOSED. All five §5 open questions answered
inline. The questions are kept, not deleted — several of the answers overturn
a premise the gate itself stated, and that reasoning is the value:

- Oplog sizing is not the binding constraint. Cap overrun prunes and flags
  needs_snapshot (graceful resync), so the caps need no defensive sizing; the
  real ceiling is D6's 4 MB gRPC cap, binding on MaxBatchSize x row-bytes.
- LWW-vs-in-flight-send cannot arise on a correct pair (only the active node
  sweeps). The honest cost is D2's semantic change: the standby is convergent,
  no longer byte-identical.
- Migration-under-load is designed out, not managed — both mechanisms are
  deleted in the same commit and D5 forecloses rolling upgrades.
- Rollback is genuinely "revert the commit": the migration is additive and
  leaves the legacy files intact. Bounded cost is the post-cutover delta.
- One DB per process stays; every write axis is bounded.

§2's requirement to state CDC's duplicate-delivery bound is routed explicitly
to Task 21 rather than left implicit.

Task 1 re-run (safe copy-based snap(), both nodes restarted first, six 60s
intervals): sf_messages 0.80 rows/sec insert, 0/sec retry-UPDATE, oplog 0,
alarms 0, max payload_json 76 B, zero SQLite errors across 30 min, both site-a
nodes converged at 3564 rows. Independently confirms the disk-I/O storm was
observer-induced. Recorded with its limits stated: 0.80/s is ~1.6% of the 50/s
ceiling and the retry path was never exercised — acceptable only because that
ceiling is structural rather than empirical, and the rig's 721 B config rows
are explicitly NOT representative.

GATE VERDICT: PROCEED to Task 3. Binding output is MaxBatchSize 500 -> 16.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 02:15:53 -04:00
Joseph Doherty d512572dac docs(localdb): phase 2 resume state — where the plan stands and what carries forward 
Scratch handoff so the plan can be picked back up cleanly: Task 1 done and its
STOP verdict superseded, what must happen before Task 3, the four plan-premise
corrections, the rig gotchas that are not discoverable from the repo, and the
rig cleanup the Task 20 live gate needs.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 02:01:42 -04:00
Joseph Doherty 8652eab98e fix(localdb): root-cause the soak disk-I/O failure + ship the hardening follow-ups 
The Phase 2 soak's "LocalDb fails under load" blocker is NOT a product defect.
Root cause: host-side (macOS) sqlite3 reads of the live, bind-mounted WAL
databases. POSIX advisory locks do not propagate across the virtiofs boundary,
so the host reader believes it is the only connection, checkpoints on close and
resets the WAL to 0 bytes under the container. The container's still-mapped
WAL index then references frames that no longer exist and every subsequent
statement fails SQLITE_IOERR_SHORT_READ (522) -> primary code 10, permanently
until the process reopens the database.

Reproduced on demand both on the rig (one sqlite3 SELECT reset a 4.6 MiB WAL and
produced the first error one second later) and in a minimal python:3.12-alpine
repro with no LocalDb or .NET involved. Refuted the converse: a freshly-reopened
node sustained the full soak load 10+ minutes with zero errors.

The original brief's isolation was confounded - both nodes had been poisoned by
the same sampling pass, and a poisoned standby looks healthy only because it
issues almost no statements. Corrections annotated in place.

Hardening shipped alongside:
- SqliteErrorCodes.Describe: log SQLite primary AND extended codes at the
  LocalDb-adjacent catch sites (the missing extended code is what made the
  original diagnosis so slow).
- SiteAuditTelemetryActor: stop touching ActorContext across an await
  (NotSupportedException), with regression coverage.
- infra/reseed.sh: apply the MSSQL init scripts explicitly. The official
  mssql/server image does not implement /docker-entrypoint-initdb.d, so a fresh
  volume hung the reseed forever; compose mounts annotated as informational.

Deliberately NOT done: detect-and-reopen self-heal in SqliteLocalDb. It defends
only against external interference, which is now prevented at the source, and
same-kernel production readers see the locks correctly.

Build 0 warnings; SiteAuditTelemetryActorTests 9/9, SiteEventLogging 70/70.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 02:01:00 -04:00
Joseph Doherty e9e11d635e docs(known-issues): root-cause brief for the LocalDb disk-I/O-error-under-load defect 
Standalone handoff for a follow-up investigation. Records the symptom, a full
reproduction (including the two rig-tooling blockers and the CachedCall-vs-Call
detail needed to generate load at all), the isolation evidence, a code map of
the LocalDb connection model, four ranked hypotheses, and acceptance criteria.

Not root-caused. Highest-value next step identified: capture the SQLite EXTENDED
result code (logs only carry the generic primary code 10), and run the load with
LocalDb:Path off the bind mount to partition environmental vs library causes.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 01:11:08 -04:00
Joseph Doherty 82c869a1df docs(localdb): record the phase 2 task-1 gate verdict in the task record 
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 01:07:18 -04:00
Joseph Doherty fc553bd9ba docs(localdb): phase 2 rig soak findings — GATE FAILS on a phase 1 defect 
Task 1 of the Phase 2 plan. The gate stops the plan, but not for the reason it
anticipated: oplog sizing is fine and D6 is resolved.

BLOCKER: the Phase 1 consolidated LocalDb (site-localdb.db) throws SQLite
Error 10 'disk I/O error' on essentially every write on the ACTIVE node under
sustained load — site_events, OperationTracking and the audit telemetry paths
all fail, and site event logging silently drops events. Isolated to the load
(not the node, not host-side observation) by failing over between nodes, and to
LocalDb specifically (legacy store-and-forward.db / scadabridge.db in the same
bind-mounted directory take zero errors under identical load).

Phase 2 would register 8 more tables into that database — including the two
highest-volume ones — while deleting the bespoke mechanisms that currently
carry them. Must be root-caused first.

Also recorded: D6's premise corrected (largest known production config_json is
~60-70 KB, not >128 KB — but MaxBatchSize 500 is still unsafe, use 16); D4's
premise corrected (alarm writes are bounded by per-SourceReference coalescing at
a 100 ms flush); sf_messages has a hard 50 rows/sec structural ceiling; and
exceeding the oplog caps is a graceful snapshot-resync, not a failure.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-20 01:06:59 -04:00
Joseph Doherty cf46e59680 fix(rig): seed LDAP group mappings with the canonical role names 
docker/seed-sites.sh inserted the pre-rename 'Design' / 'Deployment' role
strings, but the canonical vocabulary in Roles.cs is 'Designer' / 'Deployer'.
The mismatch authorized nothing: on a freshly reseeded rig every
Designer/Deployer-gated management command failed UNAUTHORIZED, including
seed-sites.sh's own trailing `deploy artifacts` and reseed.sh's stage 6d
encrypted-secret restore.

Found while standing up the Phase 2 soak rig.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-19 17:55:04 -04:00
Joseph Doherty 25463d522f docs(localdb): phase 2 plan review pass — corrections from code verification 
Three verification sweeps over SiteRuntime, StoreAndForward, and Host/rig/docs
plus the LocalDb library source. Load-bearing corrections: D1 (the guarded write
has a second surviving caller, SiteReconciliationActor), D3 (the active node
already purges — Task 12 becomes a pin), D6 (new: the 4 MB gRPC cap vs
row-count-only batching), Task 1 (rewritten method — the Phase 2 tables are not
in the Phase 1 oplog), and Task 14 (do not register the notification/SMTP tables).

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-19 17:27:11 -04:00
Joseph Doherty f6ca82a9e2 docs(localdb): phase 2 implementation plan (21 tasks, full scope) 
Moves scadabridge.db's 9 config tables + sf_messages into the consolidated
LocalDb file and deletes SiteReplicationActor + StoreAndForward's
ReplicationService.

Resolves the phase 2 gate's five open questions from code recon (D1-D5):

D1 notify-and-fetch is DELETED, not preserved. It exists only because the
   config blob exceeds Akka's 128KB frame; LocalDb sync is gRPC. The
   deployed_at version guard protected against a stale fetch racing, so it
   dies with the fetch. Do not reproduce it on LWW - different clocks.
D2 ReplaceAllAsync is deleted and the N1 directional guard becomes
   unnecessary: LocalDb's snapshot resync merges per-row LWW and never
   deletes (SnapshotApplier has no DELETE; LwwApplier.cs:69-78 discards a
   lower-HLC incoming row). Semantic change - the standby is convergent,
   no longer byte-identical.
D3 The SMTP purge (plaintext passwords) rides the replication path and is
   re-homed to the active node BEFORE any deletion.
D4 native_alarm_state volume is measured by a rig soak, not assumed. Task 1
   gates the plan and stops it if the oplog cannot absorb the churn.
D5 No dual-mechanism period forecloses rolling site upgrades - both nodes
   must stop and start together.

Recon also found the gate doc's "two test files are the spec" undercounts:
the real specification is five files, including the N1 Critical regression
test and the only Requeue coverage.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
 2026-07-19 15:04:37 -04:00
120 changed files with 6423 additions and 3742 deletions
Expand all files Collapse all files
CLAUDE.md
+10 -3
View File
@@ -120,12 +120,19 @@ Other peers in the `scadaproj` family (see `scadaproj/CLAUDE.md` for details): `
- DCL write failures returned synchronously to calling script. - DCL write failures returned synchronously to calling script.
- Tag path resolution retried periodically for devices still booting. - Tag path resolution retried periodically for devices still booting.
- Static attribute writes persisted to local SQLite (survive restart/failover, reset on redeployment). - Static attribute writes persisted to local SQLite (survive restart/failover, reset on redeployment).
- **Consolidated site database (LocalDb Phase 1, 2026-07-19).** `OperationTracking` and `site_events` now live in ONE `ZB.MOM.WW.LocalDb`-managed SQLite file, configured by the **required** `LocalDb:Path` (`/app/data/site-localdb.db` on the rig; validated with `ValidateOnStart`, so a site config missing it fails to boot). Both are `RegisterReplicated` tables. Consequences worth knowing: - **Consolidated site database (LocalDb Phase 1 + 2, complete 2026-07-20).** **Ten** tables now live in ONE `ZB.MOM.WW.LocalDb`-managed SQLite file — the Phase 1 pair (`OperationTracking`, `site_events`) plus Phase 2's `sf_messages` and the seven site config tables (`deployed_configurations`, `static_attribute_overrides`, `shared_scripts`, `external_systems`, `database_connections`, `data_connection_definitions`, `native_alarm_state`), configured by the **required** `LocalDb:Path` (`/app/data/site-localdb.db` on the rig; validated with `ValidateOnStart`, so a site config missing it fails to boot). Both are `RegisterReplicated` tables. Consequences worth knowing:
- `site_events.id` changed from autoincrement INTEGER to an application-minted **GUID**. Last-writer-wins keys on the primary key, so two nodes independently minting `id=1,2,3…` would destroy each other's events rather than merge them. The event-log read path uses a composite `(timestamp, id)` keyset cursor with an **opaque string** continuation token; `EventLogEntry.Id` and both `ContinuationToken`s are `string`/`string?` on the site↔central Akka DTOs. - `site_events.id` changed from autoincrement INTEGER to an application-minted **GUID**. Last-writer-wins keys on the primary key, so two nodes independently minting `id=1,2,3…` would destroy each other's events rather than merge them. The event-log read path uses a composite `(timestamp, id)` keyset cursor with an **opaque string** continuation token; `EventLogEntry.Id` and both `ContinuationToken`s are `string`/`string?` on the site↔central Akka DTOs.
- `ScadaBridge:OperationTracking:ConnectionString` and `ScadaBridge:SiteEventLog:DatabasePath` are **migration-only** — nothing reads them but `SiteLocalDbLegacyMigrator`, which copies a pre-Phase-1 file in once (deterministic `mig-{NodeName}-{legacyId}` event ids, `INSERT OR IGNORE`, runs AFTER `RegisterReplicated` so migrated rows replicate) and renames it `.migrated`. Delete the keys once a node has migrated. - `ScadaBridge:OperationTracking:ConnectionString`, `ScadaBridge:SiteEventLog:DatabasePath` and — as of Phase 2 — `ScadaBridge:StoreAndForward:SqliteDbPath` + `ScadaBridge:Database:SiteDbPath` are all **migration-only** — nothing reads them but `SiteLocalDbLegacyMigrator`, which copies a pre-Phase-1 file in once (deterministic `mig-{NodeName}-{legacyId}` event ids, `INSERT OR IGNORE`, runs AFTER `RegisterReplicated` so migrated rows replicate) and renames it `.migrated`. Delete the keys once a node has migrated.
- This incidentally fixes a data-loss bug: both legacy databases defaulted to CWD-relative paths **outside** the mounted volume and were discarded on every container recreate. - This incidentally fixes a data-loss bug: both legacy databases defaulted to CWD-relative paths **outside** the mounted volume and were discarded on every container recreate.
- **Replication is default-OFF and opt-in** via `LocalDb:Replication:PeerAddress` + a matching `ApiKey` on both nodes. `LocalDbSyncAuthInterceptor` is **fail-closed**: no configured key means no sync stream is accepted at all, so a key typo does not degrade to unauthenticated replication — the pair simply stops converging. The sync endpoint shares the existing site gRPC h2c listener (8083); no new port. Rig posture: **site-a replicated, site-b/site-c deliberately not**, so both states are proven side-by-side. Status surfaces on the site health report as `LocalDbReplicationConnected` / `LocalDbOplogBacklog` (both nullable — null means "no data", NOT "disconnected with an empty backlog") and as `localdb_*` Prometheus series. Note `ZbTelemetryOptions.Meters` is an **allowlist** (`SiteServiceRegistration.ObservedMeters`); an unlisted meter exports nothing, silently. - **Replication is default-OFF and opt-in** via `LocalDb:Replication:PeerAddress` + a matching `ApiKey` on both nodes. `LocalDbSyncAuthInterceptor` is **fail-closed**: no configured key means no sync stream is accepted at all, so a key typo does not degrade to unauthenticated replication — the pair simply stops converging. The sync endpoint shares the existing site gRPC h2c listener (8083); no new port. Rig posture: **site-a replicated, site-b/site-c deliberately not**, so both states are proven side-by-side. Status surfaces on the site health report as `LocalDbReplicationConnected` / `LocalDbOplogBacklog` (both nullable — null means "no data", NOT "disconnected with an empty backlog") and as `localdb_*` Prometheus series. Note `ZbTelemetryOptions.Meters` is an **allowlist** (`SiteServiceRegistration.ObservedMeters`); an unlisted meter exports nothing, silently.
- Design: **scadaproj** `docs/plans/2026-07-19-scadabridge-localdb-design.md` (that doc lives in the umbrella repo, not here); Phase 1 plan + Phase 2 gate are here under `docs/plans/`. Phase 2 (config tables + `sf_messages`, deleting `SiteReplicationActor` + StoreAndForward `ReplicationService`) is NOT started — see `docs/plans/2026-07-19-localdb-phase2-gate.md`. - Design: **scadaproj** `docs/plans/2026-07-19-scadabridge-localdb-design.md` (that doc lives in the umbrella repo, not here); Phase 1 plan + Phase 2 gate are here under `docs/plans/`. **Phase 2 is COMPLETE** (branch `feat/localdb-phase2`, live gate PASS 2026-07-20 — all 10 checks, evidence in `docs/plans/2026-07-19-localdb-phase2-live-gate.md`). It moved the config tables + `sf_messages` in and **deleted** `SiteReplicationActor`, its `ReplicationMessages`, StoreAndForward's `ReplicationService`, and `StoreAndForwardStorage.ReplaceAllAsync`. What those did, and why nothing replaced them:
- `SiteReplicationActor` pushed config deploys to the peer and ran a **notify-and-fetch** exchange (tell the standby a deploy happened; it HTTP-fetches the config itself, with retries and a superseded-404 path). Config rows now simply replicate — **the standby makes no fetch at all** during a deploy (verified live). `SiteReconciliationActor` still fetches at node STARTUP when central reports gaps; that path survives and is a different thing.
- `ReplicationService` fanned each buffer mutation (add/remove/park/requeue) to the standby by hand. CDC triggers on `sf_messages` do it now.
- `ReplaceAllAsync` was a destructive delete-all-then-insert-all resync. It was not merely unused after the cutover but **unsafe to keep**: a mass DELETE on a now-replicated table would be captured and shipped to the peer. LocalDb's snapshot resync merges per row under LWW and never deletes, which is also why the old N1 directional-authority guard is gone — there is no wipe left to gate.
- **`notification_lists` and `smtp_configurations` are created but deliberately NOT registered.** They are permanently empty on a site (no writer since 2026-07-10, the migrator skips them, the active-node purge keeps them empty), and registering them would open a standing replication channel whose only historical payload was plaintext SMTP passwords. Pinned by a security-named test, and verified live: those two tables have **no CDC triggers** on either rig node.
- **Operational constraints (read before upgrading a site pair):** stop and start both nodes TOGETHER — rolling one at a time is no longer supported, since the legacy `SfBufferSnapshot` compatibility handler went with the replicator. And a node offline longer than `LocalDb:Replication:TombstoneRetention` (default 7 days) can resurrect deleted rows on rejoin. See `docs/deployment/topology-guide.md`.
- `LocalDb:Replication:MaxBatchSize` batches by ROW COUNT, not bytes, against a 4 MB gRPC cap — the rig pins it to **16** (~70 KB worst-case `config_json` x 16 ~= 1.1 MB). The 500 default would allow ~35 MB.
- All timestamps are UTC throughout the system. - All timestamps are UTC throughout the system.
- Inter-cluster communication uses two transports: ClusterClient for command/control (deployments, lifecycle, subscribe/unsubscribe handshake, snapshots) and gRPC server-streaming for real-time data (attribute values, alarm states). Both CentralCommunicationActor and SiteCommunicationActor registered with receptionist. Central creates one ClusterClient per site using NodeA/NodeB as contact points. Sites configure multiple central contact points for failover. Addresses cached in CentralCommunicationActor, refreshed periodically (60s) and on admin changes. Heartbeats serve health monitoring only. - Inter-cluster communication uses two transports: ClusterClient for command/control (deployments, lifecycle, subscribe/unsubscribe handshake, snapshots) and gRPC server-streaming for real-time data (attribute values, alarm states). Both CentralCommunicationActor and SiteCommunicationActor registered with receptionist. Central creates one ClusterClient per site using NodeA/NodeB as contact points. Sites configure multiple central contact points for failover. Addresses cached in CentralCommunicationActor, refreshed periodically (60s) and on admin changes. Heartbeats serve health monitoring only.
- gRPC streaming channel: SiteStreamGrpcServer on each site node (Kestrel HTTP/2, port 8083); central creates per-site SiteStreamGrpcClient via SiteStreamGrpcClientFactory. Site entity has GrpcNodeAAddress/GrpcNodeBAddress fields. Proto: sitestream.proto with SiteStreamService, SiteStreamEvent (oneof: AttributeValueUpdate, AlarmStateUpdate). DebugStreamEvent message removed (no longer flows through ClusterClient). - gRPC streaming channel: SiteStreamGrpcServer on each site node (Kestrel HTTP/2, port 8083); central creates per-site SiteStreamGrpcClient via SiteStreamGrpcClientFactory. Site entity has GrpcNodeAAddress/GrpcNodeBAddress fields. Proto: sitestream.proto with SiteStreamService, SiteStreamEvent (oneof: AttributeValueUpdate, AlarmStateUpdate). DebugStreamEvent message removed (no longer flows through ClusterClient).
Directory.Packages.props
+3 -3
View File
@@ -108,9 +108,9 @@
<PackageVersion Include="ZB.MOM.WW.Secrets.Abstractions" Version="0.2.3" /> <PackageVersion Include="ZB.MOM.WW.Secrets.Abstractions" Version="0.2.3" />
<PackageVersion Include="ZB.MOM.WW.Secrets.Ui" Version="0.2.3" /> <PackageVersion Include="ZB.MOM.WW.Secrets.Ui" Version="0.2.3" />
<PackageVersion Include="ZB.MOM.WW.Secrets.Replicator.SqlServer" Version="0.2.3" /> <PackageVersion Include="ZB.MOM.WW.Secrets.Replicator.SqlServer" Version="0.2.3" />
<PackageVersion Include="ZB.MOM.WW.LocalDb" Version="0.1.0" /> <PackageVersion Include="ZB.MOM.WW.LocalDb" Version="0.1.1" />
<PackageVersion Include="ZB.MOM.WW.LocalDb.Replication" Version="0.1.0" /> <PackageVersion Include="ZB.MOM.WW.LocalDb.Replication" Version="0.1.1" />
<PackageVersion Include="ZB.MOM.WW.LocalDb.Contracts" Version="0.1.0" /> <PackageVersion Include="ZB.MOM.WW.LocalDb.Contracts" Version="0.1.1" />
</ItemGroup> </ItemGroup>
<!-- <!--
ZB.MOM.WW.ScadaBridge.slnx
+1
View File
@@ -36,6 +36,7 @@
<Project Path="tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.csproj" /> <Project Path="tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.csproj" />
<Project Path="tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests.csproj" /> <Project Path="tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests.csproj" />
<Project Path="tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/ZB.MOM.WW.ScadaBridge.Communication.Tests.csproj" /> <Project Path="tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/ZB.MOM.WW.ScadaBridge.Communication.Tests.csproj" />
<Project Path="tests/ZB.MOM.WW.ScadaBridge.TestSupport/ZB.MOM.WW.ScadaBridge.TestSupport.csproj" />
<Project Path="tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests.csproj" /> <Project Path="tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests.csproj" />
<Project Path="tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests.csproj" /> <Project Path="tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests.csproj" />
<Project Path="tests/ZB.MOM.WW.ScadaBridge.NotificationService.Tests/ZB.MOM.WW.ScadaBridge.NotificationService.Tests.csproj" /> <Project Path="tests/ZB.MOM.WW.ScadaBridge.NotificationService.Tests/ZB.MOM.WW.ScadaBridge.NotificationService.Tests.csproj" />
docker-env2/site-x-node-a/appsettings.Site.json
+9 -2
View File
@@ -20,6 +20,10 @@
"MinNrOfMembers": 1 "MinNrOfMembers": 1
}, },
"Database": { "Database": {
// Migration-only as of LocalDb Phase 2. The site config tables now live in the
// consolidated LocalDb database (LocalDb:Path). SiteDbPath is read once at boot to drain
// a pre-Phase-2 scadabridge.db, and is unused after that - keep it until this node has
// started once.
"SiteDbPath": "/app/data/scadabridge.db" "SiteDbPath": "/app/data/scadabridge.db"
}, },
"DataConnection": { "DataConnection": {
@@ -29,8 +33,11 @@
"SeedReadTimeout": "00:00:30" "SeedReadTimeout": "00:00:30"
}, },
"StoreAndForward": { "StoreAndForward": {
"SqliteDbPath": "/app/data/store-and-forward.db", // Migration-only as of LocalDb Phase 2. The store-and-forward buffer now lives in the
"ReplicationEnabled": true // consolidated LocalDb database (LocalDb:Path) as the replicated sf_messages table.
// SqliteDbPath is read once at boot by SiteLocalDbLegacyMigrator to drain a pre-Phase-2
// file, and is unused after that - keep it until this node has started once.
"SqliteDbPath": "/app/data/store-and-forward.db"
}, },
"Communication": { "Communication": {
"CentralContactPoints": [ "CentralContactPoints": [
docker-env2/site-x-node-b/appsettings.Site.json
+9 -2
View File
@@ -20,6 +20,10 @@
"MinNrOfMembers": 1 "MinNrOfMembers": 1
}, },
"Database": { "Database": {
// Migration-only as of LocalDb Phase 2. The site config tables now live in the
// consolidated LocalDb database (LocalDb:Path). SiteDbPath is read once at boot to drain
// a pre-Phase-2 scadabridge.db, and is unused after that - keep it until this node has
// started once.
"SiteDbPath": "/app/data/scadabridge.db" "SiteDbPath": "/app/data/scadabridge.db"
}, },
"DataConnection": { "DataConnection": {
@@ -29,8 +33,11 @@
"SeedReadTimeout": "00:00:30" "SeedReadTimeout": "00:00:30"
}, },
"StoreAndForward": { "StoreAndForward": {
"SqliteDbPath": "/app/data/store-and-forward.db", // Migration-only as of LocalDb Phase 2. The store-and-forward buffer now lives in the
"ReplicationEnabled": true // consolidated LocalDb database (LocalDb:Path) as the replicated sf_messages table.
// SqliteDbPath is read once at boot by SiteLocalDbLegacyMigrator to drain a pre-Phase-2
// file, and is unused after that - keep it until this node has started once.
"SqliteDbPath": "/app/data/store-and-forward.db"
}, },
"Communication": { "Communication": {
"CentralContactPoints": [ "CentralContactPoints": [
docker/seed-sites.sh
+9 -4
View File
@@ -96,7 +96,7 @@ for ident in site-a site-b site-c; do
done done
echo "" echo ""
echo "Seeding LDAP group mappings (Design + Deployment)..." echo "Seeding LDAP group mappings (Designer + Deployer)..."
# SecurityConfiguration.HasData declares 4 mappings but the InitialSchema # SecurityConfiguration.HasData declares 4 mappings but the InitialSchema
# migration only inserts the Admin row, so a fresh ScadaBridgeConfig starts # migration only inserts the Admin row, so a fresh ScadaBridgeConfig starts
# with multi-role getting Admin only -- no Design and no Deployment access. # with multi-role getting Admin only -- no Design and no Deployment access.
@@ -106,11 +106,16 @@ docker exec -i scadabridge-mssql /opt/mssql-tools18/bin/sqlcmd \
-d ScadaBridgeConfig -Q " -d ScadaBridgeConfig -Q "
SET IDENTITY_INSERT LdapGroupMappings ON; SET IDENTITY_INSERT LdapGroupMappings ON;
IF NOT EXISTS (SELECT 1 FROM LdapGroupMappings WHERE Id = 2) IF NOT EXISTS (SELECT 1 FROM LdapGroupMappings WHERE Id = 2)
INSERT INTO LdapGroupMappings (Id, LdapGroupName, Role) VALUES (2, 'SCADA-Designers', 'Design'); INSERT INTO LdapGroupMappings (Id, LdapGroupName, Role) VALUES (2, 'SCADA-Designers', 'Designer');
IF NOT EXISTS (SELECT 1 FROM LdapGroupMappings WHERE Id = 3) IF NOT EXISTS (SELECT 1 FROM LdapGroupMappings WHERE Id = 3)
INSERT INTO LdapGroupMappings (Id, LdapGroupName, Role) VALUES (3, 'SCADA-Deploy-All', 'Deployment'); INSERT INTO LdapGroupMappings (Id, LdapGroupName, Role) VALUES (3, 'SCADA-Deploy-All', 'Deployer');
IF NOT EXISTS (SELECT 1 FROM LdapGroupMappings WHERE Id = 4) IF NOT EXISTS (SELECT 1 FROM LdapGroupMappings WHERE Id = 4)
INSERT INTO LdapGroupMappings (Id, LdapGroupName, Role) VALUES (4, 'SCADA-Deploy-SiteA', 'Deployment'); INSERT INTO LdapGroupMappings (Id, LdapGroupName, Role) VALUES (4, 'SCADA-Deploy-SiteA', 'Deployer');
-- Role strings MUST match the canonical vocabulary in
-- src/ZB.MOM.WW.ScadaBridge.Security/Roles.cs ('Designer' / 'Deployer').
-- These rows previously carried the pre-rename 'Design' / 'Deployment', which
-- authorized nothing: every Designer/Deployer-gated management command failed
-- UNAUTHORIZED on a freshly reseeded rig.
SET IDENTITY_INSERT LdapGroupMappings OFF; SET IDENTITY_INSERT LdapGroupMappings OFF;
" "
docker/site-a-node-a/appsettings.Site.json
+27 -3
View File
@@ -21,6 +21,10 @@
"MinNrOfMembers": 1 "MinNrOfMembers": 1
}, },
"Database": { "Database": {
// Migration-only as of LocalDb Phase 2. The site config tables now live in the
// consolidated LocalDb database (LocalDb:Path). SiteDbPath is read once at boot to drain
// a pre-Phase-2 scadabridge.db, and is unused after that - keep it until this node has
// started once.
"SiteDbPath": "/app/data/scadabridge.db" "SiteDbPath": "/app/data/scadabridge.db"
}, },
"DataConnection": { "DataConnection": {
@@ -30,8 +34,11 @@
"SeedReadTimeout": "00:00:30" "SeedReadTimeout": "00:00:30"
}, },
"StoreAndForward": { "StoreAndForward": {
"SqliteDbPath": "/app/data/store-and-forward.db", // Migration-only as of LocalDb Phase 2. The store-and-forward buffer now lives in the
"ReplicationEnabled": true // consolidated LocalDb database (LocalDb:Path) as the replicated sf_messages table.
// SqliteDbPath is read once at boot by SiteLocalDbLegacyMigrator to drain a pre-Phase-2
// file, and is unused after that - keep it until this node has started once.
"SqliteDbPath": "/app/data/store-and-forward.db"
}, },
"Communication": { "Communication": {
"CentralContactPoints": [ "CentralContactPoints": [
@@ -80,7 +87,24 @@
// pre-host secret expander. // pre-host secret expander.
"Replication": { "Replication": {
"PeerAddress": "http://scadabridge-site-a-b:8083", "PeerAddress": "http://scadabridge-site-a-b:8083",
"ApiKey": "dev-site-a-localdb-sync-key" "ApiKey": "dev-site-a-localdb-sync-key",
// ---- Phase 2 sizing, from the Task 1 rig soak (not from the defaults) ----
//
// MaxBatchSize (default 500) is a ROW count, not a byte budget, so the batch
// size in bytes is set by the widest replicated column. That is
// deployed_configurations.config_json: ~721 B on this rig, but up to ~60-70 KB
// in production (measured, Task 1) - and 70 KB x 500 is ~35 MB against gRPC's
// 4 MB default receive limit. 16 keeps a worst-case batch near 1.1 MB.
"MaxBatchSize": 16,
// Backlog caps bound the oplog while the peer is offline. Exceeding them is
// NOT data loss: the oplog is pruned to the ceiling and needs_snapshot is set,
// so the peer catches up by snapshot resync instead of incrementally. That
// makes tighter-than-default correct here - it trades a rare full resync for a
// bounded file. Sized from the soak's 0.80 sf_messages rows/sec (the only
// non-zero writer measured): ~69k rows/day, so 2 days is ~138k. 250,000 leaves
// room for burst without approaching the 1,000,000 default.
"MaxOplogRows": 250000,
"MaxOplogAge": "2.00:00:00"
} }
} }
} }
docker/site-a-node-b/appsettings.Site.json
+27 -3
View File
@@ -21,6 +21,10 @@
"MinNrOfMembers": 1 "MinNrOfMembers": 1
}, },
"Database": { "Database": {
// Migration-only as of LocalDb Phase 2. The site config tables now live in the
// consolidated LocalDb database (LocalDb:Path). SiteDbPath is read once at boot to drain
// a pre-Phase-2 scadabridge.db, and is unused after that - keep it until this node has
// started once.
"SiteDbPath": "/app/data/scadabridge.db" "SiteDbPath": "/app/data/scadabridge.db"
}, },
"DataConnection": { "DataConnection": {
@@ -30,8 +34,11 @@
"SeedReadTimeout": "00:00:30" "SeedReadTimeout": "00:00:30"
}, },
"StoreAndForward": { "StoreAndForward": {
"SqliteDbPath": "/app/data/store-and-forward.db", // Migration-only as of LocalDb Phase 2. The store-and-forward buffer now lives in the
"ReplicationEnabled": true // consolidated LocalDb database (LocalDb:Path) as the replicated sf_messages table.
// SqliteDbPath is read once at boot by SiteLocalDbLegacyMigrator to drain a pre-Phase-2
// file, and is unused after that - keep it until this node has started once.
"SqliteDbPath": "/app/data/store-and-forward.db"
}, },
"Communication": { "Communication": {
"CentralContactPoints": [ "CentralContactPoints": [
@@ -73,7 +80,24 @@
// fail-closed, so a typo here does not degrade to unauthenticated replication; // fail-closed, so a typo here does not degrade to unauthenticated replication;
// it rejects every stream and the pair silently stops converging. // it rejects every stream and the pair silently stops converging.
"Replication": { "Replication": {
"ApiKey": "dev-site-a-localdb-sync-key" "ApiKey": "dev-site-a-localdb-sync-key",
// ---- Phase 2 sizing, from the Task 1 rig soak (not from the defaults) ----
//
// MaxBatchSize (default 500) is a ROW count, not a byte budget, so the batch
// size in bytes is set by the widest replicated column. That is
// deployed_configurations.config_json: ~721 B on this rig, but up to ~60-70 KB
// in production (measured, Task 1) - and 70 KB x 500 is ~35 MB against gRPC's
// 4 MB default receive limit. 16 keeps a worst-case batch near 1.1 MB.
"MaxBatchSize": 16,
// Backlog caps bound the oplog while the peer is offline. Exceeding them is
// NOT data loss: the oplog is pruned to the ceiling and needs_snapshot is set,
// so the peer catches up by snapshot resync instead of incrementally. That
// makes tighter-than-default correct here - it trades a rare full resync for a
// bounded file. Sized from the soak's 0.80 sf_messages rows/sec (the only
// non-zero writer measured): ~69k rows/day, so 2 days is ~138k. 250,000 leaves
// room for burst without approaching the 1,000,000 default.
"MaxOplogRows": 250000,
"MaxOplogAge": "2.00:00:00"
} }
} }
} }
docker/site-b-node-a/appsettings.Site.json
+9 -2
View File
@@ -21,6 +21,10 @@
"MinNrOfMembers": 1 "MinNrOfMembers": 1
}, },
"Database": { "Database": {
// Migration-only as of LocalDb Phase 2. The site config tables now live in the
// consolidated LocalDb database (LocalDb:Path). SiteDbPath is read once at boot to drain
// a pre-Phase-2 scadabridge.db, and is unused after that - keep it until this node has
// started once.
"SiteDbPath": "/app/data/scadabridge.db" "SiteDbPath": "/app/data/scadabridge.db"
}, },
"DataConnection": { "DataConnection": {
@@ -30,8 +34,11 @@
"SeedReadTimeout": "00:00:30" "SeedReadTimeout": "00:00:30"
}, },
"StoreAndForward": { "StoreAndForward": {
"SqliteDbPath": "/app/data/store-and-forward.db", // Migration-only as of LocalDb Phase 2. The store-and-forward buffer now lives in the
"ReplicationEnabled": true // consolidated LocalDb database (LocalDb:Path) as the replicated sf_messages table.
// SqliteDbPath is read once at boot by SiteLocalDbLegacyMigrator to drain a pre-Phase-2
// file, and is unused after that - keep it until this node has started once.
"SqliteDbPath": "/app/data/store-and-forward.db"
}, },
"Communication": { "Communication": {
"CentralContactPoints": [ "CentralContactPoints": [
docker/site-b-node-b/appsettings.Site.json
+9 -2
View File
@@ -21,6 +21,10 @@
"MinNrOfMembers": 1 "MinNrOfMembers": 1
}, },
"Database": { "Database": {
// Migration-only as of LocalDb Phase 2. The site config tables now live in the
// consolidated LocalDb database (LocalDb:Path). SiteDbPath is read once at boot to drain
// a pre-Phase-2 scadabridge.db, and is unused after that - keep it until this node has
// started once.
"SiteDbPath": "/app/data/scadabridge.db" "SiteDbPath": "/app/data/scadabridge.db"
}, },
"DataConnection": { "DataConnection": {
@@ -30,8 +34,11 @@
"SeedReadTimeout": "00:00:30" "SeedReadTimeout": "00:00:30"
}, },
"StoreAndForward": { "StoreAndForward": {
"SqliteDbPath": "/app/data/store-and-forward.db", // Migration-only as of LocalDb Phase 2. The store-and-forward buffer now lives in the
"ReplicationEnabled": true // consolidated LocalDb database (LocalDb:Path) as the replicated sf_messages table.
// SqliteDbPath is read once at boot by SiteLocalDbLegacyMigrator to drain a pre-Phase-2
// file, and is unused after that - keep it until this node has started once.
"SqliteDbPath": "/app/data/store-and-forward.db"
}, },
"Communication": { "Communication": {
"CentralContactPoints": [ "CentralContactPoints": [
docker/site-c-node-a/appsettings.Site.json
+9 -2
View File
@@ -21,6 +21,10 @@
"MinNrOfMembers": 1 "MinNrOfMembers": 1
}, },
"Database": { "Database": {
// Migration-only as of LocalDb Phase 2. The site config tables now live in the
// consolidated LocalDb database (LocalDb:Path). SiteDbPath is read once at boot to drain
// a pre-Phase-2 scadabridge.db, and is unused after that - keep it until this node has
// started once.
"SiteDbPath": "/app/data/scadabridge.db" "SiteDbPath": "/app/data/scadabridge.db"
}, },
"DataConnection": { "DataConnection": {
@@ -30,8 +34,11 @@
"SeedReadTimeout": "00:00:30" "SeedReadTimeout": "00:00:30"
}, },
"StoreAndForward": { "StoreAndForward": {
"SqliteDbPath": "/app/data/store-and-forward.db", // Migration-only as of LocalDb Phase 2. The store-and-forward buffer now lives in the
"ReplicationEnabled": true // consolidated LocalDb database (LocalDb:Path) as the replicated sf_messages table.
// SqliteDbPath is read once at boot by SiteLocalDbLegacyMigrator to drain a pre-Phase-2
// file, and is unused after that - keep it until this node has started once.
"SqliteDbPath": "/app/data/store-and-forward.db"
}, },
"Communication": { "Communication": {
"CentralContactPoints": [ "CentralContactPoints": [
docker/site-c-node-b/appsettings.Site.json
+9 -2
View File
@@ -21,6 +21,10 @@
"MinNrOfMembers": 1 "MinNrOfMembers": 1
}, },
"Database": { "Database": {
// Migration-only as of LocalDb Phase 2. The site config tables now live in the
// consolidated LocalDb database (LocalDb:Path). SiteDbPath is read once at boot to drain
// a pre-Phase-2 scadabridge.db, and is unused after that - keep it until this node has
// started once.
"SiteDbPath": "/app/data/scadabridge.db" "SiteDbPath": "/app/data/scadabridge.db"
}, },
"DataConnection": { "DataConnection": {
@@ -30,8 +34,11 @@
"SeedReadTimeout": "00:00:30" "SeedReadTimeout": "00:00:30"
}, },
"StoreAndForward": { "StoreAndForward": {
"SqliteDbPath": "/app/data/store-and-forward.db", // Migration-only as of LocalDb Phase 2. The store-and-forward buffer now lives in the
"ReplicationEnabled": true // consolidated LocalDb database (LocalDb:Path) as the replicated sf_messages table.
// SqliteDbPath is read once at boot by SiteLocalDbLegacyMigrator to drain a pre-Phase-2
// file, and is unused after that - keep it until this node has started once.
"SqliteDbPath": "/app/data/store-and-forward.db"
}, },
"Communication": { "Communication": {
"CentralContactPoints": [ "CentralContactPoints": [
docs/components/Host.md
-1
View File
@@ -104,7 +104,6 @@ Before branching on role, `AkkaHostedService.StartAsync` creates one actor uncon
`SiteServiceRegistration.Configure` registers the site-only components. `AkkaHostedService.RegisterSiteActorsAsync` creates: `SiteServiceRegistration.Configure` registers the site-only components. `AkkaHostedService.RegisterSiteActorsAsync` creates:
- `DeploymentManagerActor` — cluster singleton scoped to `"site-{SiteId}"`. - `DeploymentManagerActor` — cluster singleton scoped to `"site-{SiteId}"`.
- `SiteCommunicationActor` — registered with `ClusterClientReceptionist`; creates a `ClusterClient` to configured central contact points. - `SiteCommunicationActor` — registered with `ClusterClientReceptionist`; creates a `ClusterClient` to configured central contact points.
- `SiteReplicationActor` — one per node (not a singleton); handles best-effort S&F replication to the standby.
- `EventLogHandlerActor` — cluster singleton scoped to `"site-{SiteId}"`. - `EventLogHandlerActor` — cluster singleton scoped to `"site-{SiteId}"`.
- `ParkedMessageHandlerActor` — bridges Akka to `StoreAndForwardService`. - `ParkedMessageHandlerActor` — bridges Akka to `StoreAndForwardService`.
- `SiteAuditTelemetryActor` — created on a dedicated `audit-telemetry-dispatcher` (2-thread `ForkJoinDispatcher`) so SQLite reads and gRPC pushes never contend with hot-path actors. - `SiteAuditTelemetryActor` — created on a dedicated `audit-telemetry-dispatcher` (2-thread `ForkJoinDispatcher`) so SQLite reads and gRPC pushes never contend with hot-path actors.
docs/components/SiteRuntime.md
+3 -3
View File
@@ -8,7 +8,7 @@ Site Runtime (#3) operates exclusively on site clusters. Its entry point is the
The component code lives in `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/`: The component code lives in `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/`:
- `Actors/` — `DeploymentManagerActor`, `InstanceActor`, `ScriptActor`, `ScriptExecutionActor`, `AlarmActor`, `AlarmExecutionActor`, `NativeAlarmActor`, `SiteReplicationActor`. - `Actors/` — `DeploymentManagerActor`, `InstanceActor`, `ScriptActor`, `ScriptExecutionActor`, `AlarmActor`, `AlarmExecutionActor`, `NativeAlarmActor`.
- `Scripts/` — `ScriptCompilationService`, `ScriptExecutionScheduler`, `SharedScriptLibrary`, `ScriptRuntimeContext`, `ScopeAccessors`, `TriggerExpressionGlobals`. - `Scripts/` — `ScriptCompilationService`, `ScriptExecutionScheduler`, `SharedScriptLibrary`, `ScriptRuntimeContext`, `ScopeAccessors`, `TriggerExpressionGlobals`.
- `Streaming/` — `SiteStreamManager` (the site-wide Akka broadcast stream). - `Streaming/` — `SiteStreamManager` (the site-wide Akka broadcast stream).
- `Persistence/` — `SiteStorageService` (raw SQLite via `Microsoft.Data.Sqlite`), `SiteStorageInitializer`. - `Persistence/` — `SiteStorageService` (raw SQLite via `Microsoft.Data.Sqlite`), `SiteStorageInitializer`.
@@ -79,7 +79,7 @@ Central sends a `DeployInstanceCommand` carrying a JSON `FlattenedConfiguration`
1. Calls `EnsureDclConnections` to push any new or changed connection definitions to the DCL manager (hash-guarded: unchanged configs are skipped). 1. Calls `EnsureDclConnections` to push any new or changed connection definitions to the DCL manager (hash-guarded: unchanged configs are skipped).
2. Calls `CreateInstanceActor`, which does `Context.ActorOf(props, instanceName)`. 2. Calls `CreateInstanceActor`, which does `Context.ActorOf(props, instanceName)`.
3. Runs an off-thread `Task` that calls `SiteStorageService.StoreDeployedConfigAsync`, clears static overrides and native alarm state, and — if `_replicationActor` is non-null (it is optional and null in isolated deployments/tests) — tells `SiteReplicationActor` to push to the peer node. 3. Runs an off-thread `Task` that calls `SiteStorageService.StoreDeployedConfigAsync` and clears static overrides and native alarm state. Nothing is pushed to the peer: as of LocalDb Phase 2 those three tables are replicated, so the writes themselves reach the standby.
4. Pipes back a `DeployPersistenceResult`; only on success does it tell the deployer `DeploymentStatus.Success`. If persistence fails, the optimistically-created actor is stopped and the error is returned to central (`SiteRuntime-005`). 4. Pipes back a `DeployPersistenceResult`; only on success does it tell the deployer `DeploymentStatus.Success`. If persistence fails, the optimistically-created actor is stopped and the error is returned to central (`SiteRuntime-005`).
For redeployment (instance already running), the existing actor is stopped and watched: For redeployment (instance already running), the existing actor is stopped and watched:
@@ -216,7 +216,7 @@ Both `AlarmActor` and `NativeAlarmActor` tell the `InstanceActor` an `AlarmState
### Standby replication ### Standby replication
`SiteReplicationActor` runs on every site node (not a singleton). The active node's `DeploymentManagerActor` tells it `ReplicateConfigDeploy`, `ReplicateConfigRemove`, `ReplicateConfigSetEnabled`, `ReplicateArtifacts`, or `ReplicateStoreAndForward`. The replication actor tracks the peer node via Akka cluster membership events and forwards each command to `/user/site-replication` on the peer via `ActorSelection`. Replication is fire-and-forget (no ack wait per design), so a failed write to the standby is logged but does not fail the primary operation. Config replication has no actor. `SiteReplicationActor` — which received `ReplicateConfigDeploy` / `ReplicateConfigRemove` / `ReplicateConfigSetEnabled` / `ReplicateArtifacts` / `ReplicateStoreAndForward` from the active node's `DeploymentManagerActor`, tracked the peer through cluster membership events, and forwarded each command to `/user/site-replication` via `ActorSelection` — was deleted in LocalDb Phase 2, together with its notify-and-fetch exchange (the standby was told a deploy had happened and then HTTP-fetched the config itself). The site's config tables are now replicated by LocalDb CDC, so a deploy on either node reaches the other as an ordinary row change. `SiteReconciliationActor` still fetches over HTTP at node startup when central reports gaps; that is a different path and it survives.
## Usage ## Usage
docs/components/StoreAndForward.md
+4 -3
View File
@@ -4,13 +4,12 @@ The Store-and-Forward Engine buffers site-originated outbound messages when a ta
## Overview ## Overview
The Store-and-Forward Engine (#6) is a site-only component. The central cluster has no equivalent buffer; it uses the Notification Outbox (#21) instead for its own queued delivery work. Every site node runs one `StoreAndForwardService` instance, backed by a `StoreAndForwardStorage` SQLite store and an optional `ReplicationService` that fans each buffer mutation to the standby. The Store-and-Forward Engine (#6) is a site-only component. The central cluster has no equivalent buffer; it uses the Notification Outbox (#21) instead for its own queued delivery work. Every site node runs one `StoreAndForwardService` instance, backed by a `StoreAndForwardStorage` store. As of LocalDb Phase 2 that store writes to the consolidated LocalDb database, and the buffer reaches the peer as the replicated `sf_messages` table — the `ReplicationService` that used to fan each mutation to the standby by hand was deleted.
The component code lives in `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/`: The component code lives in `src/ZB.MOM.WW.ScadaBridge.StoreAndForward/`:
- `StoreAndForwardService` — the core buffer: enqueue, retry sweep, park/retry/discard, and the `ICachedCallLifecycleObserver` audit hook. - `StoreAndForwardService` — the core buffer: enqueue, retry sweep, park/retry/discard, and the `ICachedCallLifecycleObserver` audit hook.
- `StoreAndForwardStorage` — the SQLite layer; all reads and writes against `sf_messages`. - `StoreAndForwardStorage` — the SQLite layer; all reads and writes against `sf_messages`.
- `ReplicationService` — fire-and-forget buffer replication to the standby.
- `ParkedMessageHandlerActor` — Akka actor bridge that exposes parked-message query/retry/discard to the `SiteCommunicationActor`. - `ParkedMessageHandlerActor` — Akka actor bridge that exposes parked-message query/retry/discard to the `SiteCommunicationActor`.
- `NotificationForwarder` — the delivery handler for the `Notification` category; forwards buffered notifications to central via the ClusterClient transport and interprets the ack. - `NotificationForwarder` — the delivery handler for the `Notification` category; forwards buffered notifications to central via the ClusterClient transport and interprets the ack.
- `StoreAndForwardOptions` — options class bound from the `StoreAndForward` configuration section. - `StoreAndForwardOptions` — options class bound from the `StoreAndForward` configuration section.
@@ -129,7 +128,9 @@ else
### Async replication to standby ### Async replication to standby
`ReplicationService` wraps each buffer mutation — add, remove, park, requeue — in a `Task.Run` fire-and-forget. The active node does not wait for standby acknowledgment. The standby applies each `ReplicationOperation` via `ApplyReplicatedOperationAsync`, which calls the same `StoreAndForwardStorage` methods. Replication failures are logged at Debug and discarded; the standby may be slightly behind the active at any moment, producing at-most a few duplicate deliveries or missed retries after a failover — an accepted trade-off for zero added latency on the enqueue path. Replication is no longer the buffer's own concern. `sf_messages` is registered with LocalDb (`SiteLocalDbSetup.OnReady`), so every insert and status change is captured by a CDC trigger and shipped to the peer on the shared sync stream. The four hand-written operations — add, remove, park, requeue — and the `Task.Run` fan-out that carried them are gone, along with `ApplyReplicatedOperationAsync` and `ReplaceAllAsync`.
The trade-off is unchanged in shape: replication is still asynchronous, so the peer may be slightly behind at any instant. What changed is the bound. Convergence is now per row under last-writer-wins with HLC-ordered tombstones, so a lagging peer converges rather than diverging, and duplicate delivery after a failover is limited to messages the old primary delivered whose status change had not yet replicated. See `Component-StoreAndForward.md` for the normative statement of that bound.
The four `ReplicationOperationType` values are `Add`, `Remove`, `Park`, and `Requeue` (requeue was added to cover the operator-initiated `Parked→Pending` transition so the standby preserves retry intent after failover). The four `ReplicationOperationType` values are `Add`, `Remove`, `Park`, and `Requeue` (requeue was added to cover the operator-initiated `Parked→Pending` transition so the standby preserves retry intent after failover).
docs/deployment/installation-guide.md
+8
View File
@@ -187,6 +187,14 @@ ALTER ROLE db_owner ADD MEMBER scadabridge_svc;
Ensure bidirectional TCP connectivity between all Akka.NET cluster peers. The remoting port (default 8081) must be open in both directions. Ensure bidirectional TCP connectivity between all Akka.NET cluster peers. The remoting port (default 8081) must be open in both directions.
## Upgrading a Site Pair
**Stop both nodes of a site pair, upgrade both, then start both.** Rolling one node at a time is
not supported as of LocalDb Phase 2 — the legacy snapshot-compatibility handler that made a
mixed-version pair converge was deleted with the bespoke replicator, and a mixed pair now diverges
silently. See `docs/deployment/topology-guide.md` for the reasoning and for the related
`TombstoneRetention` bound on how long one node may stay offline.
## Post-Installation Verification ## Post-Installation Verification
1. Start the service: `sc.exe start ScadaBridge-Central` 1. Start the service: `sc.exe start ScadaBridge-Central`
docs/deployment/topology-guide.md
+21 -1
View File
@@ -150,7 +150,27 @@ Each site has its own two-node cluster:
- Same split-brain resolver as central (keep-oldest). - Same split-brain resolver as central (keep-oldest).
- Singleton actors: Site Deployment Manager migrates on failover. - Singleton actors: Site Deployment Manager migrates on failover.
- Staggered instance startup: 50ms delay between Instance Actor creation to prevent reconnection storms. - Staggered instance startup: 50ms delay between Instance Actor creation to prevent reconnection storms.
- SQLite persistence: Both nodes access the same SQLite files (or each has its own copy with async replication). - SQLite persistence: each node owns its own consolidated LocalDb database, kept in step by
asynchronous CDC replication over a gRPC sync stream (LocalDb Phase 1 + 2). The nodes do NOT
share a SQLite file.
### Site Pair Upgrades — stop and start BOTH nodes together
**A rolling upgrade of a site pair, one node at a time, is no longer supported.** It worked while
the bespoke replicator kept a legacy `SfBufferSnapshot` compatibility handler so a new standby
could still apply an old active node's monolithic snapshot. LocalDb Phase 2 deleted that handler
along with the replicator, so a mixed-version pair has no common replication path: the two nodes
will run, but they will not converge, and the divergence is silent.
Stop both nodes of a site pair, upgrade both, then start both.
**Related bound — do not leave one node of a pair offline for long.** A node absent for longer than
`LocalDb:Replication:TombstoneRetention` (default **7 days**) can **resurrect deleted rows** when
it rejoins: deletes replicate as HLC-ordered tombstones, and once a tombstone is pruned there is
nothing left to suppress the stale row the returning node still holds. Within the retention window
a rejoin is safe and self-correcting (verified live: a node stopped and restarted mid-load rejoined
with both nodes byte-identical and zero duplicates). Beyond it, rebuild the returning node's
database from its peer rather than letting it rejoin.
### Central-Site Communication ### Central-Site Communication
docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md
+21
View File
@@ -10,6 +10,27 @@ Fixed via the **notify-and-fetch** rework (the primary recommendation below), no
- **Plan:** [`docs/plans/2026-06-26-deploy-config-notify-and-fetch.md`](../plans/2026-06-26-deploy-config-notify-and-fetch.md) - **Plan:** [`docs/plans/2026-06-26-deploy-config-notify-and-fetch.md`](../plans/2026-06-26-deploy-config-notify-and-fetch.md)
- **Validated:** live docker-cluster smoke — a previously-hanging deploy now completes in ~0.11 s; reconciliation heals single-node and concurrent-both-missing gaps. - **Validated:** live docker-cluster smoke — a previously-hanging deploy now completes in ~0.11 s; reconciliation heals single-node and concurrent-both-missing gaps.
## Amendment (2026-07-20) — LocalDb Phase 2 removed the second hop entirely
The resolution above fixed the intra-site hop by replacing it with notify-and-fetch. LocalDb
Phase 2 then deleted **notify-and-fetch itself**, along with `SiteReplicationActor`: the site's
`deployed_configurations` table is now replicated by CDC, so the config reaches the standby as an
ordinary row change over the gRPC sync stream. There is no intra-site Akka hop carrying config any
more, so the 128 000-byte frame constraint does not apply to it in any form.
The central→site hop is unchanged — it still sends a small `RefreshDeploymentCommand` and the site
still fetches over HTTP, so that half of the original fix stands.
**The successor ceiling is different in kind.** The gRPC sync stream has a 4 MB default receive
limit, and LocalDb batches by ROW COUNT (`LocalDb:Replication:MaxBatchSize`, default 500), not by
bytes. A ~70 KB `config_json` — the largest measured in production — times 500 rows is ~35 MB,
which would exceed the limit. The rig therefore pins `MaxBatchSize` to **16** (~1.1 MB worst case).
Any deployment replicating wide rows must size that key deliberately; see the Phase 2 plan (D6) and
`docs/plans/2026-07-19-localdb-phase2-live-gate.md`.
Note the failure mode differs from the one documented below: an oversized gRPC message is
**rejected**, not silently dropped.
The diagnosis below is retained as the historical record of how the bug was found and reasoned about. The diagnosis below is retained as the historical record of how the bug was found and reasoned about.
## Summary ## Summary
docs/known-issues/2026-07-20-cached-telemetry-drain-hot-loop.md
+92
View File
@@ -0,0 +1,92 @@
# Cached-telemetry drain hot-loops forever on a row whose tracking snapshot is gone
**Date:** 2026-07-20 · **Status:** OPEN · **Severity:** Medium (log flood + wasted I/O; no data loss)
· **Area:** AuditLog / Site Telemetry
## Summary
`SiteAuditTelemetryActor`'s cached-telemetry drain reads Pending audit rows, looks up each row's
tracking snapshot by `CorrelationId`, and pushes the combined packet to central. When the lookup
returns `null` the row is **skipped and deliberately left Pending**
(`SiteAuditTelemetryActor.cs:307`), on the reasoning that "central reconciliation will pick it up".
Nothing ever removes such a row from the local drain queue. The next tick re-reads it, fails the
same lookup, logs the same warning, and leaves it Pending again — **forever**. With a batch of
unresolvable rows the actor spins at its non-idle rate and emits one warning per row per pass.
Measured on the docker rig: **~2 800 warnings/minute, sustained**, surviving both a process restart
and a container restart, until the audit database itself was discarded.
```
[09:57:41 WRN] [Site/scadabridge-site-a-a] Cached-telemetry drain: no tracking snapshot for
a5392796-291f-4f5b-9fbf-5817c1ec76c7 (TrackedOperationId 59bd4bf8-…); skipping.
```
## Why the rows became unresolvable
Two independent stores must agree:
- the **audit** rows live in `auditlog.db` (site-local, and on the docker rig **inside the
container at `/app/auditlog.db`**, not on the bind-mounted data volume);
- the **tracking** rows live in `OperationTracking`, which LocalDb Phase 1 moved into the
consolidated `LocalDb:Path` database (bind-mounted).
Anything that resets one without the other strands every audit row that referenced it. The code
comment already anticipates the cause — *"possible if the audit row is older than the tracking
retention window, or the tracking store was reset"* — so this is a known-and-accepted input, not an
exotic one.
**Two realistic production triggers, neither requiring operator error:**
1. **Tracking retention expiry.** If the tracking retention window elapses before the audit drain
catches up — a long central outage, a large backlog — the snapshots are pruned out from under
still-Pending audit rows and every one of them becomes a permanent hot-loop entry.
2. **Restoring or resetting one store independently of the other**, e.g. rebuilding a node's
LocalDb file from its peer while its container-local `auditlog.db` survives.
It was hit here by (2): the two site-a LocalDb databases were dropped during a rig cleanup while
`auditlog.db` — being inside the container — survived.
## Why the current handling is not enough
Skipping the row is correct; **leaving it Pending with no other state change is not**. The row is
now in a state it can never leave:
- no attempt counter, so an unresolvable row is indistinguishable from a transiently-failing one;
- no backoff, so the actor runs at full non-idle rate against a queue that can never shrink;
- no terminal state, so it is retried for the life of the database;
- one Warning per row per pass, which buries every other log line on the node.
The "central reconciliation will pick it up" comment is about the **audit half** reaching central by
another path. That may well be true — but it does not release the row from the local drain queue,
which is what actually loops.
## Suggested fix
Give an unresolvable row somewhere to go. Roughly, in increasing order of effort:
1. **Bound the retries.** Add an attempt count; past a threshold mark the row terminal
(`TrackingUnavailable`) and stop re-reading it. Emit a single summary Warning with the count
rather than one per row per pass.
2. **Rate-limit the warning** to one per drain episode regardless of row count — the same pattern
`MaintenanceBackgroundService` already uses for the oplog caps-exceeded warning (`_snapshotFlagWarned`).
3. **Push the audit half alone** when the tracking snapshot is missing, rather than skipping the row
entirely, so the row can be marked emitted and leave the queue. Needs a decision on whether
central accepts a packet with no tracking half.
(2) alone would remove the operational damage; (1) or (3) is needed to stop the wasted I/O.
## Reproduction
1. Run a site node until it has cached-call audit rows with tracking correlations.
2. Stop the node; delete its consolidated LocalDb database (which holds `OperationTracking`);
leave `auditlog.db` in place.
3. Start the node. The drain warning repeats indefinitely; the rate does not decay.
## Notes
- **No data loss.** The audit rows are intact and still reach central by the reconciliation path;
what is broken is the local drain's ability to ever finish.
- Discovered while cleaning the rig after the LocalDb Phase 2 live gate
(`docs/plans/2026-07-19-localdb-phase2-live-gate.md`), which is also where the related
"deleting an instance orphans its buffered messages" observation is recorded.
docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md
+475
View File
@@ -0,0 +1,475 @@
# LocalDb throws `SQLite Error 10: 'disk I/O error'` on the active site node under sustained write load
**Date:** 2026-07-20 · **Status:** ROOT-CAUSED + FIX PASS COMPLETE 2026-07-20 (same day) — observer-induced, **not a LocalDb defect**; see §0 (cause) and §11a (fixes) · **Severity:** was High; resolved to an operational rule (now enforced in the tooling docs) + shipped hardening
**Area:** `ZB.MOM.WW.LocalDb` (library, `~/Desktop/scadaproj/ZB.MOM.WW.LocalDb/`) as consumed by ScadaBridge site nodes
**Found by:** the Phase 2 rig soak — [`docs/plans/2026-07-19-localdb-phase2-soak.md`](../plans/2026-07-19-localdb-phase2-soak.md)
**Branch:** `feat/localdb-phase2` (symptom observed on Phase 1 code)
> **Update 2026-07-20:** the mechanism has been identified and reproduced on demand, both in a
> minimal SQLite-only repro and on the live rig, and the follow-up fixes have shipped.
> Sections §0, §11a and §11 below are authoritative; the original brief (§1–§10) is preserved
> as written, with corrections annotated where its conclusions did not survive
> (§4.2, §4.3, §7, §9) and fix notes where they did (§5.1, §8).
---
## 0. ROOT CAUSE (verified)
**A host-side (macOS) `sqlite3` read of a live, bind-mounted WAL database checkpoints and
resets the WAL out from under the container process, permanently poisoning that process's
connections.** LocalDb, its connection handling, its UDF, and its triggers are not involved —
the mechanism reproduces with plain Python `sqlite3` and no LocalDb code at all.
Mechanism, step by step:
1. POSIX advisory locks do **not** propagate across the Docker Desktop virtiofs bind-mount
boundary. A host `sqlite3` opening the database cannot see the container's locks (and vice
versa), so it believes it is the **only** connection.
2. On close, the "last" connection in WAL mode runs a full checkpoint and **resets the WAL to
0 bytes**. Because the read happened while the container was idle (a standby node, or a gap
between write bursts), nothing blocks the checkpoint. This is exactly the file signature
found on both rig nodes: main DB mtime + 0-byte `-wal` stamped at the sampling minute.
3. The container process still holds the old WAL-index state (its per-inode `-shm` mapping,
kept alive forever by the held-open `_master` connection and the Microsoft.Data.Sqlite
connection pool). That index says the WAL contains N frames; the file now has none. Every
subsequent statement — reads and writes both consult the WAL index — fails with
**`SQLITE_IOERR_SHORT_READ` (extended code 522)**, surfaced as primary code 10
`'disk I/O error'` (some paths surface `SQLITE_NOTADB` (26) instead). The poisoning is
**permanent until the process reopens the database** (restart).
### Why the original brief's conclusions were wrong
- **"It tracks the load, not the node" (§4.1)** — confounded. *Both* nodes were poisoned by
the 04:56 UTC host-side sampling (both nodes' `site-localdb.db` main files carry the 04:56
mtime; node-b's WAL was left at 0 bytes). A poisoned **standby** shows zero errors only
because a standby issues ~zero LocalDb statements; the errors "followed the load" because
the load is what generates statements against an already-poisoned handle. Node-b's very
first write attempt after failover (04:59:37, `OperationTrackingStore.RecordAttemptAsync`)
failed — it had been poisoned for 3 minutes with nothing to say about it.
- **"It is LocalDb-specific" (§4.2)** — sampling-selection bias. The legacy WAL databases in
the same directory were healthy only because no host process ever read *them*. The minimal
repro poisons an arbitrary WAL database the same way.
- **"The observer has been ruled out" (§4.3)** — the exclusion assumed an error-free standby
was an unpoisoned standby. It wasn't; it was a poisoned node with no traffic.
### Verification (2026-07-20, all on the live rig + minimal repro)
1. **Load alone is harmless:** restarted the poisoned active node (site-a-b); freshly-reopened
site-a-a took the full soak load for **10+ minutes with zero errors** (the original model
predicted onset within ~2 min), WAL growing/checkpointing normally, DB 188 KiB → 476 KiB.
2. **One host read is sufficient and immediate:** a single
`sqlite3 docker/site-a-node-a/data/site-localdb.db "SELECT count(*) FROM site_events;"`
against the healthy loaded node reset its 4.6 MiB WAL to 0 bytes in place and produced the
first `disk I/O error` **one second later** (05:32:48 → 05:32:49), 203 errors in the next
40 s — the same one-second onset correlation as the original 04:56:36 → 04:56:37 incident.
3. **Minimal repro (no LocalDb, no .NET):** a `python:3.12-alpine` container writing a
WAL-mode SQLite DB on a bind mount (held master connection + fresh connection per op,
`synchronous=NORMAL`, `busy_timeout=5000`). A host `sqlite3 "SELECT count(*)"`:
- against the **actively-writing** DB → immediate `SQLITE_IOERR_SHORT_READ` (522) +
`SQLITE_NOTADB` burst, then recovery (checkpoint could not fully reset a hot WAL);
- during an **idle window** (connections held open, WAL populated) → WAL reset
1.2 MiB → 0 bytes, then **every fresh-connection write failed for the rest of the run
(200/200)** — the persistent variant, matching the rig.
### Consequences
1. **The operational rule in §5.3 ("do not query the databases with host-side `sqlite3`") is
the root cause, not a hygiene note.** One violation silently destroys the node's local
persistence until restart. This applies to *every* WAL SQLite file on the bind mount
(`scadabridge.db`, `store-and-forward.db` included), not just LocalDb.
2. **Safe inspection recipes:** copy the file triplet (`.db`, `-wal`, `-shm`) and open the
copy; or read from inside the container boundary (same kernel ⇒ locks visible), e.g.
`docker run --rm -v <dir>:/d alpine/sqlite3 sqlite3 /d/site-localdb.db "..."` — never the
macOS host against live files.
3. **LocalDb Phase 2 is unblocked** on this issue: the library sustained the full soak write
load indefinitely once nothing external touched its file.
4. Hardening follow-ups — **status as of the 2026-07-20 fix pass (see §11a):**
- **DONE — extended-code logging:** the LocalDb-adjacent catch sites (`SiteAuditTelemetryActor`,
`CachedCallTelemetryForwarder`, `SiteEventLogger`) now log
`SqliteException` primary/extended codes (`sqlite 10/522`-style) via
`SqliteErrorCodes.Describe` / `DescribeSqliteError`.
- **DONE — §8 async-context bug** (see §8).
- **DONE — §9.5 load regression test** (see §9).
- **NOT DONE (deliberately):** a detect-and-reopen self-heal in `SqliteLocalDb` for
persistent `SQLITE_IOERR`/`SQLITE_NOTADB`. This is a real library design change
(pool clear + master reopen + in-flight coordination) protecting against *external
interference only* — the trigger is operator/tooling action, now prevented at the
source, and on same-kernel production deployments external readers see the locks and
are safe. File as its own issue if production ever runs where a foreign-kernel reader
can touch the files.
---
> **Original brief follows, preserved as written on 2026-07-20 before root-causing.**
---
## 1. Summary
On a ScadaBridge site node, once the node is **active** and under sustained concurrent write
load, effectively every write to the consolidated LocalDb database (`site-localdb.db`) fails
with:
```
Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 10: 'disk I/O error'.
```
Observed rate: **~1 000–1 500 failures per minute**, sustained, not transient. The node stays up
and reports healthy. Ordinary (non-LocalDb) SQLite databases in the same directory, in the same
process, under the same load, are completely unaffected.
## 2. Why it matters
1. **Silent data loss today.** `SiteEventLogger` fails its inserts and logs
`[ERR] Failed to record event: script from ScriptActor:…`. Site event logging is dropping
events on the floor on the active node whenever the site is busy. `OperationTracking` writes
fail too, which breaks cached-call status tracking (`Cached-telemetry drain: no tracking
snapshot for …; skipping`).
2. **It blocks LocalDb Phase 2.** Phase 2 registers eight further tables into this same
database — including `native_alarm_state` (highest-volume table on the node) and
`sf_messages` — **and deletes the bespoke mechanisms that currently carry that data**
(`SiteReplicationActor`, `StoreAndForward.ReplicationService`) in the same commit. Cutting
over onto this store while removing the fallback would convert a logging defect into config
and buffer loss.
3. Phase 1 was previously live-gated as PASS. That gate exercised correctness and convergence,
**not sustained write load** — which is why this was not caught.
## 3. Exact symptom
Two representative stacks, both from `docker logs scadabridge-site-a-b` while that node was
active and under load:
```
Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 10: 'disk I/O error'.
at Microsoft.Data.Sqlite.SqliteDataReader.NextResult()
at Microsoft.Data.Sqlite.SqliteCommand.ExecuteReader(CommandBehavior behavior)
at ZB.MOM.WW.ScadaBridge.SiteEventLogging.SiteEventLogger.<>c__DisplayClass15_0.<ProcessWriteQueueAsync>b__0(SqliteConnection connection)
in /src/src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs:line 236
at ZB.MOM.WW.ScadaBridge.SiteEventLogging.SiteEventLogger.ProcessWriteQueueAsync()
in /src/src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs:line 221
```
```
Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 10: 'disk I/O error'.
at Microsoft.Data.Sqlite.SqliteDataReader.NextResult()
at Microsoft.Data.Sqlite.SqliteCommand.ExecuteNonQuery()
at ZB.MOM.WW.ScadaBridge.SiteRuntime.Tracking.OperationTrackingStore.RecordEnqueueAsync(...)
in /src/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Tracking/OperationTrackingStore.cs:line 137
at ZB.MOM.WW.ScadaBridge.AuditLog.Site.Telemetry.CachedCallTelemetryForwarder.TryEmitTrackingAsync(...)
in /src/src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/Telemetry/CachedCallTelemetryForwarder.cs:line 148
```
Note both fail inside `SqliteDataReader.NextResult()` — i.e. at statement execution, not at
`Open()`. Connections are being acquired successfully; the failure is on the write itself.
### Error-source distribution
Error-stack frames counted over one 3-minute window on the loaded node:
| Store | Backing file | Frames |
|---|---|---|
| `OperationTrackingStore` | `site-localdb.db` (**LocalDb**) | 13 044 |
| `SiteAuditTelemetryActor` | `site-localdb.db` (**LocalDb**) | 4 350 |
| `SiteEventLogger` | `site-localdb.db` (**LocalDb**) | 900 |
| `CachedCallTelemetryForwarder` | `site-localdb.db` (**LocalDb**) | 162 |
| `StoreAndForwardStorage` | `store-and-forward.db` (legacy) | **0** |
| `SiteStorageService` | `scadabridge.db` (legacy) | **0** |
## 4. Evidence — what has been established
### 4.1 It tracks the load, not the node
The load was moved between the two site-a nodes by restarting the active one (the surviving node
becomes oldest-up and takes over):
| Node | Role | Under load | `disk I/O error` / 4 min |
|---|---|---|---|
| site-a-a | active | yes | 2 175 |
| site-a-a | standby (after restart) | no | **0** |
| site-a-b | standby | no | **0** |
| site-a-b | active (after failover) | yes | **4 391** |
### 4.2 It is LocalDb-specific, not the filesystem or the bind mount — **WRONG, see §0**
> **Correction 2026-07-20:** sampling-selection bias — only the LocalDb file was ever read
> from the host. Any of these WAL databases is equally poisonable (minimal-repro-proven).
This is the strongest signal. `store-and-forward.db` and `scadabridge.db` live in the **same
bind-mounted directory** (`/app/data`, host `docker/site-a-node-*/data/`), are opened by the
**same process**, are also **WAL-mode**, and are being written **concurrently under the same
load** — and they log zero errors. Only the LocalDb-managed file fails.
### 4.3 The observer has been ruled out — **WRONG, see §0: the observer was the cause**
> **Correction 2026-07-20:** the exclusion below assumed an error-free standby was an
> unpoisoned standby. Node-b's files carry the 04:56 sampling-time mtimes (WAL left at
> 0 bytes); it was poisoned then and merely silent until failover gave it write traffic.
Onset (04:56:37) was **one second after** a host-side `sqlite3` read of the bind-mounted
database (04:56:36), making observer-induced `-shm` corruption the leading hypothesis. It is
excluded:
- After node-a was restarted (fresh open, `-shm` recovered) and load failed over to node-b,
**node-b** — whose files no host process had touched since a single baseline read, and which
had been error-free for the entire preceding period — began erroring immediately at a *higher*
rate.
- **node-a**, whose files *had* been sampled, dropped to zero once it stopped carrying load.
The variable that tracks the errors is load. (Host-side `sqlite3` against a live WAL database
over a bind mount is still unsafe and should be avoided — it is just not the cause here.)
### 4.4 Not disk pressure
Host had 215 GiB free throughout (`df -h`: 76 % used on the data volume). Files are small:
`site-localdb.db` 188 KiB, WAL peaked around 4.1 MiB then checkpointed to 0.
## 5. Reproduction
Fully reproducible in ~10 minutes on the local docker rig.
### 5.1 Rig prerequisites
Two rig-tooling bugs will block a fresh reseed; both are documented in the soak findings:
- `docker/seed-sites.sh` role names — **already fixed** (commit `cf46e596`).
- **`infra/mssql/setup.sql` never executes** — **FIXED 2026-07-20**: `infra/reseed.sh` now
applies the three init scripts itself via `sqlcmd` once MSSQL accepts connections (the
`/docker-entrypoint-initdb.d/` compose mounts are informational only — the official
`mcr.microsoft.com/mssql/server` image does not implement that hook; noted in the compose
file). The manual workaround below is retained for historical context / older checkouts.
Original problem: after `infra/reseed.sh` dropped the volume, nothing created
`ScadaBridgeConfig` or the `scadabridge_app` login and `reseed.sh` hung forever on its
setup.sql poll. The by-hand equivalent:
```bash
cd ~/Desktop/ScadaBridge/infra
for f in mssql/setup.sql mssql/machinedata_seed.sql mssql/setup-env2.sql; do
docker exec -i scadabridge-mssql /opt/mssql-tools18/bin/sqlcmd \
-S localhost -U sa -P 'ScadaBridge_Dev1#' -C -b < "$f"
done
```
Then restart the app containers so EF migrations run, and restart central again after
`seed-sites.sh` writes `LdapGroupMappings` (they are cached at startup).
### 5.2 Build the load generator
**The seeded `Motor Controller` template (id 4) cannot be used** — it fails pre-deployment
validation with 34 errors (30 `ConnectionBinding`, 4 `ScriptCompilation`). Build a minimal one.
**Critical:** `ExternalSystem.Call` does **not** buffer to store-and-forward in practice.
`ExternalSystem.CachedCall` is the buffering surface. Using `Call` produces HTTP traffic and no
S&F rows, and will not reproduce this.
```bash
cd ~/Desktop/ScadaBridge
SB=src/ZB.MOM.WW.ScadaBridge.CLI/bin/Debug/net10.0/scadabridge # dotnet build src/...CLI first
AUTH="--url http://localhost:9000 --username multi-role --password password"
# 1. Point the seeded external system at a refusing address (discard port).
$SB $AUTH external-system update --id 1 --name "Test REST API" \
--endpoint-url "http://127.0.0.1:9" --auth-type ApiKey --auth-config "scadabridge-test-key-1"
# 2. Minimal template: no attributes, no compositions, no connection bindings.
$SB $AUTH --format json template create --name "SoakGenerator" # -> note the id
$SB $AUTH --format json template script add --template-id <TID> --name "SoakCall" \
--trigger-type Interval --trigger-config '{"intervalMs":5000}' \
--code 'var parms = new Dictionary<string, object?> { ["a"] = 2, ["b"] = 3 }; await ExternalSystem.CachedCall("Test REST API", "Add", parms);'
# 3. Four instances on site-a (site id 1), then deploy each.
for i in 1 2 3 4; do
$SB $AUTH --format json instance create --name "soakgen-$i" --template-id <TID> --site-id 1
done
$SB $AUTH instance deploy --id <each instance id>
```
Note the CLI's `template script update` requires `--name` and `--trigger-type` even when only
changing `--code`. In zsh, do not put the auth flags in an unquoted variable — zsh does not
word-split, so pass them literally or use `${=AUTH}`.
### 5.3 Observe
```bash
# Identify the ACTIVE node — it is the one running the ScriptActors.
docker logs --since 4m scadabridge-site-a-a 2>&1 | grep -c "Connection refused"
docker logs --since 4m scadabridge-site-a-b 2>&1 | grep -c "Connection refused"
# Errors appear on that node within ~2 minutes of load starting.
docker logs --since 4m scadabridge-site-a-<active> 2>&1 | grep -c "disk I/O error"
```
Metrics (port 8084 is **not** published, and the `aspnet:10.0` image has **no `curl`**) — use a
sidecar in the container's network namespace:
```bash
docker run --rm --network container:scadabridge-site-a-a curlimages/curl:latest \
-s localhost:8084/metrics | grep '^localdb_'
```
Do **not** query the databases with host-side `sqlite3` while containers are writing them.
## 6. Code map
### Library — `~/Desktop/scadaproj/ZB.MOM.WW.LocalDb/src/ZB.MOM.WW.LocalDb/Internal/SqliteLocalDb.cs`
Facts relevant to the failure:
- **A `_master` connection is held open for the object's entire lifetime** (`:31`), explicitly to
"anchor the WAL journal". It is guarded by a `Lock _masterLock` because `SqliteConnection` is
not thread-safe.
- **`CreateConnection()` (`:83`) opens a brand-new `SqliteConnection` per call** — one per
operation, from many concurrent actors. Every call then runs
`PRAGMA synchronous=…; PRAGMA busy_timeout=…; PRAGMA foreign_keys=ON;` and registers a UDF:
```csharp
conn.CreateFunction("zb_hlc_next", () => _clock.Next());
```
- The connection string is **only** `DataSource=<path>` (`:57`) — **connection pooling is left at
the Microsoft.Data.Sqlite default (enabled)**, and no `Cache=` or `Mode=` is set.
- Effective options on the rig are the defaults: `BusyTimeoutMs = 5000`, `Synchronous = NORMAL`.
ScadaBridge's rig config (`docker/site-a-node-*/appsettings.Site.json`, `LocalDb` section) sets
only `Path` and the replication block.
- `zb_hlc_next()` is invoked **from inside the capture triggers**, i.e. on the SQLite thread
during every INSERT/UPDATE/DELETE on a registered table, and it calls into the shared
`HybridLogicalClock` from arbitrary threads.
### Failing call sites (ScadaBridge)
- `src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs:221,236` — a channel-drained
single-writer loop (`ProcessWriteQueueAsync`) using a `WithConnection(...)` helper.
- `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Tracking/OperationTrackingStore.cs:137` (`RecordEnqueueAsync`),
`:260,266` (`GetStatusAsync`).
- `src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/Telemetry/CachedCallTelemetryForwarder.cs:148`.
- `src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/Telemetry/SiteAuditTelemetryActor.cs` — also see §8.
## 7. Hypotheses, ranked
> **Resolution 2026-07-20:** none of the four below is the cause. The mechanism is a variant
> of #2's territory (bind-mount `-shm`/WAL fragility) but triggered *only* by a host-side
> reader — LocalDb's concurrency, pooling, and UDF (hypotheses 1/3/4) are exonerated. The
> extended code, since captured, is `SQLITE_IOERR_SHORT_READ` (522).
None verified. Ordered by how well they fit "LocalDb only, load-dependent, same directory as
healthy WAL databases".
1. **Connection churn × pooling × per-connection UDF registration.** LocalDb opens a fresh
`SqliteConnection` per operation with pooling enabled, and calls `CreateFunction` on every
acquisition. Under high concurrency this drives far more open/close and `-shm` mapping churn
than the legacy stores (which reuse a small number of connections), and is the clearest
structural difference between the failing and healthy databases. Suspect the interaction of
the pool with the long-lived `_master` connection and WAL index growth.
2. **`-shm` / WAL-index growth over the bind mount, triggered only at LocalDb's concurrency.**
Would explain why the same mount is fine for lower-concurrency databases. `mmap` of the shared
WAL index across virtiofs is a known-fragile area. **Distinguishing test: run the same load
with `LocalDb:Path` pointed at a container-local path (a `tmpfs` or a plain volume rather than
the bind mount).** If the errors vanish, this is confirmed and the fix is environmental /
deployment-shaped rather than a library bug. **Run this test first — it is cheap and it
partitions the hypothesis space.**
3. **`zb_hlc_next` UDF failing inside a trigger.** An exception thrown out of the managed UDF
callback during trigger execution can surface as a generic SQLite error at the statement
level. Check `HybridLogicalClock.Next()` for thread-safety and for anything that can throw
under contention (e.g. a spin/overflow path when many callers request stamps in the same
millisecond).
4. **Busy-timeout exhaustion misreported.** `BusyTimeoutMs = 5000` with heavy multi-connection
write contention on one file. This would normally surface as `SQLITE_BUSY` (5), not
`SQLITE_IOERR` (10), so it is a weaker fit — but worth excluding.
### The single highest-value next step
**Capture the extended result code.** The logs only show the primary code (`10` = `SQLITE_IOERR`),
which is generic. `SqliteException.SqliteExtendedErrorCode` names the failing syscall and would
likely settle this outright:
| Extended code | Meaning | Points at |
|---|---|---|
| `SQLITE_IOERR_SHMMAP` (6154) / `SQLITE_IOERR_SHMSIZE` (4874) | WAL index mmap/resize failed | hypothesis 2 |
| `SQLITE_IOERR_WRITE` (778) / `SQLITE_IOERR_FSYNC` (1034) | plain write/fsync failed | filesystem |
| `SQLITE_IOERR_LOCK` (3850) | file locking failed | bind mount locking |
Add the extended code to the exception logging (or attach a debugger / run the repro against a
local non-container build) before pursuing any fix.
## 8. Secondary defect in the same path — **FIXED 2026-07-20**
```
[ERROR][akka://scadabridge/user/site-audit-telemetry] There is no active ActorContext,
this is most likely due to use of async operations from within this actor.
Cause: System.NotSupportedException
```
`SiteAuditTelemetryActor` is touching `Context` (or `Self`/`Sender`) after an `await`. This is a
real bug independent of the I/O errors, though it sits in the same write path and may be
contributing. Note the family-wide rule already recorded for Akka work: never read `Self`/`Context`
after an `await` inside an actor.
> **Fixed 2026-07-20.** Root cause: both drain handlers await with `ConfigureAwait(false)`, so
> their `finally`-block re-arm (`ScheduleNext`/`ScheduleNextCached`) runs on a pool thread with
> no active ActorContext. Investigation found the failure is **bimodal**, and the second mode is
> worse than the logged one: depending on what the pool thread's thread-static cell slot holds,
> `Context`/`Self` either **throw** `NotSupportedException` (the logged variant — actor crashes
> and restarts once per drain) or **silently resolve a STALE cell of whatever actor last ran on
> that thread**, re-arming the tick at the *wrong actor* so the drain loop just stops (observed
> under TestKit: the tick landed on the TestActor). Fix: capture `Context.System.Scheduler` and
> `Self` into fields at construction (both are thread-safe immutable handles) and use only those
> from the re-arm path. Regression test
> `SiteAuditTelemetryActorTests.Drains_Whose_Awaits_Complete_Off_The_Actor_Thread_Keep_Draining_Without_Crashing`
> forces the mock awaits to complete off the actor thread — which every pre-existing test
> avoided by returning already-completed tasks — and catches **both** variants (EventFilter for
> the throw, sustained-drain counts for the silent stall). AuditLog suite 355/355 green.
## 9. What a fix must satisfy
> **Resolution 2026-07-20:** criteria 1–4 are already satisfied by the unmodified code once no
> host process touches the live files — verified 10+ min of soak load with zero errors, events
> durably written, WAL checkpointing normally. Criterion 5 (a sustained concurrent-write load
> test) would **not** have caught this — the trigger is an external reader, not load — but is
> now in place anyway: `ConcurrentWriteLoadTests` in `ZB.MOM.WW.LocalDb.Tests` (8 concurrent
> writers × 250 inserts through fresh pooled connections against a registered/triggered table on
> a real file, with concurrent readers; asserts zero failures + exact row/oplog counts; suite
> 145/145). §8's `SiteAuditTelemetryActor` async-context bug is **fixed** — see §8.
1. The §5 repro runs for **30 minutes under sustained load with zero `disk I/O error`** on the
active node.
2. No `Failed to record event` errors — site events are durably written under load.
3. `localdb_oplog_depth` rises under load and **drains** between bursts; zero dead letters.
4. Replication still converges across the site-a pair (Phase 1's existing convergence suite and
live gate still pass).
5. A regression test that would have caught this — i.e. a **concurrent-write load test** against
a real LocalDb file, not just the correctness/convergence tests Phase 1 shipped. Phase 1's
gate passed precisely because no test applied sustained concurrent write pressure.
## 10. Rig state as left
- Rig fully reseeded; central config volume dropped and replayed; site SQLite state wiped
(`reseed.sh` stage 2 does `rm -rf docker/site-*/data/*`).
- `ExternalSystemDefinitions` id 1 is **still repointed to `http://127.0.0.1:9`** — restore to
`http://scadabridge-restapi:5200` when done.
- Template `SoakGenerator` (id 2021) and instances `soakgen-1..4` (ids 5–8) are **still deployed
and still generating load** on site-a.
- `LdapGroupMappings` corrected in the live DB to the canonical `Designer`/`Deployer` names.
## 11a. Fix pass (2026-07-20, same day — all verified)
Everything actionable that this incident identified is now fixed (uncommitted on each repo's
current branch; ScadaBridge full solution builds clean, 0 warnings):
| # | Issue | Fix | Verification |
|---|---|---|---|
| 1 | §8 `SiteAuditTelemetryActor` async-context bug (bimodal: crash-per-drain OR silent tick misroute) | Capture `Context.System.Scheduler` + `Self` at construction; re-arm path never reads thread-static context | New red→green regression test forcing off-actor-thread continuations; AuditLog suite 355/355 |
| 2 | Diagnostics gap: logs carried only the primary SQLite code | `SqliteErrorCodes.Describe` (AuditLog) + `DescribeSqliteError` (SiteEventLogging) — catch sites now log `sqlite <primary>/<extended>` | Builds clean; suites green (this gap cost the investigation a from-scratch repro to learn code 522) |
| 3 | §5.1 `reseed.sh` hangs forever waiting on the initdb hook the mssql image doesn't have | `reseed.sh` applies `setup.sql`/`machinedata_seed.sql`/`setup-env2.sql` itself via `sqlcmd`; compose mounts annotated as informational | `bash -n` clean; scripts verified idempotent (`IF NOT EXISTS` guards) |
| 4 | §9.5 missing concurrent-write load test | `ConcurrentWriteLoadTests` in `ZB.MOM.WW.LocalDb.Tests` (scadaproj) — 8 writers × 250 pooled-connection inserts on a registered table + concurrent readers, exact row/oplog count asserts | LocalDb suite 145/145 |
| 5 | Root cause itself (operator/tooling host reads) | Poisonous instructions removed from the Phase 2 plan + `.tasks.json` (safe `snap()` copy-based sampling); soak-doc verdict corrected; family-wide memory rule recorded | On-demand on/off reproduction, §0 |
Deliberately **not** done: the `SqliteLocalDb` detect-and-reopen self-heal (see §0
consequence 4 for the rationale and the condition under which to file it).
## 11. Rig state after root-causing (2026-07-20 ~05:40 UTC)
- Both site-a nodes restarted during verification, curing both poisonings. End state:
**site-a-b active** carrying the soak load, site-a-a standby, **zero `disk I/O error` on
both** under sustained load.
- The §10 items still stand: `ExternalSystemDefinitions` id 1 still points at
`http://127.0.0.1:9`, and `SoakGenerator` + `soakgen-1..4` are still deployed and
generating load — the Phase 2 soak can now proceed on a clean baseline.
- Minimal-repro scripts (`writer.py` burst variant, `writer2.py` idle-window variant) lived in
the session scratchpad; the recipe is fully described in §0 and takes ~2 minutes to rebuild.
docs/plans/2026-07-19-localdb-adoption-phase2.md
+1207
View File
File diff suppressed because it is too large Load Diff
docs/plans/2026-07-19-localdb-adoption-phase2.md.tasks.json
+106
View File
@@ -0,0 +1,106 @@
{
"planPath": "docs/plans/2026-07-19-localdb-adoption-phase2.md",
"execution": {
"mode": "parallel-waves",
"implementerModel": "opus",
"isolation": "worktree",
"branch": "feat/localdb-phase2",
"baseBranch": "feat/localdb-phase1",
"note": "Phase 1's branch is NOT merged/pushed, so phase 2 branches from it. Dispatch every unblocked task concurrently per the wave table in the plan. Parallel implementers MUST use worktree isolation - concurrent git in one worktree races destructively."
},
"scopeDecision": {
"date": "2026-07-19",
"by": "user",
"choice": "Full scope as designed",
"note": "User chose both surfaces (config tables + sf_messages) in one phase, deleting both bespoke mechanisms together, over the recommended split of S&F-first. The four open design questions are therefore resolved INSIDE this plan as D1-D6 rather than deferred."
},
"reviewPass": {
"date": "2026-07-19",
"note": "Plan verified against actual code (3 verification sweeps + LocalDb library source) and corrected in place. Headline corrections: D1 (StoreDeployedConfigIfNewerAsync has a SECOND surviving caller, SiteReconciliationActor.cs:166 - the method and guard STAY; original Task 13 would also not have compiled, deleting a method whose caller dies only in Task 15), D3 (the active node ALREADY purges at DeploymentManagerActor.cs:1921 - Task 12 became a pin test, nothing is re-homed), D6 added (4 MB gRPC receive cap x row-count-only MaxBatchSize batching; config_json > 128 KB documented - measure in Task 1, size MaxBatchSize in Task 19, single row near 4 MB = stop/lib work), Task 1 rewritten (Phase 2 tables are NOT registered on the Phase 1 rig so driven churn never reaches __localdb_oplog - measure legacy-DB write rates + arithmetic; metrics port 8084 not 8080; real metric names are localdb_oplog_depth / localdb_sync_*, the plan's localdb_oplog_backlog/replication_dead_letters/sync_connected never existed; containers have no sqlite3 - sample via throwaway copies of the DB triplet, NEVER host-side sqlite3 against the live files [2026-07-20: that poisons the container's WAL state - see docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md]; branch already exists, no checkout -b), Task 14 (register 8 tables NOT 11 - notification_lists/smtp_configurations deliberately unregistered, reversing the original instruction; keep Migrate LAST in OnReady)."
},
"decisions": [
{
"id": "D1",
"subject": "Config moves to CDC; notify-and-fetch is DELETED - but the guarded write STAYS",
"evidence": "SiteReplicationActor sends id+fetch-coords only because the config blob exceeds Akka's 128KB frame (docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md, already marked RESOLVED by the notify-and-fetch rework). LocalDb sync is gRPC - no such limit (but see D6). CORRECTED: StoreDeployedConfigIfNewerAsync (SiteStorageService.cs:301-336, guard at :325) has TWO production callers - SiteReplicationActor.cs:375 (dies in Task 15) AND SiteReconciliationActor.cs:166 (per-node startup self-heal vs central, SURVIVES Phase 2, stale-fetch race still real). Method + guard stay; reconcile becomes a benign second CDC writer. Do NOT reproduce the guard on top of LWW - deployed_at and HLC are different clocks and mixing them is non-convergent."
},
{
"id": "D2",
"subject": "ReplaceAllAsync deleted; the N1 directional guard becomes unnecessary",
"evidence": "LocalDb's snapshot resync MERGES per-row LWW and never WIPES: SnapshotApplier.OnBeginAsync (SnapshotStreamer.cs:163-170) resets counters only; OnBatchAsync (:172-186) routes snapshot rows through the same LwwApplier as deltas; LwwApplier.cs:69-78 discards an incoming row whose HLC is lower. Row-level deletes DO replicate (delete-trigger tombstones, streamed by SnapshotStreamer, applied as real DELETEs by LwwApplier) - only the destructive whole-table replace is gone. Caveat: tombstones pruned after TombstoneRetention (default 7d); a node offline longer can resurrect deleted rows (runbook, Task 21). SEMANTIC CHANGE: the standby is convergent, no longer byte-identical."
},
{
"id": "D3",
"subject": "CORRECTED: the SMTP purge already runs on the active node - pin it, don't move it",
"evidence": "PurgeCentralOnlyNotificationConfigAsync (SiteStorageService.cs:811-821) has TWO callers: DeploymentManagerActor.cs:1921 (ACTIVE node's HandleDeployArtifacts, :1864-1963) and SiteReplicationActor.cs:456 (standby copy, dies with the actor). The purge never lapses; the original 're-home before any deletion' premise was false. Task 12 = pin test only (ArtifactStorageTests covers the storage method, not the actor call site Task 16 edits). No site writer to notification_lists/smtp_configurations since 2026-07-10 (verified: only test seeding inserts exist) + migrator skips them => permanently empty in the consolidated DB => Task 14 does NOT register them."
},
{
"id": "D4",
"subject": "native_alarm_state volume is MEASURED, not assumed",
"evidence": "scadabridge.db is not only config - native_alarm_state mirrors live A&C conditions (NativeAlarmActor.cs:504) and is the highest-volume table in either DB. sf_messages worst case ~50 row-writes/sec. Task 1 measures both AT THE LEGACY-DB SOURCE (they are not in the Phase 1 oplog - see reviewPass) and sets MaxOplogRows/MaxOplogAge arithmetically; Task 20 evidence 10 does the empirical post-cutover drain check. If growth is monotonic, STOP: keyed-instances escape hatch = ~/Desktop/scadaproj/docs/plans/2026-07-19-scadabridge-localdb-design.md:139-141 (adoption design doc, NOT the 07-17 lib doc), a scadaproj library effort that would suspend this plan."
},
{
"id": "D5",
"subject": "Cutover forecloses rolling site upgrades",
"evidence": "SiteReplicationActor retains a legacy monolithic SfBufferSnapshot handler for rolling upgrades. With no dual-mechanism period, one node would speak a protocol the other no longer implements. Both nodes of a site must be stopped and started together. Task 21 puts this in the deployment docs (installation-guide.md/topology-guide.md - no file literally named runbook)."
},
{
"id": "D6",
"subject": "NEW (review pass): 4 MB gRPC message cap replaces the 128 KB Akka frame as the size ceiling",
"evidence": "Neither side configures gRPC message sizes (ScadaBridge AddGrpc at Program.cs:519-521 sets only the auth interceptor; the lib's initiator channel is bare GrpcChannel.ForAddress) => 4 MB default receive limit both directions. Batching is row-count-only (MaxBatchSize default 500; SyncSession.cs:227, SnapshotStreamer.cs:55; no byte-aware chunking). deployed_configurations.config_json documented >128 KB/row; a few dozen such rows in one batch exceeds 4 MB and wedges the stream on a poison batch. Task 1 measures max/avg config_json bytes; Task 19 sets LocalDb:Replication:MaxBatchSize so max-row-bytes x MaxBatchSize << 4 MB; any single row near 4 MB = STOP (needs byte-aware batching or size knobs in the LocalDb lib - scadaproj effort)."
}
],
"reconFindings": [
"The gate doc named 2 test files as the specification; the real spec is 5 - it missed StoreAndForwardReplicationTests.cs (incl. the only Requeue coverage), ReplicationWireSerializationPinTests.cs, ResyncWireSerializationPinTests.cs, and SfBufferResyncPredicateTests.cs (the N1 Critical regression test).",
"sf_messages has NO version column today - ON CONFLICT(id) DO UPDATE has no comparison predicate (StoreAndForwardStorage.cs:331-345). 'Newest wins' is bare arrival order. LWW-by-HLC is an IMPROVEMENT here, not a regression.",
"No autoincrement-integer PKs exist anywhere in Phase 2 scope - all 9 config tables use natural TEXT or composite TEXT keys, sf_messages is TEXT. LocalDb RegisterReplicated SUPPORTS composite PKs (ordered pk ordinals) and rejects BLOB columns - no Phase 2 table has one (verified). Phase 1's site_events GUID conversion has no Phase 2 analogue.",
"SiteStorageService has NO foreign keys. RemoveDeployedConfigAsync (:343-376) is a manual 3-statement cascade in one transaction. Under CDC these become three independent delete streams that LWW may reorder - the most likely real defect in the plan (Task 18 scenario 2).",
"DeploymentManagerActor's replicationActor is an OPTIONAL POSITIONAL parameter at :169 (ctor :161-175; :184 is the field ASSIGNMENT). The same-typed IActorRef? optional dclManager sits immediately BEFORE it at :168 - that's the real silent-shift hazard. Props.Create passes it positionally at AkkaHostedService.cs:810. Check every call site by hand (Task 16).",
"ActiveNodeEvaluator must NOT be deleted - the S&F delivery gate still uses it (AkkaHostedService.cs:866 SetDeliveryGate -> SelfIsPrimary -> SelfIsOldestUp). Doc comment :14/:16 mentions replication; SiteReplicationActor.cs:288 calls it directly (dies with the actor).",
"ConfigFetchRetryCount's ONLY production reader is SiteReplicationActor.cs:157 (verified) - dead after Task 15; removed in Task 17 (removing it in Task 13, before the actor deletion, would not compile). IDeploymentConfigFetcher is KEPT: DeploymentManagerActor refresh path + SiteReconciliationActor + DI at ServiceCollectionExtensions.cs:84.",
"notification_lists and smtp_configurations are deliberately NOT migrated (Task 9) AND NOT registered (Task 14, corrected) - migrating or replicating them would resurrect/ship plaintext SMTP passwords; they are permanently empty by design (writers removed 2026-07-10; verified only test-seeding inserts exist).",
"REVIEW-PASS ADDITIONS: SiteReconciliationActor (runs on EVERY node at startup, best-effort self-heal vs central) is the second caller of both StoreDeployedConfigIfNewerAsync and IDeploymentConfigFetcher - it survives Phase 2 and constrains Tasks 11/13/20 (a startup fetch on the standby is legitimate; zero-fetch assertions must scope to the deploy window).",
"SiteStorageService has 21 (not 22) inline connection+OpenAsync pairs: 60,206,248,311,345,386,414,443,470,498,538,583,608,640,663,692,730,769,813,837,868. CreateConnection():51's only repository consumer is SiteExternalSystemRepository.",
"StoreAndForwardService: :39 is the _replication FIELD, :243 the ctor param; exactly 6 emission sites (:654,805,836,872,1120,1146). ServiceCollectionExtensions.cs:32 also resolves ReplicationService inside the StoreAndForwardService factory - must go in Task 14.",
"ReplicationMessages.cs holds ONLY the 10 Replicate*/Apply* records; the four SfBuffer resync records live at the bottom of SiteReplicationActor.cs (:678-707) and die with the actor file.",
"10 (not 9) appsettings.Site.json files set the legacy paths - deploy/wonder-app-vd03/appsettings.Site.json was missed (also sets ReplicationEnabled:false).",
"Rig facts: site metrics on port 8084 (AnyIP in-container, NOT published to host; 8080 is Traefik); real metric names localdb_oplog_depth / localdb_sync_* (meter ZB.MOM.WW.LocalDb.Replication); containers (aspnet:10.0) have NO sqlite3 - data dirs are host bind mounts but NEVER run sqlite3 host-side against the live files (poisons the container's WAL state, root cause of the 2026-07-20 disk-I/O-error incident): cp the .db/-wal/-shm triplet and query the copy (see the snap() helper in the plan); ReplicationOptions bind at LocalDb:Replication:* (lib binds the section).",
"Phase 1's LocalDbSitePairConvergenceTests uses ONE shared API key (:47) - it never did a mismatched-key non-vacuity run; wrong-key denial is unit-covered in Host.Tests/LocalDbSyncAuthInterceptorTests.cs. Task 18's non-vacuity check must be done directly for the new scenarios.",
"MigrateEvents synthesizes deterministic 'mig-{node}-{legacyId}' ids (NOT fresh GUIDs - crash-rerun idempotency). Migration runs AFTER RegisterReplicated in OnReady (order is load-bearing); Task 8's oplog pin test must register sf_messages on its own TestLocalDb since production doesn't register it until Task 14."
],
"tasks": [
{"id": 1, "subject": "Task 1: Rig soak - measure legacy-DB write rates + config_json sizes (corrected method)", "status": "completed", "classification": "high-risk", "note": "GATE CLOSED - verdict PROCEED (2026-07-20). NEVER sample host-side sqlite3 on live bind-mounted WAL files (that method poisoned the first run - use the copy-based snap() helper). Clean re-run: sf_messages 0.80 rows/sec insert, 0/sec retry-UPDATE, oplog 0, alarms 0, max payload_json 76 B, max config_json 721 B (NOT representative), zero SQLite errors in 30 min, both nodes converged. No stop condition met. Binding output: MaxBatchSize 500 -> 16 (Task 19). Empirical drain check remains Task 20 evidence 10."},
{"id": 2, "subject": "Task 2: Decision record - close the phase 2 gate", "status": "completed", "classification": "trivial", "blockedBy": [1], "note": "Gate doc status flipped NOT STARTED -> CLOSED; all 5 SS5 questions answered inline (questions kept, not deleted); SS2's N5 duplicate-bound requirement routed explicitly to Task 21."},
{"id": 3, "subject": "Task 3: Extract StoreAndForwardSchema.Apply", "status": "completed", "classification": "small", "blockedBy": [2], "note": "DEVIATION: kept PRAGMA journal_mode=WAL in InitializeAsync - the plan Step 4 snippet drops it but Task 4 says not to move the equivalent pragma; contradictory, and dropping it would regress documented concurrent-writer support before Task 5 makes it moot. Added a legacy-upgrade test beyond the plan's (the specified test asserts against a FRESH table where CREATE TABLE already lists all 16 columns - it would pass with every ALTER deleted). That test found the last_attempt_at_ms backfill lands 1 ms low: julianday() double day-fraction rounding, pre-existing, carried verbatim, asserted with 1 ms tolerance. Suite 154/154."},
{"id": 4, "subject": "Task 4: Extract SiteStorageSchema.Apply", "status": "completed", "classification": "small", "blockedBy": [2], "note": "DEVIATION: TryAddColumnAsync's catch-on-message-text ('duplicate column') became a PRAGMA table_info probe matching OperationTrackingSchema - the message form depends on a non-contractual error string and swallowed unrelated SqliteExceptions; also dropped the ILogger dep, which is what let the class be static. Cost: the per-column 'Migrated: added column' info log is gone (nothing consumes it). PRAGMA journal_mode=WAL stays in InitializeAsync per plan. Added a legacy-upgrade test for the same reason as Task 3. SiteRuntime 532/532, full solution build 0 warnings."},
{"id": 5, "subject": "Task 5: Rewire StoreAndForwardStorage onto ILocalDb", "status": "completed", "classification": "high-risk", "blockedBy": [3], "note": "Test fallout was ~7x the plan estimate: 40 files across 7 test projects, not \"fixtures\" in one. Most used Mode=Memory;Cache=Shared and LocalDb has NO in-memory mode, so all moved to real temp files. Added shared tests/ZB.MOM.WW.ScadaBridge.TestSupport lib (TestLocalDb) instead of copying the Phase 1 fixture into 7 projects. WAL test retargeted to the LocalDb-backed store; directory-creation test moved to Host.Tests (different owner - see Task 6 note)."},
{"id": 6, "subject": "Task 6: Rewire SiteStorageService onto ILocalDb", "status": "completed", "classification": "high-risk", "blockedBy": [4], "note": "LATENT PHASE-1 DEFECT FOUND+FIXED: LocalDb does NOT create the parent directory and SqliteLocalDb opens the file eagerly, so a missing dir is a HARD BOOT FAILURE (SQLite Error 14). Default site config is the RELATIVE ./data/site-localdb.db; docker escapes only because the volume mount creates /app/data. The plan wrongly assumed dir-creation moved to LocalDb with file ownership. Fixed via SiteLocalDbDirectory.Ensure(config) before AddZbLocalDb + Host.Tests/SiteLocalDbDirectoryTests (non-vacuity observed: 2 tests failed with exactly Error 14 pre-fix). ALSO a contract change: SiteStorageService.CreateConnection() used to return an UNOPENED connection; it now returns an ALREADY-OPEN one - SiteExternalSystemRepository dropped 5 OpenAsync calls. AddSiteRuntime(string) overload deleted."},
{"id": 7, "subject": "Task 7: Extend SiteLocalDbSetup with the new DDL (not yet registered)", "status": "completed", "classification": "standard", "blockedBy": [5, 6], "commit": "f8aa02e2", "note": "As planned. Two pins in SiteLocalDbWiringTests through the REAL composition root: all 12 tables exist, and ReplicatedTables is EXACTLY the Phase 1 pair (an equality check, not Contains - the 'not yet' is the assertion that matters). Task 14 should INVERT that second test, not delete it. Non-vacuity verified by removing the DDL."},
{"id": 8, "subject": "Task 8: Extend the legacy migrator for sf_messages", "status": "completed", "classification": "high-risk", "blockedBy": [7], "commit": "bdc0dffe", "note": "DEVIATION: the copy INTERSECTS the legacy column set with the current one instead of naming all 16 columns. A legacy file from an older build lacks execution_id/parent_execution_id/last_attempt_at_ms, and naming a missing column throws 'no such column' - which the existing ReadAll treats as an unrecognised shape and SILENTLY DISCARDS EVERY ROW. For undelivered messages that is real data loss. A required-column (PK) guard stops the tolerance degrading into NULL-keyed copies. Tests register sf_messages on their own harness (production OnReady does not until Task 14), so Task 14 MUST keep Migrate as the LAST call in OnReady, after all registrations. Non-vacuity verified: all 4 fail without the Migrate call."},
{"id": 9, "subject": "Task 9: Extend the legacy migrator for the config tables (skip notification/smtp)", "status": "completed", "classification": "high-risk", "blockedBy": [7], "commit": "5ddc7eed", "note": "7 of 9 tables migrated; notification_lists + smtp_configurations skipped as planned (plaintext passwords must not enter a table Task 14 makes replicated) - the skip is pinned by a test that also greps __localdb_oplog.row_json for the secret. Task 8 MigrateTable generalized to MigrateFile(many tables, one transaction, one rename). ADDED beyond the plan: MigratorColumnLists_MatchTheLiveSchema, a column-parity test vs SiteStorageSchema - the plan called for a manual column check, but a mismatch is invisible at runtime in BOTH directions (a typo is silently dropped by the intersection; a missing column silently leaves data behind). SiteStorageTables + LegacyTable are internal so the test can read them. Non-vacuity verified twice: removing the call (3 fail) and wrongly adding notification/smtp to the map (skip test fails)."},
{"id": 10, "subject": "Task 10: Port the S&F replication test intents as CDC specs", "status": "completed", "classification": "standard", "blockedBy": [8], "commit": "2bbe6631", "note": "DEVIATION: extracted the Phase 1 fixture into LocalDbSitePairHarness (abstract base) instead of duplicating ~150 lines; Phase 1 tests derive from it and pass unchanged. The harness registers the 8 Phase 2 tables itself (production OnReady does not until Task 14) - DELETE RegisterPhase2TablesUntilCutover at Task 14. The 8-table list is literal, not derived from production code, so a cutover registering the wrong set fails these tests. Non-vacuity: unregistering sf_messages failed 6 of 7 - the 7th (add-then-remove ordering) PASSED because an absent row is also what a non-replicating pair looks like; fixed with a control row that must converge in the same window."},
{"id": 11, "subject": "Task 11: Port the resync + directional-authority tests", "status": "completed", "classification": "standard", "blockedBy": [9], "note": "N1 re-expressed per D2 as a no-rows-lost property (LWW merge, never deletes), not directional authority. DEVIATION on the zero-fetch assertion: the plan wanted a fetcher double recording zero calls, but this harness has no actor system / no central / no IDeploymentConfigFetcher in the graph, so the double could not fail either way. Replaced with the positive half (config reaches B over replication alone) plus an in-file comment recording that the negative half is proved by Task 15 deleting the code and the build passing. D1 scope note recorded in-file. Non-vacuity: unregistering deployed_configurations fails all 4.", "commit": "c56bf4ae"},
{"id": 12, "subject": "Task 12: Pin the active-node SMTP purge (corrected: already exists at DeploymentManagerActor.cs:1921)", "status": "completed", "classification": "standard", "blockedBy": [10, 11], "note": "As planned - NO production change; D3 confirmed correct, the call already exists at DeploymentManagerActor.cs:1921. Pin verified RED-FIRST by commenting out the call. Test needed a using for Commons.Messages.Artifacts and polls (the apply runs on a Task.Run inside the actor).", "commit": "79ce5161"},
{"id": 13, "subject": "Task 13: Notify-and-fetch scope check - guarded write STAYS for SiteReconciliationActor", "status": "completed", "classification": "standard", "blockedBy": [12], "note": "Doc-comment only, as planned. Re-verified both callers: SiteReplicationActor:375 (dies Task 15) + SiteReconciliationActor:166 (survives). Step 2 scope check re-run: ConfigFetchRetryCount's only production reader is still SiteReplicationActor:157, so option + validator rule stay until Task 17. Also fixed a stale 'guarded standby write' header in SiteStorageServiceTests.", "commit": "79ce5161"},
{"id": 14, "subject": "Task 14: Register the 8 Phase 2 tables and delete ReplicationService", "status": "completed", "classification": "high-risk", "blockedBy": [13], "note": "LANDED AS ONE COMMIT with tasks 15+16 - PLAN DEFECT: they cannot compile separately (SiteReplicationActor takes a ReplicationService + calls ReplaceAllAsync; DeploymentManagerActor Tells ReplicationMessages types; AkkaHostedService constructs the actor). Combining also strengthens Task 14's own invariant: never both mechanisms, never neither. Registered 8 tables; notification_lists + smtp_configurations deliberately NOT registered. Deleted ReplaceAllAsync as UNSAFE (a mass DELETE on a replicated table would be captured and shipped), and its test replaced by a comment explaining that. The positional-arg hazard was REAL: 4 DeploymentManagerActor test call sites bound wrong args; converted to named where possible - Props.Create is an expression tree and rejects OUT-OF-POSITION named args, so the rest are padded positionally. Task 7's 'not yet' test INVERTED (exact in both directions) + new security-named test for the SMTP tables + composite-PK test. Harness's temporary registration deleted, so the convergence suites now prove the cutover. My expected table list was mis-sorted: ordinal puts '_' (0x5F) before 'b', so data_connection_definitions precedes database_connections.", "commit": "037798b3"},
{"id": 15, "subject": "Task 15: Delete SiteReplicationActor and its messages", "status": "completed", "classification": "high-risk", "blockedBy": [14], "note": "LANDED AS ONE COMMIT with tasks 15+16 - PLAN DEFECT: they cannot compile separately (SiteReplicationActor takes a ReplicationService + calls ReplaceAllAsync; DeploymentManagerActor Tells ReplicationMessages types; AkkaHostedService constructs the actor). Combining also strengthens Task 14's own invariant: never both mechanisms, never neither. Registered 8 tables; notification_lists + smtp_configurations deliberately NOT registered. Deleted ReplaceAllAsync as UNSAFE (a mass DELETE on a replicated table would be captured and shipped), and its test replaced by a comment explaining that. The positional-arg hazard was REAL: 4 DeploymentManagerActor test call sites bound wrong args; converted to named where possible - Props.Create is an expression tree and rejects OUT-OF-POSITION named args, so the rest are padded positionally. Task 7's 'not yet' test INVERTED (exact in both directions) + new security-named test for the SMTP tables + composite-PK test. Harness's temporary registration deleted, so the convergence suites now prove the cutover. My expected table list was mis-sorted: ordinal puts '_' (0x5F) before 'b', so data_connection_definitions precedes database_connections.", "commit": "037798b3"},
{"id": 16, "subject": "Task 16: Clean up DeploymentManagerActor and AkkaHostedService", "status": "completed", "classification": "standard", "blockedBy": [15], "note": "LANDED AS ONE COMMIT with tasks 15+16 - PLAN DEFECT: they cannot compile separately (SiteReplicationActor takes a ReplicationService + calls ReplaceAllAsync; DeploymentManagerActor Tells ReplicationMessages types; AkkaHostedService constructs the actor). Combining also strengthens Task 14's own invariant: never both mechanisms, never neither. Registered 8 tables; notification_lists + smtp_configurations deliberately NOT registered. Deleted ReplaceAllAsync as UNSAFE (a mass DELETE on a replicated table would be captured and shipped), and its test replaced by a comment explaining that. The positional-arg hazard was REAL: 4 DeploymentManagerActor test call sites bound wrong args; converted to named where possible - Props.Create is an expression tree and rejects OUT-OF-POSITION named args, so the rest are padded positionally. Task 7's 'not yet' test INVERTED (exact in both directions) + new security-named test for the SMTP tables + composite-PK test. Harness's temporary registration deleted, so the convergence suites now prove the cutover. My expected table list was mis-sorted: ordinal puts '_' (0x5F) before 'b', so data_connection_definitions precedes database_connections.", "commit": "037798b3"},
{"id": 17, "subject": "Task 17: Config-key cleanup (incl. ConfigFetchRetryCount; 10 appsettings files)", "status": "completed", "classification": "standard", "blockedBy": [16], "note": "As planned. DEVIATION: deploy/wonder-app-vd03/appsettings.Site.json sits under a GITIGNORED deploy/ tree, so its edit is local-only and must be repeated on the box at deploy time. Comment style is // (JSONC) rather than \"_comment_\" keys: every one of these files already contains // comments and .NET's json config reader accepts them, and a _comment_ key inside a bound section is a phantom config entry. Both relaxations pinned by the INVERSE of the test they replace (Site_MissingSiteDbPath_IsAccepted..., EmptySqliteDbPath_IsAccepted...), each verified red with the old rule restored. DatabaseOptionsValidator needed no change - it was already null-tolerant/blank-rejecting, which is exactly migration-only semantics.", "commit": "605e5682"},
{"id": 18, "subject": "Task 18: Two-node convergence suite for the Phase 2 tables", "status": "completed", "classification": "high-risk", "blockedBy": [17], "note": "DEVIATION: landed as a NEW file LocalDbPhase2ConvergenceTests.cs rather than extending Phase 1's LocalDbSitePairConvergenceTests.cs, and drives the REAL SiteStorageService instead of hand-written SQL - possible only post-cutover, and what makes the cascade scenario test the shipped transaction rather than a re-creation of it. Scenario 4 was retargeted onto shared_scripts/external_systems/static_attribute_overrides because the plan's version overlapped LocalDbConfigConvergenceTests' N1 scenario almost exactly; the union-survives property is per-table, so re-proving it on untouched tables is the non-redundant half. Cascade test carries a never-removed control instance (absence assertions otherwise cannot distinguish 'cascade converged' from 'node B lost these tables'). Non-vacuity PROVEN as mandated: with the 8 RegisterReplicated calls commented out, 4 failed / 0 passed; restored, 20/20 across the three LocalDb suites.", "commit": "15013156"},
{"id": 19, "subject": "Task 19: Rig configuration (MaxOplogRows/MaxOplogAge + MaxBatchSize per D6)", "status": "completed", "classification": "small", "blockedBy": [17], "note": "MaxBatchSize 500->16 (D6: row-count batching x ~70 KB production config_json vs the 4 MB gRPC cap; 16 => ~1.1 MB worst case). MaxOplogRows 1M->250,000 and MaxOplogAge 7d->2d from the soak's 0.80 rows/sec (~69k/day). Tighter-than-default is SAFE because a cap breach prunes + sets needs_snapshot (graceful snapshot resync), not data loss - the Task 1 finding that the plan's stop condition was weaker than written. site-b/site-c left unreplicated so default-OFF stays proven side by side.", "commit": "921edab4"},
{"id": 20, "subject": "Task 20: Live gate on the docker rig", "status": "completed", "classification": "high-risk", "blockedBy": [18, 19], "note": "ALL 10 CHECKS PASS. Evidence: docs/plans/2026-07-19-localdb-phase2-live-gate.md. Key blocker found and fixed mid-run: external systems reach a site ONLY via ArtifactDeploymentService, which `instance deploy` never invokes - `deploy artifacts` was needed both to deliver the probe harness and to propagate the owed ExternalSystemDefinitions restore. THREE METHOD CORRECTIONS to the plan: (1) its instruction to run DB checks host-side against the bind mounts is UNSAFE - host sqlite3 poisons the container WAL; copy the db/-wal/-shm triplet and query the copy. (2) `docker exec ... curl` cannot scrape metrics (no curl in aspnet:10.0) and with 2>/dev/null the failure is silent - it nearly became a false 'metrics missing' finding; use a network-sharing curl sidecar. (3) checks needing S&F load need CachedCall, not Call. CAVEATS recorded not glossed: check 2's zero-count is vacuous alone (legacy source was also empty) and rests on the ABSENCE of CDC triggers; check 7's native_alarm_state leg was empty live and is covered only offline; check 10's sampling is coarse and the real rise/drain evidence comes from check 6's 0->4->0.", "commit": "158e79bb"},
{"id": 21, "subject": "Task 21: Documentation truth pass", "status": "completed", "classification": "standard", "blockedBy": [20], "note": "Both CLAUDE.md files + Component-StoreAndForward.md:83 (normative resync paragraph rewritten for CDC, stating the duplicate-delivery bound explicitly: limited to messages the OLD primary delivered whose status change had not yet replicated when the gate flipped - one flush interval plus in-flight ack, and it does NOT grow with backlog depth or absence duration) + components/{StoreAndForward,SiteRuntime,Host}.md + the frame-size known-issue (amended: Phase 2 deleted notify-and-fetch itself, so the 128KB Akka frame constraint is gone from the intra-site hop entirely; successor ceiling is the 4MB gRPC cap via MaxBatchSize, and note the failure mode differs - oversized gRPC is REJECTED, not silently dropped) + deployment topology-guide.md and installation-guide.md (D5 stop-both-together, D2 TombstoneRetention resurrection bound). DoD closed: build 0 warnings, all 10 suites green (3509 tests, 0 failures). DoD grep nuance recorded in the plan: 4 matches remain in src/ and are all deliberate COMMENT prose explaining what was deleted - a literal 'no matches' would delete the explanations that stop someone re-introducing the old design.", "commit": null}
],
"knownFlakes": [
{
"test": "SiteRuntime.Tests InstanceActorChildAttributeRaceTests.ChildActors_AreSeededFromAnIsolatedCopy_NotTheLiveAttributesDictionary",
"note": "Intermittent ActorNotFoundException under full-suite load; passes in isolation. Pre-existing, carried over from Phase 1."
},
{
"test": "AuditLog.Tests ParentExecutionIdCorrelationTests.InboundRoutedRun_AllRoutedRows_CarryInboundExecutionId_AsParentExecutionId",
"note": "Cold-MSSQL-fixture timing: ~91s and AwaitAssert-times-out cold, ~1s warm. Re-run before investigating."
}
],
"lastUpdated": "2026-07-20",
"phase2Status": "UNBLOCKED - Task 1 gate CLOSED (verdict PROCEED) and Task 2 DONE, 2026-07-20. Task 1's original STOP verdict is SUPERSEDED: the 'Phase 1 disk I/O defect' was OBSERVER-INDUCED (host-side sqlite3 against live bind-mounted WAL files resets the WAL across virtiofs and permanently poisons the container's connections) - NOT a product defect. See docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md. Both earlier isolation claims were confounded: one sampling pass had already poisoned BOTH nodes, and a poisoned standby looks healthy only because it issues almost no statements; 'LocalDb-specific' was sampling-selection bias (only the LocalDb file had ever been host-read). Clean re-run 2026-07-20 with the copy-based snap() helper on restarted nodes, 6 consecutive 60s intervals: sf_messages 48 rows/min = 0.80 rows/sec dead steady, retry-UPDATE rate 0/sec (SUM(retry_count) flat at 200), oplog 0, native_alarm_state 0, max payload_json 76 B, max config_json 721 B, ZERO SQLite errors across 30 min, both site-a nodes converged at 3564 rows. Honest gap: 0.80/s is ~1.6% of the 50/s ceiling and the retry-UPDATE path was never exercised - acceptable because that ceiling is structural (SweepBatchLimit/RetryTimerInterval), not empirical. Rig config rows (721 B) are NOT representative; D6 sizing rests on the documented ~60-70 KB production config_json. Plan-premise corrections stand: D6 (MaxBatchSize 500 -> 16, the one firmly evidence-backed number; Task 19 sets it), D4 (alarm writes bounded by per-SourceReference coalescing at a 100 ms flush, NOT unbounded), sf_messages hard ceiling 50 rows/sec, oplog cap overrun = graceful snapshot resync (needs_snapshot), not data loss. NEXT: Wave 1 = Tasks 3 + 4, dispatchable in parallel (disjoint Files blocks). Tasks 3-21 untouched; no plan code written yet. Rig cleanup still owed before Task 20: restore ExternalSystemDefinitions id 1 to http://scadabridge-restapi:5200 (currently http://127.0.0.1:9), and remove SoakGenerator template 2021 + instances soakgen-1..4 (ids 5-8), still deployed and generating load."
}
docs/plans/2026-07-19-localdb-phase2-gate.md
+120 -3
View File
@@ -1,8 +1,11 @@
# LocalDb Phase 2 — Gate Document # LocalDb Phase 2 — Gate Document
> **Status: NOT STARTED.** This is the gate, not the plan. Phase 2 work must not begin > **Status: CLOSED (2026-07-20).** The implementation plan is
> until an implementation plan is written against the facts below and the open questions > [`2026-07-19-localdb-adoption-phase2.md`](2026-07-19-localdb-adoption-phase2.md); its
> in §5 are answered. > decisions D1–D6 answer this gate and its Task 1 supplied the measurements §5 demanded
> (soak record: [`2026-07-19-localdb-phase2-soak.md`](2026-07-19-localdb-phase2-soak.md)).
> Each §5 question is answered inline below. **The questions are deliberately not deleted —
> the reasoning is the value, and several of the answers overturn a premise stated above.**
**Phase 1 live gate: PASS** (2026-07-19, docker rig). See the Task 12 commit for evidence. **Phase 1 live gate: PASS** (2026-07-19, docker rig). See the Task 12 commit for evidence.
@@ -50,6 +53,11 @@ Known accepted behaviour to carry forward, not silently drop: the **N5 bounded-d
race** the bespoke replicator documents and accepts. A Phase 2 plan must state whether CDC race** the bespoke replicator documents and accepts. A Phase 2 plan must state whether CDC
inherits the same bound, a tighter one, or a different failure shape. inherits the same bound, a tighter one, or a different failure shape.
> **Routed (2026-07-20).** The failure shape changes, so N5's bound does not simply carry
> over: CDC has no chunked anti-entropy hop to duplicate across, but LWW admits re-delivery
> of a row whose park loses to a later write (see the LWW answer in §5). **Task 21 states the
> new duplicate-delivery bound explicitly** and rewrites the N5 note rather than deleting it.
## 3. Substantially harder than Phase 1 ## 3. Substantially harder than Phase 1
Phase 1 moved two tables that **nothing replicated before**, so the worst case was "no Phase 1 moved two tables that **nothing replicated before**, so the worst case was "no
@@ -84,16 +92,125 @@ class:
thresholds cannot be chosen from first principles. *This needs a Phase-1 rig soak that thresholds cannot be chosen from first principles. *This needs a Phase-1 rig soak that
has not been run.* The live gate proved correctness, not steady-state behaviour over has not been run.* The live gate proved correctness, not steady-state behaviour over
time. time.
**ANSWERED — the soak ran (Task 1); the caps are not the binding constraint, batch
*bytes* are.** Two of this question's own premises turned out to be wrong, and both
corrections relax it:
1. **The stop condition was much weaker than written.** Exceeding `MaxOplogRows` /
`MaxOplogAge` does not wedge or lose data — `OplogStore` prunes and flags the peer
`needs_snapshot` (`OplogStore.cs:109-138`, `MaintenanceBackgroundService.cs:57`), which
degrades to a **graceful snapshot resync**. Overrun is a performance event, not a
correctness event, so these caps do not need to be sized defensively.
2. **`sf_messages` has a hard ceiling, not an estimate.** `SweepBatchLimit` (500) ÷
`RetryTimerInterval` (10 s) = **≤50 row-writes/sec**, structurally. Measured rig rate
under the purpose-built `SoakGenerator` load was far below that (see the soak record).
Rows are small — max `payload_json` **76 bytes** on the rig.
`native_alarm_state` is answered under the last question in this section, not here (and
the plan's D4, whose premise it corrects). The **real** ceiling this
question was groping toward is D6's 4 MB gRPC message cap, and it binds on
`MaxBatchSize × max-row-bytes`, not on oplog depth — **Task 19 sets
`LocalDb:Replication:MaxBatchSize = 16`**, the one firmly evidence-backed number the soak
produced. The keyed-instances escape hatch is **not** needed; this plan proceeds.
Independently, the soak retired a scare: a `disk I/O error` storm initially read as a
Phase 1 library defect and STOP-gated this plan was root-caused as **observer-induced**
(host-side `sqlite3` against live bind-mounted WAL files) — see
[`docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md`](../known-issues/2026-07-20-localdb-disk-io-error-under-load.md)
§0. The unmodified library sustains the full soak load with **zero** SQLite errors.
- **LWW semantics for in-flight store-and-forward sends.** What does last-writer-wins do - **LWW semantics for in-flight store-and-forward sends.** What does last-writer-wins do
when one node parks a message the other is mid-delivery on? when one node parks a message the other is mid-delivery on?
**ANSWERED — the scenario cannot arise on a correctly-behaving pair, and D2 covers what
happens if it does.** Only the **active** node sweeps and delivers; the standby holds a
convergent copy and sends nothing. So "one node mid-delivery while the other parks" is
a split-brain symptom, not steady-state behaviour, and store-and-forward is not where it
should be defended against.
If it does happen, LWW resolves per row by HLC: the later write wins, and the losing
node's view converges to it. The concrete risk is **at-least-once delivery** — a park
that loses to a stale in-flight update can be re-swept and re-sent. That is not new;
store-and-forward is already at-least-once by construction (a send that succeeds but
whose ack is lost is retried). What **is** new is D2's semantic change, recorded here
because it is the honest cost of this gate: the standby is no longer guaranteed
**byte-identical** to the active buffer, only **convergent**. The bespoke replicator's
`ReplaceAllAsync` bought identity by wiping — and the N1 directional guard existed
precisely because that wipe was dangerous. The library's snapshot resync **merges per
row and never wipes** (`SnapshotApplier`, `LwwApplier.cs:69-78`), so the failure the
guard prevented is structurally impossible rather than merely tested against. Task 10
ports `SfBufferResyncPredicateTests` as a **convergence** assertion, not a
directional-authority one.
- **Migration-under-load.** How does the one-time copy of an actively-replicating - **Migration-under-load.** How does the one-time copy of an actively-replicating
`scadabridge.db` interact with the bespoke replicator still running during cutover? `scadabridge.db` interact with the bespoke replicator still running during cutover?
**ANSWERED — the interaction is designed out, not managed.** The two mechanisms never run
concurrently. `SiteReplicationActor` and StoreAndForward's `ReplicationService` are
**deleted in the same commit** that registers the Phase 2 tables (Tasks 14/15), and D5
forecloses rolling upgrades: **both nodes of a site stop and start together**. A process
that boots with the new code has no bespoke replicator to race, and one running the old
code has no CDC triggers. There is no window in which a row is written by one mechanism
and read by the other.
Within a single booting process the ordering is the load-bearing part, and it is the same
invariant Phase 1 established: **DDL → `RegisterReplicated` → migrate → writes**
(`SiteLocalDbSetup.OnReady`). Rows written *before* registration are invisible to the peer
forever, silently — which is why Tasks 8/9 run the migrator strictly **after**
registration, so every migrated row is captured by the CDC triggers and replicates
normally. Both nodes migrating independently is fine: they migrate the same source rows,
and LWW converges them.
- **Rollback story.** With no dual-mechanism period, what is the recovery path if the - **Rollback story.** With no dual-mechanism period, what is the recovery path if the
cutover fails in production — beyond "revert the commit"? cutover fails in production — beyond "revert the commit"?
**ANSWERED — "revert the commit" is genuinely the path, and it is safe because the
migration is additive and the legacy files are left intact.** Tasks 8/9 **copy** rows out
of `scadabridge.db` and `store-and-forward.db` into the consolidated LocalDb file; they do
not drop, truncate, or delete the source databases. Rolling back is therefore: stop both
nodes of the site, deploy the previous build, start both together (D5). The old code
reopens the legacy files and finds them exactly as it left them.
The bounded, honest cost of a rollback is **the delta** — rows written into the
consolidated DB after cutover do not flow back into the legacy files. For config tables
that self-heals: `SiteReconciliationActor` reports local inventory to central at startup
and fetches whatever it lacks, so a rolled-back node re-converges to central's truth
without operator action. For `sf_messages` the delta is **lost undelivered buffer** — the
practical mitigation is to drain the buffer before cutting over, which Task 20's live gate
and Task 21's runbook both call for.
What has **no** rollback is a site pair split across versions — hence D5. That is a
deployment-procedure constraint, and Task 21 puts it in the runbook rather than leaving it
as tribal knowledge.
- **Does the consolidated file stay appropriate?** One-DB-per-process was chosen partly - **Does the consolidated file stay appropriate?** One-DB-per-process was chosen partly
because Phase 1's tables were small. Adding config + S&F changes the size and write because Phase 1's tables were small. Adding config + S&F changes the size and write
profile of that single file. profile of that single file.
**ANSWERED — yes, and the soak is the evidence.** The write profile the consolidated file
must absorb is bounded on every axis:
| Table | Rate bound | Row size | Basis |
|---|---|---|---|
| `sf_messages` | **≤50 writes/sec** (hard) | 76 B max on the rig | `SweepBatchLimit` ÷ `RetryTimerInterval` |
| `native_alarm_state` | `distinct_source_refs × 10/sec` | small | per-`SourceReference` coalescing on a 100 ms flush (`NativeAlarmActor.cs:473-502`) |
| config tables | deploy-driven, effectively idle | ~721 B max on the rig | Task 1 measurement |
These are unremarkable for SQLite in WAL mode, and the soak ran the full generator load
against the Phase 1 consolidated file for 30 minutes with **zero** SQLite errors and no
oplog growth pathology. **One DB per process stays.**
Two caveats carried into the plan rather than hidden here. First, the rig's config rows
are tiny (max 721 B) and **cannot** be treated as representative — the size ceiling that
matters comes from the documented ~60–70 KB production `config_json`, which is what
motivates D6 and `MaxBatchSize = 16`. Second, **D4's premise was wrong**:
`native_alarm_state` is *not* "unbounded by design" and *not* "by a wide margin the
highest-volume table" — the coalescing flush bounds it, and this rig has no alarm
generator at all (measured 0 rows), so its bound is analytic rather than observed. If a
production site ever shows alarm churn that swamps the shared oplog, the keyed-instances
hatch named in D4 remains the escape — it is simply not needed to start.
## 6. Not in scope (unchanged from the design) ## 6. Not in scope (unchanged from the design)
`auditlog.db` (diverges per node by design — central pulls the union; replicating it would `auditlog.db` (diverges per node by design — central pulls the union; replicating it would
docs/plans/2026-07-19-localdb-phase2-live-gate.md
+209
View File
@@ -0,0 +1,209 @@
# LocalDb Phase 2 — live gate evidence (2026-07-20)
Gate for [Task 20](2026-07-19-localdb-adoption-phase2.md). **Status: all 10 checks captured and
passing**, with two scope caveats recorded below (check 2's count is vacuous on its own; check 7's
third table was empty).
Build under test: `feat/localdb-phase2` @ `166f07fa`, LocalDb `0.1.1` — confirmed live, not
assumed: the metrics scrape reports `otel_scope_name="ZB.MOM.WW.LocalDb.Replication"
otel_scope_version="0.1.1"`. Rig redeployed with `docker/deploy.sh`, exit 0, all 8 nodes up.
## Method notes — two corrections to the plan
**1. Do NOT run DB checks host-side against the bind mounts, as Task 20 instructs.** macOS↔container
locks do not cross virtiofs, so a host `sqlite3` open recovers the WAL out from under the container
and triggers a permanent `disk I/O error` (SQLITE_IOERR_SHORT_READ 522) storm — the root cause of
the 2026-07-20 incident. Every read below used a helper that copies the `db`/`-wal`/`-shm` triplet
and queries the **copy**; `cp` is a pure byte read with no SQLite involvement.
**2. `docker exec … curl` cannot scrape the metrics** — the `aspnet:10.0` image has no `curl`, and
with `2>/dev/null` the failure is silent and looks exactly like "no metrics exported". An earlier
pass nearly recorded that as a finding. Use a network-sharing sidecar:
```bash
docker run --rm --network container:scadabridge-site-a-a curlimages/curl:latest -s localhost:8084/metrics
```
**3. The probe harness must be delivered by `deploy artifacts`, not `instance deploy`.** External
systems reach a site only through `ArtifactDeploymentService` (`FetchGlobalArtifactsAsync` →
`GetAllExternalSystemsAsync`), which the per-instance deploy path does not invoke. A script that
names an external system that never arrived fails silently — no rows, no error.
---
## 1. Migration ran — PASS
| Node | `.migrated` markers | legacy `sf_messages` | consolidated `sf_messages` |
|---|---|---|---|
| site-a-node-a | `scadabridge.db.migrated`, `store-and-forward.db.migrated` | 11804 | 11804 |
| site-a-node-b | both | 11804 | 11804 |
Both nodes migrated their own legacy files independently and the identical-content rows then
LWW-converged — expected per the plan, not an anomaly.
**Caveat:** the migration itself ran on an earlier deploy of this build (markers dated 05:04), so
this is after-the-fact evidence. `deployed_configurations` was 0 in both legacy files, so that
table's migration path is untested here; check 3 exercises it fresh instead.
## 2. No SMTP/notification rows migrated — PASS
`smtp_configurations` and `notification_lists` are 0 on both nodes.
**That count alone is vacuous** — the legacy source also held 0 SMTP rows, so a broken exclusion
would look identical. The non-vacuous evidence is structural: CDC capture triggers exist for
exactly these ten tables on both nodes —
```
OperationTracking, data_connection_definitions, database_connections,
deployed_configurations, external_systems, native_alarm_state,
sf_messages, shared_scripts, site_events, static_attribute_overrides
```
— and `smtp_configurations` / `notification_lists` have **no triggers at all**, so no replication
channel exists for them regardless of content. That is the property the security decision rests on.
## 3. Config converges byte-identical — PASS
Deployed `gate-sensor-1` (template 3) to site-a. Both nodes:
- whole-row md5 identical: `0ec0f3cf790bfd65c45edf6c4afae50f`
- `deployment_id` `20b81607…`, `revision_hash` `sha256:253e1a56…`,
`deployed_at` `2026-07-20T09:06:11.2087221+00:00`
- `__localdb_row_version`: HLC `116951506695487488`, origin node
`cf61c688-a57f-4c5c-ba76-feff6e5679fe`, `is_tombstone` 0 — **identical on both**
Identical HLC *and* origin id is the strong form: node B holds the row A wrote, not an
independently-derived copy of it.
## 4. The standby made no config fetch — PASS
Deploy window from `2026-07-20T09:05:52Z`:
- `site-a-a` (active, deploy target): **1** fetch — `Fetching config for deployment 20b81607…
(notify-and-fetch)`. Legitimate: the active node pulling its own artifact.
- `site-a-b` (standby): **0** fetches.
Under the old architecture `SiteReplicationActor` would have notified the standby and it would have
fetched independently. It now receives the config purely by CDC — corroborated from the other
direction by check 3's shared origin id.
## 5. Store-and-forward converges — PASS
Harness: external system `GateDeadTarget` → `http://127.0.0.1:9` with method `Ping`, plus an
interval script calling `ExternalSystem.CachedCall` (only `CachedCall` buffers; plain
`ExternalSystem.Call` does not). Delivered via `deploy artifacts`.
- Buffered rows appear on **both** nodes with an identical rowset md5 `a484e369f6e0f00e7debff9b64464566`.
- `__localdb_row_version` for `sf_messages`: 2509 rows from **1 origin** on both nodes.
- The Pending→Parked transition of the pre-existing backlog also replicated identically
(10280 Pending / 1524 Parked on both), so **UPDATE** replicates, not just INSERT.
## 6. Site failover — PASS
Stopped the active node `site-a-a` at 09:18:37. Ten seconds later `site-a-b` logged
`InstanceActor started for gate-churn-1` and `ScriptActor GateBufferProbe started` — the standby
picked up the workload and kept buffering. While partitioned, node B's oplog rose to **4** unacked
entries.
Restarted `site-a-a` at 09:18:57. After rejoin:
| | node-a | node-b |
|---|---|---|
| probe rows | 48 | 48 |
| total `sf_messages` | 11852 | 11852 |
| oplog | 0 | 0 |
| dead letters | 0 | 0 |
| duplicate ids | 0 | 0 |
| rowset md5 | `54ded633c28134222bf034f3d0cec680` | `54ded633c28134222bf034f3d0cec680` |
Buffered messages survived the flip, the backlog drained on rejoin, and **zero duplicate ids** on
either node is the exactly-once evidence.
## 7. Cascade delete — PASS
Seeded `gate-sensor-1` with a `static_attribute_overrides` row (via an `Instance.SetAttribute`
script — the attribute-override deploy path writes into the config JSON, not this table), confirmed
it replicated, then deleted the instance.
| | pre-delete | post-delete |
|---|---|---|
| `deployed_configurations` | 1 / 1 | 0 / 0 |
| `static_attribute_overrides` | 1 / 1 | 0 / 0 |
| `native_alarm_state` | 0 / 0 | 0 / 0 |
Crucially, **both nodes hold explicit tombstones** — `deployed_configurations=1`,
`static_attribute_overrides=1` in `__localdb_row_version WHERE is_tombstone=1`. The rows are not
merely absent on B; B applied the deletes.
**Caveat:** `native_alarm_state` was empty for this instance, so the third leg of the cascade is
untested live. It is covered offline by
`LocalDbPhase2ConvergenceTests.RemovingAnInstance_ConvergesAllThreeCascadeTables`, which was
verified non-vacuous.
## 8. Dead letters, oplog, metrics — PASS
- `__localdb_oplog` = 0 and `__localdb_dead_letter` = 0 on both nodes at every sampling point.
- `localdb_oplog_depth` 0; `localdb_sync_applied_total` 11808 (a) / 224 (b);
`localdb_sync_reconnects_total` 1.
## 9. Both nodes stopped and started together (D5) — PASS
Stopped `site-a-a` and `site-a-b` together, started them together.
| | node-a | node-b |
|---|---|---|
| `sf_messages` | 11845 | 11845 |
| probe rows | 41 | 41 |
| oplog | 0 | 0 |
| dead letters | 0 | 0 |
Zero occurrences of `disk I/O error` / `database disk image` / `SQLITE_IOERR` / `corrupt` in either
node's log after restart. Clean rejoin.
## 10. Drain under churn — PASS
Sampled every 8 s under sustained probe load:
```
t=8s oplog_a=0 oplog_b=0 probe_a=23 probe_b=23
t=16s oplog_a=0 oplog_b=0 probe_a=25 probe_b=25
t=24s oplog_a=0 oplog_b=0 probe_a=26 probe_b=26
t=32s oplog_a=0 oplog_b=0 probe_a=28 probe_b=28
t=40s oplog_a=0 oplog_b=0 probe_a=30 probe_b=30
t=48s oplog_a=0 oplog_b=0 probe_a=31 probe_b=31
t=56s oplog_a=0 oplog_b=0 probe_a=33 probe_b=33
t=64s oplog_a=0 oplog_b=0 probe_a=35 probe_b=35
```
Writes climb steadily and the two nodes stay in lockstep at **every** sample, with no oplog
accumulation — flush plus ack outpaces the write rate, so no backlog forms. The stop condition the
plan cared about (monotonic growth that never drains) does not occur.
**On its own this sampling is weak** — an 8 s interval is far coarser than the flush interval, so a
transient non-zero depth would be missed, and "always 0" is also what a dead pump looks like. The
rise-and-drain is proven instead by check 6, where the oplog demonstrably went 0 → 4 while the peer
was down and back to 0 after rejoin.
---
## Observations outside the gate
- **A central external-system delete does not remove the row from sites.** After deleting
`GateDeadTarget` centrally and re-running `deploy artifacts`, both site nodes still carry it.
Artifact application is an upsert with no reconciliation of removals. Pre-existing behaviour in
the artifact pipeline, unrelated to LocalDb — but it means site config tables accumulate orphans.
- **Deleting an instance orphans its buffered messages.** Removing the `soakgen-*` instances left
their 11,804 `sf_messages` with no tracking snapshot, producing a continuous
`Cached-telemetry drain: no tracking snapshot for …` warning flood. Also pre-existing, and worth
a cleanup path.
## Rig state as left
- Owed cleanup **done**: `SoakGenerator` (2021) and `soakgen-1..4` deleted;
`ExternalSystemDefinitions` id 1 restored to `http://scadabridge-restapi:5200` **and confirmed
propagated to both site nodes**.
- Gate harness removed: `gate-sensor-1`, `gate-churn-1`, template-3 scripts `GateBufferProbe` /
`GateSetStatic`, and the `GateDeadTarget` external system are deleted centrally. The site-side
`external_systems` row for `GateDeadTarget` remains, per the observation above.
- Remaining instances: `soak-motor-1..4` (template 4, not deployable) — untouched, as found.
- The ~11.8k orphaned `sf_messages` remain on both site-a nodes.
docs/plans/2026-07-19-localdb-phase2-soak.md
+348
View File
@@ -0,0 +1,348 @@
# LocalDb Phase 2 — rig soak findings
**Run date:** 2026-07-19 / 2026-07-20 (UTC) · **Rig:** local 8-node docker cluster, site-a pair
**Task:** Task 1 of [`2026-07-19-localdb-adoption-phase2.md`](2026-07-19-localdb-adoption-phase2.md)
> ## GATE VERDICT: **STOP — do not proceed to Task 3.** *(superseded — see update)*
>
> Not for the reason the plan anticipated. Oplog sizing is fine and D6 is resolved. The soak
> instead surfaced what looked like a **pre-existing Phase 1 defect**: the consolidated LocalDb
> database (`site-localdb.db`) throws `SQLite Error 10: 'disk I/O error'` on essentially every
> write on the **active** node under sustained concurrent load. See
> [Finding 1](#finding-1-blocker).
>
> **Update 2026-07-20: Finding 1 is root-caused and is NOT a product defect.** The soak's own
> host-side `sqlite3` sampling poisoned both nodes (WAL reset across the virtiofs boundary);
> the unmodified code sustains the full soak load indefinitely with zero errors —
> reproduced/refuted on demand, see
> [`docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md`](../known-issues/2026-07-20-localdb-disk-io-error-under-load.md) §0.
> Finding 1 therefore no longer blocks Task 3. What still stands before proceeding: re-run the
> cut-short sampling (Findings 4/5 write-rate numbers) using the safe copy-based `snap` recipe
> now in the plan, on a rig where both site-a nodes have been restarted since any host-side read.
>
> ## GATE VERDICT (final, 2026-07-20): **PROCEED to Task 3.**
>
> The re-run is done — [§1b](#1b-clean-re-run-2026-07-20-post-root-cause). Safe copy-based
> sampling, both nodes restarted first, six clean 60-second intervals: **0.80 `sf_messages`
> rows/sec sustained, zero SQLite errors across the full window, both nodes converged.** No stop
> condition from any of D1–D6 is met. The one binding number Phase 2 must honour is
> **`LocalDb:Replication:MaxBatchSize = 16`** (Finding 2 / D6), which Task 19 sets.
---
## 1. Method as actually executed
The plan's method needed four corrections before it would run. Recorded here so the next run
does not rediscover them.
| Plan said | Reality |
|---|---|
| `docker exec … curl -s localhost:8084/metrics` | **No `curl` in the `aspnet:10.0` image.** Use a sidecar sharing the container netns: `docker run --rm --network container:scadabridge-site-a-a curlimages/curl:latest -s localhost:8084/metrics` |
| Sample with host-side `sqlite3` against the bind mounts | **This IS the cause of Finding 1** (root-caused 2026-07-20 — the "ruled out" verdict below did not survive; see the known-issue doc §0). Host↔container POSIX locks don't propagate over virtiofs; a host-side read checkpoints + resets the WAL under the container and permanently poisons its connections. Copy the file triplet and query the copy, or read counters from `/metrics`. |
| Drive alarm churn on a deployed instance | **Not possible on this rig.** `infra/mssql/seed-config.sql` seeds zero `TemplateNativeAlarmSources`, opc-plc runs with no alarm flags, and no simulator or harness exists anywhere in the repo. `native_alarm_state` stayed at 0 rows throughout. Bounded analytically instead — see [Finding 3](#finding-3). |
| Drive S&F churn via template 4's `TestExternalSystem` script | Template 4 exists after a reseed but is **not deployable** (34 pre-deployment validation errors: 30 `ConnectionBinding` + 4 `ScriptCompilation`). A purpose-built `SoakGenerator` template was used instead — see below. |
### The generator that worked
`ExternalSystem.Call` does **not** buffer to store-and-forward in practice; `ExternalSystem.CachedCall`
is the buffering surface. This is the single most important operational detail for reproducing
S&F load.
- Template `SoakGenerator` (id 2021), one `Interval` script at `{"intervalMs":5000}`:
```csharp
var parms = new Dictionary<string, object?> { ["a"] = 2, ["b"] = 3 };
await ExternalSystem.CachedCall("Test REST API", "Add", parms);
```
- No attributes, no compositions, no connection bindings — deliberately, so it deploys cleanly.
- `ExternalSystemDefinitions` id 1 repointed to `http://127.0.0.1:9` (discard port → connection
refused → classified transient → buffered).
- 4 instances (`soakgen-1..4`) deployed to site-a.
Sustained rate observed: ~**2.9 HTTP attempts/sec** (688–864 connection-refused per 4 min).
### Two rig-tooling bugs found and fixed en route
1. **`docker/seed-sites.sh` seeded stale role names** — `Design`/`Deployment` instead of the
canonical `Designer`/`Deployer` (`src/ZB.MOM.WW.ScadaBridge.Security/Roles.cs:46-47`). Every
Designer/Deployer-gated management command failed `UNAUTHORIZED` on a freshly reseeded rig,
including `seed-sites.sh`'s own trailing `deploy artifacts` and `reseed.sh` stage 6d.
**Fixed** (commit `cf46e596`).
2. **`infra/mssql/setup.sql` never executes.** It is mounted into
`/docker-entrypoint-initdb.d/`, a convention the official `mcr.microsoft.com/mssql/server`
image does not implement. After `reseed.sh` drops the volume (`docker compose down -v`),
nothing recreates `ScadaBridgeConfig` or the `scadabridge_app` login, so `reseed.sh` hangs
forever on its "Waiting for setup.sql to create ScadaBridgeConfig" poll. Worked around by
applying the three init scripts by hand. **NOT yet fixed in the repo.**
---
## Finding 1 (BLOCKER) — *root-caused 2026-07-20: observer-induced, not a product defect; see the gate-verdict update and the known-issue doc §0*
### The Phase 1 consolidated LocalDb fails under sustained write load on the active node
`site-localdb.db` throws `SQLite Error 10: 'disk I/O error'` on essentially every write once the
active node is under concurrent load. Both Phase 1 tables and the audit telemetry paths are
affected.
Representative stacks (`docker logs scadabridge-site-a-b`):
```
Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 10: 'disk I/O error'.
at ZB.MOM.WW.ScadaBridge.SiteEventLogging.SiteEventLogger.ProcessWriteQueueAsync()
SiteEventLogger.cs:line 221/236
Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 10: 'disk I/O error'.
at ZB.MOM.WW.ScadaBridge.SiteRuntime.Tracking.OperationTrackingStore.RecordEnqueueAsync(...)
OperationTrackingStore.cs:line 137
at ...CachedCallTelemetryForwarder.TryEmitTrackingAsync(...) line 148
```
User-visible symptom: `[ERR] Failed to record event: script from ScriptActor:SoakCall` — **site
event logging is silently dropping events on the floor under load.**
#### It follows the load, not the node, not the observer
The failure was isolated by moving the load between nodes:
| Node | Role | Under load | `disk I/O error` in 4 min |
|---|---|---|---|
| site-a-a | active | yes | 2 175 |
| site-a-a | standby (after restart) | no | **0** |
| site-a-b | standby | no | **0** |
| site-a-b | active (after failover) | yes | **4 391** |
#### It is LocalDb-specific, not the filesystem — *wrong: sampling-selection bias; only the LocalDb file was ever host-read*
The decisive control. Under identical load, on the same node, in the same bind-mounted
directory, counting error-stack frames over 3 minutes:
| Store | Backing file | Errors |
|---|---|---|
| `OperationTrackingStore` | `site-localdb.db` (LocalDb) | 13 044 |
| `SiteAuditTelemetryActor` | `site-localdb.db` (LocalDb) | 4 350 |
| `SiteEventLogger` | `site-localdb.db` (LocalDb) | 900 |
| `CachedCallTelemetryForwarder` | `site-localdb.db` (LocalDb) | 162 |
| `StoreAndForwardStorage` | `store-and-forward.db` (legacy) | **0** |
| `SiteStorageService` | `scadabridge.db` (legacy) | **0** |
Ordinary SQLite on the same bind mount is completely healthy. Only the LocalDb-managed
database fails.
#### Ruling out the observer — **RETRACTED 2026-07-20: the observer was the cause**
Onset (04:56:37) was one second after a host-side `sqlite3` sample (04:56:36), which made
observer-induced `-shm` corruption the leading hypothesis. The original run "excluded" it:
after node-a was restarted and the load failed over to node-b, node-b began erroring while no
host process touched its files, and sampled node-a went to zero once idle.
**That exclusion was wrong.** The 04:56 sampling had poisoned *both* nodes' files (both carry
the 04:56 main-DB mtime; node-b's WAL was left at 0 bytes) — node-b was silent only because a
standby issues ~no LocalDb statements, and erupted on its first post-failover write. Verified
2026-07-20: 10+ min of full soak load on a freshly-reopened node with zero errors, then a
single host `sqlite3` read reset its 4.6 MiB WAL to 0 bytes and started the error storm one
second later (`SQLITE_IOERR_SHORT_READ`, 522). Full mechanism + minimal repro:
[`docs/known-issues/2026-07-20-localdb-disk-io-error-under-load.md`](../known-issues/2026-07-20-localdb-disk-io-error-under-load.md) §0.
#### Secondary defect, same area
```
[ERROR][akka://scadabridge/user/site-audit-telemetry] There is no active ActorContext,
this is most likely due to use of async operations from within this actor.
Cause: System.NotSupportedException
```
`SiteAuditTelemetryActor` is closing over `ActorContext` across an `await`. Likely a
contributing cause rather than a separate issue — it is in the same write path — but it is a
real bug on its own terms.
#### Why this blocks Phase 2
Phase 2 registers **eight further tables** into this database, including `native_alarm_state`
(the highest-volume table in either DB) and `sf_messages`. It also **deletes** the bespoke
mechanisms (`SiteReplicationActor`, `ReplicationService`) that currently carry that data
independently of LocalDb. Cutting over onto a store that cannot absorb the *current* write
load — and doing so in the same commit that removes the fallback — would convert a logging
defect into site-wide config and buffer loss.
~~This is a Phase 1 defect. It must be root-caused and fixed before Phase 2's Task 3.~~
**Root-caused 2026-07-20: not a Phase 1 defect** — observer-induced WAL reset across the
virtiofs bind-mount boundary; see the gate-verdict update at the top of this document.
---
## Finding 2 — D6 resolved: no stop condition, but `MaxBatchSize` must be lowered
The plan asserts `deployed_configurations.config_json` is "documented to exceed 128 KB per row."
That misreads the source. `docs/known-issues/2026-06-26-deploy-config-exceeds-akka-frame-size.md`
says the *escaped Akka envelope* exceeded the 128 KB frame, and that the default serializer
double-escapes the payload, so **"the raw flattened JSON only needs to be ~60-70 KB to blow the
128 KB frame."**
So the largest known real production `config_json` is on the order of **60–70 KB**.
Measured on the rig (4 deployed configs): **max 715 B, avg 714 B** — trivially small, as the plan
predicted, hence the production figure above is the one to size against. (A direct measurement
against wonder was attempted; `wonder-app-vd03.zmr.zimmer.com` resolves over the VPN but the
servecli SSH service on :2222 refuses connections, so the box was unreachable.)
**Verdict: no stop condition.** Nothing approaches the 4 MB single-row ceiling.
**But the batching risk is real.** Batching is row-count-only (`MaxBatchSize` default **500**),
and neither side configures gRPC message limits, so the 4 MB default receive cap applies:
```
500 rows × 70 KB ≈ 35 MB ≫ 4 MB → poison batch, stream wedged
```
Recommended for Task 19:
```
LocalDb:Replication:MaxBatchSize = 16
```
`16 × 128 KB = 2 MB` — 2× headroom on row size over the known worst case, and 2× headroom
against the 4 MB cap.
---
## Finding 3 — D4 corrected: alarm write rate is bounded, not unbounded
The plan calls `native_alarm_state` "unbounded by design." It is not.
`NativeAlarmActor.MarkDirtyUpsert` (`NativeAlarmActor.cs:473-502`) coalesces into a dictionary
**keyed by `SourceReference`** and flushes on a timer (`_persistFlushInterval`, default
**100 ms**). One flush writes at most one row per distinct source reference, regardless of how
many transitions occurred in that window.
```
worst-case rows/sec = distinct_source_refs × 10
```
An alarm storm on N sources costs N rows per 100 ms flush, not N × transition-rate. This makes
the table analytically sizeable without a generator — which matters, because this rig cannot
produce alarm load at all (see §1).
**Not empirically validated.** `native_alarm_state` held 0 rows for the entire run.
---
## Finding 4 — `sf_messages` has a hard structural ceiling of 50 rows/sec
Confirmed by code rather than measurement, which is stronger here. The retry sweep takes at most
`SweepBatchLimit` messages every `RetryTimerInterval`, and each failed attempt is one `UPDATE`
incrementing `retry_count`:
```
SweepBatchLimit (500) ÷ RetryTimerInterval (10 s) = 50 row-writes/sec, hard ceiling
```
This confirms the plan's "~50 row-writes/sec worst case" — and it is a ceiling, not an estimate.
`DefaultMaxRetries = 50` at `DefaultRetryInterval = 30 s` bounds each message to 50 updates over
25 minutes.
Observed during the run: ~2.9 attempts/sec, far below the ceiling. Row counts could not be
sampled reliably (see §1) and the run was cut short by Finding 1.
---
## Finding 5 — exceeding the oplog caps is a graceful degradation, not a failure
The plan's Task 1 step 7 treats "the shared oplog cannot absorb this write profile" as a hard
stop requiring the keyed-instances escape hatch. It is much weaker than that.
`OplogStore.EnforceCapsAsync` (`OplogStore.cs:109-138`) prunes to the ceiling and sets
`needs_snapshot`; `MaintenanceBackgroundService.cs:57` logs *"Oplog backlog/age caps exceeded:
pruned to the ceiling and flagged needs_snapshot — the peer must snapshot-resync."*
`SyncSession.ComputeSnapshotRequiredAsync` (`:413-415`) then forces a snapshot resync.
So overrunning the caps costs a **full snapshot resync**, not data loss and not a wedged stream.
The caps therefore express *"how long may a peer be absent before it needs a full resync"*, and
should be sized to the longest tolerable peer outage rather than treated as a correctness
boundary. In healthy two-node operation the oplog drains continuously — `localdb_oplog_depth`
read **0** throughout.
### Provisional cap recommendation (Task 19)
Sized for a ~3-hour peer outage at a conservative 100 rows/sec aggregate, pending re-measurement
after Finding 1 is fixed:
```
LocalDb:Replication:MaxOplogRows = 1000000 # default; ≈2.8 h at 100 rows/s
LocalDb:Replication:MaxOplogAge = 3.00:00:00
LocalDb:Replication:MaxBatchSize = 16 # Finding 2 — this one is NOT optional
```
Only `MaxBatchSize` is firmly evidence-backed. The other two rest on an assumed aggregate write
rate that this run could not measure.
---
## 1b. Clean re-run (2026-07-20, post-root-cause)
The original sampling was cut short by Finding 1 and was itself the cause of it. Re-run after
both site-a nodes were restarted, using the plan's copy-based `snap()` helper — **no host-side
`sqlite3` ever touched a live file.** Six consecutive 60-second intervals, generator load
unchanged (`SoakGenerator` ×4 against a refusing endpoint):
| UTC | `sf_messages` rows | `SUM(retry_count)` | status=0 | `__localdb_oplog` | `native_alarm_state` |
|---|---|---|---|---|---|
| 06:07:57 | 3384 | 200 | 3380 | 0 | 0 |
| 06:08:57 | 3432 | 200 | 3428 | 0 | 0 |
| 06:09:57 | 3480 | 200 | 3476 | 0 | 0 |
| 06:10:57 | 3528 | 200 | 3524 | 0 | 0 |
| 06:11:57 | 3576 | 200 | 3572 | 0 | 0 |
| 06:12:57 | 3624 | 200 | 3620 | 0 | 0 |
| 06:13:57 | 3672 | 200 | 3668 | 0 | 0 |
**Measured:**
- **`sf_messages` insert rate: exactly 48 rows/min = 0.80 rows/sec**, dead steady across all six
intervals (+288 rows over 360 s, zero variance).
- **Retry-UPDATE rate: 0/sec.** `SUM(retry_count)` never moved. Only 4 rows ever reached
`retry_count = 50` (status 2, dead-lettered); the other ~3.6 k sit at `retry_count = 0`,
status 0 — **enqueued but never swept**. So this generator exercises the *insert* path only.
- **Row sizes:** `sf_messages.payload_json` max **76 B**; `deployed_configurations.config_json`
max **721 B** over 4 rows (rig config is trivially small — see the caveat below).
- **`native_alarm_state`: 0 rows** — no alarm generator on this rig, as previously recorded.
- **`__localdb_oplog`: 0 throughout** — the Phase 1 tables are genuinely idle under this load,
which is the expected result and not a measurement failure (Phase 2's tables are not yet
registered, so this load cannot reach the oplog by construction).
- **Zero SQLite errors** in `docker logs scadabridge-site-a-a` across the full 30-minute window
(`grep -ciE "disk I/O|SQLITE_IOERR|NOTADB"` → **0**). This is the direct confirmation that
Finding 1 was observer-induced: identical load, identical library, nothing host-reading the
files, no errors.
- **Both site-a nodes converged identically** (3564 rows on each at a common sample point) under
the bespoke replicator — the pre-cutover baseline Task 20 should reproduce under CDC.
**Honest limits of this measurement.** 0.80 rows/sec is **~1.6 % of the 50 rows/sec structural
ceiling**, and the retry path — the expensive one, one `UPDATE` per message per sweep — was never
exercised at all. This re-run therefore confirms *steady-state health and the absence of the
Finding 1 pathology*; it does **not** probe the ceiling. That is acceptable because the ceiling is
**structural rather than empirical** (Finding 4: `SweepBatchLimit` ÷ `RetryTimerInterval`), so
sizing does not depend on observing it. Likewise the rig's 721 B config rows are **not**
representative — D6's sizing rests on the documented ~60–70 KB production `config_json`, not on
this number.
## 2. What still owes measurement
Carry forward (items 1–2 partially discharged by §1b above):
1. ~~Sustained `sf_messages` rows/sec~~ — **measured** (§1b: 0.80/s insert, 0/s retry). Still
unmeasured: the **retry-UPDATE** path under a saturating generator approaching the 50/s
ceiling. Not required for sizing (the ceiling is structural), but it is the honest gap.
2. Any `native_alarm_state` measurement at all — requires building alarm-source seeding plus an
A&C-capable server. The `OpcUaAlarmLiveSmokeTests` **passed**, so the rig's opc-plc *does*
answer ConditionRefresh with a `SnapshotComplete`; the missing piece is ongoing transitions
and a seeded `TemplateNativeAlarmSource`. (The test's own doc comment claiming the simulator
"does not reliably expose A&C" is stale.)
3. A production-representative `config_json` from wonder, to replace the ~60–70 KB inference.
4. The empirical oplog drain-under-churn check — Task 20 evidence item 10.
## 3. Rig state left behind
- Fully reseeded (central config volume dropped and replayed; site SQLite state wiped by
`reseed.sh` stage 2).
- `ExternalSystemDefinitions` id 1 is **repointed to `http://127.0.0.1:9`** — restore to
`http://scadabridge-restapi:5200` before using the rig for anything else.
- Template `SoakGenerator` (2021) and instances `soakgen-1..4` (ids 5–8) remain deployed on
site-a and are **still generating load**. Undeploy or delete them before the Task 20 live gate.
- `LdapGroupMappings` corrected in the live DB to the canonical role names.
docs/requirements/Component-StoreAndForward.md
+1 -1
View File
@@ -80,7 +80,7 @@ There is **no maximum buffer size**. Messages accumulate in the buffer until del
- The standby node applies the same operations to its own local SQLite database but is **passive**: it never runs the delivery sweep. The retry sweep is **gated to the active node** (the oldest Up member / singleton host, re-evaluated every sweep tick), so only one node delivers at a time. The standby applies replicated operations purely to keep its copy warm for a future failover. - The standby node applies the same operations to its own local SQLite database but is **passive**: it never runs the delivery sweep. The retry sweep is **gated to the active node** (the oldest Up member / singleton host, re-evaluated every sweep tick), so only one node delivers at a time. The standby applies replicated operations purely to keep its copy warm for a future failover.
- On failover, the new active node has a near-complete copy of the buffer. In rare cases, the most recent operations may not have been replicated (e.g., a message added or removed just before failover). This can result in a few **duplicate deliveries** (message delivered but its `Remove` not yet replicated) or a few **missed retries** (message added but not replicated). Duplicate deliveries are therefore confined to the **failover window** — an in-flight delivery whose `Remove` had not yet replicated — and never occur in steady-state operation (the standby's gate keeps it from delivering the same rows). Both are acceptable trade-offs for the latency benefit. - On failover, the new active node has a near-complete copy of the buffer. In rare cases, the most recent operations may not have been replicated (e.g., a message added or removed just before failover). This can result in a few **duplicate deliveries** (message delivered but its `Remove` not yet replicated) or a few **missed retries** (message added but not replicated). Duplicate deliveries are therefore confined to the **failover window** — an in-flight delivery whose `Remove` had not yet replicated — and never occur in steady-state operation (the standby's gate keeps it from delivering the same rows). Both are acceptable trade-offs for the latency benefit.
- On failover, the new active node's gate flips to active within one sweep interval and it resumes delivery from its local copy. - On failover, the new active node's gate flips to active within one sweep interval and it resumes delivery from its local copy.
- **Peer-join anti-entropy resync (chunked, ack-confirmed).** Asynchronous, no-ack replication keeps the standby warm in steady state, but a standby that was **down for an extended period** (a crash, a long maintenance window) misses every operation replicated while it was gone and would otherwise diverge from the active node's buffer forever. To close that gap, whenever a node **(re)tracks its peer**, a **standby** requests a full-buffer snapshot (`RequestSfBufferResync`); the **active** node loads up to `MaxResyncRows` (10 000) of its oldest rows and answers with a **sequence of byte-budgeted chunks** (`SfBufferSnapshotChunk`), each carrying a shared `ResyncId`, a 1-based `Sequence`, and the `TotalChunks` count. Chunking is mandatory because the monolithic snapshot exceeds Akka remoting's **default 128 000-byte frame** for any realistic backlog and `BuildHocon` sets no override — a single oversized message is silently undeliverable (review 02 round 2, **N2 High**). Rows accumulate into a chunk until the estimated payload budget (`MaxResyncChunkBytes` = 64 000, ≈50% frame headroom) or the row cap (`MaxResyncChunkRows` = 200) is hit; a single row whose payload alone exceeds the budget ships solo with a Warning. The standby **assembles all chunks of one `ResyncId`** (a new `ResyncId` discards any stale partial assembly; a partial that never completes is dropped after `resyncAssemblyTimeout`, default 30 s, and counted a replication failure), then **replaces its entire local buffer** with the assembled snapshot (`ReplaceAllAsync`, one transaction) and returns a delivery confirmation (`SfBufferResyncAck`). The active node arms an ack window (`resyncAckTimeout`, default 60 s): an acknowledged resync increments `scadabridge.store_and_forward.resync.completed`; an unacknowledged one (lost chunks / dead peer) logs a Warning and increments `scadabridge.store_and_forward.resync.ack_missing` — closing N2's silent-loss mode (previously nothing counted a lost snapshot and nothing retried until the next peer-track). Only the active node answers; only a standby applies, and both sides re-check at apply time (a mid-flight active-flip aborts the wipe) — each side checks the repo-standard **oldest-Up member** active-node predicate (singleton-host semantics via the shared `ActiveNodeEvaluator`, the **same predicate as the S&F delivery gate**; review 02 round 2, **N1 Critical** — using cluster *leadership* here let a rolling restart of the lower-address node make the delivering node wipe its own live buffer). Because replicated applies are **upserts** (see the replication apply path), any Add/Remove/Park that lands after the resync merges cleanly onto the resynced state — no primary-key conflict, no lost delta; the one accepted exception is the rare **N5** orphan-row race (a replicated `Remove` ordered before the snapshot chunks can re-add the removed row, re-delivered once and self-corrected at the next resync — inherent to no-ack replication). If the buffer exceeds the 10 000-row cap the snapshot is flagged `Truncated` and the standby logs a Warning; the residual divergence beyond the cap drains naturally as the active node delivers. The legacy monolithic `SfBufferSnapshot` message + standby handler are **retained** for rolling-upgrade compatibility (an old active node's monolithic snapshot is still applied by a new standby). - **Peer-join anti-entropy resync (LocalDb CDC).** *(Rewritten for LocalDb Phase 2, 2026-07-20. The previous specification of a chunked, ack-confirmed `SfBufferSnapshotChunk` protocol described the bespoke `ReplicationService`, which Phase 2 deleted. The discussion is rewritten rather than removed, because the failure modes it reasoned about still exist — they are simply bounded differently now.)* The buffer lives in the consolidated LocalDb database as the replicated `sf_messages` table, and both nodes exchange changes over a gRPC sync stream rather than Akka remoting. A node that was down for an extended period no longer requests a full-buffer snapshot and **replaces** its local buffer; LocalDb's snapshot resync merges **per row under last-writer-wins and never deletes**. Several of the old hazards are therefore structurally gone rather than guarded against: **(a) the 128 000-byte Akka frame limit no longer applies** — the transport is gRPC, whose successor ceiling is the 4 MB default receive limit, managed by bounding `LocalDb:Replication:MaxBatchSize` (set to 16 on the rig, sized against a ~70 KB worst-case `config_json`; see the Phase 2 plan, D6). Chunking, `ResyncId` assembly, assembly timeouts, and the truncation flag are all retired with it. **(b) The N1 directional-authority hazard is gone.** That guard existed because the bespoke resync applied a destructive delete-all-then-insert-all, so running it in the wrong direction wiped a live buffer. With a non-destructive merge there is no wipe to gate, and replication is symmetric — either node may write. `ActiveNodeEvaluator` survives, but only for the **delivery** gate and the heartbeat, which still genuinely need a single active node. **(c) The N5 orphan-row race is gone.** A `Remove` ordered before a re-add can no longer resurrect a row: deletes are tombstoned with an HLC, and a tombstone beats any older write for the same key. **The duplicate-delivery bound, stated explicitly.** Delivery remains single-node: only the primary runs the sweep (`IClusterNodeProvider.SelfIsPrimary`). A message can therefore be delivered twice only when the OLD primary delivered it and the resulting status change had not yet replicated at the instant the gate flipped. The window is one replication flush interval plus the in-flight ack, not an unbounded divergence — and unlike the old model it does not grow with backlog depth or with how long a node was absent. **One new bound replaces the old ones:** a node offline longer than `LocalDb:Replication:TombstoneRetention` (default 7 days) may resurrect deleted rows on rejoin, because the tombstones that would have suppressed them have been pruned. Stop-and-start a site pair together, and do not leave one node of a pair offline across that horizon.
### Operation Tracking Table (lives in Site Runtime, not here) ### Operation Tracking Table (lives in Site Runtime, not here)
infra/docker-compose.yml
+3
View File
@@ -68,6 +68,9 @@ services:
MSSQL_PID: "Developer" MSSQL_PID: "Developer"
volumes: volumes:
- scadabridge-mssql-data:/var/opt/mssql - scadabridge-mssql-data:/var/opt/mssql
# NOTE: the official mssql/server image does NOT run
# /docker-entrypoint-initdb.d — these mounts are informational only;
# infra/reseed.sh applies the scripts explicitly via sqlcmd.
- ./mssql/setup.sql:/docker-entrypoint-initdb.d/setup.sql:ro - ./mssql/setup.sql:/docker-entrypoint-initdb.d/setup.sql:ro
- ./mssql/machinedata_seed.sql:/docker-entrypoint-initdb.d/machinedata_seed.sql:ro - ./mssql/machinedata_seed.sql:/docker-entrypoint-initdb.d/machinedata_seed.sql:ro
- ./mssql/setup-env2.sql:/docker-entrypoint-initdb.d/setup-env2.sql:ro - ./mssql/setup-env2.sql:/docker-entrypoint-initdb.d/setup-env2.sql:ro
infra/reseed.sh
+9 -6
View File
@@ -82,12 +82,15 @@ if ! $SKIP_TEARDOWN; then
done done
echo " MSSQL ready." echo " MSSQL ready."
echo " Waiting for setup.sql to create ScadaBridgeConfig..." # The official mcr.microsoft.com/mssql/server image does NOT implement
until docker exec scadabridge-mssql /opt/mssql-tools18/bin/sqlcmd \ # /docker-entrypoint-initdb.d, so the compose-mounted init scripts never run
-S localhost -U sa -P 'ScadaBridge_Dev1#' -C \ # on their own — waiting for them here hangs forever on a fresh volume.
-Q "IF DB_ID('ScadaBridgeConfig') IS NULL THROW 50000, 'not ready', 1;" \ # Apply them explicitly instead (all are idempotent).
>/dev/null 2>&1; do echo " Applying MSSQL init scripts (the mssql/server image has no initdb hook)..."
sleep 2 for f in mssql/setup.sql mssql/machinedata_seed.sql mssql/setup-env2.sql; do
echo " $f"
docker exec -i scadabridge-mssql /opt/mssql-tools18/bin/sqlcmd \
-S localhost -U sa -P 'ScadaBridge_Dev1#' -C -b < "$SCRIPT_DIR/$f"
done done
echo " ScadaBridgeConfig present." echo " ScadaBridgeConfig present."
src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/Telemetry/CachedCallTelemetryForwarder.cs
+4 -4
View File
@@ -114,8 +114,8 @@ public sealed class CachedCallTelemetryForwarder : ICachedCallTelemetryForwarder
// Kind/Status are domain fields carried in DetailsJson — decompose to log them. // Kind/Status are domain fields carried in DetailsJson — decompose to log them.
var d = AuditRowProjection.Decompose(telemetry.Audit); var d = AuditRowProjection.Decompose(telemetry.Audit);
_logger.LogWarning(ex, _logger.LogWarning(ex,
"CachedCallTelemetryForwarder: audit emission threw for EventId {EventId} (Kind {Kind}, Status {Status})", "CachedCallTelemetryForwarder: audit emission threw for EventId {EventId} (Kind {Kind}, Status {Status}, sqlite {SqliteError})",
d.EventId, d.Kind, d.Status); d.EventId, d.Kind, d.Status, SqliteErrorCodes.Describe(ex));
} }
} }
@@ -192,8 +192,8 @@ public sealed class CachedCallTelemetryForwarder : ICachedCallTelemetryForwarder
catch (Exception ex) catch (Exception ex)
{ {
_logger.LogWarning(ex, _logger.LogWarning(ex,
"CachedCallTelemetryForwarder: tracking-store emission threw for TrackedOperationId {Id} (Status {Status})", "CachedCallTelemetryForwarder: tracking-store emission threw for TrackedOperationId {Id} (Status {Status}, sqlite {SqliteError})",
telemetry.Operational.TrackedOperationId, telemetry.Operational.Status); telemetry.Operational.TrackedOperationId, telemetry.Operational.Status, SqliteErrorCodes.Describe(ex));
} }
} }
} }
src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/Telemetry/SiteAuditTelemetryActor.cs
+27 -10
View File
@@ -68,6 +68,15 @@ public class SiteAuditTelemetryActor : ReceiveActor
private readonly IOperationTrackingStore? _trackingStore; private readonly IOperationTrackingStore? _trackingStore;
private readonly SiteAuditTelemetryOptions _options; private readonly SiteAuditTelemetryOptions _options;
private readonly ILogger<SiteAuditTelemetryActor> _logger; private readonly ILogger<SiteAuditTelemetryActor> _logger;
// Captured at construction (both are thread-safe immutable handles) because
// ScheduleNext/ScheduleNextCached run from the drains' finally blocks, whose
// ConfigureAwait(false) continuations complete on pool threads with no
// active ActorContext — reading Context/Self there either throws
// NotSupportedException or, worse, silently resolves a STALE cell left in
// the thread-static slot and re-arms the tick at the wrong actor
// (known-issues/2026-07-20-localdb-disk-io-error-under-load.md §8).
private readonly IScheduler _scheduler;
private readonly IActorRef _self;
private ICancelable? _pendingTick; private ICancelable? _pendingTick;
private ICancelable? _pendingCachedTick; private ICancelable? _pendingCachedTick;
// Per-actor lifecycle CTS so an in-flight drain (queue read, // Per-actor lifecycle CTS so an in-flight drain (queue read,
@@ -108,6 +117,8 @@ public class SiteAuditTelemetryActor : ReceiveActor
_options = options.Value; _options = options.Value;
_logger = logger; _logger = logger;
_trackingStore = trackingStore; _trackingStore = trackingStore;
_scheduler = Context.System.Scheduler;
_self = Self;
ReceiveAsync<Drain>(_ => OnDrainAsync()); ReceiveAsync<Drain>(_ => OnDrainAsync());
ReceiveAsync<CachedDrain>(_ => OnCachedDrainAsync()); ReceiveAsync<CachedDrain>(_ => OnCachedDrainAsync());
@@ -197,7 +208,9 @@ public class SiteAuditTelemetryActor : ReceiveActor
{ {
// Catch-all so a SQLite hiccup or mapper bug never crashes the // Catch-all so a SQLite hiccup or mapper bug never crashes the
// actor. The next tick is still scheduled in the finally block. // actor. The next tick is still scheduled in the finally block.
_logger.LogError(ex, "Unexpected error during audit-log telemetry drain."); _logger.LogError(ex,
"Unexpected error during audit-log telemetry drain (sqlite {SqliteError}).",
SqliteErrorCodes.Describe(ex));
} }
finally finally
{ {
@@ -278,8 +291,8 @@ public class SiteAuditTelemetryActor : ReceiveActor
// batch — the audit half is best-effort. Log and skip // batch — the audit half is best-effort. Log and skip
// this row; it stays Pending for the next drain. // this row; it stays Pending for the next drain.
_logger.LogWarning(ex, _logger.LogWarning(ex,
"Cached-telemetry drain: tracking lookup threw for {EventId} (TrackedOperationId {Tid}); skipping.", "Cached-telemetry drain: tracking lookup threw for {EventId} (TrackedOperationId {Tid}, sqlite {SqliteError}); skipping.",
auditRow.EventId, auditRow.CorrelationId); auditRow.EventId, auditRow.CorrelationId, SqliteErrorCodes.Describe(ex));
continue; continue;
} }
@@ -332,7 +345,9 @@ public class SiteAuditTelemetryActor : ReceiveActor
} }
catch (Exception ex) catch (Exception ex)
{ {
_logger.LogError(ex, "Unexpected error during cached-telemetry drain."); _logger.LogError(ex,
"Unexpected error during cached-telemetry drain (sqlite {SqliteError}).",
SqliteErrorCodes.Describe(ex));
} }
finally finally
{ {
@@ -428,24 +443,26 @@ public class SiteAuditTelemetryActor : ReceiveActor
return list; return list;
} }
// Must stay off Context/Self: called from off-context continuations — see
// the _scheduler/_self field comment.
private void ScheduleNext(TimeSpan delay) private void ScheduleNext(TimeSpan delay)
{ {
_pendingTick?.Cancel(); _pendingTick?.Cancel();
_pendingTick = Context.System.Scheduler.ScheduleTellOnceCancelable( _pendingTick = _scheduler.ScheduleTellOnceCancelable(
delay, delay,
Self, _self,
Drain.Instance, Drain.Instance,
Self); _self);
} }
private void ScheduleNextCached(TimeSpan delay) private void ScheduleNextCached(TimeSpan delay)
{ {
_pendingCachedTick?.Cancel(); _pendingCachedTick?.Cancel();
_pendingCachedTick = Context.System.Scheduler.ScheduleTellOnceCancelable( _pendingCachedTick = _scheduler.ScheduleTellOnceCancelable(
delay, delay,
Self, _self,
CachedDrain.Instance, CachedDrain.Instance,
Self); _self);
} }
/// <summary>Self-tick message that triggers an audit-only drain cycle.</summary> /// <summary>Self-tick message that triggers an audit-only drain cycle.</summary>
src/ZB.MOM.WW.ScadaBridge.AuditLog/Site/Telemetry/SqliteErrorCodes.cs
+31
View File
@@ -0,0 +1,31 @@
using Microsoft.Data.Sqlite;
namespace ZB.MOM.WW.ScadaBridge.AuditLog.Site.Telemetry;
/// <summary>
/// Renders the primary/extended SQLite result codes of a
/// <see cref="SqliteException"/> for log messages. The exception's own message
/// carries only the primary code ("SQLite Error 10: 'disk I/O error'"), which
/// is too generic to act on — the 2026-07-20 disk-I/O incident
/// (known-issues/2026-07-20-localdb-disk-io-error-under-load.md) had to be
/// reproduced from scratch to learn the extended code (522 =
/// SQLITE_IOERR_SHORT_READ) that names the failing operation.
/// </summary>
internal static class SqliteErrorCodes
{
/// <summary>
/// "primary/extended" (e.g. "10/522") for a <see cref="SqliteException"/>
/// anywhere in the exception chain; "n/a" for non-SQLite failures.
/// </summary>
public static string Describe(Exception ex)
{
for (Exception? e = ex; e is not null; e = e.InnerException)
{
if (e is SqliteException se)
{
return $"{se.SqliteErrorCode}/{se.SqliteExtendedErrorCode}";
}
}
return "n/a";
}
}
src/ZB.MOM.WW.ScadaBridge.Communication/ClusterState/ActiveNodeEvaluator.cs
+14 -5
View File
@@ -10,11 +10,20 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.ClusterState;
/// once the original first node restarts and rejoins; every product-level active/standby /// once the original first node restarts and rejoins; every product-level active/standby
/// decision must use this evaluator, never <c>cluster.State.Leader</c>. /// decision must use this evaluator, never <c>cluster.State.Leader</c>.
/// <para> /// <para>
/// Lives in Communication (not Host) so BOTH <c>SiteCommunicationActor</c> and /// Lives in Communication (not Host) so <c>SiteCommunicationActor</c> can default to it —
/// <c>SiteReplicationActor</c> can default to it — Host cannot be referenced from either. /// Host cannot be referenced from there. The Host's
/// The Host's <c>ClusterActivityEvaluator.SelfIsOldest</c> delegates here, so the S&amp;F /// <c>ClusterActivityEvaluator.SelfIsOldest</c> delegates here, so the S&amp;F delivery gate
/// delivery gate (<c>IClusterNodeProvider.SelfIsPrimary</c>), the resync authority checks, /// (<c>IClusterNodeProvider.SelfIsPrimary</c>) and the heartbeat IsActive stamp share one
/// and the heartbeat IsActive stamp all share one implementation. /// implementation.
/// <para>
/// It also backed <c>SiteReplicationActor</c>'s resync authority checks until LocalDb
/// Phase 2 deleted that actor. Those checks existed because the bespoke resync applied a
/// destructive delete-all-then-insert-all, so running it in the wrong direction wiped a
/// live store-and-forward buffer. LocalDb's snapshot resync merges per row under
/// last-writer-wins and never deletes, so there is no destructive apply left to gate — the
/// evaluator survives for the delivery gate and the heartbeat, which still genuinely need
/// a single active node.
/// </para>
/// </para> /// </para>
/// </summary> /// </summary>
public static class ActiveNodeEvaluator public static class ActiveNodeEvaluator
src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs
+9 -28
View File
@@ -766,37 +766,18 @@ akka {{
var deploymentConfigFetcher = var deploymentConfigFetcher =
_serviceProvider.GetService<ZB.MOM.WW.ScadaBridge.SiteRuntime.Deployment.IDeploymentConfigFetcher>(); _serviceProvider.GetService<ZB.MOM.WW.ScadaBridge.SiteRuntime.Deployment.IDeploymentConfigFetcher>();
// Create SiteReplicationActor on every node (not a singleton) // ONE active-node predicate instance governs the S&F delivery gate and the
var sfStorage = _serviceProvider.GetRequiredService<StoreAndForwardStorage>(); // heartbeat IsActive stamp (SiteCommunicationActor, wired below) — review 02
var replicationService = _serviceProvider.GetRequiredService<ReplicationService>(); // round 2, N1. It also governed SiteReplicationActor's resync authority until
var replicationLogger = _serviceProvider.GetRequiredService<ILoggerFactory>() // LocalDb Phase 2 deleted that actor: the library's snapshot resync merges per row
.CreateLogger<SiteReplicationActor>(); // under last-writer-wins and never deletes, so there is no destructive apply left
// to need an authority check. Null in non-clustered test hosts: the consumers fall
// ONE active-node predicate instance governs the S&F delivery gate, the resync // back to the shared oldest-Up evaluator, never to a leader check.
// authority checks (SiteReplicationActor), and the heartbeat IsActive stamp
// (SiteCommunicationActor, wired below) — review 02 round 2, N1. Null in
// non-clustered test hosts: the actors fall back to the shared oldest-Up
// evaluator, never to a leader check.
var clusterNodeProvider = _serviceProvider.GetService<ZB.MOM.WW.ScadaBridge.HealthMonitoring.IClusterNodeProvider>(); var clusterNodeProvider = _serviceProvider.GetService<ZB.MOM.WW.ScadaBridge.HealthMonitoring.IClusterNodeProvider>();
Func<bool>? activeNodeCheck = clusterNodeProvider != null Func<bool>? activeNodeCheck = clusterNodeProvider != null
? () => clusterNodeProvider.SelfIsPrimary ? () => clusterNodeProvider.SelfIsPrimary
: null; : null;
var replicationActor = _actorSystem!.ActorOf(
Props.Create(() => new SiteReplicationActor(
storage, sfStorage, replicationService, siteRole, replicationLogger,
deploymentConfigFetcher, activeNodeCheck, siteRuntimeOptionsValue, null)),
"site-replication");
// Wire S&F replication handler to forward operations via the replication actor
replicationService.SetReplicationHandler(op =>
{
replicationActor.Tell(new ReplicateStoreAndForward(op));
return Task.CompletedTask;
});
_logger.LogInformation("SiteReplicationActor created and S&F replication handler wired");
// Deployment Manager — role-scoped singleton via SingletonRegistrar // Deployment Manager — role-scoped singleton via SingletonRegistrar
// (review 01 round-2 N5): previously hand-rolled with bare PoisonPill // (review 01 round-2 N5): previously hand-rolled with bare PoisonPill
// termination and NO PhaseClusterLeave drain, so in-flight SQLite // termination and NO PhaseClusterLeave drain, so in-flight SQLite
@@ -807,7 +788,7 @@ akka {{
_actorSystem!, "deployment-manager", _actorSystem!, "deployment-manager",
Props.Create(() => new DeploymentManagerActor( Props.Create(() => new DeploymentManagerActor(
storage, compilationService, sharedScriptLibrary, streamManager, storage, compilationService, sharedScriptLibrary, streamManager,
siteRuntimeOptionsValue, dmLogger, dclManager, replicationActor, siteRuntimeOptionsValue, dmLogger, dclManager,
siteHealthCollector, _serviceProvider, null, deploymentConfigFetcher)), siteHealthCollector, _serviceProvider, null, deploymentConfigFetcher)),
_logger, role: siteRole); _logger, role: siteRole);
var dmProxy = dm.Proxy; var dmProxy = dm.Proxy;
@@ -1053,7 +1034,7 @@ akka {{
// SetReady asserts a deliberately narrow contract. By this point the // SetReady asserts a deliberately narrow contract. By this point the
// actor system exists, SiteStreamManager.Initialize has run, and every // actor system exists, SiteStreamManager.Initialize has run, and every
// role actor (SiteCommunicationActor, deployment-manager singleton, // role actor (SiteCommunicationActor, deployment-manager singleton,
// SiteReplicationActor, the ClusterClient) has been created with ActorOf — // the ClusterClient) has been created with ActorOf —
// creation and the registration Tells are synchronous and strictly ordered. // creation and the registration Tells are synchronous and strictly ordered.
// What is NOT guaranteed is completion of each actor's PreStart or the // What is NOT guaranteed is completion of each actor's PreStart or the
// ClusterClient's initial-contact handshake with central: those are // ClusterClient's initial-contact handshake with central: those are
src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbLegacyMigrator.cs
+259 -3
View File
@@ -5,8 +5,9 @@ using ZB.MOM.WW.LocalDb;
namespace ZB.MOM.WW.ScadaBridge.Host; namespace ZB.MOM.WW.ScadaBridge.Host;
/// <summary> /// <summary>
/// One-time copy of the pre-Phase-1 site databases (<c>site-tracking.db</c> and /// One-time copy of the pre-consolidation site databases — Phase 1's
/// <c>site_events.db</c>) into the consolidated <c>ZB.MOM.WW.LocalDb</c> database. /// <c>site-tracking.db</c> and <c>site_events.db</c>, and Phase 2's
/// <c>store-and-forward.db</c> — into the consolidated <c>ZB.MOM.WW.LocalDb</c> database.
/// </summary> /// </summary>
/// <remarks> /// <remarks>
/// <para> /// <para>
@@ -31,13 +32,20 @@ namespace ZB.MOM.WW.ScadaBridge.Host;
/// half-migrated state to reason about. A second boot sees the renamed file and no-ops. /// half-migrated state to reason about. A second boot sees the renamed file and no-ops.
/// </para> /// </para>
/// <para> /// <para>
/// <b>Finding nothing is the expected case on the docker rig.</b> Neither legacy config key /// <b>Finding nothing is the expected case on the docker rig for the two Phase 1 files.</b>
/// Neither legacy config key
/// is set in any rig appsettings, so both fall back to CWD-relative code defaults /// is set in any rig appsettings, so both fall back to CWD-relative code defaults
/// (<c>/app/site-tracking.db</c>, <c>/app/site_events.db</c>) that sit OUTSIDE the mounted /// (<c>/app/site-tracking.db</c>, <c>/app/site_events.db</c>) that sit OUTSIDE the mounted
/// data volume — meaning they were already being discarded on every container recreate. /// data volume — meaning they were already being discarded on every container recreate.
/// Phase 1 incidentally fixes that data-loss bug by consolidating into /// Phase 1 incidentally fixes that data-loss bug by consolidating into
/// <c>/app/data/site-localdb.db</c>. A no-op here is a legitimate result, not a failure. /// <c>/app/data/site-localdb.db</c>. A no-op here is a legitimate result, not a failure.
/// </para> /// </para>
/// <para>
/// <b>Store-and-forward is the exception.</b> Its default path <i>is</i> inside the data
/// volume (<c>./data/store-and-forward.db</c>), so a real deployment has a real file there
/// holding undelivered messages. That migration genuinely moves data, and dropping it would
/// silently discard exactly the buffered calls store-and-forward exists to protect.
/// </para>
/// </remarks> /// </remarks>
public static class SiteLocalDbLegacyMigrator public static class SiteLocalDbLegacyMigrator
{ {
@@ -49,6 +57,80 @@ public static class SiteLocalDbLegacyMigrator
/// <summary>Default legacy event-log path (<c>SiteEventLogOptions.DatabasePath</c>).</summary> /// <summary>Default legacy event-log path (<c>SiteEventLogOptions.DatabasePath</c>).</summary>
private const string DefaultEventLogPath = "site_events.db"; private const string DefaultEventLogPath = "site_events.db";
/// <summary>Default legacy store-and-forward path (<c>StoreAndForwardOptions.SqliteDbPath</c>).</summary>
private const string DefaultStoreAndForwardPath = "./data/store-and-forward.db";
/// <summary>Default legacy site configuration path (<c>appsettings.Site.json</c>).</summary>
private const string DefaultSiteStoragePath = "./data/scadabridge.db";
/// <summary>
/// Every column of the current <c>sf_messages</c> schema, in a fixed order. Columns
/// absent from an older legacy file are dropped from the copy rather than failing it —
/// see <see cref="PresentColumns"/>.
/// </summary>
private static readonly string[] StoreAndForwardColumns =
[
"id", "category", "target", "payload_json",
"retry_count", "max_retries", "retry_interval_ms",
"created_at", "last_attempt_at", "status", "last_error", "origin_instance",
"execution_id", "source_script", "parent_execution_id", "last_attempt_at_ms",
];
/// <summary>
/// The site configuration tables copied out of the legacy <c>scadabridge.db</c>, with
/// the current schema's columns for each.
/// </summary>
/// <remarks>
/// <b><c>notification_lists</c> and <c>smtp_configurations</c> are deliberately absent.</b>
/// Both are purged on every deploy and are permanently empty by design — the site-side
/// write paths were removed on 2026-07-10. A pre-fix legacy file can still hold rows, and
/// <c>smtp_configurations.password</c> is plaintext.
/// <para>
/// Skipping them here is one half of a pair: the cutover also declines to register them
/// for replication, for the same reason. Migrating them would leave plaintext SMTP
/// passwords sitting in the consolidated database — one future <c>RegisterReplicated</c>
/// away from being shipped to a peer — in exchange for resurrecting config that nothing
/// reads. Keeping the tables permanently empty is what makes both decisions safe.
/// </para>
/// The tables themselves are still created (see <c>SiteStorageSchema</c>); only their
/// historical contents are left behind.
/// </remarks>
internal static readonly LegacyTable[] SiteStorageTables =
[
new("deployed_configurations", "instance_unique_name",
[
"instance_unique_name", "config_json", "deployment_id", "revision_hash",
"is_enabled", "deployed_at",
]),
new("static_attribute_overrides", "instance_unique_name",
[
"instance_unique_name", "attribute_name", "override_value", "updated_at",
]),
new("shared_scripts", "name",
[
"name", "code", "parameter_definitions", "return_definition", "updated_at",
]),
new("external_systems", "name",
[
"name", "endpoint_url", "auth_type", "auth_configuration", "method_definitions",
"updated_at", "timeout_seconds",
]),
new("database_connections", "name",
[
"name", "connection_string", "max_retries", "retry_delay_ms", "updated_at",
]),
new("data_connection_definitions", "name",
[
"name", "protocol", "configuration", "backup_configuration",
"failover_retry_count", "updated_at",
]),
new("native_alarm_state", "instance_unique_name",
[
"instance_unique_name", "source_canonical_name", "source_reference",
"condition_json", "last_transition_at", "metadata_json",
]),
];
/// <summary> /// <summary>
/// Copies any legacy site databases into <paramref name="db"/>, then renames them. /// Copies any legacy site databases into <paramref name="db"/>, then renames them.
/// </summary> /// </summary>
@@ -63,6 +145,8 @@ public static class SiteLocalDbLegacyMigrator
MigrateTracking(db, ResolveTrackingPath(config)); MigrateTracking(db, ResolveTrackingPath(config));
MigrateEvents(db, ResolveEventLogPath(config), nodeName); MigrateEvents(db, ResolveEventLogPath(config), nodeName);
MigrateStoreAndForward(db, ResolveStoreAndForwardPath(config));
MigrateSiteStorage(db, ResolveSiteStoragePath(config));
} }
/// <summary> /// <summary>
@@ -113,6 +197,121 @@ public static class SiteLocalDbLegacyMigrator
return Path.GetFullPath(path); return Path.GetFullPath(path);
} }
/// <summary>
/// Resolves the legacy store-and-forward database path from the OLD key, falling back
/// to the code default. Unlike the two Phase 1 paths, this default sits INSIDE the
/// mounted data volume (<c>./data/</c>), so on the docker rig there is a real file here
/// with real buffered messages — this migration is not the usual no-op.
/// </summary>
internal static string ResolveStoreAndForwardPath(IConfiguration config)
{
var path = config["ScadaBridge:StoreAndForward:SqliteDbPath"] ?? DefaultStoreAndForwardPath;
if (string.IsNullOrWhiteSpace(path) ||
path.Equals(":memory:", StringComparison.OrdinalIgnoreCase) ||
path.StartsWith("file:", StringComparison.OrdinalIgnoreCase))
{
return string.Empty;
}
return Path.GetFullPath(path);
}
/// <summary>
/// Resolves the legacy site configuration database path from the OLD key, falling back
/// to the code default. Like store-and-forward — and unlike the two Phase 1 paths — this
/// default is inside the mounted data volume, so a real deployment has real config here.
/// </summary>
internal static string ResolveSiteStoragePath(IConfiguration config)
{
var path = config["ScadaBridge:Database:SiteDbPath"] ?? DefaultSiteStoragePath;
if (string.IsNullOrWhiteSpace(path) ||
path.Equals(":memory:", StringComparison.OrdinalIgnoreCase) ||
path.StartsWith("file:", StringComparison.OrdinalIgnoreCase))
{
return string.Empty;
}
return Path.GetFullPath(path);
}
/// <summary>
/// Copies buffered store-and-forward messages out of the legacy file.
/// </summary>
/// <remarks>
/// No id synthesis, unlike <see cref="MigrateEvents"/>: <c>sf_messages.id</c> is already
/// a caller-assigned TEXT primary key, so <c>INSERT OR IGNORE</c> is naturally idempotent
/// across a crash-then-rerun.
/// <para>
/// These are undelivered messages, so dropping them is real data loss — a buffered call
/// that never reaches its external system is exactly what store-and-forward exists to
/// prevent. That is why the copy tolerates an older column set rather than bailing.
/// </para>
/// </remarks>
private static void MigrateStoreAndForward(ILocalDb db, string legacyPath)
=> MigrateFile(db, legacyPath, [new LegacyTable("sf_messages", "id", StoreAndForwardColumns)]);
/// <summary>
/// Copies the site's configuration tables out of the legacy <c>scadabridge.db</c>.
/// </summary>
/// <remarks>
/// All seven migrated tables live in one file, so they are copied inside a single
/// transaction and the file is renamed once: a partial config migration would leave a
/// site node running against half its old configuration, which is worse than failing
/// startup outright.
/// </remarks>
private static void MigrateSiteStorage(ILocalDb db, string legacyPath)
=> MigrateFile(db, legacyPath, SiteStorageTables);
/// <summary>
/// Copies one table from a legacy file into the consolidated database, then renames the
/// legacy file.
/// </summary>
/// <remarks>
/// The copy is restricted to the columns the legacy table <i>actually has</i>. A file
/// written by an older build predates some columns, and naming a missing column in the
/// SELECT would throw "no such column" — which the reader treats as an unrecognised
/// shape, silently discarding every row in the table. Intersecting first means an old
/// file migrates its data and simply leaves the newer columns at their schema defaults.
/// </remarks>
/// <param name="db">The consolidated database.</param>
/// <param name="legacyPath">The legacy file, which may not exist.</param>
/// <param name="tables">Every table to copy out of this file, in order.</param>
private static void MigrateFile(ILocalDb db, string legacyPath, IReadOnlyList<LegacyTable> tables)
{
if (!ShouldMigrate(legacyPath)) return;
using (var legacy = OpenLegacyReadOnly(legacyPath))
{
using var connection = db.CreateConnection();
using var transaction = connection.BeginTransaction();
foreach (var table in tables)
{
var present = PresentColumns(legacy, table.Table, table.Columns);
// An absent table probes as zero columns, so this one guard covers both
// "old file predating the table" and "file we do not recognise".
if (present.Contains(table.RequiredColumn))
CopyRows(legacy, connection, transaction, table.Table, present);
}
transaction.Commit();
}
MarkMigrated(legacyPath);
}
/// <summary>One table to copy out of a legacy file.</summary>
/// <param name="Table">Table name, identical on both sides.</param>
/// <param name="RequiredColumn">
/// A column without which the table is not the one we mean — normally the primary key.
/// Copying rows with a NULL PK would be worse than copying nothing.
/// </param>
/// <param name="Columns">The current schema's full column list, in a fixed order.</param>
internal sealed record LegacyTable(string Table, string RequiredColumn, string[] Columns);
private static void MigrateTracking(ILocalDb db, string legacyPath) private static void MigrateTracking(ILocalDb db, string legacyPath)
{ {
if (!ShouldMigrate(legacyPath)) return; if (!ShouldMigrate(legacyPath)) return;
@@ -249,6 +448,63 @@ public static class SiteLocalDbLegacyMigrator
return rows; return rows;
} }
private static SqliteConnection OpenLegacyReadOnly(string legacyPath)
{
var connection = new SqliteConnection($"Data Source={legacyPath};Mode=ReadOnly");
connection.Open();
return connection;
}
/// <summary>
/// Returns the subset of <paramref name="wanted"/> that the legacy table actually has,
/// in the caller's order. An absent table yields an empty list rather than throwing.
/// </summary>
private static List<string> PresentColumns(
SqliteConnection legacy, string table, IReadOnlyList<string> wanted)
{
var present = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
using (var probe = legacy.CreateCommand())
{
// Table name is a caller-controlled constant, never user input — safe to
// interpolate (parameters are not permitted as a pragma-function argument).
probe.CommandText = $"SELECT name FROM pragma_table_info('{table}')";
using var reader = probe.ExecuteReader();
while (reader.Read()) present.Add(reader.GetString(0));
}
return [.. wanted.Where(present.Contains)];
}
/// <summary>Streams every row of <paramref name="columns"/> from the legacy table into the target.</summary>
private static void CopyRows(
SqliteConnection legacy,
SqliteConnection target,
SqliteTransaction transaction,
string table,
IReadOnlyList<string> columns)
{
var columnList = string.Join(", ", columns);
var parameterList = string.Join(", ", columns.Select((_, i) => $"$p{i}"));
using var read = legacy.CreateCommand();
read.CommandText = $"SELECT {columnList} FROM {table};";
using var reader = read.ExecuteReader();
while (reader.Read())
{
using var write = target.CreateCommand();
write.Transaction = transaction;
write.CommandText =
$"INSERT OR IGNORE INTO {table} ({columnList}) VALUES ({parameterList});";
for (var i = 0; i < columns.Count; i++)
write.Parameters.AddWithValue($"$p{i}", reader.IsDBNull(i) ? DBNull.Value : reader.GetValue(i));
write.ExecuteNonQuery();
}
}
private static void Bind(SqliteCommand cmd, object?[] row) private static void Bind(SqliteCommand cmd, object?[] row)
{ {
for (var i = 0; i < row.Length; i++) for (var i = 0; i < row.Length; i++)
src/ZB.MOM.WW.ScadaBridge.Host/SiteLocalDbSetup.cs
+32
View File
@@ -1,7 +1,9 @@
using Microsoft.Extensions.Configuration; using Microsoft.Extensions.Configuration;
using ZB.MOM.WW.LocalDb; using ZB.MOM.WW.LocalDb;
using ZB.MOM.WW.ScadaBridge.SiteEventLogging; using ZB.MOM.WW.ScadaBridge.SiteEventLogging;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Tracking; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Tracking;
using ZB.MOM.WW.ScadaBridge.StoreAndForward;
namespace ZB.MOM.WW.ScadaBridge.Host; namespace ZB.MOM.WW.ScadaBridge.Host;
@@ -44,6 +46,11 @@ public static class SiteLocalDbSetup
{ {
OperationTrackingSchema.Apply(connection); OperationTrackingSchema.Apply(connection);
SiteEventLogSchema.Apply(connection); SiteEventLogSchema.Apply(connection);
// Phase 2: the site's configuration tables and the store-and-forward buffer
// now live in this file too.
SiteStorageSchema.Apply(connection);
StoreAndForwardSchema.Apply(connection);
} }
// Both tables qualify: each has an explicit primary key (RegisterReplicated // Both tables qualify: each has an explicit primary key (RegisterReplicated
@@ -52,6 +59,31 @@ public static class SiteLocalDbSetup
db.RegisterReplicated("OperationTracking"); db.RegisterReplicated("OperationTracking");
db.RegisterReplicated("site_events"); db.RegisterReplicated("site_events");
// Phase 2: the store-and-forward buffer and the seven site configuration tables.
// These replaced the bespoke SiteReplicationActor and StoreAndForward
// ReplicationService, which shipped hand-written Add/Remove/Park/Requeue operations
// over Akka; both were deleted in the same commit that added these lines, so the
// two mechanisms never ran at once.
//
// Both composite-PK tables are fine: RegisterReplicated orders multi-column PKs by
// ordinal. No Phase 2 table has a BLOB column, which it would reject.
db.RegisterReplicated("sf_messages");
db.RegisterReplicated("deployed_configurations");
db.RegisterReplicated("static_attribute_overrides");
db.RegisterReplicated("shared_scripts");
db.RegisterReplicated("external_systems");
db.RegisterReplicated("database_connections");
db.RegisterReplicated("data_connection_definitions");
db.RegisterReplicated("native_alarm_state");
// notification_lists and smtp_configurations are created but deliberately NOT
// registered. They are permanently empty by design — the site-side write paths were
// removed on 2026-07-10, the legacy migrator skips them, and the active node's
// artifact apply purges them on every deploy. Registering them would open a standing
// replication channel whose only historical payload was plaintext SMTP passwords, in
// exchange for replicating nothing. Anyone adding them here should first establish
// that a site has a legitimate reason to hold SMTP credentials at all.
// AFTER registration, so migrated rows enter the oplog and reach the peer like // AFTER registration, so migrated rows enter the oplog and reach the peer like
// any other write. Before it, they would be invisible to replication forever. // any other write. Before it, they would be invisible to replication forever.
SiteLocalDbLegacyMigrator.Migrate(db, config); SiteLocalDbLegacyMigrator.Migrate(db, config);
src/ZB.MOM.WW.ScadaBridge.Host/SiteServiceRegistration.cs
+15 -4
View File
@@ -65,10 +65,13 @@ public static class SiteServiceRegistration
services.AddSingleton<ISiteIdentityProvider, SiteIdentityProvider>(); services.AddSingleton<ISiteIdentityProvider, SiteIdentityProvider>();
services.AddSingleton<IHealthReportTransport, AkkaHealthReportTransport>(); services.AddSingleton<IHealthReportTransport, AkkaHealthReportTransport>();
// Site-only components — AddSiteRuntime registers SiteStorageService with SQLite path // Site-only components — AddSiteRuntime registers SiteStorageService and the
// and site-local repository implementations (IExternalSystemRepository, INotificationRepository) // site-local repository implementations (IExternalSystemRepository,
var siteDbPath = config["ScadaBridge:Database:SiteDbPath"] ?? "site.db"; // INotificationRepository). It takes no connection string any more:
services.AddSiteRuntime($"Data Source={siteDbPath}"); // SiteStorageService persists to the consolidated LocalDb database registered
// just below (LocalDb:Path). ScadaBridge:Database:SiteDbPath survives only as
// the legacy migrator's source location.
services.AddSiteRuntime();
// Consolidated site database (LocalDb Phase 1). Holds OperationTracking and // Consolidated site database (LocalDb Phase 1). Holds OperationTracking and
// site_events as replicated tables so the pair stops losing them on failover. // site_events as replicated tables so the pair stops losing them on failover.
@@ -78,6 +81,14 @@ public static class SiteServiceRegistration
// initiator idles. // initiator idles.
// //
// Design: scadaproj docs/plans/2026-07-19-scadabridge-localdb-design.md // Design: scadaproj docs/plans/2026-07-19-scadabridge-localdb-design.md
//
// The parent directory of LocalDb:Path is created by the library as of 0.1.1.
// ScadaBridge carried a SiteLocalDbDirectory shim for it during Phase 2, because
// SQLite creates the database file on demand but not its directory and
// SqliteLocalDb opens the file eagerly — a missing directory was a hard boot
// failure, and the default site path is the relative "./data/site-localdb.db".
// The shim is gone; SiteLocalDbDirectoryTests still pins the outcome here, since
// what this host needs is the guarantee, not any particular owner of it.
services.AddZbLocalDb(config, db => SiteLocalDbSetup.OnReady(db, config)); services.AddZbLocalDb(config, db => SiteLocalDbSetup.OnReady(db, config));
// The replication engine, likewise unconditional but INERT by default: with no // The replication engine, likewise unconditional but INERT by default: with no
src/ZB.MOM.WW.ScadaBridge.Host/StartupValidator.cs
+7 -4
View File
@@ -94,7 +94,7 @@ public static class StartupValidator
_ => seedNodes != null && seedNodes.Count >= 2, _ => seedNodes != null && seedNodes.Count >= 2,
"must have at least 2 entries") "must have at least 2 entries")
// The big Site-only block: GrpcPort/MetricsPort validity + cross-field // The big Site-only block: GrpcPort/MetricsPort validity + cross-field
// collisions + SiteDbPath + seed-node-port loop, in the original order. // collisions + seed-node-port loop, in the original order.
.When(role == "Site", p => .When(role == "Site", p =>
{ {
// GrpcPort range, then GrpcPort vs RemotingPort. // GrpcPort range, then GrpcPort vs RemotingPort.
@@ -110,9 +110,12 @@ public static class StartupValidator
p.Require("ScadaBridge:Node:MetricsPort", _ => metricsPort != port, "must differ from RemotingPort"); p.Require("ScadaBridge:Node:MetricsPort", _ => metricsPort != port, "must differ from RemotingPort");
p.Require("ScadaBridge:Node:MetricsPort", _ => metricsPort != grpcPort, "must differ from GrpcPort"); p.Require("ScadaBridge:Node:MetricsPort", _ => metricsPort != grpcPort, "must differ from GrpcPort");
p.Require("ScadaBridge:Database:SiteDbPath", // ScadaBridge:Database:SiteDbPath was required here until LocalDb
_ => !string.IsNullOrEmpty(configuration.GetSection("ScadaBridge:Database")["SiteDbPath"]), // Phase 2. The site's tables now live in the consolidated LocalDb
"required for Site nodes"); // database (LocalDb:Path, which SiteServiceRegistration requires),
// and SiteDbPath survives only as the legacy migration source — so
// its absence means "nothing to migrate", not a misconfiguration.
// DatabaseOptionsValidator still rejects a present-but-blank value.
// A seed node must reference an Akka.Remote endpoint, never the // A seed node must reference an Akka.Remote endpoint, never the
// Kestrel HTTP/2 gRPC port. A seed entry whose port equals this node's // Kestrel HTTP/2 gRPC port. A seed entry whose port equals this node's
src/ZB.MOM.WW.ScadaBridge.Host/appsettings.Site.json
+9 -2
View File
@@ -23,6 +23,10 @@
"MinNrOfMembers": 1 "MinNrOfMembers": 1
}, },
"Database": { "Database": {
// Migration-only as of LocalDb Phase 2. The site config tables now live in the
// consolidated LocalDb database (LocalDb:Path). SiteDbPath is read once at boot to drain
// a pre-Phase-2 scadabridge.db, and is unused after that - keep it until this node has
// started once.
"SiteDbPath": "./data/scadabridge.db" "SiteDbPath": "./data/scadabridge.db"
}, },
"DataConnection": { "DataConnection": {
@@ -31,8 +35,11 @@
"WriteTimeout": "00:00:30" "WriteTimeout": "00:00:30"
}, },
"StoreAndForward": { "StoreAndForward": {
"SqliteDbPath": "./data/store-and-forward.db", // Migration-only as of LocalDb Phase 2. The store-and-forward buffer now lives in the
"ReplicationEnabled": true // consolidated LocalDb database (LocalDb:Path) as the replicated sf_messages table.
// SqliteDbPath is read once at boot by SiteLocalDbLegacyMigrator to drain a pre-Phase-2
// file, and is unused after that - keep it until this node has started once.
"SqliteDbPath": "./data/store-and-forward.db"
}, },
"Communication": { "Communication": {
"_centralContactPoints": "Host-016: each entry MUST be a central node's remoting endpoint, NOT this site's own remoting port. The single dev-loopback default below points only at central-a (localhost:8081). In a multi-central deployment add the second central node here (e.g. 'akka.tcp://scadabridge@central-b-host:8081') so ClusterClient can fail over when central-a is down. The previous template listed localhost:8082 as the second contact — that is THIS site's own RemotingPort and is a permanent failure in the initial-contact rotation.", "_centralContactPoints": "Host-016: each entry MUST be a central node's remoting endpoint, NOT this site's own remoting port. The single dev-loopback default below points only at central-a (localhost:8081). In a multi-central deployment add the second central node here (e.g. 'akka.tcp://scadabridge@central-b-host:8081') so ClusterClient can fail over when central-a is down. The previous template listed localhost:8082 as the second contact — that is THIS site's own RemotingPort and is a permanent failure in the initial-contact rotation.",
src/ZB.MOM.WW.ScadaBridge.SiteEventLogging/SiteEventLogger.cs
+21 -2
View File
@@ -257,8 +257,8 @@ public class SiteEventLogger : ISiteEventLogger, IDisposable
// (Health Monitoring reads FailedWriteCount) and fault the caller's // (Health Monitoring reads FailedWriteCount) and fault the caller's
// Task instead of silently discarding the exception. // Task instead of silently discarding the exception.
Interlocked.Increment(ref _failedWriteCount); Interlocked.Increment(ref _failedWriteCount);
_logger.LogError(ex, "Failed to record event: {EventType} from {Source}", _logger.LogError(ex, "Failed to record event: {EventType} from {Source} (sqlite {SqliteError})",
pending.EventType, pending.Source); pending.EventType, pending.Source, DescribeSqliteError(ex));
pending.Completion.TrySetException(ex); pending.Completion.TrySetException(ex);
} }
} }
@@ -297,6 +297,25 @@ public class SiteEventLogger : ISiteEventLogger, IDisposable
} }
} }
/// <summary>
/// "primary/extended" SQLite result codes (e.g. "10/522") for a
/// <see cref="SqliteException"/> anywhere in the chain; "n/a" otherwise.
/// The exception message alone carries only the primary code, which proved
/// too generic to diagnose the 2026-07-20 disk-I/O incident
/// (known-issues/2026-07-20-localdb-disk-io-error-under-load.md).
/// </summary>
private static string DescribeSqliteError(Exception ex)
{
for (Exception? e = ex; e is not null; e = e.InnerException)
{
if (e is SqliteException se)
{
return $"{se.SqliteErrorCode}/{se.SqliteExtendedErrorCode}";
}
}
return "n/a";
}
/// <summary>An event awaiting persistence by the background writer.</summary> /// <summary>An event awaiting persistence by the background writer.</summary>
private sealed record PendingEvent( private sealed record PendingEvent(
string Id, string Id,
src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs
+8 -16
View File
@@ -52,7 +52,6 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
/// </summary> /// </summary>
private readonly ILoggerFactory _loggerFactory; private readonly ILoggerFactory _loggerFactory;
private readonly IActorRef? _dclManager; private readonly IActorRef? _dclManager;
private readonly IActorRef? _replicationActor;
private readonly ISiteHealthCollector? _healthCollector; private readonly ISiteHealthCollector? _healthCollector;
private readonly IServiceProvider? _serviceProvider; private readonly IServiceProvider? _serviceProvider;
/// <summary> /// <summary>
@@ -166,7 +165,6 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
SiteRuntimeOptions options, SiteRuntimeOptions options,
ILogger<DeploymentManagerActor> logger, ILogger<DeploymentManagerActor> logger,
IActorRef? dclManager = null, IActorRef? dclManager = null,
IActorRef? replicationActor = null,
ISiteHealthCollector? healthCollector = null, ISiteHealthCollector? healthCollector = null,
IServiceProvider? serviceProvider = null, IServiceProvider? serviceProvider = null,
ILoggerFactory? loggerFactory = null, ILoggerFactory? loggerFactory = null,
@@ -181,7 +179,6 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
_streamManager = streamManager; _streamManager = streamManager;
_options = options; _options = options;
_dclManager = dclManager; _dclManager = dclManager;
_replicationActor = replicationActor;
_healthCollector = healthCollector; _healthCollector = healthCollector;
_serviceProvider = serviceProvider; _serviceProvider = serviceProvider;
_configFetcher = configFetcher; _configFetcher = configFetcher;
@@ -785,15 +782,14 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
await _storage.ClearStaticOverridesAsync(instanceName); await _storage.ClearStaticOverridesAsync(instanceName);
await _storage.ClearNativeAlarmsForInstanceAsync(instanceName); await _storage.ClearNativeAlarmsForInstanceAsync(instanceName);
// Replicate to standby node — notify-and-fetch: send only the deployment id + // No explicit replication step: deployed_configurations is a replicated table,
// central fetch coordinates (NOT the config JSON). The standby fetches the // so the write above is captured and shipped to the peer like any other row.
// config over HTTP itself, so a large config never crosses the intra-site Akka // This used to be a notify-and-fetch Tell — the standby was sent the deployment
// hop (which would silently drop on the 128 KB frame trap). When the coords are // id plus central fetch coordinates and pulled the config over HTTP itself,
// absent (deploy paths other than RefreshDeployment), the standby fetch is a // because a large config would silently drop on the intra-site Akka hop's
// no-op miss and reconciliation is the durable backstop. // 128 KB frame trap. Replication carries the row directly and has no such limit,
_replicationActor?.Tell(new ReplicateConfigDeploy( // so the standby no longer fetches on deploy. (SiteReconciliationActor still
instanceName, command.DeploymentId, command.RevisionHash, true, // fetches at node startup when central reports gaps — a different path.)
command.CentralFetchBaseUrl ?? "", command.FetchToken ?? ""));
return new DeployPersistenceResult( return new DeployPersistenceResult(
command.DeploymentId, instanceName, true, null, sender, isRedeploy); command.DeploymentId, instanceName, true, null, sender, isRedeploy);
@@ -967,7 +963,6 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
{ {
if (t.IsCompletedSuccessfully) if (t.IsCompletedSuccessfully)
{ {
_replicationActor?.Tell(new ReplicateConfigSetEnabled(instanceName, false));
// Operational `deployment` event — disable succeeded. // Operational `deployment` event — disable succeeded.
LogDeploymentEvent("Info", instanceName, $"Instance {instanceName} disabled"); LogDeploymentEvent("Info", instanceName, $"Instance {instanceName} disabled");
} }
@@ -1001,7 +996,6 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
Task.Run(async () => Task.Run(async () =>
{ {
await _storage.SetInstanceEnabledAsync(instanceName, true); await _storage.SetInstanceEnabledAsync(instanceName, true);
_replicationActor?.Tell(new ReplicateConfigSetEnabled(instanceName, true));
var configs = await _storage.GetAllDeployedConfigsAsync(); var configs = await _storage.GetAllDeployedConfigsAsync();
var config = configs.FirstOrDefault(c => c.InstanceUniqueName == instanceName); var config = configs.FirstOrDefault(c => c.InstanceUniqueName == instanceName);
return new EnableResult(command, config, null, sender); return new EnableResult(command, config, null, sender);
@@ -1099,7 +1093,6 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
{ {
if (t.IsCompletedSuccessfully) if (t.IsCompletedSuccessfully)
{ {
_replicationActor?.Tell(new ReplicateConfigRemove(instanceName));
// Operational `deployment` event — delete succeeded. // Operational `deployment` event — delete succeeded.
LogDeploymentEvent("Info", instanceName, $"Instance {instanceName} deleted"); LogDeploymentEvent("Info", instanceName, $"Instance {instanceName} deleted");
} }
@@ -1949,7 +1942,6 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
// central-only and is never stored on a site (see the purge above). // central-only and is never stored on a site (see the purge above).
// Replicate artifacts to standby node // Replicate artifacts to standby node
_replicationActor?.Tell(new ReplicateArtifacts(command));
return new ArtifactDeploymentResponse( return new ArtifactDeploymentResponse(
command.DeploymentId, "", true, null, DateTimeOffset.UtcNow); command.DeploymentId, "", true, null, DateTimeOffset.UtcNow);
src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/SiteReplicationActor.cs
-707
View File
@@ -1,707 +0,0 @@
using Akka.Actor;
using Akka.Cluster;
using Akka.Event;
using Microsoft.Extensions.Logging;
using ZB.MOM.WW.ScadaBridge.Commons.Observability;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Deployment;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Messages;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
using ZB.MOM.WW.ScadaBridge.StoreAndForward;
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors;
/// <summary>
/// Runs on every site node (not a singleton). Handles both config and S&amp;F replication
/// between site cluster peers.
///
/// Outbound: receives local replication requests and forwards to peer via ActorSelection.
/// Inbound: receives replicated operations from peer and applies to local SQLite.
/// Uses fire-and-forget (Tell) — no ack wait per design.
/// </summary>
public class SiteReplicationActor : ReceiveActor, IWithTimers
{
private readonly SiteStorageService _storage;
private readonly StoreAndForwardStorage _sfStorage;
private readonly ReplicationService _replicationService;
private readonly IDeploymentConfigFetcher? _configFetcher;
private readonly string _siteRole;
private readonly ILogger<SiteReplicationActor> _logger;
private readonly Cluster _cluster;
private readonly Func<bool> _isActive;
private readonly int _configFetchRetryCount;
private readonly TimeSpan _configFetchRetryDelay;
private Address? _peerAddress;
/// <summary>Akka timer scheduler injected by the framework via <see cref="IWithTimers"/>.</summary>
public ITimerScheduler Timers { get; set; } = null!;
// ── Chunked-resync assembly (standby side; actor-thread only) ──
private string? _assemblingResyncId;
private int _assemblingTotalChunks;
private bool _assemblingTruncated;
private readonly Dictionary<int, List<StoreAndForwardMessage>> _assemblingChunks = new();
private const string ResyncAssemblyTimerKey = "sf-resync-assembly-timeout";
/// <summary>How long a partial chunk assembly may wait for its missing chunks before
/// being discarded (a lost chunk = lost resync; the next peer-track retries). Ctor
/// test seam; production default 30 s.</summary>
private readonly TimeSpan _resyncAssemblyTimeout;
// ── Resync delivery confirmation (active side; actor-thread only) ──
private string? _pendingAckResyncId;
private const string ResyncAckTimerKey = "sf-resync-ack-timeout";
/// <summary>How long the active node waits for the standby's <see cref="SfBufferResyncAck"/>
/// before warning + counting the resync as unacknowledged (lost chunks / dead peer). Ctor
/// test seam; production default 60 s.</summary>
private readonly TimeSpan _resyncAckTimeout;
/// <summary>
/// Maximum rows an active node returns in a single anti-entropy resync snapshot.
/// A standby whose buffer exceeded this (Truncated snapshot) resyncs the oldest
/// 10 000 rows; further divergence drains naturally as the active node delivers.
/// </summary>
private const int MaxResyncRows = 10_000;
/// <summary>
/// Estimated per-chunk payload budget for a resync snapshot. Akka remoting's default
/// <c>maximum-frame-size</c> is 128 000 bytes and <c>BuildHocon</c> sets no override,
/// so the monolithic <see cref="SfBufferSnapshot"/> is silently undeliverable for any
/// realistic backlog (review 02 round 2, N2). 64 000 bytes leaves ≈50% headroom for
/// the JSON envelope, CLR type manifests, and the non-payload columns.
/// </summary>
internal const int MaxResyncChunkBytes = 64_000;
/// <summary>Row cap per resync chunk (bounds a chunk even when every row is tiny).</summary>
internal const int MaxResyncChunkRows = 200;
/// <summary>
/// Splits a resync snapshot into chunks that fit Akka remoting's default
/// 128 000-byte frame (review 02 round 2, N2): rows accumulate until the estimated
/// payload budget or the row cap is hit. Estimation is payload-dominated
/// (payload_json length + 512 bytes fixed overhead per row); a single row whose
/// payload exceeds the budget ships alone (Warning at the call site). Order is
/// preserved (oldest-first, matching GetAllMessagesAsync).
/// </summary>
internal static List<List<StoreAndForwardMessage>> ChunkForRemoting(
IReadOnlyList<StoreAndForwardMessage> rows, int maxChunkBytes, int maxChunkRows)
{
var chunks = new List<List<StoreAndForwardMessage>>();
var current = new List<StoreAndForwardMessage>();
var currentBytes = 0;
foreach (var row in rows)
{
var estimate = (row.PayloadJson?.Length ?? 0) + 512;
if (current.Count > 0 && (currentBytes + estimate > maxChunkBytes || current.Count >= maxChunkRows))
{
chunks.Add(current);
current = new List<StoreAndForwardMessage>();
currentBytes = 0;
}
current.Add(row);
currentBytes += estimate;
}
if (current.Count > 0) chunks.Add(current);
return chunks;
}
/// <summary>
/// Initializes a new <see cref="SiteReplicationActor"/> and registers Akka message handlers.
/// </summary>
/// <param name="storage">Service for accessing local site storage.</param>
/// <param name="sfStorage">Store-and-forward SQLite storage for replication of buffered messages.</param>
/// <param name="replicationService">Service providing replication transport logic.</param>
/// <param name="siteRole">Akka cluster role used to identify peer nodes to replicate to.</param>
/// <param name="logger">Logger instance.</param>
/// <param name="configFetcher">
/// Fetches a deployed instance's config JSON from central over HTTP. Used by the
/// notify-and-fetch standby apply path (<see cref="HandleApplyConfigDeploy"/>): the peer
/// replicates only the deployment id, and the standby fetches the config itself so a large
/// config never crosses the intra-site Akka hop. Null on nodes/tests without a fetcher.
/// </param>
/// <param name="isActiveOverride">
/// Active-node check that gates the buffer-resync roles (a standby requests a
/// resync, the active node answers). Production wiring passes the Host's
/// <c>IClusterNodeProvider.SelfIsPrimary</c> delegate (the same instance gating the
/// S&amp;F delivery sweep); null falls back to the shared oldest-Up evaluator
/// (<see cref="Communication.ClusterState.ActiveNodeEvaluator"/>).
/// </param>
/// <param name="options">Site runtime options, including the config-fetch retry count; production defaults apply when null.</param>
/// <param name="configFetchRetryDelay">Delay between config-fetch retry attempts; defaults to 2 seconds when null.</param>
public SiteReplicationActor(
SiteStorageService storage,
StoreAndForwardStorage sfStorage,
ReplicationService replicationService,
string siteRole,
ILogger<SiteReplicationActor> logger,
IDeploymentConfigFetcher? configFetcher = null,
Func<bool>? isActiveOverride = null,
SiteRuntimeOptions? options = null,
TimeSpan? configFetchRetryDelay = null,
TimeSpan? resyncAssemblyTimeout = null,
TimeSpan? resyncAckTimeout = null)
{
_storage = storage;
_sfStorage = sfStorage;
_replicationService = replicationService;
_configFetcher = configFetcher;
_siteRole = siteRole;
_logger = logger;
_cluster = Cluster.Get(Context.System);
_isActive = isActiveOverride ?? DefaultIsActive;
_resyncAssemblyTimeout = resyncAssemblyTimeout ?? TimeSpan.FromSeconds(30);
_resyncAckTimeout = resyncAckTimeout ?? TimeSpan.FromSeconds(60);
// UA2: bound the standby's replicated-config fetch retries. At least one
// attempt always runs; the fixed inter-attempt delay is a test seam
// (production default 2 s).
_configFetchRetryCount = Math.Max(1, options?.ConfigFetchRetryCount ?? 1);
_configFetchRetryDelay = configFetchRetryDelay ?? TimeSpan.FromSeconds(2);
// Cluster member events
Receive<ClusterEvent.MemberUp>(HandleMemberUp);
Receive<ClusterEvent.MemberRemoved>(HandleMemberRemoved);
Receive<ClusterEvent.CurrentClusterState>(HandleCurrentClusterState);
// Outbound — forward to peer
Receive<ReplicateConfigDeploy>(msg => SendToPeer(new ApplyConfigDeploy(
msg.InstanceName, msg.DeploymentId, msg.RevisionHash, msg.IsEnabled,
msg.CentralFetchBaseUrl, msg.FetchToken)));
Receive<ReplicateConfigRemove>(msg => SendToPeer(new ApplyConfigRemove(msg.InstanceName)));
Receive<ReplicateConfigSetEnabled>(msg => SendToPeer(new ApplyConfigSetEnabled(
msg.InstanceName, msg.IsEnabled)));
Receive<ReplicateArtifacts>(msg => SendToPeer(new ApplyArtifacts(msg.Command)));
Receive<ReplicateStoreAndForward>(msg => SendToPeer(new ApplyStoreAndForward(msg.Operation)));
// Inbound — apply from peer
Receive<ApplyConfigDeploy>(HandleApplyConfigDeploy);
Receive<ApplyConfigRemove>(HandleApplyConfigRemove);
Receive<ApplyConfigSetEnabled>(HandleApplyConfigSetEnabled);
Receive<ApplyArtifacts>(HandleApplyArtifacts);
Receive<ApplyStoreAndForward>(HandleApplyStoreAndForward);
// Anti-entropy — full S&F buffer resync on peer (re)join
Receive<RequestSfBufferResync>(HandleRequestSfBufferResync);
Receive<SfResyncSnapshotLoaded>(HandleSfResyncSnapshotLoaded);
Receive<SfBufferSnapshotChunk>(HandleSfBufferSnapshotChunk);
Receive<ResyncAssemblyTimedOut>(HandleResyncAssemblyTimedOut);
Receive<SfBufferResyncAck>(msg =>
{
if (msg.ResyncId == _pendingAckResyncId)
{
_pendingAckResyncId = null;
Timers.Cancel(ResyncAckTimerKey);
}
ScadaBridgeTelemetry.RecordSfResyncCompleted();
_logger.LogInformation("S&F resync {ResyncId} acknowledged by standby: {Rows} row(s) applied",
msg.ResyncId, msg.RowCount);
});
Receive<ResyncAckTimedOut>(msg =>
{
if (msg.ResyncId != _pendingAckResyncId) return;
_pendingAckResyncId = null;
ScadaBridgeTelemetry.RecordSfResyncAckMissing();
_logger.LogWarning(
"S&F resync {ResyncId} was never acknowledged within {Window} — snapshot chunks may have been lost (frame drop / dead peer); the next peer-track retries",
msg.ResyncId, _resyncAckTimeout);
});
Receive<SfBufferSnapshot>(HandleSfBufferSnapshot); // legacy monolithic handler — retained for rolling compat
}
/// <inheritdoc />
protected override void PreStart()
{
base.PreStart();
_cluster.Subscribe(Self, ClusterEvent.SubscriptionInitialStateMode.InitialStateAsSnapshot,
typeof(ClusterEvent.MemberUp),
typeof(ClusterEvent.MemberRemoved));
_logger.LogInformation("SiteReplicationActor started, subscribing to cluster events for role {Role}", _siteRole);
}
/// <inheritdoc />
protected override void PostStop()
{
_cluster.Unsubscribe(Self);
base.PostStop();
}
private void HandleCurrentClusterState(ClusterEvent.CurrentClusterState state)
{
foreach (var member in state.Members)
{
if (member.Status == MemberStatus.Up)
TryTrackPeer(member);
}
}
private void HandleMemberUp(ClusterEvent.MemberUp evt)
{
TryTrackPeer(evt.Member);
}
private void HandleMemberRemoved(ClusterEvent.MemberRemoved evt)
{
if (evt.Member.Address.Equals(_peerAddress))
{
_logger.LogInformation("Peer node removed: {Address}", _peerAddress);
_peerAddress = null;
}
}
private void TryTrackPeer(Member member)
{
// Must have our site role, and must not be self
if (member.HasRole(_siteRole) && !member.Address.Equals(_cluster.SelfAddress))
{
_peerAddress = member.Address;
_logger.LogInformation("Peer node tracked: {Address}", _peerAddress);
OnPeerTracked();
}
}
/// <summary>
/// Side-effect run whenever a peer is (re)tracked. A <b>standby</b> requests a
/// full S&amp;F buffer snapshot for anti-entropy resync — this closes the "a
/// standby down for an hour rejoins and diverges forever" gap: it may have missed
/// replicated Add/Remove/Park ops while it was gone. The active node never
/// requests. <see langword="protected virtual"/> so tests can drive it without a
/// real two-node cluster.
/// </summary>
protected virtual void OnPeerTracked()
{
if (!SafeIsActive())
{
SendToPeer(new RequestSfBufferResync());
}
}
/// <summary>
/// Repo-standard active-node check: this node is active when it is the OLDEST Up
/// member carrying the site role — the same oldest-Up semantics as the S&F delivery
/// gate (IClusterNodeProvider.SelfIsPrimary → ClusterActivityEvaluator → shared
/// ActiveNodeEvaluator). NEVER the cluster leader: leadership is lowest-address and
/// diverges from singleton/delivery placement permanently after the lower-address
/// node restarts — the divergence that made the delivering node wipe its own live
/// buffer via a wrong-direction resync (review 02 round 2, N1 Critical). Any other
/// state reports standby — safe-by-default.
/// </summary>
private bool DefaultIsActive() =>
Communication.ClusterState.ActiveNodeEvaluator.SelfIsOldestUp(_cluster, _siteRole);
/// <summary>
/// Evaluates the active-node check, treating a throwing check as standby
/// (safe-by-default: a standby never delivers or answers resyncs).
/// </summary>
private bool SafeIsActive()
{
try
{
return _isActive();
}
catch (Exception ex)
{
_logger.LogWarning(ex, "Active-node check threw; treating node as standby");
return false;
}
}
/// <summary>
/// Forwards a replication message to the tracked peer node's <c>site-replication</c> actor
/// (fire-and-forget, dropped when no peer is tracked). <see langword="protected virtual"/>
/// so tests can intercept the peer send without standing up a real two-node cluster.
/// </summary>
/// <param name="message">The replication message to forward to the peer.</param>
protected virtual void SendToPeer(object message)
{
if (_peerAddress == null)
{
// A dropped op is a lost delta — surface it at Warning + a metric so a
// never-tracked peer (a dead standby) is visible, not silent (arch
// review 02). In single-node dev the peer is legitimately absent; the
// per-op warning is rate-tolerable (accepted per review).
ScadaBridgeTelemetry.RecordReplicationFailure();
_logger.LogWarning("No peer available, dropping replication message {Type}", message.GetType().Name);
return;
}
var path = new RootActorPath(_peerAddress) / "user" / "site-replication";
Context.ActorSelection(path).Tell(message);
}
// ── Inbound handlers ──
private void HandleApplyConfigDeploy(ApplyConfigDeploy msg)
{
if (string.IsNullOrEmpty(msg.CentralFetchBaseUrl))
{
// The direct DeployInstanceCommand cross-cluster wire path was retired.
// This guard is a defensive fallback: skip quietly rather than calling FetchAsync("")
// and logging a spurious error. Reconciliation backstops any missed writes.
_logger.LogDebug(
"No fetch coords for {Instance} (deployment {DeploymentId}) — skipping replicated fetch; T18 reconciliation is the backstop",
msg.InstanceName, msg.DeploymentId);
return;
}
if (_configFetcher is null)
{
_logger.LogWarning(
"No config fetcher available; cannot apply replicated config for {Instance} (deployment {DeploymentId}) — reconciliation will backstop",
msg.InstanceName, msg.DeploymentId);
return;
}
_logger.LogInformation(
"Replicating config for {Instance} (deployment {DeploymentId}) — fetching from central",
msg.InstanceName, msg.DeploymentId);
// Notify-and-fetch: the peer sent only the id, so the standby fetches the config
// itself (off-thread; best-effort fire-and-forget, matching the no-ack replication
// model). The guarded write only overwrites a strictly-older local row. The fetch
// is retried up to ConfigFetchRetryCount times with a fixed delay (UA2) — a transient
// central hiccup no longer defers to the slower reconciliation backstop, which still
// covers a total failure after the last attempt.
_ = FetchWithRetryAsync();
return;
async Task FetchWithRetryAsync()
{
for (var attempt = 1; attempt <= _configFetchRetryCount; attempt++)
{
try
{
// Non-null: the outer method returns early when _configFetcher is null.
var json = await _configFetcher!.FetchAsync(
msg.CentralFetchBaseUrl, msg.DeploymentId, msg.FetchToken, CancellationToken.None);
await _storage.StoreDeployedConfigIfNewerAsync(
msg.InstanceName, json, msg.DeploymentId, msg.RevisionHash, msg.IsEnabled);
return;
}
catch (DeploymentConfigFetchException fex) when (fex.IsSuperseded)
{
// A superseded/expired fetch never heals by retrying — a newer deploy
// will replicate its own id.
_logger.LogInformation(
"Skip replicated config for {Instance}: superseded/expired (a newer deploy will replicate)",
msg.InstanceName);
return;
}
catch (Exception ex)
{
if (attempt < _configFetchRetryCount)
{
_logger.LogWarning(ex,
"Replicated config fetch attempt {Attempt}/{Max} failed for {Instance} (deployment {DeploymentId}) — retrying in {Delay}",
attempt, _configFetchRetryCount, msg.InstanceName, msg.DeploymentId, _configFetchRetryDelay);
await Task.Delay(_configFetchRetryDelay);
}
else
{
_logger.LogError(ex,
"Replicated config fetch failed after {Attempts} attempt(s) for {Instance} (deployment {DeploymentId}) — reconciliation will backstop",
_configFetchRetryCount, msg.InstanceName, msg.DeploymentId);
}
}
}
}
}
private void HandleApplyConfigRemove(ApplyConfigRemove msg)
{
_logger.LogInformation("Applying replicated config remove for {Instance}", msg.InstanceName);
_storage.RemoveDeployedConfigAsync(msg.InstanceName)
.ContinueWith(t =>
{
if (t.IsFaulted)
_logger.LogError(t.Exception, "Failed to apply replicated remove for {Instance}", msg.InstanceName);
});
}
private void HandleApplyConfigSetEnabled(ApplyConfigSetEnabled msg)
{
_logger.LogInformation("Applying replicated set-enabled={Enabled} for {Instance}", msg.IsEnabled, msg.InstanceName);
_storage.SetInstanceEnabledAsync(msg.InstanceName, msg.IsEnabled)
.ContinueWith(t =>
{
if (t.IsFaulted)
_logger.LogError(t.Exception, "Failed to apply replicated set-enabled for {Instance}", msg.InstanceName);
});
}
private void HandleApplyArtifacts(ApplyArtifacts msg)
{
var command = msg.Command;
_logger.LogInformation("Applying replicated artifacts, deploymentId={DeploymentId}", command.DeploymentId);
Task.Run(async () =>
{
try
{
if (command.SharedScripts != null)
foreach (var s in command.SharedScripts)
await _storage.StoreSharedScriptAsync(s.Name, s.Code, s.ParameterDefinitions, s.ReturnDefinition);
if (command.ExternalSystems != null)
foreach (var es in command.ExternalSystems)
await _storage.StoreExternalSystemAsync(es.Name, es.EndpointUrl, es.AuthType, es.AuthConfiguration, es.MethodDefinitionsJson, es.TimeoutSeconds);
if (command.DatabaseConnections != null)
foreach (var db in command.DatabaseConnections)
await _storage.StoreDatabaseConnectionAsync(db.Name, db.ConnectionString, db.MaxRetries, db.RetryDelay);
// Notification lists and SMTP
// configuration are central-only and are never persisted on a site.
// Mirror the primary apply path: purge any pre-fix rows (including the
// plaintext SMTP password) instead of writing the command's
// (now-always-null) NotificationLists/SmtpConfigurations.
await _storage.PurgeCentralOnlyNotificationConfigAsync();
if (command.DataConnections != null)
foreach (var dc in command.DataConnections)
await _storage.StoreDataConnectionDefinitionAsync(dc.Name, dc.Protocol, dc.PrimaryConfigurationJson, dc.BackupConfigurationJson, dc.FailoverRetryCount);
}
catch (Exception ex)
{
_logger.LogError(ex, "Failed to apply replicated artifacts");
}
});
}
private void HandleApplyStoreAndForward(ApplyStoreAndForward msg)
{
_logger.LogDebug("Applying replicated S&F operation {OpType} for message {Id}",
msg.Operation.OperationType, msg.Operation.MessageId);
_replicationService.ApplyReplicatedOperationAsync(msg.Operation, _sfStorage)
.ContinueWith(t =>
{
if (t.IsFaulted)
_logger.LogError(t.Exception, "Failed to apply replicated S&F operation {Id}", msg.Operation.MessageId);
});
}
/// <summary>
/// Active-node side of the anti-entropy resync: answers a standby's
/// <see cref="RequestSfBufferResync"/> with a sequence of byte-budgeted
/// <see cref="SfBufferSnapshotChunk"/>s (up to <see cref="MaxResyncRows"/> oldest rows).
/// A non-active node ignores the request — only the authoritative node may answer.
/// The snapshot is piped back to Self so chunking + ack bookkeeping stays actor-safe.
/// </summary>
private void HandleRequestSfBufferResync(RequestSfBufferResync msg)
{
if (!SafeIsActive())
{
_logger.LogDebug("Ignoring S&F buffer resync request — this node is not active");
return;
}
var replyTo = Sender;
_sfStorage.GetAllMessagesAsync(MaxResyncRows).PipeTo(
Self,
failure: ex => new Status.Failure(ex),
success: result => new SfResyncSnapshotLoaded(replyTo, result.Messages, result.Truncated));
}
/// <summary>
/// Active-node continuation: the resync snapshot finished loading; chunk it to fit the
/// remoting frame and send the sequenced chunks to the requester (all sharing one
/// resyncId). Task 7 arms the ack-timeout here.
/// </summary>
private void HandleSfResyncSnapshotLoaded(SfResyncSnapshotLoaded msg)
{
var chunks = ChunkForRemoting(msg.Messages, MaxResyncChunkBytes, MaxResyncChunkRows);
if (chunks.Count == 0) chunks.Add(new List<StoreAndForwardMessage>()); // empty buffer still resyncs (clears the standby)
var resyncId = Guid.NewGuid().ToString("N");
for (var i = 0; i < chunks.Count; i++)
{
if (chunks[i].Count == 1 && (chunks[i][0].PayloadJson?.Length ?? 0) + 512 > MaxResyncChunkBytes)
_logger.LogWarning(
"Resync row {Id} alone exceeds the chunk budget ({Bytes}B payload); sending solo — it may exceed the remoting frame",
chunks[i][0].Id, chunks[i][0].PayloadJson?.Length ?? 0);
msg.ReplyTo.Tell(new SfBufferSnapshotChunk(resyncId, i + 1, chunks.Count, chunks[i], msg.Truncated), Self);
}
_logger.LogInformation(
"Answered S&F resync request with {Rows} row(s) in {Chunks} chunk(s), resyncId={ResyncId}",
msg.Messages.Count, chunks.Count, resyncId);
// Arm the ack window: absence of an SfBufferResyncAck within it surfaces as a
// Warning + counter (the silent-loss mode N2 flagged). Single-outstanding-resync
// bookkeeping: a new request supersedes by overwriting the id and restarting the timer.
_pendingAckResyncId = resyncId;
Timers.StartSingleTimer(ResyncAckTimerKey, new ResyncAckTimedOut(resyncId), _resyncAckTimeout);
}
/// <summary>
/// Standby-node side of the anti-entropy resync: replaces the local buffer
/// wholesale with the active node's snapshot. Combined with the upsert-based
/// replicated applies (arch review 02), any replicated op that lands
/// after this resync merges cleanly onto the resynced state. An active node
/// ignores a snapshot — it is the source of truth, never a resync target.
/// </summary>
private void HandleSfBufferSnapshot(SfBufferSnapshot msg)
{
if (SafeIsActive())
{
_logger.LogDebug("Ignoring S&F buffer snapshot — this node is active");
return;
}
if (msg.Truncated)
{
_logger.LogWarning(
"S&F buffer resync snapshot truncated at {Cap} rows; divergence beyond the cap drains naturally",
MaxResyncRows);
}
_logger.LogInformation(
"Applying S&F buffer resync snapshot ({Count} rows), replacing local buffer", msg.Messages.Count);
Task.Run(async () =>
{
// Belt-and-braces (N1): re-check at apply time. ReplaceAllAsync discards
// every in-flight row (StoreAndForwardStorage.cs "Never call on an active
// node"); a flip between message receipt and this point must abort.
if (SafeIsActive())
{
_logger.LogWarning(
"Discarding S&F buffer resync snapshot: this node became active before apply");
return;
}
await _sfStorage.ReplaceAllAsync(msg.Messages);
})
.ContinueWith(t =>
{
if (t.IsFaulted)
_logger.LogError(t.Exception, "Failed to apply S&F buffer resync snapshot");
});
}
/// <summary>
/// Standby-node side of the chunked anti-entropy resync: accumulates the chunks of one
/// <c>ResyncId</c>, and once all have arrived, assembles them in sequence order and
/// replaces the local buffer atomically, then acks. A new <c>ResyncId</c> discards any
/// stale partial assembly (review 02 round 2, N2). An active node ignores chunks.
/// </summary>
private void HandleSfBufferSnapshotChunk(SfBufferSnapshotChunk msg)
{
if (SafeIsActive())
{
_logger.LogDebug("Ignoring S&F resync chunk — this node is active");
return;
}
if (_assemblingResyncId != msg.ResyncId)
{
if (_assemblingResyncId != null)
_logger.LogWarning(
"Discarding partial S&F resync assembly {Old} ({Have}/{Want} chunks): a new resync {New} superseded it",
_assemblingResyncId, _assemblingChunks.Count, _assemblingTotalChunks, msg.ResyncId);
_assemblingResyncId = msg.ResyncId;
_assemblingTotalChunks = msg.TotalChunks;
_assemblingTruncated = msg.Truncated;
_assemblingChunks.Clear();
}
_assemblingChunks[msg.Sequence] = msg.Messages;
Timers.StartSingleTimer(ResyncAssemblyTimerKey, new ResyncAssemblyTimedOut(msg.ResyncId), _resyncAssemblyTimeout);
if (_assemblingChunks.Count < _assemblingTotalChunks)
return;
// Complete: assemble in sequence order and apply atomically.
var assembled = Enumerable.Range(1, _assemblingTotalChunks)
.SelectMany(seq => _assemblingChunks[seq])
.ToList();
var resyncId = _assemblingResyncId!;
var truncated = _assemblingTruncated;
_assemblingResyncId = null;
_assemblingChunks.Clear();
Timers.Cancel(ResyncAssemblyTimerKey);
if (truncated)
_logger.LogWarning(
"S&F buffer resync snapshot truncated at {Cap} rows; divergence beyond the cap drains naturally",
MaxResyncRows);
_logger.LogInformation(
"Applying chunked S&F resync {ResyncId} ({Count} rows), replacing local buffer", resyncId, assembled.Count);
// KNOWN, ACCEPTED race (review 02 round 2, N5 — do NOT "fix" this into something
// worse): a replicated Remove sent after the active node read its snapshot but
// before the snapshot's chunks is ordered BEFORE them on the wire (same
// sender/receiver pair), so this apply can re-add the removed row → an orphan
// Pending row that a later failover re-delivers ONCE. Bounded, self-correcting at
// the next resync, and inherent to no-ack replication; a delivered-side dedup or
// op-sequencing scheme would cost far more than one rare duplicate.
var replyTo = Sender;
Task.Run(async () =>
{
if (SafeIsActive()) // belt-and-braces, mirrors the monolithic path (T3)
{
_logger.LogWarning("Discarding chunked S&F resync {ResyncId}: node became active before apply", resyncId);
return;
}
await _sfStorage.ReplaceAllAsync(assembled);
replyTo.Tell(new SfBufferResyncAck(resyncId, assembled.Count));
})
.ContinueWith(t =>
{
if (t.IsFaulted)
_logger.LogError(t.Exception, "Failed to apply chunked S&F resync {ResyncId}", resyncId);
});
}
private void HandleResyncAssemblyTimedOut(ResyncAssemblyTimedOut msg)
{
if (_assemblingResyncId != msg.ResyncId) return; // superseded already
_logger.LogWarning(
"S&F resync assembly {ResyncId} timed out with {Have}/{Want} chunks — discarding partial (a lost chunk; the next peer-track retries)",
msg.ResyncId, _assemblingChunks.Count, _assemblingTotalChunks);
ScadaBridgeTelemetry.RecordReplicationFailure();
_assemblingResyncId = null;
_assemblingChunks.Clear();
}
/// <summary>Internal: the resync snapshot finished loading; chunk and send to the requester.</summary>
internal sealed record SfResyncSnapshotLoaded(
IActorRef ReplyTo, List<StoreAndForwardMessage> Messages, bool Truncated);
/// <summary>Internal: a partial chunk assembly for <paramref name="ResyncId"/> exceeded its window.</summary>
internal sealed record ResyncAssemblyTimedOut(string ResyncId);
/// <summary>Internal: the active node's ack window for <paramref name="ResyncId"/> expired.</summary>
internal sealed record ResyncAckTimedOut(string ResyncId);
}
/// <summary>
/// Standby→active: request a full S&amp;F buffer snapshot for anti-entropy resync
/// (sent when a standby (re)tracks a peer). Crosses Akka remoting between the two
/// site nodes; the POCO rides the default serializer.
/// </summary>
public sealed record RequestSfBufferResync;
/// <summary>
/// Active→standby: full-buffer snapshot. <paramref name="Truncated"/> is true when
/// the active node's buffer exceeded <c>MaxResyncRows</c> (the standby logs a Warning —
/// divergence beyond the cap drains naturally as the active node delivers). Crosses
/// Akka remoting; the message list rides the default serializer.
/// </summary>
public sealed record SfBufferSnapshot(List<StoreAndForwardMessage> Messages, bool Truncated);
/// <summary>
/// Active→standby: one sequenced chunk of a full-buffer anti-entropy snapshot
/// (review 02 round 2, N2 — the monolithic <see cref="SfBufferSnapshot"/> exceeds Akka
/// remoting's default 128 000-byte frame for any realistic backlog). All chunks of one
/// resync share <paramref name="ResyncId"/>; <paramref name="Sequence"/> is 1-based up to
/// <paramref name="TotalChunks"/>. Additive message — the legacy monolithic snapshot
/// handler is retained for rolling upgrades. Crosses intra-site Akka remoting (NOT
/// ClusterClient — ClusterClientContractLockTests is intentionally not involved).
/// </summary>
public sealed record SfBufferSnapshotChunk(
string ResyncId, int Sequence, int TotalChunks,
List<StoreAndForwardMessage> Messages, bool Truncated);
/// <summary>
/// Standby→active: delivery confirmation — the standby assembled all chunks of
/// <paramref name="ResyncId"/> and applied them atomically (<paramref name="RowCount"/>
/// rows installed). Absence within the ack window is surfaced by the active node
/// (Warning + counter) — the silent-loss mode N2 flagged.
/// </summary>
public sealed record SfBufferResyncAck(string ResyncId, int RowCount);
src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Messages/ReplicationMessages.cs
-44
View File
@@ -1,44 +0,0 @@
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts;
using ZB.MOM.WW.ScadaBridge.StoreAndForward;
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Messages;
// Outbound messages — sent by local DeploymentManagerActor/S&F service
// to the local SiteReplicationActor for forwarding to the peer node.
/// <summary>Outbound: tell the peer to fetch+apply a deployed instance config by id (notify-and-fetch; no inline config).</summary>
public record ReplicateConfigDeploy(
string InstanceName, string DeploymentId, string RevisionHash, bool IsEnabled,
string CentralFetchBaseUrl, string FetchToken);
/// <summary>Outbound: replicate removal of a deployed instance config to the peer node.</summary>
public record ReplicateConfigRemove(string InstanceName);
/// <summary>Outbound: replicate an instance enabled/disabled flag change to the peer node.</summary>
public record ReplicateConfigSetEnabled(string InstanceName, bool IsEnabled);
/// <summary>Outbound: replicate a system-wide artifact deployment (shared scripts, external systems, etc.) to the peer node.</summary>
public record ReplicateArtifacts(DeployArtifactsCommand Command);
/// <summary>Outbound: replicate a store-and-forward buffer mutation (enqueue/dequeue/park/etc.) to the peer node.</summary>
public record ReplicateStoreAndForward(ReplicationOperation Operation);
// Inbound messages — received from the peer's SiteReplicationActor
// and applied to local SQLite storage.
/// <summary>Inbound: peer-replicated config deploy — the standby fetches the config by id and writes it (guarded).</summary>
public record ApplyConfigDeploy(
string InstanceName, string DeploymentId, string RevisionHash, bool IsEnabled,
string CentralFetchBaseUrl, string FetchToken);
/// <summary>Inbound: apply peer-replicated removal of a deployed instance config to local SQLite.</summary>
public record ApplyConfigRemove(string InstanceName);
/// <summary>Inbound: apply a peer-replicated instance enabled/disabled flag change to local SQLite.</summary>
public record ApplyConfigSetEnabled(string InstanceName, bool IsEnabled);
/// <summary>Inbound: apply a peer-replicated system-wide artifact deployment to local SQLite.</summary>
public record ApplyArtifacts(DeployArtifactsCommand Command);
/// <summary>Inbound: apply a peer-replicated store-and-forward buffer mutation to the local buffer.</summary>
public record ApplyStoreAndForward(ReplicationOperation Operation);
src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageSchema.cs
+169
View File
@@ -0,0 +1,169 @@
using Microsoft.Data.Sqlite;
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
/// <summary>
/// DDL for the site's nine configuration tables, extracted from
/// <see cref="SiteStorageService"/> so it can be applied by whoever owns the database
/// file.
/// <para>
/// Deliberately depends only on <c>Microsoft.Data.Sqlite</c>, not on the LocalDb library.
/// The Host applies this DDL to a LocalDb-managed connection before
/// <c>RegisterReplicated</c> installs the capture triggers; nothing about the schema
/// itself is LocalDb-specific, and the service still calls it so a directly-constructed
/// service (tests, tooling) remains self-sufficient.
/// </para>
/// <para>
/// Journal-mode and other connection pragmas are deliberately NOT set here — LocalDb owns
/// the connection's pragmas, and <see cref="SiteStorageService"/> keeps its own WAL set-up
/// for the databases it opens itself.
/// </para>
/// </summary>
public static class SiteStorageSchema
{
/// <summary>
/// Creates the nine site configuration tables when absent, and additively upgrades
/// tables created by an older build. Idempotent — safe to run on every startup.
/// </summary>
/// <param name="connection">An open connection to the database that should hold the tables.</param>
public static void Apply(SqliteConnection connection)
{
ArgumentNullException.ThrowIfNull(connection);
using (var command = connection.CreateCommand())
{
command.CommandText = @"
CREATE TABLE IF NOT EXISTS deployed_configurations (
instance_unique_name TEXT PRIMARY KEY,
config_json TEXT NOT NULL,
deployment_id TEXT NOT NULL,
revision_hash TEXT NOT NULL,
is_enabled INTEGER NOT NULL DEFAULT 1,
deployed_at TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS static_attribute_overrides (
instance_unique_name TEXT NOT NULL,
attribute_name TEXT NOT NULL,
override_value TEXT NOT NULL,
updated_at TEXT NOT NULL,
PRIMARY KEY (instance_unique_name, attribute_name)
);
CREATE TABLE IF NOT EXISTS shared_scripts (
name TEXT PRIMARY KEY,
code TEXT NOT NULL,
parameter_definitions TEXT,
return_definition TEXT,
updated_at TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS external_systems (
name TEXT PRIMARY KEY,
endpoint_url TEXT NOT NULL,
auth_type TEXT NOT NULL,
auth_configuration TEXT,
method_definitions TEXT,
updated_at TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS database_connections (
name TEXT PRIMARY KEY,
connection_string TEXT NOT NULL,
max_retries INTEGER NOT NULL DEFAULT 3,
retry_delay_ms INTEGER NOT NULL DEFAULT 1000,
updated_at TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS notification_lists (
name TEXT PRIMARY KEY,
recipient_emails TEXT NOT NULL,
updated_at TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS data_connection_definitions (
name TEXT PRIMARY KEY,
protocol TEXT NOT NULL,
configuration TEXT,
backup_configuration TEXT,
failover_retry_count INTEGER NOT NULL DEFAULT 3,
updated_at TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS smtp_configurations (
name TEXT PRIMARY KEY,
server TEXT NOT NULL,
port INTEGER NOT NULL,
auth_mode TEXT NOT NULL,
from_address TEXT NOT NULL,
username TEXT,
password TEXT,
oauth_config TEXT,
updated_at TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS native_alarm_state (
instance_unique_name TEXT NOT NULL,
source_canonical_name TEXT NOT NULL,
source_reference TEXT NOT NULL,
condition_json TEXT NOT NULL,
last_transition_at TEXT NOT NULL,
metadata_json TEXT,
PRIMARY KEY (instance_unique_name, source_canonical_name, source_reference)
);
";
command.ExecuteNonQuery();
}
// Schema migrations — add columns that may not exist on older databases
MigrateSchema(connection);
}
private static void MigrateSchema(SqliteConnection connection)
{
// Add backup_configuration and failover_retry_count to data_connection_definitions
// (added in primary/backup data connections feature)
AddColumnIfMissing(connection, "data_connection_definitions", "backup_configuration", "TEXT");
AddColumnIfMissing(connection, "data_connection_definitions", "failover_retry_count", "INTEGER NOT NULL DEFAULT 3");
// Native-alarm display metadata (UA4) — restored on rehydration so a persisted condition
// renders fully (type/category/message/values) before the first source snapshot arrives.
AddColumnIfMissing(connection, "native_alarm_state", "metadata_json", "TEXT");
// Per-external-system call timeout (ExternalSystemGateway Timeout) — carried through the
// artifact pipeline so site-side calls honor it; 0 = use the site default.
AddColumnIfMissing(connection, "external_systems", "timeout_seconds", "INTEGER NOT NULL DEFAULT 0");
}
/// <summary>
/// Additively adds a column only when it is not already present. SQLite lacks
/// <c>ADD COLUMN IF NOT EXISTS</c>, so the schema is probed via
/// <c>PRAGMA table_info</c> first. Mirrors <c>OperationTrackingSchema</c>.
/// <para>
/// This replaces the previous try/catch on <c>SqliteException</c> message text
/// containing "duplicate column": probing is the same precedent the other schema
/// classes use, does not depend on an error string that is not part of SQLite's
/// contract, and does not swallow unrelated failures of the same exception type.
/// </para>
/// </summary>
private static void AddColumnIfMissing(
SqliteConnection connection, string table, string columnName, string columnDefinition)
{
using (var probe = connection.CreateCommand())
{
// Table + column names are caller-controlled constants, never user input —
// safe to interpolate (parameters are not permitted in DDL or in a
// pragma-function argument).
probe.CommandText = $"SELECT COUNT(*) FROM pragma_table_info('{table}') WHERE name = $name";
probe.Parameters.AddWithValue("$name", columnName);
if (Convert.ToInt32(probe.ExecuteScalar()) > 0)
{
return;
}
}
using var alter = connection.CreateCommand();
alter.CommandText = $"ALTER TABLE {table} ADD COLUMN {columnName} {columnDefinition}";
alter.ExecuteNonQuery();
}
}
src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Persistence/SiteStorageService.cs
+78 -205
View File
@@ -1,5 +1,6 @@
using Microsoft.Data.Sqlite; using Microsoft.Data.Sqlite;
using Microsoft.Extensions.Logging; using Microsoft.Extensions.Logging;
using ZB.MOM.WW.LocalDb;
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
@@ -10,189 +11,68 @@ namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
/// </summary> /// </summary>
public class SiteStorageService public class SiteStorageService
{ {
private readonly string _connectionString; private readonly ILocalDb _localDb;
private readonly ILogger<SiteStorageService> _logger; private readonly ILogger<SiteStorageService> _logger;
/// <summary> /// <summary>
/// Initializes a new instance of the SiteStorageService with the specified SQLite connection string and logger. /// Initializes the service over the consolidated site database.
/// </summary> /// </summary>
/// <param name="connectionString">SQLite connection string for the site database.</param> /// <param name="localDb">
/// The consolidated site database. Every connection it hands out is already open and carries
/// the per-connection pragmas (including the busy timeout that this class used to pin itself)
/// plus the <c>zb_hlc_next()</c> UDF that the config tables' capture triggers call — which is
/// exactly why the service no longer builds its own connection string. A raw connection would
/// lack the UDF and every write to a replicated table would fail closed.
/// </param>
/// <param name="logger">Logger instance for diagnostic messages.</param> /// <param name="logger">Logger instance for diagnostic messages.</param>
public SiteStorageService(string connectionString, ILogger<SiteStorageService> logger) public SiteStorageService(ILocalDb localDb, ILogger<SiteStorageService> logger)
{ {
// Normalize the connection string and pin a busy-timeout floor (S8). WAL lets a reader ArgumentNullException.ThrowIfNull(localDb);
// and a writer proceed concurrently, but two writers still contend; Microsoft.Data.Sqlite ArgumentNullException.ThrowIfNull(logger);
// drives its busy handler off the command timeout, so a floor of 5 s means a briefly _localDb = localDb;
// busy database is waited-on rather than failing fast with SQLITE_BUSY. Only raise a
// caller-supplied value that is lower than the floor (the library default of 30 s stays).
var builder = new SqliteConnectionStringBuilder(connectionString);
if (builder.DefaultTimeout < BusyTimeoutFloorSeconds)
builder.DefaultTimeout = BusyTimeoutFloorSeconds;
_connectionString = builder.ToString();
_logger = logger; _logger = logger;
} }
/// <summary>Busy-timeout floor in seconds applied to the site connection string (S8).</summary> /// <summary>
private const int BusyTimeoutFloorSeconds = 5; /// Returns an <b>already-open</b> connection against the site database.
/// Exposed so site-local repositories can get their own connections without reaching
/// into private state via reflection. The caller owns the connection and is
/// responsible for disposing it.
/// </summary>
/// <remarks>
/// <b>Contract change:</b> this used to return an <i>unopened</i> connection that the
/// caller opened. LocalDb hands out connections already open, pragma-configured, and
/// carrying the <c>zb_hlc_next()</c> UDF — calling <c>Open</c>/<c>OpenAsync</c> on one
/// throws. Callers must not open it.
/// </remarks>
/// <returns>An open <see cref="SqliteConnection"/> against the site database.</returns>
public SqliteConnection CreateConnection() => _localDb.CreateConnection();
/// <summary> /// <summary>
/// Creates a new (unopened) SQLite connection against the site database. /// Convenience alias used internally, mirroring the other LocalDb-backed stores.
/// Exposed so site-local repositories can open their own connections without
/// reaching into private state via reflection. The caller owns
/// the connection and is responsible for opening and disposing it.
/// <para>
/// The database runs in WAL journal mode (set once in <see cref="InitializeAsync"/>) with a
/// busy-timeout floor (see the constructor), so connections handed out here inherit
/// concurrent-reader/writer behavior and wait out a briefly-busy database instead of
/// failing with SQLITE_BUSY.
/// </para>
/// </summary> /// </summary>
/// <returns>A new, unopened <see cref="SqliteConnection"/> against the site database.</returns> private SqliteConnection OpenConnection() => _localDb.CreateConnection();
public SqliteConnection CreateConnection() => new(_connectionString);
/// <summary> /// <summary>
/// Creates the SQLite tables if they do not exist. /// Creates the SQLite tables if they do not exist.
/// Called once on site startup. /// Called once on site startup.
/// </summary> /// </summary>
/// <returns>A task that completes when all tables have been created or verified.</returns> /// <returns>A task that completes when all tables have been created or verified.</returns>
public async Task InitializeAsync() public Task InitializeAsync()
{ {
await using var connection = new SqliteConnection(_connectionString); // No journal-mode pragma and no busy-timeout normalization here any more:
await connection.OpenAsync(); // LocalDb owns the file and sets WAL plus the per-connection pragmas on every
// connection it hands out.
using var connection = OpenConnection();
// Switch to WAL journal mode (S8). WAL is persistent (database-level, survives across // The DDL itself lives in SiteStorageSchema so the Host can apply it to a
// connections) so this one-time set is enough. It lets readers and a writer proceed // LocalDb-managed connection before RegisterReplicated installs the capture
// concurrently instead of every access serializing on the rollback journal. A // triggers. The service still calls it, so a directly-constructed service
// :memory: database or a filesystem that cannot support WAL falls back to its prior // (tests, tooling) remains self-sufficient.
// mode — we log the result rather than throwing. SiteStorageSchema.Apply(connection);
await using (var walCommand = connection.CreateCommand())
{
walCommand.CommandText = "PRAGMA journal_mode=WAL;";
var resultingMode = (await walCommand.ExecuteScalarAsync()) as string ?? "unknown";
if (!string.Equals(resultingMode, "wal", StringComparison.OrdinalIgnoreCase))
_logger.LogWarning(
"Site SQLite could not enable WAL journal mode (got '{Mode}') — concurrent access may serialize",
resultingMode);
}
await using var command = connection.CreateCommand(); _logger.LogInformation("Site SQLite storage initialized");
command.CommandText = @" return Task.CompletedTask;
CREATE TABLE IF NOT EXISTS deployed_configurations (
instance_unique_name TEXT PRIMARY KEY,
config_json TEXT NOT NULL,
deployment_id TEXT NOT NULL,
revision_hash TEXT NOT NULL,
is_enabled INTEGER NOT NULL DEFAULT 1,
deployed_at TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS static_attribute_overrides (
instance_unique_name TEXT NOT NULL,
attribute_name TEXT NOT NULL,
override_value TEXT NOT NULL,
updated_at TEXT NOT NULL,
PRIMARY KEY (instance_unique_name, attribute_name)
);
CREATE TABLE IF NOT EXISTS shared_scripts (
name TEXT PRIMARY KEY,
code TEXT NOT NULL,
parameter_definitions TEXT,
return_definition TEXT,
updated_at TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS external_systems (
name TEXT PRIMARY KEY,
endpoint_url TEXT NOT NULL,
auth_type TEXT NOT NULL,
auth_configuration TEXT,
method_definitions TEXT,
updated_at TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS database_connections (
name TEXT PRIMARY KEY,
connection_string TEXT NOT NULL,
max_retries INTEGER NOT NULL DEFAULT 3,
retry_delay_ms INTEGER NOT NULL DEFAULT 1000,
updated_at TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS notification_lists (
name TEXT PRIMARY KEY,
recipient_emails TEXT NOT NULL,
updated_at TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS data_connection_definitions (
name TEXT PRIMARY KEY,
protocol TEXT NOT NULL,
configuration TEXT,
backup_configuration TEXT,
failover_retry_count INTEGER NOT NULL DEFAULT 3,
updated_at TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS smtp_configurations (
name TEXT PRIMARY KEY,
server TEXT NOT NULL,
port INTEGER NOT NULL,
auth_mode TEXT NOT NULL,
from_address TEXT NOT NULL,
username TEXT,
password TEXT,
oauth_config TEXT,
updated_at TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS native_alarm_state (
instance_unique_name TEXT NOT NULL,
source_canonical_name TEXT NOT NULL,
source_reference TEXT NOT NULL,
condition_json TEXT NOT NULL,
last_transition_at TEXT NOT NULL,
metadata_json TEXT,
PRIMARY KEY (instance_unique_name, source_canonical_name, source_reference)
);
";
await command.ExecuteNonQueryAsync();
// Schema migrations — add columns that may not exist on older databases
await MigrateSchemaAsync(connection);
_logger.LogInformation("Site SQLite storage initialized at {ConnectionString}", _connectionString);
}
private async Task MigrateSchemaAsync(SqliteConnection connection)
{
// Add backup_configuration and failover_retry_count to data_connection_definitions
// (added in primary/backup data connections feature)
await TryAddColumnAsync(connection, "data_connection_definitions", "backup_configuration", "TEXT");
await TryAddColumnAsync(connection, "data_connection_definitions", "failover_retry_count", "INTEGER NOT NULL DEFAULT 3");
// Native-alarm display metadata (UA4) — restored on rehydration so a persisted condition
// renders fully (type/category/message/values) before the first source snapshot arrives.
await TryAddColumnAsync(connection, "native_alarm_state", "metadata_json", "TEXT");
// Per-external-system call timeout (ExternalSystemGateway Timeout) — carried through the
// artifact pipeline so site-side calls honor it; 0 = use the site default.
await TryAddColumnAsync(connection, "external_systems", "timeout_seconds", "INTEGER NOT NULL DEFAULT 0");
}
private async Task TryAddColumnAsync(SqliteConnection connection, string table, string column, string type)
{
try
{
await using var cmd = connection.CreateCommand();
cmd.CommandText = $"ALTER TABLE {table} ADD COLUMN {column} {type}";
await cmd.ExecuteNonQueryAsync();
_logger.LogInformation("Migrated: added column {Column} to {Table}", column, table);
}
catch (SqliteException ex) when (ex.Message.Contains("duplicate column"))
{
// Column already exists — no action needed
}
} }
// ── Deployed Configuration CRUD ── // ── Deployed Configuration CRUD ──
@@ -203,8 +83,7 @@ public class SiteStorageService
/// <returns>A task that resolves to the list of all deployed instance configurations.</returns> /// <returns>A task that resolves to the list of all deployed instance configurations.</returns>
public async Task<List<DeployedInstance>> GetAllDeployedConfigsAsync() public async Task<List<DeployedInstance>> GetAllDeployedConfigsAsync()
{ {
await using var connection = new SqliteConnection(_connectionString); await using var connection = OpenConnection();
await connection.OpenAsync();
await using var command = connection.CreateCommand(); await using var command = connection.CreateCommand();
command.CommandText = @" command.CommandText = @"
@@ -245,8 +124,7 @@ public class SiteStorageService
string revisionHash, string revisionHash,
bool isEnabled) bool isEnabled)
{ {
await using var connection = new SqliteConnection(_connectionString); await using var connection = OpenConnection();
await connection.OpenAsync();
await using var command = connection.CreateCommand(); await using var command = connection.CreateCommand();
command.CommandText = @" command.CommandText = @"
@@ -278,9 +156,22 @@ public class SiteStorageService
/// clause so the guard is atomic with no application-level read-modify-write. /// clause so the guard is atomic with no application-level read-modify-write.
/// </summary> /// </summary>
/// <remarks> /// <remarks>
/// This is the standby-node write path for replicated configs. The active-node /// <b>This is the reconciliation write path.</b> <c>SiteReconciliationActor</c> runs a
/// apply path (<see cref="StoreDeployedConfigAsync"/>) remains unguarded and always /// per-node startup self-heal against central: it asks central what this node should be
/// overwrites, because the active node's write is always authoritative. /// running and fetches anything missing. That fetch races real deploys, so the
/// <c>deployed_at</c> guard is what stops a slow reconcile response from overwriting a
/// newer config that landed while it was in flight. The active-node apply path
/// (<see cref="StoreDeployedConfigAsync"/>) remains unguarded and always overwrites,
/// because a deploy is always authoritative.
/// <para>
/// It was originally the <i>standby</i> write path as well, under notify-and-fetch: the
/// standby was told a deploy had happened and fetched the config itself. LocalDb Phase 2
/// replaced that with change-data-capture — the config row simply replicates — so the
/// standby no longer writes here at all, and last-writer-wins on the primary key (not
/// this guard) is what orders concurrent writes between the two nodes. Reconciliation is
/// the reason the method survives; do not port the <c>deployed_at</c> guard onto the
/// replication path, where it would fight the HLC rather than help it.
/// </para>
/// <para> /// <para>
/// <paramref name="deployedAtOverride"/> is exposed for testing so that the exact /// <paramref name="deployedAtOverride"/> is exposed for testing so that the exact
/// <c>deployed_at</c> value can be controlled without sleeping between calls. /// <c>deployed_at</c> value can be controlled without sleeping between calls.
@@ -308,8 +199,7 @@ public class SiteStorageService
{ {
var deployedAt = (deployedAtOverride ?? DateTimeOffset.UtcNow).ToString("O"); var deployedAt = (deployedAtOverride ?? DateTimeOffset.UtcNow).ToString("O");
await using var connection = new SqliteConnection(_connectionString); await using var connection = OpenConnection();
await connection.OpenAsync();
await using var command = connection.CreateCommand(); await using var command = connection.CreateCommand();
command.CommandText = @" command.CommandText = @"
@@ -342,8 +232,7 @@ public class SiteStorageService
/// <returns>A task that completes when the configuration and its overrides have been removed.</returns> /// <returns>A task that completes when the configuration and its overrides have been removed.</returns>
public async Task RemoveDeployedConfigAsync(string instanceName) public async Task RemoveDeployedConfigAsync(string instanceName)
{ {
await using var connection = new SqliteConnection(_connectionString); await using var connection = OpenConnection();
await connection.OpenAsync();
await using var transaction = await connection.BeginTransactionAsync(); await using var transaction = await connection.BeginTransactionAsync();
@@ -383,8 +272,7 @@ public class SiteStorageService
/// <returns>A task that completes when the enabled flag has been updated.</returns> /// <returns>A task that completes when the enabled flag has been updated.</returns>
public async Task SetInstanceEnabledAsync(string instanceName, bool isEnabled) public async Task SetInstanceEnabledAsync(string instanceName, bool isEnabled)
{ {
await using var connection = new SqliteConnection(_connectionString); await using var connection = OpenConnection();
await connection.OpenAsync();
await using var command = connection.CreateCommand(); await using var command = connection.CreateCommand();
command.CommandText = @" command.CommandText = @"
@@ -411,8 +299,7 @@ public class SiteStorageService
/// <returns>A task that resolves to a dictionary mapping attribute names to their override values.</returns> /// <returns>A task that resolves to a dictionary mapping attribute names to their override values.</returns>
public async Task<Dictionary<string, string>> GetStaticOverridesAsync(string instanceName) public async Task<Dictionary<string, string>> GetStaticOverridesAsync(string instanceName)
{ {
await using var connection = new SqliteConnection(_connectionString); await using var connection = OpenConnection();
await connection.OpenAsync();
await using var command = connection.CreateCommand(); await using var command = connection.CreateCommand();
command.CommandText = @" command.CommandText = @"
@@ -440,8 +327,7 @@ public class SiteStorageService
/// <returns>A task that completes when the override has been saved.</returns> /// <returns>A task that completes when the override has been saved.</returns>
public async Task SetStaticOverrideAsync(string instanceName, string attributeName, string value) public async Task SetStaticOverrideAsync(string instanceName, string attributeName, string value)
{ {
await using var connection = new SqliteConnection(_connectionString); await using var connection = OpenConnection();
await connection.OpenAsync();
await using var command = connection.CreateCommand(); await using var command = connection.CreateCommand();
command.CommandText = @" command.CommandText = @"
@@ -467,8 +353,7 @@ public class SiteStorageService
/// <returns>A task that completes when all overrides for the instance have been deleted.</returns> /// <returns>A task that completes when all overrides for the instance have been deleted.</returns>
public async Task ClearStaticOverridesAsync(string instanceName) public async Task ClearStaticOverridesAsync(string instanceName)
{ {
await using var connection = new SqliteConnection(_connectionString); await using var connection = OpenConnection();
await connection.OpenAsync();
await using var command = connection.CreateCommand(); await using var command = connection.CreateCommand();
command.CommandText = "DELETE FROM static_attribute_overrides WHERE instance_unique_name = @name"; command.CommandText = "DELETE FROM static_attribute_overrides WHERE instance_unique_name = @name";
@@ -495,8 +380,7 @@ public class SiteStorageService
string instanceName, string sourceCanonicalName, string sourceReference, string instanceName, string sourceCanonicalName, string sourceReference,
string conditionJson, DateTimeOffset lastTransitionAt, string? metadataJson = null) string conditionJson, DateTimeOffset lastTransitionAt, string? metadataJson = null)
{ {
await using var connection = new SqliteConnection(_connectionString); await using var connection = OpenConnection();
await connection.OpenAsync();
await using var command = connection.CreateCommand(); await using var command = connection.CreateCommand();
command.CommandText = @" command.CommandText = @"
@@ -535,8 +419,7 @@ public class SiteStorageService
if (rows.Count == 0) if (rows.Count == 0)
return; return;
await using var connection = new SqliteConnection(_connectionString); await using var connection = OpenConnection();
await connection.OpenAsync();
await using var transaction = (SqliteTransaction)await connection.BeginTransactionAsync(); await using var transaction = (SqliteTransaction)await connection.BeginTransactionAsync();
await using var command = connection.CreateCommand(); await using var command = connection.CreateCommand();
@@ -580,8 +463,7 @@ public class SiteStorageService
/// <returns>A task that completes when the alarm condition row has been deleted.</returns> /// <returns>A task that completes when the alarm condition row has been deleted.</returns>
public async Task DeleteNativeAlarmAsync(string instanceName, string sourceCanonicalName, string sourceReference) public async Task DeleteNativeAlarmAsync(string instanceName, string sourceCanonicalName, string sourceReference)
{ {
await using var connection = new SqliteConnection(_connectionString); await using var connection = OpenConnection();
await connection.OpenAsync();
await using var command = connection.CreateCommand(); await using var command = connection.CreateCommand();
command.CommandText = @" command.CommandText = @"
@@ -605,8 +487,7 @@ public class SiteStorageService
/// <returns>A task that resolves to the list of stored native alarm condition rows for the binding.</returns> /// <returns>A task that resolves to the list of stored native alarm condition rows for the binding.</returns>
public async Task<IReadOnlyList<NativeAlarmRow>> GetNativeAlarmsAsync(string instanceName, string sourceCanonicalName) public async Task<IReadOnlyList<NativeAlarmRow>> GetNativeAlarmsAsync(string instanceName, string sourceCanonicalName)
{ {
await using var connection = new SqliteConnection(_connectionString); await using var connection = OpenConnection();
await connection.OpenAsync();
await using var command = connection.CreateCommand(); await using var command = connection.CreateCommand();
command.CommandText = @" command.CommandText = @"
@@ -637,8 +518,7 @@ public class SiteStorageService
/// <returns>A task that completes when all native alarm rows for the instance have been deleted.</returns> /// <returns>A task that completes when all native alarm rows for the instance have been deleted.</returns>
public async Task ClearNativeAlarmsForInstanceAsync(string instanceName) public async Task ClearNativeAlarmsForInstanceAsync(string instanceName)
{ {
await using var connection = new SqliteConnection(_connectionString); await using var connection = OpenConnection();
await connection.OpenAsync();
await using var command = connection.CreateCommand(); await using var command = connection.CreateCommand();
command.CommandText = "DELETE FROM native_alarm_state WHERE instance_unique_name = @name"; command.CommandText = "DELETE FROM native_alarm_state WHERE instance_unique_name = @name";
@@ -660,8 +540,7 @@ public class SiteStorageService
/// <returns>A task that completes when the shared script has been stored or updated.</returns> /// <returns>A task that completes when the shared script has been stored or updated.</returns>
public async Task StoreSharedScriptAsync(string name, string code, string? parameterDefs, string? returnDef) public async Task StoreSharedScriptAsync(string name, string code, string? parameterDefs, string? returnDef)
{ {
await using var connection = new SqliteConnection(_connectionString); await using var connection = OpenConnection();
await connection.OpenAsync();
await using var command = connection.CreateCommand(); await using var command = connection.CreateCommand();
command.CommandText = @" command.CommandText = @"
@@ -689,8 +568,7 @@ public class SiteStorageService
/// <returns>A task that resolves to the list of all stored shared scripts.</returns> /// <returns>A task that resolves to the list of all stored shared scripts.</returns>
public async Task<List<StoredSharedScript>> GetAllSharedScriptsAsync() public async Task<List<StoredSharedScript>> GetAllSharedScriptsAsync()
{ {
await using var connection = new SqliteConnection(_connectionString); await using var connection = OpenConnection();
await connection.OpenAsync();
await using var command = connection.CreateCommand(); await using var command = connection.CreateCommand();
command.CommandText = "SELECT name, code, parameter_definitions, return_definition FROM shared_scripts"; command.CommandText = "SELECT name, code, parameter_definitions, return_definition FROM shared_scripts";
@@ -727,8 +605,7 @@ public class SiteStorageService
string name, string endpointUrl, string authType, string? authConfig, string? methodDefs, string name, string endpointUrl, string authType, string? authConfig, string? methodDefs,
int timeoutSeconds = 0) int timeoutSeconds = 0)
{ {
await using var connection = new SqliteConnection(_connectionString); await using var connection = OpenConnection();
await connection.OpenAsync();
await using var command = connection.CreateCommand(); await using var command = connection.CreateCommand();
command.CommandText = @" command.CommandText = @"
@@ -766,8 +643,7 @@ public class SiteStorageService
public async Task StoreDatabaseConnectionAsync( public async Task StoreDatabaseConnectionAsync(
string name, string connectionString, int maxRetries, TimeSpan retryDelay) string name, string connectionString, int maxRetries, TimeSpan retryDelay)
{ {
await using var connection = new SqliteConnection(_connectionString); await using var connection = OpenConnection();
await connection.OpenAsync();
await using var command = connection.CreateCommand(); await using var command = connection.CreateCommand();
command.CommandText = @" command.CommandText = @"
@@ -810,8 +686,7 @@ public class SiteStorageService
/// <returns>A task that completes when both tables have been emptied.</returns> /// <returns>A task that completes when both tables have been emptied.</returns>
public async Task PurgeCentralOnlyNotificationConfigAsync() public async Task PurgeCentralOnlyNotificationConfigAsync()
{ {
await using var connection = new SqliteConnection(_connectionString); await using var connection = OpenConnection();
await connection.OpenAsync();
await using var command = connection.CreateCommand(); await using var command = connection.CreateCommand();
command.CommandText = @" command.CommandText = @"
@@ -834,8 +709,7 @@ public class SiteStorageService
public async Task StoreDataConnectionDefinitionAsync( public async Task StoreDataConnectionDefinitionAsync(
string name, string protocol, string? configJson, string? backupConfigJson = null, int failoverRetryCount = 3) string name, string protocol, string? configJson, string? backupConfigJson = null, int failoverRetryCount = 3)
{ {
await using var connection = new SqliteConnection(_connectionString); await using var connection = OpenConnection();
await connection.OpenAsync();
await using var command = connection.CreateCommand(); await using var command = connection.CreateCommand();
command.CommandText = @" command.CommandText = @"
@@ -865,8 +739,7 @@ public class SiteStorageService
/// <returns>A task that resolves to the list of all stored data connection definitions.</returns> /// <returns>A task that resolves to the list of all stored data connection definitions.</returns>
public async Task<List<StoredDataConnectionDefinition>> GetAllDataConnectionDefinitionsAsync() public async Task<List<StoredDataConnectionDefinition>> GetAllDataConnectionDefinitionsAsync()
{ {
await using var connection = new SqliteConnection(_connectionString); await using var connection = OpenConnection();
await connection.OpenAsync();
await using var command = connection.CreateCommand(); await using var command = connection.CreateCommand();
command.CommandText = "SELECT name, protocol, configuration, backup_configuration, failover_retry_count FROM data_connection_definitions"; command.CommandText = "SELECT name, protocol, configuration, backup_configuration, failover_retry_count FROM data_connection_definitions";
src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Repositories/SiteExternalSystemRepository.cs
+10 -5
View File
@@ -31,8 +31,9 @@ public class SiteExternalSystemRepository : IExternalSystemRepository
public async Task<IReadOnlyList<ExternalSystemDefinition>> GetAllExternalSystemsAsync( public async Task<IReadOnlyList<ExternalSystemDefinition>> GetAllExternalSystemsAsync(
CancellationToken cancellationToken = default) CancellationToken cancellationToken = default)
{ {
// Already open — SiteStorageService.CreateConnection now hands out a
// LocalDb-managed connection. Opening it again would throw.
await using var connection = CreateConnection(); await using var connection = CreateConnection();
await connection.OpenAsync(cancellationToken);
await using var command = connection.CreateCommand(); await using var command = connection.CreateCommand();
command.CommandText = @" command.CommandText = @"
@@ -63,8 +64,9 @@ public class SiteExternalSystemRepository : IExternalSystemRepository
public async Task<ExternalSystemDefinition?> GetExternalSystemByNameAsync( public async Task<ExternalSystemDefinition?> GetExternalSystemByNameAsync(
string name, CancellationToken cancellationToken = default) string name, CancellationToken cancellationToken = default)
{ {
// Already open — SiteStorageService.CreateConnection now hands out a
// LocalDb-managed connection. Opening it again would throw.
await using var connection = CreateConnection(); await using var connection = CreateConnection();
await connection.OpenAsync(cancellationToken);
await using var command = connection.CreateCommand(); await using var command = connection.CreateCommand();
command.CommandText = @" command.CommandText = @"
@@ -91,8 +93,9 @@ public class SiteExternalSystemRepository : IExternalSystemRepository
if (system is null) if (system is null)
return Array.Empty<ExternalSystemMethod>(); return Array.Empty<ExternalSystemMethod>();
// Already open — SiteStorageService.CreateConnection now hands out a
// LocalDb-managed connection. Opening it again would throw.
await using var connection = CreateConnection(); await using var connection = CreateConnection();
await connection.OpenAsync(cancellationToken);
await using var command = connection.CreateCommand(); await using var command = connection.CreateCommand();
command.CommandText = @" command.CommandText = @"
@@ -140,8 +143,9 @@ public class SiteExternalSystemRepository : IExternalSystemRepository
public async Task<IReadOnlyList<DatabaseConnectionDefinition>> GetAllDatabaseConnectionsAsync( public async Task<IReadOnlyList<DatabaseConnectionDefinition>> GetAllDatabaseConnectionsAsync(
CancellationToken cancellationToken = default) CancellationToken cancellationToken = default)
{ {
// Already open — SiteStorageService.CreateConnection now hands out a
// LocalDb-managed connection. Opening it again would throw.
await using var connection = CreateConnection(); await using var connection = CreateConnection();
await connection.OpenAsync(cancellationToken);
await using var command = connection.CreateCommand(); await using var command = connection.CreateCommand();
command.CommandText = @" command.CommandText = @"
@@ -178,8 +182,9 @@ public class SiteExternalSystemRepository : IExternalSystemRepository
public async Task<DatabaseConnectionDefinition?> GetDatabaseConnectionByNameAsync( public async Task<DatabaseConnectionDefinition?> GetDatabaseConnectionByNameAsync(
string name, CancellationToken cancellationToken = default) string name, CancellationToken cancellationToken = default)
{ {
// Already open — SiteStorageService.CreateConnection now hands out a
// LocalDb-managed connection. Opening it again would throw.
await using var connection = CreateConnection(); await using var connection = CreateConnection();
await connection.OpenAsync(cancellationToken);
await using var command = connection.CreateCommand(); await using var command = connection.CreateCommand();
command.CommandText = @" command.CommandText = @"
src/ZB.MOM.WW.ScadaBridge.SiteRuntime/ServiceCollectionExtensions.cs
+9 -21
View File
@@ -1,6 +1,7 @@
using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging; using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options; using Microsoft.Extensions.Options;
using ZB.MOM.WW.LocalDb;
using ZB.MOM.WW.ScadaBridge.Communication.Grpc; using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories; using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Deployment; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Deployment;
@@ -14,32 +15,19 @@ namespace ZB.MOM.WW.ScadaBridge.SiteRuntime;
public static class ServiceCollectionExtensions public static class ServiceCollectionExtensions
{ {
/// <summary> /// <summary>
/// Registers Site Runtime services including SiteStorageService for SQLite persistence. /// Registers Site Runtime services including SiteStorageService, which persists to the
/// The caller must register an <see cref="ISiteStorageConnectionProvider"/> or call the /// consolidated site database resolved from <c>ILocalDb</c> (<c>LocalDb:Path</c>).
/// overload with an explicit connection string.
/// </summary> /// </summary>
/// <param name="services">The DI service collection to register services into.</param> /// <param name="services">The DI service collection to register services into.</param>
/// <returns>The same <see cref="IServiceCollection"/> to allow chaining.</returns> /// <returns>The same <see cref="IServiceCollection"/> to allow chaining.</returns>
public static IServiceCollection AddSiteRuntime(this IServiceCollection services) public static IServiceCollection AddSiteRuntime(this IServiceCollection services)
{ {
// SiteStorageService is registered by the Host using AddSiteRuntime(connectionString) // SiteStorageService takes ILocalDb (the consolidated site database at
// This overload is for backward compatibility / skeleton placeholder // LocalDb:Path) rather than a connection string, so there is nothing left for a
return services; // caller to supply — the string overload is gone and this is the only entry point.
} services.AddSingleton(sp => new SiteStorageService(
sp.GetRequiredService<ILocalDb>(),
/// <summary> sp.GetRequiredService<ILogger<SiteStorageService>>()));
/// Registers Site Runtime services with an explicit SQLite connection string.
/// </summary>
/// <param name="services">The DI service collection to register services into.</param>
/// <param name="siteDbConnectionString">The SQLite connection string for the site local storage database.</param>
/// <returns>The same <see cref="IServiceCollection"/> to allow chaining.</returns>
public static IServiceCollection AddSiteRuntime(this IServiceCollection services, string siteDbConnectionString)
{
services.AddSingleton(sp =>
{
var logger = sp.GetRequiredService<ILogger<SiteStorageService>>();
return new SiteStorageService(siteDbConnectionString, logger);
});
services.AddHostedService<SiteStorageInitializer>(); services.AddHostedService<SiteStorageInitializer>();
src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptions.cs
-7
View File
@@ -60,13 +60,6 @@ public class SiteRuntimeOptions
/// <summary>HTTP timeout (seconds) for fetching a deployment config from central (notify-and-fetch).</summary> /// <summary>HTTP timeout (seconds) for fetching a deployment config from central (notify-and-fetch).</summary>
public int ConfigFetchTimeoutSeconds { get; set; } = 30; public int ConfigFetchTimeoutSeconds { get; set; } = 30;
/// <summary>
/// Bounded attempt count (including the first) for the standby's replicated-config
/// fetch; a 2 s fixed delay separates attempts and superseded fetches never retry.
/// Consumed by <c>SiteReplicationActor.HandleApplyConfigDeploy</c> (UA2). Default: 3.
/// </summary>
public int ConfigFetchRetryCount { get; set; } = 3;
/// <summary> /// <summary>
/// Fixed interval (ms) at which an Instance Actor re-sends a tag-subscribe /// Fixed interval (ms) at which an Instance Actor re-sends a tag-subscribe
/// request that either failed or whose response was lost (S4/UA6). The retry is /// request that either failed or whose response was lost (S4/UA6). The retry is
src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptionsValidator.cs
-4
View File
@@ -53,10 +53,6 @@ public sealed class SiteRuntimeOptionsValidator : OptionsValidatorBase<SiteRunti
$"ScadaBridge:SiteRuntime:ConfigFetchTimeoutSeconds must be greater than 0 " + $"ScadaBridge:SiteRuntime:ConfigFetchTimeoutSeconds must be greater than 0 " +
$"(was {options.ConfigFetchTimeoutSeconds})."); $"(was {options.ConfigFetchTimeoutSeconds}).");
builder.RequireThat(options.ConfigFetchRetryCount >= 0,
$"ScadaBridge:SiteRuntime:ConfigFetchRetryCount must be >= 0 " +
$"(was {options.ConfigFetchRetryCount}).");
builder.RequireThat(options.TagSubscribeRetryIntervalMs > 0, builder.RequireThat(options.TagSubscribeRetryIntervalMs > 0,
$"ScadaBridge:SiteRuntime:TagSubscribeRetryIntervalMs must be greater than 0 " + $"ScadaBridge:SiteRuntime:TagSubscribeRetryIntervalMs must be greater than 0 " +
$"(was {options.TagSubscribeRetryIntervalMs}); a zero interval hot-loops the tag-subscribe " + $"(was {options.TagSubscribeRetryIntervalMs}); a zero interval hot-loops the tag-subscribe " +
src/ZB.MOM.WW.ScadaBridge.StoreAndForward/ReplicationService.cs
-194
View File
@@ -1,194 +0,0 @@
using Microsoft.Extensions.Logging;
using ZB.MOM.WW.ScadaBridge.Commons.Observability;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
namespace ZB.MOM.WW.ScadaBridge.StoreAndForward;
/// <summary>
/// Async replication of buffer operations to standby node.
///
/// - Forwards add/remove/park operations to standby via a replication handler.
/// - No ack wait (fire-and-forget per design).
/// - Standby applies operations to its own SQLite.
/// - On failover, standby resumes delivery from its replicated state.
/// </summary>
public class ReplicationService
{
private readonly StoreAndForwardOptions _options;
private readonly ILogger<ReplicationService> _logger;
private Func<ReplicationOperation, Task>? _replicationHandler;
/// <summary>Initializes a new instance of <see cref="ReplicationService"/>.</summary>
/// <param name="options">Store-and-forward configuration options.</param>
/// <param name="logger">Logger instance.</param>
public ReplicationService(
StoreAndForwardOptions options,
ILogger<ReplicationService> logger)
{
_options = options;
_logger = logger;
}
/// <summary>
/// Sets the handler for forwarding replication operations to the standby node.
/// Typically wraps Akka Tell to the standby's replication actor.
/// </summary>
/// <param name="handler">The async delegate that forwards each replication operation to the standby.</param>
public void SetReplicationHandler(Func<ReplicationOperation, Task> handler)
{
_replicationHandler = handler;
}
/// <summary>
/// Replicates an enqueue operation to standby (fire-and-forget).
/// </summary>
/// <param name="message">The message that was enqueued on the active node.</param>
public void ReplicateEnqueue(StoreAndForwardMessage message)
{
if (!_options.ReplicationEnabled || _replicationHandler == null) return;
FireAndForget(new ReplicationOperation(
ReplicationOperationType.Add,
message.Id,
message));
}
/// <summary>
/// Replicates a remove operation to standby (fire-and-forget).
/// </summary>
/// <param name="messageId">The identifier of the message to remove from the standby buffer.</param>
public void ReplicateRemove(string messageId)
{
if (!_options.ReplicationEnabled || _replicationHandler == null) return;
FireAndForget(new ReplicationOperation(
ReplicationOperationType.Remove,
messageId,
null));
}
/// <summary>
/// Replicates a park operation to standby (fire-and-forget).
/// </summary>
/// <param name="message">The message that was parked on the active node.</param>
public void ReplicatePark(StoreAndForwardMessage message)
{
if (!_options.ReplicationEnabled || _replicationHandler == null) return;
FireAndForget(new ReplicationOperation(
ReplicationOperationType.Park,
message.Id,
message));
}
/// <summary>
/// Replicates an operator-initiated requeue (a parked
/// message moved back to the pending queue) to standby (fire-and-forget). The
/// carried message reflects the active node's post-requeue state (Pending,
/// retry_count = 0) so the standby's copy can be brought into sync.
/// </summary>
/// <param name="message">The message in its post-requeue (Pending, retry_count=0) state.</param>
public void ReplicateRequeue(StoreAndForwardMessage message)
{
if (!_options.ReplicationEnabled || _replicationHandler == null) return;
FireAndForget(new ReplicationOperation(
ReplicationOperationType.Requeue,
message.Id,
message));
}
/// <summary>
/// Applies a replicated operation received from the active node.
/// Used by the standby node to keep its SQLite in sync.
///
/// Add/Park/Requeue are applied as <b>upserts</b> (<see cref="StoreAndForwardStorage.UpsertMessageAsync"/>),
/// not blind INSERT/UPDATE: the full message rides in every one of those operations,
/// so a Park/Requeue whose original Add was lost (fire-and-forget replication is
/// best-effort) self-heals by materialising the row, and a duplicate Add (e.g.
/// re-issued by an anti-entropy resync) applies newest-wins instead of throwing a
/// primary-key violation. Remove is a plain delete.
/// </summary>
/// <param name="operation">The replication operation to apply.</param>
/// <param name="storage">The standby node's store-and-forward storage to update.</param>
/// <returns>A task representing the asynchronous apply operation.</returns>
public async Task ApplyReplicatedOperationAsync(
ReplicationOperation operation,
StoreAndForwardStorage storage)
{
switch (operation.OperationType)
{
case ReplicationOperationType.Add when operation.Message != null:
await storage.UpsertMessageAsync(operation.Message);
break;
case ReplicationOperationType.Remove:
await storage.RemoveMessageAsync(operation.MessageId);
break;
case ReplicationOperationType.Park when operation.Message != null:
operation.Message.Status = StoreAndForwardMessageStatus.Parked;
await storage.UpsertMessageAsync(operation.Message);
break;
case ReplicationOperationType.Requeue when operation.Message != null:
operation.Message.Status = StoreAndForwardMessageStatus.Pending;
operation.Message.RetryCount = 0;
await storage.UpsertMessageAsync(operation.Message);
break;
}
}
private void FireAndForget(ReplicationOperation operation)
{
// Invoked inline, NOT via Task.Run: the handler is a non-blocking Akka
// Tell, and thread-pool hand-off destroyed Add/Remove ordering for the
// same message id (arch review 02). Inline invocation preserves issue
// order; Akka's per-sender/receiver guarantee preserves it on the wire.
try
{
var task = _replicationHandler!.Invoke(operation);
if (!task.IsCompletedSuccessfully)
{
task.ContinueWith(t =>
{
ScadaBridgeTelemetry.RecordReplicationFailure();
_logger.LogWarning(t.Exception,
"Replication of {OpType} for message {MessageId} failed (best-effort); standby buffer may be diverging",
operation.OperationType, operation.MessageId);
},
TaskContinuationOptions.OnlyOnFaulted);
}
}
catch (Exception ex)
{
ScadaBridgeTelemetry.RecordReplicationFailure();
_logger.LogWarning(ex,
"Replication of {OpType} for message {MessageId} failed (best-effort); standby buffer may be diverging",
operation.OperationType, operation.MessageId);
}
}
}
/// <summary>
/// Represents a buffer operation to be replicated to standby.
/// </summary>
public record ReplicationOperation(
ReplicationOperationType OperationType,
string MessageId,
StoreAndForwardMessage? Message);
/// <summary>
/// Types of buffer operations that are replicated.
/// </summary>
public enum ReplicationOperationType
{
Add,
Remove,
Park,
/// <summary>
/// An operator moved a parked message back to the pending
/// queue. The standby resets its matching row to Pending with retry_count = 0.
/// </summary>
Requeue
}
src/ZB.MOM.WW.ScadaBridge.StoreAndForward/ServiceCollectionExtensions.cs
+7 -17
View File
@@ -1,6 +1,7 @@
using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging; using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options; using Microsoft.Extensions.Options;
using ZB.MOM.WW.LocalDb;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services; using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services;
using ZB.MOM.WW.ScadaBridge.SiteEventLogging; using ZB.MOM.WW.ScadaBridge.SiteEventLogging;
@@ -15,21 +16,18 @@ public static class ServiceCollectionExtensions
/// <returns>The same <paramref name="services"/> collection, for chaining.</returns> /// <returns>The same <paramref name="services"/> collection, for chaining.</returns>
public static IServiceCollection AddStoreAndForward(this IServiceCollection services) public static IServiceCollection AddStoreAndForward(this IServiceCollection services)
{ {
services.AddSingleton<StoreAndForwardStorage>(sp => // The buffer now lives in the consolidated LocalDb database at LocalDb:Path, not
{ // at StoreAndForwardOptions.SqliteDbPath — that option survives only as the
var options = sp.GetRequiredService<IOptions<StoreAndForwardOptions>>().Value; // migrator's source location (Task 8).
var logger = sp.GetRequiredService<ILogger<StoreAndForwardStorage>>(); services.AddSingleton<StoreAndForwardStorage>(sp => new StoreAndForwardStorage(
return new StoreAndForwardStorage( sp.GetRequiredService<ILocalDb>(),
$"Data Source={options.SqliteDbPath}", sp.GetRequiredService<ILogger<StoreAndForwardStorage>>()));
logger);
});
services.AddSingleton<StoreAndForwardService>(sp => services.AddSingleton<StoreAndForwardService>(sp =>
{ {
var storage = sp.GetRequiredService<StoreAndForwardStorage>(); var storage = sp.GetRequiredService<StoreAndForwardStorage>();
var options = sp.GetRequiredService<IOptions<StoreAndForwardOptions>>().Value; var options = sp.GetRequiredService<IOptions<StoreAndForwardOptions>>().Value;
var logger = sp.GetRequiredService<ILogger<StoreAndForwardService>>(); var logger = sp.GetRequiredService<ILogger<StoreAndForwardService>>();
var replication = sp.GetRequiredService<ReplicationService>();
// Wire the cached-call lifecycle // Wire the cached-call lifecycle
// observer + site identity through DI so the S&F retry loop emits // observer + site identity through DI so the S&F retry loop emits
// per-attempt + terminal telemetry under the same TrackedOperationId // per-attempt + terminal telemetry under the same TrackedOperationId
@@ -54,19 +52,11 @@ public static class ServiceCollectionExtensions
storage, storage,
options, options,
logger, logger,
replication,
cachedCallObserver, cachedCallObserver,
siteId, siteId,
siteEventLogger); siteEventLogger);
}); });
services.AddSingleton<ReplicationService>(sp =>
{
var options = sp.GetRequiredService<IOptions<StoreAndForwardOptions>>().Value;
var logger = sp.GetRequiredService<ILogger<ReplicationService>>();
return new ReplicationService(options, logger);
});
return services; return services;
} }
src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardOptions.cs
+7 -4
View File
@@ -5,12 +5,15 @@ namespace ZB.MOM.WW.ScadaBridge.StoreAndForward;
/// </summary> /// </summary>
public class StoreAndForwardOptions public class StoreAndForwardOptions
{ {
/// <summary>Path to the SQLite database for S&amp;F message persistence.</summary> /// <summary>
/// Path to the legacy standalone store-and-forward SQLite file. <b>Migration-only.</b>
/// The buffer itself lives in the consolidated LocalDb database (<c>LocalDb:Path</c>)
/// as of LocalDb Phase 2; this path is read once at boot by
/// <c>SiteLocalDbLegacyMigrator</c> to drain a pre-Phase-2 file, and is otherwise
/// unused. A node that has already migrated may leave it set or unset.
/// </summary>
public string SqliteDbPath { get; set; } = "./data/store-and-forward.db"; public string SqliteDbPath { get; set; } = "./data/store-and-forward.db";
/// <summary>Whether to replicate buffer operations to standby node.</summary>
public bool ReplicationEnabled { get; set; } = true;
/// <summary>Default retry interval for messages without per-source settings.</summary> /// <summary>Default retry interval for messages without per-source settings.</summary>
public TimeSpan DefaultRetryInterval { get; set; } = TimeSpan.FromSeconds(30); public TimeSpan DefaultRetryInterval { get; set; } = TimeSpan.FromSeconds(30);
src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardOptionsValidator.cs
+11 -9
View File
@@ -5,21 +5,23 @@ namespace ZB.MOM.WW.ScadaBridge.StoreAndForward;
/// <summary> /// <summary>
/// Validates <see cref="StoreAndForwardOptions"/> at startup. The retry intervals /// Validates <see cref="StoreAndForwardOptions"/> at startup. The retry intervals
/// feed the background sweep timer (a zero/negative period trips /// feed the background sweep timer (a zero/negative period trips
/// <see cref="ArgumentOutOfRangeException"/> in the timer constructor) and the /// <see cref="ArgumentOutOfRangeException"/> in the timer constructor). Registered
/// SQLite path is opened for the S&amp;F buffer; an empty path yields an opaque /// with <c>ValidateOnStart()</c> so a bad <c>ScadaBridge:StoreAndForward</c> section
/// connection failure at first enqueue. Registered with <c>ValidateOnStart()</c> /// fails fast at boot with a clear, key-naming message.
/// so a bad <c>ScadaBridge:StoreAndForward</c> section fails fast at boot with a /// <para>
/// clear, key-naming message. /// <see cref="StoreAndForwardOptions.SqliteDbPath"/> is deliberately NOT validated.
/// Before LocalDb Phase 2 it was the live buffer file, so an empty value produced an
/// opaque connection failure at first enqueue; the buffer now lives in the
/// consolidated LocalDb database and the key survives only as the legacy migration
/// source. An empty value there is a legitimate "nothing to migrate", so requiring
/// it would make every already-migrated node carry a dead key forever.
/// </para>
/// </summary> /// </summary>
public sealed class StoreAndForwardOptionsValidator : OptionsValidatorBase<StoreAndForwardOptions> public sealed class StoreAndForwardOptionsValidator : OptionsValidatorBase<StoreAndForwardOptions>
{ {
/// <inheritdoc /> /// <inheritdoc />
protected override void Validate(ValidationBuilder builder, StoreAndForwardOptions options) protected override void Validate(ValidationBuilder builder, StoreAndForwardOptions options)
{ {
builder.RequireThat(!string.IsNullOrWhiteSpace(options.SqliteDbPath),
"ScadaBridge:StoreAndForward:SqliteDbPath must be a non-empty path; " +
"it is the SQLite file backing the store-and-forward buffer.");
builder.RequireThat(options.DefaultRetryInterval > TimeSpan.Zero, builder.RequireThat(options.DefaultRetryInterval > TimeSpan.Zero,
$"ScadaBridge:StoreAndForward:DefaultRetryInterval must be a positive duration " + $"ScadaBridge:StoreAndForward:DefaultRetryInterval must be a positive duration " +
$"(was {options.DefaultRetryInterval}); it is the default per-message retry interval."); $"(was {options.DefaultRetryInterval}); it is the default per-message retry interval.");
src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardSchema.cs
+124
View File
@@ -0,0 +1,124 @@
using Microsoft.Data.Sqlite;
namespace ZB.MOM.WW.ScadaBridge.StoreAndForward;
/// <summary>
/// DDL for the <c>sf_messages</c> table, extracted from
/// <see cref="StoreAndForwardStorage"/> so it can be applied by whoever owns the
/// database file.
/// <para>
/// Deliberately depends only on <c>Microsoft.Data.Sqlite</c>, not on the LocalDb library.
/// The Host applies this DDL to a LocalDb-managed connection before
/// <c>RegisterReplicated</c> installs the capture triggers; nothing about the schema
/// itself is LocalDb-specific, and the store still calls it so a directly-constructed
/// store (tests, tooling) remains self-sufficient.
/// </para>
/// </summary>
public static class StoreAndForwardSchema
{
/// <summary>
/// Creates the <c>sf_messages</c> table and its indexes when absent, and additively
/// upgrades a table created by an older build. Idempotent — safe to run on every
/// startup.
/// </summary>
/// <param name="connection">An open connection to the database that should hold the table.</param>
public static void Apply(SqliteConnection connection)
{
ArgumentNullException.ThrowIfNull(connection);
using (var command = connection.CreateCommand())
{
command.CommandText = @"
CREATE TABLE IF NOT EXISTS sf_messages (
id TEXT PRIMARY KEY,
category INTEGER NOT NULL,
target TEXT NOT NULL,
payload_json TEXT NOT NULL,
retry_count INTEGER NOT NULL DEFAULT 0,
max_retries INTEGER NOT NULL DEFAULT 50,
retry_interval_ms INTEGER NOT NULL DEFAULT 30000,
created_at TEXT NOT NULL,
last_attempt_at TEXT,
status INTEGER NOT NULL DEFAULT 0,
last_error TEXT,
origin_instance TEXT
);
CREATE INDEX IF NOT EXISTS idx_sf_messages_status ON sf_messages(status);
CREATE INDEX IF NOT EXISTS idx_sf_messages_category ON sf_messages(category);
";
command.ExecuteNonQuery();
}
// Additively add the execution_id /
// source_script columns. CREATE TABLE IF NOT EXISTS above does NOT add
// columns to a table that already exists from before these fields, so a
// databases created by an older build needs the columns ALTER-ed in.
// SQLite has no "ADD COLUMN IF NOT EXISTS"; the column presence is
// probed first and the ALTER skipped when already there. Both columns
// are nullable with no default, so any row buffered before this
// migration reads back ExecutionId/SourceScript = null (back-compat).
AddColumnIfMissing(connection, "execution_id", "TEXT");
AddColumnIfMissing(connection, "source_script", "TEXT");
// Additively add the
// parent_execution_id column the same way — a sibling to execution_id.
// Nullable with no default, so any row buffered before this migration
// reads back ParentExecutionId = null (back-compat).
AddColumnIfMissing(connection, "parent_execution_id", "TEXT");
// Additively add the epoch-ms sibling of last_attempt_at. The
// ISO-8601 text column stays authoritative for reads / back-compat; this
// INTEGER column drives the due predicate so the sweep no longer parses
// julianday() per row.
AddColumnIfMissing(connection, "last_attempt_at_ms", "INTEGER");
// One-time backfill for rows persisted before the ms column existed: derive
// epoch-ms from the text timestamp. The "... IS NULL" guard makes this run
// once per legacy DB and never match again — this is the only remaining
// julianday() use in the store.
using (var backfill = connection.CreateCommand())
{
backfill.CommandText = @"
UPDATE sf_messages
SET last_attempt_at_ms = CAST((julianday(last_attempt_at) - 2440587.5) * 86400000 AS INTEGER)
WHERE last_attempt_at IS NOT NULL AND last_attempt_at_ms IS NULL";
backfill.ExecuteNonQuery();
}
// Covering index for the due query (status filter + ms ordering column).
// Created after the ALTER above so the column exists.
using (var dueIndex = connection.CreateCommand())
{
dueIndex.CommandText =
"CREATE INDEX IF NOT EXISTS idx_sf_messages_status_due ON sf_messages(status, last_attempt_at_ms)";
dueIndex.ExecuteNonQuery();
}
}
/// <summary>
/// Adds a column to <c>sf_messages</c>
/// only when it is not already present. SQLite lacks <c>ADD COLUMN IF NOT
/// EXISTS</c>, so the schema is probed via <c>PRAGMA table_info</c> first.
/// Idempotent — safe to run on every <see cref="Apply"/>.
/// </summary>
private static void AddColumnIfMissing(
SqliteConnection connection, string columnName, string columnType)
{
using (var probe = connection.CreateCommand())
{
probe.CommandText = "SELECT COUNT(*) FROM pragma_table_info('sf_messages') WHERE name = @name";
probe.Parameters.AddWithValue("@name", columnName);
if (Convert.ToInt32(probe.ExecuteScalar()) > 0)
{
return;
}
}
using var alter = connection.CreateCommand();
// Column name + type are caller-controlled constants, never user input —
// safe to interpolate (parameters are not permitted in DDL).
alter.CommandText = $"ALTER TABLE sf_messages ADD COLUMN {columnName} {columnType}";
alter.ExecuteNonQuery();
}
}
src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardService.cs
+9 -23
View File
@@ -36,7 +36,6 @@ public class StoreAndForwardService
{ {
private readonly StoreAndForwardStorage _storage; private readonly StoreAndForwardStorage _storage;
private readonly StoreAndForwardOptions _options; private readonly StoreAndForwardOptions _options;
private readonly ReplicationService? _replication;
private readonly ILogger<StoreAndForwardService> _logger; private readonly ILogger<StoreAndForwardService> _logger;
/// <summary> /// <summary>
/// Site-side observer notified /// Site-side observer notified
@@ -107,7 +106,7 @@ public class StoreAndForwardService
/// <c>null</c> when no sweep is currently running. Captured when the timer /// <c>null</c> when no sweep is currently running. Captured when the timer
/// callback starts a sweep so <see cref="StopAsync"/> can wait for it to /// callback starts a sweep so <see cref="StopAsync"/> can wait for it to
/// finish before the host disposes downstream dependencies /// finish before the host disposes downstream dependencies
/// (<see cref="_storage"/>, <see cref="_replication"/>) that the sweep is /// (<see cref="_storage"/>) that the sweep is
/// still touching. Written from the timer thread and from /// still touching. Written from the timer thread and from
/// <see cref="StopAsync"/>, so reads are synchronised via the /// <see cref="StopAsync"/>, so reads are synchronised via the
/// <see cref="Volatile"/> APIs. /// <see cref="Volatile"/> APIs.
@@ -227,7 +226,6 @@ public class StoreAndForwardService
/// <param name="storage">The storage backend for buffered messages.</param> /// <param name="storage">The storage backend for buffered messages.</param>
/// <param name="options">Configuration options.</param> /// <param name="options">Configuration options.</param>
/// <param name="logger">Logger instance.</param> /// <param name="logger">Logger instance.</param>
/// <param name="replication">Optional replication service for standby synchronization.</param>
/// <param name="cachedCallObserver">Optional observer for cached call lifecycle events.</param> /// <param name="cachedCallObserver">Optional observer for cached call lifecycle events.</param>
/// <param name="siteId">The site identifier this service belongs to.</param> /// <param name="siteId">The site identifier this service belongs to.</param>
/// <param name="siteEventLogger"> /// <param name="siteEventLogger">
@@ -240,7 +238,6 @@ public class StoreAndForwardService
StoreAndForwardStorage storage, StoreAndForwardStorage storage,
StoreAndForwardOptions options, StoreAndForwardOptions options,
ILogger<StoreAndForwardService> logger, ILogger<StoreAndForwardService> logger,
ReplicationService? replication = null,
ICachedCallLifecycleObserver? cachedCallObserver = null, ICachedCallLifecycleObserver? cachedCallObserver = null,
string siteId = "", string siteId = "",
ISiteEventLogger? siteEventLogger = null) ISiteEventLogger? siteEventLogger = null)
@@ -248,7 +245,6 @@ public class StoreAndForwardService
_storage = storage; _storage = storage;
_options = options; _options = options;
_logger = logger; _logger = logger;
_replication = replication;
_cachedCallObserver = cachedCallObserver; _cachedCallObserver = cachedCallObserver;
_siteId = string.IsNullOrWhiteSpace(siteId) ? UnknownSiteSentinel : siteId; _siteId = string.IsNullOrWhiteSpace(siteId) ? UnknownSiteSentinel : siteId;
_siteEventLogger = siteEventLogger; _siteEventLogger = siteEventLogger;
@@ -651,7 +647,6 @@ public class StoreAndForwardService
private async Task BufferAsync(StoreAndForwardMessage message) private async Task BufferAsync(StoreAndForwardMessage message)
{ {
await _storage.EnqueueAsync(message); await _storage.EnqueueAsync(message);
_replication?.ReplicateEnqueue(message);
Interlocked.Increment(ref _bufferedCount); Interlocked.Increment(ref _bufferedCount);
} }
@@ -802,7 +797,6 @@ public class StoreAndForwardService
if (success) if (success)
{ {
await _storage.RemoveMessageAsync(message.Id); await _storage.RemoveMessageAsync(message.Id);
_replication?.ReplicateRemove(message.Id);
Interlocked.Decrement(ref _bufferedCount); Interlocked.Decrement(ref _bufferedCount);
RaiseActivity("Delivered", message.Category, RaiseActivity("Delivered", message.Category,
$"Delivered to {message.Target} after {message.RetryCount} retries"); $"Delivered to {message.Target} after {message.RetryCount} retries");
@@ -833,7 +827,6 @@ public class StoreAndForwardService
return RetryOutcome.Skipped; return RetryOutcome.Skipped;
} }
Interlocked.Decrement(ref _bufferedCount); Interlocked.Decrement(ref _bufferedCount);
_replication?.ReplicatePark(message);
RaiseActivity("Parked", message.Category, RaiseActivity("Parked", message.Category,
$"Permanent failure for {message.Target}: handler returned false"); $"Permanent failure for {message.Target}: handler returned false");
@@ -869,7 +862,6 @@ public class StoreAndForwardService
return RetryOutcome.Skipped; return RetryOutcome.Skipped;
} }
Interlocked.Decrement(ref _bufferedCount); Interlocked.Decrement(ref _bufferedCount);
_replication?.ReplicatePark(message);
RaiseActivity("Parked", message.Category, RaiseActivity("Parked", message.Category,
$"Max retries ({message.MaxRetries}) reached for {message.Target}"); $"Max retries ({message.MaxRetries}) reached for {message.Target}");
_logger.LogWarning( _logger.LogWarning(
@@ -1077,17 +1069,12 @@ public class StoreAndForwardService
/// <summary> /// <summary>
/// Retries a parked message (moves back to pending queue). /// Retries a parked message (moves back to pending queue).
/// ///
/// An operator requeue is a buffer state change and is /// An operator requeue is a buffer state change, and the peer picks it up because
/// replicated to the standby (as a <see cref="ReplicationOperationType.Requeue"/>) /// <c>sf_messages</c> is a replicated table: the row update is captured and shipped
/// so a failover preserves the operator's retry intent. /// like any other write, so a failover preserves the operator's retry intent. This
/// used to be an explicit Requeue operation sent to the standby.
/// The activity-log entry carries the message's true /// The activity-log entry carries the message's true
/// category rather than a hard-coded one. /// category rather than a hard-coded one.
/// The parked row is captured <i>before</i> the local
/// requeue write rather than re-read after it, so a concurrent
/// <c>RemoveMessageAsync</c> or <c>DiscardParkedMessageAsync</c> running
/// between the two storage calls cannot leave the standby in <c>Parked</c>
/// while the active node has already requeued — we always have the row in
/// hand for the <c>Requeue</c> replication.
/// </summary> /// </summary>
/// <param name="messageId">The identifier of the message to retry.</param> /// <param name="messageId">The identifier of the message to retry.</param>
/// <returns>True if successfully retried, false otherwise.</returns> /// <returns>True if successfully retried, false otherwise.</returns>
@@ -1117,7 +1104,6 @@ public class StoreAndForwardService
captured.RetryCount = 0; captured.RetryCount = 0;
captured.LastError = null; captured.LastError = null;
captured.LastAttemptAt = null; captured.LastAttemptAt = null;
_replication?.ReplicateRequeue(captured);
RaiseActivity("Retry", captured.Category, RaiseActivity("Retry", captured.Category,
$"Parked message {messageId} moved back to queue"); $"Parked message {messageId} moved back to queue");
@@ -1127,9 +1113,10 @@ public class StoreAndForwardService
/// <summary> /// <summary>
/// Permanently discards a parked message. /// Permanently discards a parked message.
/// ///
/// An operator discard is a buffer removal and is replicated /// An operator discard is a buffer removal. The delete is captured as a tombstone on
/// to the standby (as a <see cref="ReplicationOperationType.Remove"/>) so the /// the replicated <c>sf_messages</c> table and carries the later HLC, so the discarded
/// discarded message does not reappear after a failover. /// message cannot reappear after a failover regardless of arrival order. This used to
/// be an explicit Remove operation sent to the standby.
/// The activity-log entry carries the message's true /// The activity-log entry carries the message's true
/// category rather than a hard-coded one. /// category rather than a hard-coded one.
/// </summary> /// </summary>
@@ -1143,7 +1130,6 @@ public class StoreAndForwardService
var success = await _storage.DiscardParkedMessageAsync(messageId); var success = await _storage.DiscardParkedMessageAsync(messageId);
if (success) if (success)
{ {
_replication?.ReplicateRemove(messageId);
RaiseActivity("Discard", message?.Category ?? StoreAndForwardCategory.ExternalSystem, RaiseActivity("Discard", message?.Category ?? StoreAndForwardCategory.ExternalSystem,
$"Parked message {messageId} discarded"); $"Parked message {messageId} discarded");
} }
src/ZB.MOM.WW.ScadaBridge.StoreAndForward/StoreAndForwardStorage.cs
+47 -201
View File
@@ -1,5 +1,6 @@
using Microsoft.Data.Sqlite; using Microsoft.Data.Sqlite;
using Microsoft.Extensions.Logging; using Microsoft.Extensions.Logging;
using ZB.MOM.WW.LocalDb;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
namespace ZB.MOM.WW.ScadaBridge.StoreAndForward; namespace ZB.MOM.WW.ScadaBridge.StoreAndForward;
@@ -11,17 +12,25 @@ namespace ZB.MOM.WW.ScadaBridge.StoreAndForward;
/// </summary> /// </summary>
public class StoreAndForwardStorage public class StoreAndForwardStorage
{ {
private readonly string _connectionString; private readonly ILocalDb _localDb;
private readonly ILogger<StoreAndForwardStorage> _logger; private readonly ILogger<StoreAndForwardStorage> _logger;
/// <summary> /// <summary>
/// Initializes a new instance of <see cref="StoreAndForwardStorage"/> with the given SQLite connection string. /// Initializes the store over the consolidated site database and applies the schema.
/// </summary> /// </summary>
/// <param name="connectionString">SQLite connection string for the store-and-forward database.</param> /// <param name="localDb">
/// The consolidated site database. Every connection it hands out is already open and carries
/// the per-connection pragmas (including <c>busy_timeout</c>) plus the <c>zb_hlc_next()</c>
/// UDF that <c>sf_messages</c>' capture triggers call — which is exactly why the store no
/// longer opens its own <see cref="SqliteConnection"/> from a connection string. A raw
/// connection would lack the UDF and every write to the replicated table would fail closed.
/// </param>
/// <param name="logger">Logger for diagnostics.</param> /// <param name="logger">Logger for diagnostics.</param>
public StoreAndForwardStorage(string connectionString, ILogger<StoreAndForwardStorage> logger) public StoreAndForwardStorage(ILocalDb localDb, ILogger<StoreAndForwardStorage> logger)
{ {
_connectionString = connectionString; ArgumentNullException.ThrowIfNull(localDb);
ArgumentNullException.ThrowIfNull(logger);
_localDb = localDb;
_logger = logger; _logger = logger;
} }
@@ -29,164 +38,34 @@ public class StoreAndForwardStorage
/// Creates the sf_messages table if it does not exist. /// Creates the sf_messages table if it does not exist.
/// </summary> /// </summary>
/// <returns>A task that represents the asynchronous operation.</returns> /// <returns>A task that represents the asynchronous operation.</returns>
public async Task InitializeAsync() public Task InitializeAsync()
{ {
EnsureDatabaseDirectoryExists(); // No directory creation and no journal-mode pragma here any more: LocalDb owns
// the file (it creates the directory) and sets WAL plus the per-connection
// pragmas on every connection it hands out.
using var connection = OpenConnection();
await using var connection = await OpenConnectionAsync(); // The DDL itself lives in StoreAndForwardSchema so the Host can apply it to a
// LocalDb-managed connection before RegisterReplicated installs the capture
// Enable WAL so the concurrent writers this store has by design (script // triggers. The store still calls it, so a directly-constructed store (tests,
// enqueues, the retry sweep's per-target lanes, standby replication applies, // tooling) remains self-sufficient.
// central pull queries) read/write without "database is locked". WAL is StoreAndForwardSchema.Apply(connection);
// persistent + file-scoped; in-memory DBs report "memory" instead — harmless,
// so it is not asserted here.
await using (var walCmd = connection.CreateCommand())
{
walCmd.CommandText = "PRAGMA journal_mode=WAL";
await walCmd.ExecuteNonQueryAsync();
}
await using var command = connection.CreateCommand();
command.CommandText = @"
CREATE TABLE IF NOT EXISTS sf_messages (
id TEXT PRIMARY KEY,
category INTEGER NOT NULL,
target TEXT NOT NULL,
payload_json TEXT NOT NULL,
retry_count INTEGER NOT NULL DEFAULT 0,
max_retries INTEGER NOT NULL DEFAULT 50,
retry_interval_ms INTEGER NOT NULL DEFAULT 30000,
created_at TEXT NOT NULL,
last_attempt_at TEXT,
status INTEGER NOT NULL DEFAULT 0,
last_error TEXT,
origin_instance TEXT
);
CREATE INDEX IF NOT EXISTS idx_sf_messages_status ON sf_messages(status);
CREATE INDEX IF NOT EXISTS idx_sf_messages_category ON sf_messages(category);
";
await command.ExecuteNonQueryAsync();
// Additively add the execution_id /
// source_script columns. CREATE TABLE IF NOT EXISTS above does NOT add
// columns to a table that already exists from before these fields, so a
// databases created by an older build needs the columns ALTER-ed in.
// SQLite has no "ADD COLUMN IF NOT EXISTS"; the column presence is
// probed first and the ALTER skipped when already there. Both columns
// are nullable with no default, so any row buffered before this
// migration reads back ExecutionId/SourceScript = null (back-compat).
await AddColumnIfMissingAsync(connection, "execution_id", "TEXT");
await AddColumnIfMissingAsync(connection, "source_script", "TEXT");
// Additively add the
// parent_execution_id column the same way — a sibling to execution_id.
// Nullable with no default, so any row buffered before this migration
// reads back ParentExecutionId = null (back-compat).
await AddColumnIfMissingAsync(connection, "parent_execution_id", "TEXT");
// Additively add the epoch-ms sibling of last_attempt_at. The
// ISO-8601 text column stays authoritative for reads / back-compat; this
// INTEGER column drives the due predicate so the sweep no longer parses
// julianday() per row.
await AddColumnIfMissingAsync(connection, "last_attempt_at_ms", "INTEGER");
// One-time backfill for rows persisted before the ms column existed: derive
// epoch-ms from the text timestamp. The "... IS NULL" guard makes this run
// once per legacy DB and never match again — this is the only remaining
// julianday() use in the store.
await using (var backfill = connection.CreateCommand())
{
backfill.CommandText = @"
UPDATE sf_messages
SET last_attempt_at_ms = CAST((julianday(last_attempt_at) - 2440587.5) * 86400000 AS INTEGER)
WHERE last_attempt_at IS NOT NULL AND last_attempt_at_ms IS NULL";
await backfill.ExecuteNonQueryAsync();
}
// Covering index for the due query (status filter + ms ordering column).
// Created after the ALTER above so the column exists.
await using (var dueIndex = connection.CreateCommand())
{
dueIndex.CommandText =
"CREATE INDEX IF NOT EXISTS idx_sf_messages_status_due ON sf_messages(status, last_attempt_at_ms)";
await dueIndex.ExecuteNonQueryAsync();
}
_logger.LogInformation("Store-and-forward SQLite storage initialized"); _logger.LogInformation("Store-and-forward SQLite storage initialized");
return Task.CompletedTask;
} }
/// <summary> /// <summary>
/// Adds a column to <c>sf_messages</c> /// CreateConnection returns an ALREADY-OPEN, pragma-configured connection with the
/// only when it is not already present. SQLite lacks <c>ADD COLUMN IF NOT /// <c>zb_hlc_next()</c> UDF registered. Calling <c>OpenAsync</c> on it throws, and a raw
/// EXISTS</c>, so the schema is probed via <c>PRAGMA table_info</c> first. /// <see cref="SqliteConnection"/> would lack the UDF, making every capture trigger fail
/// Idempotent — safe to run on every <see cref="InitializeAsync"/>. /// closed.
/// </summary> /// </summary>
private static async Task AddColumnIfMissingAsync( private SqliteConnection OpenConnection() => _localDb.CreateConnection();
SqliteConnection connection, string columnName, string columnType)
{
await using var probe = connection.CreateCommand();
probe.CommandText = "SELECT COUNT(*) FROM pragma_table_info('sf_messages') WHERE name = @name";
probe.Parameters.AddWithValue("@name", columnName);
var exists = Convert.ToInt32(await probe.ExecuteScalarAsync()) > 0;
if (exists)
{
return;
}
await using var alter = connection.CreateCommand();
// Column name + type are caller-controlled constants, never user input —
// safe to interpolate (parameters are not permitted in DDL).
alter.CommandText = $"ALTER TABLE sf_messages ADD COLUMN {columnName} {columnType}";
await alter.ExecuteNonQueryAsync();
}
/// <summary>
/// Ensures the directory for a file-backed SQLite database exists. SQLite creates
/// the database file on demand but not its parent directory, so a configured path
/// such as "./data/store-and-forward.db" fails to open ("unable to open database
/// file") when the "data" directory does not yet exist. In-memory databases and
/// bare filenames in the working directory have no directory to create and are
/// skipped.
/// </summary>
private void EnsureDatabaseDirectoryExists()
{
var builder = new SqliteConnectionStringBuilder(_connectionString);
if (builder.Mode == SqliteOpenMode.Memory)
return;
var dataSource = builder.DataSource;
if (string.IsNullOrEmpty(dataSource) || dataSource == ":memory:")
return;
var directory = System.IO.Path.GetDirectoryName(System.IO.Path.GetFullPath(dataSource));
if (!string.IsNullOrEmpty(directory) && !System.IO.Directory.Exists(directory))
{
System.IO.Directory.CreateDirectory(directory);
_logger.LogInformation("Created store-and-forward database directory: {Directory}", directory);
}
}
/// <summary>
/// Opens a connection with a 5s busy_timeout. Concurrent writers exist by
/// design (script enqueues, the sweep's lanes, standby replication applies,
/// central pull queries); with connection-per-operation the pragma must be
/// set per open (it is per-connection, and Microsoft.Data.Sqlite pooling
/// makes the extra statement cheap on a pooled physical connection).
/// </summary>
private async Task<SqliteConnection> OpenConnectionAsync()
{
var connection = new SqliteConnection(_connectionString);
await connection.OpenAsync();
await using var pragma = connection.CreateCommand();
pragma.CommandText = "PRAGMA busy_timeout = 5000";
await pragma.ExecuteNonQueryAsync();
return connection;
}
/// <summary> /// <summary>
/// INSERT statement for a full message row. Shared by <see cref="EnqueueAsync"/> /// INSERT statement for a full message row. Shared by <see cref="EnqueueAsync"/>
/// and <see cref="ReplaceAllAsync"/>; bind with <see cref="BindMessageParameters"/> /// and <see cref="UpsertMessageAsync"/>; bind with <see cref="BindMessageParameters"/>
/// so the column list and the parameters never drift apart. /// so the column list and the parameters never drift apart.
/// </summary> /// </summary>
private const string InsertMessageSql = @" private const string InsertMessageSql = @"
@@ -235,7 +114,7 @@ public class StoreAndForwardStorage
/// <returns>A task that represents the asynchronous operation.</returns> /// <returns>A task that represents the asynchronous operation.</returns>
public async Task EnqueueAsync(StoreAndForwardMessage message) public async Task EnqueueAsync(StoreAndForwardMessage message)
{ {
await using var connection = await OpenConnectionAsync(); await using var connection = OpenConnection();
await using var cmd = connection.CreateCommand(); await using var cmd = connection.CreateCommand();
cmd.CommandText = InsertMessageSql; cmd.CommandText = InsertMessageSql;
@@ -255,7 +134,7 @@ public class StoreAndForwardStorage
/// <returns>The oldest-first page (at most <paramref name="limit"/> rows) and whether more rows exist beyond it.</returns> /// <returns>The oldest-first page (at most <paramref name="limit"/> rows) and whether more rows exist beyond it.</returns>
public async Task<(List<StoreAndForwardMessage> Messages, bool Truncated)> GetAllMessagesAsync(int limit) public async Task<(List<StoreAndForwardMessage> Messages, bool Truncated)> GetAllMessagesAsync(int limit)
{ {
await using var connection = await OpenConnectionAsync(); await using var connection = OpenConnection();
await using var cmd = connection.CreateCommand(); await using var cmd = connection.CreateCommand();
cmd.CommandText = @" cmd.CommandText = @"
@@ -272,39 +151,6 @@ public class StoreAndForwardStorage
return (rows.Take(limit).ToList(), truncated); return (rows.Take(limit).ToList(), truncated);
} }
/// <summary>
/// Atomically replaces the entire buffer with <paramref name="messages"/> in a
/// single transaction (delete-all then insert-all). Standby-side anti-entropy
/// apply: a peer-join resync overwrites the standby's divergent copy with the
/// active node's authoritative snapshot. <b>Never call on an active node</b> —
/// it discards every in-flight row.
/// </summary>
/// <param name="messages">The full buffer snapshot to install.</param>
/// <returns>A task that represents the asynchronous operation.</returns>
public async Task ReplaceAllAsync(IReadOnlyList<StoreAndForwardMessage> messages)
{
await using var connection = await OpenConnectionAsync();
await using var transaction = (SqliteTransaction)await connection.BeginTransactionAsync();
await using (var deleteCmd = connection.CreateCommand())
{
deleteCmd.Transaction = transaction;
deleteCmd.CommandText = "DELETE FROM sf_messages";
await deleteCmd.ExecuteNonQueryAsync();
}
foreach (var message in messages)
{
await using var insertCmd = connection.CreateCommand();
insertCmd.Transaction = transaction;
insertCmd.CommandText = InsertMessageSql;
BindMessageParameters(insertCmd, message);
await insertCmd.ExecuteNonQueryAsync();
}
await transaction.CommitAsync();
}
/// <summary> /// <summary>
/// Inserts a message, or updates every mutable column in place if a row with the /// Inserts a message, or updates every mutable column in place if a row with the
/// same id already exists (<c>ON CONFLICT(id) DO UPDATE</c>). Used by the standby /// same id already exists (<c>ON CONFLICT(id) DO UPDATE</c>). Used by the standby
@@ -318,7 +164,7 @@ public class StoreAndForwardStorage
/// <returns>A task that represents the asynchronous operation.</returns> /// <returns>A task that represents the asynchronous operation.</returns>
public async Task UpsertMessageAsync(StoreAndForwardMessage message) public async Task UpsertMessageAsync(StoreAndForwardMessage message)
{ {
await using var connection = await OpenConnectionAsync(); await using var connection = OpenConnection();
await using var cmd = connection.CreateCommand(); await using var cmd = connection.CreateCommand();
cmd.CommandText = @" cmd.CommandText = @"
@@ -375,7 +221,7 @@ public class StoreAndForwardStorage
/// <returns>A task that resolves to the list of messages due for retry, ordered by creation time ascending.</returns> /// <returns>A task that resolves to the list of messages due for retry, ordered by creation time ascending.</returns>
public async Task<List<StoreAndForwardMessage>> GetMessagesForRetryAsync(int limit = 0) public async Task<List<StoreAndForwardMessage>> GetMessagesForRetryAsync(int limit = 0)
{ {
await using var connection = await OpenConnectionAsync(); await using var connection = OpenConnection();
await using var cmd = connection.CreateCommand(); await using var cmd = connection.CreateCommand();
cmd.CommandText = @" cmd.CommandText = @"
@@ -409,7 +255,7 @@ public class StoreAndForwardStorage
/// <returns>A task that represents the asynchronous operation.</returns> /// <returns>A task that represents the asynchronous operation.</returns>
public async Task UpdateMessageAsync(StoreAndForwardMessage message) public async Task UpdateMessageAsync(StoreAndForwardMessage message)
{ {
await using var connection = await OpenConnectionAsync(); await using var connection = OpenConnection();
await using var cmd = connection.CreateCommand(); await using var cmd = connection.CreateCommand();
cmd.CommandText = @" cmd.CommandText = @"
@@ -446,7 +292,7 @@ public class StoreAndForwardStorage
StoreAndForwardMessage message, StoreAndForwardMessage message,
StoreAndForwardMessageStatus expectedStatus) StoreAndForwardMessageStatus expectedStatus)
{ {
await using var connection = await OpenConnectionAsync(); await using var connection = OpenConnection();
await using var cmd = connection.CreateCommand(); await using var cmd = connection.CreateCommand();
cmd.CommandText = @" cmd.CommandText = @"
@@ -479,7 +325,7 @@ public class StoreAndForwardStorage
/// <returns>A task that represents the asynchronous operation.</returns> /// <returns>A task that represents the asynchronous operation.</returns>
public async Task RemoveMessageAsync(string messageId) public async Task RemoveMessageAsync(string messageId)
{ {
await using var connection = await OpenConnectionAsync(); await using var connection = OpenConnection();
await using var cmd = connection.CreateCommand(); await using var cmd = connection.CreateCommand();
cmd.CommandText = "DELETE FROM sf_messages WHERE id = @id"; cmd.CommandText = "DELETE FROM sf_messages WHERE id = @id";
@@ -500,7 +346,7 @@ public class StoreAndForwardStorage
int pageNumber = 1, int pageNumber = 1,
int pageSize = 50) int pageSize = 50)
{ {
await using var connection = await OpenConnectionAsync(); await using var connection = OpenConnection();
await using var transaction = (SqliteTransaction)await connection.BeginTransactionAsync(); await using var transaction = (SqliteTransaction)await connection.BeginTransactionAsync();
@@ -545,7 +391,7 @@ public class StoreAndForwardStorage
/// <returns>A task that resolves to <c>true</c> if the message was found and reset to Pending; <c>false</c> if not found or not in Parked status.</returns> /// <returns>A task that resolves to <c>true</c> if the message was found and reset to Pending; <c>false</c> if not found or not in Parked status.</returns>
public async Task<bool> RetryParkedMessageAsync(string messageId) public async Task<bool> RetryParkedMessageAsync(string messageId)
{ {
await using var connection = await OpenConnectionAsync(); await using var connection = OpenConnection();
await using var cmd = connection.CreateCommand(); await using var cmd = connection.CreateCommand();
cmd.CommandText = @" cmd.CommandText = @"
@@ -569,7 +415,7 @@ public class StoreAndForwardStorage
/// <returns>A task that resolves to <c>true</c> if the message was found and deleted; <c>false</c> if not found or not in Parked status.</returns> /// <returns>A task that resolves to <c>true</c> if the message was found and deleted; <c>false</c> if not found or not in Parked status.</returns>
public async Task<bool> DiscardParkedMessageAsync(string messageId) public async Task<bool> DiscardParkedMessageAsync(string messageId)
{ {
await using var connection = await OpenConnectionAsync(); await using var connection = OpenConnection();
await using var cmd = connection.CreateCommand(); await using var cmd = connection.CreateCommand();
cmd.CommandText = "DELETE FROM sf_messages WHERE id = @id AND status = @parked"; cmd.CommandText = "DELETE FROM sf_messages WHERE id = @id AND status = @parked";
@@ -586,7 +432,7 @@ public class StoreAndForwardStorage
/// <returns>A task that resolves to a dictionary mapping each category to its pending message count.</returns> /// <returns>A task that resolves to a dictionary mapping each category to its pending message count.</returns>
public async Task<Dictionary<StoreAndForwardCategory, int>> GetBufferDepthByCategoryAsync() public async Task<Dictionary<StoreAndForwardCategory, int>> GetBufferDepthByCategoryAsync()
{ {
await using var connection = await OpenConnectionAsync(); await using var connection = OpenConnection();
await using var cmd = connection.CreateCommand(); await using var cmd = connection.CreateCommand();
cmd.CommandText = @" cmd.CommandText = @"
@@ -616,7 +462,7 @@ public class StoreAndForwardStorage
/// <returns>A task that resolves to the number of messages whose origin instance matches <paramref name="instanceName"/>.</returns> /// <returns>A task that resolves to the number of messages whose origin instance matches <paramref name="instanceName"/>.</returns>
public async Task<int> GetMessageCountByOriginInstanceAsync(string instanceName) public async Task<int> GetMessageCountByOriginInstanceAsync(string instanceName)
{ {
await using var connection = await OpenConnectionAsync(); await using var connection = OpenConnection();
await using var cmd = connection.CreateCommand(); await using var cmd = connection.CreateCommand();
cmd.CommandText = @" cmd.CommandText = @"
@@ -635,7 +481,7 @@ public class StoreAndForwardStorage
/// <returns>A task that resolves to the matching message, or <c>null</c> if not found.</returns> /// <returns>A task that resolves to the matching message, or <c>null</c> if not found.</returns>
public async Task<StoreAndForwardMessage?> GetMessageByIdAsync(string messageId) public async Task<StoreAndForwardMessage?> GetMessageByIdAsync(string messageId)
{ {
await using var connection = await OpenConnectionAsync(); await using var connection = OpenConnection();
await using var cmd = connection.CreateCommand(); await using var cmd = connection.CreateCommand();
cmd.CommandText = @" cmd.CommandText = @"
@@ -656,7 +502,7 @@ public class StoreAndForwardStorage
/// <returns>A task that resolves to the number of messages currently in Parked status.</returns> /// <returns>A task that resolves to the number of messages currently in Parked status.</returns>
public async Task<int> GetParkedMessageCountAsync() public async Task<int> GetParkedMessageCountAsync()
{ {
await using var conn = await OpenConnectionAsync(); await using var conn = OpenConnection();
await using var cmd = conn.CreateCommand(); await using var cmd = conn.CreateCommand();
cmd.CommandText = "SELECT COUNT(*) FROM sf_messages WHERE status = @parked"; cmd.CommandText = "SELECT COUNT(*) FROM sf_messages WHERE status = @parked";
cmd.Parameters.AddWithValue("@parked", (int)StoreAndForwardMessageStatus.Parked); cmd.Parameters.AddWithValue("@parked", (int)StoreAndForwardMessageStatus.Parked);
@@ -674,7 +520,7 @@ public class StoreAndForwardStorage
/// <returns>A task that resolves to the oldest parked row's creation time, or <c>null</c> if none are parked.</returns> /// <returns>A task that resolves to the oldest parked row's creation time, or <c>null</c> if none are parked.</returns>
public async Task<DateTimeOffset?> GetOldestParkedCreatedAtAsync() public async Task<DateTimeOffset?> GetOldestParkedCreatedAtAsync()
{ {
await using var conn = await OpenConnectionAsync(); await using var conn = OpenConnection();
await using var cmd = conn.CreateCommand(); await using var cmd = conn.CreateCommand();
cmd.CommandText = "SELECT MIN(created_at) FROM sf_messages WHERE status = @parked"; cmd.CommandText = "SELECT MIN(created_at) FROM sf_messages WHERE status = @parked";
cmd.Parameters.AddWithValue("@parked", (int)StoreAndForwardMessageStatus.Parked); cmd.Parameters.AddWithValue("@parked", (int)StoreAndForwardMessageStatus.Parked);
@@ -691,7 +537,7 @@ public class StoreAndForwardStorage
/// <returns>A task that resolves to the count of messages with the specified status.</returns> /// <returns>A task that resolves to the count of messages with the specified status.</returns>
public async Task<int> GetMessageCountByStatusAsync(StoreAndForwardMessageStatus status) public async Task<int> GetMessageCountByStatusAsync(StoreAndForwardMessageStatus status)
{ {
await using var connection = await OpenConnectionAsync(); await using var connection = OpenConnection();
await using var cmd = connection.CreateCommand(); await using var cmd = connection.CreateCommand();
cmd.CommandText = "SELECT COUNT(*) FROM sf_messages WHERE status = @status"; cmd.CommandText = "SELECT COUNT(*) FROM sf_messages WHERE status = @status";
tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Integration/ParentExecutionIdCorrelationTests.cs
+15 -4
View File
@@ -35,6 +35,7 @@ using ZB.MOM.WW.ScadaBridge.NotificationOutbox.Delivery;
using ZB.MOM.WW.ScadaBridge.NotificationOutbox.Messages; using ZB.MOM.WW.ScadaBridge.NotificationOutbox.Messages;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts;
using ZB.MOM.WW.ScadaBridge.StoreAndForward; using ZB.MOM.WW.ScadaBridge.StoreAndForward;
using ZB.MOM.WW.ScadaBridge.TestSupport;
namespace ZB.MOM.WW.ScadaBridge.AuditLog.Tests.Integration; namespace ZB.MOM.WW.ScadaBridge.AuditLog.Tests.Integration;
@@ -197,11 +198,15 @@ public class ParentExecutionIdCorrelationTests : TestKit, IClassFixture<MsSqlMig
stubClient); stubClient);
// Site Store-and-Forward — Notify.Send buffers a NotificationSubmit here. // Site Store-and-Forward — Notify.Send buffers a NotificationSubmit here.
using var safKeepAlive = new Microsoft.Data.Sqlite.SqliteConnection( // The storage takes an ILocalDb and LocalDb has no in-memory mode, so this is a
$"Data Source=parentexec-saf-{Guid.NewGuid():N};Mode=Memory;Cache=Shared"); // real temp file, disposed (the master connection anchors the WAL) and deleted in
safKeepAlive.Open(); // the finally below.
var safLocalDb = TestLocalDb.CreateTemp("parentexec-saf");
var safDbPath = safLocalDb.Path;
try
{
var safStorage = new StoreAndForwardStorage( var safStorage = new StoreAndForwardStorage(
safKeepAlive.ConnectionString, NullLogger<StoreAndForwardStorage>.Instance); safLocalDb.Db, NullLogger<StoreAndForwardStorage>.Instance);
await safStorage.InitializeAsync(); await safStorage.InitializeAsync();
var storeAndForward = new StoreAndForwardService( var storeAndForward = new StoreAndForwardService(
safStorage, safStorage,
@@ -366,6 +371,12 @@ public class ParentExecutionIdCorrelationTests : TestKit, IClassFixture<MsSqlMig
AssertChain(treeFromRoot, inboundExecutionId, routedExecutionId); AssertChain(treeFromRoot, inboundExecutionId, routedExecutionId);
}, TimeSpan.FromSeconds(90)); }, TimeSpan.FromSeconds(90));
} }
finally
{
safLocalDb.Dispose();
TestLocalDb.DeleteFiles(safDbPath);
}
}
/// <summary> /// <summary>
/// Asserts the execution tree is the expected two-node inbound→routed chain: /// Asserts the execution tree is the expected two-node inbound→routed chain:
tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/Site/Telemetry/SiteAuditTelemetryActorTests.cs
+52
View File
@@ -5,6 +5,7 @@ using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options; using Microsoft.Extensions.Options;
using NSubstitute; using NSubstitute;
using NSubstitute.ExceptionExtensions; using NSubstitute.ExceptionExtensions;
using NSubstitute.ReceivedExtensions;
using ZB.MOM.WW.ScadaBridge.AuditLog.Site.Telemetry; using ZB.MOM.WW.ScadaBridge.AuditLog.Site.Telemetry;
using ZB.MOM.WW.Audit; using ZB.MOM.WW.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Audit; using ZB.MOM.WW.ScadaBridge.Commons.Types.Audit;
@@ -451,4 +452,55 @@ public class SiteAuditTelemetryActorTests : TestKit
await _client.DidNotReceiveWithAnyArgs().IngestCachedTelemetryAsync(default!, default); await _client.DidNotReceiveWithAnyArgs().IngestCachedTelemetryAsync(default!, default);
await _queue.DidNotReceiveWithAnyArgs().ReadPendingCachedTelemetryAsync(default, default); await _queue.DidNotReceiveWithAnyArgs().ReadPendingCachedTelemetryAsync(default, default);
} }
/// <summary>
/// Regression for the 2026-07-20 rig finding (known-issues/2026-07-20-localdb-disk-io-error-under-load.md §8):
/// both drain handlers await with <c>ConfigureAwait(false)</c>, so when a
/// dependency call completes off the actor thread (as real SQLite/gRPC
/// always do) the <c>finally</c>-block re-arm runs on a pool thread with no
/// active ActorContext. Depending on what that thread's thread-static cell
/// slot holds, <c>Context</c>/<c>Self</c> either throw
/// <c>NotSupportedException: There is no active ActorContext</c> (crashing
/// the actor once per drain — the rig's logged variant) or silently resolve
/// to a STALE cell of some other actor, misrouting the tick so the drain
/// loop stops. The two assertions below catch one variant each. Every other
/// test in this class masks the bug by returning already-completed tasks
/// from the mocks, which keeps the continuations on the actor thread.
/// </summary>
[Fact]
public async Task Drains_Whose_Awaits_Complete_Off_The_Actor_Thread_Keep_Draining_Without_Crashing()
{
// Task.Delay completes on a timer thread; with ConfigureAwait(false)
// everything after the await — including the finally-block re-arm —
// stays off the actor context.
_queue.ReadPendingAsync(Arg.Any<int>(), Arg.Any<CancellationToken>())
.Returns(async _ =>
{
await Task.Delay(25).ConfigureAwait(false);
return (IReadOnlyList<AuditEvent>)Array.Empty<AuditEvent>();
});
_queue.ReadPendingCachedTelemetryAsync(Arg.Any<int>(), Arg.Any<CancellationToken>())
.Returns(async _ =>
{
await Task.Delay(25).ConfigureAwait(false);
return (IReadOnlyList<AuditEvent>)Array.Empty<AuditEvent>();
});
// Variant 1 (throw → restart storm): any NotSupportedException logged
// during the run fails the filter. Variant 2 (silent misroute → the
// drain loop stalls after the first tick): the sustained-drain
// assertion inside fails because reads stop at 1.
await EventFilter.Exception<NotSupportedException>().ExpectAsync(0, async () =>
{
CreateActorWithCachedDrain(Opts(busySeconds: 1, idleSeconds: 1));
// Three full cycles prove the finally-block re-arm works from an
// off-context thread: tick → drain → re-arm → tick → …
await AwaitAssertAsync(async () =>
{
await _queue.Received(Quantity.Within(3, int.MaxValue)).ReadPendingAsync(Arg.Any<int>(), Arg.Any<CancellationToken>());
await _queue.Received(Quantity.Within(3, int.MaxValue)).ReadPendingCachedTelemetryAsync(Arg.Any<int>(), Arg.Any<CancellationToken>());
}, TimeSpan.FromSeconds(10));
});
}
} }
tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests/ZB.MOM.WW.ScadaBridge.AuditLog.Tests.csproj
+2 -1
View File
@@ -76,7 +76,8 @@
central MSSQL AuditLog. central MSSQL AuditLog.
--> -->
<ProjectReference Include="../../src/ZB.MOM.WW.ScadaBridge.InboundAPI/ZB.MOM.WW.ScadaBridge.InboundAPI.csproj" /> <ProjectReference Include="../../src/ZB.MOM.WW.ScadaBridge.InboundAPI/ZB.MOM.WW.ScadaBridge.InboundAPI.csproj" />
</ItemGroup> <ProjectReference Include="../ZB.MOM.WW.ScadaBridge.TestSupport/ZB.MOM.WW.ScadaBridge.TestSupport.csproj" />
</ItemGroup>
<ItemGroup> <ItemGroup>
<!-- M4 Bundle E (Task E3): need ASP.NET Core for the TestHost-based middleware E2E. --> <!-- M4 Bundle E (Task E3): need ASP.NET Core for the TestHost-based middleware E2E. -->
tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/DatabaseGatewayTests.cs
+71 -50
View File
@@ -3,18 +3,56 @@ using System.Data.Common;
using System.Text.Json; using System.Text.Json;
using Microsoft.Extensions.Logging.Abstractions; using Microsoft.Extensions.Logging.Abstractions;
using NSubstitute; using NSubstitute;
using ZB.MOM.WW.LocalDb;
using ZB.MOM.WW.ScadaBridge.Commons.Entities.ExternalSystems; using ZB.MOM.WW.ScadaBridge.Commons.Entities.ExternalSystems;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories; using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
using ZB.MOM.WW.ScadaBridge.TestSupport;
namespace ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests; namespace ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests;
/// <summary> /// <summary>
/// WP-9: Tests for Database access — connection resolution, cached writes. /// WP-9: Tests for Database access — connection resolution, cached writes.
/// </summary> /// </summary>
public class DatabaseGatewayTests public class DatabaseGatewayTests : IDisposable
{ {
private readonly IExternalSystemRepository _repository = Substitute.For<IExternalSystemRepository>(); private readonly IExternalSystemRepository _repository = Substitute.For<IExternalSystemRepository>();
/// <summary>
/// Temp local databases opened by the Store-and-Forward tests, torn down in
/// <see cref="Dispose"/>.
/// </summary>
private readonly List<TestLocalDb> _localDbs = new();
/// <summary>
/// Opens a real <c>ILocalDb</c> over a fresh temp file for a test that needs a live
/// <see cref="ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardStorage"/>. The
/// store takes an <c>ILocalDb</c> and LocalDb has no in-memory mode, so each test gets
/// its own file rather than a shared-cache in-memory database held open by a
/// keep-alive connection.
/// </summary>
private TestLocalDb CreateLocalDb(string prefix)
{
var localDb = TestLocalDb.CreateTemp(prefix);
_localDbs.Add(localDb);
return localDb;
}
/// <summary>
/// Disposes every temp local database and then deletes its files — in that order,
/// because the master connection LocalDb holds anchors the WAL sidecars.
/// </summary>
public void Dispose()
{
foreach (var localDb in _localDbs)
{
var path = localDb.Path;
localDb.Dispose();
TestLocalDb.DeleteFiles(path);
}
GC.SuppressFinalize(this);
}
/// <summary> /// <summary>
/// Configures the repository substitute for the name-keyed connection-resolution /// Configures the repository substitute for the name-keyed connection-resolution
/// path used by <c>DatabaseGateway</c> (ExternalSystemGateway-011). A <c>null</c> /// path used by <c>DatabaseGateway</c> (ExternalSystemGateway-011). A <c>null</c>
@@ -84,12 +122,9 @@ public class DatabaseGatewayTests
}; };
StubConnection(conn); StubConnection(conn);
var dbName = $"EsgCachedWrite_{Guid.NewGuid():N}"; var localDb = CreateLocalDb("EsgCachedWrite");
var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared";
using var keepAlive = new Microsoft.Data.Sqlite.SqliteConnection(connStr);
keepAlive.Open();
var storage = new ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardStorage( var storage = new ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardStorage(
connStr, NullLogger<ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardStorage>.Instance); localDb.Db, NullLogger<ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardStorage>.Instance);
await storage.InitializeAsync(); await storage.InitializeAsync();
var sfOptions = new ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardOptions var sfOptions = new ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardOptions
{ {
@@ -121,7 +156,7 @@ public class DatabaseGatewayTests
var depth = await storage.GetBufferDepthByCategoryAsync(); var depth = await storage.GetBufferDepthByCategoryAsync();
Assert.Equal(1, depth[ZB.MOM.WW.ScadaBridge.Commons.Types.Enums.StoreAndForwardCategory.CachedDbWrite]); Assert.Equal(1, depth[ZB.MOM.WW.ScadaBridge.Commons.Types.Enums.StoreAndForwardCategory.CachedDbWrite]);
var buffered = ReadBufferedRetrySettings(connStr); var buffered = ReadBufferedRetrySettings(localDb.Db);
Assert.Equal(5, buffered.MaxRetries); Assert.Equal(5, buffered.MaxRetries);
Assert.Equal((long)TimeSpan.FromSeconds(12).TotalMilliseconds, buffered.RetryIntervalMs); Assert.Equal((long)TimeSpan.FromSeconds(12).TotalMilliseconds, buffered.RetryIntervalMs);
@@ -148,12 +183,9 @@ public class DatabaseGatewayTests
}; };
StubConnection(conn); StubConnection(conn);
var dbName = $"EsgCachedWriteZero_{Guid.NewGuid():N}"; var localDb = CreateLocalDb("EsgCachedWriteZero");
var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared";
using var keepAlive = new Microsoft.Data.Sqlite.SqliteConnection(connStr);
keepAlive.Open();
var storage = new ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardStorage( var storage = new ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardStorage(
connStr, NullLogger<ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardStorage>.Instance); localDb.Db, NullLogger<ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardStorage>.Instance);
await storage.InitializeAsync(); await storage.InitializeAsync();
var sfOptions = new ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardOptions var sfOptions = new ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardOptions
{ {
@@ -172,7 +204,7 @@ public class DatabaseGatewayTests
await gateway.CachedWriteAsync("testDb", "INSERT INTO t VALUES (1)"); await gateway.CachedWriteAsync("testDb", "INSERT INTO t VALUES (1)");
var (maxRetries, _, _, _) = ReadBufferedRetrySettings(connStr); var (maxRetries, _, _, _) = ReadBufferedRetrySettings(localDb.Db);
// Must be the bounded S&F default, never 0 — a stored 0 would mean retry-forever. // Must be the bounded S&F default, never 0 — a stored 0 would mean retry-forever.
Assert.Equal(99, maxRetries); Assert.Equal(99, maxRetries);
Assert.NotEqual(0, maxRetries); Assert.NotEqual(0, maxRetries);
@@ -182,19 +214,15 @@ public class DatabaseGatewayTests
// cached-write attempt + the buffered retry path ── // cached-write attempt + the buffered retry path ──
/// <summary> /// <summary>
/// Builds a real, initialised in-memory store-and-forward service plus a /// Builds a real, initialised store-and-forward service over a fresh temp local
/// keep-alive connection (the SQLite shared-cache DB lives only while a /// database. The database is torn down by <see cref="Dispose"/>.
/// connection is open). The caller disposes <paramref name="keepAlive"/>.
/// </summary> /// </summary>
private static (ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardService Sf, string ConnStr, Microsoft.Data.Sqlite.SqliteConnection KeepAlive) private (ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardService Sf, ILocalDb Db)
NewStoreAndForward() NewStoreAndForward()
{ {
var dbName = $"EsgCachedWriteClassify_{Guid.NewGuid():N}"; var localDb = CreateLocalDb("EsgCachedWriteClassify");
var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared";
var keepAlive = new Microsoft.Data.Sqlite.SqliteConnection(connStr);
keepAlive.Open();
var storage = new ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardStorage( var storage = new ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardStorage(
connStr, NullLogger<ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardStorage>.Instance); localDb.Db, NullLogger<ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardStorage>.Instance);
storage.InitializeAsync().GetAwaiter().GetResult(); storage.InitializeAsync().GetAwaiter().GetResult();
var sfOptions = new ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardOptions var sfOptions = new ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardOptions
{ {
@@ -204,7 +232,7 @@ public class DatabaseGatewayTests
}; };
var sf = new ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardService( var sf = new ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardService(
storage, sfOptions, NullLogger<ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardService>.Instance); storage, sfOptions, NullLogger<ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardService>.Instance);
return (sf, connStr, keepAlive); return (sf, localDb.Db);
} }
[Fact] [Fact]
@@ -217,8 +245,7 @@ public class DatabaseGatewayTests
var conn = new DatabaseConnectionDefinition("testDb", "Server=localhost;Database=test") { Id = 1 }; var conn = new DatabaseConnectionDefinition("testDb", "Server=localhost;Database=test") { Id = 1 };
StubConnection(conn); StubConnection(conn);
var (sf, connStr, keepAlive) = NewStoreAndForward(); var (sf, sfDb) = NewStoreAndForward();
using var _ = keepAlive;
var gateway = new ExecuteStubGateway( var gateway = new ExecuteStubGateway(
_repository, _repository,
@@ -233,7 +260,7 @@ public class DatabaseGatewayTests
Assert.NotNull(result.ErrorMessage); Assert.NotNull(result.ErrorMessage);
// Nothing buffered — the permanent failure short-circuited S&F. // Nothing buffered — the permanent failure short-circuited S&F.
Assert.Equal(0, ReadBufferDepth(connStr)); Assert.Equal(0, ReadBufferDepth(sfDb));
} }
[Fact] [Fact]
@@ -249,8 +276,7 @@ public class DatabaseGatewayTests
}; };
StubConnection(conn); StubConnection(conn);
var (sf, connStr, keepAlive) = NewStoreAndForward(); var (sf, sfDb) = NewStoreAndForward();
using var _ = keepAlive;
var gateway = new ExecuteStubGateway( var gateway = new ExecuteStubGateway(
_repository, _repository,
@@ -265,7 +291,7 @@ public class DatabaseGatewayTests
Assert.True(result.WasBuffered); // handed to S&F, not synchronously failed Assert.True(result.WasBuffered); // handed to S&F, not synchronously failed
Assert.Null(result.ErrorMessage); Assert.Null(result.ErrorMessage);
Assert.Equal(1, ReadBufferDepth(connStr)); Assert.Equal(1, ReadBufferDepth(sfDb));
} }
[Fact] [Fact]
@@ -277,8 +303,7 @@ public class DatabaseGatewayTests
var conn = new DatabaseConnectionDefinition("testDb", "Server=localhost;Database=test") { Id = 1 }; var conn = new DatabaseConnectionDefinition("testDb", "Server=localhost;Database=test") { Id = 1 };
StubConnection(conn); StubConnection(conn);
var (sf, connStr, keepAlive) = NewStoreAndForward(); var (sf, sfDb) = NewStoreAndForward();
using var _ = keepAlive;
var gateway = new ExecuteStubGateway(_repository, sf, onExecute: () => { /* succeeds */ }); var gateway = new ExecuteStubGateway(_repository, sf, onExecute: () => { /* succeeds */ });
@@ -288,7 +313,7 @@ public class DatabaseGatewayTests
Assert.False(result.WasBuffered); Assert.False(result.WasBuffered);
Assert.Null(result.ErrorMessage); Assert.Null(result.ErrorMessage);
Assert.Equal(0, ReadBufferDepth(connStr)); Assert.Equal(0, ReadBufferDepth(sfDb));
} }
[Fact] [Fact]
@@ -381,8 +406,7 @@ public class DatabaseGatewayTests
}; };
StubConnection(conn); StubConnection(conn);
var (sf, connStr, keepAlive) = NewStoreAndForward(); var (sf, sfDb) = NewStoreAndForward();
using var _ = keepAlive;
// RawExecuteStubGateway routes the raw throw through the PRODUCTION // RawExecuteStubGateway routes the raw throw through the PRODUCTION
// ExecuteWriteAsync classification (the seam under test), unlike // ExecuteWriteAsync classification (the seam under test), unlike
@@ -395,7 +419,7 @@ public class DatabaseGatewayTests
Assert.True(result.WasBuffered); // handed to S&F as transient Assert.True(result.WasBuffered); // handed to S&F as transient
Assert.Null(result.ErrorMessage); // not a permanent Failed result Assert.Null(result.ErrorMessage); // not a permanent Failed result
Assert.Equal(1, ReadBufferDepth(connStr)); Assert.Equal(1, ReadBufferDepth(sfDb));
} }
[Fact] [Fact]
@@ -408,8 +432,7 @@ public class DatabaseGatewayTests
var conn = new DatabaseConnectionDefinition("testDb", "Server=localhost;Database=test") { Id = 1 }; var conn = new DatabaseConnectionDefinition("testDb", "Server=localhost;Database=test") { Id = 1 };
StubConnection(conn); StubConnection(conn);
var (sf, connStr, keepAlive) = NewStoreAndForward(); var (sf, sfDb) = NewStoreAndForward();
using var _ = keepAlive;
using var cts = new CancellationTokenSource(); using var cts = new CancellationTokenSource();
cts.Cancel(); cts.Cancel();
@@ -421,7 +444,7 @@ public class DatabaseGatewayTests
() => gateway.CachedWriteAsync("testDb", "INSERT INTO t VALUES (1)", cancellationToken: cts.Token)); () => gateway.CachedWriteAsync("testDb", "INSERT INTO t VALUES (1)", cancellationToken: cts.Token));
// Cancellation is not a transient failure — nothing must have been buffered. // Cancellation is not a transient failure — nothing must have been buffered.
Assert.Equal(0, ReadBufferDepth(connStr)); Assert.Equal(0, ReadBufferDepth(sfDb));
} }
[Fact] [Fact]
@@ -434,8 +457,7 @@ public class DatabaseGatewayTests
var conn = new DatabaseConnectionDefinition("testDb", "Server=localhost;Database=test") { Id = 1 }; var conn = new DatabaseConnectionDefinition("testDb", "Server=localhost;Database=test") { Id = 1 };
StubConnection(conn); StubConnection(conn);
var (sf, connStr, keepAlive) = NewStoreAndForward(); var (sf, sfDb) = NewStoreAndForward();
using var _ = keepAlive;
var gateway = new RawExecuteStubGateway( var gateway = new RawExecuteStubGateway(
_repository, sf, onRunSql: () => throw new ArgumentException("authoring bug")); _repository, sf, onRunSql: () => throw new ArgumentException("authoring bug"));
@@ -443,7 +465,7 @@ public class DatabaseGatewayTests
await Assert.ThrowsAsync<ArgumentException>( await Assert.ThrowsAsync<ArgumentException>(
() => gateway.CachedWriteAsync("testDb", "INSERT INTO t VALUES (1)")); () => gateway.CachedWriteAsync("testDb", "INSERT INTO t VALUES (1)"));
Assert.Equal(0, ReadBufferDepth(connStr)); Assert.Equal(0, ReadBufferDepth(sfDb));
} }
[Fact] [Fact]
@@ -486,8 +508,7 @@ public class DatabaseGatewayTests
var conn = new DatabaseConnectionDefinition("testDb", "Server=localhost;Database=test") { Id = 1 }; var conn = new DatabaseConnectionDefinition("testDb", "Server=localhost;Database=test") { Id = 1 };
StubConnection(conn); StubConnection(conn);
var (sf, connStr, keepAlive) = NewStoreAndForward(); var (sf, sfDb) = NewStoreAndForward();
using var _ = keepAlive;
using var cts = new CancellationTokenSource(); using var cts = new CancellationTokenSource();
cts.Cancel(); cts.Cancel();
@@ -505,7 +526,7 @@ public class DatabaseGatewayTests
// The cancel won — it must NOT have been classified as transient (buffered) // The cancel won — it must NOT have been classified as transient (buffered)
// nor returned as a permanent Failed result. // nor returned as a permanent Failed result.
Assert.Equal(0, ReadBufferDepth(connStr)); Assert.Equal(0, ReadBufferDepth(sfDb));
} }
/// <summary> /// <summary>
@@ -542,10 +563,10 @@ public class DatabaseGatewayTests
/// Reads the current buffered-message count off the S&amp;F SQLite DB by /// Reads the current buffered-message count off the S&amp;F SQLite DB by
/// counting <c>sf_messages</c> rows (the engine's persistence table). /// counting <c>sf_messages</c> rows (the engine's persistence table).
/// </summary> /// </summary>
private static int ReadBufferDepth(string connStr) private static int ReadBufferDepth(ILocalDb localDb)
{ {
using var conn = new Microsoft.Data.Sqlite.SqliteConnection(connStr); // CreateConnection hands back an already-open, pragma-configured connection.
conn.Open(); using var conn = localDb.CreateConnection();
using var cmd = conn.CreateCommand(); using var cmd = conn.CreateCommand();
cmd.CommandText = "SELECT COUNT(*) FROM sf_messages"; cmd.CommandText = "SELECT COUNT(*) FROM sf_messages";
return Convert.ToInt32(cmd.ExecuteScalar()); return Convert.ToInt32(cmd.ExecuteScalar());
@@ -615,10 +636,10 @@ public class DatabaseGatewayTests
} }
private static (int MaxRetries, long RetryIntervalMs, Guid? ExecutionId, string? SourceScript) private static (int MaxRetries, long RetryIntervalMs, Guid? ExecutionId, string? SourceScript)
ReadBufferedRetrySettings(string connStr) ReadBufferedRetrySettings(ILocalDb localDb)
{ {
using var conn = new Microsoft.Data.Sqlite.SqliteConnection(connStr); // CreateConnection hands back an already-open, pragma-configured connection.
conn.Open(); using var conn = localDb.CreateConnection();
using var cmd = conn.CreateCommand(); using var cmd = conn.CreateCommand();
cmd.CommandText = cmd.CommandText =
"SELECT max_retries, retry_interval_ms, execution_id, source_script FROM sf_messages"; "SELECT max_retries, retry_interval_ms, execution_id, source_script FROM sf_messages";
tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/ExternalSystemClientTests.cs
+53 -32
View File
@@ -1,24 +1,60 @@
using System.Net; using System.Net;
using System.Net.Http.Headers; using System.Net.Http.Headers;
using System.Text; using System.Text;
using Microsoft.Data.Sqlite;
using Microsoft.Extensions.Logging; using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Logging.Abstractions; using Microsoft.Extensions.Logging.Abstractions;
using NSubstitute; using NSubstitute;
using ZB.MOM.WW.LocalDb;
using ZB.MOM.WW.ScadaBridge.Commons.Entities.ExternalSystems; using ZB.MOM.WW.ScadaBridge.Commons.Entities.ExternalSystems;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories; using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
using ZB.MOM.WW.ScadaBridge.StoreAndForward; using ZB.MOM.WW.ScadaBridge.StoreAndForward;
using ZB.MOM.WW.ScadaBridge.TestSupport;
namespace ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests; namespace ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests;
/// <summary> /// <summary>
/// WP-6/7: Tests for ExternalSystemClient — HTTP client, call modes, error handling. /// WP-6/7: Tests for ExternalSystemClient — HTTP client, call modes, error handling.
/// </summary> /// </summary>
public class ExternalSystemClientTests public class ExternalSystemClientTests : IDisposable
{ {
private readonly IExternalSystemRepository _repository = Substitute.For<IExternalSystemRepository>(); private readonly IExternalSystemRepository _repository = Substitute.For<IExternalSystemRepository>();
private readonly IHttpClientFactory _httpClientFactory = Substitute.For<IHttpClientFactory>(); private readonly IHttpClientFactory _httpClientFactory = Substitute.For<IHttpClientFactory>();
/// <summary>
/// Temp local databases opened by the Store-and-Forward tests, torn down in
/// <see cref="Dispose"/>.
/// </summary>
private readonly List<TestLocalDb> _localDbs = new();
/// <summary>
/// Opens a real <c>ILocalDb</c> over a fresh temp file for a test that needs a live
/// <see cref="StoreAndForwardStorage"/>. The store takes an <c>ILocalDb</c> and LocalDb
/// has no in-memory mode, so each test gets its own file rather than a shared-cache
/// in-memory database held open by a keep-alive connection.
/// </summary>
private TestLocalDb CreateLocalDb(string prefix)
{
var localDb = TestLocalDb.CreateTemp(prefix);
_localDbs.Add(localDb);
return localDb;
}
/// <summary>
/// Disposes every temp local database and then deletes its files — in that order,
/// because the master connection LocalDb holds anchors the WAL sidecars.
/// </summary>
public void Dispose()
{
foreach (var localDb in _localDbs)
{
var path = localDb.Path;
localDb.Dispose();
TestLocalDb.DeleteFiles(path);
}
GC.SuppressFinalize(this);
}
/// <summary> /// <summary>
/// Configures the repository substitute for the name-keyed resolution path used by /// Configures the repository substitute for the name-keyed resolution path used by
/// <c>ExternalSystemClient</c> (ExternalSystemGateway-011). A <c>null</c> system or /// <c>ExternalSystemClient</c> (ExternalSystemGateway-011). A <c>null</c> system or
@@ -275,11 +311,8 @@ public class ExternalSystemClientTests
_httpClientFactory.CreateClient(Arg.Any<string>()).Returns(httpClient); _httpClientFactory.CreateClient(Arg.Any<string>()).Returns(httpClient);
// A real S&F service with a registered delivery handler that counts invocations. // A real S&F service with a registered delivery handler that counts invocations.
var dbName = $"EsgDoubleDispatch_{Guid.NewGuid():N}"; var localDb = CreateLocalDb("EsgDoubleDispatch");
var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared"; var storage = new StoreAndForwardStorage(localDb.Db, NullLogger<StoreAndForwardStorage>.Instance);
using var keepAlive = new SqliteConnection(connStr);
keepAlive.Open();
var storage = new StoreAndForwardStorage(connStr, NullLogger<StoreAndForwardStorage>.Instance);
await storage.InitializeAsync(); await storage.InitializeAsync();
var sfOptions = new StoreAndForwardOptions var sfOptions = new StoreAndForwardOptions
{ {
@@ -422,11 +455,8 @@ public class ExternalSystemClientTests
var httpClient = new HttpClient(new MockHttpMessageHandler(HttpStatusCode.InternalServerError, "boom")); var httpClient = new HttpClient(new MockHttpMessageHandler(HttpStatusCode.InternalServerError, "boom"));
_httpClientFactory.CreateClient(Arg.Any<string>()).Returns(httpClient); _httpClientFactory.CreateClient(Arg.Any<string>()).Returns(httpClient);
var dbName = $"EsgRetry_{Guid.NewGuid():N}"; var localDb = CreateLocalDb("EsgRetry");
var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared"; var storage = new StoreAndForwardStorage(localDb.Db, NullLogger<StoreAndForwardStorage>.Instance);
using var keepAlive = new SqliteConnection(connStr);
keepAlive.Open();
var storage = new StoreAndForwardStorage(connStr, NullLogger<StoreAndForwardStorage>.Instance);
await storage.InitializeAsync(); await storage.InitializeAsync();
// S&F defaults deliberately different from the system's settings. // S&F defaults deliberately different from the system's settings.
var sfOptions = new StoreAndForwardOptions var sfOptions = new StoreAndForwardOptions
@@ -454,7 +484,7 @@ public class ExternalSystemClientTests
var depth = await storage.GetBufferDepthByCategoryAsync(); var depth = await storage.GetBufferDepthByCategoryAsync();
Assert.Equal(1, depth[ZB.MOM.WW.ScadaBridge.Commons.Types.Enums.StoreAndForwardCategory.ExternalSystem]); Assert.Equal(1, depth[ZB.MOM.WW.ScadaBridge.Commons.Types.Enums.StoreAndForwardCategory.ExternalSystem]);
var buffered = ReadBufferedRetrySettings(connStr); var buffered = ReadBufferedRetrySettings(localDb.Db);
Assert.Equal(7, buffered.MaxRetries); Assert.Equal(7, buffered.MaxRetries);
Assert.Equal((long)TimeSpan.FromSeconds(42).TotalMilliseconds, buffered.RetryIntervalMs); Assert.Equal((long)TimeSpan.FromSeconds(42).TotalMilliseconds, buffered.RetryIntervalMs);
@@ -466,10 +496,10 @@ public class ExternalSystemClientTests
} }
private static (int MaxRetries, long RetryIntervalMs, Guid? ExecutionId, string? SourceScript) private static (int MaxRetries, long RetryIntervalMs, Guid? ExecutionId, string? SourceScript)
ReadBufferedRetrySettings(string connStr) ReadBufferedRetrySettings(ILocalDb localDb)
{ {
using var conn = new SqliteConnection(connStr); // CreateConnection hands back an already-open, pragma-configured connection.
conn.Open(); using var conn = localDb.CreateConnection();
using var cmd = conn.CreateCommand(); using var cmd = conn.CreateCommand();
cmd.CommandText = cmd.CommandText =
"SELECT max_retries, retry_interval_ms, execution_id, source_script FROM sf_messages"; "SELECT max_retries, retry_interval_ms, execution_id, source_script FROM sf_messages";
@@ -505,11 +535,8 @@ public class ExternalSystemClientTests
var httpClient = new HttpClient(new MockHttpMessageHandler(HttpStatusCode.InternalServerError, "boom")); var httpClient = new HttpClient(new MockHttpMessageHandler(HttpStatusCode.InternalServerError, "boom"));
_httpClientFactory.CreateClient(Arg.Any<string>()).Returns(httpClient); _httpClientFactory.CreateClient(Arg.Any<string>()).Returns(httpClient);
var dbName = $"EsgRetryZero_{Guid.NewGuid():N}"; var localDb = CreateLocalDb("EsgRetryZero");
var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared"; var storage = new StoreAndForwardStorage(localDb.Db, NullLogger<StoreAndForwardStorage>.Instance);
using var keepAlive = new SqliteConnection(connStr);
keepAlive.Open();
var storage = new StoreAndForwardStorage(connStr, NullLogger<StoreAndForwardStorage>.Instance);
await storage.InitializeAsync(); await storage.InitializeAsync();
var sfOptions = new StoreAndForwardOptions var sfOptions = new StoreAndForwardOptions
{ {
@@ -525,7 +552,7 @@ public class ExternalSystemClientTests
await client.CachedCallAsync("TestAPI", "postData"); await client.CachedCallAsync("TestAPI", "postData");
var (maxRetries, _, _, _) = ReadBufferedRetrySettings(connStr); var (maxRetries, _, _, _) = ReadBufferedRetrySettings(localDb.Db);
// Must be the bounded S&F default, never 0 — a stored 0 would mean retry-forever. // Must be the bounded S&F default, never 0 — a stored 0 would mean retry-forever.
Assert.Equal(99, maxRetries); Assert.Equal(99, maxRetries);
Assert.NotEqual(0, maxRetries); Assert.NotEqual(0, maxRetries);
@@ -651,11 +678,8 @@ public class ExternalSystemClientTests
var httpClient = new HttpClient(new HangingHttpMessageHandler(TimeSpan.FromMinutes(10))); var httpClient = new HttpClient(new HangingHttpMessageHandler(TimeSpan.FromMinutes(10)));
_httpClientFactory.CreateClient(Arg.Any<string>()).Returns(httpClient); _httpClientFactory.CreateClient(Arg.Any<string>()).Returns(httpClient);
var dbName = $"EsgCancel_{Guid.NewGuid():N}"; var localDb = CreateLocalDb("EsgCancel");
var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared"; var storage = new StoreAndForwardStorage(localDb.Db, NullLogger<StoreAndForwardStorage>.Instance);
using var keepAlive = new SqliteConnection(connStr);
keepAlive.Open();
var storage = new StoreAndForwardStorage(connStr, NullLogger<StoreAndForwardStorage>.Instance);
await storage.InitializeAsync(); await storage.InitializeAsync();
var sfOptions = new StoreAndForwardOptions var sfOptions = new StoreAndForwardOptions
{ {
@@ -1060,11 +1084,8 @@ public class ExternalSystemClientTests
.Returns(_ => new HttpClient(new MockHttpMessageHandler(HttpStatusCode.OK, hugeBody))); .Returns(_ => new HttpClient(new MockHttpMessageHandler(HttpStatusCode.OK, hugeBody)));
// A real S&F service so we can assert the oversized response is NOT buffered. // A real S&F service so we can assert the oversized response is NOT buffered.
var dbName = $"EsgOversize_{Guid.NewGuid():N}"; var localDb = CreateLocalDb("EsgOversize");
var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared"; var storage = new StoreAndForwardStorage(localDb.Db, NullLogger<StoreAndForwardStorage>.Instance);
using var keepAlive = new SqliteConnection(connStr);
keepAlive.Open();
var storage = new StoreAndForwardStorage(connStr, NullLogger<StoreAndForwardStorage>.Instance);
await storage.InitializeAsync(); await storage.InitializeAsync();
var sfOptions = new StoreAndForwardOptions var sfOptions = new StoreAndForwardOptions
{ {
tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.Tests.csproj
+2 -1
View File
@@ -24,6 +24,7 @@
<ProjectReference Include="../../src/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.csproj" /> <ProjectReference Include="../../src/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway/ZB.MOM.WW.ScadaBridge.ExternalSystemGateway.csproj" />
<ProjectReference Include="../../src/ZB.MOM.WW.ScadaBridge.Commons/ZB.MOM.WW.ScadaBridge.Commons.csproj" /> <ProjectReference Include="../../src/ZB.MOM.WW.ScadaBridge.Commons/ZB.MOM.WW.ScadaBridge.Commons.csproj" />
<ProjectReference Include="../../src/ZB.MOM.WW.ScadaBridge.StoreAndForward/ZB.MOM.WW.ScadaBridge.StoreAndForward.csproj" /> <ProjectReference Include="../../src/ZB.MOM.WW.ScadaBridge.StoreAndForward/ZB.MOM.WW.ScadaBridge.StoreAndForward.csproj" />
</ItemGroup> <ProjectReference Include="../ZB.MOM.WW.ScadaBridge.TestSupport/ZB.MOM.WW.ScadaBridge.TestSupport.csproj" />
</ItemGroup>
</Project> </Project>
tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests/HealthReportSenderTests.cs
+13 -8
View File
@@ -1,10 +1,10 @@
using Microsoft.Data.Sqlite;
using Microsoft.Extensions.Logging; using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Logging.Abstractions; using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options; using Microsoft.Extensions.Options;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health; using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
using ZB.MOM.WW.ScadaBridge.StoreAndForward; using ZB.MOM.WW.ScadaBridge.StoreAndForward;
using ZB.MOM.WW.ScadaBridge.TestSupport;
namespace ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests; namespace ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests;
@@ -191,13 +191,11 @@ public class HealthReportSenderTests
[Fact] [Fact]
public async Task ReportsIncludeStoreAndForwardBufferDepthsFromStorage() public async Task ReportsIncludeStoreAndForwardBufferDepthsFromStorage()
{ {
var dbName = $"HealthSfDepth_{Guid.NewGuid():N}"; // LocalDb has no in-memory mode, so the store runs over a real temp file.
var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared"; var localDb = TestLocalDb.CreateTemp("HealthSfDepth");
// Keep one connection alive so the in-memory DB persists for the test. try
using var keepAlive = new SqliteConnection(connStr); {
keepAlive.Open(); var storage = new StoreAndForwardStorage(localDb.Db, NullLogger<StoreAndForwardStorage>.Instance);
var storage = new StoreAndForwardStorage(connStr, NullLogger<StoreAndForwardStorage>.Instance);
await storage.InitializeAsync(); await storage.InitializeAsync();
// Two pending ExternalSystem messages and one pending Notification message. // Two pending ExternalSystem messages and one pending Notification message.
@@ -236,6 +234,13 @@ public class HealthReportSenderTests
Assert.Equal(1, depths[nameof(StoreAndForwardCategory.Notification)]); Assert.Equal(1, depths[nameof(StoreAndForwardCategory.Notification)]);
Assert.False(depths.ContainsKey(nameof(StoreAndForwardCategory.CachedDbWrite))); Assert.False(depths.ContainsKey(nameof(StoreAndForwardCategory.CachedDbWrite)));
} }
finally
{
// The master connection anchors the WAL — dispose before deleting.
localDb.Dispose();
TestLocalDb.DeleteFiles(localDb.Path);
}
}
private static StoreAndForwardMessage MakePendingMessage(string id, StoreAndForwardCategory category) => private static StoreAndForwardMessage MakePendingMessage(string id, StoreAndForwardCategory category) =>
new() new()
tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests/ZB.MOM.WW.ScadaBridge.HealthMonitoring.Tests.csproj
+2 -1
View File
@@ -28,6 +28,7 @@
<ItemGroup> <ItemGroup>
<ProjectReference Include="../../src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/ZB.MOM.WW.ScadaBridge.HealthMonitoring.csproj" /> <ProjectReference Include="../../src/ZB.MOM.WW.ScadaBridge.HealthMonitoring/ZB.MOM.WW.ScadaBridge.HealthMonitoring.csproj" />
</ItemGroup> <ProjectReference Include="../ZB.MOM.WW.ScadaBridge.TestSupport/ZB.MOM.WW.ScadaBridge.TestSupport.csproj" />
</ItemGroup>
</Project> </Project>
tests/ZB.MOM.WW.ScadaBridge.Host.Tests/CompositionRootTests.cs
-1
View File
@@ -471,7 +471,6 @@ public class SiteCompositionRootTests : IDisposable
new object[] { typeof(IDataConnectionFactory) }, new object[] { typeof(IDataConnectionFactory) },
new object[] { typeof(StoreAndForwardStorage) }, new object[] { typeof(StoreAndForwardStorage) },
new object[] { typeof(StoreAndForwardService) }, new object[] { typeof(StoreAndForwardService) },
new object[] { typeof(ReplicationService) },
new object[] { typeof(ISiteEventLogger) }, new object[] { typeof(ISiteEventLogger) },
new object[] { typeof(IEventLogQueryService) }, new object[] { typeof(IEventLogQueryService) },
new object[] { typeof(ISiteIdentityProvider) }, new object[] { typeof(ISiteIdentityProvider) },
tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbDirectoryTests.cs
+72
View File
@@ -0,0 +1,72 @@
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using ZB.MOM.WW.LocalDb;
namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
/// <summary>
/// Pins that a site node creates the directory holding <c>LocalDb:Path</c> before the
/// database is opened.
/// </summary>
/// <remarks>
/// <para>
/// The guarantee has now had three owners: <c>StoreAndForwardStorage</c> created the
/// directory for its own SQLite file, LocalDb Phase 2 folded that file into the
/// consolidated database and the Host took it over as a <c>SiteLocalDbDirectory</c> shim,
/// and LocalDb 0.1.1 took it into the library itself. This test survived all three moves
/// unchanged in intent, because what a site node needs is the OUTCOME — resolving
/// <c>ILocalDb</c> from the site registration must not fail on a fresh machine — not any
/// particular implementer of it. It is deliberately written against the registration path
/// rather than the mechanism, so a future move costs nothing here.
/// </para>
/// <para>
/// This is not hypothetical: the default site configuration points at the <i>relative</i>
/// path <c>./data/site-localdb.db</c>, so any site node whose working directory has no
/// <c>data/</c> subdirectory hits it. The docker rig escapes only because its volume mount
/// creates <c>/app/data</c> — which is exactly the kind of coincidence that hides a defect
/// until a bare-metal or fresh deployment.
/// </para>
/// </remarks>
public class SiteLocalDbDirectoryTests
{
[Fact]
public void SiteRegistration_CreatesTheLocalDbDirectory_WhenItDoesNotExist()
{
var root = Path.Combine(Path.GetTempPath(), "localdb-dir-test-" + Guid.NewGuid().ToString("N"));
var dbPath = Path.Combine(root, "nested", "site-localdb.db");
Assert.False(Directory.Exists(root));
try
{
var config = new ConfigurationBuilder()
.AddInMemoryCollection(new Dictionary<string, string?>
{
["LocalDb:Path"] = dbPath,
})
.Build();
// The registration path under test. Resolving ILocalDb is what actually opens
// the file, so this fails with SQLite Error 14 if the directory step regresses
// — wherever it lives.
var services = new ServiceCollection();
services.AddZbLocalDb(config);
using var provider = services.BuildServiceProvider();
using var scope = provider.CreateScope();
var db = scope.ServiceProvider.GetRequiredService<ILocalDb>();
Assert.NotNull(db);
Assert.True(Directory.Exists(Path.GetDirectoryName(dbPath)!));
Assert.True(File.Exists(dbPath));
}
finally
{
SqliteConnectionPoolCleanup();
if (Directory.Exists(root))
Directory.Delete(root, recursive: true);
}
}
private static void SqliteConnectionPoolCleanup() =>
Microsoft.Data.Sqlite.SqliteConnection.ClearAllPools();
}
tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbLegacyMigratorTests.cs
+289 -2
View File
@@ -40,7 +40,14 @@ public class SiteLocalDbLegacyMigratorTests : IDisposable
private string Path_(string name) => System.IO.Path.Combine(_root, name); private string Path_(string name) => System.IO.Path.Combine(_root, name);
/// <summary>A consolidated database with both tables created and registered, as the host has it.</summary> /// <summary>A consolidated database with the tables created and registered, as the host has it.</summary>
/// <remarks>
/// <c>sf_messages</c> is registered here even though production <c>OnReady</c> does not
/// register it until the Task 14 cutover. The oplog assertion below needs live capture
/// triggers to mean anything, and the property under test — that the migrator runs after
/// registration — is the same either way. The corollary is a real constraint on Task 14:
/// <c>Migrate</c> must stay the LAST call in <c>OnReady</c>, after every registration.
/// </remarks>
private ILocalDb CreateConsolidated(IConfiguration config) private ILocalDb CreateConsolidated(IConfiguration config)
{ {
var provider = new ServiceCollection() var provider = new ServiceCollection()
@@ -49,8 +56,13 @@ public class SiteLocalDbLegacyMigratorTests : IDisposable
using var connection = db.CreateConnection(); using var connection = db.CreateConnection();
ZB.MOM.WW.ScadaBridge.SiteRuntime.Tracking.OperationTrackingSchema.Apply(connection); ZB.MOM.WW.ScadaBridge.SiteRuntime.Tracking.OperationTrackingSchema.Apply(connection);
ZB.MOM.WW.ScadaBridge.SiteEventLogging.SiteEventLogSchema.Apply(connection); ZB.MOM.WW.ScadaBridge.SiteEventLogging.SiteEventLogSchema.Apply(connection);
ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardSchema.Apply(connection);
ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence.SiteStorageSchema.Apply(connection);
db.RegisterReplicated("OperationTracking"); db.RegisterReplicated("OperationTracking");
db.RegisterReplicated("site_events"); db.RegisterReplicated("site_events");
db.RegisterReplicated("sf_messages");
foreach (var table in SiteLocalDbLegacyMigrator.SiteStorageTables)
db.RegisterReplicated(table.Table);
}) })
.BuildServiceProvider(); .BuildServiceProvider();
_providers.Add(provider); _providers.Add(provider);
@@ -59,12 +71,26 @@ public class SiteLocalDbLegacyMigratorTests : IDisposable
} }
private IConfiguration Config( private IConfiguration Config(
string? trackingPath = null, string? eventsPath = null, string nodeName = "node-a") string? trackingPath = null,
string? eventsPath = null,
string nodeName = "node-a",
string? storeAndForwardPath = null,
string? siteDbPath = null)
{ {
var values = new Dictionary<string, string?> var values = new Dictionary<string, string?>
{ {
["LocalDb:Path"] = Path_("consolidated.db"), ["LocalDb:Path"] = Path_("consolidated.db"),
["ScadaBridge:Node:NodeName"] = nodeName, ["ScadaBridge:Node:NodeName"] = nodeName,
// Always pinned inside the test's own directory, even when a test does not care
// about it. Left unset, the resolver falls back to the CWD-relative code default
// "./data/store-and-forward.db" — and the test run's CWD is the test binary's
// output directory, so an unlucky run could migrate (and RENAME) a real file.
["ScadaBridge:StoreAndForward:SqliteDbPath"] =
storeAndForwardPath ?? Path_("absent-store-and-forward.db"),
// Same reasoning: the site-storage default is "./data/scadabridge.db".
["ScadaBridge:Database:SiteDbPath"] = siteDbPath ?? Path_("absent-scadabridge.db"),
}; };
if (trackingPath is not null) if (trackingPath is not null)
@@ -75,6 +101,28 @@ public class SiteLocalDbLegacyMigratorTests : IDisposable
return new ConfigurationBuilder().AddInMemoryCollection(values).Build(); return new ConfigurationBuilder().AddInMemoryCollection(values).Build();
} }
/// <summary>Seeds a legacy store-and-forward file with the CURRENT column set.</summary>
private static void SeedLegacyStoreAndForward(string path, params string[] ids)
{
using var connection = new SqliteConnection($"Data Source={path}");
connection.Open();
ZB.MOM.WW.ScadaBridge.StoreAndForward.StoreAndForwardSchema.Apply(connection);
foreach (var id in ids)
{
using var cmd = connection.CreateCommand();
cmd.CommandText = """
INSERT INTO sf_messages (
id, category, target, payload_json, retry_count, max_retries,
retry_interval_ms, created_at, status, execution_id)
VALUES ($id, 0, 'ERP', '{"order":1}', 0, 50, 30000, $now, 0, 'exec-1');
""";
cmd.Parameters.AddWithValue("$id", id);
cmd.Parameters.AddWithValue("$now", DateTime.UtcNow.ToString("o"));
cmd.ExecuteNonQuery();
}
}
private static void SeedLegacyTracking(string path, params string[] ids) private static void SeedLegacyTracking(string path, params string[] ids)
{ {
using var connection = new SqliteConnection($"Data Source={path}"); using var connection = new SqliteConnection($"Data Source={path}");
@@ -261,6 +309,245 @@ public class SiteLocalDbLegacyMigratorTests : IDisposable
Assert.Equal(2L, (long)cmd.ExecuteScalar()!); Assert.Equal(2L, (long)cmd.ExecuteScalar()!);
} }
[Fact]
public void SfMessages_AreCopiedFromTheLegacyFile()
{
// Phase 2 (Task 8). Unlike the two Phase 1 files, the store-and-forward default path
// is inside the data volume, so on a real deployment this actually moves data:
// undelivered messages that a lost migration would silently discard.
var sfPath = Path_("store-and-forward.db");
SeedLegacyStoreAndForward(sfPath, "msg-1", "msg-2", "msg-3");
var config = Config(storeAndForwardPath: sfPath);
var db = CreateConsolidated(config);
SiteLocalDbLegacyMigrator.Migrate(db, config);
Assert.Equal(3, CountRows(db, "sf_messages"));
Assert.Equal(["msg-1", "msg-2", "msg-3"], SelectIds(db, "sf_messages", "id"));
// Renamed, not deleted — same contract as the Phase 1 files.
Assert.False(File.Exists(sfPath));
Assert.True(File.Exists(sfPath + ".migrated"));
}
[Fact]
public void SfMessages_Migration_IsIdempotent_WhenRerunAfterACrashBeforeRename()
{
// The crash window: copy committed, rename did not. sf_messages.id is a natural
// TEXT key, so INSERT OR IGNORE absorbs the re-copy with no id synthesis needed.
var sfPath = Path_("store-and-forward.db");
SeedLegacyStoreAndForward(sfPath, "msg-1", "msg-2");
var config = Config(storeAndForwardPath: sfPath);
var db = CreateConsolidated(config);
SiteLocalDbLegacyMigrator.Migrate(db, config);
File.Move(sfPath + ".migrated", sfPath);
SiteLocalDbLegacyMigrator.Migrate(db, config);
Assert.Equal(2, CountRows(db, "sf_messages"));
}
[Fact]
public void MigratedSfMessages_EnterTheOplog_SoTheyActuallyReplicate()
{
// Same ordering trap as the Phase 1 tables: migrate before RegisterReplicated and
// the rows never enter the oplog, never reach the peer, and nothing errors. Only an
// assertion on __localdb_oplog catches the ordering being reversed.
var sfPath = Path_("store-and-forward.db");
SeedLegacyStoreAndForward(sfPath, "msg-1", "msg-2");
var config = Config(storeAndForwardPath: sfPath);
var db = CreateConsolidated(config);
SiteLocalDbLegacyMigrator.Migrate(db, config);
using var connection = db.CreateConnection();
using var cmd = connection.CreateCommand();
cmd.CommandText = "SELECT COUNT(*) FROM __localdb_oplog WHERE table_name = 'sf_messages'";
Assert.Equal(2L, (long)cmd.ExecuteScalar()!);
}
[Fact]
public void SfMessages_FromAnOlderBuild_MigrateDespiteMissingColumns()
{
// A legacy file written before execution_id / parent_execution_id / last_attempt_at_ms
// existed. Naming a missing column in the SELECT throws "no such column", and the
// reader treats that as an unrecognised shape — which would silently discard every
// buffered message. The copy intersects the column sets instead.
var sfPath = Path_("store-and-forward.db");
using (var legacy = new SqliteConnection($"Data Source={sfPath}"))
{
legacy.Open();
using var ddl = legacy.CreateCommand();
ddl.CommandText = """
CREATE TABLE sf_messages (
id TEXT PRIMARY KEY, category INTEGER NOT NULL, target TEXT NOT NULL,
payload_json TEXT NOT NULL, retry_count INTEGER NOT NULL DEFAULT 0,
max_retries INTEGER NOT NULL DEFAULT 50,
retry_interval_ms INTEGER NOT NULL DEFAULT 30000,
created_at TEXT NOT NULL, last_attempt_at TEXT,
status INTEGER NOT NULL DEFAULT 0, last_error TEXT, origin_instance TEXT
);
INSERT INTO sf_messages (id, category, target, payload_json, created_at)
VALUES ('old-1', 0, 'ERP', '{}', '2026-01-01T00:00:00Z');
""";
ddl.ExecuteNonQuery();
}
var config = Config(storeAndForwardPath: sfPath);
var db = CreateConsolidated(config);
SiteLocalDbLegacyMigrator.Migrate(db, config);
Assert.Equal(1, CountRows(db, "sf_messages"));
Assert.Equal(["old-1"], SelectIds(db, "sf_messages", "id"));
}
/// <summary>Seeds a legacy scadabridge.db with the CURRENT config schema and one row per table.</summary>
private static void SeedLegacySiteStorage(string path)
{
using var connection = new SqliteConnection($"Data Source={path}");
connection.Open();
ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence.SiteStorageSchema.Apply(connection);
using var cmd = connection.CreateCommand();
cmd.CommandText = """
INSERT INTO deployed_configurations
(instance_unique_name, config_json, deployment_id, revision_hash, is_enabled, deployed_at)
VALUES ('inst-1', '{"a":1}', 'dep-1', 'hash-1', 1, '2026-01-01T00:00:00Z');
INSERT INTO static_attribute_overrides
(instance_unique_name, attribute_name, override_value, updated_at)
VALUES ('inst-1', 'Setpoint', '42', '2026-01-01T00:00:00Z');
INSERT INTO shared_scripts (name, code, updated_at)
VALUES ('helper', 'return 1;', '2026-01-01T00:00:00Z');
INSERT INTO external_systems (name, endpoint_url, auth_type, updated_at, timeout_seconds)
VALUES ('ERP', 'http://erp:5200', 'None', '2026-01-01T00:00:00Z', 30);
INSERT INTO database_connections (name, connection_string, updated_at)
VALUES ('BT', 'Server=sql;Database=BT', '2026-01-01T00:00:00Z');
INSERT INTO data_connection_definitions (name, protocol, updated_at)
VALUES ('opc-1', 'OpcUa', '2026-01-01T00:00:00Z');
INSERT INTO native_alarm_state
(instance_unique_name, source_canonical_name, source_reference, condition_json, last_transition_at)
VALUES ('inst-1', 'Area.Line', 'ref-1', '{"active":true}', '2026-01-01T00:00:00Z');
-- The two that must NOT be migrated. A pre-2026-07-10 file really can hold these.
INSERT INTO notification_lists (name, recipient_emails, updated_at)
VALUES ('ops', 'ops@example.com', '2026-01-01T00:00:00Z');
INSERT INTO smtp_configurations
(name, server, port, auth_mode, from_address, username, password, updated_at)
VALUES ('smtp', 'smtp.example.com', 587, 'Basic', 'a@b.c', 'user',
'PLAINTEXT-SECRET', '2026-01-01T00:00:00Z');
""";
cmd.ExecuteNonQuery();
}
[Fact]
public void SiteConfigTables_AreCopiedFromTheLegacyFile()
{
// Phase 2 (Task 9). All seven migrated tables live in one legacy file and are copied
// in a single transaction: a partial config migration would leave a site node running
// against half its old configuration, which is worse than failing startup.
var sitePath = Path_("scadabridge.db");
SeedLegacySiteStorage(sitePath);
var config = Config(siteDbPath: sitePath);
var db = CreateConsolidated(config);
SiteLocalDbLegacyMigrator.Migrate(db, config);
foreach (var table in SiteLocalDbLegacyMigrator.SiteStorageTables)
Assert.Equal(1, CountRows(db, table.Table));
Assert.False(File.Exists(sitePath));
Assert.True(File.Exists(sitePath + ".migrated"));
}
[Fact]
public void Migration_DoesNotCopyNotificationOrSmtpRows_EvenWhenTheLegacyFileHasThem()
{
// smtp_configurations.password is PLAINTEXT. Both tables are purged on every deploy
// and permanently empty by design since the site write paths were removed
// (2026-07-10), but a pre-fix legacy file can still hold rows. The cutover also
// declines to REGISTER these two for replication, for the same reason — so migrating
// them would leave plaintext SMTP passwords in the consolidated database, one future
// RegisterReplicated away from being shipped to a peer, for config nothing reads.
var sitePath = Path_("scadabridge.db");
SeedLegacySiteStorage(sitePath);
var config = Config(siteDbPath: sitePath);
var db = CreateConsolidated(config);
SiteLocalDbLegacyMigrator.Migrate(db, config);
Assert.Equal(0, CountRows(db, "notification_lists"));
Assert.Equal(0, CountRows(db, "smtp_configurations"));
// Belt and braces: the secret must not be anywhere in the consolidated file,
// including the oplog, whose row_json is a json_object copy of every captured row.
using var connection = db.CreateConnection();
using var cmd = connection.CreateCommand();
cmd.CommandText = "SELECT COUNT(*) FROM __localdb_oplog WHERE row_json LIKE '%PLAINTEXT-SECRET%'";
Assert.Equal(0L, (long)cmd.ExecuteScalar()!);
}
[Fact]
public void MigratedConfigRows_EnterTheOplog_SoTheyActuallyReplicate()
{
var sitePath = Path_("scadabridge.db");
SeedLegacySiteStorage(sitePath);
var config = Config(siteDbPath: sitePath);
var db = CreateConsolidated(config);
SiteLocalDbLegacyMigrator.Migrate(db, config);
using var connection = db.CreateConnection();
using var cmd = connection.CreateCommand();
cmd.CommandText =
"SELECT COUNT(*) FROM __localdb_oplog WHERE table_name = 'deployed_configurations'";
Assert.Equal(1L, (long)cmd.ExecuteScalar()!);
}
[Fact]
public void SiteConfigMigration_IsIdempotent_WhenRerunAfterACrashBeforeRename()
{
var sitePath = Path_("scadabridge.db");
SeedLegacySiteStorage(sitePath);
var config = Config(siteDbPath: sitePath);
var db = CreateConsolidated(config);
SiteLocalDbLegacyMigrator.Migrate(db, config);
File.Move(sitePath + ".migrated", sitePath);
SiteLocalDbLegacyMigrator.Migrate(db, config);
foreach (var table in SiteLocalDbLegacyMigrator.SiteStorageTables)
Assert.Equal(1, CountRows(db, table.Table));
}
[Fact]
public void MigratorColumnLists_MatchTheLiveSchema()
{
// A column named here but absent from the schema fails at runtime on a real
// deployment and NOWHERE else — the intersection logic hides a typo (the column is
// simply dropped from the copy) and every other test still passes. A column in the
// schema but missing here is worse: that data is silently left behind.
//
// So this asserts set equality against the schema the host actually applies.
using var connection = new SqliteConnection($"Data Source={Path_("schema-probe.db")}");
connection.Open();
ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence.SiteStorageSchema.Apply(connection);
foreach (var table in SiteLocalDbLegacyMigrator.SiteStorageTables)
{
using var probe = connection.CreateCommand();
probe.CommandText = $"SELECT name FROM pragma_table_info('{table.Table}')";
using var reader = probe.ExecuteReader();
var actual = new List<string>();
while (reader.Read()) actual.Add(reader.GetString(0));
Assert.Equal(
actual.OrderBy(c => c, StringComparer.Ordinal).ToArray(),
table.Columns.OrderBy(c => c, StringComparer.Ordinal).ToArray());
}
}
[Fact] [Fact]
public void LegacyFileWithoutTheExpectedTable_IsTreatedAsEmpty_NotAFailure() public void LegacyFileWithoutTheExpectedTable_IsTreatedAsEmpty_NotAFailure()
{ {
tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbWiringTests.cs
+99
View File
@@ -215,6 +215,105 @@ public class SiteLocalDbWiringTests : IDisposable
Assert.Contains(LocalDbMetrics.MeterName, SiteServiceRegistration.ObservedMeters); Assert.Contains(LocalDbMetrics.MeterName, SiteServiceRegistration.ObservedMeters);
} }
[Fact]
public void Site_LocalDb_CreatesThePhase2Tables_InTheConsolidatedFile()
{
// Phase 2 (Task 7). The nine configuration tables and the store-and-forward buffer
// now live in the consolidated file rather than in scadabridge.db and
// store-and-forward.db. Asserting through the REAL composition root is what proves
// OnReady applies both schemas — the stores themselves also call Apply, so a test
// that went through a store would pass even if OnReady never touched them.
var db = _host.Services.GetRequiredService<ILocalDb>();
var tables = TableNames(db);
// One from each Phase 1 schema, then every Phase 2 table.
Assert.Contains("OperationTracking", tables);
Assert.Contains("site_events", tables);
Assert.Contains("sf_messages", tables);
foreach (var table in new[]
{
"deployed_configurations", "static_attribute_overrides", "shared_scripts",
"external_systems", "database_connections", "notification_lists",
"data_connection_definitions", "smtp_configurations", "native_alarm_state",
})
{
Assert.Contains(table, tables);
}
}
[Fact]
public void Site_LocalDb_RegistersExactlyTheTenReplicatedTables()
{
// The Phase 2 cutover. This is an EXACTLY check rather than a Contains in both
// directions, and both directions are load-bearing.
//
// Too few: the table silently stops replicating. Storage still works, every other
// test still passes, and the pair just quietly diverges — the failure mode Phase 1
// existed to end.
//
// Too many: notification_lists and smtp_configurations must NOT be here. They are
// permanently empty by design, and registering them would open a standing
// replication channel whose only historical payload was plaintext SMTP passwords.
// A Contains-based test would never catch that.
var db = _host.Services.GetRequiredService<ILocalDb>();
Assert.Equal(
[
// Ordinal order: '_' (0x5F) sorts before 'b' (0x62), so
// data_connection_definitions precedes database_connections.
"OperationTracking", "data_connection_definitions", "database_connections",
"deployed_configurations", "external_systems", "native_alarm_state",
"sf_messages", "shared_scripts", "site_events", "static_attribute_overrides",
],
db.ReplicatedTables.Keys.OrderBy(k => k, StringComparer.Ordinal).ToArray());
}
[Fact]
public void Site_LocalDb_DoesNotReplicateTheCentralOnlyNotificationTables()
{
// Stated separately from the exact-set test above because this one is a security
// property, not a wiring property, and deserves to fail with its own name. The
// tables exist (SiteStorageSchema creates them) — they simply must never be
// captured. See SiteLocalDbSetup.OnReady for the rationale.
var db = _host.Services.GetRequiredService<ILocalDb>();
Assert.DoesNotContain("notification_lists", db.ReplicatedTables.Keys);
Assert.DoesNotContain("smtp_configurations", db.ReplicatedTables.Keys);
}
[Fact]
public void Site_LocalDb_ReplicatedPhase2TablesHaveTheirExpectedPrimaryKeys()
{
// RegisterReplicated refuses a table with no explicit PK, so reaching these
// assertions proves the DDL ran before registration. The composite keys are the
// interesting ones: LWW conflict resolution keys on the FULL PK, so a wrong or
// truncated key set would silently collapse distinct rows into one.
var db = _host.Services.GetRequiredService<ILocalDb>();
Assert.Equal(["id"], db.ReplicatedTables["sf_messages"].PkColumns);
Assert.Equal(
["instance_unique_name"], db.ReplicatedTables["deployed_configurations"].PkColumns);
Assert.Equal(
["instance_unique_name", "attribute_name"],
db.ReplicatedTables["static_attribute_overrides"].PkColumns);
Assert.Equal(
["instance_unique_name", "source_canonical_name", "source_reference"],
db.ReplicatedTables["native_alarm_state"].PkColumns);
}
private static HashSet<string> TableNames(ILocalDb db)
{
using var connection = db.CreateConnection();
using var cmd = connection.CreateCommand();
cmd.CommandText = "SELECT name FROM sqlite_master WHERE type = 'table'";
using var reader = cmd.ExecuteReader();
var names = new HashSet<string>(StringComparer.Ordinal);
while (reader.Read()) names.Add(reader.GetString(0));
return names;
}
[Fact] [Fact]
public void Site_LocalDb_CreatesTheConfiguredFile() public void Site_LocalDb_CreatesTheConfiguredFile()
{ {
tests/ZB.MOM.WW.ScadaBridge.Host.Tests/StartupValidatorTests.cs
+10 -3
View File
@@ -232,15 +232,22 @@ public class StartupValidatorTests
Assert.Null(ex); Assert.Null(ex);
} }
/// <summary>
/// The inverse of the rule this replaces. SiteDbPath was mandatory for Site
/// nodes until LocalDb Phase 2 moved the site tables into the consolidated
/// LocalDb database; it now names only the legacy file the boot-time migrator
/// drains, so its absence means "nothing to migrate". Requiring it would make
/// every already-migrated node carry a dead key forever.
/// </summary>
[Fact] [Fact]
public void Site_MissingSiteDbPath_FailsValidation() public void Site_MissingSiteDbPath_IsAccepted_BecauseThePathIsMigrationOnly()
{ {
var values = ValidSiteConfig(); var values = ValidSiteConfig();
values.Remove("ScadaBridge:Database:SiteDbPath"); values.Remove("ScadaBridge:Database:SiteDbPath");
var config = BuildConfig(values); var config = BuildConfig(values);
var ex = Assert.Throws<InvalidOperationException>(() => StartupValidator.Validate(config)); var ex = Record.Exception(() => StartupValidator.Validate(config));
Assert.Contains("SiteDbPath required for Site nodes", ex.Message); Assert.Null(ex);
} }
[Fact] [Fact]
tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/SfBufferResyncPredicateTests.cs
-94
View File
@@ -1,94 +0,0 @@
using Akka.Actor;
using Microsoft.Extensions.Logging.Abstractions;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
using ZB.MOM.WW.ScadaBridge.StoreAndForward;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
namespace ZB.MOM.WW.ScadaBridge.IntegrationTests.Cluster;
/// <summary>
/// N1 regression (review 02 round 2, Critical): the resync authority must use the same
/// oldest-Up predicate as the S&F delivery gate. Divergence scenario = the delivering node
/// is OLDEST but not LEADER (leader = lowest address), the exact state a rolling restart of
/// the lower-address node produces. Pre-fix the delivering node requests a resync from the
/// stale peer and ReplaceAllAsync wipes its live buffer.
/// </summary>
public class SfBufferResyncPredicateTests
{
[Fact]
public async Task OldestButNotLeaderNode_KeepsItsBuffer_AndSeedsTheJoiner()
{
// Two explicit ports, deliberately assigned so the FIRST-started (oldest,
// delivering) node has the HIGHER address → the second node is cluster leader.
var p1 = TwoNodeClusterFixture.GetFreeTcpPort();
var p2 = TwoNodeClusterFixture.GetFreeTcpPort();
var (portHigh, portLow) = p1 > p2 ? (p1, p2) : (p2, p1);
await using var fixture = await TwoNodeClusterFixture.StartAsync(
role: "site-int", portA: portHigh, portB: portLow);
// Real S&F storage + replication actor per node, production default predicate
// (no isActiveOverride) — the exact wiring under test.
var (storageOldest, _) = await CreateReplicationActorAsync(fixture.NodeA, "oldest");
var (storageJoiner, _) = await CreateReplicationActorAsync(fixture.NodeB, "joiner");
// The delivering (oldest) node has a live buffered row the standby never saw.
await storageOldest.EnqueueAsync(NewMessage("live-row"));
// Trigger peer (re)tracking on both sides: each actor got InitialStateAsSnapshot
// in PreStart, but the enqueue raced it — re-deliver via a fresh MemberUp is not
// needed; OnPeerTracked already fired on join. The resync exchange is async:
// wait until the JOINER holds the row (proves the snapshot flowed oldest→joiner,
// the correct direction). Pre-fix this times out (the joiner, as leader, never
// requests) AND the oldest node's row is deleted by the stale wipe.
await AwaitAsync(async () => await storageJoiner.GetMessageByIdAsync("live-row") != null,
TimeSpan.FromSeconds(20),
"joiner never received the resync snapshot (resync ran in the wrong direction)");
// And the delivering node's buffer is untouched — the N1 wipe assertion.
Assert.NotNull(await storageOldest.GetMessageByIdAsync("live-row"));
}
private static async Task<(StoreAndForwardStorage Storage, IActorRef Actor)> CreateReplicationActorAsync(
ActorSystem node, string tag)
{
var sfDb = Path.Combine(Path.GetTempPath(), $"sf-resync-{tag}-{Guid.NewGuid():N}.db");
var siteDb = Path.Combine(Path.GetTempPath(), $"site-resync-{tag}-{Guid.NewGuid():N}.db");
var sfStorage = new StoreAndForwardStorage($"Data Source={sfDb}",
NullLogger<StoreAndForwardStorage>.Instance);
await sfStorage.InitializeAsync();
var siteStorage = new SiteStorageService($"Data Source={siteDb}",
NullLogger<SiteStorageService>.Instance);
var replicationService = new ReplicationService(
new StoreAndForwardOptions(), NullLogger<ReplicationService>.Instance);
// Name MUST be "site-replication" — SendToPeer targets /user/site-replication.
var actor = node.ActorOf(Props.Create(() => new SiteReplicationActor(
siteStorage, sfStorage, replicationService, "site-int",
NullLogger<SiteReplicationActor>.Instance, null, null, null, null)),
"site-replication");
return (sfStorage, actor);
}
private static StoreAndForwardMessage NewMessage(string id) => new()
{
Id = id,
Category = StoreAndForwardCategory.Notification,
Target = "central",
PayloadJson = "{}",
CreatedAt = DateTimeOffset.UtcNow,
Status = StoreAndForwardMessageStatus.Pending,
MaxRetries = 0,
};
private static async Task AwaitAsync(Func<Task<bool>> condition, TimeSpan timeout, string why)
{
var deadline = DateTime.UtcNow + timeout;
while (DateTime.UtcNow < deadline)
{
if (await condition()) return;
await Task.Delay(250);
}
throw new TimeoutException(why);
}
}
tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/DualNodeRecoveryTests.cs
+36 -15
View File
@@ -1,6 +1,7 @@
using Microsoft.Extensions.Logging.Abstractions; using Microsoft.Extensions.Logging.Abstractions;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
using ZB.MOM.WW.ScadaBridge.StoreAndForward; using ZB.MOM.WW.ScadaBridge.StoreAndForward;
using ZB.MOM.WW.ScadaBridge.TestSupport;
namespace ZB.MOM.WW.ScadaBridge.IntegrationTests; namespace ZB.MOM.WW.ScadaBridge.IntegrationTests;
@@ -18,12 +19,17 @@ public class DualNodeRecoveryTests
// Scenario: both site nodes crash. First node to restart opens the existing // Scenario: both site nodes crash. First node to restart opens the existing
// SQLite database and finds all buffered S&F messages intact. // SQLite database and finds all buffered S&F messages intact.
var dbPath = Path.Combine(Path.GetTempPath(), $"sf_dual_{Guid.NewGuid():N}.db"); var dbPath = Path.Combine(Path.GetTempPath(), $"sf_dual_{Guid.NewGuid():N}.db");
var connStr = $"Data Source={dbPath}";
// The store takes an ILocalDb; the "restarted node" opens its own local database
// over the SAME file, which is how this test models recovery from disk.
TestLocalDb? crashedDb = null;
TestLocalDb? recoveryDb = null;
try try
{ {
// Setup: populate SQLite with messages (simulating pre-crash state) // Setup: populate SQLite with messages (simulating pre-crash state)
var storage = new StoreAndForwardStorage(connStr, NullLogger<StoreAndForwardStorage>.Instance); crashedDb = TestLocalDb.Create(dbPath);
var storage = new StoreAndForwardStorage(crashedDb.Db, NullLogger<StoreAndForwardStorage>.Instance);
await storage.InitializeAsync(); await storage.InitializeAsync();
var messageIds = new List<string>(); var messageIds = new List<string>();
@@ -48,7 +54,8 @@ public class DualNodeRecoveryTests
// Both nodes down — simulate by creating a fresh storage instance // Both nodes down — simulate by creating a fresh storage instance
// (new process connecting to same SQLite file) // (new process connecting to same SQLite file)
var recoveryStorage = new StoreAndForwardStorage(connStr, NullLogger<StoreAndForwardStorage>.Instance); recoveryDb = TestLocalDb.Create(dbPath);
var recoveryStorage = new StoreAndForwardStorage(recoveryDb.Db, NullLogger<StoreAndForwardStorage>.Instance);
await recoveryStorage.InitializeAsync(); await recoveryStorage.InitializeAsync();
// Verify all messages are available for retry // Verify all messages are available for retry
@@ -69,8 +76,10 @@ public class DualNodeRecoveryTests
} }
finally finally
{ {
if (File.Exists(dbPath)) // Dispose before deleting — the master connections anchor the WAL sidecars.
File.Delete(dbPath); recoveryDb?.Dispose();
crashedDb?.Dispose();
TestLocalDb.DeleteFiles(dbPath);
} }
} }
@@ -79,11 +88,14 @@ public class DualNodeRecoveryTests
public async Task SiteTopology_DualCrash_ParkedMessagesPreserved() public async Task SiteTopology_DualCrash_ParkedMessagesPreserved()
{ {
var dbPath = Path.Combine(Path.GetTempPath(), $"sf_dual_parked_{Guid.NewGuid():N}.db"); var dbPath = Path.Combine(Path.GetTempPath(), $"sf_dual_parked_{Guid.NewGuid():N}.db");
var connStr = $"Data Source={dbPath}";
TestLocalDb? crashedDb = null;
TestLocalDb? recoveryDb = null;
try try
{ {
var storage = new StoreAndForwardStorage(connStr, NullLogger<StoreAndForwardStorage>.Instance); crashedDb = TestLocalDb.Create(dbPath);
var storage = new StoreAndForwardStorage(crashedDb.Db, NullLogger<StoreAndForwardStorage>.Instance);
await storage.InitializeAsync(); await storage.InitializeAsync();
// Mix of pending and parked messages // Mix of pending and parked messages
@@ -114,7 +126,8 @@ public class DualNodeRecoveryTests
}); });
// Dual crash recovery // Dual crash recovery
var recoveryStorage = new StoreAndForwardStorage(connStr, NullLogger<StoreAndForwardStorage>.Instance); recoveryDb = TestLocalDb.Create(dbPath);
var recoveryStorage = new StoreAndForwardStorage(recoveryDb.Db, NullLogger<StoreAndForwardStorage>.Instance);
await recoveryStorage.InitializeAsync(); await recoveryStorage.InitializeAsync();
var pendingCount = await recoveryStorage.GetMessageCountByStatusAsync(StoreAndForwardMessageStatus.Pending); var pendingCount = await recoveryStorage.GetMessageCountByStatusAsync(StoreAndForwardMessageStatus.Pending);
@@ -134,8 +147,10 @@ public class DualNodeRecoveryTests
} }
finally finally
{ {
if (File.Exists(dbPath)) // Dispose before deleting — the master connections anchor the WAL sidecars.
File.Delete(dbPath); recoveryDb?.Dispose();
crashedDb?.Dispose();
TestLocalDb.DeleteFiles(dbPath);
} }
} }
@@ -176,11 +191,14 @@ public class DualNodeRecoveryTests
{ {
// CREATE TABLE IF NOT EXISTS is idempotent — safe to call on recovery // CREATE TABLE IF NOT EXISTS is idempotent — safe to call on recovery
var dbPath = Path.Combine(Path.GetTempPath(), $"sf_idempotent_{Guid.NewGuid():N}.db"); var dbPath = Path.Combine(Path.GetTempPath(), $"sf_idempotent_{Guid.NewGuid():N}.db");
var connStr = $"Data Source={dbPath}";
TestLocalDb? localDb1 = null;
TestLocalDb? localDb2 = null;
try try
{ {
var storage1 = new StoreAndForwardStorage(connStr, NullLogger<StoreAndForwardStorage>.Instance); localDb1 = TestLocalDb.Create(dbPath);
var storage1 = new StoreAndForwardStorage(localDb1.Db, NullLogger<StoreAndForwardStorage>.Instance);
await storage1.InitializeAsync(); await storage1.InitializeAsync();
await storage1.EnqueueAsync(new StoreAndForwardMessage await storage1.EnqueueAsync(new StoreAndForwardMessage
@@ -196,7 +214,8 @@ public class DualNodeRecoveryTests
}); });
// Second InitializeAsync on same DB should be safe (no data loss) // Second InitializeAsync on same DB should be safe (no data loss)
var storage2 = new StoreAndForwardStorage(connStr, NullLogger<StoreAndForwardStorage>.Instance); localDb2 = TestLocalDb.Create(dbPath);
var storage2 = new StoreAndForwardStorage(localDb2.Db, NullLogger<StoreAndForwardStorage>.Instance);
await storage2.InitializeAsync(); await storage2.InitializeAsync();
var msg = await storage2.GetMessageByIdAsync("test-1"); var msg = await storage2.GetMessageByIdAsync("test-1");
@@ -205,8 +224,10 @@ public class DualNodeRecoveryTests
} }
finally finally
{ {
if (File.Exists(dbPath)) // Dispose before deleting — the master connections anchor the WAL sidecars.
File.Delete(dbPath); localDb2?.Dispose();
localDb1?.Dispose();
TestLocalDb.DeleteFiles(dbPath);
} }
} }
} }
tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/IntegrationSurfaceTests.cs
+5 -5
View File
@@ -91,12 +91,12 @@ public class IntegrationSurfaceTests
{ {
// Notification Outbox: Notify.Send enqueues into the site Store-and-Forward // Notification Outbox: Notify.Send enqueues into the site Store-and-Forward
// Engine and returns the NotificationId handle immediately. // Engine and returns the NotificationId handle immediately.
var dbName = $"NotifyWired_{Guid.NewGuid():N}"; // A real temp-file LocalDb, not the shared-cache in-memory database this used
var connStr = $"Data Source={dbName};Mode=Memory;Cache=Shared"; // before: StoreAndForwardStorage takes ILocalDb now, and LocalDb has no
using var keepAlive = new Microsoft.Data.Sqlite.SqliteConnection(connStr); // in-memory mode.
keepAlive.Open(); using var localDb = ZB.MOM.WW.ScadaBridge.TestSupport.TestLocalDb.CreateTemp("NotifyWired");
var storage = new StoreAndForward.StoreAndForwardStorage( var storage = new StoreAndForward.StoreAndForwardStorage(
connStr, Microsoft.Extensions.Logging.Abstractions.NullLogger<StoreAndForward.StoreAndForwardStorage>.Instance); localDb.Db, Microsoft.Extensions.Logging.Abstractions.NullLogger<StoreAndForward.StoreAndForwardStorage>.Instance);
await storage.InitializeAsync(); await storage.InitializeAsync();
var saf = new StoreAndForward.StoreAndForwardService( var saf = new StoreAndForward.StoreAndForwardService(
storage, new StoreAndForward.StoreAndForwardOptions(), storage, new StoreAndForward.StoreAndForwardOptions(),
tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbConfigConvergenceTests.cs
+177
View File
@@ -0,0 +1,177 @@
using ZB.MOM.WW.LocalDb;
namespace ZB.MOM.WW.ScadaBridge.IntegrationTests;
/// <summary>
/// Phase 2 convergence: deployed configuration across a real site pair, plus the resync
/// behaviour that the bespoke replicator needed a directional guard for.
/// </summary>
/// <remarks>
/// Companion to <see cref="LocalDbStoreAndForwardConvergenceTests"/>: same
/// specifications-before-deletion role, for the intents held by
/// <c>SiteReplicationActorTests</c> and <c>SfBufferResyncPredicateTests</c> rather than by
/// the store-and-forward <c>ReplicationService</c>.
/// </remarks>
[Collection("LocalDbSitePairConvergence")]
public sealed class LocalDbConfigConvergenceTests : LocalDbSitePairHarness
{
// ---- data helpers -----------------------------------------------------------------
private static Task DeployConfigAsync(ILocalDb db, string instance, string configJson, string deploymentId)
=> db.ExecuteAsync(
"""
INSERT INTO deployed_configurations (
instance_unique_name, config_json, deployment_id, revision_hash,
is_enabled, deployed_at)
VALUES (@instance, @configJson, @deploymentId, @deploymentId, 1, @now)
ON CONFLICT(instance_unique_name) DO UPDATE SET
config_json = excluded.config_json,
deployment_id = excluded.deployment_id,
revision_hash = excluded.revision_hash,
deployed_at = excluded.deployed_at;
""",
new { instance, configJson, deploymentId, now = DateTime.UtcNow.ToString("o") });
private static async Task<string?> ReadConfigAsync(ILocalDb db, string instance)
{
var rows = await db.QueryAsync(
"SELECT config_json FROM deployed_configurations WHERE instance_unique_name = @instance",
static r => r.GetString(0), new { instance });
return rows.Count == 0 ? null : rows[0];
}
private static Task EnqueueAsync(ILocalDb db, string id)
=> db.ExecuteAsync(
"""
INSERT INTO sf_messages (
id, category, target, payload_json, retry_count, max_retries,
retry_interval_ms, created_at, status)
VALUES (@id, 0, 'central', '{}', 0, 50, 30000, @now, 0);
""",
new { id, now = DateTime.UtcNow.ToString("o") });
private static async Task<bool> MessageExistsAsync(ILocalDb db, string id)
{
var rows = await db.QueryAsync(
"SELECT 1 FROM sf_messages WHERE id = @id", static r => r.GetInt32(0), new { id });
return rows.Count > 0;
}
// ---- scenarios --------------------------------------------------------------------
[Fact]
public async Task ANodeWithNewerLocalRows_KeepsThem_WhenAPeerSnapshotArrives()
{
// N1 Critical (was: SfBufferResyncPredicateTests). That test existed because the
// bespoke replicator's ReplaceAllAsync was a destructive DELETE-then-INSERT: a resync
// running in the wrong direction WIPED a live buffer, so the code needed a
// same-oldest-Up predicate to decide who had authority. The divergence case was a
// rolling restart leaving the delivering node oldest but not leader.
//
// Under LocalDb that failure mode is structurally impossible rather than guarded
// against: snapshot resync merges per row under LWW and never deletes. So this is
// deliberately NOT a directional-authority test — there is no active/standby
// asymmetry left to enforce. It asserts the property the guard was protecting: no
// node loses rows to a peer's snapshot.
//
// The setup is the wipe scenario made concrete. Each node holds a row the other has
// never seen, and they disagree about a third; then they resync.
await DeployConfigAsync(A, "shared-instance", """{"v":1}""", "dep-1");
await WaitUntilAsync(
async () => await ReadConfigAsync(B, "shared-instance") is not null,
"the shared row to exist on both nodes before they diverge");
await StopPassiveAsync();
// A keeps working while B is down: a brand-new buffered message and a newer version
// of the shared config.
await EnqueueAsync(A, "live-row-on-a");
await DeployConfigAsync(A, "shared-instance", """{"v":2}""", "dep-2");
// B is offline but its database is still writable — it also has local state the
// snapshot exchange must not destroy.
await EnqueueAsync(B, "live-row-on-b");
await RestartPairAsync();
await WaitUntilAsync(
async () =>
await MessageExistsAsync(A, "live-row-on-a") &&
await MessageExistsAsync(A, "live-row-on-b") &&
await MessageExistsAsync(B, "live-row-on-a") &&
await MessageExistsAsync(B, "live-row-on-b"),
"both nodes to hold the UNION of the rows each wrote while partitioned");
// And the contended row converges to the newer write rather than either node's
// snapshot flattening the other.
await WaitUntilAsync(
async () => await ReadConfigAsync(A, "shared-instance") == """{"v":2}"""
&& await ReadConfigAsync(B, "shared-instance") == """{"v":2}""",
"both nodes to converge on the newer config for the contended instance");
}
[Fact]
public async Task ConfigDeployedOnA_ConvergesToB_WithoutAnyFetch()
{
// Was: SiteReplicationActorTests #1-#5 (notify-and-fetch — the standby is told a
// deploy happened, then HTTP-fetches the config itself, with retries and a
// superseded-404 path). Under CDC the config row simply replicates, so that whole
// exchange is deleted at Task 15.
//
// The plan called for asserting a fetcher test double records zero invocations. That
// would be THEATRE here and is deliberately not done: this harness has no actor
// system, no central, and no IDeploymentConfigFetcher in the graph at all, so a
// double would record zero calls whether or not the fetch path still existed. An
// assertion that cannot fail is worse than none.
//
// What this test honestly proves is the positive half: the config reaches node B
// through replication ALONE, in a process where fetching is not merely unused but
// absent. The negative half — that no fetch path survives — is proved by Task 15
// deleting the code and the build still passing, which is a stronger check than any
// runtime counter.
//
// Scope note (D1): SiteReconciliationActor survives Phase 2 and legitimately fetches
// over HTTP at node startup when central reports gaps. "The standby never fetches,
// ever" would be a FALSE claim; the claim is about the deploy path only.
await DeployConfigAsync(A, "inst-deploy", """{"setpoint":42}""", "dep-100");
await WaitUntilAsync(
async () => await ReadConfigAsync(B, "inst-deploy") == """{"setpoint":42}""",
"the deployed config to reach node B over replication alone");
}
[Fact]
public async Task RedeployingAnInstance_ConvergesToTheNewRevision_OnBothNodes()
{
// The steady-state deploy loop, and the reason the old path needed a deployed_at
// guard: a standby that applied a STALE fetch would pin an instance to superseded
// config until the next deploy. Under LWW the newer write wins on the primary key,
// so the guard is not what protects this — but the outcome still has to hold.
await DeployConfigAsync(A, "inst-redeploy", """{"rev":"a"}""", "dep-a");
await WaitUntilAsync(
async () => await ReadConfigAsync(B, "inst-redeploy") == """{"rev":"a"}""",
"the first revision to reach node B");
await DeployConfigAsync(A, "inst-redeploy", """{"rev":"b"}""", "dep-b");
await WaitUntilAsync(
async () => await ReadConfigAsync(A, "inst-redeploy") == """{"rev":"b"}"""
&& await ReadConfigAsync(B, "inst-redeploy") == """{"rev":"b"}""",
"both nodes to converge on the redeployed revision");
}
[Fact]
public async Task ConfigDeployedOnB_ConvergesToA()
{
// Deliberately the reverse direction. The bespoke replicator was asymmetric by
// design — the active node pushed, the standby fetched — so "which node deployed it"
// was a meaningful question. Under CDC it is not, and this test is what says so.
// After a failover the surviving node must be able to deploy without waiting to be
// promoted to some authority role.
await DeployConfigAsync(B, "inst-from-b", """{"origin":"b"}""", "dep-b1");
await WaitUntilAsync(
async () => await ReadConfigAsync(A, "inst-from-b") == """{"origin":"b"}""",
"a config deployed on node B to reach node A");
}
}
tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbPhase2ConvergenceTests.cs
+193
View File
@@ -0,0 +1,193 @@
using Microsoft.Extensions.Logging.Abstractions;
using ZB.MOM.WW.LocalDb;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
namespace ZB.MOM.WW.ScadaBridge.IntegrationTests;
/// <summary>
/// Phase 2 convergence driven through the REAL <see cref="SiteStorageService"/>, over a real
/// site pair, rather than through hand-written SQL.
/// </summary>
/// <remarks>
/// <para>
/// The sibling suites (<see cref="LocalDbConfigConvergenceTests"/>,
/// <see cref="LocalDbStoreAndForwardConvergenceTests"/>) were written as specifications
/// BEFORE the cutover, so they necessarily use hand-written SQL — the production writers were
/// still going through the bespoke replicator at the time. This suite runs after the cutover
/// and therefore drives the production writer, which is what makes the cascade scenario below
/// meaningful: it is the shipped multi-statement transaction under test, not a re-creation
/// of it.
/// </para>
/// <para>
/// <see cref="SiteStorageService"/> is constructed directly on the fixture's
/// <see cref="ILocalDb"/> — the same instance the host registers — so its writes flow through
/// the same CDC triggers.
/// </para>
/// </remarks>
[Collection("LocalDbSitePairConvergence")]
public sealed class LocalDbPhase2ConvergenceTests : LocalDbSitePairHarness
{
private SiteStorageService Storage(ILocalDb db) =>
new(db, NullLogger<SiteStorageService>.Instance);
// ---- read helpers -----------------------------------------------------------------
private static Task<long> CountAsync(ILocalDb db, string table, string instance) =>
ScalarAsync(db, $"SELECT COUNT(*) FROM {table} WHERE instance_unique_name = @instance",
new { instance });
private static async Task<long> ScalarAsync(ILocalDb db, string sql, object? args = null)
{
var rows = await db.QueryAsync(sql, static r => r.GetInt64(0), args);
return rows[0];
}
/// <summary>
/// The whole <c>deployed_configurations</c> row as one comparable string, so a scenario
/// can assert every column replicated rather than just the payload it happened to check.
/// </summary>
private static async Task<string?> ReadWholeConfigRowAsync(ILocalDb db, string instance)
{
var rows = await db.QueryAsync(
"""
SELECT instance_unique_name || '|' || config_json || '|' || deployment_id || '|' ||
revision_hash || '|' || is_enabled || '|' || deployed_at
FROM deployed_configurations WHERE instance_unique_name = @instance
""",
static r => r.GetString(0), new { instance });
return rows.Count == 0 ? null : rows[0];
}
/// <summary>Unacked oplog backlog. Drains to zero once the peer acknowledges.</summary>
private static Task<long> OplogDepthAsync(ILocalDb db) =>
ScalarAsync(db,
"SELECT COUNT(*) FROM __localdb_oplog WHERE seq > " +
"(SELECT last_acked_seq FROM __localdb_peer_state WHERE id = 1)");
// ---- scenarios --------------------------------------------------------------------
[Fact]
public async Task DeployedConfigRow_ConvergesToB_ColumnForColumn()
{
// LocalDbConfigConvergenceTests asserts config_json converges. That is the payload
// only: a capture that dropped, defaulted or reordered any other column would still
// pass it. deployed_at is the one that matters most — SiteReconciliationActor's
// guarded write compares it, so a node whose replicated copy carried a different
// timestamp would make different staleness decisions from its peer.
await Storage(A).StoreDeployedConfigAsync(
"inst-columns", """{"setpoint":7}""", "dep-col-1", "hash-col-1", isEnabled: true);
var onA = await ReadWholeConfigRowAsync(A, "inst-columns");
Assert.NotNull(onA);
await WaitUntilAsync(
async () => await ReadWholeConfigRowAsync(B, "inst-columns") == onA,
"node B's deployed_configurations row to match node A's in EVERY column");
}
[Fact]
public async Task RemovingAnInstance_ConvergesAllThreeCascadeTables()
{
// The plan flagged this as the most likely real defect in Phase 2, and the reasoning
// is sound: RemoveDeployedConfigAsync deletes from three tables in one transaction,
// the schema has NO foreign keys, and CDC captures the three deletes as independent
// per-table streams that LWW may apply to the peer in any order. Nothing in the
// system re-derives an orphan, so a dropped delete leaves a permanently stale
// static_attribute_overrides or native_alarm_state row on the standby — invisible
// until that instance name is redeployed and picks up ghost overrides.
var storageA = Storage(A);
// A second instance that is never removed. Without it this test cannot distinguish
// "the cascade converged" from "node B lost these tables entirely" — a registration
// bug that wiped B would satisfy the absence assertions on its own.
foreach (var instance in new[] { "inst-cascade", "inst-survivor" })
{
await storageA.StoreDeployedConfigAsync(
instance, """{"v":1}""", $"dep-{instance}", $"hash-{instance}", isEnabled: true);
await storageA.SetStaticOverrideAsync(instance, "Setpoint", "42");
await storageA.SetStaticOverrideAsync(instance, "Mode", "Auto");
await storageA.UpsertNativeAlarmAsync(
instance, "Area1.Line1.Alarm", "ref-1", """{"active":true}""", DateTimeOffset.UtcNow);
}
await WaitUntilAsync(
async () =>
await CountAsync(B, "deployed_configurations", "inst-cascade") == 1 &&
await CountAsync(B, "static_attribute_overrides", "inst-cascade") == 2 &&
await CountAsync(B, "native_alarm_state", "inst-cascade") == 1,
"all three tables to reach node B before the instance is removed");
await storageA.RemoveDeployedConfigAsync("inst-cascade");
await WaitUntilAsync(
async () =>
await CountAsync(B, "deployed_configurations", "inst-cascade") == 0 &&
await CountAsync(B, "static_attribute_overrides", "inst-cascade") == 0 &&
await CountAsync(B, "native_alarm_state", "inst-cascade") == 0,
"the 3-table cascade delete to converge on node B with NO orphans left behind");
// The control instance is untouched on both nodes: the cascade is scoped, not a wipe.
Assert.Equal(1, await CountAsync(B, "deployed_configurations", "inst-survivor"));
Assert.Equal(2, await CountAsync(B, "static_attribute_overrides", "inst-survivor"));
Assert.Equal(1, await CountAsync(B, "native_alarm_state", "inst-survivor"));
}
[Fact]
public async Task ANativeAlarmBurst_Converges_AndTheOplogDrains()
{
// native_alarm_state is the highest-rate replicated table (Task 1 found it bounded by
// per-SourceReference coalescing on a 100 ms flush, correcting the plan's D4). This
// asserts the burst converges AND that the oplog goes back to empty afterwards —
// convergence alone would still pass if entries replicated but were never acked, and
// an oplog that only grows eventually trips the caps and forces a snapshot resync.
const int burst = 200;
var storageA = Storage(A);
var at = DateTimeOffset.UtcNow;
for (var i = 0; i < burst; i++)
{
await storageA.UpsertNativeAlarmAsync(
"inst-burst", "Area1.Line1.Alarm", $"ref-{i}", $$"""{"seq":{{i}}}""", at);
}
await WaitUntilAsync(
async () => await CountAsync(B, "native_alarm_state", "inst-burst") == burst,
$"all {burst} alarm rows to reach node B");
await WaitUntilAsync(
async () => await OplogDepthAsync(A) == 0,
"node A's oplog to drain once node B acknowledges the burst");
}
[Fact]
public async Task RowsWrittenOnBWhileItsListenerIsDown_SurviveTheRejoin()
{
// Companion to LocalDbConfigConvergenceTests' N1 scenario, which covers the same
// rejoin for sf_messages and deployed_configurations. This one exercises the tables
// that scenario does not touch, because "the union survives" is a per-table property:
// it holds only if each table is actually registered, and an unregistered table is
// silently local-only rather than an error.
await Storage(A).StoreSharedScriptAsync("script-before", "return 1;", null, null);
await WaitUntilAsync(
async () => await ScalarAsync(B,
"SELECT COUNT(*) FROM shared_scripts WHERE name = 'script-before'") == 1,
"the pre-outage script to reach node B");
await StopPassiveAsync();
// Each node writes to a different Phase 2 table while they cannot see each other.
await Storage(A).StoreExternalSystemAsync(
"sys-from-a", "http://a.invalid", "None", null, null);
await Storage(B).SetStaticOverrideAsync("inst-offline", "WrittenWhileDown", "yes");
await RestartPairAsync();
await WaitUntilAsync(
async () =>
await ScalarAsync(A,
"SELECT COUNT(*) FROM static_attribute_overrides WHERE attribute_name = 'WrittenWhileDown'") == 1 &&
await ScalarAsync(B,
"SELECT COUNT(*) FROM external_systems WHERE name = 'sys-from-a'") == 1,
"both nodes to hold the union of what each wrote while partitioned");
}
}
tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbSitePairConvergenceTests.cs
+7 -203
View File
@@ -1,202 +1,23 @@
using System.Net;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Hosting.Server;
using Microsoft.AspNetCore.Hosting.Server.Features;
using Microsoft.AspNetCore.Server.Kestrel.Core;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using ZB.MOM.WW.LocalDb; using ZB.MOM.WW.LocalDb;
using ZB.MOM.WW.LocalDb.Replication;
using ZB.MOM.WW.ScadaBridge.Host;
namespace ZB.MOM.WW.ScadaBridge.IntegrationTests; namespace ZB.MOM.WW.ScadaBridge.IntegrationTests;
/// <summary> /// <summary>
/// Serializes the site-pair convergence tests against each other: each one stands up a real /// Phase 1 convergence: operation tracking and site events across a real site pair.
/// Kestrel listener plus two SQLite files, and running them concurrently under CI
/// contention is a flakiness risk.
/// </summary>
[CollectionDefinition("LocalDbSitePairConvergence")]
public sealed class LocalDbSitePairConvergenceCollection;
/// <summary>
/// Two ScadaBridge site nodes replicating the consolidated site database over a REAL
/// loopback gRPC transport, through the REAL fail-closed auth interceptor.
/// </summary> /// </summary>
/// <remarks> /// <remarks>
/// <para>
/// This is the test that answers the question Phase 1 exists to answer: does a site node /// This is the test that answers the question Phase 1 exists to answer: does a site node
/// pair actually stop losing operation-tracking and site-event state? Everything upstream /// pair actually stop losing operation-tracking and site-event state? Everything upstream
/// of it — schema helpers, DI wiring, the interceptor — can be individually green while the /// of it — schema helpers, DI wiring, the interceptor — can be individually green while the
/// pair still fails to converge. /// pair still fails to converge.
/// </para>
/// <para> /// <para>
/// It uses <see cref="SiteLocalDbSetup.OnReady"/>, not a hand-written schema, so the tables, /// The fixture lives in <see cref="LocalDbSitePairHarness"/>, shared with the Phase 2
/// their primary keys, and the registration ORDER under test are the ones the host actually /// convergence suites.
/// runs. A separate schema here would prove only that the test agrees with itself.
/// </para>
/// <para>
/// Offline: no docker, no external services. Loopback Kestrel with h2c.
/// </para> /// </para>
/// </remarks> /// </remarks>
[Collection("LocalDbSitePairConvergence")] [Collection("LocalDbSitePairConvergence")]
public sealed class LocalDbSitePairConvergenceTests : IAsyncLifetime public sealed class LocalDbSitePairConvergenceTests : LocalDbSitePairHarness
{ {
private const string SharedApiKey = "site-pair-convergence-key";
private static readonly TimeSpan ConvergeTimeout = TimeSpan.FromSeconds(30);
private readonly string _pathA = Path.Combine(Path.GetTempPath(), $"sitepairA-{Guid.NewGuid():N}.db");
private readonly string _pathB = Path.Combine(Path.GetTempPath(), $"sitepairB-{Guid.NewGuid():N}.db");
// The databases are owned by the fixture, in their own providers, and registered into the
// hosts as pre-constructed instances. MS.DI does not dispose instances it did not create,
// so tearing a host down (the offline-peer scenario) leaves the databases intact and
// writable — which is exactly what lets node A accumulate writes while B is down.
private ServiceProvider _dbProviderA = null!;
private ServiceProvider _dbProviderB = null!;
private IHost? _serverHost; // node B — passive
private IHost? _initiatorHost; // node A — dials the peer
static LocalDbSitePairConvergenceTests() =>
// Grpc.Net.Client dials the loopback server over HTTP/2 cleartext.
AppContext.SetSwitch("System.Net.Http.SocketsHttpHandler.Http2UnencryptedSupport", true);
private ILocalDb A => _dbProviderA.GetRequiredService<ILocalDb>();
private ILocalDb B => _dbProviderB.GetRequiredService<ILocalDb>();
public async Task InitializeAsync()
{
_dbProviderA = BuildDatabaseProvider(_pathA, "node-a");
_dbProviderB = BuildDatabaseProvider(_pathB, "node-b");
// Force construction (and therefore OnReady) before anything replicates.
_ = A;
_ = B;
await StartPassiveAsync();
await StartInitiatorAsync();
}
public async Task DisposeAsync()
{
await StopHostAsync(_initiatorHost);
await StopHostAsync(_serverHost);
await _dbProviderA.DisposeAsync();
await _dbProviderB.DisposeAsync();
Microsoft.Data.Sqlite.SqliteConnection.ClearAllPools();
foreach (var path in new[] { _pathA, _pathB })
{
foreach (var suffix in new[] { "", "-wal", "-shm" })
{
try { File.Delete(path + suffix); } catch { /* best effort */ }
}
}
}
// ---- fixture internals ------------------------------------------------------------
/// <summary>
/// A provider owning one consolidated site database, initialized through the host's own
/// <see cref="SiteLocalDbSetup.OnReady"/> — same schema, same registration order.
/// </summary>
private static ServiceProvider BuildDatabaseProvider(string path, string nodeName)
{
var config = new ConfigurationBuilder()
.AddInMemoryCollection(new Dictionary<string, string?>
{
["LocalDb:Path"] = path,
["ScadaBridge:Node:NodeName"] = nodeName,
// Point the legacy migrator at paths that do not exist, so it no-ops rather
// than picking up stray files from the test working directory.
["ScadaBridge:OperationTracking:ConnectionString"] =
$"Data Source={Path.Combine(Path.GetTempPath(), $"absent-{Guid.NewGuid():N}.db")}",
["ScadaBridge:SiteEventLog:DatabasePath"] =
Path.Combine(Path.GetTempPath(), $"absent-{Guid.NewGuid():N}.db"),
})
.Build();
return new ServiceCollection()
.AddZbLocalDb(config, db => SiteLocalDbSetup.OnReady(db, config))
.BuildServiceProvider();
}
private static IConfiguration ReplicationConfig(string? peerAddress)
{
var values = new Dictionary<string, string?>
{
// Tight flush + bounded reconnect backoff so convergence is observable well
// inside the poll deadline. The 60 s production default would let the doubling
// backoff overrun it after a peer outage.
["LocalDb:Replication:FlushInterval"] = "00:00:00.050",
["LocalDb:Replication:ReconnectBackoffMax"] = "00:00:02",
// Both nodes share one key — the interceptor is fail-closed, so a mismatch here
// turns every scenario below red (verified by deliberately breaking it).
["LocalDb:Replication:ApiKey"] = SharedApiKey,
};
if (peerAddress is not null)
values["LocalDb:Replication:PeerAddress"] = peerAddress;
return new ConfigurationBuilder().AddInMemoryCollection(values).Build();
}
private async Task StartPassiveAsync()
{
var config = ReplicationConfig(peerAddress: null);
_serverHost = await new HostBuilder()
.ConfigureWebHost(web =>
{
web.UseKestrel(o =>
o.Listen(IPAddress.Loopback, 0, listen => listen.Protocols = HttpProtocols.Http2));
web.ConfigureServices(services =>
{
services.AddLogging();
services.AddRouting();
// The REAL interceptor, not a stand-in. If it rejected legitimate peer
// traffic, every scenario below would fail — which is the point.
services.AddGrpc(o => o.Interceptors.Add<LocalDbSyncAuthInterceptor>());
services.AddSingleton(B);
services.AddZbLocalDbReplication(config);
});
web.Configure(app =>
{
app.UseRouting();
app.UseEndpoints(e => e.MapZbLocalDbSync());
});
})
.StartAsync();
}
private async Task StartInitiatorAsync()
{
var config = ReplicationConfig(PassiveAddress());
_initiatorHost = await new HostBuilder()
.ConfigureServices(services =>
{
services.AddLogging();
services.AddSingleton(A);
services.AddZbLocalDbReplication(config);
})
.StartAsync();
}
private string PassiveAddress()
=> _serverHost!.Services.GetRequiredService<IServer>()
.Features.Get<IServerAddressesFeature>()!.Addresses.Single();
private static async Task StopHostAsync(IHost? host)
{
if (host is null) return;
try { await host.StopAsync(TimeSpan.FromSeconds(5)); } catch { /* teardown */ }
host.Dispose();
}
// ---- data helpers ----------------------------------------------------------------- // ---- data helpers -----------------------------------------------------------------
private static Task WriteTrackingAsync(ILocalDb db, string id, string status, string target) private static Task WriteTrackingAsync(ILocalDb db, string id, string status, string target)
@@ -232,19 +53,6 @@ public sealed class LocalDbSitePairConvergenceTests : IAsyncLifetime
private static Task<IReadOnlyList<string>> ReadEventIdsAsync(ILocalDb db) private static Task<IReadOnlyList<string>> ReadEventIdsAsync(ILocalDb db)
=> db.QueryAsync("SELECT id FROM site_events ORDER BY id", static r => r.GetString(0)); => db.QueryAsync("SELECT id FROM site_events ORDER BY id", static r => r.GetString(0));
/// <summary>Polls <paramref name="condition"/> until true or the deadline passes.</summary>
private static async Task WaitUntilAsync(Func<Task<bool>> condition, string because)
{
var deadline = DateTime.UtcNow + ConvergeTimeout;
while (DateTime.UtcNow < deadline)
{
if (await condition()) return;
await Task.Delay(50);
}
Assert.Fail($"Timed out after {ConvergeTimeout.TotalSeconds:0}s waiting for: {because}");
}
// ---- scenarios -------------------------------------------------------------------- // ---- scenarios --------------------------------------------------------------------
[Fact] [Fact]
@@ -322,19 +130,15 @@ public sealed class LocalDbSitePairConvergenceTests : IAsyncLifetime
{ {
// The failover case that motivates Phase 1: one node is down while the other keeps // The failover case that motivates Phase 1: one node is down while the other keeps
// working, and nothing written during the outage may be lost. // working, and nothing written during the outage may be lost.
await StopHostAsync(_serverHost); await StopPassiveAsync();
_serverHost = null;
var idsDuringOutage = Enumerable.Range(0, 5).Select(_ => Guid.NewGuid().ToString("N")).ToList(); var idsDuringOutage = Enumerable.Range(0, 5).Select(_ => Guid.NewGuid().ToString("N")).ToList();
foreach (var id in idsDuringOutage) await WriteEventAsync(A, id, "written while B was down"); foreach (var id in idsDuringOutage) await WriteEventAsync(A, id, "written while B was down");
await WriteTrackingAsync(A, "op-during-outage", "Delivered", "ERP.GetOrder"); await WriteTrackingAsync(A, "op-during-outage", "Delivered", "ERP.GetOrder");
// Node B's database survived the host teardown (pre-constructed instance), so this is // Node B's database survived the host teardown (pre-constructed instance), so this is
// a genuine rejoin rather than a fresh node. It comes back on a NEW loopback port; // a genuine rejoin rather than a fresh node.
// the initiator's channel factory re-reads the peer address on each reconnect. await RestartPairAsync();
await StartPassiveAsync();
await StopHostAsync(_initiatorHost);
await StartInitiatorAsync();
await WaitUntilAsync( await WaitUntilAsync(
async () => async () =>
tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbSitePairHarness.cs
+260
View File
@@ -0,0 +1,260 @@
using System.Net;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Hosting.Server;
using Microsoft.AspNetCore.Hosting.Server.Features;
using Microsoft.AspNetCore.Server.Kestrel.Core;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using ZB.MOM.WW.LocalDb;
using ZB.MOM.WW.LocalDb.Replication;
using ZB.MOM.WW.ScadaBridge.Host;
namespace ZB.MOM.WW.ScadaBridge.IntegrationTests;
/// <summary>
/// Serializes the site-pair convergence tests against each other: each one stands up a real
/// Kestrel listener plus two SQLite files, and running them concurrently under CI
/// contention is a flakiness risk.
/// </summary>
[CollectionDefinition("LocalDbSitePairConvergence")]
public sealed class LocalDbSitePairConvergenceCollection;
/// <summary>
/// Two ScadaBridge site nodes replicating the consolidated site database over a REAL
/// loopback gRPC transport, through the REAL fail-closed auth interceptor.
/// </summary>
/// <remarks>
/// <para>
/// Extracted from the Phase 1 convergence tests once Phase 2 needed the same pair for the
/// store-and-forward buffer and the configuration tables. Everything here is fixture; the
/// derived classes hold only their own data helpers and scenarios.
/// </para>
/// <para>
/// It uses <see cref="SiteLocalDbSetup.OnReady"/>, not a hand-written schema, so the tables,
/// their primary keys, and the registration ORDER under test are the ones the host actually
/// runs. A separate schema here would prove only that the test agrees with itself.
/// </para>
/// <para>
/// Offline: no docker, no external services. Loopback Kestrel with h2c.
/// </para>
/// </remarks>
public abstract class LocalDbSitePairHarness : IAsyncLifetime
{
private const string SharedApiKey = "site-pair-convergence-key";
/// <summary>How long a scenario waits for the pair to agree before failing.</summary>
protected static readonly TimeSpan ConvergeTimeout = TimeSpan.FromSeconds(30);
private readonly string _pathA = Path.Combine(Path.GetTempPath(), $"sitepairA-{Guid.NewGuid():N}.db");
private readonly string _pathB = Path.Combine(Path.GetTempPath(), $"sitepairB-{Guid.NewGuid():N}.db");
// The databases are owned by the fixture, in their own providers, and registered into the
// hosts as pre-constructed instances. MS.DI does not dispose instances it did not create,
// so tearing a host down (the offline-peer scenario) leaves the databases intact and
// writable — which is exactly what lets node A accumulate writes while B is down.
private ServiceProvider _dbProviderA = null!;
private ServiceProvider _dbProviderB = null!;
private IHost? _serverHost; // node B — passive
private IHost? _initiatorHost; // node A — dials the peer
static LocalDbSitePairHarness() =>
// Grpc.Net.Client dials the loopback server over HTTP/2 cleartext.
AppContext.SetSwitch("System.Net.Http.SocketsHttpHandler.Http2UnencryptedSupport", true);
/// <summary>Node A — the initiator, which dials the peer.</summary>
protected ILocalDb A => _dbProviderA.GetRequiredService<ILocalDb>();
/// <summary>Node B — the passive node, which listens.</summary>
protected ILocalDb B => _dbProviderB.GetRequiredService<ILocalDb>();
public async Task InitializeAsync()
{
_dbProviderA = BuildDatabaseProvider(_pathA, "node-a");
_dbProviderB = BuildDatabaseProvider(_pathB, "node-b");
// Force construction (and therefore OnReady) before anything replicates.
_ = A;
_ = B;
await StartPassiveAsync();
await StartInitiatorAsync();
}
public async Task DisposeAsync()
{
await StopHostAsync(_initiatorHost);
await StopHostAsync(_serverHost);
await _dbProviderA.DisposeAsync();
await _dbProviderB.DisposeAsync();
Microsoft.Data.Sqlite.SqliteConnection.ClearAllPools();
foreach (var path in new[] { _pathA, _pathB })
{
foreach (var suffix in new[] { "", "-wal", "-shm" })
{
try { File.Delete(path + suffix); } catch { /* best effort */ }
}
}
}
// ---- fixture internals ------------------------------------------------------------
/// <summary>
/// A provider owning one consolidated site database, initialized through the host's own
/// <see cref="SiteLocalDbSetup.OnReady"/> — same schema, same registration order.
/// </summary>
private static ServiceProvider BuildDatabaseProvider(string path, string nodeName)
{
var config = new ConfigurationBuilder()
.AddInMemoryCollection(new Dictionary<string, string?>
{
["LocalDb:Path"] = path,
["ScadaBridge:Node:NodeName"] = nodeName,
// Point the legacy migrators at paths that do not exist, so they no-op rather
// than picking up stray files from the test working directory. The two Phase 2
// defaults matter most: unlike the Phase 1 pair they resolve inside ./data/,
// and a migration would also RENAME whatever it found.
["ScadaBridge:OperationTracking:ConnectionString"] =
$"Data Source={Path.Combine(Path.GetTempPath(), $"absent-{Guid.NewGuid():N}.db")}",
["ScadaBridge:SiteEventLog:DatabasePath"] =
Path.Combine(Path.GetTempPath(), $"absent-{Guid.NewGuid():N}.db"),
["ScadaBridge:StoreAndForward:SqliteDbPath"] =
Path.Combine(Path.GetTempPath(), $"absent-{Guid.NewGuid():N}.db"),
["ScadaBridge:Database:SiteDbPath"] =
Path.Combine(Path.GetTempPath(), $"absent-{Guid.NewGuid():N}.db"),
})
.Build();
return new ServiceCollection()
.AddZbLocalDb(config, db => SiteLocalDbSetup.OnReady(db, config))
.BuildServiceProvider();
}
/// <summary>
/// The eight tables the Phase 2 cutover registers. Listed literally rather than derived
/// from production code, so a registration that drifts fails these tests instead of
/// agreeing with itself.
/// </summary>
/// <remarks>
/// <c>notification_lists</c> and <c>smtp_configurations</c> are absent by design. They
/// are permanently empty (no site writer since 2026-07-10, the migrator skips them, the
/// active-node purge keeps them empty), so registering them would open a standing
/// replication channel whose only historical payload was plaintext SMTP passwords.
/// </remarks>
protected static readonly string[] Phase2ReplicatedTables =
[
"sf_messages",
"deployed_configurations", "static_attribute_overrides", "shared_scripts",
"external_systems", "database_connections", "data_connection_definitions",
"native_alarm_state",
];
private static IConfiguration ReplicationConfig(string? peerAddress)
{
var values = new Dictionary<string, string?>
{
// Tight flush + bounded reconnect backoff so convergence is observable well
// inside the poll deadline. The 60 s production default would let the doubling
// backoff overrun it after a peer outage.
["LocalDb:Replication:FlushInterval"] = "00:00:00.050",
["LocalDb:Replication:ReconnectBackoffMax"] = "00:00:02",
// Both nodes share one key — the interceptor is fail-closed, so a mismatch here
// turns every scenario below red (verified by deliberately breaking it).
["LocalDb:Replication:ApiKey"] = SharedApiKey,
};
if (peerAddress is not null)
values["LocalDb:Replication:PeerAddress"] = peerAddress;
return new ConfigurationBuilder().AddInMemoryCollection(values).Build();
}
/// <summary>Starts node B, the passive listener.</summary>
protected async Task StartPassiveAsync()
{
var config = ReplicationConfig(peerAddress: null);
_serverHost = await new HostBuilder()
.ConfigureWebHost(web =>
{
web.UseKestrel(o =>
o.Listen(IPAddress.Loopback, 0, listen => listen.Protocols = HttpProtocols.Http2));
web.ConfigureServices(services =>
{
services.AddLogging();
services.AddRouting();
// The REAL interceptor, not a stand-in. If it rejected legitimate peer
// traffic, every scenario below would fail — which is the point.
services.AddGrpc(o => o.Interceptors.Add<LocalDbSyncAuthInterceptor>());
services.AddSingleton(B);
services.AddZbLocalDbReplication(config);
});
web.Configure(app =>
{
app.UseRouting();
app.UseEndpoints(e => e.MapZbLocalDbSync());
});
})
.StartAsync();
}
/// <summary>Starts node A, which dials the passive node.</summary>
protected async Task StartInitiatorAsync()
{
var config = ReplicationConfig(PassiveAddress());
_initiatorHost = await new HostBuilder()
.ConfigureServices(services =>
{
services.AddLogging();
services.AddSingleton(A);
services.AddZbLocalDbReplication(config);
})
.StartAsync();
}
/// <summary>Takes node B's listener down, leaving its database intact and writable.</summary>
protected async Task StopPassiveAsync()
{
await StopHostAsync(_serverHost);
_serverHost = null;
}
/// <summary>
/// Brings node B back on a NEW loopback port and re-dials from A. The initiator's channel
/// factory re-reads the peer address on each reconnect, so this is a genuine rejoin.
/// </summary>
protected async Task RestartPairAsync()
{
await StartPassiveAsync();
await StopHostAsync(_initiatorHost);
await StartInitiatorAsync();
}
private string PassiveAddress()
=> _serverHost!.Services.GetRequiredService<IServer>()
.Features.Get<IServerAddressesFeature>()!.Addresses.Single();
private static async Task StopHostAsync(IHost? host)
{
if (host is null) return;
try { await host.StopAsync(TimeSpan.FromSeconds(5)); } catch { /* teardown */ }
host.Dispose();
}
/// <summary>Polls <paramref name="condition"/> until true or the deadline passes.</summary>
protected static async Task WaitUntilAsync(Func<Task<bool>> condition, string because)
{
var deadline = DateTime.UtcNow + ConvergeTimeout;
while (DateTime.UtcNow < deadline)
{
if (await condition()) return;
await Task.Delay(50);
}
Assert.Fail($"Timed out after {ConvergeTimeout.TotalSeconds:0}s waiting for: {because}");
}
}
tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/LocalDbStoreAndForwardConvergenceTests.cs
+221
View File
@@ -0,0 +1,221 @@
using ZB.MOM.WW.LocalDb;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
namespace ZB.MOM.WW.ScadaBridge.IntegrationTests;
/// <summary>
/// Phase 2 convergence: the store-and-forward buffer across a real site pair.
/// </summary>
/// <remarks>
/// <para>
/// These are <b>specifications, written before the deletion they justify</b>. The bespoke
/// <c>ReplicationService</c> (an explicit Add/Remove/Park/Requeue operation stream over Akka)
/// is deleted at Task 14 and replaced by LocalDb's trigger-based change capture. Each test
/// here is one behaviour the old mechanism provided, restated as an outcome the replacement
/// must still deliver — so the cutover has something to be judged against other than "it
/// compiles".
/// </para>
/// <para>
/// They are written in terms of <i>rows</i>, not operations, which is the whole point:
/// under CDC there is no Add or Park message to observe, only a row that must end up in the
/// right state on both nodes.
/// </para>
/// <para>
/// <b>One intent is deliberately not ported:</b>
/// <c>ReplicationServiceTests.ReplicationOperations_AreDispatchedInIssueOrder</c> — 200
/// interleaved operations dispatched synchronously, observed in strict issue order. That
/// asserts the <i>mechanism</i> (inline fire-and-forget dispatch), not an outcome, and CDC
/// capture is asynchronous and batched by construction, so no honest port exists. Its
/// portable content is the ordering <i>outcome</i>: an add followed by a remove must never
/// converge to present. That is
/// <see cref="MessageAddedThenRemoved_NeverConvergesToPresent"/>. This paragraph exists so a
/// future reader does not conclude the test was dropped by accident.
/// </para>
/// </remarks>
[Collection("LocalDbSitePairConvergence")]
public sealed class LocalDbStoreAndForwardConvergenceTests : LocalDbSitePairHarness
{
// ---- data helpers -----------------------------------------------------------------
/// <summary>Buffers a message, as <c>StoreAndForwardStorage.EnqueueAsync</c> does.</summary>
private static Task EnqueueAsync(ILocalDb db, string id, string target = "ERP.GetOrder")
=> db.ExecuteAsync(
"""
INSERT INTO sf_messages (
id, category, target, payload_json, retry_count, max_retries,
retry_interval_ms, created_at, status)
VALUES (@id, 0, @target, '{"order":1}', 0, 50, 30000, @now, @pending)
ON CONFLICT(id) DO UPDATE SET
status = excluded.status,
retry_count = excluded.retry_count;
""",
new
{
id,
target,
now = DateTime.UtcNow.ToString("o"),
pending = (int)StoreAndForwardMessageStatus.Pending,
});
/// <summary>Deletes a delivered message, as the successful-retry path does.</summary>
private static Task DeleteAsync(ILocalDb db, string id)
=> db.ExecuteAsync("DELETE FROM sf_messages WHERE id = @id;", new { id });
private static Task SetStatusAsync(ILocalDb db, string id, StoreAndForwardMessageStatus status, int retryCount)
=> db.ExecuteAsync(
"UPDATE sf_messages SET status = @status, retry_count = @retryCount WHERE id = @id;",
new { id, status = (int)status, retryCount });
private static async Task<(int Status, int RetryCount)?> ReadAsync(ILocalDb db, string id)
{
var rows = await db.QueryAsync(
"SELECT status, retry_count FROM sf_messages WHERE id = @id",
static r => (r.GetInt32(0), r.GetInt32(1)), new { id });
return rows.Count == 0 ? null : rows[0];
}
private static async Task<bool> ExistsAsync(ILocalDb db, string id)
=> await ReadAsync(db, id) is not null;
private static async Task<bool> HasStatusAsync(
ILocalDb db, string id, StoreAndForwardMessageStatus status)
=> await ReadAsync(db, id) is { } row && row.Status == (int)status;
// ---- scenarios --------------------------------------------------------------------
[Fact]
public async Task BufferedMessage_MaterialisesOnThePeer()
{
// Was: BufferingAMessage_ReplicatesAnAddOperation.
await EnqueueAsync(A, "msg-add");
await WaitUntilAsync(
() => ExistsAsync(B, "msg-add"),
"a message buffered on node A to appear on node B");
}
[Fact]
public async Task DeliveredMessage_DisappearsFromThePeer()
{
// Was: SuccessfulRetry_ReplicatesARemoveOperation. Row deletes propagate as
// tombstones under CDC — the peer must not keep re-delivering a message that node A
// already succeeded on.
await EnqueueAsync(A, "msg-remove");
await WaitUntilAsync(() => ExistsAsync(B, "msg-remove"), "the message to arrive on B first");
await DeleteAsync(A, "msg-remove");
await WaitUntilAsync(
async () => !await ExistsAsync(B, "msg-remove"),
"the delivered message to be gone from node B");
}
[Fact]
public async Task ParkedMessage_ShowsAsParkedOnThePeer()
{
// Was: ParkedMessage_ReplicatesAParkOperation.
await EnqueueAsync(A, "msg-park");
await WaitUntilAsync(() => ExistsAsync(B, "msg-park"), "the message to arrive on B first");
await SetStatusAsync(A, "msg-park", StoreAndForwardMessageStatus.Parked, retryCount: 50);
await WaitUntilAsync(
() => HasStatusAsync(B, "msg-park", StoreAndForwardMessageStatus.Parked),
"the parked status to reach node B");
}
[Fact]
public async Task RequeuedMessage_ResetsStatusAndRetryCountOnThePeer()
{
// Was: RetryingAParkedMessage_ReplicatesARequeueOperation +
// ApplyReplicatedOperation_Requeue_MovesStandbyRowBackToPending. RetryCount matters
// as much as status: a peer that took Pending but kept retry_count at max would park
// the message again on its first attempt after a failover.
await EnqueueAsync(A, "msg-requeue");
await SetStatusAsync(A, "msg-requeue", StoreAndForwardMessageStatus.Parked, retryCount: 50);
await WaitUntilAsync(
() => HasStatusAsync(B, "msg-requeue", StoreAndForwardMessageStatus.Parked),
"the message to be parked on B before it is requeued");
await SetStatusAsync(A, "msg-requeue", StoreAndForwardMessageStatus.Pending, retryCount: 0);
await WaitUntilAsync(
async () => await ReadAsync(B, "msg-requeue")
is { Status: (int)StoreAndForwardMessageStatus.Pending, RetryCount: 0 },
"node B to show the requeued message as Pending with retry_count 0");
}
[Fact]
public async Task MessageAddedThenRemoved_NeverConvergesToPresent()
{
// The portable intent of ReplicationOperations_AreDispatchedInIssueOrder (see the
// class remarks). Ordering only ever mattered because a remove overtaken by its own
// add would resurrect a delivered message and send it twice. Under LWW that
// reordering is not preventable by dispatch discipline — it is prevented because the
// tombstone carries the later HLC and therefore wins regardless of arrival order.
//
// Asserting the endpoint rather than the sequence is what makes this portable: the
// old test would fail on any async transport even when the outcome was correct.
await EnqueueAsync(A, "msg-ordering");
await DeleteAsync(A, "msg-ordering");
// A control row written in the same window. Without it this test is VACUOUS: an
// absent row is also what a pair that replicates nothing at all looks like, so it
// would pass with capture switched off entirely (observed — it was the only one of
// these seven that survived unregistering sf_messages). The control converging is
// what makes the absence of msg-ordering evidence rather than silence.
await EnqueueAsync(A, "msg-ordering-control");
await WaitUntilAsync(
() => ExistsAsync(B, "msg-ordering-control"),
"the control message to reach node B, proving the pair was replicating");
await WaitUntilAsync(
async () => !await ExistsAsync(B, "msg-ordering") && !await ExistsAsync(A, "msg-ordering"),
"the add-then-remove pair to settle as absent on both nodes");
// Absence has to persist, not just occur once: a late-arriving add would flip the row
// back and the poll above could have observed a gap it never actually converged to.
await Task.Delay(TimeSpan.FromSeconds(2));
Assert.False(await ExistsAsync(B, "msg-ordering"));
Assert.False(await ExistsAsync(A, "msg-ordering"));
Assert.True(await ExistsAsync(B, "msg-ordering-control"));
}
[Fact]
public async Task SameMessage_BufferedTwice_ConvergesToTheNewerState()
{
// Was: ApplyReplicatedAdd_Twice_IsIdempotent_NewestWins. Under CDC this is not a
// property the application has to implement — LWW on the primary key gives it — but
// it is still a property the pair must HAVE, so it is asserted rather than assumed.
await EnqueueAsync(A, "msg-twice");
await WaitUntilAsync(() => ExistsAsync(B, "msg-twice"), "the first write to reach B");
await SetStatusAsync(A, "msg-twice", StoreAndForwardMessageStatus.InFlight, retryCount: 3);
await WaitUntilAsync(
async () => await ReadAsync(B, "msg-twice")
is { Status: (int)StoreAndForwardMessageStatus.InFlight, RetryCount: 3 },
"the newer state to win on node B");
}
[Fact]
public async Task ParkArrivingWithoutItsAdd_StillMaterialisesTheRow()
{
// Was: ApplyReplicatedPark_WhenAddWasLost_MaterializesTheParkedRow. The bespoke
// replicator needed explicit upsert semantics so a lost Add did not leave a Park with
// nothing to update. The CDC equivalent is a peer that was offline for the add and
// only ever sees the row in its parked state — it must still end up with the row,
// not skip it as an update to something it never had.
await StopPassiveAsync();
await EnqueueAsync(A, "msg-park-no-add");
await SetStatusAsync(A, "msg-park-no-add", StoreAndForwardMessageStatus.Parked, retryCount: 50);
await RestartPairAsync();
await WaitUntilAsync(
() => HasStatusAsync(B, "msg-park-no-add", StoreAndForwardMessageStatus.Parked),
"node B to materialise a row it only ever saw in its parked state");
}
}
tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/RecoveryDrillTests.cs
+23 -9
View File
@@ -3,6 +3,7 @@ using Microsoft.Extensions.Logging.Abstractions;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment; using ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
using ZB.MOM.WW.ScadaBridge.StoreAndForward; using ZB.MOM.WW.ScadaBridge.StoreAndForward;
using ZB.MOM.WW.ScadaBridge.TestSupport;
namespace ZB.MOM.WW.ScadaBridge.IntegrationTests; namespace ZB.MOM.WW.ScadaBridge.IntegrationTests;
@@ -59,11 +60,15 @@ public class RecoveryDrillTests
// Scenario: Communication drops while deploying system-wide artifacts. // Scenario: Communication drops while deploying system-wide artifacts.
// The deployment command is buffered by S&F and retried when connection restores. // The deployment command is buffered by S&F and retried when connection restores.
var dbPath = Path.Combine(Path.GetTempPath(), $"sf_commdrop_{Guid.NewGuid():N}.db"); var dbPath = Path.Combine(Path.GetTempPath(), $"sf_commdrop_{Guid.NewGuid():N}.db");
var connStr = $"Data Source={dbPath}";
// The store takes an ILocalDb (LocalDb has no in-memory mode), so the buffer
// lives in a real temp file for the duration of the drill.
TestLocalDb? localDb = null;
try try
{ {
var storage = new StoreAndForwardStorage(connStr, NullLogger<StoreAndForwardStorage>.Instance); localDb = TestLocalDb.Create(dbPath);
var storage = new StoreAndForwardStorage(localDb.Db, NullLogger<StoreAndForwardStorage>.Instance);
await storage.InitializeAsync(); await storage.InitializeAsync();
var options = new StoreAndForwardOptions var options = new StoreAndForwardOptions
@@ -102,8 +107,9 @@ public class RecoveryDrillTests
} }
finally finally
{ {
if (File.Exists(dbPath)) // Dispose before deleting — the master connection anchors the WAL sidecars.
File.Delete(dbPath); localDb?.Dispose();
TestLocalDb.DeleteFiles(dbPath);
} }
} }
@@ -115,12 +121,17 @@ public class RecoveryDrillTests
// On startup, the Deployment Manager Actor reads configs from SQLite and // On startup, the Deployment Manager Actor reads configs from SQLite and
// recreates Instance Actors. // recreates Instance Actors.
var dbPath = Path.Combine(Path.GetTempPath(), $"sf_restart_{Guid.NewGuid():N}.db"); var dbPath = Path.Combine(Path.GetTempPath(), $"sf_restart_{Guid.NewGuid():N}.db");
var connStr = $"Data Source={dbPath}";
// The restarted "process" opens its own local database over the SAME file — that
// is what this drill verifies survives.
TestLocalDb? preRestartDb = null;
TestLocalDb? restartedDb = null;
try try
{ {
// Pre-restart: S&F messages in buffer // Pre-restart: S&F messages in buffer
var storage = new StoreAndForwardStorage(connStr, NullLogger<StoreAndForwardStorage>.Instance); preRestartDb = TestLocalDb.Create(dbPath);
var storage = new StoreAndForwardStorage(preRestartDb.Db, NullLogger<StoreAndForwardStorage>.Instance);
await storage.InitializeAsync(); await storage.InitializeAsync();
for (var i = 0; i < 3; i++) for (var i = 0; i < 3; i++)
@@ -140,7 +151,8 @@ public class RecoveryDrillTests
} }
// Post-restart: new storage instance reads same DB // Post-restart: new storage instance reads same DB
var restartedStorage = new StoreAndForwardStorage(connStr, NullLogger<StoreAndForwardStorage>.Instance); restartedDb = TestLocalDb.Create(dbPath);
var restartedStorage = new StoreAndForwardStorage(restartedDb.Db, NullLogger<StoreAndForwardStorage>.Instance);
await restartedStorage.InitializeAsync(); await restartedStorage.InitializeAsync();
var pending = await restartedStorage.GetMessagesForRetryAsync(); var pending = await restartedStorage.GetMessagesForRetryAsync();
@@ -153,8 +165,10 @@ public class RecoveryDrillTests
} }
finally finally
{ {
if (File.Exists(dbPath)) // Dispose before deleting — the master connections anchor the WAL sidecars.
File.Delete(dbPath); restartedDb?.Dispose();
preRestartDb?.Dispose();
TestLocalDb.DeleteFiles(dbPath);
} }
} }
tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/SiteFailoverTests.cs
+37 -15
View File
@@ -4,6 +4,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Streaming; using ZB.MOM.WW.ScadaBridge.Commons.Messages.Streaming;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
using ZB.MOM.WW.ScadaBridge.StoreAndForward; using ZB.MOM.WW.ScadaBridge.StoreAndForward;
using ZB.MOM.WW.ScadaBridge.TestSupport;
namespace ZB.MOM.WW.ScadaBridge.IntegrationTests; namespace ZB.MOM.WW.ScadaBridge.IntegrationTests;
@@ -21,12 +22,18 @@ public class SiteFailoverTests
// Simulates site failover: messages buffered in SQLite survive process restart. // Simulates site failover: messages buffered in SQLite survive process restart.
// The standby node picks up the same SQLite file and retries pending messages. // The standby node picks up the same SQLite file and retries pending messages.
var dbPath = Path.Combine(Path.GetTempPath(), $"sf_failover_{Guid.NewGuid():N}.db"); var dbPath = Path.Combine(Path.GetTempPath(), $"sf_failover_{Guid.NewGuid():N}.db");
var connStr = $"Data Source={dbPath}";
// The store takes an ILocalDb; each "node" opens its own local database over the
// SAME file, which is how this test models the standby picking up the primary's
// buffer after failover.
TestLocalDb? primaryDb = null;
TestLocalDb? standbyDb = null;
try try
{ {
// Phase 1: Buffer messages on "primary" node // Phase 1: Buffer messages on "primary" node
var storage = new StoreAndForwardStorage(connStr, NullLogger<StoreAndForwardStorage>.Instance); primaryDb = TestLocalDb.Create(dbPath);
var storage = new StoreAndForwardStorage(primaryDb.Db, NullLogger<StoreAndForwardStorage>.Instance);
await storage.InitializeAsync(); await storage.InitializeAsync();
var message = new StoreAndForwardMessage var message = new StoreAndForwardMessage
@@ -46,7 +53,8 @@ public class SiteFailoverTests
await storage.EnqueueAsync(message); await storage.EnqueueAsync(message);
// Phase 2: "Standby" node opens the same database (simulating failover) // Phase 2: "Standby" node opens the same database (simulating failover)
var standbyStorage = new StoreAndForwardStorage(connStr, NullLogger<StoreAndForwardStorage>.Instance); standbyDb = TestLocalDb.Create(dbPath);
var standbyStorage = new StoreAndForwardStorage(standbyDb.Db, NullLogger<StoreAndForwardStorage>.Instance);
await standbyStorage.InitializeAsync(); await standbyStorage.InitializeAsync();
var pending = await standbyStorage.GetMessagesForRetryAsync(); var pending = await standbyStorage.GetMessagesForRetryAsync();
@@ -57,8 +65,10 @@ public class SiteFailoverTests
} }
finally finally
{ {
if (File.Exists(dbPath)) // Dispose before deleting — the master connections anchor the WAL sidecars.
File.Delete(dbPath); standbyDb?.Dispose();
primaryDb?.Dispose();
TestLocalDb.DeleteFiles(dbPath);
} }
} }
@@ -67,11 +77,14 @@ public class SiteFailoverTests
public async Task StoreAndForward_ParkedMessages_SurviveFailover() public async Task StoreAndForward_ParkedMessages_SurviveFailover()
{ {
var dbPath = Path.Combine(Path.GetTempPath(), $"sf_parked_{Guid.NewGuid():N}.db"); var dbPath = Path.Combine(Path.GetTempPath(), $"sf_parked_{Guid.NewGuid():N}.db");
var connStr = $"Data Source={dbPath}";
TestLocalDb? primaryDb = null;
TestLocalDb? standbyDb = null;
try try
{ {
var storage = new StoreAndForwardStorage(connStr, NullLogger<StoreAndForwardStorage>.Instance); primaryDb = TestLocalDb.Create(dbPath);
var storage = new StoreAndForwardStorage(primaryDb.Db, NullLogger<StoreAndForwardStorage>.Instance);
await storage.InitializeAsync(); await storage.InitializeAsync();
var parkedMsg = new StoreAndForwardMessage var parkedMsg = new StoreAndForwardMessage
@@ -93,7 +106,8 @@ public class SiteFailoverTests
await storage.EnqueueAsync(parkedMsg); await storage.EnqueueAsync(parkedMsg);
// Standby opens same DB // Standby opens same DB
var standbyStorage = new StoreAndForwardStorage(connStr, NullLogger<StoreAndForwardStorage>.Instance); standbyDb = TestLocalDb.Create(dbPath);
var standbyStorage = new StoreAndForwardStorage(standbyDb.Db, NullLogger<StoreAndForwardStorage>.Instance);
await standbyStorage.InitializeAsync(); await standbyStorage.InitializeAsync();
var (parked, count) = await standbyStorage.GetParkedMessagesAsync(); var (parked, count) = await standbyStorage.GetParkedMessagesAsync();
@@ -102,8 +116,10 @@ public class SiteFailoverTests
} }
finally finally
{ {
if (File.Exists(dbPath)) // Dispose before deleting — the master connections anchor the WAL sidecars.
File.Delete(dbPath); standbyDb?.Dispose();
primaryDb?.Dispose();
TestLocalDb.DeleteFiles(dbPath);
} }
} }
@@ -160,11 +176,14 @@ public class SiteFailoverTests
public async Task StoreAndForward_BufferDepth_ReportedAfterFailover() public async Task StoreAndForward_BufferDepth_ReportedAfterFailover()
{ {
var dbPath = Path.Combine(Path.GetTempPath(), $"sf_depth_{Guid.NewGuid():N}.db"); var dbPath = Path.Combine(Path.GetTempPath(), $"sf_depth_{Guid.NewGuid():N}.db");
var connStr = $"Data Source={dbPath}";
TestLocalDb? primaryDb = null;
TestLocalDb? standbyDb = null;
try try
{ {
var storage = new StoreAndForwardStorage(connStr, NullLogger<StoreAndForwardStorage>.Instance); primaryDb = TestLocalDb.Create(dbPath);
var storage = new StoreAndForwardStorage(primaryDb.Db, NullLogger<StoreAndForwardStorage>.Instance);
await storage.InitializeAsync(); await storage.InitializeAsync();
// Enqueue messages in different categories // Enqueue messages in different categories
@@ -199,7 +218,8 @@ public class SiteFailoverTests
} }
// After failover, standby reads buffer depths // After failover, standby reads buffer depths
var standbyStorage = new StoreAndForwardStorage(connStr, NullLogger<StoreAndForwardStorage>.Instance); standbyDb = TestLocalDb.Create(dbPath);
var standbyStorage = new StoreAndForwardStorage(standbyDb.Db, NullLogger<StoreAndForwardStorage>.Instance);
await standbyStorage.InitializeAsync(); await standbyStorage.InitializeAsync();
var depths = await standbyStorage.GetBufferDepthByCategoryAsync(); var depths = await standbyStorage.GetBufferDepthByCategoryAsync();
@@ -208,8 +228,10 @@ public class SiteFailoverTests
} }
finally finally
{ {
if (File.Exists(dbPath)) // Dispose before deleting — the master connections anchor the WAL sidecars.
File.Delete(dbPath); standbyDb?.Dispose();
primaryDb?.Dispose();
TestLocalDb.DeleteFiles(dbPath);
} }
} }
} }
tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/ZB.MOM.WW.ScadaBridge.IntegrationTests.csproj
+2 -1
View File
@@ -42,6 +42,7 @@
<ProjectReference Include="../../src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.csproj" /> <ProjectReference Include="../../src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.csproj" />
<ProjectReference Include="../../src/ZB.MOM.WW.ScadaBridge.ManagementService/ZB.MOM.WW.ScadaBridge.ManagementService.csproj" /> <ProjectReference Include="../../src/ZB.MOM.WW.ScadaBridge.ManagementService/ZB.MOM.WW.ScadaBridge.ManagementService.csproj" />
<ProjectReference Include="../../src/ZB.MOM.WW.ScadaBridge.SiteRuntime/ZB.MOM.WW.ScadaBridge.SiteRuntime.csproj" /> <ProjectReference Include="../../src/ZB.MOM.WW.ScadaBridge.SiteRuntime/ZB.MOM.WW.ScadaBridge.SiteRuntime.csproj" />
</ItemGroup> <ProjectReference Include="../ZB.MOM.WW.ScadaBridge.TestSupport/ZB.MOM.WW.ScadaBridge.TestSupport.csproj" />
</ItemGroup>
</Project> </Project>
tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerActorTests.cs
+105 -12
View File
@@ -2,6 +2,7 @@ using Akka.Actor;
using Akka.TestKit.Xunit2; using Akka.TestKit.Xunit2;
using Microsoft.Extensions.Logging; using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Logging.Abstractions; using Microsoft.Extensions.Logging.Abstractions;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment; using ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.DebugView; using ZB.MOM.WW.ScadaBridge.Commons.Messages.DebugView;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.InboundApi; using ZB.MOM.WW.ScadaBridge.Commons.Messages.InboundApi;
@@ -14,6 +15,7 @@ using ZB.MOM.WW.ScadaBridge.SiteRuntime.Deployment;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.TestSupport; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.TestSupport;
using ZB.MOM.WW.ScadaBridge.TestSupport;
using System.Text.Json; using System.Text.Json;
using System.Threading; using System.Threading;
@@ -28,13 +30,13 @@ public class DeploymentManagerActorTests : TestKit, IDisposable
private readonly SiteStorageService _storage; private readonly SiteStorageService _storage;
private readonly ScriptCompilationService _compilationService; private readonly ScriptCompilationService _compilationService;
private readonly SharedScriptLibrary _sharedScriptLibrary; private readonly SharedScriptLibrary _sharedScriptLibrary;
private readonly string _dbFile; private readonly TestLocalDb _localDb;
public DeploymentManagerActorTests() public DeploymentManagerActorTests()
{ {
_dbFile = Path.Combine(Path.GetTempPath(), $"dm-test-{Guid.NewGuid():N}.db"); _localDb = TestLocalDb.CreateTemp("dm-test");
_storage = new SiteStorageService( _storage = new SiteStorageService(
$"Data Source={_dbFile}", _localDb.Db,
NullLogger<SiteStorageService>.Instance); NullLogger<SiteStorageService>.Instance);
_storage.InitializeAsync().GetAwaiter().GetResult(); _storage.InitializeAsync().GetAwaiter().GetResult();
_compilationService = new ScriptCompilationService( _compilationService = new ScriptCompilationService(
@@ -45,8 +47,12 @@ public class DeploymentManagerActorTests : TestKit, IDisposable
void IDisposable.Dispose() void IDisposable.Dispose()
{ {
// TestKit teardown first, so no in-flight actor can reach a disposed ILocalDb;
// then dispose the database before deleting — the master connection anchors the WAL.
Shutdown(); Shutdown();
try { File.Delete(_dbFile); } catch { /* cleanup */ } var path = _localDb.Path;
_localDb.Dispose();
TestLocalDb.DeleteFiles(path);
} }
private IActorRef CreateDeploymentManager( private IActorRef CreateDeploymentManager(
@@ -61,12 +67,14 @@ public class DeploymentManagerActorTests : TestKit, IDisposable
null, // no stream manager in tests null, // no stream manager in tests
options, options,
NullLogger<DeploymentManagerActor>.Instance, NullLogger<DeploymentManagerActor>.Instance,
null, // Named from here on. These trailing parameters are all optional and several
null, // share a type, so a positional list silently binds the wrong argument when the
null, // signature changes — which is exactly what removing replicationActor did.
serviceProvider, dclManager: null,
null, healthCollector: null,
configFetcher))); serviceProvider: serviceProvider,
loggerFactory: null,
configFetcher: configFetcher)));
} }
private static string MakeConfigJson(string instanceName) private static string MakeConfigJson(string instanceName)
@@ -295,8 +303,11 @@ public class DeploymentManagerActorTests : TestKit, IDisposable
var dm = ActorOf(Props.Create(() => new DeploymentManagerActor( var dm = ActorOf(Props.Create(() => new DeploymentManagerActor(
_storage, _compilationService, _sharedScriptLibrary, null, _storage, _compilationService, _sharedScriptLibrary, null,
new SiteRuntimeOptions(), NullLogger<DeploymentManagerActor>.Instance, new SiteRuntimeOptions(), NullLogger<DeploymentManagerActor>.Instance,
null, null, null, null, null, null, // dclManager, healthCollector, serviceProvider, loggerFactory, configFetcher.
TimeSpan.FromMilliseconds(200), loader))); // Props.Create builds an expression tree, which rejects named arguments that
// are out of position, so the optional tail has to be padded positionally.
null, null, null, null, null,
startupLoadRetryInterval: TimeSpan.FromMilliseconds(200), configLoader: loader)));
AwaitAssert(() => AwaitAssert(() =>
{ {
@@ -878,6 +889,88 @@ public class DeploymentManagerActorTests : TestKit, IDisposable
Assert.Equal("SenderPump", response.InstanceUniqueName); Assert.Equal("SenderPump", response.InstanceUniqueName);
} }
// ── LocalDb Phase 2 / Task 12: the active-node central-only purge ──
[Fact]
public async Task ApplyingArtifacts_PurgesCentralOnlyNotificationConfig()
{
// Security cleanup. notification_lists and smtp_configurations can hold plaintext
// SMTP passwords written by a pre-2026-07-10 build, and the ACTIVE node's artifact
// apply is what clears them (DeploymentManagerActor.HandleDeployArtifacts). The
// standby used to hold a second copy of this call in SiteReplicationActor; LocalDb
// Phase 2 deleted that actor, so this is now the ONLY call site that keeps the
// tables empty.
//
// ArtifactStorageTests covers the storage method, not the actor's call to it, so
// without this test the call could be dropped and every suite would stay green.
// That is precisely the kind of silent security regression it exists to prevent —
// verified red-first by commenting out the call.
await SeedCentralOnlyRowsAsync();
Assert.Equal(1, await RowCountAsync("notification_lists"));
Assert.Equal(1, await RowCountAsync("smtp_configurations"));
var manager = CreateDeploymentManager();
manager.Tell(new DeployArtifactsCommand(
DeploymentId: "dep-purge-1",
SharedScripts: null,
ExternalSystems: null,
DatabaseConnections: null,
NotificationLists: null,
DataConnections: null,
SmtpConfigurations: null,
Timestamp: DateTimeOffset.UtcNow));
// The apply runs on a Task.Run inside the actor, so poll rather than assert once.
await AwaitPurgedAsync();
}
private async Task SeedCentralOnlyRowsAsync()
{
// Seeded through the service's own (already-open) LocalDb connection — a raw
// SqliteConnection would lack the pragmas and the zb_hlc_next() UDF the site
// tables' capture triggers call. Raw SQL is the only way these rows can exist at
// all now that the site-side write paths are gone.
await using var connection = _storage.CreateConnection();
await using var command = connection.CreateCommand();
command.CommandText = """
INSERT INTO notification_lists (name, recipient_emails, updated_at)
VALUES ('Ops Team', '["ops@example.com"]', @u);
INSERT INTO smtp_configurations
(name, server, port, auth_mode, from_address, username, password, oauth_config, updated_at)
VALUES ('smtp.example.com:587', 'smtp.example.com', 587, 'BasicAuth',
'noreply@example.com', 'smtpuser', 'PLAINTEXT-SECRET', NULL, @u);
""";
command.Parameters.AddWithValue("@u", DateTimeOffset.UtcNow.ToString("O"));
await command.ExecuteNonQueryAsync();
}
private async Task<long> RowCountAsync(string table)
{
await using var connection = _storage.CreateConnection();
await using var command = connection.CreateCommand();
command.CommandText = $"SELECT COUNT(*) FROM {table}";
return (long)(await command.ExecuteScalarAsync())!;
}
private async Task AwaitPurgedAsync()
{
var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(10);
while (DateTime.UtcNow < deadline)
{
if (await RowCountAsync("notification_lists") == 0 &&
await RowCountAsync("smtp_configurations") == 0)
{
return;
}
await Task.Delay(50);
}
Assert.Fail(
"Artifact apply did not purge the central-only notification/SMTP rows. " +
"The plaintext SMTP password is still on disk.");
}
/// <summary> /// <summary>
/// In-test fake <see cref="IDeploymentConfigFetcher"/>: returns a canned config JSON /// In-test fake <see cref="IDeploymentConfigFetcher"/>: returns a canned config JSON
/// (notify-and-fetch success) or throws a canned exception (fetch failure), and records /// (notify-and-fetch success) or throws a canned exception (fetch failure), and records
tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerCertReconcileTests.cs
+10 -6
View File
@@ -6,6 +6,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Messages.Management;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts;
using ZB.MOM.WW.ScadaBridge.TestSupport;
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors;
@@ -46,13 +47,13 @@ akka {
private readonly SiteStorageService _storage; private readonly SiteStorageService _storage;
private readonly ScriptCompilationService _compilationService; private readonly ScriptCompilationService _compilationService;
private readonly SharedScriptLibrary _sharedScriptLibrary; private readonly SharedScriptLibrary _sharedScriptLibrary;
private readonly string _dbFile; private readonly TestLocalDb _localDb;
public DeploymentManagerCertReconcileTests() : base(ClusterConfig, "dm-cert-reconcile") public DeploymentManagerCertReconcileTests() : base(ClusterConfig, "dm-cert-reconcile")
{ {
_dbFile = Path.Combine(Path.GetTempPath(), $"dm-cert-test-{Guid.NewGuid():N}.db"); _localDb = TestLocalDb.CreateTemp("dm-cert-test");
_storage = new SiteStorageService( _storage = new SiteStorageService(
$"Data Source={_dbFile}", NullLogger<SiteStorageService>.Instance); _localDb.Db, NullLogger<SiteStorageService>.Instance);
_storage.InitializeAsync().GetAwaiter().GetResult(); _storage.InitializeAsync().GetAwaiter().GetResult();
_compilationService = new ScriptCompilationService( _compilationService = new ScriptCompilationService(
NullLogger<ScriptCompilationService>.Instance); NullLogger<ScriptCompilationService>.Instance);
@@ -62,8 +63,12 @@ akka {
void IDisposable.Dispose() void IDisposable.Dispose()
{ {
// TestKit teardown first, so no in-flight actor can reach a disposed ILocalDb;
// then dispose the database before deleting — the master connection anchors the WAL.
Shutdown(); Shutdown();
try { File.Delete(_dbFile); } catch { /* cleanup */ } var path = _localDb.Path;
_localDb.Dispose();
TestLocalDb.DeleteFiles(path);
} }
/// <summary>Forwards every message it receives to a probe, preserving the original sender.</summary> /// <summary>Forwards every message it receives to a probe, preserving the original sender.</summary>
@@ -75,8 +80,7 @@ akka {
private IActorRef CreateDeploymentManager() => private IActorRef CreateDeploymentManager() =>
ActorOf(Props.Create(() => new DeploymentManagerActor( ActorOf(Props.Create(() => new DeploymentManagerActor(
_storage, _compilationService, _sharedScriptLibrary, _storage, _compilationService, _sharedScriptLibrary,
null, new SiteRuntimeOptions(), NullLogger<DeploymentManagerActor>.Instance, null, new SiteRuntimeOptions(), NullLogger<DeploymentManagerActor>.Instance)));
null, null, null, null, null, null, null, null)));
[Fact] [Fact]
public void SiteNodeJoined_PushesLocalTrustedCertsToJoinedNode() public void SiteNodeJoined_PushesLocalTrustedCertsToJoinedNode()
tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerLoggerFactoryTests.cs
+13 -9
View File
@@ -8,6 +8,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Types.Flattening;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts;
using ZB.MOM.WW.ScadaBridge.TestSupport;
using System.Text.Json; using System.Text.Json;
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors;
@@ -23,13 +24,13 @@ public class DeploymentManagerLoggerFactoryTests : TestKit, IDisposable
private readonly SiteStorageService _storage; private readonly SiteStorageService _storage;
private readonly ScriptCompilationService _compilationService; private readonly ScriptCompilationService _compilationService;
private readonly SharedScriptLibrary _sharedScriptLibrary; private readonly SharedScriptLibrary _sharedScriptLibrary;
private readonly string _dbFile; private readonly TestLocalDb _localDb;
public DeploymentManagerLoggerFactoryTests() public DeploymentManagerLoggerFactoryTests()
{ {
_dbFile = Path.Combine(Path.GetTempPath(), $"dm-loggerfactory-test-{Guid.NewGuid():N}.db"); _localDb = TestLocalDb.CreateTemp("dm-loggerfactory-test");
_storage = new SiteStorageService( _storage = new SiteStorageService(
$"Data Source={_dbFile}", _localDb.Db,
NullLogger<SiteStorageService>.Instance); NullLogger<SiteStorageService>.Instance);
_storage.InitializeAsync().GetAwaiter().GetResult(); _storage.InitializeAsync().GetAwaiter().GetResult();
_compilationService = new ScriptCompilationService( _compilationService = new ScriptCompilationService(
@@ -40,8 +41,12 @@ public class DeploymentManagerLoggerFactoryTests : TestKit, IDisposable
void IDisposable.Dispose() void IDisposable.Dispose()
{ {
// TestKit teardown first, so no in-flight actor can reach a disposed ILocalDb;
// then dispose the database before deleting — the master connection anchors the WAL.
Shutdown(); Shutdown();
try { File.Delete(_dbFile); } catch { /* cleanup */ } var path = _localDb.Path;
_localDb.Dispose();
TestLocalDb.DeleteFiles(path);
} }
private static string MakeConfigJson(string instanceName) private static string MakeConfigJson(string instanceName)
@@ -98,11 +103,10 @@ public class DeploymentManagerLoggerFactoryTests : TestKit, IDisposable
null, null,
new SiteRuntimeOptions { StartupBatchSize = 100, StartupBatchDelayMs = 5 }, new SiteRuntimeOptions { StartupBatchSize = 100, StartupBatchDelayMs = 5 },
NullLogger<DeploymentManagerActor>.Instance, NullLogger<DeploymentManagerActor>.Instance,
null, // dclManager, healthCollector, serviceProvider — padded positionally because
null, // Props.Create is an expression tree and rejects out-of-position named args.
null, null, null, null,
null, loggerFactory: loggerFactory)));
loggerFactory)));
// Allow async startup (load configs + staggered creation). // Allow async startup (load configs + staggered creation).
await Task.Delay(2000); await Task.Delay(2000);
tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerMediumFindingsTests.cs
+35 -14
View File
@@ -1,12 +1,14 @@
using Akka.Actor; using Akka.Actor;
using Akka.TestKit.Xunit2; using Akka.TestKit.Xunit2;
using Microsoft.Extensions.Logging.Abstractions; using Microsoft.Extensions.Logging.Abstractions;
using ZB.MOM.WW.LocalDb;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment; using ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Flattening; using ZB.MOM.WW.ScadaBridge.Commons.Types.Flattening;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts;
using ZB.MOM.WW.ScadaBridge.TestSupport;
using System.Text.Json; using System.Text.Json;
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors;
@@ -20,11 +22,11 @@ public class DeploymentManagerMediumFindingsTests : TestKit, IDisposable
{ {
private readonly ScriptCompilationService _compilationService; private readonly ScriptCompilationService _compilationService;
private readonly SharedScriptLibrary _sharedScriptLibrary; private readonly SharedScriptLibrary _sharedScriptLibrary;
private readonly string _dbFile; private readonly TestLocalDb _localDb;
public DeploymentManagerMediumFindingsTests() public DeploymentManagerMediumFindingsTests()
{ {
_dbFile = Path.Combine(Path.GetTempPath(), $"dm-medium-test-{Guid.NewGuid():N}.db"); _localDb = TestLocalDb.CreateTemp("dm-medium-test");
_compilationService = new ScriptCompilationService( _compilationService = new ScriptCompilationService(
NullLogger<ScriptCompilationService>.Instance); NullLogger<ScriptCompilationService>.Instance);
_sharedScriptLibrary = new SharedScriptLibrary( _sharedScriptLibrary = new SharedScriptLibrary(
@@ -33,12 +35,16 @@ public class DeploymentManagerMediumFindingsTests : TestKit, IDisposable
void IDisposable.Dispose() void IDisposable.Dispose()
{ {
// TestKit teardown first, so no in-flight actor can reach a disposed ILocalDb;
// then dispose the database before deleting — the master connection anchors the WAL.
Shutdown(); Shutdown();
try { File.Delete(_dbFile); } catch { /* cleanup */ } var path = _localDb.Path;
_localDb.Dispose();
TestLocalDb.DeleteFiles(path);
} }
private SiteStorageService NewStorage(string connectionString) private SiteStorageService NewStorage(ILocalDb localDb)
=> new(connectionString, NullLogger<SiteStorageService>.Instance); => new(localDb, NullLogger<SiteStorageService>.Instance);
private IActorRef CreateDeploymentManager(SiteStorageService storage, IActorRef? dclManager = null) private IActorRef CreateDeploymentManager(SiteStorageService storage, IActorRef? dclManager = null)
{ {
@@ -100,11 +106,19 @@ public class DeploymentManagerMediumFindingsTests : TestKit, IDisposable
[Fact] [Fact]
public async Task Deploy_PersistenceFailure_ReportsFailedNotSuccess() public async Task Deploy_PersistenceFailure_ReportsFailedNotSuccess()
{ {
// A connection string pointing at an unwritable path makes every storage // A storage over a database whose site tables were never created makes every
// write throw, so StoreDeployedConfigAsync fails. // storage operation throw ("no such table"), so StoreDeployedConfigAsync fails.
var badPath = Path.Combine( //
Path.GetTempPath(), $"no-such-dir-{Guid.NewGuid():N}", "site.db"); // This replaces the old "connection string pointing at an unwritable path" trick:
var storage = NewStorage($"Data Source={badPath}"); // the service now takes an ILocalDb rather than a connection string, and LocalDb
// opens its file eagerly in its own constructor — so an unopenable path fails
// while building the fixture, before there is a storage to hand the actor at all.
// Skipping InitializeAsync is the equivalent lever on the new seam, and fails the
// same way the old one did: reads AND writes both throw.
var uninitialized = TestLocalDb.CreateTemp("dm-medium-persist-fail");
try
{
var storage = NewStorage(uninitialized.Db);
var actor = CreateDeploymentManager(storage); var actor = CreateDeploymentManager(storage);
await Task.Delay(500); // empty startup await Task.Delay(500); // empty startup
@@ -117,6 +131,13 @@ public class DeploymentManagerMediumFindingsTests : TestKit, IDisposable
Assert.Equal(DeploymentStatus.Failed, response.Status); Assert.Equal(DeploymentStatus.Failed, response.Status);
Assert.False(string.IsNullOrEmpty(response.ErrorMessage)); Assert.False(string.IsNullOrEmpty(response.ErrorMessage));
} }
finally
{
var path = uninitialized.Path;
uninitialized.Dispose();
TestLocalDb.DeleteFiles(path);
}
}
/// <summary> /// <summary>
/// SiteRuntime-005: a successful deployment must still report /// SiteRuntime-005: a successful deployment must still report
@@ -126,7 +147,7 @@ public class DeploymentManagerMediumFindingsTests : TestKit, IDisposable
[Fact] [Fact]
public async Task Deploy_Success_ReportsSuccessAndPersistsConfig() public async Task Deploy_Success_ReportsSuccessAndPersistsConfig()
{ {
var storage = NewStorage($"Data Source={_dbFile}"); var storage = NewStorage(_localDb.Db);
await storage.InitializeAsync(); await storage.InitializeAsync();
var actor = CreateDeploymentManager(storage); var actor = CreateDeploymentManager(storage);
@@ -152,7 +173,7 @@ public class DeploymentManagerMediumFindingsTests : TestKit, IDisposable
[Fact] [Fact]
public async Task EnsureDclConnections_ConnectionConfigChanged_ReissuesCreateCommand() public async Task EnsureDclConnections_ConnectionConfigChanged_ReissuesCreateCommand()
{ {
var storage = NewStorage($"Data Source={_dbFile}"); var storage = NewStorage(_localDb.Db);
await storage.InitializeAsync(); await storage.InitializeAsync();
var dcl = CreateTestProbe(); var dcl = CreateTestProbe();
@@ -191,7 +212,7 @@ public class DeploymentManagerMediumFindingsTests : TestKit, IDisposable
[Fact] [Fact]
public async Task EnsureDclConnections_UnchangedConfig_DoesNotReissueCreateCommand() public async Task EnsureDclConnections_UnchangedConfig_DoesNotReissueCreateCommand()
{ {
var storage = NewStorage($"Data Source={_dbFile}"); var storage = NewStorage(_localDb.Db);
await storage.InitializeAsync(); await storage.InitializeAsync();
var dcl = CreateTestProbe(); var dcl = CreateTestProbe();
@@ -223,7 +244,7 @@ public class DeploymentManagerMediumFindingsTests : TestKit, IDisposable
[Fact] [Fact]
public async Task Startup_WithSharedScripts_LoadsConfigsAndStaysResponsive() public async Task Startup_WithSharedScripts_LoadsConfigsAndStaysResponsive()
{ {
var storage = NewStorage($"Data Source={_dbFile}"); var storage = NewStorage(_localDb.Db);
await storage.InitializeAsync(); await storage.InitializeAsync();
// Several shared scripts to compile during startup. // Several shared scripts to compile during startup.
tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/DeploymentManagerRedeployTests.cs
+13 -7
View File
@@ -11,6 +11,7 @@ using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.TestSupport; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.TestSupport;
using ZB.MOM.WW.ScadaBridge.TestSupport;
using System.Text.Json; using System.Text.Json;
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors;
@@ -25,13 +26,13 @@ public class DeploymentManagerRedeployTests : TestKit, IDisposable
private readonly SiteStorageService _storage; private readonly SiteStorageService _storage;
private readonly ScriptCompilationService _compilationService; private readonly ScriptCompilationService _compilationService;
private readonly SharedScriptLibrary _sharedScriptLibrary; private readonly SharedScriptLibrary _sharedScriptLibrary;
private readonly string _dbFile; private readonly TestLocalDb _localDb;
public DeploymentManagerRedeployTests() public DeploymentManagerRedeployTests()
{ {
_dbFile = Path.Combine(Path.GetTempPath(), $"dm-redeploy-test-{Guid.NewGuid():N}.db"); _localDb = TestLocalDb.CreateTemp("dm-redeploy-test");
_storage = new SiteStorageService( _storage = new SiteStorageService(
$"Data Source={_dbFile}", _localDb.Db,
NullLogger<SiteStorageService>.Instance); NullLogger<SiteStorageService>.Instance);
_storage.InitializeAsync().GetAwaiter().GetResult(); _storage.InitializeAsync().GetAwaiter().GetResult();
_compilationService = new ScriptCompilationService( _compilationService = new ScriptCompilationService(
@@ -42,8 +43,12 @@ public class DeploymentManagerRedeployTests : TestKit, IDisposable
void IDisposable.Dispose() void IDisposable.Dispose()
{ {
// TestKit teardown first, so no in-flight actor can reach a disposed ILocalDb;
// then dispose the database before deleting — the master connection anchors the WAL.
Shutdown(); Shutdown();
try { File.Delete(_dbFile); } catch { /* cleanup */ } var path = _localDb.Path;
_localDb.Dispose();
TestLocalDb.DeleteFiles(path);
} }
private IActorRef CreateDeploymentManager( private IActorRef CreateDeploymentManager(
@@ -56,10 +61,11 @@ public class DeploymentManagerRedeployTests : TestKit, IDisposable
null, null,
new SiteRuntimeOptions(), new SiteRuntimeOptions(),
NullLogger<DeploymentManagerActor>.Instance, NullLogger<DeploymentManagerActor>.Instance,
// dclManager — padded positionally because Props.Create is an expression tree
// and rejects out-of-position named args.
null, null,
null, healthCollector: healthCollector,
healthCollector, serviceProvider: serviceProvider)));
serviceProvider)));
} }
/// <summary> /// <summary>
tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorChildAttributeRaceTests.cs
+9 -4
View File
@@ -7,6 +7,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Types.Flattening;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts;
using ZB.MOM.WW.ScadaBridge.TestSupport;
using System.Reflection; using System.Reflection;
using System.Text.Json; using System.Text.Json;
@@ -39,13 +40,13 @@ public class InstanceActorChildAttributeRaceTests : TestKit, IDisposable
private readonly ScriptCompilationService _compilationService; private readonly ScriptCompilationService _compilationService;
private readonly SharedScriptLibrary _sharedScriptLibrary; private readonly SharedScriptLibrary _sharedScriptLibrary;
private readonly SiteRuntimeOptions _options; private readonly SiteRuntimeOptions _options;
private readonly string _dbFile; private readonly TestLocalDb _localDb;
public InstanceActorChildAttributeRaceTests() public InstanceActorChildAttributeRaceTests()
{ {
_dbFile = Path.Combine(Path.GetTempPath(), $"instance-race-test-{Guid.NewGuid():N}.db"); _localDb = TestLocalDb.CreateTemp("instance-race-test");
_storage = new SiteStorageService( _storage = new SiteStorageService(
$"Data Source={_dbFile}", _localDb.Db,
NullLogger<SiteStorageService>.Instance); NullLogger<SiteStorageService>.Instance);
_storage.InitializeAsync().GetAwaiter().GetResult(); _storage.InitializeAsync().GetAwaiter().GetResult();
_compilationService = new ScriptCompilationService( _compilationService = new ScriptCompilationService(
@@ -61,8 +62,12 @@ public class InstanceActorChildAttributeRaceTests : TestKit, IDisposable
void IDisposable.Dispose() void IDisposable.Dispose()
{ {
// TestKit teardown first, so no in-flight actor can reach a disposed ILocalDb;
// then dispose the database before deleting — the master connection anchors the WAL.
Shutdown(); Shutdown();
try { File.Delete(_dbFile); } catch { /* cleanup */ } var path = _localDb.Path;
_localDb.Dispose();
TestLocalDb.DeleteFiles(path);
} }
private static FlattenedConfiguration BuildConfig(string instanceName) private static FlattenedConfiguration BuildConfig(string instanceName)
tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorChildRoutingTests.cs
+9 -4
View File
@@ -5,6 +5,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Types.Flattening;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts;
using ZB.MOM.WW.ScadaBridge.TestSupport;
using System.Text.Json; using System.Text.Json;
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors;
@@ -22,13 +23,13 @@ public class InstanceActorChildRoutingTests : TestKit, IDisposable
private readonly ScriptCompilationService _compilationService; private readonly ScriptCompilationService _compilationService;
private readonly SharedScriptLibrary _sharedScriptLibrary; private readonly SharedScriptLibrary _sharedScriptLibrary;
private readonly SiteRuntimeOptions _options; private readonly SiteRuntimeOptions _options;
private readonly string _dbFile; private readonly TestLocalDb _localDb;
public InstanceActorChildRoutingTests() public InstanceActorChildRoutingTests()
{ {
_dbFile = Path.Combine(Path.GetTempPath(), $"instance-routing-test-{Guid.NewGuid():N}.db"); _localDb = TestLocalDb.CreateTemp("instance-routing-test");
_storage = new SiteStorageService( _storage = new SiteStorageService(
$"Data Source={_dbFile}", _localDb.Db,
NullLogger<SiteStorageService>.Instance); NullLogger<SiteStorageService>.Instance);
_storage.InitializeAsync().GetAwaiter().GetResult(); _storage.InitializeAsync().GetAwaiter().GetResult();
_compilationService = new ScriptCompilationService( _compilationService = new ScriptCompilationService(
@@ -44,8 +45,12 @@ public class InstanceActorChildRoutingTests : TestKit, IDisposable
void IDisposable.Dispose() void IDisposable.Dispose()
{ {
// TestKit teardown first, so no in-flight actor can reach a disposed ILocalDb;
// then dispose the database before deleting — the master connection anchors the WAL.
Shutdown(); Shutdown();
try { File.Delete(_dbFile); } catch { /* cleanup */ } var path = _localDb.Path;
_localDb.Dispose();
TestLocalDb.DeleteFiles(path);
} }
private static FlattenedConfiguration ScriptsAB_and_Expression(string instanceName) private static FlattenedConfiguration ScriptsAB_and_Expression(string instanceName)
tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorIntegrationTests.cs
+9 -4
View File
@@ -9,6 +9,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Types.Flattening;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts;
using ZB.MOM.WW.ScadaBridge.TestSupport;
using System.Text.Json; using System.Text.Json;
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors;
@@ -22,13 +23,13 @@ public class InstanceActorIntegrationTests : TestKit, IDisposable
private readonly ScriptCompilationService _compilationService; private readonly ScriptCompilationService _compilationService;
private readonly SharedScriptLibrary _sharedScriptLibrary; private readonly SharedScriptLibrary _sharedScriptLibrary;
private readonly SiteRuntimeOptions _options; private readonly SiteRuntimeOptions _options;
private readonly string _dbFile; private readonly TestLocalDb _localDb;
public InstanceActorIntegrationTests() public InstanceActorIntegrationTests()
{ {
_dbFile = Path.Combine(Path.GetTempPath(), $"instance-int-test-{Guid.NewGuid():N}.db"); _localDb = TestLocalDb.CreateTemp("instance-int-test");
_storage = new SiteStorageService( _storage = new SiteStorageService(
$"Data Source={_dbFile}", _localDb.Db,
NullLogger<SiteStorageService>.Instance); NullLogger<SiteStorageService>.Instance);
_storage.InitializeAsync().GetAwaiter().GetResult(); _storage.InitializeAsync().GetAwaiter().GetResult();
_compilationService = new ScriptCompilationService( _compilationService = new ScriptCompilationService(
@@ -45,8 +46,12 @@ public class InstanceActorIntegrationTests : TestKit, IDisposable
void IDisposable.Dispose() void IDisposable.Dispose()
{ {
// TestKit teardown first, so no in-flight actor can reach a disposed ILocalDb;
// then dispose the database before deleting — the master connection anchors the WAL.
Shutdown(); Shutdown();
try { File.Delete(_dbFile); } catch { /* cleanup */ } var path = _localDb.Path;
_localDb.Dispose();
TestLocalDb.DeleteFiles(path);
} }
private IActorRef CreateInstanceWithScripts( private IActorRef CreateInstanceWithScripts(
tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorNativeAlarmTests.cs
+9 -4
View File
@@ -12,6 +12,7 @@ using ZB.MOM.WW.ScadaBridge.SiteRuntime;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts;
using ZB.MOM.WW.ScadaBridge.TestSupport;
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors;
@@ -25,12 +26,12 @@ public class InstanceActorNativeAlarmTests : TestKit, IDisposable
private readonly ScriptCompilationService _compilationService; private readonly ScriptCompilationService _compilationService;
private readonly SharedScriptLibrary _sharedScriptLibrary; private readonly SharedScriptLibrary _sharedScriptLibrary;
private readonly SiteRuntimeOptions _options = new(); private readonly SiteRuntimeOptions _options = new();
private readonly string _dbFile; private readonly TestLocalDb _localDb;
public InstanceActorNativeAlarmTests() public InstanceActorNativeAlarmTests()
{ {
_dbFile = Path.Combine(Path.GetTempPath(), $"instance-native-{Guid.NewGuid():N}.db"); _localDb = TestLocalDb.CreateTemp("instance-native");
_storage = new SiteStorageService($"Data Source={_dbFile}", NullLogger<SiteStorageService>.Instance); _storage = new SiteStorageService(_localDb.Db, NullLogger<SiteStorageService>.Instance);
_storage.InitializeAsync().GetAwaiter().GetResult(); _storage.InitializeAsync().GetAwaiter().GetResult();
_compilationService = new ScriptCompilationService(NullLogger<ScriptCompilationService>.Instance); _compilationService = new ScriptCompilationService(NullLogger<ScriptCompilationService>.Instance);
_sharedScriptLibrary = new SharedScriptLibrary(_compilationService, NullLogger<SharedScriptLibrary>.Instance); _sharedScriptLibrary = new SharedScriptLibrary(_compilationService, NullLogger<SharedScriptLibrary>.Instance);
@@ -149,7 +150,11 @@ public class InstanceActorNativeAlarmTests : TestKit, IDisposable
void IDisposable.Dispose() void IDisposable.Dispose()
{ {
// TestKit teardown first, so no in-flight actor can reach a disposed ILocalDb;
// then dispose the database before deleting — the master connection anchors the WAL.
Shutdown(); Shutdown();
try { File.Delete(_dbFile); } catch { /* cleanup */ } var path = _localDb.Path;
_localDb.Dispose();
TestLocalDb.DeleteFiles(path);
} }
} }
tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorSetAttributeTests.cs
+9 -4
View File
@@ -8,6 +8,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Types.Flattening;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts;
using ZB.MOM.WW.ScadaBridge.TestSupport;
using System.Text.Json; using System.Text.Json;
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors;
@@ -23,13 +24,13 @@ public class InstanceActorSetAttributeTests : TestKit, IDisposable
private readonly ScriptCompilationService _compilationService; private readonly ScriptCompilationService _compilationService;
private readonly SharedScriptLibrary _sharedScriptLibrary; private readonly SharedScriptLibrary _sharedScriptLibrary;
private readonly SiteRuntimeOptions _options; private readonly SiteRuntimeOptions _options;
private readonly string _dbFile; private readonly TestLocalDb _localDb;
public InstanceActorSetAttributeTests() public InstanceActorSetAttributeTests()
{ {
_dbFile = Path.Combine(Path.GetTempPath(), $"instance-setattr-test-{Guid.NewGuid():N}.db"); _localDb = TestLocalDb.CreateTemp("instance-setattr-test");
_storage = new SiteStorageService( _storage = new SiteStorageService(
$"Data Source={_dbFile}", _localDb.Db,
NullLogger<SiteStorageService>.Instance); NullLogger<SiteStorageService>.Instance);
_storage.InitializeAsync().GetAwaiter().GetResult(); _storage.InitializeAsync().GetAwaiter().GetResult();
_compilationService = new ScriptCompilationService( _compilationService = new ScriptCompilationService(
@@ -41,8 +42,12 @@ public class InstanceActorSetAttributeTests : TestKit, IDisposable
void IDisposable.Dispose() void IDisposable.Dispose()
{ {
// TestKit teardown first, so no in-flight actor can reach a disposed ILocalDb;
// then dispose the database before deleting — the master connection anchors the WAL.
Shutdown(); Shutdown();
try { File.Delete(_dbFile); } catch { /* cleanup */ } var path = _localDb.Path;
_localDb.Dispose();
TestLocalDb.DeleteFiles(path);
} }
private IActorRef CreateInstanceActor(string instanceName, FlattenedConfiguration config, IActorRef? dclManager) private IActorRef CreateInstanceActor(string instanceName, FlattenedConfiguration config, IActorRef? dclManager)
tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorTests.cs
+9 -4
View File
@@ -12,6 +12,7 @@ using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.TestSupport; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.TestSupport;
using ZB.MOM.WW.ScadaBridge.TestSupport;
using System.Text.Json; using System.Text.Json;
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors;
@@ -25,13 +26,13 @@ public class InstanceActorTests : TestKit, IDisposable
private readonly ScriptCompilationService _compilationService; private readonly ScriptCompilationService _compilationService;
private readonly SharedScriptLibrary _sharedScriptLibrary; private readonly SharedScriptLibrary _sharedScriptLibrary;
private readonly SiteRuntimeOptions _options; private readonly SiteRuntimeOptions _options;
private readonly string _dbFile; private readonly TestLocalDb _localDb;
public InstanceActorTests() public InstanceActorTests()
{ {
_dbFile = Path.Combine(Path.GetTempPath(), $"instance-actor-test-{Guid.NewGuid():N}.db"); _localDb = TestLocalDb.CreateTemp("instance-actor-test");
_storage = new SiteStorageService( _storage = new SiteStorageService(
$"Data Source={_dbFile}", _localDb.Db,
NullLogger<SiteStorageService>.Instance); NullLogger<SiteStorageService>.Instance);
_storage.InitializeAsync().GetAwaiter().GetResult(); _storage.InitializeAsync().GetAwaiter().GetResult();
_compilationService = new ScriptCompilationService( _compilationService = new ScriptCompilationService(
@@ -56,8 +57,12 @@ public class InstanceActorTests : TestKit, IDisposable
void IDisposable.Dispose() void IDisposable.Dispose()
{ {
// TestKit teardown first, so no in-flight actor can reach a disposed ILocalDb;
// then dispose the database before deleting — the master connection anchors the WAL.
Shutdown(); Shutdown();
try { File.Delete(_dbFile); } catch { /* cleanup */ } var path = _localDb.Path;
_localDb.Dispose();
TestLocalDb.DeleteFiles(path);
} }
// ── M1.6: site event log `instance_lifecycle` category ────────────────── // ── M1.6: site event log `instance_lifecycle` category ──────────────────
tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/InstanceActorWaitForAttributeTests.cs
+9 -4
View File
@@ -10,6 +10,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Types.Flattening;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts;
using ZB.MOM.WW.ScadaBridge.TestSupport;
using System.Text.Json; using System.Text.Json;
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors;
@@ -26,13 +27,13 @@ public class InstanceActorWaitForAttributeTests : TestKit, IDisposable
private readonly ScriptCompilationService _compilationService; private readonly ScriptCompilationService _compilationService;
private readonly SharedScriptLibrary _sharedScriptLibrary; private readonly SharedScriptLibrary _sharedScriptLibrary;
private readonly SiteRuntimeOptions _options; private readonly SiteRuntimeOptions _options;
private readonly string _dbFile; private readonly TestLocalDb _localDb;
public InstanceActorWaitForAttributeTests() public InstanceActorWaitForAttributeTests()
{ {
_dbFile = Path.Combine(Path.GetTempPath(), $"instance-waitfor-test-{Guid.NewGuid():N}.db"); _localDb = TestLocalDb.CreateTemp("instance-waitfor-test");
_storage = new SiteStorageService( _storage = new SiteStorageService(
$"Data Source={_dbFile}", _localDb.Db,
NullLogger<SiteStorageService>.Instance); NullLogger<SiteStorageService>.Instance);
_storage.InitializeAsync().GetAwaiter().GetResult(); _storage.InitializeAsync().GetAwaiter().GetResult();
_compilationService = new ScriptCompilationService( _compilationService = new ScriptCompilationService(
@@ -57,8 +58,12 @@ public class InstanceActorWaitForAttributeTests : TestKit, IDisposable
void IDisposable.Dispose() void IDisposable.Dispose()
{ {
// TestKit teardown first, so no in-flight actor can reach a disposed ILocalDb;
// then dispose the database before deleting — the master connection anchors the WAL.
Shutdown(); Shutdown();
try { File.Delete(_dbFile); } catch { /* cleanup */ } var path = _localDb.Path;
_localDb.Dispose();
TestLocalDb.DeleteFiles(path);
} }
// ── 1. Fast-path: attribute already at target ──────────────────────────── // ── 1. Fast-path: attribute already at target ────────────────────────────
tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/NativeAlarmActorTests.cs
+13 -7
View File
@@ -12,6 +12,7 @@ using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Messages; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Messages;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.TestSupport; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.TestSupport;
using ZB.MOM.WW.ScadaBridge.TestSupport;
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors;
@@ -21,14 +22,16 @@ namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors;
/// </summary> /// </summary>
public class NativeAlarmActorTests : TestKit, IDisposable public class NativeAlarmActorTests : TestKit, IDisposable
{ {
private readonly string _dbFile; private readonly TestLocalDb _localDb;
private readonly SiteStorageService _storage; private readonly SiteStorageService _storage;
private readonly SiteRuntimeOptions _options = new(); private readonly SiteRuntimeOptions _options = new();
public NativeAlarmActorTests() public NativeAlarmActorTests()
{ {
_dbFile = Path.Combine(Path.GetTempPath(), $"naa-{Guid.NewGuid():N}.db"); // SiteStorageService takes an ILocalDb now; LocalDb has no in-memory mode, so the
_storage = new SiteStorageService($"Data Source={_dbFile}", NullLogger<SiteStorageService>.Instance); // fixture is a real temp file (deleted in Dispose, after the TestKit shutdown).
_localDb = TestLocalDb.CreateTemp("naa");
_storage = new SiteStorageService(_localDb.Db, NullLogger<SiteStorageService>.Instance);
_storage.InitializeAsync().GetAwaiter().GetResult(); _storage.InitializeAsync().GetAwaiter().GetResult();
} }
@@ -442,11 +445,14 @@ public class NativeAlarmActorTests : TestKit, IDisposable
void IDisposable.Dispose() void IDisposable.Dispose()
{ {
// Shut the actor system down FIRST: in-flight alarm actors still hold the
// ILocalDb, and their coalesced flush would hit a disposed database otherwise.
Shutdown(); Shutdown();
if (File.Exists(_dbFile)) // Then dispose — the master connection anchors the WAL, so the sidecars cannot
{ // be removed while it is open.
File.Delete(_dbFile); var path = _localDb.Path;
} _localDb.Dispose();
TestLocalDb.DeleteFiles(path);
} }
[Fact] [Fact]
tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/SiteReconciliationActorTests.cs
+13 -4
View File
@@ -6,6 +6,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Messages.Deployment;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Deployment; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Deployment;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
using ZB.MOM.WW.ScadaBridge.TestSupport;
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors;
@@ -22,20 +23,28 @@ public class SiteReconciliationActorTests : TestKit, IDisposable
private const string NodeId = "node-a"; private const string NodeId = "node-a";
private readonly SiteStorageService _storage; private readonly SiteStorageService _storage;
private readonly string _dbFile; private readonly TestLocalDb _localDb;
public SiteReconciliationActorTests() public SiteReconciliationActorTests()
{ {
_dbFile = Path.Combine(Path.GetTempPath(), $"site-reconcile-test-{Guid.NewGuid():N}.db"); // SiteStorageService takes an ILocalDb now; LocalDb has no in-memory mode, so the
// fixture is a real temp file (deleted in Dispose, after the TestKit shutdown).
_localDb = TestLocalDb.CreateTemp("site-reconcile-test");
_storage = new SiteStorageService( _storage = new SiteStorageService(
$"Data Source={_dbFile}", Microsoft.Extensions.Logging.Abstractions.NullLogger<SiteStorageService>.Instance); _localDb.Db, Microsoft.Extensions.Logging.Abstractions.NullLogger<SiteStorageService>.Instance);
_storage.InitializeAsync().GetAwaiter().GetResult(); _storage.InitializeAsync().GetAwaiter().GetResult();
} }
void IDisposable.Dispose() void IDisposable.Dispose()
{ {
// Shut the actor system down FIRST: a reconcile continuation may still be
// writing through the ILocalDb, which must outlive the actors.
Shutdown(); Shutdown();
try { File.Delete(_dbFile); } catch { /* cleanup */ } // Then dispose — the master connection anchors the WAL, so the sidecars cannot
// be removed while it is open.
var path = _localDb.Path;
_localDb.Dispose();
TestLocalDb.DeleteFiles(path);
} }
private IActorRef CreateReconciliationActor( private IActorRef CreateReconciliationActor(
tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/SiteReplicationActorTests.cs
-559
View File
@@ -1,559 +0,0 @@
using System.Collections.Concurrent;
using System.Diagnostics.Metrics;
using Akka.Actor;
using Akka.TestKit.Xunit2;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Logging.Abstractions;
using ZB.MOM.WW.ScadaBridge.SiteRuntime;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Actors;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Deployment;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Messages;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
using ZB.MOM.WW.ScadaBridge.StoreAndForward;
using ZB.MOM.WW.ScadaBridge.Commons.Observability;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors;
/// <summary>
/// Tests for <see cref="SiteReplicationActor"/>'s notify-and-fetch config replication:
/// the active node now replicates an id-only <see cref="ReplicateConfigDeploy"/> (no inline
/// config JSON — killing the intra-site 128 KB frame trap), and the standby fetches the
/// config from central over HTTP and writes it with the older-write guard.
/// </summary>
public class SiteReplicationActorTests : TestKit, IDisposable
{
// Cluster provider is required because SiteReplicationActor calls Cluster.Get in its ctor
// and subscribes to cluster events in PreStart. We use the in-memory TestTransport (not
// dot-netty) so no real socket is bound and no DNS lookup happens — the actor only needs
// the cluster extension to load; these tests never form a real two-node cluster.
private const string ClusterConfig = @"
akka {
actor { provider = cluster }
remote {
enabled-transports = [""akka.remote.test""]
test {
transport-class = ""Akka.Remote.Transport.TestTransport, Akka.Remote""
applied-adapters = []
registry-key = site-repl-test
local-address = ""test://site-repl@localhost:1""
maximum-payload-bytes = 128000b
scheme-identifier = test
}
}
cluster { roles = [""site-test""] }
loglevel = WARNING
}";
private const string SiteRole = "site-test";
private readonly SiteStorageService _storage;
private readonly StoreAndForwardStorage _sfStorage;
private readonly ReplicationService _replicationService;
private readonly string _dbFile;
private readonly string _sfDbFile;
public SiteReplicationActorTests() : base(ClusterConfig, "site-repl")
{
_dbFile = Path.Combine(Path.GetTempPath(), $"site-repl-test-{Guid.NewGuid():N}.db");
_sfDbFile = Path.Combine(Path.GetTempPath(), $"site-repl-sf-{Guid.NewGuid():N}.db");
_storage = new SiteStorageService(
$"Data Source={_dbFile}", NullLogger<SiteStorageService>.Instance);
_storage.InitializeAsync().GetAwaiter().GetResult();
_sfStorage = new StoreAndForwardStorage(
$"Data Source={_sfDbFile}", NullLogger<StoreAndForwardStorage>.Instance);
_sfStorage.InitializeAsync().GetAwaiter().GetResult();
_replicationService = new ReplicationService(
new StoreAndForwardOptions(), NullLogger<ReplicationService>.Instance);
}
void IDisposable.Dispose()
{
Shutdown();
try { File.Delete(_dbFile); } catch { /* cleanup */ }
try { File.Delete(_sfDbFile); } catch { /* cleanup */ }
}
private IActorRef CreateReplicationActor(IDeploymentConfigFetcher fetcher) =>
ActorOf(Props.Create(() => new SiteReplicationActor(
_storage, _sfStorage, _replicationService, SiteRole,
NullLogger<SiteReplicationActor>.Instance, fetcher)));
private IActorRef CreateReplicationActor(
IDeploymentConfigFetcher fetcher, SiteRuntimeOptions options, TimeSpan retryDelay) =>
ActorOf(Props.Create(() => new SiteReplicationActor(
_storage, _sfStorage, _replicationService, SiteRole,
NullLogger<SiteReplicationActor>.Instance, fetcher, null, options, retryDelay)));
[Fact]
public async Task ReplicatedFetch_RetriesUpToConfigFetchRetryCount()
{
// The first two fetches fail transiently; the third succeeds. With
// ConfigFetchRetryCount = 3 the standby must retry to the third attempt and
// then guarded-write the fetched config (a short retry delay keeps the test fast).
var attempts = 0;
var fetcher = new FakeConfigFetcher(_ =>
Interlocked.Increment(ref attempts) < 3
? Task.FromException<string>(new InvalidOperationException("central hiccup"))
: Task.FromResult("{\"instanceUniqueName\":\"RetryPump\"}"));
var actor = CreateReplicationActor(
fetcher, new SiteRuntimeOptions { ConfigFetchRetryCount = 3 },
TimeSpan.FromMilliseconds(50));
actor.Tell(new ApplyConfigDeploy(
"RetryPump", "dep-r1", "sha256:r1", true,
"http://central:9000", "tok-r1"));
await AwaitAssertAsync(async () =>
{
Assert.Equal(3, Volatile.Read(ref attempts));
var configs = await _storage.GetAllDeployedConfigsAsync();
Assert.Single(configs, c => c.InstanceUniqueName == "RetryPump");
}, TimeSpan.FromSeconds(10));
}
[Fact]
public async Task ApplyConfigDeploy_StandbyFetchesConfigAndGuardedWrites()
{
// The standby receives an id-only ApplyConfigDeploy; it fetches the config from
// central using the message's coords, then guarded-writes the fetched config.
const string configJson = "{\"instanceUniqueName\":\"Pump1\"}";
var fetcher = new FakeConfigFetcher(_ => Task.FromResult(configJson));
var actor = CreateReplicationActor(fetcher);
actor.Tell(new ApplyConfigDeploy(
"Pump1", "dep-100", "sha256:abc", true,
"http://central:9000", "tok-xyz"));
// The continuation runs off-thread; await the guarded write landing.
await AwaitAssertAsync(async () =>
{
var configs = await _storage.GetAllDeployedConfigsAsync();
var row = Assert.Single(configs, c => c.InstanceUniqueName == "Pump1");
Assert.Equal(configJson, row.ConfigJson);
Assert.Equal("dep-100", row.DeploymentId);
Assert.Equal("sha256:abc", row.RevisionHash);
Assert.True(row.IsEnabled);
}, TimeSpan.FromSeconds(5));
// The fetcher was called with the message's coords.
var call = Assert.Single(fetcher.Calls);
Assert.Equal("http://central:9000", call.BaseUrl);
Assert.Equal("dep-100", call.DeploymentId);
Assert.Equal("tok-xyz", call.Token);
}
[Fact]
public async Task ApplyConfigDeploy_Superseded404_SkipsWriteAndActorSurvives()
{
// A 404 (superseded/expired) surfaces as DeploymentConfigFetchException{IsSuperseded}.
// The standby must skip the write, observe the exception (no crash), and stay alive.
var fetcher = new FakeConfigFetcher(_ =>
Task.FromException<string>(
new DeploymentConfigFetchException("expired", isSuperseded: true)));
var actor = CreateReplicationActor(fetcher);
actor.Tell(new ApplyConfigDeploy(
"GonePump", "dep-stale", "sha256:gone", true,
"http://central:9000", "tok-stale"));
// The fetch was attempted...
await AwaitAssertAsync(() =>
{
Assert.Single(fetcher.Calls);
return Task.CompletedTask;
}, TimeSpan.FromSeconds(5));
// ...the actor did not crash (no Terminated to its watcher within the window)...
Watch(actor);
ExpectNoMsg(TimeSpan.FromMilliseconds(500));
// ...and nothing was written for the superseded instance.
var configs = await _storage.GetAllDeployedConfigsAsync();
Assert.DoesNotContain(configs, c => c.InstanceUniqueName == "GonePump");
}
[Fact]
public async Task ApplyConfigDeploy_EmptyFetchCoords_SkipsFetchAndWrite()
{
// The direct DeployInstanceCommand cross-cluster wire path was retired in Task 14.
// This tests the defensive guard: if empty coords arrive, the actor must skip quietly
// — no FetchAsync("") call, no write — rather than erroring.
var fetcher = new FakeConfigFetcher(_ => Task.FromResult("never"));
var actor = CreateReplicationActor(fetcher);
actor.Tell(new ApplyConfigDeploy(
"NoCoordsPump", "dep-direct", "sha256:nc", true,
CentralFetchBaseUrl: "", FetchToken: ""));
// Give any (erroneous) async continuation time to run, then prove neither happened.
Watch(actor);
ExpectNoMsg(TimeSpan.FromMilliseconds(500));
Assert.Empty(fetcher.Calls);
var configs = await _storage.GetAllDeployedConfigsAsync();
Assert.DoesNotContain(configs, c => c.InstanceUniqueName == "NoCoordsPump");
}
[Fact]
public void ReplicateConfigDeploy_MapsToIdOnlyApplyConfigDeploy_ForPeer()
{
// The outbound mapping must forward an id-only ApplyConfigDeploy carrying the fetch
// coords (and NO inline config) to the peer.
var probe = CreateTestProbe();
var fetcher = new FakeConfigFetcher(_ => Task.FromResult("unused"));
var actor = ActorOf(Props.Create(() => new ProbeForwardingReplicationActor(
_storage, _sfStorage, _replicationService, SiteRole,
NullLogger<SiteReplicationActor>.Instance, fetcher, probe.Ref)));
actor.Tell(new ReplicateConfigDeploy(
"Pump2", "dep-200", "sha256:def", false,
"http://central:9000", "tok-abc"));
var applied = probe.ExpectMsg<ApplyConfigDeploy>(TimeSpan.FromSeconds(3));
Assert.Equal("Pump2", applied.InstanceName);
Assert.Equal("dep-200", applied.DeploymentId);
Assert.Equal("sha256:def", applied.RevisionHash);
Assert.False(applied.IsEnabled);
Assert.Equal("http://central:9000", applied.CentralFetchBaseUrl);
Assert.Equal("tok-abc", applied.FetchToken);
}
// ── Task 21: peer-join S&F buffer resync (anti-entropy) ──
[Fact]
public void StandbyTrackingPeer_SendsResyncRequest()
{
var probe = CreateTestProbe();
var actor = ActorOf(Props.Create(() => new ResyncTestActor(
_storage, _sfStorage, _replicationService, SiteRole,
NullLogger<SiteReplicationActor>.Instance, probe.Ref, () => false)));
actor.Tell(new TriggerPeerTracked()); // stands in for TryTrackPeer's MemberUp path
probe.ExpectMsg<RequestSfBufferResync>(TimeSpan.FromSeconds(3));
}
[Fact]
public void ActiveTrackingPeer_DoesNotRequestResync()
{
var probe = CreateTestProbe();
var actor = ActorOf(Props.Create(() => new ResyncTestActor(
_storage, _sfStorage, _replicationService, SiteRole,
NullLogger<SiteReplicationActor>.Instance, probe.Ref, () => true)));
actor.Tell(new TriggerPeerTracked());
probe.ExpectNoMsg(TimeSpan.FromMilliseconds(300)); // active node never requests a resync
}
[Fact]
public async Task ActiveNode_AnswersResyncRequest_WithChunkedSnapshot()
{
// Post-R2-T5 the active node answers with byte-budgeted SfBufferSnapshotChunk(s)
// (a single small row rides one chunk) rather than the monolithic SfBufferSnapshot.
await _sfStorage.EnqueueAsync(NewSfMessage("m1"));
var probe = CreateTestProbe();
var actor = ActorOf(Props.Create(() => new ResyncTestActor(
_storage, _sfStorage, _replicationService, SiteRole,
NullLogger<SiteReplicationActor>.Instance, probe.Ref, () => true)));
actor.Tell(new RequestSfBufferResync(), TestActor);
var chunk = ExpectMsg<SfBufferSnapshotChunk>(TimeSpan.FromSeconds(3));
Assert.Equal(1, chunk.TotalChunks);
Assert.Equal(1, chunk.Sequence);
Assert.Single(chunk.Messages);
Assert.False(chunk.Truncated);
}
[Fact]
public async Task StandbyNode_AppliesSnapshot_ReplacingItsBuffer()
{
await _sfStorage.EnqueueAsync(NewSfMessage("stale"));
var probe = CreateTestProbe();
var actor = ActorOf(Props.Create(() => new ResyncTestActor(
_storage, _sfStorage, _replicationService, SiteRole,
NullLogger<SiteReplicationActor>.Instance, probe.Ref, () => false)));
actor.Tell(new SfBufferSnapshot(new List<StoreAndForwardMessage> { NewSfMessage("fresh") }, false));
await AwaitAssertAsync(async () =>
{
Assert.Null(await _sfStorage.GetMessageByIdAsync("stale"));
Assert.NotNull(await _sfStorage.GetMessageByIdAsync("fresh"));
}, TimeSpan.FromSeconds(5));
}
// ── R2 T5: chunked resync answer ──
[Fact]
public void ChunkForRemoting_SplitsByByteBudget_PreservingOrderAndSequence()
{
var rows = Enumerable.Range(0, 10)
.Select(i => NewMessage($"m{i}", payloadJson: new string('x', 20_000)))
.ToList();
var chunks = SiteReplicationActor.ChunkForRemoting(rows, maxChunkBytes: 64_000, maxChunkRows: 200);
Assert.True(chunks.Count > 1); // 10 × 20 KB cannot ride one 64 KB chunk
Assert.Equal(rows.Select(r => r.Id), chunks.SelectMany(c => c).Select(r => r.Id)); // order preserved
Assert.All(chunks, c => Assert.True(
c.Sum(r => r.PayloadJson.Length) <= 64_000 || c.Count == 1)); // budget honored (oversized row isolated)
}
[Fact]
public void ChunkForRemoting_RowCapHonored_AndSingleOversizedRowIsolated()
{
var many = Enumerable.Range(0, 500).Select(i => NewMessage($"s{i}", payloadJson: "{}")).ToList();
Assert.All(SiteReplicationActor.ChunkForRemoting(many, 64_000, 200), c => Assert.True(c.Count <= 200));
var oversized = new List<StoreAndForwardMessage>
{ NewMessage("big", payloadJson: new string('y', 100_000)), NewMessage("small", payloadJson: "{}") };
var chunks = SiteReplicationActor.ChunkForRemoting(oversized, 64_000, 200);
Assert.Equal(2, chunks.Count); // the oversized row rides alone
}
[Fact]
public async Task ActiveNode_AnswersResyncRequest_WithSequencedChunks_SharingOneResyncId()
{
for (var i = 0; i < 3; i++)
await _sfStorage.EnqueueAsync(NewMessage($"c{i}", payloadJson: new string('z', 30_000)));
var actor = CreateResyncActor(isActive: () => true);
actor.Tell(new RequestSfBufferResync(), TestActor);
var first = ExpectMsg<SfBufferSnapshotChunk>(TimeSpan.FromSeconds(5));
var rest = Enumerable.Range(1, first.TotalChunks - 1)
.Select(_ => ExpectMsg<SfBufferSnapshotChunk>(TimeSpan.FromSeconds(5)))
.Prepend(first)
.ToList();
Assert.True(first.TotalChunks > 1);
Assert.All(rest, c => Assert.Equal(first.ResyncId, c.ResyncId));
Assert.Equal(Enumerable.Range(1, first.TotalChunks), rest.Select(c => c.Sequence));
Assert.Equal(3, rest.Sum(c => c.Messages.Count));
}
// ── R2 T6: standby chunk assembly + atomic apply + ack ──
[Fact]
public async Task StandbyNode_AssemblesChunks_AppliesOnce_AndAcks()
{
await _sfStorage.EnqueueAsync(NewMessage("stale"));
var actor = CreateResyncActor(isActive: () => false);
var resyncId = "r1";
actor.Tell(new SfBufferSnapshotChunk(resyncId, 1, 2,
new List<StoreAndForwardMessage> { NewMessage("f1") }, false), TestActor);
actor.Tell(new SfBufferSnapshotChunk(resyncId, 2, 2,
new List<StoreAndForwardMessage> { NewMessage("f2") }, false), TestActor);
var ack = ExpectMsg<SfBufferResyncAck>(TimeSpan.FromSeconds(5));
Assert.Equal(resyncId, ack.ResyncId);
Assert.Equal(2, ack.RowCount);
await AwaitAssertAsync(async () =>
{
Assert.Null(await _sfStorage.GetMessageByIdAsync("stale")); // replaced wholesale
Assert.NotNull(await _sfStorage.GetMessageByIdAsync("f1"));
Assert.NotNull(await _sfStorage.GetMessageByIdAsync("f2"));
});
}
[Fact]
public async Task StandbyNode_NewResyncId_DiscardsStalePartialAssembly()
{
var actor = CreateResyncActor(isActive: () => false);
actor.Tell(new SfBufferSnapshotChunk("old", 1, 2,
new List<StoreAndForwardMessage> { NewMessage("orphan") }, false), TestActor);
actor.Tell(new SfBufferSnapshotChunk("new", 1, 1,
new List<StoreAndForwardMessage> { NewMessage("fresh") }, false), TestActor);
ExpectMsg<SfBufferResyncAck>(TimeSpan.FromSeconds(5)); // "new" completed
await AwaitAssertAsync(async () =>
{
Assert.NotNull(await _sfStorage.GetMessageByIdAsync("fresh"));
Assert.Null(await _sfStorage.GetMessageByIdAsync("orphan")); // stale partial never applied
});
}
[Fact]
public void ActiveNode_IgnoresChunks_NeverAcks()
{
var actor = CreateResyncActor(isActive: () => true);
actor.Tell(new SfBufferSnapshotChunk("r", 1, 1,
new List<StoreAndForwardMessage> { NewMessage("x") }, false), TestActor);
ExpectNoMsg(TimeSpan.FromMilliseconds(300));
}
// ── R2 T7: active-side resync ack confirmation + telemetry ──
//
// NOTE (deviation from plan): the actor logs via Microsoft ILogger (NullLogger in
// tests), NOT Akka's EventStream, so the plan's EventFilter.Warning assertions can
// never observe these warnings. We observe the two OTel counters via a MeterListener
// instead — the equivalent, and stronger, observable signal.
[Fact]
public async Task ActiveNode_ReceivingAck_CountsResyncCompleted()
{
long completed = 0;
using var listener = ListenCounter("scadabridge.store_and_forward.resync.completed",
m => Interlocked.Add(ref completed, m));
await _sfStorage.EnqueueAsync(NewMessage("m1"));
var actor = CreateResyncActor(isActive: () => true, ackTimeout: TimeSpan.FromSeconds(30));
actor.Tell(new RequestSfBufferResync(), TestActor);
var chunk = ExpectMsg<SfBufferSnapshotChunk>(TimeSpan.FromSeconds(5));
actor.Tell(new SfBufferResyncAck(chunk.ResyncId, 1), TestActor);
await AwaitAssertAsync(() =>
{
Assert.True(Interlocked.Read(ref completed) >= 1); // ack recorded the completion
return Task.CompletedTask;
}, TimeSpan.FromSeconds(5));
}
[Fact]
public async Task ActiveNode_MissingAck_WarnsAfterAckTimeout()
{
long ackMissing = 0;
using var listener = ListenCounter("scadabridge.store_and_forward.resync.ack_missing",
m => Interlocked.Add(ref ackMissing, m));
await _sfStorage.EnqueueAsync(NewMessage("m1"));
var actor = CreateResyncActor(isActive: () => true, ackTimeout: TimeSpan.FromMilliseconds(200));
actor.Tell(new RequestSfBufferResync(), TestActor);
ExpectMsg<SfBufferSnapshotChunk>(TimeSpan.FromSeconds(5));
// No ack is sent → the ack window expires and the resync is counted unacknowledged.
await AwaitAssertAsync(() =>
{
Assert.True(Interlocked.Read(ref ackMissing) >= 1);
return Task.CompletedTask;
}, TimeSpan.FromSeconds(5));
}
/// <summary>Attaches a <see cref="MeterListener"/> to a single ScadaBridge counter by name,
/// forwarding each recorded increment to <paramref name="onMeasurement"/>.</summary>
private static MeterListener ListenCounter(string instrumentName, Action<long> onMeasurement)
{
var listener = new MeterListener();
listener.InstrumentPublished = (inst, l) =>
{
if (inst.Meter.Name == ScadaBridgeTelemetry.MeterName && inst.Name == instrumentName)
l.EnableMeasurementEvents(inst);
};
listener.SetMeasurementEventCallback<long>((_, m, _, _) => onMeasurement(m));
listener.Start();
return listener;
}
private static StoreAndForwardMessage NewSfMessage(string id) => new()
{
Id = id,
Category = StoreAndForwardCategory.ExternalSystem,
Target = "t",
PayloadJson = "{}",
RetryCount = 0,
MaxRetries = 50,
RetryIntervalMs = 30000,
CreatedAt = DateTimeOffset.UtcNow,
Status = StoreAndForwardMessageStatus.Pending,
};
/// <summary>
/// Builds a resync-test message with a settable payload (additive to
/// <see cref="NewSfMessage"/> — the chunker sizes on <c>PayloadJson</c> length).
/// </summary>
private static StoreAndForwardMessage NewMessage(string id, string payloadJson = "{}") => new()
{
Id = id,
Category = StoreAndForwardCategory.ExternalSystem,
Target = "t",
PayloadJson = payloadJson,
RetryCount = 0,
MaxRetries = 50,
RetryIntervalMs = 30000,
CreatedAt = DateTimeOffset.UtcNow,
Status = StoreAndForwardMessageStatus.Pending,
};
/// <summary>Constructs a <see cref="ResyncTestActor"/> with the given active-node check
/// (the resync chunk/ack tests Tell to and expect from <see cref="TestKit.TestActor"/>).
/// <paramref name="ackTimeout"/> is the active-side ack window seam (T7).</summary>
private IActorRef CreateResyncActor(Func<bool> isActive, TimeSpan? ackTimeout = null) =>
ActorOf(Props.Create(() => new ResyncTestActor(
_storage, _sfStorage, _replicationService, SiteRole,
NullLogger<SiteReplicationActor>.Instance, CreateTestProbe().Ref, isActive, ackTimeout)));
/// <summary>Test message: drives <see cref="SiteReplicationActor.OnPeerTracked"/> directly,
/// standing in for the MemberUp→TryTrackPeer path (a single-node TestKit cannot form a real peer).</summary>
private sealed record TriggerPeerTracked;
/// <summary>
/// Test subclass for the resync tests: captures peer sends to a probe, injects the
/// active-node check, and exposes <see cref="OnPeerTracked"/> via a test message.
/// </summary>
private sealed class ResyncTestActor : SiteReplicationActor
{
private readonly IActorRef _peerProbe;
public ResyncTestActor(
SiteStorageService storage, StoreAndForwardStorage sfStorage,
ReplicationService replicationService, string siteRole,
ILogger<SiteReplicationActor> logger, IActorRef peerProbe, Func<bool> isActive,
TimeSpan? ackTimeout = null)
: base(storage, sfStorage, replicationService, siteRole, logger,
configFetcher: null, isActiveOverride: isActive, resyncAckTimeout: ackTimeout)
{
_peerProbe = peerProbe;
Receive<TriggerPeerTracked>(_ => OnPeerTracked());
}
protected override void SendToPeer(object message) => _peerProbe.Tell(message, Self);
}
/// <summary>
/// Test subclass exposing the peer send: <see cref="SiteReplicationActor.SendToPeer"/> is
/// overridden to forward to a probe so the outbound mapping can be asserted without a real
/// two-node cluster (a single-node TestKit has no peer address, so the real send is dropped).
/// </summary>
private sealed class ProbeForwardingReplicationActor : SiteReplicationActor
{
private readonly IActorRef _peerProbe;
public ProbeForwardingReplicationActor(
SiteStorageService storage, StoreAndForwardStorage sfStorage,
ReplicationService replicationService, string siteRole,
ILogger<SiteReplicationActor> logger, IDeploymentConfigFetcher configFetcher,
IActorRef peerProbe)
: base(storage, sfStorage, replicationService, siteRole, logger, configFetcher)
=> _peerProbe = peerProbe;
protected override void SendToPeer(object message) => _peerProbe.Tell(message, Self);
}
/// <summary>
/// In-test fake <see cref="IDeploymentConfigFetcher"/>: runs a per-deploymentId behavior
/// (return config JSON or throw, as a Task — mirroring the real async HTTP fetcher) and
/// records every call's coords thread-safely (the continuation runs on a pool thread).
/// </summary>
private sealed class FakeConfigFetcher : IDeploymentConfigFetcher
{
private readonly Func<string, Task<string>> _behavior;
public ConcurrentQueue<(string BaseUrl, string DeploymentId, string Token)> Calls { get; } = new();
public FakeConfigFetcher(Func<string, Task<string>> behavior) => _behavior = behavior;
public async Task<string> FetchAsync(
string centralFetchBaseUrl, string deploymentId, string token, CancellationToken ct)
{
Calls.Enqueue((centralFetchBaseUrl, deploymentId, token));
await Task.Yield();
return await _behavior(deploymentId);
}
}
}
tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/NegativeTests.cs
+20 -1
View File
@@ -1,6 +1,7 @@
using Microsoft.Data.Sqlite; using Microsoft.Data.Sqlite;
using Microsoft.Extensions.Logging.Abstractions; using Microsoft.Extensions.Logging.Abstractions;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
using ZB.MOM.WW.ScadaBridge.TestSupport;
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests;
@@ -14,8 +15,17 @@ public class NegativeTests
{ {
// Per design decision: no alarm state table in site SQLite schema. // Per design decision: no alarm state table in site SQLite schema.
// The site SQLite stores only deployed configs and static attribute overrides. // The site SQLite stores only deployed configs and static attribute overrides.
//
// SiteStorageService takes an ILocalDb now, and LocalDb has no in-memory mode
// (its Path is a filesystem path), so the service is initialized over a real
// temp file instead of the "Data Source=:memory:" it used before. The manually
// built schema subset below is still a plain in-memory SqliteConnection — it is
// not a LocalDb store, just a scratch database this test asserts against.
var localDb = TestLocalDb.CreateTemp("negative-schema");
try
{
var storage = new SiteStorageService( var storage = new SiteStorageService(
"Data Source=:memory:", localDb.Db,
NullLogger<SiteStorageService>.Instance); NullLogger<SiteStorageService>.Instance);
await storage.InitializeAsync(); await storage.InitializeAsync();
@@ -49,6 +59,15 @@ public class NegativeTests
var result = await checkCmd.ExecuteScalarAsync(); var result = await checkCmd.ExecuteScalarAsync();
Assert.Null(result); Assert.Null(result);
} }
finally
{
// Dispose first — the master connection anchors the WAL, so the sidecars
// cannot be removed while it is open.
var path = localDb.Path;
localDb.Dispose();
TestLocalDb.DeleteFiles(path);
}
}
[Fact] [Fact]
public async Task Schema_NoLocalConfigAuthoring() public async Task Schema_NoLocalConfigAuthoring()
tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/ArtifactStorageTests.cs
+19 -10
View File
@@ -1,5 +1,6 @@
using Microsoft.Extensions.Logging.Abstractions; using Microsoft.Extensions.Logging.Abstractions;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
using ZB.MOM.WW.ScadaBridge.TestSupport;
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Persistence; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Persistence;
@@ -7,20 +8,24 @@ namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Persistence;
/// WP-33: Local Artifact Storage tests — shared scripts, external systems, /// WP-33: Local Artifact Storage tests — shared scripts, external systems,
/// database connections, notification lists. /// database connections, notification lists.
/// </summary> /// </summary>
/// <remarks>
/// Backed by a real temp-file LocalDb: <see cref="SiteStorageService"/> takes an
/// <c>ILocalDb</c> rather than a connection string, and LocalDb has no in-memory mode.
/// </remarks>
public class ArtifactStorageTests : IAsyncLifetime, IDisposable public class ArtifactStorageTests : IAsyncLifetime, IDisposable
{ {
private readonly string _dbFile; private readonly TestLocalDb _localDb;
private SiteStorageService _storage = null!; private SiteStorageService _storage = null!;
public ArtifactStorageTests() public ArtifactStorageTests()
{ {
_dbFile = Path.Combine(Path.GetTempPath(), $"artifact-test-{Guid.NewGuid():N}.db"); _localDb = TestLocalDb.CreateTemp("artifact-test");
} }
public async Task InitializeAsync() public async Task InitializeAsync()
{ {
_storage = new SiteStorageService( _storage = new SiteStorageService(
$"Data Source={_dbFile}", _localDb.Db,
NullLogger<SiteStorageService>.Instance); NullLogger<SiteStorageService>.Instance);
await _storage.InitializeAsync(); await _storage.InitializeAsync();
} }
@@ -29,7 +34,11 @@ public class ArtifactStorageTests : IAsyncLifetime, IDisposable
public void Dispose() public void Dispose()
{ {
try { File.Delete(_dbFile); } catch { /* cleanup */ } // Dispose first — the master connection anchors the WAL, so the sidecars
// cannot be removed while it is open.
var path = _localDb.Path;
_localDb.Dispose();
TestLocalDb.DeleteFiles(path);
} }
// ── Shared Script Storage ── // ── Shared Script Storage ──
@@ -132,8 +141,10 @@ public class ArtifactStorageTests : IAsyncLifetime, IDisposable
private async Task SeedNotificationRowAsync(string name, string emailsJson) private async Task SeedNotificationRowAsync(string name, string emailsJson)
{ {
await using var connection = new Microsoft.Data.Sqlite.SqliteConnection($"Data Source={_dbFile}"); // Seeded through the service's own (already-open) LocalDb connection — a raw
await connection.OpenAsync(); // SqliteConnection would lack the pragmas and the zb_hlc_next() UDF the site
// tables' capture triggers call.
await using var connection = _storage.CreateConnection();
await using var command = connection.CreateCommand(); await using var command = connection.CreateCommand();
command.CommandText = command.CommandText =
"INSERT INTO notification_lists (name, recipient_emails, updated_at) VALUES (@n, @e, @u)"; "INSERT INTO notification_lists (name, recipient_emails, updated_at) VALUES (@n, @e, @u)";
@@ -145,8 +156,7 @@ public class ArtifactStorageTests : IAsyncLifetime, IDisposable
private async Task SeedSmtpRowAsync(string name, string password) private async Task SeedSmtpRowAsync(string name, string password)
{ {
await using var connection = new Microsoft.Data.Sqlite.SqliteConnection($"Data Source={_dbFile}"); await using var connection = _storage.CreateConnection();
await connection.OpenAsync();
await using var command = connection.CreateCommand(); await using var command = connection.CreateCommand();
command.CommandText = command.CommandText =
@"INSERT INTO smtp_configurations (name, server, port, auth_mode, from_address, username, password, oauth_config, updated_at) @"INSERT INTO smtp_configurations (name, server, port, auth_mode, from_address, username, password, oauth_config, updated_at)
@@ -159,8 +169,7 @@ public class ArtifactStorageTests : IAsyncLifetime, IDisposable
private async Task<long> RowCountAsync(string table) private async Task<long> RowCountAsync(string table)
{ {
await using var connection = new Microsoft.Data.Sqlite.SqliteConnection($"Data Source={_dbFile}"); await using var connection = _storage.CreateConnection();
await connection.OpenAsync();
await using var command = connection.CreateCommand(); await using var command = connection.CreateCommand();
command.CommandText = $"SELECT COUNT(*) FROM {table}"; command.CommandText = $"SELECT COUNT(*) FROM {table}";
return (long)(await command.ExecuteScalarAsync())!; return (long)(await command.ExecuteScalarAsync())!;
tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/NativeAlarmStateStoreTests.cs
+13 -7
View File
@@ -1,5 +1,6 @@
using Microsoft.Extensions.Logging.Abstractions; using Microsoft.Extensions.Logging.Abstractions;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
using ZB.MOM.WW.ScadaBridge.TestSupport;
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Persistence; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Persistence;
@@ -7,19 +8,23 @@ namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Persistence;
/// Task 14: site-local SQLite <c>native_alarm_state</c> store — mirrored native alarm /// Task 14: site-local SQLite <c>native_alarm_state</c> store — mirrored native alarm
/// condition snapshots keyed by (instance, source canonical name, source reference). /// condition snapshots keyed by (instance, source canonical name, source reference).
/// </summary> /// </summary>
/// <remarks>
/// Backed by a real temp-file LocalDb: <see cref="SiteStorageService"/> takes an
/// <c>ILocalDb</c> rather than a connection string, and LocalDb has no in-memory mode.
/// </remarks>
public class NativeAlarmStateStoreTests : IAsyncLifetime, IDisposable public class NativeAlarmStateStoreTests : IAsyncLifetime, IDisposable
{ {
private readonly string _dbFile; private readonly TestLocalDb _localDb;
private SiteStorageService _storage = null!; private SiteStorageService _storage = null!;
public NativeAlarmStateStoreTests() public NativeAlarmStateStoreTests()
{ {
_dbFile = Path.Combine(Path.GetTempPath(), $"nas-{Guid.NewGuid():N}.db"); _localDb = TestLocalDb.CreateTemp("nas");
} }
public async Task InitializeAsync() public async Task InitializeAsync()
{ {
_storage = new SiteStorageService($"Data Source={_dbFile}", NullLogger<SiteStorageService>.Instance); _storage = new SiteStorageService(_localDb.Db, NullLogger<SiteStorageService>.Instance);
await _storage.InitializeAsync(); await _storage.InitializeAsync();
} }
@@ -93,9 +98,10 @@ public class NativeAlarmStateStoreTests : IAsyncLifetime, IDisposable
public void Dispose() public void Dispose()
{ {
if (File.Exists(_dbFile)) // Dispose first — the master connection anchors the WAL, so the sidecars
{ // cannot be removed while it is open.
File.Delete(_dbFile); var path = _localDb.Path;
} _localDb.Dispose();
TestLocalDb.DeleteFiles(path);
} }
} }
tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/SiteStorageSchemaTests.cs
+128
View File
@@ -0,0 +1,128 @@
using Microsoft.Data.Sqlite;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Persistence;
/// <summary>
/// Pins <see cref="SiteStorageSchema"/> as the single owner of the site config DDL.
/// The Host applies this to a LocalDb-managed connection before
/// <c>RegisterReplicated</c> installs the capture triggers, so the shape it produces
/// is the shape that replicates.
/// </summary>
public class SiteStorageSchemaTests
{
[Fact]
public void Apply_IsIdempotent_AndCreatesEveryTable()
{
var path = Path.Combine(Path.GetTempPath(), $"site-schema-{Guid.NewGuid():N}.db");
try
{
using var connection = new SqliteConnection($"Data Source={path}");
connection.Open();
SiteStorageSchema.Apply(connection);
SiteStorageSchema.Apply(connection); // second run must not throw
// Every table is named explicitly on purpose — a loop asserting "9 tables"
// would still pass if one were renamed.
foreach (var table in new[]
{
"deployed_configurations", "static_attribute_overrides", "shared_scripts",
"external_systems", "database_connections", "notification_lists",
"data_connection_definitions", "smtp_configurations", "native_alarm_state",
})
{
Assert.True(TableExists(connection, table), $"missing table: {table}");
}
}
finally
{
SqliteConnection.ClearAllPools();
File.Delete(path);
}
}
/// <summary>
/// The migration columns exist to upgrade a database created by an older build in
/// place. Asserting only against a freshly-created table would pass even if every
/// <c>ALTER</c> were deleted, because <c>CREATE TABLE</c> already lists them — so
/// this starts from the pre-migration shape and proves the migration path runs.
/// </summary>
[Fact]
public void Apply_AddsMigrationColumns_ToALegacyDatabase()
{
var path = Path.Combine(Path.GetTempPath(), $"site-schema-legacy-{Guid.NewGuid():N}.db");
try
{
using var connection = new SqliteConnection($"Data Source={path}");
connection.Open();
// The shapes as they existed before backup_configuration /
// failover_retry_count / metadata_json / timeout_seconds were added.
using (var legacy = connection.CreateCommand())
{
legacy.CommandText = """
CREATE TABLE data_connection_definitions (
name TEXT PRIMARY KEY,
protocol TEXT NOT NULL,
configuration TEXT,
updated_at TEXT NOT NULL
);
CREATE TABLE native_alarm_state (
instance_unique_name TEXT NOT NULL,
source_canonical_name TEXT NOT NULL,
source_reference TEXT NOT NULL,
condition_json TEXT NOT NULL,
last_transition_at TEXT NOT NULL,
PRIMARY KEY (instance_unique_name, source_canonical_name, source_reference)
);
CREATE TABLE external_systems (
name TEXT PRIMARY KEY,
endpoint_url TEXT NOT NULL,
auth_type TEXT NOT NULL,
auth_configuration TEXT,
method_definitions TEXT,
updated_at TEXT NOT NULL
);
INSERT INTO external_systems (name, endpoint_url, auth_type, updated_at)
VALUES ('legacy-system', 'http://example.invalid', 'None', '2026-01-01T00:00:00Z');
""";
legacy.ExecuteNonQuery();
}
SiteStorageSchema.Apply(connection);
Assert.True(ColumnExists(connection, "data_connection_definitions", "backup_configuration"));
Assert.True(ColumnExists(connection, "data_connection_definitions", "failover_retry_count"));
Assert.True(ColumnExists(connection, "native_alarm_state", "metadata_json"));
Assert.True(ColumnExists(connection, "external_systems", "timeout_seconds"));
// The pre-existing row survives the upgrade and reads back the NOT NULL
// column's default rather than failing.
using var check = connection.CreateCommand();
check.CommandText = "SELECT timeout_seconds FROM external_systems WHERE name = 'legacy-system'";
Assert.Equal(0, Convert.ToInt32(check.ExecuteScalar()));
}
finally
{
SqliteConnection.ClearAllPools();
File.Delete(path);
}
}
private static bool TableExists(SqliteConnection connection, string table)
{
using var cmd = connection.CreateCommand();
cmd.CommandText = "SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name = $name";
cmd.Parameters.AddWithValue("$name", table);
return Convert.ToInt32(cmd.ExecuteScalar()) > 0;
}
private static bool ColumnExists(SqliteConnection connection, string table, string column)
{
using var cmd = connection.CreateCommand();
cmd.CommandText = $"SELECT COUNT(*) FROM pragma_table_info('{table}') WHERE name = $name";
cmd.Parameters.AddWithValue("$name", column);
return Convert.ToInt32(cmd.ExecuteScalar()) > 0;
}
}
tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Persistence/SiteStorageServiceTests.cs
+36 -12
View File
@@ -1,7 +1,7 @@
using Microsoft.Data.Sqlite;
using Microsoft.Extensions.Logging; using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Logging.Abstractions; using Microsoft.Extensions.Logging.Abstractions;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
using ZB.MOM.WW.ScadaBridge.TestSupport;
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Persistence; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Persistence;
@@ -9,20 +9,26 @@ namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Persistence;
/// Tests for SiteStorageService using file-based SQLite (temp files). /// Tests for SiteStorageService using file-based SQLite (temp files).
/// Validates the schema, CRUD operations, and constraint behavior. /// Validates the schema, CRUD operations, and constraint behavior.
/// </summary> /// </summary>
/// <remarks>
/// The service now takes an <c>ILocalDb</c> rather than a connection string, so the fixture
/// is a real temp-file LocalDb. It stays a file (never in-memory): LocalDb has no in-memory
/// mode, and the connections it hands out carry the pragmas and the <c>zb_hlc_next()</c> UDF
/// the site tables' capture triggers depend on.
/// </remarks>
public class SiteStorageServiceTests : IAsyncLifetime, IDisposable public class SiteStorageServiceTests : IAsyncLifetime, IDisposable
{ {
private readonly string _dbFile; private readonly TestLocalDb _localDb;
private SiteStorageService _storage = null!; private SiteStorageService _storage = null!;
public SiteStorageServiceTests() public SiteStorageServiceTests()
{ {
_dbFile = Path.Combine(Path.GetTempPath(), $"site-storage-test-{Guid.NewGuid():N}.db"); _localDb = TestLocalDb.CreateTemp("site-storage-test");
} }
public async Task InitializeAsync() public async Task InitializeAsync()
{ {
_storage = new SiteStorageService( _storage = new SiteStorageService(
$"Data Source={_dbFile}", _localDb.Db,
NullLogger<SiteStorageService>.Instance); NullLogger<SiteStorageService>.Instance);
await _storage.InitializeAsync(); await _storage.InitializeAsync();
} }
@@ -31,7 +37,11 @@ public class SiteStorageServiceTests : IAsyncLifetime, IDisposable
public void Dispose() public void Dispose()
{ {
try { File.Delete(_dbFile); } catch { /* cleanup */ } // Dispose first — the master connection anchors the WAL, so the sidecars
// cannot be removed while it is open.
var path = _localDb.Path;
_localDb.Dispose();
TestLocalDb.DeleteFiles(path);
} }
[Fact] [Fact]
@@ -45,10 +55,17 @@ public class SiteStorageServiceTests : IAsyncLifetime, IDisposable
[Fact] [Fact]
public async Task Initialize_EnablesWalJournalMode() public async Task Initialize_EnablesWalJournalMode()
{ {
// WAL is set once at InitializeAsync (persistent, database-level). A file-backed DB // ── Invariant that moved owner ──
// is required — WAL is not available for :memory: databases. // WAL used to be SiteStorageService's own job (an explicit PRAGMA in
await using var conn = _storage.CreateConnection(); // InitializeAsync). LocalDb now owns the file and its pragmas, so the service no
await conn.OpenAsync(); // longer sets it. The guarantee production depends on has NOT moved: without WAL
// the site's concurrent readers and writers start serializing on "database is
// locked". So rather than deleting this test with the code that used to provide
// the pragma, it is retargeted to assert the same guarantee against the new,
// LocalDb-backed service. journal_mode is persistent and file-scoped, so any
// connection observes it. A file-backed DB is still required — WAL is not
// available for :memory: databases, which is also why LocalDb has no in-memory mode.
await using var conn = _storage.CreateConnection(); // already open — do NOT call OpenAsync
await using var cmd = conn.CreateCommand(); await using var cmd = conn.CreateCommand();
cmd.CommandText = "PRAGMA journal_mode;"; cmd.CommandText = "PRAGMA journal_mode;";
var mode = (string)(await cmd.ExecuteScalarAsync())!; var mode = (string)(await cmd.ExecuteScalarAsync())!;
@@ -209,7 +226,12 @@ public class SiteStorageServiceTests : IAsyncLifetime, IDisposable
Assert.Empty(overrides); Assert.Empty(overrides);
} }
// ── Task 13: StoreDeployedConfigIfNewerAsync (guarded standby write) ── // ── StoreDeployedConfigIfNewerAsync (the deployed_at-guarded write) ──
//
// Originally the standby's notify-and-fetch write path. LocalDb Phase 2 replaced that
// with change-data-capture, so the surviving caller is SiteReconciliationActor's
// startup self-heal against central, where the guard still stops a slow reconcile
// response from overwriting a newer config that landed while it was in flight.
/// <summary> /// <summary>
/// Seeds a deployed_configurations row with an explicit deployed_at timestamp using the same /// Seeds a deployed_configurations row with an explicit deployed_at timestamp using the same
@@ -219,8 +241,10 @@ public class SiteStorageServiceTests : IAsyncLifetime, IDisposable
string instanceName, string configJson, string deploymentId, string instanceName, string configJson, string deploymentId,
string revisionHash, bool isEnabled, DateTimeOffset deployedAt) string revisionHash, bool isEnabled, DateTimeOffset deployedAt)
{ {
await using var conn = new SqliteConnection($"Data Source={_dbFile}"); // Seeded through the service's own (already-open) LocalDb connection: a raw
await conn.OpenAsync(); // SqliteConnection would lack the pragmas and the zb_hlc_next() UDF the site
// tables' capture triggers call.
await using var conn = _storage.CreateConnection();
await using var cmd = conn.CreateCommand(); await using var cmd = conn.CreateCommand();
cmd.CommandText = @" cmd.CommandText = @"
INSERT INTO deployed_configurations INSERT INTO deployed_configurations
tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Repositories/SiteRepositoryTests.cs
+16 -4
View File
@@ -1,6 +1,7 @@
using Microsoft.Extensions.Logging.Abstractions; using Microsoft.Extensions.Logging.Abstractions;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Repositories; using ZB.MOM.WW.ScadaBridge.SiteRuntime.Repositories;
using ZB.MOM.WW.ScadaBridge.TestSupport;
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Repositories; namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Repositories;
@@ -17,21 +18,32 @@ namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Repositories;
/// </summary> /// </summary>
public class SiteRepositoryTests : IDisposable public class SiteRepositoryTests : IDisposable
{ {
private readonly string _dbFile; private readonly TestLocalDb _localDb;
public SiteRepositoryTests() public SiteRepositoryTests()
{ {
_dbFile = Path.Combine(Path.GetTempPath(), $"site-repo-test-{Guid.NewGuid():N}.db"); _localDb = TestLocalDb.CreateTemp("site-repo-test");
} }
public void Dispose() public void Dispose()
{ {
try { File.Delete(_dbFile); } catch { /* cleanup */ } // Dispose first — the master connection anchors the WAL, so the sidecars
// cannot be removed while it is open.
var path = _localDb.Path;
_localDb.Dispose();
TestLocalDb.DeleteFiles(path);
GC.SuppressFinalize(this); GC.SuppressFinalize(this);
} }
/// <summary>
/// A brand-new <see cref="SiteStorageService"/> instance over the same site database.
/// The service now takes an <c>ILocalDb</c> instead of a connection string, so the
/// single fixture database is shared while each call still yields a fresh service
/// object — which is what the restart tests below actually vary (the synthetic IDs are
/// derived per service/repository instance, not per connection).
/// </summary>
private SiteStorageService NewStorage() private SiteStorageService NewStorage()
=> new($"Data Source={_dbFile}", NullLogger<SiteStorageService>.Instance); => new(_localDb.Db, NullLogger<SiteStorageService>.Instance);
/// <summary> /// <summary>
/// SiteRuntime-006: an external system stored via <see cref="SiteStorageService"/> /// SiteRuntime-006: an external system stored via <see cref="SiteStorageService"/>

Some files were not shown because too many files have changed in this diff Show More