Commit Graph

2241 Commits

Author SHA1 Message Date
Joseph Doherty ac5eb12cce refactor(site): extract SiteStorageSchema.Apply from SiteStorageService
Task 4 of the Phase 2 plan. All nine table definitions plus MigrateSchemaAsync
and TryAddColumnAsync move into a static sync Apply(SqliteConnection) depending
only on Microsoft.Data.Sqlite, so the Host can apply the DDL to a LocalDb-managed
connection before RegisterReplicated installs the capture triggers.

Per the plan, PRAGMA journal_mode=WAL stays in InitializeAsync — LocalDb owns
the connection's pragmas, and the service still opens its own connections until
Task 6.

One deviation: TryAddColumnAsync's `catch (SqliteException) when
(ex.Message.Contains("duplicate column"))` becomes a PRAGMA table_info probe,
matching OperationTrackingSchema. The message-matching form depends on an error
string that is not part of SQLite's contract, and it swallowed every
SqliteException whose message happened to contain that substring. The probe also
drops the ILogger dependency, which is what let the class become static. Cost:
the per-column "Migrated: added column" info log is gone — it fired once per
column per legacy database and nothing consumes it.

Also added a test beyond the plan's specified one, for the same reason as Task 3:
the specified test asserts against a freshly-created database, where CREATE TABLE
already lists the migration columns, so it would pass with every ALTER deleted.
The added test starts from the pre-migration shapes with a row present and proves
the migration path runs and preserves data across a NOT NULL DEFAULT add.

SiteRuntime suite 532 passed / 0 failed; full solution build 0 warnings.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
2026-07-20 02:20:50 -04:00
Joseph Doherty 9e3239c5d9 refactor(sf): extract StoreAndForwardSchema.Apply from the storage class
Task 3 of the Phase 2 plan. Mirrors OperationTrackingSchema: the sf_messages
DDL, the four additive ALTERs, the last_attempt_at_ms backfill and the due-index
move verbatim into a static sync Apply(SqliteConnection), depending only on
Microsoft.Data.Sqlite. The Host needs to apply this to a LocalDb-managed
connection before RegisterReplicated installs the capture triggers; the store
still calls it so a directly-constructed store stays self-sufficient.

Two deviations from the plan as written, both deliberate:

1. The PRAGMA journal_mode=WAL in InitializeAsync STAYS. The plan's Step 4
   snippet drops it, but Task 4 explicitly says not to move the equivalent
   pragma ("LocalDb owns the connection's pragmas") — the two tasks contradict
   each other. Keeping it preserves behaviour for the intermediate commits and
   for directly-constructed stores; it becomes moot in Task 5 when the store
   stops opening its own connections. Dropping it now would quietly regress the
   concurrent-writer support the pragma's own comment documents.

2. Added a second test beyond the plan's. The specified test asserts only
   against a freshly-created table, where CREATE TABLE already lists all 16
   columns — it would pass with every ALTER deleted. The added test starts from
   the pre-upgrade 12-column shape with a row in it and proves the upgrade path
   runs and preserves data.

That second test also surfaced that the backfill lands 1 ms low: julianday()'s
double day-fraction cannot represent every millisecond. Pre-existing behaviour,
carried over verbatim, asserted with a 1 ms tolerance rather than pinning a
precision the implementation never had.

Suite: 154 passed, 0 failed.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
2026-07-20 02:18:05 -04:00
Joseph Doherty 12eb97e07a docs(localdb): close the phase 2 gate — questions answered, plan written
Task 2 of the Phase 2 plan, plus the clean re-run of Task 1's cut-short
sampling that Task 2 was waiting on.

Gate doc: status NOT STARTED -> CLOSED. All five §5 open questions answered
inline. The questions are kept, not deleted — several of the answers overturn
a premise the gate itself stated, and that reasoning is the value:

- Oplog sizing is not the binding constraint. Cap overrun prunes and flags
  needs_snapshot (graceful resync), so the caps need no defensive sizing; the
  real ceiling is D6's 4 MB gRPC cap, binding on MaxBatchSize x row-bytes.
- LWW-vs-in-flight-send cannot arise on a correct pair (only the active node
  sweeps). The honest cost is D2's semantic change: the standby is convergent,
  no longer byte-identical.
- Migration-under-load is designed out, not managed — both mechanisms are
  deleted in the same commit and D5 forecloses rolling upgrades.
- Rollback is genuinely "revert the commit": the migration is additive and
  leaves the legacy files intact. Bounded cost is the post-cutover delta.
- One DB per process stays; every write axis is bounded.

§2's requirement to state CDC's duplicate-delivery bound is routed explicitly
to Task 21 rather than left implicit.

Task 1 re-run (safe copy-based snap(), both nodes restarted first, six 60s
intervals): sf_messages 0.80 rows/sec insert, 0/sec retry-UPDATE, oplog 0,
alarms 0, max payload_json 76 B, zero SQLite errors across 30 min, both site-a
nodes converged at 3564 rows. Independently confirms the disk-I/O storm was
observer-induced. Recorded with its limits stated: 0.80/s is ~1.6% of the 50/s
ceiling and the retry path was never exercised — acceptable only because that
ceiling is structural rather than empirical, and the rig's 721 B config rows
are explicitly NOT representative.

GATE VERDICT: PROCEED to Task 3. Binding output is MaxBatchSize 500 -> 16.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
2026-07-20 02:15:53 -04:00
Joseph Doherty d512572dac docs(localdb): phase 2 resume state — where the plan stands and what carries forward
Scratch handoff so the plan can be picked back up cleanly: Task 1 done and its
STOP verdict superseded, what must happen before Task 3, the four plan-premise
corrections, the rig gotchas that are not discoverable from the repo, and the
rig cleanup the Task 20 live gate needs.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
2026-07-20 02:01:42 -04:00
Joseph Doherty 8652eab98e fix(localdb): root-cause the soak disk-I/O failure + ship the hardening follow-ups
The Phase 2 soak's "LocalDb fails under load" blocker is NOT a product defect.
Root cause: host-side (macOS) sqlite3 reads of the live, bind-mounted WAL
databases. POSIX advisory locks do not propagate across the virtiofs boundary,
so the host reader believes it is the only connection, checkpoints on close and
resets the WAL to 0 bytes under the container. The container's still-mapped
WAL index then references frames that no longer exist and every subsequent
statement fails SQLITE_IOERR_SHORT_READ (522) -> primary code 10, permanently
until the process reopens the database.

Reproduced on demand both on the rig (one sqlite3 SELECT reset a 4.6 MiB WAL and
produced the first error one second later) and in a minimal python:3.12-alpine
repro with no LocalDb or .NET involved. Refuted the converse: a freshly-reopened
node sustained the full soak load 10+ minutes with zero errors.

The original brief's isolation was confounded - both nodes had been poisoned by
the same sampling pass, and a poisoned standby looks healthy only because it
issues almost no statements. Corrections annotated in place.

Hardening shipped alongside:
- SqliteErrorCodes.Describe: log SQLite primary AND extended codes at the
  LocalDb-adjacent catch sites (the missing extended code is what made the
  original diagnosis so slow).
- SiteAuditTelemetryActor: stop touching ActorContext across an await
  (NotSupportedException), with regression coverage.
- infra/reseed.sh: apply the MSSQL init scripts explicitly. The official
  mssql/server image does not implement /docker-entrypoint-initdb.d, so a fresh
  volume hung the reseed forever; compose mounts annotated as informational.

Deliberately NOT done: detect-and-reopen self-heal in SqliteLocalDb. It defends
only against external interference, which is now prevented at the source, and
same-kernel production readers see the locks correctly.

Build 0 warnings; SiteAuditTelemetryActorTests 9/9, SiteEventLogging 70/70.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
2026-07-20 02:01:00 -04:00
Joseph Doherty e9e11d635e docs(known-issues): root-cause brief for the LocalDb disk-I/O-error-under-load defect
Standalone handoff for a follow-up investigation. Records the symptom, a full
reproduction (including the two rig-tooling blockers and the CachedCall-vs-Call
detail needed to generate load at all), the isolation evidence, a code map of
the LocalDb connection model, four ranked hypotheses, and acceptance criteria.

Not root-caused. Highest-value next step identified: capture the SQLite EXTENDED
result code (logs only carry the generic primary code 10), and run the load with
LocalDb:Path off the bind mount to partition environmental vs library causes.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
2026-07-20 01:11:08 -04:00
Joseph Doherty 82c869a1df docs(localdb): record the phase 2 task-1 gate verdict in the task record
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
2026-07-20 01:07:18 -04:00
Joseph Doherty fc553bd9ba docs(localdb): phase 2 rig soak findings — GATE FAILS on a phase 1 defect
Task 1 of the Phase 2 plan. The gate stops the plan, but not for the reason it
anticipated: oplog sizing is fine and D6 is resolved.

BLOCKER: the Phase 1 consolidated LocalDb (site-localdb.db) throws SQLite
Error 10 'disk I/O error' on essentially every write on the ACTIVE node under
sustained load — site_events, OperationTracking and the audit telemetry paths
all fail, and site event logging silently drops events. Isolated to the load
(not the node, not host-side observation) by failing over between nodes, and to
LocalDb specifically (legacy store-and-forward.db / scadabridge.db in the same
bind-mounted directory take zero errors under identical load).

Phase 2 would register 8 more tables into that database — including the two
highest-volume ones — while deleting the bespoke mechanisms that currently
carry them. Must be root-caused first.

Also recorded: D6's premise corrected (largest known production config_json is
~60-70 KB, not >128 KB — but MaxBatchSize 500 is still unsafe, use 16); D4's
premise corrected (alarm writes are bounded by per-SourceReference coalescing at
a 100 ms flush); sf_messages has a hard 50 rows/sec structural ceiling; and
exceeding the oplog caps is a graceful snapshot-resync, not a failure.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
2026-07-20 01:06:59 -04:00
Joseph Doherty cf46e59680 fix(rig): seed LDAP group mappings with the canonical role names
docker/seed-sites.sh inserted the pre-rename 'Design' / 'Deployment' role
strings, but the canonical vocabulary in Roles.cs is 'Designer' / 'Deployer'.
The mismatch authorized nothing: on a freshly reseeded rig every
Designer/Deployer-gated management command failed UNAUTHORIZED, including
seed-sites.sh's own trailing `deploy artifacts` and reseed.sh's stage 6d
encrypted-secret restore.

Found while standing up the Phase 2 soak rig.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
2026-07-19 17:55:04 -04:00
Joseph Doherty 25463d522f docs(localdb): phase 2 plan review pass — corrections from code verification
Three verification sweeps over SiteRuntime, StoreAndForward, and Host/rig/docs
plus the LocalDb library source. Load-bearing corrections: D1 (the guarded write
has a second surviving caller, SiteReconciliationActor), D3 (the active node
already purges — Task 12 becomes a pin), D6 (new: the 4 MB gRPC cap vs
row-count-only batching), Task 1 (rewritten method — the Phase 2 tables are not
in the Phase 1 oplog), and Task 14 (do not register the notification/SMTP tables).

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
2026-07-19 17:27:11 -04:00
Joseph Doherty f6ca82a9e2 docs(localdb): phase 2 implementation plan (21 tasks, full scope)
Moves scadabridge.db's 9 config tables + sf_messages into the consolidated
LocalDb file and deletes SiteReplicationActor + StoreAndForward's
ReplicationService.

Resolves the phase 2 gate's five open questions from code recon (D1-D5):

D1 notify-and-fetch is DELETED, not preserved. It exists only because the
   config blob exceeds Akka's 128KB frame; LocalDb sync is gRPC. The
   deployed_at version guard protected against a stale fetch racing, so it
   dies with the fetch. Do not reproduce it on LWW - different clocks.
D2 ReplaceAllAsync is deleted and the N1 directional guard becomes
   unnecessary: LocalDb's snapshot resync merges per-row LWW and never
   deletes (SnapshotApplier has no DELETE; LwwApplier.cs:69-78 discards a
   lower-HLC incoming row). Semantic change - the standby is convergent,
   no longer byte-identical.
D3 The SMTP purge (plaintext passwords) rides the replication path and is
   re-homed to the active node BEFORE any deletion.
D4 native_alarm_state volume is measured by a rig soak, not assumed. Task 1
   gates the plan and stops it if the oplog cannot absorb the churn.
D5 No dual-mechanism period forecloses rolling site upgrades - both nodes
   must stop and start together.

Recon also found the gate doc's "two test files are the spec" undercounts:
the real specification is five files, including the N1 Critical regression
test and the only Requeue coverage.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
2026-07-19 15:04:37 -04:00
Joseph Doherty b64857c8d6 docs(localdb): Phase 2 gate + final Phase 1 task record
Task 14. A GATE document, not an implementation plan - the plan explicitly wants
Phase 2 planned against post-Phase-1 reality, which means after this record
exists rather than from the original design alone.

Records what Phase 1 actually established (ordering constraints, where
registration lives and why, the fail-closed interceptor, the silent meter
allowlist), what must be ported before anything is deleted (both bespoke
replicator test files, treated as specifications, plus the accepted N5
bounded-duplicate race), and why Phase 2 is a harder risk class than Phase 1:
it REPLACES a working mechanism rather than adding one where none existed,
sf_messages has external side effects, config tables drive deployment, there is
no dual-mechanism period, and the migration is real this time rather than the
usual no-op.

The most important line is an open question, not an answer: choosing
MaxOplogRows / MaxOplogAge / snapshot-resync thresholds for sf_messages write
rates needs a Phase-1 rig SOAK that has not been run. The live gate proved
correctness, not steady-state behaviour over time. Phase 2 should not be planned
without it.

tasks.json finalized: all 15 tasks completed, every deviation from the plan
recorded with its reason.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
2026-07-19 10:00:50 -04:00
Joseph Doherty 568735a41e docs(localdb): Phase 1 truth pass
Task 13.

SiteEventLogger's XML doc said 'Not replicated to standby. On failover, the new
active node starts a fresh log.' That is now false and was the exact behaviour
Phase 1 set out to fix.

OperationTrackingOptions.ConnectionString and SiteEventLogOptions.DatabasePath
marked MIGRATION-ONLY: nothing reads them but SiteLocalDbLegacyMigrator, and
they are safe to delete from a config once a node has migrated.

ScadaBridge CLAUDE.md gains a consolidated-site-database entry covering the
GUID/opaque-token contract change, the migration-only keys, the incidental
CWD-outside-the-volume data-loss fix, the fail-closed default-OFF replication
posture, and the fact that ZbTelemetryOptions.Meters is a silent allowlist.

scadaproj CLAUDE.md's LocalDb row goes from 'no app adoption yet' to the honest
current state, per the component-status-honesty convention: ScadaBridge Phase 1
only, on an UNMERGED branch, replication live-proven on the rig's site-a pair
ONLY, default-OFF everywhere else, Phase 2 not started, no other app adopted, no
production deployment replicating.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
2026-07-19 09:51:55 -04:00
Joseph Doherty 95c108f1d7 fix(localdb): allowlist the replication meter + record the Task 12 live gate
Task 12 (live gate on the docker rig) found one real defect, which this fixes.

THE DEFECT. The plan asserted LocalDb metrics "need no work - the
ZB.MOM.WW.LocalDb.Replication meter flows through the existing AddZbTelemetry
Prometheus export". It does not. ZbTelemetryOptions.Meters is an ALLOWLIST: an
unlisted meter's instruments are never observed, with no error, no warning, and
no missing-metric signal anywhere. The rig's /metrics scrape returned 301 lines
and zero localdb_* series. Only looking for them found it.

Fixed by hoisting the allowlist to SiteServiceRegistration.ObservedMeters and
adding LocalDbMetrics.MeterName. The pin test asserts on that constant rather
than on OTel's observed set, because AddZbTelemetry applies its options to a
local instance and never registers IOptions - there is nothing resolvable to
interrogate. The end-to-end proof is the live scrape below; the test exists to
stop the entry being dropped again. Called out as such in the test.

LIVE GATE EVIDENCE (docker rig, 8 nodes, rebuilt via docker/deploy.sh):

1. Consolidated DB created on every site node at /app/data/site-localdb.db -
   inside the mounted volume, unlike the legacy CWD-relative databases.

2. Replication proven REAL, not two independent local writes. Both site-a nodes
   hold the identical row and, decisively, the identical originating node_id and
   HLC in __localdb_row_version:
     site_events|{"id":"8b06b37c..."}|116946936661147648|65b00319-...|0
   __localdb_peer_state shows a completed handshake in both directions.

3. Convergence: 5/5 events on both nodes, __localdb_row_version byte-identical
   across the pair, oplog drained to 0, dead_letter 0 on both.

4. SITE failover drill. Note the repo's failover-drill.sh targets CENTRAL nodes
   and does not exercise this claim, so the site-pair flip was run directly:
   stopped site-a-a (the node holding the history); site-a-b survived with the
   complete pre-flip history (3/3 events) plus its own takeover event. Restarted
   site-a-a; it caught up on what it missed and both nodes reconverged to the
   same 5 ids. This is the failure Phase 1 exists to prevent.

5. localdb_* now exported after the fix:
     localdb_sync_reconnects_total{...} 1
     localdb_oplog_depth{...} 0

6. DEFAULT-OFF pin, live and side-by-side: site-b (no Replication section) boots
   identically, creates its consolidated DB, registers the engine - and stays
   inert. Zero replication log lines, no reconnect counter at all, so it never
   dialled anything.

One transient WRN at startup on site-a-a ("Connection refused", reconnecting) is
node A dialling before node B's listener was up; it reconnected within a second.
Expected for a pair that starts together.

Verified: build 0 warnings; Host wiring tests 11/11.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
2026-07-19 09:50:11 -04:00
Joseph Doherty 9cf7c0f007 feat(localdb): enable replication on the site-a rig pair
Task 11. site-a-a is the initiator (dials scadabridge-site-a-b:8083, the
existing site gRPC h2c listener - no new port); site-a-b is passive. Both carry
the SAME dev ApiKey, which is load-bearing: LocalDbSyncAuthInterceptor is
fail-closed, so a mismatch does not degrade to unauthenticated replication, it
rejects every stream and the pair silently stops converging.

site-b and site-c stay unreplicated on purpose, so the default-OFF posture is
proven side-by-side on one rig rather than asserted.

Plain dev key matches the rig's existing posture; production uses a
${secret:...} reference through the pre-host expander. Noted in both files.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
2026-07-19 09:43:04 -04:00
Joseph Doherty 3c395d794a test(localdb): two-node site-pair convergence over a real gRPC transport
Task 10 of the LocalDb Phase 1 adoption plan. Two ScadaBridge site nodes
replicating the consolidated site database over loopback Kestrel h2c, through
the REAL fail-closed auth interceptor - not a stand-in.

This is the test that answers the question Phase 1 exists to answer. Every piece
upstream of it - schema helpers, DI wiring, the interceptor - can be
individually green while the pair still fails to converge.

It initializes both databases through SiteLocalDbSetup.OnReady rather than a
hand-written schema, so the tables, primary keys and registration ORDER under
test are the ones the host actually runs. A local schema would prove only that
the test agrees with itself.

Scenarios: A->B, B->A (bidirectional even though only A dials, which is what
makes failover safe in either direction), LWW convergence on a contended
operation, the event union across both nodes, and a peer-offline-then-rejoin
catch-up.

The event-union scenario is the one that pins the GUID id change: under the old
autoincrement scheme both nodes independently mint id=1,2,3..., and
last-writer-wins on the primary key would silently DESTROY one node's events
rather than merge them. Asserting the union count is what catches that.

The whole suite passes in ~0.6s, which is fast enough to be suspicious of a
vacuous pass, so it was verified red-first the hard way: deliberately mismatching
the two nodes' ApiKey turns all 5 red with 2m30s of timeouts. The tests really do
run through the interceptor and the real transport.

Node B's database survives its host teardown (registered as a pre-constructed
instance, which MS.DI does not dispose), so the rejoin scenario is a genuine
rejoin rather than a fresh node.

Offline - no docker, no external services.

Verified: build 0 warnings; IntegrationTests 80/80.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
2026-07-19 09:42:21 -04:00
Joseph Doherty 98b94771ae feat(localdb): one-time migrator for the pre-Phase-1 site databases
Task 6 of the LocalDb Phase 1 adoption plan. Copies site-tracking.db and
site_events.db into the consolidated database, then renames each source to
<name>.migrated.

Runs AFTER RegisterReplicated, deliberately: capture is trigger-based, so rows
inserted before registration would never enter the oplog and would never reach
the peer - silently. There is a test asserting migrated rows land in
__localdb_oplog, because every other test in the file would still pass with that
ordering reversed.

Idempotent twice over. Tracking rows keep their existing TEXT primary key, so
INSERT OR IGNORE dedupes naturally. Legacy events had autoincrement integer ids,
which the consolidated schema replaced with GUIDs; they get DETERMINISTIC ids of
the form mig-{NodeName}-{legacyId} rather than fresh GUIDs, so a migration that
crashes between commit and rename cannot duplicate history on the next boot. The
NodeName prefix keeps the two nodes of a pair from colliding on their independent
legacy id sequences. Both properties are covered, including an explicit
crash-window test that undoes the rename and re-runs.

All-or-nothing: the copy commits in one transaction and the rename happens only
afterwards, so a failure throws out of OnReady, fails host startup, and leaves
the legacy files untouched. Deviation from the plan: it specified
BeginTransactionAsync, but OnReady is a synchronous Action<ILocalDb>. A raw
transaction on a CreateConnection() connection gives identical atomicity and
identical trigger capture without blocking on async in a startup callback.

Legacy paths come from the OLD config keys and fall back to the CWD-relative code
defaults, because that is where the old code actually wrote them. Note this means
both legacy databases have always lived outside the mounted data volume and were
already discarded on every container recreate - so on the rig this migrator will
usually find nothing, which is a legitimate no-op rather than a failure. Phase 1
incidentally fixes that data-loss bug.

Non-failure cases are covered too: missing files, a legacy file with no such
table, and in-memory legacy connection strings from test/dev configs.

Verified: build 0 warnings; Host 316/316 (9 migrator tests new).

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
2026-07-19 09:36:38 -04:00
Joseph Doherty 59c695191c feat(localdb): fail-closed auth on the sync endpoint + replication health signal
Tasks 8 and 9 of the LocalDb Phase 1 adoption plan.

Task 8 - LocalDbSyncAuthInterceptor. The replication library's LocalDbSyncService
verifies nothing; inbound auth is explicitly the host's job. Without this,
anything able to reach a site node's gRPC port could stream arbitrary rows into
the consolidated site database - including OperationTracking, which central
reconciles from.

Scoped strictly to /localdb_sync.v1.LocalDbSync/; SiteStream shares the same
AddGrpc pipeline and passes through untouched. Fail-closed: with no
LocalDb:Replication:ApiKey configured NO sync stream is accepted, authenticated
or not. That is deliberate - "no key" is the default every site node ships with,
so treating it as "no auth required" would expose the endpoint on precisely the
most common configuration. Comparison is FixedTimeEquals over UTF-8 bytes.

All four server handler shapes are gated, not just unary: the sync RPC is a
bidirectional stream, so gating only the unary path would leave the real endpoint
open while every unary test still passed. There is a test for that.

Deviation from the plan: it specified Grpc.Core.Testing for the fake
ServerCallContext. That type ships in the retired native Grpc.Core package and
does not exist on the grpc-dotnet stack this solution uses; a minimal
FakeServerCallContext in the test file was the better trade than adding a dead
dependency.

Task 9 - ISyncStatus onto the site health report as LocalDbReplicationConnected
and LocalDbOplogBacklog, via a delegate-seam hosted service following the
AddSiteEventLogHealthMetricsBridge precedent (HealthMonitoring takes no reference
on the replication library). Both are additive init properties, so the
Akka-remoted SiteHealthReport constructor signature is untouched.

Both fields are nullable and the distinction is load-bearing:
  - null = the reporter has not run / replication is not wired ("no data");
  - false/0 = a real reading. On a node with no peer that IS the healthy
    default-OFF state, not an outage.
OplogBacklog is passed through nullable end-to-end because ISyncStatus returns
null when the poll fails - flattening it to 0 would report a replication pair
that cannot read its own oplog as perfectly healthy. The collector stores both
values as one tuple and CollectReport reads it once, so a torn read cannot pair a
fresh Connected with a stale backlog.

Verified: build 0 warnings; Host 307/307 (8 interceptor + 3 health tests new),
HealthMonitoring 97/97, Commons 684/684.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
2026-07-19 09:29:22 -04:00
Joseph Doherty e62b076f2e docs(localdb): record wave 4 completion (Tasks 4, 5a, 7) + known flakes
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
2026-07-19 09:15:52 -04:00
Joseph Doherty a560e9eaf9 feat(localdb): wire 2-node replication for the site database, default OFF
Task 7 of the LocalDb Phase 1 adoption plan.

AddZbLocalDbReplication registers the engine; MapZbLocalDbSync exposes the
passive endpoint the peer node dials. Both are unconditional but INERT by
default: with no LocalDb:Replication:PeerAddress the initiating
SyncBackgroundService starts and immediately idles, and nothing dials the
endpoint. A site pair replicates only once an operator sets a peer on BOTH
nodes. The endpoint shares the Kestrel h2c listener the site gRPC server already
uses, so there are no listener or port changes.

Deviation from the plan: it put AddZbLocalDbReplication in Program.cs. Doing so
would have left it untested - the composition-root tests build
SiteServiceRegistration's graph, not Program.cs - so a registration that silently
went missing would still show green. It now sits next to AddZbLocalDb in
SiteServiceRegistration, where the DI-graph tests actually cover it.
MapZbLocalDbSync stays in Program.cs because it needs the WebApplication.

Two pins added to SiteLocalDbWiringTests, which configures no peer:
  - the engine resolves and is idle (not connected, no peer, never synced) -
    default-off means inert, not unregistered;
  - OplogBacklog is 0, not null. It is deliberately nullable so a failed poll
    reads as "unknown" instead of a healthy zero; a real 0 proves the provider is
    wired to the oplog, which Task 9's health signal depends on.

Inbound auth on the sync endpoint is Task 8. Until that lands the endpoint is
unauthenticated - which is precisely why replication ships default-OFF and no
config in this repo sets a peer.

Verified: build 0 warnings; Host 296/296, SiteEventLogging 70/70,
SiteRuntime 530/530, Commons 684/684, Communication 312/312.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
2026-07-19 09:15:26 -04:00
Joseph Doherty 0d8a80aa27 feat(localdb): rewire SiteEventLogger onto the consolidated site database
Completes Task 5a of the LocalDb Phase 1 adoption plan. The GUID-id half landed
with Task 2 (727fa48c) because leaving it out would have committed a writer that
could not satisfy site_events.id NOT NULL; this is the connection half, which
was blocked on Task 3.

SiteEventLogger owned a SqliteConnection built from SiteEventLogOptions
.DatabasePath. It now takes ILocalDb and gets that connection from
CreateConnection() - already open, pragma-configured, and carrying the
zb_hlc_next() UDF that site_events' capture triggers call. It still owns and
disposes the connection; WithConnection's signature and locking semantics are
untouched, so EventLogQueryService and EventLogPurgeService are unaffected.

The connectionStringOverride seam is deleted rather than preserved. site_events
is a replicated table, so a raw connection lacks the UDF and fails closed on
every insert - an override could only produce a logger that cannot write. It had
no callers outside this class (the connectionStringOverride hits elsewhere in the
tree belong to SqliteAuditWriter, a different type).

Tests: a shared TestLocalDb helper gives each fixture a real ILocalDb over a temp
file. Real rather than stubbed on purpose - a stub handing back a bare
SqliteConnection would pass today and fail closed the moment the table is
registered, which is exactly the silent-wiring failure mode this family has hit
three times. ServiceWiringTests' DI path also needed AddZbLocalDb, mirroring
SiteServiceRegistration.

Verified: build 0 warnings; SiteEventLogging 70/70, Host 294/294,
SiteRuntime 530/530.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
2026-07-19 09:07:49 -04:00
Joseph Doherty 0b5e9b44f6 feat(localdb): rewire OperationTrackingStore onto the consolidated site database
Task 4 of the LocalDb Phase 1 adoption plan.

OperationTrackingStore took IOptions<OperationTrackingOptions> and opened its own
SqliteConnection from ConnectionString. It now takes ILocalDb and gets every
connection - writer and the two ad-hoc reader paths - from CreateConnection().

This is not cosmetic. OperationTracking is a RegisterReplicated table, so its
capture triggers call zb_hlc_next(). That UDF is registered per connection by
ILocalDb and by nothing else, so a raw SqliteConnection would fail closed on
every write. Connections from CreateConnection() also arrive already open -
calling Open()/OpenAsync() on one throws - hence the removed OpenAsync calls on
the reader paths.

OperationTrackingOptions.ConnectionString is now vestigial for this store; the
database location is LocalDb:Path. The options class stays (retention settings)
and the config key stays bound for the site config DB.

InitializeSchema is kept but is now always a no-op in the host: onReady runs
while ILocalDb is being constructed, strictly before this constructor can
receive it. It remains so a directly-constructed store (tests, tooling) is
self-sufficient.

Tests: the store fixture moves off mode=memory&cache=shared onto a real ILocalDb
over a temp file. There is no in-memory mode - LocalDbOptions.Path is a
filesystem path - and testing through a raw in-memory connection would no longer
resemble the host. Verifier connections stay raw on purpose: this fixture never
registers the table, so no trigger fires and the UDF is never reached.

Three DI fixtures needed LocalDb:Path added, because resolving
IOperationTrackingStore now forces ILocalDb construction:
SiteCompositionRootTests, SiteAuditWiringTests, and the AuditLog
CombinedTelemetryHarness.

Verified: build 0 warnings; Host 294/294, SiteRuntime 530/530, AuditLog 354/354.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
2026-07-19 08:59:38 -04:00
Joseph Doherty 2bff527247 docs(localdb): record Task 3 completion
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
2026-07-19 03:37:03 -04:00
Joseph Doherty 2977e6c45c feat(localdb): register the consolidated site database in the composition root
Task 3. Site nodes now open one ZB.MOM.WW.LocalDb-managed database holding
OperationTracking and site_events as replicated tables.

SiteLocalDbSetup.OnReady applies both schemas and then RegisterReplicated's them.
That order is load-bearing and easy to get silently wrong: capture is
trigger-based CDC, so rows written before registration are never captured or
snapshotted - they would be invisible to the peer forever, with no error anywhere.

Storage ships UNCONDITIONALLY, no feature flag. With no peer configured LocalDb is
just a local SQLite file, so this is behaviour-equivalent to what site nodes do
today. Replication is opt-in and lands in Program.cs (Task 7), which owns the gRPC
host the sync endpoint needs.

Config: LocalDb:Path added to all EIGHT site-node appsettings (the plan said six -
docker-env2's site-x pair exists too), plus the dev template and the wonder-app-vd03
production sample. All ten are required, not optional: LocalDbOptions.Path is
validated with ValidateOnStart, so a site node without it fails fast at startup.
That is the desired posture, and it is what caught ActorPathTests - the one Host
fixture that actually starts a host - whose config now sets the path too.

Note the path choice fixes an existing data-loss bug in passing: site-tracking.db
and site_events.db both defaulted to CWD-relative paths (/app/...), outside the
mounted volume, so they were silently lost on every container recreate. The
consolidated database lives on /app/data.

Tests are container-built against the REAL SiteServiceRegistration.Configure, not a
hand-assembled ServiceCollection - this family has shipped three wiring defects that
only a real graph would catch (inert Secrets replicator 0.2.0, singleton deadlock
0.2.2, Secrets.Ui missing IAuditWriter). They assert the library's own view of what
is registered rather than that we called the right method.

Verified (all after the change):
  dotnet build ZB.MOM.WW.ScadaBridge.slnx -> 0 Error(s), 0 Warning(s)
  Host.Tests            -> 294 passed (5 new, red-first)
  SiteRuntime.Tests     -> 530 passed
  CentralUI.Tests       -> 925 passed
  Commons.Tests         -> 684 passed
  Communication.Tests   -> 312 passed
  SiteEventLogging.Tests-> 70 passed

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
2026-07-19 03:36:54 -04:00
Joseph Doherty 2742d54c9f docs(localdb): record Task 1/2/5b completion + prerequisite commits
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
2026-07-19 03:15:50 -04:00
Joseph Doherty 727fa48cba feat(localdb): move site_events to application-minted GUID ids
Tasks 2 + 5a-writer + 5b of the LocalDb Phase 1 plan, landed together because
they are one indivisible change: the schema, the writer that fills it, and every
consumer that assumed the old id semantics.

WHY the id changes. Site pairs will replicate site_events with last-writer-wins on
the primary key. With INTEGER PRIMARY KEY AUTOINCREMENT both nodes independently
mint 1, 2, 3... for unrelated events, so sync would treat them as the same row and
silently overwrite. A GUID makes the event log a pure union across the pair.

Task 2 - schema extracted so the Host's AddZbLocalDb onReady can create the tables
before RegisterReplicated installs capture triggers (pre-registration rows are
never captured):
  - new OperationTrackingSchema.Apply / SiteEventLogSchema.Apply, plain
    Microsoft.Data.Sqlite, no LocalDb dependency
  - both stores delegate their InitializeSchema to them (idempotent, so a
    directly-constructed store still works)
  - OperationTracking is unchanged - it already had a TEXT PK and replicates as-is

Task 5a - SiteEventLogger mints Guid.NewGuid("N") per event and inserts it.

Task 5b - three consumers assumed a monotonic integer id. All three move to
timestamp ordering; leaving any one behind would be a live bug:
  - EventLogQueryService: "id > $afterId" would return an ARBITRARY subset of a
    GUID-keyed table and SILENTLY DROP ROWS from page-through. Now a composite
    (timestamp, id) keyset cursor with an opaque string token; timestamps are not
    unique, so id is the tie-break that guarantees exactly-once paging.
  - EventLogPurgeService: "ORDER BY id ASC LIMIT 1000" would delete a RANDOM batch
    instead of the oldest. Now orders by timestamp.
  - EventLogEntry.Id and both ContinuationTokens: long -> string.

WIRE COMPATIBILITY. Those DTOs cross the site<->central Akka boundary
(SiteCommunicationActor -> CommunicationService -> ManagementActor / CentralUI).
No rolling-upgrade shim is needed because both sides ship in the same deployable
and the rig redeploys as a unit. Checked: no Akka serializer binding pins these
types by name. A stale numeric token degrades to "start from the beginning"
(a visible repeat) rather than throwing or losing rows.

Tests: the two that encoded the old semantics were rewritten to guard the new
invariant rather than deleted - uniqueness instead of monotonicity, and
oldest-purged-first keyed on timestamp. That second test also exposed a latent
weakness: the bulk seed stamped every row with the same UtcNow, so "oldest" was
never actually well-defined; rows now get distinct increasing timestamps, kept
inside the retention window so the retention purge does not eat them first.

Verified:
  dotnet build ZB.MOM.WW.ScadaBridge.slnx -> 0 Error(s), 0 Warning(s)
  SiteEventLogging.Tests -> 70 passed (12 new schema tests, red-first)
  CentralUI.Tests        -> 925 passed
  Commons.Tests          -> 684 passed
  SiteRuntime.Tests      -> 529 passed, 1 pre-existing flaky failure
                            (InstanceActorChildAttributeRaceTests - passes 3/3 in
                            isolation with and without this change)

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
2026-07-19 03:15:17 -04:00
Joseph Doherty f056b67e9a fix(build): suppress AngleSharp advisory instead of pinning (corrects ecf6b628)
ecf6b628 pinned AngleSharp to 1.5.2 to clear GHSA-pgww-w46g-26qg. That fixed the
build but BROKE 33 CentralUI bunit tests at runtime with

    MissingMethodException: AngleSharp.Dom.IHtmlCollection`1.get_Item(Int32)

because 1.5.x changed IHtmlCollection<T>'s indexer and bunit is compiled against
1.1.x. My mistake: I verified `dotnet build` was clean and did not run the test
suite before committing, so a runtime-only break sailed through.

There is no working patched combination upstream. Verified empirically:
  AngleSharp 1.1.2 / 1.2.0 / 1.3.0 / 1.4.0 -> still flagged NU1902
  AngleSharp 1.5.0 / 1.5.2                 -> patched, but breaks bunit at runtime
  bunit up to 2.7.2 (latest)               -> still resolves AngleSharp 1.4.0

So the real choice is a suppressed advisory or an unbuildable suite. Scoped
suppression wins here because the reach is nil: AngleSharp is an HTML parser bunit
uses to assert on rendered markup, it ships in no production project, and the only
"documents" it parses are our own components' output.

Explicitly NOT the same call as GHSA-2m69-gcr7-jv3q (SQLitePCLRaw), whose
suppression was removed in 2026-07: that one masked a vulnerability on PRODUCTION
code paths and had a working patched version available. Neither is true here.
Revisit when bunit ships against AngleSharp 1.5+.

Verified:
  dotnet build ZB.MOM.WW.ScadaBridge.slnx -> 0 Error(s), 0 Warning(s)
  CentralUI.Tests                          -> 925 passed, 0 failed (was 33 failing)

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
2026-07-19 03:14:58 -04:00
Joseph Doherty 7370b33186 feat(localdb): reference ZB.MOM.WW.LocalDb 0.1.0 packages
Task 1 of the Phase 1 adoption plan. Wiring only - no behavior change yet.

  - Directory.Packages.props: LocalDb / .Replication / .Contracts @ 0.1.0
  - NuGet.config: ZB.MOM.WW.LocalDb + ZB.MOM.WW.LocalDb.* mapped to the
    dohertj2-gitea feed (central package management requires explicit mapping)
  - SiteRuntime + SiteEventLogging: PackageReference ZB.MOM.WW.LocalDb
  - Host: ZB.MOM.WW.LocalDb + ZB.MOM.WW.LocalDb.Replication

Required two prerequisite commits, both landed ahead of this one:
  ecf6b628 - AngleSharp pin (pre-existing break; the solution would not build)
  4715f2f9 - gRPC 2.71.0 -> 2.76.0 (LocalDb.Replication's nuspec floor)

SQLitePCLRaw stays pinned at 2.1.12 (GHSA-2m69-gcr7-jv3q) - verified intact.

Verified:
  dotnet build ZB.MOM.WW.ScadaBridge.slnx -> 0 Error(s), 0 Warning(s)
  all three LocalDb packages restored from the Gitea feed

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
2026-07-19 02:39:52 -04:00
Joseph Doherty 4715f2f921 chore(deps): raise gRPC stack 2.71.0 -> 2.76.0 (Protobuf 3.34.1)
Forced by ZB.MOM.WW.LocalDb.Replication 0.1.0, whose nuspec floors
Grpc.AspNetCore / Grpc.Net.Client / Grpc.Core.Api / Grpc.Tools at 2.76.0 and
Google.Protobuf at 3.34.1. Restore fails NU1605 (package downgrade) below that
floor, so LocalDb adoption cannot proceed without it.

Landed as its own commit deliberately. The SQLitePCLRaw note in
Directory.Packages.props deferred precisely this bump as belonging in "their own
reviewed commit - not smuggled in under a SQLite security fix"; this honors that.

Scope kept minimal:
  - Grpc.Core.Api pinned explicitly so the family stays on one version instead
    of floating in transitively.
  - Microsoft.Data.SqlClient and Newtonsoft.Json, also named in that note, are
    NOT bumped - nothing forces them and they carry their own risk profile.

Direct consumer surface is two projects: Grpc.AspNetCore in Host,
Grpc.Net.Client in Communication.

Verified:
  dotnet build ZB.MOM.WW.ScadaBridge.slnx     -> 0 Error(s), 0 Warning(s)
  Communication.Tests                          -> 312 passed, 0 failed
  Transport.IntegrationTests                   -> 103 passed, 0 failed

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
2026-07-19 02:39:11 -04:00
Joseph Doherty ecf6b62850 fix(build): pin AngleSharp 1.5.2 to unbreak the solution build
GHSA-pgww-w46g-26qg (NU1902, moderate) on AngleSharp 1.1.1, pulled in
transitively by bunit into CentralUI.Tests. Under TreatWarningsAsErrors this
made `dotnet build ZB.MOM.WW.ScadaBridge.slnx` fail on a CLEAN main - the whole
suite was unbuildable and every "0 warnings / suite green" gate unverifiable.

Pre-existing, not introduced here; surfaced while starting LocalDb Phase 1,
whose per-task verification depends on a green baseline.

Same fix as the identical break in HistorianGateway (historiangw @ 6bc005d):
pin the leaf, test-only. AngleSharp is a bunit HTML-parsing dependency and
reaches no production project. Bumping bunit does not help - 2.0.33-preview is
the current preview line and still resolves the vulnerable AngleSharp.

Verified: dotnet build ZB.MOM.WW.ScadaBridge.slnx -> 0 Error(s), 0 Warning(s)
(was 1 Error before this commit).

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
2026-07-19 02:37:50 -04:00
Joseph Doherty 1556ae04e4 docs(localdb): amend Phase 1 plan - split Task 5 (event-log id change)
Execution review against the code found the integer site_events.id is
load-bearing in three places the original Task 5 did not list:

  - keyset cursor      EventLogQueryService.cs:70,141,153  (id > $afterId)
  - storage-cap purge  EventLogPurgeService.cs:164         (ORDER BY id ASC)
  - wire contracts     EventLogEntry.Id, both ContinuationTokens (long)
                       cross the site<->central Akka boundary

A GUID PK against the existing cursor silently drops rows, and against the
purge deletes arbitrary rows instead of the oldest. Task 5 is therefore split
into 5a (writer, standard) and 5b (read path + DTOs, high-risk), with 5b's
full file set spelled out and Task 10 gated on it.

Also recorded: Task 4's real registration site, and that both legacy DB paths
are CWD-relative and unset in docker today (so they live outside the data
volume and are already lost on container recreate - Phase 1 fixes that).

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
2026-07-19 02:34:38 -04:00
Joseph Doherty 84170b63ec docs(localdb): verify lib assumptions against source (CreateConnection open+UDF, ReplicationOptions binding)
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
2026-07-19 02:02:42 -04:00
Joseph Doherty b0d4e2edac docs(localdb): parallel-wave execution config, Opus implementers
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
2026-07-19 02:00:05 -04:00
Joseph Doherty ed9644fa3d docs(localdb): Phase 1 implementation plan for LocalDb adoption
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
2026-07-19 01:57:53 -04:00
Joseph Doherty 6351077898 chore(secrets): bump to Secrets 0.2.3 - visible delete modal (scadaproj#2)
0.2.3's Secrets.Ui ships the ConfirmDeleteModal's own styles under
collision-proof zb-secrets-* class names; on 0.2.1 Bootstrap's
.modal{display:none} made the delete modal permanently invisible on this
host. Also picks up 0.2.2's Akka-replicator DI-cycle fix (inert here -
ScadaBridge uses the SqlServer replicator, never affected).

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
2026-07-19 01:09:17 -04:00
Joseph Doherty fc86e8bfe1 fix(secrets): bridge the shared audit seam so /admin/secrets renders (#22)
Secrets.Ui components inject the SHARED ZB.MOM.WW.Audit.IAuditWriter seam,
which is deliberately distinct from the repo's own Commons IAuditWriter —
the G-4 adoption mounted the UI without registering it, so component
activation threw and the page 500'd on every render (hidden behind the
login wall; it plausibly never worked).

Fix: CentralSharedSeamAuditWriter forwards the shared seam into the central
direct-write path (ICentralAuditWriter -> dbo.AuditLog) — NOT the site
SQLite chain, which is never resolved on central and has no forwarder
there. Registered in the Central composition root next to the Secrets UI
authorization wiring.

Regression pin: CentralCompositionRootTests resolves the full Secrets.Ui
component injection set (ISecretStore / ISecretCipher / shared
IAuditWriter) out of the REAL Program.cs via WebApplicationFactory, plus a
type check that the seam is the central bridge. Red on the previous
commit; the defect class is invisible until the DI graph is actually
built.

Live-proven on the local docker cluster: /admin/secrets 200 + interactive
(Blazor circuit + working @onclick), and secret.add / secret.delete events
(both outcomes) landed in central dbo.AuditLog via the bridge.

Closes #22

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
2026-07-19 00:57:50 -04:00
Joseph Doherty 0083ce3560 fix(secrets): consume Secrets 0.2.1
Picks up the upstream fix for the Akka replicator's inert registration. Does not
affect this repo's SQL-Server hub path, which always registered ISecretReplicator
before AddZbSecrets and was never inert - but keeps the family on one version.

Verified: 55 projects build 0 warnings / 0 errors; Host.Tests 285/285 pass.
2026-07-18 12:19:55 -04:00
Joseph Doherty 8e12f99432 feat(secrets): opt-in SQL-Server hub replication for the host secret store
Routes both host-container secret registrations (central role in Program.cs,
site role in SiteServiceRegistration) through a new SecretsRegistration
composition seam that optionally enables ZB.MOM.WW.Secrets.Replicator.SqlServer
hub-replication mode: each node keeps a LOCAL store that syncs bidirectionally
with a shared central SQL hub, so a site cluster keeps resolving secrets
straight through a WAN outage to central.

OPT-IN GATE (the load-bearing part). AddZbSecretsSqlServerReplication validates
its options EAGERLY at registration time, so wiring it unconditionally would
make ScadaBridge fail to START anywhere Secrets:SqlServer:ConnectionString is
unset -- every dev box, every docker node, every existing deployment. The
SQL-Server package is therefore only touched when BOTH Secrets:Replication:
Enabled is true AND a non-blank connection string is present; otherwise the
registration is byte-identical to the previous plain AddZbSecrets call.
Enabled-without-a-connection-string falls back to local-only and logs a warning
rather than failing the node or silently looking healthy.

BOOTSTRAP CYCLE. The hub connection string is itself a secret and can never come
from the hub -- a node cannot read the hub to learn how to reach the hub. It must
arrive from outside the replicated set: an environment variable, or a ${secret:}
reference seeded in that node's own LOCAL store. appsettings.json therefore ships
ConnectionString empty with a _comment saying so (leaf keys starting with '_' are
skipped by the reference expander, verified in SecretReferenceExpander). No real
connection string is committed. The pre-host ${secret:} expander in Program.cs is
deliberately left on a plain local SQLite store for the same reason.

Per-node docker appsettings are intentionally NOT modified -- replication stays
off there for now.

Tests written before the wiring and confirmed red first (2 failed / 4 passed),
green after (6/6). They BUILD a container and RESOLVE from it rather than
asserting over ServiceDescriptors: a decorator can look correct as a descriptor
list and still throw on first resolve because the undecorated concrete store it
depends on is missing -- that exact defect shipped once in this library with all
descriptor-level tests green, so the undecorated SqliteSecretStore gets its own
resolution test.

Verified: Host.Tests 285/285 pass; full build (all projects except the
pre-existing AngleSharp NU1902 CentralUI.Tests restore break) 0 warnings,
0 errors.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
2026-07-18 11:10:48 -04:00
Joseph Doherty 59805f8551 feat(secrets): add ZB.MOM.WW.Secrets.Replicator.SqlServer 0.2.0 package reference 2026-07-18 05:36:02 -04:00
Joseph Doherty 2a0faeab6f fix(security): actually fix GHSA-2m69-gcr7-jv3q instead of suppressing it
The NuGetAuditSuppress in Directory.Packages.props was masking a LIVE high-severity
vulnerability, not documenting an accepted one. Only the Host project resolved a
patched SQLitePCLRaw.lib.e_sqlite3 2.1.12 (transitively, via ZB.MOM.WW.Auth.ApiKeys).
Every other SQLite consumer - AuditLog, SiteRuntime, StoreAndForward, SiteEventLogging
and 11 test projects - still resolved the vulnerable 2.1.11.

The suppression's rationale was factually wrong: it claimed 'the only patched native
lib is the SQLitePCLRaw 3.x line'. 2.1.12 patches this advisory within the 2.1.x line,
so the feared risky force-override of the whole family to 3.x is unnecessary.

Fix: explicit PackageReference to the patched 2.1.12 in each SQLite-consuming project,
plus the central PackageVersion row. Suppression removed, so the advisory is audited
again rather than silenced.

Rejected alternative: CentralPackageTransitivePinningEnabled. It clears the advisory in
one line but makes every central version a ceiling for transitive resolution, demanding
bumps to Google.Protobuf, Grpc.Net.Client, Microsoft.Data.SqlClient and Newtonsoft.Json.
That is a gRPC/data-access change to a production SCADA platform and deserves its own
reviewed commit.

Verified: forced full-solution restore reports no NU1903 (only the pre-existing,
unrelated AngleSharp NU1902 in CentralUI.Tests); all four previously-vulnerable src
projects now resolve 2.1.12; 55 projects build 0 warnings / 0 errors; 1108 tests pass
across the SQLite layer (AuditLog 354, ConfigurationDatabase 357, Security 181,
StoreAndForward 152, SiteEventLogging 64).
2026-07-18 05:30:56 -04:00
Joseph Doherty 86d48ff8a5 chore(secrets): bump ZB.MOM.WW.Secrets 0.1.2 -> 0.2.0
Version hygiene + picks up the G-8 KEK-rotation surface, and is the precondition
for adopting clustered secret replication.

NOT a security fix for this repo, despite what the original message said. A/B
against the 0.1.2 baseline shows SQLitePCLRaw.lib.e_sqlite3 already resolved
2.1.12, supplied transitively by ZB.MOM.WW.Auth.ApiKeys 0.1.5 (commit 50d79ed1).

Note: Directory.Packages.props suppresses GHSA-2m69-gcr7-jv3q and its comment
asserts 'the only patched native lib is the SQLitePCLRaw 3.x line'. That appears
incorrect for this advisory - 2.1.12 patches it. Settle before removing the
suppression; HistorianGateway's separate 3.50.3 pin cites a DIFFERENT CVE
(CVE-2025-6965), which may be the source of the confusion.
2026-07-18 05:23:42 -04:00
Joseph Doherty 50d79ed1c0 fix(deps): bump ZB.MOM.WW.Auth 0.1.3 -> 0.1.5 for the SQLitePCLRaw security fix
Auth.ApiKeys pulled SQLitePCLRaw.lib.e_sqlite3 2.1.11, which carries high-severity
advisory GHSA-2m69-gcr7-jv3q. ScadaBridge was genuinely exposed -- verified 2.1.11
resolving before the bump and 2.1.12 after, with the vulnerability scan now clean.

This jump crosses 0.1.4 (ExpiresUtc verifier enforcement), but that change was
additive: Security, Security.Tests, InboundAPI and InboundAPI.Tests all build clean
with no source changes. Suites match their documented baselines exactly --
Security 181/181, InboundAPI 269/269.

Note: a full-solution build is currently blocked by a PRE-EXISTING and unrelated
NU1902 error (AngleSharp 1.1.1 in CentralUI.Tests under TreatWarningsAsErrors),
confirmed present on a clean tree before this change.

Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
2026-07-18 03:20:14 -04:00
Joseph Doherty b1b4874090 fix(siteruntime): injectable ScriptExecutionScheduler + un-leakable expression-fault test
Gitea #18: the process-wide ScriptExecutionScheduler was reached via a static
singleton (Shared) directly inside ScriptActor, ScriptExecutionActor, AlarmActor,
AlarmExecutionActor, and ScriptSchedulerStatsReporter, so it could not be
substituted — a shared mutable global for anything running more than one logical
site in a process, most visibly the test assembly. Add an optional scheduler
injection seam to all five: the Host injects nothing and gets the shared pool
(byte-for-byte unchanged), while a test (or a future multi-site host) hands an
actor its own instance. Spawning actors thread their scheduler down to the
execution children they create. Shared() now also recreates a disposed cached
instance rather than returning it (new IsDisposed guard), so a disposed scheduler
can never silently poison every later script/alarm execution.

Gitea #16: ScriptActorTests now runs on its OWN scheduler (disposed at class
teardown), so the faulted-task test can no longer strand a worker on the
process-wide pool and starve unrelated test classes (one prior run failed 22
tests). EvalGate's wait is bounded to 30 s (was unbounded — the actual leak
mechanism) and reset per test; the teardown releases one permit per worker
instead of Release(Entries), which under-released in exactly the failure case.
Full SiteRuntime suite: 6/6 runs green (524/524); was 3/6 failing on clean main.

Fixes: #18
Fixes: #16
2026-07-17 15:37:00 -04:00
Joseph Doherty 3e84eee195 fix(dcl): route native OPC UA alarms by binding identity, not event name
A native OPC UA alarm source's SourceReference has to be two things at once:
it is parsed as a NodeId to open the monitored item, and matched as a plain-name
prefix against the event's SourceName to route the transition to an instance. No
string is both, so a NodeId-form binding subscribed correctly and then silently
dropped every transition — "pymodbus/plc/HR200".StartsWith("nsu=...;s=pymodbus/plc/HR200")
is false. Only the empty (Server-object) binding worked, because StartsWith("")
matches everything, which is why the sole live smoke test never caught it.

Each OPC UA alarm feed is opened for exactly one binding, so every transition on
it belongs to that binding. The adapter now tags each transition's routing
identity (SourceObjectReference) with the binding string verbatim via the pure
OpcUaAlarmMapper.BuildIdentity, making DataConnectionActor's routing key an exact
match regardless of whether the binding is stored as ns=<index> or the durable
nsu=<uri> form. The Server-object aggregate feed keeps an empty routing identity,
so it reaches only "mirror everything" subscribers and never leaks into a
specific-node binding. The per-condition SourceReference key stays the readable
SourceName.ConditionName, so persistence and display are unchanged, and MxGateway
is untouched — its bindings are names and its mapper already emits matching names.

Unblocked by lmxopcua#473 (OtOpcUa now populates SourceNode/SourceName/EventType
on conditions); SourceName is the RawPath, so the per-condition key is unique.
Live end-to-end verification against native alarms still needs a v3 rig.

Fixes: Gitea #17
2026-07-17 15:07:43 -04:00
Joseph Doherty 30196d1ab8 feat(dcl): bind OPC UA nodes by namespace URI, not baked index
A stored ns=<index> reference is only meaningful against one server's namespace
table. A server that adds, removes or reorders a namespace silently re-points
every binding: best case BadNodeIdUnknown, worst case the index now names a
different namespace holding a colliding identifier and the binding resolves to
the wrong node with Good quality. Nothing could detect that, because ScadaBridge
stored no namespace URI anywhere.

OtOpcUa v3.0 makes this concrete: v2's sole custom namespace and v3's raw
namespace both sit at index 2, so v2-era bindings resolve against v3 without
error while meaning something else entirely.

Adds OpcUaNodeReference as the single translation seam between stored references
and the wire. Resolve() accepts both ns=<index>;s=<id> (existing bindings, which
keep working unchanged) and the durable nsu=<uri>;s=<id>, mapping the URI to the
live index at use time; it is wired into every parse site — subscribe, read,
write, alarm-subscribe and browse. An unpublished URI now throws naming the URI
rather than binding to whatever occupies that index, and a svr= reference to
another server is rejected instead of being resolved against the wrong address
space.

The browser emits ToDurable(), so what the picker shows is what gets stored and
newly-authored bindings are index-proof from the start. That also closes a
round-trip gap: browse previously emitted ExpandedNodeId.ToString(), which for a
URI- or server-index-carrying reference produced a string NodeId.Parse could not
read back — the same method already resolved it correctly 57 lines later.

Bindings stored before this change keep their ns= form and keep working; they are
only as durable as the server's namespace order. Re-authoring against the picker
is what makes them durable, and that re-bind still needs a live v3 rig.

Refs: Gitea #14
2026-07-17 15:07:43 -04:00
Joseph Doherty 4d869de9c2 docs: sister-repo index update + working notes
CLAUDE.md: the historian SDK is now owned inside HistorianGateway at
histsdk/ (the separate ~/Desktop/histsdk repo was folded in with history),
so the sister-project list no longer points at a standalone repo.

Also checks in working notes that were sitting untracked in the tree:
deferred.md (remaining deferred work snapshot), ScadaBridge-docs-fixed.md and
ScadaBridge-docs-issues.md (documentation-analysis report output).
2026-07-16 23:31:34 -04:00
Joseph Doherty d2a6107cdb test(host): serialize Central-boot fixtures to close env-var race
CentralDbTestEnvironment sets five process-wide environment variables, and
Program's AddEnvironmentVariables() reads them at an unpredictable point during
host boot. With xUnit collection parallelization on, one fixture's teardown
could clear a var mid-boot for a sibling. Since the secrets adoption (G-4)
three of those keys are ${secret:...} references that fail closed, turning a
previously benign empty value into a SecretNotFoundException that aborts the
boot — an intermittent CI failure.

Adds a HostBootCollection that serializes every fixture booting a real host
while depending on that shared state, folding in the narrower "ActorSystem"
collection so its members stay serialized with each other as before. Site-role
fixtures stay parallel: they call Configuration.Sources.Clear(), dropping the
env-var provider, so they cannot participate in the race.

CentralDbTestEnvironment now also fails fast if two instances are ever live at
once, making a regression (a fixture added outside the collection) deterministic
rather than intermittent — this is what surfaced the disposed-CTS defect fixed
in the previous commit. Fixture teardown is try/finally so a throwing host
teardown can no longer strand the vars for the rest of the run.

Cost: Host.Tests runs ~2m30s -> ~4m10s; the serialized Central-boot classes are
most of the assembly's parallelism.

Fixes: Gitea #15
2026-07-16 23:31:28 -04:00
Joseph Doherty 9110a4eb01 fix(auditlog,health): harden hosted-service shutdown against disposed CTS
The host does not guarantee IHostedService.StopAsync is driven before the DI
container is disposed — WebApplicationFactory's teardown reaches Dispose first
— so cancelling the internal CTS from StopAsync threw ObjectDisposedException
and aborted the host's whole shutdown sequence. Four services shared the same
copy-pasted lifecycle and the same two races: StopAsync cancelling an already-
disposed CTS, and StartAsync reading _cts.Token lazily inside the Task.Run
lambda, which faults the loop task the host awaits when Dispose wins that race.

Each service now captures the token on the caller's thread, tolerates a
disposed CTS, and cancels-before-disposing so the loop is always signalled and
its pending Task.Delay sees a cancelled token rather than a dead source.
SiteAuditBacklogReporter also gains the outer OperationCanceledException guard
its sibling SiteAuditRetentionService already carried (arch-review 04 R2, R7),
without which a shutdown landing mid-probe threw TaskCanceledException out of
Host.StopAsync.

Surfaced while verifying the Gitea #15 test-harness fix: in Host.Tests the
aborted teardown skipped the fixture's env-var restore, contaminating every
later test in the run.

Refs: Gitea #15
2026-07-16 23:31:19 -04:00
Joseph Doherty 128f159692 feat(secrets): resolve MxGateway ApiKey secret: refs at connect time (ScadaBridge G-3 T9) 2026-07-16 15:37:01 -04:00
Joseph Doherty 56b428d2ef docs(secrets): clustered master-key posture runbook + Central pointer (ScadaBridge G-5 T8) 2026-07-16 15:21:54 -04:00