ef03a0dca784a4b036e267bef85d20132d5cc127
107 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Joseph Doherty
|
bc29dc412b |
fix(secrets): review round 1 — pre-host expander reads the same store the runtime serves, raw-string Replication:Enabled parse, leftover-key warning
- SecretsRegistration: UsesCentralSharedStore predicate shared by AddScadaBridgeSecrets and the
new AddPreHostExpander, so a Central node's Layer-A ${secret:} expander composes against the
shared SQL store whenever Secrets:Store:Provider=SqlServer does — closing the gap where a
central migrated off the retired Grpc topology would boot-time-resolve config from per-node
SQLite while serving the shared store.
- Program.cs: expander block now calls AddPreHostExpander + resolves the store-agnostic
ISecretsStoreMigrator seam; stale Grpc-hub/replication comment rewritten.
- DeployedSecretsMirror.cs: stale "gRPC-hub follower half" doc comment rewritten to state the
now-permanent local-SQLite-only invariant.
- StartupValidator: Secrets:Replication:Enabled read as a raw string (any non-empty, non-"false"
value rejected) instead of GetValue<bool>, which threw a raw binder crash on "yes"/"1"; added a
CollectWarnings entry for leftover Secrets:Replication/Secrets:GrpcHub child keys once Enabled
is false/unset.
- Tests: +15 (AddPreHostExpander/UsesCentralSharedStore wiring, Replication:Enabled edge values,
leftover-key warnings). Full solution build and full Host.Tests green (593/593).
|
||
|
Joseph Doherty
|
2bac13170c | feat(secrets)!: retire the gRPC secrets hub in ScadaBridge — sites get secrets via Deploy Artifacts; rig central on the shared SQL store | ||
|
Joseph Doherty
|
a35ee663c8 | feat(site): deployed_secrets (LocalDb-replicated ciphertext) + node-local mirrored_secrets ledger | ||
|
Joseph Doherty
|
a57c36f504 | fix(secrets): review round 1 — gate the clustered-central secrets warning on Replication:Enabled, name the real trigger in the shared-store connection-string message, pin Grpc-vs-Store:Provider precedence | ||
|
Joseph Doherty
|
35235ecfed | feat(secrets): Secrets:Store:Provider=SqlServer — central pair shares one SQL secret store without the hub | ||
|
Joseph Doherty
|
9ee402bbcd | Resolve all 46 low-severity architecture review findings | ||
|
Joseph Doherty
|
9a87d7a328 |
Resolve all 33 medium-severity architecture review findings
Second slice of the 2026-09-09 whole-system review remediation
(docs/plans/2026-09-09-arch-review.md §10), on top of the High slice
|
||
|
Joseph Doherty
|
146a13238f | Resolve all high-severity architecture review findings | ||
|
Joseph Doherty
|
d6fd8c400c |
feat(host): StartupValidator warns when a clustered central keeps per-node inbound API keys; rejects an unknown ApiKeyStore:Provider at boot; Program.cs logs CollectWarnings once Serilog is up
Claude-Session: https://claude.ai/code/session_013L3XdmVii9i5sY4FChU8pB |
||
|
Joseph Doherty
|
06be0b9e64 |
feat(host): inbound API key store provider — SqlServer shares one zbauth row set across the central pair; one-shot SQLite import at boot (file renamed .imported, one retry on a concurrent-peer PK race, failure never blocks boot)
Claude-Session: https://claude.ai/code/session_013L3XdmVii9i5sY4FChU8pB |
||
|
Joseph Doherty
|
afe59a5518 |
chore(host): pin ZB.MOM.WW.Auth 0.3.0 + ApiKeys.SqlServer; InboundApiKeyStoreOptions (Provider Sqlite|SqlServer, ReadProvider fails the boot on an unknown value)
Claude-Session: https://claude.ai/code/session_013L3XdmVii9i5sY4FChU8pB |
||
|
Joseph Doherty
|
25c15dd164 |
ops(docker)+test(host)+docs: close register rows 104 + 105 — offline VACUUM recipe applied to the rig; build-time pin that no replicated table is a cascading-FK parent; record row 106 (LocalDb as_of_seq=0 full-snapshot loop)
Row 104: docker/vacuum-site-localdb.sh stops both nodes of a pair together, sets auto_vacuum=INCREMENTAL + VACUUM on each node volume with integrity_check before/after, then starts both (DRY_RUN=1 reports). Run on all three rig pairs: six files 0 -> 2, ok, ~11% smaller; site-a replication reconnected with backlog 0 and no watermark change — VACUUM is replication-invisible (ledger keys by declared PK, oplog seq is INTEGER PRIMARY KEY AUTOINCREMENT). Topology guide gains the recipe. Row 105: SiteLocalDbReplicatedTableFkGuardTests builds the real site schema via OnReady (unreplicated config so 0.3.3's own guard cannot mask the culprit) and asserts every SiteLocalDbSetup.ReplicatedTables entry (now internal) exists and has no inbound CASCADE/SET NULL/SET DEFAULT foreign key, with a negative control. The constraint is enforced at build instead of at the first replicated boot. Row 106 (found during 104's live verification, NOT caused by it — the identical 127k-row full snapshot also landed at the earlier deploy restart): SnapshotStreamer.SendAsync stamps as_of_seq = MAX(seq), which is 0 once a converged sender's oplog is ack-pruned to empty; a zero-watermark receiver (rebuilt node) is never lifted, so the empty-oplog horizon rule re-streams a full snapshot on every reconnect. Sender-side one-line fix for LocalDb 0.3.4: as_of_seq = max(MAX(seq), last_acked_seq). |
||
|
Joseph Doherty
|
428b8db024 |
test(dcl+communication+storeandforward+kpihistory+host): close register row-40-class sleeps + row-32 provider leak (arch-review 2026-08-19 P8-8 + P8-9)
P8-8: replace the remaining wall-clock sleep-then-assert instances outside Communication.Tests' Medium-wave sweep with the same row-40 house pattern — continuous polling (AssertHoldsFor) for ABSENCE assertions instead of a single end-of-window sample, which cannot distinguish "stayed quiet throughout" from "hadn't happened yet when the one sample landed" and cannot catch a value that flips wrong and back before that sample. - DataConnectionActorBatchTests: QualityCounters_ArePushedOnTransitionOnly_AndCoalesced (:206/:214 class) converted to AssertHoldsFor; ResolutionProbes_BackOff... (:181/:184, a genuine real-time backoff-RATE assertion, not a sync-primitive stand-in) converted to AssertRoundsWithinWindow, which enforces the ceiling continuously (fails fast on a no-backoff regression) while the floor still necessarily waits out the window. - SiteAlarmLiveCacheServiceTests (:452-454, :470-472): converted to the project's own AssertHoldsFor (mirrors SiteAlarmAggregatorActorTests', same assembly). - StoreAndForwardServiceTests (:891-892): async AssertHoldsForAsync variant (the class runs on the async test thread). - KpiHistoryRecorderActorTests (:1043-1049): two skip-path FoldCount==0 absence checks converted. Each helper is a private per-file copy, matching the Medium-wave T17 precedent (SiteAlarmAggregatorActorTests.AssertHoldsFor) rather than a shared TestSupport utility. Proof of the mechanism: a standalone (non-repo) harness run 100 trials of a real background thread flipping a boolean true then back to false inside a 300ms window — Thread.Sleep(300)+one sample missed the violation 100/100 times (false pass), continuous 20ms polling caught it 100/100 times. Most of the actual conversions here guard monotonic counters (their single-sample variant would eventually see the final value too), so each converted test carries an inline honesty note on what specifically the conversion buys for that assertion (fail-fast vs. true flip-back detection). P8-9: ScadaBridgeTelemetryTests.EmitHelpers_DoNotThrow called SetQueueDepthProvider(() => 5) against the process-global row-32 provider slot with no clear and no [Collection] — a landmine for any future same-process reader of scadabridge.store_and_forward.queue.depth. Fixed with compare-and-clear in a finally via the existing (production, untouched) ClearQueueDepthProvider. Searched this assembly and the whole test tree for a second same-process reader that asserts on the value: none exists today (MetricsEndpointTests scrapes /metrics but only asserts on the response shape, and the only value-asserting test, QueueDepthGaugeTests, lives in StoreAndForward.Tests — a separate test-host process the leak cannot reach), so no [Collection] was added; the test's doc comment records the search and flags it for re-check if that changes. Test-only change, zero production edits. Full suites green: DCL.Tests 342/342, Communication.Tests 729/729, StoreAndForward.Tests 135/135, KpiHistory.Tests 54/54, Host.Tests 522/522. Each converted test additionally looped 20x in isolation with zero flakes. |
||
|
Joseph Doherty
|
abbbcd1a8d |
fix(kpihistory)+fix(notificationoutbox)+fix(sitecallaudit)+fix(host): P5-5 — rate-fold interval triple made self-consistent + cross-validated
Defect: the hourly KPI rollup fold (KpiHistoryRepository.FoldHourlyRollupsAsync) sums every Rate-classified sample recorded in an hour, correct only if each sample is a non-overlapping delta covering exactly one recorder tick (KpiHistoryOptions.SampleInterval). That invariant was an unenforced silent assumption across three independently-owned config keys — SampleInterval, NotificationOutboxOptions.DeliveredKpiWindow, SiteCallAuditOptions.KpiInterval — so widening DeliveredKpiWindow to e.g. 15 min would silently over-count the rollup ~15x. Robust fix: NotificationOutboxKpiSampleSource and SiteCallAuditKpiSampleSource now rescale their Rate-classified samples (deliveredLastInterval, failedLastInterval) by SampleInterval / (DeliveredKpiWindow or KpiInterval) before recording, so the rollup's per-hour Sum is correct regardless of how the other two keys are tuned — the live dashboard tile's own cutoff/window is untouched. Both sources gain a project reference to KpiHistory to read SampleInterval; no cycle (KpiHistory references only Commons). Eager cross-validation (defense-in-depth): KpiRateFoldIntervalCrossValidator (Host, the one composition root referencing all three component projects) fails boot naming all three config keys when they diverge, protecting the separate (still-real) invariant that the recorded/charted KPI-history sample equals the live tile at the same instant — which the rescale trades away whenever the scale factor isn't 1. Rides KpiHistoryOptions' existing ValidateOnStart trigger. Also: NotificationOutboxOptions had no registered validator at all before this (the gap this task's "whichever options class owns DeliveredKpiWindow" file scope pointed at) — added NotificationOutboxOptionsValidator mirroring SiteCallAuditOptionsValidator's positive-duration/count convention, wired with ValidateOnStart. Tests: new rescale tests in both sample-source suites prove the fold is numerically correct at a non-default SampleInterval (30/40 counted over a 15/20-minute window correctly divides down to a 1-minute-tick delta); new KpiRateFoldIntervalCrossValidatorTests prove divergent config fails with all three keys named; new NotificationOutboxOptionsValidatorTests. Every new test proven to fail against the unfixed computation first (temporarily short-circuited the scale factor / cross-check to the pre-fix behavior, confirmed failure, restored) — not via git stash (repo-wide, shared across concurrent worktrees this session) but via file-copy swap. Full suites green: Host.Tests (527), ConfigurationDatabase.Tests (401), KpiHistory.Tests (54), NotificationOutbox.Tests (163), SiteCallAudit.Tests (67). |
||
|
Joseph Doherty
|
11675285af |
fix(siteeventlog)+fix(host): purge startup yield + auto_vacuum reality (arch-review 2026-08-19 P4-5 + P4-6)
P4-5: EventLogPurgeService.ExecuteAsync now yields (Task.Yield()) before running the initial RunPurge(), making "the purge never runs synchronously on the caller's thread" a property of the method itself rather than an artifact of .NET 8+'s BackgroundService scheduling (verified empirically not to reproduce the literal "StartAsync blocks" symptom on this repo's net10.0 target — a 1.5s synchronous Thread.Sleep in ExecuteAsync still lets StartAsync return in ~2ms). Proven instead via a reflection-invoked ExecuteAsync test that bypasses BackgroundService's Task.Run wrapper, which is timing-independent. Also: a throwing isActive predicate now fails CLOSED (logged as inactive, at Warning) instead of silently claiming "active" — the predicate only affects the tick's log line since P4-4, but a throw must not masquerade as a successful check. P4-6: PRAGMA auto_vacuum = INCREMENTAL only takes effect on a table-less database, and LocalDbSchema.EnsureCreated (inside SqliteLocalDb's constructor) always creates LocalDb's own bookkeeping tables before SiteEventLogSchema.Apply ever gets a connection — so the pragma there had never once taken effect, in any real call path. Real fix per plan decision 2: SiteLocalDbSetup.EnsureAutoVacuumOnFreshFile pre-creates a genuinely fresh LocalDb file with the pragma set, called from SiteServiceRegistration.Configure BEFORE AddZbLocalDb's lazy singleton ever constructs SqliteLocalDb — no LocalDb API change needed. Only helps NEW site databases; an existing file (every node on the docker rig today) is left untouched rather than VACUUM'd, per the plan's explicit rejection of an unconditional VACUUM against a live ~1 GB file. The now-dead pragma call in SiteEventLogSchema.Apply is removed and its comment corrected. Tests proven to fail against the unfixed code (git-diff-and-restore, not stash — this repo shares one stash ref across all worktrees): ExecuteAsync_YieldsBeforeRunningTheInitial- PurgeSynchronously and RunPurge_ActiveNodeCheckThrows_TickLogsAsInactive_NotActive fail without the EventLogPurgeService fix; SiteLocalDbAutoVacuumTests fails to compile without EnsureAutoVacuumOnFreshFile. SiteEventLogging.Tests 112/112 and Host.Tests 526/526 green with the fix applied. |
||
|
Joseph Doherty
|
b68f553f1e | fix(host): h2 keepalive on site gRPC listener (arch-review 2026-08-19 P4-3) | ||
|
Joseph Doherty
|
6c11362d9f |
fix(auditlog)+fix(siteeventlog): complete audit write tasks only after commit; run event-log retention on standby (arch-review 2026-08-19 P4-2 + P4-4)
P4-2: SqliteAuditWriter.FlushBatch completed each pending row's Task (TrySetResult) inside the per-row insert loop, before transaction.Commit() ran. A later failure in the same batch (another row's insert throwing, or the commit itself failing) rolled the whole transaction back, but the earlier rows' Tasks had already completed successfully -- the caller and FallbackAuditWriter (whose ring-buffer fallback exists precisely for this outage) both saw false success while the rows were actually gone from the database; the outer catch's TrySetException became a silent no-op against an already-completed TaskCompletionSource. Every row's Completion is now deferred and only resolved (TrySetResult for the whole batch, or TrySetException for the whole batch) after transaction.Commit() succeeds or the transaction rolls back, mirroring SiteEventLogger.FlushBatch. P4-4: EventLogPurgeService gated the daily site_events purge on the active-node check, early-exiting on the standby, on the stale premise that "the standby receives no writes". DataConnectionActor runs on both site nodes and writes site_events regardless of active/standby status (register rows 89/90/97), so an unreplicated standby (the rig's site-b/site-c posture) grew its site_events file forever with no retention and no cap. The gate is removed; every node now purges its own file every tick. Safe on a replicated pair because the retention/cap deletes are CDC-captured and converge under LWW regardless of which node issues them; required on an unreplicated pair because each node's file is independent. The active-node check is retained only to annotate the purge tick's log line. Tests: new SqliteAuditWriterCommitFailureTests forces a genuine non-swallowed mid-batch INSERT failure via a real SQLite trigger + custom scalar function on the writer's own connection (reflection seam), proving (a) a row whose own insert succeeded still faults when a later row in the same batch fails, and (b) FallbackAuditWriter rings every faulted row and increments the failure counter for each -- both assertions fail against the pre-fix code (proof-of-failure verified by temporarily reverting the fix). EventLogPurgeServiceTests' standby case is inverted to assert the purge now runs (not skips) on a standby node. |
||
|
Joseph Doherty
|
d2d4db492b |
test(infra): close register row 96 — per-suite xunit parallelism caps for the resource-heavy suites
maxParallelThreads was unset everywhere and only 3 of ~20 test projects carried an xunit.runner.json at all, so xunit ran Environment.ProcessorCount collections at once (14 on the dev box) in every suite. That is the general load multiplier behind the row-82/91/95 flake family. Policy: per-suite caps, NOT a global one. Pure unit suites stay uncapped for speed; a suite gets a config only when a test CLASS costs a real external or OS-level resource, so ProcessorCount of them at once is a resource question rather than a CPU question. Four suites qualify, all capped at maxParallelThreads: 4. - ConfigurationDatabase.Tests, AuditLog.Tests, SiteCallAudit.Tests — verified as the three that actually take MsSqlMigrationFixture (the latter two by project reference). Every MSSQL-backed test class means a CREATE DATABASE plus the ~100-migration chain against the one shared dev SQL Server. 4 is chosen to MATCH the default slot count of row 82's cross-process sp_getapplock migrate semaphore, so the in-process and cross-process limits agree rather than fighting. - Host.Tests — classes boot real ASP.NET Core hosts and real ActorSystems; Akka.Remote binds real sockets, Kestrel binds real ports, gRPC dials them. Threads are capped, collections are NOT serialized. IntegrationTests (parallelizeAssembly/parallelizeTestCollections false) and CentralUI.PlaywrightTests (maxParallelThreads 1) already carry stricter pre-existing settings and are left alone — the Playwright suite drives a browser against a live rig, where shared UI state, not CPU, is the constraint. Each capped csproj gains <Content Include="xunit.runner.json" CopyToOutputDirectory="PreserveNewest" />; the SDK does not copy it automatically and without it the file is silently ignored. Verified by moving the copied file out of bin/ to take an uncapped baseline: ConfigurationDatabase.Tests ran at 216 % CPU uncapped vs 145 % capped, so the cap demonstrably takes effect. Cost is close to nil — these suites are latency-bound on an external resource, not CPU-bound (14-25 % utilisation for the two slowest). Standalone, uncapped -> capped: AuditLog 383 tests 1m52s -> 1m51s; Host 520 tests 4m05s -> 4m08s; ConfigurationDatabase 396 tests 10s -> 12s; SiteCallAudit 66 tests 8s -> 8s. All green. docs/test_infra/test_infra.md gains a Test Parallelism Policy section: the two axes (in-process collection threads = this cap; cross-process solution-level dotnet test = unbounded in general, bounded on the DB-migration path by the sp_getapplock semaphore), the per-suite table with reasons, the rules (cap threads rather than serialize; a cap is not a substitute for test isolation; the Content copy is required), and the default for new projects (no config until the project acquires a per-class external resource). test_infra_db.md cross-links it from the semaphore bullet. |
||
|
Joseph Doherty
|
4d39ad3cd7 |
Merge rows 80+73+71: nullable LocalDb health fields on unreplicated sites (Host supplies ReplicationIsConfigured), StartupReconcilerActive on the health report (proto field 29) with central No-self-heal badge, standby active-node ERR downgraded via narrowly-keyed Serilog filter
# Conflicts: # CLAUDE.md |
||
|
Joseph Doherty
|
8b761841bc |
fix(logging): row 71 — stop logging the standby's by-design active-node verdict at ERROR
A healthy standby logged "Health check active-node with status Unhealthy … 'not the site pair primary (standby)'" at Error on every probe — every 10s on a site node, and (discovered while fixing this) every 5s on a central standby, which Traefik health-probes at that cadence. Correct-by-design state at an alarming severity: it floods the log and trains operators to ignore ERR. Mechanism: DefaultHealthCheckService logs one line per check per probe and picks the level from the verdict — Debug/Warning/Error for Healthy/Degraded/Unhealthy. Both active-node checks answer Unhealthy on the standby because 503 IS the contract /health/active publishes. There is no IHealthCheckPublisher in this repo, so each line is one inbound HTTP probe. Fix: one Serilog exclusion filter in LoggerConfigurationFactory. A level override could not work — Serilog's MinimumLevel.Override only RAISES a per-source floor, and a LogEvent's Level is immutable, so neither an override nor an enricher can rewrite Error as Information. Changing the check to report Degraded was rejected: it would alter the HTTP status that /health/active publishes, which is load-bearing for Traefik and the family dashboard. The constraint (genuine failures must still log at Error) is preserved by keying the predicate on the check NAME rather than the logger category: localdb, akka-cluster, database and required-singletons keep their Error lines, and the active-node check has no error path of its own — the active node always answers Healthy — so the only event this can ever suppress is a standby reporting that it is a standby. The verdict stays fully observable via /health/active's own HTTP status, the readiness probe's cluster view, and the health dashboard. Also corrects a stale CLAUDE.md claim found while verifying what consumes this check: "/health/active is central-only (site nodes map no /health/* at all)" is wrong — SiteServiceRegistration registers akka-cluster + localdb + the role-scoped SitePairActiveNodeHealthCheck, and Program.cs's Site branch calls MapZbHealth() on the HTTP/1.1 listener. Fail-before evidence: with the filter behaviourally disabled, StandbyActiveNodeVerdict_IsNotLogged fails "Collection was not empty" while the other 9 tests in the file — the must-still-log constraints — pass either way, which is what proves the filter is narrow rather than a blanket suppression. Tests: Host 518 green (10 new). |
||
|
Joseph Doherty
|
a2f0086fa1 |
fix(health): rows 80+73 — unconfigured replication reports null, reconciler presence surfaced
ROW 80 (observability correctness). An unreplicated site published
LocalDbReplicationConnected=false / LocalDbOplogBacklog=0, contradicting the
nullable semantics every doc and the proto already state ("null means no
reading, NOT disconnected with an empty backlog").
Root cause: the replication engine is registered unconditionally and idles
without a peer, so ISyncStatus answers false/0 on a node that was never meant to
replicate, and LocalDbReplicationStatusReporter passed that straight through.
Two consequences: an operator read connected:false on a healthy site as a fault,
and central's null branches — row 33's conditional KPI emission and
sustained-badge anchor exclusion — never received the null they exist for, so
rig site-b/site-c emitted a permanently misleading
localDbReplicationConnected=0 series.
Fix mirrors the SiteEventLogReplicationCheck precedent: the Host evaluates
SiteLocalDbSetup.ReplicationIsConfigured ONCE and passes it to
AddLocalDbReplicationHealthBridge as a third provider; unconfigured
short-circuits the probe to (null, null). HealthMonitoring never reads LocalDb
config itself, and the same captured value now also feeds
SiteEventLogReplicationCheck so the two surfaces cannot disagree.
ISiteHealthCollector.SetLocalDbReplicationStatus widens to bool? — the
non-nullable bool was the type-level root cause (null was inexpressible).
false now means exactly one thing: configured, peer session down = a real fault.
Wire/central/UI needed no change: the proto already carries BoolValue/Int64Value
wrappers and the aggregator + KPI source already branch on null. This makes code
match doc, so no doc claim was weakened.
Watch the trap that caused this: configured-ness is PeerAddress OR ApiKey (only
the dialling half sets a peer address). The pre-existing SiteLocalDbWiringTests
fixture sets ApiKey and IS a replicating node, so its false/0 assertion is
correct and is kept, with a comment saying so; the new
UnreplicatedSiteLocalDbHealthTests fixture is the genuinely-unreplicated one.
ROW 73 (silent loss of self-heal). AkkaHostedService skips
SiteReconciliationActor when no IDeploymentConfigFetcher is resolvable, and that
skip was a lone boot Warning — a node with no self-heal path at all went on
reporting itself perfectly healthy forever.
Adds tri-state SiteHealthReport.StartupReconcilerActive (null = not reported,
i.e. the central self-report or a pre-row-73 site build; true = running;
false = booted without it), latched once at boot by the Host in both branches.
Additive proto field 29 (BoolValue, vendored C# regenerated via
docker/regen-proto.sh centralcontrol; csproj restored, verified empty diff).
The skip branch also raises its log to Error — no configuration makes the
fetcher legitimately absent on a site node — and writes a best-effort
deployment/Error site event so the evidence outlives the boot log.
Central treatment: a "No self-heal" WARNING badge on the site tile, not a
failure. Config still reaches the node via LocalDb replication while it is up;
what it lost is recovery from a deploy that landed while it was down — degraded,
not offline. Only the explicit false is badged (badging null would light up the
whole fleet the moment central is upgraded ahead of the sites), and offline
sites are skipped, matching the replication-backlog badge.
Fail-before evidence: with the row-80 gate behaviourally reverted,
UnreplicatedSite_HealthReport_ReportsNullForBothLocalDbFields fails
"Expected: null / Actual: False" and 3 of the 8 new reporter unit tests fail.
Tests: HealthMonitoring 128, Host 518, CentralUI 1051, Communication 717,
Commons 715, SiteEventLogging 81 — all green.
|
||
|
Joseph Doherty
|
9d668861ad |
test(host): row 62 — pin ISiteEventLogger's Singleton lifetime in the site graph
Six SiteRuntime call sites resolve ISiteEventLogger straight off the root
IServiceProvider with no scope: AlarmActor (cached in a field for the actor's
lifetime), ScriptActor (x2), InstanceActor (x2), NativeAlarmActor,
DeploymentManagerActor, and Scripts/ScriptRunLauncher. All correct today, and
correct for exactly one reason — the registration is AddSingleton, so a root
resolve yields what a scope would.
That makes the lifetime a load-bearing invariant with no enforcement anywhere.
If it ever changes, all six silently become register row 45, and nothing in the
tree notices: ValidateOnBuild walks CONSTRUCTOR graphs and these are imperative
GetService calls in method bodies; ValidateScopes would fire, but at runtime on
a site node on whichever path logs an event first, not at boot and not in CI;
CaptiveDependencyGuard catches the AlarmActor field case by design but cannot
see the five per-call resolves at all. This pin is the tripwire, and its failure
message names the six call sites that must be fixed alongside any such change.
Asserts the REGISTRATION, not the call sites — the SiteRuntime actor files are
correct as written and are untouched here.
Two facts, not one, because the interface registration is a FACTORY
(AddSingleton<ISiteEventLogger>(sp => sp.GetRequiredService<SiteEventLogger>())):
a Singleton lifetime on the interface alone does not prove callers share an
instance, since downgrading the concrete SiteEventLogger would make that
singleton factory resolve a scoped service from the root.
Fail-before, both halves, by temporarily flipping the real registration:
- AddScoped<ISiteEventLogger> -> first pin FAILS, "Actual lifetime: Scoped",
with the full row-62 pointer in the message.
- AddScoped<SiteEventLogger> (interface left Singleton) -> the first pin stays
GREEN and only the second catches it. That is the evidence the second fact
is load-bearing rather than redundant.
Registration restored; both green.
Registration-only inspection — the provider is never built, so the pin costs
~30 ms and opens no SQLite file.
Host.Tests: 507/507 green.
|
||
|
Joseph Doherty
|
c26b52ad68 | merge: observability (register rows 33, 54, 56, 63, 66) — LocalDb backlog KPI history + sustained-threshold badge, honest stream-subscriber gauge rename + real data-connection gauges, central metrics port on the rig, ASPNETCORE_ENVIRONMENT on site containers, import-session rejection counter | ||
|
Joseph Doherty
|
92fe5acca7 |
refactor(auditlog+sitecallaudit): delete the _injectedRepository test ctors — register row 61
AuditLogIngestActor and SiteCallAuditActor each exposed constructors taking a scoped
repository and caching it on a field for the actor's lifetime. Safe only because no
production call site picked those overloads — i.e. one call site away from re-introducing
the row-45 captive-dependency bug — which is why CaptiveDependencyGuard had to allowlist
both fields.
Preferred fix applied: the test fixtures fold onto the IServiceProvider overload with a
stub provider, and the constructors are gone.
Product deletions:
- AuditLogIngestActor: the (IAuditLogRepository, ILogger) ctor, the _injectedRepository
field and its branch in OnIngestAsync, and OnCachedTelemetryWithoutDualWriteAsync (the
"this ctor cannot dual-write" fallback handler that only that ctor installed).
_serviceProvider is now non-nullable and the null-forgiving operators are gone.
- SiteCallAuditActor: both test ctors (repo-only, and repo + reconciliation
collaborators), the _injectedRepository field and its four branches, and the private
SerializedRepository class. _backgroundTimersEnabled went with them: only the deleted
repo-only ctor ever set it false, so it was a constant and its two guards are removed.
The reconciliation tick is still gated — on the collaborators being resolvable, which
is how production reaches the same state.
Test fixtures (12 call sites): two small TestSupport helpers build the stub providers.
SiteCallAuditActor's production wiring resolves a FRESH repository per scope, so the
serializing decorator existed only to protect fixtures that share one DbContext-backed
instance across every scope — it moves into the test project as
SerializingSiteCallAuditRepository, which is where the hazard is actually created.
Fixtures that relied on it keep it. The relay test's ICentralAuditWriter now arrives via
the provider, exactly as production resolves it.
Behaviour is preserved on the paths under test: the deleted audit ctor passed
redactor:null / failureCounter:null, and a provider registering neither resolves both to
null, so the SafeDefaultAuditRedactor fallback and counter-free logging are unchanged. No
fixture on the folded sites sends IngestCachedTelemetryCommand, so swapping to the real
dual-write handler is inert. The SiteCallAudit purge timer now arms in fixtures that
previously had timers off, but every one of them uses default purge settings whose first
tick is 5 minutes out — nothing fires inside a test.
One reconciliation test was renamed and rewritten rather than deleted:
TestCtor_RepositoryOnly_DoesNotStartReconciliationTick became
NoReconciliationCollaboratorsRegistered_DoesNotStartReconciliationTick — the same
guarantee, expressed the way production actually reaches it.
Guard: the allowlist is now EMPTY. The staleness self-test (every entry must still match a
flagged field) is replaced by two stronger pins — Allowlist_IsEmpty asserts nothing is
exempt at all, and FormerlyAllowlistedActors_NowScanClean scans both actors with the
allowlist DISABLED, so the empty allowlist is evidence the fields are gone rather than
evidence the scan stopped looking.
Negative control: re-adding an ISiteCallAuditRepository field to SiteCallAuditActor fails
both FormerlyAllowlistedActors_NowScanClean and the real
Central_NoActorHoldsCaptiveScopedState scan.
Verification: solution builds clean. AuditLog.Tests 3/3 (383), SiteCallAudit.Tests 3/3
(66), Host.Tests 2/2 (505), IntegrationTests 2/2 (115), Communication.Tests 2/2 (717).
|
||
|
Joseph Doherty
|
b5f120c370 |
feat(observability): honest stream-subscriber name + real data-connection gauges (row 54)
scadabridge_site_connection_up never counted data connections. It counts open central->site gRPC subscriber streams (Debug View + the site-wide alarm feed), so it correctly reads 0 on a healthy site nobody is watching — and the name invited reading it as device health. That misreading has already cost a live debugging session: register row 48's 'connection_up is 0 on all six site nodes' was healthy steady state, not the fault it was diagnosed as. Two halves, per the row: RENAME. scadabridge.site.connection.up -> scadabridge.site.stream.subscribers.active (helpers SiteConnectionOpened/Closed -> SiteStreamSubscriberOpened/Closed), and the instrument gains the unit:"1" the other gauges already carried. Every reference updated: the gRPC server, both test suites, Component-Communication.md, components/Commons.md, docker/README.md and docker/seed-live-probe.sh. A test now pins the new name AND asserts the old one is gone, so it cannot creep back. NEW. scadabridge.site.data_connections.connected / .total — the first exported data-connection signal (the state existed on the health report and in the CLI, but never as a metric). connected < total is the alert condition; neither number alone separates 'a device is down' from 'nothing is configured', which is why the denominator ships with the numerator. Sourcing is deliberate on two counts. It is fed by the DCL at its four state transitions rather than derived from the health report, so the metric survives a dead report pipeline — the exact failure the 'metrics stale' flag exists to catch, and a derived gauge would go blind at the same moment. And it uses a static registry reached through ScadaBridgeTelemetry (the SiteConnectionOpened precedent) rather than a DI-injected holder: the DCL actors already take ISiteHealthCollector through their constructors, and threading a second dependency through both would ripple into every test that builds them. It also keeps SiteHealthCollector untouched, which matters right now — another agent is concurrently editing that file for the tag-resolution path. Removal is wired too: an undeployed connection leaves the registry, so it stops counting toward the total instead of latching connected < total on forever. Both gauges read 0 on the standby (connections open on the active node only) — healthy, documented, and called out in the docs so it does not become the next site_connection_up. No ObservedMeters allowlist change: all three instruments ride the existing ZB.MOM.WW.ScadaBridge meter. (An unlisted meter exports nothing, silently.) Tests: Commons 711 passed (9 new, delta-based against a baseline with unique per-test connection names, so they do not inherit the register row 32 QueueDepthGaugeTests static-state flake); Communication 717; DCL 319. |
||
|
Joseph Doherty
|
00a804e6d9 |
guard(di): captive-dependency tripwires — scope validation + an actor field scan
Register row 50. Row 45 (fixed
|
||
|
Joseph Doherty
|
9fb52153fd |
fix(test): order three more unsynchronized assertions behind the observables they follow
Deferred flake-pattern sweep of tests/ for the class fixed in |
||
|
Joseph Doherty
|
cca7f1786d | chore(deps): LocalDb 0.2.0 — dereg cleanup, late-opt-in baselining, byte-budget replication | ||
|
Joseph Doherty
|
a5882753dd | perf(comms+audit): close phase-2 residuals — direct ingest path, monotonic timeouts, synthetic probe, not-reporting set, cursor-exact audit pull | ||
|
Joseph Doherty
|
c7a89cbe88 | Merge branch 'worktree-agent-a95eaaa8a3181ad4c' into arch-review-remediation | ||
|
Joseph Doherty
|
2e4e41a8f7 |
fix(auditlog): site audit DB onto the data volume; required path + soft flush
Closes WP1.2 of the arch-review remediation plan (finding #2, High): SqliteAuditWriterOptions.DatabasePath defaulted to CWD-relative "auditlog.db", which on the docker rig resolves onto the container's ephemeral overlayfs (not the mounted /app/data volume), silently discarding the pending audit forward-state backlog on every recreate; nothing in docker/ or docker-env2/ overrode it; FlushIntervalMs was validated but never read by the writer loop (one commit per event even at trickle rate); and no PRAGMA synchronous was set (SQLite's FULL default fsyncs every commit). - DatabasePath now has no default (mirrors ZB.MOM.WW.LocalDb's LocalDbOptions.Path) and is required pre-host for Site nodes only, via a new StartupValidator raw-config check (top-level "AuditLog:SiteWriter:DatabasePath", NOT nested under ScadaBridge: AddAuditLog binds that section off the configuration root). SqliteAuditWriterOptionsValidator deliberately does NOT check DatabasePath itself, because AddAuditLog runs its ValidateOnStart on both Central and Site composition roots but only Site nodes ever resolve the writer — checking it there would fail Central's boot too. - All 8 site-node appsettings under docker/ and docker-env2/ now set AuditLog:SiteWriter:DatabasePath to /app/data/auditlog.db (mounted volume, survives container recreate, same convention as LocalDb:Path); the local-dev base appsettings.Site.json sets ./data/auditlog.db to match. - The writer loop now honors FlushIntervalMs: after draining the immediately available burst, it keeps the transaction open (bounded by FlushIntervalMs from the first event) waiting for more trickle-rate events before committing, instead of flushing (and fsyncing) per event. - PRAGMA synchronous = NORMAL on the write connection — audit is best-effort by design (CLAUDE.md: "Audit-write failure NEVER aborts the user-facing action"), so NORMAL's narrower power-loss window is an acceptable trade for far fewer fsyncs; WAL mode still guarantees no corruption. - Tests: StartupValidator site-required/blank/central-exempt cases; writer trickle-load single-transaction coalescing + beyond-interval separate-transaction regression (new FlushCountForTests seam); options-validator doc updates reflecting the moved responsibility. Full suite runs green: AuditLog.Tests 368/368, Host.Tests 480/480. One-time migration note: the existing container-local auditlog.db (wherever it landed under CWD) is abandoned by this change, not migrated — already-forwarded rows are safe centrally (AuditLog is the durable copy), and any still-Pending rows on the abandoned path are lost once. This is the exact bug being fixed, not a new loss: those rows were already living outside the mounted volume and would not have survived the next container recreate regardless. Cross-reference docs/known-issues/2026-07-20-cached-telemetry-drain-hot-loop.md, which this placement bug caused. |
||
|
Joseph Doherty
|
7ebdcd370a |
perf(host): install CDC capture only when replication is configured
SiteLocalDbSetup.OnReady registered all ten replicated tables unconditionally, so a deliberately unreplicated site node (site-b and site-c on the rig) carried the full 30-trigger CDC set forever. Every write to those tables paid two extra INSERTs plus a json_object serialization of the whole row, inside the caller's own transaction, and appended to an oplog nothing ever drains. Arch-review finding #5 (High), repo half; the library half — trigger cleanup API and O(1) backlog — is WP3.3. The ten RegisterReplicated calls are now behind a guard on whether the node has LocalDb:Replication:PeerAddress OR LocalDb:Replication:ApiKey. Either key counts, and the OR is load-bearing rather than defensive: replication is one bidirectional stream that exactly one side dials, so only the initiator sets PeerAddress. Verified against the rig — site-a node-a has PeerAddress + ApiKey, site-a node-b (passive) has ApiKey alone, site-b/site-c have no Replication section at all. Keying on PeerAddress alone would have stripped capture from every passive node and silently made each pair converge in one direction only. The load-bearing ordering documented in the file is preserved: DDL still precedes registration, and the legacy migrator still runs unconditionally after it — an unreplicated node must still absorb its pre-Phase-1 files, and it has no peer for those rows to be invisible to. Known residual, documented in-file and in the topology guide: a database file first created by an older build keeps its stale __localdb_* triggers. The guard decides whether triggers are installed, not whether existing ones are removed, and the library has no removal API until WP3.3. Moot on the docker rig, where a schema-change redeploy recreates the volumes. The inverse is also now documented: enabling replication on a site that has run without it does not baseline existing rows, since CDC never recorded them in __localdb_row_version and the snapshot resync streams from that ledger. Tests: new SiteLocalDbCdcRegistrationTests asserts trigger presence and absence via sqlite_master across all four config shapes (none, ApiKey only, PeerAddress + ApiKey, and the notification-table exclusion), plus DDL-still-runs and migrator-still-runs on the unreplicated branch. SiteLocalDbWiringTests and the integration site-pair harness now configure an ApiKey — mirroring the rig's passive node — so their registration and convergence assertions still describe a replicating node. 483/483 Host.Tests pass; the 20 offline LocalDb convergence tests still pass. |
||
|
Joseph Doherty
|
006202f3c7 |
fix(audit): fail closed when a configured redactor is unavailable (#35)
Component-AuditLog.md has always required "we over-redact, never under-redact, on configuration faults", but the body / SQL-parameter redactors violated it. AuditRegexCache rejects a pattern that is malformed OR whose compile exceeds a 100 ms budget, caching the rejection for the process lifetime. ScadaBridgeAuditRedactor then simply dropped the rejected pattern from its redactor set and emitted the payload anyway — publishing precisely the values the operator configured it to suppress, onto a row that looks entirely normal downstream. Recovery required a process restart and the only signal was one Warning line. The SQL path was worse: TryGetSqlParamRedactor returned a bare false for both "no redactor configured for this connection" and "the configured one will not compile", and CLAUDE.md records SQL parameter capture as on by default. Two changes: 1. Fail closed. A pattern that is CONFIGURED but unavailable now over-redacts the whole payload and increments AuditRedactionFailure, reusing the existing safety-net path. "Not configured at all" stays permissive — conflating those two states is the actual defect, so both are pinned by tests. 2. Precompile off the hot path. The audit-log roadmap specifies patterns are "precompiled at startup; rejected if compile takes >100ms"; the implementation had drifted to compiling lazily on first event, which put a wall-clock budget on a hot path under production load. RegexOptions.Compiled emits IL during construction, so a busy node could blow the budget on a perfectly valid pattern. Warm-up now runs at construction and on every options reload. The residual window between a reload and its warm-up is safe because that path now fails closed. Warm-up deliberately does not fail the boot — an unusable pattern degrades the node to over-redaction (safe, loud) rather than refusing to start. Reading CurrentValue happens inside the warm-up try so an options provider that throws still surfaces via Apply's over-redact path, not the constructor (OuterCatch_OptionsThrows_NeverLeaks_AllSensitiveFieldsOverRedacted). Also de-flakes GrpcCentralTransportTests.DeadlineExceeded_IsNotRetriedOnThePeer, which is how this was found. It black-holed node A behind a 300 ms deadline, but on a saturated machine the call could fail to even START — a genuinely-unsent failure that IsConnectFailure correctly fails over on, so node B's ack arrived instead of the expected Status.Failure. The test read as a flake while actually reporting that its own premise had not held. Split in two: the hard rule now injects an explicit DeadlineExceeded via a trailers-only response (deterministic, load-independent), and a new BlackHoledNode_DoesNotHang covers the deadline-is-actually-applied half with both nodes black-holed so no ack can arrive down any path. Verified: both fixes were confirmed to fail before they pass — reverting the fail-closed guard fails exactly the 5 fail-closed tests while the 4 controls still pass, and adding DeadlineExceeded to IsConnectFailure fails the rewritten transport test. AuditLog 367/367, Host.Tests GrpcCentralTransport 8/8, solution build clean. The previously-intermittent Filter_PicksUp_NewBodyRedactor_OnConfigReload is green in a full sweep for the first time. Not addressed here, and noted on #35: the 100 ms wall-clock budget remains a weak proxy for catastrophic backtracking (RegexOptions.Compiled defers JIT to first match, so construction time measures the wrong thing), and a rejection is still cached permanently. Both are now safe rather than dangerous, so they are hardening rather than a leak. |
||
|
Joseph Doherty
|
fdc6b0c2bb |
chore(secrets): adopt ZB.MOM.WW.Secrets 0.6.2 and close the pre-host guard gap
0.6.x refuses a secret store whose path is relative or inside the content root,
because a store in the deployment directory is destroyed by an ordinary upgrade —
the failure that wiped the MxGateway API-key store on 2026-08-09 and read as an
auth outage rather than a deployment error.
The pin alone would not have protected this repo. Program.cs expands ${secret:}
before the host exists, composing secrets into a throwaway ServiceCollection with
no IHostEnvironment, so the guard would not run at the moment the migrator creates
the store. That composition now lives in SecretsRegistration with an explicit
content root — resolved to match what the host resolves later, including the
Windows-Service case where the pre-host CWD is still system32 — and is covered by
PreHostSecretsContentRootTests, verified by simulating the regression and
confirming it fails on the leftover file rather than on the exception.
The docker rig needed a fix too: /app/data is absolute but inside the container's
content root, so all 8 nodes would have failed to boot. Each node's data directory
is now mounted a second time at /data; same host directory, so existing stores
carry over untouched.
Verified: build clean, 29 test assemblies green (Playwright's 159 failures are the
pre-existing SEC-36 login baseline). Not yet deployed — the rig runs the old
config until someone redeploys.
|
||
|
Joseph Doherty
|
e9c412e528 |
fix(host): unhandled boot exception now kills the process instead of wedging the container (#34)
Root cause: dotnet runs as container PID 1 and Linux ignores default-action signals sent to PID 1, so the runtime's unhandled-exception path (banner, then abort() -> SIGABRT) could never terminate the process — it printed the trace and spun the main thread at 100% CPU with the container `running`, so `restart: unless-stopped` never fired. Reproduced deterministically: same StartupValidator throw exits 134 under an init process and wedges without one. Two layers, each covering the other's gap: - Program.cs registers an AppDomain.UnhandledException handler before the first statement that can throw: prints the trace, best-effort flushes Serilog, Environment.Exit(134) — exit() is a syscall PID 1 CAN perform, 134 preserves the 128+SIGABRT crash code, and it covers every thread, not just the boot window. It cannot fire under WebApplicationFactory (the test host catches entry-point exceptions), so the designed boot-refusal exceptions still propagate to tests unchanged. - docker-compose: init: true on all 8 nodes for the crash paths that bypass the managed event (Environment.FailFast, runtime-internal aborts). The CoordinatedShutdown no-Environment.Exit guard gains a precise carve-out (exactly one call, only inside the handler); Environment.Exit still fires the CLR shutdown hook Akka binds via run-by-clr-shutdown-hook = on, so the crash path skips nothing abort() kept. New pin test keeps the handler ahead of the configuration build. Live-verified on the rig image: crash now yields Exited (134) + RestartCount climbing under `unless-stopped`, trace intact, with and without init; full 8-node rig redeployed healthy with docker-init as PID 1. Closes #34. Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1 |
||
|
Joseph Doherty
|
68f812eaa4 |
feat(secrets): bump ZB.MOM.WW.Secrets family to 0.5.0 and wire hub fallback endpoints
Pin all five ZB.MOM.WW.Secrets* packages 0.4.1 -> 0.5.0, which brings SecretsGrpcHubClientOptions.FallbackEndpoints and the package's internal FailoverSecretsHubReader. A site whose GrpcHub section lists fallback endpoints now fails a sweep over to the next central instead of stalling on a downed primary - safe ONLY because both central nodes serve one shared SQL secret store (scadaproj#4), so either hub answers with the same manifest; the appsettings comments say so and warn against listing endpoints backed by independent stores. appsettings.json gains "FallbackEndpoints": [] with a _fallbackEndpoints comment, and the _endpoint note's single-endpoint-stall caveat is scoped to the empty-list case it now only applies to. Wiring pins (red first on 0.4.1): site + Grpc + one fallback resolves ISecretsHubReader to FailoverSecretsHubReader with the "zb-secrets-grpc-hub:fallback:0" keyed channel present; zero fallbacks keeps the plain GrpcSecretsHubClient and no fallback channel - the pre-0.5.0 container shape byte-identical. Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1 |
||
|
Joseph Doherty
|
aebdd56b52 |
docs(secrets): review follow-ups — pre-Serilog window note, ThrowIfNull hygiene, honest site-purity scan bound
The expander comment now records that a SQL outage at central boot exits pre-Serilog with a bare stderr trace (honest, restart-retryable, unenriched) and why the role-reading predicate deliberately stays in Program.cs rather than SecretsRegistration (that class refuses config-read roles by design). EnsureCentralSharedStoreConnectionString gains the file's standard ThrowIfNull. The site-purity scan's doc no longer overclaims: two factory-lambda descriptors evade it individually; the pin holds because three concrete-type registrations from the same call cannot. Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1 |
||
|
Joseph Doherty
|
43e87a7492 |
feat(secrets): central's Grpc-mode store is the SHARED SQL-Server store (scadaproj#4)
In Secrets:Replication:Mode=Grpc a central node's ISecretStore is now the
shared SQL-Server store (AddZbSecretsSqlServerStore) instead of a per-node
local SQLite store. Both central hubs read and write ONE copy of every row,
so they serve identical manifests by construction — the 2026-08-07 live gate
observed central-b answering an authenticated GetManifest with an EMPTY
manifest while central-a held every secret, which would turn site-side hub
failover into a silent convergence stop.
- SecretsRegistration: two fail-closed pre-checks before any registration on
the central+Grpc path — a blank Secrets:SqlServer:ConnectionString throws
naming the key (an independent store per central node is the recorded
defect), and a value containing ${secret: throws naming the bootstrap
circularity (the expander needs this store to resolve references). Site
registrations are byte-identical to before; SqlServer mode and
replication-off are untouched.
- Program.cs Layer-A expander follows the store swap: central+Grpc with a
non-blank connection string migrates and resolves pre-host ${secret:}
references through the shared SQL store, so expanded values can never
diverge from what the running node serves. Every other case keeps the
SQLite path unchanged; blank-connstr central deliberately falls through so
the clear AddScadaBridgeSecrets message is the one that fails the boot.
- appsettings.json: Secrets:SqlServer _comment now documents the Grpc-mode
central requirement (literal/env value only, sites leave it empty).
- SecretsReplicationWiringTests: +5 pins (shared store resolves, blank and
${secret:} connstrings fail naming the key, sites-have-no-SqlServer-types
descriptor sweep), central fixtures carry the now-required connstr.
Full suite green (7,474 passed across 30 projects, 0 warnings); the two
failures are pre-existing and unrelated: the Playwright live-rig suite fails
identically on unmodified main (cluster not running), and
GrpcCentralTransportTests.DeadlineExceeded_IsNotRetriedOnThePeer is a timing
flake that passes 3/3 in isolation and 470/470 on the first run of this code.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
|
||
|
Joseph Doherty
|
c8e90daafb |
test(secrets): pin both hub halves, the mode key, and the mapping gate
Extends the existing build-and-resolve wiring suite rather than asserting over ServiceDescriptors, for the reason that file already documents: a registration can look correct as a descriptor list and still fail on first resolve. Role separation is pinned in both directions — central registers the hub's fail-closed interceptor and NOT the sweep, a site registers the sweep and NOT the hub — because only one of those is a security failure and testing the happy half would not catch it. The sweep is asserted by resolving IHostedService, which builds its whole graph (reader, keyed channel, local store) and is where a forgotten AddZbSecrets would surface; GrpcSecretSyncService is internal to the package, so it is matched by assembly + type name the way the SqlServer replicator's services already are. SqlServer mode gets regression pins with the mode key both unset and named explicitly, plus one asserting it stays role-agnostic — both nodes sync bidirectionally against the same database, and the role parameter added for the hub must not have quietly changed that. The mapping tests assert over the app's real endpoint data sources, on the WIRE route (/zb.mom.ww.secrets.hub.v1.SecretsHub/...) rather than the C# type, since the route is what a follower addresses. One of them pins exactly the two READ methods: pull-only is a property of the contract, and this is where a future package version growing a write RPC would become visible instead of silently opening a path for a site to overwrite central. Numeric mode values get their own test. Enum.TryParse accepts any integer, including ones outside the enum, so "7" would otherwise select a mode that does not exist and fall through to the SqlServer branch. Verified red-first by mutation on the finished implementation: swapping the two role branches reds 8 (both role pins, both fail-closed pins, both mapping pins); deleting the UsesGrpcHub check in the map extension reds exactly the two "maps no hub endpoint" cases — the unauthenticated-hub scenario; dropping Enum.IsDefined and letting UsesGrpcHub ignore Enabled reds the out-of-range value and the flag-off-with-full-hub-config cases. 31/31 green restored. Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1 |
||
|
Joseph Doherty
|
7caa8bfd99 |
fix(localdb): disable SQLite pooling on legacy reads - a Windows site node could not boot
SiteLocalDbLegacyMigrator opens each pre-Phase-2 file read-only, copies its rows into
the consolidated site database, then renames the file so a later boot skips it.
Microsoft.Data.Sqlite POOLS connections, so disposing one returns it to the pool and
leaves the underlying sqlite3 handle - and the OS file handle - open. On Windows the
subsequent File.Move throws IOException ("being used by another process"), the
migration faults out of AddZbLocalDb's factory, and the site node does not start at
all. There is no partial-migration path: it is a hard boot failure, once, on the first
upgrade past LocalDb Phase 2.
Why it shipped: POSIX rename ignores open handles, so this cannot reproduce on Linux or
macOS. Every existing rename assertion in SiteLocalDbLegacyMigratorTests - including
LegacyTrackingRows_AreCopiedAndTheFileIsRenamed - passes with the bug fully present, and
the docker rig migrates cleanly. It was found by pre-flighting the wonder-app-vd03
upgrade against COPIES of that box's real site databases; an earlier pre-flight pass
with empty data directories had nothing to drain and passed clean.
Fix: both legacy read-only connection strings now go through one
LegacyReadOnlyConnectionString helper carrying Pooling=False.
The accompanying test is deliberately white-box. A behavioural assertion cannot
discriminate here on the platform this suite runs on, so it pins the connection string
instead; it is red without the fix.
Verified: Host.Tests 440/440, 0 warnings. The build deployed to wonder-app-vd03 carries
this change (as the then-uncommitted fix) and has been running since 2026-08-05.
|
||
|
Joseph Doherty
|
5d4853a1f0 |
refactor(health): adopt the shared active-node check (Health 0.3.0)
OldestNodeActiveHealthCheck existed because the shared ActiveNodeHealthCheck
selected by cluster leadership (review 01 [High]): leadership is address-ordered
and diverges from singleton placement after a restart, and during a partition
both sides compute themselves leader so Traefik served both. Health 0.3.0 makes
the shared check age-based — it is now this host's own rule, promoted — so the
private copy is deleted and central registers the shared type.
ActiveNodeEvaluator, the "THE single definition of active node" this repo already
maintained, now delegates to the shared ClusterActiveNode rather than
re-implementing it. That keeps the delivery gate, the heartbeat IsActive stamp,
the inbound-API gate and the /health/active tier on one rule, and it means the
rule is shared with OtOpcUa, which had independently written a third copy.
Communication takes a ZB.MOM.WW.Health.Akka reference for it; the layering
trade-off is recorded at the PackageReference.
SitePairActiveNodeHealthCheck is deliberately KEPT. It is not a duplicate of the
rule — it is a thin adapter over the site's own IClusterNodeProvider, which
scopes to site-{SiteId} and is itself now backed by ClusterActiveNode. Registering
the shared check directly would have to re-derive the site role and would lose the
property that the tier and singleton placement come from the same provider.
Central stays unscoped: every central member competes for one active slot.
No behaviour change on any node — same rule, one implementation instead of two.
Verified: Host.Tests 439/439, Communication ActiveNode 2/2.
|
||
|
Joseph Doherty
|
47850c0f53 |
feat(health): serve MapZbHealth on site nodes
Site nodes served no health surface at all — gRPC on the HTTP/2-only listener and
/metrics on the HTTP/1.1 one — so nothing outside the cluster could ask a site
node whether it was ready or which half of the pair was active. The family
overview dashboard probes every instance the same way, and this is the one gap.
Three checks, registered in SiteServiceRegistration.Configure (not Program.cs, so
the composition-root tests that build this graph actually cover them) and mapped
by app.MapZbHealth() on the site's HTTP/1.1 listener (default :8084) alongside
/metrics:
akka-cluster [Ready] the shared AkkaClusterHealthCheck. Also carries the
cluster-view data (leader/memberCount/...) the dashboard
reads, free with ZB.MOM.WW.Health 0.2.0.
localdb [Ready] NEW SiteLocalDbHealthCheck — SELECT 1 through the
registered ILocalDb. Site has no EF context; central's
DatabaseHealthCheck<ScadaBridgeDbContext> is central-only.
Replication state rides along as data ENRICHMENT only:
replication is default-OFF, so failing on it would mark
every correctly-configured node unready.
active-node [Active] NEW SitePairActiveNodeHealthCheck, delegating to
IClusterNodeProvider.SelfIsPrimary. Deliberately NOT
central's OldestNodeActiveHealthCheck: that one calls
SelfIsOldest(cluster) with no role argument and would
compute "oldest" across the wrong member set on a mesh
carrying more than one site.
Anonymous, as central's are: the site pipeline runs no authentication middleware
and has no FallbackPolicy, so nothing extra was needed.
Also bumps ZB.MOM.WW.Health* 0.1.0 -> 0.2.0 (central gains data.leader for free).
Tests: SiteHealthCheckTests builds the REAL site container and activates every
registration through its factory (exact-set names, one tier tag each, resolved
types, central-only checks absent behind a positive control) plus behaviour for
both new checks. SiteHealthEndpointTests boots the real site Program over
WebApplicationFactory and proves the endpoints are MAPPED — without it, deleting
MapZbHealth would leave every registration test green while site nodes 404'd.
Prereq for scadaproj docs/plans/2026-07-22-overview-dashboard-impl-plan.md Phase 1.
|
||
|
Joseph Doherty
|
7fd5cb2b56 |
feat(comm): Phase 4 — delete Akka ClusterClient site↔central transport, gRPC-only
ClusterClient→gRPC migration Phase 4 (docs/plans/2026-07-22-clusterclient-to-grpc-plan.md). Phases 2/3 proved both directions on gRPC; this removes the Akka transport underneath. Deleted: - AkkaCentralTransport, AkkaSiteTransport (+ their dedicated tests) - ISiteClientFactory + DefaultSiteClientFactory; CentralCommunicationActor legacy ctor + SelectTransport (Host now builds GrpcSiteTransport and injects it) - ClusterClient creation + both ClusterClientReceptionist.RegisterService calls in AkkaHostedService; the RegisterCentralClient message + receive block - CommunicationOptions.CentralContactPoints; the CentralTransport/SiteTransport coexistence flags; the CentralTransportMode/SiteTransportKind enums gRPC is now the only site↔central transport (site→central CentralControlService via GrpcCentralTransport; central→site SiteCommandService via GrpcSiteTransport), both built unconditionally by the Host. NoOpCentralTransport is the fail-loud null-default so TestKit command-dispatch suites still construct the site actor without a wired transport; production always injects GrpcCentralTransport. Config: CentralGrpcEndpoints is now unconditional — CommunicationOptionsValidator rejects blank entries (role-agnostic), and StartupValidator requires a Site node to list >=1 endpoint (fail-fast, mirrors GrpcPsk). Rig configs moved CentralContactPoints -> CentralGrpcEndpoints (docker x6, docker-env2 x2, Host default, deploy/wonder-app-vd03). Kept Akka.Cluster.Tools (ClusterSingleton still used). Tests: build 0/0; Communication.Tests 640, Host.Tests 421 green. Removed the ClusterClient.Send per-site-routing tests (covered by the transport suites), swapped the ISiteClientFactory-based ctors to a substitute ISiteCommandTransport, converted the audit-push integration relay to an in-process bridge transport. Docs: Component-Communication/Host/StoreAndForward, components/Communication, topology-guide, grpc_streams (SUPERSEDED note), the frame-size known-issue (retired amendment), and CLAUDE.md transport decisions. Not included: the dead IntegrationCallRequest path (#32) is a separate user-owned behavioral decision — SiteEnvelope routing is transport-agnostic so it still compiles. |
||
|
Joseph Doherty
|
86ad4d5c8e |
feat(comm): T1B.3 — central-side gRPC site-command transport seam (default Akka)
Extract the central→site send path in CentralCommunicationActor behind a new ISiteCommandTransport, selected by ScadaBridge:Communication:SiteTransport (Akka | Grpc, default Akka — rollback = flip the flag). CommunicationService's 27 commands, SiteCallAuditActor's 2 parked relays and DebugStreamBridgeActor's subscribe/unsubscribe are untouched; the seam sits below SiteEnvelope. - AkkaSiteTransport: today's per-site ClusterClient path extracted verbatim (the _siteClients lookup + ClusterClient.Send with the reply-to sender preserved, and the "no client ⇒ warn + drop, caller's Ask times out" path). - GrpcSiteTransport: dials the site SiteCommandService (T1B.1 proto client) via SiteCommandDtoMapper, PSK + x-scadabridge-site on the channel through ControlPlaneCredentials, per-command deadlines set EQUAL to today's CommunicationService Ask timeouts (per-command, not per-group: DeploymentState query and TriggerSiteFailover use QueryTimeout; the two parked relays map to QueryTimeout so SiteCallAudit's inner RelayTimeout 10s < 30s ordering holds; WaitForAttribute keeps its dynamic Timeout + IntegrationTimeout). - SitePairChannelProvider: per-site A/B channel pair with sticky failover (flip only on Unavailable — NEVER on DeadlineExceeded, a write/deploy/failover may have run), background failback probe to the preferred node with 1s→60s doubling backoff, PSK invalidation on site removal. Fed by the SAME DB refresh loop (extended to carry GrpcNodeA/GrpcNodeBAddress) — no second poll. Tests: actor-with-substitute-transport (routing, Ask-reply plumbing, per-site lifecycle across refreshes), ResolveDeadline pinned to each command's current Ask timeout, and GrpcSiteTransport/SitePairChannelProvider over dual in-process TestServers (PSK+header+deadline attached, Unavailable failover + stickiness, failback to preferred, no-retry-on-DeadlineExceeded). Proto csproj untouched (no active <Protobuf> item). Full solution builds 0 warnings; Communication.Tests 607 green with Akka default. |
||
|
Joseph Doherty
|
518c699b90 |
feat(comm): extract SiteCommandDispatcher; site serves commands over gRPC too (T1B.2)
Refactor SiteCommunicationActor's central→site routing table into one SiteCommandDispatcher — the single routing truth for the 28 migrated commands (IntegrationCallRequest, the dead 29th, stays on the actor and out of the dispatcher). The Akka actor and the new SiteCommandGrpcService both route through one dispatcher instance so the two transports can never drift on where a command goes. Server-side only: nothing central flips to gRPC yet (that is T1B.3); ClusterClient remains the live path. Decisions worth recording: - Targets preserved byte-for-byte. Lifecycle/OPC UA/query/route → the Deployment Manager singleton proxy; DeployArtifacts/EventLog/parked → their null-guarded handlers with the exact same "handler not available" replies; the parked handler stays NODE-LOCAL (per-node replicated-store owner), never the singleton proxy — pinned by a dispatcher test that asserts the target is the parked probe and NOT the dm proxy. - Sender preservation intact. The actor's command handlers became thin DispatchCommand delegations that still Forward (central Ask → reply routes straight back); the existing SiteCommunicationActorTests pass unchanged, which is the regression guard for that plumbing. UnsubscribeDebugView keeps its fire-and-forget shape: the actor Forwards, the gRPC service Tells + returns the synthetic UnsubscribeDebugViewAck so a unary RPC still answers. - Ack-before-Leave on failover. The dispatcher's PrepareFailover resolves the standby with a DRY-RUN (no leave) to build the ack, and hands back a deferred CommitLeave; the gRPC service returns the ack, then schedules the real Cluster.Leave — so a caller reaching the very node about to leave still gets its ack instead of a broken stream. The actor path keeps today's coupled resolve-and-leave (over ClusterClient the ack Tell only enqueues, so order is immaterial). Proven at both levels: a dispatcher test asserts the ack is built before CommitLeave runs, and a TestServer test asserts the recorded seam order is resolve-then-leave. - ControlPlaneAuthInterceptor gates SiteCommandService by EXTENDING DefaultGatedPrefixes (descriptor-derived), not by adding a constructor — the one-public-ctor invariant and its test stay green. Tests: SiteCommandDispatcherTests (28-command routing incl. parked node-locality and both failover paths) and SiteCommandGrpcService TestServer tests (auth, readiness→Unavailable, one command per oneof group, failover ordering). Full solution build 0/0; Communication.Tests 574 and Host.Tests 377 green. No active <Protobuf> item. |
||
|
Joseph Doherty
|
0e162cb250 |
fix(grpc): central Kestrel gRPC listener was dropping the :5000 HTTP surface (T1A.2 regression)
Caught on the Phase 1A rig proof: central-a logged only "Now listening on: http://[::]:8083" and nothing else. Central UI, the Management + Inbound API, and the /health/* endpoints Traefik + IActiveNodeGate depend on were all gone, with no startup error — the node booted, joined the cluster and served gRPC fine. Cause: calling options.ListenAnyIP in ConfigureKestrel puts Kestrel into explicit-endpoints mode, which SUPPRESSES the URLs from ASPNETCORE_URLS/--urls entirely — it is not additive, contrary to the comment T1A.2 shipped. Central's whole HTTP/1 surface lives on that URL (http://+:5000 on the rig, a different port in prod), so binding only the gRPC port silently deleted it. The site branch has the same shape but no ASPNETCORE_URLS surface to lose — it binds every port it needs explicitly. Fix: parse the port(s) from the configured URLs and re-declare them (Http1AndHttp2) alongside the gRPC port (Http2) in the one ConfigureKestrel call. New Program.ParseHttpBindPorts + CentralHttpBindPortsTests (14 cases: wildcard/ipv6/ hostname hosts, multi-URL, de-dupe, scheme-default, null/blank, unparseable-skipped). Why no test caught it originally: unit/E2E tests use TestServer, which never binds real Kestrel. Only a live node exposes the missing listener — which is exactly what the rig proof is for. |
||
|
Joseph Doherty
|
33b15f10a4 |
feat(grpc): site-side ICentralTransport seam + gRPC transport (T1A.3)
Introduce ICentralTransport as the site->central choke point inside SiteCommunicationActor. The seven site->central sends (notification submit/ status, audit + cached-telemetry ingest, reconcile, health, heartbeat) now delegate to an injected transport instead of owning ClusterClient.Send inline. - AkkaCentralTransport: verbatim extraction of today's ClusterClient.Send path, including the exact sender-forwarding that routes central's reply straight back to the waiting Ask. Default when no transport is injected -> behaviour unchanged, existing SiteCommunicationActorTests pass as-is. - GrpcCentralTransport + CentralChannelProvider: dial CentralControlService with sticky failover + background failback (1s-doubling-cap-60s), PSK + x-scadabridge-site via ControlPlaneCredentials, per-call deadlines mirroring today's Ask timeouts. Cross-node retry ONLY on provably-unsent connect failures; never on DeadlineExceeded. Heartbeat stays fire-and-forget. - StaticSitePskProvider: site's single own-key provider (fail-closed). - CommunicationOptions: CentralTransport flag (default Akka) + CentralGrpcEndpoints (validator: required when transport=Grpc). Host selects the impl; the ClusterClient is created only on the Akka path. Tests: actor-with-fake-transport (7 delegations + fault routing + heartbeat no-fault), AkkaCentralTransport sender-forwarding, GrpcCentralTransport over in-process TestServer (failover flip, sticky, failback, PSK+header, deadline, no-retry-on-deadline), validator. Communication.Tests 371 green, Host.Tests 391 green; the three above-seam suites pass unmodified. |
||
|
Joseph Doherty
|
780bb9c369 |
feat(grpc): host CentralControlService on the central node (T1A.2)
Central now ALSO listens for the seven site→central control messages over gRPC, alongside the existing ClusterClient path. Nothing flips to gRPC yet — sites keep CentralTransport=Akka (T1A.3's job); central simply starts also accepting. - CentralControlGrpcService (Communication.Grpc): decodes each RPC onto the SAME in-process message the ClusterClient path carries, Asks the existing CentralCommunicationActor (zero handler-logic changes), encodes the reply via the T1A.1 mapper. Readiness-gated like SiteStreamGrpcServer.SetReady — Unavailable until AkkaHostedService hands the actor over. Heartbeat stays fire-and-forget (Tell, always-OK, never gated on readiness). Ingest reuses the shared SiteStreamGrpcServer.AuditIngestAskTimeout constant. Fault→status mapping is retry-aware: Unavailable (never dispatched, safe to cross-node retry) vs DeadlineExceeded/Internal (it ran, do not re-send elsewhere). - CentralControlAuthInterceptor (Host): a SEPARATE interceptor class, not a variant constructor on ControlPlaneAuthInterceptor. Central's model is per-site (verify the Bearer token against the key for the site in the required x-scadabridge-site header, via ISitePskProvider) where a site verifies its one own-key — a genuinely different model. Fail-closed on every branch: missing or blank header, unresolvable key, and mismatched token all → PermissionDenied, never pass-through. One public constructor only (the explicit-prefix ctor is internal), pinned by a reflection test — a second public ctor makes Grpc.AspNetCore's GetFactory() throw per-call and silently disables the gate. - Explicit Kestrel h2c listener on new option ScadaBridge:Node:CentralGrpcPort (default 8083, symmetric with sites), mirroring the Site branch. Additive to central's :5000 HTTP/1 surface, which is untouched — gRPC does NOT go through Traefik (HTTP/1 only). Registered by type on AddGrpc; service mapped with MapGrpcService. Port range-validated by NodeOptionsValidator. - Rig: publish the central gRPC port 9013:8083 / 9014:8083 on both central nodes so a later task can exercise it. Tests: CentralControlEndToEndTests (Host.Tests, TestServer + real interceptor + real service over a stub actor) proves auth positives/negatives are distinguishable and covers unary + the ingest bridge shapes; the interceptor is registered BY TYPE, never in DI. CentralControlAuthInterceptorTests pins the per-site gate + one-public-ctor invariant. CentralControlGrpcServiceTests (Communication.Tests, TestKit) covers the readiness gate, fire-and-forget heartbeat, and the DeadlineExceeded-vs-Unavailable status mapping. No active <Protobuf> item. Communication.Tests (356) + Host.Tests (384) green. |
||
|
Joseph Doherty
|
228ff8b428 |
fix(grpc): one public constructor on ControlPlaneAuthInterceptor — two made the gate inert
Caught by the Phase 0 live gate, not by the suite. Grpc.AspNetCore registers the interceptor BY TYPE, and InterceptorRegistration.GetFactory() throws "Multiple constructors accepting all given argument types have been found" when more than one public constructor is applicable. The interceptor had two: the DI one and a prefix-set overload added for later phases. The failure mode is nasty. The throw happens inside the interceptor pipeline on every call, so nothing fails at startup — the site node boots, joins, reports healthy. Every gated call then dies with Unknown / "Exception was thrown by handler", which reads as a handler bug rather than an auth bug. And it fails OPEN in the sense that matters least and closed in the sense that matters most: no call is ever authorized, but no call is ever correctly REFUSED either, so the rig showed identical errors for a correct key, a wrong key and no key at all. Live evidence, site-a: three PullAuditEvents calls, three identical InvalidOperationExceptions in the node log. Fix: the prefix-set constructor is internal (Host.Tests already has InternalsVisibleTo). Later phases extend DefaultGatedPrefixes rather than adding a second public registration shape. Why the tests missed it, and what changed: ControlPlaneAuthEndToEndTests registered the interceptor with AddSingleton alongside AddGrpc, so DI handed back the instance and Grpc.AspNetCore's activation path — the thing that throws — never ran. The harness now registers exactly as Program.cs does, by type and not in DI. Plus a direct reflection assertion that the type has exactly one public constructor, since that is the real invariant and it is cheap to pin. |
||
|
Joseph Doherty
|
2ee84af1c0 |
feat(grpc): PSK-authenticate the site gRPC control plane; drop the vestigial management receptionist registration
Phase 0 of the ClusterClient→gRPC migration
(docs/plans/2026-07-22-clusterclient-to-grpc-plan.md). Standalone hardening: it
closes a gap that exists today and is a precondition for moving command/control
onto gRPC in later phases.
T0.1 — delete the ManagementActor ClusterClientReceptionist registration.
It was built for an out-of-cluster CLI that was never written: the shipped CLI
speaks HTTP Basic to /management, which asks the actor in-process through
ManagementActorHolder. Nothing in the repo ever sent to /user/management. The
actor still runs there; only the cross-boundary advertisement is gone. Six
documents claimed the CLI used ClusterClient — including the CLI's own README
"Architecture Notes" — and are corrected here rather than left to rot.
T0.2 — record, do not port, the dead integration-routing path.
IntegrationCallRequest is unwired at BOTH ends: RouteIntegrationCallAsync has
zero callers anywhere, and RegisterLocalHandler(Integration, …) appears only in
a test, so production always answers "Integration handler not available". It is
excluded from the gRPC contract (28 of 29 commands migrate) rather than
enshrined on an additive-only wire format, and deleting it during a
transport migration would mix a behavioural change into a change whose whole
value is that behaviour is identical. See
docs/known-issues/2026-07-22-integration-call-routing-is-dead-code.md.
T0.3 — preshared-key authentication on SiteStreamService.
The service shipped with no auth at all: plaintext h2c, no interceptor, so
anything that could reach a site node's :8083 could open a live data stream or
read audit rows back via PullAuditEvents/PullSiteCalls. ControlPlaneAuthInterceptor
now gates /sitestream.SiteStreamService/ — modeled on LocalDbSyncAuthInterceptor
(constant-time compare, fail-closed, PermissionDenied) but gating a SET of
service prefixes so phases 1A/1B add services rather than interceptors. LocalDb
sync keeps its own separate key: it authenticates the pair partner, not central,
and collapsing the two would make a site's central-facing key also admit writes
into its database.
Keys are per site (SB-GRPC-PSK-<siteId>), never fleet-wide, so a compromised
site yields only its own. Central attaches them through ControlPlaneCredentials,
which binds CallCredentials to the channel — covering unary and streaming
uniformly, and letting the key resolve asynchronously, which a client
interceptor could not do without blocking. All three central→site channel
creation sites go through it (SiteStreamGrpcClient and both audit pull invokers);
the pull invokers' channel caches are re-keyed by (site, endpoint) because
credentials are per-site and bound to the channel.
Two decisions beyond the plan:
* StartupValidator now requires GrpcPsk on Site nodes. The plan specified only
the runtime gate, but fail-closed with no boot check produces a node that
joins, answers heartbeats and reports healthy while refusing every stream,
audit pull and telemetry ingest — silent and total. Same reasoning as the
existing inbound API-key pepper rule.
* Added Communication:SitePsks as a central-side key map. The plan assumed
central would read the store, seeded via a dev KEK; the docker rig
deliberately boots with no master key, so store-only resolution would leave
it unable to dial its own sites. The store stays primary — it is the only
source that can serve a site added at runtime — with the map covering
key-less hosts and one-off pins. Neither source falling back to
"unauthenticated" is the invariant.
T0.4 — dev keys on both rigs and tests.
34 tests. The seven that matter most exercise a real in-process gRPC stack over
TestServer: the unit tests on either side of the wire would both stay green if
the halves disagreed, and gRPC refuses call credentials on a plaintext channel
by default — the UnsafeUseInsecureChannelCallCredentials opt-in is only provable
by making a real call. They confirm correct key passes on unary AND streaming,
wrong key and no-credentials both get PermissionDenied, and an unresolvable key
fails the call with nothing reaching the service.
OPERATIONAL: a site node upgraded to this build without a key will not boot.
That includes the gitignored deploy/wonder-app-vd03/ overlay.
|