Task 7 of the LocalDb Phase 1 adoption plan.
AddZbLocalDbReplication registers the engine; MapZbLocalDbSync exposes the
passive endpoint the peer node dials. Both are unconditional but INERT by
default: with no LocalDb:Replication:PeerAddress the initiating
SyncBackgroundService starts and immediately idles, and nothing dials the
endpoint. A site pair replicates only once an operator sets a peer on BOTH
nodes. The endpoint shares the Kestrel h2c listener the site gRPC server already
uses, so there are no listener or port changes.
Deviation from the plan: it put AddZbLocalDbReplication in Program.cs. Doing so
would have left it untested - the composition-root tests build
SiteServiceRegistration's graph, not Program.cs - so a registration that silently
went missing would still show green. It now sits next to AddZbLocalDb in
SiteServiceRegistration, where the DI-graph tests actually cover it.
MapZbLocalDbSync stays in Program.cs because it needs the WebApplication.
Two pins added to SiteLocalDbWiringTests, which configures no peer:
- the engine resolves and is idle (not connected, no peer, never synced) -
default-off means inert, not unregistered;
- OplogBacklog is 0, not null. It is deliberately nullable so a failed poll
reads as "unknown" instead of a healthy zero; a real 0 proves the provider is
wired to the oplog, which Task 9's health signal depends on.
Inbound auth on the sync endpoint is Task 8. Until that lands the endpoint is
unauthenticated - which is precisely why replication ships default-OFF and no
config in this repo sets a peer.
Verified: build 0 warnings; Host 296/296, SiteEventLogging 70/70,
SiteRuntime 530/530, Commons 684/684, Communication 312/312.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Task 4 of the LocalDb Phase 1 adoption plan.
OperationTrackingStore took IOptions<OperationTrackingOptions> and opened its own
SqliteConnection from ConnectionString. It now takes ILocalDb and gets every
connection - writer and the two ad-hoc reader paths - from CreateConnection().
This is not cosmetic. OperationTracking is a RegisterReplicated table, so its
capture triggers call zb_hlc_next(). That UDF is registered per connection by
ILocalDb and by nothing else, so a raw SqliteConnection would fail closed on
every write. Connections from CreateConnection() also arrive already open -
calling Open()/OpenAsync() on one throws - hence the removed OpenAsync calls on
the reader paths.
OperationTrackingOptions.ConnectionString is now vestigial for this store; the
database location is LocalDb:Path. The options class stays (retention settings)
and the config key stays bound for the site config DB.
InitializeSchema is kept but is now always a no-op in the host: onReady runs
while ILocalDb is being constructed, strictly before this constructor can
receive it. It remains so a directly-constructed store (tests, tooling) is
self-sufficient.
Tests: the store fixture moves off mode=memory&cache=shared onto a real ILocalDb
over a temp file. There is no in-memory mode - LocalDbOptions.Path is a
filesystem path - and testing through a raw in-memory connection would no longer
resemble the host. Verifier connections stay raw on purpose: this fixture never
registers the table, so no trigger fires and the UDF is never reached.
Three DI fixtures needed LocalDb:Path added, because resolving
IOperationTrackingStore now forces ILocalDb construction:
SiteCompositionRootTests, SiteAuditWiringTests, and the AuditLog
CombinedTelemetryHarness.
Verified: build 0 warnings; Host 294/294, SiteRuntime 530/530, AuditLog 354/354.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Task 3. Site nodes now open one ZB.MOM.WW.LocalDb-managed database holding
OperationTracking and site_events as replicated tables.
SiteLocalDbSetup.OnReady applies both schemas and then RegisterReplicated's them.
That order is load-bearing and easy to get silently wrong: capture is
trigger-based CDC, so rows written before registration are never captured or
snapshotted - they would be invisible to the peer forever, with no error anywhere.
Storage ships UNCONDITIONALLY, no feature flag. With no peer configured LocalDb is
just a local SQLite file, so this is behaviour-equivalent to what site nodes do
today. Replication is opt-in and lands in Program.cs (Task 7), which owns the gRPC
host the sync endpoint needs.
Config: LocalDb:Path added to all EIGHT site-node appsettings (the plan said six -
docker-env2's site-x pair exists too), plus the dev template and the wonder-app-vd03
production sample. All ten are required, not optional: LocalDbOptions.Path is
validated with ValidateOnStart, so a site node without it fails fast at startup.
That is the desired posture, and it is what caught ActorPathTests - the one Host
fixture that actually starts a host - whose config now sets the path too.
Note the path choice fixes an existing data-loss bug in passing: site-tracking.db
and site_events.db both defaulted to CWD-relative paths (/app/...), outside the
mounted volume, so they were silently lost on every container recreate. The
consolidated database lives on /app/data.
Tests are container-built against the REAL SiteServiceRegistration.Configure, not a
hand-assembled ServiceCollection - this family has shipped three wiring defects that
only a real graph would catch (inert Secrets replicator 0.2.0, singleton deadlock
0.2.2, Secrets.Ui missing IAuditWriter). They assert the library's own view of what
is registered rather than that we called the right method.
Verified (all after the change):
dotnet build ZB.MOM.WW.ScadaBridge.slnx -> 0 Error(s), 0 Warning(s)
Host.Tests -> 294 passed (5 new, red-first)
SiteRuntime.Tests -> 530 passed
CentralUI.Tests -> 925 passed
Commons.Tests -> 684 passed
Communication.Tests -> 312 passed
SiteEventLogging.Tests-> 70 passed
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Secrets.Ui components inject the SHARED ZB.MOM.WW.Audit.IAuditWriter seam,
which is deliberately distinct from the repo's own Commons IAuditWriter —
the G-4 adoption mounted the UI without registering it, so component
activation threw and the page 500'd on every render (hidden behind the
login wall; it plausibly never worked).
Fix: CentralSharedSeamAuditWriter forwards the shared seam into the central
direct-write path (ICentralAuditWriter -> dbo.AuditLog) — NOT the site
SQLite chain, which is never resolved on central and has no forwarder
there. Registered in the Central composition root next to the Secrets UI
authorization wiring.
Regression pin: CentralCompositionRootTests resolves the full Secrets.Ui
component injection set (ISecretStore / ISecretCipher / shared
IAuditWriter) out of the REAL Program.cs via WebApplicationFactory, plus a
type check that the seam is the central bridge. Red on the previous
commit; the defect class is invisible until the DI graph is actually
built.
Live-proven on the local docker cluster: /admin/secrets 200 + interactive
(Blazor circuit + working @onclick), and secret.add / secret.delete events
(both outcomes) landed in central dbo.AuditLog via the bridge.
Closes#22
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Routes both host-container secret registrations (central role in Program.cs,
site role in SiteServiceRegistration) through a new SecretsRegistration
composition seam that optionally enables ZB.MOM.WW.Secrets.Replicator.SqlServer
hub-replication mode: each node keeps a LOCAL store that syncs bidirectionally
with a shared central SQL hub, so a site cluster keeps resolving secrets
straight through a WAN outage to central.
OPT-IN GATE (the load-bearing part). AddZbSecretsSqlServerReplication validates
its options EAGERLY at registration time, so wiring it unconditionally would
make ScadaBridge fail to START anywhere Secrets:SqlServer:ConnectionString is
unset -- every dev box, every docker node, every existing deployment. The
SQL-Server package is therefore only touched when BOTH Secrets:Replication:
Enabled is true AND a non-blank connection string is present; otherwise the
registration is byte-identical to the previous plain AddZbSecrets call.
Enabled-without-a-connection-string falls back to local-only and logs a warning
rather than failing the node or silently looking healthy.
BOOTSTRAP CYCLE. The hub connection string is itself a secret and can never come
from the hub -- a node cannot read the hub to learn how to reach the hub. It must
arrive from outside the replicated set: an environment variable, or a ${secret:}
reference seeded in that node's own LOCAL store. appsettings.json therefore ships
ConnectionString empty with a _comment saying so (leaf keys starting with '_' are
skipped by the reference expander, verified in SecretReferenceExpander). No real
connection string is committed. The pre-host ${secret:} expander in Program.cs is
deliberately left on a plain local SQLite store for the same reason.
Per-node docker appsettings are intentionally NOT modified -- replication stays
off there for now.
Tests written before the wiring and confirmed red first (2 failed / 4 passed),
green after (6/6). They BUILD a container and RESOLVE from it rather than
asserting over ServiceDescriptors: a decorator can look correct as a descriptor
list and still throw on first resolve because the undecorated concrete store it
depends on is missing -- that exact defect shipped once in this library with all
descriptor-level tests green, so the undecorated SqliteSecretStore gets its own
resolution test.
Verified: Host.Tests 285/285 pass; full build (all projects except the
pre-existing AngleSharp NU1902 CentralUI.Tests restore break) 0 warnings,
0 errors.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
CentralDbTestEnvironment sets five process-wide environment variables, and
Program's AddEnvironmentVariables() reads them at an unpredictable point during
host boot. With xUnit collection parallelization on, one fixture's teardown
could clear a var mid-boot for a sibling. Since the secrets adoption (G-4)
three of those keys are ${secret:...} references that fail closed, turning a
previously benign empty value into a SecretNotFoundException that aborts the
boot — an intermittent CI failure.
Adds a HostBootCollection that serializes every fixture booting a real host
while depending on that shared state, folding in the narrower "ActorSystem"
collection so its members stay serialized with each other as before. Site-role
fixtures stay parallel: they call Configuration.Sources.Clear(), dropping the
env-var provider, so they cannot participate in the race.
CentralDbTestEnvironment now also fails fast if two instances are ever live at
once, making a regression (a fixture added outside the collection) deterministic
rather than intermittent — this is what surfaced the disposed-CTS defect fixed
in the previous commit. Fixture teardown is try/finally so a throwing host
teardown can no longer strand the vars for the rest of the run.
Cost: Host.Tests runs ~2m30s -> ~4m10s; the serialized Central-boot classes are
most of the assembly's parallelism.
Fixes: Gitea #15
Switch the three literal ${SCADABRIDGE_*} placeholders in appsettings.Central.json
to ${secret:} references resolved by the pre-host secrets expander, and update the
_secrets doc note. Fix Host.Tests fixtures that boot the real Program pipeline against
Central config: CentralDbTestEnvironment now also supplies the LDAP service-account
password and JWT signing key as whole-key env overrides (rollback path) so the expander
skips the tokens without a seeded store; the three fixtures that managed env vars
inline now use CentralDbTestEnvironment.
VERIFY-THEN-FIX: the plan-authoring discovery is CONFIRMED. No code anywhere in
src/ registered IOperationTrackingStore in DI, yet AkkaHostedService and
AuditLog SCE both comment that AddSiteRuntime provides it. Every consumer
resolved it via GetService (null-tolerant) and silently ran degraded:
cached-drain scheduler never armed, PullSiteCalls reconciliation seam never
wired, Tracking.Status degraded to audit-only. This is a FUNCTIONAL FIX riding
the hygiene plan: AddSiteRuntime now registers the store (site-only), so
site-local cached-call tracking runs in its intended mode. OperationTrackingOptions
is now bound + eagerly validated in the Host site-options block.
Extend CommunicationOptionsValidator with eager bounds for the four T4
live-alarm-cache options (linger >= 0, reconcile > 0, seed concurrency 1..64,
subscribers-per-site >= 1). Enforce the per-site viewer cap fail-safe in
SiteAlarmLiveCacheService.Subscribe (reject excess viewers with a no-op
disposable rather than growing the list or throwing into the Blazor render
path). Surface two telemetry instruments on the existing ScadaBridgeTelemetry
meter: an active-aggregator observable gauge and a reconnect counter, wired from
the aggregator actor's PreStart/PostStop and its NodeA<->NodeB flip /
reconcile-driven reopen.
Claude-Session: https://claude.ai/code/session_01MtdgwpEeCUn6cUA5f1LMPj
New Commons seam: ScriptArtifactsChanged record + ScriptArtifactKinds + the
IScriptArtifactChangeBus pub/sub interface. Host ships InProcessScriptArtifactChangeBus
(lock-free copy-on-write, swallow-and-log per subscriber) registered as a central-role
singleton. BundleImporter gains an optional IScriptArtifactChangeBus? and, AFTER
tx.CommitAsync, publishes one notification per script-bearing kind (ApiMethod/
SharedScript/Template) whose resolution was Overwrite/Rename, using post-resolution
names — fully guarded so a bad subscriber can't fail a committed import. Contract +
plan-06 handoff table documented. No consumer yet (additive; publisher is correct).
Claude-Session: https://claude.ai/code/session_01MtdgwpEeCUn6cUA5f1LMPj
WhenTerminated watchdog calls IHostApplicationLifetime.StopApplication() when
the ActorSystem dies outside StopAsync (SBR self-down + run-coordinated-shutdown-when-down),
so the service supervisor restarts the node as a fresh incarnation. Optional
ctor param keeps every existing construction site compiling.
Plan's deploy/wonder-app-vd03/install.ps1 service-recovery-actions edit is
skipped: that production deploy artifact is not tracked in this repo (Task 16
skipped its appsettings.Central.json for the same reason).
- Remove redundant linked CancellationTokenSource in ProbeAsync; pass the
framework cancellationToken and ProbeTimeout directly to Ask (the two-CTS
pattern was redundant — Ask already honours both the timeout and the token).
- Add EchoActor XML <remarks> explaining why no Receive<Identify> handler is
needed (ActorBase answers Identify automatically).
- Add PreCancelledToken_ReportsUnhealthy_DoesNotThrow test: verifies the
never-throws guarantee on the shutdown-race path (token already cancelled
before CheckHealthAsync is invoked).
REQ-HOST-4a lists "required cluster singletons running (if applicable)" as a
readiness criterion, but /health/ready only checked database + akka-cluster.
Add a third Ready-tagged check, RequiredSingletonsHealthCheck, registered in the
Central-role AddHealthChecks() chain (so it is naturally role-scoped — site nodes
never run it).
Probe: for each required central singleton, Ask its local ClusterSingletonProxy
an Identify with a short bounded per-singleton timeout (~2s, probes run
concurrently via Task.WhenAll). A non-null ActorIdentity.Subject within the
timeout means the singleton is running and reachable through the proxy; a null
subject or a timeout means unreachable → Unhealthy, naming the unreachable
singleton(s). The check never throws (catch-all → Unhealthy) and resolves
ActorSystem lazily from DI per probe (Unhealthy if Akka not yet up).
Required-always set = the five singleton proxies created unconditionally in
AkkaHostedService.RegisterCentralActors: notification-outbox, audit-log-ingest,
site-call-audit, audit-log-purge, site-audit-reconciliation. There are no
feature/config-gated central singletons today; any future gated singleton is the
"if applicable" case and must NOT be added to the required set.
Leadership-agnostic: the proxy reaches the singleton from either central node, so
a ready standby still reports ready (readiness must not require cluster
leadership — that is the Active tier's job). During a brief singleton handover the
probe may time out and the node flaps to not-ready, which is correct (a node
mid-handover is legitimately not fully ready); no retries, to keep the probe fast.
Tests (TDD): RequiredSingletonsHealthCheckTests exercises the probe against a
TestKit ActorSystem — all proxies present+reachable → Healthy; one missing →
Unhealthy naming it; ActorSystem absent → Unhealthy, no throw. HealthCheckTests
regression-guards the Ready tag + absence of the Active tag on the new check.
REQ-HOST-3/REQ-HOST-4 require a MachineDataDb connection string for Central nodes.
The shipped docker appsettings (docker/central-node-a/appsettings.Central.json and
central-node-b) already carry the key. Host-008 had removed the fail-fast Require
because MachineDataDb had no consumer yet; this commit reverses that decision so a
misconfigured or missing connection string is caught at startup with a clear error.
Changes:
- DatabaseOptions: add MachineDataDb property with XML doc comment
- StartupValidator: add .Require for ScadaBridge:Database:MachineDataDb inside the
existing Central .When block, immediately after the ConfigurationDb Require
- StartupValidatorTests: rename Central_MissingMachineDataDb_PassesValidation ->
FailsValidation and flip to Assert.Throws; update comment to cite REQ-HOST-3/4,
shipped docker appsettings, and the Host-008 reversal; add MachineDataDb to
ValidCentralConfig() so all other Central tests remain green
- CentralDbTestEnvironment: supply ScadaBridge__Database__MachineDataDb env var
(mirrors ConfigurationDb pattern) so HostStartupTests, HealthCheckTests, and
MetricsEndpointTests pass through the new Require
- CompositionRootTests, AkkaHostedServiceAuditWiringTests, ActorPathTests: set
ScadaBridge__Database__MachineDataDb env var alongside the pepper env var and
clear it in Dispose, matching the existing pepper handling pattern
Build: 0 warnings, 0 errors. dotnet test Host.Tests: 233/233 passed.
Replace dc=scadabridge,dc=local with dc=zb,dc=local in all dev/test LDAP
references — app config, docker test-cluster node configs (docker/ and
docker-env2/), GLAuth fixture, dev tooling, Host.Tests fixtures,
IntegrationTests factory, and operational test_infra docs. OU structure
(ou=SCADA-Admins,ou=users,etc.) preserved throughout. Email domains
(@scadabridge.local), hostnames, and container names are untouched.
Historical plan docs (2026-05-24-second-environment.md,
2026-05-31-folder-repo-rename-scadabridge-design.md) excluded as
point-in-time records. No synthetic dc=example,dc=com placeholders touched.
Commit 1fcc4f5 added a Central-only Require for ScadaBridge:InboundApi:ApiKeyPepper
(>=16 chars) to StartupValidator. That Require fires in Program.cs before WebApplicationFactory
can apply any WithWebHostBuilder config overlays, so it must be satisfied via environment
variables (which ARE in the pre-host AddEnvironmentVariables() pass).
Fix (test-only, no src/ changes):
- CentralDbTestEnvironment: add ScadaBridge__InboundApi__ApiKeyPepper env var (TestPepper
constant, 23 chars) alongside the existing db connection string; restore on Dispose.
Fixes HealthCheckTests, MetricsEndpointTests, and HostStartupTests.CentralRole_StartsWithoutError
which all use CentralDbTestEnvironment.
- CentralActorPathTests.InitializeAsync: set the pepper env var before WebApplicationFactory
is constructed (the class uses IAsyncLifetime directly, not CentralDbTestEnvironment).
- CentralCompositionRootTests ctor + Dispose: same env-var pattern; those tests already had
the pepper in AddInMemoryCollection (DI-layer only, too late for pre-host validation).
- CentralAuditWiringTests ctor + Dispose: same env-var pattern for the same reason.
- StartupValidatorTests.ValidCentralConfig(): add pepper so the unit tests that call
StartupValidator.Validate() directly with a Central config stop failing.
- Add guard tests: Central_MissingApiKeyPepper_FailsValidation,
Central_ShortApiKeyPepper_FailsValidation, Site_ApiKeyPepper_NotRequired — these lock
the production behavior introduced by 1fcc4f5.
Joseph Doherty