CentralDbTestEnvironment sets five process-wide environment variables, and
Program's AddEnvironmentVariables() reads them at an unpredictable point during
host boot. With xUnit collection parallelization on, one fixture's teardown
could clear a var mid-boot for a sibling. Since the secrets adoption (G-4)
three of those keys are ${secret:...} references that fail closed, turning a
previously benign empty value into a SecretNotFoundException that aborts the
boot — an intermittent CI failure.
Adds a HostBootCollection that serializes every fixture booting a real host
while depending on that shared state, folding in the narrower "ActorSystem"
collection so its members stay serialized with each other as before. Site-role
fixtures stay parallel: they call Configuration.Sources.Clear(), dropping the
env-var provider, so they cannot participate in the race.
CentralDbTestEnvironment now also fails fast if two instances are ever live at
once, making a regression (a fixture added outside the collection) deterministic
rather than intermittent — this is what surfaced the disposed-CTS defect fixed
in the previous commit. Fixture teardown is try/finally so a throwing host
teardown can no longer strand the vars for the rest of the run.
Cost: Host.Tests runs ~2m30s -> ~4m10s; the serialized Central-boot classes are
most of the assembly's parallelism.
Fixes: Gitea #15
Switch the three literal ${SCADABRIDGE_*} placeholders in appsettings.Central.json
to ${secret:} references resolved by the pre-host secrets expander, and update the
_secrets doc note. Fix Host.Tests fixtures that boot the real Program pipeline against
Central config: CentralDbTestEnvironment now also supplies the LDAP service-account
password and JWT signing key as whole-key env overrides (rollback path) so the expander
skips the tokens without a seeded store; the three fixtures that managed env vars
inline now use CentralDbTestEnvironment.
VERIFY-THEN-FIX: the plan-authoring discovery is CONFIRMED. No code anywhere in
src/ registered IOperationTrackingStore in DI, yet AkkaHostedService and
AuditLog SCE both comment that AddSiteRuntime provides it. Every consumer
resolved it via GetService (null-tolerant) and silently ran degraded:
cached-drain scheduler never armed, PullSiteCalls reconciliation seam never
wired, Tracking.Status degraded to audit-only. This is a FUNCTIONAL FIX riding
the hygiene plan: AddSiteRuntime now registers the store (site-only), so
site-local cached-call tracking runs in its intended mode. OperationTrackingOptions
is now bound + eagerly validated in the Host site-options block.
Extend CommunicationOptionsValidator with eager bounds for the four T4
live-alarm-cache options (linger >= 0, reconcile > 0, seed concurrency 1..64,
subscribers-per-site >= 1). Enforce the per-site viewer cap fail-safe in
SiteAlarmLiveCacheService.Subscribe (reject excess viewers with a no-op
disposable rather than growing the list or throwing into the Blazor render
path). Surface two telemetry instruments on the existing ScadaBridgeTelemetry
meter: an active-aggregator observable gauge and a reconnect counter, wired from
the aggregator actor's PreStart/PostStop and its NodeA<->NodeB flip /
reconcile-driven reopen.
Claude-Session: https://claude.ai/code/session_01MtdgwpEeCUn6cUA5f1LMPj
New Commons seam: ScriptArtifactsChanged record + ScriptArtifactKinds + the
IScriptArtifactChangeBus pub/sub interface. Host ships InProcessScriptArtifactChangeBus
(lock-free copy-on-write, swallow-and-log per subscriber) registered as a central-role
singleton. BundleImporter gains an optional IScriptArtifactChangeBus? and, AFTER
tx.CommitAsync, publishes one notification per script-bearing kind (ApiMethod/
SharedScript/Template) whose resolution was Overwrite/Rename, using post-resolution
names — fully guarded so a bad subscriber can't fail a committed import. Contract +
plan-06 handoff table documented. No consumer yet (additive; publisher is correct).
Claude-Session: https://claude.ai/code/session_01MtdgwpEeCUn6cUA5f1LMPj
WhenTerminated watchdog calls IHostApplicationLifetime.StopApplication() when
the ActorSystem dies outside StopAsync (SBR self-down + run-coordinated-shutdown-when-down),
so the service supervisor restarts the node as a fresh incarnation. Optional
ctor param keeps every existing construction site compiling.
Plan's deploy/wonder-app-vd03/install.ps1 service-recovery-actions edit is
skipped: that production deploy artifact is not tracked in this repo (Task 16
skipped its appsettings.Central.json for the same reason).
- Remove redundant linked CancellationTokenSource in ProbeAsync; pass the
framework cancellationToken and ProbeTimeout directly to Ask (the two-CTS
pattern was redundant — Ask already honours both the timeout and the token).
- Add EchoActor XML <remarks> explaining why no Receive<Identify> handler is
needed (ActorBase answers Identify automatically).
- Add PreCancelledToken_ReportsUnhealthy_DoesNotThrow test: verifies the
never-throws guarantee on the shutdown-race path (token already cancelled
before CheckHealthAsync is invoked).
REQ-HOST-4a lists "required cluster singletons running (if applicable)" as a
readiness criterion, but /health/ready only checked database + akka-cluster.
Add a third Ready-tagged check, RequiredSingletonsHealthCheck, registered in the
Central-role AddHealthChecks() chain (so it is naturally role-scoped — site nodes
never run it).
Probe: for each required central singleton, Ask its local ClusterSingletonProxy
an Identify with a short bounded per-singleton timeout (~2s, probes run
concurrently via Task.WhenAll). A non-null ActorIdentity.Subject within the
timeout means the singleton is running and reachable through the proxy; a null
subject or a timeout means unreachable → Unhealthy, naming the unreachable
singleton(s). The check never throws (catch-all → Unhealthy) and resolves
ActorSystem lazily from DI per probe (Unhealthy if Akka not yet up).
Required-always set = the five singleton proxies created unconditionally in
AkkaHostedService.RegisterCentralActors: notification-outbox, audit-log-ingest,
site-call-audit, audit-log-purge, site-audit-reconciliation. There are no
feature/config-gated central singletons today; any future gated singleton is the
"if applicable" case and must NOT be added to the required set.
Leadership-agnostic: the proxy reaches the singleton from either central node, so
a ready standby still reports ready (readiness must not require cluster
leadership — that is the Active tier's job). During a brief singleton handover the
probe may time out and the node flaps to not-ready, which is correct (a node
mid-handover is legitimately not fully ready); no retries, to keep the probe fast.
Tests (TDD): RequiredSingletonsHealthCheckTests exercises the probe against a
TestKit ActorSystem — all proxies present+reachable → Healthy; one missing →
Unhealthy naming it; ActorSystem absent → Unhealthy, no throw. HealthCheckTests
regression-guards the Ready tag + absence of the Active tag on the new check.
REQ-HOST-3/REQ-HOST-4 require a MachineDataDb connection string for Central nodes.
The shipped docker appsettings (docker/central-node-a/appsettings.Central.json and
central-node-b) already carry the key. Host-008 had removed the fail-fast Require
because MachineDataDb had no consumer yet; this commit reverses that decision so a
misconfigured or missing connection string is caught at startup with a clear error.
Changes:
- DatabaseOptions: add MachineDataDb property with XML doc comment
- StartupValidator: add .Require for ScadaBridge:Database:MachineDataDb inside the
existing Central .When block, immediately after the ConfigurationDb Require
- StartupValidatorTests: rename Central_MissingMachineDataDb_PassesValidation ->
FailsValidation and flip to Assert.Throws; update comment to cite REQ-HOST-3/4,
shipped docker appsettings, and the Host-008 reversal; add MachineDataDb to
ValidCentralConfig() so all other Central tests remain green
- CentralDbTestEnvironment: supply ScadaBridge__Database__MachineDataDb env var
(mirrors ConfigurationDb pattern) so HostStartupTests, HealthCheckTests, and
MetricsEndpointTests pass through the new Require
- CompositionRootTests, AkkaHostedServiceAuditWiringTests, ActorPathTests: set
ScadaBridge__Database__MachineDataDb env var alongside the pepper env var and
clear it in Dispose, matching the existing pepper handling pattern
Build: 0 warnings, 0 errors. dotnet test Host.Tests: 233/233 passed.
Replace dc=scadabridge,dc=local with dc=zb,dc=local in all dev/test LDAP
references — app config, docker test-cluster node configs (docker/ and
docker-env2/), GLAuth fixture, dev tooling, Host.Tests fixtures,
IntegrationTests factory, and operational test_infra docs. OU structure
(ou=SCADA-Admins,ou=users,etc.) preserved throughout. Email domains
(@scadabridge.local), hostnames, and container names are untouched.
Historical plan docs (2026-05-24-second-environment.md,
2026-05-31-folder-repo-rename-scadabridge-design.md) excluded as
point-in-time records. No synthetic dc=example,dc=com placeholders touched.
Commit 1fcc4f5 added a Central-only Require for ScadaBridge:InboundApi:ApiKeyPepper
(>=16 chars) to StartupValidator. That Require fires in Program.cs before WebApplicationFactory
can apply any WithWebHostBuilder config overlays, so it must be satisfied via environment
variables (which ARE in the pre-host AddEnvironmentVariables() pass).
Fix (test-only, no src/ changes):
- CentralDbTestEnvironment: add ScadaBridge__InboundApi__ApiKeyPepper env var (TestPepper
constant, 23 chars) alongside the existing db connection string; restore on Dispose.
Fixes HealthCheckTests, MetricsEndpointTests, and HostStartupTests.CentralRole_StartsWithoutError
which all use CentralDbTestEnvironment.
- CentralActorPathTests.InitializeAsync: set the pepper env var before WebApplicationFactory
is constructed (the class uses IAsyncLifetime directly, not CentralDbTestEnvironment).
- CentralCompositionRootTests ctor + Dispose: same env-var pattern; those tests already had
the pepper in AddInMemoryCollection (DI-layer only, too late for pre-host validation).
- CentralAuditWiringTests ctor + Dispose: same env-var pattern for the same reason.
- StartupValidatorTests.ValidCentralConfig(): add pepper so the unit tests that call
StartupValidator.Validate() directly with a Central config stop failing.
- Add guard tests: Central_MissingApiKeyPepper_FailsValidation,
Central_ShortApiKeyPepper_FailsValidation, Site_ApiKeyPepper_NotRequired — these lock
the production behavior introduced by 1fcc4f5.
Joseph Doherty