WhenTerminated watchdog calls IHostApplicationLifetime.StopApplication() when
the ActorSystem dies outside StopAsync (SBR self-down + run-coordinated-shutdown-when-down),
so the service supervisor restarts the node as a fresh incarnation. Optional
ctor param keeps every existing construction site compiling.
Plan's deploy/wonder-app-vd03/install.ps1 service-recovery-actions edit is
skipped: that production deploy artifact is not tracked in this repo (Task 16
skipped its appsettings.Central.json for the same reason).
Remove project bookkeeping citations from shipped code comments across the
solution: hyphenated task IDs (WP-14, StoreAndForward-025), milestone/task/
issue refs (M3, Task 4, Audit Log #23, #21), Bundle X task-bundle labels,
and C/D/K/S/T phase labels.
Comment text only — no code logic, string/log literals, or XML-doc structure
changed. Genuine descriptions are preserved (only the citation is stripped),
and technical lookalikes are retained (UTF-8, SHA-256, T00:00:00, M365,
UTC-5, pre-C4/pre-C5 schema versions). Flagged by the new CommentChecker
TaskReferenceInComment / TrackingReferenceInComment checks plus targeted
grep passes; full solution builds clean, append-only guard tests pass.
Notify-and-fetch follow-ups:
- PendingDeploymentPurgeActor: a central cluster singleton (not
readiness-gated, best-effort) that sweeps expired PendingDeployment
staging rows on CommunicationOptions.PendingDeploymentPurgeInterval
(default 1h). Modeled on the kpi-history-recorder pattern: self-scheduling
timer, per-tick DI scope -> IDeploymentManagerRepository, continue-on-error.
Wired in AkkaHostedService.RegisterCentralActors (manager + proxy + drain);
resolves the deferred TODO in DeploymentService. Correctness never depends
on it (supersession bounds rows to <=1/instance; the fetch endpoint enforces
the TTL), so it is deliberately absent from RequiredSingletonsHealthCheck.
- SQL Server integration test for StagePendingIfAbsentAsync re-staging an
instance's OWN DeploymentId over an expired row against the real UNIQUE
index on DeploymentId — confirms EF orders DELETE before INSERT in one
SaveChanges (SQLite's constraint timing differs from SQL Server's). Plus
a same-instance supersession variant on real SQL Server.
Tests: 2 TestKit actor tests + 2 SQL Server integration tests (both ran
green against the infra MSSQL container); 235 Communication + 15
PendingDeployment tests pass; Host builds 0 warnings.
Populates ScadaBridge:Communication:CentralFetchBaseUrl in every central
appsettings for the notify-and-fetch deploy flow. An empty value now
causes a fail-fast on deploy; this config prevents that regression.
- docker/: http://scadabridge-traefik (in-network Traefik LB, port 80)
- docker-env2/: http://scadabridge-env2-traefik (env2 Traefik LB, port 80)
- src/Host base: http://localhost:5000 (ASP.NET Core default for single-host dev)
- deploy/wonder-app-vd03: http://localhost:8085 (gitignored; edited in main repo)
RUNBOOK Upgrading section updated with note on this setting.
Remediation from the full per-module code review at 4307c381 (findings recorded
separately in code-reviews/).
Highs fixed:
- DeploymentManager-025/SiteRuntime-031: stop broadcasting notification lists + SMTP
configs (incl. credentials) to sites; site purges already-persisted rows on apply
(enforces the central-only delivery design; clears plaintext SMTP creds at rest).
- DataConnectionLayer-023: guard the native-alarm subscribe path against the
mid-flight-unsubscribe adapter-feed leak (mirrors the DCL-021 tag-path fix).
- SiteEventLogging-024: normalize From/To query bounds to UTC (the -016 fix the
audit trail claimed but never committed).
- KpiHistory-001: add an in-flight guard to the recorder sample tick.
- ScriptAnalysis-001: harden the trust analyzer's TPA-absent fallback (resolve
forbidden anchors in the minimal reference set; warn on degraded mode) — anchors
added to validation references only, never the compile gate.
(InboundAPI-026 left to the feat/ipsen-movein effort per owner decision.)
Medium/Low: DM-026 deterministic deploy-status tiebreaker; SR-027/028/029/030
native-alarm leak/phantom-active/delete-during-redeploy fixes; AL-013/014/016;
TE-024 (folder-mutation audit rows now persisted)/025; SF-025 gauge-provider
clear-on-stop; ESG-025/026; SEC-023/024/025; SCA-007/008/009; plus doc/test
accuracy COM-023/024, HOST-025/026, HM-024/025, NS-027/028.
Full-solution build 0 warnings; ~3560 tests across 18 touched suites green.
Wire the M6 KPI History recorder into the central composition path:
- Program.cs: call services.AddKpiHistory(configuration) on the central-only
branch alongside AddNotificationOutbox/AddAuditLog/AddSiteCallAudit.
- AkkaHostedService.cs: register KpiHistoryRecorderActor as a central,
non-role-scoped ClusterSingletonManager + ClusterSingletonProxy + a
PhaseClusterLeave CoordinatedShutdown graceful-stop drain (singleton name
'kpi-history-recorder'), copied/adapted from the audit-log-purge block.
- appsettings.Central.json (Host + docker + docker-env2 central nodes): add a
ScadaBridge:KpiHistory section (SampleInterval 00:01:00, RetentionDays 90,
PurgeInterval 1.00:00:00, DefaultMaxSeriesPoints 200).
KPI history is observability/best-effort and MUST NOT gate readiness: the
recorder is deliberately NOT added to RequiredSingletonsHealthCheck or any
other readiness gate.
Adds a "Dev Disable-Login Flag" subsection to Component-Security.md covering
ScadaBridge:Security:Auth:DisableLogin / User, the AutoLoginAuthenticationHandler
mechanism, and the no-environment-guard / startup-warning production risk.
Ships DisableLogin: false under ScadaBridge → Security → Auth in:
- src/.../Host/appsettings.json (canonical default)
- docker/central-node-a/appsettings.Central.json
- docker/central-node-b/appsettings.Central.json
Also records DL-3 commit SHAs in the plan tasks file.
- Remove redundant linked CancellationTokenSource in ProbeAsync; pass the
framework cancellationToken and ProbeTimeout directly to Ask (the two-CTS
pattern was redundant — Ask already honours both the timeout and the token).
- Add EchoActor XML <remarks> explaining why no Receive<Identify> handler is
needed (ActorBase answers Identify automatically).
- Add PreCancelledToken_ReportsUnhealthy_DoesNotThrow test: verifies the
never-throws guarantee on the shutdown-race path (token already cancelled
before CheckHealthAsync is invoked).
REQ-HOST-4a lists "required cluster singletons running (if applicable)" as a
readiness criterion, but /health/ready only checked database + akka-cluster.
Add a third Ready-tagged check, RequiredSingletonsHealthCheck, registered in the
Central-role AddHealthChecks() chain (so it is naturally role-scoped — site nodes
never run it).
Probe: for each required central singleton, Ask its local ClusterSingletonProxy
an Identify with a short bounded per-singleton timeout (~2s, probes run
concurrently via Task.WhenAll). A non-null ActorIdentity.Subject within the
timeout means the singleton is running and reachable through the proxy; a null
subject or a timeout means unreachable → Unhealthy, naming the unreachable
singleton(s). The check never throws (catch-all → Unhealthy) and resolves
ActorSystem lazily from DI per probe (Unhealthy if Akka not yet up).
Required-always set = the five singleton proxies created unconditionally in
AkkaHostedService.RegisterCentralActors: notification-outbox, audit-log-ingest,
site-call-audit, audit-log-purge, site-audit-reconciliation. There are no
feature/config-gated central singletons today; any future gated singleton is the
"if applicable" case and must NOT be added to the required set.
Leadership-agnostic: the proxy reaches the singleton from either central node, so
a ready standby still reports ready (readiness must not require cluster
leadership — that is the Active tier's job). During a brief singleton handover the
probe may time out and the node flaps to not-ready, which is correct (a node
mid-handover is legitimately not fully ready); no retries, to keep the probe fast.
Tests (TDD): RequiredSingletonsHealthCheckTests exercises the probe against a
TestKit ActorSystem — all proxies present+reachable → Healthy; one missing →
Unhealthy naming it; ActorSystem absent → Unhealthy, no throw. HealthCheckTests
regression-guards the Ready tag + absence of the Active tag on the new check.
REQ-HOST-3/REQ-HOST-4 require a MachineDataDb connection string for Central nodes.
The shipped docker appsettings (docker/central-node-a/appsettings.Central.json and
central-node-b) already carry the key. Host-008 had removed the fail-fast Require
because MachineDataDb had no consumer yet; this commit reverses that decision so a
misconfigured or missing connection string is caught at startup with a clear error.
Changes:
- DatabaseOptions: add MachineDataDb property with XML doc comment
- StartupValidator: add .Require for ScadaBridge:Database:MachineDataDb inside the
existing Central .When block, immediately after the ConfigurationDb Require
- StartupValidatorTests: rename Central_MissingMachineDataDb_PassesValidation ->
FailsValidation and flip to Assert.Throws; update comment to cite REQ-HOST-3/4,
shipped docker appsettings, and the Host-008 reversal; add MachineDataDb to
ValidCentralConfig() so all other Central tests remain green
- CentralDbTestEnvironment: supply ScadaBridge__Database__MachineDataDb env var
(mirrors ConfigurationDb pattern) so HostStartupTests, HealthCheckTests, and
MetricsEndpointTests pass through the new Require
- CompositionRootTests, AkkaHostedServiceAuditWiringTests, ActorPathTests: set
ScadaBridge__Database__MachineDataDb env var alongside the pepper env var and
clear it in Dispose, matching the existing pepper handling pattern
Build: 0 warnings, 0 errors. dotnet test Host.Tests: 233/233 passed.
Site Call Audit (#22): build the documented periodic reconciliation PULL
self-heal path for the eventually-consistent central SiteCalls mirror, as a
dedicated PullSiteCalls gRPC RPC kept separate from the audit pull. This is the
pull PLUMBING only; the central reconciliation tick is a separate follow-up.
- IOperationTrackingStore.ReadChangedSinceAsync(sinceUtc, batchSize): inclusive
UpdatedAtUtc cursor, oldest-first, batch-capped; SQLite impl projects tracking
rows onto SiteCallOperational (Kind->Channel, TargetSummary->Target, SourceSite
left empty - the store has no site-id column).
- sitestream.proto: rpc PullSiteCalls + PullSiteCallsRequest/Response, mirroring
PullAuditEvents; regenerated checked-in SiteStreamGrpc/*.cs.
- SiteCallDtoMapper.ToDto(SiteCallOperational): inverse of FromDto for the handler.
- SiteStreamGrpcServer.PullSiteCalls handler + SetOperationTrackingStore seam;
Host wires the seam alongside SetSiteAuditQueue (site roles only).
- Central IPullSiteCallsClient + GrpcPullSiteCallsClient (home: AuditLog/Central to
reuse ISiteEnumerator; SiteCallAudit does not reference AuditLog). Re-stamps
SourceSite from the dialed siteId; no-throw on tolerable transport faults;
SpecifyKind (not ToUniversalTime) cursor handling. Central-only DI registration.
Tests: ReadChangedSinceAsync (4), PullSiteCalls handler (6), GrpcPullSiteCallsClient
(8). Full solution build 0 warnings/0 errors (TreatWarningsAsErrors).
Per-probe health-check child scopes were disposing the AddTransient-bridged
ActorSystem (IDisposable), terminating the live cluster node ~4s after boot and
leaving every singleton-proxy Ask to hang the full 30s QueryTimeout — the central
report pages (/notifications, /site-calls, /monitoring/health) loaded in ~30s.
Bridge it as a singleton via a new lazy AkkaHostedService.GetOrCreateActorSystem()
so child-scope disposal never touches it. Verified: 0 post-startup terminates,
healthy active/standby, report pages ~0.05s, Playwright 68 passed / 0 failed.
Resolve all 622 issues flagged by the enhanced CommentChecker: add missing
<returns> tags (incl. the standard phrasing on non-generic Task methods),
add missing <summary> tags, and replace misused/redundant <inheritdoc/> on
members that override or implement nothing with real documentation.
Documentation-only — no behavior change; solution builds clean.
Host/App.razor uses the kit's <ThemeHead/>/<ThemeScripts/>, but Host had no direct
PackageReference — it relied on CentralUI re-exporting the package transitively.
Add a versionless <PackageReference Include="ZB.MOM.WW.Theme"/> (version pinned by
central PM at Directory.Packages.props) so the declared dependency matches actual
usage and survives any future PrivateAssets/refactor on CentralUI. Additive only;
Host builds clean (0/0).
Replace dc=scadabridge,dc=local with dc=zb,dc=local in all dev/test LDAP
references — app config, docker test-cluster node configs (docker/ and
docker-env2/), GLAuth fixture, dev tooling, Host.Tests fixtures,
IntegrationTests factory, and operational test_infra docs. OU structure
(ou=SCADA-Admins,ou=users,etc.) preserved throughout. Email domains
(@scadabridge.local), hostnames, and container names are untouched.
Historical plan docs (2026-05-24-second-environment.md,
2026-05-31-folder-repo-rename-scadabridge-design.md) excluded as
point-in-time records. No synthetic dc=example,dc=com placeholders touched.
Joseph Doherty