Central re-bound its HTTP surface from ASPNETCORE_URLS keeping only the
port, so https://+:443 silently served plaintext HTTP. The parser now keeps
the scheme (ParseHttpBindEndpoints); an https entry binds UseHttps with the
node certificate from the new ScadaBridge:Node:Tls section (CertificatePath
+ CertificatePassword, or CertificateThumbprint + StoreName + StoreLocation;
default unset) and the node refuses to boot, with a clear error, when https
is requested without a certificate. http entries bind exactly as before.
Tests: scheme parsing, refuse-to-boot, http endpoint config identical to the
legacy inline binding (reflected effective Kestrel listen options), and a
real Kestrel HTTPS listener with a generated .pfx serving h2 and HTTP/1.1 by
ALPN while plaintext to the TLS port fails. Reverting the fix turns the
refuse-to-boot and real-listener tests red.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
ScadaBridge:Cluster:RemotingTls (ClusterRemotingTlsOptions) renders
akka.remote.dot-netty.tcp enable-ssl + ssl { certificate, mutual auth,
hostname validation } only when Enabled. Chain validation is always on
(no suppress-validation switch). Certificate is a .pfx path + password
(${secret:} reference) or an OS-store thumbprint.
ClusterOptionsValidator, TLS on only: exactly one certificate source, a
legal store location, and akka.ssl.tcp:// on every seed - Akka.Remote
advertises akka.ssl.tcp addresses once SSL is on, so an akka.tcp seed is
never answered and the join retries forever with no startup error.
ClusterBootstrapGuard.SeedPattern accepted only one scheme segment, so
akka.ssl.tcp seeds parsed as nothing and the guard would silently never
engage on a TLS pair; it now accepts any number.
Default HOCON is byte-identical: 12 rendered documents (3 downing
strategies x guard on/off x Central/Site) captured before the change and
diffed after, all identical; a permanent test pins that a disabled block
with every field set renders the default document. A real ActorSystem
booted on the emitted TLS document comes up as akka.ssl.tcp (mutation
check: forcing enable-ssl=false turns it red).
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Adds LeaderAutoDowningProvider (Host/Actors), a line-for-line port of Akka.NET
1.5.62 AutoDowning/AutoDown/AutoDownBase read from the decompiled assembly: the
leader among the reachable members downs a peer unreachable for StableAfter; a
non-leader records it as pending and downs it on becoming leader; reachable-again
or removed members are forgotten; Down/Exiting members are skipped. Its window is
carried under scadabridge.cluster.leader-auto-down.unreachable-after, so no
deprecated auto-down-unreachable-after key is emitted and Akka's boot-time
"keep-majority will be used instead" warning goes away. DownRemovalMargin reads
akka.cluster.down-removal-margin exactly as AutoDowning does (default off).
New SplitBrainResolverStrategy value "leader-auto-down", wired through BuildHocon
and ClusterOptionsValidator. The default stays "auto-down"; auto-down and
keep-oldest HOCON is byte-identical to before. Placed in Host, not
ClusterInfrastructure, per the Code Placement rule (ClusterInfrastructure stays
Akka-free).
Tests: HOCON + validator units; LeaderAutoDownTests ports Akka's AutoDownSpec
cases (leader-only, pending-on-leader-change, losing leadership, reachable again,
removed, Down/Exiting skipped, single Down per member); LeaderAutoDownFailoverTests
clones SbrFailoverTests' oldest-crash and younger-crash cases on a real two-node
cluster plus a test that Akka actually loaded the provider and its downing actor.
Mutation-checked: a provider that never calls Cluster.Down fails both failover
tests; never-downs, leader-check-dropped, reachable-ignored and pending-not-flushed
mutations each fail the unit tests.
Remaining gate before switching the default: the docker failover drill
(DRILL_MODE=active and standby mode) on leader-auto-down.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The #96 site-psk-invalidation topic was the first cross-node invalidation broadcast.
It is generalised into ONE central DistributedPubSub topic, central-cache-invalidation,
carrying typed messages (Commons ICentralCacheInvalidation: SitePskInvalidated,
RoleMappingsInvalidated, ExternalDefinitionsInvalidated, InboundApiMethodInvalidated)
in an envelope stamped with the publisher's process-unique origin id.
- Host: CentralCacheInvalidationBus (ICentralCacheInvalidationPublisher, no-op until
the mediator is set, never throws) and CentralCacheInvalidationSubscriber, which
skips its own node's envelopes and hands the peer's to every registered
ICentralCacheInvalidationHandler (a failing handler does not stop the rest).
Replaces SitePskInvalidationSubscriber.
- Migration of #96: SitePskRotationInvalidator publishes through the bus and is the
PSK handler; SitePskInvalidated keeps its type name and is also published bare on
site-psk-invalidation, which the subscriber still listens to, so a mixed-version
central pair keeps exchanging PSK invalidations. Remove after one release.
- Audit D1: RoleMappingCache is a handler; ManagementActor's post-commit role-mapping
invalidation also publishes RoleMappingsInvalidated, so a revocation is immediate on
the peer instead of after RoleMappingCacheSeconds.
- Audit D3 / #83 (peer half): ExternalDefinitionLookupCache is a handler; the 9
ManagementActor handlers and the Transport bundle importer publish
ExternalDefinitionsInvalidated after commit. (#83's local import half was already in.)
TTLs remain the fallback for a lost broadcast or a direct SQL edit. Tests: bus and
subscriber unit tests (origin filter, legacy topic), peer-node tests through the real
caches in ManagementService.Tests, importer publish, composition root, and a real
two-node cluster from the production HOCON (CentralCacheInvalidationTwoNodeTests)
proving every message type crosses with the production serializer. Docs: Host,
Communication, Security, ManagementService, ESG, and the #85 cache audit.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
SitePskProvider cached each site's gRPC preshared key forever; only site removal
invalidated it, so rotating SB-GRPC-PSK-{site} broke the central<->site link on
both central nodes until they restarted. The cached key is now dropped when:
- the secret is written through /admin/secrets: central replaces the secrets
library's ISecretCacheInvalidator with SitePskRotationInvalidator, which also
publishes SitePskInvalidated on a new DistributedPubSub topic
(site-psk-invalidation) so the peer central node drops its copy and its
resolver cache entry too (SitePskInvalidationSubscriber);
- a site rejects a central->site call with PermissionDenied/Unauthenticated:
WithSiteCredentials wraps every per-site channel's handler in
PskRejectionRefreshHandler, so the next call re-reads the key (covers
out-of-process writes such as the secret CLI);
- a site->central call presents a token that does not match the cached key:
CentralControlAuthInterceptor invalidates and re-reads once before refusing;
- ScadaBridge:Communication:SitePsks reloads.
Invalidation is case-insensitive because SecretName lowercases names. Rotation
runbook (topology-guide), Component-Communication/Host and the #85 cache audit
(D2) updated.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The site script and alarm on-trigger launchers push the run identity onto LogContext for the
run's lifetime and the host now enables Enrich.FromLogContext(), so log lines written on the
run's own async call chain carry the id. Bounded part only: LogContext does not cross an Akka
message hop, so Instance Actor/DCL/S&F lines and a site_events execution-id column remain open
(documented in the step-logs design doc section 9).
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Replace the private SitePairActiveNodeHealthCheck (503 whenever no ActorSystem, no data)
with the shared ZB.MOM.WW.Health ActiveNodeHealthCheck constructed with the site-{SiteId}
role: Degraded (200) during startup per the Health spec, and it publishes
selfAddress/activeRole/activeNode so the Overview dashboard can name the active site node.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The InboundAPI carried its own IActiveNodeGate (implemented by Host/Health/ActiveNodeGate)
whose rule already matched the shared Health 0.3.0 AkkaActiveNodeGate — both route to
ClusterActiveNode (oldest Up member). The private interface and implementation are deleted;
InboundAPI references ZB.MOM.WW.Health (0.3.0, already pinned) and InboundApiEndpointFilter,
AuditWriteMiddleware, InboundMethodWarmupService and FlowDebugRunner consume
ZB.MOM.WW.Health.IActiveNodeGate. The Central branch of Program.cs registers
AkkaActiveNodeGate (unscoped, like the active-node health check).
Warm-up behaviour is kept: the shared gate answers false while no ActorSystem resolves, while
this member is not Up, and on any cluster-init fault. It resolves the ActorSystem through the
existing DI singleton bridge (GetOrCreateActorSystem), the same path the health probes use.
Tests: CompositionRoot asserts the shared gate type (fails with the registration removed);
the evaluator wiring test now drives AkkaActiveNodeGate; new warm-up test (no ActorSystem ->
not active). Nothing was missing from the Health 0.3.0 package.
(ScadaBridge#80)
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Interface now lives in Commons.Interfaces.Services; AuditLog no longer gets it
transitively through Communication. Implementation stays in HealthMonitoring.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
An OperationCanceledException escaping RunAsync after ApplicationStopping has fired
(WindowsServiceLifetime.StopAsync gets a cancelled token after shutdown finished) is a
normal stop. Program.IsGracefulShutdown now routes it to an Information line; any other
exception, or a cancellation with no shutdown requested, still logs Fatal and rethrows.
- ManagementAuthenticator accepts Authorization: Bearer <jwt> alongside Basic on
/management, the audit REST endpoints and the debug-stream hub. Tokens are
validated locally (HS256 only, alg=none rejected, issuer/audience, lifetime with
30 s skew); roles and site scope inside a valid token are authoritative for its
lifetime, so a bearer call costs no LDAP bind and no role re-map.
- /auth/token returns expires_in + management_bearer:true, carries username, display
name, roles and permitted sites, and answers 404 under DisableLogin.
- CLI exchanges credentials once per process (in-memory CliTokenStore), sends Bearer,
re-exchanges once on 401 then falls back to Basic; older/login-disabled servers are
Basic-only; bad credentials are not retried with Basic. Debug stream uses the token
via AccessTokenProvider. SCADABRIDGE_AUTH_MODE=basic opts out.
- StartupValidator fails a Central boot on a JwtSigningKey under 32 UTF-8 bytes;
fixed the false 'fail-fast at construction' comment. Removed the dead
RefreshToken/RecordActivity/ShouldRefresh/IsIdleTimedOut members and the inert
JwtRefreshThresholdMinutes; JwtExpiryMinutes now documented as the bearer lifetime
(validated > 0).
- RoleMapper reads through a process-wide RoleMappingCache (RoleMappingCacheSeconds,
default 30 s): one refresh loads the table plus every Deployer mapping's scope
rules, Invalidate() drops it (generation-guarded against in-flight refreshes).
- Component-Security.md / Component-CLI.md / CLI README describe the real model
(cookie claims ticket, bearer for CLI) instead of a cookie-embedded JWT.
Comment/XML-doc changes only, no code logic:
- strip project-tracking references (WP*, P*-*, row-NN, #NN, Task N) from
comments, keeping the explanatory rationale
- add missing summaries, <param>, <returns>, <typeparam> and <inheritdoc/>
- replace misused <inheritdoc/> with real docs; where <inheritdoc/> was
combined with useful remarks, move the rationale into // comments
- drop stale replicationActor <param> on DeploymentManagerActor ctor
Remaining 5 are false positives (CWE-1236 x2, failures-1, year-0001, M365).
Build 0 errors / 0 warnings.
A DeployedSecretsMirror failure on the STANDBY node (e.g. a master-key mismatch
that stops a deployed secret decrypting there) was only logged on that node:
site health reports are sent by the active node only, so central never saw it.
- New 12th LocalDb-replicated table secret_mirror_status (node_name PK,
failures_json of {SecretName, Message}, checked_at; no FKs). Each node's
mirror writes only its own row, and only when its failure set changes plus
once per start-up (every write replicates via CDC). Best-effort: a failed
write is a Warning and is retried next pass. Node key = INodeIdentityProvider
NodeName, falling back to the machine name.
- SecretMirrorStatusReporter (SiteRuntime, every node, 30 s) reads every row
into ISiteHealthCollector.SetSecretMirrorStatuses -> additive nullable
SiteHealthReport.SecretMirrorStatuses (null = no data).
- Wire: SiteHealthReportDto.secret_mirror_statuses = 33 (presence wrapper so
null != empty); vendored CentralControl.cs regenerated; mapper both ways.
- Central UI Health card: 'Secret mirror failing' badge + compact per-node
node / secret / message list.
- Tests: mirror status write-on-change/first-pass/failure isolation, storage
round-trip, reporter/collector, DTO mapper round-trip incl. null/empty,
bUnit, replicated-table set (twelve) + FK guard wording.
- SecretsRegistration: UsesCentralSharedStore predicate shared by AddScadaBridgeSecrets and the
new AddPreHostExpander, so a Central node's Layer-A ${secret:} expander composes against the
shared SQL store whenever Secrets:Store:Provider=SqlServer does — closing the gap where a
central migrated off the retired Grpc topology would boot-time-resolve config from per-node
SQLite while serving the shared store.
- Program.cs: expander block now calls AddPreHostExpander + resolves the store-agnostic
ISecretsStoreMigrator seam; stale Grpc-hub/replication comment rewritten.
- DeployedSecretsMirror.cs: stale "gRPC-hub follower half" doc comment rewritten to state the
now-permanent local-SQLite-only invariant.
- StartupValidator: Secrets:Replication:Enabled read as a raw string (any non-empty, non-"false"
value rejected) instead of GetValue<bool>, which threw a raw binder crash on "yes"/"1"; added a
CollectWarnings entry for leftover Secrets:Replication/Secrets:GrpcHub child keys once Enabled
is false/unset.
- Tests: +15 (AddPreHostExpander/UsesCentralSharedStore wiring, Replication:Enabled edge values,
leftover-key warnings). Full solution build and full Host.Tests green (593/593).
Second slice of the 2026-09-09 whole-system review remediation
(docs/plans/2026-09-09-arch-review.md §10), on top of the High slice 146a1323.
Runtime/devices: SR-2 alarm on-trigger integration scope; SR-3 correlated NativeAlarmActor
subscribe retry; SR-4 immutable audited DB target; SR-5 ordered InstanceActor stop/flush;
SR-12 rehydration fence; SR-13 S&F gate recheck per delivery; DC-4 no OPC UA security
downgrade; DC-5 alarm snapshot for every new subscriber; ORCH-1 sync-dispose joins the pump.
Management/UI: MG-2 site scope on audit tree + Flow Debugger Replay; MG-3 stale-probe memo
removed; MG-4 server-owned bulk-deploy deadline; UI-5 sandbox runs in a killable child
process (new CentralUI.SandboxWorker project); UI-14 bounded sandbox output/IPC.
Edge/audit: EI-1 relay audit ownership; EI-2 SkipBodyCapture suppresses error text; EI-3/EI-11
error-field redaction + caps; EI-4 single relay deadline, pre-execution-only retry; EI-5
authenticated relay provenance; EI-10 body cap before audit read.
Persistence/contracts: DL-1 global Instance.UniqueName unique index + conflict translation;
DL-2 DataConnection protocol config encrypted at rest (migration
20260909203046_EnforceGlobalInstanceNamesAndProtectProtocolConfiguration — coordinated
central upgrade, old binaries cannot read protected rows); DL-3 bounded Replay paging;
DL-6 cross-site Overwrite of a deployed instance rejected; XC-4 proto descriptor contract test.
Config/hygiene: XC-1 central listener isolation by local port; XC-2/XC-3 inert SMTP + cron
keys removed from rig settings; CM-2 env2 bootstrap guard enabled; DC-6 deploy site exit code;
DC-7 notifier retries only explicit standby rejection; XC-5 ClusterClient purged from docs.
Gates recorded in the report: build 0/0, 9 397 passed / 0 failed, rig redeployed on all
eight nodes with the migration applied. 46 Low findings remain open.
Row 104: docker/vacuum-site-localdb.sh stops both nodes of a pair together, sets
auto_vacuum=INCREMENTAL + VACUUM on each node volume with integrity_check before/after,
then starts both (DRY_RUN=1 reports). Run on all three rig pairs: six files 0 -> 2, ok,
~11% smaller; site-a replication reconnected with backlog 0 and no watermark change —
VACUUM is replication-invisible (ledger keys by declared PK, oplog seq is INTEGER PRIMARY
KEY AUTOINCREMENT). Topology guide gains the recipe.
Row 105: SiteLocalDbReplicatedTableFkGuardTests builds the real site schema via OnReady
(unreplicated config so 0.3.3's own guard cannot mask the culprit) and asserts every
SiteLocalDbSetup.ReplicatedTables entry (now internal) exists and has no inbound
CASCADE/SET NULL/SET DEFAULT foreign key, with a negative control. The constraint is
enforced at build instead of at the first replicated boot.
Row 106 (found during 104's live verification, NOT caused by it — the identical 127k-row
full snapshot also landed at the earlier deploy restart): SnapshotStreamer.SendAsync stamps
as_of_seq = MAX(seq), which is 0 once a converged sender's oplog is ack-pruned to empty;
a zero-watermark receiver (rebuilt node) is never lifted, so the empty-oplog horizon rule
re-streams a full snapshot on every reconnect. Sender-side one-line fix for LocalDb 0.3.4:
as_of_seq = max(MAX(seq), last_acked_seq).
Defect: the hourly KPI rollup fold (KpiHistoryRepository.FoldHourlyRollupsAsync)
sums every Rate-classified sample recorded in an hour, correct only if each
sample is a non-overlapping delta covering exactly one recorder tick
(KpiHistoryOptions.SampleInterval). That invariant was an unenforced silent
assumption across three independently-owned config keys — SampleInterval,
NotificationOutboxOptions.DeliveredKpiWindow, SiteCallAuditOptions.KpiInterval —
so widening DeliveredKpiWindow to e.g. 15 min would silently over-count the
rollup ~15x.
Robust fix: NotificationOutboxKpiSampleSource and SiteCallAuditKpiSampleSource
now rescale their Rate-classified samples (deliveredLastInterval,
failedLastInterval) by SampleInterval / (DeliveredKpiWindow or KpiInterval)
before recording, so the rollup's per-hour Sum is correct regardless of how
the other two keys are tuned — the live dashboard tile's own cutoff/window is
untouched. Both sources gain a project reference to KpiHistory to read
SampleInterval; no cycle (KpiHistory references only Commons).
Eager cross-validation (defense-in-depth): KpiRateFoldIntervalCrossValidator
(Host, the one composition root referencing all three component projects)
fails boot naming all three config keys when they diverge, protecting the
separate (still-real) invariant that the recorded/charted KPI-history sample
equals the live tile at the same instant — which the rescale trades away
whenever the scale factor isn't 1. Rides KpiHistoryOptions' existing
ValidateOnStart trigger.
Also: NotificationOutboxOptions had no registered validator at all before this
(the gap this task's "whichever options class owns DeliveredKpiWindow" file
scope pointed at) — added NotificationOutboxOptionsValidator mirroring
SiteCallAuditOptionsValidator's positive-duration/count convention, wired with
ValidateOnStart.
Tests: new rescale tests in both sample-source suites prove the fold is
numerically correct at a non-default SampleInterval (30/40 counted over a
15/20-minute window correctly divides down to a 1-minute-tick delta); new
KpiRateFoldIntervalCrossValidatorTests prove divergent config fails with all
three keys named; new NotificationOutboxOptionsValidatorTests. Every new test
proven to fail against the unfixed computation first (temporarily
short-circuited the scale factor / cross-check to the pre-fix behavior,
confirmed failure, restored) — not via git stash (repo-wide, shared across
concurrent worktrees this session) but via file-copy swap.
Full suites green: Host.Tests (527), ConfigurationDatabase.Tests (401),
KpiHistory.Tests (54), NotificationOutbox.Tests (163),
SiteCallAudit.Tests (67).
P4-5: EventLogPurgeService.ExecuteAsync now yields (Task.Yield()) before running the
initial RunPurge(), making "the purge never runs synchronously on the caller's thread"
a property of the method itself rather than an artifact of .NET 8+'s BackgroundService
scheduling (verified empirically not to reproduce the literal "StartAsync blocks" symptom
on this repo's net10.0 target — a 1.5s synchronous Thread.Sleep in ExecuteAsync still lets
StartAsync return in ~2ms). Proven instead via a reflection-invoked ExecuteAsync test that
bypasses BackgroundService's Task.Run wrapper, which is timing-independent. Also: a
throwing isActive predicate now fails CLOSED (logged as inactive, at Warning) instead of
silently claiming "active" — the predicate only affects the tick's log line since P4-4,
but a throw must not masquerade as a successful check.
P4-6: PRAGMA auto_vacuum = INCREMENTAL only takes effect on a table-less database, and
LocalDbSchema.EnsureCreated (inside SqliteLocalDb's constructor) always creates LocalDb's
own bookkeeping tables before SiteEventLogSchema.Apply ever gets a connection — so the
pragma there had never once taken effect, in any real call path. Real fix per plan
decision 2: SiteLocalDbSetup.EnsureAutoVacuumOnFreshFile pre-creates a genuinely fresh
LocalDb file with the pragma set, called from SiteServiceRegistration.Configure BEFORE
AddZbLocalDb's lazy singleton ever constructs SqliteLocalDb — no LocalDb API change
needed. Only helps NEW site databases; an existing file (every node on the docker rig
today) is left untouched rather than VACUUM'd, per the plan's explicit rejection of an
unconditional VACUUM against a live ~1 GB file. The now-dead pragma call in
SiteEventLogSchema.Apply is removed and its comment corrected.
Tests proven to fail against the unfixed code (git-diff-and-restore, not stash — this repo
shares one stash ref across all worktrees): ExecuteAsync_YieldsBeforeRunningTheInitial-
PurgeSynchronously and RunPurge_ActiveNodeCheckThrows_TickLogsAsInactive_NotActive fail
without the EventLogPurgeService fix; SiteLocalDbAutoVacuumTests fails to compile without
EnsureAutoVacuumOnFreshFile. SiteEventLogging.Tests 112/112 and Host.Tests 526/526 green
with the fix applied.
P4-2: SqliteAuditWriter.FlushBatch completed each pending row's Task
(TrySetResult) inside the per-row insert loop, before transaction.Commit()
ran. A later failure in the same batch (another row's insert throwing, or
the commit itself failing) rolled the whole transaction back, but the
earlier rows' Tasks had already completed successfully -- the caller and
FallbackAuditWriter (whose ring-buffer fallback exists precisely for this
outage) both saw false success while the rows were actually gone from the
database; the outer catch's TrySetException became a silent no-op against
an already-completed TaskCompletionSource. Every row's Completion is now
deferred and only resolved (TrySetResult for the whole batch, or
TrySetException for the whole batch) after transaction.Commit() succeeds
or the transaction rolls back, mirroring SiteEventLogger.FlushBatch.
P4-4: EventLogPurgeService gated the daily site_events purge on the
active-node check, early-exiting on the standby, on the stale premise that
"the standby receives no writes". DataConnectionActor runs on both site
nodes and writes site_events regardless of active/standby status (register
rows 89/90/97), so an unreplicated standby (the rig's site-b/site-c
posture) grew its site_events file forever with no retention and no cap.
The gate is removed; every node now purges its own file every tick. Safe
on a replicated pair because the retention/cap deletes are CDC-captured
and converge under LWW regardless of which node issues them; required on
an unreplicated pair because each node's file is independent. The
active-node check is retained only to annotate the purge tick's log line.
Tests: new SqliteAuditWriterCommitFailureTests forces a genuine
non-swallowed mid-batch INSERT failure via a real SQLite trigger + custom
scalar function on the writer's own connection (reflection seam), proving
(a) a row whose own insert succeeded still faults when a later row in the
same batch fails, and (b) FallbackAuditWriter rings every faulted row and
increments the failure counter for each -- both assertions fail against
the pre-fix code (proof-of-failure verified by temporarily reverting the
fix). EventLogPurgeServiceTests' standby case is inverted to assert the
purge now runs (not skips) on a standby node.
A healthy standby logged "Health check active-node with status Unhealthy …
'not the site pair primary (standby)'" at Error on every probe — every 10s on a
site node, and (discovered while fixing this) every 5s on a central standby,
which Traefik health-probes at that cadence. Correct-by-design state at an
alarming severity: it floods the log and trains operators to ignore ERR.
Mechanism: DefaultHealthCheckService logs one line per check per probe and picks
the level from the verdict — Debug/Warning/Error for Healthy/Degraded/Unhealthy.
Both active-node checks answer Unhealthy on the standby because 503 IS the
contract /health/active publishes. There is no IHealthCheckPublisher in this
repo, so each line is one inbound HTTP probe.
Fix: one Serilog exclusion filter in LoggerConfigurationFactory. A level
override could not work — Serilog's MinimumLevel.Override only RAISES a
per-source floor, and a LogEvent's Level is immutable, so neither an override
nor an enricher can rewrite Error as Information. Changing the check to report
Degraded was rejected: it would alter the HTTP status that /health/active
publishes, which is load-bearing for Traefik and the family dashboard.
The constraint (genuine failures must still log at Error) is preserved by keying
the predicate on the check NAME rather than the logger category: localdb,
akka-cluster, database and required-singletons keep their Error lines, and the
active-node check has no error path of its own — the active node always answers
Healthy — so the only event this can ever suppress is a standby reporting that
it is a standby. The verdict stays fully observable via /health/active's own
HTTP status, the readiness probe's cluster view, and the health dashboard.
Also corrects a stale CLAUDE.md claim found while verifying what consumes this
check: "/health/active is central-only (site nodes map no /health/* at all)" is
wrong — SiteServiceRegistration registers akka-cluster + localdb + the
role-scoped SitePairActiveNodeHealthCheck, and Program.cs's Site branch calls
MapZbHealth() on the HTTP/1.1 listener.
Fail-before evidence: with the filter behaviourally disabled,
StandbyActiveNodeVerdict_IsNotLogged fails "Collection was not empty" while the
other 9 tests in the file — the must-still-log constraints — pass either way,
which is what proves the filter is narrow rather than a blanket suppression.
Tests: Host 518 green (10 new).
ROW 80 (observability correctness). An unreplicated site published
LocalDbReplicationConnected=false / LocalDbOplogBacklog=0, contradicting the
nullable semantics every doc and the proto already state ("null means no
reading, NOT disconnected with an empty backlog").
Root cause: the replication engine is registered unconditionally and idles
without a peer, so ISyncStatus answers false/0 on a node that was never meant to
replicate, and LocalDbReplicationStatusReporter passed that straight through.
Two consequences: an operator read connected:false on a healthy site as a fault,
and central's null branches — row 33's conditional KPI emission and
sustained-badge anchor exclusion — never received the null they exist for, so
rig site-b/site-c emitted a permanently misleading
localDbReplicationConnected=0 series.
Fix mirrors the SiteEventLogReplicationCheck precedent: the Host evaluates
SiteLocalDbSetup.ReplicationIsConfigured ONCE and passes it to
AddLocalDbReplicationHealthBridge as a third provider; unconfigured
short-circuits the probe to (null, null). HealthMonitoring never reads LocalDb
config itself, and the same captured value now also feeds
SiteEventLogReplicationCheck so the two surfaces cannot disagree.
ISiteHealthCollector.SetLocalDbReplicationStatus widens to bool? — the
non-nullable bool was the type-level root cause (null was inexpressible).
false now means exactly one thing: configured, peer session down = a real fault.
Wire/central/UI needed no change: the proto already carries BoolValue/Int64Value
wrappers and the aggregator + KPI source already branch on null. This makes code
match doc, so no doc claim was weakened.
Watch the trap that caused this: configured-ness is PeerAddress OR ApiKey (only
the dialling half sets a peer address). The pre-existing SiteLocalDbWiringTests
fixture sets ApiKey and IS a replicating node, so its false/0 assertion is
correct and is kept, with a comment saying so; the new
UnreplicatedSiteLocalDbHealthTests fixture is the genuinely-unreplicated one.
ROW 73 (silent loss of self-heal). AkkaHostedService skips
SiteReconciliationActor when no IDeploymentConfigFetcher is resolvable, and that
skip was a lone boot Warning — a node with no self-heal path at all went on
reporting itself perfectly healthy forever.
Adds tri-state SiteHealthReport.StartupReconcilerActive (null = not reported,
i.e. the central self-report or a pre-row-73 site build; true = running;
false = booted without it), latched once at boot by the Host in both branches.
Additive proto field 29 (BoolValue, vendored C# regenerated via
docker/regen-proto.sh centralcontrol; csproj restored, verified empty diff).
The skip branch also raises its log to Error — no configuration makes the
fetcher legitimately absent on a site node — and writes a best-effort
deployment/Error site event so the evidence outlives the boot log.
Central treatment: a "No self-heal" WARNING badge on the site tile, not a
failure. Config still reaches the node via LocalDb replication while it is up;
what it lost is recovery from a deploy that landed while it was down — degraded,
not offline. Only the explicit false is badged (badging null would light up the
whole fleet the moment central is upgraded ahead of the sites), and offline
sites are skipped, matching the replication-backlog badge.
Fail-before evidence: with the row-80 gate behaviourally reverted,
UnreplicatedSite_HealthReport_ReportsNullForBothLocalDbFields fails
"Expected: null / Actual: False" and 3 of the 8 new reporter unit tests fail.
Tests: HealthMonitoring 128, Host 518, CentralUI 1051, Communication 717,
Commons 715, SiteEventLogging 81 — all green.
An external system's MaxRetries/RetryDelay existed on the central entity but never
reached a site and had no management surface, so every ExternalSystem.CachedCall
buffered with the S&F engine default (DefaultMaxRetries 50 x DefaultRetryInterval
30s ~= 25 min to park) whatever central had configured. The sibling
DatabaseConnectionArtifact had always carried its equivalents.
The consumption end was already wired: ExternalSystemClient.CachedCallAsync already
read system.MaxRetries/RetryDelay and passed them to EnqueueAsync with the 0-as-unset
convention. Only the transport and persistence were missing.
Commons
- ExternalSystemArtifact gains MaxRetries/RetryDelay as optional positional params
(0 / TimeSpan.Zero = unset), matching DatabaseConnectionArtifact rather than going
nullable, so it maps 1:1 onto the entity and the proto3 scalars.
- Create/UpdateExternalSystemCommand gain nullable MaxRetries/RetryDelay, following
the existing TimeoutSeconds "omit to preserve" convention. Both additive.
Central
- ManagementActor create maps both; update preserves-if-null.
- ArtifactDeploymentService projects them onto the artifact.
Wire
- site_command.proto ExternalSystemArtifactDto gains max_retries=7 /
retry_delay=8 (google.protobuf.Duration), mirroring DatabaseConnectionArtifactDto.
Additive: a pre-existing site decodes them as 0/absent, which IS the unset value,
so old behaviour is preserved. Vendored C# regenerated via docker/regen-proto.sh
sitecommand; csproj Protobuf item re-commented, no message types added or removed.
Site
- external_systems gains max_retries / retry_delay_ms via AddColumnIfMissing,
DEFAULT 0 = unset. Storage upsert, repository SELECT/mapping (new ordinals 6/7
appended so existing ones are untouched), apply path, and the legacy migrator's
explicit column list all carry them.
Replicated-table decision: external_systems IS in SiteLocalDbSetup.ReplicatedTables,
so this was checked against the AckTime/native_alarm_state precedent. Real columns
are correct here, not a JSON bag: the table has no general-purpose config column
(auth_configuration and method_definitions are both semantically specific),
timeout_seconds already set the additive-column precedent on this exact table, and
sibling database_connections stores the same two values as plain columns. Verified in
LocalDb source that RegisterReplicated re-reads PRAGMA table_info every boot and its
install script is DROP-then-CREATE, so capture triggers rebuild over the new column
set automatically, and SiteStorageSchema.Apply runs before registration on the same
boot. The only cost is the ReplicatedTable.ComputeDigest change, which the sync
handshake compares fail-closed - a mixed-version pair refuses to sync loudly instead
of diverging silently. Both-nodes-together is already the standing site-pair rule, so
no new operational constraint; recorded in topology-guide.md.
Also fixed: ArtifactDiff.CompareExternalSystem never compared MaxRetries/RetryDelay,
so a bundle import that changed a system's retry budget previewed as "Identical" -
harmless while the values were inert, a live behaviour change now.
No EF migration: the central ExternalSystemDefinitions columns already exist
(InitialSchema). Central UI already had both form fields.
Tests: site storage round-trip + unset-default, site apply path, central artifact
projection, proto round-trip (via the shared sample), management create/preserve,
CLI option parse + alias, and diff reporting. Fail-before verified on the two core
tests (expected 7 got 0 on the read path; expected 9 got 0 on the apply path).
Docs: Component-ExternalSystemGateway, Component-StoreAndForward, Component-CLI,
components/StoreAndForward, components/SiteRuntime, deployment/topology-guide,
CLI README (which was also missing --timeout-seconds entirely).
The row read 'central nodes expose no Prometheus endpoint'. That is not quite
what was wrong, and the difference matters. Central already maps /metrics
(Program.cs central branch) and telemetry is registered for both roles through
the shared BindSharedOptions, so the endpoint rides endpoint routing and has
always been served on central's web port (:5000 on the rig, anonymous — there
is no FallbackPolicy). What central lacked was the site branch's dedicated
MetricsPort, which is the port the established live-scrape recipe probes:
docker exec <node> bash -c 'exec 3<>/dev/tcp/localhost/8084 ...'
So a probe of central on 8084 failed and central-only series (KPI/outbox
counters, the bundle-import session gauge + its new rejection counter,
alarm-cache gauges) read as unscrapeable while they were in fact being
exported on a different port.
Central now binds a dedicated HTTP/1.1 listener on ScadaBridge:Node:MetricsPort,
so one probe recipe covers all eight rig nodes. HTTP/1.1-capable because a
standard Prometheus scraper cannot negotiate h2c, which is why this cannot
share the gRPC port.
Read as a PRESENCE check, not GetValue's defaulted read. NodeOptions defaults
MetricsPort to 8084, so a defaulted read would make every existing central node
start binding a new port on upgrade — and a port already in use is a hard boot
failure, not a degraded one. Central gains the listener only where an operator
asked for it; site nodes are unchanged and still bind theirs unconditionally.
The bind is also skipped when the port would collide with the gRPC port or an
ASPNETCORE_URLS port, since Kestrel throws on a duplicate ListenAnyIP and
/metrics is reachable on those ports anyway.
Config only — no host port mapping added, because the probe runs inside the
container. Applied to both docker/ centrals and both docker-env2/ centrals.
FILE EDITS ONLY: no docker command was run, and deploy.sh was not invoked.
SiteReconciliationActor's node-STARTUP fetch healed the config FILE but never the
RUNTIME. Instance Actors are created from exactly two paths -- the Deployment
Manager's boot-time SQLite load and an explicit deploy/enable command -- and the
reconcile pass runs AFTER that boot load has already read storage. So a config
reconcile restored started no actor: an enabled, successfully-deployed instance
whose config was missing at boot (lost volume, first boot after a restore,
rehomed data dir) stayed dead until the NEXT restart, whose load finally picked
it up. Proven live on site-a 2026-08-15: "Reconcile: fetched + stored config"
with no "InstanceActor started" after it, and a debug snapshot answering
instanceNotFound.
The pass now raises a fire-and-forget EnsureInstanceStartedCommand (new Commons
Deployment message) for every ENABLED instance it healed, routed through the
deployment-manager singleton PROXY so Instance Actors stay under the singleton no
matter which node's reconcile found the gap.
The handler is deliberately advisory:
- idempotent -- no-op when the actor is already running, which both nodes'
reconcile actors legitimately trigger for the same instance;
- enabled-state-aware -- it re-reads the durable row, which is the authority on
both the config body and enabled/disabled, so a disable landing after the
fetch still wins and the instance stays stopped;
- it defers to any in-flight deploy/redeploy/warm for the instance, which keeps
the deploy gate's ordering guarantees and the all-or-nothing-per-instance rule
intact -- this never applies a config, it only starts one already persisted;
- a missing row is a quiet no-op, the normal case on an unreplicated site whose
peer node healed its own file.
The heal is no longer silent: a successful reconcile start emits a Warning log
line and an operational `deployment` site event, because reaching it means the
node booted without a config for an instance that should have been running.
Register row 50. Row 45 (fixed 9e769735) was a live captive dependency —
NotificationOutboxActor cached delivery adapters resolved from an actor-lifetime
IServiceScope, so MaxParallelDeliveries concurrent deliveries shared one EF
DbContext and central-b logged EF's "second operation was started on this context
instance" continuously. Actors take the ROOT IServiceProvider rather than typed
services, which is invisible to build-time analysis, so the bug class had no
mechanical guard at all. This adds both of the register's mitigations; neither
subsumes the other.
1. ValidateScopes + ValidateOnBuild in Development, on BOTH branches, via
ServiceProviderValidation.Configure. WebApplication.CreateBuilder already sets
both flags to IsDevelopment(), so on paper this restates a default — it is
written out because that default is silent (a future move off CreateBuilder
drops it with no diagnostic) and because this host carries three
environment-ish signals (SCADABRIDGE_CONFIG, DOTNET_ENVIRONMENT,
ASPNETCORE_ENVIRONMENT), so "is this node in Development" is not obvious at the
call site. Production stays unvalidated deliberately: a validation failure is
fail-closed, and CI now proves what a production node will not check.
This surfaced two REAL latent violations on the SITE role — not captive
dependencies but unconstructible descriptors, which fail ValidateOnBuild for
the entire graph and are therefore what kept the site branch from being
validatable at all (SiteHealthEndpointTests had pinned UseEnvironment
("Production") to work around exactly this):
- ReconcileService (scoped, needs the central-only IDeploymentManagerRepository)
moved out of the shared AddCommunication() into a new AddCentralCommunication().
- AuditLogKpiSampleSource (scoped IKpiSampleSource, needs the central-only
IAuditLogRepository) moved out of the shared AddAuditLog() into a new
AddAuditLogCentralKpi().
Both were registered on sites under the argument "harmless — only the central
actor ever resolves it". True of the runtime, false of the container: a
registration nothing can build is what stops the validator being usable, and it
masks whatever real defect the validator would have found. Both are now
central-only and SiteHealthEndpointTests boots the site branch validated again.
2. CaptiveDependencyGuard: a reflection scan over every actor class in the shipped
assemblies (28 today; the reference graph is walked from the Host assembly
because referenced assemblies load lazily). Fails on any instance field — auto-
property backing fields included, they are ordinary private fields — typed as
IServiceScope, as a DbContext, or as a service the REAL composition roots
register Scoped. The scoped set is derived from the built Central and Site
containers rather than hand-listed, so a lifetime change automatically re-scopes
the check. IServiceProvider / IServiceScopeFactory fields are explicitly allowed:
that IS the correct per-operation-scope pattern.
No live violation exists — every actor opens a scope per operation. Two fields
are allowlisted with their safety argument rather than exempted silently:
AuditLogIngestActor._injectedRepository and SiteCallAuditActor._injectedRepository
are scoped repository types assigned ONLY by explicitly test-mode constructors,
null under production wiring (AkkaHostedService uses the IServiceProvider
overload in both cases), and SiteCallAuditActor additionally wraps its injected
instance in a SerializedRepository. The shape is one constructor call away from
being row 45 again, so it is recorded, not dismissed.
Tests: 10 new. Six permanent self-tests of the guard, including positive controls
that reproduce the pre-row-45 shape (an actor caching a scoped service, an
IServiceScope, a DbContext, and a scoped auto-property) and a negative control for
the allowed root-provider pattern — a reflection guard that matches nothing passes
just as green as one that works. Plus an actor-count floor with cross-assembly
spot-checks, an allowlist-staleness test that fails any entry no longer matching a
flagged field, and the two per-role scans. Verified non-vacuous by emptying the
allowlist: the Central scan then names both real fields. Two composition-root
build tests force both validation flags on per role regardless of environment.
Doc: REQ-HOST-11 in Component-Host.md records all three rules and their guards.
LocalDb 0.3.0 (scadaproj branch localdb-per-table-snapshot) narrows the
snapshot-resync flag from one per-database bit to one per table, closing the
residual 0.2.1 explicitly deferred. On-disk bookkeeping schema goes to v3
(__localdb_snapshot_state, upgraded in place on open); the wire stays compatible
with 0.1.x/0.2.x, negotiated by capability rather than by a lib_schema_version
bump — which is not available, since the handshake compares that field
fail-closed for equality.
SiteLocalDbSetup needed no restructuring: it already calls RegisterReplicated
once per table, which is exactly what per-table flagging keys off. The comment
now records that the loop shape is load-bearing, and what it buys — adding an
eleventh table to ReplicatedTables on an already-replicating site snapshots that
one table, where through 0.2.x the same edit re-streamed all eleven in full in
both directions. First boot is unchanged (all ten seed at once, so the flagged
set is every registered table and the library sends an ordinary full snapshot),
and upgrading the rig in place is a no-op (the ten are already ledgered, so
nothing seeds and nothing is flagged).
Suites: Host 490/490, SiteRuntime 604/604, StoreAndForward 134/134,
SiteEventLogging 76/76. Solution build clean.
topology-guide + CLAUDE.md LocalDb bullet updated; the umbrella scadaproj
CLAUDE.md travels in that repo's commit.
The daily site_events retention purge (and the storage-cap trim) is CDC-captured
on a replication-enabled site node exactly like any other write — correct by
design, since LocalDb Phase 2 deliberately has no purge-exemption path — so the
backlog jumps by the deleted batch size at purge time. LocalDbOplogBacklog /
localdb_oplog_depth spike, drain, and an operator watching the gauge with no
context reads it as a replication fault.
Documentation + one log line, no behaviour change:
- topology-guide.md gains "Reading the replication backlog — the daily
site_events purge burst": when it fires (PurgeInterval 24h, anchored to the
active node's PROCESS START, not a wall-clock hour, so it moves after every
failover), where it shows (replicated nodes only — not rig site-b/site-c),
the healthy signature (LocalDbReplicationConnected stays true, backlog
returns to ~0) and what a genuine fault looks like instead.
- Component-SiteEventLogging.md Storage records the same under retention/purge;
Component-HealthMonitoring.md gains the two previously-undocumented
LocalDbReplicationConnected / LocalDbOplogBacklog metric rows carrying the
caveat, with cross-references both ways.
- EventLogPurgeService emits one Information line naming the row count and the
expected transient backlog when a purge deleted rows on a replication-enabled
node, so the spike is correlatable in the log. Replication-awareness comes in
as a Host-supplied SiteEventLogReplicationCheck delegate, mirroring the
existing SiteEventLogActiveNodeCheck seam: SiteLocalDbSetup.ReplicationIsConfigured
goes internal so the PeerAddress-OR-ApiKey rule stays in one place and
SiteEventLogging never learns to read LocalDb config. Unregistered ⇒ no note,
matching the default that replication is opt-in and off.
Both delete paths carry the note (a cap trim is usually the larger burst); the
predicate is try/caught since a log-wording check must never break the purge.
Tests: 5 new EventLogPurgeServiceTests cases (replicated logs it, unreplicated
does not, zero-rows does not, cap purge logs it, throwing predicate still purges
and swallows) via a local capturing ILogger. SiteEventLogging 81/81 green,
Host 490/490 green, full solution build clean (0 warnings).
Implements WP3.2 stage (b) per docs/plans/2026-08-15-site-events-policy-design.md.
- Per-run instance-script Started/Completed Info site events are now off by
default (SiteRuntimeOptions.PerRunScriptEvents=false) instead of firing on
every run, closing the dominant site_events writer. Gated at the ScriptRunLauncher
call sites (moved there from ScriptExecutionActor by WP3.1). Error-level events
(timeout/failure/stuck-watchdog/recursion-limit) remain unconditional.
- ScriptRunSummaryRecorder accumulates per-(instance, script) run counters and a
new site-only ScriptRunSummaryFlushService emits one aggregate "script" Info
site event per ScriptRunSummaryIntervalSeconds (default 300s), top-50-script
breakdown with an "others" rollup, zero-activity intervals emit nothing.
- Per-script opt-in via PerRunScriptEventScripts ("Instance/Script" exact or
"Instance/*" wildcard), matched by the new pure ScriptRunEventPolicy. All three
options are read from IOptionsMonitor<SiteRuntimeOptions> per run, so the
policy is hot-togglable without a restart.
- Fixed the stale "event log is not replicated" comment at AkkaHostedService.cs
(~905): site_events IS registered in SiteLocalDbSetup.ReplicatedTables — the
singleton is what makes queries always hit the actively-written copy;
replication is what gives the singleton history to read after a failover
(memo Decision (b)). site_events replication itself is unchanged (still
registered) and already pinned by
tests/ZB.MOM.WW.ScadaBridge.Host.Tests/SiteLocalDbCdcRegistrationTests.cs.
- Updated Component-SiteEventLogging.md (Volume Policy section, corrected
Storage/replication rationale) and Component-SiteRuntime.md (Script Run
Launch + Error Handling sections).
Joseph Doherty