A healthy standby logged "Health check active-node with status Unhealthy …
'not the site pair primary (standby)'" at Error on every probe — every 10s on a
site node, and (discovered while fixing this) every 5s on a central standby,
which Traefik health-probes at that cadence. Correct-by-design state at an
alarming severity: it floods the log and trains operators to ignore ERR.
Mechanism: DefaultHealthCheckService logs one line per check per probe and picks
the level from the verdict — Debug/Warning/Error for Healthy/Degraded/Unhealthy.
Both active-node checks answer Unhealthy on the standby because 503 IS the
contract /health/active publishes. There is no IHealthCheckPublisher in this
repo, so each line is one inbound HTTP probe.
Fix: one Serilog exclusion filter in LoggerConfigurationFactory. A level
override could not work — Serilog's MinimumLevel.Override only RAISES a
per-source floor, and a LogEvent's Level is immutable, so neither an override
nor an enricher can rewrite Error as Information. Changing the check to report
Degraded was rejected: it would alter the HTTP status that /health/active
publishes, which is load-bearing for Traefik and the family dashboard.
The constraint (genuine failures must still log at Error) is preserved by keying
the predicate on the check NAME rather than the logger category: localdb,
akka-cluster, database and required-singletons keep their Error lines, and the
active-node check has no error path of its own — the active node always answers
Healthy — so the only event this can ever suppress is a standby reporting that
it is a standby. The verdict stays fully observable via /health/active's own
HTTP status, the readiness probe's cluster view, and the health dashboard.
Also corrects a stale CLAUDE.md claim found while verifying what consumes this
check: "/health/active is central-only (site nodes map no /health/* at all)" is
wrong — SiteServiceRegistration registers akka-cluster + localdb + the
role-scoped SitePairActiveNodeHealthCheck, and Program.cs's Site branch calls
MapZbHealth() on the HTTP/1.1 listener.
Fail-before evidence: with the filter behaviourally disabled,
StandbyActiveNodeVerdict_IsNotLogged fails "Collection was not empty" while the
other 9 tests in the file — the must-still-log constraints — pass either way,
which is what proves the filter is narrow rather than a blanket suppression.
Tests: Host 518 green (10 new).
Register row 50. Row 45 (fixed 9e769735) was a live captive dependency —
NotificationOutboxActor cached delivery adapters resolved from an actor-lifetime
IServiceScope, so MaxParallelDeliveries concurrent deliveries shared one EF
DbContext and central-b logged EF's "second operation was started on this context
instance" continuously. Actors take the ROOT IServiceProvider rather than typed
services, which is invisible to build-time analysis, so the bug class had no
mechanical guard at all. This adds both of the register's mitigations; neither
subsumes the other.
1. ValidateScopes + ValidateOnBuild in Development, on BOTH branches, via
ServiceProviderValidation.Configure. WebApplication.CreateBuilder already sets
both flags to IsDevelopment(), so on paper this restates a default — it is
written out because that default is silent (a future move off CreateBuilder
drops it with no diagnostic) and because this host carries three
environment-ish signals (SCADABRIDGE_CONFIG, DOTNET_ENVIRONMENT,
ASPNETCORE_ENVIRONMENT), so "is this node in Development" is not obvious at the
call site. Production stays unvalidated deliberately: a validation failure is
fail-closed, and CI now proves what a production node will not check.
This surfaced two REAL latent violations on the SITE role — not captive
dependencies but unconstructible descriptors, which fail ValidateOnBuild for
the entire graph and are therefore what kept the site branch from being
validatable at all (SiteHealthEndpointTests had pinned UseEnvironment
("Production") to work around exactly this):
- ReconcileService (scoped, needs the central-only IDeploymentManagerRepository)
moved out of the shared AddCommunication() into a new AddCentralCommunication().
- AuditLogKpiSampleSource (scoped IKpiSampleSource, needs the central-only
IAuditLogRepository) moved out of the shared AddAuditLog() into a new
AddAuditLogCentralKpi().
Both were registered on sites under the argument "harmless — only the central
actor ever resolves it". True of the runtime, false of the container: a
registration nothing can build is what stops the validator being usable, and it
masks whatever real defect the validator would have found. Both are now
central-only and SiteHealthEndpointTests boots the site branch validated again.
2. CaptiveDependencyGuard: a reflection scan over every actor class in the shipped
assemblies (28 today; the reference graph is walked from the Host assembly
because referenced assemblies load lazily). Fails on any instance field — auto-
property backing fields included, they are ordinary private fields — typed as
IServiceScope, as a DbContext, or as a service the REAL composition roots
register Scoped. The scoped set is derived from the built Central and Site
containers rather than hand-listed, so a lifetime change automatically re-scopes
the check. IServiceProvider / IServiceScopeFactory fields are explicitly allowed:
that IS the correct per-operation-scope pattern.
No live violation exists — every actor opens a scope per operation. Two fields
are allowlisted with their safety argument rather than exempted silently:
AuditLogIngestActor._injectedRepository and SiteCallAuditActor._injectedRepository
are scoped repository types assigned ONLY by explicitly test-mode constructors,
null under production wiring (AkkaHostedService uses the IServiceProvider
overload in both cases), and SiteCallAuditActor additionally wraps its injected
instance in a SerializedRepository. The shape is one constructor call away from
being row 45 again, so it is recorded, not dismissed.
Tests: 10 new. Six permanent self-tests of the guard, including positive controls
that reproduce the pre-row-45 shape (an actor caching a scoped service, an
IServiceScope, a DbContext, and a scoped auto-property) and a negative control for
the allowed root-provider pattern — a reflection guard that matches nothing passes
just as green as one that works. Plus an actor-count floor with cross-assembly
spot-checks, an allowlist-staleness test that fails any entry no longer matching a
flagged field, and the two per-role scans. Verified non-vacuous by emptying the
allowlist: the Central scan then names both real fields. Two composition-root
build tests force both validation flags on per role regardless of environment.
Doc: REQ-HOST-11 in Component-Host.md records all three rules and their guards.
ClusterClient→gRPC migration Phase 4 (docs/plans/2026-07-22-clusterclient-to-grpc-plan.md).
Phases 2/3 proved both directions on gRPC; this removes the Akka transport underneath.
Deleted:
- AkkaCentralTransport, AkkaSiteTransport (+ their dedicated tests)
- ISiteClientFactory + DefaultSiteClientFactory; CentralCommunicationActor legacy
ctor + SelectTransport (Host now builds GrpcSiteTransport and injects it)
- ClusterClient creation + both ClusterClientReceptionist.RegisterService calls in
AkkaHostedService; the RegisterCentralClient message + receive block
- CommunicationOptions.CentralContactPoints; the CentralTransport/SiteTransport
coexistence flags; the CentralTransportMode/SiteTransportKind enums
gRPC is now the only site↔central transport (site→central CentralControlService via
GrpcCentralTransport; central→site SiteCommandService via GrpcSiteTransport), both
built unconditionally by the Host. NoOpCentralTransport is the fail-loud null-default
so TestKit command-dispatch suites still construct the site actor without a wired
transport; production always injects GrpcCentralTransport.
Config: CentralGrpcEndpoints is now unconditional — CommunicationOptionsValidator
rejects blank entries (role-agnostic), and StartupValidator requires a Site node to
list >=1 endpoint (fail-fast, mirrors GrpcPsk). Rig configs moved
CentralContactPoints -> CentralGrpcEndpoints (docker x6, docker-env2 x2, Host default,
deploy/wonder-app-vd03). Kept Akka.Cluster.Tools (ClusterSingleton still used).
Tests: build 0/0; Communication.Tests 640, Host.Tests 421 green. Removed the
ClusterClient.Send per-site-routing tests (covered by the transport suites), swapped
the ISiteClientFactory-based ctors to a substitute ISiteCommandTransport, converted
the audit-push integration relay to an in-process bridge transport.
Docs: Component-Communication/Host/StoreAndForward, components/Communication,
topology-guide, grpc_streams (SUPERSEDED note), the frame-size known-issue (retired
amendment), and CLAUDE.md transport decisions.
Not included: the dead IntegrationCallRequest path (#32) is a separate user-owned
behavioral decision — SiteEnvelope routing is transport-agnostic so it still compiles.
Phase 0 of the ClusterClient→gRPC migration
(docs/plans/2026-07-22-clusterclient-to-grpc-plan.md). Standalone hardening: it
closes a gap that exists today and is a precondition for moving command/control
onto gRPC in later phases.
T0.1 — delete the ManagementActor ClusterClientReceptionist registration.
It was built for an out-of-cluster CLI that was never written: the shipped CLI
speaks HTTP Basic to /management, which asks the actor in-process through
ManagementActorHolder. Nothing in the repo ever sent to /user/management. The
actor still runs there; only the cross-boundary advertisement is gone. Six
documents claimed the CLI used ClusterClient — including the CLI's own README
"Architecture Notes" — and are corrected here rather than left to rot.
T0.2 — record, do not port, the dead integration-routing path.
IntegrationCallRequest is unwired at BOTH ends: RouteIntegrationCallAsync has
zero callers anywhere, and RegisterLocalHandler(Integration, …) appears only in
a test, so production always answers "Integration handler not available". It is
excluded from the gRPC contract (28 of 29 commands migrate) rather than
enshrined on an additive-only wire format, and deleting it during a
transport migration would mix a behavioural change into a change whose whole
value is that behaviour is identical. See
docs/known-issues/2026-07-22-integration-call-routing-is-dead-code.md.
T0.3 — preshared-key authentication on SiteStreamService.
The service shipped with no auth at all: plaintext h2c, no interceptor, so
anything that could reach a site node's :8083 could open a live data stream or
read audit rows back via PullAuditEvents/PullSiteCalls. ControlPlaneAuthInterceptor
now gates /sitestream.SiteStreamService/ — modeled on LocalDbSyncAuthInterceptor
(constant-time compare, fail-closed, PermissionDenied) but gating a SET of
service prefixes so phases 1A/1B add services rather than interceptors. LocalDb
sync keeps its own separate key: it authenticates the pair partner, not central,
and collapsing the two would make a site's central-facing key also admit writes
into its database.
Keys are per site (SB-GRPC-PSK-<siteId>), never fleet-wide, so a compromised
site yields only its own. Central attaches them through ControlPlaneCredentials,
which binds CallCredentials to the channel — covering unary and streaming
uniformly, and letting the key resolve asynchronously, which a client
interceptor could not do without blocking. All three central→site channel
creation sites go through it (SiteStreamGrpcClient and both audit pull invokers);
the pull invokers' channel caches are re-keyed by (site, endpoint) because
credentials are per-site and bound to the channel.
Two decisions beyond the plan:
* StartupValidator now requires GrpcPsk on Site nodes. The plan specified only
the runtime gate, but fail-closed with no boot check produces a node that
joins, answers heartbeats and reports healthy while refusing every stream,
audit pull and telemetry ingest — silent and total. Same reasoning as the
existing inbound API-key pepper rule.
* Added Communication:SitePsks as a central-side key map. The plan assumed
central would read the store, seeded via a dev KEK; the docker rig
deliberately boots with no master key, so store-only resolution would leave
it unable to dial its own sites. The store stays primary — it is the only
source that can serve a site added at runtime — with the map covering
key-less hosts and one-off pins. Neither source falling back to
"unauthenticated" is the invariant.
T0.4 — dev keys on both rigs and tests.
34 tests. The seven that matter most exercise a real in-process gRPC stack over
TestServer: the unit tests on either side of the wire would both stay green if
the halves disagreed, and gRPC refuses call credentials on a plaintext channel
by default — the UnsafeUseInsecureChannelCallCredentials opt-in is only provable
by making a real call. They confirm correct key passes on unary AND streaming,
wrong key and no-credentials both get PermissionDenied, and an unresolvable key
fails the call with nothing reaching the service.
OPERATIONAL: a site node upgraded to this build without a key will not boot.
That includes the gitignored deploy/wonder-app-vd03/ overlay.
Remediation from the full per-module code review at 4307c381 (findings recorded
separately in code-reviews/).
Highs fixed:
- DeploymentManager-025/SiteRuntime-031: stop broadcasting notification lists + SMTP
configs (incl. credentials) to sites; site purges already-persisted rows on apply
(enforces the central-only delivery design; clears plaintext SMTP creds at rest).
- DataConnectionLayer-023: guard the native-alarm subscribe path against the
mid-flight-unsubscribe adapter-feed leak (mirrors the DCL-021 tag-path fix).
- SiteEventLogging-024: normalize From/To query bounds to UTC (the -016 fix the
audit trail claimed but never committed).
- KpiHistory-001: add an in-flight guard to the recorder sample tick.
- ScriptAnalysis-001: harden the trust analyzer's TPA-absent fallback (resolve
forbidden anchors in the minimal reference set; warn on degraded mode) — anchors
added to validation references only, never the compile gate.
(InboundAPI-026 left to the feat/ipsen-movein effort per owner decision.)
Medium/Low: DM-026 deterministic deploy-status tiebreaker; SR-027/028/029/030
native-alarm leak/phantom-active/delete-during-redeploy fixes; AL-013/014/016;
TE-024 (folder-mutation audit rows now persisted)/025; SF-025 gauge-provider
clear-on-stop; ESG-025/026; SEC-023/024/025; SCA-007/008/009; plus doc/test
accuracy COM-023/024, HOST-025/026, HM-024/025, NS-027/028.
Full-solution build 0 warnings; ~3560 tests across 18 touched suites green.
REQ-HOST-4a lists "required cluster singletons running (if applicable)" as a
readiness criterion, but /health/ready only checked database + akka-cluster.
Add a third Ready-tagged check, RequiredSingletonsHealthCheck, registered in the
Central-role AddHealthChecks() chain (so it is naturally role-scoped — site nodes
never run it).
Probe: for each required central singleton, Ask its local ClusterSingletonProxy
an Identify with a short bounded per-singleton timeout (~2s, probes run
concurrently via Task.WhenAll). A non-null ActorIdentity.Subject within the
timeout means the singleton is running and reachable through the proxy; a null
subject or a timeout means unreachable → Unhealthy, naming the unreachable
singleton(s). The check never throws (catch-all → Unhealthy) and resolves
ActorSystem lazily from DI per probe (Unhealthy if Akka not yet up).
Required-always set = the five singleton proxies created unconditionally in
AkkaHostedService.RegisterCentralActors: notification-outbox, audit-log-ingest,
site-call-audit, audit-log-purge, site-audit-reconciliation. There are no
feature/config-gated central singletons today; any future gated singleton is the
"if applicable" case and must NOT be added to the required set.
Leadership-agnostic: the proxy reaches the singleton from either central node, so
a ready standby still reports ready (readiness must not require cluster
leadership — that is the Active tier's job). During a brief singleton handover the
probe may time out and the node flaps to not-ready, which is correct (a node
mid-handover is legitimately not fully ready); no retries, to keep the probe fast.
Tests (TDD): RequiredSingletonsHealthCheckTests exercises the probe against a
TestKit ActorSystem — all proxies present+reachable → Healthy; one missing →
Unhealthy naming it; ActorSystem absent → Unhealthy, no throw. HealthCheckTests
regression-guards the Ready tag + absence of the Active tag on the new check.
Switch site host to WebApplicationBuilder with Kestrel HTTP/2 gRPC server,
add GrpcPort/keepalive config, wire SiteStreamManager as ISiteStreamSubscriber,
expose gRPC ports in docker-compose, add site seed script, update all 10
requirement docs + CLAUDE.md + README.md for the new dual-transport architecture.
Organize documentation by moving requirements (HighLevelReqs, Component-*,
lmxproxy_protocol) to docs/requirements/ and test infrastructure docs to
docs/test_infra/. Updates all cross-references in README, CLAUDE.md,
infra/README, component docs, and 23 plan files.
Joseph Doherty