fix(logging): row 71 — stop logging the standby's by-design active-node verdict at ERROR

A healthy standby logged "Health check active-node with status Unhealthy …
'not the site pair primary (standby)'" at Error on every probe — every 10s on a
site node, and (discovered while fixing this) every 5s on a central standby,
which Traefik health-probes at that cadence. Correct-by-design state at an
alarming severity: it floods the log and trains operators to ignore ERR.

Mechanism: DefaultHealthCheckService logs one line per check per probe and picks
the level from the verdict — Debug/Warning/Error for Healthy/Degraded/Unhealthy.
Both active-node checks answer Unhealthy on the standby because 503 IS the
contract /health/active publishes. There is no IHealthCheckPublisher in this
repo, so each line is one inbound HTTP probe.

Fix: one Serilog exclusion filter in LoggerConfigurationFactory. A level
override could not work — Serilog's MinimumLevel.Override only RAISES a
per-source floor, and a LogEvent's Level is immutable, so neither an override
nor an enricher can rewrite Error as Information. Changing the check to report
Degraded was rejected: it would alter the HTTP status that /health/active
publishes, which is load-bearing for Traefik and the family dashboard.

The constraint (genuine failures must still log at Error) is preserved by keying
the predicate on the check NAME rather than the logger category: localdb,
akka-cluster, database and required-singletons keep their Error lines, and the
active-node check has no error path of its own — the active node always answers
Healthy — so the only event this can ever suppress is a standby reporting that
it is a standby. The verdict stays fully observable via /health/active's own
HTTP status, the readiness probe's cluster view, and the health dashboard.

Also corrects a stale CLAUDE.md claim found while verifying what consumes this
check: "/health/active is central-only (site nodes map no /health/* at all)" is
wrong — SiteServiceRegistration registers akka-cluster + localdb + the
role-scoped SitePairActiveNodeHealthCheck, and Program.cs's Site branch calls
MapZbHealth() on the HTTP/1.1 listener.

Fail-before evidence: with the filter behaviourally disabled,
StandbyActiveNodeVerdict_IsNotLogged fails "Collection was not empty" while the
other 9 tests in the file — the must-still-log constraints — pass either way,
which is what proves the filter is narrow rather than a blanket suppression.

Tests: Host 518 green (10 new).
This commit is contained in:
Joseph Doherty
2026-08-15 21:41:01 -04:00
parent a2f0086fa1
commit 8b761841bc
4 changed files with 287 additions and 2 deletions
+54
View File
@@ -195,6 +195,60 @@ Akka actors are constructed by the actor system, not by the DI container, so the
---
### Boot-time facts the Host supplies to Health Monitoring
Health Monitoring reports two things the Host alone knows, and in both cases the Host reads the
configuration and hands over a captured value rather than teaching the component to read it. This
is the `SiteEventLogReplicationCheck` pattern generalised: configuration-reading rules live in one
place, and a component that merely *reports* a fact never grows a dependency on where it comes
from.
**LocalDb replication configured-ness** (`SiteServiceRegistration`, register row 80). The
replication engine is registered unconditionally and idles without a peer, so `ISyncStatus` answers
`Connected = false` with a real `0` backlog on a node that was never meant to replicate. Passing
that through made every deliberately-unreplicated site publish "disconnected, empty backlog". The
Host now evaluates `SiteLocalDbSetup.ReplicationIsConfigured(config)` **once** and passes it into
`AddLocalDbReplicationHealthBridge` as a third provider; an unconfigured node publishes `null` for
both LocalDb health fields. The same captured value feeds `SiteEventLogReplicationCheck`, so the
two surfaces can never disagree about whether this node replicates. Note the rule is `PeerAddress`
**OR** `ApiKey` — only the dialling half sets a peer address.
**Startup reconciler presence** (`AkkaHostedService`, register row 73). The Host decides whether to
spawn `SiteReconciliationActor` — it needs an `IDeploymentConfigFetcher`, and skips the actor when
none is resolvable. That skip was previously a lone boot Warning, so a node with no self-heal path
at all reported itself perfectly healthy. Both branches now call
`ISiteHealthCollector.SetStartupReconcilerActive(...)`, latching the fact onto every subsequent
health report as tri-state `StartupReconcilerActive`; the skip branch additionally logs at **Error**
(raised from Warning — no configuration makes the fetcher legitimately absent on a site node) and
writes a best-effort `deployment`/`Error` site event so the evidence outlives the boot log. Central
renders it as a *No self-heal* warning badge. See
[Component-HealthMonitoring.md](Component-HealthMonitoring.md) → *Startup Reconciler Presence*.
### Standby health-check log noise
`LoggerConfigurationFactory` installs one Serilog exclusion filter, for the by-design standby
verdict of the `active-node` probe (register row 71).
`DefaultHealthCheckService` logs one line per check per probe and picks the level from the verdict —
Debug / Warning / **Error** for Healthy / Degraded / Unhealthy. Both active-node checks
(`SitePairActiveNodeHealthCheck` on a site, the shared `ActiveNodeHealthCheck` on central) answer
Unhealthy on the standby, because 503 is exactly the contract `/health/active` publishes to Traefik
and the family dashboard. A perfectly healthy standby therefore logged ERR on every probe — every
10 s on a site node, every 5 s on a central node behind Traefik's active-node probe — which floods
the log and trains operators to ignore ERR.
Serilog's `MinimumLevel.Override` can only *raise* a per-source floor and a `LogEvent`'s level is
immutable, so neither an override nor an enricher can rewrite the event; excluding it is the only
surgical option. The predicate matches on `Level == Error` **and** the health-check logger category
**and** `HealthCheckName == "active-node"`. Keying on the check name rather than the category is
what preserves the constraint: a failing `localdb`, `akka-cluster`, `database` or
`required-singletons` check still logs at Error, and the active-node check has no error path of its
own (the active node always answers Healthy), so the only event this can ever suppress is a standby
reporting that it is a standby. The verdict remains fully observable through the HTTP status of
`/health/active` itself, the readiness probe's cluster view, and the health dashboard.
---
## Component Registration Matrix
| Component | Central | Site | DI (`AddXxx`) | Actors (`AddXxxActors`) | Endpoints (`MapXxx`) |