fix(logging): row 71 — stop logging the standby's by-design active-node verdict at ERROR
A healthy standby logged "Health check active-node with status Unhealthy … 'not the site pair primary (standby)'" at Error on every probe — every 10s on a site node, and (discovered while fixing this) every 5s on a central standby, which Traefik health-probes at that cadence. Correct-by-design state at an alarming severity: it floods the log and trains operators to ignore ERR. Mechanism: DefaultHealthCheckService logs one line per check per probe and picks the level from the verdict — Debug/Warning/Error for Healthy/Degraded/Unhealthy. Both active-node checks answer Unhealthy on the standby because 503 IS the contract /health/active publishes. There is no IHealthCheckPublisher in this repo, so each line is one inbound HTTP probe. Fix: one Serilog exclusion filter in LoggerConfigurationFactory. A level override could not work — Serilog's MinimumLevel.Override only RAISES a per-source floor, and a LogEvent's Level is immutable, so neither an override nor an enricher can rewrite Error as Information. Changing the check to report Degraded was rejected: it would alter the HTTP status that /health/active publishes, which is load-bearing for Traefik and the family dashboard. The constraint (genuine failures must still log at Error) is preserved by keying the predicate on the check NAME rather than the logger category: localdb, akka-cluster, database and required-singletons keep their Error lines, and the active-node check has no error path of its own — the active node always answers Healthy — so the only event this can ever suppress is a standby reporting that it is a standby. The verdict stays fully observable via /health/active's own HTTP status, the readiness probe's cluster view, and the health dashboard. Also corrects a stale CLAUDE.md claim found while verifying what consumes this check: "/health/active is central-only (site nodes map no /health/* at all)" is wrong — SiteServiceRegistration registers akka-cluster + localdb + the role-scoped SitePairActiveNodeHealthCheck, and Program.cs's Site branch calls MapZbHealth() on the HTTP/1.1 listener. Fail-before evidence: with the filter behaviourally disabled, StandbyActiveNodeVerdict_IsNotLogged fails "Collection was not empty" while the other 9 tests in the file — the must-still-log constraints — pass either way, which is what proves the filter is narrow rather than a blanket suppression. Tests: Host 518 green (10 new).
This commit is contained in:
@@ -195,6 +195,60 @@ Akka actors are constructed by the actor system, not by the DI container, so the
|
||||
|
||||
---
|
||||
|
||||
### Boot-time facts the Host supplies to Health Monitoring
|
||||
|
||||
Health Monitoring reports two things the Host alone knows, and in both cases the Host reads the
|
||||
configuration and hands over a captured value rather than teaching the component to read it. This
|
||||
is the `SiteEventLogReplicationCheck` pattern generalised: configuration-reading rules live in one
|
||||
place, and a component that merely *reports* a fact never grows a dependency on where it comes
|
||||
from.
|
||||
|
||||
**LocalDb replication configured-ness** (`SiteServiceRegistration`, register row 80). The
|
||||
replication engine is registered unconditionally and idles without a peer, so `ISyncStatus` answers
|
||||
`Connected = false` with a real `0` backlog on a node that was never meant to replicate. Passing
|
||||
that through made every deliberately-unreplicated site publish "disconnected, empty backlog". The
|
||||
Host now evaluates `SiteLocalDbSetup.ReplicationIsConfigured(config)` **once** and passes it into
|
||||
`AddLocalDbReplicationHealthBridge` as a third provider; an unconfigured node publishes `null` for
|
||||
both LocalDb health fields. The same captured value feeds `SiteEventLogReplicationCheck`, so the
|
||||
two surfaces can never disagree about whether this node replicates. Note the rule is `PeerAddress`
|
||||
**OR** `ApiKey` — only the dialling half sets a peer address.
|
||||
|
||||
**Startup reconciler presence** (`AkkaHostedService`, register row 73). The Host decides whether to
|
||||
spawn `SiteReconciliationActor` — it needs an `IDeploymentConfigFetcher`, and skips the actor when
|
||||
none is resolvable. That skip was previously a lone boot Warning, so a node with no self-heal path
|
||||
at all reported itself perfectly healthy. Both branches now call
|
||||
`ISiteHealthCollector.SetStartupReconcilerActive(...)`, latching the fact onto every subsequent
|
||||
health report as tri-state `StartupReconcilerActive`; the skip branch additionally logs at **Error**
|
||||
(raised from Warning — no configuration makes the fetcher legitimately absent on a site node) and
|
||||
writes a best-effort `deployment`/`Error` site event so the evidence outlives the boot log. Central
|
||||
renders it as a *No self-heal* warning badge. See
|
||||
[Component-HealthMonitoring.md](Component-HealthMonitoring.md) → *Startup Reconciler Presence*.
|
||||
|
||||
### Standby health-check log noise
|
||||
|
||||
`LoggerConfigurationFactory` installs one Serilog exclusion filter, for the by-design standby
|
||||
verdict of the `active-node` probe (register row 71).
|
||||
|
||||
`DefaultHealthCheckService` logs one line per check per probe and picks the level from the verdict —
|
||||
Debug / Warning / **Error** for Healthy / Degraded / Unhealthy. Both active-node checks
|
||||
(`SitePairActiveNodeHealthCheck` on a site, the shared `ActiveNodeHealthCheck` on central) answer
|
||||
Unhealthy on the standby, because 503 is exactly the contract `/health/active` publishes to Traefik
|
||||
and the family dashboard. A perfectly healthy standby therefore logged ERR on every probe — every
|
||||
10 s on a site node, every 5 s on a central node behind Traefik's active-node probe — which floods
|
||||
the log and trains operators to ignore ERR.
|
||||
|
||||
Serilog's `MinimumLevel.Override` can only *raise* a per-source floor and a `LogEvent`'s level is
|
||||
immutable, so neither an override nor an enricher can rewrite the event; excluding it is the only
|
||||
surgical option. The predicate matches on `Level == Error` **and** the health-check logger category
|
||||
**and** `HealthCheckName == "active-node"`. Keying on the check name rather than the category is
|
||||
what preserves the constraint: a failing `localdb`, `akka-cluster`, `database` or
|
||||
`required-singletons` check still logs at Error, and the active-node check has no error path of its
|
||||
own (the active node always answers Healthy), so the only event this can ever suppress is a standby
|
||||
reporting that it is a standby. The verdict remains fully observable through the HTTP status of
|
||||
`/health/active` itself, the readiness probe's cluster view, and the health dashboard.
|
||||
|
||||
---
|
||||
|
||||
## Component Registration Matrix
|
||||
|
||||
| Component | Central | Site | DI (`AddXxx`) | Actors (`AddXxxActors`) | Endpoints (`MapXxx`) |
|
||||
|
||||
Reference in New Issue
Block a user