Merge rows 80+73+71: nullable LocalDb health fields on unreplicated sites (Host supplies ReplicationIsConfigured), StartupReconcilerActive on the health report (proto field 29) with central No-self-heal badge, standby active-node ERR downgraded via narrowly-keyed Serilog filter
# Conflicts: # CLAUDE.md
This commit is contained in:
@@ -37,7 +37,8 @@ Site clusters (metric collection and reporting). Central cluster (aggregation an
|
||||
| `SiteAuditBacklog` | Audit Log (site) | Count of `Pending` rows in the site-local `AuditLog` plus oldest-pending-age plus on-disk bytes. Surfaced on the Audit KPI tiles (which border-highlight any non-zero backlog) and sampled into KPI history as `auditBacklogPending`. **No configurable per-site threshold badge exists for this metric** — earlier revisions of this document described one, but none was ever implemented and no such option exists. The only configurable sustained-threshold site badge is the LocalDb replication backlog (register row 33, below). |
|
||||
| `SiteAuditWriteFailures` | Audit Log (site) | Count of failed hot-path audit appends at the site since the last health report. |
|
||||
| `AuditRedactionFailure` | Audit Log (central) | Count of payload redactor errors (over-redacted payloads, safety-net hit) since the last interval. |
|
||||
| `LocalDbReplicationConnected` | Consolidated site LocalDb | Whether a peer sync session is currently running on this node. **Nullable — null means "no reading", not "disconnected".** Reported only by a node with replication configured. |
|
||||
| `LocalDbReplicationConnected` | Consolidated site LocalDb | Whether a peer sync session is currently running on this node. **Nullable — null means "no reading", not "disconnected".** Reported only by a node with replication configured; an unconfigured node reports **null**, never `false` (register row 80 — see *Unconfigured replication reports null* below). |
|
||||
| `StartupReconcilerActive` | Host (site) | Whether this node spawned its `SiteReconciliationActor` at boot. **Tri-state:** null = not reported (the central self-report, or a pre-row-73 site build), `true` = running, `false` = the node booted with no `IDeploymentConfigFetcher` and therefore has **no startup self-heal**. Latched once at boot by the Host — not polled. Drives the *No self-heal* dashboard badge (register row 73). |
|
||||
| `LocalDbOplogBacklog` | Consolidated site LocalDb | Unacked oplog depth (Prometheus `localdb_oplog_depth`). **Nullable — null means "no reading", NOT "connected with an empty backlog"** (a failed poll rendered as 0 would report a pair that cannot read its own oplog as perfectly healthy). **Expected transient spikes:** the daily `site_events` retention purge and the storage-cap trim are CDC-captured like any other write, so the backlog jumps by the batch size at purge time and drains as the peer acks it — see `Component-SiteEventLogging.md` → Storage and `docs/deployment/topology-guide.md` → *Reading the replication backlog* for the healthy-versus-faulty signature. |
|
||||
|
||||
## Reporting Protocol
|
||||
@@ -141,6 +142,57 @@ reporting". Note `false` and `null` are different states for the connected flag
|
||||
that is down samples as `0`; no link at all samples as nothing. Both fold as **Gauge** in the hourly
|
||||
rollup catalog (summing an oplog depth, or a 0/1 flag, per hour would be meaningless).
|
||||
|
||||
## Unconfigured Replication Reports Null
|
||||
|
||||
The two LocalDb fields are produced by `LocalDbReplicationStatusReporter`, a 30 s site-side poll
|
||||
over `ISyncStatus`. That poll is **gated on whether the node is configured to replicate at all**,
|
||||
and an unconfigured node publishes `null` for *both* fields.
|
||||
|
||||
The gate is load-bearing because the replication engine is registered **unconditionally** on every
|
||||
site node and sits inert without a peer — so `ISyncStatus` answers `Connected = false` with a real
|
||||
`0` backlog on a site that was never meant to replicate. Passing that pair through (the behaviour
|
||||
before register row 80) had two consequences:
|
||||
|
||||
- An operator reading a healthy, deliberately-unreplicated site saw `connected: false` and read a
|
||||
fault that did not exist.
|
||||
- Every null branch downstream was dead code on the rig. The conditional KPI emission above and the
|
||||
sustained-badge anchor exclusion below both exist to handle "no reading", but rig site-b/site-c
|
||||
never sent one — they emitted a permanently misleading `localDbReplicationConnected = 0` series
|
||||
instead.
|
||||
|
||||
`false` therefore has exactly one meaning on this field: **replication is configured on this node
|
||||
and the peer session is currently down** — a genuine fault. Beware the specific trap that produced
|
||||
the bug: configured-ness is `PeerAddress` **OR** `ApiKey` (only the dialling half sets a peer
|
||||
address; the passive half carries the key alone), so "no `PeerAddress`" does **not** mean
|
||||
"unreplicated".
|
||||
|
||||
The gate value is supplied by the **Host** from `SiteLocalDbSetup.ReplicationIsConfigured`, exactly
|
||||
as `SiteEventLogReplicationCheck` is supplied to Site Event Logging. Health Monitoring never reads
|
||||
LocalDb configuration itself, and the PeerAddress-OR-ApiKey rule stays in one place.
|
||||
|
||||
## Startup Reconciler Presence
|
||||
|
||||
`SiteReconciliationActor` is the per-node startup self-heal: a standby that was DOWN during a
|
||||
deploy reports its local inventory to central on its next boot, fetches the gap and writes it
|
||||
(and, since register row 67, restarts Instance Actors the boot-time load missed). The Host skips
|
||||
it when no `IDeploymentConfigFetcher` is resolvable.
|
||||
|
||||
That skip used to be a single boot Warning and nothing else, so a node that had silently lost its
|
||||
entire self-heal path went on reporting itself perfectly healthy (register row 73). It is now
|
||||
surfaced three ways:
|
||||
|
||||
- **`StartupReconcilerActive` on the health report**, latched at boot by the Host in both branches.
|
||||
- **A boot-time `Error` log line** (raised from Warning — no configuration makes the fetcher
|
||||
legitimately absent on a site node) plus a **site event** (`deployment` / `Error`), so the
|
||||
evidence survives after the boot log has scrolled away.
|
||||
- **A `No self-heal` warning badge** on the site's Health dashboard tile.
|
||||
|
||||
Central's treatment is deliberately **Warning, not failure**. Config still reaches the node through
|
||||
LocalDb replication while it is up; what it has lost is recovery from a deploy that landed while it
|
||||
was *down*. That is degraded, not offline. Only the explicit `false` is badged — `null` means "not
|
||||
reported", and badging it would light up the whole fleet the moment central is upgraded ahead of
|
||||
the sites. Offline sites are not badged, matching the replication-backlog badge.
|
||||
|
||||
## Prometheus Metrics (site node)
|
||||
|
||||
Distinct from the health *report* above: these are scraped from the site node's `/metrics`
|
||||
@@ -182,6 +234,7 @@ endpoint (`MetricsPort`, default 8084) rather than sent to central.
|
||||
- **Site Runtime (site)**: Provides script error rate and alarm evaluation error rate metrics.
|
||||
- **Store-and-Forward Engine (site)**: Provides buffer depth metrics, including the notification backlog awaiting forward to central.
|
||||
- **Cluster Infrastructure (site)**: Provides node role status.
|
||||
- **Host (site)**: Supplies the two boot-time facts Health Monitoring must not derive for itself — the LocalDb replication-configured gate behind `LocalDbReplicationConnected`/`LocalDbOplogBacklog` (from `SiteLocalDbSetup.ReplicationIsConfigured`), and `StartupReconcilerActive`, latched where the Host decides whether to spawn `SiteReconciliationActor`. Both follow the `SiteEventLogReplicationCheck` precedent: the Host owns the configuration reading, the component owns the reporting.
|
||||
- **Notification Outbox (central)**: Provides central-computed outbox KPIs — queue depth, stuck count, parked count — for the headline dashboard tiles.
|
||||
- **Site Call Audit (central)**: Provides central-computed cached-call KPIs — buffered count, parked count, failed/delivered (last interval), oldest pending age, stuck count — for the headline dashboard tiles.
|
||||
- **Site Event Logging (site)**: Provides `SiteEventLogWriteFailures`. Its daily retention purge and storage-cap trim are also the expected cause of transient `LocalDbOplogBacklog` spikes on a replicated node — see [Component-SiteEventLogging.md](Component-SiteEventLogging.md) → Storage.
|
||||
|
||||
@@ -195,6 +195,60 @@ Akka actors are constructed by the actor system, not by the DI container, so the
|
||||
|
||||
---
|
||||
|
||||
### Boot-time facts the Host supplies to Health Monitoring
|
||||
|
||||
Health Monitoring reports two things the Host alone knows, and in both cases the Host reads the
|
||||
configuration and hands over a captured value rather than teaching the component to read it. This
|
||||
is the `SiteEventLogReplicationCheck` pattern generalised: configuration-reading rules live in one
|
||||
place, and a component that merely *reports* a fact never grows a dependency on where it comes
|
||||
from.
|
||||
|
||||
**LocalDb replication configured-ness** (`SiteServiceRegistration`, register row 80). The
|
||||
replication engine is registered unconditionally and idles without a peer, so `ISyncStatus` answers
|
||||
`Connected = false` with a real `0` backlog on a node that was never meant to replicate. Passing
|
||||
that through made every deliberately-unreplicated site publish "disconnected, empty backlog". The
|
||||
Host now evaluates `SiteLocalDbSetup.ReplicationIsConfigured(config)` **once** and passes it into
|
||||
`AddLocalDbReplicationHealthBridge` as a third provider; an unconfigured node publishes `null` for
|
||||
both LocalDb health fields. The same captured value feeds `SiteEventLogReplicationCheck`, so the
|
||||
two surfaces can never disagree about whether this node replicates. Note the rule is `PeerAddress`
|
||||
**OR** `ApiKey` — only the dialling half sets a peer address.
|
||||
|
||||
**Startup reconciler presence** (`AkkaHostedService`, register row 73). The Host decides whether to
|
||||
spawn `SiteReconciliationActor` — it needs an `IDeploymentConfigFetcher`, and skips the actor when
|
||||
none is resolvable. That skip was previously a lone boot Warning, so a node with no self-heal path
|
||||
at all reported itself perfectly healthy. Both branches now call
|
||||
`ISiteHealthCollector.SetStartupReconcilerActive(...)`, latching the fact onto every subsequent
|
||||
health report as tri-state `StartupReconcilerActive`; the skip branch additionally logs at **Error**
|
||||
(raised from Warning — no configuration makes the fetcher legitimately absent on a site node) and
|
||||
writes a best-effort `deployment`/`Error` site event so the evidence outlives the boot log. Central
|
||||
renders it as a *No self-heal* warning badge. See
|
||||
[Component-HealthMonitoring.md](Component-HealthMonitoring.md) → *Startup Reconciler Presence*.
|
||||
|
||||
### Standby health-check log noise
|
||||
|
||||
`LoggerConfigurationFactory` installs one Serilog exclusion filter, for the by-design standby
|
||||
verdict of the `active-node` probe (register row 71).
|
||||
|
||||
`DefaultHealthCheckService` logs one line per check per probe and picks the level from the verdict —
|
||||
Debug / Warning / **Error** for Healthy / Degraded / Unhealthy. Both active-node checks
|
||||
(`SitePairActiveNodeHealthCheck` on a site, the shared `ActiveNodeHealthCheck` on central) answer
|
||||
Unhealthy on the standby, because 503 is exactly the contract `/health/active` publishes to Traefik
|
||||
and the family dashboard. A perfectly healthy standby therefore logged ERR on every probe — every
|
||||
10 s on a site node, every 5 s on a central node behind Traefik's active-node probe — which floods
|
||||
the log and trains operators to ignore ERR.
|
||||
|
||||
Serilog's `MinimumLevel.Override` can only *raise* a per-source floor and a `LogEvent`'s level is
|
||||
immutable, so neither an override nor an enricher can rewrite the event; excluding it is the only
|
||||
surgical option. The predicate matches on `Level == Error` **and** the health-check logger category
|
||||
**and** `HealthCheckName == "active-node"`. Keying on the check name rather than the category is
|
||||
what preserves the constraint: a failing `localdb`, `akka-cluster`, `database` or
|
||||
`required-singletons` check still logs at Error, and the active-node check has no error path of its
|
||||
own (the active node always answers Healthy), so the only event this can ever suppress is a standby
|
||||
reporting that it is a standby. The verdict remains fully observable through the HTTP status of
|
||||
`/health/active` itself, the readiness probe's cluster view, and the health dashboard.
|
||||
|
||||
---
|
||||
|
||||
## Component Registration Matrix
|
||||
|
||||
| Component | Central | Site | DI (`AddXxx`) | Actors (`AddXxxActors`) | Endpoints (`MapXxx`) |
|
||||
|
||||
Reference in New Issue
Block a user