feat(health): show site event-log/audit write and redaction failures on the Health dashboard (ScadaBridge#88)

New SiteWriteFailureBadges component renders warning badges for
SiteEventLogWriteFailures, SiteAuditWriteFailures and AuditRedactionFailure
when > 0 on online sites. The CLI 'health summary' already serialises the full
latest report, so these counters are visible there unchanged.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-09 13:13:57 -05:00
parent c4186509f6
commit 004bf3cf06
4 changed files with 114 additions and 2 deletions
@@ -35,8 +35,8 @@ Site clusters (metric collection and reporting). Central cluster (aggregation an
| Notification Outbox stuck count | Notification Outbox (central) | Count of `Pending` / `Retrying` notifications older than the configurable stuck-age threshold — central-computed, not site-reported |
| Notification Outbox parked count | Notification Outbox (central) | Count of `Parked` notifications — central-computed, not site-reported |
| `SiteAuditBacklog` | Audit Log (site) | Count of `Pending` rows in the site-local `AuditLog` plus oldest-pending-age plus on-disk bytes. Surfaced on the Audit KPI tiles (which border-highlight any non-zero backlog) and sampled into KPI history as `auditBacklogPending`. **No configurable per-site threshold badge exists for this metric** — earlier revisions of this document described one, but none was ever implemented and no such option exists. The only configurable sustained-threshold site badge is the LocalDb replication backlog (register row 33, below). Also carries `RejectedCount` (optional, wire field `rejected_count` = 4): rows central permanently refused (`IngestAck.rejected_event_ids`) that the site moved to the `Rejected` forward state — a subset of the pending count; the Health page shows an "Audit rows rejected" badge when non-zero (Gitea #69). |
| `SiteAuditWriteFailures` | Audit Log (site) | Count of failed hot-path audit appends at the site since the last health report. |
| `AuditRedactionFailure` | Audit Log (central) | Count of payload redactor errors (over-redacted payloads, safety-net hit) since the last interval. |
| `SiteAuditWriteFailures` | Audit Log (site) | Count of failed hot-path audit appends at the site since the last health report. Shown as an *Audit write failures* warning badge on the site card when > 0 (Gitea #88). |
| `AuditRedactionFailure` | Audit Log (central) | Count of payload redactor errors (over-redacted payloads, safety-net hit) since the last interval. Shown as an *Audit redaction failures* warning badge on the site card when > 0 (Gitea #88). |
| `LocalDbReplicationConnected` | Consolidated site LocalDb | Whether a peer sync session is currently running on this node. **Nullable — null means "no reading", not "disconnected".** Reported only by a node with replication configured; an unconfigured node reports **null**, never `false` (register row 80 — see *Unconfigured replication reports null* below). |
| `StartupReconcilerActive` | Host (site) | Whether this node spawned its `SiteReconciliationActor` at boot. **Tri-state:** null = not reported (the central self-report, or a pre-row-73 site build), `true` = running, `false` = the node booted with no `IDeploymentConfigFetcher` and therefore has **no startup self-heal**. Latched once at boot by the Host — not polled. Drives the *No self-heal* dashboard badge (register row 73). |
| `SecretMirrorStatuses` | Site Runtime (`DeployedSecretsMirror` via the replicated `secret_mirror_status` table) | Every site node's deployed-secrets mirror status — node name, when the node last wrote the row (on a failure-set change, at start-up, or on the 10-min refresh), and its current failures (`{SecretName, Message}`; names and messages only, never values). Review finding M1: the **standby's** failures ride the **active** node's report, because each node writes its own row to a LocalDb-replicated table and the active node's `SecretMirrorStatusReporter` reads them all. **Nullable — null = no data** (central self-report, a pre-M1 site, reporter not yet run); an empty list = wired, no row yet; a node with an empty `Failures` list is healthy. Point-in-time (not drained on collect). Wire: `SiteHealthReportDto.secret_mirror_statuses` (field **33**, a presence wrapper so null ≠ empty). Drives the *Secret mirror failing* dashboard badge + per-node failure list. On an unreplicated pair only the active node's own row appears. A node that has not refreshed its row for `SiteRuntime:SecretMirrorStatusStaleAfter` (default 1 h — decommissioned, renamed or down) is left off and its row deleted. |