feat(health): show site event-log/audit write and redaction failures on the Health dashboard (ScadaBridge#88)
New SiteWriteFailureBadges component renders warning badges for SiteEventLogWriteFailures, SiteAuditWriteFailures and AuditRedactionFailure when > 0 on online sites. The CLI 'health summary' already serialises the full latest report, so these counters are visible there unchanged. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
@@ -35,8 +35,8 @@ Site clusters (metric collection and reporting). Central cluster (aggregation an
|
||||
| Notification Outbox stuck count | Notification Outbox (central) | Count of `Pending` / `Retrying` notifications older than the configurable stuck-age threshold — central-computed, not site-reported |
|
||||
| Notification Outbox parked count | Notification Outbox (central) | Count of `Parked` notifications — central-computed, not site-reported |
|
||||
| `SiteAuditBacklog` | Audit Log (site) | Count of `Pending` rows in the site-local `AuditLog` plus oldest-pending-age plus on-disk bytes. Surfaced on the Audit KPI tiles (which border-highlight any non-zero backlog) and sampled into KPI history as `auditBacklogPending`. **No configurable per-site threshold badge exists for this metric** — earlier revisions of this document described one, but none was ever implemented and no such option exists. The only configurable sustained-threshold site badge is the LocalDb replication backlog (register row 33, below). Also carries `RejectedCount` (optional, wire field `rejected_count` = 4): rows central permanently refused (`IngestAck.rejected_event_ids`) that the site moved to the `Rejected` forward state — a subset of the pending count; the Health page shows an "Audit rows rejected" badge when non-zero (Gitea #69). |
|
||||
| `SiteAuditWriteFailures` | Audit Log (site) | Count of failed hot-path audit appends at the site since the last health report. |
|
||||
| `AuditRedactionFailure` | Audit Log (central) | Count of payload redactor errors (over-redacted payloads, safety-net hit) since the last interval. |
|
||||
| `SiteAuditWriteFailures` | Audit Log (site) | Count of failed hot-path audit appends at the site since the last health report. Shown as an *Audit write failures* warning badge on the site card when > 0 (Gitea #88). |
|
||||
| `AuditRedactionFailure` | Audit Log (central) | Count of payload redactor errors (over-redacted payloads, safety-net hit) since the last interval. Shown as an *Audit redaction failures* warning badge on the site card when > 0 (Gitea #88). |
|
||||
| `LocalDbReplicationConnected` | Consolidated site LocalDb | Whether a peer sync session is currently running on this node. **Nullable — null means "no reading", not "disconnected".** Reported only by a node with replication configured; an unconfigured node reports **null**, never `false` (register row 80 — see *Unconfigured replication reports null* below). |
|
||||
| `StartupReconcilerActive` | Host (site) | Whether this node spawned its `SiteReconciliationActor` at boot. **Tri-state:** null = not reported (the central self-report, or a pre-row-73 site build), `true` = running, `false` = the node booted with no `IDeploymentConfigFetcher` and therefore has **no startup self-heal**. Latched once at boot by the Host — not polled. Drives the *No self-heal* dashboard badge (register row 73). |
|
||||
| `SecretMirrorStatuses` | Site Runtime (`DeployedSecretsMirror` via the replicated `secret_mirror_status` table) | Every site node's deployed-secrets mirror status — node name, when the node last wrote the row (on a failure-set change, at start-up, or on the 10-min refresh), and its current failures (`{SecretName, Message}`; names and messages only, never values). Review finding M1: the **standby's** failures ride the **active** node's report, because each node writes its own row to a LocalDb-replicated table and the active node's `SecretMirrorStatusReporter` reads them all. **Nullable — null = no data** (central self-report, a pre-M1 site, reporter not yet run); an empty list = wired, no row yet; a node with an empty `Failures` list is healthy. Point-in-time (not drained on collect). Wire: `SiteHealthReportDto.secret_mirror_statuses` (field **33**, a presence wrapper so null ≠ empty). Drives the *Secret mirror failing* dashboard badge + per-node failure list. On an unreplicated pair only the active node's own row appears. A node that has not refreshed its row for `SiteRuntime:SecretMirrorStatusStaleAfter` (default 1 h — decommissioned, renamed or down) is left off and its row deleted. |
|
||||
|
||||
@@ -0,0 +1,42 @@
|
||||
@using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health
|
||||
|
||||
@* Gitea #88: warning badges for the three site-report failure counters that were collected
|
||||
and shipped to central but never rendered — event-log write failures (cumulative since the
|
||||
logger started), audit write failures (per-interval) and audit redaction failures
|
||||
(per-interval). Each renders only when its counter is > 0, like the "Audit rows rejected"
|
||||
badge (#69). Offline sites render nothing: the Offline badge is the signal that matters. *@
|
||||
@if (Report is not null && IsOnline)
|
||||
{
|
||||
@if (Report.SiteEventLogWriteFailures > 0)
|
||||
{
|
||||
<span class="badge bg-warning text-dark"
|
||||
data-test="event-log-write-failures-badge"
|
||||
title="The site failed to write @Report.SiteEventLogWriteFailures event-log row(s) since its logger started (SQLite error, disk full, or queue overflow). Site events are being lost — check the site node's disk and logs.">
|
||||
Event log write failures
|
||||
</span>
|
||||
}
|
||||
@if (Report.SiteAuditWriteFailures > 0)
|
||||
{
|
||||
<span class="badge bg-warning text-dark"
|
||||
data-test="audit-write-failures-badge"
|
||||
title="The site's primary audit writer failed @Report.SiteAuditWriteFailures time(s) in the last report interval; rows were routed to the drop-oldest fallback ring and may be lost. Check the site's audit SQLite store.">
|
||||
Audit write failures
|
||||
</span>
|
||||
}
|
||||
@if (Report.AuditRedactionFailure > 0)
|
||||
{
|
||||
<span class="badge bg-warning text-dark"
|
||||
data-test="audit-redaction-failures-badge"
|
||||
title="The audit payload redactor errored @Report.AuditRedactionFailure time(s) in the last report interval and fields were replaced with a redaction-error marker. Check the audit redaction regex configuration.">
|
||||
Audit redaction failures
|
||||
</span>
|
||||
}
|
||||
}
|
||||
|
||||
@code {
|
||||
/// <summary>The site's latest health report, or null when none has arrived yet.</summary>
|
||||
[Parameter] public SiteHealthReport? Report { get; set; }
|
||||
|
||||
/// <summary>Whether the site is currently online; offline sites show no warning badges.</summary>
|
||||
[Parameter] public bool IsOnline { get; set; }
|
||||
}
|
||||
@@ -293,6 +293,7 @@
|
||||
Audit rows rejected
|
||||
</span>
|
||||
}
|
||||
<SiteWriteFailureBadges Report="@state.LatestReport" IsOnline="@state.IsOnline" />
|
||||
@if (!state.IsOnline && state.LastStatusChangeAt is { } changedAt)
|
||||
{
|
||||
<small class="text-muted">offline since @changedAt.ToString("u")</small>
|
||||
|
||||
@@ -0,0 +1,69 @@
|
||||
using Bunit;
|
||||
using ZB.MOM.WW.ScadaBridge.CentralUI.Components.Health;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.CentralUI.Tests.Monitoring;
|
||||
|
||||
/// <summary>Gitea #88: failure-counter badges on the Health dashboard site card.</summary>
|
||||
public class SiteWriteFailureBadgesTests : BunitContext
|
||||
{
|
||||
private static SiteHealthReport Report(long eventLog = 0, int audit = 0, int redaction = 0) =>
|
||||
new("site-a", 1, DateTimeOffset.UtcNow,
|
||||
new Dictionary<string, ZB.MOM.WW.ScadaBridge.Commons.Types.Enums.ConnectionHealth>(),
|
||||
new Dictionary<string, ZB.MOM.WW.ScadaBridge.Commons.Messages.Health.TagResolutionStatus>(),
|
||||
0, 0, new Dictionary<string, int>(), 0, 0, 0, 0,
|
||||
SiteAuditWriteFailures: audit,
|
||||
AuditRedactionFailure: redaction,
|
||||
SiteEventLogWriteFailures: eventLog);
|
||||
|
||||
private IRenderedComponent<SiteWriteFailureBadges> Render(SiteHealthReport? report, bool online = true) =>
|
||||
Render<SiteWriteFailureBadges>(p => p.Add(c => c.Report, report).Add(c => c.IsOnline, online));
|
||||
|
||||
/// <summary>All-zero counters render nothing.</summary>
|
||||
[Fact]
|
||||
public void ZeroCounters_RenderNoBadges() =>
|
||||
Assert.Empty(Render(Report()).FindAll(".badge"));
|
||||
|
||||
/// <summary>A missing report renders nothing.</summary>
|
||||
[Fact]
|
||||
public void NullReport_RendersNoBadges() =>
|
||||
Assert.Empty(Render(null).FindAll(".badge"));
|
||||
|
||||
/// <summary>Event-log write failures raise only their own badge.</summary>
|
||||
[Fact]
|
||||
public void EventLogFailures_ShowOwnBadgeOnly()
|
||||
{
|
||||
var cut = Render(Report(eventLog: 3));
|
||||
Assert.Single(cut.FindAll("[data-test='event-log-write-failures-badge']"));
|
||||
Assert.Single(cut.FindAll(".badge"));
|
||||
Assert.Contains("3", cut.Find("[data-test='event-log-write-failures-badge']").GetAttribute("title"));
|
||||
}
|
||||
|
||||
/// <summary>Audit write failures raise only their own badge.</summary>
|
||||
[Fact]
|
||||
public void AuditWriteFailures_ShowOwnBadgeOnly()
|
||||
{
|
||||
var cut = Render(Report(audit: 2));
|
||||
Assert.Single(cut.FindAll("[data-test='audit-write-failures-badge']"));
|
||||
Assert.Single(cut.FindAll(".badge"));
|
||||
}
|
||||
|
||||
/// <summary>Audit redaction failures raise only their own badge.</summary>
|
||||
[Fact]
|
||||
public void RedactionFailures_ShowOwnBadgeOnly()
|
||||
{
|
||||
var cut = Render(Report(redaction: 1));
|
||||
Assert.Single(cut.FindAll("[data-test='audit-redaction-failures-badge']"));
|
||||
Assert.Single(cut.FindAll(".badge"));
|
||||
}
|
||||
|
||||
/// <summary>All three counters render three badges.</summary>
|
||||
[Fact]
|
||||
public void AllCounters_ShowThreeBadges() =>
|
||||
Assert.Equal(3, Render(Report(1, 1, 1)).FindAll(".badge").Count);
|
||||
|
||||
/// <summary>An offline site renders no warning badges.</summary>
|
||||
[Fact]
|
||||
public void OfflineSite_ShowsNoBadges() =>
|
||||
Assert.Empty(Render(Report(1, 1, 1), online: false).FindAll(".badge"));
|
||||
}
|
||||
Reference in New Issue
Block a user