fix(site-secrets): report every site node's secret mirror failures on the health report (M1)
A DeployedSecretsMirror failure on the STANDBY node (e.g. a master-key mismatch
that stops a deployed secret decrypting there) was only logged on that node:
site health reports are sent by the active node only, so central never saw it.
- New 12th LocalDb-replicated table secret_mirror_status (node_name PK,
failures_json of {SecretName, Message}, checked_at; no FKs). Each node's
mirror writes only its own row, and only when its failure set changes plus
once per start-up (every write replicates via CDC). Best-effort: a failed
write is a Warning and is retried next pass. Node key = INodeIdentityProvider
NodeName, falling back to the machine name.
- SecretMirrorStatusReporter (SiteRuntime, every node, 30 s) reads every row
into ISiteHealthCollector.SetSecretMirrorStatuses -> additive nullable
SiteHealthReport.SecretMirrorStatuses (null = no data).
- Wire: SiteHealthReportDto.secret_mirror_statuses = 33 (presence wrapper so
null != empty); vendored CentralControl.cs regenerated; mapper both ways.
- Central UI Health card: 'Secret mirror failing' badge + compact per-node
node / secret / message list.
- Tests: mirror status write-on-change/first-pass/failure isolation, storage
round-trip, reporter/collector, DTO mapper round-trip incl. null/empty,
bUnit, replicated-table set (twelve) + FK guard wording.
This commit is contained in:
@@ -275,6 +275,14 @@
|
||||
No self-heal
|
||||
</span>
|
||||
}
|
||||
@if (HasSecretMirrorFailures(state))
|
||||
{
|
||||
<span class="badge bg-warning text-dark"
|
||||
data-test="secret-mirror-badge"
|
||||
title="A site node could not apply and/or decrypt one or more deployed secrets. A standby node's failures surface here before a failover would expose them — see the details in the card body.">
|
||||
Secret mirror failing
|
||||
</span>
|
||||
}
|
||||
@if (!state.IsOnline && state.LastStatusChangeAt is { } changedAt)
|
||||
{
|
||||
<small class="text-muted">offline since @changedAt.ToString("u")</small>
|
||||
@@ -302,6 +310,26 @@
|
||||
@if (state.LatestReport != null)
|
||||
{
|
||||
var report = state.LatestReport;
|
||||
@if (HasSecretMirrorFailures(state))
|
||||
{
|
||||
@* Review finding M1: every site node's deployed-secrets mirror failures,
|
||||
the standby's included (carried on the active node's report). *@
|
||||
<div class="alert alert-warning py-2 px-3 mb-3 small" data-test="secret-mirror-failures" role="alert">
|
||||
<strong>Deployed secrets failing on a site node</strong>
|
||||
<ul class="mb-0 ps-3">
|
||||
@foreach (var node in report.SecretMirrorStatuses!.Where(n => n.Failures.Count > 0))
|
||||
{
|
||||
@foreach (var failure in node.Failures)
|
||||
{
|
||||
<li>
|
||||
<span class="font-monospace">@node.NodeName</span> ·
|
||||
<span class="font-monospace">@failure.SecretName</span>: @failure.Message
|
||||
</li>
|
||||
}
|
||||
}
|
||||
</ul>
|
||||
</div>
|
||||
}
|
||||
<div class="row g-3">
|
||||
@* Each logical group below sits in its own card, with the group's
|
||||
head promoted to a real card-header, instead of h6 rules stacked
|
||||
@@ -590,6 +618,15 @@
|
||||
state.IsOnline
|
||||
&& state.LatestReport?.StartupReconcilerActive == false;
|
||||
|
||||
// Review finding M1: warn when any site node (the standby included) reports deployed
|
||||
// secrets it cannot apply/decrypt. Null ("no data": central self-report, pre-M1 site) and
|
||||
// nodes with an empty failure list are healthy. Offline sites are skipped like the other
|
||||
// warning badges — the Offline badge is the signal that matters there.
|
||||
private static bool HasSecretMirrorFailures(SiteHealthState state) =>
|
||||
state.IsOnline
|
||||
&& state.LatestReport?.SecretMirrorStatuses is { } statuses
|
||||
&& statuses.Any(n => n.Failures.Count > 0);
|
||||
|
||||
// Online central nodes, from the same ClusterNodes list the Nodes column renders.
|
||||
// Fewer than two means no standby, which disables the manual-failover control. This is
|
||||
// a display-side guard only — AkkaManualFailoverService re-checks against live cluster
|
||||
|
||||
Reference in New Issue
Block a user