fix(site-secrets): report every site node's secret mirror failures on the health report (M1)

A DeployedSecretsMirror failure on the STANDBY node (e.g. a master-key mismatch
that stops a deployed secret decrypting there) was only logged on that node:
site health reports are sent by the active node only, so central never saw it.

- New 12th LocalDb-replicated table secret_mirror_status (node_name PK,
  failures_json of {SecretName, Message}, checked_at; no FKs). Each node's
  mirror writes only its own row, and only when its failure set changes plus
  once per start-up (every write replicates via CDC). Best-effort: a failed
  write is a Warning and is retried next pass. Node key = INodeIdentityProvider
  NodeName, falling back to the machine name.
- SecretMirrorStatusReporter (SiteRuntime, every node, 30 s) reads every row
  into ISiteHealthCollector.SetSecretMirrorStatuses -> additive nullable
  SiteHealthReport.SecretMirrorStatuses (null = no data).
- Wire: SiteHealthReportDto.secret_mirror_statuses = 33 (presence wrapper so
  null != empty); vendored CentralControl.cs regenerated; mapper both ways.
- Central UI Health card: 'Secret mirror failing' badge + compact per-node
  node / secret / message list.
- Tests: mirror status write-on-change/first-pass/failure isolation, storage
  round-trip, reporter/collector, DTO mapper round-trip incl. null/empty,
  bUnit, replicated-table set (twelve) + FK guard wording.
This commit is contained in:
Joseph Doherty
2026-09-28 21:33:10 -04:00
parent 23a139a46c
commit 0289d51137
24 changed files with 1718 additions and 100 deletions
@@ -1,6 +1,7 @@
using System.Text.Json;
using Microsoft.Extensions.Logging.Abstractions;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Artifacts;
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Health;
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
using ZB.MOM.WW.ScadaBridge.TestSupport;
@@ -303,6 +304,54 @@ public class DeployedSecretsStorageTests : IAsyncLifetime, IDisposable
Assert.Equal("A", mirrored.Name);
}
// ── secret_mirror_status (review finding M1) ──
[Fact]
public async Task SecretMirrorStatus_RoundTrips_OneRowPerNode_OrderedByNodeName()
{
var at = new DateTimeOffset(2026, 9, 28, 6, 0, 0, TimeSpan.Zero);
await _storage.SetSecretMirrorStatusAsync(
"node-b", [new SecretMirrorFailure("A", "did not decrypt")], at);
await _storage.SetSecretMirrorStatusAsync("node-a", [], at.AddMinutes(1));
var rows = await _storage.GetSecretMirrorStatusesAsync();
Assert.Equal(["node-a", "node-b"], rows.Select(r => r.NodeName));
Assert.Empty(rows[0].Failures);
Assert.Equal(at.AddMinutes(1), rows[0].CheckedAtUtc);
var failure = Assert.Single(rows[1].Failures);
Assert.Equal(new SecretMirrorFailure("A", "did not decrypt"), failure);
Assert.Equal(at, rows[1].CheckedAtUtc);
}
[Fact]
public async Task SecretMirrorStatus_Set_UpsertsTheNodesRow()
{
var at = new DateTimeOffset(2026, 9, 28, 6, 0, 0, TimeSpan.Zero);
await _storage.SetSecretMirrorStatusAsync("node-a", [new SecretMirrorFailure("A", "x")], at);
await _storage.SetSecretMirrorStatusAsync("node-a", [], at.AddHours(1));
var row = Assert.Single(await _storage.GetSecretMirrorStatusesAsync());
Assert.Empty(row.Failures);
Assert.Equal(at.AddHours(1), row.CheckedAtUtc);
}
[Fact]
public async Task SecretMirrorStatus_UnparseableRow_ReadsAsAFailure_NotAsHealthy()
{
await using (var connection = _storage.CreateConnection())
await using (var insert = connection.CreateCommand())
{
insert.CommandText =
"INSERT INTO secret_mirror_status (node_name, failures_json, checked_at) VALUES ('node-b', '{not json', '2026-09-28T00:00:00.0000000+00:00')";
await insert.ExecuteNonQueryAsync();
}
var row = Assert.Single(await _storage.GetSecretMirrorStatusesAsync());
Assert.Equal("node-b", row.NodeName);
Assert.Single(row.Failures);
}
private static SecretArtifact Art(string name, long revision) => new(
Name: name,
ContentType: "application/octet-stream",