fix(site-secrets): report every site node's secret mirror failures on the health report (M1)

A DeployedSecretsMirror failure on the STANDBY node (e.g. a master-key mismatch
that stops a deployed secret decrypting there) was only logged on that node:
site health reports are sent by the active node only, so central never saw it.

- New 12th LocalDb-replicated table secret_mirror_status (node_name PK,
  failures_json of {SecretName, Message}, checked_at; no FKs). Each node's
  mirror writes only its own row, and only when its failure set changes plus
  once per start-up (every write replicates via CDC). Best-effort: a failed
  write is a Warning and is retried next pass. Node key = INodeIdentityProvider
  NodeName, falling back to the machine name.
- SecretMirrorStatusReporter (SiteRuntime, every node, 30 s) reads every row
  into ISiteHealthCollector.SetSecretMirrorStatuses -> additive nullable
  SiteHealthReport.SecretMirrorStatuses (null = no data).
- Wire: SiteHealthReportDto.secret_mirror_statuses = 33 (presence wrapper so
  null != empty); vendored CentralControl.cs regenerated; mapper both ways.
- Central UI Health card: 'Secret mirror failing' badge + compact per-node
  node / secret / message list.
- Tests: mirror status write-on-change/first-pass/failure isolation, storage
  round-trip, reporter/collector, DTO mapper round-trip incl. null/empty,
  bUnit, replicated-table set (twelve) + FK guard wording.
This commit is contained in:
Joseph Doherty
2026-09-28 21:33:10 -04:00
parent 23a139a46c
commit 0289d51137
24 changed files with 1718 additions and 100 deletions
@@ -3,8 +3,8 @@ using Microsoft.Data.Sqlite;
namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
/// <summary>
/// DDL for the site's eleven configuration tables (nine original plus
/// <c>deployed_secrets</c> and <c>mirrored_secrets</c>), extracted from
/// DDL for the site's twelve configuration tables (nine original plus
/// <c>deployed_secrets</c>, <c>mirrored_secrets</c> and <c>secret_mirror_status</c>), extracted from
/// <see cref="SiteStorageService"/> so it can be applied by whoever owns the database
/// file.
/// <para>
@@ -23,7 +23,7 @@ namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Persistence;
public static class SiteStorageSchema
{
/// <summary>
/// Creates the eleven site configuration tables when absent, and additively upgrades
/// Creates the twelve site configuration tables when absent, and additively upgrades
/// tables created by an older build. Idempotent — safe to run on every startup.
/// </summary>
/// <param name="connection">An open connection to the database that should hold the tables.</param>
@@ -136,6 +136,20 @@ public static class SiteStorageSchema
revision INTEGER NOT NULL,
mirrored_at TEXT NOT NULL
);
-- Per-node deployed-secrets mirror status (review finding M1): one row per site
-- node, keyed by node name, holding that node's CURRENT mirror failures as a JSON
-- array of {SecretName, Message} (names and messages only — never values or
-- ciphertext). Unlike mirrored_secrets this IS replicated
-- (SiteLocalDbSetup.ReplicatedTables): each node writes only its own row, and the
-- replication is exactly what lets the ACTIVE node's health report carry the
-- STANDBY node's failures to central. Written only when a node's failure set
-- changes (plus once per start-up), never per mirror tick. No foreign keys.
CREATE TABLE IF NOT EXISTS secret_mirror_status (
node_name TEXT PRIMARY KEY,
failures_json TEXT NOT NULL,
checked_at TEXT NOT NULL
);
";
command.ExecuteNonQuery();
}