Merge branch 'fix/standby-secret-mirror-status' (M1: standby mirror failures on the health report)

# Conflicts:
#	CLAUDE.md
#	docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md

docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md

@@ -262,6 +262,15 @@ accurate as the spec:
     never deletes a name it did not ledger, so they were removed once by hand on the rig (both
     nodes stopped together; see the live-gate doc). QA and vd03 never ran the hub
     (`Secrets:Replication:Enabled=false`), so nothing like this exists there.
+- **Standby mirror failures reach central (follow-up, review finding M1).** A `DeployedSecretsMirror`
+  failure on the standby (e.g. a KEK mismatch) was only logged on that node — site health reports
+  are sent by the active node only. Each node now records its current failure set (names +
+  messages, never values) in a new, **12th LocalDb-replicated table `secret_mirror_status`** (one
+  row per node keyed by `node_name`), written only when the set changes plus once per start-up; a
+  per-node `SecretMirrorStatusReporter` puts every node's row on the active node's report as
+  `SiteHealthReport.SecretMirrorStatuses` (proto `SiteHealthReportDto` field 33, presence-wrapped),
+  and the Central UI health card shows a *Secret mirror failing* badge + per-node list. Requires
+  LocalDb replication for the standby's row to be visible, the same prerequisite as the secrets.
 - **Rig posture.** `docker/docker-compose.yml`'s central nodes now carry
   `Secrets__Store__Provider=SqlServer` plus a shared KEK; site-a shares that KEK (it is the
   secrets-via-artifact-deploy site); site-b/site-c deliberately carry a **mismatched** KEK so the

docs/requirements/Component-CentralUI.md

@@ -291,6 +291,7 @@ Per-leaf alarm rendering (leaf nodes are individual conditions for native alarms
 - Headline **Notification Outbox KPI tiles** — queue depth, stuck count, and parked count. These are central-computed by the Notification Outbox from the central `Notifications` table (not part of any site health report). The full outbox view is on the dedicated Notification Outbox page.
 - Headline **Site Call Audit KPI tiles** — buffered count, parked count, and failed-last-interval. These are central-computed by the Site Call Audit component from the central `SiteCalls` table (not part of any site health report). The full cached-call view is on the dedicated Site Calls page.
 - Headline **Audit KPI tiles** — three tiles in a new "Audit" KPI group: **Audit volume**, **Audit error rate**, and **Audit backlog**. These are sourced from the Audit Log component (#23) and Health Monitoring per the metric definitions in Component-HealthMonitoring.md; the dashboard simply surfaces them. The full audit query view is on the dedicated Audit Log page.
+- **Secret mirror failing** (review finding M1) — a warning badge on an online site's card when any node's `SiteHealthReport.SecretMirrorStatuses` entry lists failures, plus a compact warning panel at the top of the card body listing *node · secret: message* for each failure. The standby's failures appear here too (they ride the active node's report), so a master-key mismatch on the standby is visible before a failover exposes it. Null (no data) and all-healthy nodes render nothing; offline sites are skipped like the other warning badges.
 - **Trigger failover** (`CentralFailoverControl`, Administrator only) — on the central card and on each site card. Disabled with an explanatory tooltip when there is no standby to take over. It always confirms first, and the confirmation's red button reads **Fail over** (see the dialog-label contract under Technology); the central variant's copy states that this page is served by the node being restarted and will disconnect and reconnect, while the site variant deliberately does not, since a site is a different cluster.
 - **Shared KPI cache (arch-review WP2.4).** All four KPI families above — Notification Outbox, Site Call Audit, Audit, and their per-site/per-node breakdowns — are read through a process-level `IKpiSnapshotCache`, not queried per page load. Each accessor is independently memoized (8s TTL) with single-flight production, so N Blazor circuits polling the same KPI inside one window (e.g. ten operators with the Health dashboard open) cost the node ONE aggregate SQL round trip per KPI per window, not N. A failed query is never memoized — the next caller re-attempts it, and the calling page's existing per-tile "unavailable" degradation is unchanged. Every KPI-tile page (Health, Notification Outbox, Site Calls, Audit Log) shares the same cache instance.
 

docs/requirements/Component-HealthMonitoring.md

@@ -39,6 +39,7 @@ Site clusters (metric collection and reporting). Central cluster (aggregation an
 | `AuditRedactionFailure` | Audit Log (central) | Count of payload redactor errors (over-redacted payloads, safety-net hit) since the last interval. |
 | `LocalDbReplicationConnected` | Consolidated site LocalDb | Whether a peer sync session is currently running on this node. **Nullable — null means "no reading", not "disconnected".** Reported only by a node with replication configured; an unconfigured node reports **null**, never `false` (register row 80 — see *Unconfigured replication reports null* below). |
 | `StartupReconcilerActive` | Host (site) | Whether this node spawned its `SiteReconciliationActor` at boot. **Tri-state:** null = not reported (the central self-report, or a pre-row-73 site build), `true` = running, `false` = the node booted with no `IDeploymentConfigFetcher` and therefore has **no startup self-heal**. Latched once at boot by the Host — not polled. Drives the *No self-heal* dashboard badge (register row 73). |
+| `SecretMirrorStatuses` | Site Runtime (`DeployedSecretsMirror` via the replicated `secret_mirror_status` table) | Every site node's deployed-secrets mirror status — node name, when its failure set last changed, and its current failures (`{SecretName, Message}`; names and messages only, never values). Review finding M1: the **standby's** failures ride the **active** node's report, because each node writes its own row to a LocalDb-replicated table and the active node's `SecretMirrorStatusReporter` reads them all. **Nullable — null = no data** (central self-report, a pre-M1 site, reporter not yet run); an empty list = wired, no row yet; a node with an empty `Failures` list is healthy. Point-in-time (not drained on collect). Wire: `SiteHealthReportDto.secret_mirror_statuses` (field **33**, a presence wrapper so null ≠ empty). Drives the *Secret mirror failing* dashboard badge + per-node failure list. On an unreplicated pair only the active node's own row appears. |
 | `LocalDbOplogBacklog` | Consolidated site LocalDb | Unacked oplog depth (Prometheus `localdb_oplog_depth`). **Nullable — null means "no reading", NOT "connected with an empty backlog"** (a failed poll rendered as 0 would report a pair that cannot read its own oplog as perfectly healthy). **Expected transient spikes:** the daily `site_events` retention purge and the storage-cap trim are CDC-captured like any other write, so the backlog jumps by the batch size at purge time and drains as the peer acks it — see `Component-SiteEventLogging.md` → Storage and `docs/deployment/topology-guide.md` → *Reading the replication backlog* for the healthy-versus-faulty signature. |
 
 ## Reporting Protocol
@@ -232,6 +233,7 @@ endpoint (`MetricsPort`, default 8084) rather than sent to central.
 - **Communication Layer**: Transports health reports from sites to central.
 - **Data Connection Layer (site)**: Provides connection health metrics.
 - **Site Runtime (site)**: Provides script error rate and alarm evaluation error rate metrics.
+- **Site Runtime (site) — deployed secrets**: Provides `SecretMirrorStatuses` via its own `SecretMirrorStatusReporter` hosted service (SiteRuntime already references HealthMonitoring and owns the `secret_mirror_status` storage, following the `SiteStreamAlarmDropReporter` precedent).
 - **Store-and-Forward Engine (site)**: Provides buffer depth metrics, including the notification backlog awaiting forward to central.
 - **Cluster Infrastructure (site)**: Provides node role status.
 - **Host (site)**: Supplies the two boot-time facts Health Monitoring must not derive for itself — the LocalDb replication-configured gate behind `LocalDbReplicationConnected`/`LocalDbOplogBacklog` (from `SiteLocalDbSetup.ReplicationIsConfigured`), and `StartupReconcilerActive`, latched where the Host decides whether to spawn `SiteReconciliationActor`. Both follow the `SiteEventLogReplicationCheck` precedent: the Host owns the configuration reading, the component owns the reporting.

docs/requirements/Component-SiteRuntime.md

@@ -434,9 +434,10 @@ The Instance Actor owns native-alarm setup alongside its computed Script and Ala
 
 ## Deployed Secrets (Site SQLite / LocalDb)
 
-Site secrets ride the artifact deploy (design `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md`; see that doc's *Implementation notes (as built)* for deviations from the original design). Two tables, two different replication postures:
+Site secrets ride the artifact deploy (design `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md`; see that doc's *Implementation notes (as built)* for deviations from the original design). Three tables, two different replication postures:
 
 - **`deployed_secrets`** — the **11th LocalDb-replicated table** (`SiteLocalDbSetup.ReplicatedTables`, up from ten). Columns: `name` (PK), `row_json` (the shipped `SecretArtifact` — ciphertext, nonce, tag, wrapped DEK, `KekId`, revision; never plaintext), `revision`, `updated_utc`, `is_removed` (soft-remove flag, default 0), `deployment_id`, `applied_at`. No FKs (the standing FK guard — `SiteLocalDbReplicatedTableFkGuardTests` — applies to it like every other replicated table). `UpsertDeployedSecretsAsync` is full-set upsert-then-soft-remove, never a hard `DELETE`, so a removal is itself a replicated row change the standby node can react to.
+- **`secret_mirror_status`** — the **12th LocalDb-replicated table** (review finding M1). One row per site node: `node_name` (PK — `INodeIdentityProvider.NodeName`, i.e. `node-a`/`node-b`, falling back to the machine name when unconfigured), `failures_json` (a JSON array of `{SecretName, Message}` — the node's *current* mirror failures; names and messages only, never values or ciphertext), `checked_at`. No FKs. It is per-node *data* but replicated on purpose: each node writes only its own row, and replication is what lets the **active** node read the **standby's** row onto the site health report (only the active node sends one — see *Mirror status on the health report* below).
 - **`mirrored_secrets`** — a **node-local, unreplicated** ledger (`name` PK, `revision`, `mirrored_at`) recording which names *this specific node* has applied into its own `ISecretStore`. Deliberately not registered with LocalDb: it is a per-node fact (which names this node mirrored), not site-wide state, and replicating it would make one node's mirroring progress masquerade as the other's.
 
 **`DeployedSecretsMirror`** (`SiteRuntime/Secrets/DeployedSecretsMirror.cs`) is a per-node `BackgroundService` — registered on **every** site node, not a singleton — that reconciles `deployed_secrets` into the node's own local `ISecretStore` (site nodes never run a replication-decorated store; see Component-Host.md → *Secrets*):
@@ -450,6 +451,7 @@ Site secrets ride the artifact deploy (design `docs/plans/2026-09-28-site-secret
 - **Isolation.** One row failing to apply/verify does not block any other row in the same pass.
 - **Cache invalidation.** A successful apply or delete invalidates the shared `ISecretResolver` cache before verifying, so the verify step (and any script/connection reading the secret) sees the fresh value immediately rather than waiting out the resolver's cache TTL.
 - **Never logs a secret value** — log lines and the reported `MirrorResult` carry only names, revisions, and `KekId`s.
+- **Mirror status on the health report (review finding M1).** A failure on the **standby** — above all a secret that will not decrypt because the standby's master key differs from central's — used to live only in that node's log, invisible until a failover broke its connections. After every pass the mirror records its node's current failure set in `secret_mirror_status`, but **only when the set (sorted name+message pairs) differs from what this process last wrote**, plus once on the first pass after start-up so a restart refreshes a stale row — never on every 15 s tick, because every write is a replicated CDC change. The write is best-effort: a failure logs a Warning (repeats at Debug), never affects the pass result, and is retried next pass. `SecretMirrorStatusReporter` (`SiteRuntime/Secrets/`, every site node, 30 s, immediate first probe) reads every node's row into `ISiteHealthCollector.SetSecretMirrorStatuses`, so the active node's report carries both nodes' status as `SiteHealthReport.SecretMirrorStatuses` (see Component-HealthMonitoring.md). On an **unreplicated** pair only the active node's own row is visible — the same replication prerequisite deployed secrets already have.
 
 ---