docs(site-secrets): document secret_mirror_status and standby mirror failures on the health report (M1)

2026-09-28 21:33:10 -04:00
parent 0289d51137
commit f5756affef
5 changed files with 20 additions and 6 deletions
@@ -39,6 +39,7 @@ Site clusters (metric collection and reporting). Central cluster (aggregation an
 | `AuditRedactionFailure` | Audit Log (central) | Count of payload redactor errors (over-redacted payloads, safety-net hit) since the last interval. |
 | `LocalDbReplicationConnected` | Consolidated site LocalDb | Whether a peer sync session is currently running on this node. **Nullable — null means "no reading", not "disconnected".** Reported only by a node with replication configured; an unconfigured node reports **null**, never `false` (register row 80 — see *Unconfigured replication reports null* below). |
 | `StartupReconcilerActive` | Host (site) | Whether this node spawned its `SiteReconciliationActor` at boot. **Tri-state:** null = not reported (the central self-report, or a pre-row-73 site build), `true` = running, `false` = the node booted with no `IDeploymentConfigFetcher` and therefore has **no startup self-heal**. Latched once at boot by the Host — not polled. Drives the *No self-heal* dashboard badge (register row 73). |
+| `SecretMirrorStatuses` | Site Runtime (`DeployedSecretsMirror` via the replicated `secret_mirror_status` table) | Every site node's deployed-secrets mirror status — node name, when its failure set last changed, and its current failures (`{SecretName, Message}`; names and messages only, never values). Review finding M1: the **standby's** failures ride the **active** node's report, because each node writes its own row to a LocalDb-replicated table and the active node's `SecretMirrorStatusReporter` reads them all. **Nullable — null = no data** (central self-report, a pre-M1 site, reporter not yet run); an empty list = wired, no row yet; a node with an empty `Failures` list is healthy. Point-in-time (not drained on collect). Wire: `SiteHealthReportDto.secret_mirror_statuses` (field **33**, a presence wrapper so null ≠ empty). Drives the *Secret mirror failing* dashboard badge + per-node failure list. On an unreplicated pair only the active node's own row appears. |
 | `LocalDbOplogBacklog` | Consolidated site LocalDb | Unacked oplog depth (Prometheus `localdb_oplog_depth`). **Nullable — null means "no reading", NOT "connected with an empty backlog"** (a failed poll rendered as 0 would report a pair that cannot read its own oplog as perfectly healthy). **Expected transient spikes:** the daily `site_events` retention purge and the storage-cap trim are CDC-captured like any other write, so the backlog jumps by the batch size at purge time and drains as the peer acks it — see `Component-SiteEventLogging.md` → Storage and `docs/deployment/topology-guide.md` → *Reading the replication backlog* for the healthy-versus-faulty signature. |

 ## Reporting Protocol
@@ -232,6 +233,7 @@ endpoint (`MetricsPort`, default 8084) rather than sent to central.
 - **Communication Layer**: Transports health reports from sites to central.
 - **Data Connection Layer (site)**: Provides connection health metrics.
 - **Site Runtime (site)**: Provides script error rate and alarm evaluation error rate metrics.
+- **Site Runtime (site) — deployed secrets**: Provides `SecretMirrorStatuses` via its own `SecretMirrorStatusReporter` hosted service (SiteRuntime already references HealthMonitoring and owns the `secret_mirror_status` storage, following the `SiteStreamAlarmDropReporter` precedent).
 - **Store-and-Forward Engine (site)**: Provides buffer depth metrics, including the notification backlog awaiting forward to central.
 - **Cluster Infrastructure (site)**: Provides node role status.
 - **Host (site)**: Supplies the two boot-time facts Health Monitoring must not derive for itself — the LocalDb replication-configured gate behind `LocalDbReplicationConnected`/`LocalDbOplogBacklog` (from `SiteLocalDbSetup.ReplicationIsConfigured`), and `StartupReconcilerActive`, latched where the Host decides whether to spawn `SiteReconciliationActor`. Both follow the `SiteEventLogReplicationCheck` precedent: the Host owns the configuration reading, the component owns the reporting.