fix(site-secrets): keep a live node's mirror-status row from losing a race with the peer's prune

Rig finding on the upgrade restart: both nodes' rows were older than the stale threshold
(the old build never refreshed them), so node-a's reporter pruned node-b's row while node-b
wrote a fresh one, and the replicated delete won. The reporter now prunes nothing during its
first refresh interval after start-up, and each mirror pass re-writes its own row if it no
longer holds what the mirror last wrote.
This commit is contained in:
Joseph Doherty
2026-09-29 11:06:26 -04:00
parent 55e1741075
commit 3a217b9484
7 changed files with 100 additions and 13 deletions
+1 -1
View File
@@ -451,7 +451,7 @@ Site secrets ride the artifact deploy (design `docs/plans/2026-09-28-site-secret
- **Isolation.** One row failing to apply/verify does not block any other row in the same pass.
- **Cache invalidation.** A successful apply or delete invalidates the shared `ISecretResolver` cache before verifying, so the verify step (and any script/connection reading the secret) sees the fresh value immediately rather than waiting out the resolver's cache TTL.
- **Never logs a secret value** — log lines and the reported `MirrorResult` carry only names, revisions, and `KekId`s.
- **Mirror status on the health report (review finding M1).** A failure on the **standby** — above all a secret that will not decrypt because the standby's master key differs from central's — used to live only in that node's log, invisible until a failover broke its connections. After every pass the mirror records its node's current failure set in `secret_mirror_status`, but **only when the set (sorted name+message pairs) differs from what this process last wrote**, plus once on the first pass after start-up so a restart refreshes a stale row, plus at least every `SiteRuntime:SecretMirrorStatusRefreshInterval` (default **10 min**) as a heartbeat — never on every 15 s tick, because every write is a replicated CDC change. The write is best-effort: a failure logs a Warning (repeats at Debug), never affects the pass result, and is retried next pass. `SecretMirrorStatusReporter` (`SiteRuntime/Secrets/`, every site node, 30 s, immediate first probe) reads every node's row into `ISiteHealthCollector.SetSecretMirrorStatuses`, so the active node's report carries both nodes' status as `SiteHealthReport.SecretMirrorStatuses` (see Component-HealthMonitoring.md). On an **unreplicated** pair only the active node's own row is visible — the same replication prerequisite deployed secrets already have. **Stale rows are pruned:** the row is keyed by node name, so a decommissioned or renamed node used to leave its row (and any *Secret mirror failing* badge) behind forever. The reporter now leaves off the report, and deletes, any row whose `checked_at` is older than `SiteRuntime:SecretMirrorStatusStaleAfter` (default **1 h**, validated `≥ 2 ×` the refresh interval) — the delete is conditional on the `checked_at` it read, so a node that re-wrote its row in the meantime keeps it, and it replicates like any other write. A node down longer than that also drops off the list (its absence already shows in the cluster member count) and writes its row again on its first pass after it returns.
- **Mirror status on the health report (review finding M1).** A failure on the **standby** — above all a secret that will not decrypt because the standby's master key differs from central's — used to live only in that node's log, invisible until a failover broke its connections. After every pass the mirror records its node's current failure set in `secret_mirror_status`, but **only when the set (sorted name+message pairs) differs from what this process last wrote**, plus once on the first pass after start-up so a restart refreshes a stale row, plus at least every `SiteRuntime:SecretMirrorStatusRefreshInterval` (default **10 min**) as a heartbeat — never on every 15 s tick, because every write is a replicated CDC change. The write is best-effort: a failure logs a Warning (repeats at Debug), never affects the pass result, and is retried next pass. `SecretMirrorStatusReporter` (`SiteRuntime/Secrets/`, every site node, 30 s, immediate first probe) reads every node's row into `ISiteHealthCollector.SetSecretMirrorStatuses`, so the active node's report carries both nodes' status as `SiteHealthReport.SecretMirrorStatuses` (see Component-HealthMonitoring.md). On an **unreplicated** pair only the active node's own row is visible — the same replication prerequisite deployed secrets already have. **Stale rows are pruned:** the row is keyed by node name, so a decommissioned or renamed node used to leave its row (and any *Secret mirror failing* badge) behind forever. The reporter now leaves off the report, and deletes, any row whose `checked_at` is older than `SiteRuntime:SecretMirrorStatusStaleAfter` (default **1 h**, validated `≥ 2 ×` the refresh interval) — the delete is conditional on the `checked_at` it read, so a node that re-wrote its row in the meantime keeps it, and it replicates like any other write. A node down longer than that also drops off the list (its absence already shows in the cluster member count) and writes its row again on its first pass after it returns. Two guards keep a live node's row from being lost to the prune: the reporter prunes nothing during its first refresh interval after start-up (on a pair restarted together — an upgrade from a build without the heartbeat — every row is still old until each node's first write has replicated), and each mirror pass checks with a local read that its own row still holds what it last wrote, re-writing it otherwise (a peer's prune racing this node's write can replicate a delete over it; found on the rig).
---