fix(site-secrets): contain secret-store outages per site; audit removal-only deploys

Secret-store outage (final review M5): SiteSecretsCollector returns a store
read failure as SiteSecretsCollection.StoreError instead of throwing. A deploy
now fails only the sites that reference a secret (nothing sent, not stamped,
DeploySiteSecrets row with the error), and ComputeCurrentArtifactHashesAsync
omits only those sites (they read Out of date) instead of the exception
aborting the whole sweep and the fleet-wide pending-updates check.

Removal-only deploys: the site reply gains SecretsRemoved (proto field 7,
names only) - the deployed_secrets rows the deploy soft-removed. Central
carries it on SiteArtifactResult and writes the DeploySiteSecrets row with a
removed list whenever it is non-empty, so dropping a site's last secret:
reference is audited centrally rather than only in site logs.
This commit is contained in:
Joseph Doherty
2026-09-28 21:23:52 -04:00
parent ef03a0dca7
commit 29049c4ef6
15 changed files with 875 additions and 423 deletions
@@ -237,11 +237,32 @@ accurate as the spec:
next (re)connect (no forced reconnect).
- *One `secret:` parser* (`SecretReferenceScanner.TryParseSecretReference`) shared by the scanner
and `MxGatewayDataConnection`, so a padded/case-variant reference resolves the shipped name.
- *Audit shape (§3.7 deviation).* The `DeploySiteSecrets` row carries `secrets` and `missing` —
there is **no `removed` field** — and a removal-only deploy (empty list shipped) writes **no
central row**; removals are logged on the site by `DeployedSecretsMirror`.
- *Audit shape (§3.7).* The `DeploySiteSecrets` row carries `secrets`, `missing` and `removed`.
(Shipped first without `removed`, so a removal-only deploy wrote no central row; closed by the
follow-ups below.)
- *Deployment prerequisite:* a site pair using deployed secrets must run LocalDb replication
(topology guide), since on an unreplicated pair only the node active at deploy time gets them.
- **Follow-ups closed after merge (2026-09-28).**
- *Secret-store outage is per site (review M5).* `SiteSecretsCollector` returns a store read
failure as `SiteSecretsCollection.StoreError` instead of throwing. A deploy fails only the
sites that reference a secret (nothing is sent to them; they get a `DeploySiteSecrets` row
with the error), and `ComputeCurrentArtifactHashesAsync` leaves those sites out of its map,
which the pending-updates evaluator reads as *Out of date*. Sites with no secret references
never touch the store and are unaffected. Before this, one `GetAsync` exception aborted the
whole sweep and the fleet-wide pending-updates check.
- *Removal-only deploys are audited.* The site's reply gained `SecretsRemoved` (proto
`ArtifactDeploymentResponseDto.secrets_removed = 7`, names only; absent/empty reads as null):
the `deployed_secrets` rows that deploy soft-removed. Central carries it on
`SiteArtifactResult.SecretsRemoved` and writes the `DeploySiteSecrets` row whenever it is
non-empty, with a `removed` list. An old site never sends it, so its removals stay unaudited
centrally, as before.
- *CLI exit code.* `site deploy-artifacts` and `deploy artifacts` exit `1` when the summary's
`failureCount` is non-zero (same rule `deploy site` already had).
- *Hub-era site-a secrets (review M8).* The retired hub had copied six `hub-gate-*`/`shared-gate-*`
test secrets (2026-08-07) into both rig site-a nodes' local stores. The mirror deliberately
never deletes a name it did not ledger, so they were removed once by hand on the rig (both
nodes stopped together; see the live-gate doc). QA and vd03 never ran the hub
(`Secrets:Replication:Enabled=false`), so nothing like this exists there.
- **Rig posture.** `docker/docker-compose.yml`'s central nodes now carry
`Secrets__Store__Provider=SqlServer` plus a shared KEK; site-a shares that KEK (it is the
secrets-via-artifact-deploy site); site-b/site-c deliberately carry a **mismatched** KEK so the