fix(site-secrets): contain secret-store outages per site; audit removal-only deploys

Secret-store outage (final review M5): SiteSecretsCollector returns a store
read failure as SiteSecretsCollection.StoreError instead of throwing. A deploy
now fails only the sites that reference a secret (nothing sent, not stamped,
DeploySiteSecrets row with the error), and ComputeCurrentArtifactHashesAsync
omits only those sites (they read Out of date) instead of the exception
aborting the whole sweep and the fleet-wide pending-updates check.

Removal-only deploys: the site reply gains SecretsRemoved (proto field 7,
names only) - the deployed_secrets rows the deploy soft-removed. Central
carries it on SiteArtifactResult and writes the DeploySiteSecrets row with a
removed list whenever it is non-empty, so dropping a site's last secret:
reference is audited centrally rather than only in site logs.
This commit is contained in:
Joseph Doherty
2026-09-28 21:23:52 -04:00
parent ef03a0dca7
commit 29049c4ef6
15 changed files with 875 additions and 423 deletions
@@ -385,16 +385,25 @@ out below.
check 5). An old site ignores proto field 9, so the empty list costs it
nothing, and the hash is unaffected (empty == null above).
- **Audit.** One best-effort `DeploySiteSecrets` / `SiteSecrets` audit row is
written per site deploy that carried secrets or a missing-secret failure
(never for a site with no secret references at all) — after-state is `{
deploymentId, secrets: [{name, revision}], missing: [name], success }`, which
structurally excludes ciphertext. Written on both `DeployCoreAsync` and
`RetryForSiteAsync`, including the fail-closed path (audited even though
nothing was sent). The row carries `secrets` and `missing` only — there is
**no `removed` field** (design §3.7 named one), and a deploy that only
removes secrets (the site's last reference dropped, an empty list shipped)
writes **no central row**; removals are logged on the site by
`DeployedSecretsMirror`.
written per site deploy that carried secrets, failed on a missing secret or a
secret-store read, or retired previously-deployed secrets (never for a site
with no secret involvement at all) — after-state is `{ deploymentId,
secrets: [{name, revision}], missing: [name], removed: [name], success,
error }`, which structurally excludes ciphertext. Written on both
`DeployCoreAsync` and `RetryForSiteAsync`, including the fail-closed paths
(audited even though nothing was sent). `removed` is the site's reply field
`ArtifactDeploymentResponse.SecretsRemoved` — the `deployed_secrets` rows that
deploy soft-removed — so a deploy that only drops the site's last secret
reference is still audited centrally. A site that predates the field never
reports it.
- **Central secret-store outage.** `SiteSecretsCollector` returns a store read
failure as `StoreError` rather than throwing, so it is contained per site: a
site that references a secret fails (`Central secret store could not be read
while resolving '<name>' (...)`, nothing sent, not stamped), every other site
deploys, and `ComputeCurrentArtifactHashesAsync` omits only the affected sites
from its map (they read *Out of date* on the Sites page) instead of failing the
fleet-wide pending-updates check. A site with no secret references never
touches the store.
- **Secret rotation and live connections.** The site writes and mirrors the
shipped secrets before it pushes data connections to the DCL, so a
connection created by the same deploy resolves a new secret on first