fix(site-secrets): contain secret-store outages per site; audit removal-only deploys

Secret-store outage (final review M5): SiteSecretsCollector returns a store
read failure as SiteSecretsCollection.StoreError instead of throwing. A deploy
now fails only the sites that reference a secret (nothing sent, not stamped,
DeploySiteSecrets row with the error), and ComputeCurrentArtifactHashesAsync
omits only those sites (they read Out of date) instead of the exception
aborting the whole sweep and the fleet-wide pending-updates check.

Removal-only deploys: the site reply gains SecretsRemoved (proto field 7,
names only) - the deployed_secrets rows the deploy soft-removed. Central
carries it on SiteArtifactResult and writes the DeploySiteSecrets row with a
removed list whenever it is non-empty, so dropping a site's last secret:
reference is audited centrally rather than only in site logs.
This commit is contained in:
Joseph Doherty
2026-09-28 21:23:52 -04:00
parent ef03a0dca7
commit 29049c4ef6
15 changed files with 875 additions and 423 deletions
@@ -385,16 +385,25 @@ out below.
check 5). An old site ignores proto field 9, so the empty list costs it
nothing, and the hash is unaffected (empty == null above).
- **Audit.** One best-effort `DeploySiteSecrets` / `SiteSecrets` audit row is
written per site deploy that carried secrets or a missing-secret failure
(never for a site with no secret references at all) — after-state is `{
deploymentId, secrets: [{name, revision}], missing: [name], success }`, which
structurally excludes ciphertext. Written on both `DeployCoreAsync` and
`RetryForSiteAsync`, including the fail-closed path (audited even though
nothing was sent). The row carries `secrets` and `missing` only — there is
**no `removed` field** (design §3.7 named one), and a deploy that only
removes secrets (the site's last reference dropped, an empty list shipped)
writes **no central row**; removals are logged on the site by
`DeployedSecretsMirror`.
written per site deploy that carried secrets, failed on a missing secret or a
secret-store read, or retired previously-deployed secrets (never for a site
with no secret involvement at all) — after-state is `{ deploymentId,
secrets: [{name, revision}], missing: [name], removed: [name], success,
error }`, which structurally excludes ciphertext. Written on both
`DeployCoreAsync` and `RetryForSiteAsync`, including the fail-closed paths
(audited even though nothing was sent). `removed` is the site's reply field
`ArtifactDeploymentResponse.SecretsRemoved` — the `deployed_secrets` rows that
deploy soft-removed — so a deploy that only drops the site's last secret
reference is still audited centrally. A site that predates the field never
reports it.
- **Central secret-store outage.** `SiteSecretsCollector` returns a store read
failure as `StoreError` rather than throwing, so it is contained per site: a
site that references a secret fails (`Central secret store could not be read
while resolving '<name>' (...)`, nothing sent, not stamped), every other site
deploys, and `ComputeCurrentArtifactHashesAsync` omits only the affected sites
from its map (they read *Out of date* on the Sites page) instead of failing the
fleet-wide pending-updates check. A site with no secret references never
touches the store.
- **Secret rotation and live connections.** The site writes and mirrors the
shipped secrets before it pushes data connections to the DCL, so a
connection created by the same deploy resolves a new secret on first
@@ -166,6 +166,7 @@ flowchart TD
- `HandleDeployArtifacts` additionally accepts the command's `Secrets` list (`SecretArtifact`, ciphertext only) and writes the full shipped set to the `deployed_secrets` table via `SiteStorageService.UpsertDeployedSecretsAsync` — upsert every shipped name, soft-remove (`is_removed = 1`) any stored name absent from the set. A `null` `Secrets` field (pre-feature central) touches nothing and answers with no `SecretsApplied`. A current central always ships a list — **empty** when the site references no secrets — so dropping a site's last `secret:` reference removes the secret from both nodes. Storage detail: *Deployed Secrets (Site SQLite / LocalDb)* below.
- After writing the table, the active node runs `DeployedSecretsMirror.ReconcileAsync` **synchronously**, before replying — so the deploy response's `SecretsApplied` count and any apply/verify error reflect this deploy's own outcome, not a value left over from the mirror's background schedule. `SecretsApplied` counts only the names *this deploy shipped* that applied and verified cleanly; a mirror failure for an unrelated, already-broken name (a different deploy's leftover, or a stale removal) is logged at Warning and excluded — it neither fails this deploy nor reduces its `SecretsApplied`.
- No `DeployedSecretsMirror` registered (secrets deployment not composed on this node) with a non-empty `Secrets` list fails the deploy with `Secret deployment is not enabled on this site node`; with an empty `Secrets` list it still succeeds (`SecretsApplied = 0`).
- The reply also carries `SecretsRemoved`: the names `UpsertDeployedSecretsAsync` soft-removed on this deploy (site-wide, whichever node's mirror later deletes them), or `null` when none. Central uses it for the `removed` list of its `DeploySiteSecrets` audit row, so a deploy that only retires secrets is audited centrally.
- **Ordering against the DCL.** The shipped secrets are written and mirrored **before** the deploy's data-connection definitions are pushed to the DCL (the definitions are still stored in SQLite at the same point, and the push still goes out if the secrets step fails), so a connection created or reconfigured by the same deploy resolves a newly shipped secret on its first connect. A rotation with **unchanged** connection config does not recreate the connection (the DCL push is hash-guarded) — the new value takes effect on that connection's next (re)connect. No forced reconnect is performed.
- See *Deployed Secrets (Site SQLite / LocalDb)* below for the mirror itself.