fix(site-secrets): contain secret-store outages per site; audit removal-only deploys

Secret-store outage (final review M5): SiteSecretsCollector returns a store
read failure as SiteSecretsCollection.StoreError instead of throwing. A deploy
now fails only the sites that reference a secret (nothing sent, not stamped,
DeploySiteSecrets row with the error), and ComputeCurrentArtifactHashesAsync
omits only those sites (they read Out of date) instead of the exception
aborting the whole sweep and the fleet-wide pending-updates check.

Removal-only deploys: the site reply gains SecretsRemoved (proto field 7,
names only) - the deployed_secrets rows the deploy soft-removed. Central
carries it on SiteArtifactResult and writes the DeploySiteSecrets row with a
removed list whenever it is non-empty, so dropping a site's last secret:
reference is audited centrally rather than only in site logs.
This commit is contained in:
Joseph Doherty
2026-09-28 21:23:52 -04:00
parent ef03a0dca7
commit 29049c4ef6
15 changed files with 875 additions and 423 deletions
@@ -3352,6 +3352,7 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
// registered on this node fails the deploy rather than silently applying
// nothing.
int? secretsApplied = null;
IReadOnlyList<string>? secretsRemoved = null;
string? secretError = null;
if (command.Secrets != null)
{
@@ -3371,7 +3372,20 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
_logger.LogInformation(
"Applying {Count} deployed secret(s), deploymentId={DeploymentId}",
command.Secrets.Count, command.DeploymentId);
await _storage.UpsertDeployedSecretsAsync(command.Secrets, command.DeploymentId);
var (_, removedNames) =
await _storage.UpsertDeployedSecretsAsync(command.Secrets, command.DeploymentId);
// Reported to central for its DeploySiteSecrets audit row, so a deploy
// that only retires secrets is still audited there. These are the
// deployed_secrets rows this deploy marked removed (site-wide, the
// authoritative "no longer deployed" set) — not whichever of them this
// node's mirror happened to own and delete.
if (removedNames.Count > 0)
{
secretsRemoved = removedNames;
_logger.LogInformation(
"Retired {Count} deployed secret(s) no longer referenced, deploymentId={DeploymentId}: {Names}",
removedNames.Count, command.DeploymentId, string.Join(", ", removedNames));
}
var mirror = await _secretsMirror.ReconcileAsync(CancellationToken.None);
// The mirror reconciles the WHOLE site-wide deployed_secrets table
@@ -3418,7 +3432,7 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
return new ArtifactDeploymentResponse(
command.DeploymentId, "", secretError is null, secretError, DateTimeOffset.UtcNow,
secretsApplied);
secretsApplied, secretsRemoved);
}
catch (Exception ex)
{