fix(site-secrets): contain secret-store outages per site; audit removal-only deploys

Secret-store outage (final review M5): SiteSecretsCollector returns a store
read failure as SiteSecretsCollection.StoreError instead of throwing. A deploy
now fails only the sites that reference a secret (nothing sent, not stamped,
DeploySiteSecrets row with the error), and ComputeCurrentArtifactHashesAsync
omits only those sites (they read Out of date) instead of the exception
aborting the whole sweep and the fleet-wide pending-updates check.

Removal-only deploys: the site reply gains SecretsRemoved (proto field 7,
names only) - the deployed_secrets rows the deploy soft-removed. Central
carries it on SiteArtifactResult and writes the DeploySiteSecrets row with a
removed list whenever it is non-empty, so dropping a site's last secret:
reference is audited centrally rather than only in site logs.
This commit is contained in:
Joseph Doherty
2026-09-28 21:23:52 -04:00
parent ef03a0dca7
commit 29049c4ef6
15 changed files with 875 additions and 423 deletions
@@ -6,12 +6,23 @@ namespace ZB.MOM.WW.ScadaBridge.DeploymentManager;
/// <summary>
/// Result of <see cref="SiteSecretsCollector.CollectAsync"/>: the resolved ciphertext rows for a
/// site's referenced secrets, and the references that could not be resolved (absent or
/// tombstoned) at central.
/// site's referenced secrets, the references that could not be resolved (absent or
/// tombstoned) at central, and — when the central secret store itself could not be read — the
/// error that stopped resolution.
/// </summary>
/// <param name="Secrets">The resolved rows.</param>
/// <param name="Missing">References that are absent or tombstoned in the store.</param>
/// <param name="StoreError">
/// <see langword="null"/> when every reference was looked up. Otherwise the store read failed
/// (final review M5: e.g. the shared SQL Server secret store is unreachable), resolution stopped
/// at the first failure, and <see cref="Secrets"/>/<see cref="Missing"/> are INCOMPLETE — the
/// caller must treat this site as undeployable and its current hash as unknown. The message is
/// the exception type and message only; it never contains secret material.
/// </param>
public sealed record SiteSecretsCollection(
IReadOnlyList<SecretArtifact> Secrets,
IReadOnlyList<SecretReference> Missing);
IReadOnlyList<SecretReference> Missing,
string? StoreError = null);
/// <summary>
/// Central half of design §3.1/§3.3: derives which secrets a site's data connections reference
@@ -30,7 +41,11 @@ public sealed class SiteSecretsCollector(ISecretStore store)
/// <param name="ct">A token to cancel the operation.</param>
/// <returns>
/// The resolved rows (mapped field-for-field from <see cref="StoredSecret"/>) and the
/// references that were absent or tombstoned in the store.
/// references that were absent or tombstoned in the store. A store read failure is
/// returned as <see cref="SiteSecretsCollection.StoreError"/> rather than thrown, so one
/// site's lookup failing never aborts a whole multi-site sweep (final review M5); only
/// cancellation of <paramref name="ct"/> propagates. A site whose connections reference no
/// secret never touches the store at all.
/// </returns>
public async Task<SiteSecretsCollection> CollectAsync(
IEnumerable<DataConnectionArtifact> connections, CancellationToken ct)
@@ -40,7 +55,19 @@ public sealed class SiteSecretsCollector(ISecretStore store)
foreach (var reference in SecretReferenceScanner.Scan(connections))
{
var row = await store.GetAsync(new SecretName(reference.Name), ct);
StoredSecret? row;
try
{
row = await store.GetAsync(new SecretName(reference.Name), ct);
}
catch (Exception ex) when (!ct.IsCancellationRequested)
{
return new SiteSecretsCollection(
secrets, missing,
$"Central secret store could not be read while resolving '{reference.Name}' " +
$"({ex.GetType().Name}: {ex.Message})");
}
if (row is null || row.IsDeleted)
{
missing.Add(reference);