Site secrets ride the artifact deploy — design

Status: design, 2026-09-28. Plan: 2026-09-28-site-secrets-artifact-deploy.md.

1. Problem

A site resolves secrets at runtime (today: an MxGateway data connection's ApiKey: "secret:<name>", DataConnectionLayer/Adapters/MxGatewayDataConnection.cs:101). The only way a secret reaches a site is the pull-only gRPC secrets hub (Secrets:Replication:Mode=Grpc), which:

copies every central secret to every participating site node every 30 s — no per-site scope;
is invisible to the deployment model: the Sites page Artifacts current / out of date badge (Site.ArtifactRevisionHash, 2026-09-14) knows nothing about secrets, so a site can read Current while its data connection cannot resolve its key;
logs nothing on a successful in-sync sweep (library GrpcSecretSyncService logs only when rows were pulled or on failure) and exposes no status;
is enabled only on the docker rig's site-a; vd03 and QA run with replication OFF, so each site node's secrets.db is filled by hand.

2. Decisions (user, 2026-09-28)

#	Decision
U1	A site receives only the secrets it references — never other sites' keys.
U2	The artifact deploy replaces the gRPC hub as the way secrets reach sites (hub retired for ScadaBridge sites).
U3	Logging = deploy log lines (central + site), audit rows, and hub sweep logging (for the transition until the hub is retired, and for other hub consumers).
U4	Central deploys from one shared SQL-Server secret store used by both central nodes.

3. Design

3.1 Which secrets a site needs — derived, not configured

SecretReferenceScanner.Scan(IEnumerable<DataConnectionArtifact>) walks each connection's PrimaryConfigurationJson and BackupConfigurationJson and collects every JSON string value of the form secret:<name> (prefix case-insensitive, <name> trimmed, must be a valid SecretName). It is protocol-agnostic on purpose: a new adapter that adopts secret: references is covered with no scanner change. Result: a distinct, ordinal-sorted name set per site, each name carrying the data connections that reference it (for error messages and logs).

Deliberately NOT included: bootstrap secrets — the site's gRPC PSK (SB-GRPC-PSK-{siteId}), the LocalDb replication key, the secrets master key. The site needs them before it can receive an artifact deploy, and pushing a rotated PSK through the channel it authenticates would sever that channel. They stay out-of-band (service environment / hand-seeded store), exactly as today on vd03/QA. ${secret:X} references inside a node's own appsettings are node-local and invisible to central — also out of scope.

3.2 Central source of truth — the shared SQL store (U4)

New option Secrets:Store:Provider (Sqlite default | SqlServer), read only on the Central role. SqlServer registers the library's AddZbSecretsSqlServerStore against Secrets:SqlServer:ConnectionString (literal/env value — never ${secret:}, same bootstrap rule as today) so both central nodes read and write one store; the /admin/secrets page edits it. This is the store the hub's Grpc-mode central already uses — decoupled from the hub. A clustered Central left on Sqlite boots but StartupValidator logs a Warning (same shape as the inbound API key-store warning, row 108), because the two nodes would deploy different secret sets depending on which is active.

3.3 The wire: ciphertext rows, never plaintext

DeployArtifactsCommand gains an additive Secrets list of SecretArtifact — the library's StoredSecret fields verbatim (name, content type, ciphertext/nonce/tag, wrapped DEK/nonce/tag, KekId, Revision, created/updated UTC). Central never decrypts; the site applies the row with ISecretStore.ApplyReplicatedAsync (last-writer-wins on UpdatedUtc/Revision, the same path the hub sweep uses). Every node must therefore share the KEK (ZB_SECRETS_MASTER_KEY / master-key file) — exactly the hub's existing requirement. A mismatch is detected at deploy time (§3.5), not at the next reconnect. Proto: DeployArtifactsCommandDto.secrets = 9 (SecretArtifactListDto, absent ⇒ null) and ArtifactDeploymentResponseDto gains secrets_applied (wrapper Int32Value, absent ⇒ null).

Fail-closed on a missing secret. If a referenced name is absent or tombstoned in the central store, that site's deploy fails before anything is sent: Secret 'X' referenced by data connection(s) A, B is not in the central secret store. Shipping the rest would stamp the site Current while a connection cannot resolve its key.

3.4 Staleness — secrets are part of the artifact hash

ArtifactRevisionHashService adds Secrets → HashableSecret { ContentDigest, KekId, Missing, Name, Revision } (alphabetical — the existing ordering test covers it). ContentDigest = SHA-256 of ciphertext‖nonce‖tag‖wrappedDek, so a KEK rewrap (new wrapped DEK, same revision) correctly marks sites out of date. ComputeCurrentArtifactHashesAsync hashes missing names with Missing = true, so a site referencing a not-yet-created secret reads Out of date, never Current. The list is null-when-empty and the serializer omits nulls, so sites with no secret references keep their current hash — only sites that reference secrets flip to Out of date once after rollout.

3.5 Site apply — both nodes, via a replicated table

The artifact apply runs on the active node's DeploymentManagerActor singleton, but secrets.db is a per-node file outside LocalDb. So:

deployed_secrets — a new LocalDb table in SiteStorageSchema (name PK, row_json = the SecretArtifact ciphertext row as JSON, revision, updated_utc, is_removed 0/1, deployment_id, applied_at), added to SiteLocalDbSetup.ReplicatedTables (becomes the 11th; LocalDb 0.3.x snapshots just that table on an already-replicating pair). No FKs (row 105 guard). Ciphertext only — the no-plaintext rule that kept smtp_configurations unregistered holds. HandleDeployArtifacts applies it full-set: upsert every shipped row, mark rows absent from the set is_removed = 1 (kept, so the standby learns about the removal).
DeployedSecretsMirror — a per-node service (every site node, not a singleton) that reconciles deployed_secrets into the node's own ISecretStore: ApplyReplicatedAsync for live rows, DeleteAsync for removed rows only if the name was previously mirrored (a local, unreplicated mirrored_secrets table records names this mirror wrote — an out-of-band secret of the same name that was never deployed is never deleted). After applying, it trial-resolves each deployed secret through ISecretResolver; a decrypt failure (KEK mismatch) is an error naming the secret and the row's KekId. It runs at startup, every SiteRuntime:DeployedSecretsMirrorInterval (default 15 s — the standby's convergence path), and on demand.
The active node's apply calls the mirror synchronously after writing the table and fails the deploy response if any secret did not apply or verify; on success the response carries SecretsApplied = n. The standby converges through LocalDb replication + its own mirror within one interval.

3.6 Mixed versions

A pre-feature site ignores proto field 9 and answers Success with secrets_applied absent. Central treats command carried ≥ 1 secret AND SecretsApplied is null as a failure for that site (site build predates secret deployment — upgrade the site) and does not stamp the hash. A command with no secrets is unaffected, so an old site with no secret references deploys exactly as before. Upgrade central first, as always.

3.7 Logging and audit (U3)

Central, per site: Information Deploying {Count} secret(s) to site {SiteId}: {Name}@r{Revision},…; Error on a missing secret (names + referencing connections). Never values or ciphertext.
Site, active apply and every mirror pass that changes something: Information per secret Applied deployed secret {Name} r{Revision} (kek {KekId}) / Removed deployed secret {Name}; Error on apply/verify failure; one Information summary per pass. An in-sync mirror pass logs at Debug only.
Audit: one IAuditService row per site deploy that carried secrets — action DeploySiteSecrets, entity type SiteSecrets, entity id = site identifier, after-state { deploymentId, secrets: [{name, revision}], removed: [name], success }. Same config-change audit as the existing DeployArtifacts row.
Hub sweep (library ZB.MOM.WW.Secrets 0.6.3, scadaproj): every sweep outcome is observable — in-sync sweeps at Debug plus an Information heartbeat at most once per InSyncLogInterval (default 1 h), a Recovered after {Failures} failed sweep(s) Information line, and a public ISecretsHubSyncStatus (LastSweepUtc, LastSuccessUtc, ConsecutiveFailures, LastError). Useful for the rig until the hub is retired and for any other hub consumer.

3.8 Hub retirement (U2)

Once the artifact path is live on the rig, ScadaBridge stops composing the hub: the Site role no longer registers AddZbSecretsGrpcHubClient, the Central role no longer maps the hub endpoint, and StartupValidator refuses Secrets:Replication:Enabled=true with Mode=Grpc with a message pointing at the artifact deploy. The library keeps the hub (other consumers); only ScadaBridge stops using it. The rig's site-a moves from the hub to the artifact path; central keeps the shared SQL store through Secrets:Store:Provider=SqlServer.

4. Rollout (deployment records, not code)

vd03 and QA need, per environment: the zbsecrets store in the configuration SQL Server + both centrals on Secrets:Store:Provider=SqlServer; one KEK across every central and site node (verify before enabling — the mirror's verify step reports a mismatch by name); existing hand-seeded site secrets recreated at central; then Deploy Artifacts. Tracked as a deferred-work register row.

5. Out of scope

Per-site explicit secret lists; secrets for external systems / database connections (they ship literal values today); plaintext transport; moving bootstrap secrets through the deploy.

10 KiB Raw Blame History