ScadaBridge/docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md

# Site secrets ride the artifact deploy — design

**Status:** design, 2026-09-28. Plan: `2026-09-28-site-secrets-artifact-deploy.md`.

## 1. Problem

A site resolves secrets at runtime (today: an MxGateway data connection's `ApiKey: "secret:<name>"`,
`DataConnectionLayer/Adapters/MxGatewayDataConnection.cs:101`). The only way a secret reaches a site
is the pull-only gRPC secrets hub (`Secrets:Replication:Mode=Grpc`), which:

- copies **every** central secret to **every** participating site node every 30 s — no per-site scope;
- is invisible to the deployment model: the Sites page *Artifacts current / out of date* badge
  (`Site.ArtifactRevisionHash`, 2026-09-14) knows nothing about secrets, so a site can read *Current*
  while its data connection cannot resolve its key;
- logs nothing on a successful in-sync sweep (library `GrpcSecretSyncService` logs only when rows were
  pulled or on failure) and exposes no status;
- is enabled only on the docker rig's site-a; vd03 and QA run with replication OFF, so each site node's
  `secrets.db` is filled by hand.

## 2. Decisions (user, 2026-09-28)

| # | Decision |
|---|---|
| U1 | A site receives **only the secrets it references** — never other sites' keys. |
| U2 | The artifact deploy **replaces** the gRPC hub as the way secrets reach sites (hub retired for ScadaBridge sites). |
| U3 | Logging = **deploy log lines** (central + site), **audit rows**, and **hub sweep logging** (for the transition until the hub is retired, and for other hub consumers). |
| U4 | Central deploys from **one shared SQL-Server secret store** used by both central nodes. |

## 3. Design

### 3.1 Which secrets a site needs — derived, not configured

`SecretReferenceScanner.Scan(IEnumerable<DataConnectionArtifact>)` walks each connection's
`PrimaryConfigurationJson` and `BackupConfigurationJson` and collects every JSON **string value**
of the form `secret:<name>` (prefix case-insensitive, `<name>` trimmed, must be a valid `SecretName`).
It is protocol-agnostic on purpose: a new adapter that adopts `secret:` references is covered with
no scanner change. Result: a distinct, ordinal-sorted name set per site, each name carrying the
data connections that reference it (for error messages and logs).

**Deliberately NOT included:** bootstrap secrets — the site's gRPC PSK (`SB-GRPC-PSK-{siteId}`), the
LocalDb replication key, the secrets master key. The site needs them *before* it can receive an
artifact deploy, and pushing a rotated PSK through the channel it authenticates would sever that
channel. They stay out-of-band (service environment / hand-seeded store), exactly as today on vd03/QA.
`${secret:X}` references inside a node's own `appsettings` are node-local and invisible to central —
also out of scope.

### 3.2 Central source of truth — the shared SQL store (U4)

New option `Secrets:Store:Provider` (`Sqlite` default | `SqlServer`), read **only on the Central
role**. `SqlServer` registers the library's `AddZbSecretsSqlServerStore` against
`Secrets:SqlServer:ConnectionString` (literal/env value — never `${secret:}`, same bootstrap rule as
today) so both central nodes read and write one store; the `/admin/secrets` page edits it. This is
the store the hub's Grpc-mode central already uses — decoupled from the hub. A clustered Central left
on `Sqlite` boots but `StartupValidator` logs a Warning (same shape as the inbound API key-store
warning, row 108), because the two nodes would deploy different secret sets depending on which is
active.

### 3.3 The wire: ciphertext rows, never plaintext

`DeployArtifactsCommand` gains an additive `Secrets` list of `SecretArtifact` — the library's
`StoredSecret` fields verbatim (name, content type, ciphertext/nonce/tag, wrapped DEK/nonce/tag,
`KekId`, `Revision`, created/updated UTC). Central never decrypts; the site applies the row with
`ISecretStore.ApplyReplicatedAsync` (last-writer-wins on `UpdatedUtc`/`Revision`, the same path the
hub sweep uses). **Every node must therefore share the KEK** (`ZB_SECRETS_MASTER_KEY` / master-key
file) — exactly the hub's existing requirement. A mismatch is detected at deploy time (§3.5), not at
the next reconnect. Proto: `DeployArtifactsCommandDto.secrets = 9` (`SecretArtifactListDto`, absent ⇒
null) and `ArtifactDeploymentResponseDto` gains `secrets_applied` (wrapper `Int32Value`, absent ⇒ null).

**Fail-closed on a missing secret.** If a referenced name is absent or tombstoned in the central
store, that site's deploy fails before anything is sent: `Secret 'X' referenced by data connection(s)
A, B is not in the central secret store`. Shipping the rest would stamp the site *Current* while a
connection cannot resolve its key.

### 3.4 Staleness — secrets are part of the artifact hash

`ArtifactRevisionHashService` adds `Secrets` → `HashableSecret { ContentDigest, KekId, Missing,
Name, Revision }` (alphabetical — the existing ordering test covers it). `ContentDigest` = SHA-256 of
ciphertext‖nonce‖tag‖wrappedDek, so a KEK rewrap (new wrapped DEK, same revision) correctly marks
sites out of date. `ComputeCurrentArtifactHashesAsync` hashes missing names with `Missing = true`, so
a site referencing a not-yet-created secret reads *Out of date*, never *Current*. The list is
null-when-empty and the serializer omits nulls, so **sites with no secret references keep their
current hash** — only sites that reference secrets flip to *Out of date* once after rollout.

### 3.5 Site apply — both nodes, via a replicated table

The artifact apply runs on the active node's `DeploymentManagerActor` singleton, but `secrets.db` is
a per-node file outside LocalDb. So:

1. **`deployed_secrets`** — a new LocalDb table in `SiteStorageSchema` (`name` PK, `row_json` = the
   `SecretArtifact` ciphertext row as JSON, `revision`, `updated_utc`, `is_removed` 0/1,
   `deployment_id`, `applied_at`), added to `SiteLocalDbSetup.ReplicatedTables` (becomes the 11th;
   LocalDb 0.3.x snapshots just that table on an already-replicating pair). No FKs (row 105 guard).
   Ciphertext only — the no-plaintext rule that kept `smtp_configurations` unregistered holds.
   `HandleDeployArtifacts` applies it full-set: upsert every shipped row, mark rows absent from the
   set `is_removed = 1` (kept, so the standby learns about the removal).
2. **`DeployedSecretsMirror`** — a per-node service (every site node, not a singleton) that reconciles
   `deployed_secrets` into the node's own `ISecretStore`: `ApplyReplicatedAsync` for live rows,
   `DeleteAsync` for removed rows **only if the name was previously mirrored** (a local, unreplicated
   `mirrored_secrets` table records names this mirror wrote — an out-of-band secret of the same name
   that was never deployed is never deleted). After applying, it **trial-resolves** each deployed
   secret through `ISecretResolver`; a decrypt failure (KEK mismatch) is an error naming the secret
   and the row's `KekId`. It runs at startup, every `SiteRuntime:DeployedSecretsMirrorInterval`
   (default 15 s — the standby's convergence path), and on demand.
3. The active node's apply calls the mirror **synchronously** after writing the table and fails the
   deploy response if any secret did not apply or verify; on success the response carries
   `SecretsApplied = n`. The standby converges through LocalDb replication + its own mirror within one
   interval.

### 3.6 Mixed versions

A pre-feature site ignores proto field 9 and answers `Success` with `secrets_applied` absent. Central
treats *command carried ≥ 1 secret AND `SecretsApplied` is null* as a **failure** for that site
(`site build predates secret deployment — upgrade the site`) and does not stamp the hash. A command
with no secrets is unaffected, so an old site with no secret references deploys exactly as before.
Upgrade central first, as always.

### 3.7 Logging and audit (U3)

- **Central, per site:** Information `Deploying {Count} secret(s) to site {SiteId}: {Name}@r{Revision},…`;
  Error on a missing secret (names + referencing connections). Never values or ciphertext.
- **Site, active apply and every mirror pass that changes something:** Information per secret
  `Applied deployed secret {Name} r{Revision} (kek {KekId})` / `Removed deployed secret {Name}`;
  Error on apply/verify failure; one Information summary per pass. An in-sync mirror pass logs at
  Debug only.
- **Audit:** one `IAuditService` row per site deploy that carried secrets — action `DeploySiteSecrets`,
  entity type `SiteSecrets`, entity id = site identifier, after-state `{ deploymentId, secrets:
  [{name, revision}], removed: [name], success }`. Same config-change audit as the existing
  `DeployArtifacts` row.
- **Hub sweep (library `ZB.MOM.WW.Secrets` 0.6.3, scadaproj):** every sweep outcome is observable —
  in-sync sweeps at Debug plus an Information heartbeat at most once per `InSyncLogInterval`
  (default 1 h), a `Recovered after {Failures} failed sweep(s)` Information line, and a public
  `ISecretsHubSyncStatus` (`LastSweepUtc`, `LastSuccessUtc`, `ConsecutiveFailures`, `LastError`).
  Useful for the rig until the hub is retired and for any other hub consumer.

### 3.8 Hub retirement (U2)

Once the artifact path is live on the rig, ScadaBridge stops composing the hub: the Site role no
longer registers `AddZbSecretsGrpcHubClient`, the Central role no longer maps the hub endpoint, and
`StartupValidator` refuses `Secrets:Replication:Enabled=true` with `Mode=Grpc` with a message pointing
at the artifact deploy. The library keeps the hub (other consumers); only ScadaBridge stops using it.
The rig's site-a moves from the hub to the artifact path; central keeps the shared SQL store through
`Secrets:Store:Provider=SqlServer`.

## 4. Rollout (deployment records, not code)

vd03 and QA need, per environment: the `zbsecrets` store in the configuration SQL Server + both
centrals on `Secrets:Store:Provider=SqlServer`; **one KEK across every central and site node**
(verify before enabling — the mirror's verify step reports a mismatch by name); existing hand-seeded
site secrets recreated at central; then *Deploy Artifacts*. Tracked as a deferred-work register row.

## 5. Out of scope

Per-site explicit secret lists; secrets for external systems / database connections (they ship
literal values today); plaintext transport; moving bootstrap secrets through the deploy.