docs: site secrets ride the artifact deploy; gRPC hub retired in ScadaBridge

2026-09-28 20:22:45 -04:00
parent bc29dc412b
commit 0ccf68951d
8 changed files with 259 additions and 6 deletions
@@ -109,7 +109,7 @@ Central cluster only. Sites have no user interface.
 - **Move data connection between sites** (M9/T24): a **Move…** action on a data connection opens `MoveDataConnectionDialog`. The underlying `MoveDataConnectionCommand` guards against: target site does not exist, name collision at the target, any `InstanceConnectionBinding` referencing the connection, and any `InstanceNativeAlarmSourceOverride.ConnectionNameOverride` reference at the source site's templates. Blocked connections display the blocking reason.
 - **Server certificate management** (`/design/connections/{id}/certificates`, Admin role): a per-connection page that lists the contents of the site's OPC UA trusted-peer and rejected certificate stores (Subject / Issuer / Thumbprint / validity / Trusted-or-Rejected status) with a **Remove** action. The page makes clear the store is **node-wide for the site** (shared by every site node), not per data connection — trusting or removing a certificate affects all OPC UA connections at that site. Trust and Remove are central commands relayed to **both** site nodes so the node-local PKI stores stay consistent across failover (see Component-SiteRuntime.md, Component-DataConnectionLayer.md).
 - The site detail page exposes a new **"Audit feed"** tab that hosts the Audit Log page pre-filtered to `Site = <site>` — an in-context view of every operational audit event for that site.
- **Artifacts badge (2026-09-14, design `docs/plans/2026-09-14-pending-updates-design.md`).** Each site card carries an **Artifacts** badge next to the identifier, one of three states: `Current` (light — the site's `Site.ArtifactRevisionHash` matches what a deploy would send today), `Out of date` (warning, `bi-exclamation-triangle`, tooltip = the last successful deploy's UTC timestamp), or `Never deployed` (secondary — the site has no stamp at all, including every site immediately after the migration that added the columns, until one artifact deploy). Below the badge a muted `N of M instances out of date` line links to `/deployment/topology` whenever the site has any deployed instances, since the same fleet snapshot already carries the count.
+- **Artifacts badge (2026-09-14, design `docs/plans/2026-09-14-pending-updates-design.md`).** Each site card carries an **Artifacts** badge next to the identifier, one of three states: `Current` (light — the site's `Site.ArtifactRevisionHash` matches what a deploy would send today), `Out of date` (warning, `bi-exclamation-triangle`, tooltip = the last successful deploy's UTC timestamp), or `Never deployed` (secondary — the site has no stamp at all, including every site immediately after the migration that added the columns, until one artifact deploy). Below the badge a muted `N of M instances out of date` line links to `/deployment/topology` whenever the site has any deployed instances, since the same fleet snapshot already carries the count. **A site's secrets count toward the same badge (2026-09-28, `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md`)**: a data connection referencing a `secret:<name>` not yet created centrally hashes as `Missing` and reads `Out of date`, never `Current`, until the secret exists and a deploy ships it.
 - The page header gains a **Re-check** button (forces a fresh fleet evaluation, disabled while running) and an **"Updates as of HH:mm:ss UTC"** caption next to the page title. The snapshot loads **after** the site list, never blocking it — cards render immediately with a *checking…* placeholder for the badge/line, and a snapshot failure renders **unavailable** on the badge rather than hiding it or failing the page. A successful **Deploy Artifacts** (single-site or all-sites) forces a refresh of the snapshot so the badge flips to `Current` immediately rather than waiting out the cache TTL.

 ### Schema Library (Design Role)
@@ -189,6 +189,7 @@ Central cluster only. Sites have no user interface.
 - **Deploy all**: A bulk action deploys artifacts to all sites at once.
 - This is a **separate action** from instance deployment — system-wide artifacts are not automatically pushed when definitions change.
 - **Track per-site deployment status**, backed by `Site.ArtifactRevisionHash` / `ArtifactsDeployedAt` (Component-DeploymentManager.md → *Per-site artifact revision*), stamped **only on a successful deploy to that site**. A failed site keeps its previous stamp (or none) and keeps reading out of date on the Sites page; after an upgrade that ships the stamp columns, every site reads *Never deployed* until one artifact deploy runs — see the Artifacts badge above.
+- **Secrets (2026-09-28).** A deploy carries only the secrets that site's data connections reference (Component-DeploymentManager.md → *Site secrets*); no separate UI surface picks them. A site referencing a secret that is missing or tombstoned centrally fails **that site's** deploy (others in the same "Deploy all" sweep are unaffected) with a message naming the secret and the referencing data connection(s) — e.g. `Secret 'X' referenced by data connection(s) A, B is not in the central secret store` — surfaced on the deployment result exactly like any other per-site artifact-deploy failure. A site whose build predates secret deployment, or that applied only some of the shipped secrets, likewise fails and is not stamped `Current`; both cases resolve by upgrading the site and redeploying.

 ### Debug View (Deployment Role)
 - Select a deployed instance and open a live debug view — the instance picked through the same shared **`InstancePicker`** combobox as the Flow Debugger, with an `ItemLabel` suffix of `(State)` on each row.
@@ -281,6 +281,9 @@ Component-NotificationService.md). Notification delivery happens on the central
 cluster, so no notification artifact or SMTP credential is ever distributed to
 sites.

+A site's **secrets** are a fifth, *derived* artifact class riding the same command —
+see *Site secrets* below.
+
 ### Per-site artifact revision (2026-09-14)

 `Site` carries two nullable columns — `ArtifactRevisionHash` (`string?`, max 100)
@@ -331,6 +334,59 @@ design**: a partial map would silently report the unqueried sites as up to date,
 so the pending-updates cache instead discards the faulted flight and the UI shows
 the aggregate as unavailable.

+### Site secrets (2026-09-28)
+
+Site secrets ride the artifact deploy instead of the (now retired) gRPC secrets
+hub — design `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md`; see
+that doc's *Implementation notes (as built)* section for the deviations called
+out below.
+
+- **Derived, not configured.** `SiteSecretsCollector` scans the site's data
+  connection artifacts (`SecretReferenceScanner` over `PrimaryConfigurationJson`
+  / `BackupConfigurationJson`) for `secret:<name>` string values and resolves
+  each name against the central `ISecretStore`, producing the ciphertext-only
+  `SecretArtifact` rows the command carries (`DeployArtifactsCommand.Secrets`,
+  proto field 9) plus the set of unresolved references. A site with no `secret:`
+  references ships no secrets and is unaffected by any of this.
+- **Fail closed, per site.** A site with one or more unresolved (absent or
+  tombstoned) references fails its deploy *before* `_communicationService` is
+  called — nothing is sent, the site keeps its previous stamp, and the error
+  names the secret and the referencing data connection(s): `Secret 'X'
+  referenced by data connection(s) A, B is not in the central secret store`
+  (semicolon-joined for more than one). Other sites in the same sweep are
+  unaffected.
+- **Stamp gating.** A site is stamped `ArtifactRevisionHash`/`ArtifactsDeployedAt`
+  only when every shipped secret actually applied. Two failure shapes are
+  distinguished from an ordinary site failure and never stamped: an **old
+  site** (wire `Success = true`, secrets were shipped, but the response has no
+  `SecretsApplied` at all — a pre-feature site build) fails with `Site build
+  predates secret deployment — upgrade the site before deploying secrets`; a
+  **partial apply** (`SecretsApplied` less than the shipped count) fails with
+  `Site applied {n} of {m} secrets`. Both apply identically on the all-sites
+  sweep (`DeployCoreAsync`) and the per-site retry (`RetryForSiteAsync`).
+- **Staleness hash.** `ArtifactRevisionHashService` folds secrets into the same
+  hash as every other artifact class (`HashableSecret`: a SHA-256 digest of the
+  ciphertext/nonce/tag/wrapped-DEK material, never plaintext, plus `KekId` and
+  `Revision`) so a KEK rewrap correctly flips a site to out of date. A
+  referenced-but-not-yet-created secret hashes as an explicit `Missing` marker
+  (`DeployArtifactsCommand.MissingSecrets`, central-only — never on the wire),
+  so that site reads *Out of date*, never *Current*, until the secret exists.
+  **A `null` secrets list and an empty one hash identically** (the existing
+  null-when-empty convention for every artifact class), so a site with no
+  secret references keeps its pre-existing hash across the upgrade that added
+  this feature.
+- **Audit.** One best-effort `DeploySiteSecrets` / `SiteSecrets` audit row is
+  written per site deploy that carried secrets or a missing-secret failure
+  (never for a site with no secret references at all) — after-state is `{
+  deploymentId, secrets: [{name, revision}], missing: [name], success }`, which
+  structurally excludes ciphertext. Written on both `DeployCoreAsync` and
+  `RetryForSiteAsync`, including the fail-closed path (audited even though
+  nothing was sent).
+- **Central store.** Central resolves secrets from `ISecretStore`, composed by
+  the Host per `Secrets:Store:Provider` — see Component-Host.md → *Secrets*.
+- **Site apply and the per-node mirror** — see Component-SiteRuntime.md →
+  *Deployed Secrets*.
+
 ## Site-Side Apply Atomicity

 Applying a deployment at the site is **all-or-nothing per instance**:
@@ -420,6 +476,7 @@ The Deployment Manager sends the following commands to sites via the Communicati
 - **Configuration Database (MS SQL)**: Stores deployment status and deployed configuration snapshots.
 - **Security & Auth**: Enforces Deployment role (with optional site scoping).
 - **Configuration Database (via IAuditService)**: Logs all deployment actions, system-wide artifact deployments, and instance lifecycle changes.
+- **`ZB.MOM.WW.Secrets.Abstractions`** (`ISecretStore`, `SecretName`): `SiteSecretsCollector` resolves a site's referenced secrets against the Host-composed central store — see Component-Host.md → *Secrets*.

 ## Interactions

@@ -249,6 +249,58 @@ reporting that it is a standby. The verdict remains fully observable through the

 ---

+## Secrets (2026-09-28)
+
+The Host composes `ZB.MOM.WW.Secrets` (a `scadaproj` shared library, not a ScadaBridge component —
+hence it has no row in the Component Registration Matrix below) via `SecretsRegistration.AddScadaBridgeSecrets`,
+called on both roles from their respective composition-root `Configure` methods. Design:
+`docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md` (see its *Implementation notes (as
+built)* section for where the shipped code deviates from the original design).
+
+- **Site role — always a local, undecorated store.** A site node's `ISecretStore`/`ISecretResolver`
+  is unconditionally the plain node-local SQLite store (`AddZbSecrets`), regardless of any
+  `Secrets:Store:Provider` value — a Site never reads that key. This is a hard invariant, not
+  just a default: `DeployedSecretsMirror` (Component-SiteRuntime.md) calls `ISecretStore.DeleteAsync`
+  directly on the assumption that a delete is node-local; a replication-decorated store would have
+  published that delete fleet-wide. Secrets reach a site only via Deploy Artifacts (see
+  Component-DeploymentManager.md → *Site secrets*) — there is no other channel.
+- **Central role — `Secrets:Store:Provider`** (`Sqlite` default | `SqlServer`), read **only** on the
+  Central role:
+  - `Sqlite` (default): each central node keeps its own per-node SQLite secret store. On a
+    **clustered** central (≥ 2 seed nodes) this is a documented misconfiguration — `StartupValidator`
+    logs a boot-time Warning (same shape as the inbound-API-key-store precedent, register row 108),
+    because the two nodes would deploy different secret sets to sites depending on which is active.
+  - `SqlServer`: both central nodes read and write **one shared SQL Server store**
+    (`Secrets:SqlServer:ConnectionString`, a literal/env value — never a `${secret:}` reference,
+    the same bootstrap-circularity rule the inbound-API-key shared store follows) — the `/admin/secrets`
+    page edits it, and it is the store `SiteSecretsCollector` resolves from. The pre-host `${secret:}`
+    expander (the Layer-A pass that resolves `${secret:X}` tokens inside `appsettings` itself, before
+    the DI container is built) reads the **same** shared store when `Provider = SqlServer`, so a
+    central node's boot-time config tokens and its running `ISecretResolver` never disagree about
+    what a token resolves to.
+  - A connection string that is blank, or is itself a `${secret:}` reference (a bootstrap circularity),
+    fails fast naming the offending key.
+- **Bootstrap secrets are never deployed.** The site's gRPC PSK (`SB-GRPC-PSK-{siteId}`), the LocalDb
+  replication key, and the secrets master key (KEK) are never data-connection-referenced and are
+  never shipped through Deploy Artifacts — a site needs them *before* it can receive a deploy at all,
+  and pushing a rotated one through the channel it authenticates would sever that channel. They stay
+  out-of-band: service environment variables, a hand-seeded store, or a master-key file, exactly as
+  before this feature. Every node — every central and every site — **must share the same KEK**;
+  see `docs/deployment/topology-guide.md` → *Secrets: shared store, per-site deploy, and the KEK*
+  for where that is (and is not) checked.
+- **The gRPC secrets hub is retired in ScadaBridge (2026-09-28).** ScadaBridge no longer composes
+  `ZB.MOM.WW.Secrets.Replicator.Grpc` on either role, and the historical SQL-Server *replication*
+  mode (`Secrets:Replication:Mode=SqlServer` — distinct from `Secrets:Store:Provider=SqlServer`
+  above, which shares one store rather than replicating between two) is retired too. Setting
+  `Secrets:Replication:Enabled` to anything other than absent/empty/`false` now fails
+  `StartupValidator.Validate` outright, on **either** role, with a message pointing at Deploy
+  Artifacts (sites) and `Secrets:Store:Provider=SqlServer` (a clustered central). A leftover
+  `Secrets:Replication:Mode` or any `Secrets:GrpcHub:*` key (harmless once `Enabled` is unset, but
+  dead configuration) logs a startup Warning naming both sections so they can be cleaned up. The
+  library itself keeps the hub for other consumers; only ScadaBridge stops using it.
+
+---
+
 ## Component Registration Matrix

 | Component | Central | Site | DI (`AddXxx`) | Actors (`AddXxxActors`) | Endpoints (`MapXxx`) |
@@ -162,6 +162,12 @@ flowchart TD
 - Recompiles shared scripts and makes updated code available to all Script Actors.
 - **Reconciles deletions**: each non-null artifact list is the complete set for its class, so after upserting, any stored row absent from the set is deleted (the artifact was deleted centrally — there is no per-artifact delete command). A removed shared script is also unregistered from the compiled script library, and a removed data connection's live DCL connection actor is stopped. A `null` list means "field not shipped" and reconciles nothing.

+### Deployed Secrets (2026-09-28)
+- `HandleDeployArtifacts` additionally accepts the command's `Secrets` list (`SecretArtifact`, ciphertext only) and writes the full shipped set to the `deployed_secrets` table via `SiteStorageService.UpsertDeployedSecretsAsync` — upsert every shipped name, soft-remove (`is_removed = 1`) any stored name absent from the set. A `null` `Secrets` field (pre-feature central) touches nothing and answers with no `SecretsApplied`. Storage detail: *Deployed Secrets (Site SQLite / LocalDb)* below.
+- After writing the table, the active node runs `DeployedSecretsMirror.ReconcileAsync` **synchronously**, before replying — so the deploy response's `SecretsApplied` count and any apply/verify error reflect this deploy's own outcome, not a value left over from the mirror's background schedule. `SecretsApplied` counts only the names *this deploy shipped* that applied and verified cleanly; a mirror failure for an unrelated, already-broken name (a different deploy's leftover, or a stale removal) is logged at Warning and excluded — it neither fails this deploy nor reduces its `SecretsApplied`.
+- No `DeployedSecretsMirror` registered (secrets deployment not composed on this node) with a non-empty `Secrets` list fails the deploy with `Secret deployment is not enabled on this site node`; with an empty `Secrets` list it still succeeds (`SecretsApplied = 0`).
+- See *Deployed Secrets (Site SQLite / LocalDb)* below for the mirror itself.
+
 ### Instance Lifecycle Commands
 - **Disable**: Stops the Instance Actor and its children. Retains the deployed configuration in SQLite so the instance can be re-enabled without redeployment.
 - **Enable**: Creates a new Instance Actor from the stored configuration (same as startup).
@@ -424,6 +430,26 @@ The Instance Actor owns native-alarm setup alongside its computed Script and Ala

 ---

+## Deployed Secrets (Site SQLite / LocalDb)
+
+Site secrets ride the artifact deploy (design `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md`; see that doc's *Implementation notes (as built)* for deviations from the original design). Two tables, two different replication postures:
+
+- **`deployed_secrets`** — the **11th LocalDb-replicated table** (`SiteLocalDbSetup.ReplicatedTables`, up from ten). Columns: `name` (PK), `row_json` (the shipped `SecretArtifact` — ciphertext, nonce, tag, wrapped DEK, `KekId`, revision; never plaintext), `revision`, `updated_utc`, `is_removed` (soft-remove flag, default 0), `deployment_id`, `applied_at`. No FKs (the standing FK guard — `SiteLocalDbReplicatedTableFkGuardTests` — applies to it like every other replicated table). `UpsertDeployedSecretsAsync` is full-set upsert-then-soft-remove, never a hard `DELETE`, so a removal is itself a replicated row change the standby node can react to.
+- **`mirrored_secrets`** — a **node-local, unreplicated** ledger (`name` PK, `revision`, `mirrored_at`) recording which names *this specific node* has applied into its own `ISecretStore`. Deliberately not registered with LocalDb: it is a per-node fact (which names this node mirrored), not site-wide state, and replicating it would make one node's mirroring progress masquerade as the other's.
+
+**`DeployedSecretsMirror`** (`SiteRuntime/Secrets/DeployedSecretsMirror.cs`) is a per-node `BackgroundService` — registered on **every** site node, not a singleton — that reconciles `deployed_secrets` into the node's own local `ISecretStore` (site nodes never run a replication-decorated store; see Component-Host.md → *Secrets*):
+
+- **Schedule**: runs once immediately at start, then every `SiteRuntime:DeployedSecretsMirrorInterval` (default 15s, validated `> 0`), and synchronously on demand right after an active-node artifact apply (see *Deployed Secrets* under Deployment Manager Singleton above). This 15s interval is the **standby node's own convergence path** — it never receives the active node's synchronous call, only the replicated `deployed_secrets` rows via LocalDb, so it picks them up on its own next tick.
+- **Apply is confirm-then-ledger, not blind.** `ApplyReplicatedAsync` is strict last-writer-wins on `(UpdatedUtc, Revision)` (a tie is a no-op) and `DeleteAsync` tombstones rather than hard-deletes — so the mirror always reads the row back after applying and only ledgers it in `mirrored_secrets` when the local ciphertext matches what was shipped. A tombstoned or clock-skewed local row that the ledger already owns is safely re-stamped (`UpdatedUtc`/`Revision` bumped past the local value) and re-applied; a live row with **different** ciphertext that the ledger does **not** already own is left completely untouched — not overwritten, not ledgered, not verified — and reported as an out-of-band conflict (`"exists on this node with a newer local value; the deployed revision was not applied"`), so an operator's genuinely local secret of the same name is never silently destroyed by a later central deploy or removal.
+- **Ownership, not tombstone-state, decides re-stamping.** Once a name is ledgered in `mirrored_secrets` at any revision, this mirror treats that name as its own going forward — including re-stamping a mirror-owned name that drifted (e.g. an operator edited it locally) back to the deployed value on the next pass. Central is authoritative for every name it deploys.
+- **Removal** deletes from the local store **only if the name was previously ledgered** (present in `mirrored_secrets`) — an out-of-band local secret sharing a deployed name that was never mirrored is never touched by a central-side removal.
+- **Verify.** Every pass trial-resolves every live row through `ISecretResolver` (even one not re-applied this pass, so a later KEK change is still caught) — a decrypt failure names the secret and its `KekId` and is reported as a distinct error (de-spammed: Error the first time a given `(name, revision, KekId)` fails, Debug on identical repeats, Error again after an intervening success).
+- **Isolation.** One row failing to apply/verify does not block any other row in the same pass.
+- **Cache invalidation.** A successful apply or delete invalidates the shared `ISecretResolver` cache before verifying, so the verify step (and any script/connection reading the secret) sees the fresh value immediately rather than waiting out the resolver's cache TTL.
+- **Never logs a secret value** — log lines and the reported `MirrorResult` carry only names, revisions, and `KekId`s.
+
+---
+
 ## Enriched `AlarmStateChanged` Message

 The `AlarmStateChanged` message published by both Alarm Actors and Native Alarm Actors was extended **additively** (existing consumers keep working with computed defaults):
@@ -642,10 +668,11 @@ Per Akka.NET best practices, internal actor communication uses **Tell** (fire-an
 - **Site Event Logging**: Records script executions, alarm events, deployment events, instance lifecycle events.
 - **Health Monitoring**: Reports script error rates and alarm evaluation error rates.
 - **Local SQLite**: Persists deployed configurations, system-wide artifacts (external system definitions, database connection definitions, data connection definitions). Sites also maintain peer SQLite stores for the Store-and-Forward buffer, the site event log, the operation tracking table, and the site-local `AuditLog` (see [Component-AuditLog.md](Component-AuditLog.md)). The `AuditLog` file is purged on the same daily cadence as the others but respects the hard `ForwardState` invariant — rows still `Pending` forward are never purged, regardless of age.
+- **`ZB.MOM.WW.Secrets.Abstractions`** (`ISecretStore`, `ISecretResolver`, `ISecretCacheInvalidator`): `DeployedSecretsMirror` applies deployed secrets into, and verifies against, the node's local secret store (composed by the Host — see Component-Host.md → *Secrets*). The Site Runtime depends only on the abstractions package, never the concrete store implementation.

 ## Interactions

- **Deployment Manager (central)**: Receives flattened configurations, system-wide artifact updates, and instance lifecycle commands.
+- **Deployment Manager (central)**: Receives flattened configurations, system-wide artifact updates (including a site's referenced secrets, ciphertext only), and instance lifecycle commands.
 - **Data Connection Layer**: Bidirectional — receives value updates, sends write-back commands.
 - **Communication Layer**: Receives commands from central, sends deployment results, serves debug view data.
 - **Store-and-Forward Engine**: Scripts route cached writes, notifications, and external system calls here.