docs: site secrets ride the artifact deploy; gRPC hub retired in ScadaBridge
This commit is contained in:
Joseph Doherty
@@ -120,6 +120,14 @@ Both central nodes share state through:
|
||||
because a failover then 401s every inbound client until keys are re-issued on the survivor. Both
|
||||
docker rigs run `SqlServer`; see `docs/operations/inbound-api-key-reissue.md` → *Central pairs* to
|
||||
migrate an existing pair.
|
||||
- **Secrets store**: shared **only** when `Secrets:Store:Provider = SqlServer` (design 2026-09-28,
|
||||
`docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md`) — both nodes then read/write the
|
||||
same shared SQL Server secret store, so `/admin/secrets` create/rotate/delete changes are
|
||||
cluster-wide immediately and both nodes deploy the same secret set to sites regardless of which
|
||||
is active. The library default, `Provider = Sqlite`, stores secrets in a **per-node** file instead;
|
||||
running `Sqlite` on a central pair is a documented misconfiguration — `StartupValidator` logs a
|
||||
boot-time Warning. See *Secrets: shared store, per-site deploy, and the KEK* below for the KEK
|
||||
requirement this store does **not** by itself satisfy.
|
||||
|
||||
### Load Balancer
|
||||
|
||||
@@ -558,6 +566,39 @@ path. That is the accepted posture today — the same trusted-network assumption
|
||||
already made, now with authentication rather than none. TLS on these listeners is follow-on
|
||||
hardening and needs no change to the key design.
|
||||
|
||||
#### Secrets: shared store, per-site deploy, and the KEK
|
||||
|
||||
Site secrets ride the artifact deploy (2026-09-28, `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md`;
|
||||
see that doc's *Implementation notes (as built)* for where the shipped code deviates from the
|
||||
original design). This section is the operational summary; component detail lives in
|
||||
Component-Host.md → *Secrets*, Component-DeploymentManager.md → *Site secrets*, and
|
||||
Component-SiteRuntime.md → *Deployed Secrets*.
|
||||
|
||||
- **Central**: `Secrets:Store:Provider = SqlServer` on both central nodes — see *Shared State*
|
||||
above. `SiteSecretsCollector` resolves each site's referenced secrets from that store at deploy
|
||||
time; a missing/tombstoned secret fails that site's deploy before anything is sent.
|
||||
- **Every node must share the same KEK** (`ZB_SECRETS_MASTER_KEY` / master-key file) — **every
|
||||
central node and every site node**, not just the central pair. This is the same requirement the
|
||||
retired gRPC secrets hub had; it did not change. **Nothing checks this at boot or at the next
|
||||
reconnect** — a mismatch surfaces only at **deploy time**, on the site: `DeployedSecretsMirror`'s
|
||||
verify step trial-decrypts every applied secret and reports a distinct error naming the secret
|
||||
and its `KekId` if the local master key cannot open it. Verify the KEK is identical everywhere
|
||||
**before** turning this on for a site — the rig's `docker/docker-compose.yml` deliberately runs
|
||||
site-b/site-c on a mismatched KEK precisely to exercise this failure path live.
|
||||
- **Standby / passive-node convergence.** The active site node applies a deploy's secrets
|
||||
synchronously (so the deploy response reflects that apply immediately); the standby node never
|
||||
receives that synchronous call — it converges through ordinary LocalDb replication of the
|
||||
`deployed_secrets` table plus its **own** `DeployedSecretsMirror` running on its own schedule,
|
||||
`ScadaBridge:SiteRuntime:DeployedSecretsMirrorInterval` (default **15s**). A freshly-promoted
|
||||
standby is therefore current on secrets within one interval of the last deploy, not instantly.
|
||||
- **Rollout is a deployment record, not a code change** — see the deferred-work register for the
|
||||
vd03/QA prerequisites (shared store provisioning, `Provider=SqlServer` on both centrals, the KEK
|
||||
alignment check above, recreating any hand-seeded site secrets centrally, then *Deploy Artifacts*).
|
||||
- **The gRPC secrets hub is retired in ScadaBridge** — `Secrets:Replication:Enabled` now fails
|
||||
startup on either role (Component-Host.md → *Secrets*). QA and vd03 currently run
|
||||
`Replication:Enabled=false`, so upgrading them to a build carrying this change will not itself
|
||||
fail fast.
|
||||
|
||||
## Scaling Guidelines
|
||||
|
||||
### Target Scale
|
||||
|
||||
@@ -120,6 +120,7 @@ All 7 fix-now items landed via PLAN-04/05/06/07/08 (verified in review 08 round
|
||||
| 124 | **No template-graph-watermark short-circuit on the pending-updates fleet flatten** *(Low)*. `PendingUpdatesEvaluator.EvaluateAsync` re-flattens every deployed instance through `DeploymentService.GetInstanceStalenessAsync` on every cache-TTL expiry (`PendingUpdatesCacheTtl`, default 60 s) while any Deployer/Admin circuit is open, even when nothing in the template/instance/artifact graph has changed since the last evaluation. | `docs/plans/2026-09-14-pending-updates-design.md` §3.2 ("Not in scope") | The single-flight TTL cache already bounds the cost to one fleet flatten per TTL fleet-wide (not per viewer); a watermark short-circuit is a further optimization, not a correctness need, and needs a cheap "has anything changed" signal that does not exist yet. | The fleet flatten's cost (instance count x template depth) becomes measurable against `PendingUpdatesCacheTtl` on a large deployment. |
|
||||
| 125 | **Pending-updates counts have no Health-dashboard tile or CLI surface** *(Low)*. The aggregate (`PendingUpdatesSnapshot.StaleInstanceCount` / `StaleArtifactSiteCount`) is consumed only by the Central UI Sites page and rail indicator; there is no Health dashboard KPI tile and no `deploy`/`site` CLI command surfacing it — `site list` shows only the raw `artifactRevisionHash`/`artifactsDeployedAt` fields on each site, not whether they are current. | `docs/plans/2026-09-14-pending-updates-design.md` §3.2 ("Not in scope") | Out of scope for the UI-focused ask; the CLI/Health surfaces are a separate consumer of the same `IPendingUpdatesCache`. | An operator wants the pending-updates counts without opening the Central UI (headless monitoring, CLI scripting). |
|
||||
| 126 | ~~**TODO — roll the two outstanding migrations to the QA pair vq03/vq04**~~ **DONE 2026-09-28** — QA vq03/vq04 upgraded `2b9a7c3c` → `b3021e7c`, 55 → 57, artifacts deployed (record `docs/deployment/deployment-records/2026-09-28-wonder-app-vq03-vq04-b3021e7c.md`). *(deployment, closed)*. **DEV vd03 DONE 2026-09-28** (`fe89363b`, 55 → 57, record `docs/deployment/deployment-records/2026-09-28-wonder-app-vd03-fe89363b.md` — the `sb-mig-55to57.sql` there is the same script QA needs). QA still runs `2b9a7c3c` with the DB at migration 55 (`20260909203046_EnforceGlobalInstanceNamesAndProtectProtocolConfiguration`); `main` (`ac833f5a`, pushed 2026-09-14) carries two more: **56** `20260911135501_AddAlarmMesReportable` (computed-alarm `IsMesReportable` flag) and **57** `20260914173147_AddSiteArtifactRevision` (`Sites.ArtifactRevisionHash`/`ArtifactsDeployedAt`, both nullable). Both are additive `AddColumn`s — no data upgrade, no coordinated step like the 2026-09-10 protocol-config encryption. Recipe (same as the 55 rollout): generate ONE from→to delta script `dotnet ef migrations script EnforceGlobalInstanceNamesAndProtectProtocolConfiguration AddSiteArtifactRevision --project src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase --startup-project src/ZB.MOM.WW.ScadaBridge.Host -o sb-mig-55to57.sql` (NOT idempotent, identical for vd03 and the QA pair), take the pre-upgrade DB backup, stop the pair TOGETHER, apply via `sqlcmd -C -b -I` on the SQL host, publish/swap the `ac833f5a` binaries, start the pair, confirm the Central boot log shows 0 pending migrations and no "Applying migration" (the script already applied them). After upgrade every site reads *Artifacts never deployed* until one *Deploy Artifacts to All Sites* — run it as the last step. Write the deployment record `docs/deployment/deployment-records/<date>-wonder-app-{vd03,vq03-vq04}-ac833f5a.md`. | Pending-updates finish (2026-09-14) | ~~Blocked by the 2026-09-11 remote-access hold~~ — **unblocked 2026-09-28**: `winrm` is re-enabled for deployment activities (CLAUDE.md). vd03 done; QA done 2026-09-28. | Closed 2026-09-28. |
|
||||
| 127 | **TODO — roll out site secrets via artifact deploy to vd03 + QA (vq03/vq04)**. The gRPC secrets hub is retired in ScadaBridge as of `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md` (implemented on branch `feat/site-secrets-artifact-deploy`, rig-only so far); vd03 and QA still run the pre-feature build with `Secrets:Replication:Enabled=false`, so upgrading them to a build carrying this change will **not** itself fail fast (the retirement's `StartupValidator` rejection only fires on `Enabled=true`, which neither environment sets) — but a site still needs the four prerequisites below before *Deploy Artifacts* will actually deliver its secrets. Prerequisites, in order: (1) provision the `zbsecrets` schema/store in the environment's configuration SQL Server (the same server `ConfigurationDb` already points at); (2) set `Secrets:Store:Provider=SqlServer` on **both** central nodes of the environment; (3) verify the KEK (`ZB_SECRETS_MASTER_KEY` / master-key file) is identical across **every** node — both centrals and every site node — since a mismatch is caught only at deploy time by `DeployedSecretsMirror`'s per-secret verify, not at boot; (4) recreate, at the new central shared store, any secret an operator previously hand-seeded directly into a site's local `secrets.db` (those never round-tripped through central and would otherwise be silently absent from the referenced-secrets scan). Then run *Deploy Artifacts* against every affected site as the rollout's last step. | `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md` §4 *Rollout*; `docs/deployment/topology-guide.md` → *Secrets: shared store, per-site deploy, and the KEK* | Deployment record, not a code change — vd03/QA config is user-held and their rollout schedule is independent of the branch merging to main. | User asks for the vd03/QA secrets window. |
|
||||
|
||||
## Resolved (verified against the code 2026-07-10)
|
||||
Rows removed from the Deferred table above once confirmed shipped. Kept here for traceability.
|
||||
|
||||
@@ -152,3 +152,76 @@ site secrets recreated at central; then *Deploy Artifacts*. Tracked as a deferre
|
||||
|
||||
Per-site explicit secret lists; secrets for external systems / database connections (they ship
|
||||
literal values today); plaintext transport; moving bootstrap secrets through the deploy.
|
||||
|
||||
## 6. Implementation notes (as built)
|
||||
|
||||
Execution (tasks 1–11, ledger `.superpowers/sdd/2026-09-28-site-secrets-artifact-deploy/progress.md`)
|
||||
matched this design in shape but landed several deviations, called out here so the doc stays
|
||||
accurate as the spec:
|
||||
|
||||
- **Central collection.** The scanner and per-site resolve described in §3.1/§3.3 are implemented
|
||||
as `SiteSecretsCollector` (`ZB.MOM.WW.ScadaBridge.DeploymentManager`), not inline in
|
||||
`ArtifactDeploymentService`. Fail-closed (§3.3) is per site, not per batch: a site with an
|
||||
unresolved reference is skipped (`_communicationService` is never called for it) while every
|
||||
other site in the same sweep proceeds normally.
|
||||
- **Audit (§3.7).** The `DeploySiteSecrets` / `SiteSecrets` audit row is written on **both**
|
||||
`DeployCoreAsync` (the sweep) and `RetryForSiteAsync` (the per-site retry), including the
|
||||
fail-closed path on each — the design's §3.7 wording ("one row per site deploy that carried
|
||||
secrets") did not call out the retry path explicitly; it is in scope and covered.
|
||||
- **Stamp gating (§3.6).** Implemented as designed, plus one addition: a **partial apply**
|
||||
(`SecretsApplied` less than the shipped count, not just the mixed-version `null` case) is also
|
||||
treated as a failure and is not stamped, with its own message (`Site applied {n} of {m}
|
||||
secrets`) distinct from the "predates secret deployment" wording.
|
||||
- **Hash (§3.4).** Implemented as designed. Confirmed explicitly (not just implied by
|
||||
"null-when-empty"): an **empty** `Secrets`/`MissingSecrets` list and a **null** one hash
|
||||
identically, and the pre-existing golden hash for a site with no secret references is
|
||||
unchanged by this feature (pinned by a dedicated regression test).
|
||||
- **Site apply (§3.5).** `HandleDeployArtifacts` writes the full `deployed_secrets` set and then
|
||||
runs `DeployedSecretsMirror.ReconcileAsync` **synchronously** before replying, exactly as
|
||||
designed. `SecretsApplied` counts only the names *this deploy shipped* that applied and
|
||||
verified without error — a mirror failure for a name this deploy did not ship (e.g. another,
|
||||
already-broken deployed secret, or an unrelated removal) is logged at Warning, excluded from
|
||||
the count, and does not fail the deploy. Removal failures are likewise Warnings, never deploy
|
||||
failures.
|
||||
- **`DeployedSecretsMirror` — apply is confirm-then-ledger, not the blind `ApplyReplicatedAsync`
|
||||
§3.5 originally described.** The real library store is strict last-writer-wins on
|
||||
`(UpdatedUtc, Revision)` (a tie is a no-op) and tombstones rather than hard-deletes, which the
|
||||
naive "apply then ledger" sequence gets wrong in two ways: a routine remove-then-re-add can
|
||||
no-op against a still-live tombstone while the ledger is written anyway (silently broken), and
|
||||
an out-of-band local secret with a genuinely newer stamp can be ledgered and later deleted by a
|
||||
central tombstone (destroying the operator's own secret). The mirror now always reads the row
|
||||
back after applying and only ledgers when the local ciphertext matches what was shipped; a
|
||||
tombstoned or ledger-owned row is safely re-stamped (`UpdatedUtc`/`Revision` bumped past the
|
||||
local value, including across clock skew) and re-applied, while a live row with different
|
||||
ciphertext that the ledger does **not** already own is left completely untouched and reported
|
||||
as a distinct out-of-band-conflict failure. Ownership is decided from `mirrored_secrets`
|
||||
membership at any revision, not from tombstone state alone — so a mirror-owned name that drifts
|
||||
locally (an operator's edit over an already-deployed name) is re-stamped back to the deployed
|
||||
value on the next pass: **central is authoritative for every name it deploys.** Every apply/
|
||||
delete also invalidates the shared `ISecretResolver` cache before the verify step, and one
|
||||
poisoned row never blocks the rest of the pass.
|
||||
- **Hub retirement (§3.8) — scope widened to ALL secrets replication, not just the gRPC hub.**
|
||||
ScadaBridge also retires the historical SQL-Server *replication* mode
|
||||
(`Secrets:Replication:Mode=SqlServer`, distinct from the still-supported
|
||||
`Secrets:Store:Provider=SqlServer` shared-store feature) for the same reason the gRPC hub goes:
|
||||
a site's `ISecretStore` must be the plain, undecorated local SQLite store, because
|
||||
`DeployedSecretsMirror` calls `DeleteAsync` directly and a replication decorator would publish
|
||||
that delete fleet-wide. `Secrets:Replication:Enabled` set to anything other than
|
||||
absent/empty/`false` (a deliberately strict, non-boolean-coerced check — `"0"`/`"1"`/`"yes"`
|
||||
all count as "set") now fails `StartupValidator.Validate` on **either** role. A leftover
|
||||
`Secrets:Replication:Mode` or any `Secrets:GrpcHub:*` key logs a startup Warning naming both
|
||||
sections. The pre-host `${secret:}` expander was fixed in the same task set to read the shared
|
||||
SQL store on a `Store:Provider=SqlServer` central (it previously only did this for the retired
|
||||
Grpc mode) — without that fix a central's boot-time `${secret:}` tokens and its running
|
||||
`ISecretResolver` could resolve a token differently.
|
||||
- **`ZB.MOM.WW.Secrets` 0.6.3 (hub sweep logging, §3.7's third bullet — `EventId`s 4/5/6,
|
||||
`ISecretsHubSyncStatus`) was implemented and tested on a `scadaproj` branch but is NOT
|
||||
published to the package feed, and ScadaBridge does not consume it.** Publishing is an outward
|
||||
side effect that needs explicit user consent, and it would be a dead pin here regardless —
|
||||
hub retirement (above) removes ScadaBridge's only dependency on the hub client entirely. It
|
||||
remains available for the library's other hub consumers once published.
|
||||
- **Rig posture.** `docker/docker-compose.yml`'s central nodes now carry
|
||||
`Secrets__Store__Provider=SqlServer` plus a shared KEK; site-a shares that KEK (it is the
|
||||
secrets-via-artifact-deploy site); site-b/site-c deliberately carry a **mismatched** KEK so the
|
||||
`DeployedSecretsMirror` verify-failure path has a real, provable negative on the rig rather than
|
||||
an assumed one.
|
||||
|
||||
@@ -109,7 +109,7 @@ Central cluster only. Sites have no user interface.
|
||||
- **Move data connection between sites** (M9/T24): a **Move…** action on a data connection opens `MoveDataConnectionDialog`. The underlying `MoveDataConnectionCommand` guards against: target site does not exist, name collision at the target, any `InstanceConnectionBinding` referencing the connection, and any `InstanceNativeAlarmSourceOverride.ConnectionNameOverride` reference at the source site's templates. Blocked connections display the blocking reason.
|
||||
- **Server certificate management** (`/design/connections/{id}/certificates`, Admin role): a per-connection page that lists the contents of the site's OPC UA trusted-peer and rejected certificate stores (Subject / Issuer / Thumbprint / validity / Trusted-or-Rejected status) with a **Remove** action. The page makes clear the store is **node-wide for the site** (shared by every site node), not per data connection — trusting or removing a certificate affects all OPC UA connections at that site. Trust and Remove are central commands relayed to **both** site nodes so the node-local PKI stores stay consistent across failover (see Component-SiteRuntime.md, Component-DataConnectionLayer.md).
|
||||
- The site detail page exposes a new **"Audit feed"** tab that hosts the Audit Log page pre-filtered to `Site = <site>` — an in-context view of every operational audit event for that site.
|
||||
- **Artifacts badge (2026-09-14, design `docs/plans/2026-09-14-pending-updates-design.md`).** Each site card carries an **Artifacts** badge next to the identifier, one of three states: `Current` (light — the site's `Site.ArtifactRevisionHash` matches what a deploy would send today), `Out of date` (warning, `bi-exclamation-triangle`, tooltip = the last successful deploy's UTC timestamp), or `Never deployed` (secondary — the site has no stamp at all, including every site immediately after the migration that added the columns, until one artifact deploy). Below the badge a muted `N of M instances out of date` line links to `/deployment/topology` whenever the site has any deployed instances, since the same fleet snapshot already carries the count.
|
||||
- **Artifacts badge (2026-09-14, design `docs/plans/2026-09-14-pending-updates-design.md`).** Each site card carries an **Artifacts** badge next to the identifier, one of three states: `Current` (light — the site's `Site.ArtifactRevisionHash` matches what a deploy would send today), `Out of date` (warning, `bi-exclamation-triangle`, tooltip = the last successful deploy's UTC timestamp), or `Never deployed` (secondary — the site has no stamp at all, including every site immediately after the migration that added the columns, until one artifact deploy). Below the badge a muted `N of M instances out of date` line links to `/deployment/topology` whenever the site has any deployed instances, since the same fleet snapshot already carries the count. **A site's secrets count toward the same badge (2026-09-28, `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md`)**: a data connection referencing a `secret:<name>` not yet created centrally hashes as `Missing` and reads `Out of date`, never `Current`, until the secret exists and a deploy ships it.
|
||||
- The page header gains a **Re-check** button (forces a fresh fleet evaluation, disabled while running) and an **"Updates as of HH:mm:ss UTC"** caption next to the page title. The snapshot loads **after** the site list, never blocking it — cards render immediately with a *checking…* placeholder for the badge/line, and a snapshot failure renders **unavailable** on the badge rather than hiding it or failing the page. A successful **Deploy Artifacts** (single-site or all-sites) forces a refresh of the snapshot so the badge flips to `Current` immediately rather than waiting out the cache TTL.
|
||||
|
||||
### Schema Library (Design Role)
|
||||
@@ -189,6 +189,7 @@ Central cluster only. Sites have no user interface.
|
||||
- **Deploy all**: A bulk action deploys artifacts to all sites at once.
|
||||
- This is a **separate action** from instance deployment — system-wide artifacts are not automatically pushed when definitions change.
|
||||
- **Track per-site deployment status**, backed by `Site.ArtifactRevisionHash` / `ArtifactsDeployedAt` (Component-DeploymentManager.md → *Per-site artifact revision*), stamped **only on a successful deploy to that site**. A failed site keeps its previous stamp (or none) and keeps reading out of date on the Sites page; after an upgrade that ships the stamp columns, every site reads *Never deployed* until one artifact deploy runs — see the Artifacts badge above.
|
||||
- **Secrets (2026-09-28).** A deploy carries only the secrets that site's data connections reference (Component-DeploymentManager.md → *Site secrets*); no separate UI surface picks them. A site referencing a secret that is missing or tombstoned centrally fails **that site's** deploy (others in the same "Deploy all" sweep are unaffected) with a message naming the secret and the referencing data connection(s) — e.g. `Secret 'X' referenced by data connection(s) A, B is not in the central secret store` — surfaced on the deployment result exactly like any other per-site artifact-deploy failure. A site whose build predates secret deployment, or that applied only some of the shipped secrets, likewise fails and is not stamped `Current`; both cases resolve by upgrading the site and redeploying.
|
||||
|
||||
### Debug View (Deployment Role)
|
||||
- Select a deployed instance and open a live debug view — the instance picked through the same shared **`InstancePicker`** combobox as the Flow Debugger, with an `ItemLabel` suffix of `(State)` on each row.
|
||||
|
||||
@@ -281,6 +281,9 @@ Component-NotificationService.md). Notification delivery happens on the central
|
||||
cluster, so no notification artifact or SMTP credential is ever distributed to
|
||||
sites.
|
||||
|
||||
A site's **secrets** are a fifth, *derived* artifact class riding the same command —
|
||||
see *Site secrets* below.
|
||||
|
||||
### Per-site artifact revision (2026-09-14)
|
||||
|
||||
`Site` carries two nullable columns — `ArtifactRevisionHash` (`string?`, max 100)
|
||||
@@ -331,6 +334,59 @@ design**: a partial map would silently report the unqueried sites as up to date,
|
||||
so the pending-updates cache instead discards the faulted flight and the UI shows
|
||||
the aggregate as unavailable.
|
||||
|
||||
### Site secrets (2026-09-28)
|
||||
|
||||
Site secrets ride the artifact deploy instead of the (now retired) gRPC secrets
|
||||
hub — design `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md`; see
|
||||
that doc's *Implementation notes (as built)* section for the deviations called
|
||||
out below.
|
||||
|
||||
- **Derived, not configured.** `SiteSecretsCollector` scans the site's data
|
||||
connection artifacts (`SecretReferenceScanner` over `PrimaryConfigurationJson`
|
||||
/ `BackupConfigurationJson`) for `secret:<name>` string values and resolves
|
||||
each name against the central `ISecretStore`, producing the ciphertext-only
|
||||
`SecretArtifact` rows the command carries (`DeployArtifactsCommand.Secrets`,
|
||||
proto field 9) plus the set of unresolved references. A site with no `secret:`
|
||||
references ships no secrets and is unaffected by any of this.
|
||||
- **Fail closed, per site.** A site with one or more unresolved (absent or
|
||||
tombstoned) references fails its deploy *before* `_communicationService` is
|
||||
called — nothing is sent, the site keeps its previous stamp, and the error
|
||||
names the secret and the referencing data connection(s): `Secret 'X'
|
||||
referenced by data connection(s) A, B is not in the central secret store`
|
||||
(semicolon-joined for more than one). Other sites in the same sweep are
|
||||
unaffected.
|
||||
- **Stamp gating.** A site is stamped `ArtifactRevisionHash`/`ArtifactsDeployedAt`
|
||||
only when every shipped secret actually applied. Two failure shapes are
|
||||
distinguished from an ordinary site failure and never stamped: an **old
|
||||
site** (wire `Success = true`, secrets were shipped, but the response has no
|
||||
`SecretsApplied` at all — a pre-feature site build) fails with `Site build
|
||||
predates secret deployment — upgrade the site before deploying secrets`; a
|
||||
**partial apply** (`SecretsApplied` less than the shipped count) fails with
|
||||
`Site applied {n} of {m} secrets`. Both apply identically on the all-sites
|
||||
sweep (`DeployCoreAsync`) and the per-site retry (`RetryForSiteAsync`).
|
||||
- **Staleness hash.** `ArtifactRevisionHashService` folds secrets into the same
|
||||
hash as every other artifact class (`HashableSecret`: a SHA-256 digest of the
|
||||
ciphertext/nonce/tag/wrapped-DEK material, never plaintext, plus `KekId` and
|
||||
`Revision`) so a KEK rewrap correctly flips a site to out of date. A
|
||||
referenced-but-not-yet-created secret hashes as an explicit `Missing` marker
|
||||
(`DeployArtifactsCommand.MissingSecrets`, central-only — never on the wire),
|
||||
so that site reads *Out of date*, never *Current*, until the secret exists.
|
||||
**A `null` secrets list and an empty one hash identically** (the existing
|
||||
null-when-empty convention for every artifact class), so a site with no
|
||||
secret references keeps its pre-existing hash across the upgrade that added
|
||||
this feature.
|
||||
- **Audit.** One best-effort `DeploySiteSecrets` / `SiteSecrets` audit row is
|
||||
written per site deploy that carried secrets or a missing-secret failure
|
||||
(never for a site with no secret references at all) — after-state is `{
|
||||
deploymentId, secrets: [{name, revision}], missing: [name], success }`, which
|
||||
structurally excludes ciphertext. Written on both `DeployCoreAsync` and
|
||||
`RetryForSiteAsync`, including the fail-closed path (audited even though
|
||||
nothing was sent).
|
||||
- **Central store.** Central resolves secrets from `ISecretStore`, composed by
|
||||
the Host per `Secrets:Store:Provider` — see Component-Host.md → *Secrets*.
|
||||
- **Site apply and the per-node mirror** — see Component-SiteRuntime.md →
|
||||
*Deployed Secrets*.
|
||||
|
||||
## Site-Side Apply Atomicity
|
||||
|
||||
Applying a deployment at the site is **all-or-nothing per instance**:
|
||||
@@ -420,6 +476,7 @@ The Deployment Manager sends the following commands to sites via the Communicati
|
||||
- **Configuration Database (MS SQL)**: Stores deployment status and deployed configuration snapshots.
|
||||
- **Security & Auth**: Enforces Deployment role (with optional site scoping).
|
||||
- **Configuration Database (via IAuditService)**: Logs all deployment actions, system-wide artifact deployments, and instance lifecycle changes.
|
||||
- **`ZB.MOM.WW.Secrets.Abstractions`** (`ISecretStore`, `SecretName`): `SiteSecretsCollector` resolves a site's referenced secrets against the Host-composed central store — see Component-Host.md → *Secrets*.
|
||||
|
||||
## Interactions
|
||||
|
||||
|
||||
@@ -249,6 +249,58 @@ reporting that it is a standby. The verdict remains fully observable through the
|
||||
|
||||
---
|
||||
|
||||
## Secrets (2026-09-28)
|
||||
|
||||
The Host composes `ZB.MOM.WW.Secrets` (a `scadaproj` shared library, not a ScadaBridge component —
|
||||
hence it has no row in the Component Registration Matrix below) via `SecretsRegistration.AddScadaBridgeSecrets`,
|
||||
called on both roles from their respective composition-root `Configure` methods. Design:
|
||||
`docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md` (see its *Implementation notes (as
|
||||
built)* section for where the shipped code deviates from the original design).
|
||||
|
||||
- **Site role — always a local, undecorated store.** A site node's `ISecretStore`/`ISecretResolver`
|
||||
is unconditionally the plain node-local SQLite store (`AddZbSecrets`), regardless of any
|
||||
`Secrets:Store:Provider` value — a Site never reads that key. This is a hard invariant, not
|
||||
just a default: `DeployedSecretsMirror` (Component-SiteRuntime.md) calls `ISecretStore.DeleteAsync`
|
||||
directly on the assumption that a delete is node-local; a replication-decorated store would have
|
||||
published that delete fleet-wide. Secrets reach a site only via Deploy Artifacts (see
|
||||
Component-DeploymentManager.md → *Site secrets*) — there is no other channel.
|
||||
- **Central role — `Secrets:Store:Provider`** (`Sqlite` default | `SqlServer`), read **only** on the
|
||||
Central role:
|
||||
- `Sqlite` (default): each central node keeps its own per-node SQLite secret store. On a
|
||||
**clustered** central (≥ 2 seed nodes) this is a documented misconfiguration — `StartupValidator`
|
||||
logs a boot-time Warning (same shape as the inbound-API-key-store precedent, register row 108),
|
||||
because the two nodes would deploy different secret sets to sites depending on which is active.
|
||||
- `SqlServer`: both central nodes read and write **one shared SQL Server store**
|
||||
(`Secrets:SqlServer:ConnectionString`, a literal/env value — never a `${secret:}` reference,
|
||||
the same bootstrap-circularity rule the inbound-API-key shared store follows) — the `/admin/secrets`
|
||||
page edits it, and it is the store `SiteSecretsCollector` resolves from. The pre-host `${secret:}`
|
||||
expander (the Layer-A pass that resolves `${secret:X}` tokens inside `appsettings` itself, before
|
||||
the DI container is built) reads the **same** shared store when `Provider = SqlServer`, so a
|
||||
central node's boot-time config tokens and its running `ISecretResolver` never disagree about
|
||||
what a token resolves to.
|
||||
- A connection string that is blank, or is itself a `${secret:}` reference (a bootstrap circularity),
|
||||
fails fast naming the offending key.
|
||||
- **Bootstrap secrets are never deployed.** The site's gRPC PSK (`SB-GRPC-PSK-{siteId}`), the LocalDb
|
||||
replication key, and the secrets master key (KEK) are never data-connection-referenced and are
|
||||
never shipped through Deploy Artifacts — a site needs them *before* it can receive a deploy at all,
|
||||
and pushing a rotated one through the channel it authenticates would sever that channel. They stay
|
||||
out-of-band: service environment variables, a hand-seeded store, or a master-key file, exactly as
|
||||
before this feature. Every node — every central and every site — **must share the same KEK**;
|
||||
see `docs/deployment/topology-guide.md` → *Secrets: shared store, per-site deploy, and the KEK*
|
||||
for where that is (and is not) checked.
|
||||
- **The gRPC secrets hub is retired in ScadaBridge (2026-09-28).** ScadaBridge no longer composes
|
||||
`ZB.MOM.WW.Secrets.Replicator.Grpc` on either role, and the historical SQL-Server *replication*
|
||||
mode (`Secrets:Replication:Mode=SqlServer` — distinct from `Secrets:Store:Provider=SqlServer`
|
||||
above, which shares one store rather than replicating between two) is retired too. Setting
|
||||
`Secrets:Replication:Enabled` to anything other than absent/empty/`false` now fails
|
||||
`StartupValidator.Validate` outright, on **either** role, with a message pointing at Deploy
|
||||
Artifacts (sites) and `Secrets:Store:Provider=SqlServer` (a clustered central). A leftover
|
||||
`Secrets:Replication:Mode` or any `Secrets:GrpcHub:*` key (harmless once `Enabled` is unset, but
|
||||
dead configuration) logs a startup Warning naming both sections so they can be cleaned up. The
|
||||
library itself keeps the hub for other consumers; only ScadaBridge stops using it.
|
||||
|
||||
---
|
||||
|
||||
## Component Registration Matrix
|
||||
|
||||
| Component | Central | Site | DI (`AddXxx`) | Actors (`AddXxxActors`) | Endpoints (`MapXxx`) |
|
||||
|
||||
@@ -162,6 +162,12 @@ flowchart TD
|
||||
- Recompiles shared scripts and makes updated code available to all Script Actors.
|
||||
- **Reconciles deletions**: each non-null artifact list is the complete set for its class, so after upserting, any stored row absent from the set is deleted (the artifact was deleted centrally — there is no per-artifact delete command). A removed shared script is also unregistered from the compiled script library, and a removed data connection's live DCL connection actor is stopped. A `null` list means "field not shipped" and reconciles nothing.
|
||||
|
||||
### Deployed Secrets (2026-09-28)
|
||||
- `HandleDeployArtifacts` additionally accepts the command's `Secrets` list (`SecretArtifact`, ciphertext only) and writes the full shipped set to the `deployed_secrets` table via `SiteStorageService.UpsertDeployedSecretsAsync` — upsert every shipped name, soft-remove (`is_removed = 1`) any stored name absent from the set. A `null` `Secrets` field (pre-feature central) touches nothing and answers with no `SecretsApplied`. Storage detail: *Deployed Secrets (Site SQLite / LocalDb)* below.
|
||||
- After writing the table, the active node runs `DeployedSecretsMirror.ReconcileAsync` **synchronously**, before replying — so the deploy response's `SecretsApplied` count and any apply/verify error reflect this deploy's own outcome, not a value left over from the mirror's background schedule. `SecretsApplied` counts only the names *this deploy shipped* that applied and verified cleanly; a mirror failure for an unrelated, already-broken name (a different deploy's leftover, or a stale removal) is logged at Warning and excluded — it neither fails this deploy nor reduces its `SecretsApplied`.
|
||||
- No `DeployedSecretsMirror` registered (secrets deployment not composed on this node) with a non-empty `Secrets` list fails the deploy with `Secret deployment is not enabled on this site node`; with an empty `Secrets` list it still succeeds (`SecretsApplied = 0`).
|
||||
- See *Deployed Secrets (Site SQLite / LocalDb)* below for the mirror itself.
|
||||
|
||||
### Instance Lifecycle Commands
|
||||
- **Disable**: Stops the Instance Actor and its children. Retains the deployed configuration in SQLite so the instance can be re-enabled without redeployment.
|
||||
- **Enable**: Creates a new Instance Actor from the stored configuration (same as startup).
|
||||
@@ -424,6 +430,26 @@ The Instance Actor owns native-alarm setup alongside its computed Script and Ala
|
||||
|
||||
---
|
||||
|
||||
## Deployed Secrets (Site SQLite / LocalDb)
|
||||
|
||||
Site secrets ride the artifact deploy (design `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md`; see that doc's *Implementation notes (as built)* for deviations from the original design). Two tables, two different replication postures:
|
||||
|
||||
- **`deployed_secrets`** — the **11th LocalDb-replicated table** (`SiteLocalDbSetup.ReplicatedTables`, up from ten). Columns: `name` (PK), `row_json` (the shipped `SecretArtifact` — ciphertext, nonce, tag, wrapped DEK, `KekId`, revision; never plaintext), `revision`, `updated_utc`, `is_removed` (soft-remove flag, default 0), `deployment_id`, `applied_at`. No FKs (the standing FK guard — `SiteLocalDbReplicatedTableFkGuardTests` — applies to it like every other replicated table). `UpsertDeployedSecretsAsync` is full-set upsert-then-soft-remove, never a hard `DELETE`, so a removal is itself a replicated row change the standby node can react to.
|
||||
- **`mirrored_secrets`** — a **node-local, unreplicated** ledger (`name` PK, `revision`, `mirrored_at`) recording which names *this specific node* has applied into its own `ISecretStore`. Deliberately not registered with LocalDb: it is a per-node fact (which names this node mirrored), not site-wide state, and replicating it would make one node's mirroring progress masquerade as the other's.
|
||||
|
||||
**`DeployedSecretsMirror`** (`SiteRuntime/Secrets/DeployedSecretsMirror.cs`) is a per-node `BackgroundService` — registered on **every** site node, not a singleton — that reconciles `deployed_secrets` into the node's own local `ISecretStore` (site nodes never run a replication-decorated store; see Component-Host.md → *Secrets*):
|
||||
|
||||
- **Schedule**: runs once immediately at start, then every `SiteRuntime:DeployedSecretsMirrorInterval` (default 15s, validated `> 0`), and synchronously on demand right after an active-node artifact apply (see *Deployed Secrets* under Deployment Manager Singleton above). This 15s interval is the **standby node's own convergence path** — it never receives the active node's synchronous call, only the replicated `deployed_secrets` rows via LocalDb, so it picks them up on its own next tick.
|
||||
- **Apply is confirm-then-ledger, not blind.** `ApplyReplicatedAsync` is strict last-writer-wins on `(UpdatedUtc, Revision)` (a tie is a no-op) and `DeleteAsync` tombstones rather than hard-deletes — so the mirror always reads the row back after applying and only ledgers it in `mirrored_secrets` when the local ciphertext matches what was shipped. A tombstoned or clock-skewed local row that the ledger already owns is safely re-stamped (`UpdatedUtc`/`Revision` bumped past the local value) and re-applied; a live row with **different** ciphertext that the ledger does **not** already own is left completely untouched — not overwritten, not ledgered, not verified — and reported as an out-of-band conflict (`"exists on this node with a newer local value; the deployed revision was not applied"`), so an operator's genuinely local secret of the same name is never silently destroyed by a later central deploy or removal.
|
||||
- **Ownership, not tombstone-state, decides re-stamping.** Once a name is ledgered in `mirrored_secrets` at any revision, this mirror treats that name as its own going forward — including re-stamping a mirror-owned name that drifted (e.g. an operator edited it locally) back to the deployed value on the next pass. Central is authoritative for every name it deploys.
|
||||
- **Removal** deletes from the local store **only if the name was previously ledgered** (present in `mirrored_secrets`) — an out-of-band local secret sharing a deployed name that was never mirrored is never touched by a central-side removal.
|
||||
- **Verify.** Every pass trial-resolves every live row through `ISecretResolver` (even one not re-applied this pass, so a later KEK change is still caught) — a decrypt failure names the secret and its `KekId` and is reported as a distinct error (de-spammed: Error the first time a given `(name, revision, KekId)` fails, Debug on identical repeats, Error again after an intervening success).
|
||||
- **Isolation.** One row failing to apply/verify does not block any other row in the same pass.
|
||||
- **Cache invalidation.** A successful apply or delete invalidates the shared `ISecretResolver` cache before verifying, so the verify step (and any script/connection reading the secret) sees the fresh value immediately rather than waiting out the resolver's cache TTL.
|
||||
- **Never logs a secret value** — log lines and the reported `MirrorResult` carry only names, revisions, and `KekId`s.
|
||||
|
||||
---
|
||||
|
||||
## Enriched `AlarmStateChanged` Message
|
||||
|
||||
The `AlarmStateChanged` message published by both Alarm Actors and Native Alarm Actors was extended **additively** (existing consumers keep working with computed defaults):
|
||||
@@ -642,10 +668,11 @@ Per Akka.NET best practices, internal actor communication uses **Tell** (fire-an
|
||||
- **Site Event Logging**: Records script executions, alarm events, deployment events, instance lifecycle events.
|
||||
- **Health Monitoring**: Reports script error rates and alarm evaluation error rates.
|
||||
- **Local SQLite**: Persists deployed configurations, system-wide artifacts (external system definitions, database connection definitions, data connection definitions). Sites also maintain peer SQLite stores for the Store-and-Forward buffer, the site event log, the operation tracking table, and the site-local `AuditLog` (see [Component-AuditLog.md](Component-AuditLog.md)). The `AuditLog` file is purged on the same daily cadence as the others but respects the hard `ForwardState` invariant — rows still `Pending` forward are never purged, regardless of age.
|
||||
- **`ZB.MOM.WW.Secrets.Abstractions`** (`ISecretStore`, `ISecretResolver`, `ISecretCacheInvalidator`): `DeployedSecretsMirror` applies deployed secrets into, and verifies against, the node's local secret store (composed by the Host — see Component-Host.md → *Secrets*). The Site Runtime depends only on the abstractions package, never the concrete store implementation.
|
||||
|
||||
## Interactions
|
||||
|
||||
- **Deployment Manager (central)**: Receives flattened configurations, system-wide artifact updates, and instance lifecycle commands.
|
||||
- **Deployment Manager (central)**: Receives flattened configurations, system-wide artifact updates (including a site's referenced secrets, ciphertext only), and instance lifecycle commands.
|
||||
- **Data Connection Layer**: Bidirectional — receives value updates, sends write-back commands.
|
||||
- **Communication Layer**: Receives commands from central, sends deployment results, serves debug view data.
|
||||
- **Store-and-Forward Engine**: Scripts route cached writes, notifications, and external system calls here.
|
||||
|
||||
Reference in New Issue
Block a user