docs: site secrets ride the artifact deploy; gRPC hub retired in ScadaBridge

2026-09-28 20:22:45 -04:00
parent bc29dc412b
commit 0ccf68951d
8 changed files with 259 additions and 6 deletions
@@ -90,7 +90,7 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
 - DCL write failures returned synchronously to calling script.
 - Tag path resolution retried periodically for devices still booting.
 - Static attribute writes persisted to local SQLite (survive restart/failover, reset on redeployment).
- **Consolidated site database (LocalDb Phase 1 + 2, complete 2026-07-20).** Ten tables now live in ONE `ZB.MOM.WW.LocalDb`-managed SQLite file (the `SiteLocalDbSetup.ReplicatedTables` list), configured by the **required** `LocalDb:Path` (`/app/data/site-localdb.db` on the rig; validated with `ValidateOnStart`, so a site config missing it fails to boot). Consequences worth knowing:
+- **Consolidated site database (LocalDb Phase 1 + 2, complete 2026-07-20).** Eleven tables now live in ONE `ZB.MOM.WW.LocalDb`-managed SQLite file (the `SiteLocalDbSetup.ReplicatedTables` list — the 11th, `deployed_secrets`, was added 2026-09-28 by the site-secrets-via-artifact-deploy change, see *Templates & Deployment* below), configured by the **required** `LocalDb:Path` (`/app/data/site-localdb.db` on the rig; validated with `ValidateOnStart`, so a site config missing it fails to boot). Consequences worth knowing:
  - `site_events.id` changed from autoincrement INTEGER to an application-minted **GUID**. Last-writer-wins keys on the primary key, so two nodes independently minting `id=1,2,3…` would destroy each other's events rather than merge them. The event-log read path uses a composite `(timestamp, id)` keyset cursor with an **opaque string** continuation token; `EventLogEntry.Id` and both `ContinuationToken`s are `string`/`string?` on the site↔central Akka DTOs.
  - `ScadaBridge:OperationTracking:ConnectionString`, `ScadaBridge:SiteEventLog:DatabasePath` and — as of Phase 2 — `ScadaBridge:StoreAndForward:SqliteDbPath` + `ScadaBridge:Database:SiteDbPath` are all **migration-only** — nothing reads them but `SiteLocalDbLegacyMigrator`, which copies a pre-Phase-1 file in once (deterministic `mig-{NodeName}-{legacyId}` event ids, `INSERT OR IGNORE`, runs AFTER `RegisterReplicated` so migrated rows replicate) and renames it `.migrated`. Delete the keys once a node has migrated.
  - This incidentally fixes a data-loss bug: both legacy databases defaulted to CWD-relative paths **outside** the mounted volume and were discarded on every container recreate.
@@ -101,10 +101,10 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
    - **`notification_lists` and `smtp_configurations` are created but deliberately NOT registered.** They are permanently empty on a site (no writer since 2026-07-10, the migrator skips them, the active-node purge keeps them empty), and registering them would open a standing replication channel whose only historical payload was plaintext SMTP passwords. Pinned by a security-named test, and verified live: those two tables have **no CDC triggers** on either rig node.
    - **Operational constraints (read before upgrading a site pair):** stop and start both nodes TOGETHER — rolling one at a time is no longer supported, since the legacy `SfBufferSnapshot` compatibility handler went with the replicator. And a node offline longer than `LocalDb:Replication:TombstoneRetention` (default 7 days) can resurrect deleted rows on rejoin. See `docs/deployment/topology-guide.md`.
    - **Batching is by BYTE BUDGET as of LocalDb 0.2.0** — `LocalDb:Replication:MaxBatchBytes` (default **2 MB**, sized under the 4 MB gRPC cap) bounds a delta/snapshot message by summed serialized size via a per-message split in `SyncSession.PumpLoopAsync`, with `MaxBatchSize` demoted to a secondary row cap; a single row over budget is sent alone rather than stalling the stream. **The rig's old `MaxBatchSize = 16` pin (a hand-computed byte-budget proxy: ~70 KB worst-case `config_json` x the 500 default is ~35 MB) is retired, but `MaxBatchSize` is NOT fully redundant with `MaxBatchBytes`** — it also bounds the separate DB READ page in `OplogStore.ReadBatchAboveAsync`/`SnapshotStreamer`, which materializes the whole page into memory *before* the byte-budget split runs, so an unset (500-default) `MaxBatchSize` still lets a reconnect drain transiently allocate ~35 MB per read even though every wire message stays under `MaxBatchBytes` (arch-review adversarial finding F2). Both site-a nodes on `docker/` therefore pin an explicit `"MaxBatchSize": 64` to bound that transient allocation, while `MaxBatchBytes` stays unset (its 2 MB default) to bound the wire message; site-b/site-c stay unreplicated so the key doesn't apply there.
-    - **CDC registration is conditional, and both directions self-heal at boot** (arch-review WP1.3 + WP3.3, `Host/SiteLocalDbSetup.cs`). Capture triggers are installed only when this node has replication configured — `PeerAddress` **or** `ApiKey`, an OR because only the dialling half sets `PeerAddress` while the passive half carries the key alone. An unreplicated node calls **`DeregisterReplicated`** on all ten tables at boot, dropping triggers an earlier build left behind and pruning their oplog/row-version rows (idempotent; logs once at Information when something was actually cleaned). A replicating node registers with **`baselineExistingRows: true`**, which seeds `__localdb_row_version` for pre-existing rows at the LWW floor (HLC `0`, this node's id) and flags a snapshot resync — so turning replication ON for a site that has been running without it now converges on the rows already in the file instead of only on writes made after the restart. Deregistration must be symmetric (the handshake compares registered-table digests fail-closed), so replication is a both-nodes-together change in either direction. LocalDb 0.2.0 also makes backlog depth O(1) and drops the unused `__localdb_oplog_hlc` index; the on-disk bookkeeping schema goes to v2, upgraded in place on open, with no wire change (0.1.x peers still sync).
+    - **CDC registration is conditional, and both directions self-heal at boot** (arch-review WP1.3 + WP3.3, `Host/SiteLocalDbSetup.cs`). Capture triggers are installed only when this node has replication configured — `PeerAddress` **or** `ApiKey`, an OR because only the dialling half sets `PeerAddress` while the passive half carries the key alone. An unreplicated node calls **`DeregisterReplicated`** on all eleven tables at boot, dropping triggers an earlier build left behind and pruning their oplog/row-version rows (idempotent; logs once at Information when something was actually cleaned). A replicating node registers with **`baselineExistingRows: true`**, which seeds `__localdb_row_version` for pre-existing rows at the LWW floor (HLC `0`, this node's id) and flags a snapshot resync — so turning replication ON for a site that has been running without it now converges on the rows already in the file instead of only on writes made after the restart. Deregistration must be symmetric (the handshake compares registered-table digests fail-closed), so replication is a both-nodes-together change in either direction. LocalDb 0.2.0 also makes backlog depth O(1) and drops the unused `__localdb_oplog_hlc` index; the on-disk bookkeeping schema goes to v2, upgraded in place on open, with no wire change (0.1.x peers still sync).
    - **LocalDb 0.3.0 (2026-08-15) makes that snapshot flag PER-TABLE** — the residual 0.2.1 explicitly deferred. On-disk bookkeeping schema goes to **v3** (a new `__localdb_snapshot_state` table, upgraded in place on open); still wire-compatible, and `lib_schema_version` deliberately stays `1` because the handshake compares it fail-closed for *equality*. The extension is negotiated by **capability** (`Handshake.supports_partial_snapshot`, `SnapshotBegin.tables` where **empty == all tables**), so a peer that does not advertise it is served a full snapshot — a scoped one would make it advance `last_applied_remote_seq` past deltas it never received. For the same reason **a partial snapshot moves no watermark on either side**; the overlapping deltas are still pumped and absorbed by LWW. Snapshot debt now clears on the receiver's new `SnapshotAck` rather than on enqueue, so a crash mid-snapshot re-streams the same scope. **`SiteLocalDbSetup.cs` needed no restructuring** — it already calls `RegisterReplicated` once per table in a loop, which is exactly what per-table flagging keys off. The consequence to know: **adding an eleventh table to `ReplicatedTables` on an already-replicating site now snapshots that one table**, where through 0.2.x the same edit re-streamed all eleven in full in both directions. First-boot behaviour is unchanged (all ten seed at once ⇒ the flagged set is every registered table ⇒ an ordinary full snapshot), and so is upgrading the rig in place (the ten tables are already ledgered, nothing seeds, nothing is flagged). A future "register the whole list in one call" API would silently give the coarse behaviour back.
    - **LocalDb 0.3.2 (2026-08-19, ScadaBridge arch-review P4-1 + P4-3; wire-compatible, no schema change):** (1) the sync pump now **faults the live session when a cap-prune invalidates its position** (horizon check on the oplog's min seq before each read batch + on the batch head after it; throws the new public `LocalDbOplogGapException` through the 0.3.1 fault-teardown) — previously a slow-but-alive peer silently skipped the pruned gap and the pair diverged until the next reconnect; (2) **transport keepalives both sides** — client `SocketsHttpHandler` pings via new validated `LocalDb:Replication:KeepAlivePingDelay`/`KeepAlivePingTimeout` (defaults 60s/20s, `Always` policy), and ScadaBridge's site gRPC listener sets Kestrel h2 keepalive (`Host/SiteKestrelSetup.cs` — server-wide, also governs the metrics port); (3) the server's single-stream guard is now **newest-wins with an awaited handover** — a fresh authenticated `Sync` cancels a dead/wedged incumbent and *waits* for it to finish (`TakeoverTimeout` 10s) before starting, so a half-open link can no longer wedge the pair behind `AlreadyExists` while the zombie side reports `LocalDbReplicationConnected=true`.
-    - **LocalDb 0.3.3 (2026-08-20, ScadaBridge arch-review P4-7; wire-compatible, no schema change — superseded by 0.3.4 below in the same bullet):** two latent `LwwApplier` hazards closed before any table could hit them. (1) **`RegisterReplicated` now REFUSES a table that is the parent of a foreign key with a row-mutating `ON DELETE` action (`CASCADE`/`SET NULL`/`SET DEFAULT`) from any other table** (`LocalDbRegistrationException` naming `child.column ON DELETE <action>`; `RESTRICT`/`NO ACTION` still allowed) — the applier's `INSERT OR REPLACE` runs with `foreign_keys=ON`, so the implicit delete fired those actions while CDC capture was suppressed, i.e. child rows changed on one node and never replicated. Registration-time only: SQLite can't add an FK to an existing table, so a later-created child is caught at the next boot's registration. None of ScadaBridge's ten tables has an FK (verified on the live rig file: zero `REFERENCES` across 29 `CREATE TABLE`s) — **keep it that way; a cascading-FK parent can never be one of `ReplicatedTables`** (register row 105). (2) The per-entry apply (delete-or-upsert + row-version ledger write) is now **atomic under a `SAVEPOINT`** — a fault between the two statements used to leave a deleted row with a live ledger entry, which `SnapshotStreamer` warns-and-skips forever; poison-catch semantics unchanged (narrow `SqliteException`, per-entry dead-letter, batch continues). Cost: two extra in-memory statements per applied entry on a backlog drain. **Both are pinned here** (2026-08-20): `SiteLocalDbReplicatedTableFkGuardTests` asserts at build time that no `ReplicatedTables` entry has an inbound row-mutating FK (register row 105), and **pre-2026-08-20 site files must be VACUUMed once offline** to get `auto_vacuum=INCREMENTAL` — `docker/vacuum-site-localdb.sh` (pair stopped together; done on all three rig pairs; topology guide → *one-off offline VACUUM*; row 104). **LocalDb 0.3.4 (2026-08-20, register row 106; wire-compatible, sender-side only):** a converged pair whose sender oplog was ack-pruned to empty used to stamp snapshots `as_of_seq 0`, so a peer with a zero watermark (a rebuilt node — the rig's site-a-a since 2026-08-15) received a FULL snapshot on every reconnect until the sender wrote again (127 k rows per restart, observed). The as-of now floors at `last_acked_seq`; proven live — second restart after the pin exchanged zero snapshots. Lesson for rebuilding a node from its peer: the first reconnect snapshots, the second must not.
+    - **LocalDb 0.3.3 (2026-08-20, ScadaBridge arch-review P4-7; wire-compatible, no schema change — superseded by 0.3.4 below in the same bullet):** two latent `LwwApplier` hazards closed before any table could hit them. (1) **`RegisterReplicated` now REFUSES a table that is the parent of a foreign key with a row-mutating `ON DELETE` action (`CASCADE`/`SET NULL`/`SET DEFAULT`) from any other table** (`LocalDbRegistrationException` naming `child.column ON DELETE <action>`; `RESTRICT`/`NO ACTION` still allowed) — the applier's `INSERT OR REPLACE` runs with `foreign_keys=ON`, so the implicit delete fired those actions while CDC capture was suppressed, i.e. child rows changed on one node and never replicated. Registration-time only: SQLite can't add an FK to an existing table, so a later-created child is caught at the next boot's registration. None of ScadaBridge's eleven tables has an FK (verified on the live rig file: zero `REFERENCES` across 29 `CREATE TABLE`s) — **keep it that way; a cascading-FK parent can never be one of `ReplicatedTables`** (register row 105). (2) The per-entry apply (delete-or-upsert + row-version ledger write) is now **atomic under a `SAVEPOINT`** — a fault between the two statements used to leave a deleted row with a live ledger entry, which `SnapshotStreamer` warns-and-skips forever; poison-catch semantics unchanged (narrow `SqliteException`, per-entry dead-letter, batch continues). Cost: two extra in-memory statements per applied entry on a backlog drain. **Both are pinned here** (2026-08-20): `SiteLocalDbReplicatedTableFkGuardTests` asserts at build time that no `ReplicatedTables` entry has an inbound row-mutating FK (register row 105), and **pre-2026-08-20 site files must be VACUUMed once offline** to get `auto_vacuum=INCREMENTAL` — `docker/vacuum-site-localdb.sh` (pair stopped together; done on all three rig pairs; topology guide → *one-off offline VACUUM*; row 104). **LocalDb 0.3.4 (2026-08-20, register row 106; wire-compatible, sender-side only):** a converged pair whose sender oplog was ack-pruned to empty used to stamp snapshots `as_of_seq 0`, so a peer with a zero watermark (a rebuilt node — the rig's site-a-a since 2026-08-15) received a FULL snapshot on every reconnect until the sender wrote again (127 k rows per restart, observed). The as-of now floors at `last_acked_seq`; proven live — second restart after the pin exchanged zero snapshots. Lesson for rebuilding a node from its peer: the first reconnect snapshots, the second must not.
    - **LocalDb 0.2.1** (two adversarial-review findings against 0.2.0, wire-compatible, no schema change) fixes: (1) `DeregisterReplicated` was deleting the HLC clock's only durable crash anchor along with the table rows — an ungraceful exit (crash, SIGKILL, power loss) on a fully-deregistered file left nothing to recover from, so the reopened clock could restart from wall-clock time and re-issue stamps below HLCs peers already hold (silently discarded by LWW); the current clock is now flushed inside the deregistration transaction, before the prune. (2) The sync inbox was an **unbounded** channel, so inbound memory was a function of what the peer sends — a mutual full snapshot (both sides of a pair baseline unconditionally when replication is enabled late) buffered whole, in memory, on both nodes at once; it is now **bounded at 64 messages** with wait-mode backpressure (unrelated to, and not to be confused with, the rig's separate `MaxBatchSize: 64` DB-read-page pin above), which required moving the snapshot to run alongside the receive loop (not inside the pre-loop handshake) so the inbox is drained for the whole session and can't deadlock itself.
 - All timestamps are UTC throughout the system.
 - Inter-cluster communication uses **three** transports, not two — **all cross-cluster command/control and data now rides gRPC** after the ClusterClient→gRPC migration's Phase 4 (`docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`) deleted Akka `ClusterClient`/`ClusterClientReceptionist`: (1) **gRPC command/control** — site→central over the central-hosted `CentralControlService` (`GrpcCentralTransport`, sticky central-a→central-b channel pair; deployments/notifications/health/heartbeat/audit-ingest/reconcile), and central→site over the site-hosted `SiteCommandService` (`GrpcSiteTransport`, per-site NodeA→NodeB channel pair; the 28 lifecycle/OPC-UA/query/parked/route/failover commands); (2) **gRPC** server-streaming for real-time data (attribute values, alarm states, `SiteStreamService`); and (3) **plain token-gated HTTP** for the deployment config itself — notify-and-fetch, the site pulls the config from `DeploymentConfigEndpoints` (`ManagementService/DeploymentConfigEndpoints.cs`) with an `X-Deployment-Token` header, `AllowAnonymous` with the per-deployment token as the entire security boundary. The gRPC boundary is per-site PSK-authenticated (`ControlPlaneAuthInterceptor`, unchanged). There is **no receptionist registration** — discovery is by dialling configured endpoints; central builds one `SitePairChannelProvider` per site (addresses from `Site.GrpcNodeAAddress`/`GrpcNodeBAddress`, refreshed from the DB every 60s and on admin changes), sites dial `ScadaBridge:Communication:CentralGrpcEndpoints` (both central nodes, h2c on `CentralGrpcPort` 8083, **NOT** via Traefik). **Discovery is asymmetric by design:** central discovers site gRPC addresses from the *database* (refreshable at runtime), sites discover central from *appsettings* (`CentralGrpcEndpoints`, static — restart required; `StartupValidator` requires a Site node to list at least one). `Akka.Cluster.Tools` stays for ClusterSingleton; only the ClusterClient part is gone. **Central never buffers for an unreachable site** — the send fails with the caller's Ask/deadline timing out; a `ConnectionStateChanged` mechanism built for this was deleted as dead code.
@@ -141,6 +141,7 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
 - Per-instance operation lock covers all mutating commands (deploy, disable, enable, delete).
 - **Bulk site deployment (`DeploymentService.DeploySiteAsync`, arch-review WP2.5)** redeploys every out-of-date instance at a site through one flatten session (a template chain shared by N instances is walked once) in three phases — serial Prepare, bounded-parallel Send (`SiteDeploymentMaxParallelism`, default 4), serial Finalize — still **not all-or-nothing**: any one instance's failure doesn't block the rest. **Disabled instances are SKIPPED by bulk deploy as of 2026-08-19 (arch-review P7-2)** — they are reported in the summary's additive `SkippedInstances` list, never flattened/sent/re-enabled (single-instance deploy remains the explicit operator path that deploys+enables a Disabled instance); **Up-to-date instances are ALSO skipped as of 2026-08-20 (arch-review P7-6)** — a redeploy is not idempotent at the site (it resets static overrides and clears native alarm state), so `DeploySiteAsync` pre-filters through `IStaleInstanceProbe` (trustworthy since P7-4 fixed its memo timing) and reports unchanged instances in `SkippedInstances` with reason `up-to-date`; the filter is conservative (every uncertainty deploys: missing snapshot, `ShouldQuerySiteBeforeRedeploy`, null/faulting probe). Additive `MgmtDeploySiteCommand.Force` / CLI `deploy site --force` restores the unconditional push for what central cannot see (restored volume, rebuilt node) — **Disabled still wins under `--force`**. Also P7-5: the three phases share one scoped DbContext and a failed `SaveChanges` leaves the entity tracked, so one permanent failure used to cascade to every later instance and strand records `InProgress`; `MarkDeploymentFailedAsync` now detaches pending changes and retries the `Failed` write once on the clean tracker; Prepare's per-instance reconcile also runs under a short `SiteDeploymentReconcileTimeout` (5s) with a once-per-batch unreachable-site circuit-breaker, so a down site no longer costs 30s × N serial before the first send (P7-3). The `PendingDeployment` staging row (5-minute TTL) is written per-instance immediately before that instance's round-trip in the Send phase, not upfront in Prepare, so a large batch's tail instances don't have their TTL clock already half-spent by the time they're reached.
 - **Site artifact revision + fleet-wide pending-updates indicator (2026-09-14, `docs/plans/2026-09-14-pending-updates-design.md`).** `Site.ArtifactRevisionHash`/`ArtifactsDeployedAt` (migration `AddSiteArtifactRevision`) are stamped by `ArtifactDeploymentService` only on a site whose artifact deploy succeeds, so a redeployed fleet's artifacts are trackable per site the same way instance staleness already is. `PendingUpdatesEvaluator` combines that with `DeploymentService.GetInstanceStalenessAsync` (one shared `FlattenSession` for the whole fleet) into a `PendingUpdatesSnapshot`, held behind a process-wide singleton `IPendingUpdatesCache` (`ScadaBridge:DeploymentManager:PendingUpdatesCacheTtl`, default 60s) so N Blazor circuits cost one fleet flatten per TTL, not one each. Surfaces: the Sites page's per-site **Artifacts** badge (`Current`/`Out of date`/`Never deployed`) + instance-staleness line + header **Re-check**, and a rail-footer **pending updates** indicator (Deployer/Administrator only, 30s poll, site-scoped). **Upgrade consequence:** every site reads *Never deployed* until one artifact deploy runs post-upgrade — deliberate, not a bug. Also shipped: the rail nav now sits in its own scroll pane (`NavMenu`'s `.rail-scroll` + `site.css`) so a long nav can't push the footer/indicator off-screen; the `ZB.MOM.WW.Theme` kit itself is untouched (register row 123 proposes upstreaming it).
 - **Site secrets ride the artifact deploy; the gRPC secrets hub is retired in ScadaBridge (2026-09-28, `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md`).** `Deploy Artifacts` now ships each site **only the secrets it references** — `SiteSecretsCollector` scans a site's data-connection JSON for `secret:<name>` (`SecretReferenceScanner`, protocol-agnostic, bootstrap names like `SB-GRPC-PSK-*` excluded) and resolves each against the central `ISecretStore`; a missing/tombstoned secret fails **that site's** deploy before anything is sent (fail-closed), ciphertext only ever crosses the wire, and secrets factor into `ArtifactRevisionHashService`'s staleness hash (a missing secret hashes as a distinguishable marker, so an unresolvable reference never reads *Current*; a site with no secret references hashes identically whether the field is null or empty). A pre-feature site (answers with no `SecretsApplied`) or a partial apply is also treated as a deploy failure and is never stamped; one `DeploySiteSecrets`/`SiteSecrets` audit row (name + revision only, never ciphertext) is written per site deploy that carried secrets, on both the sweep and the per-site retry paths. Site-side: `deployed_secrets` is the 11th LocalDb-replicated table (ciphertext rows only, upsert-then-soft-remove); the per-node `DeployedSecretsMirror` (`SiteRuntime:DeployedSecretsMirrorInterval`, default 15s, also run synchronously right after an active-node apply) reconciles it into that node's local `ISecretStore`, ledgers what it mirrored in the node-local `mirrored_secrets` table so an out-of-band local secret sharing a deployed name is never silently touched (a mirror-owned name that drifts locally is re-stamped back to the deployed value), and trial-decrypts every pass to catch a KEK mismatch. Central runs one shared secret store, `Secrets:Store:Provider=SqlServer` (Central role only; a clustered central left on `Sqlite` gets a boot Warning) — the pre-host `${secret:}` expander reads the same store the running node serves. **ALL secrets replication is retired in ScadaBridge** — the pull-only gRPC hub and the SQL-Server replication mode both — `Secrets:Replication:Enabled` set to anything but empty/`false` now fails `StartupValidator` on either role with a message pointing at Deploy Artifacts / `Secrets:Store:Provider=SqlServer`.
 - **Create Instance picks its template from the Templates folder tree** (2026-08-21): the flat `<select>` is gone. The Template field is a read-only folder-qualified path box (`TemplatePathFormatter`, "Folder / Sub / Name") + a **Choose…** button, both opening `TemplatePickerDialog` — an `IDialogService.ShowAsync` body wrapping the **same shared `TemplateFolderTree`** the Templates page renders, with its own search, expand/collapse-all, templates-only selection and double-click-to-pick. It opens on the current pick via the tree's additive **`RevealKeys`** parameter (ancestors of the named `f:`/`t:` keys start expanded when no `Filter` is active; an active filter's reveal set still wins; callers passing nothing are byte-identical). `InstanceCreate` gained `[SupplyParameterFromQuery] TemplateId`, which the Templates tree's new **Deployer-gated "New Instance…"** context-menu item navigates with; combined with `?siteId=` it populates the Connection Bindings card on first render. **The trap for anyone extending the picker:** in `TreeViewSelectionMode.Single` the inner `TreeView` reads the *singular* `SelectedKey`/`SelectedKeyChanged` pair, but `TemplateFolderTree` forwards only the plural (Checkbox-mode) `SelectedKeys` pair — so a Single-mode caller cannot get a row highlight through the shared tree and must own selection in its own `NodeContent`, as the picker does. Design: `docs/plans/2026-08-21-template-picker-design.md`.
 - **Create Instance binds module groups at creation** (2026-08-21): the create page's Connection Bindings card flattens template+site through the additive `IFlatteningPipeline.PreviewAsync` (an unsaved instance — no validation, no hash, nothing persisted, instance repo never touched) and reuses `BindingGroupBuilder`; `NodeBrowserDialog.SelectableNodeClasses` lets it pick **Object** nodes (default stays Variable-only for every existing caller). Create = `CreateInstanceAsync` → `SetConnectionBindingsAsync` (the existing upsert, no new server surface) → always navigate to Configure with `?created=1&bound=N|&bindError=…`, because the row exists whether or not the second call succeeded. Partial binding is legal: a group missing a connection or an Object is skipped and stays unbound. Design: `docs/plans/2026-08-21-create-instance-bindings-design.md`.
 - Site-side apply is all-or-nothing per instance.
@@ -174,7 +175,7 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
 - One row per lifecycle event; cached calls produce 4+ rows per operation (`Submitted`, `Forwarded`, `Attempted`, `Delivered`/`Parked`/`Discarded`).
 - `ExecutionId` (`uniqueidentifier NULL`) is the universal per-run correlation value — every audit row emitted by one script execution / inbound request shares it; `CorrelationId` remains the per-operation lifecycle id (NULL for sync one-shots).
 - `ParentExecutionId` (`uniqueidentifier NULL`) is the cross-execution spawn pointer — every row of a spawned run carries the spawner's `ExecutionId`; bridges inbound API → routed-site-script, alarm-triggered on-trigger scripts, and nested `CallScript`/`CallShared` invocations; `IX_AuditLog_ParentExecution` backs the filter + the recursive execution-tree walk. **Tag-cascade (alarm leg) is populated, not just plumbed** — M5.4 T4 threaded the `parentExecutionId` parameter but every `AlarmActor.SpawnAlarmExecution` call site passed null, so alarm runs were silently always roots; the id now rides site-locally as `SetStaticAttributeCommand.SourceExecutionId` → `AttributeValueChanged.SourceExecutionId` → `SpawnAlarmExecution` (all additive/nullable, no wire/proto/schema change; `Expression` triggers capture the writer *with* the evaluated snapshot since the eval completes off-dispatcher). Sources are `ScriptRuntimeContext.SetAttribute` (the run's own `ExecutionId`) and `Route.To(...).SetAttributes(...)` (the inbound request's). **Still roots by design, not omission:** alarms fired by DCL data (external values have no spawning execution — including the device echo of a script write to a *data-sourced* attribute, so only **static** writes cascade), and `ScriptActor` value-change/conditional/expression/timer trigger runs (a timer tick has no spawner; a `WhileTrue`/interval run has no single identifiable write).
- Site SQLite hot-path first, then gRPC telemetry to central; ingest is idempotent on `EventId`; periodic reconciliation pull as fallback when telemetry is lost. **The site audit DB is a separate, unreplicated SQLite file — NOT one of LocalDb's ten consolidated tables** — at the **required** `AuditLog:SiteWriter:DatabasePath` (arch-review WP1.2, finding #2 High), `/app/data/auditlog.db` on the rig; `StartupValidator` refuses to boot a Site node without it. `DatabasePath` used to default to CWD-relative `auditlog.db`, which on the docker rig landed on the container's ephemeral overlay, not the mounted volume — the same data-loss pattern the pre-LocalDb legacy S&F/tracking databases hit (see the LocalDb bullet above), independently rediscovered here because this file sits outside LocalDb's management. The writer loop now also honors `FlushIntervalMs` (was validated but never read — one fsync'ing commit per event even at trickle rate) and sets `PRAGMA synchronous = NORMAL` (audit is best-effort by design, so the narrower power-loss window is an acceptable trade for far fewer fsyncs; WAL still guarantees no corruption).
+- Site SQLite hot-path first, then gRPC telemetry to central; ingest is idempotent on `EventId`; periodic reconciliation pull as fallback when telemetry is lost. **The site audit DB is a separate, unreplicated SQLite file — NOT one of LocalDb's eleven consolidated tables** — at the **required** `AuditLog:SiteWriter:DatabasePath` (arch-review WP1.2, finding #2 High), `/app/data/auditlog.db` on the rig; `StartupValidator` refuses to boot a Site node without it. `DatabasePath` used to default to CWD-relative `auditlog.db`, which on the docker rig landed on the container's ephemeral overlay, not the mounted volume — the same data-loss pattern the pre-LocalDb legacy S&F/tracking databases hit (see the LocalDb bullet above), independently rediscovered here because this file sits outside LocalDb's management. The writer loop now also honors `FlushIntervalMs` (was validated but never read — one fsync'ing commit per event even at trickle rate) and sets `PRAGMA synchronous = NORMAL` (audit is best-effort by design, so the narrower power-loss window is an acceptable trade for far fewer fsyncs; WAL still guarantees no corruption).
 - Cached operations: site emits a single additively-extended `CachedCallTelemetry` packet carrying both audit events and operational state; central writes `AuditLog` + `SiteCalls` in one transaction.
 - Payload cap 8 KB by default / 64 KB on error rows; auth headers redacted by default; SQL parameter values captured by default; per-target redaction opt-in. Inbound API: full verbatim capture up to `InboundMaxBytes` (default 1 MiB); request headers stored in `Extra.requestHeaders` (post-redaction); per-method `SkipBodyCapture` flag suppresses bodies while still recording headers + metadata; `AuditInboundCeilingHits` counter surfaced on health snapshot. (M5.3 T7)
 - Audit-write failure NEVER aborts the user-facing action — audit is best-effort, the action's own success/failure path is authoritative.
@@ -120,6 +120,14 @@ Both central nodes share state through:
  because a failover then 401s every inbound client until keys are re-issued on the survivor. Both
  docker rigs run `SqlServer`; see `docs/operations/inbound-api-key-reissue.md` → *Central pairs* to
  migrate an existing pair.
 - **Secrets store**: shared **only** when `Secrets:Store:Provider = SqlServer` (design 2026-09-28,
  `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md`) — both nodes then read/write the
  same shared SQL Server secret store, so `/admin/secrets` create/rotate/delete changes are
  cluster-wide immediately and both nodes deploy the same secret set to sites regardless of which
  is active. The library default, `Provider = Sqlite`, stores secrets in a **per-node** file instead;
  running `Sqlite` on a central pair is a documented misconfiguration — `StartupValidator` logs a
  boot-time Warning. See *Secrets: shared store, per-site deploy, and the KEK* below for the KEK
  requirement this store does **not** by itself satisfy.
 ### Load Balancer
@@ -558,6 +566,39 @@ path. That is the accepted posture today — the same trusted-network assumption
 already made, now with authentication rather than none. TLS on these listeners is follow-on
 hardening and needs no change to the key design.
 #### Secrets: shared store, per-site deploy, and the KEK
 Site secrets ride the artifact deploy (2026-09-28, `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md`;
 see that doc's *Implementation notes (as built)* for where the shipped code deviates from the
 original design). This section is the operational summary; component detail lives in
 Component-Host.md → *Secrets*, Component-DeploymentManager.md → *Site secrets*, and
 Component-SiteRuntime.md → *Deployed Secrets*.
 - **Central**: `Secrets:Store:Provider = SqlServer` on both central nodes — see *Shared State*
  above. `SiteSecretsCollector` resolves each site's referenced secrets from that store at deploy
  time; a missing/tombstoned secret fails that site's deploy before anything is sent.
 - **Every node must share the same KEK** (`ZB_SECRETS_MASTER_KEY` / master-key file) — **every
  central node and every site node**, not just the central pair. This is the same requirement the
  retired gRPC secrets hub had; it did not change. **Nothing checks this at boot or at the next
  reconnect** — a mismatch surfaces only at **deploy time**, on the site: `DeployedSecretsMirror`'s
  verify step trial-decrypts every applied secret and reports a distinct error naming the secret
  and its `KekId` if the local master key cannot open it. Verify the KEK is identical everywhere
  **before** turning this on for a site — the rig's `docker/docker-compose.yml` deliberately runs
  site-b/site-c on a mismatched KEK precisely to exercise this failure path live.
 - **Standby / passive-node convergence.** The active site node applies a deploy's secrets
  synchronously (so the deploy response reflects that apply immediately); the standby node never
  receives that synchronous call — it converges through ordinary LocalDb replication of the
  `deployed_secrets` table plus its **own** `DeployedSecretsMirror` running on its own schedule,
  `ScadaBridge:SiteRuntime:DeployedSecretsMirrorInterval` (default **15s**). A freshly-promoted
  standby is therefore current on secrets within one interval of the last deploy, not instantly.
 - **Rollout is a deployment record, not a code change** — see the deferred-work register for the
  vd03/QA prerequisites (shared store provisioning, `Provider=SqlServer` on both centrals, the KEK
  alignment check above, recreating any hand-seeded site secrets centrally, then *Deploy Artifacts*).
 - **The gRPC secrets hub is retired in ScadaBridge** — `Secrets:Replication:Enabled` now fails
  startup on either role (Component-Host.md → *Secrets*). QA and vd03 currently run
  `Replication:Enabled=false`, so upgrading them to a build carrying this change will not itself
  fail fast.
 ## Scaling Guidelines
 ### Target Scale
@@ -120,6 +120,7 @@ All 7 fix-now items landed via PLAN-04/05/06/07/08 (verified in review 08 round
 | 124 | **No template-graph-watermark short-circuit on the pending-updates fleet flatten** *(Low)*. `PendingUpdatesEvaluator.EvaluateAsync` re-flattens every deployed instance through `DeploymentService.GetInstanceStalenessAsync` on every cache-TTL expiry (`PendingUpdatesCacheTtl`, default 60 s) while any Deployer/Admin circuit is open, even when nothing in the template/instance/artifact graph has changed since the last evaluation. | `docs/plans/2026-09-14-pending-updates-design.md` §3.2 ("Not in scope") | The single-flight TTL cache already bounds the cost to one fleet flatten per TTL fleet-wide (not per viewer); a watermark short-circuit is a further optimization, not a correctness need, and needs a cheap "has anything changed" signal that does not exist yet. | The fleet flatten's cost (instance count x template depth) becomes measurable against `PendingUpdatesCacheTtl` on a large deployment. |
 | 125 | **Pending-updates counts have no Health-dashboard tile or CLI surface** *(Low)*. The aggregate (`PendingUpdatesSnapshot.StaleInstanceCount` / `StaleArtifactSiteCount`) is consumed only by the Central UI Sites page and rail indicator; there is no Health dashboard KPI tile and no `deploy`/`site` CLI command surfacing it — `site list` shows only the raw `artifactRevisionHash`/`artifactsDeployedAt` fields on each site, not whether they are current. | `docs/plans/2026-09-14-pending-updates-design.md` §3.2 ("Not in scope") | Out of scope for the UI-focused ask; the CLI/Health surfaces are a separate consumer of the same `IPendingUpdatesCache`. | An operator wants the pending-updates counts without opening the Central UI (headless monitoring, CLI scripting). |
 | 126 | ~~**TODO — roll the two outstanding migrations to the QA pair vq03/vq04**~~ **DONE 2026-09-28** — QA vq03/vq04 upgraded `2b9a7c3c` → `b3021e7c`, 55 → 57, artifacts deployed (record `docs/deployment/deployment-records/2026-09-28-wonder-app-vq03-vq04-b3021e7c.md`). *(deployment, closed)*. **DEV vd03 DONE 2026-09-28** (`fe89363b`, 55 → 57, record `docs/deployment/deployment-records/2026-09-28-wonder-app-vd03-fe89363b.md` — the `sb-mig-55to57.sql` there is the same script QA needs). QA still runs `2b9a7c3c` with the DB at migration 55 (`20260909203046_EnforceGlobalInstanceNamesAndProtectProtocolConfiguration`); `main` (`ac833f5a`, pushed 2026-09-14) carries two more: **56** `20260911135501_AddAlarmMesReportable` (computed-alarm `IsMesReportable` flag) and **57** `20260914173147_AddSiteArtifactRevision` (`Sites.ArtifactRevisionHash`/`ArtifactsDeployedAt`, both nullable). Both are additive `AddColumn`s — no data upgrade, no coordinated step like the 2026-09-10 protocol-config encryption. Recipe (same as the 55 rollout): generate ONE from→to delta script `dotnet ef migrations script EnforceGlobalInstanceNamesAndProtectProtocolConfiguration AddSiteArtifactRevision --project src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase --startup-project src/ZB.MOM.WW.ScadaBridge.Host -o sb-mig-55to57.sql` (NOT idempotent, identical for vd03 and the QA pair), take the pre-upgrade DB backup, stop the pair TOGETHER, apply via `sqlcmd -C -b -I` on the SQL host, publish/swap the `ac833f5a` binaries, start the pair, confirm the Central boot log shows 0 pending migrations and no "Applying migration" (the script already applied them). After upgrade every site reads *Artifacts never deployed* until one *Deploy Artifacts to All Sites* — run it as the last step. Write the deployment record `docs/deployment/deployment-records/<date>-wonder-app-{vd03,vq03-vq04}-ac833f5a.md`. | Pending-updates finish (2026-09-14) | ~~Blocked by the 2026-09-11 remote-access hold~~ — **unblocked 2026-09-28**: `winrm` is re-enabled for deployment activities (CLAUDE.md). vd03 done; QA done 2026-09-28. | Closed 2026-09-28. |
 | 127 | **TODO — roll out site secrets via artifact deploy to vd03 + QA (vq03/vq04)**. The gRPC secrets hub is retired in ScadaBridge as of `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md` (implemented on branch `feat/site-secrets-artifact-deploy`, rig-only so far); vd03 and QA still run the pre-feature build with `Secrets:Replication:Enabled=false`, so upgrading them to a build carrying this change will **not** itself fail fast (the retirement's `StartupValidator` rejection only fires on `Enabled=true`, which neither environment sets) — but a site still needs the four prerequisites below before *Deploy Artifacts* will actually deliver its secrets. Prerequisites, in order: (1) provision the `zbsecrets` schema/store in the environment's configuration SQL Server (the same server `ConfigurationDb` already points at); (2) set `Secrets:Store:Provider=SqlServer` on **both** central nodes of the environment; (3) verify the KEK (`ZB_SECRETS_MASTER_KEY` / master-key file) is identical across **every** node — both centrals and every site node — since a mismatch is caught only at deploy time by `DeployedSecretsMirror`'s per-secret verify, not at boot; (4) recreate, at the new central shared store, any secret an operator previously hand-seeded directly into a site's local `secrets.db` (those never round-tripped through central and would otherwise be silently absent from the referenced-secrets scan). Then run *Deploy Artifacts* against every affected site as the rollout's last step. | `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md` §4 *Rollout*; `docs/deployment/topology-guide.md` → *Secrets: shared store, per-site deploy, and the KEK* | Deployment record, not a code change — vd03/QA config is user-held and their rollout schedule is independent of the branch merging to main. | User asks for the vd03/QA secrets window. |
 ## Resolved (verified against the code 2026-07-10)
 Rows removed from the Deferred table above once confirmed shipped. Kept here for traceability.
@@ -152,3 +152,76 @@ site secrets recreated at central; then *Deploy Artifacts*. Tracked as a deferre
 Per-site explicit secret lists; secrets for external systems / database connections (they ship
 literal values today); plaintext transport; moving bootstrap secrets through the deploy.
 ## 6. Implementation notes (as built)
 Execution (tasks 1–11, ledger `.superpowers/sdd/2026-09-28-site-secrets-artifact-deploy/progress.md`)
 matched this design in shape but landed several deviations, called out here so the doc stays
 accurate as the spec:
 - **Central collection.** The scanner and per-site resolve described in §3.1/§3.3 are implemented
  as `SiteSecretsCollector` (`ZB.MOM.WW.ScadaBridge.DeploymentManager`), not inline in
  `ArtifactDeploymentService`. Fail-closed (§3.3) is per site, not per batch: a site with an
  unresolved reference is skipped (`_communicationService` is never called for it) while every
  other site in the same sweep proceeds normally.
 - **Audit (§3.7).** The `DeploySiteSecrets` / `SiteSecrets` audit row is written on **both**
  `DeployCoreAsync` (the sweep) and `RetryForSiteAsync` (the per-site retry), including the
  fail-closed path on each — the design's §3.7 wording ("one row per site deploy that carried
  secrets") did not call out the retry path explicitly; it is in scope and covered.
 - **Stamp gating (§3.6).** Implemented as designed, plus one addition: a **partial apply**
  (`SecretsApplied` less than the shipped count, not just the mixed-version `null` case) is also
  treated as a failure and is not stamped, with its own message (`Site applied {n} of {m}
  secrets`) distinct from the "predates secret deployment" wording.
 - **Hash (§3.4).** Implemented as designed. Confirmed explicitly (not just implied by
  "null-when-empty"): an **empty** `Secrets`/`MissingSecrets` list and a **null** one hash
  identically, and the pre-existing golden hash for a site with no secret references is
  unchanged by this feature (pinned by a dedicated regression test).
 - **Site apply (§3.5).** `HandleDeployArtifacts` writes the full `deployed_secrets` set and then
  runs `DeployedSecretsMirror.ReconcileAsync` **synchronously** before replying, exactly as
  designed. `SecretsApplied` counts only the names *this deploy shipped* that applied and
  verified without error — a mirror failure for a name this deploy did not ship (e.g. another,
  already-broken deployed secret, or an unrelated removal) is logged at Warning, excluded from
  the count, and does not fail the deploy. Removal failures are likewise Warnings, never deploy
  failures.
 - **`DeployedSecretsMirror` — apply is confirm-then-ledger, not the blind `ApplyReplicatedAsync`
  §3.5 originally described.** The real library store is strict last-writer-wins on
  `(UpdatedUtc, Revision)` (a tie is a no-op) and tombstones rather than hard-deletes, which the
  naive "apply then ledger" sequence gets wrong in two ways: a routine remove-then-re-add can
  no-op against a still-live tombstone while the ledger is written anyway (silently broken), and
  an out-of-band local secret with a genuinely newer stamp can be ledgered and later deleted by a
  central tombstone (destroying the operator's own secret). The mirror now always reads the row
  back after applying and only ledgers when the local ciphertext matches what was shipped; a
  tombstoned or ledger-owned row is safely re-stamped (`UpdatedUtc`/`Revision` bumped past the
  local value, including across clock skew) and re-applied, while a live row with different
  ciphertext that the ledger does **not** already own is left completely untouched and reported
  as a distinct out-of-band-conflict failure. Ownership is decided from `mirrored_secrets`
  membership at any revision, not from tombstone state alone — so a mirror-owned name that drifts
  locally (an operator's edit over an already-deployed name) is re-stamped back to the deployed
  value on the next pass: **central is authoritative for every name it deploys.** Every apply/
  delete also invalidates the shared `ISecretResolver` cache before the verify step, and one
  poisoned row never blocks the rest of the pass.
 - **Hub retirement (§3.8) — scope widened to ALL secrets replication, not just the gRPC hub.**
  ScadaBridge also retires the historical SQL-Server *replication* mode
  (`Secrets:Replication:Mode=SqlServer`, distinct from the still-supported
  `Secrets:Store:Provider=SqlServer` shared-store feature) for the same reason the gRPC hub goes:
  a site's `ISecretStore` must be the plain, undecorated local SQLite store, because
  `DeployedSecretsMirror` calls `DeleteAsync` directly and a replication decorator would publish
  that delete fleet-wide. `Secrets:Replication:Enabled` set to anything other than
  absent/empty/`false` (a deliberately strict, non-boolean-coerced check — `"0"`/`"1"`/`"yes"`
  all count as "set") now fails `StartupValidator.Validate` on **either** role. A leftover
  `Secrets:Replication:Mode` or any `Secrets:GrpcHub:*` key logs a startup Warning naming both
  sections. The pre-host `${secret:}` expander was fixed in the same task set to read the shared
  SQL store on a `Store:Provider=SqlServer` central (it previously only did this for the retired
  Grpc mode) — without that fix a central's boot-time `${secret:}` tokens and its running
  `ISecretResolver` could resolve a token differently.
 - **`ZB.MOM.WW.Secrets` 0.6.3 (hub sweep logging, §3.7's third bullet — `EventId`s 4/5/6,
  `ISecretsHubSyncStatus`) was implemented and tested on a `scadaproj` branch but is NOT
  published to the package feed, and ScadaBridge does not consume it.** Publishing is an outward
  side effect that needs explicit user consent, and it would be a dead pin here regardless —
  hub retirement (above) removes ScadaBridge's only dependency on the hub client entirely. It
  remains available for the library's other hub consumers once published.
 - **Rig posture.** `docker/docker-compose.yml`'s central nodes now carry
  `Secrets__Store__Provider=SqlServer` plus a shared KEK; site-a shares that KEK (it is the
  secrets-via-artifact-deploy site); site-b/site-c deliberately carry a **mismatched** KEK so the
  `DeployedSecretsMirror` verify-failure path has a real, provable negative on the rig rather than
  an assumed one.
@@ -109,7 +109,7 @@ Central cluster only. Sites have no user interface.
 - **Move data connection between sites** (M9/T24): a **Move…** action on a data connection opens `MoveDataConnectionDialog`. The underlying `MoveDataConnectionCommand` guards against: target site does not exist, name collision at the target, any `InstanceConnectionBinding` referencing the connection, and any `InstanceNativeAlarmSourceOverride.ConnectionNameOverride` reference at the source site's templates. Blocked connections display the blocking reason.
 - **Server certificate management** (`/design/connections/{id}/certificates`, Admin role): a per-connection page that lists the contents of the site's OPC UA trusted-peer and rejected certificate stores (Subject / Issuer / Thumbprint / validity / Trusted-or-Rejected status) with a **Remove** action. The page makes clear the store is **node-wide for the site** (shared by every site node), not per data connection — trusting or removing a certificate affects all OPC UA connections at that site. Trust and Remove are central commands relayed to **both** site nodes so the node-local PKI stores stay consistent across failover (see Component-SiteRuntime.md, Component-DataConnectionLayer.md).
 - The site detail page exposes a new **"Audit feed"** tab that hosts the Audit Log page pre-filtered to `Site = <site>` — an in-context view of every operational audit event for that site.
- **Artifacts badge (2026-09-14, design `docs/plans/2026-09-14-pending-updates-design.md`).** Each site card carries an **Artifacts** badge next to the identifier, one of three states: `Current` (light — the site's `Site.ArtifactRevisionHash` matches what a deploy would send today), `Out of date` (warning, `bi-exclamation-triangle`, tooltip = the last successful deploy's UTC timestamp), or `Never deployed` (secondary — the site has no stamp at all, including every site immediately after the migration that added the columns, until one artifact deploy). Below the badge a muted `N of M instances out of date` line links to `/deployment/topology` whenever the site has any deployed instances, since the same fleet snapshot already carries the count.
+- **Artifacts badge (2026-09-14, design `docs/plans/2026-09-14-pending-updates-design.md`).** Each site card carries an **Artifacts** badge next to the identifier, one of three states: `Current` (light — the site's `Site.ArtifactRevisionHash` matches what a deploy would send today), `Out of date` (warning, `bi-exclamation-triangle`, tooltip = the last successful deploy's UTC timestamp), or `Never deployed` (secondary — the site has no stamp at all, including every site immediately after the migration that added the columns, until one artifact deploy). Below the badge a muted `N of M instances out of date` line links to `/deployment/topology` whenever the site has any deployed instances, since the same fleet snapshot already carries the count. **A site's secrets count toward the same badge (2026-09-28, `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md`)**: a data connection referencing a `secret:<name>` not yet created centrally hashes as `Missing` and reads `Out of date`, never `Current`, until the secret exists and a deploy ships it.
 - The page header gains a **Re-check** button (forces a fresh fleet evaluation, disabled while running) and an **"Updates as of HH:mm:ss UTC"** caption next to the page title. The snapshot loads **after** the site list, never blocking it — cards render immediately with a *checking…* placeholder for the badge/line, and a snapshot failure renders **unavailable** on the badge rather than hiding it or failing the page. A successful **Deploy Artifacts** (single-site or all-sites) forces a refresh of the snapshot so the badge flips to `Current` immediately rather than waiting out the cache TTL.
 ### Schema Library (Design Role)
@@ -189,6 +189,7 @@ Central cluster only. Sites have no user interface.
 - **Deploy all**: A bulk action deploys artifacts to all sites at once.
 - This is a **separate action** from instance deployment — system-wide artifacts are not automatically pushed when definitions change.
 - **Track per-site deployment status**, backed by `Site.ArtifactRevisionHash` / `ArtifactsDeployedAt` (Component-DeploymentManager.md → *Per-site artifact revision*), stamped **only on a successful deploy to that site**. A failed site keeps its previous stamp (or none) and keeps reading out of date on the Sites page; after an upgrade that ships the stamp columns, every site reads *Never deployed* until one artifact deploy runs — see the Artifacts badge above.
 - **Secrets (2026-09-28).** A deploy carries only the secrets that site's data connections reference (Component-DeploymentManager.md → *Site secrets*); no separate UI surface picks them. A site referencing a secret that is missing or tombstoned centrally fails **that site's** deploy (others in the same "Deploy all" sweep are unaffected) with a message naming the secret and the referencing data connection(s) — e.g. `Secret 'X' referenced by data connection(s) A, B is not in the central secret store` — surfaced on the deployment result exactly like any other per-site artifact-deploy failure. A site whose build predates secret deployment, or that applied only some of the shipped secrets, likewise fails and is not stamped `Current`; both cases resolve by upgrading the site and redeploying.
 ### Debug View (Deployment Role)
 - Select a deployed instance and open a live debug view — the instance picked through the same shared **`InstancePicker`** combobox as the Flow Debugger, with an `ItemLabel` suffix of `(State)` on each row.
@@ -281,6 +281,9 @@ Component-NotificationService.md). Notification delivery happens on the central
 cluster, so no notification artifact or SMTP credential is ever distributed to
 sites.
 A site's **secrets** are a fifth, *derived* artifact class riding the same command —
 see *Site secrets* below.
 ### Per-site artifact revision (2026-09-14)
 `Site` carries two nullable columns — `ArtifactRevisionHash` (`string?`, max 100)
@@ -331,6 +334,59 @@ design**: a partial map would silently report the unqueried sites as up to date,
 so the pending-updates cache instead discards the faulted flight and the UI shows
 the aggregate as unavailable.
 ### Site secrets (2026-09-28)
 Site secrets ride the artifact deploy instead of the (now retired) gRPC secrets
 hub — design `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md`; see
 that doc's *Implementation notes (as built)* section for the deviations called
 out below.
 - **Derived, not configured.** `SiteSecretsCollector` scans the site's data
  connection artifacts (`SecretReferenceScanner` over `PrimaryConfigurationJson`
  / `BackupConfigurationJson`) for `secret:<name>` string values and resolves
  each name against the central `ISecretStore`, producing the ciphertext-only
  `SecretArtifact` rows the command carries (`DeployArtifactsCommand.Secrets`,
  proto field 9) plus the set of unresolved references. A site with no `secret:`
  references ships no secrets and is unaffected by any of this.
 - **Fail closed, per site.** A site with one or more unresolved (absent or
  tombstoned) references fails its deploy *before* `_communicationService` is
  called — nothing is sent, the site keeps its previous stamp, and the error
  names the secret and the referencing data connection(s): `Secret 'X'
  referenced by data connection(s) A, B is not in the central secret store`
  (semicolon-joined for more than one). Other sites in the same sweep are
  unaffected.
 - **Stamp gating.** A site is stamped `ArtifactRevisionHash`/`ArtifactsDeployedAt`
  only when every shipped secret actually applied. Two failure shapes are
  distinguished from an ordinary site failure and never stamped: an **old
  site** (wire `Success = true`, secrets were shipped, but the response has no
  `SecretsApplied` at all — a pre-feature site build) fails with `Site build
  predates secret deployment — upgrade the site before deploying secrets`; a
  **partial apply** (`SecretsApplied` less than the shipped count) fails with
  `Site applied {n} of {m} secrets`. Both apply identically on the all-sites
  sweep (`DeployCoreAsync`) and the per-site retry (`RetryForSiteAsync`).
 - **Staleness hash.** `ArtifactRevisionHashService` folds secrets into the same
  hash as every other artifact class (`HashableSecret`: a SHA-256 digest of the
  ciphertext/nonce/tag/wrapped-DEK material, never plaintext, plus `KekId` and
  `Revision`) so a KEK rewrap correctly flips a site to out of date. A
  referenced-but-not-yet-created secret hashes as an explicit `Missing` marker
  (`DeployArtifactsCommand.MissingSecrets`, central-only — never on the wire),
  so that site reads *Out of date*, never *Current*, until the secret exists.
  **A `null` secrets list and an empty one hash identically** (the existing
  null-when-empty convention for every artifact class), so a site with no
  secret references keeps its pre-existing hash across the upgrade that added
  this feature.
 - **Audit.** One best-effort `DeploySiteSecrets` / `SiteSecrets` audit row is
  written per site deploy that carried secrets or a missing-secret failure
  (never for a site with no secret references at all) — after-state is `{
  deploymentId, secrets: [{name, revision}], missing: [name], success }`, which
  structurally excludes ciphertext. Written on both `DeployCoreAsync` and
  `RetryForSiteAsync`, including the fail-closed path (audited even though
  nothing was sent).
 - **Central store.** Central resolves secrets from `ISecretStore`, composed by
  the Host per `Secrets:Store:Provider` — see Component-Host.md → *Secrets*.
 - **Site apply and the per-node mirror** — see Component-SiteRuntime.md →
  *Deployed Secrets*.
 ## Site-Side Apply Atomicity
 Applying a deployment at the site is **all-or-nothing per instance**:
@@ -420,6 +476,7 @@ The Deployment Manager sends the following commands to sites via the Communicati
 - **Configuration Database (MS SQL)**: Stores deployment status and deployed configuration snapshots.
 - **Security & Auth**: Enforces Deployment role (with optional site scoping).
 - **Configuration Database (via IAuditService)**: Logs all deployment actions, system-wide artifact deployments, and instance lifecycle changes.
 - **`ZB.MOM.WW.Secrets.Abstractions`** (`ISecretStore`, `SecretName`): `SiteSecretsCollector` resolves a site's referenced secrets against the Host-composed central store — see Component-Host.md → *Secrets*.
 ## Interactions
@@ -249,6 +249,58 @@ reporting that it is a standby. The verdict remains fully observable through the
 ---
 ## Secrets (2026-09-28)
 The Host composes `ZB.MOM.WW.Secrets` (a `scadaproj` shared library, not a ScadaBridge component —
 hence it has no row in the Component Registration Matrix below) via `SecretsRegistration.AddScadaBridgeSecrets`,
 called on both roles from their respective composition-root `Configure` methods. Design:
 `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md` (see its *Implementation notes (as
 built)* section for where the shipped code deviates from the original design).
 - **Site role — always a local, undecorated store.** A site node's `ISecretStore`/`ISecretResolver`
  is unconditionally the plain node-local SQLite store (`AddZbSecrets`), regardless of any
  `Secrets:Store:Provider` value — a Site never reads that key. This is a hard invariant, not
  just a default: `DeployedSecretsMirror` (Component-SiteRuntime.md) calls `ISecretStore.DeleteAsync`
  directly on the assumption that a delete is node-local; a replication-decorated store would have
  published that delete fleet-wide. Secrets reach a site only via Deploy Artifacts (see
  Component-DeploymentManager.md → *Site secrets*) — there is no other channel.
 - **Central role — `Secrets:Store:Provider`** (`Sqlite` default | `SqlServer`), read **only** on the
  Central role:
  - `Sqlite` (default): each central node keeps its own per-node SQLite secret store. On a
    **clustered** central (≥ 2 seed nodes) this is a documented misconfiguration — `StartupValidator`
    logs a boot-time Warning (same shape as the inbound-API-key-store precedent, register row 108),
    because the two nodes would deploy different secret sets to sites depending on which is active.
  - `SqlServer`: both central nodes read and write **one shared SQL Server store**
    (`Secrets:SqlServer:ConnectionString`, a literal/env value — never a `${secret:}` reference,
    the same bootstrap-circularity rule the inbound-API-key shared store follows) — the `/admin/secrets`
    page edits it, and it is the store `SiteSecretsCollector` resolves from. The pre-host `${secret:}`
    expander (the Layer-A pass that resolves `${secret:X}` tokens inside `appsettings` itself, before
    the DI container is built) reads the **same** shared store when `Provider = SqlServer`, so a
    central node's boot-time config tokens and its running `ISecretResolver` never disagree about
    what a token resolves to.
  - A connection string that is blank, or is itself a `${secret:}` reference (a bootstrap circularity),
    fails fast naming the offending key.
 - **Bootstrap secrets are never deployed.** The site's gRPC PSK (`SB-GRPC-PSK-{siteId}`), the LocalDb
  replication key, and the secrets master key (KEK) are never data-connection-referenced and are
  never shipped through Deploy Artifacts — a site needs them *before* it can receive a deploy at all,
  and pushing a rotated one through the channel it authenticates would sever that channel. They stay
  out-of-band: service environment variables, a hand-seeded store, or a master-key file, exactly as
  before this feature. Every node — every central and every site — **must share the same KEK**;
  see `docs/deployment/topology-guide.md` → *Secrets: shared store, per-site deploy, and the KEK*
  for where that is (and is not) checked.
 - **The gRPC secrets hub is retired in ScadaBridge (2026-09-28).** ScadaBridge no longer composes
  `ZB.MOM.WW.Secrets.Replicator.Grpc` on either role, and the historical SQL-Server *replication*
  mode (`Secrets:Replication:Mode=SqlServer` — distinct from `Secrets:Store:Provider=SqlServer`
  above, which shares one store rather than replicating between two) is retired too. Setting
  `Secrets:Replication:Enabled` to anything other than absent/empty/`false` now fails
  `StartupValidator.Validate` outright, on **either** role, with a message pointing at Deploy
  Artifacts (sites) and `Secrets:Store:Provider=SqlServer` (a clustered central). A leftover
  `Secrets:Replication:Mode` or any `Secrets:GrpcHub:*` key (harmless once `Enabled` is unset, but
  dead configuration) logs a startup Warning naming both sections so they can be cleaned up. The
  library itself keeps the hub for other consumers; only ScadaBridge stops using it.
 ---
 ## Component Registration Matrix
 | Component | Central | Site | DI (`AddXxx`) | Actors (`AddXxxActors`) | Endpoints (`MapXxx`) |
@@ -162,6 +162,12 @@ flowchart TD
 - Recompiles shared scripts and makes updated code available to all Script Actors.
 - **Reconciles deletions**: each non-null artifact list is the complete set for its class, so after upserting, any stored row absent from the set is deleted (the artifact was deleted centrally — there is no per-artifact delete command). A removed shared script is also unregistered from the compiled script library, and a removed data connection's live DCL connection actor is stopped. A `null` list means "field not shipped" and reconciles nothing.
 ### Deployed Secrets (2026-09-28)
 - `HandleDeployArtifacts` additionally accepts the command's `Secrets` list (`SecretArtifact`, ciphertext only) and writes the full shipped set to the `deployed_secrets` table via `SiteStorageService.UpsertDeployedSecretsAsync` — upsert every shipped name, soft-remove (`is_removed = 1`) any stored name absent from the set. A `null` `Secrets` field (pre-feature central) touches nothing and answers with no `SecretsApplied`. Storage detail: *Deployed Secrets (Site SQLite / LocalDb)* below.
 - After writing the table, the active node runs `DeployedSecretsMirror.ReconcileAsync` **synchronously**, before replying — so the deploy response's `SecretsApplied` count and any apply/verify error reflect this deploy's own outcome, not a value left over from the mirror's background schedule. `SecretsApplied` counts only the names *this deploy shipped* that applied and verified cleanly; a mirror failure for an unrelated, already-broken name (a different deploy's leftover, or a stale removal) is logged at Warning and excluded — it neither fails this deploy nor reduces its `SecretsApplied`.
 - No `DeployedSecretsMirror` registered (secrets deployment not composed on this node) with a non-empty `Secrets` list fails the deploy with `Secret deployment is not enabled on this site node`; with an empty `Secrets` list it still succeeds (`SecretsApplied = 0`).
 - See *Deployed Secrets (Site SQLite / LocalDb)* below for the mirror itself.
 ### Instance Lifecycle Commands
 - **Disable**: Stops the Instance Actor and its children. Retains the deployed configuration in SQLite so the instance can be re-enabled without redeployment.
 - **Enable**: Creates a new Instance Actor from the stored configuration (same as startup).
@@ -424,6 +430,26 @@ The Instance Actor owns native-alarm setup alongside its computed Script and Ala
 ---
 ## Deployed Secrets (Site SQLite / LocalDb)
 Site secrets ride the artifact deploy (design `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md`; see that doc's *Implementation notes (as built)* for deviations from the original design). Two tables, two different replication postures:
 - **`deployed_secrets`** — the **11th LocalDb-replicated table** (`SiteLocalDbSetup.ReplicatedTables`, up from ten). Columns: `name` (PK), `row_json` (the shipped `SecretArtifact` — ciphertext, nonce, tag, wrapped DEK, `KekId`, revision; never plaintext), `revision`, `updated_utc`, `is_removed` (soft-remove flag, default 0), `deployment_id`, `applied_at`. No FKs (the standing FK guard — `SiteLocalDbReplicatedTableFkGuardTests` — applies to it like every other replicated table). `UpsertDeployedSecretsAsync` is full-set upsert-then-soft-remove, never a hard `DELETE`, so a removal is itself a replicated row change the standby node can react to.
 - **`mirrored_secrets`** — a **node-local, unreplicated** ledger (`name` PK, `revision`, `mirrored_at`) recording which names *this specific node* has applied into its own `ISecretStore`. Deliberately not registered with LocalDb: it is a per-node fact (which names this node mirrored), not site-wide state, and replicating it would make one node's mirroring progress masquerade as the other's.
 **`DeployedSecretsMirror`** (`SiteRuntime/Secrets/DeployedSecretsMirror.cs`) is a per-node `BackgroundService` — registered on **every** site node, not a singleton — that reconciles `deployed_secrets` into the node's own local `ISecretStore` (site nodes never run a replication-decorated store; see Component-Host.md → *Secrets*):
 - **Schedule**: runs once immediately at start, then every `SiteRuntime:DeployedSecretsMirrorInterval` (default 15s, validated `> 0`), and synchronously on demand right after an active-node artifact apply (see *Deployed Secrets* under Deployment Manager Singleton above). This 15s interval is the **standby node's own convergence path** — it never receives the active node's synchronous call, only the replicated `deployed_secrets` rows via LocalDb, so it picks them up on its own next tick.
 - **Apply is confirm-then-ledger, not blind.** `ApplyReplicatedAsync` is strict last-writer-wins on `(UpdatedUtc, Revision)` (a tie is a no-op) and `DeleteAsync` tombstones rather than hard-deletes — so the mirror always reads the row back after applying and only ledgers it in `mirrored_secrets` when the local ciphertext matches what was shipped. A tombstoned or clock-skewed local row that the ledger already owns is safely re-stamped (`UpdatedUtc`/`Revision` bumped past the local value) and re-applied; a live row with **different** ciphertext that the ledger does **not** already own is left completely untouched — not overwritten, not ledgered, not verified — and reported as an out-of-band conflict (`"exists on this node with a newer local value; the deployed revision was not applied"`), so an operator's genuinely local secret of the same name is never silently destroyed by a later central deploy or removal.
 - **Ownership, not tombstone-state, decides re-stamping.** Once a name is ledgered in `mirrored_secrets` at any revision, this mirror treats that name as its own going forward — including re-stamping a mirror-owned name that drifted (e.g. an operator edited it locally) back to the deployed value on the next pass. Central is authoritative for every name it deploys.
 - **Removal** deletes from the local store **only if the name was previously ledgered** (present in `mirrored_secrets`) — an out-of-band local secret sharing a deployed name that was never mirrored is never touched by a central-side removal.
 - **Verify.** Every pass trial-resolves every live row through `ISecretResolver` (even one not re-applied this pass, so a later KEK change is still caught) — a decrypt failure names the secret and its `KekId` and is reported as a distinct error (de-spammed: Error the first time a given `(name, revision, KekId)` fails, Debug on identical repeats, Error again after an intervening success).
 - **Isolation.** One row failing to apply/verify does not block any other row in the same pass.
 - **Cache invalidation.** A successful apply or delete invalidates the shared `ISecretResolver` cache before verifying, so the verify step (and any script/connection reading the secret) sees the fresh value immediately rather than waiting out the resolver's cache TTL.
 - **Never logs a secret value** — log lines and the reported `MirrorResult` carry only names, revisions, and `KekId`s.
 ---
 ## Enriched `AlarmStateChanged` Message
 The `AlarmStateChanged` message published by both Alarm Actors and Native Alarm Actors was extended **additively** (existing consumers keep working with computed defaults):
@@ -642,10 +668,11 @@ Per Akka.NET best practices, internal actor communication uses **Tell** (fire-an
 - **Site Event Logging**: Records script executions, alarm events, deployment events, instance lifecycle events.
 - **Health Monitoring**: Reports script error rates and alarm evaluation error rates.
 - **Local SQLite**: Persists deployed configurations, system-wide artifacts (external system definitions, database connection definitions, data connection definitions). Sites also maintain peer SQLite stores for the Store-and-Forward buffer, the site event log, the operation tracking table, and the site-local `AuditLog` (see [Component-AuditLog.md](Component-AuditLog.md)). The `AuditLog` file is purged on the same daily cadence as the others but respects the hard `ForwardState` invariant — rows still `Pending` forward are never purged, regardless of age.
 - **`ZB.MOM.WW.Secrets.Abstractions`** (`ISecretStore`, `ISecretResolver`, `ISecretCacheInvalidator`): `DeployedSecretsMirror` applies deployed secrets into, and verifies against, the node's local secret store (composed by the Host — see Component-Host.md → *Secrets*). The Site Runtime depends only on the abstractions package, never the concrete store implementation.
 ## Interactions
- **Deployment Manager (central)**: Receives flattened configurations, system-wide artifact updates, and instance lifecycle commands.
+- **Deployment Manager (central)**: Receives flattened configurations, system-wide artifact updates (including a site's referenced secrets, ciphertext only), and instance lifecycle commands.
 - **Data Connection Layer**: Bidirectional — receives value updates, sends write-back commands.
 - **Communication Layer**: Receives commands from central, sends deployment results, serves debug view data.
 - **Store-and-Forward Engine**: Scripts route cached writes, notifications, and external system calls here.