test(rig): site secrets artifact-deploy live gate PASS 8/8

docs/plans/2026-09-28-site-secrets-artifact-deploy-live-gate.md

@@ -0,0 +1,137 @@
+# Site secrets ride the artifact deploy — live gate (Task 13)
+
+**Date:** 2026-09-28/29 (UTC 00:28–00:50 on 2026-09-29) · **Branch:** `ScadaBridge-site-secrets` worktree ·
+**Result:** `bash docker/verify-site-secrets.sh` → **PASS 8/8** (on the image carrying fix `ad19d4b1`).
+The first run, on the pre-fix image, was **PASS 7/8** — check 5 found a product defect, fixed below.
+
+Design: [`2026-09-28-site-secrets-artifact-deploy-design.md`](2026-09-28-site-secrets-artifact-deploy-design.md).
+
+## 1. Method
+
+### Rig bring-up
+
+`docker/deploy.sh` was deliberately **not** run from the worktree (its relative bind mounts would rehome
+to the worktree). Instead:
+
+1. `bash docker/build.sh` from the worktree → `scadabridge:latest` built from branch code.
+2. `docker compose -f <worktree>/docker/docker-compose.yml --project-directory ~/Desktop/ScadaBridge/docker up -d --force-recreate`
+   — the branch compose (new `Secrets__*` env: central `Store:Provider=SqlServer` + shared KEK, site-a shared KEK,
+   site-b/site-c a deliberately mismatched KEK, no hub keys anywhere) with bind mounts resolved against the primary
+   checkout's `docker/` directory. `git diff 2aede140 HEAD --stat -- docker/` shows only `docker-compose.yml` changed,
+   so the mounted node appsettings are identical to the branch's. `docker compose config` resolved project name
+   `docker`, the running project, so the containers were replaced, not duplicated.
+3. All 8 nodes answered `/health/ready` 200 (centrals on 9001/9002; sites probed on their internal 8084 from the
+   Traefik container, which is on `scadabridge-net`).
+
+The same procedure was repeated after the fix (image rebuild + force-recreate); all 8 ready within ~10 s.
+
+### Boot-log review (secrets)
+
+- **Central (both):** no secrets-related warning or error. `Secrets:Store:Provider=SqlServer` against the rig's
+  existing `ZbSecretsHub` database came up clean; no leftover-key warning (the rig carries no `Secrets:GrpcHub:*` or
+  `Replication:Mode`).
+- **site-a:** no secrets-related warning or error. Its local `secrets.db` still holds six rows swept from the retired
+  hub in August (`hub-gate-smoke-2/3`, `shared-gate-*`, KEK `sha256:f94d4ae2c4b1`) — inert, not ledgered by the
+  mirror, so never touched by it.
+- **site-b / site-c:** booted cleanly with a KEK for the first time; no secrets warnings.
+- The only secrets line after the second recreate was an expected `ERR` on site-b-a — the mirror's startup pass
+  re-verifying the `sb-gate-mxgw` row left live there by the pre-fix run (the defect's residue); the next gate
+  run's pre-clean removed it.
+- Unrelated noise present on every boot: singleton-proxy "not found after 30 s" warnings during cluster formation,
+  one `akka-cluster ... Removed` health-check ERR per pair during the recreate, OPC UA auto-accept warnings.
+
+### How each surface is driven
+
+| Surface | Tool | Notes |
+|---|---|---|
+| Central secret create / rotate / delete | Central UI **`/admin/secrets`** (ZB.MOM.WW.Secrets.Ui), headless Chromium via Python Playwright | The supported operator path; there is no CLI or management verb for central secrets, and the library's `secret` CLI only talks to a SQLite store headless. Value goes through the environment into the masked input, never argv/stdout. |
+| Data connections, artifact deploys, audit query | ScadaBridge CLI via Traefik (`localhost:9000`) | `data-connection create/update/delete`, `site deploy-artifacts --site-id`, `audit-config query --action DeploySiteSecrets`. |
+| Artifacts badge (Current / Out of date) | Sites page, **Re-check** pressed, badge `aria-label` read | The pending-updates evaluation has no CLI surface and is Blazor-interactive. |
+| Central row | `sqlcmd` **read-only** on `ZbSecretsHub.zbsecrets.secret` | Revision, KekId, `HASHBYTES('SHA2_256', ciphertext)`. |
+| Site node stores | `docker cp` of `/app/data/scadabridge-secrets.db` (library table `secret`) and `/app/data/site-localdb.db` (`deployed_secrets`, `mirrored_secrets`, `data_connection_definitions`) + WAL, read by host python `sqlite3` | The image has no sqlite3. The `-shm` is not copied (SQLite rebuilds the wal-index); each read is repeated until two consecutive copies agree. |
+| Out-of-band seed on site-a-a (check 5) | The library's own **`secret` CLI** (`ZB.MOM.WW.Secrets.Cli`, published from scadaproj main = 0.6.2) run in a throwaway `scadabridge:latest` container with `--volumes-from scadabridge-site-a-a`, `Secrets__SqlitePath=/data/scadabridge-secrets.db` and site-a's KEK | i.e. the library's normal local SQLite write path on that node's store — not a raw SQL insert. |
+
+Gate objects: central secret `sb-gate-mxgw`; never-created `sb-gate-missing`; out-of-band `sb-gate-oob` on site-a-a;
+MxGateway connections `sb-gate-conn-a` (site-a) and `sb-gate-conn-b` (site-b), unbound, endpoint
+`http://sb-gate-no-such-gateway:5000`. The trap-based teardown (also run up-front as a pre-clean) deletes both
+connections, redeploys site-a and site-b so the removal reaches them, deletes the central secret, removes the
+out-of-band secret with the `secret` CLI, then redeploys artifacts fleet-wide; the run ended with all three sites
+**Current**. The library tombstones rather than hard-deletes, so after teardown the stores hold only
+`is_deleted = 1` rows for the gate names (verified: site-a-a, site-a-b, site-b-a and central all `deleted`;
+no gate connection left).
+
+## 2. Per-check evidence (run 2, PASS 8/8)
+
+| # | Check | Evidence |
+|---|---|---|
+| 1 | Secret + referencing connection → site-a Out of date, b/c unchanged | `sb-gate-mxgw` created via `/admin/secrets` (central r3, KEK `sha256:f94d4ae2c4b1` — the revision continues past run 1's tombstone); after `sb-gate-conn-a` (`"ApiKey":"secret:sb-gate-mxgw"`) badges `site-a: out of date, site-b: current, site-c: current`. |
+| 2 | Deploy site-a → success, `secretsDeployed: 1`, Current | `{"siteId":"site-a","success":true,"secretsDeployed":1}` (0.2 s); badge `current`. |
+| 3 | Both nodes at the central revision; standby ≤ 30 s | site-a-a r3, ciphertext SHA-256 `707ae73ba872…` == central (visible 4.0 s after the deploy returned — dominated by the docker-cp read loop; the active node applies synchronously before replying); **site-a-b converged 5.5 s** after the deploy returned; its `mirrored_secrets` ledger at r3. |
+| 4 | Rotate → Out of date → deploy → both at new revision | central r3 → r4 with new ciphertext; badge `out of date`; redeploy `secretsDeployed 1`; site-a-a r4 (0.3 s), **site-a-b r4 at 6.8 s**; badge back to `current`. |
+| 5 | Drop the reference → both nodes drop it; out-of-band survives | `sb-gate-oob` seeded on site-a-a via the `secret` CLI; `sb-gate-conn-a` switched to a literal ApiKey; deploy succeeded (no `secretsDeployed`); site-a-a no longer holds `sb-gate-mxgw` (0.2 s), **site-a-b at 9.0 s**; `sb-gate-oob` still live on site-a-a; `deployed_secrets.sb-gate-mxgw.is_removed = 1` (kept for the standby). |
+| 6 | Missing secret → fail closed, stays Out of date, site untouched | Error `Secret 'sb-gate-missing' referenced by data connection(s) sb-gate-conn-a is not in the central secret store`; badge `out of date`; stamped hash, both nodes' secret rows, `deployed_secrets`, ledgers and the site's `sb-gate-conn-a` definition byte-identical before/after. |
+| 7 | Same reference on site-b (mismatched KEK) → fails naming KekId | Error `Secret 'sb-gate-mxgw' (kek sha256:f94d4ae2c4b1) did not decrypt on this node — the node's master key differs from central's`; site-b not stamped (`out of date`). |
+| 8 | Audit rows; no material in logs | Exactly 4 `DeploySiteSecrets`/`SiteSecrets` rows with distinct deploymentIds: `site-a ok sb-gate-mxgw@r3`, `site-a ok @r4`, `site-a fail missing sb-gate-missing`, `site-b fail @r4`; the no-secret deploy of check 5 wrote none. Grep of `docker logs --since <gate start>` on all 8 nodes and of the audit payloads for both plaintext values, the out-of-band value and both base64 ciphertexts: **0 hits**. |
+
+### Standby convergence (site-a-b, seconds after the deploy call returned)
+
+| Run | Check 3 (first deploy) | Check 4 (rotation) | Check 5 (removal) |
+|---|---|---|---|
+| 1 (pre-fix) | 9.4 | 5.4 | never (defect) |
+| 2 (fixed) | 5.5 | 6.8 | 9.0 |
+
+All inside the 30 s window, consistent with LocalDb CDC replication of `deployed_secrets` plus the standby's
+15 s `DeployedSecretsMirrorInterval`. The figures include up to ~1–2 s of read-loop overhead (docker cp + double
+read per poll).
+
+## 3. Defects found
+
+### 3.1 Dropping a site's last `secret:` reference never removed the secret (fixed, `ad19d4b1`)
+
+Run 1, check 5: after the removal deploy both site-a nodes still held `sb-gate-mxgw` r1 live,
+`deployed_secrets.is_removed = 0`, ledger r1. Cause: `ArtifactDeploymentService.BuildDeployArtifactsCommandAsync`
+shipped `Secrets = null` whenever a site had no references ("null-when-empty"), and the site deliberately treats
+`null` as "central shipped no secrets" (pre-feature contract) and touches nothing — so the full-set soft-remove
+never ran. Fix: a secrets-aware central always ships a list (empty when there are no references). Old sites
+ignore proto field 9; the artifact hash is unchanged (empty and null hash identically); `SecretsDeployed` stays
+null and no audit row is written when nothing shipped. Regression test
+`NoSecretReferences_ShipsEmptySecretsList_SoTheSiteRemovesPreviouslyDeployedOnes` (red on the old code, green on
+the fix). Spec updated in `Component-DeploymentManager.md`, `Component-SiteRuntime.md` and the design doc's
+as-built notes.
+
+### 3.2 `ManagementActorTests` did not compile a valid proxy (fixed, `1444f859`)
+
+Four `DeployArtifacts_*` tests built an NSubstitute proxy of `ArtifactDeploymentService` with the pre-Task-6
+constructor arguments; they now pass a `SiteSecretsCollector`. Test-only.
+
+## 4. Test suite
+
+`dotnet build ZB.MOM.WW.ScadaBridge.slnx`: 0 warnings, 0 errors.
+`dotnet test ZB.MOM.WW.ScadaBridge.slnx` (30 test assemblies, run against the live rig): **10 006 tests, 9 979
+passed, 27 failed** — 4 were the `ManagementActorTests` proxy failures fixed in `1444f859`
+(ManagementService.Tests re-run: 668/668), the other 23 are Playwright E2E. After the fix commit
+DeploymentManager.Tests is 234/234 (one new test). Every non-Playwright assembly is green.
+
+Playwright residue (23 of 184):
+
+| Class | Tests | Classification |
+|---|---|---|
+| `RoleNavigationTests` | `AdminUser_DoesNotSeeDeploymentSection`, `AdminUser_DoesNotSeeDesignSection`, `AdminUser_SeesHealthDashboard_NotDeploymentMonitoring`, `DeploymentUser_DoesNotSeeAdminSection`, `DeploymentUser_DoesNotSeeDesignSection`, `DeploymentUser_SeesMonitoringButNotConfigurationAuditLog`, `DesignUser_DoesNotSeeAdminSection`, `DesignUser_DoesNotSeeDeploymentSection`, `DesignUser_SeesHealthDashboard_NotDeploymentMonitoringOrAudit` | Known env-only — role-negative under `DisableLogin=true` |
+| `LoginTests` | `UnauthenticatedUser_RedirectsToLogin` | Known env-only — `DisableLogin` |
+| `AuditLogPageTests` | `PermissionGating_DesignerWithoutOperationalAudit_IsDeniedAccess` | Known env-only — `DisableLogin` |
+| `DeploymentActionTests` | `Delete_Instance_RemovesFromTree`, `Deploy_Instance_ShowsOutcomeToast`, `Disable_Instance_ShowsOutcomeToast`, `Enable_Instance_ShowsOutcomeToast` | Known residue (pagination helper — row not found) |
+| `TopologyAreaTests` | `CreateArea_ViaToolbar_AppearsInTreeAndPersists`, `Diff_OnDeployedInstance_OpensDialog`, `MoveArea_UnderAnotherArea_ShowsMovedToast`, `MoveInstance_ToArea_ShowsMovedToast`, `RenameArea_EnterCommits_PersistsNewName`, `RenameArea_Escape_RevertsToOriginalLabel` | Known residue (SignalR/visibility timeout) |
+| `AuditLogPageTests` | `ExportCsv_LinkVisibleForAuditExportUserAndTriggersDownload` | **Not on the known list** — `Download` event timeout (30 s), reproducible in isolation. Audit export is untouched by this branch; central logs showed audit-ingest SQL timeouts against the large `AuditLog` table at the time. Judged environmental. |
+| `TransportImportTests` | `ImportSyntheticBundle_AppliesAndShowsAuditDrillIn` | **Not on the known list** — reproducible in isolation; central logs `Cannot insert duplicate key row in object 'dbo.Sites' ... (site-a)` during the import, i.e. the synthetic bundle's site mapping collides with the rig's existing site-a. Transport and site mapping are untouched by this branch. Judged rig-data related. |
+
+## 5. Observations (not gated)
+
+- **Standby of an unreplicated site pair never receives deployed secrets.** `deployed_secrets` reaches the
+  standby only through LocalDb CDC replication; on site-b (deliberately unreplicated on this rig) the gate secret
+  landed in site-b-a's store only — site-b-b's store never saw it. A failover on an unreplicated pair would leave
+  the new active node without the site's secrets until the next artifact deploy. This follows from the design
+  (§3.5) but is worth stating in the topology guide: secrets-bearing sites should run with LocalDb replication on.
+- The CLI's `site deploy-artifacts` exits 0 even when a site result is `success: false`; the gate reads the JSON
+  `siteResults` rather than the exit code.
+- Re-adding a name that is tombstoned at central continues its revision (run 2 started at r3 after run 1's r1 +
+  delete), which is why the gate compares revisions to the live central row rather than to fixed numbers.