# Site secrets ride the artifact deploy — live gate (Task 13)

**Date:** 2026-09-28/29 (UTC 00:28–00:50 on 2026-09-29) · **Branch:** `ScadaBridge-site-secrets` worktree ·
**Result:** `bash docker/verify-site-secrets.sh` → **PASS 8/8** (on the image carrying fix `ad19d4b1`).
The first run, on the pre-fix image, was **PASS 7/8** — check 5 found a product defect, fixed below.

Design: [`2026-09-28-site-secrets-artifact-deploy-design.md`](2026-09-28-site-secrets-artifact-deploy-design.md).

## 1. Method

### Rig bring-up

`docker/deploy.sh` was deliberately **not** run from the worktree (its relative bind mounts would rehome
to the worktree). Instead:

1. `bash docker/build.sh` from the worktree → `scadabridge:latest` built from branch code.
2. `docker compose -f <worktree>/docker/docker-compose.yml --project-directory ~/Desktop/ScadaBridge/docker up -d --force-recreate`
   — the branch compose (new `Secrets__*` env: central `Store:Provider=SqlServer` + shared KEK, site-a shared KEK,
   site-b/site-c a deliberately mismatched KEK, no hub keys anywhere) with bind mounts resolved against the primary
   checkout's `docker/` directory. `git diff 2aede140 HEAD --stat -- docker/` shows only `docker-compose.yml` changed,
   so the mounted node appsettings are identical to the branch's. `docker compose config` resolved project name
   `docker`, the running project, so the containers were replaced, not duplicated.
3. All 8 nodes answered `/health/ready` 200 (centrals on 9001/9002; sites probed on their internal 8084 from the
   Traefik container, which is on `scadabridge-net`).

The same procedure was repeated after the fix (image rebuild + force-recreate); all 8 ready within ~10 s.

### Boot-log review (secrets)

- **Central (both):** no secrets-related warning or error. `Secrets:Store:Provider=SqlServer` against the rig's
  existing `ZbSecretsHub` database came up clean; no leftover-key warning (the rig carries no `Secrets:GrpcHub:*` or
  `Replication:Mode`).
- **site-a:** no secrets-related warning or error. Its local `secrets.db` still holds six rows swept from the retired
  hub in August (`hub-gate-smoke-2/3`, `shared-gate-*`, KEK `sha256:f94d4ae2c4b1`) — inert, not ledgered by the
  mirror, so never touched by it.
- **site-b / site-c:** booted cleanly with a KEK for the first time; no secrets warnings.
- The only secrets line after the second recreate was an expected `ERR` on site-b-a — the mirror's startup pass
  re-verifying the `sb-gate-mxgw` row left live there by the pre-fix run (the defect's residue); the next gate
  run's pre-clean removed it.
- Unrelated noise present on every boot: singleton-proxy "not found after 30 s" warnings during cluster formation,
  one `akka-cluster ... Removed` health-check ERR per pair during the recreate, OPC UA auto-accept warnings.

### How each surface is driven

| Surface | Tool | Notes |
|---|---|---|
| Central secret create / rotate / delete | Central UI **`/admin/secrets`** (ZB.MOM.WW.Secrets.Ui), headless Chromium via Python Playwright | The supported operator path; there is no CLI or management verb for central secrets, and the library's `secret` CLI only talks to a SQLite store headless. Value goes through the environment into the masked input, never argv/stdout. |
| Data connections, artifact deploys, audit query | ScadaBridge CLI via Traefik (`localhost:9000`) | `data-connection create/update/delete`, `site deploy-artifacts --site-id`, `audit-config query --action DeploySiteSecrets`. |
| Artifacts badge (Current / Out of date) | Sites page, **Re-check** pressed, badge `aria-label` read | The pending-updates evaluation has no CLI surface and is Blazor-interactive. |
| Central row | `sqlcmd` **read-only** on `ZbSecretsHub.zbsecrets.secret` | Revision, KekId, `HASHBYTES('SHA2_256', ciphertext)`. |
| Site node stores | `docker cp` of `/app/data/scadabridge-secrets.db` (library table `secret`) and `/app/data/site-localdb.db` (`deployed_secrets`, `mirrored_secrets`, `data_connection_definitions`) + WAL, read by host python `sqlite3` | The image has no sqlite3. The `-shm` is not copied (SQLite rebuilds the wal-index); each read is repeated until two consecutive copies agree. |
| Out-of-band seed on site-a-a (check 5) | The library's own **`secret` CLI** (`ZB.MOM.WW.Secrets.Cli`, published from scadaproj main = 0.6.2) run in a throwaway `scadabridge:latest` container with `--volumes-from scadabridge-site-a-a`, `Secrets__SqlitePath=/data/scadabridge-secrets.db` and site-a's KEK | i.e. the library's normal local SQLite write path on that node's store — not a raw SQL insert. |

Gate objects: central secret `sb-gate-mxgw`; never-created `sb-gate-missing`; out-of-band `sb-gate-oob` on site-a-a;
MxGateway connections `sb-gate-conn-a` (site-a) and `sb-gate-conn-b` (site-b), unbound, endpoint
`http://sb-gate-no-such-gateway:5000`. The trap-based teardown (also run up-front as a pre-clean) deletes both
connections, redeploys site-a and site-b so the removal reaches them, deletes the central secret, removes the
out-of-band secret with the `secret` CLI, then redeploys artifacts fleet-wide; the run ended with all three sites
**Current**. The library tombstones rather than hard-deletes, so after teardown the stores hold only
`is_deleted = 1` rows for the gate names (verified: site-a-a, site-a-b, site-b-a and central all `deleted`;
no gate connection left).

## 2. Per-check evidence (run 2, PASS 8/8)

| # | Check | Evidence |
|---|---|---|
| 1 | Secret + referencing connection → site-a Out of date, b/c unchanged | `sb-gate-mxgw` created via `/admin/secrets` (central r3, KEK `sha256:f94d4ae2c4b1` — the revision continues past run 1's tombstone); after `sb-gate-conn-a` (`"ApiKey":"secret:sb-gate-mxgw"`) badges `site-a: out of date, site-b: current, site-c: current`. |
| 2 | Deploy site-a → success, `secretsDeployed: 1`, Current | `{"siteId":"site-a","success":true,"secretsDeployed":1}` (0.2 s); badge `current`. |
| 3 | Both nodes at the central revision; standby ≤ 30 s | site-a-a r3, ciphertext SHA-256 `707ae73ba872…` == central (visible 4.0 s after the deploy returned — dominated by the docker-cp read loop; the active node applies synchronously before replying); **site-a-b converged 5.5 s** after the deploy returned; its `mirrored_secrets` ledger at r3. |
| 4 | Rotate → Out of date → deploy → both at new revision | central r3 → r4 with new ciphertext; badge `out of date`; redeploy `secretsDeployed 1`; site-a-a r4 (0.3 s), **site-a-b r4 at 6.8 s**; badge back to `current`. |
| 5 | Drop the reference → both nodes drop it; out-of-band survives | `sb-gate-oob` seeded on site-a-a via the `secret` CLI; `sb-gate-conn-a` switched to a literal ApiKey; deploy succeeded (no `secretsDeployed`); site-a-a no longer holds `sb-gate-mxgw` (0.2 s), **site-a-b at 9.0 s**; `sb-gate-oob` still live on site-a-a; `deployed_secrets.sb-gate-mxgw.is_removed = 1` (kept for the standby). |
| 6 | Missing secret → fail closed, stays Out of date, site untouched | Error `Secret 'sb-gate-missing' referenced by data connection(s) sb-gate-conn-a is not in the central secret store`; badge `out of date`; stamped hash, both nodes' secret rows, `deployed_secrets`, ledgers and the site's `sb-gate-conn-a` definition byte-identical before/after. |
| 7 | Same reference on site-b (mismatched KEK) → fails naming KekId | Error `Secret 'sb-gate-mxgw' (kek sha256:f94d4ae2c4b1) did not decrypt on this node — the node's master key differs from central's`; site-b not stamped (`out of date`). |
| 8 | Audit rows; no material in logs | Exactly 4 `DeploySiteSecrets`/`SiteSecrets` rows with distinct deploymentIds: `site-a ok sb-gate-mxgw@r3`, `site-a ok @r4`, `site-a fail missing sb-gate-missing`, `site-b fail @r4`; the no-secret deploy of check 5 wrote none. Grep of `docker logs --since <gate start>` on all 8 nodes and of the audit payloads for both plaintext values, the out-of-band value and both base64 ciphertexts: **0 hits**. |

### Standby convergence (site-a-b, seconds after the deploy call returned)

| Run | Check 3 (first deploy) | Check 4 (rotation) | Check 5 (removal) |
|---|---|---|---|
| 1 (pre-fix) | 9.4 | 5.4 | never (defect) |
| 2 (fixed) | 5.5 | 6.8 | 9.0 |

All inside the 30 s window, consistent with LocalDb CDC replication of `deployed_secrets` plus the standby's
15 s `DeployedSecretsMirrorInterval`. The figures include up to ~1–2 s of read-loop overhead (docker cp + double
read per poll).

## 3. Defects found

### 3.1 Dropping a site's last `secret:` reference never removed the secret (fixed, `ad19d4b1`)

Run 1, check 5: after the removal deploy both site-a nodes still held `sb-gate-mxgw` r1 live,
`deployed_secrets.is_removed = 0`, ledger r1. Cause: `ArtifactDeploymentService.BuildDeployArtifactsCommandAsync`
shipped `Secrets = null` whenever a site had no references ("null-when-empty"), and the site deliberately treats
`null` as "central shipped no secrets" (pre-feature contract) and touches nothing — so the full-set soft-remove
never ran. Fix: a secrets-aware central always ships a list (empty when there are no references). Old sites
ignore proto field 9; the artifact hash is unchanged (empty and null hash identically); `SecretsDeployed` stays
null and no audit row is written when nothing shipped. Regression test
`NoSecretReferences_ShipsEmptySecretsList_SoTheSiteRemovesPreviouslyDeployedOnes` (red on the old code, green on
the fix). Spec updated in `Component-DeploymentManager.md`, `Component-SiteRuntime.md` and the design doc's
as-built notes.

### 3.2 `ManagementActorTests` did not compile a valid proxy (fixed, `1444f859`)

Four `DeployArtifacts_*` tests built an NSubstitute proxy of `ArtifactDeploymentService` with the pre-Task-6
constructor arguments; they now pass a `SiteSecretsCollector`. Test-only.

## 4. Test suite

`dotnet build ZB.MOM.WW.ScadaBridge.slnx`: 0 warnings, 0 errors.
`dotnet test ZB.MOM.WW.ScadaBridge.slnx` (30 test assemblies, run against the live rig): **10 006 tests, 9 979
passed, 27 failed** — 4 were the `ManagementActorTests` proxy failures fixed in `1444f859`
(ManagementService.Tests re-run: 668/668), the other 23 are Playwright E2E. After the fix commit
DeploymentManager.Tests is 234/234 (one new test). Every non-Playwright assembly is green.

Playwright residue (23 of 184):

| Class | Tests | Classification |
|---|---|---|
| `RoleNavigationTests` | `AdminUser_DoesNotSeeDeploymentSection`, `AdminUser_DoesNotSeeDesignSection`, `AdminUser_SeesHealthDashboard_NotDeploymentMonitoring`, `DeploymentUser_DoesNotSeeAdminSection`, `DeploymentUser_DoesNotSeeDesignSection`, `DeploymentUser_SeesMonitoringButNotConfigurationAuditLog`, `DesignUser_DoesNotSeeAdminSection`, `DesignUser_DoesNotSeeDeploymentSection`, `DesignUser_SeesHealthDashboard_NotDeploymentMonitoringOrAudit` | Known env-only — role-negative under `DisableLogin=true` |
| `LoginTests` | `UnauthenticatedUser_RedirectsToLogin` | Known env-only — `DisableLogin` |
| `AuditLogPageTests` | `PermissionGating_DesignerWithoutOperationalAudit_IsDeniedAccess` | Known env-only — `DisableLogin` |
| `DeploymentActionTests` | `Delete_Instance_RemovesFromTree`, `Deploy_Instance_ShowsOutcomeToast`, `Disable_Instance_ShowsOutcomeToast`, `Enable_Instance_ShowsOutcomeToast` | Known residue (pagination helper — row not found) |
| `TopologyAreaTests` | `CreateArea_ViaToolbar_AppearsInTreeAndPersists`, `Diff_OnDeployedInstance_OpensDialog`, `MoveArea_UnderAnotherArea_ShowsMovedToast`, `MoveInstance_ToArea_ShowsMovedToast`, `RenameArea_EnterCommits_PersistsNewName`, `RenameArea_Escape_RevertsToOriginalLabel` | Known residue (SignalR/visibility timeout) |
| `AuditLogPageTests` | `ExportCsv_LinkVisibleForAuditExportUserAndTriggersDownload` | **Not on the known list** — `Download` event timeout (30 s), reproducible in isolation. Audit export is untouched by this branch; central logs showed audit-ingest SQL timeouts against the large `AuditLog` table at the time. Judged environmental. |
| `TransportImportTests` | `ImportSyntheticBundle_AppliesAndShowsAuditDrillIn` | **Not on the known list** — reproducible in isolation; central logs `Cannot insert duplicate key row in object 'dbo.Sites' ... (site-a)` during the import, i.e. the synthetic bundle's site mapping collides with the rig's existing site-a. Transport and site mapping are untouched by this branch. Judged rig-data related. |

## 5. Observations (not gated)

- **Standby of an unreplicated site pair never receives deployed secrets.** `deployed_secrets` reaches the
  standby only through LocalDb CDC replication; on site-b (deliberately unreplicated on this rig) the gate secret
  landed in site-b-a's store only — site-b-b's store never saw it. A failover on an unreplicated pair would leave
  the new active node without the site's secrets until the next artifact deploy. This follows from the design
  (§3.5) but is worth stating in the topology guide: secrets-bearing sites should run with LocalDb replication on.
- The CLI's `site deploy-artifacts` exits 0 even when a site result is `success: false`; the gate reads the JSON
  `siteResults` rather than the exit code.
- Re-adding a name that is tombstoned at central continues its revision (run 2 started at r3 after run 1's r1 +
  delete), which is why the gate compares revisions to the live central row rather than to fixed numbers.

## 6. Re-run after final-review fixes (2026-09-28)

Code under test: `d6411863` (fixes `6f1787ab` mirror drift heal + unreadable-row isolation, `b1b49bb3`
secrets-before-DCL ordering, `67a3950f` shared `secret:` parser + hash-parity test). Same procedure as §1:
`bash docker/build.sh` from the worktree (image `sha256:9de1db5c…`), then
`docker compose -f <worktree>/docker/docker-compose.yml --project-directory ~/Desktop/ScadaBridge/docker up -d
--force-recreate`; all 8 nodes answered `/health/ready` 200 within ~15 s.

`bash docker/verify-site-secrets.sh` → **PASS 8/8**. Standby convergence on site-a-b (seconds after the deploy
call returned): check 3 = 8.1, check 4 = 5.4, check 5 = 9.0. Check 8: exactly the 4 expected `DeploySiteSecrets`
rows, 0 plaintext/ciphertext hits across all 8 node logs and the audit payloads. Teardown left all three sites
*Current*.

Tests before the rig run: `dotnet build ZB.MOM.WW.ScadaBridge.slnx` 0 errors; Commons.Tests 875/875,
SiteRuntime.Tests 892/892, DeploymentManager.Tests 235/235, DataConnectionLayer.Tests 383/383, Host.Tests 593/593.

## 7. Follow-ups run (2026-09-28)

Code under test: branch `fix/site-secrets-followups` — per-site secret-store outage handling and
removal-only audit rows (`29049c4e`), artifact-deploy CLI exit codes (`ef03a0dc`), per-node mirror status on
the site health report (`0289d511`, merged in `4e640918`). Image rebuilt from the worktree with
`bash docker/build.sh`, all 8 nodes recreated from the primary checkout's compose file.

**Hub-era secrets removed from site-a (review M8), before the run.** The retired gRPC hub had left nine rows
dated 2026-08-07 in both site-a nodes' local stores (`/app/data/scadabridge-secrets.db`, table `secret`):
six live (`hub-gate-smoke-2`, `hub-gate-smoke-3`, `shared-gate-051`, `shared-gate-2`, `shared-gate-2b`,
`shared-gate-final`) and three tombstones (`expander-gate-pepper`, `hub-gate-smoke-1`, `shared-gate-1`).
Nothing in the rig config referenced them. With the site-a pair stopped together, each node's file was edited
from a throwaway `python:3.13-alpine` container mounting that node's data volume (`DELETE … WHERE name IN (…)
AND updated_utc < '2026-08-08'`, then `wal_checkpoint(TRUNCATE)`); 9 rows deleted per node, only the `sb-gate-*`
gate tombstones left. Both nodes came back ready, site-a-a active. site-b/site-c never ran the hub and had no
such rows; QA and vd03 never enabled replication.

**The gate grew to nine checks:**
- new check 5 (M1): site-a-b is recreated with the site-b/site-c dev KEK (a compose override, removed again in
  the same check and in teardown), and central's `health site --identifier site-a` must show node-b failing
  `sb-gate-mxgw` while node-a is clean, then both clean once site-a-b has site-a's KEK again;
- check 6 (was 5) also asserts the reply's `secretsRemoved` names the dropped secret;
- checks 7 and 8 (were 6 and 7) also assert the CLI exits 1;
- check 9 (was 8) now expects five `DeploySiteSecrets` rows — the removal-only deploy of check 6 writes one,
  with `removed: [sb-gate-mxgw]`.

`bash docker/verify-site-secrets.sh` → **PASS 9/9**. Before the check, central reported both site-a node
rows with no failures; site-b and site-c (unreplicated) report their active node only. With the wrong KEK,
node-b's `sb-gate-mxgw` failure appeared on central's site-a report 57 s after site-a-b was back in the cluster
(mirror pass 15 s + status reporter 30 s + health report 30 s), and cleared 57 s after the KEK was restored.
Standby convergence: check 3 = 4.3 s, check 4 = 4.2 s, check 6 = 15.1 s. Audit: the 5 expected rows, 5 distinct
deployment ids; 0 plaintext/ciphertext hits in all 8 node logs and the audit payloads. Teardown left all three
sites *Current* and site-a-b on site-a's KEK.

Not exercised live: the central secret-store outage (review M5) — taking the shared SQL store down on the rig
also takes down central's configuration database. Covered by unit tests (`SiteSecretsCollectorTests`,
`ArtifactDeploymentServiceTests.StoreOutage_*`, `ComputeCurrentHashes_StoreOutage_*`,
`RetryForSite_StoreOutage_*`).

Tests: `dotnet build ZB.MOM.WW.ScadaBridge.slnx` 0 errors / 0 warnings; every non-Playwright test project green
(CLI.Tests 553 after updating `DeploymentDeadlineParityTests`' stub server to reply with a real zero-failure
summary — a bare `{}` now reads as an invalid artifact-deploy summary and exits 1). Playwright not run.