ScadaBridge/docs/plans/2026-09-28-site-secrets-artifact-deploy-live-gate.md

Site secrets ride the artifact deploy — live gate (Task 13)

Date: 2026-09-28/29 (UTC 00:28–00:50 on 2026-09-29) · Branch: ScadaBridge-site-secrets worktree · Result: bash docker/verify-site-secrets.sh → PASS 8/8 (on the image carrying fix ad19d4b1). The first run, on the pre-fix image, was PASS 7/8 — check 5 found a product defect, fixed below.

Design: 2026-09-28-site-secrets-artifact-deploy-design.md.

1. Method

Rig bring-up

docker/deploy.sh was deliberately not run from the worktree (its relative bind mounts would rehome to the worktree). Instead:

1. bash docker/build.sh from the worktree → scadabridge:latest built from branch code.
2. docker compose -f <worktree>/docker/docker-compose.yml --project-directory ~/Desktop/ScadaBridge/docker up -d --force-recreate — the branch compose (new Secrets__* env: central Store:Provider=SqlServer + shared KEK, site-a shared KEK, site-b/site-c a deliberately mismatched KEK, no hub keys anywhere) with bind mounts resolved against the primary checkout's docker/ directory. git diff 2aede140 HEAD --stat -- docker/ shows only docker-compose.yml changed, so the mounted node appsettings are identical to the branch's. docker compose config resolved project name docker, the running project, so the containers were replaced, not duplicated.
3. All 8 nodes answered /health/ready 200 (centrals on 9001/9002; sites probed on their internal 8084 from the Traefik container, which is on scadabridge-net).

The same procedure was repeated after the fix (image rebuild + force-recreate); all 8 ready within ~10 s.

Boot-log review (secrets)

• Central (both): no secrets-related warning or error. Secrets:Store:Provider=SqlServer against the rig's existing ZbSecretsHub database came up clean; no leftover-key warning (the rig carries no Secrets:GrpcHub:* or Replication:Mode).
• site-a: no secrets-related warning or error. Its local secrets.db still holds six rows swept from the retired hub in August (hub-gate-smoke-2/3, shared-gate-*, KEK sha256:f94d4ae2c4b1) — inert, not ledgered by the mirror, so never touched by it.
• site-b / site-c: booted cleanly with a KEK for the first time; no secrets warnings.
• The only secrets line after the second recreate was an expected ERR on site-b-a — the mirror's startup pass re-verifying the sb-gate-mxgw row left live there by the pre-fix run (the defect's residue); the next gate run's pre-clean removed it.
• Unrelated noise present on every boot: singleton-proxy "not found after 30 s" warnings during cluster formation, one akka-cluster ... Removed health-check ERR per pair during the recreate, OPC UA auto-accept warnings.

How each surface is driven

Surface	Tool	Notes
Central secret create / rotate / delete	Central UI /admin/secrets (ZB.MOM.WW.Secrets.Ui), headless Chromium via Python Playwright	The supported operator path; there is no CLI or management verb for central secrets, and the library's secret CLI only talks to a SQLite store headless. Value goes through the environment into the masked input, never argv/stdout.
Data connections, artifact deploys, audit query	ScadaBridge CLI via Traefik (localhost:9000)	data-connection create/update/delete, site deploy-artifacts --site-id, audit-config query --action DeploySiteSecrets.
Artifacts badge (Current / Out of date)	Sites page, Re-check pressed, badge aria-label read	The pending-updates evaluation has no CLI surface and is Blazor-interactive.
Central row	sqlcmd read-only on ZbSecretsHub.zbsecrets.secret	Revision, KekId, HASHBYTES('SHA2_256', ciphertext).
Site node stores	docker cp of /app/data/scadabridge-secrets.db (library table secret) and /app/data/site-localdb.db (deployed_secrets, mirrored_secrets, data_connection_definitions) + WAL, read by host python sqlite3	The image has no sqlite3. The -shm is not copied (SQLite rebuilds the wal-index); each read is repeated until two consecutive copies agree.
Out-of-band seed on site-a-a (check 5)	The library's own secret CLI (ZB.MOM.WW.Secrets.Cli, published from scadaproj main = 0.6.2) run in a throwaway scadabridge:latest container with --volumes-from scadabridge-site-a-a, Secrets__SqlitePath=/data/scadabridge-secrets.db and site-a's KEK	i.e. the library's normal local SQLite write path on that node's store — not a raw SQL insert.

Gate objects: central secret sb-gate-mxgw; never-created sb-gate-missing; out-of-band sb-gate-oob on site-a-a; MxGateway connections sb-gate-conn-a (site-a) and sb-gate-conn-b (site-b), unbound, endpoint http://sb-gate-no-such-gateway:5000. The trap-based teardown (also run up-front as a pre-clean) deletes both connections, redeploys site-a and site-b so the removal reaches them, deletes the central secret, removes the out-of-band secret with the secret CLI, then redeploys artifacts fleet-wide; the run ended with all three sites Current. The library tombstones rather than hard-deletes, so after teardown the stores hold only is_deleted = 1 rows for the gate names (verified: site-a-a, site-a-b, site-b-a and central all deleted; no gate connection left).

2. Per-check evidence (run 2, PASS 8/8)

#	Check	Evidence
1	Secret + referencing connection → site-a Out of date, b/c unchanged	sb-gate-mxgw created via /admin/secrets (central r3, KEK sha256:f94d4ae2c4b1 — the revision continues past run 1's tombstone); after sb-gate-conn-a ("ApiKey":"secret:sb-gate-mxgw") badges site-a: out of date, site-b: current, site-c: current.
2	Deploy site-a → success, secretsDeployed: 1, Current	{"siteId":"site-a","success":true,"secretsDeployed":1} (0.2 s); badge current.
3	Both nodes at the central revision; standby ≤ 30 s	site-a-a r3, ciphertext SHA-256 707ae73ba872… == central (visible 4.0 s after the deploy returned — dominated by the docker-cp read loop; the active node applies synchronously before replying); site-a-b converged 5.5 s after the deploy returned; its mirrored_secrets ledger at r3.
4	Rotate → Out of date → deploy → both at new revision	central r3 → r4 with new ciphertext; badge out of date; redeploy secretsDeployed 1; site-a-a r4 (0.3 s), site-a-b r4 at 6.8 s; badge back to current.
5	Drop the reference → both nodes drop it; out-of-band survives	sb-gate-oob seeded on site-a-a via the secret CLI; sb-gate-conn-a switched to a literal ApiKey; deploy succeeded (no secretsDeployed); site-a-a no longer holds sb-gate-mxgw (0.2 s), site-a-b at 9.0 s; sb-gate-oob still live on site-a-a; deployed_secrets.sb-gate-mxgw.is_removed = 1 (kept for the standby).
6	Missing secret → fail closed, stays Out of date, site untouched	Error Secret 'sb-gate-missing' referenced by data connection(s) sb-gate-conn-a is not in the central secret store; badge out of date; stamped hash, both nodes' secret rows, deployed_secrets, ledgers and the site's sb-gate-conn-a definition byte-identical before/after.
7	Same reference on site-b (mismatched KEK) → fails naming KekId	Error Secret 'sb-gate-mxgw' (kek sha256:f94d4ae2c4b1) did not decrypt on this node — the node's master key differs from central's; site-b not stamped (out of date).
8	Audit rows; no material in logs	Exactly 4 DeploySiteSecrets/SiteSecrets rows with distinct deploymentIds: site-a ok sb-gate-mxgw@r3, site-a ok @r4, site-a fail missing sb-gate-missing, site-b fail @r4; the no-secret deploy of check 5 wrote none. Grep of docker logs --since <gate start> on all 8 nodes and of the audit payloads for both plaintext values, the out-of-band value and both base64 ciphertexts: 0 hits.

Standby convergence (site-a-b, seconds after the deploy call returned)

Run	Check 3 (first deploy)	Check 4 (rotation)	Check 5 (removal)
1 (pre-fix)	9.4	5.4	never (defect)
2 (fixed)	5.5	6.8	9.0

All inside the 30 s window, consistent with LocalDb CDC replication of deployed_secrets plus the standby's 15 s DeployedSecretsMirrorInterval. The figures include up to ~1–2 s of read-loop overhead (docker cp + double read per poll).

3. Defects found

3.1 Dropping a site's last secret: reference never removed the secret (fixed, ad19d4b1)

Run 1, check 5: after the removal deploy both site-a nodes still held sb-gate-mxgw r1 live, deployed_secrets.is_removed = 0, ledger r1. Cause: ArtifactDeploymentService.BuildDeployArtifactsCommandAsync shipped Secrets = null whenever a site had no references ("null-when-empty"), and the site deliberately treats null as "central shipped no secrets" (pre-feature contract) and touches nothing — so the full-set soft-remove never ran. Fix: a secrets-aware central always ships a list (empty when there are no references). Old sites ignore proto field 9; the artifact hash is unchanged (empty and null hash identically); SecretsDeployed stays null and no audit row is written when nothing shipped. Regression test NoSecretReferences_ShipsEmptySecretsList_SoTheSiteRemovesPreviouslyDeployedOnes (red on the old code, green on the fix). Spec updated in Component-DeploymentManager.md, Component-SiteRuntime.md and the design doc's as-built notes.

3.2 ManagementActorTests did not compile a valid proxy (fixed, 1444f859)

Four DeployArtifacts_* tests built an NSubstitute proxy of ArtifactDeploymentService with the pre-Task-6 constructor arguments; they now pass a SiteSecretsCollector. Test-only.

4. Test suite

dotnet build ZB.MOM.WW.ScadaBridge.slnx: 0 warnings, 0 errors. dotnet test ZB.MOM.WW.ScadaBridge.slnx (30 test assemblies, run against the live rig): 10 006 tests, 9 979 passed, 27 failed — 4 were the ManagementActorTests proxy failures fixed in 1444f859 (ManagementService.Tests re-run: 668/668), the other 23 are Playwright E2E. After the fix commit DeploymentManager.Tests is 234/234 (one new test). Every non-Playwright assembly is green.

Playwright residue (23 of 184):

Class	Tests	Classification
RoleNavigationTests	AdminUser_DoesNotSeeDeploymentSection, AdminUser_DoesNotSeeDesignSection, AdminUser_SeesHealthDashboard_NotDeploymentMonitoring, DeploymentUser_DoesNotSeeAdminSection, DeploymentUser_DoesNotSeeDesignSection, DeploymentUser_SeesMonitoringButNotConfigurationAuditLog, DesignUser_DoesNotSeeAdminSection, DesignUser_DoesNotSeeDeploymentSection, DesignUser_SeesHealthDashboard_NotDeploymentMonitoringOrAudit	Known env-only — role-negative under DisableLogin=true
LoginTests	UnauthenticatedUser_RedirectsToLogin	Known env-only — DisableLogin
AuditLogPageTests	PermissionGating_DesignerWithoutOperationalAudit_IsDeniedAccess	Known env-only — DisableLogin
DeploymentActionTests	Delete_Instance_RemovesFromTree, Deploy_Instance_ShowsOutcomeToast, Disable_Instance_ShowsOutcomeToast, Enable_Instance_ShowsOutcomeToast	Known residue (pagination helper — row not found)
TopologyAreaTests	CreateArea_ViaToolbar_AppearsInTreeAndPersists, Diff_OnDeployedInstance_OpensDialog, MoveArea_UnderAnotherArea_ShowsMovedToast, MoveInstance_ToArea_ShowsMovedToast, RenameArea_EnterCommits_PersistsNewName, RenameArea_Escape_RevertsToOriginalLabel	Known residue (SignalR/visibility timeout)
AuditLogPageTests	ExportCsv_LinkVisibleForAuditExportUserAndTriggersDownload	Not on the known list — Download event timeout (30 s), reproducible in isolation. Audit export is untouched by this branch; central logs showed audit-ingest SQL timeouts against the large AuditLog table at the time. Judged environmental.
TransportImportTests	ImportSyntheticBundle_AppliesAndShowsAuditDrillIn	Not on the known list — reproducible in isolation; central logs Cannot insert duplicate key row in object 'dbo.Sites' ... (site-a) during the import, i.e. the synthetic bundle's site mapping collides with the rig's existing site-a. Transport and site mapping are untouched by this branch. Judged rig-data related.

5. Observations (not gated)

• Standby of an unreplicated site pair never receives deployed secrets. deployed_secrets reaches the standby only through LocalDb CDC replication; on site-b (deliberately unreplicated on this rig) the gate secret landed in site-b-a's store only — site-b-b's store never saw it. A failover on an unreplicated pair would leave the new active node without the site's secrets until the next artifact deploy. This follows from the design (§3.5) but is worth stating in the topology guide: secrets-bearing sites should run with LocalDb replication on.
• The CLI's site deploy-artifacts exits 0 even when a site result is success: false; the gate reads the JSON siteResults rather than the exit code.
• Re-adding a name that is tombstoned at central continues its revision (run 2 started at r3 after run 1's r1 + delete), which is why the gate compares revisions to the live central row rather than to fixed numbers.

6. Re-run after final-review fixes (2026-09-28)

Code under test: d6411863 (fixes 6f1787ab mirror drift heal + unreadable-row isolation, b1b49bb3 secrets-before-DCL ordering, 67a3950f shared secret: parser + hash-parity test). Same procedure as §1: bash docker/build.sh from the worktree (image sha256:9de1db5c…), then docker compose -f <worktree>/docker/docker-compose.yml --project-directory ~/Desktop/ScadaBridge/docker up -d --force-recreate; all 8 nodes answered /health/ready 200 within ~15 s.

bash docker/verify-site-secrets.sh → PASS 8/8. Standby convergence on site-a-b (seconds after the deploy call returned): check 3 = 8.1, check 4 = 5.4, check 5 = 9.0. Check 8: exactly the 4 expected DeploySiteSecrets rows, 0 plaintext/ciphertext hits across all 8 node logs and the audit payloads. Teardown left all three sites Current.

Tests before the rig run: dotnet build ZB.MOM.WW.ScadaBridge.slnx 0 errors; Commons.Tests 875/875, SiteRuntime.Tests 892/892, DeploymentManager.Tests 235/235, DataConnectionLayer.Tests 383/383, Host.Tests 593/593.

7. Follow-ups run (2026-09-28)

Code under test: branch fix/site-secrets-followups — per-site secret-store outage handling and removal-only audit rows (29049c4e), artifact-deploy CLI exit codes (ef03a0dc), per-node mirror status on the site health report (0289d511, merged in 4e640918). Image rebuilt from the worktree with bash docker/build.sh, all 8 nodes recreated from the primary checkout's compose file.

Hub-era secrets removed from site-a (review M8), before the run. The retired gRPC hub had left nine rows dated 2026-08-07 in both site-a nodes' local stores (/app/data/scadabridge-secrets.db, table secret): six live (hub-gate-smoke-2, hub-gate-smoke-3, shared-gate-051, shared-gate-2, shared-gate-2b, shared-gate-final) and three tombstones (expander-gate-pepper, hub-gate-smoke-1, shared-gate-1). Nothing in the rig config referenced them. With the site-a pair stopped together, each node's file was edited from a throwaway python:3.13-alpine container mounting that node's data volume (DELETE … WHERE name IN (…) AND updated_utc < '2026-08-08', then wal_checkpoint(TRUNCATE)); 9 rows deleted per node, only the sb-gate-* gate tombstones left. Both nodes came back ready, site-a-a active. site-b/site-c never ran the hub and had no such rows; QA and vd03 never enabled replication.

The gate grew to nine checks:

• new check 5 (M1): site-a-b is recreated with the site-b/site-c dev KEK (a compose override, removed again in the same check and in teardown), and central's health site --identifier site-a must show node-b failing sb-gate-mxgw while node-a is clean, then both clean once site-a-b has site-a's KEK again;
• check 6 (was 5) also asserts the reply's secretsRemoved names the dropped secret;
• checks 7 and 8 (were 6 and 7) also assert the CLI exits 1;
• check 9 (was 8) now expects five DeploySiteSecrets rows — the removal-only deploy of check 6 writes one, with removed: [sb-gate-mxgw].

bash docker/verify-site-secrets.sh → PASS 9/9. Before the check, central reported both site-a node rows with no failures; site-b and site-c (unreplicated) report their active node only. With the wrong KEK, node-b's sb-gate-mxgw failure appeared on central's site-a report 57 s after site-a-b was back in the cluster (mirror pass 15 s + status reporter 30 s + health report 30 s), and cleared 57 s after the KEK was restored. Standby convergence: check 3 = 4.3 s, check 4 = 4.2 s, check 6 = 15.1 s. Audit: the 5 expected rows, 5 distinct deployment ids; 0 plaintext/ciphertext hits in all 8 node logs and the audit payloads. Teardown left all three sites Current and site-a-b on site-a's KEK.

Not exercised live: the central secret-store outage (review M5) — taking the shared SQL store down on the rig also takes down central's configuration database. Covered by unit tests (SiteSecretsCollectorTests, ArtifactDeploymentServiceTests.StoreOutage_*, ComputeCurrentHashes_StoreOutage_*, RetryForSite_StoreOutage_*).

Tests: dotnet build ZB.MOM.WW.ScadaBridge.slnx 0 errors / 0 warnings; every non-Playwright test project green (CLI.Tests 553 after updating DeploymentDeadlineParityTests' stub server to reply with a real zero-failure summary — a bare {} now reads as an invalid artifact-deploy summary and exits 1). Playwright not run.