test(rig): site secrets gate covers the follow-ups — standby KEK mismatch on the health report, removal audit, CLI exit codes — PASS 9/9
Also: DeploymentDeadlineParityTests' stub replies with a real zero-failure summary (the artifact aliases now derive their exit code from failureCount), and register row 127 reflects the merge and the two new replicated site tables.
This commit is contained in:
Joseph Doherty
@@ -120,7 +120,7 @@ All 7 fix-now items landed via PLAN-04/05/06/07/08 (verified in review 08 round
|
||||
| 124 | **No template-graph-watermark short-circuit on the pending-updates fleet flatten** *(Low)*. `PendingUpdatesEvaluator.EvaluateAsync` re-flattens every deployed instance through `DeploymentService.GetInstanceStalenessAsync` on every cache-TTL expiry (`PendingUpdatesCacheTtl`, default 60 s) while any Deployer/Admin circuit is open, even when nothing in the template/instance/artifact graph has changed since the last evaluation. | `docs/plans/2026-09-14-pending-updates-design.md` §3.2 ("Not in scope") | The single-flight TTL cache already bounds the cost to one fleet flatten per TTL fleet-wide (not per viewer); a watermark short-circuit is a further optimization, not a correctness need, and needs a cheap "has anything changed" signal that does not exist yet. | The fleet flatten's cost (instance count x template depth) becomes measurable against `PendingUpdatesCacheTtl` on a large deployment. |
|
||||
| 125 | **Pending-updates counts have no Health-dashboard tile or CLI surface** *(Low)*. The aggregate (`PendingUpdatesSnapshot.StaleInstanceCount` / `StaleArtifactSiteCount`) is consumed only by the Central UI Sites page and rail indicator; there is no Health dashboard KPI tile and no `deploy`/`site` CLI command surfacing it — `site list` shows only the raw `artifactRevisionHash`/`artifactsDeployedAt` fields on each site, not whether they are current. | `docs/plans/2026-09-14-pending-updates-design.md` §3.2 ("Not in scope") | Out of scope for the UI-focused ask; the CLI/Health surfaces are a separate consumer of the same `IPendingUpdatesCache`. | An operator wants the pending-updates counts without opening the Central UI (headless monitoring, CLI scripting). |
|
||||
| 126 | ~~**TODO — roll the two outstanding migrations to the QA pair vq03/vq04**~~ **DONE 2026-09-28** — QA vq03/vq04 upgraded `2b9a7c3c` → `b3021e7c`, 55 → 57, artifacts deployed (record `docs/deployment/deployment-records/2026-09-28-wonder-app-vq03-vq04-b3021e7c.md`). *(deployment, closed)*. **DEV vd03 DONE 2026-09-28** (`fe89363b`, 55 → 57, record `docs/deployment/deployment-records/2026-09-28-wonder-app-vd03-fe89363b.md` — the `sb-mig-55to57.sql` there is the same script QA needs). QA still runs `2b9a7c3c` with the DB at migration 55 (`20260909203046_EnforceGlobalInstanceNamesAndProtectProtocolConfiguration`); `main` (`ac833f5a`, pushed 2026-09-14) carries two more: **56** `20260911135501_AddAlarmMesReportable` (computed-alarm `IsMesReportable` flag) and **57** `20260914173147_AddSiteArtifactRevision` (`Sites.ArtifactRevisionHash`/`ArtifactsDeployedAt`, both nullable). Both are additive `AddColumn`s — no data upgrade, no coordinated step like the 2026-09-10 protocol-config encryption. Recipe (same as the 55 rollout): generate ONE from→to delta script `dotnet ef migrations script EnforceGlobalInstanceNamesAndProtectProtocolConfiguration AddSiteArtifactRevision --project src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase --startup-project src/ZB.MOM.WW.ScadaBridge.Host -o sb-mig-55to57.sql` (NOT idempotent, identical for vd03 and the QA pair), take the pre-upgrade DB backup, stop the pair TOGETHER, apply via `sqlcmd -C -b -I` on the SQL host, publish/swap the `ac833f5a` binaries, start the pair, confirm the Central boot log shows 0 pending migrations and no "Applying migration" (the script already applied them). After upgrade every site reads *Artifacts never deployed* until one *Deploy Artifacts to All Sites* — run it as the last step. Write the deployment record `docs/deployment/deployment-records/<date>-wonder-app-{vd03,vq03-vq04}-ac833f5a.md`. | Pending-updates finish (2026-09-14) | ~~Blocked by the 2026-09-11 remote-access hold~~ — **unblocked 2026-09-28**: `winrm` is re-enabled for deployment activities (CLAUDE.md). vd03 done; QA done 2026-09-28. | Closed 2026-09-28. |
|
||||
| 127 | **TODO — roll out site secrets via artifact deploy to vd03 + QA (vq03/vq04)**. The gRPC secrets hub is retired in ScadaBridge as of `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md` (implemented on branch `feat/site-secrets-artifact-deploy`, rig-only so far); vd03 and QA still run the pre-feature build with `Secrets:Replication:Enabled=false`, so upgrading them will **not** fail the boot (the retirement's `StartupValidator` rejection only fires on `Enabled=true`). **Upgrade impact:** once upgraded, any site whose data connections carry `secret:` references fails its **whole** artifact deploy (scripts, external systems, DB connections too) and reads *Out of date* until every referenced secret exists in the central store — so before upgrading vd03/QA, check whether their connection configs use `secret:` references and whether their site pairs run LocalDb replication, and do the prerequisites in the same window. Prerequisites, in order: (1) provision the `zbsecrets` schema/store in the environment's configuration SQL Server (the same server `ConfigurationDb` already points at); (2) set `Secrets:Store:Provider=SqlServer` on **both** central nodes of the environment; (3) verify the KEK (`ZB_SECRETS_MASTER_KEY` / master-key file) is identical across **every** node — both centrals and every site node — since a mismatch is caught only at deploy time by `DeployedSecretsMirror`'s per-secret verify, not at boot; (4) confirm every site pair that uses secrets runs LocalDb replication — on an unreplicated pair only the node active at deploy time gets the secrets, so a failover leaves the new active node without them until the next *Deploy Artifacts*; (5) recreate, at the new central shared store, any secret an operator previously hand-seeded directly into a site's local `secrets.db` (those never round-tripped through central and would otherwise be silently absent from the referenced-secrets scan). Then run *Deploy Artifacts* against every affected site as the rollout's last step. | `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md` §4 *Rollout*; `docs/deployment/topology-guide.md` → *Secrets: shared store, per-site deploy, and the KEK* | Deployment record, not a code change — vd03/QA config is user-held and their rollout schedule is independent of the branch merging to main. | User asks for the vd03/QA secrets window. |
|
||||
| 127 | **TODO — roll out site secrets via artifact deploy to vd03 + QA (vq03/vq04)**. The gRPC secrets hub is retired in ScadaBridge as of `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md` (merged to `main` @ `23a139a4` 2026-09-28, plus the same-day follow-ups: per-site secret-store outage handling, removal audit rows, per-node mirror status on the health report, artifact-deploy CLI exit codes; rig-only so far); vd03 and QA still run the pre-feature build with `Secrets:Replication:Enabled=false`, so upgrading them will **not** fail the boot (the retirement's `StartupValidator` rejection only fires on `Enabled=true`). **Upgrade impact:** once upgraded, any site whose data connections carry `secret:` references fails its **whole** artifact deploy (scripts, external systems, DB connections too) and reads *Out of date* until every referenced secret exists in the central store — so before upgrading vd03/QA, check whether their connection configs use `secret:` references and whether their site pairs run LocalDb replication, and do the prerequisites in the same window. Prerequisites, in order: (1) provision the `zbsecrets` schema/store in the environment's configuration SQL Server (the same server `ConfigurationDb` already points at); (2) set `Secrets:Store:Provider=SqlServer` on **both** central nodes of the environment; (3) verify the KEK (`ZB_SECRETS_MASTER_KEY` / master-key file) is identical across **every** node — both centrals and every site node — since a mismatch is caught only at deploy time by `DeployedSecretsMirror`'s per-secret verify, not at boot (after the first deploy a mismatch on either node — the standby included — shows as a *Secret mirror failing* badge on that site's Health card); (4) confirm every site pair that uses secrets runs LocalDb replication — on an unreplicated pair only the node active at deploy time gets the secrets, so a failover leaves the new active node without them until the next *Deploy Artifacts*; the upgrade also adds two replicated site tables (`deployed_secrets`, `secret_mirror_status`), so stop and start each site pair together as usual; (5) recreate, at the new central shared store, any secret an operator previously hand-seeded directly into a site's local `secrets.db` (those never round-tripped through central and would otherwise be silently absent from the referenced-secrets scan). Then run *Deploy Artifacts* against every affected site as the rollout's last step. | `docs/plans/2026-09-28-site-secrets-artifact-deploy-design.md` §4 *Rollout*; `docs/deployment/topology-guide.md` → *Secrets: shared store, per-site deploy, and the KEK* | Deployment record, not a code change — vd03/QA config is user-held and their rollout schedule is independent of the branch merging to main. | User asks for the vd03/QA secrets window. |
|
||||
|
||||
## Resolved (verified against the code 2026-07-10)
|
||||
Rows removed from the Deferred table above once confirmed shipped. Kept here for traceability.
|
||||
|
||||
@@ -151,3 +151,46 @@ rows, 0 plaintext/ciphertext hits across all 8 node logs and the audit payloads.
|
||||
|
||||
Tests before the rig run: `dotnet build ZB.MOM.WW.ScadaBridge.slnx` 0 errors; Commons.Tests 875/875,
|
||||
SiteRuntime.Tests 892/892, DeploymentManager.Tests 235/235, DataConnectionLayer.Tests 383/383, Host.Tests 593/593.
|
||||
|
||||
## 7. Follow-ups run (2026-09-28)
|
||||
|
||||
Code under test: branch `fix/site-secrets-followups` — per-site secret-store outage handling and
|
||||
removal-only audit rows (`29049c4e`), artifact-deploy CLI exit codes (`ef03a0dc`), per-node mirror status on
|
||||
the site health report (`0289d511`, merged in `4e640918`). Image rebuilt from the worktree with
|
||||
`bash docker/build.sh`, all 8 nodes recreated from the primary checkout's compose file.
|
||||
|
||||
**Hub-era secrets removed from site-a (review M8), before the run.** The retired gRPC hub had left nine rows
|
||||
dated 2026-08-07 in both site-a nodes' local stores (`/app/data/scadabridge-secrets.db`, table `secret`):
|
||||
six live (`hub-gate-smoke-2`, `hub-gate-smoke-3`, `shared-gate-051`, `shared-gate-2`, `shared-gate-2b`,
|
||||
`shared-gate-final`) and three tombstones (`expander-gate-pepper`, `hub-gate-smoke-1`, `shared-gate-1`).
|
||||
Nothing in the rig config referenced them. With the site-a pair stopped together, each node's file was edited
|
||||
from a throwaway `python:3.13-alpine` container mounting that node's data volume (`DELETE … WHERE name IN (…)
|
||||
AND updated_utc < '2026-08-08'`, then `wal_checkpoint(TRUNCATE)`); 9 rows deleted per node, only the `sb-gate-*`
|
||||
gate tombstones left. Both nodes came back ready, site-a-a active. site-b/site-c never ran the hub and had no
|
||||
such rows; QA and vd03 never enabled replication.
|
||||
|
||||
**The gate grew to nine checks:**
|
||||
- new check 5 (M1): site-a-b is recreated with the site-b/site-c dev KEK (a compose override, removed again in
|
||||
the same check and in teardown), and central's `health site --identifier site-a` must show node-b failing
|
||||
`sb-gate-mxgw` while node-a is clean, then both clean once site-a-b has site-a's KEK again;
|
||||
- check 6 (was 5) also asserts the reply's `secretsRemoved` names the dropped secret;
|
||||
- checks 7 and 8 (were 6 and 7) also assert the CLI exits 1;
|
||||
- check 9 (was 8) now expects five `DeploySiteSecrets` rows — the removal-only deploy of check 6 writes one,
|
||||
with `removed: [sb-gate-mxgw]`.
|
||||
|
||||
`bash docker/verify-site-secrets.sh` → **PASS 9/9**. Before the check, central reported both site-a node
|
||||
rows with no failures; site-b and site-c (unreplicated) report their active node only. With the wrong KEK,
|
||||
node-b's `sb-gate-mxgw` failure appeared on central's site-a report 57 s after site-a-b was back in the cluster
|
||||
(mirror pass 15 s + status reporter 30 s + health report 30 s), and cleared 57 s after the KEK was restored.
|
||||
Standby convergence: check 3 = 4.3 s, check 4 = 4.2 s, check 6 = 15.1 s. Audit: the 5 expected rows, 5 distinct
|
||||
deployment ids; 0 plaintext/ciphertext hits in all 8 node logs and the audit payloads. Teardown left all three
|
||||
sites *Current* and site-a-b on site-a's KEK.
|
||||
|
||||
Not exercised live: the central secret-store outage (review M5) — taking the shared SQL store down on the rig
|
||||
also takes down central's configuration database. Covered by unit tests (`SiteSecretsCollectorTests`,
|
||||
`ArtifactDeploymentServiceTests.StoreOutage_*`, `ComputeCurrentHashes_StoreOutage_*`,
|
||||
`RetryForSite_StoreOutage_*`).
|
||||
|
||||
Tests: `dotnet build ZB.MOM.WW.ScadaBridge.slnx` 0 errors / 0 warnings; every non-Playwright test project green
|
||||
(CLI.Tests 553 after updating `DeploymentDeadlineParityTests`' stub server to reply with a real zero-failure
|
||||
summary — a bare `{}` now reads as an invalid artifact-deploy summary and exits 1). Playwright not run.
|
||||
|
||||
Reference in New Issue
Block a user