From e697477c1f00a258ac41a4a280ef9214839751c7 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Fri, 7 Aug 2026 11:39:28 -0400 Subject: [PATCH] feat(secrets): bump ZB.MOM.WW.Secrets family to 0.5.1; gate re-drill PASS on the fixed migrator MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 0.5.1 (scadaproj 31ca940) fixes the virgin-DB concurrent migrator race this gate found: the retry filter now covers 2714/1913/2627 alongside deadlock 1205. Rig rebuilt on the bumped pins and the exact trigger re-drilled — ZbSecretsHub dropped and recreated empty, both centrals started in one docker invocation — and both booted clean in the same second (schema provisioned once, /health/ready 200 both, no 2714, no wedge), where 0.5.0 crashed central-a under identical conditions. Convergence re-smoked on the new image (13 s, decrypt-verified). Gate doc amended: defect 1 disposition FIXED in 0.5.1 with the re-drill evidence; defect 2 (pre-Serilog wedge) remains open pending its own issue. Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1 --- Directory.Packages.props | 10 +-- ...-secrets-central-shared-store-live-gate.md | 83 ++++++++++++++----- 2 files changed, 67 insertions(+), 26 deletions(-) diff --git a/Directory.Packages.props b/Directory.Packages.props index e80bb109..699030d3 100644 --- a/Directory.Packages.props +++ b/Directory.Packages.props @@ -104,11 +104,11 @@ - - - - - + + + + + diff --git a/docs/plans/2026-08-07-secrets-central-shared-store-live-gate.md b/docs/plans/2026-08-07-secrets-central-shared-store-live-gate.md index e76b2fbe..95d8af62 100644 --- a/docs/plans/2026-08-07-secrets-central-shared-store-live-gate.md +++ b/docs/plans/2026-08-07-secrets-central-shared-store-live-gate.md @@ -1,11 +1,17 @@ # Secrets central shared SQL store + hub failover — live gate on the docker cluster -**Result: PASS — 5/5 checks.** Run 2026-08-07 on the local 8-node docker cluster. -Two defects were found and are **documented, not patched** (standing rule): one in the -`ZB.MOM.WW.Secrets.Replicator.SqlServer` library (concurrent first-boot migration race), -one in the ScadaBridge Host's pre-Serilog crash path (unhandled boot exception wedges the -process instead of exiting). Neither is a failure of a gate check — every check's own -criterion passed — but both need a controller decision. See [Defects](#defects-found). +**Result: PASS — 5/5 checks, plus the 0.5.1 re-drill.** Run 2026-08-07 on the local +8-node docker cluster in two passes: the **first pass** (secrets libs **0.5.0**, image +`b41249cc7bf8`) passed all five checks and surfaced two defects, **documented, not +patched** (standing rule) — one in the `ZB.MOM.WW.Secrets.Replicator.SqlServer` library +(concurrent first-boot migration race), one in the ScadaBridge Host's pre-Serilog crash +path (unhandled boot exception wedges the process instead of exiting). The library +defect was **fixed at the library and shipped as 0.5.1** (`scadaproj` main `31ca940`); +the **second pass** (libs **0.5.1**, image `4b0c36398cb7`) re-ran the exact virgin-DB +concurrent bring-up drill that exposed it — both centrals booted clean in the same +second — and re-smoked end-to-end convergence. The Host wedge (defect 2) **remains +open**, to be filed as its own ScadaBridge issue at closeout. See +[Defects](#defects-found). ## Purpose @@ -28,14 +34,19 @@ negative (`EnsureCentralSharedStoreConnectionString` throwing the designed messa ## Rig state +| Fact | First pass (checks 1–5) | Second pass (0.5.1 re-drill) | +|---|---|---| +| Repo / branch | `ScadaBridge` @ `main` | same | +| Git SHA gated | `68f812eaa4b29b0657ef2d967b1fa0dfc105c214` (`68f812ea`) | `68f812ea` + the 0.5.1 pin bump (committed with this amendment) | +| Image | `scadabridge:latest` = `b41249cc7bf8`, built 15:04:42Z | `4b0c36398cb7`, built 15:36:41Z | +| Secrets libs | `ZB.MOM.WW.Secrets*` **0.5.0** (Grpc dll informational version `0.5.0+7a2980e`, `FailoverSecretsHubReader` present) | **0.5.1** — verified in-container: `0.5.1+e62b4a5…` (SourceRevisionId is a pack-before-commit stamp; the fix is provably in the binary — `IsRetryableMigrationError` present, the old `MaxDeadlockRetries` gone) | + +First-pass binary identity: + | Fact | Value | |---|---| -| Repo / branch | `ScadaBridge` @ `main` | -| Git SHA gated | `68f812eaa4b29b0657ef2d967b1fa0dfc105c214` (`68f812ea`) | -| Image | `scadabridge:latest` = `b41249cc7bf8`, built 15:04:42Z | | `ZB.MOM.WW.ScadaBridge.Host.dll` | `9dba0869ff4dfa7c…` — byte-identical on all 4 participating nodes | -| `ZB.MOM.WW.Secrets.Replicator.Grpc.dll` | `4c837f627e901641…`, informational version **`0.5.0+7a2980e`**, `FailoverSecretsHubReader` present — byte-identical on all 4 | -| Secrets libs | `ZB.MOM.WW.Secrets*` **0.5.0** | +| `ZB.MOM.WW.Secrets.Replicator.Grpc.dll` | `4c837f627e901641…` — byte-identical on all 4 | | Hub enabled on | `central-a`, `central-b` (hub, **shared SQL store**) and `site-a-a`, `site-a-b` (followers, local SQLite) | | Hub NOT enabled on | `site-b-*`, `site-c-*` — verified: zero `Secrets__*` env vars, zero hub/sweep log lines | | Sweep interval | `00:00:30` — the product default, not shortened; timings below are real | @@ -285,8 +296,9 @@ lines. ## Defects found **None against any check's criterion.** Two real defects surfaced around the checks — -per the gate's standing rule, **documented and NOT patched**; both need a controller -decision. +per the gate's standing rule, documented and not patched by the gate itself. Defect 1 +has since been **fixed at the library (0.5.1) and re-drilled on this rig**; defect 2 +**remains open** pending its own ScadaBridge issue. ### 1. Library — `SqlServerSecretsStoreMigrator` concurrent first-boot race (error 2714) @@ -313,9 +325,36 @@ restart at 15:09:59Z and every recreate since booted clean), so the blast radius one-time-per-virgin-DB — but it lands on the very first production bring-up, and combined with defect 2 it left the node wedged rather than restart-recovered. -**Suggested direction (for the controller, not applied):** include 2714 (and arguably -2617/1913-class "already exists" errors) in the retry filter, or catch-and-recheck — -retrying is exactly as correct for this race as for the deadlock it already retries. +**Disposition: FIXED in 0.5.1** (`scadaproj` main `31ca940`, released to the Gitea feed +the same day). The retry filter now retries 1205/2714/1913/2627, scanning the whole +`SqlException.Errors` collection (2714 arrives bundled with 2759 "CREATE SCHEMA failed +due to previous errors"), capped at 5 attempts, and the class's "safe … concurrently" +doc overclaim is corrected. ScadaBridge bumped all five `ZB.MOM.WW.Secrets*` pins to +0.5.1 and the rig was rebuilt (image `4b0c36398cb7`; in-container +`ZB.MOM.WW.Secrets.Replicator.SqlServer.dll` verified to carry +`IsRetryableMigrationError` and not the old `MaxDeadlockRetries`). + +**Re-drill on 0.5.1 — PASS.** The exact trigger was reproduced: both centrals stopped, +`ZbSecretsHub` **dropped and recreated empty** (0 tables, same `scadabridge_app` grant), +then both centrals started **in the same `docker start` invocation** (15:37:41Z): + +``` +central-a [15:37:42 INF] Application started. Press Ctrl+C to shut down. +central-b [15:37:42 INF] Application started. Press Ctrl+C to shut down. +zbsecrets.schema_version: version=1 applied_utc=2026-08-07T15:37:42.7099417+00:00 +central-a /health/ready=200 central-b /health/ready=200 RestartCount=0 both, CPU ~4–5% +``` + +Both nodes booted **in the same second** against the virgin database, the schema was +provisioned exactly once, and there was no 2714, no crash, no wedge — where the 0.5.0 +first pass crashed central-a at 15:04:55Z under identical conditions. Honest bound: the +migrator runs pre-Serilog with no logger, so a fired retry is not itself visible in +logs; the evidence is the trigger conditions reproduced plus the outcome flipped. +Convergence re-smoked end-to-end on the 0.5.1 image: `shared-gate-051` seeded into the +(freshly wiped) shared store at 15:38:11Z, pulled by `site-a-a` at 15:38:24Z (**13 s**) +and decrypt-verified. (The DB drop wiped the first pass's rows from the hub; site stores +keep their local copies — pull-only never deletes — so the final-state table above +reflects the first pass.) ### 2. ScadaBridge Host — pre-Serilog unhandled boot exception wedges the process instead of exiting @@ -334,8 +373,10 @@ So it is not specific to SqlClient or to the new SQL path — any throw in the p window (including the pre-existing SQLite path's failure modes) turns a clean fail-fast into a silent wedge that needs a manual restart, and monitoring that watches container state sees a healthy-looking `running` container. Mechanism not root-caused here (gate -scope); the observable and two reproductions are recorded. Worth a dedicated issue — -fail-closed is only honest if the process actually exits. +scope); the observable and two reproductions are recorded. + +**Disposition: OPEN** — to be filed as its own ScadaBridge issue at closeout (controller +decision 2026-08-07). Fail-closed is only honest if the process actually exits. ### Gate-method notes (not defects) @@ -349,9 +390,9 @@ fail-closed is only honest if the process actually exits. ## Residuals -1. **Defects 1 and 2 above** — filed for controller disposition; neither blocks the - topology (defect 1 is one-time-per-virgin-DB and recoverable by restart; defect 2 is - pre-existing behavior of the pre-host window, now with two live reproductions). +1. **Defect 2 above** (pre-Serilog wedge) — open, pending its own ScadaBridge issue; + pre-existing behavior of the pre-host window, now with two live reproductions. + Defect 1 is closed: fixed in 0.5.1 and re-drilled on this rig (see its disposition). 2. **`site-a-b`'s one-tick sweep failure after a central restart** (stale subchannel) — same bounded, self-healing artifact the previous gate recorded; worst-case post-outage convergence is two sweep intervals.