dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 2 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

feat(secrets): bump ZB.MOM.WW.Secrets family to 0.5.1; gate re-drill PASS on the fixed migrator

Browse Source 
0.5.1 (scadaproj 31ca940) fixes the virgin-DB concurrent migrator race this gate
found: the retry filter now covers 2714/1913/2627 alongside deadlock 1205. Rig
rebuilt on the bumped pins and the exact trigger re-drilled — ZbSecretsHub dropped
and recreated empty, both centrals started in one docker invocation — and both
booted clean in the same second (schema provisioned once, /health/ready 200 both,
no 2714, no wedge), where 0.5.0 crashed central-a under identical conditions.
Convergence re-smoked on the new image (13 s, decrypt-verified). Gate doc amended:
defect 1 disposition FIXED in 0.5.1 with the re-drill evidence; defect 2
(pre-Serilog wedge) remains open pending its own issue.

Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
This commit is contained in:
Joseph Doherty
2026-08-07 11:39:28 -04:00
parent 4d7f09d550
commit e697477c1f
2 changed files with 67 additions and 26 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Directory.Packages.props
+5 -5
View File
@@ -104,11 +104,11 @@
<PackageVersion Include="ZB.MOM.WW.Auth.AspNetCore" Version="0.1.5" />
<PackageVersion Include="ZB.MOM.WW.Audit" Version="0.1.0" />
<PackageVersion Include="ZB.MOM.WW.Theme" Version="0.3.1" />
<PackageVersion Include="ZB.MOM.WW.Secrets" Version="0.5.0" />
<PackageVersion Include="ZB.MOM.WW.Secrets.Abstractions" Version="0.5.0" />
<PackageVersion Include="ZB.MOM.WW.Secrets.Ui" Version="0.5.0" />
<PackageVersion Include="ZB.MOM.WW.Secrets.Replicator.SqlServer" Version="0.5.0" />
<PackageVersion Include="ZB.MOM.WW.Secrets.Replicator.Grpc" Version="0.5.0" />
<PackageVersion Include="ZB.MOM.WW.Secrets" Version="0.5.1" />
<PackageVersion Include="ZB.MOM.WW.Secrets.Abstractions" Version="0.5.1" />
<PackageVersion Include="ZB.MOM.WW.Secrets.Ui" Version="0.5.1" />
<PackageVersion Include="ZB.MOM.WW.Secrets.Replicator.SqlServer" Version="0.5.1" />
<PackageVersion Include="ZB.MOM.WW.Secrets.Replicator.Grpc" Version="0.5.1" />
<PackageVersion Include="ZB.MOM.WW.LocalDb" Version="0.1.3" />
<PackageVersion Include="ZB.MOM.WW.LocalDb.Replication" Version="0.1.3" />
<PackageVersion Include="ZB.MOM.WW.LocalDb.Contracts" Version="0.1.3" />
docs/plans/2026-08-07-secrets-central-shared-store-live-gate.md
+62 -21
View File
@@ -1,11 +1,17 @@
# Secrets central shared SQL store + hub failover — live gate on the docker cluster
**Result: PASS — 5/5 checks.** Run 2026-08-07 on the local 8-node docker cluster.
Two defects were found and are **documented, not patched** (standing rule): one in the
`ZB.MOM.WW.Secrets.Replicator.SqlServer` library (concurrent first-boot migration race),
one in the ScadaBridge Host's pre-Serilog crash path (unhandled boot exception wedges the
process instead of exiting). Neither is a failure of a gate check — every check's own
criterion passed — but both need a controller decision. See [Defects](#defects-found).
**Result: PASS — 5/5 checks, plus the 0.5.1 re-drill.** Run 2026-08-07 on the local
8-node docker cluster in two passes: the **first pass** (secrets libs **0.5.0**, image
`b41249cc7bf8`) passed all five checks and surfaced two defects, **documented, not
patched** (standing rule) — one in the `ZB.MOM.WW.Secrets.Replicator.SqlServer` library
(concurrent first-boot migration race), one in the ScadaBridge Host's pre-Serilog crash
path (unhandled boot exception wedges the process instead of exiting). The library
defect was **fixed at the library and shipped as 0.5.1** (`scadaproj` main `31ca940`);
the **second pass** (libs **0.5.1**, image `4b0c36398cb7`) re-ran the exact virgin-DB
concurrent bring-up drill that exposed it — both centrals booted clean in the same
second — and re-smoked end-to-end convergence. The Host wedge (defect 2) **remains
open**, to be filed as its own ScadaBridge issue at closeout. See
[Defects](#defects-found).
## Purpose
@@ -28,14 +34,19 @@ negative (`EnsureCentralSharedStoreConnectionString` throwing the designed messa
## Rig state
| Fact | First pass (checks 1–5) | Second pass (0.5.1 re-drill) |
|---|---|---|
| Repo / branch | `ScadaBridge` @ `main` | same |
| Git SHA gated | `68f812eaa4b29b0657ef2d967b1fa0dfc105c214` (`68f812ea`) | `68f812ea` + the 0.5.1 pin bump (committed with this amendment) |
| Image | `scadabridge:latest` = `b41249cc7bf8`, built 15:04:42Z | `4b0c36398cb7`, built 15:36:41Z |
| Secrets libs | `ZB.MOM.WW.Secrets*` **0.5.0** (Grpc dll informational version `0.5.0+7a2980e`, `FailoverSecretsHubReader` present) | **0.5.1** — verified in-container: `0.5.1+e62b4a5…` (SourceRevisionId is a pack-before-commit stamp; the fix is provably in the binary — `IsRetryableMigrationError` present, the old `MaxDeadlockRetries` gone) |
First-pass binary identity:
| Fact | Value |
|---|---|
| Repo / branch | `ScadaBridge` @ `main` |
| Git SHA gated | `68f812eaa4b29b0657ef2d967b1fa0dfc105c214` (`68f812ea`) |
| Image | `scadabridge:latest` = `b41249cc7bf8`, built 15:04:42Z |
| `ZB.MOM.WW.ScadaBridge.Host.dll` | `9dba0869ff4dfa7c…` — byte-identical on all 4 participating nodes |
| `ZB.MOM.WW.Secrets.Replicator.Grpc.dll` | `4c837f627e901641…`, informational version **`0.5.0+7a2980e`**, `FailoverSecretsHubReader` present — byte-identical on all 4 |
| Secrets libs | `ZB.MOM.WW.Secrets*` **0.5.0** |
| `ZB.MOM.WW.Secrets.Replicator.Grpc.dll` | `4c837f627e901641…` — byte-identical on all 4 |
| Hub enabled on | `central-a`, `central-b` (hub, **shared SQL store**) and `site-a-a`, `site-a-b` (followers, local SQLite) |
| Hub NOT enabled on | `site-b-*`, `site-c-*` — verified: zero `Secrets__*` env vars, zero hub/sweep log lines |
| Sweep interval | `00:00:30` — the product default, not shortened; timings below are real |
@@ -285,8 +296,9 @@ lines.
## Defects found
**None against any check's criterion.** Two real defects surfaced around the checks —
per the gate's standing rule, **documented and NOT patched**; both need a controller
decision.
per the gate's standing rule, documented and not patched by the gate itself. Defect 1
has since been **fixed at the library (0.5.1) and re-drilled on this rig**; defect 2
**remains open** pending its own ScadaBridge issue.
### 1. Library — `SqlServerSecretsStoreMigrator` concurrent first-boot race (error 2714)
@@ -313,9 +325,36 @@ restart at 15:09:59Z and every recreate since booted clean), so the blast radius
one-time-per-virgin-DB — but it lands on the very first production bring-up, and combined
with defect 2 it left the node wedged rather than restart-recovered.
**Suggested direction (for the controller, not applied):** include 2714 (and arguably
2617/1913-class "already exists" errors) in the retry filter, or catch-and-recheck —
retrying is exactly as correct for this race as for the deadlock it already retries.
**Disposition: FIXED in 0.5.1** (`scadaproj` main `31ca940`, released to the Gitea feed
the same day). The retry filter now retries 1205/2714/1913/2627, scanning the whole
`SqlException.Errors` collection (2714 arrives bundled with 2759 "CREATE SCHEMA failed
due to previous errors"), capped at 5 attempts, and the class's "safe … concurrently"
doc overclaim is corrected. ScadaBridge bumped all five `ZB.MOM.WW.Secrets*` pins to
0.5.1 and the rig was rebuilt (image `4b0c36398cb7`; in-container
`ZB.MOM.WW.Secrets.Replicator.SqlServer.dll` verified to carry
`IsRetryableMigrationError` and not the old `MaxDeadlockRetries`).
**Re-drill on 0.5.1 — PASS.** The exact trigger was reproduced: both centrals stopped,
`ZbSecretsHub` **dropped and recreated empty** (0 tables, same `scadabridge_app` grant),
then both centrals started **in the same `docker start` invocation** (15:37:41Z):
```
central-a [15:37:42 INF] Application started. Press Ctrl+C to shut down.
central-b [15:37:42 INF] Application started. Press Ctrl+C to shut down.
zbsecrets.schema_version: version=1 applied_utc=2026-08-07T15:37:42.7099417+00:00
central-a /health/ready=200 central-b /health/ready=200 RestartCount=0 both, CPU ~4–5%
```
Both nodes booted **in the same second** against the virgin database, the schema was
provisioned exactly once, and there was no 2714, no crash, no wedge — where the 0.5.0
first pass crashed central-a at 15:04:55Z under identical conditions. Honest bound: the
migrator runs pre-Serilog with no logger, so a fired retry is not itself visible in
logs; the evidence is the trigger conditions reproduced plus the outcome flipped.
Convergence re-smoked end-to-end on the 0.5.1 image: `shared-gate-051` seeded into the
(freshly wiped) shared store at 15:38:11Z, pulled by `site-a-a` at 15:38:24Z (**13 s**)
and decrypt-verified. (The DB drop wiped the first pass's rows from the hub; site stores
keep their local copies — pull-only never deletes — so the final-state table above
reflects the first pass.)
### 2. ScadaBridge Host — pre-Serilog unhandled boot exception wedges the process instead of exiting
@@ -334,8 +373,10 @@ So it is not specific to SqlClient or to the new SQL path — any throw in the p
window (including the pre-existing SQLite path's failure modes) turns a clean fail-fast
into a silent wedge that needs a manual restart, and monitoring that watches container
state sees a healthy-looking `running` container. Mechanism not root-caused here (gate
scope); the observable and two reproductions are recorded. Worth a dedicated issue —
fail-closed is only honest if the process actually exits.
scope); the observable and two reproductions are recorded.
**Disposition: OPEN** — to be filed as its own ScadaBridge issue at closeout (controller
decision 2026-08-07). Fail-closed is only honest if the process actually exits.
### Gate-method notes (not defects)
@@ -349,9 +390,9 @@ fail-closed is only honest if the process actually exits.
## Residuals
1. **Defects 1 and 2 above** — filed for controller disposition; neither blocks the
topology (defect 1 is one-time-per-virgin-DB and recoverable by restart; defect 2 is
pre-existing behavior of the pre-host window, now with two live reproductions).
1. **Defect 2 above** (pre-Serilog wedge) — open, pending its own ScadaBridge issue;
pre-existing behavior of the pre-host window, now with two live reproductions.
Defect 1 is closed: fixed in 0.5.1 and re-drilled on this rig (see its disposition).
2. **`site-a-b`'s one-tick sweep failure after a central restart** (stale subchannel) —
same bounded, self-healing artifact the previous gate recorded; worst-case post-outage
convergence is two sweep intervals.