Secrets:SqlitePath is now validated at startup by SecretsOptionsValidator using two rules in the new public SecretsStorePathRules. Both come from production incidents and neither implies the other. Must be absolute: a relative value resolves against the process working directory, which for a Windows service is C:\Windows\System32, so the store lands outside the installation; in a dev loop it lands in the source tree (a gateway was observed writing its secrets database into src/, tripping that repo's own tree-hygiene test). Must not be inside the content root: an absolute path can still point into the directory every deployment procedure in this family renames on upgrade. On 2026-08-09 a gateway's credential store sat inside its Server\ directory; the deploy renamed Server to Server.bak.*, the database went with it, and the process created a fresh empty one at the same path. Every consumer lost its credential, silently, for two days. The existing "must be absolute" convention did not help — the path was absolute. The rules reject rather than relocate. Silently moving a credential store presents later as an authentication failure pointing nowhere near the configuration that caused it; a boot error is seen immediately by whoever caused it. Validation is deliberately not exempted in Development: a guard that never runs in a dev loop or in CI rots, and the configuration behind the outage looked entirely reasonable. BREAKING for configuration, not API. SecretsOptions.SqlitePath now defaults to an absolute per-user path under LocalApplicationData; it was "secrets.db", which the new rules reject. An application configuring a relative path will fail to boot — remove the key or set an absolute one outside the content root. All five deployed instances of this family were checked first and already set an absolute path outside their content root, so the migration edge (a host relying on the OLD default now finding nothing and creating an empty store — this bug, re-introduced by its own fix) does not apply to any of them. Per-user rather than machine-wide is deliberate: CommonApplicationData is /usr/share on Linux and macOS, which a normal user cannot write, so a machine-wide library default would fail every consuming application's boot on a developer machine. An application may compute its own default; the configured value always wins. Validator registration is idempotent. AddZbSecrets is legitimately called more than once in real compositions, and TryAddEnumerable cannot de-duplicate a factory-built descriptor — it de-duplicates on implementation type, which a factory does not carry, so it throws on the second call rather than skipping. The existing suite caught this. Tests cover both rules including the segment-boundary case (/srv/app-data must not read as inside /srv/app — a false positive on a fail-closed startup rule is an application that will not boot), that the shipped default satisfies the rules it ships with, that the default is not machine-wide, idempotent registration, and — most importantly — that a bad path actually throws when IOptions is resolved, since a correctly implemented validator that is never invoked is the same "guard that does not fire" failure this change exists to prevent. Family bumps to 0.6.0. Adds a dependency on ZB.MOM.WW.Configuration 0.1.0 for the shared OptionsValidatorBase/ValidationBuilder vocabulary, which every consumer already references.
12 KiB
Changelog
All notable changes to the ZB.MOM.WW.Secrets family are recorded here. Format loosely
follows Keep a Changelog; versions are shared across every
package in the family (see Directory.Build.props) even when a given release only touches
one of them.
0.6.0 — 2026-08-11
Added
-
ZB.MOM.WW.Secrets— fail-closed validation of the store path.Secrets:SqlitePathis now validated at startup (ValidateOnStart) bySecretsOptionsValidator, using two rules in the new publicSecretsStorePathRules. Both come from production incidents and neither implies the other:- Must be absolute. A relative value resolves against the process working directory —
for a Windows service that is
C:\Windows\System32, so the store lands outside the installation entirely; in a dev loop it lands in the source tree (observed: a gateway wrote its secrets database intosrc/and tripped that repo's own tree-hygiene test). - Must not be inside the application content root. An absolute path can still point
into the directory every deployment procedure in this family replaces or renames on
upgrade. On 2026-08-09 a gateway's credential store sat inside its
Server\directory; the deploy renamedServertoServer.bak.*, the database left with it, and the process created a fresh empty one at the same path. Every consumer lost its credential, silently, for two days. The existing "absolute path" convention did not help — the path was absolute.
The rules reject rather than relocate: silently moving a credential store presents later as an authentication failure pointing nowhere near the configuration that caused it, whereas a boot error is seen immediately by whoever caused it. Validation is deliberately not exempted in Development — a guard that never runs in a dev loop or in CI rots, and the configuration behind the outage looked entirely reasonable.
The content-root rule is skipped when no
IHostEnvironmentis available (the CLI composes the subsystem without a host); the absolute-path rule still applies. - Must be absolute. A relative value resolves against the process working directory —
for a Windows service that is
Changed
-
BREAKING (configuration, not API):
SecretsOptions.SqlitePathnow defaults to an absolute per-user path —ZB.MOM.WW/secrets.dbunderLocalApplicationData(~/.local/shareon Linux and macOS,%LOCALAPPDATA%on Windows), exposed asSecretsOptions.DefaultSqlitePath. It wassecrets.db, which the new rules reject.A consuming application that configures a relative
Secrets:SqlitePathwill now fail to boot. The fix is to remove the key (accepting the default) or set an absolute path outside the content root. Every deployment of this family already sets an absolute path, so this bites dev loops and unconfigured installs, not production — but verify before upgrading: a host that relied on the old default now looks in a new location, finds nothing, and creates an empty store, which is the same failure this release exists to prevent.Per-user rather than machine-wide is deliberate.
CommonApplicationDatais/usr/shareon Linux and macOS, which a normal user cannot write; as a library default that would fail every consuming application's boot on a developer machine. An application is free to compute its own default (a machine-wide Windows service reasonably prefersCommonApplicationDataalongside its other stores) — the configured value always wins. -
AddZbSecretsregistration of the validator is idempotent.AddZbSecretsis legitimately called more than once in real compositions (a replication package calls it after the application already has), andTryAddEnumerablecannot de-duplicate a factory-built descriptor — it de-duplicates on implementation type, which a factory does not carry, so it throws on the second call rather than skipping. -
ZB.MOM.WW.Secretsnow depends onZB.MOM.WW.Configuration(0.1.0) for the sharedOptionsValidatorBase/ValidationBuildervocabulary. Every consumer already references it.
0.5.1 — 2026-08-07
Fixed
ZB.MOM.WW.Secrets.Replicator.SqlServer— virgin-database concurrent migrator race (present since 0.2.0; found by the ScadaBridge central shared-store live gate). The migrator's doc claimed simultaneous provisioning of a virgin database converges, but its retry filter covered only deadlock victims (1205): Serializable isolation does not serialize theIF NOT EXISTSmetadata probes against another session's in-flightCREATE, so of two nodes booting together against an empty database the loser died withSqlException2714 ("There is already an object named 'zbsecrets'") and aborted its host — observed live on an ordinarydocker compose upof the ScadaBridge central pair. The retry filter now also treats the lost-CREATE/lost-INSERT race outcomes as retryable — 2714 (schema/table already exists), 1913 (index already exists), 2627 (schema_version MERGE-insert PK violation) — since losing such a race means the object now exists, which is the desired end state; the retry re-runs the idempotentIF NOT EXISTSpath and proceeds. Classification scans the wholeSqlException.Errorscollection (a failed DDL batch stacks follow-on errors such as 2759, so the retryable number is not guaranteed first), and the retry cap (5) keeps a genuine conflict — e.g. an unrelated pre-existing object colliding with the schema name — failing closed. 2705 (duplicate column) is deliberately not retried: every column lives inside a singleCREATE TABLEstatement, so no cross-session race can produce it. The error-number predicate is now an internal static pinned by offline tests, and the live suite gains an N-concurrent-migrators virgin-database test.
0.5.0 — 2026-08-07
Added
ZB.MOM.WW.Secrets.Replicator.Grpc— hub endpoint failover for followers (scadaproj#4).SecretsGrpcHubClientOptionsgainsFallbackEndpoints: additional hub addresses tried, in order, when a call to the primaryEndpoint(or an earlier fallback) fails with a transport-level gRPC error. The answering endpoint is sticky — a sweep that failed over does not re-pay the dead endpoint's fullCallDeadlineon every subsequent call — and rotation wraps, so a recovered primary is retried the next time the stuck-to endpoint fails. OnlyRpcExceptiontriggers failover; cancellation and everything else propagate untouched, and when every endpoint fails the lastRpcExceptionis rethrown so the sweep degrades to last-known-good exactly as before. Each failover logs a Warning naming the failed and next endpoint URIs (never any credential material).- Same-store requirement: list only endpoints that serve the SAME store — for ScadaBridge, the central pair over one shared SQL Server store. Endpoints backed by independent stores turn a primary outage into a silent convergence stop (failing over to an emptier hub "succeeds"), which is the exact defect scadaproj#4 recorded.
- Validation fails startup on a blank/relative fallback or one duplicating the primary or another fallback (a duplicated endpoint silently halves the configured failover).
Unchanged
- With
FallbackEndpointsempty (the default), the registration graph is identical to 0.4.x — one keyed channel, oneGrpcSecretsHubClient, no failover wrapper — and the wire behavior of every call is byte-identical. Every other package in the family is behaviorally unchanged in this release.
0.4.1 — 2026-08-07
Fixed
ZB.MOM.WW.Secrets.Replicator.Grpc— the hub now logs its denials. A ScadaBridge live gate found thatSecretsHubAuthInterceptorrefused calls correctly but wrote nothing about it: the only trace of a refusal was a frameworkGrpc.AspNetCore.ServerInformation line, so a follower whose bearer token had been mis-rotated silently stopped converging and the hub showed it only at INF. Every denial is now counted, and aWarningsummarising the window is emitted on the first denial after startup or after 60 quiet seconds — carrying the number of denials since the previous summary plus a breakdown by cause (missing/malformed credential, wrong credential, hub token unconfigured). A single probe is therefore visible immediately, while a sustained attack is bounded to one log line per minute — the endpoint is unauthenticated by construction, so a per-denial warning would be an unbounded log write driven by an unauthenticated caller.
Unchanged
- Wire behavior is byte-identical. The cause breakdown is server-side diagnostics only; every
rejection still returns the same uniform
Unauthenticatedstatus with the same detail, and the credential comparison is still constant-time. No token material, presented or configured, appears in any log line at any level. - The existing warn-once-on-unconfigured-token line is unchanged and still emitted exactly once;
denials in that state also count toward the new summary's
unconfiguredcategory. - Every other package in the family is behaviorally unchanged in this release.
0.4.0 — 2026-08-07
Added
- New package
ZB.MOM.WW.Secrets.Replicator.Grpc— a central-hostedsecrets_hub.v1gRPC hub for the sites-talk-to-central production topology (scadaproj#3; design docdocs/plans/2026-08-07-secrets-production-topology-and-grpc-hub.md).- Pull-only by construction: the proto exposes no write RPC. Updates originate at central only; a site never pushes a secret upstream.
- Fail-closed bearer auth on the hub server: constant-time comparison, uniform denial response (no signal distinguishing "wrong token" from "missing token"), a warn-once log on an unconfigured/empty token rather than a silent open door.
GetSecretsenforces a batch-size cap, rejecting an oversized request withResourceExhaustedinstead of serving it partially or unbounded.- Site-side
GrpcSecretSyncServicesweeps the hub on an interval with per-row parse isolation — one malformed/undecryptable row logs and is skipped, it does not fault the whole sweep. - Keyed
GrpcChannelregistration so multiple hub targets can be wired in one process without channel collisions. - Offline behavior is last-known-good: if the hub is unreachable, the site keeps serving the most recently synced values rather than failing secret resolution.
Unchanged
ZB.MOM.WW.Secrets.Abstractions,ZB.MOM.WW.Secrets,ZB.MOM.WW.Secrets.Ui,ZB.MOM.WW.Secrets.Replicator.SqlServer, andZB.MOM.WW.Secrets.Replicator.AkkaDotNetare behaviorally unchanged in this release — no breaking changes to any existing package.
0.3.0 — 2026-07-19
- Menu-driven Spectre.Console interactive operator console added to the
secretCLI (list / set / reveal / delete, reference audit & seed, KEK doctor, ciphertext-only bundle export/import). Seedocs/operations/interactive-console.md.
0.2.3 — 2026-07-19
ZB.MOM.WW.Secrets.Ui: fixed the delete-confirmation modal being permanently invisible on every Bootstrap host (ConfirmDeleteModalused bareclass="modal"with no shipped CSS; Bootstrap's.modal{display:none}hid it everywhere but mxaccessgw). Fixed via azb-secrets-modal*class rename plus a component-emitted<style>block.
0.2.2 — 2026-07-18
- Fixed the Akka replicator's hosted-process DI deadlock (a circular singleton dependency
hidden behind factory lambdas that MS.DI's
StackGuardturned into a silent cross-thread hang rather than a stack overflow). NewDeferredSecretCacheInvalidatordefers the one cycle-closing edge to first eviction.
0.2.1 — 2026-07-18
- Fixed the Akka replicator being silently inert since 0.2.0 (a
TryAddregistration-order bug published into a no-op sink and never spawned an actor). SQL-Server replication was never affected.
0.2.0 — 2026-07-18
- Added clustered replication (G-7):
ZB.MOM.WW.Secrets.Replicator.SqlServer(shared store, or local-store-with-hub) andZB.MOM.WW.Secrets.Replicator.AkkaDotNet(peer-to-peer over distributed pub/sub).