Joseph Doherty
cbfe296279
Found by the HistorianGateway adoption of 0.6.0, which probed the rejected paths
instead of observing a clean boot — a clean boot being equally consistent with the
validator not being wired at all.
An application that expands ${secret:} references before building its host composes
this subsystem in a throwaway ServiceCollection with no IHostEnvironment in it, then
migrates the store from that container. SecretsOptionsValidator resolves the content
root from IHostEnvironment and skips the under-content-root rule when it is absent, so
pre-host an absolute in-content-root path passed the only rule still applying, the
migrator CREATED the database, and the boot failed a second later once the real host
registered IHostEnvironment and the rule finally fired.
The observable result was a rejected boot that still left an empty store, with -wal and
-shm siblings, at the rejected path — the same artifact that made the MxGateway outage
read as "the database is there, it's just empty", which is precisely what this guard
exists to prevent. A relative path was unaffected because the absolute-path rule fires
with or without a content root; that asymmetry is what exposed it.
The mistake was conflating "no IHostEnvironment registered" with "no content root
exists". New AddZbSecrets overload takes the content root explicitly; the 3-argument
overload is unchanged and still falls back to IHostEnvironment, so this is additive.
SecretsOptionsValidator gains a string? constructor alongside the IHostEnvironment? one.
Reproduced first in an isolated harness before changing anything: a bare host rejects
both bad paths leaving no artifacts, and a bare ServiceCollection with no
IHostEnvironment migrates an in-content-root path to disk. Both are now regression tests,
the second paired with a positive control asserting the old behaviour still holds when no
content root is supplied — so the test is about the explicit content root rather than
about paths in general.
Also documents an upgrade note that applies to 0.6.0: a store seeded under the old
relative default is orphaned rather than migrated, and nothing signals it.
243 lines
15 KiB
Markdown
243 lines
15 KiB
Markdown
# Changelog
|
|
|
|
All notable changes to the `ZB.MOM.WW.Secrets` family are recorded here. Format loosely
|
|
follows [Keep a Changelog](https://keepachangelog.com/); versions are shared across every
|
|
package in the family (see `Directory.Build.props`) even when a given release only touches
|
|
one of them.
|
|
|
|
## 0.6.1 — 2026-08-11
|
|
|
|
### Fixed
|
|
|
|
- **The 0.6.0 guard had a hole at exactly the moment the store file is created** (found by the
|
|
HistorianGateway adoption of 0.6.0, which probed the rejected paths rather than observing a
|
|
clean boot). An application that expands `${secret:}` references *before* building its host
|
|
composes this subsystem in a throwaway `ServiceCollection` with **no `IHostEnvironment` in it**,
|
|
then migrates the store from that container. `SecretsOptionsValidator` resolves the content root
|
|
from `IHostEnvironment` and skips the under-content-root rule when it is absent — so pre-host an
|
|
absolute in-content-root path passed the only rule still applying, the migrator **created the
|
|
database**, and the boot then failed a second later once the real host registered
|
|
`IHostEnvironment` and the rule finally fired.
|
|
|
|
The observable result was a rejected boot that *still left an empty store, with `-wal`/`-shm`
|
|
siblings, sitting at the rejected path* — the same artifact that made the outage this guard
|
|
exists to prevent read as "the database is there, it's just empty". A relative path was
|
|
unaffected, because the absolute-path rule fires with or without a content root; that asymmetry
|
|
is what exposed it.
|
|
|
|
Conflating "no `IHostEnvironment` registered" with "no content root exists" was the mistake. New
|
|
overload **`AddZbSecrets(services, config, sectionPath, contentRootPath)`** takes the content
|
|
root explicitly; the 3-argument overload is unchanged and still falls back to `IHostEnvironment`.
|
|
**Pre-host composition should pass it** — e.g. `builder.Environment.ContentRootPath`. Passing
|
|
`null` remains correct only where there genuinely is no content root, such as the CLI.
|
|
`SecretsOptionsValidator` gains a `string?` constructor alongside the `IHostEnvironment?` one.
|
|
|
|
### Upgrade note (applies to 0.6.0 as well)
|
|
|
|
- **A store seeded under the pre-0.6.0 default is orphaned, not migrated, and nothing says so.**
|
|
The old default was the relative `secrets.db`, which resolved against the process working
|
|
directory; the new default is absolute. An application that relied on the old default silently
|
|
starts using a different file, and anything previously seeded into the old one is stranded. This
|
|
is a dev-loop concern for the apps in this family (every deployment sets an explicit absolute
|
|
path), but check for a stray `*-secrets.db` beside your project before assuming an empty store
|
|
after upgrade means the store was lost. Deleting the key from configuration, rather than leaving
|
|
a relative value, is what makes the new default apply.
|
|
|
|
## 0.6.0 — 2026-08-11
|
|
|
|
### Added
|
|
|
|
- **`ZB.MOM.WW.Secrets` — fail-closed validation of the store path.** `Secrets:SqlitePath` is
|
|
now validated at startup (`ValidateOnStart`) by `SecretsOptionsValidator`, using two rules in
|
|
the new public `SecretsStorePathRules`. Both come from production incidents and neither
|
|
implies the other:
|
|
- **Must be absolute.** A relative value resolves against the process working directory —
|
|
for a Windows service that is `C:\Windows\System32`, so the store lands outside the
|
|
installation entirely; in a dev loop it lands in the source tree (observed: a gateway wrote
|
|
its secrets database into `src/` and tripped that repo's own tree-hygiene test).
|
|
- **Must not be inside the application content root.** An *absolute* path can still point
|
|
into the directory every deployment procedure in this family replaces or renames on
|
|
upgrade. On 2026-08-09 a gateway's credential store sat inside its `Server\` directory; the
|
|
deploy renamed `Server` to `Server.bak.*`, the database left with it, and the process
|
|
created a fresh empty one at the same path. Every consumer lost its credential, silently,
|
|
for two days. The existing "absolute path" convention did not help — the path *was*
|
|
absolute.
|
|
|
|
The rules **reject rather than relocate**: silently moving a credential store presents later
|
|
as an authentication failure pointing nowhere near the configuration that caused it, whereas
|
|
a boot error is seen immediately by whoever caused it. Validation is deliberately **not**
|
|
exempted in Development — a guard that never runs in a dev loop or in CI rots, and the
|
|
configuration behind the outage looked entirely reasonable.
|
|
|
|
The content-root rule is skipped when no `IHostEnvironment` is available (the CLI composes
|
|
the subsystem without a host); the absolute-path rule still applies.
|
|
|
|
### Changed
|
|
|
|
- **BREAKING (configuration, not API): `SecretsOptions.SqlitePath` now defaults to an absolute
|
|
per-user path** — `ZB.MOM.WW/secrets.db` under `LocalApplicationData` (`~/.local/share` on
|
|
Linux and macOS, `%LOCALAPPDATA%` on Windows), exposed as `SecretsOptions.DefaultSqlitePath`.
|
|
It was `secrets.db`, which the new rules reject.
|
|
|
|
**A consuming application that configures a relative `Secrets:SqlitePath` will now fail to
|
|
boot.** The fix is to remove the key (accepting the default) or set an absolute path outside
|
|
the content root. Every deployment of this family already sets an absolute path, so this
|
|
bites dev loops and unconfigured installs, not production — but verify before upgrading: a
|
|
host that relied on the *old* default now looks in a new location, finds nothing, and creates
|
|
an empty store, which is the same failure this release exists to prevent.
|
|
|
|
Per-user rather than machine-wide is deliberate. `CommonApplicationData` is `/usr/share` on
|
|
Linux and macOS, which a normal user cannot write; as a library default that would fail every
|
|
consuming application's boot on a developer machine. An application is free to compute its own
|
|
default (a machine-wide Windows service reasonably prefers `CommonApplicationData` alongside
|
|
its other stores) — the configured value always wins.
|
|
|
|
- `AddZbSecrets` registration of the validator is idempotent. `AddZbSecrets` is legitimately
|
|
called more than once in real compositions (a replication package calls it after the
|
|
application already has), and `TryAddEnumerable` cannot de-duplicate a factory-built
|
|
descriptor — it de-duplicates on implementation type, which a factory does not carry, so it
|
|
throws on the second call rather than skipping.
|
|
|
|
- `ZB.MOM.WW.Secrets` now depends on `ZB.MOM.WW.Configuration` (0.1.0) for the shared
|
|
`OptionsValidatorBase`/`ValidationBuilder` vocabulary. Every consumer already references it.
|
|
|
|
## 0.5.1 — 2026-08-07
|
|
|
|
### Fixed
|
|
|
|
- **`ZB.MOM.WW.Secrets.Replicator.SqlServer` — virgin-database concurrent migrator race**
|
|
(present since 0.2.0; found by the ScadaBridge central shared-store live gate). The
|
|
migrator's doc claimed simultaneous provisioning of a virgin database converges, but its
|
|
retry filter covered only deadlock victims (1205): Serializable isolation does not
|
|
serialize the `IF NOT EXISTS` metadata probes against another session's in-flight
|
|
`CREATE`, so of two nodes booting together against an empty database the loser died with
|
|
`SqlException` 2714 ("There is already an object named 'zbsecrets'") and aborted its
|
|
host — observed live on an ordinary `docker compose up` of the ScadaBridge central pair.
|
|
The retry filter now also treats the lost-CREATE/lost-INSERT race outcomes as retryable —
|
|
2714 (schema/table already exists), 1913 (index already exists), 2627 (schema_version
|
|
MERGE-insert PK violation) — since losing such a race means the object now exists, which
|
|
is the desired end state; the retry re-runs the idempotent `IF NOT EXISTS` path and
|
|
proceeds. Classification scans the whole `SqlException.Errors` collection (a failed DDL
|
|
batch stacks follow-on errors such as 2759, so the retryable number is not guaranteed
|
|
first), and the retry cap (5) keeps a genuine conflict — e.g. an unrelated pre-existing
|
|
object colliding with the schema name — failing closed. 2705 (duplicate column) is
|
|
deliberately not retried: every column lives inside a single `CREATE TABLE` statement,
|
|
so no cross-session race can produce it. The error-number predicate is now an internal
|
|
static pinned by offline tests, and the live suite gains an N-concurrent-migrators
|
|
virgin-database test.
|
|
|
|
## 0.5.0 — 2026-08-07
|
|
|
|
### Added
|
|
|
|
- **`ZB.MOM.WW.Secrets.Replicator.Grpc` — hub endpoint failover for followers**
|
|
(scadaproj#4). `SecretsGrpcHubClientOptions` gains `FallbackEndpoints`: additional hub
|
|
addresses tried, in order, when a call to the primary `Endpoint` (or an earlier fallback)
|
|
fails with a transport-level gRPC error. The answering endpoint is sticky — a sweep that
|
|
failed over does not re-pay the dead endpoint's full `CallDeadline` on every subsequent
|
|
call — and rotation wraps, so a recovered primary is retried the next time the stuck-to
|
|
endpoint fails. Only `RpcException` triggers failover; cancellation and everything else
|
|
propagate untouched, and when every endpoint fails the last `RpcException` is rethrown so
|
|
the sweep degrades to last-known-good exactly as before. Each failover logs a Warning
|
|
naming the failed and next endpoint URIs (never any credential material).
|
|
- **Same-store requirement:** list only endpoints that serve the SAME store — for
|
|
ScadaBridge, the central pair over one shared SQL Server store. Endpoints backed by
|
|
independent stores turn a primary outage into a silent convergence stop (failing over
|
|
to an emptier hub "succeeds"), which is the exact defect scadaproj#4 recorded.
|
|
- Validation fails startup on a blank/relative fallback or one duplicating the primary
|
|
or another fallback (a duplicated endpoint silently halves the configured failover).
|
|
|
|
### Unchanged
|
|
|
|
- With `FallbackEndpoints` empty (the default), the registration graph is identical to
|
|
0.4.x — one keyed channel, one `GrpcSecretsHubClient`, no failover wrapper — and the wire
|
|
behavior of every call is byte-identical. Every other package in the family is
|
|
behaviorally unchanged in this release.
|
|
|
|
## 0.4.1 — 2026-08-07
|
|
|
|
### Fixed
|
|
|
|
- **`ZB.MOM.WW.Secrets.Replicator.Grpc` — the hub now logs its denials.** A ScadaBridge live
|
|
gate found that `SecretsHubAuthInterceptor` refused calls correctly but wrote nothing about
|
|
it: the only trace of a refusal was a framework `Grpc.AspNetCore.Server` Information line, so
|
|
a follower whose bearer token had been mis-rotated silently stopped converging and the hub
|
|
showed it only at INF. Every denial is now counted, and a `Warning` summarising the window is
|
|
emitted on the first denial after startup or after 60 quiet seconds — carrying the number of
|
|
denials since the previous summary plus a breakdown by cause (missing/malformed credential,
|
|
wrong credential, hub token unconfigured). A single probe is therefore visible immediately,
|
|
while a sustained attack is bounded to one log line per minute — the endpoint is
|
|
unauthenticated by construction, so a per-denial warning would be an unbounded log write
|
|
driven by an unauthenticated caller.
|
|
|
|
### Unchanged
|
|
|
|
- **Wire behavior is byte-identical.** The cause breakdown is server-side diagnostics only; every
|
|
rejection still returns the same uniform `Unauthenticated` status with the same detail, and the
|
|
credential comparison is still constant-time. No token material, presented or configured,
|
|
appears in any log line at any level.
|
|
- The existing warn-once-on-unconfigured-token line is unchanged and still emitted exactly once;
|
|
denials in that state also count toward the new summary's `unconfigured` category.
|
|
- Every other package in the family is behaviorally unchanged in this release.
|
|
|
|
## 0.4.0 — 2026-08-07
|
|
|
|
### Added
|
|
|
|
- **New package `ZB.MOM.WW.Secrets.Replicator.Grpc`** — a central-hosted `secrets_hub.v1`
|
|
gRPC hub for the sites-talk-to-central production topology (scadaproj#3; design doc
|
|
[`docs/plans/2026-08-07-secrets-production-topology-and-grpc-hub.md`](../docs/plans/2026-08-07-secrets-production-topology-and-grpc-hub.md)).
|
|
- **Pull-only by construction**: the proto exposes no write RPC. Updates originate at
|
|
central only; a site never pushes a secret upstream.
|
|
- **Fail-closed bearer auth** on the hub server: constant-time comparison, uniform denial
|
|
response (no signal distinguishing "wrong token" from "missing token"), a warn-once log
|
|
on an unconfigured/empty token rather than a silent open door.
|
|
- `GetSecrets` enforces a batch-size cap, rejecting an oversized request with
|
|
`ResourceExhausted` instead of serving it partially or unbounded.
|
|
- Site-side `GrpcSecretSyncService` sweeps the hub on an interval with **per-row parse
|
|
isolation** — one malformed/undecryptable row logs and is skipped, it does not fault the
|
|
whole sweep.
|
|
- Keyed `GrpcChannel` registration so multiple hub targets can be wired in one process
|
|
without channel collisions.
|
|
- **Offline behavior is last-known-good**: if the hub is unreachable, the site keeps
|
|
serving the most recently synced values rather than failing secret resolution.
|
|
|
|
### Unchanged
|
|
|
|
- `ZB.MOM.WW.Secrets.Abstractions`, `ZB.MOM.WW.Secrets`, `ZB.MOM.WW.Secrets.Ui`,
|
|
`ZB.MOM.WW.Secrets.Replicator.SqlServer`, and `ZB.MOM.WW.Secrets.Replicator.AkkaDotNet` are
|
|
behaviorally unchanged in this release — no breaking changes to any existing package.
|
|
|
|
## 0.3.0 — 2026-07-19
|
|
|
|
- Menu-driven Spectre.Console interactive operator console added to the `secret` CLI (list /
|
|
set / reveal / delete, reference audit & seed, KEK doctor, ciphertext-only bundle
|
|
export/import). See
|
|
[`docs/operations/interactive-console.md`](docs/operations/interactive-console.md).
|
|
|
|
## 0.2.3 — 2026-07-19
|
|
|
|
- `ZB.MOM.WW.Secrets.Ui`: fixed the delete-confirmation modal being permanently invisible on
|
|
every Bootstrap host (`ConfirmDeleteModal` used bare `class="modal"` with no shipped CSS;
|
|
Bootstrap's `.modal{display:none}` hid it everywhere but mxaccessgw). Fixed via a
|
|
`zb-secrets-modal*` class rename plus a component-emitted `<style>` block.
|
|
|
|
## 0.2.2 — 2026-07-18
|
|
|
|
- Fixed the Akka replicator's hosted-process DI deadlock (a circular singleton dependency
|
|
hidden behind factory lambdas that MS.DI's `StackGuard` turned into a silent cross-thread
|
|
hang rather than a stack overflow). New `DeferredSecretCacheInvalidator` defers the one
|
|
cycle-closing edge to first eviction.
|
|
|
|
## 0.2.1 — 2026-07-18
|
|
|
|
- Fixed the Akka replicator being silently inert since 0.2.0 (a `TryAdd` registration-order
|
|
bug published into a no-op sink and never spawned an actor). SQL-Server replication was
|
|
never affected.
|
|
|
|
## 0.2.0 — 2026-07-18
|
|
|
|
- Added clustered replication (G-7): `ZB.MOM.WW.Secrets.Replicator.SqlServer` (shared store,
|
|
or local-store-with-hub) and `ZB.MOM.WW.Secrets.Replicator.AkkaDotNet` (peer-to-peer over
|
|
distributed pub/sub).
|