Joseph Doherty
c5e894d9b3
Suggested by the OtOpcUa session, the only real containerized deployment in this family and the one that hit the false positive. A store on a volume mounted inside the content root is durable across container replacement but is rejected anyway, because the rule cannot tell a mount from an ordinary directory. The message now says so, gives the fix, and forecloses the wrong one by stating that no override exists. A false positive's entire cost is the confusion in the moment, so an error that names its own fix costs nothing on the true-positive path and turns the false positive into a two-minute edit. Message only, no behaviour change. Their case for keeping the rule strict is recorded in the CHANGELOG because it is better than the trade-off I had written: the workaround is an improvement rather than a concession, an override flag has the lifecycle of a knob set once and never revisited, and mount detection would make the verdict depend on runtime platform state so the same path would pass on one host and fail on another.
289 lines
18 KiB
Markdown
289 lines
18 KiB
Markdown
# Changelog
|
|
|
|
All notable changes to the `ZB.MOM.WW.Secrets` family are recorded here. Format loosely
|
|
follows [Keep a Changelog](https://keepachangelog.com/); versions are shared across every
|
|
package in the family (see `Directory.Build.props`) even when a given release only touches
|
|
one of them.
|
|
|
|
## 0.6.2 — 2026-08-11
|
|
|
|
### Changed
|
|
|
|
- **The under-content-root rejection message now teaches the container case.** Suggested by the
|
|
OtOpcUa session, which is the only real containerized deployment in this family and the one that
|
|
hit the false positive. A store on a volume mounted inside the content root is durable across
|
|
container replacement but is rejected anyway, because the rule cannot tell a mount from an
|
|
ordinary directory. The message now says so, gives the fix (mount the same volume at a second
|
|
path outside the content root; the file sits at the volume root either way, so an existing store
|
|
carries over), and forecloses the wrong fix by stating that no override exists.
|
|
|
|
A false positive's entire cost is the confusion in the moment, so a boot error that names its own
|
|
fix costs nothing on the true-positive path and converts the false positive into a two-minute
|
|
edit. **Message only — no behaviour change.** Existing 0.6.1 consumers can take this whenever they
|
|
next touch the pin; it does not warrant a re-pin round on its own.
|
|
|
|
Their argument for keeping the rule strict is recorded because it is better than the trade-off I
|
|
had written down: the workaround is an improvement rather than a concession (it separates the
|
|
application from its durable state at the filesystem level — the exact distinction the incident
|
|
blurred); an `AllowStoreUnderContentRoot` flag has the lifecycle of a knob set once to clear a
|
|
symptom and never revisited, surviving the redeploy that changes the layout it was reasoned
|
|
against; and mount detection would make the verdict depend on runtime platform state, so the same
|
|
path would pass on one host and fail on another — strictly worse than a stable false positive
|
|
with a cheap documented fix.
|
|
|
|
## 0.6.1 — 2026-08-11
|
|
|
|
### Fixed
|
|
|
|
- **The 0.6.0 guard had a hole at exactly the moment the store file is created** (found by the
|
|
HistorianGateway adoption of 0.6.0, which probed the rejected paths rather than observing a
|
|
clean boot). An application that expands `${secret:}` references *before* building its host
|
|
composes this subsystem in a throwaway `ServiceCollection` with **no `IHostEnvironment` in it**,
|
|
then migrates the store from that container. `SecretsOptionsValidator` resolves the content root
|
|
from `IHostEnvironment` and skips the under-content-root rule when it is absent — so pre-host an
|
|
absolute in-content-root path passed the only rule still applying, the migrator **created the
|
|
database**, and the boot then failed a second later once the real host registered
|
|
`IHostEnvironment` and the rule finally fired.
|
|
|
|
The observable result was a rejected boot that *still left an empty store, with `-wal`/`-shm`
|
|
siblings, sitting at the rejected path* — the same artifact that made the outage this guard
|
|
exists to prevent read as "the database is there, it's just empty". A relative path was
|
|
unaffected, because the absolute-path rule fires with or without a content root; that asymmetry
|
|
is what exposed it.
|
|
|
|
Conflating "no `IHostEnvironment` registered" with "no content root exists" was the mistake. New
|
|
overload **`AddZbSecrets(services, config, sectionPath, contentRootPath)`** takes the content
|
|
root explicitly; the 3-argument overload is unchanged and still falls back to `IHostEnvironment`.
|
|
**Pre-host composition should pass it** — e.g. `builder.Environment.ContentRootPath`. Passing
|
|
`null` remains correct only where there genuinely is no content root, such as the CLI.
|
|
`SecretsOptionsValidator` gains a `string?` constructor alongside the `IHostEnvironment?` one.
|
|
|
|
### Known interaction: containers, where the rule deliberately over-rejects
|
|
|
|
- **A containerized deployment can put the store on a mounted volume *inside* the content root, and
|
|
0.6.1 will reject it.** Observed on the OtOpcUa docker-dev rig: `Secrets__SqlitePath` was
|
|
`/app/data/…` with `/app` the container content root and `/app/data` a mounted volume — absolute,
|
|
durable across container replacement, and entirely deliberate. The rule rejects it anyway.
|
|
|
|
**This is a true positive for the rule as written and a false positive for the operator's
|
|
intent**, and the difference is real: in a container the image is replaced on upgrade but the
|
|
volume is not, so "inside the content root" is a weaker proxy for "destroyed on upgrade" there
|
|
than it is for a Windows service whose whole directory gets renamed.
|
|
|
|
It is not being relaxed, and no opt-out flag is being added. Distinguishing a mount from an
|
|
ordinary directory means reading platform-specific mount tables, and an
|
|
`AllowStoreUnderContentRoot` escape hatch is the kind of switch that gets set once to clear a
|
|
boot failure and then outlives the reason — which is precisely the failure mode the guard exists
|
|
to close. **The workaround is one line**: mount the same volume at a second path outside the
|
|
content root (the rig used `/data`) and point `Secrets:SqlitePath` there. The file is at the
|
|
volume root either way, so seeded stores carry over with no migration.
|
|
|
|
### Upgrade note (applies to 0.6.0 as well)
|
|
|
|
- **A store seeded under the pre-0.6.0 default is orphaned, not migrated, and nothing says so.**
|
|
The old default was the relative `secrets.db`, which resolved against the process working
|
|
directory; the new default is absolute. An application that relied on the old default silently
|
|
starts using a different file, and anything previously seeded into the old one is stranded. This
|
|
is a dev-loop concern for the apps in this family (every deployment sets an explicit absolute
|
|
path), but check for a stray `*-secrets.db` beside your project before assuming an empty store
|
|
after upgrade means the store was lost. Deleting the key from configuration, rather than leaving
|
|
a relative value, is what makes the new default apply.
|
|
|
|
## 0.6.0 — 2026-08-11
|
|
|
|
### Added
|
|
|
|
- **`ZB.MOM.WW.Secrets` — fail-closed validation of the store path.** `Secrets:SqlitePath` is
|
|
now validated at startup (`ValidateOnStart`) by `SecretsOptionsValidator`, using two rules in
|
|
the new public `SecretsStorePathRules`. Both come from production incidents and neither
|
|
implies the other:
|
|
- **Must be absolute.** A relative value resolves against the process working directory —
|
|
for a Windows service that is `C:\Windows\System32`, so the store lands outside the
|
|
installation entirely; in a dev loop it lands in the source tree (observed: a gateway wrote
|
|
its secrets database into `src/` and tripped that repo's own tree-hygiene test).
|
|
- **Must not be inside the application content root.** An *absolute* path can still point
|
|
into the directory every deployment procedure in this family replaces or renames on
|
|
upgrade. On 2026-08-09 a gateway's credential store sat inside its `Server\` directory; the
|
|
deploy renamed `Server` to `Server.bak.*`, the database left with it, and the process
|
|
created a fresh empty one at the same path. Every consumer lost its credential, silently,
|
|
for two days. The existing "absolute path" convention did not help — the path *was*
|
|
absolute.
|
|
|
|
The rules **reject rather than relocate**: silently moving a credential store presents later
|
|
as an authentication failure pointing nowhere near the configuration that caused it, whereas
|
|
a boot error is seen immediately by whoever caused it. Validation is deliberately **not**
|
|
exempted in Development — a guard that never runs in a dev loop or in CI rots, and the
|
|
configuration behind the outage looked entirely reasonable.
|
|
|
|
The content-root rule is skipped when no `IHostEnvironment` is available (the CLI composes
|
|
the subsystem without a host); the absolute-path rule still applies.
|
|
|
|
### Changed
|
|
|
|
- **BREAKING (configuration, not API): `SecretsOptions.SqlitePath` now defaults to an absolute
|
|
per-user path** — `ZB.MOM.WW/secrets.db` under `LocalApplicationData` (`~/.local/share` on
|
|
Linux and macOS, `%LOCALAPPDATA%` on Windows), exposed as `SecretsOptions.DefaultSqlitePath`.
|
|
It was `secrets.db`, which the new rules reject.
|
|
|
|
**A consuming application that configures a relative `Secrets:SqlitePath` will now fail to
|
|
boot.** The fix is to remove the key (accepting the default) or set an absolute path outside
|
|
the content root. Every deployment of this family already sets an absolute path, so this
|
|
bites dev loops and unconfigured installs, not production — but verify before upgrading: a
|
|
host that relied on the *old* default now looks in a new location, finds nothing, and creates
|
|
an empty store, which is the same failure this release exists to prevent.
|
|
|
|
Per-user rather than machine-wide is deliberate. `CommonApplicationData` is `/usr/share` on
|
|
Linux and macOS, which a normal user cannot write; as a library default that would fail every
|
|
consuming application's boot on a developer machine. An application is free to compute its own
|
|
default (a machine-wide Windows service reasonably prefers `CommonApplicationData` alongside
|
|
its other stores) — the configured value always wins.
|
|
|
|
- `AddZbSecrets` registration of the validator is idempotent. `AddZbSecrets` is legitimately
|
|
called more than once in real compositions (a replication package calls it after the
|
|
application already has), and `TryAddEnumerable` cannot de-duplicate a factory-built
|
|
descriptor — it de-duplicates on implementation type, which a factory does not carry, so it
|
|
throws on the second call rather than skipping.
|
|
|
|
- `ZB.MOM.WW.Secrets` now depends on `ZB.MOM.WW.Configuration` (0.1.0) for the shared
|
|
`OptionsValidatorBase`/`ValidationBuilder` vocabulary. Every consumer already references it.
|
|
|
|
## 0.5.1 — 2026-08-07
|
|
|
|
### Fixed
|
|
|
|
- **`ZB.MOM.WW.Secrets.Replicator.SqlServer` — virgin-database concurrent migrator race**
|
|
(present since 0.2.0; found by the ScadaBridge central shared-store live gate). The
|
|
migrator's doc claimed simultaneous provisioning of a virgin database converges, but its
|
|
retry filter covered only deadlock victims (1205): Serializable isolation does not
|
|
serialize the `IF NOT EXISTS` metadata probes against another session's in-flight
|
|
`CREATE`, so of two nodes booting together against an empty database the loser died with
|
|
`SqlException` 2714 ("There is already an object named 'zbsecrets'") and aborted its
|
|
host — observed live on an ordinary `docker compose up` of the ScadaBridge central pair.
|
|
The retry filter now also treats the lost-CREATE/lost-INSERT race outcomes as retryable —
|
|
2714 (schema/table already exists), 1913 (index already exists), 2627 (schema_version
|
|
MERGE-insert PK violation) — since losing such a race means the object now exists, which
|
|
is the desired end state; the retry re-runs the idempotent `IF NOT EXISTS` path and
|
|
proceeds. Classification scans the whole `SqlException.Errors` collection (a failed DDL
|
|
batch stacks follow-on errors such as 2759, so the retryable number is not guaranteed
|
|
first), and the retry cap (5) keeps a genuine conflict — e.g. an unrelated pre-existing
|
|
object colliding with the schema name — failing closed. 2705 (duplicate column) is
|
|
deliberately not retried: every column lives inside a single `CREATE TABLE` statement,
|
|
so no cross-session race can produce it. The error-number predicate is now an internal
|
|
static pinned by offline tests, and the live suite gains an N-concurrent-migrators
|
|
virgin-database test.
|
|
|
|
## 0.5.0 — 2026-08-07
|
|
|
|
### Added
|
|
|
|
- **`ZB.MOM.WW.Secrets.Replicator.Grpc` — hub endpoint failover for followers**
|
|
(scadaproj#4). `SecretsGrpcHubClientOptions` gains `FallbackEndpoints`: additional hub
|
|
addresses tried, in order, when a call to the primary `Endpoint` (or an earlier fallback)
|
|
fails with a transport-level gRPC error. The answering endpoint is sticky — a sweep that
|
|
failed over does not re-pay the dead endpoint's full `CallDeadline` on every subsequent
|
|
call — and rotation wraps, so a recovered primary is retried the next time the stuck-to
|
|
endpoint fails. Only `RpcException` triggers failover; cancellation and everything else
|
|
propagate untouched, and when every endpoint fails the last `RpcException` is rethrown so
|
|
the sweep degrades to last-known-good exactly as before. Each failover logs a Warning
|
|
naming the failed and next endpoint URIs (never any credential material).
|
|
- **Same-store requirement:** list only endpoints that serve the SAME store — for
|
|
ScadaBridge, the central pair over one shared SQL Server store. Endpoints backed by
|
|
independent stores turn a primary outage into a silent convergence stop (failing over
|
|
to an emptier hub "succeeds"), which is the exact defect scadaproj#4 recorded.
|
|
- Validation fails startup on a blank/relative fallback or one duplicating the primary
|
|
or another fallback (a duplicated endpoint silently halves the configured failover).
|
|
|
|
### Unchanged
|
|
|
|
- With `FallbackEndpoints` empty (the default), the registration graph is identical to
|
|
0.4.x — one keyed channel, one `GrpcSecretsHubClient`, no failover wrapper — and the wire
|
|
behavior of every call is byte-identical. Every other package in the family is
|
|
behaviorally unchanged in this release.
|
|
|
|
## 0.4.1 — 2026-08-07
|
|
|
|
### Fixed
|
|
|
|
- **`ZB.MOM.WW.Secrets.Replicator.Grpc` — the hub now logs its denials.** A ScadaBridge live
|
|
gate found that `SecretsHubAuthInterceptor` refused calls correctly but wrote nothing about
|
|
it: the only trace of a refusal was a framework `Grpc.AspNetCore.Server` Information line, so
|
|
a follower whose bearer token had been mis-rotated silently stopped converging and the hub
|
|
showed it only at INF. Every denial is now counted, and a `Warning` summarising the window is
|
|
emitted on the first denial after startup or after 60 quiet seconds — carrying the number of
|
|
denials since the previous summary plus a breakdown by cause (missing/malformed credential,
|
|
wrong credential, hub token unconfigured). A single probe is therefore visible immediately,
|
|
while a sustained attack is bounded to one log line per minute — the endpoint is
|
|
unauthenticated by construction, so a per-denial warning would be an unbounded log write
|
|
driven by an unauthenticated caller.
|
|
|
|
### Unchanged
|
|
|
|
- **Wire behavior is byte-identical.** The cause breakdown is server-side diagnostics only; every
|
|
rejection still returns the same uniform `Unauthenticated` status with the same detail, and the
|
|
credential comparison is still constant-time. No token material, presented or configured,
|
|
appears in any log line at any level.
|
|
- The existing warn-once-on-unconfigured-token line is unchanged and still emitted exactly once;
|
|
denials in that state also count toward the new summary's `unconfigured` category.
|
|
- Every other package in the family is behaviorally unchanged in this release.
|
|
|
|
## 0.4.0 — 2026-08-07
|
|
|
|
### Added
|
|
|
|
- **New package `ZB.MOM.WW.Secrets.Replicator.Grpc`** — a central-hosted `secrets_hub.v1`
|
|
gRPC hub for the sites-talk-to-central production topology (scadaproj#3; design doc
|
|
[`docs/plans/2026-08-07-secrets-production-topology-and-grpc-hub.md`](../docs/plans/2026-08-07-secrets-production-topology-and-grpc-hub.md)).
|
|
- **Pull-only by construction**: the proto exposes no write RPC. Updates originate at
|
|
central only; a site never pushes a secret upstream.
|
|
- **Fail-closed bearer auth** on the hub server: constant-time comparison, uniform denial
|
|
response (no signal distinguishing "wrong token" from "missing token"), a warn-once log
|
|
on an unconfigured/empty token rather than a silent open door.
|
|
- `GetSecrets` enforces a batch-size cap, rejecting an oversized request with
|
|
`ResourceExhausted` instead of serving it partially or unbounded.
|
|
- Site-side `GrpcSecretSyncService` sweeps the hub on an interval with **per-row parse
|
|
isolation** — one malformed/undecryptable row logs and is skipped, it does not fault the
|
|
whole sweep.
|
|
- Keyed `GrpcChannel` registration so multiple hub targets can be wired in one process
|
|
without channel collisions.
|
|
- **Offline behavior is last-known-good**: if the hub is unreachable, the site keeps
|
|
serving the most recently synced values rather than failing secret resolution.
|
|
|
|
### Unchanged
|
|
|
|
- `ZB.MOM.WW.Secrets.Abstractions`, `ZB.MOM.WW.Secrets`, `ZB.MOM.WW.Secrets.Ui`,
|
|
`ZB.MOM.WW.Secrets.Replicator.SqlServer`, and `ZB.MOM.WW.Secrets.Replicator.AkkaDotNet` are
|
|
behaviorally unchanged in this release — no breaking changes to any existing package.
|
|
|
|
## 0.3.0 — 2026-07-19
|
|
|
|
- Menu-driven Spectre.Console interactive operator console added to the `secret` CLI (list /
|
|
set / reveal / delete, reference audit & seed, KEK doctor, ciphertext-only bundle
|
|
export/import). See
|
|
[`docs/operations/interactive-console.md`](docs/operations/interactive-console.md).
|
|
|
|
## 0.2.3 — 2026-07-19
|
|
|
|
- `ZB.MOM.WW.Secrets.Ui`: fixed the delete-confirmation modal being permanently invisible on
|
|
every Bootstrap host (`ConfirmDeleteModal` used bare `class="modal"` with no shipped CSS;
|
|
Bootstrap's `.modal{display:none}` hid it everywhere but mxaccessgw). Fixed via a
|
|
`zb-secrets-modal*` class rename plus a component-emitted `<style>` block.
|
|
|
|
## 0.2.2 — 2026-07-18
|
|
|
|
- Fixed the Akka replicator's hosted-process DI deadlock (a circular singleton dependency
|
|
hidden behind factory lambdas that MS.DI's `StackGuard` turned into a silent cross-thread
|
|
hang rather than a stack overflow). New `DeferredSecretCacheInvalidator` defers the one
|
|
cycle-closing edge to first eviction.
|
|
|
|
## 0.2.1 — 2026-07-18
|
|
|
|
- Fixed the Akka replicator being silently inert since 0.2.0 (a `TryAdd` registration-order
|
|
bug published into a no-op sink and never spawned an actor). SQL-Server replication was
|
|
never affected.
|
|
|
|
## 0.2.0 — 2026-07-18
|
|
|
|
- Added clustered replication (G-7): `ZB.MOM.WW.Secrets.Replicator.SqlServer` (shared store,
|
|
or local-store-with-hub) and `ZB.MOM.WW.Secrets.Replicator.AkkaDotNet` (peer-to-peer over
|
|
distributed pub/sub).
|