scadaproj/ZB.MOM.WW.Secrets/CHANGELOG.md

# Changelog

All notable changes to the `ZB.MOM.WW.Secrets` family are recorded here. Format loosely
follows [Keep a Changelog](https://keepachangelog.com/); versions are shared across every
package in the family (see `Directory.Build.props`) even when a given release only touches
one of them.

## 0.6.6 — unreleased (NOT published to the feed)

Fixes the scadaproj#15 code-review findings (`code-reviews/Secrets/findings.md`): six P2s (scadaproj#30–#35),
the P3 batch (scadaproj#45: Secrets-004–009, 011, 012, 014–016, 018, 019) and the scadaproj#49 follow-ups.

### Deprecated

- **`ZB.MOM.WW.Secrets.Replicator.Grpc` is deprecated** (scadaproj#8, owner decision 2026-11-09). The
  pull-only `secrets_hub.v1` hub has had no consumer since ScadaBridge retired it (2026-09-28, site
  secrets now ride its artifact deploy); OtOpcUa never used it (Akka pair-local). The project and its
  test project are removed from `ZB.MOM.WW.Secrets.slnx`, so the family build, test run and pack
  (`build/push.sh`) no longer include it, and the csproj is `IsPackable=false`. `SecretsGrpcHubServerExtensions`
  (`AddZbSecretsGrpcHub`, `MapZbSecretsHub`) and `SecretsGrpcHubClientExtensions` (`AddZbSecretsGrpcHubClient`)
  are `[Obsolete]`. **No `0.6.6` of this package will be published**; `0.4.1`–`0.6.3` stay on the feed
  until the owner unlists them. Source and tests stay in the tree and still build standalone
  (`dotnet test tests/ZB.MOM.WW.Secrets.Replicator.Grpc.Tests`).

### scadaproj#52 follow-ups

- **The CLI reference audit reports malformed `${secret:}` tokens.** New `ReferenceStatus.Malformed`:
  an empty/blank name, an unterminated `${secret:name` or a nested marker — exactly the shapes
  `SecretReferenceExpander` now refuses at startup — yields one finding (`${secret:...} (malformed)`)
  listing every config path, and the audit flow tells the operator to fix it. Not seedable.
- **`SecretName` is capped at 450 characters** (`SecretName.MaxLength`, measured after trim/lowercase),
  matching the hub's `NVARCHAR(450)` name column. Enforced in the constructor, so every store/API
  entry point rejects an over-long name up front instead of the hub refusing the row later.
  Behaviour change: an already-stored name longer than 450 can no longer be constructed.
- **Secrets-005 follow-up: an ordinary local SMB share.** On Windows, `\\localhost\share\...` /
  `\\<this machine>\share\...` for a non-administrative share is mapped to the directory the share
  publishes (`NetShareGetInfo` level 2), so a store reached through a share that points back at the
  content root is refused. Limits (documented in `SecretsStorePathRules`): the API needs
  Administrators / Server Operators rights (denied => compared as written); a share on another machine
  is never resolved; on non-Windows an NFS/SMB mount cannot be resolved to its export, so that case is
  not caught. Test is Windows-only and skips when it cannot create a share.

### Fixed

- **SQL-Server and Akka registration no longer depends on call order** (scadaproj#30, Secrets-001).
  `AddZbSecretsSqlServerStore`, `AddZbSecretsSqlServerReplication` and `AddZbSecretsAkkaReplication`
  used to `TryAdd` their `ISecretStore` / `ISecretsStoreMigrator` / `ISecretReplicator` ahead of their
  internal `AddZbSecrets` call, so an application that had already called `AddZbSecrets` silently kept
  the SQLite store (shared-store mode) or the `NoOpSecretReplicator` (hub and Akka modes). They now
  replace those registrations after the core call; a later `AddZbSecrets` only `TryAdd`s, so either
  order yields the same graph, and exactly one `ISecretStore` is registered.
- **The SQL-Server and Akka extensions take a `contentRootPath`** (scadaproj#31, Secrets-002), and the
  store-path validator's content root is now **the last explicitly supplied one** across all
  `AddZbSecrets` calls (a `null` root never erases an earlier explicit one). Before, the first call to
  register the validator captured the root, so a package's internal 3-arg call that ran first silently
  dropped the application's 4-arg root and the pre-host under-content-root guard was skipped — in hub
  mode no registration order applied it. New trailing optional parameter on
  `AddZbSecretsSqlServerStore`, `AddZbSecretsSqlServerReplication` and `AddZbSecretsAkkaReplication`
  (source-compatible; callers recompile). Pass it from every pre-host composition.
- **`${secret:}` expansion survives configuration reload** (scadaproj#32, Secrets-003).
  `ExpandConfigurationAsync` used to write through `config[key] = …`, which a file provider's reload
  replaces, so editing `appsettings.json` on a running service put the `${secret:…}` placeholder back
  into every reader. Given a `ConfigurationManager` (`builder.Configuration`), it now pins the expanded
  values in an in-memory overlay source appended last that never reloads; given any other
  `IConfigurationRoot` it writes in place as before and re-applies the values from the root's reload
  token. Behaviour changes: a referencing key keeps its startup value until restart (edits to it are
  ignored on reload); every reference is resolved before anything is written, so a fail-closed miss
  leaves the configuration untouched instead of half expanded.
- **The Akka replication actor starts with the host** (scadaproj#33, Secrets-010). It used to be created
  only when something resolved `ISecretStore`, so a node that expanded its references pre-host and never
  touched the store at runtime had no actor and silently missed every rotation. `AddZbSecretsAkkaReplication`
  now registers a hosted starter that creates it on `IHostApplicationLifetime.ApplicationStarted` (after
  every hosted service, including an Akka.Hosting `AkkaHostedService` registered later), or in its own
  `StartAsync` when no lifetime is registered. Behaviour change: the application's `ActorSystem` must be
  resolvable once the host has started even if no secret is ever touched; a failure there is logged.
  OtOpcUa's own `SecretReplicationStarter` becomes redundant (harmless — same actor).
- **UI: a committed add/rotate/delete is no longer reported as failed when its audit write throws**
  (scadaproj#34, Secrets-013). `SecretEditor` and `ConfirmDeleteModal` wrote the success audit inside the
  store write's `try`, so a throwing `IAuditWriter` produced an error message, a `Failure` audit row and no
  cache eviction for a change that had committed. The success audit is now best-effort after the commit:
  the change is reported saved, `OnSaved`/`OnDeleted` fire, the resolver cache is evicted, and the audit
  failure is logged as a warning (`ILogger<SecretEditor>` / `ILogger<ConfirmDeleteModal>`, when the host
  registers logging).

- **A malformed row fails as `SecretDecryptionException`** (scadaproj#45, Secrets-004). A wrong-length
  nonce, tag or wrapped DEK made `AesGcmEnvelopeCipher.Decrypt` throw a raw `ArgumentException`, which
  skipped the resolver's `decryption-failed` audit.
- **One master-key reading per cipher operation, zeroed afterwards** (scadaproj#45, Secrets-007). The KEK
  id and the key used to be two separate reads of the source, so a key file replaced mid-`set` could stamp
  a row with the old key's id and wrap it under the new key — permanently undecryptable. New
  `MasterKeySnapshot` + default-implemented `IMasterKeyProvider.GetSnapshot()` (additive; existing
  implementations keep working, their key is copied, never zeroed in place); `MasterKeyProviderBase`
  resolves once and derives the id from that key. **Stored data is unchanged** — a golden row written by
  the 0.6.5 cipher is pinned by a test. `MasterKeyProviderBase.ResolveKeyBytes` must return a fresh
  buffer (it is zeroed after use); every shipped provider already did.
- **Store-path rules compare locations, not spellings** (scadaproj#45, Secrets-005). Rooted now means
  fully qualified (Windows `C:secrets.db` / `\secrets.db` are rejected), and the under-content-root rule
  also resolves symlinks/junctions, `\\?\` / `\\.\` prefixes, loopback admin shares and 8.3 names.
  **Behaviour change:** a store that reaches the content root through a link — including macOS's
  `/var` → `/private/var` — now fails startup validation where it used to pass.
- **Resolver cache** (scadaproj#45, Secrets-006): a resolve in flight across an `Invalidate` no longer
  re-caches the old plaintext for a full TTL, expired entries are evicted (on lookup and by a periodic
  sweep), and a non-positive TTL caches nothing. `DefaultSecretResolver` is now `IDisposable`.
- **Hub push sweep is isolated per row** (scadaproj#45, Secrets-008): one row the hub refuses (a
  >450-char name) no longer stops every later name from being pushed; it is logged (EventId 4) and retried.
- **Malformed `${secret:}` references fail closed** (scadaproj#45, Secrets-009): `${secret:}`, a blank
  name, an unterminated `${secret:name` or a nested marker throws `FormatException` naming the key
  (never the value) instead of passing the literal through as the credential. **Behaviour change** for a
  host whose configuration holds such a typo: it now fails at startup. The README sample no longer uses a
  relative `SqlitePath`.
- **Akka announce carries a sender; anti-entropy is chunked** (scadaproj#45, Secrets-011/012). The
  periodic announce is published from the actor's mailbox (it lost its sender under a genuinely async
  store, so peers replied to deadLetters); a failed manifest read is logged. Pushed rows, pull replies and
  pull requests are split into ≤64 KB messages, so a cold node catches up past Akka.Remote's 128000-byte
  frame. The announce itself is not split (see `docs/operations/clustered-secrets.md`).
- **UI** (scadaproj#45, Secrets-014/015/016): "Add secret" refuses a name that already exists (use
  Rotate) and restores a deleted one only on a second submit; Reveal **fails closed** — no value is shown
  unless its `secret.reveal` audit was written, and the failure is logged; `RevealButton`, `SecretEditor`
  and `ConfirmDeleteModal` re-check their own policy (`RevealPolicy` / `ManagePolicy`) and refuse without
  it, instead of relying on the host's `AuthorizeView`.
- **Bundle import validates the whole bundle first** (scadaproj#45, Secrets-018): a malformed or tampered
  entry anywhere aborts with nothing written; a store fault mid-apply names the rows already written.
- **Hub-call cancellation is cancellation** (scadaproj#45, Secrets-019): the gRPC hub channels set
  `ThrowOperationCanceledOnCancellation`, so a shutdown mid-sweep is no longer recorded in
  `ISecretsHubSyncStatus` as a failure, and the failover reader no longer dials every fallback with a
  cancelled token.
- **The CLI's shipped `appsettings.json` no longer sets a relative `SqlitePath`** (scadaproj#49), which
  failed validation; the per-user default applies unless the working directory's `appsettings.json` or
  `Secrets__SqlitePath` says otherwise.

### Documentation

- scadaproj#49: the plain-`IConfigurationRoot` reload gap is documented, not fixed (README "Reload"); no
  family host is exposed. `clustered-secrets.md` notes that OtOpcUa's own `SecretReplicationStarter` is
  redundant once it adopts 0.6.6, and documents the Akka message-size bounds.

### Changed

- **CLI `set`/`rotate` take the value on stdin or a masked prompt** (scadaproj#35, Secrets-017).
  `secret set <name> -` / `--value-stdin` reads stdin (one trailing newline dropped); `secret set <name>`
  reads piped stdin, or prompts twice (masked) at a terminal. The positional `<value>` still works but
  prints a deprecation warning to stderr. The CLI host is no longer built from argv, so a value starting
  with `-` and containing `=` no longer crashes it with the value in an unhandled `FormatException`,
  and argv can no longer override configuration (`--Secrets:SqlitePath=…`, `--environment …`: use
  environment variables). A startup failure now prints `secret: startup failed: <message>` and exits 1
  instead of an unhandled stack trace. An unexpected trailing argument is reported by position, not
  echoed.

## 0.6.5 — unreleased (NOT published to the feed)

### Changed

- **`Microsoft.Data.Sqlite` 10.0.7 -> 10.0.13** (scadaproj#26): 10.0.7's connection pool can hand one
  handle to two users (dotnet/efcore#39009). 10.0.13 floors `SQLitePCLRaw.bundle_e_sqlite3` at 2.1.13, so the
  central transitive pin on `SQLitePCLRaw.lib.e_sqlite3` moves 2.1.12 -> 2.1.13 (the old pin is now a hard
  NU1109 downgrade error). NuGetAudit clean.
- Tests (scadaproj#27): the two Unix-permission tests are `[SkippableFact]` (they threw `SkipException` from a plain
  `[Fact]` and failed on Windows); `Resolver_Works_EndToEnd_ThroughDi` clears the SQLite pools before deleting its temp DB.

## 0.6.4 — unreleased (NOT published to the feed)

### Fixed

- **SQLite store initialisation is safe when several setups run at once on one store**
  (scadaproj#25). mxaccessgw's parallel test hosts, each migrating the same per-process store,
  failed intermittently on the Windows nightly with `SqliteException: cannot start a transaction
  within a transaction` from `SqliteSecretsStoreMigrator.MigrateAsync`. The migrator now:
  - serialises migrations of one store within a process behind a process-wide gate keyed on the
    normalised store path (`Path.GetFullPath`; case-insensitive on Windows and macOS), however
    many factories, migrators or DI containers point at it;
  - runs each attempt on a dedicated **unpooled** connection inside `BEGIN IMMEDIATE`, so it can
    neither be handed a pooled handle another owner is still using nor leave one behind in a
    transaction. The pinned Microsoft.Data.Sqlite `10.0.7` can hand one pooled handle to two
    owners (dotnet/efcore#39009, fixed in `10.0.13`), and no `10.0.x` refuses to re-pool a handle
    whose rollback failed (dotnet/efcore#39082); both surface as exactly this error on the next
    `BEGIN`;
  - retries `SQLITE_BUSY` / `SQLITE_LOCKED` (another process holding the write lock past the 5 s
    busy timeout) up to 6 attempts on a fresh connection with equal-jitter exponential backoff.

No schema, wire or configuration change; store operations other than the migration are unchanged.
The Microsoft.Data.Sqlite pin stays at `10.0.7` so this patch does not raise the consumer floor.

## 0.6.3 — 2026-09-28

### Added

- **gRPC hub follower sweeps are observable.** `GrpcSecretSyncService` (package
  `ZB.MOM.WW.Secrets.Replicator.Grpc`) now logs every sweep outcome instead of only the sweeps
  that pull a change:
  - `EventId 4` (Debug) — a sweep found nothing new; logged on every in-sync sweep.
  - `EventId 5` (Information) — an in-sync heartbeat, at most once per
    `SecretsGrpcHubClientOptions.InSyncLogInterval` (new, default 1 hour, validated positive), so
    "the loop is alive and still agrees with the hub" is visible at the default log level without
    a line every 30 seconds.
  - `EventId 6` (Information) — the first successful sweep after one or more failures, naming
    the failure count.
- **`ISecretsHubSyncStatus`** — a public, DI-registered read model of the follower loop:
  `LastSweepUtc`, `LastSuccessUtc`, `ConsecutiveFailures`, `LastError`. Getters are synchronized
  with the writer, so a health check or status page reads a consistent snapshot.

No behaviour change to syncing itself, and no wire or schema change. Only hub followers gain
anything from this release; other consumers can stay on 0.6.2.

## 0.6.2 — 2026-08-11

### Changed

- **The under-content-root rejection message now teaches the container case.** Suggested by the
  OtOpcUa session, which is the only real containerized deployment in this family and the one that
  hit the false positive. A store on a volume mounted inside the content root is durable across
  container replacement but is rejected anyway, because the rule cannot tell a mount from an
  ordinary directory. The message now says so, gives the fix (mount the same volume at a second
  path outside the content root; the file sits at the volume root either way, so an existing store
  carries over), and forecloses the wrong fix by stating that no override exists.

  A false positive's entire cost is the confusion in the moment, so a boot error that names its own
  fix costs nothing on the true-positive path and converts the false positive into a two-minute
  edit. **Message only — no behaviour change.** Existing 0.6.1 consumers can take this whenever they
  next touch the pin; it does not warrant a re-pin round on its own.

  Their argument for keeping the rule strict is recorded because it is better than the trade-off I
  had written down: the workaround is an improvement rather than a concession (it separates the
  application from its durable state at the filesystem level — the exact distinction the incident
  blurred); an `AllowStoreUnderContentRoot` flag has the lifecycle of a knob set once to clear a
  symptom and never revisited, surviving the redeploy that changes the layout it was reasoned
  against; and mount detection would make the verdict depend on runtime platform state, so the same
  path would pass on one host and fail on another — strictly worse than a stable false positive
  with a cheap documented fix.

## 0.6.1 — 2026-08-11

### Fixed

- **The 0.6.0 guard had a hole at exactly the moment the store file is created** (found by the
  HistorianGateway adoption of 0.6.0, which probed the rejected paths rather than observing a
  clean boot). An application that expands `${secret:}` references *before* building its host
  composes this subsystem in a throwaway `ServiceCollection` with **no `IHostEnvironment` in it**,
  then migrates the store from that container. `SecretsOptionsValidator` resolves the content root
  from `IHostEnvironment` and skips the under-content-root rule when it is absent — so pre-host an
  absolute in-content-root path passed the only rule still applying, the migrator **created the
  database**, and the boot then failed a second later once the real host registered
  `IHostEnvironment` and the rule finally fired.

  The observable result was a rejected boot that *still left an empty store, with `-wal`/`-shm`
  siblings, sitting at the rejected path* — the same artifact that made the outage this guard
  exists to prevent read as "the database is there, it's just empty". A relative path was
  unaffected, because the absolute-path rule fires with or without a content root; that asymmetry
  is what exposed it.

  Conflating "no `IHostEnvironment` registered" with "no content root exists" was the mistake. New
  overload **`AddZbSecrets(services, config, sectionPath, contentRootPath)`** takes the content
  root explicitly; the 3-argument overload is unchanged and still falls back to `IHostEnvironment`.
  **Pre-host composition should pass it** — e.g. `builder.Environment.ContentRootPath`. Passing
  `null` remains correct only where there genuinely is no content root, such as the CLI.
  `SecretsOptionsValidator` gains a `string?` constructor alongside the `IHostEnvironment?` one.

### Known interaction: containers, where the rule deliberately over-rejects

- **A containerized deployment can put the store on a mounted volume *inside* the content root, and
  0.6.1 will reject it.** Observed on the OtOpcUa docker-dev rig: `Secrets__SqlitePath` was
  `/app/data/…` with `/app` the container content root and `/app/data` a mounted volume — absolute,
  durable across container replacement, and entirely deliberate. The rule rejects it anyway.

  **This is a true positive for the rule as written and a false positive for the operator's
  intent**, and the difference is real: in a container the image is replaced on upgrade but the
  volume is not, so "inside the content root" is a weaker proxy for "destroyed on upgrade" there
  than it is for a Windows service whose whole directory gets renamed.

  It is not being relaxed, and no opt-out flag is being added. Distinguishing a mount from an
  ordinary directory means reading platform-specific mount tables, and an
  `AllowStoreUnderContentRoot` escape hatch is the kind of switch that gets set once to clear a
  boot failure and then outlives the reason — which is precisely the failure mode the guard exists
  to close. **The workaround is one line**: mount the same volume at a second path outside the
  content root (the rig used `/data`) and point `Secrets:SqlitePath` there. The file is at the
  volume root either way, so seeded stores carry over with no migration.

### Upgrade note (applies to 0.6.0 as well)

- **A store seeded under the pre-0.6.0 default is orphaned, not migrated, and nothing says so.**
  The old default was the relative `secrets.db`, which resolved against the process working
  directory; the new default is absolute. An application that relied on the old default silently
  starts using a different file, and anything previously seeded into the old one is stranded. This
  is a dev-loop concern for the apps in this family (every deployment sets an explicit absolute
  path), but check for a stray `*-secrets.db` beside your project before assuming an empty store
  after upgrade means the store was lost. Deleting the key from configuration, rather than leaving
  a relative value, is what makes the new default apply.

## 0.6.0 — 2026-08-11

### Added

- **`ZB.MOM.WW.Secrets` — fail-closed validation of the store path.** `Secrets:SqlitePath` is
  now validated at startup (`ValidateOnStart`) by `SecretsOptionsValidator`, using two rules in
  the new public `SecretsStorePathRules`. Both come from production incidents and neither
  implies the other:
  - **Must be absolute.** A relative value resolves against the process working directory —
    for a Windows service that is `C:\Windows\System32`, so the store lands outside the
    installation entirely; in a dev loop it lands in the source tree (observed: a gateway wrote
    its secrets database into `src/` and tripped that repo's own tree-hygiene test).
  - **Must not be inside the application content root.** An *absolute* path can still point
    into the directory every deployment procedure in this family replaces or renames on
    upgrade. On 2026-08-09 a gateway's credential store sat inside its `Server\` directory; the
    deploy renamed `Server` to `Server.bak.*`, the database left with it, and the process
    created a fresh empty one at the same path. Every consumer lost its credential, silently,
    for two days. The existing "absolute path" convention did not help — the path *was*
    absolute.

  The rules **reject rather than relocate**: silently moving a credential store presents later
  as an authentication failure pointing nowhere near the configuration that caused it, whereas
  a boot error is seen immediately by whoever caused it. Validation is deliberately **not**
  exempted in Development — a guard that never runs in a dev loop or in CI rots, and the
  configuration behind the outage looked entirely reasonable.

  The content-root rule is skipped when no `IHostEnvironment` is available (the CLI composes
  the subsystem without a host); the absolute-path rule still applies.

### Changed

- **BREAKING (configuration, not API): `SecretsOptions.SqlitePath` now defaults to an absolute
  per-user path** — `ZB.MOM.WW/secrets.db` under `LocalApplicationData` (`~/.local/share` on
  Linux and macOS, `%LOCALAPPDATA%` on Windows), exposed as `SecretsOptions.DefaultSqlitePath`.
  It was `secrets.db`, which the new rules reject.

  **A consuming application that configures a relative `Secrets:SqlitePath` will now fail to
  boot.** The fix is to remove the key (accepting the default) or set an absolute path outside
  the content root. Every deployment of this family already sets an absolute path, so this
  bites dev loops and unconfigured installs, not production — but verify before upgrading: a
  host that relied on the *old* default now looks in a new location, finds nothing, and creates
  an empty store, which is the same failure this release exists to prevent.

  Per-user rather than machine-wide is deliberate. `CommonApplicationData` is `/usr/share` on
  Linux and macOS, which a normal user cannot write; as a library default that would fail every
  consuming application's boot on a developer machine. An application is free to compute its own
  default (a machine-wide Windows service reasonably prefers `CommonApplicationData` alongside
  its other stores) — the configured value always wins.

- `AddZbSecrets` registration of the validator is idempotent. `AddZbSecrets` is legitimately
  called more than once in real compositions (a replication package calls it after the
  application already has), and `TryAddEnumerable` cannot de-duplicate a factory-built
  descriptor — it de-duplicates on implementation type, which a factory does not carry, so it
  throws on the second call rather than skipping.

- `ZB.MOM.WW.Secrets` now depends on `ZB.MOM.WW.Configuration` (0.1.0) for the shared
  `OptionsValidatorBase`/`ValidationBuilder` vocabulary. Every consumer already references it.

## 0.5.1 — 2026-08-07

### Fixed

- **`ZB.MOM.WW.Secrets.Replicator.SqlServer` — virgin-database concurrent migrator race**
  (present since 0.2.0; found by the ScadaBridge central shared-store live gate). The
  migrator's doc claimed simultaneous provisioning of a virgin database converges, but its
  retry filter covered only deadlock victims (1205): Serializable isolation does not
  serialize the `IF NOT EXISTS` metadata probes against another session's in-flight
  `CREATE`, so of two nodes booting together against an empty database the loser died with
  `SqlException` 2714 ("There is already an object named 'zbsecrets'") and aborted its
  host — observed live on an ordinary `docker compose up` of the ScadaBridge central pair.
  The retry filter now also treats the lost-CREATE/lost-INSERT race outcomes as retryable —
  2714 (schema/table already exists), 1913 (index already exists), 2627 (schema_version
  MERGE-insert PK violation) — since losing such a race means the object now exists, which
  is the desired end state; the retry re-runs the idempotent `IF NOT EXISTS` path and
  proceeds. Classification scans the whole `SqlException.Errors` collection (a failed DDL
  batch stacks follow-on errors such as 2759, so the retryable number is not guaranteed
  first), and the retry cap (5) keeps a genuine conflict — e.g. an unrelated pre-existing
  object colliding with the schema name — failing closed. 2705 (duplicate column) is
  deliberately not retried: every column lives inside a single `CREATE TABLE` statement,
  so no cross-session race can produce it. The error-number predicate is now an internal
  static pinned by offline tests, and the live suite gains an N-concurrent-migrators
  virgin-database test.

## 0.5.0 — 2026-08-07

### Added

- **`ZB.MOM.WW.Secrets.Replicator.Grpc` — hub endpoint failover for followers**
  (scadaproj#4). `SecretsGrpcHubClientOptions` gains `FallbackEndpoints`: additional hub
  addresses tried, in order, when a call to the primary `Endpoint` (or an earlier fallback)
  fails with a transport-level gRPC error. The answering endpoint is sticky — a sweep that
  failed over does not re-pay the dead endpoint's full `CallDeadline` on every subsequent
  call — and rotation wraps, so a recovered primary is retried the next time the stuck-to
  endpoint fails. Only `RpcException` triggers failover; cancellation and everything else
  propagate untouched, and when every endpoint fails the last `RpcException` is rethrown so
  the sweep degrades to last-known-good exactly as before. Each failover logs a Warning
  naming the failed and next endpoint URIs (never any credential material).
  - **Same-store requirement:** list only endpoints that serve the SAME store — for
    ScadaBridge, the central pair over one shared SQL Server store. Endpoints backed by
    independent stores turn a primary outage into a silent convergence stop (failing over
    to an emptier hub "succeeds"), which is the exact defect scadaproj#4 recorded.
  - Validation fails startup on a blank/relative fallback or one duplicating the primary
    or another fallback (a duplicated endpoint silently halves the configured failover).

### Unchanged

- With `FallbackEndpoints` empty (the default), the registration graph is identical to
  0.4.x — one keyed channel, one `GrpcSecretsHubClient`, no failover wrapper — and the wire
  behavior of every call is byte-identical. Every other package in the family is
  behaviorally unchanged in this release.

## 0.4.1 — 2026-08-07

### Fixed

- **`ZB.MOM.WW.Secrets.Replicator.Grpc` — the hub now logs its denials.** A ScadaBridge live
  gate found that `SecretsHubAuthInterceptor` refused calls correctly but wrote nothing about
  it: the only trace of a refusal was a framework `Grpc.AspNetCore.Server` Information line, so
  a follower whose bearer token had been mis-rotated silently stopped converging and the hub
  showed it only at INF. Every denial is now counted, and a `Warning` summarising the window is
  emitted on the first denial after startup or after 60 quiet seconds — carrying the number of
  denials since the previous summary plus a breakdown by cause (missing/malformed credential,
  wrong credential, hub token unconfigured). A single probe is therefore visible immediately,
  while a sustained attack is bounded to one log line per minute — the endpoint is
  unauthenticated by construction, so a per-denial warning would be an unbounded log write
  driven by an unauthenticated caller.

### Unchanged

- **Wire behavior is byte-identical.** The cause breakdown is server-side diagnostics only; every
  rejection still returns the same uniform `Unauthenticated` status with the same detail, and the
  credential comparison is still constant-time. No token material, presented or configured,
  appears in any log line at any level.
- The existing warn-once-on-unconfigured-token line is unchanged and still emitted exactly once;
  denials in that state also count toward the new summary's `unconfigured` category.
- Every other package in the family is behaviorally unchanged in this release.

## 0.4.0 — 2026-08-07

### Added

- **New package `ZB.MOM.WW.Secrets.Replicator.Grpc`** — a central-hosted `secrets_hub.v1`
  gRPC hub for the sites-talk-to-central production topology (scadaproj#3; design doc
  [`docs/plans/2026-08-07-secrets-production-topology-and-grpc-hub.md`](../docs/plans/2026-08-07-secrets-production-topology-and-grpc-hub.md)).
  - **Pull-only by construction**: the proto exposes no write RPC. Updates originate at
    central only; a site never pushes a secret upstream.
  - **Fail-closed bearer auth** on the hub server: constant-time comparison, uniform denial
    response (no signal distinguishing "wrong token" from "missing token"), a warn-once log
    on an unconfigured/empty token rather than a silent open door.
  - `GetSecrets` enforces a batch-size cap, rejecting an oversized request with
    `ResourceExhausted` instead of serving it partially or unbounded.
  - Site-side `GrpcSecretSyncService` sweeps the hub on an interval with **per-row parse
    isolation** — one malformed/undecryptable row logs and is skipped, it does not fault the
    whole sweep.
  - Keyed `GrpcChannel` registration so multiple hub targets can be wired in one process
    without channel collisions.
  - **Offline behavior is last-known-good**: if the hub is unreachable, the site keeps
    serving the most recently synced values rather than failing secret resolution.

### Unchanged

- `ZB.MOM.WW.Secrets.Abstractions`, `ZB.MOM.WW.Secrets`, `ZB.MOM.WW.Secrets.Ui`,
  `ZB.MOM.WW.Secrets.Replicator.SqlServer`, and `ZB.MOM.WW.Secrets.Replicator.AkkaDotNet` are
  behaviorally unchanged in this release — no breaking changes to any existing package.

## 0.3.0 — 2026-07-19

- Menu-driven Spectre.Console interactive operator console added to the `secret` CLI (list /
  set / reveal / delete, reference audit & seed, KEK doctor, ciphertext-only bundle
  export/import). See
  [`docs/operations/interactive-console.md`](docs/operations/interactive-console.md).

## 0.2.3 — 2026-07-19

- `ZB.MOM.WW.Secrets.Ui`: fixed the delete-confirmation modal being permanently invisible on
  every Bootstrap host (`ConfirmDeleteModal` used bare `class="modal"` with no shipped CSS;
  Bootstrap's `.modal{display:none}` hid it everywhere but mxaccessgw). Fixed via a
  `zb-secrets-modal*` class rename plus a component-emitted `<style>` block.

## 0.2.2 — 2026-07-18

- Fixed the Akka replicator's hosted-process DI deadlock (a circular singleton dependency
  hidden behind factory lambdas that MS.DI's `StackGuard` turned into a silent cross-thread
  hang rather than a stack overflow). New `DeferredSecretCacheInvalidator` defers the one
  cycle-closing edge to first eviction.

## 0.2.1 — 2026-07-18

- Fixed the Akka replicator being silently inert since 0.2.0 (a `TryAdd` registration-order
  bug published into a no-op sink and never spawned an actor). SQL-Server replication was
  never affected.

## 0.2.0 — 2026-07-18

- Added clustered replication (G-7): `ZB.MOM.WW.Secrets.Replicator.SqlServer` (shared store,
  or local-store-with-hub) and `ZB.MOM.WW.Secrets.Replicator.AkkaDotNet` (peer-to-peer over
  distributed pub/sub).