Owner chose option (b). ZB.MOM.WW.Secrets.Replicator.Grpc has had no consumer
since ScadaBridge retired the gRPC secrets hub (2026-09-28); OtOpcUa never used it.
- removed the project and its test project from ZB.MOM.WW.Secrets.slnx, so the
family build/test and build/push.sh's `dotnet pack` skip it;
- csproj IsPackable=false (a direct pack of the project yields nothing);
- SecretsGrpcHubServerExtensions / SecretsGrpcHubClientExtensions [Obsolete];
- README, CHANGELOG (0.6.6 Deprecated), CLAUDE.md and clustered-secrets.md say so.
Source + tests stay in the tree and still build/run standalone (test csproj
suppresses CS0618): 200/200. Feed versions 0.4.1-0.6.3 are left for the owner
to unlist.
Also zeroes the solution's pre-existing warnings: CA1416 on two Unix-only
permission tests (guarded by Skip.IfNot, which the analyzer cannot see — now
[UnsupportedOSPlatform("windows")]) and two xUnit2031 Assert.Single(Where).
Build 0 warnings; tests 429 passed / 21 skipped / 0 failed; pack = 5 nupkgs.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The CLI host takes its working directory as the content root, and the rig put the store under it.
The old lexical store-path rule missed that on macOS only because the temp path is spelled
/var/... while the working directory reports /private/var/...; with Secrets-005's link resolution
the rule (correctly) refused it. The rig now runs from a sibling work/ directory.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Secrets-018: BundleService.ImportAsync now rehydrates every entry and re-wraps every foreign-KEK
row before the first write, so a malformed or tampered entry anywhere fails the import with the
store untouched ("Nothing was imported"). A store fault during the apply phase (no multi-row
transaction) now reports the row it stopped at and every row already written, which the
interactive flow prints, instead of a bare "Import failed".
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Secrets-009: SecretReferenceExpander.ExpandAsync throws FormatException when a value holds a
${secret: marker that is not a complete reference with a non-blank name (${secret:}, an
unterminated ${secret:name, a nested marker), instead of passing the literal text through as the
credential. ExpandConfigurationAsync names the offending key (never the value) and writes nothing.
The README wiring sample drops the relative "SqlitePath": "secrets.db" (which failed validation)
for the per-user default and shows an absolute example; a test validates the sample.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The CLI's own appsettings.json set "SqlitePath": "secrets.db", which the store-path validator
rejects as relative, so the headless CLI failed at startup when run from its own directory unless
Secrets__SqlitePath overrode it. The key is removed: the per-user default
(<LocalApplicationData>/ZB.MOM.WW/secrets.db) applies, and an app's appsettings.json in the working
directory or the environment still override it. Regression test validates the shipped file.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Secrets-006: DefaultSecretResolver keeps a per-key invalidation generation; Invalidate bumps it
before evicting, and a resolve that read the store under an older generation withdraws its insert,
so a rotation mid-resolve is not undone for another TTL. Expired entries are removed on lookup and
by a periodic TimeProvider sweep (every TTL, at least 1 s); a non-positive TTL caches nothing.
The resolver is now IDisposable (stops the sweep timer).
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Secrets-005: AddIfNotRooted now uses Path.IsPathFullyQualified, so Windows drive-relative
(C:secrets.db) and root-relative (\secrets.db) paths are rejected. AddIfUnderContentRoot refuses a
store if EITHER its spelled location (device prefix stripped, loopback admin share mapped to its
drive) OR its resolved location (symlinks/junctions followed, 8.3 names expanded on Windows) is
under the likewise-resolved content root. A store spelled through the content root stays refused
even if a link inside carries it elsewhere. UNC paths through other shares are compared as written.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Secrets-004: AesGcmEnvelopeCipher.Decrypt now wraps the ArgumentException AesGcm throws for a
wrong-length nonce/tag/wrapped DEK as SecretDecryptionException (as Rewrap already did), so the
resolver's decryption-failed audit fires.
Secrets-007: new MasterKeySnapshot (key + KekId from one reading, zeroed on Dispose) and a
default-implemented IMasterKeyProvider.GetSnapshot(); MasterKeyProviderBase overrides it to
resolve the source once and derive the id from that same key. Encrypt/Decrypt/Rewrap take one
snapshot per KEK and dispose it, so a key file swapped mid-operation can no longer stamp a row
with one key's id and wrap it under another. Intermediate key buffers are zeroed. On-disk format
unchanged: a golden row written by the 0.6.5 cipher still decrypts.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
`secret set`/`rotate` took the plaintext as a positional argument (shell
history, process list), and Program handed argv to
Host.CreateApplicationBuilder, whose command-line config provider crashed on
a value like "-p=Hunter2Secret" with the value in the unhandled
FormatException on stderr.
- SetCommandLine: `set|rotate <name> [-|--value-stdin]`; no value reads piped
stdin or prompts twice (masked) at a TTY; one trailing newline dropped;
empty refused. The positional <value> still works and warns on stderr.
- host built from HostApplicationBuilderSettings without argv; startup
failures print `secret: startup failed: <msg>` and exit 1
- unexpected trailing arguments reported by position, not echoed
- tests: SetCommandLineTests incl. two child-process runs of the built CLI;
the -p=Hunter2Secret probe fails (leaks) on the old Program.cs
- docs: README, clustered-secrets runbook, CHANGELOG
- code-reviews: Secrets-017 Resolved; README regenerated
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
ExpandConfigurationAsync wrote through config[key] = ..., which only sets the
providers' in-memory Data; a file provider's reload replaces it, so editing
appsettings.json on a running service reverted every expanded key to its
${secret:...} placeholder.
- ConfigurationManager: expanded values are pinned in an internal overlay
source appended last; its provider's Load() is a no-op and Build always
returns the same provider, so reloads (and source rebuilds) keep the pins
- any other IConfigurationRoot: in-place write as before, re-applied from the
root's reload token
- all references resolve before anything is written (a fail-closed miss no
longer leaves the config half expanded)
- tests: SecretExpansionReloadTests (real JSON file + Reload(), options
monitor, plain root); three fail without the fix
- docs: README reload note, CHANGELOG
- code-reviews: Secrets-003 Resolved; README regenerated
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The SQL-Server extensions called the 3-arg AddZbSecrets, and the store-path
validator captured the content root of whichever call registered it first.
In hub mode no registration order applied the pre-host under-content-root
guard: extension-first dropped the app's 4-arg root, core-first lost the
replicator (#30). That is the 0.6.0 store-path hole, reopened.
- AddZbSecretsSqlServerStore / AddZbSecretsSqlServerReplication /
AddZbSecretsAkkaReplication gain a trailing optional contentRootPath,
forwarded to the 4-arg AddZbSecrets
- the root now lives in one SecretsContentRoot instance holder per service
collection; the last explicit root wins, null never erases one; the
IHostEnvironment fallback is unchanged
- tests: ContentRootTests (hub mode, three compositions, artifact asserted
before the throw, positive control), last-explicit-wins theory in
SecretsStorePathRulesTests, Akka forwarding test
- code-reviews: Secrets-002 Resolved; README regenerated
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The two Unix-permission tests called Skip.IfNot from a plain [Fact] (xunit v2 +
Xunit.SkippableFact) so they failed on Windows; now [SkippableFact].
Resolver_Works_EndToEnd_ThroughDi calls SqliteConnection.ClearAllPools() before
deleting its temp DB.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Several setups migrating one SQLite secret store at once (mxaccessgw's parallel test hosts on the
per-process store) failed intermittently on Windows with "cannot start a transaction within a
transaction" from SqliteSecretsStoreMigrator.MigrateAsync. That message means the BEGIN ran on a
handle that was already in a transaction: the pinned Microsoft.Data.Sqlite 10.0.7 can hand one
pooled handle to two owners (dotnet/efcore#39009, fixed in 10.0.13) and no 10.0.x refuses to
re-pool a handle whose rollback failed (dotnet/efcore#39082).
The migrator now serialises migrations of one store per process behind a gate keyed on the
normalised path, runs each attempt on a dedicated unpooled connection inside BEGIN IMMEDIATE, and
retries SQLITE_BUSY/SQLITE_LOCKED (another process past the busy timeout) with equal-jitter
exponential backoff. Store operations and the Microsoft.Data.Sqlite pin are unchanged.
Tests: concurrent-init (16 hosts x 20 rounds on fresh and existing stores, two path spellings,
16 AddZbSecrets containers running the pre-host migrator + hosted service), gate sharing/waiting,
BUSY retry past the 5 s busy timeout, transient-code classification.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Suggested by the OtOpcUa session, the only real containerized deployment in this family
and the one that hit the false positive. A store on a volume mounted inside the content
root is durable across container replacement but is rejected anyway, because the rule
cannot tell a mount from an ordinary directory. The message now says so, gives the fix,
and forecloses the wrong one by stating that no override exists.
A false positive's entire cost is the confusion in the moment, so an error that names its
own fix costs nothing on the true-positive path and turns the false positive into a
two-minute edit. Message only, no behaviour change.
Their case for keeping the rule strict is recorded in the CHANGELOG because it is better
than the trade-off I had written: the workaround is an improvement rather than a
concession, an override flag has the lifecycle of a knob set once and never revisited,
and mount detection would make the verdict depend on runtime platform state so the same
path would pass on one host and fail on another.
Raised by the mxaccessgw session, which went back to its own tests after I reported a
vacuous assertion in mine and found the same class one step further from the fuse.
The rejecting test asserts no store file is left at a rejected path. That claim is only
meaningful if something proves the write path is live at all: "no file was created" is
indistinguishable from "nothing in this container ever creates files". The positive
control was supposed to establish that and did not -- it asserted Assert.NotNull on a
resolved SecretsSqliteConnectionFactory, which is close to a tautology once no exception
escaped, and it never ran the migration at all. So if the pre-host container ever stopped
touching the store, the rejecting test would have gone quietly vacuous while the positive
control kept passing. Neither would have failed.
The positive control now runs SqliteSecretsStoreMigrator.MigrateAsync and asserts the
database IS created. The pair is mutually checking: the migration writes the file when
allowed and does not when rejected, and neither assertion stands on its own.
Verified the same way as the last one, by simulating the failure rather than reasoning
about it: replacing the migration with a no-op makes the positive control fail with "the
accepted path must actually get a store file". Reverted after measuring.
No library change; 0.6.1 as published is unaffected. Suite 502 passing.
Two defects in the 0.6.1 regression test, the first raised by the mxaccessgw session
from its own adoption, the second exposed by acting on it.
Assertion order. The test asserted the throw before the artifact. That hides the
stronger finding: a regression that stops the throw but still creates the store fails
with "no exception was thrown" and never mentions the database sitting at the rejected
path -- which IS the defect. Now captures with Record.ExceptionAsync, asserts no file
created first, and asserts the exception type last. Notable because the defect under
test was itself "a boot outcome was observed instead of the artifact being probed", so
closing it with an outcome-first assertion would have repeated the mistake.
Vacuous assertions. Simulating the regression -- replacing the 4-argument call with the
3-argument one, which still compiles, exactly as the call-site comments warn -- showed
the test failing on the exception assertion rather than the file assertion. No file had
been created in EITHER case: the test resolved SecretsSqliteConnectionFactory, whose
constructor never touches the filesystem. The File.Exists assertions could not fail. The
test now resolves SqliteSecretsStoreMigrator and runs MigrateAsync, which is what a
pre-host expander actually does and what actually creates the database.
Verified by simulating the regression again: the test now fails with "a rejected path
must not be left holding a store file" rather than "no exception was thrown". Reverted
after measuring.
No library change; 0.6.1 as published is unaffected. Suite 502 passing.
Found by the HistorianGateway adoption of 0.6.0, which probed the rejected paths
instead of observing a clean boot — a clean boot being equally consistent with the
validator not being wired at all.
An application that expands ${secret:} references before building its host composes
this subsystem in a throwaway ServiceCollection with no IHostEnvironment in it, then
migrates the store from that container. SecretsOptionsValidator resolves the content
root from IHostEnvironment and skips the under-content-root rule when it is absent, so
pre-host an absolute in-content-root path passed the only rule still applying, the
migrator CREATED the database, and the boot failed a second later once the real host
registered IHostEnvironment and the rule finally fired.
The observable result was a rejected boot that still left an empty store, with -wal and
-shm siblings, at the rejected path — the same artifact that made the MxGateway outage
read as "the database is there, it's just empty", which is precisely what this guard
exists to prevent. A relative path was unaffected because the absolute-path rule fires
with or without a content root; that asymmetry is what exposed it.
The mistake was conflating "no IHostEnvironment registered" with "no content root
exists". New AddZbSecrets overload takes the content root explicitly; the 3-argument
overload is unchanged and still falls back to IHostEnvironment, so this is additive.
SecretsOptionsValidator gains a string? constructor alongside the IHostEnvironment? one.
Reproduced first in an isolated harness before changing anything: a bare host rejects
both bad paths leaving no artifacts, and a bare ServiceCollection with no
IHostEnvironment migrates an in-content-root path to disk. Both are now regression tests,
the second paired with a positive control asserting the old behaviour still holds when no
content root is supplied — so the test is about the explicit content root rather than
about paths in general.
Also documents an upgrade note that applies to 0.6.0: a store seeded under the old
relative default is orphaned rather than migrated, and nothing signals it.
Secrets:SqlitePath is now validated at startup by SecretsOptionsValidator using two
rules in the new public SecretsStorePathRules. Both come from production incidents and
neither implies the other.
Must be absolute: a relative value resolves against the process working directory,
which for a Windows service is C:\Windows\System32, so the store lands outside the
installation; in a dev loop it lands in the source tree (a gateway was observed writing
its secrets database into src/, tripping that repo's own tree-hygiene test).
Must not be inside the content root: an absolute path can still point into the
directory every deployment procedure in this family renames on upgrade. On 2026-08-09 a
gateway's credential store sat inside its Server\ directory; the deploy renamed Server
to Server.bak.*, the database went with it, and the process created a fresh empty one
at the same path. Every consumer lost its credential, silently, for two days. The
existing "must be absolute" convention did not help — the path was absolute.
The rules reject rather than relocate. Silently moving a credential store presents later
as an authentication failure pointing nowhere near the configuration that caused it; a
boot error is seen immediately by whoever caused it. Validation is deliberately not
exempted in Development: a guard that never runs in a dev loop or in CI rots, and the
configuration behind the outage looked entirely reasonable.
BREAKING for configuration, not API. SecretsOptions.SqlitePath now defaults to an
absolute per-user path under LocalApplicationData; it was "secrets.db", which the new
rules reject. An application configuring a relative path will fail to boot — remove the
key or set an absolute one outside the content root. All five deployed instances of this
family were checked first and already set an absolute path outside their content root,
so the migration edge (a host relying on the OLD default now finding nothing and
creating an empty store — this bug, re-introduced by its own fix) does not apply to any
of them.
Per-user rather than machine-wide is deliberate: CommonApplicationData is /usr/share on
Linux and macOS, which a normal user cannot write, so a machine-wide library default
would fail every consuming application's boot on a developer machine. An application may
compute its own default; the configured value always wins.
Validator registration is idempotent. AddZbSecrets is legitimately called more than once
in real compositions, and TryAddEnumerable cannot de-duplicate a factory-built
descriptor — it de-duplicates on implementation type, which a factory does not carry, so
it throws on the second call rather than skipping. The existing suite caught this.
Tests cover both rules including the segment-boundary case (/srv/app-data must not read
as inside /srv/app — a false positive on a fail-closed startup rule is an application
that will not boot), that the shipped default satisfies the rules it ships with, that
the default is not machine-wide, idempotent registration, and — most importantly — that
a bad path actually throws when IOptions is resolved, since a correctly implemented
validator that is never invoked is the same "guard that does not fire" failure this
change exists to prevent.
Family bumps to 0.6.0. Adds a dependency on ZB.MOM.WW.Configuration 0.1.0 for the shared
OptionsValidatorBase/ValidationBuilder vocabulary, which every consumer already
references.
Environment.SetEnvironmentVariable/GetEnvironmentVariable mutate the
process-wide environ; concurrent setenv/getenv across parallel xunit test
classes is a documented crash/flake source on glibc even with unique
variable names. Group the six classes that touch process env vars into a
DisableParallelization=true collection so they run serially against each
other while everything else keeps running in parallel.
Secrets were per-node SQLite, so a secret written on one node was invisible to
the rest of a cluster. G-7's design resolved the "shared SQL store vs Akka
replicator" fork to build only the former; both are built here so the choice is
a deployment decision (availability vs partition tolerance) rather than a
library limitation.
Two new packages — ZB.MOM.WW.Secrets.Replicator.SqlServer (shared store, plus a
local-store-with-hub mode) and .Replicator.AkkaDotNet (peer-to-peer over
distributed pub/sub). Core gains ISecretsStoreMigrator, one shared
SecretLastWriterWins predicate so no two stores can disagree on a tie, the
transport-agnostic reconciler, and ReplicatingSecretStore — which closes a real
gap: nothing had ever called ISecretReplicator.PublishAsync, so the seam was
inert and local writes would not have propagated at all.
Verified 182 pass / 1 skip / 0 warnings, including 15 live tests against a real
SQL Server 2022 (the SQLite suite ported case-for-case, so any behavioural
divergence between the stores fails) and a 9-test in-process 2-node Akka
cluster over real remoting. A post-build review caught six defects, all fixed
and now covered: both replication modes could not resolve from the container
(no test had built one), an unbounded fetch that broke past SQL Server's
2100-parameter cap, a poison row that aborted the rest of its batch forever,
Enum.Parse on peer input that could restart the actor in a loop, null crypto
blobs crossing the trust boundary, and a silently dropped pull-read failure.
Packed at 0.2.0 and vulnerability-scanned clean; not yet published to the feed.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
G-8 (KEK rotation) — built in ZB.MOM.WW.Secrets, lib 0.1.2->0.1.3:
- ISecretCipher.Rewrap(row, oldKek, newKek): re-wraps the per-secret DEK only
(bodies never re-encrypted; revision/timestamps preserved -> invisible to
cluster LWW). Fail-closed on wrong old-KEK id, wrong key bytes, and malformed
wraps; DEK zeroed on all paths.
- ISecretStore.ApplyRewrapAsync(rewrappedRow, expectedCurrentWrappedDek):
updates only the 4 wrap columns + kek_id, compare-and-swap on the current
wrapped DEK so a concurrent set/rotate cannot corrupt a row (closes a
review-caught TOCTOU).
- KekRotationService.RewrapAllAsync + RewrapReport: enumerate all rows incl.
tombstones, idempotent/resumable skip-already-current, bounded CAS-retry,
fail-closed on unknown/identical KEK.
- `secret rewrap-all` CLI verb: key material only via env-var name / file path,
JSON counts report; README section + operator runbook.
Verified: full offline suite green (82 core + 15 UI, 0 regressions) + end-to-end
CLI smoke + adversarial crypto review (all 7 categories PASS; TOCTOU fixed).
G-7 (clustered replication) — designed + planned, no code:
- Fork resolved to build Option A (shared SQL-Server ISecretStore); Akka
replicator ZB.MOM.WW.Secrets.Akka is a deferred phase-2. Design doc +
executable plan + .tasks.json under docs/plans/2026-07-17-secrets-g7-*.
Tracking: components/secrets/GAPS.md + CLAUDE.md secrets row updated.
Live HistorianGateway boot caught that the expander scanned a '_secretsComment' appsettings
value documenting the ${secret:...} syntax, tried to resolve the literal example token, and
crashed boot. The family uses _*Comment keys pervasively, so skip any config key whose leaf
segment starts with '_' (the _comment convention). Functional keys never lead with '_'.
Joseph Doherty