Owner chose option (b). ZB.MOM.WW.Secrets.Replicator.Grpc has had no consumer
since ScadaBridge retired the gRPC secrets hub (2026-09-28); OtOpcUa never used it.
- removed the project and its test project from ZB.MOM.WW.Secrets.slnx, so the
family build/test and build/push.sh's `dotnet pack` skip it;
- csproj IsPackable=false (a direct pack of the project yields nothing);
- SecretsGrpcHubServerExtensions / SecretsGrpcHubClientExtensions [Obsolete];
- README, CHANGELOG (0.6.6 Deprecated), CLAUDE.md and clustered-secrets.md say so.
Source + tests stay in the tree and still build/run standalone (test csproj
suppresses CS0618): 200/200. Feed versions 0.4.1-0.6.3 are left for the owner
to unlist.
Also zeroes the solution's pre-existing warnings: CA1416 on two Unix-only
permission tests (guarded by Skip.IfNot, which the analyzer cannot see — now
[UnsupportedOSPlatform("windows")]) and two xUnit2031 Assert.Single(Where).
Build 0 warnings; tests 429 passed / 21 skipped / 0 failed; pack = 5 nupkgs.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Secrets-004–009, 011, 012, 014–016, 018 and 019 marked Resolved with commit, fix and regression
evidence; Secrets open findings 13 -> 0; code-reviews/README.md regenerated (--check passes).
CHANGELOG 0.6.6 records the fixes, behaviour changes and the #49 follow-ups.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
`secret set`/`rotate` took the plaintext as a positional argument (shell
history, process list), and Program handed argv to
Host.CreateApplicationBuilder, whose command-line config provider crashed on
a value like "-p=Hunter2Secret" with the value in the unhandled
FormatException on stderr.
- SetCommandLine: `set|rotate <name> [-|--value-stdin]`; no value reads piped
stdin or prompts twice (masked) at a TTY; one trailing newline dropped;
empty refused. The positional <value> still works and warns on stderr.
- host built from HostApplicationBuilderSettings without argv; startup
failures print `secret: startup failed: <msg>` and exit 1
- unexpected trailing arguments reported by position, not echoed
- tests: SetCommandLineTests incl. two child-process runs of the built CLI;
the -p=Hunter2Secret probe fails (leaks) on the old Program.cs
- docs: README, clustered-secrets runbook, CHANGELOG
- code-reviews: Secrets-017 Resolved; README regenerated
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
SecretEditor and ConfirmDeleteModal wrote the success audit inside the same
try as the store write, so a throwing IAuditWriter after a committed
add/rotate/delete showed an error, wrote a Failure audit and skipped the
resolver-cache eviction - the old plaintext kept resolving for the TTL.
The mutation try now covers only the cipher + store call; the success audit
is best-effort afterwards and logs a warning (no value) on failure, so the
change is reported saved, the callback fires and the cache is evicted.
- tests: AuditFailureAfterCommitTests (add, rotate, delete) with a throwing
success audit; all fail without the fix
- code-reviews: Secrets-013 Resolved; README regenerated
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The replication actor was created only when something resolved ISecretStore
(store -> replicator -> SecretReplicationActorProvider.ActorRef). A node that
expanded its ${secret:} references pre-host and never touched the store at
runtime had no actor: no topic subscription, no announces, no pull replies.
AddZbSecretsAkkaReplication now registers SecretReplicationActorStarter, which
creates the actor on IHostApplicationLifetime.ApplicationStarted (after every
hosted service, incl. an Akka.Hosting AkkaHostedService registered later), or
in StartAsync when no lifetime is registered.
- tests: EagerActorStartTests (generic host in both registration orders with a
no-actor-before-start control; bare container) - all fail without the fix
- docs: README, CHANGELOG
- code-reviews: Secrets-010 Resolved; README regenerated
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
ExpandConfigurationAsync wrote through config[key] = ..., which only sets the
providers' in-memory Data; a file provider's reload replaces it, so editing
appsettings.json on a running service reverted every expanded key to its
${secret:...} placeholder.
- ConfigurationManager: expanded values are pinned in an internal overlay
source appended last; its provider's Load() is a no-op and Build always
returns the same provider, so reloads (and source rebuilds) keep the pins
- any other IConfigurationRoot: in-place write as before, re-applied from the
root's reload token
- all references resolve before anything is written (a fail-closed miss no
longer leaves the config half expanded)
- tests: SecretExpansionReloadTests (real JSON file + Reload(), options
monitor, plain root); three fail without the fix
- docs: README reload note, CHANGELOG
- code-reviews: Secrets-003 Resolved; README regenerated
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The SQL-Server extensions called the 3-arg AddZbSecrets, and the store-path
validator captured the content root of whichever call registered it first.
In hub mode no registration order applied the pre-host under-content-root
guard: extension-first dropped the app's 4-arg root, core-first lost the
replicator (#30). That is the 0.6.0 store-path hole, reopened.
- AddZbSecretsSqlServerStore / AddZbSecretsSqlServerReplication /
AddZbSecretsAkkaReplication gain a trailing optional contentRootPath,
forwarded to the 4-arg AddZbSecrets
- the root now lives in one SecretsContentRoot instance holder per service
collection; the last explicit root wins, null never erases one; the
IHostEnvironment fallback is unchanged
- tests: ContentRootTests (hub mode, three compositions, artifact asserted
before the throw, positive control), last-explicit-wins theory in
SecretsStorePathRulesTests, Akka forwarding test
- code-reviews: Secrets-002 Resolved; README regenerated
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
AddZbSecretsSqlServerStore / AddZbSecretsSqlServerReplication (and the Akka
extension, same defect class) TryAdd-ed their ISecretStore, ISecretsStoreMigrator
and ISecretReplicator ahead of their internal AddZbSecrets call. An application
that had already called AddZbSecrets therefore silently kept the SQLite store
(shared-store mode) or NoOpSecretReplicator (hub/Akka modes).
They now call AddZbSecrets first and RemoveAll + AddSingleton their own
registrations; a later AddZbSecrets only TryAdds, so either order yields the
same graph with exactly one ISecretStore registration.
- tests: RegistrationOrderTests (SqlServer + Akka) build and resolve the real
container in both orders; core-first fails without the fix
- Directory.Build.props 0.6.5 -> 0.6.6 (NOT packed or published)
- code-reviews: Secrets-001 Resolved; README regenerated
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
10.0.7's pool can hand one handle to two users (efcore#39009). 10.0.13 floors
SQLitePCLRaw 2.1.13, so the transitive lib.e_sqlite3 pin moves 2.1.12 -> 2.1.13
(the old pin is a NU1109 downgrade error). NuGetAudit clean; all suites green.
Versions bumped, NOT published: Secrets 0.6.5, LocalDb 0.3.5, Auth 0.4.3.
GalaxyRepository and Overview do not use Microsoft.Data.Sqlite.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Several setups migrating one SQLite secret store at once (mxaccessgw's parallel test hosts on the
per-process store) failed intermittently on Windows with "cannot start a transaction within a
transaction" from SqliteSecretsStoreMigrator.MigrateAsync. That message means the BEGIN ran on a
handle that was already in a transaction: the pinned Microsoft.Data.Sqlite 10.0.7 can hand one
pooled handle to two owners (dotnet/efcore#39009, fixed in 10.0.13) and no 10.0.x refuses to
re-pool a handle whose rollback failed (dotnet/efcore#39082).
The migrator now serialises migrations of one store per process behind a gate keyed on the
normalised path, runs each attempt on a dedicated unpooled connection inside BEGIN IMMEDIATE, and
retries SQLITE_BUSY/SQLITE_LOCKED (another process past the busy timeout) with equal-jitter
exponential backoff. Store operations and the Microsoft.Data.Sqlite pin are unchanged.
Tests: concurrent-init (16 hosts x 20 rounds on fresh and existing stores, two path spellings,
16 AddZbSecrets containers running the pre-host migrator + hosted service), gate sharing/waiting,
BUSY retry past the 5 s busy timeout, transient-code classification.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Suggested by the OtOpcUa session, the only real containerized deployment in this family
and the one that hit the false positive. A store on a volume mounted inside the content
root is durable across container replacement but is rejected anyway, because the rule
cannot tell a mount from an ordinary directory. The message now says so, gives the fix,
and forecloses the wrong one by stating that no override exists.
A false positive's entire cost is the confusion in the moment, so an error that names its
own fix costs nothing on the true-positive path and turns the false positive into a
two-minute edit. Message only, no behaviour change.
Their case for keeping the rule strict is recorded in the CHANGELOG because it is better
than the trade-off I had written: the workaround is an improvement rather than a
concession, an override flag has the lifecycle of a knob set once and never revisited,
and mount detection would make the verdict depend on runtime platform state so the same
path would pass on one host and fail on another.
Found by the HistorianGateway adoption of 0.6.0, which probed the rejected paths
instead of observing a clean boot — a clean boot being equally consistent with the
validator not being wired at all.
An application that expands ${secret:} references before building its host composes
this subsystem in a throwaway ServiceCollection with no IHostEnvironment in it, then
migrates the store from that container. SecretsOptionsValidator resolves the content
root from IHostEnvironment and skips the under-content-root rule when it is absent, so
pre-host an absolute in-content-root path passed the only rule still applying, the
migrator CREATED the database, and the boot failed a second later once the real host
registered IHostEnvironment and the rule finally fired.
The observable result was a rejected boot that still left an empty store, with -wal and
-shm siblings, at the rejected path — the same artifact that made the MxGateway outage
read as "the database is there, it's just empty", which is precisely what this guard
exists to prevent. A relative path was unaffected because the absolute-path rule fires
with or without a content root; that asymmetry is what exposed it.
The mistake was conflating "no IHostEnvironment registered" with "no content root
exists". New AddZbSecrets overload takes the content root explicitly; the 3-argument
overload is unchanged and still falls back to IHostEnvironment, so this is additive.
SecretsOptionsValidator gains a string? constructor alongside the IHostEnvironment? one.
Reproduced first in an isolated harness before changing anything: a bare host rejects
both bad paths leaving no artifacts, and a bare ServiceCollection with no
IHostEnvironment migrates an in-content-root path to disk. Both are now regression tests,
the second paired with a positive control asserting the old behaviour still holds when no
content root is supplied — so the test is about the explicit content root rather than
about paths in general.
Also documents an upgrade note that applies to 0.6.0: a store seeded under the old
relative default is orphaned rather than migrated, and nothing signals it.
Secrets:SqlitePath is now validated at startup by SecretsOptionsValidator using two
rules in the new public SecretsStorePathRules. Both come from production incidents and
neither implies the other.
Must be absolute: a relative value resolves against the process working directory,
which for a Windows service is C:\Windows\System32, so the store lands outside the
installation; in a dev loop it lands in the source tree (a gateway was observed writing
its secrets database into src/, tripping that repo's own tree-hygiene test).
Must not be inside the content root: an absolute path can still point into the
directory every deployment procedure in this family renames on upgrade. On 2026-08-09 a
gateway's credential store sat inside its Server\ directory; the deploy renamed Server
to Server.bak.*, the database went with it, and the process created a fresh empty one
at the same path. Every consumer lost its credential, silently, for two days. The
existing "must be absolute" convention did not help — the path was absolute.
The rules reject rather than relocate. Silently moving a credential store presents later
as an authentication failure pointing nowhere near the configuration that caused it; a
boot error is seen immediately by whoever caused it. Validation is deliberately not
exempted in Development: a guard that never runs in a dev loop or in CI rots, and the
configuration behind the outage looked entirely reasonable.
BREAKING for configuration, not API. SecretsOptions.SqlitePath now defaults to an
absolute per-user path under LocalApplicationData; it was "secrets.db", which the new
rules reject. An application configuring a relative path will fail to boot — remove the
key or set an absolute one outside the content root. All five deployed instances of this
family were checked first and already set an absolute path outside their content root,
so the migration edge (a host relying on the OLD default now finding nothing and
creating an empty store — this bug, re-introduced by its own fix) does not apply to any
of them.
Per-user rather than machine-wide is deliberate: CommonApplicationData is /usr/share on
Linux and macOS, which a normal user cannot write, so a machine-wide library default
would fail every consuming application's boot on a developer machine. An application may
compute its own default; the configured value always wins.
Validator registration is idempotent. AddZbSecrets is legitimately called more than once
in real compositions, and TryAddEnumerable cannot de-duplicate a factory-built
descriptor — it de-duplicates on implementation type, which a factory does not carry, so
it throws on the second call rather than skipping. The existing suite caught this.
Tests cover both rules including the segment-boundary case (/srv/app-data must not read
as inside /srv/app — a false positive on a fail-closed startup rule is an application
that will not boot), that the shipped default satisfies the rules it ships with, that
the default is not machine-wide, idempotent registration, and — most importantly — that
a bad path actually throws when IOptions is resolved, since a correctly implemented
validator that is never invoked is the same "guard that does not fire" failure this
change exists to prevent.
Family bumps to 0.6.0. Adds a dependency on ZB.MOM.WW.Configuration 0.1.0 for the shared
OptionsValidatorBase/ValidationBuilder vocabulary, which every consumer already
references.
The ScadaBridge central shared-store live gate proved SqlServerSecretsStoreMigrator's
"safe to run concurrently" claim false on a virgin database: Serializable isolation does
not serialize the IF NOT EXISTS metadata probes against another session's in-flight
CREATE, so of two central nodes booting together the loser died with SqlException 2714
("There is already an object named 'zbsecrets'") — the retry filter covered only
deadlock 1205. Losing such a race means the object now exists (the desired end state),
so 2714 (schema/table), 1913 (index) and 2627 (schema_version MERGE-insert PK) are now
retryable alongside 1205; the retry re-runs the idempotent IF NOT EXISTS path.
Classification scans the whole SqlException.Errors collection (2714 arrives stacked with
2759), the retry cap keeps genuine conflicts failing closed, and 2705 is deliberately
excluded (unreachable — columns live in a single CREATE TABLE statement). Predicate
extracted as an internal static pinned by offline tests; live suite gains an env-gated
4-concurrent-migrators virgin-database test. Six packages pushed to the Gitea feed at
0.5.1.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Bump ZB.MOM.WW.Secrets family 0.4.1 -> 0.5.0 and date the CHANGELOG
entry. All 6 packages (Abstractions, Secrets, Replicator.SqlServer,
Replicator.AkkaDotNet, Replicator.Grpc, Ui) packed and pushed to the
dohertj2-gitea feed; restore-verified from a scratch consumer with a
fresh package cache. Suite 470 passed / 0 failed / 15 skipped.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
SecretsGrpcHubClientOptions gains a getter-only FallbackEndpoints list:
additional hub addresses tried in order when a call fails with a
transport-level RpcException. Validation fails startup on blank/relative
entries and on any duplicate of the primary or another fallback (a
duplicated endpoint silently halves the configured failover).
New FailoverSecretsHubReader composes one GrpcSecretsHubClient per
endpoint: the answering endpoint is sticky (a sweep is GetManifest + N
GetSecrets batches — without stickiness every call after a primary
outage re-pays a full CallDeadline before reaching the healthy
fallback), rotation wraps so a recovered primary is retried when the
stuck-to endpoint fails, only RpcException triggers failover
(cancellation and defects propagate untouched), and when every endpoint
fails the last RpcException is rethrown so the sweep degrades to
last-known-good exactly as today. Failover warnings name only the two
endpoint URIs and the status code — never any credential material.
DI: zero fallbacks keeps the registration graph byte-identical to 0.4.x
(single keyed channel under ChannelServiceKey, plain
GrpcSecretsHubClient); with fallbacks, one container-owned keyed channel
per endpoint (fallback i under "{ChannelServiceKey}:fallback:{i}") wired
into the failover reader. GrpcSecretsHubClient itself is unchanged.
Safe only when every listed endpoint serves the SAME store — for
ScadaBridge, the central pair over one shared SQL Server store; the
options doc and CHANGELOG carry the warning.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
The ScadaBridge live gate (check 4) found SecretsHubAuthInterceptor denying
correctly but logging nothing: the only trace of a refusal was a framework
Grpc.AspNetCore.Server Information line. A follower whose bearer token was
mis-rotated therefore stopped converging silently, with the hub showing it
only at INF.
Every denial is now counted and summarised in a Warning emitted on the first
denial after startup or after a 60-second quiet window, carrying the denials
since the previous summary plus a breakdown by cause. A single probe is
visible immediately; a sustained flood is bounded to one line per window,
which is what makes warning safe at all — the endpoint is unauthenticated by
construction, so a per-denial warning would be an unbounded log write driven
by an unauthenticated caller. Counters are Interlocked and the emit is
decided by a compare-exchange on the last-emit timestamp, so exactly one
thread emits per window and a denial counted after the drain carries into
the next summary rather than being lost.
The cause breakdown is server-side diagnostics only. Denied() is untouched:
every rejection still returns the identical uniform Unauthenticated status,
the comparison is still constant-time, and no token material appears in any
log line. The existing warn-once-on-unconfigured-token signal is unchanged
and still emitted exactly once; denials in that state count toward the new
summary's "unconfigured" category.
TimeProvider is an optional default-valued parameter rather than a second
overload — the type must keep exactly one public constructor or the gRPC
interceptor activator silently stops invoking the auth gate.
Two existing tests scoped their assertions to the configuration warning,
since both kinds now share a category: the new line is deliberate behavior,
not drift.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Bumps the ZB.MOM.WW.Secrets family from 0.3.0 to 0.4.0 and adds the new
ZB.MOM.WW.Secrets.Replicator.Grpc package to the packable set. Publishes
all six 0.4.0 nupkgs to the dohertj2-gitea feed and restore-verifies the
new package resolves from it in a fresh --packages directory. Existing
packages are behaviorally unchanged — no breaking changes.
Adds ZB.MOM.WW.Secrets/CHANGELOG.md (none existed previously) to record
this and prior releases, and lists the new package in README.md's
Packages table. Also includes the controller's pending ledger updates
for docs/plans/2026-08-07-secrets-grpc-hub-implementation.md (Tasks
1-4 and 8 marked completed).
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Joseph Doherty