Owner chose option (b). ZB.MOM.WW.Secrets.Replicator.Grpc has had no consumer
since ScadaBridge retired the gRPC secrets hub (2026-09-28); OtOpcUa never used it.
- removed the project and its test project from ZB.MOM.WW.Secrets.slnx, so the
family build/test and build/push.sh's `dotnet pack` skip it;
- csproj IsPackable=false (a direct pack of the project yields nothing);
- SecretsGrpcHubServerExtensions / SecretsGrpcHubClientExtensions [Obsolete];
- README, CHANGELOG (0.6.6 Deprecated), CLAUDE.md and clustered-secrets.md say so.
Source + tests stay in the tree and still build/run standalone (test csproj
suppresses CS0618): 200/200. Feed versions 0.4.1-0.6.3 are left for the owner
to unlist.
Also zeroes the solution's pre-existing warnings: CA1416 on two Unix-only
permission tests (guarded by Skip.IfNot, which the analyzer cannot see — now
[UnsupportedOSPlatform("windows")]) and two xUnit2031 Assert.Single(Where).
Build 0 warnings; tests 429 passed / 21 skipped / 0 failed; pack = 5 nupkgs.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The CLI host takes its working directory as the content root, and the rig put the store under it.
The old lexical store-path rule missed that on macOS only because the temp path is spelled
/var/... while the working directory reports /private/var/...; with Secrets-005's link resolution
the rule (correctly) refused it. The rig now runs from a sibling work/ directory.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Secrets-019: the hub channels are created with ThrowOperationCanceledOnCancellation = true, so a
call cancelled through its token throws OperationCanceledException. Defensively, the failover
reader rethrows RpcException(Cancelled)-with-a-cancelled-token as cancellation instead of dialing
every fallback with the same cancelled token, and the sweep treats it as shutdown, not a failure
recorded in ISecretsHubSyncStatus. A server-side Cancelled status (token not cancelled) still fails
over.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Secrets-018: BundleService.ImportAsync now rehydrates every entry and re-wraps every foreign-KEK
row before the first write, so a malformed or tampered entry anywhere fails the import with the
store untouched ("Nothing was imported"). A store fault during the apply phase (no multi-row
transaction) now reports the row it stopped at and every row already written, which the
interactive flow prints, instead of a bare "Import failed".
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Secrets-016: RevealButton (RevealPolicy), SecretEditor and ConfirmDeleteModal (ManagePolicy) now
evaluate their policy through IAuthorizationService at the top of the revealing/mutating handler
and fail closed (no auth state, no IAuthorizationService, or a failed policy), instead of relying
on the host to wrap them in AuthorizeView. New tests build the REAL AddSecretsAuthorization
policies and evaluate each role against each policy, so secrets-manager satisfying RevealPolicy
would now fail a test.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Secrets-015: RevealButton now writes the secret.reveal Success audit before showing the value and
shows nothing ("audit record could not be written") if that write throws; the failure is logged as
a warning with actor and name, never the value. The cancellation token is captured up front, so a
component disposed mid-reveal can no longer turn the audit into a swallowed ObjectDisposedException.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Secrets-014: SecretEditor in add mode reads the name first. A live row is refused ("already
exists. Use Rotate") with no write and no audit, so a pasted name can no longer replace a live
credential and reset its description/content type under a "secret.add" audit. A tombstoned name is
restored only when the operator submits the same name a second time after the warning.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Secrets-011: HandleAnnounceTick pipes the manifest read back to the actor and publishes the
announce from the mailbox with Self as sender. Under a genuinely async store it used to publish
from a pool continuation with NoSender, so every peer replied to deadLetters; a faulted manifest
read was dropped silently and is now logged.
Secrets-012: the exchange's push rows, the pull reply and the pull request are split into
messages of at most 64 KB (measured with the wire encoding), half Akka.Remote's default 128000-byte
frame, so a cold node converges against a large store. A single over-budget row is still sent
alone; the manifest announce is not split (documented in clustered-secrets.md).
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Secrets-009: SecretReferenceExpander.ExpandAsync throws FormatException when a value holds a
${secret: marker that is not a complete reference with a non-blank name (${secret:}, an
unterminated ${secret:name, a nested marker), instead of passing the literal text through as the
credential. ExpandConfigurationAsync names the offending key (never the value) and writes nothing.
The README wiring sample drops the relative "SqlitePath": "secrets.db" (which failed validation)
for the per-user default and shows an absolute example; a test validates the sample.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The CLI's own appsettings.json set "SqlitePath": "secrets.db", which the store-path validator
rejects as relative, so the headless CLI failed at startup when run from its own directory unless
Secrets__SqlitePath overrode it. The key is removed: the per-user default
(<LocalApplicationData>/ZB.MOM.WW/secrets.db) applies, and an app's appsettings.json in the working
directory or the environment still override it. Regression test validates the shipped file.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Secrets-008: SqlServerSecretSyncService.SweepAsync now catches a per-row push failure, logs it
(EventId 4, name only) and continues, like the pull side. One row the hub refuses (a >450-char
name, an over-long explicit KEK id) no longer stops every later name from reaching the hub.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Secrets-006: DefaultSecretResolver keeps a per-key invalidation generation; Invalidate bumps it
before evicting, and a resolve that read the store under an older generation withdraws its insert,
so a rotation mid-resolve is not undone for another TTL. Expired entries are removed on lookup and
by a periodic TimeProvider sweep (every TTL, at least 1 s); a non-positive TTL caches nothing.
The resolver is now IDisposable (stops the sweep timer).
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Secrets-005: AddIfNotRooted now uses Path.IsPathFullyQualified, so Windows drive-relative
(C:secrets.db) and root-relative (\secrets.db) paths are rejected. AddIfUnderContentRoot refuses a
store if EITHER its spelled location (device prefix stripped, loopback admin share mapped to its
drive) OR its resolved location (symlinks/junctions followed, 8.3 names expanded on Windows) is
under the likewise-resolved content root. A store spelled through the content root stays refused
even if a link inside carries it elsewhere. UNC paths through other shares are compared as written.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Secrets-004: AesGcmEnvelopeCipher.Decrypt now wraps the ArgumentException AesGcm throws for a
wrong-length nonce/tag/wrapped DEK as SecretDecryptionException (as Rewrap already did), so the
resolver's decryption-failed audit fires.
Secrets-007: new MasterKeySnapshot (key + KekId from one reading, zeroed on Dispose) and a
default-implemented IMasterKeyProvider.GetSnapshot(); MasterKeyProviderBase overrides it to
resolve the source once and derive the id from that same key. Encrypt/Decrypt/Rewrap take one
snapshot per KEK and dispose it, so a key file swapped mid-operation can no longer stamp a row
with one key's id and wrap it under another. Intermediate key buffers are zeroed. On-disk format
unchanged: a golden row written by the 0.6.5 cipher still decrypts.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
`secret set`/`rotate` took the plaintext as a positional argument (shell
history, process list), and Program handed argv to
Host.CreateApplicationBuilder, whose command-line config provider crashed on
a value like "-p=Hunter2Secret" with the value in the unhandled
FormatException on stderr.
- SetCommandLine: `set|rotate <name> [-|--value-stdin]`; no value reads piped
stdin or prompts twice (masked) at a TTY; one trailing newline dropped;
empty refused. The positional <value> still works and warns on stderr.
- host built from HostApplicationBuilderSettings without argv; startup
failures print `secret: startup failed: <msg>` and exit 1
- unexpected trailing arguments reported by position, not echoed
- tests: SetCommandLineTests incl. two child-process runs of the built CLI;
the -p=Hunter2Secret probe fails (leaks) on the old Program.cs
- docs: README, clustered-secrets runbook, CHANGELOG
- code-reviews: Secrets-017 Resolved; README regenerated
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
SecretEditor and ConfirmDeleteModal wrote the success audit inside the same
try as the store write, so a throwing IAuditWriter after a committed
add/rotate/delete showed an error, wrote a Failure audit and skipped the
resolver-cache eviction - the old plaintext kept resolving for the TTL.
The mutation try now covers only the cipher + store call; the success audit
is best-effort afterwards and logs a warning (no value) on failure, so the
change is reported saved, the callback fires and the cache is evicted.
- tests: AuditFailureAfterCommitTests (add, rotate, delete) with a throwing
success audit; all fail without the fix
- code-reviews: Secrets-013 Resolved; README regenerated
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The replication actor was created only when something resolved ISecretStore
(store -> replicator -> SecretReplicationActorProvider.ActorRef). A node that
expanded its ${secret:} references pre-host and never touched the store at
runtime had no actor: no topic subscription, no announces, no pull replies.
AddZbSecretsAkkaReplication now registers SecretReplicationActorStarter, which
creates the actor on IHostApplicationLifetime.ApplicationStarted (after every
hosted service, incl. an Akka.Hosting AkkaHostedService registered later), or
in StartAsync when no lifetime is registered.
- tests: EagerActorStartTests (generic host in both registration orders with a
no-actor-before-start control; bare container) - all fail without the fix
- docs: README, CHANGELOG
- code-reviews: Secrets-010 Resolved; README regenerated
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
ExpandConfigurationAsync wrote through config[key] = ..., which only sets the
providers' in-memory Data; a file provider's reload replaces it, so editing
appsettings.json on a running service reverted every expanded key to its
${secret:...} placeholder.
- ConfigurationManager: expanded values are pinned in an internal overlay
source appended last; its provider's Load() is a no-op and Build always
returns the same provider, so reloads (and source rebuilds) keep the pins
- any other IConfigurationRoot: in-place write as before, re-applied from the
root's reload token
- all references resolve before anything is written (a fail-closed miss no
longer leaves the config half expanded)
- tests: SecretExpansionReloadTests (real JSON file + Reload(), options
monitor, plain root); three fail without the fix
- docs: README reload note, CHANGELOG
- code-reviews: Secrets-003 Resolved; README regenerated
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The SQL-Server extensions called the 3-arg AddZbSecrets, and the store-path
validator captured the content root of whichever call registered it first.
In hub mode no registration order applied the pre-host under-content-root
guard: extension-first dropped the app's 4-arg root, core-first lost the
replicator (#30). That is the 0.6.0 store-path hole, reopened.
- AddZbSecretsSqlServerStore / AddZbSecretsSqlServerReplication /
AddZbSecretsAkkaReplication gain a trailing optional contentRootPath,
forwarded to the 4-arg AddZbSecrets
- the root now lives in one SecretsContentRoot instance holder per service
collection; the last explicit root wins, null never erases one; the
IHostEnvironment fallback is unchanged
- tests: ContentRootTests (hub mode, three compositions, artifact asserted
before the throw, positive control), last-explicit-wins theory in
SecretsStorePathRulesTests, Akka forwarding test
- code-reviews: Secrets-002 Resolved; README regenerated
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
AddZbSecretsSqlServerStore / AddZbSecretsSqlServerReplication (and the Akka
extension, same defect class) TryAdd-ed their ISecretStore, ISecretsStoreMigrator
and ISecretReplicator ahead of their internal AddZbSecrets call. An application
that had already called AddZbSecrets therefore silently kept the SQLite store
(shared-store mode) or NoOpSecretReplicator (hub/Akka modes).
They now call AddZbSecrets first and RemoveAll + AddSingleton their own
registrations; a later AddZbSecrets only TryAdds, so either order yields the
same graph with exactly one ISecretStore registration.
- tests: RegistrationOrderTests (SqlServer + Akka) build and resolve the real
container in both orders; core-first fails without the fix
- Directory.Build.props 0.6.5 -> 0.6.6 (NOT packed or published)
- code-reviews: Secrets-001 Resolved; README regenerated
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
The two Unix-permission tests called Skip.IfNot from a plain [Fact] (xunit v2 +
Xunit.SkippableFact) so they failed on Windows; now [SkippableFact].
Resolver_Works_EndToEnd_ThroughDi calls SqliteConnection.ClearAllPools() before
deleting its temp DB.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Several setups migrating one SQLite secret store at once (mxaccessgw's parallel test hosts on the
per-process store) failed intermittently on Windows with "cannot start a transaction within a
transaction" from SqliteSecretsStoreMigrator.MigrateAsync. That message means the BEGIN ran on a
handle that was already in a transaction: the pinned Microsoft.Data.Sqlite 10.0.7 can hand one
pooled handle to two owners (dotnet/efcore#39009, fixed in 10.0.13) and no 10.0.x refuses to
re-pool a handle whose rollback failed (dotnet/efcore#39082).
The migrator now serialises migrations of one store per process behind a gate keyed on the
normalised path, runs each attempt on a dedicated unpooled connection inside BEGIN IMMEDIATE, and
retries SQLITE_BUSY/SQLITE_LOCKED (another process past the busy timeout) with equal-jitter
exponential backoff. Store operations and the Microsoft.Data.Sqlite pin are unchanged.
Tests: concurrent-init (16 hosts x 20 rounds on fresh and existing stores, two path spellings,
16 AddZbSecrets containers running the pre-host migrator + hosted service), gate sharing/waiting,
BUSY retry past the 5 s busy timeout, transient-code classification.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Flat 100*attempt backoff let concurrent boots retry in lockstep (Auth's twin loop
exhausted retries ~1/60 eight-way boots). Backoff is now exponential with equal
jitter, via an injectable Random for deterministic tests.
Refs scadaproj#18
GrpcSecretSyncService's background sweep now records outcomes on a new
SecretsHubSyncStatus (registered singleton, exposed as public
ISecretsHubSyncStatus) and logs three new events on the outcome-recording
path: EventId 4 (Debug, every in-sync sweep), EventId 5 (Information, the
same outcome throttled to at most once per the new
SecretsGrpcHubClientOptions.InSyncLogInterval, default 1h), and EventId 6
(Information, the first sweep to succeed after one or more failures).
EventIds 1-3 are unchanged; the raw internal SweepAsync still throws on
failure exactly as before. Not yet published to the Gitea feed and
ScadaBridge is not re-pinned — that is a separate, later step.
Suggested by the OtOpcUa session, the only real containerized deployment in this family
and the one that hit the false positive. A store on a volume mounted inside the content
root is durable across container replacement but is rejected anyway, because the rule
cannot tell a mount from an ordinary directory. The message now says so, gives the fix,
and forecloses the wrong one by stating that no override exists.
A false positive's entire cost is the confusion in the moment, so an error that names its
own fix costs nothing on the true-positive path and turns the false positive into a
two-minute edit. Message only, no behaviour change.
Their case for keeping the rule strict is recorded in the CHANGELOG because it is better
than the trade-off I had written: the workaround is an improvement rather than a
concession, an override flag has the lifecycle of a knob set once and never revisited,
and mount detection would make the verdict depend on runtime platform state so the same
path would pass on one host and fail on another.
Raised by the mxaccessgw session, which went back to its own tests after I reported a
vacuous assertion in mine and found the same class one step further from the fuse.
The rejecting test asserts no store file is left at a rejected path. That claim is only
meaningful if something proves the write path is live at all: "no file was created" is
indistinguishable from "nothing in this container ever creates files". The positive
control was supposed to establish that and did not -- it asserted Assert.NotNull on a
resolved SecretsSqliteConnectionFactory, which is close to a tautology once no exception
escaped, and it never ran the migration at all. So if the pre-host container ever stopped
touching the store, the rejecting test would have gone quietly vacuous while the positive
control kept passing. Neither would have failed.
The positive control now runs SqliteSecretsStoreMigrator.MigrateAsync and asserts the
database IS created. The pair is mutually checking: the migration writes the file when
allowed and does not when rejected, and neither assertion stands on its own.
Verified the same way as the last one, by simulating the failure rather than reasoning
about it: replacing the migration with a no-op makes the positive control fail with "the
accepted path must actually get a store file". Reverted after measuring.
No library change; 0.6.1 as published is unaffected. Suite 502 passing.
Two defects in the 0.6.1 regression test, the first raised by the mxaccessgw session
from its own adoption, the second exposed by acting on it.
Assertion order. The test asserted the throw before the artifact. That hides the
stronger finding: a regression that stops the throw but still creates the store fails
with "no exception was thrown" and never mentions the database sitting at the rejected
path -- which IS the defect. Now captures with Record.ExceptionAsync, asserts no file
created first, and asserts the exception type last. Notable because the defect under
test was itself "a boot outcome was observed instead of the artifact being probed", so
closing it with an outcome-first assertion would have repeated the mistake.
Vacuous assertions. Simulating the regression -- replacing the 4-argument call with the
3-argument one, which still compiles, exactly as the call-site comments warn -- showed
the test failing on the exception assertion rather than the file assertion. No file had
been created in EITHER case: the test resolved SecretsSqliteConnectionFactory, whose
constructor never touches the filesystem. The File.Exists assertions could not fail. The
test now resolves SqliteSecretsStoreMigrator and runs MigrateAsync, which is what a
pre-host expander actually does and what actually creates the database.
Verified by simulating the regression again: the test now fails with "a rejected path
must not be left holding a store file" rather than "no exception was thrown". Reverted
after measuring.
No library change; 0.6.1 as published is unaffected. Suite 502 passing.
Found by the HistorianGateway adoption of 0.6.0, which probed the rejected paths
instead of observing a clean boot — a clean boot being equally consistent with the
validator not being wired at all.
An application that expands ${secret:} references before building its host composes
this subsystem in a throwaway ServiceCollection with no IHostEnvironment in it, then
migrates the store from that container. SecretsOptionsValidator resolves the content
root from IHostEnvironment and skips the under-content-root rule when it is absent, so
pre-host an absolute in-content-root path passed the only rule still applying, the
migrator CREATED the database, and the boot failed a second later once the real host
registered IHostEnvironment and the rule finally fired.
The observable result was a rejected boot that still left an empty store, with -wal and
-shm siblings, at the rejected path — the same artifact that made the MxGateway outage
read as "the database is there, it's just empty", which is precisely what this guard
exists to prevent. A relative path was unaffected because the absolute-path rule fires
with or without a content root; that asymmetry is what exposed it.
The mistake was conflating "no IHostEnvironment registered" with "no content root
exists". New AddZbSecrets overload takes the content root explicitly; the 3-argument
overload is unchanged and still falls back to IHostEnvironment, so this is additive.
SecretsOptionsValidator gains a string? constructor alongside the IHostEnvironment? one.
Reproduced first in an isolated harness before changing anything: a bare host rejects
both bad paths leaving no artifacts, and a bare ServiceCollection with no
IHostEnvironment migrates an in-content-root path to disk. Both are now regression tests,
the second paired with a positive control asserting the old behaviour still holds when no
content root is supplied — so the test is about the explicit content root rather than
about paths in general.
Also documents an upgrade note that applies to 0.6.0: a store seeded under the old
relative default is orphaned rather than migrated, and nothing signals it.
Secrets:SqlitePath is now validated at startup by SecretsOptionsValidator using two
rules in the new public SecretsStorePathRules. Both come from production incidents and
neither implies the other.
Must be absolute: a relative value resolves against the process working directory,
which for a Windows service is C:\Windows\System32, so the store lands outside the
installation; in a dev loop it lands in the source tree (a gateway was observed writing
its secrets database into src/, tripping that repo's own tree-hygiene test).
Must not be inside the content root: an absolute path can still point into the
directory every deployment procedure in this family renames on upgrade. On 2026-08-09 a
gateway's credential store sat inside its Server\ directory; the deploy renamed Server
to Server.bak.*, the database went with it, and the process created a fresh empty one
at the same path. Every consumer lost its credential, silently, for two days. The
existing "must be absolute" convention did not help — the path was absolute.
The rules reject rather than relocate. Silently moving a credential store presents later
as an authentication failure pointing nowhere near the configuration that caused it; a
boot error is seen immediately by whoever caused it. Validation is deliberately not
exempted in Development: a guard that never runs in a dev loop or in CI rots, and the
configuration behind the outage looked entirely reasonable.
BREAKING for configuration, not API. SecretsOptions.SqlitePath now defaults to an
absolute per-user path under LocalApplicationData; it was "secrets.db", which the new
rules reject. An application configuring a relative path will fail to boot — remove the
key or set an absolute one outside the content root. All five deployed instances of this
family were checked first and already set an absolute path outside their content root,
so the migration edge (a host relying on the OLD default now finding nothing and
creating an empty store — this bug, re-introduced by its own fix) does not apply to any
of them.
Per-user rather than machine-wide is deliberate: CommonApplicationData is /usr/share on
Linux and macOS, which a normal user cannot write, so a machine-wide library default
would fail every consuming application's boot on a developer machine. An application may
compute its own default; the configured value always wins.
Validator registration is idempotent. AddZbSecrets is legitimately called more than once
in real compositions, and TryAddEnumerable cannot de-duplicate a factory-built
descriptor — it de-duplicates on implementation type, which a factory does not carry, so
it throws on the second call rather than skipping. The existing suite caught this.
Tests cover both rules including the segment-boundary case (/srv/app-data must not read
as inside /srv/app — a false positive on a fail-closed startup rule is an application
that will not boot), that the shipped default satisfies the rules it ships with, that
the default is not machine-wide, idempotent registration, and — most importantly — that
a bad path actually throws when IOptions is resolved, since a correctly implemented
validator that is never invoked is the same "guard that does not fire" failure this
change exists to prevent.
Family bumps to 0.6.0. Adds a dependency on ZB.MOM.WW.Configuration 0.1.0 for the shared
OptionsValidatorBase/ValidationBuilder vocabulary, which every consumer already
references.
The ScadaBridge central shared-store live gate proved SqlServerSecretsStoreMigrator's
"safe to run concurrently" claim false on a virgin database: Serializable isolation does
not serialize the IF NOT EXISTS metadata probes against another session's in-flight
CREATE, so of two central nodes booting together the loser died with SqlException 2714
("There is already an object named 'zbsecrets'") — the retry filter covered only
deadlock 1205. Losing such a race means the object now exists (the desired end state),
so 2714 (schema/table), 1913 (index) and 2627 (schema_version MERGE-insert PK) are now
retryable alongside 1205; the retry re-runs the idempotent IF NOT EXISTS path.
Classification scans the whole SqlException.Errors collection (2714 arrives stacked with
2759), the retry cap keeps genuine conflicts failing closed, and 2705 is deliberately
excluded (unreachable — columns live in a single CREATE TABLE statement). Predicate
extracted as an internal static pinned by offline tests; live suite gains an env-gated
4-concurrent-migrators virgin-database test. Six packages pushed to the Gitea feed at
0.5.1.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Review follow-ups: the warning now carries only the two endpoint URIs and the
status code — attaching the exception object would let any real sink render
its ToString(), which for RpcException includes server-authored Status.Detail.
The leak pin test now also asserts no exception rides on the warning, via a
new WarningExceptions capture on RecordingLogger. Plus two cosmetic nits:
unified ThrowIfNull idiom, documented that FallbackEndpoints mutation after
Validate is unsupported.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
SecretsGrpcHubClientOptions gains a getter-only FallbackEndpoints list:
additional hub addresses tried in order when a call fails with a
transport-level RpcException. Validation fails startup on blank/relative
entries and on any duplicate of the primary or another fallback (a
duplicated endpoint silently halves the configured failover).
New FailoverSecretsHubReader composes one GrpcSecretsHubClient per
endpoint: the answering endpoint is sticky (a sweep is GetManifest + N
GetSecrets batches — without stickiness every call after a primary
outage re-pays a full CallDeadline before reaching the healthy
fallback), rotation wraps so a recovered primary is retried when the
stuck-to endpoint fails, only RpcException triggers failover
(cancellation and defects propagate untouched), and when every endpoint
fails the last RpcException is rethrown so the sweep degrades to
last-known-good exactly as today. Failover warnings name only the two
endpoint URIs and the status code — never any credential material.
DI: zero fallbacks keeps the registration graph byte-identical to 0.4.x
(single keyed channel under ChannelServiceKey, plain
GrpcSecretsHubClient); with fallbacks, one container-owned keyed channel
per endpoint (fallback i under "{ChannelServiceKey}:fallback:{i}") wired
into the failover reader. GrpcSecretsHubClient itself is unchanged.
Safe only when every listed endpoint serves the SAME store — for
ScadaBridge, the central pair over one shared SQL Server store; the
options doc and CHANGELOG carry the warning.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
The ScadaBridge live gate (check 4) found SecretsHubAuthInterceptor denying
correctly but logging nothing: the only trace of a refusal was a framework
Grpc.AspNetCore.Server Information line. A follower whose bearer token was
mis-rotated therefore stopped converging silently, with the hub showing it
only at INF.
Every denial is now counted and summarised in a Warning emitted on the first
denial after startup or after a 60-second quiet window, carrying the denials
since the previous summary plus a breakdown by cause. A single probe is
visible immediately; a sustained flood is bounded to one line per window,
which is what makes warning safe at all — the endpoint is unauthenticated by
construction, so a per-denial warning would be an unbounded log write driven
by an unauthenticated caller. Counters are Interlocked and the emit is
decided by a compare-exchange on the last-emit timestamp, so exactly one
thread emits per window and a denial counted after the drain carries into
the next summary rather than being lost.
The cause breakdown is server-side diagnostics only. Denied() is untouched:
every rejection still returns the identical uniform Unauthenticated status,
the comparison is still constant-time, and no token material appears in any
log line. The existing warn-once-on-unconfigured-token signal is unchanged
and still emitted exactly once; denials in that state count toward the new
summary's "unconfigured" category.
TimeProvider is an optional default-valued parameter rather than a second
overload — the type must keep exactly one public constructor or the gRPC
interceptor activator silently stops invoking the auth gate.
Two existing tests scoped their assertions to the configuration warning,
since both kinds now share a category: the new line is deliberate behavior,
not drift.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
An internal validateEagerly seam on AddZbSecretsGrpcHub replaces the harness's
hand-copied registration fragment, so the deny-all second-layer tests exercise
the production wiring and cannot drift from it.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Adds the end-to-end layer the unit suites structurally cannot reach: a real
in-process ASP.NET Core host serving SecretsHubService through
AddZbSecretsGrpcHub/MapZbSecretsHub, a follower built by the shipped
AddZbSecretsGrpcHubClient, a real gRPC call between them, and real SQLite
stores on both sides. The class of defect this catches is pieces that are each
individually correct and are never connected to one another — for instance the
interceptor's warn-once, which only holds if the container really hands gRPC
the singleton it registered.
Covered: byte-identical convergence over the wire (every ciphertext/nonce/tag/
wrapped-DEK byte, kek_id, revision, both timestamps INCLUDING their offsets,
across a tombstone, a nulls-everywhere row and an empty-string-description row);
delete propagation; a hub unreachable at follower boot leaving the last-known-
good store intact and converging once it answers; the auth negatives fail-closed
across the wire with one uniform denial detail; the per-request name cap as
ResourceExhausted; and an RpcException from the call invoker escaping the hub
client unchanged, so a transport fault stays distinguishable from a parse fault.
The credential-hygiene assertions are made against logs captured at Trace on
BOTH hosts, so "no token material ever reaches a sink" is asserted at the most
verbose setting a deployment could switch on.
SecretsHubContractPinTests reads the generated descriptor — not the .proto text
— and pins the RPC set to exactly {GetManifest, GetSecrets}, plus unary-only and
the package-qualified service name. The hub is pull-only by construction:
updates originate at central only, and the absence of a write RPC is what
enforces that, not authorization a mis-wired follower may hold more of than
intended. A third RPC silently retires the guarantee, so the failure message
says so.
Microsoft.AspNetCore.TestHost pinned at 10.0.7 — the version ZB.MOM.WW.LocalDb,
OtOpcUa and ScadaBridge already pin, matching the Microsoft.Extensions.* line.
Full suite: 436 passed / 0 failed / 15 skipped, no new warnings.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
A single malformed row or manifest entry poisoned the whole sweep, permanently.
WireToDomain.ToRows/ToManifest materialized eagerly, so one unparseable element
threw out of the fetch callback, past SecretReplicationReconciler's per-batch
handling, into SweepSafelyAsync's whole-sweep catch. The follower's pull set is
derived from the hub's manifest, so the identical bad element arrived in the
identical batch on every interval and nothing ever converged — logged as "sync
failed", indistinguishable from a hub outage.
Isolation now matches the Akka sibling's split exactly: WireToDomain stays a
strict per-item parser and loses its two batch entry points (their absence is
pinned by a reflection test), while GrpcSecretsHubClient converts each item
inside its own try — the same shape as SecretReplicationActor.Materialize.
A dropped item costs one warning naming the secret and a reason category, and
every well-formed sibling still converges. Only parse-shaped exceptions are
caught (ArgumentException, FormatException); a defect still propagates. A row
whose own name is unparseable is logged under a marker rather than echoing the
untrusted string, and no ciphertext, DEK or credential reaches a log line.
GrpcSecretsHubClient had zero coverage — the existing fake implements the
ISecretsHubReader seam directly, bypassing the header, the deadline and the
conversion, which are the only things that class does. It is now driven for
real through a fake CallInvoker handed to the generated client.
The stale doc comment claiming the reconciler isolated the failure is replaced
by one describing the mechanism that actually exists, and Endpoint gains a note
that a plain-http hub sends the bearer token cleartext (doc only, no validation
change).
Tests 52 -> 67 in the package suite; solution green, no new warnings.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Five review items on the Task 2 server half.
1. GetSecrets now enforces a server-side cap on request.Names, refusing an
over-cap batch with ResourceExhausted BEFORE any store read runs. The
well-behaved follower chunks at 500, but client discipline is not a server
defence — the family precedent (HistorianGateway's max_values) is that the
server declines regardless. The new SecretsGrpcHubServerOptions.
MaxNamesPerRequest defaults to 1000, comfortably above that chunk so a
compliant sweep never meets it, and Validate() refuses a non-positive value
at startup rather than letting it read at runtime as a broken follower. The
refusal states the cap and the received count — neither is secret, and an
operator cannot resize a batch without them — and never a name. The cap sits
behind the interceptor, so it is defence in depth against a misbehaving
token holder, not an anonymous-DoS control.
2. AddZbSecretsGrpcHub now refuses a second registration. The two calls do not
compose: options binding MERGES (a key absent from the second section leaves
the first section's value standing) while the SectionPath stamp is
last-one-wins, so a hub registered twice could serve a token bound from one
section while every failure message named the other. Detected via a private
marker registration. The extension also gains real DI coverage — the intended
single-call path asserted for token/SectionPath coherence through a built
provider, a custom section path stamped end to end, a missing token throwing
from the registration call itself, and the duplicate call throwing.
3. Client-streaming and duplex-streaming denial tests, so all four gated handler
shapes are pinned rather than only unary and server-streaming, plus a
valid-token test that reaches all three streaming continuations.
4. A cancellation test: GetSecrets hands the ServerCallContext token through to
the store, pinned against a fake that records it.
5. The proto's service comment stated the ISecretReplicationHub mapping in prose
that no longer matched the RPC name; it now says GetManifestAsync ->
GetManifest and GetManyAsync -> GetSecrets explicitly. Comment-only — no
message or rpc definition is touched. The GetSecrets rpc comment also notes
the new RESOURCE_EXHAUSTED cap, which a client author has to know about.
Grpc server suite 50 -> 73 tests; full solution suite green, 0 new warnings.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Client half of the central-hosted secrets hub: a site node follows central and
never pushes.
- `ISecretsHubReader` + `GrpcSecretsHubClient` — the read half of the hub as a
follower sees it (manifest + bulk fetch, nothing else). Bearer credential and a
per-call deadline are attached per call; responses convert through
`WireToDomain`. Faults are deliberately not caught here, so an unreachable hub
can never be mistaken for an empty one.
- `WireToDomain` — strict, per-row proto -> domain parse mirroring the Akka
replicator's `SecretRowDto`: a name the domain would reject, a content type
naming no enum member, or a non-round-trip timestamp is refused at the boundary.
Explicit proto3 presence keeps null distinguishable from empty.
- `GrpcSecretSyncService` — `SqlServerSecretSyncService` with the push half
removed. One `ReconcileAsync` per tick (pull-only by the reconciler's own
contract, and batched, so a cold-start pull set cannot exceed gRPC's 4 MB cap),
never throws out of `ExecuteAsync`, warns once and retries on the next interval.
The local store is left completely undecorated — the sweep is its only hub-ward
writer.
- `SecretsGrpcHubClientOptions` — fail-closed `Validate()` naming the exact
configuration key, including the section a caller actually bound. The bearer
token is never echoed by any message.
- `AddZbSecretsGrpcHubClient` — binds, eagerly validates, and registers the
channel under a service KEY so it cannot lose a `TryAdd` race to an
application's own `GrpcChannel` and dial the wrong endpoint with the hub
credential attached.
Tests 52/52 (red-first), including a source-scan guard pinning that no client file
names `ComputePushSet`, `ReadLocalAsync`, or `ApplyReplicatedAsync`, and a
reflection pin that the reader seam exposes exactly two read operations.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
The server half of the central-hosted pull-only secrets hub. SecretsHubService
adapts the node's own ISecretStore onto secrets_hub.v1 using exactly two read
members (GetManifestAsync, GetAsync) — the proto exposes no write RPC, so a
follower has no path to put a row into central whatever its credentials allow.
GetSecrets omits a name the hub does not hold and skips one that will not parse
as a SecretName; neither is an error, because both describe a follower acting on
a manifest that was true a moment ago. Failing the call would stall the sweep and
let one bad name deny the follower every other row in the batch. Names are
re-validated server-side (the wire type is a plain string) and de-duplicated on
the normalized value.
DomainToWire mirrors the Akka replicator's SecretRowDto discipline exactly:
round-trip "O" timestamps so a DateTimeOffset's offset and ticks survive the
last-writer-wins equality compare, the content type as its enum MEMBER NAME so an
unknown value is rejected rather than decoded to the zero member, and nullable
fields left absent rather than written as "" so null and empty stay distinct.
SecretsHubAuthInterceptor fails closed: with no token configured every call is
denied and one warning names the exact configuration key — never a token value,
presented or configured, at any level. Denial is uniform (one Status, one detail,
no "why") so a caller gets no oracle. Comparison is constant-time over UTF-8
bytes. All four handler shapes are gated, not just unary, so a future RPC cannot
arrive unauthenticated by omission; more than one Authorization header is refused
as a smuggling shape. The type keeps EXACTLY ONE public constructor — Grpc's
activator silently stops invoking an interceptor it cannot construct
unambiguously, and the failure mode is an auth gate that quietly does not run — so
a test pins the count.
Tests: 50, red-first (compile-red, then a fail-closed mutation verified 5 of them
bite). Full Secrets suite 328 passed / 0 failed / 15 live-gated skips, 0 new
warnings.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Environment.SetEnvironmentVariable/GetEnvironmentVariable mutate the
process-wide environ; concurrent setenv/getenv across parallel xunit test
classes is a documented crash/flake source on glibc even with unique
variable names. Group the six classes that touch process env vars into a
DisableParallelization=true collection so they run serially against each
other while everything else keeps running in parallel.
Joseph Doherty