SecretsGrpcHubClientOptions gains a getter-only FallbackEndpoints list:
additional hub addresses tried in order when a call fails with a
transport-level RpcException. Validation fails startup on blank/relative
entries and on any duplicate of the primary or another fallback (a
duplicated endpoint silently halves the configured failover).
New FailoverSecretsHubReader composes one GrpcSecretsHubClient per
endpoint: the answering endpoint is sticky (a sweep is GetManifest + N
GetSecrets batches — without stickiness every call after a primary
outage re-pays a full CallDeadline before reaching the healthy
fallback), rotation wraps so a recovered primary is retried when the
stuck-to endpoint fails, only RpcException triggers failover
(cancellation and defects propagate untouched), and when every endpoint
fails the last RpcException is rethrown so the sweep degrades to
last-known-good exactly as today. Failover warnings name only the two
endpoint URIs and the status code — never any credential material.
DI: zero fallbacks keeps the registration graph byte-identical to 0.4.x
(single keyed channel under ChannelServiceKey, plain
GrpcSecretsHubClient); with fallbacks, one container-owned keyed channel
per endpoint (fallback i under "{ChannelServiceKey}:fallback:{i}") wired
into the failover reader. GrpcSecretsHubClient itself is unchanged.
Safe only when every listed endpoint serves the SAME store — for
ScadaBridge, the central pair over one shared SQL Server store; the
options doc and CHANGELOG carry the warning.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Task 9 (closeout) of the Secrets gRPC-hub program. Bookkeeping only; no code.
- clustered-secrets.md, Production enablement: the gRPC hub is no longer
pending. It is BUILT, SHIPPED (0.4.1) and LIVE-GATED PASS 4/4 (ScadaBridge
main @ f6c3f7c5, docs/plans/2026-08-07-secrets-grpc-hub-live-gate.md), so the
"until that ships, ScadaBridge replication stays off" line is replaced with
the truth: the code path is merged and gated, and production enablement is a
separate operator action behind a still-default-off flag. OtOpcUa's
pair-local topology is recorded as live-gated (lmxopcua master @ 1bd79c8e)
rather than pending re-gate. Rotation fan-out corrected: on ScadaBridge it is
BOTH central nodes plus both nodes of every site pair, because each central
node has its own independent store.
- components/secrets/GAPS.md: new "gRPC hub delivered" section recording the
0.4.0/0.4.1 publish, both gates with SHAs and gate-doc paths, and the
residual. Existing narrative left intact and amended, not rewritten.
- CLAUDE.md Secrets row: lib version 0.3.0 -> 0.4.1 and the app-pin claim
corrected (ScadaBridge pins 0.4.1 for the Grpc replicator; the other three
still pin 0.2.3); the "production topologies DECIDED" region now records
delivery, both gates, default-off, and the residual.
- Plan ledger: task 9 completed.
Residual carried forward as scadaproj#4 — ScadaBridge's central pair does not
converge with itself (central-b served an authenticated but EMPTY manifest all
gate while central-a held every secret) and the site client dials a single
endpoint; these are one design question, since failover to an empty hub would
succeed and silently stop convergence. scadaproj#3 is closed.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
The ScadaBridge live gate (check 4) found SecretsHubAuthInterceptor denying
correctly but logging nothing: the only trace of a refusal was a framework
Grpc.AspNetCore.Server Information line. A follower whose bearer token was
mis-rotated therefore stopped converging silently, with the hub showing it
only at INF.
Every denial is now counted and summarised in a Warning emitted on the first
denial after startup or after a 60-second quiet window, carrying the denials
since the previous summary plus a breakdown by cause. A single probe is
visible immediately; a sustained flood is bounded to one line per window,
which is what makes warning safe at all — the endpoint is unauthenticated by
construction, so a per-denial warning would be an unbounded log write driven
by an unauthenticated caller. Counters are Interlocked and the emit is
decided by a compare-exchange on the last-emit timestamp, so exactly one
thread emits per window and a denial counted after the drain carries into
the next summary rather than being lost.
The cause breakdown is server-side diagnostics only. Denied() is untouched:
every rejection still returns the identical uniform Unauthenticated status,
the comparison is still constant-time, and no token material appears in any
log line. The existing warn-once-on-unconfigured-token signal is unchanged
and still emitted exactly once; denials in that state count toward the new
summary's "unconfigured" category.
TimeProvider is an optional default-valued parameter rather than a second
overload — the type must keep exactly one public constructor or the gRPC
interceptor activator silently stops invoking the auth gate.
Two existing tests scoped their assertions to the configuration warning,
since both kinds now share a category: the new line is deliberate behavior,
not drift.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Bumps the ZB.MOM.WW.Secrets family from 0.3.0 to 0.4.0 and adds the new
ZB.MOM.WW.Secrets.Replicator.Grpc package to the packable set. Publishes
all six 0.4.0 nupkgs to the dohertj2-gitea feed and restore-verifies the
new package resolves from it in a fresh --packages directory. Existing
packages are behaviorally unchanged — no breaking changes.
Adds ZB.MOM.WW.Secrets/CHANGELOG.md (none existed previously) to record
this and prior releases, and lists the new package in README.md's
Packages table. Also includes the controller's pending ledger updates
for docs/plans/2026-08-07-secrets-grpc-hub-implementation.md (Tasks
1-4 and 8 marked completed).
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
An internal validateEagerly seam on AddZbSecretsGrpcHub replaces the harness's
hand-copied registration fragment, so the deny-all second-layer tests exercise
the production wiring and cannot drift from it.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Adds the end-to-end layer the unit suites structurally cannot reach: a real
in-process ASP.NET Core host serving SecretsHubService through
AddZbSecretsGrpcHub/MapZbSecretsHub, a follower built by the shipped
AddZbSecretsGrpcHubClient, a real gRPC call between them, and real SQLite
stores on both sides. The class of defect this catches is pieces that are each
individually correct and are never connected to one another — for instance the
interceptor's warn-once, which only holds if the container really hands gRPC
the singleton it registered.
Covered: byte-identical convergence over the wire (every ciphertext/nonce/tag/
wrapped-DEK byte, kek_id, revision, both timestamps INCLUDING their offsets,
across a tombstone, a nulls-everywhere row and an empty-string-description row);
delete propagation; a hub unreachable at follower boot leaving the last-known-
good store intact and converging once it answers; the auth negatives fail-closed
across the wire with one uniform denial detail; the per-request name cap as
ResourceExhausted; and an RpcException from the call invoker escaping the hub
client unchanged, so a transport fault stays distinguishable from a parse fault.
The credential-hygiene assertions are made against logs captured at Trace on
BOTH hosts, so "no token material ever reaches a sink" is asserted at the most
verbose setting a deployment could switch on.
SecretsHubContractPinTests reads the generated descriptor — not the .proto text
— and pins the RPC set to exactly {GetManifest, GetSecrets}, plus unary-only and
the package-qualified service name. The hub is pull-only by construction:
updates originate at central only, and the absence of a write RPC is what
enforces that, not authorization a mis-wired follower may hold more of than
intended. A third RPC silently retires the guarantee, so the failure message
says so.
Microsoft.AspNetCore.TestHost pinned at 10.0.7 — the version ZB.MOM.WW.LocalDb,
OtOpcUa and ScadaBridge already pin, matching the Microsoft.Extensions.* line.
Full suite: 436 passed / 0 failed / 15 skipped, no new warnings.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
A single malformed row or manifest entry poisoned the whole sweep, permanently.
WireToDomain.ToRows/ToManifest materialized eagerly, so one unparseable element
threw out of the fetch callback, past SecretReplicationReconciler's per-batch
handling, into SweepSafelyAsync's whole-sweep catch. The follower's pull set is
derived from the hub's manifest, so the identical bad element arrived in the
identical batch on every interval and nothing ever converged — logged as "sync
failed", indistinguishable from a hub outage.
Isolation now matches the Akka sibling's split exactly: WireToDomain stays a
strict per-item parser and loses its two batch entry points (their absence is
pinned by a reflection test), while GrpcSecretsHubClient converts each item
inside its own try — the same shape as SecretReplicationActor.Materialize.
A dropped item costs one warning naming the secret and a reason category, and
every well-formed sibling still converges. Only parse-shaped exceptions are
caught (ArgumentException, FormatException); a defect still propagates. A row
whose own name is unparseable is logged under a marker rather than echoing the
untrusted string, and no ciphertext, DEK or credential reaches a log line.
GrpcSecretsHubClient had zero coverage — the existing fake implements the
ISecretsHubReader seam directly, bypassing the header, the deadline and the
conversion, which are the only things that class does. It is now driven for
real through a fake CallInvoker handed to the generated client.
The stale doc comment claiming the reconciler isolated the failure is replaced
by one describing the mechanism that actually exists, and Endpoint gains a note
that a plain-http hub sends the bearer token cleartext (doc only, no validation
change).
Tests 52 -> 67 in the package suite; solution green, no new warnings.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Five review items on the Task 2 server half.
1. GetSecrets now enforces a server-side cap on request.Names, refusing an
over-cap batch with ResourceExhausted BEFORE any store read runs. The
well-behaved follower chunks at 500, but client discipline is not a server
defence — the family precedent (HistorianGateway's max_values) is that the
server declines regardless. The new SecretsGrpcHubServerOptions.
MaxNamesPerRequest defaults to 1000, comfortably above that chunk so a
compliant sweep never meets it, and Validate() refuses a non-positive value
at startup rather than letting it read at runtime as a broken follower. The
refusal states the cap and the received count — neither is secret, and an
operator cannot resize a batch without them — and never a name. The cap sits
behind the interceptor, so it is defence in depth against a misbehaving
token holder, not an anonymous-DoS control.
2. AddZbSecretsGrpcHub now refuses a second registration. The two calls do not
compose: options binding MERGES (a key absent from the second section leaves
the first section's value standing) while the SectionPath stamp is
last-one-wins, so a hub registered twice could serve a token bound from one
section while every failure message named the other. Detected via a private
marker registration. The extension also gains real DI coverage — the intended
single-call path asserted for token/SectionPath coherence through a built
provider, a custom section path stamped end to end, a missing token throwing
from the registration call itself, and the duplicate call throwing.
3. Client-streaming and duplex-streaming denial tests, so all four gated handler
shapes are pinned rather than only unary and server-streaming, plus a
valid-token test that reaches all three streaming continuations.
4. A cancellation test: GetSecrets hands the ServerCallContext token through to
the store, pinned against a fake that records it.
5. The proto's service comment stated the ISecretReplicationHub mapping in prose
that no longer matched the RPC name; it now says GetManifestAsync ->
GetManifest and GetManyAsync -> GetSecrets explicitly. Comment-only — no
message or rpc definition is touched. The GetSecrets rpc comment also notes
the new RESOURCE_EXHAUSTED cap, which a client author has to know about.
Grpc server suite 50 -> 73 tests; full solution suite green, 0 new warnings.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Client half of the central-hosted secrets hub: a site node follows central and
never pushes.
- `ISecretsHubReader` + `GrpcSecretsHubClient` — the read half of the hub as a
follower sees it (manifest + bulk fetch, nothing else). Bearer credential and a
per-call deadline are attached per call; responses convert through
`WireToDomain`. Faults are deliberately not caught here, so an unreachable hub
can never be mistaken for an empty one.
- `WireToDomain` — strict, per-row proto -> domain parse mirroring the Akka
replicator's `SecretRowDto`: a name the domain would reject, a content type
naming no enum member, or a non-round-trip timestamp is refused at the boundary.
Explicit proto3 presence keeps null distinguishable from empty.
- `GrpcSecretSyncService` — `SqlServerSecretSyncService` with the push half
removed. One `ReconcileAsync` per tick (pull-only by the reconciler's own
contract, and batched, so a cold-start pull set cannot exceed gRPC's 4 MB cap),
never throws out of `ExecuteAsync`, warns once and retries on the next interval.
The local store is left completely undecorated — the sweep is its only hub-ward
writer.
- `SecretsGrpcHubClientOptions` — fail-closed `Validate()` naming the exact
configuration key, including the section a caller actually bound. The bearer
token is never echoed by any message.
- `AddZbSecretsGrpcHubClient` — binds, eagerly validates, and registers the
channel under a service KEY so it cannot lose a `TryAdd` race to an
application's own `GrpcChannel` and dial the wrong endpoint with the hub
credential attached.
Tests 52/52 (red-first), including a source-scan guard pinning that no client file
names `ComputePushSet`, `ReadLocalAsync`, or `ApplyReplicatedAsync`, and a
reflection pin that the reader seam exposes exactly two read operations.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
The server half of the central-hosted pull-only secrets hub. SecretsHubService
adapts the node's own ISecretStore onto secrets_hub.v1 using exactly two read
members (GetManifestAsync, GetAsync) — the proto exposes no write RPC, so a
follower has no path to put a row into central whatever its credentials allow.
GetSecrets omits a name the hub does not hold and skips one that will not parse
as a SecretName; neither is an error, because both describe a follower acting on
a manifest that was true a moment ago. Failing the call would stall the sweep and
let one bad name deny the follower every other row in the batch. Names are
re-validated server-side (the wire type is a plain string) and de-duplicated on
the normalized value.
DomainToWire mirrors the Akka replicator's SecretRowDto discipline exactly:
round-trip "O" timestamps so a DateTimeOffset's offset and ticks survive the
last-writer-wins equality compare, the content type as its enum MEMBER NAME so an
unknown value is rejected rather than decoded to the zero member, and nullable
fields left absent rather than written as "" so null and empty stay distinct.
SecretsHubAuthInterceptor fails closed: with no token configured every call is
denied and one warning names the exact configuration key — never a token value,
presented or configured, at any level. Denial is uniform (one Status, one detail,
no "why") so a caller gets no oracle. Comparison is constant-time over UTF-8
bytes. All four handler shapes are gated, not just unary, so a future RPC cannot
arrive unauthenticated by omission; more than one Authorization header is refused
as a smuggling shape. The type keeps EXACTLY ONE public constructor — Grpc's
activator silently stops invoking an interceptor it cannot construct
unambiguously, and the failure mode is an auth gate that quietly does not run — so
a test pins the count.
Tests: 50, red-first (compile-red, then a fail-closed mutation verified 5 of them
bite). Full Secrets suite 328 passed / 0 failed / 15 live-gated skips, 0 new
warnings.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Adds the third replicator sibling next to Replicator.SqlServer and
Replicator.AkkaDotNet: the central-hosted gRPC secrets hub. This commit is
scaffold only -- project, wire contract, solution wiring -- no service or
client implementation.
The proto mirrors exactly the read half of ISecretReplicationHub
(GetManifestAsync, GetManyAsync). ApplyReplicatedAsync is deliberately NOT
mirrored: in this topology every update originates at central and a site node
is a strict follower, so the narrow contract IS the enforcement -- with no
write RPC on the wire, a mis-wired or compromised site node has no way to push
a row into the hub regardless of what its authorization allows.
Messages are hand-written mirrors of StoredSecret and SecretManifestEntry
rather than serializations of the domain types, following the Akka
replicator's SecretRowDto precedent: the trust boundary stays structural, and
a field added to StoredSecret later cannot start crossing the network merely
by existing. Timestamps are ISO-8601 "O" strings (not google.protobuf.
Timestamp, which cannot round-trip a DateTimeOffset's offset) so a relayed row
is byte-identical to the one central holds; nullable .NET fields use proto3
explicit presence so null and empty string stay distinguishable.
Grpc 2.76.0 / Google.Protobuf 3.34.1 match what ScadaBridge, OtOpcUa and the
sibling ZB.MOM.WW.LocalDb libraries already resolve, so a consumer taking both
gets one Grpc rather than a downgrade warning.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Design doc + runbook Production-enablement section + GAPS/index updates.
The previously-untracked production-KEK decision is now recorded: one
fleet-wide KEK via appsettings/env (never committed); ScadaBridge rejects
the SqlServer hub on topology grounds (sites talk to central, never the
DB) in favor of a new ZB.MOM.WW.Secrets.Replicator.Grpc package —
pull-only, updates originate at central only, by design.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Environment.SetEnvironmentVariable/GetEnvironmentVariable mutate the
process-wide environ; concurrent setenv/getenv across parallel xunit test
classes is a documented crash/flake source on glibc even with unique
variable names. Group the six classes that touch process env vars into a
DisableParallelization=true collection so they run serially against each
other while everything else keeps running in parallel.
Fixes scadaproj#2. ConfirmDeleteModal used the bare `modal` class and the
Secrets.Ui RCL shipped no CSS at all, silently depending on the host having
no opinion about that selector. Bootstrap 5 does: `.modal { display: none }`
made the delete modal permanently invisible on every Bootstrap host
(ScadaBridge, OtOpcUa, HistorianGateway — confirmed live on the first two)
while every @onclick handler kept working. Only MxGateway, the lone
Bootstrap-free host, could ever have rendered it.
Fix, entirely inside the RCL so a package bump repairs every host with no
host-side changes:
- class vocabulary renamed to zb-secrets-modal / -backdrop / -card / -title
so no host framework selector can match the elements;
- the component emits its own <style> block alongside the markup. Scoped
.razor.css was deliberately NOT used: three of the four family hosts never
link a scoped-CSS bundle, so isolation CSS would silently fail to load —
the exact defect class being fixed. Theme tokens with fallbacks keep the
card legible even on a host without the kit stylesheet.
- data-testids unchanged, so existing Playwright gates and bUnit tests keep
their selectors.
New bUnit pins: no Bootstrap-reserved class name appears in the rendered
markup, and the component ships a style block that both positions the
overlay and gives it a display mode. Suite 182 pass / 0 fail / 0.2.3.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
ActorContextAfterAwaitTests asserted that a ConfigureAwait(false)
continuation on the shared thread pool always throws NotSupportedException
when reading Self/Context. That property is a timing accident, not an Akka
guarantee: pool threads are exactly where Akka and the TestKit legitimately
install the [ThreadStatic] actor cell during mailbox runs and async
continuations, and Akka 1.5.62 has two non-throwing states besides — a
cleared cell makes ActorBase.Context return null (NullReferenceException on
.Self, not NotSupportedException) and ActorBase.Self return _clearedSelf
without any throw. Under parallel suite load the assertion failed once at
exactly that seam (2026-07-18); ironically the test's own doc comment said
the behaviour "does not reliably throw" and then asserted reliability.
The illegal reads now run on a dedicated new thread (LongRunning), the one
place the no-context state is guaranteed, while the realistic
ConfigureAwait(false) escape from the mailbox is kept. If the reads ever
unexpectedly succeed again, the failure message reports whose context the
thread was carrying. Verified 3 consecutive full-project runs green (38/38)
after 8 instrumented runs hunting the original repro.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Closes the defect in scadaproj#1. The hang was never Akka: the package's DI
wiring closed a circular singleton dependency the container cannot see through
factory lambdas — ISecretStore (ReplicatingSecretStore decorator) ->
ISecretReplicator -> SecretReplicationActorProvider -> ISecretCacheInvalidator
-> DefaultSecretResolver -> ISecretStore. Resolution recurses around the loop
until MS.DI's StackGuard hops it onto a fresh thread-pool thread, which then
blocks forever on a singleton call-site lock the first thread still holds:
a silent permanent hang instead of a stack overflow. Managed stacks from
dotnet-dump show the repeating cycle and both parked threads; both candidate
causes in the issue (DistributedPubSub.Get vs the Lazy lock, missing
Akka.Cluster.Tools HOCON) are disproven — the actor constructor was never
reached, and the deadlock reproduces on a single non-clustered node.
Fix: defer the one cycle-closing edge. The provider now gets a
DeferredSecretCacheInvalidator that resolves the real invalidator on first
eviction — which only happens when a replicated row is applied, strictly after
graph resolution. Severing the edge instead is wrong: a null-invalidator
experiment ran the live gate at 5/6, with deleted secrets still resolving on
the peer. The SqlServer package never had the cycle (its replicator chain
never touches the invalidator), which is why the hub gate always passed.
Verified: live 2-node convergence gate now 6/6 (was: infinite hang), including
the delete-visibility check that proves the deferred invalidator really evicts.
New HostedProcessResolutionTests builds the graph as a host does (container-
registered ActorSystem, hosted services, watchdogged resolves) and fails on
0.2.1; DeferredSecretCacheInvalidatorTests pins the wrapper contract. Full
suite 180 passed / 0 failed / 15 skipped (env-gated live SQL).
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
AddZbSecretsAkkaReplication called AddZbSecrets FIRST, which does
TryAddSingleton<ISecretReplicator, NoOpSecretReplicator>(). The package's own
TryAddSingleton<ISecretReplicator> therefore found a descriptor already present
and was silently discarded.
Consequence: ISecretReplicator resolved to the no-op sink, so every write
published into nothing; and because SecretReplicationActor is only spawned as a
side effect of constructing AkkaSecretReplicator, no actor was ever created
either. No exception, no log line - a cluster that reports healthy and silently
never converges. The worst available failure mode for a secrets store.
Fix: register ISecretReplicator BEFORE AddZbSecrets, matching what the SQL-Server
package already did. Found during OtOpcUa adoption (Task 6), which is the first
code that ever built a container around this extension.
Root cause of the gap: the SQL-Server package had a DI test asserting its
replicator type (AddZbSecretsSqlServerTests:58) and the correct order; the Akka
package had neither. Every Akka test exercised the actor, serializer and
reconciler in isolation - none built a container, so nothing could see it. This
is the third instance of the same defect class in this library (the inert
ISecretReplicator seam, the unregistered concrete SqliteSecretStore, and now
this), all of which share one cause: unit tests that never construct the DI graph.
Adds AddZbSecretsAkkaReplicationTests (5 tests) asserting registration at the
ServiceCollection level. Verified to discriminate: with the 0.2.0 order restored,
2 of the 5 fail; with the fix, all 5 pass. They assert descriptors rather than
resolving from a provider on purpose - resolving ISecretReplicator eagerly spawns
the actor, whose PreStart needs DistributedPubSub and therefore a joined cluster,
which would make the test hang rather than fail.
Full suite Release-green: 175 passed, 15 skipped, no new warnings.
Joseph Doherty