Bumps the ZB.MOM.WW.Secrets family from 0.3.0 to 0.4.0 and adds the new
ZB.MOM.WW.Secrets.Replicator.Grpc package to the packable set. Publishes
all six 0.4.0 nupkgs to the dohertj2-gitea feed and restore-verifies the
new package resolves from it in a fresh --packages directory. Existing
packages are behaviorally unchanged — no breaking changes.
Adds ZB.MOM.WW.Secrets/CHANGELOG.md (none existed previously) to record
this and prior releases, and lists the new package in README.md's
Packages table. Also includes the controller's pending ledger updates
for docs/plans/2026-08-07-secrets-grpc-hub-implementation.md (Tasks
1-4 and 8 marked completed).
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
An internal validateEagerly seam on AddZbSecretsGrpcHub replaces the harness's
hand-copied registration fragment, so the deny-all second-layer tests exercise
the production wiring and cannot drift from it.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Adds the end-to-end layer the unit suites structurally cannot reach: a real
in-process ASP.NET Core host serving SecretsHubService through
AddZbSecretsGrpcHub/MapZbSecretsHub, a follower built by the shipped
AddZbSecretsGrpcHubClient, a real gRPC call between them, and real SQLite
stores on both sides. The class of defect this catches is pieces that are each
individually correct and are never connected to one another — for instance the
interceptor's warn-once, which only holds if the container really hands gRPC
the singleton it registered.
Covered: byte-identical convergence over the wire (every ciphertext/nonce/tag/
wrapped-DEK byte, kek_id, revision, both timestamps INCLUDING their offsets,
across a tombstone, a nulls-everywhere row and an empty-string-description row);
delete propagation; a hub unreachable at follower boot leaving the last-known-
good store intact and converging once it answers; the auth negatives fail-closed
across the wire with one uniform denial detail; the per-request name cap as
ResourceExhausted; and an RpcException from the call invoker escaping the hub
client unchanged, so a transport fault stays distinguishable from a parse fault.
The credential-hygiene assertions are made against logs captured at Trace on
BOTH hosts, so "no token material ever reaches a sink" is asserted at the most
verbose setting a deployment could switch on.
SecretsHubContractPinTests reads the generated descriptor — not the .proto text
— and pins the RPC set to exactly {GetManifest, GetSecrets}, plus unary-only and
the package-qualified service name. The hub is pull-only by construction:
updates originate at central only, and the absence of a write RPC is what
enforces that, not authorization a mis-wired follower may hold more of than
intended. A third RPC silently retires the guarantee, so the failure message
says so.
Microsoft.AspNetCore.TestHost pinned at 10.0.7 — the version ZB.MOM.WW.LocalDb,
OtOpcUa and ScadaBridge already pin, matching the Microsoft.Extensions.* line.
Full suite: 436 passed / 0 failed / 15 skipped, no new warnings.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
A single malformed row or manifest entry poisoned the whole sweep, permanently.
WireToDomain.ToRows/ToManifest materialized eagerly, so one unparseable element
threw out of the fetch callback, past SecretReplicationReconciler's per-batch
handling, into SweepSafelyAsync's whole-sweep catch. The follower's pull set is
derived from the hub's manifest, so the identical bad element arrived in the
identical batch on every interval and nothing ever converged — logged as "sync
failed", indistinguishable from a hub outage.
Isolation now matches the Akka sibling's split exactly: WireToDomain stays a
strict per-item parser and loses its two batch entry points (their absence is
pinned by a reflection test), while GrpcSecretsHubClient converts each item
inside its own try — the same shape as SecretReplicationActor.Materialize.
A dropped item costs one warning naming the secret and a reason category, and
every well-formed sibling still converges. Only parse-shaped exceptions are
caught (ArgumentException, FormatException); a defect still propagates. A row
whose own name is unparseable is logged under a marker rather than echoing the
untrusted string, and no ciphertext, DEK or credential reaches a log line.
GrpcSecretsHubClient had zero coverage — the existing fake implements the
ISecretsHubReader seam directly, bypassing the header, the deadline and the
conversion, which are the only things that class does. It is now driven for
real through a fake CallInvoker handed to the generated client.
The stale doc comment claiming the reconciler isolated the failure is replaced
by one describing the mechanism that actually exists, and Endpoint gains a note
that a plain-http hub sends the bearer token cleartext (doc only, no validation
change).
Tests 52 -> 67 in the package suite; solution green, no new warnings.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Five review items on the Task 2 server half.
1. GetSecrets now enforces a server-side cap on request.Names, refusing an
over-cap batch with ResourceExhausted BEFORE any store read runs. The
well-behaved follower chunks at 500, but client discipline is not a server
defence — the family precedent (HistorianGateway's max_values) is that the
server declines regardless. The new SecretsGrpcHubServerOptions.
MaxNamesPerRequest defaults to 1000, comfortably above that chunk so a
compliant sweep never meets it, and Validate() refuses a non-positive value
at startup rather than letting it read at runtime as a broken follower. The
refusal states the cap and the received count — neither is secret, and an
operator cannot resize a batch without them — and never a name. The cap sits
behind the interceptor, so it is defence in depth against a misbehaving
token holder, not an anonymous-DoS control.
2. AddZbSecretsGrpcHub now refuses a second registration. The two calls do not
compose: options binding MERGES (a key absent from the second section leaves
the first section's value standing) while the SectionPath stamp is
last-one-wins, so a hub registered twice could serve a token bound from one
section while every failure message named the other. Detected via a private
marker registration. The extension also gains real DI coverage — the intended
single-call path asserted for token/SectionPath coherence through a built
provider, a custom section path stamped end to end, a missing token throwing
from the registration call itself, and the duplicate call throwing.
3. Client-streaming and duplex-streaming denial tests, so all four gated handler
shapes are pinned rather than only unary and server-streaming, plus a
valid-token test that reaches all three streaming continuations.
4. A cancellation test: GetSecrets hands the ServerCallContext token through to
the store, pinned against a fake that records it.
5. The proto's service comment stated the ISecretReplicationHub mapping in prose
that no longer matched the RPC name; it now says GetManifestAsync ->
GetManifest and GetManyAsync -> GetSecrets explicitly. Comment-only — no
message or rpc definition is touched. The GetSecrets rpc comment also notes
the new RESOURCE_EXHAUSTED cap, which a client author has to know about.
Grpc server suite 50 -> 73 tests; full solution suite green, 0 new warnings.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Client half of the central-hosted secrets hub: a site node follows central and
never pushes.
- `ISecretsHubReader` + `GrpcSecretsHubClient` — the read half of the hub as a
follower sees it (manifest + bulk fetch, nothing else). Bearer credential and a
per-call deadline are attached per call; responses convert through
`WireToDomain`. Faults are deliberately not caught here, so an unreachable hub
can never be mistaken for an empty one.
- `WireToDomain` — strict, per-row proto -> domain parse mirroring the Akka
replicator's `SecretRowDto`: a name the domain would reject, a content type
naming no enum member, or a non-round-trip timestamp is refused at the boundary.
Explicit proto3 presence keeps null distinguishable from empty.
- `GrpcSecretSyncService` — `SqlServerSecretSyncService` with the push half
removed. One `ReconcileAsync` per tick (pull-only by the reconciler's own
contract, and batched, so a cold-start pull set cannot exceed gRPC's 4 MB cap),
never throws out of `ExecuteAsync`, warns once and retries on the next interval.
The local store is left completely undecorated — the sweep is its only hub-ward
writer.
- `SecretsGrpcHubClientOptions` — fail-closed `Validate()` naming the exact
configuration key, including the section a caller actually bound. The bearer
token is never echoed by any message.
- `AddZbSecretsGrpcHubClient` — binds, eagerly validates, and registers the
channel under a service KEY so it cannot lose a `TryAdd` race to an
application's own `GrpcChannel` and dial the wrong endpoint with the hub
credential attached.
Tests 52/52 (red-first), including a source-scan guard pinning that no client file
names `ComputePushSet`, `ReadLocalAsync`, or `ApplyReplicatedAsync`, and a
reflection pin that the reader seam exposes exactly two read operations.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
The server half of the central-hosted pull-only secrets hub. SecretsHubService
adapts the node's own ISecretStore onto secrets_hub.v1 using exactly two read
members (GetManifestAsync, GetAsync) — the proto exposes no write RPC, so a
follower has no path to put a row into central whatever its credentials allow.
GetSecrets omits a name the hub does not hold and skips one that will not parse
as a SecretName; neither is an error, because both describe a follower acting on
a manifest that was true a moment ago. Failing the call would stall the sweep and
let one bad name deny the follower every other row in the batch. Names are
re-validated server-side (the wire type is a plain string) and de-duplicated on
the normalized value.
DomainToWire mirrors the Akka replicator's SecretRowDto discipline exactly:
round-trip "O" timestamps so a DateTimeOffset's offset and ticks survive the
last-writer-wins equality compare, the content type as its enum MEMBER NAME so an
unknown value is rejected rather than decoded to the zero member, and nullable
fields left absent rather than written as "" so null and empty stay distinct.
SecretsHubAuthInterceptor fails closed: with no token configured every call is
denied and one warning names the exact configuration key — never a token value,
presented or configured, at any level. Denial is uniform (one Status, one detail,
no "why") so a caller gets no oracle. Comparison is constant-time over UTF-8
bytes. All four handler shapes are gated, not just unary, so a future RPC cannot
arrive unauthenticated by omission; more than one Authorization header is refused
as a smuggling shape. The type keeps EXACTLY ONE public constructor — Grpc's
activator silently stops invoking an interceptor it cannot construct
unambiguously, and the failure mode is an auth gate that quietly does not run — so
a test pins the count.
Tests: 50, red-first (compile-red, then a fail-closed mutation verified 5 of them
bite). Full Secrets suite 328 passed / 0 failed / 15 live-gated skips, 0 new
warnings.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Adds the third replicator sibling next to Replicator.SqlServer and
Replicator.AkkaDotNet: the central-hosted gRPC secrets hub. This commit is
scaffold only -- project, wire contract, solution wiring -- no service or
client implementation.
The proto mirrors exactly the read half of ISecretReplicationHub
(GetManifestAsync, GetManyAsync). ApplyReplicatedAsync is deliberately NOT
mirrored: in this topology every update originates at central and a site node
is a strict follower, so the narrow contract IS the enforcement -- with no
write RPC on the wire, a mis-wired or compromised site node has no way to push
a row into the hub regardless of what its authorization allows.
Messages are hand-written mirrors of StoredSecret and SecretManifestEntry
rather than serializations of the domain types, following the Akka
replicator's SecretRowDto precedent: the trust boundary stays structural, and
a field added to StoredSecret later cannot start crossing the network merely
by existing. Timestamps are ISO-8601 "O" strings (not google.protobuf.
Timestamp, which cannot round-trip a DateTimeOffset's offset) so a relayed row
is byte-identical to the one central holds; nullable .NET fields use proto3
explicit presence so null and empty string stay distinguishable.
Grpc 2.76.0 / Google.Protobuf 3.34.1 match what ScadaBridge, OtOpcUa and the
sibling ZB.MOM.WW.LocalDb libraries already resolve, so a consumer taking both
gets one Grpc rather than a downgrade warning.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Design doc + runbook Production-enablement section + GAPS/index updates.
The previously-untracked production-KEK decision is now recorded: one
fleet-wide KEK via appsettings/env (never committed); ScadaBridge rejects
the SqlServer hub on topology grounds (sites talk to central, never the
DB) in favor of a new ZB.MOM.WW.Secrets.Replicator.Grpc package —
pull-only, updates originate at central only, by design.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Environment.SetEnvironmentVariable/GetEnvironmentVariable mutate the
process-wide environ; concurrent setenv/getenv across parallel xunit test
classes is a documented crash/flake source on glibc even with unique
variable names. Group the six classes that touch process env vars into a
DisableParallelization=true collection so they run serially against each
other while everything else keeps running in parallel.
Fixes scadaproj#2. ConfirmDeleteModal used the bare `modal` class and the
Secrets.Ui RCL shipped no CSS at all, silently depending on the host having
no opinion about that selector. Bootstrap 5 does: `.modal { display: none }`
made the delete modal permanently invisible on every Bootstrap host
(ScadaBridge, OtOpcUa, HistorianGateway — confirmed live on the first two)
while every @onclick handler kept working. Only MxGateway, the lone
Bootstrap-free host, could ever have rendered it.
Fix, entirely inside the RCL so a package bump repairs every host with no
host-side changes:
- class vocabulary renamed to zb-secrets-modal / -backdrop / -card / -title
so no host framework selector can match the elements;
- the component emits its own <style> block alongside the markup. Scoped
.razor.css was deliberately NOT used: three of the four family hosts never
link a scoped-CSS bundle, so isolation CSS would silently fail to load —
the exact defect class being fixed. Theme tokens with fallbacks keep the
card legible even on a host without the kit stylesheet.
- data-testids unchanged, so existing Playwright gates and bUnit tests keep
their selectors.
New bUnit pins: no Bootstrap-reserved class name appears in the rendered
markup, and the component ships a style block that both positions the
overlay and gives it a display mode. Suite 182 pass / 0 fail / 0.2.3.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
ActorContextAfterAwaitTests asserted that a ConfigureAwait(false)
continuation on the shared thread pool always throws NotSupportedException
when reading Self/Context. That property is a timing accident, not an Akka
guarantee: pool threads are exactly where Akka and the TestKit legitimately
install the [ThreadStatic] actor cell during mailbox runs and async
continuations, and Akka 1.5.62 has two non-throwing states besides — a
cleared cell makes ActorBase.Context return null (NullReferenceException on
.Self, not NotSupportedException) and ActorBase.Self return _clearedSelf
without any throw. Under parallel suite load the assertion failed once at
exactly that seam (2026-07-18); ironically the test's own doc comment said
the behaviour "does not reliably throw" and then asserted reliability.
The illegal reads now run on a dedicated new thread (LongRunning), the one
place the no-context state is guaranteed, while the realistic
ConfigureAwait(false) escape from the mailbox is kept. If the reads ever
unexpectedly succeed again, the failure message reports whose context the
thread was carrying. Verified 3 consecutive full-project runs green (38/38)
after 8 instrumented runs hunting the original repro.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Closes the defect in scadaproj#1. The hang was never Akka: the package's DI
wiring closed a circular singleton dependency the container cannot see through
factory lambdas — ISecretStore (ReplicatingSecretStore decorator) ->
ISecretReplicator -> SecretReplicationActorProvider -> ISecretCacheInvalidator
-> DefaultSecretResolver -> ISecretStore. Resolution recurses around the loop
until MS.DI's StackGuard hops it onto a fresh thread-pool thread, which then
blocks forever on a singleton call-site lock the first thread still holds:
a silent permanent hang instead of a stack overflow. Managed stacks from
dotnet-dump show the repeating cycle and both parked threads; both candidate
causes in the issue (DistributedPubSub.Get vs the Lazy lock, missing
Akka.Cluster.Tools HOCON) are disproven — the actor constructor was never
reached, and the deadlock reproduces on a single non-clustered node.
Fix: defer the one cycle-closing edge. The provider now gets a
DeferredSecretCacheInvalidator that resolves the real invalidator on first
eviction — which only happens when a replicated row is applied, strictly after
graph resolution. Severing the edge instead is wrong: a null-invalidator
experiment ran the live gate at 5/6, with deleted secrets still resolving on
the peer. The SqlServer package never had the cycle (its replicator chain
never touches the invalidator), which is why the hub gate always passed.
Verified: live 2-node convergence gate now 6/6 (was: infinite hang), including
the delete-visibility check that proves the deferred invalidator really evicts.
New HostedProcessResolutionTests builds the graph as a host does (container-
registered ActorSystem, hosted services, watchdogged resolves) and fails on
0.2.1; DeferredSecretCacheInvalidatorTests pins the wrapper contract. Full
suite 180 passed / 0 failed / 15 skipped (env-gated live SQL).
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
AddZbSecretsAkkaReplication called AddZbSecrets FIRST, which does
TryAddSingleton<ISecretReplicator, NoOpSecretReplicator>(). The package's own
TryAddSingleton<ISecretReplicator> therefore found a descriptor already present
and was silently discarded.
Consequence: ISecretReplicator resolved to the no-op sink, so every write
published into nothing; and because SecretReplicationActor is only spawned as a
side effect of constructing AkkaSecretReplicator, no actor was ever created
either. No exception, no log line - a cluster that reports healthy and silently
never converges. The worst available failure mode for a secrets store.
Fix: register ISecretReplicator BEFORE AddZbSecrets, matching what the SQL-Server
package already did. Found during OtOpcUa adoption (Task 6), which is the first
code that ever built a container around this extension.
Root cause of the gap: the SQL-Server package had a DI test asserting its
replicator type (AddZbSecretsSqlServerTests:58) and the correct order; the Akka
package had neither. Every Akka test exercised the actor, serializer and
reconciler in isolation - none built a container, so nothing could see it. This
is the third instance of the same defect class in this library (the inert
ISecretReplicator seam, the unregistered concrete SqliteSecretStore, and now
this), all of which share one cause: unit tests that never construct the DI graph.
Adds AddZbSecretsAkkaReplicationTests (5 tests) asserting registration at the
ServiceCollection level. Verified to discriminate: with the 0.2.0 order restored,
2 of the 5 fail; with the fix, all 5 pass. They assert descriptors rather than
resolving from a provider on purpose - resolving ISecretReplicator eagerly spawns
the actor, whose PreStart needs DistributedPubSub and therefore a joined cluster,
which would make the test hang rather than fail.
Full suite Release-green: 175 passed, 15 skipped, no new warnings.
I dismissed this finding from the code review as a false positive, reasoning
that Akka's ActorBase caches Self in a field and that three passing
anti-entropy tests traverse the path. Both premises were wrong. Self resolves
through Context, which is [ThreadStatic], and throws NotSupportedException once
a continuation resumes on a thread-pool thread.
The tests passed because a local SQLite store usually completes await
SYNCHRONOUSLY, so the continuation stayed on the mailbox thread and the context
was still intact. Correctness therefore depended on store latency and
thread-pool timing: green here, broken under a slower or contended store, with
the only symptom a per-peer warning every announce interval while nodes
silently stopped converging. The live-broadcast fast path masked it further —
only the anti-entropy repair path was dead.
Captures self on the actor thread and passes it in. Adds
GenuinelyAsyncSecretStore to force the async path, a regression test that fails
on the unfixed code (20s timeout) and passes in 3s after, and
ActorContextAfterAwaitTests pinning the underlying Akka behaviour so the wrong
assumption cannot be made again. Audited every remaining Self/Sender access in
the actor.
170 pass offline / 184 with the live SQL suite / 1 skip / 0 warnings.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Secrets were per-node SQLite, so a secret written on one node was invisible to
the rest of a cluster. G-7's design resolved the "shared SQL store vs Akka
replicator" fork to build only the former; both are built here so the choice is
a deployment decision (availability vs partition tolerance) rather than a
library limitation.
Two new packages — ZB.MOM.WW.Secrets.Replicator.SqlServer (shared store, plus a
local-store-with-hub mode) and .Replicator.AkkaDotNet (peer-to-peer over
distributed pub/sub). Core gains ISecretsStoreMigrator, one shared
SecretLastWriterWins predicate so no two stores can disagree on a tie, the
transport-agnostic reconciler, and ReplicatingSecretStore — which closes a real
gap: nothing had ever called ISecretReplicator.PublishAsync, so the seam was
inert and local writes would not have propagated at all.
Verified 182 pass / 1 skip / 0 warnings, including 15 live tests against a real
SQL Server 2022 (the SQLite suite ported case-for-case, so any behavioural
divergence between the stores fails) and a 9-test in-process 2-node Akka
cluster over real remoting. A post-build review caught six defects, all fixed
and now covered: both replication modes could not resolve from the container
(no test had built one), an unbounded fetch that broke past SQL Server's
2100-parameter cap, a poison row that aborted the rest of its batch forever,
Enum.Parse on peer input that could restart the actor in a loop, null crypto
blobs crossing the trust boundary, and a silently dropped pull-read failure.
Packed at 0.2.0 and vulnerability-scanned clean; not yet published to the feed.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Packing Secrets 0.1.3 surfaced NU1903: Microsoft.Data.Sqlite 10.0.7 pulls
SQLitePCLRaw.lib.e_sqlite3 2.1.11, which carries high-severity advisory
GHSA-2m69-gcr7-jv3q. Auth.ApiKeys had the same exposure and was already shipped
at 0.1.4 to three consumers, so both libs are fixed rather than only the one
being published.
Fixed the way ZB.MOM.WW.LocalDb already had it: CentralPackageTransitivePinningEnabled
plus a pin to the patched 2.1.12. Bumping Microsoft.Data.Sqlite does not help --
even 10.0.10 still resolves 2.1.11 -- so the pin is the actual fix. The pin reaches
consumers: both nuspecs now declare SQLitePCLRaw.lib.e_sqlite3 >= 2.1.12 as a direct
dependency, verified by restoring the published packages into a scratch project,
which resolves 2.1.12 and scans clean.
Auth goes 0.1.4 -> 0.1.5 with no API change (0.1.4 plus the pin). Suites re-run
after the native-lib swap with identical counts, so no behavioral regression:
Secrets 97 pass/1 skip, Auth 215 pass/1 skip (both skips are Windows-only DPAPI
and opt-in LDAP).
Consumers are NOT yet bumped and remain on vulnerable versions: mxaccessgw 0.1.4,
HistorianGateway 0.1.4, ScadaBridge 0.1.3. Secrets consumers all sit at 0.1.2 and
also lack KEK rotation.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Joseph Doherty