The ScadaBridge central shared-store live gate proved SqlServerSecretsStoreMigrator's
"safe to run concurrently" claim false on a virgin database: Serializable isolation does
not serialize the IF NOT EXISTS metadata probes against another session's in-flight
CREATE, so of two central nodes booting together the loser died with SqlException 2714
("There is already an object named 'zbsecrets'") — the retry filter covered only
deadlock 1205. Losing such a race means the object now exists (the desired end state),
so 2714 (schema/table), 1913 (index) and 2627 (schema_version MERGE-insert PK) are now
retryable alongside 1205; the retry re-runs the idempotent IF NOT EXISTS path.
Classification scans the whole SqlException.Errors collection (2714 arrives stacked with
2759), the retry cap keeps genuine conflicts failing closed, and 2705 is deliberately
excluded (unreachable — columns live in a single CREATE TABLE statement). Predicate
extracted as an internal static pinned by offline tests; live suite gains an env-gated
4-concurrent-migrators virgin-database test. Six packages pushed to the Gitea feed at
0.5.1.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Bump ZB.MOM.WW.Secrets family 0.4.1 -> 0.5.0 and date the CHANGELOG
entry. All 6 packages (Abstractions, Secrets, Replicator.SqlServer,
Replicator.AkkaDotNet, Replicator.Grpc, Ui) packed and pushed to the
dohertj2-gitea feed; restore-verified from a scratch consumer with a
fresh package cache. Suite 470 passed / 0 failed / 15 skipped.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
The ScadaBridge live gate (check 4) found SecretsHubAuthInterceptor denying
correctly but logging nothing: the only trace of a refusal was a framework
Grpc.AspNetCore.Server Information line. A follower whose bearer token was
mis-rotated therefore stopped converging silently, with the hub showing it
only at INF.
Every denial is now counted and summarised in a Warning emitted on the first
denial after startup or after a 60-second quiet window, carrying the denials
since the previous summary plus a breakdown by cause. A single probe is
visible immediately; a sustained flood is bounded to one line per window,
which is what makes warning safe at all — the endpoint is unauthenticated by
construction, so a per-denial warning would be an unbounded log write driven
by an unauthenticated caller. Counters are Interlocked and the emit is
decided by a compare-exchange on the last-emit timestamp, so exactly one
thread emits per window and a denial counted after the drain carries into
the next summary rather than being lost.
The cause breakdown is server-side diagnostics only. Denied() is untouched:
every rejection still returns the identical uniform Unauthenticated status,
the comparison is still constant-time, and no token material appears in any
log line. The existing warn-once-on-unconfigured-token signal is unchanged
and still emitted exactly once; denials in that state count toward the new
summary's "unconfigured" category.
TimeProvider is an optional default-valued parameter rather than a second
overload — the type must keep exactly one public constructor or the gRPC
interceptor activator silently stops invoking the auth gate.
Two existing tests scoped their assertions to the configuration warning,
since both kinds now share a category: the new line is deliberate behavior,
not drift.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Bumps the ZB.MOM.WW.Secrets family from 0.3.0 to 0.4.0 and adds the new
ZB.MOM.WW.Secrets.Replicator.Grpc package to the packable set. Publishes
all six 0.4.0 nupkgs to the dohertj2-gitea feed and restore-verifies the
new package resolves from it in a fresh --packages directory. Existing
packages are behaviorally unchanged — no breaking changes.
Adds ZB.MOM.WW.Secrets/CHANGELOG.md (none existed previously) to record
this and prior releases, and lists the new package in README.md's
Packages table. Also includes the controller's pending ledger updates
for docs/plans/2026-08-07-secrets-grpc-hub-implementation.md (Tasks
1-4 and 8 marked completed).
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Fixes scadaproj#2. ConfirmDeleteModal used the bare `modal` class and the
Secrets.Ui RCL shipped no CSS at all, silently depending on the host having
no opinion about that selector. Bootstrap 5 does: `.modal { display: none }`
made the delete modal permanently invisible on every Bootstrap host
(ScadaBridge, OtOpcUa, HistorianGateway — confirmed live on the first two)
while every @onclick handler kept working. Only MxGateway, the lone
Bootstrap-free host, could ever have rendered it.
Fix, entirely inside the RCL so a package bump repairs every host with no
host-side changes:
- class vocabulary renamed to zb-secrets-modal / -backdrop / -card / -title
so no host framework selector can match the elements;
- the component emits its own <style> block alongside the markup. Scoped
.razor.css was deliberately NOT used: three of the four family hosts never
link a scoped-CSS bundle, so isolation CSS would silently fail to load —
the exact defect class being fixed. Theme tokens with fallbacks keep the
card legible even on a host without the kit stylesheet.
- data-testids unchanged, so existing Playwright gates and bUnit tests keep
their selectors.
New bUnit pins: no Bootstrap-reserved class name appears in the rendered
markup, and the component ships a style block that both positions the
overlay and gives it a display mode. Suite 182 pass / 0 fail / 0.2.3.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Closes the defect in scadaproj#1. The hang was never Akka: the package's DI
wiring closed a circular singleton dependency the container cannot see through
factory lambdas — ISecretStore (ReplicatingSecretStore decorator) ->
ISecretReplicator -> SecretReplicationActorProvider -> ISecretCacheInvalidator
-> DefaultSecretResolver -> ISecretStore. Resolution recurses around the loop
until MS.DI's StackGuard hops it onto a fresh thread-pool thread, which then
blocks forever on a singleton call-site lock the first thread still holds:
a silent permanent hang instead of a stack overflow. Managed stacks from
dotnet-dump show the repeating cycle and both parked threads; both candidate
causes in the issue (DistributedPubSub.Get vs the Lazy lock, missing
Akka.Cluster.Tools HOCON) are disproven — the actor constructor was never
reached, and the deadlock reproduces on a single non-clustered node.
Fix: defer the one cycle-closing edge. The provider now gets a
DeferredSecretCacheInvalidator that resolves the real invalidator on first
eviction — which only happens when a replicated row is applied, strictly after
graph resolution. Severing the edge instead is wrong: a null-invalidator
experiment ran the live gate at 5/6, with deleted secrets still resolving on
the peer. The SqlServer package never had the cycle (its replicator chain
never touches the invalidator), which is why the hub gate always passed.
Verified: live 2-node convergence gate now 6/6 (was: infinite hang), including
the delete-visibility check that proves the deferred invalidator really evicts.
New HostedProcessResolutionTests builds the graph as a host does (container-
registered ActorSystem, hosted services, watchdogged resolves) and fails on
0.2.1; DeferredSecretCacheInvalidatorTests pins the wrapper contract. Full
suite 180 passed / 0 failed / 15 skipped (env-gated live SQL).
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
AddZbSecretsAkkaReplication called AddZbSecrets FIRST, which does
TryAddSingleton<ISecretReplicator, NoOpSecretReplicator>(). The package's own
TryAddSingleton<ISecretReplicator> therefore found a descriptor already present
and was silently discarded.
Consequence: ISecretReplicator resolved to the no-op sink, so every write
published into nothing; and because SecretReplicationActor is only spawned as a
side effect of constructing AkkaSecretReplicator, no actor was ever created
either. No exception, no log line - a cluster that reports healthy and silently
never converges. The worst available failure mode for a secrets store.
Fix: register ISecretReplicator BEFORE AddZbSecrets, matching what the SQL-Server
package already did. Found during OtOpcUa adoption (Task 6), which is the first
code that ever built a container around this extension.
Root cause of the gap: the SQL-Server package had a DI test asserting its
replicator type (AddZbSecretsSqlServerTests:58) and the correct order; the Akka
package had neither. Every Akka test exercised the actor, serializer and
reconciler in isolation - none built a container, so nothing could see it. This
is the third instance of the same defect class in this library (the inert
ISecretReplicator seam, the unregistered concrete SqliteSecretStore, and now
this), all of which share one cause: unit tests that never construct the DI graph.
Adds AddZbSecretsAkkaReplicationTests (5 tests) asserting registration at the
ServiceCollection level. Verified to discriminate: with the 0.2.0 order restored,
2 of the 5 fail; with the fix, all 5 pass. They assert descriptors rather than
resolving from a provider on purpose - resolving ISecretReplicator eagerly spawns
the actor, whose PreStart needs DistributedPubSub and therefore a joined cluster,
which would make the test hang rather than fail.
Full suite Release-green: 175 passed, 15 skipped, no new warnings.
Secrets were per-node SQLite, so a secret written on one node was invisible to
the rest of a cluster. G-7's design resolved the "shared SQL store vs Akka
replicator" fork to build only the former; both are built here so the choice is
a deployment decision (availability vs partition tolerance) rather than a
library limitation.
Two new packages — ZB.MOM.WW.Secrets.Replicator.SqlServer (shared store, plus a
local-store-with-hub mode) and .Replicator.AkkaDotNet (peer-to-peer over
distributed pub/sub). Core gains ISecretsStoreMigrator, one shared
SecretLastWriterWins predicate so no two stores can disagree on a tie, the
transport-agnostic reconciler, and ReplicatingSecretStore — which closes a real
gap: nothing had ever called ISecretReplicator.PublishAsync, so the seam was
inert and local writes would not have propagated at all.
Verified 182 pass / 1 skip / 0 warnings, including 15 live tests against a real
SQL Server 2022 (the SQLite suite ported case-for-case, so any behavioural
divergence between the stores fails) and a 9-test in-process 2-node Akka
cluster over real remoting. A post-build review caught six defects, all fixed
and now covered: both replication modes could not resolve from the container
(no test had built one), an unbounded fetch that broke past SQL Server's
2100-parameter cap, a poison row that aborted the rest of its batch forever,
Enum.Parse on peer input that could restart the actor in a loop, null crypto
blobs crossing the trust boundary, and a silently dropped pull-read failure.
Packed at 0.2.0 and vulnerability-scanned clean; not yet published to the feed.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
G-8 (KEK rotation) — built in ZB.MOM.WW.Secrets, lib 0.1.2->0.1.3:
- ISecretCipher.Rewrap(row, oldKek, newKek): re-wraps the per-secret DEK only
(bodies never re-encrypted; revision/timestamps preserved -> invisible to
cluster LWW). Fail-closed on wrong old-KEK id, wrong key bytes, and malformed
wraps; DEK zeroed on all paths.
- ISecretStore.ApplyRewrapAsync(rewrappedRow, expectedCurrentWrappedDek):
updates only the 4 wrap columns + kek_id, compare-and-swap on the current
wrapped DEK so a concurrent set/rotate cannot corrupt a row (closes a
review-caught TOCTOU).
- KekRotationService.RewrapAllAsync + RewrapReport: enumerate all rows incl.
tombstones, idempotent/resumable skip-already-current, bounded CAS-retry,
fail-closed on unknown/identical KEK.
- `secret rewrap-all` CLI verb: key material only via env-var name / file path,
JSON counts report; README section + operator runbook.
Verified: full offline suite green (82 core + 15 UI, 0 regressions) + end-to-end
CLI smoke + adversarial crypto review (all 7 categories PASS; TOCTOU fixed).
G-7 (clustered replication) — designed + planned, no code:
- Fork resolved to build Option A (shared SQL-Server ISecretStore); Akka
replicator ZB.MOM.WW.Secrets.Akka is a deferred phase-2. Design doc +
executable plan + .tasks.json under docs/plans/2026-07-17-secrets-g7-*.
Tracking: components/secrets/GAPS.md + CLAUDE.md secrets row updated.
Live HistorianGateway boot caught that the expander scanned a '_secretsComment' appsettings
value documenting the ${secret:...} syntax, tried to resolve the literal example token, and
crashed boot. The family uses _*Comment keys pervasively, so skip any config key whose leaf
segment starts with '_' (the _comment convention). Functional keys never lead with '_'.
Reference-consumer (HistorianGateway) caught that SecretsAuthorization.AdminRole was
'administrator' (lowercase) while the family canonical role is CanonicalRole.Administrator.
ASP.NET role claims compare case-sensitively (Ordinal), so a real admin got 403 on the
secrets UI. Now nameof(CanonicalRole.Administrator).
Joseph Doherty