Closes the defect in scadaproj#1. The hang was never Akka: the package's DI
wiring closed a circular singleton dependency the container cannot see through
factory lambdas — ISecretStore (ReplicatingSecretStore decorator) ->
ISecretReplicator -> SecretReplicationActorProvider -> ISecretCacheInvalidator
-> DefaultSecretResolver -> ISecretStore. Resolution recurses around the loop
until MS.DI's StackGuard hops it onto a fresh thread-pool thread, which then
blocks forever on a singleton call-site lock the first thread still holds:
a silent permanent hang instead of a stack overflow. Managed stacks from
dotnet-dump show the repeating cycle and both parked threads; both candidate
causes in the issue (DistributedPubSub.Get vs the Lazy lock, missing
Akka.Cluster.Tools HOCON) are disproven — the actor constructor was never
reached, and the deadlock reproduces on a single non-clustered node.
Fix: defer the one cycle-closing edge. The provider now gets a
DeferredSecretCacheInvalidator that resolves the real invalidator on first
eviction — which only happens when a replicated row is applied, strictly after
graph resolution. Severing the edge instead is wrong: a null-invalidator
experiment ran the live gate at 5/6, with deleted secrets still resolving on
the peer. The SqlServer package never had the cycle (its replicator chain
never touches the invalidator), which is why the hub gate always passed.
Verified: live 2-node convergence gate now 6/6 (was: infinite hang), including
the delete-visibility check that proves the deferred invalidator really evicts.
New HostedProcessResolutionTests builds the graph as a host does (container-
registered ActorSystem, hosted services, watchdogged resolves) and fails on
0.2.1; DeferredSecretCacheInvalidatorTests pins the wrapper contract. Full
suite 180 passed / 0 failed / 15 skipped (env-gated live SQL).
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
- All 4 apps on 0.2.1 (local branches, unpushed).
- Records the corrected security story: the version bump closed NO advisory (all four
repos already resolved patched 2.1.12); the real live vulnerability was in ScadaBridge,
masked by a NuGetAuditSuppress, and was fixed by separate work.
- Records the upstream 0.2.0 inert-Akka-replicator defect, its root cause (DI extensions
with no container-building test - third instance of that class), and the 0.2.1 fix.
- Marks clustered topology WIRED-but-default-OFF and explicitly NOT live-validated;
Task 9 remains open and the topology is not 'adopted' until it passes.
I dismissed this finding from the code review as a false positive, reasoning
that Akka's ActorBase caches Self in a field and that three passing
anti-entropy tests traverse the path. Both premises were wrong. Self resolves
through Context, which is [ThreadStatic], and throws NotSupportedException once
a continuation resumes on a thread-pool thread.
The tests passed because a local SQLite store usually completes await
SYNCHRONOUSLY, so the continuation stayed on the mailbox thread and the context
was still intact. Correctness therefore depended on store latency and
thread-pool timing: green here, broken under a slower or contended store, with
the only symptom a per-peer warning every announce interval while nodes
silently stopped converging. The live-broadcast fast path masked it further —
only the anti-entropy repair path was dead.
Captures self on the actor thread and passes it in. Adds
GenuinelyAsyncSecretStore to force the async path, a regression test that fails
on the unfixed code (20s timeout) and passes in 3s after, and
ActorContextAfterAwaitTests pinning the underlying Akka behaviour so the wrong
assumption cannot be made again. Audited every remaining Self/Sender access in
the actor.
170 pass offline / 184 with the live SQL suite / 1 skip / 0 warnings.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Secrets were per-node SQLite, so a secret written on one node was invisible to
the rest of a cluster. G-7's design resolved the "shared SQL store vs Akka
replicator" fork to build only the former; both are built here so the choice is
a deployment decision (availability vs partition tolerance) rather than a
library limitation.
Two new packages — ZB.MOM.WW.Secrets.Replicator.SqlServer (shared store, plus a
local-store-with-hub mode) and .Replicator.AkkaDotNet (peer-to-peer over
distributed pub/sub). Core gains ISecretsStoreMigrator, one shared
SecretLastWriterWins predicate so no two stores can disagree on a tie, the
transport-agnostic reconciler, and ReplicatingSecretStore — which closes a real
gap: nothing had ever called ISecretReplicator.PublishAsync, so the seam was
inert and local writes would not have propagated at all.
Verified 182 pass / 1 skip / 0 warnings, including 15 live tests against a real
SQL Server 2022 (the SQLite suite ported case-for-case, so any behavioural
divergence between the stores fails) and a 9-test in-process 2-node Akka
cluster over real remoting. A post-build review caught six defects, all fixed
and now covered: both replication modes could not resolve from the container
(no test had built one), an unbounded fetch that broke past SQL Server's
2100-parameter cap, a poison row that aborted the rest of its batch forever,
Enum.Parse on peer input that could restart the actor in a loop, null crypto
blobs crossing the trust boundary, and a silently dropped pull-read failure.
Packed at 0.2.0 and vulnerability-scanned clean; not yet published to the feed.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Packing Secrets 0.1.3 surfaced NU1903: Microsoft.Data.Sqlite 10.0.7 pulls
SQLitePCLRaw.lib.e_sqlite3 2.1.11, which carries high-severity advisory
GHSA-2m69-gcr7-jv3q. Auth.ApiKeys had the same exposure and was already shipped
at 0.1.4 to three consumers, so both libs are fixed rather than only the one
being published.
Fixed the way ZB.MOM.WW.LocalDb already had it: CentralPackageTransitivePinningEnabled
plus a pin to the patched 2.1.12. Bumping Microsoft.Data.Sqlite does not help --
even 10.0.10 still resolves 2.1.11 -- so the pin is the actual fix. The pin reaches
consumers: both nuspecs now declare SQLitePCLRaw.lib.e_sqlite3 >= 2.1.12 as a direct
dependency, verified by restoring the published packages into a scratch project,
which resolves 2.1.12 and scans clean.
Auth goes 0.1.4 -> 0.1.5 with no API change (0.1.4 plus the pin). Suites re-run
after the native-lib swap with identical counts, so no behavioral regression:
Secrets 97 pass/1 skip, Auth 215 pass/1 skip (both skips are Windows-only DPAPI
and opt-in LDAP).
Consumers are NOT yet bumped and remain on vulnerable versions: mxaccessgw 0.1.4,
HistorianGateway 0.1.4, ScadaBridge 0.1.3. Secrets consumers all sit at 0.1.2 and
also lack KEK rotation.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Publishes the three ZB.MOM.WW.LocalDb packages to the Gitea feed (restore-verified
from a scratch consumer) and adds the build/push.sh the other shared libs already have.
The status prose across CLAUDE.md, README.md and components/*/GAPS.md had drifted from
reality, so it was re-derived from the feed listing and the actual PackageReferences +
registration calls on each consumer's default branch rather than from prior claims.
Five claims were false: Health "not yet adopted" (all four apps wire MapZbHealth),
GalaxyRepository's mxaccessgw adoption "a follow-on" (its Server wires
AddZbGalaxyRepository), Configuration "not yet pushed", Secrets G-8 "not yet
committed", and Theme pinned at 0.2.0 (all four are on 0.3.1). Every doc also said
"three apps" while HistorianGateway is a fourth consumer of seven libs, and all
eight libraries' test counts were stale (re-ran each suite; all green).
Surfaces one previously unrecorded gap: Secrets source is at 0.1.3 with KEK rotation
committed, but the feed tops out at 0.1.2, so no app can consume rotation until it
is published.
Health and observability divergence tables are labelled historical, not re-verified —
the libraries are proven wired, but per-app probe coverage vs spec was not re-walked.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
G-8 (KEK rotation) — built in ZB.MOM.WW.Secrets, lib 0.1.2->0.1.3:
- ISecretCipher.Rewrap(row, oldKek, newKek): re-wraps the per-secret DEK only
(bodies never re-encrypted; revision/timestamps preserved -> invisible to
cluster LWW). Fail-closed on wrong old-KEK id, wrong key bytes, and malformed
wraps; DEK zeroed on all paths.
- ISecretStore.ApplyRewrapAsync(rewrappedRow, expectedCurrentWrappedDek):
updates only the 4 wrap columns + kek_id, compare-and-swap on the current
wrapped DEK so a concurrent set/rotate cannot corrupt a row (closes a
review-caught TOCTOU).
- KekRotationService.RewrapAllAsync + RewrapReport: enumerate all rows incl.
tombstones, idempotent/resumable skip-already-current, bounded CAS-retry,
fail-closed on unknown/identical KEK.
- `secret rewrap-all` CLI verb: key material only via env-var name / file path,
JSON counts report; README section + operator runbook.
Verified: full offline suite green (82 core + 15 UI, 0 regressions) + end-to-end
CLI smoke + adversarial crypto review (all 7 categories PASS; TOCTOU fixed).
G-7 (clustered replication) — designed + planned, no code:
- Fork resolved to build Option A (shared SQL-Server ISecretStore); Akka
replicator ZB.MOM.WW.Secrets.Akka is a deferred phase-2. Design doc +
executable plan + .tasks.json under docs/plans/2026-07-17-secrets-g7-*.
Tracking: components/secrets/GAPS.md + CLAUDE.md secrets row updated.
Record the OtOpcUa live wonder gate (item 2). A throwaway harness drove OtOpcUa's
real GalaxyDriverBrowser (secret: ApiKeySecretRef resolved via ISecretResolver) against
the real production MxGateway: dummy secret: ref -> MxGatewayAuthenticationException,
real metadata:read key -> CONNECTED + browsed the real Galaxy root (10 nodes). Temp key
minted via the dashboard, then revoked (functionally proven) + deleted; no prod key
touched; harness + scratch stores destroyed. All four apps now G-2..G-6 done + live-proven.
Record the OtOpcUa secrets adoption (G-2/G-4/G-5/G-6, the last app): FF-merged +
pushed to lmxopcua origin/master @872cf7e3. Update the components/secrets/GAPS.md
OtOpcUa bullet (execution record + the three verified plan deviations + the regression
the gate caught) and the CLAUDE.md secrets-component-table row. All four apps
(HistorianGateway, mxaccessgw, ScadaBridge, OtOpcUa) now adopt ZB.MOM.WW.Secrets.
Update the OtOpcUa umbrella entry for v3.0: the single .../ns namespace is
replaced by .../raw (s=<RawPath>) + .../uns (s=<Area>/<Line>/<Equipment>/<Eff>),
single-source fan-out to both NodeIds, Organizes UNS->Raw, writes/HistoryRead via
either NodeId, multi-notifier alarms, EquipmentNodeIds retired. Flags the wire-
contract change as a pending ScadaBridge Data-Connection-Layer cutover.
Claude-Session: https://claude.ai/code/session_01LVneM3eh1UtJxEisFXgmox
Add components/secrets/ (SPEC, realized shared-contract, code-verified
current-state for OtOpcUa/mxaccessgw/ScadaBridge, GAPS adoption backlog) and
register the Secrets row in CLAUDE.md + components/README.md. The
ZB.MOM.WW.Secrets lib is built + published 0.1.2 + reference-consumer-proven
(HistorianGateway, live vs the wonder historian); per-app adoption is the
tracked follow-on.
Full *.IntegrationTests sweep done 2026-07-15: every driver + server suite GREEN,
zero OtOpcUa production regressions (all fixture/harness/test rot). PRs #446-#452.
Cross-repo highlights: Host harness LDAP swapped bitnami/openldap:2.6 -> GLAuth
(PR #451, own :3894 container, unifies the family on GLAuth); deploy-E2E 'emulated-
SQL timing' was misdiagnosed -> missing FK-parent seed + SQL/LDAP host overrides
(PR #452).
R2-04 T13/T15 (PR #444) close the last behavior-affecting round-2
verification gate (S4 primary-gate default-deny, live-verified on the
2-node rig + in-process delivery-path test). S7.Cli phrase-scan test
fixed (PR #443). Round-2 remainder is now infra-gated only.
The R2-06 live gate found the historian's Int1 analog-tag creation path
is server-degenerate; filed gateway-side as historiangw #11 (documented
as an explicit exclusion, PR #16), routed back as lmxopcua #441.
OtOpcUa's HistorianTypeMapper now maps Boolean → Int2. Updates the
OtOpcUa entry: master tip 7f79cd59 → 38e21df2, live gate 6/6.
HistorianGateway's Later tier (roadmap 16-21) landed on main via a coordinated
two-PR merge (origin/main @ 6a24a30); update the umbrella index from the stale
'open PRs pending merge' framing to merged. Only the 1.0.0 TLS-default flip remains.
ApiKeyRecord/ApiKeyListItem gain a nullable ExpiresUtc (NULL = never
expires); ApiKeyFailure gains KeyExpired. ApiKeyVerifier rejects a key
whose ExpiresUtc is at-or-before now (inclusive, injected clock), before
the secret comparison. CreateKeyAsync gets an expiresUtc overload; rotate
preserves existing expiry. SQLite schema bumps to v3: a nullable
expires_utc column, added to fresh DBs via CREATE and to existing v1/v2
DBs via an idempotent guarded ALTER — donor v2 gateway-auth.db upgrades
in place, no key invalidated. Version 0.1.3 -> 0.1.4 (not yet published;
nuget push is human-gated). Consumers (HistorianGateway, mxaccessgw) bump
to 0.1.4 to set/enforce expiry. 146 Auth.ApiKeys tests pass.
histsdk is now owned/imported (not vendored) — retire the 'vendored/re-vendor'
language; add the Client.Cli smoke CLI and record origin/main @ 93c6051 (the
repo-wide XML-doc sweep).
The ScadaBridge entry showed a `_git_` placeholder in the Repo column; set it
to the actual remote (gitea.dohertylan.com/dohertj2/ScadaBridge) to match the
peer rows. Keeps the umbrella index in sync with the repo, per the reciprocal
propagate-cross-repo-changes rule now recorded in ScadaBridge's CLAUDE.md.
Record the new HistorianGateway/docker/ local container deployment in the
umbrella inventory: HistorianGateway table row (real wonder datasources,
login + gRPC API-key auth disabled, runtime-only image / VPN / isolated
compose project) + a publish+compose recipe in the per-project commands.
OtOpcUa now consumes HistorianGateway as its sole historian read/write backend
(ZB.MOM.WW.HistorianGateway.Client; continuous historization + alarms), retiring
its bespoke Wonderware historian driver — merged to OtOpcUa master (PR #423).
Gateway follow-ups merged to historiangw main: WriteLiveValues UTC->server-local
timestamp fix + SendEvent Source_Object protocol-limitation doc (PR #6, pending.md
C4), the Plan-1 client lib + packable Contracts (PR #5), and a C2 cross-ref docs
PR (#4). Updated the OtOpcUa + HistorianGateway entries and the coupling note from
"independent / not a dependency" to "OtOpcUa depends on HistorianGateway".
Pre-existing OtOpcUa test failure tracked as lmxopcua issue #424.
Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii
Part C + C2 + the SQL-path ReadEvents are all merged to origin/main (gateway
fabab1a, histsdk f0a1b04). Event reads now work via Runtime.dbo.Events behind
RuntimeDb:EventReadsEnabled (the practical workaround for the C2 server gate).
Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii
Replaces the wrong "WCF not served on 2023 R2" (a test error vs the reverse
tunnel). WCF cert transport + auth reach the historian; rows are server-gated on
both transports (same per-connection gate as gRPC). Not client-fixable.
Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii
Event reads unavailable on the 2023 R2 historian over both transports — gRPC
retrieval-server-gated, legacy WCF net.tcp not served (live-disproven 2026-06-25).
Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii
Records the Part C minimal work landed on the gateway's local main (unpushed):
Int8/UInt8 live write types un-gated + live-proven; 2023 R2 gRPC interface-version
evidence recorded; UInt1 server-blocked (degenerate analog tag) → re-gated.
Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii
Joseph Doherty