ZbOtelCollector (otelcol-contrib 0.162.0, NSSM, NT SERVICE\ZbOtelCollector, every
listener on 127.0.0.1, file + Prometheus exporters, size-capped rotation) on DEV
vd03 and QA vq03/vq04. OtOpcUa, ScadaBridge Central + Site, MxAccessGateway and
HistorianGateway export OTLP to http://127.0.0.1:4317 via two service-env entries
each; rolling restarts, standby-first on the QA pairs. Overview has no exporter
binding and is unchanged. Config, install/switch/verify scripts, runbook and the
dated rollout record under infra/otel-collector/wonder/; evidence doc, GAPS,
index row and deployment.md updated.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Metrics, traces and logs from both rig centrals with node.role=Central and site.id=central;
deployments.applied and inbound_api.requests arrive. Follow-ups scadaproj#56 (log-export
self-spans) and ScadaBridge#134 (rig image lacks git SHA).
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Evidence doc gains the #55 section (root cause of no logs, 0.3.0 contents + SHA256,
per-app commits, collector output per app: service.version, site.id, node.role and
log records on OtOpcUa, HistorianGateway, ScadaBridge Site and MxAccessGateway).
GAPS, current-state, SPEC, the Telemetry CLAUDE.md and the index row updated;
infra/otel-collector gains otelidentity.py (Resource per signal + log samples).
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
ILdapGroupLookup.GetUserGroupsAsync(username, ct) re-reads a known user's
groups with the service account: same extraction, canonicalisation, endpoint
walk (FallbackServers), timeouts and TLS policy as AuthenticateAsync, via one
shared WalkEndpoints<T>. Explicit statuses: Found, UserNotFound, UserDisabled
(AD userAccountControl), AmbiguousUser, DirectoryUnavailable, NotSupported
(BindMode=User / no service account / foreign ILdapAuthService),
ProviderDisabled. AddZbLdapAuth registers it as the same singleton.
0.5.0 is the first published version carrying 0.4.2-0.4.4; scadaproj#51 was
live-verified against the NAM AD (24/24, identity check mutation-proven).
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Both packages published to the feed 2026-11-12, packed from 01c6dc1 (the 0.2.0
code + corrected RepositoryUrl, before the 0.2.1 commit); SHA256 of the feed
download matches the local pack. ScadaBridge switched to AddZbSerilog with the
MinimumLevelGovernor + ConfigureLogger hooks (main 2d2e9f35) and deleted its
LoggerConfigurationFactory. Updates the Observability row, Telemetry/CLAUDE.md
and components/observability/GAPS.md.
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
SPEC 2.4 named OtOpcUa and MxGateway as users; neither is (verified against every
app's default branch; MxGateway worker IPC is named pipes, OtOpcUa deliberately
does not gate readiness on gateway channels). Keep the check (removal is a breaking
API change; small, tested building block), mark it as no-current-consumers in SPEC
2.4/6/7, both READMEs, the shared-contract doc and GAPS.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Audited the registration and mapping code of each app at its origin default branch: OtOpcUa
c904472a, MxAccessGateway de86f8b8, ScadaBridge c7d868c5, HistorianGateway 4d6a9ad (all on Health
0.3.0, all MapZbHealth with default paths: /healthz runs nothing, /health/ready and /health/active
by tag). GAPS.md now carries the verified per-endpoint coverage matrix and a SPEC-by-app table;
each current-state doc lists exactly what each endpoint runs, and HistorianGateway gets its first
current-state doc.
Recorded: mxaccessgw#206 (worker-executable + galaxy-sql readiness checks, on /health/ready) is
merged; historiangw#87 (session-pool check latches 503 until restart) is still open at 4d6a9ad.
Deferred P2 downstream probes are now per-app; ScadaBridge IActiveNodeGate unification stays
deferred (shared rule already used, only the interface is local).
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Adds an additive AddZbSerilog(builder, configure, configureSerilog) overload
taking ZbSerilogOptions:
- MinimumLevelGovernor: Func<IConfiguration, LogEventLevel?> applied with
MinimumLevel.Is AFTER ReadFrom.Configuration, so an app-owned key (e.g.
ScadaBridge:Logging:MinimumLevel) outranks Serilog:MinimumLevel; null defers
to config; Serilog:MinimumLevel:Override:* entries still apply.
- ConfigureLogger: runs last, for app filters/enrichers/sinks (ScadaBridge's
standby active-node exclusion).
The two-argument overload is unchanged. 8 new tests (4 fail with the hook
neutralized). Telemetry + Telemetry.Serilog 0.1.0 -> 0.2.0, not published.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Adds the family backlog standard to the root index (open work = Gitea issues,
shared label set, BTDB excluded) and converts components/*/GAPS, hosted-lib
CLAUDE.md notes, code-reviews, infra/glauth and plan docs to banner + issue
pointers (scadaproj #5-#19, ScadaBridge#79).
Program closeout: scadaproj#4 CLOSED (comment issuecomment-5751) — ScadaBridge's
central pair now runs ONE shared SQL-Server store in Grpc mode (0.5.0
FallbackEndpoints, 0.5.1 virgin-DB migrator race fix; ScadaBridge @ e697477c),
live-gated PASS 5/5 + re-drill. The pre-Serilog container-wedge defect the gate
reproduced twice is filed as ScadaBridge#34 (pre-existing, out of program scope).
- clustered-secrets.md: central pair = one shared SQL store (connstr literal/env,
never ${secret:}); rewrap = ONE central run + both nodes of each site pair;
FallbackEndpoints same-store-only rule; 0.5.1 minimum for central pairs
- components/secrets/GAPS.md: resolution section appended; stale wait-on-#4 struck
- CLAUDE.md Secrets row: #4 CLOSED, family 0.5.1, gate + residual recorded
- tasks.json: all 6 tasks completed
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Task 9 (closeout) of the Secrets gRPC-hub program. Bookkeeping only; no code.
- clustered-secrets.md, Production enablement: the gRPC hub is no longer
pending. It is BUILT, SHIPPED (0.4.1) and LIVE-GATED PASS 4/4 (ScadaBridge
main @ f6c3f7c5, docs/plans/2026-08-07-secrets-grpc-hub-live-gate.md), so the
"until that ships, ScadaBridge replication stays off" line is replaced with
the truth: the code path is merged and gated, and production enablement is a
separate operator action behind a still-default-off flag. OtOpcUa's
pair-local topology is recorded as live-gated (lmxopcua master @ 1bd79c8e)
rather than pending re-gate. Rotation fan-out corrected: on ScadaBridge it is
BOTH central nodes plus both nodes of every site pair, because each central
node has its own independent store.
- components/secrets/GAPS.md: new "gRPC hub delivered" section recording the
0.4.0/0.4.1 publish, both gates with SHAs and gate-doc paths, and the
residual. Existing narrative left intact and amended, not rewritten.
- CLAUDE.md Secrets row: lib version 0.3.0 -> 0.4.1 and the app-pin claim
corrected (ScadaBridge pins 0.4.1 for the Grpc replicator; the other three
still pin 0.2.3); the "production topologies DECIDED" region now records
delivery, both gates, default-off, and the residual.
- Plan ledger: task 9 completed.
Residual carried forward as scadaproj#4 — ScadaBridge's central pair does not
converge with itself (central-b served an authenticated but EMPTY manifest all
gate while central-a held every secret) and the site client dials a single
endpoint; these are one design question, since failover to an empty hub would
succeed and silently stop convergence. scadaproj#3 is closed.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Design doc + runbook Production-enablement section + GAPS/index updates.
The previously-untracked production-KEK decision is now recorded: one
fleet-wide KEK via appsettings/env (never committed); ScadaBridge rejects
the SqlServer hub on topology grounds (sites talk to central, never the
DB) in favor of a new ZB.MOM.WW.Secrets.Replicator.Grpc package —
pull-only, updates originate at central only, by design.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Rewrites the active-node sections of the spec and shared contract, which
described the leader-based design that both consumers rejected.
- SPEC §2.3 replaced: the rule, the ordered RolePreference, why a node holding
none of the preferred roles is Unhealthy rather than Healthy, and the data
payload. §4 (IActiveNodeGate) and the tier table follow.
- SPEC §6 marked historical: it still tells adopters to use
ActiveNodeHealthCheck(role: "admin") and AkkaActiveNodeGate, which is the
advice that produced the defect. Kept for provenance, annotated so nobody
follows it again.
- GAPS: gaps L1/L2 recorded as closed the wrong way, reopened, then closed at
0.3.0 — including the part worth remembering, that both apps adopted the
shared check as specified, hit it live, and reverted to private copies, so the
surface reached zero consumers by being avoided rather than unused.
- Index rows updated with the consumer-visible changes and both app branches.
Closes the defect in scadaproj#1. The hang was never Akka: the package's DI
wiring closed a circular singleton dependency the container cannot see through
factory lambdas — ISecretStore (ReplicatingSecretStore decorator) ->
ISecretReplicator -> SecretReplicationActorProvider -> ISecretCacheInvalidator
-> DefaultSecretResolver -> ISecretStore. Resolution recurses around the loop
until MS.DI's StackGuard hops it onto a fresh thread-pool thread, which then
blocks forever on a singleton call-site lock the first thread still holds:
a silent permanent hang instead of a stack overflow. Managed stacks from
dotnet-dump show the repeating cycle and both parked threads; both candidate
causes in the issue (DistributedPubSub.Get vs the Lazy lock, missing
Akka.Cluster.Tools HOCON) are disproven — the actor constructor was never
reached, and the deadlock reproduces on a single non-clustered node.
Fix: defer the one cycle-closing edge. The provider now gets a
DeferredSecretCacheInvalidator that resolves the real invalidator on first
eviction — which only happens when a replicated row is applied, strictly after
graph resolution. Severing the edge instead is wrong: a null-invalidator
experiment ran the live gate at 5/6, with deleted secrets still resolving on
the peer. The SqlServer package never had the cycle (its replicator chain
never touches the invalidator), which is why the hub gate always passed.
Verified: live 2-node convergence gate now 6/6 (was: infinite hang), including
the delete-visibility check that proves the deferred invalidator really evicts.
New HostedProcessResolutionTests builds the graph as a host does (container-
registered ActorSystem, hosted services, watchdogged resolves) and fails on
0.2.1; DeferredSecretCacheInvalidatorTests pins the wrapper contract. Full
suite 180 passed / 0 failed / 15 skipped (env-gated live SQL).
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
scadaproj#1 holds the defect (the library lives here); lmxopcua#482 is the
consumer-side tracker keeping Secrets:Replication:Enabled=false until it is fixed.
SQL-Server hub topology (ScadaBridge) is live-validated against a real SQL Server
2022 with two nodes: write->resolve via hub, tombstone propagation without
resurrection, bidirectional sweep, partitioned-node local resolution (site
autonomy - the reason hub mode was chosen), and wrong-KEK fail-closed via
SecretDecryptionException.
Akka peer-to-peer topology (OtOpcUa) FAILS the gate: resolving ISecretReplicator
hangs indefinitely in a real clustered process. Reproduced twice on a healthy
2-node cluster; isolated to constructing AkkaSecretReplicator -> ActorRef ->
system.ActorOf, NOT nested DI resolution. The library's own 2-node suite passes,
so the trigger is creation through DI in a hosted process. Root cause not yet
identified.
Impact: OtOpcUa would hang at startup if replication were enabled, since the
startup hook resolves ISecretStore. Harmless today only because the flag defaults
false. Do NOT enable Secrets:Replication:Enabled in OtOpcUa.
The gate did its job - it caught a blocking defect that every offline suite,
including the library's own real-cluster tests, had passed over.
- All 4 apps on 0.2.1 (local branches, unpushed).
- Records the corrected security story: the version bump closed NO advisory (all four
repos already resolved patched 2.1.12); the real live vulnerability was in ScadaBridge,
masked by a NuGetAuditSuppress, and was fixed by separate work.
- Records the upstream 0.2.0 inert-Akka-replicator defect, its root cause (DI extensions
with no container-building test - third instance of that class), and the 0.2.1 fix.
- Marks clustered topology WIRED-but-default-OFF and explicitly NOT live-validated;
Task 9 remains open and the topology is not 'adopted' until it passes.
I dismissed this finding from the code review as a false positive, reasoning
that Akka's ActorBase caches Self in a field and that three passing
anti-entropy tests traverse the path. Both premises were wrong. Self resolves
through Context, which is [ThreadStatic], and throws NotSupportedException once
a continuation resumes on a thread-pool thread.
The tests passed because a local SQLite store usually completes await
SYNCHRONOUSLY, so the continuation stayed on the mailbox thread and the context
was still intact. Correctness therefore depended on store latency and
thread-pool timing: green here, broken under a slower or contended store, with
the only symptom a per-peer warning every announce interval while nodes
silently stopped converging. The live-broadcast fast path masked it further —
only the anti-entropy repair path was dead.
Captures self on the actor thread and passes it in. Adds
GenuinelyAsyncSecretStore to force the async path, a regression test that fails
on the unfixed code (20s timeout) and passes in 3s after, and
ActorContextAfterAwaitTests pinning the underlying Akka behaviour so the wrong
assumption cannot be made again. Audited every remaining Self/Sender access in
the actor.
170 pass offline / 184 with the live SQL suite / 1 skip / 0 warnings.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Secrets were per-node SQLite, so a secret written on one node was invisible to
the rest of a cluster. G-7's design resolved the "shared SQL store vs Akka
replicator" fork to build only the former; both are built here so the choice is
a deployment decision (availability vs partition tolerance) rather than a
library limitation.
Two new packages — ZB.MOM.WW.Secrets.Replicator.SqlServer (shared store, plus a
local-store-with-hub mode) and .Replicator.AkkaDotNet (peer-to-peer over
distributed pub/sub). Core gains ISecretsStoreMigrator, one shared
SecretLastWriterWins predicate so no two stores can disagree on a tie, the
transport-agnostic reconciler, and ReplicatingSecretStore — which closes a real
gap: nothing had ever called ISecretReplicator.PublishAsync, so the seam was
inert and local writes would not have propagated at all.
Verified 182 pass / 1 skip / 0 warnings, including 15 live tests against a real
SQL Server 2022 (the SQLite suite ported case-for-case, so any behavioural
divergence between the stores fails) and a 9-test in-process 2-node Akka
cluster over real remoting. A post-build review caught six defects, all fixed
and now covered: both replication modes could not resolve from the container
(no test had built one), an unbounded fetch that broke past SQL Server's
2100-parameter cap, a poison row that aborted the rest of its batch forever,
Enum.Parse on peer input that could restart the actor in a loop, null crypto
blobs crossing the trust boundary, and a silently dropped pull-read failure.
Packed at 0.2.0 and vulnerability-scanned clean; not yet published to the feed.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts