ScriptOptions.WithReferences(Assembly[]) resolves each assembly through
MetadataReference.CreateFromFile, which does not cache: every call mints a
fresh AssemblyMetadata -> PEReader -> NativeHeapMemoryBlock holding an
unmanaged copy of the assembly metadata that nothing disposes. Building the
options per compile therefore leaked native memory permanently — invisible to
the GC, to gcdump and to the managed allocation counters, so the working set
grew while the GC heap did not.
Diagnosed from a live dump of a wonder-app-vd03 Site node: 2,885 MB working
set 78 min after a cold start, only 150 MB live GC heap, ~2,469 MB on the
default process heap across ~6,700 undisposed AssemblyMetadata instances
against 473 DLLs on disk.
Three sites, all hoisted to static readonly:
- SiteRuntime ScriptCompilationService (the dumped one)
- InboundAPI InboundScriptExecutor — same defect on the central node; method
compiles recur on every re-registration and revision change
- CentralUI ScriptAnalysisService — CreateFromFile per sandbox run
ScriptAnalysis RoslynScriptCompiler also builds options per call but draws
from the static ScriptTrustPolicy.DefaultReferences, so it mints no metadata
and is left alone.
Guarded by reference-equality on the artifact rather than by watching memory:
a bytes-watching test would be flaky, and the leak is native so the managed
counters cannot see it at all. The test is proven to fail before the fix.
This does NOT close the AddTemplateScript OOM — that path was shown twice not
to compile scripts. It explains how a long-running node reaches a native
memory state where a large allocation fails with gigabytes free, which is a
lead worth re-testing, not a closure.
CLAUDE.md points at .claude/skills/scadabridge-components/SKILL.md as the
component catalog and .claude/skills/scadabridge-cluster-ops/SKILL.md as the
source for the management URL, credentials and rebuild/redeploy commands — but
neither file was tracked, so a fresh clone got instructions referencing content
that was not there.
Only the skill files are added; .claude/settings.local.json and
scheduled_tasks.lock stay ignored via .gitignore as before.
Component-AuditLog.md has always required "we over-redact, never under-redact,
on configuration faults", but the body / SQL-parameter redactors violated it.
AuditRegexCache rejects a pattern that is malformed OR whose compile exceeds a
100 ms budget, caching the rejection for the process lifetime.
ScadaBridgeAuditRedactor then simply dropped the rejected pattern from its
redactor set and emitted the payload anyway — publishing precisely the values
the operator configured it to suppress, onto a row that looks entirely normal
downstream. Recovery required a process restart and the only signal was one
Warning line. The SQL path was worse: TryGetSqlParamRedactor returned a bare
false for both "no redactor configured for this connection" and "the configured
one will not compile", and CLAUDE.md records SQL parameter capture as on by
default.
Two changes:
1. Fail closed. A pattern that is CONFIGURED but unavailable now over-redacts
the whole payload and increments AuditRedactionFailure, reusing the existing
safety-net path. "Not configured at all" stays permissive — conflating those
two states is the actual defect, so both are pinned by tests.
2. Precompile off the hot path. The audit-log roadmap specifies patterns are
"precompiled at startup; rejected if compile takes >100ms"; the implementation
had drifted to compiling lazily on first event, which put a wall-clock budget
on a hot path under production load. RegexOptions.Compiled emits IL during
construction, so a busy node could blow the budget on a perfectly valid
pattern. Warm-up now runs at construction and on every options reload. The
residual window between a reload and its warm-up is safe because that path
now fails closed.
Warm-up deliberately does not fail the boot — an unusable pattern degrades the
node to over-redaction (safe, loud) rather than refusing to start. Reading
CurrentValue happens inside the warm-up try so an options provider that throws
still surfaces via Apply's over-redact path, not the constructor
(OuterCatch_OptionsThrows_NeverLeaks_AllSensitiveFieldsOverRedacted).
Also de-flakes GrpcCentralTransportTests.DeadlineExceeded_IsNotRetriedOnThePeer,
which is how this was found. It black-holed node A behind a 300 ms deadline, but
on a saturated machine the call could fail to even START — a genuinely-unsent
failure that IsConnectFailure correctly fails over on, so node B's ack arrived
instead of the expected Status.Failure. The test read as a flake while actually
reporting that its own premise had not held. Split in two: the hard rule now
injects an explicit DeadlineExceeded via a trailers-only response (deterministic,
load-independent), and a new BlackHoledNode_DoesNotHang covers the
deadline-is-actually-applied half with both nodes black-holed so no ack can
arrive down any path.
Verified: both fixes were confirmed to fail before they pass — reverting the
fail-closed guard fails exactly the 5 fail-closed tests while the 4 controls
still pass, and adding DeadlineExceeded to IsConnectFailure fails the rewritten
transport test. AuditLog 367/367, Host.Tests GrpcCentralTransport 8/8, solution
build clean. The previously-intermittent
Filter_PicksUp_NewBodyRedactor_OnConfigReload is green in a full sweep for the
first time.
Not addressed here, and noted on #35: the 100 ms wall-clock budget remains a
weak proxy for catastrophic backtracking (RegexOptions.Compiled defers JIT to
first match, so construction time measures the wrong thing), and a rejection is
still cached permanently. Both are now safe rather than dangerous, so they are
hardening rather than a leak.
Brings the two shared-library families that had fallen behind the Gitea feed up
to latest; the other 20 ZB.MOM.WW.* packages were already at feed-latest.
Auth 0.1.5 -> 0.2.0 is LDAP failover only: a new LdapOptions.FallbackServers
list, an endpoint walk with sticky preference in LdapAuthService, and per-hop
warning logs. Purely additive — with FallbackServers empty (the committed
default everywhere) the walk collapses to exactly one attempt, so behaviour is
unchanged until someone opts in. Nothing in Auth.ApiKeys changed, so there is
no key-store schema migration to sequence.
MxGateway 0.1.1 -> 0.2.0 matters more than the version gap suggests: upstream
records that four clients "had drifted onto the already-published 0.1.2/0.1.1
while their APIs kept changing underneath", so 0.1.1 was a stale label rather
than a stable point. Real deltas since: status/HRESULT reply validation made
conformant across clients (CLI-37/38), the ReplayGap reconnect sentinel
surfaced as a typed signal (CLI-15), and typed single-item command parity
(CLI-04/30).
The one change touching the write path — correlating OnWriteComplete onto plain
Write/Write2 replies — is server-side. Its proto diff is comment-only and the
behaviour lives in the Worker, so ScadaBridge gains the correlated statuses when
the gateway server is upgraded, not from this bump. No wire-contract break.
Verified: restore and full solution build clean (0 warnings, 0 errors); all 22
shared packages resolve to a single version each across all 57 projects, with no
split resolution. Security 181/181, InboundAPI 269/269, DataConnectionLayer
277/277 — the three suites that consume these packages directly.
Full-sweep residue is pre-existing and unrelated, confirmed 3/3 green in
isolation for the first two:
- GrpcCentralTransportTests.DeadlineExceeded_IsNotRetriedOnThePeer — a
saturated TestServer can fail to START the call, which IsConnectFailure
correctly treats as provably-unsent and fails over; the harness cannot
guarantee the DeadlineExceeded it means to exercise.
- AuditLogOptionsBindingTests.Filter_PicksUp_NewBodyRedactor_OnConfigReload —
not a reload race (the test is synchronous); it trips the 100 ms compile
budget in AuditRegexCache, which fails OPEN. Filed as its own issue.
- CentralUI Playwright 159/173 — login failures against the running rig,
which was built from the pre-bump image and so cannot be affected by this
change: 20 from the known SEC-36 GLAuth password rotation, 136 cascading
from the login throttle.
The rig still runs the previous libraries; it needs a docker/deploy.sh rebuild
to pick these up.
The 0.6.2 adoption set Secrets__SqlitePath inside x-secrets-hub-env, which only the
central pair and site-a merge — clustered replication is deliberately enabled on four
nodes so the default-OFF posture is proven side by side. site-b and site-c merge no
secrets anchor at all, so they fell back to the shipped default and crash-looped at
boot (exit 134, nine times each) while the other four came up clean.
Splitting the store path into its own anchor that all eight merge fixes it. The path
is not optional config: every node has a store, only four have a hub.
Also corrects the assumption behind deleting the key from appsettings.json. Measured
here: with a Secrets section present but no SqlitePath, the effective value is the
RELATIVE "secrets.db", which fails both validation rules — the per-user default does
not take over. A deployment must set it explicitly.
Verified live: all 8 nodes up, zero validation errors, stores at /data on the same
host directory as before, all four pairs converged to one cluster each through a
simultaneous recreate, central-a active / central-b standby, secrets hub sweeps
recovered after the expected boot race.
Records that the SharedSchema resolver is fully consumed at deploy, runtime and in
the UI value-entry forms, but has no authoring path: SchemaBuilder is the only
schema editor on all four surfaces and offers no library-reference option, so the
page creates entries nothing can point at. SchemaBuilder also collapses a $ref it
is shown to {"type":"string"} — currently unreachable, since no ref exists and
there is no UI or CLI to make one, but a trap for the first one authored.
Includes live usage counts (zero entries, zero refs, both environments) and the
resulting recommendation: leave dormant with a trigger, rather than finish or
delete it now.
0.6.x refuses a secret store whose path is relative or inside the content root,
because a store in the deployment directory is destroyed by an ordinary upgrade —
the failure that wiped the MxGateway API-key store on 2026-08-09 and read as an
auth outage rather than a deployment error.
The pin alone would not have protected this repo. Program.cs expands ${secret:}
before the host exists, composing secrets into a throwaway ServiceCollection with
no IHostEnvironment, so the guard would not run at the moment the migrator creates
the store. That composition now lives in SecretsRegistration with an explicit
content root — resolved to match what the host resolves later, including the
Windows-Service case where the pre-host CWD is still system32 — and is covered by
PreHostSecretsContentRootTests, verified by simulating the regression and
confirming it fails on the leftover file rather than on the exception.
The docker rig needed a fix too: /app/data is absolute but inside the container's
content root, so all 8 nodes would have failed to boot. Each node's data directory
is now mounted a second time at /data; same host directory, so existing stores
carry over untouched.
Verified: build clean, 29 test assemblies green (Playwright's 159 failures are the
pre-existing SEC-36 login baseline). Not yet deployed — the rig runs the old
config until someone redeploys.
Gating the detail modal/drawer on a held id rather than on the row resolving
(d14e0ee4) made it the surface's own job to clear that id when navigation
invalidates page-scoped state. ParkedMessages and ConfigurationAuditLog did not,
so paging away from an open row left the surface mounted on a notice it could
never recover from — reachable only by paging back.
The criterion is per-surface and comes down to whether the modal has content of
its own:
ParkedMessages, ConfigurationAuditLog — no keyed detail fetch; content resolves
from the loaded page alone. An entry paged out of view can never resolve again,
so these must clear on paging. They already cleared it on Search/OnSiteChanged
for the same reason, and clear _selectedIds on paging for the same reason
again; paging was simply missed.
NotificationReport, SiteCallsReport — fetch detail by id, so the modal still
shows real content after its row leaves the page. These deliberately do NOT
clear on paging and are unchanged.
Clearing on an explicit navigation action is user intent, not a resolve-driven
unmount, so this cannot reopen the handler-disposal race that d14e0ee4 closed.
PageScopedDetailStateTests covers all three paging entry points and records the
criterion so the next reader can tell why two surfaces clear and two do not. Run
against both clears reverted, all three fail; restored, all three pass.
CentralUI.Tests 994/994, solution build 0/0.
Also corrects two comments in ParkedMessages left stale by d14e0ee4 — they still
described the drawer as self-closing when a row stops resolving, which is the
behaviour that change deliberately removed.
The sweep's modal re-key (holding the row's id and re-resolving it, rather than
holding the record) also used that resolve as the modal's visibility gate. That
makes the modal's existence a function of list contents: any render where the
row is momentarily unresolvable unmounts the whole subtree and disposes every
event-handler id inside it, Close's included. A click already in flight against
a disposed handler makes the renderer throw GetRequiredEventBindingEntry during
DispatchEventAsync — which is how this surfaced, as an intermittent failure of
CloseButton_DismissesModal (989/990 on one run, green on re-run).
The record-held form made that structurally impossible: the modal existed
because the user opened it, and no list mutation could retract that. This
restores the property while keeping the re-key's actual benefit. Visibility now
gates on the held id; the resolve drives only content. An unresolvable row
degrades to an explicit notice and hides the row-scoped actions, while the frame
and Close stay mounted. Detail fetched by id still renders, so the user does not
lose the body they opened.
Applied to all four surfaces that shared the construction: NotificationReport,
ConfigurationAuditLog, ParkedMessages (offcanvas drawer) and SiteCallsReport.
Modal_StaysOpen_WhenItsRowLeavesThePage drops the opened row from the next query
and asserts the modal survives, keeps its fetched body, hides Retry/Discard, and
that Close still works. It was run against a deliberately restored defective
gate and failed there before passing here — a regression test that passes both
ways would be worthless against a race. 20 consecutive runs of the previously
flaky class: no failures. CentralUI.Tests 991/991, solution build 0/0.
The plan doc gains a section recording that the sweep was reported as
behaviour-preserving when it was not, and why the merge review missed it.
Applies the family-wide admin-UI cleanup playbook to the Central UI so the
Blazor surfaces stop diverging from the shared kit: buttons are grouped rather
than individually sized, long cell values are contained instead of widening
tables, and hard-coded colours give way to theme tokens.
The headline fix is that MainLayout passed Accent="#2f5fd0" to ThemeShell,
which the kit emits as an inline style on the shell root. Being a descendant of
<html>, it beat the [data-bs-theme="dark"] override for the entire app, so the
dark accent had never rendered. Declaring --accent in site.css :root instead
lets both schemes resolve; light is unchanged because the value already matched
the kit's light default.
Theme pins to 0.4.1, which upstreams the local .btn sizing block verbatim, so
that block is deleted here rather than duplicated. Verified byte-identical
before removal; the repo now declares no --bs-btn-* anywhere.
NOT purely cosmetic, contrary to the sweep's stated scope: four detail-modal
surfaces (NotificationReport, ConfigurationAuditLog, ParkedMessages,
SiteCallsReport) were additionally refactored from holding the selected record
to holding its id and re-resolving from the current page each render, with the
resolve doubling as the visibility gate. A background refresh that drops the
row now closes the modal instead of showing a stale snapshot. This is a
behaviour change and is called out rather than buried: a full-suite run turned
up one intermittent CentralUI failure, CloseButton_DismissesModal, whose stack
(GetRequiredEventBindingEntry during DispatchEventAsync) indicates the handler
was disposed between render and click — a window the previous field-held record
made structurally impossible. Treat the modal lifecycle here as unreviewed.
Build 0/0; suite green apart from that one intermittent failure.
The review asked for a regression test pinning DTD-prohibited parsing. Writing
it showed the premise was wrong: XDocument.Parse permits an internal DTD subset
and expands its entities, so a DOCTYPE-bearing response body parsed fine and the
guard did not exist (entity-expansion DoS on external input).
Parse now goes through XmlReader with DtdProcessing.Prohibit and a null
XmlResolver. The regression test feeds a DOCTYPE + ENTITY payload shaped as a
well-formed EWS error response, so it fails if DtdProcessing is ever loosened
rather than passing for the unrelated-XML reason.
On-prem Exchange 2013 EWS (Basic over HTTPS, live-probed) becomes a second
selectable email transport beside SMTP: additive SmtpConfiguration.Transport
discriminator, hand-rolled CreateItem SOAP sender (no SDK, BCC-only,
SendOnly), EwsErrorClassifier mirroring the SMTP transient/permanent split,
CLI/UI transport selector, fake-EWS unit stub + one-off live gate. The
pending O365 SMTP-OAuth2 verification (Q12) is superseded — real mail infra
is on-prem EWS. Design only; no implementation. Also gitignore the untracked
dev-credential file email_details.txt.
Root cause: dotnet runs as container PID 1 and Linux ignores default-action
signals sent to PID 1, so the runtime's unhandled-exception path (banner,
then abort() -> SIGABRT) could never terminate the process — it printed the
trace and spun the main thread at 100% CPU with the container `running`,
so `restart: unless-stopped` never fired. Reproduced deterministically:
same StartupValidator throw exits 134 under an init process and wedges
without one.
Two layers, each covering the other's gap:
- Program.cs registers an AppDomain.UnhandledException handler before the
first statement that can throw: prints the trace, best-effort flushes
Serilog, Environment.Exit(134) — exit() is a syscall PID 1 CAN perform,
134 preserves the 128+SIGABRT crash code, and it covers every thread,
not just the boot window. It cannot fire under WebApplicationFactory
(the test host catches entry-point exceptions), so the designed
boot-refusal exceptions still propagate to tests unchanged.
- docker-compose: init: true on all 8 nodes for the crash paths that
bypass the managed event (Environment.FailFast, runtime-internal aborts).
The CoordinatedShutdown no-Environment.Exit guard gains a precise carve-out
(exactly one call, only inside the handler); Environment.Exit still fires
the CLR shutdown hook Akka binds via run-by-clr-shutdown-hook = on, so the
crash path skips nothing abort() kept. New pin test keeps the handler ahead
of the configuration build.
Live-verified on the rig image: crash now yields Exited (134) +
RestartCount climbing under `unless-stopped`, trace intact, with and
without init; full 8-node rig redeployed healthy with docker-init as PID 1.
Closes#34.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
0.5.1 (scadaproj 31ca940) fixes the virgin-DB concurrent migrator race this gate
found: the retry filter now covers 2714/1913/2627 alongside deadlock 1205. Rig
rebuilt on the bumped pins and the exact trigger re-drilled — ZbSecretsHub dropped
and recreated empty, both centrals started in one docker invocation — and both
booted clean in the same second (schema provisioned once, /health/ready 200 both,
no 2714, no wedge), where 0.5.0 crashed central-a under identical conditions.
Convergence re-smoked on the new image (13 s, decrypt-verified). Gate doc amended:
defect 1 disposition FIXED in 0.5.1 with the re-drill evidence; defect 2
(pre-Serilog wedge) remains open pending its own issue.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Rig config: central pair gains Secrets__SqlServer__ConnectionString (dedicated
ZbSecretsHub database on the existing scadabridge-mssql, dev credentials); site-a
pair gains Secrets__GrpcHub__FallbackEndpoints__0 = central-b. Gate doc records
5/5 PASS (parity-by-construction, both-direction failover incl. recovered-primary
wrap, delete-while-follower-offline with no resurrection, Layer-A expander
provably reading the shared store via a stale-SQLite decoy discrimination, and
fail-closed negatives), discharging the Program.cs SQL-expander offline-test
residual, plus two defects documented NOT patched: the SqlServer migrator's
concurrent virgin-DB CREATE SCHEMA race (error 2714 not in the retry filter) and
the Host's pre-Serilog crash path wedging at 100% CPU instead of exiting.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Pin all five ZB.MOM.WW.Secrets* packages 0.4.1 -> 0.5.0, which brings
SecretsGrpcHubClientOptions.FallbackEndpoints and the package's internal
FailoverSecretsHubReader. A site whose GrpcHub section lists fallback
endpoints now fails a sweep over to the next central instead of stalling
on a downed primary - safe ONLY because both central nodes serve one
shared SQL secret store (scadaproj#4), so either hub answers with the
same manifest; the appsettings comments say so and warn against listing
endpoints backed by independent stores.
appsettings.json gains "FallbackEndpoints": [] with a _fallbackEndpoints
comment, and the _endpoint note's single-endpoint-stall caveat is scoped
to the empty-list case it now only applies to.
Wiring pins (red first on 0.4.1): site + Grpc + one fallback resolves
ISecretsHubReader to FailoverSecretsHubReader with the
"zb-secrets-grpc-hub:fallback:0" keyed channel present; zero fallbacks
keeps the plain GrpcSecretsHubClient and no fallback channel - the
pre-0.5.0 container shape byte-identical.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
The expander comment now records that a SQL outage at central boot exits
pre-Serilog with a bare stderr trace (honest, restart-retryable, unenriched)
and why the role-reading predicate deliberately stays in Program.cs rather
than SecretsRegistration (that class refuses config-read roles by design).
EnsureCentralSharedStoreConnectionString gains the file's standard
ThrowIfNull. The site-purity scan's doc no longer overclaims: two
factory-lambda descriptors evade it individually; the pin holds because
three concrete-type registrations from the same call cannot.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
The blank/${secret:} pre-checks lived only in AddScadaBridgeSecrets, but on
a real central boot the Layer-A expander runs FIRST and would hand a bad
value to SqlConnection, burying the designed message under a generic
'initialization string' format error. Extracted both checks into
EnsureCentralSharedStoreConnectionString — one definition, called by the
expander (earliest point) and by registration (covers embedded/test
composition) — same single-source lesson as the UsesGrpcHub predicate.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
In Secrets:Replication:Mode=Grpc a central node's ISecretStore is now the
shared SQL-Server store (AddZbSecretsSqlServerStore) instead of a per-node
local SQLite store. Both central hubs read and write ONE copy of every row,
so they serve identical manifests by construction — the 2026-08-07 live gate
observed central-b answering an authenticated GetManifest with an EMPTY
manifest while central-a held every secret, which would turn site-side hub
failover into a silent convergence stop.
- SecretsRegistration: two fail-closed pre-checks before any registration on
the central+Grpc path — a blank Secrets:SqlServer:ConnectionString throws
naming the key (an independent store per central node is the recorded
defect), and a value containing ${secret: throws naming the bootstrap
circularity (the expander needs this store to resolve references). Site
registrations are byte-identical to before; SqlServer mode and
replication-off are untouched.
- Program.cs Layer-A expander follows the store swap: central+Grpc with a
non-blank connection string migrates and resolves pre-host ${secret:}
references through the shared SQL store, so expanded values can never
diverge from what the running node serves. Every other case keeps the
SQLite path unchanged; blank-connstr central deliberately falls through so
the clear AddScadaBridgeSecrets message is the one that fails the boot.
- appsettings.json: Secrets:SqlServer _comment now documents the Grpc-mode
central requirement (literal/env value only, sites leave it empty).
- SecretsReplicationWiringTests: +5 pins (shared store resolves, blank and
${secret:} connstrings fail naming the key, sites-have-no-SqlServer-types
descriptor sweep), central fixtures carry the now-required connstr.
Full suite green (7,474 passed across 30 projects, 0 warnings); the two
failures are pre-existing and unrelated: the Playwright live-rig suite fails
identically on unmodified main (cluster not running), and
GrpcCentralTransportTests.DeadlineExceeded_IsNotRetriedOnThePeer is a timing
flake that passes 3/3 in isolation and 470/470 on the first run of this code.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Second pass on the 0.4.1 image, from a clean start with zero denial warnings on
central-a. Both auth negatives are still refused with a byte-identical
Unauthenticated status and detail - 0.4.1 changed what the server writes down,
not what a caller sees - and the no-bearer call now produces a WRN in the same
second it is refused, with the cause attributed. No interval has to elapse for
the first denial to be visible, which is the whole point: a follower with a
mis-rotated token is refused on every sweep, and central now says so immediately.
The rate limit was proven to DEFER rather than drop, not assumed to. The
wrong-bearer call one second later fell inside the 60 s window and produced no
line of its own; the window was waited out and one further wrong-bearer call
issued, whose summary reported TWO wrong-credential denials - the deferred one
plus the new one. Summing the two lines gives exactly the three negatives issued,
correctly attributed by cause. N is a per-window delta, so a reader must sum the
lines rather than quote the last one; that is recorded as a follow-up because it
is the kind of thing an alert gets wrong.
Log hygiene re-run fleet-wide and widened: all eight nodes' docker logs and every
on-disk Serilog file were grepped for the dev token, the dev KEK, all three secret
plaintexts AND both wrong tokens the negatives presented. Zero hits everywhere.
The presented-credential check is deliberate - echoing a rejected credential back
into a log is its own leak and a free oracle, and the new warning counts denials
by cause without carrying any credential material.
Checks 2 and 3 were not repeated: 0.4.1 touches the hub's denial logging and
nothing else - no wire change, no store change, no sweep change. Convergence was
re-smoked instead so the new image is not merely assumed to replicate: a fresh
secret reached both followers in 17 s, byte-identical and decrypt-verified on
both, and the first pass's live secret and tombstone survived the image swap
unchanged on all three nodes.
The first-pass FAIL evidence is kept intact rather than overwritten. The fix only
means anything against the failure it answers, and a gate doc that shows only the
green run cannot be audited.
Residuals stand as recorded: the hub client dials a single endpoint with no
failover, and the central pair does not converge with itself - central-b answered
an authenticated GetManifest with an empty manifest for the whole run. Those are
one question, not two.
4/4. Merging.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
The 2026-08-07 live gate's check 4 failed one clause of three: both auth
negatives were denied with a byte-identical Unauthenticated status and no token
or plaintext reached any log, but the hub recorded a denial only at Information
level, via Grpc.AspNetCore.Server rather than the gate itself. A follower whose
token was mis-rotated would therefore stop converging while central showed
nothing above INF.
That was a property of the library, not of this branch, so it was fixed there and
shipped as 0.4.1 (scadaproj main c86cead): SecretsHubAuthInterceptor now emits a
rate-limited Warning summarising each denial window, breaking the count down by
cause, with the first denial after startup or a quiet window warning immediately
so a single probe is never silent. Wire behaviour is untouched - the denial is
still one uniform Unauthenticated with one detail, and still carries no token
material - so this is additive on the server's diagnostics only and nothing a
follower observes changes.
All five ZB.MOM.WW.Secrets* pins move together. Splitting them is not an option
worth having: Abstractions carries the StoredSecret shape the Grpc wire mirror is
written against, so a mixed set is a silent structural mismatch rather than a
build error.
Build clean at 0 warnings; the secrets wiring + hub-mapping pins are 31/31.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Rig config enabling the pull-only hub on the docker cluster (central pair hosts,
site-a pair follows; site-b and site-c deliberately left off so the default-OFF
posture is proven side by side), plus the gate record.
Checks 1-3 PASS. A central write reaches both site-a nodes in 5 s with a
byte-identical ciphertext row and decrypts correctly on both; a site pair boots
and serves its full last-known-good store with the entire central pair stopped,
warning once per interval without crashing, and resumes convergence unaided when
central returns; a tombstone propagates in under 9 s and survives a pair restart
with central up and sweeping, without resurrecting.
Check 4 FAILS one clause of three. Both auth negatives - absent bearer and wrong
bearer - are denied with a byte-identical Unauthenticated status and detail, and
a fleet-wide grep of all eight nodes' docker logs and on-disk Serilog files finds
ZERO occurrences of the dev token, the dev KEK or either plaintext. But the
criterion also asks for a server-side WARNING on denial, and there is none: the
only record is one Information line per call from Grpc.AspNetCore.Server, because
SecretsHubAuthInterceptor deliberately logs nothing on a denial and warns only
when no token is configured at all. That is a property of the 0.4.0 library, not
of this branch, and it is not patched here - a host-side interceptor would
contradict a documented library decision at the wrong layer and put an unbounded
log write on an unauthenticated endpoint.
The merge condition is 4/4, so this branch is NOT merged. The library's denial
logging is the only thing between this result and a merge.
Two residuals worth carrying: the hub client dials a single endpoint and does not
fail over (observed live, and contrasted against CentralGrpcEndpoints failing over
on the same node in the same minute), and the central pair does not converge with
itself - central-b answered an authenticated GetManifest with an empty manifest
for the whole run while central-a held both secrets. Together those make "which
central node is authoritative for secrets" one question, not two.
Rig config notes: Secrets__SqlitePath points at /app/data because the appsettings
default resolves to /app inside the image's writable layer, so the central pair
gained the per-node data volume the site pairs already had. All values are
dev-only and committed under the same exception the mesh PSKs already use.
Also recorded: a gate-METHOD defect. Seeding the bind-mounted store from the macOS
host is not coherent with the running container - the row was visible to the host
and to a fresh container but never to the node, and was lost outright on restart.
Every store access was redone from a throwaway container. The failure mode is a
convincing false negative that looks exactly like a broken hub.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
AddScadaBridgeSecrets now branches on UsesGrpcHub instead of re-deriving the
same condition, so the doc's single-predicate claim is enforced rather than
aspirational; the role split is a switch with a default-throw so a future
third role must choose its hub half explicitly.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Extends the existing build-and-resolve wiring suite rather than asserting over
ServiceDescriptors, for the reason that file already documents: a registration
can look correct as a descriptor list and still fail on first resolve.
Role separation is pinned in both directions — central registers the hub's
fail-closed interceptor and NOT the sweep, a site registers the sweep and NOT
the hub — because only one of those is a security failure and testing the
happy half would not catch it. The sweep is asserted by resolving IHostedService,
which builds its whole graph (reader, keyed channel, local store) and is where
a forgotten AddZbSecrets would surface; GrpcSecretSyncService is internal to
the package, so it is matched by assembly + type name the way the SqlServer
replicator's services already are.
SqlServer mode gets regression pins with the mode key both unset and named
explicitly, plus one asserting it stays role-agnostic — both nodes sync
bidirectionally against the same database, and the role parameter added for the
hub must not have quietly changed that.
The mapping tests assert over the app's real endpoint data sources, on the WIRE
route (/zb.mom.ww.secrets.hub.v1.SecretsHub/...) rather than the C# type, since
the route is what a follower addresses. One of them pins exactly the two READ
methods: pull-only is a property of the contract, and this is where a future
package version growing a write RPC would become visible instead of silently
opening a path for a site to overwrite central.
Numeric mode values get their own test. Enum.TryParse accepts any integer,
including ones outside the enum, so "7" would otherwise select a mode that does
not exist and fall through to the SqlServer branch.
Verified red-first by mutation on the finished implementation: swapping the two
role branches reds 8 (both role pins, both fail-closed pins, both mapping
pins); deleting the UsesGrpcHub check in the map extension reds exactly the two
"maps no hub endpoint" cases — the unauthenticated-hub scenario; dropping
Enum.IsDefined and letting UsesGrpcHub ignore Enabled reds the out-of-range
value and the flag-off-with-full-hub-config cases. 31/31 green restored.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
scadaproj#3: the production secrets topology is central hosting a pull-only
gRPC hub with site nodes sweeping it. The SqlServer replicator stays in the
codebase and keeps working exactly as it did, but it is not the production
path — it needs every site node to hold a connection string to central's
database, which breaks ScadaBridge's standing rule that sites talk to central,
not to central's DB.
Selection is a new Secrets:Replication:Mode key alongside the existing
Secrets:Replication:Enabled flag. Absent or blank means SqlServer, so an
existing configuration that sets only Enabled behaves identically; an
unrecognised value is refused at startup naming the key, and refused whenever
it is present rather than only when replication is on — a typo should fail the
boot that introduced it, not some later boot that flips an unrelated flag.
Which HALF a node composes is a parameter, not a config key. Both composition
roots already know statically which they are (Program.cs is central,
SiteServiceRegistration is a site), and a role read from configuration is a
role that can be got wrong in the one direction that matters: a site hosting
the hub would serve central's whole secret inventory from inside the site
network to anything holding the shared token.
The hub is mapped onto the EXISTING central h2c control-plane listener
(ScadaBridge:Node:CentralGrpcPort, default 8083) beside CentralControlService
— the same listener and the same addressing convention sites already use, and
the same shape as the site's LocalDb sync endpoint sharing its gRPC port: two
disjoint service prefixes, two independent fail-closed gates. The
CentralControlAuthInterceptor on AddGrpc is prefix-scoped and passes hub calls
through; the hub's own SecretsHubAuthInterceptor, attached per-service by
AddZbSecretsGrpcHub, gates them on Secrets:GrpcHub:BearerToken.
Registration and mapping share one predicate (UsesGrpcHub) deliberately.
Mapping without registering would map a hub whose interceptor was never
attached — an anonymous endpoint serving every secret central holds, on a node
that looks completely healthy — so the two must not be able to drift.
gRPC mode fails CLOSED where SqlServer mode degrades: a missing endpoint or
bearer token is a startup failure, not a warning plus a local-only store. The
degraded outcome is precisely what the hub exists to prevent (a site quietly
serving secrets that never converge), and the package's own errors name the
exact key, so they are left unwrapped.
Templates are default-OFF: Enabled stays false, Mode stays SqlServer, and
Secrets:GrpcHub ships with an empty BearerToken and Endpoint. Empty is
fail-closed, not open. The token is documented as appsettings/env only — never
a ${secret:} reference, since resolving one is what the hub exists to make
possible — the same bootstrap rule the mesh pre-shared keys follow.
Known asymmetry, recorded in the template: CentralGrpcEndpoints is a LIST that
fails over across the central pair, but the hub client dials a SINGLE endpoint,
so a sweep against a downed central node stalls instead of failing over. That
is survivable — the sweep is best-effort and the site keeps serving its full
local last-known-good store — but secrets stop converging until that node is
back.
Claude-Session: https://claude.ai/code/session_014WNM4vjoVksyyBraTXSZE1
Joseph Doherty