Closes WP1.2 of the arch-review remediation plan (finding #2, High):
SqliteAuditWriterOptions.DatabasePath defaulted to CWD-relative "auditlog.db",
which on the docker rig resolves onto the container's ephemeral overlayfs
(not the mounted /app/data volume), silently discarding the pending audit
forward-state backlog on every recreate; nothing in docker/ or docker-env2/
overrode it; FlushIntervalMs was validated but never read by the writer loop
(one commit per event even at trickle rate); and no PRAGMA synchronous was
set (SQLite's FULL default fsyncs every commit).
- DatabasePath now has no default (mirrors ZB.MOM.WW.LocalDb's LocalDbOptions.Path)
and is required pre-host for Site nodes only, via a new StartupValidator raw-config
check (top-level "AuditLog:SiteWriter:DatabasePath", NOT nested under ScadaBridge:
AddAuditLog binds that section off the configuration root). SqliteAuditWriterOptionsValidator
deliberately does NOT check DatabasePath itself, because AddAuditLog runs its
ValidateOnStart on both Central and Site composition roots but only Site nodes
ever resolve the writer — checking it there would fail Central's boot too.
- All 8 site-node appsettings under docker/ and docker-env2/ now set
AuditLog:SiteWriter:DatabasePath to /app/data/auditlog.db (mounted volume,
survives container recreate, same convention as LocalDb:Path); the local-dev
base appsettings.Site.json sets ./data/auditlog.db to match.
- The writer loop now honors FlushIntervalMs: after draining the immediately
available burst, it keeps the transaction open (bounded by FlushIntervalMs
from the first event) waiting for more trickle-rate events before committing,
instead of flushing (and fsyncing) per event.
- PRAGMA synchronous = NORMAL on the write connection — audit is best-effort by
design (CLAUDE.md: "Audit-write failure NEVER aborts the user-facing action"),
so NORMAL's narrower power-loss window is an acceptable trade for far fewer
fsyncs; WAL mode still guarantees no corruption.
- Tests: StartupValidator site-required/blank/central-exempt cases; writer
trickle-load single-transaction coalescing + beyond-interval separate-transaction
regression (new FlushCountForTests seam); options-validator doc updates reflecting
the moved responsibility. Full suite runs green: AuditLog.Tests 368/368,
Host.Tests 480/480.
One-time migration note: the existing container-local auditlog.db (wherever it
landed under CWD) is abandoned by this change, not migrated — already-forwarded
rows are safe centrally (AuditLog is the durable copy), and any still-Pending
rows on the abandoned path are lost once. This is the exact bug being fixed, not
a new loss: those rows were already living outside the mounted volume and would
not have survived the next container recreate regardless. Cross-reference
docs/known-issues/2026-07-20-cached-telemetry-drain-hot-loop.md, which this
placement bug caused.
SiteLocalDbSetup.OnReady registered all ten replicated tables
unconditionally, so a deliberately unreplicated site node (site-b and
site-c on the rig) carried the full 30-trigger CDC set forever. Every
write to those tables paid two extra INSERTs plus a json_object
serialization of the whole row, inside the caller's own transaction, and
appended to an oplog nothing ever drains. Arch-review finding #5 (High),
repo half; the library half — trigger cleanup API and O(1) backlog — is
WP3.3.
The ten RegisterReplicated calls are now behind a guard on whether the
node has LocalDb:Replication:PeerAddress OR LocalDb:Replication:ApiKey.
Either key counts, and the OR is load-bearing rather than defensive:
replication is one bidirectional stream that exactly one side dials, so
only the initiator sets PeerAddress. Verified against the rig — site-a
node-a has PeerAddress + ApiKey, site-a node-b (passive) has ApiKey
alone, site-b/site-c have no Replication section at all. Keying on
PeerAddress alone would have stripped capture from every passive node and
silently made each pair converge in one direction only.
The load-bearing ordering documented in the file is preserved: DDL still
precedes registration, and the legacy migrator still runs unconditionally
after it — an unreplicated node must still absorb its pre-Phase-1 files,
and it has no peer for those rows to be invisible to.
Known residual, documented in-file and in the topology guide: a database
file first created by an older build keeps its stale __localdb_* triggers.
The guard decides whether triggers are installed, not whether existing
ones are removed, and the library has no removal API until WP3.3. Moot on
the docker rig, where a schema-change redeploy recreates the volumes.
The inverse is also now documented: enabling replication on a site that
has run without it does not baseline existing rows, since CDC never
recorded them in __localdb_row_version and the snapshot resync streams
from that ledger.
Tests: new SiteLocalDbCdcRegistrationTests asserts trigger presence and
absence via sqlite_master across all four config shapes (none, ApiKey
only, PeerAddress + ApiKey, and the notification-table exclusion), plus
DDL-still-runs and migrator-still-runs on the unreplicated branch.
SiteLocalDbWiringTests and the integration site-pair harness now
configure an ApiKey — mirroring the rig's passive node — so their
registration and convergence assertions still describe a replicating
node. 483/483 Host.Tests pass; the 20 offline LocalDb convergence tests
still pass.
The Secrets management UI (ZB.MOM.WW.Secrets Secrets.Ui, mounted at
/admin/secrets) has been linked from the NavMenu Admin section since the
Theme adoption, but no NavMenu test asserted it. Add bUnit coverage that
the item renders for an Administrator and is absent for a
Designer+Deployer principal, matching the existing role-gate test style.
The rig pointed at the shared 10.100.0.35 GLAuth whose serviceaccount password
was rotated (SEC-36), so central login had been failing ('Authentication service
is misconfigured') and a TEMP DisableLogin workaround was pending. Central nodes
now point at the local redundant pair (scadaproj/infra/glauth-redundant,
host.docker.internal:3893 + FallbackServers :3894), where the dev bind password
is correct — live-gated on the redeployed rig: login OK, primary-kill failover,
sticky preference (bind-count proven), walk-back on backup-kill.
AuthFlowTests factory bound as cn=admin for search-then-bind, but the current
directory grants the search capability only to serviceaccount (admin searches
return 50 Insufficient access) — stale since the GLAuth config evolved; the test
had been skipping on the closed port and failed once anything answered :3893.
Now binds as serviceaccount; AuthFlowTests 5/5 against the pair.
ScriptOptions.WithReferences(Assembly[]) resolves each assembly through
MetadataReference.CreateFromFile, which does not cache: every call mints a
fresh AssemblyMetadata -> PEReader -> NativeHeapMemoryBlock holding an
unmanaged copy of the assembly metadata that nothing disposes. Building the
options per compile therefore leaked native memory permanently — invisible to
the GC, to gcdump and to the managed allocation counters, so the working set
grew while the GC heap did not.
Diagnosed from a live dump of a wonder-app-vd03 Site node: 2,885 MB working
set 78 min after a cold start, only 150 MB live GC heap, ~2,469 MB on the
default process heap across ~6,700 undisposed AssemblyMetadata instances
against 473 DLLs on disk.
Three sites, all hoisted to static readonly:
- SiteRuntime ScriptCompilationService (the dumped one)
- InboundAPI InboundScriptExecutor — same defect on the central node; method
compiles recur on every re-registration and revision change
- CentralUI ScriptAnalysisService — CreateFromFile per sandbox run
ScriptAnalysis RoslynScriptCompiler also builds options per call but draws
from the static ScriptTrustPolicy.DefaultReferences, so it mints no metadata
and is left alone.
Guarded by reference-equality on the artifact rather than by watching memory:
a bytes-watching test would be flaky, and the leak is native so the managed
counters cannot see it at all. The test is proven to fail before the fix.
This does NOT close the AddTemplateScript OOM — that path was shown twice not
to compile scripts. It explains how a long-running node reaches a native
memory state where a large allocation fails with gigabytes free, which is a
lead worth re-testing, not a closure.
CLAUDE.md points at .claude/skills/scadabridge-components/SKILL.md as the
component catalog and .claude/skills/scadabridge-cluster-ops/SKILL.md as the
source for the management URL, credentials and rebuild/redeploy commands — but
neither file was tracked, so a fresh clone got instructions referencing content
that was not there.
Only the skill files are added; .claude/settings.local.json and
scheduled_tasks.lock stay ignored via .gitignore as before.
Component-AuditLog.md has always required "we over-redact, never under-redact,
on configuration faults", but the body / SQL-parameter redactors violated it.
AuditRegexCache rejects a pattern that is malformed OR whose compile exceeds a
100 ms budget, caching the rejection for the process lifetime.
ScadaBridgeAuditRedactor then simply dropped the rejected pattern from its
redactor set and emitted the payload anyway — publishing precisely the values
the operator configured it to suppress, onto a row that looks entirely normal
downstream. Recovery required a process restart and the only signal was one
Warning line. The SQL path was worse: TryGetSqlParamRedactor returned a bare
false for both "no redactor configured for this connection" and "the configured
one will not compile", and CLAUDE.md records SQL parameter capture as on by
default.
Two changes:
1. Fail closed. A pattern that is CONFIGURED but unavailable now over-redacts
the whole payload and increments AuditRedactionFailure, reusing the existing
safety-net path. "Not configured at all" stays permissive — conflating those
two states is the actual defect, so both are pinned by tests.
2. Precompile off the hot path. The audit-log roadmap specifies patterns are
"precompiled at startup; rejected if compile takes >100ms"; the implementation
had drifted to compiling lazily on first event, which put a wall-clock budget
on a hot path under production load. RegexOptions.Compiled emits IL during
construction, so a busy node could blow the budget on a perfectly valid
pattern. Warm-up now runs at construction and on every options reload. The
residual window between a reload and its warm-up is safe because that path
now fails closed.
Warm-up deliberately does not fail the boot — an unusable pattern degrades the
node to over-redaction (safe, loud) rather than refusing to start. Reading
CurrentValue happens inside the warm-up try so an options provider that throws
still surfaces via Apply's over-redact path, not the constructor
(OuterCatch_OptionsThrows_NeverLeaks_AllSensitiveFieldsOverRedacted).
Also de-flakes GrpcCentralTransportTests.DeadlineExceeded_IsNotRetriedOnThePeer,
which is how this was found. It black-holed node A behind a 300 ms deadline, but
on a saturated machine the call could fail to even START — a genuinely-unsent
failure that IsConnectFailure correctly fails over on, so node B's ack arrived
instead of the expected Status.Failure. The test read as a flake while actually
reporting that its own premise had not held. Split in two: the hard rule now
injects an explicit DeadlineExceeded via a trailers-only response (deterministic,
load-independent), and a new BlackHoledNode_DoesNotHang covers the
deadline-is-actually-applied half with both nodes black-holed so no ack can
arrive down any path.
Verified: both fixes were confirmed to fail before they pass — reverting the
fail-closed guard fails exactly the 5 fail-closed tests while the 4 controls
still pass, and adding DeadlineExceeded to IsConnectFailure fails the rewritten
transport test. AuditLog 367/367, Host.Tests GrpcCentralTransport 8/8, solution
build clean. The previously-intermittent
Filter_PicksUp_NewBodyRedactor_OnConfigReload is green in a full sweep for the
first time.
Not addressed here, and noted on #35: the 100 ms wall-clock budget remains a
weak proxy for catastrophic backtracking (RegexOptions.Compiled defers JIT to
first match, so construction time measures the wrong thing), and a rejection is
still cached permanently. Both are now safe rather than dangerous, so they are
hardening rather than a leak.
Brings the two shared-library families that had fallen behind the Gitea feed up
to latest; the other 20 ZB.MOM.WW.* packages were already at feed-latest.
Auth 0.1.5 -> 0.2.0 is LDAP failover only: a new LdapOptions.FallbackServers
list, an endpoint walk with sticky preference in LdapAuthService, and per-hop
warning logs. Purely additive — with FallbackServers empty (the committed
default everywhere) the walk collapses to exactly one attempt, so behaviour is
unchanged until someone opts in. Nothing in Auth.ApiKeys changed, so there is
no key-store schema migration to sequence.
MxGateway 0.1.1 -> 0.2.0 matters more than the version gap suggests: upstream
records that four clients "had drifted onto the already-published 0.1.2/0.1.1
while their APIs kept changing underneath", so 0.1.1 was a stale label rather
than a stable point. Real deltas since: status/HRESULT reply validation made
conformant across clients (CLI-37/38), the ReplayGap reconnect sentinel
surfaced as a typed signal (CLI-15), and typed single-item command parity
(CLI-04/30).
The one change touching the write path — correlating OnWriteComplete onto plain
Write/Write2 replies — is server-side. Its proto diff is comment-only and the
behaviour lives in the Worker, so ScadaBridge gains the correlated statuses when
the gateway server is upgraded, not from this bump. No wire-contract break.
Verified: restore and full solution build clean (0 warnings, 0 errors); all 22
shared packages resolve to a single version each across all 57 projects, with no
split resolution. Security 181/181, InboundAPI 269/269, DataConnectionLayer
277/277 — the three suites that consume these packages directly.
Full-sweep residue is pre-existing and unrelated, confirmed 3/3 green in
isolation for the first two:
- GrpcCentralTransportTests.DeadlineExceeded_IsNotRetriedOnThePeer — a
saturated TestServer can fail to START the call, which IsConnectFailure
correctly treats as provably-unsent and fails over; the harness cannot
guarantee the DeadlineExceeded it means to exercise.
- AuditLogOptionsBindingTests.Filter_PicksUp_NewBodyRedactor_OnConfigReload —
not a reload race (the test is synchronous); it trips the 100 ms compile
budget in AuditRegexCache, which fails OPEN. Filed as its own issue.
- CentralUI Playwright 159/173 — login failures against the running rig,
which was built from the pre-bump image and so cannot be affected by this
change: 20 from the known SEC-36 GLAuth password rotation, 136 cascading
from the login throttle.
The rig still runs the previous libraries; it needs a docker/deploy.sh rebuild
to pick these up.
The 0.6.2 adoption set Secrets__SqlitePath inside x-secrets-hub-env, which only the
central pair and site-a merge — clustered replication is deliberately enabled on four
nodes so the default-OFF posture is proven side by side. site-b and site-c merge no
secrets anchor at all, so they fell back to the shipped default and crash-looped at
boot (exit 134, nine times each) while the other four came up clean.
Splitting the store path into its own anchor that all eight merge fixes it. The path
is not optional config: every node has a store, only four have a hub.
Also corrects the assumption behind deleting the key from appsettings.json. Measured
here: with a Secrets section present but no SqlitePath, the effective value is the
RELATIVE "secrets.db", which fails both validation rules — the per-user default does
not take over. A deployment must set it explicitly.
Verified live: all 8 nodes up, zero validation errors, stores at /data on the same
host directory as before, all four pairs converged to one cluster each through a
simultaneous recreate, central-a active / central-b standby, secrets hub sweeps
recovered after the expected boot race.
Records that the SharedSchema resolver is fully consumed at deploy, runtime and in
the UI value-entry forms, but has no authoring path: SchemaBuilder is the only
schema editor on all four surfaces and offers no library-reference option, so the
page creates entries nothing can point at. SchemaBuilder also collapses a $ref it
is shown to {"type":"string"} — currently unreachable, since no ref exists and
there is no UI or CLI to make one, but a trap for the first one authored.
Includes live usage counts (zero entries, zero refs, both environments) and the
resulting recommendation: leave dormant with a trigger, rather than finish or
delete it now.
0.6.x refuses a secret store whose path is relative or inside the content root,
because a store in the deployment directory is destroyed by an ordinary upgrade —
the failure that wiped the MxGateway API-key store on 2026-08-09 and read as an
auth outage rather than a deployment error.
The pin alone would not have protected this repo. Program.cs expands ${secret:}
before the host exists, composing secrets into a throwaway ServiceCollection with
no IHostEnvironment, so the guard would not run at the moment the migrator creates
the store. That composition now lives in SecretsRegistration with an explicit
content root — resolved to match what the host resolves later, including the
Windows-Service case where the pre-host CWD is still system32 — and is covered by
PreHostSecretsContentRootTests, verified by simulating the regression and
confirming it fails on the leftover file rather than on the exception.
The docker rig needed a fix too: /app/data is absolute but inside the container's
content root, so all 8 nodes would have failed to boot. Each node's data directory
is now mounted a second time at /data; same host directory, so existing stores
carry over untouched.
Verified: build clean, 29 test assemblies green (Playwright's 159 failures are the
pre-existing SEC-36 login baseline). Not yet deployed — the rig runs the old
config until someone redeploys.
Gating the detail modal/drawer on a held id rather than on the row resolving
(d14e0ee4) made it the surface's own job to clear that id when navigation
invalidates page-scoped state. ParkedMessages and ConfigurationAuditLog did not,
so paging away from an open row left the surface mounted on a notice it could
never recover from — reachable only by paging back.
The criterion is per-surface and comes down to whether the modal has content of
its own:
ParkedMessages, ConfigurationAuditLog — no keyed detail fetch; content resolves
from the loaded page alone. An entry paged out of view can never resolve again,
so these must clear on paging. They already cleared it on Search/OnSiteChanged
for the same reason, and clear _selectedIds on paging for the same reason
again; paging was simply missed.
NotificationReport, SiteCallsReport — fetch detail by id, so the modal still
shows real content after its row leaves the page. These deliberately do NOT
clear on paging and are unchanged.
Clearing on an explicit navigation action is user intent, not a resolve-driven
unmount, so this cannot reopen the handler-disposal race that d14e0ee4 closed.
PageScopedDetailStateTests covers all three paging entry points and records the
criterion so the next reader can tell why two surfaces clear and two do not. Run
against both clears reverted, all three fail; restored, all three pass.
CentralUI.Tests 994/994, solution build 0/0.
Also corrects two comments in ParkedMessages left stale by d14e0ee4 — they still
described the drawer as self-closing when a row stops resolving, which is the
behaviour that change deliberately removed.
Joseph Doherty