Second slice of the 2026-09-09 whole-system review remediation
(docs/plans/2026-09-09-arch-review.md §10), on top of the High slice 146a1323.
Runtime/devices: SR-2 alarm on-trigger integration scope; SR-3 correlated NativeAlarmActor
subscribe retry; SR-4 immutable audited DB target; SR-5 ordered InstanceActor stop/flush;
SR-12 rehydration fence; SR-13 S&F gate recheck per delivery; DC-4 no OPC UA security
downgrade; DC-5 alarm snapshot for every new subscriber; ORCH-1 sync-dispose joins the pump.
Management/UI: MG-2 site scope on audit tree + Flow Debugger Replay; MG-3 stale-probe memo
removed; MG-4 server-owned bulk-deploy deadline; UI-5 sandbox runs in a killable child
process (new CentralUI.SandboxWorker project); UI-14 bounded sandbox output/IPC.
Edge/audit: EI-1 relay audit ownership; EI-2 SkipBodyCapture suppresses error text; EI-3/EI-11
error-field redaction + caps; EI-4 single relay deadline, pre-execution-only retry; EI-5
authenticated relay provenance; EI-10 body cap before audit read.
Persistence/contracts: DL-1 global Instance.UniqueName unique index + conflict translation;
DL-2 DataConnection protocol config encrypted at rest (migration
20260909203046_EnforceGlobalInstanceNamesAndProtectProtocolConfiguration — coordinated
central upgrade, old binaries cannot read protected rows); DL-3 bounded Replay paging;
DL-6 cross-site Overwrite of a deployed instance rejected; XC-4 proto descriptor contract test.
Config/hygiene: XC-1 central listener isolation by local port; XC-2/XC-3 inert SMTP + cron
keys removed from rig settings; CM-2 env2 bootstrap guard enabled; DC-6 deploy site exit code;
DC-7 notifier retries only explicit standby rejection; XC-5 ClusterClient purged from docs.
Gates recorded in the report: build 0/0, 9 397 passed / 0 failed, rig redeployed on all
eight nodes with the migration applied. 46 Low findings remain open.
Measuring the tracking-store queue on the rig moved nothing: the DB-done -> Notify.Send
gap stayed at 108 ms. The cost was the audit half of CachedCallTelemetryForwarder:
SqliteAuditWriter.WriteAsync enqueues synchronously but returns the BATCH completion,
and the writer loop coalesces for FlushIntervalMs (50 ms) before it commits, so each
awaited audit row cost ~50 ms + fsync and a cached call awaited three (submit,
attempted, resolve) on the script thread. Every other script-path emitter already
fired-and-observed; only this forwarder awaited.
TryEmitAuditAsync now hands the row to the writer and observes the completion on a
fault-logging continuation (the FallbackAuditWriter ring-buffer fallback still runs
inside that task). Best-effort contract unchanged; per-store FIFO unchanged; nothing
joins the audit and tracking stores by commit order.
Rig, FlowOutboundDemo, 5-run medians: inbound 194 -> 25 ms, site script run 176 -> 17 ms,
CachedWrite submit->done 61 -> 11 ms, DB done -> Notify.Send 108 -> 0.1 ms. The tracking
queue stays (three fewer fsyncs per call, batched). Docs corrected to attribute the cost
to the flush wait, with the trap recorded in CLAUDE.md and Component-SiteRuntime.md.
Tests: ForwardAsync returns while the audit commit is still pending (never-completing
writer task) and still records tracking; a late commit fault is swallowed.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
GetTraceEventsAsync widens its channel filter to ScriptTrace ∪ ApiOutbound ∪
DbOutbound ∪ Notification ∪ (ApiInbound & InboundRequest) (plan D6); SecuredWrite,
Cluster and InboundAuthFailure rows in the same tree stay excluded. The site
SqliteAuditWriter.ReadByExecutionIdsAsync already had no channel predicate — pinned
by a test. Interface doc updated to match.
Claude-Session: https://claude.ai/code/session_01NcMG6xXdsJbmD2wYAgkK29
Flow Debugger (design 2026-09-05 §7.1): IAuditLogRepository.GetTraceEventsAsync walks any
entry id to its root and down the Edges/Chain CTE exactly as GetExecutionTreeAsync, then
returns the chain's ScriptTrace rows plus the ApiInbound.InboundRequest row via the canonical
row mapper, ordered (OccurredAtUtc, EventId). Test doubles implement the new member with a
NotSupported stub.
Claude-Session: https://claude.ai/code/session_01GSLkkHKz9JrAtmJjmQK3cP
Flow Debugger (design 2026-09-05 §5.3/§5.4): AttributeValueUpdate fields 7/8 stamped by
StreamRelayActor and parsed by SiteStreamGrpcClient (older frames => null/false). New
PullTraceEvents unary RPC on SiteStreamService (same PSK-gated prefix as PullAuditEvents):
a point read of the site audit SQLite by ExecutionId OR ParentExecutionId
(ISiteAuditQueue.ReadByExecutionIdsAsync, json_extract on the codec's camelCase keys, new
IX_audit_event_occurred index, 64-id / 2000-row caps) that moves no cursor and flips no
ForwardState. Central IPullTraceEventsClient/GrpcPullTraceEventsClient mirror the audit pull
client (endpoint resolution, NodeB failover, fault => empty) and register beside it.
Claude-Session: https://claude.ai/code/session_01GSLkkHKz9JrAtmJjmQK3cP
SiteAuditReconciliationActor minted a fresh random EventId for the synthetic
ReconciliationAbandoned row every time a permanently-abandoned row was
re-abandoned. Because the reconciliation cursor is held at `since` by ANY
co-batch unresolved row (see PullSiteAsync's hasUnresolvedFailure gate), a
poisoned row's per-EventId attempt counter restarts every time it reappears
in a stuck batch, so it gets re-abandoned roughly every
MaxPermanentInsertAttempts ticks — minting an unbounded number of duplicate
central audit rows for the SAME lost event.
Fix: DeriveAbandonmentEventId(Guid lostEventId) derives the synthetic
EventId deterministically (SHA-256 over a fixed namespace GUID + the lost
row's bytes, RFC4122-v5-shaped). Re-abandonment of the same lost row now
always produces the same synthetic EventId, so
IAuditLogRepository.InsertIfNotExistsAsync's existing idempotency collapses
every repeat into the first insert. Also adds an in-memory _abandonedEventIds
set as a CHEAP SHORT-CIRCUIT ONLY (skips the wasted ~25 min retry-then-abandon
cycle for a row already known abandoned this process's lifetime) — it is not
the correctness mechanism; it resets on singleton restart/failover, at which
point the deterministic id alone still keeps re-abandonment idempotent.
Tests (SiteAuditReconciliationActorTests):
- ReAbandonment_SameLostEventId_YieldsExactlyOneCentralRow: end-to-end repro
of the defect trigger (poison row + a rotating-EventId co-batch row that
never itself reaches the threshold, holding the cursor forever) across two
full abandon cycles. Proof of failure: run against the unfixed source
(git stash of the source file only, pure-function tests temporarily
#if false'd to keep the file compiling), the test failed with exactly the
predicted defect shape — 2 distinct synthetic rows for one lost EventId.
Passes post-fix (1 row).
- DeriveAbandonmentEventId_DifferentLostRows_YieldDistinctSyntheticIds and
DeriveAbandonmentEventId_IsPureAndStableAcrossIndependentInvocations:
pure-function coverage of the derivation directly, including a pinned
golden-value regression guard proving it is a function of the input GUID's
value alone (no clock/randomness/process state).
P5-1: AuditLogIngestActor.OnCachedTelemetryAsync's per-entry fallback loop
(used when the cached-telemetry batch fast path fails) ran with no
budget/cancellation, unlike OnIngestAsync's fallback. A slow/wedged DB could
freeze the central ingest singleton's mailbox for minutes. The loop now runs
under its own fresh IngestFallbackBudget CancellationTokenSource (mirroring
OnIngestAsync), passes the token into BeginTransactionAsync/
InsertIfNotExistsAsync/UpsertAsync, and stops early on budget exhaustion,
logging how many entries were left for the periodic reconciliation pull to
pick up.
P5-2: SiteCallAuditRepository.QueryAsync ordered by
(CreatedAtUtc DESC, TrackedOperationId DESC) with no index leading
unconditionally on CreatedAtUtc, forcing a full scan+sort on an unfiltered
page. Added IX_SiteCalls_Created_TrackedOperationId (migration
AddSiteCallsListSortIndex + matching hand-written production-parity script
under docs/plans/sql/). Also switched the query from FromSqlInterpolated
(which binds string filters as nvarchar(4000)) to FromSqlRaw with explicitly
SqlDbType.VarChar-typed parameters, matching the SiteCalls columns'
IsUnicode(false) declaration and the write path's existing Param(...)
pattern, so seeks — including the keyset predicate — are no longer blocked
by an implicit conversion.
Tests: new AuditLogIngestActorCombinedTelemetryTests case proves the
fallback budget bounds the wait (hanging repo stub, ~5s not indefinite),
that remaining entries are skipped, and that a non-default/cancellable
token reaches the repository call; verified fail-unfixed by reverting the
actor change (test times out at its 20s bound). Existing
SiteCallAuditRepositoryTests QueryAsync/keyset/stuck-cutoff cases (real
MSSQL) confirm the raw-SQL rewrite still returns correct pages.
P4-2: SqliteAuditWriter.FlushBatch completed each pending row's Task
(TrySetResult) inside the per-row insert loop, before transaction.Commit()
ran. A later failure in the same batch (another row's insert throwing, or
the commit itself failing) rolled the whole transaction back, but the
earlier rows' Tasks had already completed successfully -- the caller and
FallbackAuditWriter (whose ring-buffer fallback exists precisely for this
outage) both saw false success while the rows were actually gone from the
database; the outer catch's TrySetException became a silent no-op against
an already-completed TaskCompletionSource. Every row's Completion is now
deferred and only resolved (TrySetResult for the whole batch, or
TrySetException for the whole batch) after transaction.Commit() succeeds
or the transaction rolls back, mirroring SiteEventLogger.FlushBatch.
P4-4: EventLogPurgeService gated the daily site_events purge on the
active-node check, early-exiting on the standby, on the stale premise that
"the standby receives no writes". DataConnectionActor runs on both site
nodes and writes site_events regardless of active/standby status (register
rows 89/90/97), so an unreplicated standby (the rig's site-b/site-c
posture) grew its site_events file forever with no retention and no cap.
The gate is removed; every node now purges its own file every tick. Safe
on a replicated pair because the retention/cap deletes are CDC-captured
and converge under LWW regardless of which node issues them; required on
an unreplicated pair because each node's file is independent. The
active-node check is retained only to annotate the purge tick's log line.
Tests: new SqliteAuditWriterCommitFailureTests forces a genuine
non-swallowed mid-batch INSERT failure via a real SQLite trigger + custom
scalar function on the writer's own connection (reflection seam), proving
(a) a row whose own insert succeeded still faults when a later row in the
same batch fails, and (b) FallbackAuditWriter rings every faulted row and
increments the failure counter for each -- both assertions fail against
the pre-fix code (proof-of-failure verified by temporarily reverting the
fix). EventLogPurgeServiceTests' standby case is inverted to assert the
purge now runs (not skips) on a standby node.
maxParallelThreads was unset everywhere and only 3 of ~20 test projects carried
an xunit.runner.json at all, so xunit ran Environment.ProcessorCount collections
at once (14 on the dev box) in every suite. That is the general load multiplier
behind the row-82/91/95 flake family.
Policy: per-suite caps, NOT a global one. Pure unit suites stay uncapped for
speed; a suite gets a config only when a test CLASS costs a real external or
OS-level resource, so ProcessorCount of them at once is a resource question
rather than a CPU question. Four suites qualify, all capped at
maxParallelThreads: 4.
- ConfigurationDatabase.Tests, AuditLog.Tests, SiteCallAudit.Tests — verified as
the three that actually take MsSqlMigrationFixture (the latter two by project
reference). Every MSSQL-backed test class means a CREATE DATABASE plus the
~100-migration chain against the one shared dev SQL Server. 4 is chosen to
MATCH the default slot count of row 82's cross-process sp_getapplock migrate
semaphore, so the in-process and cross-process limits agree rather than
fighting.
- Host.Tests — classes boot real ASP.NET Core hosts and real ActorSystems;
Akka.Remote binds real sockets, Kestrel binds real ports, gRPC dials them.
Threads are capped, collections are NOT serialized. IntegrationTests
(parallelizeAssembly/parallelizeTestCollections false) and
CentralUI.PlaywrightTests (maxParallelThreads 1) already carry stricter
pre-existing settings and are left alone — the Playwright suite drives a browser
against a live rig, where shared UI state, not CPU, is the constraint.
Each capped csproj gains
<Content Include="xunit.runner.json" CopyToOutputDirectory="PreserveNewest" />;
the SDK does not copy it automatically and without it the file is silently
ignored. Verified by moving the copied file out of bin/ to take an uncapped
baseline: ConfigurationDatabase.Tests ran at 216 % CPU uncapped vs 145 % capped,
so the cap demonstrably takes effect.
Cost is close to nil — these suites are latency-bound on an external resource,
not CPU-bound (14-25 % utilisation for the two slowest). Standalone, uncapped ->
capped: AuditLog 383 tests 1m52s -> 1m51s; Host 520 tests 4m05s -> 4m08s;
ConfigurationDatabase 396 tests 10s -> 12s; SiteCallAudit 66 tests 8s -> 8s. All
green.
docs/test_infra/test_infra.md gains a Test Parallelism Policy section: the two
axes (in-process collection threads = this cap; cross-process solution-level
dotnet test = unbounded in general, bounded on the DB-migration path by the
sp_getapplock semaphore), the per-suite table with reasons, the rules (cap
threads rather than serialize; a cap is not a substitute for test isolation; the
Content copy is required), and the default for new projects (no config until the
project acquires a per-class external resource). test_infra_db.md cross-links it
from the semaphore bullet.
Row 82 (da23f2dc) fixed this leak shape in PartitionPurgeTests and
NotifyDispatcherAuditTrailTests; three classes in the same assembly still had
it. Each creates an actor with a fast periodic tick (100 ms purge override,
~0 s reconciliation override) against a shared MsSqlMigrationFixture database
and never stops it deterministically, relying on TestKit's Dispose -> Shutdown
— which is timeout-bounded and does NOT fail when it expires. A survivor keeps
mutating the fixture DB while LATER test classes run.
All three now use the same Watch(actor) / Sys.Stop(actor) /
ExpectTerminated(actor) helper da23f2dc established, applied at EVERY
actor-creation site — 11 in AuditLogPurgeActorTests, 10 in
SiteAuditReconciliationActorTests, 2 in OutageReconciliationTests.
The two real-DB cases are the ones that could corrupt another class:
AuditLogPurgeActorTests' EndToEnd_RealPartition_… keeps switching partitions
out of the shared DB, and SiteAuditReconciliationActorTests'
Tick_Duplicates_NotDoubleInserted holds a live AuditLogRepository over a
DbContext the test is about to dispose. The stub-repo cases are cheaper but not
free: each burns a DI scope and a thread every 100 ms for the rest of the
assembly run.
OutageReconciliationTests needed more than the helper. It already called
Sys.Stop, but (a) fire-and-forget, so the call returns before the in-flight tick
has finished writing to MSSQL, and (b) from the LAST line of the enclosing
`await using` block, so any earlier assertion failure skipped it entirely. Both
stops move to immediately after the last step that needs the actor alive —
before the read-back that counts rows — and the AuditLogIngestActor that
BuildCentralPipeline creates is now stopped too (test 2 was discarding its ref
outright, leaving it holding a service provider the block then disposed).
Where a test's remaining assertions read a collection the actor appends to
(client.Calls indexing, Assert.Empty on a switched-boundaries list), the stop is
placed BEFORE them so they no longer read a moving target.
AuditLog.Tests: 383/383 green (1 m 51 s), 0 skipped.
AuditLog.Tests flaked under parallel full-solution runs. Three distinct
mechanisms, all load-sensitive and all green in isolation.
1. MsSqlMigrationFixture create+migrate saturation (the register's headline).
Database NAMING was already isolated (a guid per fixture), but nothing
bounded how many fixtures ran CREATE DATABASE + the ~100-migration chain at
once. xunit parallelises processor-count collections PER PROCESS, and a
solution-level `dotnet test` starts several MSSQL-backed processes together,
so the dev container saw dozens of simultaneous fresh-database migrations.
That is resource saturation, not object-level locking, and it pushed
individual DDL statements past EF's default 30s command timeout — every
failure had the identical fingerprint the register recorded:
MsSqlMigrationFixture..ctor -> Migrator.MigrateAsync -> SqlException
"Execution Timeout Expired" (error 258), which xunit surfaces as
"Class fixture type ... threw in its constructor" against every test in the
class.
Fix: a machine-wide counting semaphore over the create+migrate window, built
from sp_getapplock on master (ScadaBridgeTestMigrate/0..N-1, default 4 slots,
SCADABRIDGE_MSSQL_TEST_MIGRATE_SLOTS to override). An xunit collection cannot
do this — the contention is cross-PROCESS, so the arbiter has to live on the
shared server. The gate connection is unpooled (a pooled connection defers the
implicit release of a session-scoped app lock until reuse) and releases
explicitly; a crashed process drops its connection and SQL releases the lock,
so the gate cannot wedge the suite, and a deadline miss proceeds ungated
rather than failing. Migration commands also get a 300s timeout and
CREATE/DROP DATABASE 120s: under a bounded queue the server is busy but
progressing, so waiting is the correct response. Plus a once-per-process
best-effort sweep of ScadaBridgeAuditMigTest_* databases older than 2h so a
killed run does not leave strays, and the connectivity probe now retries
(3 attempts) — a busy server misread as an absent one would silently turn
every MSSQL test into a Skip, which reads green and is worse than a red.
2. PartitionPurgeTests leaked AuditLogPurgeActors. Each ticks every 100ms and
switches out the Jan-2026 partition; they were only stopped implicitly by
TestKit Dispose -> Shutdown, which is timeout-bounded and does not fail when
it expires. Under load a previous test's actor was still ticking against the
shared per-class fixture DB, so PerChannelOverride_... found its seeded
Jan-15-2026 row already gone (Expected: 1, Actual: 0).
3. NotifyDispatcherAuditTrailTests leaked NotificationOutboxActors, same shape.
The dispatcher claims EVERY due row, not just the test's siteId, so a leaked
actor delivered the next test's notification through its own exhausted
QueuedOutcomeAdapter — a transient outcome, leaving the row in Retrying
(Expected: Delivered, Actual: Retrying) and audit counts short.
Both fixed by stopping the actor inside the test that created it
(Watch/Stop/ExpectTerminated), removing the dependency on shutdown timing.
Evidence. Reproducer: 10 concurrent ConfigurationDatabase.Tests processes
against the one dev SQL Server — before, 10/10 processes failed (17-77 failures
each, 100% the ctor->MigrateAsync timeout); after, 3 consecutive runs 10/10
green, 395/395 each. Realistic shape (3x AuditLog.Tests + 1x
ConfigurationDatabase.Tests concurrent): first post-fix run still showed the
NotifyDispatcher leak, then 3 consecutive clean runs. Standalone: AuditLog.Tests
383/383, ConfigurationDatabase.Tests 395/395 (12s, unchanged from baseline),
SiteCallAudit.Tests 66/66. No orphan databases or stuck app locks left behind.
docs/test_infra/test_infra_db.md documents the ephemeral per-test-class
databases, the throttle and its env override, the timeouts, and the stray sweep.
AuditLogIngestActor and SiteCallAuditActor each exposed constructors taking a scoped
repository and caching it on a field for the actor's lifetime. Safe only because no
production call site picked those overloads — i.e. one call site away from re-introducing
the row-45 captive-dependency bug — which is why CaptiveDependencyGuard had to allowlist
both fields.
Preferred fix applied: the test fixtures fold onto the IServiceProvider overload with a
stub provider, and the constructors are gone.
Product deletions:
- AuditLogIngestActor: the (IAuditLogRepository, ILogger) ctor, the _injectedRepository
field and its branch in OnIngestAsync, and OnCachedTelemetryWithoutDualWriteAsync (the
"this ctor cannot dual-write" fallback handler that only that ctor installed).
_serviceProvider is now non-nullable and the null-forgiving operators are gone.
- SiteCallAuditActor: both test ctors (repo-only, and repo + reconciliation
collaborators), the _injectedRepository field and its four branches, and the private
SerializedRepository class. _backgroundTimersEnabled went with them: only the deleted
repo-only ctor ever set it false, so it was a constant and its two guards are removed.
The reconciliation tick is still gated — on the collaborators being resolvable, which
is how production reaches the same state.
Test fixtures (12 call sites): two small TestSupport helpers build the stub providers.
SiteCallAuditActor's production wiring resolves a FRESH repository per scope, so the
serializing decorator existed only to protect fixtures that share one DbContext-backed
instance across every scope — it moves into the test project as
SerializingSiteCallAuditRepository, which is where the hazard is actually created.
Fixtures that relied on it keep it. The relay test's ICentralAuditWriter now arrives via
the provider, exactly as production resolves it.
Behaviour is preserved on the paths under test: the deleted audit ctor passed
redactor:null / failureCounter:null, and a provider registering neither resolves both to
null, so the SafeDefaultAuditRedactor fallback and counter-free logging are unchanged. No
fixture on the folded sites sends IngestCachedTelemetryCommand, so swapping to the real
dual-write handler is inert. The SiteCallAudit purge timer now arms in fixtures that
previously had timers off, but every one of them uses default purge settings whose first
tick is 5 minutes out — nothing fires inside a test.
One reconciliation test was renamed and rewritten rather than deleted:
TestCtor_RepositoryOnly_DoesNotStartReconciliationTick became
NoReconciliationCollaboratorsRegistered_DoesNotStartReconciliationTick — the same
guarantee, expressed the way production actually reaches it.
Guard: the allowlist is now EMPTY. The staleness self-test (every entry must still match a
flagged field) is replaced by two stronger pins — Allowlist_IsEmpty asserts nothing is
exempt at all, and FormerlyAllowlistedActors_NowScanClean scans both actors with the
allowlist DISABLED, so the empty allowlist is evidence the fields are gone rather than
evidence the scan stopped looking.
Negative control: re-adding an ISiteCallAuditRepository field to SiteCallAuditActor fails
both FormerlyAllowlistedActors_NowScanClean and the real
Central_NoActorHoldsCaptiveScopedState scan.
Verification: solution builds clean. AuditLog.Tests 3/3 (383), SiteCallAudit.Tests 3/3
(66), Host.Tests 2/2 (505), IntegrationTests 2/2 (115), Communication.Tests 2/2 (717).
Deferred flake-pattern sweep of tests/ for the class fixed in c4caebe9 and
cfa6acbf — a bounded wait on observable A followed by a bare assert on an
observable B that the product only reaches strictly after A. Three clear
instances, each reproduced deterministically by delaying only the later step
and each re-verified green with that same delay still injected.
AlarmOnTriggerRuns_ShedAtTheSameCap_WithAnAlarmScopedSiteEvent gated on the
rate-limited shed site event and then asserted the shed COUNT bare.
AlarmActor.ShedAlarmRun increments the counter and only then emits the event,
and the event fires on the first shed only — so the gate observed Flap(4)'s
shed and ordered nothing with respect to Flap(5)'s, which is a separate
mailbox message with no observable of its own (an alarm on-trigger run has no
Ask caller to reply to, unlike ScriptActor.ShedRun, whose sibling test is
correctly ordered by its ScriptCallResult and is left alone). Deferring
Flap(5) by 2 s failed it with "Expected: 2 / Actual: 1". The count is now
ACCUMULATED across polls rather than re-read, because
SiteHealthCollector.CollectReport DRAINS the interval counters — a poll loop
that simply re-read it would consume the first shed and never reach 2.
EndToEnd_GrpcStubError_RowStays_Pending_NextTick_Succeeds gated on the
central row arriving and then asserted bare that the site SQLite row had left
Pending. SiteAuditTelemetryActor pushes via IngestAuditEventsAsync (which is
what writes the central row) and calls MarkForwardedAsync only after parsing
the ack. Delaying just that post-push step failed it with
"Assert.DoesNotContain() Failure: Filter matched in collection".
PreSnapshotBuffer_IsCapped_DropsOldest_AndCountsTheDrops gated on
"Count >= cap" and then asserted "Count == cap + 1" bare — a gate strictly
weaker than the assertion it guards, so it ordered nothing with respect to
the last event of a FlushBuffer loop that delivers one at a time. Parking
that loop after its 19,999th delivery failed it with
"Expected: 20001 / Actual: 20000".
Also hardens GrpcCentralTransportTests.WaitUntil, which returned silently on
timeout; today's single caller re-asserts immediately, so this only sharpens
the message rather than fixing a live flake.
Cleared with evidence, not guessed: SiteAlarmLiveCacheService's LingerStop
removes the site entry inside one lock, so IsLive and GetCurrentAlarms flip
atomically; and SiteReconciliationActor walks response.Gap with a sequential
foreach in which the asserted "Gone" log precedes the awaited "Good" row, the
inverse of this class.
Test-only; every ordering named above is correct as written.
F1 (HIGH) DebugStreamBridgeActor: the 5-minute orphan net measured the MAILBOX
(SetReceiveTimeout), and once stream events were correctly marked
INotInfluenceReceiveTimeout nothing recurring reset it — the snapshot lands once
and GrpcStreamStable once — so every healthy session self-terminated at ~6 min
with a false "Site disconnected". Replaced with a periodic self-tick
(ConsumerLivenessCheckInterval, 30s) over a consumer-last-seen stamp renewed only
by DebugStreamConsumerAlive, which DebugStreamService Tells on a shared timer to
every session still in its registry (holding a session there IS "a consumer is
attached" — both the Blazor view and the SignalR hub release it on
dispose/disconnect, and it works headless). Reverting the wrapper was rejected: it
would restore the quiet-instance orphan bug.
F2 (MED) SiteStreamGrpcClient: the RpcException(Cancelled) filter now requires
cts.IsCancellationRequested. A peer-originated / channel-dispose Cancelled fired
none of onError/onCompleted/onConnected, leaving SiteAlarmAggregatorActor with
_streamDown=false forever (IsLive stuck true, reconcile reopen guard never fired).
F3 (MED) SiteStreamGrpcClient: a header TIMEOUT is no longer reported as
connected — that shape is exactly what an unreachable site produces, and it
cleared _streamDown, consumed _seedOnConnect and launched a full snapshot fan-out
at a dead site. AwaitHeadersAsync returns bool; the first received event is the
fallback connected signal, fired at most once from headers OR first event.
F4 (LOW-MED) SqliteAuditWriter.MarkReconciledUpToAsync: the blanket below-cursor
UPDATE retired late-stamped inserts that were never served (then age-purged —
silent loss). The flip is now bounded by insertion order: a Pending row retires
only if its rowid is at or below the high-water mark of rows this instance has
served from ReadPendingSinceAsync (clamped on purge, since SQLite reuses rowids);
Forwarded rows are exempt (central ACKed them over the push path). At-least-once
is unchanged.
F5 (LOW) Documented the liveness dependency (a served row never covered by a later
cursor stays Pending forever; PurgeExpiredAsync never purges Pending) in
ISiteAuditQueue + Component-AuditLog.md, and added a cheap site-health signal:
SiteAuditBacklogReporter logs a rate-limited warning when the existing
oldest-pending metric exceeds 24h.
F6 (MED) SiteAlarmAggregatorActor: _fanoutSinceLastTick was armed by the
reconcile's OWN fan-out, so steady state ran fan-out→skip→fan-out→skip — one
reconcile per 2x interval (120s), halving the not-reporting refresh and the alarm
reconcile backstop. The skip is now armed only by connect/failover-driven seeds
(initial, _seedOnConnect, and a re-seed queued behind one).
Tests: Communication.Tests 691 passed (+13), AuditLog.Tests 382 passed (+5).
Six adversarial-review findings in the central SQL/ingest layer.
F1 (AuditLogRepository.InsertChunkAsync) — the set-based ingest declared each
string parameter at its COLUMN width (Actor/Target 256, Action 64, Outcome 16,
Category 32, SourceNode 64), so SqlClient truncated an over-long value at bind
time and committed the mutilated row — silent, in an append-only store, with no
PayloadTruncated flag — while the per-row and reconciliation paths sent the same
value in full and let the server reject it with 2628. Bind at the value's own
length instead; explicit SqlDbType is kept (it fixes the VALUES constructor's
derived column types and datetime2 precision). Design: reject everywhere,
truncate nowhere — matching today's per-row behaviour.
F2 (SiteCallAuditRepository.UpsertAsync) — the single-statement upsert ran the
monotonic UPDATE first and INSERTed only if nothing matched. Two writers racing
the first packet of one TrackedOperationId (the cached dual-write and the
reconciliation pull carry DIFFERENT lifecycle states) both matched nothing, and
the loser then skipped its INSERT or swallowed a 2627 — dropping its
Status/RetryCount/HttpStatus/TerminalAtUtc. Legs swapped to
`IF NOT EXISTS … INSERT; UPDATE <monotonic>` — still one round trip, and the
loser's UPDATE now lands on the winner's row. The duplicate-key catch re-runs
the monotonic UPDATE for the same reason. Moved to raw SQL with explicitly-typed
parameters so the intricate rank predicate exists in exactly one place (an
untyped DateTime would bind as `datetime` and round the freshness tiebreaker).
F3 (docs/plans/sql/*.sql) — filtered-index DDL failed with error 1934 under the
documented `docker exec … sqlcmd` path, which defaults QUOTED_IDENTIFIER OFF;
once IX_Notifications_Delivered exists, QI-OFF DML on Notifications fails too.
All four scripts now open with `SET QUOTED_IDENTIFIER ON; SET ANSI_NULLS ON; GO`
(own batch, so it is in force when the next batch parses), and the migration
convention in Component-ConfigurationDatabase.md documents `sqlcmd -I`. Verified
live: the pre-fix script fails 1934 without -I, the fixed one applies.
F4 (SiteCallAuditActor) — the off-mailbox reconciliation/purge passes reuse the
injected repository, so tests drove one DbContext from the pass and a mailbox
handler concurrently. Serialized at the CALL via a private SerializedRepository
wrapper applied only by the test constructors, rather than running the pass
on-mailbox: production keeps its PipeTo shape untouched, and the existing
"a blocked drain does not stall ingest/query/KPI" regression tests stay
meaningful (they would have been invalidated by suspending the mailbox).
F5 (AuditLogIngestActor) — when the batch failed because the 20 s IngestBudget
expired, the per-row fallback reused the same expired token: N instant failures,
N counter bumps, zero accepted. The fallback now gets a fresh 5 s budget (inside
the 30 s outer Ask), and a blown budget bumps the failure counter ONCE for the
batch instead of once per row.
F6 (NotificationOutboxRepository.UpdateAsync) — ExecuteUpdate's row count was
discarded, so an operator Retry/Discard of a notification the retention purge had
already deleted reported success (the pre-ExecuteUpdate code threw
DbUpdateConcurrencyException). UpdateAsync now returns whether a row matched; the
operator one-shots answer "notification not found" and emit no audit row for the
action that did not happen, while the dispatcher logs a warning (its delivery
already happened; nothing to retry). GetByIdAsync switched to AsNoTracking since
the write is out-of-band.
Tests: 5 new SQL-backed regressions (over-long Target rejected on both paths +
boundary round-trip; concurrent first-write and already-created-by-another-writer
upserts; vanished-row UpdateAsync), a token-identity pin on the ingest fallback,
a repository-concurrency detector for the SiteCallAudit passes, and vanished-row
operator-path tests. The F1/F2/F4 regressions were each confirmed failing against
the pre-fix code. Suites: ConfigurationDatabase 369, AuditLog 378, SiteCallAudit
66, NotificationOutbox 152 — all green, solution builds with 0 warnings.
CachedDrain_OrphanRow_PastGrace_IsAbandoned_AndTheValidRowStillFlows gated on
IngestCachedTelemetryAsync being received once and then asserted, bare, that the
valid row had been marked Forwarded. The drain does that strictly AFTER the push
returns: OnCachedDrainAsync abandons the orphan (:351), pushes the batch (:366),
then parses the ack and marks the accepted ids (:380). Observing the push
therefore orders nothing with respect to the second MarkForwardedAsync — under a
loaded parallel run the post-push continuation can be scheduled after the poll
that saw the push, and the assertion fails fast with "Actually received no
matching calls" while the orphan's own earlier call is reported as the single
non-matching one.
Reproduced deterministically by delaying only the post-push step, which fails
exactly this test (11 siblings still pass) at ~1.3s into the assembly run —
matching the observed failure's fast-fail signature and pointing at line 430.
With the fix the same injected delay passes; suppressing the valid row's
MarkForwarded entirely still fails the test with the identical message, so the
claim (orphan abandoned in its own call, valid row pushed and marked, exactly
once each with exactly the same arguments) is unchanged in force.
Same unsynchronized-assertion class as c4caebe9, different actor. Test-only; the
drain's abandon/push/mark ordering is correct as written.
Both NotifyDispatcher_AuditWriter_Throws_DeliveryStillSucceeds and
NotificationDispatch_BrokenAuditWriter_StillTransitionsToDelivered read the
throwing writer's attempt counter with a bare Assert immediately after an
AwaitAssert on the Notifications row reaching Delivered. That assumes the audit
writes happen no later than the operational status write, which the dispatcher
deliberately does NOT guarantee: DeliverOneAsync persists the delivery state
first (NotificationOutboxActor.cs:657) and only then emits the Attempted
(:663) and terminal (:676) audit rows — audit is best-effort and must never
gate the user-facing action. Observing Delivered therefore establishes no
happens-before edge with the writer, and under a loaded full-solution parallel
run the continuation after the DB write can be scheduled after the poll that
saw Delivered, so the counter reads 0 and the test fails with "saw 0".
Reproduced deterministically by delaying only the post-update audit emission,
which yields both observed failure messages verbatim; with the fix in place the
same injected delay passes, and suppressing the emissions entirely still fails
both tests with the identical messages — the claims (delivery despite audit
failure, and attempts >= N) are unchanged in force, only the ordering
assumption is gone.
Test-only change; the update-then-audit ordering predates the remediation
(#23 M4) and is correct as written.
Closes WP1.2 of the arch-review remediation plan (finding #2, High):
SqliteAuditWriterOptions.DatabasePath defaulted to CWD-relative "auditlog.db",
which on the docker rig resolves onto the container's ephemeral overlayfs
(not the mounted /app/data volume), silently discarding the pending audit
forward-state backlog on every recreate; nothing in docker/ or docker-env2/
overrode it; FlushIntervalMs was validated but never read by the writer loop
(one commit per event even at trickle rate); and no PRAGMA synchronous was
set (SQLite's FULL default fsyncs every commit).
- DatabasePath now has no default (mirrors ZB.MOM.WW.LocalDb's LocalDbOptions.Path)
and is required pre-host for Site nodes only, via a new StartupValidator raw-config
check (top-level "AuditLog:SiteWriter:DatabasePath", NOT nested under ScadaBridge:
AddAuditLog binds that section off the configuration root). SqliteAuditWriterOptionsValidator
deliberately does NOT check DatabasePath itself, because AddAuditLog runs its
ValidateOnStart on both Central and Site composition roots but only Site nodes
ever resolve the writer — checking it there would fail Central's boot too.
- All 8 site-node appsettings under docker/ and docker-env2/ now set
AuditLog:SiteWriter:DatabasePath to /app/data/auditlog.db (mounted volume,
survives container recreate, same convention as LocalDb:Path); the local-dev
base appsettings.Site.json sets ./data/auditlog.db to match.
- The writer loop now honors FlushIntervalMs: after draining the immediately
available burst, it keeps the transaction open (bounded by FlushIntervalMs
from the first event) waiting for more trickle-rate events before committing,
instead of flushing (and fsyncing) per event.
- PRAGMA synchronous = NORMAL on the write connection — audit is best-effort by
design (CLAUDE.md: "Audit-write failure NEVER aborts the user-facing action"),
so NORMAL's narrower power-loss window is an acceptable trade for far fewer
fsyncs; WAL mode still guarantees no corruption.
- Tests: StartupValidator site-required/blank/central-exempt cases; writer
trickle-load single-transaction coalescing + beyond-interval separate-transaction
regression (new FlushCountForTests seam); options-validator doc updates reflecting
the moved responsibility. Full suite runs green: AuditLog.Tests 368/368,
Host.Tests 480/480.
One-time migration note: the existing container-local auditlog.db (wherever it
landed under CWD) is abandoned by this change, not migrated — already-forwarded
rows are safe centrally (AuditLog is the durable copy), and any still-Pending
rows on the abandoned path are lost once. This is the exact bug being fixed, not
a new loss: those rows were already living outside the mounted volume and would
not have survived the next container recreate regardless. Cross-reference
docs/known-issues/2026-07-20-cached-telemetry-drain-hot-loop.md, which this
placement bug caused.
Component-AuditLog.md has always required "we over-redact, never under-redact,
on configuration faults", but the body / SQL-parameter redactors violated it.
AuditRegexCache rejects a pattern that is malformed OR whose compile exceeds a
100 ms budget, caching the rejection for the process lifetime.
ScadaBridgeAuditRedactor then simply dropped the rejected pattern from its
redactor set and emitted the payload anyway — publishing precisely the values
the operator configured it to suppress, onto a row that looks entirely normal
downstream. Recovery required a process restart and the only signal was one
Warning line. The SQL path was worse: TryGetSqlParamRedactor returned a bare
false for both "no redactor configured for this connection" and "the configured
one will not compile", and CLAUDE.md records SQL parameter capture as on by
default.
Two changes:
1. Fail closed. A pattern that is CONFIGURED but unavailable now over-redacts
the whole payload and increments AuditRedactionFailure, reusing the existing
safety-net path. "Not configured at all" stays permissive — conflating those
two states is the actual defect, so both are pinned by tests.
2. Precompile off the hot path. The audit-log roadmap specifies patterns are
"precompiled at startup; rejected if compile takes >100ms"; the implementation
had drifted to compiling lazily on first event, which put a wall-clock budget
on a hot path under production load. RegexOptions.Compiled emits IL during
construction, so a busy node could blow the budget on a perfectly valid
pattern. Warm-up now runs at construction and on every options reload. The
residual window between a reload and its warm-up is safe because that path
now fails closed.
Warm-up deliberately does not fail the boot — an unusable pattern degrades the
node to over-redaction (safe, loud) rather than refusing to start. Reading
CurrentValue happens inside the warm-up try so an options provider that throws
still surfaces via Apply's over-redact path, not the constructor
(OuterCatch_OptionsThrows_NeverLeaks_AllSensitiveFieldsOverRedacted).
Also de-flakes GrpcCentralTransportTests.DeadlineExceeded_IsNotRetriedOnThePeer,
which is how this was found. It black-holed node A behind a 300 ms deadline, but
on a saturated machine the call could fail to even START — a genuinely-unsent
failure that IsConnectFailure correctly fails over on, so node B's ack arrived
instead of the expected Status.Failure. The test read as a flake while actually
reporting that its own premise had not held. Split in two: the hard rule now
injects an explicit DeadlineExceeded via a trailers-only response (deterministic,
load-independent), and a new BlackHoledNode_DoesNotHang covers the
deadline-is-actually-applied half with both nodes black-holed so no ack can
arrive down any path.
Verified: both fixes were confirmed to fail before they pass — reverting the
fail-closed guard fails exactly the 5 fail-closed tests while the 4 controls
still pass, and adding DeadlineExceeded to IsConnectFailure fails the rewritten
transport test. AuditLog 367/367, Host.Tests GrpcCentralTransport 8/8, solution
build clean. The previously-intermittent
Filter_PicksUp_NewBodyRedactor_OnConfigReload is green in a full sweep for the
first time.
Not addressed here, and noted on #35: the 100 ms wall-clock budget remains a
weak proxy for catastrophic backtracking (RegexOptions.Compiled defers JIT to
first match, so construction time measures the wrong thing), and a rejection is
still cached permanently. Both are now safe rather than dangerous, so they are
hardening rather than a leak.
Phase 4 of the ClusterClient -> gRPC migration deleted the Akka transports
but left the naming behind: `ClusterClientSiteAuditClient` was transport-
agnostic and worked unchanged, so it survived the deletion under a name
that now describes a transport the repo no longer has. Same for a scatter
of doc-comments still framing gRPC as "the new transport" beside an Akka
one that is gone.
Renames it to `SiteCommunicationAuditClient` (and its test file) and
rewrites the stale comments to describe the single transport that exists.
Also tightens CLAUDE.md: drops the self-describing directory listing and
the 27-component enumeration in favour of the non-obvious parts only.
Behaviour-neutral: names and prose only. Recorded as the Phase 4
follow-up in docs/plans/2026-07-22-clusterclient-to-grpc-plan.md.
Phase 0 of the ClusterClient→gRPC migration
(docs/plans/2026-07-22-clusterclient-to-grpc-plan.md). Standalone hardening: it
closes a gap that exists today and is a precondition for moving command/control
onto gRPC in later phases.
T0.1 — delete the ManagementActor ClusterClientReceptionist registration.
It was built for an out-of-cluster CLI that was never written: the shipped CLI
speaks HTTP Basic to /management, which asks the actor in-process through
ManagementActorHolder. Nothing in the repo ever sent to /user/management. The
actor still runs there; only the cross-boundary advertisement is gone. Six
documents claimed the CLI used ClusterClient — including the CLI's own README
"Architecture Notes" — and are corrected here rather than left to rot.
T0.2 — record, do not port, the dead integration-routing path.
IntegrationCallRequest is unwired at BOTH ends: RouteIntegrationCallAsync has
zero callers anywhere, and RegisterLocalHandler(Integration, …) appears only in
a test, so production always answers "Integration handler not available". It is
excluded from the gRPC contract (28 of 29 commands migrate) rather than
enshrined on an additive-only wire format, and deleting it during a
transport migration would mix a behavioural change into a change whose whole
value is that behaviour is identical. See
docs/known-issues/2026-07-22-integration-call-routing-is-dead-code.md.
T0.3 — preshared-key authentication on SiteStreamService.
The service shipped with no auth at all: plaintext h2c, no interceptor, so
anything that could reach a site node's :8083 could open a live data stream or
read audit rows back via PullAuditEvents/PullSiteCalls. ControlPlaneAuthInterceptor
now gates /sitestream.SiteStreamService/ — modeled on LocalDbSyncAuthInterceptor
(constant-time compare, fail-closed, PermissionDenied) but gating a SET of
service prefixes so phases 1A/1B add services rather than interceptors. LocalDb
sync keeps its own separate key: it authenticates the pair partner, not central,
and collapsing the two would make a site's central-facing key also admit writes
into its database.
Keys are per site (SB-GRPC-PSK-<siteId>), never fleet-wide, so a compromised
site yields only its own. Central attaches them through ControlPlaneCredentials,
which binds CallCredentials to the channel — covering unary and streaming
uniformly, and letting the key resolve asynchronously, which a client
interceptor could not do without blocking. All three central→site channel
creation sites go through it (SiteStreamGrpcClient and both audit pull invokers);
the pull invokers' channel caches are re-keyed by (site, endpoint) because
credentials are per-site and bound to the channel.
Two decisions beyond the plan:
* StartupValidator now requires GrpcPsk on Site nodes. The plan specified only
the runtime gate, but fail-closed with no boot check produces a node that
joins, answers heartbeats and reports healthy while refusing every stream,
audit pull and telemetry ingest — silent and total. Same reasoning as the
existing inbound API-key pepper rule.
* Added Communication:SitePsks as a central-side key map. The plan assumed
central would read the store, seeded via a dev KEK; the docker rig
deliberately boots with no master key, so store-only resolution would leave
it unable to dial its own sites. The store stays primary — it is the only
source that can serve a site added at runtime — with the map covering
key-less hosts and one-off pins. Neither source falling back to
"unauthenticated" is the invariant.
T0.4 — dev keys on both rigs and tests.
34 tests. The seven that matter most exercise a real in-process gRPC stack over
TestServer: the unit tests on either side of the wire would both stay green if
the halves disagreed, and gRPC refuses call credentials on a plaintext channel
by default — the UnsafeUseInsecureChannelCallCredentials opt-in is only provable
by making a real call. They confirm correct key passes on unary AND streaming,
wrong key and no-credentials both get PermissionDenied, and an unresolvable key
fails the call with nothing reaching the service.
OPERATIONAL: a site node upgraded to this build without a key will not boot.
That includes the gitignored deploy/wonder-app-vd03/ overlay.
A cached-telemetry audit row whose OperationTracking snapshot could not be
resolved was skipped and left Pending. Nothing ever removed it, so the next
drain re-read it, failed identically, and logged again — forever.
The severity was worse than the original write-up said. The queue is read
oldest-first with a fixed BatchSize (default 256), so once a batch's worth of
permanently-unresolvable rows collected at the head, every drain re-read
exactly those rows and NEVER REACHED the newer rows behind them. That is a
permanent stall of the cached-telemetry path, not the "log flood, no data loss"
the issue was first filed as. Measured on the rig at ~2,800 warnings/minute,
surviving both a process restart and a container restart.
Fix: after a grace period (CachedTrackingGraceSeconds, default 300 s) the
operational half is abandoned and the row marked Forwarded. A row with no
CorrelationId is abandoned immediately — it can never resolve.
Marking Forwarded does not drop audit data. That state means "no longer owed by
the drain, still eligible for reconciliation", which is exactly this situation:
ReadPendingSinceAsync covers Forwarded as well as Pending and central dedups on
EventId, so the reconciliation pull still delivers the audit half. What is lost
is the operational (SiteCalls) half, which is unrecoverable anyway once the
tracking row is gone.
Three deliberate boundaries:
- Inside the grace window the row is still retried — a missing snapshot is
normally a brief write race, and abandoning on the first failed lookup would
discard the operational half of every cached call that lost that race.
- A tracking-store THROW never abandons, however old the row: a throw is a
store fault (locked, corrupt, mid-restore), not a verdict about the row.
- Logging is per drain pass, not per row. Deferred rows dropped to Debug —
being inside the grace window is normal operation, not a warning.
CachedDrain_OrphanRow_NoTrackingSnapshot_IsSkipped_DoesNotCrash was RETARGETED
rather than deleted: it pinned the defect ("skipped and stays Pending"), so its
orphan assertion is inverted and the half that still holds is kept verbatim.
Three tests added, including the starvation regression. Both "must NOT abandon"
tests carry a positive control on the read count, so they cannot pass by virtue
of a drain that never ran.
Verified non-vacuous: with abandonment disabled the two abandonment tests go
red and the two guards stay green. Build 0 warnings; AuditLog 358, Host 330,
SiteRuntime 512, StoreAndForward 130, Communication 312, Commons 684,
Integration 94 — all pass.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Tasks 5 and 6 of the Phase 2 plan, committed together because their test
fallout is entangled — several fixtures construct both stores.
StoreAndForwardStorage and SiteStorageService now take ILocalDb. Connections
come from ILocalDb.CreateConnection(), which hands out an already-open,
pragma-configured connection carrying the zb_hlc_next() UDF the capture triggers
call; a raw connection would lack the UDF and every write to a replicated table
would fail closed. Deleted with the connection strings: S&F's
EnsureDatabaseDirectoryExists and its per-open busy_timeout pragma, and the site
service's BusyTimeoutFloorSeconds normalization — LocalDb owns all of it now.
DI: AddSiteRuntime's string overload is gone (nothing left to supply), so the
Host calls the no-arg form. ScadaBridge:Database:SiteDbPath and
StoreAndForwardOptions.SqliteDbPath survive only as the migrator's source
locations in Tasks 8/9.
Two things the plan did not anticipate, both worth reading:
1. FOUND A REAL LATENT DEFECT, from Phase 1, now fixed. The plan assumed
directory creation simply moved to LocalDb along with file ownership. It did
not: the LocalDb library never creates the parent directory, and
SqliteLocalDb opens the file eagerly in its constructor — so a missing
directory is a hard boot failure ("SQLite Error 14: unable to open database
file"), not a degraded start. The default site config points at the RELATIVE
path ./data/site-localdb.db, so any site node without a pre-existing data/
directory fails to boot. The docker rig escapes only because its volume mount
happens to create /app/data — a coincidence that would have hidden this until
a bare-metal or fresh deployment. This has been latent since Phase 1 made
LocalDb:Path required; deleting S&F's EnsureDatabaseDirectoryExists here
would have widened it. Re-established the guarantee at the layer that now
owns the path (SiteLocalDbDirectory.Ensure, called before AddZbLocalDb) and
pinned it with SiteLocalDbDirectoryTests. Non-vacuity is not assumed: two
tests written against the wrong assumption failed with exactly this
SQLite Error 14 before the fix existed.
2. Test fallout was ~7x the plan's estimate. The plan named "fixtures" in one
project; the constructor change actually reaches 40 files across 7 test
projects, and most used Mode=Memory;Cache=Shared — which LocalDb has no
equivalent for, so every one had to move to a real temp file. Rather than
copy the Phase 1 TestLocalDb fixture into 7 projects, added a shared
tests/ZB.MOM.WW.ScadaBridge.TestSupport library (not a test project) so the
WAL-sidecar cleanup and the "real, not stubbed" rationale live in one place.
Retargeted rather than deleted, in both directions: the S&F WAL test now asserts
against the LocalDb-backed store (WAL genuinely is LocalDb's job), while the
directory-creation test moved to Host.Tests (that guarantee is NOT LocalDb's).
SiteStorageServiceTests.Initialize_EnablesWalJournalMode got the same treatment.
DeploymentManagerMediumFindingsTests induced a persistence failure via an
unopenable path, which no longer reaches the assertion since the fixture now
throws first; it induces the same failure shape via an uninitialized store.
Verified: full solution build 0 warnings; SiteRuntime 532, Host 318,
AuditLog 355, ExternalSystemGateway 142, HealthMonitoring 97,
StoreAndForward 153 — 1597 passed, 0 failed.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
The Phase 2 soak's "LocalDb fails under load" blocker is NOT a product defect.
Root cause: host-side (macOS) sqlite3 reads of the live, bind-mounted WAL
databases. POSIX advisory locks do not propagate across the virtiofs boundary,
so the host reader believes it is the only connection, checkpoints on close and
resets the WAL to 0 bytes under the container. The container's still-mapped
WAL index then references frames that no longer exist and every subsequent
statement fails SQLITE_IOERR_SHORT_READ (522) -> primary code 10, permanently
until the process reopens the database.
Reproduced on demand both on the rig (one sqlite3 SELECT reset a 4.6 MiB WAL and
produced the first error one second later) and in a minimal python:3.12-alpine
repro with no LocalDb or .NET involved. Refuted the converse: a freshly-reopened
node sustained the full soak load 10+ minutes with zero errors.
The original brief's isolation was confounded - both nodes had been poisoned by
the same sampling pass, and a poisoned standby looks healthy only because it
issues almost no statements. Corrections annotated in place.
Hardening shipped alongside:
- SqliteErrorCodes.Describe: log SQLite primary AND extended codes at the
LocalDb-adjacent catch sites (the missing extended code is what made the
original diagnosis so slow).
- SiteAuditTelemetryActor: stop touching ActorContext across an await
(NotSupportedException), with regression coverage.
- infra/reseed.sh: apply the MSSQL init scripts explicitly. The official
mssql/server image does not implement /docker-entrypoint-initdb.d, so a fresh
volume hung the reseed forever; compose mounts annotated as informational.
Deliberately NOT done: detect-and-reopen self-heal in SqliteLocalDb. It defends
only against external interference, which is now prevented at the source, and
same-kernel production readers see the locks correctly.
Build 0 warnings; SiteAuditTelemetryActorTests 9/9, SiteEventLogging 70/70.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
Task 4 of the LocalDb Phase 1 adoption plan.
OperationTrackingStore took IOptions<OperationTrackingOptions> and opened its own
SqliteConnection from ConnectionString. It now takes ILocalDb and gets every
connection - writer and the two ad-hoc reader paths - from CreateConnection().
This is not cosmetic. OperationTracking is a RegisterReplicated table, so its
capture triggers call zb_hlc_next(). That UDF is registered per connection by
ILocalDb and by nothing else, so a raw SqliteConnection would fail closed on
every write. Connections from CreateConnection() also arrive already open -
calling Open()/OpenAsync() on one throws - hence the removed OpenAsync calls on
the reader paths.
OperationTrackingOptions.ConnectionString is now vestigial for this store; the
database location is LocalDb:Path. The options class stays (retention settings)
and the config key stays bound for the site config DB.
InitializeSchema is kept but is now always a no-op in the host: onReady runs
while ILocalDb is being constructed, strictly before this constructor can
receive it. It remains so a directly-constructed store (tests, tooling) is
self-sufficient.
Tests: the store fixture moves off mode=memory&cache=shared onto a real ILocalDb
over a temp file. There is no in-memory mode - LocalDbOptions.Path is a
filesystem path - and testing through a raw in-memory connection would no longer
resemble the host. Verifier connections stay raw on purpose: this fixture never
registers the table, so no trigger fires and the UDF is never reached.
Three DI fixtures needed LocalDb:Path added, because resolving
IOperationTrackingStore now forces ILocalDb construction:
SiteCompositionRootTests, SiteAuditWiringTests, and the AuditLog
CombinedTelemetryHarness.
Verified: build 0 warnings; Host 294/294, SiteRuntime 530/530, AuditLog 354/354.
Claude-Session: https://claude.ai/code/session_01BL2Vu1ESDQ9SCN4gVKkdts
The NuGetAuditSuppress in Directory.Packages.props was masking a LIVE high-severity
vulnerability, not documenting an accepted one. Only the Host project resolved a
patched SQLitePCLRaw.lib.e_sqlite3 2.1.12 (transitively, via ZB.MOM.WW.Auth.ApiKeys).
Every other SQLite consumer - AuditLog, SiteRuntime, StoreAndForward, SiteEventLogging
and 11 test projects - still resolved the vulnerable 2.1.11.
The suppression's rationale was factually wrong: it claimed 'the only patched native
lib is the SQLitePCLRaw 3.x line'. 2.1.12 patches this advisory within the 2.1.x line,
so the feared risky force-override of the whole family to 3.x is unnecessary.
Fix: explicit PackageReference to the patched 2.1.12 in each SQLite-consuming project,
plus the central PackageVersion row. Suppression removed, so the advisory is audited
again rather than silenced.
Rejected alternative: CentralPackageTransitivePinningEnabled. It clears the advisory in
one line but makes every central version a ceiling for transitive resolution, demanding
bumps to Google.Protobuf, Grpc.Net.Client, Microsoft.Data.SqlClient and Newtonsoft.Json.
That is a gRPC/data-access change to a production SCADA platform and deserves its own
reviewed commit.
Verified: forced full-solution restore reports no NU1903 (only the pre-existing,
unrelated AngleSharp NU1902 in CentralUI.Tests); all four previously-vulnerable src
projects now resolve 2.1.12; 55 projects build 0 warnings / 0 errors; 1108 tests pass
across the SQLite layer (AuditLog 354, ConfigurationDatabase 357, Security 181,
StoreAndForward 152, SiteEventLogging 64).
The host does not guarantee IHostedService.StopAsync is driven before the DI
container is disposed — WebApplicationFactory's teardown reaches Dispose first
— so cancelling the internal CTS from StopAsync threw ObjectDisposedException
and aborted the host's whole shutdown sequence. Four services shared the same
copy-pasted lifecycle and the same two races: StopAsync cancelling an already-
disposed CTS, and StartAsync reading _cts.Token lazily inside the Task.Run
lambda, which faults the loop task the host awaits when Dispose wins that race.
Each service now captures the token on the caller's thread, tolerates a
disposed CTS, and cancels-before-disposing so the loop is always signalled and
its pending Task.Delay sees a cancelled token rather than a dead source.
SiteAuditBacklogReporter also gains the outer OperationCanceledException guard
its sibling SiteAuditRetentionService already carried (arch-review 04 R2, R7),
without which a shutdown landing mid-probe threw TaskCanceledException out of
Host.StopAsync.
Surfaced while verifying the Gitea #15 test-harness fix: in Host.Tests the
aborted teardown skipped the fixture's env-var restore, contaminating every
later test in the run.
Refs: Gitea #15
Consolidates the reporter's poll cadence onto SqliteAuditWriterOptions instead of
the hard-coded 30 s constant, closing deferred-work register item #21 (the config-
shape cleanup its own XML doc flagged as a follow-up).
- SqliteAuditWriterOptions.BacklogPollIntervalSeconds (default 30).
- Reporter reads it via IOptions; precedence explicit-override (tests) > configured
> 30 s default; a non-positive value falls back to the default. Corrected the
stale "hard-code / tunable in a follow-up" class-doc.
- Cadence tests + register updated (row 21 -> Resolved).
Claude-Session: https://claude.ai/code/session_01MtdgwpEeCUn6cUA5f1LMPj
SiteEntry gains an additive FallbackGrpcEndpoint; SiteEnumerator now picks NodeA
else NodeB as primary (a NodeB-only site is no longer skipped) and carries a
distinct NodeB as the fallback. Both GrpcPull{AuditEvents,SiteCalls} clients dial
the primary first and, on a transport fault (Unavailable/DeadlineExceeded/
Cancelled/HttpRequestException/SocketException/OperationCanceled), fail over to
NodeB once before collapsing to empty — a mapping/unexpected fault does NOT fail
over. Keeps the reconciliation loss-recovery net available during a NodeA outage.
When a pulled AuditEvent fails to insert on every retry up to the permanent-
abandon threshold, the reconciliation actor now writes ONE synthetic
ReconciliationAbandoned row (new AuditKind) alongside the Critical log line —
fresh EventId, Status=Failed, channel preserved from the lost row, Extra carrying
the abandoned EventId + source site + final error — via the same
ScadaBridgeAuditEventFactory the ingest path uses. Best-effort in its own
try/catch so it never blocks the cursor twice. The permanent loss is now
queryable in the Audit Log, not only in a rotating log file.
Reconciliation cursor becomes composite (UpdatedAtUtc, TrackedOperationId);
IPullSiteCallsClient/GrpcPullSiteCallsClient forward the Task-15 after_id keyset
(additive param, null preserves the legacy inclusive >= contract). A burst
sharing one exact UpdatedAtUtc now drains via the id tiebreak instead of pinning
forever. A legacy site that ignores after_id is latched + published as
SiteCallReconciliationPinnedChanged on the EventStream (transition-only),
replacing the prior silent log line.
Joseph Doherty