`dotnet ef migrations script --idempotent` puts each migration in ONE batch
behind IF NOT EXISTS guards, and SQL Server compiles the whole batch before any
guard runs. AddAuditLogHashChain adds ChainSeq and then creates two filtered
indexes whose WHERE clause references it in the same batch, so the script
failed with Msg 207 "Invalid column name 'ChainSeq'" on any DB at migration
<= 60 (the 2026-11-11 vd03 deploy hand-inserted GO separators instead).
- AddAuditLogHashChain: the two CREATE INDEX statements go through EXEC(N'...')
so they compile at execution time. Same indexes, same snapshot; Migrate() /
database update unaffected; DBs that already ran the migration never run it
again. The column-level GRANT compiles fine (checked at execution).
- Audit of 58-64: no other migration hits the pattern (the idempotent script
from 57 now runs clean end to end; AddConfigGraphRowVersions' plain CREATE
INDEX on a new column is not compile-checked).
- docs/plans/sql/AddAuditLogHashChain.sql: same EXEC wrap. It, and
PrefixSearchIndexes.sql / AddConfigGraphRowVersions.sql, also carried a
stray UTF-8 BOM after the prepended header comment that made sqlcmd fail
with Msg 102 on line 1 of the batch; removed.
- New IdempotentMigrationScriptTests: migrates the fixture DB to 57, runs the
generated idempotent script batch by batch, asserts every migration applied,
then re-runs it as a no-op. Fails with Msg 207 without the migration fix.
Co-Authored-By: Claude <noreply@anthropic.com>
Variant B2 of the #78 benchmark: migration FlipAuditLogClusteredKey drops the
twelve non-clustered indexes, re-adds PK_AuditLog as (OccurredAtUtc, EventId)
on ps_AuditLog_Month, adds the aligned unique UX_AuditLog_EventId_Occurred
(EventId, OccurredAtUtc) for the idempotency probe, and rebuilds the twelve
indexes unchanged. Guarded (re-runnable) Up, inverse-rebuild Down; production
scripts docs/plans/sql/FlipAuditLogClusteredKey{,.rollback}.sql.
The partition-switch purge staging table now reads the live clustered key
order from sys.index_columns, so it switches either shape (old binaries
hard-code the old order: ship with the binaries, both centrals stopped).
Evidence: 300k rows, page 1 90,532 -> 34 logical reads (ordered backward
clustered scan under Top), keyset page 90,532 -> 46, probe 14 -> 14;
FlipAuditLogClusteredKeyMigrationTests pins indexes, grants, Down/Up, the
staging key (mutation: hard-coded old key fails SWITCH) and the read ratio.
Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
Second slice of the 2026-09-09 whole-system review remediation
(docs/plans/2026-09-09-arch-review.md §10), on top of the High slice 146a1323.
Runtime/devices: SR-2 alarm on-trigger integration scope; SR-3 correlated NativeAlarmActor
subscribe retry; SR-4 immutable audited DB target; SR-5 ordered InstanceActor stop/flush;
SR-12 rehydration fence; SR-13 S&F gate recheck per delivery; DC-4 no OPC UA security
downgrade; DC-5 alarm snapshot for every new subscriber; ORCH-1 sync-dispose joins the pump.
Management/UI: MG-2 site scope on audit tree + Flow Debugger Replay; MG-3 stale-probe memo
removed; MG-4 server-owned bulk-deploy deadline; UI-5 sandbox runs in a killable child
process (new CentralUI.SandboxWorker project); UI-14 bounded sandbox output/IPC.
Edge/audit: EI-1 relay audit ownership; EI-2 SkipBodyCapture suppresses error text; EI-3/EI-11
error-field redaction + caps; EI-4 single relay deadline, pre-execution-only retry; EI-5
authenticated relay provenance; EI-10 body cap before audit read.
Persistence/contracts: DL-1 global Instance.UniqueName unique index + conflict translation;
DL-2 DataConnection protocol config encrypted at rest (migration
20260909203046_EnforceGlobalInstanceNamesAndProtectProtocolConfiguration — coordinated
central upgrade, old binaries cannot read protected rows); DL-3 bounded Replay paging;
DL-6 cross-site Overwrite of a deployed instance rejected; XC-4 proto descriptor contract test.
Config/hygiene: XC-1 central listener isolation by local port; XC-2/XC-3 inert SMTP + cron
keys removed from rig settings; CM-2 env2 bootstrap guard enabled; DC-6 deploy site exit code;
DC-7 notifier retries only explicit standby rejection; XC-5 ClusterClient purged from docs.
Gates recorded in the report: build 0/0, 9 397 passed / 0 failed, rig redeployed on all
eight nodes with the migration applied. 46 Low findings remain open.
AuditLog.Tests flaked under parallel full-solution runs. Three distinct
mechanisms, all load-sensitive and all green in isolation.
1. MsSqlMigrationFixture create+migrate saturation (the register's headline).
Database NAMING was already isolated (a guid per fixture), but nothing
bounded how many fixtures ran CREATE DATABASE + the ~100-migration chain at
once. xunit parallelises processor-count collections PER PROCESS, and a
solution-level `dotnet test` starts several MSSQL-backed processes together,
so the dev container saw dozens of simultaneous fresh-database migrations.
That is resource saturation, not object-level locking, and it pushed
individual DDL statements past EF's default 30s command timeout — every
failure had the identical fingerprint the register recorded:
MsSqlMigrationFixture..ctor -> Migrator.MigrateAsync -> SqlException
"Execution Timeout Expired" (error 258), which xunit surfaces as
"Class fixture type ... threw in its constructor" against every test in the
class.
Fix: a machine-wide counting semaphore over the create+migrate window, built
from sp_getapplock on master (ScadaBridgeTestMigrate/0..N-1, default 4 slots,
SCADABRIDGE_MSSQL_TEST_MIGRATE_SLOTS to override). An xunit collection cannot
do this — the contention is cross-PROCESS, so the arbiter has to live on the
shared server. The gate connection is unpooled (a pooled connection defers the
implicit release of a session-scoped app lock until reuse) and releases
explicitly; a crashed process drops its connection and SQL releases the lock,
so the gate cannot wedge the suite, and a deadline miss proceeds ungated
rather than failing. Migration commands also get a 300s timeout and
CREATE/DROP DATABASE 120s: under a bounded queue the server is busy but
progressing, so waiting is the correct response. Plus a once-per-process
best-effort sweep of ScadaBridgeAuditMigTest_* databases older than 2h so a
killed run does not leave strays, and the connectivity probe now retries
(3 attempts) — a busy server misread as an absent one would silently turn
every MSSQL test into a Skip, which reads green and is worse than a red.
2. PartitionPurgeTests leaked AuditLogPurgeActors. Each ticks every 100ms and
switches out the Jan-2026 partition; they were only stopped implicitly by
TestKit Dispose -> Shutdown, which is timeout-bounded and does not fail when
it expires. Under load a previous test's actor was still ticking against the
shared per-class fixture DB, so PerChannelOverride_... found its seeded
Jan-15-2026 row already gone (Expected: 1, Actual: 0).
3. NotifyDispatcherAuditTrailTests leaked NotificationOutboxActors, same shape.
The dispatcher claims EVERY due row, not just the test's siteId, so a leaked
actor delivered the next test's notification through its own exhausted
QueuedOutcomeAdapter — a transient outcome, leaving the row in Retrying
(Expected: Delivered, Actual: Retrying) and audit counts short.
Both fixed by stopping the actor inside the test that created it
(Watch/Stop/ExpectTerminated), removing the dependency on shutdown timing.
Evidence. Reproducer: 10 concurrent ConfigurationDatabase.Tests processes
against the one dev SQL Server — before, 10/10 processes failed (17-77 failures
each, 100% the ctor->MigrateAsync timeout); after, 3 consecutive runs 10/10
green, 395/395 each. Realistic shape (3x AuditLog.Tests + 1x
ConfigurationDatabase.Tests concurrent): first post-fix run still showed the
NotifyDispatcher leak, then 3 consecutive clean runs. Standalone: AuditLog.Tests
383/383, ConfigurationDatabase.Tests 395/395 (12s, unchanged from baseline),
SiteCallAudit.Tests 66/66. No orphan databases or stuck app locks left behind.
docs/test_infra/test_infra_db.md documents the ephemeral per-test-class
databases, the throttle and its env override, the timeouts, and the stray sweep.
Perf re-baseline (HotPathLatencyTests): empirical p95 on Apple M-series Release
build: 4KB DetailsJson slow path ≈14 µs, small-DetailsJson no-redactors ≈2 µs,
true no-op fast path ≈0 µs. Thresholds updated: 200 µs / 30 µs / 5 µs (≈15×
headroom for contested CI runners). Old thresholds (50 µs / 10 µs) were set for
the pre-C3 typed-field path; canonical JSON parse+rewrite is empirically faster.
Adds a third test (Filter_Apply_NoDetailsJson_FastPath) that asserts same-instance
return on the DetailsJson-null + within-cap fast path. Env-var overrides retained.
CollapseAuditLogToCanonicalMigrationTests (new): three MSSQL-gated [SkippableFact]
tests verifying Action/Category/Outcome projection, NULL Actor, DetailsJson codec
round-trip, and all six persisted computed columns (Kind/Status/SourceSiteId/
ExecutionId/ParentExecutionId) for ApiOutbound, InboundAuthFailure, and Failed-
status rows.
AddAuditLogTableMigrationTests: rename CreatesFiveNamedIndexes →
CreatesNineNamedIndexes; expand coverage from 5 original indexes to all 9 named
non-clustered indexes present after CollapseAuditLogToCanonical (adds
IX_AuditLog_Execution, IX_AuditLog_ParentExecution, IX_AuditLog_Node_Occurred,
UX_AuditLog_EventId).
Dead-cref cleanup: zero references to the deleted IAuditPayloadFilter /
DefaultAuditPayloadFilter / SafeDefaultAuditPayloadFilter types remain in any
.cs file (source or test). 26 occurrences across 13 files replaced with correct
references to IAuditRedactor / ScadaBridgeAuditRedactor / SafeDefaultAuditRedactor
or reworded as plain prose.
Residual sweep: no unused transitional code found beyond the acknowledged
"C3 transitional shim" comments on IngestedAtUtc stamping (active code, not dead).
Renames the 13 SCADALINK_* runtime env vars → SCADABRIDGE_*, the ScadaLink__
.NET config keys → ScadaBridge__, the stale ScadaLink.Host.exe assembly name
→ ZB.MOM.WW.ScadaBridge.Host.exe, the scadalink_app SQL login → scadabridge_app,
and residual identifiers/comments/docs. Migration records (prior rename
tooling/design, DB-rename helper, this scrub script) carved out.
Adds tools/scrub-scadalink-refs.sh.
Joseph Doherty