From dcfcf822af08a1396f17c51e467c5c41454301b1 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Mon, 17 Aug 2026 05:41:56 -0400 Subject: [PATCH] =?UTF-8?q?docs(register):=20Wave=205=20close-out=20?= =?UTF-8?q?=E2=80=94=20close=20rows=2037/86/87/88=20(deployment-read=20con?= =?UTF-8?q?solidation=20+=20Administrator=20system-wide=20fix,=20live-prob?= =?UTF-8?q?ed),=2089/90=20(DCL=20definitions-overlay=20arbitration,=20prem?= =?UTF-8?q?ise=20corrected=20twice,=20live-probed),=2091/95/96=20(flake=20?= =?UTF-8?q?family=20+=20parallelism=20policy);=20add=20rows=2097-100=20(li?= =?UTF-8?q?ve-connection=20reconfigure=20gap,=20fail-open=20scope=20gate,?= =?UTF-8?q?=20case-sensitive=20audit=20--status,=20process-global=20compil?= =?UTF-8?q?e-cache=20Clear)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../2026-07-08-deferred-work-register.md | 22 +++++++++++-------- 1 file changed, 13 insertions(+), 9 deletions(-) diff --git a/docs/plans/2026-07-08-deferred-work-register.md b/docs/plans/2026-07-08-deferred-work-register.md index b9e95434..5a1a23c7 100644 --- a/docs/plans/2026-07-08-deferred-work-register.md +++ b/docs/plans/2026-07-08-deferred-work-register.md @@ -30,7 +30,7 @@ All 7 fix-now items landed via PLAN-04/05/06/07/08 (verified in review 08 round | 34 | ~~**`site_events` purge burst is unbounded in the oplog and unmeasured at scale**~~ **RESOLVED 2026-08-16 (`d9a35846`) — measured, and paced.** Measured with the REAL `EventLogPurgeService` over a real `RegisterReplicated` LocalDb file + a loopback sync pair at the rig's `MaxBatchSize: 64`: 393 B/row incl. indexes ⇒ a 1 GB event log holds **~2.73 M rows**, and expiring them in one tick took 285 s and queued **2 731 947 oplog rows = 2.7x LocalDb's `MaxOplogRows` default (1 M)** — enough to make the maintenance tick prune the oplog and flag a FULL ten-table snapshot resync, bought by a routine retention purge. The decisive nuance: the *resident* backlog is produced−minus−drained, so a healthy peer blunts it (391 k peak on loopback, drained 6.0 min) — the old safety margin was a function of peer throughput and vanished with the peer down. Fix: `MaxPurgedRowsPerTick` (default **250 000**, `0` = unlimited escape hatch), one budget SHARED across retention + cap trim; a ceiling-limited tick reschedules after `PurgeCatchUpInterval` (**5 min**, validated `> 0` and `<= PurgeInterval`) so worst-case retention converges in ~55 min instead of 11 days; purge loop is now a delay loop anchored to the END of the previous purge; rate-limited Warning + counter `scadabridge.site.event_log.purge_ceiling_hits`. No purge-exemption path added. Row-33 interplay verified empirically (500 k drained in 46 s vs the 15-min dwell). Fail-before: `PurgeBudget` neutralised → 6 of the new tests fail, all 19 pre-existing pass. Measurement + decision record: `docs/plans/2026-08-15-site-events-purge-burst-measurement.md`. Side-finding recorded as row 81 (storage cap unenforceable on a replicated node). | Found by R7 | Closed 2026-08-16. | — | | 35 | ~~**`DeploymentRecords` index does not cover the deployment-list sort key**~~ **RESOLVED 2026-08-15 (`be29bf47`).** Migration `20260815165452_AddDeploymentRecordListSortIndex` + hand-written `docs/plans/sql/AddDeploymentRecordListSortIndex.sql`: composite `IX_DeploymentRecords_DeployedAt_Id (DeployedAt DESC, Id DESC)`; the old single-column index DROPPED as strictly redundant (`Id` is the clustered PK and was already in the old index's leaf as the row locator — same leaf width, now a declared ordered key; keeping it would cost a second index on every insert/status transition for no read benefit). Serves the three reads sharing the key (`QueryDeploymentListPageAsync`, `QueryDeploymentSummariesAsync`, `GetCurrentDeploymentStatusAsync`). `Id` in the key is also a CORRECTNESS matter: a bulk site deploy stamps many rows in one `DeployedAt` instant, so without the tie-break keyset pages can repeat or drop rows. Recorded caveat: the optimizer may already have served the ordering via a backward scan of the old index, so the "sorts per page" premise was possibly optimizer-dependent — the composite makes it deterministic and self-documenting; an actual plan capture against the rig would quantify the before/after. The ConfigurationDatabase suite exercises the descending composite as real DDL via SQLite `EnsureCreated()`, proving provider portability. | Found by R3 (`35ce1413`) | Closed 2026-08-15. | — | | 36 | **Deployments free-text search is unindexable by construction** | Found by R3; re-verified 2026-08-17 (Low-tail sweep) | Four `LIKE '%term%'` predicates (instance name, deployment id, revision hash, user) — leading wildcards preclude index seeks, so a search is a scan of the join (`DeploymentManagerRepository.cs:100-104`; only searched column with any index is `DeploymentId`, unseekable under a leading wildcard). Acceptable at the purge-bounded row count. **2026-08-17 refresh: the row undercounted the cost by half** — `BuildListQuery` is shared with `GetDeploymentStatusCountsAsync`, so one search = TWO scans of the join (list + status tiles), mitigated by the UI's trailing-edge debounce (one pair per pause, not per keystroke). Anchoring the id/hash legs was evaluated and REJECTED as a micro-fix: `DeploymentListFilter.Search`'s doc comment promises substring matching and a 4-case `[Theory]` deliberately pins mid-string matches — changing it is a contract renegotiation, not an optimization. Fix if hot: renegotiate to prefix on id/hash legs, or full-text. *(Low)* | Search latency complaints on the Deployments page | -| 37 | **Two paged reads of `DeploymentRecords` with different contracts** | Found by R3; re-verified 2026-08-17 (Low-tail sweep: both shapes intact, `DeploymentManagerRepository.cs:110-149` and `:399-449`) | `QueryDeploymentSummariesAsync` (CLI/ManagementActor: offset, no total, no instance join, instance-id-list scoping) and R3's `QueryDeploymentListPageAsync` (UI: total + instance join + site-id scoping) duplicate query logic. Consolidating on one contract — giving the CLI a total, both scoping by site — removes the duplication. **2026-08-17: the duplication has now DRIFTED into real behavioral divergences, recorded as rows 86–88 (site-scope authorization divergence, CLI unbounded read + non-transactional scope resolution, silent `--status` parse fallback) — a consolidation should resolve those rows with it.** *(Low structurally; the drift itself is row 86, Medium)* | Rows 86–88, or next time either query's shape changes | +| 37 | ~~**Two paged reads of `DeploymentRecords` with different contracts**~~ **RESOLVED 2026-08-17 (merge `ad0100f2`, agent commit `9f67147e`).** `QueryDeploymentSummariesAsync` and `DeploymentRecordSummary` are DELETED; the CLI/HTTP `QueryDeployments` handler now rides `QueryDeploymentListPageAsync` + `DeploymentListFilter` — one row shape (`DeploymentListRow`), one scoping mechanism (site-id via the query join), total count available to both surfaces (`DeploymentListPage` now also owns `MaxPageSize` + `DefaultPageSize = 50`). Rows 86–88 were resolved with the consolidation, as this row anticipated. Row 36's 4-case substring-search Theory untouched and still passing. Remaining UI-vs-CLI asymmetry (`GetDeploymentStatusCountsAsync` status tiles are UI-only) is deliberate, not drift. **Live-probed on the rig:** CLI `deploy status --status Success --page-size 5` returns the consolidated page — rows now carry `siteId` + `instanceUniqueName`, and the response carries `totalCount` (11) the CLI never had. | Found by R3; consolidation shipped with rows 86–88 | Closed 2026-08-17. | — | | 38 | **OtOpcUa: `RoslynVirtualTagEvaluatorTests` racing-clear test is load-fragile** (OtOpcUa repo, not this one) | Found by R4 during the full-suite bump verification | `Evaluate_racing_ClearCompiledScripts_never_fails_with_disposed` fails under full-solution parallel test load, passes 3/3 isolated: its design assumes at most one `ClearCompiledScripts` lands inside one evaluation (so the product's one-shot retry suffices); under contention two can. Same timing-assumption family as the R5 sweep, but lives in OtOpcUa. Test-side fix: bound clear cadence relative to observed evaluation latency, or assert on retry-exhaustion instead of zero failures. | Next OtOpcUa test session, or when it fails a run that matters | | 39 | **OtOpcUa: late-enabled replication cannot baseline pre-existing rows** (OtOpcUa repo, not this one) | Found by R4; documented in OtOpcUa's CLAUDE.md | `LocalDbSetup.OnReady` registers all four tables unconditionally on every driver node, so it can never pass `baselineExistingRows: true` (only correct when gated on replication being configured). Turning replication ON for a long-running default-OFF node converges on subsequent writes only — existing rows are never baselined. ScadaBridge closed this class with conditional registration + boot-time `DeregisterReplicated` self-heal (WP1.3/WP3.3); porting it is a behavior change beyond a pin bump (registered-set pinned by tests, deregistration must be pair-symmetric). Low current impact: site-a was enabled from fresh. | Before enabling replication on an established OtOpcUa node | | 40 | ~~**Wall-clock sleeps used as synchronization in tests (absence-assertion class)**~~ **RESOLVED 2026-08-15 (`7a7d345b`).** All six listed sites replaced with observable edges + positive controls, strengthening proven by stash-based negative controls: `ScriptDeadlineAtEnqueueTests:128` now pins the watchdog via a genuinely-wedged positive control (armed strictly later, self-releasing) + per-script absence assertion + detach counter pinned at 1 — **disabling the entire stuck-script watchdog passed the OLD test 2/2 and fails the new one**; `:105` now proves the run was still queued when its budget expired via `QueueDepth` edges (the actual design claim — the old sleep asserted nothing about it); `DeploymentWarmThenGateTests:117` (+2 sibling `Task.Delay(500)`s) — the empty-site zero is now the manager's own observation (phantom-startup control: old passed 2/2, new fails); `SiteAlarmLiveCacheServiceTests:195,216` poll a new subscription-cancellation observable across 3 linger windows, each closing with a positive control (control: linger cancellation disabled + viewer guard dropped — `Resubscribe_Within_Linger` old passed 2/2, new fails; honest caveat: the other case's old later assertion did already catch that break, so its gain is the observable + control, not a false-pass fix); `DataConnectionActorTests:435` waits for the Connected health publication so the subscribe provably runs against Connected (the sleep guarded no assertion — determinism gain only). Bonus: the de-padded schedule exposed a LATENT race in `InstanceActorChildAttributeRaceTests` (~1-in-6 under load, A/B'd 14/14 green on the unmodified tree so pre-existing) — fixed with a bounded-deadline resolution retry; pattern recorded as row 77. | Found by the R5 sweep (`9fb52153`) | Closed 2026-08-15. | — | @@ -78,17 +78,21 @@ All 7 fix-now items landed via PLAN-04/05/06/07/08 (verified in review 08 round | 83 | ~~**Post-boot start-path tests must gate on the DM's first instance-count report — `ReconcileStartsInstanceTests` still uses a bare `Task.Delay(500)`** *(Low, test-infra)*~~ **RESOLVED 2026-08-17 (`524bdbbe`).** `ReconcileStartsInstanceTests` converted to the first-`SetInstanceCounts`-report gate; the pattern was extracted into a shared `TestSupport/DeploymentManagerStartupProbe` now used by both this suite and `InstanceSweepSelfHealTests`, so the convention has one implementation to copy instead of two to keep in sync. SiteRuntime 633/633 green. | Row-78 fix session | Closed 2026-08-17. | — | | 84 | ~~**`LogReplicationBacklogNote` can fire twice in one purge tick** *(Low, cosmetic)*~~ **RESOLVED 2026-08-17 (`ddb3c32e`).** The note is emitted at most once per tick. Fail-before: `RunPurge_WhenBothPhasesDeleteRows_LogsTheBacklogNoteOnceForTheWholeTick` fails pre-fix with `Assert.Single(): collection contained 2 matching items`. | Row-34 session | Closed 2026-08-17. | — | | 85 | ~~**DCL connection actors are not spawned for connection configs that arrive via LocalDb replication — row 78's gap, one layer down** *(Medium-Low, availability; proven live 2026-08-16 by the row-78 close-out probe)*~~ **RESOLVED 2026-08-17 (`8f2c2813`) — with a premise correction AND a widened blast radius.** The row's mechanism was wrong: `data_connection_definitions` has NO production reader (its getter is test-only) — connection actors are created solely from the `Connections` map in an instance's own flattened config via `EnsureDclConnections`, so there was never a store→actor path to repair. The real defect: that ensure was the CALLER'S job and only 2 of 4 instance-start paths did it (boot batch ✅, `ApplyDeployment` ✅, `StartStoredInstance` ❌ — both the row-67 ensure-started and the row-78 sweep — and `HandleEnableResult` ❌; the two misses are exactly the two paths added later). So the bug was NOT replication-specific: an ordinary unreplicated site that boots with an instance disabled and enables it later hit the same dead end, because the boot load reads only enabled configs. Fix is structural: `EnsureDclConnections` moved INTO `CreateInstanceActor` (the documented single creation funnel), before the already-exists guard so the two working paths keep exact semantics; reuses the per-name hash guard so steady state is silent; a fifth path cannot forget. Store-level sweep pass explicitly REJECTED (memo §7.4): it wouldn't have fixed this (the table is empty on deploy-only sites) and the unguarded form ping-pongs the hash slot against the deploy path, dropping live subscriptions every tick. Fail-before by behavioral revert: 4 of 7 new tests fail (sweep-started, reconcile-ensure-started, enable-after-disabled-boot, exactly-once-across-sweeps). SiteRuntime 633/633, DCL 320/320. Sibling gaps found during the fix recorded as rows 89/90. | Proven live 2026-08-16 by the row-78 close-out probe (sweep started `rig-probe-1`/`soak-notify-1` but every tag subscribe failed for 6+ min with `Unknown connection: OPC PLC Simulator — will retry`; a second pair restart healed it) | Closed 2026-08-17. **Live-proven on the rig same day** in the exact original failure sequence (pair stopped, node-a `site-localdb.db` quarantined as `.probe85-quarantine-2026-08-17`, pair restarted): reconcile "0 fetched" at 08:29:59, then at the 08:30:54 sweep tick `Created DCL connection OPC PLC Simulator` fired ALONGSIDE the two self-heal Warnings (pre-fix they appeared alone), zero `No connection actor`/`Unknown connection` lines on either node, all three `rig-probe-1` tags Good with live values ~2 min after restart — NO second restart; steady-state silence confirmed (0 Created/Updated lines across subsequent ticks); replication reseeded (`localDbReplicationConnected: true`, backlog 0), site-b/c still correctly omit the fields. | — | -| 86 | **CLI and UI disagree on deployment site-scoping for Administrators** *(Medium, authorization)* | Found by the 2026-08-17 Low-tail sweep while re-verifying row 37 | `ManagementActor.cs:2586` grants an **Administrator role bypass** of site scoping on `deploy status`, while the UI's `SiteScopeService` is role-independent (system-wide iff the user carries zero `SiteId` claims). An Administrator carrying site claims therefore sees the whole fleet via the CLI but only their sites on the Deployments page — same user, same data, two answers, and one of them is wrong. This is row 37's duplication having drifted into behavior; decide which semantic is intended (role bypass vs claims-only) and enforce it on BOTH paths, ideally as part of the row-37 consolidation. | Next ManagementService/CLI session, or the row-37 consolidation — whichever comes first | -| 87 | **CLI deployment scoping is non-transactional and unbounded** *(Low)* | Found by the 2026-08-17 Low-tail sweep | The CLI path resolves site grants to an instance-id list in a separate, non-transactional read (`ManagementActor.cs:2590-2596`), reloading EVERY instance per scoped query; and `PageSize: 0` on the wire maps to `take: null` = "no limit" (`DeploymentManagerRepository.cs:434`), materializing the entire retention window in one query. The UI path is structurally incapable of both. Bounded today by the purge window and low CLI traffic. | Row-37 consolidation, or first slow/OOM `deploy status` report | -| 88 | **Unparseable CLI `--status` silently means "no filter"** *(Low, UX)* | Found by the 2026-08-17 Low-tail sweep | `ManagementActor.ParseDeploymentStatus` returns null on an unparseable status value, so a typo (`--status Sucess`) silently returns ALL statuses instead of an error — the caller cannot distinguish "filter applied, nothing matched" from "filter ignored". Fix: fail loudly on an unrecognized value (mirror the row-46 principle: loud failure over silent empty/full results). | Next CLI touch to the deploy command group | -| 89 | **Connection-only artifact deploy does not reach the peer node's DCL** *(Medium-Low, availability)* | Found by the 2026-08-17 row-85 fix session | `data_connection_definitions` IS one of the ten replicated tables, but only the node that received the deploy runs `HandleApplyArtifactDataConnectionsToDcl` — the peer gets the row via CDC and keeps serving the OLD endpoint until a restart or an instance redeploy touches it. Not fixable by another sweep: applying a definition change to a live connection needs an arbitration decision first (does the instance's flattened-config `Connections` hash or the standalone definition hash win when they disagree?), because the row-85 fix deliberately made the flattened config the sole creation source. | Next artifact-deploy or DCL session — decide the arbitration, then apply-on-CDC becomes safe | -| 90 | **No removal path for a connection deleted on the peer** *(Low, mirror image of row 89)* | Found by the 2026-08-17 row-85 fix session | `RemoveConnectionCommand` is sent only from the artifact-deploy reconcile on the node that processed the deploy; a connection deleted on the peer leaves a live connection actor here until restart. Bounded: the orphan actor serves a connection no instance references, wasting a device session but corrupting nothing. Same arbitration prerequisite as row 89 — fix them together. | With row 89 | -| 91 | **`SiteScriptCompileCacheTests.GetOrAdd_SameCodeAndGlobals_ComputesOnce` is parallel-load fragile** *(Low, test-infra; row-82 family)* | Found by the 2026-08-17 row-85 fix session: failed once in 4 full-suite runs, green in isolation AND green on base — pre-existing, probability merely raised by the suite growing | `SiteScriptCompileCache` is a process-wide static and the test asserts an absolute `Hits == 1` after a global `Clear()`; xUnit runs other classes in parallel in the same process, so any concurrently-compiling test breaks the absolute count. Recommended fix: row-74's established delta-over-scoped-action assertion pattern (a collection fixture would serialize more than needed). | Next SiteRuntime test touch, or first flake in a run that matters | +| 86 | ~~**CLI and UI disagree on deployment site-scoping for Administrators**~~ **RESOLVED 2026-08-17 (merge `ad0100f2`, agent commit `9f67147e`) — Administrator = system-wide is the intended semantic; the UI was the defect.** The rule was already documented on BOTH sides (`ManagementActor.cs` ~:522 "Users with the Administrator role, or system-wide Deployer, are not restricted"; `SiteScopeService`'s own XML doc "any Admin/Design user — is system-wide") and enforced at 6+ ManagementActor sites — only `SiteScopeService.ResolveAsync` ignored it. It now treats an Administrator as system-wide regardless of `SiteId` claims, read via explicit `FindAll(RoleClaimType)` + `OrdinalIgnoreCase` (NOT `IsInRole`, which compares ordinally and needs a `roleType` the harness identities don't set); an Administrator's site claims aren't collected at all, preserving *system-wide ⇒ empty permitted-id set*. All 15 UI consumer call sites funnel through `ResolveAsync` (surveyed). Fail-before verified: 6/6 new Administrator pins failed with only `SiteScopeService.cs` reverted; `DesignerWithSiteClaims_IsStillScoped` pins that the bypass was NOT widened to Designer (the XML doc's "Design user" claim was inaccurate — corrected). | Found by the 2026-08-17 Low-tail sweep; fixed with the row-37 consolidation | Closed 2026-08-17. | — | +| 87 | ~~**CLI deployment scoping is non-transactional and unbounded**~~ **RESOLVED 2026-08-17 (merge `ad0100f2`, agent commit `9f67147e`), with the row-37 consolidation.** Site grants now ride as `SiteIdScope` resolved inside the query's join — the whole-table `GetAllInstancesAsync` read is gone (pinned by `..._ReadsNoInstancesAtAll`, a strictly stronger replacement for the old "bulk load exactly once" pin, consciously migrated). An unparseable permitted-site id is *dropped* (narrows the scope) rather than discarding the grant (which would widen to fleet-wide). `PageSize <= 0` no longer means unlimited: it takes the CLI's own default 50, capped at `MaxPageSize` 1000. | Found by the 2026-08-17 Low-tail sweep; fixed with the row-37 consolidation | Closed 2026-08-17. | — | +| 88 | ~~**Unparseable CLI `--status` silently means "no filter"**~~ **RESOLVED 2026-08-17 (merge `ad0100f2`, agent commit `9f67147e`).** An unrecognized non-empty status now produces a `ManagementError` naming the valid values (row-46 principle: loud failure over silent empty/full results) — enforced actor-side so HTTP API callers get it too. Case-insensitive parse retained; numeric strings rejected via `Enum.IsDefined` (`Enum.TryParse` accepts `"7"`). CLI help text documents the valid set; deliberately NOT `AcceptOnlyFromAmong`, whose case-sensitive matching is a live minor defect on `audit query --status` (row 99). **Live-probed on the rig:** `deploy status --status Sucess` → `Unknown deployment status 'Sucess'. Valid values: Pending, InProgress, Success, Failed.` | Found by the 2026-08-17 Low-tail sweep | Closed 2026-08-17. | — | +| 89 | ~~**Connection-only artifact deploy does not reach the peer node's DCL**~~ **RESOLVED 2026-08-17 (merge `6cfb6a79`, agent commit `c57e74e6`) — premise corrected TWICE, then fixed.** (1) The standby's DCL is **EMPTY**, not "serving the old endpoint": `DeploymentManagerActor` (role-scoped singleton) is the sole sender of `CreateConnectionCommand` and only populates its own node's `dcl-manager`. The real defect was **create-time and not replication-specific**: `_createdConnections` is per singleton *incarnation*, and `data_connection_definitions` had NO production reader — so any restart/failover rebuilt every connection from the flattened-config snapshots and silently discarded newer connection-only artifact deploys (identical loss on an unreplicated single-node restart). (2) Arbitration decided and shipped: **the definitions store wins wherever it has an opinion; flattened config is authoritative only for names the store never owned** (deploy-only sites unchanged — memo §7.4's objection). A `_connectionDefinitions` overlay refreshes from `ReadStoreSnapshotAsync` at boot AND every sweep tick, applied before instance creation; ping-pong killed by construction (every ensure path resolves the *winner* before hashing, so one config per hash slot). Fail-before: 5/8 new `DclConnectionArbitrationTests` fail with the overlay inert. Decision memo: `docs/plans/2026-08-17-dcl-connection-arbitration-decision.md` (records the accepted residuals: re-adds land at next instance start, not instantly — mirrors the removal's own "bad quality until redeploy" semantic, pinned by test). The discovery that a *live* connection actor is never reconfigured in place on ANY node is a separate pre-existing defect → row 97. **Live rig probe at wave close-out (post-redeploy):** the singleton node logged exactly ONE boot-time `Created DCL connection OPC PLC Simulator` with zero update churn across later sweep ticks (no ping-pong from the overlay), the standby stayed silent (topology as corrected), all three rig-probe-1 tags Good, site-a replication connected / backlog 0. | Found by the 2026-08-17 row-85 fix session | Closed 2026-08-17. | — | +| 90 | ~~**No removal path for a connection deleted on the peer**~~ **RESOLVED 2026-08-17 (merge `6cfb6a79`, agent commit `c57e74e6`), with row 89.** Both removal paths now share `RemoveConnectionCentrally` + an in-memory tombstone that `EnsureDclConnections` honors — which also closed a pre-existing bug where an artifact removal was **undone by the very next instance start**. The sweep detects a previously-definition-sourced row's disappearance via its own overlay bookkeeping. Accepted residuals, documented in the decision memo: tombstones are not durable across a restart (by decision — matches the long-standing restart-resurrection semantic), and a delete is undetectable on a node that never adopted the row (the stronger "non-empty table ⇒ complete set" inference would stop live connections on a guess; deliberately not taken). | Found by the 2026-08-17 row-85 fix session | Closed 2026-08-17. | — | +| 91 | ~~**`SiteScriptCompileCacheTests.GetOrAdd_SameCodeAndGlobals_ComputesOnce` is parallel-load fragile**~~ **RESOLVED 2026-08-17 (merge `106e467d`, agent commit `26e3a1b9`) — row-74 pattern applied (exemplar `98c94588`), and wider than the named test.** Two habits: a per-test-instance guid key token (no other class can collide with or serve this test's entries, making `factoryCalls`/`Assert.Same` exact and owned) + one-sided bounds on the process-global counters (`>= baseline + n`; safe because all `Clear()` callers share the serialized collection). Four assertions changed, two got STRONGER (`GetOrAdd_SameCode_DifferentGlobals` had a second absolute pair; `Overflow_StaysWithinTheBound` gained an eviction probe over its own keys — a slack bound is not proof a sweep ran; its `ForeignEntrySlack = 512` is a judgement call, the eviction probe is the load-bearing assertion). Sibling absolutes in the same class and in `DeploymentWarmThenGateTests` fixed too. SiteRuntime.Tests run twice at full parallelism, 633/633 both. The underlying process-global `Clear()` cross-talk hazard remains → row 100. | Found by the 2026-08-17 row-85 fix session | Closed 2026-08-17. | — | | 92 | **The event-log cap's size model is sample-based and blind to mid-history width shifts** *(Low, accuracy limitation — documented in memo §3.1)* | Row-81 fix session (2026-08-17) | Row width is sampled 500 rows from each END of `site_events`; a stream whose row width changes sharply mid-history is mis-sized until the trim reaches that region. Self-correcting per tick (each trim exposes new tail rows to the sample), and the model's deliberate +8–13% high bias absorbs modest shifts, but a pathological width profile could over- or under-trim for several ticks. | First field report of the cap trimming visibly wide of its target, or any rework of the size model (see rows 93/94) | | 93 | **The cap model's page-layout constants are empirical for the CURRENT `site_events` schema** *(Low, maintenance trap)* | Row-81 fix session (2026-08-17) | Fill factors (0.78/0.75), cell overheads (16/8) and especially `IndexedColumnCount = 4` are calibrated to today's table + four indexes. A schema change — new column, new index — silently invalidates them; a fifth index in particular would be silently under-counted. Any `site_events` schema migration must re-run the memo §4 calibration and update the constants together. | Any schema change to `site_events` or its indexes | | 94 | **`SqliteCapabilityTests` is a deliberate tripwire — do not "fix" it by weakening the assertion** *(Low, standing note)* | Row-81 fix session (2026-08-17) | The test FAILS BY DESIGN if a future SQLite bump ships `dbstat`/`sqlite_dbpage` support. The correct response at that point is to REPLACE the sampled page-layout model with exact `dbstat` measurement (retiring rows 92/93 with it), not to relax the test. The test's own comment says the same. | `SqliteCapabilityTests` fails after a SQLite/provider bump | -| 95 | **The leaked 100 ms-tick test-actor pattern is systemic — three more classes carry it** *(Low, test-infra; row-82 residue)* | Row-82 fix session (2026-08-17): two classes fixed (`PartitionPurgeTests`, `NotifyDispatcherAuditTrailTests`), three more found carrying the same shape | `AuditLogPurgeActorTests` (benign today only because exactly ONE test there touches the fixture DB — adding a second DB-backed test reintroduces row 82), `OutageReconciliationTests`, and `SiteAuditReconciliationActorTests` all create 100 ms-tick actors against a shared fixture DB with no deterministic stop (only timeout-bounded `TestKit.Dispose`, which does not fail on expiry). None failed across ~15 parallel runs, so left untouched for proportionate blast radius. Fix pattern is already written: `Watch`/`Stop`/`ExpectTerminated` in the creating test. | Next touch to any of the three classes, or first flake matching the leaked-actor fingerprint (a prior test's rows mutated mid-test) | -| 96 | **xunit parallelism is uncapped almost everywhere — `MaxParallelThreads` unset, only 3 of ~20 test projects have an `xunit.runner.json`** *(Low, test-infra policy)* | Row-82 fix session (2026-08-17) | Every suite runs at processor-count collection parallelism, and a solution-level `dotnet test` multiplies that across concurrent processes — the general multiplier behind the row-82/91 flake class. The row-82 fix bounds the DB-migration axis specifically; a project-wide parallelism default (or per-suite runner configs for the resource-heavy suites) is a policy decision worth making once, deliberately, rather than per-flake. Supporting evidence from the 2026-08-17 close-out run: `AuditLogPageTests.DrillInFromParentExecution_FiltersGridToSpawnerExecution` (Playwright, untouched by the wave) hit a 30 s navigation timeout under full-suite parallel load and passed isolated in 1 s — one more member of this class. | Next test-infra session, or CI adoption of parallel suite runs | +| 95 | ~~**The leaked 100 ms-tick test-actor pattern is systemic — three more classes carry it**~~ **RESOLVED 2026-08-17 (merge `106e467d`, agent commit `34a8079d`).** All three classes live in AuditLog.Tests (premise correction: none in SiteCallAudit.Tests). `Watch`/`Stop`/`ExpectTerminated` replicated from `da23f2dc` at every creation site: 11 in `AuditLogPurgeActorTests`, 10 in `SiteAuditReconciliationActorTests`, 2 in `OutageReconciliationTests`. Second premise correction: `OutageReconciliationTests` DID call `Sys.Stop` — but fire-and-forget AND from the last line of the `await using` block, so an in-flight tick could still be writing to MSSQL during read-back and any earlier assertion failure skipped the stop entirely; stops moved before the read-backs, plus the `AuditLogIngestActor` that test 2 was discarding outright (holding a disposed provider) is now stopped too. Known lesser sibling habit left in place: EventStream probes are never unsubscribed (bounded by the per-test ActorSystem — noted, not a row). AuditLog.Tests 383/383. | Row-82 fix session (2026-08-17) | Closed 2026-08-17. | — | +| 96 | ~~**xunit parallelism is uncapped almost everywhere**~~ **RESOLVED 2026-08-17 (merge `106e467d`, agent commit `d2d4db49`) — policy made and shipped: per-suite caps for the resource-heavy suites, pure unit suites stay uncapped.** The four MSSQL-fixture users (`ConfigurationDatabase.Tests`, `AuditLog.Tests`, `SiteCallAudit.Tests`, `Host.Tests` — verified before capping) get `xunit.runner.json` with `maxParallelThreads: 4`, **deliberately matching row-82's `sp_getapplock` slot default** so the in-process and cross-process limits agree; no collections serialized. `IntegrationTests` (already fully serialized) and `CentralUI.PlaywrightTests` (already `maxParallelThreads: 1`, stricter than needed) left alone. Cap verified live (216% CPU uncapped vs 145% capped on ConfigurationDatabase.Tests); cost is nil — these suites are latency-bound on MSSQL/sockets (14–25% CPU), durations unchanged within noise. Policy documented in `docs/test_infra/test_infra.md` → *Test Parallelism Policy* (two-axis model, per-suite table, default for new projects), cross-linked from `test_infra_db.md`. Found-not-fixed: `StoreAndForward.Tests/xunit.runner.json` is a no-op (pins the default `parallelizeTestCollections: true`) — harmless, tidy on next touch. | Row-82 fix session (2026-08-17) | Closed 2026-08-17. | — | +| 97 | **A live DCL connection actor is never reconfigured in place — on ANY node** *(Medium, availability)* | Found by the 2026-08-17 rows-89/90 fix session; documented in `docs/plans/2026-08-17-dcl-connection-arbitration-decision.md` | `DataConnectionManagerActor.HandleCreateConnection` is create-only (`if (_connectionActors.ContainsKey(name)) { warn; return; }`), so `EnsureDclConnection`'s documented "re-issues the create so the DCL adopts the new configuration" has NEVER been true — including on the deploy-handling node — while the hash cache recorded the change as applied. The rows-89/90 fix made this *announced* (Warning, once per changed config) instead of silent, but a changed endpoint still only takes effect at instance redeploy or singleton restart. Fix sketch (from the memo): `ReconfigureConnectionCommand` handled *inside* `DataConnectionActor`, reusing the primary→backup failover adapter swap (dispose → `_factory.Create` → bump `_adapterGeneration` → re-enter the connect cycle, which re-issues from `_subscriptionsByInstance`). The naive stop-and-recreate is DESTRUCTIVE: subscriptions live in `DataConnectionActor` and `InstanceActor` cancels its subscribe-retry timer on first success, so nothing re-subscribes — it would trade "stale endpoint, data flowing" for "correct endpoint, no data until redeploy". | Next DataConnectionLayer session, or first field report that an artifact-deployed endpoint change didn't take effect on a live connection | +| 98 | **`EnforceSiteScopeForInstance` is fail-open for nonexistent instance ids** *(Low, authorization shape)* | Found by the 2026-08-17 rows-37/86-88 fix session (Agent A survey) | A site-scoped user querying a nonexistent instance id passes the scope gate rather than being denied (`ManagementActor.cs` ~:540); `EnforceSiteScopeForIdentifier` shares the shape. Harmless for the deployment read (no records reference a missing instance), but fail-open is the wrong default for an authorization gate — a future caller reusing it for something with side effects inherits the hole. | Next ManagementActor authorization touch, or any new caller of either enforce helper | +| 99 | **CLI `audit query --status` client-side validation is case-sensitive** *(Low, UX)* | Found by the 2026-08-17 rows-37/86-88 fix session | `AuditCommands` uses `AcceptOnlyFromAmong` with capitalised values, and that check matches case-*sensitively* — `audit query --status delivered` fails client-side while the server would accept it. The row-88 fix deliberately avoided introducing the same trap on `deploy status` (documented valid set in help text + case-insensitive server-side check instead). Fix: same treatment for the audit command group. | Next CLI audit-command touch | +| 100 | **`SiteScriptCompileCache.Clear()` is process-global — test classes wipe each other's entries** *(Low, test-infra hazard)* | Found by the 2026-08-17 row-91 fix session | Any test calling `Clear()` evicts entries other parallel classes just inserted. Harmless to correctness now that row 91 removed the absolute-count assertions (all `Clear()` callers share a serialized collection), but it remains latent cross-talk: a future test outside that collection calling `Clear()` mid-run degrades other classes to recompiles (perf noise) and re-opens the row-91 class if anyone reintroduces absolute counts. A `ClearKeys(prefix)` or an instance-scoped cache seam would remove the last shared mutable state. | Next `SiteScriptCompileCache` API touch, or a flake with the row-91 fingerprint from a class outside the serialized collection | | 68 | ~~**`deploy.sh` run from a git worktree silently rehomes every node's bind-mount data dir**~~ **RESOLVED 2026-08-15 (guard added in the same close-out).** The compose files' bind mounts are relative (`./site-a-node-a/data`), so a deploy from a worktree checkout mounts the worktree's copies — the rig keeps running and looks healthy, but its SQLite stores split from the main checkout's and everything written there is orphaned by the next deploy from main. Bit the rows-46–53 close-out live: the seeding agent's `rig-probe-1` deploy persisted into worktree-homed dirs and "vanished" after the orchestrator's redeploy (no tombstone, no product bug — LWW/tombstone forensics ruled out deletion; the row sat in `.claude/worktrees/…/docker/site-a-node-a/data/site-localdb.db` the whole time). Both `docker/deploy.sh` and `docker-env2/deploy.sh` now refuse to run from a `.claude/worktrees/` path (override: `SCADABRIDGE_ALLOW_WORKTREE_DEPLOY=1`). | Found during the rows-46–53 close-out | Closed 2026-08-15. | — | ## Resolved (verified against the code 2026-07-10)