# Deferred-Work Register (established 2026-07-08, from architecture review 08)

Single tracked list of consciously-deferred work. Rules: every deferral gets a row
(rationale + revisit trigger); fix-now items reference the archreview plan that owns
them and are removed from this table when that plan's task lands.

## Fix-now (owned by archreview plans)

All 7 fix-now items landed via PLAN-04/05/06/07/08 (verified in review 08 round 2, 2026-07-12) — rows removed per the rule above; see the round-2 report §1 for the per-item evidence.

## Deferred (with rationale + revisit trigger)
| # | Item | Where noted | Rationale for deferral | Revisit trigger |
|---|------|-------------|------------------------|-----------------|
| 8 | Hash-chain tamper evidence (T1); CLI verify-chain is a no-op stub | audit-log roadmap :12 | v1.x by locked decision; append-only DB roles are the control | Compliance requirement for cryptographic tamper evidence |
| 9 | Parquet audit archival (T2); endpoint returns 501 | AuditEndpoints.cs:204 | v1.x; 501 + CLI messaging are honest | AuditLog partition volume nears retention ceiling |
| 11 | Central-persisted OPC UA cert-trust audit | m7 follow-ups | Broadcast-to-both-nodes covers HA | Governance/audit requirement for trust decisions |
| 17 | Unified notifications+site-calls outbox page | stillpending :118 | Explicit M9 decision to keep two pages | Operator confusion reports |
| 19 | Bundle signing / cluster-to-cluster pull / differential bundles | transport-design :402 | v1 manifest hash + AES-GCM held sufficient | Non-repudiation requirement across orgs |
| 23 | Live LDAP group-membership re-query for an active session | `docs/requirements/Component-Security.md` :61-69 (+ :78-79) | Blocked on an external package. The mid-session refresh re-maps the **stored** groups against the central DB with **no LDAP call**, so a directory group-membership change lands only at next login. A live re-query needs a passwordless service-account group-search method on the shared `ZB.MOM.WW.Auth.Ldap` library — an external NuGet `PackageReference` (`src/ZB.MOM.WW.ScadaBridge.Security/…csproj:23`) exposing only `AuthenticateAsync(username, password, ct)`. Central role-mapping/scope changes still apply within ~15 min (`RoleRefreshThresholdMinutes`). | `ZB.MOM.WW.Auth.Ldap` gains a standalone group-search API, or a requirement that a directory-side group revocation take effect mid-session rather than at next login |
| 24 | M8 large-bundle performance hardening | `docs/plans/2026-06-15-stillpending-completion-design.md:106` — "Small follow-ups logged (not blocking): … large-bundle/perf hardening" | Logged as a non-blocking follow-up when M8 shipped and never given an artifact: **no plan, no task entry, no perf/load test exists** (`tests/…Transport.Tests/Import/BundleImporterLoadTests.cs` is a `LoadAsync` unit suite despite the name). No measured problem; the only sizing controls in place are the 5-minute CLI transport timeout, `LineDiffer`'s `MaxInputLines`=4000 summary-only cap, and `MaxConcurrentImportSessions`=8. | First real bundle that times out, exhausts memory, or makes the import wizard's diff step unusable |
| 25 | ~~**Phase-8 WP-4 target-scale load test** (10 sites x 500 instances x 75 tags = 37,500 subscriptions/site, 375,000 total)~~ **RESOLVED 2026-08-15 — run, measured, evidenced.** The 107-byte stub is replaced by a real full-scale run: **10 x 500 x 75 = 375,000 live tag subscriptions**, 37,518 tag updates/s achieved against 37,500 nominal (100.0%), **45,021,375 updates offered** over a 20-minute steady-state window on an M4 Pro / 14-core / 48 GB host with the 8-node `docker/` rig still running. Headline numbers: tag update latency **P50 0.88 ms / P99 4.57 ms / max 37.41 ms** end-to-end (DCL boundary to stream subscriber, the emit instant carried verbatim through production code); **0 events dropped** at 100 live subscribers; health report collect+ingest **P99 0.31 ms** with 10/10 sites tracked; debug view snapshot under load **P99 2.19 ms, 0 timeouts**; 500 instances deployed to a site in **2.6 s**; CPU **2.9% of the box**; working-set slope **+8.83 MB/min**. **11 clean passes, 1 pass with a caveat (F1: 20 min with zero gen-2 collections cannot fully settle the leak question — the 1-hour run would), 0 failures.** Harness: `tests/ZB.MOM.WW.ScadaBridge.LoadHarness/` (standalone Exe — the `Category=Performance` trait enables a filter but does NOT exclude by default, so a 20-minute test could not live in `PerformanceTests`) plus a CI-scale smoke `[Fact]` (`PerformanceTests/TargetScale/TargetScaleHarnessSmokeTests.cs`). Real: per-site ActorSystem + LocalDb, the real DCL via the documented `DataConnectionFactory.RegisterAdapter` seam, real `InstanceActor`s, real `SiteStreamManager`, real `StreamRelayActor` + production-capacity bounded channel, real S&F, real health collector/aggregator; only the socket hops are stood in for. **Results: `docs/plans/2026-08-15-target-scale-load-test-results.md`; design + deviations + thresholds: `docs/plans/2026-08-15-target-scale-load-test-design.md`.** | as before | Closed. Residual scope: `[xc-3]` site-event-log retention volume, `[xc-4]` audit-vs-central degradation, `[xc-5]` large-template flattening and `[xc-8]` UI responsiveness are NOT covered by this harness (central-cluster/browser concerns) and remain open WP-4 scope — see results doc SS6. | Closed 2026-08-15. |
| 26 | Ipsen MES MoveIn tail: leak-test (`-LT`) receivers + routing, PLC-output-flag writes, `Z28062` BTDB data completeness | `docs/plans/2026-06-16-ipsen-mes-movein.md:409` ("Out of scope (future)"); design `2026-06-16-ipsen-mes-movein-design.md:58-60, :196-198` | Customer-site scope, not a platform gap. `-LT` routing needs an MES-receiver child + Galaxy reference that do not exist on the reactor template (any `-LT`/unknown suffix returns `WasSuccessful=false` with an "unsupported side/target" message by decision); `MoveInComplete`/`Successful`/`ErrorText` are **PLC-owned** by locked decision, so ScadaBridge deliberately does not write them; `Z28062` completeness is an operational data fix, not code. Note the separate alarm-status path already handles the suffix — `_LT` is stripped before side-scoping (`2026-06-30-mes-alarm-status-api.md:158`). | Ipsen creates the leak-test receiver + Galaxy reference, or asks ScadaBridge to own the PLC-output flags — otherwise a **candidate won't-do** (`[PERM]`) at the next Ipsen scope review |
| 27 | ~~External-system per-system retry config (`MaxRetries`/`RetryDelay`) never reaches sites, and has no CLI/management surface~~ **RESOLVED 2026-08-15 (`41a5ecf1`).** The gap was transport + persistence only — the consumption end already worked (`ExternalSystemClient.CachedCallAsync` already passed `system.MaxRetries`/`RetryDelay` to `EnqueueAsync` with 0-as-unset, and the Central UI form already collected both fields, inert until now with defaults 3×5s that will take effect on next deploy). Shipped: artifact fields, additive `site_command.proto` fields 7/8 (regen via `docker/regen-proto.sh`, csproj left clean), site `external_systems` columns `max_retries`/`retry_delay_ms` as REAL COLUMNS — not a JSON bag; verified safe against the AckTime precedent in the LocalDb source (capture triggers are DROP-then-CREATE from `PRAGMA table_info` each boot, and the fail-closed handshake digest turns a mixed-version pair into a loud sync refusal, no new constraint since both-nodes-together is already the standing pair rule; `timeout_seconds` set the additive-column precedent on this exact table) — apply path, and CLI `--max-retries`/`--retry-delay-seconds` (+`--retry-delay` alias). Bonus fix: `ArtifactDiff.CompareExternalSystem` never compared the retry fields, so a bundle import changing a retry budget previewed as "Identical". Fail-before verified on the site round-trip + apply tests; 12 suites green. | Found live 2026-08-01 (rig session, #11 gRPC live checks) | Closed 2026-08-15. | — |
| 28 | ~~Health-dashboard "Trigger failover" confirm dialog's confirm button is labeled **"Delete"**~~ **RESOLVED 2026-08-15 (`0db253b5`).** Structural, not a missing argument: `IDialogService.ConfirmAsync` had NO label parameter — `DialogHost` derived the verb from `danger` alone (`danger ? "Delete" : "Confirm"`), so no call site could ever have passed one. New optional `confirmLabel`; `danger` now selects styling only. Full audit of all 34 `ConfirmAsync` call sites (the only confirmation path — no `window.confirm`, no per-page modals): NINE were wrongly labeled "Delete" — both failover buttons, secured-write **Approve** and **Reject**, instance Disable, and four Discard actions — all now name their action; 18 genuine deletes + 7 non-danger sites verified correct. The `danger→"Delete"` fallback deliberately kept: forcing an explicit label would churn ~20 correct sites, and a missed site would silently degrade to "Confirm", detectable only by the rig-dependent Playwright suite — contract pinned in the interface XML doc, Component-CentralUI.md, and a fallback test instead. Gotcha recorded in the tests: adding an optional parameter silently un-matches existing 3-arg NSubstitute stubs (returns null Task → NRE). Follow-ups: rows 75 (Approve's red styling), and the bUnit real-dialog deadlock hazard documented in `FailoverConfirmLabelTests`. | Found live 2026-08-01 (rig session, #11 TriggerSiteFailover check) | Closed 2026-08-15. | — |
| 29 | ~~**1-hour sustained run to settle the memory-slope ambiguity (load-test finding F1)**~~ **RESOLVED 2026-08-15 — run, and F1 is settled: NOT a leak.** Full 1-hour protocol executed (same harness/scale, `--sustain-minutes 60`, 135,041,250 updates at 37,511/s = 100.0 % of nominal, concurrent with the live docker rig AND a central failover drill on the same host). Decisive: **working-set slope −28.89 MB/min** (2,722 → 1,757 MB over the 3,600 s window, peak 3,258 MB) with **one gen-2 collection observed** and threads flat at 59 — a working set that FALLS ~1 GB across a sustained hour is incompatible with a slow leak; the positive managed-heap least-squares slope was the sawtooth-segment artifact F1 hypothesised. All other metrics held or improved vs the 20-min run: P99 4.01 ms, 0 healthy-subscriber drops, 744/744 debug snapshots, S&F drain to depth 0. Full numbers appended to the results doc F1 section. | `docs/plans/2026-08-15-target-scale-load-test-results.md` F1 postscript | Closed 2026-08-15. | — |
| 30 | **WP-4 residual scope: `[xc-3]` site-event-log retention volume, `[xc-4]` audit-load-vs-central degradation, `[xc-5]` large-template flattening time, `[xc-8]` UI responsiveness under load** | `docs/plans/2026-08-15-target-scale-load-test-results.md` §6 + row 25 closure note | The row-25 harness is a site-runtime rig; these four WP-4 sub-criteria are central-cluster/browser concerns it deliberately does not cover. Recorded so the row-25 closure never reads as "WP-4 fully done" — phase-8 WP-4 is closed for the site-runtime data path only. | Before declaring phase-8 WP-4 complete, or production go-live at target scale |
| 31 | **Unpaced-burst absorption is bounded by the shared publish stage, not per-subscriber buffers (load-test finding F3 caveat)** | `docs/plans/2026-08-15-target-scale-load-test-results.md` F3 | Measurement-backed design note, no defect: slow-subscriber isolation is structural and total (a stalled reader loses events only in its own bounded `DropOldest` channel; healthy subscribers delivered 100.00%), but the publish `Source.ActorRef` **upstream** of the `BroadcastHub` is shared, so an unpaced burst degrades every subscriber equally — that stage, not per-subscriber buffering, is the burst ceiling. | First field report of event loss on *healthy* subscribers during a burst, or any redesign of the site stream publish stage |
| 32 | ~~**`QueueDepthGaugeTests` full-suite flake — shared static gauge (load-test finding F4)**~~ **RESOLVED 2026-08-15 (`65b19516`).** Reproduced first (2 failures in 5 parallel runs — the gauge read a sibling class's provider). The register's suggested fix (per-test meter) was rejected on inspection: the slot is process-global by product design and 3 of the 6 facts assert the slot-ownership semantics themselves (compare-and-clear identity, stop/restart re-seed), so per-test instances would change what they prove. Correct isolation is mutual exclusion — the three `StartAsync`-calling classes now share xunit collection `QueueDepthGaugeCollection` (with a `[CallerFilePath]`-anchored source-scan drift guard that names an offending file), the other eight classes stay parallel, and `parallelizeTestCollections` was flipped back ON so the fix is exercised rather than masked. Negative control: dropping the attribute fails the guard AND resurfaces the original flake in the same run. 8/8 green parallel runs. | `tests/…StoreAndForward.Tests/QueueDepthGaugeTests.cs`; surfaced by the row-25 verification runs | Closed 2026-08-15. | — |
| 33 | ~~**`LocalDbOplogBacklog` has no threshold, KPI history, or dashboard treatment**~~ **RESOLVED 2026-08-15 (`8da60f57`).** Both halves shipped: sampled into KPI history via `SiteHealthKpiSampleSource` (conditionally — 12 samples unreplicated / 14 replicated, so a series gap truthfully means "not replicated") plus a backlog trend chart on the site health panel, and a SUSTAINED threshold badge (`LocalDbBacklogThreshold` 1000 held for `LocalDbBacklogSustainedFor` 15 min, anchored by `LocalDbBacklogAboveSince` in the aggregator) — a bare instantaneous threshold would fire on every replicated site every day from the documented purge burst. Null backlog is a deliberate third branch: it leaves the anchor unchanged (neither evidence of a drain nor of a backlog), so unreplicated sites can never anchor and can never be badged. NOTE the row's premise was false — the `SiteAuditBacklog` "configurable threshold badge" it told us to mirror was described twice in `Component-HealthMonitoring.md` but NEVER BUILT; both stale doc claims corrected, and the new badge was modeled on the real Metrics-stale badge instead. | Found by R7 while documenting the purge burst | Closed 2026-08-15. | — |
| 34 | ~~**`site_events` purge burst is unbounded in the oplog and unmeasured at scale**~~ **RESOLVED 2026-08-16 (`d9a35846`) — measured, and paced.** Measured with the REAL `EventLogPurgeService` over a real `RegisterReplicated` LocalDb file + a loopback sync pair at the rig's `MaxBatchSize: 64`: 393 B/row incl. indexes ⇒ a 1 GB event log holds **~2.73 M rows**, and expiring them in one tick took 285 s and queued **2 731 947 oplog rows = 2.7x LocalDb's `MaxOplogRows` default (1 M)** — enough to make the maintenance tick prune the oplog and flag a FULL ten-table snapshot resync, bought by a routine retention purge. The decisive nuance: the *resident* backlog is produced−minus−drained, so a healthy peer blunts it (391 k peak on loopback, drained 6.0 min) — the old safety margin was a function of peer throughput and vanished with the peer down. Fix: `MaxPurgedRowsPerTick` (default **250 000**, `0` = unlimited escape hatch), one budget SHARED across retention + cap trim; a ceiling-limited tick reschedules after `PurgeCatchUpInterval` (**5 min**, validated `> 0` and `<= PurgeInterval`) so worst-case retention converges in ~55 min instead of 11 days; purge loop is now a delay loop anchored to the END of the previous purge; rate-limited Warning + counter `scadabridge.site.event_log.purge_ceiling_hits`. No purge-exemption path added. Row-33 interplay verified empirically (500 k drained in 46 s vs the 15-min dwell). Fail-before: `PurgeBudget` neutralised → 6 of the new tests fail, all 19 pre-existing pass. Measurement + decision record: `docs/plans/2026-08-15-site-events-purge-burst-measurement.md`. Side-finding recorded as row 81 (storage cap unenforceable on a replicated node). | Found by R7 | Closed 2026-08-16. | — |
| 35 | ~~**`DeploymentRecords` index does not cover the deployment-list sort key**~~ **RESOLVED 2026-08-15 (`be29bf47`).** Migration `20260815165452_AddDeploymentRecordListSortIndex` + hand-written `docs/plans/sql/AddDeploymentRecordListSortIndex.sql`: composite `IX_DeploymentRecords_DeployedAt_Id (DeployedAt DESC, Id DESC)`; the old single-column index DROPPED as strictly redundant (`Id` is the clustered PK and was already in the old index's leaf as the row locator — same leaf width, now a declared ordered key; keeping it would cost a second index on every insert/status transition for no read benefit). Serves the three reads sharing the key (`QueryDeploymentListPageAsync`, `QueryDeploymentSummariesAsync`, `GetCurrentDeploymentStatusAsync`). `Id` in the key is also a CORRECTNESS matter: a bulk site deploy stamps many rows in one `DeployedAt` instant, so without the tie-break keyset pages can repeat or drop rows. Recorded caveat: the optimizer may already have served the ordering via a backward scan of the old index, so the "sorts per page" premise was possibly optimizer-dependent — the composite makes it deterministic and self-documenting; an actual plan capture against the rig would quantify the before/after. The ConfigurationDatabase suite exercises the descending composite as real DDL via SQLite `EnsureCreated()`, proving provider portability. | Found by R3 (`35ce1413`) | Closed 2026-08-15. | — |
| 36 | **Deployments free-text search is unindexable by construction** | Found by R3; re-verified 2026-08-17 (Low-tail sweep) | Four `LIKE '%term%'` predicates (instance name, deployment id, revision hash, user) — leading wildcards preclude index seeks, so a search is a scan of the join (`DeploymentManagerRepository.cs:100-104`; only searched column with any index is `DeploymentId`, unseekable under a leading wildcard). Acceptable at the purge-bounded row count. **2026-08-17 refresh: the row undercounted the cost by half** — `BuildListQuery` is shared with `GetDeploymentStatusCountsAsync`, so one search = TWO scans of the join (list + status tiles), mitigated by the UI's trailing-edge debounce (one pair per pause, not per keystroke). Anchoring the id/hash legs was evaluated and REJECTED as a micro-fix: `DeploymentListFilter.Search`'s doc comment promises substring matching and a 4-case `[Theory]` deliberately pins mid-string matches — changing it is a contract renegotiation, not an optimization. Fix if hot: renegotiate to prefix on id/hash legs, or full-text. *(Low)* | Search latency complaints on the Deployments page |
| 37 | ~~**Two paged reads of `DeploymentRecords` with different contracts**~~ **RESOLVED 2026-08-17 (merge `ad0100f2`, agent commit `9f67147e`).** `QueryDeploymentSummariesAsync` and `DeploymentRecordSummary` are DELETED; the CLI/HTTP `QueryDeployments` handler now rides `QueryDeploymentListPageAsync` + `DeploymentListFilter` — one row shape (`DeploymentListRow`), one scoping mechanism (site-id via the query join), total count available to both surfaces (`DeploymentListPage` now also owns `MaxPageSize` + `DefaultPageSize = 50`). Rows 86–88 were resolved with the consolidation, as this row anticipated. Row 36's 4-case substring-search Theory untouched and still passing. Remaining UI-vs-CLI asymmetry (`GetDeploymentStatusCountsAsync` status tiles are UI-only) is deliberate, not drift. **Live-probed on the rig:** CLI `deploy status --status Success --page-size 5` returns the consolidated page — rows now carry `siteId` + `instanceUniqueName`, and the response carries `totalCount` (11) the CLI never had. | Found by R3; consolidation shipped with rows 86–88 | Closed 2026-08-17. | — |
| 38 | ~~**OtOpcUa: `RoslynVirtualTagEvaluatorTests` racing-clear test is load-fragile**~~ **RESOLVED 2026-08-18 in OtOpcUa (`34829ea0` + `837d8a17`, branch perf/followups-r3).** Rebuilt on the assert-on-retry-exhaustion option, renamed `…never_leaks_a_raw_disposed_failure`. Two findings beyond the brief: (1) the DOMINANT load-failure mode was the test's own 45 s `WaitAsync` budget — every clear forces a recompile, so the fixed 12×400 workload's wall-clock tracked spare CPU; reproduced RED 4/4 at 4-way self-concurrency, fixed by duration-bounding the loops (10 s window OR 400 iters), GREEN 4/4 and 8/8 after. (2) A second flake vector: `TimedScriptEvaluator`'s 5 s clock starts at enqueue, so pool starvation surfaces "script timed out" from healthy evaluations — now its own bounded bucket, not counted as a leak; failure-reason literals promoted to shared consts. Honest caveat recorded: the disposed race itself does not reproduce on the arm64 macOS box (zero ODEs even with the product retry disabled), so the raw-escape assertion is the load-independent guarantee. Commit stats cross-checked from this repo. | Found by R4; fixed cross-session by the OtOpcUa agent 2026-08-18 | Closed 2026-08-18. | — |
| 39 | ~~**OtOpcUa: late-enabled replication cannot baseline pre-existing rows**~~ **RESOLVED 2026-08-18 in OtOpcUa (`3e62f161` + review round `df533916`, branch perf/followups-r3).** ScadaBridge's WP1.3/WP3.3 pattern ported: registration follows replication config — configured ⇒ `RegisterReplicated(..., baselineExistingRows: true)` (probed live on 0.2.1: seeds `__localdb_row_version` at HLC 0), unconfigured ⇒ boot-time `DeregisterReplicated` self-heal, pair-symmetric; review round added deregister-preserves-data + partial-flip-fails-loudly tests. Green: `LocalDbSetupTests` 15/15, Host.IntegrationTests 226 pass/0 fail. **Stayed pinned to LocalDb 0.2.1 by deliberate choice** — the 0.2.1 API already carries both needed methods, and the v2→v3 one-way bookkeeping upgrade makes a 0.3.x bump a both-nodes-together operation; recorded caveat in OtOpcUa's CLAUDE.md: on 0.2.1 the snapshot-resync flag is per-DATABASE, so a late enable snapshots the whole file rather than per-table (0.3.1 available; bump logged as a deliberate future change in OtOpcUa's own follow-up register). Commit stats cross-checked from this repo. | Found by R4; fixed cross-session by the OtOpcUa agent 2026-08-18 | Closed 2026-08-18. | — |
| 40 | ~~**Wall-clock sleeps used as synchronization in tests (absence-assertion class)**~~ **RESOLVED 2026-08-15 (`7a7d345b`).** All six listed sites replaced with observable edges + positive controls, strengthening proven by stash-based negative controls: `ScriptDeadlineAtEnqueueTests:128` now pins the watchdog via a genuinely-wedged positive control (armed strictly later, self-releasing) + per-script absence assertion + detach counter pinned at 1 — **disabling the entire stuck-script watchdog passed the OLD test 2/2 and fails the new one**; `:105` now proves the run was still queued when its budget expired via `QueueDepth` edges (the actual design claim — the old sleep asserted nothing about it); `DeploymentWarmThenGateTests:117` (+2 sibling `Task.Delay(500)`s) — the empty-site zero is now the manager's own observation (phantom-startup control: old passed 2/2, new fails); `SiteAlarmLiveCacheServiceTests:195,216` poll a new subscription-cancellation observable across 3 linger windows, each closing with a positive control (control: linger cancellation disabled + viewer guard dropped — `Resubscribe_Within_Linger` old passed 2/2, new fails; honest caveat: the other case's old later assertion did already catch that break, so its gain is the observable + control, not a false-pass fix); `DataConnectionActorTests:435` waits for the Connected health publication so the subscribe provably runs against Connected (the sleep guarded no assertion — determinism gain only). Bonus: the de-padded schedule exposed a LATENT race in `InstanceActorChildAttributeRaceTests` (~1-in-6 under load, A/B'd 14/14 green on the unmodified tree so pre-existing) — fixed with a bounded-deadline resolution retry; pattern recorded as row 77. | Found by the R5 sweep (`9fb52153`) | Closed 2026-08-15. | — |
| 41 | ~~**LocalDb (scadaproj) sync-session tests flake under parallel load**~~ **RESOLVED 2026-08-18 in scadaproj (`3e5a5fd`, LocalDb 0.3.1) — hypothesis CONFIRMED as a product bug, fixed on the product path.** The enqueue→receive→validate→fault path in `HandshakeAsync` has no yields while the writer's wakeup sits in the thread-pool queue, so the catch's `linked.Cancel()` beat the writer 39/40 times in a new pinning test (`Handshake_FaultTeardown_StillTransmitsTheLocalHandshake`) — mismatch detection is symmetric (each side computes its verdict FROM the other's Handshake), so the healthy peer learned nothing: 15 s stall in tests; in production a permanent schema mismatch would be misclassified as a generic disconnect and reconnect-loop at fast backoff instead of the FailedPrecondition/max-backoff path. Fix: `RunAsync`'s fault teardown completes both lanes so the writer drains what's enqueued and exits on its own, with a 2 s deferred cancel as the wedged-transport backstop; caller-initiated cancellation unchanged; **no wire change** (a 0.3.0 peer just finally receives the Handshake it was owed). A/B under doubled load: pre-fix 2/6 full runs failed (exact predicted stack), post-fix 6/6 green (200/200) + 5/5 filtered. The OTHER named test (`BatchBudgetTests.Pump_RowLargerThanTheBudget…`) was a harness defect, not this root cause (scripted peer, no handshake fault): it awaited the peer channel without observing the session task, so any session fault surfaced as an opaque 15 s timeout — hardened with a run-observing `NextAsync` that races the session task at every read site. *Residual (scadaproj-side, pre-existing class):* two non-recurring one-offs in 10 post-fix runs (`Handshake_Exchanged_BothProceed` @53 ms, `Metrics_NullSafe…` @208 ms; detail lost, never recurred in 12 further loaded runs either side) — both share the `SwallowAsync`-catches-only-OCE teardown shape, best guess a cancelled SQLite op surfacing `SqliteException` at teardown; not attributable to this change, not proven either. **0.3.1 published to the feed 2026-08-18** (all three packages, restore-verified from a scratch consumer with a positive control confirming the restored Replication assembly carries the fix; scadaproj main pushed — `3e5a5fd`, `b62981f`, index `dbcbc5b`). ScadaBridge's pin bumped 0.3.0 → 0.3.1 the same day: build clean, all six LocalDb-consuming test suites green (SiteRuntime 652, IntegrationTests 115, HealthMonitoring/ExternalSystemGateway/AuditLog/StoreAndForward all pass); a 0.3.0/0.3.1 mixed pair is wire-safe (teardown-ordering change only). | Found by R1; confirmed pre-existing at base `f627cee`; fixed cross-session by the scadaproj agent 2026-08-18 | Closed 2026-08-18. | — |
| 42 | ~~**`build/push.sh`-class publish bug across the family repos**~~ **RESOLVED 2026-08-18 in scadaproj (`b62981f`).** The bug shape: `dotnet pack -c Release -o ./artifacts` with no project argument and no `cd`, so the script packs the CALLER'S CWD. Fixed in LocalDb's script (cd to own repo root, name the solution, absolute output path). **AUDITED 2026-08-15 (read-only, this wave): SEVEN scadaproj scripts still carry the exact shape** — `ZB.MOM.WW.Secrets/build/push.sh`, `ZB.MOM.WW.Theme/build/{pack,push}.sh`, `ZB.MOM.WW.Audit/build/{pack,push}.sh`, `ZB.MOM.WW.Auth/build/{pack,push}.sh` (all `dotnet pack -c Release -o ./artifacts` with no project argument and no `cd`, plus a CWD-relative `nuget push "./artifacts/*.nupkg"` that would publish whatever sits in the caller's artifacts dir). MxAccessGateway's `scripts/pack-clients.ps1` is SAFE (`$PSScriptRoot`-anchored, explicit csproj paths); HistorianGateway and OtOpcUa have no pack scripts. Fix = copy LocalDb's `0038d46` pattern into the seven scripts; lives in the scadaproj repo, outside this wave's edit scope. Fixed: all seven scripts now carry the `0038d46` pattern (cd to own repo root, named `.slnx`, absolute artifacts output, anchored push glob); verified by `bash -n` on all seven plus a positive control on the pack half only (Audit's `pack.sh` run from a foreign CWD packed only `ZB.MOM.WW.Audit` into its own artifacts dir, caller's directory untouched); no push script executed, no credentials touched. Commit stat cross-checked from this repo: exactly the seven listed files. | Found by R1 the hard way (incident: 25 unrelated packages briefly published to the feed, deleted + verified same session); fixed cross-session by the scadaproj agent 2026-08-18 | Closed 2026-08-18. | — |
| 43 | ~~**Central UI import wizard leaks a `BundleSession` slot on every abandoned import — imports lock out for 30 min**~~ **RESOLVED 2026-08-15 (`70838982`).** New `IBundleImporter.Abandon(sessionId)` does the same zero-then-remove release `ApplyAsync` already did (T-007 discipline), idempotent (double/unknown/expired ids are no-ops) and only ever given the caller's own random-GUID id, so it cannot evict a concurrent circuit's session. Wired into EVERY abandonment path: wizard new-upload, `TryLoadAsync` pre-overwrite, Back-to-Upload/`ResetSessionState`, 3-strike lockout reset, `Dispose` — plus two CLI leaks the original row didn't know about: `transport preview` released nothing at all (every preview burned a slot for 30 min) and `transport import` stranded the session on pre-apply aborts; both now release in a `finally`. `BundleSessionEvictionService` is documented as the TTL backstop, not an abandonment path. Fail-before-fix stash-verified (3 of 4 new bUnit tests fail pre-fix). 10 new tests. | Found live by R6 | Closed 2026-08-15. | — |
| 44 | ~~**`BundleUnlockRateLimiter` is dead code — the documented per-IP unlock throttle is not enforced**~~ **RESOLVED 2026-08-15 (`5ab26a42`).** `BundleImporter.LoadAsync` now registers every unlock attempt against `MaxUnlockAttemptsPerIpPerHour` (10). Ordering is load-bearing: AFTER the per-bundle 3-strike check (hammering a locked bundle spends no hourly budget) and BEFORE the decrypt (a throttle denial burns no per-bundle strike, so the throttle can't be abused to lock out a legitimate import); successful unlocks count too. Denial = `BundleUnlockThrottledException` (placed in Commons so the management surface returns a curated message). Client key: Central UI via a scoped `CircuitClientAddressProvider` snapshotting the IP at circuit establishment; CLI via additive `ManagementEnvelope.ClientIp`. No attributable IP falls into one shared `<unattributed>` bucket — stricter, never a bypass; per-client isolation is only real behind a ForwardedHeaders-trusted proxy (same caveat as the login throttle, documented). Fail-before-fix verified (4 of 6 new tests fail with the limiter neutralised; 2 negative controls). | Found by R6 | Closed 2026-08-15. | — |
| 45 | ~~**Notification Outbox dispatcher shares one `DbContext` across up to 4 parallel deliveries**~~ **RESOLVED 2026-08-15 (`9e769735`).** `_adaptersCache`/`_adaptersScope`/`ResolveAdapters()` deleted; the typed adapter is resolved inside the per-delivery DI scope `DeliverGatedAsync` already opens (static `ResolveAdapter` preserving DI last-wins + the null-adapter park path), so each concurrent delivery gets its own adapter and `DbContext`. Nothing shared was lost: the Email adapter builds a fresh SMTP client per attempt from an injected factory, the SMS adapter draws from the singleton `IHttpClientFactory`, and CentralUI's adapter-gated Type selector (`NotificationChannelCatalog`) is an independent scoped consumer, untouched. Fail-before-fix stash-verified (simulated DbContext throwing EF's exact second-operation exception under a rendezvous-guaranteed overlap: 1 of 2 rows persisted pre-fix; 1-vs-4 distinct adapter instances). Root cause of note: the bug was INTRODUCED by code-review finding NotificationOutbox-006's caching "fix" (2026-05-28), whose own regression test pinned the broken behavior and predates bounded parallel delivery — that finding is marked SUPERSEDED. NotificationOutbox 153, AuditLog 383 green. | Found by R6; observed live on rig central-b | Closed 2026-08-15. | — |
| 46 | ~~**CLI `audit-config query` returns `{}` for every filter combination**~~ **RESOLVED 2026-08-15 (`09108018`).** Not a CLI bug: `ManagementActor.HandleQueryAuditLog` returned the repository's ValueTuple `(Entries, TotalCount)` straight to `SerializeResult`; System.Text.Json writes properties only and a ValueTuple's `Item1`/`Item2` are **fields**, so the whole payload serialized to the literal `{}` — every filter combination looked empty with no error anywhere in the stack. Fixed with a named Commons record `AuditLogQueryResult` (mirroring `SecuredWriteListResult`) plus a class-wide guard: `SerializeResult` now throws on any `ITuple` result, so the next tuple-returning handler fails loudly instead of silently emitting `{}`. Fail-before verified (test asserted exactly `"{}"` pre-fix); live re-verified on the rig (`totalCount 2725` unfiltered, `712` for `--entity-type Template --action Delete`). The only other tuple-returning repo method in Commons (`INotificationOutboxRepository.QueryAsync`) is never serialized. | Found by R6 against the rig; underlying `AuditLogEntries` rows verified present | Closed 2026-08-15. | — |
| 47 | ~~**`SiteStreamGrpcServer` duplicate-stream replacement disposed another handler's `CancellationTokenSource`**~~ **RESOLVED 2026-08-15 (found and fixed in-flight by R2, `e6842c10`).** `SubscribeInstance`'s duplicate-prevention path called `Cancel()` AND `Dispose()` on the replaced stream's CTS, which belongs to that still-running handler's `using var` and is still read — escaping as an unhandled `ObjectDisposedException` under scheduling pressure (confirmed pre-existing on `2b74851f`; load-only, passed 10/10 isolated). Fixed to cancel-only (the owner's `using` disposes exactly once), `Cancel` wrapped for the converse race, mirroring `CancelAllStreams()`. Deterministic regression test via a gate inside the setup window, fail-before/pass-after verified by reinstating the `Dispose()` as a negative control. Recorded here (though closed) because it is a product-bug instance of the disposal-race class, useful precedent for future stream-lifecycle work. | Closed 2026-08-15. |
| 48 | ~~**Docker rig has no running site instances — live stream probes have no traffic**~~ **RESOLVED 2026-08-15 (`b5e6821c`).** Working set re-seeded on site-a entirely via the CLI (`soak-*` rows left alone): template `RigProbe` + instance `rig-probe-1`, bound to the OPC PLC sim's built-in moving tags (`SpikeData`, `RandomSignedInt32`) plus a 5s heartbeat script. Deploy Success; `debug snapshot` returns live Good-quality data; `health site` shows `OPC PLC Simulator: Connected`, `good:2 bad:0 uncertain:0`; a 60s `debug stream` produced 239 events; and **`scadabridge.site.stream.batch_size` recorded live for the first time** (count 70 / sum 238, batches to ~10 events) — the R2 batching histogram is now observable on the rig, which was this row's purpose. Reproducible via `docker/seed-live-probe.sh`. Two diagnostic corrections worth keeping: the custom tags in `infra/opcua/nodes.json` are declared but **not simulated** by opc-plc (a healthy-looking connection with zero traffic — bind to the built-ins: `SpikeData`, `RandomSignedInt32`, `StepUp`, `FastUInt1..5`, `SlowUInt1..5`); and `scadabridge_site_connection_up` counts open central→site gRPC subscriber streams, not data connections, so this row's original "0 on all six site nodes" was healthy steady state, not the fault (row 54). Close-out postscript: the seeding agent's live verification ran against worktree-homed data dirs (row 68), so after the orchestrator's redeploy from main the config was absent at boot and re-fetched by the startup reconcile — which exposed row 67 — and `rig-probe-1` was re-verified live on the main checkout's data dirs (snapshot Good ×3, heartbeat ticking, sim values moving). | Found during the R2 close-out live probe (2026-08-15) | Closed 2026-08-15. | — |
| 49 | ~~**Sweep code-review "Resolved" entries that cached or lifetime-widened a DI-scoped service**~~ **RESOLVED 2026-08-15 — swept, no further bug found (`75520bee`).** All 576 resolved findings across the 25 component findings files were re-read for the row-45 generalization (a resolution that caches, hoists, or widens the lifetime of a DI-resolved service, later invalidated by a concurrency/lifetime feature). Nine matched or nearly matched the shape; **NotificationOutbox-006 is the only one that ever became a bug** and was already fixed as row 45. The other eight verify STILL-SAFE or SUPERSEDED-BENIGN — the two with real concurrency nearby each anticipated it explicitly (`DeploymentService`'s `dbGate` serialises the one repository touch inside the bounded-parallel Send phase; `ArtifactDeploymentService` builds all per-site commands serially before fanning out, commenting "DbContext is not thread-safe"). An independent product-code cross-check confirms the pattern is absent: no `IServiceScope` is stored on a field anywhere in `src/` (every long-lived holder takes `IServiceScopeFactory` and opens a scope per operation), every `Task.WhenAll`/`Parallel.*` site either touches no scoped service or opens a per-operation scope, and no scoped-registered service is captured by a singleton. One stale archive corrected: NotificationOutbox-004 was marked Resolved with an `_Unresolved._` body; the real resolution is now recorded. Near-miss shapes recorded as rows 58–60. | Found while fixing row 45 | Closed 2026-08-15. | — |
| 50 | ~~**No guard against capturing a scoped service on a singleton actor (captive dependency)**~~ **RESOLVED 2026-08-15 (`00a804e6`).** Both mitigations shipped, and they are complementary. (1) `ValidateScopes`/`ValidateOnBuild` in Development on **both** role branches via `Host/ServiceProviderValidation.cs` — written out rather than left to `CreateBuilder`'s silent default; production stays unvalidated deliberately (fail-closed at boot). Switching it on surfaced **two real Site-role defects**: `ReconcileService` and `AuditLogKpiSampleSource` were registered on sites but need central-only repositories, so neither could ever be constructed there — one unconstructible descriptor fails `ValidateOnBuild` for the entire site graph, which is exactly why `SiteHealthEndpointTests` had quietly pinned `UseEnvironment("Production")` to dodge validation. Both moved to central-only registrations (`AddCentralCommunication()` / `AddAuditLogCentralKpi()`); the workaround is gone and the site graph now boots validated. (2) `CaptiveDependencyGuard` reflects over all 28 shipped actor classes and fails on any instance field (backing fields included) typed `IServiceScope`, `DbContext`, or Scoped-in-the-real-containers — the scoped set is derived from the built Central and Site graphs, so a lifetime change auto-re-scopes the check; `IServiceProvider`/`IServiceScopeFactory` fields are allowed (the correct pattern). Proven non-vacuous via permanent positive-control dummy actors. No live violation; the two test-only `_injectedRepository` ctor fields allowlisted with written safety arguments plus a staleness self-test (row 61). Doc: REQ-HOST-11 in `Component-Host.md`. | Found while fixing row 45 | Closed 2026-08-15. | — |
| 51 | ~~**`BundleLockedException` reaches the CLI as "An internal error occurred"**~~ **RESOLVED 2026-08-15 (`b449fe20`).** Relocated to `Commons/Types/Transport/` beside `BundleUnlockThrottledException` (row 44's pattern); both consumers already carried the namespace, so no call-site edits. `ManagementActor.HandlePreviewBundle`/`HandleImportBundle` now rethrow it as `ManagementCommandException`, so CLI and UI print the lockout reason instead of a correlation id. Fault-mapping tests mirrored from row 44 on both handlers. Follow-up in the same diff (`73f518d6`): deleted the dead `BundleUnlockRateLimitedException` (superseded by row 44, zero references — three similarly-named unlock exceptions with one inert was a live trap). | Found while fixing row 44 | Closed 2026-08-15. | — |
| 52 | ~~**The unlock throttle's counters are per-node and per-process**~~ **RESOLVED 2026-08-15 (`677263af`).** The trailing-hour window now lives in the central MS SQL `BundleUnlockAttempt` table — the HA store both central nodes already share, so no new infra, port or consensus. `MaxUnlockAttemptsPerIpPerHour` is a true fleet cap and survives a node restart; config keys, semantics and the rig's 500 override are unchanged, as are all row-44 behaviours (lockout checked first so locked bundles spend no budget; denials burn no strike and are not recorded; `<unattributed>` bucket; attempts-not-failures). Three recorded trades: **fails open** (store error → Warning naming client + cap, attempt permitted — availability of legitimate imports outranks a best-effort cap over an already-authenticated Administrator-gated surface; a process-local fallback was considered and rejected as it would still block an operator's legitimate 11th import); **approximate** (count-then-insert is not atomic across nodes; overshoot bounded by in-flight count, self-corrects within the window — row 64); **opportunistic prune on the unlock path** (CAS-claimed ≤ every 15 min per process, ≤1000 rows, retains 2× the window so a prune can't race a count — row 65) rather than a new background service. EF migration `20260815154803_AddBundleUnlockAttemptTable` + hand-written production script `docs/plans/sql/AddBundleUnlockAttemptTable.sql`. Integration test proves two limiter instances over one shared store enforce one combined window, restart durability, and store-failure fail-open. | Row 44 implementation note; documented in `docs/components/Transport.md` | Closed 2026-08-15. | — |
| 53 | ~~**`MaxConcurrentImportSessions` has no observability**~~ **RESOLVED 2026-08-15 (`372ea4df`).** Gauge `scadabridge.central.transport.import_sessions.open` on the `ZB.MOM.WW.ScadaBridge` meter — already in `SiteServiceRegistration.ObservedMeters` (the single shared allowlist source for both roles; no appsettings pins one), so no allowlist change was needed and the series is not silently dropped. Provider-backed like the S&F queue-depth gauge: registered by `BundleSessionEvictionService.StartAsync`, identity-checked-cleared on `StopAsync`. `IBundleSessionStore.OpenSessionCount` counts only **unexpired** sessions, so the gauge doesn't read "pinned at the cap" for a sweep interval after a burst drains. `BundleSessionStore.Open` logs Warning with open count, cap and TTL on a genuine cap hit only (a store full of expired sessions is reclaimed silently). No alertable rejection counter yet — row 66. | Found while fixing row 43 — R6 could only infer the leak from rotating test failures | Closed 2026-08-15. | — |
| 54 | ~~**`scadabridge_site_connection_up` is misleadingly named** *(Low, observability)*~~ **RESOLVED 2026-08-15 (`b5f120c3`).** Renamed to `scadabridge.site.stream.subscribers.active` (a test pins the new name AND asserts the old one is gone; doc/script references updated) and BOTH real data-connection gauges added: `scadabridge.site.data_connections.connected` / `.total` — the denominator ships with the numerator because neither alone separates "device down" from "nothing configured"; `connected < total` is the alert condition. Sourced from the DCL's four state transitions, deliberately NOT from the health report, so the metric survives a dead report pipeline (and stays clear of the concurrent row-55 fix in `SiteHealthCollector`). All on the already-allowlisted `ZB.MOM.WW.ScadaBridge` meter. Expected rig shape: connected==total on the active site node, 0/0 on the standby (healthy — connections open on the active node only). | Row-48 close-out (2026-08-15) — the row's original mis-diagnosis traced directly to this | Closed 2026-08-15. | — |
| 55 | ~~**`tagResolutionCounts` reads `{totalSubscribed:0, successfullyResolved:0}` for a connection with 2 live Good-quality subscribed tags** *(Low)*~~ **RESOLVED 2026-08-15 (`5da27e05`).** All three suspected reporting-path hypotheses were WRONG — live debugging showed the zeros and the `good:2` on the SAME report from the SAME collector on the active node, ruling out standby shadowing, keying races, and the central merge in one stroke. The real bug is a DCL omission, not a regression: `DataConnectionActor.HandleSubscribeCompleted` — the first-time, everything-resolves-cleanly subscribe path — was the missing FOURTH `UpdateTagResolution` push site. `BecomeConnected`'s legitimate 0/0 push fires before the stashed subscribe unstashes (proven in live logs: "Entering Connected state" follows the subscribe line), `ScheduleTagResolutionRetry` cancels outright at zero unresolved, and any reconnect masks the bug via the batch path — so the diagnostic was blindest on a perfectly HEALTHY connection, exactly when it should read clean rather than empty. One mirrored push + regression test (stash-verified fail-before: exactly one `0/0` call received pre-fix while the actor's own `GetHealthReport` already said 2/2, pinning the actor-correct/collector-stale asymmetry). Also corrected `Component-DataConnectionLayer.md`, which claimed the counts were "derived at report time" — they are pushed, and a cached push is only as fresh as its last push site. | Row-48 live probe; root-caused live 2026-08-15 | Closed 2026-08-15. | — |
| 56 | ~~**The rig's central nodes expose no Prometheus endpoint** *(Low, rig config)*~~ **RESOLVED 2026-08-15 (`103f2f19`).** Premise refined on inspection: central already served `/metrics` on its web port (anonymous, no fallback policy) — what was missing was the dedicated 8084 listener the documented docker-exec scrape recipe targets. Both rigs' central appsettings now set `MetricsPort`, and the Host binds it as a PRESENCE check, not a defaulted read — `NodeOptions` defaults to 8084, so a defaulted read would make every existing production central surprise-bind a new port on upgrade, and port-in-use is a hard boot failure. | Row-48 live probe | Closed 2026-08-15. | — |
| 57 | ~~**`audit-config query` has no CLI table projection** *(Low, cosmetic)*~~ **RESOLVED 2026-08-15 (`b2cab59c`).** `AuditConfigTableProjection` mirrors `TemplateTableProjection`: unwraps the `{entries, totalCount}` envelope to rows (`id, timestamp, user, action, entityType, entityName, entityId`), drops the fat `afterStateJson`, adds `--detail` to skip projection (matching `template list`), and surfaces `totalCount` as a trailing note row (`showing N of M — use --page/--page-size`) emitted ONLY when the page is a strict subset, with no `id` cell so it can't be mistaken for an entry. The deprecated `audit-log` alias shares the command tree and gets the projector too. Tests serialize the REAL `AuditLogQueryResult`/`AuditLogEntry` types, so a server-side property rename fails the test instead of silently blanking a column. Stale-README follow-up recorded as row 76. | Row-46 fix (2026-08-15) | Closed 2026-08-15. | — |
| 58 | ~~**`IDeploymentStatusNotifier` singleton retains delegates targeting circuit-scoped Blazor components** *(Low)*~~ **RESOLVED 2026-08-17 (`2807b5a3`).** Still exactly one subscriber (`Deployments.razor`), but the cited pin was BLIND: `Deployments_Dispose_UnsubscribesFromNotifier` was satisfied by the `_disposed` guard alone — mutation-verified (with the `-=` deleted, the old test stayed green while the leak was wide open). Closed by two real pins: `DeploymentsPushUpdateTests.Deployments_Dispose_ActuallyDetachesTheDelegate_NotJustGuardsIt` (structural — the invocation list must shrink to 0) and `EveryCentralUiSubscriberToTheNotifier_UnsubscribesAndIsDisposable` (a source scan pinning this row's own trigger: any new subscriber without a matching Dispose unsubscribe fails the suite). Both fail on the mutation. | Row-49 sweep (CentralUI-006 re-verification) | Closed 2026-08-17. | — |
| 59 | **`SiteCallAuditActor` per-tick scope spans all sites × ≤50 upsert pages** *(Low)* | Row-49 sweep (SiteCallAudit-009 re-verification); re-verified 2026-08-17 (Low-tail sweep — loop still serial, line numbers drifted to `SiteCallAuditActor.cs:528-545`) | Safe only because site reconciliation is a serial `foreach … await`; one scope across a parallel fan-out would be row 45 again. 2026-08-17: the missing in-code constraint remark was added at the loop (`2807b5a3` — "serial by design … do not fan out"); a source-scan pin was judged too brittle, so the comment + this row remain the tripwire. | `ReconcileSiteAsync` is ever fanned out with `Task.WhenAll` |
| 60 | ~~**`ArtifactDeploymentService.GlobalArtifactSnapshot` holds EF-materialised entities across the parallel per-site send** *(Low)*~~ **RESOLVED 2026-08-17 (`2807b5a3`) — with a premise correction so the row is never reopened on the wrong reasoning.** The row's premise was factually wrong: `GlobalArtifactSnapshot` holds pure Commons DTOs, not EF entities (`ArtifactDeploymentService.cs:170-212`); the EF state that actually crosses the fan-out is the tracked `Site` entities the per-site lambda closes over. The safety conclusion still holds — commands are built serially before the fan-out and lazy loading is confirmed absent — and the load-bearing half of the trigger is now permanently pinned by `ConfigurationDatabase.Tests.AddConfigurationDatabase_DoesNotEnableLazyLoading`. | Row-49 sweep (DeploymentManager-023 re-verification) | Closed 2026-08-17. | — |
| 61 | ~~**`_injectedRepository` test-mode constructors are one call site from re-introducing row 45** *(Low)*~~ **RESOLVED 2026-08-15 (`92fe5acc`).** Preferred fix done: both actors now have exactly ONE constructor (`IServiceProvider`); 12 test call sites folded onto stub providers via two `TestSupport` helpers. Also deleted as newly-dead: `OnCachedTelemetryWithoutDualWriteAsync`, `SiteCallAuditActor`'s `_backgroundTimersEnabled` (a constant once the test ctors went), and the private `SerializedRepository` — moved into the test project, since production resolves a fresh repository per scope and the concurrency hazard existed only for fixtures sharing one instance. `CaptiveDependencyGuard` allowlist is now EMPTY; the staleness self-test replaced by two stronger pins — `Allowlist_IsEmpty` plus `FormerlyAllowlistedActors_NowScanClean`, which scans both actors with the allowlist DISABLED, so emptiness is evidence the fields are gone, not that the scan stopped looking. Control: re-adding a field fails both pins and the real scan. | Row-50 guard allowlist | Closed 2026-08-15. | — |
| 62 | ~~**Five SiteRuntime actors resolve `ISiteEventLogger` off the root provider without a scope** *(Low)*~~ **RESOLVED 2026-08-16 (`9d668861`) — pinned, and the row undercounted: there are SIX sites.** `SiteEventLoggerLifetimePinTests` (Host tests, registration-only inspection, no provider built) pins TWO facts — the interface registration is a *factory* forwarding to concrete `SiteEventLogger`, so pinning the interface lifetime alone doesn't prove callers share an instance; the concrete registration is pinned too, and flipping only the concrete to Scoped is caught ONLY by the second pin (verified both directions, registration restored). The pin's failure message names all six root-resolve sites: the five in this row plus `SiteRuntime/Scripts/ScriptRunLauncher.cs:224` — the run body WP3.1 lifted verbatim out of the deleted execution actors, which inherited the pattern unnoticed. | Row-50 guard allowlist | Closed 2026-08-16. | — |
| 63 | ~~**Rig site containers run without `ASPNETCORE_ENVIRONMENT`, so the row-50 Development validation never fires there** *(Low, rig config)*~~ **RESOLVED 2026-08-15 (`0d2d6de6`).** Set on all six `docker/` site containers and both `docker-env2` site containers, mirroring the centrals; both compose files re-parsed to prove well-formedness; the now-stale `Program.cs` comment corrected. A captive dependency in the site graph now fails the container at boot on the rig, not just in CI. | Row-50 implementation | Closed 2026-08-15. | — |
| 64 | **The fleet-wide unlock throttle's count-then-insert is not atomic across central nodes** *(Low, accepted by design)* | Row-52 implementation trade, documented in both Transport docs (re-verified 2026-08-17: `Component-Transport.md:341`, `docs/components/Transport.md:140`, plus the code XML doc) | Simultaneous attempts can overshoot `MaxUnlockAttemptsPerIpPerHour` by the number in flight; bounded and self-correcting within the window. A conditional single-statement insert would close it at the cost of provider-specific raw SQL. | A threat model needing a hard cap rather than a rate damper, or evidence of a real brute-force exploiting the overshoot |
| 65 | ~~**`BundleUnlockAttempt` rows are pruned only by nodes actively serving unlocks** *(Low)*~~ **RESOLVED 2026-08-17 (`2807b5a3`) — the revisit trigger had FIRED.** Transport gained exactly the central scheduler the trigger named: `BundleSessionEvictionService` (central-only `BackgroundService`; `AddTransport` is called only from the central branch of `Program.cs`). The prune tick now rides it, so an idle cluster's rows age out without an unlock ever occurring; the opportunistic unlock-path prune remains. Operational note: the service now has two maintenance duties, deliberately in separate `try` blocks so neither can mask the other. Tests: `FleetWideUnlockThrottleTests` additions + `BundleSessionObservabilityTests` extension, both green. | Row-52 implementation trade | Closed 2026-08-17. | — |
| 66 | ~~**No counter for import-session cap rejections** *(Low)*~~ **RESOLVED 2026-08-15 (`90ce0999`).** `scadabridge.central.transport.import_sessions.rejected` on the existing allowlisted meter, emitted ONLY at the post-eviction recheck — the cap is tested twice and a store full of expired sessions reclaims and succeeds, so counting the first test would fire routinely on any node whose sweep hadn't run; duplicate-session-id re-`Open` is also excluded. New tests use delta-over-scoped-action with unique per-test keys (the pre-existing gauge tests' absolute asserts are row 74). | Row-53 implementation note | Closed 2026-08-15. | — |
| 67 | ~~**Startup reconcile stores a fetched instance config but does not start the InstanceActor** *(Medium-Low, availability)*~~ **RESOLVED 2026-08-15 (`482165cb`).** Root cause: InstanceActors are created from exactly two paths — the boot-time SQLite load in `DeploymentManagerActor.PreStart` and an explicit deploy/enable command — and the reconcile pass fires ~5s AFTER the boot load with a pure file write (`StoreDeployedConfigIfNewerAsync`) as its only side effect. Fix: reconcile collects the enabled instances it stored and Tells a new `EnsureInstanceStartedCommand` to the DM **singleton proxy**, so the actor is created under the singleton no matter which node's pass found the gap. Handler is advisory and idempotent: no-op if already running; re-reads the durable row (authority on config body AND `is_enabled`, so a disable landing after the fetch wins); defers to `_deployWarms`/`_terminatingActorsByName`/`_inFlightDeployments` so deploy-gate ordering and all-or-nothing-per-instance are untouched (it starts an already-persisted config, never applies one); quiet no-op on an absent row (normal on an unreplicated site whose peer healed its own file). The heal is no longer silent — a Warning site event `Instance <X> started by startup reconcile self-heal` fires. Fail-before by behavioral revert: 3 of 8 new tests fail pre-fix, including the exact live symptom (`debug snapshot` → `instanceNotFound`); the 5 negative guards (disabled / failed-fetch / unknown / idempotence) green in both states. SiteRuntime 613/613, Host 504/504. Sibling gaps found during the fix recorded as rows 72/73. | Proven live during the rows-46–53 close-out (2026-08-15) | Closed 2026-08-15. | — |
| 69 | ~~**Both site-a nodes' LocalDb files corrupt — replication down, one node crash-looping**~~ **RESOLVED 2026-08-15 (operational recovery, no code change).** Found live during this wave: site-a-b threw `SQLite Error 11: 'database disk image is malformed'` on every sync call (replication `connected: false`, backlog 892, faulted session retried ~every 60s) and, after the redeploy, site-a-a crash-looped (SIGABRT ×12) on the same error at `SqliteLocalDb` construction — the new boot-time re-registration reads pages the old build never touched, so site-a-a's corruption (present but latent) became fatal only on upgrade. Recovery: pair stopped together, both files quarantined (`*.corrupt-2026-08-15`, kept for forensics; 394 MB / 416 MB), fresh files reseeded — site-a-b via its startup reconcile (which live-proved the row-67 fix, see row 67) and CDC snapshot resync; converged to `connected: true`, backlog 0, with the row-27 schema change proving the digest handshake green. Suspected cause is host-level, not product: see row 79. | Found live by the row-55 debug + this wave's close-out redeploy | Closed 2026-08-15. | — |
| 70 | **`auto-down-unreachable-after` is deprecated — Akka warns "keep-majority will be used instead" on every boot** *(Medium info — verified NOT currently true, but a live removal risk)* | Row-55 live session; investigated + drill-verified at this wave's close-out | The warning is real (fires at Cluster start on every node, Akka.NET 1.5.62 — same version as the 2026-07-21 decision, so nothing changed underneath; it fires on the presence of the `auto-down-unreachable-after` key). **Drill-proven that the substitution does NOT happen when `downing-provider-class = "Akka.Cluster.AutoDowning"` is EXPLICIT (as `BuildHocon` sets):** `DRILL_MODE=active` kills central-a (oldest AND lowest address) — under keep-majority a 2-node even split keeps the lowest-address side, so surviving central-b would have downed ITSELF (total outage); instead central-b observed OldestChanged → became Oldest, took over all singletons, and the victim rejoined ready 2s after restart (2026-08-15 drill, post-merge rig). The 2026-07-21 decision therefore still holds exactly as documented. The residual risk is the deprecation itself: a future Akka.NET bump can delete `AutoDowning`, silently or loudly breaking the availability-first strategy. | **Any Akka.NET version bump** — re-run `DRILL_MODE=active bash docker/failover-drill.sh` and re-read the release notes before accepting it |
| 71 | ~~**Standby nodes log the active-node health check at ERR every 10s** *(Low, log noise)*~~ **RESOLVED 2026-08-16 (`8b761841`).** One narrowly-keyed Serilog `Filter.ByExcluding` in `LoggerConfigurationFactory`: `Level == Error` AND the `DefaultHealthCheckService` category AND `HealthCheckName == "active-node"` — check-NAME keying is what preserves the constraint (`localdb`/`akka-cluster`/`database`/`required-singletons` keep their Error lines, and active-node has no error path of its own on the active side, so the only suppressible event is a standby reporting it's a standby). Rejected on evidence: `MinimumLevel.Override` can only raise a floor (can't rewrite Error→Information), and reporting `Degraded` would change the HTTP status `/health/active` publishes, which is load-bearing for Traefik. Bonus reach: the CENTRAL standby was logging the same Error 2x as often (Traefik probes `/health/active` every 5 s) — covered by the same filter. Fail-before: filter disabled → `StandbyActiveNodeVerdict_IsNotLogged` fails; the 9 must-still-log constraint tests pass either way, proving narrowness. Side-finding: the observed ~10 s SITE-side cadence comes from an out-of-repo prober (nothing in this repo probes a site's `/health/active`), and CLAUDE.md's "site nodes map no `/health/*`" claim was stale — corrected. | Row-55 live session (2026-08-15) | Closed 2026-08-16. | — |
| 72 | ~~**Reconcile refreshes a STALE config but does not restart the running InstanceActor** *(Medium-Low — same silent-divergence class as row 67, arguably wider blast radius)*~~ **RESOLVED 2026-08-16 (`e606b24a`, with row 78 — one fix, the DM instance sweep).** Design decision made and recorded (`docs/plans/2026-08-15-instance-sweep-self-heal-design.md`): **converge, don't merely warn** — the sweep re-applies the stored config through **`HandleDeploy`**, the exact path a central redeploy uses, so the compile gate, warm-then-gate ordering, redeploy-supersede, static-override/native-alarm reset and all-or-nothing-per-instance are the production ones (a stop still does not cancel an in-flight run). Rationale: the stored row IS what central successfully deployed; surface-only leaves the operator to hand-issue the very redeploy the sweep would perform. Ordering hazards checked: the sweep never acts on an instance a deploy pipeline owns; a gate-rejected restart is inert, not partial; the synthetic deploy replies to `Self`, never `NoSender` (dead letters are a health metric). "Nothing surfaces it" is now false in BOTH paths — even a failed restart yields two `deployment` site events naming both revisions. Safety property beyond the brief: **one repair attempt per (instance, revision) pair**, so the self-heal can never become a restart loop for an init-crashing actor or a config the gate can never accept. Fail-before: sweep unarmed → `Sweep_RestartsInstance_ServingAStaleRevision` fails showing rev-1's value served while the store holds rev-2. | Row-67 fix session (2026-08-15) | Closed 2026-08-16. | — |
| 73 | ~~**`SiteReconciliationActor` is silently skipped when `IDeploymentConfigFetcher` is null** *(Low)*~~ **RESOLVED 2026-08-16 (`a2f0086f`, with row 80).** New tri-state health-report field **`StartupReconcilerActive`** (`bool?`, matching the `LocalDbReplicationConnected` nullable precedent; additive proto **field 29**, vendored C# regenerated via `docker/regen-proto.sh centralcontrol` with the csproj verified restored): null = not reported (central self-report or pre-row-73 site build), false = booted with no fetcher. Latched once at boot by the Host in both branches. The skip branch was raised **Warning → Error** (no configuration legitimately omits the fetcher on a site node) plus a best-effort `deployment`/`Error` site event so evidence outlives the boot log. Central treatment: a **"No self-heal" Warning badge**, deliberately not a failure — config still arrives via replication while the node is up; what's lost is recovery from a deploy landing while it's down. Only explicit `false` is badged (badging null would light the fleet the moment central upgrades ahead of the sites); offline sites skipped, matching the backlog badge. | Row-67 fix session | Closed 2026-08-16. | — |
| 74 | ~~**`BundleSessionObservabilityTests` gauge tests assert absolute values — latent row-32-class race** *(Low, test-only)*~~ **RESOLVED 2026-08-16 (`98c94588`).** Both gauge facts converted to delta-over-scoped-action via a `ReadGaugeDelta` helper — and two assertions got STRONGER: the post-clear check (`Assert.Equal(0, …)` — satisfied by any unregistered slot) now asserts opens on the detached store move the gauge by nothing (the actual claim), and the compare-and-clear fact gained a negative control (the superseded store's opens must NOT reach the instrument), without which it passed by coincidence whenever both stores held equal counts. Fail-before: a temporary foreign-provider probe returning 7 pre-start — old body fails `Expected: 0 / Actual: 7`, new body passes the identical scenario (probe deleted, not committed). Documented residual: a registrant clobbering the slot BETWEEN the two reads still needs row-32's mutual-exclusion pattern; cannot arise today (sole constructor of `BundleSessionEvictionService` in the assembly). | Row-66 implementation (2026-08-15) | Closed 2026-08-16. | — |
| 75 | ~~**SecuredWrites Approve renders as a red destructive button** *(Low, UI)*~~ **RESOLVED 2026-08-16 (`40944a0d`).** The real defect was deeper than "red means delete": Approve and Reject rendered as IDENTICAL red buttons, so at the moment of a two-person-control decision colour carried no discriminating information — because `danger` was a two-state flag for a three-state concept. Fix: `DialogSeverity` (`Normal→btn-primary`, `Caution→btn-warning`, `Destructive→btn-danger`); **Approve → Caution amber** (executes a real write to live plant equipment — consequential, not destroying), **Reject keeps Destructive red** (terminal for the request; and holding it red keeps the two opposite decisions distinct — dropping it to neutral would just invert the bias). Only `Destructive` inherits row-28's "Delete" label fallback, or row 28 returns through the new door. Implemented as an ADDITIVE overload deliberately: a 5th optional parameter would silently un-match every 4-arg NSubstitute stub (the row-28 gotcha, null Task → NRE); the overload's only breakage was 6 hand-written fakes failing to COMPILE — the loud mode. `DialogState.Danger` became `Severity` (one field can't disagree with itself). Playwright `.btn-danger` selectors stay valid (`Destructive` mapping unchanged). Fail-before: call site reverted to `danger: true` → both Approve pins fail; `Caution→btn-danger` remap → amber styling pin fails. Decision recorded in `Component-CentralUI.md`. | Row-28 audit (2026-08-15) | Closed 2026-08-16. | — |
| 76 | ~~**CLI README's `audit-config query` JSON sample omits `afterStateJson`/`bundleImportId`** *(Low, docs)*~~ **RESOLVED 2026-08-16 (`4ae57bdb`).** Sample now shows both fields with their nullability semantics, and was grown to two entries mirroring the table-output sample below it row-for-row — which is what makes the "the table drops these two fields" sentence checkable. | Row-57 fix (2026-08-15) | Closed 2026-08-16. | — |
| 77 | ~~**Test pattern: reflecting `ActorCell.Actor` right after `ResolveOne` is unordered vs child construction** *(Low, test-infra)*~~ **RESOLVED 2026-08-16 — swept clean, no commit needed.** Full `tests/` sweep: `ActorCell` (1 hit — the already-fixed file's comment), `ActorRefWithCell` (1 — fixed file :135), `.Underlying` (1 — same line), `BindingFlags` near actor/cell (2 — same file), `.Cell` (0), `ResolveOne` (3 files: the fixed one; `SiteAlarmLiveCacheServiceTests:432` resolves-then-`Sys.Stop`s, benign; `ReconcileStartsInstanceTests:363` uses the ref only for `IActorRef` equality, benign), `UnderlyingActor` (6 files — all Akka.TestKit `TestActorRef<T>` via `ActorOfAsTestActorRef`, an ORDERED mechanism, not this pattern). The unordered idiom exists in exactly one place and already carries the bounded-deadline retry. | Row-40 session (2026-08-15) | Closed 2026-08-16. | — |
| 78 | ~~**Configs arriving via LocalDb replication start no InstanceActor — row 67's gap through a side door** *(Medium-Low, availability; proven live 2026-08-15)*~~ **RESOLVED 2026-08-16 (`e606b24a`, with row 72 — the DM instance sweep).** A periodic sweep on the Deployment Manager **singleton** (`ScadaBridge:SiteRuntime:InstanceSweepIntervalSeconds`, default 60, validated `>= 0`, `0` disables — default is live everywhere with no rig config change, which is intended): each tick re-reads the durable rows off-thread and, on the actor thread, starts any **enabled** stored config with no running actor. Arrival-path-agnostic by design — covers replication seeding, post-boot snapshot resyncs, and any future path. A LocalDb change hook was considered and rejected on evidence: it covers only the replication path and only the replicated half of the fleet (site-b/c aren't), AND `ILocalDb` 0.3.0 exposes no change-notification API at all, so a hook means a cross-repo upstream capability to close a site-local gap. Warning site event on a real heal names the sweep as the trigger; silent when there is nothing to do. Fail-before (sweep unarmed): 5 of 10 new tests fail including the exact live symptom ("A replication-seeded config for an enabled instance left no running Instance Actor"); 5 negative guards (disabled / in-flight-deploy deferral / matching-revision no-op / silence / interval-0) green in both states. Design note: `docs/plans/2026-08-15-instance-sweep-self-heal-design.md`. **LIVE-PROVEN at close-out (2026-08-16) in the exact original failure sequence:** site-a pair stopped, node-a's LocalDb file quarantined, pair restarted together — node-a's 08:45:55 reconcile reported "0 fetched" (replication had seeded the fresh store; central rightly saw no gap — the precise side door), and at 08:46:50, one sweep interval later, BOTH enabled instances started with the Warning `Instance rig-probe-1 started by deployment-manager sweep self-heal — its config is present in the deployed-config store but no Instance Actor was running until now`; `debug snapshot` live with no extra restart. The probe also exposed the DCL sibling gap — row 85. | This wave's close-out (2026-08-15) | Closed 2026-08-16. | — |
| 79 | ~~**OrbStack virtiofs is a live corruption + availability hazard for the rig's bind-mounted SQLite files**~~ **RESOLVED 2026-08-18 — the prescribed remedy executed without waiting for a second corruption: all ten rig data dirs (8 `docker/` nodes + 2 `docker-env2/` site nodes) moved to named docker volumes** (`scadabridge-<node>-data` / `scadabridge-env2-<node>-data`, explicit `name:`s in both compose files), seeded from the old bind dirs with both rigs stopped, then redeployed and live-verified: site-a `localDbReplicationConnected: true` / backlog 0 over the migrated files, deployed instances resubscribed on boot (data continuity proven), env2 stable with 0 restarts. The trade-off that had deferred this — losing host-side inspectability — turned out to be mostly moot: OrbStack exposes every volume on the Mac at `~/OrbStack/docker/volumes/<name>/` (read-only inspection; never write through it). The rebuild also surfaced latent env2 drift: env2 was running a pre-Secrets-0.6.2 image, so on the fresh image all four env2 app nodes failed the new store-path validation at boot — fixed by porting the main rig's `x-secrets-store-env` posture (`Secrets__SqlitePath=/data/…` + `/data` second mounts, incl. NEW env2 central data volumes; env2 centrals previously kept data on the ephemeral overlay). Residual accepted: failure mode 2 (VM idle-sleep, `orb restart docker` recovery) is availability-only and unaffected — data now survives it on ext4. Hazard history + recovery runbook retained in `docs/deployment/topology-guide.md` (section updated). Old `docker/<node>/data/` dirs left in place as dormant pre-migration backups. | This wave's close-out (2026-08-15); runbook 2026-08-16; migration executed 2026-08-18 | Closed 2026-08-18. | — |
| 80 | ~~**Unreplicated sites report NON-NULL `LocalDbReplicationConnected: false` / `LocalDbOplogBacklog: 0` — contradicting the documented nullable semantics** *(Low-Medium, observability correctness)*~~ **RESOLVED 2026-08-16 (`a2f0086f`, with row 73).** Root cause was type-level: `ISiteHealthCollector.SetLocalDbReplicationStatus(bool …)` made null INEXPRESSIBLE for the connected flag, and the Host wired the bridge unconditionally — the replication engine is registered on every site node and idles answering `false`/`0` with no peer. Fix mirrors the `SiteEventLogReplicationCheck` precedent exactly: the Host evaluates `SiteLocalDbSetup.ReplicationIsConfigured` once and passes it as a third provider to `AddLocalDbReplicationHealthBridge` (the SAME captured value now also feeds `SiteEventLogReplicationCheck`, so the two surfaces can't disagree); unconfigured short-circuits to `(null, null)`; the collector widens to `bool?`. The rest of the chain needed NO change — proto already `BoolValue`/`Int64Value` wrappers, `NextBacklogAboveSince` already leaves the anchor unchanged on null, KPI source already conditional: row 33's machinery was correct and had simply never received a null (its null branch was dead code in production until now). `false` now means exactly one thing: configured, peer session down = real fault. Trap that produced the bug, now pinned in tests: configured-ness is `PeerAddress` **OR** `ApiKey` — "no PeerAddress" is NOT "unreplicated" (a passive-half node carries the key alone); the pre-existing Host fixture asserting `false`/`0` was exactly such a node and its assertion is CORRECT — kept with an explaining comment, plus a new genuinely-unreplicated fixture. Fail-before: gate behaviourally reverted → `UnreplicatedSite_HealthReport_ReportsNullForBothLocalDbFields` fails `Expected: null / Actual: False`. | Found at the previous wave's close-out verifying row 33 | Closed 2026-08-16. | — |
| 81 | ~~**The site-event-log storage cap cannot be enforced on a replicated node with an undrained oplog** *(Medium-Low, measured 2026-08-16 during the row-34 work)*~~ **RESOLVED 2026-08-17 (`ddb3c32e`) — but NOT via the row's proposed `dbstat` fix, which is impossible on this stack.** Probed BEFORE implementing: the bundled SQLite 3.53.3 has no `ENABLE_DBSTAT_VTAB` (`pragma_module_list` returns 10 modules, none of them `dbstat`/`sqlite_dbpage`); pinned by `SqliteCapabilityTests`, written to FAIL if a future bump enables it (row 94). Switching SQLite builds rejected — the provider is shared with `ZB.MOM.WW.LocalDb` fleet-wide. The cap instead measures `site_events` + its four indexes via a sampled page-layout model: row width sampled 500 rows from each end per tick (`octet_length`), whole-page table cost (linear models under-estimate near-half-page rows by ~40%), clamped by whole-file logical size (a true upper bound). Index pages count against the cap (~34% of footprint — the 1 GB default was sized against a figure that included them). Measured model error −0.8%/+8.4%/+12.8%, biased high on purpose; cross-checks row 34's independent 393 B/row at 397.2. Guard semantics re-based: (1) unsampleable row width ⇒ refuse to trim (checked BEFORE the cap comparison); (2) the old "size did not decrease" guard now runs on the table-scoped estimate, so replication bookkeeping can no longer trip it — it now means exactly one thing: events arriving as fast as the trim removes them. Bytes-per-row held fixed per tick (re-sampling per batch would jitter the estimate through guard 2 — the same self-defeating measurement as the row itself). LocalDb bookkeeping explicitly NOT managed here (`MaxOplogRows`/tombstone retention own it); topology guide gained a volume-sizing subsection stating the honest budget. Fail-before: reverting to whole-file semantics fails 5 new tests and reproduces the exact original signature (rows destroyed, cap still unmet). SiteEventLogging 104/104, Host 520/520. No config keys added or renamed. Decision memo: `docs/plans/2026-08-17-site-event-log-cap-semantics-decision.md`. Model limitations recorded as rows 92/93. | Row-34 measurement harness; §5 of the measurement doc (postscript appended) | Closed 2026-08-17. | — |
| 82 | ~~**`AuditLog.Tests` flakes under parallel full-solution runs — `MsSqlMigrationFixture` contention** *(Low, test-infra)*~~ **RESOLVED 2026-08-17 (`da23f2dc`) — three mechanisms, and the row's "per-run database name" prescription was already in place (the fixture has minted `ScadaBridgeAuditMigTest_<guid>` per class since it was written).** (1) The headline fingerprint was create+migrate SATURATION: nothing bounded how many fixtures ran `CREATE DATABASE` + the ~100-migration chain at once (measured peak: 45 concurrent test DBs / 19 blocked requests / 98 sessions on one container), pushing DDL past EF's default 30 s command timeout — every observed failure was `..ctor → MigrateAsync → SqlException 258 Execution Timeout Expired`, which xunit reports against every test in the class (hence "29 failures"). Fix: a machine-wide counting semaphore on `sp_getapplock` in `master` (`ScadaBridgeTestMigrate/0..N-1`, 4 slots, `SCADABRIDGE_MSSQL_TEST_MIGRATE_SLOTS` override) held across create+migrate — the arbiter must live on the shared SERVER because the contention is cross-process; gate connection is deliberately UNPOOLED (a pooled one defers the session-lock release until reuse, stranding a slot); crashed process ⇒ SQL releases the lock; a gate deadline miss proceeds UNGATED so the throttle can never itself be a red. Timeouts raised to 300 s migrate / 120 s create-drop (bounded queue = busy but progressing); once-per-process sweep of orphaned `ScadaBridgeAuditMigTest_*` >2 h; connectivity probe retries 3× (a busy server misread as absent flips every MSSQL test to a green-looking Skip — worse than a red). (2)+(3) Two more flakes behind it: `PartitionPurgeTests` and `NotifyDispatcherAuditTrailTests` LEAKED 100 ms-tick actors (purge / outbox dispatcher) whose only stop was timeout-bounded `TestKit.Dispose` — a leaked dispatcher even delivered the NEXT test's notification through its own exhausted adapter, stranding it `Retrying`. Fixed with `Watch`/`Stop`/`ExpectTerminated` in the creating test. Evidence: 10× concurrent ConfigurationDatabase.Tests went from 10/10 processes failing (17–77 failures each) to 3 consecutive 10/10-green runs; the AuditLog 3×+1× mix from failing to 3 clean runs; standalone AuditLog 383/383, ConfigurationDatabase 395/395 (no runtime regression), SiteCallAudit 66/66 (inherits the fixture); zero orphan DBs or stuck app locks left. Host.Tests/PlaywrightTests hit the PERSISTENT DB on an already-serialized axis — untouched. Systemic residue recorded as rows 95/96. Doc: `docs/test_infra/test_infra_db.md`. | Found independently by two agents in the 2026-08-16 wave | Closed 2026-08-17. | — |
| 83 | ~~**Post-boot start-path tests must gate on the DM's first instance-count report — `ReconcileStartsInstanceTests` still uses a bare `Task.Delay(500)`** *(Low, test-infra)*~~ **RESOLVED 2026-08-17 (`524bdbbe`).** `ReconcileStartsInstanceTests` converted to the first-`SetInstanceCounts`-report gate; the pattern was extracted into a shared `TestSupport/DeploymentManagerStartupProbe` now used by both this suite and `InstanceSweepSelfHealTests`, so the convention has one implementation to copy instead of two to keep in sync. SiteRuntime 633/633 green. | Row-78 fix session | Closed 2026-08-17. | — |
| 84 | ~~**`LogReplicationBacklogNote` can fire twice in one purge tick** *(Low, cosmetic)*~~ **RESOLVED 2026-08-17 (`ddb3c32e`).** The note is emitted at most once per tick. Fail-before: `RunPurge_WhenBothPhasesDeleteRows_LogsTheBacklogNoteOnceForTheWholeTick` fails pre-fix with `Assert.Single(): collection contained 2 matching items`. | Row-34 session | Closed 2026-08-17. | — |
| 85 | ~~**DCL connection actors are not spawned for connection configs that arrive via LocalDb replication — row 78's gap, one layer down** *(Medium-Low, availability; proven live 2026-08-16 by the row-78 close-out probe)*~~ **RESOLVED 2026-08-17 (`8f2c2813`) — with a premise correction AND a widened blast radius.** The row's mechanism was wrong: `data_connection_definitions` has NO production reader (its getter is test-only) — connection actors are created solely from the `Connections` map in an instance's own flattened config via `EnsureDclConnections`, so there was never a store→actor path to repair. The real defect: that ensure was the CALLER'S job and only 2 of 4 instance-start paths did it (boot batch ✅, `ApplyDeployment` ✅, `StartStoredInstance` ❌ — both the row-67 ensure-started and the row-78 sweep — and `HandleEnableResult` ❌; the two misses are exactly the two paths added later). So the bug was NOT replication-specific: an ordinary unreplicated site that boots with an instance disabled and enables it later hit the same dead end, because the boot load reads only enabled configs. Fix is structural: `EnsureDclConnections` moved INTO `CreateInstanceActor` (the documented single creation funnel), before the already-exists guard so the two working paths keep exact semantics; reuses the per-name hash guard so steady state is silent; a fifth path cannot forget. Store-level sweep pass explicitly REJECTED (memo §7.4): it wouldn't have fixed this (the table is empty on deploy-only sites) and the unguarded form ping-pongs the hash slot against the deploy path, dropping live subscriptions every tick. Fail-before by behavioral revert: 4 of 7 new tests fail (sweep-started, reconcile-ensure-started, enable-after-disabled-boot, exactly-once-across-sweeps). SiteRuntime 633/633, DCL 320/320. Sibling gaps found during the fix recorded as rows 89/90. | Proven live 2026-08-16 by the row-78 close-out probe (sweep started `rig-probe-1`/`soak-notify-1` but every tag subscribe failed for 6+ min with `Unknown connection: OPC PLC Simulator — will retry`; a second pair restart healed it) | Closed 2026-08-17. **Live-proven on the rig same day** in the exact original failure sequence (pair stopped, node-a `site-localdb.db` quarantined as `.probe85-quarantine-2026-08-17`, pair restarted): reconcile "0 fetched" at 08:29:59, then at the 08:30:54 sweep tick `Created DCL connection OPC PLC Simulator` fired ALONGSIDE the two self-heal Warnings (pre-fix they appeared alone), zero `No connection actor`/`Unknown connection` lines on either node, all three `rig-probe-1` tags Good with live values ~2 min after restart — NO second restart; steady-state silence confirmed (0 Created/Updated lines across subsequent ticks); replication reseeded (`localDbReplicationConnected: true`, backlog 0), site-b/c still correctly omit the fields. | — |
| 86 | ~~**CLI and UI disagree on deployment site-scoping for Administrators**~~ **RESOLVED 2026-08-17 (merge `ad0100f2`, agent commit `9f67147e`) — Administrator = system-wide is the intended semantic; the UI was the defect.** The rule was already documented on BOTH sides (`ManagementActor.cs` ~:522 "Users with the Administrator role, or system-wide Deployer, are not restricted"; `SiteScopeService`'s own XML doc "any Admin/Design user — is system-wide") and enforced at 6+ ManagementActor sites — only `SiteScopeService.ResolveAsync` ignored it. It now treats an Administrator as system-wide regardless of `SiteId` claims, read via explicit `FindAll(RoleClaimType)` + `OrdinalIgnoreCase` (NOT `IsInRole`, which compares ordinally and needs a `roleType` the harness identities don't set); an Administrator's site claims aren't collected at all, preserving *system-wide ⇒ empty permitted-id set*. All 15 UI consumer call sites funnel through `ResolveAsync` (surveyed). Fail-before verified: 6/6 new Administrator pins failed with only `SiteScopeService.cs` reverted; `DesignerWithSiteClaims_IsStillScoped` pins that the bypass was NOT widened to Designer (the XML doc's "Design user" claim was inaccurate — corrected). | Found by the 2026-08-17 Low-tail sweep; fixed with the row-37 consolidation | Closed 2026-08-17. | — |
| 87 | ~~**CLI deployment scoping is non-transactional and unbounded**~~ **RESOLVED 2026-08-17 (merge `ad0100f2`, agent commit `9f67147e`), with the row-37 consolidation.** Site grants now ride as `SiteIdScope` resolved inside the query's join — the whole-table `GetAllInstancesAsync` read is gone (pinned by `..._ReadsNoInstancesAtAll`, a strictly stronger replacement for the old "bulk load exactly once" pin, consciously migrated). An unparseable permitted-site id is *dropped* (narrows the scope) rather than discarding the grant (which would widen to fleet-wide). `PageSize <= 0` no longer means unlimited: it takes the CLI's own default 50, capped at `MaxPageSize` 1000. | Found by the 2026-08-17 Low-tail sweep; fixed with the row-37 consolidation | Closed 2026-08-17. | — |
| 88 | ~~**Unparseable CLI `--status` silently means "no filter"**~~ **RESOLVED 2026-08-17 (merge `ad0100f2`, agent commit `9f67147e`).** An unrecognized non-empty status now produces a `ManagementError` naming the valid values (row-46 principle: loud failure over silent empty/full results) — enforced actor-side so HTTP API callers get it too. Case-insensitive parse retained; numeric strings rejected via `Enum.IsDefined` (`Enum.TryParse` accepts `"7"`). CLI help text documents the valid set; deliberately NOT `AcceptOnlyFromAmong`, whose case-sensitive matching is a live minor defect on `audit query --status` (row 99). **Live-probed on the rig:** `deploy status --status Sucess` → `Unknown deployment status 'Sucess'. Valid values: Pending, InProgress, Success, Failed.` **Postscript 2026-08-17 (Wave 6, `16056f8e`): the "numeric strings rejected via `Enum.IsDefined`" claim was WRONG** — `Enum.IsDefined(parsed)` is true for any valid ordinal, so `--status 2` silently resolved to `InProgress`. Fixed with the same `LooksNumeric` first-character guard the row-99 strict parser uses; live-probed: `deploy status --status 2` now fails loud naming the valid values. | Found by the 2026-08-17 Low-tail sweep | Closed 2026-08-17. | — |
| 89 | ~~**Connection-only artifact deploy does not reach the peer node's DCL**~~ **RESOLVED 2026-08-17 (merge `6cfb6a79`, agent commit `c57e74e6`) — premise corrected TWICE, then fixed.** (1) The standby's DCL is **EMPTY**, not "serving the old endpoint": `DeploymentManagerActor` (role-scoped singleton) is the sole sender of `CreateConnectionCommand` and only populates its own node's `dcl-manager`. The real defect was **create-time and not replication-specific**: `_createdConnections` is per singleton *incarnation*, and `data_connection_definitions` had NO production reader — so any restart/failover rebuilt every connection from the flattened-config snapshots and silently discarded newer connection-only artifact deploys (identical loss on an unreplicated single-node restart). (2) Arbitration decided and shipped: **the definitions store wins wherever it has an opinion; flattened config is authoritative only for names the store never owned** (deploy-only sites unchanged — memo §7.4's objection). A `_connectionDefinitions` overlay refreshes from `ReadStoreSnapshotAsync` at boot AND every sweep tick, applied before instance creation; ping-pong killed by construction (every ensure path resolves the *winner* before hashing, so one config per hash slot). Fail-before: 5/8 new `DclConnectionArbitrationTests` fail with the overlay inert. Decision memo: `docs/plans/2026-08-17-dcl-connection-arbitration-decision.md` (records the accepted residuals: re-adds land at next instance start, not instantly — mirrors the removal's own "bad quality until redeploy" semantic, pinned by test). The discovery that a *live* connection actor is never reconfigured in place on ANY node is a separate pre-existing defect → row 97. **Live rig probe at wave close-out (post-redeploy):** the singleton node logged exactly ONE boot-time `Created DCL connection OPC PLC Simulator` with zero update churn across later sweep ticks (no ping-pong from the overlay), the standby stayed silent (topology as corrected), all three rig-probe-1 tags Good, site-a replication connected / backlog 0. | Found by the 2026-08-17 row-85 fix session | Closed 2026-08-17. | — |
| 90 | ~~**No removal path for a connection deleted on the peer**~~ **RESOLVED 2026-08-17 (merge `6cfb6a79`, agent commit `c57e74e6`), with row 89.** Both removal paths now share `RemoveConnectionCentrally` + an in-memory tombstone that `EnsureDclConnections` honors — which also closed a pre-existing bug where an artifact removal was **undone by the very next instance start**. The sweep detects a previously-definition-sourced row's disappearance via its own overlay bookkeeping. Accepted residuals, documented in the decision memo: tombstones are not durable across a restart (by decision — matches the long-standing restart-resurrection semantic), and a delete is undetectable on a node that never adopted the row (the stronger "non-empty table ⇒ complete set" inference would stop live connections on a guess; deliberately not taken). | Found by the 2026-08-17 row-85 fix session | Closed 2026-08-17. | — |
| 91 | ~~**`SiteScriptCompileCacheTests.GetOrAdd_SameCodeAndGlobals_ComputesOnce` is parallel-load fragile**~~ **RESOLVED 2026-08-17 (merge `106e467d`, agent commit `26e3a1b9`) — row-74 pattern applied (exemplar `98c94588`), and wider than the named test.** Two habits: a per-test-instance guid key token (no other class can collide with or serve this test's entries, making `factoryCalls`/`Assert.Same` exact and owned) + one-sided bounds on the process-global counters (`>= baseline + n`; safe because all `Clear()` callers share the serialized collection). Four assertions changed, two got STRONGER (`GetOrAdd_SameCode_DifferentGlobals` had a second absolute pair; `Overflow_StaysWithinTheBound` gained an eviction probe over its own keys — a slack bound is not proof a sweep ran; its `ForeignEntrySlack = 512` is a judgement call, the eviction probe is the load-bearing assertion). Sibling absolutes in the same class and in `DeploymentWarmThenGateTests` fixed too. SiteRuntime.Tests run twice at full parallelism, 633/633 both. The underlying process-global `Clear()` cross-talk hazard remains → row 100. | Found by the 2026-08-17 row-85 fix session | Closed 2026-08-17. | — |
| 92 | **The event-log cap's size model is sample-based and blind to mid-history width shifts** *(Low, accuracy limitation — documented in memo §3.1)* | Row-81 fix session (2026-08-17) | Row width is sampled 500 rows from each END of `site_events`; a stream whose row width changes sharply mid-history is mis-sized until the trim reaches that region. Self-correcting per tick (each trim exposes new tail rows to the sample), and the model's deliberate +8–13% high bias absorbs modest shifts, but a pathological width profile could over- or under-trim for several ticks. **Reviewed 2026-08-18: accepted by design** — inherent to any sampled model, not closable by code short of the row-94 `dbstat` replacement; the row-93 schema pin does not (and cannot) cover it. | First field report of the cap trimming visibly wide of its target, or the row-94 `dbstat` replacement (which retires this row with the model) |
| 93 | ~~**The cap model's page-layout constants are empirical for the CURRENT `site_events` schema**~~ **RESOLVED 2026-08-18 — the silent trap is now loud.** `SiteEventsSchemaCalibrationTests` (SiteEventLogging.Tests) pins the exact shape the calibration assumed: the eight-column list the sampler's `octet_length` sum covers, the index set as exactly the four secondary indexes `IndexedColumnCount = 4` charges plus the PK autoindex (absorbed into the measured fill factors), and each index's single covered column. Any `site_events` schema migration — new column, fifth index, composite widening — now fails the suite with a message directing to the memo §4 recalibration and the constants + sampling-SQL update, instead of silently mis-sizing the cap. Suite green 110/110. | Row-81 fix session (2026-08-17); pinned by test 2026-08-18 | Closed 2026-08-18. | — |
| 94 | **`SqliteCapabilityTests` is a deliberate tripwire — do not "fix" it by weakening the assertion** *(Low, standing note)* | Row-81 fix session (2026-08-17) | The test FAILS BY DESIGN if a future SQLite bump ships `dbstat`/`sqlite_dbpage` support. The correct response at that point is to REPLACE the sampled page-layout model with exact `dbstat` measurement (retiring row 92 — and the row-93 schema pin `SiteEventsSchemaCalibrationTests`, added 2026-08-18, goes with the model too), not to relax the test. The test's own comment says the same. | `SqliteCapabilityTests` fails after a SQLite/provider bump |
| 95 | ~~**The leaked 100 ms-tick test-actor pattern is systemic — three more classes carry it**~~ **RESOLVED 2026-08-17 (merge `106e467d`, agent commit `34a8079d`).** All three classes live in AuditLog.Tests (premise correction: none in SiteCallAudit.Tests). `Watch`/`Stop`/`ExpectTerminated` replicated from `da23f2dc` at every creation site: 11 in `AuditLogPurgeActorTests`, 10 in `SiteAuditReconciliationActorTests`, 2 in `OutageReconciliationTests`. Second premise correction: `OutageReconciliationTests` DID call `Sys.Stop` — but fire-and-forget AND from the last line of the `await using` block, so an in-flight tick could still be writing to MSSQL during read-back and any earlier assertion failure skipped the stop entirely; stops moved before the read-backs, plus the `AuditLogIngestActor` that test 2 was discarding outright (holding a disposed provider) is now stopped too. Known lesser sibling habit left in place: EventStream probes are never unsubscribed (bounded by the per-test ActorSystem — noted, not a row). AuditLog.Tests 383/383. **Postscript 2026-08-19 (arch-review P8-6): the premise correction was itself WRONG — SiteCallAudit.Tests DOES leak 100 ms-tick actors.** `SiteCallAuditPurgeTests` (4 creation sites incl. a discarded `Sys.ActorOf`) and `SiteCallAuditReconciliationTests` (11 sites, two in one test) had ZERO `Watch`/`Stop`/`ExpectTerminated`. Reopened and re-closed 2026-08-19 (agent commit `102e7a2b`): row-82 treatment via a per-file `StopTicker` helper (`try/finally`, blocked test doubles released first) at every site; the same sweep also fixed `PendingDeploymentPurgeActorTests.cs:90` in Communication.Tests (same class), capped Communication.Tests with the row-96 `xunit.runner.json` (`maxParallelThreads: 4` — it was the last uncapped TestKit+Kestrel suite), collection-serialized the two `aggregators.active` gauge-moving classes (row-102 shape), and reworked 11 sleep-then-assert-absence sites in `SiteAlarmAggregatorActorTests` per row-40 (`AssertHoldsFor` polling helper; settle-sleeps before *actions* left alone). Both suites 3× green (717/717, 66/66), zero flakes observed. | Row-82 fix session (2026-08-17) | Closed 2026-08-17; reopened + re-closed 2026-08-19 (arch-review P8-6). | — |
| 96 | ~~**xunit parallelism is uncapped almost everywhere**~~ **RESOLVED 2026-08-17 (merge `106e467d`, agent commit `d2d4db49`) — policy made and shipped: per-suite caps for the resource-heavy suites, pure unit suites stay uncapped.** The four MSSQL-fixture users (`ConfigurationDatabase.Tests`, `AuditLog.Tests`, `SiteCallAudit.Tests`, `Host.Tests` — verified before capping) get `xunit.runner.json` with `maxParallelThreads: 4`, **deliberately matching row-82's `sp_getapplock` slot default** so the in-process and cross-process limits agree; no collections serialized. `IntegrationTests` (already fully serialized) and `CentralUI.PlaywrightTests` (already `maxParallelThreads: 1`, stricter than needed) left alone. Cap verified live (216% CPU uncapped vs 145% capped on ConfigurationDatabase.Tests); cost is nil — these suites are latency-bound on MSSQL/sockets (14–25% CPU), durations unchanged within noise. Policy documented in `docs/test_infra/test_infra.md` → *Test Parallelism Policy* (two-axis model, per-suite table, default for new projects), cross-linked from `test_infra_db.md`. Found-not-fixed: `StoreAndForward.Tests/xunit.runner.json` is a no-op (pins the default `parallelizeTestCollections: true`) — harmless, tidy on next touch. | Row-82 fix session (2026-08-17) | Closed 2026-08-17. | — |
| 97 | ~~**A live DCL connection actor is never reconfigured in place — on ANY node**~~ **RESOLVED 2026-08-17 (merge `9dbda4a2`, agent commit `7c1268d8`) — the memo's §6 sketch shipped, plus two things the sketch missed.** `EnsureDclConnection` now has three outcomes off its one hash slot: unchanged → nothing; new → create; **changed → `ReconfigureConnectionCommand`** (arbitration/tombstones byte-for-byte untouched). The manager forwards to the owning actor (unknown name falls back to create); `DataConnectionActor` handles it in **all three** lifecycle states and never stashes it, via a new shared `SwapAdapter` helper now also used by both failover sites. The sketch's two gaps, found in implementation: **endpoint selection must be re-armed to Primary** (a connection sitting on its old *backup* would otherwise dial the new backup first), and the deliberate link drop must **suppress the unstable-disconnect accounting** (else the swap itself could trip a failover). Also a premise-sharpening: "reconfigure while mid-connect" is the *primary* case, not an edge — a wrong endpoint loops in `Connecting` forever and never reaches `Connected`. Fail-before: 5/7 new `DataConnectionReconfigureTests` + 3 SiteRuntime ensure-path tests fail on the reverted behavior. **Live rig probe (post-redeploy, round trip):** connection 1's config changed (operationTimeoutMs 15000→16000→back) fired exactly TWO in-place reconfigures on the active node — `Reconfiguring DataConnectionActor for OPC PLC Simulator in place` → `Reconnected successfully on Primary endpoint` in 45 ms → `Re-subscribing 2 tags` — with all rig-probe-1 tags Good after each, the standby silent (0 reconfigure lines, topology as designed), and the old dead-end "has a new configuration" Warning never appearing. Rig config restored to original. | Found by the 2026-08-17 rows-89/90 fix session; memo `docs/plans/2026-08-17-dcl-connection-arbitration-decision.md` (§9 postscript appended) | Closed 2026-08-17. | — |
| 98 | ~~**`EnforceSiteScopeForInstance` is fail-open for nonexistent instance ids**~~ **RESOLVED 2026-08-17 (merge `152ec85e`, agent commit `f33f25a1`) — premise corrected: the defect had FIVE instances, not two, and the worst three were worse than "harmless".** Beyond the two named helpers, the same `if (target != null) EnforceSiteScope(...)` shape sat **inline** in `HandleGetInstance` (:879), `HandleGetSite` (:1702) and `HandleGetDataConnection` (:1844) — and those returned `ManagementSuccess{JsonData=null}` for a nonexistent id vs `ManagementUnauthorized` for an out-of-scope one, i.e. a working **id-enumeration oracle across the whole fleet** for a site-scoped caller. All five now route through one shared `DenyUnresolvedTargetIfScoped`: a scoped caller's unresolvable target is DENIED, indistinguishable from out-of-scope (no existence leak); system-wide callers (Administrator / system-wide Deployer) are untouched. Also checked: `Instance.SiteId` is non-nullable `int`, so no secondary fail-open path through `EnforceSiteScope(null)`. Fail-before: 5 of the 16 new ManagementActorTests fail on the old code. | Found by the 2026-08-17 rows-37/86-88 fix session (Agent A survey) | Closed 2026-08-17. | — |
| 99 | ~~**CLI `audit query --status` client-side validation is case-sensitive**~~ **RESOLVED 2026-08-17 (merge `152ec85e`, agent commit `f33f25a1`) — premise corrected: "the server would accept it" was FALSE, and removing the client gate alone would have been a regression.** The server-side parser (`AuditQueryParamParsers.ParseEnumList`) is documented-and-tested **lax** — it silently drops unparseable values and collapses to null (filter unconstrained), i.e. exactly the row-46/88 silent-wrong-answer trap. Fix shipped both halves: **seven** `AcceptOnlyFromAmong` gates removed across `audit query`/`audit export` (`--channel`/`--kind`/`--status` on both, plus `export --format`, which rejected `CSV` the server lowercases anyway), valid sets moved to help text, and a new **strict** `ParseEnumListStrict` (throws naming the valid values; `LooksNumeric` guard rejects ordinals) wired into both ManagementService audit endpoints. The lax parser deliberately survives for the CentralUI drill-in/export query strings (bookmarked fragments must render, not fault — documented as non-reconcilable). Left alone with reasons in code: `--direction up|down` (no server parse to defer to), `--format json|table` (CLI-local), NotificationCommands' literals (unsurveyed). Four pre-existing tests were passing *vacuously* (no-credentials exit codes) and now pin the real contract; fail-before 21/26. **Found-and-fixed follow-on (orchestrator, `16056f8e`):** row 88's own `ParseDeploymentStatus` accepted ordinals (`--status 2` → InProgress) — same `LooksNumeric` guard applied + 3 tests. **Live rig probes (post-redeploy):** `audit query --status delivered` returns rows; `--status Nonsense` and `--status 3` both fail loud naming the 8 valid values; `deploy status --status 2` fails loud naming the 4 valid values. **Postscript 2026-08-19 (arch-review P6-3): the strict parser this row shipped was itself fail-open — `Enum.TryParse` splits and ORs comma-joined names inside ONE token for any enum (not just `[Flags]`), and dense 0-based `AuditStatus` lands composites on defined members:** `--status Forwarded,Attempted` = 1|2 = 3 = **Delivered** (silently wrong data), `Submitted,Parked` = 0|5 drops Submitted; the server passes raw query values whole (ASP.NET Core does not split commas), so the trap was live on `/api/audit/query` + export. Reopened and re-closed 2026-08-19 (agent commit `db174633`): any token containing `,` is rejected before `TryParse` with the same valid-values error; proof-of-failure verified on the two name+name composites. Nuance pinned by probe: the `"Failed,7"` name+ordinal variant does NOT reproduce — `TryParse` rejects mixed name/numeric comma lists outright — the test is kept as a regression guard. Repeated-param lists of valid names unaffected. | Found by the 2026-08-17 rows-37/86-88 fix session | Closed 2026-08-17; reopened + re-closed 2026-08-19 (arch-review P6-3). | — |
| 100 | ~~**`SiteScriptCompileCache.Clear()` is process-global — test classes wipe each other's entries**~~ **RESOLVED 2026-08-17 (merge `8a9a8d29`, agent commit `84b405ff`) — premise corrected: there were ZERO production callers of `Clear()`** (the only production entry point is `ScriptCompilationService.CompileCore` → `GetOrAdd`; `Clear()` existed solely for tests), so neither register option was taken as written — `Clear()` was **deleted outright** and the cache became an instance-scoped `sealed class` owned by `ScriptCompilationService`. The N1 "one compile per body per process" property now rests on `AddSiteRuntime` registering that service as a **DI singleton** — a lifetime that became load-bearing, so new `SiteRuntimeServiceRegistrationTests` pins it (a Scoped/Transient downgrade would reintroduce WP3.1's recompile storms with no behavioral test failing). The 37 test sites doing `new ScriptCompilationService(logger)` each silently get their own cache — isolation for the whole assembly, not just the row-91 classes. Row-91 workarounds RETIRED, not preserved: serialized collection removed from all three classes, exact `Hits`/`Count` assertions restored, guid key token gone, `ForeignEntrySlack` 512→8; new `SeparateCaches_DoNotShareEntriesOrCounters` pins the scoping. SiteRuntime.Tests green three consecutive full-parallel runs. | Found by the 2026-08-17 row-91 fix session | Closed 2026-08-17. | — |
| 101 | ~~**`ApplyingArtifacts_RemovedDataConnection_StopsLiveDclConnection` flaked once — suspected overlay-sweep read racing the artifact-apply write**~~ **RESOLVED 2026-08-17 (`0d3a90a9`) — hypothesis VERIFIED, and the production consequence was WORSE than the row guessed.** The race is real: the snapshot is read off the actor thread but compared against the overlay only when the piped message lands, so a read starting before an artifact apply's store writes can land after `HandleApplyArtifactDataConnectionsToDcl` populated the overlay — its stale (even empty) snapshot then "proves" the just-applied names disappeared. Overlay iteration order explains the exact fingerprint (`PlcA` removed first). Two corrections to the row's guesses: (1) it is NOT "self-healing at the next tick" — the removal evicts the hash slot the overlay's push is gated on, so the next fresh sweep clears the tombstone but re-creates nothing; the stopped connection stays dead **until an instance redeploy**; (2) the mid-write "momentarily not seeing the row" framing was the lesser half — the whole-snapshot staleness (read-before-write, land-after-apply) is the dominant window. Fix: `_definitionsWriteGeneration`, bumped on the actor thread by every artifact apply and captured at every snapshot-read dispatch (boot load + sweep); a result carrying a stale generation is discarded WHOLE — additions included, since a stale row pushed at a live connection would reconfigure it backward and flip-flop with the next fresh read. Deterministic repro fails before the fix (`StaleSweepSnapshot_RacingAnArtifactApply_DoesNotRemoveJustAppliedConnections`); `FreshEmptySweepSnapshot_StillRemovesDisappearedConnections` pins that row-90's legitimate disappearance still fires. SiteRuntime.Tests 647/647; rig redeployed, zero spurious removals/discards at boot, all rig-probe-1 tags Good. Memo §10 records the mechanism. **Postscript 2026-08-18 (merge `505f3569`, agent commit `f02e1166`): extended to the WHOLE snapshot.** The instance-config half had the same TOCTOU, and all three hazards were PROVEN real pre-fix: a stale sweep restarts an instance **backward** (`TryRestartStaleInstance` feeds the snapshot's old `ConfigJson` to `HandleDeploy` — a literal old-config redeploy; pipeline ownership only covers in-flight deploys and the old revision was never claimed), a read-pre-delete/disable **resurrects** a just-stopped instance (delete even clears the sweep claim), and the boot path skews `_deployedInstanceNames`. Fix: one `_storeWriteGeneration` over both tables, bumped at BOTH ends of every actor-thread config mutation (deploy persist, enable, disable, delete, rollback, artifact apply) — both ends because `ApplyDeployment`/`HandleDisable`/`HandleDelete` advance actor state BEFORE the off-thread write, so a completion-only bump (the row-101 original argument, which holds only for the artifact apply) leaves a window. Stale sweep discarded whole; stale boot load re-dispatched, capped at 5 consecutive (then applied with a Warning). CDC/`SiteReconciliationActor` writes deliberately never bump — discovering them is the sweep's purpose. 3 new fail-before tests + 2 fresh-generation guards pinning that row-72/78 legitimate repairs survive; SiteRuntime.Tests 652/652 x3; full suite green (one unrelated Communication.Tests gauge flake → row 102); rig probe: disable/enable/redeploy churn over multiple sweep ticks produced zero self-heal events on either site-a node, tags Good. Memo §11. | Observed by the 2026-08-17 row-100 fix session | Closed 2026-08-17; whole-snapshot extension 2026-08-18. | — |
| 102 | ~~**`SiteStreamSubscribersActiveGauge_GoesToOneOnConnect_AndBackToZeroOnCancel` is delta-hardened but still flakes — the baseline can't survive other contributors CLOSING**~~ **RESOLVED 2026-08-18 via the preferred option (row-96 pattern).** The survey found the gauge moves ONLY inside the real `SiteStreamGrpcServer.SubscribeInstance` handler, and exactly two test classes drive it: `SiteStreamGrpcServerTests` (the gauge test's own class) and `CleanupVerificationTests` — `GrpcOptionsWiringTests` instantiates the server but never opens a stream, and the client/aggregator suites subscribe against fakes, so none of those can perturb the counter. Both drivers now share the `[Collection("SiteStreamSubscriberGauge")]` xunit collection (comments on both classes + at the baseline poll explain why the relative read alone was insufficient), so no other contributor can close a stream between the baseline capture and the assertions. Verified 4/4 consecutive full-suite runs green (717/717), durations unchanged (22–25 s vs the prior 24 s) — the serialization is free. Original evidence: the test polls `baseline + 1`, but the process-global gauge's baseline is captured ONCE; capture while parallel classes' streams are up → they close → the gauge settles at this test's own 1 and `baseline+1` is unreachable forever — a fixed-baseline delta only survives contributors that hold constant or increase. | Observed 2026-08-18 (one full-suite failure, `Expected: 4, Actual: 1`; 3/3 green in isolation); fixed same day | Closed 2026-08-18. | — |
| 103 | ~~**Two lesser TTL-bounded circuit-teardown-vs-async-setup races remain in CentralUI**~~ **RESOLVED 2026-08-20 (Low-wave T12, agent commit `c8452136`).** Both instances closed with the Medium-wave T10 take-once discipline: `TransportImport.razor.cs` re-checks disposal after `TryLoadAsync`'s await and releases the import slot through exactly-once ownership (dispose-during-start frees the `BundleSession` slot immediately instead of stranding it for the 30-min TTL), and `Topology.razor` creates its timer under the same post-await re-check so a disposed page leaves no live timer (asserted event-style, not by sleep). Original note: (arch-review 2026-08-19 P6-6's optional tail, deliberately deferred by the fix). The permanent debug-stream leak was closed at `DebugView.razor` + `DebugStreamHub` (dispose-vs-connect take-once pattern, commit `11e623c2`), but two same-family instances with a *different shape* were left: (1) `TransportImport.razor.cs TryLoadAsync` (~:244) — a circuit torn down mid-load strands 1 of the 8 `MaxConcurrentImportSessions` slots until the 30-min `BundleSession` TTL reaps it (partially defeats row 43); (2) `Topology.razor` (~:201-214) — timer created before a disposal re-check. Both are TTL/lifetime-bounded (no permanent leak), which is why they missed the Medium bar. Fix is the same post-await `_disposed` re-check + take-once teardown, but routed through `BundleSession.ReleaseSession`/the wizard-step lifecycle rather than a session-stop call. | Arch-review 2026-08-19 P6-6 remediation (T10 explicitly scoped them out) | Open. | Next CentralUI hardening pass, or opportunistically when either file is next touched. |
| 104 | ~~**Existing site LocalDb files stay `auto_vacuum=NONE`; `PRAGMA incremental_vacuum` after a cap trim is a no-op on them**~~ **RESOLVED 2026-08-20 — recipe shipped and applied to the rig.** New `docker/vacuum-site-localdb.sh` (stop both nodes of a pair together → `PRAGMA auto_vacuum=INCREMENTAL; VACUUM` each volume with integrity checks before/after → start both; `DRY_RUN=1` reports) plus the topology-guide section *Site LocalDb files created before 2026-08-20 need a one-off offline VACUUM*. Run on all three rig pairs: all six files `auto_vacuum 0 → 2`, `integrity_check=ok`, ~11 % smaller (site-a 20.3 → 18.1 MB), site-a replication back `Connected=true`/backlog 0 with no watermark or schema change — the VACUUM was replication-invisible as predicted (ledger keys by declared PK; oplog `seq` is an `INTEGER PRIMARY KEY AUTOINCREMENT`). The same maintenance surfaced the unrelated row 106. Original: | Arch-review 2026-08-19 P4-6 (Low-wave T4, `47f03fe6`) | `auto_vacuum` is only honoured on a table-less database, and `LocalDbSchema.EnsureCreated` had always created LocalDb's bookkeeping tables before `SiteEventLogSchema.Apply` ran — so the pragma had never once taken effect on any real file. The fix (`SiteLocalDbSetup.EnsureAutoVacuumOnFreshFile`, run before `AddZbLocalDb`'s lazy singleton constructs `SqliteLocalDb`) only helps **fresh** files; an unconditional `VACUUM` against a live ~1 GB replicated file was explicitly rejected (long exclusive lock on the active node; the rewrite is CDC-invisible but blocks the sync pump). Every node on the docker rig today predates the fix, so freed `site_events` pages there are reused, not returned to the OS — the storage cap (row 81) still bounds the data volume, only the file size stops shrinking. | Operator-scheduled `VACUUM` in a maintenance window (both nodes of a pair stopped together), or when a site's file is next recreated. |
| 105 | ~~**Design constraint (not a defect): a LocalDb-replicated table must never be the parent of a row-mutating `ON DELETE` foreign key (`CASCADE`/`SET NULL`/`SET DEFAULT`) from another table**~~ **PINNED 2026-08-20.** Enforcement moved from the first replicated boot to the build: `SiteLocalDbReplicatedTableFkGuardTests` (Host.Tests) builds the real site schema via `OnReady` on an unreplicated config (so 0.3.3's own guard can't mask which FK is at fault), walks `SiteLocalDbSetup.ReplicatedTables` (now `internal`), asserts each exists and has no inbound row-mutating `ON DELETE` FK with the same `sqlite_master JOIN pragma_foreign_key_list` query LocalDb uses, and carries a negative control proving the checker finds a cascading child. The constraint itself stands — the row is kept as its record. Original: | Arch-review 2026-08-19 P4-7 → LocalDb 0.3.3 (scadaproj `122940c` + review follow-up `9663279`, Low-wave T5) | `LwwApplier` upserts with `INSERT OR REPLACE` under `foreign_keys=ON`, so the implicit delete fires the child action while CDC capture is suppressed — the child rows change on the applying node only and the pair diverges silently. 0.3.3 makes `RegisterReplicated` refuse such a parent at registration (clear exception naming the child FK), which is a BOOT failure for the site node, so this is enforced, not advisory. None of the ten `SiteLocalDbSetup.ReplicatedTables` has any FK today (zero `REFERENCES` on the live rig file). Model cross-table cleanup as explicit application deletes on each table (as the 3-table deploy cascade already does), never as SQLite FK actions. | Whenever a schema change adds a foreign key between site tables, or an eleventh table joins `ReplicatedTables`. |
| 106 | ~~**LocalDb: a converged pair re-streams a FULL snapshot on EVERY reconnect once the sender's oplog has been ack-pruned to empty and the receiver's watermark is 0**~~ **RESOLVED 2026-08-20 — LocalDb 0.3.4 (scadaproj `75c1897`), published and pinned.** `SnapshotStreamer.SendAsync` now stamps `as_of_seq = max(MAX(seq), last_acked_seq)`; pinned by `WipedPeer_SnapshotLiftsItsWatermark_SoTheNextReconnectSendsNoSecondSnapshot` (fails on 0.3.3 exactly as the rig did: watermark 0, expected 3). Live proof on the rig: the first restart on 0.3.4 sent one last full snapshot stamped `as_of_seq 149` (was 0) and node-a's `last_applied_remote_seq` went 0 → 149; a second restart of the pair exchanged **zero** snapshots, `Connected=true`, backlog 0. Original: | Found 2026-08-20 while verifying row 104 on the rig (site-a-a received the same 127 k-row full snapshot at both the 13:09 deploy restart and the 13:25 VACUUM restart) | Sender-side defect in `SnapshotStreamer.SendAsync`: `as_of_seq = MAX(seq)` of the oplog, which is **0** once the pair has converged and ack-pruning emptied it. The receiver's `SetLastAppliedRemoteSeqAsync` is MAX-guarded (no regression — good), but a peer whose watermark is genuinely 0 (site-a-a was rebuilt after the 2026-08-15 corruption; any wiped/re-imaged node is the same) can never be lifted by an `as_of_seq 0` snapshot, so the 0.1.2 empty-oplog horizon rule (`streamableFrom = last_acked_seq + 1`) sees `0 < 149` as a gap on every handshake and re-streams everything. Rig state: node-b `last_acked_seq=149`, oplog empty; node-a `last_applied_remote_seq=0`. Not a divergence (LWW absorbs the re-delivered rows) but a cost/availability bug — ~1 s and 127 k rows here, minutes and a full-file read on a 1 GB production pair, on every reconnect, until the sender happens to write a new row. Fix is sender-side and wire-compatible: stamp `as_of_seq = max(MAX(seq), last_acked_seq)` (a full snapshot taken now covers every delta ever pruned below `last_acked_seq`), so the receiver's MAX-guarded watermark lifts to 149 and the loop ends; add a pinning test (converged+pruned sender, zero-watermark receiver → exactly one snapshot across two handshakes). Ships as LocalDb 0.3.4. | Next LocalDb release (0.3.4) — small, well-understood, and worth doing before any production pair is rebuilt from its peer. |
| 107 | **Prod `CvdReactor.ProcessRecipeDownload` still addresses `Children["DelmiaRecv"]` after the 2026-06-27 `DelmiaRecv` → `DelmiaReceiver` composition rename** — inferred from the deployed script (`docs/api-specs/Delmia-Recipe-Download-API.md` §7 item 1): `ChildrenAccessor` concatenates the name into a path, an unknown path yields null, so every call answers `Receiver ready flag not set to true` regardless of the real flag. No call has hit prod since the rename, so this is inferred, not observed. The docker rig's stand-in (`docker/seed-delmia-sim.sh`) uses `Children["DelmiaReceiver"]` and the same handshake answers correctly, which is the working reference. | Found 2026-09-02 while capturing the as-deployed spec; QA copy unread (credentials) | Separate change with its own deployment record: `scadabridge template script update` on the `CvdReactor` template + redeploy Z28061/Z28062/Z28061Sim, then a live `WWNotifier` call. | Before the first production Delmia recipe download. |
| 108 | ~~**Inbound API keys are a PER-NODE SQLite store — a central failover invalidates every inbound client's key**~~ **RESOLVED 2026-09-02 — `ZB.MOM.WW.Auth` 0.3.0 (new `ZB.MOM.WW.Auth.ApiKeys.SqlServer` package) + ScadaBridge `ScadaBridge:InboundApi:ApiKeyStore:Provider=SqlServer`.** A central pair now shares one row set: both nodes' key stores, admin stores, and audit stores point at the same `zbauth` schema in the configuration database (connection string defaults to `ScadaBridge:Database:ConfigurationDb` — never a `${secret:}` reference, to avoid a circular dependency on the SQL-backed secrets store), so create/revoke/rotate/scope changes are cluster-wide immediately instead of per-node. Each node's existing SQLite file is imported once at boot (insert-where-absent, so both nodes importing is safe and the result is the union) and renamed `*.imported`. `Provider` defaults to `Sqlite` (unchanged behavior for single-node installs, MxAccessGateway, HistorianGateway); a clustered Central left on `Sqlite` now gets a boot-time `StartupValidator` Warning naming this design instead of silently keeping the gap. Both docker rigs (`docker/`, `docker-env2/`) run `Provider=SqlServer` on both centrals; `docker/sync-inbound-api-keys.sh` is deleted, superseded by the shared store. Rig proof 2026-09-03 (`docker/`, both centrals redeployed on `Provider=SqlServer`): both nodes imported their SQLite file at the same second with no PK race — central-a `1 key / 264 audit rows inserted`, central-b `0 / 26 inserted, 1 / 264 already present` (the union), files renamed `*.imported`; a key minted via `:9001` answered `200` via `:9002`, `--enabled false` via `:9001` made both nodes answer `401` and `:9002`'s own list showed `enabled=False`, re-enable via `:9002` restored `200` on `:9001`, delete via `:9002` made `:9001` answer `401`; `verify-delmia-notifier.sh` with a freshly minted key and NO file copy: A/B/E/F `YES`, C/G `NO`, active-central `docker kill` takeover **28 s**, rejoin as standby in 2 s. `docker-env2` (second pair, `ScadaBridgeConfig2`): both centrals booted, `zbauth` v1 created, empty files imported. QA (`vq03`/`vq04`) and prod (`vd03`) still run `Provider=Sqlite` — their move is a planned deployment record (`docs/deployment/deployment-records/2026-09-XX-zb-servers-inbound-key-store.md`), not yet applied. Design: `docs/plans/2026-09-02-inbound-api-key-shared-store-design.md`. Original: | `docs/api-specs/Delmia-Recipe-Download-API.md` §8; `docker/sync-inbound-api-keys.sh` header | Fix is an architecture decision, not a patch: move the store to the shared SQL Server (as Secrets 0.5.0 did for its central store) or replicate the file; either way revocation must become cluster-wide. Needs the `ZB.MOM.WW.Auth.ApiKeys` library (scadaproj), not just this repo. | Before any 2-node central goes live with inbound API clients; before the next arch-review wave. |
| 109 | **`ZB.MOM.WW.Secrets.Replicator.SqlServer`'s store migrator retries its DDL with a flat `100 * attempt` backoff and no jitter — a latent concurrent-boot flake, open in Secrets** *(Low)*. Porting that migrator's retry loop verbatim for the Auth 0.3.0 SQL Server API-key store (register row 108) reproduced an intermittent exhaustion — about once in sixty 8-way concurrent-boot trials the five flat-backoff retries ran out before a DDL/lock conflict cleared; adding jitter to the Auth port's backoff took it to 0 failures in 180 trials. Secrets' own migrator has no concurrent-migrate test, so the same defect is presumed still open there — it just hasn't been exercised at 8-way concurrency, and a ScadaBridge central pair is only ever 2-way, so this is unlikely to bite in practice. | Found 2026-09-02 while implementing `docs/plans/2026-09-02-inbound-api-key-shared-store.md` (Auth 0.3.0 `SqlServerApiKeyStoreMigrator`) | Add jitter to `SqlServerSecretsStoreMigrator`'s backoff in `~/Desktop/scadaproj/ZB.MOM.WW.Secrets` (mirror the Auth fix) and add a `Migrate_Concurrently_FromEightTasks_AllSucceed`-shaped live test to catch a regression. Lives entirely in scadaproj, not this repo. | Next time Secrets' SQL Server migrator is touched, or if a wider-than-2-way concurrent boot of a Secrets-backed cluster is ever planned. |
| 110 | **Flow Debugger follow-ups: the *Vertical trace* list toggle (mockup Direction B) and console capture on `ScriptRun`** *(Low)*. v1 ships the Direction A swimlane only; the mockup's Direction B — the same steps as a vertical, filterable list with the lane as a column — was kept as an alternate for long flows and screen readers, and the detail panel's *Console* tab is absent because script `Console`/log output is not captured per run (`ScriptRun` rows carry parameters, return value, error and timing, not stdout). Both are additive: a list view over the existing `FlowGraph`, and an extra `Extra.console` field on the `ScriptRun` row's `ScriptTraceExtra` (payload cap applies). | Design `docs/plans/2026-09-05-flow-debugger-design.md` §11; plan `2026-09-05-flow-debugger.md` Task 23 | Neither was asked for by an operator; the swimlane + Timing/Writes tabs cover the acceptance flow (Delmia recipe download, rig step H), and console capture needs a per-run writer seam in `ScriptRuntimeContext` plus a redaction decision. | First operator request for a list view or for script log lines in the step detail. |
| 111 | ~~**Flow Debugger outbound nodes: no Retry/Discard on a parked node, and no `?correlationId=` deep link from the node to the Site Calls / Notification report pages** *(Low)*~~ **RESOLVED 2026-09-06 (plan `docs/plans/2026-09-06-flow-debugger-followups.md`).** `FlowStep` now carries the row's `CorrelationId`; the detail panel deep-links to `/site-calls/report?correlationId=` / `/notifications/report?correlationId=` (both pages accept it through additive per-id query filters — `SiteCallQueryRequest.TrackedOperationIdFilter`, `NotificationOutboxQueryRequest.NotificationIdFilter` — pre-filter to the row and auto-open its detail), and a `Parked` node offers Retry / Discard in place through `IFlowDebugService.RetryOutboundAsync` / `DiscardOutboundAsync`, which reuse the report pages' relay, site-scope re-check and confirm wording, after which the page re-polls Replay until the node leaves Parked. Per-attempt request/response diff is still not shown (the lifecycle rows carry the error summary only). Found on the way: the prediction circuit deadlock (`4769e192`). | Plan 2026-09-06 "Out of scope"; `FlowStepDetail.razor` parked hint | The Site Calls page owns the retry semantics and its audit rows; duplicating the action on the debugger node would need the same Deployer gate + confirm flow. | First operator who reaches a parked node from the debugger and asks for the button. |
| 112 | **Flow Debugger: a SITE-side inline trace truncation is invisible to the operator** *(Low)*. `ScriptCallResult` / `RouteToCallResponse` carry `TraceEvents` but no `Truncated` flag, so the page's *event list truncated* badge reflects central's `FlowTraceCollector` only — a site run whose own collector dropped rows returns a silently short list. The new 1 MB byte budget (arch-review F1) makes site truncation reachable sooner than the old 500-event count cap did. The fix is additive: a trailing `bool Truncated = false` on both records, a new field on `RouteToCallResponseDto` (regen via `docker/regen-proto.sh sitecommand`), and `FlowDebugRunner` ORing it into `FlowDebugRunResult.Truncated`. | Arch-review remediation code review 2026-09-06 | Needs a proto regeneration plus a rig rebuild to prove end-to-end, and nothing is lost meanwhile: the site's audit rows are written in full, so a Replay of the same execution is never truncated. | The first Live run that shows fewer site steps than its Replay of the same execution. |
| 113 | **Flow Debugger: the method-entry site scope (F2) gates ROUTED site traffic only** *(Low)*. A site-scoped Deployer who Live-runs a method belonging to another plant is still refused at `RouteTarget.ResolveSiteAsync`, but the method's central-side side effects — its `Database.*` calls, `ExternalSystem` calls, notifications — execute before or regardless of that refusal. Only routing is scoped. | Arch-review remediation code review 2026-09-06 | F2's stated property ("no site traffic outside the operator's scope") holds as written. Going further needs a per-method site affinity, which does not exist in the model — ApiMethods are central objects with no site column — so there is nothing to scope a method against without inventing that concept. | A decision to give `ApiMethod` a site affinity, or an operator report of a scoped user reaching another plant's central-side effects. |
| 68 | ~~**`deploy.sh` run from a git worktree silently rehomes every node's bind-mount data dir**~~ **RESOLVED 2026-08-15 (guard added in the same close-out).** The compose files' bind mounts are relative (`./site-a-node-a/data`), so a deploy from a worktree checkout mounts the worktree's copies — the rig keeps running and looks healthy, but its SQLite stores split from the main checkout's and everything written there is orphaned by the next deploy from main. Bit the rows-46–53 close-out live: the seeding agent's `rig-probe-1` deploy persisted into worktree-homed dirs and "vanished" after the orchestrator's redeploy (no tombstone, no product bug — LWW/tombstone forensics ruled out deletion; the row sat in `.claude/worktrees/…/docker/site-a-node-a/data/site-localdb.db` the whole time). Both `docker/deploy.sh` and `docker-env2/deploy.sh` now refuse to run from a `.claude/worktrees/` path (override: `SCADABRIDGE_ALLOW_WORKTREE_DEPLOY=1`). | Found during the rows-46–53 close-out | Closed 2026-08-15. | — |

## Resolved (verified against the code 2026-07-10)
Rows removed from the Deferred table above once confirmed shipped. Kept here for traceability.

| # | Item | Resolution |
|---|------|-----------|
| 10 | Aggregated live alarm stream for Alarm Summary | Shipped 2026-07-10 (`docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.md`): a **transient, in-memory** per-site central live alarm cache (`ISiteAlarmLiveCache`/`SiteAlarmLiveCacheService` + per-site `SiteAlarmAggregatorActor`) fed by a new site-wide, alarm-only `SubscribeSite` gRPC stream (`SiteStreamManager.SubscribeSiteAlarms`), seed-then-stream with dedup + NodeA↔NodeB re-seed + periodic reconcile. Alarm Summary is now live-cache-driven (`AlarmSummaryService.BuildFromLiveAlarms`) with the 15s poll retained as fallback + `NotReporting` authority. Honors the `[PERM]` no-central-store rule — nothing persisted (no EF table/migration). Options on `CommunicationOptions` (eagerly validated) + two `ScadaBridgeTelemetry` signals. |
| 7 | SecuredWrite audit rows leave SourceNode NULL | Resolved (PLAN-07): `ManagementActor.EmitSecuredWriteAuditAsync` routes through `ICentralAuditWriter`, which stamps `SourceNode` (`central-a`/`central-b`) from `INodeIdentityProvider`. |
| 12 (CLI/API) | Native-alarm-source-override CSV import | Shipped 2026-07-10: shared `CsvLineSplitter`, `NativeAlarmSourceOverrideCsvParser`, bulk all-or-nothing `SetInstanceNativeAlarmSourceOverridesCommand` + ManagementActor handler (Deployer-gated), CLI `instance native-alarm-source import --file`, parser/CLI/handler tests. **UI upload affordance shipped 2026-08-01** — second `InputFile` on the `InstanceConfigure` Native Alarm Source Overrides card reusing the shared parser, mirroring the attribute importer's UX and the server's all-or-nothing merge semantics (`InstanceConfigureNativeAlarmCsvImportTests`); row 12 removed from the Deferred table. |
| 13 | WaitForAttribute quality-gated ("Good"-only) mode | Already implemented (Commons `WaitForAttribute.RequireGoodQuality`, enforced in `InstanceActor`, threaded through `ScriptRuntimeContext`, tested in `InstanceActorWaitForAttributeTests`). Stale "planned enhancement" doc line corrected 2026-07-10. |
| 14 | WaitForAttribute in Test-Run sandbox | Shipped 2026-07-10 (full fidelity): sandbox `Attributes.WaitAsync`/`WaitForAsync` (value-equality) route to the bound instance via `ISandboxInstanceGateway.WaitForAttributeAsync` → the existing `CommunicationService.RouteToWaitForAttributeAsync` cross-site route. Additive `RouteToWaitForAttributeRequest.RequireGoodQuality` (honored by the site handler) makes quality-gated waits route too. **Predicate-form waits stay unsupported** (an in-process lambda can't be routed) and throw a labelled `ScriptSandboxException`. Tests: sandbox accessor routing (CentralUI), site-handler quality-flag threading (SiteRuntime). |
| 15 | BrowseNext final-page signal not surfaced | Already surfaced (M7 browse work): `RealOpcUaClient` sets `Truncated=false`/`ContinuationToken=null` on the last page; `BrowseNodeResult` carries both; `TreeRow.razor` renders "Load more" only when a continuation token remains — no wasted BrowseNext. |
| 16 | StubOpcUaClient throws on browse | Already resolved: `StubOpcUaClient` supports browse + address-space search, covered by `StubOpcUaClientBrowseTests`/`StubOpcUaClientSearchTests`. |
| 18 | Folder drag-drop | Closed — **permanently deferred (`[PERM]`)** by the M9 decision (`docs/plans/2026-06-15-stillpending-completion-design.md:122`): menu-based reorder (T23) shipped instead, and the folder-hierarchy design fixed the reorganization UX as "right-click context menus only (no drag-drop)" (`2026-05-11-templates-folder-hierarchy-design.md:27`). Row removed from the Deferred table 2026-08-01 — nothing left to revisit. |
| 20 | Deployment EXPIRED-row purge | Already resolved (PLAN-04): `PendingDeploymentPurgeActor` central singleton (spawned in `AkkaHostedService`) ticks `IDeploymentManagerRepository.PurgeExpiredPendingDeploymentsAsync` every `CommunicationOptions.PendingDeploymentPurgeInterval` (default 1h), options-validated, tested. |
| 21 | SiteAuditBacklogReporter threshold consolidation | Shipped 2026-07-10: `SqliteAuditWriterOptions.BacklogPollIntervalSeconds` (default 30) now drives the reporter's poll cadence; explicit ctor override still wins (tests), non-positive falls back to the 30 s default. Cadence tests added; stale "hard-code / follow-up" class-doc corrected. |
| 22 | KPI history hourly rollups | Shipped 2026-07-10 (`docs/plans/2026-07-10-kpi-history-hourly-rollups-plan.md`, T1–T8): new `KpiRollupHourly` table (migration `20260710153953`) folded by a third recorder tick (`kpi-rollup`, `RollupInterval` default 1h) over a re-folded `RollupLookbackHours` window via an idempotent, failover-self-healing upsert; per-metric gauge-vs-rate aggregation (`KpiMetricAggregationCatalog`); a one-shot backfill of the retention window on start; raw-vs-rollup query routing by `RollupThresholdHours` (default 168h); longer rollup retention (`RollupRetentionDays` default 365 ≥ `RetentionDays`, dual daily purge); and 30 d / 90 d trend windows added to the four surfaces. Options + validator, docs (`Component-KpiHistory.md`), and tests shipped. |

## New deferrals from review 08 (this plan)
| Item | Rationale | Revisit trigger |
|------|-----------|-----------------|
| Communication → HealthMonitoring layering (ICentralHealthAggregator consumed by CentralCommunicationActor.cs:351) | Moving the interface + SiteHealthState to Commons ripples across 5 projects for a cosmetic inversion | Next breaking change to ICentralHealthAggregator |
| docs/components reference docs for ScriptAnalysis, KpiHistory, DelmiaNotifier | Reference docs are substantial (StyleGuide-conformant); README claim scoped instead (PLAN-08 Task 10) | Next doc-writing session touching those components |
| Test-coverage backfill: SiteCallAudit.Tests (31 tests/1.6k LOC), DeploymentManager.Tests | No defect identified; coverage partly lives in ManagementService/Host/Integration suites | First regression escaping either component |
| ~~Failover-timing measurement (the "~25s total failover" envelope)~~ **RESOLVED 2026-08-01** — split out of the combined row and closed. `tests/ZB.MOM.WW.ScadaBridge.PerformanceTests/Failover/FailoverTimingTests.cs` is no longer a skipped placeholder: it runs as a live `[Fact]` (`Category=Performance`) on the real two-node in-process rig (`TwoNodeClusterFixture`, production `BuildHocon`) at production timings — 2s heartbeat / 10s failure-detection threshold / 15s stable-after — hard-killing the younger node and timing the survivor's member REMOVAL with singleton continuity asserted on the oldest. Delivered by **PLAN-R2-01 Task 4** (`archreview/plans/PLAN-R2-01-cluster-host-failover.md:226`). The oldest-crash direction is covered behaviorally by `SbrFailoverTests.AutoDown_HardCrashOfOldestNode_*` and by `docker/failover-drill.sh`. | The 2026-07-08 "PLAN-01 rig landing" trigger had fired unnoticed (NF2); PLAN-R2-01 T4 wired the placeholder to the fixture rig rather than recording a blocker. | Closed. |
| ~~Broader perf envelope — **S&F drain rate + per-subscriber stream backpressure**~~ **RESOLVED 2026-08-15 — absorbed into the row-25 target-scale run, exactly as this row's trigger anticipated.** Both halves measured at full scale. **(a) S&F drain:** 20,000 messages buffered concurrently from 25 origin instances at **15,368 msg/s**, draining at **3,533 msg/s** to depth 0. Reported as two numbers because the naive single number is misleading — a deferred backlog sits for one full `DefaultRetryInterval` (measured 28.9 s) before *anything* drains, since `EnqueueAsync(attemptImmediateDelivery: false)` stamps `LastAttemptAt` and the due-predicate then holds the row for one interval (correct by design; `Notify.Send`'s `deferToSweep: true` leaves it null and skips the wait). Filed as informational finding **F2**. **(b) Slow-subscriber backpressure:** with 5 subscribers on the SAME instance and one reader stalled at 50 ms/event, the four healthy subscribers delivered **100.00% with zero drops** while the stalled one lost 197,028 of 200,000 events **entirely within its own bounded channel**. Isolation is structural and total — `Buffer(DropHead)` never backpressures the `BroadcastHub`, `Sink.ForEach`'s `Tell` never blocks, and `StreamRelayActor.WriteToChannel`'s `TryWrite` on a `DropOldest` channel never blocks — so a slow WAN link degrades only its own feed, visibly (per-stream eviction counter + `RecordSiteStreamEventDropped`). Finding **F3**. Caveat also recorded: an *unpaced* burst costs every subscriber, because the publish `Source.ActorRef` upstream of the hub is shared — that shared stage, not per-subscriber buffering, bounds burst absorption. | Closed by the row-25 run; no defect found in either half. | Closed 2026-08-15 — `docs/plans/2026-08-15-target-scale-load-test-results.md`. |

## Deferred — operational risk (from the initiative tracker, folded in 2026-07-12)
Two live items previously tracked ONLY in `archreview/plans/00-MASTER-TRACKER.md`'s registry are folded in here (NF5) so this register is the single tracking place. The tracker's narrative subsections remain as the historical evidence.

| # | Item | Where noted | Rationale for deferral | Revisit trigger |
|---|------|-------------|------------------------|-----------------|
| SBR | ~~**SBR oldest-crash total-outage gap**~~ **RESOLVED 2026-07-21 (owner decision — availability over partition-safety).** All clusters switched from `keep-oldest` to the `auto-down` downing strategy (Akka `AutoDowning`, `auto-down-unreachable-after` = 15s): a hard crash of EITHER node — active/oldest included — now fails over to the survivor in ~25s with no operator action. Accepted trade: a real network partition produces dual-active until an operator restarts one side. Decision record + evidence (live keep-oldest `DownReachable … including myself` log, Akka.NET 1.5.62 `KeepOldest.OldestDecision` source, rejected alternatives incl. the static-quorum-1 `DownAll` trap): `docs/plans/2026-07-21-auto-down-availability-decision.md`. | `archreview/plans/00-MASTER-TRACKER.md:194` + auto-memory `sbr-keep-oldest-2node-active-crash-gap` (both now historical) | — | Closed. Residual: seed-node boot-alone constraint (unchanged, documented in `Component-ClusterInfrastructure.md`); dual-active recovery is operator-driven. |
| vd03 | **`deploy/wonder-app-vd03/` overlay edits unapplied** — `appsettings.Central.json` needs `AllowSingleNodeCluster: true` + phantom-seed removal + `NodeName: central-a`; `install.ps1` needs `sc.exe failure` recovery actions. The `deploy/wonder-app-vd03/` artifact directory is intentionally untracked (production config out of source control), so the repo cannot ship the fix. | `archreview/plans/00-MASTER-TRACKER.md:198` (PLAN-01 T16/T20/T23) | Needs on-host access; without `NodeName` that deployment's audit rows stamp NULL `SourceNode` — **partially mitigated once PLAN-R2-08 Task 7 lands: the host now FAILS AT BOOT with a key-naming error instead of silently NULLing, so applying the overlay becomes mandatory at the next upgrade.** Owner: whoever maintains the host (user). | Next wonder-app-vd03 deployment/upgrade — **the Task 7 validator makes this row unskippable then.** |