ScadaBridge/docs/plans/2026-07-08-deferred-work-register.md

# Deferred-Work Register (established 2026-07-08, from architecture review 08)

Single tracked list of consciously-deferred work. Rules: every deferral gets a row
(rationale + revisit trigger); fix-now items reference the archreview plan that owns
them and are removed from this table when that plan's task lands.

## Fix-now (owned by archreview plans)

All 7 fix-now items landed via PLAN-04/05/06/07/08 (verified in review 08 round 2, 2026-07-12) — rows removed per the rule above; see the round-2 report §1 for the per-item evidence.

## Deferred (with rationale + revisit trigger)
| # | Item | Where noted | Rationale for deferral | Revisit trigger |
|---|------|-------------|------------------------|-----------------|
| 8 | Hash-chain tamper evidence (T1); CLI verify-chain is a no-op stub | audit-log roadmap :12 | v1.x by locked decision; append-only DB roles are the control | Compliance requirement for cryptographic tamper evidence |
| 9 | Parquet audit archival (T2); endpoint returns 501 | AuditEndpoints.cs:204 | v1.x; 501 + CLI messaging are honest | AuditLog partition volume nears retention ceiling |
| 11 | Central-persisted OPC UA cert-trust audit | m7 follow-ups | Broadcast-to-both-nodes covers HA | Governance/audit requirement for trust decisions |
| 17 | Unified notifications+site-calls outbox page | stillpending :118 | Explicit M9 decision to keep two pages | Operator confusion reports |
| 19 | Bundle signing / cluster-to-cluster pull / differential bundles | transport-design :402 | v1 manifest hash + AES-GCM held sufficient | Non-repudiation requirement across orgs |
| 23 | Live LDAP group-membership re-query for an active session | `docs/requirements/Component-Security.md` :61-69 (+ :78-79) | Blocked on an external package. The mid-session refresh re-maps the **stored** groups against the central DB with **no LDAP call**, so a directory group-membership change lands only at next login. A live re-query needs a passwordless service-account group-search method on the shared `ZB.MOM.WW.Auth.Ldap` library — an external NuGet `PackageReference` (`src/ZB.MOM.WW.ScadaBridge.Security/…csproj:23`) exposing only `AuthenticateAsync(username, password, ct)`. Central role-mapping/scope changes still apply within ~15 min (`RoleRefreshThresholdMinutes`). | `ZB.MOM.WW.Auth.Ldap` gains a standalone group-search API, or a requirement that a directory-side group revocation take effect mid-session rather than at next login |
| 24 | M8 large-bundle performance hardening | `docs/plans/2026-06-15-stillpending-completion-design.md:106` — "Small follow-ups logged (not blocking): … large-bundle/perf hardening" | Logged as a non-blocking follow-up when M8 shipped and never given an artifact: **no plan, no task entry, no perf/load test exists** (`tests/…Transport.Tests/Import/BundleImporterLoadTests.cs` is a `LoadAsync` unit suite despite the name). No measured problem; the only sizing controls in place are the 5-minute CLI transport timeout, `LineDiffer`'s `MaxInputLines`=4000 summary-only cap, and `MaxConcurrentImportSessions`=8. | First real bundle that times out, exhausts memory, or makes the import wizard's diff step unusable |
| 25 | ~~**Phase-8 WP-4 target-scale load test** (10 sites x 500 instances x 75 tags = 37,500 subscriptions/site, 375,000 total)~~ **RESOLVED 2026-08-15 — run, measured, evidenced.** The 107-byte stub is replaced by a real full-scale run: **10 x 500 x 75 = 375,000 live tag subscriptions**, 37,518 tag updates/s achieved against 37,500 nominal (100.0%), **45,021,375 updates offered** over a 20-minute steady-state window on an M4 Pro / 14-core / 48 GB host with the 8-node `docker/` rig still running. Headline numbers: tag update latency **P50 0.88 ms / P99 4.57 ms / max 37.41 ms** end-to-end (DCL boundary to stream subscriber, the emit instant carried verbatim through production code); **0 events dropped** at 100 live subscribers; health report collect+ingest **P99 0.31 ms** with 10/10 sites tracked; debug view snapshot under load **P99 2.19 ms, 0 timeouts**; 500 instances deployed to a site in **2.6 s**; CPU **2.9% of the box**; working-set slope **+8.83 MB/min**. **11 clean passes, 1 pass with a caveat (F1: 20 min with zero gen-2 collections cannot fully settle the leak question — the 1-hour run would), 0 failures.** Harness: `tests/ZB.MOM.WW.ScadaBridge.LoadHarness/` (standalone Exe — the `Category=Performance` trait enables a filter but does NOT exclude by default, so a 20-minute test could not live in `PerformanceTests`) plus a CI-scale smoke `[Fact]` (`PerformanceTests/TargetScale/TargetScaleHarnessSmokeTests.cs`). Real: per-site ActorSystem + LocalDb, the real DCL via the documented `DataConnectionFactory.RegisterAdapter` seam, real `InstanceActor`s, real `SiteStreamManager`, real `StreamRelayActor` + production-capacity bounded channel, real S&F, real health collector/aggregator; only the socket hops are stood in for. **Results: `docs/plans/2026-08-15-target-scale-load-test-results.md`; design + deviations + thresholds: `docs/plans/2026-08-15-target-scale-load-test-design.md`.** | as before | Closed. Residual scope: `[xc-3]` site-event-log retention volume, `[xc-4]` audit-vs-central degradation, `[xc-5]` large-template flattening and `[xc-8]` UI responsiveness are NOT covered by this harness (central-cluster/browser concerns) and remain open WP-4 scope — see results doc SS6. | Closed 2026-08-15. |
| 26 | Ipsen MES MoveIn tail: leak-test (`-LT`) receivers + routing, PLC-output-flag writes, `Z28062` BTDB data completeness | `docs/plans/2026-06-16-ipsen-mes-movein.md:409` ("Out of scope (future)"); design `2026-06-16-ipsen-mes-movein-design.md:58-60, :196-198` | Customer-site scope, not a platform gap. `-LT` routing needs an MES-receiver child + Galaxy reference that do not exist on the reactor template (any `-LT`/unknown suffix returns `WasSuccessful=false` with an "unsupported side/target" message by decision); `MoveInComplete`/`Successful`/`ErrorText` are **PLC-owned** by locked decision, so ScadaBridge deliberately does not write them; `Z28062` completeness is an operational data fix, not code. Note the separate alarm-status path already handles the suffix — `_LT` is stripped before side-scoping (`2026-06-30-mes-alarm-status-api.md:158`). | Ipsen creates the leak-test receiver + Galaxy reference, or asks ScadaBridge to own the PLC-output flags — otherwise a **candidate won't-do** (`[PERM]`) at the next Ipsen scope review |
| 27 | External-system per-system retry config (`MaxRetries`/`RetryDelay`) never reaches sites, and has no CLI/management surface | Found live 2026-08-01 (rig session, #11 gRPC live checks) | Two stacked gaps: (a) `ExternalSystemArtifact` (Commons) carries `TimeoutSeconds` but NOT `MaxRetries`/`RetryDelay`, and the site `external_systems` table has no such columns — so a centrally-configured retry policy is silently ignored on sites; every cached call buffers with the S&F default (`DefaultMaxRetries` 50 × `DefaultRetryInterval` 30s ≈ 25 min to park). (b) `Create/UpdateExternalSystemCommand` don't expose the fields either — the only way to set them today is a direct DB edit of `ExternalSystemDefinitions`. Transport bundles DO carry them (arch-review 05 "ES retry config"), which masks the gap in export/import round-trips. Fix is additive: extend the artifact + site schema + apply path, and add `--max-retries`/`--retry-delay` to the CLI. | First operator who tunes retry policy on an external system and expects site cached calls to honor it |
| 28 | Health-dashboard "Trigger failover" confirm dialog's confirm button is labeled **"Delete"** | Found live 2026-08-01 (rig session, #11 TriggerSiteFailover check) | The DialogService confirmation host's default destructive-action label leaks through — the dialog copy is correct but the red confirm button says "Delete" for a failover. One-line fix: pass an explicit confirm label ("Fail over") at the Health-dashboard call site (and audit other confirm-dialog call sites for the same default). | Next Central UI session |
| 29 | **1-hour sustained run to settle the memory-slope ambiguity (load-test finding F1)** | `docs/plans/2026-08-15-target-scale-load-test-results.md` F1 (Low) | The row-25 20-minute run PASSED the working-set threshold (+8.83 MB/min < 20) but recorded **zero gen-2 collections across 45M events**, so the +19.52 MB/min managed-heap drift on an uncompacted gen-2 is ambiguous — the heap demonstrably sawtooths, but 20 min cannot distinguish "gen-2 not yet collected" from a slow leak. Not a defect finding; an evidence gap. The harness exists (`tests/ZB.MOM.WW.ScadaBridge.LoadHarness/`) and the full 1-hour protocol is one documented command (results doc §7) — deferred only because the measured window was scaled for practicality. | Before production go-live at target scale, or first field report of unexplained site-node memory growth |
| 30 | **WP-4 residual scope: `[xc-3]` site-event-log retention volume, `[xc-4]` audit-load-vs-central degradation, `[xc-5]` large-template flattening time, `[xc-8]` UI responsiveness under load** | `docs/plans/2026-08-15-target-scale-load-test-results.md` §6 + row 25 closure note | The row-25 harness is a site-runtime rig; these four WP-4 sub-criteria are central-cluster/browser concerns it deliberately does not cover. Recorded so the row-25 closure never reads as "WP-4 fully done" — phase-8 WP-4 is closed for the site-runtime data path only. | Before declaring phase-8 WP-4 complete, or production go-live at target scale |
| 31 | **Unpaced-burst absorption is bounded by the shared publish stage, not per-subscriber buffers (load-test finding F3 caveat)** | `docs/plans/2026-08-15-target-scale-load-test-results.md` F3 | Measurement-backed design note, no defect: slow-subscriber isolation is structural and total (a stalled reader loses events only in its own bounded `DropOldest` channel; healthy subscribers delivered 100.00%), but the publish `Source.ActorRef` **upstream** of the `BroadcastHub` is shared, so an unpaced burst degrades every subscriber equally — that stage, not per-subscriber buffering, is the burst ceiling. | First field report of event loss on *healthy* subscribers during a burst, or any redesign of the site stream publish stage |
| 32 | **`QueueDepthGaugeTests` full-suite flake — shared static gauge (load-test finding F4)** | `tests/ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests/QueueDepthGaugeTests.cs`; surfaced by the row-25 verification runs (`docs/plans/2026-08-15-target-scale-load-test-results.md` F4) | Pre-existing, test-only: fails when the full suite runs concurrently, passes in isolation — the gauge under test is process-global static state shared across test classes. Zero `src/` involvement; needs test isolation (per-test meter/collector instance), not a product fix. | Next test-infra session, or the first time it fails a CI/full-suite run that matters |
| 33 | **`LocalDbOplogBacklog` has no threshold, KPI history, or dashboard treatment** | Found by R7 (`docs/plans/2026-08-15-residual-remediation-plan.md`) while documenting the purge burst | `SiteAuditBacklog` drives a configurable threshold warning on the site tile, but the LocalDb backlog is reported and Prometheus-exported with nothing consuming it for alerting or trending — it is not among the metrics `SiteHealthKpiSampleSource` samples into KPI history, so the purge-burst spike R7 documents cannot be reviewed after the fact; an operator must be watching live. Candidate: add it to the KPI sample source and/or a sustained-backlog threshold badge. Pairs naturally with the R1 per-table-snapshot work, which changes snapshot/backlog behavior. | Next health-dashboard session, or first operator question about a backlog spike they didn't see live |
| 34 | **`site_events` purge burst is unbounded in the oplog and unmeasured at scale** | Found by R7 | Retention deletes are sliced at 1000 rows/statement, but the whole expired set is deleted in one tick, so a replicated site's oplog gains one row per deleted event with no pacing — a first purge after enabling replication (or after a long low-purge period) queues the entire batch at once. The wire path is bounded (`MaxBatchBytes`), but the oplog write itself is not paced, and no measurement exists for batch size or drain time at target scale. Candidate: measure on the rig or in the load harness; consider a per-tick delete ceiling if drain time is material. | Before enabling replication on a site with a large existing `site_events` backlog, or the next load-harness session |
| 35 | **`DeploymentRecords` index does not cover the deployment-list sort key** | Found by R3 (`35ce1413`) | The table has a single-column `IX DeployedAt`; the new list query orders `DeployedAt DESC, Id DESC` and joins `Instance`, so SQL Server sorts per page. A composite `(DeployedAt DESC, Id DESC)` index would serve the ordering directly — deliberately not done in R3 because it needs an EF migration and the terminal-record purge bounds the table. *(Low)* | Deployment list measurably slow at a real fleet's retention depth |
| 36 | **Deployments free-text search is unindexable by construction** | Found by R3 | Four `LIKE '%term%'` predicates (instance name, deployment id, revision hash, user) — leading wildcards preclude index seeks, so a search is a scan of the join. Acceptable at the purge-bounded row count. Fix if hot: anchor the id/hash legs as prefix matches, or full-text. *(Low)* | Search latency complaints on the Deployments page |
| 37 | **Two paged reads of `DeploymentRecords` with different contracts** | Found by R3 | `QueryDeploymentSummariesAsync` (CLI/ManagementActor: offset, no total, no instance join, instance-id-list scoping) and R3's `QueryDeploymentListPageAsync` (UI: total + instance join + site-id scoping) duplicate query logic. Consolidating on one contract — giving the CLI a total, both scoping by site — removes the duplication. Genuinely different contracts today, so not urgent. *(Low)* | Next time either query's shape changes |
| 38 | **OtOpcUa: `RoslynVirtualTagEvaluatorTests` racing-clear test is load-fragile** (OtOpcUa repo, not this one) | Found by R4 during the full-suite bump verification | `Evaluate_racing_ClearCompiledScripts_never_fails_with_disposed` fails under full-solution parallel test load, passes 3/3 isolated: its design assumes at most one `ClearCompiledScripts` lands inside one evaluation (so the product's one-shot retry suffices); under contention two can. Same timing-assumption family as the R5 sweep, but lives in OtOpcUa. Test-side fix: bound clear cadence relative to observed evaluation latency, or assert on retry-exhaustion instead of zero failures. | Next OtOpcUa test session, or when it fails a run that matters |
| 39 | **OtOpcUa: late-enabled replication cannot baseline pre-existing rows** (OtOpcUa repo, not this one) | Found by R4; documented in OtOpcUa's CLAUDE.md | `LocalDbSetup.OnReady` registers all four tables unconditionally on every driver node, so it can never pass `baselineExistingRows: true` (only correct when gated on replication being configured). Turning replication ON for a long-running default-OFF node converges on subsequent writes only — existing rows are never baselined. ScadaBridge closed this class with conditional registration + boot-time `DeregisterReplicated` self-heal (WP1.3/WP3.3); porting it is a behavior change beyond a pin bump (registered-set pinned by tests, deregistration must be pair-symmetric). Low current impact: site-a was enabled from fresh. | Before enabling replication on an established OtOpcUa node |
| 40 | **Wall-clock sleeps used as synchronization in tests (absence-assertion class)** | Found by the R5 sweep (`9fb52153`) — a different class from the one swept, deliberately left unfixed | Tests that sleep, then assert an *absence*: `ScriptDeadlineAtEnqueueTests.cs:105,128` (two `Thread.Sleep(2s)`, with `Assert.Equal(0, scheduler.DetachedThreadCount)` depending on the sleep out-racing the watchdog — worst offender), `DeploymentWarmThenGateTests.cs:117`, `SiteAlarmLiveCacheServiceTests.cs:195,216`, `DataConnectionActorTests.cs:435` (`Task.Delay(200)` to "settle" after an `AwaitCondition`). These fail toward false-pass rather than flake, so they don't trip CI — but each is a timing pin that weakens what the test proves. Fix pattern: replace the sleep with an explicit observable edge (or assert on the positive event with a bounded wait plus a negative control). | Next test-infra session, or the first time one of these areas regresses without its test catching it |
| 41 | **LocalDb (scadaproj) sync-session tests flake under parallel load** (scadaproj repo, not this one) | Found by R1; confirmed pre-existing at base `f627cee` (1 failure in 3 full runs) | `SyncSessionTests.Handshake_DigestMismatch_FailsClosed` and `BatchBudgetTests.Pump_RowLargerThanTheBudget_IsSentAloneRatherThanStallingTheStream` intermittently time out. Root cause looks structural, not a timing pin: on a handshake fault, `RunAsync`'s catch cancels the writer loop, which can kill the session's OWN Handshake before it transmits — the healthy peer then waits out its full 15 s cap for a message that never comes. Product-adjacent (the same early-cancel could delay a real peer's fail-closed verdict), so worth a look beyond the tests. | Next LocalDb session |
| 42 | **`build/push.sh`-class publish bug across the family repos** | Found by R1 the hard way (incident: 25 unrelated packages briefly published to the feed, deleted + verified same session; script fixed in scadaproj `0038d46`) | The bug shape: `dotnet pack -c Release -o ./artifacts` with no project argument and no `cd`, so the script packs the CALLER'S CWD. Fixed in LocalDb's script (cd to own repo root, name the solution, absolute output path). Peer repos' publish scripts (MxAccessGateway, HistorianGateway, OtOpcUa, others in scadaproj) should be checked for the same shape — the recovery only worked because the token happened to carry delete rights. | Before anyone runs a family publish script from outside its repo, or next umbrella-repo session |
| 43 | **Central UI import wizard leaks a `BundleSession` slot on every abandoned import — imports lock out for 30 min** *(High — availability + secret lifetime)* | Found live by R6 (reproduced: 3 back-to-back suite runs → 6 of 8 slots held, rotating pair of `TransportImportTests` failures; central-b restart cleared it) | `TransportImport.razor.cs`: `ResetSessionState()` nulls the local field and `Dispose()` deletes only the staged temp file — neither calls `IBundleSessionStore.Remove(sessionId)`; only `BundleImporter.ApplyAsync` removes sessions. An operator who unlocks a bundle then navigates away burns one of `MaxConcurrentImportSessions` (8) for the full `BundleSessionTtlMinutes` (30); eight abandonments lock out ALL imports on the active node. The leaked session also holds the **decrypted** bundle content that T-007 deliberately zeroes on the Apply paths — so this is a secret-lifetime hole too. The class doc's "an abandoned wizard does not leak" claim is true of the plaintext file only. Fix: remove the session (and zero its content) from `Dispose`/`ResetSessionState`/back-navigation. | Fix-now candidate — next session touching Transport or Central UI |
| 44 | **`BundleUnlockRateLimiter` is dead code — the documented per-IP unlock throttle is not enforced** *(Medium — security)* | Found by R6 | Registered, injected into `BundleImporter`, unit-tested, documented on `TransportOptions.MaxUnlockAttemptsPerIpPerHour = 10`, pinned in `appsettings.Central.json` — but `_unlockRateLimiter` is never invoked anywhere in `src/`. A brute-force passphrase attacker is bounded only by the per-bundle 3-strike lockout, not the advertised 10/hour/IP cap. Fix: call it on the unlock path (or delete it and the config surface honestly). | Fix-now candidate — next Transport/security session |
| 45 | **Notification Outbox dispatcher shares one `DbContext` across up to 4 parallel deliveries** *(Medium — correctness under load)* | Found by R6; observed continuously on rig central-b: `InvalidOperationException: A second operation was started on this context instance` on dispatch | `NotificationOutboxActor.ResolveAdapters()` caches adapter instances in an actor-lifetime `IServiceScope` while the Email/Sms adapters are `AddScoped` and hold a scoped `INotificationRepository` (`ScadaBridgeDbContext`); `MaxParallelDeliveries` (4) then shares one context across concurrent deliveries. The per-delivery scope added in `DeliverGatedAsync` for exactly this hazard covers only `INotificationOutboxRepository` — the adapter escapes it. Effect today: failed attempts + retries + error noise; sustained contention could burn a row to `Parked`. Fix: resolve adapters inside the per-delivery scope. | Fix-now candidate — next Notification Outbox session, or first `Parked` row attributed to it |
| 46 | **CLI `audit-config query` returns `{}` for every filter combination** *(Low)* | Found by R6 against the rig; underlying `AuditLogEntries` rows verified present | Possibly a CLI output-shape/serialization issue; not investigated (out of R6 scope). | Next CLI session |

## Resolved (verified against the code 2026-07-10)
Rows removed from the Deferred table above once confirmed shipped. Kept here for traceability.

| # | Item | Resolution |
|---|------|-----------|
| 10 | Aggregated live alarm stream for Alarm Summary | Shipped 2026-07-10 (`docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.md`): a **transient, in-memory** per-site central live alarm cache (`ISiteAlarmLiveCache`/`SiteAlarmLiveCacheService` + per-site `SiteAlarmAggregatorActor`) fed by a new site-wide, alarm-only `SubscribeSite` gRPC stream (`SiteStreamManager.SubscribeSiteAlarms`), seed-then-stream with dedup + NodeA↔NodeB re-seed + periodic reconcile. Alarm Summary is now live-cache-driven (`AlarmSummaryService.BuildFromLiveAlarms`) with the 15s poll retained as fallback + `NotReporting` authority. Honors the `[PERM]` no-central-store rule — nothing persisted (no EF table/migration). Options on `CommunicationOptions` (eagerly validated) + two `ScadaBridgeTelemetry` signals. |
| 7 | SecuredWrite audit rows leave SourceNode NULL | Resolved (PLAN-07): `ManagementActor.EmitSecuredWriteAuditAsync` routes through `ICentralAuditWriter`, which stamps `SourceNode` (`central-a`/`central-b`) from `INodeIdentityProvider`. |
| 12 (CLI/API) | Native-alarm-source-override CSV import | Shipped 2026-07-10: shared `CsvLineSplitter`, `NativeAlarmSourceOverrideCsvParser`, bulk all-or-nothing `SetInstanceNativeAlarmSourceOverridesCommand` + ManagementActor handler (Deployer-gated), CLI `instance native-alarm-source import --file`, parser/CLI/handler tests. **UI upload affordance shipped 2026-08-01** — second `InputFile` on the `InstanceConfigure` Native Alarm Source Overrides card reusing the shared parser, mirroring the attribute importer's UX and the server's all-or-nothing merge semantics (`InstanceConfigureNativeAlarmCsvImportTests`); row 12 removed from the Deferred table. |
| 13 | WaitForAttribute quality-gated ("Good"-only) mode | Already implemented (Commons `WaitForAttribute.RequireGoodQuality`, enforced in `InstanceActor`, threaded through `ScriptRuntimeContext`, tested in `InstanceActorWaitForAttributeTests`). Stale "planned enhancement" doc line corrected 2026-07-10. |
| 14 | WaitForAttribute in Test-Run sandbox | Shipped 2026-07-10 (full fidelity): sandbox `Attributes.WaitAsync`/`WaitForAsync` (value-equality) route to the bound instance via `ISandboxInstanceGateway.WaitForAttributeAsync` → the existing `CommunicationService.RouteToWaitForAttributeAsync` cross-site route. Additive `RouteToWaitForAttributeRequest.RequireGoodQuality` (honored by the site handler) makes quality-gated waits route too. **Predicate-form waits stay unsupported** (an in-process lambda can't be routed) and throw a labelled `ScriptSandboxException`. Tests: sandbox accessor routing (CentralUI), site-handler quality-flag threading (SiteRuntime). |
| 15 | BrowseNext final-page signal not surfaced | Already surfaced (M7 browse work): `RealOpcUaClient` sets `Truncated=false`/`ContinuationToken=null` on the last page; `BrowseNodeResult` carries both; `TreeRow.razor` renders "Load more" only when a continuation token remains — no wasted BrowseNext. |
| 16 | StubOpcUaClient throws on browse | Already resolved: `StubOpcUaClient` supports browse + address-space search, covered by `StubOpcUaClientBrowseTests`/`StubOpcUaClientSearchTests`. |
| 18 | Folder drag-drop | Closed — **permanently deferred (`[PERM]`)** by the M9 decision (`docs/plans/2026-06-15-stillpending-completion-design.md:122`): menu-based reorder (T23) shipped instead, and the folder-hierarchy design fixed the reorganization UX as "right-click context menus only (no drag-drop)" (`2026-05-11-templates-folder-hierarchy-design.md:27`). Row removed from the Deferred table 2026-08-01 — nothing left to revisit. |
| 20 | Deployment EXPIRED-row purge | Already resolved (PLAN-04): `PendingDeploymentPurgeActor` central singleton (spawned in `AkkaHostedService`) ticks `IDeploymentManagerRepository.PurgeExpiredPendingDeploymentsAsync` every `CommunicationOptions.PendingDeploymentPurgeInterval` (default 1h), options-validated, tested. |
| 21 | SiteAuditBacklogReporter threshold consolidation | Shipped 2026-07-10: `SqliteAuditWriterOptions.BacklogPollIntervalSeconds` (default 30) now drives the reporter's poll cadence; explicit ctor override still wins (tests), non-positive falls back to the 30 s default. Cadence tests added; stale "hard-code / follow-up" class-doc corrected. |
| 22 | KPI history hourly rollups | Shipped 2026-07-10 (`docs/plans/2026-07-10-kpi-history-hourly-rollups-plan.md`, T1–T8): new `KpiRollupHourly` table (migration `20260710153953`) folded by a third recorder tick (`kpi-rollup`, `RollupInterval` default 1h) over a re-folded `RollupLookbackHours` window via an idempotent, failover-self-healing upsert; per-metric gauge-vs-rate aggregation (`KpiMetricAggregationCatalog`); a one-shot backfill of the retention window on start; raw-vs-rollup query routing by `RollupThresholdHours` (default 168h); longer rollup retention (`RollupRetentionDays` default 365 ≥ `RetentionDays`, dual daily purge); and 30 d / 90 d trend windows added to the four surfaces. Options + validator, docs (`Component-KpiHistory.md`), and tests shipped. |

## New deferrals from review 08 (this plan)
| Item | Rationale | Revisit trigger |
|------|-----------|-----------------|
| Communication → HealthMonitoring layering (ICentralHealthAggregator consumed by CentralCommunicationActor.cs:351) | Moving the interface + SiteHealthState to Commons ripples across 5 projects for a cosmetic inversion | Next breaking change to ICentralHealthAggregator |
| docs/components reference docs for ScriptAnalysis, KpiHistory, DelmiaNotifier | Reference docs are substantial (StyleGuide-conformant); README claim scoped instead (PLAN-08 Task 10) | Next doc-writing session touching those components |
| Test-coverage backfill: SiteCallAudit.Tests (31 tests/1.6k LOC), DeploymentManager.Tests | No defect identified; coverage partly lives in ManagementService/Host/Integration suites | First regression escaping either component |
| ~~Failover-timing measurement (the "~25s total failover" envelope)~~ **RESOLVED 2026-08-01** — split out of the combined row and closed. `tests/ZB.MOM.WW.ScadaBridge.PerformanceTests/Failover/FailoverTimingTests.cs` is no longer a skipped placeholder: it runs as a live `[Fact]` (`Category=Performance`) on the real two-node in-process rig (`TwoNodeClusterFixture`, production `BuildHocon`) at production timings — 2s heartbeat / 10s failure-detection threshold / 15s stable-after — hard-killing the younger node and timing the survivor's member REMOVAL with singleton continuity asserted on the oldest. Delivered by **PLAN-R2-01 Task 4** (`archreview/plans/PLAN-R2-01-cluster-host-failover.md:226`). The oldest-crash direction is covered behaviorally by `SbrFailoverTests.AutoDown_HardCrashOfOldestNode_*` and by `docker/failover-drill.sh`. | The 2026-07-08 "PLAN-01 rig landing" trigger had fired unnoticed (NF2); PLAN-R2-01 T4 wired the placeholder to the fixture rig rather than recording a blocker. | Closed. |
| ~~Broader perf envelope — **S&F drain rate + per-subscriber stream backpressure**~~ **RESOLVED 2026-08-15 — absorbed into the row-25 target-scale run, exactly as this row's trigger anticipated.** Both halves measured at full scale. **(a) S&F drain:** 20,000 messages buffered concurrently from 25 origin instances at **15,368 msg/s**, draining at **3,533 msg/s** to depth 0. Reported as two numbers because the naive single number is misleading — a deferred backlog sits for one full `DefaultRetryInterval` (measured 28.9 s) before *anything* drains, since `EnqueueAsync(attemptImmediateDelivery: false)` stamps `LastAttemptAt` and the due-predicate then holds the row for one interval (correct by design; `Notify.Send`'s `deferToSweep: true` leaves it null and skips the wait). Filed as informational finding **F2**. **(b) Slow-subscriber backpressure:** with 5 subscribers on the SAME instance and one reader stalled at 50 ms/event, the four healthy subscribers delivered **100.00% with zero drops** while the stalled one lost 197,028 of 200,000 events **entirely within its own bounded channel**. Isolation is structural and total — `Buffer(DropHead)` never backpressures the `BroadcastHub`, `Sink.ForEach`'s `Tell` never blocks, and `StreamRelayActor.WriteToChannel`'s `TryWrite` on a `DropOldest` channel never blocks — so a slow WAN link degrades only its own feed, visibly (per-stream eviction counter + `RecordSiteStreamEventDropped`). Finding **F3**. Caveat also recorded: an *unpaced* burst costs every subscriber, because the publish `Source.ActorRef` upstream of the hub is shared — that shared stage, not per-subscriber buffering, bounds burst absorption. | Closed by the row-25 run; no defect found in either half. | Closed 2026-08-15 — `docs/plans/2026-08-15-target-scale-load-test-results.md`. |

## Deferred — operational risk (from the initiative tracker, folded in 2026-07-12)
Two live items previously tracked ONLY in `archreview/plans/00-MASTER-TRACKER.md`'s registry are folded in here (NF5) so this register is the single tracking place. The tracker's narrative subsections remain as the historical evidence.

| # | Item | Where noted | Rationale for deferral | Revisit trigger |
|---|------|-------------|------------------------|-----------------|
| SBR | ~~**SBR oldest-crash total-outage gap**~~ **RESOLVED 2026-07-21 (owner decision — availability over partition-safety).** All clusters switched from `keep-oldest` to the `auto-down` downing strategy (Akka `AutoDowning`, `auto-down-unreachable-after` = 15s): a hard crash of EITHER node — active/oldest included — now fails over to the survivor in ~25s with no operator action. Accepted trade: a real network partition produces dual-active until an operator restarts one side. Decision record + evidence (live keep-oldest `DownReachable … including myself` log, Akka.NET 1.5.62 `KeepOldest.OldestDecision` source, rejected alternatives incl. the static-quorum-1 `DownAll` trap): `docs/plans/2026-07-21-auto-down-availability-decision.md`. | `archreview/plans/00-MASTER-TRACKER.md:194` + auto-memory `sbr-keep-oldest-2node-active-crash-gap` (both now historical) | — | Closed. Residual: seed-node boot-alone constraint (unchanged, documented in `Component-ClusterInfrastructure.md`); dual-active recovery is operator-driven. |
| vd03 | **`deploy/wonder-app-vd03/` overlay edits unapplied** — `appsettings.Central.json` needs `AllowSingleNodeCluster: true` + phantom-seed removal + `NodeName: central-a`; `install.ps1` needs `sc.exe failure` recovery actions. The `deploy/wonder-app-vd03/` artifact directory is intentionally untracked (production config out of source control), so the repo cannot ship the fix. | `archreview/plans/00-MASTER-TRACKER.md:198` (PLAN-01 T16/T20/T23) | Needs on-host access; without `NodeName` that deployment's audit rows stamp NULL `SourceNode` — **partially mitigated once PLAN-R2-08 Task 7 lands: the host now FAILS AT BOOT with a key-naming error instead of silently NULLing, so applying the overlay becomes mandatory at the next upgrade.** Owner: whoever maintains the host (user). | Next wonder-app-vd03 deployment/upgrade — **the Task 7 validator makes this row unskippable then.** |