Resolve all 46 low-severity architecture review findings

2026-09-09 22:56:36 -04:00
parent fef3e7675f
commit 9ee402bbcd
139 changed files with 2621 additions and 1082 deletions
@@ -38,7 +38,7 @@ contract changes since August are additive; all eight migrations are sound and t
 matches. The test suite is broad; fresh execution evidence and its limits are in §7.

 The reviewed baseline contained **92 unique findings: 13 High, 33 Medium, 46 Low**.
-**Remediation status: all 13 High and 33 Medium findings completed; 46 Low findings remain open.**
+**Remediation status: all 13 High and 33 Medium findings completed; 46 / 46 Low findings completed (see §11).**
 Completion evidence and limits are recorded in §§9–10.
 This includes low-priority test/documentation gaps and latent risks, not just reproduced failures.
 The reviewed High-priority findings were:
@@ -983,4 +983,110 @@ cross-reviews rechecked lifecycle, sandbox containment, relay provenance and mig
  explicit field/service metadata. Earlier fixture/build failures were corrected without
  weakening production guards: audit test doubles accept caller scope, and protocol-writing
  fixtures use stable ephemeral key rings. These are not claimed as before-fix reproductions.
- The 46 Low findings remain open. Focused test counts overlap the full suite and are not additive.
+- Low findings are tracked in §11. Focused test counts overlap the full suite and are not additive.
+
+
+## 11. Low-severity remediation progress
+
+**Completed: 46 / 46 Low findings.** All 46 Low findings are in scope. This register supersedes the
+baseline finding status only when a row is marked **Completed** with verification.
+High and Medium completion records in §§9–10 remain authoritative.
+
+Implementation follows the existing finding-specific fixes: preserve wire compatibility,
+bound maintenance and debugging work, enforce authorization at application boundaries,
+and correct documentation against current source. Independent UI, Flow Debugger, and
+runtime/device groups run in parallel with exclusive file ownership. Transport,
+maintenance, package/configuration work, and this register have one coordinating owner.
+Behavior changes receive focused regression checks; documentation and configuration
+changes receive source/structure validation. Final gates are full build, relevant/full
+solution tests, diff review, and a local rig rebuild for shipping runtime changes.
+
+| Finding | Status | Progress / evidence |
+|---|---|---|
+| SR-6 | **Completed** | Device-to-script linkage is explicitly inferred; alarm causal parents remain reserved for actual writes. Runtime/Commons regressions pass. |
+| SR-7 | **Completed** | Trace failure diagnostics use the undecorated logger; recursive trace capture regression passes. |
+| SR-8 | **Completed** | Thread-static scheduler identity prevents cross-scheduler inlining; saturation regression passes. |
+| SR-9 | **Completed** | Shared trigger gate has explicit process ownership; consumer Dispose cannot poison later callers. Lifecycle regression passes. |
+| SR-10 | **Completed** | Per-context helpers and guard closures are lazily cached; identity regression passes. |
+| CM-3 | **Completed** | Raw audit/site-call pulls reject limits outside 1–2000 before reading; both reconciliation option validators enforce the same cap. Boundary regressions pass. |
+| CM-4 | **Completed** | Bootstrap checks cancellation before resolving the actor system or joining seeds, and propagates canceled probes. Host suite 598 passed. |
+| CM-5 | **Completed** | Retired site ingest RPCs always return Unimplemented, including empty payloads; dead actor wiring removed. Contract regressions pass. |
+| CM-6 | **Completed** | Central and site HTTP/2 listeners share finite keepalive settings; metrics listener isolation remains enforced. Build and Host 598 tests pass. |
+| CM-7 | **Completed** | Channel graveyard retains active calls beyond grace; lifecycle locking closes shutdown/requeue races. Completion and shutdown regressions pass. |
+| CM-8 | **Completed** | Connected-header timeout is immutable per client, bound through validated CommunicationOptions. Parallel-instance regression passes. |
+| MG-5 | **Completed** | Total live-command budget includes lookup/write/audit/pull, reserves response grace, and propagates audit cancellation. Management suite 658 passed. |
+| MG-6 | **Completed** | Clock-independent keyset rescans capture late/backdated completion and descendant rows. Indexed key union bounds payload pages; real SQLite plans cover 1/64 IDs and malformed legacy JSON. Management 660 and AuditLog 407 tests pass. |
+| MG-7 | **Completed** | Live write results use bounded collector with explicit event/log truncation; Management suite 658 passed. |
+| MG-8 | **Completed** | Shared-script create/update uses cached compile/trust validation; TemplateEngine suite 483 passed. |
+| MG-9 | **Completed** | Positive page sizes clamp to 1000; null/nonpositive semantics preserved and boundaries tested. |
+| MG-10 | **Completed** | Management component catalog now documents all four FlowDebug commands, authority, paging and budget. |
+| UI-4 | **Completed** | Poll cleanup runs in outer finally and observes rejected renderer work; full UI suite 1416 passed. |
+| UI-6 | **Completed** | Scoped failover facade requires current Administrator authority before invoking the host backend; positive/negative tests pass. |
+| UI-7 | **Completed** | Exact route-policy sweep covers UI and external Secrets components; full UI suite 1416 passed. |
+| UI-8 | **Completed** | Shared parked-call relay resolves site ownership and denies unknown/out-of-scope targets for scoped principals; regressions pass. |
+| UI-9 | **Completed** | Tree value snapshots skip unchanged forest rebuilds while preserving in-place edits, child metadata and filtered key/inheritance behavior. |
+| UI-10 | **Completed** | Cookie login explicitly validates antiforgery; missing token rejected and valid real token issues cookie. CLI bearer-token endpoint preserved. |
+| UI-11 | **Completed** | Lazy site/area queries retain one untracked 50-row instance page per loaded branch; shared async gate serializes branch/refresh/staleness. SQLite paging and controlled interleaving regressions pass. |
+| UI-12 | **Completed** | Viewer sees no empty Notifications heading and Flow Debugger navigation matches its policy; regression passes. |
+| EI-6 | **Completed** | Purge actor forwards per-channel 50000-row tick budget; repository caps final partial batch. Purge/options 20 and repository/guard 10 tests pass. |
+| EI-7 | **Completed** | Root-DI admission registry retains one unfinished collection per registration across actor recreation; actual wait is bounded and task owns scope until completion. Full KPI suite 57 passed. |
+| EI-8 | **Completed** | Relay strips Accept-Encoding because it forwards raw response bytes without Content-Encoding; regression and full InboundAPI suite 360 pass. |
+| EI-9 | **Completed** | Outbox contract explicitly documents repeated duplicate batches during dual-active operation and lack of row claims. |
+| DL-4 | **Completed** | Echo edges replace ordinary duplicate endpoint pairs; unique-edge and echo-semantic regressions pass. |
+| DL-5 | **Completed** | Refresh bases graph order on authoritative Replay and appends live-only rows; order regression passes. |
+| DL-7 | **Completed** | Graph build parses each audit payload once and reuses projections; Commons suite 851 passed. |
+| DL-8 | **Completed** | Existing Medium chain-anchor fix revalidated by timestamp-skewed child/root regression. |
+| DL-9 | **Completed** | Commons Debugging namespace, per-run bounded collector ownership and redaction documented. |
+| DL-10 | **Completed** | Enum name/value serialization contracts and display ordering have separate regression coverage. |
+| DL-11 | **Completed** | Folded steps/lifecycle records retain latest exception detail and clear stale details after success; regressions pass. |
+| DC-8 | **Completed** | CLI harness covers all 19 groups plus browse/search/verify/certs; secured-write README added; CLI suite 475 passed. |
+| DC-9 | **Completed** | DCL docs corrected against current unsubscribe, write rejection, atomic disconnect and snapshot behavior. |
+| DC-10 | **Completed** | OPC shard plans are computed once per resolved chunk and reused for each monitored item; DCL suite 379 passed. |
+| DC-11 | **Completed** | All notifier base URLs are validated as absolute HTTP(S) before delivery; notifier suite 36 passed. |
+| XC-7 | **Completed** | Removed both migration-only sections from all eight rig site JSON configurations; parsed and verified each file. |
+| XC-8 | **Completed** | Source-discovered service maps and external LocalDb/Secrets mappings are compared against generated method descriptors and auth gates, with gate-removal negative control. Host suite 598 passed. |
+| XC-9 | **Completed** | Central SqlClient 6.1.1 removes per-test overrides; SignalR.Client 10.0.7 aligns with runtime; bUnit 2.7.2 uses stable syntax. Restore, zero-warning build and all solution suites pass. |
+| XC-10 | **Completed** | Global OPC/Mx options validate required identity strings at startup, preserving optional path defaults; DCL suite 379 passed. |
+| XC-11 | **Completed** | Umbrella index now includes instance picker/v1.4 logs and live remediation status. Fresh origin/main fetch confirms current main fef3e767 is already pushed (0 ahead/behind); no new push needed for the historical finding. |
+| XC-12 | **Completed** | Health-check remarks now distinguish seven central singleton proxies, five required and two best-effort exclusions. |
+
+
+### Final verification — completed 2026-09-09
+
+- Full solution restore succeeded after centralizing Microsoft.Data.SqlClient 6.1.1,
+  Microsoft.AspNetCore.SignalR.Client 10.0.7, and stable bUnit 2.7.2.
+- Full solution build: **0 warnings, 0 errors**. Canonical `docker/regen-proto.sh sitestream`
+  preserves the source descriptor path for the additive trace cursor; all 766 Communication
+  tests pass, including the complete source/vendored descriptor comparison.
+- Initial solution run (excluding deployed Playwright tests): 9474 passed, three failures.
+  One was the generated descriptor path, corrected and verified above. Two login integration
+  fixtures omitted the now-required antiforgery token; they now fetch the real rendered token
+  and cookie, preserve the original login assertions, and add missing-token rejection coverage.
+  Full Integration rerun: **118 passed, zero failed/skipped**. Across the full run with
+  corrected Communication and Integration suite reruns: **9478 passed, zero outstanding failures
+  or skips**. Counts replace earlier suite results rather than adding repeated tests. The deployed
+  Playwright smoke adds **1 passed**: real antiforgery login, initially unloaded topology, bounded
+  branch loading/page controls, and Flow Debugger navigation without browser errors.
+- `bash docker/deploy.sh` succeeded from the canonical repository. All eight primary rig nodes run
+  image `sha256:cfa919dd343ffa090995336f2854ac4d79d4c2bb919514aa7e72abf579b0b4ed`, with
+  zero restarts, `/health/ready` 200, two members and zero unreachable members per pair, and
+  exactly one `/health/active` 200 per pair (standby 503). Existing bind mounts and volumes
+  were preserved. Env2 containers were not redeployed.
+- Final completion audit verifies all 46 unique Low IDs have Completed rows, with no remaining
+  Low findings. `git diff --check` passes. The verified remediation is committed directly on `main`.
+- Validation logs: `/tmp/low-final-build2.log`, `/tmp/low-full-tests.log`,
+  `/tmp/low-comm-final.log`, `/tmp/low-integration-final.log`, `/tmp/low-rig-deploy.log`,
+  `/tmp/low-rig-health.json`, and `/tmp/low-smoke.log`. Focused regression evidence above overlaps
+  the full suite and is not additive.
+
+### Operational notes
+
+- MG-6 intentionally rescans matching execution-ID batches on each poll to capture completion
+  rows inserted late with earlier timestamps. Indexed SQLite key queries avoid scanning unrelated
+  retained audit payloads. Discovery memory grows with the traversed tree during the bounded
+  command; returned results have fixed count/byte caps. No central/site clock agreement is assumed.
+- EI-7 bounds the caller wait and admits at most one unfinished source collection per registration
+  across actor recreation on the same root provider. A source that ignores cancellation retains
+  its task and scope until actual completion; subsequent ticks do not create duplicate work.
+- EI-9 is resolved through the requested contract clarification: dual-active outbox writers can
+  deliver repeated duplicate batches; database row claims are not implemented by this change.
@@ -641,7 +641,10 @@ MS SQL for direct-write events). Unredacted secrets never persist.
  silently skipped (the global switch already covers them). The DELETE runs under
  `scadabridge_audit_purger` (the maintenance role); the append-only writer role
  is unaffected. Batch size is configurable via
-  `AuditLog:Purge:ChannelPurgeBatchSize` (default 5000). Each channel override
+  `AuditLog:Purge:ChannelPurgeBatchSize` (default 5000). Each channel deletes at
+  most `AuditLog:Purge:MaxPurgedRowsPerChannelPerTick` rows (default 50,000;
+  must be positive), including a smaller final batch when needed. Remaining
+  backlog resumes on the next tick so later channels are not starved. Each channel override
  runs in its own try/catch, mirroring the per-boundary error-isolation of the
  partition switch-out loop. Values are validated to be in
  `[30, RetentionDays]`; keys that are not a recognized `AuditChannel` enum name
@@ -772,7 +775,8 @@ size:
 "AuditLog": {
  "Purge": {
    "IntervalHours": 24,
-    "ChannelPurgeBatchSize": 5000
+    "ChannelPurgeBatchSize": 5000,
+    "MaxPurgedRowsPerChannelPerTick": 50000
  }
 }
 ```
@@ -20,6 +20,16 @@ Central cluster only. Sites have no user interface.
  - Colour is never the only signal — the button always carries its verb (see the label contract above), so the mapping is an accessibility affordance, not the meaning.
  - **`DialogSeverity` is an additive overload**, not a replacement: the boolean form remains the signature for the ~34 existing call sites, which are genuinely two-state. Adding a fifth optional parameter to the boolean form instead would have silently un-matched every existing four-argument NSubstitute stub — the row-28 gotcha, where a stub that stops matching returns a null `Task` and the caller NREs with no compile error. Reach for the overload only when `Caution` is what you mean.

+## Review hardening (2026-09-09)
+
+- Cookie login validates an antiforgery token before reading credentials or calling LDAP; the login form supplies the token. The JSON token endpoint retains its non-cookie client contract.
+- Manual failover goes through a scoped `AuthorizedManualFailoverService` that refreshes the circuit's Administrator grant before calling the host singleton. The component's visibility policy is an additional presentation guard.
+- Flow Debugger retry/discard validates the current Deployer grant and resolves site scope inside its service. Scoped callers cannot relay an unknown site. Refresh merges preserve Replay's event ordering and append only rows known exclusively to the live run.
+- Health and Alarm Summary observe the whole posted refresh task and release their poll gate even when renderer dispatch fails during disposal.
+- The Topology initially loads permitted sites. Expand a site and select **Load instances** to load that site's area metadata and first root-instance page; each expanded area has its own loader. **Previous** and **Next** replace that branch's 50-instance page. Queries are untracked and omit member collections; templates are not loaded. Live refresh and staleness checks cover loaded pages. Search and counts explicitly describe loaded data.
+- `TemplateFolderTree` compares immutable value snapshots of folder/template hierarchy and extra children before rebuilding. In-place renames, moves, derived-state changes, and composition edits remain visible; filter and reveal changes are applied independently.
+- Navigation grants Flow Debugger its own policy and hides Notifications unless the principal holds a role that admits a child. Route-policy coverage freezes every CentralUI route and the external Secrets route.
+
 ## Failover Behavior

 - A **load balancer** sits in front of the central cluster and routes to the active node.
@@ -197,6 +197,7 @@ ZB.MOM.WW.ScadaBridge.Commons/
 │   │                               #   AuditQueryParamParsers, ExecutionTreeNode,
 │   │                               #   SiteCallKpiSnapshot, SiteCallPaging,
 │   │                               #   SiteCallQueryFilter, SiteCallSiteKpiSnapshot
+│   ├── Debugging/                  # FlowStep/FlowGraph, projector/builder, trace collector, ScriptTraceExtra
 │   ├── DataConnections/            # OPC UA endpoint config value objects + enums
 │   ├── Flattening/                 # FlattenedConfiguration, ConfigurationDiff,
 │   │                               #   DeploymentPackage, ValidationResult
@@ -326,6 +327,8 @@ It must **not** contain any business logic that orchestrates other components, s

 Examples currently in Commons that fall under this carve-out: `Result<T>`, `ScriptParameters` and `ScriptArgs` (script-parameter shaping), `ValueFormatter` (value-to-string formatting), `DynamicJsonElement` (dynamic JSON access), `StaleTagMonitor` (a self-contained heartbeat watchdog), `OpcUaEndpointConfigSerializer` (typed↔legacy JSON conversion of a Commons value object) and `OpcUaEndpointConfigValidator` (rule checks over a Commons value object). These are intentionally placed in Commons so every consuming component shares one implementation rather than duplicating the logic. Anything that would require an I/O dependency, mutable cross-call state, or knowledge of another component's behavior does **not** qualify and must live in the owning component.

+**Debugging helpers and ownership.** `Types/Debugging` contains `FlowStep`, `FlowGraph`, lane/kind/outcome enums, lifecycle/log projections, `ScriptTraceExtra`, `FlowStepProjector`, `FlowGraphBuilder`, and `FlowTraceCollector`. Projection and graph construction are pure transformations of audit events. The collector is created and owned by one traced run, never shared across runs or registered as a process singleton. It holds only bounded in-memory rows, applies the supplied `IAuditRedactor` abstraction, reserves capacity for step rows ahead of log lines, and freezes collection on `Complete()`. Callers report its event/log truncation flags and own persistence separately. A null redactor is appropriate for rows already redacted by the site. This self-contained per-run state fits REQ-COM-6; it performs no I/O or component orchestration.
+
 ### REQ-COM-7: Minimal Dependencies

 Commons must depend only on core .NET libraries (`System.*`, `Microsoft.Extensions.Primitives` if needed). It must **not** reference:
@@ -499,3 +499,16 @@ mapped gRPC methods can execute on the control-plane socket; they are rejected o
 sockets. Dedicated metrics sockets expose metrics and health only. Decisions use LocalPort and
 endpoint metadata, never the caller's Host/authority header. The env2 central/site pairs now enable
 the same deterministic bootstrap guard as the main rig for simultaneous container recreation.
+
+
+### Transport hardening (2026-09-09 Low review)
+
+Central and site Kestrel servers use the same HTTP/2 keepalive settings. Central's optional
+metrics listener exposes metrics/health only under listener isolation; it does not grant access
+to the control plane or UI. The gRPC socket likewise rejects ordinary HTTP endpoints.
+`ScadaBridge:Communication:ConnectedHeaderTimeout` (default 10 seconds) belongs to each client;
+when headers do not arrive, only an actual first event signals a connected stream.
+Reconciliation pulls accept 1–2000 rows and also obey the serialized wire-byte budget.
+SiteStreamService's legacy ingest RPCs return `Unimplemented`; central ingestion uses
+CentralControlService. Replaced channel pairs remain parked until both the idle grace and
+all current calls have finished, including long attribute waits.
@@ -320,7 +320,7 @@ The registered `SourceReference` is a **NodeId** for OPC UA (the picker, CSV and
 - **Reconnect handling**: on entering **Reconnecting**, the actor pushes a `NativeAlarmSourceUnavailable` to every alarm subscriber (consumers mark mirrored alarms uncertain rather than clearing them). On successful reconnection it re-subscribes the feed; the adapter re-emits a snapshot, reconciling state.
 - **Shared condition filter (last-subscriber-wins)**: because the feed is opened once per source, it carries a single condition filter. A second subscriber that registers a *different* filter for the same source overwrites it (last writer wins) and the actor logs a warning — co-subscribers to one source are expected to agree on the filter.
 - **Routing index**: subscribed sources are bucketed by the **first path segment** of their reference (everything before the first `.`), and a transition is matched only against the bucket for its own first segment, plus a residue list of sources that carry no separator at all (a prefix shorter than one segment, which can match other buckets). This is sound because a source reference containing a separator can only prefix a reference sharing its entire first segment. The per-source `StartsWith` test and the condition-type gate inside the bucket are unchanged, so routing decisions are identical to the previous linear scan over every subscribed source; snapshot boundaries bypass the condition filter, but OPC UA boundaries are sent only to the exact source binding. Connection-wide Mx boundaries broadcast to all subscribers while preserving their refresh epoch.
- **Gateway union filter (MxGateway)**: `StreamAlarms` carries a single `alarm_filter_prefix`, so the adapter opens the stream on the **longest common prefix** of the currently subscribed source references and restarts it for each new source, including one covered by the existing prefix, so every newly bound source receives standing state. Each transport refresh has connection-wide start/end markers and a distinct epoch. An unsubscribe never restarts the stream — the prefix it leaves behind is at worst too broad, which costs bandwidth, not correctness. The prefix is a bandwidth optimisation only; the actor's per-source + condition-type gate remains authoritative, mirroring the OPC UA server-side WhereClause stance.
+- **Gateway union filter (MxGateway)**: `StreamAlarms` carries a single `alarm_filter_prefix`, so the adapter opens the stream on the **longest common prefix** of the currently subscribed source references and restarts it for each new source, including one covered by the existing prefix, so every newly bound source receives standing state. Each transport refresh has connection-wide start/end markers and a distinct epoch. An unsubscribe never restarts the stream — the prefix it leaves behind is at worst too broad, which may increase bandwidth. Correctness also depends on the epoch-scoped snapshot cache merging concurrent live updates and rejecting stale snapshot ends (DC-5); a broad filter alone provides no such guarantee. The prefix is a bandwidth optimisation only; the actor's per-source + condition-type gate remains authoritative, mirroring the OPC UA server-side WhereClause stance.

 ### Protocol-Neutral Types & Messages

@@ -387,7 +387,7 @@ Instance Actors and NativeAlarmActors send explicit unsubscribe requests from `P

 OPC UA refreshes carry an exact `SnapshotSourceReference` (including the empty Server binding) and a `SnapshotId` shared by `SnapshotStart`, entries and `SnapshotComplete`. Each NativeAlarmActor replaces its buffer on a new start and commits only a matching end; duplicate or superseded ends cannot clear a committed snapshot. The adapter tags each monitored item's refresh separately even though ConditionRefresh targets the shared event subscription. Mx snapshots retain their connection-wide framing. An interrupted refresh buffer is discarded on disconnect.

-This pattern ensures no messages are lost during connection transitions and is the standard Akka.NET approach for actors with I/O lifecycle dependencies.
+Subscription demand survives connection transitions through coalescing and owner tracking. Device writes during those transitions fail immediately so stale setpoints cannot be replayed.

 **OPC UA-specific notes**: The `RealOpcUaClient` uses the OPC Foundation SDK's `Session.KeepAlive` event for proactive disconnect detection. The SDK sends keep-alive requests at the subscription's `KeepAliveCount × PublishingInterval` (default: 10s). When keep-alive fails, the `ConnectionLost` event fires, triggering the same reconnection flow. On reconnection, the DCL re-creates the OPC UA session and subscription, then re-adds all monitored items.

@@ -413,7 +413,7 @@ Each adapter implements the `IDataConnection.Disconnected` event to proactively

 **Event marshalling**: The `DataConnectionActor` subscribes to `_adapter.Disconnected` in `PreStart()`. Since `Disconnected` may fire from a background thread (gRPC stream task, OPC UA keep-alive timer), the handler sends an `AdapterDisconnected` message to `Self`, marshalling the notification onto the actor's message loop. This triggers `BecomeReconnecting()` → bad quality push → retry timer.

-**Once-only guard**: `OpcUaDataConnection` uses a `volatile bool _disconnectFired` flag to ensure `RaiseDisconnected()` fires exactly once per connection session. The flag resets on successful reconnection (`ConnectAsync`).
+**Once-only guard**: `OpcUaDataConnection` uses an `int _disconnectFired` guard with `Interlocked.Exchange(ref _disconnectFired, 1)` to ensure `RaiseDisconnected()` fires exactly once per connection session. The guard resets atomically with `Interlocked.Exchange(ref _disconnectFired, 0)` on successful reconnection (`ConnectAsync`).

 ## Write Failure Handling

@@ -80,14 +80,16 @@ The table is non-partitioned, on the standard `[PRIMARY]` filegroup, with no DB-

 The recorder is the Akka.NET cluster singleton **`kpi-history-recorder`** (singleton-manager actor `kpi-history-recorder-singleton`), running on the active central node. It is **not readiness-gated** — the recorder is pure observability and must never gate `/health/ready`, so it is started outside the readiness barrier (unlike the operational singletons). On graceful shutdown it drains via a `CoordinatedShutdown` task for clean singleton handover.

-A timer fires every `SampleInterval` (default 60s; an immediate first tick primes the series, then it settles into the periodic cadence). On each tick the recorder:
+A timer fires every `SampleInterval` (default 60s; a first tick after five seconds primes the series, then it settles into the periodic cadence). On each tick the recorder:

 1. Opens a **per-tick DI scope** (scoped `DbContext`/repository — the same scope-per-sweep pattern as the `NotificationOutboxActor`).
-2. Enumerates the registered `IEnumerable<IKpiSampleSource>`. Each source returns an `IReadOnlyList<KpiSample>` stamped with the tick's single `CapturedAtUtc`.
+2. Enumerates the registered `IEnumerable<IKpiSampleSource>`. Each collection runs with its own DI scope and returns an `IReadOnlyList<KpiSample>` stamped with the tick's single `CapturedAtUtc`.
 3. Writes all collected samples via `IKpiHistoryRepository.RecordSamplesAsync`.

 **Best-effort, per-source isolation.** Each source call and the write are individually guarded. A throwing source is logged and its samples skipped for that tick; it never aborts the tick, the other sources, or the source component itself. This is the same `IEnumerable<>`-of-adapters decoupling pattern used by `INotificationDeliveryAdapter`.

+Each source has a `SourceTimeout` (default 10 seconds) bounding the actual wait as well as requesting cancellation. A source that ignores cancellation keeps its task, cancellation source, and DI scope until it actually finishes; later ticks skip that registration instead of starting replacements. Healthy sources continue with separate scopes, so they do not share a busy `DbContext` with unfinished work. A root-DI singleton admits at most one unfinished collection per registration across recorder instances, including actor stop/recreation on the same provider. Admission is atomic; replacing an actor does not start another call against a still-running source. Different root providers have independent admission state.
+
 **Hourly rollup tick.** A third timer (`kpi-rollup`) fires every `RollupInterval` (default 1h). On each tick the recorder opens a per-tick DI scope and calls `IKpiHistoryRepository.FoldHourlyRollupsAsync` over the window `[TruncateToHour(now) − RollupLookbackHours, TruncateToHour(now))` — an **exclusive** upper bound at the current hour start, so the in-progress hour is never folded. The fold groups raw `KpiSample` rows by `(series, hour)`, applies the per-metric aggregation intent (sum for rate metrics, last-value for gauges — see Query), and **idempotently upserts** each `(Source, Metric, Scope, ScopeKey, HourStartUtc)` row. Re-folding the trailing lookback window (default 3h) rather than only the last hour is the self-heal for a **singleton-failover-missed tick**: because the upsert overwrites the aggregate in place, re-running the same window produces identical values (no double-count), so any complete hour a missed tick skipped is refilled on the next fold. A `_rollupInFlight` guard (mirroring `_sampleInFlight`) skips a tick while a prior fold is still running so two idempotent upserts never race on overlapping rows. During a **failover overlap** — the old singleton incarnation's in-flight fold and the new node's first fold both `Add`-ing the same `(series, hour)` row — the loser of the unique `IX_KpiRollupHourly_Series` upsert race discards its **whole fold pass** (a single `SaveChanges`), logged at Information and repaired by the next idempotent fold; when reading fold logs after a failover, the failure grain is the **pass, not the row** (arch-review 04 round 2, R5).

 **One-shot backfill.** So the 30 d / 90 d charts are not blank until enough wall-clock passes, the recorder runs a backfill on start (`kpi-rollup-backfill` timer) that folds the retention window, reusing the same `FoldHourlyRollupsAsync` path. The `FoldHourlyRollupsAsync` repository call materializes its whole window in memory, so the backfill folds the window in **≤24 h slices, oldest-first** — one bounded slice per fold — rather than handing the fold the full 90-day window at once (which at fleet volume is tens of millions of rows in one tracked pass). Between slices it lowers its in-flight guard so periodic folds **interleave** instead of stalling behind the historical pass; a periodic fold and a backfill slice strictly alternate through the single `_rollupInFlight`/`_backfillInFlight` gate pair, so the two idempotent upserts never race on overlapping rows. On a **failover restart** the plan first consults the `IKpiHistoryRepository.GetLatestRollupHourAsync` watermark: if the newest folded hour is already within `RollupLookbackHours` of now (the common case) the whole historical pass is **skipped** — the periodic fold's lookback covers the tail — otherwise the window is **shrunk to the un-rolled tail** `[watermark, TruncateToHour(now))` (re-folding the watermark hour itself as an idempotent safety margin) instead of the full `RetentionDays` floor. A `_backfillDone` latch makes the whole pass at-most-once per actor lifetime, and it defers (re-arms) while a periodic fold is in flight. Being an idempotent upsert, it is safe (idempotent) to re-run on a later failover restart; the watermark fast-path makes the failover re-run cheap as well.
@@ -201,6 +203,7 @@ Bound from the `ScadaBridge:KpiHistory` section on the central host (Options pat
 | Option | Default | Notes |
 |---|---|---|
 | `SampleInterval` | `60s` | Recorder tick cadence. Must be `> 0`. |
+| `SourceTimeout` | `10s` | Actual maximum wait per source; must be `> 0` and `<= 1 day`. Unfinished work remains owned and is skipped on subsequent ticks. |
 | `RetentionDays` | `90` | Raw `KpiSample` rows older than this are purged. Bounded to `[1, 3650]` days. |
 | `PurgeInterval` | `1d` | Daily purge cadence (drives **both** raw and rollup purges). Must be `> 0`. |
 | `DefaultMaxSeriesPoints` | `200` | Default bucket cap for a series query when the caller does not override it. Bounded to `[2, 5000]`. |
@@ -230,6 +230,19 @@ The two-person authorization workflow for writes through the MxAccess Gateway. B
 - **VerifyEndpointCommand**: Ask a site to probe an OPC UA endpoint (temporary client, short timeout) and report success / typed failure / a captured-but-untrusted server cert. Read-only — never trusts. Actor-dispatched, `SiteIdentifier`-required + site-scope, relayed via `CommunicationService.VerifyEndpointAsync` (arch-review C4). (`Design` role — runs inside the Admin-gated connection editor.)
 - **TrustServerCertCommand** / **RemoveServerCertCommand** / **ListServerCertsCommand**: Manage the site's OPC UA trusted-peer PKI store; Trust/Remove broadcast to **both** site nodes (see Component-SiteRuntime.md). Actor-dispatched and `Admin`-gated at the actor (arch-review C4) — each handler requires the command's `SiteIdentifier`, enforces site-scope, then relays via `CommunicationService.TrustServerCertAsync` / `RemoveServerCertAsync` / `ListServerCertsAsync`; the site-side reconcile across both nodes is owned by the site `CertStoreActor`. (`Admin` role.)

+### Flow Debugger
+
+All four commands return `FlowDebugRunResult`: the chain root execution ID, completion/error metadata, audit events, timestamps, and independent event/log truncation flags.
+
+| Command | Behaviour | Required role |
+|---|---|---|
+| `FlowDebugRunMethodCommand` | Validates an inbound API method's parameters and runs it with tracing; routed calls enforce the caller's site scope. | Deployer |
+| `FlowDebugRunValueChangeCommand` | Writes an instance attribute through the real site path and polls for the triggered execution tree. | Deployer |
+| `FlowDebugRunScriptCommand` | Resolves the effective script; writes its trigger attribute for write-trigger kinds, otherwise invokes it directly with validated parameters. | Deployer |
+| `FlowDebugReplayCommand` | Reads the stored execution tree for any execution ID, enforcing caller scope and retaining the repository's chain anchor. | Any of Deployer, Administrator, Viewer |
+
+Live commands require the active central node and return `STANDBY_NODE` elsewhere. They perform real script/device operations and write operator-attributed audit roots. One cancellation budget starts at runner entry, covers lookups, execution/write, audit and trace pulls, and reserves five seconds below `LongRunningCommandTimeout` for the reply. Configuration must exceed that grace. Write-trigger waits are clamped to 1–295 seconds and the remaining total budget. Polling drains all `(OccurredAtUtc, EventId)` keyset pages in execution-ID batches of at most 64. Each subsequent poll intentionally rescans those batches: closing rows are inserted at completion but stamped with their start time, so a persistent timestamp cursor would miss them. The root is a fresh execution GUID; scans use no timestamp lower bound and do not require central/site clock agreement. This removes fixed-page starvation but deliberately retransfers known rows. Site reads use partial valid-JSON execution/parent-expression indexes; an indexed union of keys deduplicates both matches and limits the page before loading payloads. Malformed legacy details are excluded from trace matching without preventing index initialization. Discovery retains deduplicated rows for the bounded command duration, so working memory grows with the traversed tree; only the returned reply has a fixed collector cap. Inline results use `FlowTraceCollector` count/byte caps and expose truncation. Replay also has repository count/UTF-8 bounds; it does not execute scripts or write to devices.
+
 ## Authorization

 Every incoming message carries the authenticated user's identity and roles. The ManagementActor enforces the same role-based authorization rules as the Central UI:
@@ -109,7 +109,7 @@ Terminal rows (`Delivered`, `Parked`, `Discarded`) are removed by a **daily purg

 The site→central handoff is **at-least-once**. Central ingests an inbound notification submission with an insert-if-not-exists on `NotificationId`, then acks the site; the site S&F engine clears the message only on that ack. Because central acks only after the row is persisted (ack-after-persist), a lost ack causes the site to resend, and the GUID `NotificationId` idempotency key makes the resend harmless — the duplicate insert is a no-op.

-A rare central failover mid-delivery could re-send one already-`Delivered` notification. This is an accepted trade-off, consistent with the duplicate-delivery trade-off the Store-and-Forward Engine already accepts.
+Delivery has no row-level claim or lease: dispatchers read due rows and persist delivery status after sending. An interrupted delivery can therefore be sent again after failover. During a dual-active partition permitted by the auto-down policy, both central singletons can repeatedly select overlapping batches and deliver duplicate notifications across multiple sweeps; exposure is not limited to one notification or one handover. The `NotificationId` deduplication above protects ingestion, not external Email/SMS delivery. This remains an accepted at-least-once delivery limitation; there is no dispatcher fencing or exactly-once guarantee.

 ## Dispatcher