Resolve all high-severity architecture review findings

2026-09-09 16:13:58 -04:00
parent 8bfe65512e
commit 146a13238f
192 changed files with 3160 additions and 753 deletions
@@ -1,6 +1,6 @@
 # ScadaBridge — architectural / code review (whole system)

-**Date:** 2026-09-09 · **Status:** SECOND-PASS REVIEW REPORT (documentation only; no production code changed)
+**Date:** 2026-09-09 · **Status:** ALL 13 HIGH FINDINGS COMPLETED (see §9)
 **Reviewed revision:** `main` @ `c078bc8f`. Its only difference from the first review's
 `d8229916` is the original findings document; production source/configuration is identical.
 **Scope:** all 26 `src/` projects, 32 test/support projects, both Docker rig configurations,
@@ -13,7 +13,8 @@ rechecked all High source paths, ran build/test/format/package checks, and execu
 probes. Each area revalidated its existing IDs and searched for additional defects. Source-traced
 mechanisms, runtime probes, latent risks, and unproven operational impact are distinguished below.
 This is a focused whole-system review, not a claim that every line or production topology was
-exercised. No rig rebuild, device writes, failover drill, or remote QA/production inspection ran.
+exercised. The review phase ran no rig rebuild, device writes, failover drill, or remote QA/production inspection.
+Sections 1–8 preserve that reviewed baseline; §9 records the subsequent implementation and local rig verification.

 **Second-pass changes:** 11 new findings; UI-3 withdrawn as unsupported; XC-6 merged into CM-5;
 UI-4/6/7 reclassified as Low hardening gaps. Existing IDs are preserved for traceability. The
@@ -36,9 +37,11 @@ references; every mapped gRPC service is behind a fail-closed constant-time inte
 contract changes since August are additive; all eight migrations are sound and the model snapshot
 matches. The test suite is broad; fresh execution evidence and its limits are in §7.

-The updated register has **92 unique active findings: 13 High, 33 Medium, 46 Low**.
+The reviewed baseline contained **92 unique findings: 13 High, 33 Medium, 46 Low**.
+**Remediation status: all 13 High findings completed; 79 findings remain open (33 Medium, 46 Low).**
+Completion evidence and limits are recorded in §9.
 This includes low-priority test/documentation gaps and latent risks, not just reproduced failures.
-The High-priority findings are:
+The reviewed High-priority findings were:

 | # | Sev | Finding | Area |
 |---|---|---|---|
@@ -821,3 +824,75 @@ A `pass` is limited to the cited check, not a blanket claim that the component i
 Add these to the repository's chosen CI system with existing build/test gates; keep formatting
 cleanup and framework/package migration in separate maintenance work so they do not obscure the
 correctness/security remediation.
+
+
+## 9. High-severity remediation completion
+
+**Completed 2026-09-09. All 13 High findings are resolved and verified.** Implementation
+started from `8bfe6551` on `codex/resolve-high-review-findings`, with parallel device/runtime,
+transport, and UI/security agents plus primary-agent integration and independent reviews.
+The findings above preserve the reviewed baseline; this section supersedes their open status.
+Medium/Low findings remain outside this remediation unless a change was required for a High fix.
+
+| Finding | Status | Implementation / verification |
+|---|---|---|
+| DC-1 | **Completed** | Owner-checked PostStop unsubscribe and DeathWatch release dead demand. Shared/replacement demand survives old-owner termination and in-flight completion. Deterministic lifecycle/race regressions pass. |
+| DC-2 | **Completed** | Initial subscribe and resolution exclude each other’s in-flight tags; equal live handles are never released. Mx reverse-map removal is atomic and identity-checked. Adapter and controlled-race regressions pass. |
+| DC-3 | **Completed** | Connecting/reconnecting reject single and batch writes immediately; subscription retries are deduplicated by owner/source. No stale setpoint replay after reconnect. Full DCL suite: 365 passed. |
+| DC-12 | **Completed** | OPC snapshot start/entries/end retain source and epoch; NativeAlarmActor ignores unrelated/duplicate ends and drops partial snapshots on disconnect. Legacy Mx framing remains supported. Actor tests prove standing-alarm preservation; live two-boiler feed verifies distinct wire scopes and epochs. |
+| CM-1 | **Completed** | Servers and all six application channels use explicit 32 MiB send/receive limits. Protobuf-exact 2 MiB audit budgets cover pending/cached drain and reconciliation paging. Both 256×64 KiB lanes drain without loss; real gRPC 17 MiB response passes. Oversize-row policy is explicit below. |
+| CM-9 | **Completed** | Authenticated site identity guards RPC payloads, audit rows and notification status. SQL upsert rejects another site’s stored operation ID, including duplicate-insert retry. Real gRPC denial, outbox ownership and 29 SQL Server lifecycle/ownership tests pass. |
+| CM-10 | **Completed** | Reads/idempotent ingests may retry; mutations retry only after explicit server pre-dispatch rejection. Ambiguous primary/secondary response loss returns `OUTCOME_UNKNOWN` without replay. Real transport and effect-count tests pass. |
+| UI-1 | **Completed** | All 16 configuration repository-bypass pages use typed commands. One transaction spans entity saves and final scrubbed audit flush, including generated IDs; cache publication follows commit. Direct instance/area paths retain domain checks within the same boundary. LDAP CRUD and audit-failure rollback regressions pass; deployed browser CRUD passes. |
+| UI-2 | **Completed** | Passive ping/Blazor validation neither slides cookies nor advances activity. Trusted browser input posts with antiforgery; per-login activity reaches existing circuits. Stable NameIdentifier preserves antiforgery across role refresh. Real HTTP fake-clock tests and deployed trusted/synthetic-input browser gate pass. |
+| UI-13 | **Completed** | Circuit authority remaps stored groups in fresh scopes; mutations require current roles and persisted-site scope. API keys, reports, failover, parked messages, instances and Topology recheck before dispatch. Revoked-scope and authorized-positive tests pass, including nonnumeric site identifiers. Explicit server-configured auto-login retains synthetic grants under its existing startup guard. |
+| SR-1 | **Completed** | Attribute get/set/wait/batch continuations use `ConfigureAwait(false)`. Saturated two-worker synchronous read/write regressions pass without watchdog assistance; wait/trace regression coverage remains green. |
+| SR-11 | **Completed** | Sync/async transaction and command connection/transaction readback retain auditing wrappers. SQLite regressions prove persisted writes and exactly one audit through each former escape. |
+| MG-1 | **Completed** | Trust gate denies Unsafe, RuntimeHelpers and Runtime.Loader, including alias/static imports, while preserving async and CallerMemberName. ScriptAnalysis suite: 81 passed. |
+
+### Final verification
+
+| Gate | Result / evidence |
+|---|---|
+| Final solution build | `dotnet build ZB.MOM.WW.ScadaBridge.slnx --no-restore`: **0 warnings, 0 errors**. `/tmp/high-final-build3.log` |
+| Selected full solution | `dotnet test ZB.MOM.WW.ScadaBridge.slnx --no-build --no-restore --filter 'FullyQualifiedName!~PlaywrightTests' --logger trx --results-directory /tmp/scadabridge-high-final-green`: **9,310 passed, 0 failed, 0 skipped**; exit 0. `/tmp/high-final-tests2.log` |
+| Final UI follow-up | After the final auto-login compatibility fix: **1,392 passed, 0 failed, 0 skipped** in the full CentralUI suite. This reruns the 1,391 UI cases included above plus the new regression; counts are not additive. `/tmp/high-ui-final.log` |
+| Deployed browser | LDAP create/edit/delete round trip and trusted-input/passive-input/antiforgery/throttle scenario: **2 passed, 0 failed, 0 skipped**. `/tmp/high-browser-final.log` |
+| Independent reviews | Runtime/trust, transport/ownership, and UI transaction/authorization reviews passed after corrections; final auto-login guard follow-up also passed. `/tmp/scadabridge-ui-independent-review.md` |
+| Local rig | Final image `sha256:b9d6b60ab77093ee537470992fb891f78cc9888170d290021855c902c0115eb6` on **all eight nodes**. All ready (200), one active per pair (200) and four standbys (503), two members per pair, zero unreachable/restarts, all five central singletons reachable. Site-A replication connected with backlog 0. `/tmp/scadabridge-final-loginfix-postdeploy-health.json` |
+| Change hygiene | `git diff --check` clean. Canonical original-repository config/log mounts and named data volumes preserved; env2 remains untouched. Component requirements updated with implementation. |
+
+The local image was built with `bash docker/build.sh` from the implementation worktree, then
+all eight nodes were recreated using the canonical original repository’s Compose file. This
+keeps existing configuration, logs and data in place rather than rebinding them to the worktree.
+No device writes, failover drill, remote deployment, commit or push was performed by this remediation.
+
+### Regression evidence and limits
+
+- SR-1/SR-11 reproduced **five failures before** their fixes; the runtime follow-up passed 39
+  targeted tests. MG-1 reproduced five forbidden-call failures while two benign cases passed;
+  all 81 ScriptAnalysis cases pass afterward. Full SiteRuntime: **774 passed**.
+- CM-1’s 17 MiB response failed at the previous default limit. CM-9’s stored-owner SQL regression
+  failed before its guard. Full Communication **756**, ConfigurationDatabase **411**, AuditLog
+  **396**, Host **590**, and integration **117** tests pass. Two historical Host tests expected
+  blind mutation retry on `Unavailable`; they now model explicit pre-dispatch rejection, with
+  a separate real-transport test proving ambiguous failure never dispatches to the peer.
+- Device tests control owner death, replacement, shared-tag demand, subscribe/resolve races,
+  disconnected writes and interleaved snapshots. A stale failover test assertion was corrected:
+  backup demand must be restored while an original subscribe remains pending. The full DCL
+  suite subsequently passed. The local two-boiler snapshots were empty; standing-condition
+  preservation is proven by deterministic actor tests, **not by live raised alarms**.
+- UI revocation and Topology denied-mutation tests reproduced failures before correction.
+  The first LDAP baseline test stopped at an incorrect seeded-row count, so it does **not**
+  establish a before-fix missing-audit reproduction. Corrected real-SQLite CRUD/audit/rollback
+  tests pass. A real HTTP regression found antiforgery failure after role refresh; stable user
+  identity fixed it. The deployed browser found the synthetic auto-login role-loss regression;
+  the explicit trusted-configuration guard passed its red/green test and repeat live CRUD gate.
+- A single audit row above 2 MiB travels alone up to the 32 MiB wire ceiling. A larger row remains
+  durable/unacknowledged with an explicit operator-action error, without truncation or silent
+  loss. Stored LDAP groups are remapped live; directory group-membership changes still require
+  login. Pre-upgrade cookies without stable NameIdentifier may require a fresh sign-in.
+- No separate formatter cleanup or framework/package migration was included. The original
+  Medium/Low register and baseline formatting debt remain. The local rig has an existing
+  `zztest-conn-5d93ea81` test connection retrying unreachable `localhost:4840`; healthy cluster
+  readiness does not assert that every configured device endpoint is reachable.
@@ -881,3 +881,20 @@ orphaned entries) and in the CLI's `audit tree` output.
  reusing the existing audit-KPI reads; the resulting `totalEventsLastHour` /
  `errorEventsLastHour` / `backlogTotal` series render as trends on the Audit Log
  page via `KpiTrendChart`.
+
+### Serialized-byte budgets for telemetry and reconciliation (2026-09-09)
+
+The audit-only drain, combined cached telemetry drain, and site reconciliation responses use
+`AuditWireBudget`: the complete protobuf row (including error message/detail, summaries, extra,
+provenance and nested operational fields), repeated-message tag/length prefixes, five-byte gRPC
+frame and the continuation flag must fit a normal **2 MiB** budget. A drain sends a prefix and
+marks only acknowledged rows forwarded. A byte-limited pull returns `MoreAvailable = true`;
+its cursor advances from the last returned row, so unreturned rows remain available.
+
+A single row exceeding 2 MiB is sent alone if it fits the coordinated **32 MiB** channel ceiling.
+A row exceeding that ceiling is never truncated, falsely acknowledged or silently skipped:
+the operation reports `ResourceExhausted` with its row identifier and required byte count, and
+the row remains durable locally. Good rows preceding it can still advance. Operators must
+reduce that oversized payload or deploy coordinated larger limits to resume past it; repeated
+drain failures are logged. A lost response retains the existing at-least-once retry/cursor
+semantics. The separate Flow Debugger trace pull remains governed by its own contract.
@@ -330,3 +330,11 @@ Per-leaf alarm rendering (leaf nodes are individual conditions for native alarms
 - **Site Call Audit**: Serves the `SiteCalls` table queries and relays Retry/Discard actions to sites for the Site Calls page.
 - **Audit Log (#23)**: Serves all `AuditLog` table queries (filter / grid / drilldown / CSV export) for the new Audit Log page and the drill-in surfaces on Notifications, Site Calls, External Systems, Inbound API keys, Sites, and Instances. Payload capture, redaction, and per-site authorization follow the Audit Log component's "Payload Capture Policy" and "Security & Tamper-Evidence" sections.
 - **KPI History (#26)**: The Central UI hosts the `KpiHistoryQueryService` (scoped-repository read over `IKpiHistoryRepository.GetRawSeriesAsync` + `KpiSeriesBucketer`, dual-ctor test seam) and renders the reusable custom-SVG `KpiTrendChart` fed by it. Trend sections appear on the Notification Outbox, Site Calls, and Audit Log pages and in a per-site panel on the Health Monitoring dashboard; a query failure degrades to an unavailable-chart placeholder rather than breaking the page. See [Component-KpiHistory.md](Component-KpiHistory.md).
+
+### Configuration mutations and live circuit authority (2026-09-09)
+
+Page configuration handlers use `ConfigurationCommandService` and the guarded management command boundary. Database changes and their scrubbed audit entries commit in one transaction, including generated entity IDs. Existing instance service operations execute through the same transaction seam and retain their domain validation and audit ownership. Circuit read tracking is cleared after a separate transaction scope changes configuration.
+
+Cookie circuits retain their initial principal without retaining `HttpContext`; authoritative reads re-map stored directory groups against current grants. Mutation handlers recheck role and persisted target site immediately before dispatch. Passive session pings do not count as user activity. Trusted browser input posts an antiforgery-protected activity signal; the server activity tracker also makes it visible to the existing circuit.
+
+The explicit server-configured development auto-login mode preserves its synthetic all-role principal. It uses the Cookies scheme but has no LDAP group claims; it must not enter directory re-mapping. This exception follows validated `AuthDisableLoginOptions`, never a caller-supplied claim.
@@ -452,3 +452,33 @@ Disconnect is detected at the **transport layer**, never via an application-leve
 - **Audit Log (#23)**: Sites forward audit-event telemetry (push) and serve the `PullAuditEvents` gRPC reconciliation pull from their `ISiteAuditQueue`; the central `AuditLogIngestActor` is the ingest target for both the push path and the combined cached-call telemetry packet.
 - **Site Event Logging**: Event log queries are routed through communication.
 - **Management Service**: The ManagementActor runs at `/user/management` on central nodes and is reached **in-process** via `ManagementActorHolder`. The CLI communicates with it over the central HTTP `/management` endpoints (HTTP Basic), not over any cluster transport. It is no longer registered with ClusterClientReceptionist (that registration, and the whole ClusterClient surface, is gone).
+
+### Control-plane ownership, retry safety and message limits (2026-09-09)
+
+Both gRPC servers and every application client channel explicitly set maximum send and receive
+message sizes to **32 MiB** through `GrpcMessageLimits`. Audit push and reconciliation normally
+use **2 MiB** batches measured from the complete protobuf message fields, repeated-message
+length/tag prefixes and gRPC framing. The shared limits must move together on both peers.
+
+On the central control service, the PSK interceptor stores a verified site identity in the
+server call context. Before dispatch it compares that identity to notification source site,
+reconciliation site identifier, health report and heartbeat site, **each** audit row's source
+site and **both** halves of every cached telemetry packet. A mismatch or missing owner denies
+the request with `PermissionDenied` before any handler or partial batch runs. Execution and
+parent-execution linkage may cross sites; those identifiers are not ownership fields.
+Notification status queries carry the verified identity to the outbox, which returns no result
+for a notification stored under another site. An unrecognized protobuf request is denied until
+its ownership rule is explicitly implemented.
+
+`Unavailable` alone does not prove a command was unsent: it can represent a connection loss
+after a device write, deployment or script call ran. The site transport automatically retries
+only an explicit read-only command whitelist, or a server readiness rejection carrying the
+`scadabridge-dispatch: not-started` trailer emitted before actor dispatch. An ambiguous mutation
+returns `Aborted` with `OUTCOME_UNKNOWN`; the caller must inspect the operation before retrying.
+The provider selects the peer for subsequent independent calls, without replaying the failed
+mutation. A secondary attempt that loses its response has the same uncertain-outcome behavior.
+
+Site-to-central notification submission and audit ingestion can retry because their durable
+identifiers are idempotent; status reads and sequence-guarded health reports can retry as well.
+Reconciliation mints fetch tokens and only retries a proven pre-dispatch rejection. Attempts
+share the original absolute deadline. No transport retries `DeadlineExceeded` across nodes.
@@ -460,3 +460,7 @@ The `AuditLog` table is append-only and grows by every script-trust-boundary eve
 - **Central UI**: Uses `ICentralUiRepository` for read-oriented queries across domain areas, including config-audit queries for the Configuration Audit Log Viewer (the operational Audit Log page is owned by #23).
 - **All central components that modify state**: Call `IAuditService.LogAsync()` after successful operations to record audit entries within the same transaction.
 - **Host**: Provides database connection configuration. Registers DbContext, repository implementations, and `IAuditService` implementation in the DI container. Triggers auto-migration in development or validates schema version in production.
+
+### Configuration command transactions (2026-09-09)
+
+`ConfigurationCommandTransaction` creates a fresh dependency scope and EF context for each execution-strategy attempt. Entity changes and staged audit entries share one transaction, including intermediate saves needed for generated keys. A failure before commit rolls back both. An uncertain commit is not replayed, preventing duplicate creates; the caller must reconcile its result. Postcommit cache actions execute while their dependency scope remains alive; failures are logged without replaying the committed operation.
@@ -304,7 +304,7 @@ The `AlarmTransitionCallback` delivers a protocol-neutral `NativeAlarmTransition

 ### Protocol Adapters

- **OPC UA** (`OpcUaDataConnection` + `RealOpcUaClient`): a single **event MonitoredItem** (`AttributeId = EventNotifier`) on the Server object, with an `EventFilter` selecting `EventType` / `SourceNode` / `Severity` plus the `ConditionType` / `AcknowledgeableConditionType` / `AlarmConditionType` state fields. `ConditionRefresh` is invoked on subscribe to replay active conditions as the snapshot. The OPC UA field → `NativeAlarmTransition` mapping is isolated in the pure helper `OpcUaAlarmMapper`, unit-testable without a live server.
+- **OPC UA** (`OpcUaDataConnection` + `RealOpcUaClient`): one **event MonitoredItem per source binding** (`AttributeId = EventNotifier`), sharing an event subscription (the empty binding selects the Server object), with an `EventFilter` selecting `EventType` / `SourceNode` / `Severity` plus the `ConditionType` / `AcknowledgeableConditionType` / `AlarmConditionType` state fields. `ConditionRefresh` is invoked on subscribe to replay active conditions as the snapshot. The OPC UA field → `NativeAlarmTransition` mapping is isolated in the pure helper `OpcUaAlarmMapper`, unit-testable without a live server.
 - **MxGateway** (`MxGatewayDataConnection` + `RealMxGatewayClient`): mirrors over the gateway package's `StreamAlarmsAsync` — a resumable background stream whose reconnect re-sends a snapshot. The field mapping lives in `MxGatewayAlarmMapper`.

 Other/custom protocols do not implement the capability; a subscribe request against such a connection is replied to with a failure (`SubscribeAlarmsResponse.Success = false`).
@@ -315,11 +315,11 @@ The `DataConnectionActor` opens **one alarm feed per source** (not per subscribe

 The registered `SourceReference` is a **NodeId** for OPC UA (the picker, CSV and config all store NodeIds), so the transition's routing identity must live in the same space or nothing matches. **Gitea #17**: the OPC UA adapter previously set `SourceObjectReference` from the event's `SourceName` (a plain name — the RawPath for OtOpcUa), which never prefix-matches a NodeId binding, so every native OPC UA alarm transition was silently dropped. Because each OPC UA feed is opened for exactly one binding, the adapter now tags every transition on that feed with the **binding string verbatim** (via the pure `OpcUaAlarmMapper.BuildIdentity`), making the routing key an exact match independent of whether the binding is stored as `ns=<index>` or the durable `nsu=<uri>` form. The Server-object aggregate feed (empty binding) keeps an empty routing identity, so it reaches only "mirror everything" subscribers and never leaks into a specific-node binding. The per-condition `SourceReference` key stays the human-readable `SourceName.ConditionName`, so persistence and display are unchanged. MxGateway is unaffected — its bindings are object names and its mapper already emits matching names.

- **State gating**: `SubscribeAlarmsRequest` is handled only in the **Connected** state; requests arriving while **Connecting**/**Reconnecting** are stashed (standard Become/Stash) and processed on entering Connected.
+- **State gating**: `SubscribeAlarmsRequest` is handled only in the **Connected** state; requests arriving while **Connecting**/**Reconnecting** are coalesced to the latest request per subscriber actor/source and processed on entering Connected. Unsubscribes and subscriber termination are handled in every state.
 - **Capability check**: if `_adapter is not IAlarmSubscribableConnection`, the actor replies `SubscribeAlarmsResponse(Success = false, ...)`.
 - **Reconnect handling**: on entering **Reconnecting**, the actor pushes a `NativeAlarmSourceUnavailable` to every alarm subscriber (consumers mark mirrored alarms uncertain rather than clearing them). On successful reconnection it re-subscribes the feed; the adapter re-emits a snapshot, reconciling state.
 - **Shared condition filter (last-subscriber-wins)**: because the feed is opened once per source, it carries a single condition filter. A second subscriber that registers a *different* filter for the same source overwrites it (last writer wins) and the actor logs a warning — co-subscribers to one source are expected to agree on the filter.
- **Routing index**: subscribed sources are bucketed by the **first path segment** of their reference (everything before the first `.`), and a transition is matched only against the bucket for its own first segment, plus a residue list of sources that carry no separator at all (a prefix shorter than one segment, which can match other buckets). This is sound because a source reference containing a separator can only prefix a reference sharing its entire first segment. The per-source `StartsWith` test and the condition-type gate inside the bucket are unchanged, so routing decisions are identical to the previous linear scan over every subscribed source; the `SnapshotComplete` sentinel still bypasses the index entirely and is broadcast to every subscriber.
+- **Routing index**: subscribed sources are bucketed by the **first path segment** of their reference (everything before the first `.`), and a transition is matched only against the bucket for its own first segment, plus a residue list of sources that carry no separator at all (a prefix shorter than one segment, which can match other buckets). This is sound because a source reference containing a separator can only prefix a reference sharing its entire first segment. The per-source `StartsWith` test and the condition-type gate inside the bucket are unchanged, so routing decisions are identical to the previous linear scan over every subscribed source; snapshot boundaries bypass the condition filter, but OPC UA boundaries are sent only to the exact source binding. Only legacy connection-wide Mx boundaries broadcast to all subscribers.
 - **Gateway union filter (MxGateway)**: `StreamAlarms` carries a single `alarm_filter_prefix`, so the adapter opens the stream on the **longest common prefix** of the currently subscribed source references and restarts it only when a NEW source falls outside that prefix (the source then replays a fresh snapshot, which `NativeAlarmActor` already handles). An unsubscribe never restarts the stream — the prefix it leaves behind is at worst too broad, which costs bandwidth, not correctness. The prefix is a bandwidth optimisation only; the actor's per-source + condition-type gate remains authoritative, mirroring the OPC UA server-side WhereClause stance.

 ### Protocol-Neutral Types & Messages
@@ -330,7 +330,7 @@ All defined in Commons so the feed is identical across protocols:
 |------|-------|
 | `NativeAlarmTransition` | `SourceReference`, `SourceObjectReference`, `AlarmTypeName`, `Kind`, `Condition`, `Category`, `Description`, `Message`, `OperatorUser`, `OperatorComment`, `OriginalRaiseTime?`, `TransitionTime`, `CurrentValue`, `LimitValue`, `AckTime?` |
 | `AlarmConditionState` | `Active`, `Acknowledged`, `Confirmed?` (null when not confirmable), `Shelve`, `Suppressed`, `Severity` (0–1000) |
-| `AlarmTransitionKind` (enum) | `Snapshot`, `SnapshotComplete`, `Raise`, `Acknowledge`, `Clear`, `Retrigger`, `StateChange` |
+| `AlarmTransitionKind` (enum) | `Snapshot`, `SnapshotComplete`, `Raise`, `Acknowledge`, `Clear`, `Retrigger`, `StateChange`, `SnapshotStart` (appended) |

 `OperatorUser` / `OperatorComment` and `CurrentValue` / `LimitValue` are display-only mirrors from the source.

@@ -377,11 +377,15 @@ Each value update delivered to an Instance Actor includes:
 >
 > **Stop-and-recreate was rejected, and the reason is worth keeping.** Subscriptions are held inside `DataConnectionActor`, and `InstanceActor` cancels its per-connection subscribe-retry timer on the first success — so a rebuilt connection actor would have *no* subscribers and nothing would ever re-subscribe. That trades "stale endpoint, data flowing" for "correct endpoint, no data until redeploy". See *In-place reconfiguration* below for what the actor does instead.

-Each data connection is managed by a dedicated connection actor that uses the Akka.NET **Become/Stash** pattern to model its lifecycle as a state machine:
+Each data connection is managed by a dedicated connection actor that uses the Akka.NET **Become** pattern to model its lifecycle as a state machine:

- **Connecting**: The actor attempts to establish the connection. Subscription requests and write commands received during this phase are **stashed** (buffered in the actor's stash).
- **Connected**: The actor is actively servicing subscriptions. On entering this state, all stashed messages are unstashed and processed.
- **Reconnecting**: The connection was lost. The actor transitions back to a connecting-like state, stashing new requests while it retries.
+- **Connecting**: The actor attempts to establish the connection. It retains only the latest tag-subscribe request per instance and alarm-subscribe request per subscriber/source. Single and batch device writes receive an immediate failed response; they are never retained.
+- **Connected**: The actor actively services subscriptions and processes the retained latest subscription requests.
+- **Reconnecting**: The connection was lost. Existing subscribers receive bad quality, the latest subscription requests are coalesced while the actor retries, and single/batch writes fail immediately. A caller timeout never leaves a queued setpoint to replay on recovery.
+
+Instance Actors and NativeAlarmActors send explicit unsubscribe requests from `PostStop`, with their own actor references as sender. DCL also watches every subscriber. A stopped actor releases only its own registrations; a delayed unsubscribe or termination from an old actor cannot remove the replacement actor with the same instance name. A narrower replacement drops tags it no longer requests. Completion of outstanding device subscribe/resolution I/O adopts handles only for tags still requested and releases orphan handles. Existing identical Mx handles are never released as redundant, and adapter reverse-map removals compare the handle atomically before removing a tag mapping.
+
+OPC UA refreshes carry an exact `SnapshotSourceReference` (including the empty Server binding) and a `SnapshotId` shared by `SnapshotStart`, entries and `SnapshotComplete`. Each NativeAlarmActor replaces its buffer on a new start and commits only a matching end; duplicate or superseded ends cannot clear a committed snapshot. The adapter tags each monitored item's refresh separately even though ConditionRefresh targets the shared event subscription. Mx snapshots retain their connection-wide framing. An interrupted refresh buffer is discarded on disconnect.

 This pattern ensures no messages are lost during connection transitions and is the standard Akka.NET approach for actors with I/O lifecycle dependencies.

@@ -309,3 +309,7 @@ The ManagementActor receives the following services and repositories via DI (inj
 - **Communication Layer**: Deployment commands and remote site queries flow through communication actors.
 - **Configuration Database (via IAuditService)**: All configuration changes are audited.
 - **Security & Auth**: The ManagementActor enforces authorization using user identity passed in messages. For HTTP API access, the Management endpoint authenticates the user via LDAP and resolves roles before dispatching to the ManagementActor.
+
+### Atomic configuration command boundary (2026-09-09)
+
+Database-only configuration commands may execute through `ExecuteConfigurationCommandAsync` from the actor or Central UI. The boundary requires role authorization and `IConfigurationCommandTransaction`; missing transaction configuration fails closed. Repository mutations may flush generated keys within the transaction; a final flush persists the audit before commit. Existing service-owned audits are retained. Site cache refresh and inbound API handler publication run only after a successful commit.
@@ -199,3 +199,12 @@ Delivery max-retry-count and retry interval are not part of `NotificationOutboxO
 - **Central UI**: Queries the `Notifications` table for the Notification Outbox page and issues operator Retry/Discard actions on parked notifications.
 - **Health Monitoring**: Polls the outbox for KPI tiles on the health dashboard.
 - **KPI History (#26)**: Emits `IKpiSampleSource` (`NotificationOutboxKpiSampleSource`, Global + per-Site + per-Node) consumed by the KpiHistory recorder (#26), reusing the existing `Compute…KpisAsync` reads; the resulting `queueDepth` / `stuckCount` / `parkedCount` / `deliveredLastInterval` / `oldestPendingAgeSeconds` series render as trends on the Notification Outbox page via `KpiTrendChart`. See [Component-KpiHistory.md](Component-KpiHistory.md).
+
+### Site-scoped delivery-status queries (2026-09-09)
+
+The central gRPC ingress adds its credential-verified site identity to the internal status
+query; the site cannot supply or override this field on the wire. The outbox compares it to
+the stored notification's `SourceSiteId` before returning delivery state or error details.
+A foreign or unknown notification returns the same `Found = false` response. Existing trusted
+in-process central callers may omit the site constraint. Notification ingest remains
+idempotent on `NotificationId`; this does not imply row-level dispatch claiming.
@@ -38,6 +38,8 @@ The following namespace/type prefixes are forbidden in all scripts:
 | `System.Reflection` | Reflection — forbidden entirely |
 | `System.Net` | Raw network access — forbidden entirely (scripts must use `ExternalSystem.Call`) |
 | `System.Runtime.InteropServices` | Native interop — forbidden entirely |
+| `System.Runtime.CompilerServices.Unsafe`, `System.Runtime.CompilerServices.RuntimeHelpers` | Type-punning and constructor/runtime bypass — forbidden types; compiler attributes and async support remain allowed |
+| `System.Runtime.Loader` | Assembly loading and load-context control — forbidden entirely |
 | `Microsoft.Win32` | Win32 API access — forbidden entirely |
 | `System.Environment` | Whole type — forbidden entirely; exposes process control (`Exit`/`FailFast`), the host environment, and secrets via environment variables (e.g. `SCADABRIDGE_API_KEY`). Scripts that need a line break use `"\n"` instead of `Environment.NewLine`. |
 | `System.GC` | Whole type — forbidden entirely; GC control (`Collect`, `KeepAlive`, memory-pressure knobs) has no legitimate script use. |
@@ -51,12 +51,12 @@ Set in a local or docker-dev environment via the environment variable `ScadaBrid
 > intent and is the mechanism for any non-cookie bearer surface (e.g. `/auth/token`), but
 > it is **not** the transport for the cookie principal.

- **Idle timeout**: Configurable, default **30 minutes**. If no requests are made within the idle window, the session is rejected and the user must re-login. Tracked via a `LastActivity` last-activity timestamp claim. The cookie's `ExpireTimeSpan` is set to the idle timeout and `SlidingExpiration` renews it on activity, so the cookie window and the explicit `OnValidatePrincipal` idle check use the **same** value and cannot contradict each other.
- **Role-mapping refresh (LDAP-free)**: Configurable, default **15 minutes** (`SecurityOptions.RoleRefreshThresholdMinutes`). At login the session stores the user's raw LDAP groups (one `zb:group` claim each) plus a `zb:lastrolerefresh` anchor. Once the anchor is older than the threshold, `OnValidatePrincipal` re-runs the **DB-backed** `RoleMapper` on the stored groups — **with no LDAP call** — rebuilds the role/scope claims via the shared claim-builder, advances the anchor, and re-issues the cookie. Central role-mapping (DB) changes — including a **revoked** mapping that drops the user's roles, and changed site-scope rules — take effect within this window. Roles derived from central mappings are never more than ~15 minutes stale.
+- **Idle timeout**: Configurable, default **30 minutes**. Without eligible activity within the idle window, the session is rejected and the user must re-login. Passive `/auth/ping`, activity prevalidation, and Blazor transport requests preserve the activity anchor and suppress sliding renewal; the activity endpoint renews only after authentication and antiforgery validation. Tracked via a `LastActivity` last-activity timestamp claim. The cookie's `ExpireTimeSpan` is set to the idle timeout and `SlidingExpiration` renews it on activity, so the cookie window and the explicit `OnValidatePrincipal` idle check use the **same** value and cannot contradict each other.
+- **Role-mapping refresh (LDAP-free)**: Configurable, default **15 minutes** (`SecurityOptions.RoleRefreshThresholdMinutes`). At login the session stores the user's raw LDAP groups (one `zb:group` claim each) plus a `zb:lastrolerefresh` anchor. Once the anchor is older than the threshold, `OnValidatePrincipal` re-runs the **DB-backed** `RoleMapper` on the stored groups — **with no LDAP call** — rebuilds the role/scope claims via the shared claim-builder, advances the anchor, and re-issues the cookie. Central role-mapping (DB) changes — including a **revoked** mapping that drops the user's roles, and changed site-scope rules — take effect within this window. Ordinary HTTP validation uses this threshold. A live circuit refreshes mappings in fresh scopes on authentication-state reads, and mutation entrypoints require a successful current mapping before dispatch.

 #### Session Validation (`OnValidatePrincipal`)
 - The cookie principal is built at login by a **single shared claim-builder** (`SessionClaimBuilder`). The `OnValidatePrincipal` role-refresh path rebuilds the principal through the **same** builder, so the login and refresh claim shapes cannot drift.
- **Failure policy**: the refresh is best-effort. Any error during the refresh (e.g. the configuration database is unreachable) **keeps the existing principal with its current roles** — it never signs the user out and never throws out of the request pipeline. This mirrors the **Active sessions** stance under *LDAP Connection Failure* below. Only the explicit idle-timeout path rejects the principal.
+- **Failure policy**: ordinary cookie validation and display-state refresh retain existing grants on a mapping lookup error. Authoritative circuit mutation checks fail closed on the same error, so a mutation cannot proceed using grants that could not be refreshed. Idle timeout rejects the principal.

 > **Residual limitation — live LDAP group-membership changes (follow-up).** The
 > mid-session refresh re-maps the **stored** groups against the central database; it does
@@ -66,7 +66,7 @@ Set in a local or docker-dev environment via the environment variable `ScadaBrid
 > shared `ZB.MOM.WW.Auth.Ldap` library, which is an **external NuGet package** and exposes
 > only `AuthenticateAsync(username, password, ct)` (no standalone group search). Adding
 > that method is tracked as a follow-up. Until then: central role-mapping/scope changes are
-> reflected within ~15 minutes; directory group-membership changes require re-login.
+> reflected during HTTP refresh and immediately at circuit mutation checks; directory group-membership changes require re-login.

 ### Load Balancer Compatibility
 - The authentication cookie carries a self-contained JWT — no server-side session state. A load balancer in front of the central cluster can route requests to either node without sticky sessions or a shared session store.
@@ -213,3 +213,7 @@ To blunt online password-spray and brute-force against the directory, every LDAP
 - **Proxy topologies and the username-lockout DoS.** The key is deliberately `{username}|{IP}`, not `{username}` alone: keying on username alone would let any network-adjacent actor lock any operator out of a SCADA control surface with five wrong passwords, repeatable indefinitely. That isolation only exists when the throttle sees real client IPs — behind a proxy **without** ForwardedHeaders trust, all clients collapse onto the proxy's IP and the DoS returns. Mitigations: (a) the shipped Traefik topologies enable trusted-proxy ForwardedHeaders (so per-IP isolation is real in the documented deployment); (b) the lockout window is short by default (`LoginLockoutMinutes` = 5) and never touches the directory account itself; (c) residual risk — an attacker spraying from their *own* IP locking out a victim username *at that IP only* — is the throttle working as designed. A success-path bypass ("a correct password unlocks") was considered and rejected: it would let an attacker who has the password bypass the lockout entirely, and it converts the throttle into a password oracle during the lockout window.
 - **Behaviour when locked**: the surface refuses the bind without contacting LDAP. The Basic-Auth surfaces (`ManagementAuthenticator`) and `POST /auth/token` return HTTP `429` with `code = "AUTH_THROTTLED"`; `POST /auth/login` redirects to `/login` with a "Too many failed attempts. Try again later." message.
 - **Best-effort, per-node**: the throttle is in-memory with no external state. The two central nodes maintain independent counters (a spray hitting both simply burns its budget on each). Memory is bounded — writes prune expired keys and cap the tracked-key count. This is a rate-limit guard, not an account-lockout policy; it never disables the directory account itself.
+
+### Passive traffic and circuit refresh (2026-09-09)
+
+`/auth/ping`, `/auth/activity`, and Blazor transport validation do not advance inactivity timestamps or trigger sliding renewal. The explicit activity endpoint validates antiforgery before renewing the cookie and recording activity by protected session ID. A bounded process-local tracker lets a live circuit observe successful activity renewal. Role refresh preserves the session ID and activity timestamp; current circuit mutation checks re-map stored groups with fail-closed lookup behavior.
@@ -148,6 +148,8 @@ and the mailbox supplies the memory barrier between consecutive passes. The
 daily terminal-row purge uses the same shape. (`NotificationOutboxActor`'s
 dispatch sweep is the in-repo reference.)

+**Stored site ownership is immutable.** Authenticated control-plane requests must claim their verified site in both audit and operational halves. An existing `TrackedOperationId` cannot be advanced by another site even when the incoming packet claims its own site consistently: the SQL upsert rejects the owner mismatch before its monotonic update, including the duplicate-insert retry path. The update predicate independently requires an exact owner match. A rejection rolls back the paired audit/operational ingest transaction and produces no acknowledgment.
+
 **The central upsert is one round trip, INSERT-first.**
 `SiteCallAuditRepository.UpsertAsync` ships both statements in a single command
 text — `IF NOT EXISTS … INSERT;` then the monotonic `UPDATE` — so a packet costs
@@ -216,7 +216,7 @@ flowchart TD
 - **Alarm state changes**: `[InstanceUniqueName].[AlarmName]`, alarm state (active/normal), priority, timestamp.

 ### GetAttribute / SetAttribute
- **GetAttribute**: Returns the current in-memory value for the requested attribute.
+- **GetAttribute**: Returns the current in-memory value for the requested attribute. Attribute read/write/wait helper continuations do not capture the calling scheduler (`ConfigureAwait(false)`), so synchronous `Attributes[key]` access can complete even when every script worker is blocked. AsyncLocal trace attribution still flows with execution context.
 - **SetAttribute** (for attributes with data source reference): Sends a write request to the Data Connection Layer. The DCL writes to the physical device. If the write fails (connection down, device rejection, timeout), the error is returned synchronously to the calling script for handling. On success, the existing subscription picks up the confirmed value from the device and sends it back as a value update, which then updates the in-memory value. The in-memory value is **not** optimistically updated.
 - **SetAttribute** (for static attributes): Updates the in-memory value and **persists the override to local SQLite**. On restart (or failover), the Instance Actor loads persisted overrides on top of the deployed configuration, preserving runtime-modified values. A redeployment of the instance resets all persisted overrides to the new deployed configuration values.

@@ -379,6 +379,8 @@ that owns the Instance Actor, and central reads the linkage from audit rows.
 - **`NativeAlarmSourceUnavailable`**: the source connection has dropped. The actor **retains its last-known mirrored conditions** but marks them uncertain rather than purging them, so a transient disconnect does not flap every condition to normal. The set is reconciled against truth by the next reconnect snapshot.

 ### Transition Handling (`NativeAlarmTransitionUpdate`)
+- **Owned subscription cleanup**: Instance Actors send tag unsubscribes for each bound connection in `PostStop`; Native Alarm Actors send their alarm-source unsubscribe with their own actor identity. DCL DeathWatch provides cleanup if an owner terminates without a graceful message. Late cleanup from an old actor cannot remove its same-name replacement.
+- **OPC UA refresh scope**: snapshot start, entries and completion preserve the exact source binding and refresh epoch. Native Alarm Actors ignore unrelated feeds and mismatched/duplicate completion epochs, and discard a partial refresh on source disconnect. MxGateway keeps its connection-wide snapshot framing.
 - **Snapshot / SnapshotComplete (reconnect reconciliation)**: `Snapshot` updates buffer into a staging set; `SnapshotComplete` performs an **atomic swap** of the mirrored set with the staged set. Any condition that was previously mirrored but is **not present** in the new snapshot emits a return-to-normal `AlarmStateChanged` and drops out. This is how the mirror self-corrects after an outage.
 - **Live transitions** (`Raise` / `Ack` / `Clear` / `Retrigger` / `StateChange`): upsert the condition by `SourceReference`. Updates carrying a `TransitionTime` **older** than the currently held transition are ignored (out-of-order protection). Accepted transitions persist to SQLite and emit an enriched `AlarmStateChanged` upward to the Instance Actor.
 - **Retention**: a mirrored condition is dropped once it is both inactive **and** acknowledged (`!Active && Acknowledged`) — the alarm has fully run its course at the source and no longer needs mirroring. The drop emits a final state change and deletes the SQLite row.
@@ -499,7 +501,7 @@ Available to all script runs and alarm on-trigger runs:
 - `Notify.Status("trackedOperationId")` — A thin alias of `Tracking.Status(id)` retained for the notification domain. Returns a status record (status, retry count, last error, key timestamps). While the notification is still in the site store-and-forward buffer the site answers locally (status `Forwarding`); once forwarded the query round-trips to central.

 ### Database Access
- `Database.Connection("connectionName")` — Obtain a raw MS SQL client connection (ADO.NET) for synchronous read/write.
+- `Database.Connection("connectionName")` — Obtain an audited ADO.NET connection for read/write. Commands and synchronous/asynchronous transactions retain the auditing connection on public readback; provider transactions are unwrapped only internally for command assignment. Each command execution emits one audit event, including commands reached through `transaction.Connection` or `command.Connection`. Commit, rollback, savepoints and disposal retain provider semantics.
 - `Database.CachedWrite("connectionName", "sql", parameters)` — Submit a write operation for store-and-forward delivery. Returns a `TrackedOperationId` tracking handle immediately; pass it to `Tracking.Status(id)` to observe delivery progress.

 ### Operation Tracking