Resolve all 33 medium-severity architecture review findings

Second slice of the 2026-09-09 whole-system review remediation
(docs/plans/2026-09-09-arch-review.md §10), on top of the High slice 146a1323.

Runtime/devices: SR-2 alarm on-trigger integration scope; SR-3 correlated NativeAlarmActor
subscribe retry; SR-4 immutable audited DB target; SR-5 ordered InstanceActor stop/flush;
SR-12 rehydration fence; SR-13 S&F gate recheck per delivery; DC-4 no OPC UA security
downgrade; DC-5 alarm snapshot for every new subscriber; ORCH-1 sync-dispose joins the pump.

Management/UI: MG-2 site scope on audit tree + Flow Debugger Replay; MG-3 stale-probe memo
removed; MG-4 server-owned bulk-deploy deadline; UI-5 sandbox runs in a killable child
process (new CentralUI.SandboxWorker project); UI-14 bounded sandbox output/IPC.

Edge/audit: EI-1 relay audit ownership; EI-2 SkipBodyCapture suppresses error text; EI-3/EI-11
error-field redaction + caps; EI-4 single relay deadline, pre-execution-only retry; EI-5
authenticated relay provenance; EI-10 body cap before audit read.

Persistence/contracts: DL-1 global Instance.UniqueName unique index + conflict translation;
DL-2 DataConnection protocol config encrypted at rest (migration
20260909203046_EnforceGlobalInstanceNamesAndProtectProtocolConfiguration — coordinated
central upgrade, old binaries cannot read protected rows); DL-3 bounded Replay paging;
DL-6 cross-site Overwrite of a deployed instance rejected; XC-4 proto descriptor contract test.

Config/hygiene: XC-1 central listener isolation by local port; XC-2/XC-3 inert SMTP + cron
keys removed from rig settings; CM-2 env2 bootstrap guard enabled; DC-6 deploy site exit code;
DC-7 notifier retries only explicit standby rejection; XC-5 ClusterClient purged from docs.

Gates recorded in the report: build 0/0, 9 397 passed / 0 failed, rig redeployed on all
eight nodes with the migration applied. 46 Low findings remain open.
This commit is contained in:
Joseph Doherty
2026-09-09 22:05:30 -04:00
parent 146a13238f
commit 9a87d7a328
136 changed files with 5441 additions and 648 deletions
+10 -1
View File
@@ -506,7 +506,7 @@ Available to all script runs and alarm on-trigger runs:
### Operation Tracking
- `Tracking.Status("trackedOperationId")` — Returns a status record (status, retry count, last error, key timestamps) for any tracked operation: a cached external system call, a cached database write, or a notification. For cached calls and writes the answer is always site-local and authoritative — the site owns the operation tracking table. (`Notify.Status(...)` is a thin alias scoped to the notification domain.)
- **Cached-call telemetry is off the script thread (2026-09-06).** A `Database.CachedWrite` / `ExternalSystem.CachedCall` emits three `CachedCallTelemetry` packets (submit, attempted, resolve) through `CachedCallTelemetryForwarder`, each an audit row plus a tracking row. Both halves used to be awaited to commit on the script thread; the audit half was the real cost — `SqliteAuditWriter.WriteAsync` returns the batch completion, and the writer coalesces for `FlushIntervalMs` (50 ms) before committing — so a cached call paid ~160 ms of a 176 ms run for bookkeeping (measured on the rig; now 17 ms). The forwarder now **enqueues the audit row and observes its completion on a fault-logging continuation** (the same pattern every other script-path audit emission already used), and `OperationTrackingStore` is a **single-writer queue**: `RecordEnqueue/Attempt/TerminalAsync` return once the row is *queued*, and one background pump applies each drained batch in one SQLite transaction. Preserved: FIFO order across the three lifecycle rows; **read-your-writes** — `GetStatusAsync` (behind `Tracking.Status`, and the telemetry actor's status join) and `ReadChangedSinceAsync` (central's `PullSiteCalls`) wait for every write queued before them; timestamps are stamped when recorded, not when applied; `PurgeTerminalAsync` returns after the purge lands; `DisposeAsync` drains the queue (the sync `Dispose()` does not — host shutdown uses the async path). Accepted: a row queued but unapplied is lost on a hard crash (milliseconds), leaving the tracking row at its previous state until reconciled; the S&F buffer and audit rows are unaffected. The queue is bounded (8 192) with wait backpressure — it never drops. `FlushAsync()` is the explicit barrier for callers that go to the file directly. **Trap for new audit emitters on a script path:** never `await` `IAuditWriter.WriteAsync` there — it waits for the writer's flush, not the enqueue. Design + measurements: `docs/plans/2026-09-06-tracking-store-async-writes-design.md`.
- **Cached-call telemetry is off the script thread (2026-09-06).** A `Database.CachedWrite` / `ExternalSystem.CachedCall` emits three `CachedCallTelemetry` packets (submit, attempted, resolve) through `CachedCallTelemetryForwarder`, each an audit row plus a tracking row. Both halves used to be awaited to commit on the script thread; the audit half was the real cost — `SqliteAuditWriter.WriteAsync` returns the batch completion, and the writer coalesces for `FlushIntervalMs` (50 ms) before committing — so a cached call paid ~160 ms of a 176 ms run for bookkeeping (measured on the rig; now 17 ms). The forwarder now **enqueues the audit row and observes its completion on a fault-logging continuation** (the same pattern every other script-path audit emission already used), and `OperationTrackingStore` is a **single-writer queue**: `RecordEnqueue/Attempt/TerminalAsync` return once the row is *queued*, and one background pump applies each drained batch in one SQLite transaction. Preserved: FIFO order across the three lifecycle rows; **read-your-writes** — `GetStatusAsync` (behind `Tracking.Status`, and the telemetry actor's status join) and `ReadChangedSinceAsync` (central's `PullSiteCalls`) wait for every write queued before them; timestamps are stamped when recorded, not when applied; `PurgeTerminalAsync` returns after the purge lands; Both `DisposeAsync` and sync `Dispose()` drain the queue before closing the writer; host shutdown uses the async path. Accepted: a row queued but unapplied is lost on a hard crash (milliseconds), leaving the tracking row at its previous state until reconciled; the S&F buffer and audit rows are unaffected. The queue is bounded (8 192) with wait backpressure — it never drops. `FlushAsync()` is the explicit barrier for callers that go to the file directly. **Trap for new audit emitters on a script path:** never `await` `IAuditWriter.WriteAsync` there — it waits for the writer's flush, not the enqueue. Design + measurements: `docs/plans/2026-09-06-tracking-store-async-writes-design.md`.
### Alarm State (read-only)
- `Alarms.CurrentAsync()` — Returns a snapshot of the instance's **current** alarm conditions as `IReadOnlyList<ScriptAlarm>` (`Commons.Types.Scripts`), covering computed alarms and mirrored native (OPC UA A&C / MxAccess Gateway) conditions alike. Served by a **local Ask** (`GetAlarmSnapshotRequest`/`GetAlarmSnapshotResponse`) to the script's own Instance Actor — the same mechanism attribute reads use, never a cross-cluster hop.
@@ -637,3 +637,12 @@ Per Akka.NET best practices, internal actor communication uses **Tell** (fire-an
- **Communication Layer**: Receives commands from central, sends deployment results, serves debug view data.
- **Store-and-Forward Engine**: Scripts route cached writes, notifications, and external system calls here.
- **Health Monitoring**: Periodically reports error rate metrics.
### Medium architecture review lifecycle guarantees (2026-09-09)
- Alarm on-trigger runs resolve the same scoped integration services as ordinary scripts, including external systems, databases, notifications, operation tracking, auditing, and site/node provenance. The scope remains alive for the run and is disposed on every exit.
- Native alarm subscription retries retain the current correlation ID until its response deadline, reject stale replies, and back off to a bounded interval. Rehydration cannot overwrite a condition touched by a live event, including deletion, or a completed authoritative snapshot.
- Audited raw ADO connections expose no connection string and reject `ConnectionString` assignment and `ChangeDatabase`; their configured database name remains the audit target.
- Static override persistence runs in order off the actor dispatcher. Redeployment waits for both predecessor termination and successful final persistence acknowledgement before clearing old state; failed persistence fails the redeployment. At most four redeployment predecessor stops are in progress per deployment manager. Delete and disable retain the same-name stop barrier until the final flush and lifecycle persistence finish; an overlapping deploy/enable is rejected with a retryable busy diagnostic. Timer flushes keep coalescing while a previous SQLite write is pending.
- Synchronous operation-tracking disposal drains and joins its thread-pool writer before disposing SQLite resources, including when invoked under a synchronization context.