docs: arch-review remediation — component docs sweep, execution log, residuals register

Final consistency sweep per plan §6: verified component docs against shipped
WP1-WP3 + adversarial-review-fix state, corrected drift found in SiteRuntime
(recursion-exempt run cap, stale ScriptExecutionActor/AlarmExecutionActor
references), TemplateEngine (BundleImporter watermark path), DeploymentManager
(phase-2 PendingDeployment staging), CentralUI (shared KPI cache, dedup'd alarm
poll, render coalescing), StoreAndForward (rate-limited drop logging), and
ConfigurationDatabase (documented DbContext-pooling non-adoption). Updated the
docs/components/ developer-reference set (SiteRuntime, SiteEventLogging,
InboundAPI) to drop the deleted per-run actor classes. Amended one known-issue
for the superseding MaxBatchSize:64 read-page pin. Added CLAUDE.md bullets for
stream graceful-completion reconnect, the required site audit DB path, honest
CLI HTTP timeouts, bulk DeploySiteAsync, and LocalDb 0.2.1. New execution log
records the phase→commit map, gate results, adversarial-review tally, the
three test-flake root causes, and the nine-item residuals register.
This commit is contained in:
Joseph Doherty
2026-08-15 01:30:59 -04:00
parent a9ca51e008
commit 7804fe7958
12 changed files with 244 additions and 49 deletions
+5 -4
View File
@@ -227,6 +227,7 @@ When the Instance Actor is stopped (due to disable, delete, or redeployment), Ak
- **Pool sizing is instance-scaled and grow-only (WP3.1).** The pool was a fixed 8 threads regardless of load. It is now `clamp(max(ScriptExecutionThreadCount, ceil(enabledInstances / 8)), 1, ScriptExecutionMaxThreadCount)` — the existing `ScriptExecutionThreadCount` (default 8) becomes the **floor**, so configurations at or below 64 instances behave exactly as before, and the new `ScriptExecutionMaxThreadCount` (default 32) is the ceiling. Beyond the ceiling the per-script cap below is the real regulator. `DeploymentManagerActor.UpdateInstanceCounts` calls `EnsureCapacity` on every deploy / undeploy / enable / disable and once per staggered startup batch. Growth only: undeploying leaves idle threads, which cost nothing measurable and avoid drain/steal complexity.
- **The deadline is armed at enqueue, not at dequeue (WP3.1).** The run's timeout `CancellationTokenSource` is created on the actor thread *before* the body is queued, so queue wait consumes the script's own budget. A body that dequeues past its deadline **skips execution entirely** and takes the existing timeout path (site event, script-error counter, error reply, completion message): a saturated pool sheds stale work instead of running it late with a fresh full budget.
- **Concurrent runs per script are capped (WP3.1).** `MaxConcurrentRunsPerScript` (default 4) bounds runs in flight — queued or executing — for any one script or alarm on-trigger script. Over the cap the **newest** run is shed: the four already in flight are closest to their own deadlines and already charged against them, so nothing is ever reordered and no extra queue is needed (the scheduler's FIFO already is the queue). A shed increments `ISiteHealthCollector.IncrementScriptRunShed` (surfaced as `ScriptRunShedCount`), emits a `script`/`Warning` site event **rate-limited to one per script per minute** so a hot trigger cannot flood `site_events`, and — for an Ask-based `CallScript` — replies with an explicit error so a nested call or inbound-API route fails fast instead of hanging.
- **The cap gates only depth-0 launches (review fix).** A trigger fire or a depth-0 `CallScript`/inbound-API route counts against `MaxConcurrentRunsPerScript`; a *nested* `Instance.CallScript` self-recursion (`callDepth > 0`) is exempt — it is already bounded by `MaxScriptCallDepth` instead, and the calling run is holding one of the four cap slots itself while it awaits the callee, so counting the nested launch too would spuriously shed legitimate self-recursion.
### Handling `Instance.CallScript`
- When an external caller (another script run, an alarm on-trigger run, or a routed call from the Inbound API) sends a `CallScript` message to the Script Actor, it launches a run to handle the call.
@@ -252,7 +253,7 @@ When the Instance Actor is stopped (due to disable, delete, or redeployment), Ak
- **Expression** trigger evaluation runs on the shared .NET thread pool behind a process-wide concurrency gate (`TriggerEvalGate`, sized by `TriggerEvalMaxConcurrency`, default `max(2, ProcessorCount)`) — **not** on the bounded script-execution pool. This is the WP3.1 fix for arch-review finding #4 (High): when evaluation shared that pool, N script bodies blocked in synchronous I/O stalled *every* Expression trigger on the node — scripts and alarms alike — for an unbounded time, and the evaluation's own 2 s timeout was constructed inside the queued body, so it did not start ticking until dequeue. An alarm that should have raised in milliseconds simply never raised, with neither a raise nor a timeout visible to the operator. Trigger expressions are non-blocking **by construction** (`TriggerExpressionGlobals` exposes only reads over an in-memory snapshot, and the script trust gate has already denied I/O, network, threading, and reflection), so the shared pool is where they belong; a second dedicated pool was considered and rejected as adding threads, gauges, and a second starvation surface for no isolation gain. The evaluation deadline (`TriggerEvalTimeoutSeconds`, default 2, previously hardcoded) is now armed **at enqueue**, so gate-wait time burns the same budget and a saturated gate yields a timely `false` rather than an unbounded stall. Per-actor coalescing (one evaluation in flight, one pending) is unchanged and caps waiters at one per Expression trigger, so the gate queue is bounded by trigger count.
- For binary trigger types (ValueMatch / RangeViolation / RateOfChange), when the condition is met and the alarm is currently in **normal** state, the alarm transitions to **active**:
- Updates the alarm state on the parent Instance Actor (which publishes to the Akka stream).
- If an on-trigger script is defined, spawns an Alarm Execution Actor to execute it.
- If an on-trigger script is defined, launches its run via `ScriptRunLauncher` (see Alarm On-Trigger Run below) — no per-run child actor.
- When the condition clears and the alarm is in **active** state, the alarm transitions to **normal**.
- For HiLo triggers, the actor tracks the current `AlarmLevel` (None / Low / LowLow / High / HighHigh). Each level transition emits a fresh `AlarmStateChanged` with the new level and its priority; level escalations (e.g., High → HighHigh) and de-escalations (HighHigh → High) both produce events. The on-trigger script fires only on the Normal → non-None edge, not on escalations between alarm bands.
- No script execution on clear in any trigger type.
@@ -507,14 +508,14 @@ Per Akka.NET best practices, internal actor communication uses **Tell** (fire-an
**Ask** is reserved for system boundaries where a synchronous response is needed:
- **`Instance.CallScript()`**: Ask pattern from Script Execution Actor to sibling Script Actor. The caller needs the return value. Acceptable because script calls are infrequent relative to tag updates.
- **`Instance.CallScript()`**: Ask pattern from the running script (via `ScriptRuntimeContext.CallScript`) to sibling Script Actor. The caller needs the return value. Acceptable because script calls are infrequent relative to tag updates.
- **`Route.To().Call()`**: Ask from Inbound API to site Instance Actor via Communication Layer. External caller needs a response.
- **Debug view snapshot**: Ask from Communication Layer to Instance Actor for initial state.
## Concurrency & Serialization
- The Instance Actor processes messages **sequentially** (standard Akka actor model). This means `SetAttribute` calls from concurrent Script Execution Actors are serialized at the Instance Actor, preventing race conditions on attribute state.
- Script Execution Actors may run concurrently, but all state mutations (attribute reads/writes, alarm state updates) are mediated through the parent Instance Actor's message queue.
- The Instance Actor processes messages **sequentially** (standard Akka actor model). This means `SetAttribute` calls from concurrent script runs are serialized at the Instance Actor, preventing race conditions on attribute state.
- Script runs (launched via `ScriptRunLauncher`, no per-run child actor) may run concurrently, but all state mutations (attribute reads/writes, alarm state updates) are mediated through the parent Instance Actor's message queue.
- External side effects (external system calls, notifications, database writes) are not serialized — concurrent scripts may produce interleaved side effects. This is acceptable because each side effect is independent.
## SiteStreamManager and gRPC Integration