docs+rig(flow-debugger): step logs v1.4 specs, register rows, FlowLogDemo seed + verification step J

This commit is contained in:
Joseph Doherty
2026-09-09 12:56:39 -04:00
parent f174725557
commit 82739ec912
12 changed files with 221 additions and 12 deletions
+2 -1
View File
@@ -57,9 +57,10 @@ boundary:
| `POST /api/{method}` (Inbound API) | External | Inbound (invokes a script) | No (gap) |
| `Route.To(x).Call(y)` (Inbound API → site) | Script | Central → site | Yes — `ScriptTrace.RouteCall`, **always-on** (2026-09-05) |
| Site script run / attribute write / attribute wait under a **traced** run | Script | Site-local | Yes — `ScriptTrace.ScriptRun` / `AttributeWrite` / `AttributeWait`, **gated** by the `Trace` flag (Flow Debugger Live runs, `ApiMethod.TraceAllCalls`) |
| Script `Log.Debug/Info/Warn/Error(...)` — and the runtime's own Information+ entries during a **traced** run | Script / runtime | Site-local or central | Yes — `ScriptTrace.ScriptLog`, one row per line, **gated** by the same `Trace` flag (2026-09-09) |
| Central method-script `Database.QuerySingle/Query/Execute` (`InboundDatabaseHelper`) | Script | Outbound | Yes — `DbOutbound.DbWrite` via `ICentralAuditWriter` (2026-09-06; was an unaudited gap) — `Target` = connection name, `Actor` = `api:{method}`, the request's `ExecutionId`, `Extra` = `{op, rowsAffected\|rowsReturned}` |
**`AuditChannel.ScriptTrace` (Flow Debugger, design `docs/plans/2026-09-05-flow-debugger-design.md` §5.1).** Trace steps ARE audit events — no parallel store. One kind is always-on (`RouteCall`, a central direct write per `Route.To().Call`, carrying the child run's id in `Extra.childExecutionId`); the three site kinds are emitted only when the run's `Trace` flag is set, ride the ordinary site `IAuditWriter` → SQLite → telemetry → central path, share the run's `ExecutionId` / `ParentExecutionId`, and are additionally returned inline on the routed reply so a Live run does not wait on the 5 s telemetry cadence. The Flow Debugger's graph (`FlowStep` / `FlowGraphBuilder`, Commons) is a central-side projection of these rows — **and, since 2026-09-06, of the run's `ApiOutbound` / `DbOutbound` / `Notification` rows too** (plan `docs/plans/2026-09-06-flow-debugger-outbound-calls.md`): the outbound rows already sit in the execution tree under the run's `ExecutionId`, so the replay query (`GetTraceEventsAsync`) simply widens its channel filter to the four channels plus the inbound row, and a traced site run also passes the outbound rows it emits through the same inline trace collector as its `ScriptTrace` rows (same `AuditEvent` instance the writer receives — no duplicate construction; a collector that throws can neither fail the call nor skip the durable write). Rows written after the run returned (the S&F forwarder's `Forwarded` / `Attempted` / `CachedResolve`, the Notification Outbox's `NotifyDeliver`) are not inline — the page re-reads the tree for them.
**`AuditChannel.ScriptTrace` (Flow Debugger, design `docs/plans/2026-09-05-flow-debugger-design.md` §5.1).** Trace steps ARE audit events — no parallel store. One kind is always-on (`RouteCall`, a central direct write per `Route.To().Call`, carrying the child run's id in `Extra.childExecutionId`); the three site kinds are emitted only when the run's `Trace` flag is set, ride the ordinary site `IAuditWriter` → SQLite → telemetry → central path, share the run's `ExecutionId` / `ParentExecutionId`, and are additionally returned inline on the routed reply so a Live run does not wait on the 5 s telemetry cadence. The Flow Debugger's graph (`FlowStep` / `FlowGraphBuilder`, Commons) is a central-side projection of these rows — **and, since 2026-09-06, of the run's `ApiOutbound` / `DbOutbound` / `Notification` rows too** (plan `docs/plans/2026-09-06-flow-debugger-outbound-calls.md`): the outbound rows already sit in the execution tree under the run's `ExecutionId`, so the replay query (`GetTraceEventsAsync`) simply widens its channel filter to the four channels plus the inbound row, and a traced site run also passes the outbound rows it emits through the same inline trace collector as its `ScriptTrace` rows (same `AuditEvent` instance the writer receives — no duplicate construction; a collector that throws can neither fail the call nor skip the durable write). Rows written after the run returned (the S&F forwarder's `Forwarded` / `Attempted` / `CachedResolve`, the Notification Outbox's `NotifyDeliver`) are not inline — the page re-reads the tree for them. **Since 2026-09-09 the channel also carries `ScriptLog`** (design `docs/plans/2026-09-09-flow-debugger-step-logs-design.md` D1) — one row per line a traced script or the runtime logged, with `RequestSummary` = the message, `ErrorMessage` / `ErrorDetail` = the attached exception's message / `ToString()`, and `Extra` (`ScriptTraceExtra`) carrying `logLevel` (`debug|info|warn|error`), `logSource` (`script|runtime`), the owning `stepEventId` and a per-run `seq`. It is **never a step**: `FlowGraphBuilder` folds it into its owning step's `Logs` list, and the same rows carry the exception the failing step threw via the `errorDetail` field that already existed on `AuditDetails` (site `ScriptRun` rows and the central inbound row now populate it). Redaction, the 8 KB / 64 KB-on-error payload caps and `AuditLog:PerChannelRetentionDays:ScriptTrace` all cover the new kind unchanged — a log line is an ordinary audit row, not a side channel.
Out of scope — framework traffic is not audited:
+1
View File
@@ -219,6 +219,7 @@ Per-leaf alarm rendering (leaf nodes are individual conditions for native alarms
- **`/deployment/flow-debugger`** (nav: Deployment → Flow Debugger; design `docs/plans/2026-09-05-flow-debugger-design.md`, plan `docs/plans/2026-09-05-flow-debugger.md`). Runs one entry point with operator-supplied inputs and renders every step the run passes through as a **four-lane swimlane** (Central / Site / Tags / **External**) — request, route call, site script run, attribute writes, attribute waits, **outbound API / database / notification calls**, response — each node carrying its input, output, duration and outcome; a tabbed detail panel (Input · Output · Writes *or* Lifecycle · Timing, plus a *Where this ran* column) opens on the selected node with `Open in Debug View` (script entries, where the instance is known) and `Audit rows (n)` (→ `/audit/log?executionId=`) drill-ins.
- **Two modes.** *Live* is a real run — the method script executes on the active central node with tracing on, or a real attribute write goes through the routed-write path and the site's triggered run is polled back — audited like any other call. *Replay* executes nothing: it rebuilds the flow from an existing execution's `ScriptTrace` audit rows plus its `ApiOutbound` / `DbOutbound` / `Notification` rows (`?executionId=` drill-in from the Audit Log drawer's and the Execution Chain page's **Open in Flow Debugger** actions).
- **External lane (v1.1, plan `docs/plans/2026-09-06-flow-debugger-outbound-calls.md`).** Every `ExternalSystem.Call/CachedCall`, `Database.*` and `Notify.Send` a traced run makes is a node on the fourth lane — kinds `ApiCall` / `DbCall` / `Notify` (eyebrows `API CALL` / `DATABASE` / `NOTIFY`), title = the system.method / connection / list, a status pill (`Ok` / `Pending` / `Parked` / `Failed`) plus `sync` | `cached` | `notify`. **One node per operation:** a cached call's or notification's later lifecycle rows (Forwarded → Attempted → Delivered / Parked, written minutes later by the S&amp;F forwarder or the Notification Outbox) fold into the node (`FlowStep.Lifecycle`, a Lifecycle tab in the detail panel with *Delivered by*) instead of drawing four boxes. A sync call sits in the execution chain; a cached call or notification hangs off it on a dashed **async** connector, and a trigger cascade's spawn edge carries a **bolt badge on the connector's riser** — the edge itself keeps an ordinary arrowhead (the bolt was the arrowhead until 2026-09-08, and on the upward Tags→Site hop it read as a broken arrow). The spawn edge starts at the parent's **last write of the attribute the child's `triggerAttribute` names** (`FlowGraphBuilder`); only when the child names no attribute, or the parent never wrote it, does the timestamp rule (last non-async parent step at or before the child) decide — a burst of writes landing in one millisecond used to draw the cascade from the last of them rather than the one that fired it. A Live run returns only the rows the run itself emitted, so a graph left with `Pending` nodes **refreshes itself** through the Replay command every 5 s for up to 2 min (page constants), and a *Refresh outcomes* button re-reads on demand in either mode; the state chip stays *Completed* — pending outbound work is not a failed run. A refresh **merges** the re-read rows into the run's known rows by `EventId` (fresh copy wins) and rebuilds the graph from the union, so it only ever adds nodes: the site's outbound rows reach central's audit store by telemetry seconds after the run returned them inline, and a Replay read taken in between would otherwise erase an already-drawn node (observed on the rig 2026-09-06 with a resolved cached DB write). The union is exact because the inline rows are the very `AuditEvent` objects the site writer persists. **v1.2 (follow-ups plan `docs/plans/2026-09-06-flow-debugger-followups.md`, register row 111 closed):** a folded node carries its `CorrelationId` (`FlowStep.CorrelationId` — the `TrackedOperationId` / `NotificationId`), so the detail panel deep-links to **`/site-calls/report?correlationId=<id>`** / **`/notifications/report?correlationId=<id>`** (*Open in Site Calls* / *Open in Notification report*; both pages accept that query, pre-filter to the one row through an additive per-id query filter — `SiteCallQueryRequest.TrackedOperationIdFilter` / `NotificationOutboxQueryRequest.NotificationIdFilter` — show a removable *Operation* / *Notification* pill and auto-open the row's detail modal; note `NotificationId` is minted as a Guid in **`N` form** — no dashes — so the Notify link passes `N` and the report page normalises any Guid-shaped value to `N`, while `TrackedOperationId` is a `varchar(36)` `D` string and the Site Calls filter binds a `varchar` parameter for the PK seek), and a **`Parked` node offers Retry / Discard in place**: the same site-scope re-check, confirm dialog (Discard = danger) and central→site / Notification Outbox relay the report pages use (`IFlowDebugService.RetryOutboundAsync` / `DiscardOutboundAsync` → `CommunicationService`), then the page re-polls Replay (5 s / 2 min) until the node leaves `Parked`. **Prediction is async-safe:** `FlowDebugService.PredictMethodAsync` resolves route targets by a resolve-and-rewalk loop — its original blocking resolver deadlocked the whole Blazor circuit on the renderer sync context the moment a method routing to a site script was chosen (found 2026-09-06; pinned by a single-thread-pump unit test and the Playwright `FlowDebuggerTests`, which log in through the fixture and run a Live run of an ephemeral method in a real browser).
- **Step logs and exceptions (v1.4, 2026-09-09, design `docs/plans/2026-09-09-flow-debugger-step-logs-design.md`).** A traced run shows, **on the node that produced them**, the exception a failing step threw and every line the script (or the runtime on its behalf) logged. A log line is an ordinary audit row — `ScriptTrace` / `ScriptLog`, one per line — that `FlowGraphBuilder` folds into its owning step's `FlowStep.Logs` and **never draws as a node**; attribution is `Extra.stepEventId` when the emitting helper pre-minted its step id, else the execution's anchor row (`ScriptRun` on a site, `InboundRequest` / the debugger's synthesized `RouteCall` root centrally), else that execution's first step, else `FlowGraph.UnattachedLogLines`. Rendering: a node footer badge (`✕ n` errors, `! n` warnings, `≡ n` info, highest severity first, `data-test="flow-node-logs"`, hidden with no lines); a failed node's subtitle becomes `ExceptionType: message` (namespace stripped, innermost exception, generic-safe) fed by the new `FlowStep.ErrorDetail`; a **Log (n)** tab in the detail panel (`flow-step-tab-log`) carrying the full exception block first, then the lines under an `All | Warn +` filter with a copy button and a *"n more on &lt;step&gt;"* cross-step hint that selects the named step; and a `Log: n errors · n warnings · n info` count in the status strip (`flow-log-counts`) that counts LINES, not steps — a `Log.Error` line is not a failed step and never moves the state chip. Two truncation notes sit beside it: `flow-logs-truncated` (*inline log lines capped — refresh outcomes pulls the rest*, from `FlowDebugRunResult.LogsTruncated`; it clears once a refresh merges in more log rows, because Replay reads the complete set) and `flow-unattached-logs`. Live and Replay read the same rows, so the two modes render identically once telemetry has landed.
- **Authorization (arch-review F3).** The page is `RequireFlowDebugger` — `AuthorizationPolicies.FlowDebuggerAccessRoles` = Deployer, Administrator, Viewer — so an auditor can open it and Replay a past execution without the ability to run anything, and `FlowDebugReplayCommand` accepts the same set. **Everything that executes stays Deployer-only:** the three run commands (`FlowDebugRunMethodCommand` / `FlowDebugRunScriptCommand` / `FlowDebugRunValueChangeCommand`), the Live entry-point controls, the parked-node Retry/Discard actions and the `?methodId=` / `?instanceId=` / `?script=` drill-ins; for a non-Deployer the page pins itself to Replay mode.
- **Two entry points** (script entry redesigned 2026-09-08, `docs/plans/2026-09-08-flow-debugger-script-entry-design.md`). An inbound API method (inputs form generated from the method's `ParameterDefinitions`, values coerced by declared type) or a **site script** (Site → Instance → Script pickers, site-scope filtered): a script whose trigger is ValueChange or Conditional shows one *New value for `<attribute>`* field, with the attribute derived from the script's own trigger configuration rather than picked; every other script kind (Call, Interval, Expression, or no trigger) shows a `ParameterValueForm` generated from the script's `ParameterDefinitions`, or a "this script takes no parameters" note when there are none. A method-entry Live run is additionally site-scoped **at route time** on the server (`RouteHelper.WithSiteScope` → `SITE_SCOPE_DENIED`, arch-review F2), so an operator cannot reach a site outside their scope by choosing a method that routes there. Both entry points pick the target instance through the shared **`InstancePicker`** combobox (2026-09-08, `docs/plans/2026-09-08-instance-picker-design.md`): a text field whose dropdown is the site's area tree (`InstanceTreeBuilder` — all areas collapsed on open, un-areaed instances at the root, orphaned rows surface at the root), typed text filters by instance name and expands to the matches, Enter picks the first visible match, Escape restores the selection.
- **Predicted graph first.** Choosing an entry point renders the statically predicted shape (`FlowShapePredictor`, Script Analysis REQ-SA-7) before anything runs — nodes whose target depends on inputs render dashed; after a run, predicted nodes the run never produced render greyed.
+28 -1
View File
@@ -152,7 +152,7 @@ The `sbk_<keyId>_<secret>` design already supports zero-downtime rotation — a
| `TIMEOUT` | 500 | The method script exceeded its execution timeout. |
| `SITE_UNREACHABLE` | 500 | The target site did not respond to the routed request within the routing timeout (no contact / down / partitioned) — typed from the routed Ask expiring (`AskTimeoutException`), never inferred from an error-message substring. A `Success=false` response means the site answered and is classified `SCRIPT_ERROR`. |
| `SCRIPT_COMPILE_FAILED` | 500 | The method's script failed to compile (returned consistently on every node until a compiling version is saved). |
| `SCRIPT_ERROR` | 500 | Catch-all for any other script-execution failure (details are logged centrally, never leaked to the caller). |
| `SCRIPT_ERROR` | 500 | Catch-all for any other script-execution failure (details are logged centrally, never leaked to the caller — the exception's `ToString()` is captured on the audit row as `ErrorDetail`, never in the response body). |
### Extended Type System
- API method parameter and return type definitions support an **extended type system** beyond the four template attribute types (Boolean, Integer, Float, String):
@@ -295,6 +295,33 @@ Containment rules (enforced, not advisory):
For everything else, scripts interact with the system through the curated `Route` and
`Parameters` surfaces above.
#### Logging (`Log`)
Method scripts get the same leveled `Log` global site scripts do (design
`docs/plans/2026-09-09-flow-debugger-step-logs-design.md` D4/D5) — `InboundScriptContext.Log`,
backed by `InboundScriptLogger`:
- `Log.Debug(message, exception?)` / `Log.Info` / `Log.Warn` / `Log.Error` — synchronous, `void`,
never throws into the script.
- **Always** forwards to the central node's `ILogger`. A `ScriptTrace` / `ScriptLog` audit row is
written **only when the request is traced** (a Flow Debugger Live run, or a real call to a method
with `TraceAllCalls`), through the executor's inline trace sink and `ICentralAuditWriter`; an
ordinary call pays a log entry and nothing more.
- Lines with no explicit step id are attributed to the request's anchor row (the `InboundRequest`
row, or the debugger's synthesized `RouteCall` root), so they render on the inbound node.
- **Bounded:** `ScadaBridge:InboundApi:MaxLogLinesPerRun` (default 200, validated `> 0`) caps the
rows one request may emit, messages are clipped at 2 000 characters, and one closing `warn` row
names the dropped count.
#### Script exception detail
`InboundScriptResult` carries an internal `ErrorDetail` (`ex.ToString()` of the fault the method
script threw). It is written to the request's audit row — by `FlowDebugRunner` for a Live run, and
on the ordinary HTTP path via `HttpContext.Items[AuditWriteMiddleware.ErrorDetailItemKey]` so a
`TraceAllCalls` Replay has it too — and is projected onto `FlowStep.ErrorDetail` for the Flow
Debugger's exception block. **It is never returned to the HTTP caller:** the response body stays the
generic `SCRIPT_ERROR` envelope, unchanged.
### Routing Behavior
- The `Route.To()` helper resolves the instance's site assignment from the configuration database and routes the request to the correct site cluster via the Communication Layer.
- The call is **synchronous from the API caller's perspective** — the API method blocks until the site responds or the **method-level timeout** is reached.
@@ -148,6 +148,8 @@ var errors = RoslynScriptCompiler.Compile(code, typeof(ScriptCompileSurface));
```
This allows the compile to bind `Attributes["name"]`, `Notify.To("x").Send(...)`, `ExternalSystem.Call(...)`, `Alarms.CurrentAsync()`, and similar API calls against real types, catching undefined-symbol and type-mismatch errors before deployment.
The surface also mirrors the **`Log` global** (step logs design `docs/plans/2026-09-09-flow-debugger-step-logs-design.md` D4) as the compile-only `ScriptCompileSurface.CompileLog` — `Debug` / `Info` / `Warn` / `Error(string message, Exception? exception = null)`, including the optional exception parameter, so `Log.Warn("…", ex)` binds at the design-time gate exactly as it does at the site. `Log` needs no `ScriptTrustPolicy` change for the same reason `Alarms` did not: the trust boundary is a deny-list over API roots, not an allow-list of context members.
Where an accessor returns a domain type rather than a primitive, the stub returns the **same** type the runtime returns (not a compile-only stand-in) — `Alarms.CurrentAsync()` yields `IReadOnlyList<ScriptAlarm>` on both surfaces, so a script reading `a.Severity` / `a.AckTime` binds identically at the gate and at the site. This works because those types live in Commons, which is already in `DefaultAssemblies`.
**Trust-model note:** the script trust boundary is a **deny-list over API roots**, not an allow-list of context members, so adding a globals member such as `Alarms` requires no `ScriptTrustPolicy` change. What must hold is that the member's return type resolves in a *permitted* namespace — pinned by a test asserting no entry in `ForbiddenScopes` prefixes the Commons script-surface namespace.
+10 -1
View File
@@ -516,6 +516,15 @@ Available to all script runs and alarm on-trigger runs:
- The type lives in Commons so the runtime accessor and the Script Analysis compile-only surface (`ScriptCompileSurface.CompileAlarmsAccessor`) project to the **same** type — a script binding at the design-time gate binds identically at the site. Parity is pinned by `CompileSurfaceParityTests`.
- Design record: `docs/plans/2026-06-30-mes-alarm-status-api.md` §5.2.
### Logging (`Log`)
- `Log.Debug(message, exception?)` / `Log.Info` / `Log.Warn` / `Log.Error` — leveled, per-run logging available to **every** script, alongside the `Alarms` accessor above (design `docs/plans/2026-09-09-flow-debugger-step-logs-design.md` D4/D5). Synchronous, `void`, and it **never throws into the script**: a failing sink is swallowed, because a debug observer must not fail a production run.
- **Every line always reaches the node's `ILogger`** (structured `Instance` / `Script` / `ExecutionId` / message), traced or not — an untraced run is an ordinary logging surface and pays nothing else. A `ScriptTrace` / `ScriptLog` **audit row** is emitted **only when the run is traced** (a Flow Debugger Live run, or a real call to a method with `TraceAllCalls`), so a timer or value-change script logging at high rate costs no audit volume. There is no site-event-log sink for `Log.*`.
- The implementation is `ScriptLogger` (`SiteRuntime/Scripts/ScriptLogger.cs`), created by `ScriptRuntimeContext` and exposed as `ScriptGlobals.Log`. **One logger instance is one run's counter** — `CallShared` shares the calling context, so an inline shared script shares the caller's counter, execution id and anchor row rather than starting its own.
- **Runtime lines land on the step that produced them.** While a run is traced, `ScriptRuntimeContext` wraps its own `ILogger` in a `TraceCapturingLogger`, which forwards everything to the inner logger unchanged and additionally copies **Information and above** into `ScriptLog` rows with `logSource = "runtime"` (Debug/Trace are deliberately not captured — internal chatter). `IsEnabled` returns true for Information and above regardless of the node's minimum level, because those entries are trace data rather than node log output. Attribution rides an ambient `TraceStepScope` (`AsyncLocal<Guid?>`): `SetAttribute`, `WriteBatchAndWait`, `WaitInternal` and the synchronous `ExternalSystem.Call` pre-mint their step row's `EventId`, open the scope around their body and pass the id to `EmitTrace`, so a warning the runtime logs inside a write shows on that write's node. Everything else falls back to the run's anchor `ScriptRun` row. **Build `ScriptLogger` over the UNDECORATED logger** — `TraceCapturingLogger`'s constructor rejects an inner `TraceCapturingLogger`, since double wrapping would emit two rows per line.
- **Bounded (D7).** `ScadaBridge:SiteRuntime:MaxLogLinesPerRun` (default 200, validated `> 0`) caps the rows one run may emit; messages are clipped at 2 000 characters. Lines past the cap are counted and dropped, and a single closing `warn` / `runtime` row naming the dropped count is flushed at run end (it bypasses the counter — it is the one row that must always land).
- **Exception detail.** A failed run's `ScriptRun` row now carries `errorDetail = ex.ToString()` alongside the existing one-line `errorMessage`, so the Flow Debugger can show type, message, stack and inner exception. A timeout emits no detail — an `OperationCanceledException` stack is noise.
- **`Console.WriteLine` is not the logging surface on a site.** `System` is imported, so it compiles, and the Central UI Test Run sandbox does capture it; but on a site node `Console.Out` is process-global and shared by every concurrent script run, so per-run capture is unsound. `Log.*` is the leveled, per-run replacement, and the sandbox's mirror writes to the same console pane.
### Parameter Access
- `Parameters["key"]` — Raw dictionary access (returns `object?`, requires manual casting).
- `Parameters.Get<T>("key")` — Typed access with descriptive error messages. Throws `ScriptParameterException` if parameter is missing, null, or cannot be converted to `T`.
@@ -535,7 +544,7 @@ Available to all script runs and alarm on-trigger runs:
Scripts execute **in-process** with constrained access. The following restrictions are enforced at compilation and runtime:
- **Allowed**: Access to the Script Runtime API (GetAttribute, SetAttribute, CallScript, CallShared, ExternalSystem, Notify, Database, Tracking, Alarms), standard C# language features, basic .NET types (collections, string manipulation, math, date/time). `System.Diagnostics.Stopwatch`, `Debug`, and `Activity` are permitted.
- **Allowed**: Access to the Script Runtime API (GetAttribute, SetAttribute, CallScript, CallShared, ExternalSystem, Notify, Database, Tracking, Alarms, Log), standard C# language features, basic .NET types (collections, string manipulation, math, date/time). `System.Diagnostics.Stopwatch`, `Debug`, and `Activity` are permitted.
- **Forbidden**: File system access (`System.IO`), process spawning (`System.Diagnostics.Process`), threading (`System.Threading` — except `Tasks`, `CancellationToken`, and `CancellationTokenSource`), reflection (`System.Reflection`), all raw network access (`System.Net` — must use `ExternalSystem.Call`), native interop (`System.Runtime.InteropServices`, `Microsoft.Win32`), assembly loading, unsafe code, `dynamic`, `Activator`.
- **Execution timeout**: Configurable per-script maximum execution time, **measured from enqueue** (WP3.1) — a run that spends its whole budget queued behind blocked scripts is shed at dequeue without executing, rather than being granted a fresh full budget for work whose triggering condition is already stale. Exceeding the timeout cancels the script **cooperatively** (S2/UA5): a script blocked in synchronous I/O or a tight CPU loop does not observe cancellation and continues to occupy its dedicated script-execution thread. A **watchdog names the script** once its thread has not returned within a grace period after the timeout — and, as of WP3.1, **detaches and replaces that worker thread**, so a wedged script costs the pool a thread only until the watchdog fires rather than for the rest of the process. The detached worker exits (instead of pulling more work) when its body finally returns, so capacity never silently doubles. Live detached workers are capped at the pool size — at the cap no replacement is started and an Error site event says so, because bounded starvation is preferable to unbounded thread growth when scripts wedge en masse. The scheduler's **queue depth / busy-thread / oldest-busy-age / detached-thread gauges** and the **per-interval shed count** are reported through Health Monitoring so a saturated, stuck, or shedding scheduler is visible.
- **Memory**: Scripts share the host process memory. No per-script memory limit, but the execution timeout prevents runaway allocations.