feat(scripting): Scripting:Runner=Shadow — in-process results published, each compiled run mirrored to the ScriptWorker pool and compared (lmxopcua#780 P2)

- ShadowScriptRunner (Core.Scripting): IScriptShadow.Mirror copies the inputs and queues the shadow run on the
  pool; compares value (structural), CLR type, status and failure class (compile / budget / threw+message /
  unsupported_type); runner-fault shadow outcomes (Unavailable, worker busy) are not compared. Bounded:
  Scripting:Shadow:MaxInFlight (256) and one per script; drops counted (behind | key_busy).
- Metrics otopcua.script.shadow.{compared,mismatch{script},dropped,inflight}; one rate-limited Warning per
  script and reason with a diff summary (Scripting:Shadow:MismatchLogsPerMinute, re-armed at the seam's deploy).
- The VirtualTag, Calculation and scripted-alarm seams recognise the shadow and keep their in-process path
  byte-for-byte (no coalescer, one-at-a-time pump, in-process engine), mirroring after each compiled run.
- ScriptRunnerSelection builds the shadow over an OutOfProcessScriptRunner whose lines carry
  ScriptRunnerRole=Shadow; Status() is null under Shadow so readiness never sees the shadow pool; the
  script-runner health entry stays Healthy and reports the shadow counters as data.
- docker-dev/docker-compose.script-runner-shadow.yml; docs/VirtualTags.md "Shadow mode".
- Tests: comparison table, per-script metric + once-per-reason log, rate limit, hung/blocking/throwing shadow
  never reaches the caller, MaxInFlight drop accounting, input copy; real-worker divergent + unsupported-type
  scripts counted against exactly those scripts, a worker-only runaway killed/quarantined there with every
  published value intact, alarm transitions identical to in process across a kill -9 of the shadow worker.
  Mutation-proved (16 mutants, 15 red; the one survivor added an assertion).

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-10 07:18:40 -05:00
parent 0796928504
commit 03686eb5d7
17 changed files with 1477 additions and 52 deletions
+29 -2
View File
@@ -45,8 +45,7 @@ The sandbox cannot pre-empt a script. **Memory:** each run's own-thread allocati
#### Out-of-process runner (`Scripting:Runner`, lmxopcua#780)
`Scripting:Runner` selects where compiled scripts run: **`InProcess`** (the default and the rollback — everything
above, unchanged), **`OutOfProcess`**, or `Shadow` (design P2; not implemented yet, so it runs `InProcess` with a
startup Warning). `Scripting:Runner:Seams` selects the seams (`VirtualTag`, `Calculation`, `ScriptedAlarm`). Unset or
above, unchanged), **`OutOfProcess`**, or **`Shadow`** (design P2, see "Shadow mode" below). `Scripting:Runner:Seams` selects the seams (`VirtualTag`, `Calculation`, `ScriptedAlarm`). Unset or
empty means `VirtualTag` + `Calculation`: the **`ScriptedAlarm` seam runs out of process only when it is listed
explicitly** (alarms go last, design §14 #10). Under `OutOfProcess` the selected seams hand their compiled runs to
`OutOfProcessScriptRunner` (`Core.Scripting.OutOfProcess`), which runs them in a pool of `ZB.MOM.WW.OtOpcUa.ScriptWorker`
@@ -156,6 +155,34 @@ processes:
Pinned by `tests/Server/ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests` (`Category=ScriptWorker`: real worker processes).
#### Shadow mode (`Scripting:Runner=Shadow`, lmxopcua#780 P2)
A time-boxed diagnostic before the flip to `OutOfProcess`. The selected seams (same `Scripting:Runner:Seams` rule:
unset = VirtualTag + Calculation, `ScriptedAlarm` only when listed) keep their **in-process path, unchanged — that is
what is published** — and each compiled evaluation is then mirrored to the `ScriptWorker` pool by `ShadowScriptRunner`
(`Core.Scripting`), whose result is only compared. The passthrough fast path is never mirrored (it runs no user code).
- **Compared:** value (structural for arrays), exact CLR type, status (Ok vs failed) and failure class (`compile`,
`budget` = timed out / killed / refused / quarantined, `threw` with its message, `unsupported_type`). A shadow outcome
that is the runner's own fault (worker restarted, none ready, worker busy) is **not compared**
(`result = shadow_unavailable`). An input or result the protocol cannot carry (design §14 #11) counts as a mismatch
with reason `unsupported_type` — out of process that evaluation would publish Bad.
- **Never reaches a published value or readiness.** The mirror copies the inputs and queues the shadow run on the pool;
a shadow-side timeout, kill, crash, quarantine, or a runner that hangs or throws stays on the shadow side.
`/health/ready`'s `script-runner` entry stays **Healthy** under Shadow and carries the shadow pool's state and the
counters as data. Kill / quarantine lines of the shadow pool go to the host log tagged `ScriptRunnerRole=Shadow`,
never to the script log; shadow runs' `ctx.Logger` lines are dropped (the in-process run wrote them).
- **Bounded.** At most `Scripting:Shadow:MaxInFlight` (default 256) shadow runs outstanding and at most one per script;
the rest are dropped and counted (`otopcua.script.shadow.dropped{kind, reason = behind | key_busy}`).
- **Metrics:** `otopcua.script.shadow.compared{kind, result = match | mismatch | shadow_unavailable}`,
`otopcua.script.shadow.mismatch{kind, reason, script}`, `otopcua.script.shadow.dropped`, `otopcua.script.shadow.inflight`.
The shadow pool's own `otopcua.script.worker.*` / `otopcua.script.quarantined` series are emitted as usual.
- **Log:** the first mismatch of each script for each reason logs one Warning `Script shadow mismatch: <kind> <key>
(<reason>) — published …; shadow …`, re-armed when the seam redeploys, capped at `Scripting:Shadow:MismatchLogsPerMinute`
(default 20) lines a minute; the next line reports how many were suppressed.
- **Cost.** Every selected script runs twice and the in-process risk stays. docker-dev:
`docker-dev/docker-compose.script-runner-shadow.yml` (all three seams).
#### Quarantine: what a client sees (status code)
A quarantined script's output reads **`BadInternalError`** on OPC UA, the same code as a script that