feat(scripting): self-contained script worker publish, docker-dev drill, §8 perf harness (lmxopcua#780 P1)

- A self-contained Host publish (-r win-x64 --self-contained, the Windows installs) now publishes the ScriptWorker self-contained for the same RID into script-worker/. The launcher starts its apphost directly when runtimeconfig.json lists includedFrameworks. A framework-dependent worker (docker-dev) keeps running through the dotnet muxer. - docker-dev/docker-compose.script-runner.yml: OutOfProcess for the central pair's VT + Calc seams. Live drill recorded in design §12.1 (runaway kill + quarantine, kill -9 recovery, open circuit with /health/ready still 200, metrics). - ScriptRunnerPerfTests (Category=Perf, opt-in via OTOPCUA_SCRIPT_PERF=1): InProcess vs OutOfProcess at K=1/2/4. The accepted 5k evals/s/node gate passes on macOS and windev. The host-CPU, RSS and Windows latency budget misses are tracked as #789. - Fix: the single-flight slot is released before the awaiting caller resumes. Before, a caller that started its next run right after the previous one returned could be refused with "previous evaluation still running", which the seams publish as Bad. - Fix: the worker compiles on dedicated threads, not the thread pool. A burst of cold loads starved the heartbeat, and the host killed the shard (found on windev). Load-timeout queueing is #790. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-10 02:53:53 -05:00
parent f50b29abae
commit 1ead8f458a
13 changed files with 836 additions and 20 deletions
@@ -98,7 +98,10 @@ At startup the host kills workers a dead predecessor of the same install left be
 (pid + start time + image, and only when the marker's host is gone). `Scripting:Worker:PeCacheDir`, when set, holds the
 compiled-PE cache, its HMAC key and the launch markers: put it outside the install folder (startup refuses a relative path
 or one inside the content root) on a directory only the service account can read. The `script-runner` readiness component
-is at worst Degraded. The worker currently runs under the host's account through the dotnet muxer; the dedicated low-privilege account,
+is at worst Degraded. A self-contained host publish (`dotnet publish … -r win-x64 --self-contained`, the Windows installs, which have no
+.NET runtime) ships a **self-contained** worker in `script-worker/` (~97 MB: it carries its own runtime) and the host starts its
+`ZB.MOM.WW.OtOpcUa.ScriptWorker.exe` apphost directly; a framework-dependent host (docker-dev) runs the worker's dll through the
+dotnet muxer. The worker currently runs under the host's account; the dedicated low-privilege account,
 pipe ACL and firewall outbound rule (design §14 #5/#6) are installer work still open on lmxopcua#780. See
 `docs/VirtualTags.md` §"Out-of-process runner".

@@ -114,11 +114,24 @@ and Calculation evaluators hand each compiled run to `OutOfProcessScriptRunner`
  is on the **ready** tier and is at worst **Degraded** (still 200): a sick runner never fails readiness, so a pair
  does not move ServiceLevel to its peer over a script problem (design §14 #12).
 - **Deployment.** The Host build copies the worker into `script-worker/` beside `OtOpcUa.Host.dll`
-  (`ScriptWorker.Deploy.targets`, Build and Publish); `Scripting:Worker:ExecutablePath` overrides it. It is run through
-  the dotnet muxer, so it needs a framework-dependent host (the docker-dev image). Not yet done (lmxopcua#780): a
-  self-contained worker for the Windows installs, the low-privilege worker account, pipe ACL and firewall rule
-  (installer), the docker-dev live drill, the scripted-alarm seam, the per-seam batch submit (design §4.5) and the §8
-  perf harness.
+  (`ScriptWorker.Deploy.targets`, Build and Publish); `Scripting:Worker:ExecutablePath` overrides it. A framework-dependent
+  host (the docker-dev image) gets the worker's framework-dependent build, run through the dotnet muxer. A **self-contained**
+  host publish (`-r win-x64 --self-contained`, the Windows installs, which have no .NET runtime) publishes the worker
+  self-contained for the same RID (~97 MB, its own runtime; not trimmed or single-file, because Roslyn compiles against the
+  runtime's reference assemblies on disk), and the launcher starts its apphost `ZB.MOM.WW.OtOpcUa.ScriptWorker.exe`
+  directly: `OutOfProcessScriptRunner.ResolveWorkerEntry` picks the apphost only when the worker's `runtimeconfig.json` lists
+  `includedFrameworks`. Gated on windev against a real `-r win-x64` publish (`OTOPCUA_SCRIPTWORKER_TEST_PATH` points the
+  worker suite at the published apphost). Not yet done (lmxopcua#780): the low-privilege worker account, pipe ACL, firewall
+  rule and PeCacheDir ACL (installer), the scripted-alarm seam and the per-seam batch submit (design §4.5).
+- **docker-dev.** `docker-dev/docker-compose.script-runner.yml` turns `OutOfProcess` on for the central pair's
+  VirtualTag and Calculation seams. The 2026-11-10 live drill (runaway kill and quarantine, `kill -9` recovery, open circuit
+  with readiness still 200) is recorded in the design doc's rollout section.
+- **Perf harness.** `ScriptRunnerPerfTests` (`Category=Perf`, skipped unless `OTOPCUA_SCRIPT_PERF=1`) measures InProcess
+  against OutOfProcess at K = 1/2/4 against the accepted 5,000 evals/s/node gate and the design §8.2 budgets. The numbers
+  are in the design doc's rollout section (§12.1). The gate passes; the host-CPU and RSS budgets miss (lmxopcua#789).
+- **Compile threads.** A worker compiles on `min(8, max(1, CPUs/2))` dedicated threads, never on the thread pool, so a
+  burst of cold loads cannot starve its heartbeat or its pipe writer. The host load timeout still counts the time a load
+  spends in that queue (lmxopcua#790).

 Pinned by `tests/Server/ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests` (`Category=ScriptWorker`: real worker processes).

@@ -686,6 +686,12 @@ per-shard state, restarts in the last hour and the quarantine list. Quarantines
     §10 gauges and histograms, the degraded-but-ready `script-runner` health check (§14 #12), and the orphan sweep by
     launch marker (pid + start time + image, host gone). Still open: self-contained worker publish, installer (account,
     pipe ACL, firewall), docker-dev live drill, scripted-alarm seam, the §4.5 batch submit, the §8 perf harness.
+   - **Third P1 slice (lmxopcua#780, record in §12.1):** a self-contained worker publish for self-contained (Windows)
+     Host publishes, started through its apphost; the docker-dev live drill (`docker-dev/docker-compose.script-runner.yml`);
+     the §8 perf harness (`Category=Perf`, opt-in), which passes the accepted 5k evals/s/node gate; and two runner fixes
+     the harness found (a caller refused by its own finished run; heartbeat starvation under a compile burst). Still open:
+     installer (account, pipe ACL, firewall, PeCacheDir ACL), scripted-alarm seam, §4.5 batch submit, the §8.2 budget
+     misses (lmxopcua#789) and the load-timeout queue issue (lmxopcua#790).
 3. **P2, shadow mode on docker-dev and QA.**
   - `Shadow` runs the **in-process** evaluation as the published result, and *also* submits to the worker.
   - It compares value, status class and failure kind per evaluation, and counts
@@ -713,6 +719,160 @@ The PE cache directory can be deleted at any time.
 - `docs/Telemetry.md`;
 - the scadaproj index row, if the status-code decision changes the ScadaBridge-visible contract.

+### 12.1 P1 record: self-contained publish, docker-dev live drill, §8 perf (2026-11-10)
+
+**Self-contained worker publish.**
+- A `-r win-x64 --self-contained` Host publish now publishes the worker self-contained for the same RID into
+  `script-worker/`: 236 files, about 97 MB, with its own runtime. It is not trimmed and not single-file.
+- The launcher starts the worker's apphost (`ZB.MOM.WW.OtOpcUa.ScriptWorker.exe`) directly. It does so only when the
+  worker's `runtimeconfig.json` lists `includedFrameworks`. Otherwise (framework-dependent) it keeps `dotnet exec <dll>`.
+- A framework-dependent publish is unchanged: 51 files, no `includedFrameworks`, run through the muxer.
+- Gate on windev: the Host was published win-x64 self-contained under `C:\build`, and the whole
+  `ScriptWorker.Tests` suite was run against the published apphost (`OTOPCUA_SCRIPTWORKER_TEST_PATH`).
+  - Result: **132/132**.
+  - It includes `A_published_self_contained_worker_runs_as_its_own_process`. That test checks that the worker
+    process's `MainModule` is the `.exe`, so no muxer is involved.
+
+**docker-dev live drill.** Image `otopcua-host:dev` was built from the branch at `f50b29ab` plus the self-contained
+change. That build is from before the two runner fixes below. The drill ran with `docker-compose.script-runner.yml` on
+`central-1`/`central-2`.
+
+| Step | Observed |
+|---|---|
+| Normal VT + Calc | Workers start on the first compiled evaluation (`dotnet exec /app/script-worker/…dll`). `okvt` (HR100+1) stays Good and tracks HR100. Writing HR5 = 21 on the fixture gives Calc `HR5x2` **42 Good** on both nodes, and the sibling VT `drillsib` (HR5×3) **63 Good**. |
+| `while (true) { }` VT (1 s timer, T = 2 s) | Logged `exceeded its hard deadline (4000 ms); killing ScriptWorker[1]` twice, 5 s apart. After the second: `quarantined: "it exceeded its hard deadline and its worker was killed (2 time(s) in the last hour)…"`. Both nodes did the same. `okvt` and Calc share that shard and kept updating Good (26 Good samples, no Bad) because neither was in flight at the kills. `run_refused` then counts the quarantined trigger. |
+| `kill -9` both workers on central-1 | A 1.5 s busy VT (`drillslow`) was in flight on shard 1. It published **BadResourceUnavailable** at 05:41:22.28, became a suspect and moved to isolation. It was Good again 30 ms later, re-evaluated in isolation. After HR5 = 22 it read 1022. Shards 0 and 1 restarted within about 250 ms. Isolation probation (20 clean runs) returned it to its shard. Calc gave 44 Good and the sibling 66 Good. |
+| Circuit open (worker dll renamed, all workers killed) | `/health/ready` stayed **200**. The overall status went to `Degraded`, and `script-runner` showed `Degraded` → `Unavailable (circuit open)` after 2 min. Logged `Script runner circuit OPEN …` once. `otopcua_script_runner_state` = **2**, `worker_up` = 0 per shard. VTs and Calc (after an HR5 change) published **BadResourceUnavailable**. Driver tags (HR100) stayed Good, so OPC UA kept serving. |
+| Recovery (dll restored) | The next backoff retry (cap 30 s) started the workers. Logged `circuit closed`. Calc **46** (HR5 = 23 × 2), sibling **69**, `drillslow` 1023, `okvt` Good, all without any input change. |
+| Metrics on `:9000/metrics` | `otopcua_script_runner_state`, `worker_up`/`inflight`/`queue_depth`/`heap_bytes{shard}`, `worker_starts`/`exits{reason}`, `compile{cache}`, `ipc_roundtrip`/`eval_duration` histograms, `quarantined{reason}`, `isolation_suspects`. |
+
+On the wire, a Bad publish carries a null Value. This holds for every Bad status in this server, including
+BadInternalError and BadOutOfRange. The seams keep the last value, but an OPC UA client does not see it. This was
+noted for ScadaBridge#112.
+
+**Rig state afterwards.**
+- Drill rows were deleted, HR5 was set back to 5, and the config was redeployed. The revision hash is `e221b179…`,
+  byte-identical to the pre-drill `gate-cleanup` deployment.
+- `otopcua-host:dev` was re-tagged to the pre-drill image `24f9d7287ed2`.
+- `central-1`/`central-2` were recreated from the base compose file: InProcess, no `Scripting__*` env. The four site
+  nodes, sql and traefik were never touched.
+- The two central containers were recreated (twice), so their container-local secret stores and their PE cache dirs
+  under /tmp are gone.
+
+Two defects outside the runner were found and filed:
+- lmxopcua#787: a disabled VirtualTag is still deployed and evaluated.
+- lmxopcua#788: worker start failures log an Error with a stack trace on every retry.
+
+**§8 perf harness.**
+- `ScriptRunnerPerfTests`, `Category=Perf`, opt-in with `OTOPCUA_SCRIPT_PERF=1`.
+- The runners are driven directly: trivial one-input script, 2,000 keys over 50 sources.
+- Phases:
+  - idle sequential;
+  - paced open loop at 5k/s and at 10k/s;
+  - 64-way closed loop.
+- Host CPU is this process's CPU time as a share of one core, harness included. Under the same harness InProcess shows
+  60–75 % at 5–10k/s, and that is the harness's own floor.
+
+macOS:
+
+**Script worker restart / RSS / kill latency (Unix Arm64, 14 logical CPUs, .NET 10.0.14, phase 10 s)**
+
+| Measure | Result | §8.2 budget |
+|---|---:|---|
+| 1000 scripts, empty cache, all first results (compile + write) | 1,633 ms | — |
+| 1000 scripts, fresh runner on a warm cache, all first results | 217 ms | ≤ 2,000 ms |
+| kill both shards → every script's next correct result | 482 ms | ≤ 2,000 ms |
+| worker RSS after first eval (max of 2 shards) | 188 MB | ≤ 80 MB + scripts |
+| worker RSS with 500 scripts loaded per shard (max) | 358 MB | — |
+| kill latency after T+G (5 runs, ms) | 31 / 35 / 31 / 32 / 33 | ≤ 100 ms |
+
+**Script runner perf (Unix Arm64, 14 logical CPUs, .NET 10.0.14, phase 10 s)**
+
+| Runner | Phase | evals/s | p50 µs | p99 µs | host CPU | ok / not ok |
+|---|---|---:|---:|---:|---:|---|
+| InProcess | idle, sequential |   189,154 |        4 |         8 |    446 % | 5,000 / 0 |
+| InProcess | paced 5,000/s |     4,998 |       32 |       223 |     60 % | 49,999 / 0 |
+| InProcess | paced 10,000/s |     9,991 |       41 |       364 |     74 % | 99,986 / 0 |
+| InProcess | closed loop ×64 |   789,533 |        4 |        15 |    409 % | 7,896,214 / 0 |
+| OutOfProcess K=1 | idle, sequential |    21,641 |       45 |        64 |    264 % | 5,000 / 0 |
+| OutOfProcess K=1 | paced 5,000/s |     4,999 |      149 |       860 |    108 % | 49,998 / 0 |
+| OutOfProcess K=1 | paced 10,000/s |     9,995 |      205 |     1,763 |    159 % | 99,989 / 0 |
+| OutOfProcess K=1 | closed loop ×64 |   206,862 |      258 |     1,368 |    526 % | 2,068,749 / 0 |
+| OutOfProcess K=2 | idle, sequential |    22,642 |       41 |        70 |    267 % | 5,000 / 0 |
+| OutOfProcess K=2 | paced 5,000/s |     4,999 |      182 |       722 |    125 % | 49,994 / 0 |
+| OutOfProcess K=2 | paced 10,000/s |     9,996 |      232 |       720 |    237 % | 99,998 / 0 |
+| OutOfProcess K=2 | closed loop ×64 |   336,519 |      150 |       993 |    544 % | 3,365,226 / 0 |
+| OutOfProcess K=4 | idle, sequential |    18,461 |       50 |        97 |    303 % | 5,000 / 0 |
+| OutOfProcess K=4 | paced 5,000/s |     4,999 |      201 |       455 |    197 % | 49,997 / 0 |
+| OutOfProcess K=4 | paced 10,000/s |     9,997 |      126 |       732 |    176 % | 99,989 / 0 |
+| OutOfProcess K=4 | closed loop ×64 |   297,094 |      178 |     1,015 |    507 % | 2,970,998 / 0 |
+
+windev (Windows VM; framework-dependent worker built beside the tests):
+
+**Script runner perf (Win32NT X64, 36 logical CPUs, .NET 10.0.8, phase 10 s)**
+
+| Runner | Phase | evals/s | p50 µs | p99 µs | host CPU | ok / not ok |
+|---|---|---:|---:|---:|---:|---|
+| InProcess | idle, sequential |     5,876 |      142 |       253 |    204 % | 5,000 / 0 |
+| InProcess | paced 5,000/s |     4,997 |      136 |       335 |     55 % | 49,996 / 0 |
+| InProcess | paced 10,000/s |     9,989 |      105 |       274 |      6 % | 99,984 / 0 |
+| InProcess | closed loop ×64 |   177,782 |       26 |    10,062 |    543 % | 1,778,824 / 0 |
+| OutOfProcess K=1 | idle, sequential |     1,276 |      750 |     1,263 |     74 % | 5,000 / 0 |
+| OutOfProcess K=1 | paced 5,000/s |     4,997 |    1,073 |     3,028 |     68 % | 49,990 / 0 |
+| OutOfProcess K=1 | paced 10,000/s |     9,992 |    1,139 |     2,840 |    102 % | 99,999 / 0 |
+| OutOfProcess K=1 | closed loop ×64 |    38,464 |    1,514 |     6,547 |    350 % | 384,727 / 0 |
+| OutOfProcess K=2 | idle, sequential |     1,207 |      795 |     1,376 |     80 % | 5,000 / 0 |
+| OutOfProcess K=2 | paced 5,000/s |     4,998 |      956 |     1,856 |    111 % | 49,997 / 0 |
+| OutOfProcess K=2 | paced 10,000/s |     9,962 |      942 |     1,916 |    153 % | 99,992 / 0 |
+| OutOfProcess K=2 | closed loop ×64 |    51,241 |    1,129 |     5,894 |    594 % | 512,504 / 0 |
+| OutOfProcess K=4 | idle, sequential |     1,163 |      828 |     1,366 |     61 % | 5,000 / 0 |
+| OutOfProcess K=4 | paced 5,000/s |     4,997 |      804 |     1,472 |    172 % | 49,991 / 0 |
+| OutOfProcess K=4 | paced 10,000/s |     9,992 |      757 |     1,591 |    233 % | 99,992 / 0 |
+| OutOfProcess K=4 | closed loop ×64 |    69,503 |      774 |     6,321 |  1,051 % | 695,094 / 0 |
+
+
+**Script worker restart / RSS / kill latency (Win32NT X64, 36 logical CPUs, .NET 10.0.8, phase 10 s)**
+
+| Measure | Result | §8.2 budget |
+|---|---:|---|
+| 1000 scripts, empty cache, all first results (compile + write) | 11,625 ms | — |
+| 1000 scripts, fresh runner on a warm cache, all first results | 1,715 ms | ≤ 2,000 ms |
+| kill both shards → every script's next correct result | 2,028 ms | ≤ 2,000 ms |
+| worker RSS after first eval (max of 2 shards) | 158 MB | ≤ 80 MB + scripts |
+| worker RSS with 500 scripts loaded per shard (max) | 232 MB | — |
+| kill latency after T+G (5 runs, ms) | 257 / 210 / 200 / 217 / 209 | ≤ 100 ms |
+
+Verdict against §8.2 and the accepted gate:
+- **Accepted gate (≥ 5,000 evals/s/node at K = 2): PASS** on both hosts (macOS 336k/s, windev 51k/s closed loop), with every result correct.
+- Throughput ≥ 10k/s: PASS.
+- Idle p50 ≤ 150 µs: PASS on macOS (41 µs), MISS on windev (795 µs; InProcess there is already 142 µs).
+- p99 ≤ 2 ms at 10k/s: PASS (macOS 0.7 ms, windev 1.9 ms).
+- Warm-cache restart of 1,000 scripts ≤ 2 s: PASS (0.2 s / 1.7 s).
+- Kill latency ≤ 100 ms: PASS on macOS (about 32 ms), MISS on windev (about 210 ms).
+- **Host CPU ≤ 25 % of a core at 10k/s: MISS.** windev runs K = 2 at about 153 %. On macOS, K = 2 runs about 240 % against InProcess's 74 % under the same
+  harness. A sequential probe measured 156 µs of host CPU per evaluation, and 63 µs with
+  `DOTNET_ThreadPool_UnfairSemaphoreSpinLimit=0`. Thread-pool spinning is therefore most of it.
+- **Worker idle RSS ≤ 80 MB: MISS.** It is about 190 MB on macOS and 158 MB on windev, because Roslyn is resident since compiles run in the worker.
+
+Both misses → lmxopcua#789 (the §4.5 batching and per-eval allocation work).
+
+**Two runner defects found by the harness and fixed in this slice.**
+1. **Refused own run.** The single-flight slot was freed by a continuation that was not ordered against the caller's
+   await. A caller that awaited one run and then started the next was sometimes refused with
+   `previous evaluation still running`: 32 of 200 back-to-back runs, and about 80 % in the perf idle phase. The VT and
+   Calc seams publish that as Bad. The slot is now freed before the caller resumes, idempotently.
+   - Pinned by `A_caller_that_awaited_its_run_is_never_refused_by_it` (1,000 runs).
+   - Mutation red.
+2. **Heartbeat starvation under a compile burst.** Each load was compiled synchronously on its own thread-pool work item.
+   On windev, 500 cold loads starved the heartbeat timer and the pipe writer. The host killed the shard
+   (`KilledHeartbeat`), and all 500 keys became suspects. Compiles now run on `min(8, max(1, CPUs/2))` dedicated
+   `ScriptWorker.Compile` threads.
+   - Pinned by `A_burst_of_cold_compiles_does_not_starve_the_heartbeat` (1,500 cold scripts, 50 ms heartbeat).
+   - Red on macOS with the thread-pool version. Green 3/3 on macOS and windev; on windev 1,500 cold compiles take
+     about 18 s.
+
+The host load timeout still counts the time a load waits behind other compiles → lmxopcua#790.
+
 ---

 ## 13. Test plan