From 2688bd5cfd44d49fbb9494707214b29579b77a20 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Fri, 13 Nov 2026 03:15:34 -0500 Subject: [PATCH] docs(deploy): #780 QA ScriptedAlarm seam applied (drill pending) and vd03 runner flip to OutOfProcess Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi --- ...1-13-780-scriptedalarm-qa-and-vd03-flip.md | 134 ++++++++++++++++++ 1 file changed, 134 insertions(+) create mode 100644 docs/deploy/2026-11-13-780-scriptedalarm-qa-and-vd03-flip.md diff --git a/docs/deploy/2026-11-13-780-scriptedalarm-qa-and-vd03-flip.md b/docs/deploy/2026-11-13-780-scriptedalarm-qa-and-vd03-flip.md new file mode 100644 index 000000000..0ea37e719 --- /dev/null +++ b/docs/deploy/2026-11-13-780-scriptedalarm-qa-and-vd03-flip.md @@ -0,0 +1,134 @@ +# lmxopcua#780: ScriptedAlarm seam on QA vq03/vq04 and the runner flip on DEV vd03 (2026-11-13) + +> **Status: Part 1 (QA seam) applied, ScriptedAlarm drill PENDING. Part 2 (vd03 flip) DONE.** All three nodes run +> `c9640f5c` (see [`2026-11-13-vd03-qa-to-c9640f5c.md`](2026-11-13-vd03-qa-to-c9640f5c.md)). The owner approved this on +> 2026-11-13, with WinRM limited to the deploy-type steps: env change, `.reg` backups, restarts, `-Verify`, and the +> temporary AdminUI role-mapping row. +> +> - Health, OPC UA, `/metrics` and ConfigDb checks ran from the Mac over HTTP, opc.tcp and SQL (vd03). +> - Logs were read over SMB. +> - No other service was touched. +> - No Galaxy writes, no historian writes. + +## Part 1: QA, `Scripting__Runner__Seams__2=ScriptedAlarm` + +Stage `E:\psrem-stage\otopcua-780-scriptedalarm\` on each node: + +- `seam780.ps1 -Mode plan|apply|rollback`; +- `restart780.ps1`, which restarts the service, waits for `/health/ready`, then runs `-Verify` from the `c9640f5c` stage's `install\`; +- `sql780.ps1 -Action show|addrole|delrole`, which uses the service's own ConfigDb connection, with the password in `SQLCMDPASSWORD` and never printed; +- `envdigest.ps1`. + +The backup folder `backup\` has a protected DACL (SYSTEM + Administrators). + +**Before:** + +- ServiceLevel vq03 **240**, vq04 **250**: Primary vq04, as left by this morning's deploy. +- Both nodes: `/health/ready` 200, `Out-of-process script runner ready.`, shards 0 and 1 up. +- Canary Good on both nodes. +- ConfigDb (QA): 2 scripts (the canary), 1 VT, **0 scripted alarms**, 0 role mappings. Current deployment `5905003e` rev + `82dd0598c37f41f8…`, the canary-only revision. + +| Step | vq03 | vq04 | +|---|---|---| +| `.reg` export | `backup\OtOpcUaHost-pre-780-seam2.reg` (31,002 B) | same name (30,244 B) | +| Env before (names + SHA only) | 48 entries, ordered `a04ca3f1fecb` / sorted `78e6acfab539` (= this morning's post-deploy digest) | 47 entries, `c6adaf68ef48` / `5df6eacf7152` (= this morning's) | +| Flags before | `Scripting__Runner=OutOfProcess`, `Seams__0=Calculation`, `Seams__1=VirtualTag`, `Seams__2` absent, `Scripting__Worker__*` present | same | +| Change | `Scripting__Runner__Seams__2=ScriptedAlarm` inserted right after `Seams__1`, so the installer's two `Scripting__Worker__*` entries stay last | same | +| Env after | 49 entries, `5750bbeb4734` / `4263c3ac5a46`; diff = exactly the one added entry, 0 removed | 48 entries, `ef146b2681b4` / `d19e968212f3`; same diff | +| Restart (one at a time, Secondary first) | 08:06:30.9Z → ready 08:06:41.1Z (**10.2 s**). SL after: vq03 240, vq04 250 | 08:07:21.2Z → ready 08:08:00.4Z (**39.2 s**; the stop alone took ~30 s, see the deviation below) | +| `-Verify` after the restart | **VERIFY OK 21/21** | **VERIFY OK 21/21** | + +**Failover** (vq03 sampled every ~4 s while vq04 restarted): + +- vq03 stayed **240 / `/health/active` 503** from 08:07:19Z to 08:07:48Z. +- From 08:07:52Z it read **250 / 200** on every sample. +- After vq04 was back: vq03 **250** (Primary), vq04 **240**, with no failback. + +**After both restarts:** + +- Both nodes: `/health/ready` 200, 5/5 components, `Out-of-process script runner ready.`, akka 2 members / 0 unreachable. +- `/health/active` 200 on vq03 and 503 on vq04. +- Canary `canary-calc-seconds` and `canary-vt-second` Good and advancing on both endpoints. + +There is no startup log line that lists the selected seams (`ScriptRunnerSelection` logs only warnings), so the env entry +is the evidence until a scripted alarm runs. The drill's proof that the seam is live is +`otopcua_script_ipc_roundtrip{kind="alarm"}` counting on both nodes. + +### Deviation: a planned stop of the Primary takes ~30 s (filed as lmxopcua#906) + +vq04 logged `Application is shutting down...` at 03:07:21.289 EST. It then logged nothing but DbCommand lines until 03:07:51.292: + +- `The request stream was aborted`; +- `HTTP POST /localdb_sync.v1.LocalDbSync/Sync responded 200 in 76909 ms`. + +Only after that did it run Akka `CoordinatedShutdown` → `Leaving` → OldestChanged to vq03. + +The host waits on the peer's open LocalDb sync stream until Kestrel's 30 s shutdown timeout. Throughout that wait the stopping Primary keeps vq03 at 240. The sync client (vq03) stops in about 1 s, so the earlier failover gates, which stopped vq03, never showed this. + +### ScriptedAlarm drill: PENDING + +Prepared, not run. The QA AdminUI needs a signed-in session: QA has no `Security:DeployApiKey`, and authoring and deploying +go through the AdminUI. The temporary `LdapGroupRoleMapping` row (`SCADA-Admins` → `Administrator`, system-wide, Notes +`lmxopcua#780 drill temp`) was added with `sql780.ps1 -Action addrole`, and the local forwarder `localhost:19003` → +`wonder-app-vq03:9000` was started. The agent does not type the LDAP password itself, so the drill waits for an operator +sign-in. + +Planned content (all on the canary equipment `EQ-ec3979550d90`; **HistorizeToAveva off** on every alarm, so no historian write): + +| Object | Value | +|---|---| +| Script `ScriptRunnerCanary-AlarmPred` | `return ((int)ctx.GetTag("script-runner-canary/Device1/canary-calc-seconds").Value) % 20 >= 10;` (input changes every second; active for 10 s, inactive for 10 s) | +| Alarm `canary-alarm-cycle` | AlarmCondition, severity 100, predicate above, Retain on, HistorizeToAveva **off** | +| Script `ScriptRunnerCanary-AlarmRunaway` | `var v = ctx.GetTag("script-runner-canary/Device1/canary-calc-seconds").Value; while (true) { }` | +| Alarm `canary-alarm-runaway` | AlarmCondition, severity 100, HistorizeToAveva **off**, deployed only after the normal alarm has fired and cleared and `ipc_roundtrip{kind="alarm"}` counts on both nodes | + +Cleanup: delete both alarms and both scripts, deploy, and expect the revision back at `82dd0598…`. Delete the role row +(`sql780.ps1 -Action delrole`) and stop the forwarder. + +## Part 2: DEV vd03, runner flip to OutOfProcess + +**ConfigDb (read-only SQL from the Mac):** + +- 0 scripts, 0 virtual tags, 0 scripted alarms. The only driver is FOCAS. +- Current deployment `595cb909` rev `fa9a2e28…`. The draft still drifts from it because of this morning's #903 edit-and-revert. + +**Seams.** All three seams were set, the same as QA, even though vd03 has no scripts of any kind. The seam list only decides +where a future script runs, and DEV should match QA. Dropping every seam for kinds with no scripts would have left the flip +with no effect. + +**No canary on vd03.** Any deploy would seal the drifted draft as a new revision, so `fa9a2e28…` could not be restored +afterwards. The worker start is therefore not observable on DEV: shards stay `not started` until a first script. + +Stage `E:\ApiInstall\_stage-otopcua-780-flip\` (`flip780vd03.ps1 -Mode plan|apply|rollback`, `restart780vd03.ps1`, `envdigest.ps1`). + +| Step | Result | +|---|---| +| Plan | Env **28** entries, `05db5d058fee` / `e3f876a407a8` (= this morning's). No `Scripting__*`. #901 record: InstallRoot `E:\ApiInstall\OtOpcUa`, LocalSystem, PeCacheDir `C:\ProgramData\OtOpcUa\script-pe-cache`, **ConfigureHostService 0**, 5 DenyPaths. Installer hashes `7D4078CB…` / `AE9D84FD…` (= the pack). | +| Backup | `E:\ApiInstall\OtOpcUa-backups\pre-20261113-780-runner-flip\{OtOpcUaHost.reg, OtOpcUaScriptWorker.reg}` (protected) | +| Installer, supported path | Full recorded arguments plus `-ConfigureHostService`, service up. 14 steps: **`SettingsRecord` changed** (now records ConfigureHostService=1) and **`HostServiceEnv` changed** (added `Scripting__Worker__Identity=NT SERVICE\OtOpcUaScriptWorker`, `Scripting__Worker__PeCacheDir=C:\ProgramData\OtOpcUa\script-pe-cache`). The other 12 steps were unchanged. 0 env entries removed. | +| Runner entries | `Scripting__Runner=OutOfProcess`, `Seams__0=Calculation`, `Seams__1=VirtualTag`, `Seams__2=ScriptedAlarm`, inserted ahead of the Worker entries | +| Env after | **34** entries, `69a493345099` / `caa047aaaf42`; diff = exactly those 6 added, 0 removed | +| Restart | 08:11:53.2Z → ready 08:12:02.3Z (**9.1 s**) | +| `-Verify` | **VERIFY OK 14/14** (13 before + `HostServiceEnv`) | + +### vd03 gates + +| Gate | Result | +|---|---| +| Health | `/health/ready` 200, 5/5 components; `script-runner`: **`Out-of-process script runner ready.`** (`state=Ready`, `peCache=true`, shards `not started`); `/health/active` 200 | +| ServiceLevel | **250** on three fresh-session reads (`opc.tcp://10.220.157.247:4840`) | +| Runner metrics | `otopcua_script_runner_state` 0; `otopcua_script_worker_up` 0 for every shard (idle, no scripts) | +| FOCAS | 22 raw variables: **21 Good**, `Program/BlockCount` `BadNoData` (as before) | +| Boot log | `roles=[admin,driver]`; `raw subtree materialised (containers=10, tags=22, … failed=0)`; credential-reference preflight clean; no FTL, no ScriptWorker line. ERR lines were only the certificate-domain noise from the gate's own reads by IP. | + +## Rollback + +- **QA seam:** + 1. Run `E:\psrem-stage\otopcua-780-scriptedalarm\seam780.ps1 -Mode rollback` on each node. It removes only `Seams__2`. + 2. Restart the Secondary, then the Primary (`restart780.ps1`). + 3. Alternatively, import `backup\OtOpcUaHost-pre-780-seam2.reg`. +- **vd03 flip:** + 1. Run `E:\ApiInstall\_stage-otopcua-780-flip\flip780vd03.ps1 -Mode rollback`. It removes the four `Scripting__Runner*` entries and keeps the Worker entries, which are harmless under InProcess. + 2. Restart. + 3. For the full pre-flip state, import both `.reg` files from `E:\ApiInstall\OtOpcUa-backups\pre-20261113-780-runner-flip\`. They restore the host env and the #901 record with ConfigureHostService=0.