Files
lmxopcua/docs/deploy/2026-11-13-780-scriptedalarm-qa-and-vd03-flip.md
T

173 lines
13 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# lmxopcua#780: ScriptedAlarm seam on QA vq03/vq04 and the runner flip on DEV vd03 (2026-11-13)
> **Status: EXECUTED. Part 1 (QA ScriptedAlarm seam + alarm drill) PASS; Part 2 (vd03 flip) DONE.** All three nodes run
> `c9640f5c` (see [`2026-11-13-vd03-qa-to-c9640f5c.md`](2026-11-13-vd03-qa-to-c9640f5c.md)). The owner approved this on
> 2026-11-13, with WinRM limited to the deploy-type steps: env change, `.reg` backups, restarts, `-Verify`, and the
> temporary AdminUI role-mapping row.
>
> - Health, OPC UA, `/metrics` and ConfigDb checks ran from the Mac over HTTP, opc.tcp and SQL (vd03).
> - Logs were read over SMB.
> - No other service was touched.
> - No Galaxy writes, no historian writes.
## Part 1: QA, `Scripting__Runner__Seams__2=ScriptedAlarm`
Stage `E:\psrem-stage\otopcua-780-scriptedalarm\` on each node:
- `seam780.ps1 -Mode plan|apply|rollback`;
- `restart780.ps1`, which restarts the service, waits for `/health/ready`, then runs `-Verify` from the `c9640f5c` stage's `install\`;
- `sql780.ps1 -Action show|addrole|delrole`, which uses the service's own ConfigDb connection, with the password in `SQLCMDPASSWORD` and never printed;
- `envdigest.ps1`.
The backup folder `backup\` has a protected DACL (SYSTEM + Administrators).
**Before:**
- ServiceLevel vq03 **240**, vq04 **250**: Primary vq04, as left by this morning's deploy.
- Both nodes: `/health/ready` 200, `Out-of-process script runner ready.`, shards 0 and 1 up.
- Canary Good on both nodes.
- ConfigDb (QA): 2 scripts (the canary), 1 VT, **0 scripted alarms**, 0 role mappings. Current deployment `5905003e` rev
`82dd0598c37f41f8…`, the canary-only revision.
| Step | vq03 | vq04 |
|---|---|---|
| `.reg` export | `backup\OtOpcUaHost-pre-780-seam2.reg` (31,002 B) | same name (30,244 B) |
| Env before (names + SHA only) | 48 entries, ordered `a04ca3f1fecb` / sorted `78e6acfab539` (= this morning's post-deploy digest) | 47 entries, `c6adaf68ef48` / `5df6eacf7152` (= this morning's) |
| Flags before | `Scripting__Runner=OutOfProcess`, `Seams__0=Calculation`, `Seams__1=VirtualTag`, `Seams__2` absent, `Scripting__Worker__*` present | same |
| Change | `Scripting__Runner__Seams__2=ScriptedAlarm` inserted right after `Seams__1`, so the installer's two `Scripting__Worker__*` entries stay last | same |
| Env after | 49 entries, `5750bbeb4734` / `4263c3ac5a46`; diff = exactly the one added entry, 0 removed | 48 entries, `ef146b2681b4` / `d19e968212f3`; same diff |
| Restart (one at a time, Secondary first) | 08:06:30.9Z → ready 08:06:41.1Z (**10.2 s**). SL after: vq03 240, vq04 250 | 08:07:21.2Z → ready 08:08:00.4Z (**39.2 s**; the stop alone took ~30 s, see the deviation below) |
| `-Verify` after the restart | **VERIFY OK 21/21** | **VERIFY OK 21/21** |
**Failover** (vq03 sampled every ~4 s while vq04 restarted):
- vq03 stayed **240 / `/health/active` 503** from 08:07:19Z to 08:07:48Z.
- From 08:07:52Z it read **250 / 200** on every sample.
- After vq04 was back: vq03 **250** (Primary), vq04 **240**, with no failback.
**After both restarts:**
- Both nodes: `/health/ready` 200, 5/5 components, `Out-of-process script runner ready.`, akka 2 members / 0 unreachable.
- `/health/active` 200 on vq03 and 503 on vq04.
- Canary `canary-calc-seconds` and `canary-vt-second` Good and advancing on both endpoints.
There is no startup log line that lists the selected seams (`ScriptRunnerSelection` logs only warnings), so the env entry
is the evidence until a scripted alarm runs. The drill's proof that the seam is live is
`otopcua_script_ipc_roundtrip{kind="alarm"}` counting on both nodes.
### Deviation: a planned stop of the Primary takes ~30 s (filed as lmxopcua#906)
vq04 logged `Application is shutting down...` at 03:07:21.289 EST. It then logged nothing but DbCommand lines until 03:07:51.292:
- `The request stream was aborted`;
- `HTTP POST /localdb_sync.v1.LocalDbSync/Sync responded 200 in 76909 ms`.
Only after that did it run Akka `CoordinatedShutdown` → `Leaving` → OldestChanged to vq03.
The host waits on the peer's open LocalDb sync stream until Kestrel's 30 s shutdown timeout. Throughout that wait the stopping Primary keeps vq03 at 240. The sync client (vq03) stops in about 1 s, so the earlier failover gates, which stopped vq03, never showed this.
### ScriptedAlarm drill (2026-11-13, 11:44–11:53Z): PASS
**Access.**
- The temporary `LdapGroupRoleMapping` row (`SCADA-Admins` → `Administrator`, system-wide, Notes `lmxopcua#780 drill temp`)
was added with `sql780.ps1 -Action addrole` (on-box SQL with the service's own connection).
- The local forwarder `localhost:19003` → `wonder-app-vq03:9000` (socat) was started.
- **The owner signed in to the AdminUI** (session `multi-role`, Administrator). The agent did not type a credential.
- QA OtOpcUa authenticates against the box-local GLAuth (`127.0.0.1:3893`, `dc=scadabridge,dc=local`), not the shared
`dc=zb` directory.
Authored in the AdminUI on the canary equipment `EQ-ec3979550d90`. **HistorizeToAveva was off on both alarms**, checked in
the ConfigDb (`historize=0`) before each deploy.
| Object | Value |
|---|---|
| Script `ScriptRunnerCanary-AlarmPred` (`SC-3e4f868120bb`) | `return ((int)ctx.GetTag("script-runner-canary/Device1/canary-calc-seconds").Value) % 20 >= 10;` |
| Alarm `canary-alarm-cycle` | AlarmCondition, severity 100, Retain on, HistorizeToAveva off |
| Script `ScriptRunnerCanary-AlarmRunaway` (`SC-d3f7ed7b8d74`) | `var v = ctx.GetTag("script-runner-canary/Device1/canary-calc-seconds").Value;` then `while (true) { }` |
| Alarm `canary-alarm-runaway` | AlarmCondition, severity 100, Retain on, HistorizeToAveva off |
During the drill vq03 was the Primary (**250**) and vq04 the Secondary (**240**), as left by Part 1.
**1. Normal scripted alarm.** Deployed as `fbfba004` (rev `5c8ea584…`, 11:46:13Z).
- `alarms --refresh` (OtOpcUa CLI, anonymous) on both endpoints, `ns=3;s=canary-alarm-cycle`: Inactive at 11:46:20 →
**Active 11:46:30 → Inactive 11:46:40 → Active …**, the same 10 s cycle on vq03 and vq04.
- The predicate runs in the worker: `otopcua_script_ipc_roundtrip_microseconds_count{kind="alarm"}` was 66 on vq03 and 67
on vq04 within a minute. This is the proof that the `ScriptedAlarm` seam is live on both nodes.
- The events carry severity **250**: the engine bands the configured 100 into `AlarmSeverity` Low, as designed.
**2. Runaway scripted alarm.** Added and deployed as `b0177ec4` (rev `a9320d56…`, 11:48:53Z).
During the run, each endpoint was sampled 22 times over about 2 min 15 s (11:48:42–11:50:57Z), reading ServiceLevel, both canary values and `/health/ready`. Alarm events were captured on both endpoints the whole time.
| Observed | vq03 (Primary) | vq04 (Secondary) |
|---|---|---|
| Kills | 06:48:59.225 EST: `Script 3:canary-alarm-runaway exceeded its hard deadline (4000 ms); killing ScriptWorker[0] pid 5368` → `ended: "KilledDeadline" (exit code -1, culprit 3:canary-alarm-runaway, 1 evaluation(s) and 0 compile(s) in flight)`; second kill at 06:49:04.163 (pid 14720) | 06:48:59.423 (pid 18096) and 06:49:04.495 (pid 18472), same lines |
| Quarantine | 06:49:04.167: `Script "3:canary-alarm-runaway" quarantined: "it exceeded its hard deadline and its worker was killed (2 time(s) in the last hour); the script is not evaluated again until its source changes or it is redeployed"` | 06:49:04.499, same |
| Shard 0 restart | `ScriptWorker[0] started` at +0.58 s (pid 14720) and +0.75 s (pid 32204) | +0.61 s (pid 18472) and +0.80 s (pid 14432) |
| Metrics after | `worker_exits_total{reason="killed_deadline",shard="0"}` 2, `quarantined_total{reason="killed",script="3:canary-alarm-runaway"}` 1, `run_refused_total` 6, `runner_state` 0, `worker_up` shards 0/1 = 1, `compile_total` hit 8 / miss 2 | identical |
| Sibling alarm `canary-alarm-cycle` (same shard family) | **kept its exact 10 s cycle straight through both kills**: Active 11:48:50, Inactive 11:49:00, Active 11:49:10 … Inactive 11:51:00; no transition missed | same, about 0.3 s later each |
| Sibling Calc + VT canaries | 22/22 samples **Good** and advancing | 22/22 Good |
| Node | `/health/ready` 200 on 22/22; `script-runner` Healthy, `quarantined: 1`; `/health/active` 200 | 200 on 22/22; `quarantined: 1`; `/health/active` 503 |
| ServiceLevel | **250** on 22/22 | **240** on 22/22 |
| Host log, other ERR/FTL in 06:46–06:59 EST | only `TCPSERVERCHANNEL … BadConnectionClosed` at 06:47:15, from the drill's own CLI alarm subscriber being killed | same |
| HistorianGateway `/metrics` (vq03 `:5220`) | `HistorianStatus/Probe` 755 → 813 and nothing else changed: **no historian write or tag-creation RPC** | |
**3. Cleanup.**
- Both alarms and both drill scripts were deleted in the AdminUI. ConfigDb: 2 scripts (the canary), 1 VT, 0 scripted alarms.
- Deployed as **`a8e078f0`, rev `82dd0598c37f41f8…`**, identical to the canary-only revision (`bcaba714` / `5905003e`).
- After the deploy:
- `ns=3;s=canary-alarm-cycle` reads `BadNodeIdUnknown`.
- Both canary values are Good on both endpoints.
- `script-runner` reports `quarantined: 0`.
- ServiceLevel is 250/240.
- The temporary role row was deleted (`sql780.ps1 -Action delrole`: deleted=1, QA role rows 0).
- The forwarder was stopped and the browser tab closed.
## Part 2: DEV vd03, runner flip to OutOfProcess
**ConfigDb (read-only SQL from the Mac):**
- 0 scripts, 0 virtual tags, 0 scripted alarms. The only driver is FOCAS.
- Current deployment `595cb909` rev `fa9a2e28…`. The draft still drifts from it because of this morning's #903 edit-and-revert.
**Seams.** All three seams were set, the same as QA, even though vd03 has no scripts of any kind. The seam list only decides
where a future script runs, and DEV should match QA. Dropping every seam for kinds with no scripts would have left the flip
with no effect.
**No canary on vd03.** Any deploy would seal the drifted draft as a new revision, so `fa9a2e28…` could not be restored
afterwards. The worker start is therefore not observable on DEV: shards stay `not started` until a first script.
Stage `E:\ApiInstall\_stage-otopcua-780-flip\` (`flip780vd03.ps1 -Mode plan|apply|rollback`, `restart780vd03.ps1`, `envdigest.ps1`).
| Step | Result |
|---|---|
| Plan | Env **28** entries, `05db5d058fee` / `e3f876a407a8` (= this morning's). No `Scripting__*`. #901 record: InstallRoot `E:\ApiInstall\OtOpcUa`, LocalSystem, PeCacheDir `C:\ProgramData\OtOpcUa\script-pe-cache`, **ConfigureHostService 0**, 5 DenyPaths. Installer hashes `7D4078CB…` / `AE9D84FD…` (= the pack). |
| Backup | `E:\ApiInstall\OtOpcUa-backups\pre-20261113-780-runner-flip\{OtOpcUaHost.reg, OtOpcUaScriptWorker.reg}` (protected) |
| Installer, supported path | Full recorded arguments plus `-ConfigureHostService`, service up. 14 steps: **`SettingsRecord` changed** (now records ConfigureHostService=1) and **`HostServiceEnv` changed** (added `Scripting__Worker__Identity=NT SERVICE\OtOpcUaScriptWorker`, `Scripting__Worker__PeCacheDir=C:\ProgramData\OtOpcUa\script-pe-cache`). The other 12 steps were unchanged. 0 env entries removed. |
| Runner entries | `Scripting__Runner=OutOfProcess`, `Seams__0=Calculation`, `Seams__1=VirtualTag`, `Seams__2=ScriptedAlarm`, inserted ahead of the Worker entries |
| Env after | **34** entries, `69a493345099` / `caa047aaaf42`; diff = exactly those 6 added, 0 removed |
| Restart | 08:11:53.2Z → ready 08:12:02.3Z (**9.1 s**) |
| `-Verify` | **VERIFY OK 14/14** (13 before + `HostServiceEnv`) |
### vd03 gates
| Gate | Result |
|---|---|
| Health | `/health/ready` 200, 5/5 components; `script-runner`: **`Out-of-process script runner ready.`** (`state=Ready`, `peCache=true`, shards `not started`); `/health/active` 200 |
| ServiceLevel | **250** on three fresh-session reads (`opc.tcp://10.220.157.247:4840`) |
| Runner metrics | `otopcua_script_runner_state` 0; `otopcua_script_worker_up` 0 for every shard (idle, no scripts) |
| FOCAS | 22 raw variables: **21 Good**, `Program/BlockCount` `BadNoData` (as before) |
| Boot log | `roles=[admin,driver]`; `raw subtree materialised (containers=10, tags=22, … failed=0)`; credential-reference preflight clean; no FTL, no ScriptWorker line. ERR lines were only the certificate-domain noise from the gate's own reads by IP. |
## Rollback
- **QA seam:**
1. Run `E:\psrem-stage\otopcua-780-scriptedalarm\seam780.ps1 -Mode rollback` on each node. It removes only `Seams__2`.
2. Restart the Secondary, then the Primary (`restart780.ps1`).
3. Alternatively, import `backup\OtOpcUaHost-pre-780-seam2.reg`.
- **vd03 flip:**
1. Run `E:\ApiInstall\_stage-otopcua-780-flip\flip780vd03.ps1 -Mode rollback`. It removes the four `Scripting__Runner*` entries and keeps the Worker entries, which are harmless under InProcess.
2. Restart.
3. For the full pre-flip state, import both `.reg` files from `E:\ApiInstall\OtOpcUa-backups\pre-20261113-780-runner-flip\`. They restore the host env and the #901 record with ConfigureHostService=0.