003c7f8954
v2-ci / unit-tests (push) Has been cancelled
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
173 lines
13 KiB
Markdown
173 lines
13 KiB
Markdown
# lmxopcua#780: ScriptedAlarm seam on QA vq03/vq04 and the runner flip on DEV vd03 (2026-11-13)
|
||
|
||
> **Status: EXECUTED. Part 1 (QA ScriptedAlarm seam + alarm drill) PASS; Part 2 (vd03 flip) DONE.** All three nodes run
|
||
> `c9640f5c` (see [`2026-11-13-vd03-qa-to-c9640f5c.md`](2026-11-13-vd03-qa-to-c9640f5c.md)). The owner approved this on
|
||
> 2026-11-13, with WinRM limited to the deploy-type steps: env change, `.reg` backups, restarts, `-Verify`, and the
|
||
> temporary AdminUI role-mapping row.
|
||
>
|
||
> - Health, OPC UA, `/metrics` and ConfigDb checks ran from the Mac over HTTP, opc.tcp and SQL (vd03).
|
||
> - Logs were read over SMB.
|
||
> - No other service was touched.
|
||
> - No Galaxy writes, no historian writes.
|
||
|
||
## Part 1: QA, `Scripting__Runner__Seams__2=ScriptedAlarm`
|
||
|
||
Stage `E:\psrem-stage\otopcua-780-scriptedalarm\` on each node:
|
||
|
||
- `seam780.ps1 -Mode plan|apply|rollback`;
|
||
- `restart780.ps1`, which restarts the service, waits for `/health/ready`, then runs `-Verify` from the `c9640f5c` stage's `install\`;
|
||
- `sql780.ps1 -Action show|addrole|delrole`, which uses the service's own ConfigDb connection, with the password in `SQLCMDPASSWORD` and never printed;
|
||
- `envdigest.ps1`.
|
||
|
||
The backup folder `backup\` has a protected DACL (SYSTEM + Administrators).
|
||
|
||
**Before:**
|
||
|
||
- ServiceLevel vq03 **240**, vq04 **250**: Primary vq04, as left by this morning's deploy.
|
||
- Both nodes: `/health/ready` 200, `Out-of-process script runner ready.`, shards 0 and 1 up.
|
||
- Canary Good on both nodes.
|
||
- ConfigDb (QA): 2 scripts (the canary), 1 VT, **0 scripted alarms**, 0 role mappings. Current deployment `5905003e` rev
|
||
`82dd0598c37f41f8…`, the canary-only revision.
|
||
|
||
| Step | vq03 | vq04 |
|
||
|---|---|---|
|
||
| `.reg` export | `backup\OtOpcUaHost-pre-780-seam2.reg` (31,002 B) | same name (30,244 B) |
|
||
| Env before (names + SHA only) | 48 entries, ordered `a04ca3f1fecb` / sorted `78e6acfab539` (= this morning's post-deploy digest) | 47 entries, `c6adaf68ef48` / `5df6eacf7152` (= this morning's) |
|
||
| Flags before | `Scripting__Runner=OutOfProcess`, `Seams__0=Calculation`, `Seams__1=VirtualTag`, `Seams__2` absent, `Scripting__Worker__*` present | same |
|
||
| Change | `Scripting__Runner__Seams__2=ScriptedAlarm` inserted right after `Seams__1`, so the installer's two `Scripting__Worker__*` entries stay last | same |
|
||
| Env after | 49 entries, `5750bbeb4734` / `4263c3ac5a46`; diff = exactly the one added entry, 0 removed | 48 entries, `ef146b2681b4` / `d19e968212f3`; same diff |
|
||
| Restart (one at a time, Secondary first) | 08:06:30.9Z → ready 08:06:41.1Z (**10.2 s**). SL after: vq03 240, vq04 250 | 08:07:21.2Z → ready 08:08:00.4Z (**39.2 s**; the stop alone took ~30 s, see the deviation below) |
|
||
| `-Verify` after the restart | **VERIFY OK 21/21** | **VERIFY OK 21/21** |
|
||
|
||
**Failover** (vq03 sampled every ~4 s while vq04 restarted):
|
||
|
||
- vq03 stayed **240 / `/health/active` 503** from 08:07:19Z to 08:07:48Z.
|
||
- From 08:07:52Z it read **250 / 200** on every sample.
|
||
- After vq04 was back: vq03 **250** (Primary), vq04 **240**, with no failback.
|
||
|
||
**After both restarts:**
|
||
|
||
- Both nodes: `/health/ready` 200, 5/5 components, `Out-of-process script runner ready.`, akka 2 members / 0 unreachable.
|
||
- `/health/active` 200 on vq03 and 503 on vq04.
|
||
- Canary `canary-calc-seconds` and `canary-vt-second` Good and advancing on both endpoints.
|
||
|
||
There is no startup log line that lists the selected seams (`ScriptRunnerSelection` logs only warnings), so the env entry
|
||
is the evidence until a scripted alarm runs. The drill's proof that the seam is live is
|
||
`otopcua_script_ipc_roundtrip{kind="alarm"}` counting on both nodes.
|
||
|
||
### Deviation: a planned stop of the Primary takes ~30 s (filed as lmxopcua#906)
|
||
|
||
vq04 logged `Application is shutting down...` at 03:07:21.289 EST. It then logged nothing but DbCommand lines until 03:07:51.292:
|
||
|
||
- `The request stream was aborted`;
|
||
- `HTTP POST /localdb_sync.v1.LocalDbSync/Sync responded 200 in 76909 ms`.
|
||
|
||
Only after that did it run Akka `CoordinatedShutdown` → `Leaving` → OldestChanged to vq03.
|
||
|
||
The host waits on the peer's open LocalDb sync stream until Kestrel's 30 s shutdown timeout. Throughout that wait the stopping Primary keeps vq03 at 240. The sync client (vq03) stops in about 1 s, so the earlier failover gates, which stopped vq03, never showed this.
|
||
|
||
### ScriptedAlarm drill (2026-11-13, 11:44–11:53Z): PASS
|
||
|
||
**Access.**
|
||
- The temporary `LdapGroupRoleMapping` row (`SCADA-Admins` → `Administrator`, system-wide, Notes `lmxopcua#780 drill temp`)
|
||
was added with `sql780.ps1 -Action addrole` (on-box SQL with the service's own connection).
|
||
- The local forwarder `localhost:19003` → `wonder-app-vq03:9000` (socat) was started.
|
||
- **The owner signed in to the AdminUI** (session `multi-role`, Administrator). The agent did not type a credential.
|
||
- QA OtOpcUa authenticates against the box-local GLAuth (`127.0.0.1:3893`, `dc=scadabridge,dc=local`), not the shared
|
||
`dc=zb` directory.
|
||
|
||
Authored in the AdminUI on the canary equipment `EQ-ec3979550d90`. **HistorizeToAveva was off on both alarms**, checked in
|
||
the ConfigDb (`historize=0`) before each deploy.
|
||
|
||
| Object | Value |
|
||
|---|---|
|
||
| Script `ScriptRunnerCanary-AlarmPred` (`SC-3e4f868120bb`) | `return ((int)ctx.GetTag("script-runner-canary/Device1/canary-calc-seconds").Value) % 20 >= 10;` |
|
||
| Alarm `canary-alarm-cycle` | AlarmCondition, severity 100, Retain on, HistorizeToAveva off |
|
||
| Script `ScriptRunnerCanary-AlarmRunaway` (`SC-d3f7ed7b8d74`) | `var v = ctx.GetTag("script-runner-canary/Device1/canary-calc-seconds").Value;` then `while (true) { }` |
|
||
| Alarm `canary-alarm-runaway` | AlarmCondition, severity 100, Retain on, HistorizeToAveva off |
|
||
|
||
During the drill vq03 was the Primary (**250**) and vq04 the Secondary (**240**), as left by Part 1.
|
||
|
||
**1. Normal scripted alarm.** Deployed as `fbfba004` (rev `5c8ea584…`, 11:46:13Z).
|
||
|
||
- `alarms --refresh` (OtOpcUa CLI, anonymous) on both endpoints, `ns=3;s=canary-alarm-cycle`: Inactive at 11:46:20 →
|
||
**Active 11:46:30 → Inactive 11:46:40 → Active …**, the same 10 s cycle on vq03 and vq04.
|
||
- The predicate runs in the worker: `otopcua_script_ipc_roundtrip_microseconds_count{kind="alarm"}` was 66 on vq03 and 67
|
||
on vq04 within a minute. This is the proof that the `ScriptedAlarm` seam is live on both nodes.
|
||
- The events carry severity **250**: the engine bands the configured 100 into `AlarmSeverity` Low, as designed.
|
||
|
||
**2. Runaway scripted alarm.** Added and deployed as `b0177ec4` (rev `a9320d56…`, 11:48:53Z).
|
||
|
||
During the run, each endpoint was sampled 22 times over about 2 min 15 s (11:48:42–11:50:57Z), reading ServiceLevel, both canary values and `/health/ready`. Alarm events were captured on both endpoints the whole time.
|
||
|
||
| Observed | vq03 (Primary) | vq04 (Secondary) |
|
||
|---|---|---|
|
||
| Kills | 06:48:59.225 EST: `Script 3:canary-alarm-runaway exceeded its hard deadline (4000 ms); killing ScriptWorker[0] pid 5368` → `ended: "KilledDeadline" (exit code -1, culprit 3:canary-alarm-runaway, 1 evaluation(s) and 0 compile(s) in flight)`; second kill at 06:49:04.163 (pid 14720) | 06:48:59.423 (pid 18096) and 06:49:04.495 (pid 18472), same lines |
|
||
| Quarantine | 06:49:04.167: `Script "3:canary-alarm-runaway" quarantined: "it exceeded its hard deadline and its worker was killed (2 time(s) in the last hour); the script is not evaluated again until its source changes or it is redeployed"` | 06:49:04.499, same |
|
||
| Shard 0 restart | `ScriptWorker[0] started` at +0.58 s (pid 14720) and +0.75 s (pid 32204) | +0.61 s (pid 18472) and +0.80 s (pid 14432) |
|
||
| Metrics after | `worker_exits_total{reason="killed_deadline",shard="0"}` 2, `quarantined_total{reason="killed",script="3:canary-alarm-runaway"}` 1, `run_refused_total` 6, `runner_state` 0, `worker_up` shards 0/1 = 1, `compile_total` hit 8 / miss 2 | identical |
|
||
| Sibling alarm `canary-alarm-cycle` (same shard family) | **kept its exact 10 s cycle straight through both kills**: Active 11:48:50, Inactive 11:49:00, Active 11:49:10 … Inactive 11:51:00; no transition missed | same, about 0.3 s later each |
|
||
| Sibling Calc + VT canaries | 22/22 samples **Good** and advancing | 22/22 Good |
|
||
| Node | `/health/ready` 200 on 22/22; `script-runner` Healthy, `quarantined: 1`; `/health/active` 200 | 200 on 22/22; `quarantined: 1`; `/health/active` 503 |
|
||
| ServiceLevel | **250** on 22/22 | **240** on 22/22 |
|
||
| Host log, other ERR/FTL in 06:46–06:59 EST | only `TCPSERVERCHANNEL … BadConnectionClosed` at 06:47:15, from the drill's own CLI alarm subscriber being killed | same |
|
||
| HistorianGateway `/metrics` (vq03 `:5220`) | `HistorianStatus/Probe` 755 → 813 and nothing else changed: **no historian write or tag-creation RPC** | |
|
||
|
||
**3. Cleanup.**
|
||
- Both alarms and both drill scripts were deleted in the AdminUI. ConfigDb: 2 scripts (the canary), 1 VT, 0 scripted alarms.
|
||
- Deployed as **`a8e078f0`, rev `82dd0598c37f41f8…`**, identical to the canary-only revision (`bcaba714` / `5905003e`).
|
||
- After the deploy:
|
||
- `ns=3;s=canary-alarm-cycle` reads `BadNodeIdUnknown`.
|
||
- Both canary values are Good on both endpoints.
|
||
- `script-runner` reports `quarantined: 0`.
|
||
- ServiceLevel is 250/240.
|
||
- The temporary role row was deleted (`sql780.ps1 -Action delrole`: deleted=1, QA role rows 0).
|
||
- The forwarder was stopped and the browser tab closed.
|
||
|
||
## Part 2: DEV vd03, runner flip to OutOfProcess
|
||
|
||
**ConfigDb (read-only SQL from the Mac):**
|
||
|
||
- 0 scripts, 0 virtual tags, 0 scripted alarms. The only driver is FOCAS.
|
||
- Current deployment `595cb909` rev `fa9a2e28…`. The draft still drifts from it because of this morning's #903 edit-and-revert.
|
||
|
||
**Seams.** All three seams were set, the same as QA, even though vd03 has no scripts of any kind. The seam list only decides
|
||
where a future script runs, and DEV should match QA. Dropping every seam for kinds with no scripts would have left the flip
|
||
with no effect.
|
||
|
||
**No canary on vd03.** Any deploy would seal the drifted draft as a new revision, so `fa9a2e28…` could not be restored
|
||
afterwards. The worker start is therefore not observable on DEV: shards stay `not started` until a first script.
|
||
|
||
Stage `E:\ApiInstall\_stage-otopcua-780-flip\` (`flip780vd03.ps1 -Mode plan|apply|rollback`, `restart780vd03.ps1`, `envdigest.ps1`).
|
||
|
||
| Step | Result |
|
||
|---|---|
|
||
| Plan | Env **28** entries, `05db5d058fee` / `e3f876a407a8` (= this morning's). No `Scripting__*`. #901 record: InstallRoot `E:\ApiInstall\OtOpcUa`, LocalSystem, PeCacheDir `C:\ProgramData\OtOpcUa\script-pe-cache`, **ConfigureHostService 0**, 5 DenyPaths. Installer hashes `7D4078CB…` / `AE9D84FD…` (= the pack). |
|
||
| Backup | `E:\ApiInstall\OtOpcUa-backups\pre-20261113-780-runner-flip\{OtOpcUaHost.reg, OtOpcUaScriptWorker.reg}` (protected) |
|
||
| Installer, supported path | Full recorded arguments plus `-ConfigureHostService`, service up. 14 steps: **`SettingsRecord` changed** (now records ConfigureHostService=1) and **`HostServiceEnv` changed** (added `Scripting__Worker__Identity=NT SERVICE\OtOpcUaScriptWorker`, `Scripting__Worker__PeCacheDir=C:\ProgramData\OtOpcUa\script-pe-cache`). The other 12 steps were unchanged. 0 env entries removed. |
|
||
| Runner entries | `Scripting__Runner=OutOfProcess`, `Seams__0=Calculation`, `Seams__1=VirtualTag`, `Seams__2=ScriptedAlarm`, inserted ahead of the Worker entries |
|
||
| Env after | **34** entries, `69a493345099` / `caa047aaaf42`; diff = exactly those 6 added, 0 removed |
|
||
| Restart | 08:11:53.2Z → ready 08:12:02.3Z (**9.1 s**) |
|
||
| `-Verify` | **VERIFY OK 14/14** (13 before + `HostServiceEnv`) |
|
||
|
||
### vd03 gates
|
||
|
||
| Gate | Result |
|
||
|---|---|
|
||
| Health | `/health/ready` 200, 5/5 components; `script-runner`: **`Out-of-process script runner ready.`** (`state=Ready`, `peCache=true`, shards `not started`); `/health/active` 200 |
|
||
| ServiceLevel | **250** on three fresh-session reads (`opc.tcp://10.220.157.247:4840`) |
|
||
| Runner metrics | `otopcua_script_runner_state` 0; `otopcua_script_worker_up` 0 for every shard (idle, no scripts) |
|
||
| FOCAS | 22 raw variables: **21 Good**, `Program/BlockCount` `BadNoData` (as before) |
|
||
| Boot log | `roles=[admin,driver]`; `raw subtree materialised (containers=10, tags=22, … failed=0)`; credential-reference preflight clean; no FTL, no ScriptWorker line. ERR lines were only the certificate-domain noise from the gate's own reads by IP. |
|
||
|
||
## Rollback
|
||
|
||
- **QA seam:**
|
||
1. Run `E:\psrem-stage\otopcua-780-scriptedalarm\seam780.ps1 -Mode rollback` on each node. It removes only `Seams__2`.
|
||
2. Restart the Secondary, then the Primary (`restart780.ps1`).
|
||
3. Alternatively, import `backup\OtOpcUaHost-pre-780-seam2.reg`.
|
||
- **vd03 flip:**
|
||
1. Run `E:\ApiInstall\_stage-otopcua-780-flip\flip780vd03.ps1 -Mode rollback`. It removes the four `Scripting__Runner*` entries and keeps the Worker entries, which are harmless under InProcess.
|
||
2. Restart.
|
||
3. For the full pre-flip state, import both `.reg` files from `E:\ApiInstall\OtOpcUa-backups\pre-20261113-780-runner-flip\`. They restore the host env and the #901 record with ConfigureHostService=0.
|