dohertj2/lmxopcua
1
0
Fork 0
Code Issues 5 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(deploy): #780 QA scripted-alarm drill passed (kill x2, quarantine, siblings Good, 250/240); config back to rev 82dd0598
v2-ci / unit-tests (push) Has been cancelled
Details

Browse Source 
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
Joseph Doherty
2026-11-13 06:54:09 -05:00
parent 36eb6d558d
commit 003c7f8954
1 changed files with 52 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/deploy/2026-11-13-780-scriptedalarm-qa-and-vd03-flip.md
+52 -14
View File
@@ -1,6 +1,6 @@
# lmxopcua#780: ScriptedAlarm seam on QA vq03/vq04 and the runner flip on DEV vd03 (2026-11-13)
> **Status: Part 1 (QA seam) applied, ScriptedAlarm drill PENDING. Part 2 (vd03 flip) DONE.** All three nodes run
> **Status: EXECUTED. Part 1 (QA ScriptedAlarm seam + alarm drill) PASS; Part 2 (vd03 flip) DONE.** All three nodes run
> `c9640f5c` (see [`2026-11-13-vd03-qa-to-c9640f5c.md`](2026-11-13-vd03-qa-to-c9640f5c.md)). The owner approved this on
> 2026-11-13, with WinRM limited to the deploy-type steps: env change, `.reg` backups, restarts, `-Verify`, and the
> temporary AdminUI role-mapping row.
@@ -66,25 +66,63 @@ Only after that did it run Akka `CoordinatedShutdown` → `Leaving` → OldestCh
The host waits on the peer's open LocalDb sync stream until Kestrel's 30 s shutdown timeout. Throughout that wait the stopping Primary keeps vq03 at 240. The sync client (vq03) stops in about 1 s, so the earlier failover gates, which stopped vq03, never showed this.
### ScriptedAlarm drill: PENDING
### ScriptedAlarm drill (2026-11-13, 11:44–11:53Z): PASS
Prepared, not run. The QA AdminUI needs a signed-in session: QA has no `Security:DeployApiKey`, and authoring and deploying
go through the AdminUI. The temporary `LdapGroupRoleMapping` row (`SCADA-Admins` → `Administrator`, system-wide, Notes
`lmxopcua#780 drill temp`) was added with `sql780.ps1 -Action addrole`, and the local forwarder `localhost:19003` →
`wonder-app-vq03:9000` was started. The agent does not type the LDAP password itself, so the drill waits for an operator
sign-in.
**Access.**
- The temporary `LdapGroupRoleMapping` row (`SCADA-Admins` → `Administrator`, system-wide, Notes `lmxopcua#780 drill temp`)
was added with `sql780.ps1 -Action addrole` (on-box SQL with the service's own connection).
- The local forwarder `localhost:19003` → `wonder-app-vq03:9000` (socat) was started.
- **The owner signed in to the AdminUI** (session `multi-role`, Administrator). The agent did not type a credential.
- QA OtOpcUa authenticates against the box-local GLAuth (`127.0.0.1:3893`, `dc=scadabridge,dc=local`), not the shared
`dc=zb` directory.
Planned content (all on the canary equipment `EQ-ec3979550d90`; **HistorizeToAveva off** on every alarm, so no historian write):
Authored in the AdminUI on the canary equipment `EQ-ec3979550d90`. **HistorizeToAveva was off on both alarms**, checked in
the ConfigDb (`historize=0`) before each deploy.
| Object | Value |
|---|---|
| Script `ScriptRunnerCanary-AlarmPred` | `return ((int)ctx.GetTag("script-runner-canary/Device1/canary-calc-seconds").Value) % 20 >= 10;` (input changes every second; active for 10 s, inactive for 10 s) |
| Alarm `canary-alarm-cycle` | AlarmCondition, severity 100, predicate above, Retain on, HistorizeToAveva **off** |
| Script `ScriptRunnerCanary-AlarmRunaway` | `var v = ctx.GetTag("script-runner-canary/Device1/canary-calc-seconds").Value; while (true) { }` |
| Alarm `canary-alarm-runaway` | AlarmCondition, severity 100, HistorizeToAveva **off**, deployed only after the normal alarm has fired and cleared and `ipc_roundtrip{kind="alarm"}` counts on both nodes |
| Script `ScriptRunnerCanary-AlarmPred` (`SC-3e4f868120bb`) | `return ((int)ctx.GetTag("script-runner-canary/Device1/canary-calc-seconds").Value) % 20 >= 10;` |
| Alarm `canary-alarm-cycle` | AlarmCondition, severity 100, Retain on, HistorizeToAveva off |
| Script `ScriptRunnerCanary-AlarmRunaway` (`SC-d3f7ed7b8d74`) | `var v = ctx.GetTag("script-runner-canary/Device1/canary-calc-seconds").Value;` then `while (true) { }` |
| Alarm `canary-alarm-runaway` | AlarmCondition, severity 100, Retain on, HistorizeToAveva off |
Cleanup: delete both alarms and both scripts, deploy, and expect the revision back at `82dd0598…`. Delete the role row
(`sql780.ps1 -Action delrole`) and stop the forwarder.
During the drill vq03 was the Primary (**250**) and vq04 the Secondary (**240**), as left by Part 1.
**1. Normal scripted alarm.** Deployed as `fbfba004` (rev `5c8ea584…`, 11:46:13Z).
- `alarms --refresh` (OtOpcUa CLI, anonymous) on both endpoints, `ns=3;s=canary-alarm-cycle`: Inactive at 11:46:20 →
**Active 11:46:30 → Inactive 11:46:40 → Active …**, the same 10 s cycle on vq03 and vq04.
- The predicate runs in the worker: `otopcua_script_ipc_roundtrip_microseconds_count{kind="alarm"}` was 66 on vq03 and 67
on vq04 within a minute. This is the proof that the `ScriptedAlarm` seam is live on both nodes.
- The events carry severity **250**: the engine bands the configured 100 into `AlarmSeverity` Low, as designed.
**2. Runaway scripted alarm.** Added and deployed as `b0177ec4` (rev `a9320d56…`, 11:48:53Z).
During the run, each endpoint was sampled 22 times over about 2 min 15 s (11:48:42–11:50:57Z), reading ServiceLevel, both canary values and `/health/ready`. Alarm events were captured on both endpoints the whole time.
| Observed | vq03 (Primary) | vq04 (Secondary) |
|---|---|---|
| Kills | 06:48:59.225 EST: `Script 3:canary-alarm-runaway exceeded its hard deadline (4000 ms); killing ScriptWorker[0] pid 5368` → `ended: "KilledDeadline" (exit code -1, culprit 3:canary-alarm-runaway, 1 evaluation(s) and 0 compile(s) in flight)`; second kill at 06:49:04.163 (pid 14720) | 06:48:59.423 (pid 18096) and 06:49:04.495 (pid 18472), same lines |
| Quarantine | 06:49:04.167: `Script "3:canary-alarm-runaway" quarantined: "it exceeded its hard deadline and its worker was killed (2 time(s) in the last hour); the script is not evaluated again until its source changes or it is redeployed"` | 06:49:04.499, same |
| Shard 0 restart | `ScriptWorker[0] started` at +0.58 s (pid 14720) and +0.75 s (pid 32204) | +0.61 s (pid 18472) and +0.80 s (pid 14432) |
| Metrics after | `worker_exits_total{reason="killed_deadline",shard="0"}` 2, `quarantined_total{reason="killed",script="3:canary-alarm-runaway"}` 1, `run_refused_total` 6, `runner_state` 0, `worker_up` shards 0/1 = 1, `compile_total` hit 8 / miss 2 | identical |
| Sibling alarm `canary-alarm-cycle` (same shard family) | **kept its exact 10 s cycle straight through both kills**: Active 11:48:50, Inactive 11:49:00, Active 11:49:10 … Inactive 11:51:00; no transition missed | same, about 0.3 s later each |
| Sibling Calc + VT canaries | 22/22 samples **Good** and advancing | 22/22 Good |
| Node | `/health/ready` 200 on 22/22; `script-runner` Healthy, `quarantined: 1`; `/health/active` 200 | 200 on 22/22; `quarantined: 1`; `/health/active` 503 |
| ServiceLevel | **250** on 22/22 | **240** on 22/22 |
| Host log, other ERR/FTL in 06:46–06:59 EST | only `TCPSERVERCHANNEL … BadConnectionClosed` at 06:47:15, from the drill's own CLI alarm subscriber being killed | same |
| HistorianGateway `/metrics` (vq03 `:5220`) | `HistorianStatus/Probe` 755 → 813 and nothing else changed: **no historian write or tag-creation RPC** | |
**3. Cleanup.**
- Both alarms and both drill scripts were deleted in the AdminUI. ConfigDb: 2 scripts (the canary), 1 VT, 0 scripted alarms.
- Deployed as **`a8e078f0`, rev `82dd0598c37f41f8…`**, identical to the canary-only revision (`bcaba714` / `5905003e`).
- After the deploy:
- `ns=3;s=canary-alarm-cycle` reads `BadNodeIdUnknown`.
- Both canary values are Good on both endpoints.
- `script-runner` reports `quarantined: 0`.
- ServiceLevel is 250/240.
- The temporary role row was deleted (`sql780.ps1 -Action delrole`: deleted=1, QA role rows 0).
- The forwarder was stopped and the browser tab closed.
## Part 2: DEV vd03, runner flip to OutOfProcess