Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
13 KiB
lmxopcua#780: ScriptedAlarm seam on QA vq03/vq04 and the runner flip on DEV vd03 (2026-11-13)
Status: EXECUTED. Part 1 (QA ScriptedAlarm seam + alarm drill) PASS; Part 2 (vd03 flip) DONE. All three nodes run
c9640f5c(see2026-11-13-vd03-qa-to-c9640f5c.md). The owner approved this on 2026-11-13, with WinRM limited to the deploy-type steps: env change,.regbackups, restarts,-Verify, and the temporary AdminUI role-mapping row.
- Health, OPC UA,
/metricsand ConfigDb checks ran from the Mac over HTTP, opc.tcp and SQL (vd03).- Logs were read over SMB.
- No other service was touched.
- No Galaxy writes, no historian writes.
Part 1: QA, Scripting__Runner__Seams__2=ScriptedAlarm
Stage E:\psrem-stage\otopcua-780-scriptedalarm\ on each node:
seam780.ps1 -Mode plan|apply|rollback;restart780.ps1, which restarts the service, waits for/health/ready, then runs-Verifyfrom thec9640f5cstage'sinstall\;sql780.ps1 -Action show|addrole|delrole, which uses the service's own ConfigDb connection, with the password inSQLCMDPASSWORDand never printed;envdigest.ps1.
The backup folder backup\ has a protected DACL (SYSTEM + Administrators).
Before:
- ServiceLevel vq03 240, vq04 250: Primary vq04, as left by this morning's deploy.
- Both nodes:
/health/ready200,Out-of-process script runner ready., shards 0 and 1 up. - Canary Good on both nodes.
- ConfigDb (QA): 2 scripts (the canary), 1 VT, 0 scripted alarms, 0 role mappings. Current deployment
5905003erev82dd0598c37f41f8…, the canary-only revision.
| Step | vq03 | vq04 |
|---|---|---|
.reg export |
backup\OtOpcUaHost-pre-780-seam2.reg (31,002 B) |
same name (30,244 B) |
| Env before (names + SHA only) | 48 entries, ordered a04ca3f1fecb / sorted 78e6acfab539 (= this morning's post-deploy digest) |
47 entries, c6adaf68ef48 / 5df6eacf7152 (= this morning's) |
| Flags before | Scripting__Runner=OutOfProcess, Seams__0=Calculation, Seams__1=VirtualTag, Seams__2 absent, Scripting__Worker__* present |
same |
| Change | Scripting__Runner__Seams__2=ScriptedAlarm inserted right after Seams__1, so the installer's two Scripting__Worker__* entries stay last |
same |
| Env after | 49 entries, 5750bbeb4734 / 4263c3ac5a46; diff = exactly the one added entry, 0 removed |
48 entries, ef146b2681b4 / d19e968212f3; same diff |
| Restart (one at a time, Secondary first) | 08:06:30.9Z → ready 08:06:41.1Z (10.2 s). SL after: vq03 240, vq04 250 | 08:07:21.2Z → ready 08:08:00.4Z (39.2 s; the stop alone took ~30 s, see the deviation below) |
-Verify after the restart |
VERIFY OK 21/21 | VERIFY OK 21/21 |
Failover (vq03 sampled every ~4 s while vq04 restarted):
- vq03 stayed 240 /
/health/active503 from 08:07:19Z to 08:07:48Z. - From 08:07:52Z it read 250 / 200 on every sample.
- After vq04 was back: vq03 250 (Primary), vq04 240, with no failback.
After both restarts:
- Both nodes:
/health/ready200, 5/5 components,Out-of-process script runner ready., akka 2 members / 0 unreachable. /health/active200 on vq03 and 503 on vq04.- Canary
canary-calc-secondsandcanary-vt-secondGood and advancing on both endpoints.
There is no startup log line that lists the selected seams (ScriptRunnerSelection logs only warnings), so the env entry
is the evidence until a scripted alarm runs. The drill's proof that the seam is live is
otopcua_script_ipc_roundtrip{kind="alarm"} counting on both nodes.
Deviation: a planned stop of the Primary takes ~30 s (filed as lmxopcua#906)
vq04 logged Application is shutting down... at 03:07:21.289 EST. It then logged nothing but DbCommand lines until 03:07:51.292:
The request stream was aborted;HTTP POST /localdb_sync.v1.LocalDbSync/Sync responded 200 in 76909 ms.
Only after that did it run Akka CoordinatedShutdown → Leaving → OldestChanged to vq03.
The host waits on the peer's open LocalDb sync stream until Kestrel's 30 s shutdown timeout. Throughout that wait the stopping Primary keeps vq03 at 240. The sync client (vq03) stops in about 1 s, so the earlier failover gates, which stopped vq03, never showed this.
ScriptedAlarm drill (2026-11-13, 11:44–11:53Z): PASS
Access.
- The temporary
LdapGroupRoleMappingrow (SCADA-Admins→Administrator, system-wide, Noteslmxopcua#780 drill temp) was added withsql780.ps1 -Action addrole(on-box SQL with the service's own connection). - The local forwarder
localhost:19003→wonder-app-vq03:9000(socat) was started. - The owner signed in to the AdminUI (session
multi-role, Administrator). The agent did not type a credential. - QA OtOpcUa authenticates against the box-local GLAuth (
127.0.0.1:3893,dc=scadabridge,dc=local), not the shareddc=zbdirectory.
Authored in the AdminUI on the canary equipment EQ-ec3979550d90. HistorizeToAveva was off on both alarms, checked in
the ConfigDb (historize=0) before each deploy.
| Object | Value |
|---|---|
Script ScriptRunnerCanary-AlarmPred (SC-3e4f868120bb) |
return ((int)ctx.GetTag("script-runner-canary/Device1/canary-calc-seconds").Value) % 20 >= 10; |
Alarm canary-alarm-cycle |
AlarmCondition, severity 100, Retain on, HistorizeToAveva off |
Script ScriptRunnerCanary-AlarmRunaway (SC-d3f7ed7b8d74) |
var v = ctx.GetTag("script-runner-canary/Device1/canary-calc-seconds").Value; then while (true) { } |
Alarm canary-alarm-runaway |
AlarmCondition, severity 100, Retain on, HistorizeToAveva off |
During the drill vq03 was the Primary (250) and vq04 the Secondary (240), as left by Part 1.
1. Normal scripted alarm. Deployed as fbfba004 (rev 5c8ea584…, 11:46:13Z).
alarms --refresh(OtOpcUa CLI, anonymous) on both endpoints,ns=3;s=canary-alarm-cycle: Inactive at 11:46:20 → Active 11:46:30 → Inactive 11:46:40 → Active …, the same 10 s cycle on vq03 and vq04.- The predicate runs in the worker:
otopcua_script_ipc_roundtrip_microseconds_count{kind="alarm"}was 66 on vq03 and 67 on vq04 within a minute. This is the proof that theScriptedAlarmseam is live on both nodes. - The events carry severity 250: the engine bands the configured 100 into
AlarmSeverityLow, as designed.
2. Runaway scripted alarm. Added and deployed as b0177ec4 (rev a9320d56…, 11:48:53Z).
During the run, each endpoint was sampled 22 times over about 2 min 15 s (11:48:42–11:50:57Z), reading ServiceLevel, both canary values and /health/ready. Alarm events were captured on both endpoints the whole time.
| Observed | vq03 (Primary) | vq04 (Secondary) |
|---|---|---|
| Kills | 06:48:59.225 EST: Script 3:canary-alarm-runaway exceeded its hard deadline (4000 ms); killing ScriptWorker[0] pid 5368 → ended: "KilledDeadline" (exit code -1, culprit 3:canary-alarm-runaway, 1 evaluation(s) and 0 compile(s) in flight); second kill at 06:49:04.163 (pid 14720) |
06:48:59.423 (pid 18096) and 06:49:04.495 (pid 18472), same lines |
| Quarantine | 06:49:04.167: Script "3:canary-alarm-runaway" quarantined: "it exceeded its hard deadline and its worker was killed (2 time(s) in the last hour); the script is not evaluated again until its source changes or it is redeployed" |
06:49:04.499, same |
| Shard 0 restart | ScriptWorker[0] started at +0.58 s (pid 14720) and +0.75 s (pid 32204) |
+0.61 s (pid 18472) and +0.80 s (pid 14432) |
| Metrics after | worker_exits_total{reason="killed_deadline",shard="0"} 2, quarantined_total{reason="killed",script="3:canary-alarm-runaway"} 1, run_refused_total 6, runner_state 0, worker_up shards 0/1 = 1, compile_total hit 8 / miss 2 |
identical |
Sibling alarm canary-alarm-cycle (same shard family) |
kept its exact 10 s cycle straight through both kills: Active 11:48:50, Inactive 11:49:00, Active 11:49:10 … Inactive 11:51:00; no transition missed | same, about 0.3 s later each |
| Sibling Calc + VT canaries | 22/22 samples Good and advancing | 22/22 Good |
| Node | /health/ready 200 on 22/22; script-runner Healthy, quarantined: 1; /health/active 200 |
200 on 22/22; quarantined: 1; /health/active 503 |
| ServiceLevel | 250 on 22/22 | 240 on 22/22 |
| Host log, other ERR/FTL in 06:46–06:59 EST | only TCPSERVERCHANNEL … BadConnectionClosed at 06:47:15, from the drill's own CLI alarm subscriber being killed |
same |
HistorianGateway /metrics (vq03 :5220) |
HistorianStatus/Probe 755 → 813 and nothing else changed: no historian write or tag-creation RPC |
3. Cleanup.
- Both alarms and both drill scripts were deleted in the AdminUI. ConfigDb: 2 scripts (the canary), 1 VT, 0 scripted alarms.
- Deployed as
a8e078f0, rev82dd0598c37f41f8…, identical to the canary-only revision (bcaba714/5905003e). - After the deploy:
ns=3;s=canary-alarm-cyclereadsBadNodeIdUnknown.- Both canary values are Good on both endpoints.
script-runnerreportsquarantined: 0.- ServiceLevel is 250/240.
- The temporary role row was deleted (
sql780.ps1 -Action delrole: deleted=1, QA role rows 0). - The forwarder was stopped and the browser tab closed.
Part 2: DEV vd03, runner flip to OutOfProcess
ConfigDb (read-only SQL from the Mac):
- 0 scripts, 0 virtual tags, 0 scripted alarms. The only driver is FOCAS.
- Current deployment
595cb909revfa9a2e28…. The draft still drifts from it because of this morning's #903 edit-and-revert.
Seams. All three seams were set, the same as QA, even though vd03 has no scripts of any kind. The seam list only decides where a future script runs, and DEV should match QA. Dropping every seam for kinds with no scripts would have left the flip with no effect.
No canary on vd03. Any deploy would seal the drifted draft as a new revision, so fa9a2e28… could not be restored
afterwards. The worker start is therefore not observable on DEV: shards stay not started until a first script.
Stage E:\ApiInstall\_stage-otopcua-780-flip\ (flip780vd03.ps1 -Mode plan|apply|rollback, restart780vd03.ps1, envdigest.ps1).
| Step | Result |
|---|---|
| Plan | Env 28 entries, 05db5d058fee / e3f876a407a8 (= this morning's). No Scripting__*. #901 record: InstallRoot E:\ApiInstall\OtOpcUa, LocalSystem, PeCacheDir C:\ProgramData\OtOpcUa\script-pe-cache, ConfigureHostService 0, 5 DenyPaths. Installer hashes 7D4078CB… / AE9D84FD… (= the pack). |
| Backup | E:\ApiInstall\OtOpcUa-backups\pre-20261113-780-runner-flip\{OtOpcUaHost.reg, OtOpcUaScriptWorker.reg} (protected) |
| Installer, supported path | Full recorded arguments plus -ConfigureHostService, service up. 14 steps: SettingsRecord changed (now records ConfigureHostService=1) and HostServiceEnv changed (added Scripting__Worker__Identity=NT SERVICE\OtOpcUaScriptWorker, Scripting__Worker__PeCacheDir=C:\ProgramData\OtOpcUa\script-pe-cache). The other 12 steps were unchanged. 0 env entries removed. |
| Runner entries | Scripting__Runner=OutOfProcess, Seams__0=Calculation, Seams__1=VirtualTag, Seams__2=ScriptedAlarm, inserted ahead of the Worker entries |
| Env after | 34 entries, 69a493345099 / caa047aaaf42; diff = exactly those 6 added, 0 removed |
| Restart | 08:11:53.2Z → ready 08:12:02.3Z (9.1 s) |
-Verify |
VERIFY OK 14/14 (13 before + HostServiceEnv) |
vd03 gates
| Gate | Result |
|---|---|
| Health | /health/ready 200, 5/5 components; script-runner: Out-of-process script runner ready. (state=Ready, peCache=true, shards not started); /health/active 200 |
| ServiceLevel | 250 on three fresh-session reads (opc.tcp://10.220.157.247:4840) |
| Runner metrics | otopcua_script_runner_state 0; otopcua_script_worker_up 0 for every shard (idle, no scripts) |
| FOCAS | 22 raw variables: 21 Good, Program/BlockCount BadNoData (as before) |
| Boot log | roles=[admin,driver]; raw subtree materialised (containers=10, tags=22, … failed=0); credential-reference preflight clean; no FTL, no ScriptWorker line. ERR lines were only the certificate-domain noise from the gate's own reads by IP. |
Rollback
- QA seam:
- Run
E:\psrem-stage\otopcua-780-scriptedalarm\seam780.ps1 -Mode rollbackon each node. It removes onlySeams__2. - Restart the Secondary, then the Primary (
restart780.ps1). - Alternatively, import
backup\OtOpcUaHost-pre-780-seam2.reg.
- Run
- vd03 flip:
- Run
E:\ApiInstall\_stage-otopcua-780-flip\flip780vd03.ps1 -Mode rollback. It removes the fourScripting__Runner*entries and keeps the Worker entries, which are harmless under InProcess. - Restart.
- For the full pre-flip state, import both
.regfiles fromE:\ApiInstall\OtOpcUa-backups\pre-20261113-780-runner-flip\. They restore the host env and the #901 record with ConfigureHostService=0.
- Run