Files
lmxopcua/docs/deploy/2026-11-13-780-scriptedalarm-qa-and-vd03-flip.md
T

13 KiB
Raw Blame History

lmxopcua#780: ScriptedAlarm seam on QA vq03/vq04 and the runner flip on DEV vd03 (2026-11-13)

Status: EXECUTED. Part 1 (QA ScriptedAlarm seam + alarm drill) PASS; Part 2 (vd03 flip) DONE. All three nodes run c9640f5c (see 2026-11-13-vd03-qa-to-c9640f5c.md). The owner approved this on 2026-11-13, with WinRM limited to the deploy-type steps: env change, .reg backups, restarts, -Verify, and the temporary AdminUI role-mapping row.

  • Health, OPC UA, /metrics and ConfigDb checks ran from the Mac over HTTP, opc.tcp and SQL (vd03).
  • Logs were read over SMB.
  • No other service was touched.
  • No Galaxy writes, no historian writes.

Part 1: QA, Scripting__Runner__Seams__2=ScriptedAlarm

Stage E:\psrem-stage\otopcua-780-scriptedalarm\ on each node:

  • seam780.ps1 -Mode plan|apply|rollback;
  • restart780.ps1, which restarts the service, waits for /health/ready, then runs -Verify from the c9640f5c stage's install\;
  • sql780.ps1 -Action show|addrole|delrole, which uses the service's own ConfigDb connection, with the password in SQLCMDPASSWORD and never printed;
  • envdigest.ps1.

The backup folder backup\ has a protected DACL (SYSTEM + Administrators).

Before:

  • ServiceLevel vq03 240, vq04 250: Primary vq04, as left by this morning's deploy.
  • Both nodes: /health/ready 200, Out-of-process script runner ready., shards 0 and 1 up.
  • Canary Good on both nodes.
  • ConfigDb (QA): 2 scripts (the canary), 1 VT, 0 scripted alarms, 0 role mappings. Current deployment 5905003e rev 82dd0598c37f41f8…, the canary-only revision.
Step vq03 vq04
.reg export backup\OtOpcUaHost-pre-780-seam2.reg (31,002 B) same name (30,244 B)
Env before (names + SHA only) 48 entries, ordered a04ca3f1fecb / sorted 78e6acfab539 (= this morning's post-deploy digest) 47 entries, c6adaf68ef48 / 5df6eacf7152 (= this morning's)
Flags before Scripting__Runner=OutOfProcess, Seams__0=Calculation, Seams__1=VirtualTag, Seams__2 absent, Scripting__Worker__* present same
Change Scripting__Runner__Seams__2=ScriptedAlarm inserted right after Seams__1, so the installer's two Scripting__Worker__* entries stay last same
Env after 49 entries, 5750bbeb4734 / 4263c3ac5a46; diff = exactly the one added entry, 0 removed 48 entries, ef146b2681b4 / d19e968212f3; same diff
Restart (one at a time, Secondary first) 08:06:30.9Z → ready 08:06:41.1Z (10.2 s). SL after: vq03 240, vq04 250 08:07:21.2Z → ready 08:08:00.4Z (39.2 s; the stop alone took ~30 s, see the deviation below)
-Verify after the restart VERIFY OK 21/21 VERIFY OK 21/21

Failover (vq03 sampled every ~4 s while vq04 restarted):

  • vq03 stayed 240 / /health/active 503 from 08:07:19Z to 08:07:48Z.
  • From 08:07:52Z it read 250 / 200 on every sample.
  • After vq04 was back: vq03 250 (Primary), vq04 240, with no failback.

After both restarts:

  • Both nodes: /health/ready 200, 5/5 components, Out-of-process script runner ready., akka 2 members / 0 unreachable.
  • /health/active 200 on vq03 and 503 on vq04.
  • Canary canary-calc-seconds and canary-vt-second Good and advancing on both endpoints.

There is no startup log line that lists the selected seams (ScriptRunnerSelection logs only warnings), so the env entry is the evidence until a scripted alarm runs. The drill's proof that the seam is live is otopcua_script_ipc_roundtrip{kind="alarm"} counting on both nodes.

Deviation: a planned stop of the Primary takes ~30 s (filed as lmxopcua#906)

vq04 logged Application is shutting down... at 03:07:21.289 EST. It then logged nothing but DbCommand lines until 03:07:51.292:

  • The request stream was aborted;
  • HTTP POST /localdb_sync.v1.LocalDbSync/Sync responded 200 in 76909 ms.

Only after that did it run Akka CoordinatedShutdown → Leaving → OldestChanged to vq03.

The host waits on the peer's open LocalDb sync stream until Kestrel's 30 s shutdown timeout. Throughout that wait the stopping Primary keeps vq03 at 240. The sync client (vq03) stops in about 1 s, so the earlier failover gates, which stopped vq03, never showed this.

ScriptedAlarm drill (2026-11-13, 11:44–11:53Z): PASS

Access.

  • The temporary LdapGroupRoleMapping row (SCADA-Admins → Administrator, system-wide, Notes lmxopcua#780 drill temp) was added with sql780.ps1 -Action addrole (on-box SQL with the service's own connection).
  • The local forwarder localhost:19003 → wonder-app-vq03:9000 (socat) was started.
  • The owner signed in to the AdminUI (session multi-role, Administrator). The agent did not type a credential.
  • QA OtOpcUa authenticates against the box-local GLAuth (127.0.0.1:3893, dc=scadabridge,dc=local), not the shared dc=zb directory.

Authored in the AdminUI on the canary equipment EQ-ec3979550d90. HistorizeToAveva was off on both alarms, checked in the ConfigDb (historize=0) before each deploy.

Object Value
Script ScriptRunnerCanary-AlarmPred (SC-3e4f868120bb) return ((int)ctx.GetTag("script-runner-canary/Device1/canary-calc-seconds").Value) % 20 >= 10;
Alarm canary-alarm-cycle AlarmCondition, severity 100, Retain on, HistorizeToAveva off
Script ScriptRunnerCanary-AlarmRunaway (SC-d3f7ed7b8d74) var v = ctx.GetTag("script-runner-canary/Device1/canary-calc-seconds").Value; then while (true) { }
Alarm canary-alarm-runaway AlarmCondition, severity 100, Retain on, HistorizeToAveva off

During the drill vq03 was the Primary (250) and vq04 the Secondary (240), as left by Part 1.

1. Normal scripted alarm. Deployed as fbfba004 (rev 5c8ea584…, 11:46:13Z).

  • alarms --refresh (OtOpcUa CLI, anonymous) on both endpoints, ns=3;s=canary-alarm-cycle: Inactive at 11:46:20 → Active 11:46:30 → Inactive 11:46:40 → Active …, the same 10 s cycle on vq03 and vq04.
  • The predicate runs in the worker: otopcua_script_ipc_roundtrip_microseconds_count{kind="alarm"} was 66 on vq03 and 67 on vq04 within a minute. This is the proof that the ScriptedAlarm seam is live on both nodes.
  • The events carry severity 250: the engine bands the configured 100 into AlarmSeverity Low, as designed.

2. Runaway scripted alarm. Added and deployed as b0177ec4 (rev a9320d56…, 11:48:53Z).

During the run, each endpoint was sampled 22 times over about 2 min 15 s (11:48:42–11:50:57Z), reading ServiceLevel, both canary values and /health/ready. Alarm events were captured on both endpoints the whole time.

Observed vq03 (Primary) vq04 (Secondary)
Kills 06:48:59.225 EST: Script 3:canary-alarm-runaway exceeded its hard deadline (4000 ms); killing ScriptWorker[0] pid 5368 → ended: "KilledDeadline" (exit code -1, culprit 3:canary-alarm-runaway, 1 evaluation(s) and 0 compile(s) in flight); second kill at 06:49:04.163 (pid 14720) 06:48:59.423 (pid 18096) and 06:49:04.495 (pid 18472), same lines
Quarantine 06:49:04.167: Script "3:canary-alarm-runaway" quarantined: "it exceeded its hard deadline and its worker was killed (2 time(s) in the last hour); the script is not evaluated again until its source changes or it is redeployed" 06:49:04.499, same
Shard 0 restart ScriptWorker[0] started at +0.58 s (pid 14720) and +0.75 s (pid 32204) +0.61 s (pid 18472) and +0.80 s (pid 14432)
Metrics after worker_exits_total{reason="killed_deadline",shard="0"} 2, quarantined_total{reason="killed",script="3:canary-alarm-runaway"} 1, run_refused_total 6, runner_state 0, worker_up shards 0/1 = 1, compile_total hit 8 / miss 2 identical
Sibling alarm canary-alarm-cycle (same shard family) kept its exact 10 s cycle straight through both kills: Active 11:48:50, Inactive 11:49:00, Active 11:49:10 … Inactive 11:51:00; no transition missed same, about 0.3 s later each
Sibling Calc + VT canaries 22/22 samples Good and advancing 22/22 Good
Node /health/ready 200 on 22/22; script-runner Healthy, quarantined: 1; /health/active 200 200 on 22/22; quarantined: 1; /health/active 503
ServiceLevel 250 on 22/22 240 on 22/22
Host log, other ERR/FTL in 06:46–06:59 EST only TCPSERVERCHANNEL … BadConnectionClosed at 06:47:15, from the drill's own CLI alarm subscriber being killed same
HistorianGateway /metrics (vq03 :5220) HistorianStatus/Probe 755 → 813 and nothing else changed: no historian write or tag-creation RPC

3. Cleanup.

  • Both alarms and both drill scripts were deleted in the AdminUI. ConfigDb: 2 scripts (the canary), 1 VT, 0 scripted alarms.
  • Deployed as a8e078f0, rev 82dd0598c37f41f8…, identical to the canary-only revision (bcaba714 / 5905003e).
  • After the deploy:
    • ns=3;s=canary-alarm-cycle reads BadNodeIdUnknown.
    • Both canary values are Good on both endpoints.
    • script-runner reports quarantined: 0.
    • ServiceLevel is 250/240.
  • The temporary role row was deleted (sql780.ps1 -Action delrole: deleted=1, QA role rows 0).
  • The forwarder was stopped and the browser tab closed.

Part 2: DEV vd03, runner flip to OutOfProcess

ConfigDb (read-only SQL from the Mac):

  • 0 scripts, 0 virtual tags, 0 scripted alarms. The only driver is FOCAS.
  • Current deployment 595cb909 rev fa9a2e28…. The draft still drifts from it because of this morning's #903 edit-and-revert.

Seams. All three seams were set, the same as QA, even though vd03 has no scripts of any kind. The seam list only decides where a future script runs, and DEV should match QA. Dropping every seam for kinds with no scripts would have left the flip with no effect.

No canary on vd03. Any deploy would seal the drifted draft as a new revision, so fa9a2e28… could not be restored afterwards. The worker start is therefore not observable on DEV: shards stay not started until a first script.

Stage E:\ApiInstall\_stage-otopcua-780-flip\ (flip780vd03.ps1 -Mode plan|apply|rollback, restart780vd03.ps1, envdigest.ps1).

Step Result
Plan Env 28 entries, 05db5d058fee / e3f876a407a8 (= this morning's). No Scripting__*. #901 record: InstallRoot E:\ApiInstall\OtOpcUa, LocalSystem, PeCacheDir C:\ProgramData\OtOpcUa\script-pe-cache, ConfigureHostService 0, 5 DenyPaths. Installer hashes 7D4078CB… / AE9D84FD… (= the pack).
Backup E:\ApiInstall\OtOpcUa-backups\pre-20261113-780-runner-flip\{OtOpcUaHost.reg, OtOpcUaScriptWorker.reg} (protected)
Installer, supported path Full recorded arguments plus -ConfigureHostService, service up. 14 steps: SettingsRecord changed (now records ConfigureHostService=1) and HostServiceEnv changed (added Scripting__Worker__Identity=NT SERVICE\OtOpcUaScriptWorker, Scripting__Worker__PeCacheDir=C:\ProgramData\OtOpcUa\script-pe-cache). The other 12 steps were unchanged. 0 env entries removed.
Runner entries Scripting__Runner=OutOfProcess, Seams__0=Calculation, Seams__1=VirtualTag, Seams__2=ScriptedAlarm, inserted ahead of the Worker entries
Env after 34 entries, 69a493345099 / caa047aaaf42; diff = exactly those 6 added, 0 removed
Restart 08:11:53.2Z → ready 08:12:02.3Z (9.1 s)
-Verify VERIFY OK 14/14 (13 before + HostServiceEnv)

vd03 gates

Gate Result
Health /health/ready 200, 5/5 components; script-runner: Out-of-process script runner ready. (state=Ready, peCache=true, shards not started); /health/active 200
ServiceLevel 250 on three fresh-session reads (opc.tcp://10.220.157.247:4840)
Runner metrics otopcua_script_runner_state 0; otopcua_script_worker_up 0 for every shard (idle, no scripts)
FOCAS 22 raw variables: 21 Good, Program/BlockCount BadNoData (as before)
Boot log roles=[admin,driver]; raw subtree materialised (containers=10, tags=22, … failed=0); credential-reference preflight clean; no FTL, no ScriptWorker line. ERR lines were only the certificate-domain noise from the gate's own reads by IP.

Rollback

  • QA seam:
    1. Run E:\psrem-stage\otopcua-780-scriptedalarm\seam780.ps1 -Mode rollback on each node. It removes only Seams__2.
    2. Restart the Secondary, then the Primary (restart780.ps1).
    3. Alternatively, import backup\OtOpcUaHost-pre-780-seam2.reg.
  • vd03 flip:
    1. Run E:\ApiInstall\_stage-otopcua-780-flip\flip780vd03.ps1 -Mode rollback. It removes the four Scripting__Runner* entries and keeps the Worker entries, which are harmless under InProcess.
    2. Restart.
    3. For the full pre-flip state, import both .reg files from E:\ApiInstall\OtOpcUa-backups\pre-20261113-780-runner-flip\. They restore the host env and the #901 record with ConfigureHostService=0.