dohertj2/lmxopcua
1
0
Fork 0
Code Issues 5 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(deploy): #780 QA ScriptedAlarm seam applied (drill pending) and vd03 runner flip to OutOfProcess
v2-ci / unit-tests (push) Has been cancelled
Details

Browse Source 
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
Joseph Doherty
2026-11-13 03:15:34 -05:00
parent 8667ce58aa
commit 2688bd5cfd
1 changed files with 134 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/deploy/2026-11-13-780-scriptedalarm-qa-and-vd03-flip.md
+134
View File
@@ -0,0 +1,134 @@
# lmxopcua#780: ScriptedAlarm seam on QA vq03/vq04 and the runner flip on DEV vd03 (2026-11-13)
> **Status: Part 1 (QA seam) applied, ScriptedAlarm drill PENDING. Part 2 (vd03 flip) DONE.** All three nodes run
> `c9640f5c` (see [`2026-11-13-vd03-qa-to-c9640f5c.md`](2026-11-13-vd03-qa-to-c9640f5c.md)). The owner approved this on
> 2026-11-13, with WinRM limited to the deploy-type steps: env change, `.reg` backups, restarts, `-Verify`, and the
> temporary AdminUI role-mapping row.
>
> - Health, OPC UA, `/metrics` and ConfigDb checks ran from the Mac over HTTP, opc.tcp and SQL (vd03).
> - Logs were read over SMB.
> - No other service was touched.
> - No Galaxy writes, no historian writes.
## Part 1: QA, `Scripting__Runner__Seams__2=ScriptedAlarm`
Stage `E:\psrem-stage\otopcua-780-scriptedalarm\` on each node:
- `seam780.ps1 -Mode plan|apply|rollback`;
- `restart780.ps1`, which restarts the service, waits for `/health/ready`, then runs `-Verify` from the `c9640f5c` stage's `install\`;
- `sql780.ps1 -Action show|addrole|delrole`, which uses the service's own ConfigDb connection, with the password in `SQLCMDPASSWORD` and never printed;
- `envdigest.ps1`.
The backup folder `backup\` has a protected DACL (SYSTEM + Administrators).
**Before:**
- ServiceLevel vq03 **240**, vq04 **250**: Primary vq04, as left by this morning's deploy.
- Both nodes: `/health/ready` 200, `Out-of-process script runner ready.`, shards 0 and 1 up.
- Canary Good on both nodes.
- ConfigDb (QA): 2 scripts (the canary), 1 VT, **0 scripted alarms**, 0 role mappings. Current deployment `5905003e` rev
`82dd0598c37f41f8…`, the canary-only revision.
| Step | vq03 | vq04 |
|---|---|---|
| `.reg` export | `backup\OtOpcUaHost-pre-780-seam2.reg` (31,002 B) | same name (30,244 B) |
| Env before (names + SHA only) | 48 entries, ordered `a04ca3f1fecb` / sorted `78e6acfab539` (= this morning's post-deploy digest) | 47 entries, `c6adaf68ef48` / `5df6eacf7152` (= this morning's) |
| Flags before | `Scripting__Runner=OutOfProcess`, `Seams__0=Calculation`, `Seams__1=VirtualTag`, `Seams__2` absent, `Scripting__Worker__*` present | same |
| Change | `Scripting__Runner__Seams__2=ScriptedAlarm` inserted right after `Seams__1`, so the installer's two `Scripting__Worker__*` entries stay last | same |
| Env after | 49 entries, `5750bbeb4734` / `4263c3ac5a46`; diff = exactly the one added entry, 0 removed | 48 entries, `ef146b2681b4` / `d19e968212f3`; same diff |
| Restart (one at a time, Secondary first) | 08:06:30.9Z → ready 08:06:41.1Z (**10.2 s**). SL after: vq03 240, vq04 250 | 08:07:21.2Z → ready 08:08:00.4Z (**39.2 s**; the stop alone took ~30 s, see the deviation below) |
| `-Verify` after the restart | **VERIFY OK 21/21** | **VERIFY OK 21/21** |
**Failover** (vq03 sampled every ~4 s while vq04 restarted):
- vq03 stayed **240 / `/health/active` 503** from 08:07:19Z to 08:07:48Z.
- From 08:07:52Z it read **250 / 200** on every sample.
- After vq04 was back: vq03 **250** (Primary), vq04 **240**, with no failback.
**After both restarts:**
- Both nodes: `/health/ready` 200, 5/5 components, `Out-of-process script runner ready.`, akka 2 members / 0 unreachable.
- `/health/active` 200 on vq03 and 503 on vq04.
- Canary `canary-calc-seconds` and `canary-vt-second` Good and advancing on both endpoints.
There is no startup log line that lists the selected seams (`ScriptRunnerSelection` logs only warnings), so the env entry
is the evidence until a scripted alarm runs. The drill's proof that the seam is live is
`otopcua_script_ipc_roundtrip{kind="alarm"}` counting on both nodes.
### Deviation: a planned stop of the Primary takes ~30 s (filed as lmxopcua#906)
vq04 logged `Application is shutting down...` at 03:07:21.289 EST. It then logged nothing but DbCommand lines until 03:07:51.292:
- `The request stream was aborted`;
- `HTTP POST /localdb_sync.v1.LocalDbSync/Sync responded 200 in 76909 ms`.
Only after that did it run Akka `CoordinatedShutdown` → `Leaving` → OldestChanged to vq03.
The host waits on the peer's open LocalDb sync stream until Kestrel's 30 s shutdown timeout. Throughout that wait the stopping Primary keeps vq03 at 240. The sync client (vq03) stops in about 1 s, so the earlier failover gates, which stopped vq03, never showed this.
### ScriptedAlarm drill: PENDING
Prepared, not run. The QA AdminUI needs a signed-in session: QA has no `Security:DeployApiKey`, and authoring and deploying
go through the AdminUI. The temporary `LdapGroupRoleMapping` row (`SCADA-Admins` → `Administrator`, system-wide, Notes
`lmxopcua#780 drill temp`) was added with `sql780.ps1 -Action addrole`, and the local forwarder `localhost:19003` →
`wonder-app-vq03:9000` was started. The agent does not type the LDAP password itself, so the drill waits for an operator
sign-in.
Planned content (all on the canary equipment `EQ-ec3979550d90`; **HistorizeToAveva off** on every alarm, so no historian write):
| Object | Value |
|---|---|
| Script `ScriptRunnerCanary-AlarmPred` | `return ((int)ctx.GetTag("script-runner-canary/Device1/canary-calc-seconds").Value) % 20 >= 10;` (input changes every second; active for 10 s, inactive for 10 s) |
| Alarm `canary-alarm-cycle` | AlarmCondition, severity 100, predicate above, Retain on, HistorizeToAveva **off** |
| Script `ScriptRunnerCanary-AlarmRunaway` | `var v = ctx.GetTag("script-runner-canary/Device1/canary-calc-seconds").Value; while (true) { }` |
| Alarm `canary-alarm-runaway` | AlarmCondition, severity 100, HistorizeToAveva **off**, deployed only after the normal alarm has fired and cleared and `ipc_roundtrip{kind="alarm"}` counts on both nodes |
Cleanup: delete both alarms and both scripts, deploy, and expect the revision back at `82dd0598…`. Delete the role row
(`sql780.ps1 -Action delrole`) and stop the forwarder.
## Part 2: DEV vd03, runner flip to OutOfProcess
**ConfigDb (read-only SQL from the Mac):**
- 0 scripts, 0 virtual tags, 0 scripted alarms. The only driver is FOCAS.
- Current deployment `595cb909` rev `fa9a2e28…`. The draft still drifts from it because of this morning's #903 edit-and-revert.
**Seams.** All three seams were set, the same as QA, even though vd03 has no scripts of any kind. The seam list only decides
where a future script runs, and DEV should match QA. Dropping every seam for kinds with no scripts would have left the flip
with no effect.
**No canary on vd03.** Any deploy would seal the drifted draft as a new revision, so `fa9a2e28…` could not be restored
afterwards. The worker start is therefore not observable on DEV: shards stay `not started` until a first script.
Stage `E:\ApiInstall\_stage-otopcua-780-flip\` (`flip780vd03.ps1 -Mode plan|apply|rollback`, `restart780vd03.ps1`, `envdigest.ps1`).
| Step | Result |
|---|---|
| Plan | Env **28** entries, `05db5d058fee` / `e3f876a407a8` (= this morning's). No `Scripting__*`. #901 record: InstallRoot `E:\ApiInstall\OtOpcUa`, LocalSystem, PeCacheDir `C:\ProgramData\OtOpcUa\script-pe-cache`, **ConfigureHostService 0**, 5 DenyPaths. Installer hashes `7D4078CB…` / `AE9D84FD…` (= the pack). |
| Backup | `E:\ApiInstall\OtOpcUa-backups\pre-20261113-780-runner-flip\{OtOpcUaHost.reg, OtOpcUaScriptWorker.reg}` (protected) |
| Installer, supported path | Full recorded arguments plus `-ConfigureHostService`, service up. 14 steps: **`SettingsRecord` changed** (now records ConfigureHostService=1) and **`HostServiceEnv` changed** (added `Scripting__Worker__Identity=NT SERVICE\OtOpcUaScriptWorker`, `Scripting__Worker__PeCacheDir=C:\ProgramData\OtOpcUa\script-pe-cache`). The other 12 steps were unchanged. 0 env entries removed. |
| Runner entries | `Scripting__Runner=OutOfProcess`, `Seams__0=Calculation`, `Seams__1=VirtualTag`, `Seams__2=ScriptedAlarm`, inserted ahead of the Worker entries |
| Env after | **34** entries, `69a493345099` / `caa047aaaf42`; diff = exactly those 6 added, 0 removed |
| Restart | 08:11:53.2Z → ready 08:12:02.3Z (**9.1 s**) |
| `-Verify` | **VERIFY OK 14/14** (13 before + `HostServiceEnv`) |
### vd03 gates
| Gate | Result |
|---|---|
| Health | `/health/ready` 200, 5/5 components; `script-runner`: **`Out-of-process script runner ready.`** (`state=Ready`, `peCache=true`, shards `not started`); `/health/active` 200 |
| ServiceLevel | **250** on three fresh-session reads (`opc.tcp://10.220.157.247:4840`) |
| Runner metrics | `otopcua_script_runner_state` 0; `otopcua_script_worker_up` 0 for every shard (idle, no scripts) |
| FOCAS | 22 raw variables: **21 Good**, `Program/BlockCount` `BadNoData` (as before) |
| Boot log | `roles=[admin,driver]`; `raw subtree materialised (containers=10, tags=22, … failed=0)`; credential-reference preflight clean; no FTL, no ScriptWorker line. ERR lines were only the certificate-domain noise from the gate's own reads by IP. |
## Rollback
- **QA seam:**
1. Run `E:\psrem-stage\otopcua-780-scriptedalarm\seam780.ps1 -Mode rollback` on each node. It removes only `Seams__2`.
2. Restart the Secondary, then the Primary (`restart780.ps1`).
3. Alternatively, import `backup\OtOpcUaHost-pre-780-seam2.reg`.
- **vd03 flip:**
1. Run `E:\ApiInstall\_stage-otopcua-780-flip\flip780vd03.ps1 -Mode rollback`. It removes the four `Scripting__Runner*` entries and keeps the Worker entries, which are harmless under InProcess.
2. Restart.
3. For the full pre-flip state, import both `.reg` files from `E:\ApiInstall\OtOpcUa-backups\pre-20261113-780-runner-flip\`. They restore the host env and the #901 record with ConfigureHostService=0.