Files
lmxopcua/docs/deploy/qa-upgrade-b8b4c18f-to-fea7de0a.md
T
Joseph Doherty c80d187c18
v2-ci / unit-tests (push) Has been cancelled
docs(deploy): QA vq03/vq04 + DEV vd03 upgraded to master fea7de0a (SDK 2.0.1) - #801 #793 #794 #780
DEV vd03 5fb4b4db -> fea7de0a (ServiceLevel 255 -> 250, 23/24 values as before).
QA pair b8b4c18f -> fea7de0a with stop-both/start-both: ConfigDb preflight all ok
(empty configuration), snapshot-isolation migration, isolated preflight boot +
pepper negative control, ScriptWorker isolation installed (idempotent, rollback
drill), runner OutOfProcess on Calculation + VirtualTag. Real pair 250/240,
LocalDb 0.3.7 sync connected with an auth negative control, failover and no-failback
verified. Follow-up #890 (isolation does not cover sibling pki/data).

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 17:03:05 -05:00

179 lines
15 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# QA wonder-app-vq03/vq04 upgrade `b8b4c18f` → master `fea7de0a` (SDK 2.0.1), with the DEV rehearsal
> **Status: EXECUTED 2026-11-12.** DEV vd03 runs `fea7de0a` (from `5fb4b4db`). The QA pair vq03 + vq04 runs
> `fea7de0a` (from `b8b4c18f`) with the out-of-process script runner on (Calculation + VirtualTag seams) and the
> ScriptWorker isolation installed. Tracking: #801 (SDK 2 live gate), #793 (ConfigDb preflight), #794 (worker
> isolation install), #780 (P3 runner flip). Template: [`vd03-upgrade-b8b4c18f-to-master.md`](vd03-upgrade-b8b4c18f-to-master.md).
Channel: WinRM + SMB from the Mac (the `winrm` skill), used only for the on-box steps of this deploy, which the owner
requested on 2026-11-12. Health, OPC UA, ScadaBridge and Overview checks ran from the Mac over HTTP / opc.tcp.
## Build
- Source: `origin/master` **`fea7de0a`** (`fea7de0aefd905522d187dad495c3c8163a5ec03`), built in an isolated worktree.
- `dotnet publish src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj -c Release -r win-x64 --self-contained true`
on the Mac: 0 warnings, **1396 files**, `OtOpcUa.Host.exe` ProductVersion `1.0.0+fea7de0a…`, self-contained
`script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.exe`.
- Packages in `deps.json`: LocalDb + LocalDb.Replication **0.3.7**, MxGateway.Client **0.4.0**, HistorianGateway.Client **1.0.0**.
- Zip `otopcua-fea7de0a.zip`, SHA-256 `170E7392…8F8024`; the hash was checked on every box before expanding.
- `5fb4b4db..fea7de0a`: 39 commits, **no ConfigDb migration**, no new required setting.
## DEV: wonder-app-vd03 (`5fb4b4db` → `fea7de0a`)
Same layout and scripts as §(h) of the vd03 pack (single fused node, `E:\ApiInstall\OtOpcUa`, data in `E:\ApiInstall\OtOpcUa-data`).
- **Prepare (service up):** registry export; `BACKUP DATABASE … COPY_ONLY` →
`OtOpcUaConfig-pre-20261112-otopcua-fea7de0a.bak` (instance default backup dir); the ConfigDb was already at
`20261022102459_EnableSnapshotIsolation`, snapshot ON (9 migrations). The pepper was already present (26 env vars, unchanged).
w32tm source USWAP-VDC-ZMR1.
- **Swap:** stop 2 s, `OtOpcUa-data` robocopied to `OtOpcUa-backups\pre-20261112-otopcua-fea7de0a`, previous install at
`E:\ApiInstall\OtOpcUa.bak.20261112-pre-fea7de0a`, box `appsettings*`/`pki`/`logs` carried. **Downtime about 10 s.**
Rollback: `E:\ApiInstall\_stage-otopcua-fea7de0a\rollback.ps1`.
- **Verified:**
- `/healthz`, `/health/ready` (opcua-endpoint, script-runner InProcess, configdb, config-publish, akka) and `/health/active` all 200.
- **ServiceLevel 250** on the single node (was 255 on `5fb4b4db`, #884/#885).
- Raw + UNS browse identical to before (ns=2/ns=3 NodeIds).
- 23/24 variables Good, the same set as before. The one non-Good is `Program/BlockCount`, BadNoData, as before.
`parts-count` (raw and UNS) Good, `Spindle/S1/MotorSpeedPct` Good, `Program/Name` Good.
- AdminUI `/`, `/hosts`, `/fleet`, `/deployments`, `/raw` all 200 (login disabled on DEV).
- Boot log: `roles=[admin,driver]`, the #815 preflight line, no `RegisterServer` noise (#876), no new ERR. The repeated
`domain … is not listed in the server certificate` ERR comes from CLI clients dialling the FQDN against the
`CN=OtOpcUa` certificate. It was there before the upgrade too (36 lines earlier the same day).
- ScadaBridge on vd03: `/health/ready` 200. It has no OPC UA link to OtOpcUa (MxGateway only), so it is unaffected.
- Not run on DEV: a post-deploy AdminUI deploy (no configuration change) and Galaxy alarm history (vd03 has no Galaxy driver).
## QA preflight (#793)
Run on the box with the service's own SQL connection (`wwadmin` → `wonder-sql-vq03`, `OtOpcUaConfig`), the Mac
login being unable to open the DB.
- **ConfigDb checks** (`scripts/preflight/vd03-preupgrade-b8b4c18f-configdb-checks.sql`, `sqlcmd -I`): **all 15 checks
`ok`**, including #783 VirtualTagNoEffectiveTrigger, #791 VT→VT and VirtualTagDependencyCycle, #792/E2 forbidden
script APIs, and the credential/MTConnect/Sql/Heidenhain/UNS/FOCAS rules.
- **The QA ConfigDb is empty:** 0 deployments, 0 driver instances, 0 tags, 0 virtual tags, 0 scripts, 0 scripted
alarms, 0 API keys, 0 NodeAcl rows.
- Two ClusterNode rows: `10.220.158.2:4053` (vq03) and `10.220.158.3:4053` (vq04), cluster `DEV`, ServiceLevelBase 200.
- Migrations 8 (last `AddApiKeyTable`), snapshot isolation **OFF**, compat 160.
- **PKI / CA KeyUsage (#782):** each node has only its own self-signed `CN=OtOpcUa` certificate (CA:FALSE). vq03 is
`06498CB2…`, SAN `urn:wonder-app-vq03:OtOpcUa` + IP 10.220.158.2. vq04 is `CE4859A4…`, SAN `urn:wonder-app-vq04:OtOpcUa`
+ IP 10.220.158.3. Both are valid to 2027-08-11. There are no CA or issuer certificates, so nothing for the stricter
rule to reject. ScadaBridge QA has no OPC UA connection to OtOpcUa (its one data connection is `MxGateway`), so
there is no trust chain to check.
- **Service environment:** 41 entries on vq03, 40 on vq04 (vq04 has no `PeerAddress`). Compared by SHA-256 prefix only:
- `Security__ApiKeys__Pepper` is already set (44 chars) and identical on both nodes.
- **`LocalDb__Replication__ApiKey`** is already set (44 chars) and **byte-identical on both nodes**. This is what
LocalDb 0.3.7 requires; it lives in the service registry environment, not in a file.
- The JWT key is 64 bytes, with the default Issuer/Audience.
- `ServerHistorian` is enabled → `http://10.220.158.2:5221` (vq03's HistorianGateway `d8d890c`), `UseTls=false`.
- No `OpcUa:EnabledSecurityProfiles` anywhere, so #879 changes nothing here.
- **Clocks (#873):** vq03 → USWAP-VDC-ZMR3, vq04 → USSBN-VDC-ZMR1, both synced within minutes.
- **HistorianGateway: not upgraded.** Master uses only RPCs that `d8d890c` serves (ReadRaw/Aggregate/AtTime/Events,
WriteLiveValues, SendEvent, EnsureTags, Probe). It sends its own event GUIDs and does not use the new store-forward
fields or string backfill (`docs/Historian.md`). The preflight boot logged `Historian gateway probe: the gateway is reachable`
against `d8d890c` with Client 1.0.0 over h2c. QA has no historized tags. historiangw#78 stays its own deploy.
- **mxaccessgw: not upgraded.** The #877 stream-attached ack is capability-negotiated (`stream-events-attached-ack`;
`SupportsStreamAttachedAck` defaults false → the old best-effort wait). The #749 operator password is an additive
proto field that the old gateway ignores. QA OtOpcUa has no Galaxy driver at all, so `4f931e4` stays.
- **Isolated preflight boot on vq04** (live service running): the candidate ran from a copy with the node's real
service environment, using its own ports (HTTP 19000, Akka 14053 seeded to itself, OPC UA 14840, sync 19101), a
clone `OtOpcUaConfig_pf` (snapshot migration applied) and copies of `data\`/`pki\`.
- **A:** `/health/ready` 200 in 11 s, all five components Healthy.
- **B (negative control):** a 5-character pepper → exit 1, `FATAL … Security:ApiKeys:Pepper must be at least 16 characters`.
- The clone was dropped afterwards.
- **Finding:** the first A run also overrode `OpcUa:PublicHostname=127.0.0.1`. SDK 2.0.1 then refused to start the
endpoint (`The certificate with subject CN=OtOpcUa in the configuration is invalid`), because 127.0.0.1 is not in
the certificate SAN. `/health/ready` went 503 (opcua-endpoint Unhealthy) while **`/health/active` stayed 200**. So
`OpcUa:PublicHostname` must stay a name or IP in the certificate SAN, and only readiness shows a dead endpoint.
## QA window (stop both, swap both, start both; owner decision, no mixed-SDK rolling restart)
Layout: `E:\ApiInstall\OtOpcUa\{app,data,pki}`, service `OtOpcUaHost` (LocalSystem), staging `E:\psrem-stage\otopcua-fea7de0a`.
1. **Prepare (services up):**
- Candidate expanded to `app.new-fea7de0a` (1396 files, hash ok) and the box's five `appsettings*.json` carried into it
(SHA-identical; the publisher's copies were not used).
- Isolation `-PlanOnly`: 7 steps.
- Registry export `backup\OtOpcUaHost-pre-fea7de0a.reg` on each node.
- `BACKUP DATABASE OtOpcUaConfig … COPY_ONLY, CHECKSUM, COMPRESSION` → `OtOpcUaConfig_pre-fea7de0a_20261112.bak` (default
backup dir on wonder-sql-vq03), `RESTORE VERIFYONLY` ok.
- **Migration `20261022102459_EnableSnapshotIsolation` applied to the live DB** (snapshot ON, 9 migrations), while
`b8b4c18f` was still running. The old build does not use it.
2. **Stop + swap** (16:48:13–16:48:20 EST): vq03 (Secondary) stopped in 1 s, then vq04 (Primary) in 1 s.
- `data\` and `pki\` robocopied to `backup\{data,pki}-pre-fea7de0a` (stopped, so the SQLite triplets are consistent).
- `app` → `app.bak.20261112-pre-fea7de0a`, `app.new-fea7de0a` → `app`, `logs\` carried; appsettings matched.
3. **#794 isolation on the new `app\`** (the swap drops folder ACEs), both nodes, services stopped. Run with
`Install-ScriptWorkerIsolation.ps1 -InstallRoot E:\ApiInstall\OtOpcUa\app -HostServiceAccount LocalSystem -ConfigureHostService`:
- `-WhatIf`: 7 operations, nothing changed.
- Apply: 7/7 `changed`. **Second apply: 7/7 `unchanged`.**
- Rollback drill: `-Uninstall`: 6 changed (the PE cache is kept). **Second uninstall: 6/6 `unchanged`.**
- Re-apply: 6 changed + PeCacheDir unchanged. **Second re-apply: 7/7 `unchanged`.**
- State verified on both nodes:
- `sc qc OtOpcUaScriptWorker`: DISABLED, `NT SERVICE\OtOpcUaScriptWorker`. `showsid`: `S-1-5-80-1846135827-…-3023753135`.
- `C:\ProgramData\OtOpcUa\script-pe-cache`: SYSTEM and Administrators full, no Users. Its `pe\` adds the identity `(OI)(CI)(M)`.
- `app\`: identity `(OI)(CI)(DENY)(S,RD)`. `app\script-worker`: identity `(OI)(CI)(RX)` (explicit, ahead of the inherited deny).
- Firewall rule `OtOpcUa-ScriptWorker-BlockOutbound`: Outbound / Block / Any / enabled, program
`E:\ApiInstall\OtOpcUa\app\script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.exe`.
- Service env gained `Scripting__Worker__Identity` and `Scripting__Worker__PeCacheDir`, with the other entries intact.
4. **#780 P3 flip** (registry env):
- `Scripting__Runner=OutOfProcess` and `Scripting__Runner__Seams__0=Calculation`.
- After the start below, widened to `Scripting__Runner__Seams__1=VirtualTag`. **ScriptedAlarm is not enabled.**
- Env count is 46 on vq03 and 45 on vq04.
5. **Start** vq04 (16:49:34 → ready 16:50:14, 39 s: it waits for its peer), then vq03 (ready 16:50:23 in 8 s).
**Pair outage 16:48:13 → 16:50:14, about 2 min**, of which about 70 s was the isolation drill.
6. Cleanup: the clone DB and the preflight copy were removed. The staging dir and backups stay on each node.
## QA verification (#801)
| Check | Result |
|---|---|
| Version | `OtOpcUa.Host.exe` `1.0.0+fea7de0a…` on both. |
| ServiceLevel on the real pair | **vq04 250 (Primary), vq03 240 (Secondary)**, stable over repeated fresh-session reads. ApplicationUri per node unchanged. |
| Health | Both `/health/ready` 200 with opcua-endpoint, script-runner (**`Out-of-process script runner ready.`**), configdb, config-publish and akka Healthy. `/health/active` 200 on the Primary, 503 on the Secondary (by design). |
| LocalDb sync (0.3.7, ApiKey enforced) | `localdb-replication`: **connected, oplog backlog 0** on both nodes, after the window and again after the failover. **Negative control:** a sync call from the Mac with no bearer token and one with a bogus token both get `grpc-status 7 LocalDb sync authentication failed` on both nodes, and each node logs `Rejected a LocalDb sync call`. |
| Restart Secondary (to pick up the VirtualTag seam) | vq03 down → 100 → **240** within about 8 s of start. vq04 held 250 throughout. |
| **Failover** | `Stop-Service` on the Primary vq04 at 16:53:42. The graceful leave took 30 s, and vq03 became oldest at 16:54:13 (`OldestChanged … -> myself`). From the first sample: **vq03 250, `/health/active` 200**. |
| Old Primary restart | vq04 started 16:55:32: **100 → 240** at its first sample after ready, and never above 240 (no 255, #885). **No failback**: vq03 stays 250 / active 200, vq04 240 / active 503. |
| AdminUI | Login posture unchanged: `/` → 302 `/login`, `/login` 200, `/api/deployments` 302. |
| Boot logs | `roles=[admin,driver]`, the #815 preflight line, `Historian gateway probe: the gateway is reachable`. No FTL. The ERR lines are bootstrap-time Akka `Removed` (vq04 before join) and the CLI certificate-domain noise. |
| Overview (vq03 + vq04 `:5320`) | OtOpcUa **Up**, 2 instances, 2 members / 0 unreachable, 5/5 checks healthy (now including opcua-endpoint and script-runner). |
| ScadaBridge QA | `/health/ready` 200 on both nodes before and after. Its only data connection is `MxGateway`, so **it reads no OtOpcUa data on QA**. The consumer check is not applicable here. |
| AVEVA history via HG | **Not applicable on QA**: no Galaxy driver, no historized tags. The real-AVEVA gate passed separately against vd03's gateway (#801 comment, `008f0b36`). |
**Not exercisable on QA (empty configuration):**
- No Calculation, VirtualTag or scripted-alarm exists, so no worker has started on either node. The following were
therefore not observed: the restricted-token worker under the real service session, the window-station grant, the
PE-cache fill, the firewall drop log, and "Calculation tags stay Good".
- The runaway drill was **skipped**: running it needs a script in the configuration, and authoring QA configuration
was out of scope for this deploy.
- `script-runner` reports `Out-of-process script runner ready` on both nodes.
**Isolation coverage note.** The installer denies the worker identity on `InstallRoot` (`app\`) only. On QA the
siblings `E:\ApiInstall\OtOpcUa\pki` (including `own\private\*.pfx`) and `data\` (`secrets.db`, LocalDb) inherit
`BUILTIN\Users:ReadAndExecute`. Users is one of the worker token's restricting SIDs, so those folders are not covered
by the deny. File-level ACLs were not inspected. Follow-up: #890.
## Configuration set (service registry environment; no file changes, no secrets added)
| Node | Added |
|---|---|
| vq03, vq04 | `Scripting__Worker__Identity=NT SERVICE\OtOpcUaScriptWorker`, `Scripting__Worker__PeCacheDir=C:\ProgramData\OtOpcUa\script-pe-cache` (installer), `Scripting__Runner=OutOfProcess`, `Scripting__Runner__Seams__0=Calculation`, `Scripting__Runner__Seams__1=VirtualTag` |
| vq03, vq04 | Kept, already present and identical: `Security__ApiKeys__Pepper`, `LocalDb__Replication__ApiKey` |
| ConfigDb (QA) | `EnableSnapshotIsolation` applied |
| vd03 | No change |
## Rollback
- **Runner only:** set `Scripting__Runner=InProcess` (or remove the four `Scripting__Runner*` entries) and restart the
node. To remove the isolation, run `Install-ScriptWorkerIsolation.ps1 -InstallRoot E:\ApiInstall\OtOpcUa\app
-HostServiceAccount LocalSystem -ConfigureHostService -Uninstall`.
- **QA binaries:** run `E:\psrem-stage\otopcua-fea7de0a\rollback.ps1` on **both** nodes (stop both first), then start both.
The script does three things:
1. `app` → `app.failed-fea7de0a`, and `app.bak.20261112-pre-fea7de0a` → `app`;
2. mirrors `backup\data-pre-fea7de0a` back over `data\`;
3. imports `backup\OtOpcUaHost-pre-fea7de0a.reg`, which restores the pre-deploy environment, without the Scripting entries.
The isolation ACEs, firewall rule and anchor service are harmless to `b8b4c18f`. No schema undo is needed.
`OtOpcUaConfig_pre-fea7de0a_20261112.bak` is the DB fallback.
- **DEV:** `E:\ApiInstall\_stage-otopcua-fea7de0a\rollback.ps1` (→ `5fb4b4db`).
- SDK 1.5 tooling caveat after a rollback: see the rollout runbook §Rollback item 4 (#880).