From c80d187c183dff23107cf4b853a1bce2c242ce37 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Thu, 12 Nov 2026 17:03:05 -0500 Subject: [PATCH] docs(deploy): QA vq03/vq04 + DEV vd03 upgraded to master fea7de0a (SDK 2.0.1) - #801 #793 #794 #780 DEV vd03 5fb4b4db -> fea7de0a (ServiceLevel 255 -> 250, 23/24 values as before). QA pair b8b4c18f -> fea7de0a with stop-both/start-both: ConfigDb preflight all ok (empty configuration), snapshot-isolation migration, isolated preflight boot + pepper negative control, ScriptWorker isolation installed (idempotent, rollback drill), runner OutOfProcess on Calculation + VirtualTag. Real pair 250/240, LocalDb 0.3.7 sync connected with an auth negative control, failover and no-failback verified. Follow-up #890 (isolation does not cover sibling pki/data). Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi --- .../deploy/qa-upgrade-b8b4c18f-to-fea7de0a.md | 178 ++++++++++++++++++ .../deploy/vd03-upgrade-b8b4c18f-to-master.md | 3 + 2 files changed, 181 insertions(+) create mode 100644 docs/deploy/qa-upgrade-b8b4c18f-to-fea7de0a.md diff --git a/docs/deploy/qa-upgrade-b8b4c18f-to-fea7de0a.md b/docs/deploy/qa-upgrade-b8b4c18f-to-fea7de0a.md new file mode 100644 index 000000000..975142af4 --- /dev/null +++ b/docs/deploy/qa-upgrade-b8b4c18f-to-fea7de0a.md @@ -0,0 +1,178 @@ +# QA wonder-app-vq03/vq04 upgrade `b8b4c18f` → master `fea7de0a` (SDK 2.0.1), with the DEV rehearsal + +> **Status: EXECUTED 2026-11-12.** DEV vd03 runs `fea7de0a` (from `5fb4b4db`). The QA pair vq03 + vq04 runs +> `fea7de0a` (from `b8b4c18f`) with the out-of-process script runner on (Calculation + VirtualTag seams) and the +> ScriptWorker isolation installed. Tracking: #801 (SDK 2 live gate), #793 (ConfigDb preflight), #794 (worker +> isolation install), #780 (P3 runner flip). Template: [`vd03-upgrade-b8b4c18f-to-master.md`](vd03-upgrade-b8b4c18f-to-master.md). + +Channel: WinRM + SMB from the Mac (the `winrm` skill), used only for the on-box steps of this deploy, which the owner +requested on 2026-11-12. Health, OPC UA, ScadaBridge and Overview checks ran from the Mac over HTTP / opc.tcp. + +## Build + +- Source: `origin/master` **`fea7de0a`** (`fea7de0aefd905522d187dad495c3c8163a5ec03`), built in an isolated worktree. +- `dotnet publish src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj -c Release -r win-x64 --self-contained true` + on the Mac: 0 warnings, **1396 files**, `OtOpcUa.Host.exe` ProductVersion `1.0.0+fea7de0a…`, self-contained + `script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.exe`. +- Packages in `deps.json`: LocalDb + LocalDb.Replication **0.3.7**, MxGateway.Client **0.4.0**, HistorianGateway.Client **1.0.0**. +- Zip `otopcua-fea7de0a.zip`, SHA-256 `170E7392…8F8024`; the hash was checked on every box before expanding. +- `5fb4b4db..fea7de0a`: 39 commits, **no ConfigDb migration**, no new required setting. + +## DEV: wonder-app-vd03 (`5fb4b4db` → `fea7de0a`) + +Same layout and scripts as §(h) of the vd03 pack (single fused node, `E:\ApiInstall\OtOpcUa`, data in `E:\ApiInstall\OtOpcUa-data`). + +- **Prepare (service up):** registry export; `BACKUP DATABASE … COPY_ONLY` → + `OtOpcUaConfig-pre-20261112-otopcua-fea7de0a.bak` (instance default backup dir); the ConfigDb was already at + `20261022102459_EnableSnapshotIsolation`, snapshot ON (9 migrations). The pepper was already present (26 env vars, unchanged). + w32tm source USWAP-VDC-ZMR1. +- **Swap:** stop 2 s, `OtOpcUa-data` robocopied to `OtOpcUa-backups\pre-20261112-otopcua-fea7de0a`, previous install at + `E:\ApiInstall\OtOpcUa.bak.20261112-pre-fea7de0a`, box `appsettings*`/`pki`/`logs` carried. **Downtime about 10 s.** + Rollback: `E:\ApiInstall\_stage-otopcua-fea7de0a\rollback.ps1`. +- **Verified:** + - `/healthz`, `/health/ready` (opcua-endpoint, script-runner InProcess, configdb, config-publish, akka) and `/health/active` all 200. + - **ServiceLevel 250** on the single node (was 255 on `5fb4b4db`, #884/#885). + - Raw + UNS browse identical to before (ns=2/ns=3 NodeIds). + - 23/24 variables Good, the same set as before. The one non-Good is `Program/BlockCount`, BadNoData, as before. + `parts-count` (raw and UNS) Good, `Spindle/S1/MotorSpeedPct` Good, `Program/Name` Good. + - AdminUI `/`, `/hosts`, `/fleet`, `/deployments`, `/raw` all 200 (login disabled on DEV). + - Boot log: `roles=[admin,driver]`, the #815 preflight line, no `RegisterServer` noise (#876), no new ERR. The repeated + `domain … is not listed in the server certificate` ERR comes from CLI clients dialling the FQDN against the + `CN=OtOpcUa` certificate. It was there before the upgrade too (36 lines earlier the same day). + - ScadaBridge on vd03: `/health/ready` 200. It has no OPC UA link to OtOpcUa (MxGateway only), so it is unaffected. +- Not run on DEV: a post-deploy AdminUI deploy (no configuration change) and Galaxy alarm history (vd03 has no Galaxy driver). + +## QA preflight (#793) + +Run on the box with the service's own SQL connection (`wwadmin` → `wonder-sql-vq03`, `OtOpcUaConfig`), the Mac +login being unable to open the DB. + +- **ConfigDb checks** (`scripts/preflight/vd03-preupgrade-b8b4c18f-configdb-checks.sql`, `sqlcmd -I`): **all 15 checks + `ok`**, including #783 VirtualTagNoEffectiveTrigger, #791 VT→VT and VirtualTagDependencyCycle, #792/E2 forbidden + script APIs, and the credential/MTConnect/Sql/Heidenhain/UNS/FOCAS rules. + - **The QA ConfigDb is empty:** 0 deployments, 0 driver instances, 0 tags, 0 virtual tags, 0 scripts, 0 scripted + alarms, 0 API keys, 0 NodeAcl rows. + - Two ClusterNode rows: `10.220.158.2:4053` (vq03) and `10.220.158.3:4053` (vq04), cluster `DEV`, ServiceLevelBase 200. + - Migrations 8 (last `AddApiKeyTable`), snapshot isolation **OFF**, compat 160. +- **PKI / CA KeyUsage (#782):** each node has only its own self-signed `CN=OtOpcUa` certificate (CA:FALSE). vq03 is + `06498CB2…`, SAN `urn:wonder-app-vq03:OtOpcUa` + IP 10.220.158.2. vq04 is `CE4859A4…`, SAN `urn:wonder-app-vq04:OtOpcUa` + + IP 10.220.158.3. Both are valid to 2027-08-11. There are no CA or issuer certificates, so nothing for the stricter + rule to reject. ScadaBridge QA has no OPC UA connection to OtOpcUa (its one data connection is `MxGateway`), so + there is no trust chain to check. +- **Service environment:** 41 entries on vq03, 40 on vq04 (vq04 has no `PeerAddress`). Compared by SHA-256 prefix only: + - `Security__ApiKeys__Pepper` is already set (44 chars) and identical on both nodes. + - **`LocalDb__Replication__ApiKey`** is already set (44 chars) and **byte-identical on both nodes**. This is what + LocalDb 0.3.7 requires; it lives in the service registry environment, not in a file. + - The JWT key is 64 bytes, with the default Issuer/Audience. + - `ServerHistorian` is enabled → `http://10.220.158.2:5221` (vq03's HistorianGateway `d8d890c`), `UseTls=false`. + - No `OpcUa:EnabledSecurityProfiles` anywhere, so #879 changes nothing here. +- **Clocks (#873):** vq03 → USWAP-VDC-ZMR3, vq04 → USSBN-VDC-ZMR1, both synced within minutes. +- **HistorianGateway: not upgraded.** Master uses only RPCs that `d8d890c` serves (ReadRaw/Aggregate/AtTime/Events, + WriteLiveValues, SendEvent, EnsureTags, Probe). It sends its own event GUIDs and does not use the new store-forward + fields or string backfill (`docs/Historian.md`). The preflight boot logged `Historian gateway probe: the gateway is reachable` + against `d8d890c` with Client 1.0.0 over h2c. QA has no historized tags. historiangw#78 stays its own deploy. +- **mxaccessgw: not upgraded.** The #877 stream-attached ack is capability-negotiated (`stream-events-attached-ack`; + `SupportsStreamAttachedAck` defaults false → the old best-effort wait). The #749 operator password is an additive + proto field that the old gateway ignores. QA OtOpcUa has no Galaxy driver at all, so `4f931e4` stays. +- **Isolated preflight boot on vq04** (live service running): the candidate ran from a copy with the node's real + service environment, using its own ports (HTTP 19000, Akka 14053 seeded to itself, OPC UA 14840, sync 19101), a + clone `OtOpcUaConfig_pf` (snapshot migration applied) and copies of `data\`/`pki\`. + - **A:** `/health/ready` 200 in 11 s, all five components Healthy. + - **B (negative control):** a 5-character pepper → exit 1, `FATAL … Security:ApiKeys:Pepper must be at least 16 characters`. + - The clone was dropped afterwards. + - **Finding:** the first A run also overrode `OpcUa:PublicHostname=127.0.0.1`. SDK 2.0.1 then refused to start the + endpoint (`The certificate with subject CN=OtOpcUa in the configuration is invalid`), because 127.0.0.1 is not in + the certificate SAN. `/health/ready` went 503 (opcua-endpoint Unhealthy) while **`/health/active` stayed 200**. So + `OpcUa:PublicHostname` must stay a name or IP in the certificate SAN, and only readiness shows a dead endpoint. + +## QA window (stop both, swap both, start both; owner decision, no mixed-SDK rolling restart) + +Layout: `E:\ApiInstall\OtOpcUa\{app,data,pki}`, service `OtOpcUaHost` (LocalSystem), staging `E:\psrem-stage\otopcua-fea7de0a`. + +1. **Prepare (services up):** + - Candidate expanded to `app.new-fea7de0a` (1396 files, hash ok) and the box's five `appsettings*.json` carried into it + (SHA-identical; the publisher's copies were not used). + - Isolation `-PlanOnly`: 7 steps. + - Registry export `backup\OtOpcUaHost-pre-fea7de0a.reg` on each node. + - `BACKUP DATABASE OtOpcUaConfig … COPY_ONLY, CHECKSUM, COMPRESSION` → `OtOpcUaConfig_pre-fea7de0a_20261112.bak` (default + backup dir on wonder-sql-vq03), `RESTORE VERIFYONLY` ok. + - **Migration `20261022102459_EnableSnapshotIsolation` applied to the live DB** (snapshot ON, 9 migrations), while + `b8b4c18f` was still running. The old build does not use it. +2. **Stop + swap** (16:48:13–16:48:20 EST): vq03 (Secondary) stopped in 1 s, then vq04 (Primary) in 1 s. + - `data\` and `pki\` robocopied to `backup\{data,pki}-pre-fea7de0a` (stopped, so the SQLite triplets are consistent). + - `app` → `app.bak.20261112-pre-fea7de0a`, `app.new-fea7de0a` → `app`, `logs\` carried; appsettings matched. +3. **#794 isolation on the new `app\`** (the swap drops folder ACEs), both nodes, services stopped. Run with + `Install-ScriptWorkerIsolation.ps1 -InstallRoot E:\ApiInstall\OtOpcUa\app -HostServiceAccount LocalSystem -ConfigureHostService`: + - `-WhatIf`: 7 operations, nothing changed. + - Apply: 7/7 `changed`. **Second apply: 7/7 `unchanged`.** + - Rollback drill: `-Uninstall`: 6 changed (the PE cache is kept). **Second uninstall: 6/6 `unchanged`.** + - Re-apply: 6 changed + PeCacheDir unchanged. **Second re-apply: 7/7 `unchanged`.** + - State verified on both nodes: + - `sc qc OtOpcUaScriptWorker`: DISABLED, `NT SERVICE\OtOpcUaScriptWorker`. `showsid`: `S-1-5-80-1846135827-…-3023753135`. + - `C:\ProgramData\OtOpcUa\script-pe-cache`: SYSTEM and Administrators full, no Users. Its `pe\` adds the identity `(OI)(CI)(M)`. + - `app\`: identity `(OI)(CI)(DENY)(S,RD)`. `app\script-worker`: identity `(OI)(CI)(RX)` (explicit, ahead of the inherited deny). + - Firewall rule `OtOpcUa-ScriptWorker-BlockOutbound`: Outbound / Block / Any / enabled, program + `E:\ApiInstall\OtOpcUa\app\script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.exe`. + - Service env gained `Scripting__Worker__Identity` and `Scripting__Worker__PeCacheDir`, with the other entries intact. +4. **#780 P3 flip** (registry env): + - `Scripting__Runner=OutOfProcess` and `Scripting__Runner__Seams__0=Calculation`. + - After the start below, widened to `Scripting__Runner__Seams__1=VirtualTag`. **ScriptedAlarm is not enabled.** + - Env count is 46 on vq03 and 45 on vq04. +5. **Start** vq04 (16:49:34 → ready 16:50:14, 39 s: it waits for its peer), then vq03 (ready 16:50:23 in 8 s). + **Pair outage 16:48:13 → 16:50:14, about 2 min**, of which about 70 s was the isolation drill. +6. Cleanup: the clone DB and the preflight copy were removed. The staging dir and backups stay on each node. + +## QA verification (#801) + +| Check | Result | +|---|---| +| Version | `OtOpcUa.Host.exe` `1.0.0+fea7de0a…` on both. | +| ServiceLevel on the real pair | **vq04 250 (Primary), vq03 240 (Secondary)**, stable over repeated fresh-session reads. ApplicationUri per node unchanged. | +| Health | Both `/health/ready` 200 with opcua-endpoint, script-runner (**`Out-of-process script runner ready.`**), configdb, config-publish and akka Healthy. `/health/active` 200 on the Primary, 503 on the Secondary (by design). | +| LocalDb sync (0.3.7, ApiKey enforced) | `localdb-replication`: **connected, oplog backlog 0** on both nodes, after the window and again after the failover. **Negative control:** a sync call from the Mac with no bearer token and one with a bogus token both get `grpc-status 7 LocalDb sync authentication failed` on both nodes, and each node logs `Rejected a LocalDb sync call`. | +| Restart Secondary (to pick up the VirtualTag seam) | vq03 down → 100 → **240** within about 8 s of start. vq04 held 250 throughout. | +| **Failover** | `Stop-Service` on the Primary vq04 at 16:53:42. The graceful leave took 30 s, and vq03 became oldest at 16:54:13 (`OldestChanged … -> myself`). From the first sample: **vq03 250, `/health/active` 200**. | +| Old Primary restart | vq04 started 16:55:32: **100 → 240** at its first sample after ready, and never above 240 (no 255, #885). **No failback**: vq03 stays 250 / active 200, vq04 240 / active 503. | +| AdminUI | Login posture unchanged: `/` → 302 `/login`, `/login` 200, `/api/deployments` 302. | +| Boot logs | `roles=[admin,driver]`, the #815 preflight line, `Historian gateway probe: the gateway is reachable`. No FTL. The ERR lines are bootstrap-time Akka `Removed` (vq04 before join) and the CLI certificate-domain noise. | +| Overview (vq03 + vq04 `:5320`) | OtOpcUa **Up**, 2 instances, 2 members / 0 unreachable, 5/5 checks healthy (now including opcua-endpoint and script-runner). | +| ScadaBridge QA | `/health/ready` 200 on both nodes before and after. Its only data connection is `MxGateway`, so **it reads no OtOpcUa data on QA**. The consumer check is not applicable here. | +| AVEVA history via HG | **Not applicable on QA**: no Galaxy driver, no historized tags. The real-AVEVA gate passed separately against vd03's gateway (#801 comment, `008f0b36`). | + +**Not exercisable on QA (empty configuration):** +- No Calculation, VirtualTag or scripted-alarm exists, so no worker has started on either node. The following were + therefore not observed: the restricted-token worker under the real service session, the window-station grant, the + PE-cache fill, the firewall drop log, and "Calculation tags stay Good". +- The runaway drill was **skipped**: running it needs a script in the configuration, and authoring QA configuration + was out of scope for this deploy. +- `script-runner` reports `Out-of-process script runner ready` on both nodes. + +**Isolation coverage note.** The installer denies the worker identity on `InstallRoot` (`app\`) only. On QA the +siblings `E:\ApiInstall\OtOpcUa\pki` (including `own\private\*.pfx`) and `data\` (`secrets.db`, LocalDb) inherit +`BUILTIN\Users:ReadAndExecute`. Users is one of the worker token's restricting SIDs, so those folders are not covered +by the deny. File-level ACLs were not inspected. Follow-up: #890. + +## Configuration set (service registry environment; no file changes, no secrets added) + +| Node | Added | +|---|---| +| vq03, vq04 | `Scripting__Worker__Identity=NT SERVICE\OtOpcUaScriptWorker`, `Scripting__Worker__PeCacheDir=C:\ProgramData\OtOpcUa\script-pe-cache` (installer), `Scripting__Runner=OutOfProcess`, `Scripting__Runner__Seams__0=Calculation`, `Scripting__Runner__Seams__1=VirtualTag` | +| vq03, vq04 | Kept, already present and identical: `Security__ApiKeys__Pepper`, `LocalDb__Replication__ApiKey` | +| ConfigDb (QA) | `EnableSnapshotIsolation` applied | +| vd03 | No change | + +## Rollback + +- **Runner only:** set `Scripting__Runner=InProcess` (or remove the four `Scripting__Runner*` entries) and restart the + node. To remove the isolation, run `Install-ScriptWorkerIsolation.ps1 -InstallRoot E:\ApiInstall\OtOpcUa\app + -HostServiceAccount LocalSystem -ConfigureHostService -Uninstall`. +- **QA binaries:** run `E:\psrem-stage\otopcua-fea7de0a\rollback.ps1` on **both** nodes (stop both first), then start both. + The script does three things: + 1. `app` → `app.failed-fea7de0a`, and `app.bak.20261112-pre-fea7de0a` → `app`; + 2. mirrors `backup\data-pre-fea7de0a` back over `data\`; + 3. imports `backup\OtOpcUaHost-pre-fea7de0a.reg`, which restores the pre-deploy environment, without the Scripting entries. + + The isolation ACEs, firewall rule and anchor service are harmless to `b8b4c18f`. No schema undo is needed. + `OtOpcUaConfig_pre-fea7de0a_20261112.bak` is the DB fallback. +- **DEV:** `E:\ApiInstall\_stage-otopcua-fea7de0a\rollback.ps1` (→ `5fb4b4db`). +- SDK 1.5 tooling caveat after a rollback: see the rollout runbook §Rollback item 4 (#880). diff --git a/docs/deploy/vd03-upgrade-b8b4c18f-to-master.md b/docs/deploy/vd03-upgrade-b8b4c18f-to-master.md index 6523e4f45..15aa8fee7 100644 --- a/docs/deploy/vd03-upgrade-b8b4c18f-to-master.md +++ b/docs/deploy/vd03-upgrade-b8b4c18f-to-master.md @@ -4,6 +4,9 @@ > `363f0d81`; nothing between that and `5fb4b4db` changed the schema, configuration or preflight. The execution > record is §(h); the owner-action table and "Not determined" list below carry the outcomes. **All six owner actions > are closed and nothing is left undetermined (close-out 2026-11-12, §h last bullet).** +> +> **Superseded on vd03 the same day:** vd03 was upgraded again, `5fb4b4db` → `fea7de0a` (ServiceLevel 255 → 250), as the +> DEV rehearsal for the QA pair. Record: [`qa-upgrade-b8b4c18f-to-fea7de0a.md`](qa-upgrade-b8b4c18f-to-fea7de0a.md). Upgrade-readiness notes for the DEV box wonder-app-vd03. The box runs the `OtOpcUaHost` service from `E:\ApiInstall\OtOpcUa`: one fused admin+driver node with one FOCAS driver (Makino Z-34184). The ConfigDb