docs(deploy): QA vq03/vq04 + DEV vd03 upgraded to master fea7de0a (SDK 2.0.1) - #801 #793 #794 #780
v2-ci / unit-tests (push) Has been cancelled
v2-ci / unit-tests (push) Has been cancelled
DEV vd035fb4b4db->fea7de0a(ServiceLevel 255 -> 250, 23/24 values as before). QA pairb8b4c18f->fea7de0awith stop-both/start-both: ConfigDb preflight all ok (empty configuration), snapshot-isolation migration, isolated preflight boot + pepper negative control, ScriptWorker isolation installed (idempotent, rollback drill), runner OutOfProcess on Calculation + VirtualTag. Real pair 250/240, LocalDb 0.3.7 sync connected with an auth negative control, failover and no-failback verified. Follow-up #890 (isolation does not cover sibling pki/data). Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,178 @@
|
||||
# QA wonder-app-vq03/vq04 upgrade `b8b4c18f` → master `fea7de0a` (SDK 2.0.1), with the DEV rehearsal
|
||||
|
||||
> **Status: EXECUTED 2026-11-12.** DEV vd03 runs `fea7de0a` (from `5fb4b4db`). The QA pair vq03 + vq04 runs
|
||||
> `fea7de0a` (from `b8b4c18f`) with the out-of-process script runner on (Calculation + VirtualTag seams) and the
|
||||
> ScriptWorker isolation installed. Tracking: #801 (SDK 2 live gate), #793 (ConfigDb preflight), #794 (worker
|
||||
> isolation install), #780 (P3 runner flip). Template: [`vd03-upgrade-b8b4c18f-to-master.md`](vd03-upgrade-b8b4c18f-to-master.md).
|
||||
|
||||
Channel: WinRM + SMB from the Mac (the `winrm` skill), used only for the on-box steps of this deploy, which the owner
|
||||
requested on 2026-11-12. Health, OPC UA, ScadaBridge and Overview checks ran from the Mac over HTTP / opc.tcp.
|
||||
|
||||
## Build
|
||||
|
||||
- Source: `origin/master` **`fea7de0a`** (`fea7de0aefd905522d187dad495c3c8163a5ec03`), built in an isolated worktree.
|
||||
- `dotnet publish src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj -c Release -r win-x64 --self-contained true`
|
||||
on the Mac: 0 warnings, **1396 files**, `OtOpcUa.Host.exe` ProductVersion `1.0.0+fea7de0a…`, self-contained
|
||||
`script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.exe`.
|
||||
- Packages in `deps.json`: LocalDb + LocalDb.Replication **0.3.7**, MxGateway.Client **0.4.0**, HistorianGateway.Client **1.0.0**.
|
||||
- Zip `otopcua-fea7de0a.zip`, SHA-256 `170E7392…8F8024`; the hash was checked on every box before expanding.
|
||||
- `5fb4b4db..fea7de0a`: 39 commits, **no ConfigDb migration**, no new required setting.
|
||||
|
||||
## DEV: wonder-app-vd03 (`5fb4b4db` → `fea7de0a`)
|
||||
|
||||
Same layout and scripts as §(h) of the vd03 pack (single fused node, `E:\ApiInstall\OtOpcUa`, data in `E:\ApiInstall\OtOpcUa-data`).
|
||||
|
||||
- **Prepare (service up):** registry export; `BACKUP DATABASE … COPY_ONLY` →
|
||||
`OtOpcUaConfig-pre-20261112-otopcua-fea7de0a.bak` (instance default backup dir); the ConfigDb was already at
|
||||
`20261022102459_EnableSnapshotIsolation`, snapshot ON (9 migrations). The pepper was already present (26 env vars, unchanged).
|
||||
w32tm source USWAP-VDC-ZMR1.
|
||||
- **Swap:** stop 2 s, `OtOpcUa-data` robocopied to `OtOpcUa-backups\pre-20261112-otopcua-fea7de0a`, previous install at
|
||||
`E:\ApiInstall\OtOpcUa.bak.20261112-pre-fea7de0a`, box `appsettings*`/`pki`/`logs` carried. **Downtime about 10 s.**
|
||||
Rollback: `E:\ApiInstall\_stage-otopcua-fea7de0a\rollback.ps1`.
|
||||
- **Verified:**
|
||||
- `/healthz`, `/health/ready` (opcua-endpoint, script-runner InProcess, configdb, config-publish, akka) and `/health/active` all 200.
|
||||
- **ServiceLevel 250** on the single node (was 255 on `5fb4b4db`, #884/#885).
|
||||
- Raw + UNS browse identical to before (ns=2/ns=3 NodeIds).
|
||||
- 23/24 variables Good, the same set as before. The one non-Good is `Program/BlockCount`, BadNoData, as before.
|
||||
`parts-count` (raw and UNS) Good, `Spindle/S1/MotorSpeedPct` Good, `Program/Name` Good.
|
||||
- AdminUI `/`, `/hosts`, `/fleet`, `/deployments`, `/raw` all 200 (login disabled on DEV).
|
||||
- Boot log: `roles=[admin,driver]`, the #815 preflight line, no `RegisterServer` noise (#876), no new ERR. The repeated
|
||||
`domain … is not listed in the server certificate` ERR comes from CLI clients dialling the FQDN against the
|
||||
`CN=OtOpcUa` certificate. It was there before the upgrade too (36 lines earlier the same day).
|
||||
- ScadaBridge on vd03: `/health/ready` 200. It has no OPC UA link to OtOpcUa (MxGateway only), so it is unaffected.
|
||||
- Not run on DEV: a post-deploy AdminUI deploy (no configuration change) and Galaxy alarm history (vd03 has no Galaxy driver).
|
||||
|
||||
## QA preflight (#793)
|
||||
|
||||
Run on the box with the service's own SQL connection (`wwadmin` → `wonder-sql-vq03`, `OtOpcUaConfig`), the Mac
|
||||
login being unable to open the DB.
|
||||
|
||||
- **ConfigDb checks** (`scripts/preflight/vd03-preupgrade-b8b4c18f-configdb-checks.sql`, `sqlcmd -I`): **all 15 checks
|
||||
`ok`**, including #783 VirtualTagNoEffectiveTrigger, #791 VT→VT and VirtualTagDependencyCycle, #792/E2 forbidden
|
||||
script APIs, and the credential/MTConnect/Sql/Heidenhain/UNS/FOCAS rules.
|
||||
- **The QA ConfigDb is empty:** 0 deployments, 0 driver instances, 0 tags, 0 virtual tags, 0 scripts, 0 scripted
|
||||
alarms, 0 API keys, 0 NodeAcl rows.
|
||||
- Two ClusterNode rows: `10.220.158.2:4053` (vq03) and `10.220.158.3:4053` (vq04), cluster `DEV`, ServiceLevelBase 200.
|
||||
- Migrations 8 (last `AddApiKeyTable`), snapshot isolation **OFF**, compat 160.
|
||||
- **PKI / CA KeyUsage (#782):** each node has only its own self-signed `CN=OtOpcUa` certificate (CA:FALSE). vq03 is
|
||||
`06498CB2…`, SAN `urn:wonder-app-vq03:OtOpcUa` + IP 10.220.158.2. vq04 is `CE4859A4…`, SAN `urn:wonder-app-vq04:OtOpcUa`
|
||||
+ IP 10.220.158.3. Both are valid to 2027-08-11. There are no CA or issuer certificates, so nothing for the stricter
|
||||
rule to reject. ScadaBridge QA has no OPC UA connection to OtOpcUa (its one data connection is `MxGateway`), so
|
||||
there is no trust chain to check.
|
||||
- **Service environment:** 41 entries on vq03, 40 on vq04 (vq04 has no `PeerAddress`). Compared by SHA-256 prefix only:
|
||||
- `Security__ApiKeys__Pepper` is already set (44 chars) and identical on both nodes.
|
||||
- **`LocalDb__Replication__ApiKey`** is already set (44 chars) and **byte-identical on both nodes**. This is what
|
||||
LocalDb 0.3.7 requires; it lives in the service registry environment, not in a file.
|
||||
- The JWT key is 64 bytes, with the default Issuer/Audience.
|
||||
- `ServerHistorian` is enabled → `http://10.220.158.2:5221` (vq03's HistorianGateway `d8d890c`), `UseTls=false`.
|
||||
- No `OpcUa:EnabledSecurityProfiles` anywhere, so #879 changes nothing here.
|
||||
- **Clocks (#873):** vq03 → USWAP-VDC-ZMR3, vq04 → USSBN-VDC-ZMR1, both synced within minutes.
|
||||
- **HistorianGateway: not upgraded.** Master uses only RPCs that `d8d890c` serves (ReadRaw/Aggregate/AtTime/Events,
|
||||
WriteLiveValues, SendEvent, EnsureTags, Probe). It sends its own event GUIDs and does not use the new store-forward
|
||||
fields or string backfill (`docs/Historian.md`). The preflight boot logged `Historian gateway probe: the gateway is reachable`
|
||||
against `d8d890c` with Client 1.0.0 over h2c. QA has no historized tags. historiangw#78 stays its own deploy.
|
||||
- **mxaccessgw: not upgraded.** The #877 stream-attached ack is capability-negotiated (`stream-events-attached-ack`;
|
||||
`SupportsStreamAttachedAck` defaults false → the old best-effort wait). The #749 operator password is an additive
|
||||
proto field that the old gateway ignores. QA OtOpcUa has no Galaxy driver at all, so `4f931e4` stays.
|
||||
- **Isolated preflight boot on vq04** (live service running): the candidate ran from a copy with the node's real
|
||||
service environment, using its own ports (HTTP 19000, Akka 14053 seeded to itself, OPC UA 14840, sync 19101), a
|
||||
clone `OtOpcUaConfig_pf` (snapshot migration applied) and copies of `data\`/`pki\`.
|
||||
- **A:** `/health/ready` 200 in 11 s, all five components Healthy.
|
||||
- **B (negative control):** a 5-character pepper → exit 1, `FATAL … Security:ApiKeys:Pepper must be at least 16 characters`.
|
||||
- The clone was dropped afterwards.
|
||||
- **Finding:** the first A run also overrode `OpcUa:PublicHostname=127.0.0.1`. SDK 2.0.1 then refused to start the
|
||||
endpoint (`The certificate with subject CN=OtOpcUa in the configuration is invalid`), because 127.0.0.1 is not in
|
||||
the certificate SAN. `/health/ready` went 503 (opcua-endpoint Unhealthy) while **`/health/active` stayed 200**. So
|
||||
`OpcUa:PublicHostname` must stay a name or IP in the certificate SAN, and only readiness shows a dead endpoint.
|
||||
|
||||
## QA window (stop both, swap both, start both; owner decision, no mixed-SDK rolling restart)
|
||||
|
||||
Layout: `E:\ApiInstall\OtOpcUa\{app,data,pki}`, service `OtOpcUaHost` (LocalSystem), staging `E:\psrem-stage\otopcua-fea7de0a`.
|
||||
|
||||
1. **Prepare (services up):**
|
||||
- Candidate expanded to `app.new-fea7de0a` (1396 files, hash ok) and the box's five `appsettings*.json` carried into it
|
||||
(SHA-identical; the publisher's copies were not used).
|
||||
- Isolation `-PlanOnly`: 7 steps.
|
||||
- Registry export `backup\OtOpcUaHost-pre-fea7de0a.reg` on each node.
|
||||
- `BACKUP DATABASE OtOpcUaConfig … COPY_ONLY, CHECKSUM, COMPRESSION` → `OtOpcUaConfig_pre-fea7de0a_20261112.bak` (default
|
||||
backup dir on wonder-sql-vq03), `RESTORE VERIFYONLY` ok.
|
||||
- **Migration `20261022102459_EnableSnapshotIsolation` applied to the live DB** (snapshot ON, 9 migrations), while
|
||||
`b8b4c18f` was still running. The old build does not use it.
|
||||
2. **Stop + swap** (16:48:13–16:48:20 EST): vq03 (Secondary) stopped in 1 s, then vq04 (Primary) in 1 s.
|
||||
- `data\` and `pki\` robocopied to `backup\{data,pki}-pre-fea7de0a` (stopped, so the SQLite triplets are consistent).
|
||||
- `app` → `app.bak.20261112-pre-fea7de0a`, `app.new-fea7de0a` → `app`, `logs\` carried; appsettings matched.
|
||||
3. **#794 isolation on the new `app\`** (the swap drops folder ACEs), both nodes, services stopped. Run with
|
||||
`Install-ScriptWorkerIsolation.ps1 -InstallRoot E:\ApiInstall\OtOpcUa\app -HostServiceAccount LocalSystem -ConfigureHostService`:
|
||||
- `-WhatIf`: 7 operations, nothing changed.
|
||||
- Apply: 7/7 `changed`. **Second apply: 7/7 `unchanged`.**
|
||||
- Rollback drill: `-Uninstall`: 6 changed (the PE cache is kept). **Second uninstall: 6/6 `unchanged`.**
|
||||
- Re-apply: 6 changed + PeCacheDir unchanged. **Second re-apply: 7/7 `unchanged`.**
|
||||
- State verified on both nodes:
|
||||
- `sc qc OtOpcUaScriptWorker`: DISABLED, `NT SERVICE\OtOpcUaScriptWorker`. `showsid`: `S-1-5-80-1846135827-…-3023753135`.
|
||||
- `C:\ProgramData\OtOpcUa\script-pe-cache`: SYSTEM and Administrators full, no Users. Its `pe\` adds the identity `(OI)(CI)(M)`.
|
||||
- `app\`: identity `(OI)(CI)(DENY)(S,RD)`. `app\script-worker`: identity `(OI)(CI)(RX)` (explicit, ahead of the inherited deny).
|
||||
- Firewall rule `OtOpcUa-ScriptWorker-BlockOutbound`: Outbound / Block / Any / enabled, program
|
||||
`E:\ApiInstall\OtOpcUa\app\script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.exe`.
|
||||
- Service env gained `Scripting__Worker__Identity` and `Scripting__Worker__PeCacheDir`, with the other entries intact.
|
||||
4. **#780 P3 flip** (registry env):
|
||||
- `Scripting__Runner=OutOfProcess` and `Scripting__Runner__Seams__0=Calculation`.
|
||||
- After the start below, widened to `Scripting__Runner__Seams__1=VirtualTag`. **ScriptedAlarm is not enabled.**
|
||||
- Env count is 46 on vq03 and 45 on vq04.
|
||||
5. **Start** vq04 (16:49:34 → ready 16:50:14, 39 s: it waits for its peer), then vq03 (ready 16:50:23 in 8 s).
|
||||
**Pair outage 16:48:13 → 16:50:14, about 2 min**, of which about 70 s was the isolation drill.
|
||||
6. Cleanup: the clone DB and the preflight copy were removed. The staging dir and backups stay on each node.
|
||||
|
||||
## QA verification (#801)
|
||||
|
||||
| Check | Result |
|
||||
|---|---|
|
||||
| Version | `OtOpcUa.Host.exe` `1.0.0+fea7de0a…` on both. |
|
||||
| ServiceLevel on the real pair | **vq04 250 (Primary), vq03 240 (Secondary)**, stable over repeated fresh-session reads. ApplicationUri per node unchanged. |
|
||||
| Health | Both `/health/ready` 200 with opcua-endpoint, script-runner (**`Out-of-process script runner ready.`**), configdb, config-publish and akka Healthy. `/health/active` 200 on the Primary, 503 on the Secondary (by design). |
|
||||
| LocalDb sync (0.3.7, ApiKey enforced) | `localdb-replication`: **connected, oplog backlog 0** on both nodes, after the window and again after the failover. **Negative control:** a sync call from the Mac with no bearer token and one with a bogus token both get `grpc-status 7 LocalDb sync authentication failed` on both nodes, and each node logs `Rejected a LocalDb sync call`. |
|
||||
| Restart Secondary (to pick up the VirtualTag seam) | vq03 down → 100 → **240** within about 8 s of start. vq04 held 250 throughout. |
|
||||
| **Failover** | `Stop-Service` on the Primary vq04 at 16:53:42. The graceful leave took 30 s, and vq03 became oldest at 16:54:13 (`OldestChanged … -> myself`). From the first sample: **vq03 250, `/health/active` 200**. |
|
||||
| Old Primary restart | vq04 started 16:55:32: **100 → 240** at its first sample after ready, and never above 240 (no 255, #885). **No failback**: vq03 stays 250 / active 200, vq04 240 / active 503. |
|
||||
| AdminUI | Login posture unchanged: `/` → 302 `/login`, `/login` 200, `/api/deployments` 302. |
|
||||
| Boot logs | `roles=[admin,driver]`, the #815 preflight line, `Historian gateway probe: the gateway is reachable`. No FTL. The ERR lines are bootstrap-time Akka `Removed` (vq04 before join) and the CLI certificate-domain noise. |
|
||||
| Overview (vq03 + vq04 `:5320`) | OtOpcUa **Up**, 2 instances, 2 members / 0 unreachable, 5/5 checks healthy (now including opcua-endpoint and script-runner). |
|
||||
| ScadaBridge QA | `/health/ready` 200 on both nodes before and after. Its only data connection is `MxGateway`, so **it reads no OtOpcUa data on QA**. The consumer check is not applicable here. |
|
||||
| AVEVA history via HG | **Not applicable on QA**: no Galaxy driver, no historized tags. The real-AVEVA gate passed separately against vd03's gateway (#801 comment, `008f0b36`). |
|
||||
|
||||
**Not exercisable on QA (empty configuration):**
|
||||
- No Calculation, VirtualTag or scripted-alarm exists, so no worker has started on either node. The following were
|
||||
therefore not observed: the restricted-token worker under the real service session, the window-station grant, the
|
||||
PE-cache fill, the firewall drop log, and "Calculation tags stay Good".
|
||||
- The runaway drill was **skipped**: running it needs a script in the configuration, and authoring QA configuration
|
||||
was out of scope for this deploy.
|
||||
- `script-runner` reports `Out-of-process script runner ready` on both nodes.
|
||||
|
||||
**Isolation coverage note.** The installer denies the worker identity on `InstallRoot` (`app\`) only. On QA the
|
||||
siblings `E:\ApiInstall\OtOpcUa\pki` (including `own\private\*.pfx`) and `data\` (`secrets.db`, LocalDb) inherit
|
||||
`BUILTIN\Users:ReadAndExecute`. Users is one of the worker token's restricting SIDs, so those folders are not covered
|
||||
by the deny. File-level ACLs were not inspected. Follow-up: #890.
|
||||
|
||||
## Configuration set (service registry environment; no file changes, no secrets added)
|
||||
|
||||
| Node | Added |
|
||||
|---|---|
|
||||
| vq03, vq04 | `Scripting__Worker__Identity=NT SERVICE\OtOpcUaScriptWorker`, `Scripting__Worker__PeCacheDir=C:\ProgramData\OtOpcUa\script-pe-cache` (installer), `Scripting__Runner=OutOfProcess`, `Scripting__Runner__Seams__0=Calculation`, `Scripting__Runner__Seams__1=VirtualTag` |
|
||||
| vq03, vq04 | Kept, already present and identical: `Security__ApiKeys__Pepper`, `LocalDb__Replication__ApiKey` |
|
||||
| ConfigDb (QA) | `EnableSnapshotIsolation` applied |
|
||||
| vd03 | No change |
|
||||
|
||||
## Rollback
|
||||
|
||||
- **Runner only:** set `Scripting__Runner=InProcess` (or remove the four `Scripting__Runner*` entries) and restart the
|
||||
node. To remove the isolation, run `Install-ScriptWorkerIsolation.ps1 -InstallRoot E:\ApiInstall\OtOpcUa\app
|
||||
-HostServiceAccount LocalSystem -ConfigureHostService -Uninstall`.
|
||||
- **QA binaries:** run `E:\psrem-stage\otopcua-fea7de0a\rollback.ps1` on **both** nodes (stop both first), then start both.
|
||||
The script does three things:
|
||||
1. `app` → `app.failed-fea7de0a`, and `app.bak.20261112-pre-fea7de0a` → `app`;
|
||||
2. mirrors `backup\data-pre-fea7de0a` back over `data\`;
|
||||
3. imports `backup\OtOpcUaHost-pre-fea7de0a.reg`, which restores the pre-deploy environment, without the Scripting entries.
|
||||
|
||||
The isolation ACEs, firewall rule and anchor service are harmless to `b8b4c18f`. No schema undo is needed.
|
||||
`OtOpcUaConfig_pre-fea7de0a_20261112.bak` is the DB fallback.
|
||||
- **DEV:** `E:\ApiInstall\_stage-otopcua-fea7de0a\rollback.ps1` (→ `5fb4b4db`).
|
||||
- SDK 1.5 tooling caveat after a rollback: see the rollout runbook §Rollback item 4 (#880).
|
||||
@@ -4,6 +4,9 @@
|
||||
> `363f0d81`; nothing between that and `5fb4b4db` changed the schema, configuration or preflight. The execution
|
||||
> record is §(h); the owner-action table and "Not determined" list below carry the outcomes. **All six owner actions
|
||||
> are closed and nothing is left undetermined (close-out 2026-11-12, §h last bullet).**
|
||||
>
|
||||
> **Superseded on vd03 the same day:** vd03 was upgraded again, `5fb4b4db` → `fea7de0a` (ServiceLevel 255 → 250), as the
|
||||
> DEV rehearsal for the QA pair. Record: [`qa-upgrade-b8b4c18f-to-fea7de0a.md`](qa-upgrade-b8b4c18f-to-fea7de0a.md).
|
||||
|
||||
Upgrade-readiness notes for the DEV box wonder-app-vd03. The box runs the `OtOpcUaHost` service from
|
||||
`E:\ApiInstall\OtOpcUa`: one fused admin+driver node with one FOCAS driver (Makino Z-34184). The ConfigDb
|
||||
|
||||
Reference in New Issue
Block a user