DEV vd035fb4b4db->fea7de0a(ServiceLevel 255 -> 250, 23/24 values as before). QA pairb8b4c18f->fea7de0awith stop-both/start-both: ConfigDb preflight all ok (empty configuration), snapshot-isolation migration, isolated preflight boot + pepper negative control, ScriptWorker isolation installed (idempotent, rollback drill), runner OutOfProcess on Calculation + VirtualTag. Real pair 250/240, LocalDb 0.3.7 sync connected with an auth negative control, failover and no-failback verified. Follow-up #890 (isolation does not cover sibling pki/data). Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
15 KiB
QA wonder-app-vq03/vq04 upgrade b8b4c18f → master fea7de0a (SDK 2.0.1), with the DEV rehearsal
Status: EXECUTED 2026-11-12. DEV vd03 runs
fea7de0a(from5fb4b4db). The QA pair vq03 + vq04 runsfea7de0a(fromb8b4c18f) with the out-of-process script runner on (Calculation + VirtualTag seams) and the ScriptWorker isolation installed. Tracking: #801 (SDK 2 live gate), #793 (ConfigDb preflight), #794 (worker isolation install), #780 (P3 runner flip). Template:vd03-upgrade-b8b4c18f-to-master.md.
Channel: WinRM + SMB from the Mac (the winrm skill), used only for the on-box steps of this deploy, which the owner
requested on 2026-11-12. Health, OPC UA, ScadaBridge and Overview checks ran from the Mac over HTTP / opc.tcp.
Build
- Source:
origin/masterfea7de0a(fea7de0aefd905522d187dad495c3c8163a5ec03), built in an isolated worktree. dotnet publish src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj -c Release -r win-x64 --self-contained trueon the Mac: 0 warnings, 1396 files,OtOpcUa.Host.exeProductVersion1.0.0+fea7de0a…, self-containedscript-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.exe.- Packages in
deps.json: LocalDb + LocalDb.Replication 0.3.7, MxGateway.Client 0.4.0, HistorianGateway.Client 1.0.0. - Zip
otopcua-fea7de0a.zip, SHA-256170E7392…8F8024; the hash was checked on every box before expanding. 5fb4b4db..fea7de0a: 39 commits, no ConfigDb migration, no new required setting.
DEV: wonder-app-vd03 (5fb4b4db → fea7de0a)
Same layout and scripts as §(h) of the vd03 pack (single fused node, E:\ApiInstall\OtOpcUa, data in E:\ApiInstall\OtOpcUa-data).
- Prepare (service up): registry export;
BACKUP DATABASE … COPY_ONLY→OtOpcUaConfig-pre-20261112-otopcua-fea7de0a.bak(instance default backup dir); the ConfigDb was already at20261022102459_EnableSnapshotIsolation, snapshot ON (9 migrations). The pepper was already present (26 env vars, unchanged). w32tm source USWAP-VDC-ZMR1. - Swap: stop 2 s,
OtOpcUa-datarobocopied toOtOpcUa-backups\pre-20261112-otopcua-fea7de0a, previous install atE:\ApiInstall\OtOpcUa.bak.20261112-pre-fea7de0a, boxappsettings*/pki/logscarried. Downtime about 10 s. Rollback:E:\ApiInstall\_stage-otopcua-fea7de0a\rollback.ps1. - Verified:
/healthz,/health/ready(opcua-endpoint, script-runner InProcess, configdb, config-publish, akka) and/health/activeall 200.- ServiceLevel 250 on the single node (was 255 on
5fb4b4db, #884/#885). - Raw + UNS browse identical to before (ns=2/ns=3 NodeIds).
- 23/24 variables Good, the same set as before. The one non-Good is
Program/BlockCount, BadNoData, as before.parts-count(raw and UNS) Good,Spindle/S1/MotorSpeedPctGood,Program/NameGood. - AdminUI
/,/hosts,/fleet,/deployments,/rawall 200 (login disabled on DEV). - Boot log:
roles=[admin,driver], the #815 preflight line, noRegisterServernoise (#876), no new ERR. The repeateddomain … is not listed in the server certificateERR comes from CLI clients dialling the FQDN against theCN=OtOpcUacertificate. It was there before the upgrade too (36 lines earlier the same day). - ScadaBridge on vd03:
/health/ready200. It has no OPC UA link to OtOpcUa (MxGateway only), so it is unaffected.
- Not run on DEV: a post-deploy AdminUI deploy (no configuration change) and Galaxy alarm history (vd03 has no Galaxy driver).
QA preflight (#793)
Run on the box with the service's own SQL connection (wwadmin → wonder-sql-vq03, OtOpcUaConfig), the Mac
login being unable to open the DB.
- ConfigDb checks (
scripts/preflight/vd03-preupgrade-b8b4c18f-configdb-checks.sql,sqlcmd -I): all 15 checksok, including #783 VirtualTagNoEffectiveTrigger, #791 VT→VT and VirtualTagDependencyCycle, #792/E2 forbidden script APIs, and the credential/MTConnect/Sql/Heidenhain/UNS/FOCAS rules.- The QA ConfigDb is empty: 0 deployments, 0 driver instances, 0 tags, 0 virtual tags, 0 scripts, 0 scripted alarms, 0 API keys, 0 NodeAcl rows.
- Two ClusterNode rows:
10.220.158.2:4053(vq03) and10.220.158.3:4053(vq04), clusterDEV, ServiceLevelBase 200. - Migrations 8 (last
AddApiKeyTable), snapshot isolation OFF, compat 160.
- PKI / CA KeyUsage (#782): each node has only its own self-signed
CN=OtOpcUacertificate (CA:FALSE). vq03 is06498CB2…, SANurn:wonder-app-vq03:OtOpcUa+ IP 10.220.158.2. vq04 isCE4859A4…, SANurn:wonder-app-vq04:OtOpcUa- IP 10.220.158.3. Both are valid to 2027-08-11. There are no CA or issuer certificates, so nothing for the stricter
rule to reject. ScadaBridge QA has no OPC UA connection to OtOpcUa (its one data connection is
MxGateway), so there is no trust chain to check.
- IP 10.220.158.3. Both are valid to 2027-08-11. There are no CA or issuer certificates, so nothing for the stricter
rule to reject. ScadaBridge QA has no OPC UA connection to OtOpcUa (its one data connection is
- Service environment: 41 entries on vq03, 40 on vq04 (vq04 has no
PeerAddress). Compared by SHA-256 prefix only:Security__ApiKeys__Pepperis already set (44 chars) and identical on both nodes.LocalDb__Replication__ApiKeyis already set (44 chars) and byte-identical on both nodes. This is what LocalDb 0.3.7 requires; it lives in the service registry environment, not in a file.- The JWT key is 64 bytes, with the default Issuer/Audience.
ServerHistorianis enabled →http://10.220.158.2:5221(vq03's HistorianGatewayd8d890c),UseTls=false.- No
OpcUa:EnabledSecurityProfilesanywhere, so #879 changes nothing here.
- Clocks (#873): vq03 → USWAP-VDC-ZMR3, vq04 → USSBN-VDC-ZMR1, both synced within minutes.
- HistorianGateway: not upgraded. Master uses only RPCs that
d8d890cserves (ReadRaw/Aggregate/AtTime/Events, WriteLiveValues, SendEvent, EnsureTags, Probe). It sends its own event GUIDs and does not use the new store-forward fields or string backfill (docs/Historian.md). The preflight boot loggedHistorian gateway probe: the gateway is reachableagainstd8d890cwith Client 1.0.0 over h2c. QA has no historized tags. historiangw#78 stays its own deploy. - mxaccessgw: not upgraded. The #877 stream-attached ack is capability-negotiated (
stream-events-attached-ack;SupportsStreamAttachedAckdefaults false → the old best-effort wait). The #749 operator password is an additive proto field that the old gateway ignores. QA OtOpcUa has no Galaxy driver at all, so4f931e4stays. - Isolated preflight boot on vq04 (live service running): the candidate ran from a copy with the node's real
service environment, using its own ports (HTTP 19000, Akka 14053 seeded to itself, OPC UA 14840, sync 19101), a
clone
OtOpcUaConfig_pf(snapshot migration applied) and copies ofdata\/pki\.- A:
/health/ready200 in 11 s, all five components Healthy. - B (negative control): a 5-character pepper → exit 1,
FATAL … Security:ApiKeys:Pepper must be at least 16 characters. - The clone was dropped afterwards.
- Finding: the first A run also overrode
OpcUa:PublicHostname=127.0.0.1. SDK 2.0.1 then refused to start the endpoint (The certificate with subject CN=OtOpcUa in the configuration is invalid), because 127.0.0.1 is not in the certificate SAN./health/readywent 503 (opcua-endpoint Unhealthy) while/health/activestayed 200. SoOpcUa:PublicHostnamemust stay a name or IP in the certificate SAN, and only readiness shows a dead endpoint.
- A:
QA window (stop both, swap both, start both; owner decision, no mixed-SDK rolling restart)
Layout: E:\ApiInstall\OtOpcUa\{app,data,pki}, service OtOpcUaHost (LocalSystem), staging E:\psrem-stage\otopcua-fea7de0a.
- Prepare (services up):
- Candidate expanded to
app.new-fea7de0a(1396 files, hash ok) and the box's fiveappsettings*.jsoncarried into it (SHA-identical; the publisher's copies were not used). - Isolation
-PlanOnly: 7 steps. - Registry export
backup\OtOpcUaHost-pre-fea7de0a.regon each node. BACKUP DATABASE OtOpcUaConfig … COPY_ONLY, CHECKSUM, COMPRESSION→OtOpcUaConfig_pre-fea7de0a_20261112.bak(default backup dir on wonder-sql-vq03),RESTORE VERIFYONLYok.- Migration
20261022102459_EnableSnapshotIsolationapplied to the live DB (snapshot ON, 9 migrations), whileb8b4c18fwas still running. The old build does not use it.
- Candidate expanded to
- Stop + swap (16:48:13–16:48:20 EST): vq03 (Secondary) stopped in 1 s, then vq04 (Primary) in 1 s.
data\andpki\robocopied tobackup\{data,pki}-pre-fea7de0a(stopped, so the SQLite triplets are consistent).app→app.bak.20261112-pre-fea7de0a,app.new-fea7de0a→app,logs\carried; appsettings matched.
- #794 isolation on the new
app\(the swap drops folder ACEs), both nodes, services stopped. Run withInstall-ScriptWorkerIsolation.ps1 -InstallRoot E:\ApiInstall\OtOpcUa\app -HostServiceAccount LocalSystem -ConfigureHostService:-WhatIf: 7 operations, nothing changed.- Apply: 7/7
changed. Second apply: 7/7unchanged. - Rollback drill:
-Uninstall: 6 changed (the PE cache is kept). Second uninstall: 6/6unchanged. - Re-apply: 6 changed + PeCacheDir unchanged. Second re-apply: 7/7
unchanged. - State verified on both nodes:
sc qc OtOpcUaScriptWorker: DISABLED,NT SERVICE\OtOpcUaScriptWorker.showsid:S-1-5-80-1846135827-…-3023753135.C:\ProgramData\OtOpcUa\script-pe-cache: SYSTEM and Administrators full, no Users. Itspe\adds the identity(OI)(CI)(M).app\: identity(OI)(CI)(DENY)(S,RD).app\script-worker: identity(OI)(CI)(RX)(explicit, ahead of the inherited deny).- Firewall rule
OtOpcUa-ScriptWorker-BlockOutbound: Outbound / Block / Any / enabled, programE:\ApiInstall\OtOpcUa\app\script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.exe. - Service env gained
Scripting__Worker__IdentityandScripting__Worker__PeCacheDir, with the other entries intact.
- #780 P3 flip (registry env):
Scripting__Runner=OutOfProcessandScripting__Runner__Seams__0=Calculation.- After the start below, widened to
Scripting__Runner__Seams__1=VirtualTag. ScriptedAlarm is not enabled. - Env count is 46 on vq03 and 45 on vq04.
- Start vq04 (16:49:34 → ready 16:50:14, 39 s: it waits for its peer), then vq03 (ready 16:50:23 in 8 s). Pair outage 16:48:13 → 16:50:14, about 2 min, of which about 70 s was the isolation drill.
- Cleanup: the clone DB and the preflight copy were removed. The staging dir and backups stay on each node.
QA verification (#801)
| Check | Result |
|---|---|
| Version | OtOpcUa.Host.exe 1.0.0+fea7de0a… on both. |
| ServiceLevel on the real pair | vq04 250 (Primary), vq03 240 (Secondary), stable over repeated fresh-session reads. ApplicationUri per node unchanged. |
| Health | Both /health/ready 200 with opcua-endpoint, script-runner (Out-of-process script runner ready.), configdb, config-publish and akka Healthy. /health/active 200 on the Primary, 503 on the Secondary (by design). |
| LocalDb sync (0.3.7, ApiKey enforced) | localdb-replication: connected, oplog backlog 0 on both nodes, after the window and again after the failover. Negative control: a sync call from the Mac with no bearer token and one with a bogus token both get grpc-status 7 LocalDb sync authentication failed on both nodes, and each node logs Rejected a LocalDb sync call. |
| Restart Secondary (to pick up the VirtualTag seam) | vq03 down → 100 → 240 within about 8 s of start. vq04 held 250 throughout. |
| Failover | Stop-Service on the Primary vq04 at 16:53:42. The graceful leave took 30 s, and vq03 became oldest at 16:54:13 (OldestChanged … -> myself). From the first sample: vq03 250, /health/active 200. |
| Old Primary restart | vq04 started 16:55:32: 100 → 240 at its first sample after ready, and never above 240 (no 255, #885). No failback: vq03 stays 250 / active 200, vq04 240 / active 503. |
| AdminUI | Login posture unchanged: / → 302 /login, /login 200, /api/deployments 302. |
| Boot logs | roles=[admin,driver], the #815 preflight line, Historian gateway probe: the gateway is reachable. No FTL. The ERR lines are bootstrap-time Akka Removed (vq04 before join) and the CLI certificate-domain noise. |
Overview (vq03 + vq04 :5320) |
OtOpcUa Up, 2 instances, 2 members / 0 unreachable, 5/5 checks healthy (now including opcua-endpoint and script-runner). |
| ScadaBridge QA | /health/ready 200 on both nodes before and after. Its only data connection is MxGateway, so it reads no OtOpcUa data on QA. The consumer check is not applicable here. |
| AVEVA history via HG | Not applicable on QA: no Galaxy driver, no historized tags. The real-AVEVA gate passed separately against vd03's gateway (#801 comment, 008f0b36). |
Not exercisable on QA (empty configuration):
- No Calculation, VirtualTag or scripted-alarm exists, so no worker has started on either node. The following were therefore not observed: the restricted-token worker under the real service session, the window-station grant, the PE-cache fill, the firewall drop log, and "Calculation tags stay Good".
- The runaway drill was skipped: running it needs a script in the configuration, and authoring QA configuration was out of scope for this deploy.
script-runnerreportsOut-of-process script runner readyon both nodes.
Isolation coverage note. The installer denies the worker identity on InstallRoot (app\) only. On QA the
siblings E:\ApiInstall\OtOpcUa\pki (including own\private\*.pfx) and data\ (secrets.db, LocalDb) inherit
BUILTIN\Users:ReadAndExecute. Users is one of the worker token's restricting SIDs, so those folders are not covered
by the deny. File-level ACLs were not inspected. Follow-up: #890.
Configuration set (service registry environment; no file changes, no secrets added)
| Node | Added |
|---|---|
| vq03, vq04 | Scripting__Worker__Identity=NT SERVICE\OtOpcUaScriptWorker, Scripting__Worker__PeCacheDir=C:\ProgramData\OtOpcUa\script-pe-cache (installer), Scripting__Runner=OutOfProcess, Scripting__Runner__Seams__0=Calculation, Scripting__Runner__Seams__1=VirtualTag |
| vq03, vq04 | Kept, already present and identical: Security__ApiKeys__Pepper, LocalDb__Replication__ApiKey |
| ConfigDb (QA) | EnableSnapshotIsolation applied |
| vd03 | No change |
Rollback
-
Runner only: set
Scripting__Runner=InProcess(or remove the fourScripting__Runner*entries) and restart the node. To remove the isolation, runInstall-ScriptWorkerIsolation.ps1 -InstallRoot E:\ApiInstall\OtOpcUa\app -HostServiceAccount LocalSystem -ConfigureHostService -Uninstall. -
QA binaries: run
E:\psrem-stage\otopcua-fea7de0a\rollback.ps1on both nodes (stop both first), then start both. The script does three things:app→app.failed-fea7de0a, andapp.bak.20261112-pre-fea7de0a→app;- mirrors
backup\data-pre-fea7de0aback overdata\; - imports
backup\OtOpcUaHost-pre-fea7de0a.reg, which restores the pre-deploy environment, without the Scripting entries.
The isolation ACEs, firewall rule and anchor service are harmless to
b8b4c18f. No schema undo is needed.OtOpcUaConfig_pre-fea7de0a_20261112.bakis the DB fallback. -
DEV:
E:\ApiInstall\_stage-otopcua-fea7de0a\rollback.ps1(→5fb4b4db). -
SDK 1.5 tooling caveat after a rollback: see the rollout runbook §Rollback item 4 (#880).