Files
lmxopcua/docs/deploy/qa-upgrade-b8b4c18f-to-fea7de0a.md
T
Joseph Doherty c80d187c18
v2-ci / unit-tests (push) Has been cancelled
docs(deploy): QA vq03/vq04 + DEV vd03 upgraded to master fea7de0a (SDK 2.0.1) - #801 #793 #794 #780
DEV vd03 5fb4b4db -> fea7de0a (ServiceLevel 255 -> 250, 23/24 values as before).
QA pair b8b4c18f -> fea7de0a with stop-both/start-both: ConfigDb preflight all ok
(empty configuration), snapshot-isolation migration, isolated preflight boot +
pepper negative control, ScriptWorker isolation installed (idempotent, rollback
drill), runner OutOfProcess on Calculation + VirtualTag. Real pair 250/240,
LocalDb 0.3.7 sync connected with an auth negative control, failover and no-failback
verified. Follow-up #890 (isolation does not cover sibling pki/data).

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 17:03:05 -05:00

15 KiB
Raw Blame History

QA wonder-app-vq03/vq04 upgrade b8b4c18f → master fea7de0a (SDK 2.0.1), with the DEV rehearsal

Status: EXECUTED 2026-11-12. DEV vd03 runs fea7de0a (from 5fb4b4db). The QA pair vq03 + vq04 runs fea7de0a (from b8b4c18f) with the out-of-process script runner on (Calculation + VirtualTag seams) and the ScriptWorker isolation installed. Tracking: #801 (SDK 2 live gate), #793 (ConfigDb preflight), #794 (worker isolation install), #780 (P3 runner flip). Template: vd03-upgrade-b8b4c18f-to-master.md.

Channel: WinRM + SMB from the Mac (the winrm skill), used only for the on-box steps of this deploy, which the owner requested on 2026-11-12. Health, OPC UA, ScadaBridge and Overview checks ran from the Mac over HTTP / opc.tcp.

Build

  • Source: origin/master fea7de0a (fea7de0aefd905522d187dad495c3c8163a5ec03), built in an isolated worktree.
  • dotnet publish src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj -c Release -r win-x64 --self-contained true on the Mac: 0 warnings, 1396 files, OtOpcUa.Host.exe ProductVersion 1.0.0+fea7de0a…, self-contained script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.exe.
  • Packages in deps.json: LocalDb + LocalDb.Replication 0.3.7, MxGateway.Client 0.4.0, HistorianGateway.Client 1.0.0.
  • Zip otopcua-fea7de0a.zip, SHA-256 170E7392…8F8024; the hash was checked on every box before expanding.
  • 5fb4b4db..fea7de0a: 39 commits, no ConfigDb migration, no new required setting.

DEV: wonder-app-vd03 (5fb4b4db → fea7de0a)

Same layout and scripts as §(h) of the vd03 pack (single fused node, E:\ApiInstall\OtOpcUa, data in E:\ApiInstall\OtOpcUa-data).

  • Prepare (service up): registry export; BACKUP DATABASE … COPY_ONLY → OtOpcUaConfig-pre-20261112-otopcua-fea7de0a.bak (instance default backup dir); the ConfigDb was already at 20261022102459_EnableSnapshotIsolation, snapshot ON (9 migrations). The pepper was already present (26 env vars, unchanged). w32tm source USWAP-VDC-ZMR1.
  • Swap: stop 2 s, OtOpcUa-data robocopied to OtOpcUa-backups\pre-20261112-otopcua-fea7de0a, previous install at E:\ApiInstall\OtOpcUa.bak.20261112-pre-fea7de0a, box appsettings*/pki/logs carried. Downtime about 10 s. Rollback: E:\ApiInstall\_stage-otopcua-fea7de0a\rollback.ps1.
  • Verified:
    • /healthz, /health/ready (opcua-endpoint, script-runner InProcess, configdb, config-publish, akka) and /health/active all 200.
    • ServiceLevel 250 on the single node (was 255 on 5fb4b4db, #884/#885).
    • Raw + UNS browse identical to before (ns=2/ns=3 NodeIds).
    • 23/24 variables Good, the same set as before. The one non-Good is Program/BlockCount, BadNoData, as before. parts-count (raw and UNS) Good, Spindle/S1/MotorSpeedPct Good, Program/Name Good.
    • AdminUI /, /hosts, /fleet, /deployments, /raw all 200 (login disabled on DEV).
    • Boot log: roles=[admin,driver], the #815 preflight line, no RegisterServer noise (#876), no new ERR. The repeated domain … is not listed in the server certificate ERR comes from CLI clients dialling the FQDN against the CN=OtOpcUa certificate. It was there before the upgrade too (36 lines earlier the same day).
    • ScadaBridge on vd03: /health/ready 200. It has no OPC UA link to OtOpcUa (MxGateway only), so it is unaffected.
  • Not run on DEV: a post-deploy AdminUI deploy (no configuration change) and Galaxy alarm history (vd03 has no Galaxy driver).

QA preflight (#793)

Run on the box with the service's own SQL connection (wwadmin → wonder-sql-vq03, OtOpcUaConfig), the Mac login being unable to open the DB.

  • ConfigDb checks (scripts/preflight/vd03-preupgrade-b8b4c18f-configdb-checks.sql, sqlcmd -I): all 15 checks ok, including #783 VirtualTagNoEffectiveTrigger, #791 VT→VT and VirtualTagDependencyCycle, #792/E2 forbidden script APIs, and the credential/MTConnect/Sql/Heidenhain/UNS/FOCAS rules.
    • The QA ConfigDb is empty: 0 deployments, 0 driver instances, 0 tags, 0 virtual tags, 0 scripts, 0 scripted alarms, 0 API keys, 0 NodeAcl rows.
    • Two ClusterNode rows: 10.220.158.2:4053 (vq03) and 10.220.158.3:4053 (vq04), cluster DEV, ServiceLevelBase 200.
    • Migrations 8 (last AddApiKeyTable), snapshot isolation OFF, compat 160.
  • PKI / CA KeyUsage (#782): each node has only its own self-signed CN=OtOpcUa certificate (CA:FALSE). vq03 is 06498CB2…, SAN urn:wonder-app-vq03:OtOpcUa + IP 10.220.158.2. vq04 is CE4859A4…, SAN urn:wonder-app-vq04:OtOpcUa
    • IP 10.220.158.3. Both are valid to 2027-08-11. There are no CA or issuer certificates, so nothing for the stricter rule to reject. ScadaBridge QA has no OPC UA connection to OtOpcUa (its one data connection is MxGateway), so there is no trust chain to check.
  • Service environment: 41 entries on vq03, 40 on vq04 (vq04 has no PeerAddress). Compared by SHA-256 prefix only:
    • Security__ApiKeys__Pepper is already set (44 chars) and identical on both nodes.
    • LocalDb__Replication__ApiKey is already set (44 chars) and byte-identical on both nodes. This is what LocalDb 0.3.7 requires; it lives in the service registry environment, not in a file.
    • The JWT key is 64 bytes, with the default Issuer/Audience.
    • ServerHistorian is enabled → http://10.220.158.2:5221 (vq03's HistorianGateway d8d890c), UseTls=false.
    • No OpcUa:EnabledSecurityProfiles anywhere, so #879 changes nothing here.
  • Clocks (#873): vq03 → USWAP-VDC-ZMR3, vq04 → USSBN-VDC-ZMR1, both synced within minutes.
  • HistorianGateway: not upgraded. Master uses only RPCs that d8d890c serves (ReadRaw/Aggregate/AtTime/Events, WriteLiveValues, SendEvent, EnsureTags, Probe). It sends its own event GUIDs and does not use the new store-forward fields or string backfill (docs/Historian.md). The preflight boot logged Historian gateway probe: the gateway is reachable against d8d890c with Client 1.0.0 over h2c. QA has no historized tags. historiangw#78 stays its own deploy.
  • mxaccessgw: not upgraded. The #877 stream-attached ack is capability-negotiated (stream-events-attached-ack; SupportsStreamAttachedAck defaults false → the old best-effort wait). The #749 operator password is an additive proto field that the old gateway ignores. QA OtOpcUa has no Galaxy driver at all, so 4f931e4 stays.
  • Isolated preflight boot on vq04 (live service running): the candidate ran from a copy with the node's real service environment, using its own ports (HTTP 19000, Akka 14053 seeded to itself, OPC UA 14840, sync 19101), a clone OtOpcUaConfig_pf (snapshot migration applied) and copies of data\/pki\.
    • A: /health/ready 200 in 11 s, all five components Healthy.
    • B (negative control): a 5-character pepper → exit 1, FATAL … Security:ApiKeys:Pepper must be at least 16 characters.
    • The clone was dropped afterwards.
    • Finding: the first A run also overrode OpcUa:PublicHostname=127.0.0.1. SDK 2.0.1 then refused to start the endpoint (The certificate with subject CN=OtOpcUa in the configuration is invalid), because 127.0.0.1 is not in the certificate SAN. /health/ready went 503 (opcua-endpoint Unhealthy) while /health/active stayed 200. So OpcUa:PublicHostname must stay a name or IP in the certificate SAN, and only readiness shows a dead endpoint.

QA window (stop both, swap both, start both; owner decision, no mixed-SDK rolling restart)

Layout: E:\ApiInstall\OtOpcUa\{app,data,pki}, service OtOpcUaHost (LocalSystem), staging E:\psrem-stage\otopcua-fea7de0a.

  1. Prepare (services up):
    • Candidate expanded to app.new-fea7de0a (1396 files, hash ok) and the box's five appsettings*.json carried into it (SHA-identical; the publisher's copies were not used).
    • Isolation -PlanOnly: 7 steps.
    • Registry export backup\OtOpcUaHost-pre-fea7de0a.reg on each node.
    • BACKUP DATABASE OtOpcUaConfig … COPY_ONLY, CHECKSUM, COMPRESSION → OtOpcUaConfig_pre-fea7de0a_20261112.bak (default backup dir on wonder-sql-vq03), RESTORE VERIFYONLY ok.
    • Migration 20261022102459_EnableSnapshotIsolation applied to the live DB (snapshot ON, 9 migrations), while b8b4c18f was still running. The old build does not use it.
  2. Stop + swap (16:48:13–16:48:20 EST): vq03 (Secondary) stopped in 1 s, then vq04 (Primary) in 1 s.
    • data\ and pki\ robocopied to backup\{data,pki}-pre-fea7de0a (stopped, so the SQLite triplets are consistent).
    • app → app.bak.20261112-pre-fea7de0a, app.new-fea7de0a → app, logs\ carried; appsettings matched.
  3. #794 isolation on the new app\ (the swap drops folder ACEs), both nodes, services stopped. Run with Install-ScriptWorkerIsolation.ps1 -InstallRoot E:\ApiInstall\OtOpcUa\app -HostServiceAccount LocalSystem -ConfigureHostService:
    • -WhatIf: 7 operations, nothing changed.
    • Apply: 7/7 changed. Second apply: 7/7 unchanged.
    • Rollback drill: -Uninstall: 6 changed (the PE cache is kept). Second uninstall: 6/6 unchanged.
    • Re-apply: 6 changed + PeCacheDir unchanged. Second re-apply: 7/7 unchanged.
    • State verified on both nodes:
      • sc qc OtOpcUaScriptWorker: DISABLED, NT SERVICE\OtOpcUaScriptWorker. showsid: S-1-5-80-1846135827-…-3023753135.
      • C:\ProgramData\OtOpcUa\script-pe-cache: SYSTEM and Administrators full, no Users. Its pe\ adds the identity (OI)(CI)(M).
      • app\: identity (OI)(CI)(DENY)(S,RD). app\script-worker: identity (OI)(CI)(RX) (explicit, ahead of the inherited deny).
      • Firewall rule OtOpcUa-ScriptWorker-BlockOutbound: Outbound / Block / Any / enabled, program E:\ApiInstall\OtOpcUa\app\script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.exe.
      • Service env gained Scripting__Worker__Identity and Scripting__Worker__PeCacheDir, with the other entries intact.
  4. #780 P3 flip (registry env):
    • Scripting__Runner=OutOfProcess and Scripting__Runner__Seams__0=Calculation.
    • After the start below, widened to Scripting__Runner__Seams__1=VirtualTag. ScriptedAlarm is not enabled.
    • Env count is 46 on vq03 and 45 on vq04.
  5. Start vq04 (16:49:34 → ready 16:50:14, 39 s: it waits for its peer), then vq03 (ready 16:50:23 in 8 s). Pair outage 16:48:13 → 16:50:14, about 2 min, of which about 70 s was the isolation drill.
  6. Cleanup: the clone DB and the preflight copy were removed. The staging dir and backups stay on each node.

QA verification (#801)

Check Result
Version OtOpcUa.Host.exe 1.0.0+fea7de0a… on both.
ServiceLevel on the real pair vq04 250 (Primary), vq03 240 (Secondary), stable over repeated fresh-session reads. ApplicationUri per node unchanged.
Health Both /health/ready 200 with opcua-endpoint, script-runner (Out-of-process script runner ready.), configdb, config-publish and akka Healthy. /health/active 200 on the Primary, 503 on the Secondary (by design).
LocalDb sync (0.3.7, ApiKey enforced) localdb-replication: connected, oplog backlog 0 on both nodes, after the window and again after the failover. Negative control: a sync call from the Mac with no bearer token and one with a bogus token both get grpc-status 7 LocalDb sync authentication failed on both nodes, and each node logs Rejected a LocalDb sync call.
Restart Secondary (to pick up the VirtualTag seam) vq03 down → 100 → 240 within about 8 s of start. vq04 held 250 throughout.
Failover Stop-Service on the Primary vq04 at 16:53:42. The graceful leave took 30 s, and vq03 became oldest at 16:54:13 (OldestChanged … -> myself). From the first sample: vq03 250, /health/active 200.
Old Primary restart vq04 started 16:55:32: 100 → 240 at its first sample after ready, and never above 240 (no 255, #885). No failback: vq03 stays 250 / active 200, vq04 240 / active 503.
AdminUI Login posture unchanged: / → 302 /login, /login 200, /api/deployments 302.
Boot logs roles=[admin,driver], the #815 preflight line, Historian gateway probe: the gateway is reachable. No FTL. The ERR lines are bootstrap-time Akka Removed (vq04 before join) and the CLI certificate-domain noise.
Overview (vq03 + vq04 :5320) OtOpcUa Up, 2 instances, 2 members / 0 unreachable, 5/5 checks healthy (now including opcua-endpoint and script-runner).
ScadaBridge QA /health/ready 200 on both nodes before and after. Its only data connection is MxGateway, so it reads no OtOpcUa data on QA. The consumer check is not applicable here.
AVEVA history via HG Not applicable on QA: no Galaxy driver, no historized tags. The real-AVEVA gate passed separately against vd03's gateway (#801 comment, 008f0b36).

Not exercisable on QA (empty configuration):

  • No Calculation, VirtualTag or scripted-alarm exists, so no worker has started on either node. The following were therefore not observed: the restricted-token worker under the real service session, the window-station grant, the PE-cache fill, the firewall drop log, and "Calculation tags stay Good".
  • The runaway drill was skipped: running it needs a script in the configuration, and authoring QA configuration was out of scope for this deploy.
  • script-runner reports Out-of-process script runner ready on both nodes.

Isolation coverage note. The installer denies the worker identity on InstallRoot (app\) only. On QA the siblings E:\ApiInstall\OtOpcUa\pki (including own\private\*.pfx) and data\ (secrets.db, LocalDb) inherit BUILTIN\Users:ReadAndExecute. Users is one of the worker token's restricting SIDs, so those folders are not covered by the deny. File-level ACLs were not inspected. Follow-up: #890.

Configuration set (service registry environment; no file changes, no secrets added)

Node Added
vq03, vq04 Scripting__Worker__Identity=NT SERVICE\OtOpcUaScriptWorker, Scripting__Worker__PeCacheDir=C:\ProgramData\OtOpcUa\script-pe-cache (installer), Scripting__Runner=OutOfProcess, Scripting__Runner__Seams__0=Calculation, Scripting__Runner__Seams__1=VirtualTag
vq03, vq04 Kept, already present and identical: Security__ApiKeys__Pepper, LocalDb__Replication__ApiKey
ConfigDb (QA) EnableSnapshotIsolation applied
vd03 No change

Rollback

  • Runner only: set Scripting__Runner=InProcess (or remove the four Scripting__Runner* entries) and restart the node. To remove the isolation, run Install-ScriptWorkerIsolation.ps1 -InstallRoot E:\ApiInstall\OtOpcUa\app -HostServiceAccount LocalSystem -ConfigureHostService -Uninstall.

  • QA binaries: run E:\psrem-stage\otopcua-fea7de0a\rollback.ps1 on both nodes (stop both first), then start both. The script does three things:

    1. app → app.failed-fea7de0a, and app.bak.20261112-pre-fea7de0a → app;
    2. mirrors backup\data-pre-fea7de0a back over data\;
    3. imports backup\OtOpcUaHost-pre-fea7de0a.reg, which restores the pre-deploy environment, without the Scripting entries.

    The isolation ACEs, firewall rule and anchor service are harmless to b8b4c18f. No schema undo is needed. OtOpcUaConfig_pre-fea7de0a_20261112.bak is the DB fallback.

  • DEV: E:\ApiInstall\_stage-otopcua-fea7de0a\rollback.ps1 (→ 5fb4b4db).

  • SDK 1.5 tooling caveat after a rollback: see the rollout runbook §Rollback item 4 (#880).