lmxopcua/docs/deploy/vd03-upgrade-b8b4c18f-to-master.md

wonder-app-vd03 upgrade: b8b4c18f → master 363f0d81

Upgrade-readiness notes for the DEV box wonder-app-vd03. The box runs the OtOpcUaHost service from E:\ApiInstall\OtOpcUa: one fused admin+driver node with one FOCAS driver (Makino Z-34184). The ConfigDb is OtOpcUaConfig on the box's SQL instance (docs/deployments/wonder-app-vd03-makino-z-34184.md). Tracking issue: Gitea #793.

Baseline. The deployed build is b8b4c18f (2026-09-03), not the e82881c1 recorded in the 2026-08-11 notes. The evidence:

• All 409 source documents in six deployed PDBs match b8b4c18f.
• deps.json pins LocalDb 0.3.4 and MxGateway.Client 0.2.1, which only b8b4c18f used.

Everything below is the b8b4c18f..master delta. The ConfigDb schema is the same at both commits. The last migration at both is 20260731162342_AddApiKeyTable.

⛔ The remote-exec hold on the wonder boxes still applies (root CLAUDE.md): no psbridge and no WinRM. The owner runs every step on the box. SMB copies and direct SQL logins are allowed.

Owner actions, in order

#	Action	Blocks	Source
1	Run scripts/preflight/vd03-preupgrade-b8b4c18f-configdb-checks.sql against OtOpcUaConfig, then fix every attention + blocking = YES row.	The first deploy on master	§g
2	Add Security__ApiKeys__Pepper to the service environment (at least 16 characters, or a ${secret:} reference). The box does not have this setting.	Host start: the host exits with code 1	§b
3	Check that Security__Jwt__SigningKey is at least 32 UTF-8 bytes.	Host start	§b
4	Apply the EnableSnapshotIsolation migration to OtOpcUaConfig.	Every deploy (SQL error 3952)	§a
5	Check the 22 FOCAS tags for the MaxRpm → MotorSpeedPct and AA_1/AA_2 → AA6/AA7 renames, and for String on MACRO/PMC/PARAM addresses. Tell ScadaBridge about the FOCAS value changes.	Client-visible	§d, check 13
6	Make the #782 go/hold decision for SDK 2.0.1. The rollout runbook still marks it HOLD.	Policy	§c

(a) ConfigDb migration

• One migration is in range: 20261022102459_EnableSnapshotIsolation, from commit 73e74e27.
  • It runs ALTER DATABASE CURRENT SET ALLOW_SNAPSHOT_ISOLATION ON with suppressTransaction: true.
  • It changes no tables. Down is deliberately a no-op.
  • Source: src/Core/ZB.MOM.WW.OtOpcUa.Configuration/Migrations/20261022102459_EnableSnapshotIsolation.cs.
• Why it is required. DraftSnapshotFactory.FromConfigDbAsync now reads the whole draft inside one IsolationLevel.Snapshot transaction. Without the option, the first read fails with SQL error 3952 and the deploy is rejected. Source: the class remarks in Configuration/Validation/DraftSnapshotFactory.cs.
• The migration is not applied automatically. The host never migrates the ConfigDb: Program.cs:129 only migrates the secrets store. docker-dev runs a one-shot migrator container because "production owns schema changes" (docker-dev/README.md, docker-dev/docker-compose.yml). The owner applies it by hand, using either of these:
  • Option 1: dotnet ef database update 20261022102459_EnableSnapshotIsolation from src/Core/ZB.MOM.WW.OtOpcUa.Configuration, with OTOPCUA_CONFIG_CONNECTION pointing at OtOpcUaConfig (DesignTimeDbContextFactory, docs/Configuration.md).
  • Option 2: run this idempotent script in SSMS. It is the output of dotnet ef migrations script 20260731162342_AddApiKeyTable 20261022102459_EnableSnapshotIsolation --idempotent:

    IF NOT EXISTS (SELECT * FROM [__EFMigrationsHistory] WHERE [MigrationId] = N'20261022102459_EnableSnapshotIsolation')
    BEGIN ALTER DATABASE CURRENT SET ALLOW_SNAPSHOT_ISOLATION ON; END;
    GO
    IF NOT EXISTS (SELECT * FROM [__EFMigrationsHistory] WHERE [MigrationId] = N'20261022102459_EnableSnapshotIsolation')
    BEGIN INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) VALUES (N'20261022102459_EnableSnapshotIsolation', N'10.0.7'); END;
    GO
    

• Login and timing. The login needs ALTER on the database. Turning the option on waits for in-flight transactions but does not need exclusive access, so it can run before the binary swap while the old build is still up. The old build is unaffected by it: it uses no IsolationLevel and ALLOW_SNAPSHOT_ISOLATION does not change default reads. A rollback needs no schema undo.

(b) Configuration: environment variables and appsettings

The 23 service environment variables against master (names only; values were not inspected):

Variable	On master
ASPNETCORE_URLS, Cluster__Hostname, Cluster__Port, Cluster__PublicHostname, Cluster__SeedNodes__0, ConnectionStrings__ConfigDb, DOTNET_ENVIRONMENT, LocalDb__Path, Security__Auth__DisableLogin, Security__DeployApiKey	Unchanged. Security:DeployApiKey now also gates the new GET /api/deployments/{id} (AdminUI/Api/DeployApiEndpoints.cs).
Cluster__Roles__0/1 + OTOPCUA_ROLES	Unchanged (EffectiveRoles is already in the baseline). Cluster:Roles wins. If the two sets disagree, master logs a Warning. If a role were a cluster-… role, #678.6 (f292cf3b) now still loads appsettings.admin-driver.json (docs/Configuration.md).
Secrets__SqlitePath	Unchanged. The Secrets 0.6.2 absolute, outside-content-root path rule (b037cc7a) is already in the baseline, and the box boots with it.
Security__Jwt__SigningKey/Issuer/Audience	Re-interpreted: now validated at start (f72af76e, 3ddb6876; Security/Jwt/JwtOptions.Validate). When a key is set, it must be at least 32 UTF-8 bytes, and Issuer/Audience must be non-blank, or the host does not start. Bearer validation is now the first-party AddJwtBearer, pinned to HS256.
ServerHistorian__Enabled/Endpoint/ApiKey/UseTls	Same keys. The HistorianGateway client goes 0.3.0 → 1.0.0 (230cef4e, historiangw#131). On UseTls=false, the cert options are ignored, not refused. A partially valid CaCertificatePath bundle now fails client creation. Not determined: the oldest gateway build that master works with. If Enabled=true on the box, check the box's gateway version first.
GALAXY_MXGW_API_KEY	Not read by any code directly. A Galaxy config that references env:GALAXY_MXGW_API_KEY is in scope for #815: GALAXY_MXGW_ is an allowed Galaxy prefix and API_KEY matches [A-Z0-9_]. It is also allowed under #835/#849 as a shared name, so the deploy gate does not refuse it. It would only collide with a Galaxy driver whose id or name has the env form API or API_KEY (DriverCredentialPolicy.cs, the GalaxyScope definition and the comment at line 374). The box has no Galaxy driver. The 2026-08-11 notes say this key was already rejected by the gateway.
Authentication__Ldap__DevStubMode	Dead at both commits. No code reads an Authentication: section; the live key is Security:Ldap:DevStubMode, which the deployed appsettings.admin-driver.json sets to false, with Transport=Ldaps. Harmless.

New settings master needs that are not in that list:

• Security__ApiKeys__Pepper: REQUIRED. It is missing on the box.
  • ApiKeyOptionsValidator (f72af76e, Host/Configuration/ApiKeyOptionsValidator.cs) fails host start on any admin or driver node when the pepper is blank or shorter than 16 characters.
  • The box has no pepper in its environment and none in the deployed appsettings*.json. Without it the upgraded service exits during start-up. Because that happens before Serilog starts, the reason only reaches stderr and the Event Log.
  • On b8b4c18f an empty pepper already made minting refuse and verification reject every key (ApiKeySecret.cs, ApiKeyOpcUaUserAuthenticator.cs). So no working data-plane API keys exist today, and choosing a fresh pepper invalidates nothing. Check 14 counts the ApiKey rows.
  • Supply it as a ${secret:} reference or an environment variable. Never commit it.
• Everything else is optional and has defaults:
  • OpcUa:CredentialLockout:* (#833/#844/#845; UserName/API-key only, never Anonymous).
  • Security:ApiKeys miss budgets, StoreTimeoutMs and StoreCacheSeconds.
  • DriverWrites:Timeout (10 s, #777).
  • ServerHistorian:EventReadHorizon (00:00:00 = off, #694).
  • ServerHistorian:Gateway:*.
  • Scripting:Runner: the default stays InProcess. #793 says not to enable Shadow or OutOfProcess on the wonder boxes yet.
  • Out-of-range values for these log a Warning or fail start only when they are set (docs/Configuration.md, docs/security.md).

Deployed appsettings vs master. The deployed files were diffed against the b8b4c18f and master copies, with secrets redacted.

• appsettings.json:
  • ContinuousHistorization still carries the pre-b8b4c18f values (Capacity 0, DrainBatchSize 64; no DrainConcurrency, MaxWriteAttempts or CaptureWhilePrimaryUnreachable). Master's in-code defaults fill the absent keys, and the box's explicit values win. This only matters if continuous historization is enabled.
  • DriverWrites and EventReadHorizon are absent, so the defaults apply.
  • The box uses Secrets:MasterKey FilePath instead of the shipped EnvVarName. That is a site choice and is unchanged.
• appsettings.driver.json: master lowers Opc.Ua from Debug to Information. The fused node loads the admin-driver overlay instead, which is unchanged.
• Keep the box's own appsettings*.json across the swap. Do not take the publisher's copies.

Other host-level changes:

• The publish output now contains script-worker/ (about 97 MB). It is unused while the runner is InProcess (docs/ServiceHosting.md §Script worker processes).

• /health/ready gains three checks:

  • OPC UA endpoint started;
  • script runner (at worst Degraded);
  • deploy coordinator (at worst Degraded).

  Source: Host/Health/HealthEndpoints.cs.

• No new ports are opened.

(c) PKI and OPC UA SDK 2.0.1 (#782)

• SDK version. It moves from 1.5.378.106 to 2.0.1 (Directory.Packages.props), passing through 1.5.378.182 at 2da58d03. Namespace URIs and NodeIds are unchanged.
• Certificates and policies. The PKI store is still the Directory store under OpcUa:PkiStoreRoot (default pki, beside the exe). The existing application certificate is reused with the same thumbprint and private key. Security profiles are unchanged: None, Basic256Sha256Sign and Basic256Sha256SignAndEncrypt. Sources: docs/security.md, docs/plans/2026-11-10-opcua-sdk-2-upgrade.md.
• CA KeyUsage. #793 says CA certificates must carry the keyCertSign/cRLSign KeyUsage, or SDK 1.5.378.182+ rejects them. The upgrade plan (line 159) lists it as a preflight check. Before the swap, check every CA in pki\issuer and pki\trusted, and ScadaBridge's trust of this server's chain. A self-signed application certificate is not a CA and is not affected. The repo states no further KeyUsage rule.
• Release status. docs/plans/2026-11-10-opcua-sdk-2-rollout.md still marks the release HOLD. Its open gates are the memory soak (#799), Windows service validation (#800) and live consumer testing (#801). vd03 would be the first Windows host to run SDK 2. Record the go/hold decision in #782.
• ScadaBridge. The adapter passed its 18 tests on None + API key. SignAndEncrypt from ScadaBridge is still untested because ScadaBridge has no client certificate yet (ScadaBridge#121; the issue predates this upgrade).

(d) What ScadaBridge will see (b8b4c18f..master)

The detail is in ../scadaproj/.claude/skills/sister-project-history/otopcua.md; the commits are cited below.

On this FOCAS box:

• Macros. Macros are now decoded at the right offset (665ecd55), so parts-count/parts-required (MACRO:3901/3902) change magnitude. A vacant macro reads BadNoData. A fractional value on an integer tag reads BadTypeMismatch.
• FixedTree.
  • Positions are scaled by their envelope (69e0a622; for example 2801574 → 280.1574).
  • Servo/spindle Load is now a real percent, and spindle Load is Float64 (1dba46bd).
  • Spindle/*/MaxRpm is renamed MotorSpeedPct (1dba46bd).
  • Axes/AA_1/AA_2 are renamed AA6/AA7 (9460da51).
  • Unreported program fields read BadNoData instead of a Good 0 (26b512ad).
  • A CNC refusal makes only that tag Bad and no longer Degrades the driver (2724d7da, #740).
  • String on a PMC/PARAM/MACRO address is refused, and the tag is skipped (cc140593).
  • New nodes: a Status/ subtree (#257) and optional ALARM: conditions (#647).
• ServiceLevel drops to 0 if the OPC UA endpoint fails to start (#762).
• Virtual tags. A disabled virtual tag reads BadOutOfService (#787). VT→VT reads now produce values (#791).
• Logins and API keys.
  • Credential lockout now covers UserName/API-key logins only, never Anonymous (#833/#844/#845).
  • Unknown API keys are rate-limited per client (#751).
  • Expired or disabled keys are refused for writes and acks (#563).
• ACL. Event and subscription filtering by ACL changes only if Security:Acl:Enabled is on (#586/#588, #797/#808, #826). Whether it is on for vd03 was not determined.
• Address-space changes. Surgically added raw nodes are now announced with model-change events (#599). Native condition state survives a rebuild (#641), and the native alarm feed survives a deploy (#662).
• Alarms and history. Each method transition emits exactly one event (#587). Overload deferral is per condition (#838/#854/#855). Event history now carries the event type and condition name (#746). Processed reads with no historian return BadAggregateNotSupported (#682).

Not reachable here (no such driver):

• Galaxy: OPC DA quality grading of status rows (#866, 363f0d81), first-value race (#859), acks wait for confirmation (#869).
• MTConnect: blank string reads Good "" (#824).
• OpcUaClient: host-name check (#816).
• Heidenhain: new driver (#803).
• AbCip: the native transport is the default.

(e) Install, swap and rollback

The source is docs/plans/2026-11-10-opcua-sdk-2-rollout.md (§Sequence, §Rollback). docs/ServiceHosting.md §Install/Uninstall describes a fresh install only.

1. Publish a fresh directory: dotnet publish src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj -c Release -r win-x64 --self-contained true -o <dir>. Hash it into a manifest stored outside the directory, then copy it to the box over SMB.

2. Back up these:

   • the whole install, with the service stopped: robocopy E:\ApiInstall\OtOpcUa <backup> /E /COPYALL /DCOPY:DAT. This covers pki\, appsettings*.json, data\ (LocalDb) and logs\;
   • the secrets SQLite file at Secrets__SqlitePath (it is outside the install), and its master-key file;
   • reg export HKLM\SYSTEM\CurrentControlSet\Services\OtOpcUaHost. It holds secrets, so protect the file;
   • BACKUP DATABASE OtOpcUaConfig.

3. Add the pepper (action 2) and check the JWT key length (action 3) on the service key. Apply the migration (§a).

4. Swap the directories:

   • Stop-Service OtOpcUaHost and wait for Stopped.
   • Move-Item the live install to a fresh sibling $PreviousInstall, then Move-Item the candidate into E:\ApiInstall\OtOpcUa.
   • Copy back the box's appsettings*.json, pki\ and data\.
   • Start-Service and wait for Running.

5. Never run Install-Services.ps1, Uninstall-Services.ps1 or Refresh-Services.ps1 here.

   • sc delete wipes the 23 environment variables.
   • Install-Services.ps1 rewrites the Environment value with only 2–3 entries.
   • Refresh-Services.ps1 assumes NSSM and C:\publish.
   • The folder swap leaves the registry key untouched.

6. Rollback.

   • Stop the service and move the candidate to a quarantine path.
   • Restore $PreviousInstall (or the robocopy backup), including its appsettings*.json, pki\ and data\.
   • Check the certificate thumbprint, then start the service.
   • Clients reconnect. Do not replay writes or acks.

   No schema undo is needed (§a). Remove the pepper only if it got in the way; b8b4c18f ignores it. The baseline already reads gzip artifacts (9dbd5e96 predates b8b4c18f), so artifacts written by master stay readable after a rollback.

   Not determined: whether the b8b4c18f build reads a LocalDb or alarm-outbox file after master has written to it. Restore data\ from the backup.

(f) Post-deploy checks

• Boot log logs\otopcua-YYYYMMDD.log. Read its contents; the listed size shows 0 while the service holds it open.

  • Expect OtOpcUa.Host starting with roles=[admin,driver], with no roles-disagreement warning.
  • About 15 s later expect Credential-reference preflight (lmxopcua#815): every stored and deployed reference is namespaced. This comes from CredentialReferencePreflightService in AdminUI/Deployments/CredentialReferencePreflight.cs.
  • If the service stops immediately, check the Windows Event Log and stderr for the pepper/JWT FATAL: message (Program.cs, #618).

• Health, at http://wonder-app-vd03.zmr.zimmer.com:9000:

  • /healthz: process up.
  • /health/ready: ConfigDb, Akka Up, OPC UA endpoint started, script runner, deploy coordinator.
  • /health/active.

  All three should return 200 on this single node (Host/Health/HealthEndpoints.cs, docs/ServiceHosting.md §Health endpoints).

• /raw credential banner (#834, warn-only by owner decision). It should be absent. FOCAS has no credential fields (docs/security.md §"Credential references are namespaced per driver"). A banner listing rows means a stored, deployed or running out-of-namespace reference that the next deploy will refuse.

• /hosts. The FOCAS driver row should be Healthy, with the per-node chips (#847/#851). Galaxy host statuses do not apply here (no Galaxy driver).

• Deploy once from the AdminUI, then check /fleet and /deployments.

  • Expect Applied, not Applied (degraded) (#601, docs/Redundancy.md). Node rows show Applying until the address-space outcome is reported (#868).
  • A 3952 error means §a was skipped.
  • A VirtualTagNoEffectiveTrigger, CredentialReferenceOutOfScope or similar rejection means a preflight row was missed.

• OPC UA at opc.tcp://wonder-app-vd03.zmr.zimmer.com:4840/OtOpcUa:

  • The server certificate thumbprint is unchanged (A873DF51…) and ServiceLevel is non-zero.
  • Browse the raw and UNS trees and read parts-count with the Client CLI (docs/Client.CLI.md). Expect the new magnitude.
  • From ScadaBridge: it reconnects, values are Good, and any MaxRpm/AA_x/Load bindings have been re-pointed.

(g) Preflight SQL: scripts/preflight/vd03-preupgrade-b8b4c18f-configdb-checks.sql

What it is. The script is read-only: it uses SELECT plus table variables. It is written against the b8b4c18f schema and needs SQL Server 2016+ at compatibility level 130 or higher.

How to run it: sqlcmd -S <instance> -d OtOpcUaConfig -E -i vd03-preupgrade-b8b4c18f-configdb-checks.sql -W -s "|" -o preflight.txt

Output.

• Each check produces a header row (ok / attention / info, the row count, whether it blocks the deploy, and its source), followed by its detail rows.
• Credential values are never printed: reference names are shown, and literals are redacted.

#	Check	Blocks deploy?	Enforced on master by	Source
0	Environment: database, compatibility level, last migration, snapshot-isolation state	—	—	—
1	VirtualTagNoEffectiveTrigger	yes	DraftValidator.ValidateVirtualTagTriggers	#783 d9b95dcf
2	CalculationNoEffectiveTrigger	yes	DraftValidator.ValidateCalculationTags rule 3	4d3fc9c8
3	Scripts that read a VirtualTag: they now resolve and start producing values	no (behaviour)	VirtualTagHostActor via dependency mux	#791 000d7f55
4	VirtualTagDependencyCycle (approximate: bounded search, 25 hops)	yes	DraftValidator.ValidateVirtualTagDependencyCycles	#791 000d7f55
5	Newly forbidden script APIs (#793 list plus E2's System.Threading, Type reflection, RuntimeHelpers)	no (compile failure at runtime → node Bad)	Core.Scripting/ForbiddenTypeAnalyzer.cs	#792 0e96dd08, E2 2e916031
6	CredentialReferenceOutOfScope, plus the #836 UserCertificatePath rule	yes	CredentialReferenceDeployChecks, DriverCredentialPolicy	#815 a224fd63, #836 63331b04, #850 3dbfe6ec
7	CredentialReferenceOtherInstance (secret: segment / env: part names another same-type driver)	yes	DriverCredentialPolicy.DescribeOtherInstanceProblems	#835 63331b04, #849 4a6229e4/d68dff09
8	Reserved driver names (drv, DRV-<12 hex>)	no (save/rename refused)	DriverCredentialPolicy.IsReservedInstanceName	#848/#849 d08e59ee/d68dff09
9	MTConnectMultipleDevices	yes	DraftValidator.ValidateMTConnectSingleDevice	#628 dcc827ca
10	Sql: SqlQueryNotDeclared (10a), the write block (10b, review), writable mismatch (10c, warning)	10a/10b yes	SqlTagDeployChecks	#781 241318e7, f5c30ed0
11	Heidenhain drivers present (expected none; the type is new)	yes	HeidenhainDeployChecks	#803 2b45694f
12	UNS effective-name collisions (an unchanged rule, re-checked)	yes	DraftValidator.ValidateUnsEffectiveLeafUniqueness	—
13	FOCAS renames, String refusals and value changes	no (deploy warning; client-visible)	FocasTagDefinitionFactory, FOCAS wire client	1dba46bd, 9460da51, cc140593, 69e0a622, 665ecd55, 26b512ad
14	Inventory: drivers, tags, VTs, scripts, API keys	—	—	—

Approximations. T-SQL cannot reproduce the C# regexes or System.Text.Json exactly; the script header lists every approximation. In short:

• ctx.GetTag("…") literals are extracted after whitespace normalisation.
• Cycles are found by a bounded path search.
• ISJSON is stricter than System.Text.Json (comments, trailing commas).
• The #850 merged driver+device path check is shown as review.
• Check 5 is a text search, so its review patterns can be false positives (for example JsonElement.GetProperty).

Some branches cannot occur under the database's own CHECK constraints but are kept for completeness. For example, CK on VirtualTag makes ChangeTriggered = 0 without a timer impossible, so only the "no literal read" branch of check 1 can fire.

Not covered. EquipmentTagConfigInspector findings are warnings only (Deployment:TagConfigValidationMode=Warn by default, AdminOperationsActor). Cluster topology, ACL scope and EquipReferenceUnresolved are unchanged since the baseline.

How it was verified (2026-11-11):

• A throwaway database otopcua_vd03preflight_<hex> was built on the local docker-dev SQL Server 2022 (localhost,14330) at compatibility level 130. It was migrated with dotnet ef database update 20260731162342_AddApiKeyTable, which is the b8b4c18f schema.
• It was seeded with one or more rows that trigger each check. Every check fired as expected, and check 11 was the expected ok. The database was then dropped.
• The script also ran clean against the docker-dev OtOpcUa database (master schema, 53 drivers). Every gate check was ok there, which matches master's own gate accepting that config.

Not determined

• Whether the box's ServerHistorian__Enabled is true, and if so, which HistorianGateway build it points at (client 1.0.0 compatibility).
• Whether the box's JWT signing key is at least 32 bytes.
• Whether Security:Acl:Enabled is on.
• ScadaBridge's security mode against vd03.
• Whether b8b4c18f can reopen LocalDb or outbox files after master has written them; restore them from backup on rollback.
• The current #782 go/hold state. The repo runbook says HOLD.