lmxopcua/docs/ServiceHosting.md

Service Hosting (v2)

Overview

A production OtOpcUa deployment runs one binary per node. The historian backend is the external ZB.MOM.WW.HistorianGateway, deployed separately (not installed by this repo's scripts):

Process	Project	Runtime	Platform	Responsibility
OtOpcUa Host	src/Server/ZB.MOM.WW.OtOpcUa.Host	.NET 10	AnyCPU	Single fused binary. The resolved EffectiveRoles decides what to mount: admin (Blazor + auth + control-plane singletons), driver (OPC UA endpoint + per-driver actors), or both.
ZB.MOM.WW.HistorianGateway (external — separate deployment)	not in this repo	.NET 10	—	The default historian backend (ServerHistorian:Backend=Gateway; SqlServer needs no sidecar). OtOpcUa talks gRPC to it (via the ZB.MOM.WW.HistorianGateway.Client package) for HistoryRead, alarm SendEvent, and continuous WriteLiveValues. Must run RuntimeDb:Enabled=true + RuntimeDb:EventReadsEnabled=true; the API key must carry historian:read + historian:write + historian:tags:write.

Galaxy access still uses the separately-installed mxaccessgw sidecar (see docs/v2/Galaxy.ParityRig.md); the gateway owns the MXAccess COM bitness constraint (its worker is x86 net48). Nothing in the OtOpcUa repo carries that constraint anymore. (The bespoke Wonderware historian sidecar this deployment used to ship was retired — see drivers/Historian.Wonderware.md.)

v2 change. v1's separate OtOpcUa.Server + OtOpcUa.Admin Windows services merged into a single role-gated OtOpcUa.Host binary. Two installers became one (with a -Roles parameter). The whole DI graph is composed in OtOpcUa.Host/Program.cs; per-role wiring is conditional on the env var.

Role gating

Program.cs resolves EffectiveRoles.Resolve(builder.Configuration) (ZB.MOM.WW.OtOpcUa.Cluster) — the single role authority (archreview R3-04): Cluster:Roles when non-empty, else OTOPCUA_ROLES, else neither role. The SAME resolved instance drives both the conditional wiring below AND the Akka gossiped roles (AkkaClusterOptions.Roles), so the two cannot disagree — and conditionally registers services:

Role present	Wires
admin	AddOtOpcUaAuth, AddAdminUI, AddSignalR, AddOtOpcUaAdminClients, MapOtOpcUaAuth, MapAdminUI<App>, MapOtOpcUaHubs, WithOtOpcUaControlPlaneSingletons (5 admin singletons via Akka.Hosting)
driver	WithOtOpcUaRuntimeActors (DriverHostActor + DbHealthProbeActor) — and the OPC UA endpoint on port 4840
Either / both	AddOtOpcUaConfigDb, AddOtOpcUaCluster, AddOtOpcUaHealth (/health/ready, /health/active, /healthz)

Single-node dev: OTOPCUA_ROLES=admin,driver (or the Cluster:Roles equivalent). Production: typically two admin nodes (HA pair) + N driver nodes.

Per-role configuration overlays

Program.cs builds a role suffix (RoleOverlay.Suffix) by joining the resolved EffectiveRoles.Gossiped plain roles (admin/driver/dev) alphabetically with - and loads appsettings.{roleSuffix}.json as an optional overlay on top of base appsettings.json. A plain role is always lower-case in Gossiped regardless of how the operator spelled it, so the overlay filename is stable (appsettings.driver.json). cluster-{ClusterId} roles are NOT part of the filename (#678.6): they are mesh-membership tags, so Cluster:Roles: ["driver", "cluster-SITE-A"] loads appsettings.driver.json and ["admin", "driver", "cluster-MAIN"] loads appsettings.admin-driver.json. (Before #678.6 the cluster role was folded into the name, producing e.g. appsettings.admin-cluster-MAIN-driver.json, which never exists — so on every node with a cluster role no shipped overlay ever loaded.) Three overlays ship in src/Server/ZB.MOM.WW.OtOpcUa.Host/:

• appsettings.admin.json — admin-only nodes
• appsettings.driver.json — driver-only nodes
• appsettings.admin-driver.json — fused single-node dev / small deployments

All three carry Serilog log-level overrides + Security:Ldap:DevStubMode = false + Security:Ldap:Transport = Ldaps (audit table: Configuration.md § Role overlays). Configuration loading order (lowest to highest precedence) is:

appsettings.json < appsettings.{Environment}.json < appsettings.{role}.json < environment variables < command-line args

The role overlay intentionally outranks appsettings.{Environment}.json so role-level security defaults (such as DevStubMode = false) cannot be silently overridden by a developer's local appsettings.Development.json; environment variables and command-line args still outrank everything for deployment-level overrides. Overlays are optional; the base file boots a node on its own.

Akka cluster

The host joins an Akka.NET cluster bound to the address in appsettings.json::Cluster:

{
  "Cluster": {
    "Hostname": "0.0.0.0",
    "Port": 4053,
    "PublicHostname": "node-a.lan",
    "SeedNodes": ["akka.tcp://otopcua@node-a.lan:4053"],
    "Roles": ["admin", "driver"]
  }
}

• SeedNodes is ordered: a node listed in its own seed list must be entry 0, since Akka only lets seed-nodes[0] form a new cluster. AkkaClusterOptionsValidator fails startup otherwise — see Redundancy.md § Bootstrap: self-first seed ordering.
• WithOtOpcUaClusterBootstrap (in OtOpcUa.Cluster) loads the embedded HOCON (split-brain resolver, pinned dispatcher, failure detector tuning) and overlays remote endpoint + cluster options.
• All cluster singletons + per-node actors live on this single ActorSystem — there is no second Akka instance.

See Redundancy.md for the role-leader + ServiceLevel story.

Health endpoints

Both admin and driver nodes expose:

Path	Status meaning
/healthz	Process alive.
/health/ready	ConfigDb reachable + cluster member state is Up.
/health/active	Admin-role leader (the node Traefik or an HA LB should route traffic to).

Used by Traefik for the active-leader-only routing pattern (see Architecture-v2.md).

Historian backend (HistorianGateway — external)

The historian backend is the external ZB.MOM.WW.HistorianGateway, deployed and operated separately (not installed by Install-Services.ps1). OtOpcUa connects to it over gRPC via the ZB.MOM.WW.HistorianGateway.Client package — configure the ServerHistorian:Endpoint (https://host:5222) and supply ServerHistorian__ApiKey via the environment on the OtOpcUa host side. The gateway must run with RuntimeDb:Enabled=true + RuntimeDb:EventReadsEnabled=true and an API key carrying historian:read + historian:write + historian:tags:write. See Historian.md for the full config-key and deployment-prerequisite reference. (The retired Wonderware TCP sidecar: Historian.Wonderware.md.)

Script worker processes (Scripting:Runner=OutOfProcess, lmxopcua#780)

Off by default. When enabled, a driver node launches up to Scripting:Worker:Shards + 1 child processes (ZB.MOM.WW.OtOpcUa.ScriptWorker, from script-worker/ beside the host) on its first compiled script evaluation and kills them with it: a Job Object with kill-on-close on Windows, pipe end of stream plus a parent-pid poll elsewhere, and a Shutdown frame then a kill on host stop. Their stderr lands in the host log prefixed ScriptWorker[<shard>]. At startup the host kills workers a dead predecessor of the same install left behind, matched by launch marker (pid + start time + image, and only when the marker's host is gone). Scripting:Worker:PeCacheDir, when set, holds the compiled-PE cache, its HMAC key and the launch markers: put it outside the install folder (startup refuses a relative path or one inside the content root) on a directory only the service account can read. The script-runner readiness component is at worst Degraded. A self-contained host publish (dotnet publish … -r win-x64 --self-contained, the Windows installs, which have no .NET runtime) ships a self-contained worker in script-worker/ (~97 MB: it carries its own runtime) and the host starts its ZB.MOM.WW.OtOpcUa.ScriptWorker.exe apphost directly; a framework-dependent host (docker-dev) runs the worker's dll through the dotnet muxer. See docs/VirtualTags.md §"Out-of-process runner".

Worker isolation on Windows (Install-ScriptWorkerIsolation.ps1, design §14 #5/#6)

scripts/install/Install-ScriptWorkerIsolation.ps1 -InstallRoot <dir> -HostServiceAccount <account> [-DenyPaths <dir>,…] [-ConfigureHostService] (idempotent; -WhatIf prints every change; -Uninstall rolls back; -PlanOnly prints the plan on any OS) sets up:

Step	Change
Worker identity	A disabled anchor service OtOpcUaScriptWorker running as NT SERVICE\OtOpcUaScriptWorker, so the virtual account exists. It is never started.
Scripting:Worker:Identity	With NT SERVICE\OtOpcUaScriptWorker set (by -ConfigureHostService, or in appsettings), the host starts every worker under a restricted token derived from its own: every privilege but SeChangeNotifyPrivilege removed, and restricting SIDs = the identity + Everyone + BUILTIN\Users + RESTRICTED. Every access the worker makes must pass for the host account and for that set. The token's default DACL is host + SYSTEM + identity; the worker is created suspended and put in its Job Object before it runs; TEMP/TMP point at its per-launch working directory (ACL: host + SYSTEM full, identity Modify); DOTNET_EnableDiagnostics=0. The host also adds the identity to its own window station and desktop DACL (enumerate/read, and create-window on the desktop), once per process. USER32 connects to them in its DllMain and the runtime imports USER32, so without the grant coreclr.dll fails to load (ERROR_DLL_INIT_FAILED, measured on windev). The grant lives on the session's kernel objects and ends with the session; nothing persistent changes. The logon SID, which those objects normally grant, is deliberately not a restricting SID: the host's process DACL gives it PROCESS_VM_READ. Windows cannot start an ad-hoc child process as a virtual account (only the SCM can, for a service), and a separate local user would need a stored password plus SeAssignPrimaryTokenPrivilege on the host account; a restricted copy of the host's own token needs neither. Startup refuses the key off Windows or when malformed.
Pipe ACL	On Windows the worker pipe always carries an explicit, protected DACL: the host account and (with an identity) the worker identity, read/write, nobody else. No SYSTEM, no Administrators, no Everyone. The owner is the host token's owner, so the worker's CurrentUserOnly check of the server end still holds.
PeCacheDir	Created with a protected ACL: SYSTEM, Administrators and the host account full control, no Users. pe\ grants the identity Modify (workers compile and write entries). The HMAC key and workers\ markers are host-only.
Install folder	The identity is denied read and write on <InstallRoot> (inherited: ReadData/List, WriteData/AddFile, AppendData/AddSubdirectory, write attributes/EA, Delete, DeleteChild, ChangePermissions, TakeOwnership; Traverse/Execute, ReadAttributes and ReadPermissions stay so the loader reaches script-worker\). The worker cannot read appsettings*.json, logs or host binaries even where Users can. script-worker\ grants it read + execute.
Sensitive folders (-DenyPaths, lmxopcua#890)	The same inherited deny on every listed folder. Name every folder outside <InstallRoot> that holds keys, secrets, stores, logs or backups: the worker's restricting SIDs include Everyone and BUILTIN\Users, so anything Users can read is otherwise readable to it (and a data drive usually lets Users create files in subfolders, e.g. a certificate in pki\trusted\certs). Name the deployment folder around <InstallRoot> (one ACE covers pki\, data\, logs\, backup\, old app.bak.*) and pki\ and data\ themselves, so the deny holds even on a folder whose DACL does not inherit. Drive and share roots, script-worker\ and the PE cache are refused; each folder must exist. The list is recorded under HKLM\…\Services\OtOpcUaScriptWorker\Parameters (DenyPaths): a re-run removes the deny from folders dropped from the list, and -Uninstall removes every recorded one without repeating -DenyPaths. After an apply the installer WARNs about any item under a denied folder whose DACL is protected and has no deny of its own (the inherited deny does not reach it); without -DenyPaths it WARNs about pki/data/logs/backup/config/secrets siblings of <InstallRoot>. Wonder layout: -InstallRoot E:\ApiInstall\OtOpcUa\app -DenyPaths E:\ApiInstall\OtOpcUa,E:\ApiInstall\OtOpcUa\pki,E:\ApiInstall\OtOpcUa\data,C:\ProgramData\ZB\kek plus, box-wide (lmxopcua#891), every other app folder under E:\ApiInstall and the staging folders E:\psrem-stage, E:\psrem; keep deploy backups (.reg exports, appsettings copies) in a folder with a protected SYSTEM + Administrators ACL.
Firewall	Outbound block rule OtOpcUa-ScriptWorker-BlockOutbound on script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.exe, every profile. It covers the self-contained apphost the Windows installs run. A framework-dependent worker runs under dotnet.exe and is not covered, so there the analyzer's System.Net ban is the only network control.

The plan is pure data asserted by scripts/install/tests/Test-ScriptWorkerIsolation.ps1 (pwsh -File …, any OS). Test-ScriptWorkerIsolation.Windows.ps1 applies it for real in a scratch folder (elevated), and InstallerLayoutIsolationTests (ScriptWorker.Tests, OTOPCUA_ISOLATION_INSTALLER_E2E=1, elevated) proves under the real restricted token that a worker cannot read pki\/data\/logs\/backup\ of a scratch wonder layout and still runs. Rollback: -Uninstall -ConfigureHostService, then restart the host (-Uninstall alone also works from the record).

After an upgrade that swaps the install folder (lmxopcua#901)

A rename/swap drops the explicit ACEs on script-worker\, on the install folder and on any denied folder that is recreated or copied in (vd03's pki\): a moved-in folder carries its own DACL. Files copied into an existing folder inherit the ACEs. The installer therefore records its effective settings (InstallRoot, HostServiceAccount, PeCacheDir, HostServiceName, ConfigureHostService, beside DenyPaths) under HKLM\SYSTEM\CurrentControlSet\Services\OtOpcUaScriptWorker\Parameters, and sets that key's DACL protected, SYSTEM + Administrators only (no secrets are recorded; Users, a restricting SID of the worker token, can no longer read the list). Two argument-free modes use the record:

Command	Does
Install-ScriptWorkerIsolation.ps1 -Reapply	Re-applies with the recorded settings; one [changed]/[unchanged] line per step and a summary: line. After a swap: the three folder ACEs changed, the rest unchanged; a second run all unchanged.
Install-ScriptWorkerIsolation.ps1 -Verify	Check only. [ok]/[MISSING] per step (anchor service, record + key ACL, PE cache ACL, pe\, script-worker\, install-folder deny, every -DenyPaths deny, firewall rule and the worker exe it names, the host env entries when recorded). Exit 1 with VERIFY FAILED: … when anything is missing, exit 0 with VERIFY OK. Protected-DACL coverage gaps are WARNed, not failed.
… -Reapply -Scope InstallRoot / -Verify -Scope InstallRoot	(lmxopcua#905) Only what a swap can undo: the ACEs on the install folder, script-worker\ and a denied folder inside it (vd03's pki\). The coverage scan walks the install folder only.
… -Reapply -Scope Outside / -Verify -Scope Outside	Every other step (anchor service, records, PE cache, denies outside the install folder, firewall rule, host env). The firewall rule names the worker exe by a path a swap does not change, and its query costs 3–5 s; the swap checks in its preflight that the candidate carries the exe. The scan walks the denied folders and skips the install folder. InstallRoot + Outside = the full plan.

Both refuse to run without a full record. A node isolated by an installer older than #901 recorded only DenyPaths: run the install once with its full arguments (the per-box deploy record lists them) to write the record.

scripts/install/Invoke-OtOpcUaInstallSwap.ps1 is the swap with the gate built in: preflight (candidate carries script-worker\, PreviousInstall is fresh and on the same volume, isolation record present when isolation is installed, record InstallRoot equals -InstallRoot) → -Reapply -Scope Outside + -Verify -Scope Outside while the old build still serves (a failure stops here: nothing stopped or moved) → stop → move live → previous, candidate → live → copy -Carry items back (default appsettings*.json, pki, logs; QA: 'appsettings*.json','logs'; rollback: -NoCarry) → -Reapply -Scope InstallRoot → -Verify -Scope InstallRoot → start (-NoStart for a pair window). The two scoped gates together check every install step, so the new build never starts unverified; a failed post-swap -Verify leaves the service stopped. The script prints the stopped window and its parts (stopped window (stop issued -> start issued): 6.0 s [stop …, move 0.0 s, carry 0.1 s, isolation 5.8 s]). Backups, data\ restore, migrations and health checks stay in the deploy's own steps. Every deploy pack that swaps the folder runs it (or -Reapply + -Verify right after its own swap), and -Verify is a post-deploy gate. Test-ScriptWorkerIsolation.Windows.ps1 exercises the whole sequence (swap → -Verify fails → -Reapply → -Verify passes → -Reapply unchanged → the swap script → the scoped modes, the pre-stop gate, -NoStart, a -NoCarry rollback) on a scratch tree; -TimingFiles N times the swap on a vd03-like tree (-ScriptsDir for a before/after).

Expected stopped window (lmxopcua#905). Before #905 the swap ran the full -Reapply and -Verify while the service was stopped, and each of them walked every denied folder for the protected-DACL coverage scan with Get-Acl per item (about 3.4 ms an item on Windows PowerShell 5.1). vd03 denies all of E:\ApiInstall, so that was most of its 101 s outage on 2026-11-13 (QA: 50–60 s per node). The scan now reads only the DACL through .NET (about 20× faster per item), walks each denied folder once (a folder nested in another denied folder is no longer walked again), and inside the stopped window walks only the swapped-in install folder. Setting the ACEs themselves was never the cost: an unchanged deny is read, not rewritten, and the re-applied install-folder deny propagates over about 1,400 files in about 0.5 s. The firewall query (3–5 s a run) moved to the Outside scope. Measured on windev with a vd03-like tree (17,851 items under the denied folder, 48 protected files): the swap's stopped window went from 153.1 s to 6.0 s on Windows PowerShell 5.1 and from 101.6 s to 6.2 s on PowerShell 7 (Test-ScriptWorkerIsolation.Windows.ps1 -TimingFiles 17000). Of the 6 s, about 1.5 s per installer run is process start + script parse + module import. The Outside part (full scan, firewall, anchor-service query) takes about 23 s before the stop, with the old build serving. Add the stop, the deploy's own steps inside the window (a data\ backup) and the host start-up to /health/ready.

Install / Uninstall

• scripts/install/Install-Services.ps1 -Roles admin,driver — installs OtOpcUaHost.
• scripts/install/Install-ScriptWorkerIsolation.ps1 — the ScriptWorker identity, ACLs and firewall rule (above); only for Scripting:Runner ≠ InProcess.
• scripts/install/Invoke-OtOpcUaInstallSwap.ps1 — upgrade/rollback folder swap with the isolation -Reapply + -Verify gate built in (lmxopcua#901), scoped so only the swapped-in folder is handled while the service is stopped (lmxopcua#905, above).
• scripts/install/Uninstall-Services.ps1 — stops + removes the host service. (The historian backend is the external HistorianGateway — not installed/removed by these scripts.)

Logging

Serilog with rolling-daily file sinks. Each host writes to logs/otopcua-*.log plus stdout (NSSM/systemd-friendly). Per-environment log level overrides go in appsettings.{Environment}.json.

Depth reference

For the full host-architecture rationale (why fused vs. split, role-gating tradeoffs, multi-node deployment shapes), see docs/plans/2026-05-26-akka-hosting-alignment-design.md §3-4.