lmxopcua/docs/ServiceHosting.md

Service Hosting (v2)

Overview

A production OtOpcUa deployment runs one binary per node. The historian backend is the external ZB.MOM.WW.HistorianGateway, deployed separately (not installed by this repo's scripts):

Process	Project	Runtime	Platform	Responsibility
OtOpcUa Host	src/Server/ZB.MOM.WW.OtOpcUa.Host	.NET 10	AnyCPU	Single fused binary. The resolved EffectiveRoles decides what to mount: admin (Blazor + auth + control-plane singletons), driver (OPC UA endpoint + per-driver actors), or both.
ZB.MOM.WW.HistorianGateway (external — separate deployment)	not in this repo	.NET 10	—	The default historian backend (ServerHistorian:Backend=Gateway; SqlServer needs no sidecar). OtOpcUa talks gRPC to it (via the ZB.MOM.WW.HistorianGateway.Client package) for HistoryRead, alarm SendEvent, and continuous WriteLiveValues. Must run RuntimeDb:Enabled=true + RuntimeDb:EventReadsEnabled=true; the API key must carry historian:read + historian:write + historian:tags:write.

Galaxy access still uses the separately-installed mxaccessgw sidecar (see docs/v2/Galaxy.ParityRig.md); the gateway owns the MXAccess COM bitness constraint (its worker is x86 net48). Nothing in the OtOpcUa repo carries that constraint anymore. (The bespoke Wonderware historian sidecar this deployment used to ship was retired — see drivers/Historian.Wonderware.md.)

v2 change. v1's separate OtOpcUa.Server + OtOpcUa.Admin Windows services merged into a single role-gated OtOpcUa.Host binary. Two installers became one (with a -Roles parameter). The whole DI graph is composed in OtOpcUa.Host/Program.cs; per-role wiring is conditional on the env var.

Role gating

Program.cs resolves EffectiveRoles.Resolve(builder.Configuration) (ZB.MOM.WW.OtOpcUa.Cluster) — the single role authority (archreview R3-04): Cluster:Roles when non-empty, else OTOPCUA_ROLES, else neither role. The SAME resolved instance drives both the conditional wiring below AND the Akka gossiped roles (AkkaClusterOptions.Roles), so the two cannot disagree — and conditionally registers services:

Role present	Wires
admin	AddOtOpcUaAuth, AddAdminUI, AddSignalR, AddOtOpcUaAdminClients, MapOtOpcUaAuth, MapAdminUI<App>, MapOtOpcUaHubs, WithOtOpcUaControlPlaneSingletons (5 admin singletons via Akka.Hosting)
driver	WithOtOpcUaRuntimeActors (DriverHostActor + DbHealthProbeActor) — and the OPC UA endpoint on port 4840
Either / both	AddOtOpcUaConfigDb, AddOtOpcUaCluster, AddOtOpcUaHealth (/health/ready, /health/active, /healthz)

Single-node dev: OTOPCUA_ROLES=admin,driver (or the Cluster:Roles equivalent). Production: typically two admin nodes (HA pair) + N driver nodes.

Per-role configuration overlays

Program.cs builds a role suffix (RoleOverlay.Suffix) by joining the resolved EffectiveRoles.Gossiped plain roles (admin/driver/dev) alphabetically with - and loads appsettings.{roleSuffix}.json as an optional overlay on top of base appsettings.json. A plain role is always lower-case in Gossiped regardless of how the operator spelled it, so the overlay filename is stable (appsettings.driver.json). cluster-{ClusterId} roles are NOT part of the filename (#678.6): they are mesh-membership tags, so Cluster:Roles: ["driver", "cluster-SITE-A"] loads appsettings.driver.json and ["admin", "driver", "cluster-MAIN"] loads appsettings.admin-driver.json. (Before #678.6 the cluster role was folded into the name, producing e.g. appsettings.admin-cluster-MAIN-driver.json, which never exists — so on every node with a cluster role no shipped overlay ever loaded.) Three overlays ship in src/Server/ZB.MOM.WW.OtOpcUa.Host/:

• appsettings.admin.json — admin-only nodes
• appsettings.driver.json — driver-only nodes
• appsettings.admin-driver.json — fused single-node dev / small deployments

All three carry Serilog log-level overrides + Security:Ldap:DevStubMode = false + Security:Ldap:Transport = Ldaps (audit table: Configuration.md § Role overlays). Configuration loading order (lowest to highest precedence) is:

appsettings.json < appsettings.{Environment}.json < appsettings.{role}.json < environment variables < command-line args

The role overlay intentionally outranks appsettings.{Environment}.json so role-level security defaults (such as DevStubMode = false) cannot be silently overridden by a developer's local appsettings.Development.json; environment variables and command-line args still outrank everything for deployment-level overrides. Overlays are optional; the base file boots a node on its own.

Akka cluster

The host joins an Akka.NET cluster bound to the address in appsettings.json::Cluster:

{
  "Cluster": {
    "Hostname": "0.0.0.0",
    "Port": 4053,
    "PublicHostname": "node-a.lan",
    "SeedNodes": ["akka.tcp://otopcua@node-a.lan:4053"],
    "Roles": ["admin", "driver"]
  }
}

• SeedNodes is ordered: a node listed in its own seed list must be entry 0, since Akka only lets seed-nodes[0] form a new cluster. AkkaClusterOptionsValidator fails startup otherwise — see Redundancy.md § Bootstrap: self-first seed ordering.
• WithOtOpcUaClusterBootstrap (in OtOpcUa.Cluster) loads the embedded HOCON (split-brain resolver, pinned dispatcher, failure detector tuning) and overlays remote endpoint + cluster options.
• All cluster singletons + per-node actors live on this single ActorSystem — there is no second Akka instance.

See Redundancy.md for the role-leader + ServiceLevel story.

Health endpoints

Both admin and driver nodes expose:

Path	Status meaning
/healthz	Process alive.
/health/ready	ConfigDb reachable + cluster member state is Up.
/health/active	Admin-role leader (the node Traefik or an HA LB should route traffic to).

Used by Traefik for the active-leader-only routing pattern (see Architecture-v2.md).

Historian backend (HistorianGateway — external)

The historian backend is the external ZB.MOM.WW.HistorianGateway, deployed and operated separately (not installed by Install-Services.ps1). OtOpcUa connects to it over gRPC via the ZB.MOM.WW.HistorianGateway.Client package — configure the ServerHistorian:Endpoint (https://host:5222) and supply ServerHistorian__ApiKey via the environment on the OtOpcUa host side. The gateway must run with RuntimeDb:Enabled=true + RuntimeDb:EventReadsEnabled=true and an API key carrying historian:read + historian:write + historian:tags:write. See Historian.md for the full config-key and deployment-prerequisite reference. (The retired Wonderware TCP sidecar: Historian.Wonderware.md.)

Script worker processes (Scripting:Runner=OutOfProcess, lmxopcua#780)

Off by default. When enabled, a driver node launches up to Scripting:Worker:Shards + 1 child processes (ZB.MOM.WW.OtOpcUa.ScriptWorker, from script-worker/ beside the host) on its first compiled script evaluation and kills them with it: a Job Object with kill-on-close on Windows, pipe end of stream plus a parent-pid poll elsewhere, and a Shutdown frame then a kill on host stop. Their stderr lands in the host log prefixed ScriptWorker[<shard>]. At startup the host kills workers a dead predecessor of the same install left behind, matched by launch marker (pid + start time + image, and only when the marker's host is gone). Scripting:Worker:PeCacheDir, when set, holds the compiled-PE cache, its HMAC key and the launch markers: put it outside the install folder (startup refuses a relative path or one inside the content root) on a directory only the service account can read. The script-runner readiness component is at worst Degraded. A self-contained host publish (dotnet publish … -r win-x64 --self-contained, the Windows installs, which have no .NET runtime) ships a self-contained worker in script-worker/ (~97 MB: it carries its own runtime) and the host starts its ZB.MOM.WW.OtOpcUa.ScriptWorker.exe apphost directly; a framework-dependent host (docker-dev) runs the worker's dll through the dotnet muxer. See docs/VirtualTags.md §"Out-of-process runner".

Worker isolation on Windows (Install-ScriptWorkerIsolation.ps1, design §14 #5/#6)

scripts/install/Install-ScriptWorkerIsolation.ps1 -InstallRoot <dir> -HostServiceAccount <account> [-ConfigureHostService] (idempotent; -WhatIf prints every change; -Uninstall rolls back; -PlanOnly prints the plan on any OS) sets up:

Step	Change
Worker identity	A disabled anchor service OtOpcUaScriptWorker running as NT SERVICE\OtOpcUaScriptWorker, so the virtual account exists. It is never started.
Scripting:Worker:Identity	With NT SERVICE\OtOpcUaScriptWorker set (by -ConfigureHostService, or in appsettings), the host starts every worker under a restricted token derived from its own: every privilege but SeChangeNotifyPrivilege removed, and restricting SIDs = the identity + Everyone + BUILTIN\Users + RESTRICTED. Every access the worker makes must pass for the host account and for that set. The token's default DACL is host + SYSTEM + identity; the worker is created suspended and put in its Job Object before it runs; TEMP/TMP point at its per-launch working directory (ACL: host + SYSTEM full, identity Modify); DOTNET_EnableDiagnostics=0. The host also adds the identity to its own window station and desktop DACL (enumerate/read, and create-window on the desktop), once per process. USER32 connects to them in its DllMain and the runtime imports USER32, so without the grant coreclr.dll fails to load (ERROR_DLL_INIT_FAILED, measured on windev). The grant lives on the session's kernel objects and ends with the session; nothing persistent changes. The logon SID, which those objects normally grant, is deliberately not a restricting SID: the host's process DACL gives it PROCESS_VM_READ. Windows cannot start an ad-hoc child process as a virtual account (only the SCM can, for a service), and a separate local user would need a stored password plus SeAssignPrimaryTokenPrivilege on the host account; a restricted copy of the host's own token needs neither. Startup refuses the key off Windows or when malformed.
Pipe ACL	On Windows the worker pipe always carries an explicit, protected DACL: the host account and (with an identity) the worker identity, read/write, nobody else. No SYSTEM, no Administrators, no Everyone. The owner is the host token's owner, so the worker's CurrentUserOnly check of the server end still holds.
PeCacheDir	Created with a protected ACL: SYSTEM, Administrators and the host account full control, no Users. pe\ grants the identity Modify (workers compile and write entries). The HMAC key and workers\ markers are host-only.
Install folder	The identity is denied ReadData on <InstallRoot> (inherited), so the worker cannot read appsettings*.json, logs or host binaries even where Users can. script-worker\ grants it read + execute.
Firewall	Outbound block rule OtOpcUa-ScriptWorker-BlockOutbound on script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.exe, every profile. It covers the self-contained apphost the Windows installs run. A framework-dependent worker runs under dotnet.exe and is not covered, so there the analyzer's System.Net ban is the only network control.

The plan is pure data asserted by scripts/install/tests/Test-ScriptWorkerIsolation.ps1 (pwsh -File …, any OS). Re-run the installer after a deploy that replaces the install folder: a rename/swap drops the folder ACEs, but files copied into the existing folder inherit them. Rollback: -Uninstall -ConfigureHostService, then restart the host.

Install / Uninstall

• scripts/install/Install-Services.ps1 -Roles admin,driver — installs OtOpcUaHost.
• scripts/install/Install-ScriptWorkerIsolation.ps1 — the ScriptWorker identity, ACLs and firewall rule (above); only for Scripting:Runner ≠ InProcess.
• scripts/install/Uninstall-Services.ps1 — stops + removes the host service. (The historian backend is the external HistorianGateway — not installed/removed by these scripts.)

Logging

Serilog with rolling-daily file sinks. Each host writes to logs/otopcua-*.log plus stdout (NSSM/systemd-friendly). Per-environment log level overrides go in appsettings.{Environment}.json.

Depth reference

For the full host-architecture rationale (why fused vs. split, role-gating tradeoffs, multi-node deployment shapes), see docs/plans/2026-05-26-akka-hosting-alignment-design.md §3-4.