lmxopcua/docs/ServiceHosting.md

# Service Hosting (v2)

## Overview

A production OtOpcUa deployment runs **one binary per node**. The historian backend is the external
`ZB.MOM.WW.HistorianGateway`, deployed separately (not installed by this repo's scripts):

| Process | Project | Runtime | Platform | Responsibility |
|---|---|---|---|---|
| **OtOpcUa Host** | `src/Server/ZB.MOM.WW.OtOpcUa.Host` | .NET 10 | AnyCPU | Single fused binary. The resolved [`EffectiveRoles`](#role-gating) decides what to mount: `admin` (Blazor + auth + control-plane singletons), `driver` (OPC UA endpoint + per-driver actors), or both. |
| **ZB.MOM.WW.HistorianGateway** *(external — separate deployment)* | not in this repo | .NET 10 | — | The default historian backend (`ServerHistorian:Backend=Gateway`; `SqlServer` needs no sidecar). OtOpcUa talks gRPC to it (via the `ZB.MOM.WW.HistorianGateway.Client` package) for HistoryRead, alarm `SendEvent`, and continuous `WriteLiveValues`. Must run `RuntimeDb:Enabled=true` + `RuntimeDb:EventReadsEnabled=true`; the API key must carry `historian:read` + `historian:write` + `historian:tags:write`. |

Galaxy access still uses the separately-installed **mxaccessgw** sidecar (see `docs/v2/Galaxy.ParityRig.md`); the gateway owns the MXAccess COM bitness constraint (its worker is x86 net48). Nothing in the OtOpcUa repo carries that constraint anymore. (The bespoke Wonderware historian sidecar this deployment used to ship was retired — see [drivers/Historian.Wonderware.md](drivers/Historian.Wonderware.md).)

> **v2 change.** v1's separate `OtOpcUa.Server` + `OtOpcUa.Admin` Windows services merged into a single role-gated `OtOpcUa.Host` binary. Two installers became one (with a `-Roles` parameter). The whole DI graph is composed in `OtOpcUa.Host/Program.cs`; per-role wiring is conditional on the env var.

## Role gating

`Program.cs` resolves `EffectiveRoles.Resolve(builder.Configuration)` (`ZB.MOM.WW.OtOpcUa.Cluster`) —
the single role authority (archreview R3-04): `Cluster:Roles` when non-empty, else `OTOPCUA_ROLES`,
else neither role. The SAME resolved instance drives both the conditional wiring below AND the Akka
gossiped roles (`AkkaClusterOptions.Roles`), so the two cannot disagree — and conditionally registers
services:

| Role present | Wires |
|---|---|
| `admin` | `AddOtOpcUaAuth`, `AddAdminUI`, `AddSignalR`, `AddOtOpcUaAdminClients`, `MapOtOpcUaAuth`, `MapAdminUI<App>`, `MapOtOpcUaHubs`, `WithOtOpcUaControlPlaneSingletons` (5 admin singletons via `Akka.Hosting`) |
| `driver` | `WithOtOpcUaRuntimeActors` (DriverHostActor + DbHealthProbeActor) — and the OPC UA endpoint on port 4840 |
| Either / both | `AddOtOpcUaConfigDb`, `AddOtOpcUaCluster`, `AddOtOpcUaHealth` (`/health/ready`, `/health/active`, `/healthz`) |

Single-node dev: `OTOPCUA_ROLES=admin,driver` (or the `Cluster:Roles` equivalent). Production: typically two admin nodes (HA pair) + N driver nodes.

### Per-role configuration overlays

`Program.cs` builds a role suffix (`RoleOverlay.Suffix`) by joining the resolved `EffectiveRoles.Gossiped` **plain** roles (`admin`/`driver`/`dev`) alphabetically with `-` and loads `appsettings.{roleSuffix}.json` as an optional overlay on top of base `appsettings.json`. A plain role is always lower-case in `Gossiped` regardless of how the operator spelled it, so the overlay filename is stable (`appsettings.driver.json`). **`cluster-{ClusterId}` roles are NOT part of the filename** (#678.6): they are mesh-membership tags, so `Cluster:Roles: ["driver", "cluster-SITE-A"]` loads `appsettings.driver.json` and `["admin", "driver", "cluster-MAIN"]` loads `appsettings.admin-driver.json`. (Before #678.6 the cluster role was folded into the name, producing e.g. `appsettings.admin-cluster-MAIN-driver.json`, which never exists — so on every node with a cluster role no shipped overlay ever loaded.) Three overlays ship in `src/Server/ZB.MOM.WW.OtOpcUa.Host/`:

- `appsettings.admin.json` — admin-only nodes
- `appsettings.driver.json` — driver-only nodes
- `appsettings.admin-driver.json` — fused single-node dev / small deployments

All three carry Serilog log-level overrides + `Security:Ldap:DevStubMode = false` + `Security:Ldap:Transport = Ldaps` (audit table: [Configuration.md § Role overlays](Configuration.md#role-overlay-audit)). Configuration loading order (lowest to highest precedence) is:

`appsettings.json` < `appsettings.{Environment}.json` < `appsettings.{role}.json` < environment variables < command-line args

The role overlay intentionally outranks `appsettings.{Environment}.json` so role-level security defaults (such as `DevStubMode = false`) cannot be silently overridden by a developer's local `appsettings.Development.json`; environment variables and command-line args still outrank everything for deployment-level overrides. Overlays are optional; the base file boots a node on its own.

## Akka cluster

The host joins an Akka.NET cluster bound to the address in `appsettings.json::Cluster`:

```json
{
  "Cluster": {
    "Hostname": "0.0.0.0",
    "Port": 4053,
    "PublicHostname": "node-a.lan",
    "SeedNodes": ["akka.tcp://otopcua@node-a.lan:4053"],
    "Roles": ["admin", "driver"]
  }
}
```

- `SeedNodes` is **ordered**: a node listed in its own seed list must be entry 0, since Akka only lets `seed-nodes[0]` form a new cluster. `AkkaClusterOptionsValidator` fails startup otherwise — see [Redundancy.md § Bootstrap: self-first seed ordering](Redundancy.md#bootstrap-self-first-seed-ordering).
- `WithOtOpcUaClusterBootstrap` (in `OtOpcUa.Cluster`) loads the embedded HOCON (split-brain resolver, pinned dispatcher, failure detector tuning) and overlays remote endpoint + cluster options.
- All cluster singletons + per-node actors live on this single ActorSystem — there is no second Akka instance.

See [Redundancy.md](Redundancy.md) for the role-leader + ServiceLevel story.

## Health endpoints

Both admin and driver nodes expose:

| Path | Status meaning |
|---|---|
| `/healthz` | Process alive. |
| `/health/ready` | ConfigDb reachable + cluster member state is `Up`. |
| `/health/active` | Admin-role leader (the node Traefik or an HA LB should route traffic to). |

Used by Traefik for the active-leader-only routing pattern (see [Architecture-v2.md](v2/Architecture-v2.md)).

## Historian backend (HistorianGateway — external)

The historian backend is the external `ZB.MOM.WW.HistorianGateway`, deployed and operated separately (not
installed by `Install-Services.ps1`). OtOpcUa connects to it over gRPC via the
`ZB.MOM.WW.HistorianGateway.Client` package — configure the `ServerHistorian:Endpoint` (`https://host:5222`)
and supply `ServerHistorian__ApiKey` via the environment on the OtOpcUa host side. The gateway must run with
`RuntimeDb:Enabled=true` + `RuntimeDb:EventReadsEnabled=true` and an API key carrying `historian:read` +
`historian:write` + `historian:tags:write`. See [Historian.md](Historian.md) for the full config-key and
deployment-prerequisite reference. (The retired Wonderware TCP sidecar: [Historian.Wonderware.md](drivers/Historian.Wonderware.md).)

## Script worker processes (`Scripting:Runner=OutOfProcess`, lmxopcua#780)

Off by default. When enabled, a driver node launches up to `Scripting:Worker:Shards` + 1 child processes
(`ZB.MOM.WW.OtOpcUa.ScriptWorker`, from `script-worker/` beside the host) on its first compiled script evaluation and
kills them with it: a Job Object with kill-on-close on Windows, pipe end of stream plus a parent-pid poll elsewhere,
and a Shutdown frame then a kill on host stop. Their stderr lands in the host log prefixed `ScriptWorker[<shard>]`.
At startup the host kills workers a dead predecessor of the same install left behind, matched by launch marker
(pid + start time + image, and only when the marker's host is gone). `Scripting:Worker:PeCacheDir`, when set, holds the
compiled-PE cache, its HMAC key and the launch markers: put it outside the install folder (startup refuses a relative path
or one inside the content root) on a directory only the service account can read. The `script-runner` readiness component
is at worst Degraded. A self-contained host publish (`dotnet publish … -r win-x64 --self-contained`, the Windows installs, which have no
.NET runtime) ships a **self-contained** worker in `script-worker/` (~97 MB: it carries its own runtime) and the host starts its
`ZB.MOM.WW.OtOpcUa.ScriptWorker.exe` apphost directly; a framework-dependent host (docker-dev) runs the worker's dll through the
dotnet muxer. See `docs/VirtualTags.md` §"Out-of-process runner".

### Worker isolation on Windows (`Install-ScriptWorkerIsolation.ps1`, design §14 #5/#6)

`scripts/install/Install-ScriptWorkerIsolation.ps1 -InstallRoot <dir> -HostServiceAccount <account> [-DenyPaths <dir>,…] [-ConfigureHostService]`
(idempotent; `-WhatIf` prints every change; `-Uninstall` rolls back; `-PlanOnly` prints the plan on any OS) sets up:

| Step | Change |
|---|---|
| Worker identity | A **disabled** anchor service `OtOpcUaScriptWorker` running as `NT SERVICE\OtOpcUaScriptWorker`, so the virtual account exists. It is never started. |
| `Scripting:Worker:Identity` | With `NT SERVICE\OtOpcUaScriptWorker` set (by `-ConfigureHostService`, or in appsettings), the host starts every worker under a **restricted token** derived from its own: every privilege but `SeChangeNotifyPrivilege` removed, and restricting SIDs = the identity + `Everyone` + `BUILTIN\Users` + `RESTRICTED`. Every access the worker makes must pass for the host account **and** for that set. The token's default DACL is host + SYSTEM + identity; the worker is created suspended and put in its Job Object before it runs; `TEMP`/`TMP` point at its per-launch working directory (ACL: host + SYSTEM full, identity Modify); `DOTNET_EnableDiagnostics=0`. The host also adds the identity to its own window station and desktop DACL (enumerate/read, and create-window on the desktop), once per process. USER32 connects to them in its DllMain and the runtime imports USER32, so without the grant `coreclr.dll` fails to load (`ERROR_DLL_INIT_FAILED`, measured on windev). The grant lives on the session's kernel objects and ends with the session; nothing persistent changes. The logon SID, which those objects normally grant, is deliberately not a restricting SID: the host's process DACL gives it `PROCESS_VM_READ`. Windows cannot start an ad-hoc child process *as* a virtual account (only the SCM can, for a service), and a separate local user would need a stored password plus `SeAssignPrimaryTokenPrivilege` on the host account; a restricted copy of the host's own token needs neither. Startup refuses the key off Windows or when malformed. |
| Pipe ACL | On Windows the worker pipe always carries an explicit, protected DACL: the host account and (with an identity) the worker identity, read/write, nobody else. No SYSTEM, no Administrators, no Everyone. The owner is the host token's owner, so the worker's `CurrentUserOnly` check of the server end still holds. |
| `PeCacheDir` | Created with a protected ACL: SYSTEM, Administrators and the host account full control, no Users. `pe\` grants the identity Modify (workers compile and write entries). The HMAC key and `workers\` markers are host-only. |
| Install folder | The identity is **denied** read and write on `<InstallRoot>` (inherited: ReadData/List, WriteData/AddFile, AppendData/AddSubdirectory, write attributes/EA, Delete, DeleteChild, ChangePermissions, TakeOwnership; Traverse/Execute, ReadAttributes and ReadPermissions stay so the loader reaches `script-worker\`). The worker cannot read `appsettings*.json`, logs or host binaries even where `Users` can. `script-worker\` grants it read + execute. |
| Sensitive folders (`-DenyPaths`, lmxopcua#890) | The same inherited deny on every listed folder. **Name every folder outside `<InstallRoot>` that holds keys, secrets, stores, logs or backups**: the worker's restricting SIDs include `Everyone` and `BUILTIN\Users`, so anything Users can read is otherwise readable to it (and a data drive usually lets Users create files in subfolders, e.g. a certificate in `pki\trusted\certs`). Name the deployment folder around `<InstallRoot>` (one ACE covers `pki\`, `data\`, `logs\`, `backup\`, old `app.bak.*`) **and** `pki\` and `data\` themselves, so the deny holds even on a folder whose DACL does not inherit. Drive and share roots, `script-worker\` and the PE cache are refused; each folder must exist. The list is recorded under `HKLM\…\Services\OtOpcUaScriptWorker\Parameters` (`DenyPaths`): a re-run removes the deny from folders dropped from the list, and `-Uninstall` removes every recorded one without repeating `-DenyPaths`. After an apply the installer WARNs about any item under a denied folder whose DACL is protected and has no deny of its own (the inherited deny does not reach it); without `-DenyPaths` it WARNs about `pki`/`data`/`logs`/`backup`/`config`/`secrets` siblings of `<InstallRoot>`. Wonder layout: `-InstallRoot E:\ApiInstall\OtOpcUa\app -DenyPaths E:\ApiInstall\OtOpcUa,E:\ApiInstall\OtOpcUa\pki,E:\ApiInstall\OtOpcUa\data,C:\ProgramData\ZB\kek` plus, box-wide (lmxopcua#891), every other app folder under `E:\ApiInstall` and the staging folders `E:\psrem-stage`, `E:\psrem`; keep deploy backups (`.reg` exports, appsettings copies) in a folder with a protected SYSTEM + Administrators ACL. |
| Firewall | Outbound **block** rule `OtOpcUa-ScriptWorker-BlockOutbound` on `script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.exe`, every profile. It covers the self-contained apphost the Windows installs run. A framework-dependent worker runs under `dotnet.exe` and is not covered, so there the analyzer's `System.Net` ban is the only network control. |

The plan is pure data asserted by `scripts/install/tests/Test-ScriptWorkerIsolation.ps1` (`pwsh -File …`, any OS).
`Test-ScriptWorkerIsolation.Windows.ps1` applies it for real in a scratch folder (elevated), and
`InstallerLayoutIsolationTests` (ScriptWorker.Tests, `OTOPCUA_ISOLATION_INSTALLER_E2E=1`, elevated) proves under the real
restricted token that a worker cannot read `pki\`/`data\`/`logs\`/`backup\` of a scratch wonder layout and still runs.
Rollback: `-Uninstall -ConfigureHostService`, then restart the host (`-Uninstall` alone also works from the record).

#### After an upgrade that swaps the install folder (lmxopcua#901)

A rename/swap drops the explicit ACEs on `script-worker\`, on the install folder and on any denied folder that is
recreated or copied in (vd03's `pki\`): a moved-in folder carries its own DACL. Files copied into an existing folder
inherit the ACEs. The installer therefore records its effective settings (`InstallRoot`, `HostServiceAccount`,
`PeCacheDir`, `HostServiceName`, `ConfigureHostService`, beside `DenyPaths`) under
`HKLM\SYSTEM\CurrentControlSet\Services\OtOpcUaScriptWorker\Parameters`, and sets that key's DACL protected, SYSTEM +
Administrators only (no secrets are recorded; Users, a restricting SID of the worker token, can no longer read the list).
Two argument-free modes use the record:

| Command | Does |
|---|---|
| `Install-ScriptWorkerIsolation.ps1 -Reapply` | Re-applies with the recorded settings; one `[changed]`/`[unchanged]` line per step and a `summary:` line. After a swap: the three folder ACEs `changed`, the rest `unchanged`; a second run all `unchanged`. |
| `Install-ScriptWorkerIsolation.ps1 -Verify` | Check only. `[ok]`/`[MISSING]` per step (anchor service, record + key ACL, PE cache ACL, `pe\`, `script-worker\`, install-folder deny, every `-DenyPaths` deny, firewall rule and the worker exe it names, the host env entries when recorded). **Exit 1** with `VERIFY FAILED: …` when anything is missing, exit 0 with `VERIFY OK`. Protected-DACL coverage gaps are WARNed, not failed. |

Both refuse to run without a full record. A node isolated by an installer older than #901 recorded only `DenyPaths`:
run the install once with its full arguments (the per-box deploy record lists them) to write the record.

`scripts/install/Invoke-OtOpcUaInstallSwap.ps1` is the swap with the gate built in: preflight (candidate carries
`script-worker\`, `PreviousInstall` is fresh and on the same volume, isolation record present when isolation is
installed, record `InstallRoot` equals `-InstallRoot`) → stop → move live → previous, candidate → live → copy `-Carry`
items back (default `appsettings*.json`, `pki`, `logs`; QA: `'appsettings*.json','logs'`; rollback: `-NoCarry`) →
`-Reapply` → `-Verify` → start (`-NoStart` for a pair window). A failed `-Verify` leaves the service **stopped**. Backups,
`data\` restore, migrations and health checks stay in the deploy's own steps. Every deploy pack that swaps the folder runs
it (or `-Reapply` + `-Verify` right after its own swap), and `-Verify` is a post-deploy gate.
`Test-ScriptWorkerIsolation.Windows.ps1` exercises the whole sequence (swap → `-Verify` fails → `-Reapply` → `-Verify`
passes → `-Reapply` unchanged → the swap script) on a scratch tree.

## Install / Uninstall

- `scripts/install/Install-Services.ps1 -Roles admin,driver` — installs `OtOpcUaHost`.
- `scripts/install/Install-ScriptWorkerIsolation.ps1` — the ScriptWorker identity, ACLs and firewall rule (above); only for `Scripting:Runner` ≠ `InProcess`.
- `scripts/install/Invoke-OtOpcUaInstallSwap.ps1` — upgrade/rollback folder swap with the isolation `-Reapply` + `-Verify` gate built in (lmxopcua#901, above).
- `scripts/install/Uninstall-Services.ps1` — stops + removes the host service. (The historian backend is the external HistorianGateway — not installed/removed by these scripts.)

## Logging

Serilog with rolling-daily file sinks. Each host writes to `logs/otopcua-*.log` plus stdout (NSSM/systemd-friendly). Per-environment log level overrides go in `appsettings.{Environment}.json`.

## Depth reference

For the full host-architecture rationale (why fused vs. split, role-gating tradeoffs, multi-node deployment shapes), see `docs/plans/2026-05-26-akka-hosting-alignment-design.md` §3-4.