lmxopcua/docs/ServiceHosting.md

# Service Hosting (v2)

## Overview

A production OtOpcUa deployment runs **one binary per node**. The historian backend is the external
`ZB.MOM.WW.HistorianGateway`, deployed separately (not installed by this repo's scripts):

| Process | Project | Runtime | Platform | Responsibility |
|---|---|---|---|---|
| **OtOpcUa Host** | `src/Server/ZB.MOM.WW.OtOpcUa.Host` | .NET 10 | AnyCPU | Single fused binary. The resolved [`EffectiveRoles`](#role-gating) decides what to mount: `admin` (Blazor + auth + control-plane singletons), `driver` (OPC UA endpoint + per-driver actors), or both. |
| **ZB.MOM.WW.HistorianGateway** *(external — separate deployment)* | not in this repo | .NET 10 | — | The default historian backend (`ServerHistorian:Backend=Gateway`; `SqlServer` needs no sidecar). OtOpcUa talks gRPC to it (via the `ZB.MOM.WW.HistorianGateway.Client` package) for HistoryRead, alarm `SendEvent`, and continuous `WriteLiveValues`. Must run `RuntimeDb:Enabled=true` + `RuntimeDb:EventReadsEnabled=true`; the API key must carry `historian:read` + `historian:write` + `historian:tags:write`. |

Galaxy access still uses the separately-installed **mxaccessgw** sidecar (see `docs/v2/Galaxy.ParityRig.md`); the gateway owns the MXAccess COM bitness constraint (its worker is x86 net48). Nothing in the OtOpcUa repo carries that constraint anymore. (The bespoke Wonderware historian sidecar this deployment used to ship was retired — see [drivers/Historian.Wonderware.md](drivers/Historian.Wonderware.md).)

> **v2 change.** v1's separate `OtOpcUa.Server` + `OtOpcUa.Admin` Windows services merged into a single role-gated `OtOpcUa.Host` binary. Two installers became one (with a `-Roles` parameter). The whole DI graph is composed in `OtOpcUa.Host/Program.cs`; per-role wiring is conditional on the env var.

## Role gating

`Program.cs` resolves `EffectiveRoles.Resolve(builder.Configuration)` (`ZB.MOM.WW.OtOpcUa.Cluster`) —
the single role authority (archreview R3-04): `Cluster:Roles` when non-empty, else `OTOPCUA_ROLES`,
else neither role. The SAME resolved instance drives both the conditional wiring below AND the Akka
gossiped roles (`AkkaClusterOptions.Roles`), so the two cannot disagree — and conditionally registers
services:

| Role present | Wires |
|---|---|
| `admin` | `AddOtOpcUaAuth`, `AddAdminUI`, `AddSignalR`, `AddOtOpcUaAdminClients`, `MapOtOpcUaAuth`, `MapAdminUI<App>`, `MapOtOpcUaHubs`, `WithOtOpcUaControlPlaneSingletons` (5 admin singletons via `Akka.Hosting`) |
| `driver` | `WithOtOpcUaRuntimeActors` (DriverHostActor + DbHealthProbeActor) — and the OPC UA endpoint on port 4840 |
| Either / both | `AddOtOpcUaConfigDb`, `AddOtOpcUaCluster`, `AddOtOpcUaHealth` (`/health/ready`, `/health/active`, `/healthz`) |

Single-node dev: `OTOPCUA_ROLES=admin,driver` (or the `Cluster:Roles` equivalent). Production: typically two admin nodes (HA pair) + N driver nodes.

### Per-role configuration overlays

`Program.cs` builds a role suffix (`RoleOverlay.Suffix`) by joining the resolved `EffectiveRoles.Gossiped` **plain** roles (`admin`/`driver`/`dev`) alphabetically with `-` and loads `appsettings.{roleSuffix}.json` as an optional overlay on top of base `appsettings.json`. A plain role is always lower-case in `Gossiped` regardless of how the operator spelled it, so the overlay filename is stable (`appsettings.driver.json`). **`cluster-{ClusterId}` roles are NOT part of the filename** (#678.6): they are mesh-membership tags, so `Cluster:Roles: ["driver", "cluster-SITE-A"]` loads `appsettings.driver.json` and `["admin", "driver", "cluster-MAIN"]` loads `appsettings.admin-driver.json`. (Before #678.6 the cluster role was folded into the name, producing e.g. `appsettings.admin-cluster-MAIN-driver.json`, which never exists — so on every node with a cluster role no shipped overlay ever loaded.) Three overlays ship in `src/Server/ZB.MOM.WW.OtOpcUa.Host/`:

- `appsettings.admin.json` — admin-only nodes
- `appsettings.driver.json` — driver-only nodes
- `appsettings.admin-driver.json` — fused single-node dev / small deployments

All three carry Serilog log-level overrides + `Security:Ldap:DevStubMode = false` + `Security:Ldap:Transport = Ldaps` (audit table: [Configuration.md § Role overlays](Configuration.md#role-overlay-audit)). Configuration loading order (lowest to highest precedence) is:

`appsettings.json` < `appsettings.{Environment}.json` < `appsettings.{role}.json` < environment variables < command-line args

The role overlay intentionally outranks `appsettings.{Environment}.json` so role-level security defaults (such as `DevStubMode = false`) cannot be silently overridden by a developer's local `appsettings.Development.json`; environment variables and command-line args still outrank everything for deployment-level overrides. Overlays are optional; the base file boots a node on its own.

## Akka cluster

The host joins an Akka.NET cluster bound to the address in `appsettings.json::Cluster`:

```json
{
  "Cluster": {
    "Hostname": "0.0.0.0",
    "Port": 4053,
    "PublicHostname": "node-a.lan",
    "SeedNodes": ["akka.tcp://otopcua@node-a.lan:4053"],
    "Roles": ["admin", "driver"]
  }
}
```

- `SeedNodes` is **ordered**: a node listed in its own seed list must be entry 0, since Akka only lets `seed-nodes[0]` form a new cluster. `AkkaClusterOptionsValidator` fails startup otherwise — see [Redundancy.md § Bootstrap: self-first seed ordering](Redundancy.md#bootstrap-self-first-seed-ordering).
- `WithOtOpcUaClusterBootstrap` (in `OtOpcUa.Cluster`) loads the embedded HOCON (split-brain resolver, pinned dispatcher, failure detector tuning) and overlays remote endpoint + cluster options.
- All cluster singletons + per-node actors live on this single ActorSystem — there is no second Akka instance.

See [Redundancy.md](Redundancy.md) for the role-leader + ServiceLevel story.

## Health endpoints

Both admin and driver nodes expose:

| Path | Status meaning |
|---|---|
| `/healthz` | Process alive. |
| `/health/ready` | ConfigDb reachable + cluster member state is `Up`. |
| `/health/active` | Admin-role leader (the node Traefik or an HA LB should route traffic to). |

Used by Traefik for the active-leader-only routing pattern (see [Architecture-v2.md](v2/Architecture-v2.md)).

## Historian backend (HistorianGateway — external)

The historian backend is the external `ZB.MOM.WW.HistorianGateway`, deployed and operated separately (not
installed by `Install-Services.ps1`). OtOpcUa connects to it over gRPC via the
`ZB.MOM.WW.HistorianGateway.Client` package — configure the `ServerHistorian:Endpoint` (`https://host:5222`)
and supply `ServerHistorian__ApiKey` via the environment on the OtOpcUa host side. The gateway must run with
`RuntimeDb:Enabled=true` + `RuntimeDb:EventReadsEnabled=true` and an API key carrying `historian:read` +
`historian:write` + `historian:tags:write`. See [Historian.md](Historian.md) for the full config-key and
deployment-prerequisite reference. (The retired Wonderware TCP sidecar: [Historian.Wonderware.md](drivers/Historian.Wonderware.md).)

## Script worker processes (`Scripting:Runner=OutOfProcess`, lmxopcua#780)

Off by default. When enabled, a driver node launches up to `Scripting:Worker:Shards` + 1 child processes
(`ZB.MOM.WW.OtOpcUa.ScriptWorker`, from `script-worker/` beside the host) on its first compiled script evaluation and
kills them with it: a Job Object with kill-on-close on Windows, pipe end of stream plus a parent-pid poll elsewhere,
and a Shutdown frame then a kill on host stop. Their stderr lands in the host log prefixed `ScriptWorker[<shard>]`.
At startup the host kills workers a dead predecessor of the same install left behind, matched by launch marker
(pid + start time + image, and only when the marker's host is gone). `Scripting:Worker:PeCacheDir`, when set, holds the
compiled-PE cache, its HMAC key and the launch markers: put it outside the install folder (startup refuses a relative path
or one inside the content root) on a directory only the service account can read. The `script-runner` readiness component
is at worst Degraded. A self-contained host publish (`dotnet publish … -r win-x64 --self-contained`, the Windows installs, which have no
.NET runtime) ships a **self-contained** worker in `script-worker/` (~97 MB: it carries its own runtime) and the host starts its
`ZB.MOM.WW.OtOpcUa.ScriptWorker.exe` apphost directly; a framework-dependent host (docker-dev) runs the worker's dll through the
dotnet muxer. See `docs/VirtualTags.md` §"Out-of-process runner".

### Worker isolation on Windows (`Install-ScriptWorkerIsolation.ps1`, design §14 #5/#6)

`scripts/install/Install-ScriptWorkerIsolation.ps1 -InstallRoot <dir> -HostServiceAccount <account> [-ConfigureHostService]`
(idempotent; `-WhatIf` prints every change; `-Uninstall` rolls back; `-PlanOnly` prints the plan on any OS) sets up:

| Step | Change |
|---|---|
| Worker identity | A **disabled** anchor service `OtOpcUaScriptWorker` running as `NT SERVICE\OtOpcUaScriptWorker`, so the virtual account exists. It is never started. |
| `Scripting:Worker:Identity` | With `NT SERVICE\OtOpcUaScriptWorker` set (by `-ConfigureHostService`, or in appsettings), the host starts every worker under a **restricted token** derived from its own: every privilege but `SeChangeNotifyPrivilege` removed, and restricting SIDs = the identity + `Everyone` + `BUILTIN\Users` + `RESTRICTED`. Every access the worker makes must pass for the host account **and** for that set. The token's default DACL is host + SYSTEM + identity; the worker is created suspended and put in its Job Object before it runs; `TEMP`/`TMP` point at its per-launch working directory (ACL: host + SYSTEM full, identity Modify); `DOTNET_EnableDiagnostics=0`. The host also adds the identity to its own window station and desktop DACL (enumerate/read, and create-window on the desktop), once per process. USER32 connects to them in its DllMain and the runtime imports USER32, so without the grant `coreclr.dll` fails to load (`ERROR_DLL_INIT_FAILED`, measured on windev). The grant lives on the session's kernel objects and ends with the session; nothing persistent changes. The logon SID, which those objects normally grant, is deliberately not a restricting SID: the host's process DACL gives it `PROCESS_VM_READ`. Windows cannot start an ad-hoc child process *as* a virtual account (only the SCM can, for a service), and a separate local user would need a stored password plus `SeAssignPrimaryTokenPrivilege` on the host account; a restricted copy of the host's own token needs neither. Startup refuses the key off Windows or when malformed. |
| Pipe ACL | On Windows the worker pipe always carries an explicit, protected DACL: the host account and (with an identity) the worker identity, read/write, nobody else. No SYSTEM, no Administrators, no Everyone. The owner is the host token's owner, so the worker's `CurrentUserOnly` check of the server end still holds. |
| `PeCacheDir` | Created with a protected ACL: SYSTEM, Administrators and the host account full control, no Users. `pe\` grants the identity Modify (workers compile and write entries). The HMAC key and `workers\` markers are host-only. |
| Install folder | The identity is **denied** ReadData on `<InstallRoot>` (inherited), so the worker cannot read `appsettings*.json`, logs or host binaries even where `Users` can. `script-worker\` grants it read + execute. |
| Firewall | Outbound **block** rule `OtOpcUa-ScriptWorker-BlockOutbound` on `script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.exe`, every profile. It covers the self-contained apphost the Windows installs run. A framework-dependent worker runs under `dotnet.exe` and is not covered, so there the analyzer's `System.Net` ban is the only network control. |

The plan is pure data asserted by `scripts/install/tests/Test-ScriptWorkerIsolation.ps1` (`pwsh -File …`, any OS).
Re-run the installer after a deploy that replaces the install folder: a rename/swap drops the folder ACEs, but files
copied into the existing folder inherit them. Rollback: `-Uninstall -ConfigureHostService`, then restart the host.

## Install / Uninstall

- `scripts/install/Install-Services.ps1 -Roles admin,driver` — installs `OtOpcUaHost`.
- `scripts/install/Install-ScriptWorkerIsolation.ps1` — the ScriptWorker identity, ACLs and firewall rule (above); only for `Scripting:Runner` ≠ `InProcess`.
- `scripts/install/Uninstall-Services.ps1` — stops + removes the host service. (The historian backend is the external HistorianGateway — not installed/removed by these scripts.)

## Logging

Serilog with rolling-daily file sinks. Each host writes to `logs/otopcua-*.log` plus stdout (NSSM/systemd-friendly). Per-environment log level overrides go in `appsettings.{Environment}.json`.

## Depth reference

For the full host-architecture rationale (why fused vs. split, role-gating tradeoffs, multi-node deployment shapes), see `docs/plans/2026-05-26-akka-hosting-alignment-design.md` §3-4.