Joseph Doherty
f75ee7d089
v2-ci / unit-tests (push) Waiting to run
The installer denied NT SERVICE\OtOpcUaScriptWorker only on -InstallRoot. On the wonder layout the node's keys and secrets sit beside it (E:\ApiInstall\OtOpcUa\pki with the application certificate's private key, data\ with secrets.db and LocalDb), inheriting BUILTIN\Users:RX. Users is a restricting SID of the worker token, so a worker could read them. - Install-ScriptWorkerIsolation.ps1 -DenyPaths <dir>,...: each folder gets the same inherited deny as InstallRoot (new SensitiveDirDeny steps). Refuses drive/share roots, script-worker\ and the PE cache; folders must exist; a comma/semicolon string works for powershell -File. - The deny is read AND write (0xD0157: ReadData, WriteData, AppendData, WriteEA, DeleteChild, WriteAttributes, Delete, WRITE_DAC, WRITE_OWNER). Traverse/Execute, ReadAttributes, ReadControl and Synchronize stay so the loader still reaches script-worker\. The write half stops a worker planting a cert in pki\trusted\certs where a data drive lets Users create files in subfolders. - The list is recorded under the anchor service's key (Parameters\DenyPaths, step DenyPathRecord): a re-run removes the deny from folders dropped from the list, -Uninstall removes every recorded one without repeating -DenyPaths. -WhatIf/-PlanOnly print every step. - After an apply, WARN for each protected-DACL item under a denied folder that the inherited deny cannot reach; without -DenyPaths, WARN for pki/data/logs/backup/config/secrets siblings of InstallRoot. Tests: - Dry-run harness 46 -> 86 assertions (QA layout plan, mask bits, refusals, stale removal, record-driven rollback, host account never denied). Mutations (no sibling deny, deny->allow, mask back to ReadData, no stale removal, rollback ignoring the record, script-worker not refused) each turn it red. - Test-ScriptWorkerIsolation.Windows.ps1 (Windows PowerShell 5.1, windev): 34/34, incl. -DenyPaths apply, 11/11 unchanged re-apply, shorter-list re-apply, uninstall from the record, second uninstall unchanged. - New InstallerLayoutIsolationTests (OTOPCUA_ISOLATION_INSTALLER_E2E=1, elevated Windows): a scratch wonder layout under the real restricted worker token. Before install and after uninstall the token reads pki/data/ logs/backup/appsettings and plants a trust-list file; installed it can do neither, still reads script-worker\, a real worker launched from the layout runs a script, and the host still reads everything. Mutating the -DenyPaths denies to a no-op mask turns it red on pki\own\private\otopcua.pfx (reproduces #890). QA vq03/vq04 are NOT re-applied; the command is in docs/deploy/qa-upgrade-b8b4c18f-to-fea7de0a.md. Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
141 lines
15 KiB
Markdown
141 lines
15 KiB
Markdown
# Service Hosting (v2)
|
|
|
|
## Overview
|
|
|
|
A production OtOpcUa deployment runs **one binary per node**. The historian backend is the external
|
|
`ZB.MOM.WW.HistorianGateway`, deployed separately (not installed by this repo's scripts):
|
|
|
|
| Process | Project | Runtime | Platform | Responsibility |
|
|
|---|---|---|---|---|
|
|
| **OtOpcUa Host** | `src/Server/ZB.MOM.WW.OtOpcUa.Host` | .NET 10 | AnyCPU | Single fused binary. The resolved [`EffectiveRoles`](#role-gating) decides what to mount: `admin` (Blazor + auth + control-plane singletons), `driver` (OPC UA endpoint + per-driver actors), or both. |
|
|
| **ZB.MOM.WW.HistorianGateway** *(external — separate deployment)* | not in this repo | .NET 10 | — | The default historian backend (`ServerHistorian:Backend=Gateway`; `SqlServer` needs no sidecar). OtOpcUa talks gRPC to it (via the `ZB.MOM.WW.HistorianGateway.Client` package) for HistoryRead, alarm `SendEvent`, and continuous `WriteLiveValues`. Must run `RuntimeDb:Enabled=true` + `RuntimeDb:EventReadsEnabled=true`; the API key must carry `historian:read` + `historian:write` + `historian:tags:write`. |
|
|
|
|
Galaxy access still uses the separately-installed **mxaccessgw** sidecar (see `docs/v2/Galaxy.ParityRig.md`); the gateway owns the MXAccess COM bitness constraint (its worker is x86 net48). Nothing in the OtOpcUa repo carries that constraint anymore. (The bespoke Wonderware historian sidecar this deployment used to ship was retired — see [drivers/Historian.Wonderware.md](drivers/Historian.Wonderware.md).)
|
|
|
|
> **v2 change.** v1's separate `OtOpcUa.Server` + `OtOpcUa.Admin` Windows services merged into a single role-gated `OtOpcUa.Host` binary. Two installers became one (with a `-Roles` parameter). The whole DI graph is composed in `OtOpcUa.Host/Program.cs`; per-role wiring is conditional on the env var.
|
|
|
|
## Role gating
|
|
|
|
`Program.cs` resolves `EffectiveRoles.Resolve(builder.Configuration)` (`ZB.MOM.WW.OtOpcUa.Cluster`) —
|
|
the single role authority (archreview R3-04): `Cluster:Roles` when non-empty, else `OTOPCUA_ROLES`,
|
|
else neither role. The SAME resolved instance drives both the conditional wiring below AND the Akka
|
|
gossiped roles (`AkkaClusterOptions.Roles`), so the two cannot disagree — and conditionally registers
|
|
services:
|
|
|
|
| Role present | Wires |
|
|
|---|---|
|
|
| `admin` | `AddOtOpcUaAuth`, `AddAdminUI`, `AddSignalR`, `AddOtOpcUaAdminClients`, `MapOtOpcUaAuth`, `MapAdminUI<App>`, `MapOtOpcUaHubs`, `WithOtOpcUaControlPlaneSingletons` (5 admin singletons via `Akka.Hosting`) |
|
|
| `driver` | `WithOtOpcUaRuntimeActors` (DriverHostActor + DbHealthProbeActor) — and the OPC UA endpoint on port 4840 |
|
|
| Either / both | `AddOtOpcUaConfigDb`, `AddOtOpcUaCluster`, `AddOtOpcUaHealth` (`/health/ready`, `/health/active`, `/healthz`) |
|
|
|
|
Single-node dev: `OTOPCUA_ROLES=admin,driver` (or the `Cluster:Roles` equivalent). Production: typically two admin nodes (HA pair) + N driver nodes.
|
|
|
|
### Per-role configuration overlays
|
|
|
|
`Program.cs` builds a role suffix (`RoleOverlay.Suffix`) by joining the resolved `EffectiveRoles.Gossiped` **plain** roles (`admin`/`driver`/`dev`) alphabetically with `-` and loads `appsettings.{roleSuffix}.json` as an optional overlay on top of base `appsettings.json`. A plain role is always lower-case in `Gossiped` regardless of how the operator spelled it, so the overlay filename is stable (`appsettings.driver.json`). **`cluster-{ClusterId}` roles are NOT part of the filename** (#678.6): they are mesh-membership tags, so `Cluster:Roles: ["driver", "cluster-SITE-A"]` loads `appsettings.driver.json` and `["admin", "driver", "cluster-MAIN"]` loads `appsettings.admin-driver.json`. (Before #678.6 the cluster role was folded into the name, producing e.g. `appsettings.admin-cluster-MAIN-driver.json`, which never exists — so on every node with a cluster role no shipped overlay ever loaded.) Three overlays ship in `src/Server/ZB.MOM.WW.OtOpcUa.Host/`:
|
|
|
|
- `appsettings.admin.json` — admin-only nodes
|
|
- `appsettings.driver.json` — driver-only nodes
|
|
- `appsettings.admin-driver.json` — fused single-node dev / small deployments
|
|
|
|
All three carry Serilog log-level overrides + `Security:Ldap:DevStubMode = false` + `Security:Ldap:Transport = Ldaps` (audit table: [Configuration.md § Role overlays](Configuration.md#role-overlay-audit)). Configuration loading order (lowest to highest precedence) is:
|
|
|
|
`appsettings.json` < `appsettings.{Environment}.json` < `appsettings.{role}.json` < environment variables < command-line args
|
|
|
|
The role overlay intentionally outranks `appsettings.{Environment}.json` so role-level security defaults (such as `DevStubMode = false`) cannot be silently overridden by a developer's local `appsettings.Development.json`; environment variables and command-line args still outrank everything for deployment-level overrides. Overlays are optional; the base file boots a node on its own.
|
|
|
|
## Akka cluster
|
|
|
|
The host joins an Akka.NET cluster bound to the address in `appsettings.json::Cluster`:
|
|
|
|
```json
|
|
{
|
|
"Cluster": {
|
|
"Hostname": "0.0.0.0",
|
|
"Port": 4053,
|
|
"PublicHostname": "node-a.lan",
|
|
"SeedNodes": ["akka.tcp://otopcua@node-a.lan:4053"],
|
|
"Roles": ["admin", "driver"]
|
|
}
|
|
}
|
|
```
|
|
|
|
- `SeedNodes` is **ordered**: a node listed in its own seed list must be entry 0, since Akka only lets `seed-nodes[0]` form a new cluster. `AkkaClusterOptionsValidator` fails startup otherwise — see [Redundancy.md § Bootstrap: self-first seed ordering](Redundancy.md#bootstrap-self-first-seed-ordering).
|
|
- `WithOtOpcUaClusterBootstrap` (in `OtOpcUa.Cluster`) loads the embedded HOCON (split-brain resolver, pinned dispatcher, failure detector tuning) and overlays remote endpoint + cluster options.
|
|
- All cluster singletons + per-node actors live on this single ActorSystem — there is no second Akka instance.
|
|
|
|
See [Redundancy.md](Redundancy.md) for the role-leader + ServiceLevel story.
|
|
|
|
## Health endpoints
|
|
|
|
Both admin and driver nodes expose:
|
|
|
|
| Path | Status meaning |
|
|
|---|---|
|
|
| `/healthz` | Process alive. |
|
|
| `/health/ready` | ConfigDb reachable + cluster member state is `Up`. |
|
|
| `/health/active` | Admin-role leader (the node Traefik or an HA LB should route traffic to). |
|
|
|
|
Used by Traefik for the active-leader-only routing pattern (see [Architecture-v2.md](v2/Architecture-v2.md)).
|
|
|
|
## Historian backend (HistorianGateway — external)
|
|
|
|
The historian backend is the external `ZB.MOM.WW.HistorianGateway`, deployed and operated separately (not
|
|
installed by `Install-Services.ps1`). OtOpcUa connects to it over gRPC via the
|
|
`ZB.MOM.WW.HistorianGateway.Client` package — configure the `ServerHistorian:Endpoint` (`https://host:5222`)
|
|
and supply `ServerHistorian__ApiKey` via the environment on the OtOpcUa host side. The gateway must run with
|
|
`RuntimeDb:Enabled=true` + `RuntimeDb:EventReadsEnabled=true` and an API key carrying `historian:read` +
|
|
`historian:write` + `historian:tags:write`. See [Historian.md](Historian.md) for the full config-key and
|
|
deployment-prerequisite reference. (The retired Wonderware TCP sidecar: [Historian.Wonderware.md](drivers/Historian.Wonderware.md).)
|
|
|
|
## Script worker processes (`Scripting:Runner=OutOfProcess`, lmxopcua#780)
|
|
|
|
Off by default. When enabled, a driver node launches up to `Scripting:Worker:Shards` + 1 child processes
|
|
(`ZB.MOM.WW.OtOpcUa.ScriptWorker`, from `script-worker/` beside the host) on its first compiled script evaluation and
|
|
kills them with it: a Job Object with kill-on-close on Windows, pipe end of stream plus a parent-pid poll elsewhere,
|
|
and a Shutdown frame then a kill on host stop. Their stderr lands in the host log prefixed `ScriptWorker[<shard>]`.
|
|
At startup the host kills workers a dead predecessor of the same install left behind, matched by launch marker
|
|
(pid + start time + image, and only when the marker's host is gone). `Scripting:Worker:PeCacheDir`, when set, holds the
|
|
compiled-PE cache, its HMAC key and the launch markers: put it outside the install folder (startup refuses a relative path
|
|
or one inside the content root) on a directory only the service account can read. The `script-runner` readiness component
|
|
is at worst Degraded. A self-contained host publish (`dotnet publish … -r win-x64 --self-contained`, the Windows installs, which have no
|
|
.NET runtime) ships a **self-contained** worker in `script-worker/` (~97 MB: it carries its own runtime) and the host starts its
|
|
`ZB.MOM.WW.OtOpcUa.ScriptWorker.exe` apphost directly; a framework-dependent host (docker-dev) runs the worker's dll through the
|
|
dotnet muxer. See `docs/VirtualTags.md` §"Out-of-process runner".
|
|
|
|
### Worker isolation on Windows (`Install-ScriptWorkerIsolation.ps1`, design §14 #5/#6)
|
|
|
|
`scripts/install/Install-ScriptWorkerIsolation.ps1 -InstallRoot <dir> -HostServiceAccount <account> [-DenyPaths <dir>,…] [-ConfigureHostService]`
|
|
(idempotent; `-WhatIf` prints every change; `-Uninstall` rolls back; `-PlanOnly` prints the plan on any OS) sets up:
|
|
|
|
| Step | Change |
|
|
|---|---|
|
|
| Worker identity | A **disabled** anchor service `OtOpcUaScriptWorker` running as `NT SERVICE\OtOpcUaScriptWorker`, so the virtual account exists. It is never started. |
|
|
| `Scripting:Worker:Identity` | With `NT SERVICE\OtOpcUaScriptWorker` set (by `-ConfigureHostService`, or in appsettings), the host starts every worker under a **restricted token** derived from its own: every privilege but `SeChangeNotifyPrivilege` removed, and restricting SIDs = the identity + `Everyone` + `BUILTIN\Users` + `RESTRICTED`. Every access the worker makes must pass for the host account **and** for that set. The token's default DACL is host + SYSTEM + identity; the worker is created suspended and put in its Job Object before it runs; `TEMP`/`TMP` point at its per-launch working directory (ACL: host + SYSTEM full, identity Modify); `DOTNET_EnableDiagnostics=0`. The host also adds the identity to its own window station and desktop DACL (enumerate/read, and create-window on the desktop), once per process. USER32 connects to them in its DllMain and the runtime imports USER32, so without the grant `coreclr.dll` fails to load (`ERROR_DLL_INIT_FAILED`, measured on windev). The grant lives on the session's kernel objects and ends with the session; nothing persistent changes. The logon SID, which those objects normally grant, is deliberately not a restricting SID: the host's process DACL gives it `PROCESS_VM_READ`. Windows cannot start an ad-hoc child process *as* a virtual account (only the SCM can, for a service), and a separate local user would need a stored password plus `SeAssignPrimaryTokenPrivilege` on the host account; a restricted copy of the host's own token needs neither. Startup refuses the key off Windows or when malformed. |
|
|
| Pipe ACL | On Windows the worker pipe always carries an explicit, protected DACL: the host account and (with an identity) the worker identity, read/write, nobody else. No SYSTEM, no Administrators, no Everyone. The owner is the host token's owner, so the worker's `CurrentUserOnly` check of the server end still holds. |
|
|
| `PeCacheDir` | Created with a protected ACL: SYSTEM, Administrators and the host account full control, no Users. `pe\` grants the identity Modify (workers compile and write entries). The HMAC key and `workers\` markers are host-only. |
|
|
| Install folder | The identity is **denied** read and write on `<InstallRoot>` (inherited: ReadData/List, WriteData/AddFile, AppendData/AddSubdirectory, write attributes/EA, Delete, DeleteChild, ChangePermissions, TakeOwnership; Traverse/Execute, ReadAttributes and ReadPermissions stay so the loader reaches `script-worker\`). The worker cannot read `appsettings*.json`, logs or host binaries even where `Users` can. `script-worker\` grants it read + execute. |
|
|
| Sensitive folders (`-DenyPaths`, lmxopcua#890) | The same inherited deny on every listed folder. **Name every folder outside `<InstallRoot>` that holds keys, secrets, stores, logs or backups**: the worker's restricting SIDs include `Everyone` and `BUILTIN\Users`, so anything Users can read is otherwise readable to it (and a data drive usually lets Users create files in subfolders, e.g. a certificate in `pki\trusted\certs`). Name the deployment folder around `<InstallRoot>` (one ACE covers `pki\`, `data\`, `logs\`, `backup\`, old `app.bak.*`) **and** `pki\` and `data\` themselves, so the deny holds even on a folder whose DACL does not inherit. Drive and share roots, `script-worker\` and the PE cache are refused; each folder must exist. The list is recorded under `HKLM\…\Services\OtOpcUaScriptWorker\Parameters` (`DenyPaths`): a re-run removes the deny from folders dropped from the list, and `-Uninstall` removes every recorded one without repeating `-DenyPaths`. After an apply the installer WARNs about any item under a denied folder whose DACL is protected and has no deny of its own (the inherited deny does not reach it); without `-DenyPaths` it WARNs about `pki`/`data`/`logs`/`backup`/`config`/`secrets` siblings of `<InstallRoot>`. Wonder layout: `-InstallRoot E:\ApiInstall\OtOpcUa\app -DenyPaths E:\ApiInstall\OtOpcUa,E:\ApiInstall\OtOpcUa\pki,E:\ApiInstall\OtOpcUa\data`. |
|
|
| Firewall | Outbound **block** rule `OtOpcUa-ScriptWorker-BlockOutbound` on `script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.exe`, every profile. It covers the self-contained apphost the Windows installs run. A framework-dependent worker runs under `dotnet.exe` and is not covered, so there the analyzer's `System.Net` ban is the only network control. |
|
|
|
|
The plan is pure data asserted by `scripts/install/tests/Test-ScriptWorkerIsolation.ps1` (`pwsh -File …`, any OS).
|
|
`Test-ScriptWorkerIsolation.Windows.ps1` applies it for real in a scratch folder (elevated), and
|
|
`InstallerLayoutIsolationTests` (ScriptWorker.Tests, `OTOPCUA_ISOLATION_INSTALLER_E2E=1`, elevated) proves under the real
|
|
restricted token that a worker cannot read `pki\`/`data\`/`logs\`/`backup\` of a scratch wonder layout and still runs.
|
|
Re-run the installer after a deploy that replaces the install folder: a rename/swap drops the folder ACEs, but files
|
|
copied into the existing folder inherit them. Rollback: `-Uninstall -ConfigureHostService`, then restart the host.
|
|
|
|
## Install / Uninstall
|
|
|
|
- `scripts/install/Install-Services.ps1 -Roles admin,driver` — installs `OtOpcUaHost`.
|
|
- `scripts/install/Install-ScriptWorkerIsolation.ps1` — the ScriptWorker identity, ACLs and firewall rule (above); only for `Scripting:Runner` ≠ `InProcess`.
|
|
- `scripts/install/Uninstall-Services.ps1` — stops + removes the host service. (The historian backend is the external HistorianGateway — not installed/removed by these scripts.)
|
|
|
|
## Logging
|
|
|
|
Serilog with rolling-daily file sinks. Each host writes to `logs/otopcua-*.log` plus stdout (NSSM/systemd-friendly). Per-environment log level overrides go in `appsettings.{Environment}.json`.
|
|
|
|
## Depth reference
|
|
|
|
For the full host-architecture rationale (why fused vs. split, role-gating tradeoffs, multi-node deployment shapes), see `docs/plans/2026-05-26-akka-hosting-alignment-design.md` §3-4.
|