Joseph Doherty
f50b29abae
v2-ci / unit-tests (push) Failing after 1h12m10s
Second P1 slice of docs/plans/2026-11-09-out-of-process-script-runner-design.md; still opt-in behind Scripting:Runner=OutOfProcess (InProcess unchanged). - Compiled-PE cache (design §7, §14 #8): Scripting:Worker:PeCacheDir holds <dir>/pe/<id>.pe entries authenticated with HMAC-SHA256 under a per-node random key (<dir>/pe-cache.key, created by the host, 0600/0700 on Unix), handed to the worker in Hello. Entry id = SHA-256(compile fingerprint + kind + source); the fingerprint covers the Roslyn version, the Core.Scripting build (analyzer, wrapper, stack guard), the sandbox reference set and the runtime. Edited, truncated, other-key or renamed entries are rejected, deleted and recompiled; the worker never loads unverified bytes. Pruned by recency at startup (30 d / 50 000). ScriptEvaluator splits into CompileToImage + FromImage. Measured on 40 scripts, restart to every first result: macOS 820 ms no cache / 840 ms empty / 85 ms warm; windev (Windows) 9.5 s / 9.4 s / 1.0 s. - Quarantine race: WorkerConnection runs exit attribution synchronously BEFORE completing the evaluations/loads it held (and before Exited), and the shard passes it straight to the runner — a kill strike is in place before the killed key's slot is released. - Supervision: a dead worker restarts by itself after its backoff; Unavailable outcomes carry ScriptRunOutcome.Recovery (-> VirtualTagEvalResult.RetryWhen) and the VT actor / Calculation driver re-evaluate once against current inputs when it completes. Isolation worker stops after Scripting:Worker:IsolationIdleTimeout (10 min). Keys compiling when a worker dies are suspects too (isolation finds a poison compile and quarantines it). Startup orphan sweep by launch marker: kills only a process whose marker's host is gone and whose pid + start time + image match. - Crash-loop circuit (Scripting:Worker:CircuitOpenAfter, 2 min): runner Unavailable, one Error, evaluations answered at once while shards keep retrying at the cap. - Observability (§10): otopcua.script.runner.state, worker.up/inflight/queue_depth/heap_bytes (shard), ipc.roundtrip + eval.duration (kind), compile{cache}, isolation.suspects, exits reason=idle. `script-runner` health check on the ready tier, at worst Degraded (§14 #12: degraded but still ready). Left on #780: self-contained worker publish, installer (account, pipe ACL, firewall), docker-dev live drill, scripted-alarm seam, the §4.5 per-seam batch submit, the §8 perf harness. Tests: SupervisionTests, PeCacheStoreTests/PeCacheWorkerTests (real workers), ScriptRunnerHealthCheckTests, VT + Calc recovery seam tests; 14 mutations each turned their test red. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
117 lines
8.9 KiB
Markdown
117 lines
8.9 KiB
Markdown
# Service Hosting (v2)
|
|
|
|
## Overview
|
|
|
|
A production OtOpcUa deployment runs **one binary per node**. The historian backend is the external
|
|
`ZB.MOM.WW.HistorianGateway`, deployed separately (not installed by this repo's scripts):
|
|
|
|
| Process | Project | Runtime | Platform | Responsibility |
|
|
|---|---|---|---|---|
|
|
| **OtOpcUa Host** | `src/Server/ZB.MOM.WW.OtOpcUa.Host` | .NET 10 | AnyCPU | Single fused binary. The resolved [`EffectiveRoles`](#role-gating) decides what to mount: `admin` (Blazor + auth + control-plane singletons), `driver` (OPC UA endpoint + per-driver actors), or both. |
|
|
| **ZB.MOM.WW.HistorianGateway** *(external — separate deployment)* | not in this repo | .NET 10 | — | The default historian backend (`ServerHistorian:Backend=Gateway`; `SqlServer` needs no sidecar). OtOpcUa talks gRPC to it (via the `ZB.MOM.WW.HistorianGateway.Client` package) for HistoryRead, alarm `SendEvent`, and continuous `WriteLiveValues`. Must run `RuntimeDb:Enabled=true` + `RuntimeDb:EventReadsEnabled=true`; the API key must carry `historian:read` + `historian:write` + `historian:tags:write`. |
|
|
|
|
Galaxy access still uses the separately-installed **mxaccessgw** sidecar (see `docs/v2/Galaxy.ParityRig.md`); the gateway owns the MXAccess COM bitness constraint (its worker is x86 net48). Nothing in the OtOpcUa repo carries that constraint anymore. (The bespoke Wonderware historian sidecar this deployment used to ship was retired — see [drivers/Historian.Wonderware.md](drivers/Historian.Wonderware.md).)
|
|
|
|
> **v2 change.** v1's separate `OtOpcUa.Server` + `OtOpcUa.Admin` Windows services merged into a single role-gated `OtOpcUa.Host` binary. Two installers became one (with a `-Roles` parameter). The whole DI graph is composed in `OtOpcUa.Host/Program.cs`; per-role wiring is conditional on the env var.
|
|
|
|
## Role gating
|
|
|
|
`Program.cs` resolves `EffectiveRoles.Resolve(builder.Configuration)` (`ZB.MOM.WW.OtOpcUa.Cluster`) —
|
|
the single role authority (archreview R3-04): `Cluster:Roles` when non-empty, else `OTOPCUA_ROLES`,
|
|
else neither role. The SAME resolved instance drives both the conditional wiring below AND the Akka
|
|
gossiped roles (`AkkaClusterOptions.Roles`), so the two cannot disagree — and conditionally registers
|
|
services:
|
|
|
|
| Role present | Wires |
|
|
|---|---|
|
|
| `admin` | `AddOtOpcUaAuth`, `AddAdminUI`, `AddSignalR`, `AddOtOpcUaAdminClients`, `MapOtOpcUaAuth`, `MapAdminUI<App>`, `MapOtOpcUaHubs`, `WithOtOpcUaControlPlaneSingletons` (5 admin singletons via `Akka.Hosting`) |
|
|
| `driver` | `WithOtOpcUaRuntimeActors` (DriverHostActor + DbHealthProbeActor) — and the OPC UA endpoint on port 4840 |
|
|
| Either / both | `AddOtOpcUaConfigDb`, `AddOtOpcUaCluster`, `AddOtOpcUaHealth` (`/health/ready`, `/health/active`, `/healthz`) |
|
|
|
|
Single-node dev: `OTOPCUA_ROLES=admin,driver` (or the `Cluster:Roles` equivalent). Production: typically two admin nodes (HA pair) + N driver nodes.
|
|
|
|
### Per-role configuration overlays
|
|
|
|
`Program.cs` builds a role suffix (`RoleOverlay.Suffix`) by joining the resolved `EffectiveRoles.Gossiped` **plain** roles (`admin`/`driver`/`dev`) alphabetically with `-` and loads `appsettings.{roleSuffix}.json` as an optional overlay on top of base `appsettings.json`. A plain role is always lower-case in `Gossiped` regardless of how the operator spelled it, so the overlay filename is stable (`appsettings.driver.json`). **`cluster-{ClusterId}` roles are NOT part of the filename** (#678.6): they are mesh-membership tags, so `Cluster:Roles: ["driver", "cluster-SITE-A"]` loads `appsettings.driver.json` and `["admin", "driver", "cluster-MAIN"]` loads `appsettings.admin-driver.json`. (Before #678.6 the cluster role was folded into the name, producing e.g. `appsettings.admin-cluster-MAIN-driver.json`, which never exists — so on every node with a cluster role no shipped overlay ever loaded.) Three overlays ship in `src/Server/ZB.MOM.WW.OtOpcUa.Host/`:
|
|
|
|
- `appsettings.admin.json` — admin-only nodes
|
|
- `appsettings.driver.json` — driver-only nodes
|
|
- `appsettings.admin-driver.json` — fused single-node dev / small deployments
|
|
|
|
All three carry Serilog log-level overrides + `Security:Ldap:DevStubMode = false` + `Security:Ldap:Transport = Ldaps` (audit table: [Configuration.md § Role overlays](Configuration.md#role-overlay-audit)). Configuration loading order (lowest to highest precedence) is:
|
|
|
|
`appsettings.json` < `appsettings.{Environment}.json` < `appsettings.{role}.json` < environment variables < command-line args
|
|
|
|
The role overlay intentionally outranks `appsettings.{Environment}.json` so role-level security defaults (such as `DevStubMode = false`) cannot be silently overridden by a developer's local `appsettings.Development.json`; environment variables and command-line args still outrank everything for deployment-level overrides. Overlays are optional; the base file boots a node on its own.
|
|
|
|
## Akka cluster
|
|
|
|
The host joins an Akka.NET cluster bound to the address in `appsettings.json::Cluster`:
|
|
|
|
```json
|
|
{
|
|
"Cluster": {
|
|
"Hostname": "0.0.0.0",
|
|
"Port": 4053,
|
|
"PublicHostname": "node-a.lan",
|
|
"SeedNodes": ["akka.tcp://otopcua@node-a.lan:4053"],
|
|
"Roles": ["admin", "driver"]
|
|
}
|
|
}
|
|
```
|
|
|
|
- `SeedNodes` is **ordered**: a node listed in its own seed list must be entry 0, since Akka only lets `seed-nodes[0]` form a new cluster. `AkkaClusterOptionsValidator` fails startup otherwise — see [Redundancy.md § Bootstrap: self-first seed ordering](Redundancy.md#bootstrap-self-first-seed-ordering).
|
|
- `WithOtOpcUaClusterBootstrap` (in `OtOpcUa.Cluster`) loads the embedded HOCON (split-brain resolver, pinned dispatcher, failure detector tuning) and overlays remote endpoint + cluster options.
|
|
- All cluster singletons + per-node actors live on this single ActorSystem — there is no second Akka instance.
|
|
|
|
See [Redundancy.md](Redundancy.md) for the role-leader + ServiceLevel story.
|
|
|
|
## Health endpoints
|
|
|
|
Both admin and driver nodes expose:
|
|
|
|
| Path | Status meaning |
|
|
|---|---|
|
|
| `/healthz` | Process alive. |
|
|
| `/health/ready` | ConfigDb reachable + cluster member state is `Up`. |
|
|
| `/health/active` | Admin-role leader (the node Traefik or an HA LB should route traffic to). |
|
|
|
|
Used by Traefik for the active-leader-only routing pattern (see [Architecture-v2.md](v2/Architecture-v2.md)).
|
|
|
|
## Historian backend (HistorianGateway — external)
|
|
|
|
The historian backend is the external `ZB.MOM.WW.HistorianGateway`, deployed and operated separately (not
|
|
installed by `Install-Services.ps1`). OtOpcUa connects to it over gRPC via the
|
|
`ZB.MOM.WW.HistorianGateway.Client` package — configure the `ServerHistorian:Endpoint` (`https://host:5222`)
|
|
and supply `ServerHistorian__ApiKey` via the environment on the OtOpcUa host side. The gateway must run with
|
|
`RuntimeDb:Enabled=true` + `RuntimeDb:EventReadsEnabled=true` and an API key carrying `historian:read` +
|
|
`historian:write` + `historian:tags:write`. See [Historian.md](Historian.md) for the full config-key and
|
|
deployment-prerequisite reference. (The retired Wonderware TCP sidecar: [Historian.Wonderware.md](drivers/Historian.Wonderware.md).)
|
|
|
|
## Script worker processes (`Scripting:Runner=OutOfProcess`, lmxopcua#780)
|
|
|
|
Off by default. When enabled, a driver node launches up to `Scripting:Worker:Shards` + 1 child processes
|
|
(`ZB.MOM.WW.OtOpcUa.ScriptWorker`, from `script-worker/` beside the host) on its first compiled script evaluation and
|
|
kills them with it: a Job Object with kill-on-close on Windows, pipe end of stream plus a parent-pid poll elsewhere,
|
|
and a Shutdown frame then a kill on host stop. Their stderr lands in the host log prefixed `ScriptWorker[<shard>]`.
|
|
At startup the host kills workers a dead predecessor of the same install left behind, matched by launch marker
|
|
(pid + start time + image, and only when the marker's host is gone). `Scripting:Worker:PeCacheDir`, when set, holds the
|
|
compiled-PE cache, its HMAC key and the launch markers: put it outside the install folder (startup refuses a relative path
|
|
or one inside the content root) on a directory only the service account can read. The `script-runner` readiness component
|
|
is at worst Degraded. The worker currently runs under the host's account through the dotnet muxer; the dedicated low-privilege account,
|
|
pipe ACL and firewall outbound rule (design §14 #5/#6) are installer work still open on lmxopcua#780. See
|
|
`docs/VirtualTags.md` §"Out-of-process runner".
|
|
|
|
## Install / Uninstall
|
|
|
|
- `scripts/install/Install-Services.ps1 -Roles admin,driver` — installs `OtOpcUaHost`.
|
|
- `scripts/install/Uninstall-Services.ps1` — stops + removes the host service. (The historian backend is the external HistorianGateway — not installed/removed by these scripts.)
|
|
|
|
## Logging
|
|
|
|
Serilog with rolling-daily file sinks. Each host writes to `logs/otopcua-*.log` plus stdout (NSSM/systemd-friendly). Per-environment log level overrides go in `appsettings.{Environment}.json`.
|
|
|
|
## Depth reference
|
|
|
|
For the full host-architecture rationale (why fused vs. split, role-gating tradeoffs, multi-node deployment shapes), see `docs/plans/2026-05-26-akka-hosting-alignment-design.md` §3-4.
|