Files
lmxopcua/docs/Configuration.md
Joseph Doherty 8a173bbaf6 docs(mesh): Phase 6 — mark done, remove single-mesh caveats, document pair-local meshes
Redundancy.md: KNOWN-LIMITATION (per-Akka-cluster election) marked RESOLVED;
new 'Per-cluster meshes (Phase 6)' section + pair-local secrets note; manual-
failover mesh-scope caveat removed; auto-down 1-vs-1 note updated (every mesh is
now two nodes, drill deferred to Phase 7). IManualFailoverService + ClusterRedundancy
razor caveats updated to pair-local reality. Configuration.md documents the
cluster-{ClusterId} role, per-pair self-first seeds, SplitTopologyTransportValidator,
and the intentional Dps default. Program + design status tables + CLAUDE.md marked
Phase 6 DONE.

(Task 8 subagent completed the edits but died on an API error before committing;
reviewed and committed by the controller.)

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 03:02:35 -04:00

371 lines
36 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Configuration Reference
> **v3 in progress (Batch 1 landed — schema + RawPath identity).** The config **DB schema** was
> replaced by the greenfield v3 model: a driver-centric **Raw tree**
> (`Cluster → RawFolder → DriverInstance → Device → TagGroup → Tag`) plus a reference-only **UNS**
> projection (`UnsTagReference`). A tag's identity is now its **RawPath**
> (`<Folder/…>/<Driver>/<Device>/<TagGroup/…>/<Tag>`), not a `TagConfig.FullName` blob; the
> `Namespace`/`NamespaceKind`/`EquipmentImportBatch` entities are retired, and connection endpoints
> moved from `DriverConfig` into each `Device`'s `DeviceConfig`. The two OPC UA namespaces (`ns=Raw` /
> `ns=UNS`) and the `/raw` authoring UI arrive in later v3 batches; **Batch 1 leaves the address space
> deliberately dark** (drivers connect, but no tag variables materialize yet). The `appsettings*.json`
> surface below is unchanged. Full design: `docs/plans/2026-07-15-raw-uns-two-subtree-v3-design.md`.
This is the live configuration reference for the OtOpcUa Host (`src/Server/ZB.MOM.WW.OtOpcUa.Host/`). It enumerates the `appsettings*.json` sections, the bound Options classes, and the `OTOPCUA_*` / sim-endpoint environment variables — every entry grounded in source.
Two related concerns get their own dedicated pages and are **only summarised + linked** here, not duplicated:
- **Transport security, OPC UA authentication, LDAP, data-/control-plane authorization** → [`security.md`](security.md)
- **Redundancy + the `Cluster` section** → [`Redundancy.md`](Redundancy.md)
## How configuration is layered
The Host (`Program.cs`) loads `appsettings.json`, then overlays a **per-role** file chosen from the cluster roles:
- A single role → `appsettings.{role}.json` (e.g. `appsettings.driver.json`, `appsettings.admin.json`).
- Both roles → `appsettings.admin-driver.json` (roles joined with `-`, ordinal-sorted).
- `appsettings.{ASPNETCORE_ENVIRONMENT}.json` (e.g. `appsettings.Development.json`) is layered on by the host builder.
All role overlays are **optional** — the base `appsettings.json` plus the Options-class C# defaults are enough to boot. The roles themselves come from the `OTOPCUA_ROLES` env var (see [`ServiceHosting.md`](ServiceHosting.md) and the table below).
The checked-in `appsettings*.json` files are deliberately thin: they carry only `Serilog` and the `Security:Ldap` overlay. Everything else (`OpcUa`, `Cluster`, `ConnectionStrings`/`ConfigDb`) binds from the Options-class defaults documented below unless an operator adds the section explicitly or supplies the corresponding environment variable.
---
## `appsettings` sections
### `Serilog`
- **Purpose:** logging. Console + rolling daily file sink, layered with the shared `ZB.MOM.WW.Telemetry` enrichers (`AddZbSerilog` in `Program.cs`).
- **Where bound:** `builder.AddZbSerilog(...)` reads `Serilog` from configuration (`ReadFrom.Configuration`).
- **Checked-in shape** (`appsettings.json`): `Using` = `[ "Serilog.Sinks.Console", "Serilog.Sinks.File" ]`, `WriteTo` = a `Console` sink and a `File` sink (`path: logs/otopcua-.log`, `rollingInterval: Day`). Role overlays add `MinimumLevel` / `Override` blocks (e.g. `Opc.Ua: Debug`, `Akka: Information`).
### `OpcUa`
- **Purpose:** the OPC UA server endpoint identity, listening port, PKI, transport-security profiles, and redundancy peer advertising.
- **Options class:** `OpcUaApplicationHostOptions``src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OpcUaApplicationHost.cs`.
- **Bound by:** `AddValidatedOptions<OpcUaApplicationHostOptions, OpcUaApplicationHostOptionsValidator>(config, "OpcUa")` in `Program.cs` (driver-role only). Validated fail-fast at startup by `OpcUaApplicationHostOptionsValidator` (`src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/OpcUaApplicationHostOptionsValidator.cs`).
| Key | Type | Default | Meaning |
|---|---|---|---|
| `ApplicationName` | string | `OtOpcUa` | Server application name. Required (validated). |
| `ApplicationUri` | string | `urn:OtOpcUa` | Server application URI. Must be unique per redundancy node. Required. |
| `ProductUri` | string | `https://zb.com/otopcua` | Product URI. Not validated. |
| `OpcUaPort` | int | `4840` | Binary endpoint listen port. Validated as a port. |
| `PublicHostname` | string | `0.0.0.0` | Hostname/IP advertised in endpoint descriptions. Required. |
| `ApplicationConfigPath` | string? | `null` | Optional path to an application config XML; loaded instead of building from defaults. |
| `PkiStoreRoot` | string | `pki` | Root of the PKI hierarchy (`own`/`issuer`/`trusted`/`rejected` substores created under it). Required. See [`security.md`](security.md). |
| `EnabledSecurityProfiles` | list of `OpcUaSecurityProfile` | `[None, Basic256Sha256Sign, Basic256Sha256SignAndEncrypt]` | Transport-security profiles, one endpoint per entry. Must contain ≥1. Profile detail in [`security.md`](security.md). |
| `AutoAcceptUntrustedClientCertificates` | bool | `false` | Auto-trust unknown client certs on first connect (dev convenience). Not validated. See [`security.md`](security.md). |
| `PeerApplicationUris` | list of string | `[]` (empty) | Partner node `ApplicationUri`s published in `Server.ServerArray` for redundancy discovery. See [`Redundancy.md`](Redundancy.md). |
> **Transport security profiles** (the values in `EnabledSecurityProfiles` — `None`, `Basic256Sha256Sign`, `Basic256Sha256SignAndEncrypt`) and the PKI trust flow are documented in full in [`security.md`](security.md). This page does not duplicate them.
### `Security`
- **Purpose:** Admin-UI and OPC UA authentication. Three subsections, each its own Options class:
| Subsection | Options class (`SectionName`) | Purpose |
|---|---|---|
| `Security:Ldap` | `LdapOptions``src/Server/ZB.MOM.WW.OtOpcUa.Security/Ldap/LdapOptions.cs` | LDAP bind for Admin cookie login + OPC UA UserName tokens. Bound by `AddValidatedOptions<LdapOptions, LdapOptionsValidator>` in `Program.cs`. |
| `Security:Jwt` | `JwtOptions``src/Server/ZB.MOM.WW.OtOpcUa.Security/Jwt/JwtOptions.cs` | Signing config for the JWT minted at `/auth/token` for **external** consumers (OPC UA clients / automation). |
| `Security:Cookie` | `OtOpcUaCookieOptions``src/Server/ZB.MOM.WW.OtOpcUa.Security/CookieOptions.cs` | The Admin-UI auth cookie (`AddOtOpcUaAuth` copies these onto `CookieAuthenticationOptions`). |
**`Security:Ldap` — see [`security.md`](security.md) for the full field-by-field reference and bind-flow.** The checked-in role overlays set only `DevStubMode` and `Transport`; the remaining `LdapOptions` fields (`Enabled`, `Server`, `Port`, `AllowInsecure`, `SearchBase`, `ServiceAccountDn`, `ServiceAccountPassword`, `GroupAttribute`, `DisplayNameAttribute`, `UserNameAttribute`, `GroupToRole`) are covered there.
**`Security:Jwt`** key fields (`JwtOptions`):
| Key | Type | Default | Meaning |
|---|---|---|---|
| `SigningKey` | string | `""` | HS256 signing key; must be ≥32 bytes UTF-8. Set from your secret store — never commit a value. |
| `Issuer` | string | `otopcua` | JWT issuer. |
| `Audience` | string | `otopcua` | JWT audience. |
| `ExpiryMinutes` | int | `15` | Token lifetime. |
**`Security:Cookie`** key fields (`OtOpcUaCookieOptions`):
| Key | Type | Default | Meaning |
|---|---|---|---|
| `Name` | string | `ZB.MOM.WW.OtOpcUa.Auth` | Auth cookie name. Changing it invalidates existing sessions on next deploy. |
| `ExpiryMinutes` | int | `30` | Idle sliding-window length. |
| `RequireHttpsCookie` | bool | `true` | `SecurePolicy = Always`. Set `false` only for plain-HTTP local dev (emits a startup Warning). |
> Authentication, data-plane authorization (`NodeAcl` / `PermissionTrie`), and control-plane Admin roles are all in [`security.md`](security.md).
### `Cluster`
- **Purpose:** Akka.NET cluster identity, transport, and roles — the backbone of redundancy.
- **Options class:** `AkkaClusterOptions` (`SectionName = "Cluster"`) — `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/AkkaClusterOptions.cs`. Bound by `AddOtOpcUaCluster(config)` in `Program.cs`.
| Key | Type | Default | Meaning |
|---|---|---|---|
| `SystemName` | string | `otopcua` | Akka actor-system name. |
| `Hostname` | string | `0.0.0.0` | Bind hostname. |
| `Port` | int | `4053` | Cluster transport port. **Duplicated as `ClusterNode.AkkaPort` in the Config DB** — see the note below. |
| `PublicHostname` | string | `127.0.0.1` | Hostname advertised in cluster gossip; must be reachable by peers. **Duplicated as `ClusterNode.Host`** — see the note below. |
| `SeedNodes` | string[] | `[]` | Seed nodes for bootstrapping. **Ordered:** a node that appears in its own list must be entry 0 (only `seed-nodes[0]` can form a new cluster), or the host refuses to start — see [Redundancy.md § Bootstrap: self-first seed ordering](Redundancy.md#bootstrap-self-first-seed-ordering). **Per-pair since Phase 6:** each application `Cluster`'s pair is its own Akka mesh, so a node's `SeedNodes` lists only itself (first) and its own pair partner (second) — never a node from another cluster's pair. |
| `Roles` | string[] | `[]` | Cluster roles for this node. When empty, falls back to `OTOPCUA_ROLES`. Allowed values: `admin`, `driver`, `dev`, plus a per-cluster `cluster-{ClusterId}` role (e.g. `cluster-MAIN`, `cluster-SITE-A`) — see [Per-cluster mesh split (Phase 6)](#per-cluster-mesh-split-phase-6-cluster-clusterid-roles--splittopologytransportvalidator) below. |
> The full redundancy model (ServiceLevel tiers, split-brain, peer discovery) is in [`Redundancy.md`](Redundancy.md). The OPC UA peer-URI advertising lives in the `OpcUa:PeerApplicationUris` key above.
> **`Port` / `PublicHostname` are stored twice.** The node binds from these keys; the fleet's `ClusterNode` row records the same address as `AkkaPort` / `Host` so that **central can dial the node without sharing a gossip ring with it** — which is what [per-cluster mesh Phase 2](plans/2026-07-21-per-cluster-mesh-design.md) needs. Nothing in the schema makes the two agree, so `ClusterNodeAddressReconcilerActor` (admin-role singleton) compares them against live membership and logs an Error on mismatch. If you change `Cluster:Port` or `Cluster:PublicHostname` on a node, **update its `ClusterNode` row too** — a node binding 4054 with a row saying 4053 still gossips fine today, and fails silently in Phase 2. The row's `NodeId` is `host:port` and is also the deploy path's ack identity, so a drifted node's deployment acks stop matching as well. See [`config-db-schema.md` § `ClusterNode`](v2/config-db-schema.md#clusternode).
### Per-cluster mesh split (Phase 6): `cluster-{ClusterId}` roles + `SplitTopologyTransportValidator`
- **Purpose:** [per-cluster mesh](plans/2026-07-22-per-cluster-mesh-program.md) **Phase 6** split the
single fleet-wide Akka mesh into one independent 2-node mesh per application `Cluster`. A node's
`Cluster:Roles` now carries a `cluster-{ClusterId}` role alongside `driver` (and `admin` for a fused
central node) — e.g. `driver,cluster-SITE-A` — and its `Cluster:SeedNodes` lists only itself and its
own pair partner (see the `Roles`/`SeedNodes` rows above). Nodes in different clusters never gossip
with each other.
- **`SplitTopologyTransportValidator`** (`src/Core/ZB.MOM.WW.OtOpcUa.Cluster/`, wired at
`ValidateOnStart` alongside the other `Cluster`-family validators) fails host start on any node
carrying a `cluster-{ClusterId}` role unless it is also configured for the split-safe transports:
`MeshTransport:Mode = ClusterClient`, `Telemetry:Mode = Grpc` on a driver node, `TelemetryDial:Mode =
Grpc` on an admin node. A DPS-mode transport on a cluster-role node would try to gossip across a
partition boundary that no longer exists, so the validator fails closed rather than let commands or
telemetry silently vanish. A legacy node with no `cluster-{ClusterId}` role is exempt and may stay on
`Dps`.
- **The compiled transport-mode defaults were deliberately NOT flipped.** `MeshTransport:Mode`,
`Telemetry:Mode`, and `TelemetryDial:Mode` (documented below) all still default to `Dps` — a
blast-radius assessment found that flipping the compiled default would break every integration test
and every existing `appsettings.json` profile that leaves these keys unset. `SplitTopologyTransportValidator`
plus explicit per-deployment configuration is the guardrail, not a default flip; every node that
actually carries a `cluster-{ClusterId}` role must set the split-safe modes explicitly.
- See [`Redundancy.md` § Per-cluster meshes (Phase 6)](Redundancy.md#per-cluster-meshes-phase-6) for the
full runtime picture (one Primary per pair, the cluster-scoped `RedundancyStateActor` singleton,
`ClusterNodeAddressReconciler` own-cluster-scoping, and central's one-`ClusterClient`-per-`Cluster`
fan-out).
### `MeshTransport` (central↔node command transport)
- **Purpose:** selects how central's three command channels (`deployments`, `driver-control`, `alarm-commands`) and the node's `deployment-acks` reply reach the other side — over the mesh-wide DistributedPubSub, or over an Akka `ClusterClient` that does **not** require central and the node to share a gossip ring. The second option is what [per-cluster mesh](plans/2026-07-21-per-cluster-mesh-design.md) Phase 6 needs once the meshes split.
- **Options class:** `MeshTransportOptions` (`SectionName = "MeshTransport"`) — `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/MeshTransportOptions.cs`. Bound and validated by `AddOtOpcUaCluster(config)`; `MeshTransportOptionsValidator` runs at `ValidateOnStart`.
| Key | Type | Default | Meaning |
|---|---|---|---|
| `Mode` | string | `Dps` | `Dps` = DistributedPubSub (the transport this repo has always used). `ClusterClient` = the mesh boundary. Any other value **refuses to start**. |
| `CentralContactPoints` | string[] | `[]` | Node-side only: central node **addresses** (`akka.tcp://<system>@<host>:<port>`). Required under `ClusterClient`, ignored under `Dps`. |
| `ContactRefreshSeconds` | int | `60` | Central-side only: how often central rebuilds its ClusterClient contact set from the `ClusterNode` rows. |
**Addresses only — no actor-path suffix.** `/system/receptionist` is appended at construction time; a contact carrying `/user/` or `/system/` is rejected at startup, because a doubled suffix produces a path that resolves to nothing and drops every command in silence.
**It is a dark switch, and both sides are always wired.** The comm actors are spawned and registered with the `ClusterClientReceptionist` on every node in *both* modes, so flipping `Mode` is a config change rather than a redeploy. Under `Dps` the registration is simply idle.
**Central's contact set comes from the enabled, non-maintenance `ClusterNode` rows** — the same source Phase 1 gave the deploy ack set. A node in `MaintenanceMode` is not dialled, but note that **the contact set does not scope delivery**: a receptionist serves its whole cluster, so `SendToAll` still reaches every registered node exactly as today's DPS broadcast does. The filter is about which receptionists are worth dialling, not about who receives.
**Buffering is off by design** (`akka.cluster.client.buffer-size = 0`, set in the shipped `akka.conf`). If no receptionist is known, the command is dropped rather than queued. Akka's default of 1000 would replay a `DispatchDeployment` on reconnect — silently, on the client's own timer, at an unbounded remove from the dispatch. Do not "fix" it back to quiet a flaky test.
> **What this does *not* buy you.** It does not guarantee a node never applies a deployment the DB records as `TimedOut`. A node that missed a dispatch keeps the orphan `Applying` row the coordinator pre-created, and `DriverHostActor.Bootstrap()` replays that row on the node's next restart — live-observed applying a deployment 44 s after it was sealed `TimedOut` (see the [Phase 2 live gate](plans/2026-07-22-mesh-phase2-live-gate.md), finding B). The boot replay is the better-behaved of the two — deliberate, Warning-logged, and bounded to a restart — but it is still a replay. `buffer-size = 0` removes the *silent, timer-driven* one only.
On the docker-dev rig every node carries these keys already; flip the whole rig with `OTOPCUA_MESH_MODE=ClusterClient` at `docker compose up`.
### `ConfigSource` / `ConfigServe` (config fetch-and-cache)
- **Purpose:** [per-cluster mesh](plans/2026-07-21-per-cluster-mesh-design.md) **Phase 3**. `ConfigSource:Mode` selects where a **driver** node reads its deployed configuration: `Direct` (read `Deployment.ArtifactBlob` from central SQL, today's behaviour) or `FetchAndCache` (fetch the artifact bytes from central over a gRPC stream, verify SHA-256, cache in LocalDb, and read **only** the cache — no central-SQL config read). `ConfigServe` is the **central** (admin-role) serve side that a `FetchAndCache` node dials.
- **Options classes:** `ConfigSourceOptions` (`SectionName = "ConfigSource"`) + `ConfigServeOptions` (`SectionName = "ConfigServe"`), both in `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/`. Bound by `AddOtOpcUaCluster(config)`; `ConfigSourceOptionsValidator` runs at `ValidateOnStart`.
**`ConfigSource` (driver-side fetch):**
| Key | Type | Default | Meaning |
|---|---|---|---|
| `Mode` | string | `Direct` | `Direct` = read central SQL (unchanged). `FetchAndCache` = fetch from central over gRPC, read the LocalDb cache. Any other value **refuses to start**. |
| `CentralFetchEndpoints` | string[] | `[]` | Central artifact-gRPC base addresses, e.g. `http://central-1:4055` (h2c ⇒ the `http` scheme). Tried in order for failover. **Required** under `FetchAndCache`. |
| `ApiKey` | string | `""` | Shared bearer key; must equal central's `ConfigServe:ApiKey`. **Supply via env `ConfigSource__ApiKey` — never commit.** Required under `FetchAndCache`. |
| `FetchTimeoutSeconds` | int | `30` | Per-fetch deadline. Must be positive under `FetchAndCache`. |
**`ConfigServe` (central serve):**
| Key | Type | Default | Meaning |
|---|---|---|---|
| `GrpcListenPort` | int | `0` | Dedicated **h2c-only** Kestrel port for the artifact gRPC service. `0` = disabled (nothing bound). Must differ from the main HTTP port and from `LocalDb:SyncListenPort` (h2c cannot share a cleartext HTTP/1 port). Mapped only on admin-role nodes. |
| `ApiKey` | string | `""` | Shared bearer key the serve-side interceptor checks (constant-time, **fail-closed**: an unset key rejects every call). **Supply via env `ConfigServe__ApiKey` — never commit.** |
**It is a dark switch; the serve side is always wired.** Central maps the `DeploymentArtifactService` behind the `ConfigServeAuthInterceptor` (path-scoped, `FixedTimeEquals`, fail-closed) whenever `ConfigServe:GrpcListenPort > 0`, in both modes — so flipping a node to `FetchAndCache` is a config change, not a redeploy. A `FetchAndCache` node that cannot fetch (all endpoints down, `NotFound`, SHA mismatch) **fails the apply and keeps last-known-good** — a null fetch is "no answer," never an empty configuration (the [#485](plans/2026-07-15-v3-batch4-address-space-plan.md) contract). At boot it restores served state from the LocalDb pointer with no SQL read; an empty or unreadable cache lands Steady-with-no-revision (the first dispatch fetches), never Stale.
**The listener is dedicated h2c and coexists with the LocalDb-sync listener.** A fused admin+driver node with **both** `ConfigServe:GrpcListenPort` and `LocalDb:SyncListenPort` set re-binds its existing HTTP surface exactly once and adds both dedicated HTTP/2-only ports on top — see the same Kestrel-takeover caveat under [`LocalDb`](#localdb-pair-local-config-cache--driver-role-only).
**`RevisionHash == SHA-256(artifact bytes)` is load-bearing.** `ConfigComposer` computes the deployment's revision hash as the lowercase-hex SHA-256 of the exact artifact bytes, so the node verifies a fetch against the `RevisionHash` the dispatch already carries — no hash rides in the proto. If a future change makes the revision hash anything else, every fetch fails verification.
On the docker-dev rig the central pair carries `ConfigServe` (port `4055`, dev key) and stays `Direct`; the four site nodes carry `ConfigSource` (endpoints + matching key) defaulting to `Direct`. Flip only the site nodes with `OTOPCUA_CONFIG_MODE=FetchAndCache` at `docker compose up` — central keeps SQL and models the target topology.
**`FetchAndCache` is mandatory on a driver-only node — per-cluster mesh Phase 4.** `Program.cs` now
registers `AddOtOpcUaConfigDb` only when the node carries the `admin` role, so a **driver-only**
node (`Cluster:Roles` has `driver`, not `admin`) holds **no `ConfigDb` connection string at all**
there is nothing for `Direct` to read. `ConfigSourceOptionsValidator` enforces this at
`ValidateOnStart`: a driver-only node whose `ConfigSource:Mode` is `Direct` **fails host start**
with an error naming the fix (`ConfigSource:Mode=FetchAndCache`). A **fused** `admin,driver` node
still owns its ConfigDb via the admin role and may stay `Direct` — the validator only fires on
`driver` without `admin`. On a driver-only node the deployed configuration, the scripted-alarm Part
9 condition state (`LocalDbAlarmConditionStateStore`, replacing the ConfigDb-backed
`EfAlarmConditionStateStore`), and the alarm store-and-forward buffer all live in the node's
LocalDb; central remains the system of record for deploy acks (`ConfigPublishCoordinator.
PersistNodeAck`) and LDAP group→role DB grants (a driver-only node maps LDAP groups to roles from
the `Security:Ldap:GroupToRole` appsettings baseline only — see `docs/Redundancy.md` and
`CLAUDE.md` §"Phase 4" for the full picture, including the client-visible ServiceLevel change).
### `Telemetry` / `TelemetryDial` (live telemetry transport)
- **Purpose:** [per-cluster mesh](plans/2026-07-21-per-cluster-mesh-design.md) **Phase 5**. Selects how
the four live-observability channels (`alerts`, `script-logs`, `driver-health`,
`driver-resilience-status`) reach central: `Dps` (DistributedPubSub — the four existing SignalR
bridges, today's behaviour) or `Grpc` (a driver node hosts one gRPC server-streaming contract over a
dedicated h2c port; central dials in and feeds the same in-process sinks). `Telemetry` is the
**node/serve** side; `TelemetryDial` is the **central/dial** side.
- **Options classes:** `TelemetryOptions` (`SectionName = "Telemetry"`) + `TelemetryDialOptions`
(`SectionName = "TelemetryDial"`), both in `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/`. Bound by
`AddOtOpcUaCluster(config)`; their validators run at `ValidateOnStart`.
**`Telemetry` (node-side, serve):**
| Key | Type | Default | Meaning |
|---|---|---|---|
| `Mode` | string | `Dps` | `Dps` = no change, node just keeps publishing DPS as it always has. `Grpc` = central-ingest hint (see `TelemetryDial:Mode` below — this key does not gate anything on the node itself). Any other value **refuses to start**. |
| `GrpcListenPort` | int | `0` | Dedicated **h2c-only** Kestrel port for the telemetry stream service. `0` = disabled (nothing bound). The node hosts the server whenever this is `> 0`, **regardless of `Mode`**. |
| `ApiKey` | string | `""` | Shared bearer key the serve-side interceptor checks (constant-time, **fail-closed**: an unset key rejects every call). **Supply via env `Telemetry__ApiKey` — never commit.** |
**`TelemetryDial` (central-side, dial):**
| Key | Type | Default | Meaning |
|---|---|---|---|
| `Mode` | string | `Dps` | `Dps` = the four DPS SignalR bridges (alert, script-log, driver-status, resilience) subscribe as today. `Grpc` = those four bridges are **not spawned**; the `TelemetryDialSupervisor` actor dials every driver node instead and feeds the same sinks. Any other value **refuses to start**. |
| `ApiKey` | string | `""` | Shared bearer key; must equal the nodes' `Telemetry:ApiKey`. **Supply via env `TelemetryDial__ApiKey` — never commit.** Required under `Grpc`. |
| `ContactRefreshSeconds` | int | `60` | How often central rebuilds its dial-target set from the `ClusterNode` rows (`Host` + `GrpcPort`). |
| `CallTimeoutSeconds` | int | `30` | Per-call deadline for the streaming RPC's keepalive/health bookkeeping. |
**It is a dark switch; the node always hosts.** Unlike `ConfigServe`/`MeshTransport`, where the serve
side is gated on a role, the telemetry server binds whenever `Telemetry:GrpcListenPort > 0` — hosting
is unconditional on a driver-role node, in both modes, because the node also always emits into its
local hub and always publishes DPS. Only `TelemetryDial:Mode` on central actually changes behaviour:
which of the two already-always-available sources (DPS bridges vs. the gRPC dial supervisor) feeds the
AdminUI sinks. Flipping either side is a config change plus restart, not a redeploy — the same
discipline as `MeshTransport:Mode` and `ConfigSource:Mode`.
**A driver-role node with `Telemetry:Mode=Grpc` must set `GrpcListenPort > 0`** (nothing to serve
otherwise) — enforced by `TelemetryOptionsValidator`. **`Grpc` mode on either side requires a
non-empty `ApiKey`** — fail-closed, same as `ConfigServe`/`LocalDb:Replication`.
**Three DPS observability topics stay out of scope for this transport, on purpose:**
`redundancy-state` (pair-local, built from `Cluster.State`, stays on DPS in both modes — see
[`Redundancy.md`](Redundancy.md#command-transport-centralnode)), `fleet-status` (central-internal;
`Fleet.razor` polls the Config DB, not any stream), and `deployment-acks` (already rides the Phase 2
`ClusterClient` transport as a command-plane reply). See
[`docs/Telemetry.md`](Telemetry.md) for the full architecture, the reconnect/generation-stamped
dialer story, and the rationale for the node-hosts/central-dials inversion.
On the docker-dev rig every driver-role node carries `Telemetry:GrpcListenPort` (`4056`) + a
committed dev key, and central carries the matching `TelemetryDial:ApiKey`; both `Mode` keys stay
unset (⇒ `Dps`) until the rig is flipped with `Telemetry__Mode=Grpc` / `TelemetryDial__Mode=Grpc` at
`docker compose up`.
### `ConnectionStrings` → `ConfigDb`
- **Purpose:** the central Config DB connection string. **Required on admin-role nodes only**`Program.cs` calls `AddOtOpcUaConfigDb` when `hasAdmin` (per-cluster mesh Phase 4). A driver-only node holds no `ConfigDb` connection string; it fetches its configuration via `ConfigSource:Mode=FetchAndCache` instead (see above). A fused `admin,driver` node still requires it.
- **Bound by:** `AddOtOpcUaConfigDb(config)` (`src/Core/ZB.MOM.WW.OtOpcUa.Configuration/ServiceCollectionExtensions.cs`). The connection-string name constant is `ConnectionStringName = "ConfigDb"`, read via `configuration.GetConnectionString("ConfigDb")`. If absent, startup throws with a message pointing to either `appsettings.json` or the `OTOPCUA_CONFIG_CONNECTION` env var.
- **Shape:** standard `ConnectionStrings:ConfigDb` SQL Server connection string. There is no checked-in default in the thin `appsettings*.json` — supply it per environment.
The Config DB itself (the EF Core `OtOpcUaConfigDbContext`, entities, draft/publish generations, `NodeAcl`, `LdapGroupRoleMapping`, migrations) is the durable home for the fleet's drivers, UNS hierarchy, ACLs, and audit log. For the **full schema** see [`docs/v2/config-db-schema.md`](v2/config-db-schema.md). This page does not duplicate it.
### Galaxy / MxAccess driver config (`DriverConfig` JSON, not `appsettings`)
The Galaxy/MxAccess connection settings are **not an `appsettings` section.** They are driver-instance options stored in the `DriverConfig` JSON column of the Config DB (edited via the Admin UI), bound to `GalaxyDriverOptions` (`src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Contracts/GalaxyDriverOptions.cs`, namespace `...Driver.Galaxy.Config`). It decomposes into nested records:
| Record | Key fields (default) | Meaning |
|---|---|---|
| `GalaxyGatewayOptions` (`Gateway`) | `Endpoint`; `ApiKeySecretRef`; `UseTls` (`true`); `CaCertificatePath` (`null`); `ConnectTimeoutSeconds` (`10`); `DefaultCallTimeoutSeconds` (`30`); `StreamTimeoutSeconds` (`0` = unlimited) | mxaccessgw gateway connection. `ApiKeySecretRef` supports `env:NAME` / `file:PATH` / `dev:KEY` / literal forms (resolved at `InitializeAsync`); prefer `env:`/`file:` in production. Never store a cleartext key. |
| `GalaxyMxAccessOptions` (`MxAccess`) | `ClientName`; `PublishingIntervalMs` (`1000`); `WriteUserId` (`0` = anonymous); `EventPumpChannelCapacity` (`50000`) | MXAccess client identity + tuning. `ClientName` **must be unique per OtOpcUa instance** (redundancy pairs enforce this). |
| `GalaxyRepositoryOptions` (`Repository`) | `DiscoverPageSize` (`5000`); `WatchDeployEvents` (`true`) | Galaxy Repository browse paging + deploy-event watching. |
| `GalaxyReconnectOptions` (`Reconnect`) | `InitialBackoffMs` (`500`); `MaxBackoffMs` (`30000`); `ReplayOnSessionLost` (`true`) | In-driver reconnect-supervisor backoff. |
| (top-level) | `ProbeTimeoutSeconds` (`30`, range 160) | AdminUI Test-Connect probe timeout. |
> The `OTOPCUA_GALAXY_*` environment variables that v1's in-process `Galaxy.Host` consumed **no longer live in this repo** — they moved into the separately-installed mxaccessgw gateway's own config (see the v1 archive pointer in `docs/README.md` and the Galaxy overview at [`docs/drivers/Galaxy.md`](drivers/Galaxy.md)). The only Galaxy connection secret this repo touches is the gateway API key via `ApiKeySecretRef` above.
### Historian config (HistorianGateway)
The historian backend is the external **`ZB.MOM.WW.HistorianGateway`** sidecar, consumed as the
`ZB.MOM.WW.HistorianGateway.Client` gRPC package (the retired Wonderware TCP sidecar is documented at
[`docs/drivers/Historian.Wonderware.md`](drivers/Historian.Wonderware.md)). The OtOpcUa host reads three
appsettings sections — `ServerHistorian` (read path + gateway connection), `ContinuousHistorization`
(FasterLog outbox + recorder draining to `WriteLiveValues`), and `AlarmHistorian` (the store-and-forward
alarm sink draining to `SendEvent`; its buffer lives in the node's `LocalDb:Path` database, not a
file of its own). The gateway connection (endpoint / key / TLS) lives **only** in
`ServerHistorian`; the other two sections source it from there.
The gateway API key is supplied via the environment variable **`ServerHistorian__ApiKey`** — never committed
to config. The target gateway must run `RuntimeDb:Enabled=true` + `RuntimeDb:EventReadsEnabled=true`, and the
key must carry the scopes `historian:read`, `historian:write`, `historian:tags:write`. See
[`docs/Historian.md`](Historian.md) for the full key reference, the migration note (old Wonderware keys →
gateway keys), and the deployment prerequisites.
### `LocalDb` (pair-local config cache — driver role only)
Driver-role nodes keep a consolidated **`ZB.MOM.WW.LocalDb`** SQLite database that caches the deployed
configuration artifact (chunked) so the node can **boot from cache when central SQL is unreachable**. The
section is bound by `AddOtOpcUaLocalDb` (`Host/Configuration/LocalDbRegistration.cs`) inside the `hasDriver`
branch — **admin-only nodes never register it, and never require `LocalDb:Path`**. Storage is unconditional;
replication stays inert until the sync port + peer are set.
| Key | Default | Meaning |
|---|---|---|
| `LocalDb:Path` | `./data/otopcua-localdb.db` | SQLite file path. `ValidateOnStart`-required once `AddZbLocalDb` runs (driver nodes only). In containers put it on a durable volume. |
| `LocalDb:SyncListenPort` | `0` | `0` = replication off (no sync listener). A non-zero port binds a dedicated **h2c** sync listener; set the **same** port on both pair nodes. Setting it makes the Host add an explicit Kestrel `Listen*` and re-bind the primary HTTP port (an explicit `Listen*` otherwise discards `ASPNETCORE_URLS`). |
| `LocalDb:Replication:PeerAddress` | `""` | The peer this node **dials** (`http://<peer>:<port>`). Set on the initiator only; leave empty on the passive node. The stream is bidirectional. |
| `LocalDb:Replication:ApiKey` | `""` | Bearer token for the sync stream. **Fail-closed:** no key ⇒ the passive endpoint refuses everything; a mismatch ⇒ the pair silently stops converging. **Must be byte-identical on both nodes.** Supply via `${secret:...}` / env — never a cleartext literal in production. |
| `LocalDb:Replication:MaxBatchSize` | (library default) | Rows per replication batch — **row-count-only** against gRPC's 4 MB cap. `16` for the artifact cache (chunk rows ≈171 KB, so 16 × 171 KB ≈ 2.7 MB). |
Replication is **default-OFF** across the fleet; it is enabled per-pair as an opt-in. See
[`docs/operations/2026-07-20-localdb-pair-replication.md`](operations/2026-07-20-localdb-pair-replication.md)
for the enablement runbook (ApiKey handling, stop/start-together rule, tombstone-retention window, and the
never-`sqlite3`-a-live-WAL-DB inspection rules) and the "pair-local config cache" section of
[`docs/Redundancy.md`](Redundancy.md) for what boot-from-cache does and does not cover.
---
## Environment variables
All names are read in this repo's source via `Environment.GetEnvironmentVariable(...)` unless noted otherwise. Defaults shown are the in-source fallbacks.
### Host / cluster / Config DB
| Variable | Read by | Effect / default |
|---|---|---|
| `OTOPCUA_ROLES` | `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs` (`RoleParser.Parse`) | Comma-separated cluster roles for the node (`admin`, `driver`, `dev`). Drives the conditional wiring and the per-role appsettings overlay. Used when `Cluster:Roles` is empty. |
| `OTOPCUA_CONFIG_CONNECTION` | `src/Core/ZB.MOM.WW.OtOpcUa.Configuration/DesignTimeDbContextFactory.cs` (design-time / `dotnet ef` only) | Read at **design time** by `DesignTimeDbContextFactory.cs` for `dotnet ef` migrations. At **runtime** the server resolves the connection string from `ConnectionStrings:ConfigDb` (env form: `ConnectionStrings__ConfigDb`) via `configuration.GetConnectionString("ConfigDb")` in `ServiceCollectionExtensions.cs``OTOPCUA_CONFIG_CONNECTION` appears there only as a hint in an error message, not via `GetEnvironmentVariable`. No credential is embedded in source. |
| `ASPNETCORE_ENVIRONMENT` | ASP.NET host builder (framework) | Selects `appsettings.{Environment}.json` (e.g. `Development`). |
### Historian (`ServerHistorian__ApiKey`)
The retired Wonderware sidecar's `OTOPCUA_HISTORIAN_*` environment variables are **gone** — no source reads
them anymore. The historian backend is now the external HistorianGateway, configured through the
`ServerHistorian` / `ContinuousHistorization` / `AlarmHistorian` appsettings sections (above). The single
historian secret this repo reads from the environment is the gateway API key:
| Variable | Effect / default |
|---|---|
| `ServerHistorian__ApiKey` | The HistorianGateway peppered-HMAC key (`histgw_<id>_<secret>`) sent as `Authorization: Bearer`. Supply via environment — **never commit**. Required when `ServerHistorian:Enabled=true`. |
### Driver integration-test / fixture sim endpoints
These are consumed by the driver **integration-test fixtures** (under `tests/Drivers/...IntegrationTests/`), not by the production server. Each overrides the simulator endpoint a fixture TCP-probes; defaults point at the shared Docker host `10.100.0.35` (see `CLAUDE.md` Docker Workflow).
| Variable | Read by (fixture) | Default |
|---|---|---|
| `MODBUS_SIM_ENDPOINT` | `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus.IntegrationTests/ModbusSimulatorFixture.cs` | `10.100.0.35:5020` |
| `AB_SERVER_ENDPOINT` | `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.AbCip.IntegrationTests/AbServerFixture.cs` | `10.100.0.35:44818` |
| `S7_SIM_ENDPOINT` | `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.S7.IntegrationTests/Snap7ServerFixture.cs` | `10.100.0.35:1102` (non-privileged; not S7-standard 102) |
| `OPCUA_SIM_ENDPOINT` | `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.IntegrationTests/OpcPlcFixture.cs` | `opc.tcp://10.100.0.35:50000` |
| `OTOPCUA_FOCAS_SIM_ENDPOINT` | `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.FOCAS.IntegrationTests/FocasSimFixture.cs` | `localhost:8193` |
> Additional harness/parity/soak env vars (`OTOPCUA_FOCAS_*`, `OTOPCUA_PARITY_*`, `OTOPCUA_SOAK_*`, `OTOPCUA_HARNESS_USE_SQL`) exist only in the test/parity/soak harnesses, not in production source, and are out of scope for this reference.
---
## See also
- [`security.md`](security.md) — transport security, OPC UA authentication, LDAP (`Security:Ldap`), data-plane ACLs, control-plane roles.
- [`Redundancy.md`](Redundancy.md) — the `Cluster` section in the context of warm/hot redundancy, ServiceLevel, peer discovery.
- [`Telemetry.md`](Telemetry.md) — the `Telemetry`/`TelemetryDial` gRPC live-telemetry stream: direction, dark switch, the four migrated channels + three deferrals, auth, reconnect story.
- [`ServiceHosting.md`](ServiceHosting.md) — role-based host wiring and `OTOPCUA_ROLES`.
- [`docs/drivers/Galaxy.md`](drivers/Galaxy.md) — Galaxy/MxAccess driver overview.
- [`docs/v2/config-db-schema.md`](v2/config-db-schema.md) — the full Config DB schema.