lmxopcua/docs/Configuration.md

# Configuration Reference

> **v3 in progress (Batch 1 landed — schema + RawPath identity).** The config **DB schema** was
> replaced by the greenfield v3 model: a driver-centric **Raw tree**
> (`Cluster → RawFolder → DriverInstance → Device → TagGroup → Tag`) plus a reference-only **UNS**
> projection (`UnsTagReference`). A tag's identity is now its **RawPath**
> (`<Folder/…>/<Driver>/<Device>/<TagGroup/…>/<Tag>`), not a `TagConfig.FullName` blob; the
> `Namespace`/`NamespaceKind`/`EquipmentImportBatch` entities are retired, and connection endpoints
> moved from `DriverConfig` into each `Device`'s `DeviceConfig`. The two OPC UA namespaces (`ns=Raw` /
> `ns=UNS`) and the `/raw` authoring UI arrive in later v3 batches; **Batch 1 leaves the address space
> deliberately dark** (drivers connect, but no tag variables materialize yet). The `appsettings*.json`
> surface below is unchanged. Full design: `docs/plans/2026-07-15-raw-uns-two-subtree-v3-design.md`.

This is the live configuration reference for the OtOpcUa Host (`src/Server/ZB.MOM.WW.OtOpcUa.Host/`). It enumerates the `appsettings*.json` sections, the bound Options classes, and the `OTOPCUA_*` / sim-endpoint environment variables — every entry grounded in source.

Two related concerns get their own dedicated pages and are **only summarised + linked** here, not duplicated:

- **Transport security, OPC UA authentication, LDAP, data-/control-plane authorization** → [`security.md`](security.md)
- **Redundancy + the `Cluster` section** → [`Redundancy.md`](Redundancy.md)

## How configuration is layered

The Host (`Program.cs`) loads `appsettings.json`, then overlays a **per-role** file chosen from the cluster roles:

- A single role → `appsettings.{role}.json` (e.g. `appsettings.driver.json`, `appsettings.admin.json`).
- Both roles → `appsettings.admin-driver.json` (roles joined with `-`, ordinal-sorted).
- `appsettings.{ASPNETCORE_ENVIRONMENT}.json` (e.g. `appsettings.Development.json`) is layered on by the host builder.

All role overlays are **optional** — the base `appsettings.json` plus the Options-class C# defaults are enough to boot. The roles themselves come from the `OTOPCUA_ROLES` env var (see [`ServiceHosting.md`](ServiceHosting.md) and the table below).

The checked-in `appsettings*.json` files are deliberately thin: they carry only `Serilog` and the `Security:Ldap` overlay. Everything else (`OpcUa`, `Cluster`, `ConnectionStrings`/`ConfigDb`) binds from the Options-class defaults documented below unless an operator adds the section explicitly or supplies the corresponding environment variable.

---

## `appsettings` sections

### `Serilog`

- **Purpose:** logging. Console + rolling daily file sink, layered with the shared `ZB.MOM.WW.Telemetry` enrichers (`AddZbSerilog` in `Program.cs`).
- **Where bound:** `builder.AddZbSerilog(...)` reads `Serilog` from configuration (`ReadFrom.Configuration`).
- **Checked-in shape** (`appsettings.json`): `Using` = `[ "Serilog.Sinks.Console", "Serilog.Sinks.File" ]`, `WriteTo` = a `Console` sink and a `File` sink (`path: logs/otopcua-.log`, `rollingInterval: Day`). Role overlays add `MinimumLevel` / `Override` blocks (e.g. `Opc.Ua: Debug`, `Akka: Information`).

### `OpcUa`

- **Purpose:** the OPC UA server endpoint identity, listening port, PKI, transport-security profiles, and redundancy peer advertising.
- **Options class:** `OpcUaApplicationHostOptions` — `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OpcUaApplicationHost.cs`.
- **Bound by:** `AddValidatedOptions<OpcUaApplicationHostOptions, OpcUaApplicationHostOptionsValidator>(config, "OpcUa")` in `Program.cs` (driver-role only). Validated fail-fast at startup by `OpcUaApplicationHostOptionsValidator` (`src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/OpcUaApplicationHostOptionsValidator.cs`).

| Key | Type | Default | Meaning |
|---|---|---|---|
| `ApplicationName` | string | `OtOpcUa` | Server application name. Required (validated). |
| `ApplicationUri` | string | `urn:OtOpcUa` | Server application URI. Must be unique per redundancy node. Required. |
| `ProductUri` | string | `https://zb.com/otopcua` | Product URI. Not validated. |
| `OpcUaPort` | int | `4840` | Binary endpoint listen port. Validated as a port. |
| `PublicHostname` | string | `0.0.0.0` | Hostname/IP advertised in endpoint descriptions. Required. |
| `ApplicationConfigPath` | string? | `null` | Optional path to an application config XML; loaded instead of building from defaults. |
| `PkiStoreRoot` | string | `pki` | Root of the PKI hierarchy (`own`/`issuer`/`trusted`/`rejected` substores created under it). Required. See [`security.md`](security.md). |
| `EnabledSecurityProfiles` | list of `OpcUaSecurityProfile` | `[None, Basic256Sha256Sign, Basic256Sha256SignAndEncrypt]` | Transport-security profiles, one endpoint per entry. Must contain ≥1. Profile detail in [`security.md`](security.md). |
| `AutoAcceptUntrustedClientCertificates` | bool | `false` | Auto-trust unknown client certs on first connect (dev convenience). Not validated. See [`security.md`](security.md). |
| `PeerApplicationUris` | list of string | `[]` (empty) | Partner node `ApplicationUri`s published in `Server.ServerArray` for redundancy discovery. See [`Redundancy.md`](Redundancy.md). |

> **Transport security profiles** (the values in `EnabledSecurityProfiles` — `None`, `Basic256Sha256Sign`, `Basic256Sha256SignAndEncrypt`) and the PKI trust flow are documented in full in [`security.md`](security.md). This page does not duplicate them.

### `Security`

- **Purpose:** Admin-UI and OPC UA authentication + data-plane authorization. Five subsections, each its own Options class:

| Subsection | Options class (`SectionName`) | Purpose |
|---|---|---|
| `Security:Ldap` | `LdapOptions` — `src/Server/ZB.MOM.WW.OtOpcUa.Security/Ldap/LdapOptions.cs` | LDAP bind for **Admin cookie login only**. The OPC UA data plane moved to API keys in #520. Bound by `AddValidatedOptions<LdapOptions, LdapOptionsValidator>` in `Program.cs`. |
| `Security:ApiKeys` | `ApiKeyOptions` — `src/Core/ZB.MOM.WW.OtOpcUa.Core/Authorization/ApiKeyOptions.cs` | OPC UA data-plane identity: the static `Username` every key-bearing client presents, and the server `Pepper` the stored key hashes are computed under. Bound unconditionally — a driver node needs the pepper to verify keys, an admin node to mint them. |
| `Security:Acl` | `AclOptions` — `src/Core/ZB.MOM.WW.OtOpcUa.Core/Authorization/AclOptions.cs` | Whether node ACL grants decide data-plane authorization. See below. |
| `Security:Jwt` | `JwtOptions` — `src/Server/ZB.MOM.WW.OtOpcUa.Security/Jwt/JwtOptions.cs` | Signing config for the JWT minted at `/auth/token` for **external** consumers (OPC UA clients / automation). |
| `Security:Cookie` | `OtOpcUaCookieOptions` — `src/Server/ZB.MOM.WW.OtOpcUa.Security/CookieOptions.cs` | The Admin-UI auth cookie (`AddOtOpcUaAuth` copies these onto `CookieAuthenticationOptions`). |

**`Security:ApiKeys`** key fields (`ApiKeyOptions`):

| Key | Type | Default | Meaning |
|---|---|---|---|
| `Username` | string | `apikey` | The fixed username a key-bearing client sends in its OPC UA UserName token. Carries no information — the key identifies the principal — but is checked rather than ignored so a typo is an explicit denial. |
| `Pepper` | string | `""` | Server-side pepper for the stored key hashes. **Supply via `Security__ApiKeys__Pepper` or `${secret:}` — never commit it.** No default on purpose: an empty pepper still hashes "successfully", so `ApiKeySecret.ComputeHash` throws on one rather than producing a deployment with none of the protection it believes it has. Rotating it invalidates every existing key. |

**`Security:Acl`** key fields (`AclOptions`):

| Key | Type | Default | Meaning |
|---|---|---|---|
| `Enabled` | bool | `false` | `true` makes scoped `NodeAcl` grants decide **write**, **alarm-command** and **HistoryRead** authorization, *replacing* the server-wide `WriteOperate` / `AlarmAck` role checks. `false` keeps those role checks and leaves HistoryRead ungated — byte-for-byte the pre-stage-5 behaviour. Read, Browse and Subscribe are ungated either way. |

> **Flipping `Security:Acl:Enabled` is a config change, not a redeploy** — both paths are wired in both modes (the node builds its permission trie on every deploy regardless), the same dark-switch discipline as [`MeshTransport`](#meshtransport), [`ConfigSource`](#configsource) and [`Telemetry`](#telemetry). It defaults off because the first deployment carrying the code would otherwise start denying every client that has no key, **including ScadaBridge's Data Connection Layer, which consumes this address space from another repo.** Enforcement fails closed on every unresolvable case, so a wiring or config mistake presents as *everything denied* — check the `OpcUaPublish: data-plane authority generation N installed` line for a non-zero grant count before debugging an individual rule. See [`security.md` § Node ACL enforcement](security.md#node-acl-enforcement-securityaclenabled).

**`Security:Ldap` — see [`security.md`](security.md) for the full field-by-field reference and bind-flow.** The checked-in role overlays set only `DevStubMode` and `Transport`; the remaining `LdapOptions` fields (`Enabled`, `Server`, `Port`, `AllowInsecure`, `SearchBase`, `ServiceAccountDn`, `ServiceAccountPassword`, `GroupAttribute`, `DisplayNameAttribute`, `UserNameAttribute`, `GroupToRole`) are covered there.

**`Security:Jwt`** key fields (`JwtOptions`):

| Key | Type | Default | Meaning |
|---|---|---|---|
| `SigningKey` | string | `""` | HS256 signing key; must be ≥32 bytes UTF-8. Set from your secret store — never commit a value. |
| `Issuer` | string | `otopcua` | JWT issuer. |
| `Audience` | string | `otopcua` | JWT audience. |
| `ExpiryMinutes` | int | `15` | Token lifetime. |

**`Security:Cookie`** key fields (`OtOpcUaCookieOptions`):

| Key | Type | Default | Meaning |
|---|---|---|---|
| `Name` | string | `ZB.MOM.WW.OtOpcUa.Auth` | Auth cookie name. Changing it invalidates existing sessions on next deploy. |
| `ExpiryMinutes` | int | `30` | Idle sliding-window length. |
| `RequireHttpsCookie` | bool | `true` | `SecurePolicy = Always`. Set `false` only for plain-HTTP local dev (emits a startup Warning). |

> Authentication, data-plane authorization (`NodeAcl` / `PermissionTrie`), and control-plane Admin roles are all in [`security.md`](security.md).

### `Cluster`

- **Purpose:** Akka.NET cluster identity, transport, and roles — the backbone of redundancy.
- **Options class:** `AkkaClusterOptions` (`SectionName = "Cluster"`) — `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/AkkaClusterOptions.cs`. Bound by `AddOtOpcUaCluster(config)` in `Program.cs`.

| Key | Type | Default | Meaning |
|---|---|---|---|
| `SystemName` | string | `otopcua` | Akka actor-system name. |
| `Hostname` | string | `0.0.0.0` | Bind hostname. |
| `Port` | int | `4053` | Cluster transport port. **Duplicated as `ClusterNode.AkkaPort` in the Config DB** — see the note below. |
| `PublicHostname` | string | `127.0.0.1` | Hostname advertised in cluster gossip; must be reachable by peers. **Duplicated as `ClusterNode.Host`** — see the note below. |
| `SeedNodes` | string[] | `[]` | Seed nodes for bootstrapping. **Ordered:** a node that appears in its own list must be entry 0 (only `seed-nodes[0]` can form a new cluster), or the host refuses to start — see [Redundancy.md § Bootstrap: self-first seed ordering](Redundancy.md#bootstrap-self-first-seed-ordering). **Per-pair since Phase 6:** each application `Cluster`'s pair is its own Akka mesh, so a node's `SeedNodes` lists only itself (first) and its own pair partner (second) — never a node from another cluster's pair. |
| `Roles` | string[] | `[]` | Cluster roles for this node. When empty, falls back to `OTOPCUA_ROLES` (see the note below). Allowed values: `admin`, `driver`, `dev`, plus a per-cluster `cluster-{ClusterId}` role (e.g. `cluster-MAIN`, `cluster-SITE-A`) — see [Per-cluster mesh split (Phase 6)](#per-cluster-mesh-split-phase-6-cluster-clusterid-roles--splittopologytransportvalidator) below. |

> **`Cluster:Roles` and `OTOPCUA_ROLES` are two knobs for one thing, and only one of them is the Akka role list.** `OTOPCUA_ROLES` drives all of `Program.cs`'s conditional wiring (which subsystems mount); `Cluster:Roles` becomes the node's **gossiped Akka member roles**. Since Gitea #532 `AddOtOpcUaCluster` applies the documented fallback — an empty `Cluster:Roles` takes the env var — so setting only `OTOPCUA_ROLES` now works. Before that fix it did not, silently: the node joined the mesh carrying **no roles at all**, which reads as healthy locally and as *"no driver-role nodes are Up"* at central, and every deployment to it timed out ([`deployments/wonder-app-vd03-makino-z-34184.md`](deployments/wonder-app-vd03-makino-z-34184.md)).
>
> Setting `Cluster:Roles` explicitly is still the recommendation, because the two sources normalize differently: `Cluster:Roles` binds **verbatim** while the env path is **lower-cased** by `RoleParser.Parse`. Akka compares role names ordinally, so a redundant pair configured through *different* sources with a mixed-case `cluster-{ClusterId}` (e.g. `cluster-SITE-A` vs `cluster-site-a`) would gossip two different role names and elect two singletons. Configure both nodes of a pair the same way.

> The full redundancy model (ServiceLevel tiers, split-brain, peer discovery) is in [`Redundancy.md`](Redundancy.md). The OPC UA peer-URI advertising lives in the `OpcUa:PeerApplicationUris` key above.

> **`Port` / `PublicHostname` are stored twice.** The node binds from these keys; the fleet's `ClusterNode` row records the same address as `AkkaPort` / `Host` so that **central can dial the node without sharing a gossip ring with it** — which is what [per-cluster mesh Phase 2](plans/2026-07-21-per-cluster-mesh-design.md) needs. Nothing in the schema makes the two agree, so `ClusterNodeAddressReconcilerActor` (admin-role singleton) compares them against live membership and logs an Error on mismatch. If you change `Cluster:Port` or `Cluster:PublicHostname` on a node, **update its `ClusterNode` row too** — a node binding 4054 with a row saying 4053 still gossips fine today, and fails silently in Phase 2. The row's `NodeId` is `host:port` and is also the deploy path's ack identity, so a drifted node's deployment acks stop matching as well. See [`config-db-schema.md` § `ClusterNode`](v2/config-db-schema.md#clusternode).

### Per-cluster mesh split (Phase 6): `cluster-{ClusterId}` roles + `SplitTopologyTransportValidator`

- **Purpose:** [per-cluster mesh](plans/2026-07-22-per-cluster-mesh-program.md) **Phase 6** split the
  single fleet-wide Akka mesh into one independent 2-node mesh per application `Cluster`. A node's
  `Cluster:Roles` now carries a `cluster-{ClusterId}` role alongside `driver` (and `admin` for a fused
  central node) — e.g. `driver,cluster-SITE-A` — and its `Cluster:SeedNodes` lists only itself and its
  own pair partner (see the `Roles`/`SeedNodes` rows above). Nodes in different clusters never gossip
  with each other.
- **`SplitTopologyTransportValidator`** (`src/Core/ZB.MOM.WW.OtOpcUa.Cluster/`, wired at
  `ValidateOnStart` alongside the other `Cluster`-family validators) fails host start on any node
  carrying a `cluster-{ClusterId}` role unless it is also configured for the split-safe transports:
  `MeshTransport:Mode = ClusterClient`, `Telemetry:Mode = Grpc` on a driver node, `TelemetryDial:Mode =
  Grpc` on an admin node. A DPS-mode transport on a cluster-role node would try to gossip across a
  partition boundary that no longer exists, so the validator fails closed rather than let commands or
  telemetry silently vanish. A legacy node with no `cluster-{ClusterId}` role is exempt and may stay on
  `Dps`.
- **The compiled transport-mode defaults were deliberately NOT flipped.** `MeshTransport:Mode`,
  `Telemetry:Mode`, and `TelemetryDial:Mode` (documented below) all still default to `Dps` — a
  blast-radius assessment found that flipping the compiled default would break every integration test
  and every existing `appsettings.json` profile that leaves these keys unset. `SplitTopologyTransportValidator`
  plus explicit per-deployment configuration is the guardrail, not a default flip; every node that
  actually carries a `cluster-{ClusterId}` role must set the split-safe modes explicitly.
- See [`Redundancy.md` § Per-cluster meshes (Phase 6)](Redundancy.md#per-cluster-meshes-phase-6) for the
  full runtime picture (one Primary per pair, the cluster-scoped `RedundancyStateActor` singleton,
  `ClusterNodeAddressReconciler` own-cluster-scoping, and central's one-`ClusterClient`-per-`Cluster`
  fan-out).

### `MeshTransport` (central↔node command transport)

- **Purpose:** selects how central's three command channels (`deployments`, `driver-control`, `alarm-commands`) and the node's `deployment-acks` reply reach the other side — over the mesh-wide DistributedPubSub, or over an Akka `ClusterClient` that does **not** require central and the node to share a gossip ring. The second option is what [per-cluster mesh](plans/2026-07-21-per-cluster-mesh-design.md) Phase 6 needs once the meshes split.
- **Options class:** `MeshTransportOptions` (`SectionName = "MeshTransport"`) — `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/MeshTransportOptions.cs`. Bound and validated by `AddOtOpcUaCluster(config)`; `MeshTransportOptionsValidator` runs at `ValidateOnStart`.

| Key | Type | Default | Meaning |
|---|---|---|---|
| `Mode` | string | `Dps` | `Dps` = DistributedPubSub (the transport this repo has always used). `ClusterClient` = the mesh boundary. Any other value **refuses to start**. |
| `CentralContactPoints` | string[] | `[]` | Node-side only: central node **addresses** (`akka.tcp://<system>@<host>:<port>`). Required under `ClusterClient`, ignored under `Dps`. |
| `ContactRefreshSeconds` | int | `60` | Central-side only: how often central rebuilds its ClusterClient contact set from the `ClusterNode` rows. |

**Addresses only — no actor-path suffix.** `/system/receptionist` is appended at construction time; a contact carrying `/user/` or `/system/` is rejected at startup, because a doubled suffix produces a path that resolves to nothing and drops every command in silence.

**It is a dark switch, and both sides are always wired.** The comm actors are spawned and registered with the `ClusterClientReceptionist` on every node in *both* modes, so flipping `Mode` is a config change rather than a redeploy. Under `Dps` the registration is simply idle.

**Central's contact set comes from the enabled, non-maintenance `ClusterNode` rows** — the same source Phase 1 gave the deploy ack set. A node in `MaintenanceMode` is not dialled, but note that **the contact set does not scope delivery**: a receptionist serves its whole cluster, so `SendToAll` still reaches every registered node exactly as today's DPS broadcast does. The filter is about which receptionists are worth dialling, not about who receives.

**Buffering is off by design** (`akka.cluster.client.buffer-size = 0`, set in the shipped `akka.conf`). If no receptionist is known, the command is dropped rather than queued. Akka's default of 1000 would replay a `DispatchDeployment` on reconnect — silently, on the client's own timer, at an unbounded remove from the dispatch. Do not "fix" it back to quiet a flaky test.

> **What this does *not* buy you.** It does not guarantee a node never applies a deployment the DB records as `TimedOut`. A node that missed a dispatch keeps the orphan `Applying` row the coordinator pre-created, and `DriverHostActor.Bootstrap()` replays that row on the node's next restart — live-observed applying a deployment 44 s after it was sealed `TimedOut` (see the [Phase 2 live gate](plans/2026-07-22-mesh-phase2-live-gate.md), finding B). The boot replay is the better-behaved of the two — deliberate, Warning-logged, and bounded to a restart — but it is still a replay. `buffer-size = 0` removes the *silent, timer-driven* one only.

On the docker-dev rig every node carries these keys already; flip the whole rig with `OTOPCUA_MESH_MODE=ClusterClient` at `docker compose up`.

### `ConfigSource` / `ConfigServe` (config fetch-and-cache)

- **Purpose:** [per-cluster mesh](plans/2026-07-21-per-cluster-mesh-design.md) **Phase 3**. `ConfigSource:Mode` selects where a **driver** node reads its deployed configuration: `Direct` (read `Deployment.ArtifactBlob` from central SQL, today's behaviour) or `FetchAndCache` (fetch the artifact bytes from central over a gRPC stream, verify SHA-256, cache in LocalDb, and read **only** the cache — no central-SQL config read). `ConfigServe` is the **central** (admin-role) serve side that a `FetchAndCache` node dials.
- **Options classes:** `ConfigSourceOptions` (`SectionName = "ConfigSource"`) + `ConfigServeOptions` (`SectionName = "ConfigServe"`), both in `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/`. Bound by `AddOtOpcUaCluster(config)`; `ConfigSourceOptionsValidator` runs at `ValidateOnStart`.

**`ConfigSource` (driver-side fetch):**

| Key | Type | Default | Meaning |
|---|---|---|---|
| `Mode` | string | `Direct` | `Direct` = read central SQL (unchanged). `FetchAndCache` = fetch from central over gRPC, read the LocalDb cache. Any other value **refuses to start**. |
| `CentralFetchEndpoints` | string[] | `[]` | Central artifact-gRPC base addresses, e.g. `http://central-1:4055` (h2c ⇒ the `http` scheme). Tried in order for failover. **Required** under `FetchAndCache`. |
| `ApiKey` | string | `""` | Shared bearer key; must equal central's `ConfigServe:ApiKey`. **Supply via env `ConfigSource__ApiKey` — never commit.** Required under `FetchAndCache`. |
| `FetchTimeoutSeconds` | int | `30` | Per-fetch deadline. Must be positive under `FetchAndCache`. |

**`ConfigServe` (central serve):**

| Key | Type | Default | Meaning |
|---|---|---|---|
| `GrpcListenPort` | int | `0` | Dedicated **h2c-only** Kestrel port for the artifact gRPC service. `0` = disabled (nothing bound). Must differ from the main HTTP port and from `LocalDb:SyncListenPort` (h2c cannot share a cleartext HTTP/1 port). Mapped only on admin-role nodes. |
| `ApiKey` | string | `""` | Shared bearer key the serve-side interceptor checks (constant-time, **fail-closed**: an unset key rejects every call). **Supply via env `ConfigServe__ApiKey` — never commit.** |

**It is a dark switch; the serve side is always wired.** Central maps the `DeploymentArtifactService` behind the `ConfigServeAuthInterceptor` (path-scoped, `FixedTimeEquals`, fail-closed) whenever `ConfigServe:GrpcListenPort > 0`, in both modes — so flipping a node to `FetchAndCache` is a config change, not a redeploy. A `FetchAndCache` node that cannot fetch (all endpoints down, `NotFound`, SHA mismatch) **fails the apply and keeps last-known-good** — a null fetch is "no answer," never an empty configuration (the [#485](plans/2026-07-15-v3-batch4-address-space-plan.md) contract). At boot it restores served state from the LocalDb pointer with no SQL read; an empty or unreadable cache lands Steady-with-no-revision (the first dispatch fetches), never Stale.

**The listener is dedicated h2c and coexists with the LocalDb-sync listener.** A fused admin+driver node with **both** `ConfigServe:GrpcListenPort` and `LocalDb:SyncListenPort` set re-binds its existing HTTP surface exactly once and adds both dedicated HTTP/2-only ports on top — see the same Kestrel-takeover caveat under [`LocalDb`](#localdb-pair-local-config-cache--driver-role-only).

**`RevisionHash == SHA-256(artifact bytes)` is load-bearing.** `ConfigComposer` computes the deployment's revision hash as the lowercase-hex SHA-256 of the exact artifact bytes, so the node verifies a fetch against the `RevisionHash` the dispatch already carries — no hash rides in the proto. If a future change makes the revision hash anything else, every fetch fails verification.

On the docker-dev rig the central pair carries `ConfigServe` (port `4055`, dev key) and stays `Direct`; the four site nodes carry `ConfigSource` (endpoints + matching key) defaulting to `Direct`. Flip only the site nodes with `OTOPCUA_CONFIG_MODE=FetchAndCache` at `docker compose up` — central keeps SQL and models the target topology.

**`FetchAndCache` is mandatory on a driver-only node — per-cluster mesh Phase 4.** `Program.cs` now
registers `AddOtOpcUaConfigDb` only when the node carries the `admin` role, so a **driver-only**
node (`Cluster:Roles` has `driver`, not `admin`) holds **no `ConfigDb` connection string at all** —
there is nothing for `Direct` to read. `ConfigSourceOptionsValidator` enforces this at
`ValidateOnStart`: a driver-only node whose `ConfigSource:Mode` is `Direct` **fails host start**
with an error naming the fix (`ConfigSource:Mode=FetchAndCache`). A **fused** `admin,driver` node
still owns its ConfigDb via the admin role and may stay `Direct` — the validator only fires on
`driver` without `admin`. On a driver-only node the deployed configuration, the scripted-alarm Part
9 condition state (`LocalDbAlarmConditionStateStore`, replacing the ConfigDb-backed
`EfAlarmConditionStateStore`), and the alarm store-and-forward buffer all live in the node's
LocalDb; central remains the system of record for deploy acks (`ConfigPublishCoordinator.
PersistNodeAck`) and LDAP group→role DB grants (a driver-only node maps LDAP groups to roles from
the `Security:Ldap:GroupToRole` appsettings baseline only — see `docs/Redundancy.md` and
`CLAUDE.md` §"Phase 4" for the full picture, including the client-visible ServiceLevel change).

### `Telemetry` / `TelemetryDial` (live telemetry transport)

- **Purpose:** [per-cluster mesh](plans/2026-07-21-per-cluster-mesh-design.md) **Phase 5**. Selects how
  the four live-observability channels (`alerts`, `script-logs`, `driver-health`,
  `driver-resilience-status`) reach central: `Dps` (DistributedPubSub — the four existing SignalR
  bridges, today's behaviour) or `Grpc` (a driver node hosts one gRPC server-streaming contract over a
  dedicated h2c port; central dials in and feeds the same in-process sinks). `Telemetry` is the
  **node/serve** side; `TelemetryDial` is the **central/dial** side.
- **Options classes:** `TelemetryOptions` (`SectionName = "Telemetry"`) + `TelemetryDialOptions`
  (`SectionName = "TelemetryDial"`), both in `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/`. Bound by
  `AddOtOpcUaCluster(config)`; their validators run at `ValidateOnStart`.

**`Telemetry` (node-side, serve):**

| Key | Type | Default | Meaning |
|---|---|---|---|
| `Mode` | string | `Dps` | `Dps` = no change, node just keeps publishing DPS as it always has. `Grpc` = central-ingest hint (see `TelemetryDial:Mode` below — this key does not gate anything on the node itself). Any other value **refuses to start**. |
| `GrpcListenPort` | int | `0` | Dedicated **h2c-only** Kestrel port for the telemetry stream service. `0` = disabled (nothing bound). The node hosts the server whenever this is `> 0`, **regardless of `Mode`**. |
| `ApiKey` | string | `""` | Shared bearer key the serve-side interceptor checks (constant-time, **fail-closed**: an unset key rejects every call). **Supply via env `Telemetry__ApiKey` — never commit.** |

**`TelemetryDial` (central-side, dial):**

| Key | Type | Default | Meaning |
|---|---|---|---|
| `Mode` | string | `Dps` | `Dps` = the four DPS SignalR bridges (alert, script-log, driver-status, resilience) subscribe as today. `Grpc` = those four bridges are **not spawned**; the `TelemetryDialSupervisor` actor dials every driver node instead and feeds the same sinks. Any other value **refuses to start**. |
| `ApiKey` | string | `""` | Shared bearer key; must equal the nodes' `Telemetry:ApiKey`. **Supply via env `TelemetryDial__ApiKey` — never commit.** Required under `Grpc`. |
| `ContactRefreshSeconds` | int | `60` | How often central rebuilds its dial-target set from the `ClusterNode` rows (`Host` + `GrpcPort`). |
| `CallTimeoutSeconds` | int | `30` | Per-call deadline for the streaming RPC's keepalive/health bookkeeping. |

**It is a dark switch; the node always hosts.** Unlike `ConfigServe`/`MeshTransport`, where the serve
side is gated on a role, the telemetry server binds whenever `Telemetry:GrpcListenPort > 0` — hosting
is unconditional on a driver-role node, in both modes, because the node also always emits into its
local hub and always publishes DPS. Only `TelemetryDial:Mode` on central actually changes behaviour:
which of the two already-always-available sources (DPS bridges vs. the gRPC dial supervisor) feeds the
AdminUI sinks. Flipping either side is a config change plus restart, not a redeploy — the same
discipline as `MeshTransport:Mode` and `ConfigSource:Mode`.

**A driver-role node with `Telemetry:Mode=Grpc` must set `GrpcListenPort > 0`** (nothing to serve
otherwise) — enforced by `TelemetryOptionsValidator`. **`Grpc` mode on either side requires a
non-empty `ApiKey`** — fail-closed, same as `ConfigServe`/`LocalDb:Replication`.

**Three DPS observability topics stay out of scope for this transport, on purpose:**
`redundancy-state` (pair-local, built from `Cluster.State`, stays on DPS in both modes — see
[`Redundancy.md`](Redundancy.md#command-transport-centralnode)), `fleet-status` (central-internal;
`Fleet.razor` polls the Config DB, not any stream), and `deployment-acks` (already rides the Phase 2
`ClusterClient` transport as a command-plane reply). See
[`docs/Telemetry.md`](Telemetry.md) for the full architecture, the reconnect/generation-stamped
dialer story, and the rationale for the node-hosts/central-dials inversion.

On the docker-dev rig every driver-role node carries `Telemetry:GrpcListenPort` (`4056`) + a
committed dev key, and central carries the matching `TelemetryDial:ApiKey`; both `Mode` keys stay
unset (⇒ `Dps`) until the rig is flipped with `Telemetry__Mode=Grpc` / `TelemetryDial__Mode=Grpc` at
`docker compose up`.

### `ConnectionStrings` → `ConfigDb`

- **Purpose:** the central Config DB connection string. **Required on admin-role nodes only** — `Program.cs` calls `AddOtOpcUaConfigDb` when `hasAdmin` (per-cluster mesh Phase 4). A driver-only node holds no `ConfigDb` connection string; it fetches its configuration via `ConfigSource:Mode=FetchAndCache` instead (see above). A fused `admin,driver` node still requires it.
- **Bound by:** `AddOtOpcUaConfigDb(config)` (`src/Core/ZB.MOM.WW.OtOpcUa.Configuration/ServiceCollectionExtensions.cs`). The connection-string name constant is `ConnectionStringName = "ConfigDb"`, read via `configuration.GetConnectionString("ConfigDb")`. If absent, startup throws with a message pointing to either `appsettings.json` or the `OTOPCUA_CONFIG_CONNECTION` env var.
- **Shape:** standard `ConnectionStrings:ConfigDb` SQL Server connection string. There is no checked-in default in the thin `appsettings*.json` — supply it per environment.

The Config DB itself (the EF Core `OtOpcUaConfigDbContext`, entities, draft/publish generations, `NodeAcl`, `LdapGroupRoleMapping`, migrations) is the durable home for the fleet's drivers, UNS hierarchy, ACLs, and audit log. For the **full schema** see [`docs/v2/config-db-schema.md`](v2/config-db-schema.md). This page does not duplicate it.

### Galaxy / MxAccess driver config (`DriverConfig` JSON, not `appsettings`)

The Galaxy/MxAccess connection settings are **not an `appsettings` section.** They are driver-instance options stored in the `DriverConfig` JSON column of the Config DB (edited via the Admin UI), bound to `GalaxyDriverOptions` (`src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Contracts/GalaxyDriverOptions.cs`, namespace `...Driver.Galaxy.Config`). It decomposes into nested records:

| Record | Key fields (default) | Meaning |
|---|---|---|
| `GalaxyGatewayOptions` (`Gateway`) | `Endpoint`; `ApiKeySecretRef`; `UseTls` (`true`); `CaCertificatePath` (`null`); `ConnectTimeoutSeconds` (`10`); `DefaultCallTimeoutSeconds` (`30`); `StreamTimeoutSeconds` (`0` = unlimited) | mxaccessgw gateway connection. `ApiKeySecretRef` supports `env:NAME` / `file:PATH` / `dev:KEY` / literal forms (resolved at `InitializeAsync`); prefer `env:`/`file:` in production. Never store a cleartext key. |
| `GalaxyMxAccessOptions` (`MxAccess`) | `ClientName`; `PublishingIntervalMs` (`1000`); `WriteUserId` (`0` = anonymous); `EventPumpChannelCapacity` (`50000`) | MXAccess client identity + tuning. `ClientName` **must be unique per OtOpcUa instance** (redundancy pairs enforce this). |
| `GalaxyRepositoryOptions` (`Repository`) | `DiscoverPageSize` (`5000`); `WatchDeployEvents` (`true`) | Galaxy Repository browse paging + deploy-event watching. |
| `GalaxyReconnectOptions` (`Reconnect`) | `InitialBackoffMs` (`500`); `MaxBackoffMs` (`30000`); `ReplayOnSessionLost` (`true`) | In-driver reconnect-supervisor backoff. |
| (top-level) | `ProbeTimeoutSeconds` (`30`, range 1–60) | AdminUI Test-Connect probe timeout. |

> The `OTOPCUA_GALAXY_*` environment variables that v1's in-process `Galaxy.Host` consumed **no longer live in this repo** — they moved into the separately-installed mxaccessgw gateway's own config (see the v1 archive pointer in `docs/README.md` and the Galaxy overview at [`docs/drivers/Galaxy.md`](drivers/Galaxy.md)). The only Galaxy connection secret this repo touches is the gateway API key via `ApiKeySecretRef` above.

### Historian config (HistorianGateway)

The historian backend is the external **`ZB.MOM.WW.HistorianGateway`** sidecar, consumed as the
`ZB.MOM.WW.HistorianGateway.Client` gRPC package (the retired Wonderware TCP sidecar is documented at
[`docs/drivers/Historian.Wonderware.md`](drivers/Historian.Wonderware.md)). The OtOpcUa host reads three
appsettings sections — `ServerHistorian` (read path + gateway connection), `ContinuousHistorization`
(FasterLog outbox + recorder draining to `WriteLiveValues`), and `AlarmHistorian` (the store-and-forward
alarm sink draining to `SendEvent`; its buffer lives in the node's `LocalDb:Path` database, not a
file of its own). The gateway connection (endpoint / key / TLS) lives **only** in
`ServerHistorian`; the other two sections source it from there.

The gateway API key is supplied via the environment variable **`ServerHistorian__ApiKey`** — never committed
to config. The target gateway must run `RuntimeDb:Enabled=true` + `RuntimeDb:EventReadsEnabled=true`, and the
key must carry the scopes `historian:read`, `historian:write`, `historian:tags:write`. See
[`docs/Historian.md`](Historian.md) for the full key reference, the migration note (old Wonderware keys →
gateway keys), and the deployment prerequisites.

### `LocalDb` (pair-local config cache — driver role only)

Driver-role nodes keep a consolidated **`ZB.MOM.WW.LocalDb`** SQLite database that caches the deployed
configuration artifact (chunked) so the node can **boot from cache when central SQL is unreachable**. The
section is bound by `AddOtOpcUaLocalDb` (`Host/Configuration/LocalDbRegistration.cs`) inside the `hasDriver`
branch — **admin-only nodes never register it, and never require `LocalDb:Path`**. Storage is unconditional;
replication stays inert until the sync port + peer are set.

| Key | Default | Meaning |
|---|---|---|
| `LocalDb:Path` | `./data/otopcua-localdb.db` | SQLite file path. `ValidateOnStart`-required once `AddZbLocalDb` runs (driver nodes only). In containers put it on a durable volume. |
| `LocalDb:SyncListenPort` | `0` | `0` = replication off (no sync listener). A non-zero port binds a dedicated **h2c** sync listener; set the **same** port on both pair nodes. Setting it makes the Host add an explicit Kestrel `Listen*` and re-bind the primary HTTP port (an explicit `Listen*` otherwise discards `ASPNETCORE_URLS`). |
| `LocalDb:Replication:PeerAddress` | `""` | The peer this node **dials** (`http://<peer>:<port>`). Set on the initiator only; leave empty on the passive node. The stream is bidirectional. |
| `LocalDb:Replication:ApiKey` | `""` | Bearer token for the sync stream. **Fail-closed:** no key ⇒ the passive endpoint refuses everything; a mismatch ⇒ the pair silently stops converging. **Must be byte-identical on both nodes.** Supply via `${secret:...}` / env — never a cleartext literal in production. |
| `LocalDb:Replication:MaxBatchSize` | (library default) | Rows per replication batch — **row-count-only** against gRPC's 4 MB cap. `16` for the artifact cache (chunk rows ≈171 KB, so 16 × 171 KB ≈ 2.7 MB). |

Replication is **default-OFF** across the fleet; it is enabled per-pair as an opt-in. See
[`docs/operations/2026-07-20-localdb-pair-replication.md`](operations/2026-07-20-localdb-pair-replication.md)
for the enablement runbook (ApiKey handling, stop/start-together rule, tombstone-retention window, and the
never-`sqlite3`-a-live-WAL-DB inspection rules) and the "pair-local config cache" section of
[`docs/Redundancy.md`](Redundancy.md) for what boot-from-cache does and does not cover.

---

## Environment variables

All names are read in this repo's source via `Environment.GetEnvironmentVariable(...)` unless noted otherwise. Defaults shown are the in-source fallbacks.

### Host / cluster / Config DB

| Variable | Read by | Effect / default |
|---|---|---|
| `OTOPCUA_ROLES` | `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs` (`RoleParser.Parse`) and `AddOtOpcUaCluster`'s `AkkaClusterOptions` post-configure (`RoleParser.ResolveEffectiveRoles`) | Comma-separated cluster roles for the node (`admin`, `driver`, `dev`, `cluster-{ClusterId}`). Drives the conditional wiring and the per-role appsettings overlay, and — when `Cluster:Roles` is empty — the node's Akka member roles too. See the note under `Cluster:Roles` for why the two sources are not interchangeable. |
| `OTOPCUA_CONFIG_CONNECTION` | `src/Core/ZB.MOM.WW.OtOpcUa.Configuration/DesignTimeDbContextFactory.cs` (design-time / `dotnet ef` only) | Read at **design time** by `DesignTimeDbContextFactory.cs` for `dotnet ef` migrations. At **runtime** the server resolves the connection string from `ConnectionStrings:ConfigDb` (env form: `ConnectionStrings__ConfigDb`) via `configuration.GetConnectionString("ConfigDb")` in `ServiceCollectionExtensions.cs` — `OTOPCUA_CONFIG_CONNECTION` appears there only as a hint in an error message, not via `GetEnvironmentVariable`. No credential is embedded in source. |
| `ASPNETCORE_ENVIRONMENT` | ASP.NET host builder (framework) | Selects `appsettings.{Environment}.json` (e.g. `Development`). |

### Historian (`ServerHistorian__ApiKey`)

The retired Wonderware sidecar's `OTOPCUA_HISTORIAN_*` environment variables are **gone** — no source reads
them anymore. The historian backend is now the external HistorianGateway, configured through the
`ServerHistorian` / `ContinuousHistorization` / `AlarmHistorian` appsettings sections (above). The single
historian secret this repo reads from the environment is the gateway API key:

| Variable | Effect / default |
|---|---|
| `ServerHistorian__ApiKey` | The HistorianGateway peppered-HMAC key (`histgw_<id>_<secret>`) sent as `Authorization: Bearer`. Supply via environment — **never commit**. Required when `ServerHistorian:Enabled=true`. |

### Driver integration-test / fixture sim endpoints

These are consumed by the driver **integration-test fixtures** (under `tests/Drivers/...IntegrationTests/`), not by the production server. Each overrides the simulator endpoint a fixture TCP-probes; defaults point at the shared Docker host `10.100.0.35` (see `CLAUDE.md` Docker Workflow).

| Variable | Read by (fixture) | Default |
|---|---|---|
| `MODBUS_SIM_ENDPOINT` | `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus.IntegrationTests/ModbusSimulatorFixture.cs` | `10.100.0.35:5020` |
| `AB_SERVER_ENDPOINT` | `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.AbCip.IntegrationTests/AbServerFixture.cs` | `10.100.0.35:44818` |
| `S7_SIM_ENDPOINT` | `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.S7.IntegrationTests/Snap7ServerFixture.cs` | `10.100.0.35:1102` (non-privileged; not S7-standard 102) |
| `OPCUA_SIM_ENDPOINT` | `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.IntegrationTests/OpcPlcFixture.cs` | `opc.tcp://10.100.0.35:50000` |
| `OTOPCUA_FOCAS_SIM_ENDPOINT` | `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.FOCAS.IntegrationTests/FocasSimFixture.cs` | `localhost:8193` |

> Additional harness/parity/soak env vars (`OTOPCUA_FOCAS_*`, `OTOPCUA_PARITY_*`, `OTOPCUA_SOAK_*`, `OTOPCUA_HARNESS_USE_SQL`) exist only in the test/parity/soak harnesses, not in production source, and are out of scope for this reference.

---

## See also

- [`security.md`](security.md) — transport security, OPC UA authentication, LDAP (`Security:Ldap`), data-plane ACLs, control-plane roles.
- [`Redundancy.md`](Redundancy.md) — the `Cluster` section in the context of warm/hot redundancy, ServiceLevel, peer discovery.
- [`Telemetry.md`](Telemetry.md) — the `Telemetry`/`TelemetryDial` gRPC live-telemetry stream: direction, dark switch, the four migrated channels + three deferrals, auth, reconnect story.
- [`ServiceHosting.md`](ServiceHosting.md) — role-based host wiring and `OTOPCUA_ROLES`.
- [`docs/drivers/Galaxy.md`](drivers/Galaxy.md) — Galaxy/MxAccess driver overview.
- [`docs/v2/config-db-schema.md`](v2/config-db-schema.md) — the full Config DB schema.