lmxopcua/docs/Configuration.md

# Configuration Reference

> **v3 in progress (Batch 1 landed — schema + RawPath identity).** The config **DB schema** was
> replaced by the greenfield v3 model: a driver-centric **Raw tree**
> (`Cluster → RawFolder → DriverInstance → Device → TagGroup → Tag`) plus a reference-only **UNS**
> projection (`UnsTagReference`). A tag's identity is now its **RawPath**
> (`<Folder/…>/<Driver>/<Device>/<TagGroup/…>/<Tag>`), not a `TagConfig.FullName` blob; the
> `Namespace`/`NamespaceKind`/`EquipmentImportBatch` entities are retired, and connection endpoints
> moved from `DriverConfig` into each `Device`'s `DeviceConfig`. The two OPC UA namespaces (`ns=Raw` /
> `ns=UNS`) and the `/raw` authoring UI arrive in later v3 batches; **Batch 1 leaves the address space
> deliberately dark** (drivers connect, but no tag variables materialize yet). The `appsettings*.json`
> surface below is unchanged. Full design: `docs/plans/2026-07-15-raw-uns-two-subtree-v3-design.md`.

This is the live configuration reference for the OtOpcUa Host (`src/Server/ZB.MOM.WW.OtOpcUa.Host/`). It enumerates the `appsettings*.json` sections, the bound Options classes, and the `OTOPCUA_*` / sim-endpoint environment variables — every entry grounded in source.

Two related concerns get their own dedicated pages and are **only summarised + linked** here, not duplicated:

- **Transport security, OPC UA authentication, LDAP, data-/control-plane authorization** → [`security.md`](security.md)
- **Redundancy + the `Cluster` section** → [`Redundancy.md`](Redundancy.md)

## How configuration is layered

The Host (`Program.cs`) loads `appsettings.json`, then overlays a **per-role** file chosen from the cluster roles:

- A single plain role → `appsettings.{role}.json` (e.g. `appsettings.driver.json`, `appsettings.admin.json`).
- Both roles → `appsettings.admin-driver.json` (plain roles joined with `-`, ordinal-sorted).
- `cluster-{ClusterId}` roles are **ignored** when naming the overlay (#678.6) — `[driver, cluster-SITE-A]` loads `appsettings.driver.json`.
- `appsettings.{ASPNETCORE_ENVIRONMENT}.json` (e.g. `appsettings.Development.json`) is layered on by the host builder.

All role overlays are **optional** — the base `appsettings.json` plus the Options-class C# defaults are enough to boot. The roles themselves come from `EffectiveRoles.Resolve` (`ZB.MOM.WW.OtOpcUa.Cluster`) — `Cluster:Roles`, falling back to `OTOPCUA_ROLES` when that is empty (see [`ServiceHosting.md`](ServiceHosting.md) and the table below).

The checked-in `appsettings*.json` files are deliberately thin: they carry only `Serilog` and the `Security:Ldap` overlay. Everything else (`OpcUa`, `Cluster`, `ConnectionStrings`/`ConfigDb`) binds from the Options-class defaults documented below unless an operator adds the section explicitly or supplies the corresponding environment variable.

<a id="role-overlay-audit"></a>
#### Role overlay audit (#678.6)

Until #678.6 the overlay never loaded on a node carrying a `cluster-*` role (the cluster role leaked into the filename). Fixing that **activates** the shipped overlays on those nodes (the docker-dev rig and any per-cluster-mesh production node), so each key was audited against base `appsettings.json` / the Options defaults. Env vars and command-line args are re-appended after the overlay, so any value a deployment sets explicitly (the rig sets every `Security__Ldap__*` key) still wins.

| Key | Overlay(s) | Base / default | Effect on activation |
|---|---|---|---|
| `Serilog:MinimumLevel:Default` = `Information` | all three | Serilog default `Information` | no-op (in `Development` it outranks `appsettings.Development.json`'s `Debug`, by design) |
| `…Override:Akka` = `Information` | all three | inherits `Default` (`Information`) | no-op |
| `…Override:Microsoft.AspNetCore` = `Warning` | `admin`, `admin-driver` | unset (`Information`) | quieter ASP.NET Core logs; log-level only, matches what the rig already pins |
| `…Override:Opc.Ua` = `Information` | `admin-driver`, `driver` | inherits `Default` | no-op. **`driver` shipped `Debug` — changed to `Information` in #678.6**: activating a Debug OPC UA stack trace on every production driver node would flood Serilog (the same pipeline starvation the rig's EF/AspNetCore pins guard against). Raise it per node with `Serilog__MinimumLevel__Override__Opc.Ua=Debug`. |
| `Security:Ldap:DevStubMode` = `false` | all three | `false` | no-op; the overlay's point is that it outranks a developer's `appsettings.Development.json` (`true`) |
| `Security:Ldap:Transport` = `Ldaps` | all three | `LdapOptions.Transport` default `None` | **behaviour-changing, kept deliberately** (documented secure production default). A cluster-role node that configures `Security:Ldap:Server`/`Port` for a plaintext LDAP but never sets `Transport` would now dial LDAPS and fail AdminUI sign-in; set `Security__Ldap__Transport` (the rig already sets `None` explicitly). Data-plane auth is API-key based and unaffected; driver-only nodes have no AdminUI. |


---

## `appsettings` sections

### `Serilog`

- **Purpose:** logging. Console + rolling daily file sink, layered with the shared `ZB.MOM.WW.Telemetry` enrichers (`AddZbSerilog` in `Program.cs`).
- **Where bound:** `builder.AddZbSerilog(...)` reads `Serilog` from configuration (`ReadFrom.Configuration`).
- **Checked-in shape** (`appsettings.json`): `Using` = `[ "Serilog.Sinks.Console", "Serilog.Sinks.File" ]`, `WriteTo` = a `Console` sink and a `File` sink (`path: logs/otopcua-.log`, `rollingInterval: Day`). Role overlays add `MinimumLevel` / `Override` blocks (e.g. `Opc.Ua: Information`, `Akka: Information`).

### `OpcUa`

- **Purpose:** the OPC UA server endpoint identity, listening port, PKI, transport-security profiles, and redundancy peer advertising.
- **Options class:** `OpcUaApplicationHostOptions` — `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OpcUaApplicationHost.cs`.
- **Bound by:** `AddValidatedOptions<OpcUaApplicationHostOptions, OpcUaApplicationHostOptionsValidator>(config, "OpcUa")` in `Program.cs` (driver-role only). Validated fail-fast at startup by `OpcUaApplicationHostOptionsValidator` (`src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/OpcUaApplicationHostOptionsValidator.cs`).

| Key | Type | Default | Meaning |
|---|---|---|---|
| `ApplicationName` | string | `OtOpcUa` | Server application name. Required (validated). |
| `ApplicationUri` | string | `urn:OtOpcUa` | Server application URI. Must be unique per redundancy node. Required. |
| `ProductUri` | string | `https://zb.com/otopcua` | Product URI. Not validated. |
| `OpcUaPort` | int | `4840` | Binary endpoint listen port. Validated as a port. |
| `PublicHostname` | string | `0.0.0.0` | Hostname/IP advertised in endpoint descriptions. Required. |
| `ApplicationConfigPath` | string? | `null` | Optional path to an application config XML; loaded instead of building from defaults. |
| `PkiStoreRoot` | string | `pki` | Root of the PKI hierarchy (`own`/`issuer`/`trusted`/`rejected` substores created under it). Required. See [`security.md`](security.md). |
| `EnabledSecurityProfiles` | list of `OpcUaSecurityProfile` | `[None, Basic256Sha256Sign, Basic256Sha256SignAndEncrypt]` | Transport-security profiles, one endpoint per entry. Must contain ≥1. Profile detail in [`security.md`](security.md). |
| `AutoAcceptUntrustedClientCertificates` | bool | `false` | Auto-trust unknown client certs on first connect (dev convenience). Not validated. See [`security.md`](security.md). |
| `PeerApplicationUris` | list of string | `[]` (empty) | Partner node `ApplicationUri`s published in `Server.ServerArray` for redundancy discovery. See [`Redundancy.md`](Redundancy.md). |

> **Transport security profiles** (the values in `EnabledSecurityProfiles` — `None`, `Basic256Sha256Sign`, `Basic256Sha256SignAndEncrypt`) and the PKI trust flow are documented in full in [`security.md`](security.md). This page does not duplicate them.

### `Security`

- **Purpose:** Admin-UI and OPC UA authentication + data-plane authorization. Five subsections, each its own Options class:

| Subsection | Options class (`SectionName`) | Purpose |
|---|---|---|
| `Security:Ldap` | `LdapOptions` — `src/Server/ZB.MOM.WW.OtOpcUa.Security/Ldap/LdapOptions.cs` | LDAP bind for **Admin cookie login only**. The OPC UA data plane moved to API keys in #520. Bound by `AddValidatedOptions<LdapOptions, LdapOptionsValidator>` in `Program.cs`. |
| `Security:ApiKeys` | `ApiKeyOptions` — `src/Core/ZB.MOM.WW.OtOpcUa.Core/Authorization/ApiKeyOptions.cs` | OPC UA data-plane identity: the static `Username` every key-bearing client presents, and the server `Pepper` the stored key hashes are computed under. Bound unconditionally — a driver node needs the pepper to verify keys, an admin node to mint them. |
| `Security:Acl` | `AclOptions` — `src/Core/ZB.MOM.WW.OtOpcUa.Core/Authorization/AclOptions.cs` | Whether node ACL grants decide data-plane authorization. See below. |
| `Security:Jwt` | `JwtOptions` — `src/Server/ZB.MOM.WW.OtOpcUa.Security/Jwt/JwtOptions.cs` | Signing config for the JWT minted at `/auth/token` for **external** consumers (OPC UA clients / automation). |
| `Security:Cookie` | `OtOpcUaCookieOptions` — `src/Server/ZB.MOM.WW.OtOpcUa.Security/CookieOptions.cs` | The Admin-UI auth cookie (`AddOtOpcUaAuth` copies these onto `CookieAuthenticationOptions`). |

**`Security:ApiKeys`** key fields (`ApiKeyOptions`):

| Key | Type | Default | Meaning |
|---|---|---|---|
| `Username` | string | `apikey` | The fixed username a key-bearing client sends in its OPC UA UserName token. Carries no information — the key identifies the principal — but is checked rather than ignored so a typo is an explicit denial. |
| `Pepper` | string | `""` | Server-side pepper for the stored key hashes. **Supply via `Security__ApiKeys__Pepper` or `${secret:}` — never commit it.** No default on purpose: an empty pepper still hashes "successfully", so `ApiKeySecret.ComputeHash` throws on one rather than producing a deployment with none of the protection it believes it has. Rotating it invalidates every existing key. |
| `StoreTimeoutMs` | int | `2000` | Hard per-call deadline on the API-key store lookup that runs inline on session activation (arch-review 03/S18). Exceeding it **denies** the activation, logs a distinct Warning and meters `otopcua.opcua.apikey_store.bounded{outcome=timeout}`. `≤ 0` disables the deadline — do not, on a node with a ConfigDb: the SDK holds a `SessionManager`-wide lock across that lookup, so an unanswered query stalls **every** session activation on the server. Applies only to `ConfigDbApiKeyStore`; the in-memory artifact store on a driver-only node is not wrapped. |
| `StoreCacheSeconds` | int | `5` | TTL of the read-through cache in front of that store, so a burst of activations costs one round trip. `0` disables it (a store read per activation). Bounds how long an `Enabled` revocation authored in the ConfigDb takes to reach a **new** activation; it has no effect on already-open sessions (those re-check the artifact snapshot) and cannot delay an `ExpiresUtc` — a cached hit that has since expired is re-read rather than served. |

**`Security:Acl`** key fields (`AclOptions`):

| Key | Type | Default | Meaning |
|---|---|---|---|
| `Enabled` | bool | `false` | `true` makes scoped `NodeAcl` grants decide **write**, **alarm-command** and **HistoryRead** authorization, *replacing* the server-wide `WriteOperate` / `AlarmAck` role checks. `false` keeps those role checks and leaves HistoryRead ungated — byte-for-byte the pre-stage-5 behaviour. Read, Browse and Subscribe are ungated either way. |
| `RevalidationIntervalSeconds` | int | `60` | How often live monitored items are re-authorized against the clock, in addition to on every deployment (arch-review 03/S19). A key's `ExpiresUtc` passes with the clock, so without this an expired key kept feeding a live subscription until a deployment happened to land — indefinitely on a fleet deployed once and left running. `0` disables the timer (deploy-triggered only). **Inert while `Enabled` is false**, since the pass early-returns when enforcement is off. An affected session is silenced with `BadUserAccessDenied`, **not closed**, and recovers on re-activation. |

> **Flipping `Security:Acl:Enabled` is a config change, not a redeploy** — both paths are wired in both modes (the node builds its permission trie on every deploy regardless), the same dark-switch discipline as [`MeshTransport`](#meshtransport), [`ConfigSource`](#configsource) and [`Telemetry`](#telemetry). It defaults off because the first deployment carrying the code would otherwise start denying every client that has no key, **including ScadaBridge's Data Connection Layer, which consumes this address space from another repo.** Enforcement fails closed on every unresolvable case, so a wiring or config mistake presents as *everything denied* — check the `OpcUaPublish: data-plane authority generation N installed` line for a non-zero grant count before debugging an individual rule. See [`security.md` § Node ACL enforcement](security.md#node-acl-enforcement-securityaclenabled).

**`Security:Ldap` — see [`security.md`](security.md) for the full field-by-field reference and bind-flow.** The checked-in role overlays set only `DevStubMode` and `Transport`; the remaining `LdapOptions` fields (`Enabled`, `Server`, `Port`, `AllowInsecure`, `SearchBase`, `ServiceAccountDn`, `ServiceAccountPassword`, `GroupAttribute`, `DisplayNameAttribute`, `UserNameAttribute`, `GroupToRole`) are covered there.

**`Security:Jwt`** key fields (`JwtOptions`):

| Key | Type | Default | Meaning |
|---|---|---|---|
| `SigningKey` | string | `""` | HS256 signing key; must be ≥32 bytes UTF-8. Set from your secret store — never commit a value. |
| `Issuer` | string | `otopcua` | JWT issuer. |
| `Audience` | string | `otopcua` | JWT audience. |
| `ExpiryMinutes` | int | `15` | Token lifetime. |

**`Security:Cookie`** key fields (`OtOpcUaCookieOptions`):

| Key | Type | Default | Meaning |
|---|---|---|---|
| `Name` | string | `ZB.MOM.WW.OtOpcUa.Auth` | Auth cookie name. Changing it invalidates existing sessions on next deploy. |
| `ExpiryMinutes` | int | `30` | Idle sliding-window length. |
| `RequireHttpsCookie` | bool | `true` | `SecurePolicy = Always`. Set `false` only for plain-HTTP local dev (emits a startup Warning). |

> Authentication, data-plane authorization (`NodeAcl` / `PermissionTrie`), and control-plane Admin roles are all in [`security.md`](security.md).

### `Cluster`

- **Purpose:** Akka.NET cluster identity, transport, and roles — the backbone of redundancy.
- **Options class:** `AkkaClusterOptions` (`SectionName = "Cluster"`) — `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/AkkaClusterOptions.cs`. Bound by `AddOtOpcUaCluster(config)` in `Program.cs`.

| Key | Type | Default | Meaning |
|---|---|---|---|
| `SystemName` | string | `otopcua` | Akka actor-system name. |
| `Hostname` | string | `0.0.0.0` | Bind hostname. |
| `Port` | int | `4053` | Cluster transport port. **Duplicated as `ClusterNode.AkkaPort` in the Config DB** — see the note below. |
| `PublicHostname` | string | `127.0.0.1` | Hostname advertised in cluster gossip; must be reachable by peers. **Duplicated as `ClusterNode.Host`** — see the note below. |
| `SeedNodes` | string[] | `[]` | Seed nodes for bootstrapping. **Ordered:** a node that appears in its own list must be entry 0 (only `seed-nodes[0]` can form a new cluster), or the host refuses to start — see [Redundancy.md § Bootstrap: self-first seed ordering](Redundancy.md#bootstrap-self-first-seed-ordering). **Per-pair since Phase 6:** each application `Cluster`'s pair is its own Akka mesh, so a node's `SeedNodes` lists only itself (first) and its own pair partner (second) — never a node from another cluster's pair. |
| `Roles` | string[] | `[]` | Cluster roles for this node. When empty, falls back to `OTOPCUA_ROLES` (see the note below). Values are trimmed and (for a plain role) lower-cased; a cluster role's ClusterId suffix keeps the operator's spelling. An unknown role from either source **fails host start**. Allowed values: `admin`, `driver`, `dev`, plus a per-cluster `cluster-{ClusterId}` role (e.g. `cluster-MAIN`, `cluster-SITE-A`) — see [Per-cluster mesh split (Phase 6)](#per-cluster-mesh-split-phase-6-cluster-clusterid-roles--splittopologytransportvalidator) below. |

> **One role authority, one precedence, both consumers (archreview R3-04).** `EffectiveRoles.Resolve` (`ZB.MOM.WW.OtOpcUa.Cluster`) is the SINGLE place a node's roles are decided: `Cluster:Roles` wins outright when non-empty, else `OTOPCUA_ROLES`, else empty. Both `Program.cs`'s conditional host wiring (which subsystems mount) AND the Akka `AkkaClusterOptions.Roles` gossiped to the cluster are the SAME `EffectiveRoles` instance's `Gossiped` list — not two separate derivations that happen to usually agree. The boot banner (`OtOpcUa.Host starting with roles=[...] source=...`) names which source won. Setting both sources to *different* normalized role sets logs a Warning at boot (still resolves from `Cluster:Roles`) — almost certainly not what an operator intended.
>
> Before R3-04, `Program.cs` read `OTOPCUA_ROLES` only and never looked at `Cluster:Roles` at all — a node configured solely through `Cluster:Roles` (the shape Gitea #532's fix made viable on the Akka side) joined the mesh gossiping the right roles while `Program.cs`'s own conditional wiring mounted NONE of them, silently. See [`deployments/wonder-app-vd03-makino-z-34184.md`](deployments/wonder-app-vd03-makino-z-34184.md) for the inverse (env-only, pre-#532) incident this family of defect belongs to.
>
> **A role overlay may not itself set `Cluster:Roles`.** `Program.cs` resolves `EffectiveRoles` off the *pre-overlay* configuration — it needs the answer to pick the overlay's filename — so a role overlay (or a `${secret:}` expansion) that changed `Cluster:Roles` would leave the bound options disagreeing with every consumer holding the registered instance. `AkkaClusterOptionsValidator` fails host start if this ever happens, naming both views.

> The full redundancy model (ServiceLevel tiers, split-brain, peer discovery) is in [`Redundancy.md`](Redundancy.md). The OPC UA peer-URI advertising lives in the `OpcUa:PeerApplicationUris` key above.

> **`Port` / `PublicHostname` are stored twice.** The node binds from these keys; the fleet's `ClusterNode` row records the same address as `AkkaPort` / `Host` so that **central can dial the node without sharing a gossip ring with it** — which is what [per-cluster mesh Phase 2](plans/2026-07-21-per-cluster-mesh-design.md) needs. Nothing in the schema makes the two agree, so `ClusterNodeAddressReconcilerActor` (admin-role singleton) compares them against live membership and logs an Error on mismatch. If you change `Cluster:Port` or `Cluster:PublicHostname` on a node, **update its `ClusterNode` row too** — a node binding 4054 with a row saying 4053 still gossips fine today, and fails silently in Phase 2. The row's `NodeId` is `host:port` and is also the deploy path's ack identity, so a drifted node's deployment acks stop matching as well. See [`config-db-schema.md` § `ClusterNode`](v2/config-db-schema.md#clusternode).

### Per-cluster mesh split (Phase 6): `cluster-{ClusterId}` roles + `SplitTopologyTransportValidator`

- **Purpose:** [per-cluster mesh](plans/2026-07-22-per-cluster-mesh-program.md) **Phase 6** split the
  single fleet-wide Akka mesh into one independent 2-node mesh per application `Cluster`. A node's
  `Cluster:Roles` now carries a `cluster-{ClusterId}` role alongside `driver` (and `admin` for a fused
  central node) — e.g. `driver,cluster-SITE-A` — and its `Cluster:SeedNodes` lists only itself and its
  own pair partner (see the `Roles`/`SeedNodes` rows above). Nodes in different clusters never gossip
  with each other.
- **`SplitTopologyTransportValidator`** (`src/Core/ZB.MOM.WW.OtOpcUa.Cluster/`, wired at
  `ValidateOnStart` alongside the other `Cluster`-family validators) fails host start on any node
  carrying a `cluster-{ClusterId}` role unless it is also configured for the split-safe transports:
  `MeshTransport:Mode = Grpc`, `Telemetry:Mode = Grpc` on a driver node, `TelemetryDial:Mode =
  Grpc` on an admin node. A DPS-mode transport on a cluster-role node would try to gossip across a
  partition boundary that no longer exists, so the validator fails closed rather than let commands or
  telemetry silently vanish. A legacy node with no `cluster-{ClusterId}` role is exempt and may stay on
  `Dps`.
- **The compiled transport-mode defaults were deliberately NOT flipped.** `MeshTransport:Mode`,
  `Telemetry:Mode`, and `TelemetryDial:Mode` (documented below) all still default to `Dps` — a
  blast-radius assessment found that flipping the compiled default would break every integration test
  and every existing `appsettings.json` profile that leaves these keys unset. `SplitTopologyTransportValidator`
  plus explicit per-deployment configuration is the guardrail, not a default flip; every node that
  actually carries a `cluster-{ClusterId}` role must set the split-safe modes explicitly.
- See [`Redundancy.md` § Per-cluster meshes (Phase 6)](Redundancy.md#per-cluster-meshes-phase-6) for the
  full runtime picture (one Primary per pair, the cluster-scoped `RedundancyStateActor` singleton,
  `ClusterNodeAddressReconciler` own-cluster-scoping, and central's per-node gRPC command dial —
  one `NodeCommandService.Execute` call per enabled `ClusterNode` row carrying a `GrpcPort`).

### `MeshTransport` (central↔node command transport)

- **Purpose:** selects how central's three command channels (`deployments`, `driver-control`, `alarm-commands`) and the node's `deployment-acks` reply reach the other side — over the mesh-wide DistributedPubSub (pair-local only), or over unary gRPC calls that do **not** require central and the node to share a gossip ring. The second option is what [per-cluster mesh](plans/2026-07-21-per-cluster-mesh-design.md) Phase 6 needs, now that the meshes are split. The Akka `ClusterClient` transport that used to be the second option here is **deleted** ([ClusterClient→gRPC migration](plans/2026-08-06-clusterclient-to-grpc-design.md), Phase 3) — `Grpc` is its replacement, not a third option alongside it.
- **Options class:** `MeshTransportOptions` (`SectionName = "MeshTransport"`) — `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/MeshTransportOptions.cs`. Bound and validated by `AddOtOpcUaCluster(config)`; `MeshTransportOptionsValidator` runs at `ValidateOnStart`.

| Key | Type | Default | Meaning |
|---|---|---|---|
| `Mode` | string | `Dps` | `Dps` = DistributedPubSub, pair-local only — cannot cross an application-`Cluster` boundary. `Grpc` = the `mesh_transport.v1` gRPC contract (below), required on any node carrying a `cluster-{ClusterId}` role. Any other value **refuses to start**, including the retired `ClusterClient` value — a node still configured for it is refused at boot rather than silently demoted to `Dps`, which on a split-topology node would deliver nothing at all. |
| `ContactRefreshSeconds` | int | `60` | Central-side only: how often central rebuilds its node dispatch set — the gRPC dial targets and the known-cluster PSK gate — from the `ClusterNode` rows. Runs in every mode; under `Dps` it still keeps the inbound-ack endpoint's known-cluster gate current even though nothing dials out. While the known-cluster gate is still unpublished (PSK provider not yet resolvable at boot, or the first `ClusterNode` load failed) central also retries a refresh every 2 s, at most 15 times, until the first publish (#654); steady state adds no traffic. Load failures and an unresolvable provider share the one 15-retry (~30 s) budget; after it is spent, healing falls to the periodic refresh. |

**`Grpc` mode: central dials each node, a node posts its ack to central.** `CentralCommunicationActor` fans one `NodeCommandService.Execute` call per node — the target set is every enabled, non-maintenance `ClusterNode` row that carries a `GrpcPort` (dialled at `Host`+`GrpcPort`), rebuilt every `ContactRefreshSeconds`. There is no receptionist and no single fleet-wide (or per-cluster) client object: it is a plain per-node dial. A node posts its `ApplyAck` with `GrpcApplyAckSender`, which walks `CentralGrpcEndpoints` in order and stops at the first success — an availability list, not a fan-out list, so central normally sees one ack per node per dispatch (the walk as a whole is at-least-once on a slow-then-retried endpoint; `ConfigPublishCoordinator.HandleAck` absorbs a duplicate idempotently). Both directions are **fire-and-forget, with no retry**: a failed dispatch is logged (naming the node and the gRPC status) and dropped in the dispatch second — the deploy path's 2-minute apply deadline and a node's boot-time replay (`DriverHostActor.Bootstrap()` re-applying an orphan `Applying` row) are the only recovery mechanisms, exactly as they were under `ClusterClient`.

The ack itself is sent only once the node's address-space apply outcome is known, and carries an additive `degraded` flag (`ApplyAckMessage` fields 6/7; trailing `ApplyAck.Degraded`/`DegradedReason` on DPS) — an old node's ack decodes clean, exactly as before. See [Redundancy.md § "What an `Applied` ack means"](Redundancy.md#what-an-applied-ack-means-the-degraded-apply-amendment-gitea-601) (Gitea #601).

| Key | Type | Default | Meaning |
|---|---|---|---|
| `GrpcPsk` | string? | `null` (unset) | Node-side only, required under `Grpc` on a driver-role node: this node's pre-shared key for its own `Cluster`, presented as the bearer on outbound acks and required on inbound `NodeCommandService` calls. Production form is `${secret:OT-MESH-PSK-<clusterId>}`; supply a literal only on a rig, via `MeshTransport__GrpcPsk` — never commit one. |
| `CentralGrpcEndpoints` | string[] | `[]` | Node-side only, required under `Grpc` on a driver-role node: full gRPC base URLs (`scheme://host:port`, e.g. `http://central-1:4055` for h2c) of the central pair's `CentralCommandService`, tried in order, first success wins. There is no Akka-side counterpart any more — the old `MeshTransport:CentralContactPoints` (the `ClusterClient` contact-point list) is deleted along with `ClusterClient` itself, not merely superseded. |
| `DispatchTimeoutSeconds` | int | `10` | Central-side per-node deadline for one `NodeCommandService.Execute` call. Must be positive under `Grpc`; validated on driver-role nodes (covers the central pair, which is always fused). |
| `AckTimeoutSeconds` | int | `10` | Node-side per-endpoint deadline for one `CentralCommandService.ReportApply` call. Must be positive under `Grpc`. Applied per endpoint walked in `CentralGrpcEndpoints`, not per ack, so a dead first endpoint costs this deadline before the second is tried. |
| `ClusterPsks` | `Dictionary<string,string>` | `{}` (case-insensitive keys) | **Central-side only** override map, keyed by `Cluster` id: `IMeshPskProvider` consults this map *before* the Secrets store when central builds a channel to (or verifies an ack from) a given cluster. Production leaves this empty and resolves `OT-MESH-PSK-{clusterId}` from Secrets instead; the map exists so the docker-dev rig and integration tests can supply literals via the environment (`MeshTransport__ClusterPsks__SITE-A`) without standing up a KEK. |

**Auth is per-cluster PSK bearer, both directions, fail-closed.** `GrpcPsk` (node-side) and `ClusterPsks` (central-side) are two ends of the same key, one per application `Cluster` — not a fleet-wide shared bearer, because this channel carries `RestartDriver`, `AlarmCommand` and deployment dispatch, and a single fleet key would let one compromised site node command every other site or forge acks fleet-wide. Both `MeshCommandAuthInterceptor` (node-side, gates `NodeCommandService`) and `CentralCommandAuthInterceptor` (central-side, gates `CentralCommandService`) reject every call when no key is configured for the relevant cluster, compare in constant time, and never log key material. **PSK seeding:** production form is a secret reference, `${secret:OT-MESH-PSK-<clusterId>}` (e.g. `${secret:OT-MESH-PSK-SITE-A}`), resolved by the Secrets store so it works on a driver-only node with no ConfigDb; the docker-dev rig instead supplies three literal dev keys via environment — `dev-mesh-psk-MAIN`, `dev-mesh-psk-SITE-A`, `dev-mesh-psk-SITE-B` — one per cluster, node-side via `MeshTransport__GrpcPsk` and central-side via `MeshTransport__ClusterPsks__<clusterId>`. Never commit a literal key.

**`ClusterPsks` / `OT-MESH-PSK-<clusterId>` and `SecretName`'s allow-list.** `SecretName` (`ZB.MOM.WW.Secrets.Abstractions`) lowercases the secret name and restricts it to `[a-z0-9._/-]` — a `Cluster` id that carries any character outside that set cannot be expressed as `OT-MESH-PSK-<clusterId>` at all, so `MeshPskProvider.TryGetPsk` fails closed for that cluster on every call that reaches the store (`SecretName`'s constructor throws, `MeshPskProvider.ResolveFromStore` catches it, logs a warning, and returns no key) — `MeshTransport:ClusterPsks` is the only path that still works for such an id. The rig's three ids (`MAIN`, `SITE-A`, `SITE-B`) are all within the allow-list and resolve normally either way.

**Both command-plane gRPC services ride an existing listener — neither claims a port of its own.** `NodeCommandService` is mapped alongside `TelemetryStreamGrpcService` on the driver-role node's `Telemetry:GrpcListenPort` listener (the docker-dev rig's `:4056`) whenever that port is bound, deliberately NOT gated on `MeshTransport:Mode` — mapping it under `Dps` costs nothing (nothing dials it), whereas gating it would mean a running fleet needs a central restart before any node could ack after flipping the mode. `CentralCommandService` (the node→central ack ingress) is mapped alongside `DeploymentArtifactService` on the admin-role node's `ConfigServe:GrpcListenPort` listener (the rig's `:4055`), for the same reason — see [`docs/Telemetry.md`](Telemetry.md) for the shared driver-role listener and [§ `ConfigSource`/`ConfigServe`](#configsource--configserve-config-fetch-and-cache) for the shared admin-role one.

**The transport underneath is cleartext h2c, deliberately.** Both command-plane listeners are HTTP/2-only on a plaintext port and every dial is a prior-knowledge `http://host:port` address, so the PSK bearer crosses the wire in the clear on a link assumed private. That posture is **recorded and accepted** for these two planes and the three others that share their shape — see [`security.md` § Internal gRPC planes (h2c)](security.md#internal-grpc-planes-h2c--accepted-cleartext-posture) for the full five-plane table, what each one exposes, and why a TLS option would have to cover all five at once.

On the docker-dev rig every node carries these keys already; the rig's compose default is `Grpc` (`OTOPCUA_MESH_MODE:-Grpc`, cutover commit `caf2eec6`) — the compiled class default in `MeshTransportOptions` stays `Dps`, so an unconfigured deployment outside the rig still needs `Mode=Grpc` set explicitly. Every rig node carries a `cluster-{ClusterId}` role (Phase 6), so `OTOPCUA_MESH_MODE=Dps` is not a usable override on this rig as shipped — `SplitTopologyTransportValidator` refuses to boot any of them on `Dps`.

See the [ClusterClient→gRPC migration design](plans/2026-08-06-clusterclient-to-grpc-design.md), [implementation plan](plans/2026-08-06-clusterclient-to-grpc.md), [tracking doc](plans/2026-08-06-clusterclient-to-grpc-tracking.md) and [live gate](plans/2026-08-06-clusterclient-to-grpc-live-gate.md) for the migration's phase-by-phase history — those are historical records of the cutover and are not updated further by this sweep.

**On an admin-role node, `NodeCommandService` and `CentralCommandService` are `AllowAnonymous` by design** — the AdminUI cookie `FallbackPolicy` would otherwise 401/redirect a correctly-PSK'd call before the interceptor above ever ran (archreview 03/S14). See [`security.md` § Mesh gRPC ingress is `AllowAnonymous` by design](security.md#mesh-grpc-ingress-is-allowanonymous-by-design).

### `ConfigSource` / `ConfigServe` (config fetch-and-cache)

- **Purpose:** [per-cluster mesh](plans/2026-07-21-per-cluster-mesh-design.md) **Phase 3**. `ConfigSource:Mode` selects where a **driver** node reads its deployed configuration: `Direct` (read `Deployment.ArtifactBlob` from central SQL, today's behaviour) or `FetchAndCache` (fetch the artifact bytes from central over a gRPC stream, verify SHA-256, cache in LocalDb, and read **only** the cache — no central-SQL config read). `ConfigServe` is the **central** (admin-role) serve side that a `FetchAndCache` node dials.
- **Options classes:** `ConfigSourceOptions` (`SectionName = "ConfigSource"`) + `ConfigServeOptions` (`SectionName = "ConfigServe"`), both in `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/`. Bound by `AddOtOpcUaCluster(config)`; `ConfigSourceOptionsValidator` runs at `ValidateOnStart`.

**`ConfigSource` (driver-side fetch):**

| Key | Type | Default | Meaning |
|---|---|---|---|
| `Mode` | string | `Direct` | `Direct` = read central SQL (unchanged). `FetchAndCache` = fetch from central over gRPC, read the LocalDb cache. Any other value **refuses to start**. |
| `CentralFetchEndpoints` | string[] | `[]` | Central artifact-gRPC base addresses, e.g. `http://central-1:4055` (h2c ⇒ the `http` scheme). Tried in order for failover. **Required** under `FetchAndCache`. |
| `ApiKey` | string | `""` | Shared bearer key; must equal central's `ConfigServe:ApiKey`. **Supply via env `ConfigSource__ApiKey` — never commit.** Required under `FetchAndCache`. |
| `FetchTimeoutSeconds` | int | `30` | Per-fetch deadline. Must be positive under `FetchAndCache`. |

**`ConfigServe` (central serve):**

| Key | Type | Default | Meaning |
|---|---|---|---|
| `GrpcListenPort` | int | `0` | Dedicated **h2c-only** Kestrel port for the artifact gRPC service. `0` = disabled (nothing bound). Must differ from the main HTTP port and from `LocalDb:SyncListenPort` (h2c cannot share a cleartext HTTP/1 port). Mapped only on admin-role nodes. |
| `ApiKey` | string | `""` | Shared bearer key the serve-side interceptor checks (constant-time, **fail-closed**: an unset key rejects every call). **Supply via env `ConfigServe__ApiKey` — never commit.** |

**It is a dark switch; the serve side is always wired.** Central maps the `DeploymentArtifactService` behind the `ConfigServeAuthInterceptor` (path-scoped, `FixedTimeEquals`, fail-closed) whenever `ConfigServe:GrpcListenPort > 0`, in both modes — so flipping a node to `FetchAndCache` is a config change, not a redeploy. A `FetchAndCache` node that cannot fetch (all endpoints down, `NotFound`, SHA mismatch) **fails the apply and keeps last-known-good** — a null fetch is "no answer," never an empty configuration (the [#485](plans/2026-07-15-v3-batch4-address-space-plan.md) contract). At boot it restores served state from the LocalDb pointer with no SQL read; an empty or unreadable cache — including a cached row whose bytes are malformed (review R1 m4) — lands Steady-with-no-revision (the first dispatch fetches, even of the same revision), never Stale.

**The listener is dedicated h2c and coexists with the LocalDb-sync listener.** A fused admin+driver node with **both** `ConfigServe:GrpcListenPort` and `LocalDb:SyncListenPort` set re-binds its existing HTTP surface exactly once and adds both dedicated HTTP/2-only ports on top — see the same Kestrel-takeover caveat under [`LocalDb`](#localdb-pair-local-config-cache--driver-role-only).

**`RevisionHash == SHA-256(artifact bytes)` is load-bearing.** `ConfigComposer` computes the deployment's revision hash as the lowercase-hex SHA-256 of the exact artifact bytes, so the node verifies a fetch against the `RevisionHash` the dispatch already carries — no hash rides in the proto. If a future change makes the revision hash anything else, every fetch fails verification.

**`DeploymentArtifactService` is `AllowAnonymous` by design** — same reasoning as `NodeCommandService`/`CentralCommandService` above: the AdminUI cookie `FallbackPolicy` on an admin-role node would otherwise block a correctly-keyed fetch before `ConfigServeAuthInterceptor`'s verdict matters. See [`security.md` § Mesh gRPC ingress is `AllowAnonymous` by design](security.md#mesh-grpc-ingress-is-allowanonymous-by-design).

#### Artifact compression (2026-08-15)

The artifact blob is **gzip-compressed, inside the hashed bytes**. `ConfigComposer.Pack` gzips the
serialised snapshot (`CompressionLevel.Fastest`, ~10–20x on this highly repetitive JSON) immediately
after `SerializeToUtf8Bytes` and **then** takes the SHA-256, so the compressed bytes *are* the artifact
everywhere — `Deployment.ArtifactBlob`, the LocalDb cache chunks, the gRPC fetch stream, and the
`RevisionHash` a `FetchAndCache` node verifies against. The invariant above is preserved exactly, which
is why the fetcher, the cache and the serve path are untouched: nothing between the composer and
`DeploymentArtifact.Open` looks inside the blob.

Decompression lives in exactly one place, `DeploymentArtifact.Open`, which **sniffs the gzip magic
`1F 8B`** at offset 0 — valid JSON can never start with `0x1F`, so no framing of our own is needed and
**pre-upgrade uncompressed rows keep parsing unchanged**. Inflation runs under a 512 MiB guard so a
corrupt or hostile blob cannot OOM the node, and every decompression failure (corrupt, truncated,
over-limit) is reported as `JsonException` — the same exception family the malformed-JSON path already
owns, so no caller's decision changes.

Two operational facts:

- **The first deploy after the upgrade re-hashes every revision.** The same configuration now serialises
  to different (compressed) bytes, so it produces a different `RevisionHash`. Expect a **one-time full
  re-dispatch to every node**, not the usual no-op — this is normal, and it happens once.
- **Rollout ordering matters: upgrade the NODES first.** A node must be running artifact-sniffing code
  *before* central starts emitting compressed blobs. A pre-upgrade node handed a gzip artifact cannot
  parse it and **fails the apply**, which means it keeps serving last-known-good (the #485 contract) —
  degraded but not destructive, and it recovers as soon as it is upgraded. Rolling back central is
  likewise safe in the other direction, since sniffing code still reads uncompressed blobs.

(gzip output is not guaranteed byte-stable across .NET releases. Harmless here: the hash is computed
from the bytes emitted at compose time and never re-derived from the JSON, so an implementation change
costs at most one spurious "changed" revision after a runtime upgrade. Snapshot determinism is
unaffected — compression wraps the final bytes and reorders nothing.)

On the docker-dev rig the central pair carries `ConfigServe` (port `4055`, dev key) and stays hardcoded `Direct` via its admin role; the four site nodes carry `ConfigSource` (endpoints + matching key) hardcoded `FetchAndCache` — mandatory for driver-only nodes since Phase 4, not a flip. There is no `OTOPCUA_CONFIG_MODE` env var in the compose file; central keeps SQL and models the target topology.

**`FetchAndCache` is mandatory on a driver-only node — per-cluster mesh Phase 4.** `Program.cs` now
registers `AddOtOpcUaConfigDb` only when the node carries the `admin` role, so a **driver-only**
node (`Cluster:Roles` has `driver`, not `admin`) holds **no `ConfigDb` connection string at all** —
there is nothing for `Direct` to read. `ConfigSourceOptionsValidator` enforces this at
`ValidateOnStart`: a driver-only node whose `ConfigSource:Mode` is `Direct` **fails host start**
with an error naming the fix (`ConfigSource:Mode=FetchAndCache`). A **fused** `admin,driver` node
still owns its ConfigDb via the admin role and may stay `Direct` — the validator only fires on
`driver` without `admin`. On a driver-only node the deployed configuration, the scripted-alarm Part
9 condition state (`LocalDbAlarmConditionStateStore`, replacing the ConfigDb-backed
`EfAlarmConditionStateStore`), and the alarm store-and-forward buffer all live in the node's
LocalDb; central remains the system of record for deploy acks (`ConfigPublishCoordinator.
PersistNodeAck`) and LDAP group→role DB grants (a driver-only node maps LDAP groups to roles from
the `Security:Ldap:GroupToRole` appsettings baseline only — see `docs/Redundancy.md` and
`CLAUDE.md` §"Phase 4" for the full picture, including the client-visible ServiceLevel change).

### `Telemetry` / `TelemetryDial` (live telemetry transport)

- **Purpose:** [per-cluster mesh](plans/2026-07-21-per-cluster-mesh-design.md) **Phase 5**. Selects how
  the four live-observability channels (`alerts`, `script-logs`, `driver-health`,
  `driver-resilience-status`) reach central: `Dps` (DistributedPubSub — the four existing SignalR
  bridges, today's behaviour) or `Grpc` (a driver node hosts one gRPC server-streaming contract over a
  dedicated h2c port; central dials in and feeds the same in-process sinks). `Telemetry` is the
  **node/serve** side; `TelemetryDial` is the **central/dial** side.
- **Options classes:** `TelemetryOptions` (`SectionName = "Telemetry"`) + `TelemetryDialOptions`
  (`SectionName = "TelemetryDial"`), both in `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/`. Bound by
  `AddOtOpcUaCluster(config)`; their validators run at `ValidateOnStart`.

**`Telemetry` (node-side, serve):**

| Key | Type | Default | Meaning |
|---|---|---|---|
| `Mode` | string | `Dps` | `Dps` = no change, node just keeps publishing DPS as it always has. `Grpc` = central-ingest hint (see `TelemetryDial:Mode` below — this key does not gate anything on the node itself). Any other value **refuses to start**. |
| `GrpcListenPort` | int | `0` | Dedicated **h2c-only** Kestrel port for the telemetry stream service. `0` = disabled (nothing bound). The node hosts the server whenever this is `> 0`, **regardless of `Mode`**. |
| `ApiKey` | string | `""` | Shared bearer key the serve-side interceptor checks (constant-time, **fail-closed**: an unset key rejects every call). **Supply via env `Telemetry__ApiKey` — never commit.** |

**`TelemetryDial` (central-side, dial):**

| Key | Type | Default | Meaning |
|---|---|---|---|
| `Mode` | string | `Dps` | `Dps` = the four DPS SignalR bridges (alert, script-log, driver-status, resilience) subscribe as today. `Grpc` = those four bridges are **not spawned**; the `TelemetryDialSupervisor` actor dials every driver node instead and feeds the same sinks. Any other value **refuses to start**. |
| `ApiKey` | string | `""` | Shared bearer key; must equal the nodes' `Telemetry:ApiKey`. **Supply via env `TelemetryDial__ApiKey` — never commit.** Required under `Grpc`. |
| `ContactRefreshSeconds` | int | `60` | How often central rebuilds its dial-target set from the `ClusterNode` rows (`Host` + `GrpcPort`). |
| `CallTimeoutSeconds` | int | `30` | Per-call deadline for the streaming RPC's keepalive/health bookkeeping. |

**It is a dark switch; the node always hosts.** Unlike `ConfigServe`/`MeshTransport`, where the serve
side is gated on a role, the telemetry server binds whenever `Telemetry:GrpcListenPort > 0` — hosting
is unconditional on a driver-role node, in both modes, because the node also always emits into its
local hub and always publishes DPS. Only `TelemetryDial:Mode` on central actually changes behaviour:
which of the two already-always-available sources (DPS bridges vs. the gRPC dial supervisor) feeds the
AdminUI sinks. Flipping either side is a config change plus restart, not a redeploy — the same
discipline as `MeshTransport:Mode` and `ConfigSource:Mode`.

**A driver-role node with `Telemetry:Mode=Grpc` must set `GrpcListenPort > 0`** (nothing to serve
otherwise) — enforced by `TelemetryOptionsValidator`. **`Grpc` mode on either side requires a
non-empty `ApiKey`** — fail-closed, same as `ConfigServe`/`LocalDb:Replication`.

**Two DPS observability topics stay out of scope for this transport, on purpose:**
`redundancy-state` (pair-local, built from `Cluster.State`, stays on DPS in both modes — see
[`Redundancy.md`](Redundancy.md#command-transport-centralnode)) and `deployment-acks` (already rides
the Phase 2 gRPC command transport, `CentralCommandService.ReportApply`, as a command-plane reply).
A third, `fleet-status`, was **deleted** rather than deferred (2026-08-06, Gitea #542 — it had no
client end to end; `Fleet.razor` polls the Config DB). See
[`docs/Telemetry.md`](Telemetry.md) for the full architecture, the reconnect/generation-stamped
dialer story, and the rationale for the node-hosts/central-dials inversion.

On the docker-dev rig every driver-role node carries `Telemetry:GrpcListenPort` (`4056`) + a
committed dev key, and central carries the matching `TelemetryDial:ApiKey`; both `Mode` keys stay
unset (⇒ `Dps`) until the rig is flipped with `Telemetry__Mode=Grpc` / `TelemetryDial__Mode=Grpc` at
`docker compose up`.

### `ConnectionStrings` → `ConfigDb`

- **Purpose:** the central Config DB connection string. **Required on admin-role nodes only** — `Program.cs` calls `AddOtOpcUaConfigDb` when `hasAdmin` (per-cluster mesh Phase 4). A driver-only node holds no `ConfigDb` connection string; it fetches its configuration via `ConfigSource:Mode=FetchAndCache` instead (see above). A fused `admin,driver` node still requires it.
- **Bound by:** `AddOtOpcUaConfigDb(config)` (`src/Core/ZB.MOM.WW.OtOpcUa.Configuration/ServiceCollectionExtensions.cs`). The connection-string name constant is `ConnectionStringName = "ConfigDb"`, read via `configuration.GetConnectionString("ConfigDb")`. If absent, startup throws with a message pointing to either `appsettings.json` or the `OTOPCUA_CONFIG_CONNECTION` env var.
- **Shape:** standard `ConnectionStrings:ConfigDb` SQL Server connection string. There is no checked-in default in the thin `appsettings*.json` — supply it per environment.

The Config DB itself (the EF Core `OtOpcUaConfigDbContext`, entities, draft/publish generations, `NodeAcl`, `LdapGroupRoleMapping`, migrations) is the durable home for the fleet's drivers, UNS hierarchy, ACLs, and audit log. For the **full schema** see [`docs/v2/config-db-schema.md`](v2/config-db-schema.md). This page does not duplicate it.

### Audit log (`ConfigAuditLog`)

The cluster audit log lives in the Config DB and is written by `ActorAuditWriter`; the AdminUI reads it
per cluster at `/clusters/{id}/audit`. It has no `appsettings` section — it is on whenever an admin node
has a `ConfigDb` — but one schema property is worth recording here because it constrains how callers may
write to it:

- **`EventId` gives cross-restart deduplication, and the DB enforces it.** Each row carries the
  stable per-event `AuditEvent.EventId`, under a *filtered* unique index
  (`UX_ConfigAuditLog_EventId`, `WHERE [EventId] IS NOT NULL` — so the nullable pre-v2 backfill rows
  do not collide with each other). That is what makes an Akka actor safe to retry an apply against
  the audit log: the batched writer flushing again after a process crash, a supervision restart, or
  an at-least-once redelivery re-sends the same `EventId` and does **not** insert a duplicate row.
  `CorrelationId` is the companion column — it joins a row to its originating request/workflow, so
  the several rows of one logical operation group together.
- Consequently, **mint the `EventId` per event, never per write attempt.** An id generated inside
  the retry path defeats the unique index rather than tripping it, and nothing downstream flags the
  result — the log simply grows a second copy of a deployment's audit trail. Carry the id with the
  command.

For the full column list see [`docs/v2/config-db-schema.md`](v2/config-db-schema.md).

### Galaxy / MxAccess driver config (`DriverConfig` JSON, not `appsettings`)

The Galaxy/MxAccess connection settings are **not an `appsettings` section.** They are driver-instance options stored in the `DriverConfig` JSON column of the Config DB (edited via the Admin UI), bound to `GalaxyDriverOptions` (`src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Contracts/GalaxyDriverOptions.cs`, namespace `...Driver.Galaxy.Config`). It decomposes into nested records:

| Record | Key fields (default) | Meaning |
|---|---|---|
| `GalaxyGatewayOptions` (`Gateway`) | `Endpoint`; `ApiKeySecretRef`; `UseTls` (`true`); `CaCertificatePath` (`null`); `ConnectTimeoutSeconds` (`10`); `DefaultCallTimeoutSeconds` (`30`); `StreamTimeoutSeconds` (`0` = unlimited) | mxaccessgw gateway connection. `ApiKeySecretRef` supports `env:NAME` / `file:PATH` / `dev:KEY` / literal forms (resolved at `InitializeAsync`); prefer `env:`/`file:` in production. Never store a cleartext key. |
| `GalaxyMxAccessOptions` (`MxAccess`) | `ClientName`; `PublishingIntervalMs` (`1000`); `WriteUserId` (`0` = anonymous); `EventPumpChannelCapacity` (`50000`) | MXAccess client identity + tuning. `ClientName` **must be unique per OtOpcUa instance** (redundancy pairs enforce this). |
| `GalaxyRepositoryOptions` (`Repository`) | `DiscoverPageSize` (`5000`); `WatchDeployEvents` (`true`) | Galaxy Repository browse paging + deploy-event watching. |
| `GalaxyReconnectOptions` (`Reconnect`) | `InitialBackoffMs` (`500`); `MaxBackoffMs` (`30000`); `ReplayOnSessionLost` (`true`) | In-driver reconnect-supervisor backoff. |
| (top-level) | `ProbeTimeoutSeconds` (`30`, range 1–60) | AdminUI Test-Connect probe timeout. |

> The `OTOPCUA_GALAXY_*` environment variables that v1's in-process `Galaxy.Host` consumed **no longer live in this repo** — they moved into the separately-installed mxaccessgw gateway's own config (see the v1 archive pointer in `docs/README.md` and the Galaxy overview at [`docs/drivers/Galaxy.md`](drivers/Galaxy.md)). The only Galaxy connection secret this repo touches is the gateway API key via `ApiKeySecretRef` above.

### Historian config (`ServerHistorian:Backend` — HistorianGateway, SQL Server or QuestDB)

The historian backend is **pluggable**, selected by `ServerHistorian:Backend` (case-insensitive; default `Gateway`;
unknown value ⇒ host start fails listing the allowed values): `Gateway` is the external **`ZB.MOM.WW.HistorianGateway`**
sidecar below, `SqlServer` is a plain SQL Server database (see [`ServerHistorian:SqlServer`](#serverhistoriansqlserver-backendsqlserver)),
and `QuestDb` is a QuestDB time-series database reached over the Postgres wire protocol (see
[`ServerHistorian:QuestDb`](#serverhistorianquestdb-backendquestdb)).
The default backend is the gateway, consumed as the
`ZB.MOM.WW.HistorianGateway.Client` gRPC package (the retired Wonderware TCP sidecar is documented at
[`docs/drivers/Historian.Wonderware.md`](drivers/Historian.Wonderware.md)). The OtOpcUa host reads three
appsettings sections — `ServerHistorian` (read path + gateway connection), `ContinuousHistorization`
(FasterLog outbox + recorder draining to `WriteLiveValues`), and `AlarmHistorian` (the store-and-forward
alarm sink draining to `SendEvent`; its buffer lives in the node's `LocalDb:Path` database, not a
file of its own). The gateway connection (endpoint / key / TLS) lives **only** in
`ServerHistorian`; the other two sections source it from there.

The gateway API key is supplied via the environment variable **`ServerHistorian__ApiKey`** — never committed
to config. The target gateway must run `RuntimeDb:Enabled=true` + `RuntimeDb:EventReadsEnabled=true`, and the
key must carry the scopes `historian:read`, `historian:write`, `historian:tags:write`. See
[`docs/Historian.md`](Historian.md) for the full key reference, the migration note (old Wonderware keys →
gateway keys), and the deployment prerequisites.

#### `ServerHistorian:SqlServer` (`Backend=SqlServer`)

Read only when `ServerHistorian:Backend` is `SqlServer`; the gateway keys (`Endpoint`, `ApiKey`, `UseTls`, …) are then not required.
Validated at host start (driver-role nodes) when `ServerHistorian:Enabled` **or** `AlarmHistorian:Enabled` is true.

| Key | Default | Validation / meaning |
|---|---|---|
| `ServerHistorian:Backend` | `Gateway` | `Gateway`, `SqlServer` or `QuestDb`; anything else fails host start (checked even when the historian is disabled). |
| `ServerHistorian:SqlServer:ConnectionString` | `""` | **Required.** Historian database connection string (not the ConfigDb). Supply via `${secret:OT-HISTORIAN-SQL}` or env `ServerHistorian__SqlServer__ConnectionString`; the database must already exist. |
| `ServerHistorian:SqlServer:Schema` | `hist` | Must match `^[A-Za-z_][A-Za-z0-9_]{0,127}$` (spliced into DDL/DML). |
| `ServerHistorian:SqlServer:CommandTimeout` | `00:00:30` | Must be `> 0`; rounded up to whole seconds. |
| `ServerHistorian:SqlServer:AutoCreateSchema` | `true` | `true` applies the embedded schema scripts under an app-lock on first use (upgrading a v1 schema to v2); `false` only verifies `SchemaVersion >= 1` and fails every historian call (health `LastError`, no crash) if behind. A verified **v1** schema (no `002_sample_modified.sql`) is served with modified-value reads and HistoryUpdate refused `BadHistoryOperationUnsupported`, retention skipping `SampleModified`, and one Warning per boot (#674). |
| `ServerHistorian:SqlServer:RetentionDays` | unset | Days of history kept (source time), `>= 1` when set; unset keeps everything. Enables the Primary-gated purge of `hist.Sample`/`hist.Event` (see [Historian.md](Historian.md#retention-retentiondays)). |
| `ServerHistorian:SqlServer:PurgeInterval` | `01:00:00` | Purge cadence; must be `> 0`. |
| `ServerHistorian:SqlServer:PurgeBatchSize` | `4000` | Rows per `DELETE TOP (n)` batch; `>= 1`. Keep under ~5000 to avoid lock escalation. |

#### `ServerHistorian:QuestDb` (`Backend=QuestDb`)

Read only when `ServerHistorian:Backend` is `QuestDb`; the gateway keys are then not required. Validated at host start (driver-role nodes)
when `ServerHistorian:Enabled` **or** `AlarmHistorian:Enabled` is true (`QuestDbHistorianOptions.Validate()`).

| Key | Default | Validation / meaning |
|---|---|---|
| `ServerHistorian:QuestDb:ConnectionString` | `""` | **Required.** Npgsql connection string for QuestDB's PG-wire port, e.g. `Host=questdb;Port=8812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading`. Env `ServerHistorian__QuestDb__ConnectionString` / `${secret:}` — never commit a real one. When the string does not set Npgsql `Maximum Pool Size`, the backend caps it at **24** (Npgsql's own default of 100 exceeds QuestDB's 64-connection default limit); an explicit value is kept as written, and the effective value is logged at Information. Keep the per-node total of a pair under QuestDB's limit. |
| `ServerHistorian:QuestDb:TablePrefix` | `hist_` | Must match `^[A-Za-z0-9_]+$` (spliced into DDL/DML). Tables are `{prefix}sample`, `{prefix}event`, `{prefix}tag`. |
| `ServerHistorian:QuestDb:RetentionDays` | unset | `>= 1` when set. Applied as table TTL to the sample and event tables (enforced on write commits only — a quiet table never drops). Unset keeps data forever. |
| `ServerHistorian:QuestDb:CommandTimeout` | `00:00:30` | Must be `> 0`; rounded up to whole seconds. |
| `ServerHistorian:QuestDb:AutoCreateSchema` | `true` | `true` creates the tables and reconciles the TTL on first use; `false` only verifies the tables exist and fails every historian call (health `LastError`, no crash) if one is missing. |
| `ServerHistorian:QuestDb:VisibilityTimeout` | `00:00:05` | Must be `>= 0`. How long a write waits for its rows to become visible to queries (WAL apply); `0` disables the wait. A timeout only logs a Warning. |

HistoryUpdate (SqlServer and QuestDb only; QuestDb deletes are soft — tombstones) additionally needs `ServerHistorian:Enabled=true` and a key
whose principal holds the `HistoryUpdate` permission — see [`docs/security.md`](security.md#historyupdate-historian-writes). It is served
only on the redundancy **Primary** (`BadInvalidState` elsewhere, #674) and is bounded by one backend-independent key:

| Key | Default | Meaning |
|---|---|---|
| `ServerHistorian:HistoryUpdateMaxFutureSkew` | `1.00:00:00` (1 day) | TimeSpan. A HistoryUpdate (Insert / Replace / Update) item whose `SourceTimestamp` is later than server-clock now + this is refused `BadOutOfRange` before any backend sees it (#674 — a far-future typo could otherwise age out a QuestDB TTL table or poison the history). Applies to every backend; the live recorder is not bounded. Non-positive disables the bound. | Full semantics,
schema, aggregates and the docker-dev recipe: [`docs/Historian.md`](Historian.md#sql-server-backend) (SqlServer) and
[`docs/Historian.md`](Historian.md#questdb-backend) (QuestDb: the µs timestamp rules, pre-1970 limits, TTL and the docker-dev overlay).

#### `ContinuousHistorization` (recorder + outbox knobs)

Only the knobs; the backend connection comes from `ServerHistorian`. Outbox records are format v2 — **drain or delete `OutboxPath` before downgrading a node** to a pre-v2 build (see [`docs/Historian.md` §Outbox format v2 and rollback](Historian.md#outbox-format-v2-and-rollback)); undecodable records are skipped and counted by the gauge `otopcua.historian.outbox.unreadable`. Full descriptions in
[`docs/Historian.md` §ContinuousHistorization configuration](Historian.md#continuoushistorization-configuration).

| Key | Default | Meaning |
|---|---|---|
| `ContinuousHistorization:Enabled` | `false` | `true` **and** a configured `ServerHistorian` wires the FasterLog outbox and spawns the recorder. |
| `ContinuousHistorization:OutboxPath` | `""` | **Directory** for the FasterLog segment + commit files. Required when enabled; use an **absolute** path on durable storage. |
| `ContinuousHistorization:CommitMode` | `PerEntry` | `PerEntry` fsyncs before the append returns; `Periodic` commits on a timer every `CommitIntervalMs`. **`PerEntry` is one fsync per ingest *burst*, not per value** — the recorder appends a burst through `AppendBatchAsync`, which enqueues every entry and issues a single group commit. Durability is unchanged: the call does not return until that commit lands. |
| `ContinuousHistorization:CommitIntervalMs` | `100` | Periodic-mode cadence (ms); must be `> 0` under `Periodic`. |
| `ContinuousHistorization:DrainBatchSize` | `1024` | Entries peeked + written per drain pass. **Raised from 64**: the peek is an in-memory scan (widening it costs allocation, not I/O) and the pass is now bounded-parallel rather than sequential, so a wide batch no longer costs `tags × latency`. 64 drew down ≤ 64 entries per `DrainIntervalSeconds`, so any gateway hiccup left the outbox permanently behind, growing toward `Capacity` and shedding oldest samples. |
| `ContinuousHistorization:DrainIntervalSeconds` | `2` | Steady drain cadence (and post-success reschedule). |
| `ContinuousHistorization:DrainConcurrency` | `8` | Historian tags one drain pass writes **concurrently** — a batch is grouped per tag and each group is its own gateway round-trip, so this turns a sequential `tags × latency` pass into `⌈tags / concurrency⌉ × latency` while bounding outstanding round-trips. `1` = fully sequential; `< 1` is clamped to 1 with a `Validate()` warning. The ack is unchanged (all-or-nothing after every group reports), as is the #538 quarantine rule. |
| `ContinuousHistorization:Capacity` | `500000` | Max un-acked entries before drop-oldest; `0` = unbounded. The #538 disk backstop for a historian refusing **every** write — the one case the per-tag quarantine cannot resolve. |
| `ContinuousHistorization:MaxWriteAttempts` | `5` | #538 poison-tag budget: consecutive failed passes one tag may accumulate — **counted only across passes where *other* tags wrote successfully** — before its entries are abandoned so the FIFO head can advance. |
| `ContinuousHistorization:CaptureWhilePrimaryUnreachable` | `true` | #672: a standby stood down for its mesh-partner Primary (#537) resumes capture as soon as the failure detector reports that Primary **unreachable**, rather than after auto-down + a fresh snapshot — a crash-failover gap of ~25–30 s becomes the failure-detector window (~10–12 s). Cost: duplicate capture while a live Primary is merely unreachable and under a real partition (absorbed idempotently on SqlServer/QuestDB only for source-timestamped drivers; Modbus-style poll-time stamps and the Gateway backend store duplicates). Affects only the capture verdict. `false` = pre-#672 behaviour. |
| `ContinuousHistorization:MinBackoffSeconds` / `MaxBackoffSeconds` | `1` / `30` | Initial and capped exponential retry backoff after a failed drain pass. |

### `LocalDb` (pair-local config cache — driver role only)

Driver-role nodes keep a consolidated **`ZB.MOM.WW.LocalDb`** SQLite database that caches the deployed
configuration artifact (chunked) so the node can **boot from cache when central SQL is unreachable**. The
section is bound by `AddOtOpcUaLocalDb` (`Host/Configuration/LocalDbRegistration.cs`) inside the `hasDriver`
branch — **admin-only nodes never register it, and never require `LocalDb:Path`**. Storage is unconditional;
replication stays inert until the sync port + peer are set.

| Key | Default | Meaning |
|---|---|---|
| `LocalDb:Path` | `./data/otopcua-localdb.db` | SQLite file path. `ValidateOnStart`-required once `AddZbLocalDb` runs (driver nodes only). In containers put it on a durable volume. |
| `LocalDb:SyncListenPort` | `0` | `0` = replication off (no sync listener). A non-zero port binds a dedicated **h2c** sync listener; set the **same** port on both pair nodes. Setting it makes the Host add an explicit Kestrel `Listen*` and re-bind the primary HTTP port (an explicit `Listen*` otherwise discards `ASPNETCORE_URLS`). |
| `LocalDb:Replication:PeerAddress` | `""` | The peer this node **dials** (`http://<peer>:<port>`). Set on the initiator only; leave empty on the passive node. The stream is bidirectional. |
| `LocalDb:Replication:ApiKey` | `""` | Bearer token for the sync stream. **Fail-closed:** no key ⇒ the passive endpoint refuses everything; a mismatch ⇒ the pair silently stops converging. **Must be byte-identical on both nodes.** Supply via `${secret:...}` / env — never a cleartext literal in production. |
| `LocalDb:Replication:MaxBatchSize` | `500` (library) | **Secondary** row-count cap on a batch — *and* the DB read-page size. `16` on the replicating pair. Since LocalDb 0.2.0 the wire message is bounded by `MaxBatchBytes`, not by this, so it is no longer the hand-tuned byte proxy the 4 MB cap once forced. It still matters because `OplogStore.ReadBatchAboveAsync`/`SnapshotStreamer` materialise a whole read page **before** the byte-budget split runs — `16` bounds that transient allocation at ≈2.7 MB for ≈171 KB artifact chunk rows. |
| `LocalDb:Replication:MaxBatchBytes` | `2 MB` (library) | **Primary** cap on one outbound delta/snapshot message: the summed serialized size of its entries, sized under gRPC's 4 MB receive limit. Left unset across the fleet. A single row larger than the budget is sent **alone** rather than stalling the stream. Added in LocalDb 0.2.0; needs no wire change, so a node on this version interoperates with any earlier peer in both directions. |

**These two keys also decide whether the node captures at all.** `LocalDbSetup.ReplicationIsConfigured` is
`PeerAddress` **OR** `ApiKey` non-blank — the OR because only the *dialling* half sets `PeerAddress`, so
keying on it alone would strip change capture from every passive node and leave the pair converging in one
direction only. `SyncListenPort` is deliberately **not** part of it (a port with no `ApiKey` is fail-closed
and syncs with nobody). (The alarm-historian **drain gate** uses a stricter predicate on purpose —
`ApiKey` AND (`PeerAddress` OR `SyncListenPort > 0`), `LocalDbReplicationPolicy.SharesRowsWithPeer` — because
gating a node that shares its queue with nobody strands its alarm history; see `docs/AlarmHistorian.md`
§"The drain gate".) A configured node registers all four tables with `baselineExistingRows: true`, so
turning replication on converges the rows **already in the file** and not merely subsequent writes; an
unconfigured node **deregisters** them at boot, dropping capture triggers an earlier build installed. ⚠️ The
sync handshake compares registered-table digests **fail-closed**, so flipping either key is a
both-nodes-together change: set it on both and restart both. As of LocalDb 0.3.0 (this repo pins
0.3.4) baselining flags a **per-table** snapshot resync — enabling replication on one table no
longer forces a full-database resync of every registered table in both directions.

Replication is **default-OFF** across the fleet; it is enabled per-pair as an opt-in. See
[`docs/operations/2026-07-20-localdb-pair-replication.md`](operations/2026-07-20-localdb-pair-replication.md)
for the enablement runbook (ApiKey handling, stop/start-together rule, tombstone-retention window, and the
never-`sqlite3`-a-live-WAL-DB inspection rules) and the "pair-local config cache" section of
[`docs/Redundancy.md`](Redundancy.md) for what boot-from-cache does and does not cover.

### `Drivers:AbCip` (node-level AbCip defaults)

- **Purpose:** pin the wire transport an **AbCip driver with no `transport` key** runs on, for every such
  driver on this node, without editing driver configs (final review I1 of the native AB transport). The
  per-driver `DriverConfig` key `transport` still wins when present. The main use is an upgrade: the old
  AdminUI cannot author the per-driver key (its typed form drops unknown keys, #576), so this is the way to
  keep a fleet on libplctag across the upgrade restart. See
  [`drivers/AbCip.md` § Transport](drivers/AbCip.md#transport-transport-option).
- **Options class:** `AbCipDriverDefaultsOptions` (`SectionName = "Drivers:AbCip"`) —
  `src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/AbCipDriverDefaultsOptions.cs`. Bound and validated
  unconditionally in `Program.cs` (`AddValidatedOptions` + `ValidateOnStart`), because both consumers need
  it: the driver factory (driver role) and the AdminUI Test Connect probe (admin role), so the probe
  exercises the stack the driver would run.

| Key | Type | Default | Meaning |
|---|---|---|---|
| `DefaultTransport` | string | `Native` | `Native` or `Libplctag`, case-insensitive. Blank/absent = `Native`. **Any other value fails host start** (`AbCipDriverDefaultsOptionsValidator`, naming the key), including numeric (`1`) and flags (`Native,Libplctag`) forms. Env form: `Drivers__AbCip__DefaultTransport`. |

Precedence for an AbCip driver: explicit `transport` in its `DriverConfig` → this key → compiled `Native`.
Like the per-driver key, it takes effect when a driver instance is **constructed** (process start, or a
deploy that recreates the driver), not on `ReinitializeAsync`. The init log line reads
`transport=Libplctag (node default)` when the pin is what selected the stack. There is **no AbLegacy
counterpart**: an AbLegacy driver with no key always runs libplctag (Native is opt-in per driver).

---

## Environment variables

All names are read in this repo's source via `Environment.GetEnvironmentVariable(...)` unless noted otherwise. Defaults shown are the in-source fallbacks.

### Host / cluster / Config DB

| Variable | Read by | Effect / default |
|---|---|---|
| `OTOPCUA_ROLES` | `EffectiveRoles.Resolve` (`ZB.MOM.WW.OtOpcUa.Cluster`) — the single role authority both `Program.cs` and `AddOtOpcUaCluster` consume | Comma-separated cluster roles for the node (`admin`, `driver`, `dev`, `cluster-{ClusterId}`), read as the **fallback** only when `Cluster:Roles` is empty. Drives the conditional wiring, the per-role appsettings overlay, and the node's Akka member roles — all from the same resolved instance. See the note under `Cluster:Roles` above. |
| `OTOPCUA_CONFIG_CONNECTION` | `src/Core/ZB.MOM.WW.OtOpcUa.Configuration/DesignTimeDbContextFactory.cs` (design-time / `dotnet ef` only) | Read at **design time** by `DesignTimeDbContextFactory.cs` for `dotnet ef` migrations. At **runtime** the server resolves the connection string from `ConnectionStrings:ConfigDb` (env form: `ConnectionStrings__ConfigDb`) via `configuration.GetConnectionString("ConfigDb")` in `ServiceCollectionExtensions.cs` — `OTOPCUA_CONFIG_CONNECTION` appears there only as a hint in an error message, not via `GetEnvironmentVariable`. No credential is embedded in source. |
| `ASPNETCORE_ENVIRONMENT` | ASP.NET host builder (framework) | Selects `appsettings.{Environment}.json` (e.g. `Development`). |
| `Drivers__AbCip__DefaultTransport` | configuration binder (`AbCipDriverDefaultsOptions`) | Env form of [`Drivers:AbCip:DefaultTransport`](#driversabcip-node-level-abcip-defaults): `Native` (default) or `Libplctag` for AbCip drivers with no `transport` key. A bad value fails host start. |

### Historian (`ServerHistorian__ApiKey`)

The retired Wonderware sidecar's `OTOPCUA_HISTORIAN_*` environment variables are **gone** — no source reads
them anymore. The historian backend is now the external HistorianGateway, configured through the
`ServerHistorian` / `ContinuousHistorization` / `AlarmHistorian` appsettings sections (above). The single
historian secret this repo reads from the environment is the gateway API key:

| Variable | Effect / default |
|---|---|
| `ServerHistorian__ApiKey` | The HistorianGateway peppered-HMAC key (`histgw_<id>_<secret>`) sent as `Authorization: Bearer`. Supply via environment — **never commit**. Required when `ServerHistorian:Enabled=true` **and `Backend=Gateway`**. |
| `ServerHistorian__Backend` | `Gateway` (default), `SqlServer` or `QuestDb`. |
| `ServerHistorian__SqlServer__ConnectionString` | The SQL Server historian connection string. Required when `Backend=SqlServer` and the historian (or alarm history) is enabled. Supply via environment / `${secret:}` — **never commit**. |
| `ServerHistorian__QuestDb__ConnectionString` | The QuestDB historian Npgsql connection string. Required when `Backend=QuestDb` and the historian (or alarm history) is enabled. Supply via environment / `${secret:}` — **never commit**. |

### Driver integration-test / fixture sim endpoints

These are consumed by the driver **integration-test fixtures** (under `tests/Drivers/...IntegrationTests/`), not by the production server. Each overrides the simulator endpoint a fixture TCP-probes; defaults point at the shared Docker host `10.100.0.35` (see `CLAUDE.md` Docker Workflow).

| Variable | Read by (fixture) | Default |
|---|---|---|
| `MODBUS_SIM_ENDPOINT` | `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus.IntegrationTests/ModbusSimulatorFixture.cs` | `10.100.0.35:5020` |
| `AB_SERVER_ENDPOINT` | `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.AbCip.IntegrationTests/AbServerFixture.cs` | `10.100.0.35:44818` |
| `S7_SIM_ENDPOINT` | `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.S7.IntegrationTests/Snap7ServerFixture.cs` | `10.100.0.35:1102` (non-privileged; not S7-standard 102) |
| `OPCUA_SIM_ENDPOINT` | `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.IntegrationTests/OpcPlcFixture.cs` | `opc.tcp://10.100.0.35:50000` |
| `OTOPCUA_FOCAS_SIM_ENDPOINT` | `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.FOCAS.IntegrationTests/FocasSimFixture.cs` | `10.100.0.35:8193` |

> Additional harness/parity/soak env vars (`OTOPCUA_FOCAS_*`, `OTOPCUA_PARITY_*`, `OTOPCUA_SOAK_*`, `OTOPCUA_HARNESS_USE_SQL`) exist only in the test/parity/soak harnesses, not in production source, and are out of scope for this reference.

---

## See also

- [`security.md`](security.md) — transport security, OPC UA authentication, LDAP (`Security:Ldap`), data-plane ACLs, control-plane roles.
- [`Redundancy.md`](Redundancy.md) — the `Cluster` section in the context of warm/hot redundancy, ServiceLevel, peer discovery.
- [`Telemetry.md`](Telemetry.md) — the `Telemetry`/`TelemetryDial` gRPC live-telemetry stream: direction, dark switch, the four migrated channels + three deferrals, auth, reconnect story.
- [`ServiceHosting.md`](ServiceHosting.md) — role-based host wiring and `OTOPCUA_ROLES`.
- [`docs/drivers/Galaxy.md`](drivers/Galaxy.md) — Galaxy/MxAccess driver overview.
- [`docs/v2/config-db-schema.md`](v2/config-db-schema.md) — the full Config DB schema.