Joseph Doherty
06c56d7719
Two independent defects surfaced in review of #520 stage 3.
#532 — Cluster:Roles had no OTOPCUA_ROLES fallback, despite two doc sites
promising one. A node configured only through the env var — which drives ALL of
Program.cs's conditional wiring, so it reads as the authoritative knob — bound an
empty Roles array and joined the cluster carrying no Akka roles at all. It ran,
served, and logged nothing wrong, while central reported "no driver-role nodes
are Up" and every deployment to it timed out. Not hypothetical: the field fix on
wonder-app-vd03 was to hand-add Cluster__Roles__0/1 to a Windows service's
registry environment. Anyone reading the code to diagnose that box would have
concluded from the XML doc that the fallback was already applied.
AddOtOpcUaCluster now post-configures AkkaClusterOptions through the shared
RoleParser.ResolveEffectiveRoles, so BuildClusterOptions (Akka's own member
roles), ClusterRoleInfo, the ControlPlane singleton scope and
ConfigDbApiKeyStore's ClusterId all see one answer. Two sibling validators
(ConfigSource, Telemetry) read Cluster:Roles directly and were exempting the
exact node they exist for — an env-only driver-only node has no ConfigDb and
could sit on ConfigSource:Mode=Direct unchallenged. They resolve the same way now.
Also collapsed the four open-coded `Roles.FirstOrDefault(IsClusterRole)` +
ClusterIdFromRole copies into RoleParser.ClusterIdFromRoles. They had drifted:
three classified the cluster role ordinally, so a verbatim-bound `Cluster-SITE-A`
yielded a null ClusterId — and a null ClusterId applies NO cluster filter, which
made #528's cross-cluster over-grant guard fail OPEN. IsClusterRole is now
case-insensitive; the ClusterId suffix keeps its authored spelling, because
normalizing it would silently rename the Akka role of a running deployment.
#533 — three string-identity rules in one authorization path. AclScopeRef used
default (ordinal) equality while the trie builder filtered ClusterId and hashed
principals OrdinalIgnoreCase; the deploy gate's cross-cluster check was ordinal
too, which was the worse half — it could reject a whole deploy over a ClusterId
case difference the runtime would have accepted. It only worked because resolver
and builder were handed the same NodeAcl instances.
The rule is now stated once, on AclScopeRef, and applied throughout:
identifiers (ScopeId, ClusterId, Principal) compare OrdinalIgnoreCase; path
segments compare Ordinal. AclScopePathResolver re-keys its own maps rather than
documenting a comparer contract for callers, so the two independent producers —
EF at the gate, artifact JSON on the node — cannot diverge from the outside.
PermissionTrieNode.Children stays Ordinal: those are browse names, and Speed /
speed are two nodes that can coexist under one device.
Split PermissionTrieBuildDiagnosticReason.RetiredScopeKind out of
UnresolvedScope. A legacy FolderSegment row and a typo'd id had the same message
and different remedies.
Falsified: five mutations, each reddening only its own guard — remove the
PostConfigure (3 tests), revert IsClusterRole to ordinal (3), revert AclScopeRef
equality (1), revert the resolver's map comparer (6), revert the gate's
comparisons (2), remove the FolderSegment short-circuit (1).
Test isolation: OTOPCUA_ROLES is process-global and xUnit runs classes in
parallel, so the three validator suites that now resolve effective roles share a
non-parallel collection. Without it, cases passing no roles would intermittently
observe another class's env var.
Full solution build clean; full suite green apart from two pre-existing failures
confirmed identical on 3a6b2a1d (PrimaryGateFailoverTests, and
DriverProbeRegistrationTests expecting 10 probe types where 12 now register)
plus the AbLegacy integration suite, which needs a docker fixture.
Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
375 lines
38 KiB
Markdown
375 lines
38 KiB
Markdown
# Configuration Reference
|
||
|
||
> **v3 in progress (Batch 1 landed — schema + RawPath identity).** The config **DB schema** was
|
||
> replaced by the greenfield v3 model: a driver-centric **Raw tree**
|
||
> (`Cluster → RawFolder → DriverInstance → Device → TagGroup → Tag`) plus a reference-only **UNS**
|
||
> projection (`UnsTagReference`). A tag's identity is now its **RawPath**
|
||
> (`<Folder/…>/<Driver>/<Device>/<TagGroup/…>/<Tag>`), not a `TagConfig.FullName` blob; the
|
||
> `Namespace`/`NamespaceKind`/`EquipmentImportBatch` entities are retired, and connection endpoints
|
||
> moved from `DriverConfig` into each `Device`'s `DeviceConfig`. The two OPC UA namespaces (`ns=Raw` /
|
||
> `ns=UNS`) and the `/raw` authoring UI arrive in later v3 batches; **Batch 1 leaves the address space
|
||
> deliberately dark** (drivers connect, but no tag variables materialize yet). The `appsettings*.json`
|
||
> surface below is unchanged. Full design: `docs/plans/2026-07-15-raw-uns-two-subtree-v3-design.md`.
|
||
|
||
This is the live configuration reference for the OtOpcUa Host (`src/Server/ZB.MOM.WW.OtOpcUa.Host/`). It enumerates the `appsettings*.json` sections, the bound Options classes, and the `OTOPCUA_*` / sim-endpoint environment variables — every entry grounded in source.
|
||
|
||
Two related concerns get their own dedicated pages and are **only summarised + linked** here, not duplicated:
|
||
|
||
- **Transport security, OPC UA authentication, LDAP, data-/control-plane authorization** → [`security.md`](security.md)
|
||
- **Redundancy + the `Cluster` section** → [`Redundancy.md`](Redundancy.md)
|
||
|
||
## How configuration is layered
|
||
|
||
The Host (`Program.cs`) loads `appsettings.json`, then overlays a **per-role** file chosen from the cluster roles:
|
||
|
||
- A single role → `appsettings.{role}.json` (e.g. `appsettings.driver.json`, `appsettings.admin.json`).
|
||
- Both roles → `appsettings.admin-driver.json` (roles joined with `-`, ordinal-sorted).
|
||
- `appsettings.{ASPNETCORE_ENVIRONMENT}.json` (e.g. `appsettings.Development.json`) is layered on by the host builder.
|
||
|
||
All role overlays are **optional** — the base `appsettings.json` plus the Options-class C# defaults are enough to boot. The roles themselves come from the `OTOPCUA_ROLES` env var (see [`ServiceHosting.md`](ServiceHosting.md) and the table below).
|
||
|
||
The checked-in `appsettings*.json` files are deliberately thin: they carry only `Serilog` and the `Security:Ldap` overlay. Everything else (`OpcUa`, `Cluster`, `ConnectionStrings`/`ConfigDb`) binds from the Options-class defaults documented below unless an operator adds the section explicitly or supplies the corresponding environment variable.
|
||
|
||
---
|
||
|
||
## `appsettings` sections
|
||
|
||
### `Serilog`
|
||
|
||
- **Purpose:** logging. Console + rolling daily file sink, layered with the shared `ZB.MOM.WW.Telemetry` enrichers (`AddZbSerilog` in `Program.cs`).
|
||
- **Where bound:** `builder.AddZbSerilog(...)` reads `Serilog` from configuration (`ReadFrom.Configuration`).
|
||
- **Checked-in shape** (`appsettings.json`): `Using` = `[ "Serilog.Sinks.Console", "Serilog.Sinks.File" ]`, `WriteTo` = a `Console` sink and a `File` sink (`path: logs/otopcua-.log`, `rollingInterval: Day`). Role overlays add `MinimumLevel` / `Override` blocks (e.g. `Opc.Ua: Debug`, `Akka: Information`).
|
||
|
||
### `OpcUa`
|
||
|
||
- **Purpose:** the OPC UA server endpoint identity, listening port, PKI, transport-security profiles, and redundancy peer advertising.
|
||
- **Options class:** `OpcUaApplicationHostOptions` — `src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OpcUaApplicationHost.cs`.
|
||
- **Bound by:** `AddValidatedOptions<OpcUaApplicationHostOptions, OpcUaApplicationHostOptionsValidator>(config, "OpcUa")` in `Program.cs` (driver-role only). Validated fail-fast at startup by `OpcUaApplicationHostOptionsValidator` (`src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/OpcUaApplicationHostOptionsValidator.cs`).
|
||
|
||
| Key | Type | Default | Meaning |
|
||
|---|---|---|---|
|
||
| `ApplicationName` | string | `OtOpcUa` | Server application name. Required (validated). |
|
||
| `ApplicationUri` | string | `urn:OtOpcUa` | Server application URI. Must be unique per redundancy node. Required. |
|
||
| `ProductUri` | string | `https://zb.com/otopcua` | Product URI. Not validated. |
|
||
| `OpcUaPort` | int | `4840` | Binary endpoint listen port. Validated as a port. |
|
||
| `PublicHostname` | string | `0.0.0.0` | Hostname/IP advertised in endpoint descriptions. Required. |
|
||
| `ApplicationConfigPath` | string? | `null` | Optional path to an application config XML; loaded instead of building from defaults. |
|
||
| `PkiStoreRoot` | string | `pki` | Root of the PKI hierarchy (`own`/`issuer`/`trusted`/`rejected` substores created under it). Required. See [`security.md`](security.md). |
|
||
| `EnabledSecurityProfiles` | list of `OpcUaSecurityProfile` | `[None, Basic256Sha256Sign, Basic256Sha256SignAndEncrypt]` | Transport-security profiles, one endpoint per entry. Must contain ≥1. Profile detail in [`security.md`](security.md). |
|
||
| `AutoAcceptUntrustedClientCertificates` | bool | `false` | Auto-trust unknown client certs on first connect (dev convenience). Not validated. See [`security.md`](security.md). |
|
||
| `PeerApplicationUris` | list of string | `[]` (empty) | Partner node `ApplicationUri`s published in `Server.ServerArray` for redundancy discovery. See [`Redundancy.md`](Redundancy.md). |
|
||
|
||
> **Transport security profiles** (the values in `EnabledSecurityProfiles` — `None`, `Basic256Sha256Sign`, `Basic256Sha256SignAndEncrypt`) and the PKI trust flow are documented in full in [`security.md`](security.md). This page does not duplicate them.
|
||
|
||
### `Security`
|
||
|
||
- **Purpose:** Admin-UI and OPC UA authentication. Three subsections, each its own Options class:
|
||
|
||
| Subsection | Options class (`SectionName`) | Purpose |
|
||
|---|---|---|
|
||
| `Security:Ldap` | `LdapOptions` — `src/Server/ZB.MOM.WW.OtOpcUa.Security/Ldap/LdapOptions.cs` | LDAP bind for Admin cookie login + OPC UA UserName tokens. Bound by `AddValidatedOptions<LdapOptions, LdapOptionsValidator>` in `Program.cs`. |
|
||
| `Security:Jwt` | `JwtOptions` — `src/Server/ZB.MOM.WW.OtOpcUa.Security/Jwt/JwtOptions.cs` | Signing config for the JWT minted at `/auth/token` for **external** consumers (OPC UA clients / automation). |
|
||
| `Security:Cookie` | `OtOpcUaCookieOptions` — `src/Server/ZB.MOM.WW.OtOpcUa.Security/CookieOptions.cs` | The Admin-UI auth cookie (`AddOtOpcUaAuth` copies these onto `CookieAuthenticationOptions`). |
|
||
|
||
**`Security:Ldap` — see [`security.md`](security.md) for the full field-by-field reference and bind-flow.** The checked-in role overlays set only `DevStubMode` and `Transport`; the remaining `LdapOptions` fields (`Enabled`, `Server`, `Port`, `AllowInsecure`, `SearchBase`, `ServiceAccountDn`, `ServiceAccountPassword`, `GroupAttribute`, `DisplayNameAttribute`, `UserNameAttribute`, `GroupToRole`) are covered there.
|
||
|
||
**`Security:Jwt`** key fields (`JwtOptions`):
|
||
|
||
| Key | Type | Default | Meaning |
|
||
|---|---|---|---|
|
||
| `SigningKey` | string | `""` | HS256 signing key; must be ≥32 bytes UTF-8. Set from your secret store — never commit a value. |
|
||
| `Issuer` | string | `otopcua` | JWT issuer. |
|
||
| `Audience` | string | `otopcua` | JWT audience. |
|
||
| `ExpiryMinutes` | int | `15` | Token lifetime. |
|
||
|
||
**`Security:Cookie`** key fields (`OtOpcUaCookieOptions`):
|
||
|
||
| Key | Type | Default | Meaning |
|
||
|---|---|---|---|
|
||
| `Name` | string | `ZB.MOM.WW.OtOpcUa.Auth` | Auth cookie name. Changing it invalidates existing sessions on next deploy. |
|
||
| `ExpiryMinutes` | int | `30` | Idle sliding-window length. |
|
||
| `RequireHttpsCookie` | bool | `true` | `SecurePolicy = Always`. Set `false` only for plain-HTTP local dev (emits a startup Warning). |
|
||
|
||
> Authentication, data-plane authorization (`NodeAcl` / `PermissionTrie`), and control-plane Admin roles are all in [`security.md`](security.md).
|
||
|
||
### `Cluster`
|
||
|
||
- **Purpose:** Akka.NET cluster identity, transport, and roles — the backbone of redundancy.
|
||
- **Options class:** `AkkaClusterOptions` (`SectionName = "Cluster"`) — `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/AkkaClusterOptions.cs`. Bound by `AddOtOpcUaCluster(config)` in `Program.cs`.
|
||
|
||
| Key | Type | Default | Meaning |
|
||
|---|---|---|---|
|
||
| `SystemName` | string | `otopcua` | Akka actor-system name. |
|
||
| `Hostname` | string | `0.0.0.0` | Bind hostname. |
|
||
| `Port` | int | `4053` | Cluster transport port. **Duplicated as `ClusterNode.AkkaPort` in the Config DB** — see the note below. |
|
||
| `PublicHostname` | string | `127.0.0.1` | Hostname advertised in cluster gossip; must be reachable by peers. **Duplicated as `ClusterNode.Host`** — see the note below. |
|
||
| `SeedNodes` | string[] | `[]` | Seed nodes for bootstrapping. **Ordered:** a node that appears in its own list must be entry 0 (only `seed-nodes[0]` can form a new cluster), or the host refuses to start — see [Redundancy.md § Bootstrap: self-first seed ordering](Redundancy.md#bootstrap-self-first-seed-ordering). **Per-pair since Phase 6:** each application `Cluster`'s pair is its own Akka mesh, so a node's `SeedNodes` lists only itself (first) and its own pair partner (second) — never a node from another cluster's pair. |
|
||
| `Roles` | string[] | `[]` | Cluster roles for this node. When empty, falls back to `OTOPCUA_ROLES` (see the note below). Allowed values: `admin`, `driver`, `dev`, plus a per-cluster `cluster-{ClusterId}` role (e.g. `cluster-MAIN`, `cluster-SITE-A`) — see [Per-cluster mesh split (Phase 6)](#per-cluster-mesh-split-phase-6-cluster-clusterid-roles--splittopologytransportvalidator) below. |
|
||
|
||
> **`Cluster:Roles` and `OTOPCUA_ROLES` are two knobs for one thing, and only one of them is the Akka role list.** `OTOPCUA_ROLES` drives all of `Program.cs`'s conditional wiring (which subsystems mount); `Cluster:Roles` becomes the node's **gossiped Akka member roles**. Since Gitea #532 `AddOtOpcUaCluster` applies the documented fallback — an empty `Cluster:Roles` takes the env var — so setting only `OTOPCUA_ROLES` now works. Before that fix it did not, silently: the node joined the mesh carrying **no roles at all**, which reads as healthy locally and as *"no driver-role nodes are Up"* at central, and every deployment to it timed out ([`deployments/wonder-app-vd03-makino-z-34184.md`](deployments/wonder-app-vd03-makino-z-34184.md)).
|
||
>
|
||
> Setting `Cluster:Roles` explicitly is still the recommendation, because the two sources normalize differently: `Cluster:Roles` binds **verbatim** while the env path is **lower-cased** by `RoleParser.Parse`. Akka compares role names ordinally, so a redundant pair configured through *different* sources with a mixed-case `cluster-{ClusterId}` (e.g. `cluster-SITE-A` vs `cluster-site-a`) would gossip two different role names and elect two singletons. Configure both nodes of a pair the same way.
|
||
|
||
> The full redundancy model (ServiceLevel tiers, split-brain, peer discovery) is in [`Redundancy.md`](Redundancy.md). The OPC UA peer-URI advertising lives in the `OpcUa:PeerApplicationUris` key above.
|
||
|
||
> **`Port` / `PublicHostname` are stored twice.** The node binds from these keys; the fleet's `ClusterNode` row records the same address as `AkkaPort` / `Host` so that **central can dial the node without sharing a gossip ring with it** — which is what [per-cluster mesh Phase 2](plans/2026-07-21-per-cluster-mesh-design.md) needs. Nothing in the schema makes the two agree, so `ClusterNodeAddressReconcilerActor` (admin-role singleton) compares them against live membership and logs an Error on mismatch. If you change `Cluster:Port` or `Cluster:PublicHostname` on a node, **update its `ClusterNode` row too** — a node binding 4054 with a row saying 4053 still gossips fine today, and fails silently in Phase 2. The row's `NodeId` is `host:port` and is also the deploy path's ack identity, so a drifted node's deployment acks stop matching as well. See [`config-db-schema.md` § `ClusterNode`](v2/config-db-schema.md#clusternode).
|
||
|
||
### Per-cluster mesh split (Phase 6): `cluster-{ClusterId}` roles + `SplitTopologyTransportValidator`
|
||
|
||
- **Purpose:** [per-cluster mesh](plans/2026-07-22-per-cluster-mesh-program.md) **Phase 6** split the
|
||
single fleet-wide Akka mesh into one independent 2-node mesh per application `Cluster`. A node's
|
||
`Cluster:Roles` now carries a `cluster-{ClusterId}` role alongside `driver` (and `admin` for a fused
|
||
central node) — e.g. `driver,cluster-SITE-A` — and its `Cluster:SeedNodes` lists only itself and its
|
||
own pair partner (see the `Roles`/`SeedNodes` rows above). Nodes in different clusters never gossip
|
||
with each other.
|
||
- **`SplitTopologyTransportValidator`** (`src/Core/ZB.MOM.WW.OtOpcUa.Cluster/`, wired at
|
||
`ValidateOnStart` alongside the other `Cluster`-family validators) fails host start on any node
|
||
carrying a `cluster-{ClusterId}` role unless it is also configured for the split-safe transports:
|
||
`MeshTransport:Mode = ClusterClient`, `Telemetry:Mode = Grpc` on a driver node, `TelemetryDial:Mode =
|
||
Grpc` on an admin node. A DPS-mode transport on a cluster-role node would try to gossip across a
|
||
partition boundary that no longer exists, so the validator fails closed rather than let commands or
|
||
telemetry silently vanish. A legacy node with no `cluster-{ClusterId}` role is exempt and may stay on
|
||
`Dps`.
|
||
- **The compiled transport-mode defaults were deliberately NOT flipped.** `MeshTransport:Mode`,
|
||
`Telemetry:Mode`, and `TelemetryDial:Mode` (documented below) all still default to `Dps` — a
|
||
blast-radius assessment found that flipping the compiled default would break every integration test
|
||
and every existing `appsettings.json` profile that leaves these keys unset. `SplitTopologyTransportValidator`
|
||
plus explicit per-deployment configuration is the guardrail, not a default flip; every node that
|
||
actually carries a `cluster-{ClusterId}` role must set the split-safe modes explicitly.
|
||
- See [`Redundancy.md` § Per-cluster meshes (Phase 6)](Redundancy.md#per-cluster-meshes-phase-6) for the
|
||
full runtime picture (one Primary per pair, the cluster-scoped `RedundancyStateActor` singleton,
|
||
`ClusterNodeAddressReconciler` own-cluster-scoping, and central's one-`ClusterClient`-per-`Cluster`
|
||
fan-out).
|
||
|
||
### `MeshTransport` (central↔node command transport)
|
||
|
||
- **Purpose:** selects how central's three command channels (`deployments`, `driver-control`, `alarm-commands`) and the node's `deployment-acks` reply reach the other side — over the mesh-wide DistributedPubSub, or over an Akka `ClusterClient` that does **not** require central and the node to share a gossip ring. The second option is what [per-cluster mesh](plans/2026-07-21-per-cluster-mesh-design.md) Phase 6 needs once the meshes split.
|
||
- **Options class:** `MeshTransportOptions` (`SectionName = "MeshTransport"`) — `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/MeshTransportOptions.cs`. Bound and validated by `AddOtOpcUaCluster(config)`; `MeshTransportOptionsValidator` runs at `ValidateOnStart`.
|
||
|
||
| Key | Type | Default | Meaning |
|
||
|---|---|---|---|
|
||
| `Mode` | string | `Dps` | `Dps` = DistributedPubSub (the transport this repo has always used). `ClusterClient` = the mesh boundary. Any other value **refuses to start**. |
|
||
| `CentralContactPoints` | string[] | `[]` | Node-side only: central node **addresses** (`akka.tcp://<system>@<host>:<port>`). Required under `ClusterClient`, ignored under `Dps`. |
|
||
| `ContactRefreshSeconds` | int | `60` | Central-side only: how often central rebuilds its ClusterClient contact set from the `ClusterNode` rows. |
|
||
|
||
**Addresses only — no actor-path suffix.** `/system/receptionist` is appended at construction time; a contact carrying `/user/` or `/system/` is rejected at startup, because a doubled suffix produces a path that resolves to nothing and drops every command in silence.
|
||
|
||
**It is a dark switch, and both sides are always wired.** The comm actors are spawned and registered with the `ClusterClientReceptionist` on every node in *both* modes, so flipping `Mode` is a config change rather than a redeploy. Under `Dps` the registration is simply idle.
|
||
|
||
**Central's contact set comes from the enabled, non-maintenance `ClusterNode` rows** — the same source Phase 1 gave the deploy ack set. A node in `MaintenanceMode` is not dialled, but note that **the contact set does not scope delivery**: a receptionist serves its whole cluster, so `SendToAll` still reaches every registered node exactly as today's DPS broadcast does. The filter is about which receptionists are worth dialling, not about who receives.
|
||
|
||
**Buffering is off by design** (`akka.cluster.client.buffer-size = 0`, set in the shipped `akka.conf`). If no receptionist is known, the command is dropped rather than queued. Akka's default of 1000 would replay a `DispatchDeployment` on reconnect — silently, on the client's own timer, at an unbounded remove from the dispatch. Do not "fix" it back to quiet a flaky test.
|
||
|
||
> **What this does *not* buy you.** It does not guarantee a node never applies a deployment the DB records as `TimedOut`. A node that missed a dispatch keeps the orphan `Applying` row the coordinator pre-created, and `DriverHostActor.Bootstrap()` replays that row on the node's next restart — live-observed applying a deployment 44 s after it was sealed `TimedOut` (see the [Phase 2 live gate](plans/2026-07-22-mesh-phase2-live-gate.md), finding B). The boot replay is the better-behaved of the two — deliberate, Warning-logged, and bounded to a restart — but it is still a replay. `buffer-size = 0` removes the *silent, timer-driven* one only.
|
||
|
||
On the docker-dev rig every node carries these keys already; flip the whole rig with `OTOPCUA_MESH_MODE=ClusterClient` at `docker compose up`.
|
||
|
||
### `ConfigSource` / `ConfigServe` (config fetch-and-cache)
|
||
|
||
- **Purpose:** [per-cluster mesh](plans/2026-07-21-per-cluster-mesh-design.md) **Phase 3**. `ConfigSource:Mode` selects where a **driver** node reads its deployed configuration: `Direct` (read `Deployment.ArtifactBlob` from central SQL, today's behaviour) or `FetchAndCache` (fetch the artifact bytes from central over a gRPC stream, verify SHA-256, cache in LocalDb, and read **only** the cache — no central-SQL config read). `ConfigServe` is the **central** (admin-role) serve side that a `FetchAndCache` node dials.
|
||
- **Options classes:** `ConfigSourceOptions` (`SectionName = "ConfigSource"`) + `ConfigServeOptions` (`SectionName = "ConfigServe"`), both in `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/`. Bound by `AddOtOpcUaCluster(config)`; `ConfigSourceOptionsValidator` runs at `ValidateOnStart`.
|
||
|
||
**`ConfigSource` (driver-side fetch):**
|
||
|
||
| Key | Type | Default | Meaning |
|
||
|---|---|---|---|
|
||
| `Mode` | string | `Direct` | `Direct` = read central SQL (unchanged). `FetchAndCache` = fetch from central over gRPC, read the LocalDb cache. Any other value **refuses to start**. |
|
||
| `CentralFetchEndpoints` | string[] | `[]` | Central artifact-gRPC base addresses, e.g. `http://central-1:4055` (h2c ⇒ the `http` scheme). Tried in order for failover. **Required** under `FetchAndCache`. |
|
||
| `ApiKey` | string | `""` | Shared bearer key; must equal central's `ConfigServe:ApiKey`. **Supply via env `ConfigSource__ApiKey` — never commit.** Required under `FetchAndCache`. |
|
||
| `FetchTimeoutSeconds` | int | `30` | Per-fetch deadline. Must be positive under `FetchAndCache`. |
|
||
|
||
**`ConfigServe` (central serve):**
|
||
|
||
| Key | Type | Default | Meaning |
|
||
|---|---|---|---|
|
||
| `GrpcListenPort` | int | `0` | Dedicated **h2c-only** Kestrel port for the artifact gRPC service. `0` = disabled (nothing bound). Must differ from the main HTTP port and from `LocalDb:SyncListenPort` (h2c cannot share a cleartext HTTP/1 port). Mapped only on admin-role nodes. |
|
||
| `ApiKey` | string | `""` | Shared bearer key the serve-side interceptor checks (constant-time, **fail-closed**: an unset key rejects every call). **Supply via env `ConfigServe__ApiKey` — never commit.** |
|
||
|
||
**It is a dark switch; the serve side is always wired.** Central maps the `DeploymentArtifactService` behind the `ConfigServeAuthInterceptor` (path-scoped, `FixedTimeEquals`, fail-closed) whenever `ConfigServe:GrpcListenPort > 0`, in both modes — so flipping a node to `FetchAndCache` is a config change, not a redeploy. A `FetchAndCache` node that cannot fetch (all endpoints down, `NotFound`, SHA mismatch) **fails the apply and keeps last-known-good** — a null fetch is "no answer," never an empty configuration (the [#485](plans/2026-07-15-v3-batch4-address-space-plan.md) contract). At boot it restores served state from the LocalDb pointer with no SQL read; an empty or unreadable cache lands Steady-with-no-revision (the first dispatch fetches), never Stale.
|
||
|
||
**The listener is dedicated h2c and coexists with the LocalDb-sync listener.** A fused admin+driver node with **both** `ConfigServe:GrpcListenPort` and `LocalDb:SyncListenPort` set re-binds its existing HTTP surface exactly once and adds both dedicated HTTP/2-only ports on top — see the same Kestrel-takeover caveat under [`LocalDb`](#localdb-pair-local-config-cache--driver-role-only).
|
||
|
||
**`RevisionHash == SHA-256(artifact bytes)` is load-bearing.** `ConfigComposer` computes the deployment's revision hash as the lowercase-hex SHA-256 of the exact artifact bytes, so the node verifies a fetch against the `RevisionHash` the dispatch already carries — no hash rides in the proto. If a future change makes the revision hash anything else, every fetch fails verification.
|
||
|
||
On the docker-dev rig the central pair carries `ConfigServe` (port `4055`, dev key) and stays `Direct`; the four site nodes carry `ConfigSource` (endpoints + matching key) defaulting to `Direct`. Flip only the site nodes with `OTOPCUA_CONFIG_MODE=FetchAndCache` at `docker compose up` — central keeps SQL and models the target topology.
|
||
|
||
**`FetchAndCache` is mandatory on a driver-only node — per-cluster mesh Phase 4.** `Program.cs` now
|
||
registers `AddOtOpcUaConfigDb` only when the node carries the `admin` role, so a **driver-only**
|
||
node (`Cluster:Roles` has `driver`, not `admin`) holds **no `ConfigDb` connection string at all** —
|
||
there is nothing for `Direct` to read. `ConfigSourceOptionsValidator` enforces this at
|
||
`ValidateOnStart`: a driver-only node whose `ConfigSource:Mode` is `Direct` **fails host start**
|
||
with an error naming the fix (`ConfigSource:Mode=FetchAndCache`). A **fused** `admin,driver` node
|
||
still owns its ConfigDb via the admin role and may stay `Direct` — the validator only fires on
|
||
`driver` without `admin`. On a driver-only node the deployed configuration, the scripted-alarm Part
|
||
9 condition state (`LocalDbAlarmConditionStateStore`, replacing the ConfigDb-backed
|
||
`EfAlarmConditionStateStore`), and the alarm store-and-forward buffer all live in the node's
|
||
LocalDb; central remains the system of record for deploy acks (`ConfigPublishCoordinator.
|
||
PersistNodeAck`) and LDAP group→role DB grants (a driver-only node maps LDAP groups to roles from
|
||
the `Security:Ldap:GroupToRole` appsettings baseline only — see `docs/Redundancy.md` and
|
||
`CLAUDE.md` §"Phase 4" for the full picture, including the client-visible ServiceLevel change).
|
||
|
||
### `Telemetry` / `TelemetryDial` (live telemetry transport)
|
||
|
||
- **Purpose:** [per-cluster mesh](plans/2026-07-21-per-cluster-mesh-design.md) **Phase 5**. Selects how
|
||
the four live-observability channels (`alerts`, `script-logs`, `driver-health`,
|
||
`driver-resilience-status`) reach central: `Dps` (DistributedPubSub — the four existing SignalR
|
||
bridges, today's behaviour) or `Grpc` (a driver node hosts one gRPC server-streaming contract over a
|
||
dedicated h2c port; central dials in and feeds the same in-process sinks). `Telemetry` is the
|
||
**node/serve** side; `TelemetryDial` is the **central/dial** side.
|
||
- **Options classes:** `TelemetryOptions` (`SectionName = "Telemetry"`) + `TelemetryDialOptions`
|
||
(`SectionName = "TelemetryDial"`), both in `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/`. Bound by
|
||
`AddOtOpcUaCluster(config)`; their validators run at `ValidateOnStart`.
|
||
|
||
**`Telemetry` (node-side, serve):**
|
||
|
||
| Key | Type | Default | Meaning |
|
||
|---|---|---|---|
|
||
| `Mode` | string | `Dps` | `Dps` = no change, node just keeps publishing DPS as it always has. `Grpc` = central-ingest hint (see `TelemetryDial:Mode` below — this key does not gate anything on the node itself). Any other value **refuses to start**. |
|
||
| `GrpcListenPort` | int | `0` | Dedicated **h2c-only** Kestrel port for the telemetry stream service. `0` = disabled (nothing bound). The node hosts the server whenever this is `> 0`, **regardless of `Mode`**. |
|
||
| `ApiKey` | string | `""` | Shared bearer key the serve-side interceptor checks (constant-time, **fail-closed**: an unset key rejects every call). **Supply via env `Telemetry__ApiKey` — never commit.** |
|
||
|
||
**`TelemetryDial` (central-side, dial):**
|
||
|
||
| Key | Type | Default | Meaning |
|
||
|---|---|---|---|
|
||
| `Mode` | string | `Dps` | `Dps` = the four DPS SignalR bridges (alert, script-log, driver-status, resilience) subscribe as today. `Grpc` = those four bridges are **not spawned**; the `TelemetryDialSupervisor` actor dials every driver node instead and feeds the same sinks. Any other value **refuses to start**. |
|
||
| `ApiKey` | string | `""` | Shared bearer key; must equal the nodes' `Telemetry:ApiKey`. **Supply via env `TelemetryDial__ApiKey` — never commit.** Required under `Grpc`. |
|
||
| `ContactRefreshSeconds` | int | `60` | How often central rebuilds its dial-target set from the `ClusterNode` rows (`Host` + `GrpcPort`). |
|
||
| `CallTimeoutSeconds` | int | `30` | Per-call deadline for the streaming RPC's keepalive/health bookkeeping. |
|
||
|
||
**It is a dark switch; the node always hosts.** Unlike `ConfigServe`/`MeshTransport`, where the serve
|
||
side is gated on a role, the telemetry server binds whenever `Telemetry:GrpcListenPort > 0` — hosting
|
||
is unconditional on a driver-role node, in both modes, because the node also always emits into its
|
||
local hub and always publishes DPS. Only `TelemetryDial:Mode` on central actually changes behaviour:
|
||
which of the two already-always-available sources (DPS bridges vs. the gRPC dial supervisor) feeds the
|
||
AdminUI sinks. Flipping either side is a config change plus restart, not a redeploy — the same
|
||
discipline as `MeshTransport:Mode` and `ConfigSource:Mode`.
|
||
|
||
**A driver-role node with `Telemetry:Mode=Grpc` must set `GrpcListenPort > 0`** (nothing to serve
|
||
otherwise) — enforced by `TelemetryOptionsValidator`. **`Grpc` mode on either side requires a
|
||
non-empty `ApiKey`** — fail-closed, same as `ConfigServe`/`LocalDb:Replication`.
|
||
|
||
**Three DPS observability topics stay out of scope for this transport, on purpose:**
|
||
`redundancy-state` (pair-local, built from `Cluster.State`, stays on DPS in both modes — see
|
||
[`Redundancy.md`](Redundancy.md#command-transport-centralnode)), `fleet-status` (central-internal;
|
||
`Fleet.razor` polls the Config DB, not any stream), and `deployment-acks` (already rides the Phase 2
|
||
`ClusterClient` transport as a command-plane reply). See
|
||
[`docs/Telemetry.md`](Telemetry.md) for the full architecture, the reconnect/generation-stamped
|
||
dialer story, and the rationale for the node-hosts/central-dials inversion.
|
||
|
||
On the docker-dev rig every driver-role node carries `Telemetry:GrpcListenPort` (`4056`) + a
|
||
committed dev key, and central carries the matching `TelemetryDial:ApiKey`; both `Mode` keys stay
|
||
unset (⇒ `Dps`) until the rig is flipped with `Telemetry__Mode=Grpc` / `TelemetryDial__Mode=Grpc` at
|
||
`docker compose up`.
|
||
|
||
### `ConnectionStrings` → `ConfigDb`
|
||
|
||
- **Purpose:** the central Config DB connection string. **Required on admin-role nodes only** — `Program.cs` calls `AddOtOpcUaConfigDb` when `hasAdmin` (per-cluster mesh Phase 4). A driver-only node holds no `ConfigDb` connection string; it fetches its configuration via `ConfigSource:Mode=FetchAndCache` instead (see above). A fused `admin,driver` node still requires it.
|
||
- **Bound by:** `AddOtOpcUaConfigDb(config)` (`src/Core/ZB.MOM.WW.OtOpcUa.Configuration/ServiceCollectionExtensions.cs`). The connection-string name constant is `ConnectionStringName = "ConfigDb"`, read via `configuration.GetConnectionString("ConfigDb")`. If absent, startup throws with a message pointing to either `appsettings.json` or the `OTOPCUA_CONFIG_CONNECTION` env var.
|
||
- **Shape:** standard `ConnectionStrings:ConfigDb` SQL Server connection string. There is no checked-in default in the thin `appsettings*.json` — supply it per environment.
|
||
|
||
The Config DB itself (the EF Core `OtOpcUaConfigDbContext`, entities, draft/publish generations, `NodeAcl`, `LdapGroupRoleMapping`, migrations) is the durable home for the fleet's drivers, UNS hierarchy, ACLs, and audit log. For the **full schema** see [`docs/v2/config-db-schema.md`](v2/config-db-schema.md). This page does not duplicate it.
|
||
|
||
### Galaxy / MxAccess driver config (`DriverConfig` JSON, not `appsettings`)
|
||
|
||
The Galaxy/MxAccess connection settings are **not an `appsettings` section.** They are driver-instance options stored in the `DriverConfig` JSON column of the Config DB (edited via the Admin UI), bound to `GalaxyDriverOptions` (`src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Contracts/GalaxyDriverOptions.cs`, namespace `...Driver.Galaxy.Config`). It decomposes into nested records:
|
||
|
||
| Record | Key fields (default) | Meaning |
|
||
|---|---|---|
|
||
| `GalaxyGatewayOptions` (`Gateway`) | `Endpoint`; `ApiKeySecretRef`; `UseTls` (`true`); `CaCertificatePath` (`null`); `ConnectTimeoutSeconds` (`10`); `DefaultCallTimeoutSeconds` (`30`); `StreamTimeoutSeconds` (`0` = unlimited) | mxaccessgw gateway connection. `ApiKeySecretRef` supports `env:NAME` / `file:PATH` / `dev:KEY` / literal forms (resolved at `InitializeAsync`); prefer `env:`/`file:` in production. Never store a cleartext key. |
|
||
| `GalaxyMxAccessOptions` (`MxAccess`) | `ClientName`; `PublishingIntervalMs` (`1000`); `WriteUserId` (`0` = anonymous); `EventPumpChannelCapacity` (`50000`) | MXAccess client identity + tuning. `ClientName` **must be unique per OtOpcUa instance** (redundancy pairs enforce this). |
|
||
| `GalaxyRepositoryOptions` (`Repository`) | `DiscoverPageSize` (`5000`); `WatchDeployEvents` (`true`) | Galaxy Repository browse paging + deploy-event watching. |
|
||
| `GalaxyReconnectOptions` (`Reconnect`) | `InitialBackoffMs` (`500`); `MaxBackoffMs` (`30000`); `ReplayOnSessionLost` (`true`) | In-driver reconnect-supervisor backoff. |
|
||
| (top-level) | `ProbeTimeoutSeconds` (`30`, range 1–60) | AdminUI Test-Connect probe timeout. |
|
||
|
||
> The `OTOPCUA_GALAXY_*` environment variables that v1's in-process `Galaxy.Host` consumed **no longer live in this repo** — they moved into the separately-installed mxaccessgw gateway's own config (see the v1 archive pointer in `docs/README.md` and the Galaxy overview at [`docs/drivers/Galaxy.md`](drivers/Galaxy.md)). The only Galaxy connection secret this repo touches is the gateway API key via `ApiKeySecretRef` above.
|
||
|
||
### Historian config (HistorianGateway)
|
||
|
||
The historian backend is the external **`ZB.MOM.WW.HistorianGateway`** sidecar, consumed as the
|
||
`ZB.MOM.WW.HistorianGateway.Client` gRPC package (the retired Wonderware TCP sidecar is documented at
|
||
[`docs/drivers/Historian.Wonderware.md`](drivers/Historian.Wonderware.md)). The OtOpcUa host reads three
|
||
appsettings sections — `ServerHistorian` (read path + gateway connection), `ContinuousHistorization`
|
||
(FasterLog outbox + recorder draining to `WriteLiveValues`), and `AlarmHistorian` (the store-and-forward
|
||
alarm sink draining to `SendEvent`; its buffer lives in the node's `LocalDb:Path` database, not a
|
||
file of its own). The gateway connection (endpoint / key / TLS) lives **only** in
|
||
`ServerHistorian`; the other two sections source it from there.
|
||
|
||
The gateway API key is supplied via the environment variable **`ServerHistorian__ApiKey`** — never committed
|
||
to config. The target gateway must run `RuntimeDb:Enabled=true` + `RuntimeDb:EventReadsEnabled=true`, and the
|
||
key must carry the scopes `historian:read`, `historian:write`, `historian:tags:write`. See
|
||
[`docs/Historian.md`](Historian.md) for the full key reference, the migration note (old Wonderware keys →
|
||
gateway keys), and the deployment prerequisites.
|
||
|
||
### `LocalDb` (pair-local config cache — driver role only)
|
||
|
||
Driver-role nodes keep a consolidated **`ZB.MOM.WW.LocalDb`** SQLite database that caches the deployed
|
||
configuration artifact (chunked) so the node can **boot from cache when central SQL is unreachable**. The
|
||
section is bound by `AddOtOpcUaLocalDb` (`Host/Configuration/LocalDbRegistration.cs`) inside the `hasDriver`
|
||
branch — **admin-only nodes never register it, and never require `LocalDb:Path`**. Storage is unconditional;
|
||
replication stays inert until the sync port + peer are set.
|
||
|
||
| Key | Default | Meaning |
|
||
|---|---|---|
|
||
| `LocalDb:Path` | `./data/otopcua-localdb.db` | SQLite file path. `ValidateOnStart`-required once `AddZbLocalDb` runs (driver nodes only). In containers put it on a durable volume. |
|
||
| `LocalDb:SyncListenPort` | `0` | `0` = replication off (no sync listener). A non-zero port binds a dedicated **h2c** sync listener; set the **same** port on both pair nodes. Setting it makes the Host add an explicit Kestrel `Listen*` and re-bind the primary HTTP port (an explicit `Listen*` otherwise discards `ASPNETCORE_URLS`). |
|
||
| `LocalDb:Replication:PeerAddress` | `""` | The peer this node **dials** (`http://<peer>:<port>`). Set on the initiator only; leave empty on the passive node. The stream is bidirectional. |
|
||
| `LocalDb:Replication:ApiKey` | `""` | Bearer token for the sync stream. **Fail-closed:** no key ⇒ the passive endpoint refuses everything; a mismatch ⇒ the pair silently stops converging. **Must be byte-identical on both nodes.** Supply via `${secret:...}` / env — never a cleartext literal in production. |
|
||
| `LocalDb:Replication:MaxBatchSize` | (library default) | Rows per replication batch — **row-count-only** against gRPC's 4 MB cap. `16` for the artifact cache (chunk rows ≈171 KB, so 16 × 171 KB ≈ 2.7 MB). |
|
||
|
||
Replication is **default-OFF** across the fleet; it is enabled per-pair as an opt-in. See
|
||
[`docs/operations/2026-07-20-localdb-pair-replication.md`](operations/2026-07-20-localdb-pair-replication.md)
|
||
for the enablement runbook (ApiKey handling, stop/start-together rule, tombstone-retention window, and the
|
||
never-`sqlite3`-a-live-WAL-DB inspection rules) and the "pair-local config cache" section of
|
||
[`docs/Redundancy.md`](Redundancy.md) for what boot-from-cache does and does not cover.
|
||
|
||
---
|
||
|
||
## Environment variables
|
||
|
||
All names are read in this repo's source via `Environment.GetEnvironmentVariable(...)` unless noted otherwise. Defaults shown are the in-source fallbacks.
|
||
|
||
### Host / cluster / Config DB
|
||
|
||
| Variable | Read by | Effect / default |
|
||
|---|---|---|
|
||
| `OTOPCUA_ROLES` | `src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs` (`RoleParser.Parse`) and `AddOtOpcUaCluster`'s `AkkaClusterOptions` post-configure (`RoleParser.ResolveEffectiveRoles`) | Comma-separated cluster roles for the node (`admin`, `driver`, `dev`, `cluster-{ClusterId}`). Drives the conditional wiring and the per-role appsettings overlay, and — when `Cluster:Roles` is empty — the node's Akka member roles too. See the note under `Cluster:Roles` for why the two sources are not interchangeable. |
|
||
| `OTOPCUA_CONFIG_CONNECTION` | `src/Core/ZB.MOM.WW.OtOpcUa.Configuration/DesignTimeDbContextFactory.cs` (design-time / `dotnet ef` only) | Read at **design time** by `DesignTimeDbContextFactory.cs` for `dotnet ef` migrations. At **runtime** the server resolves the connection string from `ConnectionStrings:ConfigDb` (env form: `ConnectionStrings__ConfigDb`) via `configuration.GetConnectionString("ConfigDb")` in `ServiceCollectionExtensions.cs` — `OTOPCUA_CONFIG_CONNECTION` appears there only as a hint in an error message, not via `GetEnvironmentVariable`. No credential is embedded in source. |
|
||
| `ASPNETCORE_ENVIRONMENT` | ASP.NET host builder (framework) | Selects `appsettings.{Environment}.json` (e.g. `Development`). |
|
||
|
||
### Historian (`ServerHistorian__ApiKey`)
|
||
|
||
The retired Wonderware sidecar's `OTOPCUA_HISTORIAN_*` environment variables are **gone** — no source reads
|
||
them anymore. The historian backend is now the external HistorianGateway, configured through the
|
||
`ServerHistorian` / `ContinuousHistorization` / `AlarmHistorian` appsettings sections (above). The single
|
||
historian secret this repo reads from the environment is the gateway API key:
|
||
|
||
| Variable | Effect / default |
|
||
|---|---|
|
||
| `ServerHistorian__ApiKey` | The HistorianGateway peppered-HMAC key (`histgw_<id>_<secret>`) sent as `Authorization: Bearer`. Supply via environment — **never commit**. Required when `ServerHistorian:Enabled=true`. |
|
||
|
||
### Driver integration-test / fixture sim endpoints
|
||
|
||
These are consumed by the driver **integration-test fixtures** (under `tests/Drivers/...IntegrationTests/`), not by the production server. Each overrides the simulator endpoint a fixture TCP-probes; defaults point at the shared Docker host `10.100.0.35` (see `CLAUDE.md` Docker Workflow).
|
||
|
||
| Variable | Read by (fixture) | Default |
|
||
|---|---|---|
|
||
| `MODBUS_SIM_ENDPOINT` | `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus.IntegrationTests/ModbusSimulatorFixture.cs` | `10.100.0.35:5020` |
|
||
| `AB_SERVER_ENDPOINT` | `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.AbCip.IntegrationTests/AbServerFixture.cs` | `10.100.0.35:44818` |
|
||
| `S7_SIM_ENDPOINT` | `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.S7.IntegrationTests/Snap7ServerFixture.cs` | `10.100.0.35:1102` (non-privileged; not S7-standard 102) |
|
||
| `OPCUA_SIM_ENDPOINT` | `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.OpcUaClient.IntegrationTests/OpcPlcFixture.cs` | `opc.tcp://10.100.0.35:50000` |
|
||
| `OTOPCUA_FOCAS_SIM_ENDPOINT` | `tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.FOCAS.IntegrationTests/FocasSimFixture.cs` | `localhost:8193` |
|
||
|
||
> Additional harness/parity/soak env vars (`OTOPCUA_FOCAS_*`, `OTOPCUA_PARITY_*`, `OTOPCUA_SOAK_*`, `OTOPCUA_HARNESS_USE_SQL`) exist only in the test/parity/soak harnesses, not in production source, and are out of scope for this reference.
|
||
|
||
---
|
||
|
||
## See also
|
||
|
||
- [`security.md`](security.md) — transport security, OPC UA authentication, LDAP (`Security:Ldap`), data-plane ACLs, control-plane roles.
|
||
- [`Redundancy.md`](Redundancy.md) — the `Cluster` section in the context of warm/hot redundancy, ServiceLevel, peer discovery.
|
||
- [`Telemetry.md`](Telemetry.md) — the `Telemetry`/`TelemetryDial` gRPC live-telemetry stream: direction, dark switch, the four migrated channels + three deferrals, auth, reconnect story.
|
||
- [`ServiceHosting.md`](ServiceHosting.md) — role-based host wiring and `OTOPCUA_ROLES`.
|
||
- [`docs/drivers/Galaxy.md`](drivers/Galaxy.md) — Galaxy/MxAccess driver overview.
|
||
- [`docs/v2/config-db-schema.md`](v2/config-db-schema.md) — the full Config DB schema.
|